Back to series
· 12 min read

Building an AI Chat App from Scratch: From Frontend Interaction to Backend Model Serving


1. Overall Architecture of an AI Chat Application

A complete AI chat application is not just an input box plus a large model API. It is in fact an application system encompassing user interaction, session management, model invocation, context organization, and security controls.

1.1 Core Components

A typical AI chat application can be broken down into the following modules:

Module Core Responsibility
Frontend interaction layer Message display, input box, session list, streaming rendering
Backend API layer Receive requests, authentication, parameter validation, business orchestration
Session management layer Create sessions, save messages, maintain context
Model adaptation layer Connect to different large model services, unify requests and responses
Model service layer Call models such as GPT, Claude, Gemini, Qwen, and DeepSeek
Data storage layer Store users, sessions, messages, and configuration
Security governance layer Permission validation, content safety, rate limiting, auditing
Observability layer Record requests, tokens, latency, errors, and cost

1.2 System Architecture

flowchart TB
    A["User"] --> B["Frontend App"]
    B --> C["Backend API Gateway"]
    C --> D["Authentication & Permission Validation"]
    D --> E["Conversation Business Service"]
    E --> F["Session Management"]
    E --> G["Context Construction"]
    E --> H["Model Adaptation Layer"]
    H --> I["LLM API"]
    F --> J["Database"]
    E --> K["Cache"]
    H --> L["Logging & Monitoring"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

1.3 The Full Path of a Single Chat Request

After a user sends a message, the system needs to complete several steps:

  1. The frontend collects the user's input.
  2. The backend verifies identity and request parameters.
  3. Retrieve the current session and historical messages.
  4. Build the context required by the model.
  5. Call the large model API.
  6. Receive the model's response.
  7. Persist the response.
  8. Return the result to the frontend.
  9. The frontend updates the chat interface.
sequenceDiagram
    autonumber
    participant U as User
    participant F as Frontend
    participant B as Backend
    participant D as Database
    participant M as LLM

    U->>F: Enter a question
    F->>B: Send chat request
    B->>B: Authentication and parameter validation
    B->>D: Query session and historical messages
    D-->>B: Return historical messages
    B->>B: Build context
    B->>M: Send model request
    M-->>B: Return model response
    B->>D: Save user message and model reply
    B-->>F: Return response
    F-->>U: Display the answer

Production systems typically use streaming responses, so model generation and frontend display do not necessarily wait until all content has been generated before starting.


2. Technology Selection and Project Structure

To build an application from scratch, the first step is to decide on the tech stack.

2.1 Recommended Tech Stack

Technology Layer Options Primary Responsibility
Frontend React, Vue, Next.js Chat interface and interaction
Backend Node.js, Python, Java, Go API and business logic
Model SDK Official SDK, HTTP Client Call model services
Database PostgreSQL, MySQL Persist users and messages
Cache Redis Session caching, rate limiting, temporary state
Real-time Communication SSE, WebSocket Streaming responses
Deployment Docker, Kubernetes Application runtime and scaling
Monitoring OpenTelemetry, Prometheus Metrics, tracing, and logs

For the first version, we recommend using:

  • React: build the chat interface.
  • Node.js: implement the backend API.
  • PostgreSQL: store users, conversations, and messages.
  • Redis: use later for caching and rate limiting.
  • LLM API: provide model inference capabilities.

There is no need to introduce microservices, message queues, and complex agent frameworks from the start. Build a well-structured monolith first, then split it according to business scale.

2.2 Project Directory Structure

A relatively clean project can be organized as follows:

ai-chat-app/
├── frontend/
│   ├── src/
│   │   ├── components/
│   │   │   ├── ChatWindow.tsx
│   │   │   ├── MessageList.tsx
│   │   │   ├── MessageItem.tsx
│   │   │   ├── ChatInput.tsx
│   │   │   └── ConversationList.tsx
│   │   ├── hooks/
│   │   │   └── useChat.ts
│   │   ├── services/
│   │   │   └── chatApi.ts
│   │   ├── types/
│   │   │   └── chat.ts
│   │   └── App.tsx
│   └── package.json
│
├── backend/
│   ├── src/
│   │   ├── routes/
│   │   │   └── chat.ts
│   │   ├── services/
│   │   │   ├── chatService.ts
│   │   │   ├── modelService.ts
│   │   │   └── contextService.ts
│   │   ├── adapters/
│   │   │   └── openaiAdapter.ts
│   │   ├── repositories/
│   │   │   ├── conversationRepository.ts
│   │   │   └── messageRepository.ts
│   │   ├── middleware/
│   │   │   ├── auth.ts
│   │   │   └── rateLimit.ts
│   │   ├── config/
│   │   │   └── model.ts
│   │   └── app.ts
│   └── package.json
│
├── database/
│   └── migrations/
│
├── .env
└── README.md

Here are several important design principles:

  • Routes handle HTTP requests and do not directly carry complex business logic.
  • Services handle conversation orchestration and model invocation.
  • Adapters isolate protocol differences between different model providers.
  • Repositories handle database access.
  • Middleware handles cross-cutting concerns such as authentication and rate limiting.

This layering prevents business code from becoming tightly coupled to model providers and database implementations.


3. Designing the Conversation Data Model

An AI chat application first needs to solve a fundamental problem: how to store users, conversations, and messages?

You cannot simply store an entire conversation as a single string. As conversations grow, you need to support paginated queries, history restoration, message status, model tracking, and data analysis.

3.1 Core Data Entities

We recommend designing at least three core entities:

User

Field Type Description
id UUID Unique user identifier
username VARCHAR Username
email VARCHAR Email
created_at TIMESTAMP Created at

Conversation

Field Type Description
id UUID Unique conversation identifier
user_id UUID Owning user
title VARCHAR Conversation title
model VARCHAR Currently used model
system_prompt TEXT Conversation-level system instruction, optional
created_at TIMESTAMP Created at
updated_at TIMESTAMP Updated at

Message: Message

Field Type Description
id UUID Unique message identifier
conversation_id UUID Owning conversation
role VARCHAR Message role
content TEXT / JSONB Message content
status VARCHAR Message status
input_tokens INTEGER Input token count
output_tokens INTEGER Output token count
model VARCHAR Model actually used
created_at TIMESTAMP Created at

3.2 Database Relationships

erDiagram
    USER ||--o{ CONVERSATION : owns
    CONVERSATION ||--o{ MESSAGE : contains

    USER {
        uuid id PK
        string username
        string email
        timestamp created_at
    }

    CONVERSATION {
        uuid id PK
        uuid user_id FK
        string title
        string model
        timestamp created_at
        timestamp updated_at
    }

    MESSAGE {
        uuid id PK
        uuid conversation_id FK
        string role
        jsonb content
        string status
        integer input_tokens
        integer output_tokens
        timestamp created_at
    }

3.3 Why can Message Content use JSONB?

If the application only supports plain text, using TEXT is sufficient.

However, if multimodal messages are supported later, message content may include:

  • Text.
  • Image references.
  • Audio content.
  • File references.
  • Tool calls.
  • Tool execution results.

Therefore, structured JSON can be used to store message content.

For example:

{
  "type": "text",
  "text": "请解释一下 Transformer 的工作原理"
}

A message containing an image can be abstracted as:

{
  "parts": [
    {
      "type": "text",
      "text": "请分析这张图片"
    },
    {
      "type": "image",
      "url": "https://example.com/image.png"
    }
  ]
}

Note that the message structure in the database is not necessarily the same as the request structure of the model provider. The application should define its own message model internally, and then the model adapter layer converts it into the provider protocol.

3.4 Role Design

The most basic conversation roles include:

Role Meaning
system System-level instruction
user User message
assistant Model reply
tool Tool execution result

Different model APIs may support roles and message structures differently; you cannot assume that all providers fully follow the same protocol.

For tool calls, you also need to save the tool call ID, tool name, parameters, execution status, and result, so that the complete lifecycle of a tool call can be traced.


IV. Integrating with an LLM API

The model API is the core capability source for AI chat applications.

The application itself is responsible for user interaction and business logic, while the LLM performs reasoning based on the input context and generates responses.

4.1 Configure the Model Service

First, prepare the following:

  • API Base URL.
  • API Key.
  • Model ID.
  • Request timeout.
  • Maximum output tokens.
  • Sampling parameters such as Temperature.

Configuration can be managed through environment variables:

MODEL_API_KEY=your_api_key
MODEL_BASE_URL=https://api.example.com/v1
MODEL_NAME=your-model-name

In production environments, do not commit real API keys to Git or write them into frontend code.

4.2 Calling the Model from the Backend

The following demonstrates using JavaScript and an OpenAI SDK-style interface. In actual use, you need to install the corresponding SDK and adjust the configuration and parameters according to the protocol of the chosen provider.

npm install openai

Model service wrapper:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.MODEL_API_KEY,
  baseURL: process.env.MODEL_BASE_URL,
});

export async function generateAnswer(
  messages: Array<{
    role: "system" | "user" | "assistant";
    content: string;
  }>
) {
  const response = await client.chat.completions.create({
    model: process.env.MODEL_NAME!,
    messages,
    temperature: 0.7,
    max_tokens: 2048,
  });

  return {
    content: response.choices[0]?.message?.content ?? "",
    usage: response.usage,
    model: response.model,
  };
}

This uses a Chat Completions-style API example, which does not mean that all model providers support exactly the same fields and parameters.

4.3 Why Encapsulate a Model Adapter?

If business code directly calls a model provider's SDK, switching models later will require many changes.

For example:

ChatService
    ├── Directly call OpenAI SDK
    ├── Directly process OpenAI responses
    └── Directly depend on OpenAI streaming events

This design couples the business layer to a specific provider.

A more reasonable approach is to define a unified interface:

export interface ModelAdapter {
  generate(
    messages: ChatMessage[],
    options?: ModelOptions
  ): Promise<ModelResponse>;

  stream(
    messages: ChatMessage[],
    options?: ModelOptions
  ): AsyncIterable<ModelStreamEvent>;
}

Then implement adapters for different providers separately:

ModelAdapter
    ├── OpenAIAdapter
    ├── AnthropicAdapter
    ├── GeminiAdapter
    ├── QwenAdapter
    └── DeepSeekAdapter

The business layer depends only on the unified interface, not directly on any provider's SDK.

Pay special attention: a unified interface should not smooth over all capability differences. Parameters, tool calls, or multimodal capabilities that a model does not support should be explicitly declared, rejected, or degraded, rather than silently ignored.


V. Implementing the Core Chat API

After model integration is complete, you need to provide stable business APIs for the frontend.

5.1 API Design

It is recommended to implement the following APIs first:

HTTP Method API Purpose
POST /api/conversations Create conversation
GET /api/conversations List conversations
GET /api/conversations/:id Get conversation details
PATCH /api/conversations/:id Update conversation information
DELETE /api/conversations/:id Delete conversation
GET /api/conversations/:id/messages Get message history
POST /api/chat Send a message
POST /api/chat/stream Send a message as a stream

These endpoints are business APIs internal to the application; they do not need to directly expose the model provider's API.

5.2 Conversation Request Structure

The frontend can send:

{
  "conversation_id": "conv_123",
  "message": "什么是 Transformer?",
  "model": "your-model-name"
}

The backend needs to validate:

  • Whether the current user is logged in.
  • Whether the current conversation belongs to the current user.
  • Whether the message content is empty.
  • Whether the message length exceeds the limit.
  • Whether the model is within the allowed range.
  • Whether the current user still has available quota.

Do not directly trust the user_id, model name, or permission information submitted by the frontend. User identity should be obtained from verified authentication credentials.

5.3 Conversation Service Implementation

The conversation service is responsible for connecting conversations, messages, and models:

export async function chat(
  userId: string,
  conversationId: string,
  userContent: string
) {
  const conversation =
    await conversationRepository.findById(conversationId);

  if (!conversation || conversation.userId !== userId) {
    throw new Error("Conversation not found");
  }

  const history =
    await messageRepository.findRecentMessages(conversationId);

  const userMessage =
    await messageRepository.create({
      conversationId,
      role: "user",
      content: userContent,
      status: "completed",
    });

  const context = await contextService.build({
    conversation,
    history,
    userContent,
  });

  const result = await modelService.generate({
    model: conversation.model,
    messages: context,
  });

  const assistantMessage =
    await messageRepository.create({
      conversationId,
      role: "assistant",
      content: result.content,
      status: "completed",
      inputTokens: result.usage?.prompt_tokens,
      outputTokens: result.usage?.completion_tokens,
    });

  return {
    userMessage,
    assistantMessage,
    usage: result.usage,
  };
}

This is a simplified example for illustrating responsibility boundaries. A production implementation also needs to handle model call failures, timeouts, concurrent messages, transaction consistency, request idempotency, and message state recovery.

Pay particular attention to this: if the user message has already been saved but the model call fails, the conversation must not be left in an unexplainable state. Message states can be designed as follows:

  • pending: waiting to be processed.
  • streaming: currently generating.
  • completed: generation completed.
  • failed: generation failed.
  • cancelled: actively cancelled by the user.

6. Implementing the Frontend Chat Interface

The frontend is the main entry point for user interaction with the AI.

A basic chat page usually includes four areas:

  1. Conversation list on the left.
  2. Conversation title and model information at the top.
  3. Message display area in the middle.
  4. Message input area at the bottom.

6.1 Page Component Design

ChatPage
├── Sidebar
│   ├── NewConversationButton
│   └── ConversationList
│       └── ConversationItem
│
├── ChatMain
│   ├── ChatHeader
│   ├── MessageList
│   │   └── MessageItem
│   └── ChatInput
│       ├── TextArea
│       ├── AttachmentButton
│       └── SendButton
│
└── ModelSelector

Components should communicate through clear data structures and callbacks as much as possible, avoiding placing all conversation state in a single page component.

6.2 Message Data Structure

The frontend can define:

export interface ChatMessage {
  id: string;
  conversationId: string;
  role: "user" | "assistant" | "tool";
  content: string;
  status: "pending" | "streaming" | "completed" | "failed";
  createdAt: string;
}

The message list needs to support:

  • Different styles for user messages and AI messages.
  • Markdown rendering.
  • Code highlighting.
  • Copying messages.
  • Regenerating messages.
  • Message feedback.
  • Loading and error states.
  • Scrolling and performance optimization for long messages.

For Markdown rendering, untrusted content must be handled securely to avoid injecting model output directly into the page as unsanitized HTML.

6.3 Conversation State Management

A chat application not only needs to store the message array but also needs to manage request state.

Typical states include:

interface ChatState {
  conversationId: string | null;
  messages: ChatMessage[];
  input: string;
  isLoading: boolean;
  isStreaming: boolean;
  error: string | null;
}

Where:

  • isLoading: the request is being prepared or is waiting for a response.
  • isStreaming: the model is continuously outputting content.
  • error: the error that occurred in the current request.

These two states should not simply be merged into a single boolean value. The application may need to distinguish between phases such as waiting for the first Token, receiving content, completion, and failure.


7. Implementing Streaming Responses

If the application waits until the model has generated the complete answer before returning it each time, users will have to wait for a relatively long time.

Streaming can transmit the content generated by the model to the frontend incrementally, significantly improving the interactive experience.

7.1 Basic Principles of Streaming Responses

Taking SSE as an example:

  • The backend initiates a streaming request to the model service.
  • The model service continuously produces events or content fragments.
  • The backend reads upstream data.
  • The backend converts the content into internal application events.
  • The frontend continuously receives events and updates the message content.
flowchart TB
    A["User sends message"] --> B["Backend establishes streaming request to model"]
    B --> C["Model generates Token"]
    C --> D["Model returns streaming data"]
    D --> E["Backend parses data chunks"]
    E --> F["Convert to application events"]
    F --> G["SSE pushes to frontend"]
    G --> H["Frontend appends message content"]
    H --> I{"Is it finished?"}
    I -->|"No"| C
    I -->|"Yes"| J["End response and save message"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

7.2 Backend Streaming Interface

The following uses Node.js's Web API ReadableStream and SSE as an example to demonstrate how the application layer sends events to the browser. modelService.stream() is the abstract method of the unified model adapter layer described earlier.

export async function streamChat(
  userId: string,
  conversationId: string,
  content: string
) {
  await conversationService.assertOwnedBy(
    conversationId,
    userId
  );

  const context = await contextService.buildForNewMessage(
    conversationId,
    content
  );

  return new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();

      const send = (event: string, data: unknown) => {
        controller.enqueue(
          encoder.encode(
            `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`
          )
        );
      };

      try {
        send("start", {
          conversationId,
        });

        for await (const chunk of modelService.stream({
          messages: context,
        })) {
          if (chunk.type === "text_delta") {
            send("delta", {
              content: chunk.text,
            });
          }
        }

        send("done", {
          conversationId,
        });
      } catch (error) {
        send("error", {
          message: "Model generation failed",
        });
      } finally {
        controller.close();
      }
    },
  });
}

Message persistence, cancellation signals, disconnect detection, and resource cleanup are omitted here. A production implementation needs to fold these into the complete request lifecycle.

7.3 Receiving SSE on the Frontend

The browser can use fetch to read a streaming response:

async function sendStreamMessage(
  conversationId: string,
  message: string,
  onDelta: (text: string) => void
) {
  const response = await fetch("/api/chat/stream", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      conversationId,
      message,
    }),
  });

  if (!response.ok || !response.body) {
    throw new Error("Stream request failed");
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();

    if (done) break;

    buffer += decoder.decode(value, {
      stream: true,
    });

    const events = buffer.split("\n\n");
    buffer = events.pop() ?? "";

    for (const event of events) {
      const dataLine = event
        .split("\n")
        .find((line) => line.startsWith("data: "));

      if (!dataLine) continue;

      const data = JSON.parse(
        dataLine.slice(6)
      );

      if (data.content) {
        onDelta(data.content);
      }
    }
  }
}