1. Overall Architecture of an AI Chat Application
A complete AI chat application is not just an input box plus a large model API. It is in fact an application system encompassing user interaction, session management, model invocation, context organization, and security controls.
1.1 Core Components
A typical AI chat application can be broken down into the following modules:
| Module | Core Responsibility |
|---|---|
| Frontend interaction layer | Message display, input box, session list, streaming rendering |
| Backend API layer | Receive requests, authentication, parameter validation, business orchestration |
| Session management layer | Create sessions, save messages, maintain context |
| Model adaptation layer | Connect to different large model services, unify requests and responses |
| Model service layer | Call models such as GPT, Claude, Gemini, Qwen, and DeepSeek |
| Data storage layer | Store users, sessions, messages, and configuration |
| Security governance layer | Permission validation, content safety, rate limiting, auditing |
| Observability layer | Record requests, tokens, latency, errors, and cost |
1.2 System Architecture
flowchart TB
A["User"] --> B["Frontend App"]
B --> C["Backend API Gateway"]
C --> D["Authentication & Permission Validation"]
D --> E["Conversation Business Service"]
E --> F["Session Management"]
E --> G["Context Construction"]
E --> H["Model Adaptation Layer"]
H --> I["LLM API"]
F --> J["Database"]
E --> K["Cache"]
H --> L["Logging & Monitoring"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
1.3 The Full Path of a Single Chat Request
After a user sends a message, the system needs to complete several steps:
- The frontend collects the user's input.
- The backend verifies identity and request parameters.
- Retrieve the current session and historical messages.
- Build the context required by the model.
- Call the large model API.
- Receive the model's response.
- Persist the response.
- Return the result to the frontend.
- The frontend updates the chat interface.
sequenceDiagram
autonumber
participant U as User
participant F as Frontend
participant B as Backend
participant D as Database
participant M as LLM
U->>F: Enter a question
F->>B: Send chat request
B->>B: Authentication and parameter validation
B->>D: Query session and historical messages
D-->>B: Return historical messages
B->>B: Build context
B->>M: Send model request
M-->>B: Return model response
B->>D: Save user message and model reply
B-->>F: Return response
F-->>U: Display the answer
Production systems typically use streaming responses, so model generation and frontend display do not necessarily wait until all content has been generated before starting.
2. Technology Selection and Project Structure
To build an application from scratch, the first step is to decide on the tech stack.
2.1 Recommended Tech Stack
| Technology Layer | Options | Primary Responsibility |
|---|---|---|
| Frontend | React, Vue, Next.js | Chat interface and interaction |
| Backend | Node.js, Python, Java, Go | API and business logic |
| Model SDK | Official SDK, HTTP Client | Call model services |
| Database | PostgreSQL, MySQL | Persist users and messages |
| Cache | Redis | Session caching, rate limiting, temporary state |
| Real-time Communication | SSE, WebSocket | Streaming responses |
| Deployment | Docker, Kubernetes | Application runtime and scaling |
| Monitoring | OpenTelemetry, Prometheus | Metrics, tracing, and logs |
For the first version, we recommend using:
- React: build the chat interface.
- Node.js: implement the backend API.
- PostgreSQL: store users, conversations, and messages.
- Redis: use later for caching and rate limiting.
- LLM API: provide model inference capabilities.
There is no need to introduce microservices, message queues, and complex agent frameworks from the start. Build a well-structured monolith first, then split it according to business scale.
2.2 Project Directory Structure
A relatively clean project can be organized as follows:
ai-chat-app/
├── frontend/
│ ├── src/
│ │ ├── components/
│ │ │ ├── ChatWindow.tsx
│ │ │ ├── MessageList.tsx
│ │ │ ├── MessageItem.tsx
│ │ │ ├── ChatInput.tsx
│ │ │ └── ConversationList.tsx
│ │ ├── hooks/
│ │ │ └── useChat.ts
│ │ ├── services/
│ │ │ └── chatApi.ts
│ │ ├── types/
│ │ │ └── chat.ts
│ │ └── App.tsx
│ └── package.json
│
├── backend/
│ ├── src/
│ │ ├── routes/
│ │ │ └── chat.ts
│ │ ├── services/
│ │ │ ├── chatService.ts
│ │ │ ├── modelService.ts
│ │ │ └── contextService.ts
│ │ ├── adapters/
│ │ │ └── openaiAdapter.ts
│ │ ├── repositories/
│ │ │ ├── conversationRepository.ts
│ │ │ └── messageRepository.ts
│ │ ├── middleware/
│ │ │ ├── auth.ts
│ │ │ └── rateLimit.ts
│ │ ├── config/
│ │ │ └── model.ts
│ │ └── app.ts
│ └── package.json
│
├── database/
│ └── migrations/
│
├── .env
└── README.md
Here are several important design principles:
- Routes handle HTTP requests and do not directly carry complex business logic.
- Services handle conversation orchestration and model invocation.
- Adapters isolate protocol differences between different model providers.
- Repositories handle database access.
- Middleware handles cross-cutting concerns such as authentication and rate limiting.
This layering prevents business code from becoming tightly coupled to model providers and database implementations.
3. Designing the Conversation Data Model
An AI chat application first needs to solve a fundamental problem: how to store users, conversations, and messages?
You cannot simply store an entire conversation as a single string. As conversations grow, you need to support paginated queries, history restoration, message status, model tracking, and data analysis.
3.1 Core Data Entities
We recommend designing at least three core entities:
User
| Field | Type | Description |
|---|---|---|
| id | UUID | Unique user identifier |
| username | VARCHAR | Username |
| VARCHAR | ||
| created_at | TIMESTAMP | Created at |
Conversation
| Field | Type | Description |
|---|---|---|
| id | UUID | Unique conversation identifier |
| user_id | UUID | Owning user |
| title | VARCHAR | Conversation title |
| model | VARCHAR | Currently used model |
| system_prompt | TEXT | Conversation-level system instruction, optional |
| created_at | TIMESTAMP | Created at |
| updated_at | TIMESTAMP | Updated at |
Message: Message
| Field | Type | Description |
|---|---|---|
| id | UUID | Unique message identifier |
| conversation_id | UUID | Owning conversation |
| role | VARCHAR | Message role |
| content | TEXT / JSONB | Message content |
| status | VARCHAR | Message status |
| input_tokens | INTEGER | Input token count |
| output_tokens | INTEGER | Output token count |
| model | VARCHAR | Model actually used |
| created_at | TIMESTAMP | Created at |
3.2 Database Relationships
erDiagram
USER ||--o{ CONVERSATION : owns
CONVERSATION ||--o{ MESSAGE : contains
USER {
uuid id PK
string username
string email
timestamp created_at
}
CONVERSATION {
uuid id PK
uuid user_id FK
string title
string model
timestamp created_at
timestamp updated_at
}
MESSAGE {
uuid id PK
uuid conversation_id FK
string role
jsonb content
string status
integer input_tokens
integer output_tokens
timestamp created_at
}
3.3 Why can Message Content use JSONB?
If the application only supports plain text, using TEXT is sufficient.
However, if multimodal messages are supported later, message content may include:
- Text.
- Image references.
- Audio content.
- File references.
- Tool calls.
- Tool execution results.
Therefore, structured JSON can be used to store message content.
For example:
{
"type": "text",
"text": "请解释一下 Transformer 的工作原理"
}
A message containing an image can be abstracted as:
{
"parts": [
{
"type": "text",
"text": "请分析这张图片"
},
{
"type": "image",
"url": "https://example.com/image.png"
}
]
}
Note that the message structure in the database is not necessarily the same as the request structure of the model provider. The application should define its own message model internally, and then the model adapter layer converts it into the provider protocol.
3.4 Role Design
The most basic conversation roles include:
| Role | Meaning |
|---|---|
| system | System-level instruction |
| user | User message |
| assistant | Model reply |
| tool | Tool execution result |
Different model APIs may support roles and message structures differently; you cannot assume that all providers fully follow the same protocol.
For tool calls, you also need to save the tool call ID, tool name, parameters, execution status, and result, so that the complete lifecycle of a tool call can be traced.
IV. Integrating with an LLM API
The model API is the core capability source for AI chat applications.
The application itself is responsible for user interaction and business logic, while the LLM performs reasoning based on the input context and generates responses.
4.1 Configure the Model Service
First, prepare the following:
- API Base URL.
- API Key.
- Model ID.
- Request timeout.
- Maximum output tokens.
- Sampling parameters such as Temperature.
Configuration can be managed through environment variables:
MODEL_API_KEY=your_api_key
MODEL_BASE_URL=https://api.example.com/v1
MODEL_NAME=your-model-name
In production environments, do not commit real API keys to Git or write them into frontend code.
4.2 Calling the Model from the Backend
The following demonstrates using JavaScript and an OpenAI SDK-style interface. In actual use, you need to install the corresponding SDK and adjust the configuration and parameters according to the protocol of the chosen provider.
npm install openai
Model service wrapper:
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.MODEL_API_KEY,
baseURL: process.env.MODEL_BASE_URL,
});
export async function generateAnswer(
messages: Array<{
role: "system" | "user" | "assistant";
content: string;
}>
) {
const response = await client.chat.completions.create({
model: process.env.MODEL_NAME!,
messages,
temperature: 0.7,
max_tokens: 2048,
});
return {
content: response.choices[0]?.message?.content ?? "",
usage: response.usage,
model: response.model,
};
}
This uses a Chat Completions-style API example, which does not mean that all model providers support exactly the same fields and parameters.
4.3 Why Encapsulate a Model Adapter?
If business code directly calls a model provider's SDK, switching models later will require many changes.
For example:
ChatService
├── Directly call OpenAI SDK
├── Directly process OpenAI responses
└── Directly depend on OpenAI streaming events
This design couples the business layer to a specific provider.
A more reasonable approach is to define a unified interface:
export interface ModelAdapter {
generate(
messages: ChatMessage[],
options?: ModelOptions
): Promise<ModelResponse>;
stream(
messages: ChatMessage[],
options?: ModelOptions
): AsyncIterable<ModelStreamEvent>;
}
Then implement adapters for different providers separately:
ModelAdapter
├── OpenAIAdapter
├── AnthropicAdapter
├── GeminiAdapter
├── QwenAdapter
└── DeepSeekAdapter
The business layer depends only on the unified interface, not directly on any provider's SDK.
Pay special attention: a unified interface should not smooth over all capability differences. Parameters, tool calls, or multimodal capabilities that a model does not support should be explicitly declared, rejected, or degraded, rather than silently ignored.
V. Implementing the Core Chat API
After model integration is complete, you need to provide stable business APIs for the frontend.
5.1 API Design
It is recommended to implement the following APIs first:
| HTTP Method | API | Purpose |
|---|---|---|
| POST | /api/conversations | Create conversation |
| GET | /api/conversations | List conversations |
| GET | /api/conversations/:id | Get conversation details |
| PATCH | /api/conversations/:id | Update conversation information |
| DELETE | /api/conversations/:id | Delete conversation |
| GET | /api/conversations/:id/messages | Get message history |
| POST | /api/chat | Send a message |
| POST | /api/chat/stream | Send a message as a stream |
These endpoints are business APIs internal to the application; they do not need to directly expose the model provider's API.
5.2 Conversation Request Structure
The frontend can send:
{
"conversation_id": "conv_123",
"message": "什么是 Transformer?",
"model": "your-model-name"
}
The backend needs to validate:
- Whether the current user is logged in.
- Whether the current conversation belongs to the current user.
- Whether the message content is empty.
- Whether the message length exceeds the limit.
- Whether the model is within the allowed range.
- Whether the current user still has available quota.
Do not directly trust the user_id, model name, or permission information submitted by the frontend. User identity should be obtained from verified authentication credentials.
5.3 Conversation Service Implementation
The conversation service is responsible for connecting conversations, messages, and models:
export async function chat(
userId: string,
conversationId: string,
userContent: string
) {
const conversation =
await conversationRepository.findById(conversationId);
if (!conversation || conversation.userId !== userId) {
throw new Error("Conversation not found");
}
const history =
await messageRepository.findRecentMessages(conversationId);
const userMessage =
await messageRepository.create({
conversationId,
role: "user",
content: userContent,
status: "completed",
});
const context = await contextService.build({
conversation,
history,
userContent,
});
const result = await modelService.generate({
model: conversation.model,
messages: context,
});
const assistantMessage =
await messageRepository.create({
conversationId,
role: "assistant",
content: result.content,
status: "completed",
inputTokens: result.usage?.prompt_tokens,
outputTokens: result.usage?.completion_tokens,
});
return {
userMessage,
assistantMessage,
usage: result.usage,
};
}
This is a simplified example for illustrating responsibility boundaries. A production implementation also needs to handle model call failures, timeouts, concurrent messages, transaction consistency, request idempotency, and message state recovery.
Pay particular attention to this: if the user message has already been saved but the model call fails, the conversation must not be left in an unexplainable state. Message states can be designed as follows:
- pending: waiting to be processed.
- streaming: currently generating.
- completed: generation completed.
- failed: generation failed.
- cancelled: actively cancelled by the user.
6. Implementing the Frontend Chat Interface
The frontend is the main entry point for user interaction with the AI.
A basic chat page usually includes four areas:
- Conversation list on the left.
- Conversation title and model information at the top.
- Message display area in the middle.
- Message input area at the bottom.
6.1 Page Component Design
ChatPage
├── Sidebar
│ ├── NewConversationButton
│ └── ConversationList
│ └── ConversationItem
│
├── ChatMain
│ ├── ChatHeader
│ ├── MessageList
│ │ └── MessageItem
│ └── ChatInput
│ ├── TextArea
│ ├── AttachmentButton
│ └── SendButton
│
└── ModelSelector
Components should communicate through clear data structures and callbacks as much as possible, avoiding placing all conversation state in a single page component.
6.2 Message Data Structure
The frontend can define:
export interface ChatMessage {
id: string;
conversationId: string;
role: "user" | "assistant" | "tool";
content: string;
status: "pending" | "streaming" | "completed" | "failed";
createdAt: string;
}
The message list needs to support:
- Different styles for user messages and AI messages.
- Markdown rendering.
- Code highlighting.
- Copying messages.
- Regenerating messages.
- Message feedback.
- Loading and error states.
- Scrolling and performance optimization for long messages.
For Markdown rendering, untrusted content must be handled securely to avoid injecting model output directly into the page as unsanitized HTML.
6.3 Conversation State Management
A chat application not only needs to store the message array but also needs to manage request state.
Typical states include:
interface ChatState {
conversationId: string | null;
messages: ChatMessage[];
input: string;
isLoading: boolean;
isStreaming: boolean;
error: string | null;
}
Where:
- isLoading: the request is being prepared or is waiting for a response.
- isStreaming: the model is continuously outputting content.
- error: the error that occurred in the current request.
These two states should not simply be merged into a single boolean value. The application may need to distinguish between phases such as waiting for the first Token, receiving content, completion, and failure.
7. Implementing Streaming Responses
If the application waits until the model has generated the complete answer before returning it each time, users will have to wait for a relatively long time.
Streaming can transmit the content generated by the model to the frontend incrementally, significantly improving the interactive experience.
7.1 Basic Principles of Streaming Responses
Taking SSE as an example:
- The backend initiates a streaming request to the model service.
- The model service continuously produces events or content fragments.
- The backend reads upstream data.
- The backend converts the content into internal application events.
- The frontend continuously receives events and updates the message content.
flowchart TB
A["User sends message"] --> B["Backend establishes streaming request to model"]
B --> C["Model generates Token"]
C --> D["Model returns streaming data"]
D --> E["Backend parses data chunks"]
E --> F["Convert to application events"]
F --> G["SSE pushes to frontend"]
G --> H["Frontend appends message content"]
H --> I{"Is it finished?"}
I -->|"No"| C
I -->|"Yes"| J["End response and save message"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
7.2 Backend Streaming Interface
The following uses Node.js's Web API ReadableStream and SSE as an example to demonstrate how the application layer sends events to the browser. modelService.stream() is the abstract method of the unified model adapter layer described earlier.
export async function streamChat(
userId: string,
conversationId: string,
content: string
) {
await conversationService.assertOwnedBy(
conversationId,
userId
);
const context = await contextService.buildForNewMessage(
conversationId,
content
);
return new ReadableStream({
async start(controller) {
const encoder = new TextEncoder();
const send = (event: string, data: unknown) => {
controller.enqueue(
encoder.encode(
`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`
)
);
};
try {
send("start", {
conversationId,
});
for await (const chunk of modelService.stream({
messages: context,
})) {
if (chunk.type === "text_delta") {
send("delta", {
content: chunk.text,
});
}
}
send("done", {
conversationId,
});
} catch (error) {
send("error", {
message: "Model generation failed",
});
} finally {
controller.close();
}
},
});
}
Message persistence, cancellation signals, disconnect detection, and resource cleanup are omitted here. A production implementation needs to fold these into the complete request lifecycle.
7.3 Receiving SSE on the Frontend
The browser can use fetch to read a streaming response:
async function sendStreamMessage(
conversationId: string,
message: string,
onDelta: (text: string) => void
) {
const response = await fetch("/api/chat/stream", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
conversationId,
message,
}),
});
if (!response.ok || !response.body) {
throw new Error("Stream request failed");
}
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, {
stream: true,
});
const events = buffer.split("\n\n");
buffer = events.pop() ?? "";
for (const event of events) {
const dataLine = event
.split("\n")
.find((line) => line.startsWith("data: "));
if (!dataLine) continue;
const data = JSON.parse(
dataLine.slice(6)
);
if (data.content) {
onDelta(data.content);
}
}
}
}