Back to series
· 11 min read

How Do Large Language Models Work? The Complete Technical Principles from Input Processing to Text Generation


1. Overview of How Large Language Models Work

The core mechanism of a Large Language Model (LLM) can be summarized as:

Receive input → Convert to Tokens → Vector representation → Transformer computation → Predict the next Token → Loop generation → Output text.

When a user inputs the following to the model:

What is machine learning?

The model does not directly understand this sentence and then search for an answer in some knowledge base.

Instead, it first converts the text into Token IDs, maps them to vectors through Embedding, computes contextual representations through multiple layers of Transformer networks, and finally predicts the probability distribution of the next Token.

The model keeps selecting and generating new Tokens until a stopping condition is met.

The overall process:

flowchart TB
    A["User input text"] --> B["Tokenizer: text segmentation"]
    B --> C["Token ID: integer sequence"]
    C --> D["Embedding: vector mapping"]
    D --> E["Transformer: context computation"]
    E --> F["LM Head: generate Logits"]
    F --> G["Softmax: compute probability distribution"]
    G --> H["Decoding strategy: select Token"]
    H --> I["Append to current sequence"]
    I --> J{"End generation?"}
    J -->|"No"| E
    J -->|"Yes"| K["Tokenizer decoding"]
    K --> L["Return text answer"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F

To understand this flow, you need to distinguish two phases:

Phase Core task Whether model parameters are updated
Training Learn language patterns and knowledge representations from data Yes
Inference Use the trained model to generate predictions Usually not

Training determines what the model has learned; inference determines how the model uses those learned capabilities based on the current input.

2. Step One: How Is Input Text Converted into Tokens?

Computers cannot feed natural language directly into a neural network as numerical input, so the text must first be encoded.

This process is carried out by the Tokenizer.

2.1 What Is a Token?

A Token is the basic unit of text that a large language model processes.

It does not necessarily equal a single Chinese character or a single English word; that depends on the tokenization algorithm and vocabulary the model uses.

For example:

Input text:
Artificial intelligence is powerful.

It might be split into:

["Artificial", " intelligence", " is", " powerful", "."]

Chinese text may likewise be split by character, by word, or into subword fragments.

Different models do not necessarily use the same Tokenizer, so the same piece of text may correspond to different numbers of Tokens in different models.

2.2 The Tokenizer's Encoding Process

A Tokenizer typically handles conversion in two directions:

  • Encoding: converting text into Token IDs.
  • Decoding: converting Token IDs back into text.

For example:

Token Token ID
Artificial 1024
intelligence 2056
is 3108
powerful 4216

The data above is for illustration only.

After encoding, the input text becomes a sequence of integers:

X=[1024,2056,3108,4216]

These integers are indices in the vocabulary, and do not directly express meaning.

2.3 How a Tokenizer Works Internally

Common Tokenizer algorithms include:

Algorithm Basic Mechanism
BPE Builds a subword vocabulary by repeatedly merging high-frequency adjacent symbols
WordPiece Builds a subword vocabulary based on word-piece scores
Unigram Selects a more suitable vocabulary from a set of candidate subwords
SentencePiece Performs subword modeling directly on raw text

The design of the Tokenizer affects:

  • The size of the model vocabulary.
  • The length of the input sequence.
  • The encoding efficiency for different languages.
  • The computational cost of training and inference.
  • How special symbols and code are handled.

Note that the Tokenizer itself is not responsible for understanding the meaning of the text. It is only responsible for mapping between text and Token IDs according to vocabulary rules.

III. Step Two: Embedding Converts Token IDs into Vectors

After Tokenizer encoding, what the model gets is a sequence of integers.

But neural networks need to compute with continuous numerical values, so an Embedding layer is also required.

3.1 The Basic Principle of Embedding

Embedding is a mechanism that maps discrete IDs to continuous vectors.

Suppose the model vocabulary size is V and the hidden dimension is d. Then the Embedding matrix can be expressed as:

E\in\mathbb{R}^{V\times d}

Where:

  • V: vocabulary size.
  • d: the vector dimension corresponding to each Token.
  • E: the model's learnable embedding matrix.

For Token ID i, the model obtains the corresponding vector by table lookup:

e_i=E[i]

For example:

TokenID=1024

After Embedding mapping, it might yield:

e_{1024}= [0.12,-0.31,0.45,0.08,\ldots]

This vector usually has hundreds to thousands of dimensions, depending on the model architecture.

3.2 Why Are Vector Representations Needed?

An integer ID only indicates the Token's position in the vocabulary.

For example:

apple = 1024
banana = 2056
car = 3108

There is no inherent semantic distance relationship among these numbers.

Embedding, by contrast, learns continuous vector representations of different Tokens through training, enabling the model to process language information using vector operations.

Note:

  • Embedding vectors are not a hand-written semantic dictionary.
  • Each dimension usually cannot be directly interpreted as some clear human concept.
  • Semantic information is usually distributed across multiple dimensions and network layers.
  • Similarity between vectors is only one way of representing relationships, and is not equivalent to full semantic understanding.

3.3 Positional Representation: How Does the Model Know the Order of Tokens?

Embedding mainly provides the content representation of Tokens, but language also carries order information.

For example:

the cat chases the mouse
the mouse chases the cat

The two sentences may use the same Tokens, but in a different order, and express different meanings.

Therefore, Transformer also needs positional information.

Common approaches include:

  • Absolute Positional Encoding.
  • Relative Positional Encoding.
  • RoPE (Rotary Position Embedding).
  • ALiBi (Attention with Linear Biases).

For models that use RoPE, positional information is incorporated into the attention computation by applying a rotation transform to the Query and Key vectors.

Embedding combined with positional representation forms the input representation of the Transformer.

flowchart TB
    A["Token ID Sequence"] --> B["Embedding Lookup"]
    C["Token Position Information"] --> D["Position Representation Computation"]
    B --> E["Combined Input Representation"]
    D --> E
    E --> F["Input to Transformer"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F

IV. Step 3: How Does the Transformer Process Input?

The Transformer is the core architecture of today's mainstream generative large language models.

Through multiple layers of neural networks, it performs contextual computation on the Token representations in the input sequence.

The main components of a typical decoder-only Transformer include:

  1. Masked Self-Attention.
  2. Feed Forward Network.
  3. Residual Connection.
  4. Layer Normalization.

Multiple Transformer Blocks are stacked in sequence, processing the input representations layer by layer.

4.1 Basic Structure of a Transformer Block

A simplified Transformer Block can be represented as:

flowchart TB
    A["Input Hidden State x"] --> B["Layer Normalization"]
    B --> C["Masked Self-Attention"]
    C --> D["Residual Connection"]
    A --> D
    D --> E["Layer Normalization"]
    E --> F["Feed Forward Network"]
    F --> G["Residual Connection"]
    D --> G
    G --> H["Output Hidden State"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The figure above uses a Pre-LN structure for illustration. Actual models may use different normalization methods, attention structures, and feedforward network designs.

4.2 Self-Attention: Establishing Contextual Associations

Self-Attention is the core mechanism of the Transformer.

It allows each Token in the sequence to aggregate information from other positions according to the needs of the current task.

For example:

Xiaoming put the book on the table because it was very heavy.

When the model processes “it,” it needs to use relevant information from the preceding context to determine how this pronoun is associated with other Tokens.

Self-Attention accomplishes this process through three types of vectors: Query, Key, and Value.

Query, Key, Value

For the input hidden state matrix X, Q, K, and V are obtained through three different learnable linear mappings:

Q=XW_Q
K=XW_K
V=XW_V

Where:

  • Query: indicates what information the current position wants to obtain.
  • Key: indicates by what features the current position can be matched.
  • Value: indicates the actual information content provided by the current position.

These three names are abstractions that help understand the computation process; they do not mean that the model is actually performing database queries.

Attention Computation Process

First, compute the dot product of Query and Key:

S=QK^T

The dot product reflects the degree of matching between representations at different positions.

Then perform scaling:

S'=\frac{QK^T}{\sqrt{d_k}}

Then use Softmax to convert it into attention weights:

A=softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)

Finally, use the attention weights to compute a weighted sum of Value:

Attention(Q,K,V)=AV

After combining:

Attention(Q,K,V)= softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Where d_k is the dimension of the Key vector.

Why divide by the square root?

As the vector dimension increases, the dot product values of Query and Key may become larger.

If Softmax is applied directly, it may cause the probability distribution to become overly concentrated, making gradients smaller and affecting training stability.

Dividing by \sqrt{d_k} can control the numerical scale of the dot product.

4.3 Causal Mask: Why can't we see future Tokens?

For autoregressive language models, when predicting the current position, the model is only allowed to use the current and previous Tokens.

For example:

I like learning artificial intelligence

When the model predicts the Token after "learning", it cannot access "artificial intelligence" in advance.

Therefore, a causal mask needs to be used.

For a sequence of length 4, its attention visibility relationships can be represented as:

Current Position Token 1 Token 2 Token 3 Token 4
Token 1 Visible Not Visible Not Visible Not Visible
Token 2 Visible Visible Not Visible Not Visible
Token 3 Visible Visible Visible Not Visible
Token 4 Visible Visible Visible Visible

In attention computation, positions that are not allowed to be attended to are typically set to negative infinity or an equivalent extremely small value, so that after Softmax these positions have zero weight.

This guarantees the causal constraint during training and autoregressive generation.

4.4 Multi-Head Attention: Why do we need multiple attention heads?

A single attention head can only perform attention computation in one projection subspace.

Multi-head attention computes different attention results in parallel through multiple independent projection matrices.

head_i=Attention(QW_i^Q,KW_i^K,VW_i^V)

After concatenating the outputs of multiple heads, they pass through the output projection matrix:

MultiHead(Q,K,V)= Concat(head_1,\ldots,head_h)W^O

Multiple attention heads can learn different association patterns, but one cannot simply assume that each head is only responsible for one fixed semantic relationship.

4.5 Feed Forward Network: Further Transforming Features

Attention is responsible for information interaction between Tokens, while FFN performs nonlinear transformation on the hidden representation of each position.

Typical form:

FFN(x)=\sigma(xW_1+b_1)W_2+b_2

Where:

  • W_1, W_2: Weight matrices.
  • b_1, b_2: Biases.
  • \sigma: Activation function.

Modern models may also use gated structures such as SwiGLU.

4.6 Residual Connections and Normalization

Deep Transformers typically contain a large number of repeated computational modules.

Residual connections allow information to pass across sublayers:

y=x+F(x)

It helps improve gradient propagation and optimization stability in deep networks.

Layer Normalization normalizes the hidden representation, helping to stabilize network training.

V. Step 4: How Does a Multi-Layer Transformer Process Information Layer by Layer?

A single Transformer Block can only complete one round of context interaction and feature transformation.

Actual large language models typically stack dozens or even more layers of Transformer Blocks.

For example, a model containing N layers of Transformer Blocks can be represented as:

H_0=Embedding(X)+Position(X)
H_1=Block_1(H_0)
H_2=Block_2(H_1)
\cdots
H_N=Block_N(H_{N-1})

Finally, we obtain:

H_N

Where:

  • H_0: Initial input representation.
  • H_i: The hidden state of the i-th layer.
  • N: The number of Transformer Blocks.

Each layer performs further computation on the hidden representation.

Overall, the model continuously integrates information and transforms features of the input, forming a context representation more suitable for the current prediction task.

It should be noted that one cannot simply interpret the lower, middle, and upper layers as being fixed to syntax, semantics, and reasoning respectively. The representation functions of different layers may overlap, and the actual mechanisms are far more complex than this layered description.

6. Step 5: How Does the Model Predict the Next Token?

After computation through multiple Transformer layers, the model obtains the final hidden state.

For autoregressive generation, the hidden state at the last valid position is typically used to predict the next Token.

6.1 LM Head: The Output Projection Layer

Suppose the hidden state at the last position is:

h_t\in\mathbb{R}^{d}

With a vocabulary of size V, the model obtains the following through the output projection layer:

z_t=h_tW_{vocab}+b

Where:

  • h_t: the hidden state at the current position.
  • W_{vocab}: the output projection matrix.
  • b: the bias.
  • z_t: the Logits vector.

The dimension of the Logits is:

z_t\in\mathbb{R}^{V}

In other words, the model computes an unnormalized score for every Token in the vocabulary.

For example:

Candidate Token Logit
data 2.8
model 2.1
algorithm 1.5
system 0.8

These scores are not probabilities in themselves.

6.2 Softmax: Converting Logits into Probabilities

Softmax can convert the Logits into a probability distribution:

P(x_i)= \frac{e^{z_i}}{\sum_{j=1}^{V}e^{z_j}}

Where:

  • z_i: the Logit of the i-th Token.
  • V: the vocabulary size.
  • P(x_i): the predicted probability of the corresponding Token.

For example:

Candidate Token Example Probability
data 0.55
model 0.25
algorithm 0.15
system 0.05

These values are only used to illustrate the probability distribution.

The probabilities output by Softmax sum to 1.

6.3 Why Not Just Pick the Token with the Highest Probability?

Choosing the Token with the highest probability every time is Greedy Decoding.

This approach is simple, but it is prone to repetition, a lack of diversity, or local optima.

Therefore, the model can also use different decoding strategies:

  • Temperature.
  • Top-K Sampling.
  • Top-P Sampling.
  • Beam Search.
  • Greedy Search.

These methods determine how the next Token is selected from the probability distribution.

7. Step 6: How Does the Model Generate a Complete Answer Token by Token?

Large language models usually do not generate the entire answer at once; instead, they generate it step by step in an autoregressive manner.

Suppose the user input is:

The core technologies of artificial intelligence include

The model might generate the output through the following steps:

Step Current Context Generated Token
1 The core technologies of artificial intelligence include machine learning
2 The core technologies of artificial intelligence include machine learning ,
3 The core technologies of artificial intelligence include machine learning, deep learning
4 The core technologies of artificial intelligence include machine learning, deep learning and
5 The core technologies of artificial intelligence include machine learning, deep learning and natural language processing

Each time the model generates a Token, it appends it to the current context for the next prediction.

7.1 The Mathematical Formulation of Autoregressive Generation

Suppose the existing context is:

x_1,x_2,\ldots,x_t

The model predicts the next Token:

P(x_{t+1}|x_1,\ldots,x_t)

After selecting one token, the sequence becomes:

x_1,x_2,\ldots,x_t,x_{t+1}

Then prediction continues:

P(x_{t+2}|x_1,\ldots,x_t,x_{t+1})

Ultimately, the joint probability of the entire sequence can be expressed as:

P(X)=\prod_{t=1}^{T}P(x_t|x_{<t})

This is the core mechanism of autoregressive language models.

7.2 When Does Generation End?

Common stopping conditions include:

  • Generating the EOS (End of Sequence) end token.
  • Generating a stop token configured for the model.
  • Reaching the maximum number of generated tokens.
  • Hitting a stop string configured by the user or application.
  • The external inference service actively terminating generation.

Actual models may use dedicated conversation-end tokens, for example distinguishing the end of a single message from the end of an entire assistant turn.

Therefore, the stopping condition is not necessarily just a single EOS token.

8. Step 7: How Does KV Cache Accelerate Generation?

Autoregressive generation has an important performance issue:

Each time a new token is generated, the preceding context must be used.

If the Key and Value of all historical tokens were recomputed every time, this would produce a large amount of duplicated computation.

KV Cache exists to solve this problem.

8.1 The Case Without KV Cache

Suppose the model has already generated:

A B C

Now it needs to predict D.

Without KV Cache, the model must reprocess:

A B C

After generating D, when predicting E, it must again reprocess:

A B C D

The Key and Value of historical tokens are computed repeatedly.

8.2 The Case With KV Cache

Transformer attention computation produces Keys and Values.

KV Cache stores the Keys and Values corresponding to historical tokens that have already been processed.

When generating the next token:

  1. Read the cache of historical tokens.
  2. Compute only the Query, Key and Value corresponding to the new token.
  3. Append the new Key and Value to the cache.
  4. Perform attention computation using the historical cache and the current token.

The process is as follows:

flowchart TB
    A["Input historical tokens"] --> B["Compute historical K, V"]
    B --> C["Save to KV Cache"]
    C --> D["Input new token"]
    D --> E["Compute current Q, K, V"]
    E --> F["Read historical KV Cache"]
    F --> G["Concatenate historical and current K, V"]
    G --> H["Execute Attention"]
    H --> I["Predict next token"]
    I --> J["Append new K, V to cache"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

8.3 The Performance Value of KV Cache

KV Cache can significantly reduce duplicated computation during the autoregressive decoding phase.

However, it also increases GPU memory usage.

Its memory requirement roughly depends on the following factors:

Memory_{KV} \propto 2\times L\times B\times T\times H_{kv}\times d_h

Where:

  • L: number of Transformer layers.
  • B: Batch Size.
  • T: number of cached tokens.
  • H_{kv}: number of Key/Value heads.
  • d_h: dimension of each head.
  • 2: stores Key and Value separately.

If BF16 or FP16 is used, each element typically occupies 2 bytes.

A more complete estimate would also need to account for data type, cache layout, alignment, and the implementation of the inference framework.

8.4 MHA, GQA and MQA

The size of the KV Cache is also related to the attention structure.

Structure Characteristics
MHA Each Query Head corresponds to an independent Key/Value Head
GQA Multiple Query Heads share one set of Key/Value Heads
MQA All Query Heads share one set of Key/Value Heads

GQA and MQA can reduce the number of Key/Value heads, thereby lowering KV Cache memory usage and the associated memory access pressure.

9. What Is the Difference Between the Training Phase and the Inference Phase?

Although training and inference use the same model architecture, their execution methods and computational goals differ.

9.1 Training Phase

The goal of the training phase is to optimize model parameters.

Typical process:

  1. Input training samples.
  2. Perform forward propagation.
  3. Compute predictions.
  4. Compute the loss function.
  5. Backpropagate to compute gradients.
  6. The optimizer updates the parameters.