I. What Is Large Language Model Pre-training?
Pre-training is a core stage in the process of building a large language model.
Its goal is to use large-scale, widely distributed text data to let the model learn the statistical regularities, grammatical structures, semantic relations, knowledge associations, and to some degree the reasoning patterns present in language.
Unlike traditional supervised learning, pre-training usually does not require humans to label each piece of text with a category or a reference answer; instead, the training objective is constructed from the text itself.
For example, given a passage of text:
Artificial intelligence is changing the way software is developed.
The model can learn to predict the next Token:
Artificial intelligence is changing the way software is [developed]
In this process, the original text serves as both the input to the model and the source of the training target.
This training approach is called self-supervised learning.
1. What Problem Does Pre-training Solve?
Pre-training aims to give the model a foundational representation with broad transferability, rather than solving only one specific task.
The main things it learns include:
| Learning dimension | Specific content |
|---|---|
| Language structure | Lexicon, grammar, syntax, and the rules of language organization |
| Semantic representation | Word meaning, contextual relations, semantic similarity |
| Knowledge association | Connections among people, places, events, and concepts |
| Context modeling | Long-distance dependencies, anaphoric relations, contextual information |
| Text generation | Generating coherent text from the existing context |
| Task patterns | Patterns required for tasks such as question answering, summarization, translation, and code completion |
Note: pre-training does not write all knowledge into the model like records in a database; rather, by optimizing a large number of parameters, it gradually shapes distributed knowledge representations and language modeling ability in the model.
2. How Pre-training Differs from Other Training Stages
Modern large language models usually go through multiple training stages.
| Stage | Core objective | Typical approach |
|---|---|---|
| Pre-training | Acquire general language modeling ability | Self-supervised learning |
| Supervised fine-tuning (SFT) | Learn instruction understanding and task execution | Supervision on instruction–response data |
| Preference alignment | Make outputs conform to human preferences | DPO, reward models, etc. |
| Reinforcement learning | Optimize behavior on complex tasks | Methods such as PPO and GRPO |
These stages are not a completely fixed pipeline. Different models may use different data, training objectives, and optimization methods, and may skip or combine certain stages.
The overall relationship is as follows:
flowchart TB
A["Large-scale raw data"] --> B["Pre-training"]
B --> C["Base language model"]
C --> D["Supervised fine-tuning SFT"]
D --> E["Instruction model"]
E --> F["Preference alignment and reinforcement learning"]
F --> G["Aligned model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
Pre-training determines the model's foundational capabilities and knowledge coverage, while the later stages further shape its interaction style, task behavior, and output preferences.
II. Where Does Pre-training Data Come From?
The capabilities of a large language model depend heavily on the scale, quality, coverage, and distribution of its training data.
Pre-training data usually comes from a variety of public, licensed, or internally and lawfully obtained data sources.
1. Common Data Sources
| Data type | Typical content | Main value |
|---|---|---|
| Web text | News, blogs, forums, encyclopedias | Broad coverage of natural-language knowledge |
| Books | Novels, textbooks, professional works | Long-form text and systematic knowledge |
| Academic papers | Research papers, technical reports | Specialized knowledge and scientific expression |
| Technical documentation | API docs, developer manuals | Programming and engineering knowledge |
| Code data | Git repositories, code examples | Programming language and code generation ability |
| Q&A data | Questions, discussions, answers | Question understanding and answering patterns |
| Multilingual corpora | Text in different languages | Multilingual understanding and generation ability |
Different data types do not contribute equally to a model.
For example:
- Books help with learning long-form structure and coherent expression.
- Code data helps with learning programming language syntax and program structure.
- Academic papers help with learning specialized terminology and technical expression.
- Multilingual corpora help improve cross-lingual modeling capabilities.
2. Data Scale and Data Quality
Data scale is important, but data scale does not equal the amount of effective information.
Suppose a training dataset contains:
- large numbers of duplicated web pages;
- low-quality machine-generated text;
- meaningless characters;
- spam advertisements;
- incorrect code snippets;
- articles with highly similar content.
Even if such data can increase the number of tokens, it may not effectively improve model capabilities.
Therefore, one of the core goals of pretraining data engineering is to increase the proportion of effective data.
Data value can be roughly understood as:
This is only a conceptual expression to aid understanding; it is not a general formula that can be directly calculated in actual training.
3. Why Is Data Distribution Important?
The capabilities a model ultimately learns are closely related to the training data distribution.
For example:
- If the proportion of code is too low, it may affect code generation capabilities.
- Insufficient high-quality Chinese corpus may affect Chinese comprehension and expression.
- Insufficient domain-specific corpus may lead to limited coverage of professional knowledge.
- If data is overly concentrated in certain content types, it may cause distribution bias in the model.
For training data, we should not only pursue scale but also consider the proportions across different domains, languages, text lengths, and content types.
III. Pretraining Data Processing Workflow
Raw data usually cannot be fed directly into model training and needs to go through a series of processing steps.
A typical data processing workflow is as follows:
flowchart TB
A["Raw data collection"] --> B["Format parsing and text extraction"]
B --> C["Content cleaning"]
C --> D["Quality filtering"]
D --> E["Duplicate data detection and deduplication"]
E --> F["Sensitive and non-compliant content handling"]
F --> G["Data mixing and sampling"]
G --> H["Tokenizer encoding"]
H --> I["Token sequence preparation"]
I --> J["Training dataset"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
1. Format Parsing and Text Extraction
Different data sources may use different formats:
- HTML
- Markdown
- JSON
- XML
- Plain text
- Source code files
First, this content needs to be parsed into text sequences that the model can process.
Web page data especially requires handling of:
- HTML tags;
- navigation bars and footers;
- advertising areas;
- page templates;
- irrelevant scripts and styles;
- repeated page elements.
If raw web page content is fed directly into the model, it may cause a large number of invalid tokens to consume training resources.
2. Data Cleaning
Data cleaning is mainly used to remove obviously invalid or low-quality content.
Common operations include:
- Deleting empty text;
- Deleting garbled text;
- Filtering based on the proportion of abnormal characters;
- Cleaning up repeated punctuation;
- Removing meaningless template text;
- Handling malformed content;
- Filtering out obvious spam information.
However, more aggressive cleaning is not always better.
For example, indentation, special symbols, comments, and Markdown markup in code may all contain valid information. If ordinary natural language cleaning rules are applied to code data, they may damage the code structure.
Therefore, different cleaning strategies usually need to be developed according to data type.
3. Data Quality Filtering
Quality filtering is more complex than basic cleaning.
Common methods include:
Rule-based filtering
Screening based on rules such as text length, character ratio, language, repetition rate, and format.
Classifier-based filtering
Using a quality classification model to determine whether text has training value.
Model-assisted scoring
Using models to evaluate text readability, coherence, and information density.
Domain-specific quality filtering
Developing specialized standards for data such as code, academic papers, and technical documents.
It should be noted that quality filtering models themselves may also have biases. If filtering rules are overly biased toward a certain language, writing style, or mode of expression, they may reduce data diversity.
4. Data Deduplication
Deduplication is an important step in pretraining data engineering.
Duplicate data can cause several problems:
- Waste training compute resources;
- Cause the model to overfit to repeated content;
- Cause certain viewpoints or knowledge to be over-reinforced;
- Increase the risk of data leakage between the training set and the evaluation set.
Deduplication is usually divided into two levels.
| Type | Target | Common methods |
|---|---|---|
| Exact deduplication | Identical text | Hash, content fingerprint |
| Approximate deduplication | Highly similar text | MinHash, SimHash, locality-sensitive hashing |
Exact deduplication can use text hashing:
If two pieces of text produce the same hash value after normalization, they can be considered duplicate text.
Approximate deduplication is more complex.
For example, two articles may differ only in title, punctuation, or a small number of words, but their main content is basically the same. In this case, text similarity can be estimated using methods such as Shingling and MinHash.
5. Data Mixing and Sampling
Data from different sources needs to be mixed according to a certain strategy.
For example:
| Data type | Example weight |
|---|---|
| General web pages | 45% |
| Books and long-form text | 15% |
| Code | 20% |
| Academic and technical documents | 15% |
| Other high-quality corpora | 5% |
The above proportions are only used to illustrate the data mixing mechanism and do not represent the actual training mix of any mainstream model.
Data mixing weights directly affect how frequently the model is exposed to different data types during training.
Actual training also needs to consider:
- Data quality;
- Data scale;
- Domain coverage;
- Language distribution;
- Repeated sampling;
- Changes across training stages.
Some training schemes increase the sampling proportion of high-quality data in the later stages of training to improve model performance.
IV. Tokenization: Converting Text into Model Input
Neural networks cannot directly process raw strings; text must first be converted into Token IDs.
This process is called Tokenization.
Suppose the input text is:
Large Language Models
The Tokenizer may split it into:
["Large", " Language", " Models"]
Then map each Token to an integer ID in the vocabulary:
[1024, 5832, 917]
The Token segmentation results and IDs here are only illustrative; the actual results depend on the specific Tokenizer.
1. Token ID and Embedding
After receiving a Token ID, the model maps it to a continuous vector through the Embedding matrix.
Let the vocabulary size be V, the hidden dimension be d, and the Embedding matrix be:
For Token ID i, the corresponding vector is:
Therefore, the Token ID itself is only a discrete index; what actually enters Transformer computation is the Embedding vector.
2. Why Is a Unified Tokenizer Needed?
The pretraining and inference stages must use compatible Tokenizers.
If one Tokenizer is used during training and another during inference, even if both process the same piece of text, they may produce different Token segmentations and IDs.
This causes the input representation to be inconsistent with the vocabulary definition used during model training.
A Tokenizer is usually released and used together with the model weights, vocabulary, and special Token configuration.
V. Core Objective of Pretraining: Next Token Prediction
For current mainstream Decoder-only large language models, the most typical pretraining objective is to predict the next Token.
Given a Token sequence:
The model learns:
That is, predict the Token at the current position based on the context before the current position.
1. Autoregressive Probability Decomposition
According to the chain rule of probability, the joint probability of the entire sequence can be decomposed as:
where:
This means the model only needs to learn the conditional probability distribution at each position to perform probabilistic modeling of the entire sequence.
2. Input and Label Misalignment
Suppose the original Token sequence is:
[A, B, C, D, E]
During training, it is usually constructed as:
| Input Position | Model Input | Prediction Target |
|---|---|---|
| 1 | A | B |
| 2 | A, B | C |
| 3 | A, B, C | D |
| 4 | A, B, C, D | E |
Corresponding inputs and labels:
input_ids = tokens[:, :-1]
labels = tokens[:, 1:]
Model input:
[A, B, C, D]
Training labels:
[B, C, D, E]
The model predicts the next Token at each position.
3. Why can training be parallelized?
Although the model learns autoregressive prediction, the training phase does not need to execute Token by Token like the generation phase.
The reason is: the training data already provides the complete Token sequence.
Through the causal attention mask (Causal Mask), the model can simultaneously compute prediction results for multiple positions while preventing each position from accessing future Tokens.
For example:
Position 1: can only see A
Position 2: can only see A, B
Position 3: can only see A, B, C
Position 4: can only see A, B, C, D
The causal attention mask usually appears as a lower-triangular visible region.
flowchart TB
A["Complete Token Sequence"] --> B["Construct shifted inputs and labels"]
B --> C["Transformer parallel computation"]
C --> D["Apply Causal Mask"]
D --> E["Predict next Token at each position"]
E --> F["Compute prediction loss"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
Here, two concepts need to be distinguished:
- Autoregressive constraint: each position can only use historical context to predict the current Token.
- Parallel training: after the complete training sequence is known, outputs for multiple positions can be computed simultaneously.
The two are not contradictory.
6. Loss Function: How Does the Model Measure Prediction Performance?
The model outputs a prediction score for each Token in the vocabulary, called Logits.
Let the vocabulary size be V, and the output at position t be:
Convert it into a probability distribution via Softmax:
The model needs to increase the predicted probability corresponding to the correct Token.
1. Cross-Entropy Loss
For the true Token x_t, the loss at a single position is:
The higher the model's predicted probability for the correct Token, the lower the loss.
For example:
| Predicted Probability of Correct Token | Cross-Entropy Loss |
|---|---|
| 0.9 | 0.105 |
| 0.5 | 0.693 |
| 0.1 | 2.303 |
| 0.01 | 4.605 |
It can be seen that the more uncertain the model is about the correct Token, the larger the loss.
2. Loss for the Entire Sequence
For a sequence of length T, the average cross-entropy loss can be expressed as:
In actual training, it is usually also necessary to handle padding tokens, special tokens, and samples of different lengths.
Therefore, a more general form is:
where:
- b: sample index within the batch.
- t: token position.
- m_{b,t}: valid-token mask.
- Denominator: the number of valid tokens participating in the loss computation.
3. Why Use Cross-Entropy?
The goal of pretraining is to maximize the likelihood of the training data under the model:
To facilitate optimization, this is usually converted to negative log-likelihood:
This is directly related to the cross-entropy loss.
From an optimization perspective, by reducing cross-entropy, the model continuously increases the predicted probability of the true token in the corresponding context.
VII. How Does the Transformer Participate in Pretraining?
Pretraining does not train a separate prediction module; rather, it trains the parameters of the entire model.
Taking a decoder-only Transformer as an example, the overall computation process is as follows:
flowchart TB
A["Token ID Sequence"] --> B["Token Embedding"]
B --> C["Positional Encoding or RoPE"]
C --> D["Transformer Block"]
D --> E["Final Hidden State"]
E --> F["Output Projection Layer"]
F --> G["Logits"]
G --> H["Softmax Probability Distribution"]
H --> I["Compute Next Token Loss"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
1. Token Embedding
Convert token IDs into vector representations.
If the hidden dimension is d and the sequence length is T, the input tensor shape is usually:
where B is the batch size.
2. Transformer Block
The Transformer Block is the core structure through which the model learns contextual relationships.
A typical Pre-LN Transformer Block can be abstracted as:
where:
- LN: LayerNorm.
- Attention: causal self-attention.
- FFN: feed-forward neural network.
- Residual connection: helps information and gradient propagation.
Different models may differ in normalization methods, activation functions, attention structures, positional encoding, and FFN structures.
3. Output Projection Layer
After the Transformer outputs the hidden states, they need to be mapped to the vocabulary dimension through the output projection layer.
Let the final hidden state be:
The output weights are:
Then:
This yields:
where V is the vocabulary size.
Each position produces a logits vector of length V, used to predict the next token.
Some models use weight tying between the input embedding and the output projection to reduce the number of parameters.
8. Backpropagation: How Does the Model Update Parameters?
After obtaining the loss, the model needs to compute the gradients of the loss with respect to the parameters.
Let the model parameters be:
The training objective is:
Compute via backpropagation:
Then use an optimizer to update the parameters.
Taking the most basic gradient descent as an example:
Where:
- \theta_t: Current model parameters.
- \eta: Learning rate.
- \nabla_{\theta}L: Gradient of the loss function with respect to the parameters.
Practical large language model training usually uses optimizers such as AdamW, together with techniques such as learning rate warmup, decay strategies, and gradient clipping.
Complete training loop:
flowchart TB
A["Prepare Training Batch"] --> B["Model Forward Pass"]
B --> C["Compute Cross-Entropy Loss"]
C --> D["Backpropagation"]
D --> E["Compute Parameter Gradients"]
E --> F["Gradient Clipping and Processing"]
F --> G["Optimizer Updates Parameters"]
G --> H["Update Learning Rate Scheduler"]
H --> I{"Is Training Finished?"}
I -->|"No"| A
I -->|"Yes"| J["Save Model Checkpoint"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#E5F0E9,stroke:#83A995,color:#315D49