Back to series
· 12 min read

Pretraining Large Language Models: From Massive Text to General Language Capabilities


I. What Is Large Language Model Pre-training?

Pre-training is a core stage in the process of building a large language model.

Its goal is to use large-scale, widely distributed text data to let the model learn the statistical regularities, grammatical structures, semantic relations, knowledge associations, and to some degree the reasoning patterns present in language.

Unlike traditional supervised learning, pre-training usually does not require humans to label each piece of text with a category or a reference answer; instead, the training objective is constructed from the text itself.

For example, given a passage of text:

Artificial intelligence is changing the way software is developed.

The model can learn to predict the next Token:

Artificial intelligence is changing the way software is [developed]

In this process, the original text serves as both the input to the model and the source of the training target.

This training approach is called self-supervised learning.

1. What Problem Does Pre-training Solve?

Pre-training aims to give the model a foundational representation with broad transferability, rather than solving only one specific task.

The main things it learns include:

Learning dimension Specific content
Language structure Lexicon, grammar, syntax, and the rules of language organization
Semantic representation Word meaning, contextual relations, semantic similarity
Knowledge association Connections among people, places, events, and concepts
Context modeling Long-distance dependencies, anaphoric relations, contextual information
Text generation Generating coherent text from the existing context
Task patterns Patterns required for tasks such as question answering, summarization, translation, and code completion

Note: pre-training does not write all knowledge into the model like records in a database; rather, by optimizing a large number of parameters, it gradually shapes distributed knowledge representations and language modeling ability in the model.

2. How Pre-training Differs from Other Training Stages

Modern large language models usually go through multiple training stages.

Stage Core objective Typical approach
Pre-training Acquire general language modeling ability Self-supervised learning
Supervised fine-tuning (SFT) Learn instruction understanding and task execution Supervision on instruction–response data
Preference alignment Make outputs conform to human preferences DPO, reward models, etc.
Reinforcement learning Optimize behavior on complex tasks Methods such as PPO and GRPO

These stages are not a completely fixed pipeline. Different models may use different data, training objectives, and optimization methods, and may skip or combine certain stages.

The overall relationship is as follows:

flowchart TB
    A["Large-scale raw data"] --> B["Pre-training"]
    B --> C["Base language model"]
    C --> D["Supervised fine-tuning SFT"]
    D --> E["Instruction model"]
    E --> F["Preference alignment and reinforcement learning"]
    F --> G["Aligned model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49

Pre-training determines the model's foundational capabilities and knowledge coverage, while the later stages further shape its interaction style, task behavior, and output preferences.


II. Where Does Pre-training Data Come From?

The capabilities of a large language model depend heavily on the scale, quality, coverage, and distribution of its training data.

Pre-training data usually comes from a variety of public, licensed, or internally and lawfully obtained data sources.

1. Common Data Sources

Data type Typical content Main value
Web text News, blogs, forums, encyclopedias Broad coverage of natural-language knowledge
Books Novels, textbooks, professional works Long-form text and systematic knowledge
Academic papers Research papers, technical reports Specialized knowledge and scientific expression
Technical documentation API docs, developer manuals Programming and engineering knowledge
Code data Git repositories, code examples Programming language and code generation ability
Q&A data Questions, discussions, answers Question understanding and answering patterns
Multilingual corpora Text in different languages Multilingual understanding and generation ability

Different data types do not contribute equally to a model.

For example:

  • Books help with learning long-form structure and coherent expression.
  • Code data helps with learning programming language syntax and program structure.
  • Academic papers help with learning specialized terminology and technical expression.
  • Multilingual corpora help improve cross-lingual modeling capabilities.

2. Data Scale and Data Quality

Data scale is important, but data scale does not equal the amount of effective information.

Suppose a training dataset contains:

  • large numbers of duplicated web pages;
  • low-quality machine-generated text;
  • meaningless characters;
  • spam advertisements;
  • incorrect code snippets;
  • articles with highly similar content.

Even if such data can increase the number of tokens, it may not effectively improve model capabilities.

Therefore, one of the core goals of pretraining data engineering is to increase the proportion of effective data.

Data value can be roughly understood as:

\text{Effective training value} \approx \text{Data scale} \times \text{Data quality} \times \text{Data diversity}

This is only a conceptual expression to aid understanding; it is not a general formula that can be directly calculated in actual training.

3. Why Is Data Distribution Important?

The capabilities a model ultimately learns are closely related to the training data distribution.

For example:

  • If the proportion of code is too low, it may affect code generation capabilities.
  • Insufficient high-quality Chinese corpus may affect Chinese comprehension and expression.
  • Insufficient domain-specific corpus may lead to limited coverage of professional knowledge.
  • If data is overly concentrated in certain content types, it may cause distribution bias in the model.

For training data, we should not only pursue scale but also consider the proportions across different domains, languages, text lengths, and content types.


III. Pretraining Data Processing Workflow

Raw data usually cannot be fed directly into model training and needs to go through a series of processing steps.

A typical data processing workflow is as follows:

flowchart TB
    A["Raw data collection"] --> B["Format parsing and text extraction"]
    B --> C["Content cleaning"]
    C --> D["Quality filtering"]
    D --> E["Duplicate data detection and deduplication"]
    E --> F["Sensitive and non-compliant content handling"]
    F --> G["Data mixing and sampling"]
    G --> H["Tokenizer encoding"]
    H --> I["Token sequence preparation"]
    I --> J["Training dataset"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

1. Format Parsing and Text Extraction

Different data sources may use different formats:

  • HTML
  • PDF
  • Markdown
  • JSON
  • XML
  • Plain text
  • Source code files

First, this content needs to be parsed into text sequences that the model can process.

Web page data especially requires handling of:

  • HTML tags;
  • navigation bars and footers;
  • advertising areas;
  • page templates;
  • irrelevant scripts and styles;
  • repeated page elements.

If raw web page content is fed directly into the model, it may cause a large number of invalid tokens to consume training resources.

2. Data Cleaning

Data cleaning is mainly used to remove obviously invalid or low-quality content.

Common operations include:

  • Deleting empty text;
  • Deleting garbled text;
  • Filtering based on the proportion of abnormal characters;
  • Cleaning up repeated punctuation;
  • Removing meaningless template text;
  • Handling malformed content;
  • Filtering out obvious spam information.

However, more aggressive cleaning is not always better.

For example, indentation, special symbols, comments, and Markdown markup in code may all contain valid information. If ordinary natural language cleaning rules are applied to code data, they may damage the code structure.

Therefore, different cleaning strategies usually need to be developed according to data type.

3. Data Quality Filtering

Quality filtering is more complex than basic cleaning.

Common methods include:

Rule-based filtering

Screening based on rules such as text length, character ratio, language, repetition rate, and format.

Classifier-based filtering

Using a quality classification model to determine whether text has training value.

Model-assisted scoring

Using models to evaluate text readability, coherence, and information density.

Domain-specific quality filtering

Developing specialized standards for data such as code, academic papers, and technical documents.

It should be noted that quality filtering models themselves may also have biases. If filtering rules are overly biased toward a certain language, writing style, or mode of expression, they may reduce data diversity.

4. Data Deduplication

Deduplication is an important step in pretraining data engineering.

Duplicate data can cause several problems:

  • Waste training compute resources;
  • Cause the model to overfit to repeated content;
  • Cause certain viewpoints or knowledge to be over-reinforced;
  • Increase the risk of data leakage between the training set and the evaluation set.

Deduplication is usually divided into two levels.

Type Target Common methods
Exact deduplication Identical text Hash, content fingerprint
Approximate deduplication Highly similar text MinHash, SimHash, locality-sensitive hashing

Exact deduplication can use text hashing:

h = Hash(\text{text})

If two pieces of text produce the same hash value after normalization, they can be considered duplicate text.

Approximate deduplication is more complex.

For example, two articles may differ only in title, punctuation, or a small number of words, but their main content is basically the same. In this case, text similarity can be estimated using methods such as Shingling and MinHash.

5. Data Mixing and Sampling

Data from different sources needs to be mixed according to a certain strategy.

For example:

Data type Example weight
General web pages 45%
Books and long-form text 15%
Code 20%
Academic and technical documents 15%
Other high-quality corpora 5%

The above proportions are only used to illustrate the data mixing mechanism and do not represent the actual training mix of any mainstream model.

Data mixing weights directly affect how frequently the model is exposed to different data types during training.

Actual training also needs to consider:

  • Data quality;
  • Data scale;
  • Domain coverage;
  • Language distribution;
  • Repeated sampling;
  • Changes across training stages.

Some training schemes increase the sampling proportion of high-quality data in the later stages of training to improve model performance.


IV. Tokenization: Converting Text into Model Input

Neural networks cannot directly process raw strings; text must first be converted into Token IDs.

This process is called Tokenization.

Suppose the input text is:

Large Language Models

The Tokenizer may split it into:

["Large", " Language", " Models"]

Then map each Token to an integer ID in the vocabulary:

[1024, 5832, 917]

The Token segmentation results and IDs here are only illustrative; the actual results depend on the specific Tokenizer.

1. Token ID and Embedding

After receiving a Token ID, the model maps it to a continuous vector through the Embedding matrix.

Let the vocabulary size be V, the hidden dimension be d, and the Embedding matrix be:

E \in \mathbb{R}^{V \times d}

For Token ID i, the corresponding vector is:

e_i = E[i]

Therefore, the Token ID itself is only a discrete index; what actually enters Transformer computation is the Embedding vector.

2. Why Is a Unified Tokenizer Needed?

The pretraining and inference stages must use compatible Tokenizers.

If one Tokenizer is used during training and another during inference, even if both process the same piece of text, they may produce different Token segmentations and IDs.

This causes the input representation to be inconsistent with the vocabulary definition used during model training.

A Tokenizer is usually released and used together with the model weights, vocabulary, and special Token configuration.


V. Core Objective of Pretraining: Next Token Prediction

For current mainstream Decoder-only large language models, the most typical pretraining objective is to predict the next Token.

Given a Token sequence:

X=(x_1,x_2,\ldots,x_T)

The model learns:

P(x_t\mid x_1,x_2,\ldots,x_{t-1})

That is, predict the Token at the current position based on the context before the current position.

1. Autoregressive Probability Decomposition

According to the chain rule of probability, the joint probability of the entire sequence can be decomposed as:

P(x_1,x_2,\ldots,x_T) = \prod_{t=1}^{T} P(x_t\mid x_{<t})

where:

x_{<t}=(x_1,x_2,\ldots,x_{t-1})

This means the model only needs to learn the conditional probability distribution at each position to perform probabilistic modeling of the entire sequence.

2. Input and Label Misalignment

Suppose the original Token sequence is:

[A, B, C, D, E]

During training, it is usually constructed as:

Input Position Model Input Prediction Target
1 A B
2 A, B C
3 A, B, C D
4 A, B, C, D E

Corresponding inputs and labels:

input_ids = tokens[:, :-1]
labels = tokens[:, 1:]

Model input:

[A, B, C, D]

Training labels:

[B, C, D, E]

The model predicts the next Token at each position.

3. Why can training be parallelized?

Although the model learns autoregressive prediction, the training phase does not need to execute Token by Token like the generation phase.

The reason is: the training data already provides the complete Token sequence.

Through the causal attention mask (Causal Mask), the model can simultaneously compute prediction results for multiple positions while preventing each position from accessing future Tokens.

For example:

Position 1: can only see A
Position 2: can only see A, B
Position 3: can only see A, B, C
Position 4: can only see A, B, C, D

The causal attention mask usually appears as a lower-triangular visible region.

flowchart TB
    A["Complete Token Sequence"] --> B["Construct shifted inputs and labels"]
    B --> C["Transformer parallel computation"]
    C --> D["Apply Causal Mask"]
    D --> E["Predict next Token at each position"]
    E --> F["Compute prediction loss"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49

Here, two concepts need to be distinguished:

  • Autoregressive constraint: each position can only use historical context to predict the current Token.
  • Parallel training: after the complete training sequence is known, outputs for multiple positions can be computed simultaneously.

The two are not contradictory.


6. Loss Function: How Does the Model Measure Prediction Performance?

The model outputs a prediction score for each Token in the vocabulary, called Logits.

Let the vocabulary size be V, and the output at position t be:

z_t \in \mathbb{R}^{V}

Convert it into a probability distribution via Softmax:

P(x_t=i\mid x_{<t}) = \frac{e^{z_{t,i}}} {\sum_{j=1}^{V}e^{z_{t,j}}}

The model needs to increase the predicted probability corresponding to the correct Token.

1. Cross-Entropy Loss

For the true Token x_t, the loss at a single position is:

L_t=-\log P(x_t\mid x_{<t})

The higher the model's predicted probability for the correct Token, the lower the loss.

For example:

Predicted Probability of Correct Token Cross-Entropy Loss
0.9 0.105
0.5 0.693
0.1 2.303
0.01 4.605

It can be seen that the more uncertain the model is about the correct Token, the larger the loss.

2. Loss for the Entire Sequence

For a sequence of length T, the average cross-entropy loss can be expressed as:

L=-\frac{1}{T}\sum_{t=1}^{T} \log P(x_t\mid x_{<t})

In actual training, it is usually also necessary to handle padding tokens, special tokens, and samples of different lengths.

Therefore, a more general form is:

L= -\frac{ \sum_{b,t}m_{b,t} \log P(x_{b,t}\mid x_{b,<t}) }{ \sum_{b,t}m_{b,t} }

where:

  • b: sample index within the batch.
  • t: token position.
  • m_{b,t}: valid-token mask.
  • Denominator: the number of valid tokens participating in the loss computation.

3. Why Use Cross-Entropy?

The goal of pretraining is to maximize the likelihood of the training data under the model:

\theta^* = \arg\max_{\theta} \prod_{t=1}^{T} P_{\theta}(x_t\mid x_{<t})

To facilitate optimization, this is usually converted to negative log-likelihood:

\theta^* = \arg\min_{\theta} -\sum_{t=1}^{T} \log P_{\theta}(x_t\mid x_{<t})

This is directly related to the cross-entropy loss.

From an optimization perspective, by reducing cross-entropy, the model continuously increases the predicted probability of the true token in the corresponding context.


VII. How Does the Transformer Participate in Pretraining?

Pretraining does not train a separate prediction module; rather, it trains the parameters of the entire model.

Taking a decoder-only Transformer as an example, the overall computation process is as follows:

flowchart TB
    A["Token ID Sequence"] --> B["Token Embedding"]
    B --> C["Positional Encoding or RoPE"]
    C --> D["Transformer Block"]
    D --> E["Final Hidden State"]
    E --> F["Output Projection Layer"]
    F --> G["Logits"]
    G --> H["Softmax Probability Distribution"]
    H --> I["Compute Next Token Loss"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

1. Token Embedding

Convert token IDs into vector representations.

X=Embedding(TokenIDs)

If the hidden dimension is d and the sequence length is T, the input tensor shape is usually:

X\in\mathbb{R}^{B\times T\times d}

where B is the batch size.

2. Transformer Block

The Transformer Block is the core structure through which the model learns contextual relationships.

A typical Pre-LN Transformer Block can be abstracted as:

H'=H+Attention(LN(H))
H_{out}=H'+FFN(LN(H'))

where:

  • LN: LayerNorm.
  • Attention: causal self-attention.
  • FFN: feed-forward neural network.
  • Residual connection: helps information and gradient propagation.

Different models may differ in normalization methods, activation functions, attention structures, positional encoding, and FFN structures.

3. Output Projection Layer

After the Transformer outputs the hidden states, they need to be mapped to the vocabulary dimension through the output projection layer.

Let the final hidden state be:

H\in\mathbb{R}^{B\times T\times d}

The output weights are:

W_{out}\in\mathbb{R}^{d\times V}

Then:

Z=HW_{out}

This yields:

Z\in\mathbb{R}^{B\times T\times V}

where V is the vocabulary size.

Each position produces a logits vector of length V, used to predict the next token.

Some models use weight tying between the input embedding and the output projection to reduce the number of parameters.


8. Backpropagation: How Does the Model Update Parameters?

After obtaining the loss, the model needs to compute the gradients of the loss with respect to the parameters.

Let the model parameters be:

\theta=\{W_1,W_2,\ldots,W_n\}

The training objective is:

L(\theta)

Compute via backpropagation:

\nabla_{\theta}L(\theta)

Then use an optimizer to update the parameters.

Taking the most basic gradient descent as an example:

\theta_{t+1} = \theta_t-\eta\nabla_{\theta}L(\theta_t)

Where:

  • \theta_t: Current model parameters.
  • \eta: Learning rate.
  • \nabla_{\theta}L: Gradient of the loss function with respect to the parameters.

Practical large language model training usually uses optimizers such as AdamW, together with techniques such as learning rate warmup, decay strategies, and gradient clipping.

Complete training loop:

flowchart TB
    A["Prepare Training Batch"] --> B["Model Forward Pass"]
    B --> C["Compute Cross-Entropy Loss"]
    C --> D["Backpropagation"]
    D --> E["Compute Parameter Gradients"]
    E --> F["Gradient Clipping and Processing"]
    F --> G["Optimizer Updates Parameters"]
    G --> H["Update Learning Rate Scheduler"]
    H --> I{"Is Training Finished?"}
    I -->|"No"| A
    I -->|"Yes"| J["Save Model Checkpoint"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49