Back to series
· 14 min read

Text Chunking Strategies: From Document Chunking to Core Techniques for High-Quality RAG Retrieval


I. What Is Chunking?

Chunking (text splitting) refers to breaking a long document into multiple smaller text units according to certain rules, so that each unit can be processed, vectorized, stored, and retrieved independently.

In a RAG system, embedding models usually have an input length limit, while a user query often only needs a small portion of a document. Therefore, the full document must first be split into fragments suitable for retrieval.

For example, the original document:

Transformer is a neural network architecture based on the attention mechanism.

It consists of components such as the Encoder and Decoder.
Self-Attention is used to establish relationships between different tokens.
FFN is used to apply a non-linear transformation to the representation at each position.

Transformer has been widely used in natural language processing and multimodal models.

After chunking:

Chunk 1:
Transformer is a neural network architecture based on the attention mechanism.

Chunk 2:
It consists of components such as the Encoder and Decoder.
Self-Attention is used to establish relationships between different tokens.

Chunk 3:
FFN is used to apply a non-linear transformation to the representation at each position.
Transformer has been widely used in natural language processing and multimodal models.

Each chunk can independently generate an embedding and be saved to a vector database.

1.1 Where Chunking Fits in RAG

Chunking sits after document parsing and before embedding; it is an important step in knowledge ingestion.

flowchart TD
    A["Original Document"] --> B["Document Parsing"]
    B --> C["Structured Text"]
    C --> D["Chunking Text Splitting"]
    D --> E["Embedding Vectorization"]
    E --> F["Vector Database"]
    F --> G["Retrieve Relevant Chunks"]
    G --> H["Context Construction"]
    H --> I["LLM Generates Answer"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Chunking is not just about controlling text length; it also involves questions of knowledge boundaries, semantic completeness, context inheritance, and retrieval granularity.

II. Why Is Chunking So Important?

The chunking strategy directly affects the retrieval quality of RAG.

The same document may yield completely different retrieval results when chunked in different ways.

2.1 Chunks That Are Too Large

Suppose a chunk contains 3,000 tokens.

Advantages:

  • Retains more context.
  • Section content is relatively complete.
  • Cross-sentence relationships are not easily lost.

Disadvantages:

  • The embedding has to summarize more topics.
  • Irrelevant content may dilute the target semantics.
  • A retrieval hit brings in a large amount of irrelevant text.
  • Consumes more of the context window.
  • Pinpointing a specific knowledge fragment is harder.

For example, a user asks:

What is the default port of Redis?

If a single chunk contains Redis installation, configuration, persistence, clustering, and performance tuning all at once, its vector may not sufficiently emphasize the local detail of the default port.

2.2 Chunks That Are Too Small

Suppose a chunk contains only 30 tokens.

Advantages:

  • Content is focused.
  • Retrieval granularity is finer.
  • Less irrelevant content.
  • Specific knowledge is easier to locate.

Disadvantages:

  • Necessary context may be lost.
  • Subject and referential relationships break easily.
  • Tables and code are easily split apart.
  • An isolated fragment may not express a complete fact.
  • Retrieval needs more chunks to reconstruct the full information.

For example:

Chunk 1:
Redis is an in-memory data structure store.

Chunk 2:
The default port is 6379.

When the user asks about the default port, Chunk 2 contains the answer but does not explicitly state that this port belongs to Redis.

If only Chunk 2 is retrieved, the model may be unable to determine accurately what it refers to.

2.3 The Core Trade-offs of Chunking

A chunking strategy must strike a balance among multiple objectives.

Goal Chunks too large Chunks too small
Semantic completeness Easier to preserve Easily lost
Retrieval precision May be diluted by irrelevant content Usually more focused
Context completeness Relatively good May be insufficient
Retrieval targeting Coarser granularity Finer granularity
Token cost Higher per result May require more results
Index size Relatively small Relatively large

To be clear: there is no optimal Chunk Size that applies to every knowledge base.

The optimal chunking granularity depends on document structure, the Embedding model, query types, retrieval strategy, and the context budget for generation.

III. Core Parameters of Chunking

Chunking usually involves multiple parameters, and Chunk Size should not be the only focus.

3.1 Chunk Size

Chunk Size indicates the target length of a single chunk.

Common units:

  • Character: number of characters.
  • Word: number of words.
  • Token: number of model Tokens.
  • Sentence: number of sentences.
  • Paragraph: number of paragraphs.

In LLM applications, Token is usually the recommended unit of length.

Reasons:

  1. Embedding models usually have Token input limits.
  2. An LLM's context window is measured in Tokens.
  3. There is no fixed conversion ratio between the number of Chinese/English characters and the number of Tokens.
  4. Code, numbers, and special symbols may produce different Token counts.

Common experimental ranges:

Scenario Initial test range
FAQ and short Q&A 100–300 Tokens
General knowledge documents 300–600 Tokens
Technical documentation 400–800 Tokens
Complex chapter content 600–1200 Tokens
Long-form analysis and reports 800–1500 Tokens

These are only starting points for experimentation, not universal standards. The final choice should be based on the limits of the target Embedding model, actual retrieval performance, and generation cost.

3.2 Chunk Overlap

Overlap indicates the length of text retained repeatedly between adjacent Chunks.

For example:

  • Chunk Size: 500 Tokens.
  • Overlap: 100 Tokens.

Adjacent chunks will share approximately 100 Tokens of content.

Original text:
[A B C D E F G H I J]

Chunk 1:
[A B C D E]

Chunk 2:
[D E F G H]

Chunk 3:
[G H I J]

Characters or fragments are used here to illustrate the overlap relationship; actual systems usually calculate it by Token count or text boundaries.

The main purposes of Overlap:

  • Reduce information loss at chunk boundaries.
  • Preserve the context of sentences that cross boundaries.
  • Improve the probability of recalling knowledge near boundaries.
  • Mitigate cases where entities and relationships are split apart.

But Overlap also has side effects:

  • Content is stored repeatedly.
  • More Embedding calls are required.
  • Vector database capacity increases.
  • Retrieval results may contain duplicate fragments.
  • The same content may be input repeatedly when assembling context.

Therefore, a larger Overlap is not always better.

3.3 Overlap Ratio

We can define:

R_{overlap}=\frac{O}{S}

Where:

  • O: Overlap length.
  • S: Chunk Size.

For example:

S=500,\quad O=100

Then:

R_{overlap}=20\%

For continuous text with a stable structure, you can start testing with a relatively low overlap ratio; for content with obvious cross-sentence dependencies, you can increase the overlap appropriately.

However, for content that has already been split along complete chapters, tables, or semantic boundaries, a large fixed Overlap may not be necessary.

3.4 Number of Chunks and Storage Overhead

Assume:

  • The total document length is N Tokens.
  • Chunk Size is S.
  • Overlap is O.

For fixed-length chunking with a fixed step size, the step size is:

Stride=S-O

When N>S, the number of chunks is approximately:

K=\left\lceil\frac{N-S}{S-O}\right\rceil+1

When N\leq S:

K=1

For example:

N=10000,\quad S=500,\quad O=100

Then:

K=\left\lceil\frac{10000-500}{400}\right\rceil+1=25

If Overlap is not used:

K=\left\lceil\frac{10000}{500}\right\rceil=20

As can be seen, Overlap increases the number of chunks, which in turn increases vector storage and retrieval overhead.

The formula here is suitable for estimating fixed-length chunking. Actual semantic chunking may produce a different number due to paragraph lengths and boundary adjustments.

4. Common Chunking Strategies

There are many ways to implement chunking, and different strategies are suitable for different types of documents.

4.1 Fixed-size Chunking

Fixed-size chunking is the simplest strategy.

Split the text continuously according to a preset length, keeping each chunk as close to the same length as possible.

For example:

Original text:
A B C D E F G H I J K L

Chunk Size = 4
Overlap = 0

Chunk 1:
A B C D

Chunk 2:
E F G H

Chunk 3:
I J K L

Implementation logic:

def fixed_chunk(text, chunk_size=500, overlap=0):
    if chunk_size <= 0:
        raise ValueError("chunk_size must be positive")

    if overlap < 0 or overlap >= chunk_size:
        raise ValueError("overlap must be >= 0 and < chunk_size")

    chunks = []
    stride = chunk_size - overlap

    for start in range(0, len(text), stride):
        chunk = text[start:start + chunk_size]

        if chunk:
            chunks.append(chunk)

        if start + chunk_size >= len(text):
            break

    return chunks

The code above chunks by character length and is intended to illustrate the algorithm logic. In production, you should use the tokenizer corresponding to the target model to calculate token length.

Advantages:

  • Simple to implement.
  • Fast processing.
  • Length is easy to control.
  • Suitable for batch processing.
  • Parameters are easy to tune.

Disadvantages:

  • May cut sentences apart.
  • May split paragraphs.
  • May break code and tables.
  • Cannot understand the semantic structure of the document.

Applicable scenarios:

  • Plain text.
  • Logs with simple structure.
  • Large-scale text preprocessing.
  • Scenarios with high requirements for chunking speed.

4.2 Recursive Character Chunking

Recursive chunking attempts larger text boundaries first, and only uses finer-grained separators when the current boundary cannot satisfy the length limit.

Common separator priority:

Paragraph separator
    ↓
Line break
    ↓
Sentence separator
    ↓
Space
    ↓
Character

Typical strategy:

  1. Split by paragraphs first.
  2. If a paragraph is still too long, split it by sentences.
  3. If a sentence is still too long, split it by finer text boundaries.
  4. Finally, ensure the chunk length meets the limit.

Illustration:

flowchart TD
    A["Input long text"] --> B["Split by paragraph"]
    B --> C{"Is the chunk too long?"}
    C -->|"No"| D["Keep current chunk"]
    C -->|"Yes"| E["Split by sentence"]
    E --> F{"Is the chunk too long?"}
    F -->|"No"| G["Keep sentence chunk"]
    F -->|"Yes"| H["Continue splitting by spaces or characters"]
    H --> I["Check length limit"]
    I --> J["Output Chunk"]
    D --> J
    G --> J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Advantages:

  • Preserve natural language boundaries as much as possible.
  • Better aligned with text structure than fixed-length chunking.
  • Relatively simple to implement.
  • Can accommodate paragraphs of different lengths.

Disadvantages:

  • Depends on delimiter quality.
  • Limited understanding of complex document structures.
  • Oversized tables, code blocks, and special formats still require dedicated handling.
  • Does not guarantee that chunks are semantically complete.

Use cases:

  • General knowledge documents.
  • Technical articles.
  • Markdown documents.
  • Long texts with relatively clear structure.

4.3 Sentence-based Chunking

Sentence-based chunking organizes text according to sentence boundaries.

Unlike fixed-length chunking, it tries not to cut content in the middle of a sentence.

For example:

Original text:

Redis is an in-memory data structure store.
It supports data types such as strings, hashes, lists, and sets.
Redis is commonly used for caching, message queues, and real-time data processing.

They can be combined based on sentence count or token budget:

Chunk 1:
Redis is an in-memory data structure store.
It supports data types such as strings, hashes, lists, and sets.

Chunk 2:
Redis is commonly used for caching, message queues, and real-time data processing.

Advantages:

  • Sentences are relatively complete.
  • Suitable for question-answer text.
  • Less likely to break local semantics.

Disadvantages:

  • Sentence lengths vary widely.
  • A single very long sentence may exceed the model limit.
  • There may be strong cross-sentence dependencies between sentences.
  • Depends on accurate sentence boundary detection.

Chinese requires special attention to:

  • Periods.
  • Question marks.
  • Exclamation marks.
  • Semicolons.
  • Punctuation inside quotation marks.
  • Periods in abbreviations and numbers.

Simply splitting on periods may break code, English abbreviations, and complex punctuation structures.

Use cases:

  • FAQs.
  • Short knowledge entries.
  • Corpora consisting mainly of single-sentence facts.
  • Texts that require high boundary integrity.

4.4 Paragraph-based Chunking

Paragraph-based chunking organizes content according to the paragraph boundaries of the original text.

A paragraph usually develops around a relatively complete topic, so it naturally has a certain degree of semantic completeness.

For example:

Paragraph 1:
Redis is an open-source in-memory data structure store,
supporting multiple data structures.

Paragraph 2:
Redis is commonly used for caching, message queues, and real-time data processing,
and has high read/write performance.

Each paragraph can be treated as a chunk, or multiple short paragraphs can be combined.

Advantages:

  • Preserves the original document structure.
  • Reduces semantic breaks between sentences.
  • Low implementation cost.
  • Suitable for document-based knowledge bases.

Disadvantages:

  • Paragraph lengths may vary widely.
  • A paragraph may contain multiple topics.
  • Some documents have poor paragraph segmentation quality.

In practice, paragraph-based chunking is often combined with a maximum token length:

  • Short paragraphs: merge adjacent paragraphs.
  • Paragraphs of appropriate length: use directly as a chunk.
  • Very long paragraphs: continue recursive splitting.

4.5 Semantic Chunking

Semantic chunking determines chunk boundaries based on semantic shifts in the text, rather than relying only on character count, sentence count, or paragraph count.

Core idea:

If adjacent text expresses the same topic, prefer to keep it in the same Chunk; if the topic changes significantly, consider splitting at the boundary.

Typical process:

flowchart TD
    A["Input document"] --> B["Sentence segmentation"]
    B --> C["Generate sentence embeddings"]
    C --> D["Compute similarity between adjacent sentences"]
    D --> E["Identify semantic change points"]
    E --> F["Incorporate length constraints"]
    F --> G["Generate semantic chunks"]
    G --> H["Check completeness"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style D fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

A common implementation approach:

Step 1: Sentence Segmentation

Convert the document into a sequence of sentences:

S=\{s_1,s_2,\ldots,s_n\}

Step 2: Generate Sentence Vectors

Use an embedding model to obtain:

E=\{e_1,e_2,\ldots,e_n\}

where:

e_i=Embedding(s_i)

Step 3: Compute Similarity Between Adjacent Sentences

Use cosine similarity:

Sim(e_i,e_{i+1})= \frac{e_i\cdot e_{i+1}} {\|e_i\|\|e_{i+1}\|}

Step 4: Identify Semantic Boundaries

If the semantic similarity between adjacent sentences drops significantly, it may indicate that the topic has changed.

For example:

Sentence 1: Redis is an in-memory data structure store.
Sentence 2: Redis supports data structures such as strings, lists, and hashes.
Sentence 3: Redis is often used in caching scenarios.
Sentence 4: Transformer uses Self-Attention to model contextual relationships.

Sentences 1–3 share the same topic, while sentence 4 shifts to another topic, so you can consider splitting between sentence 3 and sentence 4.

Advantages:

  • Better aligns with topic boundaries.
  • Reduces mixing across topics.
  • Helps with text that has irregular content structure.

Disadvantages:

  • Requires generating additional sentence embeddings.
  • Higher computational cost.
  • Similarity thresholds need to be tuned.
  • Similarity between adjacent sentences may not accurately represent topic boundaries.
  • May incorrectly split apart content with causal relationships.

Note: semantic similarity is only one signal for chunk boundaries, not full semantic understanding. Definitions, examples, counterexamples, and conclusions may have low local similarity while still belonging to the same knowledge unit.

Therefore, semantic chunking usually also needs to combine chapter structure, paragraph boundaries, and maximum length constraints.

4.6 Structure-aware Chunking

Structure-aware chunking splits according to the document's own structure.

For example:

  • Markdown: split by heading hierarchy.
  • HTML: split by DOM structure.
  • PDF: split by layout and sections.
  • Word: split by headings, paragraphs, and tables.
  • Code: split by functions, classes, and modules.
  • Technical manuals: split by chapters, steps, and parameter descriptions.

For example, Markdown:

# Redis

## I. Basic Concepts

Redis is an open-source in-memory data structure store.

## II. Data Types

### 2.1 String

String is the most basic data type in Redis.

### 2.2 Hash

Hash is used to store mappings between fields and values.

This can produce:

Chunk 1:
Redis
I. Basic Concepts
Redis is an open-source in-memory data structure store.

Chunk 2:
Redis
II. Data Types
2.1 String
String is the most basic data type in Redis.

Chunk 3:
Redis
II. Data Types
2.2 Hash
Hash is used to store mappings between fields and values.

Here, the Chunk not only preserves the body text but also inherits the section path it belongs to.

For example:

{
  "chunk_id": "chunk_001",
  "text": "String 是 Redis 最基本的数据类型。",
  "metadata": {
    "document_title": "Redis",
    "section_path": [
      "Redis",
      "二、数据类型",
      "2.1 String"
    ]
  }
}

Advantages:

  • Preserves document hierarchy.
  • Makes it easy to restore context during retrieval.
  • Suitable for technical documentation and knowledge manuals.
  • Handles tables, code, and lists better.

Disadvantages:

  • Requires reliable document parsing results.
  • Different document formats require different parsing logic.
  • Abnormal document structure may affect chunking quality.

For enterprise knowledge bases and technical knowledge bases, structured chunking is usually a strategy well worth prioritizing.

V. Advanced Chunking Strategies

In addition to basic chunking methods, there are also some advanced strategies designed for complex retrieval scenarios.

5.1 Parent-Child Chunking

Parent-Child Chunking is a parent-child chunking strategy.

It organizes documents into knowledge units of different granularities:

  • Parent Chunk: a larger parent text block that preserves the full context.
  • Child Chunk: a smaller child text block responsible for precise retrieval.

For example:

Parent Chunk:
Redis Data Types

    Child 1:
    String is the most basic data type in Redis.

    Child 2:
    Hash is used to store mappings between fields and values.

    Child 3:
    List is an ordered list of strings.

During retrieval, match Child Chunks first; when generating an answer, the corresponding Parent Chunk can be returned.

flowchart TD
    A["Complete document"] --> B["Parent Chunk"]
    B --> C["Child Chunk 1"]
    B --> D["Child Chunk 2"]
    B --> E["Child Chunk 3"]
    C --> F["Embedding"]
    D --> F
    E --> F
    F --> G["Vector retrieval"]
    G --> H["Match Child"]
    H --> I["Locate Parent"]
    I --> J["Return full context"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style D fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style E fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

This approach resolves a core contradiction:

Retrieval requires smaller granularity, while generation requires sufficient context.

Advantages:

  • Improves local retrieval precision.
  • Preserves parent-level context.
  • Reduces the problem of small fragments lacking a subject.
  • Supports multi-level knowledge organization.

Disadvantages:

  • Requires maintaining parent-child relationships.
  • Requires an additional Parent lookup after retrieval.
  • Returning parent content may increase token consumption.
  • If the parent is too large, it may still bring in irrelevant content.

Real-world systems can also adopt a multi-level structure:

Document
    ↓
Section
    ↓
Parent Chunk
    ↓
Child Chunk

Retrieval matches the smallest-granularity knowledge unit, and then the task determines which level of context to return.

5.2 Small-to-Big Retrieval

Small-to-Big Retrieval is similar in concept to Parent-Child, but places more emphasis on retrieval granularity and context expansion strategies.

Basic process:

  1. Use smaller Chunks for retrieval.
  2. Obtain the position of the matched fragment.
  3. Expand adjacent text based on the fragment position.
  4. Combine them into a larger context.
  5. Provide the expanded context to the model.

For example:

Original document:

Chunk 1: Basic concepts of Redis.
Chunk 2: Data types in Redis.
Chunk 3: Redis persistence mechanisms.
Chunk 4: Redis high availability mechanisms.

After a retrieval hit on Chunk 3, Chunk 2 and Chunk 4 can also be fetched simultaneously according to business needs.

This method does not require a strict parent-child tree to be established in advance; it can also dynamically expand based on document order, section paths, and adjacent positions.

Note: The expansion scope must be constrained by the Token budget and cannot unconditionally return the entire chapter.

5.3 Agentic Chunking

Agentic Chunking uses a large language model to assist in determining text chunking boundaries.

Unlike traditional rule-based chunking, it attempts to understand:

  • What the current paragraph discusses.
  • Which sentences belong to the same topic.
  • Which content should be kept together.
  • Which information has independent retrieval value.

For example, given the input:

Redis is an in-memory data structure store.
It supports data types such as String, Hash, and List.
Redis is commonly used in caching scenarios.
Redis also supports RDB and AOF persistence.

The model might organize it as:

Chunk 1: Definition and data types of Redis.
Chunk 2: Typical application scenarios of Redis.
Chunk 3: Persistence mechanisms of Redis.

Pros:

  • Can leverage language models to understand complex semantics.
  • Flexible for documents with irregular structure.
  • Can incorporate domain rules to define knowledge boundaries.

Cons:

  • Higher cost of model invocation.
  • Chunking results may be unstable.
  • Easily affected by prompt design.
  • Requires additional quality evaluation.
  • Throughput may be limited for large-scale processing.

Agentic Chunking does not mean that every Chunk must be decided by an Agent. Production systems can enable model-assisted chunking only for documents that are complex, low-quality, or structurally abnormal.

5.4 Late Chunking

Late Chunking is a technical approach whose order differs from traditional chunking.

Traditional process:

Full text
    ↓
Chunking first
    ↓
Each Chunk embedded independently

The core idea of Late Chunking is:

First encode a longer context with the model, then construct the vector for each Chunk based on Token-level hidden representations.

Conceptual flow:

flowchart TD
    A["Full long text"] --> B["Tokenizer"]
    B --> C["Long-context encoding"]
    C --> D["Token-level hidden representations"]
    D --> E["Group by Chunk boundaries"]
    E --> F["Pool Token representations within each group"]
    F --> G["Generate Chunk Embedding"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style D fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Suppose the model produces Token-level hidden representations for the full text:

H=[h_1,h_2,\ldots,h_n]

A given Chunk corresponds to the Token range:

C_i=[a_i,b_i]

The Chunk vector can be generated by pooling:

e_i=Pool(h_{a_i},\ldots,h_{b_i})

Common pooling methods include Mean Pooling and others; the specific method depends on the implementation requirements of the Embedding model.

Compared with traditional independent chunking, Late Chunking allows a Chunk's vector representation to leverage broader contextual information, thereby mitigating the problem of some segments lacking a subject and context.

However, note:

  • The model must support sufficiently long context encoding.
  • Token-level hidden states must be correctly obtainable.
  • The pooling method must be compatible with the model's training method.
  • Hidden states from arbitrary generative models cannot simply be treated as high-quality Embeddings.
  • Long-text encoding may incur high computational costs.
  • It cannot automatically fix incorrect chunk boundaries.

Late Chunking changes the way Chunk vectors are generated; it does not mean that chunking can be completely eliminated.

6. Chunking Strategies for Special Content

Ordinary text can use paragraph-based or semantic chunking, but technical documents also contain a large amount of special content.

Without targeted handling, the original semantics are easily broken.

6.1 Table Chunking

Tables usually have row and column relationships and cannot simply be split by character length.

For example:

Product Price Stock
A 100 20
B 200 15
C 300 10

If you only cut out:

A 100 20
B 200 15

you may lose the column names.

A more reasonable approach is to keep the table header:

Table: Product inventory information

Product | Price | Stock
A | 100 | 20
B | 200 | 15

For very long tables, rows can be grouped, but each Chunk should include the necessary table header and units.

It is recommended to preserve:

  • Table title.
  • Multi-level headers.
  • Units.
  • Row and column information.
  • Page numbers.
  • The section the table belongs to.

6.2 Code Chunking

Code has a clear syntactic structure.

It is not recommended to simply split code by a fixed number of characters.

For example:

class UserService:

    def get_user(self, user_id):
        return self.repository.find_by_id(user_id)

    def create_user(self, user):
        return self.repository.save(user)

More reasonable chunking units:

  • Module.
  • Class.
  • Function.
  • Method.
  • Code Block.

When chunking, preserve:

  • File path.
  • Class name.
  • Function name.
  • Programming language.
  • Dependencies.
  • Necessary import statements.
  • The module it belongs to.

For example:

{
  "chunk_id": "code_001",
  "language": "python",
  "file_path": "services/user_service.py",
  "symbol_type": "method",
  "class_name": "UserService",
  "symbol_name": "get_user",
  "text": "def get_user(self, user_id):\n    return self.repository.find_by_id(user_id)"
}

For large functions, you can split them into finer-grained chunks according to the abstract syntax tree (AST), but the necessary function signature and context should be preserved.

6.3 Mathematical Formula Chunking

Mathematical formulas must not be split arbitrarily.

For example:

Attention(Q,K,V)= softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V

If a formula is split into multiple fragments, the variable definitions and computational relationships may be lost.

Recommendations:

  • Keep the formula and explanatory text in the same Chunk.
  • Preserve the formula number.
  • Preserve variable definitions.
  • Preserve the conditions described before and after the formula.
  • Preserve the original LaTeX for long formulas.

6.4 List and Procedure Chunking

Operation manuals often contain consecutive steps:

1. Create a database.
2. Create a user.
3. Assign access permissions.
4. Configure connection parameters.
5. Test the connection.

Steps should not be split apart arbitrarily.

If an entire procedure has dependencies as a whole, it can be treated as a single knowledge unit.

If the procedure is too long, it can be split by stage, and the procedure title and stage information should be preserved in each Chunk.

6.5 FAQ Chunking

The natural knowledge unit of an FAQ is usually a question and its complete answer.

For example:

Question: What is the default port of Redis?

Answer: The default port of Redis is 6379.

It is recommended to keep the question and answer in the same Chunk.

If an Embedding is generated only for the answer, the question semantics may be missing during retrieval.