I. Core Concepts of Document Parsing and Knowledge Extraction
In a RAG system, document parsing and knowledge extraction are two closely related stages with different responsibilities.
Document parsing mainly addresses:
- What content does the document contain?
- In what form is the content organized?
- What relationships exist among text, tables, images, and layout?
- How can the document's original structure and order be restored?
Knowledge extraction mainly addresses:
- What key facts does the document contain?
- Which content has independent knowledge value?
- How are entities, attributes, relations, and events represented?
- How can the content be converted into data usable for downstream retrieval and reasoning?
Differences between the two:
| Dimension | Document Parsing | Knowledge Extraction |
|---|---|---|
| Core objective | Restore document content and structure | Extract valuable knowledge |
| Input | PDF, Word, HTML, images, etc. | Parsed document content |
| Processing target | Text, layout, tables, images | Entities, attributes, relations, facts |
| Output | Structured document | Knowledge records, entity relations, metadata |
| Key technologies | OCR, Layout Analysis, Parser | NER, Relation Extraction, LLM |
| Primary use | Content restoration and organization | Retrieval, question answering, knowledge reasoning |
Note: not every RAG system requires complex knowledge extraction. For ordinary document question answering, structure-preserving text parsing plus reasonable chunking may already be sufficient; for enterprise knowledge bases, complex business documents, and cross-document associative retrieval, the value of knowledge extraction becomes more apparent.
II. Overall Processing Pipeline
Document parsing and knowledge extraction typically consist of multiple processing stages.
flowchart TD
A["Raw Document<br/>PDF / Word / HTML / Images"] --> B["Document Identification & Classification"]
B --> C["Document Parsing"]
C --> D["Structure Recovery & Content Normalization"]
D --> E["Knowledge Extraction"]
E --> F["Quality Validation & Data Cleaning"]
F --> G["Structured Knowledge Data"]
G --> H["Chunking & Index Construction"]
H --> I["Vector Database / Search Index"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
The whole process can be divided into four core stages:
- Document ingestion: identify file type, source, and basic attributes.
- Document parsing: extract text, tables, images, and layout structure.
- Knowledge extraction: identify entities, relations, attributes, and key facts.
- Data governance: validate, deduplicate, standardize, and build retrieval indexes.
III. Document Ingestion and Type Identification
Document ingestion is the entry point of the entire knowledge processing pipeline.
Different document formats have different internal structures and parsing methods, so a single parsing strategy cannot handle every file.
3.1 Common Document Types
| Document type | Characteristics | Primary parsing method |
|---|---|---|
| Fixed page layout | PDF Parser, Layout Analysis | |
| DOCX | Structured Office document | XML parsing, Office Parser |
| XLSX | Worksheets, cells, and formulas | Spreadsheet Parser |
| PPTX | Slides, text boxes, and shapes | Presentation Parser |
| HTML | Tags and DOM structure | HTML Parser |
| Markdown | Headings, lists, code blocks | Markdown Parser |
| TXT | Plain text | Text reading |
| Images | Pixel data, no direct text structure | OCR, vision models |
| Scanned PDF | Composed of page images | PDF rendering, OCR |
Two types of PDF need to be distinguished in particular:
- Text-based PDF: the page contains extractable text objects that can be read by a PDF text parser.
- Scanned PDF: the page is essentially an image, and the text must be recognized through OCR.
There is also a hybrid PDF: some pages contain text objects while other pages consist of scanned images. A real-world system needs page-level detection and mixed-processing capability.
3.2 File identification cannot rely on extensions alone
File extensions are not necessarily trustworthy.
For example:
- A
.pdffile may actually be an image container or a corrupted file. - A
.docxfile is essentially a ZIP container holding XML files inside. - File extensions may have been modified manually.
- The same file may have a MIME Type that does not match its actual content.
Production systems usually combine the following information to determine file type:
- File extension.
- MIME Type.
- File header signature (Magic Number).
- Internal container structure.
- File content characteristics.
File ingestion also needs to handle size limits, corrupted files, compression bombs, malicious files and parsing resource limits.
3.3 Document metadata
Beyond the body content, the ingestion stage should also record document-level metadata.
For example:
{
"document_id": "doc_001",
"file_name": "technical_specification.pdf",
"file_type": "application/pdf",
"file_size": 2048000,
"source": "internal_knowledge_base",
"department": "engineering",
"version": "2.1",
"created_at": "2026-11-20T10:00:00Z",
"status": "pending"
}
Where:
document_id: unique document identifier.file_name: original file name.source: document source.department: the business department it belongs to.version: document version.status: current processing status.
Document metadata and body content should be managed separately.
The body is used for knowledge retrieval, while metadata is used for filtering, permission control, version management and data traceability.
IV. Core Technologies of Document Parsing
Document parsing is not simply converting a file into a string; it is about restoring, as far as possible, the content, hierarchy and spatial relationships of the original document.
4.1 Text Extraction
Text extraction is the most fundamental capability of document parsing.
Different document formats use different extraction methods:
- PDF: extract the text objects on the page along with their coordinates.
- DOCX: parse paragraphs, text and tables in the XML.
- XLSX: extract worksheets, cells, formulas and data ranges.
- PPTX: extract text boxes, titles and notes from slides.
- HTML: parse the DOM and extract the body content and structural tags.
- Markdown: parse headings, lists, quotes and code blocks.
For PDFs, text extraction usually requires not only obtaining the text content but also preserving the position of the text on the page.
For example:
{
"page": 1,
"text": "Transformer 是一种基于注意力机制的神经网络架构。",
"bbox": [100, 120, 520, 160],
"font_size": 14,
"font_name": "Arial"
}
bbox represents the bounding box of the text on the page, usually expressed as:
Preserving coordinates helps with subsequent judgments:
- Which pieces of text belong to the same paragraph.
- Which pieces of text belong to a particular table.
- Which pieces of text are headers or footers.
- How the reading order should be arranged.
- The hierarchical relationship between headings and body text.
If only the text is extracted and the coordinates are discarded, it may be impossible to accurately restore the original document structure.
4.2 OCR Optical Character Recognition
OCR (Optical Character Recognition) is used to recognize text from images.
It primarily solves the problem of being unable to extract text directly from scans, images and screenshots.
A typical OCR process:
flowchart TD
A["Input Image"] --> B["Image Preprocessing"]
B --> C["Text Region Detection"]
C --> D["Text Region Cropping"]
D --> E["Character Recognition"]
E --> F["Text Post-processing"]
F --> G["Recognition Result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The main steps of OCR include:
Step 1: Image Preprocessing
Common operations:
- Grayscale conversion.
- Binarization.
- Denoising.
- Skew correction.
- Resolution adjustment.
- Contrast enhancement.
Image quality directly affects the subsequent recognition performance.
Step 2: Text Detection
Locate text regions in the image.
The detection model needs to determine which regions contain text and the boundaries of each text region.
Common methods include:
- EAST.
- DBNet.
- CRAFT.
Step 3: Text Recognition
Perform character or sequence recognition on the detected text regions.
Common technical approaches include:
- CNN + RNN + CTC.
- Attention-based Recognition.
- Transformer-based Recognition.
Step 4: Text Post-processing
Handle formatting issues and errors in the recognition results:
- Punctuation restoration.
- Space and line break handling.
- Character error correction.
- Language rule validation.
- Text region order restoration.
OCR output should also usually retain confidence scores and coordinates to facilitate subsequent quality assessment.
For example:
{
"text": "人工智能技术",
"confidence": 0.98,
"bbox": [120, 80, 350, 120],
"page": 1
}
Note: OCR confidence is the recognition model's estimate of the result and is not equivalent to accuracy verified by humans.
For high-risk fields such as contract amounts, ID card numbers, and equipment numbers, content correctness cannot be judged based on confidence alone.
4.3 Layout Analysis
Layout analysis is used to understand the spatial layout and region types of document pages.
A document contains not only text but also:
- Titles.
- Body paragraphs.
- Tables.
- Images.
- Figure captions.
- Headers and footers.
- Page numbers.
- Formulas.
- Multi-column layouts.
The core task of layout analysis is to determine the type, position, and interrelationships of regions on a page.
For example:
┌────────────────────────────┐
│ Document Title │
├────────────────────────────┤
│ │
│ Body Paragraph │
│ Body Paragraph │
│ │
├────────────────────────────┤
│ Table │
│ ┌─────┬─────┬─────┐ │
│ │ A │ B │ C │ │
│ ├─────┼─────┼─────┤ │
│ │ 10 │ 20 │ 30 │ │
│ └─────┴─────┴─────┘ │
├────────────────────────────┤
│ Image │
│ Image Caption │
└────────────────────────────┘
Common layout element categories:
| Category | Description |
|---|---|
| Title | Document title |
| Section Header | Section title |
| Text | Body text |
| Table | Table |
| Figure | Image or chart |
| Caption | Image or table caption |
| List | List |
| Formula | Mathematical formula |
| Header | Page header |
| Footer | Page footer |
Common tools and techniques include:
- LayoutParser: an ecosystem of tools for layout analysis.
- Detectron2: an object detection and vision model framework that can be used for layout region detection.
- LayoutLM: a family of document understanding models that combine text, layout, and visual information.
- DocLayNet: a dataset for document layout analysis.
- PP-Structure: a tool suite for structured document parsing.
It is important to distinguish: layout analysis and OCR are not the same task.
OCR focuses on what the text says, while layout analysis focuses on what regions exist on the page, where those regions are, and how they are organized.
4.4 Table Extraction
Tables are the part of document parsing where errors are most likely to occur.
Plain text extraction might produce:
Product Quantity Unit Price
A 10 20
B 15 30
But the original table may contain a complex structure:
| Product | Sales Data | |
|---|---|---|
| Name | Quantity | Unit Price |
| A | 10 | 20 |
| B | 15 | 30 |
If only the plain text is retained, the following may be lost:
- The correspondence between headers and data.
- Merged cells.
- Row and column hierarchy.
- Multi-level headers.
- Unit information.
- Table titles and notes.
Therefore, table extraction requires not only recognizing text but also restoring the table structure.
A typical processing pipeline:
flowchart TD
A["Locate table region"] --> B["Recognize row and column structure"]
B --> C["Detect cell boundaries"]
C --> D["Recognize cell text"]
D --> E["Handle merged cells"]
E --> F["Associate headers with data"]
F --> G["Structured table"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The key to table extraction lies in restoring the logical structure, not just recognizing the characters in each cell.
For example:
{
"table_id": "table_001",
"headers": ["产品", "数量", "单价"],
"rows": [
["A", 10, 20],
["B", 15, 30]
]
}
For complex tables, you also need to preserve:
row_span: row-spanning information.col_span: column-spanning information.cell_bbox: cell coordinates.header_path: the path through multi-level headers.unit: the unit of a numeric value.caption: the table caption.
Pay special attention: numeric values in a table must be bound to their headers, units, and context.
For example, extracting 100 on its own does not mean much; you have to know whether it represents a quantity, an amount, a ratio, or a temperature.
4.5 Image and Chart Parsing
Images in documents may contain important knowledge:
- Architecture diagrams.
- Flowcharts.
- System topology diagrams.
- Business relationship diagrams.
- Bar charts and line charts.
- Product schematics.
- Screenshots and user interfaces.
Extracting only the text surrounding an image may lose a large amount of information.
Image parsing typically requires:
- Locating the image region.
- Extracting the image and its original coordinates.
- Identifying the image type.
- Recognizing the text in the image.
- Analyzing the structure and semantics of the image.
- Associating the image content with the body text and captions.
For charts, it is also necessary to identify:
- Coordinate axes.
- Data series.
- Legends.
- Units.
- Time ranges.
- Data points.
- Chart titles.
Vision-language models (VLMs) can help interpret complex images, but the descriptions they generate are the result of model inference and should not be treated directly as established facts from the original document.
From an engineering perspective, the original images, page positions, model descriptions and processing versions should be retained for review.
V. Document Structure Recovery and Standardization
After text, table and image extraction is complete, the logical structure of the document still needs to be recovered.
This is an important step in converting raw content into retrievable knowledge.
5.1 Why is structure recovery needed?
Consider the following content:
Chapter 1 System Architecture
1.1 Overall Design
The system adopts a microservices architecture.
1.2 Service Communication
Services communicate with each other via HTTP and message queues.
If the text is extracted directly, the result may be one continuous block of content.
But in reality, the document has a clear hierarchical relationship:
Chapter 1 System Architecture
1.1 Overall Design
The system adopts a microservices architecture.
1.2 Service Communication
Services communicate with each other via HTTP and message queues.
Structure recovery requires identifying:
- Heading levels.
- Paragraph boundaries.
- List levels.
- The chapter a table belongs to.
- The correspondence between images and captions.
- Page numbers and reading order.
Among these, reading order is especially important.
Two-column papers, newspapers and complex layouts may be arranged in an interleaved manner according to spatial position. Simply sorting text objects by their coordinates can easily mix content from different columns together.
Therefore, reading order recovery usually needs to take the following into comprehensive consideration:
- Page columns.
- Region types.
- Text block coordinates.
- Heading levels.
- Paragraph continuity.
- Document reading conventions.
5.2 Unified document intermediate representation
A production-grade system should not connect every file format directly to the downstream knowledge extraction module.
A more reasonable approach is to define a unified document intermediate representation (Document IR).
For example:
{
"document_id": "doc_001",
"title": "System Technical Design Document",
"pages": [
{
"page_number": 1,
"blocks": [
{
"block_id": "b001",
"type": "heading",
"level": 1,
"text": "Chapter 1 System Architecture",
"bbox": [80, 60, 500, 100]
},
{
"block_id": "b002",
"type": "paragraph",
"text": "The system adopts a microservices architecture.",
"parent_id": "b001",
"bbox": [80, 120, 500, 160]
},
{
"block_id": "b003",
"type": "table",
"caption": "Service Configuration Table",
"table_data": {
"headers": ["Service", "Port"],
"rows": [
["API", "8080"],
["Auth", "8081"]
]
}
}
]
}
]
}
The fields here are only examples; the actual schema should be designed according to business requirements.
The value of Document IR lies in:
- Shielding differences between underlying file formats.
- Unifying downstream processing interfaces.
- Preserving original page and region information.
- Supporting multiple knowledge extraction strategies.
- Supporting re-segmentation and re-indexing.
- Supporting problem localization and result traceability.
It is recommended to retain at least the following information:
| Field | Purpose |
|---|---|
| document_id | Unique document identifier |
| block_id | Unique content block identifier |
| parent_id | Parent content block |
| block_type | Content block type |
| text | Extracted text |
| page_number | Page number |
| bbox | Page coordinates |
| source | Original source |
| parser_version | Parser version |
| confidence | Recognition confidence |
| metadata | Extended metadata |
5.3 Content Cleaning and Normalization
Raw parsing results usually contain noise and need cleaning.
Common issues:
- Extra spaces.
- Repeated line breaks.
- Repeated headers and footers.
- Page numbers mixed into body text.
- OCR typos.
- Garbled text.
- Broken lines.
- Hyphenation splits.
- Duplicate paragraphs.
- Meaningless symbols.
For example, a PDF may contain:
A large language mod
el is a type of model based on the Transformer
architecture.
After cleaning:
A large language model is a type of model based on the Transformer architecture.
However, cleaning must not be excessive.
The following information should not be deleted arbitrarily:
- Code indentation.
- Mathematical formulas.
- Empty values in tables.
- List numbering.
- Section numbering.
- Minus signs and decimal points.
- Dates and units.
- Symbols with business meaning.
Especially in technical documents, the formatting of code, configurations, commands, and formulas is itself part of the knowledge.
Cleaning operations should preferably preserve a mapping between the original content and the normalized content to avoid losing traceability to the original text.
6. Core Technologies of Knowledge Extraction
Document parsing produces structured content, while knowledge extraction further extracts knowledge from the content that can be expressed independently, associated, and retrieved.
6.1 Entity Extraction (Named Entity Recognition)
Named entity recognition (NER) is used to identify objects with clear meaning in text.
Common entity types:
- Person.
- Organization.
- Location.
- Product.
- Technology.
- System.
- Time.
- Amount.
- Business object.
For example:
OpenAI released GPT-4 in 2023.
Entity extraction result:
{
"entities": [
{
"text": "OpenAI",
"type": "ORGANIZATION"
},
{
"text": "2023 年",
"type": "DATE"
},
{
"text": "GPT-4",
"type": "MODEL"
}
]
}
Entity recognition focuses not only on finding names but also on determining entity types, boundaries, and context.
Common technical approaches:
- Rule and dictionary matching.
- CRF.
- BiLSTM-CRF.
- BERT Token Classification.
- LLM-based structured extraction.
Traditional NER models are suitable for fixed entity types and large-scale stable tasks; LLMs are better suited to complex, open-ended entity definitions but require strict output validation.
6.2 Attribute Extraction
Entity extraction identifies objects, while attribute extraction identifies their characteristics.
For example:
GPT-4 is a large language model released by OpenAI in 2023,
supporting text and image input.
The following can be extracted:
{
"entity": "GPT-4",
"attributes": {
"developer": "OpenAI",
"release_year": 2023,
"model_type": "大语言模型",
"input_modalities": [
"text",
"image"
]
}
}
Common attributes include:
- Name.
- Type.
- Time.
- Status.
- Version.
- Specifications.
- Organization.
- Technical parameters.
- Scope of application.
Attribute extraction must take contextual constraints into account.
For example:
The system supports a maximum of 1000 concurrent connections.
You should not extract only:
{
"value": 1000
}
Instead, you should retain:
{
"entity": "system",
"attribute": "maximum number of concurrent connections",
"value": 1000,
"unit": "count"
}
6.3 Relation Extraction
Relation extraction is used to identify the semantic relations between entities.
For example:
OpenAI developed GPT-4.
This can be represented as:
(OpenAI, developed, GPT-4)
This is a triple:
Where:
- h: Head, the head entity.
- r: Relation, the relation.
- t: Tail, the tail entity.
For example:
Other common relations:
| Head entity | Relation | Tail entity |
|---|---|---|
| OpenAI | developed | GPT-4 |
| GPT-4 | belongs to | large language model |
| Transformer | uses | Self-Attention |
| Python | supports | object-oriented programming |
| Redis | provides | caching capability |
The difficulties of relation extraction are:
- Relations may span multiple sentences.
- Entities may involve coreference relations.
- Relations are directional.
- Multiple relations may exist between the same entities.
- Relations may be constrained by time and conditions.
For example:
System A calls System B.
System B caches user information through Redis.
Extracting only the entities and relations of each sentence is not enough; you also need to identify entity coreference and cross-sentence associations in the context.
6.4 Event Extraction
Event extraction focuses on what happened, as well as which participants and conditions the event involves.
For example:
In March 2026, a company released a new version of its database product.
You can extract:
{
"event_type": "PRODUCT_RELEASE",
"time": "2026-03",
"organization": "a company",
"product": "database product",
"action": "release"
}
An event usually includes:
- Event type.
- Participating entities.
- Time of occurrence.
- Location of occurrence.
- Trigger words.
- Event arguments.
Common event types:
- Product release.
- Corporate acquisition.
- System failure.
- Order creation.
- User login.
- Transaction completion.
- Contract signing.
Event extraction is valuable for news analysis, enterprise knowledge bases, audit systems, and business process analysis.
6.5 Fact Extraction
Compared with simple entity recognition, fact extraction focuses more on complete semantics.
For example:
The default port of Redis is 6379.
You can extract:
{
"subject": "Redis",
"predicate": "default port",
"object": 6379,
"unit": null,
"source": "doc_001",
"page": 12
}
A usable fact record should contain not only the subject, predicate, and object, but should also retain, as far as possible:
- The source document.
- The page number or paragraph location.
- The effective time.
- The scope of applicability.
- The confidence level.
- The document version.
This is especially true for enterprise knowledge bases, where facts are usually not permanently valid.
For example:
The system currently uses Redis 7.0.
And:
The system's historical version used Redis 6.2.
Both facts may be correct, but they correspond to different points in time and different versions.
If time and version information is not retained, the knowledge base may mistake historical facts for current facts.
7. Using Large Language Models for Knowledge Extraction
Traditional information extraction models usually require entity categories, relation types and training data to be defined in advance.
Large language models can perform more flexible knowledge extraction tasks based on natural language instructions.
7.1 Prompt-Based Structured Extraction
For example, the input:
Please extract entities, attributes and relations from the following content.
Text:
Redis is an open-source in-memory data structure storage system,
commonly used for caching, message queues and real-time data processing.
Required output:
{
"entities": [
{
"name": "Redis",
"type": "SOFTWARE"
},
{
"name": "cache",
"type": "APPLICATION"
},
{
"name": "message queue",
"type": "APPLICATION"
},
{
"name": "real-time data processing",
"type": "APPLICATION"
}
],
"relations": [
{
"subject": "Redis",
"predicate": "commonly used for",
"object": "cache"
},
{
"subject": "Redis",
"predicate": "commonly used for",
"object": "message queue"
},
{
"subject": "Redis",
"predicate": "commonly used for",
"object": "real-time data processing"
}
]
}
This approach does not require training a dedicated model for each new entity type, so it offers good flexibility.
However, note that:
- The model may generate entities that do not appear in the source text.
- It may incorrectly infer relations between entities.
- It may turn uncertain content into definite facts.
- It may omit important information.
- It may output JSON that does not conform to the schema.
Therefore, LLM extraction results must be validated and cannot be written directly into an authoritative knowledge base.
7.2 Structured Output and Schema Constraints
In engineering practice, a fixed data structure should be defined wherever possible.
For example:
{
"type": "object",
"properties": {
"entities": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
}
},
"required": [
"name",
"type"
],
"additionalProperties": false
}
},
"relations": {
"type": "array",
"items": {
"type": "object",
"properties": {
"subject": {
"type": "string"
},
"predicate": {
"type": "string"
},
"object": {
"type": "string"
}
},
"required": [
"subject",
"predicate",
"object"
],
"additionalProperties": false
}
}
},
"required": [
"entities",
"relations"
],
"additionalProperties": false
}
The schema mainly addresses structural validity:
- Whether fields exist.
- Whether field types are correct.
- Whether additional fields are allowed.
- Whether array elements conform to the definition.
However, being schema-valid does not mean the knowledge is correct.
For example:
{
"subject": "Redis",
"predicate": "开发",
"object": "OpenAI"
}
This may fully conform to JSON Schema while being semantically incorrect.
Therefore, further semantic validation is required.
7.3 Evidence-Based Extraction
A more reliable extraction method is to require the model to provide source-text evidence for each fact.
For example:
{
"subject": "Redis",
"predicate": "默认端口",
"object": "6379",
"evidence": {
"text": "Redis 默认端口为 6379。",
"page": 12,
"block_id": "b012"
}
}
Evidence can be used to:
- Check whether a fact comes from the source text.
- Locate the original passage.
- Detect model hallucinations.
- Support manual review.
- Generate citations for question answering.
- Trace the source of knowledge.
Note that the evidence text returned by the model may also be an incorrect quotation or paraphrase, so you cannot merely check whether the fields exist. Production systems should match the evidence against the original content and retain the original location identifiers.
7.4 Layered Extraction Strategy
It is not suitable to feed a complex document into a model all at once and ask it to extract all knowledge.
A more reasonable strategy is to process it in layers:
flowchart TD
A["Structured document"] --> B["Document-level analysis"]
B --> C["Section-level analysis"]
C --> D["Paragraph-level extraction"]
D --> E["Entity and attribute recognition"]
E --> F["Relation and event extraction"]
F --> G["Cross-paragraph association"]
G --> H["Fact validation and merging"]
H --> I["Knowledge data"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
style F fill:#E5EAF8,stroke:#9AA8D2,color:#3F4F7A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Hierarchical extraction solves several problems:
- Reducing the length of a single input.
- Preserving section and paragraph context.
- Avoiding missed information in long documents.
- Separating local extraction from global association.
- Making it easy to configure different models for different stages.
Among these, document-level analysis is responsible for identifying topics, sections and the overall context; paragraph-level extraction is responsible for extracting local facts; and cross-paragraph association is responsible for resolving anaphora, entity normalization and relation completion.