1. Why Data Processing and Knowledge Update Mechanisms Are Needed
In AI applications, a knowledge base is not a static dataset built once and for all, but a continuously changing data system.
Enterprise documents, product manuals, technical specifications, business databases, web content, and real-time events may all change continuously. If you focus only on the initial import and lack ongoing data governance and update mechanisms, the following problems will arise:
- Outdated knowledge: the source document has been modified, but the vector store still holds the old content.
- Duplicate knowledge: the same document is imported multiple times, producing duplicate chunks and duplicate retrieval results.
- Conflicting content: multiple data sources describe the same fact differently.
- Incomplete updates: after a document is updated, only some of its chunks are processed, leaving old and new content mixed together.
- Incomplete deletion: the original file is deleted, but the vector index, caches, and derived data still exist.
- Invalidated permissions: when a document's permissions change, the retrieval system may still return content that was originally accessible.
- Untraceable updates: it is impossible to determine which file, which version, and which processing task a piece of knowledge came from.
Therefore, a production-grade AI knowledge system must establish a complete data lifecycle management mechanism.
The core objectives include:
- Accessible data: support multiple data sources and data formats.
- Understandable data: the ability to extract text, structure, metadata, and multimodal information.
- Traceable data: every piece of knowledge can be traced back to its original source.
- Updatable data: support full synchronization, incremental updates, and partial rebuilds.
- Recoverable data: failed tasks can be retried, and erroneous data can be rolled back.
- Governable data: support permissions, deletion, deduplication, quality, and version management.
- Verifiable data: after an update is complete, you can confirm that the knowledge base is in the correct state.
2. The Overall Architecture of AI Data Processing
AI data processing is not simply file upload and vectorization; it is a complete data pipeline covering ingestion, parsing, transformation, indexing, and publishing.
2.1 Core Architecture
A typical AI data processing system can be divided into the following layers:
| Layer | Core responsibility | Typical components |
|---|---|---|
| Data source layer | Provides raw data | File systems, databases, object storage, APIs, web pages |
| Data ingestion layer | Discovers, pulls, and receives data | Connectors, webhooks, scheduled sync |
| Raw data layer | Stores original files and source information | Object storage, metadata database |
| Parsing layer | Extracts content and structure | Parsers, OCR, ASR, document parsers |
| Data governance layer | Cleaning, deduplication, validation, and normalization | Data Cleaner, Deduplicator |
| Knowledge transformation layer | Chunking, metadata enrichment, and relationship building | Chunker, Metadata Enricher |
| Feature processing layer | Generates vectors and other retrieval features | Embedding Model, Sparse Encoder |
| Index and storage layer | Stores and organizes retrieval data | Vector DB, Search Engine, Graph DB |
| Publishing management layer | Controls knowledge versions and visibility state | Version Manager, Manifest |
| Monitoring and governance layer | Monitors quality, tasks, and the data lifecycle | Metrics, Audit Log, Lineage |
The overall data processing workflow:
flowchart TD
A["Data Sources"] --> B["Data Discovery and Ingestion"]
B --> C["Raw Data Storage"]
C --> D["Format Recognition and Content Parsing"]
D --> E["Data Cleaning and Normalization"]
E --> F["Deduplication and Quality Validation"]
F --> G["Knowledge Chunking and Metadata Construction"]
G --> H["Embedding and Feature Generation"]
H --> I["Index Construction"]
I --> J["Index Validation"]
J --> K["Knowledge Version Release"]
K --> L["Online Retrieval Service"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style L fill:#E5F0E9,stroke:#83A995,color:#315D49
Here is an important architectural principle:
Successful data processing does not mean the knowledge is already available for online retrieval.
Newly added knowledge should become visible to online services only after index construction, data validation, and version release are complete.
3. Data Ingestion Mechanism
Data ingestion is responsible for continuously bringing content from external data sources into the AI knowledge system.
3.1 Common Data Source Types
| Data Source | Ingestion Method | Key Issues |
|---|---|---|
| Local files | File scanning, directory monitoring | File change detection |
| Object storage | Bucket scanning, event notifications | Object versions and deletion events |
| Relational databases | CDC, scheduled queries | Data consistency and primary keys |
| NoSQL databases | Change Stream, incremental queries | Change ordering and duplicate events |
| Enterprise knowledge platforms | API, Webhook | Pagination, permissions, and rate limiting |
| Web content | Scheduled crawling, web crawlers | Page changes and content extraction |
| SaaS systems | Official connectors, API | Tokens, permissions, and quotas |
| Messaging systems | Kafka, Pulsar, etc. | Consumer offsets and message replay |
Different data sources use different ingestion methods, but they ultimately need to be unified into internal data change events within the system.
3.2 Connector Design
A Connector is an adaptation layer between data sources and the AI data processing system.
It should not be directly responsible for text parsing, Embedding, or vector indexing; instead, it should focus on:
- Connection authentication.
- Data discovery.
- Data reading.
- Pagination and cursor management.
- Change detection.
- Deletion event identification.
- Data source metadata extraction.
- Change event standardization.
We recommend abstracting Connectors into the following unified interface:
```python from abc import ABC, abstractmethod from typing import Iterator class DataConnector(ABC): @abstractmethod def connect(self): """Establish a connection to the data source""" pass @abstractmethod def discover(self, cursor=None): """Discover newly added or changed data""" pass @abstractmethod def fetch(self, item_id): """Read the specified data""" pass @abstractmethod def get_changes(self, cursor=None): """Get data change events""" pass @abstractmethod def get_deletions(self, cursor=None): """Get deletion events""" pass @abstractmethod def checkpoint(self): """Save the sync checkpoint""" pass ``` In practice, `discover`, `get_changes`, and `get_deletions` can be combined according to what the data source supports; not every Connector is required to implement the same underlying logic.3.3 Full Scan and Incremental Discovery
A full scan is appropriate when:
- Building the knowledge base for the first time.
- The data source does not support change notifications.
- The sync state is lost.
- Periodic data consistency checks are needed.
Incremental discovery is appropriate when:
- The data volume is large.
- The data source supports querying by update time.
- The data source supports CDC.
- The cost of repeated processing needs to be reduced.
The two mechanisms usually need to be used together:
```mermaid flowchart TD A["Start data sync"] --> B{"First sync?"} B -->|"Yes"| C["Full scan"] B -->|"No"| D{"Incremental change supported?"} D -->|"Yes"| E["Read change log or update time"] D -->|"No"| F["Periodic full scan"] C --> G["Generate data change events"] E --> G F --> G G --> H["Enter the unified processing pipeline"] style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976 style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D style E fill:#EAF3EC,stroke:#83A995,color:#315D49 style F fill:#EAF3EC,stroke:#83A995,color:#315D49 style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style H fill:#E5F0E9,stroke:#83A995,color:#315D49 ```Note that using only the file modification time to determine changes is unreliable.
A file may be overwritten without its timestamp being updated correctly, and clocks may also drift. Therefore, judgments should combine content hashes, data source version numbers, ETags, or CDC positions.
IV. Data Parsing and Normalization
Data formats vary widely across sources. The goal of data parsing is to convert heterogeneous content into a unified, processable internal structure.
4.1 Format Identification
Common data formats include:
| Type | Example formats | Processing method |
|---|---|---|
| Text documents | TXT, Markdown | Plain text reading |
| Office documents | DOCX, XLSX, PPTX | Structured parsing |
| Text extraction, layout analysis, OCR | ||
| Images | PNG, JPEG | OCR, vision models |
| Audio | MP3, WAV | ASR, speaker recognition |
| Video | MP4, MOV | Frame extraction, ASR, visual understanding |
| Web pages | HTML | DOM cleaning, main content extraction |
| Structured data | JSON, CSV | Schema parsing |
| Database records | SQL, NoSQL | Field mapping and record conversion |
Format identification must not rely on file extensions alone.
For example, an object whose file extension is .pdf may not actually contain a valid PDF. A more reliable approach is to validate it using a combination of MIME type, file header, content characteristics, and the actual parser result.
4.2 Unified Data Model
After parsing is complete, it is recommended to convert different data sources into a unified Document model.
{
"document_id": "doc_10001",
"source_id": "source_001",
"source_type": "file",
"source_uri": "s3://knowledge/manual.pdf",
"source_version": "v12",
"content_hash": "sha256:abc123",
"title": "产品技术手册",
"content": "文档正文内容",
"mime_type": "application/pdf",
"metadata": {
"department": "engineering",
"language": "zh-CN",
"created_at": "2026-01-10T10:00:00Z",
"updated_at": "2026-11-20T10:00:00Z"
},
"permissions": {
"visibility": "internal",
"groups": ["engineering"]
}
}
A unified data model should at minimum distinguish:
document_id: a stable internal document identifier within the system.source_id: data source identifier.source_uri: original data location.source_version: data source version or snapshot identifier.content_hash: content fingerprint.metadata: business metadata.permissions: access control information.
The most common problem here is using a file path directly as the document_id.
File paths may change, and the same document may even be referenced through multiple paths. Therefore, internal identifiers should be as stable as possible, and paths should be stored separately as source location information.
4.3 Structured Parsing
A parser should not output only a concatenated block of plain text.
For complex documents, it is recommended to preserve the original structure:
- Document title.
- Section hierarchy.
- Paragraphs.
- Tables.
- Images.
- Image captions.
- Page numbers.
- Code blocks.
- Lists.
- Internal document links.
- Content coordinates and source locations.
For example, if a table in a PDF is directly concatenated into plain text, the row-column correspondence may be lost.
Original content:
| Product | Version | Support Status |
|---|---|---|
| Service A | 2.0 | Supported |
| Service B | 1.0 | Not supported |
If the parser outputs:
Product Version Support Status Service A 2.0 Supported Service B 1.0 Not supported
Subsequent models may be unable to accurately determine the correspondence between fields.
Therefore, structured parsing should preserve table boundaries, cells, and header relationships as much as possible.
4.4 Data Standardization
Different data sources may use different field naming, time formats, language encodings, and classification methods.
For example:
{
"updateTime": "2026/11/20 10:00:00"
}
and:
{
"updated_at": "2026-11-20T02:00:00Z"
}
may represent the same time.
Standardization needs to handle:
- Unified field naming.
- Unified time formats and time zones.
- Unified character encoding.
- Handling of null values and default values.
- Unit conversion.
- Enum value mapping.
- Text format normalization.
- Unified metadata schema.
Standardization must not arbitrarily change the original semantics. It is recommended to retain both the original data and the standardized data to facilitate troubleshooting and reprocessing.
5. Data Cleaning and Quality Governance
Data cleaning determines whether the content entering the knowledge base is usable.
5.1 Common Data Quality Issues
| Issue | Symptom | Handling |
|---|---|---|
| Empty content | Document has no valid body text | Mark as empty data |
| Duplicate content | Same file imported multiple times | Hash-based deduplication |
| Noisy text | Headers, footers, navigation bars | Rule-based cleaning |
| Encoding anomalies | Garbled characters, special characters | Encoding repair |
| Broken formatting | Table and paragraph structure lost | Re-parse |
| Outdated content | Old versions still present | Version retirement |
| Missing metadata | Source, permissions, timestamps empty | Complete or isolate |
| Content conflicts | Different conclusions from multiple sources | Conflict detection and authoritative-source governance |
5.2 Hash-Based Data Deduplication
Data deduplication is usually divided into three levels.
Level 1: File-level deduplication
Compute a hash of the original file:
If two files have the same hash, their binary content can be considered completely identical.
Level 2: Normalized content deduplication
Files may differ in format, metadata, or compression method while their actual text is the same.
Therefore, a content hash can be computed on the cleaned body text:
Level 3: Semantic similarity deduplication
Two documents may express the same meaning in completely different wording.
Candidate detection can be performed using Embedding similarity:
Once the similarity exceeds the threshold, the pair proceeds to further duplicate-content evaluation.
Note: semantic similarity does not mean the content is identical. For example, two versions of an API document may differ only in parameter types, yet precisely that difference is important knowledge and must not be merged outright.
Therefore, semantic deduplication is better suited as a candidate-discovery mechanism rather than an unconditional deletion mechanism.
5.3 Data Quality Scoring
A quality score can be established for each document:
Where:
- C: content completeness.
- S: structural completeness.
- M: metadata completeness.
- F: format validity.
- w_i: the corresponding weight.
The score is used to determine whether data enters the normal processing pipeline.
For example:
| Quality level | Processing strategy |
|---|---|
| High quality | Enters knowledge construction normally |
| Medium quality | Goes to supplementary processing or manual sampling review |
| Low quality | Quarantined, awaiting re-parsing |
| Invalid data | Rejected from ingestion, with reason logged |
A quality score cannot replace business rules. Even if a document is structurally complete, it should not be published directly if its source is untrustworthy, its permissions are unclear, or its content has expired.
6. Knowledge Segmentation and Data Version Construction
After parsing and cleaning are complete, documents need to be converted into knowledge units suitable for retrieval.
6.1 Document, Chunk, and Knowledge Unit
It is recommended to divide the data model into three levels:
| Level | Meaning | Example |
|---|---|---|
| Document | Complete original document | Product technical manual |
| Section | Internal structure of the document | Chapter 3: Deployment Architecture |
| Chunk | Content fragment actually used in retrieval | Deployment environment requirements |
A single Document can contain multiple Sections, and a Section can in turn contain multiple Chunks.
flowchart TD
A["Document"] --> B["Section 1"]
A --> C["Section 2"]
A --> D["Section 3"]
B --> E["Chunk 1"]
B --> F["Chunk 2"]
C --> G["Chunk 3"]
C --> H["Chunk 4"]
D --> I["Chunk 5"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
Each chunk should retain the following information:
chunk_iddocument_iddocument_versionsection_idchunk_indexcontentcontent_hashsource_locationmetadatapermissionembedding_version
Among these, source_location is especially important.
It can record the page number, section path, paragraph number, table coordinates, or timestamp, and it allows retrieval results to be traced back to the original content.
6.2 How to Design Chunk IDs
A common approach is to generate them from the document identifier, version, and chunk sequence number:
For example:
doc_10001:v12:chunk_0001
doc_10001:v12:chunk_0002
doc_10001:v12:chunk_0003
This approach guarantees that:
- Chunks from different versions of the same document will not be confused.
- Chunks within the same version can be located reliably.
- After a document version is updated, the old chunks can still be identified.
- Batch deletion can be performed at the version level.
However, a chunk sequence number must not be treated as a permanent identity.
If a section is inserted in the middle of a document, the sequence numbers of all subsequent chunks may change. Therefore, if content changes need to be identified precisely, content hashes, section locations, and stable structural identifiers should also be used together.
6.3 Knowledge Version and Processing Version
Two different kinds of versions need to be distinguished.
Knowledge Version
Represents the version of the original business content.
For example:
Product Manual v1
Product Manual v2
Product Manual v3
Processing Version
Represents the version of the knowledge processing logic.
For example:
Parser v1
Chunker v2
Embedding Model v3
Even if the original document has not changed, any change to the chunking strategy, parser, or embedding model may still require rebuilding the knowledge index.
Therefore, a complete processing record should contain at least:
{
"document_id": "doc_10001",
"document_version": "v12",
"parser_version": "parser_3",
"chunker_version": "chunker_2",
"embedding_model": "embedding_model_a",
"embedding_version": "embedding_v4",
"pipeline_version": "pipeline_7"
}
This prevents the business content version from being conflated with the processing algorithm version.
7. Knowledge Update Mechanisms
Knowledge updating is one of the most central engineering problems in AI data systems.
It involves not only adding new data, but also modification, deletion, rebuilding, permission changes, and upgrades to processing logic.
7.1 Four Main Types of Updates
| Update Type | Trigger Condition | Processing Strategy |
|---|---|---|
| Add | New document appears in the data source | Create new Document and Chunk |
| Modify | Original document content changes | Rebuild affected knowledge units |
| Delete | Document deleted from the data source | Delete corresponding indexes and derived data |
| Reprocess | Parsing or model strategy upgrade | Re-parse, split, or generate vectors |
7.2 Full Update
A full update rescans all data and rebuilds the knowledge base based on the current state of the data source.
Suitable for:
- Initial knowledge base construction.
- The data source lacks incremental capabilities.
- Major changes in the index structure.
- Serious data consistency issues.
- Synchronization results need to be calibrated periodically.
The main disadvantage of full updates is their high cost.
If the knowledge base contains millions of Chunks, re-parsing, splitting, and vectorizing on every update will incur substantial unnecessary computation and storage overhead.
Therefore, production systems usually do not use full rebuilds as a daily update strategy.
7.3 Incremental Update
An incremental update processes only newly added or changed data.
Its core is to identify data changes and map them to corresponding knowledge processing tasks.
flowchart TD
A["Receive data change event"] --> B["Read document metadata"]
B --> C["Calculate content fingerprint"]
C --> D{"Has content changed?"}
D -->|"No"| E["Update sync checkpoint"]
D -->|"Yes"| F{"Change type"}
F -->|"Add"| G["Create new document"]
F -->|"Modify"| H["Build new document version"]
F -->|"Delete"| I["Mark old version for deletion"]
G --> J["Parse and split"]
H --> J
J --> K["Generate Embedding"]
K --> L["Build new index"]
L --> M["Validate and publish"]
I --> N["Execute deletion cleanup"]
M --> E
N --> E
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#E5F0E9,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
The key to incremental updates is not reducing the scan volume, but avoiding unnecessary repeated computation.
For example:
- File content unchanged: skip parsing and Embedding.
- Only permissions changed: update access control information; regenerating vectors may not be necessary.
- Only metadata changed: update only the affected index fields.
- Document content changed: re-parse and rebuild the corresponding knowledge units.
- Embedding model upgraded: even if content is unchanged, vectors need to be regenerated.
7.4 Partial Update and Full-Document Rebuild
After a document is updated, does the entire document need to be reprocessed?
This depends on the scope of change and the processing approach.
Full-Document Rebuild
Suitable for:
- Document structure changes significantly.
- Chapter order is adjusted.
- Table or image structure changes.
- Chunk splitting strategy depends on full-text context.
- Local changes cannot be reliably identified.
Partial Update
Suitable for:
- Document structure is stable.
- Chapters have stable identifiers.
- Changes can be precisely located.
- Unaffected Chunks can retain their original identity.
Partial updates can reduce parsing and vectorization costs, but they are more difficult to implement.
Pay particular attention to Chunk boundary issues: if a paragraph changes, it is not necessarily only one Chunk that is affected. The overlapping areas of adjacent Chunks, context summaries, and parent-child relationships may also need to be rebuilt.
Therefore, partial updates should be based on structured content and dependencies, rather than simply replacing a single text fragment.
8. Index Update and Consistency Design
After knowledge processing is complete, the new knowledge also needs to be written into the retrieval system.
This involves consistency among the vector database, keyword index, metadata index, and relationship index.
8.1 Multi-index update problem
A knowledge unit may exist simultaneously in:
- Vector database.
- Full-text search engine.
- Relational database.
- Knowledge graph.
- Cache system.
These systems usually do not have unified transaction capabilities.
Suppose that when a document is updated:
- The new vector has already been written.
- The full-text index update fails.
- The metadata database still points to the old version.
At this point, different retrieval channels may return different versions of the knowledge.
Therefore, the write operations of multiple indexes cannot simply be treated as one atomic transaction.
8.2 Two-phase publication
A common approach is to first build a new version and then switch the online visible version.
Core steps:
- Create a new knowledge version.
- Parse and process the new data.
- Write to the new version indexes.
- Check whether each index is complete.
- Validate data counts and key fields.
- Atomically switch the current version pointer.
- Asynchronously clean up the old version.
flowchart TD
A["Current version V1 online serving"] --> B["Create new version V2"]
B --> C["Parsing and knowledge construction"]
C --> D["Write to vector index"]
C --> E["Write to full-text index"]
C --> F["Write to metadata index"]
D --> G["Index integrity validation"]
E --> G
F --> G
G --> H{"Validation passed?"}
H -->|"No"| I["Keep V1 and record failure"]
H -->|"Yes"| J["Switch current version pointer"]
J --> K["V2 provides retrieval externally"]
K --> L["Asynchronously clean up V1"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#E5F0E9,stroke:#83A995,color:#315D49
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
The advantage of this approach is that online users will not directly see partially built data during the build process.
However, it requires additional storage space and requires all retrieval channels to be version-aware.
8.3 Version pointer and release manifest
A Manifest can be used to manage the currently available knowledge versions.
{
"knowledge_base_id": "kb_001",
"active_version": "v102",
"previous_version": "v101",
"status": "READY",
"indexes": {
"vector": {
"index_name": "kb_001_v102",
"status": "READY"
},
"fulltext": {
"index_name": "kb_001_v102",
"status": "READY"
},
"metadata": {
"index_name": "kb_001_v102",
"status": "READY"
}
},
"published_at": "2026-11-20T10:00:00Z"
}
An online retrieval request first obtains the current version, then uses the index of the corresponding version.
This decouples data construction from online publication.
8.4 Eventual consistency and idempotency
Distributed data processing systems usually cannot guarantee that all components complete updates at the same moment, so they need to accept a certain degree of eventual consistency.
But eventual consistency does not mean that old and new data can be mixed arbitrarily.
Three issues need to be handled with focus:
Idempotency
Repeated execution of the same event should not produce duplicate knowledge.
You can use:
idempotency_key =
Hash(source_id, document_id, source_version, pipeline_version)
Event ordering
Old version events must not overwrite new versions.
For example, the system first receives an update for document v3, and then receives a v2 event that arrives late. It must determine whether an event is stale based on the version number, update time, or data source sequence number.
Version Isolation
Chunks and indexes from different versions must be clearly distinguished so that old and new versions are not mixed during retrieval.
An important principle is:
Tasks may be delivered redundantly, but redundant delivery must never be allowed to corrupt the final knowledge state.
9. Event-Driven Architecture in Data Updates
For large knowledge systems, it is usually necessary to decouple data ingestion, parsing, vectorization, and index building through a message queue.
9.1 Event-Driven Processing
The data processing pipeline can be split into multiple independent tasks:
flowchart TD
A["Data Source Change"] --> B["Connector"]
B --> C["Change Event Queue"]
C --> D["Parsing Task"]
D --> E["Cleaning Task"]
E --> F["Knowledge Chunking Task"]
F --> G["Embedding Task"]
G --> H["Index Writing Task"]
H --> I["Version Publishing Task"]
I --> J["Online Knowledge Base"]
D --> K["Parsing Failure Queue"]
E --> L["Cleaning Failure Queue"]
G --> M["Embedding Failure Queue"]
H --> N["Index Failure Queue"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#F7EAE7,stroke:#C98D7C,color:#754438
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
Different tasks can use different Worker clusters.
For example:
- PDF parsing tasks require substantial CPU and memory.
- OCR tasks may require GPUs or dedicated vision services.
- Embedding tasks may be constrained by model throughput and API rate limits.
- Index writing tasks are constrained by the write capacity of the database.
Once tasks are separated, they can be scaled up and down independently.
9.2 Task State Management
It is recommended to maintain an explicit state for each Document or data processing task.
| State | Meaning |
|---|---|
| DISCOVERED | Data discovered |
| FETCHED | Raw content fetched |
| PARSING | Parsing in progress |
| PARSED | Parsing complete |
| CLEANING | Cleaning in progress |
| CLEANED | Cleaning complete |
| CHUNKING | Chunking in progress |
| EMBEDDING | Generating vectors |
| INDEXING | Writing to the index |
| VALIDATING | Validating |
| PUBLISHED | Published |
| FAILED | Processing failed |
| DELETING | Deleting in progress |
| DELETED | Deletion complete |
State transitions must be constrained; tasks must not be allowed to jump between states arbitrarily.
For example, a document whose parsing has not yet completed cannot go directly into the index publishing state.
9.3 Task Retries and Dead-Letter Queues
Not all failures are suitable for immediate retry.
Exceptions can be divided into three categories:
| Exception Type | Example | Handling Strategy |
|---|---|---|
| Transient errors | Network timeouts, service rate limiting | Delayed retry |
| Data errors | Corrupted files, unsupported formats | Isolate and wait for repair |
| System errors | Program exceptions, unavailable dependent services | Alert and retry, or manual intervention |
Retries should use an exponential backoff mechanism:
In addition, add random jitter to prevent a large number of tasks from requesting the service again at the same moment.