1. What Is a Workflow Engine?
A Workflow Engine is a runtime system responsible for defining, scheduling, executing, and managing workflows.
It breaks a complex task into multiple independent execution units, determines the execution order through dependencies between nodes, and during runtime maintains task state, passes data, handles exceptions, and records execution history.
From the perspective of system responsibilities, a workflow engine mainly solves six problems:
- How tasks are organized: split business tasks into nodes and establish dependencies between nodes.
- When tasks execute: determine execution timing based on dependency conditions, trigger events, and scheduling policies.
- How tasks collaborate: manage node inputs, outputs, and shared state.
- How tasks are parallelized: identify nodes that can execute in parallel and control resources and concurrency.
- How tasks recover: handle timeouts, failures, retries, pauses, and process restarts.
- How tasks are tracked: record the execution process of each node and the lifecycle of workflow instances.
A typical workflow engine can be abstracted as:
flowchart TD
A["Workflow Definition"] --> B["Workflow Parsing"]
B --> C["Build Execution Graph"]
C --> D["Create Workflow Instance"]
D --> E["Node Scheduler"]
E --> F["Node Executor"]
F --> G["State Management"]
G --> H["Result Persistence"]
H --> I["Execution Monitoring and Tracing"]
G --> E
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
Two concepts need to be distinguished:
- Workflow Definition: describes how tasks should be executed.
- Workflow Engine: responsible for actually running workflow definitions.
A workflow definition is a static execution description, while a workflow engine is responsible for dynamic execution and state management.
2. Core Data Model of a Workflow Engine
To understand a workflow engine, you first need to establish the relationships among several core objects.
1. Workflow
A Workflow is a complete definition of a task flow.
It usually includes:
- Workflow ID.
- Workflow name.
- Set of nodes.
- Connection relationships between nodes.
- Input and output definitions.
- Conditional branches.
- Loop rules.
- Error handling policies.
- Timeout and retry configurations.
- Version information.
For example, a document analysis workflow:
{
"workflow_id": "document_analysis",
"version": 1,
"nodes": [
{
"id": "parse",
"type": "document_parser"
},
{
"id": "extract",
"type": "information_extractor"
},
{
"id": "analyze",
"type": "llm_analysis"
},
{
"id": "output",
"type": "result_output"
}
],
"edges": [
{
"source": "parse",
"target": "extract"
},
{
"source": "extract",
"target": "analyze"
},
{
"source": "analyze",
"target": "output"
}
]
}
The JSON here is only a simplified example. Production systems also need to define node parameters, input/output schemas, edge conditions, failure policies, and version compatibility rules.
2. Workflow Instance
A Workflow Instance represents a single actual run.
The same workflow definition can be used to create multiple instances, each with its own independent input, state, and execution history.
For example:
Workflow definition: document_analysis
Workflow instances:
├── instance_001: Analyze contract A
├── instance_002: Analyze contract B
└── instance_003: Analyze contract C
A workflow instance typically contains:
| Field | Meaning |
|---|---|
| instance_id | Unique instance identifier |
| workflow_id | Owning workflow |
| workflow_version | Definition version used during execution |
| status | Current instance status |
| input | Initial input |
| context | Workflow runtime context |
| created_at | Creation time |
| started_at | Execution start time |
| completed_at | End time |
| error | Instance-level error information |
3. Node: Node
A Node is the basic execution unit in a workflow.
A node typically has:
- Node ID.
- Node type.
- Node configuration.
- Input schema.
- Output schema.
- Executor function.
- Timeout configuration.
- Retry policy.
- Error handling policy.
For example:
{
"id": "llm_analysis",
"type": "llm",
"config": {
"model": "example-model",
"temperature": 0.2
},
"timeout": 30000,
"retry": {
"max_attempts": 3
}
}
A node definition describes how to execute, while a node execution record describes what happened during a specific execution.
These two objects must be distinguished.
4. Edge: Edge
An Edge represents a dependency relationship between nodes.
For example:
flowchart TD
A["Document parsing"] --> B["Information extraction"]
B --> C["Risk analysis"]
C --> D["Report generation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Edges can contain more execution semantics:
- Normal dependency: execute the downstream node after the upstream node completes.
- Conditional dependency: execute only when specific conditions are met.
- Multi-input dependency: wait for multiple upstream nodes to complete.
- Error dependency: enter an error-handling node after an upstream failure.
- Loop connection: re-execute the specified node when the loop condition is met.
5. Node Execution: Node Execution Record
A node may be executed multiple times, so an execution status cannot be recorded only on the Node object.
For example, a node may produce multiple execution attempts due to retry after failure.
It is recommended to separate node definitions from execution records:
Node Definition
└── Node Execution
├── Attempt 1: Failed
├── Attempt 2: Failed
└── Attempt 3: Completed
Node Execution typically records:
- execution_id.
- instance_id.
- node_id.
- attempt_number.
- status.
- input_snapshot.
- output_snapshot.
- started_at.
- completed_at.
- error.
- worker_id.
This design preserves the complete execution history and also facilitates troubleshooting and retry analysis.
III. Node Model: The Execution Foundation of Workflows
A node is not only a function call, but also the basic abstraction by which the workflow engine manages task execution.
1. Basic Structure of a Node
A node can be abstracted as:
where:
- ID: unique identifier of the node.
- Type: node type.
- Config: runtime configuration of the node.
- Input: input data of the node.
- Output: output data of the node.
- Executor: the actual execution logic.
Node execution can be abstracted as:
Here, Context represents the workflow runtime context.
For pure-function nodes, the same input and configuration should produce the same output; but for LLMs, random computations, external APIs, and database operations, execution results may be affected by external state and non-deterministic factors.
2. Node Types
A workflow engine usually needs to support multiple node types.
| Node Type | Primary Responsibility | Typical Scenario |
|---|---|---|
| Start Node | Start the workflow | Receive requests |
| End Node | End the workflow | Return results |
| LLM Node | Call a large model | Classification, generation, reasoning |
| Tool Node | Call external tools | APIs, databases, search |
| Code Node | Execute code | Data transformation, computation |
| Condition Node | Conditional evaluation | Branch routing |
| Loop Node | Loop execution | Batch processing |
| Parallel Node | Parallel execution | Multi-task coordination |
| Join Node | Aggregate execution results | Multi-branch merging |
| Human Node | Wait for human action | Review, approval |
| Subworkflow Node | Call a subworkflow | Reuse business processes |
Node types can be extended, but the underlying execution interface should be as unified as possible.
3. Node Execution Interface
A general node execution interface can be abstracted as:
interface NodeExecutor {
execute(
input: Record<string, unknown>,
context: WorkflowContext,
config: Record<string, unknown>
): Promise<NodeResult>;
}
interface NodeResult {
output: Record<string, unknown>;
status: "completed" | "failed";
}
The interface here is an architectural illustration; an actual production implementation should also consider:
- Cancellation signals.
- Timeout control.
- Execution metadata.
- Error classification.
- Retry policies.
- Logging and tracing.
- Resource limits.
A node executor is only responsible for completing the node's own business logic and should not directly take on the scheduling responsibilities of the entire workflow.
4. Node Execution Lifecycle
A node goes through multiple stages from being scheduled to completion.
flowchart TD
A["Node waiting"] --> B["Execution conditions met"]
B --> C["Added to scheduling queue"]
C --> D["Allocate execution resources"]
D --> E["Load node input"]
E --> F["Execute node logic"]
F --> G["Validate execution result"]
G --> H{"Execution succeeded?"}
H -->|"Yes"| I["Save output"]
I --> J["Mark as Completed"]
H -->|"No"| K["Record exception"]
K --> L{"Retry allowed?"}
L -->|"Yes"| M["Wait for retry"]
M --> C
L -->|"No"| N["Mark as Failed"]
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
Two actions need to be distinguished here:
- A node being scheduled does not mean it has started executing.
- A node function finishing execution does not mean its result has been reliably persisted.
Only after the execution result has passed the necessary validation and the state has been committed can the node be safely marked as completed.
IV. Workflow Graph Structure and Node Dependencies
Workflow engines usually use a graph structure to describe the relationships between nodes.
1. DAG: Directed Acyclic Graph
A DAG (Directed Acyclic Graph) is a very common data structure in workflow orchestration.
It has two core characteristics:
- Directed: edges have direction, representing task dependencies.
- Acyclic: following edges in their direction cannot lead back to the original node.
For example:
flowchart TD
A["Start"] --> B["Read user data"]
A --> C["Read order data"]
B --> D["Data integration"]
C --> D
D --> E["Generate analysis results"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
A DAG can express:
- Node dependencies.
- Execution order.
- Parallel relationships.
- Convergence relationships.
For a static DAG, topological sorting can be performed before the workflow starts to detect circular dependencies in advance.
2. Topological Sorting
Topological Sorting is used to determine a valid execution order for the nodes in a directed acyclic graph.
If there is an edge:
then A must complete before B.
Common algorithms include:
- Kahn's algorithm: gradually build the execution order from nodes with zero in-degree.
- DFS algorithm: build the topological sequence using depth-first search and finish times.
Basic process of Kahn's algorithm:
- Calculate the in-degree of each node.
- Add all nodes with zero in-degree to the queue.
- Take a node and add it to the topological sequence.
- Delete the node's outgoing edges and update the in-degrees of downstream nodes.
- Add any nodes that newly have zero in-degree to the queue.
- Repeat until the queue is empty.
flowchart TD
A["Build node dependency graph"] --> B["Calculate in-degree of all nodes"]
B --> C["Find nodes with in-degree zero"]
C --> D{"Are there executable nodes?"}
D -->|"Yes"| E["Take out node and add to topological order"]
E --> F["Remove outgoing edges of this node"]
F --> G["Update in-degrees of downstream nodes"]
G --> C
D -->|"No"| H{"Have all nodes been processed?"}
H -->|"Yes"| I["Obtain topological sort"]
H -->|"No"| J["Cyclic dependency detected"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
The time complexity of Kahn's algorithm is:
Where:
- V: Number of nodes.
- E: Number of edges.
If the number of nodes processed in the end is less than the total number of nodes in the graph, there is a cycle in the graph.
Note that static topological sorting is mainly suitable for DAGs. Workflows with dynamic conditional branches, loops, and runtime-generated nodes usually also need to incorporate execution state and dynamic scheduling mechanisms.
3. Node Dependencies and Execution Readiness Conditions
A node's execution condition is not merely that its upstream nodes have already executed.
It usually also needs to determine:
- Whether all required dependencies are complete.
- Whether the corresponding branch has been selected.
- Whether the node has been cancelled.
- Whether runtime and resource limits are satisfied.
- Whether the workflow instance is still allowed to continue executing.
For an ordinary multi-input node, we can define:
Where:
- Pred(v): The set of predecessor nodes of node v.
- Completed(u): Whether predecessor node u has completed.
For conditional edges, the edge conditions also need to be included in the judgment:
Here, Satisfied(e) includes not only that the upstream execution is complete, but also whether the edge is activated.
This distinction is very important: a branch that was not selected should not be treated as an incomplete dependency that waits forever.
4. Conditional Branches
Conditional branches determine the subsequent execution path based on node output.
flowchart TD
A["Receive user question"] --> B["Identify question type"]
B --> C{"Question type"}
C -->|"Order issue"| D["Query order system"]
C -->|"Product issue"| E["Search product knowledge base"]
C -->|"Technical issue"| F["Search technical documentation"]
D --> G["Consolidate results"]
E --> G
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The condition node is responsible for computing the branch result, and the scheduler is responsible for activating the corresponding edge based on the branch result.
The two should not be conflated.
5. Parallel Nodes and Join Nodes
If multiple nodes have no dependencies among them, the workflow engine can execute them in parallel.
For example:
flowchart TD
A["Start"] --> B["Read user information"]
A --> C["Read order information"]
A --> D["Read product information"]
B --> E["Aggregate and analyze"]
C --> E
D --> E
E --> F["Output result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
B, C, and D can execute simultaneously.
E is a convergence node and must wait for all activated and required upstream branches to complete.
There are three concepts that are easily confused:
- Parallel: Indicates that multiple tasks can execute simultaneously.
- Join: Indicates waiting for multiple branches to meet the convergence condition.
- Merge: Indicates merging results from multiple branches into unified data.
Join handles control dependencies, while Merge handles data merging. The two can be implemented by the same node or implemented separately.
5. Workflow State Machine: How to Manage the Execution Lifecycle
A workflow engine not only needs to execute nodes but also needs to know what state the entire task is currently in.
A state machine is one of the core mechanisms of workflow runtime.
1. Workflow Instance States
Common workflow instance states:
| State | Meaning |
|---|---|
| Created | The instance has been created but not yet started |
| Running | Executing |
| Waiting | Waiting for external events or human input |
| Paused | Paused |
| Completed | Completed successfully |
| Failed | Execution failed |
| Cancelled | Cancelled |
| TimedOut | Execution timed out |
Different engines may use different state names, and may also represent timeouts as a failure reason rather than as a separate state.
2. State Transitions
Workflow state transitions can be abstracted as:
flowchart TD
A["Created"] --> B["Running"]
B --> C["Waiting"]
C --> B
B --> D["Paused"]
D --> B
B --> E["Completed"]
B --> F["Failed"]
B --> G["Cancelled"]
B --> H["TimedOut"]
H --> F
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
This is only a simplified state diagram; an actual engine also needs to clarify:
- Which states can be paused.
- Which states can be resumed.
- How cancellation requests propagate to running nodes.
- Whether retries are allowed after a timeout.
- Whether terminal states allow re-execution.
- How waiting states are resumed through external events.
3. Core Constraints of the State Machine
The purpose of a state machine is not only to record the current state but also to restrict illegal transitions.
For example:
Created → Running allowed
Running → Completed allowed
Running → Failed allowed
Waiting → Running allowed
Completed → Running direct transition generally not allowed
Cancelled → Completed direct transition not allowed
State transitions should be performed through a unified state management interface, rather than letting different nodes modify the instance state at will.
A state transition operation can be abstracted as:
```typescript interface StateTransition { instanceId: string; expectedState: string; targetState: string; reason?: string; } ``` When updating, you should check whether the current state is still equal to `expectedState`. For example, when two Workers attempt to update the same instance at the same time, only one update that matches the expected state should succeed. This constraint can be enforced through conditional database updates, version numbers, or optimistic locking.4. Workflow state and node state must be separated
The workflow instance state and the node execution state are not at the same level. For example: ```text Workflow Instance: Running Node A: Completed Node B: Completed Node C: Running Node D: Pending ``` The workflow being in Running does not mean that all nodes are executing. A node being in Failed also does not necessarily mean that the entire workflow immediately becomes Failed. The engine may choose to retry, degrade, execute an error branch, or wait for manual handling. Therefore, the following should be maintained separately:- Workflow State: the state of the workflow instance.
- Node State: the execution state of a node.
- Execution Attempt: the state of each execution attempt of a node.
5. Node states
Node states typically include:| State | Meaning |
|---|---|
| Pending | Execution conditions are not yet satisfied |
| Ready | Dependencies are satisfied and the node can be scheduled |
| Queued | The node has entered the scheduling queue |
| Running | The node is executing |
| Waiting | Waiting for external conditions |
| Retrying | Waiting for the next retry |
| Completed | Execution succeeded |
| Failed | Execution failed |
| Skipped | Skipped according to the workflow conditions |
| Cancelled | Cancelled |
VI. The scheduling mechanism of workflow engines
The scheduler is responsible for deciding which nodes can execute and when they execute. It connects the workflow graph structure with actual execution resources, and is a core component of the workflow engine.1. Main responsibilities of the scheduler
- Find nodes whose execution conditions have already been satisfied.
- Determine whether a node is activated.
- Add nodes to the execution queue.
- Control the degree of concurrency.
- Allocate Workers.
- Handle execution timeouts.
- Listen for node completion events.
- Update node states.
- Trigger downstream nodes.
2. The scheduling process
```mermaid flowchart TD A["Workflow state update"] --> B["Find nodes pending scheduling"] B --> C["Check node dependencies"] C --> D{"Execution conditions met?"} D -->|"No"| E["Remain Pending"] D -->|"Yes"| F["Check concurrency and resource limits"] F --> G{"Resources available?"} G -->|"No"| H["Wait for resources"] G -->|"Yes"| I["Add to execution queue"] I --> J["Worker fetches task"] J --> K["Execute node"] K --> L["Submit execution result"] L --> M["Update node status"] M --> A style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F style B fill:#EAF3EC,stroke:#83A995,color:#315D49 style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976 style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style F fill:#EAF3EC,stroke:#83A995,color:#315D49 style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976 style J fill:#EAF3EC,stroke:#83A995,color:#315D49 style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D style L fill:#EAF3EC,stroke:#83A995,color:#315D49 style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F ```3. Event-driven scheduling
A simple implementation is to periodically scan all nodes pending execution.
For example, check at fixed intervals:
- Which nodes have satisfied their dependencies.
- Which nodes are waiting to be retried.
- Which nodes have timed out.
- Which instances need to be recovered.
This approach is simple to implement, but as the number of workflows grows, it produces a large amount of wasted scanning.
A more common optimization is event-driven scheduling.
Example events:
NodeCompletedNodeFailedNodeRetryDueWorkflowStartedHumanApprovalReceivedTimeoutTriggered
When a node completes, the system only needs to check the downstream nodes it affects, rather than scanning the entire workflow.
```mermaid flowchart TD A["Node execution completes"] --> B["Publish NodeCompleted event"] B --> C["Event handler receives event"] C --> D["Locate downstream dependent nodes"] D --> E["Update dependency status"] E --> F{"Execution conditions met?"} F -->|"Yes"| G["Submit scheduling task"] F -->|"No"| H["Continue waiting for other dependencies"] G --> I["Scheduler performs subsequent scheduling"] style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976 style C fill:#EAF3EC,stroke:#83A995,color:#315D49 style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style I fill:#EAF3EC,stroke:#83A995,color:#315D49 ```Event-driven scheduling can reduce wasted scanning, but it needs to handle duplicate event delivery, out-of-order events, and lost events.
Therefore, event handling should be idempotent and backed by a reliable event persistence mechanism.
4. Scheduling queues and Workers
In a distributed workflow engine, the scheduler usually does not execute all nodes directly; instead, it distributes execution tasks through a task queue.
Typical architecture:
flowchart TB
A["Workflow Engine"] --> B["Scheduler"]
B --> C["Task Queue"]
C --> D["Worker 1"]
C --> E["Worker 2"]
C --> F["Worker 3"]
D --> G["Node Executor"]
E --> G
F --> G
G --> H["Execution Result"]
H --> I["State Store"]
I --> A
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
Worker responsibilities:
- Fetch execution tasks.
- Load node inputs.
- Invoke the node executor.
- Handle execution timeouts.
- Submit execution results.
- Send execution events.
The task queue and Workers can be scaled independently, thereby improving system throughput.
5. Concurrency Control
Multiple nodes in a workflow may be eligible to execute at the same time, but this does not mean they can run with unlimited concurrency.
The following need to be considered:
- Global concurrency limit.
- Concurrency limit for a single workflow instance.
- Concurrency limit for a single node type.
- Model API rate limiting.
- Number of database connections.
- Available Worker resources.
- User-level resource quotas.
For example, a workflow can execute multiple document parsing tasks at the same time, but nodes that call external models may need to have their concurrency limited.
Common control strategies:
| Strategy | Purpose |
|---|---|
| Semaphore | Limit the number of tasks executed simultaneously |
| Token Bucket | Control the request rate over a given period |
| Priority Queue | Schedule tasks by priority |
| Rate Limiter | Limit the frequency of API calls |
| Resource Pool | Manage available execution resources |
| Backpressure | Slow task submission when overloaded |
6. Priority Scheduling
When multiple workflow instances are waiting to execute at the same time, tasks can be scheduled according to business priority.
For example:
- Tasks from paying users take priority.
- Interactive tasks take priority over batch tasks.
- Tasks approaching their deadlines take priority.
- High-priority business tasks take priority.
However, priority scheduling needs to take starvation into account.
If high-priority tasks keep arriving, low-priority tasks may be unable to execute for a long time.
Strategies such as dynamic priorities, wait-time weighting, and fair scheduling can be adopted.
7. Data Passing and Context Management in Workflows
Nodes in a workflow usually do not execute in isolation. Data produced by upstream nodes needs to be passed to downstream nodes.
1. Workflow Context
Context is the data container for a workflow instance during execution.
It usually contains:
- Initial input.
- Node outputs.
- Intermediate processing results.
- User and tenant information.
- Runtime configuration.
- Execution metadata.
- Shared state.
For example:
{
"workflow_id": "customer_service",
"instance_id": "run_1001",
"context": {
"input": {
"question": "订单什么时候发货?"
},
"intent": {
"type": "order_status"
},
"order": {
"order_id": "O10001",
"status": "shipped"
},
"answer": {
"text": "您的订单已经发货。"
}
}
}
2. Explicit Data Mapping
A more reliable approach is to pass data through explicit data mapping between nodes.
For example:
{
"source": "extract_node",
"target": "analysis_node",
"mapping": {
"contract_name": "input.name",
"parties": "input.parties",
"amount": "input.amount"
}
}
This design makes data dependencies explicit and avoids implicit coupling between nodes.
3. Three Modes of Data Passing
Shared Context Mode
All nodes access a unified Workflow Context.
The advantage is simple implementation; the disadvantage is that it easily leads to data overwrites and concurrent write conflicts.
Explicit Input/Output Mode
Nodes declare inputs and outputs, and the execution engine handles data mapping.
The advantage is that dependencies are clear, making it suitable for type checking and static analysis.
Event Passing Mode
After a node completes, it publishes an event; downstream nodes subscribe to the event and obtain the result.
Suitable for asynchronous, distributed, and event-driven architectures.
Real systems can combine these modes, but data ownership and consistency boundaries must be made explicit.
4. Concurrent Write Conflicts
Suppose two nodes update the same context at the same time:
Node A: context.user.profile
Node B: context.user.profile
Without concurrency control, data written later may overwrite data written earlier.
Common solutions:
- Each node writes only to its own namespace.
- Use immutable state and state patches.
- Use version numbers for optimistic locking.
- Use explicit merge strategies for shared fields.
- Manage critical business state within separate database transactions.
For parallel workflows, it is not recommended to let multiple nodes directly modify the same large JSON object.
8. Workflow Persistence, Checkpoints, and Breakpoint Recovery
Persistence is an important capability that distinguishes a workflow engine from an ordinary function call chain.
1. Why Is Persistence Needed?
Suppose a workflow needs to execute 20 nodes and has already completed the first 15.
If the service process crashes at this point and there is no persistence mechanism, the system may need to re-execute all tasks.
If the preceding nodes include:
- Large model calls.
- Large file parsing.
- External API requests.
- Database writes.
Re-execution incurs additional cost and may even produce duplicate side effects.
Therefore, a workflow engine needs to save execution progress.
2. Checkpoints
A checkpoint is a recoverable snapshot of workflow state saved at a certain point in time.
It usually includes:
- Workflow instance state.
- Set of completed nodes.
- Current node state.
- Node outputs.
- Workflow context.
- Pending tasks.
- Information about events being awaited.
- State version.
flowchart TD
A["Workflow Start"] --> B["Execute Node A"]
B --> C["Save Checkpoint 1"]
C --> D["Execute Node B"]
D --> E["Save Checkpoint 2"]
E --> F["Execute Node C"]
F --> G{"Is Process Healthy?"}
G -->|"Yes"| H["Continue Executing Node D"]
G -->|"No"| I["Reload Persisted State"]
I --> J["Restore Checkpoint 2"]
J --> K["Re-check Node C Execution Result"]
K --> L["Continue Execution After Recovery"]
H --> M["Workflow Complete"]
L --> M
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note: A checkpoint saves execution state, but it does not necessarily save the true state of all external systems.
If a node calls an external payment API, a checkpoint cannot automatically guarantee that the payment operation and the local state update are in the same transaction.
3. Node-Level Persistence vs. Workflow-Level Persistence
Workflow-level persistence
Saves the state and context of the entire workflow instance.
Suitable for restoring the overall execution progress of a workflow.
Node-level persistence
Saves the execution record, inputs, outputs, and status of each node.
Suitable for pinpointing node failures, execution retries, and auditing.
The two usually need to be used together.
4. The Exactly-once Problem
Workflow engines often face an important question:
How do you guarantee that a node executes only once?
In a distributed environment, strict Exactly-once execution semantics are very difficult to guarantee.
For example:
- A Worker calls an external API.
- The external API has already executed successfully.
- The Worker crashes before committing the local result.
- The scheduler considers the task incomplete.
- The task re-enters the queue.
- Another Worker executes it again.
This causes duplicate operations.
Therefore, real-world systems typically adopt:
- At-least-once delivery.
- Idempotent execution.
- Deduplication mechanisms.
- Operation status queries.
- Transactional messaging mechanisms.
For nodes with side effects, a stable business idempotency key should be used.
For example:
idempotency_key =
workflow_instance_id + node_id + business_operation_id
When the same business operation is submitted repeatedly, the external service identifies the duplicate request by the idempotency key.
Note that the scope, validity period, and storage location of the idempotency key must be explicitly agreed upon by the business system; you cannot rely solely on the workflow engine to generate a random ID.
5. Consistency Between State Commits and Task Delivery
Distributed workflow engines have another problem:
The node status update succeeds, but the downstream task is not delivered successfully; or the task has already been delivered, but the status update fails.
This leads to lost tasks or duplicate execution.
A common solution is the Transactional Outbox:
- Update the node status within the same database transaction.
- Write the pending event at the same time.
- A background event publisher reads the Outbox.
- Deliver the event to the message queue.
- Mark the event as published.
- The consumer handles duplicate events through an idempotency mechanism.