Workflow Suspension (suspend/resume)¶

At a glance

What it is — A graph run can suspend while the workflow waits — for a person (an approval, missing information) or for another system (a batch job, a downstream process): its model is persisted to an external state store and the run completes normally. A later request with the same business correlation ID resumes where the workflow left off. A long-running business process becomes a sequence of short runs — nothing stays in memory between them.

Two ways to pause — a checkpoint node does its work, then pauses; on resume the workflow continues at its next step(s) without re-running it. A decision node chooses — continue, branch, or pause — and decides again with the new input on every resume.

The store is pluggable — Redis ships as the minigraph-state-redis extension; any composable function honoring the store contract works.

It scales to complex processes — records are scoped by graph + correlation ID, so every graph (a domain's own, or each subgraph behind an orchestrating parent) suspends and resumes independently under one shared business correlation ID (the orchestrator pattern).

When a workflow must wait¶

Two everyday situations produce the same problem — a multi-step process that cannot finish in one go:

Waiting for another system (fully automatic). A step starts a long batch job — for example by publishing a Kafka message — and the processing takes minutes or hours. When it finishes, the batch processor calls back with the same correlation ID and the workflow picks up at the next step. No person is involved: suspension is not only for human-in-the-loop.
Waiting for a person. A user submits a service ticket for a warranty replacement. The workflow finds information missing, emails the user, and pauses. When the user replies, the system resumes the workflow by its correlation ID — the ticket number — and continues where it left off.

In both cases the workflow pauses rather than ends: its memory is kept safe outside the application, and any application instance can pick it up when the reply arrives.

Why short runs¶

The reply — the batch completion, the manager's approval, the user's email — may take minutes, hours or days. Parking a live graph instance for that long would pin memory, defeat timeouts, and not survive a restart. Suspension inverts the problem: the run ends — the caller gets a {"type": "suspended", "cid": ...} reply — and the workflow's durable memory (the model namespace) waits in the state store under the business correlation ID with a time-to-live you choose. The resumed run is an ordinary graph execution that happens to start with restored state. Because the record key is the graph plus the correlation ID, a workflow suspended on one application instance can resume on any instance sharing the store.

The three vocabulary pieces¶

1. The suspend node — exactly one per graph, and the alias suspend is reserved (like root and end): traversal jumps to it by name. Its skill assembles and persists the state envelope through the attached store function — no data mapping needed:

create node suspend
with type Suspend
with properties
purpose=Persist workflow state to the external state store
skill=graph.suspend
task=v1.redis.persist.model
ttl=2d

ttl is mandatory with no default — a suspended workflow may wait a minute or days, and only the workflow designer knows. It uses duration syntax (20s, 5m, 2h, 2d) and becomes the store record's expiry. The suspend node also needs an outgoing connection (normally to end): without one, the record would persist and the run would then stall instead of completing — the compiler rejects the graph.

2. A suspension point — two patterns, named after the node that pauses:

A checkpoint node pauses after doing its work. Think of it as a hand-off: the node records what just happened (the order, the uploaded document), stages the reply that tells the caller what comes next, and the workflow pauses. A checkpoint never wonders whether to pause — reaching it is the decision. On the canvas it draws two arrows (the compiler enforces both): one to suspend — pause here — and at least one continuation edge — this is where the workflow continues when it wakes up, a single step or a fan-out. A resumed run continues along the continuation and never re-executes the checkpoint itself.
A decision node chooses — and pausing is one of its choices. A graph.math decision looks at the incoming request and returns where the workflow goes next: a node name (continue there) or suspend (pause and keep waiting). It draws arrows only to its real outcome targets — never to suspend: pausing is a returned choice, not an edge (the compiler rejects a decision-to-suspend edge). On resume the decision is re-executed against the new input, so it re-decides every time — "keep waiting until a valid answer arrives" is one node with no extra wiring. When suspend is reachable only this way, anchor it behind an island (root -> island -> suspend) so the graph has no orphan nodes — traversal stops at the island, so the anchor edge is never walked.

The two patterns compose. When the pause itself is conditional, put a decision in front of a checkpoint: one outcome routes into the checkpoint (do the hand-off, then pause), another routes past it (no pause needed):

flowchart LR
    d{"decision"} -->|needs a pause| c["checkpoint"]
    d -->|no pause| n["next step(s)"]
    c -->|pause| s[["suspend"]]
    c -->|"continuation - a resumed<br>run starts here"| n

And the decision pattern alone is the wait-loop — re-evaluate each incoming request until one moves the workflow forward:

flowchart LR
    d{"decision"} -->|approved| a["next step"]
    d -->|rejected| t["terminal step"]
    d -.->|"waiting..."| d
    r(["root"]) --> i["island"] --> s[["suspend"]]

The ADRs and the compiler internals call these shapes edge mode (checkpoint node — the drawn edge to suspend is the declaration) and jump mode (decision node — the IF-THEN-ELSE returns suspend). Same behavior, engineering names.

The suspend node cannot be an exception handler (exception=suspend is rejected). The retired suspend=true property is accepted and ignored for one deprecation window (the compiler logs a WARN): every valid earlier model already draws its checkpoint edge, which now declares the same behavior — models deploy unmodified.

3. The resume node — conventionally named resume, placed right after root (or after setup nodes). When the store has a record for model.cid, it restores the model, re-arms the traversal bookkeeping (a downstream graph.join still sees branches that completed before suspension), and continues at the suspension point: past a checkpoint node along its continuation edge, or by re-executing a decision against the new input. When there is no record — a fresh transaction, the normal first-run case, or an expired one — traversal simply continues along the resume node's own forward path.

Either way, the skill records the outcome in model.run — resume when a record was restored, fresh when there was none. The engine deliberately does not distinguish absent from expired (with several checkpoints in one graph, no single fallback node could be right for all of them): whether an expired approval needs its own response is application logic. Gate the resume node's forward path with a graph.math IF-THEN-ELSE — on model.run or on the request shape, exactly as tutorial-14 does — to reject the request, advise the UI, or jump to a recovery node.

create node resume
with type Resume
with properties
purpose=Restore workflow state from the external state store
skill=graph.resume
task=v1.redis.retrieve.model

Types (Suspend, Resume, Suspensible) are visual convention — they pick the node colors in the Playground; the skill defines the behavior.

Walkthrough: the purchase workflow (tutorial-14)¶

tutorial-14 (shipped with the engine, runnable in the minigraph-playground example app) is the complete multi-checkpoint pattern — three human checkpoints, four short runs, one correlation ID:

flowchart LR
    root(["root"]) --> resume --> order["order<br>(checkpoint)"] --> check{"check-approval<br>(decision)"}
    check -->|approved| approval["approval<br>(checkpoint)"] --> delivery["delivery<br>(checkpoint)"] --> ship --> done(["end"])
    check -->|rejected| reject["manager-reject"] --> done
    check -.->|waiting: pauses,<br>re-decides on resume| check

A customer orders, the store manager approves or rejects with a reason, the delivery department releases the shipment, and the parcel ships — one suspend node serves every suspension point. The order, approval and delivery nodes are checkpoint nodes: each draws its checkpoint edge to suspend, captures its actor's input into the model and stages its own stage-specific reply (overriding the default suspended response). The manager's decision lands at a graph.math decision node on the order checkpoint's continuation with three outcomes: an approved decision routes to the next suspension point, an explicit rejection routes to a terminal node that reports the manager's reason (the workflow ends), and anything else — a missing or unrecognized decision — returns suspend and pauses. Because the decision is re-executed against the new input on every resume, an invalid request can never end a long-running workflow by accident: the workflow keeps waiting and re-decides when the next request arrives, with no extra wait nodes. Run it with Redis (e.g. helpers/redis-standalone) and drive the four runs with one correlation ID.

Run 1 — the customer orders a laptop; the run suspends at the order checkpoint and replies with "run": "fresh" (a new transaction):

curl -s -X POST http://127.0.0.1:8085/api/graph/tutorial-14 \
  -H 'content-type: application/json' -H 'x-correlation-id: order-1001' \
  -d '{"item": "laptop", "amount": 2000}'

{"stage": "order-submitted; waiting for store manager approval", "run": "fresh", "cid": "order-1001"}

Run 2 — with the same x-correlation-id, the store manager approves. The resume node restores the persisted state and continues past the order checkpoint without re-executing it, into the check-approval decision — every reply from here on carries "run": "resume":

curl -s -X POST http://127.0.0.1:8085/api/graph/tutorial-14 \
  -H 'content-type: application/json' -H 'x-correlation-id: order-1001' \
  -d '{"decision": "approved", "manager": "store-88"}'

{"stage": "approved; waiting for the delivery department to release the shipment", "run": "resume", "cid": "order-1001"}

Run 3 — the delivery department releases the shipment:

curl -s -X POST http://127.0.0.1:8085/api/graph/tutorial-14 \
  -H 'content-type: application/json' -H 'x-correlation-id: order-1001' \
  -d '{"release": true, "courier": "express"}'

{"stage": "released; waiting for shipment confirmation", "run": "resume", "cid": "order-1001"}

Run 4 — shipment confirmation completes the workflow:

curl -s -X POST http://127.0.0.1:8085/api/graph/tutorial-14 \
  -H 'content-type: application/json' -H 'x-correlation-id: order-1001' \
  -d '{"tracking": "TRK-12345"}'

{
  "stage": "shipped",
  "run": "resume",
  "order": {"item": "laptop", "amount": 2000},
  "approval": {"decision": "approved", "manager": "store-88"},
  "delivery": {"release": true, "courier": "express"},
  "shipment": {"tracking": "TRK-12345"},
  "cid": "order-1001"
}

Every stage's input crossed every suspension — the model accumulated order, approval and delivery across four separate runs, and a later checkpoint simply re-persisted the grown state under the same correlation ID.

The manager may reject instead — the alternative run 2. An explicit "decision": "rejected" routes to the terminal rejection, which reports the manager's reason together with the original order, and the workflow ends — the record was already consumed on resume and nothing re-suspends, so a further request under the same correlation ID is a fresh 404 rejection:

curl -s -X POST http://127.0.0.1:8085/api/graph/tutorial-14 \
  -H 'content-type: application/json' -H 'x-correlation-id: order-2002' \
  -d '{"decision": "rejected", "reason": "budget exceeded"}'

{
  "stage": "rejected",
  "reason": "budget exceeded",
  "order": {"item": "laptop", "amount": 2000},
  "run": "resume",
  "cid": "order-2002"
}

A missing or unrecognized decision takes the third path: the reply is "stage": "awaiting-decision; supply decision approved or rejected for the store manager" and the workflow re-suspends — check-approval returns suspend, and because a pausing decision re-executes on every resume, the next request re-evaluates the decision: only an explicit approved or rejected moves the workflow forward. This is also what a replay against a leftover suspended record now yields — a self-explanatory "still waiting" instead of a surprise.

The tutorial also validates its input: a request that is not an order submission, for a correlation ID with no suspended record, is rejected with HTTP 404 — the order must come first. Three techniques worth stealing from its model:

Null-safe presence check. The math expression engine has no null literal, but {var} substitution inside a text() constant is null-safe: MAPPING: text(={input.body.item}) -> model.order_probe always yields a present string (=null when the field is absent), which an IF can compare safely. The check-approval decision reuses the same idiom, so a missing decision safely jumps back to the checkpoint rather than raising a runtime error.
A decision can stage the caller's reply. graph.math MAPPING: statements run before its IF statements and may write output.* — check-approval stages the awaiting reply unconditionally, and the approval/rejection paths overwrite it downstream. A decision-initiated suspension therefore replies with the decision's own staged body (or the default {"type": "suspended", ...} when nothing was staged).
The run flag. graph.resume sets model.run to fresh or resume, and the tutorial stages it into every reply (model.run -> output.body.run) — so the UI always knows whether it is looking at a new transaction or a resumed continuation, and a rejected later-stage request tells the caller why ("run": "fresh" on a decision-shaped body means the record expired or never existed).
Declarative response status. A graph may stage its own HTTP status — int(404) -> output.status in the rejection node. A non-2xx status routes through the surrounding flow's exception handler, which passes a staged map body through (minus any stack key, with its status key corrected) — so give your rejection fields names other than status.

curl -s -X POST http://127.0.0.1:8085/api/graph/tutorial-14 \
  -H 'content-type: application/json' -H 'x-correlation-id: order-9999' \
  -d '{"decision": "approved"}'

{"type": "rejected", "message": "Transaction not found. Submit the order first", "run": "fresh", "status": 404}

Design rules¶

The model is the workflow's durable memory. Only the model namespace persists — a node's {node}.result scratch does not survive suspension. Map anything a later step needs into model.* before the checkpoint.
A checkpoint node is a complete working step — only its exit changes. It executes its skill in full (input mapping → skill → output mapping) before pausing, so it may carry any non-routing skill (graph.data.mapper, graph.task, graph.api.fetcher, graph.extension), capture the actor's input into model.*, and stage the caller's reply in output.* — and its continuation edge defines exactly where the next run continues after resume. What it never does is choose: a checkpoint always suspends when its skill completes.
Route the choice, don't property it. When the input decides the workflow's direction (approve vs reject vs wait), make the decision a graph.math node whose outcomes route naturally: continue to the next checkpoint, branch to a terminal node, or return suspend to keep waiting. The decision draws edges only to its real outcome targets — never to suspend (the compiler rejects that shape) — and re-executes on every resume, so each new request is re-evaluated. tutorial-14's check-approval is the pattern.
A suspension point must be the sole active branch. Do not suspend between a fan-out and its join — branches in flight cannot be persisted (the engine logs a warning); suspend after the join instead. Joins whose predecessors completed before suspension work: their completion marks are part of the persisted state.
One resume per transaction. The shipped stores consume the record atomically on retrieval (Redis GETDEL on 6.2+, or a MULTI/EXEC GET+DEL transaction on older servers — detected automatically), so a duplicate resume — a double click, a retried message — finds nothing and behaves as a fresh run instead of double-executing the continuation. A later checkpoint in the resumed run simply persists a new record under the same ID.
The correlation ID is a resume capability. Whoever presents it continues the workflow: protect resume-bearing endpoints with rest.yaml authentication, and use non-guessable IDs (the engine generates UUIDs when the caller supplies none).
Suspension is self-contained per graph. Records are scoped by graph + cid, so a resume only ever sees records written by its own graph — you cannot suspend in one subgraph and resume in another. A delegated subgraph inherits the parent's business correlation ID and is fully resumable on its own; the parent orchestrates — see the orchestrator pattern.
Reserved model keys (model.cid, model.instance, model.flow, model.ttl, model.trace, model.run) are never persisted — the resumed run's own identity is authoritative. model.run is part of the read-only flow metadata family: graph.resume is its only writer, and the flow compiler rejects any data mapping that targets it (like the other reserved keys).

The orchestrator pattern¶

A complex business process rarely pauses in one place. Model each processing path as its own subgraph and let a parent graph orchestrate:

The parent delegates each path with graph.extension. The child inherits the parent's business correlation ID automatically — the same way an Event Script sub-flow inherits model.cid — so a subgraph that suspends persists its record under its own graph + the shared cid.
Each subgraph is a complete resumable workflow on its own: a resume node after root, checkpoints or decisions where it must wait, records scoped to that subgraph. Parent and subgraphs never collide, because every graph has its own record.
A suspended subgraph replies like any suspended graph — {"type": "suspended", "cid": ...} (or whatever output it staged) — and that reply lands in the parent extension node's result, so the parent can route on it with a decision. Probe the possibly-absent key with the = guard (a completed path's reply has no type):

statement[]=MAPPING: text(={model.path.type}) -> model.path_probe
statement[]='''
IF: {model.path_probe} == '=suspended'
THEN: waiting
ELSE: done
'''

Re-invoking the parent with the same correlation ID resumes the paths: each subgraph's resume node finds its own record (or starts fresh), fast-forwards past its checkpoint, and the parent assembles the results. A subgraph is also a deployed graph, so a single path can be resumed directly at POST /api/graph/{subgraph-id} with the same correlation ID — same capability rules as any resume endpoint.
One record per graph per cid. Invoke a suspendable subgraph once per correlation ID per run — a for_each fan-out of the same subgraph under one cid would overwrite its own record. Parallel paths belong in different subgraphs.

The engine's reference models for this pattern are the unit-test-orchestrator / unit-test-sub-suspend pair in the minigraph test sources, pinned end-to-end by GraphSuspendResumeTest.

The state store contract¶

The store is an ordinary composable function named by the suspend/resume nodes' task property — the Redis module below is one implementation; PostgreSQL, DynamoDB, MongoDB or anything else plugs in the same way.

Persist — invoked by graph.suspend; headers type=put; request body:

{
  "cid":   "<business correlation ID>",
  "graph": "<the graph that suspended - cid + graph form the retrieval key>",
  "node":  "<the suspension point>",
  "ttl":   172800,
  "model": { "the model namespace minus reserved keys": "..." },
  "seen":  { "traversal bookkeeping": true },
  "run":   { "traversal bookkeeping": true }
}

Store the body opaquely (the reference implementations use MsgPack — binary values round-trip; note the platform's serialization gotchas) and reply 2xx only when the record is durable — the reply is the acknowledgement graph.suspend requires before the graph completes; any error fails the suspension.

Retrieve — invoked by graph.resume; headers type=get; body {"cid": "...", "graph": "..."}. Return the stored record as-is, or null / an empty map when absent or expired — an absent record is the normal fresh-transaction case, never an error. Consume the record atomically on retrieval (or document your replay semantics). If the store has no native TTL, implement record expiry yourself.

Scope every record by graph + cid, never by cid alone. The same business correlation ID legitimately suspends in more than one graph — one transaction may cross several domains' graphs, and an orchestrator's subgraphs each pause independently (see the orchestrator pattern). A cid-only key would collapse all of them into one record. This also makes suspension self-contained per graph by construction: a resume only ever sees records written by its own graph — you cannot suspend in one subgraph and resume in another.

The smallest possible reference implementation is the engine's test fixture — a temp-file store of ~60 lines (FileStateStore in the minigraph test sources).

The Redis store module¶

extensions/minigraph-state-redis ships v1.redis.persist.model (SETEX, native expiry) and v1.redis.retrieve.model (atomic consume-on-retrieve). Records are keyed graph:{graph_id}:{cid}, so each domain's graph and each subgraph suspends independently under a shared business correlation ID. The consume strategy is version-aware: native GETDEL on Redis 6.2+, or an equally atomic MULTI/EXEC GET+DEL transaction on older servers — detected once per connection from INFO server and stated in the startup log, since enterprise deployments rarely control their managed Redis version (and the community Windows binary used by redis-standalone is 5.0.14). Include the jar and the two functions register automatically; the connection is lazy, so the application boots normally without Redis until a workflow actually suspends. Configuration uses the same redis.* keys as the sync-over-async extension (redis.host, redis.port, redis.password, redis.ssl, redis.database, redis.timeout.ms), and the worker counts are ops-tunable via worker.instances.v1.redis.persist.model / worker.instances.v1.redis.retrieve.model. See the module README for details.