Starflow state model

A practical map of where workflow run state lives, how it gets there, who is allowed to read what, and how we detect drift between tiers. This document is the implementation counterpart to ADR-045 §5 — the ADR sets the contract; this doc shows the wiring.

Updated 2026-06-01. Reflects the codebase after task #240 (lease columns + heartbeat) lands. Sections marked (planned) are spec’d by ADRs 045/047 but not yet implemented.

The three tiers

   ┌──────────────────────────────────────────────────────┐
   │  COLD — system of record                             │
   │                                                       │
   │    SupabaseWorkflowStateAdapter   (cloud / multi-    │
   │                                    tenant prod)      │
   │    LocalWorkflowStateAdapter      (SQLite — self-    │
   │                                    hosted, single-   │
   │                                    node, dev)        │
   │    NullWorkflowStateAdapter       (no-op default)    │
   │                                                       │
   │  Tables:                                             │
   │    workflow_definitions  (graphs — planned, #241)    │
   │    workflow_runs         (one row per run)           │
   │    workflow_run_events   (append-only event log)    │
   │    workflow_session_vars (K/V session store)         │
   │    mirror_outbox         (drift-resilient bridge —   │
   │                           planned, ADR-045 §5)       │
   └──────────────────────────────────────────────────────┘
                            │
                            ▼  outbox drain (continuous, sub-second — planned)
   ┌──────────────────────────────────────────────────────┐
   │  WARM — live state overlay                            │
   │                                                       │
   │    SpacetimeDB (task #179 activation)                │
   │                                                       │
   │  Reads accept up to ~2s staleness. Used for live      │
   │  dashboards, agent peer-discovery, step progress      │
   │  streams. NOT authoritative for resume, audit,        │
   │  history, billing.                                    │
   └──────────────────────────────────────────────────────┘

The adapter selection is per-deployment configuration, not a build-time choice. A self-hosted operator picks Local for single-node simplicity; cloud picks Supabase. The interface (WorkflowStateAdapter in packages/starflow-engine/src/state-adapter.ts) is identical across all three; the engine doesn’t know which is wired.

The mirror tier wraps the cold tier (MirroringWorkflowStateAdapter) — it does not replace it. Cold is always authoritative; warm is hint.

Authority rules

Operation	Read from	Write to	Notes
Resume a paused run	cold	—	Loads checkpoint_json and the graph (definition_id, #241)
Reconcile orphaned/hung runs	cold	cold	Lease sweep updates the run row
Show run history (dashboard /orchestrate/runs)	cold	—	Direct Supabase read today
Show a single run’s detail	cold	—	Direct Supabase read; latency-tolerant
Show live run status (planned)	warm	—	Push subscription via SpacetimeDB
Agent peer-discovery (planned)	warm	cold	CQRS split — agents read warm for “what is my peer doing right now,” write cold for state transitions
Audit / billing / compliance reads	cold	—	Never trust warm — staleness window invalidates audit guarantees
Step-progress telemetry (planned)	warm	warm only	High-volume; never reaches cold

The line worth remembering: if being wrong has consequences, read cold. If being 1–2 seconds stale is fine, read warm.

Tables

workflow_runs

One row per run; the primary unit of state.

Column	Type	Purpose
id	uuid PK	Run id; matches WorkflowContext.runId
template_id	text NULL	Human-readable pipeline name (e.g. ci, nightly)
workspace_id	text NOT NULL DEFAULT ‘celestial’	Tenant scope (#224)
env	text NOT NULL DEFAULT ‘prod’	Environment scope (#224)
status	text	running / paused / success / error
started_at	timestamptz	When onRunStart fired
completed_at	timestamptz NULL	When onRunComplete / onRunError fired
duration_ms	int NULL	Wall-clock duration
checkpoint_json	jsonb NULL	Full WorkflowContext snapshot; non-null only while paused
error	text NULL	Failure message from onRunError
claim_state	text NOT NULL DEFAULT ‘running’	Lease state machine — queued/claimed/running/complete/failed/abandoned (ADR-047 §3)
runner_pool	text NULL	Pool affinity (NULL = any pool may claim; planned for ADR-047)
leased_by	text NULL	Opaque runner identity holding the lease
lease_until	timestamptz NULL	When the current lease expires; sweep target
last_heartbeat	timestamptz NULL	Last refresh; observability only
claim_attempts	int NOT NULL DEFAULT 0	Increments on claim/abandon cycles (retry policy input)
definition_id	uuid FK NULL	References workflow_definitions(id) — planned, #241

Indexes (relevant ones):

• idx_workflow_runs_workspace_started (workspace_id, env, started_at DESC) — dashboard list query
• idx_workflow_runs_lease_until (lease_until) WHERE lease_until IS NOT NULL — sweep target
• idx_workflow_runs_claim_pool (claim_state, runner_pool, started_at) WHERE claim_state = 'queued' — ADR-047 claim path

Migrations: 001 (initial), 018_workflow_runs_workspace, 019_workflow_runs_lease.

workflow_run_events

Append-only event stream. One row per node-state transition + run-level events. Powers the dashboard’s per-run timeline and the per-step API (/api/runs/:id/events, #221).

Column	Type	Purpose
id	bigserial PK	Stable ordering
run_id	uuid FK	Joins to workflow_runs
event_type	text	node_start / node_complete / node_error / run_paused / run_resumed / phase / runner_lost
node_id	text NULL	Set for node-scoped events
payload_json	jsonb NULL	Inputs / outputs / error details
created_at	timestamptz DEFAULT now()	Ordering tie-breaker

workflow_session_vars

K/V store backing the session-read / session-write step types.

Column	Type	Purpose
session_id	text	Caller-supplied scope
key	text	Variable name
value_json	jsonb	Serialised value

Default lifetime today: forever. Planned: drop on onRunComplete for the originating run, with opt-in keepUntil for cross-run state (ADR-045 §10 gap 6).

mirror_outbox (planned)

Same-transaction bridge between cold writes and warm projection. Every state-adapter call enqueues an outbox row; a background drainer pushes rows to SpacetimeDB and marks them applied. Outbox depth is the drift signal.

Column	Type	Purpose
id	bigserial PK	Drainer cursor
run_id	uuid	Owning run
kind	text	run / event / session_var / step_progress
payload	jsonb	Self-contained projection payload
created_at	timestamptz DEFAULT now()	For lag-by-age gauge
applied_at	timestamptz NULL	Set when drainer succeeds

Not implemented yet; tracked as a follow-up to #179 (SpacetimeDB activation).

Write boundaries

Every state-adapter method corresponds to a precise lifecycle moment. The engine awaits each at its boundary — there is no fire-and-forget on the cold path.

Method	Fires when	Writes
onRunStart(runId, ctx)	Engine constructed, before queue	INSERT workflow_runs (status running)
writeLease(runId, runnerId, until)	Right after onRunStart	UPDATE workflow_runs set claim_state, leased_by, lease_until, last_heartbeat
refreshLease(runId, until)	Every 10s while executing	UPDATE workflow_runs set lease_until, last_heartbeat
onNodeStart(runId, nodeId, input)	Before step handler invocation	INSERT workflow_run_events (event_type node_start)
onNodeComplete(runId, nodeId, output)	After step handler resolves	INSERT workflow_run_events (event_type node_complete)
onNodeError(runId, nodeId, error)	After step handler rejects (terminal — past retry)	INSERT workflow_run_events (event_type node_error)
saveCheckpoint(runId, ctx)	Inside enterDurableWait	UPSERT workflow_runs.checkpoint_json
onRunPaused(runId, ctx)	Inside enterDurableWait	UPDATE workflow_runs (status paused); INSERT run_paused event
onRunResumed(runId)	Inside continuePausedWaitNode	UPDATE workflow_runs (status running, clear checkpoint); INSERT run_resumed event
onRunComplete(runId, ctx, durationMs)	finalizeRun, success path	UPDATE workflow_runs (status success, duration); clearCheckpoint
onRunError(runId, error)	finalizeRun, failure path	UPDATE workflow_runs (status error, message)
clearLease(runId)	finalizeRun (terminal and paused)	UPDATE workflow_runs clear leased_by + lease_until
recordPhaseIfChanged(runId, ctx, meta)	Various lifecycle moments	INSERT phase event when variables.phase value changes
sessionGet(sessionId, key)	Inside session-read handler	SELECT from workflow_session_vars
sessionSet(sessionId, key, value)	Inside session-write handler	UPSERT workflow_session_vars

The wait/wake ordering invariant

From ADR-045 §8, codified in enterDurableWait:

1.  set context.variables[__celestialWait] = pending
2.  set node.status = "waiting"          (in-memory)
3.  await onRunPaused(...)               (DB row -> "paused")
4.  await saveCheckpoint(...)            (durable context written)
5.  await recordPhaseIfChanged(...)
6.  await emitWaitWake(...)              (HOST registers external wake)

Step 6 must be last. If the host’s wake registration fires before the checkpoint is durable (steps 4–5), a resume call that arrives early hits an empty checkpoint and the run is stuck. Regression test in tests/wait-checkpoint-before-wake.test.ts asserts the order and the fault-injection case.

Lease + heartbeat machinery

   ┌──────────────────────────────────────────────────────┐
   │  Engine (one per run)                                 │
   │                                                       │
   │   onRunStart                                          │
   │      └→ writeLease(runnerId, now+30s)                 │
   │   startHeartbeat (setInterval 10s)                    │
   │      └→ refreshLease(now+30s)  ← every 10s            │
   │   ...workflow body...                                 │
   │   finalizeRun                                         │
   │      ├→ stopHeartbeat                                 │
   │      └→ clearLease  ← paused OR terminal              │
   └──────────────────────────────────────────────────────┘

   ┌──────────────────────────────────────────────────────┐
   │  starflow-server                                      │
   │                                                       │
   │   _startLeaseSweep                                    │
   │      ├→ once-at-boot pass                             │
   │      └→ setInterval 30s                               │
   │           └→ expireOldLeases("runner_lost")           │
   │                = UPDATE workflow_runs                 │
   │                   SET status='error',                 │
   │                       claim_state='abandoned',        │
   │                       error='runner_lost'             │
   │                   WHERE lease_until < now()           │
   │                     AND status IN ('running','paused')│
   └──────────────────────────────────────────────────────┘

Numbers:

• Lease TTL: 30 s
• Heartbeat interval: 10 s
• Sweep interval: 30 s

A healthy runner has 3× refresh margin before its own lease expires. A dead runner gets cleaned up within ~60 s (one sweep cycle past expiry).

Paused runs do not hold leases. When the engine reaches enterDurableWait, finalizeRun is the next call — which clears the lease. On resume, continuePausedWaitNode re-acquires the lease and restarts the heartbeat. The interval between pause and resume is unbounded; the run row sits there with status='paused', lease_until=NULL, and the sweep ignores it.

Heartbeat failures are logged, not propagated. A transient Supabase blip shouldn’t fail an otherwise-healthy run. If the blip persists past TTL, the sweep handles it the same way it handles a crashed runner.

Defensive cleanup — execute() wraps the body in try/catch; if anything before finalizeRun throws, stopHeartbeat() runs in the catch before re-throwing. The lease itself expires naturally and the sweep cleans up.

Drift detection (planned model)

Today, the mirror adapter is a thin pass-through wrapping the cold adapter — it forwards each call to the inner adapter and additionally pushes a context snapshot to the warm-tier port. The mirror is best-effort: if SpacetimeDB is down, the cold tier still succeeds and the run completes.

ADR-045 §5 specifies the upgrade path:

1. Outbox writes are synchronous with cold writes. Every state-adapter call appends a mirror_outbox row in the same transaction.
2. Drainer is continuous. Background worker drains outbox → warm with sub-second latency under healthy load.
3. Outbox lag is the drift signal. Gauge: count(mirror_outbox WHERE applied_at IS NULL) by age_bucket. Climbing depth = drifting warm tier. Alert at >2s lag, page at >30s.
4. Periodic 30 s background sweep. Compares hash of recent cold rows vs warm rows; corrects divergence. Cheap; catches anything the outbox missed.
5. End-of-run reconciliation is the per-run safety net, not the primary mechanism.

The contract: drift is detected within seconds via outbox lag, fixed within 30 s via the background sweep, double-checked at run completion as the per-run net. Long-paused runs never trigger end-of-run reconciliation, which is why the periodic sweep exists — they’re caught regardless.

Implementation order:

1. Add mirror_outbox table + outbox enqueue inside MirroringWorkflowStateAdapter. (Cold writes already happen inside the inner adapter; outbox is the additive piece.)
2. Build outbox drainer as a starflow-server background task (similar shape to the lease sweep).
3. Wire count(mirror_outbox WHERE applied_at IS NULL) by age to telemetry (HyperDX once #162 lands).
4. Background hash-compare sweep — last.

Not started; tracked in ADR-045 §10 gap 5.

Reserved namespace

WorkflowContext.variables keys with the prefix __celestial are engine internals. User-defined graphs must not write keys with this prefix; the engine may add or rename them without notice.

Today’s reserved keys:

Key	Owner	Purpose
__celestialWait	wait-kernel	Pending durable-wait envelope (WaitPending)
__celestialCheckout	ADR-042 checkout	Materialised worktree path + ref

Planned additions: __celestialLeases (consolidated VPN/resource leases — ADR-045 §10 gap 10), __celestialIdempotency (worker-job idempotency keys — ADR-047 §7).

What today’s code does vs ADR-045’s full spec

Capability	Status
Cold adapter interface (Supabase / SQLite / Null)	✅ Implemented
Per-run WorkflowContext checkpoint	✅
Wait/wake ordering invariant (saveCheckpoint before emitWaitWake)	✅ (task #242)
Lease columns + heartbeat	✅ (task #240)
Periodic lease-expiry sweep	✅ (task #240)
Boot-time orphan reconciliation (defensive backstop)	✅
mirror_outbox synchronous enqueue	⏳ Planned, ADR-045 §5
Outbox drainer + lag metric	⏳ Planned
Warm-tier authoritative reads (live UI / agent coordination)	⏳ Planned (needs #179)
onStepProgress warm-only channel	⏳ Planned, ADR-045 §5
workflow_definitions table + graph durability	⏳ Planned (task #241)
Worker SDK job table (worker_jobs)	⏳ Planned, ADR-047 §8
HMAC-signed wait tokens	⏳ Planned (task #238)
Session-var TTL / GC	⏳ Planned, ADR-045 §10 gap 6

This doc is the operational map. The ADRs are the contract. When an ADR moves from “planned” to “shipped” the relevant rows here flip from ⏳ to ✅.

References

• ADR-045 — Starflow engine semantics (the state model contract)
• ADR-047 — Runner pool architecture (claim/lease semantics + worker SDK)
• ADR-045 §10 gap 1 — orphan detection via lease sweep (task #240)
• ADR-045 §10 gap 5 — outbox + drift detection
• packages/starflow-engine/src/state-adapter.ts — the interface
• packages/starflow-engine/src/supabase-state-adapter.ts — cloud impl
• packages/starflow-engine/src/local-state-adapter.ts — SQLite impl
• packages/starflow-engine/src/mirroring-state-adapter.ts — warm-tier wrapper
• packages/supabase/migrations/019_workflow_runs_lease.sql — current schema