Starflow checkout runbook

How the per-run git worktree pipeline works, where it lives on disk, and what to do when it misbehaves. ADR-042 is the design doc; this is the operator manual.

The two paths on disk

/var/lib/starflow/repo-mirror.git     ← bare mirror, owned by starflow:starflow
/tmp/starflow-run-<runId>/            ← per-run worktree, owned by starflow:starflow

The bare mirror is the cache. It holds objects (with --filter=blob:none so blobs come on demand). The per-run worktree is a checkout against that mirror — fresh for every run, removed when the run reaches a terminal status (success / error / cancelled). Paused runs keep theirs so resume lands in the same tree.

Boot sequence

On systemctl start starflow-server, the entrypoint prints (visible in journalctl -u starflow-server):

[starflow] checkout port: git-worktree (bootstrap starting)
[starflow] checkout port: mirror ready (N ms)

First boot on a fresh host runs a real git clone --bare --filter=blob:none — expect 20–40 s. Subsequent boots are instant (ensureMirror is idempotent).

If you instead see [starflow] checkout port: noop, the starflow.yaml config.repo key is missing. Pipelines will run but no source will be staged; shell steps land in an empty per-run scratch dir as before ADR-042.

Per-run trace

In workflow_run_events for any successful run:

seq=1  node_start    node=__trigger__
seq=2  node_complete node=__trigger__
seq=3  node_start    node=step_checkout
seq=4  node_complete node=step_checkout   payload.worktree=/tmp/starflow-run-<runId>  payload.ref=<sha>
seq=5+ node_start    node=<first real step>
…

step_checkout failing fast on worktree add failed is the most common surface — usually means the ref doesn’t resolve (typo’d SHA, deleted branch). Inspect the event’s payload for the exact git worktree add stderr.

Inspect the mirror

sudo -u starflow git -C /var/lib/starflow/repo-mirror.git log --oneline -5
sudo -u starflow git -C /var/lib/starflow/repo-mirror.git worktree list

Worktree list should show one entry per actively-running pipeline. Stale entries (process crashed before removeWorktree) get swept by the next worktree prune, which the port runs after every cleanup.

Force-fetch (when webhook lag suspects)

sudo -u starflow git -C /var/lib/starflow/repo-mirror.git fetch --prune origin '+refs/heads/*:refs/heads/*'

Equivalent to what the port runs internally before every worktree add. Manual invocation is only useful when you’re chasing a “main has the commit but the mirror doesn’t” suspicion.

Disk fill repair

If the host runs out of disk, the worktree adds and fetches will fail. To recover:

# 1. Stop the server.
sudo systemctl stop starflow-server
# 2. Sweep all per-run worktrees + prune from the mirror.
sudo rm -rf /tmp/starflow-run-*
sudo -u starflow git -C /var/lib/starflow/repo-mirror.git worktree prune
# 3. GC the mirror. Frees orphan objects, repacks. ~30 s.
sudo -u starflow git -C /var/lib/starflow/repo-mirror.git gc --aggressive
# 4. Restart.
sudo systemctl start starflow-server

If GC isn’t enough, nuke the mirror and let ensureMirror rebuild from upstream:

sudo systemctl stop starflow-server
sudo rm -rf /var/lib/starflow/repo-mirror.git
sudo systemctl start starflow-server   # first-run clone, ~20-40 s

When checkout fails for every run

Symptoms: every pipeline run shows step_checkout: error with the same message.

Common causes:

Symptom	Cause	Fix
Permission denied on /var/lib/starflow	Dir owned by root, not starflow	chown -R starflow:starflow /var/lib/starflow
unable to access 'https://...': could not resolve host	DNS / network outage	Investigate; usually transient
Authentication failed	GITHUB_TOKEN is missing or revoked	Re-materialize via vault, restart the unit
fatal: not a valid object name	Trigger context sent a stale SHA (branch force-pushed)	Re-trigger with current SHA; no host fix needed
worktree add failed ... '<path>' already exists	Prior run crashed before cleanup, leftover dir	sudo rm -rf /tmp/starflow-run-<runId> then worktree prune

Per-pipeline opt-out

Some pipelines genuinely don’t need a checkout (pure metadata / orchestration). Add their names to config.checkout.skip in starflow.yaml:

checkout:
  mirror_dir: /var/lib/starflow/repo-mirror.git
  skip: [smoke, sf-mark-orphaned-runs]

For these pipelines the engine swaps to NoopCheckoutPort at trigger time. Shell steps run with the empty per-run scratch dir as cwd — same behaviour as pre-ADR-042.

Open questions (not yet fixed)

• Resume + worktree. When a paused run resumes, the engine creates a new WorkflowEngine instance. We currently don’t restore projectRoot from checkpoint, so a resumed run won’t know its old worktree path. Symptoms: shell steps after resume cd into the wrong dir. Workaround: don’t pause shell-heavy pipelines. Real fix: persist worktree path with the checkpoint and restore on hydrate.
• GC schedule. No automatic GC today; relies on disk-fill-driven manual GC. Cron’d gc --auto would prevent the disk-fill scenario.