Runbook: deploy scribe news-watch to Hetzner (ADR-008 Phase 4)

End-to-end deploy of the scribe-newswatch headless worker onto the existing Hetzner VPS, writing to a shared Supabase backend. This is the operational tail of ADR-008 — the architecture is built; this is the sequence of ss commands + SSH steps to actually fire.

Read this before running. Several steps incur infra cost or overwrite production state. Steps marked ⚠ are destructive / costly.

Prerequisites

• The Hetzner machine at 5.161.83.131 is up (verified via ssh [email protected] systemctl status ss-agent)
• ss CLI is on PATH locally
• Hetzner API token exists in the vault:

  ss vault list-keys hetzner

  If empty: ss vault set-key hetzner api_token=<from console.hetzner.cloud>
• A Supabase project is provisioned and its credentials are in the celestial vault (already there for decider-sweep):

  ss vault list-secrets --project=celestial --env=prod | grep -E "SUPABASE|DATABASE"

  Expected: SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_CONNECTION_STRING.
• Verify the SUPABASE_CONNECTION_STRING actually works. It can go stale if you rotate the DB password in the Supabase dashboard without refreshing the vault. Quick check:

  SCRIBE_STORE=supabase SCRIBE_VAULT_PROJECT=celestial SCRIBE_VAULT_ENV=prod SCRIBE_WORKSPACE_ID=test \
    node -e "import('@celestial/scribe-memory').then(({resolveMemoryStore}) => resolveMemoryStore().then(r => r.store.listProjects()).then(p => console.log('OK', p.length)).catch(e => { console.error('FAIL:', e.message); process.exit(1) }))"

  If you see FAIL: Tenant or user not found: open the Supabase dashboard → Project Settings → Database → “Connection string” tab → copy the Transaction pooler URI → update the vault:

  ss vault set-secret SUPABASE_CONNECTION_STRING='<paste-uri>' --project=celestial --env=prod

Step 1 — apply Postgres migrations to Supabase

The scribe schema (atoms, sessions, sources, feeds, candidates, pipeline_runs, etc.) needs to exist before the SupabaseMemoryStore can write to it. The migration runner pulls credentials from ss vault itself — no env-var dance required:

node packages/scribe-memory/dist/cli.js migrate --store=supabase

(--vault-project defaults to celestial, --vault-env to prod.)

Expected output:

[migrate] → 999_migration_tracker
[migrate] → 001_memory_schema
[migrate] → 002_rls_policies
[migrate] done — 3 migration(s) checked

⚠ The migrations enable RLS on every table. After this step, anonymous queries to Supabase see zero rows. Use the service-role key (which bypasses RLS) for admin work; the scribe-newswatch worker uses workspace_id + the same key to scope to its tenant.

Step 2 — smoke test SupabaseMemoryStore locally

Before deploying, verify the adapter can talk to Supabase from your laptop. Run news-feeds-poll-all (or any pipeline) against the new backend:

node packages/scribe-memory/dist/cli.js pipeline news-feeds-poll-all \
  --store=supabase --workspace=jek-laptop

Expected output line:

[scribe-memory] store: supabase (vault project=celestial env=prod) workspace=jek-laptop

If you see auth errors or “relation does not exist”, Step 1 didn’t complete; re-check.

Step 2.5 — bootstrap the host (one-time)

ss artifact deploy ships a pre-built Node bundle. The target host needs Node 22 installed exactly once. If you’ve never deployed to a host before:

ssh -i ~/.ssh/<your-hetzner-key> root@<host> \
  'curl -fsSL https://deb.nodesource.com/setup_22.x | bash - && \
   apt-get install -y nodejs && node --version'

This will eventually move into a proper ss host bootstrap <service> command — for now it’s a one-line copy-paste.

The artifact bundle itself does NOT include a Node binary. We debated bundling node-pre-gyp style but Phase 1 of ADR-011 keeps the artifact pure-JS for portability across host distros; the host runs apt install nodejs once.

Step 3 — deploy the artifact

ADR-011 Phase 1: build a self-contained tarball locally, scp it, flip a symlink. The Hetzner box never sees git/pnpm/repo source.

# Inspect what would change (build only; no upload)
ss artifact build scribe-newswatch --env=prod

# ⚠ Build + upload + activate
ss artifact deploy scribe-newswatch --env=prod \
   [--key=~/.ssh/<your-hetzner-key>]

What this does:

1. Compiles @celestial/scribe + @celestial/scribe-memory.
2. Vendors them under node_modules/@celestial/ in a staging dir along with their pruned external production deps (~43 packages, ~5 MB).
3. Tars to .starsystem/artifacts/scribe-newswatch-<commit>-prod.tar.gz.
4. scps to /tmp/ on the target.
5. Unpacks to /opt/scribe-newswatch/<commit>/.
6. Materializes /etc/scribe-newswatch/env from _systemd.env + _systemd.env_from_vault (reads OLLAMA_* from ss vault).
7. Renders + drops /etc/systemd/system/scribe-newswatch.service (no-op if unchanged; daemon-reload only when changed).
8. Atomically flips /opt/scribe-newswatch/current → <commit>.
9. systemctl enable --now scribe-newswatch + restart.
10. Prunes old revs beyond _artifact.keep_revs (default 5).

Rollback: SSH to the host and re-point current at any prior commit dir still on disk: ln -sfn /opt/scribe-newswatch/<old-sha> /opt/scribe-newswatch/current && systemctl restart scribe-newswatch. Phase 2 will turn this into ss artifact rollback.

Step 4 — verify the worker is running

# Check systemd status
ss exec --env=prod --service=ss-agent -- systemctl status scribe-newswatch

# Tail the journal
ss logs --env=prod --service=scribe-newswatch --follow

# Confirm process-manager / ss-agent sees it
ss status --env=prod

Expected:

• systemd: active (running)
• journal: [scribe news-watch] starting (pid …) + news-watch poll loop: every 30m
  • [scribe-memory] store: supabase (vault project=celestial env=prod) workspace=hetzner-newswatch

Step 5 — verify cross-machine state via the laptop UI

On your laptop, point scribe-serve at the same Supabase and confirm news the Hetzner worker ingests appears in your UI:

SCRIBE_STORE=supabase \
SCRIBE_VAULT_PROJECT=celestial SCRIBE_VAULT_ENV=prod \
SCRIBE_WORKSPACE_ID=hetzner-newswatch \
  node packages/scribe/dist/cli.js serve

(Use the same SCRIBE_WORKSPACE_ID as the worker so RLS gives you the same scope.)

Open http://localhost:3742 → News tab. Candidates the Hetzner worker discovered should appear within a poll cycle.

Rollback

If anything is wrong:

ss exec --env=prod --service=ss-agent -- systemctl stop scribe-newswatch
ss exec --env=prod --service=ss-agent -- systemctl disable scribe-newswatch

The Supabase data persists (it’s the durable layer). To wipe and start over, the migration runner’s _scribe_migrations table tracks applied versions — manual cleanup is required to re-apply migrations from scratch.

Open follow-ups (post-deploy)

• ADR-009 (cold-layer sync) — the outbox + scribe-syncer process that gives you the “local-first with cloud mirror” experience instead of “everything writes directly to Supabase”. Right now, when the Hetzner worker loses network, ingestion just halts. ADR-009 adds the local outbox that drains when the network returns.
• ADR-010 (warm-layer SpacetimeDB fanout) — sub-second cross-agent atom visibility. Optional; needed when there’s actually a second agent in the workspace.
• Move scribe-newswatch to Railway — once ADR-009 ships, the persistent-disk argument that pinned us to Hetzner disappears. Railway gives auto-deploy-from-git and shared quota with starflow-server. See ADR-008 §“Operational trajectory note”.