Scribe processors

Scribe processors are the derivation layer of the celestial-intelligence substrate. They read from the captured event stream (chat_events, atoms, messages) and produce typed projections in dedicated Supabase tables (geo_documents, geo_issues, geo_proposals, …). The dashboard, agents, and analytics all consume those projections.

This doc is the operator’s reference: how processors fit together, how to invoke one, where each lands its output, and how to add a new one.

For the architectural background see:

• ADR-049 — conversations + turns schema (the substrate processors read from)
• ADR-051 — scribe architecture (file watcher + processor pattern)
• ADR-058 — PM substrate (the geodesic.* processors)

The shape

  ~/.claude/projects/*.jsonl
            │
            │  scribe-ingest (file watcher, ADR-051)
            ▼
  chat_events  (Supabase)        ─┐
  messages     (SQLite mirror)     │   PROCESSORS read here
  atoms        (extracted intent)  │
                                  ─┘
            │
            │  scribe-memory processors
            ▼
  derived tables (Supabase)
  ─────────────────────────────
  geo_documents       ← ADR/SPEC/PRD bodies, status, owner
  geo_issues          ← Linear-style work items
  geo_issue_deps      ← typed dependency edges
  geo_proposals       ← review-queue from feature-request mining
  geo_provenance      ← edges back to source turns/atoms
  wiki_pages          ← rendered knowledge pages
  embeddings          ← per-atom vectors for retrieval
  …
            │
            │  dashboard + agents
            ▼
  /geodesic/library /board /deps /inbox /runs
  /pm  (planned)  · agent retrieval · analytics

Each processor is a thin, stateless TypeScript class implementing the Pipeline interface (see packages/scribe-memory/src/interfaces.ts). It owns:

• A stable id (e.g. geodesic.docs) — what you pass to the CLI.
• A run(ctx) method — does one pass of work and returns a PipelineResult.
• Optional cursor tracking via the processor_cursors table so re-runs are incremental, not full scans.
• Mandatory telemetry via the processor_runs table — every run inserts a row with start/end/duration/items-created/error, which is what surfaces in /geodesic/runs in the dashboard.

Invocation

All processors run through the scribe-memory CLI:

SCRIBE_WORKSPACE_ID=celestial-intelligence \
SUPABASE_CONNECTION_STRING="$(ss vault get-secret SUPABASE_CONNECTION_STRING \
  --workspace=celestial --env=prod --raw)" \
node packages/scribe-memory/dist/cli.js pipeline <id>

You can also wire a processor into a Starflow pipeline (see the existing processor-feature-requests and processor-feature-requests- backfill pipelines in starflow.yaml — they invoke the sub-pipelines of feature-requests with cursor + artifact wiring).

For one-off testing without Supabase, set SCRIBE_MEMORY_STORE_URL=sqlite:./test.db to run against a local file.

The processor catalogue

Ingest from agent / human conversations

Id	What it does	Source → output
turns-to-atoms	LLM-extracts knowledge atoms (facts, decisions, todos, etc.) from messages since the cursor watermark.	messages → atoms
code-to-atoms	Walks a local repo (workspace_packages tree) and emits an atom per file/symbol — used for grounding code-aware retrieval.	filesystem → atoms
scip-to-atoms	Reads an SCIP index file (@sourcegraph/scip) and emits one atom per indexed symbol with definition/reference metadata.	scip → atoms

Atom → projections

Id	What it does	Source → output
atoms-to-typed	Re-classifies untyped atoms into the typed atom enum using an LLM.	atoms → atoms (in-place type field)
atoms-to-index	Builds the search index over atom bodies + titles.	atoms → atom_index
atoms-to-embeddings	Per-atom vector embedding via an embedding model. Powers semantic retrieval.	atoms → embeddings
atoms-to-graph	Heuristic graph (entity-cooccurrence) from atom titles + bodies.	atoms → atom_graph
atoms-to-graph-llm	LLM-driven graph extraction (typed edges between atoms). Slower, higher-quality than the heuristic version.	atoms → atom_graph
atoms-to-wiki	Renders a wiki page per cluster of related atoms (Obsidian-style).	atoms → wiki_pages

Feature-request mining (geodesic proposal queue)

This is the original “what feature should we build” loop, decomposed into five sub-pipelines so each shows as a discrete step in starflow’s DAG view.

Id	What it does
feature-requests:read-cursor	Reads processor_cursors.last_scanned_at, writes cursor.json artifact.
feature-requests:scan-atoms	Scans atoms within the window into atoms.jsonl.
feature-requests:llm-cluster	LLM clusters atoms into candidate feature requests → clusters.json.
feature-requests:write-proposals	Inserts one geo_proposals row per cluster → written.json.
feature-requests:record-cursor	Advances processor_cursors + writes a processor_runs row.
feature-requests:confirm-status	Reviews each pending proposal against the codebase: shipped? in progress? backlog? Updates geo_proposals.status + reviewer_notes_md. v1 — rubric in the prompt.
feature-requests:confirm-status-v2	Same job, spec-aware rubric using the docs/specs/*.md substrate. SHIPPED-verified band populates from successful cel-verify runs.

Geodesic PM substrate (ADR-058)

This is the section the dashboard’s /geodesic/library, /geodesic/board, and /geodesic/deps views consume.

Id	What it does	Source → output
geodesic.docs	Walks docs/{adr,specs,prd}/*.md, parses the H1 + frontmatter table, computes sha256(body_md), upserts geo_documents keyed on (workspace_id, source_path). Idempotent on content_hash — re-runs are O(changed).	filesystem → geo_documents
geodesic.harness-tasks	Cursor-tracked scan of chat_events for tool_name IN ('TaskCreate','TaskUpdate'). Pairs args/result rows, upserts geo_issues by external_id="harness:<session>:<task#>". Applies state transitions and addBlocks / addBlockedBy edges into geo_issue_deps. Auto-creates the synthetic HARNESS team per workspace on first run.	chat_events → geo_issues + geo_issue_deps

The geodesic processors are the substrate behind:

• /geodesic/library — 57+ ADRs / SPECs / PRDs from geo_documents
• /geodesic/board — Kanban over geo_issues (incl. 330+ historical harness tasks)
• /geodesic/deps — typed dependency edges from geo_issue_deps

Spec-driven substrate (ADR-056)

Id	What it does
specs:extract-from-adr	LLM-drafts a docs/specs/SPEC-NNN-…md skeleton from an existing ADR. Used in batch mode to seed the spec corpus from accepted ADRs.
specs:coverage-analyze	Compares the SPEC corpus against the ADR corpus and the proposal queue; flags ADRs without specs, specs without verification runs, and proposals that semantically match existing specs (the “already-shipped” filter).

Web ingest (NewsWatch / research)

Id	What it does
news-feed-poll	Polls a single RSS/Atom feed, emits new items.
news-feeds-poll-all	Fan-out over feeds table — polls every enabled feed once.
news-triage-candidate	LLM-triages a single news candidate against the workspace’s profile (relevance, novelty, threat level).
news-triage-batch	Batched version of the above for cron-driven nightly runs.
news-assess-source	Per-source quality/credibility scoring used to weight inputs.
news-assess-after-ingest	Post-ingest reassessment when an item has new context (correlations, follow-ups).
web-fetch-html	Fetches a URL, stores raw HTML + headers.
web-extract-markdown	Boilerplate-stripped Markdown body from a fetched HTML.
web-segment	Splits a long article into semantically coherent segments for embedding.
web-extract-topics	LLM topic extraction per segment.
web-embed-cluster	Embeds segments and clusters into themes for retrieval.
web-emit-atoms	Emits one atom per segment so web content joins the unified retrieval substrate.

Repo atoms (ADR-055 groundwork)

Id	What it does
repo-clone-and-index	Clones a repo to a local mirror, runs a one-shot scip indexer.
repo-ingest	Reads a previously-cloned repo + indexer output and emits atoms (commits, files, symbols, todos).
repo-walk-commits	Walks git log from the cursor watermark, emits a commit-atom per commit (sha, message, author, files).
repo-walk-todos	Scans source files for TODO: / FIXME: / XXX: comments, emits a todo-atom per hit with line context.
repo-walk-docs	Walks docs/**/*.md in a repo and emits a doc-atom per file — complements geodesic.docs for non-celestial repos.

Telemetry

Every processor run inserts a row in processor_runs:

processor_id        text     -- e.g. 'processor.geodesic.harness-tasks@v1'
workspace_id        text
status              text     -- 'success' | 'error'
proposals_emitted   int      -- how many derived items this run produced
atoms_scanned       int
scan_from           timestamptz
scan_to             timestamptz
elapsed_ms          int
prompt_tokens / completion_tokens / total_tokens   -- if LLM was used
error               text     -- summary or null
started_at / completed_at    timestamptz

Cursor watermarks (for incremental processors) live in processor_cursors:

workspace_id, processor_id, last_scanned_at, total_atoms, total_proposals

These two tables back /geodesic/runs in the dashboard — that’s where you go to see “did the nightly geodesic.harness-tasks run land? how many issues did it create? did it error?”

How to add a new processor

1. Create packages/scribe-memory/src/pipelines/<your-id>.ts.

2. Implement the Pipeline interface:

   import { pipelineRegistry } from "../interfaces.js";
   import type { Pipeline, PipelineCtx, PipelineResult } from "../interfaces.js";
   
   class MyPipeline implements Pipeline {
     readonly id    = "my.pipeline";
     readonly label = "What it does in 6 words or less";
   
     async run(ctx: PipelineCtx): Promise<PipelineResult> {
       const workspaceId = (ctx.config["workspaceId"] as string | undefined)
                       ?? "celestial-intelligence";
       // … your work here …
       return { itemsCreated: 12, summary: "did 12 things" };
     }
   }
   
   pipelineRegistry.register(new MyPipeline());

3. Add a side-effect import in packages/scribe-memory/src/cli.ts so the processor self-registers when the CLI loads:

   import "./pipelines/<your-id>.js";

4. Write processor_runs telemetry from inside run() — copy the pattern from geodesic-docs.ts. The dashboard’s /geodesic/runs view will pick it up automatically.

5. (If incremental) advance processor_cursors at the end of each successful run. Copy the pattern from geodesic-harness-tasks.ts.

6. pnpm --filter @celestial/scribe-memory build and run with the CLI invocation above. Confirm a row lands in processor_runs and your target table got the writes.

7. (Optional) Wire into a Starflow pipeline if you want it cron-fired — the existing processor-feature-requests-backfill pipeline is the template.

Operational notes

• Workspace scoping: every processor takes a workspaceId config field. SCRIBE_WORKSPACE_ID env var is the convenient way to set it at the CLI layer.
• Idempotency: most processors are idempotent on their primary key (source_path for docs, external_id for issues, content_hash for fingerprinted content). Safe to re-run. The cursor pattern is an optimization, not a correctness requirement.
• Cron: scribe-memory doesn’t have a built-in scheduler. To run a processor periodically, wire it as a Starflow pipeline and use the existing schedule mechanism, or drop a launchd plist / systemd timer on the host. See docs/src/content/docs/runbooks/uptime.md for the cron / timer pattern celestial-intelligence already uses.
• Failures: when a processor errors mid-run, the partial work is committed (every projector writes per-row in a transaction-per-row style by default). The cursor is NOT advanced, so the next run will re-scan the same window — assume idempotency is real.
• LLM cost: processors that use the LLM (atoms-to-typed, feature-requests:llm-cluster, feature-requests:confirm-status*, news-triage-*, web-* LLM-driven ones) charge tokens to the configured model. The prompt_tokens / completion_tokens / total_tokens columns in processor_runs let you trace cost back to a run.

Related ADRs

• ADR-008 — scribe-memory storage abstraction
• ADR-049 — conversations + turns schema (the substrate)
• ADR-051 — scribe architecture (file watcher + processor pattern)
• ADR-054 — agent_actions queue (planned; will surface processor outputs as HITL items for human-or-agent approval before they materialize)
• ADR-055 — repo atoms (planned; the repo-* processors are the groundwork)
• ADR-056 — spec-driven substrate (the specs:* processors)
• ADR-058 — geodesic PM substrate (the geodesic.* processors)