diff --git a/.claude/knowledge/_index.md b/.claude/knowledge/_index.md index 288bed5..17a2637 100644 --- a/.claude/knowledge/_index.md +++ b/.claude/knowledge/_index.md @@ -6,6 +6,7 @@ - `architecture/model-economics.md` — Which model per skill and why (Haiku/Sonnet/session-model) - `architecture/idempotent-scaffolding.md` — Scaffolding into shared user files: absorb unmarked sections, lazy dirs, never overwrite user content - `architecture/worktree-task-file-copy.md` — Why `/kickoff` copies the task file into the worktree (not a symlink): no skill simplification, avoids accidental-commit repo poisoning + cross-platform fragility +- `architecture/manager-worker-orchestration.md` — Design decisions for the coordinated Manager/Worker model: lane identity = worktree path, cross-agent (claude/codex/grok) git-as-uniform-bus, the `.mailbox/` protocol (outbox+inbox, hook-driven push), milestone worker autonomy, merge sequencer, roadmap-as-derived-view ## Features - `features/backfill-and-origin-metadata.md` — `/backfill-knowledge` significance bar + origin-reconstruction cascade diff --git a/.claude/knowledge/architecture/manager-worker-orchestration.md b/.claude/knowledge/architecture/manager-worker-orchestration.md new file mode 100644 index 0000000..2238d8e --- /dev/null +++ b/.claude/knowledge/architecture/manager-worker-orchestration.md @@ -0,0 +1,188 @@ +--- +title: "Manager/Worker Orchestration (design)" +createdAt: 2026-07-18 +createdFrom: "session: design-manager-worker-orchestration 2026-07-18" +pluginVersion: 1.8.1 +prime: false +--- + +# Manager/Worker Orchestration (design) + +Design decisions for evolving work-system from fire-and-forget kickoffs into a +coordinated Manager/Worker model. This is the **decision record**; the +implementation is spawned across tasks `add-lane-registry`, `spike-agent-mail-substrate`, +`add-lane-mailbox`, `add-manager-watch-loop`, `extend-worker-autonomy`, +`add-merge-sequencer`, `add-roadmap-skill`, `add-mailbox-statusline`, and +`add-agent-broadcast` (follow-up). Full working notes: the design task's `DESIGN-NOTES.md`. + +## The model +- **Manager** = the Claude Code session at the main repo root (herdr `◉` tab). A + *coordinator*, not a merge robot — the human stays merge authority unless + explicitly delegated at kickoff. +- **Worker** = one {claude|codex|grok} session per worktree, driving its task to a + reviewed, mergeable PR. +- **Lane** = `(worktree_path, task, branch)`. **Identity = worktree_path** — the one + key stable across agent types and restarts. herdr pane/tab, `agent_status`, + session UUID, PR state are live-attached attributes, **never identity**. Only the + task file persists; everything else is derived live (no lane registry file, no + stashed ids). + +## Gate verdict (herdr substrate, verified live 2026-07-16, herdr 0.7.0) +- An `agent start`-launched tab **does** expose a pollable `agent_status` ∈ + `idle|working|blocked|done|unknown`. `blocked` (worker on an input/permission + prompt) is the primary "needs the Manager" signal. Poll via `agent list`/`agent + get`; **bounded** block-wait via `agent wait --status … --timeout` (no busy loop). +- Status is produced by **installed hook integrations** merged with a TUI-scrape + rule engine (`agent explain`), NOT self-reported by the agent. +- `agent read --source recent` returns clean text incl. Claude's `※ recap:` line + + statusline. This RESOLVES the previously-UNVERIFIED agent-status flag in + [herdr-kickoff-automation](../features/herdr-kickoff-automation.md) / + [herdr-close-automation](../features/herdr-close-automation.md) and adds `blocked` + to their documented `idle|working|done` set. + +## Cross-agent constraint (workers may be claude, codex, OR grok) +The Manager is always Claude Code. Workers are tiered by herdr integration: +- **claude, codex** — installed hook (`herdr-agent-state.sh`) → authoritative status + **+ session UUID**; `agent read` carries branch+`PR#` (claude also `※ recap:`). +- **grok** — **no installable integration, no session UUID**, TUI-scrape-only status + (coarse; `blocked` unlikely), minimal `agent read`. +→ The orchestration **contract is expressed in git/PR terms**, the ONE channel +uniform across all three (every agent produces commits/branches/PRs). `agent_status` +is a coarse liveness layer; `agent read` is best-effort detail, parsed *per agent +type*, never format-required. Agent-specific skills (`/open`, `/rebase`) are each +worker's *implementation* of hitting agent-agnostic milestones — soft-coupling +([skill-composition](skill-composition.md)) applied across agent types. + +## Worker↔Manager mailbox (the coordination substrate) +A file mailbox carries judgment/intent signals that have no git artifact. +- **Central `~/.agent-mail/` — NOT per-worktree** (decided 2026-07-19; the community + "agent mail" name). Participant id = **encoded absolute path**; a hook derives its dir + from `cwd`: `~/.agent-mail/lanes//{outbox,inbox}/`. Central over + `/.mailbox/` because (a) it is the rendezvous for **broadcast / cross-Manager** + (below), (b) never-commit is automatic (outside every repo — no per-repo gitignore), + (c) no cross-worktree-boundary writes. Survives `/close` removing the worktree. +- **Prior art & substrate (researched 2026-07-19; spike-first decided).** Our design is + the recognized **inbox/outbox pattern**, and **AMQ** (`avivsinai/agent-message-queue`, + Go single binary) is startlingly close (Maildir delivery, `.agent-mail` store, + swarm/agent-teams mode, federation, presence, wake). `spike-agent-mail-substrate` + **decided (2026-07-19): adopt AMQ core messaging as the transport** behind a soft-coupled + adapter (like swarm's codex/grok). **AMQ is a prerequisite, not reimplemented** — absent ⇒ + a `brew install` hint + degrade to the git/PR floor + manual relay (no thin-Maildir + fallback; that floor already exists, and a missing mailbox ≈ today's fire-and-forget, so no + regression). We use AMQ's own envelope + store (AMQ-native). **`amq swarm` is NOT used at + all**: despite the name it is a *bridge to Claude Code's native "Agent Teams"* (shared task + list at `~/.claude/tasks/{team}/`), a competing orchestration substrate (no worktrees, no + PR/CI grounding, grok can't join without the bridge). We call only `send`/inbox-outbox/ + `who`/`receipts`/`dlq`/`wake`. AMQ owns transport; herdr-derived lanes own state. The + mechanics are fixed: **Maildir** (atomic `rename()`, + no locking, `tmp/new/cur`, one file per message — supersedes the earlier append-JSONL + + byte-offset idea; unread = files in `new/`, read = moved to `cur/`) and **AMQ's own JSON + envelope** (`{schema, id, from, to[], thread, subject, created, priority, kind}`; our 5 + semantic types map onto `kind`+`subject`+body). **CloudEvents / A2A** stay *conceptual* + alignment (A2A's Task lifecycle maps to our lane states) for a future HTTP bridge — not the + on-disk format now that we're AMQ-native. +- **Message types:** `ready-to-close`, `blocked-on-decision`, `coordination-request`, + `needs-human`, `broadcast-request`. +- **Topology — outbox + inbox, NOT inbox-only.** Each participant writes **only its + own outbox**; the Manager is the **only** writer of any inbox. Buys: single writer + per mailbox (Maildir already makes concurrent delivery lockless), no cross-participant + writes by a worker, and the **single-sequencer invariant** — worker→worker messages are + *addressable* but **route through the Manager** (it drains outboxes, delivers to + inboxes, may reorder/veto/batch). Workers never write another lane's inbox. +- **`ready-to-close` is worker-authoritative + a handoff report.** Only the worker knows + if a *second* PR is open or a post-merge TODO remains — a single PR's `✓ merged` is a + soft "near-done" hint, NOT a close trigger. The message carries the worker's fresh + context as `{summary, follow_ups[], deploy?, updates[], learnings[]}`: `follow_ups` → + Manager `/define`; `deploy` runs **from main after merge** (Manager-at-main model); + `updates` = dependency/config recommendations (e.g. a plugin bump); `learnings` → + Manager `/curate`. The Manager acts on these, then it/human runs `/close`. + +## Broadcast + cross-Manager (follow-up phase; central location adopted now) +The central store unlocks system-wide coordination across the *multiple Managers* a +machine runs (one per project). `~/.agent-mail/broadcast/global.jsonl` is the one +multi-writer append log every Manager reads (offset-tracked); `managers/.json` +presence files make Managers discoverable. Use: system-wide notices, or capability +queries ("who has Cloudflare access?") answered point-to-point into the asker's inbox. +**Single-sequencer up a level:** a worker emits `broadcast-request` and its **Manager** +decides/posts; **Managers peer-to-peer** (they are the coordinators). Aligns with +`plugin-settings-system`'s `[related_projects]` peering registry. Broadcast + registry +ship as a dedicated follow-up (`add-agent-broadcast`), not the Wave-1 core. + +## Why not Claude Code Agent Teams (the native alternative) +CC's experimental **Agent Teams** (`CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1`; lead spawns +teammate processes, shared task list at `~/.claude/tasks/{team}/`, mailbox at +`~/.claude/teams/{team}/inboxes/`) solves a *different* problem: short-lived, **Claude-only**, +single-working-tree fan-out inside one session. Four blockers rule it out as our base: (1) +teammates ARE Claude processes → no codex/grok (our hard requirement); reaches externals only +via the AMQ bridge we use natively. (2) **No worktree isolation** → can't do N parallel +independent PRs. (3) Task status is **self-reported** (documented "teammate forgets +`completed`" bug) vs. our git/PR-artifact grounding. (4) No PR/review/merge/deploy lifecycle — +our four pillars. Also session-scoped/ephemeral (team config dies with the lead) vs. our +persistent worktree-lanes. **Complementary, not competing** — and both converge on the same +Maildir envelope (AMQ mirrors AT's `kind/priority/thread/receipts`), so a lane could later be +bridged to an AT team. Worth stealing now: AT's `TeammateIdle` hook validates our +Notification-hook auto-emit (T4); its dependency-aware auto-unblock informs the merge +sequencer (T5) / roadmap waves (T6). + +## Push without polling — Claude Code hooks first +Files are pull; hooks make delivery push-like at turn boundaries, all gated by a +cheap check (Maildir ⇒ unread = files in `inbox/new/`, a `readdir`): +- **Stop hook** (turn end): new files in `inbox/new/` → `decision:block` + + `additionalContext` injects them so the worker continues, then moves them to `cur/`. + Respect `stop_hook_active` (exit 0 when true) + block only when genuinely new → no + loop / 8-block cap. Injected text is context; the worker still applies its own + autonomy gates. +- **SessionStart(`resume`)**: pull what arrived while closed. **UserPromptSubmit**: + belt-and-suspenders prepend. +- **Notification hook** (`permission_prompt`/`idle_prompt`) auto-emits + `blocked-on-decision`/`needs-human` to the outbox — worker-stuck detection becomes + push, no Manager poll. +- **Manager** drains all outboxes at its own Stop/UserPromptSubmit boundaries. +- The **one gap**: a worker idle *at the prompt* (already Stopped). Only here does a + herdr **idle-wake ping** (`agent send` onto idle) earn its keep. Optional. +Hooks are Claude-Code-specific → reinforces **claude-first**; codex/grok stay +poll/Manager-inferred until they grow equivalents. + +## Worker autonomy arc (milestone-defined, agent-tiered) +Default arc = git/PR **milestones**: `commits → push → PR open → review pass → +ready-to-close`. claude self-drives via work-system/pr-flow/swarm; codex/grok drive +to *PR open* with `gh`, and the **Manager fills the review gap** (`/swarm:review --pr +N` — any agent's PR is a valid target). Kickoff **pre-authorizes** commit/push/PR/ +review/own-branch-rebase/agreed-fixes. **Stop-early gates** (→ `blocked`/escalate): +failing CI, scope drift, any destructive/irreversible step, **merge to main** (never +without explicit delegation), force-push beyond own-branch lease, any human decision. + +## Merge sequencing — deterministic decision helper +`merge-sequencer.sh` (a *decision* helper, not an actor) over the lanes view outputs: +mergeable-now (`◆` + CI green + no conflict) and, after a merge, rebase-due lanes +(behind main ∧ file overlap). One PR at a time; the human authorizes each; the +Manager sends rebase intents. The `marketplace.json`+`plugin.json` version-line +collision is baked in. Encodes today's ROADMAP prose rule as a script (prose-drift). + +## Roadmap — a derived view, co-equal with the GH board +Source of truth = task files + git/PR state. `ROADMAP.md` and the future GH board +(see task `design-github-projects-task-source`) are **two derived views**, not +competing sources — ROADMAP is the local/offline projection, the board the +shared/online one; both read the same `states`/`task-status.sh` substrate. Inside +ROADMAP.md, a marker-delimited **Manager-auto block** (per-lane state ○●◇◆✓, PR#, +version) is regenerated from the lanes view; the **human-curated block** (waves, +serialization policy, rationale) is never machine-touched (idempotent-scaffolding: +absorb-unmarked, never overwrite user content). Owned by a new `/roadmap` skill; +`/list` stays the ephemeral snapshot. + +## Safety envelope (cross-cutting) +Fail-closed pane id (reuse `worktree-tab-state` tri-state — never inject into an +unverified pane); no force-push beyond own-branch `--force-with-lease`, never +shared/main; no merge-to-main without delegation; bounded herdr waits, survey refresh +`--cached`; grok `always-approve` ⇒ Manager sends nothing destructive to a grok lane. + +## Reused existing substrate (build ON, don't reinvent) +- `ws-statusline.sh states
[--cached]` → `task\tstate\tglyph` = a ready-made + lane registry with states; the state machine ○●◇◆✓ + `◉` and the tab↔worktree join + already exist in [herdr-tab-glyphs](../features/herdr-tab-glyphs.md). +- The PR cache (`headRef\tstate\treviewDecision`) + sync/`--cached` refresh policy. + +Related: [skill-composition](skill-composition.md), +[idempotent-scaffolding](idempotent-scaffolding.md), +[herdr-kickoff-automation](../features/herdr-kickoff-automation.md), +[herdr-tab-glyphs](../features/herdr-tab-glyphs.md).