diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0d96a39..c2d1c86 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,11 @@ jobs: run: pnpm run typecheck - name: Lint (eslint) run: pnpm run lint + # Hard complexity ceiling earned by the 2026-08 complexity program + # (ADR-0037): no function over CC 50, enforced as an error. The + # advisory warn-25 tier lives in eslint.config.mjs. + - name: Complexity ceiling (CC ≤ 50, error) + run: pnpm run lint:cc - name: Markdown lint run: pnpm run lint:md - name: Build (packaging + load validation) diff --git a/docs/TRANSCRIPTS.md b/docs/TRANSCRIPTS.md index d761127..3b84240 100644 --- a/docs/TRANSCRIPTS.md +++ b/docs/TRANSCRIPTS.md @@ -36,12 +36,12 @@ rewritten; rule 3 of the module header, `usage-index.mjs:22-29`): | Host | Store | Discovered by | |---|---|---| -| Claude Code | `~/.claude/projects//.jsonl` | `listClaude` (`usage-index.mjs:916-926`) — exactly one level of project directories | -| Codex CLI | `~/.codex/sessions///
/rollout--.jsonl` | `listCodex` (`usage-index.mjs:878-891`) — the `yyyy/mm/dd` tree walk | +| Claude Code | `~/.claude/projects//.jsonl` | `listClaude` (`usage-index.mjs:206-218`) — exactly one level of project directories | +| Codex CLI | `~/.codex/sessions///
/rollout--.jsonl` | `listCodex` (`usage-index.mjs:221-240`) — the `yyyy/mm/dd` tree walk | -Roots come from `defaultRoots()` (`usage-index.mjs:908-912`) and are injectable +Roots come from `defaultRoots()` (`usage-index.mjs:198-203`) and are injectable for tests. A malformed line is skipped, never fatal (`jsonLines`, -`usage-index.mjs:381-387` — one corrupt line must not cost a whole file). +`usage-parsers.mjs:167-173` — one corrupt line must not cost a whole file). Host evidence is not inference-provider proof. A Claude transcript may describe Anthropic-, OpenRouter-, or Ollama-served inference. ADR-0016 defines separate @@ -52,32 +52,32 @@ transcript host/parser identity unless other evidence grounds the inference prov ### 1.1 Claude entry vocabulary Each line has a top-level `type`. The parser (`parseClaude`, -`usage-index.mjs:480-563`) reads: +`usage-parsers.mjs:376-401`) reads: | `type` | What the parser takes from it | |---|---| -| `ai-title` | The model-written session title (`usage-index.mjs:488`) — preferred over the first-prompt fallback | +| `ai-title` | The model-written session title (`usage-parsers.mjs:383`) — preferred over the first-prompt fallback | | `user` | A user-**role** turn — which is *not* the same as "the human"; see §3 | -| `assistant` | A model turn: `model` id, per-turn `usage` token counts, `tool_use` blocks (`usage-index.mjs:508-551`) | -| any | Side-band fields read regardless of type: `attributionSkill`/`attributionPlugin` (`usage-index.mjs:489-490`), `isSidechain` (`usage-index.mjs:491`), `cwd` for project derivation | +| `assistant` | A model turn: `model` id, per-turn `usage` token counts, `tool_use` blocks (`usage-parsers.mjs:329-368`) | +| any | Side-band fields read regardless of type: `attributionSkill`/`attributionPlugin` (`usage-parsers.mjs:384-385`), `isSidechain` (`usage-parsers.mjs:387`), `cwd` for project derivation | An assistant entry with `isApiErrorMessage: true` is a **local placeholder** Claude Code writes when a request dies before a real completion (connection drop, rate limit, auth failure — `model: ""`, all-zero usage). It is real engaged time but not a model attempt: counted as an *exception*, never -pushed into `models` or priced (`usage-index.mjs:595-606`; the full story is +pushed into `models` or priced (`usage-parsers.mjs:345-353`; the full story is [`USAGE-SCORECARD-METRICS.md`](USAGE-SCORECARD-METRICS.md) §10). ### 1.2 Codex entry vocabulary Codex rollout lines carry `type` + `payload`. The parser (`parseCodex`, -`usage-index.mjs:580-681`) reads: +`usage-parsers.mjs:578-593`) reads: | `type` / `payload.type` | What the parser takes from it | |---|---| -| `session_meta` | Authoritative session id, `cwd`, and `thread_source` (`usage-index.mjs:592-597`) — `"subagent"` marks a thread_spawn replay whose tokens are excluded from aggregation (`usage-index.mjs:662`; `USAGE-SCORECARD-METRICS.md` Appendix A, Bug B) | -| `turn_context` | The model id in effect from this point on (`usage-index.mjs:598`) | -| `event_msg` → `token_count` | A **cumulative** usage snapshot; only the last one is kept (`usage-index.mjs:662-664`) | +| `session_meta` | Authoritative session id, `cwd`, and `thread_source` (`usage-parsers.mjs:421-428`) — `"subagent"` marks a thread_spawn replay whose tokens are excluded from aggregation (`usage-parsers.mjs:546`; `USAGE-SCORECARD-METRICS.md` Appendix A, Bug B) | +| `turn_context` | The model id in effect from this point on (`usage-parsers.mjs:431-437`) | +| `event_msg` → `token_count` | A **cumulative** usage snapshot; only the last one is kept (`usage-parsers.mjs:474-480`) | | `event_msg` → `user_message` | A legacy-format real human prompt — Codex does not route tool output through this event | | `event_msg` → `agent_message` | A legacy-format model response | | `event_msg` → `item_completed` → `UserMessage` | A current-format real human prompt; text blocks use the observed lowercase `text` discriminator | @@ -131,8 +131,8 @@ The same parsers serve two very different callers, switched by `withTurns`: | Path | Entry point | `withTurns` | Message bodies | Cached? | |---|---|---|---|---| -| **Scan** — the aggregate index behind the Scorecard/Findings/Sessions views | `buildIndex` → `parseFile` (`usage-index.mjs:925`) | `false` | never held — holding them would balloon memory across 3,000+ files (`usage-index.mjs:490-493`) | yes: per-file derived records in `~/.config/agentic-kit/usage-index.json`, keyed `(path, mtime, size)`, invalidated wholesale by `SCHEMA_VERSION` (`usage-index.mjs:77`) | -| **Reader** — one transcript for the Transcript view | `readSession` (`usage-index.mjs:1566`) | `true` | full turn list built | **never** — every call re-reads and re-parses the one file | +| **Scan** — the aggregate index behind the Scorecard/Findings/Sessions views | `buildIndex` → `parseFile` (`usage-index.mjs:255`) | `false` | never held — holding them would balloon memory across 3,000+ files (`usage-parsers.mjs:371-375`) | yes: per-file derived records in `~/.config/agentic-kit/usage-index.json`, keyed `(path, mtime, size)`, invalidated wholesale by `SCHEMA_VERSION` (`usage-index.mjs:92`) | +| **Reader** — one transcript for the Transcript view | `readSession` (`usage-index.mjs:715`) | `true` | full turn list built | **never** — every call re-reads and re-parses the one file | ![Figure: one parser, two read paths — the scan path (withTurns false) caches per-file records keyed by path, mtime and size; the reader path (withTurns true) builds full turns and is never cached](assets/transcript-read-paths.svg) @@ -140,7 +140,7 @@ The reader path being cache-free is load-bearing for maintainers: **turn-shape changes (like the `kind` field, §3) need no `SCHEMA_VERSION` bump**, because no turn is ever served from cache — whereas *session-record* fields (like `exceptions`) do, since stale cached records would otherwise sum `undefined` -into totals (`usage-index.mjs:40-50`; the incidents behind that rule are +into totals (`usage-index.mjs:59-65`; the incidents behind that rule are recorded in `USAGE-SCORECARD-METRICS.md` Appendix A). --- @@ -153,11 +153,11 @@ recorded in `USAGE-SCORECARD-METRICS.md` Appendix A). |---|---|---| | `role` | all | `"user"` or `"assistant"` — the **Messages-API role**, not the author (see below) | | `at` | all | ISO timestamp | -| `text` | all | Flattened display text (`claudeText`, `usage-index.mjs:459-479` — binary payloads dropped: a pasted screenshot renders as `[image]`, a tool result is prefixed `[tool result]`) | -| `model` | assistant | The model id; the literal string `exception` for an API-error placeholder turn (`usage-index.mjs:526`) | +| `text` | all | Flattened display text (`claudeText`, `telemetry-records.mjs:38-55` — binary payloads dropped: a pasted screenshot renders as `[image]`, a tool result is prefixed `[tool result]`) | +| `model` | assistant | The model id; the literal string `exception` for an API-error placeholder turn (`usage-parsers.mjs:350`) | | `tools` | assistant | Tool names invoked in the turn | -| `prompt` | user | `isHumanPrompt`'s verdict (`usage-index.mjs:441-450`) — drives the **prompt counts** | -| `kind` | user | `'prompt'` \| `'tool-result'` \| `'context'` — drives the **attribution label** (`userTurnKind`, `usage-index.mjs:468-479`) | +| `prompt` | user | `isHumanPrompt`'s verdict (`usage-parsers.mjs:262-271`) — drives the **prompt counts** | +| `kind` | user | `'prompt'` \| `'tool-result'` \| `'context'` — drives the **attribution label** (`userTurnKind`, `usage-parsers.mjs:290-294`) | | `exception` | assistant | `true` on API-error placeholder turns | | `truncated`, `originalChars` | any | Present **only** when the turn was abridged (§4.3) | @@ -177,7 +177,7 @@ story is [Appendix A](#appendix-a--fix-history).) ### 3.2 `kind` — the attribution field -`userTurnKind` (`usage-index.mjs:523-528`) classifies every user-role turn: +`userTurnKind` (`usage-parsers.mjs:290-294`) classifies every user-role turn: | `kind` | Test | Meaning | |---|---|---| @@ -195,12 +195,12 @@ Two deliberate subtleties: * **Harness-output envelopes are excluded from the prompt *count* too.** `isHumanPrompt` shares `HARNESS_OUTPUT_RE`, so a session's `prompts` figure never counts stdout dumps or task notifications as things the person said - (`SCHEMA_VERSION` 5, `usage-index.mjs:47-51`; the correction this shipped + (`SCHEMA_VERSION` 5, `usage-index.mjs:66-69`; the correction this shipped with is in [Appendix A](#appendix-a--fix-history)). * **`tool-result` outranks `context`**: a `tool_result` block on an `isMeta` entry is still tool feedback. -Codex user turns are `kind: 'prompt'` by construction (`usage-index.mjs:716-725`) +Codex user turns are `kind: 'prompt'` by construction (`usage-parsers.mjs:486-492`) — rollouts only record real prompts as `user_message` events (§1.2). Coverage: `tests/kit/usage-index.test.mjs` — "user-role turns carry a kind" @@ -211,31 +211,31 @@ and image-only pastes get the right kind" (the two edges). ## 4. The `readSession` pipeline — how one session becomes a payload -`readSession(id, opts)` (`usage-index.mjs:1573-1629`) is the only way +`readSession(id, opts)` (`usage-index.mjs:715-770`) is the only way transcript content leaves the module, and every step is a gate: ### 4.1 Locate, contain, bound 1. **Id grammar before any filesystem access** — `VALID_ID` (`/^[A-Za-z0-9._-]{1,128}$/`, `usage-index.mjs:95`) rejects traversal - shapes with `ERR_INVALID_SESSION_ID` (`usage-index.mjs:1568-1572`). -2. **Locate by id** across both roots (`locate`, `usage-index.mjs:1578`), + shapes with `ERR_INVALID_SESSION_ID` (`usage-index.mjs:657-662`). +2. **Locate by id** across both roots (`locate`, `usage-index.mjs:667`), consulting the scan cache when present but never requiring it — `readSession` works with no prior `buildIndex`. -3. **Realpath containment** (`usage-index.mjs:1587-1601`) — the resolved file +3. **Realpath containment** (`usage-index.mjs:736-748`) — the resolved file must live under a transcript root *after* `realpathSync` collapses symlinks; a symlink planted inside a root pointing at `/etc/anything` passes a lexical `startsWith` but fails this. Roots are realpath'd too so a symlinked dotfiles setup still works. -4. **Size cap** — `MAX_SESSION_BYTES` (64 MB, `usage-index.mjs:90`): a +4. **Size cap** — `MAX_SESSION_BYTES` (64 MB, `usage-index.mjs:102`): a transcript is read whole and JSON-expands ~5×, so an unbounded read is a memory-amplification primitive. Oversized reads as unavailable, not risky. ### 4.2 Parse and price The file is parsed with `withTurns: true` by the provider's parser -(`usage-index.mjs:1614-1618`), and `meta` is assembled -(`usage-index.mjs:1624-1653`) with the same fields the Sessions view rows +(`usage-index.mjs:762-766`), and `meta` is assembled +(`usage-aggregate.mjs:477-495`) with the same fields the Sessions view rows carry — `prompts`, `responses`, `exceptions`, `sidechain`, `threadSource`, `models`, `tools`, `skill`/`plugin`, worktree — plus a `cost` priced from the same per-model usage rows `aggregate()` uses. @@ -252,11 +252,11 @@ never renames a retained session model, changes historical token pricing, or rew ### 4.3 Mask, then truncate — both marked, differently -Every turn body is passed through `maskSecrets` (`usage-index.mjs:208` — the +Every turn body is passed through `maskSecrets` (`usage-aggregate.mjs:133-138` — the 23 secret shapes) **server-side, before serialization**, then length-capped at `MAX_TURN_CHARS` (40,000, -`usage-index.mjs:89`) with the marker appended -(`usage-index.mjs:1724-1734`). Two invariants: +`usage-aggregate.mjs:63`) with the marker appended +(`usage-aggregate.mjs:502-511`). Two invariants: * **Presence is the signal.** `truncated`/`originalChars` are emitted only when the slice fired, so a complete turn cannot be misread as abridged. @@ -404,7 +404,7 @@ was wrong before, for the curious. `isHumanPrompt` once counted `harness-output` envelopes as human prompts — 32 claimed vs 20 real on the reference session. Cached session records carried the inflated counts, hence the wholesale `SCHEMA_VERSION` 5 cache - invalidation (`usage-index.mjs:48-51`). + invalidation (`usage-index.mjs:66-69`). * **Session expander fields shipped but unrendered.** The per-session fields §6.1's expander now renders (classification `basis` + confidence, the token split, flags) once travelled on the wire and rendered nowhere. @@ -412,7 +412,7 @@ was wrong before, for the curious. assembled `meta` left `cost` undefined, and `fmtUsd(undefined)` renders the truthy string `"$0.00"` — a fixed-looking zero on a panel whose whole subject is cost. `meta.cost` is now priced via `sessionCost()` from the - same per-model usage rows `aggregate()` uses (`usage-index.mjs:1696`). + same per-model usage rows `aggregate()` uses (`usage-aggregate.mjs:494`). * **Aggregate-side incidents** (the v4/v5 cache bumps, the Codex parsing defects) are recorded in `USAGE-SCORECARD-METRICS.md` Appendix A. diff --git a/docs/USAGE-SCORECARD-METRICS.md b/docs/USAGE-SCORECARD-METRICS.md index 698aee8..6023744 100644 --- a/docs/USAGE-SCORECARD-METRICS.md +++ b/docs/USAGE-SCORECARD-METRICS.md @@ -154,14 +154,14 @@ responses = Σ over included sessions of session.responses **Source:** - Filter: a parsed record with zero assistant turns is dropped entirely — "no - assistant turn → not a session" (`usage-index.mjs:1070`) — and a record whose + assistant turn → not a session" (`usage-aggregate.mjs:307`) — and a record whose last activity falls outside the requested window is dropped too - (`usage-index.mjs:1124`). + (`usage-aggregate.mjs:308`). - `responses` accumulation: Claude increments per assistant message -(`usage-index.mjs:559-563`); Codex increments per `agent_message` event -(`usage-index.mjs:727-731`). +(`usage-parsers.mjs:328-333`); Codex increments per `agent_message` event +(`usage-parsers.mjs:494-500`). - Totals: `totals.responses += s.responses` per included session -(`usage-index.mjs:1175`). +(`usage-aggregate.mjs:358`). - Render: `kpi("sessions", fmtNum(t.sessions), fmtNum(t.responses)+" assistant turns", "")` (`dashboard/client.mjs`). @@ -239,7 +239,7 @@ already in effect on the given day, comparing ISO date strings lexicographically so no `Date` parsing is involved and the module stays clock-free. -`aggregate()` passes each usage row's own `day` (`usage-index.mjs:1129-1130`), which +`aggregate()` passes each usage row's own `day` (`usage-aggregate.mjs:228-229`), which it already has because rows are keyed by `(day, model)`. **This is the whole point:** tokens metered in August must still read as August's rate when the panel is opened in December. Pricing by *today's* date instead would restate a @@ -321,9 +321,9 @@ tokens = input + output + cacheRead + cacheWrite (summed across all rows in wi ``` **Source:** `t.tokens` from `totals`, accumulated per row at -`usage-index.mjs:1108` (`rowTokens = row.input + row.output + row.cacheRead + +`usage-aggregate.mjs:236` (`rowTokens = row.input + row.output + row.cacheRead + row.cacheWrite`) and rolled into `totals.tokens` via `addTo` -(`usage-index.mjs:1080-1086`). Rendered with `fmtTok()` +(`usage-aggregate.mjs:201-208`). Rendered with `fmtTok()` (`dashboard/client.mjs`): `≥1e9` → `"X.XB"`, `≥1e6` → `"X.XM"`, `≥1e3` → `"X.XK"`, else the rounded integer. @@ -335,9 +335,9 @@ numbers as percentages of `t.tokens` (`dashboard/client.mjs`, **What "input" excludes.** For both providers, the `input` counter recorded per row is **gross input minus cached input** — Claude's parser reads `cache_read_input_tokens` and `cache_creation_input_tokens` as separate fields -the provider already reports separately (`telemetry-records.mjs:209-212`); Codex's +the provider already reports separately (`telemetry-records.mjs:216-224`); Codex's parser subtracts `cached_input_tokens` from `input_tokens` explicitly -(`usage-index.mjs:741-750`, `input: Math.max(0, gross - cacheRead)`) because +(`usage-parsers.mjs:544-561`, `input: Math.max(0, gross - cacheRead)`) because Codex's own `input_tokens` field **includes** cached tokens and would double-count them against the separately-reported `cacheRead` figure if left as-is. This is asserted by test: @@ -422,24 +422,24 @@ session data, and each needs its own fix: human, or genuinely idle) donates its *entire* idle stretch to the span, even though no work happened during it. Fix: split each session into active sub-intervals wherever the gap between two consecutive timestamps - exceeds `IDLE_GAP_MS` (15 minutes, `usage-index.mjs:80`), then union + exceeds `IDLE_GAP_MS` (15 minutes, `usage-parsers.mjs:21`), then union *those* sub-intervals — this is `engagedSeconds`. **Source:** -- `mergeIntervals()` (`usage-index.mjs:89-114`) — the pure union primitive, +- `mergeIntervals()` (`usage-aggregate.mjs:30-55`) — the pure union primitive, sorts intervals and merges any two that overlap **or exactly touch** - (`s <= curEnd`, `usage-index.mjs:133`), returning total covered seconds + (`s <= curEnd`, `usage-aggregate.mjs:46`), returning total covered seconds rounded to the nearest second. -- `activeIntervals()` (`usage-index.mjs:427-438`) — splits one session's +- `activeIntervals()` (`usage-parsers.mjs:206-217`) — splits one session's sorted timestamp list into sub-intervals wherever a gap exceeds `IDLE_GAP_MS`; "a run of one timestamp yields a zero-length interval and so - contributes nothing" (comment, `usage-index.mjs:427-431`). + contributes nothing" (comment, `usage-parsers.mjs:200-204`). - Aggregation: `totals.engagedSeconds = mergeIntervals(sessions.flatMap(s => - s._active))` (`usage-index.mjs:1219`); `totals.spanUnionSeconds = - mergeIntervals(sessions.map(s => s._span))` (`usage-index.mjs:1218`); + s._active))` (`usage-aggregate.mjs:423`); `totals.spanUnionSeconds = + mergeIntervals(sessions.map(s => s._span))` (`usage-aggregate.mjs:422`); `totals.spanMinutes` is a running sum of `s._span[1] - s._span[0]` across - the loop (`usage-index.mjs:1179`, finalized `usage-index.mjs:1217`). + the loop (`usage-aggregate.mjs:362`, finalized `usage-aggregate.mjs:421`). - Render: `fmtHours()` (`dashboard/client.mjs`, `≥10h` rounds to the nearest hour, else one decimal place) and `fmtMins()` (`dashboard/client.mjs`, `≥60min` rounds to hours, else whole @@ -490,12 +490,12 @@ byDay[day].sessionsActive = count of distinct sessions with any usage row that d **Source:** the day key is the row's own `row.day`, computed once at parse time as **local calendar day**, not UTC -(`usage-index.mjs:589`/`usage-index.mjs:745` call `localDay(at)`) — so a +(`usage-parsers.mjs:360`/`usage-parsers.mjs:550` call `localDay(at)`) — so a session that runs from 23:58 local to 00:05 local is billed to the day its *first* row landed on (test: `tests/kit/usage-index.test.mjs:634`, "a session that opens before midnight is counted on its first billed day"). Accumulation: -`byDay[row.day].cost += rowCost` (`usage-index.mjs:1151`). Bar height: +`byDay[row.day].cost += rowCost` (`usage-aggregate.mjs:239`). Bar height: `h = maxDay ? max(2, cost/maxDay*100) : 2` (`dashboard/client.mjs`) — every non-empty day gets a visually nonzero bar (floor of 2%), so a very cheap day is never rendered as invisible. @@ -522,11 +522,11 @@ renders "no sessions in window" instead of zeroed figures (`dashboard/client.mjs`). **Formula:** identical aggregation to every other bucket -(`byProvider[s.provider]`, populated via `addTo()`, `usage-index.mjs:1036-1045`, - called once per session at `usage-index.mjs:1097`), keyed by the literal string +(`byProvider[s.provider]`, populated via `addTo()`, `usage-aggregate.mjs:201-209`, + called once per session at `usage-aggregate.mjs:365`), keyed by the literal string `"claude"` or `"codex"` assigned at parse time (`blankSession(id, 'claude')` / `blankSession(id, 'codex')`, -`usage-index.mjs:398-408`, `parseClaude`/`parseCodex` entry points). +`usage-parsers.mjs:180-191`, `parseClaude`/`parseCodex` entry points). **Why this pairing is the one under the most scrutiny.** Both providers' tokens are summed into the *same* `tokens`/`cost` fields using the *same* @@ -562,9 +562,9 @@ punchcard[dow + "-" + hour] += 1 per assistant/agent_message response, at its ``` **Source:** incremented once per Claude assistant turn -(`usage-index.mjs:559-563`, keyed by `punchKey(at)`) and once per Codex -`agent_message` (`usage-index.mjs:727-731`), merged into the window-level -`punchcard` object per session (`usage-index.mjs:1184`). Cell intensity is +(`usage-parsers.mjs:328-333`, keyed by `punchKey(at)`) and once per Codex +`agent_message` (`usage-parsers.mjs:494-500`), merged into the window-level +`punchcard` object per session (`usage-aggregate.mjs:370`). Cell intensity is linear against the single busiest cell in the window: `v = pcMax ? n/pcMax : 0` (`dashboard/client.mjs`) — this is a **relative**, not absolute, scale, so the heatmap's brightest cell is always @@ -597,9 +597,9 @@ byModel[model].sessions = count of DISTINCT sessions whose s.models includes th ``` **Source:** cost/tokens/responses accumulate inside the usage-row loop -(`usage-index.mjs:1089-1116`); the `sessions` count is deliberately computed +(`usage-aggregate.mjs:227-246`); the `sessions` count is deliberately computed **separately**, once per session over its `s.models` array -(`usage-index.mjs:1190-1194`) rather than inside the cost loop, precisely +(`usage-aggregate.mjs:321-328`) rather than inside the cost loop, precisely **so that a model can appear in `byModel` — with a nonzero session count — even in a session that contributed zero cost/tokens/responses for that model.** This is not an edge case invented for this document: it is the @@ -609,11 +609,11 @@ excluded subagent-replay session still shows up as "used," at zero cost, rather than vanishing. `byModel[...].responses` is populated from `row.responses` -(`usage-index.mjs:1154`), which in turn comes from the `responses` field +(`usage-aggregate.mjs:243`), which in turn comes from the `responses` field passed into `addUsage()` at the call site — `1` per Claude assistant turn -(`usage-index.mjs:596-608`), or `rec.responses` (the session's whole response +(`usage-parsers.mjs:357-360`), or `rec.responses` (the session's whole response count) once per Codex session, passed at the single point Codex calls -`addUsage` (`usage-index.mjs:745-804`). +`addUsage` (`usage-parsers.mjs:550-556`). **Render:** `bar(name, fmtUsd(cost), fmtTok(tokens)+" · "+fmtNum(responses)+" resp", pct(cost, topModelCost), false)` (`dashboard/client.mjs`), @@ -629,14 +629,14 @@ split `server_error` 27, `authentication_failed` 3, `rate_limit` 3 — three distinct underlying causes, one placeholder shape). The parser branches on `isApiErrorMessage === true` -(`usage-index.mjs:522-531`): the turn still increments `rec.responses` -and the punchcard (`usage-index.mjs:559-563`) — it *is* real engaged +(`usage-parsers.mjs:345-353`): the turn still increments `rec.responses` +and the punchcard (`usage-parsers.mjs:328-333`) — it *is* real engaged time, someone was genuinely waiting on it — but it is never pushed into `rec.models` and `addUsage()` is never called for it, so it can no longer create a `byModel` row of any kind. It increments a separate -`rec.exceptions` counter instead (`usage-index.mjs:523`), rolled up into -`totals.exceptions` (`usage-index.mjs:1177`) and surfaced per-session -(`usage-index.mjs:973-987`, alongside the existing `sidechain`/`threadSource` +`rec.exceptions` counter instead (`usage-parsers.mjs:346`), rolled up into +`totals.exceptions` (`usage-aggregate.mjs:358`) and surfaced per-session +(`usage-aggregate.mjs:278-279`, alongside the existing `sidechain`/`threadSource` flags — inspectable in the Sessions tab, never hidden). When `totals.exceptions > 0`, the panel header shows a small `"· N dropped/errored turns excluded"` note (`dashboard/client.mjs`); @@ -920,7 +920,7 @@ both credential-free for ak: `windowDurationMins: 10080` (the weekly). Windows are therefore keyed and labelled by duration (`windowLabel`, `quota.mjs:44`), never by slot name. The same rule applies to the historical snapshots parsed out of rollouts: the -normalizer at `usage-index.mjs:686-703` keeps a flat `windows` list keyed by +normalizer at `usage-parsers.mjs:446-458` keeps a flat `windows` list keyed by `window_minutes`. **Freshness is part of the number.** Both sides carry `fetchedAt`; the view @@ -945,13 +945,13 @@ Codex ≥0.140 maintains its own SQLite thread ledger (`~/.codex/state_N.sqlite` — the `N` is a migration generation, so `codexStateDb` (`codex-state.mjs:30`) globs and takes the newest). `readCodexState` (`:49`) reads per-thread `thread_source` (`user` vs `subagent`) plus `thread_spawn_edges`, and -`applyCodexLedger` (`usage-index.mjs:1532-1543`) overlays that onto parsed +`applyCodexLedger` (`usage-aggregate.mjs:454-466`) overlays that onto parsed sessions: a ledger-identified subagent has its token usage stripped — its rollout replays the parent's entire token history (ccusage/ccusage#950 measured up to 91× inflation) — while the session record stays visible. The rollout's own `session_meta.thread_source` sniff remains as the fallback when the ledger is absent or migrated beyond recognition. Codex sessions also carry -`reasoningOutput` (`usage-index.mjs:755`) — reasoning tokens are a **subset** +`reasoningOutput` (`usage-parsers.mjs:560`) — reasoning tokens are a **subset** of output tokens and are annotation only, never added to any sum. ## 14. Known limitations, restated as a single checklist @@ -996,14 +996,14 @@ commit `540be18` on this branch. `parseCodex`'s single `addUsage()` call never included a `responses` field — Claude's parser passes `responses: 1` per assistant turn -(`usage-index.mjs:598`, the current equivalent), but Codex's call +(`usage-parsers.mjs:360`, the current equivalent), but Codex's call passed no such field at all. Because `byModel[model].responses` is summed -directly from each usage row's `responses` field (`usage-index.mjs:1154`, +directly from each usage row's `responses` field (`usage-aggregate.mjs:243`, `m.responses += row.responses`), **every** Codex model in §10's "Models in Play" list displayed `0 resp` regardless of real token/cost volume or actual `agent_message` count. **Fix:** `parseCodex` now passes `responses: rec.responses` (the session's own tallied response count, -`usage-index.mjs:765`) on its `addUsage()` call. +`usage-parsers.mjs:555`) on its `addUsage()` call. #### Bug B — subagent thread-replay could double-bill tokens @@ -1023,13 +1023,13 @@ at face value (correctly avoiding the separate naive-summing bug **[C5]** documents, since it already used last-event-only logic — see §4's worked example) but performed **no de-duplication** against a parent session a subagent file might be replaying. **Fix:** the parser now reads -`session_meta.thread_source` (`telemetry-records.mjs:117-122`, confirmed as a real +`session_meta.thread_source` (`telemetry-records.mjs:101-110`, confirmed as a real Codex rollout field by **[C7]**) and skips the `addUsage()` call entirely -when its value is `'subagent'` (`usage-index.mjs:741`, guard condition +when its value is `'subagent'` (`usage-parsers.mjs:546`, guard condition `rec.threadSource !== 'subagent'`). The session record itself is **not** dropped — it remains visible in the Sessions tab with `threadSource` surfaced (mirroring the existing `sidechain` flag Claude sessions already -carry, `usage-index.mjs:347`), so a maintainer auditing the raw data can +carry, `usage-parsers.mjs:184`), so a maintainer auditing the raw data can still see it; it simply contributes zero tokens/cost, exactly as intended by the "models still shows up in §10's list, with zero cost" mechanism §10 describes. diff --git a/docs/adr/0017-opencode-host.md b/docs/adr/0017-opencode-host.md index 8b0e147..61e6eb9 100644 --- a/docs/adr/0017-opencode-host.md +++ b/docs/adr/0017-opencode-host.md @@ -15,7 +15,10 @@ lifecycle plugin, and managed host assets. The 2026-08-15 amendment keeps Ruflo and Agentic QE connected in stock OpenCode while blacklisting their eager tool catalogues from model requests and projecting a compact, lazy Agentic Kit gateway instead. The 2026-08-17 amendment adds a - bounded cross-assistant-message repeated-tool guard to the managed lifecycle plugin. + bounded cross-assistant-message repeated-tool guard to the managed lifecycle plugin. The + 2026-08-26 complexity-program wave 2 split the owner module's implementation across + `opencode-core.mjs`/`opencode-agents.mjs`/`opencode-artifacts.mjs`/`opencode-lifecycle.mjs` + (ADR-0037's file-size gate); `opencode.mjs` is now a re-export barrel, not a behavior change. - **Deciders:** agentic-kit maintainers > **GA amendment:** OpenCode remains opt-in, non-primary, and outside AQE inference-provider @@ -150,7 +153,13 @@ routes; it does not make OpenCode primary or an AQE provider. ### 2. One owner module: `src/lib/opencode.mjs` Every ak-managed byte on opencode's surfaces lives behind one module, following the -`settings.mjs`/`mcp.mjs` contracts (backup-first, merge-not-clobber, idempotent): +`settings.mjs`/`mcp.mjs` contracts (backup-first, merge-not-clobber, idempotent). +`src/lib/opencode.mjs` is the single import path every consumer uses; since 2026-08-26 its +implementation is split by size (not by ownership) across `opencode-core.mjs` (config-wiring), +`opencode-agents.mjs` (catalog + agent conversion/sync/status), `opencode-artifacts.mjs` +(plugin + skill deployment/teardown), and `opencode-lifecycle.mjs` (the stack composition below) +— `opencode.mjs` itself re-exports their combined public surface, so this remains one owner +module from every caller's point of view: - **`opencode.json` wiring** (`applyOpencode`): `mcp.claude-flow` (command `claude-flow-mcp` when the dedicated stdio bin is present — it answers `initialize` @@ -346,8 +355,10 @@ discrepancy until reconciled. - ADR-0016 defines the registry, lifecycle, ownership, and normalized-fact contracts implemented here. ADR-0018 records the generalized execution contract implemented by #82. -- `src/lib/opencode.mjs` (the owner module: `opencodeStack`, `retireOpencode`, - `reconcileOpencodeGuidance`), `src/lib/hosts.mjs` (adapter), +- `src/lib/opencode.mjs` (the owner module's public entry point, re-exporting + `opencodeStack`/`retireOpencode`/`reconcileOpencodeGuidance` from + `src/lib/opencode-lifecycle.mjs`, plus `src/lib/opencode-core.mjs`, + `src/lib/opencode-agents.mjs`, and `src/lib/opencode-artifacts.mjs`), `src/lib/hosts.mjs` (adapter), `src/lib/providers.mjs` (registry-derived managed host projection, `--opencode` flag handling, `hostAuthState` home seam), `src/lib/blocks.mjs` (`agents-opencode` target, new registry rows), diff --git a/docs/adr/0037-complexity-program-structural-patterns.md b/docs/adr/0037-complexity-program-structural-patterns.md index 89c2117..6fb8b31 100644 --- a/docs/adr/0037-complexity-program-structural-patterns.md +++ b/docs/adr/0037-complexity-program-structural-patterns.md @@ -74,8 +74,35 @@ repository's sanctioned structures: functions became visible to ESLint for the first time — the denominator became honest; per-function severity dropped sharply. - Known residual backlog (deliberately not part of this program): `uninstall run()` - (CC 100), `opencode.mjs` receipt-reconciliation family (`applyOpencode` 71 — the - audit's `reconcileOwnedMap` recommendation), `model-inventory` discovery and - `footprint`/`adapters` functions in the 50–65 band, and the newly visible client - functions (`system-projects` 88, `overview` 63). `providers.mjs` sits just over - the 1,000-line warning; a future `aqe-router.mjs` split would clear it. + (CC 100), `model-inventory` discovery and `footprint`/`adapters` functions in the + 50–65 band, and the newly visible client functions (`system-projects` 88, + `overview` 63). `providers.mjs` sits just over the 1,000-line warning; a future + `aqe-router.mjs` split would clear it. +- **Wave 2 update (2026-08-26):** the `opencode.mjs` receipt-reconciliation family + named above is done — `reconcileOwnedMap` (+ a `reconcileFamilyPermissions` + family-atomicity wrapper for the permission block) now backs `applyOpencode` + (71→21), and every other named function over CC 25 in `opencode.mjs` + (`normalizeManaged` 32, `opencodeConverged` 28, `undoOpencode` 26, `opencodeStack` + 44, `syncAgents` 50, `agentsStatus` 46, `removeArtifacts` 26), in + `src/lib/execution/opencode.mjs` (`terminalResult` 30, `launch` 36), and in the + emitted `src/templates/opencode-ruflo-gateway.js` template (`config` 28) is under + the gate. `opencode.mjs` (1,573 lines) was also split into five files, each under + 1,000 lines (`opencode.mjs` is now a re-export barrel; see ADR-0017 §2). +- **Wave 2 closure (2026-08-27):** the whole residual backlog above is cleared — + seven parallel tracks brought `uninstall run()` (100→5), the adapters family + (conformance 60→19, admission 55, four validators 45–49 → all <25), footprint + (`collectProjects` 59→16, `storage.mjs` split three ways), model-inventory + (`discoverOllamaApi` 65→13), the telemetry residuals (`adaptCodexLedger` 65→20, + `parseCodex` 53→11, `usage-index.mjs` split into index-I/O plus + `usage-parsers.mjs` and `usage-aggregate.mjs`, removing a latent circular + import), the AQE-router + machinery out of `providers.mjs` into `aqe-router.mjs`, and every over-25 + dashboard client function (worst `renderSysStorage` 88 → an orchestrator of ≤22 + helpers) under the gate. One real defect was found and fixed along the way — a + literal NUL byte in the Models "Used for" sort key (`mliRouteValue`), a live sort + bug. **Repo-wide there is now no function over CC 50**, and that ceiling is + enforced as an ERROR (`pnpm run lint:cc`, wired into `check` and CI's quality + job); the warn-25 tier stays advisory with ~30 functions in the 26–49 band, led + by `createLiveEvent` (49), which stays as the deliberate validation-boundary + exemption. Remaining advisory residuals live in the lint output, not in this + document. diff --git a/docs/ddd/machine-footprint.md b/docs/ddd/machine-footprint.md index 7de1bbb..8a0cc7a 100644 --- a/docs/ddd/machine-footprint.md +++ b/docs/ddd/machine-footprint.md @@ -68,7 +68,7 @@ read not on this list is a defect, and adding one is an amendment to this docume |------|-----------|---------------|---------------------| | Directory entries and `lstat` | `walk.mjs`, every collector | name, kind, size, mtime, block count | anything inside a file | | `.git/config` | `projects.mjs` | the origin remote URL | every other config key | -| `.git/worktrees//gitdir` | `storage.mjs` | one filesystem path, bounded to 4 KB | — | +| `.git/worktrees//gitdir` | `storage-reclaim-detectors.mjs` (re-exported from `storage.mjs`) | one filesystem path, bounded to 4 KB | — | | A transcript's **head** | `project-sources.mjs` | the session's `cwd` **field** | every message, prompt, tool call, tool result and model output in the file | | OpenCode's session store | `project-sources.mjs` | the `directory` column, read-only | every other column, and every message row | | A project's own manifests | `stack-detect.mjs` | dependency **keys** (and, for `path:`/`workspace:` entries, enough of the value to reject them) | manifest values, scripts, and anything executable | diff --git a/package.json b/package.json index 487ecfc..0237363 100644 --- a/package.json +++ b/package.json @@ -44,6 +44,7 @@ "test:qe-court-live": "node --test tests/live/qe-court-participant-transport.test.mjs", "typecheck": "tsc -p tsconfig.json", "lint": "eslint .", + "lint:cc": "eslint src bin --rule 'complexity: [2, 50]'", "lint:fix": "eslint . --fix", "lint:md": "markdownlint-cli2", "lint:md:fix": "markdownlint-cli2 --fix", @@ -54,7 +55,7 @@ "audit:fix": "pnpm audit --fix=override", "outdated": "pnpm outdated", "upgrade": "pnpm up", - "check": "pnpm run typecheck && pnpm run lint && pnpm run lint:md && pnpm run build && pnpm test" + "check": "pnpm run typecheck && pnpm run lint && pnpm run lint:cc && pnpm run lint:md && pnpm run build && pnpm test" }, "devDependencies": { "@eslint/js": "^10.0.1", diff --git a/src/commands/models.mjs b/src/commands/models.mjs index b6a73d4..25369a2 100644 --- a/src/commands/models.mjs +++ b/src/commands/models.mjs @@ -89,131 +89,146 @@ function noSnapshot(flags, cacheFile) { return 0; } -/** @param {{flags: Record, positionals: string[], deps?: Record}} input */ -export async function run({ flags, positionals, deps = {} }) { - const action = positionals[0] ?? 'status'; - const cacheFile = deps.cacheFile ?? modelInventoryPath(); - const readStore = deps.readStore ?? readModelStore; - const append = deps.append ?? appendModelSnapshot; - const collect = deps.collect ?? collectModelSnapshot; - const loadConfig = deps.loadConfig ?? loadKitConfig; - const cfg = loadConfig(); - - if (action === 'refresh') { - const owners = selectedOwners(flags, cfg); - const onlineContact = Boolean(flags.online && owners.includes('opencode')); - if (flags['dry-run']) { - const result = { dryRun: true, action, owners, online: onlineContact, - onlineRequested: flags.online, network: false, writes: false, cacheFile }; - if (flags.json) printJson(result); - else { - heading('ak models — refresh plan (dry-run)'); - info(`Would inspect: ${owners.join(', ')}.`); - info(onlineContact ? 'OpenCode catalog refresh would be permitted.' - : 'No online catalog refresh would be contacted.'); - info(dim('No source was contacted and no file was written.')); - } - return 0; - } - const aqeConfig = (deps.readJson ?? readJson)((deps.aqeFile ?? aqeRouterFile)(process.cwd())); - const snapshot = await collect({ - config: cfg, aqeConfig, rufloConfig: cfg, scope: { project: process.cwd() }, - discoveryOptions: { owners, online: flags.online, cwd: process.cwd() }, - }); - const store = append(snapshot, { file: cacheFile }); - const result = { status: 'refreshed', cacheFile, contacts: owners, online: onlineContact, - onlineRequested: flags.online, - snapshot: createModelReadModel(snapshot), retainedSnapshots: store.snapshots.length }; +/** `ak models refresh` — the only action that contacts sources / writes the + * cache; every other action is a pure read over the existing snapshot store. */ +async function runRefresh(ctx) { + const { flags, cfg, cacheFile, append, collect, deps } = ctx; + const owners = selectedOwners(flags, cfg); + const onlineContact = Boolean(flags.online && owners.includes('opencode')); + if (flags['dry-run']) { + const result = { dryRun: true, action: 'refresh', owners, online: onlineContact, + onlineRequested: flags.online, network: false, writes: false, cacheFile }; if (flags.json) printJson(result); else { - const health = summarizeModelHealth(snapshot); - ok(`Model inventory refreshed: ${snapshot.models.length} model(s) · ${snapshot.sources.length} source(s)`); - info(health.message); - info(dim(`private cache: ${cacheFile}`)); + heading('ak models — refresh plan (dry-run)'); + info(`Would inspect: ${owners.join(', ')}.`); + info(onlineContact ? 'OpenCode catalog refresh would be permitted.' + : 'No online catalog refresh would be contacted.'); + info(dim('No source was contacted and no file was written.')); } return 0; } + const aqeConfig = (deps.readJson ?? readJson)((deps.aqeFile ?? aqeRouterFile)(process.cwd())); + const snapshot = await collect({ + config: cfg, aqeConfig, rufloConfig: cfg, scope: { project: process.cwd() }, + discoveryOptions: { owners, online: flags.online, cwd: process.cwd() }, + }); + const store = append(snapshot, { file: cacheFile }); + const result = { status: 'refreshed', cacheFile, contacts: owners, online: onlineContact, + onlineRequested: flags.online, + snapshot: createModelReadModel(snapshot), retainedSnapshots: store.snapshots.length }; + if (flags.json) printJson(result); + else { + const health = summarizeModelHealth(snapshot); + ok(`Model inventory refreshed: ${snapshot.models.length} model(s) · ${snapshot.sources.length} source(s)`); + info(health.message); + info(dim(`private cache: ${cacheFile}`)); + } + return 0; +} - const store = readStore({ file: cacheFile }); - const latest = latestSnapshot(store); - if (!latest) return noSnapshot(flags, cacheFile); - - if (action === 'status') { - if (flags.host && !ALL_OWNERS.includes(flags.host)) { - warn(`unsupported model host: ${flags.host}`); - return 2; - } - const snapshot = visibleSnapshot(latest, flags.host); - const since = flags.since ? Date.parse(flags.since) : null; - const history = store.snapshots.filter((entry) => entry.scope.fingerprint === latest.scope.fingerprint - && (!Number.isFinite(since) || Date.parse(entry.capturedAt) >= since)); - const result = { status: 'cached', cacheFile, health: summarizeModelHealth(snapshot), - inventory: createModelReadModel(snapshot), history: history.map(({ snapshotId, capturedAt }) => ({ snapshotId, capturedAt })) }; - if (flags.json) printJson(result); - else { - heading('ak models — offline lifecycle inventory'); - const health = result.health; - (health.level === 'ok' ? ok : warn)(health.message); - for (const source of snapshot.sources) info(`${source.id}: ${source.status} · ${source.capturedAt}`); - info(dim(`snapshot ${snapshot.snapshotId} · ${history.length} retained same-scope capture(s)`)); - } - return 0; +function runStatus(ctx) { + const { flags, cacheFile, store, latest } = ctx; + if (flags.host && !ALL_OWNERS.includes(flags.host)) { + warn(`unsupported model host: ${flags.host}`); + return 2; + } + const snapshot = visibleSnapshot(latest, flags.host); + const since = flags.since ? Date.parse(flags.since) : null; + const history = store.snapshots.filter((entry) => entry.scope.fingerprint === latest.scope.fingerprint + && (!Number.isFinite(since) || Date.parse(entry.capturedAt) >= since)); + const result = { status: 'cached', cacheFile, health: summarizeModelHealth(snapshot), + inventory: createModelReadModel(snapshot), history: history.map(({ snapshotId, capturedAt }) => ({ snapshotId, capturedAt })) }; + if (flags.json) printJson(result); + else { + heading('ak models — offline lifecycle inventory'); + const health = result.health; + (health.level === 'ok' ? ok : warn)(health.message); + for (const source of snapshot.sources) info(`${source.id}: ${source.status} · ${source.capturedAt}`); + info(dim(`snapshot ${snapshot.snapshotId} · ${history.length} retained same-scope capture(s)`)); } + return 0; +} - if (action === 'diff') { - const { before, after, fromId, toId } = selectedPair(store, positionals, flags); - if (!before || !after) { - const missing = !before ? fromId ?? 'same-scope baseline' : toId ?? 'latest snapshot'; - if (flags.json) printJson({ comparable: false, reason: 'snapshot-not-found', missing }); - else warn(`Cannot diff: ${missing} not found.`); - return 1; - } - const result = diffSnapshotHistory(before, after, store.snapshots); - if (flags.json) printJson(result); - else { - heading(`ak models diff — ${before.snapshotId} → ${after.snapshotId}`); - if (!result.comparable) warn(result.diagnostics.join('; ')); - else if (!result.changes.length) ok('No model lifecycle changes.'); - else for (const change of result.changes) info(`${change.kind}: ${change.subject}${change.provisional ? ' (provisional)' : ''}`); - for (const message of result.diagnostics) info(dim(message)); - } - return result.comparable ? 0 : 1; +function runDiff(ctx) { + const { store, positionals, flags } = ctx; + const { before, after, fromId, toId } = selectedPair(store, positionals, flags); + if (!before || !after) { + const missing = !before ? fromId ?? 'same-scope baseline' : toId ?? 'latest snapshot'; + if (flags.json) printJson({ comparable: false, reason: 'snapshot-not-found', missing }); + else warn(`Cannot diff: ${missing} not found.`); + return 1; + } + const result = diffSnapshotHistory(before, after, store.snapshots); + if (flags.json) printJson(result); + else { + heading(`ak models diff — ${before.snapshotId} → ${after.snapshotId}`); + if (!result.comparable) warn(result.diagnostics.join('; ')); + else if (!result.changes.length) ok('No model lifecycle changes.'); + else for (const change of result.changes) info(`${change.kind}: ${change.subject}${change.provisional ? ' (provisional)' : ''}`); + for (const message of result.diagnostics) info(dim(message)); } + return result.comparable ? 0 : 1; +} - if (action === 'explain') { - const selector = positionals[1] ?? flags.to; - if (!selector) { warn('usage: ak models explain HOST:MODEL'); return 2; } - const result = explainModel(latest, selector); - if (flags.json) printJson(result); - else if (!result.found) warn(`Model not found: ${selector}`); - else { - heading(`ak models explain — ${selector}`); - for (const match of result.matches) { - info(`${match.key.host}${match.key.provider ? `/${match.key.provider}` : ''}: ${match.key.modelId}`); - for (const [name, dimension] of Object.entries(match.dimensions)) info(` ${name}: ${dimension.value ?? 'unknown'}`); - info(` lifecycle: ${match.lifecycle.state}${match.lifecycle.replacement ? ` → ${match.lifecycle.replacement}` : ''}`); - } +function runExplain(ctx) { + const { positionals, flags, latest } = ctx; + const selector = positionals[1] ?? flags.to; + if (!selector) { warn('usage: ak models explain HOST:MODEL'); return 2; } + const result = explainModel(latest, selector); + if (flags.json) printJson(result); + else if (!result.found) warn(`Model not found: ${selector}`); + else { + heading(`ak models explain — ${selector}`); + for (const match of result.matches) { + info(`${match.key.host}${match.key.provider ? `/${match.key.provider}` : ''}: ${match.key.modelId}`); + for (const [name, dimension] of Object.entries(match.dimensions)) info(` ${name}: ${dimension.value ?? 'unknown'}`); + info(` lifecycle: ${match.lifecycle.state}${match.lifecycle.replacement ? ` → ${match.lifecycle.replacement}` : ''}`); } - return result.found ? 0 : 1; } + return result.found ? 0 : 1; +} - if (action === 'plan') { - const activity = flags.activity; - const to = flags.to ?? positionals[1]; - if (!activity || !to) { warn('usage: ak models plan --activity ACTIVITY [--from HOST:MODEL] --to HOST:MODEL'); return 2; } - const result = planModelChange(latest, { activity, from: flags.from, to }); - if (flags.json) printJson(result); +function runPlan(ctx) { + const { flags, positionals, latest } = ctx; + const activity = flags.activity; + const to = flags.to ?? positionals[1]; + if (!activity || !to) { warn('usage: ak models plan --activity ACTIVITY [--from HOST:MODEL] --to HOST:MODEL'); return 2; } + const result = planModelChange(latest, { activity, from: flags.from, to }); + if (flags.json) printJson(result); + else { + heading(`ak models plan — ${activity}`); + if (!result.plannable) warn(`No mechanical plan: ${result.reason ?? result.compatibility?.blockers?.join('; ')}`); else { - heading(`ak models plan — ${activity}`); - if (!result.plannable) warn(`No mechanical plan: ${result.reason ?? result.compatibility?.blockers?.join('; ')}`); - else { - ok('Mechanical compatibility is supported by current evidence. Quality equivalence remains unknown.'); - info(`Copy to apply explicitly: ${result.action.command}`); - } + ok('Mechanical compatibility is supported by current evidence. Quality equivalence remains unknown.'); + info(`Copy to apply explicitly: ${result.action.command}`); } - return result.plannable ? 0 : 1; } + return result.plannable ? 0 : 1; +} + +// Actions after 'refresh' are pure reads over the existing snapshot store — +// dispatched by name once that store/latest snapshot is in hand (below). +const READ_ACTIONS = { status: runStatus, diff: runDiff, explain: runExplain, plan: runPlan }; + +/** @param {{flags: Record, positionals: string[], deps?: Record}} input */ +export async function run({ flags, positionals, deps = {} }) { + const action = positionals[0] ?? 'status'; + const cacheFile = deps.cacheFile ?? modelInventoryPath(); + const readStore = deps.readStore ?? readModelStore; + const append = deps.append ?? appendModelSnapshot; + const collect = deps.collect ?? collectModelSnapshot; + const loadConfig = deps.loadConfig ?? loadKitConfig; + const cfg = loadConfig(); + const ctx = { flags, positionals, deps, cacheFile, readStore, append, collect, cfg }; + + if (action === 'refresh') return runRefresh(ctx); + + const store = readStore({ file: cacheFile }); + const latest = latestSnapshot(store); + if (!latest) return noSnapshot(flags, cacheFile); - warn('usage: ak models status|refresh|diff|explain|plan'); - return 2; + const handler = READ_ACTIONS[action]; + if (!handler) { warn('usage: ak models status|refresh|diff|explain|plan'); return 2; } + return handler({ ...ctx, store, latest }); } diff --git a/src/commands/uninstall.mjs b/src/commands/uninstall.mjs index 9f0de6f..b1101da 100644 --- a/src/commands/uninstall.mjs +++ b/src/commands/uninstall.mjs @@ -183,81 +183,83 @@ export async function purgeDejaVuIndex({ /** @typedef {{dejaAdapter?:any,purgeDejaVuIndex?:typeof purgeDejaVuIndex}} UninstallDeps */ -/** @param {{flags:Record,deps?:UninstallDeps}} request */ -export async function run({ flags, deps = {} }) { - const dry = flags['dry-run']; - const act = (msg, fn) => { if (dry) info(`[dry-run] ${msg}`); else { fn(); ok(msg); } }; - // Ownership markers are read ONCE up front: the purge path removes kit.json - // below, and teardown decisions (opencode undo) must still see what ak owned - // (codex-review — purge ordering must not strand managed opencode.json keys). - const cfg = loadKitConfig(); - const kitCfg = cfg; - let ownershipTeardownOk = true; - let dejaVuTeardownOk = true; - - // 0. User-scoped Codex line: only values recorded as ours are candidates. - if (kitCfg.statusline?.codex) { - if (dry) info('[dry-run] release managed Codex status line (preserving user-modified keys)'); - else { - let r; - try { r = removeCodexStatusline(kitCfg.statusline.codex.lastProjection); } - catch (error) { - warn(`Codex config was not changed; status-line ownership retained: ${error.message}`); - r = null; - } - if (r) { - kitCfg.statusline.codex = null; - if (!flags.purge) saveKitConfig(kitCfg); - ok(`Codex status-line ownership released${r.changed ? ' (unchanged managed keys removed)' : ' (user-modified keys preserved)'}`); - } - } +// ── the uninstall step registry ────────────────────────────────────────── +// Mirrors sync.mjs's SYNC_STEPS idiom (ADR-0037): every teardown phase used to +// be inlined sequentially into `run()`, with real ordering invariants (the +// CLAUDE.md/skill/opencode strips before kit.json purge reads ownership; +// deja-vu targets before its data purge before its package removal; the +// registry-driven host-lifecycle loop before kit.json is ever deleted) proven +// only by source order. Each step below is `{id, when(ctx), run(ctx)}`; +// UNINSTALL_STEPS's array order *is* the ordering invariant. `ctx` is the +// shared per-invocation context built in `run()`: {flags, dry, deps, cfg, +// act, state}. `state` carries the two cross-step signals +// (`ownershipTeardownOk`, `dejaVuTeardownOk`) later steps (the kit.json purge +// decision) still need to read. +function stepCodexStatusline(ctx) { + const { dry, cfg, flags } = ctx; + if (dry) { info('[dry-run] release managed Codex status line (preserving user-modified keys)'); return; } + let r; + try { r = removeCodexStatusline(cfg.statusline.codex.lastProjection); } + catch (error) { + warn(`Codex config was not changed; status-line ownership retained: ${error.message}`); + r = null; } + if (r) { + cfg.statusline.codex = null; + if (!flags.purge) saveKitConfig(cfg); + ok(`Codex status-line ownership released${r.changed ? ' (unchanged managed keys removed)' : ' (user-modified keys preserved)'}`); + } +} - // 1. CLAUDE.md managed blocks: every built-in slug (registry-driven, so - // non-ruflo blocks like ruvnet-brain-reference are covered), the legacy - // ruflo-* pattern as a catch-all, plus any custom slugs from kit.json. +// 1. CLAUDE.md managed blocks: every built-in slug (registry-driven, so +// non-ruflo blocks like ruvnet-brain-reference are covered), the legacy +// ruflo-* pattern as a catch-all, plus any custom slugs from kit.json. +function stepClaudeMdBlocks(ctx) { const md = paths.claudeMdPath(); - if (fs.existsSync(md)) { - let content = fs.readFileSync(md, 'utf8'); - const slugs = new Set([...content.matchAll(//g)].map((m) => m[1])); - for (const b of BUILTIN_BLOCKS) if (content.includes(BEGIN(b.slug))) slugs.add(b.slug); - for (const b of kitCfg.customBlocks) if (content.includes(BEGIN(b.slug))) slugs.add(b.slug); - if (slugs.size) { - act(`stripped ${slugs.size} managed block(s) from ~/.claude/CLAUDE.md (backup written)`, () => { - fs.copyFileSync(md, `${md}.bak.${Date.now()}`); - for (const s of slugs) content = stripBlock(content, s); - fs.writeFileSync(md, content); - }); - } - } + if (!fs.existsSync(md)) return; + let content = fs.readFileSync(md, 'utf8'); + const slugs = new Set([...content.matchAll(//g)].map((m) => m[1])); + for (const b of BUILTIN_BLOCKS) if (content.includes(BEGIN(b.slug))) slugs.add(b.slug); + for (const b of ctx.cfg.customBlocks) if (content.includes(BEGIN(b.slug))) slugs.add(b.slug); + if (!slugs.size) return; + ctx.act(`stripped ${slugs.size} managed block(s) from ~/.claude/CLAUDE.md (backup written)`, () => { + fs.copyFileSync(md, `${md}.bak.${Date.now()}`); + for (const s of slugs) content = stripBlock(content, s); + fs.writeFileSync(md, content); + }); +} - // 2. deployed skill. kit.json is purged only after all receipt-dependent - // teardown succeeds; otherwise it remains the recovery proof. +// 2. deployed skill. kit.json is purged only after all receipt-dependent +// teardown succeeds; otherwise it remains the recovery proof. +function stepSkill(ctx) { const skill = path.join(paths.claudeSkillsDir(), 'ruflo-token-audit'); - if (fs.existsSync(skill)) act('removed skill ruflo-token-audit', () => fs.rmSync(skill, { recursive: true })); + if (fs.existsSync(skill)) ctx.act('removed skill ruflo-token-audit', () => fs.rmSync(skill, { recursive: true })); +} - // 2b. opencode host footprint (when ak managed it): strip the guidance blocks - // from opencode's AGENTS.md, the opencode.json wiring, and deployed artifacts. +// 2b. opencode host footprint (when ak managed it): strip the guidance +// blocks from opencode's AGENTS.md. (The opencode.json wiring and deployed +// artifacts are handled by the registry-driven host-lifecycle loop below.) +function stepOpencodeAgentsMd(ctx) { const ocMd = paths.opencodeAgentsMdPath(); - if (fs.existsSync(ocMd)) { - let content = fs.readFileSync(ocMd, 'utf8'); - const slugs = new Set([...content.matchAll(//g)].map((m) => m[1])); - if (slugs.size) { - act(`stripped ${slugs.size} managed block(s) from opencode AGENTS.md (backup written)`, () => { - fs.copyFileSync(ocMd, `${ocMd}.bak.${Date.now()}`); - for (const s of slugs) content = stripBlock(content, s); - fs.writeFileSync(ocMd, content); - }); - } - } + if (!fs.existsSync(ocMd)) return; + let content = fs.readFileSync(ocMd, 'utf8'); + const slugs = new Set([...content.matchAll(//g)].map((m) => m[1])); + if (!slugs.size) return; + ctx.act(`stripped ${slugs.size} managed block(s) from opencode AGENTS.md (backup written)`, () => { + fs.copyFileSync(ocMd, `${ocMd}.bak.${Date.now()}`); + for (const s of slugs) content = stripBlock(content, s); + fs.writeFileSync(ocMd, content); + }); +} - // 2c. Companion teardown is receipt-gated and precedes any kit.json purge. - // The sequence is load-bearing: targets first, then the optional derived - // index while `deja doctor` still exists, and only then the optional package. +/** Approvals + derived facts for the deja-vu teardown phases below, computed + * once so target/data/package phases share one confirm pass. */ +async function computeDejaVuPlan(ctx) { + const { cfg, dry, flags } = ctx; const dejaOwn = dejaVuOwnership(cfg); const ownsDeja = hasDejaVuOwnership(cfg); const ownedTargetCount = plain(dejaOwn?.targets) ? Object.keys(dejaOwn.targets).length : 0; - const dejaAdapter = deps.dejaAdapter ?? companionLifecycleFor('deja-vu'); + const dejaAdapter = ctx.deps.dejaAdapter ?? companionLifecycleFor('deja-vu'); let removePackageApproved = false; let purgeDataApproved = dry && flags['purge-deja-vu-data']; if (!dry && flags['remove-deja-vu'] && dejaOwn?.install) { @@ -274,96 +276,122 @@ export async function run({ flags, deps = {} }) { ); if (!purgeDataApproved) info('kept deja-vu derived index'); } - - const undoDejaVu = async (removePackage) => { - try { - const retired = await runLifecycle({ - adapter: dejaAdapter, - action: 'undo', - cfg, - options: { removePackage }, - }); - if (retired?.configChanged) saveKitConfig(cfg); - return retired; - } catch { - return { ok: false, changed: false, configChanged: false }; - } + return { + dejaOwn, ownsDeja, ownedTargetCount, dejaAdapter, removePackageApproved, purgeDataApproved, }; +} - // Phase 1: default uninstall always attempts only Kit-owned target receipts. - if (ownsDeja && dry) { +async function undoDejaVu(ctx, removePackage) { + try { + const retired = await runLifecycle({ + adapter: ctx.plan.dejaAdapter, + action: 'undo', + cfg: ctx.cfg, + options: { removePackage }, + }); + if (retired?.configChanged) saveKitConfig(ctx.cfg); + return retired; + } catch { + return { ok: false, changed: false, configChanged: false }; + } +} + +// Phase 1: default uninstall always attempts only Kit-owned target receipts. +async function dejaVuTargetTeardown(ctx) { + const { ownsDeja, ownedTargetCount, dejaAdapter } = ctx.plan; + if (ownsDeja && ctx.dry) { if (ownedTargetCount > 0) info('[dry-run] remove Kit-owned deja-vu target wiring'); - } else if (ownsDeja) { - if (!dejaAdapter) { - warn('deja-vu teardown unavailable — ownership receipt retained'); - dejaVuTeardownOk = false; - } else { - const retired = await undoDejaVu(false); - dejaVuTeardownOk = retired?.ok === true; - if (dejaVuTeardownOk) { - if (retired?.changed) ok('deja-vu: Kit-owned target wiring teardown complete'); - } else { - warn('deja-vu teardown incomplete — recovery ownership receipts retained'); - } - } + return; + } + if (!ownsDeja) return; + if (!dejaAdapter) { + warn('deja-vu teardown unavailable — ownership receipt retained'); + ctx.state.dejaVuTeardownOk = false; + return; } + const retired = await undoDejaVu(ctx, false); + ctx.state.dejaVuTeardownOk = retired?.ok === true; + if (ctx.state.dejaVuTeardownOk) { + if (retired?.changed) ok('deja-vu: Kit-owned target wiring teardown complete'); + } else { + warn('deja-vu teardown incomplete — recovery ownership receipts retained'); + } +} - // Phase 2: data has a separate destructive scope and is validated through a - // single offline doctor call. Dry-run performs the validation but no delete. - if (purgeDataApproved) { - if (!dry && !dejaVuTeardownOk) { - warn('deja-vu derived index retained because ownership teardown is incomplete'); - } else { - const purge = deps.purgeDejaVuIndex ?? purgeDejaVuIndex; - const removed = await purge({ homeDir: paths.home, dryRun: dry }); - if (removed?.ok) { - if (dry) info('[dry-run] validated deja-vu derived index; would delete it (path withheld)'); - else (removed.changed ? ok : info)(removed.changed - ? 'deja-vu derived index deleted (path withheld)' - : 'deja-vu derived index was already absent'); - } else { - warn(`${dry ? '[dry-run] ' : ''}deja-vu derived index refused — validation or doctor check failed (path withheld)`); - dejaVuTeardownOk = false; - } - } +// Phase 2: data has a separate destructive scope and is validated through a +// single offline doctor call. Dry-run performs the validation but no delete. +async function dejaVuDataPurge(ctx) { + if (!ctx.plan.purgeDataApproved) return; + const { dry } = ctx; + if (!dry && !ctx.state.dejaVuTeardownOk) { + warn('deja-vu derived index retained because ownership teardown is incomplete'); + return; + } + const purge = ctx.deps.purgeDejaVuIndex ?? purgeDejaVuIndex; + const removed = await purge({ homeDir: paths.home, dryRun: dry }); + if (removed?.ok) { + if (dry) info('[dry-run] validated deja-vu derived index; would delete it (path withheld)'); + else (removed.changed ? ok : info)(removed.changed + ? 'deja-vu derived index deleted (path withheld)' + : 'deja-vu derived index was already absent'); + } else { + warn(`${dry ? '[dry-run] ' : ''}deja-vu derived index refused — validation or doctor check failed (path withheld)`); + ctx.state.dejaVuTeardownOk = false; } +} - // Phase 3: package removal is possible only after target teardown and any - // requested data purge succeeded. A data failure retains the CLI for retry. - if (flags['remove-deja-vu']) { - if (!ownsDeja || !dejaOwn?.install) { - info('deja-vu package preserved — no Kit ownership receipt'); - } else if (dry) { - info('[dry-run] uninstall Kit-owned deja-vu package after target/data teardown'); - } else if (removePackageApproved && !dejaVuTeardownOk) { - warn('deja-vu package retained because target/data teardown is incomplete'); - } else if (removePackageApproved) { - const retired = await undoDejaVu(true); - dejaVuTeardownOk = retired?.ok === true; - if (dejaVuTeardownOk && retired?.changed) ok('deja-vu: Kit-owned package removed'); - else if (!dejaVuTeardownOk) warn('deja-vu package removal incomplete — ownership receipt retained'); - } +// Phase 3: package removal is possible only after target teardown and any +// requested data purge succeeded. A data failure retains the CLI for retry. +async function dejaVuPackageRemoval(ctx) { + if (!ctx.flags['remove-deja-vu']) return; + const { dry } = ctx; + const { ownsDeja, dejaOwn, removePackageApproved } = ctx.plan; + if (!ownsDeja || !dejaOwn?.install) { + info('deja-vu package preserved — no Kit ownership receipt'); + } else if (dry) { + info('[dry-run] uninstall Kit-owned deja-vu package after target/data teardown'); + } else if (removePackageApproved && !ctx.state.dejaVuTeardownOk) { + warn('deja-vu package retained because target/data teardown is incomplete'); + } else if (removePackageApproved) { + const retired = await undoDejaVu(ctx, true); + ctx.state.dejaVuTeardownOk = retired?.ok === true; + if (ctx.state.dejaVuTeardownOk && retired?.changed) ok('deja-vu: Kit-owned package removed'); + else if (!ctx.state.dejaVuTeardownOk) warn('deja-vu package removal incomplete — ownership receipt retained'); } - ownershipTeardownOk = ownershipTeardownOk && dejaVuTeardownOk; - // Registry-driven host lifecycle teardown — reached by id, never by name - // (mirrors x/host.mjs's off(), which does its undo the same way). cfg comes - // from the top of run() (read before any purge of kit.json); --purge - // removes kit.json below, so persisting cfg here would recreate it. Each - // adapter's own undo() already honors ownership/receipts (opencode's - // undoOpencode no-ops when it never held mcp:'ak', and marker-gates - // artifact removal independent of that), so a BUILT-IN's call is - // unconditional per host, same as before ADR-0031 P3 — the only kit-side - // gate is "did anything actually happen", to avoid a no-op teardown line - // (and a needless kit.json rewrite) on a host that was never enabled. An - // ADMITTED external host is different: there is no "always safe, always - // idempotent" guarantee for an arbitrary third-party hook the way there is - // for opencode's own undo, so an admitted host's teardown is gated by - // lifecycleExecutionEnabled (cfg enablement AND the experimental flag) — - // an admitted host that was never enabled/consented for this run is never - // invoked. hostsWithLifecycle() (built-ins + admitted, ADR-0031 P3) is safe - // to loop unconditionally now: lifecycle-render.mjs's renderUndoReport - // dispatches on the runLifecycle result's own shape, so this loop body - // never destructures a host-specific result directly. +} + +// 2c. Companion teardown is receipt-gated and precedes any kit.json purge. +// The sequence is load-bearing: targets first, then the optional derived +// index while `deja doctor` still exists, and only then the optional package. +async function stepDejaVu(ctx) { + ctx.plan = await computeDejaVuPlan(ctx); + await dejaVuTargetTeardown(ctx); + await dejaVuDataPurge(ctx); + await dejaVuPackageRemoval(ctx); + ctx.state.ownershipTeardownOk = ctx.state.ownershipTeardownOk && ctx.state.dejaVuTeardownOk; +} + +// Registry-driven host lifecycle teardown — reached by id, never by name +// (mirrors x/host.mjs's off(), which does its undo the same way). cfg comes +// from the top of run() (read before any purge of kit.json); --purge +// removes kit.json below, so persisting cfg here would recreate it. Each +// adapter's own undo() already honors ownership/receipts (opencode's +// undoOpencode no-ops when it never held mcp:'ak', and marker-gates +// artifact removal independent of that), so a BUILT-IN's call is +// unconditional per host, same as before ADR-0031 P3 — the only kit-side +// gate is "did anything actually happen", to avoid a no-op teardown line +// (and a needless kit.json rewrite) on a host that was never enabled. An +// ADMITTED external host is different: there is no "always safe, always +// idempotent" guarantee for an arbitrary third-party hook the way there is +// for opencode's own undo, so an admitted host's teardown is gated by +// lifecycleExecutionEnabled (cfg enablement AND the experimental flag) — +// an admitted host that was never enabled/consented for this run is never +// invoked. hostsWithLifecycle() (built-ins + admitted, ADR-0031 P3) is safe +// to loop unconditionally now: lifecycle-render.mjs's renderUndoReport +// dispatches on the runLifecycle result's own shape, so this loop body +// never destructures a host-specific result directly. +async function stepHostLifecycles(ctx) { + const { cfg, dry, flags } = ctx; for (const hostId of hostsWithLifecycle()) { if (!isBuiltinHost(hostId) && !lifecycleExecutionEnabled(hostId, cfg)) continue; const adapter = lifecycleAdapterFor(hostId); @@ -375,7 +403,7 @@ export async function run({ flags, deps = {} }) { } const retired = await runLifecycle({ adapter, action: 'undo', cfg }); const undoReport = renderUndoReport(hostId, retired); - ownershipTeardownOk = ownershipTeardownOk && undoReport.ok; + ctx.state.ownershipTeardownOk = ctx.state.ownershipTeardownOk && undoReport.ok; // Persist markers unconditionally, exactly like x/host.mjs's off()/pick(): // undo() mutates cfg's ownership markers in memory even when it rewrote // no file (`undo.changed` measures the FILE, not cfg), so gating the save @@ -385,30 +413,38 @@ export async function run({ flags, deps = {} }) { if (!flags.purge) saveKitConfig(cfg); for (const line of undoReport.lines) printReportLine(line); } - if (flags.purge) { - for (const [label, file] of [ - ['model inventory cache', modelInventoryPath()], ['model scope key', modelScopeKeyPath()], - ]) { - if (fs.existsSync(file)) act(`removed ${label}`, () => fs.rmSync(file)); - } - } - if (flags.purge && fs.existsSync(paths.kitConfigPath())) { - if (ownershipTeardownOk) act('removed kit.json', () => fs.rmSync(paths.kitConfigPath())); - else if (!dejaVuTeardownOk) { - warn('kit.json retained because deja-vu teardown is incomplete; it contains recovery ownership receipts'); - } else warn('kit.json retained because OpenCode teardown is incomplete; it contains the recovery ownership receipt'); +} + +function stepPurgeArtifacts(ctx) { + for (const [label, file] of [ + ['model inventory cache', modelInventoryPath()], ['model scope key', modelScopeKeyPath()], + ]) { + if (fs.existsSync(file)) ctx.act(`removed ${label}`, () => fs.rmSync(file)); } +} - // 3. MCP registration + deny rules - if (dry) info('[dry-run] unregister claude-flow/ruflo MCP + clean deny rules'); - else { const removed = await unregister(); ok(`MCP unregistered (deny rules cleaned: ${removed})`); } +function stepPurgeKitConfig(ctx) { + if (ctx.state.ownershipTeardownOk) { + ctx.act('removed kit.json', () => fs.rmSync(paths.kitConfigPath())); + } else if (!ctx.state.dejaVuTeardownOk) { + warn('kit.json retained because deja-vu teardown is incomplete; it contains recovery ownership receipts'); + } else warn('kit.json retained because OpenCode teardown is incomplete; it contains the recovery ownership receipt'); +} - // 4. legacy shell-kit remnants +// 3. MCP registration + deny rules +async function stepMcp(ctx) { + if (ctx.dry) { info('[dry-run] unregister claude-flow/ruflo MCP + clean deny rules'); return; } + const removed = await unregister(); + ok(`MCP unregistered (deny rules cleaned: ${removed})`); +} + +// 4. legacy shell-kit remnants +function stepLegacyShellKit(ctx) { for (const rc of ['.zshrc', '.bashrc'].map((f) => path.join(paths.home, f))) { if (!fs.existsSync(rc)) continue; const txt = fs.readFileSync(rc, 'utf8'); if (txt.includes('ruflo-functions.sh')) { - act(`removed shell-kit source line from ${rc}`, () => { + ctx.act(`removed shell-kit source line from ${rc}`, () => { fs.copyFileSync(rc, `${rc}.bak`); fs.writeFileSync(rc, txt.split('\n').filter((l) => !l.includes('ruflo-functions.sh')).join('\n')); }); @@ -418,31 +454,33 @@ export async function run({ flags, deps = {} }) { if (fs.existsSync(localBin)) { // every ruflo-* here is shell-kit era (the npm kit's bins live in npm's global bin) for (const f of fs.readdirSync(localBin).filter((f) => f.startsWith('ruflo-'))) { - act(`removed legacy ${path.join(localBin, f)}`, () => fs.rmSync(path.join(localBin, f))); + ctx.act(`removed legacy ${path.join(localBin, f)}`, () => fs.rmSync(path.join(localBin, f))); } } const cfgDir = paths.legacyConfigDir(); // shell-kit files lived in ~/.config/ruflo if (fs.existsSync(cfgDir)) { for (const f of fs.readdirSync(cfgDir).filter((f) => f.endsWith('.sh') || f.endsWith('-template.md') || f === 'ruflo-reference-full.md')) { - act(`removed legacy ${path.join(cfgDir, f)}`, () => fs.rmSync(path.join(cfgDir, f))); + ctx.act(`removed legacy ${path.join(cfgDir, f)}`, () => fs.rmSync(path.join(cfgDir, f))); } } +} - // 5. per-project revert - if (flags['this-project']) { - const sl = paths.projectStatusline(process.cwd()); - if (fs.existsSync(sl)) { - act('reverted statusline footer in this project', () => { - fs.copyFileSync(sl, `${sl}.bak`); - let s = fs.readFileSync(sl, 'utf8'); - s = s.replace(/\/\* ruflo-seg:BEGIN \*\/[\s\S]*?\/\* ruflo-seg:END \*\/\n?/, ''); - s = s.replace(/ \+ rufloActivationSegments\(process\.cwd\(\)\)/g, ''); - fs.writeFileSync(sl, s); - }); - } - } +// 5. per-project revert +function stepThisProject(ctx) { + const sl = paths.projectStatusline(process.cwd()); + if (!fs.existsSync(sl)) return; + ctx.act('reverted statusline footer in this project', () => { + fs.copyFileSync(sl, `${sl}.bak`); + let s = fs.readFileSync(sl, 'utf8'); + s = s.replace(/\/\* ruflo-seg:BEGIN \*\/[\s\S]*?\/\* ruflo-seg:END \*\/\n?/, ''); + s = s.replace(/ \+ rufloActivationSegments\(process\.cwd\(\)\)/g, ''); + fs.writeFileSync(sl, s); + }); +} - // 6. global packages (machine-wide — confirmed individually) +// 6. global packages (machine-wide — confirmed individually) +async function stepGlobalPackages(ctx) { + const { flags, dry } = ctx; const removals = []; if (flags['remove-ruflo'] || flags.purge) removals.push('ruflo'); if (flags['remove-aqe'] || flags.purge) removals.push('agentic-qe'); @@ -454,13 +492,57 @@ export async function run({ flags, deps = {} }) { (r.code === 0 ? ok : warn)(`${pkg}: ${r.code === 0 ? 'removed' : 'could not remove'}`); } else info(`kept ${pkg}`); } +} - // RuvNet Brain: a user-scope plugin + a large (~512 MB) KB cache — left in - // place (like ruflo/aqe) rather than force-deleted. Point at the manual path. +// RuvNet Brain: a user-scope plugin + a large (~512 MB) KB cache — left in +// place (like ruflo/aqe) rather than force-deleted. Point at the manual path. +function stepRuvnetBrainNotice() { if (rbPresent()) { info('RuvNet Brain left installed — remove manually: `claude plugin uninstall ruvnet-brain@ruvnet-brain` + `rm -rf ~/.cache/ruvnet-brain`'); } +} + +export const UNINSTALL_STEPS = [ + { id: 'codex-statusline', when: (ctx) => !!ctx.cfg.statusline?.codex, run: stepCodexStatusline }, + { id: 'claude-md-blocks', when: () => true, run: stepClaudeMdBlocks }, + { id: 'skill', when: () => true, run: stepSkill }, + { id: 'opencode-agents-md', when: () => true, run: stepOpencodeAgentsMd }, + { id: 'deja-vu', when: () => true, run: stepDejaVu }, + { id: 'host-lifecycles', when: () => true, run: stepHostLifecycles }, + { id: 'purge-artifacts', when: (ctx) => ctx.flags.purge, run: stepPurgeArtifacts }, + { + id: 'purge-kit-config', + when: (ctx) => ctx.flags.purge && fs.existsSync(paths.kitConfigPath()), + run: stepPurgeKitConfig, + }, + { id: 'mcp', when: () => true, run: stepMcp }, + { id: 'legacy-shell-kit', when: () => true, run: stepLegacyShellKit }, + { id: 'this-project', when: (ctx) => ctx.flags['this-project'], run: stepThisProject }, + { id: 'global-packages', when: () => true, run: stepGlobalPackages }, + { id: 'ruvnet-brain-notice', when: () => true, run: stepRuvnetBrainNotice }, +]; + +/** @param {{flags:Record,deps?:UninstallDeps}} request */ +export async function run({ flags, deps = {} }) { + const dry = flags['dry-run']; + const act = (msg, fn) => { if (dry) info(`[dry-run] ${msg}`); else { fn(); ok(msg); } }; + // Ownership markers are read ONCE up front: the purge path removes kit.json + // below, and teardown decisions (opencode undo) must still see what ak owned + // (codex-review — purge ordering must not strand managed opencode.json keys). + const cfg = loadKitConfig(); + const ctx = { + flags, + dry, + deps, + cfg, + act, + state: { ownershipTeardownOk: true, dejaVuTeardownOk: true }, + }; + + for (const step of UNINSTALL_STEPS) { + if (step.when(ctx)) await step.run(ctx); + } ok('uninstall complete — project data (.swarm/.claude-flow/.agentic-qe) untouched'); - return ownershipTeardownOk ? 0 : 1; + return ctx.state.ownershipTeardownOk ? 0 : 1; } diff --git a/src/commands/x/host-adapters-grants.mjs b/src/commands/x/host-adapters-grants.mjs index 2573a59..8e3dbe8 100644 --- a/src/commands/x/host-adapters-grants.mjs +++ b/src/commands/x/host-adapters-grants.mjs @@ -67,44 +67,69 @@ function capabilityStatusNote(capability) { return 'commandStatusline is currently inert: it reaches the effective host registry, but no runtime path reads it anywhere yet — the statusline render path is a later wave.'; } -export async function grant({ - name, capability, cfg, consent, reader, ask, isTTY, yes, grantsFile, -}) { +/** Validate the request and resolve the manifest, returning either + * {ok:false, code, message} (already-safe text, ready for `fail()`) or + * {ok:true, safeName, safeCapability, manifest, hash}. */ +async function resolveGrantSubject({ name, capability, cfg, reader }) { if (typeof name !== 'string' || !name || typeof capability !== 'string' || !capability) { - fail('usage: ak host adapters grant '); - return 2; + return { ok: false, code: 2, message: 'usage: ak host adapters grant ' }; } const safeCapability = stripControl(capability); if (!grantableCapability(capability)) { - fail(`'${safeCapability}' is not a grantable capability — ak can only grant ${Object.values(TIER_GRANTS).join(', ')}.`); - return 1; + return { + ok: false, code: 1, + message: `'${safeCapability}' is not a grantable capability — ak can only grant ${Object.values(TIER_GRANTS).join(', ')}.`, + }; } const safeName = stripControl(name); const entry = findEntry(cfg, name); - if (!entry) { fail(`no host adapter named '${safeName}' in kit.json hostAdapters`); return 1; } + if (!entry) return { ok: false, code: 1, message: `no host adapter named '${safeName}' in kit.json hostAdapters` }; const loaded = await loadAndHash(entry, { reader }); if (!loaded.ok) { - fail(`'${safeName}' manifest refused: ${stripControl(loaded.reason)} — ${stripControl(loaded.detail)}`); - return 1; + return { + ok: false, code: 1, + message: `'${safeName}' manifest refused: ${stripControl(loaded.reason)} — ${stripControl(loaded.detail)}`, + }; } const { manifest, hash } = loaded; if (capability === 'aqeProvider' && !manifest.aqe?.provider) { - fail(`grant refused: '${safeName}' does not declare manifest.aqe.provider at this adapter-content hash`); - return 1; + return { + ok: false, code: 1, + message: `grant refused: '${safeName}' does not declare manifest.aqe.provider at this adapter-content hash`, + }; } - const tier = gatingTierFor(capability); - const safeTier = stripControl(tier); + return { + ok: true, safeName, safeCapability, manifest, hash, + }; +} - // F-3 (security review): this is the highest-privilege act in the model — - // a hex hash alone told the operator nothing about what they were actually - // converting to a live capability. Disclose the ACTUAL evidence backing - // the gating tier, the manifest's declared hooks, and the manifest's trust - // state, all BEFORE the confirmation prompt. - const record = grantsFor(name, { file: grantsFile, currentHash: hash }); - const tierEntry = record?.tiers?.[tier]; - const evidence = tierEntry?.status === 'passed' ? tierEntry.evidence : undefined; +/** N-1 (security re-review): derive trust state from the hash the caller + * ALREADY resolved — never re-resolve the manifest source (stateFor would + * call loadAndHash a second time). On a mutable remote source (https:/npm:) + * a second resolve can return different bytes than the first, which would + * (a) disclose a trust state describing content that isn't what's actually + * being granted, and (b) double the fetch cost (an npm: source runs `npm + * pack` twice). Same three-state logic as stateFor, just computed from data + * already in hand. */ +function resolveGrantTrustState(consent, name, hash) { + try { + const recordedHash = consent.recordedHashFor(name); + if (recordedHash === null || recordedHash === undefined) return 'not consented'; + return recordedHash === hash ? 'trusted' : 'consent-stale'; + } catch (error) { + return `manifest error (consent-error: ${error?.message ?? String(error)})`; + } +} +/** F-3 (security review): this is the highest-privilege act in the model — + * a hex hash alone told the operator nothing about what they were actually + * converting to a live capability. Disclose the ACTUAL evidence backing the + * gating tier, the manifest's declared hooks, and the manifest's trust + * state, all BEFORE the confirmation prompt. */ +function discloseGrant({ + safeCapability, safeName, safeTier, hash, evidence, manifest, trustState, capability, +}) { console.log(bold(`grant '${safeCapability}' to '${safeName}'`)); console.log(` gating tier: ${safeTier}`); console.log(` content hash (manifest + hook files): ${hash}`); @@ -112,26 +137,33 @@ export async function grant({ const hooks = hookCommandsFor(manifest); console.log(` manifest hooks:${hooks.length ? '' : ' (none)'}`); for (const line of hooks) console.log(` ${stripControl(line)}`); - // N-1 (security re-review): derive trust state from the hash this call - // ALREADY resolved above — never re-resolve the manifest source (stateFor - // would call loadAndHash a second time). On a mutable remote source - // (https:/npm:) a second resolve can return different bytes than the - // first, which would (a) disclose a trust state describing content that - // isn't what's actually being granted, and (b) double the fetch cost (an - // npm: source runs `npm pack` twice). Same three-state logic as stateFor, - // just computed from data already in hand. - let trustState; - try { - const recordedHash = consent.recordedHashFor(name); - if (recordedHash === null || recordedHash === undefined) trustState = 'not consented'; - else trustState = recordedHash === hash ? 'trusted' : 'consent-stale'; - } catch (error) { - trustState = `manifest error (consent-error: ${error?.message ?? String(error)})`; - } console.log(` manifest trust state: ${trustState}`); info('this grant pins the combined adapter content identity: validated manifest plus every declared hook-file digest.'); info("granting a capability is a trust act, same posture as 'trust': it takes effect in the effective host registry from the next ak invocation."); info(capabilityStatusNote(capability)); +} + +export async function grant({ + name, capability, cfg, consent, reader, ask, isTTY, yes, grantsFile, +}) { + const subject = await resolveGrantSubject({ + name, capability, cfg, reader, + }); + if (!subject.ok) { fail(subject.message); return subject.code; } + const { + safeName, safeCapability, manifest, hash, + } = subject; + const tier = gatingTierFor(capability); + const safeTier = stripControl(tier); + + const record = grantsFor(name, { file: grantsFile, currentHash: hash }); + const tierEntry = record?.tiers?.[tier]; + const evidence = tierEntry?.status === 'passed' ? tierEntry.evidence : undefined; + const trustState = resolveGrantTrustState(consent, name, hash); + + discloseGrant({ + safeCapability, safeName, safeTier, hash, evidence, manifest, trustState, capability, + }); if (!yes) { if (!isTTY) { diff --git a/src/commands/x/host-adapters.mjs b/src/commands/x/host-adapters.mjs index 9657127..0264c2f 100644 --- a/src/commands/x/host-adapters.mjs +++ b/src/commands/x/host-adapters.mjs @@ -76,48 +76,50 @@ export function findEntry(cfg, name) { return (Array.isArray(cfg?.hostAdapters) ? cfg.hostAdapters : []).find((e) => e?.name === name) ?? null; } -/** Read + validate + hash one adapter entry. Never throws — reports - * {ok:false, reason, detail} on any failure, the same per-entry isolation - * posture admission.mjs's admitOne holds. `reader` may return either the - * sources.mjs `{raw, origin}` shape or a bare raw document (tests are free - * to stub either). */ -export async function loadAndHash(entry, { reader }) { - let raw; - // Fail-closed, not fail-open: a bare-raw reader result (no {raw,origin} - // wrapper — including one whose `origin` is missing/malformed) is - // 'unknown', never assumed to be 'file'. Defaulting to 'file' would let - // --yes silently skip the --expect-hash pin (finding 8) for a source that - // was never actually proven local; only an explicit origin:'file' counts. - let origin = 'unknown'; +/** Resolve one entry's raw manifest document. Fail-closed, not fail-open: a + * bare-raw reader result (no {raw,origin} wrapper — including one whose + * `origin` is missing/malformed) is 'unknown', never assumed to be 'file'. + * Defaulting to 'file' would let --yes silently skip the --expect-hash pin + * (finding 8) for a source that was never actually proven local; only an + * explicit origin:'file' counts. */ +async function resolveManifestRaw(entry, reader) { try { const resolved = await reader(entry.source); if (resolved && typeof resolved === 'object' && 'raw' in resolved) { - raw = resolved.raw; - origin = typeof resolved.origin === 'string' && resolved.origin ? resolved.origin : 'unknown'; - } else { - raw = resolved; + const origin = typeof resolved.origin === 'string' && resolved.origin ? resolved.origin : 'unknown'; + return { ok: true, raw: resolved.raw, origin }; } + return { ok: true, raw: resolved, origin: 'unknown' }; } catch (error) { return { ok: false, reason: error?.reason ?? 'manifest-unreadable', detail: error?.message ?? String(error) }; } +} - // Same distinct failure admitOne draws: cfg's pinned contract disagreeing - // with the manifest's self-declared one means the file changed underneath - // the operator, not merely "unsupported version". +/** Same distinct failure admitOne draws: cfg's pinned contract disagreeing + * with the manifest's self-declared one means the file changed underneath + * the operator, not merely "unsupported version". */ +function checkManifestContractMatch(entry, raw) { if (entry.contract !== undefined && raw?.contract !== undefined && entry.contract !== raw.contract) { return { ok: false, reason: 'contract-mismatch', detail: `cfg declares contract ${entry.contract}, manifest declares ${raw.contract}`, }; } + return { ok: true }; +} +/** Validate shape + contract version + name match + built-in shadowing. + * Same check order admitOne applies before ever computing a hash: a manifest + * whose host id collides with a built-in can never actually be admitted, so + * trusting it would record a standing consent for content admission that + * will always refuse — misleading UX for nothing gained. */ +function validateManifestShape(entry, raw) { let manifest; try { manifest = validateAdapterManifest(raw); } catch (error) { return { ok: false, reason: error?.reason ?? 'manifest-invalid', detail: error?.message ?? String(error) }; } - if (manifest.contract !== SUPPORTED_CONTRACT) { return { ok: false, reason: 'contract-version', detail: `unsupported contract ${manifest.contract}` }; } @@ -127,13 +129,28 @@ export async function loadAndHash(entry, { reader }) { detail: `cfg entry '${entry.name}' does not match manifest host id '${manifest.host.id}'`, }; } - // Same check admitOne applies before ever computing a hash: a manifest - // whose host id collides with a built-in can never actually be admitted, - // so trusting it would record a standing consent for content admission - // will always refuse — misleading UX for nothing gained. if (HOST_REGISTRY.some((host) => host.id === manifest.host.id)) { return { ok: false, reason: 'builtin-shadow', detail: `'${manifest.host.id}' is a built-in host id` }; } + return { ok: true, manifest }; +} + +/** Read + validate + hash one adapter entry. Never throws — reports + * {ok:false, reason, detail} on any failure, the same per-entry isolation + * posture admission.mjs's admitOne holds. `reader` may return either the + * sources.mjs `{raw, origin}` shape or a bare raw document (tests are free + * to stub either). */ +export async function loadAndHash(entry, { reader }) { + const resolved = await resolveManifestRaw(entry, reader); + if (!resolved.ok) return resolved; + const { raw, origin } = resolved; + + const contractMatch = checkManifestContractMatch(entry, raw); + if (!contractMatch.ok) return contractMatch; + + const validated = validateManifestShape(entry, raw); + if (!validated.ok) return validated; + const { manifest } = validated; let integrity; try { @@ -415,6 +432,50 @@ async function conformance({ return anyFailed ? 1 : 0; } +// Revocation is fail-safe and stays reachable regardless of the experimental +// flag: an operator who turns the flag OFF must still be able to withdraw a +// standing consent or grant record, or it silently reactivates the next time +// the flag is turned back on. +const FAIL_SAFE_HANDLERS = { + revoke: (ctx) => revoke({ name: ctx.name, consent: ctx.consent }), + 'revoke-grant': (ctx) => revokeGrant({ + name: ctx.name, capability: ctx.positionals[2], grantsFile: ctx.grantsFile, cfg: ctx.cfg, env: ctx.env, cwd: ctx.cwd, + ...(ctx.saveConfig ? { saveConfig: ctx.saveConfig } : {}), + ...(ctx.bootstrapAdapters ? { bootstrapAdapters: ctx.bootstrapAdapters } : {}), + ...(ctx.applyRouter ? { applyRouter: ctx.applyRouter } : {}), + }), +}; + +// `list`/`trust`/`conformance`/`grant`/`gate`/`status` stay gated behind the +// experimental flag — they're the surface that reads/records new trust, +// evidence, or capability. `resolvedCfg` is populated by run() only once the +// flag check passes (below). +const grantHandler = (ctx) => grantCap({ + name: ctx.name, capability: ctx.positionals[2], cfg: ctx.resolvedCfg, consent: ctx.consent, reader: ctx.reader, + ask: ctx.ask, isTTY: ctx.isTTY, yes: !!ctx.flags.yes, grantsFile: ctx.grantsFile, +}); +const GATED_HANDLERS = { + list: (ctx) => list({ cfg: ctx.resolvedCfg, consent: ctx.consent, reader: ctx.reader }), + trust: (ctx) => trust({ + name: ctx.name, cfg: ctx.resolvedCfg, consent: ctx.consent, reader: ctx.reader, ask: ctx.ask, isTTY: ctx.isTTY, + yes: !!ctx.flags.yes, expectHash: ctx.flags['expect-hash'], + }), + conformance: (ctx) => conformance({ + name: ctx.name, cfg: ctx.resolvedCfg, reader: ctx.reader, runTiered: ctx.runTieredConformance, + consentFile: ctx.consentFile, grantsFile: ctx.grantsFile, flags: ctx.flags, + }), + // F-8 (security review, ADR-0031-accurate naming): `bless` is the alias — + // §3 calls this exact out-of-tree grant path a "Blessed external adapter"; + // "Promoted built-in" is the separate manual registry PR this command does + // NOT do. `grant` stays the primary spelling; `promote` was dropped. + grant: grantHandler, + bless: grantHandler, + gate: (ctx) => gateTier({ + name: ctx.name, tier: ctx.positionals[2], ref: ctx.positionals[3], cfg: ctx.resolvedCfg, reader: ctx.reader, grantsFile: ctx.grantsFile, + }), + status: (ctx) => statusReport({ name: ctx.name, cfg: ctx.resolvedCfg, reader: ctx.reader, grantsFile: ctx.grantsFile }), +}; + /** * @param {{ positionals?: string[], flags?: any, env?: NodeJS.ProcessEnv, * consent?: { recordedHashFor(name:string): string|null, recordConsent(name:string, hash:string): void, revokeConsent(name:string): boolean }, @@ -434,59 +495,24 @@ export async function run({ saveConfig, bootstrapAdapters, applyRouter, } = {}) { const sub = positionals[0] ?? 'list'; - const name = positionals[1]; - - // Revocation is fail-safe and stays reachable regardless of the - // experimental flag: an operator who turns the flag OFF must still be - // able to withdraw a standing consent or grant record, or it silently - // reactivates the next time the flag is turned back on. `list`/`trust`/ - // `conformance`/`grant`/`gate`/`status` stay gated — they're the surface - // that reads/records new trust, evidence, or capability. - if (sub === 'revoke') return revoke({ name, consent }); - if (sub === 'revoke-grant') { - return revokeGrant({ - name, capability: positionals[2], grantsFile, cfg, env, cwd, - ...(saveConfig ? { saveConfig } : {}), - ...(bootstrapAdapters ? { bootstrapAdapters } : {}), - ...(applyRouter ? { applyRouter } : {}), - }); - } + const ctx = { + positionals, flags, env, consent, reader, ask, isTTY, cfg, runTieredConformance, + consentFile, grantsFile, cwd, saveConfig, bootstrapAdapters, applyRouter, + name: positionals[1], + }; + + const failSafe = FAIL_SAFE_HANDLERS[sub]; + if (failSafe) return failSafe(ctx); if (!flagEnabled(env)) { fail(`experimental host-adapter surface is disabled — set ${FLAG_ENV_VAR}=1`); return 2; } - const resolvedCfg = cfg ?? loadKitConfig(); + ctx.resolvedCfg = cfg ?? loadKitConfig(); - if (sub === 'list') return list({ cfg: resolvedCfg, consent, reader }); - if (sub === 'trust') { - return trust({ - name, cfg: resolvedCfg, consent, reader, ask, isTTY, - yes: !!flags.yes, expectHash: flags['expect-hash'], - }); - } - if (sub === 'conformance') { - return conformance({ - name, cfg: resolvedCfg, reader, runTiered: runTieredConformance, consentFile, grantsFile, flags, - }); - } - // F-8 (security review, ADR-0031-accurate naming): `bless` is the alias — - // §3 calls this exact out-of-tree grant path a "Blessed external adapter"; - // "Promoted built-in" is the separate manual registry PR this command does - // NOT do. `grant` stays the primary spelling; `promote` was dropped. - if (sub === 'grant' || sub === 'bless') { - return grantCap({ - name, capability: positionals[2], cfg: resolvedCfg, consent, reader, ask, isTTY, - yes: !!flags.yes, grantsFile, - }); - } - if (sub === 'gate') { - return gateTier({ - name, tier: positionals[2], ref: positionals[3], cfg: resolvedCfg, reader, grantsFile, - }); - } - if (sub === 'status') return statusReport({ name, cfg: resolvedCfg, reader, grantsFile }); + const gated = GATED_HANDLERS[sub]; + if (gated) return gated(ctx); fail(`unknown host adapters subcommand: ${sub} (list|trust|revoke|conformance|grant|bless|gate|status|revoke-grant)`); return 2; diff --git a/src/commands/x/host.mjs b/src/commands/x/host.mjs index 152f0dd..1235f8d 100644 --- a/src/commands/x/host.mjs +++ b/src/commands/x/host.mjs @@ -186,29 +186,9 @@ export function bindingWarnings(cfg) { `kit.json integrations.bindings[${index}].${error.path.replace(/^binding\./, '')}: ${error.code} (${JSON.stringify(error.value)})`)); } -async function status({ flags, cwd }) { - const cfg = loadKitConfig(); - const facts = await collectIntegrationFacts({ cwd, cfg }); - const hosts = facts.hosts; - const providers = facts.providers; - const { scope } = settingsTarget(cwd); - - if (flags.json) { - console.log(JSON.stringify({ - scope, - config: { - integrations: cfg.integrations, - routing: cfg.routing, - providers: cfg.providers, - }, - hosts, - providers, - }, null, 2)); - return 0; - } - - for (const message of bindingWarnings(cfg)) warn(message); - +/** 'ruflo agent hosts' section — detected CLIs, enabled/wired state, tier, + * and the auth/billing axis, one line per managed host. */ +function printHostsSection({ cfg, hosts, scope }) { const dflt = isDefault(cfg); console.log(bold('ruflo agent hosts') + dim(` (wiring scope: ${scope})`)); for (const h of HOSTS) { @@ -227,8 +207,11 @@ async function status({ flags, cwd }) { const note = hostAsymmetryNote(h.id); if (note) console.log(` ${dim(note)}`); } +} - // agentic-qe LLM provider (AQE_LLM_PROVIDER) + fallback chain +/** 'agentic-qe LLM provider' section — primary provider + fallback chain, + * flagging any rung with no discoverable credential. */ +function printAqeProviderSection({ cfg }) { const ap = cfg.providers.aqeProvider; console.log(bold('\nagentic-qe LLM provider') + dim(' (built-ins: env; external: project llm-config)')); console.log(` ${(ap ?? dim('aqe default (unset)')).padEnd(24)} ${dim(`supported: ${aqeSelectableProviderTypes().join(', ')}`)}`); @@ -247,10 +230,13 @@ async function status({ flags, cwd }) { } else { console.log(` ${dim('fallback chain: none (aqe auto-enables keyed providers)')}`); } +} - // Credential state for EVERY aqe provider type, so a provider that is - // credentialed on this machine (openrouter, say) is never invisible while an - // uncredentialed one is displayed as a configured fallback (#54). +/** 'aqe provider credentials' section — credential state for EVERY aqe + * provider type, so a provider that is credentialed on this machine + * (openrouter, say) is never invisible while an uncredentialed one is + * displayed as a configured fallback (#54). */ +function printAqeCredentialsSection() { const creds = detectAqeProviders(); console.log(bold('\naqe provider credentials') + dim(' (keys read from env; never persisted)')); for (const p of aqeSelectableProviderTypes()) { @@ -261,7 +247,10 @@ async function status({ flags, cwd }) { : `no key ${dim(`(${c.missing.join(', ')})`)}`; console.log(` ${p.padEnd(14)} ${state}${c.source && c.present && c.billing === 'metered' ? dim(` · ${c.source}`) : ''}`); } +} +/** 'ruflo LLM providers' section — registered intent per API provider. */ +function printRufloProvidersSection({ cfg, providers }) { const cm = cfg.providers.models ?? []; console.log(bold('\nruflo LLM providers') + dim(' (registered intent; direct agents select provider + model)')); for (const p of API_PROVIDERS) { @@ -271,10 +260,10 @@ async function status({ flags, cwd }) { const conf = cfgEntry ? `registered${cfgEntry.model ? ` (${cfgEntry.model})` : ''}` : dim('not registered'); console.log(` ${p.id.padEnd(10)} ${key.padEnd(12)} ${conf}`); } +} - printActivityRoutingTable(cfg); - printQeCourtStatus(cwd); - +/** Closing summary — idle-but-installed hosts, and the dual-host tips. */ +function printHostSummarySection({ cfg, hosts }) { const codexIdle = hosts.codex.present && !cfg.integrations.hosts.codex; const ocIdle = hosts.opencode.present && !cfg.integrations.hosts.opencode; console.log(''); @@ -282,6 +271,47 @@ async function status({ flags, cwd }) { if (ocIdle) info('opencode is installed but disabled — enable it with: ak host pick --host claude,opencode'); if (!codexIdle && !ocIdle) ok('host/provider config reflects installed CLIs'); printDualHostTips(cfg); +} + +// Human-readable `status()` renders as an ordered array of section printers +// over one shared ctx ({cfg, hosts, providers, scope, cwd}) — mirrors the +// house section-registry idiom (ADR-0037) so a future section is one array +// entry, not a new branch threaded through a growing function. +const STATUS_SECTIONS = [ + printHostsSection, + printAqeProviderSection, + printAqeCredentialsSection, + printRufloProvidersSection, + ({ cfg }) => printActivityRoutingTable(cfg), + ({ cwd }) => printQeCourtStatus(cwd), + printHostSummarySection, +]; + +async function status({ flags, cwd }) { + const cfg = loadKitConfig(); + const facts = await collectIntegrationFacts({ cwd, cfg }); + const hosts = facts.hosts; + const providers = facts.providers; + const { scope } = settingsTarget(cwd); + + if (flags.json) { + console.log(JSON.stringify({ + scope, + config: { + integrations: cfg.integrations, + routing: cfg.routing, + providers: cfg.providers, + }, + hosts, + providers, + }, null, 2)); + return 0; + } + + for (const message of bindingWarnings(cfg)) warn(message); + + const ctx = { cfg, hosts, providers, scope, cwd }; + for (const section of STATUS_SECTIONS) section(ctx); return 0; } diff --git a/src/commands/x/verify.mjs b/src/commands/x/verify.mjs index 6f75b87..752de61 100644 --- a/src/commands/x/verify.mjs +++ b/src/commands/x/verify.mjs @@ -260,42 +260,18 @@ function hasDejaVuOwnership(cfg) { return plain(own) && (!!own.install || (plain(own.targets) && Object.keys(own.targets).length > 0)); } -/** - * A bounded, content-free deja-vu proof. Its lifecycle adapter may run only - * presence/version checks, direct wiring observations, and - * `deja doctor --json --offline`; no search/recall command belongs here. - */ -export async function verifyDejaVu({ - cfg = loadKitConfig(), - adapter = companionLifecycleFor('deja-vu'), -} = {}) { - heading('deja-vu — content-free structural companion proof'); - const enabled = cfg?.integrations?.tools?.dejaVu?.enabled === true; - if (!enabled && !hasDejaVuOwnership(cfg)) { - warn('deja-vu disabled and unowned — skipped'); - return true; - } - if (!adapter) { - fail('deja-vu lifecycle adapter unavailable'); - return false; - } - - let result; - try { - result = await runLifecycle({ adapter, action: 'verify', cfg }); - } catch { - fail('deja-vu structural verification could not run (details redacted)'); - return false; - } - const facts = plain(result?.facts) ? result.facts : {}; - const install = plain(facts.install) ? facts.install : {}; +/** Package/CLI presence check — prints its verdict and returns whether it passed. */ +function checkDejaVuPackage(install) { const version = typeof install.version === 'string' && SAFE_VERSION.test(install.version) ? install.version.replace(/^v/, '') : 'unavailable'; const packageGood = install.binaryPresent === true && install.supported === true; const owner = SAFE_OWNERSHIP.has(install.ownership) ? install.ownership : 'unknown'; (packageGood ? ok : fail)(`CLI/package ${version === 'unavailable' ? version : `v${version}`}: ${packageGood ? 'compatible' : 'incompatible or unavailable'} (${owner})`); + return packageGood; +} - const doctor = plain(facts.doctor) ? facts.doctor : {}; +/** `deja doctor` schema + bounded component health check. */ +function checkDejaVuDoctor(doctor) { const doctorGood = doctor.state === 'ok' && doctor.schemaVersion === 2 && doctor.health?.state !== 'degraded'; (doctorGood ? ok : fail)(doctorGood @@ -303,13 +279,22 @@ export async function verifyDejaVu({ : doctor.state === 'ok' && doctor.schemaVersion === 2 ? 'doctor schema v2 accepted but bounded component health is degraded' : 'doctor schema incompatible or unavailable'); + return doctorGood; +} - const index = plain(facts.index) ? facts.index : {}; +/** Derived-index state check — a missing index is fine when disabled or + * never desired on setup. */ +function checkDejaVuIndex(index, enabled, facts) { const indexState = SAFE_INDEX_STATES.has(index.state) ? index.state : 'unknown'; const indexGood = !enabled || indexState === 'ok' || (indexState === 'missing' && facts.desired?.indexOnSetup === false); (indexGood ? ok : fail)(`index state: ${indexState}`); + return indexGood; +} +/** Per-host wiring check across every desired target (claude/codex/opencode + * × mcp/auto), printing one line per target and folding to a single verdict. */ +function checkDejaVuTargets(facts, enabled) { let targetsGood = true; const desiredHosts = enabled && Array.isArray(facts.desired?.hosts) ? facts.desired.hosts : []; const mode = facts.desired?.mode === 'auto' ? 'auto' : 'mcp'; @@ -322,7 +307,13 @@ export async function verifyDejaVu({ (wired ? ok : fail)(`${targetName}: ${wired ? 'wired' : 'not satisfied'}`); targetsGood = wired && targetsGood; } + return targetsGood; +} +/** Fold the four per-surface verdicts into one, reporting the lifecycle + * adapter's own failure count (never its raw errors — see the module + * header) when it did not report ok. */ +function finalizeDejaVuVerdict(result, packageGood, doctorGood, indexGood, targetsGood) { const good = result?.ok === true && packageGood && doctorGood && indexGood && targetsGood; if (!result?.ok) { const count = Array.isArray(result?.errors) ? Math.min(result.errors.length, 99) : 1; @@ -331,6 +322,42 @@ export async function verifyDejaVu({ return good; } +/** + * A bounded, content-free deja-vu proof. Its lifecycle adapter may run only + * presence/version checks, direct wiring observations, and + * `deja doctor --json --offline`; no search/recall command belongs here. + */ +export async function verifyDejaVu({ + cfg = loadKitConfig(), + adapter = companionLifecycleFor('deja-vu'), +} = {}) { + heading('deja-vu — content-free structural companion proof'); + const enabled = cfg?.integrations?.tools?.dejaVu?.enabled === true; + if (!enabled && !hasDejaVuOwnership(cfg)) { + warn('deja-vu disabled and unowned — skipped'); + return true; + } + if (!adapter) { + fail('deja-vu lifecycle adapter unavailable'); + return false; + } + + let result; + try { + result = await runLifecycle({ adapter, action: 'verify', cfg }); + } catch { + fail('deja-vu structural verification could not run (details redacted)'); + return false; + } + const facts = plain(result?.facts) ? result.facts : {}; + const packageGood = checkDejaVuPackage(plain(facts.install) ? facts.install : {}); + const doctorGood = checkDejaVuDoctor(plain(facts.doctor) ? facts.doctor : {}); + const indexGood = checkDejaVuIndex(plain(facts.index) ? facts.index : {}, enabled, facts); + const targetsGood = checkDejaVuTargets(facts, enabled); + + return finalizeDejaVuVerdict(result, packageGood, doctorGood, indexGood, targetsGood); +} + export async function run({ positionals }) { const which = positionals[0] ?? 'all'; const suites = { diff --git a/src/lib/adapters/admission.mjs b/src/lib/adapters/admission.mjs index 8e3bf10..9babcbf 100644 --- a/src/lib/adapters/admission.mjs +++ b/src/lib/adapters/admission.mjs @@ -19,7 +19,16 @@ export const SUPPORTED_CONTRACT = 1; const builtinIds = () => new Set(HOST_REGISTRY.map((host) => host.id)); -async function admitOne(entry, { readManifest, consent, builtins }) { +// ── admitOne decomposition ─────────────────────────────────────────────── +// admitOne is a sequential fail-closed pipeline (ADR-0028/0029): each stage +// below either refuses the entry with its own {admitted:false, reason, +// detail} shape (unchanged strings — tests assert them) or hands the next +// stage what it computed. Splitting it into small, single-purpose functions +// keeps every stage's own complexity — and admitOne's own — low, without +// altering the per-entry isolation or any failure's reason/detail wording. + +/** Shape-check the cfg.hostAdapters entry itself, before any I/O. */ +function validateEntryShape(entry) { const name = entry?.name; if (typeof name !== 'string' || !name) { return { name: name ?? '(unknown)', admitted: false, reason: 'invalid-entry', detail: 'cfg.hostAdapters entry requires a name' }; @@ -27,50 +36,62 @@ async function admitOne(entry, { readManifest, consent, builtins }) { if (typeof entry.source !== 'string' || !entry.source) { return { name, admitted: false, reason: 'invalid-entry', detail: 'cfg.hostAdapters entry requires a source' }; } + return null; +} - let raw; +async function readEntryManifest(name, source, readManifest) { try { - raw = await readManifest(entry.source); + return { ok: true, raw: await readManifest(source) }; } catch (error) { - return { name, admitted: false, reason: 'manifest-unreadable', detail: error?.message ?? String(error) }; + return { ok: false, failure: { name, admitted: false, reason: 'manifest-unreadable', detail: error?.message ?? String(error) } }; } +} - // cfg's own pinned contract (if declared) disagreeing with the manifest's - // self-declared contract is a distinct failure from "unsupported version" — - // it means the operator pinned to a manifest that changed underneath them. +/** cfg's own pinned contract (if declared) disagreeing with the manifest's + * self-declared contract is a distinct failure from "unsupported version" — + * it means the operator pinned to a manifest that changed underneath them. */ +function checkContractPin(name, entry, raw) { if (entry.contract !== undefined && raw?.contract !== undefined && entry.contract !== raw.contract) { return { name, admitted: false, reason: 'contract-mismatch', detail: `cfg declares contract ${entry.contract}, manifest declares ${raw.contract}`, }; } + return null; +} - let manifest; +function parseEntryManifest(name, raw) { try { - manifest = validateAdapterManifest(raw); + return { ok: true, manifest: validateAdapterManifest(raw) }; } catch (error) { - return { name, admitted: false, reason: error?.reason ?? 'manifest-invalid', detail: error?.message ?? String(error) }; + return { ok: false, failure: { name, admitted: false, reason: error?.reason ?? 'manifest-invalid', detail: error?.message ?? String(error) } }; } +} +function checkHostIdentity(name, manifest, builtins) { if (manifest.contract !== SUPPORTED_CONTRACT) { return { name, admitted: false, reason: 'contract-version', detail: `unsupported contract ${manifest.contract}` }; } - if (manifest.host.id !== name) { return { name, admitted: false, reason: 'name-mismatch', detail: `cfg entry '${name}' does not match manifest host id '${manifest.host.id}'` }; } - if (builtins.has(manifest.host.id)) { return { name, admitted: false, reason: 'builtin-shadow', detail: `'${manifest.host.id}' is a built-in host id` }; } + return null; +} - let integrity; +function computeEntryIntegrity(name, manifest, source) { try { - integrity = hashAdapterContent(manifest, { baseDir: baseDirForSource(entry.source) }); + return { ok: true, integrity: hashAdapterContent(manifest, { baseDir: baseDirForSource(source) }) }; } catch (error) { - return { name, admitted: false, reason: error?.reason ?? 'hook-integrity', detail: error?.message ?? String(error) }; + return { ok: false, failure: { name, admitted: false, reason: error?.reason ?? 'hook-integrity', detail: error?.message ?? String(error) } }; } - const { hash } = integrity; +} + +/** Consent gate: the adapter's content hash must be BOTH recorded and + * trusted at exactly that hash. */ +function checkConsent(name, hash, consent) { let recorded; try { recorded = consent.recordedHashFor(name); @@ -90,6 +111,35 @@ async function admitOne(entry, { readManifest, consent, builtins }) { if (!trusted || recorded !== hash) { return { name, admitted: false, reason: 'consent-stale', detail: `adapter content hash ${hash} does not match consented ${recorded}` }; } + return null; +} + +async function admitOne(entry, { readManifest, consent, builtins }) { + const shapeFailure = validateEntryShape(entry); + if (shapeFailure) return shapeFailure; + const { name } = entry; + + const manifestRead = await readEntryManifest(name, entry.source, readManifest); + if (!manifestRead.ok) return manifestRead.failure; + const { raw } = manifestRead; + + const contractPinFailure = checkContractPin(name, entry, raw); + if (contractPinFailure) return contractPinFailure; + + const manifestParsed = parseEntryManifest(name, raw); + if (!manifestParsed.ok) return manifestParsed.failure; + const { manifest } = manifestParsed; + + const identityFailure = checkHostIdentity(name, manifest, builtins); + if (identityFailure) return identityFailure; + + const integrityComputed = computeEntryIntegrity(name, manifest, entry.source); + if (!integrityComputed.ok) return integrityComputed.failure; + const { integrity } = integrityComputed; + const { hash } = integrity; + + const consentFailure = checkConsent(name, hash, consent); + if (consentFailure) return consentFailure; return { name, admitted: true, entry: manifest.host, manifest, integrity, contentHash: hash }; } @@ -133,6 +183,234 @@ async function defaultReadManifest(source) { return raw; } +// ── bootstrapHostAdapters decomposition ────────────────────────────────── +// Each helper below is one guarded, non-fatal stage of the bootstrap +// sequence (ADR-0031 §1/P2/P3) — extracted verbatim from the orchestrator's +// body so the orchestrator itself just sequences them, and each stage's own +// (independently small) complexity stays with the stage. `warnings` is a +// shared, mutated-in-place array throughout, matching the pre-decomposition +// code's own accumulation pattern. + +/** The AQE provider bridge is an exact snapshot of THIS bootstrap pass — + * cleared before every flag-on early return as well as before rebuilding, + * so removing the final adapter or losing access to the consent store + * never leaves a provider from a prior in-process bootstrap live. */ +async function resetAqeProviderBridge() { + try { + const bridge = await import('./aqe-provider.mjs'); + bridge.resetAdmittedAqeProviders(); + return bridge; + } catch { + return null; + } +} + +/** + * Resolve the consent store this bootstrap pass will use: the caller's + * injection (tests) or a dynamic import of the sibling consent.mjs. + * Returns `{ok:true, consentStore}` on success, or `{ok:false, result}` + * carrying bootstrapHostAdapters' own early-return shape (every entry + * warned 'consent-unavailable') when consent.mjs itself can't be loaded — + * never fatal. + */ +async function resolveConsentStore(consent, entries) { + if (consent) return { ok: true, consentStore: consent }; + try { + // Sibling-owned module (consent.mjs); dynamic so this file loads even + // before it lands, and so tests never pay for it unless they choose to. + // consent.mjs exports recordedHashFor/isTrusted as plain named + // functions (no wrapper object) — the module namespace itself already + // satisfies {recordedHashFor(name), isTrusted(name,hash)}; the + // consentStore/default fallbacks are just tolerance for a future + // reshape, not the shape it ships today. + const mod = /** @type {any} */ (await import('./consent.mjs')); + return { ok: true, consentStore: mod.consentStore ?? mod.default ?? mod }; + } catch (error) { + const warnings = entries.map((raw) => ({ + name: raw?.name ?? '(unknown)', reason: 'consent-unavailable', detail: error?.message ?? String(error), + })); + return { ok: false, result: { active: true, admitted: [], warnings } }; + } +} + +/** + * Keystone (ADR-0031 §1): build the per-host granted-capability lookup + * BEFORE applying the overlay, so an earned canBePrimary/commandStatusline + * is live in effectiveHostRegistry() from process start. Guarded and + * non-fatal — one host's grant lookup failing must never block another + * host's, or the admission result itself. + * + * CRITICAL: the hash passed to grantedCapabilitiesFor is the content + * identity produced by admission, not a manifest-only fallback. It pins + * both the validated manifest and any declared hook bytes, so a file edit + * cannot leave a capability grant live under the old content hash. + */ +async function buildGrantsByName(admitted, warnings) { + try { + const { grantedCapabilitiesFor } = await import('./grants.mjs'); + // Object.create(null), not {} (F-6): admitted host ids come from + // consented adapter names, which are attacker-influenceable in + // principle — a plain object literal's prototype chain would make + // 'constructor' a live (if inert) key collision. No inherited + // properties at all closes that off entirely; admitted.mjs's own + // Object.hasOwn guard on the read side is the belt to this suspenders. + const grantsByName = Object.create(null); + for (const result of admitted) { + try { + grantsByName[result.name] = grantedCapabilitiesFor(result.name, result.contentHash ?? hashManifest(result.manifest)); + } catch (error) { + warnings.push({ name: result.name, reason: 'grant-lookup-failed', detail: error?.message ?? String(error) }); + } + } + return { grantsByName, grantedCapabilitiesForCurrentHash: grantedCapabilitiesFor }; + } catch { + // grants.mjs unavailable — proceed ungranted (manifest-floor + // capabilities only), exactly like the flag-off path. Never fatal. + return { grantsByName: undefined, grantedCapabilitiesForCurrentHash: undefined }; + } +} + +/** + * AQE ADR-127 / issue #628: a manifest's aqe.provider block is only a + * candidate. It becomes a live trampoline target after ALL local gates: + * admission, explicit host enablement, and a hash-current aqeProvider + * grant. The dedicated registry keeps this non-boolean identity out of + * applyAdmitted's deliberately narrow host-capability overlay. One bad + * provider is isolated to one warning. + */ +async function registerAqeProviderCandidates({ + aqeProviderBridge, admitted, cfg, grantsByName, grantedCapabilitiesForCurrentHash, sourceByName, consentStore, + currentConfig, warnings, +}) { + if (!aqeProviderBridge) return; + const aqeCandidates = admitted.filter((result) => ( + !!result.manifest?.aqe?.provider + && cfg.integrations?.hosts?.[result.name] === true + && grantsByName && Object.hasOwn(grantsByName, result.name) + && grantsByName[result.name]?.aqeProvider === true + )); + for (const result of aqeCandidates) { + try { + aqeProviderBridge.registerAdmittedAqeProvider(result.manifest, { + baseDir: baseDirForSource(sourceByName.get(result.name)), + integrity: result.integrity, + contentHash: result.contentHash, + // Bootstrap admission is a snapshot; execution authority is not. + // The hidden AQE trampoline may wait on stdin while another + // process revokes consent/grant or disables the host. Re-read all + // three gates after prompt collection and snapshot capture, + // immediately before spawn. Any read/shape failure denies. + authorize: async () => { + try { + const liveCfg = currentConfig + ? await currentConfig() + : (await import('../config.mjs')).loadKitConfig(); + if (liveCfg?.integrations?.hosts?.[result.name] !== true) return false; + const liveHash = result.contentHash; + if (consentStore.recordedHashFor(result.name) !== liveHash + || !consentStore.isTrusted(result.name, liveHash)) return false; + return grantedCapabilitiesForCurrentHash(result.name, liveHash)?.aqeProvider === true; + } catch { + return false; + } + }, + }); + } catch (error) { + warnings.push({ + name: result.name, + reason: error?.reason ?? 'aqe-provider-registration-failed', + detail: error?.message ?? String(error), + }); + } + } +} + +/** + * P2 (ADR-0031): an admitted manifest declaring both an execution block and + * host.capabilities.canRouteActivities gets its execution adapter derived + * and registered here, so `ak run` can route to it. Same guarded, + * non-fatal posture as the rest of bootstrap: one adapter's registration + * failure never blocks the others or the admission result. + */ +async function registerExecutionCandidates(admitted, sourceByName, warnings) { + const executionCandidates = admitted.filter((result) => ( + result.manifest?.execution && result.entry?.capabilities?.canRouteActivities === true + )); + if (!executionCandidates.length) return; + try { + const { registerAdmittedExecution } = await import('../execution/admitted.mjs'); + for (const result of executionCandidates) { + // F-5 (ADR-0029 §2): a manifest that never declared the + // cli-subprocess driving surface gets no cli-subprocess execution + // adapter — refused with its own reason, before even attempting + // registration (buildAdmittedExecutionAdapter re-checks this too, + // defence-in-depth for any caller that bypasses this filter). + if (!result.manifest?.driving?.surfaces?.includes('cli-subprocess')) { + warnings.push({ + name: result.name, reason: 'surface-unsupported', + detail: `'${result.name}' declares an execution block but not driving.surfaces including 'cli-subprocess'`, + }); + continue; + } + try { + registerAdmittedExecution(result.manifest, { + baseDir: baseDirForSource(sourceByName.get(result.name)), integrity: result.integrity, + }); + } catch (error) { + warnings.push({ name: result.name, reason: error?.reason ?? 'execution-registration-failed', detail: error?.message ?? String(error) }); + } + } + } catch (error) { + for (const result of executionCandidates) { + warnings.push({ name: result.name, reason: 'execution-registration-failed', detail: error?.message ?? String(error) }); + } + } +} + +/** + * P3 (ADR-0031): an admitted manifest declaring a lifecycle block gets its + * derived lifecycle adapter registered here, so it appears in + * hostsWithLifecycle() and setup/sync/uninstall's lifecycle loops can drive + * it (gated per-run by lifecycleExecutionEnabled — registration alone + * never runs a hook). Same guarded, non-fatal posture as the + * execution-registration stage above: one adapter's registration failure + * never blocks the others or the admission result. F-1 (same as execution): + * baseDir anchors a relative lifecycle hook command to the adapter's own + * directory — without it, a relative command would resolve against the + * OPERATOR's cwd, arbitrary-code-execution with the consent hash unchanged. + * Unlike execution (one hook, all-or-nothing), lifecycle has five + * independently-optional verbs, so an unanchorable one is refused per-verb + * (buildAdmittedLifecycleAdapter never wires it to spawn) rather than + * failing the whole registration — the other, anchored/PATH-binary verbs + * still register and work. + */ +async function registerLifecycleCandidates(admitted, sourceByName, warnings) { + const lifecycleCandidates = admitted.filter((result) => !!result.manifest?.lifecycle); + if (!lifecycleCandidates.length) return; + try { + const { registerAdmittedLifecycle } = await import('./lifecycle-registry.mjs'); + for (const result of lifecycleCandidates) { + try { + const baseDir = baseDirForSource(sourceByName.get(result.name)); + const adapter = registerAdmittedLifecycle(result.manifest, { baseDir, integrity: result.integrity }); + if (adapter.unanchoredVerbs.length) { + warnings.push({ + name: result.name, reason: 'lifecycle-unanchored', + detail: `'${result.name}' lifecycle hook(s) refused (relative command, no anchored adapter ` + + `base directory): ${adapter.unanchoredVerbs.join(', ')}`, + }); + } + } catch (error) { + warnings.push({ name: result.name, reason: error?.reason ?? 'lifecycle-registration-failed', detail: error?.message ?? String(error) }); + } + } + } catch (error) { + for (const result of lifecycleCandidates) { + warnings.push({ name: result.name, reason: 'lifecycle-registration-failed', detail: error?.message ?? String(error) }); + } + } +} + /** * CLI bootstrap entry point — the only call site production code needs * (`bootstrapHostAdapters({cfg, env})`, wired from bin/agentic-kit.mjs). @@ -151,45 +429,14 @@ export async function bootstrapHostAdapters({ } = {}) { if (env?.AK_EXPERIMENTAL_HOST_ADAPTERS !== '1') return { active: false, admitted: [], warnings: [] }; - // The AQE provider bridge is an exact snapshot of THIS bootstrap pass. - // Clear it before every flag-on early return as well as before rebuilding: - // removing the final adapter or losing access to the consent store must not - // leave a provider from a prior in-process bootstrap live. Flag-off remains - // a true zero-import no-op above. - let aqeProviderBridge; - try { - aqeProviderBridge = await import('./aqe-provider.mjs'); - aqeProviderBridge.resetAdmittedAqeProviders(); - } catch { - aqeProviderBridge = null; - } + const aqeProviderBridge = await resetAqeProviderBridge(); const entries = Array.isArray(cfg?.hostAdapters) ? cfg.hostAdapters : []; if (entries.length === 0) return { active: false, admitted: [], warnings: [] }; - let consentStore = consent; - if (!consentStore) { - try { - // Sibling-owned module (consent.mjs); dynamic so this file loads even - // before it lands, and so tests never pay for it unless they choose to. - // consent.mjs exports recordedHashFor/isTrusted as plain named - // functions (no wrapper object) — the module namespace itself already - // satisfies {recordedHashFor(name), isTrusted(name,hash)}; the - // consentStore/default fallbacks are just tolerance for a future - // reshape, not the shape it ships today. - // Cast to `any`: consent.mjs's real, current shape has neither - // `consentStore` nor `default` — the fallback chain below is tolerance - // for a future reshape (see the comment above), not today's actual - // module namespace type, so a literal type there would just be wrong. - const mod = /** @type {any} */ (await import('./consent.mjs')); - consentStore = mod.consentStore ?? mod.default ?? mod; - } catch (error) { - const warnings = entries.map((raw) => ({ - name: raw?.name ?? '(unknown)', reason: 'consent-unavailable', detail: error?.message ?? String(error), - })); - return { active: true, admitted: [], warnings }; - } - } + const consentResolution = await resolveConsentStore(consent, entries); + if (!consentResolution.ok) return consentResolution.result; + const { consentStore } = consentResolution; const results = await admitAdapters({ cfg, readManifest, consent: consentStore }); const admitted = results.filter((result) => result.admitted); @@ -198,182 +445,22 @@ export async function bootstrapHostAdapters({ if (admitted.length) { const { applyAdmitted } = await import('./admitted.mjs'); - - // Keystone (ADR-0031 §1): build the per-host granted-capability lookup - // BEFORE applying the overlay, so an earned canBePrimary/commandStatusline - // is live in effectiveHostRegistry() from process start. Guarded and - // non-fatal, lazy import — the same posture as the execution/lifecycle - // registration blocks below, and a NEW sibling concern to them (it does - // not touch either). One host's grant lookup failing must never block - // another host's, or the admission result itself. - // - // CRITICAL: the hash passed to grantedCapabilitiesFor is the content - // identity produced by admission, not a manifest-only fallback. It pins - // both the validated manifest and any declared hook bytes, so a file edit - // cannot leave a capability grant live under the old content hash. - let grantsByName; - let grantedCapabilitiesForCurrentHash; - try { - const { grantedCapabilitiesFor } = await import('./grants.mjs'); - grantedCapabilitiesForCurrentHash = grantedCapabilitiesFor; - // Object.create(null), not {} (F-6): admitted host ids come from - // consented adapter names, which are attacker-influenceable in - // principle — a plain object literal's prototype chain would make - // 'constructor' a live (if inert) key collision. No inherited - // properties at all closes that off entirely; admitted.mjs's own - // Object.hasOwn guard on the read side is the belt to this suspenders. - grantsByName = Object.create(null); - for (const result of admitted) { - try { - grantsByName[result.name] = grantedCapabilitiesFor(result.name, result.contentHash ?? hashManifest(result.manifest)); - } catch (error) { - warnings.push({ name: result.name, reason: 'grant-lookup-failed', detail: error?.message ?? String(error) }); - } - } - } catch { - // grants.mjs unavailable — proceed ungranted (manifest-floor - // capabilities only), exactly like the flag-off path. Never fatal. - grantsByName = undefined; - } - + const { grantsByName, grantedCapabilitiesForCurrentHash } = await buildGrantsByName(admitted, warnings); applyAdmitted(admitted, { grantsByName }); // name -> the cfg entry's own declared source, for F-1's baseDir // derivation below (admitted results carry the validated manifest, not - // the raw cfg entry that named where it came from). Shared by both the - // execution- and lifecycle-registration blocks below — one map, not a - // second copy — so a caller correcting F-1 in one place can't drift from - // the other. + // the raw cfg entry that named where it came from). Shared by every + // registration stage below — one map, not a second copy — so a caller + // correcting F-1 in one place can't drift from the others. const sourceByName = new Map(entries.map((entry) => [entry?.name, entry?.source])); - // AQE ADR-127 / issue #628: a manifest's aqe.provider block is only a - // candidate. It becomes a live trampoline target after ALL local gates: - // admission above, explicit host enablement, and a hash-current - // aqeProvider grant. The dedicated registry keeps this non-boolean - // identity out of applyAdmitted's deliberately narrow host-capability - // overlay. One bad provider is isolated to one warning. - if (aqeProviderBridge) { - const aqeCandidates = admitted.filter((result) => ( - !!result.manifest?.aqe?.provider - && cfg.integrations?.hosts?.[result.name] === true - && grantsByName && Object.hasOwn(grantsByName, result.name) - && grantsByName[result.name]?.aqeProvider === true - )); - for (const result of aqeCandidates) { - try { - aqeProviderBridge.registerAdmittedAqeProvider(result.manifest, { - baseDir: baseDirForSource(sourceByName.get(result.name)), - integrity: result.integrity, - contentHash: result.contentHash, - // Bootstrap admission is a snapshot; execution authority is not. - // The hidden AQE trampoline may wait on stdin while another - // process revokes consent/grant or disables the host. Re-read all - // three gates after prompt collection and snapshot capture, - // immediately before spawn. Any read/shape failure denies. - authorize: async () => { - try { - const liveCfg = currentConfig - ? await currentConfig() - : (await import('../config.mjs')).loadKitConfig(); - if (liveCfg?.integrations?.hosts?.[result.name] !== true) return false; - const liveHash = result.contentHash; - if (consentStore.recordedHashFor(result.name) !== liveHash - || !consentStore.isTrusted(result.name, liveHash)) return false; - return grantedCapabilitiesForCurrentHash(result.name, liveHash)?.aqeProvider === true; - } catch { - return false; - } - }, - }); - } catch (error) { - warnings.push({ - name: result.name, - reason: error?.reason ?? 'aqe-provider-registration-failed', - detail: error?.message ?? String(error), - }); - } - } - } - - // P2 (ADR-0031): an admitted manifest declaring both an execution block - // and host.capabilities.canRouteActivities gets its execution adapter - // derived and registered here, so `ak run` can route to it. Same - // guarded, non-fatal posture as the rest of bootstrap: one adapter's - // registration failure never blocks the others or the admission result. - const executionCandidates = admitted.filter((result) => ( - result.manifest?.execution && result.entry?.capabilities?.canRouteActivities === true - )); - if (executionCandidates.length) { - try { - const { registerAdmittedExecution } = await import('../execution/admitted.mjs'); - for (const result of executionCandidates) { - // F-5 (ADR-0029 §2): a manifest that never declared the - // cli-subprocess driving surface gets no cli-subprocess execution - // adapter — refused with its own reason, before even attempting - // registration (buildAdmittedExecutionAdapter re-checks this too, - // defence-in-depth for any caller that bypasses this filter). - if (!result.manifest?.driving?.surfaces?.includes('cli-subprocess')) { - warnings.push({ - name: result.name, reason: 'surface-unsupported', - detail: `'${result.name}' declares an execution block but not driving.surfaces including 'cli-subprocess'`, - }); - continue; - } - try { - registerAdmittedExecution(result.manifest, { - baseDir: baseDirForSource(sourceByName.get(result.name)), integrity: result.integrity, - }); - } catch (error) { - warnings.push({ name: result.name, reason: error?.reason ?? 'execution-registration-failed', detail: error?.message ?? String(error) }); - } - } - } catch (error) { - for (const result of executionCandidates) { - warnings.push({ name: result.name, reason: 'execution-registration-failed', detail: error?.message ?? String(error) }); - } - } - } - - // P3 (ADR-0031): an admitted manifest declaring a lifecycle block gets its - // derived lifecycle adapter registered here, so it appears in - // hostsWithLifecycle() and setup/sync/uninstall's lifecycle loops can - // drive it (gated per-run by lifecycleExecutionEnabled — registration - // alone never runs a hook). Same guarded, non-fatal posture as the - // execution-registration block above: one adapter's registration failure - // never blocks the others or the admission result. F-1 (same as - // execution above): baseDir anchors a relative lifecycle hook command to - // the adapter's own directory — without it, a relative command would - // resolve against the OPERATOR's cwd, arbitrary-code-execution with the - // consent hash unchanged. Unlike execution (one hook, all-or-nothing), - // lifecycle has five independently-optional verbs, so an unanchorable - // one is refused per-verb (buildAdmittedLifecycleAdapter never wires it - // to spawn) rather than failing the whole registration — the other, - // anchored/PATH-binary verbs still register and work. - const lifecycleCandidates = admitted.filter((result) => !!result.manifest?.lifecycle); - if (lifecycleCandidates.length) { - try { - const { registerAdmittedLifecycle } = await import('./lifecycle-registry.mjs'); - for (const result of lifecycleCandidates) { - try { - const baseDir = baseDirForSource(sourceByName.get(result.name)); - const adapter = registerAdmittedLifecycle(result.manifest, { baseDir, integrity: result.integrity }); - if (adapter.unanchoredVerbs.length) { - warnings.push({ - name: result.name, reason: 'lifecycle-unanchored', - detail: `'${result.name}' lifecycle hook(s) refused (relative command, no anchored adapter ` - + `base directory): ${adapter.unanchoredVerbs.join(', ')}`, - }); - } - } catch (error) { - warnings.push({ name: result.name, reason: error?.reason ?? 'lifecycle-registration-failed', detail: error?.message ?? String(error) }); - } - } - } catch (error) { - for (const result of lifecycleCandidates) { - warnings.push({ name: result.name, reason: 'lifecycle-registration-failed', detail: error?.message ?? String(error) }); - } - } - } + await registerAqeProviderCandidates({ + aqeProviderBridge, admitted, cfg, grantsByName, grantedCapabilitiesForCurrentHash, sourceByName, consentStore, + currentConfig, warnings, + }); + await registerExecutionCandidates(admitted, sourceByName, warnings); + await registerLifecycleCandidates(admitted, sourceByName, warnings); } return { active: true, admitted, warnings }; diff --git a/src/lib/adapters/config.mjs b/src/lib/adapters/config.mjs index 0dc6c69..010e420 100644 --- a/src/lib/adapters/config.mjs +++ b/src/lib/adapters/config.mjs @@ -104,39 +104,48 @@ export function validateEndpoint(endpoint) { return { ok: true, normalized: endpoint }; } -export function migrateIntegrationConfig(config = {}, _options = {}) { - const out = structuredClone(plain(config) ? config : {}); - if (out.integrations !== undefined && !plain(out.integrations)) return immutable(out); - const existing = out.integrations ?? {}; - if (Number.isInteger(existing.version) && existing.version > CURRENT_INTEGRATIONS_VERSION) { - return immutable(out); - } - if (Object.hasOwn(existing, 'bindings') && !Array.isArray(existing.bindings)) { - return immutable(out); - } - if (Object.hasOwn(existing, 'hosts') && !plain(existing.hosts)) return immutable(out); - if (Object.hasOwn(existing, 'tools') && !plain(existing.tools)) return immutable(out); - if (Object.hasOwn(existing, 'ownership') && !plain(existing.ownership)) return immutable(out); +// ── migrateIntegrationConfig decomposition ─────────────────────────────── +// Each helper reproduces one slice of the original sequential body +// verbatim, in the same order, so migrateIntegrationConfig itself just +// sequences them. The shape-guard checks all produce the identical +// `immutable(out)` bail-out, so they're grouped into two predicate +// functions (existing-shaped vs providers-shaped) rather than a +// (input)->error|null rule list — nothing here is a distinct error to +// report, just "is this still safely migratable". - const providers = plain(out.providers) ? out.providers : {}; - if (own(providers, 'hosts') && !plain(providers.hosts)) return immutable(out); - if (own(providers, 'bindings') && !Array.isArray(providers.bindings)) return immutable(out); +/** Any of these existing.* shape violations means migration must leave + * `out` untouched (returned as-is) rather than guess at a shape it can't + * safely interpret. */ +function hasUnmigratableIntegrationsShape(existing) { + if (Number.isInteger(existing.version) && existing.version > CURRENT_INTEGRATIONS_VERSION) return true; + if (Object.hasOwn(existing, 'bindings') && !Array.isArray(existing.bindings)) return true; + if (Object.hasOwn(existing, 'hosts') && !plain(existing.hosts)) return true; + if (Object.hasOwn(existing, 'tools') && !plain(existing.tools)) return true; + if (Object.hasOwn(existing, 'ownership') && !plain(existing.ownership)) return true; + return false; +} + +/** Same bail-out discipline as hasUnmigratableIntegrationsShape, for the + * legacy providers.* fields this migration also reads. */ +function hasUnmigratableProvidersShape(providers) { + if (own(providers, 'hosts') && !plain(providers.hosts)) return true; + if (own(providers, 'bindings') && !Array.isArray(providers.bindings)) return true; + return false; +} - // The alpha writer's host set deliberately wins once at cutover over an - // older additive integrations.hosts snapshot. - const hosts = own(providers, 'hosts') +/** The alpha writer's host set deliberately wins once at cutover over an + * older additive integrations.hosts snapshot. */ +function resolveMigratedHosts(providers, existing) { + return own(providers, 'hosts') ? structuredClone(providers.hosts) : structuredClone(existing.hosts ?? {}); - const enabled = Object.entries(hosts).filter(([, on]) => on).map(([host]) => host); - const priorBindings = mergeBindings( - Array.isArray(existing.bindings) ? existing.bindings : [], - Array.isArray(providers.bindings) ? providers.bindings : [], - ); - const seenHosts = new Set(priorBindings.map((binding) => binding.host)); - // A host with no registered native provider (F-13) gets no inferred - // binding at all — nothing is restamped on any load, and it stays that - // way until a real binding is declared for it. - const inferred = enabled +} + +/** A host with no registered native provider (F-13) gets no inferred + * binding at all — nothing is restamped on any load, and it stays that + * way until a real binding is declared for it. */ +function deriveInferredBindings(enabled, seenHosts) { + return enabled .filter((host) => !seenHosts.has(host)) .map((host) => { const provider = NATIVE_PROVIDER_BY_HOST.get(host); @@ -152,39 +161,85 @@ export function migrateIntegrationConfig(config = {}, _options = {}) { } : null; }) .filter((binding) => binding !== null); - const ownership = structuredClone(existing.ownership ?? {}); +} + +function resolveMigratedTools(existing) { const tools = structuredClone(existing.tools ?? {}); tools.dejaVu = plain(tools.dejaVu) ? { ...structuredClone(DEFAULT_DEJA_VU_INTENT), ...tools.dejaVu } : Object.hasOwn(tools, 'dejaVu') ? tools.dejaVu : structuredClone(DEFAULT_DEJA_VU_INTENT); - const reverseMarker = 'rufloCodexMcp'; + return tools; +} + +/** Legacy providers.codexMcp / providers[reverseMarker] -> integrations. + * ownership.codex, preserving whatever the new home doesn't already own. + * Mutates `ownership` in place, and only when a legacy field is actually + * present — never invents an empty codex ownership record. */ +function applyLegacyCodexOwnership(ownership, providers, reverseMarker) { const hasLegacyCodex = (own(providers, 'codexMcp') && providers.codexMcp != null) || (own(providers, reverseMarker) && providers[reverseMarker] != null); - if (hasLegacyCodex) { - ownership.codex = { - source: 'legacy-providers', - ...(plain(ownership.codex) ? ownership.codex : {}), - ...(own(providers, 'codexMcp') ? { mcp: structuredClone(providers.codexMcp) } : {}), - ...(own(providers, reverseMarker) - ? { reverseMcp: structuredClone(providers[reverseMarker]) } : {}), - }; - } + if (!hasLegacyCodex) return; + ownership.codex = { + source: 'legacy-providers', + ...(plain(ownership.codex) ? ownership.codex : {}), + ...(own(providers, 'codexMcp') ? { mcp: structuredClone(providers.codexMcp) } : {}), + ...(own(providers, reverseMarker) + ? { reverseMcp: structuredClone(providers[reverseMarker]) } : {}), + }; +} + +/** Same discipline as applyLegacyCodexOwnership, for the three legacy + * OpenCode provider fields. */ +function applyLegacyOpenCodeOwnership(ownership, providers) { const hasLegacyOpenCode = ['opencodeMcp', 'opencodeManaged', 'opencodeCatalogDir'] .some((key) => own(providers, key) && providers[key] != null); - if (hasLegacyOpenCode) { - ownership.opencode = { - source: 'legacy-providers', - ...(plain(ownership.opencode) ? ownership.opencode : {}), - ...(own(providers, 'opencodeMcp') - ? { mcp: structuredClone(providers.opencodeMcp) } : {}), - ...(own(providers, 'opencodeManaged') - ? { managed: structuredClone(providers.opencodeManaged) } : {}), - ...(own(providers, 'opencodeCatalogDir') - ? { catalogDir: structuredClone(providers.opencodeCatalogDir) } : {}), - }; - } + if (!hasLegacyOpenCode) return; + ownership.opencode = { + source: 'legacy-providers', + ...(plain(ownership.opencode) ? ownership.opencode : {}), + ...(own(providers, 'opencodeMcp') + ? { mcp: structuredClone(providers.opencodeMcp) } : {}), + ...(own(providers, 'opencodeManaged') + ? { managed: structuredClone(providers.opencodeManaged) } : {}), + ...(own(providers, 'opencodeCatalogDir') + ? { catalogDir: structuredClone(providers.opencodeCatalogDir) } : {}), + }; +} + +function pruneLegacyProviderKeys(out, reverseMarker) { + if (!plain(out.providers)) return; + for (const key of [ + 'hosts', 'bindings', 'codexMcp', reverseMarker, + 'opencodeMcp', 'opencodeManaged', 'opencodeCatalogDir', + ]) delete out.providers[key]; +} + +export function migrateIntegrationConfig(config = {}, _options = {}) { + const out = structuredClone(plain(config) ? config : {}); + if (out.integrations !== undefined && !plain(out.integrations)) return immutable(out); + const existing = out.integrations ?? {}; + if (hasUnmigratableIntegrationsShape(existing)) return immutable(out); + + const providers = plain(out.providers) ? out.providers : {}; + if (hasUnmigratableProvidersShape(providers)) return immutable(out); + + const hosts = resolveMigratedHosts(providers, existing); + const enabled = Object.entries(hosts).filter(([, on]) => on).map(([host]) => host); + const priorBindings = mergeBindings( + Array.isArray(existing.bindings) ? existing.bindings : [], + Array.isArray(providers.bindings) ? providers.bindings : [], + ); + const seenHosts = new Set(priorBindings.map((binding) => binding.host)); + const inferred = deriveInferredBindings(enabled, seenHosts); + + const ownership = structuredClone(existing.ownership ?? {}); + const tools = resolveMigratedTools(existing); + const reverseMarker = 'rufloCodexMcp'; + applyLegacyCodexOwnership(ownership, providers, reverseMarker); + applyLegacyOpenCodeOwnership(ownership, providers); + out.integrations = { ...existing, version: CURRENT_INTEGRATIONS_VERSION, @@ -194,11 +249,6 @@ export function migrateIntegrationConfig(config = {}, _options = {}) { ...(Object.keys(ownership).length ? { ownership } : {}), }; delete out.integrations.schemaVersion; - if (plain(out.providers)) { - for (const key of [ - 'hosts', 'bindings', 'codexMcp', reverseMarker, - 'opencodeMcp', 'opencodeManaged', 'opencodeCatalogDir', - ]) delete out.providers[key]; - } + pruneLegacyProviderKeys(out, reverseMarker); return immutable(out); } diff --git a/src/lib/adapters/conformance.mjs b/src/lib/adapters/conformance.mjs index fcab9a8..b8e8632 100644 --- a/src/lib/adapters/conformance.mjs +++ b/src/lib/adapters/conformance.mjs @@ -503,6 +503,188 @@ async function checkGrantGatedTier({ }; } +// ── runTieredConformance decomposition helpers ────────────────────────── +// The orchestrator below evaluates CONFORMANCE_TIERS' six tiers in their +// fixed graduation order (ADR-0031 §2) regardless of the caller's own +// `tiers` array ordering — matching the original inline if-per-tier +// sequence. These helpers exist purely to keep the orchestrator's own +// cyclomatic complexity low: each closure it drives is one line, and the +// actual branching lives in a small, independently-testable function. + +/** True when any of `ids` is present in the caller's requested `tiers`. */ +function wantsAnyOf(tiers, ids) { + return ids.some((id) => tiers.includes(id)); +} + +/** F1 (Wave C): baseDir must be resolved before checkAdmission so it can + * thread into registerAdmittedLifecycle for the detect-hook check — the + * same anchoring the execution tier already needed. */ +function resolveBaseDir(baseDir, manifestSource) { + return baseDir !== undefined ? baseDir : baseDirForSource(manifestSource); +} + +/** + * F2 (Wave C, BLOCKER): every post-admission tier must gate on the WHOLE + * admission tier passing — not just manifest schema-validation succeeding — + * or a manifest that schema-validated but whose REAL admission + * (admitAdapters — the consent/contract/builtin-shadow gate) was refused + * (e.g. stale or missing consent) would still have its execution.run hook + * spawned for real and its result recorded 'passed' into grantsFile (which + * defaults to the operator's REAL adapter-grants.json). A manifest whose + * admission failed is treated identically to one that never validated at + * all, for every downstream tier. + */ +function resolveEffectiveManifest(admission) { + const admissionPassed = admission.checks.length > 0 && admission.checks.every((c) => c.ok); + return { admissionPassed, effectiveManifest: admissionPassed ? admission.manifest : null }; +} + +/** adrianco#131 #1: an explicit override always wins; otherwise honor the + * manifest's own declared execution.run.hook.timeoutMs (if any) as the + * OUTER runner budget too, so a manifest that already declares a longer one + * (e.g. a local-model host) isn't silently capped at the runner's 120s + * default before the hook's own tighter-of-the-two ever applies. */ +function resolveEffectiveTimeoutMs(timeoutMs, effectiveManifest) { + if (timeoutMs !== undefined) return timeoutMs; + return effectiveManifest?.execution?.run?.hook?.timeoutMs; +} + +/** + * primary-eligible and aqe-provider both require activity-routing to have + * genuinely passed THIS run — not merely to have been requested. Computed + * once, whenever either tier needs it, so a caller asking for + * `tiers: ['primary-eligible']` alone still gets the real dependency check + * rather than an unconditioned pass/skip. + */ +async function computeActivityRoutingResult(tiers, params) { + if (!wantsAnyOf(tiers, ['activity-routing', 'aqe-provider', 'primary-eligible'])) return null; + return checkActivityRouting(params); +} + +/** Reset every overlay a conformance run can touch (F7, Wave C security + * review: the host overlay, the execution overlay, and the lifecycle + * registration checkAdmission's detect-hook check creates via + * registerAdmittedLifecycle — all three, together, since resetting only the + * first two left an already-registered lifecycle adapter live for a host id + * this run's own admission may have just refused) and best-effort clean up + * any temp directories this run created. */ +function cleanupConformanceRun(tempDir, workerCwdTempDir) { + resetAdmitted(); + resetAdmittedExecution(); + resetAdmittedLifecycle(); + resetAdmittedAqeProviders(); + if (tempDir) { try { fs.rmSync(tempDir, { recursive: true, force: true }); } catch { /* best-effort cleanup */ } } + if (workerCwdTempDir) { try { fs.rmSync(workerCwdTempDir, { recursive: true, force: true }); } catch { /* best-effort cleanup */ } } +} + +/** + * Shared skip-gate for aqe-provider and primary-eligible: both require + * admission AND a genuinely-passed activity-routing result THIS RUN before + * their own real exercise can run. Identical shape for both tiers in the + * pre-decomposition code (two copy-pasted if/else-if/else blocks with + * matching detail strings) — factored out rather than duplicated. + */ +async function runActivityRoutingDependentTier(effectiveManifest, activityRoutingResult, exercise) { + if (!effectiveManifest) { + return { status: 'skipped', checks: [{ name: 'admission prerequisite', ok: false, detail: 'admission tier did not pass — cannot evaluate' }] }; + } + if (activityRoutingResult?.status !== 'passed') { + return { + status: 'skipped', + checks: [{ name: 'activity-routing prerequisite', ok: false, detail: 'activity-routing did not pass — cannot evaluate' }], + }; + } + return exercise(); +} + +/** Resolve a usable consent-store file, creating (and returning for cleanup) + * a throwaway temp one when the caller didn't supply their own. */ +function resolveConsentFile(consentFile) { + if (consentFile) return { consentFileUsed: consentFile, tempDir: null }; + const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ak-adapter-conformance-tiers-')); + return { consentFileUsed: path.join(tempDir, 'adapter-consent.json'), tempDir }; +} + +/** adrianco#131 #2: a live, often auto-approving worker must never land in + * the operator's own $PWD by accident — resolve (and return for cleanup) a + * throwaway scratch directory unless the caller supplies its own. */ +function resolveWorkerCwd(cwd) { + if (cwd) return { workerCwd: cwd, workerCwdTempDir: null }; + const workerCwdTempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ak-adapter-conformance-cwd-')); + return { workerCwd: workerCwdTempDir, workerCwdTempDir }; +} + +/** + * Persist each tier's outcome per ADR-0031 §1/§2 (see runTieredConformance's + * own doc comment for the full recording contract). Extracted verbatim from + * the orchestrator's body so it carries its own (small) CC budget instead of + * the orchestrator's. + */ +function persistTierResults(tierResults, { resolvedName, hash, grantsFile }) { + for (const tierResult of tierResults) { + try { + if (tierResult.status === 'passed') { + const evidence = tierResult.evidence ?? evidenceFromChecks(tierResult.checks) ?? tierResult.tier; + recordTierResult(resolvedName, tierResult.tier, { hash, evidence: evidence || tierResult.tier }, { file: grantsFile }); + } else if (tierResult.status === 'gated' && tierResult.gatedBy) { + recordTierGate(resolvedName, tierResult.tier, { hash, gatedBy: tierResult.gatedBy }, { file: grantsFile }); + } else if (tierResult.status === 'failed' && Object.hasOwn(TIER_GRANTS, tierResult.tier)) { + // N-1 (security-review follow-up): a grant-bearing tier that + // RE-RUNS 'failed' at the SAME (unchanged) hash means evidence a + // live capability rests on was just shown non-reproducible — void + // the stored tier and the capability together (grants.mjs's + // recordTierFailure), mirroring recordTierGate's downgrade for + // 'gated' immediately above. Deliberately NOT triggered by + // 'skipped': a skipped tier (e.g. primary-eligible + // short-circuiting because its own prerequisite — activity- + // routing — didn't run/pass THIS run) means the tier was never + // actually EVALUATED this run, which is ambiguous, not disproven + // — downgrading on an ambiguous non-evaluation would void a live + // capability on evidence that says nothing about whether it + // still holds. + recordTierFailure(resolvedName, tierResult.tier, { hash }, { file: grantsFile }); + } + } catch (error) { + tierResult.recordError = error?.message ?? String(error); + } + } +} + +/** + * Build the six tier evaluators as thin closures over this call's resolved + * context. Each closure is a single call/expression — the actual per-tier + * branching lives in checkAdmission/checkSessionDriving/checkActivityRouting/ + * checkAqeProvider/checkPrimaryEligible/checkGrantGatedTier (and the shared + * runActivityRoutingDependentTier gate above), never inline here. + */ +function buildTierRunners({ + admission, admissionPassed, effectiveManifest, activityRoutingResult, resolvedName, derivedBaseDir, + integrity, hash, haveFn, clock, workerCwd, effectiveTimeoutMs, sessionDrivingUpstreamRef, grantsFile, + exerciseStatusline, +}) { + return { + admission: () => ({ status: admissionPassed ? 'passed' : 'failed', checks: admission.checks }), + 'session-driving': () => checkSessionDriving({ manifest: effectiveManifest, upstreamRef: sessionDrivingUpstreamRef }), + 'activity-routing': () => activityRoutingResult, + 'aqe-provider': () => runActivityRoutingDependentTier(effectiveManifest, activityRoutingResult, () => checkAqeProvider({ + manifest: effectiveManifest, name: resolvedName, baseDir: derivedBaseDir, integrity, hash, cwd: workerCwd, + })), + 'primary-eligible': () => runActivityRoutingDependentTier(effectiveManifest, activityRoutingResult, () => checkPrimaryEligible({ + manifest: effectiveManifest, name: resolvedName, baseDir: derivedBaseDir, integrity, haveFn, clock, + cwd: workerCwd, timeoutMs: effectiveTimeoutMs, + })), + statusline: () => checkGrantGatedTier({ + capability: 'commandStatusline', + manifest: effectiveManifest, + name: resolvedName, + hash, + grantsFile, + exercise: exerciseStatusline, + exerciseLabel: 'renders and refreshes a command-backed footer', + }), + }; +} + /** * Run the ADR-0031 §2 tiered conformance sequence against one adapter * manifest and return a structured, per-tier report. Self-contained: builds @@ -595,192 +777,54 @@ export async function runTieredConformance({ throw new TypeError('runTieredConformance requires fixtureRoot or manifestSource'); } - let tempDir = null; - let consentFileUsed = consentFile; - if (!consentFileUsed) { - tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ak-adapter-conformance-tiers-')); - consentFileUsed = path.join(tempDir, 'adapter-consent.json'); - } - - // adrianco#131 #2: a live, often auto-approving worker must never land in - // the operator's own $PWD by accident — a throwaway scratch directory, - // never process.cwd(), unless a caller (tests) supplies its own. - let workerCwdTempDir = null; - let workerCwd = cwd; - if (!workerCwd) { - workerCwdTempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ak-adapter-conformance-cwd-')); - workerCwd = workerCwdTempDir; - } + const { consentFileUsed, tempDir } = resolveConsentFile(consentFile); + const { workerCwd, workerCwdTempDir } = resolveWorkerCwd(cwd); try { - // F1 (Wave C): baseDir is computed BEFORE checkAdmission so it can thread - // straight into registerAdmittedLifecycle for the detect-hook check — - // the same anchoring the execution tier already needed, just derived - // earlier now that admission needs it too. - const derivedBaseDir = baseDir !== undefined ? baseDir : baseDirForSource(manifestSource); + const derivedBaseDir = resolveBaseDir(baseDir, manifestSource); const admission = await checkAdmission({ name, source: manifestSource, readManifest, consentFile: consentFileUsed, baseDir: derivedBaseDir, }); const resolvedName = name ?? admission.manifest?.host?.id ?? '(unknown)'; const { hash, integrity } = admission; - // F2 (Wave C, BLOCKER): every post-admission tier gated on manifest - // validity alone (`admission.manifest != null`) would still exercise a - // manifest that schema-validated but whose REAL admission (admitAdapters - // — the consent/contract/builtin-shadow gate) was refused: e.g. stale or - // missing consent. That let a refused adapter's execution.run hook be - // spawned for real and its result recorded 'passed' into grantsFile - // (which defaults to the operator's REAL adapter-grants.json). Gating on - // the WHOLE admission tier passing — not just the manifest parsing — - // closes that: a manifest whose admission failed is treated identically - // to one that never validated at all, for every downstream tier. - const admissionPassed = admission.checks.length > 0 && admission.checks.every((c) => c.ok); - const effectiveManifest = admissionPassed ? admission.manifest : null; + const { admissionPassed, effectiveManifest } = resolveEffectiveManifest(admission); const wantTier = (tier) => tiers.includes(tier); + const effectiveTimeoutMs = resolveEffectiveTimeoutMs(timeoutMs, effectiveManifest); - // adrianco#131 #1: an explicit override always wins; otherwise honor the - // manifest's own declared execution.run.hook.timeoutMs (if any) as the - // OUTER runner budget too, so a manifest that already declares a longer - // one (e.g. a local-model host) isn't silently capped at the runner's - // 120s default before the hook's own tighter-of-the-two ever applies. - const effectiveTimeoutMs = timeoutMs !== undefined - ? timeoutMs - : effectiveManifest?.execution?.run?.hook?.timeoutMs; - - const tierResults = []; - - if (wantTier('admission')) { - tierResults.push({ - tier: 'admission', - status: admissionPassed ? 'passed' : 'failed', - checks: admission.checks, - }); - } - - if (wantTier('session-driving')) { - tierResults.push({ tier: 'session-driving', ...checkSessionDriving({ manifest: effectiveManifest, upstreamRef: sessionDrivingUpstreamRef }) }); - } - - // primary-eligible and aqe-provider both require activity-routing to have - // genuinely passed THIS run — not merely to have been requested. - // Computed here, once, whenever either tier needs it, so a caller asking - // for `tiers: ['primary-eligible']` alone still gets the real dependency - // check rather than an unconditioned pass/skip. - let activityRoutingResult = null; - if (wantTier('activity-routing') || wantTier('aqe-provider') || wantTier('primary-eligible')) { - activityRoutingResult = await checkActivityRouting({ - manifest: effectiveManifest, name: resolvedName, baseDir: derivedBaseDir, integrity, haveFn, clock, - cwd: workerCwd, timeoutMs: effectiveTimeoutMs, - }); - if (wantTier('activity-routing')) { - tierResults.push({ tier: 'activity-routing', ...activityRoutingResult }); - } - } - - if (wantTier('aqe-provider')) { - let result; - if (!effectiveManifest) { - result = { status: 'skipped', checks: [{ name: 'admission prerequisite', ok: false, detail: 'admission tier did not pass — cannot evaluate' }] }; - } else if (activityRoutingResult?.status !== 'passed') { - result = { - status: 'skipped', - checks: [{ name: 'activity-routing prerequisite', ok: false, detail: 'activity-routing did not pass — cannot evaluate' }], - }; - } else { - result = await checkAqeProvider({ - manifest: effectiveManifest, name: resolvedName, baseDir: derivedBaseDir, - integrity, hash, cwd: workerCwd, - }); - } - tierResults.push({ tier: 'aqe-provider', ...result }); - } + const activityRoutingResult = await computeActivityRoutingResult(tiers, { + manifest: effectiveManifest, name: resolvedName, baseDir: derivedBaseDir, integrity, haveFn, clock, + cwd: workerCwd, timeoutMs: effectiveTimeoutMs, + }); - if (wantTier('primary-eligible')) { - // F-1 (security review): NOT checkGrantGatedTier — that gates the - // exercise on an already-existing canBePrimary grant, which deadlocks - // against grantCapability's own requirement of an already-'passed' - // tier (see checkPrimaryEligible's header comment). checkPrimaryEligible - // runs the real exercise unconditionally once its own prerequisites - // (admission, activity-routing) are met — evidence first, grant second, - // per ADR-0031 §1/§2. - let result; - if (!effectiveManifest) { - result = { status: 'skipped', checks: [{ name: 'admission prerequisite', ok: false, detail: 'admission tier did not pass — cannot evaluate' }] }; - } else if (activityRoutingResult?.status !== 'passed') { - // Mirrors the admission short-circuit above: when admission passed - // but activity-routing (this tier's own real prerequisite) did not, - // report the SAME 'skipped' shape with a distinct reason, rather - // than attempting an exercise the host cannot actually support yet. - result = { - status: 'skipped', - checks: [{ name: 'activity-routing prerequisite', ok: false, detail: 'activity-routing did not pass — cannot evaluate' }], - }; - } else { - result = await checkPrimaryEligible({ - manifest: effectiveManifest, name: resolvedName, baseDir: derivedBaseDir, integrity, haveFn, clock, - cwd: workerCwd, timeoutMs: effectiveTimeoutMs, - }); - } - tierResults.push({ tier: 'primary-eligible', ...result }); - } + // F-1 (security review): primary-eligible is NOT checkGrantGatedTier — + // that would gate the exercise on an already-existing canBePrimary + // grant, which deadlocks against grantCapability's own requirement of an + // already-'passed' tier (see checkPrimaryEligible's header comment). + // checkPrimaryEligible runs the real exercise unconditionally once its + // own prerequisites (admission, activity-routing) are met — evidence + // first, grant second, per ADR-0031 §1/§2. Both it and aqe-provider + // share the identical prerequisite short-circuit, factored into + // runActivityRoutingDependentTier above. + const tierRunners = buildTierRunners({ + admission, admissionPassed, effectiveManifest, activityRoutingResult, resolvedName, derivedBaseDir, + integrity, hash, haveFn, clock, workerCwd, effectiveTimeoutMs, sessionDrivingUpstreamRef, grantsFile, + exerciseStatusline, + }); - if (wantTier('statusline')) { - const result = await checkGrantGatedTier({ - capability: 'commandStatusline', - manifest: effectiveManifest, - name: resolvedName, - hash, - grantsFile, - exercise: exerciseStatusline, - exerciseLabel: 'renders and refreshes a command-backed footer', - }); - tierResults.push({ tier: 'statusline', ...result }); + // CONFORMANCE_TIERS is the fixed graduation order (ADR-0031 §2) — tier + // results are always pushed in this order, independent of `tiers`' own + // element order, matching the original inline if-per-tier sequence. + const tierResults = []; + for (const tier of CONFORMANCE_TIERS.filter(wantTier)) { + tierResults.push({ tier, ...(await tierRunners[tier]()) }); } if (persist && hash) { - for (const tierResult of tierResults) { - try { - if (tierResult.status === 'passed') { - const evidence = tierResult.evidence ?? evidenceFromChecks(tierResult.checks) ?? tierResult.tier; - recordTierResult(resolvedName, tierResult.tier, { hash, evidence: evidence || tierResult.tier }, { file: grantsFile }); - } else if (tierResult.status === 'gated' && tierResult.gatedBy) { - recordTierGate(resolvedName, tierResult.tier, { hash, gatedBy: tierResult.gatedBy }, { file: grantsFile }); - } else if (tierResult.status === 'failed' && Object.hasOwn(TIER_GRANTS, tierResult.tier)) { - // N-1 (security-review follow-up): a grant-bearing tier that - // RE-RUNS 'failed' at the SAME (unchanged) hash means evidence a - // live capability rests on was just shown non-reproducible — void - // the stored tier and the capability together (grants.mjs's - // recordTierFailure), mirroring recordTierGate's downgrade for - // 'gated' immediately above. Deliberately NOT triggered by - // 'skipped': a skipped tier (e.g. primary-eligible - // short-circuiting because its own prerequisite — activity- - // routing — didn't run/pass THIS run) means the tier was never - // actually EVALUATED this run, which is ambiguous, not disproven - // — downgrading on an ambiguous non-evaluation would void a live - // capability on evidence that says nothing about whether it - // still holds. - recordTierFailure(resolvedName, tierResult.tier, { hash }, { file: grantsFile }); - } - } catch (error) { - tierResult.recordError = error?.message ?? String(error); - } - } + persistTierResults(tierResults, { resolvedName, hash, grantsFile }); } return { name: resolvedName, hash, tiers: tierResults }; } finally { - // F7 (Wave C security review): all THREE overlays a conformance run can - // touch — the host overlay, the execution overlay, and the lifecycle - // registration checkAdmission's detect-hook check creates via - // registerAdmittedLifecycle — must reset together. Resetting only the - // first two (as before) left an already-registered lifecycle adapter - // live for a host id this run's own admission may have just refused, - // the same live-edge class F-9 (execution/admitted.mjs's - // resetAllAdmitted) already closed for the other two overlays. - resetAdmitted(); - resetAdmittedExecution(); - resetAdmittedLifecycle(); - resetAdmittedAqeProviders(); - if (tempDir) { try { fs.rmSync(tempDir, { recursive: true, force: true }); } catch { /* best-effort cleanup */ } } - if (workerCwdTempDir) { try { fs.rmSync(workerCwdTempDir, { recursive: true, force: true }); } catch { /* best-effort cleanup */ } } + cleanupConformanceRun(tempDir, workerCwdTempDir); } } diff --git a/src/lib/adapters/deja-vu.mjs b/src/lib/adapters/deja-vu.mjs index eaa3c1b..41f6a7d 100644 --- a/src/lib/adapters/deja-vu.mjs +++ b/src/lib/adapters/deja-vu.mjs @@ -201,6 +201,180 @@ const indexCommand = (rebuild = false) => ({ command: DEJA_VU_BIN, args: ['index', ...(rebuild ? ['--rebuild'] : [])], }); +// ── detect() decomposition ─────────────────────────────────────────────── +// Extracted verbatim from the detect closure's body (same computed values, +// same shapes) so detect() itself just sequences these, keeping its own +// complexity low while each stage carries its own (small) CC budget. + +/** Runs the offline doctor probe only when the binary is present, and folds + * a non-zero exit over an otherwise-'ok' parse into 'degraded' — the same + * bounded, non-fatal posture detect() always had. */ +async function resolveDoctorState(binaryPresent, runner) { + if (!binaryPresent) return { state: 'skipped', reason: 'binary-missing', facts: null }; + const result = await runner(DEJA_VU_BIN, ['doctor', '--json', '--offline'], { timeout: DOCTOR_TIMEOUT_MS }); + const doctor = parseDejaVuDoctor(result.stdout); + if (result.code !== 0 && doctor.state === 'ok') { + return { state: 'degraded', reason: 'doctor-command-failed', facts: doctor.facts }; + } + return doctor; +} + +/** One host's target fact: bounded observation, receipt/ownership state, + * selection/activity, and the mode-conflict check — the detect() loop's + * per-iteration body, unchanged. */ +function buildHostFact(host, hostPresent, observedRaw, ownership, desired) { + const observed = boundedObservation(observedRaw?.[host]); + const signature = targetSignature(observed); + const receipt = ownership.targets?.[host] ?? null; + const receiptState = !receipt ? 'missing' + : receiptMatches(receipt, host, signature) ? 'current' : 'drifted'; + const selected = desired.enabled && desired.hosts.includes(host); + const active = selected && hostPresent === true; + const conflict = desired.mode === 'mcp' && (observed.direct.auto || observed.plugin.auto) + ? 'external-auto-active' + : receipt && receiptState !== 'current' ? 'ownership-drift' : null; + return { + selected, + hostPresent: hostPresent === true, + desiredTarget: selected ? DEJA_VU_TARGETS[host][desired.mode] : null, + direct: observed.direct, + projection: observed.projection, + plugin: observed.plugin, + signature, + receiptState, + ownership: receiptState === 'current' ? 'agentic-kit' + : (observed.direct.mcp || observed.direct.auto || observed.plugin.present) ? 'external' : 'none', + satisfied: active && targetSatisfied(observed, desired.mode), + conflict, + }; +} + +/** Install-facts assembly for detect(): the receipt-vs-live-version + * comparison (installReceiptState), the effective installed version + * (doctor's own report, falling back to the npm-registry read), and + * whether an owned-but-behind install can self-repair via upgrade. */ +function buildInstallFacts(ownership, binaryPresent, npmVersion, doctor, compareVersions) { + const installState = installReceiptState(ownership.install, npmVersion); + const version = doctor.facts?.version?.current ?? npmVersion; + const install = { + binaryPresent, + npmPresent: npmVersion !== null, + version: typeof version === 'string' ? version : null, + supported: typeof version === 'string' + ? compareVersions(version, DEJA_VU_MIN_VERSION) >= 0 : null, + ownership: installState === 'current' || installState === 'absent' + ? 'agentic-kit' : binaryPresent || npmVersion ? 'external' : 'none', + receiptState: installState, + }; + const ownedUpgradeCanRepair = installState === 'current' && npmVersion !== null + && compareVersions(npmVersion, DEJA_VU_MIN_VERSION) < 0; + return { install, ownedUpgradeCanRepair }; +} + +/** detect()'s error surface: an unhealthy doctor with no owned-upgrade + * self-repair path in flight is the only thing that fails detect() closed. */ +function computeDetectError(binaryPresent, doctor, ownedUpgradeCanRepair) { + if (binaryPresent && doctor.state !== 'ok' && !ownedUpgradeCanRepair) { + return `deja-doctor-${doctor.reason}`; + } + return null; +} + +// ── plan() decomposition ───────────────────────────────────────────────── +// Same discipline as detect() above: each stage extracted verbatim, plan() +// itself just sequences them. + +/** Resolve `latest` — the target install/upgrade version — from the + * registry, with the same bounded fallback ladder plan() always had: an + * unreachable/invalid registry read falls back to DEJA_VU_MIN_VERSION only + * when a version is actually needed to install or repair an unsupported + * install; otherwise it just warns and leaves `latest` null. */ +async function resolveLatestVersion({ + needsInstallVersion, needsUpgradeVersion, facts, latestVersionFn, compareVersions, warnings, +}) { + if (!needsInstallVersion && !needsUpgradeVersion) return null; + let candidate = null; + try { candidate = await latestVersionFn(DEJA_VU_PACKAGE); } catch { /* bounded fallback below */ } + if (isValidSemver(candidate) && compareVersions(candidate, DEJA_VU_MIN_VERSION) >= 0) { + return candidate; + } + if (needsInstallVersion || facts.install.supported === false) { + warnings.push('deja-package-latest-unavailable-baseline-used'); + return DEJA_VU_MIN_VERSION; + } + warnings.push('deja-package-latest-unavailable'); + return null; +} + +/** + * Decide the package install/upgrade operation (or an early-return error), + * per the desired/install-facts state machine plan() used to inline as an + * if/else-if/else chain. Each branch below still returns immediately, so + * only the first matching condition ever fires — identical priority order + * to the original chain. Returns `{error}` (plan() must return that error), + * `{operation}` (plan() should push it), or `{}` (nothing to do — a warning + * may already have been pushed). + */ +function planPackageOperation({ + facts, allowUpgrade, latest, compareVersions, warnings, +}) { + if (facts.desired.enabled && !facts.install.binaryPresent) { + if (facts.install.ownership === 'external') return { error: 'deja-external-install-unusable' }; + return { operation: commandOperation('package-install', 'package-install', packageInstallCommand(latest), { version: latest }) }; + } + if (facts.desired.enabled && facts.install.supported === false) { + if (facts.install.receiptState !== 'current') return { error: 'deja-external-version-unsupported' }; + if (!allowUpgrade) { + warnings.push('deja-package-upgrade-suppressed'); + return {}; + } + return { operation: commandOperation('package-upgrade', 'package-upgrade', packageInstallCommand(latest), { version: latest }) }; + } + if (facts.desired.enabled && allowUpgrade && facts.install.receiptState === 'current' + && latest && facts.install.version && compareVersions(latest, facts.install.version) > 0) { + return { operation: commandOperation('package-upgrade', 'package-upgrade', packageInstallCommand(latest), { version: latest }) }; + } + return {}; +} + +/** One host's target operations: the plan() loop's per-iteration body, + * unchanged — pushes onto the shared `operations`/`warnings` arrays and + * returns early (mirroring the original loop's `continue`s) once a host is + * unselected, absent, or in an unowned conflict. */ +function planHostOperations(host, fact, ownership, desired, operations, warnings) { + const receipt = ownership.targets?.[host]; + const shouldRemove = receipt && (!fact.selected || receipt.mode !== desired.mode); + if (shouldRemove) { + if (fact.receiptState !== 'current') { + warnings.push(`${host}-ownership-drift-preserved`); + } else { + operations.push(commandOperation( + `target-remove-${host}`, 'target-remove', + buildDejaVuUninstallCommand(host, receipt.mode), + { host, mode: receipt.mode, signature: fact.signature }, + )); + } + } + if (!fact.selected) return; + if (!fact.hostPresent) { warnings.push(`${host}-host-missing`); return; } + if (fact.conflict && !receipt) { warnings.push(`${host}-${fact.conflict}`); return; } + if (!fact.satisfied && !(receipt && shouldRemove && fact.receiptState !== 'current')) { + operations.push(commandOperation( + `target-install-${host}`, 'target-install', + buildDejaVuInstallCommand(host, desired.mode), + { host, mode: desired.mode }, + )); + } +} + +/** Whether plan() needs to queue an index rebuild: enabled, opted in via + * indexOnSetup, and either the index is missing/stale or the binary itself + * isn't installed yet (so a freshly-installed binary always indexes). */ +function needsIndexOperation(facts) { + return facts.desired.enabled && facts.desired.indexOnSetup + && (['missing', 'stale'].includes(facts.index.state) || !facts.install.binaryPresent); +} + export function createDejaVuLifecycleAdapter(defaults = {}) { const runner = defaults.runner ?? run; const haveFn = defaults.haveFn ?? have; @@ -221,58 +395,16 @@ export function createDejaVuLifecycleAdapter(defaults = {}) { packageVersionFn(DEJA_VU_PACKAGE), ...Object.values(HOST_BINS).map((bin) => haveFn(bin)), ]); - let doctor = { state: 'skipped', reason: 'binary-missing', facts: null }; - if (binaryPresent) { - const result = await runner(DEJA_VU_BIN, ['doctor', '--json', '--offline'], { - timeout: DOCTOR_TIMEOUT_MS, - }); - doctor = parseDejaVuDoctor(result.stdout); - if (result.code !== 0 && doctor.state === 'ok') { - doctor = { state: 'degraded', reason: 'doctor-command-failed', facts: doctor.facts }; - } - } + const doctor = await resolveDoctorState(binaryPresent, runner); const observedRaw = await observer({ cfg, doctor, binaryPresent }); const hosts = {}; for (const [index, host] of Object.keys(HOST_BINS).entries()) { - const observed = boundedObservation(observedRaw?.[host]); - const signature = targetSignature(observed); - const receipt = ownership.targets?.[host] ?? null; - const receiptState = !receipt ? 'missing' - : receiptMatches(receipt, host, signature) ? 'current' : 'drifted'; - const selected = desired.enabled && desired.hosts.includes(host); - const active = selected && hostPresence[index] === true; - const conflict = desired.mode === 'mcp' && (observed.direct.auto || observed.plugin.auto) - ? 'external-auto-active' - : receipt && receiptState !== 'current' ? 'ownership-drift' : null; - hosts[host] = { - selected, - hostPresent: hostPresence[index] === true, - desiredTarget: selected ? DEJA_VU_TARGETS[host][desired.mode] : null, - direct: observed.direct, - projection: observed.projection, - plugin: observed.plugin, - signature, - receiptState, - ownership: receiptState === 'current' ? 'agentic-kit' - : (observed.direct.mcp || observed.direct.auto || observed.plugin.present) ? 'external' : 'none', - satisfied: active && targetSatisfied(observed, desired.mode), - conflict, - }; + hosts[host] = buildHostFact(host, hostPresence[index], observedRaw, ownership, desired); } - const installState = installReceiptState(ownership.install, npmVersion); - const version = doctor.facts?.version?.current ?? npmVersion; + const { install, ownedUpgradeCanRepair } = buildInstallFacts(ownership, binaryPresent, npmVersion, doctor, compareVersions); const facts = { desired, - install: { - binaryPresent, - npmPresent: npmVersion !== null, - version: typeof version === 'string' ? version : null, - supported: typeof version === 'string' - ? compareVersions(version, DEJA_VU_MIN_VERSION) >= 0 : null, - ownership: installState === 'current' || installState === 'absent' - ? 'agentic-kit' : binaryPresent || npmVersion ? 'external' : 'none', - receiptState: installState, - }, + install, doctor: { state: doctor.state, reason: doctor.reason, @@ -282,11 +414,8 @@ export function createDejaVuLifecycleAdapter(defaults = {}) { index: doctor.facts?.index ?? { state: 'unknown', staleStores: 0 }, targets: hosts, }; - const ownedUpgradeCanRepair = installState === 'current' && npmVersion !== null - && compareVersions(npmVersion, DEJA_VU_MIN_VERSION) < 0; - if (binaryPresent && doctor.state !== 'ok' && !ownedUpgradeCanRepair) { - facts.error = `deja-doctor-${doctor.reason}`; - } + const error = computeDetectError(binaryPresent, doctor, ownedUpgradeCanRepair); + if (error) facts.error = error; return facts; }; @@ -303,73 +432,20 @@ export function createDejaVuLifecycleAdapter(defaults = {}) { && facts.install.ownership !== 'external'; const needsUpgradeVersion = facts.desired.enabled && allowUpgrade && facts.install.receiptState === 'current'; - let latest = null; - if (needsInstallVersion || needsUpgradeVersion) { - let candidate = null; - try { candidate = await latestVersionFn(DEJA_VU_PACKAGE); } catch { /* bounded fallback below */ } - if (isValidSemver(candidate) - && compareVersions(candidate, DEJA_VU_MIN_VERSION) >= 0) { - latest = candidate; - } else if (needsInstallVersion || facts.install.supported === false) { - latest = DEJA_VU_MIN_VERSION; - warnings.push('deja-package-latest-unavailable-baseline-used'); - } else { - warnings.push('deja-package-latest-unavailable'); - } - } - if (facts.desired.enabled && !facts.install.binaryPresent) { - if (facts.install.ownership === 'external') { - return { changed: false, operations, warnings, error: 'deja-external-install-unusable' }; - } - operations.push(commandOperation('package-install', 'package-install', packageInstallCommand(latest), { - version: latest, - })); - } else if (facts.desired.enabled && facts.install.supported === false) { - if (facts.install.receiptState === 'current') { - if (allowUpgrade) { - operations.push(commandOperation('package-upgrade', 'package-upgrade', packageInstallCommand(latest), { - version: latest, - })); - } else { - warnings.push('deja-package-upgrade-suppressed'); - } - } else { - return { changed: false, operations, warnings, error: 'deja-external-version-unsupported' }; - } - } else if (facts.desired.enabled && allowUpgrade && facts.install.receiptState === 'current' - && latest && facts.install.version && compareVersions(latest, facts.install.version) > 0) { - operations.push(commandOperation('package-upgrade', 'package-upgrade', packageInstallCommand(latest), { - version: latest, - })); - } + const latest = await resolveLatestVersion({ + needsInstallVersion, needsUpgradeVersion, facts, latestVersionFn, compareVersions, warnings, + }); + + const packageDecision = planPackageOperation({ + facts, allowUpgrade, latest, compareVersions, warnings, + }); + if (packageDecision.error) return { changed: false, operations, warnings, error: packageDecision.error }; + if (packageDecision.operation) operations.push(packageDecision.operation); for (const [host, fact] of Object.entries(facts.targets)) { - const receipt = ownership.targets?.[host]; - const shouldRemove = receipt && (!fact.selected || receipt.mode !== facts.desired.mode); - if (shouldRemove) { - if (fact.receiptState !== 'current') { - warnings.push(`${host}-ownership-drift-preserved`); - } else { - operations.push(commandOperation( - `target-remove-${host}`, 'target-remove', - buildDejaVuUninstallCommand(host, receipt.mode), - { host, mode: receipt.mode, signature: fact.signature }, - )); - } - } - if (!fact.selected) continue; - if (!fact.hostPresent) { warnings.push(`${host}-host-missing`); continue; } - if (fact.conflict && !receipt) { warnings.push(`${host}-${fact.conflict}`); continue; } - if (!fact.satisfied && !(receipt && shouldRemove && fact.receiptState !== 'current')) { - operations.push(commandOperation( - `target-install-${host}`, 'target-install', - buildDejaVuInstallCommand(host, facts.desired.mode), - { host, mode: facts.desired.mode }, - )); - } + planHostOperations(host, fact, ownership, facts.desired, operations, warnings); } - if (facts.desired.enabled && facts.desired.indexOnSetup - && (['missing', 'stale'].includes(facts.index.state) || !facts.install.binaryPresent)) { + if (needsIndexOperation(facts)) { operations.push(commandOperation('index', 'index', indexCommand(false))); } return { changed: operations.length > 0, operations, warnings }; diff --git a/src/lib/adapters/hook-runner.mjs b/src/lib/adapters/hook-runner.mjs index 6a51a5f..db4045a 100644 --- a/src/lib/adapters/hook-runner.mjs +++ b/src/lib/adapters/hook-runner.mjs @@ -136,21 +136,24 @@ async function killGroup(child) { } } -/** - * Run one adapter hook as a supervised subprocess and report what happened. - * Never throws for process failures (ENOENT, timeout, non-zero exit) — those - * are reported via `ok:false` and `detail`. Only malformed call arguments - * (missing/invalid `hook`, `hostId`, or `verb`) throw synchronously. - * - * @param {{hook:{command:string[], timeoutMs?:number}, hostId:string, - * verb:string, timeoutMs?:number, env?:Record, stdin?:string, - * cwd?:string, manifest?:object, integrity?:{hash:string}, baseDir?:string|null}} options - * @returns {Promise<{ok:boolean, stdout:string, stdoutText:string, stderrText:string, - * stdoutTruncated:boolean, stderrTruncated:boolean, exitCode:number|null, detail:string|null}>} - */ -export async function runAdapterHook({ - hook, hostId, verb, timeoutMs, env, stdin, cwd, manifest, integrity, baseDir, -} = /** @type {any} */ ({})) { +// ── runAdapterHook decomposition ───────────────────────────────────────── +// Each helper below reproduces one slice of the original sequential body +// verbatim, in the same order, so runAdapterHook itself just sequences +// them and its own complexity stays with the orchestration, not the +// process-lifecycle mechanics. + +/** Every early-failure result shares this shape (no output was ever + * captured, or none survives to report) — extracted so the three call + * sites that used to spell it out stay byte-identical by construction. */ +const EMPTY_HOOK_RESULT = Object.freeze({ + stdout: '', stdoutText: '', stderrText: '', stdoutTruncated: false, stderrTruncated: false, +}); + +/** Throws synchronously for malformed call arguments only — never for a + * process-level failure (that's what `ok:false` + `detail` is for). */ +function assertRunAdapterHookArgs({ + hook, hostId, verb, cwd, +}) { if (!hook || !Array.isArray(hook.command) || hook.command.length === 0 || !hook.command.every((part) => typeof part === 'string' && part.length > 0)) { throw new TypeError('runAdapterHook requires hook.command as a non-empty array of non-empty strings'); @@ -164,33 +167,37 @@ export async function runAdapterHook({ if (cwd !== undefined && (typeof cwd !== 'string' || !cwd || !pathIsAbsolute(cwd))) { throw new TypeError('runAdapterHook requires cwd to be an absolute path when provided'); } +} - const effectiveTimeoutMs = resolveTimeout(timeoutMs, hook.timeoutMs); - const [argv0, ...args] = hook.command; - const childEnv = minimalEnv(env); - - // Adrian's trust-gap finding: the manifest-only hash is not enough when a hook - // points at mutable files. Re-read the declared bytes immediately before - // spawn and fail closed if the content identity no longer matches the - // admitted/consented identity. The check is optional for direct unit-level - // callers that do not represent an admitted adapter; production registration - // always supplies all three values. - if (manifest || integrity) { - try { - verifyAdapterContent(manifest, integrity, { baseDir }); - } catch (error) { - return { - ok: false, stdout: '', stdoutText: '', stderrText: '', - stdoutTruncated: false, stderrTruncated: false, exitCode: null, - detail: `${hostId}:${verb} adapter hook integrity check failed: ${error?.message ?? String(error)}`, - }; - } +/** + * Adrian's trust-gap finding: the manifest-only hash is not enough when a + * hook points at mutable files. Re-read the declared bytes immediately + * before spawn and fail closed if the content identity no longer matches + * the admitted/consented identity. The check is optional for direct + * unit-level callers that do not represent an admitted adapter; production + * registration always supplies all three values. Returns a failure result, + * or null when there's nothing to verify or verification passes. + */ +function verifyHookIntegrityOrFailure(manifest, integrity, baseDir, hostId, verb) { + if (!manifest && !integrity) return null; + try { + verifyAdapterContent(manifest, integrity, { baseDir }); + return null; + } catch (error) { + return { + ok: false, ...EMPTY_HOOK_RESULT, exitCode: null, + detail: `${hostId}:${verb} adapter hook integrity check failed: ${error?.message ?? String(error)}`, + }; } +} - const wantsStdin = typeof stdin === 'string'; - let child; +/** Spawn the child, isolated so a synchronous spawn throw (ENOENT-class) + * becomes the same `{failure}` shape a later async 'error' event does. */ +function spawnAdapterChild({ + argv0, args, childEnv, cwd, wantsStdin, hostId, verb, +}) { try { - child = nodeSpawn(argv0, args, { + const child = nodeSpawn(argv0, args, { env: childEnv, shell: false, stdio: [wantsStdin ? 'pipe' : 'ignore', 'pipe', 'pipe'], @@ -200,34 +207,33 @@ export async function runAdapterHook({ // yet (B2 threads the real adapter-base-dir cwd through this wave). ...(cwd === undefined ? {} : { cwd }), }); + return { child }; } catch (error) { - return { - ok: false, stdout: '', stdoutText: '', stderrText: '', - stdoutTruncated: false, stderrTruncated: false, - exitCode: null, detail: describeFailure(hostId, verb, error), - }; + return { failure: { ok: false, ...EMPTY_HOOK_RESULT, exitCode: null, detail: describeFailure(hostId, verb, error) } }; } +} - const stdoutCollector = boundedCollector(OUTPUT_CAP_BYTES); - const stderrCollector = boundedCollector(OUTPUT_CAP_BYTES); - child.stdout?.on('data', (chunk) => stdoutCollector.write(chunk)); - child.stderr?.on('data', (chunk) => stderrCollector.write(chunk)); - - if (wantsStdin) { - // A child that exits before (or without) reading stdin makes the pipe - // write EPIPE — that is a normal outcome (the process's own exit code - // already reports what happened), never a reason to crash or reject - // runAdapterHook's promise. The 'close' handler below still fires and - // resolves the race normally regardless of whether this write lands. - child.stdin?.on('error', () => {}); - try { - child.stdin?.end(stdin); - } catch { - // Synchronous throw from an already-closed stream — same non-fatal - // treatment as the async 'error' event above. - } +/** Write `stdin` (when the caller declared one) and swallow both the async + * EPIPE-class error and any synchronous throw from an already-closed + * stream — a child that exits before (or without) reading stdin is a + * normal outcome (the process's own exit code already reports what + * happened), never a reason to crash or reject runAdapterHook's promise. */ +function wireAdapterStdin(child, wantsStdin, stdin) { + if (!wantsStdin) return; + child.stdin?.on('error', () => {}); + try { + child.stdin?.end(stdin); + } catch { + // Synchronous throw from an already-closed stream — same non-fatal + // treatment as the async 'error' event above. } +} +/** Race the child's close against the timeout budget. Returns + * `{closeResult, getSpawnError}` — `getSpawnError()` reads the 'error'-event + * value (if any); only meaningful to call once `closeResult` (or the + * timeout race over it) has settled. */ +function awaitAdapterClose(child) { let settled = false; let spawnError = null; const closeResult = new Promise((resolve) => { @@ -239,39 +245,37 @@ export async function runAdapterHook({ if (!settled) { settled = true; resolve({ code, signal }); } }); }); + return { closeResult, getSpawnError: () => spawnError }; +} - const raced = await raceTimeout(closeResult, effectiveTimeoutMs); - if (raced === TIMEOUT_SENTINEL) { - await killGroup(child); - await raceTimeout(closeResult, KILL_GRACE_MS); // best-effort; result unused - const stdoutCaptured = { text: stdoutCollector.text(), truncated: stdoutCollector.wasTruncated() }; - const stderrCaptured = { text: stderrCollector.text(), truncated: stderrCollector.wasTruncated() }; - return { - ok: false, - exitCode: null, - stdout: mergeCapture(stdoutCaptured, stderrCaptured), - stdoutText: boundedText(stdoutCaptured), - stderrText: boundedText(stderrCaptured), - stdoutTruncated: stdoutCaptured.truncated, - stderrTruncated: stderrCaptured.truncated, - detail: `${hostId}:${verb} adapter hook timed out after ${effectiveTimeoutMs}ms and was killed`, - }; - } - - if (spawnError) { - return { - ok: false, stdout: '', stdoutText: '', stderrText: '', - stdoutTruncated: false, stderrTruncated: false, - exitCode: null, detail: describeFailure(hostId, verb, spawnError), - }; - } +/** The timed-out-and-killed result shape, capturing whatever output had + * already accumulated before the kill. */ +async function buildAdapterTimeoutResult({ + child, effectiveTimeoutMs, stdoutCollector, stderrCollector, closeResult, hostId, verb, +}) { + await killGroup(child); + await raceTimeout(closeResult, KILL_GRACE_MS); // best-effort; result unused + const stdoutCaptured = { text: stdoutCollector.text(), truncated: stdoutCollector.wasTruncated() }; + const stderrCaptured = { text: stderrCollector.text(), truncated: stderrCollector.wasTruncated() }; + return { + ok: false, + exitCode: null, + stdout: mergeCapture(stdoutCaptured, stderrCaptured), + stdoutText: boundedText(stdoutCaptured), + stderrText: boundedText(stderrCaptured), + stdoutTruncated: stdoutCaptured.truncated, + stderrTruncated: stderrCaptured.truncated, + detail: `${hostId}:${verb} adapter hook timed out after ${effectiveTimeoutMs}ms and was killed`, + }; +} - const { code } = raced; - // F-4/R-1: stdout stays the combined stream for diagnostics/back-compat, - // but a caller parsing stdout as a structured payload (the admitted - // execution adapter) must read stdoutText instead — stdout has stderr - // folded in after a separator, which breaks JSON.parse the instant the - // hook writes anything to stderr at all. stderrText is diagnostics-only. +/** The normal-completion result shape (child closed before the timeout). + * F-4/R-1: stdout stays the combined stream for diagnostics/back-compat, + * but a caller parsing stdout as a structured payload (the admitted + * execution adapter) must read stdoutText instead — stdout has stderr + * folded in after a separator, which breaks JSON.parse the instant the + * hook writes anything to stderr at all. stderrText is diagnostics-only. */ +function buildAdapterCloseResult(code, stdoutCollector, stderrCollector, hostId, verb) { const stdoutCaptured = { text: stdoutCollector.text(), truncated: stdoutCollector.wasTruncated() }; const stderrCaptured = { text: stderrCollector.text(), truncated: stderrCollector.wasTruncated() }; const stdout = mergeCapture(stdoutCaptured, stderrCaptured); @@ -291,3 +295,59 @@ export async function runAdapterHook({ exitCode: code, detail: `${hostId}:${verb} adapter hook exited with code ${code}`, }; } + +/** + * Run one adapter hook as a supervised subprocess and report what happened. + * Never throws for process failures (ENOENT, timeout, non-zero exit) — those + * are reported via `ok:false` and `detail`. Only malformed call arguments + * (missing/invalid `hook`, `hostId`, or `verb`) throw synchronously. + * + * @param {{hook:{command:string[], timeoutMs?:number}, hostId:string, + * verb:string, timeoutMs?:number, env?:Record, stdin?:string, + * cwd?:string, manifest?:object, integrity?:{hash:string}, baseDir?:string|null}} options + * @returns {Promise<{ok:boolean, stdout:string, stdoutText:string, stderrText:string, + * stdoutTruncated:boolean, stderrTruncated:boolean, exitCode:number|null, detail:string|null}>} + */ +export async function runAdapterHook({ + hook, hostId, verb, timeoutMs, env, stdin, cwd, manifest, integrity, baseDir, +} = /** @type {any} */ ({})) { + assertRunAdapterHookArgs({ + hook, hostId, verb, cwd, + }); + + const effectiveTimeoutMs = resolveTimeout(timeoutMs, hook.timeoutMs); + const [argv0, ...args] = hook.command; + const childEnv = minimalEnv(env); + + const integrityFailure = verifyHookIntegrityOrFailure(manifest, integrity, baseDir, hostId, verb); + if (integrityFailure) return integrityFailure; + + const wantsStdin = typeof stdin === 'string'; + const spawned = spawnAdapterChild({ + argv0, args, childEnv, cwd, wantsStdin, hostId, verb, + }); + if (spawned.failure) return spawned.failure; + const { child } = spawned; + + const stdoutCollector = boundedCollector(OUTPUT_CAP_BYTES); + const stderrCollector = boundedCollector(OUTPUT_CAP_BYTES); + child.stdout?.on('data', (chunk) => stdoutCollector.write(chunk)); + child.stderr?.on('data', (chunk) => stderrCollector.write(chunk)); + + wireAdapterStdin(child, wantsStdin, stdin); + + const { closeResult, getSpawnError } = awaitAdapterClose(child); + const raced = await raceTimeout(closeResult, effectiveTimeoutMs); + if (raced === TIMEOUT_SENTINEL) { + return buildAdapterTimeoutResult({ + child, effectiveTimeoutMs, stdoutCollector, stderrCollector, closeResult, hostId, verb, + }); + } + + const spawnError = getSpawnError(); + if (spawnError) { + return { ok: false, ...EMPTY_HOOK_RESULT, exitCode: null, detail: describeFailure(hostId, verb, spawnError) }; + } + + return buildAdapterCloseResult(raced.code, stdoutCollector, stderrCollector, hostId, verb); +} diff --git a/src/lib/adapters/manifest.mjs b/src/lib/adapters/manifest.mjs index c97a6bf..d2221a7 100644 --- a/src/lib/adapters/manifest.mjs +++ b/src/lib/adapters/manifest.mjs @@ -244,15 +244,16 @@ function validateEnvNames(value, field) { return [...value]; } -/** Validate candidate data; host.id fixes identity and grants activation. */ -function validateAqe(value, host, driving, execution) { - assertRecord(value, 'aqe'); - assertNoUnknownKeys(value, ['provider'], 'aqe'); - assertRecord(value.provider, 'aqe.provider'); - assertNoUnknownKeys(value.provider, [ - 'hook', 'billingMode', 'models', 'defaultModel', 'maxConcurrency', 'stripEnv', 'displayName', - ], 'aqe.provider'); +// ── validateAqe decomposition ──────────────────────────────────────────── +// Each helper below reproduces one slice of the original sequential body +// verbatim — same reasons, same messages, same order — so validateAqe +// itself just sequences them (order matters: it IS the throw-precedence +// for a manifest with more than one violation). +/** host.id becomes the AQE provider "type" identity — must be a safe slug, + * not built-in/reserved, on a cli-subprocess-driving host that can already + * route activities via a declared execution.run hook. */ +function checkAqeProviderEligibility(host, driving, execution) { const type = host.id; if (!AQE_PROVIDER_TYPE_RE.test(type)) { throw new ManifestRejected( @@ -272,24 +273,30 @@ function validateAqe(value, host, driving, execution) { 'manifest.aqe requires host.capabilities.canRouteActivities:true and manifest.execution.run.hook', ); } +} - const provider = value.provider; - assertRecord(provider.hook, 'aqe.provider.hook'); - assertNoUnknownKeys(provider.hook, ['command', 'timeoutMs', 'files', 'passEnv'], 'aqe.provider.hook'); +/** aqe.provider.hook's command/timeoutMs/files, mirroring + * validateExecution/validateManifestLifecycle's own hook-shape checks. */ +function validateAqeProviderHook(hook) { try { - assertStringArray(provider.hook.command, 'aqe.provider.hook.command', { allowEmpty: false }); + assertStringArray(hook.command, 'aqe.provider.hook.command', { allowEmpty: false }); } catch (error) { throw new ManifestRejected('invalid-aqe-provider', error.message); } - if (provider.hook.timeoutMs !== undefined - && (!Number.isInteger(provider.hook.timeoutMs) || provider.hook.timeoutMs <= 0 - || provider.hook.timeoutMs > MAX_AQE_PROVIDER_TIMEOUT_MS)) { + if (hook.timeoutMs !== undefined + && (!Number.isInteger(hook.timeoutMs) || hook.timeoutMs <= 0 || hook.timeoutMs > MAX_AQE_PROVIDER_TIMEOUT_MS)) { throw new ManifestRejected( 'invalid-aqe-provider', `aqe.provider.hook.timeoutMs must be a positive integer <= ${MAX_AQE_PROVIDER_TIMEOUT_MS}`, ); } - validateHookFiles(provider.hook.files, 'aqe.provider.hook.files', 'invalid-aqe-provider'); + validateHookFiles(hook.files, 'aqe.provider.hook.files', 'invalid-aqe-provider'); +} + +/** passEnv/stripEnv mutual exclusion and bridge/runtime-variable safety — + * an aqe provider hook may neither forward nor strip a variable the ak + * bridge itself relies on, and the two lists may not collide. */ +function validateAqeProviderEnvPolicy(provider) { const passEnv = validateEnvNames(provider.hook.passEnv, 'aqe.provider.hook.passEnv'); const stripEnv = validateEnvNames(provider.stripEnv, 'aqe.provider.stripEnv'); const nonCanonicalStrip = stripEnv?.find((name) => name !== name.toUpperCase()); @@ -316,13 +323,12 @@ function validateAqe(value, host, driving, execution) { if (conflict) { throw new ManifestRejected('invalid-aqe-provider', `environment '${conflict}' cannot appear in both passEnv and stripEnv`); } + return { passEnv, stripEnv }; +} - if (provider.billingMode !== undefined && !AQE_BILLING_MODES.includes(provider.billingMode)) { - throw new ManifestRejected( - 'invalid-aqe-provider', - `aqe.provider.billingMode must be one of ${AQE_BILLING_MODES.join(', ')}`, - ); - } +/** aqe.provider.models/defaultModel: an optional allowlist (default + * ['default']) and an optional default that must be a member of it. */ +function validateAqeProviderModels(provider) { let models = ['default']; if (provider.models !== undefined) { try { @@ -354,6 +360,12 @@ function validateAqe(value, host, driving, execution) { throw new ManifestRejected('invalid-aqe-provider', 'aqe.provider.defaultModel must be present in aqe.provider.models'); } } + return { models, defaultModel }; +} + +/** aqe.provider.maxConcurrency/displayName: independent optional-scalar + * bounds checks, evaluated last (same order as the original body). */ +function validateAqeProviderLimits(provider) { if (provider.maxConcurrency !== undefined && (!Number.isInteger(provider.maxConcurrency) || provider.maxConcurrency <= 0 || provider.maxConcurrency > MAX_AQE_PROVIDER_CONCURRENCY)) { @@ -371,7 +383,14 @@ function validateAqe(value, host, driving, execution) { `aqe.provider.displayName must be non-empty, control-free, and <= ${MAX_AQE_DISPLAY_NAME_BYTES} UTF-8 bytes`, ); } +} +/** Assemble the validated, independent aqe.provider record — every optional + * field spread in only when the input actually declared it, never a + * fabricated default riding along undeclared. */ +function buildAqeProviderRecord(provider, { + passEnv, stripEnv, models, defaultModel, +}) { return { provider: { hook: { @@ -390,6 +409,39 @@ function validateAqe(value, host, driving, execution) { }; } +/** Validate candidate data; host.id fixes identity and grants activation. */ +function validateAqe(value, host, driving, execution) { + assertRecord(value, 'aqe'); + assertNoUnknownKeys(value, ['provider'], 'aqe'); + assertRecord(value.provider, 'aqe.provider'); + assertNoUnknownKeys(value.provider, [ + 'hook', 'billingMode', 'models', 'defaultModel', 'maxConcurrency', 'stripEnv', 'displayName', + ], 'aqe.provider'); + + checkAqeProviderEligibility(host, driving, execution); + + const { provider } = value; + assertRecord(provider.hook, 'aqe.provider.hook'); + assertNoUnknownKeys(provider.hook, ['command', 'timeoutMs', 'files', 'passEnv'], 'aqe.provider.hook'); + validateAqeProviderHook(provider.hook); + const { passEnv, stripEnv } = validateAqeProviderEnvPolicy(provider); + + // Order matters (throw precedence, unchanged from the original body): + // billingMode, then models/defaultModel, then maxConcurrency/displayName. + if (provider.billingMode !== undefined && !AQE_BILLING_MODES.includes(provider.billingMode)) { + throw new ManifestRejected( + 'invalid-aqe-provider', + `aqe.provider.billingMode must be one of ${AQE_BILLING_MODES.join(', ')}`, + ); + } + const { models, defaultModel } = validateAqeProviderModels(provider); + validateAqeProviderLimits(provider); + + return buildAqeProviderRecord(provider, { + passEnv, stripEnv, models, defaultModel, + }); +} + /** Hook file inventories are portable paths relative to the manifest's own * directory. Content is hashed later, once admission has resolved that * directory; schema validation keeps absolute/traversal paths out of the @@ -450,65 +502,66 @@ function validateManifestTrust(value) { return structuredClone(value); } -/** - * Validate one adapter manifest document. Throws ManifestRejected (a named - * `.reason`) on any structural or capability-cap violation; returns a frozen, - * fully independent copy on success. `projections`/`observability` are - * injectable for tests, defaulting to the real built-in registries. - */ -export function validateAdapterManifest(value, { projections = projectionMap, observability = observabilityMap } = {}) { - assertRecord(value, 'manifest'); - assertNoUnknownKeys(value, MANIFEST_ALLOWED_KEYS, ''); +// ── validateAdapterManifest decomposition ──────────────────────────────── +// Each helper reproduces one slice of the original sequential body +// verbatim, in the same order, so validateAdapterManifest itself just +// sequences them (order is the throw-precedence for a manifest with more +// than one violation). +function validateManifestIdentity(value) { try { assertId(value.name, 'manifest.name'); } catch (error) { throw new ManifestRejected('invalid-id', error.message); } - if (typeof value.version !== 'string' || !SEMVER_RE.test(value.version)) { throw new ManifestRejected('invalid-version', 'manifest.version must be semver (e.g. 1.2.3)'); } - if (!Number.isInteger(value.contract) || value.contract < 1) { throw new ManifestRejected('invalid-contract', 'manifest.contract must be a positive integer'); } +} - // ── host-layer allowlist wrapper, BEFORE validateHostAdapter runs ────── - // validateHostAdapter is shared with the built-in registry (registries.mjs) - // and must keep structuredClone-ing whatever it's handed — so the - // allowlisting happens here, at the manifest (external-adapter-only) layer, - // never inside that shared validator. - if (value.host && typeof value.host === 'object' && !Array.isArray(value.host)) { - assertNoUnknownKeys(value.host, HOST_ALLOWED_KEYS, 'host'); - if (value.host.install && typeof value.host.install === 'object' && !Array.isArray(value.host.install)) { - assertNoUnknownKeys(value.host.install, HOST_INSTALL_ALLOWED_KEYS, 'host.install'); - if (value.host.install.npmPackage != null) { - throw new ManifestRejected('external-npm-package', 'external adapters may not declare host.install.npmPackage — detect-never-overwrite only'); - } - try { - assertId(value.host.install.bin, 'host.install.bin'); - } catch (error) { - throw new ManifestRejected('invalid-install-bin', error.message); - } +/** + * Host-layer allowlist wrapper, BEFORE validateHostAdapter runs. + * validateHostAdapter is shared with the built-in registry (registries.mjs) + * and must keep structuredClone-ing whatever it's handed — so the + * allowlisting happens here, at the manifest (external-adapter-only) layer, + * never inside that shared validator. + */ +function assertHostAllowedKeys(hostValue) { + if (!hostValue || typeof hostValue !== 'object' || Array.isArray(hostValue)) return; + assertNoUnknownKeys(hostValue, HOST_ALLOWED_KEYS, 'host'); + if (hostValue.install && typeof hostValue.install === 'object' && !Array.isArray(hostValue.install)) { + assertNoUnknownKeys(hostValue.install, HOST_INSTALL_ALLOWED_KEYS, 'host.install'); + if (hostValue.install.npmPackage != null) { + throw new ManifestRejected('external-npm-package', 'external adapters may not declare host.install.npmPackage — detect-never-overwrite only'); + } + try { + assertId(hostValue.install.bin, 'host.install.bin'); + } catch (error) { + throw new ManifestRejected('invalid-install-bin', error.message); } - assertNoUnknownKeys(value.host.legacy, HOST_LEGACY_ALLOWED_KEYS, 'host.legacy'); - // Capability keys are allowlisted to the canonical set too, so a - // differently-cased or extra key ('CanBePrimary', 'ADMIN') can't ride - // along inert — the cap-bypass surface is provably closed, not merely - // harmless because consumers happen to read the exact lowercase flag. - assertNoUnknownKeys(value.host.capabilities, HOST_CAPABILITY_KEYS, 'host.capabilities'); } + assertNoUnknownKeys(hostValue.legacy, HOST_LEGACY_ALLOWED_KEYS, 'host.legacy'); + // Capability keys are allowlisted to the canonical set too, so a + // differently-cased or extra key ('CanBePrimary', 'ADMIN') can't ride + // along inert — the cap-bypass surface is provably closed, not merely + // harmless because consumers happen to read the exact lowercase flag. + assertNoUnknownKeys(hostValue.capabilities, HOST_CAPABILITY_KEYS, 'host.capabilities'); +} - let host; +function parseHostAdapter(hostValue, projections, observability) { try { - host = validateHostAdapter(value.host, { projections, observability }); + return validateHostAdapter(hostValue, { projections, observability }); } catch (error) { throw new ManifestRejected('invalid-host', error.message); } +} - // ── structural caps: these claims must be INEXPRESSIBLE, not merely - // refused at runtime — an external manifest can never assert them true. ── +/** Structural caps: these claims must be INEXPRESSIBLE, not merely refused + * at runtime — an external manifest can never assert them true. */ +function enforceHostCapabilityCaps(host) { if (host.capabilities.canBePrimary === true) { throw new ManifestRejected('cap-can-be-primary', 'external adapters may not claim host.capabilities.canBePrimary'); } @@ -527,13 +580,33 @@ export function validateAdapterManifest(value, { projections = projectionMap, ob throw new ManifestRejected('invalid-guidance-file', error.message); } } - // P2 structural coupling (ADR-0031): an execution hook on a host that - // cannot route activities is a contradiction the schema refuses outright, - // never silently ignores. The converse — routable, no execution block — is - // legal and degrades honestly at run time (cli_unavailable). - if (value.execution !== undefined && host.capabilities.canRouteActivities !== true) { +} + +/** P2 structural coupling (ADR-0031): an execution hook on a host that + * cannot route activities is a contradiction the schema refuses outright, + * never silently ignores. The converse — routable, no execution block — is + * legal and degrades honestly at run time (cli_unavailable). */ +function checkExecutionRoutable(executionValue, host) { + if (executionValue !== undefined && host.capabilities.canRouteActivities !== true) { throw new ManifestRejected('execution-not-routable', 'manifest.execution requires host.capabilities.canRouteActivities: true'); } +} + +/** + * Validate one adapter manifest document. Throws ManifestRejected (a named + * `.reason`) on any structural or capability-cap violation; returns a frozen, + * fully independent copy on success. `projections`/`observability` are + * injectable for tests, defaulting to the real built-in registries. + */ +export function validateAdapterManifest(value, { projections = projectionMap, observability = observabilityMap } = {}) { + assertRecord(value, 'manifest'); + assertNoUnknownKeys(value, MANIFEST_ALLOWED_KEYS, ''); + + validateManifestIdentity(value); + assertHostAllowedKeys(value.host); + const host = parseHostAdapter(value.host, projections, observability); + enforceHostCapabilityCaps(host); + checkExecutionRoutable(value.execution, host); const detection = validateDetection(value.detection); const driving = validateDriving(value.driving); diff --git a/src/lib/adapters/registries.mjs b/src/lib/adapters/registries.mjs index 9de7d19..a3c1c8f 100644 --- a/src/lib/adapters/registries.mjs +++ b/src/lib/adapters/registries.mjs @@ -444,7 +444,16 @@ export function validateActivityHost(id, hosts = HOST_REGISTRY) { ? { ok: true } : { ok: false, reason: 'capability-canRouteActivities-required' }; } -export function validateRegistries(registries) { +// ── validateRegistries decomposition ───────────────────────────────────── +// Each cross-axis invariant below is a rule `(registries) => Error[]` — +// independent of the others (none consumes another's output), so a +// registry of small check functions replaces the single sequential body. +// REGISTRY_INVARIANT_CHECKS runs them in the SAME order the original body +// evaluated its blocks in, and each function's own internal loop order is +// unchanged — both matter because tests assert the exact returned array +// (deepEqual is order-sensitive), not just its membership. + +function checkDuplicateIds(registries) { const errors = []; const axes = ['hosts', 'providers', 'projections', 'observability', 'modelDiscovery']; for (const axis of axes) { @@ -454,6 +463,15 @@ export function validateRegistries(registries) { seen.add(entry.id); } } + return errors; +} + +/** configProjection/observability referential integrity plus the + * canDriveSession -> canBePrimary/canRouteActivities dependency, per host — + * kept as one function (not three) so a host violating more than one + * invariant still reports them in the original interleaved per-host order. */ +function checkHostInvariants(registries) { + const errors = []; const projections = new Set((registries?.projections ?? []).map((entry) => entry.id)); const observability = new Set((registries?.observability ?? []).map((entry) => entry.id)); for (const [index, host] of (registries?.hosts ?? []).entries()) { @@ -473,6 +491,12 @@ export function validateRegistries(registries) { } } } + return errors; +} + +/** Billing/credential consistency plus credential-name shape, per provider. */ +function checkProviderInvariants(registries) { + const errors = []; const envName = /^[A-Z][A-Z0-9_]*$/; for (const [index, provider] of (registries?.providers ?? []).entries()) { const kind = provider.credentials?.kind; @@ -491,6 +515,11 @@ export function validateRegistries(registries) { }); } } + return errors; +} + +function checkModelDiscoveryOwners(registries) { + const errors = []; const hosts = new Set((registries?.hosts ?? []).map((entry) => entry.id)); const providers = new Set((registries?.providers ?? []).map((entry) => entry.id)); for (const [index, descriptor] of (registries?.modelDiscovery ?? []).entries()) { @@ -501,3 +530,15 @@ export function validateRegistries(registries) { } return errors; } + +export function validateRegistries(registries) { + // Array built fresh per call (not a module-level const): validateRegistries + // itself runs once at MODULE LOAD time (the construction-time self-check + // below, before this line is ever reached in top-to-bottom evaluation) — + // a top-level `const` here would still be in its temporal dead zone at + // that first call. The four functions are plain hoisted declarations, so + // referencing them is safe from anywhere in the module regardless of call + // time; only the array literal itself needs to be per-call. + const checks = [checkDuplicateIds, checkHostInvariants, checkProviderInvariants, checkModelDiscoveryOwners]; + return checks.flatMap((check) => check(registries)); +} diff --git a/src/lib/aqe-router.mjs b/src/lib/aqe-router.mjs new file mode 100644 index 0000000..9ad30e8 --- /dev/null +++ b/src/lib/aqe-router.mjs @@ -0,0 +1,681 @@ +// aqe-router.mjs — the ONE definition of the AQE-router convergence pipeline: +// `.agentic-qe/llm-config.json`'s fallback chain, default provider, external +// provider declarations/activations, and `agentOverrides` projection. +// Extracted from providers.mjs (ADR-0037) so that 1,000+ line file stays under +// its max-lines budget; providers.mjs re-imports `applyAqeRouter`, +// `aqeRouterDrift`, and `undoAqeRouter` so every existing external import path +// (`./providers.mjs`) keeps working unchanged. +// +// Five surfaces used to be braided together in one function, sharing mutable +// accumulators with implicit cross-surface feedback: `externalActive` +// (computed while reconciling external providers) constrained what the +// fallback-chain/default-provider/agentOverrides surfaces below it could +// safely reference, and `projected`/`staleOverrides` had to be recomputed +// after that same fact became known. Each surface below is a +// `(next, ctx) => {detail, error, changed, ctx?}` step, folded left-to-right +// over one shared `next` draft; a surface returns an optional `ctx` PATCH +// (applied before the next surface runs) instead of closing over an outer +// `let` — the one real cross-surface dependency (externalActive -> the +// refined `projected`/`staleOverrides`) is the only patch actually used, so +// it stays a single, explicit, ordered hand-off rather than several loose +// mutable accumulators. +import fs from 'node:fs'; +import path from 'node:path'; +import { repoRoot } from './paths.mjs'; +import { readJson, writeJsonWithBackup } from './settings.mjs'; +import { configuredPolicyToAgentOverrides, AGENT_ACTIVITY_MAP } from './routing.mjs'; +import { admittedAqeProviders } from './adapters/aqe-provider.mjs'; +import { + AQE_OWNERSHIP_KEY, EXTERNAL_PROVIDERS_MIN_AQE, stableValue, declarationHash, plainRecord, + aqeExternalProviders, aqeRouterFile, aqeSupportsExternalProviders, aqeSupportsAgentOverrides, + aqeSelectableChainProviderTypes, credentialGaps, +} from './providers.mjs'; + +// See providers.mjs's own `mergeRouterConfig`/`applyProviders` comments for +// why every field here is a REQUIRED scalar default, not an optional one: +// - mergeRouterConfig deep-merges `providers` but SHALLOW-replaces +// `fallbackChain` → ak must write a COMPLETE chain (these scalar defaults). +// - the router iterates `entry.models` → each entry needs populated models. +// - aqe refuses to persist apiKey → ak writes only `enabled` per provider; +// keys stay in the env. +const AQE_CHAIN_DEFAULTS = { maxRetries: 3, retryDelayMs: 100, backoffMultiplier: 2, maxDelayMs: 5000 }; +const AQE_MANAGED_TAG = 'agentic-kit'; + +function admittedProviderRecord(id) { + const records = admittedAqeProviders(); + return (Array.isArray(records) ? records : Object.values(records ?? {})) + .find((entry) => (entry.id ?? entry.providerId ?? entry.type) === id) ?? null; +} + +function exactlyOwnedDefault(config, receiptKey) { + const provider = config?.defaultProvider; + const receipt = plainRecord(config?.[AQE_OWNERSHIP_KEY]?.[receiptKey]); + return typeof provider === 'string' && receipt?.provider === provider + && receipt.writtenHash === declarationHash(provider) + ? provider + : null; +} + +function setDefaultOwnership(config, receiptKey, provider) { + const ownership = { ...(plainRecord(config[AQE_OWNERSHIP_KEY]) ?? {}) }; + ownership[receiptKey] = { provider, writtenHash: declarationHash(provider) }; + config[AQE_OWNERSHIP_KEY] = ownership; +} + +function clearDefaultOwnership(config, receiptKey) { + const ownership = { ...(plainRecord(config[AQE_OWNERSHIP_KEY]) ?? {}) }; + delete ownership[receiptKey]; + if (Object.keys(ownership).length) config[AQE_OWNERSHIP_KEY] = ownership; + else delete config[AQE_OWNERSHIP_KEY]; +} + +const exactlyOwnedExternalDefault = (config) => exactlyOwnedDefault(config, 'externalDefaultProvider'); +const exactlyOwnedFallbackDefault = (config) => exactlyOwnedDefault(config, 'fallbackDefaultProvider'); +const setExternalDefaultOwnership = (config, provider) => + setDefaultOwnership(config, 'externalDefaultProvider', provider); +const setFallbackDefaultOwnership = (config, provider) => + setDefaultOwnership(config, 'fallbackDefaultProvider', provider); +const clearExternalDefaultOwnership = (config) => + clearDefaultOwnership(config, 'externalDefaultProvider'); +const clearFallbackDefaultOwnership = (config) => + clearDefaultOwnership(config, 'fallbackDefaultProvider'); + +/** One `desired` entry whose live declaration differs from what ak last wrote: + * user-owned now. Its activation keeps an exact receipt only when it still + * matches what ak wrote, so a later revoke can still remove the minimal + * record ak created without ever touching the edited declaration. */ +function reconcileConflictingDeclaration(id, prior, currentActivation, receipts) { + if (prior?.providerWrittenHash && currentActivation !== undefined + && declarationHash(currentActivation) === prior.providerWrittenHash) { + receipts[id] = { providerWrittenHash: prior.providerWrittenHash }; + } else { + delete receipts[id]; + } +} + +/** Activation bookkeeping for one accepted declaration. AQE 3.13.12's MCP + * router asks whether any providers are enabled BEFORE it loads + * externalProviders (the load is what registers them) — a minimal owned + * `providers[id].enabled` record breaks that bootstrap cycle, so ak owns one + * only when the id had NO prior activation; a user-owned or already-enabled + * activation is left alone (and usable only once genuinely enabled). */ +function reconcileProviderActivation(id, currentProviders, prior, nextReceipt) { + const currentActivation = currentProviders[id]; + const priorActivationHash = prior?.providerWrittenHash; + const activationHash = currentActivation === undefined ? null : declarationHash(currentActivation); + if (currentActivation === undefined) { + currentProviders[id] = { enabled: true }; + nextReceipt.providerWrittenHash = declarationHash(currentProviders[id]); + return { added: true, ok: true }; + } + if (currentActivation?.enabled === true) { + if (priorActivationHash && activationHash === priorActivationHash) { + nextReceipt.providerWrittenHash = priorActivationHash; + } + return { added: false, ok: true }; + } + return { added: false, ok: false }; +} + +/** One `desired[id]` entry: accept it, flag it as a conflict, or reconcile its + * activation. Mutates `state`'s collections in place. */ +function reconcileDesiredProvider(id, declaration, priorReceipts, state) { + const { current, currentProviders, receipts } = state; + const prior = priorReceipts[id]; + const currentDeclaration = current[id]; + const currentHash = currentDeclaration === undefined ? null : declarationHash(currentDeclaration); + if (currentDeclaration !== undefined && (!prior || currentHash !== prior.writtenHash)) { + state.conflicts.push(id); + state.unavailable.add(id); + reconcileConflictingDeclaration(id, prior, currentProviders[id], receipts); + return; + } + current[id] = declaration; + const record = admittedProviderRecord(id); + const nextReceipt = { + hostId: record?.hostId ?? record?.host ?? record?.manifestId ?? null, + contentHash: record?.contentHash ?? record?.integrity ?? null, + writtenHash: declarationHash(declaration), + }; + if (currentDeclaration === undefined) state.added.push(id); + + const activation = reconcileProviderActivation(id, currentProviders, prior, nextReceipt); + if (activation.ok) { + state.active.add(id); + if (activation.added) state.activationsAdded.push(id); + } else { + state.conflicts.push(`${id} (providers.${id}.enabled is not true)`); + state.unavailable.add(id); + } + receipts[id] = nextReceipt; +} + +/** One `priorReceipts[id]` entry no longer in `desired`: retire it, pruning + * the declaration/activation ak wrote when they are still exactly what it + * wrote (a user edit is preserved, not silently deleted). */ +function retireStaleProvider(id, receipt, state) { + const { current, currentProviders, receipts } = state; + state.retired.push(id); + const currentDeclaration = current[id]; + if (currentDeclaration !== undefined && declarationHash(currentDeclaration) === receipt.writtenHash) { + delete current[id]; + state.pruned.push(id); + } + const currentActivation = currentProviders[id]; + if (receipt.providerWrittenHash && currentActivation !== undefined + && declarationHash(currentActivation) === receipt.providerWrittenHash) { + delete currentProviders[id]; + state.activationsPruned.push(id); + } + delete receipts[id]; +} + +/** Compare the live admitted declarations with the exact values ak previously + * wrote. Foreign entries and user-edited owned entries are never overwritten or + * removed. Returned `active` ids are safe to reference from defaults/chains. */ +function reconcileExternalProviders(existing, desired = aqeExternalProviders()) { + const current = { ...(existing.externalProviders ?? {}) }; + const currentProviders = { ...(existing.providers ?? {}) }; + // Ownership metadata is advisory proof, never trusted input. A null/array/ + // primitive receipt proves nothing and must be dropped rather than crashing + // sync or authorizing deletion of user values. + const rawReceipts = plainRecord(existing[AQE_OWNERSHIP_KEY]?.externalProviders) ?? {}; + const priorReceipts = Object.fromEntries(Object.entries(rawReceipts) + .filter(([, receipt]) => plainRecord(receipt))); + const state = { + current, + currentProviders, + receipts: { ...priorReceipts }, + active: new Set(), + conflicts: [], + unavailable: new Set(), + retired: [], + pruned: [], + added: [], + activationsAdded: [], + activationsPruned: [], + }; + + for (const [id, declaration] of Object.entries(desired)) { + reconcileDesiredProvider(id, declaration, priorReceipts, state); + } + for (const [id, receipt] of Object.entries(priorReceipts)) { + if (id in desired) continue; + retireStaleProvider(id, receipt, state); + } + + return { + externalProviders: state.current, + providers: state.currentProviders, + receipts: state.receipts, + active: state.active, + conflicts: state.conflicts, + unavailable: [...state.unavailable], + retired: state.retired, + pruned: state.pruned, + added: state.added, + activationsAdded: state.activationsAdded, + activationsPruned: state.activationsPruned, + }; +} + +/** Map kit.json `aqeFallback` entries → a complete aqe FallbackChain. Priority + * descends by list order (first = highest). Entries carry provider + models. */ +function buildChain(entries) { + return { + id: AQE_MANAGED_TAG, + entries: entries.map((e, i) => ({ + provider: e.provider, + models: e.models ?? [], + enabled: true, + priority: 100 - i * 10, + maxAttempts: 2, + timeoutMs: 30000, + })), + ...AQE_CHAIN_DEFAULTS, + }; +} + +/** The externalProviders surface's own detail line — split out only to keep + * that surface's branch count (five independent `? : ''` clauses) legible + * and under the reconciler's own complexity budget. */ +function formatExternalProvidersDetail(externalActive, reconciled) { + return `externalProviders: ${externalActive.size} managed` + + (reconciled.added.length ? ` (${reconciled.added.length} added)` : '') + + (reconciled.pruned.length ? ` (${reconciled.pruned.length} stale owned pruned)` : '') + + (reconciled.activationsAdded.length ? ` (${reconciled.activationsAdded.length} MCP activation added)` : '') + + (reconciled.activationsPruned.length ? ` (${reconciled.activationsPruned.length} stale activation pruned)` : '') + + (reconciled.conflicts.length ? ` (⚠ conflicts preserved: ${reconciled.conflicts.join(', ')})` : ''); +} + +/** Surface 1/4: reconcile admitted external-provider declarations/activations + * against the live file, prune anything that became unavailable from the + * fallback chain/defaultProvider, and refine `projected`/`staleOverrides` for + * the surfaces after it (their safe-to-reference set depends on which + * external ids ended up active here). */ +function reconcileExternalProvidersSurface(next, ctx) { + const { + existing, desiredExternal, hasExternal, hasOwnedExternal, externalSupported, + hasManagedFallback, ownedFallbackDefault, ownedExternalDefault, + priorOverrides, managedOverrideKeys, projected: priorProjected, + } = ctx; + let externalActive = new Set(); + let error = null; + let changed = false; + const detail = []; + + if (hasExternal || hasOwnedExternal) { + // A downgrade must remove only unchanged entries we previously wrote, + // plus their dangling references. Keeping declarations that this AQE + // version cannot understand would strand every router startup on drift. + const reconciled = reconcileExternalProviders(existing, externalSupported ? desiredExternal : {}); + externalActive = reconciled.active; + if (Object.keys(reconciled.externalProviders).length) next.externalProviders = reconciled.externalProviders; + else delete next.externalProviders; + if (Object.keys(reconciled.providers).length) next.providers = reconciled.providers; + else delete next.providers; + const ownership = { ...(plainRecord(next[AQE_OWNERSHIP_KEY]) ?? {}) }; + if (!ownedExternalDefault) delete ownership.externalDefaultProvider; + if (Object.keys(reconciled.receipts).length) ownership.externalProviders = reconciled.receipts; + else delete ownership.externalProviders; + if (Object.keys(ownership).length) next[AQE_OWNERSHIP_KEY] = ownership; + else delete next[AQE_OWNERSHIP_KEY]; + if (reconciled.conflicts.length) { + error = `refused conflicting foreign/user-edited external provider ids: ${reconciled.conflicts.join(', ')}`; + } + detail.push(formatExternalProvidersDetail(externalActive, reconciled)); + if (hasExternal && !externalSupported) { + error = `external providers need agentic-qe >=${EXTERNAL_PROVIDERS_MIN_AQE}`; + detail.push(`externalProviders: disabled (${error})`); + } + const unavailableExternal = new Set([...reconciled.unavailable, ...reconciled.retired]); + if (hasManagedFallback && next.fallbackChain?.entries) { + next.fallbackChain = { + ...next.fallbackChain, + entries: next.fallbackChain.entries.filter((entry) => !unavailableExternal.has(entry.provider)), + }; + if (next.fallbackChain.entries.length === 0) delete next.fallbackChain; + } + if (unavailableExternal.has(next.defaultProvider) + && (ownedFallbackDefault === next.defaultProvider || ownedExternalDefault === next.defaultProvider)) { + delete next.defaultProvider; + clearExternalDefaultOwnership(next); + clearFallbackDefaultOwnership(next); + } + changed = reconciled.added.length > 0 || reconciled.pruned.length > 0 + || reconciled.activationsAdded.length > 0 || reconciled.activationsPruned.length > 0 + || Object.keys(desiredExternal).some((id) => existing.externalProviders?.[id] + && declarationHash(existing.externalProviders[id]) !== declarationHash(desiredExternal[id])); + } + + // Admission/version/conflict filtering can make a previously projected + // external route inactive. Recompute from the safe projection so ak-owned + // overrides never retain an unusable id — this runs regardless of whether + // the branch above executed (externalActive then defaults to empty). + const projected = Object.fromEntries(Object.entries(priorProjected).filter(([, entry]) => + !(entry.provider in desiredExternal) || externalActive.has(entry.provider))); + const staleOverrides = Object.keys(priorOverrides) + .filter((agent) => managedOverrideKeys.has(agent) && !(agent in projected)); + + return { + detail, error, changed, ctx: { externalActive, projected, staleOverrides }, + }; +} + +/** Surface 2/4: retire a previously-written managed fallback chain (and its + * derived default) once the canonical `aqeFallback` intent goes empty. */ +function reconcileFallbackRetirementSurface(next, ctx) { + const { + hasChain, hasManagedFallback, ownedFallbackDefault, ownedExternalDefault, + } = ctx; + if (hasChain || !hasManagedFallback) return null; + // An empty canonical fallback intent retires the tagged chain ak previously + // wrote. Its derived default belongs to the same projection and must not + // survive independently; provider declarations/activations remain available + // for explicit selection, routes, or a future chain. + delete next.fallbackChain; + if (ownedFallbackDefault) { + delete next.defaultProvider; + if (ownedExternalDefault) clearExternalDefaultOwnership(next); + } + clearFallbackDefaultOwnership(next); + return { detail: 'chain: managed fallback retired', changed: true }; +} + +/** Surface 3/4: decide `defaultProvider` and which of the two ownership + * receipts (external vs. fallback-chain-derived) it carries, across the + * three ways it can change: explicit deselection, chain-derived assignment + * (which also builds/validates the active chain itself), and an explicit + * project-local external selection. */ +function reconcileDefaultProviderSurface(next, ctx) { + const { + cfg, existing, chain, selectedProvider, hasChain, desiredExternal, externalActive, ownedExternalDefault, + } = ctx; + const detail = []; + let error = null; + let changed = false; + + // `aqeProvider: null` is an explicit deselection. Retire only an exact + // external default that ak previously wrote, while leaving the admitted + // declaration and MCP activation intact for routes or future selection. + // A configured fallback chain owns default selection independently and is + // handled below; it must not be erased by primary-provider deselection. + if (!hasChain && selectedProvider === null && ownedExternalDefault) { + delete next.defaultProvider; + clearExternalDefaultOwnership(next); + detail.push(`defaultProvider: ${ownedExternalDefault} retired`); + changed = true; + } + + if (hasChain) { + const selectable = new Set(aqeSelectableChainProviderTypes()); + const valid = chain.filter((e) => e?.provider && selectable.has(e.provider) + && (!(e.provider in desiredExternal) || externalActive.has(e.provider))); + if (valid.length === 0) { + // A bad chain must NOT block the independent agentOverrides projection — the + // Activity routing is validated separately. Record it and carry on. + error = 'no valid providers in fallback chain'; + detail.push(`chain: ⚠ ${error}`); + } else { + const requestedDefault = cfg.providers.aqeProvider; + const requestedUnavailable = requestedDefault in desiredExternal && !externalActive.has(requestedDefault); + next.defaultProvider = requestedUnavailable ? valid[0].provider : requestedDefault ?? valid[0].provider; + setFallbackDefaultOwnership(next, next.defaultProvider); + next.providers = { ...(next.providers ?? existing.providers ?? {}) }; + for (const e of valid) { + if (!(e.provider in desiredExternal)) next.providers[e.provider] = { ...(existing.providers?.[e.provider] ?? {}), enabled: true }; + } + next.fallbackChain = buildChain(valid); + if (next.defaultProvider in desiredExternal && externalActive.has(next.defaultProvider)) { + setExternalDefaultOwnership(next, next.defaultProvider); + } else if (ownedExternalDefault) { + clearExternalDefaultOwnership(next); + } + const emptyModels = valid.filter((e) => !e.models || e.models.length === 0).map((e) => e.provider); + // Warn, never refuse: the user may export the key later, and silently + // dropping a rung is worse than writing one that is currently inert (#54). + const gaps = credentialGaps(valid); + detail.push(`chain: ${valid.map((e) => e.provider).join(' → ')}` + + (emptyModels.length ? ` (⚠ no models for: ${emptyModels.join(', ')})` : '') + + (gaps.length ? ` (⚠ no credential for: ${gaps.map((g) => `${g.provider} — needs ${g.missing.join(', ')}`).join('; ')})` : '')); + changed = true; + } + } + + // External provider selection is project-local by contract: AQE discovers it + // from this file only. managedEnv deliberately never exports an external id + // into project or user host settings. + if (selectedProvider && selectedProvider in desiredExternal) { + if (externalActive.has(selectedProvider)) { + next.defaultProvider = selectedProvider; + setExternalDefaultOwnership(next, selectedProvider); + detail.push(`defaultProvider: ${selectedProvider} (project-local external)`); + changed = true; + } else { + error ??= `external default '${selectedProvider}' is not safely managed`; + } + } + + return { detail, error, changed }; +} + +/** Surface 4/4: project `routing.routes` into aqe's `agentOverrides`, merged + * with (not replacing) foreign entries, pruning only the ak-owned entries + * the current projection no longer names (`ctx.staleOverrides`, refined by + * surface 1 against the final external-availability set). */ +function reconcileAgentOverridesSurface(next, ctx) { + const { + existing, desiredExternal, priorOverrides, projected, staleOverrides, hasPolicy, agentOverridesSupported, + } = ctx; + if ((agentOverridesSupported && Object.keys(projected).length) || staleOverrides.length) { + // MERGE, don't replace: ak owns only the curated agent-types it projects; + // preserve foreign entries (aqe's own defaults or a hand-added agent). The + // projector drops non-constructible providers (mirrors sanitizeAgentOverrides) + // and only ever emits {provider, model} — no apiKey. + next.agentOverrides = { ...priorOverrides }; + for (const agent of staleOverrides) delete next.agentOverrides[agent]; + if (agentOverridesSupported) Object.assign(next.agentOverrides, projected); + // An override naming a provider is inert until that provider is ENABLED in + // this same file: aqe enables from env keys or the `providers` map, and a + // subscription host-CLI provider (codex, claude-code) has no env key at + // all — so ak-projected codex overrides sat dead and warned on every aqe + // startup (#108 phase 3). Enable exactly the providers the projection + // references — merge-not-clobber, writing nothing beyond `enabled`. + const referenced = agentOverridesSupported + ? [...new Set(Object.values(projected).map((entry) => entry.provider))] + : []; + if (referenced.length) { + next.providers = { ...(next.providers ?? existing.providers ?? {}) }; + for (const provider of referenced) { + if (!(provider in desiredExternal)) next.providers[provider] = { ...(next.providers[provider] ?? {}), enabled: true }; + } + } + return { + changed: true, + detail: `agentOverrides: ${agentOverridesSupported ? Object.keys(projected).length : 0} agents` + + (referenced.length ? ` (providers enabled: ${referenced.join(', ')})` : '') + + (staleOverrides.length ? ` (${staleOverrides.length} stale ak entries pruned)` : '') + + (!agentOverridesSupported ? ' (new projection skipped; needs agentic-qe ≥ 3.13.1)' : ''), + }; + } + if (hasPolicy && !agentOverridesSupported) return { detail: 'agentOverrides: skipped (needs agentic-qe ≥ 3.13.1)' }; + if (hasPolicy && Object.keys(projected).length === 0) return { detail: 'agentOverrides: skipped (no safely constructible providers)' }; + return null; +} + +const AQE_ROUTER_SURFACES = [ + reconcileExternalProvidersSurface, + reconcileFallbackRetirementSurface, + reconcileDefaultProviderSurface, + reconcileAgentOverridesSurface, +]; + +/** Fold an ordered list of `(draft, ctx) => {detail, error, changed, ctx?}` + * surface reconcilers over one draft, left to right. A surface's own `ctx` + * patch (if any) is applied before the next surface runs — the only + * sanctioned channel for one surface's output to inform a later one (see the + * section comment above AQE_ROUTER_SURFACES). `draft`/`ctx` are mutated in + * place as usual; returns the accumulated {details, changed, error}. */ +function foldSurfaces(surfaces, draft, ctx) { + const details = []; + let changed = false; + let error = null; + for (const reconcile of surfaces) { + const result = reconcile(draft, ctx); + if (!result) continue; + if (result.detail) { + if (Array.isArray(result.detail)) details.push(...result.detail); + else details.push(result.detail); + } + if (result.changed) changed = true; + if (result.error) error ??= result.error; + if (result.ctx) Object.assign(ctx, result.ctx); + } + return { details, changed, error }; +} + +/** True when nothing in `cfg`/the on-disk file requires any router surface to + * run — the router file is left untouched (and unread beyond this check). */ +function aqeRouterHasNothingToApply({ + hasChain, hasPolicy, hasExternal, hasOwnedExternal, hasManagedFallback, + hasExternalDefaultReceipt, hasFallbackDefaultReceipt, staleOverrides, +}) { + return !hasChain && !hasPolicy && !hasExternal && !hasOwnedExternal && !hasManagedFallback + && !hasExternalDefaultReceipt && !hasFallbackDefaultReceipt && staleOverrides.length === 0; +} + +/** Exact receipts never regain authority. If a user changes the default away + * from the value ak wrote (external default), or the managed fallback chain + * that derived a default is gone or no longer owned, relinquish that receipt + * immediately — changing it back later is still a user write and cannot + * resurrect it. Runs before any surface, on the initial draft. */ +function clearStaleDefaultReceipts(next, { + hasExternalDefaultReceipt, ownedExternalDefault, hasFallbackDefaultReceipt, ownedFallbackDefault, hasManagedFallback, +}) { + if (hasExternalDefaultReceipt && !ownedExternalDefault) clearExternalDefaultOwnership(next); + if (hasFallbackDefaultReceipt && (!ownedFallbackDefault || !hasManagedFallback)) clearFallbackDefaultOwnership(next); +} + +/** Build the read-only context the AQE-router fold needs: repo-root resolution + * (same gate as settingsTarget — the scope gates must never disagree about + * what "in a project" means), the on-disk router file, and every derived + * fact/flag the surfaces consume. Returns null outside a project. Split out + * of applyAqeRouter so a read-only comparator (aqeRouterDrift) can ask "what + * would the writer converge this to" without ever touching disk (#129) — + * the ONE construction of this context, shared by the writer and the reader. */ +function buildAqeRouterContext(cfg, cwd) { + const chain = cfg.providers?.aqeFallback ?? []; + const policy = cfg.routing?.routes ?? {}; + const selectedProvider = cfg.providers?.aqeProvider ?? null; + const hasChain = chain.length > 0; + const hasPolicy = Object.keys(policy).length > 0; + const root = repoRoot(cwd); + if (!root) return null; + const file = aqeRouterFile(root); + const existing = readJson(file, {}) ?? {}; + const ownedExternalDefault = exactlyOwnedExternalDefault(existing); + const ownedFallbackDefault = exactlyOwnedFallbackDefault(existing); + const desiredExternal = aqeExternalProviders({ projectRoot: root }); + const hasExternal = Object.keys(desiredExternal).length > 0; + const hasOwnedExternal = Object.keys(existing[AQE_OWNERSHIP_KEY]?.externalProviders ?? {}).length > 0; + const existingOwnership = plainRecord(existing[AQE_OWNERSHIP_KEY]) ?? {}; + const hasExternalDefaultReceipt = Object.hasOwn(existingOwnership, 'externalDefaultProvider'); + const hasFallbackDefaultReceipt = Object.hasOwn(existingOwnership, 'fallbackDefaultProvider'); + const hasManagedFallback = existing.fallbackChain?.id === AQE_MANAGED_TAG; + const priorOverrides = existing.agentOverrides ?? {}; + const projected = configuredPolicyToAgentOverrides(policy); + const managedOverrideKeys = new Set(Object.keys(AGENT_ACTIVITY_MAP)); + const staleOverrides = Object.keys(priorOverrides) + .filter((agent) => managedOverrideKeys.has(agent) && !(agent in projected)); + + const facts = { + hasChain, hasPolicy, hasExternal, hasOwnedExternal, hasManagedFallback, + hasExternalDefaultReceipt, hasFallbackDefaultReceipt, staleOverrides, + }; + const ctx = { + cfg, + existing, + chain, + selectedProvider, + hasChain, + hasPolicy, + desiredExternal, + hasExternal, + hasOwnedExternal, + hasManagedFallback, + ownedExternalDefault, + ownedFallbackDefault, + priorOverrides, + managedOverrideKeys, + projected, + staleOverrides, + externalActive: new Set(), + externalSupported: aqeSupportsExternalProviders(), + agentOverridesSupported: aqeSupportsAgentOverrides(), + }; + return { + root, file, existing, facts, ctx, + }; +} + +/** Run the ordered AQE_ROUTER_SURFACES fold over a fresh draft cloned from + * `existing` — mutates neither `existing` nor disk. This is the pure "what + * would the writer converge this to" computation shared by applyAqeRouter + * (which persists the result when it differs) and aqeRouterDrift (which only + * needs to know what the writer WOULD produce). */ +function runAqeRouterFold({ existing, facts, ctx }) { + const next = { ...existing }; + clearStaleDefaultReceipts(next, { + hasExternalDefaultReceipt: facts.hasExternalDefaultReceipt, + ownedExternalDefault: ctx.ownedExternalDefault, + hasFallbackDefaultReceipt: facts.hasFallbackDefaultReceipt, + ownedFallbackDefault: ctx.ownedFallbackDefault, + hasManagedFallback: facts.hasManagedFallback, + }); + const { details, changed, error } = foldSurfaces(AQE_ROUTER_SURFACES, next, ctx); + return { + next, details, changed, error, + }; +} + +/** Write ak's managed router config into `.agentic-qe/llm-config.json`, merged + * into any existing file (backup-first, never persisting apiKey): + * - the ordered fallback chain + enabled set + default provider (from + * `aqeFallback`), and + * - the per-activity `agentOverrides` map projected from `routing.routes` + * (issue #568; only when installed aqe ≥ 3.13.1). + * No-op unless at least one of those is configured and we are in a project. + * Folds AQE_ROUTER_SURFACES over one draft (see the module comment above); + * this function is the setup (context + initial draft), the fold, and the + * final change-detect-and-write. + * Returns {ok, changed, detail}. */ +export function applyAqeRouter(cfg, cwd = process.cwd()) { + const built = buildAqeRouterContext(cfg, cwd); + if (!built) return { ok: true, changed: false, detail: 'not a project — aqe router unmanaged' }; + const { file, existing, facts } = built; + if (aqeRouterHasNothingToApply(facts)) { + return { ok: true, changed: false, detail: 'no aqe router config to apply' }; + } + + const { next, details, changed: surfacesChanged, error } = runAqeRouterFold(built); + + // One exact compare, reused for both phases below (the prior version + // stringified `existing` twice for the same never-mutated object). + const existingSnapshot = JSON.stringify(stableValue(existing)); + const changed = surfacesChanged || JSON.stringify(stableValue(next)) !== existingSnapshot; + if (!changed) return { ok: !error, changed: false, detail: details.join('; ') || 'nothing to apply' }; + next._managedBy = AQE_MANAGED_TAG; + // A surface reporting `changed: true` means this invocation owns at least + // one projection surface; it does not by itself mean the artifact changed. + // Compare the complete managed value (including the ownership tag) before + // touching disk so a converged external default/fallback/override remains + // byte- and mtime-stable across repeated syncs. + if (JSON.stringify(stableValue(next)) === existingSnapshot) { + return { ok: !error, changed: false, detail: details.join('; ') || 'nothing to apply' }; + } + fs.mkdirSync(path.dirname(file), { recursive: true }); + writeJsonWithBackup(file, next); + return { ok: !error, changed: true, detail: details.join('; ') }; +} + +/** Read-only: does the persisted fallback-chain order in + * `.agentic-qe/llm-config.json` differ from what applyAqeRouter would + * converge it to right now? Runs the SAME dry-run fold the writer runs + * (buildAqeRouterContext + runAqeRouterFold) and reads only the + * fallback-chain slice of the result — the writer's own chain-validity + * filter (reconcileDefaultProviderSurface's `valid`), not a re-derived + * approximation, so the two can never disagree (#129). Scoped to chain order + * only, same as before: agentOverrides/external-provider drift are each a + * sibling status section's own concern. + * `applicable: false` means there is no chain to compare (nothing configured, + * or outside the project scope applyAqeRouter itself declines to manage). */ +export function aqeRouterDrift(cfg, cwd = process.cwd()) { + const chain = cfg.providers?.aqeFallback ?? []; + if (chain.length === 0) return { applicable: false, drift: false, order: '' }; + const built = buildAqeRouterContext(cfg, cwd); + if (!built) return { applicable: false, drift: false, order: '' }; + const { existing } = built; + const { next } = runAqeRouterFold(built); + const order = (next.fallbackChain?.entries ?? []).map((e) => e.provider).join('→'); + const diskOrder = (existing.fallbackChain?.entries ?? []).map((e) => e.provider).join('→'); + const drift = order ? (existing._managedBy !== AQE_MANAGED_TAG || diskOrder !== order) : diskOrder !== ''; + return { applicable: true, drift, order }; +} + +/** Reversible teardown of ak's router management. Restores the pre-ak file from + * its one-time .bak, or removes an ak-created file. Never touches a file ak + * didn't write (no `_managedBy` tag). */ +export function undoAqeRouter(cwd = process.cwd()) { + const file = aqeRouterFile(cwd); + if (!fs.existsSync(file)) return { ok: true, changed: false, detail: 'no aqe router config' }; + const cur = readJson(file); + if (cur?._managedBy !== AQE_MANAGED_TAG) return { ok: true, changed: false, detail: 'llm-config.json not ak-managed — left as-is' }; + const bak = `${file}.bak`; + if (fs.existsSync(bak)) { + fs.copyFileSync(bak, file); + fs.rmSync(bak, { force: true }); + return { ok: true, changed: true, detail: 'restored pre-ak llm-config.json' }; + } + fs.rmSync(file, { force: true }); + return { ok: true, changed: true, detail: 'removed ak-created llm-config.json' }; +} diff --git a/src/lib/dashboard/client/bootstrap.mjs b/src/lib/dashboard/client/bootstrap.mjs index 753bb89..b25872c 100644 --- a/src/lib/dashboard/client/bootstrap.mjs +++ b/src/lib/dashboard/client/bootstrap.mjs @@ -169,12 +169,11 @@ import { loadUsage } from './usage.mjs'; } if(!skipHash&&activeTab==="system")syncHash(); } - export function setTab(id,focus,skipHash){ - if(TABS.indexOf(id)<0)return; - if(activeTab==="observability"&&id!=="observability"&&window.AKLive)window.AKLive.deactivate(); - activeTab=id; - try{localStorage.setItem(LS_TAB,id);}catch(e){} - if(!skipHash)syncHash(); + // setTab was one CC-26 function mixing lazy per-tab data loads, the tab + // button/area/secondary-rail paint loop, and sub-view/scroll bookkeeping. + // Split the first two out (same call order as before); each keeps its + // original logic verbatim, so behavior is unchanged. + function tabLazyLoad(id){ // Usage is LAZY (ADR-0009 §2): the index is only read once the tab is // actually opened, never on the shared status poll. if(id==="usage"&&!usageLoaded)loadUsage(); @@ -183,6 +182,9 @@ import { loadUsage } from './usage.mjs'; // tier only (ADR-0025 §3). It never triggers a deep scan — a multi-second // walk on tab-open is exactly the hang the tiering exists to prevent. if(id==="system"&&!SYSTEM&&!systemBusy)loadSystem(); + } + + function paintTabButtons(id,focus){ for(var i=0;i
No model use was observed in this window. This does not mean no models are installed or available.
'; } - export function mliDetail(model){ - var detail=document.getElementById('mli-detail'),body=document.getElementById('mli-detail-body'),title=document.getElementById('mli-detail-title'); - if(!detail||!body||!title||!model)return; - var id=mliIdentity(model),life=model.lifecycle||{},variants=model.variant||{},dimensions=model.dimensions||{},observed=dimensions.observed||{}; + // mliDetail was one CC-44 function computing several dense ternary-chain + // text fields (published/access/routable/next-step status text, the + // lifecycle-scope/availability/retirement fields, and the ollama-only + // "local install" block) and then building the whole detail-panel DOM in + // one shot. Split the pure text/HTML computations out; each keeps its + // original logic verbatim, so the rendered DOM is unchanged. + function mliDetailStatusText(dimensions){ var published=dimensions.discoverable||{},entitled=dimensions.entitled||{},routable=dimensions.routable||{},configured=dimensions.configured||{}; var publishedText=published.value===true?'Published by an accepted source':published.value===false?'Not published by the accepted source':'Not established'; var accessText=entitled.value===true?'Established for the observed account and path':entitled.value===false?'Not entitled in the accepted evidence':'Not established; public metadata is not account access'; @@ -225,26 +228,43 @@ import { MODEL_PAGE, fmtNum, modelFilters, modelRows, modelsBusy } from './usage var nextStep=routable.value===true?'No evidence step needed; this exact path was observed working at capture time.' :(configured.value===true?'Complete one successful invocation on this exact path, then run ak models refresh.' :'Configure the exact model on an intended route, authenticate its serving provider, complete one successful invocation, then run ak models refresh.'); + return {publishedText:publishedText,accessText:accessText,routableText:routableText,nextStep:nextStep}; + } + + function mliDetailVariantFields(variants){ var lifecycleScope=variants.lifecycleScope?'
Lifecycle scope
'+esc(variants.lifecycleScope)+'
':''; var availability=variants.availability?'
Published availability
'+esc(variants.availability)+'
':''; var retirement=variants.retiredAt?'
Retired
'+esc(variants.retiredAt)+'
' :(variants.retirementNotBefore?'
Retirement commitment
Not before '+esc(variants.retirementNotBefore)+'
':''); - var local=id.provider==='ollama'?'
Local installation
Installed'+(variants.modifiedAt?' · updated '+esc(variants.modifiedAt):'')+'
' + return {lifecycleScope:lifecycleScope,availability:availability,retirement:retirement}; + } + + function mliDetailLocalBlock(id,variants){ + return id.provider==='ollama'?'
Local installation
Installed'+(variants.modifiedAt?' · updated '+esc(variants.modifiedAt):'')+'
' +'
Loaded now
'+esc(variants.loaded?'Yes':'No')+(variants.expiresAt?' · expires '+esc(variants.expiresAt):'')+'
' +'
Local model build
'+esc([variants.parameterSize,variants.quantizationLevel,variants.format].filter(Boolean).join(' · ')||'Not exposed')+'
' +'
Local memory
'+esc(variants.memoryBytes!=null?fmtBytes(variants.memoryBytes)+(variants.vramBytes!=null?' · VRAM '+fmtBytes(variants.vramBytes):''):'Not loaded')+'
':''; + } + + export function mliDetail(model){ + var detail=document.getElementById('mli-detail'),body=document.getElementById('mli-detail-body'),title=document.getElementById('mli-detail-title'); + if(!detail||!body||!title||!model)return; + var id=mliIdentity(model),life=model.lifecycle||{},variants=model.variant||{},dimensions=model.dimensions||{},observed=dimensions.observed||{}; + var status=mliDetailStatusText(dimensions); + var fields=mliDetailVariantFields(variants); + var local=mliDetailLocalBlock(id,variants); title.textContent=id.name; body.innerHTML='
' +'
Exact selector
'+esc(id.selector||'Not recorded')+'
' +'
Model provider
'+esc(id.provider||'Not recorded')+'
' +'
Publisher
'+esc(id.publisher||'Not independently proven')+'
' +'
Lifecycle
'+esc(life.state||'unknown')+(life.replacementName?' → '+esc(life.replacementName):'')+'
' - +lifecycleScope+availability+retirement + +fields.lifecycleScope+fields.availability+fields.retirement +'
Observed use
'+esc(observed&&observed.value===true?'Observed locally':'Not observed')+'
' - +'
Published / discovered
'+esc(publishedText)+'
' - +'
Account access
'+esc(accessText)+'
' - +'
Local routability
'+esc(routableText)+'
' - +'
What you need to do
'+esc(nextStep)+'
' + +'
Published / discovered
'+esc(status.publishedText)+'
' + +'
Account access
'+esc(status.accessText)+'
' + +'
Local routability
'+esc(status.routableText)+'
' + +'
What you need to do
'+esc(status.nextStep)+'
' +'
Context limit
'+esc(variants.contextWindow||model.capabilities&&model.capabilities.contextLimit||'Not established by accepted sources')+'
' +'
Capabilities
'+esc(mliCapabilities(model))+'
' +'
API rate / plan use
'+mliPrice(model.pricing,id.host)+'
' @@ -339,13 +359,17 @@ import { MODEL_PAGE, fmtNum, modelFilters, modelRows, modelsBusy } from './usage } } - export function renderModelLifecycle(){ - if(!MODELS)return; - var empty=MODELS.error||MODELS.status==="empty"||!MODELS.snapshot; - var snap=MODELS.snapshot||{},attention=snap.attention||[],bindings=snap.bindings||[]; + // renderModelLifecycle was one CC-27 function writing to ~10 unrelated DOM + // targets (badge/asof, attention list, routes+observed, history, consumers+ + // impact) in sequence. Split by target group; each keeps its original logic + // verbatim, so the rendered DOM and its ordering are unchanged. + function mliRenderBadgeAndAsof(empty,snap,attention){ var badge=document.getElementById("mli-attention-n"); if(badge){badge.hidden=!attention.length;badge.textContent=attention.length?String(attention.length):"";} document.getElementById("mli-asof").textContent=empty?"not captured":("captured "+String(snap.capturedAt||"").replace("T"," ").replace(".000Z","Z")); + } + + function mliRenderAttention(empty,attention){ document.getElementById("mli-attention").innerHTML=empty ?'
'+esc(MODELS.error||"No model inventory yet. Run ak models refresh explicitly.")+"
" :attention.map(function(item){ @@ -360,18 +384,25 @@ import { MODEL_PAGE, fmtNum, modelFilters, modelRows, modelsBusy } from './usage : ''+esc(item.reason||'Evidence needs review')+'. Run ak models refresh --all for current evidence.'; return '
'+esc(title)+'
'+detail+'
'; }).join(""); + } + + function mliRenderRoutesAndObserved(bindings){ document.getElementById('mli-routes').innerHTML=mliRouteRows(bindings); var observed=MODELS.observedWindow||{days:usageDays,models:[]}; document.getElementById('mli-observed').innerHTML=mliObservedRows(observed); document.getElementById('mli-observed-note').textContent=observed.status==='unavailable' ? String(observed.days||usageDays)+' days · use unavailable' : String(observed.days||usageDays)+' days · '+String((observed.models||[]).length)+' model'+((observed.models||[]).length===1?'':'s'); - renderModelRouteSort(); - renderModelInventory(); + } + + function mliRenderHistory(snap){ var changes=snap.changes||[]; var snapshotCount=(MODELS.history||[]).length; document.getElementById("mli-history-note").textContent=changes.length+" change"+(changes.length===1?"":"s")+' · '+snapshotCount+" retained snapshot"+(snapshotCount===1?"":"s"); document.getElementById("mli-history").innerHTML=mliChangeRows(changes); + } + + function mliRenderConsumersAndImpact(bindings){ var routeBindings=bindings.filter(function(binding){return binding.role&&binding.role!=="Configured consumer";}); document.getElementById("mli-consumers").innerHTML='
'+(routeBindings.map(function(binding){var model=binding.modelName||binding.configured||'Model not pinned';return '
'+esc(binding.consumer)+'
'+esc(model)+' · '+esc(mliProviderName(binding.modelProvider||binding.provider))+''+esc(binding.lastUsed?'last used '+String(binding.lastUsed).replace('T',' ').replace('.000Z','Z'):'not observed in this window')+'
';}).join("")||'
No configured model routes.
')+"
"; document.getElementById("mli-impact").innerHTML=routeBindings.length @@ -379,3 +410,16 @@ import { MODEL_PAGE, fmtNum, modelFilters, modelRows, modelsBusy } from './usage :'
No bound consumers to assess. A plan will remain read-only and report the missing binding.
'; } + export function renderModelLifecycle(){ + if(!MODELS)return; + var empty=MODELS.error||MODELS.status==="empty"||!MODELS.snapshot; + var snap=MODELS.snapshot||{},attention=snap.attention||[],bindings=snap.bindings||[]; + mliRenderBadgeAndAsof(empty,snap,attention); + mliRenderAttention(empty,attention); + mliRenderRoutesAndObserved(bindings); + renderModelRouteSort(); + renderModelInventory(); + mliRenderHistory(snap); + mliRenderConsumersAndImpact(bindings); + } + diff --git a/src/lib/dashboard/client/overview.mjs b/src/lib/dashboard/client/overview.mjs index d01c4fb..c9fbaf5 100644 --- a/src/lib/dashboard/client/overview.mjs +++ b/src/lib/dashboard/client/overview.mjs @@ -124,11 +124,13 @@ import { fmtNum } from './usage.mjs'; } function flat(msg){return '
'+esc(msg)+"
";} - export function renderHistory(data){ - var strip=document.getElementById("history"); - var note=document.getElementById("strip-note"); - var series=[]; - if(data.health&&data.health.length){series=data.health;} + // renderHistory was one CC-63 function mixing four independent data-shaping + // computations (patterns/deltas, pattern-store series, graph series, curve + // values -- each dense with ternaries/&&/||, which is what drove the count) + // with five independent sparkline renders. Split by concern; every + // computation and every render keeps its original logic verbatim, so the + // rendered DOM is unchanged. + function historySeriesAndDeltas(data,series){ var pats=[],deltas=[]; for(var i=0;i1?sparkline(pats):flat(pats.length?String(pats[0])+" (one sample)":"no data"); + } + function renderPatternStoreSpark(storeSeries,storeTotal){ document.getElementById("spark-pattern-store").innerHTML=storeSeries.length>1?sparkline(storeSeries):flat(storeSeries.length?String(storeTotal)+" entries (one day)":"no data"); + } + function renderGraphSpark(nodesSeries,lastGraph){ document.getElementById("spark-graph").innerHTML=nodesSeries.length>1?sparkline(nodesSeries):flat(nodesSeries.length?String(nodesSeries[0])+" nodes (one sample)":"no data"); var graphMeta=document.getElementById("graph-meta"); if(graphMeta)graphMeta.textContent=lastGraph?("latest: "+fmtNum(lastGraph.nodes)+" nodes · "+fmtNum(lastGraph.edges)+" edges"):""; + } + function renderDeltaSpark(deltas,imp){ document.getElementById("spark-delta").innerHTML=deltas.length>1?sparkline(deltas):flat(deltas.length?(deltas[0]>=0?"+":"")+deltas[0]+"pp (one sample)":"no data"); var deltaMeta=document.getElementById("delta-meta"); if(deltaMeta){ @@ -210,7 +213,43 @@ import { fmtNum } from './usage.mjs'; +'p'+esc(pTxt)+" · d="+esc(dTxt)+""; } } + } + function renderCurveSpark(curveVals){ document.getElementById("spark-curve").innerHTML=curveVals.length>1?sparkline(curveVals):flat(curveVals.length?(curveVals[0]*100).toFixed(0)+"% (one sample)":"no data"); } + export function renderHistory(data){ + var strip=document.getElementById("history"); + var note=document.getElementById("strip-note"); + var series=[]; + if(data.health&&data.health.length){series=data.health;} + var sd=historySeriesAndDeltas(data,series),pats=sd.pats,deltas=sd.deltas; + var ps=historyPatternStoreSeries(data),storeSeries=ps.storeSeries,storeTotal=ps.storeTotal; + var gs=historyGraphSeries(data),nodesSeries=gs.nodesSeries,lastGraph=gs.lastGraph; + var imp=data.improvement||null; + var curveVals=historyCurveValues(imp); + + // The project PICKER lives in this strip's head, so the strip itself must + // stay visible even when the selected project has nothing to chart — + // hiding it would strand the user on an empty project with no control to + // pick a different one. Only the charts collapse. + var sparkRow=document.getElementById("spark-row"); + var emptyEl=document.getElementById("history-empty"); + var nothing=!pats.length&&!deltas.length&&!storeSeries.length&&!nodesSeries.length&&!curveVals.length; + strip.hidden=false; + if(sparkRow)sparkRow.hidden=nothing; + if(emptyEl){ + emptyEl.hidden=!nothing; + if(nothing)emptyEl.textContent="no learning history recorded for "+(selectedProjectLabel||"this project")+" yet."; + } + if(nothing){note.textContent="";return;} + note.textContent=(series.length?series.length+" samples":"snapshot")+(intelSource?" · live":""); + + renderPatternsSpark(pats); + renderPatternStoreSpark(storeSeries,storeTotal); + renderGraphSpark(nodesSeries,lastGraph); + renderDeltaSpark(deltas,imp); + renderCurveSpark(curveVals); + } + diff --git a/src/lib/dashboard/client/system-projects.mjs b/src/lib/dashboard/client/system-projects.mjs index c144502..42d8f9b 100644 --- a/src/lib/dashboard/client/system-projects.mjs +++ b/src/lib/dashboard/client/system-projects.mjs @@ -12,10 +12,14 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; // for in the panel's footnote rather than dropped silently. var REAL_HOSTS={claude:true,codex:true,opencode:true}; - function renderSysStorage(d){ - var s=d.storage,i,j; - - // ── learning stores, lifted out of the shared charts ── + // renderSysStorage was one CC-88 function mixing five independent DOM + // regions (learning stores, donut, per-host split, growth sparks, top + // sessions) plus a call into renderSysReclaim. Split one function per + // region -- each keeps its own element-null guard and reads straight off + // the storage payload `s`, so the rendered DOM is unchanged; only the + // per-region branching no longer inflates one shared complexity count. + function renderSysLearning(s){ + var i; var learn=document.getElementById("sys-learning"); if(learn){ if(!s){learn.innerHTML=sysEmpty(NOT_SCANNED);} @@ -39,6 +43,10 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; } } + } + + function renderSysDonut(s){ + var i; var donut=document.getElementById("sys-donut"); if(donut){ if(!s){donut.innerHTML=sysEmpty(NOT_SCANNED);} @@ -62,61 +70,66 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; :sysEmpty("no storage category could be measured."); } } + } + + function renderSysHostSplit(s){ + var i,j; var split=document.getElementById("sys-hostsplit"); - if(split){ - if(!s){split.innerHTML=sysEmpty(NOT_SCANNED);} - else{ - var byHost={},order=[],cats=s.categories||[],otherBytes=0,otherKeys={}; - for(i=0;i0?(row.parts[j].bytes/scale)*100:0).toFixed(2) - +"%;background:"+row.parts[j].color+'" title="'+esc(row.parts[j].label)+'">'; - } - rowsHtml+='
'+esc(row.key)+"" - +'
'+seg+"
" - +''+esc(fmtBytes(row.total))+"
"; + if(!split)return; + if(!s){split.innerHTML=sysEmpty(NOT_SCANNED);return;} + var byHost={},order=[],cats=s.categories||[],otherBytes=0,otherKeys={}; + for(i=0;i'+esc(cats[i].label)+""; - } - // Name what is NOT in the rows, with its figure. Dropping the non-host - // rows silently would leave the bars failing to add up to the donut - // beside them, with nothing on screen explaining the gap. - var otherNames=[]; - for(var ok in otherKeys)if(Object.prototype.hasOwnProperty.call(otherKeys,ok))otherNames.push(ok); - otherNames.sort(); - var footnote=otherNames.length - ?'

Hosts only. A further '+esc(fmtBytes(otherBytes))+" belongs to " - +esc(otherNames.join(" and "))+" — ak's own state, not a host's. Learning stores are on their own card.

" - :'

Learning stores are on their own card, not counted here.

'; - split.innerHTML=order.length - ?('
'+rowsHtml+'
'+catLegend+"
"+footnote) - :sysEmpty("no per-host storage node could be measured."); + if(CHART_EXCLUDED_CATEGORIES[cats[i].key])continue; + if(!byHost[kids[j].key]){byHost[kids[j].key]={key:kids[j].key,parts:[],total:0};order.push(kids[j].key);} + byHost[kids[j].key].parts.push({color:catColor(cats[i].key),bytes:b, + label:kids[j].key+" \u00b7 "+cats[i].label+" "+fmtBytes(b)}); + byHost[kids[j].key].total+=b; + } + } + order.sort(function(a,b2){return byHost[b2].total-byHost[a].total;}); + var scale=order.length?byHost[order[0]].total:0,rowsHtml=""; + for(i=0;i0?(row.parts[j].bytes/scale)*100:0).toFixed(2) + +"%;background:"+row.parts[j].color+'" title="'+esc(row.parts[j].label)+'">'; } + rowsHtml+='
'+esc(row.key)+"" + +'
'+seg+"
" + +''+esc(fmtBytes(row.total))+"
"; + } + var catLegend=""; + for(i=0;i'+esc(cats[i].label)+""; } + // Name what is NOT in the rows, with its figure. Dropping the non-host + // rows silently would leave the bars failing to add up to the donut + // beside them, with nothing on screen explaining the gap. + var otherNames=[]; + for(var ok in otherKeys)if(Object.prototype.hasOwnProperty.call(otherKeys,ok))otherNames.push(ok); + otherNames.sort(); + var footnote=otherNames.length + ?'

Hosts only. A further '+esc(fmtBytes(otherBytes))+" belongs to " + +esc(otherNames.join(" and "))+" — ak's own state, not a host's. Learning stores are on their own card.

" + :'

Learning stores are on their own card, not counted here.

'; + split.innerHTML=order.length + ?('
'+rowsHtml+'
'+catLegend+"
"+footnote) + :sysEmpty("no per-host storage node could be measured."); + } + + function renderSysGrowth(s){ + var i; var growth=document.getElementById("sys-growth"); if(growth){ var g=s&&s.growth; @@ -146,73 +159,89 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; +esc(String(g.basis||"file mtime and size only"))+""; } } - renderSysReclaim(s); + } + + function sysTopSessionsAttributable(sess){ + var attributable=[],unattributable=0,i; + for(i=0;i0?(x.bytes/ht)*100:null; + // Link to the transcript the same way Usage does, through the public + // bridge it already exposes. The id has to be normalised first: + // Storage's session is the FILE BASENAME, while /api/session wants + // Usage's form. A row we cannot address renders as plain text — a + // dead link is worse than no link. + var sid=transcriptIdOf(x); + // Strip the extension rather than truncating mid-id: a uuid cut at 34 + // characters reads as a corrupted value. + var sname=String(x.session||""); + if(sname.slice(-6)===".jsonl")sname=sname.slice(0,-6); + var cell=esc(sname); + return "" + +'' + +(sid?'":cell) + +"" + +''+esc(x.host||"\u2014")+"" + // An undecoded name says WHICH reason. "deleted project" is a + // claim, and on Windows it would be a false one for every row: the + // encoding there carries a drive prefix that the decoder refuses by + // design, so nothing is decodable and nothing has been deleted. + +""+(x.projectResolved===false + ?''+(x.projectReason==="encoding"?"name not decodable":"deleted project")+"" + :esc(x.projectLabel||x.project))+"" + +''+esc(fmtBytes(x.bytes))+"" + +""+(share==null + ?unkHtml("this host's retained total was not measured",false) + :('
')) + +""; + } + + function renderSysTopSessions(s){ var top=document.getElementById("sys-topsessions"); - if(top){ - var sess=(s&&s.topSessions)||null; - if(!s){top.innerHTML=sysEmpty(NOT_SCANNED);} - else if(!sess||!sess.length){top.innerHTML=sysEmpty("no session files were measured.");} - else{ - // Attributable rows only. A row whose project cannot be named is not a - // useful entry in a list whose whole job is "which project is holding - // these bytes" — the unattributable ones are counted in the liner - // instead, so they are excluded rather than hidden. - var attributable=[],unattributable=0; - for(i=0;i0?(x.bytes/ht)*100:null; - // Link to the transcript the same way Usage does, through the public - // bridge it already exposes. The id has to be normalised first: - // Storage's session is the FILE BASENAME, while /api/session wants - // Usage's form. A row we cannot address renders as plain text — a - // dead link is worse than no link. - var sid=transcriptIdOf(x); - // Strip the extension rather than truncating mid-id: a uuid cut at 34 - // characters reads as a corrupted value. - var sname=String(x.session||""); - if(sname.slice(-6)===".jsonl")sname=sname.slice(0,-6); - var cell=esc(sname); - body+="" - +'' - +(sid?'":cell) - +"" - +''+esc(x.host||"\u2014")+"" - // An undecoded name says WHICH reason. "deleted project" is a - // claim, and on Windows it would be a false one for every row: the - // encoding there carries a drive prefix that the decoder refuses by - // design, so nothing is decodable and nothing has been deleted. - +""+(x.projectResolved===false - ?''+(x.projectReason==="encoding"?"name not decodable":"deleted project")+"" - :esc(x.projectLabel||x.project))+"" - +''+esc(fmtBytes(x.bytes))+"" - +""+(share==null - ?unkHtml("this host's retained total was not measured",false) - :('
')) - +""; - } - top.innerHTML='
' - +'' - +body+"
SessionHostProjectSizeShare of host
" - +(unattributable?'
'+esc(fmtNum(unattributable)) - +" larger session file"+(unattributable===1?"":"s")+" could not be attributed to a project " - +"and "+(unattributable===1?"is":"are")+" not listed.
":""); - } - } + if(!top)return; + var sess=(s&&s.topSessions)||null; + if(!s){top.innerHTML=sysEmpty(NOT_SCANNED);return;} + if(!sess||!sess.length){top.innerHTML=sysEmpty("no session files were measured.");return;} + var attr=sysTopSessionsAttributable(sess),attributable=attr.attributable,unattributable=attr.unattributable; + if(!attributable.length){ + top.innerHTML=sysEmpty("no session file could be attributed to a project."); + return; } + var hostTotals=storageHostTotals(s); + var body=attributable.map(function(x){return sysTopSessionRowHtml(x,hostTotals);}).join(""); + top.innerHTML='
' + +'' + +body+"
SessionHostProjectSizeShare of host
" + +(unattributable?'
'+esc(fmtNum(unattributable)) + +" larger session file"+(unattributable===1?"":"s")+" could not be attributed to a project " + +"and "+(unattributable===1?"is":"are")+" not listed.
":""); } - function renderSysRuntime(d){ - var rt=d.runtime||{},i; + function renderSysStorage(d){ + var s=d.storage; + renderSysLearning(s); + renderSysDonut(s); + renderSysHostSplit(s); + renderSysGrowth(s); + renderSysReclaim(s); + renderSysTopSessions(s); + } + + // renderSysRuntime was one CC-39 function mixing three independent DOM + // regions (process table, memory band, daemon tiles). Split one function + // per region; each keeps its own element-null guard and reads off the + // runtime payload `rt`, so the rendered DOM is unchanged. + function renderSysProcs(rt){ + var i; var procs=document.getElementById("sys-procs"); if(procs){ var pm=rt.processes; @@ -249,6 +278,9 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; +"RSS"+body+""; } } + } + + function renderSysMem(rt){ var mem=document.getElementById("sys-mem"); if(mem){ var tot=rt.totals||{},mach=rt.machine||{}; @@ -265,6 +297,9 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; :'
'+unkHtml("the physical-memory denominator was not reported",false) +" \u2014 no share of memory can be stated
"); } + } + + function renderSysDaemons(rt){ var dae=document.getElementById("sys-daemons"); if(dae){ var dm=rt.daemons||{},ttl=Number(dm.ttlSecs)||0,oldest=mval(dm.oldestAgeSecs); @@ -287,8 +322,54 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; } } + function renderSysRuntime(d){ + var rt=d.runtime||{}; + renderSysProcs(rt); + renderSysMem(rt); + renderSysDaemons(rt); + } + + function renderSysRadar(c){ + var radar=document.getElementById("sys-radar"); + if(!radar)return; + var kinds=c.kinds||[],hosts=c.hosts||[],i,j; + var axes=[],series=[]; + for(j=0;jmax)max=v; + } + axes.push({label:KIND_LABEL[kinds[j]]||kinds[j],max:max}); + } + for(i=0;i'+esc(hosts[i])+""; + radar.innerHTML=svgRadar(axes,series)+'
'+legend+"
"; + } + + function renderSysCatCounts(c){ + var countsEl=document.getElementById("sys-catcounts"); + if(!countsEl)return; + var kinds=c.kinds||[],j; + var tiles=""; + for(j=0;j" + +'
unique '+esc(KIND_PLURAL[kinds[j]]||kinds[j])+"
"; + } + countsEl.innerHTML='
'+tiles+"
"; + } + function renderSysCatalog(d){ - var c=d.catalog,i,j; + var c=d.catalog; var radar=document.getElementById("sys-radar"); var countsEl=document.getElementById("sys-catcounts"); var matrix=document.getElementById("sys-matrix"); @@ -298,38 +379,8 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; if(matrix)matrix.innerHTML=""; return; } - var kinds=c.kinds||[],hosts=c.hosts||[]; - if(radar){ - var axes=[],series=[]; - for(j=0;jmax)max=v; - } - axes.push({label:KIND_LABEL[kinds[j]]||kinds[j],max:max}); - } - for(i=0;i'+esc(hosts[i])+""; - radar.innerHTML=svgRadar(axes,series)+'
'+legend+"
"; - } - if(countsEl){ - var tiles=""; - for(j=0;j" - +'
unique '+esc(KIND_PLURAL[kinds[j]]||kinds[j])+"
"; - } - countsEl.innerHTML='
'+tiles+"
"; - } + renderSysRadar(c); + renderSysCatCounts(c); if(matrix){ // The kind heading rows are gone: they told you what you were looking at // but gave you no way to look at less of it, and on a 318-row inventory @@ -560,6 +611,60 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; +""; } + // renderSysProjects was one CC-33 function mixing eligibility filtering, + // per-row HTML building, and final table+liner assembly. Split by concern; + // each keeps the exact original logic unchanged. (The row builder also + // drops a dead `_diskBar` computation that was built but never read in the + // original -- a no-op removal, not a behavior change.) + function sysProjectsEligible(all){ + var list=[],excluded=0; + for(var f=0;f0; + if(linked&&hosted)list.push(cand);else excluded++; + } + return {list:list,excluded:excluded}; + } + + function sysProjectNameCell(pr){ + var rem=pr.remote||null,name; + if(rem&&rem.status==="linked"&&/^https:/.test(String(rem.webUrl||""))){ + name='' + +esc(pr.label)+" ↗"; + }else{ + name=esc(pr.label); + } + return name; + } + + function sysProjectRowHtml(pr){ + var name=sysProjectNameCell(pr); + var last=mval(pr.lastActivity); + return ""+name+"" + +''+mhtml(pr.loc&&pr.loc.total,function(v){return "~"+fmtTok(v);})+"" + +""+langCell(pr.loc)+"" + +''+mhtml(pr.totalBytes,fmtBytes)+"" + +''+(last==null?unkHtml((pr.lastActivity&&pr.lastActivity.reason)||"no readable entry",false) + :esc(ago(Math.max(0,Math.round((Date.now()-last)/1000)))))+""; + } + + function sysProjectsLinerHtml(p,list,excluded){ + return '
' + +(p.everSeen + ?mhtml(p.everSeen)+" projects ever seen across all hosts, " + +(p.onDisk?mhtml(p.onDisk):"some")+" still on disk" + :mhtml(p.count)+" projects measured (this snapshot predates the ever-seen count)") + +", "+esc(fmtNum(list.length))+" listed here." + +(excluded + ? " Excluded "+esc(fmtNum(excluded))+" measured director"+(excluded===1?"y":"ies") + +" with no remote or no recorded session \u2014 agent worktrees, sub-folders of a " + +"repository already listed, and repositories with no remote." + : "") + +" Line counts are approximate: extension-bucketed, with node_modules and vendored " + +"trees excluded. Disk is the whole project directory, .git and node_modules included.
"; + } + export function renderSysProjects(d){ var el=document.getElementById("sys-projects"); if(!el)return; @@ -588,54 +693,14 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; // // A genuine local-only repository is excluded too. That is the cost of the // rule, and it is why the count is stated below rather than left implied. - var list=[],excluded=0; - for(var f=0;f0; - if(linked&&hosted)list.push(cand);else excluded++; - } + var elig=sysProjectsEligible(all),list=elig.list,excluded=elig.excluded; if(!list.length){ el.innerHTML=sysEmpty("no project with a remote and a recorded session was measured \u2014 " +fmtNum(excluded)+" measured director"+(excluded===1?"y was":"ies were")+" excluded."); return; } list=sortProjects(list,projSort.key,projSort.dir); - var body="",i; - for(i=0;i0){ - // One entity's ranked parts — shades of ONE hue, darkest for the part - // the user wrote, faintest for the reinstallable overhead. - if(tree!=null)_diskBar+=''; - if(git!=null)_diskBar+=''; - if(nm!=null)_diskBar+=''; - } - // The remote sub-line and the stack chips are gone: this table answers - // "how big is each project and what is it written in". A forge slug and a - // row of presence-only chips answered neither, and between them they owned - // half the row's height. The project still LINKS to its remote when it has - // an https one — the affordance was worth keeping, the metadata was not. - var rem=pr.remote||null,name; - if(rem&&rem.status==="linked"&&/^https:/.test(String(rem.webUrl||""))){ - name='' - +esc(pr.label)+" ↗"; - }else{ - name=esc(pr.label); - } - var last=mval(pr.lastActivity); - body+=""+name+"" - +''+mhtml(pr.loc&&pr.loc.total,function(v){return "~"+fmtTok(v);})+"" - +""+langCell(pr.loc)+"" - +''+mhtml(pr.totalBytes,fmtBytes)+"" - +''+(last==null?unkHtml((pr.lastActivity&&pr.lastActivity.reason)||"no readable entry",false) - :esc(ago(Math.max(0,Math.round((Date.now()-last)/1000)))))+""; - } + var body=list.map(sysProjectRowHtml).join(""); // Legend covers only what still renders: the language ramp. The disk column // is a single figure now, and there are no chips left to explain. el.innerHTML='
' @@ -654,24 +719,9 @@ import { fmtNum, fmtTok, limAge, pct } from './usage.mjs'; // Three numbers now, and the gap between the last two is a filter rather // than a fact about the machine — so it is named. Leaving the reader to // subtract 25 from 16 and guess is the silent exclusion ADR-0023 forbids. - +'
' - +(p.everSeen - ?mhtml(p.everSeen)+" projects ever seen across all hosts, " - +(p.onDisk?mhtml(p.onDisk):"some")+" still on disk" - :mhtml(p.count)+" projects measured (this snapshot predates the ever-seen count)") - +", "+esc(fmtNum(list.length))+" listed here." - +(excluded - ? " Excluded "+esc(fmtNum(excluded))+" measured director"+(excluded===1?"y":"ies") - +" with no remote or no recorded session \u2014 agent worktrees, sub-folders of a " - +"repository already listed, and repositories with no remote." - : "") - +" Line counts are approximate: extension-bucketed, with node_modules and vendored " - +"trees excluded. Disk is the whole project directory, .git and node_modules included.
"; + +sysProjectsLinerHtml(p,list,excluded); } - // Freshness is a contract, not a caption (ADR-0025 §3): every deep figure on - // this page was measured at ONE moment, and the label says which. Past the - // staleness horizon it nudges — but it still never scans on its own. export function renderSystemFreshness(){ var el=document.getElementById("sys-asof"),btn=document.getElementById("sys-rescan"); if(!el)return; diff --git a/src/lib/dashboard/client/system-readout.mjs b/src/lib/dashboard/client/system-readout.mjs index 7520dc1..4bbd313 100644 --- a/src/lib/dashboard/client/system-readout.mjs +++ b/src/lib/dashboard/client/system-readout.mjs @@ -384,7 +384,10 @@ import { fmtNum, fmtTok } from './usage.mjs'; if(p.truncated)note+=" The measured list is capped, so fewer rows than on-disk projects."; return note; } - export function renderSysSummary(d){ + // renderSysSummary was one CC-28 function mixing the KPI band, the disk + // gauge band, and a call into renderSysConsumers. Split by region; each + // keeps its original logic verbatim, so the rendered DOM is unchanged. + function renderSysKpis(d){ var install=d.install,storage=d.storage,catalog=d.catalog,projects=d.projects; var rt=d.runtime||{},totals=rt.totals||{}; var kpis=document.getElementById("sys-kpis"); @@ -414,6 +417,10 @@ import { fmtNum, fmtTok } from './usage.mjs'; var kpiNote=document.getElementById("sys-kpis-note"); if(kpiNote)kpiNote.innerHTML=projectsLiner(projects); + } + + function renderSysGaugeBand(d){ + var install=d.install,storage=d.storage; var band=document.getElementById("sys-gauge"); if(band){ var disk=(install&&install.disk)||null; @@ -426,6 +433,11 @@ import { fmtNum, fmtTok } from './usage.mjs'; band.innerHTML=diskBand(used,data,total==null?0:total,free); } } + } + + export function renderSysSummary(d){ + renderSysKpis(d); + renderSysGaugeBand(d); renderSysConsumers(d); } diff --git a/src/lib/dashboard/client/usage.mjs b/src/lib/dashboard/client/usage.mjs index d9cf115..8a79c56 100644 --- a/src/lib/dashboard/client/usage.mjs +++ b/src/lib/dashboard/client/usage.mjs @@ -333,7 +333,12 @@ import { renderUsage } from './usage-orchestrators.mjs'; }).join(""); } - export function renderScore(d){ + // renderScore was one CC-41 function writing ~10 independent scorecard + // regions (hero KPIs, cost-per-day bars, host cards, token bar/legend, + // punchcard, models, OpenRouter, projects, categories) in sequence. Split + // by region; each keeps its original logic verbatim, so the rendered DOM + // and its ordering are unchanged. + function renderScoreHero(d){ var t=d.totals||{}; var cacheShare=pct(t.cacheRead,t.tokens); document.getElementById("u-hero").innerHTML= @@ -346,6 +351,9 @@ import { renderUsage } from './usage-orchestrators.mjs'; +kpi("cache read",cacheShare.toFixed(1)+"%","priced at 0.1× input","warnv"); document.getElementById("u-asof").textContent=d.pricesAsOf?("rates as of "+d.pricesAsOf):""; + } + + function renderScoreDayBars(d){ // cost per day var days=[],k; for(k in (d.byDay||{}))days.push({day:k,v:d.byDay[k]}); @@ -362,6 +370,10 @@ import { renderUsage } from './usage-orchestrators.mjs'; }).join(""):'
no days in window.
'; renderTelemetryCoverage(d.sourceHealth); + } + + function renderScoreHosts(d){ + var k; // Host and inference-provider are independent canonical axes. All three // supported hosts always render (idle/grayed-out when a host has no // sessions in this window) rather than appearing/disappearing based on @@ -384,6 +396,10 @@ import { renderUsage } from './usage-orchestrators.mjs'; var hostsNoteEl=document.getElementById("u-hosts-note"); if(hostsNoteEl)hostsNoteEl.textContent=activeHosts+" active of "+order.length; + } + + function renderScoreTokBar(d){ + var t=d.totals||{}; var segs=[["cache read",t.cacheRead,"var(--warn)"],["cache write",t.cacheWrite,"var(--purple)"], ["output",t.output,"var(--accent)"],["input",t.input,"var(--ok)"]]; document.getElementById("u-tokbar").innerHTML=segs.map(function(sg){ @@ -393,6 +409,9 @@ import { renderUsage } from './usage-orchestrators.mjs'; return ''+esc(sg[0])+" "+esc(fmtTok(sg[1]))+""; }).join(""); + } + + function renderScorePunchcard(d){ // punchcard — dow 0 = Mon var DOW=["Mon","Tue","Wed","Thu","Fri","Sat","Sun"], pcMax=0, key; for(key in (d.punchcard||{}))pcMax=Math.max(pcMax,Number(d.punchcard[key])||0); @@ -412,6 +431,10 @@ import { renderUsage } from './usage-orchestrators.mjs'; pcHtml+="
"; document.getElementById("u-punch").innerHTML=pcMax?pcHtml:'
no responses in window.
'; + } + + function renderScoreModels(d){ + var t=d.totals||{}; // models + projects var models=entries(d.byModel), mMax=models.length?models[0].cost:0; document.getElementById("u-models").innerHTML=models.length?models.map(function(m){ @@ -427,6 +450,9 @@ import { renderUsage } from './usage-orchestrators.mjs'; var exc=fld(t,"exceptions"); document.getElementById("u-models-note").textContent=exc?(fmtNum(exc)+" dropped/errored turn"+(exc===1?"":"s")+" excluded"):""; + } + + function renderScoreOpenRouter(d){ // Account analytics is explicitly fetched and cached by ak usage. // OpenRouter does not provide session/host/project correlation here, so // these numbers remain a separate block and never alter t/byHost/byModel. @@ -456,6 +482,9 @@ import { renderUsage } from './usage-orchestrators.mjs'; document.getElementById("u-openrouter").innerHTML=cards+'
'+modelRows+"
"; } + } + + function renderScoreProjects(d){ var projects=entries(d.byProject), pMax=projects.length?projects[0].cost:0; var shown=projects.slice(0,8); document.getElementById("u-projects-note").textContent= @@ -465,6 +494,9 @@ import { renderUsage } from './usage-orchestrators.mjs'; pct(pr.cost,pMax),true); }).join(""):'
no projects in window.
'; + } + + function renderScoreCategories(d){ // categories — confidence is DISPLAYED, and Unclassified is never hidden. var cats=entries(d.byCategory), cMax=cats.length?cats[0].cost:0; document.getElementById("u-cats").innerHTML=cats.length?cats.map(function(c){ @@ -480,6 +512,18 @@ import { renderUsage } from './usage-orchestrators.mjs'; }).join(""):'
nothing classified in window.
'; } + export function renderScore(d){ + renderScoreHero(d); + renderScoreDayBars(d); + renderScoreHosts(d); + renderScoreTokBar(d); + renderScorePunchcard(d); + renderScoreModels(d); + renderScoreOpenRouter(d); + renderScoreProjects(d); + renderScoreCategories(d); + } + // ══ Limits view (ADR-0010) ═════════════════════════════════════════════════ export var LIMITS=null, limitsBusy=false; @@ -522,13 +566,11 @@ import { renderUsage } from './usage-orchestrators.mjs'; +''+esc(sub||resetTxt(resetSec))+""; } - function renderLimits(){ + // renderLimits was one CC-28 function mixing the Claude window, the Codex + // lane, and the insights panel. Split by region; each keeps its original + // logic verbatim, so the rendered DOM is unchanged. + function renderLimitsClaude(){ var claudeEl=document.getElementById("u-lim-claude"); - var codexEl=document.getElementById("u-lim-codex"); - if(!claudeEl||!codexEl)return; - if(!LIMITS){claudeEl.innerHTML='
loading…
'; codexEl.innerHTML='
loading…
'; return;} - if(LIMITS.error){claudeEl.innerHTML='
'+esc(LIMITS.error)+"
"; codexEl.innerHTML=""; return;} - var c=LIMITS.claude; var cn=document.getElementById("u-lim-claude-note"); if(c&&c.windows&&c.windows.length){ @@ -544,6 +586,10 @@ import { renderUsage } from './usage-orchestrators.mjs'; +"with the kit's managed statusline (Pro/Max plans only). Run one session, then revisit."; } + } + + function renderLimitsCodex(){ + var codexEl=document.getElementById("u-lim-codex"); var x=LIMITS.codex; var xn=document.getElementById("u-lim-codex-note"); if(x&&x.lanes&&x.lanes.length){ @@ -570,12 +616,27 @@ import { renderUsage } from './usage-orchestrators.mjs'; +"or app-server did not answer."; } + } + + function renderLimitsInsights(){ var ins=Array.isArray(LIMITS.insights)?LIMITS.insights:[]; document.getElementById("u-lim-insights").innerHTML=ins.length ?ins.map(insightCard).join("") :'
no limit findings — nothing is ahead of pace and no arbitrage is open.
'; } + function renderLimits(){ + var claudeEl=document.getElementById("u-lim-claude"); + var codexEl=document.getElementById("u-lim-codex"); + if(!claudeEl||!codexEl)return; + if(!LIMITS){claudeEl.innerHTML='
loading…
'; codexEl.innerHTML='
loading…
'; return;} + if(LIMITS.error){claudeEl.innerHTML='
'+esc(LIMITS.error)+"
"; codexEl.innerHTML=""; return;} + + renderLimitsClaude(); + renderLimitsCodex(); + renderLimitsInsights(); + } + export function renderFindings(d){ var ins=Array.isArray(d.insights)?d.insights:[]; var badge=document.getElementById("u-findings-n"); diff --git a/src/lib/execution/opencode.mjs b/src/lib/execution/opencode.mjs index 14403ab..1c71d40 100644 --- a/src/lib/execution/opencode.mjs +++ b/src/lib/execution/opencode.mjs @@ -365,28 +365,42 @@ function errorCategory(error) { return 'worker_error'; } +/** Map a terminal observation to {status, exitCategory, failure} — the + * branching heart of terminalResult, split out so the surrounding function's + * own complexity is just the field-mapping ternaries/optional-chains. */ +function classifyObservation(observation, assistant) { + if (observation.type === 'permission') { + return { status: 'blocked', exitCategory: 'permission_required', failure: { permission: observation.permission?.id ?? null } }; + } + if (observation.type === 'timeout') { + return { status: 'timed_out', exitCategory: 'timeout', failure: { reason: observation.reason ?? 'timeout' } }; + } + if (observation.type === 'cancelled') { + return { status: 'cancelled', exitCategory: 'cancelled', failure: { reason: 'cancelled' } }; + } + if (observation.type === 'orphaned') { + return { status: 'failed', exitCategory: 'orphaned', failure: { reason: 'owned OpenCode server did not terminate' } }; + } + if (observation.type === 'error' || assistant?.error) { + const error = observation.error ?? assistant?.error ?? null; + return { status: 'failed', exitCategory: errorCategory(error), failure: error ?? { reason: 'OpenCode session failed' } }; + } + if (observation.type !== 'idle') { + return { status: 'failed', exitCategory: 'protocol_error', failure: { reason: 'unknown terminal event' } }; + } + return { status: 'succeeded', exitCategory: 'success', failure: null }; +} + +const terminalResultUsage = (assistant) => ( + assistant?.tokens ? { tokens: assistant.tokens, cost: assistant.cost ?? null } : null +); + function terminalResult(state, observation, clock) { const endedAt = clock(); const startedAt = state.startedAt; const durationMs = Math.max(0, Date.parse(endedAt) - Date.parse(startedAt)); const assistant = observation.assistant?.info ?? null; - let status = 'succeeded'; - let exitCategory = 'success'; - let failure = null; - if (observation.type === 'permission') { - status = 'blocked'; exitCategory = 'permission_required'; failure = { permission: observation.permission?.id ?? null }; - } else if (observation.type === 'timeout') { - status = 'timed_out'; exitCategory = 'timeout'; failure = { reason: observation.reason ?? 'timeout' }; - } else if (observation.type === 'cancelled') { - status = 'cancelled'; exitCategory = 'cancelled'; failure = { reason: 'cancelled' }; - } else if (observation.type === 'orphaned') { - status = 'failed'; exitCategory = 'orphaned'; failure = { reason: 'owned OpenCode server did not terminate' }; - } else if (observation.type === 'error' || assistant?.error) { - const error = observation.error ?? assistant?.error ?? null; - status = 'failed'; exitCategory = errorCategory(error); failure = error ?? { reason: 'OpenCode session failed' }; - } else if (observation.type !== 'idle') { - status = 'failed'; exitCategory = 'protocol_error'; failure = { reason: 'unknown terminal event' }; - } + const { status, exitCategory, failure } = classifyObservation(observation, assistant); return validateWorkerResult({ workerId: state.worker.id, activity: state.worker.activity, role: state.worker.role, host: 'opencode', status, exitCategory, startedAt, endedAt, durationMs, @@ -396,7 +410,7 @@ function terminalResult(state, observation, clock) { observedModel: assistant?.modelID ?? null, sessionId: state.sessionId ?? null, transcriptRefs: state.sessionId ? [`opencode://session/${state.sessionId}`] : [], - failure, usage: assistant?.tokens ? { tokens: assistant.tokens, cost: assistant.cost ?? null } : null, + failure, usage: terminalResultUsage(assistant), }); } @@ -415,6 +429,81 @@ export function createOpenCodeExecutionAdapter({ if (!Number.isInteger(terminationGraceMs) || terminationGraceMs < 1) throw new TypeError('terminationGraceMs must be a positive integer'); if (!Number.isInteger(forceGraceMs) || forceGraceMs < 1) throw new TypeError('forceGraceMs must be a positive integer'); if (!Number.isInteger(teardownTimeoutMs) || teardownTimeoutMs < 1) throw new TypeError('teardownTimeoutMs must be a positive integer'); + + /** Spawn the OpenCode server child and acquire its OS-assigned port + * (stdout-reported, falling back to reservePort for injected children + * with no readable stdout — tests). On failure, stops the child it just + * spawned before rethrowing. */ + async function spawnServerChild(state, { signal }) { + // Port assignment WITHOUT the probe-bind-release race (S2): the child + // binds :0 itself and reports the OS-assigned port on stdout, so a + // squatter can never win a freed port against us (a rogue server would + // not know the ephemeral password anyway — but prompts/credentials must + // never reach a server we did not spawn). reservePort remains the + // fallback for injected children with no readable stdout (tests). + const invocation = resolveFn('opencode', ['serve', '--hostname', LOOPBACK, '--port', '0']); + if (typeof invocation?.command !== 'string' || !Array.isArray(invocation.args)) { + throw new TypeError('OpenCode command resolver returned an invalid invocation'); + } + if (invocation.resolved === false) { + throw new Error('OpenCode command has no safe Windows invocation'); + } + const child = spawnFn(invocation.command, invocation.args, { + cwd: state.cwd, + env: { ...process.env, OPENCODE_SERVER_USERNAME: USERNAME, OPENCODE_SERVER_PASSWORD: state.password }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + // Progressive registration: the runner owns this state and can cancel + // every resource acquired before any later launch await. + state.child = child; + try { + const port = child?.stdout && typeof child.stdout.on === 'function' + ? await boundPortFromStdout(child, { signal }) + : await reservePort({ signal }); + return { child, port }; + } catch (error) { + if (!signal?.aborted) { + const stopped = await stopChild(child, { terminationGraceMs, forceGraceMs, signalFn }); + if (stopped.stopped) state.child = null; + } + throw error; + } + } + + /** Health-check, open a session, start listening for its terminal event, + * and post the prompt. Mutates `state` (sessionId/eventAbort/terminal) as + * each step completes so a mid-flight failure leaves accurate teardown + * state for the caller's own catch block. */ + async function startSession(state, { endpoint, password, timeoutMs, signal, child }) { + await waitForHealth(fetchFn, endpoint, password, { wait, child, signal }); + signal?.throwIfAborted?.(); + const session = await requestJson(fetchFn, endpoint, password, '/session', { + method: 'POST', body: { title: `agentic-kit ${state.worker.id}` }, signal, + }); + if (typeof session?.id !== 'string' || !session.id) throw new Error('OpenCode created a session without an id'); + state.sessionId = session.id; + signal?.throwIfAborted?.(); + const headers = basicHeaders(password); + const eventAbort = new AbortController(); + state.eventAbort = eventAbort; + const eventSignal = signal ? AbortSignal.any([signal, eventAbort.signal]) : eventAbort.signal; + const eventResponse = await fetchFn(`${endpoint}/global/event`, { headers, signal: eventSignal }); + signal?.throwIfAborted?.(); + const terminal = waitForTerminalEvent(eventResponse, session.id, { signal: eventSignal }); + state.terminal = terminal; + // Every path that abandons this promise without observe() consuming it + // (a prompt post that throws, cancel/cleanup teardown) would otherwise + // leave its socket-close rejection unhandled — Node's default turns + // that into a process crash AFTER the run verdict. A no-op second + // consumer keeps teardown honest; observe() still sees the rejection. + terminal.catch(() => {}); + const model = serveModelFor(state.worker.configuredModel); + await requestWithin(fetchFn, endpoint, password, `/session/${encodeURIComponent(session.id)}/prompt_async`, { + body: { agent: 'build', ...(model ? { model } : {}), parts: [{ type: 'text', text: state.prompt }] }, + }, timeoutMs, signal); + signal?.throwIfAborted?.(); + } + const adapter = { id: 'opencode-server', async readiness({ signal, timeoutMs } = /** @type {{signal?:AbortSignal,timeoutMs?:number}} */ ({})) { @@ -432,72 +521,14 @@ export function createOpenCodeExecutionAdapter({ timeoutMs = 120_000, signal, } = /** @type {{timeoutMs?:number,signal?:AbortSignal}} */ ({})) { signal?.throwIfAborted?.(); - const password = secret(); - state.password = password; - // Port assignment WITHOUT the probe-bind-release race (S2): the child - // binds :0 itself and reports the OS-assigned port on stdout, so a - // squatter can never win a freed port against us (a rogue server would - // not know the ephemeral password anyway — but prompts/credentials must - // never reach a server we did not spawn). reservePort remains the - // fallback for injected children with no readable stdout (tests). - const invocation = resolveFn('opencode', ['serve', '--hostname', LOOPBACK, '--port', '0']); - if (typeof invocation?.command !== 'string' || !Array.isArray(invocation.args)) { - throw new TypeError('OpenCode command resolver returned an invalid invocation'); - } - if (invocation.resolved === false) { - throw new Error('OpenCode command has no safe Windows invocation'); - } - const child = spawnFn(invocation.command, invocation.args, { - cwd: state.cwd, - env: { ...process.env, OPENCODE_SERVER_USERNAME: USERNAME, OPENCODE_SERVER_PASSWORD: password }, - stdio: ['ignore', 'pipe', 'pipe'], - }); - // Progressive registration: the runner owns this state and can cancel - // every resource acquired before any later launch await. - state.child = child; - let port; - try { - port = child?.stdout && typeof child.stdout.on === 'function' - ? await boundPortFromStdout(child, { signal }) - : await reservePort({ signal }); - } catch (error) { - if (!signal?.aborted) { - const stopped = await stopChild(child, { terminationGraceMs, forceGraceMs, signalFn }); - if (stopped.stopped) state.child = null; - } - throw error; - } + state.password = secret(); + const { child, port } = await spawnServerChild(state, { signal }); signal?.throwIfAborted?.(); - const endpoint = `http://${LOOPBACK}:${port}`; - state.endpoint = endpoint; + state.endpoint = `http://${LOOPBACK}:${port}`; try { - await waitForHealth(fetchFn, endpoint, password, { wait, child, signal }); - signal?.throwIfAborted?.(); - const session = await requestJson(fetchFn, endpoint, password, '/session', { - method: 'POST', body: { title: `agentic-kit ${state.worker.id}` }, signal, + await startSession(state, { + endpoint: state.endpoint, password: state.password, timeoutMs, signal, child, }); - if (typeof session?.id !== 'string' || !session.id) throw new Error('OpenCode created a session without an id'); - state.sessionId = session.id; - signal?.throwIfAborted?.(); - const headers = basicHeaders(password); - const eventAbort = new AbortController(); - state.eventAbort = eventAbort; - const eventSignal = signal ? AbortSignal.any([signal, eventAbort.signal]) : eventAbort.signal; - const eventResponse = await fetchFn(`${endpoint}/global/event`, { headers, signal: eventSignal }); - signal?.throwIfAborted?.(); - const terminal = waitForTerminalEvent(eventResponse, session.id, { signal: eventSignal }); - state.terminal = terminal; - // Every path that abandons this promise without observe() consuming it - // (a prompt post that throws, cancel/cleanup teardown) would otherwise - // leave its socket-close rejection unhandled — Node's default turns - // that into a process crash AFTER the run verdict. A no-op second - // consumer keeps teardown honest; observe() still sees the rejection. - terminal.catch(() => {}); - const model = serveModelFor(state.worker.configuredModel); - await requestWithin(fetchFn, endpoint, password, `/session/${encodeURIComponent(session.id)}/prompt_async`, { - body: { agent: 'build', ...(model ? { model } : {}), parts: [{ type: 'text', text: state.prompt }] }, - }, timeoutMs, signal); - signal?.throwIfAborted?.(); return state; } catch (error) { if (!signal?.aborted) { diff --git a/src/lib/footprint/catalog.mjs b/src/lib/footprint/catalog.mjs index 9abbe67..9df1ee9 100644 --- a/src/lib/footprint/catalog.mjs +++ b/src/lib/footprint/catalog.mjs @@ -384,71 +384,54 @@ export function collectConfigSurface({ const itemKey = (kind, name) => `${kind}::${name.trim().toLowerCase()}`; /** - * Read every host catalog surface and fold it into deduplicated CatalogItems. - * - * A count is `partial` when any surface feeding it was unreadable or capped: the - * value is then a measured LOWER BOUND. Calling it complete would overstate the - * evidence; calling it unknown would throw away what was actually observed. - * - * @param {{ claudeRoot?: string, claudeMcpFile?: string, codexRoot?: string, - * codexConfigFile?: string, opencodeRoot?: string, opencodeConfigFile?: string, - * cwd?: string, projects?: string[], cfg?: object, now?: () => number, walk?: Function, - * limits?: object, fsImpl?: typeof fs, - * inspectCodexPlugins?: Function, includePluginSurfaces?: boolean }} [options] - * @returns {object} CatalogInventory + * Additional catalog specs contributed by installed plugins' own cache + * directories, discovered from the plugin manifests themselves (Claude's + * `installed_plugins.json`, codex's inspected plugin cache) rather than + * hardcoded. A codex read failure is that host's problem, not this catalog's — + * it must never take the rest of the specs down with it. */ -export function collectCatalog({ - claudeRoot = claudeDir(), - claudeMcpFile = claudeUserMcpPath(), - codexRoot = codexDir(), - codexConfigFile = codexConfigPath(), - opencodeRoot = opencodeDir(), - opencodeConfigFile = opencodeConfigPath(), - cwd = process.cwd(), - // On-disk project paths from the shared census (ADR-0027). Absent → user - // scope plus the launching repo only, exactly as before. - projects = [], - cfg = {}, - now = Date.now, - walk = walkTree, - limits = {}, - fsImpl = fs, - inspectCodexPlugins: inspectCodexPluginsImpl = inspectCodexPlugins, - includePluginSurfaces = true, -} = {}) { - const asOf = now(); - const io = { walk, limits, fsImpl }; - const roots = { claudeRoot, claudeMcpFile, codexRoot, codexConfigFile, opencodeRoot, opencodeConfigFile, - cwd, projects }; - const specs = surfaceSpecs(roots, io); +function pluginSurfaceSpecs({ claudeRoot, codexConfigFile, inspectCodexPlugins: inspect, io }) { + const specs = []; + const claudePlugins = readClaudePlugins(path.join(claudeRoot, 'plugins', 'installed_plugins.json'), io); + for (const { id, root } of claudePlugins.roots) { + specs.push(...pluginSubSurfaces('claude', id, root, 'claude-plugin', io)); + } + let codexPlugins = { configPresent: false, plugins: [] }; + try { codexPlugins = inspect({ configFile: codexConfigFile }) ?? codexPlugins; } + catch { codexPlugins = { configPresent: false, plugins: [] }; } + for (const plugin of codexPlugins.plugins ?? []) { + if (!plugin?.root || !plugin?.ref) continue; + const id = plugin.ref.includes('@') ? plugin.ref.slice(0, plugin.ref.lastIndexOf('@')) : plugin.ref; + specs.push(...pluginSubSurfaces('codex', id, plugin.root, 'codex-plugin', io)); + } + // Codex's enabled refs ARE that host's plugin inventory. + const enabled = (codexPlugins.plugins ?? []).map((plugin) => plugin?.ref).filter(Boolean); + specs.push({ + id: 'codex-plugins', host: 'codex', kind: 'plugin', path: codexConfigFile, + read: () => (codexPlugins.configPresent === false + ? emptyReading('absent', 'ENOENT') + : { status: 'ok', reason: null, names: enabled, partial: false, truncated: false }), + }); + return specs; +} - // Plugin sub-surfaces are discovered from the plugin manifests themselves, so - // they are appended after the base specs rather than hardcoded. - if (includePluginSurfaces) { - const claudePlugins = readClaudePlugins(path.join(claudeRoot, 'plugins', 'installed_plugins.json'), io); - for (const { id, root } of claudePlugins.roots) { - specs.push(...pluginSubSurfaces('claude', id, root, 'claude-plugin', io)); - } - let codexPlugins = { configPresent: false, plugins: [] }; - // Codex owns config.toml and its plugin cache; an unreadable one is not this - // kit's failure to fix, and it must not take the rest of the catalog with it. - try { codexPlugins = inspectCodexPluginsImpl({ configFile: codexConfigFile }) ?? codexPlugins; } - catch { codexPlugins = { configPresent: false, plugins: [] }; } - for (const plugin of codexPlugins.plugins ?? []) { - if (!plugin?.root || !plugin?.ref) continue; - const id = plugin.ref.includes('@') ? plugin.ref.slice(0, plugin.ref.lastIndexOf('@')) : plugin.ref; - specs.push(...pluginSubSurfaces('codex', id, plugin.root, 'codex-plugin', io)); - } - // Codex's enabled refs ARE that host's plugin inventory. - const enabled = (codexPlugins.plugins ?? []).map((plugin) => plugin?.ref).filter(Boolean); - specs.push({ - id: 'codex-plugins', host: 'codex', kind: 'plugin', path: codexConfigFile, - read: () => (codexPlugins.configPresent === false - ? emptyReading('absent', 'ENOENT') - : { status: 'ok', reason: null, names: enabled, partial: false, truncated: false }), - }); +/** Add one spec's names to the deduplicated CatalogItem map. A plugin-sourced + * name carries its plugin's namespace prefix, exactly as the surface declared. */ +function mergeCatalogItem(items, spec, raw) { + const name = spec.prefix ? `${spec.prefix}:${raw}` : raw; + const key = itemKey(spec.kind, name); + let item = items.get(key); + if (!item) { + item = { key, kind: spec.kind, name, hosts: [], presence: [] }; + items.set(key, item); } + if (!item.hosts.includes(spec.host)) item.hosts.push(spec.host); + item.presence.push({ host: spec.host, surface: spec.id, path: spec.path }); +} +/** Read every spec once, folding hits into deduplicated CatalogItems and + * per-surface status rows in the same pass. */ +function readCatalogSurfaces(specs) { const items = new Map(); const surfaces = []; for (const spec of specs) { @@ -466,19 +449,14 @@ export function collectCatalog({ count: reading.status === 'degraded' ? null : reading.names.length, }); if (reading.status === 'degraded') continue; - for (const raw of reading.names) { - const name = spec.prefix ? `${spec.prefix}:${raw}` : raw; - const key = itemKey(spec.kind, name); - let item = items.get(key); - if (!item) { - item = { key, kind: spec.kind, name, hosts: [], presence: [] }; - items.set(key, item); - } - if (!item.hosts.includes(spec.host)) item.hosts.push(spec.host); - item.presence.push({ host: spec.host, surface: spec.id, path: spec.path }); - } + for (const raw of reading.names) mergeCatalogItem(items, spec, raw); } + return { items, surfaces }; +} +/** Which kinds — overall, and per host — a degraded or capped surface touched. + * Feeds the `partial` flag on each count below, never a silent omission. */ +function trackIncompleteness(surfaces) { const incomplete = new Set(); const incompleteByHost = new Set(); for (const surface of surfaces) { @@ -486,17 +464,31 @@ export function collectCatalog({ incomplete.add(surface.kind); incompleteByHost.add(`${surface.host}::${surface.kind}`); } + return { incomplete, incompleteByHost }; +} - const list = [...items.values()].sort((a, b) => (a.kind === b.kind +/** Deduplicated items, ranked by kind (in the documented CATALOG_KINDS order) + * then by name. */ +function sortCatalogItems(items) { + return [...items.values()].sort((a, b) => (a.kind === b.kind ? a.name.localeCompare(b.name) : CATALOG_KINDS.indexOf(a.kind) - CATALOG_KINDS.indexOf(b.kind))); +} +/** Total count per kind, `partial` when any surface feeding that kind was + * unreadable or capped. */ +function tallyCatalogCounts(list, asOf, incomplete) { const counts = {}; - const perHost = {}; for (const kind of CATALOG_KINDS) { const value = list.filter((item) => item.kind === kind).length; counts[kind] = measured(value, { asOf, partial: incomplete.has(kind) }); } + return counts; +} + +/** The same tally, sliced per host. */ +function tallyCatalogPerHost(list, asOf, incompleteByHost) { + const perHost = {}; for (const host of CATALOG_HOSTS) { perHost[host] = {}; for (const kind of CATALOG_KINDS) { @@ -504,7 +496,56 @@ export function collectCatalog({ perHost[host][kind] = measured(value, { asOf, partial: incompleteByHost.has(`${host}::${kind}`) }); } } + return perHost; +} + +/** + * Read every host catalog surface and fold it into deduplicated CatalogItems. + * + * A count is `partial` when any surface feeding it was unreadable or capped: the + * value is then a measured LOWER BOUND. Calling it complete would overstate the + * evidence; calling it unknown would throw away what was actually observed. + * + * @param {{ claudeRoot?: string, claudeMcpFile?: string, codexRoot?: string, + * codexConfigFile?: string, opencodeRoot?: string, opencodeConfigFile?: string, + * cwd?: string, projects?: string[], cfg?: object, now?: () => number, walk?: Function, + * limits?: object, fsImpl?: typeof fs, + * inspectCodexPlugins?: Function, includePluginSurfaces?: boolean }} [options] + * @returns {object} CatalogInventory + */ +export function collectCatalog({ + claudeRoot = claudeDir(), + claudeMcpFile = claudeUserMcpPath(), + codexRoot = codexDir(), + codexConfigFile = codexConfigPath(), + opencodeRoot = opencodeDir(), + opencodeConfigFile = opencodeConfigPath(), + cwd = process.cwd(), + // On-disk project paths from the shared census (ADR-0027). Absent → user + // scope plus the launching repo only, exactly as before. + projects = [], + cfg = {}, + now = Date.now, + walk = walkTree, + limits = {}, + fsImpl = fs, + inspectCodexPlugins: inspectCodexPluginsImpl = inspectCodexPlugins, + includePluginSurfaces = true, +} = {}) { + const asOf = now(); + const io = { walk, limits, fsImpl }; + const roots = { claudeRoot, claudeMcpFile, codexRoot, codexConfigFile, opencodeRoot, opencodeConfigFile, + cwd, projects }; + const specs = surfaceSpecs(roots, io); + if (includePluginSurfaces) { + specs.push(...pluginSurfaceSpecs({ claudeRoot, codexConfigFile, inspectCodexPlugins: inspectCodexPluginsImpl, io })); + } + const { items, surfaces } = readCatalogSurfaces(specs); + const { incomplete, incompleteByHost } = trackIncompleteness(surfaces); + const list = sortCatalogItems(items); + const counts = tallyCatalogCounts(list, asOf, incomplete); + const perHost = tallyCatalogPerHost(list, asOf, incompleteByHost); const degraded = surfaces.filter((surface) => surface.status === 'degraded').map((surface) => surface.id); const truncated = surfaces.filter((surface) => surface.truncated).map((surface) => surface.id); diff --git a/src/lib/footprint/projects.mjs b/src/lib/footprint/projects.mjs index dfe4976..30748bf 100644 --- a/src/lib/footprint/projects.mjs +++ b/src/lib/footprint/projects.mjs @@ -557,56 +557,28 @@ function aggregateUnrecognized(rows) { } /** - * ProjectFootprint rows for every project that still exists on disk, plus the - * ever-seen / on-disk / git-repo counts the Summary KPI states together. + * Resolve the project catalog `collectProjects` will measure, plus the + * ever-seen / on-disk / git-repo counts that ride alongside it. * - * The TABLE is the on-disk subset by necessity — a deleted project has no bytes - * and no lines to count — while `everSeen` keeps the deleted ones, because the - * count of projects this machine has touched is a different question from the - * count it can still measure. Rendering either number alone without saying which - * one it is would misreport both, which is why `method` ships with them. - * - * @param {{ discover?: Function, sources?: object|null, walk?: Function, limits?: object, - * detect?: Function, projects?: Array<{path: string, label: string}>|object|null, - * loc?: boolean, limit?: number|null, onProgress?: Function, - * now?: () => number, fsImpl?: typeof fs }} [options] - * `sources` is an already-resolved `discoverProjectSources()` payload, so a - * caller that shares discovery with another collector pays for the transcript - * sweep once. `projects` takes EITHER shape: an explicit catalog ARRAY, whose - * rows are measured exactly as given because the caller — not discovery — chose - * them and the on-disk filter therefore does not apply; or a discovery PAYLOAD, - * which is read exactly as `sources` is, on-disk filter and KPI counts included. - * @returns {object} the ProjectFootprint section of a FootprintSnapshot + * Discovery is a candidate-path source; if it cannot run, this reports an empty + * catalog with a reason rather than taking the rest of the snapshot down with + * it. `projects` takes EITHER shape: an explicit catalog ARRAY, whose rows are + * measured exactly as given because the caller — not discovery — chose them and + * the on-disk filter therefore does not apply; or a discovery PAYLOAD, which is + * read exactly as `sources` is, on-disk filter and KPI counts included. */ -export function collectProjects({ - discover = discoverProjectSources, - sources = null, - walk = walkTree, - limits = {}, - detect = detectStack, - projects = null, - loc = true, - limit = null, - onProgress = null, - now = Date.now, - fsImpl = fs, -} = {}) { - const asOf = now(); - // Discovery is a candidate-path source; if it cannot run, this section reports - // nothing rather than taking the rest of the snapshot down with it. - let discoveryReason = null; - let catalog; - let counts; +function resolveProjectCatalog({ projects, sources, discover, fsImpl }) { if (Array.isArray(projects)) { - catalog = projects; - counts = summarizeCatalog(projects, fsImpl); - } else { - try { - const payload = (isSourcesPayload(projects) ? projects : sources) ?? discover({ fsImpl }); - // Only projects that still exist can be walked, so only they become rows — - // the vanished ones survive in `everSeen`, not as unmeasurable table rows. - catalog = (payload?.projects ?? []).filter((project) => project?.exists); - counts = { + return { catalog: projects, counts: summarizeCatalog(projects, fsImpl), discoveryReason: null }; + } + try { + const payload = (isSourcesPayload(projects) ? projects : sources) ?? discover({ fsImpl }); + // Only projects that still exist can be walked, so only they become rows — + // the vanished ones survive in `everSeen`, not as unmeasurable table rows. + const catalog = (payload?.projects ?? []).filter((project) => project?.exists); + return { + catalog, + counts: { everSeen: payload?.everSeen ?? 0, onDisk: payload?.onDisk ?? 0, gitRepos: payload?.gitRepos ?? 0, @@ -614,15 +586,17 @@ export function collectProjects({ complete: payload?.complete !== false, method: payload?.method ?? null, sources: payload?.sources ?? null, - }; - } catch (error) { - catalog = []; - discoveryReason = error?.code ?? 'discovery failed'; - } + }, + discoveryReason: null, + }; + } catch (error) { + return { catalog: [], counts: null, discoveryReason: error?.code ?? 'discovery failed' }; } - const rows = Array.isArray(catalog) ? catalog : []; - const selected = typeof limit === 'number' && limit >= 0 ? rows.slice(0, limit) : rows; +} +/** Measure every selected project, reporting progress the same way the scan + * always has: one `project` callback per row, one `done` callback at the end. */ +function measureSelectedProjects(selected, { walk, limits, detect, loc, asOf, fsImpl, onProgress }) { const out = []; for (const project of selected) { if (!project?.path) continue; @@ -630,7 +604,11 @@ export function collectProjects({ out.push(measureProject(project, { walk, limits, detect, loc, asOf, fsImpl })); } notify(onProgress, { scanned: out.length, total: selected.length, phase: 'done', path: null }); + return out; +} +/** Assemble the ProjectFootprint section from a completed measurement pass. */ +function buildProjectsSection({ asOf, out, rows, selected, counts, discoveryReason, loc }) { // A count whose sweep hit an unreadable transcript or an unrecoverable project // directory is a FLOOR, not a total — `partial` is what makes a surface render // it as "≥ N" instead of quietly overstating certainty. @@ -664,3 +642,46 @@ export function collectProjects({ && out.every((row) => row.complete), }; } + +/** + * ProjectFootprint rows for every project that still exists on disk, plus the + * ever-seen / on-disk / git-repo counts the Summary KPI states together. + * + * The TABLE is the on-disk subset by necessity — a deleted project has no bytes + * and no lines to count — while `everSeen` keeps the deleted ones, because the + * count of projects this machine has touched is a different question from the + * count it can still measure. Rendering either number alone without saying which + * one it is would misreport both, which is why `method` ships with them. + * + * @param {{ discover?: Function, sources?: object|null, walk?: Function, limits?: object, + * detect?: Function, projects?: Array<{path: string, label: string}>|object|null, + * loc?: boolean, limit?: number|null, onProgress?: Function, + * now?: () => number, fsImpl?: typeof fs }} [options] + * `sources` is an already-resolved `discoverProjectSources()` payload, so a + * caller that shares discovery with another collector pays for the transcript + * sweep once. `projects` takes EITHER shape: an explicit catalog ARRAY, whose + * rows are measured exactly as given because the caller — not discovery — chose + * them and the on-disk filter therefore does not apply; or a discovery PAYLOAD, + * which is read exactly as `sources` is, on-disk filter and KPI counts included. + * @returns {object} the ProjectFootprint section of a FootprintSnapshot + */ +export function collectProjects({ + discover = discoverProjectSources, + sources = null, + walk = walkTree, + limits = {}, + detect = detectStack, + projects = null, + loc = true, + limit = null, + onProgress = null, + now = Date.now, + fsImpl = fs, +} = {}) { + const asOf = now(); + const { catalog, counts, discoveryReason } = resolveProjectCatalog({ projects, sources, discover, fsImpl }); + const rows = Array.isArray(catalog) ? catalog : []; + const selected = typeof limit === 'number' && limit >= 0 ? rows.slice(0, limit) : rows; + const out = measureSelectedProjects(selected, { walk, limits, detect, loc, asOf, fsImpl, onProgress }); + return buildProjectsSection({ asOf, out, rows, selected, counts, discoveryReason, loc }); +} diff --git a/src/lib/footprint/stack-detect.mjs b/src/lib/footprint/stack-detect.mjs index 5918ba2..048488b 100644 --- a/src/lib/footprint/stack-detect.mjs +++ b/src/lib/footprint/stack-detect.mjs @@ -397,6 +397,48 @@ function unmeasured(reason, asOf) { }; } +/** Read every queued manifest after the walk, folding each declared dependency + * name into either the registry-matched stack or the unrecognized-dependencies + * tail. Split out of `detectStack` (2026-08 complexity program) purely to give + * this loop its own complexity budget; the matching rules are unchanged. */ +function readQueuedManifests(manifestFiles, fsImpl) { + const manifestRows = []; + const stack = new Map(); + const unrecognizedDeps = new Map(); + for (const { file, kind, bytes } of manifestFiles) { + const reading = readManifest(file, kind, bytes, fsImpl); + manifestRows.push({ + path: reading.path, kind: reading.kind, status: reading.status, reason: reading.reason, + }); + for (const name of reading.names) { + const entry = dependencyEntry(kind, name); + if (entry) { + if (!stack.has(entry.id)) stack.set(entry.id, { entry, via: kind }); + continue; + } + const key = `${kind} ${name.toLowerCase()}`; + if (!unrecognizedDeps.has(key)) unrecognizedDeps.set(key, { name, manifest: kind }); + } + } + return { manifestRows, stack, unrecognizedDeps }; +} + +/** Signature-based stack matches (a shallow file, directory, or filename prefix + * seen during the walk), added on top of whatever the manifests already + * matched — a manifest match always wins, so a signature never overrides a + * dependency-derived one. Mutates `stack` in place. */ +function matchSignatures(stack, { seenFiles, seenPaths, seenDirs }) { + for (const entry of signatureEntries()) { + const { files: names, dirs, filePrefixes } = entry.match; + const hit = names.find((name) => seenFiles.has(name) || seenPaths.has(name)) + ?? dirs.find((dir) => seenDirs.has(dir)) + ?? (filePrefixes.length + ? [...seenFiles].find((name) => filePrefixes.some((prefix) => name.startsWith(prefix))) + : undefined); + if (hit !== undefined && !stack.has(entry.id)) stack.set(entry.id, { entry, via: hit }); + } +} + /** * Detect the stack of one project directory. * @@ -501,34 +543,8 @@ export function detectStack(root, { // Manifests are read after the walk: the traversal stays a pure metadata pass, // and a slow read cannot hold the walker's entry budget open. - const manifestRows = []; - const stack = new Map(); - const unrecognizedDeps = new Map(); - for (const { file, kind, bytes } of manifestFiles) { - const reading = readManifest(file, kind, bytes, fsImpl); - manifestRows.push({ - path: reading.path, kind: reading.kind, status: reading.status, reason: reading.reason, - }); - for (const name of reading.names) { - const entry = dependencyEntry(kind, name); - if (entry) { - if (!stack.has(entry.id)) stack.set(entry.id, { entry, via: kind }); - continue; - } - const key = `${kind} ${name.toLowerCase()}`; - if (!unrecognizedDeps.has(key)) unrecognizedDeps.set(key, { name, manifest: kind }); - } - } - - for (const entry of signatureEntries()) { - const { files: names, dirs, filePrefixes } = entry.match; - const hit = names.find((name) => seenFiles.has(name) || seenPaths.has(name)) - ?? dirs.find((dir) => seenDirs.has(dir)) - ?? (filePrefixes.length - ? [...seenFiles].find((name) => filePrefixes.some((prefix) => name.startsWith(prefix))) - : undefined); - if (hit !== undefined && !stack.has(entry.id)) stack.set(entry.id, { entry, via: hit }); - } + const { manifestRows, stack, unrecognizedDeps } = readQueuedManifests(manifestFiles, fsImpl); + matchSignatures(stack, { seenFiles, seenPaths, seenDirs }); const languages = [...lines.values()] .map(({ entry, lines: count, files: fileCount }) => ({ diff --git a/src/lib/footprint/storage-reclaim-detectors.mjs b/src/lib/footprint/storage-reclaim-detectors.mjs new file mode 100644 index 0000000..b46170f --- /dev/null +++ b/src/lib/footprint/storage-reclaim-detectors.mjs @@ -0,0 +1,677 @@ +// Reclaimable-candidate DETECTORS — one function per third-party accumulation +// pattern: superseded dated snapshot copies, whole regenerable cache roots, +// superseded browser installer revisions, installed runtime-manager versions, +// transcripts for projects that no longer exist, and orphaned/idle git +// worktrees. Split out of storage.mjs (2026-08 complexity program, ADR-0037) +// purely by natural seam — each detector already took a shared `ctx` and +// returned `ReclaimableCandidate[]`, so nothing here changes shape or +// behavior. `collectReclaimables` in storage-reclaim.mjs is the orchestrator +// that calls every export below and owns the safety/advisory-only contract +// documented in storage.mjs's header; read that file first. +// +// Third-party cache conventions are spelled out here rather than in paths.mjs +// for the reason consumers.mjs states: that module owns the kit's own path +// contract, and fifty foreign tools' cache layouts would make it harder to +// audit. Platform variants are listed side by side instead of switched on +// process.platform, so the wrong-platform root simply reads absent and a machine +// carrying both (a tool that moved its cache) reports both. +import path from 'node:path'; +import { home, isWindows } from '../paths.mjs'; +import { decodeClaudeProjectDir } from './project-sources.mjs'; +import { + rootMeasurements, measured, unknown, statNode, sumMeasurements, hasValue, +} from './walk.mjs'; +import { candidate } from './storage-reclaim.mjs'; + +const xdgCache = (env) => env.XDG_CACHE_HOME || path.join(home, '.cache'); +const xdgData = (env) => env.XDG_DATA_HOME || path.join(home, '.local', 'share'); +const macCache = () => path.join(home, 'Library', 'Caches'); +const winLocalAppData = (env) => env.LOCALAPPDATA || path.join(home, 'AppData', 'Local'); + +// ── shared detector plumbing ────────────────────────────────────────────────── + +/** One node's figures: the already-measured answer when the consumers view has + * one for this exact path, otherwise one bounded walk. */ +function measureNode(target, ctx) { + const adopted = ctx.adopt?.(target); + if (adopted) return adopted; + const result = ctx.walk(target, { ...ctx.limits, fsImpl: ctx.fsImpl }); + return { + ...rootMeasurements(result, { asOf: ctx.asOf }), + newestMtimeMs: result.newestMtimeMs ?? null, + }; +} + +/** Immediate entries of a directory. ENOENT is an ABSENCE — that tool is not + * installed here, which is not a failed measurement — and every other errno is + * a degradation the caller must report rather than swallow. Symlinked entries + * are classified but never followed or measured. */ +function listMembers(dir, fsImpl) { + try { + return { status: 'ok', reason: null, entries: fsImpl.readdirSync(dir, { withFileTypes: true }) }; + } catch (err) { + const code = err?.code || 'io'; + return { status: code === 'ENOENT' ? 'absent' : 'degraded', reason: code, entries: [] }; + } +} + +/** A root that exists but could not be listed. Reported rather than dropped: + * "there is nothing to reclaim here" and "we could not look" are different + * answers, and only one of them is a measurement (invariant 2). */ +function unreadableCandidate({ id, kind, label, target, reason, safety, cleanupHint = null }) { + return candidate({ + id: `${id}:unreadable`, + kind, + label: `${label} (unreadable)`, + path: target, + bytes: unknown(reason), + files: unknown(reason), + safety, + rationale: `${target} could not be listed (${reason}), so whether anything here is ` + + 'reclaimable is unknown rather than none.', + cleanupHint, + }); +} + +/** Walk a family's members under a walk budget. A cap makes the sum a floor and + * says so through `partial`, which is what "≥ N" renders from; a budget already + * spent before this family was reached yields unknown, because zero members + * measured is not a measurement of zero bytes. */ +function measureMembers(members, ctx, limit = ctx.opts.maxFamilyWalks) { + const walked = members.slice(0, Math.max(0, limit)) + .map((member) => ({ ...member, ...measureNode(member.path, ctx) })); + const capped = members.length > walked.length; + if (members.length && !walked.length) { + const reason = 'the walk budget for this root was spent before this node was reached'; + return { walked, capped, bytes: unknown(reason), files: unknown(reason) }; + } + const bytes = sumMeasurements(walked.map((m) => m.bytes), { asOf: ctx.asOf }); + const files = sumMeasurements(walked.map((m) => m.files), { asOf: ctx.asOf }); + return { + walked, + capped, + bytes: capped && hasValue(bytes) ? { ...bytes, partial: true } : bytes, + files: capped && hasValue(files) ? { ...files, partial: true } : files, + }; +} + +/** Is there anything to advise about? A measured zero is a real zero, and a real + * zero is not a candidate — an "0 B reclaimable" row is an unknown wearing a + * number. An unmeasured figure still earns its row, because not knowing is + * itself the finding. */ +const worthListing = (bytes) => !hasValue(bytes) || bytes.value > 0; + +// ── superseded snapshot copies ──────────────────────────────────────────────── + +/** + * Families of dated copies an installer leaves beside the copy in use. The + * RuvNet Brain is the one on this machine and the largest safe win on it: five + * `kb.bak-` directories totalling ~11 GB beside a 1.9 GB active `kb/`. + * + * @param {{ env?: NodeJS.ProcessEnv }} [options] + */ +export function snapshotFamilies({ env = process.env } = {}) { + const brain = path.join(xdgCache(env), 'ruvnet-brain'); + return [{ + id: 'brain-kb-snapshots', + label: 'RuvNet Brain superseded KB copies', + dir: brain, + // `kb.bak` cannot match `kb`, so the active KB can never be enumerated as + // one of its own backups. + prefix: 'kb.bak', + active: path.join(brain, 'kb'), + activeLabel: 'active KB (kb/)', + what: 'The brain installer copies the knowledge base aside before each update and never ' + + 'removes the copy, so one accumulates per update.', + reproducible: 'A knowledge base is rebuilt by re-running the installer ' + + '(npx ruvnet-brain --doctor).', + cleanupHint: 'remove the dated kb.bak-* directories (npx ruvnet-brain --doctor rebuilds)', + }]; +} + +/** The `YYYY-MM-DD` a dated copy names, when it names one. Used only for the + * rationale's range; a member whose name carries no date is still counted. */ +const datePart = (name) => name.match(/(\d{4}-\d{2}-\d{2})/)?.[1] ?? null; + +/** + * Dated, superseded copies beside an active one — the single largest safe win + * measured on this machine. The active copy is measured too and reported in + * `keeps`, never inside the candidate figure: the row's whole credibility is + * that it can say what it is NOT proposing to touch. + */ +export function supersededSnapshotReclaimables(ctx, families) { + const rows = []; + for (const family of families ?? []) { + const listing = listMembers(family.dir, ctx.fsImpl); + if (listing.status === 'absent') continue; + if (listing.status === 'degraded') { + rows.push(unreadableCandidate({ + id: family.id, kind: 'superseded-snapshots', label: family.label, + target: family.dir, reason: listing.reason, safety: 'regenerable', + cleanupHint: family.cleanupHint, + })); + continue; + } + const members = listing.entries + .filter((entry) => entry.isDirectory() && !entry.isSymbolicLink() + && entry.name.startsWith(family.prefix)) + .map((entry) => ({ name: entry.name, path: path.join(family.dir, entry.name) })) + .sort((a, b) => a.name.localeCompare(b.name)); + if (!members.length) continue; + + const { walked, capped, bytes, files } = measureMembers(members, ctx); + if (!worthListing(bytes)) continue; + const dates = members.map((m) => datePart(m.name)).filter(Boolean); + const span = dates.length >= 2 ? ` (${dates[0]} through ${dates[dates.length - 1]})` + : (dates.length === 1 ? ` (${dates[0]})` : ''); + const active = measureNode(family.active, ctx); + rows.push(candidate({ + id: family.id, + kind: 'superseded-snapshots', + label: `${members.length} superseded copies of ${path.basename(family.active)}`, + path: family.dir, + samplePaths: walked.slice(0, ctx.opts.samplePaths).map((m) => m.path), + matchedCount: members.length, + bytes, + files, + safety: 'regenerable', + keeps: [{ + path: family.active, + label: family.activeLabel, + bytes: active.presence === 'absent' + ? unknown('the active copy is not on disk') + : active.bytes, + }], + rationale: `${members.length} dated copies${span}${capped ? ', of which only ' + + `${walked.length} were measured` : ''}. ${family.what} Nothing reads them: ` + + `the ${family.activeLabel} is measured separately, is excluded from this figure, and ` + + `is not a candidate. ${family.reproducible}`, + cleanupHint: family.cleanupHint, + })); + } + return rows; +} + +// ── regenerable caches ──────────────────────────────────────────────────────── + +/** + * Whole cache roots whose owner refetches them on demand. These are the biggest + * genuinely-safe rows on a developer machine — the npm content-addressable cache + * alone measures ~22 GB here — and they are invisible in the per-host storage + * tree because they belong to no host. + * + * @param {{ env?: NodeJS.ProcessEnv }} [options] + */ +export function regenerableCacheRoots({ env = process.env } = {}) { + const cache = xdgCache(env); + const roots = [ + { + id: 'npm-cacache', + kind: 'regenerable-cache', + label: 'npm content-addressable cache', + path: path.join(home, '.npm', '_cacache'), + what: 'Every package tarball and registry response npm has downloaded, keyed by content ' + + 'hash. It grows monotonically: npm adds to it and never prunes it.', + cleanupHint: 'npm cache clean --force', + }, + { + id: 'homebrew-downloads', + kind: 'regenerable-cache', + label: 'Homebrew download cache', + path: path.join(macCache(), 'Homebrew'), + what: 'Bottles, source tarballs and the formula API responses brew downloaded, kept after ' + + 'the install they were for.', + cleanupHint: 'brew cleanup', + }, + { + id: 'homebrew-downloads-xdg', + kind: 'regenerable-cache', + label: 'Homebrew download cache', + path: path.join(cache, 'Homebrew'), + what: 'Bottles, source tarballs and the formula API responses brew downloaded, kept after ' + + 'the install they were for.', + cleanupHint: 'brew cleanup', + }, + ]; + if (isWindows) { + roots.push({ + id: 'npm-cacache-win', + kind: 'regenerable-cache', + label: 'npm content-addressable cache', + path: path.join(winLocalAppData(env), 'npm-cache', '_cacache'), + what: 'Every package tarball and registry response npm has downloaded, keyed by content ' + + 'hash. It grows monotonically: npm adds to it and never prunes it.', + cleanupHint: 'npm cache clean --force', + }); + } + return roots; +} + +/** One row per present cache root. An absent root produces nothing at all — + * a tool that is not installed is not a reclaimable zero. */ +export function regenerableCacheReclaimables(ctx, roots) { + const rows = []; + for (const root of roots ?? []) { + const node = measureNode(root.path, ctx); + if (node.presence === 'absent') continue; + if (!worthListing(node.bytes)) continue; + rows.push(candidate({ + id: root.id, + kind: root.kind, + label: root.label, + path: root.path, + bytes: node.bytes, + files: node.files, + safety: 'regenerable', + rationale: `${root.what} Nothing here is unique: the tool refetches what it needs on the ` + + 'next install, so the cost of clearing it is download time, not data.', + cleanupHint: root.cleanupHint, + })); + } + return rows; +} + +// ── superseded browser downloads ────────────────────────────────────────────── + +/** + * Roots where a browser installer keeps one directory per revision and adds a + * new one on every version bump, never removing the old. `depth` is where the + * revision directories live: Playwright keeps them at the root + * (`chromium-1223`), Puppeteer one level down (`chrome/mac_arm-149.0.7827.22`). + * + * @param {{ env?: NodeJS.ProcessEnv }} [options] + */ +export function browserRevisionRoots({ env = process.env } = {}) { + const cache = xdgCache(env); + const playwright = { + kind: 'superseded-browser-revisions', + label: 'Playwright browser builds', + depth: 1, + installer: 'npx playwright install', + }; + return [ + { ...playwright, id: 'playwright-mac', path: path.join(macCache(), 'ms-playwright') }, + { ...playwright, id: 'playwright-xdg', path: path.join(cache, 'ms-playwright') }, + { ...playwright, id: 'playwright-win', path: path.join(winLocalAppData(env), 'ms-playwright') }, + { + kind: 'superseded-browser-revisions', + id: 'puppeteer', + label: 'Puppeteer browser builds', + path: path.join(cache, 'puppeteer'), + depth: 2, + installer: 'npx puppeteer browsers install', + }, + ]; +} + +/** Split `chromium_headless_shell-1223` into its family and its revision. The + * revision must contain a digit, which is what keeps Playwright's + * `mcp-chrome-profile` — a browser PROFILE, not a revision — out of the + * families entirely. */ +function splitRevision(name) { + const cut = name.lastIndexOf('-'); + if (cut <= 0 || cut === name.length - 1) return null; + const revision = name.slice(cut + 1); + if (!/\d/.test(revision)) return null; + return { family: name.slice(0, cut), revision }; +} + +/** Revision directories under a root, at the root itself (`depth` 1) or one + * level down (`depth` 2). Never recursive: a browser build's own contents are + * not revisions. */ +function revisionMembers(root, ctx) { + const listing = listMembers(root.path, ctx.fsImpl); + if (listing.status !== 'ok') return { status: listing.status, reason: listing.reason, members: [] }; + const holders = root.depth === 2 + ? listing.entries.filter((entry) => entry.isDirectory() && !entry.isSymbolicLink()) + .map((entry) => ({ dir: path.join(root.path, entry.name), scope: entry.name })) + : [{ dir: root.path, scope: '' }]; + const members = []; + for (const holder of holders) { + const inner = root.depth === 2 ? listMembers(holder.dir, ctx.fsImpl) : listing; + if (inner.status !== 'ok') continue; + for (const entry of inner.entries) { + if (!entry.isDirectory() || entry.isSymbolicLink()) continue; + const split = splitRevision(entry.name); + if (!split) continue; + const target = path.join(holder.dir, entry.name); + members.push({ + name: entry.name, + path: target, + family: holder.scope ? `${holder.scope}/${split.family}` : split.family, + revision: split.revision, + mtimeMs: statNode(target, { fsImpl: ctx.fsImpl }).mtimeMs ?? 0, + }); + } + } + return { status: 'ok', reason: null, members }; +} + +/** + * Older revisions in each browser family — everything except the most recently + * installed one. Ordering is by directory mtime rather than by parsing the + * revision, because the revisions are not comparable across installers + * (Playwright's `1223` is a counter, its `mcp-chrome-5b42311` is a hex build id, + * Puppeteer's is a four-part Chrome version). + * + * REVIEW tier, not regenerable, even though the installer would refetch them: a + * project's pinned playwright/puppeteer version resolves to a specific revision, + * and this module cannot see which package pins what. The row's job is to say + * "these accumulated, look at them", not "delete N GB". + */ +export function browserRevisionReclaimables(ctx, roots) { + const rows = []; + for (const root of roots ?? []) { + const { status, reason, members } = revisionMembers(root, ctx); + if (status === 'absent') continue; + if (status === 'degraded') { + rows.push(unreadableCandidate({ + id: root.id, kind: root.kind, label: root.label, + target: root.path, reason, safety: 'review', + })); + continue; + } + const families = new Map(); + for (const member of members) { + const list = families.get(member.family) ?? []; + list.push(member); + families.set(member.family, list); + } + const superseded = []; + const keeps = []; + for (const [family, list] of families) { + if (list.length < 2) continue; + const ordered = [...list].sort((a, b) => b.mtimeMs - a.mtimeMs || b.name.localeCompare(a.name)); + keeps.push({ family, ...ordered[0] }); + superseded.push(...ordered.slice(1)); + } + if (!superseded.length) continue; + const { walked, capped, bytes, files } = measureMembers(superseded, ctx); + if (!worthListing(bytes)) continue; + rows.push(candidate({ + id: `${root.id}:superseded`, + kind: root.kind, + label: `${superseded.length} superseded ${root.label.toLowerCase()}`, + path: root.path, + samplePaths: walked.slice(0, ctx.opts.samplePaths).map((m) => m.path), + matchedCount: superseded.length, + bytes, + files, + safety: 'review', + keeps: keeps.map((keep) => ({ + path: keep.path, + label: `newest ${keep.family} revision (${keep.revision})`, + bytes: unknown('the retained revision is not part of this figure'), + })), + rationale: `${keeps.length} browser famil(ies) here carry more than one revision; this ` + + `figure covers the ${superseded.length} older one(s)${capped + ? `, of which ${walked.length} were measured` : ''}, and excludes the newest of each. ` + + 'Review rather than sweep: an installed package can pin an older revision, and this ' + + 'row cannot see which package pins what. ' + + `${root.installer} refetches whatever is missing.`, + cleanupHint: `${root.installer} (confirm no project pins the older revision first)`, + })); + } + return rows; +} + +// ── installed runtime versions ──────────────────────────────────────────────── + +/** + * Version-manager install roots — one directory per installed runtime version, + * plus alias entries pointing into them. + * + * @param {{ env?: NodeJS.ProcessEnv }} [options] + */ +export function runtimeVersionRoots({ env = process.env } = {}) { + return [{ + id: 'mise-installs', + kind: 'installed-runtime-versions', + label: 'mise', + path: path.join(xdgData(env), 'mise', 'installs'), + manager: 'mise', + cleanupHint: 'mise ls, then mise uninstall @ (nothing may pin it)', + }]; +} + +/** + * Installed runtime versions, one row per managed tool that has more than one. + * + * REVIEW tier and `bytesMeaning: 'installed'`, both deliberately. On this + * machine `mise/installs/node` holds eight entries — 22, 22.22, 22.22.3, 26, + * 26.4, 26.4.0, latest, lts-jod — of which only two are real directories and six + * are ALIAS SYMLINKS resolving into them. Removing a version silently breaks + * every alias that points at it, and any `mise.toml` or `.tool-versions` on the + * machine can pin any of them, so there is no subset this module can honestly + * call reclaimable. What it can do is show what is installed and say that out + * loud; recommending the deletion of a live runtime would be worse than saying + * nothing. + * + * The alias links are counted, never followed — the walker's rule, and also the + * only reason the byte figure is not multiplied by every alias. + */ +export function runtimeVersionReclaimables(ctx, roots) { + const rows = []; + for (const root of roots ?? []) { + const listing = listMembers(root.path, ctx.fsImpl); + if (listing.status === 'absent') continue; + if (listing.status === 'degraded') { + rows.push(unreadableCandidate({ + id: root.id, kind: root.kind, label: `${root.manager} installs`, + target: root.path, reason: listing.reason, safety: 'review', + })); + continue; + } + // Every managed tool is examined — listing one is a readdir, and capping the + // LIST would drop tools in alphabetical order, which is an invisible + // truncation of exactly the kind this domain forbids. What is bounded is the + // expensive part: the walks, under one budget shared across the root. + let budget = ctx.opts.maxFamilyWalks; + const tools = listing.entries + .filter((entry) => entry.isDirectory() && !entry.isSymbolicLink() + && !entry.name.startsWith('.')); + for (const tool of tools) { + const dir = path.join(root.path, tool.name); + const inner = listMembers(dir, ctx.fsImpl); + if (inner.status !== 'ok') continue; + const versions = []; + let aliases = 0; + for (const entry of inner.entries) { + if (entry.name.startsWith('.')) continue; + if (entry.isSymbolicLink()) { aliases += 1; continue; } + if (entry.isDirectory()) versions.push({ name: entry.name, path: path.join(dir, entry.name) }); + } + // One installed version is the toolchain working as intended, not sprawl. + if (versions.length < 2) continue; + const { walked, capped, bytes, files } = measureMembers(versions, ctx, budget); + budget -= walked.length; + if (!worthListing(bytes)) continue; + rows.push(candidate({ + id: `${root.id}:${tool.name}`, + kind: root.kind, + label: `${versions.length} ${root.manager} ${tool.name} versions installed`, + path: dir, + samplePaths: walked.slice(0, ctx.opts.samplePaths).map((m) => m.path), + matchedCount: versions.length, + bytes, + files, + safety: 'review', + bytesMeaning: 'installed', + rationale: `${versions.length} installed version(s) of ${tool.name}` + + `${aliases ? ` plus ${aliases} alias link(s) resolving into them` : ''}` + + `${capped ? `, of which ${walked.length} were measured` : ''}. ` + + 'This is what is installed, not what is free: an alias points at a real version, ' + + `and any ${root.manager} config on this machine can pin any of them. Review with ` + + `\`${root.manager} ls ${tool.name}\` before removing anything.`, + cleanupHint: root.cleanupHint, + })); + } + } + return rows; +} + +// ── transcripts for projects that no longer exist ───────────────────────────── + +/** + * Claude transcript directories whose project directory is gone. + * + * A `~/.claude/projects/` directory name is a LOSSY encoding of the project path + * (`/`, `.` and a literal `-` all become `-`), so it cannot be decoded by string + * manipulation; `decodeClaudeProjectDir` walks the candidate segments against the + * real filesystem and returns a path only when the filesystem confirms one. + * A directory that decodes to nothing is therefore a project that is no longer + * on this machine — and that is exactly the evidence available, so the row says + * so rather than claiming certainty. + * + * TWO GUARDS against the failure mode that would matter, flagging live projects: + * · only `-`-leading (POSIX-encoded) names are considered, because Windows + * names encode a drive letter that this decoder does not handle and would + * otherwise report every project on the machine as dead; + * · if NOTHING decoded, the finding is about the decoder or an unreadable home + * directory, not about the projects, so no row is emitted at all. + * + * The value here is hygiene and privacy, not space: measured on this machine + * these are ~0.05 GB across 8 dead projects. The rationale says that plainly — + * selling 50 MB as a disk win would be the same dishonesty as a fabricated zero, + * just in the other direction. + */ +export function orphanedTranscriptReclaimables({ + asOf, opts, transcriptProjects, decodeDir = decodeClaudeProjectDir, fsImpl, +}) { + const byHost = new Map(); + for (const entry of transcriptProjects?.values?.() ?? []) { + if (!entry.dir.startsWith('-')) continue; + const acc = byHost.get(entry.host) + ?? { host: entry.host, root: entry.rootPath, alive: 0, dead: [] }; + if (decodeDir(entry.dir, { fsImpl })) acc.alive += 1; + else acc.dead.push(entry); + byHost.set(entry.host, acc); + } + + const rows = []; + for (const acc of byHost.values()) { + if (!acc.dead.length || !acc.alive) continue; + const bytes = acc.dead.reduce((total, entry) => total + entry.bytes, 0); + const files = acc.dead.reduce((total, entry) => total + entry.files, 0); + const newest = acc.dead.reduce((at, entry) => Math.max(at, entry.newestMtimeMs ?? 0), 0); + rows.push(candidate({ + id: `orphaned-transcripts:${acc.host}`, + kind: 'orphaned-transcripts', + label: `${acc.host} transcripts for ${acc.dead.length} project(s) that no longer exist`, + path: acc.root, + samplePaths: acc.dead.slice(0, opts.samplePaths).map((entry) => entry.path), + matchedCount: acc.dead.length, + bytes: measured(bytes, { asOf }), + files: measured(files, { asOf }), + safety: 'review', + rationale: `${files} transcript(s) belong to ${acc.dead.length} project director(ies) that ` + + `no longer resolve on this machine (${acc.alive} others still do; last activity ` + + `${newest ? `${Math.floor((asOf - newest) / 86_400_000)}d ago` : 'unknown'}). This row ` + + 'is here for hygiene and privacy, not as a space win: it is listed because those ' + + 'projects are gone, whatever the byte figure turns out to be. The transcripts are also ' + + 'the only copy of that history, and an unreadable parent directory looks identical to ' + + 'a deleted project — confirm the project is really gone before acting.', + cleanupHint: null, + })); + } + return rows; +} + +/** Orphaned git worktrees, from each project's `.git/worktrees/` admin + * records. Two honest verdicts, no git invocation: + * · the checkout the record points at no longer exists → the record is dead; + * · the checkout exists but nothing in it has been touched for the idle + * window → a candidate, with its real on-disk size. + * A record whose pointer cannot be read is reported as unverifiable rather + * than assumed dead. See storage.mjs's header for why the pointer read is in + * scope (the one deliberate content-read exception, bounded to 4 KB). */ +export function worktreeReclaimables({ asOf, projects, opts, walk, limits, fsImpl }) { + const rows = []; + let walks = 0; + for (const project of projects) { + const adminRoot = path.join(project, '.git', 'worktrees'); + let entries; + try { + entries = fsImpl.readdirSync(adminRoot, { withFileTypes: true }); + } catch { continue; } // no worktrees here (or unreadable): nothing to claim + for (const entry of entries) { + if (!entry.isDirectory()) continue; + const record = path.join(adminRoot, entry.name); + const pointer = readGitdirPointer(path.join(record, 'gitdir'), fsImpl); + if (!pointer) { + rows.push(candidate({ + id: `orphaned-worktree:${record}`, + kind: 'orphaned-worktree', + label: `worktree record "${entry.name}" (unverifiable)`, + path: record, + bytes: unknown('worktree pointer unreadable'), + files: unknown('worktree pointer unreadable'), + safety: 'review', + rationale: 'The admin record exists but its gitdir pointer could not be read, ' + + 'so whether the checkout still exists is unknown.', + cleanupHint: 'git worktree prune (verify first)', + })); + continue; + } + // gitdir points at `/.git`; the checkout is its parent. + const checkout = path.dirname(pointer); + const head = statNode(checkout, { fsImpl }); + if (head.status === 'unknown') { + rows.push(candidate({ + id: `orphaned-worktree:${record}`, + kind: 'orphaned-worktree', + label: `orphaned worktree record "${entry.name}"`, + path: record, + samplePaths: [checkout], + bytes: measured(0, { asOf }), + files: measured(0, { asOf }), + safety: 'review', + rationale: `The checkout at ${checkout} no longer exists; only the administrative ` + + 'record remains.', + cleanupHint: 'git worktree prune', + })); + continue; + } + if (walks >= opts.maxWorktreeWalks) continue; + walks += 1; + const result = walk(checkout, { ...limits, fsImpl }); + const { bytes, files } = rootMeasurements(result, { asOf }); + const idleMs = result.newestMtimeMs === null ? null : asOf - result.newestMtimeMs; + if (idleMs === null || idleMs < opts.worktreeIdleDays * 86_400_000) continue; + rows.push(candidate({ + id: `idle-worktree:${record}`, + kind: 'orphaned-worktree', + label: `idle worktree "${entry.name}"`, + path: checkout, + samplePaths: [record], + bytes, + files, + safety: 'review', + rationale: `Nothing in the checkout has changed for ${Math.floor(idleMs / 86_400_000)}d. ` + + 'Merge state is not checked here — confirm the branch is landed before removing.', + cleanupHint: 'git worktree remove (verify the branch is merged first)', + })); + } + } + return rows; +} + +/** The single narrow non-metadata read in this module: a `gitdir` file holds + * one absolute path and nothing else. Bounded to 4 KB and validated as a path + * so a corrupt file yields null rather than a bogus candidate. */ +function readGitdirPointer(file, fsImpl) { + let fd; + try { + fd = fsImpl.openSync(file, 'r'); + const buf = Buffer.alloc(4096); + const read = fsImpl.readSync(fd, buf, 0, 4096, 0); + const value = buf.toString('utf8', 0, read).trim(); + return value && path.isAbsolute(value) ? value : null; + } catch { + return null; + } finally { + if (fd !== undefined) { + try { fsImpl.closeSync(fd); } catch { /* already gone */ } + } + } +} diff --git a/src/lib/footprint/storage-reclaim.mjs b/src/lib/footprint/storage-reclaim.mjs new file mode 100644 index 0000000..9588cd0 --- /dev/null +++ b/src/lib/footprint/storage-reclaim.mjs @@ -0,0 +1,251 @@ +// Reclaimable-space ADVISORY orchestration — split out of storage.mjs (2026-08 +// complexity program, ADR-0037) by natural seam: everything below is the +// reclaim half of that module's tree (`collectStorage` calls +// `collectReclaimables` exactly as it always has). See storage.mjs's header for +// the full contract this inherits unchanged: +// +// ADVISORY ONLY (invariant 4): there is no delete, prune, or cleanup verb here, +// and none may be added. `ReclaimableCandidate` rows carry a path, a size, and a +// rationale — a `cleanupHint` names the CLI that already owns the removal, and +// that string is documentation, not a command this module runs. npx.mjs's +// pruneNpxStale is deliberately NOT imported; only its read-only scanNpxStale +// is. +// +// SAFETY IS A FIELD, NOT A TONE OF VOICE — see `RECLAIM_SAFETY_TIERS` / +// `RECLAIM_SAFETY_MEANING` below and storage.mjs's header for the full +// 'regenerable' vs 'review' distinction. +// +// The family-specific detectors (superseded snapshots, regenerable caches, +// browser revisions, runtime versions, orphaned transcripts, worktrees) live in +// storage-reclaim-detectors.mjs; this file is the orchestrator plus the shared +// bits every detector composes with: `candidate()` (the row-shape contract) and +// `adoptedConsumerFigures()` (reusing an already-measured figure from the +// ranked-consumers view instead of walking a tree twice). +import path from 'node:path'; +import { scanNpxStale } from '../npx.mjs'; +import { npxEnvNodes } from './install.mjs'; +import { decodeClaudeProjectDir } from './project-sources.mjs'; +import { measured, unknown, sumMeasurements, hasValue } from './walk.mjs'; +import { + snapshotFamilies, supersededSnapshotReclaimables, + regenerableCacheRoots, regenerableCacheReclaimables, + browserRevisionRoots, browserRevisionReclaimables, + runtimeVersionRoots, runtimeVersionReclaimables, + orphanedTranscriptReclaimables, worktreeReclaimables, +} from './storage-reclaim-detectors.mjs'; + +/** The two safety tiers, in the order a panel should present them. Two and not + * three: a "definitely dead" tier would be a claim this module cannot + * substantiate from directory metadata alone. */ +export const RECLAIM_SAFETY_TIERS = Object.freeze(['regenerable', 'review']); + +/** What each tier promises, carried in the payload so no surface has to invent + * the wording — and so the difference between the two is impossible to render + * as the same thing. */ +export const RECLAIM_SAFETY_MEANING = Object.freeze({ + regenerable: 'The owning tool refetches this on demand. Removing it costs download time, ' + + 'not data.', + review: 'Plausible but not safe to call removable: some of these may be in use. Review them ' + + 'individually — this is not a total to sweep.', +}); + +/** + * Advisory rows only — see this module's header. Nothing here removes anything; + * `cleanupHint` names the CLI that already owns the removal. + * + * @typedef {{ value: number|null, status: string, reason: string|null, + * asOf: number|null, partial: boolean }} Measurement + * @typedef {{ + * id: string, kind: string, label: string, path: string, samplePaths: string[], + * matchedCount: number|null, bytes: Measurement, files: Measurement, + * safety: 'regenerable'|'review', bytesMeaning: 'candidate'|'installed', + * keeps: Array<{ path: string, label: string, bytes: Measurement }>, + * rationale: string, cleanupHint: string|null, advisory: true, + * }} ReclaimableCandidate + */ +export function collectReclaimables({ + asOf, agedTranscripts, transcriptProjects = new Map(), projects, opts, walk, limits, + detectWorktrees, detectCaches = true, detectOrphanedTranscripts = true, + consumers = null, env = process.env, decodeDir = decodeClaudeProjectDir, fsImpl, +}) { + const rows = []; + const days = (ms) => Math.floor((asOf - ms) / 86_400_000); + const ctx = { + asOf, opts, walk, limits, fsImpl, adopt: adoptedConsumerFigures(consumers), + }; + + for (const acc of agedTranscripts.values()) { + rows.push(candidate({ + id: `aged-transcripts:${acc.host}`, + kind: 'aged-transcripts', + label: `${acc.host} transcripts older than ${opts.transcriptAgeDays}d`, + path: acc.root, + samplePaths: acc.samples, + matchedCount: acc.files, + bytes: measured(acc.bytes, { asOf }), + files: measured(acc.files, { asOf }), + // Not regenerable in any sense: a transcript is the only copy of the + // session it records, and Historical usage is denominated in them. + safety: 'review', + rationale: `${acc.files} file(s) untouched for ${opts.transcriptAgeDays}d or more; ` + + `oldest ${days(acc.oldestMtimeMs)}d. Historical usage reads these — removing them ` + + 'removes that history too.', + cleanupHint: null, + })); + } + + rows.push(...npxReclaimables({ asOf, opts, walk, limits, fsImpl })); + if (detectOrphanedTranscripts) { + rows.push(...orphanedTranscriptReclaimables({ + asOf, opts, transcriptProjects, decodeDir, fsImpl, + })); + } + if (detectCaches) { + rows.push(...supersededSnapshotReclaimables(ctx, snapshotFamilies({ env }))); + rows.push(...regenerableCacheReclaimables(ctx, regenerableCacheRoots({ env }))); + rows.push(...browserRevisionReclaimables(ctx, browserRevisionRoots({ env }))); + rows.push(...runtimeVersionReclaimables(ctx, runtimeVersionRoots({ env }))); + } + if (detectWorktrees && Array.isArray(projects)) { + rows.push(...worktreeReclaimables({ asOf, projects, opts, walk, limits, fsImpl })); + } + return rows.sort((a, b) => (b.bytes.value ?? 0) - (a.bytes.value ?? 0)); +} + +// APFS and NTFS are case-insensitive, so an adopted figure keyed by an +// exact-case path would silently miss on macOS and Windows. +const foldCase = process.platform !== 'linux'; +const pathKey = (target) => { + const abs = path.resolve(target); + return foldCase ? abs.toLowerCase() : abs; +}; + +/** Do any two of these rows describe the same path, or one inside another? Two + * such rows cover some of the same bytes — an aged transcript can also sit in a + * project that no longer exists — and adding them would report space twice. */ +function rowsOverlap(rows) { + const keys = rows.map((row) => (row.path ? pathKey(row.path) : null)).filter(Boolean); + for (let i = 0; i < keys.length; i++) { + for (let j = i + 1; j < keys.length; j++) { + if (keys[i] === keys[j]) return true; + const rel = path.relative(keys[i], keys[j]); + const nested = rel && !rel.startsWith('..') && !path.isAbsolute(rel); + const inverse = path.relative(keys[j], keys[i]); + if (nested || (inverse && !inverse.startsWith('..') && !path.isAbsolute(inverse))) return true; + } + } + return false; +} + +/** + * Per-tier totals, and deliberately NO combined figure. Summing a regenerable + * cache with a runtime tree that may be live would produce the one number a + * reader would act on and the one number this module cannot stand behind. + * + * A tier whose own rows overlap reports its total as unknown-with-reason rather + * than as a sum that counts the same bytes twice — the rowCount still stands, + * and each row still carries its own measured figure. + */ +export function summarizeReclaimables(rows, { asOf = null } = {}) { + const list = Array.isArray(rows) ? rows : []; + return { + tiers: RECLAIM_SAFETY_TIERS.map((safety) => { + const tier = list.filter((row) => row.safety === safety); + // Only 'candidate' bytes are summable: an 'installed' figure is context on + // a review row, not space the row claims is available. + const members = tier.filter((row) => row.bytesMeaning === 'candidate'); + const overlapping = rowsOverlap(members); + return { + safety, + meaning: RECLAIM_SAFETY_MEANING[safety], + rowCount: tier.length, + bytes: overlapping + ? unknown('rows in this tier describe overlapping paths, so a sum would count the ' + + 'same bytes twice') + : sumMeasurements(members.map((row) => row.bytes), { asOf }), + summedRows: overlapping ? 0 : members.length, + contextOnlyRows: tier.length - members.length, + }; + }), + combined: null, + combinedNote: 'The tiers are reported separately and never added: only the regenerable ' + + 'total is space a tool would rebuild by itself.', + }; +} + +/** Fill in the fields every row carries, so no detector can ship a candidate + * without a safety tier or a statement of what its bytes mean. Exported so + * storage-reclaim-detectors.mjs's family detectors build to the same shape. */ +export function candidate(row) { + return { + samplePaths: [], + matchedCount: null, + keeps: [], + bytesMeaning: 'candidate', + cleanupHint: null, + ...row, + advisory: true, + }; +} + +/** Stale npx cache envs. Two independent rationales, both read-only: a cached + * copy strictly older than its installed global baseline (npx.mjs's version + * verdict — the bug that kept a machine running a retired ruflo), and an env + * untouched for longer than the idle threshold. */ +export function npxReclaimables({ asOf, opts, walk, limits, fsImpl }) { + const nodes = npxEnvNodes({ walk, limits, asOf, fsImpl }); + if (nodes.presence !== 'present') return []; + let staleByVersion = new Map(); + try { + staleByVersion = new Map(scanNpxStale().map((entry) => [entry.dir, entry.stale])); + } catch { /* an unreadable cache simply yields no version verdict */ } + const idleCutoff = asOf - opts.npxEnvIdleDays * 86_400_000; + const rows = []; + for (const env of nodes.envs) { + const stale = staleByVersion.get(env.path); + const idle = env.newestMtimeMs !== null && env.newestMtimeMs < idleCutoff; + if (!stale && !idle) continue; + const why = []; + if (stale) { + why.push(`cached ${stale.map((s) => `${s.pkg}@${s.cached}`).join(', ')} ` + + `older than installed ${stale.map((s) => s.installed).join(', ')}`); + } + if (idle) why.push(`untouched for ${Math.floor((asOf - env.newestMtimeMs) / 86_400_000)}d`); + rows.push(candidate({ + id: `stale-npx-env:${env.id}`, + kind: 'stale-npx-env', + label: `npx cache env (${env.packages.join(', ') || 'unkeyed'})`, + path: env.path, + bytes: env.bytes, + files: env.files, + safety: 'regenerable', + rationale: `${why.join('; ')}. npx re-fetches on demand, so the cache is reproducible.`, + cleanupHint: 'ak sync prunes version-stale envs (npx.pruneNpxStale)', + })); + } + return rows; +} + +/** + * A lookup from absolute path to a figure the ranked-consumers view already + * measured, or a function that always misses when no such view was supplied. + * Exact paths only: a consumers row for a glob FAMILY carries one total for the + * whole family and cannot answer for an individual member. + * + * @param {{ rows?: any[] }|null} consumers a collectConsumers payload + * @returns {(target: string) => ({ presence: string, bytes: Measurement, + * files: Measurement, newestMtimeMs: number|null })|null} + */ +export function adoptedConsumerFigures(consumers) { + const index = new Map(); + for (const row of consumers?.rows ?? []) { + if (!row?.path || row.residual || row.presence !== 'present' || !hasValue(row.bytes)) continue; + index.set(pathKey(row.path), { + presence: 'present', + bytes: row.bytes, + files: row.files ?? unknown('file count not carried by the adopted figure'), + newestMtimeMs: row.newestMtimeMs ?? null, + }); + } + return (target) => (target ? index.get(pathKey(target)) ?? null : null); +} diff --git a/src/lib/footprint/storage.mjs b/src/lib/footprint/storage.mjs index f875f4f..8120564 100644 --- a/src/lib/footprint/storage.mjs +++ b/src/lib/footprint/storage.mjs @@ -1,25 +1,33 @@ -// Storage breakdown — the category → host → project → session tree, the derived -// views over the same walk (trailing-30d growth, top-N giants), and the advisory -// reclaimable rows (ADR-0025 §4, docs/ddd/machine-footprint.md "Storage -// breakdown"). +// Storage breakdown — the category → host → project → session tree, and the +// derived views over the same walk (trailing-30d growth, top-N giants) (ADR-0025 +// §4, docs/ddd/machine-footprint.md "Storage breakdown"). +// +// The reclaimable-space ADVISORY half of this domain — `collectReclaimables` and +// everything it composes — lives in storage-reclaim.mjs (orchestration + shared +// plumbing) and storage-reclaim-detectors.mjs (the per-accumulation-pattern +// detectors: superseded snapshots, regenerable caches, browser revisions, +// runtime versions, orphaned transcripts, orphaned worktrees), split out here by +// natural seam (2026-08 complexity program, ADR-0037) once this file passed +// 1,000 lines. Both are re-exported below so every existing import of +// `storage.mjs` keeps working unchanged. // // This module is ADVISORY ONLY by invariant 4: there is no delete, prune, or -// cleanup verb here, and none may be added. `ReclaimableCandidate` rows carry a -// path, a size, and a rationale — a `cleanupHint` names the CLI that already -// owns the removal, and that string is documentation, not a command this module -// runs. npx.mjs's pruneNpxStale is deliberately NOT imported; only its -// read-only scanNpxStale is. +// cleanup verb anywhere in this domain, and none may be added. `Reclaimable +// Candidate` rows carry a path, a size, and a rationale — a `cleanupHint` names +// the CLI that already owns the removal, and that string is documentation, not +// a command this module runs. npx.mjs's pruneNpxStale is deliberately NOT +// imported; only its read-only scanNpxStale is. // -// SAFETY IS A FIELD, NOT A TONE OF VOICE. Every candidate carries `safety`: -// 'regenerable' (the owning tool refetches it on demand — the npm content cache, -// the Homebrew download cache, the brain's superseded KB copies) or 'review' -// (plausible but NOT safe to state as removable — mise has eight node entries on -// this machine and some of them are the aliases a live toolchain resolves -// through; a browser revision may still be pinned by an installed package). -// A review row is a pointer at something to look at, never a figure to sweep, -// and `summarizeReclaimables` totals the two tiers SEPARATELY on purpose: a -// combined "you could free N" that mixes them would be the honest-measurement -// contract broken at the last mile. +// SAFETY IS A FIELD, NOT A TONE OF VOICE. Every reclaim candidate carries +// `safety`: 'regenerable' (the owning tool refetches it on demand — the npm +// content cache, the Homebrew download cache, the brain's superseded KB copies) +// or 'review' (plausible but NOT safe to state as removable — mise has eight +// node entries on this machine and some of them are the aliases a live +// toolchain resolves through; a browser revision may still be pinned by an +// installed package). A review row is a pointer at something to look at, never +// a figure to sweep, and `summarizeReclaimables` totals the two tiers +// SEPARATELY on purpose: a combined "you could free N" that mixes them would be +// the honest-measurement contract broken at the last mile. // // `bytesMeaning` says what the bytes on a row are: 'candidate' (the bytes the // row is actually about) or 'installed' (what is on disk, offered as context on @@ -37,21 +45,21 @@ // honest "unattributable", never as a guess and never as a zero. // // One deliberate exception, narrow and documented: orphaned-worktree detection -// reads `/.git/worktrees//gitdir`, which holds a single filesystem -// PATH. That is the same class of datum as `.git/config`'s remote URL, which -// ADR-0025 §7 already sanctions, and it is the only way an orphaned worktree -// can be identified at all. Bounded to 4 KB, parsed as a path and nothing else, -// and skipped entirely when `detectWorktrees` is false. +// (storage-reclaim-detectors.mjs) reads `/.git/worktrees//gitdir`, +// which holds a single filesystem PATH. That is the same class of datum as +// `.git/config`'s remote URL, which ADR-0025 §7 already sanctions, and it is the +// only way an orphaned worktree can be identified at all. Bounded to 4 KB, +// parsed as a path and nothing else, and skipped entirely when +// `detectWorktrees` is false. import fs from 'node:fs'; import path from 'node:path'; -import { home, claudeDir, codexDir, configDir, isWindows } from '../paths.mjs'; +import { home, claudeDir, codexDir, configDir } from '../paths.mjs'; import { defaultOpencodeDbPath } from '../usage-opencode.mjs'; -import { scanNpxStale } from '../npx.mjs'; -import { npxEnvNodes } from './install.mjs'; import { decodeClaudeProjectDir } from './project-sources.mjs'; import { - walkTree, rootMeasurements, measured, unknown, sumMeasurements, statNode, hasValue, + walkTree, rootMeasurements, measured, unknown, sumMeasurements, } from './walk.mjs'; +import { collectReclaimables, summarizeReclaimables } from './storage-reclaim.mjs'; export const STORAGE_CATEGORIES = Object.freeze([ 'transcripts', 'ledgers-and-logs', 'learning-stores', 'kit-caches', @@ -75,21 +83,6 @@ export const STORAGE_DEFAULTS = Object.freeze({ samplePaths: 5, }); -/** The two safety tiers, in the order a panel should present them. Two and not - * three: a "definitely dead" tier would be a claim this module cannot - * substantiate from directory metadata alone. */ -export const RECLAIM_SAFETY_TIERS = Object.freeze(['regenerable', 'review']); - -/** What each tier promises, carried in the payload so no surface has to invent - * the wording — and so the difference between the two is impossible to render - * as the same thing. */ -export const RECLAIM_SAFETY_MEANING = Object.freeze({ - regenerable: 'The owning tool refetches this on demand. Removing it costs download time, ' - + 'not data.', - review: 'Plausible but not safe to call removable: some of these may be in use. Review them ' - + 'individually — this is not a total to sweep.', -}); - /** Host ledger and log files that are named by generation * (`state_5.sqlite`, `logs_2.sqlite`, plus their -wal/-shm siblings). Matching * the family rather than one name is deliberate: codex bumps the generation @@ -321,48 +314,15 @@ function leafKeysFor(root, file) { } /** - * The storage section of a FootprintSnapshot. - * - * `consumers` is an already-collected ranked-consumers payload (consumers.mjs). - * When supplied, a detector that needs the size of a path that view already - * walked adopts that figure instead of walking the tree a second time — the npm - * content cache alone is ~10^5 files, and measuring it twice in one deep scan is - * pure I/O cost for an identical answer. - * - * @param {{ - * projects?: string[]|null, now?: () => number, walk?: typeof walkTree, - * roots?: StorageRoot[]|null, limits?: object, growthDays?: number, topN?: number, - * maxChildren?: number, reclaim?: object, detectWorktrees?: boolean, - * detectCaches?: boolean, detectOrphanedTranscripts?: boolean, - * consumers?: object|null, env?: NodeJS.ProcessEnv, - * decodeDir?: typeof decodeClaudeProjectDir, - * fsImpl?: typeof fs, - * }} [options] + * Walk every storage root once, folding files into the category → host → + * project → session tree and the growth / topN / aged-transcript accumulators + * in the same pass. Split out of `collectStorage` (2026-08 complexity program) + * purely to give the walk loop its own complexity budget — the accumulators and + * the per-file `onFile` handler are unchanged. Return shape left to inference + * (categories/growth/agedTranscripts/transcriptProjects are Maps; + * sessionLeaves/files are the same row arrays `collectStorage` always sorted). */ -export function collectStorage({ - projects = null, - now = Date.now, - walk = walkTree, - roots = null, - limits = {}, - growthDays = STORAGE_DEFAULTS.growthDays, - topN = STORAGE_DEFAULTS.topN, - maxChildren = STORAGE_DEFAULTS.maxChildren, - reclaim = {}, - detectWorktrees = true, - detectCaches = true, - detectOrphanedTranscripts = true, - consumers = null, - env = process.env, - decodeDir = decodeClaudeProjectDir, - fsImpl = fs, -} = {}) { - const asOf = now(); - const opts = { ...STORAGE_DEFAULTS, ...reclaim }; - const rootList = roots ?? defaultStorageRoots({ projects }); - const growthCutoff = asOf - growthDays * 86_400_000; - const agedCutoff = asOf - opts.transcriptAgeDays * 86_400_000; - +function walkStorageRoots(rootList, { walk, limits, fsImpl, topN, growthCutoff, agedCutoff, opts, asOf }) { const categories = new Map(); const growth = new Map(); const sessionLeaves = []; @@ -421,7 +381,7 @@ export function collectStorage({ }); bump(leafParent, bytes, mtimeMs); if (project && root.category === 'transcripts') { - const key = `${root.id}${project}`; + const key = `${root.id} ${project}`; const acc = transcriptProjects.get(key) ?? { host: root.host, rootPath: root.path, dir: project, path: path.join(root.path, project), @@ -494,10 +454,14 @@ export function collectStorage({ category.partial = true; } - // Every category always appears, so a missing slice is never mistaken for a - // rendering gap. A category with no roots is a measured zero — EXCEPT - // learning-stores with no project catalog supplied, whose emptiness is - // ambiguous: "we were given nowhere to look" is not "there is nothing there". + return { categories, growth, sessionLeaves, files, agedTranscripts, transcriptProjects, anyDegraded }; +} + +/** Every category always appears, so a missing slice is never mistaken for a + * rendering gap. A category with no roots is a measured zero — EXCEPT + * learning-stores with no project catalog supplied, whose emptiness is + * ambiguous: "we were given nowhere to look" is not "there is nothing there". */ +function fillMissingStorageCategories(categories, projects) { for (const id of STORAGE_CATEGORIES) { if (categories.has(id)) continue; const node = newNode({ key: id, kind: 'category', label: id }); @@ -508,6 +472,56 @@ export function collectStorage({ } categories.set(id, node); } +} + +/** + * The storage section of a FootprintSnapshot. + * + * `consumers` is an already-collected ranked-consumers payload (consumers.mjs). + * When supplied, a detector that needs the size of a path that view already + * walked adopts that figure instead of walking the tree a second time — the npm + * content cache alone is ~10^5 files, and measuring it twice in one deep scan is + * pure I/O cost for an identical answer. + * + * @param {{ + * projects?: string[]|null, now?: () => number, walk?: typeof walkTree, + * roots?: StorageRoot[]|null, limits?: object, growthDays?: number, topN?: number, + * maxChildren?: number, reclaim?: object, detectWorktrees?: boolean, + * detectCaches?: boolean, detectOrphanedTranscripts?: boolean, + * consumers?: object|null, env?: NodeJS.ProcessEnv, + * decodeDir?: typeof decodeClaudeProjectDir, + * fsImpl?: typeof fs, + * }} [options] + */ +export function collectStorage({ + projects = null, + now = Date.now, + walk = walkTree, + roots = null, + limits = {}, + growthDays = STORAGE_DEFAULTS.growthDays, + topN = STORAGE_DEFAULTS.topN, + maxChildren = STORAGE_DEFAULTS.maxChildren, + reclaim = {}, + detectWorktrees = true, + detectCaches = true, + detectOrphanedTranscripts = true, + consumers = null, + env = process.env, + decodeDir = decodeClaudeProjectDir, + fsImpl = fs, +} = {}) { + const asOf = now(); + const opts = { ...STORAGE_DEFAULTS, ...reclaim }; + const rootList = roots ?? defaultStorageRoots({ projects }); + const growthCutoff = asOf - growthDays * 86_400_000; + const agedCutoff = asOf - opts.transcriptAgeDays * 86_400_000; + + const { + categories, growth, sessionLeaves, files, agedTranscripts, transcriptProjects, anyDegraded, + } = walkStorageRoots(rootList, { walk, limits, fsImpl, topN, growthCutoff, agedCutoff, opts, asOf }); + + fillMissingStorageCategories(categories, projects); const tree = [...categories.values()] .map((node) => finalizeNode(node, { asOf, maxChildren })) @@ -622,861 +636,15 @@ export function buildGrowth(growthByHost, { asOf, growthDays }) { }; } -/** - * Advisory rows only — see this module's header. Nothing here removes anything; - * `cleanupHint` names the CLI that already owns the removal. - * - * @typedef {{ value: number|null, status: string, reason: string|null, - * asOf: number|null, partial: boolean }} Measurement - * @typedef {{ - * id: string, kind: string, label: string, path: string, samplePaths: string[], - * matchedCount: number|null, bytes: Measurement, files: Measurement, - * safety: 'regenerable'|'review', bytesMeaning: 'candidate'|'installed', - * keeps: Array<{ path: string, label: string, bytes: Measurement }>, - * rationale: string, cleanupHint: string|null, advisory: true, - * }} ReclaimableCandidate - */ -export function collectReclaimables({ - asOf, agedTranscripts, transcriptProjects = new Map(), projects, opts, walk, limits, - detectWorktrees, detectCaches = true, detectOrphanedTranscripts = true, - consumers = null, env = process.env, decodeDir = decodeClaudeProjectDir, fsImpl, -}) { - const rows = []; - const days = (ms) => Math.floor((asOf - ms) / 86_400_000); - const ctx = { - asOf, opts, walk, limits, fsImpl, adopt: adoptedConsumerFigures(consumers), - }; - - for (const acc of agedTranscripts.values()) { - rows.push(candidate({ - id: `aged-transcripts:${acc.host}`, - kind: 'aged-transcripts', - label: `${acc.host} transcripts older than ${opts.transcriptAgeDays}d`, - path: acc.root, - samplePaths: acc.samples, - matchedCount: acc.files, - bytes: measured(acc.bytes, { asOf }), - files: measured(acc.files, { asOf }), - // Not regenerable in any sense: a transcript is the only copy of the - // session it records, and Historical usage is denominated in them. - safety: 'review', - rationale: `${acc.files} file(s) untouched for ${opts.transcriptAgeDays}d or more; ` - + `oldest ${days(acc.oldestMtimeMs)}d. Historical usage reads these — removing them ` - + 'removes that history too.', - cleanupHint: null, - })); - } - - rows.push(...npxReclaimables({ asOf, opts, walk, limits, fsImpl })); - if (detectOrphanedTranscripts) { - rows.push(...orphanedTranscriptReclaimables({ - asOf, opts, transcriptProjects, decodeDir, fsImpl, - })); - } - if (detectCaches) { - rows.push(...supersededSnapshotReclaimables(ctx, snapshotFamilies({ env }))); - rows.push(...regenerableCacheReclaimables(ctx, regenerableCacheRoots({ env }))); - rows.push(...browserRevisionReclaimables(ctx, browserRevisionRoots({ env }))); - rows.push(...runtimeVersionReclaimables(ctx, runtimeVersionRoots({ env }))); - } - if (detectWorktrees && Array.isArray(projects)) { - rows.push(...worktreeReclaimables({ asOf, projects, opts, walk, limits, fsImpl })); - } - return rows.sort((a, b) => (b.bytes.value ?? 0) - (a.bytes.value ?? 0)); -} - -/** Do any two of these rows describe the same path, or one inside another? Two - * such rows cover some of the same bytes — an aged transcript can also sit in a - * project that no longer exists — and adding them would report space twice. */ -function rowsOverlap(rows) { - const keys = rows.map((row) => (row.path ? pathKey(row.path) : null)).filter(Boolean); - for (let i = 0; i < keys.length; i++) { - for (let j = i + 1; j < keys.length; j++) { - if (keys[i] === keys[j]) return true; - const rel = path.relative(keys[i], keys[j]); - const nested = rel && !rel.startsWith('..') && !path.isAbsolute(rel); - const inverse = path.relative(keys[j], keys[i]); - if (nested || (inverse && !inverse.startsWith('..') && !path.isAbsolute(inverse))) return true; - } - } - return false; -} - -/** - * Per-tier totals, and deliberately NO combined figure. Summing a regenerable - * cache with a runtime tree that may be live would produce the one number a - * reader would act on and the one number this module cannot stand behind. - * - * A tier whose own rows overlap reports its total as unknown-with-reason rather - * than as a sum that counts the same bytes twice — the rowCount still stands, - * and each row still carries its own measured figure. - */ -export function summarizeReclaimables(rows, { asOf = null } = {}) { - const list = Array.isArray(rows) ? rows : []; - return { - tiers: RECLAIM_SAFETY_TIERS.map((safety) => { - const tier = list.filter((row) => row.safety === safety); - // Only 'candidate' bytes are summable: an 'installed' figure is context on - // a review row, not space the row claims is available. - const members = tier.filter((row) => row.bytesMeaning === 'candidate'); - const overlapping = rowsOverlap(members); - return { - safety, - meaning: RECLAIM_SAFETY_MEANING[safety], - rowCount: tier.length, - bytes: overlapping - ? unknown('rows in this tier describe overlapping paths, so a sum would count the ' - + 'same bytes twice') - : sumMeasurements(members.map((row) => row.bytes), { asOf }), - summedRows: overlapping ? 0 : members.length, - contextOnlyRows: tier.length - members.length, - }; - }), - combined: null, - combinedNote: 'The tiers are reported separately and never added: only the regenerable ' - + 'total is space a tool would rebuild by itself.', - }; -} - -/** Fill in the fields every row carries, so no detector can ship a candidate - * without a safety tier or a statement of what its bytes mean. */ -function candidate(row) { - return { - samplePaths: [], - matchedCount: null, - keeps: [], - bytesMeaning: 'candidate', - cleanupHint: null, - ...row, - advisory: true, - }; -} - -/** Stale npx cache envs. Two independent rationales, both read-only: a cached - * copy strictly older than its installed global baseline (npx.mjs's version - * verdict — the bug that kept a machine running a retired ruflo), and an env - * untouched for longer than the idle threshold. */ -export function npxReclaimables({ asOf, opts, walk, limits, fsImpl }) { - const nodes = npxEnvNodes({ walk, limits, asOf, fsImpl }); - if (nodes.presence !== 'present') return []; - let staleByVersion = new Map(); - try { - staleByVersion = new Map(scanNpxStale().map((entry) => [entry.dir, entry.stale])); - } catch { /* an unreadable cache simply yields no version verdict */ } - const idleCutoff = asOf - opts.npxEnvIdleDays * 86_400_000; - const rows = []; - for (const env of nodes.envs) { - const stale = staleByVersion.get(env.path); - const idle = env.newestMtimeMs !== null && env.newestMtimeMs < idleCutoff; - if (!stale && !idle) continue; - const why = []; - if (stale) { - why.push(`cached ${stale.map((s) => `${s.pkg}@${s.cached}`).join(', ')} ` - + `older than installed ${stale.map((s) => s.installed).join(', ')}`); - } - if (idle) why.push(`untouched for ${Math.floor((asOf - env.newestMtimeMs) / 86_400_000)}d`); - rows.push(candidate({ - id: `stale-npx-env:${env.id}`, - kind: 'stale-npx-env', - label: `npx cache env (${env.packages.join(', ') || 'unkeyed'})`, - path: env.path, - bytes: env.bytes, - files: env.files, - safety: 'regenerable', - rationale: `${why.join('; ')}. npx re-fetches on demand, so the cache is reproducible.`, - cleanupHint: 'ak sync prunes version-stale envs (npx.pruneNpxStale)', - })); - } - return rows; -} - -// ── shared detector plumbing ────────────────────────────────────────────────── - -// Third-party cache conventions are spelled out here rather than in paths.mjs -// for the reason consumers.mjs states: that module owns the kit's own path -// contract, and fifty foreign tools' cache layouts would make it harder to -// audit. Platform variants are listed side by side instead of switched on -// process.platform, so the wrong-platform root simply reads absent and a machine -// carrying both (a tool that moved its cache) reports both. -const xdgCache = (env) => env.XDG_CACHE_HOME || path.join(home, '.cache'); -const xdgData = (env) => env.XDG_DATA_HOME || path.join(home, '.local', 'share'); -const macCache = () => path.join(home, 'Library', 'Caches'); -const winLocalAppData = (env) => env.LOCALAPPDATA || path.join(home, 'AppData', 'Local'); - -// APFS and NTFS are case-insensitive, so an adopted figure keyed by an -// exact-case path would silently miss on macOS and Windows. -const foldCase = process.platform !== 'linux'; -const pathKey = (target) => { - const abs = path.resolve(target); - return foldCase ? abs.toLowerCase() : abs; -}; - -/** - * A lookup from absolute path to a figure the ranked-consumers view already - * measured, or a function that always misses when no such view was supplied. - * Exact paths only: a consumers row for a glob FAMILY carries one total for the - * whole family and cannot answer for an individual member. - * - * @param {{ rows?: any[] }|null} consumers a collectConsumers payload - * @returns {(target: string) => ({ presence: string, bytes: Measurement, - * files: Measurement, newestMtimeMs: number|null })|null} - */ -export function adoptedConsumerFigures(consumers) { - const index = new Map(); - for (const row of consumers?.rows ?? []) { - if (!row?.path || row.residual || row.presence !== 'present' || !hasValue(row.bytes)) continue; - index.set(pathKey(row.path), { - presence: 'present', - bytes: row.bytes, - files: row.files ?? unknown('file count not carried by the adopted figure'), - newestMtimeMs: row.newestMtimeMs ?? null, - }); - } - return (target) => (target ? index.get(pathKey(target)) ?? null : null); -} - -/** One node's figures: the already-measured answer when the consumers view has - * one for this exact path, otherwise one bounded walk. */ -function measureNode(target, ctx) { - const adopted = ctx.adopt?.(target); - if (adopted) return adopted; - const result = ctx.walk(target, { ...ctx.limits, fsImpl: ctx.fsImpl }); - return { - ...rootMeasurements(result, { asOf: ctx.asOf }), - newestMtimeMs: result.newestMtimeMs ?? null, - }; -} - -/** Immediate entries of a directory. ENOENT is an ABSENCE — that tool is not - * installed here, which is not a failed measurement — and every other errno is - * a degradation the caller must report rather than swallow. Symlinked entries - * are classified but never followed or measured. */ -function listMembers(dir, fsImpl) { - try { - return { status: 'ok', reason: null, entries: fsImpl.readdirSync(dir, { withFileTypes: true }) }; - } catch (err) { - const code = err?.code || 'io'; - return { status: code === 'ENOENT' ? 'absent' : 'degraded', reason: code, entries: [] }; - } -} - -/** A root that exists but could not be listed. Reported rather than dropped: - * "there is nothing to reclaim here" and "we could not look" are different - * answers, and only one of them is a measurement (invariant 2). */ -function unreadableCandidate({ id, kind, label, target, reason, safety, cleanupHint = null }) { - return candidate({ - id: `${id}:unreadable`, - kind, - label: `${label} (unreadable)`, - path: target, - bytes: unknown(reason), - files: unknown(reason), - safety, - rationale: `${target} could not be listed (${reason}), so whether anything here is ` - + 'reclaimable is unknown rather than none.', - cleanupHint, - }); -} - -/** Walk a family's members under a walk budget. A cap makes the sum a floor and - * says so through `partial`, which is what "≥ N" renders from; a budget already - * spent before this family was reached yields unknown, because zero members - * measured is not a measurement of zero bytes. */ -function measureMembers(members, ctx, limit = ctx.opts.maxFamilyWalks) { - const walked = members.slice(0, Math.max(0, limit)) - .map((member) => ({ ...member, ...measureNode(member.path, ctx) })); - const capped = members.length > walked.length; - if (members.length && !walked.length) { - const reason = 'the walk budget for this root was spent before this node was reached'; - return { walked, capped, bytes: unknown(reason), files: unknown(reason) }; - } - const bytes = sumMeasurements(walked.map((m) => m.bytes), { asOf: ctx.asOf }); - const files = sumMeasurements(walked.map((m) => m.files), { asOf: ctx.asOf }); - return { - walked, - capped, - bytes: capped && hasValue(bytes) ? { ...bytes, partial: true } : bytes, - files: capped && hasValue(files) ? { ...files, partial: true } : files, - }; -} - -/** Is there anything to advise about? A measured zero is a real zero, and a real - * zero is not a candidate — an "0 B reclaimable" row is an unknown wearing a - * number. An unmeasured figure still earns its row, because not knowing is - * itself the finding. */ -const worthListing = (bytes) => !hasValue(bytes) || bytes.value > 0; - -// ── superseded snapshot copies ──────────────────────────────────────────────── - -/** - * Families of dated copies an installer leaves beside the copy in use. The - * RuvNet Brain is the one on this machine and the largest safe win on it: five - * `kb.bak-` directories totalling ~11 GB beside a 1.9 GB active `kb/`. - * - * @param {{ env?: NodeJS.ProcessEnv }} [options] - */ -export function snapshotFamilies({ env = process.env } = {}) { - const brain = path.join(xdgCache(env), 'ruvnet-brain'); - return [{ - id: 'brain-kb-snapshots', - label: 'RuvNet Brain superseded KB copies', - dir: brain, - // `kb.bak` cannot match `kb`, so the active KB can never be enumerated as - // one of its own backups. - prefix: 'kb.bak', - active: path.join(brain, 'kb'), - activeLabel: 'active KB (kb/)', - what: 'The brain installer copies the knowledge base aside before each update and never ' - + 'removes the copy, so one accumulates per update.', - reproducible: 'A knowledge base is rebuilt by re-running the installer ' - + '(npx ruvnet-brain --doctor).', - cleanupHint: 'remove the dated kb.bak-* directories (npx ruvnet-brain --doctor rebuilds)', - }]; -} - -/** The `YYYY-MM-DD` a dated copy names, when it names one. Used only for the - * rationale's range; a member whose name carries no date is still counted. */ -const datePart = (name) => name.match(/(\d{4}-\d{2}-\d{2})/)?.[1] ?? null; - -/** - * Dated, superseded copies beside an active one — the single largest safe win - * measured on this machine. The active copy is measured too and reported in - * `keeps`, never inside the candidate figure: the row's whole credibility is - * that it can say what it is NOT proposing to touch. - */ -export function supersededSnapshotReclaimables(ctx, families) { - const rows = []; - for (const family of families ?? []) { - const listing = listMembers(family.dir, ctx.fsImpl); - if (listing.status === 'absent') continue; - if (listing.status === 'degraded') { - rows.push(unreadableCandidate({ - id: family.id, kind: 'superseded-snapshots', label: family.label, - target: family.dir, reason: listing.reason, safety: 'regenerable', - cleanupHint: family.cleanupHint, - })); - continue; - } - const members = listing.entries - .filter((entry) => entry.isDirectory() && !entry.isSymbolicLink() - && entry.name.startsWith(family.prefix)) - .map((entry) => ({ name: entry.name, path: path.join(family.dir, entry.name) })) - .sort((a, b) => a.name.localeCompare(b.name)); - if (!members.length) continue; - - const { walked, capped, bytes, files } = measureMembers(members, ctx); - if (!worthListing(bytes)) continue; - const dates = members.map((m) => datePart(m.name)).filter(Boolean); - const span = dates.length >= 2 ? ` (${dates[0]} through ${dates[dates.length - 1]})` - : (dates.length === 1 ? ` (${dates[0]})` : ''); - const active = measureNode(family.active, ctx); - rows.push(candidate({ - id: family.id, - kind: 'superseded-snapshots', - label: `${members.length} superseded copies of ${path.basename(family.active)}`, - path: family.dir, - samplePaths: walked.slice(0, ctx.opts.samplePaths).map((m) => m.path), - matchedCount: members.length, - bytes, - files, - safety: 'regenerable', - keeps: [{ - path: family.active, - label: family.activeLabel, - bytes: active.presence === 'absent' - ? unknown('the active copy is not on disk') - : active.bytes, - }], - rationale: `${members.length} dated copies${span}${capped ? ', of which only ' - + `${walked.length} were measured` : ''}. ${family.what} Nothing reads them: ` - + `the ${family.activeLabel} is measured separately, is excluded from this figure, and ` - + `is not a candidate. ${family.reproducible}`, - cleanupHint: family.cleanupHint, - })); - } - return rows; -} - -// ── regenerable caches ──────────────────────────────────────────────────────── - -/** - * Whole cache roots whose owner refetches them on demand. These are the biggest - * genuinely-safe rows on a developer machine — the npm content-addressable cache - * alone measures ~22 GB here — and they are invisible in the per-host storage - * tree because they belong to no host. - * - * @param {{ env?: NodeJS.ProcessEnv }} [options] - */ -export function regenerableCacheRoots({ env = process.env } = {}) { - const cache = xdgCache(env); - const roots = [ - { - id: 'npm-cacache', - kind: 'regenerable-cache', - label: 'npm content-addressable cache', - path: path.join(home, '.npm', '_cacache'), - what: 'Every package tarball and registry response npm has downloaded, keyed by content ' - + 'hash. It grows monotonically: npm adds to it and never prunes it.', - cleanupHint: 'npm cache clean --force', - }, - { - id: 'homebrew-downloads', - kind: 'regenerable-cache', - label: 'Homebrew download cache', - path: path.join(macCache(), 'Homebrew'), - what: 'Bottles, source tarballs and the formula API responses brew downloaded, kept after ' - + 'the install they were for.', - cleanupHint: 'brew cleanup', - }, - { - id: 'homebrew-downloads-xdg', - kind: 'regenerable-cache', - label: 'Homebrew download cache', - path: path.join(cache, 'Homebrew'), - what: 'Bottles, source tarballs and the formula API responses brew downloaded, kept after ' - + 'the install they were for.', - cleanupHint: 'brew cleanup', - }, - ]; - if (isWindows) { - roots.push({ - id: 'npm-cacache-win', - kind: 'regenerable-cache', - label: 'npm content-addressable cache', - path: path.join(winLocalAppData(env), 'npm-cache', '_cacache'), - what: 'Every package tarball and registry response npm has downloaded, keyed by content ' - + 'hash. It grows monotonically: npm adds to it and never prunes it.', - cleanupHint: 'npm cache clean --force', - }); - } - return roots; -} - -/** One row per present cache root. An absent root produces nothing at all — - * a tool that is not installed is not a reclaimable zero. */ -export function regenerableCacheReclaimables(ctx, roots) { - const rows = []; - for (const root of roots ?? []) { - const node = measureNode(root.path, ctx); - if (node.presence === 'absent') continue; - if (!worthListing(node.bytes)) continue; - rows.push(candidate({ - id: root.id, - kind: root.kind, - label: root.label, - path: root.path, - bytes: node.bytes, - files: node.files, - safety: 'regenerable', - rationale: `${root.what} Nothing here is unique: the tool refetches what it needs on the ` - + 'next install, so the cost of clearing it is download time, not data.', - cleanupHint: root.cleanupHint, - })); - } - return rows; -} - -// ── superseded browser downloads ────────────────────────────────────────────── - -/** - * Roots where a browser installer keeps one directory per revision and adds a - * new one on every version bump, never removing the old. `depth` is where the - * revision directories live: Playwright keeps them at the root - * (`chromium-1223`), Puppeteer one level down (`chrome/mac_arm-149.0.7827.22`). - * - * @param {{ env?: NodeJS.ProcessEnv }} [options] - */ -export function browserRevisionRoots({ env = process.env } = {}) { - const cache = xdgCache(env); - const playwright = { - kind: 'superseded-browser-revisions', - label: 'Playwright browser builds', - depth: 1, - installer: 'npx playwright install', - }; - return [ - { ...playwright, id: 'playwright-mac', path: path.join(macCache(), 'ms-playwright') }, - { ...playwright, id: 'playwright-xdg', path: path.join(cache, 'ms-playwright') }, - { ...playwright, id: 'playwright-win', path: path.join(winLocalAppData(env), 'ms-playwright') }, - { - kind: 'superseded-browser-revisions', - id: 'puppeteer', - label: 'Puppeteer browser builds', - path: path.join(cache, 'puppeteer'), - depth: 2, - installer: 'npx puppeteer browsers install', - }, - ]; -} - -/** Split `chromium_headless_shell-1223` into its family and its revision. The - * revision must contain a digit, which is what keeps Playwright's - * `mcp-chrome-profile` — a browser PROFILE, not a revision — out of the - * families entirely. */ -function splitRevision(name) { - const cut = name.lastIndexOf('-'); - if (cut <= 0 || cut === name.length - 1) return null; - const revision = name.slice(cut + 1); - if (!/\d/.test(revision)) return null; - return { family: name.slice(0, cut), revision }; -} - -/** Revision directories under a root, at the root itself (`depth` 1) or one - * level down (`depth` 2). Never recursive: a browser build's own contents are - * not revisions. */ -function revisionMembers(root, ctx) { - const listing = listMembers(root.path, ctx.fsImpl); - if (listing.status !== 'ok') return { status: listing.status, reason: listing.reason, members: [] }; - const holders = root.depth === 2 - ? listing.entries.filter((entry) => entry.isDirectory() && !entry.isSymbolicLink()) - .map((entry) => ({ dir: path.join(root.path, entry.name), scope: entry.name })) - : [{ dir: root.path, scope: '' }]; - const members = []; - for (const holder of holders) { - const inner = root.depth === 2 ? listMembers(holder.dir, ctx.fsImpl) : listing; - if (inner.status !== 'ok') continue; - for (const entry of inner.entries) { - if (!entry.isDirectory() || entry.isSymbolicLink()) continue; - const split = splitRevision(entry.name); - if (!split) continue; - const target = path.join(holder.dir, entry.name); - members.push({ - name: entry.name, - path: target, - family: holder.scope ? `${holder.scope}/${split.family}` : split.family, - revision: split.revision, - mtimeMs: statNode(target, { fsImpl: ctx.fsImpl }).mtimeMs ?? 0, - }); - } - } - return { status: 'ok', reason: null, members }; -} - -/** - * Older revisions in each browser family — everything except the most recently - * installed one. Ordering is by directory mtime rather than by parsing the - * revision, because the revisions are not comparable across installers - * (Playwright's `1223` is a counter, its `mcp-chrome-5b42311` is a hex build id, - * Puppeteer's is a four-part Chrome version). - * - * REVIEW tier, not regenerable, even though the installer would refetch them: a - * project's pinned playwright/puppeteer version resolves to a specific revision, - * and this module cannot see which package pins what. The row's job is to say - * "these accumulated, look at them", not "delete N GB". - */ -export function browserRevisionReclaimables(ctx, roots) { - const rows = []; - for (const root of roots ?? []) { - const { status, reason, members } = revisionMembers(root, ctx); - if (status === 'absent') continue; - if (status === 'degraded') { - rows.push(unreadableCandidate({ - id: root.id, kind: root.kind, label: root.label, - target: root.path, reason, safety: 'review', - })); - continue; - } - const families = new Map(); - for (const member of members) { - const list = families.get(member.family) ?? []; - list.push(member); - families.set(member.family, list); - } - const superseded = []; - const keeps = []; - for (const [family, list] of families) { - if (list.length < 2) continue; - const ordered = [...list].sort((a, b) => b.mtimeMs - a.mtimeMs || b.name.localeCompare(a.name)); - keeps.push({ family, ...ordered[0] }); - superseded.push(...ordered.slice(1)); - } - if (!superseded.length) continue; - const { walked, capped, bytes, files } = measureMembers(superseded, ctx); - if (!worthListing(bytes)) continue; - rows.push(candidate({ - id: `${root.id}:superseded`, - kind: root.kind, - label: `${superseded.length} superseded ${root.label.toLowerCase()}`, - path: root.path, - samplePaths: walked.slice(0, ctx.opts.samplePaths).map((m) => m.path), - matchedCount: superseded.length, - bytes, - files, - safety: 'review', - keeps: keeps.map((keep) => ({ - path: keep.path, - label: `newest ${keep.family} revision (${keep.revision})`, - bytes: unknown('the retained revision is not part of this figure'), - })), - rationale: `${keeps.length} browser famil(ies) here carry more than one revision; this ` - + `figure covers the ${superseded.length} older one(s)${capped - ? `, of which ${walked.length} were measured` : ''}, and excludes the newest of each. ` - + 'Review rather than sweep: an installed package can pin an older revision, and this ' - + 'row cannot see which package pins what. ' - + `${root.installer} refetches whatever is missing.`, - cleanupHint: `${root.installer} (confirm no project pins the older revision first)`, - })); - } - return rows; -} - -// ── installed runtime versions ──────────────────────────────────────────────── - -/** - * Version-manager install roots — one directory per installed runtime version, - * plus alias entries pointing into them. - * - * @param {{ env?: NodeJS.ProcessEnv }} [options] - */ -export function runtimeVersionRoots({ env = process.env } = {}) { - return [{ - id: 'mise-installs', - kind: 'installed-runtime-versions', - label: 'mise', - path: path.join(xdgData(env), 'mise', 'installs'), - manager: 'mise', - cleanupHint: 'mise ls, then mise uninstall @ (nothing may pin it)', - }]; -} - -/** - * Installed runtime versions, one row per managed tool that has more than one. - * - * REVIEW tier and `bytesMeaning: 'installed'`, both deliberately. On this - * machine `mise/installs/node` holds eight entries — 22, 22.22, 22.22.3, 26, - * 26.4, 26.4.0, latest, lts-jod — of which only two are real directories and six - * are ALIAS SYMLINKS resolving into them. Removing a version silently breaks - * every alias that points at it, and any `mise.toml` or `.tool-versions` on the - * machine can pin any of them, so there is no subset this module can honestly - * call reclaimable. What it can do is show what is installed and say that out - * loud; recommending the deletion of a live runtime would be worse than saying - * nothing. - * - * The alias links are counted, never followed — the walker's rule, and also the - * only reason the byte figure is not multiplied by every alias. - */ -export function runtimeVersionReclaimables(ctx, roots) { - const rows = []; - for (const root of roots ?? []) { - const listing = listMembers(root.path, ctx.fsImpl); - if (listing.status === 'absent') continue; - if (listing.status === 'degraded') { - rows.push(unreadableCandidate({ - id: root.id, kind: root.kind, label: `${root.manager} installs`, - target: root.path, reason: listing.reason, safety: 'review', - })); - continue; - } - // Every managed tool is examined — listing one is a readdir, and capping the - // LIST would drop tools in alphabetical order, which is an invisible - // truncation of exactly the kind this domain forbids. What is bounded is the - // expensive part: the walks, under one budget shared across the root. - let budget = ctx.opts.maxFamilyWalks; - const tools = listing.entries - .filter((entry) => entry.isDirectory() && !entry.isSymbolicLink() - && !entry.name.startsWith('.')); - for (const tool of tools) { - const dir = path.join(root.path, tool.name); - const inner = listMembers(dir, ctx.fsImpl); - if (inner.status !== 'ok') continue; - const versions = []; - let aliases = 0; - for (const entry of inner.entries) { - if (entry.name.startsWith('.')) continue; - if (entry.isSymbolicLink()) { aliases += 1; continue; } - if (entry.isDirectory()) versions.push({ name: entry.name, path: path.join(dir, entry.name) }); - } - // One installed version is the toolchain working as intended, not sprawl. - if (versions.length < 2) continue; - const { walked, capped, bytes, files } = measureMembers(versions, ctx, budget); - budget -= walked.length; - if (!worthListing(bytes)) continue; - rows.push(candidate({ - id: `${root.id}:${tool.name}`, - kind: root.kind, - label: `${versions.length} ${root.manager} ${tool.name} versions installed`, - path: dir, - samplePaths: walked.slice(0, ctx.opts.samplePaths).map((m) => m.path), - matchedCount: versions.length, - bytes, - files, - safety: 'review', - bytesMeaning: 'installed', - rationale: `${versions.length} installed version(s) of ${tool.name}` - + `${aliases ? ` plus ${aliases} alias link(s) resolving into them` : ''}` - + `${capped ? `, of which ${walked.length} were measured` : ''}. ` - + 'This is what is installed, not what is free: an alias points at a real version, ' - + `and any ${root.manager} config on this machine can pin any of them. Review with ` - + `\`${root.manager} ls ${tool.name}\` before removing anything.`, - cleanupHint: root.cleanupHint, - })); - } - } - return rows; -} - -// ── transcripts for projects that no longer exist ───────────────────────────── - -/** - * Claude transcript directories whose project directory is gone. - * - * A `~/.claude/projects/` directory name is a LOSSY encoding of the project path - * (`/`, `.` and a literal `-` all become `-`), so it cannot be decoded by string - * manipulation; `decodeClaudeProjectDir` walks the candidate segments against the - * real filesystem and returns a path only when the filesystem confirms one. - * A directory that decodes to nothing is therefore a project that is no longer - * on this machine — and that is exactly the evidence available, so the row says - * so rather than claiming certainty. - * - * TWO GUARDS against the failure mode that would matter, flagging live projects: - * · only `-`-leading (POSIX-encoded) names are considered, because Windows - * names encode a drive letter that this decoder does not handle and would - * otherwise report every project on the machine as dead; - * · if NOTHING decoded, the finding is about the decoder or an unreadable home - * directory, not about the projects, so no row is emitted at all. - * - * The value here is hygiene and privacy, not space: measured on this machine - * these are ~0.05 GB across 8 dead projects. The rationale says that plainly — - * selling 50 MB as a disk win would be the same dishonesty as a fabricated zero, - * just in the other direction. - */ -export function orphanedTranscriptReclaimables({ - asOf, opts, transcriptProjects, decodeDir = decodeClaudeProjectDir, fsImpl, -}) { - const byHost = new Map(); - for (const entry of transcriptProjects?.values?.() ?? []) { - if (!entry.dir.startsWith('-')) continue; - const acc = byHost.get(entry.host) - ?? { host: entry.host, root: entry.rootPath, alive: 0, dead: [] }; - if (decodeDir(entry.dir, { fsImpl })) acc.alive += 1; - else acc.dead.push(entry); - byHost.set(entry.host, acc); - } - - const rows = []; - for (const acc of byHost.values()) { - if (!acc.dead.length || !acc.alive) continue; - const bytes = acc.dead.reduce((total, entry) => total + entry.bytes, 0); - const files = acc.dead.reduce((total, entry) => total + entry.files, 0); - const newest = acc.dead.reduce((at, entry) => Math.max(at, entry.newestMtimeMs ?? 0), 0); - rows.push(candidate({ - id: `orphaned-transcripts:${acc.host}`, - kind: 'orphaned-transcripts', - label: `${acc.host} transcripts for ${acc.dead.length} project(s) that no longer exist`, - path: acc.root, - samplePaths: acc.dead.slice(0, opts.samplePaths).map((entry) => entry.path), - matchedCount: acc.dead.length, - bytes: measured(bytes, { asOf }), - files: measured(files, { asOf }), - safety: 'review', - rationale: `${files} transcript(s) belong to ${acc.dead.length} project director(ies) that ` - + `no longer resolve on this machine (${acc.alive} others still do; last activity ` - + `${newest ? `${Math.floor((asOf - newest) / 86_400_000)}d ago` : 'unknown'}). This row ` - + 'is here for hygiene and privacy, not as a space win: it is listed because those ' - + 'projects are gone, whatever the byte figure turns out to be. The transcripts are also ' - + 'the only copy of that history, and an unreadable parent directory looks identical to ' - + 'a deleted project — confirm the project is really gone before acting.', - cleanupHint: null, - })); - } - return rows; -} - -/** Orphaned git worktrees, from each project's `.git/worktrees/` admin - * records. Two honest verdicts, no git invocation: - * · the checkout the record points at no longer exists → the record is dead; - * · the checkout exists but nothing in it has been touched for the idle - * window → a candidate, with its real on-disk size. - * A record whose pointer cannot be read is reported as unverifiable rather - * than assumed dead. See this module's header for why the pointer read is in - * scope. */ -export function worktreeReclaimables({ asOf, projects, opts, walk, limits, fsImpl }) { - const rows = []; - let walks = 0; - for (const project of projects) { - const adminRoot = path.join(project, '.git', 'worktrees'); - let entries; - try { - entries = fsImpl.readdirSync(adminRoot, { withFileTypes: true }); - } catch { continue; } // no worktrees here (or unreadable): nothing to claim - for (const entry of entries) { - if (!entry.isDirectory()) continue; - const record = path.join(adminRoot, entry.name); - const pointer = readGitdirPointer(path.join(record, 'gitdir'), fsImpl); - if (!pointer) { - rows.push(candidate({ - id: `orphaned-worktree:${record}`, - kind: 'orphaned-worktree', - label: `worktree record "${entry.name}" (unverifiable)`, - path: record, - bytes: unknown('worktree pointer unreadable'), - files: unknown('worktree pointer unreadable'), - safety: 'review', - rationale: 'The admin record exists but its gitdir pointer could not be read, ' - + 'so whether the checkout still exists is unknown.', - cleanupHint: 'git worktree prune (verify first)', - })); - continue; - } - // gitdir points at `/.git`; the checkout is its parent. - const checkout = path.dirname(pointer); - const head = statNode(checkout, { fsImpl }); - if (head.status === 'unknown') { - rows.push(candidate({ - id: `orphaned-worktree:${record}`, - kind: 'orphaned-worktree', - label: `orphaned worktree record "${entry.name}"`, - path: record, - samplePaths: [checkout], - bytes: measured(0, { asOf }), - files: measured(0, { asOf }), - safety: 'review', - rationale: `The checkout at ${checkout} no longer exists; only the administrative ` - + 'record remains.', - cleanupHint: 'git worktree prune', - })); - continue; - } - if (walks >= opts.maxWorktreeWalks) continue; - walks += 1; - const result = walk(checkout, { ...limits, fsImpl }); - const { bytes, files } = rootMeasurements(result, { asOf }); - const idleMs = result.newestMtimeMs === null ? null : asOf - result.newestMtimeMs; - if (idleMs === null || idleMs < opts.worktreeIdleDays * 86_400_000) continue; - rows.push(candidate({ - id: `idle-worktree:${record}`, - kind: 'orphaned-worktree', - label: `idle worktree "${entry.name}"`, - path: checkout, - samplePaths: [record], - bytes, - files, - safety: 'review', - rationale: `Nothing in the checkout has changed for ${Math.floor(idleMs / 86_400_000)}d. ` - + 'Merge state is not checked here — confirm the branch is landed before removing.', - cleanupHint: 'git worktree remove (verify the branch is merged first)', - })); - } - } - return rows; -} - -/** The single narrow non-metadata read in this module: a `gitdir` file holds - * one absolute path and nothing else. Bounded to 4 KB and validated as a path - * so a corrupt file yields null rather than a bogus candidate. */ -function readGitdirPointer(file, fsImpl) { - let fd; - try { - fd = fsImpl.openSync(file, 'r'); - const buf = Buffer.alloc(4096); - const read = fsImpl.readSync(fd, buf, 0, 4096, 0); - const value = buf.toString('utf8', 0, read).trim(); - return value && path.isAbsolute(value) ? value : null; - } catch { - return null; - } finally { - if (fd !== undefined) { - try { fsImpl.closeSync(fd); } catch { /* already gone */ } - } - } -} +// ── re-exports: the reclaim half of this domain (see this file's header) ────── +export { + RECLAIM_SAFETY_TIERS, RECLAIM_SAFETY_MEANING, + collectReclaimables, summarizeReclaimables, npxReclaimables, adoptedConsumerFigures, +} from './storage-reclaim.mjs'; +export { + snapshotFamilies, supersededSnapshotReclaimables, + regenerableCacheRoots, regenerableCacheReclaimables, + browserRevisionRoots, browserRevisionReclaimables, + runtimeVersionRoots, runtimeVersionReclaimables, + orphanedTranscriptReclaimables, worktreeReclaimables, +} from './storage-reclaim-detectors.mjs'; diff --git a/src/lib/live/claude-adapter.mjs b/src/lib/live/claude-adapter.mjs index d9e8268..d373664 100644 --- a/src/lib/live/claude-adapter.mjs +++ b/src/lib/live/claude-adapter.mjs @@ -2,76 +2,83 @@ import { createLiveEvent } from './event-schema.mjs'; import { classifyToolName } from './tool-classify.mjs'; import { artifactName, decodeClaudeRecord } from '../telemetry-records.mjs'; -/** Translate one Claude transcript record into privacy-safe metadata events. */ -export function adaptClaudeRecord(record, context = {}) { - if (!record || typeof record !== 'object') return []; - const decoded = decodeClaudeRecord(record); - const sessionId = decoded.sessionId ?? context.sessionId; - if (typeof sessionId !== 'string' || !sessionId) return []; - const actorId = decoded.agentId ?? context.agentId ?? sessionId; - const isSidechain = decoded.isSidechain || actorId !== sessionId; - const base = { +/** The `actor` half of a Claude record's live-event base. */ +function buildClaudeRecordActor({ actorId, isSidechain, record, context, decoded }) { + return { + id: actorId, kind: isSidechain ? 'subagent' : 'session', + role: isSidechain ? 'worker' : 'primary', + provider: record.provider ?? context.provider, + model: decoded.model ?? context.model, + }; +} + +/** The `source.fields` half of a Claude record's live-event base. */ +function buildClaudeRecordSourceFields({ context, record, decoded, isSidechain }) { + return { + project: context.project ? 'observed' : null, + // A provider on the record itself is observed evidence; one resolved + // from the host's configuration surface carries that resolution's own + // provenance (configured/inferred) and must not be upgraded. + provider: record.provider ? 'observed' + : (context.provider ? context.providerProvenance ?? 'configured' : null), + model: decoded.model || context.model ? 'observed' : null, + status: 'observed', + hierarchy: isSidechain ? (decoded.agentId ? 'observed' : 'inferred') : 'observed', + workspace: context.workspace ? context.workspace.confidence ?? 'observed' : null, + }; +} + +/** Fields shared by every live event derived from one decoded Claude record. */ +function buildClaudeRecordBase({ sessionId, actorId, isSidechain, record, context, decoded }) { + return { sessionId, host: 'claude', surface: 'native', project: context.project, projectKey: context.projectKey, observedAt: context.observedAt, sourceTimestamp: record.timestamp, workspace: context.workspace, + actor: buildClaudeRecordActor({ actorId, isSidechain, record, context, decoded }), + source: { + adapter: 'claude-transcript', artifact: artifactName(context.artifact), + confidence: isSidechain && !decoded.agentId ? 'inferred' : 'observed', + fields: buildClaudeRecordSourceFields({ context, record, decoded, isSidechain }), + }, + }; +} + +/** The "primary contains subagent" edge event for a newly-seen sidechain actor. */ +function buildClaudeContainmentEvent({ base, sessionId, actorId, decoded, record, context }) { + return createLiveEvent({ + ...base, actor: { - id: actorId, kind: isSidechain ? 'subagent' : 'session', - role: isSidechain ? 'worker' : 'primary', + id: sessionId, kind: 'session', role: 'primary', provider: record.provider ?? context.provider, model: decoded.model ?? context.model, }, + action: 'contains', status: 'unknown', + signal: { kind: 'relationship', phase: 'observed' }, + target: { id: actorId, kind: 'subagent', role: 'worker' }, source: { - adapter: 'claude-transcript', artifact: artifactName(context.artifact), - confidence: isSidechain && !decoded.agentId ? 'inferred' : 'observed', - fields: { - project: context.project ? 'observed' : null, - // A provider on the record itself is observed evidence; one resolved - // from the host's configuration surface carries that resolution's own - // provenance (configured/inferred) and must not be upgraded. - provider: record.provider ? 'observed' - : (context.provider ? context.providerProvenance ?? 'configured' : null), - model: decoded.model || context.model ? 'observed' : null, - status: 'observed', - hierarchy: isSidechain ? (decoded.agentId ? 'observed' : 'inferred') : 'observed', - workspace: context.workspace ? context.workspace.confidence ?? 'observed' : null, - }, + ...base.source, + confidence: decoded.agentId ? 'observed' : 'inferred', + fields: { ...base.source.fields, + hierarchy: decoded.agentId ? 'observed' : 'inferred' }, }, - }; - const out = []; + }); +} + +/** The containment edge to emit for this record, or null when the actor was + * already announced once. `context.containedActors` is mutated as a side + * effect (created on first use), the same seen-set used across records. */ +function resolveClaudeContainment({ base, sessionId, actorId, isSidechain, decoded, record, context }) { const containedActors = context.containedActors instanceof Set ? context.containedActors : (context.containedActors = new Set()); - let containment = null; - if (isSidechain && actorId !== sessionId && !containedActors.has(actorId)) { - containedActors.add(actorId); - containment = createLiveEvent({ - ...base, - actor: { - id: sessionId, kind: 'session', role: 'primary', - provider: record.provider ?? context.provider, - model: decoded.model ?? context.model, - }, - action: 'contains', status: 'unknown', - signal: { kind: 'relationship', phase: 'observed' }, - target: { id: actorId, kind: 'subagent', role: 'worker' }, - source: { - ...base.source, - confidence: decoded.agentId ? 'observed' : 'inferred', - fields: { ...base.source.fields, - hierarchy: decoded.agentId ? 'observed' : 'inferred' }, - }, - }); - } - if (context.bootstrap && (decoded.role === 'user' || decoded.role === 'assistant')) { - out.push(createLiveEvent({ ...base, action: 'session.discovered', status: 'unknown' })); - if (containment) out.push(containment); - return out; - } else if (decoded.role === 'user' || decoded.role === 'assistant') { - out.push(createLiveEvent({ - ...base, action: decoded.role === 'user' ? 'session.input' : 'agent.output', - status: 'running', - })); - } + if (!isSidechain || actorId === sessionId || containedActors.has(actorId)) return null; + containedActors.add(actorId); + return buildClaudeContainmentEvent({ base, sessionId, actorId, decoded, record, context }); +} + +/** tool_use/tool_result blocks → their tool.started/tool.completed events. */ +function claudeToolEvents(base, decoded) { + const out = []; for (const use of decoded.toolUses) { if (typeof use.id !== 'string') continue; const typed = classifyToolName(use.name); @@ -89,6 +96,34 @@ export function adaptClaudeRecord(record, context = {}) { target: { id: result.id, kind: 'tool' }, })); } + return out; +} + +/** Translate one Claude transcript record into privacy-safe metadata events. */ +export function adaptClaudeRecord(record, context = {}) { + if (!record || typeof record !== 'object') return []; + const decoded = decodeClaudeRecord(record); + const sessionId = decoded.sessionId ?? context.sessionId; + if (typeof sessionId !== 'string' || !sessionId) return []; + const actorId = decoded.agentId ?? context.agentId ?? sessionId; + const isSidechain = decoded.isSidechain || actorId !== sessionId; + const base = buildClaudeRecordBase({ sessionId, actorId, isSidechain, record, context, decoded }); + const containment = resolveClaudeContainment({ base, sessionId, actorId, isSidechain, decoded, record, context }); + + const out = []; + const isTurn = decoded.role === 'user' || decoded.role === 'assistant'; + if (context.bootstrap && isTurn) { + out.push(createLiveEvent({ ...base, action: 'session.discovered', status: 'unknown' })); + if (containment) out.push(containment); + return out; + } + if (isTurn) { + out.push(createLiveEvent({ + ...base, action: decoded.role === 'user' ? 'session.input' : 'agent.output', + status: 'running', + })); + } + out.push(...claudeToolEvents(base, decoded)); if (containment) out.push(containment); return out; } diff --git a/src/lib/live/codex-adapter.mjs b/src/lib/live/codex-adapter.mjs index e365a65..ae0424a 100644 --- a/src/lib/live/codex-adapter.mjs +++ b/src/lib/live/codex-adapter.mjs @@ -2,48 +2,50 @@ import { createLiveEvent } from './event-schema.mjs'; import { classifyToolName } from './tool-classify.mjs'; import { artifactName, decodeCodexRecord, resolveCodexProvider } from '../telemetry-records.mjs'; -export function adaptCodexRecord(record, context = {}) { - if (!record || typeof record !== 'object') return []; - const payload = record.payload && typeof record.payload === 'object' ? record.payload : {}; - const decoded = decodeCodexRecord(record); - const meta = record.type === 'session_meta' ? payload : context.meta ?? {}; - // codex spells this model_provider in rollout session_meta; bare provider is - // legacy tolerance only. `decoded.provider` already applies that same - // lookup (resolveCodexProvider) to THIS record; a carried-forward meta from - // an earlier session_meta needs the identical lookup applied directly, - // since decodeCodexRecord only ever sees one record at a time. - const metaProvider = decoded.type === 'meta' ? decoded.provider : resolveCodexProvider(meta); - const sessionId = meta.id ?? context.sessionId; - if (typeof sessionId !== 'string' || !sessionId) return []; - const subagent = meta.thread_source === 'subagent' || context.threadSource === 'subagent'; - const base = { +/** The `actor` half of a decoded-record's live-event base. */ +function buildCodexRecordActor({ sessionId, subagent, meta, metaProvider, context, decoded, record }) { + return { + id: sessionId, kind: subagent ? 'subagent' : 'session', + label: meta.agent_nickname ?? context.agentNickname, + role: meta.agent_role ?? context.agentRole ?? (subagent ? 'worker' : 'primary'), + provider: metaProvider ?? context.provider, + model: record.type === 'turn_context' + ? decoded.model ?? context.model : meta.model ?? context.model, + }; +} + +/** The `source.fields` half of a decoded-record's live-event base. */ +function buildCodexRecordSourceFields({ context, metaProvider, record, decoded, meta, subagent }) { + return { + project: context.project ? 'observed' : null, + provider: metaProvider || context.provider ? 'observed' : null, + model: (record.type === 'turn_context' ? decoded.model : meta.model) || context.model + ? 'observed' : null, + status: 'observed', + hierarchy: subagent ? (meta.thread_source === 'subagent' ? 'observed' : 'correlated') : 'observed', + workspace: context.workspace ? context.workspace.confidence ?? 'observed' : null, + }; +} + +/** Fields shared by every live event derived from one decoded rollout record. */ +function buildCodexRecordBase({ record, context, decoded, meta, metaProvider, sessionId, subagent }) { + return { sessionId, parentSessionId: context.parentSessionId, host: 'codex', surface: 'native', project: context.project, projectKey: context.projectKey, observedAt: context.observedAt, sourceTimestamp: record.timestamp, workspace: context.workspace, - actor: { - id: sessionId, kind: subagent ? 'subagent' : 'session', - label: meta.agent_nickname ?? context.agentNickname, - role: meta.agent_role ?? context.agentRole ?? (subagent ? 'worker' : 'primary'), - provider: metaProvider ?? context.provider, - model: record.type === 'turn_context' - ? decoded.model ?? context.model : meta.model ?? context.model, - }, + actor: buildCodexRecordActor({ sessionId, subagent, meta, metaProvider, context, decoded, record }), source: { adapter: 'codex-rollout', artifact: artifactName(context.artifact), confidence: subagent && meta.thread_source !== 'subagent' ? 'correlated' : 'observed', - fields: { - project: context.project ? 'observed' : null, - provider: metaProvider || context.provider ? 'observed' : null, - model: (record.type === 'turn_context' ? decoded.model : meta.model) || context.model - ? 'observed' : null, - status: 'observed', - hierarchy: subagent ? (meta.thread_source === 'subagent' ? 'observed' : 'correlated') : 'observed', - workspace: context.workspace ? context.workspace.confidence ?? 'observed' : null, - }, + fields: buildCodexRecordSourceFields({ context, metaProvider, record, decoded, meta, subagent }), }, }; +} + +/** One decoded record → zero or one live events, dispatched on decoded.type. */ +function codexRecordEvents(decoded, base, context) { if (decoded.type === 'meta') { return [createLiveEvent({ ...base, @@ -87,76 +89,120 @@ export function adaptCodexRecord(record, context = {}) { return []; } +export function adaptCodexRecord(record, context = {}) { + if (!record || typeof record !== 'object') return []; + const payload = record.payload && typeof record.payload === 'object' ? record.payload : {}; + const decoded = decodeCodexRecord(record); + const meta = record.type === 'session_meta' ? payload : context.meta ?? {}; + // codex spells this model_provider in rollout session_meta; bare provider is + // legacy tolerance only. `decoded.provider` already applies that same + // lookup (resolveCodexProvider) to THIS record; a carried-forward meta from + // an earlier session_meta needs the identical lookup applied directly, + // since decodeCodexRecord only ever sees one record at a time. + const metaProvider = decoded.type === 'meta' ? decoded.provider : resolveCodexProvider(meta); + const sessionId = meta.id ?? context.sessionId; + if (typeof sessionId !== 'string' || !sessionId) return []; + const subagent = meta.thread_source === 'subagent' || context.threadSource === 'subagent'; + const base = buildCodexRecordBase({ record, context, decoded, meta, metaProvider, sessionId, subagent }); + return codexRecordEvents(decoded, base, context); +} + +/** The ledger's own status enum, else `'unknown'` for anything it doesn't name. */ +function codexThreadStatus(thread) { + return ['queued', 'running', 'completed', 'failed', 'cancelled'].includes(thread?.status) + ? thread.status : 'unknown'; +} + +/** The `actor` half of a ledger thread's discovered event. */ +function ledgerThreadActor(id, thread, subagent) { + return { + id, kind: subagent ? 'subagent' : 'session', + label: thread?.agentNickname, + role: thread?.agentRole ?? (subagent ? 'worker' : 'primary'), + provider: thread?.provider, model: thread?.model, + }; +} + +/** The `source.fields` half of a ledger thread's discovered event. */ +function ledgerThreadSourceFields(thread, subagent) { + return { + project: thread?.project ? 'observed' : null, + provider: thread?.provider ? 'observed' : null, + model: thread?.model ? 'observed' : null, + status: thread?.status ? 'observed' : null, + hierarchy: subagent ? 'observed' : null, + workspace: thread?.workspace ? 'observed' : null, + }; +} + +/** One ledger thread → its `session.discovered` live event. */ +function buildLedgerThreadEvent(id, thread, ledger, context) { + const subagent = thread?.threadSource === 'subagent' || ledger.parents?.has(id); + return createLiveEvent({ + sessionId: id, parentSessionId: ledger.parents?.get(id), + host: 'codex', surface: 'native', project: thread?.project, + projectKey: thread?.projectKey, + observedAt: context.observedAt, + sourceTimestamp: thread?.updatedAt ?? thread?.recencyAt ?? thread?.createdAt, + workspace: thread?.workspace, + actor: ledgerThreadActor(id, thread, subagent), + action: 'session.discovered', + status: codexThreadStatus(thread), + source: { + adapter: 'codex-state', confidence: 'observed', + fields: ledgerThreadSourceFields(thread, subagent), + }, + attributes: { tokenUsage: thread?.tokensUsed }, + }); +} + +/** The `source.fields` half of a spawn-edge event. */ +function ledgerSpawnSourceFields(project, parent) { + return { + project: project && project !== 'unknown' ? 'observed' : null, + provider: parent?.provider ? 'observed' : null, + model: parent?.model ? 'observed' : null, + status: 'observed', hierarchy: 'observed', + }; +} + +/** One ledger parent/child edge → its `agent.spawned` live event. The ledger + * proves hierarchy, not that either thread is currently running — fresh + * rollout evidence owns lifecycle, so status is always 'unknown' here. */ +function buildLedgerSpawnEdgeEvent(childId, parentId, ledger, context) { + const child = ledger.threads instanceof Map ? ledger.threads.get(childId) : null; + const parent = ledger.threads instanceof Map ? ledger.threads.get(parentId) : null; + const project = parent?.project ?? context.project; + const projectKey = parent?.projectKey ?? context.projectKey; + return createLiveEvent({ + sessionId: parentId, host: 'codex', surface: 'native', + project, projectKey, observedAt: context.observedAt, + sourceTimestamp: parent?.updatedAt ?? parent?.recencyAt ?? parent?.createdAt, + actor: { + id: parentId, kind: 'session', label: parent?.agentNickname, + role: parent?.agentRole ?? 'primary', + provider: parent?.provider, model: parent?.model, + }, + action: 'agent.spawned', status: 'unknown', + target: { + id: childId, kind: 'subagent', + label: child?.agentNickname, role: child?.agentRole, + }, + source: { + adapter: 'codex-state', artifact: artifactName(context.artifact), + confidence: 'observed', + fields: ledgerSpawnSourceFields(project, parent), + }, + attributes: { tokenUsage: child?.tokensUsed }, + }); +} + /** Authoritative parent/child edges from readCodexState(). */ export function adaptCodexLedger(ledger, context = {}) { if (!(ledger?.threads instanceof Map)) return []; const out = []; - for (const [id, thread] of ledger.threads) { - const subagent = thread?.threadSource === 'subagent' || ledger.parents?.has(id); - out.push(createLiveEvent({ - sessionId: id, parentSessionId: ledger.parents?.get(id), - host: 'codex', surface: 'native', project: thread?.project, - projectKey: thread?.projectKey, - observedAt: context.observedAt, - sourceTimestamp: thread?.updatedAt ?? thread?.recencyAt ?? thread?.createdAt, - workspace: thread?.workspace, - actor: { - id, kind: subagent ? 'subagent' : 'session', - label: thread?.agentNickname, - role: thread?.agentRole ?? (subagent ? 'worker' : 'primary'), - provider: thread?.provider, model: thread?.model, - }, - action: 'session.discovered', - status: ['queued', 'running', 'completed', 'failed', 'cancelled'].includes(thread?.status) - ? thread.status : 'unknown', - source: { - adapter: 'codex-state', confidence: 'observed', - fields: { - project: thread?.project ? 'observed' : null, - provider: thread?.provider ? 'observed' : null, - model: thread?.model ? 'observed' : null, - status: thread?.status ? 'observed' : null, - hierarchy: subagent ? 'observed' : null, - workspace: thread?.workspace ? 'observed' : null, - }, - }, - attributes: { tokenUsage: thread?.tokensUsed }, - })); - } + for (const [id, thread] of ledger.threads) out.push(buildLedgerThreadEvent(id, thread, ledger, context)); if (!(ledger.parents instanceof Map)) return out; - for (const [childId, parentId] of ledger.parents) { - const child = ledger.threads instanceof Map ? ledger.threads.get(childId) : null; - const parent = ledger.threads instanceof Map ? ledger.threads.get(parentId) : null; - const project = parent?.project ?? context.project; - const projectKey = parent?.projectKey ?? context.projectKey; - out.push(createLiveEvent({ - sessionId: parentId, host: 'codex', surface: 'native', - project, projectKey, observedAt: context.observedAt, - sourceTimestamp: parent?.updatedAt ?? parent?.recencyAt ?? parent?.createdAt, - actor: { - id: parentId, kind: 'session', label: parent?.agentNickname, - role: parent?.agentRole ?? 'primary', - provider: parent?.provider, model: parent?.model, - }, - // The ledger proves hierarchy, not that either thread is currently - // running. Fresh rollout evidence owns lifecycle. - action: 'agent.spawned', status: 'unknown', - target: { - id: childId, kind: 'subagent', - label: child?.agentNickname, role: child?.agentRole, - }, - source: { - adapter: 'codex-state', artifact: artifactName(context.artifact), - confidence: 'observed', - fields: { - project: project && project !== 'unknown' ? 'observed' : null, - provider: parent?.provider ? 'observed' : null, - model: parent?.model ? 'observed' : null, - status: 'observed', hierarchy: 'observed', - }, - }, - attributes: { tokenUsage: child?.tokensUsed }, - })); - } + for (const [childId, parentId] of ledger.parents) out.push(buildLedgerSpawnEdgeEvent(childId, parentId, ledger, context)); return out; } diff --git a/src/lib/model-inventory/bindings.mjs b/src/lib/model-inventory/bindings.mjs index 1f06955..f147bbc 100644 --- a/src/lib/model-inventory/bindings.mjs +++ b/src/lib/model-inventory/bindings.mjs @@ -18,13 +18,26 @@ function record({ consumer, source, host = null, provider = null, modelRef = nul }; } -export function collectModelBindings({ - config = {}, aqeConfig, rufloConfig, -} = /** @type {any} */ ({})) { +function escalationBindings(route, activity, diagnostics) { const bindings = []; - const diagnostics = []; - const routes = config?.routing?.routes; + for (const [index, rung] of (Array.isArray(route.escalation) ? route.escalation : []).entries()) { + if (!plain(rung) || !bounded(rung.host)) { + diagnostics.push({ code: 'invalid-escalation', activity, index }); + continue; + } + bindings.push(record({ + consumer: `route:${activity}:escalation:${index}`, source: 'kit.json', host: rung.host, + modelRef: bounded(rung.model), activity, provenance: route.provenance, index, + variant: { reasoningEffort: bounded(rung.reasoningEffort) }, + })); + } + return bindings; +} + +/** Routes and their escalation rungs from `kit.json`'s `routing.routes` table. */ +function routeBindings(routes, diagnostics) { if (routes !== undefined && !plain(routes)) diagnostics.push({ code: 'invalid-routing-schema' }); + const bindings = []; for (const [activity, route] of Object.entries(plain(routes) ? routes : {})) { if (!plain(route) || !bounded(route.host)) { diagnostics.push({ code: 'invalid-route', activity }); @@ -34,20 +47,16 @@ export function collectModelBindings({ consumer: `route:${activity}`, source: 'kit.json', host: route.host, modelRef: bounded(route.model), activity, provenance: route.provenance, variant: { reasoningEffort: bounded(route.reasoningEffort) }, })); - for (const [index, rung] of (Array.isArray(route.escalation) ? route.escalation : []).entries()) { - if (!plain(rung) || !bounded(rung.host)) { - diagnostics.push({ code: 'invalid-escalation', activity, index }); - continue; - } - bindings.push(record({ - consumer: `route:${activity}:escalation:${index}`, source: 'kit.json', host: rung.host, - modelRef: bounded(rung.model), activity, provenance: route.provenance, index, - variant: { reasoningEffort: bounded(rung.reasoningEffort) }, - })); - } + bindings.push(...escalationBindings(route, activity, diagnostics)); } - for (const [index, binding] of (Array.isArray(config?.integrations?.bindings) - ? config.integrations.bindings : []).entries()) { + return bindings; +} + +/** Ad-hoc host/provider bindings from `kit.json`'s `integrations.bindings` array. */ +function integrationBindings(config, diagnostics) { + const bindings = []; + const entries = Array.isArray(config?.integrations?.bindings) ? config.integrations.bindings : []; + for (const [index, binding] of entries.entries()) { if (!plain(binding) || !bounded(binding.host)) { diagnostics.push({ code: 'invalid-integration-binding', index }); continue; @@ -58,33 +67,68 @@ export function collectModelBindings({ variant: { reasoningEffort: bounded(binding.reasoningEffort) }, })); } - if (plain(aqeConfig)) { - if (bounded(aqeConfig.defaultProvider)) bindings.push(record({ - consumer: 'aqe:default', source: '.agentic-qe/llm-config.json', provider: aqeConfig.defaultProvider, + return bindings; +} + +function aqeFallbackBindings(aqeConfig) { + const entries = Array.isArray(aqeConfig.fallbackChain) ? aqeConfig.fallbackChain + : Array.isArray(aqeConfig.fallbackChain?.entries) ? aqeConfig.fallbackChain.entries : []; + const bindings = []; + for (const [index, item] of entries.entries()) { + const entry = typeof item === 'string' ? { provider: item } : item; + if (plain(entry) && bounded(entry.provider)) bindings.push(record({ + consumer: `aqe:fallback:${index}`, source: '.agentic-qe/llm-config.json', provider: entry.provider, + modelRef: bounded(entry.model) ?? (Array.isArray(entry.models) ? bounded(entry.models[0]) : null), index, })); - const fallbackEntries = Array.isArray(aqeConfig.fallbackChain) ? aqeConfig.fallbackChain - : Array.isArray(aqeConfig.fallbackChain?.entries) ? aqeConfig.fallbackChain.entries : []; - for (const [index, item] of fallbackEntries.entries()) { - const entry = typeof item === 'string' ? { provider: item } : item; - if (plain(entry) && bounded(entry.provider)) bindings.push(record({ - consumer: `aqe:fallback:${index}`, source: '.agentic-qe/llm-config.json', provider: entry.provider, - modelRef: bounded(entry.model) ?? (Array.isArray(entry.models) ? bounded(entry.models[0]) : null), index, - })); - } - for (const [agent, entry] of Object.entries(plain(aqeConfig.agentOverrides) ? aqeConfig.agentOverrides : {})) { - if (plain(entry) && bounded(entry.provider)) bindings.push(record({ - consumer: `aqe:agent:${agent}`, source: '.agentic-qe/llm-config.json', provider: entry.provider, - modelRef: bounded(entry.model), activity: agent, - })); - } } - const rufloCandidates = Array.isArray(rufloConfig?.candidates) ? rufloConfig.candidates + return bindings; +} + +function aqeAgentOverrideBindings(aqeConfig) { + const overrides = plain(aqeConfig.agentOverrides) ? aqeConfig.agentOverrides : {}; + const bindings = []; + for (const [agent, entry] of Object.entries(overrides)) { + if (plain(entry) && bounded(entry.provider)) bindings.push(record({ + consumer: `aqe:agent:${agent}`, source: '.agentic-qe/llm-config.json', provider: entry.provider, + modelRef: bounded(entry.model), activity: agent, + })); + } + return bindings; +} + +/** Agentic-QE's own resolved config (`.agentic-qe/llm-config.json`): default provider, + * ordered fallback chain, and per-agent overrides — each an independently sourced consumer. */ +function aqeBindings(aqeConfig) { + if (!plain(aqeConfig)) return []; + const defaultBinding = bounded(aqeConfig.defaultProvider) ? [record({ + consumer: 'aqe:default', source: '.agentic-qe/llm-config.json', provider: aqeConfig.defaultProvider, + })] : []; + return [...defaultBinding, ...aqeFallbackBindings(aqeConfig), ...aqeAgentOverrideBindings(aqeConfig)]; +} + +/** Ruflo's candidate provider/model list, wherever it currently lives in its config shape. */ +function rufloBindings(rufloConfig) { + const candidates = Array.isArray(rufloConfig?.candidates) ? rufloConfig.candidates : Array.isArray(rufloConfig?.providers?.models) ? rufloConfig.providers.models : []; - for (const [index, candidate] of rufloCandidates.entries()) { + const bindings = []; + for (const [index, candidate] of candidates.entries()) { if (plain(candidate) && (bounded(candidate.provider) || bounded(candidate.model))) bindings.push(record({ consumer: `ruflo:candidate:${index}`, source: 'ruflo', provider: bounded(candidate.provider) ?? bounded(candidate.id), modelRef: bounded(candidate.model) ?? bounded(candidate.id), index, })); } + return bindings; +} + +export function collectModelBindings({ + config = {}, aqeConfig, rufloConfig, +} = /** @type {any} */ ({})) { + const diagnostics = []; + const bindings = [ + ...routeBindings(config?.routing?.routes, diagnostics), + ...integrationBindings(config, diagnostics), + ...aqeBindings(aqeConfig), + ...rufloBindings(rufloConfig), + ]; return { status: diagnostics.length ? 'partial' : 'complete', bindings, diagnostics }; } diff --git a/src/lib/model-inventory/dashboard-query.mjs b/src/lib/model-inventory/dashboard-query.mjs index 7d4fe37..c99cdd8 100644 --- a/src/lib/model-inventory/dashboard-query.mjs +++ b/src/lib/model-inventory/dashboard-query.mjs @@ -53,43 +53,52 @@ function dimensionFilter(value) { return value; } -function parseQuery(raw) { - const query = raw instanceof URLSearchParams ? raw : new URLSearchParams(raw ?? ''); - for (const key of query.keys()) if (!QUERY_KEYS.has(key)) throw invalidQuery(); - const view = one(query, 'view') ?? 'full'; - if (!['full', 'summary', 'inventory'].includes(view)) throw invalidQuery(); - if (view !== 'inventory') { - for (const key of query.keys()) if (!['view', 'token', 'days'].includes(key)) throw invalidQuery(); - const rawDays = one(query, 'days'); - if (view !== 'summary' && rawDays != null) throw invalidQuery(); - if (rawDays != null && (!/^\d+$/.test(rawDays) || Number(rawDays) < 1 || Number(rawDays) > 365)) { - throw invalidQuery(); - } - return { view, days: rawDays == null ? null : Number(rawDays) }; +function parseNonInventoryQuery(query, view) { + for (const key of query.keys()) if (!['view', 'token', 'days'].includes(key)) throw invalidQuery(); + const rawDays = one(query, 'days'); + if (view !== 'summary' && rawDays != null) throw invalidQuery(); + if (rawDays != null && (!/^\d+$/.test(rawDays) || Number(rawDays) < 1 || Number(rawDays) > 365)) { + throw invalidQuery(); } - if (one(query, 'days') != null) throw invalidQuery(); - const integer = (name, fallback) => { - const rawValue = one(query, name); - if (rawValue == null || rawValue === '') return fallback; - if (!/^\d+$/.test(rawValue)) throw invalidQuery(); - const value = Number(rawValue); - if (!Number.isSafeInteger(value) || value > 10_000_000) throw invalidQuery(); - return value; - }; + return { view, days: rawDays == null ? null : Number(rawDays) }; +} + +function queryInteger(query, name, fallback) { + const rawValue = one(query, name); + if (rawValue == null || rawValue === '') return fallback; + if (!/^\d+$/.test(rawValue)) throw invalidQuery(); + const value = Number(rawValue); + if (!Number.isSafeInteger(value) || value > 10_000_000) throw invalidQuery(); + return value; +} + +function inventorySortSelection(query) { const sort = one(query, 'sort') ?? 'displayName'; const direction = one(query, 'direction') ?? 'asc'; const relevance = one(query, 'relevance') ?? 'relevant'; if (!SORTS.has(sort) || !['asc', 'desc'].includes(direction) || !['relevant', 'catalog', 'all'].includes(relevance)) throw invalidQuery(); + return { sort, direction, relevance }; +} + +function inventoryEvidenceFilter(query) { const evidenceField = one(query, 'evidenceField'); const evidenceValue = dimensionFilter(one(query, 'evidenceValue')); if ((evidenceField == null) !== (evidenceValue == null) || (evidenceField != null && !DIMENSIONS.includes(evidenceField))) throw invalidQuery(); + return { evidenceField, evidenceValue }; +} + +function parseInventoryQuery(query, view) { + if (one(query, 'days') != null) throw invalidQuery(); + const { sort, direction, relevance } = inventorySortSelection(query); + const { evidenceField, evidenceValue } = inventoryEvidenceFilter(query); const dimensions = Object.fromEntries(DIMENSIONS.map((name) => ( [name, dimensionFilter(one(query, name))] ))); return { - view, offset: integer('offset', 0), limit: Math.min(100, Math.max(1, integer('limit', 50))), + view, offset: queryInteger(query, 'offset', 0), + limit: Math.min(100, Math.max(1, queryInteger(query, 'limit', 50))), sort, direction, relevance, search: queryText(query, 'search'), host: queryText(query, 'host'), provider: queryText(query, 'provider'), publisher: queryText(query, 'publisher'), lifecycle: queryText(query, 'lifecycle', 64), @@ -98,6 +107,14 @@ function parseQuery(raw) { }; } +function parseQuery(raw) { + const query = raw instanceof URLSearchParams ? raw : new URLSearchParams(raw ?? ''); + for (const key of query.keys()) if (!QUERY_KEYS.has(key)) throw invalidQuery(); + const view = one(query, 'view') ?? 'full'; + if (!['full', 'summary', 'inventory'].includes(view)) throw invalidQuery(); + return view === 'inventory' ? parseInventoryQuery(query, view) : parseNonInventoryQuery(query, view); +} + function isRelevant(model) { return ['configured', 'effective', 'observed'].some((name) => model.dimensions[name]?.value === true) || (model.dimensions.discoverable?.value === true diff --git a/src/lib/model-inventory/discovery/codex.mjs b/src/lib/model-inventory/discovery/codex.mjs index 4a9a98e..3bd6e2c 100644 --- a/src/lib/model-inventory/discovery/codex.mjs +++ b/src/lib/model-inventory/discovery/codex.mjs @@ -33,45 +33,41 @@ function parseConfig(raw) { return result; } -export function discoverCodex({ - cacheRaw, configRaw, capturedAt, scope = {}, scopeKey, now = Date.now(), maxAgeMs = 7 * 86_400_000, -} = /** @type {any} */ ({})) { - let cache; - let config; - try { cache = parseCache(cacheRaw); config = parseConfig(configRaw); } catch (error) { - const source = sourceRecord({ id: 'codex-cache', owner: 'codex', scope, scopeKey, capturedAt, complete: false, status: 'unsupported-schema', schema: 'codex-model-cache-v1', diagnostics: ['unsupported-schema'] }); - return { status: 'unsupported-schema', source, models: [], diagnostics: [diagnostic('unsupported-schema', error.message)] }; - } - if (!Array.isArray(cache.models)) { - const source = sourceRecord({ id: 'codex-cache', owner: 'codex', scope, scopeKey, capturedAt, complete: false, status: 'unsupported-schema', schema: 'codex-model-cache-v1', diagnostics: ['unsupported-schema'] }); - return { status: 'unsupported-schema', source, models: [], diagnostics: [diagnostic('unsupported-schema', 'models must be an array')] }; - } - const fetchedAt = Date.parse(cache.fetched_at ?? cache.fetchedAt ?? ''); - const stale = Number.isFinite(fetchedAt) && now - fetchedAt > maxAgeMs; - let complete = cache.models.length <= MAX_MODELS; +function codexUnsupportedSchemaResult(scopeCtx, message) { const source = sourceRecord({ - id: 'codex-cache', owner: 'codex', scope, scopeKey, capturedAt: capturedAt ?? (Number.isFinite(fetchedAt) ? new Date(fetchedAt).toISOString() : undefined), - complete, schema: `codex-model-cache-v1${bounded(cache.client_version, 32) ? `@${cache.client_version}` : ''}`, - freshness: stale ? 'stale' : 'current', + ...scopeCtx, id: 'codex-cache', owner: 'codex', complete: false, + status: 'unsupported-schema', schema: 'codex-model-cache-v1', diagnostics: ['unsupported-schema'], }); - const diagnostics = []; - const models = []; - for (const [index, raw] of cache.models.slice(0, MAX_MODELS).entries()) { - const modelId = bounded(raw?.slug ?? raw?.id); - const visibility = raw?.visibility ?? 'list'; - if (!modelId || !VISIBILITY.has(visibility)) { - complete = false; - diagnostics.push(diagnostic('invalid-model-schema', `models[${index}] is invalid`)); - continue; - } - const reasoningEfforts = (Array.isArray(raw.supported_reasoning_levels) ? raw.supported_reasoning_levels : []) - .map((entry) => typeof entry === 'string' ? entry : entry?.effort) - .filter((effort) => REASONING.has(effort)); - models.push(modelRecord({ + return { status: 'unsupported-schema', source, models: [], diagnostics: [diagnostic('unsupported-schema', message)] }; +} + +function codexReasoningEfforts(raw) { + const levels = Array.isArray(raw.supported_reasoning_levels) ? raw.supported_reasoning_levels : []; + const efforts = levels.map((entry) => (typeof entry === 'string' ? entry : entry?.effort)) + .filter((effort) => REASONING.has(effort)); + return [...new Set(efforts)]; +} + +function codexModelStates(config, modelId, visibility) { + return { + configured: config.model === modelId ? true : 'unknown', + effective: config.model === modelId ? true : 'unknown', + discoverable: visibility === 'list' || visibility === 'visible', entitled: 'unknown', + }; +} + +function codexModelFromCacheEntry(raw, index, config, source) { + const modelId = bounded(raw?.slug ?? raw?.id); + const visibility = raw?.visibility ?? 'list'; + if (!modelId || !VISIBILITY.has(visibility)) { + return { diagnostic: diagnostic('invalid-model-schema', `models[${index}] is invalid`) }; + } + return { + model: modelRecord({ host: 'codex', provider: null, modelId, scopeId: source.scopeId, displayName: bounded(raw.display_name) ?? modelId, source, variant: { - reasoningEfforts: [...new Set(reasoningEfforts)], + reasoningEfforts: codexReasoningEfforts(raw), contextWindow: Number.isInteger(raw.context_window) && raw.context_window > 0 ? raw.context_window : null, }, // `upgrade` is a local client hint, not a public retirement notice. It @@ -79,25 +75,68 @@ export function discoverCodex({ // route to be treated as retired. A warning needs an explicit first-party // notice URL captured by a source that can establish that fact. lifecycle: { state: visibility === 'hide' || visibility === 'hidden' ? 'hidden' : 'active', replacement: null }, - states: { - configured: config.model === modelId ? true : 'unknown', - effective: config.model === modelId ? true : 'unknown', - discoverable: visibility === 'list' || visibility === 'visible', entitled: 'unknown', - }, - })); + states: codexModelStates(config, modelId, visibility), + }), + }; +} + +function codexModelsFromCache(cacheModels, config, source) { + const models = []; + const diagnostics = []; + let complete = true; + for (const [index, raw] of cacheModels.slice(0, MAX_MODELS).entries()) { + const { model, diagnostic: rowDiagnostic } = codexModelFromCacheEntry(raw, index, config, source); + if (model) models.push(model); + if (rowDiagnostic) { complete = false; diagnostics.push(rowDiagnostic); } + } + return { models, diagnostics, complete }; +} + +function codexOverallStatus(complete, stale) { + return complete ? (stale ? 'stale' : 'complete') : 'partial'; +} + +function codexConfiguredFallbackModel(config, source) { + if (!bounded(config.model)) return null; + return modelRecord({ + host: 'codex', provider: bounded(config.model_provider), modelId: config.model, + scopeId: source.scopeId, source, + variant: { reasoningEffort: REASONING.has(config.model_reasoning_effort) ? config.model_reasoning_effort : null }, + states: { configured: true, effective: true, discoverable: 'unknown', entitled: 'unknown' }, + }); +} + +export function discoverCodex({ + cacheRaw, configRaw, capturedAt, scope = {}, scopeKey, now = Date.now(), maxAgeMs = 7 * 86_400_000, +} = /** @type {any} */ ({})) { + const scopeCtx = { scope, scopeKey, capturedAt }; + let cache; + let config; + try { cache = parseCache(cacheRaw); config = parseConfig(configRaw); } catch (error) { + return codexUnsupportedSchemaResult(scopeCtx, error.message); } - if (bounded(config.model) && !models.some((model) => model.identity.modelId === config.model)) { - models.push(modelRecord({ - host: 'codex', provider: bounded(config.model_provider), modelId: config.model, - scopeId: source.scopeId, source, - variant: { reasoningEffort: REASONING.has(config.model_reasoning_effort) ? config.model_reasoning_effort : null }, - states: { configured: true, effective: true, discoverable: 'unknown', entitled: 'unknown' }, - })); + if (!Array.isArray(cache.models)) return codexUnsupportedSchemaResult(scopeCtx, 'models must be an array'); + const fetchedAt = Date.parse(cache.fetched_at ?? cache.fetchedAt ?? ''); + const stale = Number.isFinite(fetchedAt) && now - fetchedAt > maxAgeMs; + const source = sourceRecord({ + id: 'codex-cache', owner: 'codex', scope, scopeKey, + capturedAt: capturedAt ?? (Number.isFinite(fetchedAt) ? new Date(fetchedAt).toISOString() : undefined), + complete: cache.models.length <= MAX_MODELS, + schema: `codex-model-cache-v1${bounded(cache.client_version, 32) ? `@${cache.client_version}` : ''}`, + freshness: stale ? 'stale' : 'current', + }); + const { models, diagnostics, complete: rowsComplete } = codexModelsFromCache(cache.models, config, source); + const complete = cache.models.length <= MAX_MODELS && rowsComplete; + const fallback = codexConfiguredFallbackModel(config, source); + if (fallback && !models.some((model) => model.identity.modelId === fallback.identity.modelId)) { + models.push(fallback); } if (cache.models.length > MAX_MODELS) diagnostics.push(diagnostic('model-cap', `models exceeds ${MAX_MODELS}`)); source.complete = complete; - source.status = complete ? (stale ? 'stale' : 'complete') : 'partial'; + source.status = codexOverallStatus(complete, stale); source.diagnostics = diagnostics.map(({ code }) => code); for (const model of models) model.evidence[0].completeness = complete ? 'complete' : 'partial'; - return { status: complete ? (stale ? 'stale' : 'complete') : 'partial', source, models, diagnostics }; + return { + status: codexOverallStatus(complete, stale), source, models, diagnostics, + }; } diff --git a/src/lib/model-inventory/discovery/index.mjs b/src/lib/model-inventory/discovery/index.mjs index 05411ab..d236938 100644 --- a/src/lib/model-inventory/discovery/index.mjs +++ b/src/lib/model-inventory/discovery/index.mjs @@ -35,19 +35,23 @@ export function diagnostic(code, message) { return { code, message: String(message ?? code).slice(0, 240) }; } -export function modelRecord({ host, provider = null, modelId, scopeId, displayName = null, aliases = [], - variant = {}, lifecycle = /** @type {any} */ ({ state: 'unknown', replacement: null }), - states = /** @type {any} */ ({}), capabilities = {}, pricing = null, digest = null, source }) { - const safeHost = host || 'unknown'; - const evidenceClass = source.evidenceClass ?? (source.id === 'usage-index' ? 'observed' +function resolveEvidenceClass(source) { + return source.evidenceClass ?? (source.id === 'usage-index' ? 'observed' : source.id.includes('config') ? 'configured' : 'catalog'); - const normalizedStates = { +} + +function normalizedModelStates(states) { + return { configured: 'unknown', effective: 'unknown', observed: 'unknown', discoverable: 'unknown', entitled: 'unknown', policyAllowed: 'unknown', routable: 'unknown', recommended: 'unknown', ...states, }; - const evidence = []; - const evidenceFor = (field, klass = evidenceClass) => { +} + +function makeEvidenceFor({ + source, safeHost, provider, modelId, scopeId, evidenceClass, evidence, +}) { + return (field, klass = evidenceClass) => { const id = `evidence:${createHash('sha256').update([ source.id, safeHost, provider ?? '', modelId, scopeId, field, source.capturedAt, ].join('\n')).digest('hex').slice(0, 24)}`; @@ -59,27 +63,44 @@ export function modelRecord({ host, provider = null, modelId, scopeId, displayNa }); return id; }; - const dimensions = Object.fromEntries(Object.entries(normalizedStates).map(([name, value]) => [name, { +} + +function stateDimensions(normalizedStates, evidenceFor) { + return Object.fromEntries(Object.entries(normalizedStates).map(([name, value]) => [name, { value: value === 'unknown' ? null : Boolean(value), evidenceRefs: value === 'unknown' ? [] : [evidenceFor(`dimensions.${name}`)], }])); - const replacement = lifecycle?.replacement && typeof lifecycle.replacement === 'object' +} + +function resolveReplacement(lifecycle) { + return lifecycle?.replacement && typeof lifecycle.replacement === 'object' ? lifecycle.replacement.modelId : lifecycle?.replacement ?? null; - const normalizedAliases = aliases.map((alias) => ({ +} + +function normalizedModelAliases(aliases, evidenceFor) { + return aliases.map((alias) => ({ name: alias.name, resolvesTo: alias.resolvesTo ?? null, observedAt: alias.observedAt ?? null, evidenceRefs: [evidenceFor(`aliases.${alias.name}`)], })); +} + +function lifecycleEvidenceRefs(lifecycle, replacement, evidenceClass, evidenceFor) { const lifecycleKnown = lifecycle?.state && lifecycle.state !== 'unknown'; - const lifecycleEvidence = lifecycleKnown || replacement + return lifecycleKnown || replacement ? [evidenceFor('lifecycle', replacement ? 'first-party' : evidenceClass)] : []; - if (digest) evidenceFor('key.digest'); - for (const field of Object.keys(variant)) evidenceFor(`variant.${field}`); - for (const field of Object.keys(capabilities)) evidenceFor(`capabilities.${field}`); - const normalizedPricing = pricing == null ? null : { +} + +function normalizedModelPricing(pricing, evidenceFor) { + return pricing == null ? null : { ...pricing, evidenceRefs: pricing.evidenceRefs?.length ? pricing.evidenceRefs : [evidenceFor('pricing')], }; - const edges = [ +} + +function modelEdges({ + normalizedAliases, replacement, modelId, scopeId, evidenceClass, lifecycleEvidence, +}) { + return [ ...normalizedAliases.filter(({ resolvesTo }) => resolvesTo).map((alias) => ({ kind: 'resolves-to', from: alias.name, to: alias.resolvesTo, provenance: evidenceClass === 'first-party' ? 'first-party' : 'configured', @@ -90,6 +111,35 @@ export function modelRecord({ host, provider = null, modelId, scopeId, displayNa provenance: 'first-party', scopeFingerprint: scopeId, evidenceRefs: lifecycleEvidence, }] : []), ]; +} + +export function modelRecord({ + host, provider = null, modelId, scopeId, displayName = null, aliases = [], + variant = {}, lifecycle = /** @type {any} */ ({ state: 'unknown', replacement: null }), + states = /** @type {any} */ ({}), capabilities = {}, pricing = null, digest = null, source, +}) { + const safeHost = host || 'unknown'; + const evidenceClass = resolveEvidenceClass(source); + const normalizedStates = normalizedModelStates(states); + const evidence = []; + const evidenceFor = makeEvidenceFor({ + source, safeHost, provider, modelId, scopeId, evidenceClass, evidence, + }); + // Call order below is load-bearing: dimensions, then aliases, then lifecycle, + // then digest, then variant fields, then capability fields, then pricing — + // each evidenceFor() call appends to `evidence` in this exact sequence, and + // at least one caller (Codex's discoverCodex) indexes evidence[0] positionally. + const dimensions = stateDimensions(normalizedStates, evidenceFor); + const replacement = resolveReplacement(lifecycle); + const normalizedAliases = normalizedModelAliases(aliases, evidenceFor); + const lifecycleEvidence = lifecycleEvidenceRefs(lifecycle, replacement, evidenceClass, evidenceFor); + if (digest) evidenceFor('key.digest'); + for (const field of Object.keys(variant)) evidenceFor(`variant.${field}`); + for (const field of Object.keys(capabilities)) evidenceFor(`capabilities.${field}`); + const normalizedPricing = normalizedModelPricing(pricing, evidenceFor); + const edges = modelEdges({ + normalizedAliases, replacement, modelId, scopeId, evidenceClass, lifecycleEvidence, + }); return { key: { host: safeHost, provider, modelId, scopeId, digest }, identity: { host: safeHost, provider, modelId, scopeId, digest }, @@ -97,8 +147,10 @@ export function modelRecord({ host, provider = null, modelId, scopeId, displayNa aliases: normalizedAliases, visibility: states.discoverable === true ? 'visible' : states.discoverable === false ? 'hidden' : 'unknown', variant, - lifecycle: { state: lifecycle?.state ?? 'unknown', replacement, notice: lifecycle?.notice ?? null, - effectiveAt: lifecycle?.effectiveAt ?? null, evidenceRefs: lifecycleEvidence }, + lifecycle: { + state: lifecycle?.state ?? 'unknown', replacement, notice: lifecycle?.notice ?? null, + effectiveAt: lifecycle?.effectiveAt ?? null, evidenceRefs: lifecycleEvidence, + }, capabilities, pricing: normalizedPricing, edges, dimensions, states: normalizedStates, evidence, diff --git a/src/lib/model-inventory/discovery/ollama.mjs b/src/lib/model-inventory/discovery/ollama.mjs index 66fa72b..2e89799 100644 --- a/src/lib/model-inventory/discovery/ollama.mjs +++ b/src/lib/model-inventory/discovery/ollama.mjs @@ -53,6 +53,128 @@ function runtimeByName(value) { })); } +function parseOllamaTags(tagsRaw) { + try { return { tags: json(tagsRaw, 'tags') }; } catch (error) { return { error }; } +} + +function parseOllamaRuntime(psRaw) { + try { return { runtime: runtimeByName(json(psRaw, 'ps')) }; } + catch (error) { return { runtime: new Map(), diagnostic: diagnostic(error.message, 'Ollama runtime response is invalid') }; } +} + +function resolveOllamaShow(showByModel, modelId, index) { + const shown = showByModel instanceof Map ? showByModel.get(modelId) : showByModel?.[modelId]; + try { + return { shown: shown == null ? {} : json(shown, 'show') }; + } catch (error) { + return { shown: {}, diagnostic: diagnostic(error.message, `Ollama show response for row ${index + 1} is invalid`) }; + } +} + +function ollamaFamilies(details) { + return Array.isArray(details.families) + ? details.families.map((item) => text(item, 64)).filter(Boolean).slice(0, 16) : []; +} + +function ollamaAdvertisedCapabilities(shown) { + return Array.isArray(shown.capabilities) + ? [...new Set(shown.capabilities.map((item) => text(item, 64)).filter((item) => CAPABILITIES.has(item)))] : []; +} + +/** `value` when it is not null/undefined (0 is a real, present size/window — keep it). */ +function whenPresent(key, value) { + return value != null ? { [key]: value } : {}; +} + +/** `value` when it is truthy (an empty string/array means "nothing to report"). */ +function whenTruthy(key, value) { + return value ? { [key]: value } : {}; +} + +function ollamaVariant({ + digest, row, shown, details, families, advertised, context, loaded, +}) { + const showDetails = shown.details && typeof shown.details === 'object' ? shown.details : {}; + return { + digest, + ...whenPresent('sizeBytes', positive(row.size)), + ...whenTruthy('modifiedAt', iso(row.modified_at)), + ...whenTruthy('format', text(details.format, 64)), + ...whenTruthy('family', text(details.family ?? showDetails.family, 64)), + ...whenTruthy('families', families.length ? families : null), + ...whenTruthy('parameterSize', text(details.parameter_size ?? showDetails.parameter_size, 64)), + ...whenTruthy('quantizationLevel', text(details.quantization_level ?? showDetails.quantization_level, 64)), + loaded: Boolean(loaded), + ...whenPresent('memoryBytes', positive(loaded?.size)), + ...whenPresent('vramBytes', positive(loaded?.size_vram)), + ...whenTruthy('expiresAt', iso(loaded?.expires_at)), + ...whenPresent('contextWindow', context), + ...whenTruthy('licenseSummary', licenseSummary(shown.license)), + ...whenTruthy('advertisedCapabilities', advertised.length ? advertised : null), + }; +} + +function ollamaCapabilities(advertised, context) { + return { + ...(context != null ? { contextLimit: context } : {}), + ...(advertised.includes('tools') ? { toolcall: true } : {}), + ...(advertised.includes('thinking') ? { reasoning: true } : {}), + ...(advertised.includes('vision') ? { input: { text: true, image: true } } : {}), + ...(advertised.includes('embedding') ? { embedding: true } : {}), + }; +} + +/** Evidence for runtime-observed variant fields (loaded/memory/vram/expiry/context) is + * distinct from the catalog evidence class the rest of the record carries. */ +function markOllamaRuntimeEvidence(record) { + for (const evidence of record.evidence) { + if (/^variant\.(?:loaded|memoryBytes|vramBytes|expiresAt|contextWindow)$/.test(evidence.field)) evidence.class = 'runtime'; + } +} + +/** Build one model record from a /api/tags row, joining bounded /api/show detail and + * /api/ps runtime state. Returns `{ diagnostic }` only for an invalid or unreadable row. */ +function ollamaModelFromRow({ + row, index, runtime, showByModel, source, +}) { + const modelId = text(row?.name ?? row?.model); + const digest = text(row?.digest); + if (!modelId || !TOKEN.test(modelId) || !digest || !DIGEST.test(digest)) { + return { diagnostic: diagnostic('invalid-model-row', `Ollama tags row ${index + 1} is invalid`) }; + } + const loaded = runtime.get(modelId) ?? null; + const { shown, diagnostic: showDiagnostic } = resolveOllamaShow(showByModel, modelId, index); + const details = row.details && typeof row.details === 'object' ? row.details : {}; + const families = ollamaFamilies(details); + const advertised = ollamaAdvertisedCapabilities(shown); + const context = positive(loaded?.context_length) ?? contextWindow(shown.model_info); + const variant = ollamaVariant({ + digest, row, shown, details, families, advertised, context, loaded, + }); + const record = modelRecord({ + host: 'ollama', provider: 'ollama', modelId, scopeId: source.scopeId, source, + displayName: modelId, digest, variant, capabilities: ollamaCapabilities(advertised, context), + pricing: { basis: 'local-compute', input: 0, output: 0, currency: 'USD', effectiveAt: null }, + states: { discoverable: true }, + }); + markOllamaRuntimeEvidence(record); + return { record, diagnostic: showDiagnostic }; +} + +function ollamaModelsFromRows(rows, { runtime, showByModel, source }) { + const models = []; + const diagnostics = []; + for (const [index, row] of rows.entries()) { + if (models.length >= MAX_MODELS) break; + const { record, diagnostic: rowDiagnostic } = ollamaModelFromRow({ + row, index, runtime, showByModel, source, + }); + if (record) models.push(record); + if (rowDiagnostic) diagnostics.push(rowDiagnostic); + } + return { models, diagnostics }; +} + /** Normalize bounded /api/tags, /api/show and /api/ps responses. */ export function discoverOllamaApi({ tagsRaw, psRaw = { models: [] }, showByModel = {}, capturedAt, scope = {}, scopeKey, @@ -61,76 +183,19 @@ export function discoverOllamaApi({ id: 'ollama-catalog', owner: 'ollama', ownerType: 'provider', transport: 'http', network: 'local', scope, scopeKey, capturedAt, complete: true, schema: 'ollama-api-v1', }); - const diagnostics = []; - let tags; - let runtime; - try { tags = json(tagsRaw, 'tags'); } catch (error) { - source.complete = false; source.status = 'unsupported'; source.diagnostics = [error.message]; - return { status: 'unsupported', source, models: [], diagnostics: [diagnostic(error.message, 'Ollama tags response is invalid')] }; - } - try { runtime = runtimeByName(json(psRaw, 'ps')); } catch (error) { - runtime = new Map(); diagnostics.push(diagnostic(error.message, 'Ollama runtime response is invalid')); + const { tags, error: tagsError } = parseOllamaTags(tagsRaw); + if (tagsError) { + source.complete = false; source.status = 'unsupported'; source.diagnostics = [tagsError.message]; + return { status: 'unsupported', source, models: [], diagnostics: [diagnostic(tagsError.message, 'Ollama tags response is invalid')] }; } const rows = Array.isArray(tags?.models) ? tags.models : null; if (!rows) { source.complete = false; source.status = 'unsupported-schema'; source.diagnostics = ['tags-schema-unsupported']; return { status: 'unsupported', source, models: [], diagnostics: [diagnostic('tags-schema-unsupported', 'Ollama tags response has no models array')] }; } - const models = []; - for (const [index, row] of rows.entries()) { - if (models.length >= MAX_MODELS) break; - const modelId = text(row?.name ?? row?.model); - const digest = text(row?.digest); - if (!modelId || !TOKEN.test(modelId) || !digest || !DIGEST.test(digest)) { - diagnostics.push(diagnostic('invalid-model-row', `Ollama tags row ${index + 1} is invalid`)); - continue; - } - const loaded = runtime.get(modelId) ?? null; - let shown = showByModel instanceof Map ? showByModel.get(modelId) : showByModel?.[modelId]; - try { shown = shown == null ? {} : json(shown, 'show'); } catch (error) { - shown = {}; diagnostics.push(diagnostic(error.message, `Ollama show response for row ${index + 1} is invalid`)); - } - const details = row.details && typeof row.details === 'object' ? row.details : {}; - const showDetails = shown.details && typeof shown.details === 'object' ? shown.details : {}; - const families = Array.isArray(details.families) ? details.families.map((item) => text(item, 64)).filter(Boolean).slice(0, 16) : []; - const advertised = Array.isArray(shown.capabilities) - ? [...new Set(shown.capabilities.map((item) => text(item, 64)).filter((item) => CAPABILITIES.has(item)))] : []; - const context = positive(loaded?.context_length) ?? contextWindow(shown.model_info); - const variant = { - digest, - ...(positive(row.size) != null ? { sizeBytes: row.size } : {}), - ...(iso(row.modified_at) ? { modifiedAt: iso(row.modified_at) } : {}), - ...(text(details.format, 64) ? { format: text(details.format, 64) } : {}), - ...(text(details.family ?? showDetails.family, 64) ? { family: text(details.family ?? showDetails.family, 64) } : {}), - ...(families.length ? { families } : {}), - ...(text(details.parameter_size ?? showDetails.parameter_size, 64) ? { parameterSize: text(details.parameter_size ?? showDetails.parameter_size, 64) } : {}), - ...(text(details.quantization_level ?? showDetails.quantization_level, 64) ? { quantizationLevel: text(details.quantization_level ?? showDetails.quantization_level, 64) } : {}), - loaded: Boolean(loaded), - ...(positive(loaded?.size) != null ? { memoryBytes: loaded.size } : {}), - ...(positive(loaded?.size_vram) != null ? { vramBytes: loaded.size_vram } : {}), - ...(iso(loaded?.expires_at) ? { expiresAt: iso(loaded.expires_at) } : {}), - ...(context != null ? { contextWindow: context } : {}), - ...(licenseSummary(shown.license) ? { licenseSummary: licenseSummary(shown.license) } : {}), - ...(advertised.length ? { advertisedCapabilities: advertised } : {}), - }; - const capabilities = { - ...(context != null ? { contextLimit: context } : {}), - ...(advertised.includes('tools') ? { toolcall: true } : {}), - ...(advertised.includes('thinking') ? { reasoning: true } : {}), - ...(advertised.includes('vision') ? { input: { text: true, image: true } } : {}), - ...(advertised.includes('embedding') ? { embedding: true } : {}), - }; - const record = modelRecord({ - host: 'ollama', provider: 'ollama', modelId, scopeId: source.scopeId, source, - displayName: modelId, digest, variant, capabilities, - pricing: { basis: 'local-compute', input: 0, output: 0, currency: 'USD', effectiveAt: null }, - states: { discoverable: true }, - }); - for (const evidence of record.evidence) { - if (/^variant\.(?:loaded|memoryBytes|vramBytes|expiresAt|contextWindow)$/.test(evidence.field)) evidence.class = 'runtime'; - } - models.push(record); - } + const { runtime, diagnostic: runtimeDiagnostic } = parseOllamaRuntime(psRaw); + const { models, diagnostics: rowDiagnostics } = ollamaModelsFromRows(rows, { runtime, showByModel, source }); + const diagnostics = [...(runtimeDiagnostic ? [runtimeDiagnostic] : []), ...rowDiagnostics]; if (rows.length > MAX_MODELS) diagnostics.push(diagnostic('models-truncated', `Ollama returned more than ${MAX_MODELS} models`)); const complete = diagnostics.length === 0 && rows.length <= MAX_MODELS; source.complete = complete; source.status = complete ? 'complete' : 'partial'; diff --git a/src/lib/model-inventory/discovery/opencode.mjs b/src/lib/model-inventory/discovery/opencode.mjs index d85911a..2afe32b 100644 --- a/src/lib/model-inventory/discovery/opencode.mjs +++ b/src/lib/model-inventory/discovery/opencode.mjs @@ -46,24 +46,50 @@ function selectorParts(value) { return { selector, provider, modelId }; } +function isValidCatalogModel(modelId, model) { + return plain(model) && model.id === modelId; +} + +/** -1 signals an invalid model row so the caller can fail the whole document closed. */ +function countValidCatalogModels(models) { + let count = 0; + for (const [modelId, model] of Object.entries(models)) { + if (!isValidCatalogModel(modelId, model)) return -1; + count += 1; + } + return count; +} + +function isValidCatalogProvider(providerId, provider) { + return plain(provider) && provider.id === providerId && plain(provider.models); +} + +/** -1 (not 0) distinguishes "an entry was invalid" from "zero valid models found". */ +function catalogModelCount(providers) { + let total = 0; + for (const [providerId, provider] of providers) { + if (!isValidCatalogProvider(providerId, provider)) return -1; + const count = countValidCatalogModels(provider.models); + if (count < 0) return -1; + total += count; + } + return total; +} + +function parseCatalogValue(raw) { + const text = typeof raw === 'string' ? raw : JSON.stringify(raw); + if (Buffer.byteLength(text) > MAX_CATALOG_BYTES) return null; + return typeof raw === 'string' ? JSON.parse(raw) : structuredClone(raw); +} + function catalogDocument(raw) { if (raw == null || raw === '') return null; try { - const text = typeof raw === 'string' ? raw : JSON.stringify(raw); - if (Buffer.byteLength(text) > MAX_CATALOG_BYTES) return null; - const value = typeof raw === 'string' ? JSON.parse(raw) : structuredClone(raw); + const value = parseCatalogValue(raw); if (!plain(value)) return null; const providers = Object.entries(value); if (providers.length === 0) return null; - let modelCount = 0; - for (const [providerId, provider] of providers) { - if (!plain(provider) || provider.id !== providerId || !plain(provider.models)) return null; - for (const [modelId, model] of Object.entries(provider.models)) { - if (!plain(model) || model.id !== modelId) return null; - modelCount += 1; - } - } - return modelCount > 0 ? value : null; + return catalogModelCount(providers) > 0 ? value : null; } catch { return null; } } @@ -89,28 +115,45 @@ function safeModalities(value) { return Object.keys(result).length ? result : null; } -function safeCapabilities(value, limit) { +const CAPABILITY_BOOLEAN_FIELDS = ['temperature', 'reasoning', 'attachment', 'toolcall']; + +function safeCapabilityBooleans(value) { const result = {}; - if (plain(value)) { - for (const name of ['temperature', 'reasoning', 'attachment', 'toolcall']) { - if (typeof value[name] === 'boolean') result[name] = value[name]; - } - const input = safeModalities(value.input); - const output = safeModalities(value.output); - if (input) result.input = input; - if (output) result.output = output; - if (typeof value.interleaved === 'boolean') result.interleaved = value.interleaved; - else if (plain(value.interleaved)) result.interleaved = true; - } - if (plain(limit)) { - const context = positiveSafeInteger(limit.context); - const output = positiveSafeInteger(limit.output); - if (context !== null) result.contextLimit = context; - if (output !== null) result.outputLimit = output; - } + for (const name of CAPABILITY_BOOLEAN_FIELDS) if (typeof value[name] === 'boolean') result[name] = value[name]; + return result; +} + +function safeCapabilityModalities(value) { + const result = {}; + const input = safeModalities(value.input); + const output = safeModalities(value.output); + if (input) result.input = input; + if (output) result.output = output; + return result; +} + +function safeCapabilityInterleaved(value) { + if (typeof value.interleaved === 'boolean') return { interleaved: value.interleaved }; + return plain(value.interleaved) ? { interleaved: true } : {}; +} + +function safeCapabilityLimits(limit) { + if (!plain(limit)) return {}; + const result = {}; + const context = positiveSafeInteger(limit.context); + const output = positiveSafeInteger(limit.output); + if (context !== null) result.contextLimit = context; + if (output !== null) result.outputLimit = output; return result; } +function safeCapabilities(value, limit) { + const base = plain(value) + ? { ...safeCapabilityBooleans(value), ...safeCapabilityModalities(value), ...safeCapabilityInterleaved(value) } + : {}; + return { ...base, ...safeCapabilityLimits(limit) }; +} + function catalogCapabilities(value) { if (!plain(value)) return {}; const result = {}; @@ -146,64 +189,98 @@ function lifecycleState(status) { return 'unknown'; } +/** Advance one character of brace-depth/string-aware JSON scanning. Mutates `state` + * in place and returns true once a complete top-level `{...}` block has closed. */ +function advanceJsonScan(state, char) { + if (state.inString) { + if (state.escaped) state.escaped = false; + else if (char === '\\') state.escaped = true; + else if (char === '"') state.inString = false; + return false; + } + if (char === '"') state.inString = true; + else if (char === '{') { state.depth += 1; state.started = true; } + else if (char === '}') { + state.depth -= 1; + if (state.started && state.depth === 0) return true; + } + return false; +} + function jsonBlock(lines, start) { - let depth = 0; - let inString = false; - let escaped = false; - let started = false; + const state = { + depth: 0, inString: false, escaped: false, started: false, + }; const chunks = []; for (let index = start; index < lines.length; index += 1) { - const line = lines[index]; - chunks.push(line); - for (const char of `${line}\n`) { - if (inString) { - if (escaped) escaped = false; - else if (char === '\\') escaped = true; - else if (char === '"') inString = false; - continue; - } - if (char === '"') inString = true; - else if (char === '{') { depth += 1; started = true; } - else if (char === '}') { - depth -= 1; - if (started && depth === 0) return { raw: chunks.join('\n'), end: index }; - } + chunks.push(lines[index]); + for (const char of `${lines[index]}\n`) { + if (advanceJsonScan(state, char)) return { raw: chunks.join('\n'), end: index }; } } return { raw: chunks.join('\n'), end: lines.length - 1 }; } -function metadataFor(selector, raw, catalogDocumentValue) { - let value; - try { value = JSON.parse(raw); } catch { return { error: 'invalid-model-metadata' }; } - if (!plain(value)) return { error: 'invalid-model-metadata' }; +function parseModelMetadataJson(raw) { + try { + const value = JSON.parse(raw); + return plain(value) ? { value } : { error: 'invalid-model-metadata' }; + } catch { return { error: 'invalid-model-metadata' }; } +} + +/** Selector-carried providerID/id must echo the line's own selector before any + * metadata is trusted; only then is the selector re-parsed into provider/modelId. */ +function validatedSelectorParts(value, selector) { const providerID = boundedText(value.providerID, 256); const id = boundedText(value.id, 512); - if (!providerID || !id || `${providerID}/${id}` !== selector) { - return { error: 'metadata-selector-mismatch' }; - } - const parts = selectorParts(selector); - if (!parts) return { error: 'metadata-selector-mismatch' }; - const { provider, modelId } = parts; - const proof = catalogEntry(catalogDocumentValue, provider, modelId); - const metadata = proof ?? value; + if (!providerID || !id || `${providerID}/${id}` !== selector) return null; + return selectorParts(selector); +} + +function metadataStatus(proof, metadata) { const statusValue = proof && metadata.status === undefined ? 'active' : boundedText(metadata.status, 32)?.toLowerCase() ?? null; - const status = statusValue && CATALOG_STATUSES.has(statusValue) ? statusValue : null; + return statusValue && CATALOG_STATUSES.has(statusValue) ? statusValue : null; +} + +function metadataFamily(metadata) { const familyValue = boundedText(metadata.family, 128); - const family = familyValue && CATALOG_TOKEN.test(familyValue) ? familyValue : null; - const catalog = { + return familyValue && CATALOG_TOKEN.test(familyValue) ? familyValue : null; +} + +function metadataCatalogBlock({ + proof, provider, selector, metadata, status, family, +}) { + return { source: proof ? 'models.dev' : 'opencode', public: !!proof, servingProvider: provider, publisher: null, family, selector, releaseDate: releaseDate(metadata.release_date), status, ...(proof ? { links: { catalog: 'https://models.dev/' } } : {}), }; - const availableVariants = plain(value.variants) +} + +function metadataAvailableVariants(value) { + return plain(value.variants) ? Object.keys(value.variants).filter((name) => VARIANT.test(name)).slice(0, 64).sort() : []; +} + +function metadataFor(selector, raw, catalogDocumentValue) { + const { value, error } = parseModelMetadataJson(raw); + if (error) return { error }; + const parts = validatedSelectorParts(value, selector); + if (!parts) return { error: 'metadata-selector-mismatch' }; + const { provider, modelId } = parts; + const proof = catalogEntry(catalogDocumentValue, provider, modelId); + const metadata = proof ?? value; + const status = metadataStatus(proof, metadata); + const family = metadataFamily(metadata); + const catalog = metadataCatalogBlock({ + proof, provider, selector, metadata, status, family, + }); return { metadata: { displayName: boundedText(metadata.name, 256) ?? selector, catalog, - availableVariants, + availableVariants: metadataAvailableVariants(value), capabilities: proof ? catalogCapabilities(proof) : safeCapabilities(value.capabilities, value.limit), pricing: safePricing(metadata.cost), lifecycle: { state: lifecycleState(status), replacement: null }, @@ -211,6 +288,28 @@ function metadataFor(selector, raw, catalogDocumentValue) { }; } +async function readBoundedCatalogText(response) { + const text = await response.text(); + if (Buffer.byteLength(text) > MAX_CATALOG_BYTES) throw new TypeError('catalog-too-large'); + return text; +} + +async function readBoundedCatalogStream(reader) { + const chunks = []; + let bytes = 0; + while (true) { + const { done, value } = await reader.read(); + if (done) break; + bytes += value.byteLength; + if (bytes > MAX_CATALOG_BYTES) { + await reader.cancel(); + throw new TypeError('catalog-too-large'); + } + chunks.push(value); + } + return Buffer.concat(chunks.map((value) => Buffer.from(value))).toString('utf8'); +} + async function fetchCatalog(fetchFn, timeout) { if (typeof fetchFn !== 'function') throw new TypeError('catalog-fetch-unavailable'); const controller = new AbortController(); @@ -221,33 +320,31 @@ async function fetchCatalog(fetchFn, timeout) { }); if (!response?.ok) throw new TypeError('catalog-fetch-failed'); const declared = Number(response.headers?.get?.('content-length')); - if (Number.isFinite(declared) && declared > MAX_CATALOG_BYTES) { - throw new TypeError('catalog-too-large'); - } - if (!response.body?.getReader) { - const text = await response.text(); - if (Buffer.byteLength(text) > MAX_CATALOG_BYTES) throw new TypeError('catalog-too-large'); - return text; - } - const reader = response.body.getReader(); - const chunks = []; - let bytes = 0; - while (true) { - const { done, value } = await reader.read(); - if (done) break; - bytes += value.byteLength; - if (bytes > MAX_CATALOG_BYTES) { - await reader.cancel(); - throw new TypeError('catalog-too-large'); - } - chunks.push(value); - } - return Buffer.concat(chunks.map((value) => Buffer.from(value))).toString('utf8'); + if (Number.isFinite(declared) && declared > MAX_CATALOG_BYTES) throw new TypeError('catalog-too-large'); + return response.body?.getReader + ? readBoundedCatalogStream(response.body.getReader()) : readBoundedCatalogText(response); } finally { clearTimeout(timer); } } +function nextNonBlankLineIndex(lines, from) { + let next = from; + while (next < lines.length && !lines[next].trim()) next += 1; + return next; +} + +/** A selector line may be followed by a `{...}` metadata block once the next + * non-blank line starts one; otherwise the row carries no metadata. */ +function readRowMetadata(lines, index, selector, catalog, diagnostics) { + const next = nextNonBlankLineIndex(lines, index + 1); + if (next >= lines.length || !lines[next].trimStart().startsWith('{')) return { metadata: null, end: index }; + const block = jsonBlock(lines, next); + const parsed = metadataFor(selector, block.raw, catalog); + if (parsed.error) addDiagnostic(diagnostics, diagnostic(parsed.error, `line ${next + 1} metadata is invalid`)); + return { metadata: parsed.error ? null : parsed.metadata, end: block.end }; +} + function outputRows(text, diagnostics, catalog) { const lines = text.split(/\r?\n/); const rows = []; @@ -259,16 +356,8 @@ function outputRows(text, diagnostics, catalog) { addDiagnostic(diagnostics, diagnostic('invalid-model-id', `line ${index + 1} is invalid`)); continue; } - let metadata = null; - let next = index + 1; - while (next < lines.length && !lines[next].trim()) next += 1; - if (next < lines.length && lines[next].trimStart().startsWith('{')) { - const block = jsonBlock(lines, next); - const parsed = metadataFor(selector, block.raw, catalog); - if (parsed.error) addDiagnostic(diagnostics, diagnostic(parsed.error, `line ${next + 1} metadata is invalid`)); - else metadata = parsed.metadata; - index = block.end; - } + const { metadata, end } = readRowMetadata(lines, index, selector, catalog, diagnostics); + index = end; if (!seen.has(selector)) { rows.push({ selector, metadata }); seen.add(selector); @@ -278,47 +367,109 @@ function outputRows(text, diagnostics, catalog) { return rows; } +function referenceString(value) { + if (typeof value === 'string') return value; + if (!value || typeof value !== 'object' || Array.isArray(value)) return null; + return typeof value.providerID === 'string' && typeof value.model === 'string' + ? `${value.providerID}/${value.model}` : null; +} + +function splitSelectorVariant(ref) { + const hash = ref.lastIndexOf('#'); + return hash > 0 ? { id: ref.slice(0, hash), variant: ref.slice(hash + 1) } : { id: ref, variant: null }; +} + function selection(value) { - let ref = value; - if (value && typeof value === 'object' && !Array.isArray(value)) { - ref = typeof value.providerID === 'string' && typeof value.model === 'string' - ? `${value.providerID}/${value.model}` : null; - } + const ref = referenceString(value); if (typeof ref !== 'string' || ref.length > 576) return null; - const hash = ref.lastIndexOf('#'); - const id = hash > 0 ? ref.slice(0, hash) : ref; - const variant = hash > 0 ? ref.slice(hash + 1) : null; + const { id, variant } = splitSelectorVariant(ref); return selectorParts(id) && (!variant || VARIANT.test(variant)) ? { id, variant } : null; } -function configuredRefs(raw) { - if (raw === undefined || raw === null || raw === '') return []; +function parseOpenCodeConfig(raw) { const text = typeof raw === 'string' ? raw : JSON.stringify(raw); if (Buffer.byteLength(text) > MAX_COMMAND_BYTES) throw new TypeError('config-too-large'); let config; try { config = typeof raw === 'string' ? JSON.parse(raw) : structuredClone(raw); } catch { throw new TypeError('config-invalid-json'); } if (!config || typeof config !== 'object' || Array.isArray(config)) throw new TypeError('config-schema'); - const refs = [config.model]; - for (const agent of Object.values(config.agent && typeof config.agent === 'object' ? config.agent : {})) { - if (agent && typeof agent === 'object') refs.push(agent.model); - } - for (const command of Object.values(config.command && typeof config.command === 'object' ? config.command : {})) { - if (command && typeof command === 'object') refs.push(command.model); + return config; +} + +function modelRefsFromGroup(group) { + const refs = []; + for (const entry of Object.values(group && typeof group === 'object' ? group : {})) { + if (entry && typeof entry === 'object') refs.push(entry.model); } - return refs.map(selection).filter(Boolean) - .filter((entry, index, all) => all.findIndex(({ id, variant }) => id === entry.id && variant === entry.variant) === index); + return refs; +} + +function dedupeSelections(entries) { + return entries.filter((entry, index, all) => all + .findIndex(({ id, variant }) => id === entry.id && variant === entry.variant) === index); +} + +function configuredRefs(raw) { + if (raw === undefined || raw === null || raw === '') return []; + const config = parseOpenCodeConfig(raw); + const refs = [config.model, ...modelRefsFromGroup(config.agent), ...modelRefsFromGroup(config.command)]; + return dedupeSelections(refs.map(selection).filter(Boolean)); } -export function discoverOpenCode({ raw, configRaw, catalogRaw, initialDiagnostics = [], capturedAt, - scope = {}, scopeKey, online = false } = /** @type {any} */ ({})) { +function openCodeSourceUnsupportedResult(source, online) { + source.complete = false; + source.status = 'unsupported'; + source.diagnostics = ['output-too-large']; + return { + status: 'unsupported', source, models: [], + diagnostics: [diagnostic('output-too-large', `output exceeds ${MAX_COMMAND_BYTES}`)], networkUsed: online, + }; +} + +function openCodeModelStates(qualified, { + configuredIds, configured, configRaw, ids, +}) { + return { + configured: configuredIds.includes(qualified) ? true : 'unknown', + effective: configRaw !== undefined && configured[0]?.id === qualified ? true : 'unknown', + discoverable: ids.includes(qualified) ? true : 'unknown', entitled: 'unknown', + }; +} + +function openCodeModelVariant(qualified, configured, metadata) { + return { + configuredVariants: configured.filter(({ id }) => id === qualified) + .map(({ variant }) => variant).filter(Boolean), + ...(metadata ? { catalog: metadata.catalog, availableVariants: metadata.availableVariants } : {}), + }; +} + +function openCodeModelFromId(qualified, ctx) { + const { + rows, configured, configuredIds, ids, configRaw, source, + } = ctx; + const slash = qualified.indexOf('/'); + const provider = slash > 0 ? qualified.slice(0, slash) : null; + const modelId = slash > 0 ? qualified.slice(slash + 1) : qualified; + const metadata = rows.find(({ selector }) => selector === qualified)?.metadata ?? null; + return modelRecord({ + host: 'opencode', provider, modelId, scopeId: source.scopeId, + displayName: metadata?.displayName ?? qualified, source, + states: openCodeModelStates(qualified, { + configuredIds, configured, configRaw, ids, + }), + variant: openCodeModelVariant(qualified, configured, metadata), + capabilities: metadata?.capabilities ?? {}, + pricing: metadata?.pricing ?? null, + lifecycle: metadata?.lifecycle, + }); +} + +export function discoverOpenCode({ + raw, configRaw, catalogRaw, initialDiagnostics = [], capturedAt, scope = {}, scopeKey, online = false, +} = /** @type {any} */ ({})) { const text = String(raw ?? ''); const source = sourceRecord({ id: 'opencode-models', owner: 'opencode', scope, scopeKey, capturedAt, complete: true, schema: 'opencode-models-lines-v1' }); - if (Buffer.byteLength(text) > MAX_COMMAND_BYTES) { - source.complete = false; - source.status = 'unsupported'; - source.diagnostics = ['output-too-large']; - return { status: 'unsupported', source, models: [], diagnostics: [diagnostic('output-too-large', `output exceeds ${MAX_COMMAND_BYTES}`)], networkUsed: online }; - } + if (Buffer.byteLength(text) > MAX_COMMAND_BYTES) return openCodeSourceUnsupportedResult(source, online); const diagnostics = []; for (const item of initialDiagnostics) addDiagnostic(diagnostics, item); let configured = []; @@ -335,32 +486,62 @@ export function discoverOpenCode({ raw, configRaw, catalogRaw, initialDiagnostic source.diagnostics = diagnostics.map(({ code }) => code); const configuredIds = configured.map(({ id }) => id); const allIds = [...new Set([...ids, ...configuredIds])]; - const models = allIds.map((qualified) => { - const slash = qualified.indexOf('/'); - const provider = slash > 0 ? qualified.slice(0, slash) : null; - const modelId = slash > 0 ? qualified.slice(slash + 1) : qualified; - const metadata = rows.find(({ selector }) => selector === qualified)?.metadata ?? null; - return modelRecord({ - host: 'opencode', provider, modelId, scopeId: source.scopeId, - displayName: metadata?.displayName ?? qualified, source, - states: { - configured: configuredIds.includes(qualified) ? true : 'unknown', - effective: configRaw !== undefined && configured[0]?.id === qualified ? true : 'unknown', - discoverable: ids.includes(qualified) ? true : 'unknown', entitled: 'unknown', - }, - variant: { - configuredVariants: configured.filter(({ id }) => id === qualified) - .map(({ variant }) => variant).filter(Boolean), - ...(metadata ? { catalog: metadata.catalog, availableVariants: metadata.availableVariants } : {}), - }, - capabilities: metadata?.capabilities ?? {}, - pricing: metadata?.pricing ?? null, - lifecycle: metadata?.lifecycle, - }); - }); + const models = allIds.map((qualified) => openCodeModelFromId(qualified, { + rows, configured, configuredIds, ids, configRaw, source, + })); return { status: complete ? 'complete' : 'partial', source, models, diagnostics, networkUsed: online }; } +function unavailableOpenCodeResult({ scope, scopeKey, capturedAt }, message, networkUsed) { + const source = sourceRecord({ + id: 'opencode-models', owner: 'opencode', scope, scopeKey, capturedAt, + complete: false, status: 'unavailable', schema: 'opencode-models-lines-v1', diagnostics: ['command-failed'], + }); + return { + status: 'unavailable', source, models: [], diagnostics: [diagnostic('command-failed', message)], networkUsed, + }; +} + +async function resolveOpenCodeConfigRaw(runner, timeout, initialDiagnostics) { + const configured = await runner('opencode', ['debug', 'config'], { + timeout, maxBuffer: MAX_COMMAND_BYTES, shell: false, + }); + if (configured.code === 0) return configured.stdout; + addDiagnostic(initialDiagnostics, diagnostic('config-unavailable', 'opencode resolved config unavailable')); + return undefined; +} + +async function resolveModelsDevCatalogRaw(fetchFn, timeout, initialDiagnostics) { + try { + const catalogRaw = await fetchCatalog(fetchFn, timeout); + if (!catalogDocument(catalogRaw)) throw new TypeError('catalog-invalid'); + return catalogRaw; + } catch (error) { + const invalid = error instanceof TypeError && error.message === 'catalog-invalid'; + addDiagnostic(initialDiagnostics, + diagnostic(invalid ? 'catalog-proof-invalid' : 'catalog-proof-unavailable', + invalid ? 'Models.dev identity proof is malformed or unsupported' + : 'Models.dev identity proof unavailable')); + return undefined; + } +} + +/** Returns `{ error }` (an unavailable result) on a failed --refresh, otherwise + * `{ catalogRaw }` — fetching Models.dev proof only when the caller didn't supply one. */ +async function refreshOpenCodeOnline({ + runner, baseArgs, timeout, fetchFn, catalogRaw, initialDiagnostics, scopeCtx, +}) { + const refreshed = await runner('opencode', [...baseArgs, '--refresh'], { + timeout, maxBuffer: MAX_COMMAND_BYTES, shell: false, + }); + if (refreshed.code !== 0) { + return { error: unavailableOpenCodeResult(scopeCtx, refreshed.stderr || 'opencode models refresh failed', true) }; + } + const resolvedCatalogRaw = catalogRaw === undefined + ? await resolveModelsDevCatalogRaw(fetchFn, timeout, initialDiagnostics) : catalogRaw; + return { catalogRaw: resolvedCatalogRaw }; +} + export async function collectOpenCode({ runner = run, fetchFn = globalThis.fetch, online = false, provider, configRaw, catalogRaw, capturedAt, scope = {}, scopeKey, timeout = 30_000, @@ -368,45 +549,20 @@ export async function collectOpenCode({ const providerArg = typeof provider === 'string' && provider.length <= 256 ? provider : null; const baseArgs = ['models', ...(providerArg ? [providerArg] : [])]; const initialDiagnostics = []; - if (configRaw === undefined) { - const configured = await runner('opencode', ['debug', 'config'], { - timeout, maxBuffer: MAX_COMMAND_BYTES, shell: false, - }); - if (configured.code === 0) configRaw = configured.stdout; - else addDiagnostic(initialDiagnostics, diagnostic('config-unavailable', 'opencode resolved config unavailable')); - } + const scopeCtx = { scope, scopeKey, capturedAt }; + if (configRaw === undefined) configRaw = await resolveOpenCodeConfigRaw(runner, timeout, initialDiagnostics); if (online) { - const refreshed = await runner('opencode', [...baseArgs, '--refresh'], { - timeout, maxBuffer: MAX_COMMAND_BYTES, shell: false, + const refresh = await refreshOpenCodeOnline({ + runner, baseArgs, timeout, fetchFn, catalogRaw, initialDiagnostics, scopeCtx, }); - if (refreshed.code !== 0) { - const source = sourceRecord({ id: 'opencode-models', owner: 'opencode', scope, scopeKey, - capturedAt, complete: false, status: 'unavailable', schema: 'opencode-models-lines-v1', - diagnostics: ['command-failed'] }); - return { status: 'unavailable', source, models: [], - diagnostics: [diagnostic('command-failed', refreshed.stderr || 'opencode models refresh failed')], - networkUsed: true }; - } - if (catalogRaw === undefined) { - try { - catalogRaw = await fetchCatalog(fetchFn, timeout); - if (!catalogDocument(catalogRaw)) throw new TypeError('catalog-invalid'); - } catch (error) { - const invalid = error instanceof TypeError && error.message === 'catalog-invalid'; - addDiagnostic(initialDiagnostics, - diagnostic(invalid ? 'catalog-proof-invalid' : 'catalog-proof-unavailable', - invalid ? 'Models.dev identity proof is malformed or unsupported' - : 'Models.dev identity proof unavailable')); - } - } + if (refresh.error) return refresh.error; + catalogRaw = refresh.catalogRaw; } const result = await runner('opencode', [...baseArgs, '--verbose'], { timeout, maxBuffer: MAX_COMMAND_BYTES, shell: false, }); - if (result.code !== 0) { - const source = sourceRecord({ id: 'opencode-models', owner: 'opencode', scope, scopeKey, capturedAt, complete: false, status: 'unavailable', schema: 'opencode-models-lines-v1', diagnostics: ['command-failed'] }); - return { status: 'unavailable', source, models: [], diagnostics: [diagnostic('command-failed', result.stderr || 'opencode models failed')], networkUsed: online }; - } - return discoverOpenCode({ raw: result.stdout, configRaw, catalogRaw, initialDiagnostics, - capturedAt, scope, scopeKey, online }); + if (result.code !== 0) return unavailableOpenCodeResult(scopeCtx, result.stderr || 'opencode models failed', online); + return discoverOpenCode({ + raw: result.stdout, configRaw, catalogRaw, initialDiagnostics, capturedAt, scope, scopeKey, online, + }); } diff --git a/src/lib/model-inventory/observed.mjs b/src/lib/model-inventory/observed.mjs index 255b9a4..f006724 100644 --- a/src/lib/model-inventory/observed.mjs +++ b/src/lib/model-inventory/observed.mjs @@ -3,55 +3,91 @@ import { diagnostic, modelRecord, scopeFingerprint, sourceRecord } from './disco const bounded = (value) => typeof value === 'string' && value.length > 0 && value.length <= 512 ? value : null; -export async function collectObservedModels({ - readIndexFn = readIndex, indexOptions = {}, scope = {}, scopeKey, days = 365, -} = /** @type {any} */ ({})) { - let aggregate; - try { aggregate = await readIndexFn({ days, ...indexOptions }); } catch (error) { - const capturedAt = new Date().toISOString(); - return { - status: 'unavailable', generatedAt: capturedAt, models: [], - source: sourceRecord({ id: 'usage-index', owner: 'usage', scope, scopeKey, capturedAt, - ownerType: 'usage', transport: 'index', network: 'never', mode: 'local', - complete: false, status: 'unavailable', schema: 'usage-index-v6', diagnostics: ['usage-index-unavailable'] }), - diagnostics: [diagnostic('usage-index-unavailable', error?.message ?? error)], - }; +function unavailableObservedResult(scope, scopeKey, error) { + const capturedAt = new Date().toISOString(); + return { + status: 'unavailable', generatedAt: capturedAt, models: [], + source: sourceRecord({ + id: 'usage-index', owner: 'usage', scope, scopeKey, capturedAt, + ownerType: 'usage', transport: 'index', network: 'never', mode: 'local', + complete: false, status: 'unavailable', schema: 'usage-index-v6', diagnostics: ['usage-index-unavailable'], + }), + diagnostics: [diagnostic('usage-index-unavailable', error?.message ?? error)], + }; +} + +function observedModelRecord({ + host, provider, modelId, scope, scopeKey, generatedAt, +}) { + const source = { + id: 'usage-index', capturedAt: generatedAt ?? new Date().toISOString(), + complete: true, freshness: 'current', + }; + const record = modelRecord({ + host, provider, modelId, scopeId: scopeFingerprint(host, scope, scopeKey), source, + // A successful structured invocation proves this exact observed path was + // entitled, policy-allowed, and routable at capture time. It says + // nothing about catalog completeness or other profiles/projects. + states: { observed: true, entitled: true, policyAllowed: true, routable: true }, + }); + record.observations = 1; + return record; +} + +function foldSessionModels(map, session, scope, scopeKey, generatedAt) { + const host = bounded(session.host) ?? 'unknown'; + const provider = bounded(session.provider); + for (const modelId of Array.isArray(session.models) ? session.models : []) { + if (!bounded(modelId)) continue; + const key = `${host}\0${provider ?? ''}\0${modelId}`; + const prior = map.get(key); + if (prior) { prior.observations++; continue; } + const record = observedModelRecord({ + host, provider, modelId, scope, scopeKey, generatedAt, + }); + record.evidence[0].providerProvenance = session.providerProvenance ?? 'unknown'; + map.set(key, record); } +} + +function observedModelsMap(sessions, scope, scopeKey, generatedAt) { const map = new Map(); - for (const session of Array.isArray(aggregate?.sessions) ? aggregate.sessions : []) { - const host = bounded(session.host) ?? 'unknown'; - const provider = bounded(session.provider); - for (const modelId of Array.isArray(session.models) ? session.models : []) { - if (!bounded(modelId)) continue; - const key = `${host}\0${provider ?? ''}\0${modelId}`; - const prior = map.get(key); - if (prior) { prior.observations++; continue; } - const source = { - id: 'usage-index', capturedAt: aggregate.generatedAt ?? new Date().toISOString(), - complete: true, freshness: 'current', - }; - const record = modelRecord({ - host, provider, modelId, scopeId: scopeFingerprint(host, scope, scopeKey), source, - // A successful structured invocation proves this exact observed path was - // entitled, policy-allowed, and routable at capture time. It says - // nothing about catalog completeness or other profiles/projects. - states: { observed: true, entitled: true, policyAllowed: true, routable: true }, - }); - record.observations = 1; - record.evidence[0].providerProvenance = session.providerProvenance ?? 'unknown'; - map.set(key, record); - } + for (const session of Array.isArray(sessions) ? sessions : []) { + foldSessionModels(map, session, scope, scopeKey, generatedAt); } + return map; +} + +function observedResultStatus(degraded) { + return degraded ? 'partial' : 'complete'; +} + +function finalizeObservedResult({ + aggregate, scope, scopeKey, map, +}) { const degraded = Object.values(aggregate?.sourceHealth ?? {}).some((health) => health?.status === 'degraded'); const capturedAt = aggregate?.generatedAt ?? new Date().toISOString(); const source = sourceRecord({ id: 'usage-index', owner: 'usage', scope, scopeKey, capturedAt, complete: !degraded, ownerType: 'usage', transport: 'index', network: 'never', mode: 'local', - status: degraded ? 'partial' : 'complete', schema: 'usage-index-v6', + status: observedResultStatus(degraded), schema: 'usage-index-v6', diagnostics: degraded ? ['usage-source-degraded'] : [], }); return { - status: degraded ? 'partial' : 'complete', generatedAt: capturedAt, + status: observedResultStatus(degraded), generatedAt: capturedAt, models: [...map.values()], source, diagnostics: [], sourceHealth: aggregate?.sourceHealth ?? {}, }; } + +export async function collectObservedModels({ + readIndexFn = readIndex, indexOptions = {}, scope = {}, scopeKey, days = 365, +} = /** @type {any} */ ({})) { + let aggregate; + try { aggregate = await readIndexFn({ days, ...indexOptions }); } catch (error) { + return unavailableObservedResult(scope, scopeKey, error); + } + const map = observedModelsMap(aggregate?.sessions, scope, scopeKey, aggregate?.generatedAt); + return finalizeObservedResult({ + aggregate, scope, scopeKey, map, + }); +} diff --git a/src/lib/model-inventory/read-model.mjs b/src/lib/model-inventory/read-model.mjs index a1a9438..9283221 100644 --- a/src/lib/model-inventory/read-model.mjs +++ b/src/lib/model-inventory/read-model.mjs @@ -199,43 +199,70 @@ function ownerVisibleModelText(value, max = 512) { return boundedPublicText(value, max); } -function ownerVisibleCapabilities(value) { - if (!value || typeof value !== 'object' || Array.isArray(value)) return {}; +const BOOLEAN_CAPABILITY_FIELDS = [ + 'tools', 'toolcall', 'reasoning', 'structuredOutput', 'temperature', 'attachment', 'interleaved', 'embedding', +]; +const LIMIT_CAPABILITY_FIELDS = ['contextLimit', 'outputLimit']; +const MODALITY_FIELDS = ['text', 'audio', 'image', 'video', 'pdf']; + +function booleanCapabilityFields(value) { const result = {}; - for (const name of ['tools', 'toolcall', 'reasoning', 'structuredOutput', 'temperature', 'attachment', 'interleaved', 'embedding']) { - if (typeof value[name] === 'boolean') result[name] = value[name]; - } - for (const name of ['contextLimit', 'outputLimit']) { + for (const name of BOOLEAN_CAPABILITY_FIELDS) if (typeof value[name] === 'boolean') result[name] = value[name]; + return result; +} + +function limitCapabilityFields(value) { + const result = {}; + for (const name of LIMIT_CAPABILITY_FIELDS) { if (Number.isSafeInteger(value[name]) && value[name] > 0) result[name] = value[name]; } + return result; +} + +function capabilityModalities(directionValue) { + if (!directionValue || typeof directionValue !== 'object' || Array.isArray(directionValue)) return null; + const result = {}; + for (const name of MODALITY_FIELDS) if (typeof directionValue[name] === 'boolean') result[name] = directionValue[name]; + return Object.keys(result).length ? result : null; +} + +function directionalCapabilityFields(value) { + const result = {}; for (const direction of ['input', 'output']) { - if (!value[direction] || typeof value[direction] !== 'object' || Array.isArray(value[direction])) continue; - const modalities = {}; - for (const name of ['text', 'audio', 'image', 'video', 'pdf']) { - if (typeof value[direction][name] === 'boolean') modalities[name] = value[direction][name]; - } - if (Object.keys(modalities).length) result[direction] = modalities; + const modalities = capabilityModalities(value[direction]); + if (modalities) result[direction] = modalities; } return result; } -function ownerVisiblePricing(model) { - const pricing = model.pricing; - if (pricing && ['per-token', 'per-million-tokens', 'zero', 'local-compute'].includes(pricing.basis) +function ownerVisibleCapabilities(value) { + if (!value || typeof value !== 'object' || Array.isArray(value)) return {}; + return { ...booleanCapabilityFields(value), ...limitCapabilityFields(value), ...directionalCapabilityFields(value) }; +} + +function isValidPricingRecord(pricing) { + return Boolean(pricing) && ['per-token', 'per-million-tokens', 'zero', 'local-compute'].includes(pricing.basis) && (Number.isFinite(pricing.input) || Number.isFinite(pricing.output)) - && /^[A-Z]{3}$/.test(pricing.currency ?? '')) { - const refs = new Set(pricing.evidenceRefs ?? []); - const anthropicDocs = model.evidence.some((entry) => refs.has(entry.id) - && entry.source === 'anthropic-docs' && entry.class === 'first-party'); - return { - basis: pricing.basis, input: pricing.input, output: pricing.output, currency: pricing.currency, - effectiveAt: pricing.effectiveAt, - source: pricing.basis === 'local-compute' ? 'local installation evidence' - : anthropicDocs ? 'Anthropic Models and pricing' : 'catalogue evidence', - sourceUrl: anthropicDocs ? ANTHROPIC_PRICING_URL : null, - asOf: anthropicDocs ? PRICES_AS_OF : null, matched: true, - }; - } + && /^[A-Z]{3}$/.test(pricing.currency ?? ''); +} + +function evidencedPricing(model) { + const pricing = model.pricing; + if (!isValidPricingRecord(pricing)) return null; + const refs = new Set(pricing.evidenceRefs ?? []); + const anthropicDocs = model.evidence.some((entry) => refs.has(entry.id) + && entry.source === 'anthropic-docs' && entry.class === 'first-party'); + return { + basis: pricing.basis, input: pricing.input, output: pricing.output, currency: pricing.currency, + effectiveAt: pricing.effectiveAt, + source: pricing.basis === 'local-compute' ? 'local installation evidence' + : anthropicDocs ? 'Anthropic Models and pricing' : 'catalogue evidence', + sourceUrl: anthropicDocs ? ANTHROPIC_PRICING_URL : null, + asOf: anthropicDocs ? PRICES_AS_OF : null, matched: true, + }; +} + +function publishedPricing(model) { const published = priceFor(model.key.modelId, model.key.provider); if (!published.matched) return null; const sourceUrl = published.provider === 'openai' ? openAiModelDocumentation(model.key.modelId) : null; @@ -246,6 +273,24 @@ function ownerVisiblePricing(model) { }; } +function ownerVisiblePricing(model) { + return evidencedPricing(model) ?? publishedPricing(model); +} + +function trustedModelLink(entry, labels) { + if (!entry || typeof entry !== 'object') return null; + const kind = boundedPublicText(entry.kind, 64); + const label = labels[kind] ?? null; + const raw = boundedPublicText(entry.url, 2_048); + if (!kind || !label || !raw) return null; + try { + const url = new URL(raw); + if (url.protocol !== 'https:' || url.username || url.password || url.port + || !TRUSTED_MODEL_LINK_HOSTS.has(url.hostname)) return null; + return { kind, label, url: url.href }; + } catch { return null; } +} + function trustedModelLinks(value) { const labels = { catalog: 'Models.dev', documentation: 'Documentation', provider: 'Provider', @@ -254,19 +299,7 @@ function trustedModelLinks(value) { const entries = Array.isArray(value) ? value : value && typeof value === 'object' ? Object.entries(value).flatMap(([kind, url]) => labels[kind] && typeof url === 'string' ? [{ kind, label: labels[kind], url }] : []) : []; - return entries.slice(0, 16).flatMap((entry) => { - if (!entry || typeof entry !== 'object') return []; - const kind = boundedPublicText(entry.kind, 64); - const label = labels[kind] ?? null; - const raw = boundedPublicText(entry.url, 2_048); - if (!kind || !label || !raw) return []; - try { - const url = new URL(raw); - if (url.protocol !== 'https:' || url.username || url.password || url.port - || !TRUSTED_MODEL_LINK_HOSTS.has(url.hostname)) return []; - return [{ kind, label, url: url.href }]; - } catch { return []; } - }); + return entries.slice(0, 16).map((entry) => trustedModelLink(entry, labels)).filter(Boolean); } function hasEvidence(model, predicate) { @@ -297,35 +330,42 @@ function claudeHumanName(modelId) { + (date ? ` (${date})` : ''); } -/** Public identity is fail-closed: local/custom rows need an explicit catalog-public marker. */ -function publicModelIdentity(model) { - if (model.key.host === 'codex' && hasCatalogDiscovery(model, 'codex-cache')) { - const modelDocumentation = openAiModelDocumentation(model.key.modelId); - return { - humanName: boundedPublicText(model.displayName) ?? model.key.modelId, - selector: model.key.modelId, servingProvider: 'openai', publisher: 'OpenAI', - family: boundedPublicText(model.variant?.family), - links: [{ kind: 'documentation', label: modelDocumentation ? 'OpenAI API model page' : 'Codex models', - url: modelDocumentation ?? 'https://developers.openai.com/codex/models/' }], - }; - } - if (model.key.host === 'claude' && OFFICIAL_CLAUDE_IDS.has(model.key.modelId) +function codexPublicIdentity(model) { + if (model.key.host !== 'codex' || !hasCatalogDiscovery(model, 'codex-cache')) return null; + const modelDocumentation = openAiModelDocumentation(model.key.modelId); + return { + humanName: boundedPublicText(model.displayName) ?? model.key.modelId, + selector: model.key.modelId, servingProvider: 'openai', publisher: 'OpenAI', + family: boundedPublicText(model.variant?.family), + links: [{ kind: 'documentation', label: modelDocumentation ? 'OpenAI API model page' : 'Codex models', + url: modelDocumentation ?? 'https://developers.openai.com/codex/models/' }], + }; +} + +function isOfficialClaudeIdentity(model) { + return model.key.host === 'claude' && OFFICIAL_CLAUDE_IDS.has(model.key.modelId) && hasEvidence(model, (entry) => (entry.source === 'anthropic-docs' && entry.class === 'first-party') || (entry.source === 'claude-config' && ['configured', 'first-party'].includes(entry.class)) - || (entry.source === 'usage-index' && entry.class === 'observed'))) { - const catalogPublic = hasEvidence(model, (entry) => entry.source === 'anthropic-docs' - && entry.class === 'first-party'); - return { - humanName: catalogPublic - ? boundedPublicText(model.displayName) ?? claudeHumanName(model.key.modelId) - : claudeHumanName(model.key.modelId), - selector: model.key.modelId, - servingProvider: 'anthropic', - publisher: 'Anthropic', family: model.key.modelId.split('-')[1], - links: [{ kind: 'documentation', label: 'Anthropic Models', url: ANTHROPIC_MODELS_URL }], - }; - } + || (entry.source === 'usage-index' && entry.class === 'observed')); +} + +function claudePublicIdentity(model) { + if (!isOfficialClaudeIdentity(model)) return null; + const catalogPublic = hasEvidence(model, (entry) => entry.source === 'anthropic-docs' + && entry.class === 'first-party'); + return { + humanName: catalogPublic + ? boundedPublicText(model.displayName) ?? claudeHumanName(model.key.modelId) + : claudeHumanName(model.key.modelId), + selector: model.key.modelId, + servingProvider: 'anthropic', + publisher: 'Anthropic', family: model.key.modelId.split('-')[1], + links: [{ kind: 'documentation', label: 'Anthropic Models', url: ANTHROPIC_MODELS_URL }], + }; +} + +function catalogPublicIdentity(model) { const catalog = model.variant?.catalog; if (!catalog || catalog.public !== true || !PUBLIC_CATALOG_METADATA_SOURCES.has(catalog.source) || !hasEvidence(model, (entry) => entry.field === 'variant.catalog' @@ -342,6 +382,77 @@ function publicModelIdentity(model) { }; } +/** Public identity is fail-closed: local/custom rows need an explicit catalog-public marker. */ +function publicModelIdentity(model) { + return codexPublicIdentity(model) ?? claudePublicIdentity(model) ?? catalogPublicIdentity(model); +} + +function sanitizedIdentity(model, publicIdentity) { + return { + displayName: publicIdentity?.humanName + ?? ownerVisibleModelText(model.displayName) ?? ownerVisibleModelText(model.key.modelId) ?? 'Model not recorded', + humanName: publicIdentity?.humanName + ?? ownerVisibleModelText(model.displayName) ?? ownerVisibleModelText(model.key.modelId), + servingProvider: ownerVisibleModelText(model.key.provider) ?? publicIdentity?.servingProvider ?? null, + publisher: publicIdentity?.publisher ?? null, + family: publicIdentity?.family ?? null, + selector: ownerVisibleModelText(model.key.modelId) ?? publicIdentity?.selector ?? null, + privacyClass: publicIdentity ? 'public-catalog' : 'owner-visible', + links: publicIdentity?.links ?? [], + }; +} + +function sanitizedAliases(model, key, evidenceRefs) { + return model.aliases.map((alias) => ({ + ...alias, + name: privateLabel('alias', alias.name, key), + resolvesTo: privateLabel('model', alias.resolvesTo, key), + evidenceRefs: evidenceRefs(alias.evidenceRefs), + })); +} + +function sanitizedVariantBlock(model, key, publicIdentity, privateVariant) { + return { + ...sanitizeVariant(privateVariant, key), + ...(publicIdentity && model.variant?.catalog ? { catalog: { + source: model.variant.catalog.source, + public: true, + servingProvider: publicIdentity.servingProvider, + publisher: publicIdentity.publisher, + family: publicIdentity.family, + selector: publicIdentity.selector, + links: publicIdentity.links, + } } : {}), + }; +} + +function sanitizedLifecycle(model, evidenceRefs) { + return { + ...model.lifecycle, + replacement: ownerVisibleModelText(model.lifecycle.replacement), + replacementName: ownerVisibleModelText(model.lifecycle.replacement), + replacementSelector: ownerVisibleModelText(model.lifecycle.replacement), + notice: model.lifecycle.notice ? 'Lifecycle notice available in explicit CLI evidence.' : null, + evidenceRefs: evidenceRefs(model.lifecycle.evidenceRefs), + }; +} + +function sanitizedEdges(model, key, evidenceRefs) { + return (model.edges ?? []).map((edge) => ({ + ...edge, + from: privateLabel(edge.kind === 'resolves-to' ? 'alias' : 'model', edge.from, key), + to: privateLabel('model', edge.to, key), + scopeFingerprint: privateLabel('scope', edge.scopeFingerprint, key), + evidenceRefs: evidenceRefs(edge.evidenceRefs), + })); +} + +function sanitizedDimensions(model, evidenceRefs) { + return Object.fromEntries(Object.entries(model.dimensions).map(([name, dimension]) => [name, { + ...dimension, evidenceRefs: evidenceRefs(dimension.evidenceRefs), + }])); +} + function sanitizeModel(model, key) { const evidenceIds = new Map(model.evidence.map(({ id }) => [id, privateLabel('evidence', id, key)])); const evidenceRefs = (refs = []) => refs.map((ref) => evidenceIds.get(ref) @@ -358,69 +469,43 @@ function sanitizeModel(model, key) { digest: privateLabel('digest', model.key.digest, key), }, identity: privateLabel('identity', model.identity, key), - displayName: publicIdentity?.humanName - ?? ownerVisibleModelText(model.displayName) ?? ownerVisibleModelText(model.key.modelId) ?? 'Model not recorded', - humanName: publicIdentity?.humanName - ?? ownerVisibleModelText(model.displayName) ?? ownerVisibleModelText(model.key.modelId), + ...sanitizedIdentity(model, publicIdentity), host: publicHost(model.key.host, key), - servingProvider: ownerVisibleModelText(model.key.provider) ?? publicIdentity?.servingProvider ?? null, - publisher: publicIdentity?.publisher ?? null, - family: publicIdentity?.family ?? null, - selector: ownerVisibleModelText(model.key.modelId) ?? publicIdentity?.selector ?? null, - privacyClass: publicIdentity ? 'public-catalog' : 'owner-visible', - links: publicIdentity?.links ?? [], - aliases: model.aliases.map((alias) => ({ - ...alias, - name: privateLabel('alias', alias.name, key), - resolvesTo: privateLabel('model', alias.resolvesTo, key), - evidenceRefs: evidenceRefs(alias.evidenceRefs), - })), - variant: { - ...sanitizeVariant(privateVariant, key), - ...(publicIdentity && model.variant?.catalog ? { catalog: { - source: model.variant.catalog.source, - public: true, - servingProvider: publicIdentity.servingProvider, - publisher: publicIdentity.publisher, - family: publicIdentity.family, - selector: publicIdentity.selector, - links: publicIdentity.links, - } } : {}), - }, - lifecycle: { - ...model.lifecycle, - replacement: ownerVisibleModelText(model.lifecycle.replacement), - replacementName: ownerVisibleModelText(model.lifecycle.replacement), - replacementSelector: ownerVisibleModelText(model.lifecycle.replacement), - notice: model.lifecycle.notice ? 'Lifecycle notice available in explicit CLI evidence.' : null, - evidenceRefs: evidenceRefs(model.lifecycle.evidenceRefs), - }, + aliases: sanitizedAliases(model, key, evidenceRefs), + variant: sanitizedVariantBlock(model, key, publicIdentity, privateVariant), + lifecycle: sanitizedLifecycle(model, evidenceRefs), capabilities: ownerVisibleCapabilities(model.capabilities), pricing: ownerVisiblePricing(model), - edges: (model.edges ?? []).map((edge) => ({ - ...edge, - from: privateLabel(edge.kind === 'resolves-to' ? 'alias' : 'model', edge.from, key), - to: privateLabel('model', edge.to, key), - scopeFingerprint: privateLabel('scope', edge.scopeFingerprint, key), - evidenceRefs: evidenceRefs(edge.evidenceRefs), - })), - dimensions: Object.fromEntries(Object.entries(model.dimensions).map(([name, dimension]) => [name, { - ...dimension, evidenceRefs: evidenceRefs(dimension.evidenceRefs), - }])), + edges: sanitizedEdges(model, key, evidenceRefs), + dimensions: sanitizedDimensions(model, evidenceRefs), evidence: model.evidence.map((entry) => sanitizeEvidence(entry, key)), }; } +function routeConsumerLabel(text, key) { + const route = /^route:([^:]+)(?::escalation:(\d+))?$/.exec(text); + if (!route) return null; + return `${publicActivity(route[1], key) ?? 'Route'} · ${route[2] == null ? 'primary' : `fallback ${Number(route[2]) + 1}`}`; +} + +function ordinalConsumerLabel(text, pattern, prefix) { + return pattern.test(text) ? `${prefix} ${Number(text.split(':').at(-1)) + 1}` : null; +} + +function agentOverrideConsumerLabel(text, key) { + return text.startsWith('aqe:agent:') + ? `Agentic QE · ${publicActivity(text.slice('aqe:agent:'.length), key) ?? 'override'}` : null; +} + function consumerLabel(value, key) { const text = String(value ?? 'consumer'); - const route = /^route:([^:]+)(?::escalation:(\d+))?$/.exec(text); - if (route) return `${publicActivity(route[1], key) ?? 'Route'} · ${route[2] == null ? 'primary' : `fallback ${Number(route[2]) + 1}`}`; - if (text === 'aqe:default') return 'Agentic QE · default'; - if (/^aqe:fallback:\d+$/.test(text)) return `Agentic QE · fallback ${Number(text.split(':').at(-1)) + 1}`; - if (text.startsWith('aqe:agent:')) return `Agentic QE · ${publicActivity(text.slice('aqe:agent:'.length), key) ?? 'override'}`; - if (/^ruflo:candidate:\d+$/.test(text)) return `Ruflo · candidate ${Number(text.split(':').at(-1)) + 1}`; - if (/^integration:\d+$/.test(text)) return `Integration · binding ${Number(text.split(':').at(-1)) + 1}`; - return 'Configured consumer'; + return routeConsumerLabel(text, key) + ?? (text === 'aqe:default' ? 'Agentic QE · default' : null) + ?? ordinalConsumerLabel(text, /^aqe:fallback:\d+$/, 'Agentic QE · fallback') + ?? agentOverrideConsumerLabel(text, key) + ?? ordinalConsumerLabel(text, /^ruflo:candidate:\d+$/, 'Ruflo · candidate') + ?? ordinalConsumerLabel(text, /^integration:\d+$/, 'Integration · binding') + ?? 'Configured consumer'; } function bindingRole(consumer) { @@ -429,15 +514,22 @@ function bindingRole(consumer) { return route[1] == null ? 'primary' : `fallback ${Number(route[1]) + 1}`; } +function bindingPublishedPricing(configured, effective, provider, linkedModel) { + if (linkedModel || !(configured || effective)) return null; + const published = priceFor(configured ?? effective, provider); + if (!published.matched) return null; + const sourceUrl = published.provider === 'openai' ? openAiModelDocumentation(configured ?? effective) : null; + return { + basis: 'per-million-tokens', input: published.in, output: published.out, currency: 'USD', + effectiveAt: null, source: sourceUrl ? 'OpenAI API model documentation' : 'published API list-price table', + sourceUrl, asOf: PRICES_AS_OF, matched: true, + }; +} + function sanitizeBinding(binding, key, linkedModel) { const configured = ownerVisibleModelText(binding.configured); const effective = ownerVisibleModelText(binding.effective); - const modelName = linkedModel?.displayName ?? configured ?? effective ?? 'Model not pinned'; const provider = linkedModel?.servingProvider ?? ownerVisibleModelText(binding.provider); - const published = !linkedModel && (configured || effective) - ? priceFor(configured ?? effective, provider) : null; - const publishedModel = configured ?? effective; - const sourceUrl = published?.provider === 'openai' ? openAiModelDocumentation(publishedModel) : null; return { ...binding, id: privateLabel('binding', binding.id, key), @@ -448,17 +540,13 @@ function sanitizeBinding(binding, key, linkedModel) { configured, effective, evidenceRefs: (binding.evidenceRefs ?? []).map((ref) => privateLabel('evidence', ref, key)), - modelName, + modelName: linkedModel?.displayName ?? configured ?? effective ?? 'Model not pinned', selector: linkedModel?.selector ?? configured ?? effective, modelProvider: provider, role: bindingRole(binding.consumer), lifecycle: linkedModel?.lifecycle?.state ?? 'unknown', capabilities: linkedModel?.capabilities ?? {}, - pricing: linkedModel?.pricing ?? (published?.matched ? { - basis: 'per-million-tokens', input: published.in, output: published.out, currency: 'USD', - effectiveAt: null, source: sourceUrl ? 'OpenAI API model documentation' : 'published API list-price table', - sourceUrl, asOf: PRICES_AS_OF, matched: true, - } : null), + pricing: linkedModel?.pricing ?? bindingPublishedPricing(configured, effective, provider, linkedModel), // Snapshot evidence says that use was observed, but its capturedAt is the // refresh time—not the invocation time. Windowed usage joins the actual // session timestamp onto summary bindings below. @@ -482,52 +570,71 @@ function sessionRange(session) { }; } +function matchModelForSession(exact, projected, host, modelId, provider) { + const candidates = exact.models.map((model, index) => ({ model, projected: projected.models[index] })) + .filter(({ model }) => model.key.host === host && model.key.modelId === modelId); + return candidates.find(({ projected: item }) => item?.privacyClass === 'public-catalog') + ?? candidates.find(({ model }) => model.key.provider === provider) + ?? candidates[0]; +} + +function newObservedGroupRow(host, provider, modelId, visible) { + return { + host: PUBLIC_HOSTS.has(host) ? host : 'unknown', + modelName: visible?.humanName ?? visible?.displayName ?? modelId, + selector: visible?.selector ?? modelId, + modelProvider: provider ?? visible?.servingProvider ?? visible?.publisher ?? null, + sessions: 0, responses: 0, tokens: 0, apiEquivalentCost: 0, + firstUsed: null, lastUsed: null, + }; +} + +function observedGroupRow(groups, exact, projected, host, provider, modelId) { + const groupKey = `${host}\0${provider ?? ''}\0${modelId}`; + let row = groups.get(groupKey); + if (!row) { + const match = matchModelForSession(exact, projected, host, modelId, provider); + row = newObservedGroupRow(host, provider, modelId, match?.projected); + groups.set(groupKey, row); + } + return row; +} + +function accumulateSession(row, session, range) { + row.sessions++; + row.responses += finite(session.responses); + row.tokens += finite(session.tokens); + row.apiEquivalentCost += finite(session.cost); + if (range.firstUsed && (!row.firstUsed || range.firstUsed < row.firstUsed)) row.firstUsed = range.firstUsed; + if (range.lastUsed && (!row.lastUsed || range.lastUsed > row.lastUsed)) row.lastUsed = range.lastUsed; +} + +function foldSessionIntoGroups(groups, exact, projected, session) { + const host = boundedPublicText(session?.host, 64); + const provider = boundedPublicText(session?.provider, 128); + if (!host || !Array.isArray(session?.models)) return; + const range = sessionRange(session); + for (const rawModelId of session.models) { + const modelId = ownerVisibleModelText(rawModelId); + if (!modelId) continue; + accumulateSession(observedGroupRow(groups, exact, projected, host, provider, modelId), session, range); + } +} + +function sortObservedGroups(a, b) { + return String(b.lastUsed ?? '').localeCompare(String(a.lastUsed ?? '')) + || String(a.modelName).localeCompare(String(b.modelName), 'en-US', { sensitivity: 'base' }); +} + /** Project only aggregate model-use facts; session ids, titles and projects never cross this boundary. */ function observedWindow(exact, projected, usage, days) { const sessions = Array.isArray(usage?.sessions) ? usage.sessions : []; const groups = new Map(); - for (const session of sessions) { - const host = boundedPublicText(session?.host, 64); - const provider = boundedPublicText(session?.provider, 128); - if (!host || !Array.isArray(session?.models)) continue; - const range = sessionRange(session); - for (const rawModelId of session.models) { - const modelId = ownerVisibleModelText(rawModelId); - if (!modelId) continue; - const groupKey = `${host}\0${provider ?? ''}\0${modelId}`; - let row = groups.get(groupKey); - if (!row) { - const candidates = exact.models.map((model, index) => ({ model, projected: projected.models[index] })) - .filter(({ model }) => model.key.host === host && model.key.modelId === modelId); - const match = candidates.find(({ projected: item }) => item?.privacyClass === 'public-catalog') - ?? candidates.find(({ model }) => model.key.provider === provider) - ?? candidates[0]; - const visible = match?.projected; - row = { - host: PUBLIC_HOSTS.has(host) ? host : 'unknown', - modelName: visible?.humanName ?? visible?.displayName ?? modelId, - selector: visible?.selector ?? modelId, - modelProvider: provider ?? visible?.servingProvider ?? visible?.publisher ?? null, - sessions: 0, responses: 0, tokens: 0, apiEquivalentCost: 0, - firstUsed: null, lastUsed: null, - }; - groups.set(groupKey, row); - } - row.sessions++; - row.responses += finite(session.responses); - row.tokens += finite(session.tokens); - row.apiEquivalentCost += finite(session.cost); - if (range.firstUsed && (!row.firstUsed || range.firstUsed < row.firstUsed)) row.firstUsed = range.firstUsed; - if (range.lastUsed && (!row.lastUsed || range.lastUsed > row.lastUsed)) row.lastUsed = range.lastUsed; - } - } + for (const session of sessions) foldSessionIntoGroups(groups, exact, projected, session); return { days, status: usage?.unavailable === true ? 'unavailable' : 'complete', generatedAt: typeof usage?.generatedAt === 'string' ? usage.generatedAt : null, - models: [...groups.values()].sort((a, b) => ( - String(b.lastUsed ?? '').localeCompare(String(a.lastUsed ?? '')) - || String(a.modelName).localeCompare(String(b.modelName), 'en-US', { sensitivity: 'base' }) - )), + models: [...groups.values()].sort(sortObservedGroups), }; } @@ -574,48 +681,64 @@ function humanField(value) { return safeState(value).replace(/([a-z0-9])([A-Z])/g, '$1 $2').replaceAll('-', ' ').toLowerCase(); } +function changeDetailLifecycle(change) { + const before = safeState(change.before?.state); + const after = safeState(change.after?.state); + const replacement = ownerVisibleModelText(change.after?.replacement); + return `Lifecycle ${before} → ${after}${replacement ? `; replacement ${replacement}` : ''}.`; +} + +function changeDetailVisibility(change) { + return `Catalog visibility ${safeState(change.before)} → ${safeState(change.after)}.`; +} + +function changeDetailCapability(change) { + const field = humanField(change.after?.field ?? change.before?.field); + return `${field === 'unknown' ? 'A reported capability' : `Reported ${field} support`} changed.`; +} + +function changeDetailVariant(change) { + const field = humanField(change.after?.field ?? change.before?.field); + return `${field === 'unknown' ? 'Reported model metadata' : `Reported ${field}`} changed.`; +} + +/** One formatter per change `kind`, mirroring CHANGE_LABELS' lookup-table shape above. */ +const CHANGE_DETAIL_BY_KIND = Object.freeze({ + 'model-added': () => 'Appeared in the latest inventory.', + 'model-missing': () => 'Not reported by the latest complete source; confirmation is pending.', + 'model-removed': () => 'No longer reported after repeated complete refreshes.', + 'lifecycle-changed': changeDetailLifecycle, + 'visibility-changed': changeDetailVisibility, + 'alias-target-changed': () => 'A configured alias now resolves to a different model.', + 'capability-changed': changeDetailCapability, + 'reasoning-changed': () => 'The reported reasoning options changed.', + 'context-changed': () => 'The reported context window changed.', + 'variant-changed': changeDetailVariant, + 'digest-changed': () => 'The installed model build changed; private digests remain hidden.', + 'pricing-changed': () => 'The published API rate changed.', + 'edges-changed': () => 'Compatibility or migration guidance changed.', +}); + function changeDetail(change) { - if (change.kind === 'model-added') return 'Appeared in the latest inventory.'; - if (change.kind === 'model-missing') return 'Not reported by the latest complete source; confirmation is pending.'; - if (change.kind === 'model-removed') return 'No longer reported after repeated complete refreshes.'; - if (change.kind === 'lifecycle-changed') { - const before = safeState(change.before?.state); - const after = safeState(change.after?.state); - const replacement = ownerVisibleModelText(change.after?.replacement); - return `Lifecycle ${before} → ${after}${replacement ? `; replacement ${replacement}` : ''}.`; - } - if (change.kind === 'visibility-changed') { - return `Catalog visibility ${safeState(change.before)} → ${safeState(change.after)}.`; - } - if (change.kind === 'alias-target-changed') return 'A configured alias now resolves to a different model.'; - if (change.kind === 'capability-changed') { - const field = humanField(change.after?.field ?? change.before?.field); - return `${field === 'unknown' ? 'A reported capability' : `Reported ${field} support`} changed.`; - } - if (change.kind === 'reasoning-changed') return 'The reported reasoning options changed.'; - if (change.kind === 'context-changed') return 'The reported context window changed.'; - if (change.kind === 'variant-changed') { - const field = humanField(change.after?.field ?? change.before?.field); - return `${field === 'unknown' ? 'Reported model metadata' : `Reported ${field}`} changed.`; - } - if (change.kind === 'digest-changed') return 'The installed model build changed; private digests remain hidden.'; - if (change.kind === 'pricing-changed') return 'The published API rate changed.'; - if (change.kind === 'edges-changed') return 'Compatibility or migration guidance changed.'; - return 'A model inventory fact changed.'; + const formatter = CHANGE_DETAIL_BY_KIND[change.kind]; + return formatter ? formatter(change) : 'A model inventory fact changed.'; } -function sanitizeChange(change, key, linkedModel, detectedAt) { - const rawKey = [change.after, change.before].find((value) => value && typeof value === 'object' +function changeRawSubject(change) { + return [change.after, change.before].find((value) => value && typeof value === 'object' && typeof value.modelId === 'string'); - const selector = linkedModel?.selector ?? ownerVisibleModelText(rawKey?.modelId); - const modelName = linkedModel?.displayName ?? selector ?? 'Model not recorded'; +} + +function sanitizeChange(change, key, linkedModel, detectedAt) { + const rawSubject = changeRawSubject(change); + const selector = linkedModel?.selector ?? ownerVisibleModelText(rawSubject?.modelId); return { kind: change.kind, label: changeLabel(change.kind), - modelName, + modelName: linkedModel?.displayName ?? selector ?? 'Model not recorded', selector, - modelProvider: linkedModel?.servingProvider ?? ownerVisibleModelText(rawKey?.provider), - host: linkedModel?.host ?? publicHost(rawKey?.host, key), + modelProvider: linkedModel?.servingProvider ?? ownerVisibleModelText(rawSubject?.provider), + host: linkedModel?.host ?? publicHost(rawSubject?.host, key), detail: changeDetail(change), severity: change.severity, provisional: change.provisional === true, @@ -623,6 +746,60 @@ function sanitizeChange(change, key, linkedModel, detectedAt) { }; } +function sanitizedSourceAttention(item, sourceById, key) { + return { ...item, subject: sourceById.get(item.subject)?.id ?? privateLabel('source', item.subject, key) }; +} + +function affectedRoutesFor(exactModel, exactBindings, bindingById) { + if (!exactModel) return []; + return exactBindings.filter((binding) => binding.host === exactModel.key.host + && [binding.effective, binding.configured].includes(exactModel.key.modelId)) + .map((binding) => bindingById.get(binding.id)).filter(Boolean) + .map((binding) => ({ activity: binding.activity, consumer: binding.consumer, role: binding.role })); +} + +function migrationAction(activity, host, replacement) { + return activity && PUBLIC_ACTIVITIES.has(activity) && PUBLIC_HOSTS.has(host) + ? `ak models plan --activity ${activity} --to ${host}:${replacement}` : 'ak models plan'; +} + +function sanitizedMigrationAttention(item, ctx) { + const { + modelByIdentity, exactModelByIdentity, exact, bindingById, key, + } = ctx; + const model = modelByIdentity.get(item.subject); + const exactModel = exactModelByIdentity.get(item.subject); + const affectedRoutes = affectedRoutesFor(exactModel, exact.bindings, bindingById); + const activity = affectedRoutes[0]?.activity; + const replacement = model?.lifecycle.replacementName ?? 'replacement not recorded'; + const host = model?.host; + return { + ...item, + subject: model?.identity ?? privateLabel('identity', item.subject, key), + currentModel: model?.displayName ?? model?.selector ?? 'Model not recorded', + replacementModel: replacement, + affectedRoutes, + documentationUrl: lifecycleNoticeUrl(exactModel?.lifecycle.notice), + action: migrationAction(activity, host, replacement), + reason: `${model?.displayName ?? model?.selector ?? 'Model'} is ${model?.lifecycle.state ?? 'unknown'}; recommended replacement ${replacement}`, + }; +} + +function sanitizedConsumerAttention(item, bindingById, key) { + return { ...item, subject: bindingById.get(item.subject)?.id ?? privateLabel('binding', item.subject, key) }; +} + +function sanitizedAliasAttention(item, changeBySubject, key) { + return { ...item, subject: changeBySubject.get(item.subject)?.modelName ?? privateLabel('identity', item.subject, key) }; +} + +function sanitizeAttentionItem(item, ctx) { + if (item.kind === 'source') return sanitizedSourceAttention(item, ctx.sourceById, ctx.key); + if (item.kind === 'migration') return sanitizedMigrationAttention(item, ctx); + if (item.kind === 'consumer') return sanitizedConsumerAttention(item, ctx.bindingById, ctx.key); + return sanitizedAliasAttention(item, ctx.changeBySubject, ctx.key); +} + /** * Project exact CLI evidence into an owner-visible Dashboard contract. * The caller must supply the already-existing per-install key; this function @@ -658,41 +835,10 @@ export function createDashboardModelReadModel(snapshotValue, options = {}) { const bindingById = new Map(exact.bindings.map((binding, index) => [binding.id, bindings[index]])); const sourceById = new Map(exact.sources.map((source, index) => [source.id, sources[index]])); const changeBySubject = new Map(exact.changes.map((change, index) => [change.subject, changes[index]])); - const attention = exact.attention.map((item) => { - if (item.kind === 'source') { - return { ...item, subject: sourceById.get(item.subject)?.id ?? privateLabel('source', item.subject, key) }; - } - if (item.kind === 'migration') { - const model = modelByIdentity.get(item.subject); - const exactModel = exactModelByIdentity.get(item.subject); - const affectedRoutes = exact.bindings.filter((binding) => exactModel - && binding.host === exactModel.key.host - && [binding.effective, binding.configured].includes(exactModel.key.modelId)) - .map((binding) => bindingById.get(binding.id)).filter(Boolean) - .map((binding) => ({ - activity: binding.activity, consumer: binding.consumer, role: binding.role, - })); - const activity = affectedRoutes[0]?.activity; - const replacement = model?.lifecycle.replacementName ?? 'replacement not recorded'; - const host = model?.host; - return { - ...item, - subject: model?.identity ?? privateLabel('identity', item.subject, key), - currentModel: model?.displayName ?? model?.selector ?? 'Model not recorded', - replacementModel: replacement, - affectedRoutes, - documentationUrl: lifecycleNoticeUrl(exactModel?.lifecycle.notice), - action: activity && PUBLIC_ACTIVITIES.has(activity) && PUBLIC_HOSTS.has(host) - ? `ak models plan --activity ${activity} --to ${host}:${replacement}` : 'ak models plan', - reason: `${model?.displayName ?? model?.selector ?? 'Model'} is ${model?.lifecycle.state ?? 'unknown'}; recommended replacement ${replacement}`, - }; - } - if (item.kind === 'consumer') { - return { ...item, subject: bindingById.get(item.subject)?.id ?? privateLabel('binding', item.subject, key) }; - } - return { ...item, subject: changeBySubject.get(item.subject)?.modelName - ?? privateLabel('identity', item.subject, key) }; - }); + const attentionCtx = { + modelByIdentity, exactModelByIdentity, exact, bindingById, sourceById, changeBySubject, key, + }; + const attention = exact.attention.map((item) => sanitizeAttentionItem(item, attentionCtx)); return immutable({ ...exact, snapshotId: privateLabel('snapshot', exact.snapshotId, key), @@ -710,6 +856,31 @@ export function createDashboardModelReadModel(snapshotValue, options = {}) { }); } +function timestampIso(entry) { + if (typeof entry !== 'string') return null; + const parsed = Date.parse(entry); + return Number.isFinite(parsed) ? new Date(parsed).toISOString() : null; +} + +function dashboardHistoryEntries(history, privateKey) { + return (Array.isArray(history) ? history : []).slice(0, 32).flatMap((entry) => { + const capturedAt = timestampIso(entry?.capturedAt); + const snapshotId = privateLabel('snapshot', entry?.snapshotId, privateKey); + return capturedAt && snapshotId ? [{ snapshotId, capturedAt }] : []; + }); +} + +function dashboardComparisonBlock(comparison, privateKey) { + if (!comparison) return undefined; + return { + baseline: privateLabel('snapshot', comparison.baseline, privateKey), + latest: privateLabel('snapshot', comparison.latest, privateKey), + comparable: comparison.comparable === true, + diagnostics: (Array.isArray(comparison.diagnostics) ? comparison.diagnostics : []) + .slice(0, 32).map((item) => privateLabel('diagnostic', item, privateKey)), + }; +} + /** * Sanitize a complete `/api/models` payload, including history identifiers. * @param {any} value @@ -725,27 +896,11 @@ export function createDashboardModelPayload(value, { key, usage, days = 14 } = { let snapshot = createDashboardModelReadModel(exact, { key, changes }); const window = observedWindow(exact, snapshot, usage, days); if (usage) snapshot = joinWindowedRouteUse(snapshot, window); - const timestamp = (entry) => { - if (typeof entry !== 'string') return null; - const parsed = Date.parse(entry); - return Number.isFinite(parsed) ? new Date(parsed).toISOString() : null; - }; return immutable({ status: ['cached', 'complete', 'partial', 'stale'].includes(value.status) ? value.status : 'cached', snapshot, ...(usage ? { observedWindow: window } : {}), - history: (Array.isArray(value.history) ? value.history : []).slice(0, 32) - .flatMap((entry) => { - const capturedAt = timestamp(entry?.capturedAt); - const snapshotId = privateLabel('snapshot', entry?.snapshotId, privateKey); - return capturedAt && snapshotId ? [{ snapshotId, capturedAt }] : []; - }), - comparison: value.comparison ? { - baseline: privateLabel('snapshot', value.comparison.baseline, privateKey), - latest: privateLabel('snapshot', value.comparison.latest, privateKey), - comparable: value.comparison.comparable === true, - diagnostics: (Array.isArray(value.comparison.diagnostics) ? value.comparison.diagnostics : []) - .slice(0, 32).map((item) => privateLabel('diagnostic', item, privateKey)), - } : undefined, + history: dashboardHistoryEntries(value.history, privateKey), + comparison: dashboardComparisonBlock(value.comparison, privateKey), }); } diff --git a/src/lib/model-inventory/refresh.mjs b/src/lib/model-inventory/refresh.mjs index 9178d47..aef325a 100644 --- a/src/lib/model-inventory/refresh.mjs +++ b/src/lib/model-inventory/refresh.mjs @@ -245,39 +245,70 @@ function applyBindings(models, bindings, capturedAt, fingerprint) { return result; } -export function composeModelSnapshot(collection, { - scope = {}, scopeKey, capturedAt = collection?.generatedAt ?? new Date().toISOString(), -} = /** @type {any} */ ({})) { - const discoveryResults = Object.values(collection?.discovery?.results ?? {}); - const profileFingerprints = Object.fromEntries(discoveryResults +function discoveryProfileFingerprints(discoveryResults) { + return Object.fromEntries(discoveryResults .filter((result) => result?.source?.scopeFingerprint) .map((result) => [result.source.owner ?? result.source.id, result.source.scopeFingerprint])); - const hosts = Object.keys(collection?.discovery?.results ?? {}).sort(); - const fingerprint = scopeFingerprint('inventory', { ...scope, hosts: hosts.join(',') }, scopeKey); - const sources = [ +} + +function snapshotSources(discoveryResults, observedSource, fingerprint) { + return [ ...discoveryResults.flatMap(({ source, sources: additional = [] }) => [source, ...additional]), - collection?.observed?.source, + observedSource, ] .filter(Boolean) .map((source) => ({ ...source, scopeFingerprint: fingerprint, scopeId: fingerprint })); - const discoveredAndObserved = mergeModels([ +} + +function discoveredAndObservedModels(discoveryResults, observedModels, fingerprint) { + return mergeModels([ ...discoveryResults.flatMap(({ models }) => models ?? []), - ...(collection?.observed?.models ?? []), + ...(observedModels ?? []), ].map((record) => rescopeRecord(record, fingerprint))); - const bindings = collection?.bindings?.bindings ?? []; - const models = applyBindings(discoveredAndObserved, bindings, capturedAt, fingerprint); +} + +function snapshotDiagnosticCodes(discoveryResults, collection) { const diagnostics = [ - ...discoveryResults.flatMap(({ diagnostics }) => diagnostics ?? []), + ...discoveryResults.flatMap(({ diagnostics: entryDiagnostics }) => entryDiagnostics ?? []), ...(collection?.observed?.diagnostics ?? []), ...(collection?.bindings?.diagnostics ?? []), - ].map((entry) => typeof entry === 'string' ? entry : entry.code).filter(Boolean); - const digestInput = JSON.stringify({ capturedAt, fingerprint, sources, models, bindings }); + ].map((entry) => (typeof entry === 'string' ? entry : entry.code)).filter(Boolean); + return [...new Set(diagnostics)]; +} + +function modelSnapshotId({ + capturedAt, fingerprint, sources, models, bindings, +}) { + const digestInput = JSON.stringify({ + capturedAt, fingerprint, sources, models, bindings, + }); + return `models:${createHash('sha256').update(digestInput).digest('hex').slice(0, 20)}`; +} + +export function composeModelSnapshot(collection, { + scope = {}, scopeKey, capturedAt = collection?.generatedAt ?? new Date().toISOString(), +} = /** @type {any} */ ({})) { + const discoveryResults = Object.values(collection?.discovery?.results ?? {}); + const profileFingerprints = discoveryProfileFingerprints(discoveryResults); + const hosts = Object.keys(collection?.discovery?.results ?? {}).sort(); + const fingerprint = scopeFingerprint('inventory', { ...scope, hosts: hosts.join(',') }, scopeKey); + const sources = snapshotSources(discoveryResults, collection?.observed?.source, fingerprint); + const discoveredAndObserved = discoveredAndObservedModels( + discoveryResults, collection?.observed?.models, fingerprint, + ); + const bindings = collection?.bindings?.bindings ?? []; + const models = applyBindings(discoveredAndObserved, bindings, capturedAt, fingerprint); + const diagnostics = snapshotDiagnosticCodes(discoveryResults, collection); return normalizeSnapshot({ schemaVersion: MODEL_INVENTORY_SCHEMA_VERSION, - snapshotId: `models:${createHash('sha256').update(digestInput).digest('hex').slice(0, 20)}`, + snapshotId: modelSnapshotId({ + capturedAt, fingerprint, sources, models, bindings, + }), capturedAt, - scope: { fingerprint, machine: null, project: null, hosts, profileFingerprints }, - sources, models, bindings, changes: [], opportunities: [], diagnostics: [...new Set(diagnostics)], + scope: { + fingerprint, machine: null, project: null, hosts, profileFingerprints, + }, + sources, models, bindings, changes: [], opportunities: [], diagnostics, }); } diff --git a/src/lib/opencode-agents.mjs b/src/lib/opencode-agents.mjs new file mode 100644 index 0000000..3db627e --- /dev/null +++ b/src/lib/opencode-agents.mjs @@ -0,0 +1,489 @@ +// opencode-agents.mjs — ruflo catalog resolution + the Claude Code agent .md → +// OpenCode subagent .md conversion/sync/status pipeline. Split out of +// opencode.mjs (ADR-0037's file-size gate) — behavior and export names are +// unchanged; opencode.mjs re-exports the externally-consumed names so no +// import path elsewhere in the repo needed to change. +import fs from 'node:fs'; +import path from 'node:path'; +import { readJson } from './settings.mjs'; +import * as paths from './paths.mjs'; +import { deepEqual, contentHash, hasReceiptValue, receiptMatches, asReceiptMap } from './opencode-receipts.mjs'; + +/** @typedef {{ kind: string, root: string, id: string, hasPlugins: boolean, hasPlatformSkill: boolean }} CatalogSource */ + +// ── ruflo catalog source (agents + skills) ────────────────────────────────── + +/** Resolve where ruflo's agent/skill catalog comes from. Order: explicit + * override (kit.json integrations.ownership.opencode.catalogDir) → + * $RUFLO_REPO → the claude + * marketplace clone (full repo mirror, auto-updated by claude) → the + * published @claude-flow/cli package (subset: ADR-128 agents + core skills) + * → the nested copy under global ruflo/node_modules (the layout a plain + * `npm i -g ruflo` actually produces). Candidates are LAZY thunks: the + * npm-root lookups spawn `npm root -g` (cached per process), so evaluating + * them only when earlier candidates miss keeps status probes spawn-free on + * marketplace machines. Returns {kind, root, id, hasPlugins, hasPlatformSkill} + * or null. + * @param {{ override?: string }} [opts] + * @returns {CatalogSource|null} */ +export function catalogSource({ override } = {}) { + const candidates = []; + if (override) candidates.push(() => ({ kind: 'override', root: override })); + if (process.env.RUFLO_REPO) candidates.push(() => ({ kind: 'env', root: process.env.RUFLO_REPO })); + candidates.push(() => ({ kind: 'marketplace', root: paths.rufloMarketplaceRoot() })); + candidates.push(() => ({ kind: 'npm', root: paths.rufloCliPkgRoot() })); + candidates.push(() => ({ kind: 'npm-nested', root: path.join(paths.rufloNodeModules(), '@claude-flow', 'cli') })); + for (const thunk of candidates) { + const c = thunk(); + if (!c.root || !fs.existsSync(path.join(c.root, '.claude', 'agents'))) continue; + let version = null; + try { version = JSON.parse(fs.readFileSync(path.join(c.root, 'package.json'), 'utf8')).version; } catch { /* unversioned source */ } + return { + ...c, + id: `${c.kind}@${version ?? 'unversioned'}`, + hasPlugins: fs.existsSync(path.join(c.root, 'plugins')), + hasPlatformSkill: fs.existsSync(path.join(c.root, 'SKILL.md')), + }; + } + return null; +} + +/** skills.paths entries for a catalog source (existing dirs only). + * @param {CatalogSource|null} source */ +export function skillPathsFor(source) { + if (!source) return []; + const out = [path.join(source.root, '.claude', 'skills')]; + if (source.hasPlugins) out.push(path.join(source.root, 'plugins')); + return out.filter((p) => fs.existsSync(p)); +} + +// ── agent conversion (Claude Code agent .md → opencode subagent .md) ───────── + +/** Ownership markers on generated agent files — the current ak marker plus the + * legacy standalone-script marker, so the one-time script's output is adopted + * (removed/replaced) rather than orphaned. */ +const AGENT_MARKERS = ['generated-by: agentic-kit', 'generated-by: sync-ruflo-agents.mjs']; +export const STAMP_FILE = '.ak-agents-stamp.json'; + +function* walkMd(dir) { + for (const e of fs.readdirSync(dir, { withFileTypes: true }) + .sort((left, right) => left.name.localeCompare(right.name))) { + const p = path.join(dir, e.name); + if (e.isDirectory()) yield* walkMd(p); + else if (e.isFile() && e.name.endsWith('.md')) yield p; + } +} + +/** Minimal YAML frontmatter reader: scalar fields + block scalars + * (description: | / >). A block scalar's content is every following line that + * is INDENTED, with blank lines allowed inside — terminating at the first + * blank (they'd otherwise truncate multi-paragraph descriptions and leak the + * remainder into mis-parsed fields). */ +export function parseFrontmatter(text) { + const m = text.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n?([\s\S]*)$/); + if (!m) return null; + const [, fm, body] = m; + const fields = {}; + const lines = fm.split(/\r?\n/); + for (let i = 0; i < lines.length; i++) { + const km = lines[i].match(/^([A-Za-z_][\w-]*):\s*(.*)$/); + if (!km) continue; + const [, key, raw] = km; + if (/^[>|]-?$/.test(raw) && i + 1 < lines.length) { + const buf = []; + while (i + 1 < lines.length && (lines[i + 1].trim() === '' || /^\s+\S/.test(lines[i + 1]))) { + const t = lines[++i].trim(); + if (t) buf.push(t); + } + fields[key] = buf.join(' ').trim(); + } else { + fields[key] = raw.replace(/^["']|["']$/g, '').trim(); + } + } + return { fields, body }; +} + +const collapse = (s) => String(s ?? '').replace(/\s+/g, ' ').trim(); + +const lazyGatewayCall = (family, name) => + `\`${family}_call\` with \`name=${JSON.stringify(name)}\` and \`arguments_json\` set to one JSON object string`; + +const directOpenCodeReferences = (body) => String(body) + .replace(/mcp__(?:claude-flow|claude_flow|ruflo)__([A-Za-z0-9_./:*-]+)/g, 'claude-flow_$1') + .replace(/mcp__(?:agentic-qe|agentic_qe)__([A-Za-z0-9_./:*-]+)/g, 'agentic-qe_$1'); + +/** Rewrite tool-name references inside an OpenCode-only generated agent so + * the instructions use the lazy gateway that is actually advertised. The + * Claude/Ruflo source file is never changed. Families without a managed + * gateway retain their direct OpenCode tool spelling. + * @param {string} body @param {{ruflo?:boolean,aqe?:boolean}} capabilities */ +export function rewriteAgentGatewayReferences(body, capabilities = {}) { + let result = directOpenCodeReferences(body); + if (capabilities.ruflo) { + result = result + .replace(/\b(?:claude-flow|claude_flow)_\*/g, + () => 'the Ruflo operation selected with `ak_ruflo_search`, then invoked through `ak_ruflo_call`') + .replace(/\b(?:claude-flow|claude_flow)_([A-Za-z0-9_./:-]+)/g, + (_match, name) => lazyGatewayCall('ruflo', name)); + } + if (capabilities.aqe) { + result = result + .replace(/\b(?:agentic-qe|agentic_qe)_\*/g, + () => 'the Agentic QE operation selected with `ak_aqe_search`, then invoked through `ak_aqe_call`') + .replace(/\b(?:agentic-qe|agentic_qe)_([A-Za-z0-9_./:-]+)/g, + (_match, name) => lazyGatewayCall('aqe', name)); + } + return result; +} + +/** Convert every agent under /.claude/agents into opencode form: + * frontmatter → {description, mode: subagent} (Claude's `tools:` string list + * is dropped — OpenCode applies the subagent's permissions plus inherited + * parent/session deny rules); body MCP refs + * rewritten across all catalogue spellings. Lazy-gateway conversion emits + * ak_ruflo_call/ak_aqe_call guidance; direct fallback conversion emits OpenCode's + * direct tool spelling. + * basename collisions across category dirs get the parent dir prefixed. The + * description is emitted as a JSON double-quoted scalar (valid YAML 1.2 — + * unquoted values containing ': ' or '#' would corrupt the frontmatter). + * Pure (returns content, writes nothing). + * @param {string} srcRoot */ +export function convertAgents(srcRoot, { gatewayCapabilities = {} } = {}) { + const srcDir = path.join(srcRoot, '.claude', 'agents'); + const agents = []; + let scanned = 0; + for (const file of walkMd(srcDir)) { + scanned++; + const parsed = parseFrontmatter(fs.readFileSync(file, 'utf8')); + if (!parsed) continue; + // documentation masquerading as an agent (e.g. MIGRATION_SUMMARY.md) + if (collapse(parsed.fields.type).toLowerCase() === 'documentation') continue; + const description = collapse(parsed.fields.description); + if (!description) continue; + const rel = path.relative(srcDir, file); + const dir = path.dirname(rel) === '.' ? null : path.dirname(rel).split(path.sep)[0]; + agents.push({ + base: path.basename(file, '.md'), + dir, + description, + body: rewriteAgentGatewayReferences(parsed.body, gatewayCapabilities), + }); + } + const seen = new Set(); + let renamed = 0; + for (const a of agents) { + let name = a.base; + if (seen.has(name)) { name = a.dir ? `${a.dir}-${a.base}` : `${a.base}-x`; renamed++; } + let n = 2; + while (seen.has(name)) name = `${a.dir ?? 'agent'}-${a.base}-${n++}`; + seen.add(name); + a.name = name; + a.content = `---\ndescription: ${JSON.stringify(a.description)}\nmode: subagent\n---\n\n\n${a.body}`; + } + return { agents, scanned, skipped: scanned - agents.length, renamed }; +} + +export const SPECIALIST_AGENT = { + name: 'ak-specialist', + description: 'Runs one Agentic Kit specialist profile selected lazily with ak_agent_search', + content: `--- +description: "Runs one Agentic Kit specialist profile selected lazily with ak_agent_search" +mode: subagent +--- + + +You are the Agentic Kit specialist dispatcher for stock OpenCode. + +The parent task must begin with \`PROFILE: \`. Call \`ak_agent_load\` with that exact +name before doing any work. Treat the returned receipt-owned profile as your specialist +instructions for the rest of this task. If the profile names an optional dependency that is not +available, report the missing dependency instead of inventing a result. +`, +}; + +export function specialistDispatcherState({ + destDir = paths.opencodeAgentsDir(), receipts = {}, adoptionBlocked = false, +} = {}) { + if (adoptionBlocked) return { available: false, blocked: true }; + const file = 'ak-specialist.md'; + const target = path.join(destDir, file); + if (!fs.existsSync(target)) return { available: true, blocked: false }; + let current; + try { current = fs.readFileSync(target, 'utf8'); } catch { + return { available: false, blocked: true }; + } + if (receiptMatches(current, receipts?.[file])) return { available: true, blocked: false }; + if (hasReceiptValue(receipts?.[file])) return { available: false, blocked: true }; + const adoptable = current === SPECIALIST_AGENT.content && isGeneratedContent(current); + return { available: adoptable, blocked: false }; +} + +function desiredAgentSet(source, gatewayCapabilities, lazyCatalog) { + const converted = convertAgents(source.root, { gatewayCapabilities }); + return { ...converted, agents: lazyCatalog ? [SPECIALIST_AGENT] : converted.agents }; +} + +export function gatewayAgentCatalog(source) { + if (!source) return []; + return convertAgents(source.root, { gatewayCapabilities: {} }).agents.map((agent) => ({ + name: agent.name, + description: agent.description, + body: agent.body, + })); +} + +const isGeneratedContent = (text) => AGENT_MARKERS.some((m) => text.includes(m)); + +/** Write/adopt each desired agent file, skipping any user-owned collision. + * Returns per-run counts plus the exact list of files actually deployed. */ +function deployDesiredAgents(agents, destDir, { dryRun, receiptMap, adoptionBlocked }) { + let userOwned = 0, adopted = 0, written = 0; + const deployed = []; + for (const a of agents) { + const file = `${a.name}.md`; + const p = path.join(destDir, file); + const cur = fs.existsSync(p) ? fs.readFileSync(p, 'utf8') : null; + const priorReceipt = receiptMap[file]; + const hasPriorReceipt = adoptionBlocked || hasReceiptValue(priorReceipt); + const priorOwned = cur !== null && receiptMatches(cur, priorReceipt); + const adoptable = cur === a.content && !hasPriorReceipt && isGeneratedContent(cur); + if (cur !== null && !priorOwned && !adoptable) { userOwned++; continue; } + if (adoptable) adopted++; + deployed.push(file); + if (cur !== a.content) { + written++; + if (!dryRun) fs.writeFileSync(p, a.content); + } + } + return { userOwned, adopted, written, deployed }; +} + +/** Remove receipt-owned generated agent files that are no longer desired. + * Deploy/adopt the dispatcher or complete direct set BEFORE retiring any + * receipt-owned predecessor. A write failure therefore preserves the last + * known-good eager catalogue instead of leaving no executable agent path. */ +function retireStaleGeneratedAgents(destDir, agents, receiptMap, { dryRun, lazyCatalog, deployed }) { + let removed = 0; + const removedFiles = new Set(); + if ((!lazyCatalog || deployed.includes('ak-specialist.md')) && fs.existsSync(destDir)) { + for (const f of fs.readdirSync(destDir).filter((f) => f.endsWith('.md'))) { + const p = path.join(destDir, f); + let owned = false; + try { owned = receiptMatches(fs.readFileSync(p, 'utf8'), receiptMap[f]); } catch { /* leave alone */ } + const wanted = agents.some((a) => `${a.name}.md` === f); + if (owned && !wanted) { + if (!dryRun) fs.rmSync(p); + removedFiles.add(f); + removed++; + } + } + } + return { removed, removedFiles }; +} + +/** Compute + (when ownable) write the agent-set stamp file, returning the + * per-file content hashes and the stamp's own receipt for the caller's + * managed-artifacts ledger. */ +function writeAgentStamp({ destDir, dryRun, source, gatewayCapabilities, lazyCatalog, deployed, agents, stampReceipt, adoptionBlocked }) { + const deployedHashes = Object.fromEntries(deployed.map((f) => { + const agent = agents.find((a) => `${a.name}.md` === f); + return [f, contentHash(agent.content)]; + })); + const gateway = { ruflo: !!gatewayCapabilities.ruflo, aqe: !!gatewayCapabilities.aqe }; + const stamp = { + source: source.id, gateway, lazyCatalog: !!lazyCatalog, + count: deployed.length, files: deployed.sort(), hashes: deployedHashes, + }; + const stampText = JSON.stringify(stamp, null, 2) + '\n'; + const stampPath = path.join(destDir, STAMP_FILE); + const priorStampText = fs.existsSync(stampPath) ? fs.readFileSync(stampPath, 'utf8') : null; + const hasStampReceipt = adoptionBlocked || hasReceiptValue(stampReceipt); + const stampOwned = priorStampText !== null && receiptMatches(priorStampText, stampReceipt); + // The stamp has no marker of its own, so exact bytes are adoptable only when + // every file it lists was independently receipt-owned, newly written, or + // adopted through exact marker-bearing content. + const stampAdoptable = priorStampText === stampText && !hasStampReceipt && deployed.length > 0 + && deployed.every((f) => { + try { + return contentHash(fs.readFileSync(path.join(destDir, f), 'utf8')) === deployedHashes[f]; + } catch { return false; } + }); + const mayWriteStamp = priorStampText === null || stampOwned || stampAdoptable; + if (!dryRun && mayWriteStamp && priorStampText !== stampText) { + fs.writeFileSync(stampPath, stampText); + } + return { deployedHashes, stampReceipt: mayWriteStamp ? contentHash(stampText) : stampReceipt }; +} + +/** Reconcile the converted agent set into the dest dir: rewrite generated + * files, remove stale generated ones (either marker), NEVER overwrite a file + * that carries no generated marker (a user-owned agent with a colliding name + * is preserved and reported). The stamp records the source id + the exact + * generated file list and is only rewritten when the set actually changed + * (no per-run timestamp churn — idempotent-write semantics). + * @param {{ source: CatalogSource|null, destDir?: string, dryRun?: boolean, receipts?:Record, stampReceipt?:string|null, adoptionBlocked?:boolean, gatewayCapabilities?:{ruflo?:boolean,aqe?:boolean}, lazyCatalog?:boolean }} opts */ +export function syncAgents({ + source, destDir = paths.opencodeAgentsDir(), dryRun = false, receipts = {}, stampReceipt = null, + adoptionBlocked = false, gatewayCapabilities = {}, lazyCatalog = false, +}) { + if (!source) return { ok: false, changed: false, detail: 'no ruflo catalog source (marketplace clone or @claude-flow/cli) found' }; + const receiptMap = asReceiptMap(receipts); + const { agents, scanned, skipped, renamed } = desiredAgentSet( + source, gatewayCapabilities, lazyCatalog, + ); + if (!dryRun) fs.mkdirSync(destDir, { recursive: true }); + const { userOwned, adopted, written, deployed } = deployDesiredAgents( + agents, destDir, { dryRun, receiptMap, adoptionBlocked }, + ); + const { removed, removedFiles } = retireStaleGeneratedAgents( + destDir, agents, receiptMap, { dryRun, lazyCatalog, deployed }, + ); + const changed = written > 0 || removed > 0; + // The stamp records what was ACTUALLY deployed (a user-owned file occupying + // a slot is never in it) — otherwise status would diverge forever. + // Preserve a non-null mismatched receipt while its file still exists. If it + // were dropped, a later pass could mistake the resulting absence for a + // pre-receipts install and launder an edited file back into ak ownership. + const nextReceipts = Object.fromEntries(Object.entries(receiptMap).filter(([f]) => ( + !removedFiles.has(f) && fs.existsSync(path.join(destDir, f)) + ))); + const { deployedHashes, stampReceipt: nextStampReceipt } = writeAgentStamp({ + destDir, dryRun, source, gatewayCapabilities, lazyCatalog, deployed, agents, stampReceipt, adoptionBlocked, + }); + Object.assign(nextReceipts, deployedHashes); + return { + ok: !lazyCatalog || deployed.includes('ak-specialist.md'), + changed, + receipts: nextReceipts, + stampReceipt: nextStampReceipt, + adopted, + detail: `${agents.length} ${lazyCatalog ? 'lazy dispatcher agent' : 'agents'} from ${source.id} (${written} written, ${removed} removed, ${skipped} skipped, ${renamed} collision-renamed${adopted ? `, ${adopted} adopted` : ''}${userOwned ? `, ${userOwned} user-owned preserved` : ''}; scanned ${scanned})`, + }; +} + +function readAgentStamp(destDir) { + const stampPath = path.join(destDir, STAMP_FILE); + return { + stamp: readJson(stampPath, null), + stampText: fs.existsSync(stampPath) ? fs.readFileSync(stampPath, 'utf8') : null, + }; +} + +function countGeneratedAgentFiles(destDir) { + if (!fs.existsSync(destDir)) return 0; + let count = 0; + for (const f of fs.readdirSync(destDir).filter((f) => f.endsWith('.md'))) { + try { if (isGeneratedContent(fs.readFileSync(path.join(destDir, f), 'utf8'))) count++; } catch { /* skip */ } + } + return count; +} + +/** The on-disk generated (receipt-owned or adoptable) agent file list, + * sorted. Adoptable files (exact marker-bearing desired bytes, no prior + * receipt) are appended to `adoptableFiles` as a side effect. */ +function onDiskAgentFiles(destDir, desired, { hasReceiptLedger, receiptMap, adoptionBlocked, adoptableFiles }) { + if (!fs.existsSync(destDir)) return []; + return fs.readdirSync(destDir).filter((f) => { + if (!f.endsWith('.md')) return false; + try { + const text = fs.readFileSync(path.join(destDir, f), 'utf8'); + if (hasReceiptLedger) { + const hasReceipt = adoptionBlocked || hasReceiptValue(receiptMap[f]); + const receiptOwned = receiptMatches(text, receiptMap[f]); + const adoptable = !hasReceipt && isGeneratedContent(text) && desired.get(f) === text; + if (adoptable) adoptableFiles.push(f); + return receiptOwned || adoptable; + } + return isGeneratedContent(text); + } catch { return false; } + }).sort(); +} + +function agentContentDiverged(destDir, onDisk, stamp) { + return !!stamp?.hashes && onDisk.some((f) => { + try { return contentHash(fs.readFileSync(path.join(destDir, f), 'utf8')) !== stamp.hashes[f]; } catch { return true; } + }); +} + +/** The stamp bytes syncAgents would write for today's on-disk generated set, + * or null when there's no source or the on-disk set doesn't fully match + * desired — used only for stamp-only adoption detection. */ +function expectedAgentStampText(source, onDisk, desired, gatewayCapabilities, lazyCatalog) { + if (!source || onDisk.length === 0 || !onDisk.every((f) => desired.has(f))) return null; + return `${JSON.stringify({ + source: source.id, + gateway: { ruflo: !!gatewayCapabilities.ruflo, aqe: !!gatewayCapabilities.aqe }, + lazyCatalog: !!lazyCatalog, + count: onDisk.length, + files: onDisk, + // syncAgents hashes in converter declaration order, then sorts only + // the separate files list. Preserve that byte order for exact + // stamp-only adoption detection. + hashes: Object.fromEntries([...desired.entries()] + .filter(([f]) => onDisk.includes(f)) + .map(([f, text]) => [f, contentHash(text)])), + }, null, 2)}\n`; +} + +function agentReceiptDivergence(destDir, receiptMap) { + if (!fs.existsSync(destDir)) return false; + return fs.readdirSync(destDir).filter((f) => f.endsWith('.md')).some((f) => { + try { + return hasReceiptValue(receiptMap[f]) + && !receiptMatches(fs.readFileSync(path.join(destDir, f), 'utf8'), receiptMap[f]); + } catch { return true; } + }); +} + +/** Assemble agentsStatus's final result object from its computed signals. */ +function agentsStatusResult({ + generatedCount, stamp, source, adoptionBlocked, adoptableFiles, stampAdoptable, + contentDiverged, hasReceiptLedger, destDir, receiptMap, filesDiverged, lazyCatalog, gatewayCapabilities, +}) { + return { + count: generatedCount, + stampedId: stamp?.source ?? null, + currentId: source?.id ?? null, + adoptable: !adoptionBlocked && (adoptableFiles.length > 0 || stampAdoptable), + adoptionBlocked, + modified: contentDiverged || (hasReceiptLedger && agentReceiptDivergence(destDir, receiptMap)), + stale: !stamp || stamp.source !== (source?.id ?? null) || filesDiverged + || !!stamp.lazyCatalog !== !!lazyCatalog + || !deepEqual(stamp.gateway ?? { ruflo: false, aqe: false }, { + ruflo: !!gatewayCapabilities.ruflo, aqe: !!gatewayCapabilities.aqe, + }), + }; +} + +/** Agent-set drift, honestly: stale when the stamp is missing, the catalog + * source id diverged (upgrade/marketplace pull), or the on-disk generated + * file set differs from the stamp. Count reports marker-bearing agents for + * visibility, while receipt/hash divergence is reported as `modified` so + * callers classify user edits as preserved rather than repairable drift. + * @param {{ source?: CatalogSource|null, destDir?: string, receipts?:Record|null, stampReceipt?:string|null, adoptionBlocked?:boolean, gatewayCapabilities?:{ruflo?:boolean,aqe?:boolean}, lazyCatalog?:boolean }} [opts] */ +export function agentsStatus({ + source, destDir = paths.opencodeAgentsDir(), receipts = null, stampReceipt = null, + adoptionBlocked = false, gatewayCapabilities = {}, lazyCatalog = false, +} = {}) { + const { stamp, stampText } = readAgentStamp(destDir); + const hasReceiptLedger = receipts !== null; + const receiptMap = asReceiptMap(receipts); + const desired = source + ? new Map(desiredAgentSet(source, gatewayCapabilities, lazyCatalog) + .agents.map((a) => [`${a.name}.md`, a.content])) + : new Map(); + const generatedCount = countGeneratedAgentFiles(destDir); + const adoptableFiles = []; + const onDisk = onDiskAgentFiles(destDir, desired, { hasReceiptLedger, receiptMap, adoptionBlocked, adoptableFiles }); + const stampFiles = Array.isArray(stamp?.files) ? [...stamp.files].sort() : null; + const filesDiverged = stampFiles != null && JSON.stringify(stampFiles) !== JSON.stringify(onDisk); + const contentDiverged = agentContentDiverged(destDir, onDisk, stamp); + const expectedStamp = expectedAgentStampText(source, onDisk, desired, gatewayCapabilities, lazyCatalog); + const stampAdoptable = hasReceiptLedger && !adoptionBlocked && !hasReceiptValue(stampReceipt) + && expectedStamp !== null && stampText === expectedStamp; + return agentsStatusResult({ + generatedCount, stamp, source, adoptionBlocked, adoptableFiles, stampAdoptable, + contentDiverged, hasReceiptLedger, destDir, receiptMap, filesDiverged, lazyCatalog, gatewayCapabilities, + }); +} + diff --git a/src/lib/opencode-artifacts.mjs b/src/lib/opencode-artifacts.mjs new file mode 100644 index 0000000..8699269 --- /dev/null +++ b/src/lib/opencode-artifacts.mjs @@ -0,0 +1,284 @@ +// opencode-artifacts.mjs — the plugin (lifecycle bridge + lazy rUv gateway) +// and platform-skill deployment, plus the shared teardown (removeArtifacts). +// Split out of opencode.mjs (ADR-0037's file-size gate) — behavior and +// export names are unchanged; opencode.mjs re-exports the +// externally-consumed names so no import path elsewhere in the repo needed +// to change. +import fs from 'node:fs'; +import path from 'node:path'; +import * as paths from './paths.mjs'; +import { contentHash, hasReceiptValue, receiptMatches } from './opencode-receipts.mjs'; +import { parseFrontmatter, SPECIALIST_AGENT, STAMP_FILE } from './opencode-agents.mjs'; + +/** @typedef {import('./opencode-agents.mjs').CatalogSource} CatalogSource */ + +// ── plugin (lifecycle bridge) ──────────────────────────────────────────────── + +export const PLUGIN_NAME = 'ruflo-hooks.js'; +export const GATEWAY_PLUGIN_NAME = 'ruflo-gateway.js'; +const pluginTemplate = (pkgRoot) => path.join(pkgRoot, 'src', 'templates', 'opencode-ruflo-hooks.js'); +const gatewayPluginTemplate = (pkgRoot) => path.join(pkgRoot, 'src', 'templates', 'opencode-ruflo-gateway.js'); + +/** The marker any ak-deployed plugin copy carries (from the template header). */ +const PLUGIN_MARKER = 'src/templates/opencode-ruflo-hooks.js'; +const GATEWAY_PLUGIN_MARKER = 'src/templates/opencode-ruflo-gateway.js'; + +function deployManagedPlugin({ + template, marker, name, label, pluginsDir, dryRun, receipt, adoptionBlocked, + desiredText = null, +}) { + if (!fs.existsSync(template)) return { ok: false, changed: false, detail: `template missing: ${template}` }; + const want = desiredText ?? fs.readFileSync(template, 'utf8'); + const dest = path.join(pluginsDir, name); + const cur = fs.existsSync(dest) ? fs.readFileSync(dest, 'utf8') : null; + const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); + const adoptable = cur === want && !hasReceipt && cur.includes(marker); + if (cur === want && (receiptMatches(cur, receipt) || adoptable)) { + return { + ok: true, changed: false, receipt: contentHash(cur), adopted: adoptable, + detail: adoptable ? `${label} adopted into receipt ledger` : `${label} current`, + }; + } + if (cur !== null && (!hasReceipt || !receiptMatches(cur, receipt))) { + return { ok: true, changed: false, receipt, detail: `⚠ ${dest} differs from ak's last-written receipt (user-owned/edited) — left untouched` }; + } + if (!want.includes(marker)) return { ok: false, changed: false, receipt, detail: `template marker missing: ${marker}` }; + if (!dryRun) { + fs.mkdirSync(pluginsDir, { recursive: true }); + fs.writeFileSync(dest, want); + } + return { + ok: true, + changed: true, + receipt: contentHash(want), + detail: cur == null ? `${label} deployed (${name})` : `${label} updated (${name})`, + }; +} + +function managedPluginStatus({ + template, marker, name, pluginsDir, receipt, adoptionBlocked, desiredText = null, +}) { + const dest = path.join(pluginsDir, name); + const present = fs.existsSync(dest); + const currentText = present ? fs.readFileSync(dest, 'utf8') : null; + const desired = fs.existsSync(template) ? (desiredText ?? fs.readFileSync(template, 'utf8')) : null; + const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); + const receiptOwned = present && receiptMatches(currentText, receipt); + const adoptable = present && !hasReceipt && desired !== null + && currentText === desired && currentText.includes(marker); + const foreign = present && !receiptOwned && !adoptable; + return { + present, + current: present && !foreign && desired !== null && currentText === desired, + foreign, + adoptable, + adoptionBlocked, + }; +} + +/** Deploy the lifecycle bridge plugin from the kit's template, content-diffed + * (rewrites only when the template changed — hash-stamped by content itself). + * A destination file that exists WITHOUT the ak marker is user-owned: + * preserved and reported, never overwritten. + * @param {{ pkgRoot: string, pluginsDir?: string, dryRun?: boolean, receipt?:string|null, adoptionBlocked?:boolean }} opts */ +export function deployPlugin({ + pkgRoot, pluginsDir = paths.opencodePluginsDir(), dryRun = false, receipt = null, + adoptionBlocked = false, +}) { + return deployManagedPlugin({ + template: pluginTemplate(pkgRoot), marker: PLUGIN_MARKER, name: PLUGIN_NAME, + label: 'lifecycle plugin', pluginsDir, dryRun, receipt, adoptionBlocked, + }); +} + +const GATEWAY_MCP_PLACEHOLDER = '/* AK_MANAGED_MCP_ENTRIES */ {}'; +const GATEWAY_AGENT_PLACEHOLDER = '/* AK_MANAGED_AGENT_CATALOG */ []'; +const GATEWAY_SPECIALIST_PLACEHOLDER = '/* AK_SPECIALIST_AGENT_PROMPT */ ""'; + +function gatewayDesiredText(pkgRoot, managedMcp, agentCatalog = []) { + const template = gatewayPluginTemplate(pkgRoot); + if (!fs.existsSync(template)) return null; + const source = fs.readFileSync(template, 'utf8'); + for (const [placeholder, label] of [ + [GATEWAY_MCP_PLACEHOLDER, 'managed-MCP'], + [GATEWAY_AGENT_PLACEHOLDER, 'managed-agent'], + [GATEWAY_SPECIALIST_PLACEHOLDER, 'specialist-agent'], + ]) { + const first = source.indexOf(placeholder); + if (first < 0 || source.indexOf(placeholder, first + 1) >= 0) { + throw new Error(`lazy gateway template must contain exactly one ${label} placeholder`); + } + } + const stable = Object.fromEntries(Object.entries(managedMcp ?? {}) + .sort(([a], [b]) => a.localeCompare(b))); + const specialistPrompt = parseFrontmatter(SPECIALIST_AGENT.content)?.body.trim() ?? ''; + return source + .replace(GATEWAY_MCP_PLACEHOLDER, JSON.stringify(stable)) + .replace(GATEWAY_AGENT_PLACEHOLDER, JSON.stringify(agentCatalog)) + .replace(GATEWAY_SPECIALIST_PLACEHOLDER, JSON.stringify(specialistPrompt)); +} + +/** Deploy the lazy Ruflo/Agentic-QE catalogue gateway for stock OpenCode. */ +export function deployGatewayPlugin({ + pkgRoot, managedMcp = {}, agentCatalog = [], pluginsDir = paths.opencodePluginsDir(), dryRun = false, + receipt = null, adoptionBlocked = false, +}) { + return deployManagedPlugin({ + template: gatewayPluginTemplate(pkgRoot), marker: GATEWAY_PLUGIN_MARKER, + name: GATEWAY_PLUGIN_NAME, label: 'lazy rUv gateway', pluginsDir, dryRun, + receipt, adoptionBlocked, desiredText: gatewayDesiredText(pkgRoot, managedMcp, agentCatalog), + }); +} + +/** Remove only exact receipt-owned gateway bytes when no rUv family remains + * safe to capture. User-edited/unreceipted files are preserved. */ +export function retireGatewayPlugin({ + pluginsDir = paths.opencodePluginsDir(), receipt = null, dryRun = false, +} = {}) { + const dest = path.join(pluginsDir, GATEWAY_PLUGIN_NAME); + if (!fs.existsSync(dest)) { + return { ok: true, changed: false, receipt: null, detail: 'lazy gateway not deployed' }; + } + const current = fs.readFileSync(dest, 'utf8'); + if (!receipt || !receiptMatches(current, receipt)) { + return { + ok: false, changed: false, receipt, + detail: `⚠ ${dest} is not provably ak-owned; left untouched`, + }; + } + if (!dryRun) fs.rmSync(dest, { force: true }); + return { ok: true, changed: true, receipt: null, detail: 'lazy rUv gateway retired' }; +} + +/** Plugin presence/currency against the kit template. `foreign` flags a + * user-owned file occupying the destination (status must not nag to + * overwrite it — deploy will leave it alone). */ +export function pluginStatus({ + pkgRoot, pluginsDir = paths.opencodePluginsDir(), receipt = null, adoptionBlocked = false, +}) { + return managedPluginStatus({ + template: pluginTemplate(pkgRoot), marker: PLUGIN_MARKER, name: PLUGIN_NAME, + pluginsDir, receipt, adoptionBlocked, + }); +} + +/** Lazy gateway presence/currency against its embedded, receipt-bound MCP commands. */ +export function gatewayPluginStatus({ + pkgRoot, managedMcp = {}, agentCatalog = [], pluginsDir = paths.opencodePluginsDir(), receipt = null, + adoptionBlocked = false, +}) { + return managedPluginStatus({ + template: gatewayPluginTemplate(pkgRoot), marker: GATEWAY_PLUGIN_MARKER, + name: GATEWAY_PLUGIN_NAME, pluginsDir, receipt, adoptionBlocked, + desiredText: gatewayDesiredText(pkgRoot, managedMcp, agentCatalog), + }); +} + +// ── platform skill ─────────────────────────────────────────────────────────── + +const SKILL_DEPLOYED_MARKER = ''; + +/** Deploy ruflo's platform SKILL.md (from the catalog source) into opencode's + * global skills dir, stamped with the source id for drift detection. A + * destination SKILL.md without the ak marker is user-owned: preserved. + * @param {{ source: CatalogSource|null, skillsDir?: string, dryRun?: boolean, receipt?:string|null, adoptionBlocked?:boolean }} opts */ +export function deploySkill({ + source, skillsDir = paths.opencodeSkillsDir(), dryRun = false, receipt = null, + adoptionBlocked = false, +}) { + if (!source?.hasPlatformSkill) return { ok: true, changed: false, detail: `no platform SKILL.md in catalog source${source ? ` (${source.id})` : ''}` }; + const src = path.join(source.root, 'SKILL.md'); + const dest = path.join(skillsDir, 'ruflo', 'SKILL.md'); + const want = `${fs.readFileSync(src, 'utf8').replace(/\s*$/, '')}\n\n${SKILL_DEPLOYED_MARKER} from ${source.id} — re-synced by \`ak sync\`\n`; + const cur = fs.existsSync(dest) ? fs.readFileSync(dest, 'utf8') : null; + const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); + const adoptable = cur === want && !hasReceipt && cur.includes(SKILL_DEPLOYED_MARKER); + if (cur === want && (receiptMatches(cur, receipt) || adoptable)) { + return { + ok: true, changed: false, receipt: contentHash(cur), adopted: adoptable, + detail: adoptable ? 'platform skill adopted into receipt ledger' : 'platform skill current', + }; + } + if (cur !== null && (!hasReceipt || !receiptMatches(cur, receipt))) { + return { ok: true, changed: false, receipt, detail: `⚠ ${dest} differs from ak's last-written receipt (user-owned/edited) — left untouched` }; + } + if (!dryRun) { + fs.mkdirSync(path.dirname(dest), { recursive: true }); + fs.writeFileSync(dest, want); + } + return { ok: true, changed: true, receipt: contentHash(want), detail: `platform skill deployed (skills/ruflo/SKILL.md, ${source.id})` }; +} + +/** Platform skill presence/currency against the catalog source id. `foreign` + * flags a user-owned SKILL.md at the destination. + * @param {{ source?: CatalogSource|null, skillsDir?: string, receipt?:string|null, adoptionBlocked?:boolean }} [opts] */ +export function skillStatus({ + source, skillsDir = paths.opencodeSkillsDir(), receipt = null, adoptionBlocked = false, +} = {}) { + const dest = path.join(skillsDir, 'ruflo', 'SKILL.md'); + const present = fs.existsSync(dest); + const text = present ? fs.readFileSync(dest, 'utf8') : null; + const sourceFile = source?.hasPlatformSkill ? path.join(source.root, 'SKILL.md') : null; + const desired = sourceFile && fs.existsSync(sourceFile) + ? `${fs.readFileSync(sourceFile, 'utf8').replace(/\s*$/, '')}\n\n${SKILL_DEPLOYED_MARKER} from ${source.id} — re-synced by \`ak sync\`\n` + : null; + const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); + const receiptOwned = present && receiptMatches(text, receipt); + const adoptable = present && !hasReceipt && desired !== null + && text === desired && text.includes(SKILL_DEPLOYED_MARKER); + const foreign = present && !receiptOwned && !adoptable; + return { + present, + foreign, + adoptable, + adoptionBlocked, + current: present && !foreign && desired != null && text === desired, + }; +} + +/** Remove `file` only when it exists and its content hash exactly matches + * `receipt` (provable ak ownership); push `label` onto `removed` when given. + * Returns whether the file was removed. */ +function removeIfReceiptOwned(file, receipt, label, removed) { + if (!fs.existsSync(file) || !receipt) return false; + if (contentHash(fs.readFileSync(file, 'utf8')) !== receipt) return false; + fs.rmSync(file, { force: true }); + if (label) removed.push(label); + return true; +} + +/** Remove receipt-owned generated agent files + the stamp file, pushing a + * single summary label for the count of agents removed. */ +function removeGeneratedAgents(agentsDir, receipts, removed) { + if (!fs.existsSync(agentsDir)) return; + let n = 0; + for (const f of fs.readdirSync(agentsDir)) { + const p = path.join(agentsDir, f); + if (f === STAMP_FILE) { removeIfReceiptOwned(p, receipts.agentStamp, null, removed); continue; } + if (f.endsWith('.md') && removeIfReceiptOwned(p, receipts.agents?.[f], null, removed)) n++; + } + if (n) removed.push(`${n} generated agents`); +} + +/** Remove ak-deployed artifacts (marker-gated — user files are never touched): + * the lifecycle plugin, generated agents (+ stamp), the platform skill's + * SKILL.md. Directories are pruned only when EMPTY after the managed files + * are gone — user resources placed beside them survive. + * @param {{ pluginsDir?: string, agentsDir?: string, skillsDir?: string, receipts?:any }} [opts] */ +export function removeArtifacts({ + pluginsDir = paths.opencodePluginsDir(), agentsDir = paths.opencodeAgentsDir(), + skillsDir = paths.opencodeSkillsDir(), receipts = {}, +} = {}) { + const removed = []; + const rmdirIfEmpty = (dir) => { + try { if (fs.readdirSync(dir).length === 0) fs.rmdirSync(dir); } catch { /* absent or not empty */ } + }; + removeIfReceiptOwned(path.join(pluginsDir, PLUGIN_NAME), receipts.plugin, 'plugin ruflo-hooks.js', removed); + removeIfReceiptOwned(path.join(pluginsDir, GATEWAY_PLUGIN_NAME), receipts.gateway, 'plugin ruflo-gateway.js', removed); + removeGeneratedAgents(agentsDir, receipts, removed); + const skillDir = path.join(skillsDir, 'ruflo'); + if (removeIfReceiptOwned(path.join(skillDir, 'SKILL.md'), receipts.skill, 'platform skill', removed)) { + rmdirIfEmpty(skillDir); + } + return { ok: true, changed: removed.length > 0, detail: removed.length ? `removed: ${removed.join(', ')}` : 'no ak-deployed artifacts found' }; +} diff --git a/src/lib/opencode-core.mjs b/src/lib/opencode-core.mjs new file mode 100644 index 0000000..c1a8eda --- /dev/null +++ b/src/lib/opencode-core.mjs @@ -0,0 +1,693 @@ +// opencode-core.mjs — opencode.json config-wiring: the receipt-owned +// reconcile/status/teardown pair (applyOpencode/undoOpencode, +// opencodeConverged/opencodeMcpStatus) at the heart of the third host +// adapter's I/O half. Split out of opencode.mjs (ADR-0037's file-size gate); +// opencode.mjs re-exports this module's public names, and +// opencode-lifecycle.mjs composes them with opencode-agents.mjs / +// opencode-artifacts.mjs into the full enable/retire stack. +// +// why: opencode (opencode.ai) consumes the same rUv stack as claude/codex but +// through different surfaces. This module owns every ak-managed byte on those +// surfaces, backup-first + merge-not-clobber + ownership-marked, mirroring the +// claude (settings.mjs / mcp.mjs) and codex (providers.mjs Ruflo integration) +// contracts: +// +// ~/.config/opencode/opencode.json mcp.claude-flow + mcp.agentic-qe + +// mcp.ruvnet-brain, +// skills.paths, permission patterns +// ~/.config/opencode/AGENTS.md guidance blocks (blocks.mjs target +// 'agents-opencode' — NOT here) +// ~/.config/opencode/plugins/ruflo-hooks.js lifecycle bridge (opencode has +// no settings-hooks surface; its plugin +// events are the hook spine) +// ~/.config/opencode/plugins/ruflo-gateway.js lazy bridges to the complete +// live Ruflo and Agentic QE catalogues +// ~/.config/opencode/agents/ak-specialist.md +// one stock subagent; receipt-owned rUv +// profiles stay embedded and load lazily +// ~/.config/opencode/skills/ruflo/ the platform SKILL.md +// +// Grounded: +// - opencode.json schema (https://opencode.ai/config.json): mcp local +// servers {type,command[],environment,enabled,timeout}, skills.paths[], +// permission as wildcard tool-name patterns (MCP tools surface as +// `_`, hence the claude-flow_*/agentic-qe_*/ruvnet-brain_* patterns). +// - ruflo's own init/mcp-generator.ts env block (CLAUDE_FLOW_* below). +// - `claude-flow-mcp` (the dedicated stdio bin of @claude-flow/cli) answers +// initialize directly; `ruflo mcp start` is the fallback (what ak already +// registers for claude/codex) when that bin is absent. +// - ruvnet-brain's stable-spine shim (~/.claude/ruvnet-brain/mcp/server.mjs) +// hot-swaps brain versions — the registration never needs rewriting. +// - opencode.json may legally contain JSONC comments ($schema allowComments): +// a file we cannot parse is REFUSED, never clobbered. +import fs from 'node:fs'; +import path from 'node:path'; +import { have } from './exec.mjs'; +import { writeJsonWithBackup } from './settings.mjs'; +import { CURRENT_INTEGRATIONS_VERSION } from './adapters/config.mjs'; +import * as paths from './paths.mjs'; +import { deepEqual, hasReceiptValue } from './opencode-receipts.mjs'; +import { catalogSource, skillPathsFor } from './opencode-agents.mjs'; + +export const opencodeOwnership = (cfg) => cfg?.integrations?.ownership?.opencode ?? {}; +export function mutableOpencodeOwnership(cfg) { + cfg.integrations ??= { + version: CURRENT_INTEGRATIONS_VERSION, + hosts: {}, + bindings: [], + }; + cfg.integrations.ownership ??= {}; + cfg.integrations.ownership.opencode ??= {}; + return cfg.integrations.ownership.opencode; +} + +// ── config-file wiring (opencode.json) ────────────────────────────────────── + +/** ruflo init/mcp-generator.ts's env block, mirrored for parity. */ +export const RUFLO_MCP_ENV = { + npm_config_update_notifier: 'false', + CLAUDE_FLOW_MODE: 'v3', + CLAUDE_FLOW_HOOKS_ENABLED: 'true', + CLAUDE_FLOW_TOPOLOGY: 'hierarchical-mesh', + CLAUDE_FLOW_MAX_AGENTS: '15', + CLAUDE_FLOW_MEMORY_BACKEND: 'hybrid', +}; + +/** agentic-qe init's project MCP environment, mirrored for OpenCode. */ +export const AQE_MCP_ENV = { + AQE_LEARNING_ENABLED: 'true', + AQE_WORKERS_ENABLED: 'true', + NODE_ENV: 'production', +}; + +/** Permission patterns follow the rUv capabilities AK actually projects. */ +function permissionFamiliesFor(entries) { + return [ + ...('claude-flow' in entries ? [['claude-flow_*', 'claude_flow_*']] : []), + ...('agentic-qe' in entries ? [['agentic-qe_*', 'agentic_qe_*']] : []), + ...('ruvnet-brain' in entries ? [['ruvnet-brain_*', 'ruvnet_brain_*']] : []), + ]; +} + +function permissionKeysFor(entries) { + return permissionFamiliesFor(entries).flat(); +} + +/** The brain's stable-spine shim (same registration codex carries). */ +export const brainShimPath = () => path.join(paths.home, '.claude', 'ruvnet-brain', 'mcp', 'server.mjs'); + +/** The dedicated stdio MCP server bundled inside a plain `npm i -g ruflo` + * install (nested dependency — present even when no claude-flow-mcp bin is + * on PATH). */ +export const nestedMcpServerPath = () => + path.join(paths.rufloNodeModules(), '@claude-flow', 'cli', 'bin', 'mcp-server.js'); + +/** The claude-flow MCP command, best-available-first: the claude-flow-mcp bin + * on PATH → the nested mcp-server.js via absolute node path (no PATH/cwd + * dependence — the fresh ruflo-only machine case) → `ruflo mcp start` (ak's + * claude/codex registration path, always present when ruflo is). Pure. */ +export function mcpCommandFor({ binPresent, nestedPath }) { + if (binPresent) return ['claude-flow-mcp']; + if (nestedPath && fs.existsSync(nestedPath)) return ['node', nestedPath]; + return ['ruflo', 'mcp', 'start']; +} + +/** @typedef {{ kind: string, root: string, id: string, hasPlugins: boolean, hasPlatformSkill: boolean }} CatalogSource */ + +/** The MCP server entries ak writes. `claude-flow` resolves via mcpCommandFor + * (bin on PATH → nested mcp-server.js → `ruflo mcp start`). Agentic QE is + * included by default because machine setup installs it; `--no-aqe` disables + * that projection. ruvnet-brain is included only when its shim is on disk. + * @param {{ brainShim?: string, nestedPath?: string, includeAqe?: boolean }} [opts] */ +export async function mcpEntriesFor({ + brainShim = brainShimPath(), nestedPath = nestedMcpServerPath(), includeAqe = true, +} = {}) { + const entries = { + 'claude-flow': { + type: 'local', + command: mcpCommandFor({ binPresent: await have('claude-flow-mcp'), nestedPath }), + enabled: true, + timeout: 30000, + environment: { ...RUFLO_MCP_ENV }, + }, + }; + if (includeAqe) { + entries['agentic-qe'] = { + type: 'local', + command: ['aqe-mcp'], + enabled: true, + timeout: 30000, + environment: { ...AQE_MCP_ENV }, + }; + } + if (fs.existsSync(brainShim)) { + entries['ruvnet-brain'] = { type: 'local', command: ['node', brainShim], enabled: true, timeout: 30000 }; + } + return entries; +} + +/** Strict read: distinguishes "absent/empty" from "present but not plain JSON" + * (opencode.json may legally be JSONC). NB: settings.readJson's fallback + * parameter can't express this — passing undefined re-triggers its default. */ +function readJsonStrict(file) { + try { + const raw = fs.readFileSync(file, 'utf8'); + if (!raw.trim()) return { ok: true, doc: {} }; + return { ok: true, doc: JSON.parse(raw) }; + } catch { + return { ok: false, doc: null }; + } +} + +/** OpenCode loads opencode.jsonc after opencode.json. A sibling JSONC file can + * shadow managed MCP, permission, or plugin values, and AK deliberately does + * not normalize or rewrite user comments. */ +function laterJsoncOverride(configFile) { + if (path.basename(configFile) !== 'opencode.json') return null; + const candidate = path.join(path.dirname(configFile), 'opencode.jsonc'); + return fs.existsSync(candidate) ? candidate : null; +} + +/** Registration state, spawn-free (mirrors mcp.mjs registrationStatus's + * file-read approach). `parseError` distinguishes "absent" from "present but + * not plain JSON" (JSONC) — the writer refuses the latter. + * @param {any} cfg @param {{ configFile?: string }} [opts] */ +export function opencodeMcpStatus(cfg, { configFile = paths.opencodeConfigPath() } = {}) { + const exists = fs.existsSync(configFile); + const laterOverride = laterJsoncOverride(configFile); + const { ok, doc } = exists ? readJsonStrict(configFile) : { ok: true, doc: {} }; + if (!ok) { + return { + exists, parseError: true, laterOverride, claudeFlow: false, aqe: false, brain: false, + owned: opencodeOwnership(cfg).mcp === 'ak', + }; + } + return { + exists, + parseError: false, + laterOverride, + claudeFlow: !!doc?.mcp?.['claude-flow'], + aqe: !!doc?.mcp?.['agentic-qe'], + brain: !!doc?.mcp?.['ruvnet-brain'], + paths: doc?.skills?.paths ?? [], + owned: opencodeOwnership(cfg).mcp === 'ak', + }; +} + +/** mcp. reasons: missing entirely, or present but drifted from desired. */ +function mcpConvergenceReasons(doc, entries) { + const reasons = []; + for (const [name, want] of Object.entries(entries)) { + if (!(name in (doc.mcp ?? {}))) reasons.push(`${name} missing`); + else if (!deepEqual(doc.mcp[name], want)) reasons.push(`${name} drifted`); + } + return reasons; +} + +/** A previously-owned key that fell out of the desired set but still equals + * what ak wrote is stale — shared shape for managed.mcp and + * managed.permissions convergence reasons. */ +function staleOwnedReasons(records, desiredKeys, currentValues, label) { + const reasons = []; + for (const [key, rec] of Object.entries(records)) { + if (desiredKeys.has(key) || rec.written == null) continue; + if (deepEqual(currentValues?.[key], rec.written)) reasons.push(label(key)); + } + return reasons; +} + +function skillPathReasons(doc, skillPaths) { + return skillPaths + .filter((p) => !(doc.skills?.paths ?? []).includes(p)) + .map((p) => `skills path missing: ${p}`); +} + +function permissionAllowReasons(doc, permissionKeys) { + return permissionKeys + .filter((k) => doc.permission?.[k] !== 'allow') + .map((k) => `permission ${k} not allowed`); +} + +/** Convergence check — deeper than key existence (codex-review #16): the MCP + * entries must EQUAL today's desired values (command/env/timeout drift when + * the user edits them or a kit upgrade changes the template), desired skills + * paths must all be present, desired permission patterns must be 'allow', and + * a ruvnet-brain entry whose shim has vanished is stale. Async because the + * desired entries probe the claude-flow-mcp bin (one `which`, matching the + * spawn profile of status's hosts rows). + * @param {any} cfg @param {{ configFile?: string, brainShim?: string }} [opts] */ +export async function opencodeConverged(cfg, { configFile = paths.opencodeConfigPath(), brainShim } = {}) { + const st = opencodeMcpStatus(cfg, { configFile }); + if (!st.exists || st.parseError) return { converged: false, reasons: st.parseError ? ['unparseable config'] : ['no config file'] }; + if (st.laterOverride) { + return { + converged: false, + reasons: [`later OpenCode config override is unverified: ${st.laterOverride}`], + }; + } + const doc = readJsonStrict(configFile).doc; + const entries = await mcpEntriesFor({ brainShim, includeAqe: cfg.aqe !== false }); + const managed = normalizeManaged(opencodeOwnership(cfg).managed); + const ownedEntries = Object.fromEntries(Object.entries(entries).filter( + ([name]) => managed.mcp[name]?.written != null, + )); + const permissionKeys = permissionKeysFor(ownedEntries); + const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); + const reasons = [ + ...mcpConvergenceReasons(doc, entries), + ...staleOwnedReasons(managed.mcp, new Set(Object.keys(entries)), doc.mcp, (name) => `${name} stale (no longer desired)`), + ...skillPathReasons(doc, skillPathsFor(source)), + ...permissionAllowReasons(doc, permissionKeys), + ...staleOwnedReasons(managed.permissions, new Set(permissionKeys), doc.permission, (key) => `permission ${key} stale (no longer desired)`), + ]; + return { converged: reasons.length === 0, reasons }; +} + +/** Normalize an opencodeManaged record — the current precise shape + * { mcp: {name:{prior,written}}, paths: [], permissions: {key:{prior,written}} }, + * tolerating the legacy names-only shape from the first shipped version + * (legacy entries have unknown prior/written → treated conservatively: prior + * null, written null → never auto-deleted, only re-recorded on next apply). */ +/** Normalize a legacy-tolerant names-or-{prior,written}-records collection + * (used identically for both m.mcp and m.permissions). */ +function normalizeOwnedRecordMap(source) { + const names = Array.isArray(source) ? source : Object.keys(source ?? {}); + const out = {}; + for (const n of names) { + const rec = Array.isArray(source) ? null : source[n]; + out[n] = rec && typeof rec === 'object' && 'written' in rec ? rec : { prior: null, written: null }; + } + return out; +} + +/** Normalize the artifacts sub-container into `out.artifacts`/`out.artifactState`. */ +function normalizeManagedArtifacts(m, out) { + if (hasReceiptValue(m.artifacts)) { + out.artifactState.rawContainer = structuredClone(m.artifacts); + } + if (hasReceiptValue(m.artifacts) + && (typeof m.artifacts !== 'object' || Array.isArray(m.artifacts))) { + out.artifactState.containerMalformed = true; + return; + } + const artifacts = m.artifacts ?? {}; + out.artifacts.plugin = hasReceiptValue(artifacts.plugin) ? artifacts.plugin : null; + out.artifacts.gateway = hasReceiptValue(artifacts.gateway) ? artifacts.gateway : null; + out.artifacts.agentStamp = hasReceiptValue(artifacts.agentStamp) ? artifacts.agentStamp : null; + out.artifacts.skill = hasReceiptValue(artifacts.skill) ? artifacts.skill : null; + if (hasReceiptValue(artifacts.agents) + && (typeof artifacts.agents !== 'object' || Array.isArray(artifacts.agents))) { + out.artifactState.agentsMalformed = true; + } else if (artifacts.agents) { + out.artifacts.agents = Object.fromEntries( + Object.entries(artifacts.agents).filter(([, hash]) => hasReceiptValue(hash)), + ); + } +} + +export function normalizeManaged(m) { + const out = { + mcp: {}, paths: [], permissions: {}, permissionScalar: null, + artifacts: { plugin: null, gateway: null, agents: {}, agentStamp: null, skill: null }, + artifactState: { + containerMalformed: false, agentsMalformed: false, + rawContainer: null, + }, + }; + if (!m || typeof m !== 'object') return out; + out.mcp = normalizeOwnedRecordMap(m.mcp); + out.paths = Array.isArray(m.paths) ? [...m.paths] : []; + out.permissions = normalizeOwnedRecordMap(m.permissions); + out.permissionScalar = typeof m.permissionScalar === 'string' ? m.permissionScalar : null; + normalizeManagedArtifacts(m, out); + return out; +} + +/** Read-only artifact receipt boundary shared by lifecycle and status paths. + * Only null/absent containers represent a pre-receipts migration gap; + * malformed non-null containers fail closed and block adoption. */ +export function opencodeArtifactReceiptState(managed) { + const normalized = normalizeManaged(managed); + const blocked = normalized.artifactState.containerMalformed + || normalized.artifactState.agentsMalformed; + return { + receipts: normalized.artifacts, + adoptionBlocked: blocked, + agentsAdoptionBlocked: blocked, + }; +} + +/** Exact MCP values the lazy gateway may capture. A same-name entry is not + * enough: the command and both direct permission spellings must still match + * values positively recorded as AK-written. Explicit direct-tool enablement + * is an operator opt-out from lazy capture. */ +export function managedGatewayMcp(cfg, { configFile = paths.opencodeConfigPath() } = {}) { + const managed = normalizeManaged(opencodeOwnership(cfg).managed); + const parsed = fs.existsSync(configFile) ? readJsonStrict(configFile) : { ok: true, doc: {} }; + const tools = parsed.ok ? (parsed.doc?.tools ?? {}) : {}; + const permissions = parsed.ok && typeof parsed.doc?.permission === 'object' + ? parsed.doc.permission + : {}; + const families = { + 'claude-flow': ['claude-flow_*', 'claude_flow_*'], + 'agentic-qe': ['agentic-qe_*', 'agentic_qe_*'], + }; + return Object.fromEntries(Object.entries(families) + .filter(([name, keys]) => managed.mcp[name]?.written != null + && keys.every((key) => managed.permissions[key]?.written === 'allow' + && permissions[key] === 'allow') + && !keys.some((key) => tools[key] === true)) + .map(([name]) => [name, structuredClone(managed.mcp[name].written)])); +} + +/** Generic receipt-owned key→value map reconciler: prune-stale-owned → + * collision-detect → adopt/own with {prior, written} records. This is the + * ONE algorithm behind applyOpencode's mcp block and (behind the + * family-atomicity wrapper below) its permission block — previously + * hand-instantiated per surface (the audit's `reconcileOwnedMap` finding). + * Mutates `obj` in place; returns the new {prior,written} ledger for every + * key in `desiredEntries`. + * @param {Record} obj @param {[string, any][]} desiredEntries + * @param {Record} prevRecords + * @param {{ collisions: string[], pruned: string[], pruneLabel: (key:string) => string, collisionLabel: (key:string) => string }} opts */ +function reconcileOwnedMap(obj, desiredEntries, prevRecords, { collisions, pruned, pruneLabel, collisionLabel }) { + const desiredKeys = new Set(desiredEntries.map(([key]) => key)); + for (const [key, rec] of Object.entries(prevRecords)) { + if (desiredKeys.has(key) || !(key in obj)) continue; + if (rec.written && deepEqual(obj[key], rec.written)) { + // RESTORE the prior when there was one (a user entry that happened to + // equal the old desired value is a user value, not ak's to delete); + // delete only what ak itself created (codex-review r2). + if (rec.prior != null) { obj[key] = rec.prior; pruned.push(`${pruneLabel(key)} (prior restored)`); } + else { delete obj[key]; pruned.push(pruneLabel(key)); } + } // else: user edited (or legacy record) → leave it, keep no ownership + } + const managed = {}; + for (const [key, want] of desiredEntries) { + const cur = obj[key]; + const priorRec = prevRecords[key]; + if (cur !== undefined && !deepEqual(cur, want) && !(priorRec?.written && deepEqual(cur, priorRec.written))) { + collisions.push(collisionLabel(key)); + managed[key] = { prior: cur, written: null }; // tracked but NOT ak-owned + continue; + } + // A previously-colliding (never ak-authored) value the USER has since + // aligned to the desired one stays unmanaged: adopting it with the stale + // pre-collision prior would make undo overwrite the user's own later + // choice (codex-review r2). Noted, not owned. + if (priorRec && priorRec.written == null && cur !== undefined && deepEqual(cur, want)) { + managed[key] = { prior: priorRec.prior, written: null }; + continue; + } + // prior is the ORIGINAL pre-ak value (kept across reapplies), never the + // ak-written value currently in place. + managed[key] = { prior: priorRec ? priorRec.prior : (cur ?? null), written: want }; + obj[key] = want; + } + return managed; +} + +/** Family-atomicity wrapper for permissions: a family's members (both + * claude-flow_* and claude_flow_* spellings) must ALL be free of collision + * before ANY of them join this run's desired permission set — a + * same-name MCP collision must never let AK add broad `allow` on the + * sibling spelling. A blocked family's present members are recorded + * {prior, written:null} (never applied via reconcileOwnedMap) and reported + * as a collision when their value isn't already the desired 'allow'. + * @param {Record} ownedEntries @param {Record} current + * @param {Record} prevRecords + * @param {{ collisions: string[] }} opts + * @returns {{ desiredKeys: string[], managed: Record }} */ +function reconcileFamilyPermissions(ownedEntries, current, prevRecords, { collisions }) { + const desiredKeys = []; + const managed = {}; + for (const keys of permissionFamiliesFor(ownedEntries)) { + const blocked = keys.some((key) => { + const cur = current[key]; + const priorRec = prevRecords[key]; + const conflicts = cur !== undefined && cur !== 'allow' + && !(priorRec?.written && deepEqual(cur, priorRec.written)); + const previouslyUnowned = cur !== undefined && priorRec && priorRec.written == null; + return conflicts || previouslyUnowned; + }); + if (!blocked) { + desiredKeys.push(...keys); + continue; + } + for (const key of keys) { + const cur = current[key]; + const priorRec = prevRecords[key]; + if (cur !== undefined && cur !== 'allow' + && !(priorRec?.written && deepEqual(cur, priorRec.written))) { + collisions.push(`permission.${key}`); + } + if (cur !== undefined) { + managed[key] = { prior: priorRec ? priorRec.prior : cur, written: null }; + } + } + } + return { desiredKeys, managed }; +} + +/** Reconcile ak's desired skills.paths membership: prune previously-ak-added + * paths that fell out of the desired set, then add newly desired paths not + * already present. No collision concept applies — path membership isn't + * exclusively owned the way a single mcp/permission value is. + * @param {any} next @param {string[]} skillPaths + * @param {{paths:string[]}} prevManaged @param {string[]} pruned + * @returns {string[]} the new managed paths list */ +function reconcileSkillPaths(next, skillPaths, prevManaged, pruned) { + if (next.skills?.paths && prevManaged.paths.length) { + const stale = new Set(prevManaged.paths.filter((p) => !skillPaths.includes(p))); + next.skills.paths = next.skills.paths.filter((p) => !stale.has(p)); + if (stale.size) pruned.push(`${stale.size} stale skills path(s)`); + } + if (!skillPaths.length) return []; + next.skills = { ...(next.skills ?? {}) }; + const cur = new Set(next.skills.paths ?? []); + const newlyAdded = skillPaths.filter((p) => !cur.has(p)); + // ownership = previously-recorded ak paths that are still desired + newly + // added ones (a re-apply must not erase the record of what ak added). + const managedPaths = [...new Set([...prevManaged.paths.filter((p) => skillPaths.includes(p)), ...newlyAdded])]; + next.skills.paths = [...cur, ...newlyAdded]; + return managedPaths; +} + +/** Build applyOpencode's human-readable result summary from its accumulated + * notes (wired/in-sync headline, pruned keys, preserved collisions). */ +function applyOpencodeSummary({ changed, entries, skillPaths, permissionKeys, source, pruned, collisions }) { + const aqe = entries['agentic-qe'] ? ' + agentic-qe' : ''; + const brain = entries['ruvnet-brain'] ? ' + ruvnet-brain' : ' (brain shim absent)'; + const notes = [ + changed ? `opencode.json wired: claude-flow (${entries['claude-flow'].command.join(' ')})${aqe}${brain}, ${skillPaths.length} skills path(s), ${permissionKeys.length} permission pattern(s)` + : `opencode.json in sync${source ? '' : ' — ⚠ no ruflo catalog source found for skills.paths'}`, + ]; + if (pruned.length) notes.push(`pruned: ${pruned.join(', ')}`); + if (collisions.length) notes.push(`⚠ collisions preserved (user-owned, untouched): ${collisions.join(', ')}`); + return notes.join(' — '); +} + +/** Reconcile opencode.json: ak's MCP servers, skills.paths, and permission + * patterns merged into whatever is already there. The ownership contract is + * VALUE-PRECISE, not name-precise: + * - a pre-existing entry that DIFFERS from ak's desired value (and was not + * previously ak-written) is a COLLISION: preserved, reported, never taken + * over — merge-not-clobber applies to values, not just files; + * - every key ak writes is recorded as {prior, written} so teardown can + * restore the user's original value instead of deleting it; + * - previously-managed keys that fall OUT of the desired set (brain shim + * removed, catalog source changed) are removed only while they still equal + * what ak wrote — a user-edited value is left and reported. + * Scalar `permission` ("permission":"allow") is first lifted to its + * documented object equivalent {"*":"allow"} (wildcard key semantics), never + * spread character-by-character. Backup-first, idempotent, JSONC-refusing. + * @param {any} cfg @param {{ dryRun?: boolean, configFile?: string, brainShim?: string }} [opts] */ +export async function applyOpencode(cfg, { dryRun = false, configFile = paths.opencodeConfigPath(), brainShim } = {}) { + if (!cfg.integrations?.hosts?.opencode) return { ok: true, changed: false, detail: 'opencode not enabled — unmanaged' }; + const laterOverride = laterJsoncOverride(configFile); + if (laterOverride) { + return { + ok: false, fatal: true, changed: false, + detail: `${laterOverride} loads after opencode.json — refusing to write or claim an unverified effective config; merge the Agentic Kit entries there manually or remove the override`, + }; + } + const exists = fs.existsSync(configFile); + const { ok: parsedOk, doc } = exists ? readJsonStrict(configFile) : { ok: true, doc: {} }; + if (!parsedOk) { + return { + ok: false, fatal: true, changed: false, + detail: `${configFile} is not plain JSON (JSONC comments?) — refusing to touch it; merge manually`, + }; + } + const entries = await mcpEntriesFor({ brainShim, includeAqe: cfg.aqe !== false }); + const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); + const skillPaths = skillPathsFor(source); + const prevManaged = normalizeManaged(opencodeOwnership(cfg).managed); + const collisions = []; + const pruned = []; + + const next = JSON.parse(JSON.stringify(doc)); + next.$schema ??= 'https://opencode.ai/config.json'; + + // ── mcp: prune stale ak entries, then merge desired with collision refusal ── + next.mcp = { ...(next.mcp ?? {}) }; + const managed = { + mcp: reconcileOwnedMap(next.mcp, Object.entries(entries), prevManaged.mcp, { + collisions, pruned, + pruneLabel: (name) => name, + collisionLabel: (name) => `mcp.${name}`, + }), + paths: [], permissions: {}, permissionScalar: null, + artifacts: (prevManaged.artifactState.containerMalformed + || prevManaged.artifactState.agentsMalformed) + ? structuredClone(prevManaged.artifactState.rawContainer) + : structuredClone(prevManaged.artifacts), + }; + + // ── skills.paths: remove stale ak-added paths, add desired ── + managed.paths = reconcileSkillPaths(next, skillPaths, prevManaged, pruned); + + // ── permission: lift scalar shorthand, prune stale, merge desired ── + // Record scalar ORIGIN explicitly (codex-review r2): undo restores the + // scalar form only when the file actually started scalar — a pre-existing + // {"*":"ask"} object must survive as an object, not be "restored" to "ask". + managed.permissionScalar = typeof doc.permission === 'string' + ? doc.permission + : (prevManaged.permissionScalar ?? null); + if (typeof next.permission === 'string') next.permission = { '*': next.permission }; + next.permission = { ...(next.permission ?? {}) }; + + // Permissions are family-atomic with MCP ownership. A foreign/colliding + // same-name MCP must never inherit broad AK-written `allow` patterns, and a + // collision on either spelling prevents AK from adding the other spelling. + const ownedEntries = Object.fromEntries(Object.entries(entries).filter( + ([name]) => managed.mcp[name]?.written != null, + )); + const { desiredKeys: permissionKeys, managed: blockedPermissions } = + reconcileFamilyPermissions(ownedEntries, next.permission, prevManaged.permissions, { collisions }); + Object.assign(managed.permissions, blockedPermissions); + Object.assign(managed.permissions, reconcileOwnedMap( + next.permission, permissionKeys.map((k) => [k, 'allow']), prevManaged.permissions, + { + collisions, pruned, + pruneLabel: (k) => `permission.${k}`, + collisionLabel: (k) => `permission.${k}`, + }, + )); + + const changed = JSON.stringify(next) !== JSON.stringify(doc); + if (!dryRun) { + const ownership = mutableOpencodeOwnership(cfg); + ownership.mcp = 'ak'; + ownership.managed = managed; + } + if (changed && !dryRun) writeJsonWithBackup(configFile, next); + return { + ok: collisions.length === 0, + fatal: false, + changed, + collisions, + detail: applyOpencodeSummary({ changed, entries, skillPaths, permissionKeys, source, pruned, collisions }), + }; +} + +/** Surgical teardown of ak's opencode.json wiring — ONLY the recorded managed + * keys, and ONLY when ak wrote them + * (`integrations.ownership.opencode.mcp === 'ak'`). For each + * managed key: when the current value still equals what ak wrote, the user's + * PRIOR value is restored (or the key removed if there was none); a value the + * user edited since is left and reported, never silently deleted. Scalar + * permission shorthand is restored to scalar when teardown empties the object + * but a prior '*' wildcard exists. Deployed artifacts are removed separately + * (removeArtifacts). + * @param {any} cfg @param {{ configFile?: string }} [opts] */ +/** Remove ak-managed paths that fell out of the desired set from + * doc.skills.paths, pruning the now-empty container(s) too. Returns whether + * anything changed. */ +function restoreSkillPaths(doc, managedPaths) { + if (!doc.skills?.paths || !managedPaths.length) return false; + const drop = new Set(managedPaths); + const keptPaths = doc.skills.paths.filter((p) => !drop.has(p)); + if (keptPaths.length === doc.skills.paths.length) return false; + if (keptPaths.length) doc.skills.paths = keptPaths; + else { + delete doc.skills.paths; + if (Object.keys(doc.skills).length === 0) delete doc.skills; + } + return true; +} + +/** Drop an emptied permission object, or collapse it back to the scalar + * shorthand ak lifted from (only when scalar was the ORIGIN). Returns + * whether anything changed. */ +function collapsePermissionScalar(doc, scalarOrigin) { + if (doc.permission && Object.keys(doc.permission).length === 0) { + delete doc.permission; + return false; + } + if (doc.permission && scalarOrigin != null && Object.keys(doc.permission).length === 1 && doc.permission['*'] != null) { + // restore the scalar shorthand we lifted (only when scalar was the ORIGIN; + // the current '*' value is what collapses back — '*' is never ak-managed) + doc.permission = doc.permission['*']; + return true; + } + return false; +} + +export function undoOpencode(cfg, { configFile = paths.opencodeConfigPath() } = {}) { + if (opencodeOwnership(cfg).mcp !== 'ak') { + return { ok: true, changed: false, detail: 'opencode.json left as-is (not ak-managed)' }; + } + const managed = normalizeManaged(opencodeOwnership(cfg).managed); + if (!fs.existsSync(configFile)) { + // Nothing left to strip — but the markers would otherwise survive as a lie + // (a later teardown would chase a phantom config). Clear them; the change + // is the marker cleanup itself (codex-review r3). + const ownership = mutableOpencodeOwnership(cfg); + ownership.mcp = null; + ownership.managed = null; + return { ok: true, changed: true, detail: 'opencode.json absent — ownership markers cleared (nothing to strip)' }; + } + const { ok: parsedOk, doc } = readJsonStrict(configFile); + if (!parsedOk) { + // NOT ok: the ak wiring is still ACTIVE inside a file we refuse to parse, + // and the markers are the only teardown proof — keep both, fail honestly, + // and name the manual remediation. Never report "disabled" here, and never + // null the markers (codex-review r3). + return { + ok: false, changed: false, + detail: 'opencode.json is not plain JSON (JSONC comments?) — ak wiring left ACTIVE and ownership markers retained; remove the file or make it plain JSON, then re-run the teardown', + }; + } + const kept = []; + let changed = false; + + const restore = (obj, key, rec, label) => { + if (!obj || !(key in obj)) return; + if (rec.written == null) { kept.push(`${label} (not ak-written)`); return; } + if (!deepEqual(obj[key], rec.written)) { kept.push(`${label} (edited since ak wrote it)`); return; } + if (rec.prior == null) delete obj[key]; + else obj[key] = rec.prior; + changed = true; + }; + + for (const [name, rec] of Object.entries(managed.mcp)) restore(doc.mcp, name, rec, `mcp.${name}`); + if (doc.mcp && Object.keys(doc.mcp).length === 0) delete doc.mcp; + + if (restoreSkillPaths(doc, managed.paths)) changed = true; + + const scalarOrigin = managed.permissionScalar ?? null; + for (const [k, rec] of Object.entries(managed.permissions)) restore(doc.permission, k, rec, `permission.${k}`); + if (collapsePermissionScalar(doc, scalarOrigin)) changed = true; + + if (changed) writeJsonWithBackup(configFile, doc); + const ownership = mutableOpencodeOwnership(cfg); + ownership.mcp = null; + ownership.managed = null; + const detail = [ + changed ? 'ak-managed opencode.json wiring stripped (user priors restored)' : 'nothing managed found in opencode.json', + kept.length ? `left untouched: ${kept.join(', ')}` : null, + ].filter(Boolean).join(' — '); + return { ok: true, changed, detail }; +} + diff --git a/src/lib/opencode-lifecycle.mjs b/src/lib/opencode-lifecycle.mjs new file mode 100644 index 0000000..01532d5 --- /dev/null +++ b/src/lib/opencode-lifecycle.mjs @@ -0,0 +1,313 @@ +// opencode-lifecycle.mjs — the shared enable/retire stack composition and the +// ADR-0016 lifecycle adapter for OpenCode's managed native surfaces. Split +// out of opencode.mjs (ADR-0037's file-size gate) — behavior and export +// names are unchanged; opencode.mjs re-exports the externally-consumed +// names so no import path elsewhere in the repo needed to change. +import path from 'node:path'; +import * as paths from './paths.mjs'; +import { registry, syncBlocks, blocksForTarget, retiredForTarget, guidanceTargets } from './blocks.mjs'; +import { + opencodeOwnership, mutableOpencodeOwnership, applyOpencode, undoOpencode, + opencodeArtifactReceiptState, managedGatewayMcp, opencodeConverged, normalizeManaged, +} from './opencode-core.mjs'; +import { catalogSource, specialistDispatcherState, gatewayAgentCatalog, syncAgents, agentsStatus } from './opencode-agents.mjs'; +import { + deployPlugin, deployGatewayPlugin, retireGatewayPlugin, gatewayPluginStatus, pluginStatus, + deploySkill, skillStatus, removeArtifacts, +} from './opencode-artifacts.mjs'; + +/** @typedef {import('./opencode-agents.mjs').CatalogSource} CatalogSource */ + +// ── shared stack composition (the ONE owner-module operation) ──────────────── +// setup / sync / `ak host pick` all enable opencode the same way; off / +// uninstall / pick-disable all retire it the same way. The composition itself +// (which ops, in which order) is part of the ownership contract — three copies +// would drift (codex-review: the provider-picker rework must not duplicate +// merge/ownership logic in the command). Persistence of cfg stays with the +// CALLER (applyOpencode/undoOpencode mutate the ownership markers; the command +// decides when saveKitConfig runs). + +/** Snapshot of the ownership markers used to compute `markersChanged` + * (applyOpencode re-records them on every run, so callers must compare + * before/after rather than trust `oc.changed` alone). */ +function ownershipMarkersSnapshot(cfg) { + return JSON.stringify([opencodeOwnership(cfg).mcp ?? null, opencodeOwnership(cfg).managed ?? null]); +} + +/** The shape returned for every artifact surface when the receipt ledger is + * malformed — adoption is blocked fleet-wide until it's repaired by hand. */ +function blockedArtifactResults(receipts) { + const detail = 'skipped because the artifact receipt ledger is malformed'; + return { + plugin: { ok: false, changed: false, receipt: receipts.plugin, adoptionBlocked: true, detail }, + gateway: { ok: false, changed: false, receipt: receipts.gateway, adoptionBlocked: true, detail }, + agents: { + ok: false, changed: false, receipts: receipts.agents, + stampReceipt: receipts.agentStamp, adopted: 0, adoptionBlocked: true, detail, + }, + skill: { ok: false, changed: false, receipt: receipts.skill, adoptionBlocked: true, detail }, + }; +} + +/** Deploy/converge the plugin, lazy gateway, agent set, and platform skill — + * the non-blocked body of opencodeStack's enable path. + * @param {{ cfg: any, pkgRoot: string, source: CatalogSource|null, receiptState: any, configFile?: string, pluginsDir?: string, agentsDir?: string, skillsDir?: string }} args */ +function deployOpencodeArtifacts({ cfg, pkgRoot, source, receiptState, configFile, pluginsDir, agentsDir, skillsDir }) { + const { receipts, adoptionBlocked } = receiptState; + const managedMcp = managedGatewayMcp(cfg, { ...(configFile ? { configFile } : {}) }); + const dispatcher = specialistDispatcherState({ + destDir: agentsDir ?? paths.opencodeAgentsDir(), + receipts: receipts.agents, + adoptionBlocked: receiptState.agentsAdoptionBlocked, + }); + const agentCatalog = dispatcher.available ? gatewayAgentCatalog(source) : []; + const gatewayRequired = Object.keys(managedMcp).length > 0 || agentCatalog.length > 0; + const plugin = deployPlugin({ + pkgRoot, receipt: receipts.plugin, adoptionBlocked, + ...(pluginsDir ? { pluginsDir } : {}), + }); + const gateway = gatewayRequired + ? deployGatewayPlugin({ + pkgRoot, managedMcp, agentCatalog, receipt: receipts.gateway, adoptionBlocked, + ...(pluginsDir ? { pluginsDir } : {}), + }) + : retireGatewayPlugin({ + receipt: receipts.gateway, ...(pluginsDir ? { pluginsDir } : {}), + }); + const gatewayFacts = gatewayRequired + ? gatewayPluginStatus({ + pkgRoot, managedMcp, agentCatalog, + receipt: gateway.receipt ?? receipts.gateway ?? null, + ...(pluginsDir ? { pluginsDir } : {}), + }) + : { current: false }; + const gatewayCapabilities = { + ruflo: gatewayFacts.current && managedMcp['claude-flow'] != null, + aqe: gatewayFacts.current && managedMcp['agentic-qe'] != null, + }; + const agents = dispatcher.blocked + ? { + ok: false, changed: false, receipts: receipts.agents, + stampReceipt: receipts.agentStamp, adopted: 0, adoptionBlocked: true, + detail: 'specialist dispatcher receipt mismatch; agent projection preserved', + } + : syncAgents({ + source, receipts: receipts.agents, stampReceipt: receipts.agentStamp, + adoptionBlocked: receiptState.agentsAdoptionBlocked, + gatewayCapabilities, + lazyCatalog: gatewayFacts.current && agentCatalog.length > 0, + ...(agentsDir ? { destDir: agentsDir } : {}), + }); + const skill = deploySkill({ + source, receipt: receipts.skill, adoptionBlocked, + ...(skillsDir ? { skillsDir } : {}), + }); + return { plugin, gateway, agents, skill, gatewayRequired }; +} + +/** Enable path: wire opencode.json, deploy the lifecycle and lazy-catalogue + * plugins, convert the agent set, deploy the platform skill. Callers gate on the CLI being present + * first (have('opencode')) — this never fabricates the config home for an + * absent host. Returns each step's result for the caller's own formatting, + * plus `markersChanged`: applyOpencode re-records the ownership markers on + * EVERY run (a converged file with stale/missing markers in kit.json still + * needs persisting, or the next teardown cannot prove ownership) — callers + * must save cfg when `oc.changed || markersChanged`, not on `oc.changed` + * alone (codex-review r3). + * The destination seams exist for TESTS ONLY — production callers pass none + * and get the real config home; a test that forgets them writes to the + * developer's real machine (codex-review r4). + * @param {any} cfg @param {{ pkgRoot: string, configFile?: string, brainShim?: string, pluginsDir?: string, agentsDir?: string, skillsDir?: string }} opts */ +export async function opencodeStack(cfg, { pkgRoot, configFile, brainShim, pluginsDir, agentsDir, skillsDir }) { + const before = ownershipMarkersSnapshot(cfg); + const oc = await applyOpencode(cfg, { ...(configFile ? { configFile } : {}), ...(brainShim ? { brainShim } : {}) }); + if (oc.fatal) { + const skipped = { ok: false, changed: false, detail: 'skipped because opencode.json did not converge' }; + return { + oc, plugin: skipped, gateway: skipped, agents: skipped, skill: skipped, + source: null, markersChanged: false, + }; + } + const receiptState = opencodeArtifactReceiptState(opencodeOwnership(cfg).managed); + const { receipts, adoptionBlocked } = receiptState; + const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); + if (adoptionBlocked) { + return { + oc, ...blockedArtifactResults(receipts), source, + markersChanged: ownershipMarkersSnapshot(cfg) !== before, + }; + } + const { plugin, gateway, agents, skill, gatewayRequired } = deployOpencodeArtifacts({ + cfg, pkgRoot, source, receiptState, configFile, pluginsDir, agentsDir, skillsDir, + }); + if (!adoptionBlocked) { + mutableOpencodeOwnership(cfg).managed.artifacts = { + plugin: plugin.receipt ?? receipts.plugin ?? null, + gateway: gatewayRequired + ? (gateway.receipt ?? receipts.gateway ?? null) + : (gateway.ok ? null : (gateway.receipt ?? receipts.gateway ?? null)), + agents: agents.receipts ?? receipts.agents ?? {}, + agentStamp: agents.stampReceipt ?? receipts.agentStamp ?? null, + skill: skill.receipt ?? receipts.skill ?? null, + }; + } + return { oc, plugin, gateway, agents, skill, source, markersChanged: ownershipMarkersSnapshot(cfg) !== before }; +} + +/** Retire path: strip the ak-managed opencode.json wiring (user priors + * restored; collisions and user-edited values left), then remove ak-deployed + * artifacts (marker-gated — user-owned files survive). undoOpencode nulls the + * ownership markers in cfg on success and keeps them on failure; the caller + * persists — and MUST honor undo.ok before claiming a disable (codex-review + * r3: a JSONC-refused config leaves active wiring behind). + * @param {any} cfg */ +/** @param {any} cfg + * @param {{configFile?:string,pluginsDir?:string,agentsDir?:string,skillsDir?:string}} [opts] */ +export function retireOpencode(cfg, { configFile, pluginsDir, agentsDir, skillsDir } = {}) { + const receipts = normalizeManaged(opencodeOwnership(cfg).managed).artifacts; + const undo = undoOpencode(cfg, { ...(configFile ? { configFile } : {}) }); + const artifacts = undo.ok + ? removeArtifacts({ + receipts, + ...(pluginsDir ? { pluginsDir } : {}), + ...(agentsDir ? { agentsDir } : {}), + ...(skillsDir ? { skillsDir } : {}), + }) + : { ok: false, changed: false, detail: 'retained because opencode.json teardown is incomplete' }; + return { undo, artifacts, ok: undo.ok }; +} + +/** + * ADR-0016 lifecycle adapter for OpenCode's managed native surfaces. + * Configuration lifecycle is deliberately separate from activity routing: + * this adapter drives setup/sync/status/teardown while the host-neutral runner + * separately honors the registry's explicit `canRouteActivities:true`. + * + * The factory keeps filesystem destinations injectable for hermetic conformance + * tests. `detect`, `plan`, and `verify` are read-only. `runLifecycle` owns the + * dry-run boundary, so `apply` and `undo` are never called for a dry-run. + */ +export function createOpencodeLifecycleAdapter(defaults = {}) { + const options = (request) => ({ ...defaults, ...(request.options ?? {}) }); + const detect = async (request = {}) => { + const cfg = request.cfg ?? {}; + const opts = options(request); + const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); + const convergence = await opencodeConverged(cfg, { + ...(opts.configFile ? { configFile: opts.configFile } : {}), + ...(opts.brainShim ? { brainShim: opts.brainShim } : {}), + }); + const receiptState = opencodeArtifactReceiptState(opencodeOwnership(cfg).managed); + const { receipts, adoptionBlocked } = receiptState; + const managedMcp = managedGatewayMcp(cfg, { + ...(opts.configFile ? { configFile: opts.configFile } : {}), + }); + const dispatcher = specialistDispatcherState({ + destDir: opts.agentsDir ?? paths.opencodeAgentsDir(), + receipts: receipts.agents, + adoptionBlocked: receiptState.agentsAdoptionBlocked, + }); + const agentCatalog = dispatcher.available ? gatewayAgentCatalog(source) : []; + const gatewayRequired = Object.keys(managedMcp).length > 0 || agentCatalog.length > 0; + const plugin = opts.pkgRoot + ? pluginStatus({ + pkgRoot: opts.pkgRoot, receipt: receipts.plugin, adoptionBlocked, + ...(opts.pluginsDir ? { pluginsDir: opts.pluginsDir } : {}), + }) + : { present: false, current: false, foreign: false, adoptable: false }; + const gateway = opts.pkgRoot + ? gatewayPluginStatus({ + pkgRoot: opts.pkgRoot, managedMcp, agentCatalog, + receipt: receipts.gateway, adoptionBlocked, + ...(opts.pluginsDir ? { pluginsDir: opts.pluginsDir } : {}), + }) + : { present: false, current: false, foreign: false, adoptable: false }; + gateway.required = gatewayRequired; + const agents = agentsStatus({ + source, receipts: receipts.agents, stampReceipt: receipts.agentStamp, + adoptionBlocked: receiptState.agentsAdoptionBlocked || dispatcher.blocked, + gatewayCapabilities: { + ruflo: gateway.current && managedMcp['claude-flow'] != null, + aqe: gateway.current && managedMcp['agentic-qe'] != null, + }, + lazyCatalog: gateway.current && agentCatalog.length > 0, + ...(opts.agentsDir ? { destDir: opts.agentsDir } : {}), + }); + const skill = skillStatus({ + source, receipt: receipts.skill, adoptionBlocked, + ...(opts.skillsDir ? { skillsDir: opts.skillsDir } : {}), + }); + return { + enabled: !!cfg.integrations?.hosts?.opencode, + convergence, plugin, gateway, agents, skill, + }; + }; + return { + id: 'opencode', + detect, + async plan(request = {}) { + const facts = request.facts ?? await detect(request); + const changed = facts.enabled && (!facts.convergence.converged + || facts.plugin.adoptable || (!facts.plugin.current && !facts.plugin.foreign) + || (facts.gateway.required + && (facts.gateway.adoptable || (!facts.gateway.current && !facts.gateway.foreign))) + || (!facts.gateway.required && facts.gateway.present && !facts.gateway.foreign) + || (!facts.agents.adoptionBlocked && (facts.agents.adoptable || facts.agents.stale)) + || facts.skill.adoptable || (!facts.skill.current && !facts.skill.foreign)); + return { + changed, facts, + operations: changed ? ['config', 'plugin', 'gateway', 'agents', 'skill'] : [], + }; + }, + async apply(request = {}) { + const cfg = request.cfg; + const opts = options(request); + if (!cfg || !opts.pkgRoot) throw new TypeError('opencode lifecycle apply requires cfg and pkgRoot'); + const result = await opencodeStack(cfg, opts); + return { + changed: result.oc.changed || result.plugin.changed || result.gateway.changed || result.agents.changed + || result.skill.changed || result.markersChanged, + result, + }; + }, + async verify(request = {}) { + return detect(request); + }, + async undo(request = {}) { + if (!request.cfg) throw new TypeError('opencode lifecycle undo requires cfg'); + const result = retireOpencode(request.cfg, options(request)); + return { changed: result.undo.changed || result.artifacts.changed, result }; + }, + }; +} + +export const OPENCODE_LIFECYCLE_ADAPTER = createOpencodeLifecycleAdapter(); + +/** Reconcile the opencode AGENTS.md guidance blocks for the current enablement + * state — the `agents-opencode` target only, never the claude/project files. + * Enable (`enabled: true`) upserts the enablement-gated blocks as soon as the + * config home exists; disable (`enabled: false`) strips them (the always-on + * preamble stays by design; user content is never touched). Shared by setup, + * `ak host pick` enable/disable, and `ak host off`, so every command + * converges guidance the same way sync's blocks branch does (codex-review r3). + * @param {{ pkgRoot: string, cfg: any, cwd?: string, enabled: boolean }} opts */ +export async function reconcileOpencodeGuidance({ pkgRoot, cfg, cwd = process.cwd(), enabled }) { + const target = guidanceTargets({ cwd }).find((t) => t.name === 'agents-opencode'); + if (!target) return { ok: true, changed: false, detail: 'no opencode config home — guidance skipped' }; + const rows = registry(cfg.customBlocks); + const resolve = (r) => (r.custom + ? (r.template.startsWith('~/') ? path.join(paths.home, r.template.slice(2)) : r.template) + : path.join(pkgRoot, 'claude', r.template)); + const ctx = { + flags: { + dualMode: !!cfg.integrations?.hosts?.claude && !!cfg.integrations?.hosts?.codex, + opencodeEnabled: enabled, + }, + }; + const treg = [...blocksForTarget(rows, 'agents-opencode'), ...retiredForTarget(rows, 'agents-opencode')]; + const res = await syncBlocks(target.file, treg, resolve, { context: ctx }); + const changed = res.filter((r) => r.action !== 'unchanged' && r.action !== 'skipped') + .map((r) => `${r.slug} ${r.action}`); + return { ok: true, changed: changed.length > 0, detail: changed.length ? `guidance: ${changed.join(', ')}` : 'guidance in sync' }; +} + diff --git a/src/lib/opencode-receipts.mjs b/src/lib/opencode-receipts.mjs new file mode 100644 index 0000000..78e8938 --- /dev/null +++ b/src/lib/opencode-receipts.mjs @@ -0,0 +1,27 @@ +// opencode-receipts.mjs — shared receipt/comparison primitives for the +// opencode host adapter's config-wiring, agent-conversion, and artifact +// modules (opencode.mjs, opencode-agents.mjs, opencode-artifacts.mjs, +// opencode-lifecycle.mjs). Extracted so all four share ONE definition +// instead of re-deriving content hashing / deep-equality independently. +import { createHash } from 'node:crypto'; + +/** Order-insensitive deep compare (JSON with sorted keys). */ +export function deepEqual(a, b) { + const stable = (v) => JSON.stringify(v, (k, x) => ( + x && typeof x === 'object' && !Array.isArray(x) + ? Object.fromEntries(Object.entries(x).sort(([p], [q]) => p.localeCompare(q))) + : x + )); + return stable(a) === stable(b); +} + +export const contentHash = (text) => createHash('sha256').update(text).digest('hex'); +export const hasReceiptValue = (value) => value !== null && value !== undefined; +export const receiptMatches = (text, receipt) => + typeof receipt === 'string' && contentHash(text) === receipt; + +/** Tolerate a legacy/malformed `receipts` value (array, non-object) by + * treating it as an empty ledger — shared by syncAgents and agentsStatus. */ +export const asReceiptMap = (receipts) => ( + receipts && typeof receipts === 'object' && !Array.isArray(receipts) ? receipts : {} +); diff --git a/src/lib/opencode.mjs b/src/lib/opencode.mjs index 3f51e3a..9771fa8 100644 --- a/src/lib/opencode.mjs +++ b/src/lib/opencode.mjs @@ -1,10 +1,10 @@ -// opencode host integration — the third host adapter's I/O half. +// opencode.mjs — the third host adapter's I/O half (public entry point). // // why: opencode (opencode.ai) consumes the same rUv stack as claude/codex but -// through different surfaces. This module owns every ak-managed byte on those -// surfaces, backup-first + merge-not-clobber + ownership-marked, mirroring the -// claude (settings.mjs / mcp.mjs) and codex (providers.mjs Ruflo integration) -// contracts: +// through different surfaces. This module family owns every ak-managed byte +// on those surfaces, backup-first + merge-not-clobber + ownership-marked, +// mirroring the claude (settings.mjs / mcp.mjs) and codex (providers.mjs +// Ruflo integration) contracts: // // ~/.config/opencode/opencode.json mcp.claude-flow + mcp.agentic-qe + // mcp.ruvnet-brain, @@ -21,12 +21,34 @@ // profiles stay embedded and load lazily // ~/.config/opencode/skills/ruflo/ the platform SKILL.md // +// This file is the STABLE PUBLIC ENTRY POINT every consumer already imports +// ('./opencode.mjs' / '../lib/opencode.mjs' / etc.) — it re-exports the +// public surface of four implementation modules (ADR-0037's file-size gate +// split this file, which had grown past 1,700 lines, along its natural +// seams): +// +// opencode-core.mjs config-wiring: applyOpencode/undoOpencode, +// opencodeConverged/opencodeMcpStatus, the receipt +// ledger (opencodeArtifactReceiptState), mcp entry +// resolution (mcpEntriesFor). +// opencode-agents.mjs ruflo catalog resolution (catalogSource, +// skillPathsFor) + the Claude Code agent .md → +// OpenCode subagent .md conversion/sync/status +// pipeline (convertAgents/syncAgents/agentsStatus). +// opencode-artifacts.mjs the plugin (lifecycle bridge + lazy rUv gateway) +// and platform-skill deployment/status, plus the +// shared teardown (removeArtifacts). +// opencode-lifecycle.mjs the shared enable/retire stack composition +// (opencodeStack/retireOpencode) and the ADR-0016 +// lifecycle adapter (createOpencodeLifecycleAdapter). +// // Grounded: // - opencode.json schema (https://opencode.ai/config.json): mcp local // servers {type,command[],environment,enabled,timeout}, skills.paths[], // permission as wildcard tool-name patterns (MCP tools surface as // `_`, hence the claude-flow_*/agentic-qe_*/ruvnet-brain_* patterns). -// - ruflo's own init/mcp-generator.ts env block (CLAUDE_FLOW_* below). +// - ruflo's own init/mcp-generator.ts env block (CLAUDE_FLOW_* in +// opencode-core.mjs). // - `claude-flow-mcp` (the dedicated stdio bin of @claude-flow/cli) answers // initialize directly; `ruflo mcp start` is the fallback (what ak already // registers for claude/codex) when that bin is absent. @@ -34,1540 +56,19 @@ // hot-swaps brain versions — the registration never needs rewriting. // - opencode.json may legally contain JSONC comments ($schema allowComments): // a file we cannot parse is REFUSED, never clobbered. -import fs from 'node:fs'; -import path from 'node:path'; -import { createHash } from 'node:crypto'; -import { have } from './exec.mjs'; -import { readJson, writeJsonWithBackup } from './settings.mjs'; -import { registry, syncBlocks, blocksForTarget, retiredForTarget, guidanceTargets } from './blocks.mjs'; -import { CURRENT_INTEGRATIONS_VERSION } from './adapters/config.mjs'; -import * as paths from './paths.mjs'; - -const opencodeOwnership = (cfg) => cfg?.integrations?.ownership?.opencode ?? {}; -function mutableOpencodeOwnership(cfg) { - cfg.integrations ??= { - version: CURRENT_INTEGRATIONS_VERSION, - hosts: {}, - bindings: [], - }; - cfg.integrations.ownership ??= {}; - cfg.integrations.ownership.opencode ??= {}; - return cfg.integrations.ownership.opencode; -} - -// ── config-file wiring (opencode.json) ────────────────────────────────────── - -/** ruflo init/mcp-generator.ts's env block, mirrored for parity. */ -export const RUFLO_MCP_ENV = { - npm_config_update_notifier: 'false', - CLAUDE_FLOW_MODE: 'v3', - CLAUDE_FLOW_HOOKS_ENABLED: 'true', - CLAUDE_FLOW_TOPOLOGY: 'hierarchical-mesh', - CLAUDE_FLOW_MAX_AGENTS: '15', - CLAUDE_FLOW_MEMORY_BACKEND: 'hybrid', -}; - -/** agentic-qe init's project MCP environment, mirrored for OpenCode. */ -export const AQE_MCP_ENV = { - AQE_LEARNING_ENABLED: 'true', - AQE_WORKERS_ENABLED: 'true', - NODE_ENV: 'production', -}; - -/** Permission patterns follow the rUv capabilities AK actually projects. */ -function permissionFamiliesFor(entries) { - return [ - ...('claude-flow' in entries ? [['claude-flow_*', 'claude_flow_*']] : []), - ...('agentic-qe' in entries ? [['agentic-qe_*', 'agentic_qe_*']] : []), - ...('ruvnet-brain' in entries ? [['ruvnet-brain_*', 'ruvnet_brain_*']] : []), - ]; -} - -function permissionKeysFor(entries) { - return permissionFamiliesFor(entries).flat(); -} - -/** The brain's stable-spine shim (same registration codex carries). */ -export const brainShimPath = () => path.join(paths.home, '.claude', 'ruvnet-brain', 'mcp', 'server.mjs'); - -/** The dedicated stdio MCP server bundled inside a plain `npm i -g ruflo` - * install (nested dependency — present even when no claude-flow-mcp bin is - * on PATH). */ -export const nestedMcpServerPath = () => - path.join(paths.rufloNodeModules(), '@claude-flow', 'cli', 'bin', 'mcp-server.js'); - -/** The claude-flow MCP command, best-available-first: the claude-flow-mcp bin - * on PATH → the nested mcp-server.js via absolute node path (no PATH/cwd - * dependence — the fresh ruflo-only machine case) → `ruflo mcp start` (ak's - * claude/codex registration path, always present when ruflo is). Pure. */ -export function mcpCommandFor({ binPresent, nestedPath }) { - if (binPresent) return ['claude-flow-mcp']; - if (nestedPath && fs.existsSync(nestedPath)) return ['node', nestedPath]; - return ['ruflo', 'mcp', 'start']; -} - -/** @typedef {{ kind: string, root: string, id: string, hasPlugins: boolean, hasPlatformSkill: boolean }} CatalogSource */ - -/** The MCP server entries ak writes. `claude-flow` resolves via mcpCommandFor - * (bin on PATH → nested mcp-server.js → `ruflo mcp start`). Agentic QE is - * included by default because machine setup installs it; `--no-aqe` disables - * that projection. ruvnet-brain is included only when its shim is on disk. - * @param {{ brainShim?: string, nestedPath?: string, includeAqe?: boolean }} [opts] */ -export async function mcpEntriesFor({ - brainShim = brainShimPath(), nestedPath = nestedMcpServerPath(), includeAqe = true, -} = {}) { - const entries = { - 'claude-flow': { - type: 'local', - command: mcpCommandFor({ binPresent: await have('claude-flow-mcp'), nestedPath }), - enabled: true, - timeout: 30000, - environment: { ...RUFLO_MCP_ENV }, - }, - }; - if (includeAqe) { - entries['agentic-qe'] = { - type: 'local', - command: ['aqe-mcp'], - enabled: true, - timeout: 30000, - environment: { ...AQE_MCP_ENV }, - }; - } - if (fs.existsSync(brainShim)) { - entries['ruvnet-brain'] = { type: 'local', command: ['node', brainShim], enabled: true, timeout: 30000 }; - } - return entries; -} - -/** Strict read: distinguishes "absent/empty" from "present but not plain JSON" - * (opencode.json may legally be JSONC). NB: settings.readJson's fallback - * parameter can't express this — passing undefined re-triggers its default. */ -function readJsonStrict(file) { - try { - const raw = fs.readFileSync(file, 'utf8'); - if (!raw.trim()) return { ok: true, doc: {} }; - return { ok: true, doc: JSON.parse(raw) }; - } catch { - return { ok: false, doc: null }; - } -} - -/** OpenCode loads opencode.jsonc after opencode.json. A sibling JSONC file can - * shadow managed MCP, permission, or plugin values, and AK deliberately does - * not normalize or rewrite user comments. */ -function laterJsoncOverride(configFile) { - if (path.basename(configFile) !== 'opencode.json') return null; - const candidate = path.join(path.dirname(configFile), 'opencode.jsonc'); - return fs.existsSync(candidate) ? candidate : null; -} - -/** Registration state, spawn-free (mirrors mcp.mjs registrationStatus's - * file-read approach). `parseError` distinguishes "absent" from "present but - * not plain JSON" (JSONC) — the writer refuses the latter. - * @param {any} cfg @param {{ configFile?: string }} [opts] */ -export function opencodeMcpStatus(cfg, { configFile = paths.opencodeConfigPath() } = {}) { - const exists = fs.existsSync(configFile); - const laterOverride = laterJsoncOverride(configFile); - const { ok, doc } = exists ? readJsonStrict(configFile) : { ok: true, doc: {} }; - if (!ok) { - return { - exists, parseError: true, laterOverride, claudeFlow: false, aqe: false, brain: false, - owned: opencodeOwnership(cfg).mcp === 'ak', - }; - } - return { - exists, - parseError: false, - laterOverride, - claudeFlow: !!doc?.mcp?.['claude-flow'], - aqe: !!doc?.mcp?.['agentic-qe'], - brain: !!doc?.mcp?.['ruvnet-brain'], - paths: doc?.skills?.paths ?? [], - owned: opencodeOwnership(cfg).mcp === 'ak', - }; -} - -/** Convergence check — deeper than key existence (codex-review #16): the MCP - * entries must EQUAL today's desired values (command/env/timeout drift when - * the user edits them or a kit upgrade changes the template), desired skills - * paths must all be present, desired permission patterns must be 'allow', and - * a ruvnet-brain entry whose shim has vanished is stale. Async because the - * desired entries probe the claude-flow-mcp bin (one `which`, matching the - * spawn profile of status's hosts rows). - * @param {any} cfg @param {{ configFile?: string, brainShim?: string }} [opts] */ -export async function opencodeConverged(cfg, { configFile = paths.opencodeConfigPath(), brainShim } = {}) { - const st = opencodeMcpStatus(cfg, { configFile }); - if (!st.exists || st.parseError) return { converged: false, reasons: st.parseError ? ['unparseable config'] : ['no config file'] }; - if (st.laterOverride) { - return { - converged: false, - reasons: [`later OpenCode config override is unverified: ${st.laterOverride}`], - }; - } - const doc = readJsonStrict(configFile).doc; - const reasons = []; - const entries = await mcpEntriesFor({ brainShim, includeAqe: cfg.aqe !== false }); - for (const [name, want] of Object.entries(entries)) { - if (!(name in (doc.mcp ?? {}))) reasons.push(`${name} missing`); - else if (!deepEqual(doc.mcp[name], want)) reasons.push(`${name} drifted`); - } - const managed = normalizeManaged(opencodeOwnership(cfg).managed); - const ownedEntries = Object.fromEntries(Object.entries(entries).filter( - ([name]) => managed.mcp[name]?.written != null, - )); - const permissionKeys = permissionKeysFor(ownedEntries); - for (const [name, rec] of Object.entries(managed.mcp)) { - if (name in entries || rec.written == null) continue; - if (deepEqual(doc.mcp?.[name], rec.written)) reasons.push(`${name} stale (no longer desired)`); - } - const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); - for (const p of skillPathsFor(source)) { - if (!(doc.skills?.paths ?? []).includes(p)) reasons.push(`skills path missing: ${p}`); - } - for (const k of permissionKeys) { - if (doc.permission?.[k] !== 'allow') reasons.push(`permission ${k} not allowed`); - } - for (const [key, rec] of Object.entries(managed.permissions)) { - if (permissionKeys.includes(key) || rec.written == null) continue; - if (deepEqual(doc.permission?.[key], rec.written)) reasons.push(`permission ${key} stale (no longer desired)`); - } - return { converged: reasons.length === 0, reasons }; -} - -/** Order-insensitive deep compare (JSON with sorted keys). */ -function deepEqual(a, b) { - const stable = (v) => JSON.stringify(v, (k, x) => ( - x && typeof x === 'object' && !Array.isArray(x) - ? Object.fromEntries(Object.entries(x).sort(([p], [q]) => p.localeCompare(q))) - : x - )); - return stable(a) === stable(b); -} - -const contentHash = (text) => createHash('sha256').update(text).digest('hex'); -const hasReceiptValue = (value) => value !== null && value !== undefined; -const receiptMatches = (text, receipt) => - typeof receipt === 'string' && contentHash(text) === receipt; - -/** Normalize an opencodeManaged record — the current precise shape - * { mcp: {name:{prior,written}}, paths: [], permissions: {key:{prior,written}} }, - * tolerating the legacy names-only shape from the first shipped version - * (legacy entries have unknown prior/written → treated conservatively: prior - * null, written null → never auto-deleted, only re-recorded on next apply). */ -function normalizeManaged(m) { - const out = { - mcp: {}, paths: [], permissions: {}, permissionScalar: null, - artifacts: { plugin: null, gateway: null, agents: {}, agentStamp: null, skill: null }, - artifactState: { - containerMalformed: false, agentsMalformed: false, - rawContainer: null, - }, - }; - if (!m || typeof m !== 'object') return out; - const legacyNames = Array.isArray(m.mcp) ? m.mcp : Object.keys(m.mcp ?? {}); - for (const n of legacyNames) { - const rec = Array.isArray(m.mcp) ? null : m.mcp[n]; - out.mcp[n] = rec && typeof rec === 'object' && 'written' in rec ? rec : { prior: null, written: null }; - } - out.paths = Array.isArray(m.paths) ? [...m.paths] : []; - const permKeys = Array.isArray(m.permissions) ? m.permissions : Object.keys(m.permissions ?? {}); - for (const k of permKeys) { - const rec = Array.isArray(m.permissions) ? null : m.permissions[k]; - out.permissions[k] = rec && typeof rec === 'object' && 'written' in rec ? rec : { prior: null, written: null }; - } - out.permissionScalar = typeof m.permissionScalar === 'string' ? m.permissionScalar : null; - if (hasReceiptValue(m.artifacts)) { - out.artifactState.rawContainer = structuredClone(m.artifacts); - } - if (hasReceiptValue(m.artifacts) - && (typeof m.artifacts !== 'object' || Array.isArray(m.artifacts))) { - out.artifactState.containerMalformed = true; - return out; - } - const artifacts = m.artifacts ?? {}; - out.artifacts.plugin = hasReceiptValue(artifacts.plugin) ? artifacts.plugin : null; - out.artifacts.gateway = hasReceiptValue(artifacts.gateway) ? artifacts.gateway : null; - out.artifacts.agentStamp = hasReceiptValue(artifacts.agentStamp) ? artifacts.agentStamp : null; - out.artifacts.skill = hasReceiptValue(artifacts.skill) ? artifacts.skill : null; - if (hasReceiptValue(artifacts.agents) - && (typeof artifacts.agents !== 'object' || Array.isArray(artifacts.agents))) { - out.artifactState.agentsMalformed = true; - } else if (artifacts.agents) { - out.artifacts.agents = Object.fromEntries( - Object.entries(artifacts.agents).filter(([, hash]) => hasReceiptValue(hash)), - ); - } - return out; -} - -/** Read-only artifact receipt boundary shared by lifecycle and status paths. - * Only null/absent containers represent a pre-receipts migration gap; - * malformed non-null containers fail closed and block adoption. */ -export function opencodeArtifactReceiptState(managed) { - const normalized = normalizeManaged(managed); - const blocked = normalized.artifactState.containerMalformed - || normalized.artifactState.agentsMalformed; - return { - receipts: normalized.artifacts, - adoptionBlocked: blocked, - agentsAdoptionBlocked: blocked, - }; -} - -/** Exact MCP values the lazy gateway may capture. A same-name entry is not - * enough: the command and both direct permission spellings must still match - * values positively recorded as AK-written. Explicit direct-tool enablement - * is an operator opt-out from lazy capture. */ -export function managedGatewayMcp(cfg, { configFile = paths.opencodeConfigPath() } = {}) { - const managed = normalizeManaged(opencodeOwnership(cfg).managed); - const parsed = fs.existsSync(configFile) ? readJsonStrict(configFile) : { ok: true, doc: {} }; - const tools = parsed.ok ? (parsed.doc?.tools ?? {}) : {}; - const permissions = parsed.ok && typeof parsed.doc?.permission === 'object' - ? parsed.doc.permission - : {}; - const families = { - 'claude-flow': ['claude-flow_*', 'claude_flow_*'], - 'agentic-qe': ['agentic-qe_*', 'agentic_qe_*'], - }; - return Object.fromEntries(Object.entries(families) - .filter(([name, keys]) => managed.mcp[name]?.written != null - && keys.every((key) => managed.permissions[key]?.written === 'allow' - && permissions[key] === 'allow') - && !keys.some((key) => tools[key] === true)) - .map(([name]) => [name, structuredClone(managed.mcp[name].written)])); -} - -/** Reconcile opencode.json: ak's MCP servers, skills.paths, and permission - * patterns merged into whatever is already there. The ownership contract is - * VALUE-PRECISE, not name-precise: - * - a pre-existing entry that DIFFERS from ak's desired value (and was not - * previously ak-written) is a COLLISION: preserved, reported, never taken - * over — merge-not-clobber applies to values, not just files; - * - every key ak writes is recorded as {prior, written} so teardown can - * restore the user's original value instead of deleting it; - * - previously-managed keys that fall OUT of the desired set (brain shim - * removed, catalog source changed) are removed only while they still equal - * what ak wrote — a user-edited value is left and reported. - * Scalar `permission` ("permission":"allow") is first lifted to its - * documented object equivalent {"*":"allow"} (wildcard key semantics), never - * spread character-by-character. Backup-first, idempotent, JSONC-refusing. - * @param {any} cfg @param {{ dryRun?: boolean, configFile?: string, brainShim?: string }} [opts] */ -export async function applyOpencode(cfg, { dryRun = false, configFile = paths.opencodeConfigPath(), brainShim } = {}) { - if (!cfg.integrations?.hosts?.opencode) return { ok: true, changed: false, detail: 'opencode not enabled — unmanaged' }; - const laterOverride = laterJsoncOverride(configFile); - if (laterOverride) { - return { - ok: false, fatal: true, changed: false, - detail: `${laterOverride} loads after opencode.json — refusing to write or claim an unverified effective config; merge the Agentic Kit entries there manually or remove the override`, - }; - } - const exists = fs.existsSync(configFile); - const { ok: parsedOk, doc } = exists ? readJsonStrict(configFile) : { ok: true, doc: {} }; - if (!parsedOk) { - return { - ok: false, fatal: true, changed: false, - detail: `${configFile} is not plain JSON (JSONC comments?) — refusing to touch it; merge manually`, - }; - } - const entries = await mcpEntriesFor({ brainShim, includeAqe: cfg.aqe !== false }); - const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); - const skillPaths = skillPathsFor(source); - const prevManaged = normalizeManaged(opencodeOwnership(cfg).managed); - const collisions = []; - const pruned = []; - - const next = JSON.parse(JSON.stringify(doc)); - next.$schema ??= 'https://opencode.ai/config.json'; - - // ── mcp: prune stale ak entries, then merge desired with collision refusal ── - next.mcp = { ...(next.mcp ?? {}) }; - for (const [name, rec] of Object.entries(prevManaged.mcp)) { - if (name in entries) continue; - if (!(name in next.mcp)) continue; - if (rec.written && deepEqual(next.mcp[name], rec.written)) { - // RESTORE the prior when there was one (a user entry that happened to - // equal the old desired value is a user value, not ak's to delete); - // delete only what ak itself created (codex-review r2). - if (rec.prior != null) { next.mcp[name] = rec.prior; pruned.push(`${name} (prior restored)`); } - else { delete next.mcp[name]; pruned.push(name); } - } // else: user edited (or legacy record) → leave it, keep no ownership - } - const managed = { - mcp: {}, paths: [], permissions: {}, permissionScalar: null, - artifacts: (prevManaged.artifactState.containerMalformed - || prevManaged.artifactState.agentsMalformed) - ? structuredClone(prevManaged.artifactState.rawContainer) - : structuredClone(prevManaged.artifacts), - }; - for (const [name, want] of Object.entries(entries)) { - const cur = next.mcp[name]; - const priorRec = prevManaged.mcp[name]; - if (cur !== undefined && !deepEqual(cur, want) && !(priorRec?.written && deepEqual(cur, priorRec.written))) { - collisions.push(`mcp.${name}`); - managed.mcp[name] = { prior: cur, written: null }; // tracked but NOT ak-owned - continue; - } - // A previously-colliding (never ak-authored) value the USER has since - // aligned to the desired one stays unmanaged: adopting it with the stale - // pre-collision prior would make undo overwrite the user's own later - // choice (codex-review r2). Noted, not owned. - if (priorRec && priorRec.written == null && cur !== undefined && deepEqual(cur, want)) { - managed.mcp[name] = { prior: priorRec.prior, written: null }; - continue; - } - // prior is the ORIGINAL pre-ak value (kept across reapplies), never the - // ak-written value currently in place. - managed.mcp[name] = { prior: priorRec ? priorRec.prior : (cur ?? null), written: want }; - next.mcp[name] = want; - } - - // ── skills.paths: remove stale ak-added paths, add desired ── - if (next.skills?.paths && prevManaged.paths.length) { - const stale = new Set(prevManaged.paths.filter((p) => !skillPaths.includes(p))); - next.skills.paths = next.skills.paths.filter((p) => !stale.has(p)); - if (stale.size) pruned.push(`${stale.size} stale skills path(s)`); - } - if (skillPaths.length) { - next.skills = { ...(next.skills ?? {}) }; - const cur = new Set(next.skills.paths ?? []); - const newlyAdded = skillPaths.filter((p) => !cur.has(p)); - // ownership = previously-recorded ak paths that are still desired + newly - // added ones (a re-apply must not erase the record of what ak added). - managed.paths = [...new Set([...prevManaged.paths.filter((p) => skillPaths.includes(p)), ...newlyAdded])]; - next.skills.paths = [...cur, ...newlyAdded]; - } - - // ── permission: lift scalar shorthand, prune stale, merge desired ── - // Record scalar ORIGIN explicitly (codex-review r2): undo restores the - // scalar form only when the file actually started scalar — a pre-existing - // {"*":"ask"} object must survive as an object, not be "restored" to "ask". - managed.permissionScalar = typeof doc.permission === 'string' - ? doc.permission - : (prevManaged.permissionScalar ?? null); - if (typeof next.permission === 'string') next.permission = { '*': next.permission }; - next.permission = { ...(next.permission ?? {}) }; - - // Permissions are family-atomic with MCP ownership. A foreign/colliding - // same-name MCP must never inherit broad AK-written `allow` patterns, and a - // collision on either spelling prevents AK from adding the other spelling. - const ownedEntries = Object.fromEntries(Object.entries(entries).filter( - ([name]) => managed.mcp[name]?.written != null, - )); - const permissionKeys = []; - for (const keys of permissionFamiliesFor(ownedEntries)) { - const blocked = keys.some((key) => { - const cur = next.permission[key]; - const priorRec = prevManaged.permissions[key]; - const conflicts = cur !== undefined && cur !== 'allow' - && !(priorRec?.written && deepEqual(cur, priorRec.written)); - const previouslyUnowned = cur !== undefined && priorRec && priorRec.written == null; - return conflicts || previouslyUnowned; - }); - if (!blocked) { - permissionKeys.push(...keys); - continue; - } - for (const key of keys) { - const cur = next.permission[key]; - const priorRec = prevManaged.permissions[key]; - if (cur !== undefined && cur !== 'allow' - && !(priorRec?.written && deepEqual(cur, priorRec.written))) { - collisions.push(`permission.${key}`); - } - if (cur !== undefined) { - managed.permissions[key] = { - prior: priorRec ? priorRec.prior : cur, - written: null, - }; - } - } - } - for (const [k, rec] of Object.entries(prevManaged.permissions)) { - if (permissionKeys.includes(k)) continue; - if (!(k in next.permission)) continue; - if (rec.written && deepEqual(next.permission[k], rec.written)) { - if (rec.prior != null) { - next.permission[k] = rec.prior; - pruned.push(`permission.${k} (prior restored)`); - } else { - delete next.permission[k]; - pruned.push(`permission.${k}`); - } - } - } - for (const k of permissionKeys) { - const cur = next.permission[k]; - const priorRec = prevManaged.permissions[k]; - managed.permissions[k] = { prior: priorRec ? priorRec.prior : (cur ?? null), written: 'allow' }; - next.permission[k] = 'allow'; - } - - const changed = JSON.stringify(next) !== JSON.stringify(doc); - if (!dryRun) { - const ownership = mutableOpencodeOwnership(cfg); - ownership.mcp = 'ak'; - ownership.managed = managed; - } - if (changed && !dryRun) writeJsonWithBackup(configFile, next); - const aqe = entries['agentic-qe'] ? ' + agentic-qe' : ''; - const brain = entries['ruvnet-brain'] ? ' + ruvnet-brain' : ' (brain shim absent)'; - const notes = [ - changed ? `opencode.json wired: claude-flow (${entries['claude-flow'].command.join(' ')})${aqe}${brain}, ${skillPaths.length} skills path(s), ${permissionKeys.length} permission pattern(s)` - : `opencode.json in sync${source ? '' : ' — ⚠ no ruflo catalog source found for skills.paths'}`, - ]; - if (pruned.length) notes.push(`pruned: ${pruned.join(', ')}`); - if (collisions.length) notes.push(`⚠ collisions preserved (user-owned, untouched): ${collisions.join(', ')}`); - return { - ok: collisions.length === 0, - fatal: false, - changed, - collisions, - detail: notes.join(' — '), - }; -} - -/** Surgical teardown of ak's opencode.json wiring — ONLY the recorded managed - * keys, and ONLY when ak wrote them - * (`integrations.ownership.opencode.mcp === 'ak'`). For each - * managed key: when the current value still equals what ak wrote, the user's - * PRIOR value is restored (or the key removed if there was none); a value the - * user edited since is left and reported, never silently deleted. Scalar - * permission shorthand is restored to scalar when teardown empties the object - * but a prior '*' wildcard exists. Deployed artifacts are removed separately - * (removeArtifacts). - * @param {any} cfg @param {{ configFile?: string }} [opts] */ -export function undoOpencode(cfg, { configFile = paths.opencodeConfigPath() } = {}) { - if (opencodeOwnership(cfg).mcp !== 'ak') { - return { ok: true, changed: false, detail: 'opencode.json left as-is (not ak-managed)' }; - } - const managed = normalizeManaged(opencodeOwnership(cfg).managed); - if (!fs.existsSync(configFile)) { - // Nothing left to strip — but the markers would otherwise survive as a lie - // (a later teardown would chase a phantom config). Clear them; the change - // is the marker cleanup itself (codex-review r3). - const ownership = mutableOpencodeOwnership(cfg); - ownership.mcp = null; - ownership.managed = null; - return { ok: true, changed: true, detail: 'opencode.json absent — ownership markers cleared (nothing to strip)' }; - } - const { ok: parsedOk, doc } = readJsonStrict(configFile); - if (!parsedOk) { - // NOT ok: the ak wiring is still ACTIVE inside a file we refuse to parse, - // and the markers are the only teardown proof — keep both, fail honestly, - // and name the manual remediation. Never report "disabled" here, and never - // null the markers (codex-review r3). - return { - ok: false, changed: false, - detail: 'opencode.json is not plain JSON (JSONC comments?) — ak wiring left ACTIVE and ownership markers retained; remove the file or make it plain JSON, then re-run the teardown', - }; - } - const kept = []; - let changed = false; - - const restore = (obj, key, rec, label) => { - if (!obj || !(key in obj)) return; - if (rec.written == null) { kept.push(`${label} (not ak-written)`); return; } - if (!deepEqual(obj[key], rec.written)) { kept.push(`${label} (edited since ak wrote it)`); return; } - if (rec.prior == null) delete obj[key]; - else obj[key] = rec.prior; - changed = true; - }; - - for (const [name, rec] of Object.entries(managed.mcp)) restore(doc.mcp, name, rec, `mcp.${name}`); - if (doc.mcp && Object.keys(doc.mcp).length === 0) delete doc.mcp; - - if (doc.skills?.paths && managed.paths.length) { - const drop = new Set(managed.paths); - const keptPaths = doc.skills.paths.filter((p) => !drop.has(p)); - if (keptPaths.length !== doc.skills.paths.length) { - changed = true; - if (keptPaths.length) doc.skills.paths = keptPaths; - else { delete doc.skills.paths; if (Object.keys(doc.skills).length === 0) delete doc.skills; } - } - } - - const scalarOrigin = managed.permissionScalar ?? null; - for (const [k, rec] of Object.entries(managed.permissions)) restore(doc.permission, k, rec, `permission.${k}`); - if (doc.permission && Object.keys(doc.permission).length === 0) delete doc.permission; - else if (doc.permission && scalarOrigin != null && Object.keys(doc.permission).length === 1 && doc.permission['*'] != null) { - // restore the scalar shorthand we lifted (only when scalar was the ORIGIN; - // the current '*' value is what collapses back — '*' is never ak-managed) - doc.permission = doc.permission['*']; - changed = true; - } - - if (changed) writeJsonWithBackup(configFile, doc); - const ownership = mutableOpencodeOwnership(cfg); - ownership.mcp = null; - ownership.managed = null; - const detail = [ - changed ? 'ak-managed opencode.json wiring stripped (user priors restored)' : 'nothing managed found in opencode.json', - kept.length ? `left untouched: ${kept.join(', ')}` : null, - ].filter(Boolean).join(' — '); - return { ok: true, changed, detail }; -} - -// ── shared stack composition (the ONE owner-module operation) ──────────────── -// setup / sync / `ak host pick` all enable opencode the same way; off / -// uninstall / pick-disable all retire it the same way. The composition itself -// (which ops, in which order) is part of the ownership contract — three copies -// would drift (codex-review: the provider-picker rework must not duplicate -// merge/ownership logic in the command). Persistence of cfg stays with the -// CALLER (applyOpencode/undoOpencode mutate the ownership markers; the command -// decides when saveKitConfig runs). - -/** Enable path: wire opencode.json, deploy the lifecycle and lazy-catalogue - * plugins, convert the agent set, deploy the platform skill. Callers gate on the CLI being present - * first (have('opencode')) — this never fabricates the config home for an - * absent host. Returns each step's result for the caller's own formatting, - * plus `markersChanged`: applyOpencode re-records the ownership markers on - * EVERY run (a converged file with stale/missing markers in kit.json still - * needs persisting, or the next teardown cannot prove ownership) — callers - * must save cfg when `oc.changed || markersChanged`, not on `oc.changed` - * alone (codex-review r3). - * The destination seams exist for TESTS ONLY — production callers pass none - * and get the real config home; a test that forgets them writes to the - * developer's real machine (codex-review r4). - * @param {any} cfg @param {{ pkgRoot: string, configFile?: string, brainShim?: string, pluginsDir?: string, agentsDir?: string, skillsDir?: string }} opts */ -export async function opencodeStack(cfg, { pkgRoot, configFile, brainShim, pluginsDir, agentsDir, skillsDir }) { - const before = JSON.stringify([ - opencodeOwnership(cfg).mcp ?? null, - opencodeOwnership(cfg).managed ?? null, - ]); - const oc = await applyOpencode(cfg, { ...(configFile ? { configFile } : {}), ...(brainShim ? { brainShim } : {}) }); - if (oc.fatal) { - const skipped = { ok: false, changed: false, detail: 'skipped because opencode.json did not converge' }; - return { - oc, plugin: skipped, gateway: skipped, agents: skipped, skill: skipped, - source: null, markersChanged: false, - }; - } - const receiptState = opencodeArtifactReceiptState(opencodeOwnership(cfg).managed); - const { receipts, adoptionBlocked } = receiptState; - const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); - if (adoptionBlocked) { - const detail = 'skipped because the artifact receipt ledger is malformed'; - const plugin = { ok: false, changed: false, receipt: receipts.plugin, adoptionBlocked: true, detail }; - const gateway = { ok: false, changed: false, receipt: receipts.gateway, adoptionBlocked: true, detail }; - const agents = { - ok: false, changed: false, receipts: receipts.agents, - stampReceipt: receipts.agentStamp, adopted: 0, adoptionBlocked: true, detail, - }; - const skill = { ok: false, changed: false, receipt: receipts.skill, adoptionBlocked: true, detail }; - const markersChanged = JSON.stringify([ - opencodeOwnership(cfg).mcp ?? null, - opencodeOwnership(cfg).managed ?? null, - ]) !== before; - return { oc, plugin, gateway, agents, skill, source, markersChanged }; - } - const managedMcp = managedGatewayMcp(cfg, { ...(configFile ? { configFile } : {}) }); - const dispatcher = specialistDispatcherState({ - destDir: agentsDir ?? paths.opencodeAgentsDir(), - receipts: receipts.agents, - adoptionBlocked: receiptState.agentsAdoptionBlocked, - }); - const agentCatalog = dispatcher.available ? gatewayAgentCatalog(source) : []; - const gatewayRequired = Object.keys(managedMcp).length > 0 || agentCatalog.length > 0; - const plugin = deployPlugin({ - pkgRoot, receipt: receipts.plugin, adoptionBlocked, - ...(pluginsDir ? { pluginsDir } : {}), - }); - const gateway = gatewayRequired - ? deployGatewayPlugin({ - pkgRoot, managedMcp, agentCatalog, receipt: receipts.gateway, adoptionBlocked, - ...(pluginsDir ? { pluginsDir } : {}), - }) - : retireGatewayPlugin({ - receipt: receipts.gateway, ...(pluginsDir ? { pluginsDir } : {}), - }); - const gatewayFacts = gatewayRequired - ? gatewayPluginStatus({ - pkgRoot, managedMcp, agentCatalog, - receipt: gateway.receipt ?? receipts.gateway ?? null, - ...(pluginsDir ? { pluginsDir } : {}), - }) - : { current: false }; - const gatewayCapabilities = { - ruflo: gatewayFacts.current && managedMcp['claude-flow'] != null, - aqe: gatewayFacts.current && managedMcp['agentic-qe'] != null, - }; - const agents = dispatcher.blocked - ? { - ok: false, changed: false, receipts: receipts.agents, - stampReceipt: receipts.agentStamp, adopted: 0, adoptionBlocked: true, - detail: 'specialist dispatcher receipt mismatch; agent projection preserved', - } - : syncAgents({ - source, receipts: receipts.agents, stampReceipt: receipts.agentStamp, - adoptionBlocked: receiptState.agentsAdoptionBlocked, - gatewayCapabilities, - lazyCatalog: gatewayFacts.current && agentCatalog.length > 0, - ...(agentsDir ? { destDir: agentsDir } : {}), - }); - const skill = deploySkill({ - source, receipt: receipts.skill, adoptionBlocked, - ...(skillsDir ? { skillsDir } : {}), - }); - if (!adoptionBlocked) { - mutableOpencodeOwnership(cfg).managed.artifacts = { - plugin: plugin.receipt ?? receipts.plugin ?? null, - gateway: gatewayRequired - ? (gateway.receipt ?? receipts.gateway ?? null) - : (gateway.ok ? null : (gateway.receipt ?? receipts.gateway ?? null)), - agents: agents.receipts ?? receipts.agents ?? {}, - agentStamp: agents.stampReceipt ?? receipts.agentStamp ?? null, - skill: skill.receipt ?? receipts.skill ?? null, - }; - } - const markersChanged = JSON.stringify([ - opencodeOwnership(cfg).mcp ?? null, - opencodeOwnership(cfg).managed ?? null, - ]) !== before; - return { oc, plugin, gateway, agents, skill, source, markersChanged }; -} - -/** Retire path: strip the ak-managed opencode.json wiring (user priors - * restored; collisions and user-edited values left), then remove ak-deployed - * artifacts (marker-gated — user-owned files survive). undoOpencode nulls the - * ownership markers in cfg on success and keeps them on failure; the caller - * persists — and MUST honor undo.ok before claiming a disable (codex-review - * r3: a JSONC-refused config leaves active wiring behind). - * @param {any} cfg */ -/** @param {any} cfg - * @param {{configFile?:string,pluginsDir?:string,agentsDir?:string,skillsDir?:string}} [opts] */ -export function retireOpencode(cfg, { configFile, pluginsDir, agentsDir, skillsDir } = {}) { - const receipts = normalizeManaged(opencodeOwnership(cfg).managed).artifacts; - const undo = undoOpencode(cfg, { ...(configFile ? { configFile } : {}) }); - const artifacts = undo.ok - ? removeArtifacts({ - receipts, - ...(pluginsDir ? { pluginsDir } : {}), - ...(agentsDir ? { agentsDir } : {}), - ...(skillsDir ? { skillsDir } : {}), - }) - : { ok: false, changed: false, detail: 'retained because opencode.json teardown is incomplete' }; - return { undo, artifacts, ok: undo.ok }; -} - -/** - * ADR-0016 lifecycle adapter for OpenCode's managed native surfaces. - * Configuration lifecycle is deliberately separate from activity routing: - * this adapter drives setup/sync/status/teardown while the host-neutral runner - * separately honors the registry's explicit `canRouteActivities:true`. - * - * The factory keeps filesystem destinations injectable for hermetic conformance - * tests. `detect`, `plan`, and `verify` are read-only. `runLifecycle` owns the - * dry-run boundary, so `apply` and `undo` are never called for a dry-run. - */ -export function createOpencodeLifecycleAdapter(defaults = {}) { - const options = (request) => ({ ...defaults, ...(request.options ?? {}) }); - const detect = async (request = {}) => { - const cfg = request.cfg ?? {}; - const opts = options(request); - const source = catalogSource({ override: opencodeOwnership(cfg).catalogDir }); - const convergence = await opencodeConverged(cfg, { - ...(opts.configFile ? { configFile: opts.configFile } : {}), - ...(opts.brainShim ? { brainShim: opts.brainShim } : {}), - }); - const receiptState = opencodeArtifactReceiptState(opencodeOwnership(cfg).managed); - const { receipts, adoptionBlocked } = receiptState; - const managedMcp = managedGatewayMcp(cfg, { - ...(opts.configFile ? { configFile: opts.configFile } : {}), - }); - const dispatcher = specialistDispatcherState({ - destDir: opts.agentsDir ?? paths.opencodeAgentsDir(), - receipts: receipts.agents, - adoptionBlocked: receiptState.agentsAdoptionBlocked, - }); - const agentCatalog = dispatcher.available ? gatewayAgentCatalog(source) : []; - const gatewayRequired = Object.keys(managedMcp).length > 0 || agentCatalog.length > 0; - const plugin = opts.pkgRoot - ? pluginStatus({ - pkgRoot: opts.pkgRoot, receipt: receipts.plugin, adoptionBlocked, - ...(opts.pluginsDir ? { pluginsDir: opts.pluginsDir } : {}), - }) - : { present: false, current: false, foreign: false, adoptable: false }; - const gateway = opts.pkgRoot - ? gatewayPluginStatus({ - pkgRoot: opts.pkgRoot, managedMcp, agentCatalog, - receipt: receipts.gateway, adoptionBlocked, - ...(opts.pluginsDir ? { pluginsDir: opts.pluginsDir } : {}), - }) - : { present: false, current: false, foreign: false, adoptable: false }; - gateway.required = gatewayRequired; - const agents = agentsStatus({ - source, receipts: receipts.agents, stampReceipt: receipts.agentStamp, - adoptionBlocked: receiptState.agentsAdoptionBlocked || dispatcher.blocked, - gatewayCapabilities: { - ruflo: gateway.current && managedMcp['claude-flow'] != null, - aqe: gateway.current && managedMcp['agentic-qe'] != null, - }, - lazyCatalog: gateway.current && agentCatalog.length > 0, - ...(opts.agentsDir ? { destDir: opts.agentsDir } : {}), - }); - const skill = skillStatus({ - source, receipt: receipts.skill, adoptionBlocked, - ...(opts.skillsDir ? { skillsDir: opts.skillsDir } : {}), - }); - return { - enabled: !!cfg.integrations?.hosts?.opencode, - convergence, plugin, gateway, agents, skill, - }; - }; - return { - id: 'opencode', - detect, - async plan(request = {}) { - const facts = request.facts ?? await detect(request); - const changed = facts.enabled && (!facts.convergence.converged - || facts.plugin.adoptable || (!facts.plugin.current && !facts.plugin.foreign) - || (facts.gateway.required - && (facts.gateway.adoptable || (!facts.gateway.current && !facts.gateway.foreign))) - || (!facts.gateway.required && facts.gateway.present && !facts.gateway.foreign) - || (!facts.agents.adoptionBlocked && (facts.agents.adoptable || facts.agents.stale)) - || facts.skill.adoptable || (!facts.skill.current && !facts.skill.foreign)); - return { - changed, facts, - operations: changed ? ['config', 'plugin', 'gateway', 'agents', 'skill'] : [], - }; - }, - async apply(request = {}) { - const cfg = request.cfg; - const opts = options(request); - if (!cfg || !opts.pkgRoot) throw new TypeError('opencode lifecycle apply requires cfg and pkgRoot'); - const result = await opencodeStack(cfg, opts); - return { - changed: result.oc.changed || result.plugin.changed || result.gateway.changed || result.agents.changed - || result.skill.changed || result.markersChanged, - result, - }; - }, - async verify(request = {}) { - return detect(request); - }, - async undo(request = {}) { - if (!request.cfg) throw new TypeError('opencode lifecycle undo requires cfg'); - const result = retireOpencode(request.cfg, options(request)); - return { changed: result.undo.changed || result.artifacts.changed, result }; - }, - }; -} - -export const OPENCODE_LIFECYCLE_ADAPTER = createOpencodeLifecycleAdapter(); - -/** Reconcile the opencode AGENTS.md guidance blocks for the current enablement - * state — the `agents-opencode` target only, never the claude/project files. - * Enable (`enabled: true`) upserts the enablement-gated blocks as soon as the - * config home exists; disable (`enabled: false`) strips them (the always-on - * preamble stays by design; user content is never touched). Shared by setup, - * `ak host pick` enable/disable, and `ak host off`, so every command - * converges guidance the same way sync's blocks branch does (codex-review r3). - * @param {{ pkgRoot: string, cfg: any, cwd?: string, enabled: boolean }} opts */ -export async function reconcileOpencodeGuidance({ pkgRoot, cfg, cwd = process.cwd(), enabled }) { - const target = guidanceTargets({ cwd }).find((t) => t.name === 'agents-opencode'); - if (!target) return { ok: true, changed: false, detail: 'no opencode config home — guidance skipped' }; - const rows = registry(cfg.customBlocks); - const resolve = (r) => (r.custom - ? (r.template.startsWith('~/') ? path.join(paths.home, r.template.slice(2)) : r.template) - : path.join(pkgRoot, 'claude', r.template)); - const ctx = { - flags: { - dualMode: !!cfg.integrations?.hosts?.claude && !!cfg.integrations?.hosts?.codex, - opencodeEnabled: enabled, - }, - }; - const treg = [...blocksForTarget(rows, 'agents-opencode'), ...retiredForTarget(rows, 'agents-opencode')]; - const res = await syncBlocks(target.file, treg, resolve, { context: ctx }); - const changed = res.filter((r) => r.action !== 'unchanged' && r.action !== 'skipped') - .map((r) => `${r.slug} ${r.action}`); - return { ok: true, changed: changed.length > 0, detail: changed.length ? `guidance: ${changed.join(', ')}` : 'guidance in sync' }; -} - -// ── ruflo catalog source (agents + skills) ────────────────────────────────── - -/** Resolve where ruflo's agent/skill catalog comes from. Order: explicit - * override (kit.json integrations.ownership.opencode.catalogDir) → - * $RUFLO_REPO → the claude - * marketplace clone (full repo mirror, auto-updated by claude) → the - * published @claude-flow/cli package (subset: ADR-128 agents + core skills) - * → the nested copy under global ruflo/node_modules (the layout a plain - * `npm i -g ruflo` actually produces). Candidates are LAZY thunks: the - * npm-root lookups spawn `npm root -g` (cached per process), so evaluating - * them only when earlier candidates miss keeps status probes spawn-free on - * marketplace machines. Returns {kind, root, id, hasPlugins, hasPlatformSkill} - * or null. - * @param {{ override?: string }} [opts] - * @returns {CatalogSource|null} */ -export function catalogSource({ override } = {}) { - const candidates = []; - if (override) candidates.push(() => ({ kind: 'override', root: override })); - if (process.env.RUFLO_REPO) candidates.push(() => ({ kind: 'env', root: process.env.RUFLO_REPO })); - candidates.push(() => ({ kind: 'marketplace', root: paths.rufloMarketplaceRoot() })); - candidates.push(() => ({ kind: 'npm', root: paths.rufloCliPkgRoot() })); - candidates.push(() => ({ kind: 'npm-nested', root: path.join(paths.rufloNodeModules(), '@claude-flow', 'cli') })); - for (const thunk of candidates) { - const c = thunk(); - if (!c.root || !fs.existsSync(path.join(c.root, '.claude', 'agents'))) continue; - let version = null; - try { version = JSON.parse(fs.readFileSync(path.join(c.root, 'package.json'), 'utf8')).version; } catch { /* unversioned source */ } - return { - ...c, - id: `${c.kind}@${version ?? 'unversioned'}`, - hasPlugins: fs.existsSync(path.join(c.root, 'plugins')), - hasPlatformSkill: fs.existsSync(path.join(c.root, 'SKILL.md')), - }; - } - return null; -} - -/** skills.paths entries for a catalog source (existing dirs only). - * @param {CatalogSource|null} source */ -export function skillPathsFor(source) { - if (!source) return []; - const out = [path.join(source.root, '.claude', 'skills')]; - if (source.hasPlugins) out.push(path.join(source.root, 'plugins')); - return out.filter((p) => fs.existsSync(p)); -} - -// ── agent conversion (Claude Code agent .md → opencode subagent .md) ───────── - -/** Ownership markers on generated agent files — the current ak marker plus the - * legacy standalone-script marker, so the one-time script's output is adopted - * (removed/replaced) rather than orphaned. */ -const AGENT_MARKERS = ['generated-by: agentic-kit', 'generated-by: sync-ruflo-agents.mjs']; -const STAMP_FILE = '.ak-agents-stamp.json'; - -function* walkMd(dir) { - for (const e of fs.readdirSync(dir, { withFileTypes: true }) - .sort((left, right) => left.name.localeCompare(right.name))) { - const p = path.join(dir, e.name); - if (e.isDirectory()) yield* walkMd(p); - else if (e.isFile() && e.name.endsWith('.md')) yield p; - } -} - -/** Minimal YAML frontmatter reader: scalar fields + block scalars - * (description: | / >). A block scalar's content is every following line that - * is INDENTED, with blank lines allowed inside — terminating at the first - * blank (they'd otherwise truncate multi-paragraph descriptions and leak the - * remainder into mis-parsed fields). */ -function parseFrontmatter(text) { - const m = text.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n?([\s\S]*)$/); - if (!m) return null; - const [, fm, body] = m; - const fields = {}; - const lines = fm.split(/\r?\n/); - for (let i = 0; i < lines.length; i++) { - const km = lines[i].match(/^([A-Za-z_][\w-]*):\s*(.*)$/); - if (!km) continue; - const [, key, raw] = km; - if (/^[>|]-?$/.test(raw) && i + 1 < lines.length) { - const buf = []; - while (i + 1 < lines.length && (lines[i + 1].trim() === '' || /^\s+\S/.test(lines[i + 1]))) { - const t = lines[++i].trim(); - if (t) buf.push(t); - } - fields[key] = buf.join(' ').trim(); - } else { - fields[key] = raw.replace(/^["']|["']$/g, '').trim(); - } - } - return { fields, body }; -} - -const collapse = (s) => String(s ?? '').replace(/\s+/g, ' ').trim(); - -const lazyGatewayCall = (family, name) => - `\`${family}_call\` with \`name=${JSON.stringify(name)}\` and \`arguments_json\` set to one JSON object string`; - -const directOpenCodeReferences = (body) => String(body) - .replace(/mcp__(?:claude-flow|claude_flow|ruflo)__([A-Za-z0-9_./:*-]+)/g, 'claude-flow_$1') - .replace(/mcp__(?:agentic-qe|agentic_qe)__([A-Za-z0-9_./:*-]+)/g, 'agentic-qe_$1'); - -/** Rewrite tool-name references inside an OpenCode-only generated agent so - * the instructions use the lazy gateway that is actually advertised. The - * Claude/Ruflo source file is never changed. Families without a managed - * gateway retain their direct OpenCode tool spelling. - * @param {string} body @param {{ruflo?:boolean,aqe?:boolean}} capabilities */ -export function rewriteAgentGatewayReferences(body, capabilities = {}) { - let result = directOpenCodeReferences(body); - if (capabilities.ruflo) { - result = result - .replace(/\b(?:claude-flow|claude_flow)_\*/g, - () => 'the Ruflo operation selected with `ak_ruflo_search`, then invoked through `ak_ruflo_call`') - .replace(/\b(?:claude-flow|claude_flow)_([A-Za-z0-9_./:-]+)/g, - (_match, name) => lazyGatewayCall('ruflo', name)); - } - if (capabilities.aqe) { - result = result - .replace(/\b(?:agentic-qe|agentic_qe)_\*/g, - () => 'the Agentic QE operation selected with `ak_aqe_search`, then invoked through `ak_aqe_call`') - .replace(/\b(?:agentic-qe|agentic_qe)_([A-Za-z0-9_./:-]+)/g, - (_match, name) => lazyGatewayCall('aqe', name)); - } - return result; -} - -/** Convert every agent under /.claude/agents into opencode form: - * frontmatter → {description, mode: subagent} (Claude's `tools:` string list - * is dropped — OpenCode applies the subagent's permissions plus inherited - * parent/session deny rules); body MCP refs - * rewritten across all catalogue spellings. Lazy-gateway conversion emits - * ak_ruflo_call/ak_aqe_call guidance; direct fallback conversion emits OpenCode's - * direct tool spelling. - * basename collisions across category dirs get the parent dir prefixed. The - * description is emitted as a JSON double-quoted scalar (valid YAML 1.2 — - * unquoted values containing ': ' or '#' would corrupt the frontmatter). - * Pure (returns content, writes nothing). - * @param {string} srcRoot */ -export function convertAgents(srcRoot, { gatewayCapabilities = {} } = {}) { - const srcDir = path.join(srcRoot, '.claude', 'agents'); - const agents = []; - let scanned = 0; - for (const file of walkMd(srcDir)) { - scanned++; - const parsed = parseFrontmatter(fs.readFileSync(file, 'utf8')); - if (!parsed) continue; - // documentation masquerading as an agent (e.g. MIGRATION_SUMMARY.md) - if (collapse(parsed.fields.type).toLowerCase() === 'documentation') continue; - const description = collapse(parsed.fields.description); - if (!description) continue; - const rel = path.relative(srcDir, file); - const dir = path.dirname(rel) === '.' ? null : path.dirname(rel).split(path.sep)[0]; - agents.push({ - base: path.basename(file, '.md'), - dir, - description, - body: rewriteAgentGatewayReferences(parsed.body, gatewayCapabilities), - }); - } - const seen = new Set(); - let renamed = 0; - for (const a of agents) { - let name = a.base; - if (seen.has(name)) { name = a.dir ? `${a.dir}-${a.base}` : `${a.base}-x`; renamed++; } - let n = 2; - while (seen.has(name)) name = `${a.dir ?? 'agent'}-${a.base}-${n++}`; - seen.add(name); - a.name = name; - a.content = `---\ndescription: ${JSON.stringify(a.description)}\nmode: subagent\n---\n\n\n${a.body}`; - } - return { agents, scanned, skipped: scanned - agents.length, renamed }; -} - -const SPECIALIST_AGENT = { - name: 'ak-specialist', - description: 'Runs one Agentic Kit specialist profile selected lazily with ak_agent_search', - content: `--- -description: "Runs one Agentic Kit specialist profile selected lazily with ak_agent_search" -mode: subagent ---- - - -You are the Agentic Kit specialist dispatcher for stock OpenCode. - -The parent task must begin with \`PROFILE: \`. Call \`ak_agent_load\` with that exact -name before doing any work. Treat the returned receipt-owned profile as your specialist -instructions for the rest of this task. If the profile names an optional dependency that is not -available, report the missing dependency instead of inventing a result. -`, -}; - -function specialistDispatcherState({ - destDir = paths.opencodeAgentsDir(), receipts = {}, adoptionBlocked = false, -} = {}) { - if (adoptionBlocked) return { available: false, blocked: true }; - const file = 'ak-specialist.md'; - const target = path.join(destDir, file); - if (!fs.existsSync(target)) return { available: true, blocked: false }; - let current; - try { current = fs.readFileSync(target, 'utf8'); } catch { - return { available: false, blocked: true }; - } - if (receiptMatches(current, receipts?.[file])) return { available: true, blocked: false }; - if (hasReceiptValue(receipts?.[file])) return { available: false, blocked: true }; - const adoptable = current === SPECIALIST_AGENT.content && isGeneratedContent(current); - return { available: adoptable, blocked: false }; -} - -function desiredAgentSet(source, gatewayCapabilities, lazyCatalog) { - const converted = convertAgents(source.root, { gatewayCapabilities }); - return { ...converted, agents: lazyCatalog ? [SPECIALIST_AGENT] : converted.agents }; -} - -function gatewayAgentCatalog(source) { - if (!source) return []; - return convertAgents(source.root, { gatewayCapabilities: {} }).agents.map((agent) => ({ - name: agent.name, - description: agent.description, - body: agent.body, - })); -} - -const isGeneratedContent = (text) => AGENT_MARKERS.some((m) => text.includes(m)); - -/** Reconcile the converted agent set into the dest dir: rewrite generated - * files, remove stale generated ones (either marker), NEVER overwrite a file - * that carries no generated marker (a user-owned agent with a colliding name - * is preserved and reported). The stamp records the source id + the exact - * generated file list and is only rewritten when the set actually changed - * (no per-run timestamp churn — idempotent-write semantics). - * @param {{ source: CatalogSource|null, destDir?: string, dryRun?: boolean, receipts?:Record, stampReceipt?:string|null, adoptionBlocked?:boolean, gatewayCapabilities?:{ruflo?:boolean,aqe?:boolean}, lazyCatalog?:boolean }} opts */ -export function syncAgents({ - source, destDir = paths.opencodeAgentsDir(), dryRun = false, receipts = {}, stampReceipt = null, - adoptionBlocked = false, gatewayCapabilities = {}, lazyCatalog = false, -}) { - if (!source) return { ok: false, changed: false, detail: 'no ruflo catalog source (marketplace clone or @claude-flow/cli) found' }; - const receiptMap = receipts && typeof receipts === 'object' && !Array.isArray(receipts) - ? receipts - : {}; - const { agents, scanned, skipped, renamed } = desiredAgentSet( - source, gatewayCapabilities, lazyCatalog, - ); - if (!dryRun) fs.mkdirSync(destDir, { recursive: true }); - let removed = 0, userOwned = 0, adopted = 0; - const removedFiles = new Set(); - let written = 0; - const deployed = []; - for (const a of agents) { - const file = `${a.name}.md`; - const p = path.join(destDir, file); - const cur = fs.existsSync(p) ? fs.readFileSync(p, 'utf8') : null; - const priorReceipt = receiptMap[file]; - const hasPriorReceipt = adoptionBlocked || hasReceiptValue(priorReceipt); - const priorOwned = cur !== null && receiptMatches(cur, priorReceipt); - const adoptable = cur === a.content && !hasPriorReceipt && isGeneratedContent(cur); - if (cur !== null && !priorOwned && !adoptable) { userOwned++; continue; } - if (adoptable) adopted++; - deployed.push(file); - if (cur !== a.content) { - written++; - if (!dryRun) fs.writeFileSync(p, a.content); - } - } - // Deploy/adopt the dispatcher or complete direct set before retiring any - // receipt-owned predecessor. A write failure therefore preserves the last - // known-good eager catalogue instead of leaving no executable agent path. - if ((!lazyCatalog || deployed.includes('ak-specialist.md')) && fs.existsSync(destDir)) { - for (const f of fs.readdirSync(destDir).filter((f) => f.endsWith('.md'))) { - const p = path.join(destDir, f); - let owned = false; - try { owned = receiptMatches(fs.readFileSync(p, 'utf8'), receiptMap[f]); } catch { /* leave alone */ } - const wanted = agents.some((a) => `${a.name}.md` === f); - if (owned && !wanted) { - if (!dryRun) fs.rmSync(p); - removedFiles.add(f); - removed++; - } - } - } - const changed = written > 0 || removed > 0; - // The stamp records what was ACTUALLY deployed (a user-owned file occupying - // a slot is never in it) — otherwise status would diverge forever. - // Preserve a non-null mismatched receipt while its file still exists. If it - // were dropped, a later pass could mistake the resulting absence for a - // pre-receipts install and launder an edited file back into ak ownership. - const nextReceipts = Object.fromEntries(Object.entries(receiptMap).filter(([f]) => ( - !removedFiles.has(f) && fs.existsSync(path.join(destDir, f)) - ))); - const deployedHashes = Object.fromEntries(deployed.map((f) => { - const agent = agents.find((a) => `${a.name}.md` === f); - return [f, contentHash(agent.content)]; - })); - Object.assign(nextReceipts, deployedHashes); - const gateway = { ruflo: !!gatewayCapabilities.ruflo, aqe: !!gatewayCapabilities.aqe }; - const stamp = { - source: source.id, gateway, lazyCatalog: !!lazyCatalog, - count: deployed.length, files: deployed.sort(), hashes: deployedHashes, - }; - const stampText = JSON.stringify(stamp, null, 2) + '\n'; - const stampPath = path.join(destDir, STAMP_FILE); - const priorStampText = fs.existsSync(stampPath) ? fs.readFileSync(stampPath, 'utf8') : null; - const hasStampReceipt = adoptionBlocked || hasReceiptValue(stampReceipt); - const stampOwned = priorStampText !== null && receiptMatches(priorStampText, stampReceipt); - // The stamp has no marker of its own, so exact bytes are adoptable only when - // every file it lists was independently receipt-owned, newly written, or - // adopted through exact marker-bearing content. - const stampAdoptable = priorStampText === stampText && !hasStampReceipt && deployed.length > 0 - && deployed.every((f) => { - try { - return contentHash(fs.readFileSync(path.join(destDir, f), 'utf8')) === deployedHashes[f]; - } catch { return false; } - }); - const mayWriteStamp = priorStampText === null || stampOwned || stampAdoptable; - if (!dryRun && mayWriteStamp && priorStampText !== stampText) { - fs.writeFileSync(stampPath, stampText); - } - return { - ok: !lazyCatalog || deployed.includes('ak-specialist.md'), - changed, - receipts: nextReceipts, - stampReceipt: mayWriteStamp ? contentHash(stampText) : stampReceipt, - adopted, - detail: `${agents.length} ${lazyCatalog ? 'lazy dispatcher agent' : 'agents'} from ${source.id} (${written} written, ${removed} removed, ${skipped} skipped, ${renamed} collision-renamed${adopted ? `, ${adopted} adopted` : ''}${userOwned ? `, ${userOwned} user-owned preserved` : ''}; scanned ${scanned})`, - }; -} - -/** Agent-set drift, honestly: stale when the stamp is missing, the catalog - * source id diverged (upgrade/marketplace pull), or the on-disk generated - * file set differs from the stamp. Count reports marker-bearing agents for - * visibility, while receipt/hash divergence is reported as `modified` so - * callers classify user edits as preserved rather than repairable drift. - * @param {{ source?: CatalogSource|null, destDir?: string, receipts?:Record|null, stampReceipt?:string|null, adoptionBlocked?:boolean, gatewayCapabilities?:{ruflo?:boolean,aqe?:boolean}, lazyCatalog?:boolean }} [opts] */ -export function agentsStatus({ - source, destDir = paths.opencodeAgentsDir(), receipts = null, stampReceipt = null, - adoptionBlocked = false, gatewayCapabilities = {}, lazyCatalog = false, -} = {}) { - const stampPath = path.join(destDir, STAMP_FILE); - const stamp = readJson(stampPath, null); - const stampText = fs.existsSync(stampPath) ? fs.readFileSync(stampPath, 'utf8') : null; - const hasReceiptLedger = receipts !== null; - const receiptMap = receipts && typeof receipts === 'object' && !Array.isArray(receipts) - ? receipts - : {}; - const desired = source - ? new Map(desiredAgentSet(source, gatewayCapabilities, lazyCatalog) - .agents.map((a) => [`${a.name}.md`, a.content])) - : new Map(); - const adoptableFiles = []; - let generatedCount = 0; - if (fs.existsSync(destDir)) { - for (const f of fs.readdirSync(destDir).filter((f) => f.endsWith('.md'))) { - try { if (isGeneratedContent(fs.readFileSync(path.join(destDir, f), 'utf8'))) generatedCount++; } catch { /* skip */ } - } - } - const onDisk = fs.existsSync(destDir) - ? fs.readdirSync(destDir).filter((f) => { - if (!f.endsWith('.md')) return false; - try { - const text = fs.readFileSync(path.join(destDir, f), 'utf8'); - if (hasReceiptLedger) { - const hasReceipt = adoptionBlocked || hasReceiptValue(receiptMap[f]); - const receiptOwned = receiptMatches(text, receiptMap[f]); - const adoptable = !hasReceipt && isGeneratedContent(text) && desired.get(f) === text; - if (adoptable) adoptableFiles.push(f); - return receiptOwned || adoptable; - } - return isGeneratedContent(text); - } catch { return false; } - }).sort() - : []; - const stampFiles = Array.isArray(stamp?.files) ? [...stamp.files].sort() : null; - const filesDiverged = stampFiles != null && JSON.stringify(stampFiles) !== JSON.stringify(onDisk); - const contentDiverged = !!stamp?.hashes && onDisk.some((f) => { - try { return contentHash(fs.readFileSync(path.join(destDir, f), 'utf8')) !== stamp.hashes[f]; } catch { return true; } - }); - const expectedStamp = source && onDisk.length > 0 && onDisk.every((f) => desired.has(f)) - ? `${JSON.stringify({ - source: source.id, - gateway: { ruflo: !!gatewayCapabilities.ruflo, aqe: !!gatewayCapabilities.aqe }, - lazyCatalog: !!lazyCatalog, - count: onDisk.length, - files: onDisk, - // syncAgents hashes in converter declaration order, then sorts only - // the separate files list. Preserve that byte order for exact - // stamp-only adoption detection. - hashes: Object.fromEntries([...desired.entries()] - .filter(([f]) => onDisk.includes(f)) - .map(([f, text]) => [f, contentHash(text)])), - }, null, 2)}\n` - : null; - const stampAdoptable = hasReceiptLedger && !adoptionBlocked && !hasReceiptValue(stampReceipt) - && expectedStamp !== null && stampText === expectedStamp; - return { - count: generatedCount, - stampedId: stamp?.source ?? null, - currentId: source?.id ?? null, - adoptable: !adoptionBlocked && (adoptableFiles.length > 0 || stampAdoptable), - adoptionBlocked, - modified: contentDiverged || (hasReceiptLedger && fs.existsSync(destDir) - && fs.readdirSync(destDir).filter((f) => f.endsWith('.md')).some((f) => { - try { - return hasReceiptValue(receiptMap[f]) - && !receiptMatches(fs.readFileSync(path.join(destDir, f), 'utf8'), receiptMap[f]); - } catch { return true; } - })), - stale: !stamp || stamp.source !== (source?.id ?? null) || filesDiverged - || !!stamp.lazyCatalog !== !!lazyCatalog - || !deepEqual(stamp.gateway ?? { ruflo: false, aqe: false }, { - ruflo: !!gatewayCapabilities.ruflo, aqe: !!gatewayCapabilities.aqe, - }), - }; -} - -// ── plugin (lifecycle bridge) ──────────────────────────────────────────────── - -export const PLUGIN_NAME = 'ruflo-hooks.js'; -export const GATEWAY_PLUGIN_NAME = 'ruflo-gateway.js'; -const pluginTemplate = (pkgRoot) => path.join(pkgRoot, 'src', 'templates', 'opencode-ruflo-hooks.js'); -const gatewayPluginTemplate = (pkgRoot) => path.join(pkgRoot, 'src', 'templates', 'opencode-ruflo-gateway.js'); - -/** The marker any ak-deployed plugin copy carries (from the template header). */ -const PLUGIN_MARKER = 'src/templates/opencode-ruflo-hooks.js'; -const GATEWAY_PLUGIN_MARKER = 'src/templates/opencode-ruflo-gateway.js'; - -function deployManagedPlugin({ - template, marker, name, label, pluginsDir, dryRun, receipt, adoptionBlocked, - desiredText = null, -}) { - if (!fs.existsSync(template)) return { ok: false, changed: false, detail: `template missing: ${template}` }; - const want = desiredText ?? fs.readFileSync(template, 'utf8'); - const dest = path.join(pluginsDir, name); - const cur = fs.existsSync(dest) ? fs.readFileSync(dest, 'utf8') : null; - const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); - const adoptable = cur === want && !hasReceipt && cur.includes(marker); - if (cur === want && (receiptMatches(cur, receipt) || adoptable)) { - return { - ok: true, changed: false, receipt: contentHash(cur), adopted: adoptable, - detail: adoptable ? `${label} adopted into receipt ledger` : `${label} current`, - }; - } - if (cur !== null && (!hasReceipt || !receiptMatches(cur, receipt))) { - return { ok: true, changed: false, receipt, detail: `⚠ ${dest} differs from ak's last-written receipt (user-owned/edited) — left untouched` }; - } - if (!want.includes(marker)) return { ok: false, changed: false, receipt, detail: `template marker missing: ${marker}` }; - if (!dryRun) { - fs.mkdirSync(pluginsDir, { recursive: true }); - fs.writeFileSync(dest, want); - } - return { - ok: true, - changed: true, - receipt: contentHash(want), - detail: cur == null ? `${label} deployed (${name})` : `${label} updated (${name})`, - }; -} - -function managedPluginStatus({ - template, marker, name, pluginsDir, receipt, adoptionBlocked, desiredText = null, -}) { - const dest = path.join(pluginsDir, name); - const present = fs.existsSync(dest); - const currentText = present ? fs.readFileSync(dest, 'utf8') : null; - const desired = fs.existsSync(template) ? (desiredText ?? fs.readFileSync(template, 'utf8')) : null; - const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); - const receiptOwned = present && receiptMatches(currentText, receipt); - const adoptable = present && !hasReceipt && desired !== null - && currentText === desired && currentText.includes(marker); - const foreign = present && !receiptOwned && !adoptable; - return { - present, - current: present && !foreign && desired !== null && currentText === desired, - foreign, - adoptable, - adoptionBlocked, - }; -} - -/** Deploy the lifecycle bridge plugin from the kit's template, content-diffed - * (rewrites only when the template changed — hash-stamped by content itself). - * A destination file that exists WITHOUT the ak marker is user-owned: - * preserved and reported, never overwritten. - * @param {{ pkgRoot: string, pluginsDir?: string, dryRun?: boolean, receipt?:string|null, adoptionBlocked?:boolean }} opts */ -export function deployPlugin({ - pkgRoot, pluginsDir = paths.opencodePluginsDir(), dryRun = false, receipt = null, - adoptionBlocked = false, -}) { - return deployManagedPlugin({ - template: pluginTemplate(pkgRoot), marker: PLUGIN_MARKER, name: PLUGIN_NAME, - label: 'lifecycle plugin', pluginsDir, dryRun, receipt, adoptionBlocked, - }); -} - -const GATEWAY_MCP_PLACEHOLDER = '/* AK_MANAGED_MCP_ENTRIES */ {}'; -const GATEWAY_AGENT_PLACEHOLDER = '/* AK_MANAGED_AGENT_CATALOG */ []'; -const GATEWAY_SPECIALIST_PLACEHOLDER = '/* AK_SPECIALIST_AGENT_PROMPT */ ""'; - -function gatewayDesiredText(pkgRoot, managedMcp, agentCatalog = []) { - const template = gatewayPluginTemplate(pkgRoot); - if (!fs.existsSync(template)) return null; - const source = fs.readFileSync(template, 'utf8'); - for (const [placeholder, label] of [ - [GATEWAY_MCP_PLACEHOLDER, 'managed-MCP'], - [GATEWAY_AGENT_PLACEHOLDER, 'managed-agent'], - [GATEWAY_SPECIALIST_PLACEHOLDER, 'specialist-agent'], - ]) { - const first = source.indexOf(placeholder); - if (first < 0 || source.indexOf(placeholder, first + 1) >= 0) { - throw new Error(`lazy gateway template must contain exactly one ${label} placeholder`); - } - } - const stable = Object.fromEntries(Object.entries(managedMcp ?? {}) - .sort(([a], [b]) => a.localeCompare(b))); - const specialistPrompt = parseFrontmatter(SPECIALIST_AGENT.content)?.body.trim() ?? ''; - return source - .replace(GATEWAY_MCP_PLACEHOLDER, JSON.stringify(stable)) - .replace(GATEWAY_AGENT_PLACEHOLDER, JSON.stringify(agentCatalog)) - .replace(GATEWAY_SPECIALIST_PLACEHOLDER, JSON.stringify(specialistPrompt)); -} - -/** Deploy the lazy Ruflo/Agentic-QE catalogue gateway for stock OpenCode. */ -export function deployGatewayPlugin({ - pkgRoot, managedMcp = {}, agentCatalog = [], pluginsDir = paths.opencodePluginsDir(), dryRun = false, - receipt = null, adoptionBlocked = false, -}) { - return deployManagedPlugin({ - template: gatewayPluginTemplate(pkgRoot), marker: GATEWAY_PLUGIN_MARKER, - name: GATEWAY_PLUGIN_NAME, label: 'lazy rUv gateway', pluginsDir, dryRun, - receipt, adoptionBlocked, desiredText: gatewayDesiredText(pkgRoot, managedMcp, agentCatalog), - }); -} - -/** Remove only exact receipt-owned gateway bytes when no rUv family remains - * safe to capture. User-edited/unreceipted files are preserved. */ -export function retireGatewayPlugin({ - pluginsDir = paths.opencodePluginsDir(), receipt = null, dryRun = false, -} = {}) { - const dest = path.join(pluginsDir, GATEWAY_PLUGIN_NAME); - if (!fs.existsSync(dest)) { - return { ok: true, changed: false, receipt: null, detail: 'lazy gateway not deployed' }; - } - const current = fs.readFileSync(dest, 'utf8'); - if (!receipt || !receiptMatches(current, receipt)) { - return { - ok: false, changed: false, receipt, - detail: `⚠ ${dest} is not provably ak-owned; left untouched`, - }; - } - if (!dryRun) fs.rmSync(dest, { force: true }); - return { ok: true, changed: true, receipt: null, detail: 'lazy rUv gateway retired' }; -} - -/** Plugin presence/currency against the kit template. `foreign` flags a - * user-owned file occupying the destination (status must not nag to - * overwrite it — deploy will leave it alone). */ -export function pluginStatus({ - pkgRoot, pluginsDir = paths.opencodePluginsDir(), receipt = null, adoptionBlocked = false, -}) { - return managedPluginStatus({ - template: pluginTemplate(pkgRoot), marker: PLUGIN_MARKER, name: PLUGIN_NAME, - pluginsDir, receipt, adoptionBlocked, - }); -} - -/** Lazy gateway presence/currency against its embedded, receipt-bound MCP commands. */ -export function gatewayPluginStatus({ - pkgRoot, managedMcp = {}, agentCatalog = [], pluginsDir = paths.opencodePluginsDir(), receipt = null, - adoptionBlocked = false, -}) { - return managedPluginStatus({ - template: gatewayPluginTemplate(pkgRoot), marker: GATEWAY_PLUGIN_MARKER, - name: GATEWAY_PLUGIN_NAME, pluginsDir, receipt, adoptionBlocked, - desiredText: gatewayDesiredText(pkgRoot, managedMcp, agentCatalog), - }); -} - -// ── platform skill ─────────────────────────────────────────────────────────── - -const SKILL_DEPLOYED_MARKER = ''; - -/** Deploy ruflo's platform SKILL.md (from the catalog source) into opencode's - * global skills dir, stamped with the source id for drift detection. A - * destination SKILL.md without the ak marker is user-owned: preserved. - * @param {{ source: CatalogSource|null, skillsDir?: string, dryRun?: boolean, receipt?:string|null, adoptionBlocked?:boolean }} opts */ -export function deploySkill({ - source, skillsDir = paths.opencodeSkillsDir(), dryRun = false, receipt = null, - adoptionBlocked = false, -}) { - if (!source?.hasPlatformSkill) return { ok: true, changed: false, detail: `no platform SKILL.md in catalog source${source ? ` (${source.id})` : ''}` }; - const src = path.join(source.root, 'SKILL.md'); - const dest = path.join(skillsDir, 'ruflo', 'SKILL.md'); - const want = `${fs.readFileSync(src, 'utf8').replace(/\s*$/, '')}\n\n${SKILL_DEPLOYED_MARKER} from ${source.id} — re-synced by \`ak sync\`\n`; - const cur = fs.existsSync(dest) ? fs.readFileSync(dest, 'utf8') : null; - const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); - const adoptable = cur === want && !hasReceipt && cur.includes(SKILL_DEPLOYED_MARKER); - if (cur === want && (receiptMatches(cur, receipt) || adoptable)) { - return { - ok: true, changed: false, receipt: contentHash(cur), adopted: adoptable, - detail: adoptable ? 'platform skill adopted into receipt ledger' : 'platform skill current', - }; - } - if (cur !== null && (!hasReceipt || !receiptMatches(cur, receipt))) { - return { ok: true, changed: false, receipt, detail: `⚠ ${dest} differs from ak's last-written receipt (user-owned/edited) — left untouched` }; - } - if (!dryRun) { - fs.mkdirSync(path.dirname(dest), { recursive: true }); - fs.writeFileSync(dest, want); - } - return { ok: true, changed: true, receipt: contentHash(want), detail: `platform skill deployed (skills/ruflo/SKILL.md, ${source.id})` }; -} - -/** Platform skill presence/currency against the catalog source id. `foreign` - * flags a user-owned SKILL.md at the destination. - * @param {{ source?: CatalogSource|null, skillsDir?: string, receipt?:string|null, adoptionBlocked?:boolean }} [opts] */ -export function skillStatus({ - source, skillsDir = paths.opencodeSkillsDir(), receipt = null, adoptionBlocked = false, -} = {}) { - const dest = path.join(skillsDir, 'ruflo', 'SKILL.md'); - const present = fs.existsSync(dest); - const text = present ? fs.readFileSync(dest, 'utf8') : null; - const sourceFile = source?.hasPlatformSkill ? path.join(source.root, 'SKILL.md') : null; - const desired = sourceFile && fs.existsSync(sourceFile) - ? `${fs.readFileSync(sourceFile, 'utf8').replace(/\s*$/, '')}\n\n${SKILL_DEPLOYED_MARKER} from ${source.id} — re-synced by \`ak sync\`\n` - : null; - const hasReceipt = adoptionBlocked || hasReceiptValue(receipt); - const receiptOwned = present && receiptMatches(text, receipt); - const adoptable = present && !hasReceipt && desired !== null - && text === desired && text.includes(SKILL_DEPLOYED_MARKER); - const foreign = present && !receiptOwned && !adoptable; - return { - present, - foreign, - adoptable, - adoptionBlocked, - current: present && !foreign && desired != null && text === desired, - }; -} - -/** Remove ak-deployed artifacts (marker-gated — user files are never touched): - * the lifecycle plugin, generated agents (+ stamp), the platform skill's - * SKILL.md. Directories are pruned only when EMPTY after the managed files - * are gone — user resources placed beside them survive. - * @param {{ pluginsDir?: string, agentsDir?: string, skillsDir?: string, receipts?:any }} [opts] */ -export function removeArtifacts({ - pluginsDir = paths.opencodePluginsDir(), agentsDir = paths.opencodeAgentsDir(), - skillsDir = paths.opencodeSkillsDir(), receipts = {}, -} = {}) { - const removed = []; - const rmdirIfEmpty = (dir) => { - try { if (fs.readdirSync(dir).length === 0) fs.rmdirSync(dir); } catch { /* absent or not empty */ } - }; - const plugin = path.join(pluginsDir, PLUGIN_NAME); - if (fs.existsSync(plugin) && receipts.plugin - && contentHash(fs.readFileSync(plugin, 'utf8')) === receipts.plugin) { - fs.rmSync(plugin, { force: true }); - removed.push('plugin ruflo-hooks.js'); - } - const gateway = path.join(pluginsDir, GATEWAY_PLUGIN_NAME); - if (fs.existsSync(gateway) && receipts.gateway - && contentHash(fs.readFileSync(gateway, 'utf8')) === receipts.gateway) { - fs.rmSync(gateway, { force: true }); - removed.push('plugin ruflo-gateway.js'); - } - if (fs.existsSync(agentsDir)) { - let n = 0; - for (const f of fs.readdirSync(agentsDir)) { - const p = path.join(agentsDir, f); - if (f === STAMP_FILE && receipts.agentStamp - && contentHash(fs.readFileSync(p, 'utf8')) === receipts.agentStamp) { - fs.rmSync(p, { force: true }); continue; - } - if (f.endsWith('.md') && receipts.agents?.[f] - && contentHash(fs.readFileSync(p, 'utf8')) === receipts.agents[f]) { - fs.rmSync(p, { force: true }); - n++; - } - } - if (n) removed.push(`${n} generated agents`); - } - const skillDir = path.join(skillsDir, 'ruflo'); - const skill = path.join(skillDir, 'SKILL.md'); - if (fs.existsSync(skill) && receipts.skill - && contentHash(fs.readFileSync(skill, 'utf8')) === receipts.skill) { - fs.rmSync(skill, { force: true }); - rmdirIfEmpty(skillDir); - removed.push('platform skill'); - } - return { ok: true, changed: removed.length > 0, detail: removed.length ? `removed: ${removed.join(', ')}` : 'no ak-deployed artifacts found' }; -} +export { + RUFLO_MCP_ENV, AQE_MCP_ENV, brainShimPath, nestedMcpServerPath, mcpCommandFor, mcpEntriesFor, + opencodeMcpStatus, opencodeConverged, opencodeArtifactReceiptState, managedGatewayMcp, + applyOpencode, undoOpencode, +} from './opencode-core.mjs'; +export { + catalogSource, skillPathsFor, rewriteAgentGatewayReferences, convertAgents, syncAgents, agentsStatus, +} from './opencode-agents.mjs'; +export { + PLUGIN_NAME, GATEWAY_PLUGIN_NAME, deployPlugin, deployGatewayPlugin, retireGatewayPlugin, + pluginStatus, gatewayPluginStatus, deploySkill, skillStatus, removeArtifacts, +} from './opencode-artifacts.mjs'; +export { + opencodeStack, retireOpencode, createOpencodeLifecycleAdapter, OPENCODE_LIFECYCLE_ADAPTER, + reconcileOpencodeGuidance, +} from './opencode-lifecycle.mjs'; diff --git a/src/lib/providers.mjs b/src/lib/providers.mjs index d119b59..baee17a 100644 --- a/src/lib/providers.mjs +++ b/src/lib/providers.mjs @@ -34,7 +34,7 @@ import * as paths from './paths.mjs'; import { bold, dim, cyan, reportOutcome, } from './output.mjs'; -import { configuredPolicyToAgentOverrides, seedActivityRoutes, resolveRoutes, routingSummary, divergedRoutes, migrateRetiredRoutes, ACTIVITIES, AGENT_ACTIVITY_MAP, PRIMARY_HOSTS } from './routing.mjs'; +import { seedActivityRoutes, resolveRoutes, routingSummary, divergedRoutes, migrateRetiredRoutes, ACTIVITIES, PRIMARY_HOSTS } from './routing.mjs'; import { HOST_ADAPTERS } from './hosts.mjs'; import { HOST_REGISTRY, PROVIDER_REGISTRY, normalizeIntegrationFacts, defaultHostMap, @@ -42,7 +42,13 @@ import { import { CURRENT_INTEGRATIONS_VERSION, validateEndpoint } from './adapters/config.mjs'; import { opencodeMcpStatus } from './opencode.mjs'; import { codexMcpStatus, rufloCodexMcpStatus } from './mcp.mjs'; -import { admittedAqeProviders, projectedAqeExternalProviders } from './adapters/aqe-provider.mjs'; +import { projectedAqeExternalProviders } from './adapters/aqe-provider.mjs'; +import { applyAqeRouter, aqeRouterDrift, undoAqeRouter } from './aqe-router.mjs'; + +// The AQE-router convergence pipeline itself lives in aqe-router.mjs +// (ADR-0037); re-exported here so every existing `./providers.mjs` import +// path (commands, status sections, tests) keeps working unchanged. +export { applyAqeRouter, aqeRouterDrift, undoAqeRouter }; import { DEFAULT_PRIMARY_HOST, ROUTING_SCHEMA_VERSION, @@ -414,14 +420,11 @@ export const AQE_FALLBACK_CODEX_SUGGESTION = 'claude-code:claude-opus-5; openai: export const suggestedFallbackFor = (enabledHosts) => (enabledHosts.includes('codex') ? AQE_FALLBACK_CODEX_SUGGESTION : null); // ── agentic-qe router config (.agentic-qe/llm-config.json) ────────────────── -// Grounded in aqe's router config-store + types (ADR-123): -// - mergeRouterConfig deep-merges `providers` but SHALLOW-replaces -// `fallbackChain` → ak must write a COMPLETE chain (these scalar defaults). -// - the router iterates `entry.models` → each entry needs populated models. -// - aqe refuses to persist apiKey → ak writes only `enabled` per provider; -// keys stay in the env. -const AQE_CHAIN_DEFAULTS = { maxRetries: 3, retryDelayMs: 100, backoffMultiplier: 2, maxDelayMs: 5000 }; -const AQE_MANAGED_TAG = 'agentic-kit'; +// Grounded in aqe's router config-store + types (ADR-123). The convergence +// pipeline itself (fallback chain, default provider, external provider +// declarations/activations, agentOverrides projection) lives in +// aqe-router.mjs (ADR-0037); this module keeps the version gates and the +// hash/ownership primitives aqe-router.mjs's surfaces are built from. // agentic-qe ≥ 3.13.1 shipped on-disk per-agent routing (`agentOverrides`, issue // #568). Below that, aqe ignores the key, so ak gates writing it on the version. @@ -438,170 +441,28 @@ export function aqeSupportsExternalProviders(version = installedVersion('agentic return !!version && cmpVersions(version, EXTERNAL_PROVIDERS_MIN_AQE) >= 0; } -const AQE_OWNERSHIP_KEY = '_agenticKit'; +// Ownership/hash primitives shared with aqe-router.mjs: this key/these +// functions decide whether a value on disk is still EXACTLY what ak wrote +// (never trusted input — a null/array/primitive receipt proves nothing). +// Exported for aqe-router.mjs's own reconcilers and ownership receipts; kept +// here (not there) because aqeExternalProviderState below needs them too and +// this is the direction that avoids a cycle between the two modules. +export const AQE_OWNERSHIP_KEY = '_agenticKit'; -function stableValue(value) { +export function stableValue(value) { if (Array.isArray(value)) return value.map(stableValue); if (!value || typeof value !== 'object') return value; return Object.fromEntries(Object.keys(value).sort().map((key) => [key, stableValue(value[key])])); } -function declarationHash(value) { +export function declarationHash(value) { return createHash('sha256').update(JSON.stringify(stableValue(value))).digest('hex'); } -function plainRecord(value) { +export function plainRecord(value) { return value && typeof value === 'object' && !Array.isArray(value) ? value : null; } -function exactlyOwnedDefault(config, receiptKey) { - const provider = config?.defaultProvider; - const receipt = plainRecord(config?.[AQE_OWNERSHIP_KEY]?.[receiptKey]); - return typeof provider === 'string' && receipt?.provider === provider - && receipt.writtenHash === declarationHash(provider) - ? provider - : null; -} - -function setDefaultOwnership(config, receiptKey, provider) { - const ownership = { ...(plainRecord(config[AQE_OWNERSHIP_KEY]) ?? {}) }; - ownership[receiptKey] = { provider, writtenHash: declarationHash(provider) }; - config[AQE_OWNERSHIP_KEY] = ownership; -} - -function clearDefaultOwnership(config, receiptKey) { - const ownership = { ...(plainRecord(config[AQE_OWNERSHIP_KEY]) ?? {}) }; - delete ownership[receiptKey]; - if (Object.keys(ownership).length) config[AQE_OWNERSHIP_KEY] = ownership; - else delete config[AQE_OWNERSHIP_KEY]; -} - -const exactlyOwnedExternalDefault = (config) => exactlyOwnedDefault(config, 'externalDefaultProvider'); -const exactlyOwnedFallbackDefault = (config) => exactlyOwnedDefault(config, 'fallbackDefaultProvider'); -const setExternalDefaultOwnership = (config, provider) => - setDefaultOwnership(config, 'externalDefaultProvider', provider); -const setFallbackDefaultOwnership = (config, provider) => - setDefaultOwnership(config, 'fallbackDefaultProvider', provider); -const clearExternalDefaultOwnership = (config) => - clearDefaultOwnership(config, 'externalDefaultProvider'); -const clearFallbackDefaultOwnership = (config) => - clearDefaultOwnership(config, 'fallbackDefaultProvider'); - -function admittedProviderRecord(id) { - const records = admittedAqeProviders(); - return (Array.isArray(records) ? records : Object.values(records ?? {})) - .find((entry) => (entry.id ?? entry.providerId ?? entry.type) === id) ?? null; -} - -/** Compare the live admitted declarations with the exact values ak previously - * wrote. Foreign entries and user-edited owned entries are never overwritten or - * removed. Returned `active` ids are safe to reference from defaults/chains. */ -function reconcileExternalProviders(existing, desired = aqeExternalProviders()) { - const current = { ...(existing.externalProviders ?? {}) }; - const currentProviders = { ...(existing.providers ?? {}) }; - // Ownership metadata is advisory proof, never trusted input. A null/array/ - // primitive receipt proves nothing and must be dropped rather than crashing - // sync or authorizing deletion of user values. - const rawReceipts = plainRecord(existing[AQE_OWNERSHIP_KEY]?.externalProviders) ?? {}; - const priorReceipts = Object.fromEntries(Object.entries(rawReceipts) - .filter(([, receipt]) => plainRecord(receipt))); - const receipts = { ...priorReceipts }; - const active = new Set(); - const conflicts = []; - const unavailable = new Set(); - const retired = []; - const pruned = []; - const added = []; - const activationsAdded = []; - const activationsPruned = []; - - for (const [id, declaration] of Object.entries(desired)) { - const prior = priorReceipts[id]; - const currentDeclaration = current[id]; - const currentHash = currentDeclaration === undefined ? null : declarationHash(currentDeclaration); - if (currentDeclaration !== undefined && (!prior || currentHash !== prior.writtenHash)) { - conflicts.push(id); - unavailable.add(id); - // A changed declaration becomes user-owned immediately. Its activation - // has independent ownership, though: retain only an exact activation - // receipt so a later revoke can remove the minimal record ak created - // without ever deleting the edited declaration. - const currentActivation = currentProviders[id]; - if (prior?.providerWrittenHash && currentActivation !== undefined - && declarationHash(currentActivation) === prior.providerWrittenHash) { - receipts[id] = { providerWrittenHash: prior.providerWrittenHash }; - } else { - delete receipts[id]; - } - continue; - } - current[id] = declaration; - const record = admittedProviderRecord(id); - const nextReceipt = { - hostId: record?.hostId ?? record?.host ?? record?.manifestId ?? null, - contentHash: record?.contentHash ?? record?.integrity ?? null, - writtenHash: declarationHash(declaration), - }; - if (currentDeclaration === undefined) added.push(id); - - // AQE 3.13.12's MCP router asks whether any providers are enabled BEFORE - // it loads externalProviders (the load is what registers them). A minimal - // providers[id].enabled record breaks that bootstrap cycle. Own only a - // record we created from absence; a user-owned record is preserved and is - // usable only when the user already enabled it explicitly. - const currentActivation = currentProviders[id]; - const priorActivationHash = prior?.providerWrittenHash; - const activationHash = currentActivation === undefined ? null : declarationHash(currentActivation); - if (currentActivation === undefined) { - currentProviders[id] = { enabled: true }; - nextReceipt.providerWrittenHash = declarationHash(currentProviders[id]); - activationsAdded.push(id); - active.add(id); - } else if (currentActivation?.enabled === true) { - if (priorActivationHash && activationHash === priorActivationHash) { - nextReceipt.providerWrittenHash = priorActivationHash; - } - active.add(id); - } else { - conflicts.push(`${id} (providers.${id}.enabled is not true)`); - unavailable.add(id); - } - receipts[id] = nextReceipt; - } - - for (const [id, receipt] of Object.entries(priorReceipts)) { - if (id in desired) continue; - retired.push(id); - const currentDeclaration = current[id]; - if (currentDeclaration !== undefined && declarationHash(currentDeclaration) === receipt.writtenHash) { - delete current[id]; - pruned.push(id); - } - const currentActivation = currentProviders[id]; - if (receipt.providerWrittenHash && currentActivation !== undefined - && declarationHash(currentActivation) === receipt.providerWrittenHash) { - delete currentProviders[id]; - activationsPruned.push(id); - } - // If it changed, relinquish ownership and preserve it. - delete receipts[id]; - } - - return { - externalProviders: current, - providers: currentProviders, - receipts, - active, - conflicts, - unavailable: [...unavailable], - retired, - pruned, - added, - activationsAdded, - activationsPruned, - }; -} - /** Honest, non-mutating projection state for status/verify. */ export function aqeExternalProviderState(disk = {}, { projectRoot = path.resolve(process.cwd()) } = {}) { const desired = aqeExternalProviders({ projectRoot }); @@ -683,482 +544,6 @@ export function providerExternalState(cfg, cwd = process.cwd()) { }; } -/** Map kit.json `aqeFallback` entries → a complete aqe FallbackChain. Priority - * descends by list order (first = highest). Entries carry provider + models. */ -function buildChain(entries) { - return { - id: AQE_MANAGED_TAG, - entries: entries.map((e, i) => ({ - provider: e.provider, - models: e.models ?? [], - enabled: true, - priority: 100 - i * 10, - maxAttempts: 2, - timeoutMs: 30000, - })), - ...AQE_CHAIN_DEFAULTS, - }; -} - -// ── applyAqeRouter: ordered surface reconcilers ───────────────────────────── -// Five surfaces used to be braided together in one function, sharing mutable -// accumulators with implicit cross-surface feedback: `externalActive` -// (computed while reconciling external providers) constrained what the -// fallback-chain/default-provider/agentOverrides surfaces below it could -// safely reference, and `projected`/`staleOverrides` had to be recomputed -// after that same fact became known. Each surface below is a -// `(next, ctx) => {detail, error, changed, ctx?}` step, folded left-to-right -// over one shared `next` draft; a surface returns an optional `ctx` PATCH -// (applied before the next surface runs) instead of closing over an outer -// `let` — the one real cross-surface dependency (externalActive -> the -// refined `projected`/`staleOverrides`) is the only patch actually used, so -// it stays a single, explicit, ordered hand-off rather than several loose -// mutable accumulators. - -/** The externalProviders surface's own detail line — split out only to keep - * that surface's branch count (five independent `? : ''` clauses) legible - * and under the reconciler's own complexity budget. */ -function formatExternalProvidersDetail(externalActive, reconciled) { - return `externalProviders: ${externalActive.size} managed` - + (reconciled.added.length ? ` (${reconciled.added.length} added)` : '') - + (reconciled.pruned.length ? ` (${reconciled.pruned.length} stale owned pruned)` : '') - + (reconciled.activationsAdded.length ? ` (${reconciled.activationsAdded.length} MCP activation added)` : '') - + (reconciled.activationsPruned.length ? ` (${reconciled.activationsPruned.length} stale activation pruned)` : '') - + (reconciled.conflicts.length ? ` (⚠ conflicts preserved: ${reconciled.conflicts.join(', ')})` : ''); -} - -/** Surface 1/4: reconcile admitted external-provider declarations/activations - * against the live file, prune anything that became unavailable from the - * fallback chain/defaultProvider, and refine `projected`/`staleOverrides` for - * the surfaces after it (their safe-to-reference set depends on which - * external ids ended up active here). */ -function reconcileExternalProvidersSurface(next, ctx) { - const { - existing, desiredExternal, hasExternal, hasOwnedExternal, externalSupported, - hasManagedFallback, ownedFallbackDefault, ownedExternalDefault, - priorOverrides, managedOverrideKeys, projected: priorProjected, - } = ctx; - let externalActive = new Set(); - let error = null; - let changed = false; - const detail = []; - - if (hasExternal || hasOwnedExternal) { - // A downgrade must remove only unchanged entries we previously wrote, - // plus their dangling references. Keeping declarations that this AQE - // version cannot understand would strand every router startup on drift. - const reconciled = reconcileExternalProviders(existing, externalSupported ? desiredExternal : {}); - externalActive = reconciled.active; - if (Object.keys(reconciled.externalProviders).length) next.externalProviders = reconciled.externalProviders; - else delete next.externalProviders; - if (Object.keys(reconciled.providers).length) next.providers = reconciled.providers; - else delete next.providers; - const ownership = { ...(plainRecord(next[AQE_OWNERSHIP_KEY]) ?? {}) }; - if (!ownedExternalDefault) delete ownership.externalDefaultProvider; - if (Object.keys(reconciled.receipts).length) ownership.externalProviders = reconciled.receipts; - else delete ownership.externalProviders; - if (Object.keys(ownership).length) next[AQE_OWNERSHIP_KEY] = ownership; - else delete next[AQE_OWNERSHIP_KEY]; - if (reconciled.conflicts.length) { - error = `refused conflicting foreign/user-edited external provider ids: ${reconciled.conflicts.join(', ')}`; - } - detail.push(formatExternalProvidersDetail(externalActive, reconciled)); - if (hasExternal && !externalSupported) { - error = `external providers need agentic-qe >=${EXTERNAL_PROVIDERS_MIN_AQE}`; - detail.push(`externalProviders: disabled (${error})`); - } - const unavailableExternal = new Set([...reconciled.unavailable, ...reconciled.retired]); - if (hasManagedFallback && next.fallbackChain?.entries) { - next.fallbackChain = { - ...next.fallbackChain, - entries: next.fallbackChain.entries.filter((entry) => !unavailableExternal.has(entry.provider)), - }; - if (next.fallbackChain.entries.length === 0) delete next.fallbackChain; - } - if (unavailableExternal.has(next.defaultProvider) - && (ownedFallbackDefault === next.defaultProvider || ownedExternalDefault === next.defaultProvider)) { - delete next.defaultProvider; - clearExternalDefaultOwnership(next); - clearFallbackDefaultOwnership(next); - } - changed = reconciled.added.length > 0 || reconciled.pruned.length > 0 - || reconciled.activationsAdded.length > 0 || reconciled.activationsPruned.length > 0 - || Object.keys(desiredExternal).some((id) => existing.externalProviders?.[id] - && declarationHash(existing.externalProviders[id]) !== declarationHash(desiredExternal[id])); - } - - // Admission/version/conflict filtering can make a previously projected - // external route inactive. Recompute from the safe projection so ak-owned - // overrides never retain an unusable id — this runs regardless of whether - // the branch above executed (externalActive then defaults to empty). - const projected = Object.fromEntries(Object.entries(priorProjected).filter(([, entry]) => - !(entry.provider in desiredExternal) || externalActive.has(entry.provider))); - const staleOverrides = Object.keys(priorOverrides) - .filter((agent) => managedOverrideKeys.has(agent) && !(agent in projected)); - - return { - detail, error, changed, ctx: { externalActive, projected, staleOverrides }, - }; -} - -/** Surface 2/4: retire a previously-written managed fallback chain (and its - * derived default) once the canonical `aqeFallback` intent goes empty. */ -function reconcileFallbackRetirementSurface(next, ctx) { - const { - hasChain, hasManagedFallback, ownedFallbackDefault, ownedExternalDefault, - } = ctx; - if (hasChain || !hasManagedFallback) return null; - // An empty canonical fallback intent retires the tagged chain ak previously - // wrote. Its derived default belongs to the same projection and must not - // survive independently; provider declarations/activations remain available - // for explicit selection, routes, or a future chain. - delete next.fallbackChain; - if (ownedFallbackDefault) { - delete next.defaultProvider; - if (ownedExternalDefault) clearExternalDefaultOwnership(next); - } - clearFallbackDefaultOwnership(next); - return { detail: 'chain: managed fallback retired', changed: true }; -} - -/** Surface 3/4: decide `defaultProvider` and which of the two ownership - * receipts (external vs. fallback-chain-derived) it carries, across the - * three ways it can change: explicit deselection, chain-derived assignment - * (which also builds/validates the active chain itself), and an explicit - * project-local external selection. */ -function reconcileDefaultProviderSurface(next, ctx) { - const { - cfg, existing, chain, selectedProvider, hasChain, desiredExternal, externalActive, ownedExternalDefault, - } = ctx; - const detail = []; - let error = null; - let changed = false; - - // `aqeProvider: null` is an explicit deselection. Retire only an exact - // external default that ak previously wrote, while leaving the admitted - // declaration and MCP activation intact for routes or future selection. - // A configured fallback chain owns default selection independently and is - // handled below; it must not be erased by primary-provider deselection. - if (!hasChain && selectedProvider === null && ownedExternalDefault) { - delete next.defaultProvider; - clearExternalDefaultOwnership(next); - detail.push(`defaultProvider: ${ownedExternalDefault} retired`); - changed = true; - } - - if (hasChain) { - const selectable = new Set(aqeSelectableChainProviderTypes()); - const valid = chain.filter((e) => e?.provider && selectable.has(e.provider) - && (!(e.provider in desiredExternal) || externalActive.has(e.provider))); - if (valid.length === 0) { - // A bad chain must NOT block the independent agentOverrides projection — the - // Activity routing is validated separately. Record it and carry on. - error = 'no valid providers in fallback chain'; - detail.push(`chain: ⚠ ${error}`); - } else { - const requestedDefault = cfg.providers.aqeProvider; - const requestedUnavailable = requestedDefault in desiredExternal && !externalActive.has(requestedDefault); - next.defaultProvider = requestedUnavailable ? valid[0].provider : requestedDefault ?? valid[0].provider; - setFallbackDefaultOwnership(next, next.defaultProvider); - next.providers = { ...(next.providers ?? existing.providers ?? {}) }; - for (const e of valid) { - if (!(e.provider in desiredExternal)) next.providers[e.provider] = { ...(existing.providers?.[e.provider] ?? {}), enabled: true }; - } - next.fallbackChain = buildChain(valid); - if (next.defaultProvider in desiredExternal && externalActive.has(next.defaultProvider)) { - setExternalDefaultOwnership(next, next.defaultProvider); - } else if (ownedExternalDefault) { - clearExternalDefaultOwnership(next); - } - const emptyModels = valid.filter((e) => !e.models || e.models.length === 0).map((e) => e.provider); - // Warn, never refuse: the user may export the key later, and silently - // dropping a rung is worse than writing one that is currently inert (#54). - const gaps = credentialGaps(valid); - detail.push(`chain: ${valid.map((e) => e.provider).join(' → ')}` - + (emptyModels.length ? ` (⚠ no models for: ${emptyModels.join(', ')})` : '') - + (gaps.length ? ` (⚠ no credential for: ${gaps.map((g) => `${g.provider} — needs ${g.missing.join(', ')}`).join('; ')})` : '')); - changed = true; - } - } - - // External provider selection is project-local by contract: AQE discovers it - // from this file only. managedEnv deliberately never exports an external id - // into project or user host settings. - if (selectedProvider && selectedProvider in desiredExternal) { - if (externalActive.has(selectedProvider)) { - next.defaultProvider = selectedProvider; - setExternalDefaultOwnership(next, selectedProvider); - detail.push(`defaultProvider: ${selectedProvider} (project-local external)`); - changed = true; - } else { - error ??= `external default '${selectedProvider}' is not safely managed`; - } - } - - return { detail, error, changed }; -} - -/** Surface 4/4: project `routing.routes` into aqe's `agentOverrides`, merged - * with (not replacing) foreign entries, pruning only the ak-owned entries - * the current projection no longer names (`ctx.staleOverrides`, refined by - * surface 1 against the final external-availability set). */ -function reconcileAgentOverridesSurface(next, ctx) { - const { - existing, desiredExternal, priorOverrides, projected, staleOverrides, hasPolicy, agentOverridesSupported, - } = ctx; - if ((agentOverridesSupported && Object.keys(projected).length) || staleOverrides.length) { - // MERGE, don't replace: ak owns only the curated agent-types it projects; - // preserve foreign entries (aqe's own defaults or a hand-added agent). The - // projector drops non-constructible providers (mirrors sanitizeAgentOverrides) - // and only ever emits {provider, model} — no apiKey. - next.agentOverrides = { ...priorOverrides }; - for (const agent of staleOverrides) delete next.agentOverrides[agent]; - if (agentOverridesSupported) Object.assign(next.agentOverrides, projected); - // An override naming a provider is inert until that provider is ENABLED in - // this same file: aqe enables from env keys or the `providers` map, and a - // subscription host-CLI provider (codex, claude-code) has no env key at - // all — so ak-projected codex overrides sat dead and warned on every aqe - // startup (#108 phase 3). Enable exactly the providers the projection - // references — merge-not-clobber, writing nothing beyond `enabled`. - const referenced = agentOverridesSupported - ? [...new Set(Object.values(projected).map((entry) => entry.provider))] - : []; - if (referenced.length) { - next.providers = { ...(next.providers ?? existing.providers ?? {}) }; - for (const provider of referenced) { - if (!(provider in desiredExternal)) next.providers[provider] = { ...(next.providers[provider] ?? {}), enabled: true }; - } - } - return { - changed: true, - detail: `agentOverrides: ${agentOverridesSupported ? Object.keys(projected).length : 0} agents` - + (referenced.length ? ` (providers enabled: ${referenced.join(', ')})` : '') - + (staleOverrides.length ? ` (${staleOverrides.length} stale ak entries pruned)` : '') - + (!agentOverridesSupported ? ' (new projection skipped; needs agentic-qe ≥ 3.13.1)' : ''), - }; - } - if (hasPolicy && !agentOverridesSupported) return { detail: 'agentOverrides: skipped (needs agentic-qe ≥ 3.13.1)' }; - if (hasPolicy && Object.keys(projected).length === 0) return { detail: 'agentOverrides: skipped (no safely constructible providers)' }; - return null; -} - -const AQE_ROUTER_SURFACES = [ - reconcileExternalProvidersSurface, - reconcileFallbackRetirementSurface, - reconcileDefaultProviderSurface, - reconcileAgentOverridesSurface, -]; - -/** Fold an ordered list of `(draft, ctx) => {detail, error, changed, ctx?}` - * surface reconcilers over one draft, left to right. A surface's own `ctx` - * patch (if any) is applied before the next surface runs — the only - * sanctioned channel for one surface's output to inform a later one (see the - * section comment above AQE_ROUTER_SURFACES). `draft`/`ctx` are mutated in - * place as usual; returns the accumulated {details, changed, error}. */ -function foldSurfaces(surfaces, draft, ctx) { - const details = []; - let changed = false; - let error = null; - for (const reconcile of surfaces) { - const result = reconcile(draft, ctx); - if (!result) continue; - if (result.detail) { - if (Array.isArray(result.detail)) details.push(...result.detail); - else details.push(result.detail); - } - if (result.changed) changed = true; - if (result.error) error ??= result.error; - if (result.ctx) Object.assign(ctx, result.ctx); - } - return { details, changed, error }; -} - -/** True when nothing in `cfg`/the on-disk file requires any router surface to - * run — the router file is left untouched (and unread beyond this check). */ -function aqeRouterHasNothingToApply({ - hasChain, hasPolicy, hasExternal, hasOwnedExternal, hasManagedFallback, - hasExternalDefaultReceipt, hasFallbackDefaultReceipt, staleOverrides, -}) { - return !hasChain && !hasPolicy && !hasExternal && !hasOwnedExternal && !hasManagedFallback - && !hasExternalDefaultReceipt && !hasFallbackDefaultReceipt && staleOverrides.length === 0; -} - -/** Exact receipts never regain authority. If a user changes the default away - * from the value ak wrote (external default), or the managed fallback chain - * that derived a default is gone or no longer owned, relinquish that receipt - * immediately — changing it back later is still a user write and cannot - * resurrect it. Runs before any surface, on the initial draft. */ -function clearStaleDefaultReceipts(next, { - hasExternalDefaultReceipt, ownedExternalDefault, hasFallbackDefaultReceipt, ownedFallbackDefault, hasManagedFallback, -}) { - if (hasExternalDefaultReceipt && !ownedExternalDefault) clearExternalDefaultOwnership(next); - if (hasFallbackDefaultReceipt && (!ownedFallbackDefault || !hasManagedFallback)) clearFallbackDefaultOwnership(next); -} - -/** Build the read-only context the AQE-router fold needs: repo-root resolution - * (same gate as settingsTarget — the scope gates must never disagree about - * what "in a project" means), the on-disk router file, and every derived - * fact/flag the surfaces consume. Returns null outside a project. Split out - * of applyAqeRouter so a read-only comparator (aqeRouterDrift) can ask "what - * would the writer converge this to" without ever touching disk (#129) — - * the ONE construction of this context, shared by the writer and the reader. */ -function buildAqeRouterContext(cfg, cwd) { - const chain = cfg.providers?.aqeFallback ?? []; - const policy = cfg.routing?.routes ?? {}; - const selectedProvider = cfg.providers?.aqeProvider ?? null; - const hasChain = chain.length > 0; - const hasPolicy = Object.keys(policy).length > 0; - const root = paths.repoRoot(cwd); - if (!root) return null; - const file = aqeRouterFile(root); - const existing = readJson(file, {}) ?? {}; - const ownedExternalDefault = exactlyOwnedExternalDefault(existing); - const ownedFallbackDefault = exactlyOwnedFallbackDefault(existing); - const desiredExternal = aqeExternalProviders({ projectRoot: root }); - const hasExternal = Object.keys(desiredExternal).length > 0; - const hasOwnedExternal = Object.keys(existing[AQE_OWNERSHIP_KEY]?.externalProviders ?? {}).length > 0; - const existingOwnership = plainRecord(existing[AQE_OWNERSHIP_KEY]) ?? {}; - const hasExternalDefaultReceipt = Object.hasOwn(existingOwnership, 'externalDefaultProvider'); - const hasFallbackDefaultReceipt = Object.hasOwn(existingOwnership, 'fallbackDefaultProvider'); - const hasManagedFallback = existing.fallbackChain?.id === AQE_MANAGED_TAG; - const priorOverrides = existing.agentOverrides ?? {}; - const projected = configuredPolicyToAgentOverrides(policy); - const managedOverrideKeys = new Set(Object.keys(AGENT_ACTIVITY_MAP)); - const staleOverrides = Object.keys(priorOverrides) - .filter((agent) => managedOverrideKeys.has(agent) && !(agent in projected)); - - const facts = { - hasChain, hasPolicy, hasExternal, hasOwnedExternal, hasManagedFallback, - hasExternalDefaultReceipt, hasFallbackDefaultReceipt, staleOverrides, - }; - const ctx = { - cfg, - existing, - chain, - selectedProvider, - hasChain, - hasPolicy, - desiredExternal, - hasExternal, - hasOwnedExternal, - hasManagedFallback, - ownedExternalDefault, - ownedFallbackDefault, - priorOverrides, - managedOverrideKeys, - projected, - staleOverrides, - externalActive: new Set(), - externalSupported: aqeSupportsExternalProviders(), - agentOverridesSupported: aqeSupportsAgentOverrides(), - }; - return { - root, file, existing, facts, ctx, - }; -} - -/** Run the ordered AQE_ROUTER_SURFACES fold over a fresh draft cloned from - * `existing` — mutates neither `existing` nor disk. This is the pure "what - * would the writer converge this to" computation shared by applyAqeRouter - * (which persists the result when it differs) and aqeRouterDrift (which only - * needs to know what the writer WOULD produce). */ -function runAqeRouterFold({ existing, facts, ctx }) { - const next = { ...existing }; - clearStaleDefaultReceipts(next, { - hasExternalDefaultReceipt: facts.hasExternalDefaultReceipt, - ownedExternalDefault: ctx.ownedExternalDefault, - hasFallbackDefaultReceipt: facts.hasFallbackDefaultReceipt, - ownedFallbackDefault: ctx.ownedFallbackDefault, - hasManagedFallback: facts.hasManagedFallback, - }); - const { details, changed, error } = foldSurfaces(AQE_ROUTER_SURFACES, next, ctx); - return { - next, details, changed, error, - }; -} - -/** Write ak's managed router config into `.agentic-qe/llm-config.json`, merged - * into any existing file (backup-first, never persisting apiKey): - * - the ordered fallback chain + enabled set + default provider (from - * `aqeFallback`), and - * - the per-activity `agentOverrides` map projected from `routing.routes` - * (issue #568; only when installed aqe ≥ 3.13.1). - * No-op unless at least one of those is configured and we are in a project. - * Folds AQE_ROUTER_SURFACES over one draft (see the section comment above); - * this function is the setup (context + initial draft), the fold, and the - * final change-detect-and-write. - * Returns {ok, changed, detail}. */ -export function applyAqeRouter(cfg, cwd = process.cwd()) { - const built = buildAqeRouterContext(cfg, cwd); - if (!built) return { ok: true, changed: false, detail: 'not a project — aqe router unmanaged' }; - const { file, existing, facts } = built; - if (aqeRouterHasNothingToApply(facts)) { - return { ok: true, changed: false, detail: 'no aqe router config to apply' }; - } - - const { next, details, changed: surfacesChanged, error } = runAqeRouterFold(built); - - // One exact compare, reused for both phases below (the prior version - // stringified `existing` twice for the same never-mutated object). - const existingSnapshot = JSON.stringify(stableValue(existing)); - const changed = surfacesChanged || JSON.stringify(stableValue(next)) !== existingSnapshot; - if (!changed) return { ok: !error, changed: false, detail: details.join('; ') || 'nothing to apply' }; - next._managedBy = AQE_MANAGED_TAG; - // A surface reporting `changed: true` means this invocation owns at least - // one projection surface; it does not by itself mean the artifact changed. - // Compare the complete managed value (including the ownership tag) before - // touching disk so a converged external default/fallback/override remains - // byte- and mtime-stable across repeated syncs. - if (JSON.stringify(stableValue(next)) === existingSnapshot) { - return { ok: !error, changed: false, detail: details.join('; ') || 'nothing to apply' }; - } - fs.mkdirSync(path.dirname(file), { recursive: true }); - writeJsonWithBackup(file, next); - return { ok: !error, changed: true, detail: details.join('; ') }; -} - -/** Read-only: does the persisted fallback-chain order in - * `.agentic-qe/llm-config.json` differ from what applyAqeRouter would - * converge it to right now? Runs the SAME dry-run fold the writer runs - * (buildAqeRouterContext + runAqeRouterFold) and reads only the - * fallback-chain slice of the result — the writer's own chain-validity - * filter (reconcileDefaultProviderSurface's `valid`), not a re-derived - * approximation, so the two can never disagree (#129). Scoped to chain order - * only, same as before: agentOverrides/external-provider drift are each a - * sibling status section's own concern. - * `applicable: false` means there is no chain to compare (nothing configured, - * or outside the project scope applyAqeRouter itself declines to manage). */ -export function aqeRouterDrift(cfg, cwd = process.cwd()) { - const chain = cfg.providers?.aqeFallback ?? []; - if (chain.length === 0) return { applicable: false, drift: false, order: '' }; - const built = buildAqeRouterContext(cfg, cwd); - if (!built) return { applicable: false, drift: false, order: '' }; - const { existing } = built; - const { next } = runAqeRouterFold(built); - const order = (next.fallbackChain?.entries ?? []).map((e) => e.provider).join('→'); - const diskOrder = (existing.fallbackChain?.entries ?? []).map((e) => e.provider).join('→'); - const drift = order ? (existing._managedBy !== AQE_MANAGED_TAG || diskOrder !== order) : diskOrder !== ''; - return { applicable: true, drift, order }; -} - -/** Reversible teardown of ak's router management. Restores the pre-ak file from - * its one-time .bak, or removes an ak-created file. Never touches a file ak - * didn't write (no `_managedBy` tag). */ -export function undoAqeRouter(cwd = process.cwd()) { - const file = aqeRouterFile(cwd); - if (!fs.existsSync(file)) return { ok: true, changed: false, detail: 'no aqe router config' }; - const cur = readJson(file); - if (cur?._managedBy !== AQE_MANAGED_TAG) return { ok: true, changed: false, detail: 'llm-config.json not ak-managed — left as-is' }; - const bak = `${file}.bak`; - if (fs.existsSync(bak)) { - fs.copyFileSync(bak, file); - fs.rmSync(bak, { force: true }); - return { ok: true, changed: true, detail: 'restored pre-ak llm-config.json' }; - } - fs.rmSync(file, { force: true }); - return { ok: true, changed: true, detail: 'removed ak-created llm-config.json' }; -} - // ── per-activity host routing (kit.json routing.routes) ───────────────────── // Seed/format helpers shared by `ak x host` and `ak setup`. The pure policy // core + projectors live in routing.mjs; these bridge it to kit.json + the CLI. @@ -1477,6 +862,47 @@ export function persistedRufloProvider(cwd, providerId, { env = process.env } = && entry.name.toLowerCase() === providerId.toLowerCase()) ?? null; } +/** A provider entry's id/model grammar is valid: id always required, model + * only when present (a provider entry may name a bare provider, no model). */ +function validProviderEntry(m) { + return typeof m.id === 'string' && PROVIDER_ID_RE.test(m.id) + && (!m.model || (typeof m.model === 'string' && PROVIDER_MODEL_RE.test(m.model))); +} + +/** The `-e ` args for one provider entry, when any apply: explicit, + * or Ollama's standard loopback endpoint on a fresh entry with no existing + * Ruflo-persisted baseUrl or env override. `ok: false` on an invalid + * explicit endpoint — the caller must not proceed to invoke ruflo. */ +function resolveProviderEndpointArgs(m, cwd, env) { + const existing = persistedRufloProvider(cwd, m.id, { env }); + let endpoint = m.endpoint; + if (endpoint === undefined && m.id.toLowerCase() === 'ollama' && !existing?.baseUrl + && !env.OLLAMA_BASE_URL && !env.OLLAMA_API_KEY) { + endpoint = DEFAULT_OLLAMA_ENDPOINT; + } + if (endpoint === undefined) return { ok: true, args: [] }; + const validation = typeof endpoint === 'string' + ? validateEndpoint(endpoint) + : { ok: false, reason: 'invalid-url' }; + if (!validation.ok) return { ok: false, reason: validation.reason }; + return { ok: true, args: ['-e', validation.normalized] }; +} + +/** Register one provider entry with ruflo. Returns the done-list label + * (`id`, `id(invalid)`, `id(invalid endpoint: …)`, `id(failed)`) and whether + * a ruflo invocation was actually attempted (never true for an id/model or + * endpoint that failed validation — those never reach ruflo at all). */ +async function applyOneProvider(m, cwd, env, runner) { + if (!validProviderEntry(m)) return { label: `${m.id}(invalid)`, attempted: false }; + const args = ['providers', 'configure', '-p', m.id]; + if (m.model) args.push('-m', m.model); + const endpoint = resolveProviderEndpointArgs(m, cwd, env); + if (!endpoint.ok) return { label: `${m.id}(invalid endpoint: ${endpoint.reason})`, attempted: false }; + args.push(...endpoint.args); + const r = await runner('ruflo', args, { cwd, timeout: 60_000 }); + return { label: `${m.id}${r.code === 0 ? '' : '(failed)'}`, attempted: true }; +} + /** Register configured providers with Ruflo (keys read from env, never passed * here). Idempotent — Ruflo upserts. A fresh Ollama entry receives its standard * loopback endpoint; a pre-existing custom Ruflo endpoint is preserved. */ @@ -1496,32 +922,9 @@ export async function applyProviders(cfg, cwd = process.cwd(), { let attempted = 0; for (const m of models) { if (!m?.id) continue; - if (typeof m.id !== 'string' || !PROVIDER_ID_RE.test(m.id) - || (m.model && (typeof m.model !== 'string' || !PROVIDER_MODEL_RE.test(m.model)))) { - done.push(`${m.id}(invalid)`); - continue; - } - const args = ['providers', 'configure', '-p', m.id]; - if (m.model) args.push('-m', m.model); - const existing = persistedRufloProvider(cwd, m.id, { env }); - let endpoint = m.endpoint; - if (endpoint === undefined && m.id.toLowerCase() === 'ollama' && !existing?.baseUrl - && !env.OLLAMA_BASE_URL && !env.OLLAMA_API_KEY) { - endpoint = DEFAULT_OLLAMA_ENDPOINT; - } - if (endpoint !== undefined) { - const validation = typeof endpoint === 'string' - ? validateEndpoint(endpoint) - : { ok: false, reason: 'invalid-url' }; - if (!validation.ok) { - done.push(`${m.id}(invalid endpoint: ${validation.reason})`); - continue; - } - args.push('-e', validation.normalized); - } - attempted += 1; - const r = await runner('ruflo', args, { cwd, timeout: 60_000 }); - done.push(`${m.id}${r.code === 0 ? '' : '(failed)'}`); + const result = await applyOneProvider(m, cwd, env, runner); + done.push(result.label); + if (result.attempted) attempted += 1; } const ok = done.every((d) => !d.includes('failed') && !d.includes('invalid')); const compatibility = providerSelectionSupported ? '' diff --git a/src/lib/telemetry-records.mjs b/src/lib/telemetry-records.mjs index c1b8335..8dddd00 100644 --- a/src/lib/telemetry-records.mjs +++ b/src/lib/telemetry-records.mjs @@ -95,6 +95,78 @@ function decodeCodexMessage(payload) { }; } +/** `session_meta` → the authoritative session id, cwd, and thread_source (a + * `"subagent"` value marks a thread_spawn replay whose tokens the batch + * parser excludes from aggregation). */ +function decodeSessionMeta(payload) { + return { + type: 'meta', + sessionId: payload.id, + cwd: payload.cwd, + threadSource: payload.thread_source, + provider: resolveCodexProvider(payload), + model: payload.model, + }; +} + +/** `turn_context` → the model id in effect from this point on. */ +function decodeTurnContext(payload) { + return { + type: 'turnContext', + cwd: payload.cwd, + provider: resolveCodexProvider(payload), + model: payload.model, + }; +} + +/** `event_msg` → `token_count`: a CUMULATIVE usage snapshot, so only the last + * one a caller sees should be kept. */ +function decodeTokenCount(payload) { + return { + type: 'tokenCount', + usage: { + total: payload.info?.total_token_usage && typeof payload.info.total_token_usage === 'object' + ? payload.info.total_token_usage : null, + rateLimits: payload.rate_limits && typeof payload.rate_limits === 'object' + ? payload.rate_limits : null, + }, + }; +} + +/** `event_msg` → everything but `token_count`: a message (legacy or + * item_completed), a lifecycle event, or unrecognized. */ +function decodeEventMsg(payload) { + if (payload.type === 'token_count') return decodeTokenCount(payload); + const message = decodeCodexMessage(payload); + if (message) { + return message.role + ? { type: 'message', role: message.role, text: message.text, generation: message.generation } + : { type: null, generation: message.generation, unknownItemType: message.unknownItemType }; + } + if (['task_complete', 'turn_aborted'].includes(payload.type)) { + return { type: 'lifecycle', status: payload.type === 'turn_aborted' ? 'cancelled' : 'completed' }; + } + return { type: null }; +} + +/** `response_item` → a tool call or its result. */ +function decodeResponseItem(payload) { + if (['function_call', 'custom_tool_call'].includes(payload.type)) { + return { type: 'toolCall', callId: payload.call_id ?? payload.id, toolName: payload.name }; + } + if (['function_call_output', 'custom_tool_call_output'].includes(payload.type)) { + return { type: 'toolResult', callId: payload.call_id ?? payload.id }; + } + return { type: null }; +} + +const CODEX_RECORD_DECODERS = { + session_meta: decodeSessionMeta, + turn_context: decodeTurnContext, + event_msg: decodeEventMsg, + response_item: decodeResponseItem, +}; + /** * Decode one raw Codex rollout JSONL record (`{type, payload, timestamp}`) * into a normalized description of what it is. Fields are raw pass-throughs @@ -111,62 +183,44 @@ function decodeCodexMessage(payload) { */ export function decodeCodexRecord(record) { const payload = record?.payload && typeof record.payload === 'object' ? record.payload : {}; + const decoder = CODEX_RECORD_DECODERS[record?.type]; + return decoder ? decoder(payload) : { type: null }; +} - if (record?.type === 'session_meta') { - return { - type: 'meta', - sessionId: payload.id, - cwd: payload.cwd, - threadSource: payload.thread_source, - provider: resolveCodexProvider(payload), - model: payload.model, - }; - } - - if (record?.type === 'turn_context') { - return { - type: 'turnContext', - cwd: payload.cwd, - provider: resolveCodexProvider(payload), - model: payload.model, - }; - } - - if (record?.type === 'event_msg') { - if (payload.type === 'token_count') { - return { - type: 'tokenCount', - usage: { - total: payload.info?.total_token_usage && typeof payload.info.total_token_usage === 'object' - ? payload.info.total_token_usage : null, - rateLimits: payload.rate_limits && typeof payload.rate_limits === 'object' - ? payload.rate_limits : null, - }, - }; - } - const message = decodeCodexMessage(payload); - if (message) { - return message.role - ? { type: 'message', role: message.role, text: message.text, generation: message.generation } - : { type: null, generation: message.generation, unknownItemType: message.unknownItemType }; - } - if (['task_complete', 'turn_aborted'].includes(payload.type)) { - return { type: 'lifecycle', status: payload.type === 'turn_aborted' ? 'cancelled' : 'completed' }; - } - return { type: null }; +/** Partition a Claude content-block array into its tool_use/tool_result + * blocks — the two block kinds any consumer here treats as structured + * rather than display text. */ +function splitClaudeBlocks(blocks) { + const toolUses = []; + const toolResults = []; + for (const block of blocks) { + if (!block || typeof block !== 'object') continue; + if (block.type === 'tool_use') toolUses.push({ id: block.id, name: block.name }); + else if (block.type === 'tool_result') toolResults.push({ id: block.tool_use_id, isError: block.is_error === true }); } + return { toolUses, toolResults }; +} - if (record?.type === 'response_item') { - if (['function_call', 'custom_tool_call'].includes(payload.type)) { - return { type: 'toolCall', callId: payload.call_id ?? payload.id, toolName: payload.name }; - } - if (['function_call_output', 'custom_tool_call_output'].includes(payload.type)) { - return { type: 'toolResult', callId: payload.call_id ?? payload.id }; - } - return { type: null }; - } +/** `user`/`assistant`/`null` — any other `record.type` (tool results, meta + * rows) is not a message role this module normalizes. */ +function claudeRole(record) { + if (record?.type === 'user') return 'user'; + if (record?.type === 'assistant') return 'assistant'; + return null; +} - return { type: null }; +/** Claude's four token-usage fields, normalized to zero rather than NaN. + * `cache_read_input_tokens`/`cache_creation_input_tokens` are separate + * fields the API already reports, kept apart rather than folded into + * `input_tokens` (which would double them into gross input). */ +function claudeUsage(record) { + const usage = record?.message?.usage ?? {}; + return { + input: Number(usage.input_tokens) || 0, + output: Number(usage.output_tokens) || 0, + cacheRead: Number(usage.cache_read_input_tokens) || 0, + cacheWrite: Number(usage.cache_creation_input_tokens) || 0, + }; } /** @@ -182,16 +236,9 @@ export function decodeCodexRecord(record) { export function decodeClaudeRecord(record) { const content = record?.message?.content; const blocks = Array.isArray(content) ? content : []; - const toolUses = []; - const toolResults = []; - for (const block of blocks) { - if (!block || typeof block !== 'object') continue; - if (block.type === 'tool_use') toolUses.push({ id: block.id, name: block.name }); - else if (block.type === 'tool_result') toolResults.push({ id: block.tool_use_id, isError: block.is_error === true }); - } - const usage = record?.message?.usage ?? {}; + const { toolUses, toolResults } = splitClaudeBlocks(blocks); return { - role: record?.type === 'user' ? 'user' : record?.type === 'assistant' ? 'assistant' : null, + role: claudeRole(record), sessionId: record?.sessionId, agentId: record?.agentId, isSidechain: record?.isSidechain === true, @@ -200,11 +247,6 @@ export function decodeClaudeRecord(record) { text: claudeText(content), toolUses, toolResults, - usage: { - input: Number(usage.input_tokens) || 0, - output: Number(usage.output_tokens) || 0, - cacheRead: Number(usage.cache_read_input_tokens) || 0, - cacheWrite: Number(usage.cache_creation_input_tokens) || 0, - }, + usage: claudeUsage(record), }; } diff --git a/src/lib/usage-aggregate.mjs b/src/lib/usage-aggregate.mjs new file mode 100644 index 0000000..4a553a9 --- /dev/null +++ b/src/lib/usage-aggregate.mjs @@ -0,0 +1,517 @@ +// usage-aggregate.mjs — pure arithmetic over ALREADY-PARSED session records: +// interval math, secret masking, and the two shapes usage-index.mjs hands its +// consumers (the batch Aggregate from `aggregate()`, and the single-session +// `/api/session` payload from `sessionPayload()`). No file I/O, no caching — +// that stays in usage-index.mjs. No wire-format knowledge either — that lives +// in telemetry-records.mjs and usage-parsers.mjs. This module answers "what +// do these already-decoded records add up to", nothing about how they got +// decoded. +// +// Zero imports from sibling usage-* modules by design: usage-parsers.mjs +// imports `toMs`/`maskSecrets` FROM here, so this file must not import back +// from it (or from usage-index.mjs) to keep that a one-way dependency. + +/** Milliseconds from an epoch number, Date, or ISO string; NaN when unusable. + * Exported only for usage-parsers.mjs's own timestamp parsing — not part of + * this module's documented public surface. */ +export function toMs(v) { + if (typeof v === 'number') return Number.isFinite(v) ? v : NaN; + if (v instanceof Date) return v.getTime(); + if (typeof v === 'string') return Date.parse(v); + return NaN; +} + +/** + * Total seconds covered by a set of intervals, counting overlap ONCE. + * Accepts `[start, end]` tuples or `{ start, end }` objects; each bound may be + * an epoch number, a Date, or an ISO string. Degenerate or unparseable + * intervals are dropped rather than throwing. Pure — exported for test. + */ +export function mergeIntervals(intervals) { + if (!Array.isArray(intervals)) return 0; + const spans = []; + for (const iv of intervals) { + if (!iv) continue; + const start = toMs(Array.isArray(iv) ? iv[0] : iv.start); + const end = toMs(Array.isArray(iv) ? iv[1] : iv.end); + if (!Number.isFinite(start) || !Number.isFinite(end) || end <= start) continue; + spans.push([start, end]); + } + if (!spans.length) return 0; + spans.sort((a, b) => a[0] - b[0]); + let total = 0; + let [curStart, curEnd] = spans[0]; + for (let i = 1; i < spans.length; i++) { + const [s, e] = spans[i]; + if (s <= curEnd) { // overlapping OR exactly touching → extend + if (e > curEnd) curEnd = e; + } else { + total += curEnd - curStart; + curStart = s; curEnd = e; + } + } + total += curEnd - curStart; + return Math.round(total / 1000); +} + +/** Largest single transcript readSession will pull into memory. The corpus's + * biggest real file is ~18 MB; JSON expansion runs ~5x, so 64 MB caps the + * spike near 320 MB instead of unbounded. Above this the session reads as + * unavailable rather than risking the panel's process. Lives here (not + * usage-index.mjs) only incidentally — see MAX_TURN_CHARS below for the one + * that matters to this module. */ +export const MAX_TURN_CHARS = 40_000; + +// Each replacement keeps the human-readable prefix and drops the payload. The +// replacement text cannot re-match its own pattern (the '…' is outside every +// character class), which is what makes masking idempotent. +/** @type {[RegExp, string][]} */ +// Order matters. PEM blocks are matched WHOLE and first: a later pattern that +// nibbled at the base64 body would leave the armour behind, which reads as +// "masked" on screen while the key material is still visible. +const SECRET_PATTERNS = [ + [/-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/g, + '-----BEGIN PRIVATE KEY----- …redacted -----END PRIVATE KEY-----'], + [/-----BEGIN [A-Z ]*PRIVATE KEY-----/g, '-----BEGIN PRIVATE KEY----- …redacted'], + [/\bsk-[A-Za-z0-9_-]{12,}/g, 'sk-…redacted'], + [/\bgh[pousr]_[A-Za-z0-9]{20,}/g, 'ghp_…redacted'], + [/\bAKIA[0-9A-Z]{12,}/g, 'AKIA…redacted'], + [/\bwhsec_[A-Za-z0-9_-]{16,}/g, 'whsec_…redacted'], + [/\bASIA[0-9A-Z]{12,}/g, 'ASIA…redacted'], // STS temporary credential + [/\bgithub_pat_[A-Za-z0-9_]{20,}/g, 'github_pat_…redacted'], + [/\bglpat-[A-Za-z0-9_-]{16,}/g, 'glpat-…redacted'], + [/\bhf_[A-Za-z0-9]{20,}/g, 'hf_…redacted'], + [/\bpypi-[A-Za-z0-9_-]{16,}/g, 'pypi-…redacted'], + [/\b(?:sk|rk)_live_[A-Za-z0-9]{16,}/g, 'sk_live_…redacted'], + [/\bSG\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}/g, 'SG.…redacted'], + [/\bnpm_[A-Za-z0-9]{30,}/g, 'npm_…redacted'], + [/\bxox[baprse]-[A-Za-z0-9-]{10,}/g, 'xox…redacted'], + [/\bAIza[0-9A-Za-z_-]{30,}/g, 'AIza…redacted'], + // Slack incoming-webhook URLs are bearer credentials in URL form. + [/https:\/\/hooks\.slack\.com\/services\/[A-Za-z0-9/]+/g, + 'https://hooks.slack.com/services/…redacted'], + // JWT — three base64url segments. Masked whole; the payload is the sensitive part. + [/\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}/g, 'eyJ…redacted'], + // Inline credentials in a URI (postgres://user:pass@host, https://u:p@h). The + // USERNAME is deliberately preserved — it is usually needed to identify which + // system leaked, and it is not the secret. + [/\b([a-z][a-z0-9+.-]*):\/\/([^\s:@/]+):[^\s@/]+@/g, '$1://$2:…redacted@'], + [/\b(Basic)\s+[A-Za-z0-9+/]{16,}={0,2}/g, '$1 …redacted'], + // Case-insensitive on the SCHEME only — `BEARER`/`bearer`/`Bearer` all appear + // in the wild. The token body stays case-sensitive, so this cannot widen into + // prose the way an /i on the assignment rule would. + [/\b([Bb]earer|BEARER)\s+[A-Za-z0-9._~+/=-]{16,}/g, '$1 …redacted'], + // Context-carried secrets: SCREAMING_CASE assignments whose NAME says secret. + // Deliberately case-SENSITIVE: with /i this matches prose like + // "tokens used = 10028979467", and these transcripts discuss token counts + // constantly. Uppercase-only tracks the actual env-var convention instead. + [/\b([A-Z][A-Z0-9_]*(?:SECRET|TOKEN|PASSWORD|PASSWD|API_?KEY|PRIVATE_KEY)[A-Z0-9_]*)(\s*[:=]\s*)("?)[^\s"']{8,}\3/g, + '$1$2$3…redacted$3'], + // Quoted JSON/JS object key whose name says secret — "apiKey": "…", 'client_secret': '…'. + // Case-insensitive is safe here: the quote delimiters are a shape prose never + // has, so this cannot widen into "tokens used" the way the assignment rule's + // /i would (that rule stays case-sensitive above for exactly that reason). + [/(["'][A-Za-z0-9_-]*(?:secret|token|password|passwd|api_?key|private_?key)[A-Za-z0-9_-]*["']\s*:\s*)(["'])[^"']{8,}\2/gi, + '$1$2…redacted$2'], + // Line-anchored YAML/TOML/ini assignment — api_key = …, password: … — with no + // quoting required. Anchored to line-start-through-key-through-:/= so it + // cannot match a key phrase floating mid-sentence ("tokens used = 10028979467" + // fails: "used" — not a secret-shaped word — sits directly before "="). + [/^([ \t]*[A-Za-z0-9_-]*(?:secret|token|password|passwd|api_?key|private_?key)[A-Za-z0-9_-]*[ \t]*[:=][ \t]*)(["']?)[^\s"']{8,}\2/gim, + '$1$2…redacted$2'], +]; + +// NOT masked, deliberately: a bare 40-char base64-ish AWS secret with no prefix +// and no assignment context. It is indistinguishable from a hash, a checksum, or +// a base64 blob in ordinary prose, and masking it would corrupt real transcript +// content. It is caught when it appears as AWS_SECRET_ACCESS_KEY=… above. + +/** + * Mask secret-shaped strings in transcript text (ADR-0009 §8). Idempotent, and + * a no-op on ordinary prose. Pure — exported for test. + */ +export function maskSecrets(text) { + if (text === null || text === undefined) return ''; + let out = typeof text === 'string' ? text : String(text); + for (const [re, sub] of SECRET_PATTERNS) out = out.replace(re, sub); + return out; +} + +/** + * Split the historical transcript-source `provider` field into an explicit host + * and independently evidenced inference provider. The legacy field used + * `claude`/`codex` to mean transcript host; those values therefore cannot prove + * Anthropic/OpenAI provider identity. + */ +export function normalizeSessionIdentity(record = {}) { + const legacyHost = !record.host && ['claude', 'codex'].includes(record.provider) + ? record.provider : null; + const host = record.host ?? legacyHost ?? null; + const provider = legacyHost ? null : (record.provider ?? null); + return { + ...record, + host, + provider, + model: record.model ?? null, + providerProvenance: provider ? (record.providerProvenance ?? 'unknown') : 'unknown', + }; +} + +const round = (n, p = 6) => Math.round(n * 10 ** p) / 10 ** p; + +/** Sum a record's per-model usage rows into one API-equivalent cost. Rows with + * an observed transcript cost (opencode) use it — same preference as aggregate. */ +function sessionCost(rec, deps) { + let cost = 0; + for (const row of rec.usage ?? []) { + cost += row.costObserved != null ? row.costObserved : (deps.costOf({ + model: row.model, provider: rec.provider, + input: row.input, output: row.output, cacheRead: row.cacheRead, cacheWrite: row.cacheWrite, + }) || 0); + } + return round(cost); +} + +// ── aggregation ───────────────────────────────────────────────────────────── + +// Buckets are keyed by transcript-derived strings (model ids, project names), so +// a session naming itself `__proto__` would hit Object.prototype's setter: the +// bucket never becomes an own property and vanishes from JSON.stringify, silently +// losing that model's spend. Callers build these maps with Object.create(null); +// this guard keeps the invariant local even if one forgets. +function bucket(map, key) { + if (!Object.prototype.hasOwnProperty.call(map, key)) { + map[key] = { + sessions: 0, responses: 0, input: 0, output: 0, cacheRead: 0, cacheWrite: 0, + tokens: 0, cost: 0, + // `minutes` and `confidence` are NOT decoration. Three consumers read them + // and every one of them silently rendered a zero before they existed: + // project rows showed "0m", every category showed "confidence 0.00" (which + // ADR-0009 §5 requires be DISPLAYED — a constant 0.00 is worse than + // omitting it), and high-volume-automation printed "0.0 minutes each" + // while its evidence asserted "short duration" — a duration claim it never + // measured, i.e. fabricated evidence under ADR-0009 §6 rule 1. + minutes: 0, + confidence: 0, // session-weighted mean, finalised by sealBuckets() + }; + } + return map[key]; +} + +function addTo(b, s) { + b.sessions++; b.responses += s.responses; + b.input += s.input; b.output += s.output; + b.cacheRead += s.cacheRead; b.cacheWrite += s.cacheWrite; + b.tokens += s.tokens; b.cost += s.cost; + b.minutes += Number(s.minutes) || 0; + // Accumulated as a SUM here and divided by `sessions` in sealBuckets, so the + // result is a true mean rather than a running average that drifts with order. + b.confidence += Number(s.confidence) || 0; +} + +/** Turn accumulated confidence sums into means. Must run after every addTo. */ +function sealBuckets(...maps) { + for (const map of maps) { + for (const k of Object.keys(map)) { + const b = map[k]; + b.confidence = b.sessions ? round(b.confidence / b.sessions, 3) : 0; + b.minutes = round(b.minutes, 2); + } + } +} + +/** Price one usage row (observed opencode cost wins over the pricing table — + * `day` prices it at the rate in effect WHEN THOSE TOKENS WERE SPENT, not + * today's) and fold it into a session's running sums plus the shared + * byDay/byModel buckets. Mutates `acc` and `activeDays`. */ +function foldSessionUsageRow(row, rec, deps, acc, byDay, byModel, activeDays) { + const rowCost = row.costObserved != null ? row.costObserved : (deps.costOf({ + model: row.model, provider: rec.provider, day: row.day, + input: row.input, output: row.output, cacheRead: row.cacheRead, cacheWrite: row.cacheWrite, + }) || 0); + acc.input += row.input; acc.output += row.output; + acc.cacheRead += row.cacheRead; acc.cacheWrite += row.cacheWrite; + acc.cost += rowCost; + + const rowTokens = row.input + row.output + row.cacheRead + row.cacheWrite; + if (!byDay[row.day]) byDay[row.day] = { tokens: 0, cost: 0, sessions: 0, sessionsActive: 0 }; + byDay[row.day].tokens += rowTokens; + byDay[row.day].cost = round(byDay[row.day].cost + rowCost); + activeDays.add(row.day); + + const m = bucket(byModel, row.model); + m.responses += row.responses; m.input += row.input; m.output += row.output; + m.cacheRead += row.cacheRead; m.cacheWrite += row.cacheWrite; + m.tokens += rowTokens; m.cost = round(m.cost + rowCost); +} + +/** Fold every usage row of one record; returns `{input, output, cacheRead, + * cacheWrite, cost, firstDay}` for buildSessionRow. `firstDay` is the day + * this session's tokens FIRST landed on — always a key of byDay, which keeps + * sum(byDay.sessions) === totals.sessions (a session's start day is not + * usable: it can open at 23:58 and only bill after midnight). */ +function foldSessionUsageRows(rec, deps, byDay, byModel) { + const acc = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, cost: 0 }; + let firstDay = null; + const activeDays = new Set(); + for (const row of rec.usage) { + if (firstDay === null || row.day < firstDay) firstDay = row.day; + foldSessionUsageRow(row, rec, deps, acc, byDay, byModel, activeDays); + } + for (const day of activeDays) byDay[day].sessionsActive++; + return { ...acc, firstDay }; +} + +/** One aggregate session row from a parsed record, its folded usage sums, + * and its classifier verdict. */ +function buildSessionRow(rec, usage, verdict) { + const { input, output, cacheRead, cacheWrite, cost, firstDay } = usage; + return { + id: rec.id, host: rec.host ?? rec.provider, + provider: rec.inferenceProvider ?? null, + transcriptProvider: rec.provider, + providerProvenance: rec.providerProvenance ?? 'unknown', + title: rec.title, project: rec.project, + worktree: rec.worktree ?? null, + start: new Date(rec.start ?? rec.end).toISOString(), + minutes: Math.round(((rec.end - (rec.start ?? rec.end)) / 60_000) * 10) / 10, + prompts: rec.prompts, responses: rec.responses, exceptions: rec.exceptions, + sidechain: rec.sidechain, threadSource: rec.threadSource, + models: rec.models.slice(), + input, output, cacheRead, cacheWrite, + tokens: input + output + cacheRead + cacheWrite, + cost: round(cost), + tools: { ...rec.tools }, + category: verdict.category ?? 'Unclassified', + confidence: verdict.confidence ?? 0, + basis: verdict.basis ?? 'no signal', + skill: rec.skill, plugin: rec.plugin, + // Codex-only detail (v6); zero / null on Claude sessions and on v5-cached + // records (the schema bump re-derives those). + reasoningOutput: rec.reasoningOutput ?? 0, + rateLimits: rec.rateLimits ?? null, + _span: [rec.start ?? rec.end, rec.end], + // Pre-v2 cache entries have no `active`; fall back to the whole span so a + // stale record degrades to the old figure instead of vanishing. + _active: Array.isArray(rec.active) && rec.active.length ? rec.active : [[rec.start ?? rec.end, rec.end]], + _punchcard: rec.punchcard, + _day: firstDay, + }; +} + +/** Records → aggregate session rows, folding usage into the shared byDay/ + * byModel buckets as a side effect. */ +function buildSessionRows(records, { cutoff, deps, byDay, byModel }) { + const sessions = []; + for (const rec of records) { + if (!rec || !rec.responses) continue; // no assistant turn → not a session + if (rec.end === null || rec.end < cutoff) continue; // outside the window + const usage = foldSessionUsageRows(rec, deps, byDay, byModel); + const verdict = deps.classify({ + title: rec.title, skill: rec.skill, plugin: rec.plugin, + tools: rec.tools, prompts: rec.prompts, responses: rec.responses, + }) ?? {}; + sessions.push(buildSessionRow(rec, usage, verdict)); + } + return sessions; +} + +/** A session counts once under each model it used — the token and cost + * columns already partition cleanly, session counts cannot. */ +function foldSessionByModel(byModel, s) { + for (const model of s.models) { + const b = bucket(byModel, model); + b.sessions++; + b.minutes += Number(s.minutes) || 0; + b.confidence += Number(s.confidence) || 0; + } +} + +function foldSessionIntoTree(tree, s) { + if (!tree.has(s.project)) tree.set(s.project, { project: s.project, sessions: 0, cost: 0, tokens: 0, minutes: 0, cats: new Map(), rows: [] }); + const node = tree.get(s.project); + node.sessions++; node.cost = round(node.cost + s.cost); node.tokens += s.tokens; + node.minutes = Math.round((node.minutes + s.minutes) * 10) / 10; + const cat = node.cats.get(s.category) ?? { category: s.category, sessions: 0, cost: 0 }; + cat.sessions++; cat.cost = round(cat.cost + s.cost); + node.cats.set(s.category, cat); + node.rows.push(s); +} + +/** Second pass over the (now sorted) session rows: totals, the by-host/ + * provider/project/category/model buckets, the punchcard, and the project + * tree. `byModel` is the SAME object buildSessionRows already populated + * from usage rows — this pass adds its session/minutes/confidence fields. */ +function foldSessionTotals(sessions, byDay, byModel) { + const totals = { + sessions: sessions.length, responses: 0, exceptions: 0, input: 0, output: 0, + cacheRead: 0, cacheWrite: 0, tokens: 0, cost: 0, + spanMinutes: 0, spanUnionSeconds: 0, engagedSeconds: 0, + }; + const byHost = Object.create(null), byProvider = Object.create(null); + const byProject = Object.create(null); + const byCategory = Object.create(null), punchcard = Object.create(null); + const tree = new Map(); + let spanMs = 0; + + for (const s of sessions) { + totals.responses += s.responses; totals.exceptions += s.exceptions; + totals.input += s.input; totals.output += s.output; + totals.cacheRead += s.cacheRead; totals.cacheWrite += s.cacheWrite; + totals.tokens += s.tokens; totals.cost += s.cost; + spanMs += s._span[1] - s._span[0]; + + addTo(bucket(byHost, s.host ?? 'unknown'), s); + addTo(bucket(byProvider, s.provider ?? 'unknown'), s); + addTo(bucket(byProject, s.project), s); + addTo(bucket(byCategory, s.category), s); + foldSessionByModel(byModel, s); + if (s._day && byDay[s._day]) byDay[s._day].sessions++; + for (const [k, n] of Object.entries(s._punchcard)) punchcard[k] = (punchcard[k] ?? 0) + n; + foldSessionIntoTree(tree, s); + } + + return { totals, byHost, byProvider, byProject, byCategory, punchcard, tree, spanMs }; +} + +function buildProjectTree(tree) { + return [...tree.values()] + .map((n) => ({ + project: n.project, sessions: n.sessions, cost: n.cost, tokens: n.tokens, minutes: n.minutes, + categories: [...n.cats.values()].sort((a, b) => b.cost - a.cost || b.sessions - a.sessions), + rows: n.rows, + })) + .sort((a, b) => b.cost - a.cost || b.tokens - a.tokens); +} + +/** Local rate-limit history: every Codex rollout embeds live quota snapshots + * in its token_count events, so a utilization time series is reconstructable + * retroactively with ZERO network — one point per session (its last + * snapshot), oldest first. Claude has no local analogue (its quota arrives + * via the statusline push — see quota.mjs), hence codex-prefixed. */ +function buildCodexRateLimits(sessions) { + return sessions + .filter((s) => s.host === 'codex' && s.rateLimits && Number.isFinite(s.rateLimits.at)) + .map((s) => s.rateLimits) + .sort((x, y) => x.at - y.at); +} + +/** Turn cached per-file records into the Aggregate the UI and detectors read. */ +export function aggregate(records, { days, now, cutoff, deps }) { + // Null-prototype: these are keyed by transcript-derived strings (day, model id, + // provider, project, category), so `__proto__` as a key must be an ordinary + // bucket, not a prototype write that silently discards the data. + const byDay = Object.create(null); + const byModel = Object.create(null); + + const sessions = buildSessionRows(records, { cutoff, deps, byDay, byModel }); + sessions.sort((a, b) => b.cost - a.cost || Date.parse(b.start) - Date.parse(a.start)); + + const { totals, byHost, byProvider, byProject, byCategory, punchcard, tree, spanMs } = + foldSessionTotals(sessions, byDay, byModel); + + totals.cost = round(totals.cost); + // Three tiers, each honest about a different thing: + // engagedSeconds — union of ACTIVE intervals: time actually worked + // spanUnionSeconds — union of whole spans: wall-clock with a session open + // spanMinutes — sum of spans: the double-counting figure, kept as the + // clearly-labelled secondary the ADR asks the UI to show + sealBuckets(byHost, byProvider, byProject, byCategory, byModel); + + totals.spanMinutes = Math.round((spanMs / 60_000) * 10) / 10; + totals.spanUnionSeconds = mergeIntervals(sessions.map((s) => s._span)); + totals.engagedSeconds = mergeIntervals(sessions.flatMap((s) => s._active)); + + const projectTree = buildProjectTree(tree); + for (const s of sessions) { delete s._span; delete s._active; delete s._punchcard; delete s._day; } + const codexRateLimits = buildCodexRateLimits(sessions); + + const agg = { + generatedAt: new Date(now).toISOString(), + windowDays: days, + pricesAsOf: deps.pricesAsOf ?? null, + totals, byDay, byModel, byHost, byProvider, + byProject, byCategory, + punchcard, projectTree, sessions, codexRateLimits, insights: [], + }; + agg.insights = deps.detectInsights(agg) ?? []; + return agg; +} + +/** + * Overlay Codex's SQLite thread ledger onto parsed session records. Pure — + * returns copies where anything changes, never mutates a (possibly cached) + * record. Two corrections, both attribution-only: + * - a session whose rollout carried no `thread_source` is backfilled from + * the ledger's `threads.thread_source` (or marked `subagent` when a + * spawn edge names it as a child); + * - a session the ledger says is a subagent has its token usage STRIPPED, + * mirroring the parse-time exclusion: its rollout replays the parent's + * entire token history, so keeping the tokens double-counts the parent + * (ccusage/ccusage#950). The record itself stays visible/auditable. + * Exported for test. + */ +export function applyCodexLedger(records, ledger) { + if (!ledger || !(ledger.threads instanceof Map)) return records; + return records.map((rec) => { + if (!rec || rec.provider !== 'codex') return rec; + const t = ledger.threads.get(rec.id); + const fromEdges = ledger.parents instanceof Map && ledger.parents.has(rec.id) ? 'subagent' : null; + const source = rec.threadSource ?? t?.threadSource ?? fromEdges; + if (source === rec.threadSource && (source !== 'subagent' || !rec.usage.length)) return rec; + const out = { ...rec, threadSource: source }; + if (source === 'subagent' && out.usage.length) out.usage = []; + return out; + }); +} + +/** The /api/session payload for any parsed record (claude, codex, opencode): + * meta with pricer-backed cost, and secret-masked, truncation-signalled turns. */ +export function sessionPayload(rec, turns, deps) { + const usage = (rec.usage ?? []).reduce((a, row) => ({ + input: a.input + row.input, output: a.output + row.output, + cacheRead: a.cacheRead + row.cacheRead, cacheWrite: a.cacheWrite + row.cacheWrite, + }), { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }); + + return { + meta: { + id: rec.id, host: rec.host ?? rec.provider, + provider: rec.inferenceProvider ?? null, + transcriptProvider: rec.provider, + providerProvenance: rec.providerProvenance ?? 'unknown', + title: rec.title, project: rec.project, + worktree: rec.worktree ?? null, + start: rec.start === null ? null : new Date(rec.start).toISOString(), + end: rec.end === null ? null : new Date(rec.end).toISOString(), + minutes: rec.start === null ? 0 : Math.round(((rec.end - rec.start) / 60_000) * 10) / 10, + prompts: rec.prompts, responses: rec.responses, exceptions: rec.exceptions, + sidechain: rec.sidechain, threadSource: rec.threadSource, + models: rec.models.slice(), tools: { ...rec.tools }, + skill: rec.skill, plugin: rec.plugin, + // Priced here from the same per-model rows aggregate() uses, rather than + // left undefined: the transcript header rendered a hardcoded "$0.00" on a + // panel whose whole subject is cost. `.filter(Boolean)` could not drop it + // because fmtUsd(undefined) is the truthy string "$0.00". + cost: sessionCost(rec, deps), + ...usage, tokens: usage.input + usage.output + usage.cacheRead + usage.cacheWrite, + }, + // ADR-0009 §8: truncation is the other way content is withheld, and it used + // to be silent — `truncated` was set and the renderer ignored it, so an + // abridged turn read as a complete one. Both keys are emitted only when the + // slice actually fired, so the field's *presence* is the signal and a whole + // turn cannot be misread as an abridged one. `originalChars` is measured + // after `maskSecrets`, so it describes loss due to truncation alone — it is + // not a raw-file length, and must not be rendered as one. + turns: (turns ?? []).map((t) => { + const text = maskSecrets(t.text); + const originalChars = text.length; + if (originalChars <= MAX_TURN_CHARS) return { ...t, text }; + return { + ...t, + text: `${text.slice(0, MAX_TURN_CHARS)}\n…[truncated]`, + truncated: true, + originalChars, + }; + }), + }; +} diff --git a/src/lib/usage-index.mjs b/src/lib/usage-index.mjs index ea85043..5810be8 100644 --- a/src/lib/usage-index.mjs +++ b/src/lib/usage-index.mjs @@ -28,9 +28,15 @@ // pricing/usage-classify/usage-insights are loaded LAZILY and are injectable as // `deps`. That seam is what lets the scanner be unit-tested against exact // arithmetic without importing anyone's pricing table or classification policy. +// +// This file owns index I/O (file discovery, the on-disk cache, single-session +// resolution) and the build/scan orchestration. The per-vendor transcript +// parsers live in usage-parsers.mjs; the pure aggregation/session-shape +// arithmetic lives in usage-aggregate.mjs. Both are re-exported below where a +// consumer's existing import path expects them from here. import fs from 'node:fs'; import path from 'node:path'; -import { configDir, claudeDir, codexDir, repoRoot } from './paths.mjs'; +import { configDir, claudeDir, codexDir } from './paths.mjs'; import { readCodexStateResult } from './codex-state.mjs'; import { defaultOpencodeDbPath, listSessionsResult as listOpencodeSessionsResult, @@ -40,7 +46,11 @@ import { addTelemetryDiagnostics, emptyTelemetryDiagnostics, finalizeTelemetryDiagnostics, MAX_TELEMETRY_UNKNOWN_KINDS, recordTelemetryUnit, telemetryCapabilities, } from './usage-telemetry.mjs'; -import { decodeClaudeRecord, decodeCodexRecord } from './telemetry-records.mjs'; +import { parseClaude, parseCodex } from './usage-parsers.mjs'; +import { maskSecrets, applyCodexLedger, aggregate, sessionPayload } from './usage-aggregate.mjs'; + +export { IDLE_GAP_MS, projectLabel } from './usage-parsers.mjs'; +export { MAX_TURN_CHARS, mergeIntervals, maskSecrets, normalizeSessionIdentity, applyCodexLedger } from './usage-aggregate.mjs'; /** Bump to invalidate every cached entry wholesale. * v2: cached records carry `active` sub-intervals for the idle-gap split. @@ -81,17 +91,10 @@ import { decodeClaudeRecord, decodeCodexRecord } from './telemetry-records.mjs'; * prompts/responses, so every Codex record must be re-derived. */ export const SCHEMA_VERSION = 10; -/** Silence longer than this ends a stretch of engagement. A session is split - * into active sub-intervals at gaps ABOVE this bound (exactly this much is not - * a gap), and `engagedSeconds` unions those. Named rather than inline because - * it is a judgement call the numbers depend on, not a magic constant. */ -export const IDLE_GAP_MS = 15 * 60 * 1000; - const DAY_MS = 86_400_000; // One day of slack past dashboard-server.mjs's 365-day clampDays ceiling — // see the carry-forward pruning comment in scan() below. const KEEP_MS = 366 * DAY_MS; -export const MAX_TURN_CHARS = 40_000; /** Largest single transcript readSession will pull into memory. The corpus's * biggest real file is ~18 MB; JSON expansion runs ~5x, so 64 MB caps the * spike near 320 MB instead of unbounded. Above this the session reads as @@ -99,666 +102,6 @@ export const MAX_TURN_CHARS = 40_000; const MAX_SESSION_BYTES = 64 * 1024 * 1024; const VALID_ID = /^[A-Za-z0-9._-]{1,128}$/; -// ── pure helpers ──────────────────────────────────────────────────────────── - -/** Milliseconds from an epoch number, Date, or ISO string; NaN when unusable. */ -function toMs(v) { - if (typeof v === 'number') return Number.isFinite(v) ? v : NaN; - if (v instanceof Date) return v.getTime(); - if (typeof v === 'string') return Date.parse(v); - return NaN; -} - -/** - * Total seconds covered by a set of intervals, counting overlap ONCE. - * Accepts `[start, end]` tuples or `{ start, end }` objects; each bound may be - * an epoch number, a Date, or an ISO string. Degenerate or unparseable - * intervals are dropped rather than throwing. Pure — exported for test. - */ -export function mergeIntervals(intervals) { - if (!Array.isArray(intervals)) return 0; - const spans = []; - for (const iv of intervals) { - if (!iv) continue; - const start = toMs(Array.isArray(iv) ? iv[0] : iv.start); - const end = toMs(Array.isArray(iv) ? iv[1] : iv.end); - if (!Number.isFinite(start) || !Number.isFinite(end) || end <= start) continue; - spans.push([start, end]); - } - if (!spans.length) return 0; - spans.sort((a, b) => a[0] - b[0]); - let total = 0; - let [curStart, curEnd] = spans[0]; - for (let i = 1; i < spans.length; i++) { - const [s, e] = spans[i]; - if (s <= curEnd) { // overlapping OR exactly touching → extend - if (e > curEnd) curEnd = e; - } else { - total += curEnd - curStart; - curStart = s; curEnd = e; - } - } - total += curEnd - curStart; - return Math.round(total / 1000); -} - -// Each replacement keeps the human-readable prefix and drops the payload. The -// replacement text cannot re-match its own pattern (the '…' is outside every -// character class), which is what makes masking idempotent. -/** @type {[RegExp, string][]} */ -// Order matters. PEM blocks are matched WHOLE and first: a later pattern that -// nibbled at the base64 body would leave the armour behind, which reads as -// "masked" on screen while the key material is still visible. -const SECRET_PATTERNS = [ - [/-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/g, - '-----BEGIN PRIVATE KEY----- …redacted -----END PRIVATE KEY-----'], - [/-----BEGIN [A-Z ]*PRIVATE KEY-----/g, '-----BEGIN PRIVATE KEY----- …redacted'], - [/\bsk-[A-Za-z0-9_-]{12,}/g, 'sk-…redacted'], - [/\bgh[pousr]_[A-Za-z0-9]{20,}/g, 'ghp_…redacted'], - [/\bAKIA[0-9A-Z]{12,}/g, 'AKIA…redacted'], - [/\bwhsec_[A-Za-z0-9_-]{16,}/g, 'whsec_…redacted'], - [/\bASIA[0-9A-Z]{12,}/g, 'ASIA…redacted'], // STS temporary credential - [/\bgithub_pat_[A-Za-z0-9_]{20,}/g, 'github_pat_…redacted'], - [/\bglpat-[A-Za-z0-9_-]{16,}/g, 'glpat-…redacted'], - [/\bhf_[A-Za-z0-9]{20,}/g, 'hf_…redacted'], - [/\bpypi-[A-Za-z0-9_-]{16,}/g, 'pypi-…redacted'], - [/\b(?:sk|rk)_live_[A-Za-z0-9]{16,}/g, 'sk_live_…redacted'], - [/\bSG\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}/g, 'SG.…redacted'], - [/\bnpm_[A-Za-z0-9]{30,}/g, 'npm_…redacted'], - [/\bxox[baprse]-[A-Za-z0-9-]{10,}/g, 'xox…redacted'], - [/\bAIza[0-9A-Za-z_-]{30,}/g, 'AIza…redacted'], - // Slack incoming-webhook URLs are bearer credentials in URL form. - [/https:\/\/hooks\.slack\.com\/services\/[A-Za-z0-9/]+/g, - 'https://hooks.slack.com/services/…redacted'], - // JWT — three base64url segments. Masked whole; the payload is the sensitive part. - [/\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}/g, 'eyJ…redacted'], - // Inline credentials in a URI (postgres://user:pass@host, https://u:p@h). The - // USERNAME is deliberately preserved — it is usually needed to identify which - // system leaked, and it is not the secret. - [/\b([a-z][a-z0-9+.-]*):\/\/([^\s:@/]+):[^\s@/]+@/g, '$1://$2:…redacted@'], - [/\b(Basic)\s+[A-Za-z0-9+/]{16,}={0,2}/g, '$1 …redacted'], - // Case-insensitive on the SCHEME only — `BEARER`/`bearer`/`Bearer` all appear - // in the wild. The token body stays case-sensitive, so this cannot widen into - // prose the way an /i on the assignment rule would. - [/\b([Bb]earer|BEARER)\s+[A-Za-z0-9._~+/=-]{16,}/g, '$1 …redacted'], - // Context-carried secrets: SCREAMING_CASE assignments whose NAME says secret. - // Deliberately case-SENSITIVE: with /i this matches prose like - // "tokens used = 10028979467", and these transcripts discuss token counts - // constantly. Uppercase-only tracks the actual env-var convention instead. - [/\b([A-Z][A-Z0-9_]*(?:SECRET|TOKEN|PASSWORD|PASSWD|API_?KEY|PRIVATE_KEY)[A-Z0-9_]*)(\s*[:=]\s*)("?)[^\s"']{8,}\3/g, - '$1$2$3…redacted$3'], - // Quoted JSON/JS object key whose name says secret — "apiKey": "…", 'client_secret': '…'. - // Case-insensitive is safe here: the quote delimiters are a shape prose never - // has, so this cannot widen into "tokens used" the way the assignment rule's - // /i would (that rule stays case-sensitive above for exactly that reason). - [/(["'][A-Za-z0-9_-]*(?:secret|token|password|passwd|api_?key|private_?key)[A-Za-z0-9_-]*["']\s*:\s*)(["'])[^"']{8,}\2/gi, - '$1$2…redacted$2'], - // Line-anchored YAML/TOML/ini assignment — api_key = …, password: … — with no - // quoting required. Anchored to line-start-through-key-through-:/= so it - // cannot match a key phrase floating mid-sentence ("tokens used = 10028979467" - // fails: "used" — not a secret-shaped word — sits directly before "="). - [/^([ \t]*[A-Za-z0-9_-]*(?:secret|token|password|passwd|api_?key|private_?key)[A-Za-z0-9_-]*[ \t]*[:=][ \t]*)(["']?)[^\s"']{8,}\2/gim, - '$1$2…redacted$2'], -]; - -// NOT masked, deliberately: a bare 40-char base64-ish AWS secret with no prefix -// and no assignment context. It is indistinguishable from a hash, a checksum, or -// a base64 blob in ordinary prose, and masking it would corrupt real transcript -// content. It is caught when it appears as AWS_SECRET_ACCESS_KEY=… above. - -/** - * Mask secret-shaped strings in transcript text (ADR-0009 §8). Idempotent, and - * a no-op on ordinary prose. Pure — exported for test. - */ -export function maskSecrets(text) { - if (text === null || text === undefined) return ''; - let out = typeof text === 'string' ? text : String(text); - for (const [re, sub] of SECRET_PATTERNS) out = out.replace(re, sub); - return out; -} - -/** Local calendar day, `YYYY-MM-DD`. Local because "what did I spend today" is - * a question about the user's clock, not UTC's. */ -function localDay(ms) { - const d = new Date(ms); - const p = (n) => String(n).padStart(2, '0'); - return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())}`; -} - -/** `dow-hour` punchcard key, local, with Monday as 0. */ -function punchKey(ms) { - const d = new Date(ms); - return `${(d.getDay() + 6) % 7}-${d.getHours()}`; -} - -function clip(text, max = 100) { - const t = String(text ?? '').replace(/\s+/g, ' ').trim(); - return t.length > max ? `${t.slice(0, max - 1)}…` : t; -} - -/** - * Directory names that mean "the thing below me is a WORKTREE of the repo above - * me", not a project of its own. `path.basename(cwd)` on a worktree yields the - * branch/phase name — so `keel/.autopilot/worktrees/agent-runtime/phase-1` - * reported a project called `phase-1`, and eight such rows sat beside `keel` in - * the tree as if they were peer repositories. Every project total was wrong by - * however much work happened in a worktree. - */ -const WORKTREE_MARKERS = ['.autopilot', '.claude', '.git']; - -/** - * Resolve a session's cwd to `{ project, worktree }`. - * - * The worktree name is KEPT, not discarded: "which repo" and "which branch of - * it" are different questions, and collapsing the second into the first would - * trade one wrong answer for a lossy one. Pure — exported for test. - * - * A session run in a SUB-DIRECTORY of a repo is the same problem the worktree - * markers above solve, reached by a different path shape: `emailibrium/backend` - * reports a project called `backend`, which then sits beside `emailibrium` as - * if it were a peer repository. The markers cannot catch it because there is no - * marker segment to match — only the repository boundary distinguishes them. So - * the caller may supply `repoRoot`, and when it does, the repo becomes the - * project and the sub-path takes the worktree slot ("which branch/part of it"), - * exactly as a real worktree does. - * - * `repoRoot` is a PARAMETER rather than a lookup because resolving it is a - * filesystem walk and this function is pure and called once per session; the - * index resolves it once per distinct cwd and passes it in. - * - * @param {string|null} cwd Absolute path the session ran in. - * @param {string|null} [dirName] Encoded ~/.claude/projects dir, used only as a - * fallback when the transcript carried no cwd. - * @param {string|null} [repoRoot] The repository root containing `cwd`, when known. - * @returns {{ project: string, worktree: string|null }} - */ -export function projectLabel(cwd, dirName, repoRoot) { - if (cwd && typeof cwd === 'string') { - // Split on BOTH separators, not path.sep. On Windows path.sep is '\\', but a - // transcript's recorded cwd may be POSIX-style (WSL, a synced dotfile, a - // fixture) — splitting on the host separator alone yields one segment and - // silently disables worktree detection on that platform. - const segs = cwd.split(/[\\/]+/).filter(Boolean); - - // //worktrees/<...rest> → repo = , worktree = rest - for (let i = 1; i < segs.length - 1; i++) { - if (WORKTREE_MARKERS.includes(segs[i]) && segs[i + 1] === 'worktrees') { - const rest = segs.slice(i + 2).join('/'); - return { project: segs[i - 1], worktree: rest || null }; - } - } - - // Claude Code's per-session scratchpad lives under the OS temp dir, not in - // any repo. It is genuinely not a project, so it gets its own bucket rather - // than a guessed one — the embedded path segment is `/`-encoded and cannot - // be decoded unambiguously (a `-` may be a separator or part of a name). - // Positional indexing is wrong here — the temp root varies (`/tmp/...` vs - // `/private/tmp/...`), so match the marker segment wherever it lands. - // `segs` already holds the trailing component, and unlike path.basename() it - // is separator-agnostic — basename('/a/b') is 'b' on POSIX but the whole - // string on Windows, which would have made every project label wrong there. - const base = segs[segs.length - 1]; - - if (segs.includes('scratchpad') && segs.some((seg) => /^claude-\d+$/.test(seg))) { - return { project: 'scratchpad', worktree: base || null }; - } - - // A sub-directory of a known repo is that repo, not a peer of it. Compared - // on split segments rather than string prefixes so `/a/repo-two` is never - // read as living inside `/a/repo`. - if (repoRoot && typeof repoRoot === 'string') { - const rootSegs = repoRoot.split(/[\\/]+/).filter(Boolean); - const inside = rootSegs.length < segs.length - && rootSegs.every((seg, i) => segs[i] === seg); - if (inside) { - return { project: rootSegs[rootSegs.length - 1], worktree: segs.slice(rootSegs.length).join('/') || null }; - } - } - - if (base && base !== '.') return { project: base, worktree: null }; - } - - if (!dirName) return { project: 'unknown', worktree: null }; - const parts = String(dirName).replace(/^-+/, '').split('-').filter(Boolean); - return { project: parts.length ? parts[parts.length - 1] : 'unknown', worktree: null }; -} - -/** Repo root for a cwd, memoized for the life of one index build. - * - * Indexing walks thousands of sessions but only tens of distinct working - * directories, so the filesystem walk is paid once per directory rather than - * once per session. A cwd outside any repo memoizes `null` — a miss is as - * worth caching as a hit, and `null` then leaves projectLabel on its existing - * basename path. */ -function repoRootMemo(resolve = repoRoot) { - const cache = new Map(); - return (cwd) => { - if (typeof cwd !== 'string' || !cwd) return null; - if (!cache.has(cwd)) { - let root; - try { root = resolve(cwd); } catch { root = null; } - cache.set(cwd, root ?? null); - } - return cache.get(cwd); - }; -} - -/** Module-scoped because the parse functions are called per session from - * several entry points and threading a cache through all of them would add a - * parameter to each for no behavioural gain. Safe to share: a repository root - * does not move while a process runs, and the key space is the machine's - * distinct working directories (tens), not its sessions (thousands). */ -const repoRootOf = repoRootMemo(); - -/** Sum a record's per-model usage rows into one API-equivalent cost. Rows with - * an observed transcript cost (opencode) use it — same preference as aggregate. */ -function sessionCost(rec, deps) { - let cost = 0; - for (const row of rec.usage ?? []) { - cost += row.costObserved != null ? row.costObserved : (deps.costOf({ - model: row.model, provider: rec.provider, - input: row.input, output: row.output, cacheRead: row.cacheRead, cacheWrite: row.cacheWrite, - }) || 0); - } - return round(cost); -} - -/** Write a projectLabel() result onto a record without losing the worktree. */ -function applyProject(rec, res) { - rec.project = res.project; - if (res.worktree) rec.worktree = res.worktree; -} - -/** - * Split the historical transcript-source `provider` field into an explicit host - * and independently evidenced inference provider. The legacy field used - * `claude`/`codex` to mean transcript host; those values therefore cannot prove - * Anthropic/OpenAI provider identity. - */ -export function normalizeSessionIdentity(record = {}) { - const legacyHost = !record.host && ['claude', 'codex'].includes(record.provider) - ? record.provider : null; - const host = record.host ?? legacyHost ?? null; - const provider = legacyHost ? null : (record.provider ?? null); - return { - ...record, - host, - provider, - model: record.model ?? null, - providerProvenance: provider ? (record.providerProvenance ?? 'unknown') : 'unknown', - }; -} - -// ── transcript parsing ────────────────────────────────────────────────────── - -/** Split JSONL into parsed objects, skipping anything that will not parse. */ -function* jsonLines(raw) { - for (const line of raw.split('\n')) { - if (!line || line.charCodeAt(0) !== 123 /* '{' */) continue; - let obj; - try { obj = JSON.parse(line); } catch { continue; } - if (obj && typeof obj === 'object') yield obj; - } -} - -/** A blank per-session record; `usage` rows are (day, model) buckets so byDay - * and byModel can both be derived without re-reading the transcript. Exported - * so other transcript-source parsers (usage-opencode.mjs) build the SAME - * record shape instead of hand-mirroring it. */ -export function blankSession(id, provider) { - return { - id, provider, host: provider, inferenceProvider: null, providerProvenance: 'unknown', - title: '', project: 'unknown', start: null, end: null, - prompts: 0, responses: 0, exceptions: 0, sidechain: false, threadSource: null, models: [], tools: {}, - skill: null, plugin: null, worktree: null, usage: [], punchcard: {}, active: [], stamps: [], - // Codex-only detail (v6): reasoning tokens inside output, and the last - // rate-limit snapshot the rollout carried. Claude sessions keep the zero - // and the null — absent, not unknown. - reasoningOutput: 0, rateLimits: null, - }; -} - -function noteSpan(rec, ms) { - if (!Number.isFinite(ms)) return; - if (rec.start === null || ms < rec.start) rec.start = ms; - if (rec.end === null || ms > rec.end) rec.end = ms; - rec.stamps.push(ms); -} - -/** - * Collapse a session's activity timestamps into the intervals it was actually - * working, splitting wherever the transcript went quiet for longer than - * IDLE_GAP_MS. A run of one timestamp yields a zero-length interval and so - * contributes nothing — an instant has no duration to claim. - */ -function activeIntervals(stamps) { - const ts = stamps.filter(Number.isFinite).sort((a, b) => a - b); - if (!ts.length) return []; - const out = []; - let start = ts[0]; - let prev = ts[0]; - for (let i = 1; i < ts.length; i++) { - if (ts[i] - prev > IDLE_GAP_MS) { out.push([start, prev]); start = ts[i]; } - prev = ts[i]; - } - out.push([start, prev]); - return out; -} - -/** Finish a parsed record: derive active intervals, drop the raw timestamps - * (thousands per session, and never needed again once collapsed). */ -function seal(rec) { - rec.active = activeIntervals(rec.stamps); - delete rec.stamps; - return rec; -} - -/** Add usage to a session's (day, model) bucket, creating it on first touch. - * Returns the row so a caller with a per-source extra field (opencode's - * observed `costObserved`) can set it without a second find(). Exported for - * the same reason as blankSession — one definition of "how a usage row - * accumulates", shared across transcript-source parsers. */ -export function addUsage(rec, day, model, u) { - let row = rec.usage.find((r) => r.day === day && r.model === model); - if (!row) { row = { day, model, input: 0, output: 0, cacheRead: 0, cacheWrite: 0, responses: 0 }; rec.usage.push(row); } - row.input += u.input; row.output += u.output; - row.cacheRead += u.cacheRead; row.cacheWrite += u.cacheWrite; - row.responses += u.responses ?? 0; - return row; -} - -/** - * Harness-output envelopes: user-role entries whose text the HARNESS wrote — - * background-task notifications, command stdout/stderr dumps, local-command - * caveats. They carry neither `isMeta` nor a tool_result block, so text shape - * is the only signal. Measured on the real corpus (envelope at start of user - * text): task-notification 550, bash-stdout 85, local-command-stdout 60, - * local-command-caveat 183; the stderr variants are the symmetric error-path - * siblings. NOT here: bash-input (the person typed that `! cmd`) and the - * command-name/-message/-args triple (the person invoked that slash command). - */ -const HARNESS_OUTPUT_RE = /^\s*<(task-notification|bash-stdout|bash-stderr|local-command-stdout|local-command-stderr|local-command-caveat)>/; - -function entryText(entry) { - const content = entry?.message?.content; - if (typeof content === 'string') return content; - if (!Array.isArray(content)) return ''; - const t = content.find((b) => b?.type === 'text' && typeof b.text === 'string'); - return t ? t.text : ''; -} - -/** Is this `user` entry a human prompt, or harness output being fed back? */ -function isHumanPrompt(entry) { - if (entry.isMeta) return false; - if (HARNESS_OUTPUT_RE.test(entryText(entry))) return false; - const content = entry?.message?.content; - if (typeof content === 'string') return content.trim().length > 0; - if (!Array.isArray(content)) return false; - const hasResult = content.some((b) => b?.type === 'tool_result'); - const hasText = content.some((b) => b?.type === 'text' && String(b.text ?? '').trim()); - return hasText && !hasResult; -} - -/** - * What KIND of `user`-role turn is this, for transcript attribution? The - * Messages API records tool results and harness context injections under - * `role: "user"`, so role alone must never be read as "the human typed this": - * 'tool-result' — carries a tool_result block: output the HARNESS fed back - * to the model after a tool call. - * 'context' — isMeta OR a harness-output envelope (task notifications, - * command stdout/stderr, caveats): harness-injected, not - * typed by the person — and not the model either. - * 'prompt' — the human. Deliberately broader than isHumanPrompt(): - * an image-only paste has no text block (so it is not - * COUNTED as a prompt) but it IS the person acting, and - * labeling it "tool result" would misattribute it. Also - * covers bash-input (`! cmd`) and slash-command records — - * the person initiated those. - */ -function userTurnKind(entry) { - const content = entry?.message?.content; - if (Array.isArray(content) && content.some((b) => b?.type === 'tool_result')) return 'tool-result'; - if (entry.isMeta || HARNESS_OUTPUT_RE.test(entryText(entry))) return 'context'; - return 'prompt'; -} - -/** - * Parse one Claude transcript. Returns `{ session, turns }`; `turns` is only - * populated when `withTurns` (the reader path) — the scan path does not need - * message bodies and holding them would balloon memory over 3,000 files. - */ -function parseClaude(raw, { id, dirName, withTurns = false }) { - const rec = blankSession(id, 'claude'); - const turns = []; - let firstPrompt = ''; - let aiTitle = ''; - - for (const e of jsonLines(raw)) { - const ms = toMs(e.timestamp); - if (e.type === 'ai-title') { if (typeof e.aiTitle === 'string') aiTitle = e.aiTitle; continue; } - if (typeof e.attributionSkill === 'string' && !rec.skill) rec.skill = e.attributionSkill; - if (typeof e.attributionPlugin === 'string' && !rec.plugin) rec.plugin = e.attributionPlugin; - const decoded = decodeClaudeRecord(e); - if (decoded.isSidechain) rec.sidechain = true; - if (rec.project === 'unknown' && typeof e.cwd === 'string') applyProject(rec, projectLabel(e.cwd, dirName, repoRootOf(e.cwd))); - - if (decoded.role === 'user') { - noteSpan(rec, ms); - const human = isHumanPrompt(e); - if (human) { - rec.prompts++; - if (!firstPrompt) firstPrompt = decoded.text; - } - if (withTurns && decoded.text) { - turns.push({ role: 'user', at: new Date(ms).toISOString(), text: decoded.text, prompt: human, kind: userTurnKind(e) }); - } - continue; - } - - if (decoded.role !== 'assistant' || !e.message) continue; - noteSpan(rec, ms); - rec.responses++; - const at = Number.isFinite(ms) ? ms : (rec.start ?? Date.now()); - const pk = punchKey(at); - rec.punchcard[pk] = (rec.punchcard[pk] ?? 0) + 1; - - // A dropped connection, rate limit, or auth failure makes Claude Code - // synthesize a local placeholder turn (model: "", - // isApiErrorMessage: true) with no real completion behind it — usage is - // always zero. It IS real engaged time (counted above), but it is not a - // model attempt: excluded from `models`/cost attribution so it can never - // appear as a $0 "model in play," and counted instead as an EXCEPTION so - // it stays visible rather than silently vanishing. isApiErrorMessage isn't - // reliably set on every build that emits this placeholder, so the literal - // model marker is checked directly too — it's the one part of the shape - // that's never varied in observed transcripts. - if (decoded.isApiError) { - rec.exceptions++; - if (withTurns) { - turns.push({ - role: 'assistant', at: new Date(at).toISOString(), model: 'exception', - text: decoded.text, tools: [], exception: true, - }); - } - continue; - } - - const model = typeof decoded.model === 'string' ? decoded.model : 'unknown'; - if (!rec.models.includes(model)) rec.models.push(model); - - addUsage(rec, localDay(at), model, { ...decoded.usage, responses: 1 }); - - const tools = []; - for (const use of decoded.toolUses) { - if (typeof use.name === 'string') { - tools.push(use.name); - rec.tools[use.name] = (rec.tools[use.name] ?? 0) + 1; - } - } - if (withTurns) { - turns.push({ - role: 'assistant', at: new Date(at).toISOString(), model, - text: decoded.text, tools, - }); - } - } - - rec.title = maskSecrets(aiTitle || clip(firstPrompt)) || '(untitled)'; - if (rec.project === 'unknown') applyProject(rec, projectLabel(null, dirName)); - return { session: seal(rec), turns }; -} - -function codexParseStats() { - return { - legacyEvents: 0, itemCompletedEvents: 0, tokenCountEvents: 0, - prompts: 0, responses: 0, unknownItemTypes: {}, unknownItemTypeOverflow: 0, - }; -} - -function recordCodexUnknownType(stats, type) { - if (Object.hasOwn(stats.unknownItemTypes, type)) { - stats.unknownItemTypes[type]++; - } else if (Object.keys(stats.unknownItemTypes).length < MAX_TELEMETRY_UNKNOWN_KINDS) { - stats.unknownItemTypes[type] = 1; - } else { - stats.unknownItemTypeOverflow++; - } -} - -/** - * Parse one Codex rollout. `total_token_usage` is CUMULATIVE, so the LAST - * token_count event is the session total — summing them would multiply the - * figure by the number of turns. `input_tokens` there INCLUDES - * `cached_input_tokens`, which bill as cache reads, so the two are separated. - * - * A rollout whose `session_meta.thread_source` is `subagent` is a delegated - * thread whose file replays its parent thread's ENTIRE prior token history - * as duplicate events before its own new turns (openai/codex thread_spawn - * behavior — see ccusage/ccusage#950, which measured up to 91x cost - * inflation from exactly this). Its cumulative `total_token_usage` therefore - * double-counts tokens the parent session already billed, so it is excluded - * from cost/token aggregation here; the session record itself is kept - * (`threadSource` is surfaced on it) so it stays visible/auditable. - */ -function parseCodex(raw, { id, withTurns = false }) { - const rec = blankSession(id, 'codex'); - const turns = []; - const stats = codexParseStats(); - let lastUsage = null; - let lastUsageAt = null; - let firstPrompt = ''; - - for (const e of jsonLines(raw)) { - const ms = toMs(e.timestamp); - noteSpan(rec, ms); - const decoded = decodeCodexRecord(e); - - if (decoded.type === 'meta') { - if (typeof decoded.sessionId === 'string' && decoded.sessionId) rec.id = decoded.sessionId; - if (typeof decoded.cwd === 'string') applyProject(rec, projectLabel(decoded.cwd, null, repoRootOf(decoded.cwd))); - if (typeof decoded.threadSource === 'string') rec.threadSource = decoded.threadSource; - if (decoded.provider) { - rec.inferenceProvider = decoded.provider; - rec.providerProvenance = 'observed'; - } - continue; - } - if (decoded.type === 'turnContext') { - if (typeof decoded.model === 'string' && !rec.models.includes(decoded.model)) rec.models.push(decoded.model); - if (decoded.provider) { - rec.inferenceProvider = decoded.provider; - rec.providerProvenance = 'observed'; - } - if (rec.project === 'unknown' && typeof decoded.cwd === 'string') applyProject(rec, projectLabel(decoded.cwd, null, repoRootOf(decoded.cwd))); - continue; - } - if (e.type !== 'event_msg') continue; - - if (decoded.type === 'tokenCount') { - stats.tokenCountEvents++; - const t = decoded.usage.total; - if (t) { lastUsage = t; lastUsageAt = ms; } - // Every token_count also carries a live rate-limit snapshot — keep the - // LAST one, normalized. Field names are a trap upstream: `primary` is - // whichever window the server listed first, NOT reliably the 5-hour one - // (observed live: primary = the 10080-minute weekly). So windows are - // kept as a flat list keyed by window_minutes and never by field name. - const rl = decoded.usage.rateLimits; - if (rl) { - const windows = []; - for (const w of [rl.primary, rl.secondary]) { - if (!w || typeof w !== 'object') continue; - const usedPercent = Number(w.used_percent); - const windowMinutes = Number(w.window_minutes); - if (!Number.isFinite(usedPercent) || !Number.isFinite(windowMinutes)) continue; - windows.push({ - usedPercent, windowMinutes, - resetsAt: Number.isFinite(Number(w.resets_at)) ? Number(w.resets_at) : null, - }); - } - if (windows.length) { - rec.rateLimits = { - at: Number.isFinite(ms) ? ms : null, - limitId: typeof rl.limit_id === 'string' ? rl.limit_id : null, - planType: typeof rl.plan_type === 'string' ? rl.plan_type : null, - windows, - }; - } - } - continue; - } - - if (decoded.generation === 'legacy') stats.legacyEvents++; - else if (decoded.generation === 'item') stats.itemCompletedEvents++; - if (decoded.unknownItemType) recordCodexUnknownType(stats, decoded.unknownItemType); - if (decoded.type !== 'message') continue; - - if (decoded.role === 'user') { - rec.prompts++; - stats.prompts++; - const text = decoded.text; - if (!firstPrompt) firstPrompt = text; - // Codex rollouts record only real prompts as user_message events — tool - // output travels in other event types that are not surfaced as turns — - // so every normalized Codex user turn is kind 'prompt' by construction. - if (withTurns && text) turns.push({ role: 'user', at: new Date(ms).toISOString(), text, prompt: true, kind: 'prompt' }); - continue; - } - rec.responses++; - stats.responses++; - const at = Number.isFinite(ms) ? ms : (rec.start ?? Date.now()); - const pk = punchKey(at); - rec.punchcard[pk] = (rec.punchcard[pk] ?? 0) + 1; - if (withTurns) { - turns.push({ - role: 'assistant', at: new Date(at).toISOString(), - model: rec.models[rec.models.length - 1] ?? 'unknown', - text: decoded.text, tools: [], - }); - } - } - - if (lastUsage && rec.threadSource !== 'subagent') { - const cacheRead = Number(lastUsage.cached_input_tokens) || 0; - const gross = Number(lastUsage.input_tokens) || 0; - const at = Number.isFinite(lastUsageAt) ? lastUsageAt : (rec.end ?? rec.start ?? Date.now()); - addUsage(rec, localDay(at), rec.models[rec.models.length - 1] ?? 'unknown', { - input: Math.max(0, gross - cacheRead), - output: Number(lastUsage.output_tokens) || 0, - cacheRead, - cacheWrite: 0, - responses: rec.responses, - }); - // Reasoning tokens are a SUBSET of output_tokens (they bill as output) — - // recorded as detail, never added into any token sum, or the total would - // double-count exactly the reasoning share. - rec.reasoningOutput = Number(lastUsage.reasoning_output_tokens) || 0; - } - - rec.title = maskSecrets(clip(firstPrompt)) || '(untitled)'; - return { session: seal(rec), turns, parseStats: stats }; -} - // ── file discovery ────────────────────────────────────────────────────────── function readDirSafe(dir) { @@ -1007,236 +350,6 @@ async function loadDeps(override) { return _deps; } -// ── aggregation ───────────────────────────────────────────────────────────── - -// Buckets are keyed by transcript-derived strings (model ids, project names), so -// a session naming itself `__proto__` would hit Object.prototype's setter: the -// bucket never becomes an own property and vanishes from JSON.stringify, silently -// losing that model's spend. Callers build these maps with Object.create(null); -// this guard keeps the invariant local even if one forgets. -function bucket(map, key) { - if (!Object.prototype.hasOwnProperty.call(map, key)) { - map[key] = { - sessions: 0, responses: 0, input: 0, output: 0, cacheRead: 0, cacheWrite: 0, - tokens: 0, cost: 0, - // `minutes` and `confidence` are NOT decoration. Three consumers read them - // and every one of them silently rendered a zero before they existed: - // project rows showed "0m", every category showed "confidence 0.00" (which - // ADR-0009 §5 requires be DISPLAYED — a constant 0.00 is worse than - // omitting it), and high-volume-automation printed "0.0 minutes each" - // while its evidence asserted "short duration" — a duration claim it never - // measured, i.e. fabricated evidence under ADR-0009 §6 rule 1. - minutes: 0, - confidence: 0, // session-weighted mean, finalised by sealBuckets() - }; - } - return map[key]; -} - -function addTo(b, s) { - b.sessions++; b.responses += s.responses; - b.input += s.input; b.output += s.output; - b.cacheRead += s.cacheRead; b.cacheWrite += s.cacheWrite; - b.tokens += s.tokens; b.cost += s.cost; - b.minutes += Number(s.minutes) || 0; - // Accumulated as a SUM here and divided by `sessions` in sealBuckets, so the - // result is a true mean rather than a running average that drifts with order. - b.confidence += Number(s.confidence) || 0; -} - -/** Turn accumulated confidence sums into means. Must run after every addTo. */ -function sealBuckets(...maps) { - for (const map of maps) { - for (const k of Object.keys(map)) { - const b = map[k]; - b.confidence = b.sessions ? round(b.confidence / b.sessions, 3) : 0; - b.minutes = round(b.minutes, 2); - } - } -} - -const round = (n, p = 6) => Math.round(n * 10 ** p) / 10 ** p; - -/** Turn cached per-file records into the Aggregate the UI and detectors read. */ -function aggregate(records, { days, now, cutoff, deps }) { - const sessions = []; - // Null-prototype: these are keyed by transcript-derived strings (day, model id, - // provider, project, category), so `__proto__` as a key must be an ordinary - // bucket, not a prototype write that silently discards the data. - const byDay = Object.create(null); - const byModel = Object.create(null); - - for (const rec of records) { - if (!rec || !rec.responses) continue; // no assistant turn → not a session - if (rec.end === null || rec.end < cutoff) continue; // outside the window - - let input = 0, output = 0, cacheRead = 0, cacheWrite = 0, cost = 0; - let firstDay = null; - const activeDays = new Set(); - for (const row of rec.usage) { - if (firstDay === null || row.day < firstDay) firstDay = row.day; - // `day` prices the row at the rate in effect WHEN THOSE TOKENS WERE - // SPENT, not today's. A published rate change (Sonnet 5's introductory - // period ending 2026-09-01) must not retroactively restate a finished - // window — August's spend was metered at August's rate and has to keep - // reading that way. Rows are already keyed by day, so this costs nothing. - // costObserved (opencode): the transcript's OWN metered figure for the - // row, when present, outranks the pricing table — observed truth beats a - // rate ak must guess (kimi/openrouter/local). null means no observation - // and the table applies, never a fabricated $0. - const rowCost = row.costObserved != null ? row.costObserved : (deps.costOf({ - model: row.model, provider: rec.provider, day: row.day, - input: row.input, output: row.output, cacheRead: row.cacheRead, cacheWrite: row.cacheWrite, - }) || 0); - input += row.input; output += row.output; - cacheRead += row.cacheRead; cacheWrite += row.cacheWrite; - cost += rowCost; - - const rowTokens = row.input + row.output + row.cacheRead + row.cacheWrite; - if (!byDay[row.day]) byDay[row.day] = { tokens: 0, cost: 0, sessions: 0, sessionsActive: 0 }; - byDay[row.day].tokens += rowTokens; - byDay[row.day].cost = round(byDay[row.day].cost + rowCost); - activeDays.add(row.day); - const m = bucket(byModel, row.model); - m.responses += row.responses; m.input += row.input; m.output += row.output; - m.cacheRead += row.cacheRead; m.cacheWrite += row.cacheWrite; - m.tokens += rowTokens; m.cost = round(m.cost + rowCost); - } - for (const day of activeDays) byDay[day].sessionsActive++; - - const verdict = deps.classify({ - title: rec.title, skill: rec.skill, plugin: rec.plugin, - tools: rec.tools, prompts: rec.prompts, responses: rec.responses, - }) ?? {}; - - sessions.push({ - id: rec.id, host: rec.host ?? rec.provider, - provider: rec.inferenceProvider ?? null, - transcriptProvider: rec.provider, - providerProvenance: rec.providerProvenance ?? 'unknown', - title: rec.title, project: rec.project, - worktree: rec.worktree ?? null, - start: new Date(rec.start ?? rec.end).toISOString(), - minutes: Math.round(((rec.end - (rec.start ?? rec.end)) / 60_000) * 10) / 10, - prompts: rec.prompts, responses: rec.responses, exceptions: rec.exceptions, - sidechain: rec.sidechain, threadSource: rec.threadSource, - models: rec.models.slice(), - input, output, cacheRead, cacheWrite, - tokens: input + output + cacheRead + cacheWrite, - cost: round(cost), - tools: { ...rec.tools }, - category: verdict.category ?? 'Unclassified', - confidence: verdict.confidence ?? 0, - basis: verdict.basis ?? 'no signal', - skill: rec.skill, plugin: rec.plugin, - // Codex-only detail (v6); zero / null on Claude sessions and on v5-cached - // records (the schema bump re-derives those). - reasoningOutput: rec.reasoningOutput ?? 0, - rateLimits: rec.rateLimits ?? null, - _span: [rec.start ?? rec.end, rec.end], - // Pre-v2 cache entries have no `active`; fall back to the whole span so a - // stale record degrades to the old figure instead of vanishing. - _active: Array.isArray(rec.active) && rec.active.length ? rec.active : [[rec.start ?? rec.end, rec.end]], - _punchcard: rec.punchcard, - // The day this session's tokens FIRST landed on — always a key of byDay, - // which keeps sum(byDay.sessions) === totals.sessions. Its start day is - // not usable: a session can open at 23:58 and only bill after midnight. - _day: firstDay, - }); - } - - sessions.sort((a, b) => b.cost - a.cost || Date.parse(b.start) - Date.parse(a.start)); - - const totals = { - sessions: sessions.length, responses: 0, exceptions: 0, input: 0, output: 0, - cacheRead: 0, cacheWrite: 0, tokens: 0, cost: 0, - spanMinutes: 0, spanUnionSeconds: 0, engagedSeconds: 0, - }; - const byHost = Object.create(null), byProvider = Object.create(null); - const byProject = Object.create(null); - const byCategory = Object.create(null), punchcard = Object.create(null); - const tree = new Map(); - let spanMs = 0; - - for (const s of sessions) { - totals.responses += s.responses; totals.exceptions += s.exceptions; - totals.input += s.input; totals.output += s.output; - totals.cacheRead += s.cacheRead; totals.cacheWrite += s.cacheWrite; - totals.tokens += s.tokens; totals.cost += s.cost; - spanMs += s._span[1] - s._span[0]; - - addTo(bucket(byHost, s.host ?? 'unknown'), s); - addTo(bucket(byProvider, s.provider ?? 'unknown'), s); - addTo(bucket(byProject, s.project), s); - addTo(bucket(byCategory, s.category), s); - // A session that used two models counts once under EACH — the token and - // cost columns already partition cleanly, session counts cannot. - // byModel previously got a bare sessions++ while the other three ran addTo, - // so it was the same TYPE with different FIELDS filled — the shape looked - // uniform and was not. Accumulate minutes/confidence here too. - for (const model of s.models) { - const b = bucket(byModel, model); - b.sessions++; - b.minutes += Number(s.minutes) || 0; - b.confidence += Number(s.confidence) || 0; - } - if (s._day && byDay[s._day]) byDay[s._day].sessions++; - for (const [k, n] of Object.entries(s._punchcard)) punchcard[k] = (punchcard[k] ?? 0) + n; - - if (!tree.has(s.project)) tree.set(s.project, { project: s.project, sessions: 0, cost: 0, tokens: 0, minutes: 0, cats: new Map(), rows: [] }); - const node = tree.get(s.project); - node.sessions++; node.cost = round(node.cost + s.cost); node.tokens += s.tokens; - node.minutes = Math.round((node.minutes + s.minutes) * 10) / 10; - const cat = node.cats.get(s.category) ?? { category: s.category, sessions: 0, cost: 0 }; - cat.sessions++; cat.cost = round(cat.cost + s.cost); - node.cats.set(s.category, cat); - node.rows.push(s); - } - - totals.cost = round(totals.cost); - // Three tiers, each honest about a different thing: - // engagedSeconds — union of ACTIVE intervals: time actually worked - // spanUnionSeconds — union of whole spans: wall-clock with a session open - // spanMinutes — sum of spans: the double-counting figure, kept as the - // clearly-labelled secondary the ADR asks the UI to show - sealBuckets(byHost, byProvider, byProject, byCategory, byModel); - - totals.spanMinutes = Math.round((spanMs / 60_000) * 10) / 10; - totals.spanUnionSeconds = mergeIntervals(sessions.map((s) => s._span)); - totals.engagedSeconds = mergeIntervals(sessions.flatMap((s) => s._active)); - - const projectTree = [...tree.values()] - .map((n) => ({ - project: n.project, sessions: n.sessions, cost: n.cost, tokens: n.tokens, minutes: n.minutes, - categories: [...n.cats.values()].sort((a, b) => b.cost - a.cost || b.sessions - a.sessions), - rows: n.rows, - })) - .sort((a, b) => b.cost - a.cost || b.tokens - a.tokens); - - for (const s of sessions) { delete s._span; delete s._active; delete s._punchcard; delete s._day; } - - // Local rate-limit history: every Codex rollout embeds live quota snapshots - // in its token_count events, so a utilization time series is reconstructable - // retroactively with ZERO network — one point per session (its last - // snapshot), oldest first. Claude has no local analogue (its quota arrives - // via the statusline push — see quota.mjs), hence codex-prefixed. - const codexRateLimits = sessions - .filter((s) => s.host === 'codex' && s.rateLimits && Number.isFinite(s.rateLimits.at)) - .map((s) => s.rateLimits) - .sort((x, y) => x.at - y.at); - - const agg = { - generatedAt: new Date(now).toISOString(), - windowDays: days, - pricesAsOf: deps.pricesAsOf ?? null, - totals, byDay, byModel, byHost, byProvider, - byProject, byCategory, - punchcard, projectTree, sessions, codexRateLimits, insights: [], - }; - agg.insights = deps.detectInsights(agg) ?? []; - return agg; -} - // ── build ─────────────────────────────────────────────────────────────────── // Single-flight is keyed by the options that change the RESULT, not global. @@ -1516,33 +629,6 @@ async function scan(o = {}) { return result; } -/** - * Overlay Codex's SQLite thread ledger onto parsed session records. Pure — - * returns copies where anything changes, never mutates a (possibly cached) - * record. Two corrections, both attribution-only: - * - a session whose rollout carried no `thread_source` is backfilled from - * the ledger's `threads.thread_source` (or marked `subagent` when a - * spawn edge names it as a child); - * - a session the ledger says is a subagent has its token usage STRIPPED, - * mirroring the parse-time exclusion: its rollout replays the parent's - * entire token history, so keeping the tokens double-counts the parent - * (ccusage/ccusage#950). The record itself stays visible/auditable. - * Exported for test. - */ -export function applyCodexLedger(records, ledger) { - if (!ledger || !(ledger.threads instanceof Map)) return records; - return records.map((rec) => { - if (!rec || rec.provider !== 'codex') return rec; - const t = ledger.threads.get(rec.id); - const fromEdges = ledger.parents instanceof Map && ledger.parents.has(rec.id) ? 'subagent' : null; - const source = rec.threadSource ?? t?.threadSource ?? fromEdges; - if (source === rec.threadSource && (source !== 'subagent' || !rec.usage.length)) return rec; - const out = { ...rec, threadSource: source }; - if (source === 'subagent' && out.usage.length) out.usage = []; - return out; - }); -} - /** * The Aggregate, memoized in-process and refreshed when stale. This is the * dashboard's read path — `maxAgeMs` (default 15 s) bounds how often a poll can @@ -1683,54 +769,3 @@ export async function readSession(id, o = {}) { return sessionPayload(parsed.session, parsed.turns, await loadDeps(o.deps)); } - -/** The /api/session payload for any parsed record (claude, codex, opencode): - * meta with pricer-backed cost, and secret-masked, truncation-signalled turns. */ -function sessionPayload(rec, turns, deps) { - const usage = (rec.usage ?? []).reduce((a, row) => ({ - input: a.input + row.input, output: a.output + row.output, - cacheRead: a.cacheRead + row.cacheRead, cacheWrite: a.cacheWrite + row.cacheWrite, - }), { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }); - - return { - meta: { - id: rec.id, host: rec.host ?? rec.provider, - provider: rec.inferenceProvider ?? null, - transcriptProvider: rec.provider, - providerProvenance: rec.providerProvenance ?? 'unknown', - title: rec.title, project: rec.project, - worktree: rec.worktree ?? null, - start: rec.start === null ? null : new Date(rec.start).toISOString(), - end: rec.end === null ? null : new Date(rec.end).toISOString(), - minutes: rec.start === null ? 0 : Math.round(((rec.end - rec.start) / 60_000) * 10) / 10, - prompts: rec.prompts, responses: rec.responses, exceptions: rec.exceptions, - sidechain: rec.sidechain, threadSource: rec.threadSource, - models: rec.models.slice(), tools: { ...rec.tools }, - skill: rec.skill, plugin: rec.plugin, - // Priced here from the same per-model rows aggregate() uses, rather than - // left undefined: the transcript header rendered a hardcoded "$0.00" on a - // panel whose whole subject is cost. `.filter(Boolean)` could not drop it - // because fmtUsd(undefined) is the truthy string "$0.00". - cost: sessionCost(rec, deps), - ...usage, tokens: usage.input + usage.output + usage.cacheRead + usage.cacheWrite, - }, - // ADR-0009 §8: truncation is the other way content is withheld, and it used - // to be silent — `truncated` was set and the renderer ignored it, so an - // abridged turn read as a complete one. Both keys are emitted only when the - // slice actually fired, so the field's *presence* is the signal and a whole - // turn cannot be misread as an abridged one. `originalChars` is measured - // after `maskSecrets`, so it describes loss due to truncation alone — it is - // not a raw-file length, and must not be rendered as one. - turns: (turns ?? []).map((t) => { - const text = maskSecrets(t.text); - const originalChars = text.length; - if (originalChars <= MAX_TURN_CHARS) return { ...t, text }; - return { - ...t, - text: `${text.slice(0, MAX_TURN_CHARS)}\n…[truncated]`, - truncated: true, - originalChars, - }; - }), - }; -} diff --git a/src/lib/usage-opencode.mjs b/src/lib/usage-opencode.mjs index b3ccf93..b368741 100644 --- a/src/lib/usage-opencode.mjs +++ b/src/lib/usage-opencode.mjs @@ -26,11 +26,10 @@ // double-count rule does not apply (different storage semantics). import { withDb } from './sqlite.mjs'; // Shared record shape/accumulator with parseClaude/parseCodex — see their -// definitions in usage-index.mjs. usage-index.mjs imports FROM this module -// (defaultOpencodeDbPath, parseSession, …), so this is a circular import; it -// is safe because both imports here are hoisted `function` declarations, -// resolved before either module's top-level body runs. -import { addUsage, blankSession } from './usage-index.mjs'; +// definitions in usage-parsers.mjs. usage-index.mjs imports FROM this module +// (defaultOpencodeDbPath, parseSession, …), but that is no longer a cycle: +// this module depends only on usage-parsers.mjs, not on usage-index.mjs. +import { addUsage, blankSession } from './usage-parsers.mjs'; /** The live opencode store. Overridable via roots in tests. */ export function defaultOpencodeDbPath() { @@ -121,6 +120,125 @@ function projectFromDirectory(directory) { return { project: base && base !== '.' ? base : 'unknown', worktree: null }; } +/** Group a session's `part` rows by their owning message id. */ +function buildPartsIndex(partRows) { + const partsByMessage = new Map(); + for (const p of partRows) { + const data = parseJson(p.data); + if (!data) continue; + if (!partsByMessage.has(p.message_id)) partsByMessage.set(p.message_id, []); + partsByMessage.get(p.message_id).push(data); + } + return partsByMessage; +} + +/** Joined text of a message's parts whose `type` is one of `types`. */ +function messagePartsText(partsByMessage, rowId, types) { + return (partsByMessage.get(rowId) ?? []) + .filter((p) => types.includes(p.type) && typeof p.text === 'string') + .map((p) => p.text).join('\n'); +} + +/** Extend a session record's span/stamps with one message's timestamp. */ +function noteStamp(rec, at) { + if (!at) return; + rec.stamps.push(at); + if (rec.start === null || at < rec.start) rec.start = at; + if (rec.end === null || at > rec.end) rec.end = at; +} + +function recordUserMessage(rec, turns, { rowId, at, withTurns, partsByMessage }) { + rec.prompts++; + if (!withTurns) return; + const text = messagePartsText(partsByMessage, rowId, ['text']); + turns.push({ role: 'user', at: new Date(at).toISOString(), text, prompt: true, kind: 'prompt' }); +} + +/** Model/provider/token-usage bookkeeping for one assistant message. Returns + * the model id, for the caller's turn row. */ +function recordAssistantUsage(rec, data, at) { + const model = typeof data.modelID === 'string' && data.modelID ? data.modelID : 'unknown'; + if (!rec.models.includes(model)) rec.models.push(model); + if (typeof data.providerID === 'string' && data.providerID) { + rec.inferenceProvider = data.providerID; + rec.providerProvenance = 'observed'; + } + const t = data.tokens ?? {}; + const cache = t.cache ?? {}; + const day = localDay(at || Date.now()); + const usageRow = addUsage(rec, day, model, { + input: num(t.input), output: num(t.output), + cacheRead: num(cache.read), cacheWrite: num(cache.write), responses: 1, + }); + // opencode's OWN metered cost for this message — observed truth, summed + // per (day, model) row. Rows where NO message carried a cost stay null + // (the field is always present, unlike an absent key, so a consumer + // checking `costObserved != null` never needs to guess whether this row + // was ever priced), so the aggregate falls back to the pricing table + // rather than misreporting a fabricated $0. + usageRow.costObserved ??= null; + if (Number.isFinite(Number(data.cost))) usageRow.costObserved = (usageRow.costObserved ?? 0) + Number(data.cost); + rec.reasoningOutput += num(t.reasoning); + return model; +} + +/** Turn row + tool tally for one assistant message, when withTurns. */ +function recordAssistantTurn(rec, turns, { rowId, at, model, partsByMessage }) { + const parts = partsByMessage.get(rowId) ?? []; + const tools = parts.filter((p) => p.type === 'tool' && typeof p.tool === 'string').map((p) => p.tool); + for (const name of tools) rec.tools[name] = (rec.tools[name] ?? 0) + 1; + const text = messagePartsText(partsByMessage, rowId, ['text', 'reasoning']); + turns.push({ role: 'assistant', at: new Date(at).toISOString(), model, text, tools }); +} + +function recordAssistantMessage(rec, turns, { data, rowId, at, withTurns, partsByMessage }) { + rec.responses++; + if (at) { const pk = punchKey(at); rec.punchcard[pk] = (rec.punchcard[pk] ?? 0) + 1; } + const model = recordAssistantUsage(rec, data, at); + if (withTurns) recordAssistantTurn(rec, turns, { rowId, at, model, partsByMessage }); +} + +/** One `message` row: malformed rows are skipped, never fatal. */ +function processMessageRow(rec, turns, row, { withTurns, partsByMessage }) { + const data = parseJson(row.data); + if (!data || typeof data.role !== 'string') return; + const at = num(data.time?.created) || num(row.time_created); + noteStamp(rec, at); + if (data.role === 'user') { + recordUserMessage(rec, turns, { rowId: row.id, at, withTurns, partsByMessage }); + return; + } + if (data.role !== 'assistant') return; + recordAssistantMessage(rec, turns, { data, rowId: row.id, at, withTurns, partsByMessage }); +} + +/** Tool counts without the full turn payload: one lean query on the scan path. */ +function collectScanToolCounts(db, id, rec) { + const toolRows = db.prepare(` + SELECT p.data AS data FROM part p JOIN message m ON m.id = p.message_id + WHERE m.session_id = ? AND json_extract(p.data, '$.type') = 'tool' + `).all(id); + for (const p of toolRows) { + const data = parseJson(p.data); + const name = typeof data?.tool === 'string' ? data.tool : null; + if (name) rec.tools[name] = (rec.tools[name] ?? 0) + 1; + } +} + +/** The session record's opencode-specific fields, before its messages are walked. */ +function initSessionRecord(srow) { + const { project, worktree } = projectFromDirectory(srow.directory); + // blankSession's default host/provider ('opencode' for both) already + // matches this source; only the opencode-specific fields are overridden. + const rec = blankSession(srow.id, 'opencode'); + rec.title = clip(srow.title) || '(untitled)'; + rec.project = project; + rec.sidechain = !!srow.parent_id; + rec.threadSource = srow.parent_id ? 'subagent' : null; + if (worktree) rec.worktree = worktree; + return rec; +} + /** Parse ONE opencode session into the index's per-session record shape, * built from the SAME blankSession/addUsage parseClaude and parseCodex use. * Returns { session, turns }; null when the session is gone or unreadable. @@ -138,92 +256,13 @@ export function parseSession({ dbFile, id, withTurns = false }) { WHERE m.session_id = ? ORDER BY p.rowid ASC `).all(id) : []; - const partsByMessage = new Map(); - for (const p of partRows) { - const data = parseJson(p.data); - if (!data) continue; - if (!partsByMessage.has(p.message_id)) partsByMessage.set(p.message_id, []); - partsByMessage.get(p.message_id).push(data); - } + const partsByMessage = buildPartsIndex(partRows); - const { project, worktree } = projectFromDirectory(srow.directory); - // blankSession's default host/provider ('opencode' for both) already - // matches this source; only the opencode-specific fields are overridden. - const rec = blankSession(srow.id, 'opencode'); - rec.title = clip(srow.title) || '(untitled)'; - rec.project = project; - rec.sidechain = !!srow.parent_id; - rec.threadSource = srow.parent_id ? 'subagent' : null; - if (worktree) rec.worktree = worktree; + const rec = initSessionRecord(srow); const turns = []; - let lastProviderId = null; - - for (const row of msgRows) { - const data = parseJson(row.data); - if (!data || typeof data.role !== 'string') continue; // malformed row: skipped, never fatal - const at = num(data.time?.created) || num(row.time_created); - if (at) { rec.stamps.push(at); if (rec.start === null || at < rec.start) rec.start = at; if (rec.end === null || at > rec.end) rec.end = at; } - - if (data.role === 'user') { - rec.prompts++; - if (withTurns) { - const text = (partsByMessage.get(row.id) ?? []) - .filter((p) => p.type === 'text' && typeof p.text === 'string') - .map((p) => p.text).join('\n'); - turns.push({ role: 'user', at: new Date(at).toISOString(), text, prompt: true, kind: 'prompt' }); - } - continue; - } - if (data.role !== 'assistant') continue; - - rec.responses++; - if (at) { const pk = punchKey(at); rec.punchcard[pk] = (rec.punchcard[pk] ?? 0) + 1; } - const model = typeof data.modelID === 'string' && data.modelID ? data.modelID : 'unknown'; - if (!rec.models.includes(model)) rec.models.push(model); - if (typeof data.providerID === 'string' && data.providerID) lastProviderId = data.providerID; - - const t = data.tokens ?? {}; - const cache = t.cache ?? {}; - const day = localDay(at || Date.now()); - const usageRow = addUsage(rec, day, model, { - input: num(t.input), output: num(t.output), - cacheRead: num(cache.read), cacheWrite: num(cache.write), responses: 1, - }); - // opencode's OWN metered cost for this message — observed truth, summed - // per (day, model) row. Rows where NO message carried a cost stay null - // (the field is always present, unlike an absent key, so a consumer - // checking `costObserved != null` never needs to guess whether this row - // was ever priced), so the aggregate falls back to the pricing table - // rather than misreporting a fabricated $0. - usageRow.costObserved ??= null; - if (Number.isFinite(Number(data.cost))) usageRow.costObserved = (usageRow.costObserved ?? 0) + Number(data.cost); - rec.reasoningOutput += num(t.reasoning); - - if (withTurns) { - const parts = partsByMessage.get(row.id) ?? []; - const tools = parts.filter((p) => p.type === 'tool' && typeof p.tool === 'string').map((p) => p.tool); - for (const name of tools) rec.tools[name] = (rec.tools[name] ?? 0) + 1; - const text = parts - .filter((p) => (p.type === 'text' || p.type === 'reasoning') && typeof p.text === 'string') - .map((p) => p.text).join('\n'); - turns.push({ role: 'assistant', at: new Date(at).toISOString(), model, text, tools }); - } - } - - // Tool counts without the full turn payload: one lean query on the scan path. - if (!withTurns) { - const toolRows = db.prepare(` - SELECT p.data AS data FROM part p JOIN message m ON m.id = p.message_id - WHERE m.session_id = ? AND json_extract(p.data, '$.type') = 'tool' - `).all(id); - for (const p of toolRows) { - const data = parseJson(p.data); - const name = typeof data?.tool === 'string' ? data.tool : null; - if (name) rec.tools[name] = (rec.tools[name] ?? 0) + 1; - } - } + for (const row of msgRows) processMessageRow(rec, turns, row, { withTurns, partsByMessage }); + if (!withTurns) collectScanToolCounts(db, id, rec); - if (lastProviderId) { rec.inferenceProvider = lastProviderId; rec.providerProvenance = 'observed'; } if (!rec.title) rec.title = '(untitled)'; rec.active = activeIntervals(rec.stamps); delete rec.stamps; diff --git a/src/lib/usage-parsers.mjs b/src/lib/usage-parsers.mjs new file mode 100644 index 0000000..7bd5661 --- /dev/null +++ b/src/lib/usage-parsers.mjs @@ -0,0 +1,594 @@ +// usage-parsers.mjs — the per-vendor transcript parsers for the usage index +// (ADR-0009): `parseClaude`/`parseCodex` turn one raw JSONL transcript into +// the shared per-session record shape (`blankSession`/`addUsage`), consuming +// telemetry-records.mjs's decoded records for wire-format knowledge. Also +// home to the cwd→project resolution (`projectLabel`) those two parsers +// share. Aggregation (turning many parsed records into the Aggregate) is a +// separate concern — see usage-aggregate.mjs, which this module imports +// `toMs`/`maskSecrets` from to keep that dependency one-directional. +// +// A malformed line is skipped, never fatal — one corrupt line must not cost +// a whole file, and no input here may throw. +import { repoRoot } from './paths.mjs'; +import { MAX_TELEMETRY_UNKNOWN_KINDS } from './usage-telemetry.mjs'; +import { decodeClaudeRecord, decodeCodexRecord } from './telemetry-records.mjs'; +import { toMs, maskSecrets } from './usage-aggregate.mjs'; + +/** Silence longer than this ends a stretch of engagement. A session is split + * into active sub-intervals at gaps ABOVE this bound (exactly this much is not + * a gap), and `engagedSeconds` unions those. Named rather than inline because + * it is a judgement call the numbers depend on, not a magic constant. */ +export const IDLE_GAP_MS = 15 * 60 * 1000; + +// ── pure helpers ──────────────────────────────────────────────────────────── + +/** Local calendar day, `YYYY-MM-DD`. Local because "what did I spend today" is + * a question about the user's clock, not UTC's. */ +function localDay(ms) { + const d = new Date(ms); + const p = (n) => String(n).padStart(2, '0'); + return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())}`; +} + +/** `dow-hour` punchcard key, local, with Monday as 0. */ +function punchKey(ms) { + const d = new Date(ms); + return `${(d.getDay() + 6) % 7}-${d.getHours()}`; +} + +function clip(text, max = 100) { + const t = String(text ?? '').replace(/\s+/g, ' ').trim(); + return t.length > max ? `${t.slice(0, max - 1)}…` : t; +} + +/** + * Directory names that mean "the thing below me is a WORKTREE of the repo above + * me", not a project of its own. `path.basename(cwd)` on a worktree yields the + * branch/phase name — so `keel/.autopilot/worktrees/agent-runtime/phase-1` + * reported a project called `phase-1`, and eight such rows sat beside `keel` in + * the tree as if they were peer repositories. Every project total was wrong by + * however much work happened in a worktree. + */ +const WORKTREE_MARKERS = ['.autopilot', '.claude', '.git']; + +/** + * Resolve a session's cwd to `{ project, worktree }`. + * + * The worktree name is KEPT, not discarded: "which repo" and "which branch of + * it" are different questions, and collapsing the second into the first would + * trade one wrong answer for a lossy one. Pure — exported for test. + * + * A session run in a SUB-DIRECTORY of a repo is the same problem the worktree + * markers above solve, reached by a different path shape: `emailibrium/backend` + * reports a project called `backend`, which then sits beside `emailibrium` as + * if it were a peer repository. The markers cannot catch it because there is no + * marker segment to match — only the repository boundary distinguishes them. So + * the caller may supply `repoRoot`, and when it does, the repo becomes the + * project and the sub-path takes the worktree slot ("which branch/part of it"), + * exactly as a real worktree does. + * + * `repoRoot` is a PARAMETER rather than a lookup because resolving it is a + * filesystem walk and this function is pure and called once per session; the + * index resolves it once per distinct cwd and passes it in. + * + * @param {string|null} cwd Absolute path the session ran in. + * @param {string|null} [dirName] Encoded ~/.claude/projects dir, used only as a + * fallback when the transcript carried no cwd. + * @param {string|null} [repoRoot] The repository root containing `cwd`, when known. + * @returns {{ project: string, worktree: string|null }} + */ +export function projectLabel(cwd, dirName, repoRoot) { + if (cwd && typeof cwd === 'string') { + // Split on BOTH separators, not path.sep. On Windows path.sep is '\\', but a + // transcript's recorded cwd may be POSIX-style (WSL, a synced dotfile, a + // fixture) — splitting on the host separator alone yields one segment and + // silently disables worktree detection on that platform. + const segs = cwd.split(/[\\/]+/).filter(Boolean); + + // //worktrees/<...rest> → repo = , worktree = rest + for (let i = 1; i < segs.length - 1; i++) { + if (WORKTREE_MARKERS.includes(segs[i]) && segs[i + 1] === 'worktrees') { + const rest = segs.slice(i + 2).join('/'); + return { project: segs[i - 1], worktree: rest || null }; + } + } + + // Claude Code's per-session scratchpad lives under the OS temp dir, not in + // any repo. It is genuinely not a project, so it gets its own bucket rather + // than a guessed one — the embedded path segment is `/`-encoded and cannot + // be decoded unambiguously (a `-` may be a separator or part of a name). + // Positional indexing is wrong here — the temp root varies (`/tmp/...` vs + // `/private/tmp/...`), so match the marker segment wherever it lands. + // `segs` already holds the trailing component, and unlike path.basename() it + // is separator-agnostic — basename('/a/b') is 'b' on POSIX but the whole + // string on Windows, which would have made every project label wrong there. + const base = segs[segs.length - 1]; + + if (segs.includes('scratchpad') && segs.some((seg) => /^claude-\d+$/.test(seg))) { + return { project: 'scratchpad', worktree: base || null }; + } + + // A sub-directory of a known repo is that repo, not a peer of it. Compared + // on split segments rather than string prefixes so `/a/repo-two` is never + // read as living inside `/a/repo`. + if (repoRoot && typeof repoRoot === 'string') { + const rootSegs = repoRoot.split(/[\\/]+/).filter(Boolean); + const inside = rootSegs.length < segs.length + && rootSegs.every((seg, i) => segs[i] === seg); + if (inside) { + return { project: rootSegs[rootSegs.length - 1], worktree: segs.slice(rootSegs.length).join('/') || null }; + } + } + + if (base && base !== '.') return { project: base, worktree: null }; + } + + if (!dirName) return { project: 'unknown', worktree: null }; + const parts = String(dirName).replace(/^-+/, '').split('-').filter(Boolean); + return { project: parts.length ? parts[parts.length - 1] : 'unknown', worktree: null }; +} + +/** Repo root for a cwd, memoized for the life of one index build. + * + * Indexing walks thousands of sessions but only tens of distinct working + * directories, so the filesystem walk is paid once per directory rather than + * once per session. A cwd outside any repo memoizes `null` — a miss is as + * worth caching as a hit, and `null` then leaves projectLabel on its existing + * basename path. */ +function repoRootMemo(resolve = repoRoot) { + const cache = new Map(); + return (cwd) => { + if (typeof cwd !== 'string' || !cwd) return null; + if (!cache.has(cwd)) { + let root; + try { root = resolve(cwd); } catch { root = null; } + cache.set(cwd, root ?? null); + } + return cache.get(cwd); + }; +} + +/** Module-scoped because the parse functions are called per session from + * several entry points and threading a cache through all of them would add a + * parameter to each for no behavioural gain. Safe to share: a repository root + * does not move while a process runs, and the key space is the machine's + * distinct working directories (tens), not its sessions (thousands). */ +const repoRootOf = repoRootMemo(); + +/** Write a projectLabel() result onto a record without losing the worktree. */ +function applyProject(rec, res) { + rec.project = res.project; + if (res.worktree) rec.worktree = res.worktree; +} + +// ── transcript parsing ────────────────────────────────────────────────────── + +/** Split JSONL into parsed objects, skipping anything that will not parse. */ +function* jsonLines(raw) { + for (const line of raw.split('\n')) { + if (!line || line.charCodeAt(0) !== 123 /* '{' */) continue; + let obj; + try { obj = JSON.parse(line); } catch { continue; } + if (obj && typeof obj === 'object') yield obj; + } +} + +/** A blank per-session record; `usage` rows are (day, model) buckets so byDay + * and byModel can both be derived without re-reading the transcript. Exported + * so other transcript-source parsers (usage-opencode.mjs) build the SAME + * record shape instead of hand-mirroring it. */ +export function blankSession(id, provider) { + return { + id, provider, host: provider, inferenceProvider: null, providerProvenance: 'unknown', + title: '', project: 'unknown', start: null, end: null, + prompts: 0, responses: 0, exceptions: 0, sidechain: false, threadSource: null, models: [], tools: {}, + skill: null, plugin: null, worktree: null, usage: [], punchcard: {}, active: [], stamps: [], + // Codex-only detail (v6): reasoning tokens inside output, and the last + // rate-limit snapshot the rollout carried. Claude sessions keep the zero + // and the null — absent, not unknown. + reasoningOutput: 0, rateLimits: null, + }; +} + +function noteSpan(rec, ms) { + if (!Number.isFinite(ms)) return; + if (rec.start === null || ms < rec.start) rec.start = ms; + if (rec.end === null || ms > rec.end) rec.end = ms; + rec.stamps.push(ms); +} + +/** + * Collapse a session's activity timestamps into the intervals it was actually + * working, splitting wherever the transcript went quiet for longer than + * IDLE_GAP_MS. A run of one timestamp yields a zero-length interval and so + * contributes nothing — an instant has no duration to claim. + */ +function activeIntervals(stamps) { + const ts = stamps.filter(Number.isFinite).sort((a, b) => a - b); + if (!ts.length) return []; + const out = []; + let start = ts[0]; + let prev = ts[0]; + for (let i = 1; i < ts.length; i++) { + if (ts[i] - prev > IDLE_GAP_MS) { out.push([start, prev]); start = ts[i]; } + prev = ts[i]; + } + out.push([start, prev]); + return out; +} + +/** Finish a parsed record: derive active intervals, drop the raw timestamps + * (thousands per session, and never needed again once collapsed). */ +function seal(rec) { + rec.active = activeIntervals(rec.stamps); + delete rec.stamps; + return rec; +} + +/** Add usage to a session's (day, model) bucket, creating it on first touch. + * Returns the row so a caller with a per-source extra field (opencode's + * observed `costObserved`) can set it without a second find(). Exported for + * the same reason as blankSession — one definition of "how a usage row + * accumulates", shared across transcript-source parsers. */ +export function addUsage(rec, day, model, u) { + let row = rec.usage.find((r) => r.day === day && r.model === model); + if (!row) { row = { day, model, input: 0, output: 0, cacheRead: 0, cacheWrite: 0, responses: 0 }; rec.usage.push(row); } + row.input += u.input; row.output += u.output; + row.cacheRead += u.cacheRead; row.cacheWrite += u.cacheWrite; + row.responses += u.responses ?? 0; + return row; +} + +/** + * Harness-output envelopes: user-role entries whose text the HARNESS wrote — + * background-task notifications, command stdout/stderr dumps, local-command + * caveats. They carry neither `isMeta` nor a tool_result block, so text shape + * is the only signal. Measured on the real corpus (envelope at start of user + * text): task-notification 550, bash-stdout 85, local-command-stdout 60, + * local-command-caveat 183; the stderr variants are the symmetric error-path + * siblings. NOT here: bash-input (the person typed that `! cmd`) and the + * command-name/-message/-args triple (the person invoked that slash command). + */ +const HARNESS_OUTPUT_RE = /^\s*<(task-notification|bash-stdout|bash-stderr|local-command-stdout|local-command-stderr|local-command-caveat)>/; + +function entryText(entry) { + const content = entry?.message?.content; + if (typeof content === 'string') return content; + if (!Array.isArray(content)) return ''; + const t = content.find((b) => b?.type === 'text' && typeof b.text === 'string'); + return t ? t.text : ''; +} + +/** Is this `user` entry a human prompt, or harness output being fed back? */ +function isHumanPrompt(entry) { + if (entry.isMeta) return false; + if (HARNESS_OUTPUT_RE.test(entryText(entry))) return false; + const content = entry?.message?.content; + if (typeof content === 'string') return content.trim().length > 0; + if (!Array.isArray(content)) return false; + const hasResult = content.some((b) => b?.type === 'tool_result'); + const hasText = content.some((b) => b?.type === 'text' && String(b.text ?? '').trim()); + return hasText && !hasResult; +} + +/** + * What KIND of `user`-role turn is this, for transcript attribution? The + * Messages API records tool results and harness context injections under + * `role: "user"`, so role alone must never be read as "the human typed this": + * 'tool-result' — carries a tool_result block: output the HARNESS fed back + * to the model after a tool call. + * 'context' — isMeta OR a harness-output envelope (task notifications, + * command stdout/stderr, caveats): harness-injected, not + * typed by the person — and not the model either. + * 'prompt' — the human. Deliberately broader than isHumanPrompt(): + * an image-only paste has no text block (so it is not + * COUNTED as a prompt) but it IS the person acting, and + * labeling it "tool result" would misattribute it. Also + * covers bash-input (`! cmd`) and slash-command records — + * the person initiated those. + */ +function userTurnKind(entry) { + const content = entry?.message?.content; + if (Array.isArray(content) && content.some((b) => b?.type === 'tool_result')) return 'tool-result'; + if (entry.isMeta || HARNESS_OUTPUT_RE.test(entryText(entry))) return 'context'; + return 'prompt'; +} + +/** A `user`-role Claude entry: span/prompt-count bookkeeping, plus its turn + * row when withTurns. `titleState.firstPrompt` is set from the first HUMAN + * prompt whose text is non-empty (mutated in place: later entries only fill + * it in while it is still empty). */ +function recordClaudeUserTurn(rec, turns, titleState, e, ms, decoded, withTurns) { + noteSpan(rec, ms); + const human = isHumanPrompt(e); + if (human) { + rec.prompts++; + if (!titleState.firstPrompt) titleState.firstPrompt = decoded.text; + } + if (withTurns && decoded.text) { + turns.push({ role: 'user', at: new Date(ms).toISOString(), text: decoded.text, prompt: human, kind: userTurnKind(e) }); + } +} + +/** tool_use blocks → their names, both as a turn-row list and tallied onto + * `rec.tools`. */ +function collectClaudeToolNames(rec, toolUses) { + const tools = []; + for (const use of toolUses) { + if (typeof use.name === 'string') { + tools.push(use.name); + rec.tools[use.name] = (rec.tools[use.name] ?? 0) + 1; + } + } + return tools; +} + +/** An `assistant`-role Claude entry (caller has already confirmed `e.message` + * exists): span/response-count bookkeeping, the API-error placeholder path, + * and otherwise model/usage/tool accounting plus its turn row. */ +function recordClaudeAssistantTurn(rec, turns, ms, decoded, withTurns) { + noteSpan(rec, ms); + rec.responses++; + const at = Number.isFinite(ms) ? ms : (rec.start ?? Date.now()); + const pk = punchKey(at); + rec.punchcard[pk] = (rec.punchcard[pk] ?? 0) + 1; + + // A dropped connection, rate limit, or auth failure makes Claude Code + // synthesize a local placeholder turn (model: "", + // isApiErrorMessage: true) with no real completion behind it — usage is + // always zero. It IS real engaged time (counted above), but it is not a + // model attempt: excluded from `models`/cost attribution so it can never + // appear as a $0 "model in play," and counted instead as an EXCEPTION so + // it stays visible rather than silently vanishing. isApiErrorMessage isn't + // reliably set on every build that emits this placeholder, so the literal + // model marker is checked directly too — it's the one part of the shape + // that's never varied in observed transcripts. + if (decoded.isApiError) { + rec.exceptions++; + if (withTurns) { + turns.push({ + role: 'assistant', at: new Date(at).toISOString(), model: 'exception', + text: decoded.text, tools: [], exception: true, + }); + } + return; + } + + const model = typeof decoded.model === 'string' ? decoded.model : 'unknown'; + if (!rec.models.includes(model)) rec.models.push(model); + + addUsage(rec, localDay(at), model, { ...decoded.usage, responses: 1 }); + + const tools = collectClaudeToolNames(rec, decoded.toolUses); + if (withTurns) { + turns.push({ + role: 'assistant', at: new Date(at).toISOString(), model, + text: decoded.text, tools, + }); + } +} + +/** + * Parse one Claude transcript. Returns `{ session, turns }`; `turns` is only + * populated when `withTurns` (the reader path) — the scan path does not need + * message bodies and holding them would balloon memory over 3,000 files. + */ +export function parseClaude(raw, { id, dirName, withTurns = false }) { + const rec = blankSession(id, 'claude'); + const turns = []; + const titleState = { firstPrompt: '', aiTitle: '' }; + + for (const e of jsonLines(raw)) { + const ms = toMs(e.timestamp); + if (e.type === 'ai-title') { if (typeof e.aiTitle === 'string') titleState.aiTitle = e.aiTitle; continue; } + if (typeof e.attributionSkill === 'string' && !rec.skill) rec.skill = e.attributionSkill; + if (typeof e.attributionPlugin === 'string' && !rec.plugin) rec.plugin = e.attributionPlugin; + const decoded = decodeClaudeRecord(e); + if (decoded.isSidechain) rec.sidechain = true; + if (rec.project === 'unknown' && typeof e.cwd === 'string') applyProject(rec, projectLabel(e.cwd, dirName, repoRootOf(e.cwd))); + + if (decoded.role === 'user') { + recordClaudeUserTurn(rec, turns, titleState, e, ms, decoded, withTurns); + continue; + } + + if (decoded.role !== 'assistant' || !e.message) continue; + recordClaudeAssistantTurn(rec, turns, ms, decoded, withTurns); + } + + rec.title = maskSecrets(titleState.aiTitle || clip(titleState.firstPrompt)) || '(untitled)'; + if (rec.project === 'unknown') applyProject(rec, projectLabel(null, dirName)); + return { session: seal(rec), turns }; +} + +function codexParseStats() { + return { + legacyEvents: 0, itemCompletedEvents: 0, tokenCountEvents: 0, + prompts: 0, responses: 0, unknownItemTypes: {}, unknownItemTypeOverflow: 0, + }; +} + +function recordCodexUnknownType(stats, type) { + if (Object.hasOwn(stats.unknownItemTypes, type)) { + stats.unknownItemTypes[type]++; + } else if (Object.keys(stats.unknownItemTypes).length < MAX_TELEMETRY_UNKNOWN_KINDS) { + stats.unknownItemTypes[type] = 1; + } else { + stats.unknownItemTypeOverflow++; + } +} + +function handleCodexMeta(rec, decoded) { + if (typeof decoded.sessionId === 'string' && decoded.sessionId) rec.id = decoded.sessionId; + if (typeof decoded.cwd === 'string') applyProject(rec, projectLabel(decoded.cwd, null, repoRootOf(decoded.cwd))); + if (typeof decoded.threadSource === 'string') rec.threadSource = decoded.threadSource; + if (decoded.provider) { + rec.inferenceProvider = decoded.provider; + rec.providerProvenance = 'observed'; + } +} + +function handleCodexTurnContext(rec, decoded) { + if (typeof decoded.model === 'string' && !rec.models.includes(decoded.model)) rec.models.push(decoded.model); + if (decoded.provider) { + rec.inferenceProvider = decoded.provider; + rec.providerProvenance = 'observed'; + } + if (rec.project === 'unknown' && typeof decoded.cwd === 'string') applyProject(rec, projectLabel(decoded.cwd, null, repoRootOf(decoded.cwd))); +} + +/** Normalize one token_count event's rate-limit windows (primary/secondary), + * dropping any window whose numeric fields don't parse. Field names are a + * trap upstream: `primary` is whichever window the server listed first, NOT + * reliably the 5-hour one (observed live: primary = the 10080-minute + * weekly) — so windows are kept as a flat list keyed by window_minutes and + * never by field name. */ +function codexRateLimitWindows(rl) { + const windows = []; + for (const w of [rl.primary, rl.secondary]) { + if (!w || typeof w !== 'object') continue; + const usedPercent = Number(w.used_percent); + const windowMinutes = Number(w.window_minutes); + if (!Number.isFinite(usedPercent) || !Number.isFinite(windowMinutes)) continue; + windows.push({ + usedPercent, windowMinutes, + resetsAt: Number.isFinite(Number(w.resets_at)) ? Number(w.resets_at) : null, + }); + } + return windows; +} + +function applyCodexRateLimit(rec, rl, ms) { + const windows = codexRateLimitWindows(rl); + if (!windows.length) return; + rec.rateLimits = { + at: Number.isFinite(ms) ? ms : null, + limitId: typeof rl.limit_id === 'string' ? rl.limit_id : null, + planType: typeof rl.plan_type === 'string' ? rl.plan_type : null, + windows, + }; +} + +/** `event_msg` → `token_count`: keep the LAST cumulative snapshot (see + * parseCodex's own doc comment for why last-only), plus its rate limits. */ +function handleCodexTokenCount(rec, stats, usageState, decoded, ms) { + stats.tokenCountEvents++; + const t = decoded.usage.total; + if (t) { usageState.lastUsage = t; usageState.lastUsageAt = ms; } + const rl = decoded.usage.rateLimits; + if (rl) applyCodexRateLimit(rec, rl, ms); +} + +/** `event_msg` → a `user_message`/`item_completed` user turn. Codex rollouts + * record only real prompts as user_message events — tool output travels in + * other event types that are not surfaced as turns — so every normalized + * Codex user turn is kind 'prompt' by construction. */ +function handleCodexUserMessage(rec, turns, stats, titleState, decoded, ms, withTurns) { + rec.prompts++; + stats.prompts++; + const text = decoded.text; + if (!titleState.firstPrompt) titleState.firstPrompt = text; + if (withTurns && text) turns.push({ role: 'user', at: new Date(ms).toISOString(), text, prompt: true, kind: 'prompt' }); +} + +function handleCodexAssistantMessage(rec, turns, stats, decoded, ms, withTurns) { + rec.responses++; + stats.responses++; + const at = Number.isFinite(ms) ? ms : (rec.start ?? Date.now()); + const pk = punchKey(at); + rec.punchcard[pk] = (rec.punchcard[pk] ?? 0) + 1; + if (withTurns) { + turns.push({ + role: 'assistant', at: new Date(at).toISOString(), + model: rec.models[rec.models.length - 1] ?? 'unknown', + text: decoded.text, tools: [], + }); + } +} + +/** `event_msg` → a decoded `message` (user or assistant), after generation/ + * unknown-type bookkeeping already ran in handleCodexEventMsg. */ +function handleCodexEventMessage(rec, turns, stats, titleState, decoded, ms, withTurns) { + if (decoded.role === 'user') { + handleCodexUserMessage(rec, turns, stats, titleState, decoded, ms, withTurns); + return; + } + handleCodexAssistantMessage(rec, turns, stats, decoded, ms, withTurns); +} + +/** One `event_msg` record: token_count, a message, or lifecycle/unknown + * (generation/unknown-item-type diagnostics apply to every non-token_count + * shape, so they run before the message/non-message split). */ +function handleCodexEventMsg(rec, turns, stats, titleState, usageState, decoded, ms, withTurns) { + if (decoded.type === 'tokenCount') { handleCodexTokenCount(rec, stats, usageState, decoded, ms); return; } + if (decoded.generation === 'legacy') stats.legacyEvents++; + else if (decoded.generation === 'item') stats.itemCompletedEvents++; + if (decoded.unknownItemType) recordCodexUnknownType(stats, decoded.unknownItemType); + if (decoded.type !== 'message') return; + handleCodexEventMessage(rec, turns, stats, titleState, decoded, ms, withTurns); +} + +/** One line of a Codex rollout, dispatched on its decoded type. */ +function processCodexLine(rec, turns, stats, titleState, usageState, e, ms, withTurns) { + const decoded = decodeCodexRecord(e); + if (decoded.type === 'meta') { handleCodexMeta(rec, decoded); return; } + if (decoded.type === 'turnContext') { handleCodexTurnContext(rec, decoded); return; } + if (e.type !== 'event_msg') return; + handleCodexEventMsg(rec, turns, stats, titleState, usageState, decoded, ms, withTurns); +} + +/** The session-total usage row, derived from the LAST token_count event seen + * (see parseCodex's doc comment). A no-op for a subagent thread (its + * cumulative total double-counts the parent's already-billed tokens) or a + * rollout that never carried a token_count at all. */ +function finalizeCodexUsage(rec, usageState) { + const { lastUsage, lastUsageAt } = usageState; + if (!lastUsage || rec.threadSource === 'subagent') return; + const cacheRead = Number(lastUsage.cached_input_tokens) || 0; + const gross = Number(lastUsage.input_tokens) || 0; + const at = Number.isFinite(lastUsageAt) ? lastUsageAt : (rec.end ?? rec.start ?? Date.now()); + addUsage(rec, localDay(at), rec.models[rec.models.length - 1] ?? 'unknown', { + input: Math.max(0, gross - cacheRead), + output: Number(lastUsage.output_tokens) || 0, + cacheRead, + cacheWrite: 0, + responses: rec.responses, + }); + // Reasoning tokens are a SUBSET of output_tokens (they bill as output) — + // recorded as detail, never added into any token sum, or the total would + // double-count exactly the reasoning share. + rec.reasoningOutput = Number(lastUsage.reasoning_output_tokens) || 0; +} + +/** + * Parse one Codex rollout. `total_token_usage` is CUMULATIVE, so the LAST + * token_count event is the session total — summing them would multiply the + * figure by the number of turns. `input_tokens` there INCLUDES + * `cached_input_tokens`, which bill as cache reads, so the two are separated. + * + * A rollout whose `session_meta.thread_source` is `subagent` is a delegated + * thread whose file replays its parent thread's ENTIRE prior token history + * as duplicate events before its own new turns (openai/codex thread_spawn + * behavior — see ccusage/ccusage#950, which measured up to 91x cost + * inflation from exactly this). Its cumulative `total_token_usage` therefore + * double-counts tokens the parent session already billed, so it is excluded + * from cost/token aggregation here; the session record itself is kept + * (`threadSource` is surfaced on it) so it stays visible/auditable. + */ +export function parseCodex(raw, { id, withTurns = false }) { + const rec = blankSession(id, 'codex'); + const turns = []; + const stats = codexParseStats(); + const usageState = { lastUsage: null, lastUsageAt: null }; + const titleState = { firstPrompt: '' }; + + for (const e of jsonLines(raw)) { + const ms = toMs(e.timestamp); + noteSpan(rec, ms); + processCodexLine(rec, turns, stats, titleState, usageState, e, ms, withTurns); + } + + finalizeCodexUsage(rec, usageState); + rec.title = maskSecrets(clip(titleState.firstPrompt)) || '(untitled)'; + return { session: seal(rec), turns, parseStats: stats }; +} diff --git a/src/templates/opencode-ruflo-gateway.js b/src/templates/opencode-ruflo-gateway.js index dbf946e..9e47204 100644 --- a/src/templates/opencode-ruflo-gateway.js +++ b/src/templates/opencode-ruflo-gateway.js @@ -488,6 +488,57 @@ class RufloGatewayClient { } } +function pruneUnavailableTools(plugin, available) { + if (!available.ruflo) { + delete plugin.tool.ak_ruflo_search + delete plugin.tool.ak_ruflo_call + } + if (!available.aqe) { + delete plugin.tool.ak_aqe_search + delete plugin.tool.ak_aqe_call + } + if (!available.agents) { + delete plugin.tool.ak_agent_search + delete plugin.tool.ak_agent_load + } +} + +function projectGatewayTools(cfg, available) { + // Blacklist direct catalogue exposure. The gateway tools below remain + // explicit and small; custom user policy for them is preserved. + return { + ...(cfg.tools || {}), + ...(available.ruflo ? { "claude-flow_*": false, "claude_flow_*": false } : {}), + ...(available.aqe ? { "agentic-qe_*": false, "agentic_qe_*": false } : {}), + } +} + +function projectGatewayPermissions(cfg, available) { + const permission = { + ...(cfg.permission || {}), + ...(available.ruflo ? { + ak_ruflo_search: cfg.permission?.ak_ruflo_search ?? "allow", + ak_ruflo_call: projectGatewayCallPolicy( + cfg, "ak_ruflo_call", ["claude-flow_", "claude_flow_"], + ), + } : {}), + ...(available.aqe ? { + ak_aqe_search: cfg.permission?.ak_aqe_search ?? "allow", + ak_aqe_call: projectGatewayCallPolicy( + cfg, "ak_aqe_call", ["agentic-qe_", "agentic_qe_"], + ), + } : {}), + ak_skill_search: cfg.permission?.ak_skill_search ?? "allow", + ...(available.agents ? { + ak_agent_search: cfg.permission?.ak_agent_search ?? "allow", + ak_agent_load: cfg.permission?.ak_agent_load ?? "allow", + } : {}), + } + if (available.ruflo) hideDirectFamily(permission, ["claude-flow_*", "claude_flow_*"]) + if (available.aqe) hideDirectFamily(permission, ["agentic-qe_*", "agentic_qe_*"]) + return permission +} + function renderToolResult(result, errorPrefix) { const blocks = Array.isArray(result?.content) ? result.content : [] const text = blocks @@ -507,69 +558,32 @@ export default async function rufloGateway({ directory = process.cwd() } = {}) { const aqeClient = new RufloGatewayClient("Agentic QE", directory) const skillCatalogs = new Map() let available = { ruflo: false, aqe: false, brain: false, agents: false } + function computeAvailability(cfg) { + const rufloEntry = managedEntry( + cfg, RUFLO_SERVER_NAME, + ["claude-flow_*", "claude_flow_*"], + ["claude-flow_*", "claude_flow_*"], + ) + const aqeEntry = managedEntry( + cfg, AQE_SERVER_NAME, + ["agentic-qe_*", "agentic_qe_*"], + ["agentic-qe_*", "agentic_qe_*"], + ) + return { + ruflo: rufloClient.configure(rufloEntry), + aqe: aqeClient.configure(aqeEntry), + brain: validLocalMcp(cfg.mcp?.["ruvnet-brain"]), + agents: AK_MANAGED_AGENTS.length > 0 + && typeof cfg.agent?.[SPECIALIST_AGENT_NAME]?.prompt === "string" + && cfg.agent[SPECIALIST_AGENT_NAME].prompt.trim() === AK_SPECIALIST_PROMPT.trim(), + } + } const plugin = { config(cfg) { - const rufloEntry = managedEntry( - cfg, RUFLO_SERVER_NAME, - ["claude-flow_*", "claude_flow_*"], - ["claude-flow_*", "claude_flow_*"], - ) - const aqeEntry = managedEntry( - cfg, AQE_SERVER_NAME, - ["agentic-qe_*", "agentic_qe_*"], - ["agentic-qe_*", "agentic_qe_*"], - ) - available = { - ruflo: rufloClient.configure(rufloEntry), - aqe: aqeClient.configure(aqeEntry), - brain: validLocalMcp(cfg.mcp?.["ruvnet-brain"]), - agents: AK_MANAGED_AGENTS.length > 0 - && typeof cfg.agent?.[SPECIALIST_AGENT_NAME]?.prompt === "string" - && cfg.agent[SPECIALIST_AGENT_NAME].prompt.trim() === AK_SPECIALIST_PROMPT.trim(), - } - if (!available.ruflo) { - delete plugin.tool.ak_ruflo_search - delete plugin.tool.ak_ruflo_call - } - if (!available.aqe) { - delete plugin.tool.ak_aqe_search - delete plugin.tool.ak_aqe_call - } - if (!available.agents) { - delete plugin.tool.ak_agent_search - delete plugin.tool.ak_agent_load - } - - // Blacklist direct catalogue exposure. The gateway tools below remain - // explicit and small; custom user policy for them is preserved. - cfg.tools = { - ...(cfg.tools || {}), - ...(available.ruflo ? { "claude-flow_*": false, "claude_flow_*": false } : {}), - ...(available.aqe ? { "agentic-qe_*": false, "agentic_qe_*": false } : {}), - } - const permission = { - ...(cfg.permission || {}), - ...(available.ruflo ? { - ak_ruflo_search: cfg.permission?.ak_ruflo_search ?? "allow", - ak_ruflo_call: projectGatewayCallPolicy( - cfg, "ak_ruflo_call", ["claude-flow_", "claude_flow_"], - ), - } : {}), - ...(available.aqe ? { - ak_aqe_search: cfg.permission?.ak_aqe_search ?? "allow", - ak_aqe_call: projectGatewayCallPolicy( - cfg, "ak_aqe_call", ["agentic-qe_", "agentic_qe_"], - ), - } : {}), - ak_skill_search: cfg.permission?.ak_skill_search ?? "allow", - ...(available.agents ? { - ak_agent_search: cfg.permission?.ak_agent_search ?? "allow", - ak_agent_load: cfg.permission?.ak_agent_load ?? "allow", - } : {}), - } - if (available.ruflo) hideDirectFamily(permission, ["claude-flow_*", "claude_flow_*"]) - if (available.aqe) hideDirectFamily(permission, ["agentic-qe_*", "agentic_qe_*"]) - cfg.permission = permission + available = computeAvailability(cfg) + pruneUnavailableTools(plugin, available) + cfg.tools = projectGatewayTools(cfg, available) + cfg.permission = projectGatewayPermissions(cfg, available) }, event: async ({ event }) => { if (event?.type === "session.deleted") {