Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/DASHBOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,13 @@ control, and the original masked value never reaches the browser. See
[ADR-0009](adr/0009-usage-scorecard-local-transcript-analytics.md) for the full evidence and pricing
contract.

The Scorecard view also shows a host-neutral **telemetry coverage** panel for Claude, Codex
transcript evidence, and OpenCode. It reports parsed units and observed prompt/response totals, plus
capability states (`supported`, `unsupported`, or `unavailable`). A readable source with no observed
activity is a measured zero; an absent, degraded, or old API response is disclosed as unavailable or
not reported rather than rendered as zero. The Codex transcript card does not merge the separate
`codexLedger` corrective source into its coverage counts.

## Observability

Observability separates navigation scope from playback state:
Expand Down
71 changes: 52 additions & 19 deletions docs/TRANSCRIPTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@ rewritten; rule 3 of the module header, `usage-index.mjs:22-29`):

| Host | Store | Discovered by |
|---|---|---|
| Claude Code | `~/.claude/projects/<encoded-project-dir>/<sessionId>.jsonl` | `listClaude` (`usage-index.mjs:876-886`) — exactly one level of project directories |
| Claude Code | `~/.claude/projects/<encoded-project-dir>/<sessionId>.jsonl` | `listClaude` (`usage-index.mjs:916-926`) — exactly one level of project directories |
| Codex CLI | `~/.codex/sessions/<yyyy>/<mm>/<dd>/rollout-<ts>-<uuid>.jsonl` | `listCodex` (`usage-index.mjs:891-904`) — the `yyyy/mm/dd` tree walk |

Roots come from `defaultRoots()` (`usage-index.mjs:750-754`) and are injectable
Roots come from `defaultRoots()` (`usage-index.mjs:908-912`) and are injectable
for tests. A malformed line is skipped, never fatal (`jsonLines`,
`usage-index.mjs:381-387` — one corrupt line must not cost a whole file).

Expand Down Expand Up @@ -90,6 +90,39 @@ human prompts, model responses, or existing tool metrics. Codex tool calls and
tool outputs therefore still travel in event types the parser does not surface
as turns — a fidelity gap, not an attribution bug.

### 1.3 Cross-host telemetry capability contract

The public host APIs are richer than the historical readers in this module, and
they are not interchangeable transcript schemas. This distinction is verified
against the public surfaces available on **2026-08-24**:

| Host | Public evidence | Historical adapter contract in this repository |
|---|---|---|
| Claude Code | Hooks expose `transcript_path`, `tool_name`, tool input/results, and `tool_use_id`; its monitoring surface also documents `claude_code.tool` spans and tool-result events ([hooks reference](https://code.claude.com/docs/en/hooks), [monitoring](https://code.claude.com/docs/en/monitoring-usage)) | `prompts`, `responses`, and normalized `toolCalls` are supported; command/file/MCP/collaboration subcategories remain unclaimed until their cross-host semantics are specified |
| Codex | The public `codex app-server` protocol documents typed `userMessage`, `agentMessage`, `commandExecution`, `fileChange`, `mcpToolCall`, and `collabToolCall` items ([app-server protocol](https://github.com/openai/codex/blob/main/codex-rs/app-server/README.md)) | Only prompt/response items are normalized today. The rollout parser records unknown item kinds diagnostically; Codex activity categories are `unsupported` in this historical adapter, not measured zero |
| OpenCode | The public SDK returns session messages with `parts`, and its public message model includes tool invocation parts ([SDK](https://github.com/anomalyco/opencode/blob/dev/packages/web/src/content/docs/sdk.mdx), [message model](https://github.com/anomalyco/opencode/blob/dev/packages/opencode/src/session/message.ts)) | `prompts`, `responses`, and persisted `toolCalls` are supported; command/file/MCP/collaboration subcategories remain unclaimed |

`sourceHealth.<host>.diagnostics.common` is the additive, host-neutral
coverage envelope. `unitsSeen` counts discovered session candidates in the
requested window; `unitsParsed` counts candidates parsed successfully;
`unitsWithUsage`, `unitsWithPrompts`, and `unitsWithResponses` count parsed
units carrying each kind of evidence; `prompts` and `responses` are observed
totals. `sourceHealth.<host>.capabilities` uses three states:

`unknownKinds` is capped at 32 distinct wire kinds; additional occurrences are
retained in `unknownKindOverflow` so future schema growth cannot expand the
diagnostics payload without limit.

* `supported` means this historical adapter can produce the category;
* `unsupported` means the category is intentionally not claimed by this adapter;
* `unavailable` means the adapter supports the category in principle, but the
source is absent or degraded for this scan.

Therefore a supported source with zero observations is different from an
absent/degraded source, and neither is silently converted into a host-specific
metric. Existing status/reason fields and Codex diagnostic keys remain in place
for compatibility; the common envelope and capability matrix are additive.

---

## 2. Two read paths: scan vs reader
Expand Down Expand Up @@ -154,20 +187,20 @@ story is [Appendix A](#appendix-a--fix-history).)

Two deliberate subtleties:

- **`kind` is broader than `prompt` on the image-only edge.** An image-only
* **`kind` is broader than `prompt` on the image-only edge.** An image-only
paste has no text block, so `isHumanPrompt` returns `false` (it is not
*counted* as a text prompt) — but it **is** the person acting, and
`userTurnKind` returns `'prompt'` for it. "Not countable as a text prompt"
and "not the human" are different claims.
- **Harness-output envelopes are excluded from the prompt *count* too.**
* **Harness-output envelopes are excluded from the prompt *count* too.**
`isHumanPrompt` shares `HARNESS_OUTPUT_RE`, so a session's `prompts` figure
never counts stdout dumps or task notifications as things the person said
(`SCHEMA_VERSION` 5, `usage-index.mjs:47-51`; the correction this shipped
with is in [Appendix A](#appendix-a--fix-history)).
- **`tool-result` outranks `context`**: a `tool_result` block on an `isMeta`
* **`tool-result` outranks `context`**: a `tool_result` block on an `isMeta`
entry is still tool feedback.

Codex user turns are `kind: 'prompt'` by construction (`usage-index.mjs:710`)
Codex user turns are `kind: 'prompt'` by construction (`usage-index.mjs:767-775`)
— rollouts only record real prompts as `user_message` events (§1.2).

Coverage: `tests/kit/usage-index.test.mjs` — "user-role turns carry a kind"
Expand All @@ -185,8 +218,8 @@ transcript content leaves the module, and every step is a gate:

1. **Id grammar before any filesystem access** — `VALID_ID`
(`/^[A-Za-z0-9._-]{1,128}$/`, `usage-index.mjs:95`) rejects traversal
shapes with `ERR_INVALID_SESSION_ID` (`usage-index.mjs:1508-1512`).
2. **Locate by id** across both roots (`locate`, `usage-index.mjs:1518`),
shapes with `ERR_INVALID_SESSION_ID` (`usage-index.mjs:1568-1572`).
2. **Locate by id** across both roots (`locate`, `usage-index.mjs:1578`),
consulting the scan cache when present but never requiring it —
`readSession` works with no prior `buildIndex`.
3. **Realpath containment** (`usage-index.mjs:1587-1601`) — the resolved file
Expand All @@ -213,11 +246,11 @@ Every turn body is passed through `maskSecrets` (`usage-index.mjs:208` — the
23 secret shapes) **server-side, before
serialization**, then length-capped at `MAX_TURN_CHARS` (40,000,
`usage-index.mjs:89`) with the marker appended
(`usage-index.mjs:1661-1670`). Two invariants:
(`usage-index.mjs:1721-1729`). Two invariants:

- **Presence is the signal.** `truncated`/`originalChars` are emitted only
* **Presence is the signal.** `truncated`/`originalChars` are emitted only
when the slice fired, so a complete turn cannot be misread as abridged.
- **`originalChars` is measured after masking** — it describes loss due to
* **`originalChars` is measured after masking** — it describes loss due to
truncation alone, never a raw-file length.

The two kinds of withholding keep distinct vocabulary end-to-end: masking
Expand Down Expand Up @@ -352,25 +385,25 @@ and [#59](https://github.com/pacphi/agentic-kit/issues/59).
The main body describes only current behavior; this appendix records what
was wrong before, for the curious.

- **User-role turns rendered as "you" (fixed 2026-07-26).** Before `kind`
* **User-role turns rendered as "you" (fixed 2026-07-26).** Before `kind`
existed, the Transcript view labelled every user-role turn as the person.
On the reference session that misattributed 276 tool results and 6 harness
context injections — ~93% of its "you" turns (§3.1's measured split). The
turn-`kind` machinery in §3 is the fix.
- **Prompt counts included harness output (SCHEMA_VERSION 5).**
* **Prompt counts included harness output (SCHEMA_VERSION 5).**
`isHumanPrompt` once counted `harness-output` envelopes as human prompts —
32 claimed vs 20 real on the reference session. Cached session records
carried the inflated counts, hence the wholesale `SCHEMA_VERSION` 5 cache
invalidation (`usage-index.mjs:48-51`).
- **Session expander fields shipped but unrendered.** The per-session fields
* **Session expander fields shipped but unrendered.** The per-session fields
§6.1's expander now renders (classification `basis` + confidence, the
token split, flags) once travelled on the wire and rendered nowhere.
- **Transcript header once showed a hardcoded `$0.00`.** `readSession`'s
* **Transcript header once showed a hardcoded `$0.00`.** `readSession`'s
assembled `meta` left `cost` undefined, and `fmtUsd(undefined)` renders the
truthy string `"$0.00"` — a fixed-looking zero on a panel whose whole
subject is cost. `meta.cost` is now priced via `sessionCost()` from the
same per-model usage rows `aggregate()` uses (`usage-index.mjs:1651`).
- **Aggregate-side incidents** (the v4/v5 cache bumps, the Codex parsing
same per-model usage rows `aggregate()` uses (`usage-index.mjs:1691`).
* **Aggregate-side incidents** (the v4/v5 cache bumps, the Codex parsing
defects) are recorded in `USAGE-SCORECARD-METRICS.md` Appendix A.

---
Expand All @@ -386,10 +419,10 @@ kind-attribution behavior is pinned by unit tests at both layers — parser

**Against real data** (this machine's real stores, 2026-07-26):

- A real Claude session (this feature's own working session, 884 turns):
* A real Claude session (this feature's own working session, 884 turns):
`{ prompt: 20, context: 6, 'tool-result': 276, assistant: 588 }`, zero
user turns missing `kind`.
- A real Codex rollout (8 user turns): every one `kind: 'prompt'`, as §1.2
* A real Codex rollout (8 user turns): every one `kind: 'prompt'`, as §1.2
predicts.

---
Expand Down
44 changes: 32 additions & 12 deletions docs/USAGE-SCORECARD-METRICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,26 @@ token-bearing files but zero normalized responses is degraded as
responses is degraded as `parse-yield-partial`, rather than reported as healthy
empty or complete usage.

The additive `sourceHealth.<host>.diagnostics.common` envelope makes coverage
comparable without pretending the hosts have the same wire format: it reports
discovered and parsed units, units with usage/prompts/responses, observed prompt
and response totals, warnings, and unknown kinds. The companion
`sourceHealth.<host>.capabilities` matrix distinguishes `supported`,
`unsupported`, and `unavailable`. This is intentionally the historical adapter
contract, not a claim that the hosts lack richer public APIs: Claude documents
tool hooks and OpenTelemetry tool spans ([hooks](https://code.claude.com/docs/en/hooks),
[monitoring](https://code.claude.com/docs/en/monitoring-usage)); Codex documents
typed command, file-change, MCP, and collaboration items in app-server
([protocol](https://github.com/openai/codex/blob/main/codex-rs/app-server/README.md));
and OpenCode documents session `parts` and tool invocation parts ([SDK](https://github.com/anomalyco/opencode/blob/dev/packages/web/src/content/docs/sdk.mdx),
[message model](https://github.com/anomalyco/opencode/blob/dev/packages/opencode/src/session/message.ts)).
Those richer activity categories remain unclaimed by the scorecard until a
cross-host taxonomy, nested-agent policy, and deduplication rule are accepted.

Unknown wire kinds are bounded to 32 distinct names; `unknownKindOverflow`
retains the number of additional occurrences without allowing transcript data
to expand the diagnostics payload without limit.

The current persisted field named `provider` identifies which host transcript parser produced a
session row; it is not sufficient evidence of the inference provider. The Proposed model in
[ADR-0016](adr/0016-capability-driven-integration-adapters.md) separates host, provider,
Expand Down Expand Up @@ -134,9 +154,9 @@ responses = Σ over included sessions of session.responses
**Source:**

- Filter: a parsed record with zero assistant turns is dropped entirely — "no
assistant turn → not a session" (`usage-index.mjs:1083`) — and a record whose
assistant turn → not a session" (`usage-index.mjs:1123`) — and a record whose
last activity falls outside the requested window is dropped too
(`usage-index.mjs:1084`).
(`usage-index.mjs:1124`).
- `responses` accumulation: Claude increments per assistant message
(`usage-index.mjs:568-571`); Codex increments per `agent_message` event
(`usage-index.mjs:653-662`).
Expand Down Expand Up @@ -219,7 +239,7 @@ already in effect on the given day, comparing ISO date strings
lexicographically so no `Date` parsing is involved and the module stays
clock-free.

`aggregate()` passes each usage row's own `day` (`usage-index.mjs:1100-1103`), which
`aggregate()` passes each usage row's own `day` (`usage-index.mjs:1129-1130`), which
it already has because rows are keyed by `(day, model)`. **This is the whole
point:** tokens metered in August must still read as August's rate when the
panel is opened in December. Pricing by *today's* date instead would restate a
Expand Down Expand Up @@ -409,7 +429,7 @@ session data, and each needs its own fix:

- `mergeIntervals()` (`usage-index.mjs:89-114`) — the pure union primitive,
sorts intervals and merges any two that overlap **or exactly touch**
(`s <= curEnd`, `usage-index.mjs:105`), returning total covered seconds
(`s <= curEnd`, `usage-index.mjs:133`), returning total covered seconds
rounded to the nearest second.
- `activeIntervals()` (`usage-index.mjs:427-438`) — splits one session's
sorted timestamp list into sub-intervals wherever a gap exceeds
Expand Down Expand Up @@ -470,12 +490,12 @@ byDay[day].sessionsActive = count of distinct sessions with any usage row that d

**Source:** the day key is the row's own `row.day`, computed once at parse
time as **local calendar day**, not UTC
(`usage-index.mjs:598`/`usage-index.mjs:784` call `localDay(at)`) — so a
(`usage-index.mjs:602`/`usage-index.mjs:798` call `localDay(at)`) — so a
session that runs from 23:58 local to 00:05 local is billed to the day its
*first* row landed on (test:
`tests/kit/usage-index.test.mjs:634`, "a session that opens before midnight
is counted on its first billed day"). Accumulation:
`byDay[row.day].cost += rowCost` (`usage-index.mjs:1111`). Bar height:
`byDay[row.day].cost += rowCost` (`usage-index.mjs:1151`). Bar height:
`h = maxDay ? max(2, cost/maxDay*100) : 2` (`dashboard/client.mjs`) —
every non-empty day gets a visually nonzero bar (floor of 2%), so a very
cheap day is never rendered as invisible.
Expand Down Expand Up @@ -589,11 +609,11 @@ excluded subagent-replay session still shows up as "used," at zero cost,
rather than vanishing.

`byModel[...].responses` is populated from `row.responses`
(`usage-index.mjs:1114`), which in turn comes from the `responses` field
(`usage-index.mjs:1154`), which in turn comes from the `responses` field
passed into `addUsage()` at the call site — `1` per Claude assistant turn
(`usage-index.mjs:568-604`), or `rec.responses` (the session's whole response
(`usage-index.mjs:596-608`), or `rec.responses` (the session's whole response
count) once per Codex session, passed at the single point Codex calls
`addUsage` (`usage-index.mjs:784-790`).
`addUsage` (`usage-index.mjs:798-804`).

**Render:** `bar(name, fmtUsd(cost), fmtTok(tokens)+" · "+fmtNum(responses)+"
resp", pct(cost, topModelCost), false)` (`dashboard/client.mjs`),
Expand Down Expand Up @@ -925,13 +945,13 @@ Codex ≥0.140 maintains its own SQLite thread ledger (`~/.codex/state_N.sqlite`
— the `N` is a migration generation, so `codexStateDb` (`codex-state.mjs:30`)
globs and takes the newest). `readCodexState` (`:49`) reads per-thread
`thread_source` (`user` vs `subagent`) plus `thread_spawn_edges`, and
`applyCodexLedger` (`usage-index.mjs:1469-1479`) overlays that onto parsed
`applyCodexLedger` (`usage-index.mjs:1529-1539`) overlays that onto parsed
sessions: a ledger-identified subagent has its token usage stripped — its
rollout replays the parent's entire token history (ccusage/ccusage#950
measured up to 91× inflation) — while the session record stays visible. The
rollout's own `session_meta.thread_source` sniff remains as the fallback when
the ledger is absent or migrated beyond recognition. Codex sessions also carry
`reasoningOutput` (`usage-index.mjs:794`) — reasoning tokens are a **subset**
`reasoningOutput` (`usage-index.mjs:808`) — reasoning tokens are a **subset**
of output tokens and are annotation only, never added to any sum.

## 14. Known limitations, restated as a single checklist
Expand Down Expand Up @@ -978,7 +998,7 @@ commit `540be18` on this branch.
Claude's parser passes `responses: 1` per assistant turn
(`usage-index.mjs:598`, the current equivalent), but Codex's call
passed no such field at all. Because `byModel[model].responses` is summed
directly from each usage row's `responses` field (`usage-index.mjs:1114`,
directly from each usage row's `responses` field (`usage-index.mjs:1154`,
`m.responses += row.responses`), **every** Codex model in §10's "Models in
Play" list displayed `0 resp` regardless of real token/cost volume or actual
`agent_message` count. **Fix:** `parseCodex` now passes `responses:
Expand Down
Loading
Loading