Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions EVENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Emitted once per session, immediately after `session_start` and before the first

> **Use case:** The server-side completion gate (aictrl-dev/aictrl #3216) uses `tools[]` to verify that `record_finding` and `record_review_completed` were actually in the model's function list at dispatch time, and `skills[]` to record which skill version produced the review. Detects the "silent success" failure mode where a missing MCP server lets a run complete green without ever having the review tools available.
>
> **Note on builtin tool filtering:** The `source: "builtin"` entries in `tools[]` reflect the _instance-level superset_ registered in `ToolRegistry`. Per-model filters (e.g. `apply_patch` only on gpt-*, `codesearch`/`websearch` only for aictrl provider) are applied at dispatch time inside `resolveTools` and are **not** reflected here. Consumers should treat the presence of a builtin tool name as "registered and potentially available", not as "guaranteed to appear in the model's function list". The strong structural guarantee (tool present ↔ tool in dispatch list) applies only to `source: "mcp"` entries.
> **Note on builtin tool filtering:** The `source: "builtin"` entries in `tools[]` reflect the _instance-level superset_ registered in `ToolRegistry`. Per-model filters (e.g. `apply_patch` only on gpt-\*, `codesearch`/`websearch` only for aictrl provider) are applied at dispatch time inside `resolveTools` and are **not** reflected here. Consumers should treat the presence of a builtin tool name as "registered and potentially available", not as "guaranteed to appear in the model's function list". The strong structural guarantee (tool present ↔ tool in dispatch list) applies only to `source: "mcp"` entries.

### `session_complete`

Expand Down Expand Up @@ -112,16 +112,21 @@ Emitted immediately before `session_complete` when the session terminates abnorm

### `message_complete`

Emitted when an assistant message finishes (one per LLM turn).
Emitted when a primary-session assistant message finishes (one per LLM turn). Child-session assistant messages are
excluded; use the child-session lifecycle events when tracking subagents.

```json
{
"type": "message_complete",
"messageID": "msg_01abc...",
"modelID": "claude-sonnet-4-20250514",
"providerID": "anthropic",
"agent": "default",
"status": "completed",
"usageStatus": "reported",
"cost": { "input": 0.003, "output": 0.012, "cache": { "read": 0, "write": 0 } },
"tokens": {
"total": 11360,
"input": 1024,
"output": 512,
"reasoning": 0,
Expand All @@ -132,10 +137,14 @@ Emitted when an assistant message finishes (one per LLM turn).
}
```

`finish` values: `"tool-calls"` (model wants to call tools), `"end_turn"` (model is done), `"max_tokens"` (output truncated).
- `messageID` (string, **required**) — stable assistant-message identity. A message produces at most one `message_complete`; consumers should use this field rather than timestamps for identity.
- `status` (string, **required**) — `"completed"`, `"error"`, or `"aborted"`.
- `usageStatus` (string, **required**) — `"reported"` when the provider supplied usage, `"missing"` when it did not, or `"estimated"` for an explicitly estimated future source. The CLI does not currently estimate usage.
- `finish` (string, optional) — provider finish reason, such as `"tool-calls"`, `"end_turn"`, or `"max_tokens"`. It can be absent on failed or aborted turns.

**`tokens`** (5-way breakdown, mirrors upstream `LLM.Usage`):

- `total` (number) — provider-reported total, or a finite total computed from sufficient reported components.
- `input` (number) — raw input tokens billed at the standard input rate.
- `output` (number) — output (completion) tokens.
- `reasoning` (number) — extended-thinking / reasoning tokens (0 when thinking is off).
Expand All @@ -144,12 +153,16 @@ Emitted when an assistant message finishes (one per LLM turn).

These fields are non-overlapping: a token is counted in exactly one bucket.

`tokens` is `null` when `usageStatus` is `"missing"`. This distinguishes unavailable usage from a provider-reported zero, which has `usageStatus: "reported"` and numeric zero fields. The additional identity, status, and provenance fields are additive to schema v1; successful events retain the existing `modelID`, `providerID`, `agent`, `cost`, `tokens`, `context`, and `finish` fields.

For compatibility with sessions written before usage provenance was persisted, a legacy terminal message with a finish reason is treated as `"reported"`; one without a finish reason is treated as `"missing"`.

**`context`** (context-window utilization):

- `used` (number) — tokens occupying the model's context window this turn: `input + cache.read + cache.write`.
- `limit` (number) — the model's total context-window size in tokens, sourced from the models.dev registry (`model.limit.context`).
- `ratio` (number) — `used / limit` (≥0; may exceed 1 if usage exceeds the model's registered limit). A value approaching or exceeding 1 signals context-exhaustion risk.
- `null` — emitted when the model's context limit is not known (e.g. unregistered custom endpoint).
- `null` — emitted when the model's context limit is not known (e.g. unregistered custom endpoint), or usage is missing.

### `text`

Expand Down
88 changes: 64 additions & 24 deletions packages/cli/src/cli/cmd/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@ import { bootstrap } from "../bootstrap"
import { EOL } from "os"
import { Filesystem } from "../../util/filesystem"
import { createAictrlClient } from "@aictrl/sdk"
import type { Message, ToolPart } from "@aictrl/sdk/v2"
import type { ToolPart } from "@aictrl/sdk/v2"
import { Provider } from "../../provider/provider"
import { ModelsDev } from "../../provider/models"
import { Agent } from "../../agent/agent"
import { PermissionNext } from "../../permission/next"
import { Session } from "../../session"
import type { MessageV2 } from "../../session/message-v2"
import { SessionPrompt } from "../../session/prompt"
import { Config } from "../../config/config"
import { GlobalBus } from "../../bus/global"
Expand Down Expand Up @@ -107,6 +109,46 @@ export function buildContextWindow(
}
}

export async function attachedContextLimit(
providerID: string,
modelID: string,
providers = ModelsDev.get(),
): Promise<number | null> {
return providers.then((providers) => providers[providerID]?.models[modelID]?.limit.context ?? null).catch(() => null)
}

type TerminalUsageInfo = Pick<MessageV2.Assistant, "finish" | "usageStatus" | "tokens">

function legacyTotal(info: TerminalUsageInfo) {
if (info.usageStatus !== undefined) return
return (
info.tokens.input + info.tokens.output + info.tokens.reasoning + info.tokens.cache.read + info.tokens.cache.write
)
}

function terminalUsage(info: TerminalUsageInfo) {
const usageStatus = info.usageStatus ?? (info.finish ? "reported" : "missing")
if (usageStatus === "missing") {
return {
usageStatus,
tokens: null,
}
}
return {
usageStatus,
tokens: {
total: info.tokens.total ?? legacyTotal(info),
input: info.tokens.input,
output: info.tokens.output,
reasoning: info.tokens.reasoning,
cache: {
read: info.tokens.cache.read,
write: info.tokens.cache.write,
},
},
}
}

function glob(info: ToolProps<typeof GlobTool>) {
const root = info.input.path ?? ""
const title = `Glob "${info.input.pattern}"`
Expand Down Expand Up @@ -480,6 +522,7 @@ export const RunCommand = cmd({
let sessionErrorEmitted = false
const startTime = Date.now()
const childSessions = new Set<string>()
const emitted = new Set<string>()
const seqBySession = new Map<string, number>()
function nextSeq(sid: string): number {
const n = (seqBySession.get(sid) ?? 0) + 1
Expand All @@ -494,19 +537,9 @@ export const RunCommand = cmd({
if (event.type === "message.updated" && event.properties.info.role === "assistant") {
const info = event.properties.info
if (args.format === "json") {
if (info.finish) {
// Build 5-way token breakdown mirroring upstream LLM.Usage shape.
// info.tokens already carries the full breakdown from StepFinishPart
// accumulation — reasoning and cache split are not dropped upstream.
const tokens = {
input: info.tokens.input,
output: info.tokens.output,
reasoning: info.tokens.reasoning,
cache: {
read: info.tokens.cache.read,
write: info.tokens.cache.write,
},
}
if (info.sessionID === sessionID && info.time.completed !== undefined && !emitted.has(info.id)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Child-session filter silently drops message_complete events.

Suggested change
if (info.sessionID === sessionID && info.time.completed !== undefined && !emitted.has(info.id)) {
Either document the scoping in EVENTS.md ("message_complete is emitted only for the primary session; child-session assistant messages are not reflected"), or broaden the filter to `info.sessionID === sessionID || childSessions.has(info.sessionID)` if child telemetry is desired.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #97, packages/cli/src/cli/cmd/run.ts:499):

Problem: Child-session filter silently drops message_complete events
Detail: The emit gate changed from `if (info.finish)` to `if (info.sessionID === sessionID && info.time.completed !== undefined && !emitted.has(info.id))`. The added `info.sessionID === sessionID` clause filters out assistant messages produced by child sessions (e.g. those spawned by the task tool). Previously, any assistant message with a finish reason — including child-session messages — produced a `message_complete` event. The new test explicitly asserts `msg_child` (sessionID: 'ses_child') is filtered out, so the scoping is intentional, but EVENTS.md does not document it.

Consumers that aggregate usage/cost across the primary and child sessions (e.g. for billing or CI telemetry) will silently stop receiving child-session `message_complete` events after this PR. EVENTS.md should state that `message_complete` is scoped to the primary session only, or the filter should be broadened to tracked `childSessions`.
Suggested fix: Either document the scoping in EVENTS.md ("message_complete is emitted only for the primary session; child-session assistant messages are not reflected"), or broaden the filter to `info.sessionID === sessionID || childSessions.has(info.sessionID)` if child telemetry is desired.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The emit gate changed from if (info.finish) to if (info.sessionID === sessionID && info.time.completed !== undefined && !emitted.has(info.id)). The added info.sessionID === sessionID clause filters out assistant messages produced by child sessions (e.g. those spawned by the task tool). Previously, any assistant message with a finish reason — including child-session messages — produced a message_complete event. The new test explicitly asserts msg_child (sessionID: 'ses_child') is filtered out, so the scoping is intentional, but EVENTS.md does not document it.

Consumers that aggregate usage/cost across the primary and child sessions (e.g. for billing or CI telemetry) will silently stop receiving child-session message_complete events after this PR. EVENTS.md should state that message_complete is scoped to the primary session only, or the filter should be broadened to tracked childSessions.

            if (args.format === "json") {
              if (info.sessionID === sessionID && info.time.completed !== undefined && !emitted.has(info.id)) {
                emitted.add(info.id)
                const usageStatus = info.usageStatus ?? (info.finish ? "reported" : "missing")

emitted.add(info.id)
const usage = terminalUsage(info)

// Context-window utilization: used = input + cache.read + cache.write
// (all prompt tokens that occupy the model's context window this turn).
Expand All @@ -516,25 +549,32 @@ export const RunCommand = cmd({
// undercounts utilization on the first turn of a conversation.
// limit comes from the model registry (models.dev). On lookup failure
// (or limit===0 for custom models) buildContextWindow returns null.
const contextLimit = await Provider.getModel(info.providerID, info.modelID)
.then((m) => m.limit.context)
.catch((e) => {
if (e instanceof Provider.ModelNotFoundError) return null
throw e
})
const contextUsed = tokens.input + tokens.cache.read + tokens.cache.write
const context = buildContextWindow(contextLimit, contextUsed)
const contextLimit = await (args.attach
? attachedContextLimit(info.providerID, info.modelID)
: Provider.getModel(info.providerID, info.modelID)
.then((m) => m.limit.context)
.catch((e) => {
if (e instanceof Provider.ModelNotFoundError) return null
throw e
}))
const contextUsed = usage.tokens
? usage.tokens.input + usage.tokens.cache.read + usage.tokens.cache.write
: null
const context = contextUsed === null ? null : buildContextWindow(contextLimit, contextUsed)

emit("message_complete", {
messageID: info.id,
modelID: info.modelID,
providerID: info.providerID,
agent: info.agent,
context,
// cost is sourced from info.cost which accumulates real per-step costs
// from StepFinishPart. Do NOT use the new step.ended event cost field
// which emits cost:0 and is reconciled later (the cost:0 trap).
cost: info.cost,
tokens,
context,
tokens: usage.tokens,
usageStatus: usage.usageStatus,
status: info.error?.name === "MessageAbortedError" ? "aborted" : info.error ? "error" : "completed",
finish: info.finish,
})
}
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/session/compaction.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ export namespace SessionCompaction {
root: Instance.worktree,
},
cost: 0,
usageStatus: "missing",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

usageStatus field order diverges from Session.getUsage.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #97, packages/cli/src/session/compaction.ts:127):

Problem: usageStatus field order diverges from Session.getUsage
Detail: `Session.getUsage` returns `{ usageStatus, cost, tokens }` (usageStatus first), but the manually-constructed usage object in compaction.ts inserts `usageStatus` between `cost` and `tokens` (`{ cost: 0, usageStatus: "missing", tokens: { ... } }`). Both objects represent the same shape; aligning the field order would keep the two construction sites visually consistent and make future diffs easier to scan. Semantically equivalent.
Suggested fix: Reorder compaction.ts to `{ usageStatus: "missing", cost: 0, tokens: { ... } }` to match the getUsage return shape.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

Session.getUsage returns { usageStatus, cost, tokens } (usageStatus first), but the manually-constructed usage object in compaction.ts inserts usageStatus between cost and tokens ({ cost: 0, usageStatus: "missing", tokens: { ... } }). Both objects represent the same shape; aligning the field order would keep the two construction sites visually consistent and make future diffs easier to scan. Semantically equivalent.

        root: Instance.worktree,
      },
      cost: 0,
      usageStatus: "missing",
      tokens: {
        output: 0,
        input: 0,

tokens: {
output: 0,
input: 0,
Expand Down
34 changes: 23 additions & 11 deletions packages/cli/src/session/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -808,23 +808,32 @@ export namespace Session {
metadata: z.custom<ProviderMetadata>().optional(),
}),
(input) => {
const safe = (value: number) => {
if (!Number.isFinite(value)) return 0
const safe = (value: unknown) => {
if (typeof value !== "number" || !Number.isFinite(value)) return 0
return value
}
const cacheWrite =
input.metadata?.["anthropic"]?.["cacheCreationInputTokens"] ??
// @ts-expect-error
input.metadata?.["bedrock"]?.["usage"]?.["cacheWriteInputTokens"] ??
// @ts-expect-error
input.metadata?.["venice"]?.["usage"]?.["cacheCreationInputTokens"]
const usageStatus = [
input.usage.totalTokens,
input.usage.inputTokens,
input.usage.outputTokens,
input.usage.reasoningTokens,
input.usage.cachedInputTokens,
cacheWrite,
].some(Number.isFinite)
? ("reported" as const)
: ("missing" as const)
const inputTokens = safe(input.usage.inputTokens ?? 0)
const outputTokens = safe(input.usage.outputTokens ?? 0)
const reasoningTokens = safe(input.usage.reasoningTokens ?? 0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsafe cast on untyped provider metadata.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #97, packages/cli/src/session/index.ts:834):

Problem: Unsafe cast on untyped provider metadata
Detail: `cacheWrite` is read from provider-supplied metadata via three consecutive `@ts-expect-error` accesses (untyped external data), then handed to `safe()` with an explicit `as number | undefined` cast. The cast asserts a type the runtime never validated at the access site. `safe()` happens to reject non-numbers (via `typeof value !== "number"`), so this is not exploitable, but the cast is misleading: it papers over the fact that the value originates from untrusted provider metadata. Prefer leaving it `unknown` and letting `safe()` narrow it, or validate explicitly.
Suggested fix: Drop the cast: `const cacheWriteInputTokens = safe(typeof cacheWrite === "number" ? cacheWrite : undefined)`, or have `safe()` accept `unknown`.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

cacheWrite is read from provider-supplied metadata via three consecutive @ts-expect-error accesses (untyped external data), then handed to safe() with an explicit as number | undefined cast. The cast asserts a type the runtime never validated at the access site. safe() happens to reject non-numbers (via typeof value !== "number"), so this is not exploitable, but the cast is misleading: it papers over the fact that the value originates from untrusted provider metadata. Prefer leaving it unknown and letting safe() narrow it, or validate explicitly.

      const cacheReadInputTokens = safe(input.usage.cachedInputTokens ?? 0)
      const cacheWriteInputTokens = safe(cacheWrite as number | undefined)

const cacheReadInputTokens = safe(input.usage.cachedInputTokens ?? 0)
const cacheWriteInputTokens = safe(
(input.metadata?.["anthropic"]?.["cacheCreationInputTokens"] ??
// @ts-expect-error
input.metadata?.["bedrock"]?.["usage"]?.["cacheWriteInputTokens"] ??
// @ts-expect-error
input.metadata?.["venice"]?.["usage"]?.["cacheCreationInputTokens"] ??
0) as number,
)
const cacheWriteInputTokens = safe(cacheWrite)

// Providers where inputTokens EXCLUDES cached tokens
const excludesCachedTokens = [
Expand All @@ -850,9 +859,11 @@ export namespace Session {
input.model.api.npm === "@ai-sdk/amazon-bedrock" ||
input.model.api.npm === "@ai-sdk/google-vertex/anthropic"
) {
if (!Number.isFinite(input.usage.inputTokens) || !Number.isFinite(input.usage.outputTokens)) return
return adjustedInputTokens + outputTokens + cacheReadInputTokens + cacheWriteInputTokens
}
return input.usage.totalTokens
if (Number.isFinite(input.usage.totalTokens)) return input.usage.totalTokens

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Partial-sum total labeled usageStatus "reported".

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #97, packages/cli/src/session/index.ts:865-867):

Problem: Partial-sum total labeled usageStatus "reported"
Detail: When `input.usage.totalTokens` is not finite but `inputTokens` and `outputTokens` are, the `total` falls back to `inputTokens + outputTokens`, silently excluding reasoning/cache.read/cache.write tokens. That sum is a derived partial total, yet the message is still tagged `usageStatus: "reported"` (because usageStatus only requires ANY one of the six token fields to be finite). The schema enum in message-v2.ts declares `"estimated"` and EVENTS.md describes it as "an explicitly estimated future source", but no code path emits it. Downstream consumers (billing, accounting, telemetry) will treat a computed partial sum as a provider-reported total. Either emit `"estimated"` when the totalTokens-absent fallback branch is taken, or omit `total` on that branch.
Suggested fix: When the `return inputTokens + outputTokens` fallback is taken, mark the result as estimated by extending usageStatus logic (e.g. `totalReported ? "reported" : "estimated"`), or omit `total` so consumers see `tokens.total === undefined` for the partial case.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

When input.usage.totalTokens is not finite but inputTokens and outputTokens are, the total falls back to inputTokens + outputTokens, silently excluding reasoning/cache.read/cache.write tokens. That sum is a derived partial total, yet the message is still tagged usageStatus: "reported" (because usageStatus only requires ANY one of the six token fields to be finite). The schema enum in message-v2.ts declares "estimated" and EVENTS.md describes it as "an explicitly estimated future source", but no code path emits it. Downstream consumers (billing, accounting, telemetry) will treat a computed partial sum as a provider-reported total. Either emit "estimated" when the totalTokens-absent fallback branch is taken, or omit total on that branch.

          if (!Number.isFinite(input.usage.inputTokens) || !Number.isFinite(input.usage.outputTokens)) return
          return adjustedInputTokens + outputTokens + cacheReadInputTokens + cacheWriteInputTokens
        }
        if (Number.isFinite(input.usage.totalTokens)) return input.usage.totalTokens
        if (!Number.isFinite(input.usage.inputTokens) || !Number.isFinite(input.usage.outputTokens)) return
        return inputTokens + outputTokens
      })

return
})

const tokens = {
Expand All @@ -871,6 +882,7 @@ export namespace Session {
? input.model.cost.experimentalOver200K
: input.model.cost
return {
usageStatus,
cost: safe(
new Decimal(0)
.add(new Decimal(tokens.input).mul(costInfo?.input ?? 0).div(1_000_000))
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/session/message-v2.ts
Original file line number Diff line number Diff line change
Expand Up @@ -429,6 +429,7 @@ export namespace MessageV2 {
write: z.number(),
}),
}),
usageStatus: z.enum(["reported", "missing", "estimated"]).optional(),
structured: z.any().optional(),
variant: z.string().optional(),
finish: z.string().optional(),
Expand Down
5 changes: 3 additions & 2 deletions packages/cli/src/session/processor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ export namespace SessionProcessor {
input.assistantMessage.finish = value.finishReason
input.assistantMessage.cost += usage.cost
input.assistantMessage.tokens = usage.tokens
input.assistantMessage.usageStatus = usage.usageStatus
await Session.updatePart({
id: Identifier.ascending("part"),
reason: value.finishReason,
Expand Down Expand Up @@ -371,7 +372,6 @@ export namespace SessionProcessor {
sessionID: input.assistantMessage.sessionID,
error: input.assistantMessage.error,
})
SessionStatus.set(input.sessionID, { type: "idle" })
break
}
const delay = SessionRetry.delay(attempt, error.name === "APIError" ? error : undefined)
Expand All @@ -389,7 +389,6 @@ export namespace SessionProcessor {
sessionID: input.assistantMessage.sessionID,
error: input.assistantMessage.error,
})
SessionStatus.set(input.sessionID, { type: "idle" })
}
if (snapshot) {
const patch = await Snapshot.patch(snapshot)
Expand Down Expand Up @@ -423,6 +422,8 @@ export namespace SessionProcessor {
}
}
input.assistantMessage.time.completed = Date.now()
// Persist the terminal message before SessionPrompt's deferred idle
// status; headless consumers stop reading when they observe idle.
await Session.updateMessage(input.assistantMessage)
if (needsCompaction) return "compact"
if (blocked) return "stop"
Expand Down
Loading