Meter, ledger, and gate: enforce AgentBudget at the LLM seams#554
Open
xmap wants to merge 6 commits into
Open
Conversation
The cost meter was lying for two of the three models the fleet can actually spend on: claude-opus-4-8 had no PRICING entry and the steering brain's default claude-sonnet-4-5 had none either, so both metered $0 with only a once-per-process warning; claude-opus-4-7 carried the pre-drop $15/$75 prices (Opus is $5/$25 since the 4.7 generation). Correct metering is a precondition for the budget enforcement seam that follows: a gate fed $0 costs never fires. The span and metric emission still used gen_ai.system, which the OTel GenAI semconv deprecated in favour of gen_ai.provider.name; the durable Decision-side inference record was already on the new name, so CORA was split across the two. The kwarg follows the attribute (system -> provider_name). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-call USD cost was computed at the adapter, emitted to the
cora.agent.llm.cost.usd histogram, and then discarded: the meter
existed, the ledger sink did not. Budget enforcement (unpaused
2026-07-11) needs a durable per-(agent, window) spend fact to sum, and
the inference entry is its natural home, since it already carries
agent_id, occurred_at, and the token counts the cost derives from.
Each LLM producer (RunDebriefer, CautionDrafter, regenerate slice) now
recomputes the cost via compute_cost_usd, the same pure function behind
the histogram, and carries it on AgentInferenceTrace through the
append_inferences slice onto a new nullable cost_usd column. Legacy
rows stay NULL ("not recorded then", distinct from a true $0 call).
Additive everywhere: command, REST body, MCP tool, migration, and the
regenerated OpenAPI snapshot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The budget gate needs one question answered before it permits an LLM call: how much has this agent already spent in the cap's window. This adds the cross-BC port that answers it, in the exact shape of the start_run lookup family: consumer-shaped Protocol + frozen result VO + always-pass stub in cora.infrastructure.ports, production adapter shipped by the BC that owns the data. Decision BC owns the durable spend fact (entries_decision_inferences carries agent_id, occurred_at, token counts, and now cost_usd), so PostgresSpendLookup lives in its new adapters package and SUMs directly; a proj_agent_spend read model stays deferred-with-trigger until the gate's read shows up in p95. The port is window-agnostic (the consumer computes half-open bounds per cap) so a future award-window allocation is a consumer-side change only. NULL costs and missing token counts sum as zero, keeping the coarse tier permissive rather than spuriously blocking. Kernel default is AlwaysZeroSpendLookup, so declared caps cannot block tests that don't exercise gating. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The declared AgentBudget caps were recorded-only: nothing read them at invocation, and, worse, a Suspended agent kept making LLM calls, since the subscribers checked only Access-level Actor deactivation. The operator-pause story (suspend on cost overrun, tighten, resume) was a recorded intention, not a control. Enforcement was unpaused by explicit user decision on 2026-07-11; this is the coarse post-hoc tier of the enforcement ladder for the two serialized subscriber agents. Two per-apply gates in RunDebriefer and CautionDrafter, mirrored: - Suspension gate (pre-lease): a Suspended agent takes no actions, no LLM call, no Decision, until resumed. One Agent fold serves both gates. - Budget gate (post-lease, so only the lease winner pays the lookup): cora.agent._budget_gate sums recorded spend via SpendLookup over the cap's UTC calendar window (month for monthly_usd_cap, day for daily_token_cap), computed from the terminal event's occurred_at so replays gate identically. A breach skips the call and records the refusal as the Run's deferred Decision (DebriefDeferred / NoAction, failure_error_class AgentBudgetExhausted), preserving the one-Decision-per-terminal-Run invariant so operators see WHY. Overspend is bounded to about one in-flight call per caller, the documented coarse-tier bound. A zero cap now refuses every call, making the AgentBudget VO's recorded no-spend intent real. The per-call pre-estimate tier for the steering brain follows separately (it needs a neutral port; Operation BC cannot import the Agent BC gate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five-reviewer panel (3 baseline + security + test) on the four metering slices surfaced MAJORs that all trace to one theme: the gate must be as trustworthy as the caps it enforces. - Cache-write pricing corrected to 2x base: producers pin 1h-TTL cache breakpoints and Anthropic bills 1h writes at 2x, so the old 1.25x tier under-metered exactly the calls CORA makes most. - Lifecycle gate is Versioned-only (was Suspended-only): Defined means "not yet ready for invocation" and Deprecated is terminal per the AgentStatus contract, so both now skip, not just Suspended. The integration seeds gained an explicit bootstrap-then-promote step (promote_seeded_agent) mirroring the production ceremony the cost-overrun scenario already documented. - Operator regenerate now refuses for a SUSPENDED agent (AgentSuspendedError): suspend means stop on every path, not just the subscriber path. Budget stays deliberately ungated there (an accountable human asked; the call is still metered and debited), now pinned by test. - Spend SUM overflow-proofed (numeric cast + bigint clamp) and the append_inferences boundary bounded (token le=1e9, cost finite nonnegative) so one hostile row cannot poison a window. - Migration CHECK rejects negative/NaN/infinite cost_usd at the ledger. - Docstrings now carry the trust boundary (producer-supplied agent_id unbound to principal, follow-up owed), the fail-closed lookup-error posture, and the steering brain's ungated-and-unledgered deferral. Test retrofit: versioned seeds throughout, factory wiring pins for spend_lookup on both subscribers, half-open window boundary rows and agentless-row exclusion on the Postgres lookup, deprecated-skip and under-cap-proceeds parity on both subscribers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Closes the budget-ledger inflation hole the gate review flagged: the append_inferences slice accepted a producer-supplied agent_id that was never checked against the caller, and the AgentBudget gate sums spend rows by agent_id, so any AppendInferences-authorized producer could write rows against a victim agent's id and starve its budget (denial of service by ledger inflation). The handler now rejects any entry whose agent_id differs from the calling principal (InferenceAgentMismatchError, HTTP 403), whole batch refused before any write: no logbook open, no partial append, and no decision-existence oracle since the check precedes the load. Rows with agent_id=None (operator/tool provenance, no agent claim) are untouched, and all three internal recorders already self-report with the agent's own actor id as principal, so nothing legitimate changes shape. Binding beats the persist-principal-and-filter alternative because a sum-self-reported-only ledger would let an agent evade its own cap by routing calls through a helper producer; rejecting mis-attributed rows outright closes inflation without opening that evasion path. A future on-behalf-of recorder (Tier 1.5 steering spend) gets an explicit delegation mechanism, not a relaxation. The DelegatingInferenceRecorder logs the mismatch loudly and distinctly (a mis-wired internal agent would otherwise silently drop spend rows, eroding the gate fail-open). Security review verified no bypassing write path, no UUID case/whitespace normalization asymmetry between the handler equality and the gate's SQL, and no info leak in the 403; residual precondition: legacy-mode X-Principal-Id trust means the binding is as strong as principal authentication, per the documented trust-the-proxy posture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lifts the 2026-06-20 AgentBudget enforcement pause (user-authorized 2026-07-11) by building the three missing pieces of the tier-1 coarse post-hoc ladder, then applying a five-reviewer gate panel's fixes.
Meter (6821181): the PRICING table now covers the whole fleet (opus-4-8/4-7, sonnet, haiku) with cache-write corrected to the 1h-TTL 2x tier CORA's producers actually pin, and telemetry emits
gen_ai.provider.nameper current OTel GenAI semconv (gen_ai.systemis deprecated). A fitness test derives coverage from the fleet's default-model constants so an unpriced default can never silently meter $0 again.Ledger (00e7c69): the per-call
cost_usdthe adapter already computed (and discarded after the histogram) now persists onentries_decision_inferences, guarded by a CHECK rejecting negative/NaN/infinite cost. Additive forward-only migration; legacy rows stay NULL and sum as $0.Lookup (82254ef):
SpendLookupport (consumer-shaped Protocol + frozen result VO + always-zero stub) with a Postgres adapter owned by the Decision BC, summing usd/tokens/calls per agent per half-open window. Overflow-proof SUM (numeric cast + bigint clamp).Gate (7ddaa94):
find_budget_breachchecksmonthly_usd_cap(UTC calendar month) thendaily_token_cap(UTC calendar day) at the RunDebriefer and CautionDrafter seams, windowed on the triggering event'soccurred_atso replays gate identically. A breach writes a deferred Decision (AgentBudgetExhausted) instead of calling the LLM; a lookup error propagates (never fails open). Both subscribers also gained the Versioned-only lifecycle gate theAgentStatuscontract always documented, makingsuspend_agentactually stop the agent; the operator-triggered regenerate refuses for a suspended agent (AgentSuspendedError) but stays deliberately budget-ungated (accountable human, still metered and debited).Review fixes (2bbbf60): the panel's MAJORs, plus the integration seeds now mirror the production bootstrap-then-promote ceremony (
promote_seeded_agent) since Defined agents no longer fire.Known limitations (documented in the port docstring, follow-ups filed)
Producer-suppliedFixed in cd097a8: the handler now rejects any entry whoseagent_idon inference rows is not yet bound to the calling principalagent_iddiffers from the calling principal (InferenceAgentMismatchError, 403), whole batch refused before any write; the delegating recorder logs a mismatch loudly. Residual precondition: legacy-modeX-Principal-Idtrust means the binding is as strong as principal authentication (bearer-auth mode ignores the header).Test plan
🤖 Generated with Claude Code