Measure the context baseline instead of assuming 12,000 tokens

[iamenahs]

Summary

Codex's context indicator divides by a constant that stands in for a quantity Codex can partly measure. This PR measures the part it can and uses it as the denominator where that is likely to help, keeping today's constant everywhere else.

percent_of_context_window_remaining produces the number behind the status-line indicator and the /status card. Its denominator is:

const BASELINE_TOKENS: i64 = 12000;

That value represents the part of a request the conversation cannot influence — base instructions and tool schemas. It varies per session; it is written as a global literal. The indicator is therefore off by whatever the difference happens to be.

This PR estimates part of that quantity when each request is built — the base instructions, the tool payload and the response format — and divides by it whenever it exceeds 12000. It is a heuristic correction, not a complete one, and it is aimed at the case the constant serves worst: a session with a large tool surface. It does not guarantee a fresh session reads 100%, and it is not a proof that every session's reading improves. See Scope for what is measured, what is not, and why the constant is kept as a floor rather than replaced.

The problem

The function normalizes both sides of the fraction:

let effective_window = context_window - BASELINE_TOKENS;
let used = (self.tokens_in_context_window() - BASELINE_TOKENS).max(0);
let remaining = (effective_window - used).max(0);

remaining reduces to context_window - total_tokens, so the baseline only ever moves the denominator — and the sign of the error follows the size of the session's tool surface.

A baseline that is too large. On this repository's default test configuration — suite::client::token_count_includes_rate_limits_snapshot, 8 tools — the measured base/tool/output subtotal is 7,015 tokens: 5,337 of base instructions and 1,678 of tool schemas, read from that test's serialized TokenCountEvent. Where the true fixed cost is near that, subtracting 12,000 shrinks the denominator and the indicator reports more headroom than the window has.

A baseline that is too small. Plugins and connected MCP servers add schemas. Past 12,000 tokens of fixed cost the denominator is too large, the indicator reports less room than exists, and a fresh session never reaches the 100% the function documents:

This normalizes both the numerator and denominator by subtracting the baseline, so immediately after the first prompt the UI shows 100% left

The constant is also duplicated. codex-rs/tui/src/token_usage.rs holds its own 12000 and its own copy of the arithmetic. The two agree today; nothing keeps them agreeing.

The value is display-only, which is what bounds the blast radius. On main it is defined at protocol/src/protocol.rs:2241 and tui/src/token_usage.rs:9, and every read of it sits inside the corresponding percent_of_context_window_remaining. Auto-compaction reserves its headroom separately, in codex-rs/core/src/session/context_window.rs, comparing sess.get_total_token_usage() against model_info.auto_compact_token_limit() plus TokenBudgetConfig::fallback_buffer_tokens. Changing the baseline changes what is displayed and nothing else — a property the rest of this PR is built to preserve. The source comment above the constant claims it also covers "space to call compact"; this PR corrects that line.

What this changes

An estimate of part of the fixed cost. Prompt::context_baseline prices the components the conversation cannot influence that are visible in the Prompt, at the moment the request is built:

pub struct ContextBaseline {
    pub system_prompt_tokens: i64,
    pub tool_schema_tokens: i64,
    pub output_schema_tokens: i64,
    pub tool_count: i64,   // exact
}

• Priced with approx_token_count (APPROX_BYTES_PER_TOKEN = 4), the same byte-density heuristic Codex already applies to history estimation and output truncation. No tokenizer is introduced; the result is a coarse figure of the same kind the constant stood in for.
• Every term is priced in the wire form the request will carry it in. build_responses_request folds every function and freeform spec into a single namespace object when model_info.use_responses_lite && provider.capabilities().namespace_tools, and sends the specs one after another otherwise; ToolsWireFormat::for_request mirrors that branch and renders through the same builders. The response format is likewise priced by calling create_text_param_for_request, so the TextControls wrapper — format type, schema name, strict — is counted rather than the bare schema. Verbosity travels in that same request field but is deliberately excluded: it is not part of the response format, and folding it in made a request carrying no schema at all report a non-zero output_schema_tokens.
• Measured per request rather than per session: enabling a plugin or connecting an MCP server moves the tool surface, and switching model can move the wire form it is sent in.

A denominator that never drops below today's. The estimate can land below the true fixed cost — that is the common case for a session carrying injected context — and dividing by a figure that low understates the percentage by more than the constant does. context_baseline_tokens therefore floors the estimate at LEGACY_CONTEXT_BASELINE_TOKENS. Sessions whose measured components alone exceed 12,000 get a denominator derived from what was actually sent; every other session keeps exactly the behaviour it has now. TokenUsageInfo::percent_of_context_window_remaining is the entry point that supplies it; TokenUsage::percent_of_context_window_remaining keeps the constant for callers holding no TokenUsageInfo. The status line and the /status card now go through the TokenUsageInfo form.

A baseline paired with the usage it was measured against. The baseline is staged when a request is built and published only when that request's own provider usage arrives, so a request that fails cannot pair its baseline with the previous request's usage. A request that stages none publishes none — local compaction sends a Prompt carrying no tools and no output schema (compact.rs) yet still reports provider usage, and denominating that against the previous request's tool payload would subtract a fixed cost it never paid. The same holds after a compaction, when last_token_usage is estimate_token_count_with_base_instructions, which omits the tool schemas. In every withheld case the reading falls back to the constant, which is what main shows today: dividing a total that excludes the tool surface by a baseline that includes it mixes two accountings, and the mismatch moves the reading further from the truth than the constant does.

One copy of the arithmetic. codex-protocol now exports percent_of_context_window_remaining and context_baseline_tokens as free functions. codex-tui keeps its own serde-owned token structs — it persists them in its own format — but no longer carries its own constant or its own formula, and a parity test asserts the two agree on baseline and percentage across eight input combinations.

Carried over the protocol. TokenUsageInfo.context_baseline → ThreadTokenUsage.contextBaseline, nullable. It is null whenever no measurement belongs to the request the event reports usage for — before the first request, for requests built without tools, and while the total is a local estimate — which tells a client to denominate against the constant, not to reuse the previous value. Schemas and TypeScript exports regenerated for both the stable and experimental variants.

The constant renamed to LEGACY_CONTEXT_BASELINE_TOKENS, documented as a fallback, with the compaction claim in its comment corrected.

Scope

Injected fixed context is not measured, and the reading is still short because of it. build_initial_context_with_world_state and build_skills_and_plugins emit developer instructions, plugin and app descriptions, skill fragments and extension contributions as ordinary ResponseItems, recorded through record_conversation_items. They reach the model inside Prompt.input, count toward provider total_tokens, and are indistinguishable there from what the user typed — CodexHarnessMetadata carries no provenance to separate them. So fixed_tokens() is a subtotal, and a tool-and-plugin-heavy fresh session can still read below 100%.

Pricing them properly means giving injected items a provenance at the point they are created, carrying it into Prompt, and deciding whether a skill the user invoked by name counts as fixed at all — a persisted-schema change and a product judgment, not a defect fix. The floor above is what keeps the denominator from dropping below today's in the meantime.

Auto-compaction is unchanged, by construction. estimate_token_count_with_base_instructions feeds recompute_token_usage, which writes last_token_usage.total_tokens — the field get_total_token_usage reads and token_limit_reached compares against its limit. Adding the tool schemas there would move when Codex compacts, so that estimator is left exactly as it is. It still omits the tool schemas, and the PR leaves a NOTE recording that rather than fixing it here. Three things hold the boundary:

• the measured baseline lives in its own field, and token_info() only stamps it onto the returned clone;
• nothing in the compaction path reads it — grep for the constant and for context_baseline in session/context_window.rs both come back empty;
• recording_a_baseline_does_not_move_the_history_estimate asserts the estimate is identical before and after a baseline is recorded.

Provider-reported usage is unchanged. total and last are untouched. The baseline is a denominator, never a substitute for what the provider reports consumed, and the protocol docs say so at each hop.

No /context surface. No new command, and no per-tool or per-server attribution. ContextBaseline is the minimum a denominator needs, and is named for that job.

Testing

Toolchain: Rust 1.95.0, macOS arm64.

New tests:

Test	Asserts
a_fresh_session_reads_full_only_with_a_measured_baseline	31,200-token subtotal on a 272k window: 93% with the constant, 100% measured — the constructed best case, where the subtotal is the whole fixed cost
a_measured_subtotal_below_the_constant_changes_nothing	a 5,000-token subtotal floors to 12,000 and reads exactly as main does
the_floor_never_lowers_the_baseline	across subtotals from 0 to 250,000, the resolved baseline is never below the constant
the_percentage_matches_the_protocol_for_the_same_inputs	codex-tui and codex-protocol agree on baseline and percentage over eight cases
a_staged_baseline_is_withheld_until_usage_arrives	a built-but-uncompleted request does not pair its baseline with the previous request's usage
an_estimated_usage_figure_is_not_paired_with_a_measured_baseline	the post-compaction case: 30k baseline withheld, reading is 64% — what main shows — not the 69% a mixed accounting would give
a_request_that_stages_no_baseline_does_not_inherit_the_previous_one	the compaction request itself: it carries no tools, so it reads 12% against the constant rather than 13% against the previous turn's tool payload
an_absent_baseline_keeps_the_historical_constant	no request built ⇒ the same percentage main produces today
baseline_at_or_beyond_the_window_is_zero_remaining	a tool surface larger than the window clamps to 0, never negative or >100
context_baseline_survives_a_usage_append	new_or_append preserves it
prompt_context_baseline_prices_instructions_tools_and_output_schema	all three terms priced, the response format above its bare-schema size; fixed_tokens() is their sum
prompt_context_baseline_prices_tools_in_the_form_they_are_sent	namespaced and per-tool forms do not price the same
tools_wire_format_follows_the_request_branch	all four (use_responses_lite, namespace_tools) combinations
prompt_context_baseline_is_near_zero_for_an_empty_prompt	a measured near-zero is not an absent measurement
recording_a_baseline_does_not_move_the_history_estimate	the auto-compaction boundary above

Existing suites:

• codex-protocol --lib: 306 passed.
• codex-app-server-protocol --lib: 291 passed, including the schema-fixture and precomputed-export checks.
• codex-core --lib: 2,268 passed. Two failures (guardian::tests::guardian_ephemeral_retry_preserves_parallel_trunk_and_fork_history, session::turn::tests::post_sampling_token_estimate_is_disabled_by_always_on_sinks) reproduce identically on unmodified main. Needs RUST_MIN_STACK=16777216; several agent::control tests overflow the default stack on main too.
• codex-tui: 3,806 passed, 0 failed (--test-threads=1).
• codex-core --test all, filtered to everything touching this change (token_count context_baseline context_window compact_ usage): 131 passed, 1 failed. That one, suite::otel::turn_and_completed_response_spans_record_token_usage, passes alone and passes with the whole suite::otel:: module at any thread count; it fails only when interleaved with unrelated tests, because it asserts against a captured global tracing subscriber. The unfiltered suite is not a useful signal on this host: codex-code-mode-runtime does not build here, so ~250 tests fail for reasons unrelated to any change.
• cargo fmt --all clean. cargo clippy --all-targets clean across codex-core, codex-protocol, codex-tui, codex-app-server-protocol, codex-app-server, codex-state and codex-exec; the one remaining warning is an unused import in core/tests/suite/openai_file_mcp.rs, present on main.

suite::client::token_count_includes_rate_limits_snapshot needed updating: it asserts the full serialized TokenCountEvent, which now carries context_baseline. Rather than pin sizes that any prompt edit would break, the baseline is asserted structurally and removed from the exact comparison.

Compatibility

• TokenUsageInfo.context_baseline is Option, #[serde(default)], and skip_serializing_if = "Option::is_none". Old rollouts deserialize as None and behave exactly as they do today; new rollouts carry the field.
• ThreadTokenUsage.contextBaseline is nullable rather than optional, matching modelContextWindow, so the generated TypeScript stays inside the export lint's rules for non-params types. Clients that ignore it are unaffected.
• Persisting the baseline is intentional: a resumed session knows its own fixed cost before it builds its first request.

Limitations and follow-ups

• The estimate is incomplete and unbounded in both directions. Its categories are a subset of the fixed context, as described under Scope; and approx_token_count is ceil(bytes / 4), which can land either side of what a tokenizer produces depending on the content. So this is a heuristic aimed at large-tool-surface sessions, not a guarantee that any individual reading improves. The floor bounds the denominator, not the accuracy.
• Making the pricing exact would mean adding a tokenizer dependency to Codex, which is a repo-level decision, not something this change should take.
• After a compaction the reading falls back to the constant rather than being corrected. Adding the omitted tool tokens to the displayed total would be more accurate than either, but it changes a figure that is persisted in the rollout, so it belongs with the provenance work below.
• Anything a provider adds server-side is invisible to a client-side measurement. This is a property of the number, not a task.
• estimate_token_count_with_base_instructions still omits the tool schemas, which matters after a compaction, when that estimate stands in for provider usage. It belongs in a separate change for the reason given under Scope.
• A full request breakdown would supersede ContextBaseline. #27898 asks for one, and @alainkaiser has prototyped it in feat/context-breakdown with per-tool and per-MCP-server attribution. That prototype does not change the divisor — it renames the TUI copy of the constant and keeps 12_000 in both copies — so it composes with this change rather than replacing it. If it lands, ContextBaseline should be deleted and baseline_tokens() should read its fixed terms off ContextWindowUsage.