fix: bound daemon memory so a long session cannot exhaust it - #23
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
A user reported the daemon reaching 20 GB before crashing. Nothing in the long-lived daemon bounded its in-memory footprint: not payloads, not replay, not the conversation history, and not the per-session maps. The dominant cost was transcript snapshotting. Every Claude lifecycle event (UserPromptSubmit/Stop/StopFailure/SubagentStop/SessionEnd) read the entire transcript into memory and journaled it verbatim, so a session re-recorded its whole growing transcript on every turn. On my own machine an 18 MB transcript had produced a 1.6 GB journal: 142 snapshot lines accounted for 1.54 GB while all 5,852 other lines totalled 19 MB. Replay then loaded that file with read_to_string and parsed every line into a Vec held at once, and accumulated every resulting span op -- each LLM row carrying a full history clone -- into a single Vec before emitting. That is the crash. The waste was total rather than incidental: the translator only ever consumes bytes past its cursor offset, so the entire prefix of every snapshot was dead weight in all cases. Only in-memory caches are bounded here. Nothing on disk -- journal, mirror, or conversation content -- is capped or truncated, because each is a durability record whose loss is silent; they are kept small by storing every byte once rather than by dropping data, and every in-memory structure is re-derivable from them. Changes: - Mirror transcripts instead of snapshotting them. New transcript_mirror module streams only new bytes into a daemon-owned file; the journal carries a reference plus the high-water offset. Each byte is stored once, and the offset bounds replay to exactly what the live run saw. Keeps the durability property that motivated snapshots -- the mirror is immune to the agent rewriting the path -- and makes journal growth linear in transcript size. - Stream journal replay. JournalReader reads entry by entry, bounded to the journal length recorded when the session was created, and each entry's spans are emitted as produced. Peak memory is one entry, not a whole session. - Retire idle sessions. Nothing ever removed a session from the six daemon maps, so translator state, sink handles, credential leases, and journal file handles accumulated until the process exited -- which for a continuously busy user never happened, since the idle watchdog needs every session quiet. Sessions now retire after --session-idle-timeout and rebuild from the journal on a later event. - Bound the session queue. Unbounded mpsc replaced with a bounded channel, so a stalled sink applies backpressure rather than accumulating events. The event is journaled before enqueue, so this costs latency, never data. - Drop two redundant full clones of the conversation history per turn. - Run journal/mirror GC periodically, not only at daemon startup. Backward compatible: journals written before mirroring inline the transcript as _bt_transcript_snapshot, and translators still accept that form. A test asserts both forms translate identically across all three Claude fixtures. Tests: journal growth stays flat across 40 turns on a growing transcript (would have caught the original bug), idle sessions retire and resume from their journal with stable span ids, and the mirror reference is self-contained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stephen Belanger (Qard)
force-pushed
the
fix/daemon-unbounded-memory
branch
from
August 17, 2026 05:19
e5faabe to
7575f5d
Compare
Stephen Belanger (Qard)
requested review from
Abhijeet Prasad (AbhiPrasad),
Ankur Goyal (ankrgyl) and
Andrew Kent (realark)
August 17, 2026 05:28
Abhijeet Prasad (AbhiPrasad)
approved these changes
Aug 17, 2026
Stephen Belanger (Qard)
added a commit
to braintrustdata/bt
that referenced
this pull request
Aug 20, 2026
Moves the `bt-daemon` pin from `1ca65d4` to [`abd6756`](braintrustdata/braintrust-coding-agent-plugins@abd6756), which is braintrustdata/braintrust-coding-agent-plugins#23 — *"bound daemon memory so a long session cannot exhaust it"*. ## Why A user reported the daemon reaching **20 GB** before crashing. Every Claude lifecycle event had been reading the whole transcript into memory and journaling it verbatim, so a session re-recorded its growing transcript on every turn — an 18 MB transcript produced a **1.6 GB** journal (142 snapshot lines accounted for 1.54 GB; all 5,852 other lines totalled 19 MB). Replay then loaded that file with `read_to_string` and parsed every line into a `Vec` held at once. The daemon now: - mirrors transcript bytes once into daemon-owned storage and journals only a reference plus a high-water offset, - streams journal replay entry by entry instead of materializing it, - retires idle sessions so translator state, sink handles, and credential leases are released rather than held for the process lifetime, - applies backpressure on session queues instead of using an unbounded channel. Nothing on disk is capped or truncated — only in-memory caches are bounded, and each is re-derivable from disk. ## Scope No source changes are needed in `bt`. It only constructs bt-daemon's clap-parsed command schema (`TraceArgs`), and the one new field (`ServeArgs::session_idle_timeout_secs`) carries a `default_value_t`, so this is not a breaking API change. It surfaces as a new hidden flag: ``` bt trace daemon --session-idle-timeout-secs <SECS> [default: 300] Retire a session's in-memory state after this many seconds without traffic. A later event rebuilds it from the journal. 0 disables retirement. ``` The `Cargo.lock` change is deliberately limited to the one `source` line. `cargo update -p bt-daemon` also re-resolved unrelated transitive picks (`windows-sys`, `socket2`, `getrandom`), which I reverted — the bump adds no dependencies, and pristine `main` passes `--locked` cleanly, so that churn was noise. ## Verification - `cargo check --locked --all-targets` — clean - `cargo test auth::tests` — 77 passed (what CI runs) - `cargo test trace` — 40 passed - `cargo clippy --locked --all-targets --all-features -- -D warnings` — 14 findings, identical before and after the bump. They are pre-existing `collapsible_if` lints in `src/traces.rs`, `src/sql.rs`, and `src/datasets/pipeline.rs` from a newer local clippy than CI's pinned toolchain; untouched here. - Confirmed the new flag and its default surface through the real `bt` binary. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A user reported the daemon reaching 20 GB before crashing. Nothing in the long-lived daemon bounded its in-memory footprint — not payloads, not replay, not the conversation history, and not the per-session maps.
Root cause
Every Claude lifecycle event (
UserPromptSubmit/Stop/StopFailure/SubagentStop/SessionEnd) read the entire transcript into memory and journaled it verbatim, so a session re-recorded its whole growing transcript on every turn.Measured on my own daemon state dir:
e8c6ac32-….jsonle8c6ac32-….ndjsonReplay then loaded that file with
read_to_string, parsed every line into aVecheld at once, and accumulated every resulting span op — each LLM row carrying a full conversation-history clone — into a singleVecbefore emitting. That is the crash.The waste was total rather than incidental: the translator only ever consumes bytes past its cursor offset, so the entire prefix of every snapshot was dead weight in all cases.
Scope: in-memory only
Only in-memory caches are bounded here. Nothing on disk — journal, mirror, or conversation content — is capped or truncated, because each is a durability record whose loss is silent. They are kept small by storing every byte once rather than by dropping data, and every in-memory structure is re-derivable from them.
Changes
transcript_mirrormodule streams only new bytes into a daemon-owned file; the journal carries a reference plus the high-water offset. Each byte is stored once, and the offset bounds replay to exactly what the live run saw. This keeps the durability property that motivated snapshots — the mirror is immune to the agent rewriting the path — while making journal growth linear in transcript size. The copy goes through a small fixed buffer, so mirroring an arbitrarily large transcript costs arbitrarily little memory.JournalReaderreads entry by entry, bounded to the journal length recorded when the session was created, and each entry's spans are emitted as produced. Peak memory is one entry, not a whole session.--session-idle-timeoutand rebuild from the journal on a later event. This is also what bounds the retained conversation history and the sink's open span handles.Compatibility
Journals written before mirroring inline the transcript as
_bt_transcript_snapshot, and translators still accept that form. A test asserts both forms translate identically across all three Claude fixtures, so upgrading neither changes live output nor breaks recovery from an existing journal.Tests
claude_journal_does_not_grow_with_the_transcript_on_every_turn— 40 turns on a growing transcript; asserts the journal stays flat and the mirror holds the transcript exactly once. This would have caught the original bug.idle_sessions_are_retired_and_can_resume_from_their_journal— retirement happens, and replay afterwards reuses the original span ids.mirror_and_inline_snapshot_transcripts_translate_identically— the compatibility guarantee above.claude_boundary_journal_references_a_self_contained_transcript_mirror— replaces the old snapshot assertion.122 tests pass; clippy and rustfmt clean.
Note
I confirmed the mechanism and the 1.6 GB artifact locally, but did not reproduce the user's 20 GB crash directly — their session size may differ.
🤖 Generated with Claude Code