Source
Scouted from headroom branch feat/agent-mode-lossless-densification (mature: 26 passing tests, post-landing hardening). That branch solves the same problem ctxzip does — compression for agents that resend full history every turn — but adds a lossless, deterministic, marker-free regime alongside the reversible-offload one. Three ideas hang together and are worth porting as one feature:
1. Value-factoring dictionary encoding (biggest win)
SmartCrusher-style schema headers elide repeated keys, but low-cardinality values still repeat on every row (namespace, status, file path). The transform: for each string column, collect distinct values and encode only when the byte math wins:
legend_bytes + index_bytes < raw_bytes → replace values with indices, emit `@column=[...]` legend
Header gains a __dict:K suffix; the codec reverses it exactly. Measured in headroom: search-result compression went from ~31% to ~57%, purely lossless. In ctxzip this applies to the records JSONCrusher keeps — kept rows still repeat status/namespace strings today. Requirements from their bug history: idempotent (re-compress is a no-op) and gated per-column by byte math, not a fixed threshold.
2. Densify-first stage
Densify JSON arrays reversibly (CSV-schema [N]{name:type,name:type?} + rows) before deciding what to offload; offload behind markers only if the densified form is still over budget. Mid-size outputs then need zero context_expand round-trips.
Load-bearing detail from their defect fixes — the CSV cell encoding must distinguish three states or the codec is silently lossy:
- missing key → bare
\N sentinel
null → bare empty cell"" empty string → quoted ""- literal
\N string → quoted "\\N"
3. Round-trip verify-and-revert gate
Optional verification: decode the densified form and compare against the original via canonical JSON (sorted keys, type-strict — catches bool↔int and int↔float corruption that plain equality accepts). Any message that doesn't verify reverts to the original. Result reports lossless and reverted counts. This is the guarantee that makes densification safe to enable by default.
Fit with existing architecture
All three are additive: they slot in as a rendering upgrade + pre-offload stage inside the existing crusher pipeline and change nothing about the marker/store contract, determinism (no ML, no randomness), or the fidelity floor.
Source
Scouted from headroom branch
feat/agent-mode-lossless-densification(mature: 26 passing tests, post-landing hardening). That branch solves the same problem ctxzip does — compression for agents that resend full history every turn — but adds a lossless, deterministic, marker-free regime alongside the reversible-offload one. Three ideas hang together and are worth porting as one feature:1. Value-factoring dictionary encoding (biggest win)
SmartCrusher-style schema headers elide repeated keys, but low-cardinality values still repeat on every row (namespace, status, file path). The transform: for each string column, collect distinct values and encode only when the byte math wins:
Header gains a
__dict:Ksuffix; the codec reverses it exactly. Measured in headroom: search-result compression went from ~31% to ~57%, purely lossless. In ctxzip this applies to the records JSONCrusher keeps — kept rows still repeat status/namespace strings today. Requirements from their bug history: idempotent (re-compress is a no-op) and gated per-column by byte math, not a fixed threshold.2. Densify-first stage
Densify JSON arrays reversibly (CSV-schema
[N]{name:type,name:type?}+ rows) before deciding what to offload; offload behind markers only if the densified form is still over budget. Mid-size outputs then need zerocontext_expandround-trips.Load-bearing detail from their defect fixes — the CSV cell encoding must distinguish three states or the codec is silently lossy:
\Nsentinelnull→ bare empty cell""empty string → quoted""\Nstring → quoted"\\N"3. Round-trip verify-and-revert gate
Optional verification: decode the densified form and compare against the original via canonical JSON (sorted keys, type-strict — catches bool↔int and int↔float corruption that plain equality accepts). Any message that doesn't verify reverts to the original. Result reports
losslessandrevertedcounts. This is the guarantee that makes densification safe to enable by default.Fit with existing architecture
All three are additive: they slot in as a rendering upgrade + pre-offload stage inside the existing crusher pipeline and change nothing about the marker/store contract, determinism (no ML, no randomness), or the fidelity floor.