Observed (production central server, 2026-08-10)
Maintenance compaction runs and converges (the LLP 0199 baseline gate works), but its output files are tiny. Example: an ai_gateway_messages day partition of 52,329 rows was rewritten into 230 files, avg 463KB, max 9MB - all written in the one minute of the compaction run. Every compacted hyperparam partition looks like this (~0.5MB average across 789 live files / 409MB).
Cause
compactGeneration (src/core/cache/maintenance.js) flushes its batch to a data file at COMPACT_BATCH_SIZE (10k rows) or compact_batch_bytes (32MB) of estimated in-memory bytes, whichever comes first - and writes one file per flush. The byte cap is a deliberate OOM guard (a fat denormalized column pushes a 10k-row batch into gigabytes). But gateway message rows estimate ~140KB in memory and compress ~70x in parquet, so every flush lands after ~230 rows as a ~0.5MB file.
Consequences:
target_file_bytes (128MB) is unreachable; the 32MB compact_avg_file_bytes heuristic would re-flag every partition forever - the LLP 0199 gate is the only thing stopping infinite recompaction (and before it landed, prod recompacted the same partitions 29-51 times, producing equally small files each round).- Parquet row-group min/max stats stay per-tiny-file, so the sorted-rewrite pruning wins (server LLP 0116/0117) never materialize in the cache tier.
Proposed fix
Decouple the heap bound from the file bound: keep flushing row groups at the 32MB estimated batch cap, but stream successive row groups into one open parquet file until actual written bytes reach target_file_bytes (or a new compact_file_bytes target). Peak heap stays one batch; file size becomes a function of bytes actually written, not of the in-memory estimate.
Alternative (weaker): size flushes by compressed-bytes-written feedback from the writer. Requires a running estimate but no writer-API change.
Impact
Central-server cache partitions would compact to a handful of large sorted files, matching what the server's pre-export day-compaction already produces for the archive tier, and making cache-resident reads prune the same way archived reads do.
🤖 Generated with Claude Code
Observed (production central server, 2026-08-10)
Maintenance compaction runs and converges (the LLP 0199 baseline gate works), but its output files are tiny. Example: an
ai_gateway_messagesday partition of 52,329 rows was rewritten into 230 files, avg 463KB, max 9MB - all written in the one minute of the compaction run. Every compacted hyperparam partition looks like this (~0.5MB average across 789 live files / 409MB).Cause
compactGeneration(src/core/cache/maintenance.js) flushes its batch to a data file atCOMPACT_BATCH_SIZE(10k rows) orcompact_batch_bytes(32MB) of estimated in-memory bytes, whichever comes first - and writes one file per flush. The byte cap is a deliberate OOM guard (a fat denormalized column pushes a 10k-row batch into gigabytes). But gateway message rows estimate ~140KB in memory and compress ~70x in parquet, so every flush lands after ~230 rows as a ~0.5MB file.Consequences:
target_file_bytes(128MB) is unreachable; the 32MBcompact_avg_file_bytesheuristic would re-flag every partition forever - the LLP 0199 gate is the only thing stopping infinite recompaction (and before it landed, prod recompacted the same partitions 29-51 times, producing equally small files each round).Proposed fix
Decouple the heap bound from the file bound: keep flushing row groups at the 32MB estimated batch cap, but stream successive row groups into one open parquet file until actual written bytes reach
target_file_bytes(or a newcompact_file_bytestarget). Peak heap stays one batch; file size becomes a function of bytes actually written, not of the in-memory estimate.Alternative (weaker): size flushes by compressed-bytes-written feedback from the writer. Requires a running estimate but no writer-API change.
Impact
Central-server cache partitions would compact to a handful of large sorted files, matching what the server's pre-export day-compaction already produces for the archive tier, and making cache-resident reads prune the same way archived reads do.
🤖 Generated with Claude Code