ADFA-5153: Compress Content table with a trained Brotli dictionary

[davidschachterADFA]

Summary

• Trains a shared zstd fast-cover Brotli dictionary (256 KiB) and compresses Content rows against it via the brotli CLI's -D flag (the installed Python brotli package has no dictionary API). Stored in a new single-row CompressionDictionary table inside documentation.db itself, so it always ships in sync with the content compressed against it.
• populate_db.py / insert_optimized_media.py: the Kotlin-website ingestion pipeline now compresses/decompresses through the shared dictionary, never retraining an existing one (a dictionary-compressed row is only decodable against the exact dictionary it was compressed with — verified empirically to fail, or silently produce different bytes, on any mismatch).
• New migrate_content_to_dictionary_brotli.py: one-time, idempotent, whole-database migration for every Content row the above pipeline doesn't own (reference docs, tooltip-linked pages, etc.). Idempotent because a plain decode of already-migrated content reliably fails (verified over 200 trials).
• docdb-studio/docdb_studio.py: fixes a real correctness gap this surfaced — its Content reads (get_html_anchors_for_path, fetch_content_for_path) and writes (compress_for_storage) were still plain Brotli, which was already broken against the migrated database (anchor validation / content preview were silently erroring on every real page). Now dictionary-aware, same approach as populate_db.py, with a graceful fallback for a database with no dictionary yet.
• 29 new tests total across the three changed areas. Ran the full migration against the real 299MB documentation.db: 29,748/29,751 brotli rows migrated, compressed bytes 131.1MB → 85.6MB (34.7% smaller), file overall 299.0MB → 255.3MB. Spot-checked real pages post-migration (both via the migration script and via docdb-studio) — all decode correctly.

Companion PR: appdevforall/CodeOnTheGo#1677 (WebServer.kt read-side).

Note: scripts/DocumentationDatabase.py looks obsolete against the current schema (its own schema-conformance check would reject this database outright — no templateId/Templates awareness) — flagged for a follow-up decision on deletion, not touched here.

Test plan

• docdb-studio: uv run pytest tests/ — 170/170 pass (162 existing + 8 new)
• ProcessKotlinWebsiteJSON: python3 -m unittest across both new test files — 21/21 pass
• Ran migrate_content_to_dictionary_brotli.py against the real documentation.db, verified idempotent (second run: 0 migrated, all already-dictionary-compressed)
• Spot-checked 8+ random real pages (androidx/java/kotlin doc sets) decode correctly post-migration, both standalone and through docdb-studio's fixed read paths

[davidschachterADFA]

Piggybacking one more ADFA-5141 fix onto this PR (commit b203500), since it lands in the same file this PR already touches:

While reviewing PR #25 (the docdb-studio side of ADFA-5141), I traced where the actually shipped documentation.db gets built and found scripts/DocumentationDatabase.py — which #25 fixed — is dead code: its tag-triggered publish-doc-db.yaml workflow hasn't fired since db-2025-07-16b, over a year ago.

The real pipeline is this branch's populate_db.py, which runs its own independent VACUUM (previously with no page_size pin at all). Extracted that into vacuum_and_pin_page_size(), mirroring docdb-studio's vacuum_database() — same WAL-mode workaround, same page_size pin. Added test_vacuum_and_pin_page_size.py (3 tests, all passing) covering the real 1024→2048 migration and WAL mode.

cc @alexmmiller since this builds on your fix/ADFA-4737 (PR #21) — flagging in case you want this pulled into #21 directly instead of riding in on top here.

[davidschachterADFA]

vacuum_and_pin_page_size (commit b203500) mirrored docdb-studio.py's original vacuum_database(): in-place VACUUM + a journal_mode round-trip, which requires exclusive access to the db file. SQLite refuses to switch a WAL-mode database away from WAL while any other connection has it open — even one from a function that already returned, since with sqlite3.connect(...) as conn: does not close conn on exit.

Found and fixed the identical bug in docdb_studio.py's vacuum_database on PR #25 (empirically reproduced as a deterministic deadlock), so applied the same fix here since this pipeline's own VACUUM is the one actually run against the live documentation.db.

Rewritten on VACUUM INTO + atomic os.replace — only needs a read snapshot of the source, so it works regardless of what else has db_path open. Two new tests: succeeds with both an unrelated open connection and an unclosed caller-style connection present at once (the scenario the old design was fragile against), and the original file is left untouched if VACUUM INTO fails partway.

17/17 local tests pass (test_vacuum_and_pin_page_size.py, test_populate_db_dictionary.py, test_migrate_content_to_dictionary_brotli.py).

Note: this branch predates PR #25's docdb_studio.py fixes (forked before ADFA-5141's work started), so docdb_studio.py here still has the old, page_size-unaware vacuum_database — that'll need a rebase/merge when PR #25 lands, separate from this fix.

[davidschachterADFA]

Follow-up: applied the same fixes here that a maintainer's QA and a follow-up self-review turned up on the mirrored PR #25 fix:

• Restored file permissions after the VACUUM INTO + os.replace swap (tempfile.mkstemp always creates its file mode 0600, silently dropping a 644 documentation.db to 600 on every vacuum — confirmed on real hardware in PR ADFA-5141: Pin doc DB page_size to 2048 in vacuum_database #25's QA).
• Switched VACUUM INTO's target from an f-string to a bound parameter (matching this file's own backup_database, which already does this) — a single quote in db_path's parent directory otherwise broke the statement.
• Closed the journal_mode-read connection explicitly instead of relying on it being reassigned by the next with block.

19/19 local tests pass.

[davidschachterADFA]

Applied the applicable subset of PR #25's third self-review fixes here too: chmod-before-replace ordering and explicit connection closes with consistent timeouts. 19/19 local tests passing.

[davidschachterADFA]

ADFA-5141 (the page_size=2048 change this PR's vacuum_and_pin_page_size was mirroring) has been declined — benchmarking showed essentially no performance difference between 1024 and 2048, and a negligible size difference before compression (likely even less after this PR's dictionary compression). Reverted that code out of this PR in 801f5eb6 so it stays scoped to the Brotli dictionary compression work (ADFA-5153) it's actually about. populate_db.py is back to its original plain VACUUM call; test_vacuum_and_pin_page_size.py removed along with it. 12/12 remaining local tests pass.

[alexmmiller]

QA'd this against the real ~300MB production documentation.db (post-ADFA-5141 page_size migration). The dictionary migration itself works and matches the PR's claimed numbers closely (29,748/29,751 rows, 131.08MB → 85.91MB, 34.5% smaller; spot-checked several real pages decode correctly through docdb-studio's fixed read paths). Found two bugs while doing so — left inline comments on each. Neither corrupted data (the first rolls back cleanly; the second just produces redundant, byte-identical work), but both are worth fixing before this ships.

[hal-eisen-adfa]

Code review — findings in files outside this PR's diff

This pass reviewed origin/main...HEAD, which is wider than #26's own diff (the branch has #27 merged into it). The five inline comments on this review are on files #26 actually changes.

The findings below are on the branch but live in files #26 does not modify, so GitHub can't anchor them to a line. Recording them here rather than dropping them — they may belong on a separate PR or ticket. Every one was verified against the code; several were reproduced by execution.

---

CRITICAL — scripts/sync_kotlin_stdlib_docs/sync_kdoc_json_to_db.py:54

compress_for() calls plain brotli.compress(). CI step 2 (populate_db.py) creates and uses the shared CompressionDictionary on documentation.db, then step 5 (build-kotlin-docs.yaml:267) runs this script against that same file. Every k/kotlin-stdlib/*, k/kotlin-reflect/* and k/kotlin-test/* row is left plain-brotli inside a dictionary database.

Per this PR's own note in populate_db.py:271-289, a wrong-dictionary decode "can just as easily 'succeed' while silently returning different bytes" — so this surfaces as corrupted stdlib pages with no error anywhere. This is the finding most worth fixing before #26 lands, since #26 is what makes documentation.db a dictionary database in the first place.

HIGH — sync_kdoc_json_to_db.py:166

"Source file missing ⇒ delete the row" has no sanity floor. If a Dokka upgrade changes the emitted layout so that zero files resolve, this deletes every stdlib Content row plus their parent Tooltips via cleanup_orphaned_tooltips, prints Done: updated 0, deleted N, and exits 0. The workflow's Summary step only prints row counts without asserting on them, so the gutted DB uploads to production.

HIGH — ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/md_to_json.py:84

TAG_RE = ^<(/?)(tabs|tab|note|tip|warning)([^>]*)/?>$ has no boundary after the alternation. Verified by execution: <table> → ('', 'tab', 'le'), </table> → ('/', 'tab', 'le'), <notes> → ('', 'note', 's'). A page with a hand-written HTML table (the docstring names roadmap.md) loses its <table> wrapper and the browser receives orphaned <tr>/<td> markup.

HIGH — md_to_json.py:417

Only fence is handled. markdown-it's code_block token (4-space-indented code, fully supported by the "commonmark" preset) falls through to the catch-all {"type": "html", "html": str(t.content)} and is rendered via {{ b.html|raw }}. Indented Kotlin such as fun <T> box(x: T) has <T> parsed as an unknown element and silently swallowed.

HIGH — optimize_media.py:233 and :349

dst.with_suffix(".webp") / .with_suffix(".png") collapses distinct sources onto one output path with no collision check. With --webp, a.png and a.jpg both write out/a.webp and one silently overwrites the other. build_rename_map accepts both, since its duplicate check only guards repeated old names — so pages that referenced a.jpg get rewritten to an image showing different content.

HIGH — build_nav.py:64

load_page_index runs json.loads(...).get("id") over rglob("*.json"), but line 207 writes nav.json — a top-level array — into that same directory, and the README documents exactly that invocation (build_nav.py <docs-root> <output-dir> <output-dir>). The second run dies with AttributeError: 'list' object has no attribute 'get' until the user manually deletes nav.json.

HIGH — .github/workflows/build-kotlin-docs.yaml:163

Requests only the drive.file scope. The already-working sibling workflow that downloads the same GOOGLE_DRIVE_FILE_ID through the same check-tools/download_database.py requests drive.readonly and drive.file (docdb-regression-test.yaml:46-48). drive.file covers only files the app itself created or opened, not a pre-existing shared file, so files().get() 404s and the job dies at its first Drive step on every run.

MEDIUM — sync_kdoc_json_to_db.py:164

UPDATE Content SET content = ? writes the whole blob into a single row, bypassing the CHUNK_SIZE fragmentation that populate_db.py:159 documents as having to match WebServer.kt exactly. Existing fragment rows are then deleted: relative_target_path("k/kotlin-stdlib/x.html-1") doesn't end in .html, so no source ever matches it and line 170 removes it — with those paths also fed into cleanup_orphaned_tooltips.

MEDIUM — md_to_json.py:142

The quoted-vs-bare test checks '="' in (attr_str or "") across the whole attribute string instead of per match. Verified: parse_attrs('width=500 style="block"') → {'width': '', 'style': 'block'}, and parse_attrs('kotlin-runnable="true" validate=false') → {'kotlin-runnable': 'true', 'validate': ''}. width=500 on its own works, which is what makes this easy to miss; {width=700 style="block"} emits width="" and the image renders at intrinsic size.

---

Lower-severity, not itemized above

• migrate_content_to_dictionary_brotli.py:125 — the docstring promises "a random sample drawn across the WHOLE Content table", but it's base_rows[:sample_size] over rows ORDER BY path — the first 300 alphabetically, i.e. effectively one directory. The dictionary is never retrained, so a narrow training set is permanent.
• build-kotlin-docs.yaml:367 — "Grabbing baton" is unconditional but "Dropping baton" is gated on !inputs.dry_run with no always(). dry_run defaults to true, so a default run acquires the team's lock signal and never releases it; a mid-build failure does the same.
• md_to_json.py:469 — _finalize_container keeps only type == "tab" children, so an intro paragraph before the first <tab> is dropped from the JSON with no warning.
• build_nav.py:64 — unsorted rglob with last-wins, versus md_to_json.build_topic_index's sorted/first-wins, makes nav.json non-reproducible across machines when two stems collide.
• build-stdlib-json-docs.sh:57 — under set -euo pipefail the DOKKA_VERSION="$(grep ... | sed ...)" assignment aborts on grep failure, making the if [ -z "$DOKKA_VERSION" ] diagnostic on lines 58-61 unreachable.
• run_e2e_pipeline_test.sh:105 — require_path validates user-supplied paths against the invoking cwd, then the step-2 subshell cds to $PROCESS_DIR before passing them through; the cd buys nothing, since populate_db.py resolves its assets via Path(__file__).parent.
• test_populate_db_dictionary.py:54 — asserts that a wrong dictionary decodes without raising, which the code's own docstring describes as nondeterministic. Brittle across brotli/zstd versions.
• assets/sidebar.js:114 — fetch(navSrc) never checks response.ok, so a 404 body gets injected into the sidebar as nav markup and .catch never fires.

[davidschachterADFA]

Reviewed against the shipped 299 MB documentation.db rather than from the diff alone — two blockers, both reproduced.

Blocker 1: chunked rows are silently skipped and counted as successes

reassemble_content probes <path>-1 first. But 14 of the 19 chunked chains in that database number from -2 — the very defect renumber_misnumbered_fragments.py exists to repair. For those rows the probe misses immediately, the function returns just the first 1 MiB, brotli.decompress raises on the truncated stream, and _migrate_one_row returns None — which the caller records as already_migrated. The row is never migrated and the run reports success.

The PR's own numbers corroborate this. "29,748/29,751 brotli rows migrated" — and that database has exactly three chunked rows whose content type is brotli: the notification-permission SVG, j/html/api/index-all.html, and constraint-layout-chain.mov. All three start at -2. The 3 counted as already-dictionary-compressed in a first-ever migration were not; they are the three that failed to reassemble.

Two changes, both worth making:

1. chain_fragments in the sibling script already discovers a chain correctly — LIKE query plus parsed suffix, agnostic to where the numbering starts. reassemble_content should use that approach rather than keeping a second, more fragile convention in the same PR.
2. Document the ordering dependency (renumber before migrate) in both docstrings, since getting it wrong is silent.

Blocker 2: the delete-then-insert destroys the Bookshelf

Content carries triggers:

CREATE TRIGGER DeleteBook AFTER DELETE ON Content WHEN OLD.path LIKE '%.pdf'
  BEGIN DELETE FROM Bookshelf WHERE contentID = OLD.id; END
CREATE TRIGGER AddBook AFTER INSERT ON Content WHEN NEW.path LIKE '%.pdf'
  BEGIN INSERT INTO Bookshelf (contentID, title) VALUES (NEW.id, CURRENT_TIMESTAMP || NEW.id); END

15 .pdf rows are brotli-typed, so they are in scope, and all 7 Bookshelf rows point at brotli-typed content. Running the exact delete_content + insert_chunked_content pair on a throwaway copy:

before: (53507, category 5,    'Android Notes for Professionals')
after:  (53508, category NULL, '2026-08-21 22:37:5553508')

Curated title, description and category lost for all 7 books, replaced by timestamp junk, silently, with the migration reporting success. This is also what ADFA-5212/ADFA-5179 curated.

Fix: UPDATE the base row in place instead of delete-and-reinsert. That also removes a second-order hazard — delete+insert assigns a new Content.id, so any id-keyed reference dangles even where no trigger papers over it.

Should fix

"Already migrated" is inferred, never verified. A failed plain decode is taken as proof of dictionary compression, but it is equally what truncation or corruption looks like — which is exactly how Blocker 1 hides. Confirm with a dictionary decode, and treat a row that decodes neither way as an error rather than a success.

No round-trip check before writing. recompressed is stored without confirming it decodes back to plain. One extra decode per row converts a silent bad write into a loud failure, and the compress is already the expensive part.

docdb_studio.decompress_brotli has no plain fallback. With a dictionary present it decodes dictionary-only, so plain rows in a dictionary-carrying database fail — precisely the state a partially-completed migration leaves behind, and the permanent state of plugin-inserted rows. WebServer tries dictionary-first and falls back, and that fallback is load-bearing rather than defensive: attaching a custom raw dictionary displaces the distance space brotli's built-in static dictionary occupies, so a plain row that referenced static-dictionary words genuinely fails the dictionary decode. Measured at window 22, 16 and 10 — corrupt input in all three.

Nits

The DictionaryCompressor docstring contradicts the migration's idempotency argument. It says decoding with "none" when a dictionary was used can "just as easily succeed while silently returning different bytes". Measured on 400 real migrated rows: 398 hard errors, 2 identical output (the dictionary was never referenced), 0 wrong bytes. The silent-wrong-bytes behaviour belongs to a wrong dictionary — a same-length, locally-perturbed one gave 50% errors, 38% silent wrong bytes, 12% identical. Worth separating the two cases, because the migration's idempotency depends on the no-dictionary case being loud, and this docstring tells a future reader that it is not. (ADFA-5222 covers detecting the wrong-dictionary case, which nothing currently can.)

The training sample is not random. base_rows[:sample_size] over ORDER BY C.path is the first 300 paths alphabetically — all a/... — while both the docstring and the PR description say "random sample drawn across the WHOLE Content table". This matters more than a usual doc nit, since the dictionary is trained once and load_or_create_dictionary can never retrain it.

delete_content's LIKE has no ESCAPE. _ and % are wildcards and 1,489 of 30,649 paths contain one. I checked the real database: zero rows would be wrongly deleted today, so this is latent rather than live — but chain_fragments already shows the safer pattern.

Peak memory. SELECT C.path, C.content, ... with .fetchall() holds every brotli blob (~131 MB) in one list, and each worker then re-reads the same content through its own connection. Selecting LENGTH(content) and fetching blobs per row would cut that substantially.

Base branch. This targets fix/ADFA-4737 (open PR #21), so it cannot land until that does, and its diff will shift underneath.

Worth keeping as-is

renumber_chain's single ascending pass is correct — each target was vacated by the previous rename — and simpler than the two-pass parking I used in the app-side script; I plan to simplify mine to match. _thread_compressor's per-thread reuse is the right call, and the never-retrain guarantee in load_or_create_dictionary is the right invariant in the right place.

[davidschachterADFA]

Follow-up with measurements, plus reconciliation against @hal-eisen-adfa's review above. Our passes had different scopes — that review covered origin/main...HEAD (wider than #26, including #27's files); mine covered #26's own diff and ran against the shipped documentation.db. Three findings overlap, and one premise we both relied on turns out to be wrong in a way that moves a severity.

Blocker 1 is now confirmed, not inferred

I enumerated the corpus independently while running the experiment below: the brotli set in that database is 29,751 base items and exactly 3 continuation rows. This PR reports scanning 29,751 rows and migrating 29,748. So the 3 rows it counted as "already dictionary-compressed" in a first-ever migration are precisely the 3 items that have continuations — the ones reassemble_content fails to reassemble because it probes <path>-1 and those chains start at -2. Nothing was already migrated; three rows were silently skipped and counted as successes.

Nit #7 measured: the sampling issue is worth ~19%, and the obvious fix makes it worse

@hal-eisen-adfa flagged the same base_rows[:sample_size] selection under lower-severity. It is worth more than that. I trained four 256 KiB dictionaries from the same corpus with one harness — zstd --train-fastcover, --maxdict=262144 — varying only how training rows were chosen, then compressed the same 4,000 random items (seed 11) with each:

dictionary	training material	dict size	stored	vs plain brotli	vs A'
A' current: first 300 by path	22.8 MiB, a/ only	262,144 B	10.5 MiB	36.2%	—
B stratified, 300 rows	12.7 MiB, 13 sets	165,069 B	11.0 MiB	33.2%	+4.65% worse
C stratified, 32 MiB budget	32.2 MiB, 6 sets	262,144 B	8.5 MiB	48.3%	-18.98%
D biased order, 32 MiB budget	32.1 MiB, a/ only	262,144 B	10.5 MiB	36.4%	-0.34%

D and C together isolate the mechanism. D shows more training material alone buys nothing (-0.34%, noise). C shows the spread across doc sets is the entire win. B shows why the one-line fix is a trap: stratifying at a fixed row count pulls quotas from smaller doc sets, total material drops to 12.7 MiB, and zstd cannot even fill the 256 KiB cap (165,069 B) — 4.65% worse than doing nothing.

So the fix is both changes together: stratify by doc set and control the sample by plaintext bytes rather than row count. Either alone is neutral or harmful.

For scale: my own full-corpus run with the shipped dictionary went 126.7 MiB -> 83.4 MiB (34.2%, consistent with A' here). At C's 48.3% that is roughly 65.5 MiB, about 18 MiB more off the asset. That is an extrapolation from 4,000 of 29,751 items, and my trainer settings may not match this PR's exactly.

Context worth stating: load_or_create_dictionary never retrains, so whichever dictionary a database is first minted with is permanent for that database's content. Fixing the sampler is therefore only free before the next asset is generated.

The premise correction: "no dictionary" is loud; "wrong dictionary" is what is silent

populate_db.py's DictionaryCompressor docstring says decoding with "none" when a dictionary was used "can just as easily 'succeed' while silently returning different bytes". Measured against the real database:

• No dictionary on 400 dictionary-compressed rows: 398 hard errors, 2 identical output (the compressor never referenced the dictionary), 0 wrong bytes.
• A wrong dictionary (same length, one 16 KiB region perturbed, 3 real rows x 16 regions): 50% errors, 38% decoded with no error and wrong bytes, 12% identical.

Two consequences:

1. This PR's own idempotency guarantee is sound — it rests on a plain decode failing on an already-migrated row, and that direction is reliable. The docstring currently tells a future reader otherwise, which is worth fixing precisely because someone will use it to justify tearing out the idempotency check.
2. It moves the severity of the CRITICAL above. sync_kdoc_json_to_db.py writing plain-brotli rows into a dictionary database is a real problem and should be fixed — but not because stdlib pages get silently corrupted in the app. WebServer.decompressBrotli tries the dictionary, gets a hard IOException, and falls back to a plain decode, which returns the correct bytes. What actually breaks is narrower and still worth fixing: docdb-studio's read paths, which after this PR decode dictionary-only with no fallback, so they fail on exactly those rows; those rows never get the dictionary's compression benefit; and the invariant populate_db.py claims WebServer needs ("every brotli row uses the same dictionary") is not one WebServer actually depends on, since it carries that fallback deliberately for plugin-contributed content.

Also on that thread: @hal-eisen-adfa's note that test_populate_db_dictionary.py:54 asserting a wrong-dictionary decode does not raise is brittle — the numbers above are why. That assertion is a 50/50 coin flip across brotli versions and payloads.

[davidschachterADFA]

Pushed 838ac44, which addresses every review finding that lands in files this PR touches, plus the two on sync_kdoc_json_to_db.py that this PR itself creates the conditions for. All seven inline threads are answered and resolved individually; this is the summary.

The two that were destroying data

The Bookshelf. The migration deleted and re-inserted each base row, and Content carries AddBook/DeleteBook triggers on '%.pdf' paths. Verified against the real database before the fix: (53507, category 5, "Android Notes for Professionals") became (53508, category NULL, "2026-08-21 22:37:5553508"). 15 brotli-typed .pdf rows are in scope and all 7 Bookshelf rows point at brotli-typed content, so a single run replaced the whole curated shelf with timestamp junk and reported success. Writes are now UPDATE in place, continuation rows reconciled by exact path. @hal-eisen-adfa's LIKE finding fixed this one too, since dropping delete+insert is what stops the triggers firing.

Silently skipped rows. reassemble_content probed <path>-1, so an ADFA-5171 chain numbered from -2 reassembled truncated, failed to decode, and was counted as "already dictionary-compressed". This was live, not theoretical: the corpus has 29,751 base rows and exactly 3 with continuations, and the pre-fix run reported "29,748 migrated, 3 already dictionary-compressed" on a first-ever migration. Chain discovery now lives in one place (populate_db.fragment_chain) shared with the repair script, so the two conventions cannot drift apart again.

Verified end to end

Against the real 256 MB database, journal_mode=delete, 20 workers: 29,677 rows scanned, 29,676 already dictionary-compressed, 1 migrated, 0 errors, Bookshelf still 7 rows with their titles and categories intact, integrity_check ok. That is the same run shape that previously destroyed the shelf and miscounted the chained rows.

Tests: 25 in ProcessKotlinWebsiteJSON (from 21) and 173 in docdb-studio (from 170). New coverage for the -2 chain, an undecodable row, Bookshelf survival through the real triggers, a never-referenced-dictionary row across two runs, stratified sample determinism and spread, zero-based renumbering, one bad chain not blocking other repairs, a locked database not being cached, a plain row in a dictionary database, and a missing brotli CLI.

Dictionary training

Measured with only the sampling varied, since @hal-eisen-adfa flagged the selection and the naive fix turns out to be a trap:

training sample	vs plain brotli
first 300 rows by path (all under a/)	36.2%
300 rows stratified across doc sets	33.2% -- worse
stratified, 32 MiB plaintext budget	48.3%
first-by-path, same 32 MiB budget	36.4% -- volume alone buys nothing

Stratifying at a fixed row count draws quotas from smaller doc sets, so total material falls and the trainer cannot even fill a 256 KiB dictionary. Sampling is now stratified by stored bytes and bounded by a plaintext budget, seeded for reproducibility since a stored dictionary is never retrained.

One shared premise corrected

Both reviews reasoned from DictionaryCompressor's docstring, which said a no-dictionary decode can silently return wrong bytes. Measured: decoding a dictionary row with no dictionary is loud (398 of 400 real rows raised, 2 returned identical bytes, none wrong) -- which is what makes this script's idempotency check and WebServer.kt's fallback sound. Decoding with the wrong dictionary is the silent case (50% raised, 38% returned different bytes with no error, 12% identical). The docstring now says that, and the test asserting a wrong-dictionary decode does not raise -- which was asserting a coin flip -- now asserts the invariant that actually holds.

Deliberately not in this commit

@hal-eisen-adfa's remaining findings are in files this PR does not touch and in a pipeline I cannot exercise here, so they are filed rather than bundled: ADFA-5237 (md_to_json.py: the <table> boundary bug, the code_block fallthrough, the per-match attribute test, the dropped intro paragraph) and ADFA-5238 (optimize_media.py filename collisions, build_nav.py poisoning its own index, the drive.file scope, the leaked CI baton, the chunking bypass in sync_kdoc_json_to_db.py, and the shell/JS nits). Both link to ADFA-5153.