From e0299279a344cf5e01fff5fa5c2616a60aea1d5e Mon Sep 17 00:00:00 2001 From: David Schachter Date: Thu, 20 Aug 2026 16:25:05 -0700 Subject: [PATCH 01/13] ADFA-5153: Add a parallel dictionary-recompression script for documentation.db Recompresses every brotli Content row against the dictionary already in the database's CompressionDictionary table. Written for the 20-Aug database, which has the dictionary but plain-Brotli rows, so nothing benefits from it yet. Two things about the data decided the design, both checked rather than assumed: Content over 1 MiB is not stored as independently compressed pieces. The rows are raw 1 MiB slices of a single Brotli stream -- a slice alone does not decode -- so the unit of work is a base row plus its continuations, concatenated, decoded, recompressed and re-split. A naive per-row migration would have destroyed all three such items, silently, since each slice still looks like a blob. And those continuation rows are numbered from -2 while WebServer's reassembly loop starts at -1 (ADFA-5170), so they already serve truncated. The script preserves whatever numbering it finds, keeping the migration behaviour-neutral; --renumber-continuations rewrites from -1 instead, which makes them reachable again, as an opt-in rather than a side effect. Classification tries the plain decode first, deliberately: attaching no dictionary to a stream that needs one reliably fails, so a successful plain decode proves a row is unmigrated. The reverse is not safe -- a dictionary attached to a stream that never used one can decode to different bytes without erroring. A row that decodes identically both ways is left alone; those are tiny already-compressed payloads the compressor found nothing to reference for. Every item is verified before it is written: the recompressed bytes must decode back to exactly the original plaintext, or the item is reported as an error and left as it was. Measured on a copy of the 20-Aug database, 20 workers: 29,751 items, no errors, 129.0 MiB of stored content down to 85.7 MiB (33.6%), 3.3 minutes against about 73 single-threaded. The file itself goes 313.8 MB to 267.7 MB after VACUUM, and integrity_check passes. Verified independently of the script's own accounting: 303 sampled items, including all three chunked ones, decode with the dictionary to content byte-identical to what the source decodes plainly. Re-running is cheap (0.1 min) and converges -- pass two rewrote one row 11 bytes smaller, passes three and four changed nothing. --- docs/documentation-database.md | 2 +- .../migrate_content_to_dictionary_brotli.py | 387 ++++++++++++++++++ 2 files changed, 388 insertions(+), 1 deletion(-) create mode 100755 scripts/docdb/migrate_content_to_dictionary_brotli.py diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 3055c955f0..355861fdd2 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -63,7 +63,7 @@ CREATE TABLE Tooltips ( ### Supporting tables - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. Append-only: each change is another `INSERT`, so the **row inserted last** is the current version, not the highest one ever recorded — a rebuild from an older content set is a downgrade and has to read as one (`DatabaseVersionResolver.resolveMajorVersion`, which returns null for a database predating the table). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `WebServer` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every row would then pay a failed dictionary decode before its plain one, on every request), and a migrated database that lost the table fails loudly instead of quietly. -- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). +- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). - **`Templates(id, name, content)`** — Pebble template source, keyed by id (and by `name` for well-known templates like `bookshelf`). Referenced by `Content.templateId`. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py new file mode 100755 index 0000000000..af0f6dc885 --- /dev/null +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -0,0 +1,387 @@ +#!/usr/bin/env python3 +"""Recompress documentation.db's Brotli Content rows against the shared dictionary. + +Reads the dictionary from the database's own CompressionDictionary table (id = 1) +and rewrites every `ContentTypes.compression = 'brotli'` row so it is compressed +against that dictionary instead of plainly. WebServer tries a dictionary-attached +decode first and falls back to a plain one, so a half-migrated database still +serves -- which is what makes running this incrementally safe. + +Two properties of the data shape this script, both verified against the 20-Aug +database rather than assumed: + + * Content over 1 MiB is *not* stored as independently compressed pieces. The + rows are raw 1 MiB slices of one Brotli stream: `path`, then `path-N` + continuations. A slice on its own does not decode. So the unit of work here + is a logical item -- a base row plus its continuations -- concatenated, + decoded, recompressed, and re-split. Migrating such rows one at a time would + destroy the content. + + * A few rows decode identically with and without the dictionary: tiny, + already-compressed payloads where the compressor found nothing to reference. + Those are left alone, so "already migrated" covers them as well as genuinely + dictionary-bound rows, and re-running does not churn them. + + * The continuation rows in that database are numbered from **-2**, while + WebServer's reassembly loop starts at -1 (ADFA-5170), so those items already + serve truncated. This script preserves whatever numbering it finds, keeping + the migration behaviour-neutral; --renumber-continuations rewrites them from + -1 instead, which incidentally makes them reachable again. + +Parallel by default: compression at quality 11 is the whole cost (~4 GB of +plaintext), and it parallelises perfectly across cores. + +Usage: + # inspect: what would change, nothing written + migrate_content_to_dictionary_brotli.py documentation.db --dry-run + + # migrate a copy, then swap it in + cp documentation.db migrated.db + migrate_content_to_dictionary_brotli.py migrated.db --yes + +Requires the `brotli` CLI (>= 1.0) on PATH: no Python binding exposes custom +dictionaries, so encode and decode both shell out to it with -D. +""" + +from __future__ import annotations + +import argparse +import concurrent.futures as futures +import os +import re +import sqlite3 +import subprocess +import sys +import tempfile +import time +from dataclasses import dataclass, field + +CHUNK_BYTES = 1024 * 1024 +CONTINUATION = re.compile(r"^(.*)-(\d+)$") + +# Set once per worker process: the dictionary lives in a file because the CLI +# takes a path, and writing it once per process beats once per row. +_DICTIONARY_PATH = "" + + +def _init_worker(dictionary: bytes) -> None: + global _DICTIONARY_PATH + handle, path = tempfile.mkstemp(prefix="brotli-dict-", suffix=".bin") + with os.fdopen(handle, "wb") as out: + out.write(dictionary) + _DICTIONARY_PATH = path + + +def _brotli(args: list[str], payload: bytes) -> tuple[bool, bytes, str]: + """Run the brotli CLI over stdin/stdout. Returns (ok, output, stderr).""" + done = subprocess.run(["brotli", *args], input=payload, capture_output=True) + return done.returncode == 0, done.stdout, done.stderr.decode("utf-8", "replace").strip() + + +def decode_plain(payload: bytes) -> tuple[bool, bytes]: + ok, out, _ = _brotli(["-d", "-c"], payload) + return ok, out + + +def decode_with_dictionary(payload: bytes) -> tuple[bool, bytes]: + ok, out, _ = _brotli(["-d", "-D", _DICTIONARY_PATH, "-c"], payload) + return ok, out + + +def encode_with_dictionary(payload: bytes, quality: int, window: int) -> tuple[bool, bytes, str]: + return _brotli( + ["-q", str(quality), "-w", str(window), "-D", _DICTIONARY_PATH, "-c", "-f"], + payload, + ) + + +@dataclass +class Item: + """One logical piece of content: a base row plus any continuation rows.""" + + base_path: str + base_id: int + language_id: int + content_type_id: int + template_id: int + # (row id, suffix number, byte length), ascending by suffix + continuations: list[tuple[int, int, int]] = field(default_factory=list) + base_bytes: int = 0 + + @property + def stored_bytes(self) -> int: + return self.base_bytes + sum(n for _, _, n in self.continuations) + + @property + def first_suffix(self) -> int: + return self.continuations[0][1] if self.continuations else 1 + + +@dataclass +class Result: + base_path: str + status: str # migrated | already | unchanged | error + slices: list[bytes] = field(default_factory=list) + before: int = 0 + after: int = 0 + detail: str = "" + + +def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only_if_smaller: bool) -> Result: + """Decode an item, recompress it against the dictionary, and re-split it. + + Classification deliberately tries the *plain* decode first. Attaching no + dictionary to a stream that needs one reliably fails, so a successful plain + decode proves the row is not yet migrated; the reverse test is not safe, + because a dictionary attached to a stream that never used one can decode to + different bytes without erroring. + """ + stored = b"".join(blobs) + before = len(stored) + + ok, plaintext = decode_plain(stored) + if not ok: + ok_dict, _ = decode_with_dictionary(stored) + if ok_dict: + return Result(item.base_path, "already", before=before, after=before) + return Result(item.base_path, "error", before=before, detail="decodes neither plainly nor with the dictionary") + + # A stream that decodes *both* ways is one the compressor never referenced the + # dictionary for -- small, already-compressed payloads like a 1 KB GIF. It is + # byte-identical in either form, so there is nothing to migrate, and skipping it + # keeps a re-run from recompressing it for no gain. + ok_dict, as_dict = decode_with_dictionary(stored) + if ok_dict and as_dict == plaintext: + return Result(item.base_path, "already", before=before, after=before) + + ok, recompressed, stderr = encode_with_dictionary(plaintext, quality, window) + if not ok: + return Result(item.base_path, "error", before=before, detail=f"compression failed: {stderr}") + + # The migration is only worth anything if it round-trips exactly. + ok, roundtrip = decode_with_dictionary(recompressed) + if not ok or roundtrip != plaintext: + return Result( + item.base_path, + "error", + before=before, + detail="recompressed bytes do not decode back to the original content", + ) + + if only_if_smaller and len(recompressed) >= before: + return Result(item.base_path, "unchanged", before=before, after=before, + detail=f"dictionary-compressed form is larger ({len(recompressed)} vs {before})") + + slices = [recompressed[i:i + CHUNK_BYTES] for i in range(0, len(recompressed), CHUNK_BYTES)] or [b""] + return Result(item.base_path, "migrated", slices=slices, before=before, after=len(recompressed)) + + +def load_items(connection: sqlite3.Connection) -> list[Item]: + rows = connection.execute( + """ + SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, LENGTH(C.content) + FROM Content C + JOIN ContentTypes CT ON CT.id = C.contentTypeID + WHERE CT.compression = 'brotli' + """ + ).fetchall() + + by_path = {path: row for row in rows for path in (row[1],)} + items: dict[str, Item] = {} + continuations: list[tuple[str, int, int, int]] = [] + + for row_id, path, language_id, content_type_id, template_id, length in rows: + match = CONTINUATION.match(path) + # A continuation only counts as one if its base is itself a row; a path + # that merely ends in - is ordinary content. + if match and match.group(1) in by_path: + continuations.append((match.group(1), row_id, int(match.group(2)), length)) + else: + items[path] = Item(path, row_id, language_id, content_type_id, template_id, base_bytes=length) + + for base_path, row_id, suffix, length in continuations: + owner = items.get(base_path) + if owner is not None: + owner.continuations.append((row_id, suffix, length)) + + for item in items.values(): + item.continuations.sort(key=lambda entry: entry[1]) + + return sorted(items.values(), key=lambda item: item.base_path) + + +def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes]: + ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] + placeholders = ",".join("?" * len(ids)) + found = dict(connection.execute(f"SELECT id, content FROM Content WHERE id IN ({placeholders})", ids).fetchall()) + return [found[row_id] for row_id in ids] + + +def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], renumber: bool) -> tuple[int, int]: + """Write an item's new slices back. Returns (rows inserted, rows deleted).""" + connection.execute("UPDATE Content SET content = ? WHERE id = ?", (slices[0], item.base_id)) + + start = 1 if renumber else item.first_suffix + wanted = list(enumerate(slices[1:], start=start)) + existing = {suffix: row_id for row_id, suffix, _ in item.continuations} + inserted = deleted = 0 + + for suffix, payload in wanted: + row_id = existing.pop(suffix, None) + if row_id is None: + connection.execute( + """ + INSERT INTO Content (path, languageID, content, contentTypeID, templateId) + VALUES (?, ?, ?, ?, ?) + """, + (f"{item.base_path}-{suffix}", item.language_id, payload, item.content_type_id, item.template_id), + ) + inserted += 1 + else: + connection.execute("UPDATE Content SET content = ? WHERE id = ?", (payload, row_id)) + + # Whatever is left over described slices the new stream no longer needs. + for row_id in existing.values(): + connection.execute("DELETE FROM Content WHERE id = ?", (row_id,)) + deleted += 1 + + return inserted, deleted + + +def human(n: float) -> str: + for unit in ("B", "KiB", "MiB", "GiB"): + if abs(n) < 1024 or unit == "GiB": + return f"{n:,.1f} {unit}" if unit != "B" else f"{n:,.0f} B" + n /= 1024 + return f"{n:,.1f} GiB" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("database", help="documentation.db to migrate (operate on a copy)") + parser.add_argument("--yes", action="store_true", help="actually write; without it the run is a dry run") + parser.add_argument("--dry-run", action="store_true", help="explicit no-write run (the default anyway)") + parser.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2)), help="parallel compressors") + parser.add_argument("--quality", type=int, default=11, help="brotli quality (default 11, as the pipeline uses)") + parser.add_argument("--window", type=int, default=22, help="brotli window log (default 22, the portable maximum)") + parser.add_argument("--limit", type=int, default=0, help="stop after this many items (for a smoke test)") + parser.add_argument("--path", default="", help="only items whose base path contains this substring") + parser.add_argument("--batch", type=int, default=200, help="items per write transaction") + parser.add_argument( + "--only-if-smaller", + action="store_true", + help="leave a row alone when its dictionary-compressed form is not smaller", + ) + parser.add_argument( + "--renumber-continuations", + action="store_true", + help="write continuation rows from -1 rather than preserving existing numbering " + "(fixes ADFA-5170's unreachable slices; changes behaviour, so opt-in)", + ) + args = parser.parse_args() + write = args.yes and not args.dry_run + + connection = sqlite3.connect(args.database) + connection.execute("PRAGMA foreign_keys = ON") + + dictionary_row = connection.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone() + if dictionary_row is None or not dictionary_row[0]: + print("error: this database has no CompressionDictionary row to migrate against", file=sys.stderr) + return 2 + dictionary = dictionary_row[0] + + items = load_items(connection) + if args.path: + items = [item for item in items if args.path in item.base_path] + if args.limit: + items = items[: args.limit] + chunked = [item for item in items if item.continuations] + + print(f"database {args.database}") + print(f"dictionary {human(len(dictionary))}") + print(f"items {len(items):,} ({len(chunked)} of them stored as multiple slices)") + print(f"stored now {human(sum(item.stored_bytes for item in items))}") + print(f"workers {args.workers} quality {args.quality} window {args.window}") + print(f"mode {'WRITING' if write else 'dry run (pass --yes to write)'}") + if chunked and not args.renumber_continuations: + starts = sorted({item.first_suffix for item in chunked}) + print(f"continuations preserving existing numbering (starts at {starts}); " + f"--renumber-continuations rewrites from -1") + print() + + counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + before_total = after_total = 0 + inserted_total = deleted_total = 0 + errors: list[Result] = [] + started = time.time() + + with futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) as pool: + for offset in range(0, len(items), args.batch): + batch = items[offset : offset + args.batch] + pending = { + pool.submit( + migrate_item, item, read_blobs(connection, item), args.quality, args.window, args.only_if_smaller + ): item + for item in batch + } + + for future in futures.as_completed(pending): + item = pending[future] + result = future.result() + counts[result.status] += 1 + before_total += result.before + after_total += result.after or result.before + + if result.status == "error": + errors.append(result) + elif result.status == "migrated" and write: + inserted, deleted = write_item(connection, item, result.slices, args.renumber_continuations) + inserted_total += inserted + deleted_total += deleted + + if write: + connection.commit() + + done = min(offset + args.batch, len(items)) + elapsed = time.time() - started + rate = done / elapsed if elapsed else 0 + remaining = (len(items) - done) / rate if rate else 0 + print( + f"\r{done:,}/{len(items):,} items {rate:5.1f}/s " + f"eta {remaining/60:4.1f} min saved {human(before_total - after_total)}", + end="", + flush=True, + ) + + print("\n") + print(f"migrated {counts['migrated']:,}") + print(f"already {counts['already']:,}") + if counts["unchanged"]: + print(f"left alone {counts['unchanged']:,} (not smaller with the dictionary)") + print(f"errors {counts['error']:,}") + if write: + print(f"rows inserted {inserted_total} rows deleted {deleted_total}") + print(f"stored before {human(before_total)}") + print(f"stored after {human(after_total)}") + if before_total: + print(f"saved {human(before_total - after_total)} ({100 * (before_total - after_total) / before_total:.1f}%)") + print(f"took {(time.time() - started)/60:.1f} min") + + for result in errors[:20]: + print(f" error: {result.base_path}: {result.detail}", file=sys.stderr) + if len(errors) > 20: + print(f" ... and {len(errors) - 20} more", file=sys.stderr) + + if write: + connection.commit() + print("\nRun VACUUM to reclaim the freed pages: sqlite3 %s 'VACUUM;'" % args.database) + else: + connection.rollback() + print("\nNothing written. Re-run with --yes on a copy to apply.") + + connection.close() + return 1 if errors else 0 + + +if __name__ == "__main__": + sys.exit(main()) From 2ea1ba86639411db4e8b747ebc67332bf54625ab Mon Sep 17 00:00:00 2001 From: David Schachter Date: Thu, 20 Aug 2026 16:29:39 -0700 Subject: [PATCH 02/13] ADFA-5153: Keep Spotless's shell rules off Python scripts The `shell` block targets scripts/** wholesale and runs leadingSpacesToTabs(), so adding a .py file there gets it reindented to tabs -- against PEP 8, and against every .py already in this repo, all of which are space-indented. Only the ratchet has been hiding that: those files never differ from origin/stage, so Spotless never touches them. The first edit to scripts/cloudflare-r2-upload.py or scripts/insert-ci-perf-data.py would have silently converted the whole file, which is a trap worth removing rather than working around. --- build.gradle.kts | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/build.gradle.kts b/build.gradle.kts index b0f6438ace..9c3dd7ee92 100755 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -332,7 +332,13 @@ spotless { ".githooks/**/*", "scripts/**/*", ) - targetExclude("scripts/debug-keystore/adfa-keystore.jks") + targetExclude( + "scripts/debug-keystore/adfa-keystore.jks", + // leadingSpacesToTabs() would reindent Python, which PEP 8 indents with spaces -- + // and every .py already here is space-indented. Only the ratchet has been hiding + // that mismatch: an edit to one of them would silently convert the whole file. + "**/*.py", + ) } } From 5ef824040a88b47170ac77fd287fb4edd43e4b78 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Thu, 20 Aug 2026 17:02:11 -0700 Subject: [PATCH 03/13] ADFA-5153: Repair mislabelled and mis-chunked rows before recompressing The dictionary migration now runs in three phases, because each changes what the next one sees: retype -- 74 rows hold GIF/PNG/JPEG/QuickTime payloads but are typed text/plain (ADFA-5221), so they are Brotli-compressed for no gain and served as Content-Type: text/plain. Store their plaintext and point them at the type their magic bytes prove they are. renumber -- 14 of 19 chunked items number continuations from -2 while the app's reassembly loop starts at -1 (ADFA-5170), so they serve as their first 1 MiB and nothing more. Shift them down. migrate -- the existing recompression pass, unchanged. Phase 1 feeds phase 3 for free: a row retyped to image/gif inherits that type's compression = 'none', so the compression = 'brotli' selection stops seeing it. No exclusion list needed. Extensions only nominate phase 1's candidates; magic bytes decide, and a name/content disagreement is reported rather than trusted. The four .mov files are ftypqt QuickTime, not ISO-BMFF, so --mov-type chooses between the honest video/quicktime (inserted into ContentTypes as id 28) and the video/mp4 Chromium is likelier to play. Verified on a copy of the 20-Aug database: 74/74 retyped rows byte-identical to the original plaintext, all 19 chunked items reassembling to unchanged bytes, 250/250 sampled rows decoding with the dictionary to identical content, integrity_check ok, no foreign-key violations, Content and Bookshelf row counts unchanged, and a second run reporting nothing left to do. 3.5 min at 20 workers; 313.8 -> 268.1 MB after VACUUM. Co-Authored-By: Claude Opus 5 (1M context) --- docs/documentation-database.md | 3 +- .../migrate_content_to_dictionary_brotli.py | 511 +++++++++++++++--- 2 files changed, 451 insertions(+), 63 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 355861fdd2..75674d6e2b 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -36,6 +36,7 @@ One row per file the web server can serve (HTML, CSS, JS, image, video, PDF, ... - **`path`** is the lookup key (indexed via the `UNIQUE` constraint) and is what `WebServer` matches the HTTP request path against. Paths carry a short source prefix to avoid collisions between doc sets, e.g. `k/index.html` (Kotlin) vs `j/index.html` (Java). - **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `WebServer` relies on exactly this: it tries the dictionary first and falls back to a plain decode on `IOException`, which correctly handles both dictionary-compressed and plain rows — but never rely on decode success/failure to detect a *wrong* dictionary, since that case is silent. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `WebServer` before returning. - **`templateId`**: `0` (or unset) means `content` is legacy HTML with presentation baked in (the pre-CMS Release 0/1 format). A positive value means `content` is JSON *facts only*, rendered through the matching row in `Templates` (a Pebble template) — the ongoing move to a proper CMS that de-duplicates presentation across near-identical pages (e.g. `sin`/`cos` docs). +- Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5170), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there. - The `UNIQUE(path)` constraint rejects any duplicate `path`, regardless of `languageID` — a second language for an existing path isn't supported yet (only `EN-us` currently exists). Getting there needs an upstream schema change to composite uniqueness on `(path, languageID)` (see *Known rough edges* below). Dimensions: `Languages(id, value)` (4-letter codes, e.g. `EN-us`); `ContentTypes(id, value, compression)` (MIME type + compression scheme, ~30 rows). @@ -63,7 +64,7 @@ CREATE TABLE Tooltips ( ### Supporting tables - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. Append-only: each change is another `INSERT`, so the **row inserted last** is the current version, not the highest one ever recorded — a rebuild from an older content set is a downgrade and has to read as one (`DatabaseVersionResolver.resolveMajorVersion`, which returns null for a database predating the table). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `WebServer` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every row would then pay a failed dictionary decode before its plain one, on every request), and a migrated database that lost the table fails loudly instead of quietly. -- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). +- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows, after repairing the two data defects noted above)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). - **`Templates(id, name, content)`** — Pebble template source, keyed by id (and by `name` for well-known templates like `bookshelf`). Referenced by `Content.templateId`. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index af0f6dc885..7a592d0703 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -1,43 +1,68 @@ #!/usr/bin/env python3 -"""Recompress documentation.db's Brotli Content rows against the shared dictionary. - -Reads the dictionary from the database's own CompressionDictionary table (id = 1) -and rewrites every `ContentTypes.compression = 'brotli'` row so it is compressed -against that dictionary instead of plainly. WebServer tries a dictionary-attached -decode first and falls back to a plain one, so a half-migrated database still -serves -- which is what makes running this incrementally safe. - -Two properties of the data shape this script, both verified against the 20-Aug +"""Repair mislabelled binary rows in documentation.db, then recompress its Brotli +Content rows against the shared dictionary. + +Three phases, in this order, because each one changes what the next one sees: + + 1. retype -- rows that claim to be text but hold a GIF/PNG/JPEG/QuickTime + payload (ADFA-5221). Their declared type is `text/plain`, whose + ContentTypes row says `brotli`, so they were pointlessly + compressed *and* are served as `Content-Type: text/plain`. The + fix is to store the plaintext and point the row at the honest + type, whose compression is `none`. + 2. renumber -- chunked items whose continuation rows start at -2 while + WebServer's reassembly loop starts at -1 (ADFA-5170), so they + currently serve as their first 1 MiB and nothing more. + 3. migrate -- rewrite every `ContentTypes.compression = 'brotli'` row so it is + compressed against the database's own dictionary rather than + plainly. WebServer tries a dictionary-attached decode first and + falls back to a plain one, so a half-migrated database still + serves -- which is what makes running this incrementally safe. + +Phase 1 feeds phase 3 for free: a row retyped to `image/gif` inherits that type's +`compression = 'none'`, so phase 3's `compression = 'brotli'` selection simply +stops seeing it. No exclusion list is needed. + +Properties of the data that shape this script, all verified against the 20-Aug database rather than assumed: * Content over 1 MiB is *not* stored as independently compressed pieces. The - rows are raw 1 MiB slices of one Brotli stream: `path`, then `path-N` - continuations. A slice on its own does not decode. So the unit of work here - is a logical item -- a base row plus its continuations -- concatenated, - decoded, recompressed, and re-split. Migrating such rows one at a time would - destroy the content. + rows are raw 1 MiB slices of one stream: `path`, then `path-N` continuations. + A slice on its own does not decode. So the unit of work is a logical item -- + a base row plus its continuations -- concatenated, decoded, rewritten, and + re-split. Treating such rows one at a time would destroy the content. * A few rows decode identically with and without the dictionary: tiny, already-compressed payloads where the compressor found nothing to reference. Those are left alone, so "already migrated" covers them as well as genuinely dictionary-bound rows, and re-running does not churn them. - * The continuation rows in that database are numbered from **-2**, while - WebServer's reassembly loop starts at -1 (ADFA-5170), so those items already - serve truncated. This script preserves whatever numbering it finds, keeping - the migration behaviour-neutral; --renumber-continuations rewrites them from - -1 instead, which incidentally makes them reachable again. + * Extensions nominate phase 1's candidates; magic bytes decide. A row is + retyped to what its payload actually is, not to what its name suggests, and a + name/content disagreement is reported rather than trusted. The four `.mov` + files are `ftypqt` QuickTime, not ISO-BMFF, so --mov-type picks between the + honest `video/quicktime` (inserted into ContentTypes if absent) and the + `video/mp4` that Chromium is likelier to actually play. + + * `Content` has a real UNIQUE constraint on `path` (the schema's `UNIQUE('path')` + quotes the identifier but does enforce it), so a renumbering mistake fails + loudly instead of duplicating a row. The `AddBook`/`DeleteBook` triggers fire + only for paths ending `.pdf`, which continuation paths never do. Parallel by default: compression at quality 11 is the whole cost (~4 GB of plaintext), and it parallelises perfectly across cores. Usage: - # inspect: what would change, nothing written + # inspect: what all three phases would change, nothing written migrate_content_to_dictionary_brotli.py documentation.db --dry-run - # migrate a copy, then swap it in + # do it, on a copy cp documentation.db migrated.db migrate_content_to_dictionary_brotli.py migrated.db --yes + sqlite3 migrated.db 'VACUUM;' + + # just the data repair, leaving compression alone + migrate_content_to_dictionary_brotli.py migrated.db --yes --phases retype,renumber Requires the `brotli` CLI (>= 1.0) on PATH: no Python binding exposes custom dictionaries, so encode and decode both shell out to it with -D. @@ -58,6 +83,15 @@ CHUNK_BYTES = 1024 * 1024 CONTINUATION = re.compile(r"^(.*)-(\d+)$") +ALL_PHASES = ("retype", "renumber", "migrate") + +# Extensions worth a second look when a row claims to be text. The extension only +# nominates a candidate -- sniff() decides what the row actually holds. +BINARY_EXTENSIONS = ( + ".gif", ".png", ".jpg", ".jpeg", ".webp", ".ico", ".bmp", + ".mov", ".mp4", ".m4v", ".pdf", + ".woff", ".woff2", ".ttf", ".otf", ".wasm", +) # Set once per worker process: the dictionary lives in a file because the CLI # takes a path, and writing it once per process beats once per row. @@ -95,6 +129,40 @@ def encode_with_dictionary(payload: bytes, quality: int, window: int) -> tuple[b ) +def sniff(payload: bytes) -> str: + """The MIME type the bytes themselves declare, or '' if unrecognised.""" + if payload[:6] in (b"GIF87a", b"GIF89a"): + return "image/gif" + if payload[:8] == b"\x89PNG\r\n\x1a\n": + return "image/png" + if payload[:3] == b"\xff\xd8\xff": + return "image/jpeg" + if payload[:4] == b"RIFF" and payload[8:12] == b"WEBP": + return "image/webp" + if payload[:4] == b"\x00\x00\x01\x00": + return "image/x-icon" + if payload[:4] == b"%PDF": + return "application/pdf" + if payload[4:8] == b"ftyp": + # The brand distinguishes a QuickTime container from ISO-BMFF/MP4. + return "video/quicktime" if payload[8:12] == b"qt " else "video/mp4" + if payload[:4] == b"wOF2": + return "font/woff2" + if payload[:4] == b"wOFF": + return "font/woff" + if payload[:4] == b"OTTO": + return "font/otf" + if payload[:4] in (b"\x00\x01\x00\x00", b"true", b"ttcf"): + return "font/ttf" + if payload[:4] == b"\x00asm": + return "application/wasm" + return "" + + +def slice_stream(payload: bytes) -> list[bytes]: + return [payload[i:i + CHUNK_BYTES] for i in range(0, len(payload), CHUNK_BYTES)] or [b""] + + @dataclass class Item: """One logical piece of content: a base row plus any continuation rows.""" @@ -104,6 +172,8 @@ class Item: language_id: int content_type_id: int template_id: int + content_type: str = "" + compression: str = "" # (row id, suffix number, byte length), ascending by suffix continuations: list[tuple[int, int, int]] = field(default_factory=list) base_bytes: int = 0 @@ -116,15 +186,59 @@ def stored_bytes(self) -> int: def first_suffix(self) -> int: return self.continuations[0][1] if self.continuations else 1 + @property + def suffixes(self) -> list[int]: + return [suffix for _, suffix, _ in self.continuations] + + +@dataclass +class Inspection: + """Phase 1's verdict on one candidate row.""" + + base_path: str + status: str # retype | keep | error + sniffed: str = "" + slices: list[bytes] = field(default_factory=list) + before: int = 0 + after: int = 0 + detail: str = "" + @dataclass class Result: + """Phase 3's verdict on one item.""" + base_path: str status: str # migrated | already | unchanged | error slices: list[bytes] = field(default_factory=list) before: int = 0 after: int = 0 detail: str = "" + sniffed: str = "" # set when a text-typed row turns out to hold binary + + +def inspect_item(item: Item, blobs: list[bytes]) -> Inspection: + """Decide what a text-typed candidate actually holds, and hand back its plaintext.""" + stored = b"".join(blobs) + before = len(stored) + + if item.compression == "brotli": + ok, payload = decode_plain(stored) + if not ok: + ok, payload = decode_with_dictionary(stored) + if not ok: + return Inspection(item.base_path, "error", before=before, + detail="decodes neither plainly nor with the dictionary") + else: + payload = stored + + kind = sniff(payload) + if not kind: + return Inspection(item.base_path, "keep", before=before, after=before, + detail=f"declared {item.content_type}, and the payload carries no binary signature") + + return Inspection(item.base_path, "retype", sniffed=kind, slices=slice_stream(payload), + before=before, after=len(payload)) def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only_if_smaller: bool) -> Result: @@ -146,13 +260,17 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only return Result(item.base_path, "already", before=before, after=before) return Result(item.base_path, "error", before=before, detail="decodes neither plainly nor with the dictionary") + # Decoding every row here anyway makes a mislabel sweep free: report a + # text-typed row whose payload is recognisably binary, whatever its name. + mislabelled = sniff(plaintext) if item.content_type.startswith("text") else "" + # A stream that decodes *both* ways is one the compressor never referenced the # dictionary for -- small, already-compressed payloads like a 1 KB GIF. It is # byte-identical in either form, so there is nothing to migrate, and skipping it # keeps a re-run from recompressing it for no gain. ok_dict, as_dict = decode_with_dictionary(stored) if ok_dict and as_dict == plaintext: - return Result(item.base_path, "already", before=before, after=before) + return Result(item.base_path, "already", before=before, after=before, sniffed=mislabelled) ok, recompressed, stderr = encode_with_dictionary(plaintext, quality, window) if not ok: @@ -169,35 +287,45 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only ) if only_if_smaller and len(recompressed) >= before: - return Result(item.base_path, "unchanged", before=before, after=before, + return Result(item.base_path, "unchanged", before=before, after=before, sniffed=mislabelled, detail=f"dictionary-compressed form is larger ({len(recompressed)} vs {before})") - slices = [recompressed[i:i + CHUNK_BYTES] for i in range(0, len(recompressed), CHUNK_BYTES)] or [b""] - return Result(item.base_path, "migrated", slices=slices, before=before, after=len(recompressed)) + return Result(item.base_path, "migrated", slices=slice_stream(recompressed), + before=before, after=len(recompressed), sniffed=mislabelled) -def load_items(connection: sqlite3.Connection) -> list[Item]: +def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: rows = connection.execute( - """ - SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, LENGTH(C.content) + f""" + SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, + LENGTH(C.content), CT.value, CT.compression FROM Content C JOIN ContentTypes CT ON CT.id = C.contentTypeID - WHERE CT.compression = 'brotli' + WHERE {predicate} """ ).fetchall() - by_path = {path: row for row in rows for path in (row[1],)} + paths = {row[1] for row in rows} items: dict[str, Item] = {} continuations: list[tuple[str, int, int, int]] = [] - for row_id, path, language_id, content_type_id, template_id, length in rows: + for row_id, path, language_id, type_id, template_id, length, type_value, compression in rows: match = CONTINUATION.match(path) # A continuation only counts as one if its base is itself a row; a path # that merely ends in - is ordinary content. - if match and match.group(1) in by_path: + if match and match.group(1) in paths: continuations.append((match.group(1), row_id, int(match.group(2)), length)) else: - items[path] = Item(path, row_id, language_id, content_type_id, template_id, base_bytes=length) + items[path] = Item( + base_path=path, + base_id=row_id, + language_id=language_id, + content_type_id=type_id, + template_id=template_id, + content_type=type_value, + compression=compression, + base_bytes=length, + ) for base_path, row_id, suffix, length in continuations: owner = items.get(base_path) @@ -217,15 +345,35 @@ def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes]: return [found[row_id] for row_id in ids] -def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], renumber: bool) -> tuple[int, int]: +def write_item( + connection: sqlite3.Connection, + item: Item, + slices: list[bytes], + renumber: bool, + content_type_id: int | None = None, +) -> tuple[int, int]: """Write an item's new slices back. Returns (rows inserted, rows deleted).""" - connection.execute("UPDATE Content SET content = ? WHERE id = ?", (slices[0], item.base_id)) + type_id = item.content_type_id if content_type_id is None else content_type_id + connection.execute( + "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", + (slices[0], type_id, item.base_id), + ) start = 1 if renumber else item.first_suffix wanted = list(enumerate(slices[1:], start=start)) existing = {suffix: row_id for row_id, suffix, _ in item.continuations} inserted = deleted = 0 + # Retained rows are renumbered by taking the slot they now hold, so drop every + # old continuation path first and re-create what the new stream needs. Deleting + # before inserting keeps the UNIQUE(path) constraint out of the way when the + # numbering shifts. + if renumber and item.first_suffix != 1: + for row_id in existing.values(): + connection.execute("DELETE FROM Content WHERE id = ?", (row_id,)) + deleted += 1 + existing = {} + for suffix, payload in wanted: row_id = existing.pop(suffix, None) if row_id is None: @@ -234,11 +382,14 @@ def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], INSERT INTO Content (path, languageID, content, contentTypeID, templateId) VALUES (?, ?, ?, ?, ?) """, - (f"{item.base_path}-{suffix}", item.language_id, payload, item.content_type_id, item.template_id), + (f"{item.base_path}-{suffix}", item.language_id, payload, type_id, item.template_id), ) inserted += 1 else: - connection.execute("UPDATE Content SET content = ? WHERE id = ?", (payload, row_id)) + connection.execute( + "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", + (payload, type_id, row_id), + ) # Whatever is left over described slices the new stream no longer needs. for row_id in existing.values(): @@ -248,6 +399,59 @@ def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], return inserted, deleted +def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int) -> None: + """Point an item's rows at a different content type, leaving the bytes alone.""" + ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] + placeholders = ",".join("?" * len(ids)) + connection.execute( + f"UPDATE Content SET contentTypeID = ? WHERE id IN ({placeholders})", + [content_type_id, *ids], + ) + + +def content_types(connection: sqlite3.Connection) -> dict[str, tuple[int, str]]: + return { + value: (type_id, compression) + for type_id, value, compression in connection.execute("SELECT id, value, compression FROM ContentTypes") + } + + +def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> str: + """Shift an item's continuations down so they start at -1. Returns a note, or ''.""" + shift = item.first_suffix - 1 + if shift <= 0: + return "" + + expected = list(range(item.first_suffix, item.first_suffix + len(item.continuations))) + if item.suffixes != expected: + return f"continuations are not contiguous ({item.suffixes}); left alone" + + # A row this item already owns is not a clash: it is the one being moved out of + # that slot. Only a foreign row occupying a target path blocks the shift. + own = {row_id for row_id, _, _ in item.continuations} + for _, suffix, _ in item.continuations: + target = f"{item.base_path}-{suffix - shift}" + clash = connection.execute("SELECT id FROM Content WHERE path = ?", (target,)).fetchone() + if clash and clash[0] not in own: + return f"{target} already exists and belongs to another row; left alone" + + if write: + # Two passes: park every row under a name nothing can collide with, then + # settle them into their new slots. One ascending pass would be enough only + # if the target were always already free, and for a shift of 1 it never is. + for row_id, suffix, _ in item.continuations: + connection.execute( + "UPDATE Content SET path = ? WHERE id = ?", + (f"{item.base_path}-renumbering-{suffix}", row_id), + ) + for row_id, suffix, _ in item.continuations: + connection.execute( + "UPDATE Content SET path = ? WHERE id = ?", + (f"{item.base_path}-{suffix - shift}", row_id), + ) + return "" + + def human(n: float) -> str: for unit in ("B", "KiB", "MiB", "GiB"): if abs(n) < 1024 or unit == "GiB": @@ -256,11 +460,152 @@ def human(n: float) -> str: return f"{n:,.1f} GiB" +def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[str]]: + """Retype rows that claim to be text but hold a recognisable binary payload.""" + print(f"[1/3] retype {len(items):,} text-typed rows with a binary extension") + if not items: + return set(), [] + + types = content_types(connection) + counts: dict[str, int] = {} + problems: list[str] = [] + retyped: list[tuple[Item, Inspection, str]] = [] + before_total = after_total = 0 + + pending = {pool.submit(inspect_item, item, read_blobs(connection, item)): item for item in items} + for future in futures.as_completed(pending): + item = pending[future] + found = future.result() + if found.status == "error": + problems.append(f"{item.base_path}: {found.detail}") + continue + if found.status == "keep": + counts["left as text"] = counts.get("left as text", 0) + 1 + problems.append(f"{item.base_path}: {found.detail}") + continue + + target = found.sniffed + if target == "video/quicktime" and args.mov_type == "mp4": + target = "video/mp4" + extension = item.base_path.rsplit(".", 1)[-1].lower() + if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ + and not (extension == "mov" and target.startswith("video/")): + problems.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") + + retyped.append((item, found, target)) + counts[target] = counts.get(target, 0) + 1 + before_total += found.before + after_total += found.after + + missing = sorted({target for _, _, target in retyped if target not in types}) + for value in missing: + if write: + cursor = connection.execute( + "INSERT INTO ContentTypes (value, compression) VALUES (?, 'none')", (value,) + ) + types[value] = (cursor.lastrowid, "none") + print(f" ContentTypes + id {cursor.lastrowid} {value} (compression none)") + else: + types[value] = (-1, "none") + print(f" ContentTypes would insert {value} (compression none)") + + inserted_total = deleted_total = 0 + for item, found, target in retyped: + type_id, compression = types[target] + if not write: + continue + if compression == "none": + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + inserted_total += inserted + deleted_total += deleted + else: + # The honest type is itself a compressed one, so the stored bytes stay + # as they are and phase 3 picks the row up. + retype_rows(connection, item, type_id) + if write: + connection.commit() + + for target, count in sorted(counts.items(), key=lambda kv: -kv[1]): + print(f" {target:22} {count:>4}") + print(f" stored {human(before_total)} of Brotli -> {human(after_total)} of plaintext " + f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") + if write: + print(f" rows inserted {inserted_total} deleted {deleted_total}") + return {item.base_path for item, _, _ in retyped}, problems + + +def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[int, list[str]]: + """Shift -2-based continuation numbering down to the -1 the app expects.""" + items = [item for item in load_items(connection, "1 = 1") if item.continuations] + if args.renumber_scope == "retyped": + items = [item for item in items if item.base_path in retyped_paths] + broken = [item for item in items if item.first_suffix != 1] + + starts = sorted({item.first_suffix for item in broken}) + if broken: + print(f"[2/3] renumber {len(broken)} of {len(items)} chunked items start at " + f"{', '.join('-' + str(n) for n in starts)} instead of -1") + else: + print(f"[2/3] renumber all {len(items)} chunked items already start at -1") + problems: list[str] = [] + fixed = 0 + for item in broken: + note = renumber_item(connection, item, write) + if note: + problems.append(f"{item.base_path}: {note}") + else: + fixed += 1 + if write: + connection.commit() + + for item in items: + if item.base_bytes != CHUNK_BYTES: + problems.append( + f"{item.base_path}: chunked but its base row is {item.base_bytes:,} bytes, not " + f"{CHUNK_BYTES:,} -- the app detects chunking by that exact length, so it will not reassemble" + ) + if fixed: + print(f" renumbered from -1: {fixed}") + return fixed, problems + + +def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[str]: + """Re-read what phase 1 wrote and confirm the bytes match the declared type.""" + problems: list[str] = [] + written = {item.base_path: item for item in load_items(connection, "1 = 1")} + for path in sorted(retyped_paths): + item = written.get(path) + if item is None: + problems.append(f"{path}: row vanished") + continue + payload = b"".join(read_blobs(connection, item)) + found = sniff(payload) + expected = item.content_type + if expected == "video/mp4" and mov_type == "mp4" and found == "video/quicktime": + found = "video/mp4" # deliberately typed mp4; the container really is qt + if found != expected: + problems.append(f"{path}: declared {expected} but the stored bytes sniff as {found or 'unknown'}") + if item.compression != "none": + problems.append(f"{path}: retyped to {expected}, whose compression is {item.compression}") + if item.continuations and item.suffixes != list(range(1, len(item.continuations) + 1)): + problems.append(f"{path}: continuations numbered {item.suffixes}, expected 1..n") + return problems + + def main() -> int: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("database", help="documentation.db to migrate (operate on a copy)") + parser.add_argument("database", help="documentation.db to work on (operate on a copy)") parser.add_argument("--yes", action="store_true", help="actually write; without it the run is a dry run") parser.add_argument("--dry-run", action="store_true", help="explicit no-write run (the default anyway)") + parser.add_argument("--phases", default=",".join(ALL_PHASES), + help=f"comma-separated subset of {','.join(ALL_PHASES)} (default: all, in that order)") + parser.add_argument("--mov-type", choices=("quicktime", "mp4"), default="quicktime", + help="what to call the ftypqt .mov payloads: the honest video/quicktime (inserted into " + "ContentTypes) or the video/mp4 Chromium is likelier to play (default: quicktime)") + parser.add_argument("--renumber-scope", choices=("all", "retyped"), default="all", + help="renumber every -2-based chunked item, or only the ones phase 1 retyped (default: all)") parser.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2)), help="parallel compressors") parser.add_argument("--quality", type=int, default=11, help="brotli quality (default 11, as the pipeline uses)") parser.add_argument("--window", type=int, default=22, help="brotli window log (default 22, the portable maximum)") @@ -272,15 +617,15 @@ def main() -> int: action="store_true", help="leave a row alone when its dictionary-compressed form is not smaller", ) - parser.add_argument( - "--renumber-continuations", - action="store_true", - help="write continuation rows from -1 rather than preserving existing numbering " - "(fixes ADFA-5170's unreachable slices; changes behaviour, so opt-in)", - ) args = parser.parse_args() write = args.yes and not args.dry_run + args.phase_list = [phase.strip() for phase in args.phases.split(",") if phase.strip()] + unknown = [phase for phase in args.phase_list if phase not in ALL_PHASES] + if unknown: + print(f"error: unknown phase(s) {', '.join(unknown)}; pick from {', '.join(ALL_PHASES)}", file=sys.stderr) + return 2 + connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") @@ -290,32 +635,60 @@ def main() -> int: return 2 dictionary = dictionary_row[0] - items = load_items(connection) - if args.path: - items = [item for item in items if args.path in item.base_path] - if args.limit: - items = items[: args.limit] - chunked = [item for item in items if item.continuations] + def select(items: list[Item]) -> list[Item]: + if args.path: + items = [item for item in items if args.path in item.base_path] + return items[: args.limit] if args.limit else items print(f"database {args.database}") print(f"dictionary {human(len(dictionary))}") - print(f"items {len(items):,} ({len(chunked)} of them stored as multiple slices)") - print(f"stored now {human(sum(item.stored_bytes for item in items))}") + print(f"phases {' -> '.join(args.phase_list)}") print(f"workers {args.workers} quality {args.quality} window {args.window}") print(f"mode {'WRITING' if write else 'dry run (pass --yes to write)'}") - if chunked and not args.renumber_continuations: - starts = sorted({item.first_suffix for item in chunked}) - print(f"continuations preserving existing numbering (starts at {starts}); " - f"--renumber-continuations rewrites from -1") print() - counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} - before_total = after_total = 0 - inserted_total = deleted_total = 0 - errors: list[Result] = [] + problems: list[str] = [] + retyped_paths: set[str] = set() started = time.time() with futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) as pool: + if "retype" in args.phase_list: + candidates = select([ + item for item in load_items(connection, "CT.value LIKE 'text%'") + if item.base_path.lower().endswith(BINARY_EXTENSIONS) + ]) + retyped_paths, notes = phase_retype(connection, pool, args, write, candidates) + problems += notes + if write: + problems += verify_retype(connection, retyped_paths, args.mov_type) + print() + + if "renumber" in args.phase_list: + _, notes = phase_renumber(connection, args, write, retyped_paths) + problems += notes + print() + + if "migrate" not in args.phase_list: + connection.commit() if write else connection.rollback() + connection.close() + sys.stdout.flush() + for note in problems[:30]: + print(f" note: {note}", file=sys.stderr) + return 0 + + items = select(load_items(connection, "CT.compression = 'brotli'")) + chunked = [item for item in items if item.continuations] + + print(f"[3/3] migrate {len(items):,} items ({len(chunked)} of them stored as multiple slices)") + print(f" stored now {human(sum(item.stored_bytes for item in items))}") + print() + + counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + before_total = after_total = 0 + inserted_total = deleted_total = 0 + errors: list[Result] = [] + mislabelled: list[Result] = [] + for offset in range(0, len(items), args.batch): batch = items[offset : offset + args.batch] pending = { @@ -332,10 +705,12 @@ def main() -> int: before_total += result.before after_total += result.after or result.before + if result.sniffed: + mislabelled.append(result) if result.status == "error": errors.append(result) elif result.status == "migrated" and write: - inserted, deleted = write_item(connection, item, result.slices, args.renumber_continuations) + inserted, deleted = write_item(connection, item, result.slices, renumber=False) inserted_total += inserted deleted_total += deleted @@ -347,7 +722,7 @@ def main() -> int: rate = done / elapsed if elapsed else 0 remaining = (len(items) - done) / rate if rate else 0 print( - f"\r{done:,}/{len(items):,} items {rate:5.1f}/s " + f"\r {done:,}/{len(items):,} items {rate:5.1f}/s " f"eta {remaining/60:4.1f} min saved {human(before_total - after_total)}", end="", flush=True, @@ -367,10 +742,22 @@ def main() -> int: print(f"saved {human(before_total - after_total)} ({100 * (before_total - after_total) / before_total:.1f}%)") print(f"took {(time.time() - started)/60:.1f} min") + if mislabelled: + print(f"\nstill text-typed but holding binary ({len(mislabelled)}; phase 1 nominates by extension only):") + for result in mislabelled[:20]: + print(f" {result.base_path}: {result.sniffed}") + if len(mislabelled) > 20: + print(f" ... and {len(mislabelled) - 20} more") + for result in errors[:20]: print(f" error: {result.base_path}: {result.detail}", file=sys.stderr) if len(errors) > 20: print(f" ... and {len(errors) - 20} more", file=sys.stderr) + sys.stdout.flush() + for note in problems[:30]: + print(f" note: {note}", file=sys.stderr) + if len(problems) > 30: + print(f" ... and {len(problems) - 30} more notes", file=sys.stderr) if write: connection.commit() From 81147a54f18e75d5aa07b8ebbd5d975d36813295 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Fri, 21 Aug 2026 15:57:59 -0700 Subject: [PATCH 04/13] ADFA-5153: Cite ADFA-5171 for the chunk-numbering defect, not ADFA-5170 ADFA-5171 is "Chunked Content rows numbered from -2 break reassembly"; ADFA-5170 is a separate task about peak heap when serving chunked rows. The docstring and the doc paragraph both pointed at the wrong one. Co-Authored-By: Claude Opus 5 (1M context) --- docs/documentation-database.md | 4 ++-- scripts/docdb/migrate_content_to_dictionary_brotli.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 75674d6e2b..d93f6bc9a5 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -36,7 +36,7 @@ One row per file the web server can serve (HTML, CSS, JS, image, video, PDF, ... - **`path`** is the lookup key (indexed via the `UNIQUE` constraint) and is what `WebServer` matches the HTTP request path against. Paths carry a short source prefix to avoid collisions between doc sets, e.g. `k/index.html` (Kotlin) vs `j/index.html` (Java). - **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `WebServer` relies on exactly this: it tries the dictionary first and falls back to a plain decode on `IOException`, which correctly handles both dictionary-compressed and plain rows — but never rely on decode success/failure to detect a *wrong* dictionary, since that case is silent. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `WebServer` before returning. - **`templateId`**: `0` (or unset) means `content` is legacy HTML with presentation baked in (the pre-CMS Release 0/1 format). A positive value means `content` is JSON *facts only*, rendered through the matching row in `Templates` (a Pebble template) — the ongoing move to a proper CMS that de-duplicates presentation across near-identical pages (e.g. `sin`/`cos` docs). -- Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5170), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there. +- Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5171), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there. - The `UNIQUE(path)` constraint rejects any duplicate `path`, regardless of `languageID` — a second language for an existing path isn't supported yet (only `EN-us` currently exists). Getting there needs an upstream schema change to composite uniqueness on `(path, languageID)` (see *Known rough edges* below). Dimensions: `Languages(id, value)` (4-letter codes, e.g. `EN-us`); `ContentTypes(id, value, compression)` (MIME type + compression scheme, ~30 rows). @@ -64,7 +64,7 @@ CREATE TABLE Tooltips ( ### Supporting tables - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. Append-only: each change is another `INSERT`, so the **row inserted last** is the current version, not the highest one ever recorded — a rebuild from an older content set is a downgrade and has to read as one (`DatabaseVersionResolver.resolveMajorVersion`, which returns null for a database predating the table). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `WebServer` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every row would then pay a failed dictionary decode before its plain one, on every request), and a migrated database that lost the table fails loudly instead of quietly. -- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows, after repairing the two data defects noted above)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). +- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows, after repairing the two data defects noted above) (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). - **`Templates(id, name, content)`** — Pebble template source, keyed by id (and by `name` for well-known templates like `bookshelf`). Referenced by `Content.templateId`. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 7a592d0703..0926bc4864 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -11,7 +11,7 @@ fix is to store the plaintext and point the row at the honest type, whose compression is `none`. 2. renumber -- chunked items whose continuation rows start at -2 while - WebServer's reassembly loop starts at -1 (ADFA-5170), so they + WebServer's reassembly loop starts at -1 (ADFA-5171), so they currently serve as their first 1 MiB and nothing more. 3. migrate -- rewrite every `ContentTypes.compression = 'brotli'` row so it is compressed against the database's own dictionary rather than From da69775e4c8f4e6f8b44276810de96e3483cbefd Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sat, 22 Aug 2026 00:12:37 -0700 Subject: [PATCH 05/13] ADFA-5153: Fix seven defects found reviewing the migration script --limit and --path did not reach the renumber phase, so a scoped trial run -- the first thing anyone sensibly tries -- rewrote every chunked item in the database. Verified: --limit 1 --phases renumber --yes renumbered 2 of 2 items before, 1 of 1 after. Every run demanded a CompressionDictionary, and a database without the table crashed with a traceback instead of the intended message. Only retype and migrate decode, so renumber now runs without one -- which is exactly the old database whose numbering most needs repairing -- and a missing table says so and says which phase still works. The phases-without-migrate path returned 0 whatever it had printed. Failures of the work a phase exists to do (a decode that fails, a renumber that cannot proceed, a verification mismatch) are now errors and set a non-zero exit; observations that do not make the run wrong (left as text, a name disagreeing with its payload, a chunked item whose base row is not exactly 1 MiB) stay notes. Both are labelled in the output. Worker dictionary files were never deleted: 160 of them, 40 MB, had accumulated in /tmp from earlier runs. Each worker now unlinks its own at exit. .bmp was nominated as a candidate but sniff() had no BMP signature, so a real BMP was reported as carrying no binary signature -- the opposite of the truth. The mislabel sweep used startswith("text"), the same media-type boundary bug fixed in ADFA-5241, which calls textual/example a text type. The candidate SQL had it too. Both match at the boundary now, via one helper that says why. A dry run printed "renumbered from -1: N" having written nothing, and the wording had the direction backwards. It now says "would renumber to start at -1: N". Co-Authored-By: Claude Opus 5 --- ...ntent_to_dictionary_brotli.cpython-314.pyc | Bin 0 -> 51192 bytes .../migrate_content_to_dictionary_brotli.py | 157 +++++++++++++----- 2 files changed, 114 insertions(+), 43 deletions(-) create mode 100644 scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc diff --git a/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc b/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..cd20b49529d24d4708419767af2e6c7a8c19516b GIT binary patch literal 51192 zcmce#GcU0fsP)Bm z`{tYXJNH%<3M902l6fVuP|K~moO|xs?m721<>fjA9M`&Ty!83k1mUmgMm;S2!tf|+ z6od&u65_&nK{6aQ95Kd?Rdj7UYKoiKuQ_gJzm_A`xK(*)i`&>Ud)&@`bK*Jd*AaKH zUuWFOeqC`F`^}B#;@5OE?}$6@KH`acssw3IJYO=G3X-K%81cjl&e?nF)#{~xs|Crr z*}(7PMbh4Qv9vE72@ zbs@W6D;43om|d@vN^o7uuB*m?D_)$F=fs=@Voc3mgc;<}Dq*Gtby^|))GyAeaYL2AUEm%VG0HsHF6 zU3;ZwT(_|64U!Mnt?ar<+AnRy-6p!j_svop?%LVA7HKoCx3FuUv=!Ie*mbM49oIY9 z^+stYu6MEPP3PLA-P_L!La^K-1iz6-nI4~RtkHzAYUW{D|$jCbw9;9rLzt7?5y(EW1 zV)RlV7K$}fQSnRRm>7|RA-P$+7#a+W$3kK#FnCFfjD^I(OM$WBP)xjbDZsNF3%wj? zZ(^ZPY^TE^ikp0>R{VN2B#L573$H|+k`Imq!lPn5f?UNoeRgp?E?$a^1jT@O_-JQq zPxs;0V?7;*TThLL2QT-AM?=VM#Ssl$ABhBlV&ncpo%>t1w6$&W`o!K#AwVw}LIKn~ zRyNvjfeLAjqSUbq&8TrCmL)?4R=ucijOr@J0@v~BMFvTH7K;$%u8Y?~GN2R*kHtp- z?UC!NtETn@YOo0)a$xW!@B%u(fuXPrRNo|?2p^lJz@h#h=vOsM<-*a-DnU&;?K`z>SC=qb-Ep4Kc@s6}?7? z-M~jNqi|w7dSGQDt!rbUkr1PdVk|rw#TduHi^hOsA!AsLb8e%luL~*~!GQh`- zK{4cyAV1VlL06x6nvqy>d<HY?@ef)95*2ke#6A%|LG(Glv}UbGB^ZCH$r1C<#hsKp>_0Wl6< zgnEsGRRuv@<;Zo`yHqYwWfoBDMm zG=%nyr~|$xz>89eVz36;0+kp!9VN1Q4blkx4udO%$3hCz#FH3KS&Zczt* zQ0-{25t-zQ(5M2cj+X&98bgE#qmXn!gZcQ}gz-6u&ItqgL$O^V#(Q{Bym%cq1XV!P z2S8KrE7Vs<9T@E)il7P!fx-AV1AZ8lBa8!Rv_xY8^l5B-7}88!se6EdpI)>oT?;ZB z3k-u85$PhNlta;o46M@AZP4WL7@o0lM4cgv7ks0Us~1>yLtuzNW&r=f5UA9Z_ytju zZ+NYao<7-f;6!I9qrojL__Bq_8Aj*eWf1F7{8|X34B%t=asDn|xC&TATCb3(MCf@z zY=oEx>VvvLNQtPVfiW{NJR}A##?S^Y!$CDKC6m$U<_pLlsDAL0jQ$uOrKXOAFNa3L z#GAk=R7ip&kcAya4!%wZ7k;#@kQLbh}{sg1%6;SK(64> zkBlKd;!srUm*+5!EHD!GRt&5K}0 zLttpAabQr6Kt~va7zD!XI0K{(?PTb*o}d#&s-1jj5L1flR04(Uak-!2D-)}pW!_7cW8>k4AaL(k)a@>1Z0l?;=$ub#m2pRpnPrg68w7h z_a4~^nkV*L2fG;~b&_Ff=;bJQ0WblriHth5euB)x$PbtXiI9X%3KP2s5gLjk@d3uR znCQ#6S<4}x!)r)617o1AOdp7KdyT0a0uWTOSS)41@Er*blF>pjTnJ`dn{fL`#KnZ* z7tR~Ee*sD~o>dB`43`X&K{84v$=nJR*=zX)nHy4;tAUa6P|5_FlClCKU?(Z_#Ykl2 zmVgRaF&;+)@fa0RlNxNj6afJbqF;io`^SQEBpf^)9}k8j5YJu$qQ_bfqKPNRdM||z zM_Rcs7Hb7^1~0ZgBPyp{0|QE{LjwacG<31k`4ycte`xyG5^0E$C2hOu&P6u7{ZGV12W8v3HfdjWC-lwtl2 z$TvL&nQ9Ha9DuAGy0A+d9EzfIk?}lAcEJ$|28o`b0@p5a*$JNw#(8e^n9I;iG#QS8 z^ef8y1xSa)r{!@_Kd$9D0?{bg$3ZT85qmoV+D=4TwFzv7O~8>dQ^^Snom+tm@; z6GA_3`O!n8aK!?S>Pi&Q7!lll1ViK?lKG*#*O2q0nI=b zHTg|F)XEiyUr-SprZY4B+l+eBfuUiHS^5bM?C7DfxZR7WvY@y(CJ4U&=Om8M4P{G-Zs$K&XOH5XVv$W_3t;1`ZuP*n9M3_kQX50V3EbD}>&c z!|{||4h=(?3CSt<0MurP-Y@9%cizYZ(hm~ z7jwj!gUfd3YdJS^rZz0v*M4Xc9M$iaRwT_uAJ_$p^NS}xck-(zXL{c?Z}>15ukg*s zF~Z6p6zp%e{J7nDz?AbZA^dC*$ka8molb-m@)|ll4fS6^A`SJ;hTMk&PSfij(9cXQ zyIYO=1U!cNH0~#ail-uf21*KqNNLo!3PH>&5ZzvjL9$*53mlDZ8M?hDxe6r^7o%=X z*-oDBkmZPsu}p2%(ImMJ_w2+PRLko1z-Y4M-nTnm%Xu|ts`l61|8(D3@PS~kYfi6e*5EZ|bOMb@)=Zw=poRb2tj*-iTJoZ8)XO zLGY;fY(?$?~R|AWn1J?rvUm_DmdFcDi5N zcVpjlVBT3ZQ}peMw<>0X-@Sb6@{;eldEaw)O7D6Xe7#G)7v_C0Ecp79&iNm5kc>X_V*zQBJMb|e zc}VWWh1bZyDc9gyK8(`}D3&NtB+*i*_{!Z#${MjA7ZZX4&&B$!x6R42J+}eX!%u-K zQ5J@`)O}CVl|Ze{l2x*`LZb27SKx2TrkE2_CfH+A)@x7&BiA@XZAZpW?TN1-QOwJ# z@FAF}l~|~idICRl2ieb^LiUq9dV#&<2!g6>2hG2;rucD@iPZgCI=?{`^CW}rsix&f z#x##5713u;_aPc?Nf=wM;P0Az0)?=?!`%?55)B6~Mo=f>0UO*zd}IkHg-R*9Knd4@ z2UhlHFH$m4wBiE&=`2{z$^F&X)V43ZGTr;F_L-~S+%;!j+}e59d^ec1pI)}RA6ZN; zD|na1`Y#+Y-mEl?$WhGT6WMtLAO{Q{S`(y zS4^8J59N?*ci@Wa>@<}P#nqJY-pnS(kwYb%8rXYl#(SIYy`8H0>@^1fR4eGTfk9z&--aRBBJ@x;*+9enzn*#nr>9ap26 z7`qCi5eOa4utU(0#fAnlDvytm2g2|e%#A_c&hB3j`Rz{RXw!6fL`1ARM&}!t;LmlG zJJk2lv0=QY(cjHR;Qw0wUmrX8r!PfA^v=xQ(OEj$*|}3D;rtgQ;C3?_+}ClSC!u_C zEp#!uvzw8=?iiu(ffH&u7e}KkIT@i6E?&UPEn%`zQ`U7ohdL8(SQWToI5O4>WA0A* zMZ97K^0LTw&yrt>i{i)b71sBUbNB;T(pjQwCp+5`jv?46TCYWhhT7DJorxUnA$xYR zxA$bi&Ynf$L&{Y=KEz%Mf!Jumqp#UDJduk~Gp{j5u!h^j_H6;}a!0^x;$LFevI|U2 zg!n~#a7d-}ME!J{2Bf#K@X(MvfX9ql`~xl~g!gTElk&g$RI;Rb(boK)yL?%_->_)g zkiMUaEZXYcb5|&@r-m18YiAl}Yk#|CZrftbPTiZ(qOBtRU9!ZxX!E}3UaSA`J-0}= zv48W*)U`$1y7z0FW)J=LiMhVT+TBUFn6#g`(}Yvz7a1j|^Jf)#&t0Y@PL}LhwCzdX zPrbBg6W?=Jzvs+NlfQqKIFp??f9zzs5>tboE-C(idSo2~`-g(Ta2^^Eok>=t-UPMHcAX4(sd;$RqOGtioFj5MSnqGvO0TBDT_>=Ln+e;3?R;fN6vJ2 z4IFsBx8pR#E5gT=AjF;{Wsg$+PE@=5rt2bN0r_-S6wA`;F#7 zTxbWHK!@ScyEsn>=Y+UH65_`5|0!;&MZ@A|$#~8pnd1~ zG1()PLrDk71SOwB)l)Wh9}cFRBk1gLOfn7N#;r{Z!MI8foMaIjfvpQSjGCof`Xni* zZZ_qXA>~%4$p(gCGLFdC6Xk4ZafJ*bEqSf+)46l0T*^tUYb@npHDaSM!w{kfIEF`+ zsQ2SU>sH|NEfW`~bC8ZTQYE6qhE|$hQ~3m8pMG-S|E6SQvLhYPU^zx38|Udk{=0fa z6M|wy=R6`K2Gxwj)RjkiY)e8Zr7^EeAqXDLtg1@Y`tb~eFffRyA5sqOGz2JB4aW0? z&?|I8ZYMRf3W}!R__8ozsS>)-a-src+1g?X8)W)`L&cOb`Zmf(ai21baj9ChN9OE? z3jt=IycaAW;e|Se*4tI z)*~M=p7yzdA{e;jaVpOQw-`B&DDV?N$^k-BUsRrPB?!wnTbK^7L zDx9x9Iu~5pdgR{LBMY@h7pjlG<35_SA5C|d$}FB4mE>zE;YsKtzl;R;s4i)Q>0k)c zp&(3$AzePwu|p|8=h1oGJ`Q2Xx<*G3<_MBovmw=aK`-Z?H|X!y3+FKyU|ArtdCR~c zEl=P&1I*0*Za}ex^Rb|RG*9WdF8h8{>r&Gm{7*LRnT#*FtLNR-Gu7|7YiCZq<8DaW z58c^>Q@XLA7}&rID~e-kZi3q_K_#Bdi3$9~0Ak>CfFzCAeKbt>O?%l-x;l8z4u$>Z zJR)WPaj+*Adj|6yjLG5)ZnwEo0EP;x!WgP43YMxcPj(#^_^|r7$QModEPmLylF6Ku zIkM~=-l143mk%s96%mTPLn(=e0I3Ir0QtfU45SY1FVNf;2bm-!E4*V z0H#uLfj7E|Va$?7sSQ`?WX36FCL+%9hz(#njE>|zcU}G*rTHSAzC;O&2OtGtdUyCzN~3tJYAEW())e9rc^%~K-xi&z+rgw1)OzqM=Fcl zK@y0W`ctwfk_D4J&eBWgpCr|5K(PQOm4 zH|RtVl)pkJQeEWVqSKpn`Z}F{gH9_j<#l>SrxgBThj1cH*~Br$<^Fuva$)hr;mOkr z=HmCtt6?gC z{YC_VclMIGnD-!Mn;|J$@ck=e;n@P&qMp8PC^-s^u;kWjaJi(51z5FH}#I@Q=&TWKmI6#VB ziIazz%d6s<$6R-qYiLkMel^!jilo7eE4glwa{%}9XYQ~R0Yly`K91sBkbTO6>AQ&h zaVwv^AO{>wU%+vQ%>}_@a11s+vVe219?ehrmXVKn8tEzXVrENkXgh6T+x(}rdW6w4!@X#gj|zJu9(Y^5Lc?khHatLK}LE;!) zk?6(fIpaUfip6COG8$n1`Yn3@f8s=}3Ct$QOFcr5+vu_RGG;_#gvXevq(kwbP#hx% z*f{+_!zkK6d$wK+|8(otJepXm~zqER$_S?RX*GQd|)!H={LaM zlizRnCx{S*Ws|m_J3UjSU)noy=x0v%eRtV(?KgaH_-5;Gmn^Sup6y*~**o8|H`%-| zxo;#{J&FuG5y~JZV)(!+xJssiuaDgvo4It`{FDBq^!!4{3(ITQf2V!++P~k8w5}aU zYT3a_2Mu_YK+u3=1aBj2>Eq|}h~P;!{Mn@($ssvgt%S5(>}EhGafd;kQ@$R$Wo$#j zRmx4iv;*Oxf!HWH#L6CFi5tjQ^z4P+xDieZ6f8-cbD-M&I8iiB742jPQO#K>nO2CW&1wo zH>vhbrfhvWnx}&}=C?DrQTF@^E;;@@m(XU+fR0UTXRg{#)>EHJmk>l6wC10-UxUK^ zY^|Zv9Qtz7+%vd~)fbq^55HEf-@5&WD52?@+T&#Ka_KNt%LiO7*Kfr+ugxO4(;Voj zxZ~+B&Q_~zWuUL}(>Z$!0$;`O1$%>630M}~zC)uFprcy;$d^4MfKKB;T9^ro+d(?` zvmYdHW5iLBV?XqhAmSDf!3YB*7#R8oR5lzCP=g5b9ETZ$dpPH=qmpLzI9L^hePdN?ev&Pl`1_)Gh)Z zxT2=EI8F^5Vxd6@zGbs}#Ny!!Br<6LK=3&jHB8x*$RJqf(>(fH=xq77>GXfmiA4Sk z9!^X(+UZq@m5XyU44l+ewk?U2Agw=XFm;}R5iX5XlwKF;92S~>_=NB>l;M-gv zHsBHr7bi-o6>u2c8j z#frQ!JU5i=xVq@Rmb71kt7t>}+~%eBL-Xy2mg+n1)pvZgZaQbNpyqDdMAru{VU2Hg zXrXk=yt!!N@Xu^{Q`;5`8`4>YZ(Ec8z@qzN(th#hRa@r5ck36bPR*MuCk{*vQMED( zJ^W&_|6=mR!DL5p(H%&I_UBu<=b`q2$^jlE5x83G5P+zzU{nU-#Yg&20Ey|DEQAEhm@Ds=gINeHymkX(9>iB&D^S z)Y){=C+zrVE1eEZe~j@U){hI6xE9|*?HO8s9P zgGJVkf%&rhHTj3~e@{<;_Gqj=U1D0eiMsu6f&(4FGsJ@Z?rgs6=QXD$zj+6x-N({e zv-yZVH3wUNS~$yInx?#~aPzy`&FVEKZ>|f1PEEG^ZT=j;!|(Lx`t$rS7i|M1=;%9J zk8{@)hne3BIDz6^GsBJC6SYP2qb|RtM-%zjC+y_+sAWY9`~^ekvr3_>3JQ5aPeD4B zdX*IhHh(_qto1wU42V(Bp+rsGP;*ExM1mTMs9mAIFl$Mr=`YkP$%Mr`%&L(M<<#(U z`peW5PcD&`Yr*#lZfjHDCQN->6MxSFz=#f??l|6Yuva|j3nK*ippRxt=~ta0J$i`l zl=)8zXW2tFaq)!jDyULzw;m@qV54qeZv5*5d)1nHgADEzKG;K|yI zXE5PWtMT&|estB6{|oNr|BBN~x`H?V{@&l&%L?}9rfi{?LxY&mNZE%%h{b{)oN^sJ z+1-1z`^vs$%0|IZ^oTkzWrgF$ zFk(#{>?tdZ_vfF{2dv*KD4hi{{*WdaET}^>BoYOSxnSZ*_m%t?NV%V&M4*}wmYu~* z&b9N-wGWJzT+aiCkY7IeDKN{5oCi*8j_Z-h?y`R95p3?yUi$n?5Awme9~KE>?SiZJ zjrg=_D*rYAr2mchL`Tw9yS%n)qGNf>wuuve-G5=;es1b<33v%VGna*Rpd(^#}Pm6?u~#U+Q}3 z7S0(0WJ>n!nzwr=a}aH_b!O|br}*`pn>o|o1yAjyW!dRjbXG3A@|Rpy^RB8Tc(b_b zXRUKq2ty6~-gQ0q(1yPMcP-h==k4W7_NsaOuicrnSG{ZB_0S~b zSCK8KV5;u*=9|r5@y*xoUvwTwnh$*R%RNGF9dn17&F`o;|Et)%IkNpnLI~=AX8h(B zodDf}FVip~0e}sY8iohyptVs#01#my067}BkS}oylPvH%*Gn$OEw?YTr>n&Z8rNVq zhDXO32Kfyne6KtZ_F>2t%m(4X2=ReW(IO|Z8bsmAbfpcGw6L-$leCzcr4g?lQL`lU zXf_`8DJJL)I-7^ae4BMCH704v(6I?XD-Ox%w?GgwwVC{8)$#vH;UikkR==5ArTBIs z=PUJoqkc*sEmK4y_;bODmw9fPqG)57U*joQnK@->P8s$kS`&<+@187qk6I;~D2G6c%J{3rDAPkAavqZr04)6=6k!C5^ROS!-U2Sa4phHi{- zVp4ZjTWS%Z0;137SpF8@GCZOw79u3>*LL67J?Wc^FKs(|Z`;wuievNJj=uHE%+=W| zbJbt##Lc~mWA8YQO&nYv0cIAmHbjYA4A1qpYif7R19gE)=eL#)yt3hjoq z59ze*sd-?+@B5Ct2NwLou${Z+fsL;0g1hiR4qZ8f67hkPu3Un(_+c)t`1sb17z{yW zMynCffs|k#0`@}a&N|Ld7<$c73+$IjyYj$@GG@BE^qtD)KLrs~5$zCUYSKh~s%P zA}d5KetWw?2d!KvU+5k6h~{=xa~cZ8%jvfvw-{(Fpez3$erHxSB1h8lL#g(m4pztP zrCR~FEXoC_CX^f4Z%uellE{jq?{Dy&9IaLy?OK=~`rWmnC9bXkC^OmXVMLC0J=SzV z)Mbjx|1YrZ`!fmyV^rK6br3dydfO9+``8j^ij5U>{bN|ANLkg176Q}75R!aI%Ym>} zWMGY@Ti8;u97@pZ>rU)F({pHlZ-=TYK(Xo7IKduqpT-ur5(CX)pQ!N!-0YP9Ga%jF zO%gl{#!rZcpc)~=4An&cN#ZM-ZlUTwN4rmVNWI`Oy(hJbG;*CvuTd!+YDJx}V3np; zc~@oa-+%l}$7!(ldyF>!J-uT2zDHO84^D{I3SjCsC^NoD@-=6Uc7%yzQMR$+lwIww zluhX>HmMn-!MwsAs7T!`_?%B_{t}58+5UCVZWseSLc`X3^?N3ROL;Z-@@nqvn>f5| z&Ra5<%$rM=?MIe#+*5`xmrWi1M*kcAN&AKmEC_A-&><9*-`GDLU#i+TU$t?f<9=@b zSH07nGi_f#HXB+j*u0Pn|C#0OyO*{fySM$=JMOOe?Z@WVZ=TsYw`tn?>v_1FIC9V3 zwOm%7G?z~t`OqcUJyVC4it6Tz>Sps7i#9Bmtb09jGxBap!(`{Ot7OTwZr-&nS+)O8 z=baskt`o50OCGh36M4gbSp%FTONCRElus1*t;UT?kG`fkB` zlx#2HB6_l7&mHUS>x=fIh!C6F^kvtpHdMT@7WHryC7qQ?bLB@L95x7^;|3Pj@#9^2 zow@c5@mv#i8_|DEVSdaSD#{G4oy7&_JYFQxBOqc(fO8z-cidfF(A2_9P^0kWws9o|uA_+qo zG6cbY;$JM6rwNl|_wD}M-FK^#_T$SrYoPf~x8sC?Xt93u$UEpWhOt^v?-kn)KAlMzehty)pIQ*DdUt}+lm2l67<}(M3zlvQzbhxolC}J2DomR5+~Hf62_3HmuIpAQrPbPei9L$SxQTOkJ{>VI8laU;6w!e28j?Z-mPZ{4 zHHiDW4~bMX2@hCHieaB(8-rqk$wr2UAELSP-{8a@p|y}9mdOBPjWPLSyk~&sdQ_TL z+YjHZy<7UoWX!exoi&dckSv150mk9VpEh#}%)w58Re`I3-S9}n!za;PzHZe@T1(_2 zQBI04EuR57ge{eU@z0mYc)E7=L&BL#R(3iFJGpKnkGtM-QWtlvRvC!}ja3BMr;ouF zEj|pohJjf^xUS*gmJqhcK(*i0ZiK$!Qmi2$V>+jGs==N5GiKnMdp@HM1|877TyESNBK)l8 zwuFSBca}%ZI0J{Xa$6`oR5j(G6`8R9$xh5$;TuGAgHWX1J9G|CS@X(Xf{xD`vaOz? zD#UuOt#VTJUDS-cb@f#1>o+r%7Bf?*t1){TZcamFbk@=_1T1H)<0dwfh^}F#K!iPA z31JQqC9*k0ThxNt!^eEzc}^KNmslDMn3V@=VD9_89?qnuMsqMcvnb(`#Sb%y3W03u zvPR4uen*2+*f8s_t|L%aVHC(PKG=2unUwcu@g7RDWNm|~unKOcOkp*uxdbs^$c!w6 zq$>`n@jyuCV09KP&0hD%7&TBL)Ox0zTiSJNa&K7nWb;yr4yDb z(wa4D0a43Ibf3NyrnPgqs}iI&lFYO?L#$AVWxZp{TJcP~rFA@8uok|$LQjC4UDVil zA9J2b)MRFY*m~v>!=4V0N3gtcVLQpqxdn%?JmoMPlHfOl8j;U01T=jaiN-84q?&1lN}tGWQB8@^MO(Z%JZ$GH zsH02_hr7=(iSkQydznsuLZ^>#f{#sFtR$KUy8r<>1?`qrBvsYkqx8xI0JGqylobp4 zVb&{Up*1EiF_sbE)+e#;C^A^$g<7(#ZQ0s z)6@1vo4B0cK6mK%PrQ9%Zr`1jq`N0+?}0+s(6m&){a*d{MaPc$`t3JIriP{iGx@J> z#m&UQdB=|BHI++iHY~2$aKklenA)^lv__|h{`UD~?dD|u7U+(iaz0g+Tz~AY|L)NR zkAKNCFz*>y@B}6;KXZC+Y?(SUz3Dg3&YXU?xN*6pa_Vey?dfD;?>$%V!yKWkiFE2* z)TgF?sk(i!x*biQ?3~`ODZ0-xEljNz-O@#7zX|lxLK8u4=JX;)*^Jh3 zAw_OQ!^dE)h46-#_eC#c)%g}grIi(kZjr0XL2i0Z>m#Ho8=SK=VgVu(h6se zL|2bY`Y&6M`x-5;eN6^oVl;|9gI-1cn^j`IL+@0zC)aJB>%6_?cEg?gWWmAzw*IFZ z{&>TZ>&&AZp@PR7W(+Jd>H|dlWD3kH1XjxsChFtdG(_n9R9W=_V4y8IWHD5bCR;Iy z#Ygco6=Qd41q&InL#yAf;}$;mb_|5w7zl3#$bx5@z=L~Z^BScV_=yT#dleyse{E}nT^*}>8V;mP zhr9&x3GDeumC$WS_V-w z?RAi`+=b=R_-J5Eu0~o0y?zFSKlf~yG~WjbT=~-0sYAcIXZp}g$D1dXy)AQv->H}i zOq-|9PM?~NPY)#v>+ZP@-#&Fae*4Is1Gignvuw|q?EP}is|L6W)Imb5yf z3e-NqC(we7I+CdA%25kuv#f9IyWM{Q5 zVzoB*h^tdcIV=Tm7|N;x-(-bf4{&JZAvxhzsxW$~Uf;Vtz?GGU*0cF=t@M!c7kq)$ zDqm{Yfy6;hM?Df-T5b5a@-COv4m}E}dwNujP?x!sr}m!uau^~LQhzd7yZ!mV5fA!4 z3nYOfTGS9mY)-ab?E9(TgLzklk|9-A9PxV|k0X8$WF?{RyFHD%<|&sMGue*5AN6c_ zN*ZT}q3cJA=!Mb34}T5}3k7F9ieUlVixlT7AzDDV;46oCKkU(kJlt5N7qEl z(rs1@7+_caGO2*!D)=_2c)QxPJ{NFGMSk@mex*_@+Y7IUl?ROl{~BqH8dJu~ERR-SWx0G^;cv(DebxdxKRe|K*?)V3J%IP#@m{%dypBZU^3aIev$rFD_kY?r?JP{ zvnQQGy;4(3WpG3;R|S8-v5YGfaMiBV^<@QUQYCm`k-t(}n+sJ!#VOPfQ_F;Fy436X z;dadrze7!l9)W(=L5$qApAAL` z`cfIw=(AdX2}-Z?my!~seoqtuJbYa^>5>gs!$I6O1= z7^5!5_ss;@6=>H;1tX9*@z*fs(1Zn>sbfK5ut7X_PeY2=p`{Vo{D~6$tb|1)W=*vV zHS$?9Y{o?{xy?Kh3%^Wz#pZ?`6sHaC2eu)Krb65_g3$QxVr-`@+EUEHH&0TmJW!B= zkzG6k_8rz9fC|NT=-@G8O6V1B)W&9Su+B4AzkH5F!Hb((+s`Q*k8rmOHour?)CZt5 z5U>DoEEvZ3sze|Y-_?m}V{$I~PX0D9DBd{5Rb7^qI{mN3_cmmh@M2Q zCaioJ#Y7JFF2OWVQ1m5=*!Dj(#iWLmV^%8hToykNHklwcuAXx6iaOVh2DT@hmz2e^ zP@;@C@L+m3iI^xhVe=tTu0;J4g<_+A?t;GYCaSZ2#J30I-Wa0%_h=HNefnW89ZVI& z4LgW&15pi+$opsl3JF9Oop|Jhk|O^hUMY%J%6R$eO1JXVI@aab9e@yWwX6GcC?iH9 z=7ey^cxTgpv3&n36QJck!n2R9ya6d=TU)}ocXJ|_-}CTzF_A|Ic7W}GDYont6K=ZK zg4M*vl%a{)FPW1kI*1(%Ayj7+G2z5R#UVxf#s9*AHONgbMTZf?kF`Ak8XuuY15RF1 zE}-iWY6e{#{_OvguCS%TMKauc2RHKH(i7XzI4v6huTX@C%Z zGc)7Y=_*O5H|S$K(2GqPAw7#6g<2oMrk*nBjD8)H1-c~@Whw{eMQrtU4G!?Xq=#Rl zhkT?Bu(gY5N;_pmMW~ufl%$%PvzA#ku`>hK!Iz2C%GGqD1St#k7B+$ygd=M_mNqod z0_^lEm~!zIaj=O}JTQX2b!c52e#2avVEVpfjV^VcZ(p)of{4WQB3wfdWfnc)l#n(P{<5O?$ zo-O)L;`WAx)~*Hj@w>r$ZV4|Z4kz{?Z?9dpmmbtU(~$@%%E-oSisAbI*?vTShC z6@1_Bel71t-n1FUtkUL1dkaDtzft~1`Ap~R{)IB%T>d*{ZIjl?t2gqN?M2^kz2SPt zUOVYpwimwsshgjA$6oU#=ff7EaN`FX1$Xh=rHjQqZxLf9C4#&bxKAm8fg}hS}hyOf(4Kr*n7gsG6H_R6|EPJHnnHQ41{g}5e$(!uNOz+zDOO?&@ z=$D~n#g@sAsk$5R)y_{AwagAJ`S#BH_9k2Q-M(_C_>L`k_=RM_i|@Mn->+!UCEPL> zn(IsMKA9}&dDnI7VWCh^_Mk+lZ%CT!C%UK3{draWyuD)5Jmp`ms-5wF{nPXI%1PT) z#n0SDtm{jr(NE%iaRbcrh!1IgtNC{Mo%kPp=Et9z-{DVQihN))R_BqW{m9fXs+(Ie zed!%n{UoAJ%clF@an-(h0@1O#Yo;Bu4ez?z9##lq!-tiy(=$)8vUz*iqP^nhWmPlQ zZ@b=deKT)b92t* zi8t$JTnptb$ui&U*bmR#*|o6eWO8>8VrWho@_`W>ASy%#l)r=!#tIE3&{XcQ}7cE|^o-In2XJobqF{@GyCyR$20cDDTPA zYDBfFFyhg+X#<;N(nY`8R5(GgFzcRMba$?^EGSuYz?A3GTDqR~W=^&kS;Vns=~J}Y z>36L#!l9e}FfL^%x6ZCZ9LoN*Ux2E>W-$Y*Fd;(>l+kXgKWdQ0G}y06y=1ai@B+{L zj!)DIibc+nyOaxMH!ocVyLY!~y$iLsM`@Q+gCficP&;!ue4%!FbZzFUZ9Y`n0>G}& z@61$1`@ZjY=u}Dg4JbW=Jw+1KI{5j<+)!W-zb~f?{j=?*#zf>P4Yf~2D)e)*F`xXNYKPI3$&5M zniZSVYd#kr8{uxI(N^>$O&P!>WO&qx3vFRz;B6FBMLH_1#p&zK$I6n&3Z)~z5L)sW z>dQYMGlViQ*H7Gj3b$--M?b40e;Q9y)v7$q+o03c)D@A>5>KI_z}7l0-~M*V?EdNY zZxzm*`ew;Dc7JX6@0KX*9amKwpqxQ~SGTim*_0?*4IZ*8zrr= z9jAI(`LN>gtV>iJZK`z{i`dgbr`S33`q97T{5?7A$GLb=k9%-3K z@fWz55FR49pzvES%sFl!UaUK`Q263}-J$8;r6VuAbL55B4^17y?R?>j3$A{0TESX# z<@2`k>F}bh;k}xM1y9#obP?NDBAh<*4fQVP(O<6-`+Z3v~wYM*F@*f zY}jxPB47>Q;${Z15#Kv_+w!}|=5{Cd^)2TYzg~8;Y@U!7Yuhd4Z%=x5++GJ6(tXtM zkAflR7_Eeolb7_=W9gGE-%QMmB-?tEd1n^wew{++%AM%=b598-JKdGjFU*cFxVKE2 zvFP3Vt97tAa+>#+W|Fx$WsvX}y?9I#XI=zaz zvs_sE`rey+lLa-&n&<8m{ZZAAtCFXr;w9t{hsi1w~RgH z@-C{V9b|dJz2#8>&J&nKjhb|nHdEd;D^sOzNO^~{$-90tra&QYFj>bD*Ilev=ExHO_cKSCVC9mPQL?Zp4M376)FeIy3TVQ(ZjRdfZZIc@TDtGe z+Kygohpm@S z-ZNI^>M$c?azkXY0cAMZbe| zOO|7=gR^f&^N8G~XhC_17g*oqExb?JDDDvx6nue`Hu~;w=wv`zW+_iQk#bg@i)Hk} zL)R4(yF5VYh$PY~GH~oFuben?f8);EJMUau+<0;!uV;Sa$yxJjzjEVOrsMbA>v2am z$DaG1lBrKGdFsH6+$Ch6oZh_P78z%Hb@#HX=(UqKPLc=DOn0)febKdf9&)HOuX*>}y^XA{XCXOqvLUwZy>68|q=PM#e}R*o*Z#=s_*oE7uVis`G1 z&N_XqPRupVl_z&e=`ctsn5+yfx`x;{Yq3hpJ?C01WZUvq%l9__&~iJF;VqkdvHzX@ z{R`XAEwrCssyVOAIa#~scJN1|-ygj_aQ9@gvVX~S?qQXXU;jWv805sskAAscaG&CV zTQ`e4&6#T?pc^vX(arTCE z+Cg&$Aftvy{}pOypdP+Q+N*2~48|Ui6@KVa5?^0&NP;?x8>J}ooE5KeD;e09wFChi zAlunfvDL`2DFABu#B_Fqh~_*)IxO#-l>xeV{#b?!qk;~}^{gpDEOW7zcHUF#=$2xV zN6ME9wkc42Y^w{UxKtz+>r?h-hfhh`k1D-#Ef@*)EIYEXk!sD{vzncr*MbLia_9RQU{Ftkr!XNb8>Aiz-Q!s-K}pYIHf@mDWE)&b7d^x)pc^ zA#CNbS$PntKV6^Aq?3kw5Ise^jv<|9L`vtTiW3{4nQ=nx|ARHDnvjST$K0#Tc87+0&#;Pp5@bdcB%=HV-Pv zt5UTp;XHh4`vFsa^yqCZ%Cl@u7( z9}|M8BipC0xs*!C3Td@ZJ3-f0^efaQ?0MC4qJnx(@|`HH7Nlp$Nv)HbTX2%~#TK=0 zaQkmn1yME4F#O;$`fKASgYRSDvq`N>&o;I6^sm}-C{WfAjOtK2teIhBkF8z%C#>Cy zd=Zep8OHSbc7wE~R2Z?3V1Y>;u5Qha4Nxj{Dw;0+Horx!X%G_?fFsRZu%myMdX*0F zS78F9?)Jy#xkK8SZM7NvjW_UNTY2Ek9=v&O<#Sr7UT@CU zL1C}oph1Bvjr-8T=N?xlwfCz?RcF4;{Rh?3)OS%w{~`5Rk9MU#5ri~%#G3D*IQ-~2 zqF$wc*$B@9fqp3P{6JWqLE5iM2D=Oc*rRgXG@$X{s7pGK9s1PS!CC6}UTloi5A{pQ z(JUQQ#~CPUt}@BC6DEUPe{K&ZDOMgmr&c~mFE-k9diA^hUiF!Z0csB@oMAg?bS`K# zTCPxk%~v~`*Wagp-FN(Sec!sxWVXgZ-fZ%Bqn^IkGNxqgShm%Z@gzsrm&1gZn-==Q^mW*{jODnRL z-SL>R!L5QXGg_6#cJN5hE^`m*jVlhdH7DG7A6W4cx9WRl3>kdVdmY6Ku1?o8y!u%> z@BXm*w(o|DktorwVhazn#tLJG^TIe{R-`uir}h+!U|EUTooYTR=F#H2IxL5la^)|2 zUQ)|V|Egsw7)n#ggrShA2t%WV(cE-@NQZUwQ}aiboS0-gqI*-Q)&W%OsNTS&%juYo zCXcGMk-ESIP-_)ZYCh`k(`lD<{JbEYz+X51PM#N9&Cw#}UtAn5VZWu(HK1>0_$%*^ zsO4g(+%#X%+69b@swvXHYA?XtXzYahB7B1j{1yI+?5Xv_jX{W3`YU;#u*i@OMtNL4 zv2^b0RexN)^5^znQ?L6@sJ)ERvTzAJeNIiA{>4*_kd?<~D}Un8g=zJ$(%Wm%+iAL; zSosZ6c>h|pH<-cIhBmD8ul29fyDx&CuIu^q>aW#)jaK=K{8c@_vO2{sni?gBZpX4atMgkQ?Roh1GefqlfDv{_>tLu1@2x zQX$4l%F=4-Bs26DF}hXuv^={0BOs6!deU*qM;Xbmq^0?R@+<`p^&4sp`%WO2SJk`p zujG?@RPB}dzl+goc)N=k&@J_^@mFU&Sp!q2gX2-UT*b@%iL@AA?q60*f0l8SZLF%} zB3kpT5{loc_ePJd_pjIb60HFLsPV6FrU??|AX*%)_19{tKq0~Lvd|5GjXr08?W$br zbae2PzlPCIDO*nL&1rwF)C&rG#$R`Sw>~e)kEeZUI*L(IqAyv+I#5jqwE7n5tSU`a zfp&F)`!PLc1#i*S?Rhl^^iW!JeBo@7bX-5$sg{qC#9#CmNhh9^7Vo?D?=|RxUuEb& z{VP*4*qqb5&FJcQ9;r|2^Icg-9fmS1g6`XavxqN{E}UHhxapz{bSVO9n4c|HVe~lO zBwf^{F3FzSU#zto!fQsm2h(3azJI0Q`QXa8`en=f#Tj@L0^STghA$8Q^Ty8#%%l=7 z>HFqvxzo5wQF=t*j)9lm;^K9QFGhhg&v|0&Z(4D>%*Zx|bJBP>L5D9c!(TX~kNxtnEN zi9%D_l7(4=N+FCW__dIG=wFBN_{Y0UYX!J8jUzzAiNzKMv6~R>qLp&fmP*)W3h@WD z(n>64SJ!?^v|@{AEOI(V+cHPSe8Gz}C&;3au)bF#ZR3ocD`=lmg|UraIh)Ifw(*#O0;SlSh9slcd=b3X-g`23z4T8Ii0a}=z{WfxM+t-h>@4ahKrc6yyF) zN>UQ{E^+-D8T`$S|cOYTL5u=Fa+o!7|G z(t=~ufEnl^;_+beO=htz6tQ*0-Da|sT0vAD>i zp|mW-Mm2T=RX|4is3k9I!#?HTAiPREcu9_ohR6A~SmDdqvT-P;8%iilX zrEG(jBH=+mg=RTF?XxuT$7RtLNJk+A>vza(F5(->}mjQDaU<-(1E@+Bjg_!PfPi0Dj=1$ktCnI;*gKlo)D*>4IF zP{TBwxvvxV(!{DSY}|tu3d+in+)uM~6j09Mo%uFW#-N>X?@Y(xw?8u*N)~zHQTDr^ zQT#IV3k|)Usp5kQ#6X)4X>d$@K>-x&K9z)Gi(ah6O+#2)7%}~c$WV;AOakW+_=$c8 z=$W%E#;?*Sb$`TNaK_|Kf`^9UCrd*prcqLiyF|OPWEfRIT{J9P(YocgH+3hDeDb=J z+vw^_u3Cx)6IF&sBA`0)nCW|LscmFyX`1lTeIRv7w<0(`)zFnSWr#DdEw^l*kV} z(MF&0q=G0HZQ#j?8If}MF7Z*5@F=54`JB0!Q`oCVVMOWY6x6D+mISRIuds@QXO#9a z)g38YpL7)4I2}%`Rpb^CtJKHHD<1+76PJiolkngSV=*AO*M}1!nU-@8)B1KXM!*s+ zjq1)`Ae0RhKHVUml(g->#N$W1I{5R}x(2TjZj63Ol;Jy-fP);-*POdGz^A^QeCkyy zLX}R~RX&wiPa+S%Gm1sFXtQkMnxhb$!ict2whizm3V2JE6mSv~GhOZpq1$PIL%m6p zZ=mIs{A>e9A?DD<88HWH5;0dd*ZLgtP(ADk6<2xK%TcuW5z0(c>E=^^yyebjY>@qK z%icfTqEP8I0^@@Su1s#;70;JoX!P z43<%E>IofB_9i@xAfqzsk?>>&UZXQvKsI}-24Ev`!lU#Sk1y6g;UWuv!WyLJjjxPY zfzm`RHrAo=I2K?_zr<@Z!68u(d7k|G2|tzn=_!(cw(Vi@0DO6X#zX^T0^+@JbR2_Z zie2C_6c8<-#Vs>Y(YteJ_b-UwcFQhwEMG^!Lr865i?=0m&=s^O0&Oenkf0v;_a8iS z<^&BYEV;lJWB}NPWQ=#n)x#kO)uOT&rdSsf8+iI%4T)`zZiTHX&VjUCC#ez~_ zRExi$6XP}8DWM%BC`Ri^r1B3O?cOgvKhSZux1;+sHo-g1g2U0vQnr)j$V zEjqnPr#I;IQ#$>UPG7?boe&EYC8Fm_gWo-J>&W+BSZIE3xoXGl^*?I-e&bJeF6``1?l_q|IhtHMhIPU%YbXLb zYIt8O*Ol}gxZCx>Xjm6DC<%;~D+YviTb#UV$u%32n>&*|XO`E-wJfgcvT%N9{r6(G zH~mol(JMcA?G?-JisjtX4{QtuW{VXIzq$&R zoR#y=$_3}Ti9@t|@{t=ymR#PX%Zud^uo=u^nRlY&{kE;Y-}H9VQrp4#wu5&LF18(6 zD0yMN?Z|BLPrLrO>!&Ba`st~0S`XkiKaGd;B`-`I`ohWOn*Xn?^N(rcJmdHs9N0b^ zW9+ky!NwQ}m|unv0tpZj2Z)JV643%_Lx{((@Vf((9}}U2wUz4rkYdtxm?cw=wNfrx zq`1?hFfG!ER4G)N)Ysfy&LmW@HPW>IAc@6NY0{qeaM`9w*7xq-_q}`1=kt5_Ja^CM z$-iCp+cI9;vRD=}eq9tf`1-X=flG^J_~q#q$OtXR9yvMYp5=}uT+J8OB4^#w>nqH+ zmTxS=n_c|L^QhT}v_BCBhNXdt$iRfC^slAro~5PF+ppVM=C*Co#;29ddiU~0AoG5K zs(>%0H_m!D6!iO(5!w^Jcz@>iGd%72aPq^+_x`6uQg~Jsp7qqskQX_RN~yq+&!3$;8&gmaC`1#CF>1+`EX5H^F)HciEj^NDBw`uiEn@*~RBc&RJGiVR zQYx$w3TrT|Kz~l?y&(0DMS90X%{yxbCTh%O-J&r+K<`jslE`++OIl`TSm982Ep&xW zBf3POO4lhB(pc|L-E4=ztf?=K3bq5tgWt_Y^aX($*sTtA-0EEDgrjG_W9My$k+lZV zwE|Taua$qh&?&As+;@NUtI_cJ<(iegZ%4ivS?NM_r$C+CZFg_q!rSOz8?v?|x;R)1~i);AZl7B`_h1^G=|&yBlB!Qs z^`XI0{ObC}`I)&H09~wE?CXwV&WjEW2#&!QiHQ#Fnqx4;#N)IiLqWt)z#9rBLq)_; zAwU{&L(>lAwKN8)h+}XYw|_w-W~2tZfKxcsNG^L($60i)o3DN+qNxnDaG3z|5H&eC zcTifeQ`BS!ngHPdf7>+k%~FnDu6%Ul;SoNgRn)Zxli^4C)U&BJhL~IS1-qbjt`{F( zDq8MaX+k}NeDP33;|#b1)1l7)-)MYQ1~?z57fuW6;*DHqz`F~Y7BqHlSaP`|E;sLL z_@@kBwnj8oL6fzvNxkO%2-@u}>w{H-#<8O$EcUtmtb3=Hp!>;HRo`|!?D8+Cfc4-y z$(y|p?-xi~%PZvd#}=DHjZ*1RvGnMCWw3IwDWE|5W6yq!+Pm~c4cE=O!paARr89rB zE@{z;A*3A^lq2!LzJqrS5q!tYm$dSQt$gkqDCbS2Z4;F3(BMTg5fr^WO1_xmAd%4m80Z$CmR#m1Nd#D`76!2xAY#6{_dd>yU;%?=Bs#;hl}k zeUC;Tj-s}=q&8oq&4;@CrS40S?n^)fNz-o{%+He&($cqq3zD;?B2pNj&6>3@wM1r) zl;K*t{thh3$11>!ytPq3c7Rjj$fkR2=YZ>?;XYh6;XFA9e&9H>zrG8ba+i zGT1)RQpEK=POn%d11I4uj_NZ)l`I)@vW_nsxTG(<+yq-6PG8Ccx@;x@M4Cu*)SS)g zKc8kBk*Scg{-Ga6ei z&6+s1=%|D%Uv@RBej91e3d+vCwLQP_ajnQ&hUjvE+P@LUnPjtBD7ze$H!gQApN^mD z@^E^yovq_-<$JyDZU;(hd0QE>mLnQpJ2n)C-M+sxveV%avB1qcj-%{`<%;Fr6+oyA zh%E!+i9s}SMLcl@{p>1gnnu$0(JcF3ccJ%^@GHv*Z4!tsQM&2IgYdwg3=!~IoMpul#fjLffMk90YUn*WG;x9 z3y{glC2&tMJYS__)G zO?_5qQqt!loU{z~AgBuEOBqFxj3Q*gl7WR5-1A*Xa1XigAalB6l!gkTF*yX&x_jgqSCjCQ>kSFn@3RMo{+W~lBsRtJazGn zZRkw5N&@>4Rm!U zhso=3Bm1b-JYGeus2HpVG~>0@H&im{YE8~WY7%cfI6-IxQ!-ZT@lq&~Xm=h=qK8eO zYpI5bB;{H`GS<@LB~Y|(ajPe2`nx=L4J>WgapO(Ood}b^YLc*;4uk$p=f^^gx+%&vFmwTQ93Bo(E?})qC4u(i z$(8<|aVQ(={V8bA7@o(tJ<#z=td)3vyzc^T3KqdsvOKAP!iyL2CVvVFRo#UWImrCyMFvL*oMf3xhHxsCjmSLj_Hf8IKE9b40lb~1b@H(6WkWN zJN>dXYQuBapbQ>Hf~#77@(5-Qejwn-zDzk*-w)ncZ18~d91s)!zu@xs@pi6b$-#@V zjfvk|@w>5jv7ExvhQ)_v5X&%@c`UcE+{GfBhO&Js+lR8<7sC}lVEGponIa&Ym=4?} zrku&K{JT~vHfsF%wWcQ&Fc~k(j~{j;GU1&wuv+WDU3h7x zv170k=Skp&C#ZK4(#fM+1jvg-Slp5qtKN&$mG4Fg*?BpQ4-Px#u!L-7SqZ50GW$+nn|2gA*Kgjd`u^Ez+*iL t1!Q)tjfAN4m bool: + """Whether a ContentTypes.value is a text type, matched at the boundary. + + `startswith("text")` also matches `textual/example`; the database's bare `text` + oddity is why the exact match is needed alongside `text/`. Same rule as the + app's ContentTypeHeaders (ADFA-5241). + """ + return value == "text" or value.startswith("text/") CONTINUATION = re.compile(r"^(.*)-(\d+)$") ALL_PHASES = ("retype", "renumber", "migrate") @@ -104,6 +115,17 @@ def _init_worker(dictionary: bytes) -> None: with os.fdopen(handle, "wb") as out: out.write(dictionary) _DICTIONARY_PATH = path + # One file per worker, so without this a run leaves `--workers` copies of the + # dictionary in the temp directory forever. Survives a normal pool shutdown; a + # kill -9 of a worker still leaks its file. + atexit.register(_remove_quietly, path) + + +def _remove_quietly(path: str) -> None: + try: + os.remove(path) + except OSError: + pass def _brotli(args: list[str], payload: bytes) -> tuple[bool, bytes, str]: @@ -139,6 +161,8 @@ def sniff(payload: bytes) -> str: return "image/jpeg" if payload[:4] == b"RIFF" and payload[8:12] == b"WEBP": return "image/webp" + if payload[:2] == b"BM": + return "image/bmp" if payload[:4] == b"\x00\x00\x01\x00": return "image/x-icon" if payload[:4] == b"%PDF": @@ -262,7 +286,7 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only # Decoding every row here anyway makes a mislabel sweep free: report a # text-typed row whose payload is recognisably binary, whatever its name. - mislabelled = sniff(plaintext) if item.content_type.startswith("text") else "" + mislabelled = sniff(plaintext) if is_text_type(item.content_type) else "" # A stream that decodes *both* ways is one the compressor never referenced the # dictionary for -- small, already-compressed payloads like a 1 KB GIF. It is @@ -409,6 +433,14 @@ def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int ) +def table_exists(connection: sqlite3.Connection, name: str) -> bool: + """Whether `name` is a table in this database -- checked the way WebServer does.""" + found = connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?", (name,) + ).fetchone() + return found is not None + + def content_types(connection: sqlite3.Connection) -> dict[str, tuple[int, str]]: return { value: (type_id, compression) @@ -452,6 +484,15 @@ def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> st return "" +def report(errors: list[str], notes: list[str], limit: int = 30) -> None: + """Print what went wrong and what merely deserves a look, to stderr, labelled.""" + for label, entries in (("error", errors), ("note", notes)): + for entry in entries[:limit]: + print(f" {label}: {entry}", file=sys.stderr) + if len(entries) > limit: + print(f" ... and {len(entries) - limit} more {label}s", file=sys.stderr) + + def human(n: float) -> str: for unit in ("B", "KiB", "MiB", "GiB"): if abs(n) < 1024 or unit == "GiB": @@ -460,15 +501,20 @@ def human(n: float) -> str: return f"{n:,.1f} GiB" -def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[str]]: - """Retype rows that claim to be text but hold a recognisable binary payload.""" +def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[str], list[str]]: + """Retype rows that claim to be text but hold a recognisable binary payload. + + Returns (retyped paths, errors, notes). Errors are failures of the work this + phase exists to do; notes are observations that do not make the run wrong. + """ print(f"[1/3] retype {len(items):,} text-typed rows with a binary extension") if not items: - return set(), [] + return set(), [], [] types = content_types(connection) counts: dict[str, int] = {} - problems: list[str] = [] + errors: list[str] = [] + notes: list[str] = [] retyped: list[tuple[Item, Inspection, str]] = [] before_total = after_total = 0 @@ -477,11 +523,11 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s item = pending[future] found = future.result() if found.status == "error": - problems.append(f"{item.base_path}: {found.detail}") + errors.append(f"{item.base_path}: {found.detail}") continue if found.status == "keep": counts["left as text"] = counts.get("left as text", 0) + 1 - problems.append(f"{item.base_path}: {found.detail}") + notes.append(f"{item.base_path}: {found.detail}") continue target = found.sniffed @@ -490,7 +536,7 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s extension = item.base_path.rsplit(".", 1)[-1].lower() if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ and not (extension == "mov" and target.startswith("video/")): - problems.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") + notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") retyped.append((item, found, target)) counts[target] = counts.get(target, 0) + 1 @@ -533,12 +579,17 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") if write: print(f" rows inserted {inserted_total} deleted {deleted_total}") - return {item.base_path for item, _, _ in retyped}, problems + return {item.base_path for item, _, _ in retyped}, errors, notes -def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[int, list[str]]: - """Shift -2-based continuation numbering down to the -1 the app expects.""" - items = [item for item in load_items(connection, "1 = 1") if item.continuations] +def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> tuple[int, list[str], list[str]]: + """Shift -2-based continuation numbering down to the -1 the app expects. + + [select] applies --limit and --path here as it does to the other phases. Without + it a scoped trial run -- the first thing anyone sensibly tries -- silently + rewrote every chunked item in the database. + """ + items = select([item for item in load_items(connection, "1 = 1") if item.continuations]) if args.renumber_scope == "retyped": items = [item for item in items if item.base_path in retyped_paths] broken = [item for item in items if item.first_suffix != 1] @@ -549,12 +600,14 @@ def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[in f"{', '.join('-' + str(n) for n in starts)} instead of -1") else: print(f"[2/3] renumber all {len(items)} chunked items already start at -1") - problems: list[str] = [] + errors: list[str] = [] + notes: list[str] = [] fixed = 0 for item in broken: note = renumber_item(connection, item, write) if note: - problems.append(f"{item.base_path}: {note}") + # A repair this phase exists to make and could not: an error, not an aside. + errors.append(f"{item.base_path}: {note}") else: fixed += 1 if write: @@ -562,13 +615,13 @@ def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[in for item in items: if item.base_bytes != CHUNK_BYTES: - problems.append( + notes.append( f"{item.base_path}: chunked but its base row is {item.base_bytes:,} bytes, not " f"{CHUNK_BYTES:,} -- the app detects chunking by that exact length, so it will not reassemble" ) if fixed: - print(f" renumbered from -1: {fixed}") - return fixed, problems + print(f" {'renumbered' if write else 'would renumber'} to start at -1: {fixed}") + return fixed, errors, notes def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[str]: @@ -629,11 +682,23 @@ def main() -> int: connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") - dictionary_row = connection.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone() - if dictionary_row is None or not dictionary_row[0]: - print("error: this database has no CompressionDictionary row to migrate against", file=sys.stderr) - return 2 - dictionary = dictionary_row[0] + # Only the phases that decode or encode need the dictionary. renumber only moves + # paths, so requiring one there refused to run on exactly the old databases whose + # numbering most needs repairing. + dictionary = b"" + if any(phase in ("retype", "migrate") for phase in args.phase_list): + if not table_exists(connection, "CompressionDictionary"): + print( + "error: this database has no CompressionDictionary table, so there is nothing to " + "migrate against; --phases renumber works without one", + file=sys.stderr, + ) + return 2 + dictionary_row = connection.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone() + if dictionary_row is None or not dictionary_row[0]: + print("error: this database has no CompressionDictionary row to migrate against", file=sys.stderr) + return 2 + dictionary = dictionary_row[0] def select(items: list[Item]) -> list[Item]: if args.path: @@ -641,40 +706,47 @@ def select(items: list[Item]) -> list[Item]: return items[: args.limit] if args.limit else items print(f"database {args.database}") - print(f"dictionary {human(len(dictionary))}") + print(f"dictionary {human(len(dictionary)) if dictionary else 'not needed for these phases'}") print(f"phases {' -> '.join(args.phase_list)}") print(f"workers {args.workers} quality {args.quality} window {args.window}") print(f"mode {'WRITING' if write else 'dry run (pass --yes to write)'}") print() - problems: list[str] = [] + errors: list[str] = [] + notes: list[str] = [] retyped_paths: set[str] = set() started = time.time() with futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) as pool: if "retype" in args.phase_list: candidates = select([ - item for item in load_items(connection, "CT.value LIKE 'text%'") + item for item in load_items(connection, "(CT.value = 'text' OR CT.value LIKE 'text/%')") if item.base_path.lower().endswith(BINARY_EXTENSIONS) ]) - retyped_paths, notes = phase_retype(connection, pool, args, write, candidates) - problems += notes + retyped_paths, phase_errors, phase_notes = phase_retype(connection, pool, args, write, candidates) + errors += phase_errors + notes += phase_notes if write: - problems += verify_retype(connection, retyped_paths, args.mov_type) + # A verification failure means the bytes and their declared type disagree + # after we wrote them -- the most serious thing this script can report. + errors += verify_retype(connection, retyped_paths, args.mov_type) print() if "renumber" in args.phase_list: - _, notes = phase_renumber(connection, args, write, retyped_paths) - problems += notes + _, phase_errors, phase_notes = phase_renumber(connection, args, write, retyped_paths, select) + errors += phase_errors + notes += phase_notes print() if "migrate" not in args.phase_list: connection.commit() if write else connection.rollback() connection.close() sys.stdout.flush() - for note in problems[:30]: - print(f" note: {note}", file=sys.stderr) - return 0 + report(errors, notes) + # Non-zero when something the run set out to do did not happen: this path + # used to return 0 whatever it had just printed, so a wrapper script or CI + # step could not tell a clean repair from a failed one. + return 1 if errors else 0 items = select(load_items(connection, "CT.compression = 'brotli'")) chunked = [item for item in items if item.continuations] @@ -686,7 +758,9 @@ def select(items: list[Item]) -> list[Item]: counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} before_total = after_total = 0 inserted_total = deleted_total = 0 - errors: list[Result] = [] + # Not named `errors`: that name already holds this run's phase 1 and 2 failures, + # and reusing it here would discard them. + failed_items: list[Result] = [] mislabelled: list[Result] = [] for offset in range(0, len(items), args.batch): @@ -708,7 +782,7 @@ def select(items: list[Item]) -> list[Item]: if result.sniffed: mislabelled.append(result) if result.status == "error": - errors.append(result) + failed_items.append(result) elif result.status == "migrated" and write: inserted, deleted = write_item(connection, item, result.slices, renumber=False) inserted_total += inserted @@ -749,15 +823,12 @@ def select(items: list[Item]) -> list[Item]: if len(mislabelled) > 20: print(f" ... and {len(mislabelled) - 20} more") - for result in errors[:20]: + for result in failed_items[:20]: print(f" error: {result.base_path}: {result.detail}", file=sys.stderr) - if len(errors) > 20: - print(f" ... and {len(errors) - 20} more", file=sys.stderr) + if len(failed_items) > 20: + print(f" ... and {len(failed_items) - 20} more", file=sys.stderr) sys.stdout.flush() - for note in problems[:30]: - print(f" note: {note}", file=sys.stderr) - if len(problems) > 30: - print(f" ... and {len(problems) - 30} more notes", file=sys.stderr) + report(errors, notes) if write: connection.commit() @@ -767,7 +838,7 @@ def select(items: list[Item]) -> list[Item]: print("\nNothing written. Re-run with --yes on a copy to apply.") connection.close() - return 1 if errors else 0 + return 1 if errors or failed_items else 0 if __name__ == "__main__": From 0348b4fd4f3aeb6ca79fbadd4190ff3a429ce511 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sat, 22 Aug 2026 00:30:52 -0700 Subject: [PATCH 06/13] ADFA-5153: Keep Python bytecode out of Spotless and out of git Importing the migration script -- which a test or a future module next to it does -- leaves a __pycache__/*.pyc, and spotlessShell targets scripts/**/*. It then fails the whole task on a binary file it cannot process, which fails the pre-push hook with an error that names formatting rather than the real cause. It blocked my own push. Co-Authored-By: Claude Opus 5 --- .gitignore | 4 ++++ build.gradle.kts | 4 ++++ ...content_to_dictionary_brotli.cpython-314.pyc | Bin 51192 -> 0 bytes 3 files changed, 8 insertions(+) delete mode 100644 scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc diff --git a/.gitignore b/.gitignore index af44d5bf1c..c19a6e0552 100755 --- a/.gitignore +++ b/.gitignore @@ -193,3 +193,7 @@ NATIVE_*.md TEST_*.md assets-*.zip dynamic_libs/*.aar.br + +# Python bytecode from scripts/ +__pycache__/ +*.pyc diff --git a/build.gradle.kts b/build.gradle.kts index 9c3dd7ee92..70b1567f55 100755 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -338,6 +338,10 @@ spotless { // and every .py already here is space-indented. Only the ratchet has been hiding // that mismatch: an edit to one of them would silently convert the whole file. "**/*.py", + // Python bytecode: binary, generated, and Spotless fails the whole task (and so the + // pre-push hook) on one stray file rather than skipping it. + "**/__pycache__/**", + "**/*.pyc", ) } } diff --git a/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc b/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc deleted file mode 100644 index cd20b49529d24d4708419767af2e6c7a8c19516b..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 51192 zcmce#GcU0fsP)Bm z`{tYXJNH%<3M902l6fVuP|K~moO|xs?m721<>fjA9M`&Ty!83k1mUmgMm;S2!tf|+ z6od&u65_&nK{6aQ95Kd?Rdj7UYKoiKuQ_gJzm_A`xK(*)i`&>Ud)&@`bK*Jd*AaKH zUuWFOeqC`F`^}B#;@5OE?}$6@KH`acssw3IJYO=G3X-K%81cjl&e?nF)#{~xs|Crr z*}(7PMbh4Qv9vE72@ zbs@W6D;43om|d@vN^o7uuB*m?D_)$F=fs=@Voc3mgc;<}Dq*Gtby^|))GyAeaYL2AUEm%VG0HsHF6 zU3;ZwT(_|64U!Mnt?ar<+AnRy-6p!j_svop?%LVA7HKoCx3FuUv=!Ie*mbM49oIY9 z^+stYu6MEPP3PLA-P_L!La^K-1iz6-nI4~RtkHzAYUW{D|$jCbw9;9rLzt7?5y(EW1 zV)RlV7K$}fQSnRRm>7|RA-P$+7#a+W$3kK#FnCFfjD^I(OM$WBP)xjbDZsNF3%wj? zZ(^ZPY^TE^ikp0>R{VN2B#L573$H|+k`Imq!lPn5f?UNoeRgp?E?$a^1jT@O_-JQq zPxs;0V?7;*TThLL2QT-AM?=VM#Ssl$ABhBlV&ncpo%>t1w6$&W`o!K#AwVw}LIKn~ zRyNvjfeLAjqSUbq&8TrCmL)?4R=ucijOr@J0@v~BMFvTH7K;$%u8Y?~GN2R*kHtp- z?UC!NtETn@YOo0)a$xW!@B%u(fuXPrRNo|?2p^lJz@h#h=vOsM<-*a-DnU&;?K`z>SC=qb-Ep4Kc@s6}?7? z-M~jNqi|w7dSGQDt!rbUkr1PdVk|rw#TduHi^hOsA!AsLb8e%luL~*~!GQh`- zK{4cyAV1VlL06x6nvqy>d<HY?@ef)95*2ke#6A%|LG(Glv}UbGB^ZCH$r1C<#hsKp>_0Wl6< zgnEsGRRuv@<;Zo`yHqYwWfoBDMm zG=%nyr~|$xz>89eVz36;0+kp!9VN1Q4blkx4udO%$3hCz#FH3KS&Zczt* zQ0-{25t-zQ(5M2cj+X&98bgE#qmXn!gZcQ}gz-6u&ItqgL$O^V#(Q{Bym%cq1XV!P z2S8KrE7Vs<9T@E)il7P!fx-AV1AZ8lBa8!Rv_xY8^l5B-7}88!se6EdpI)>oT?;ZB z3k-u85$PhNlta;o46M@AZP4WL7@o0lM4cgv7ks0Us~1>yLtuzNW&r=f5UA9Z_ytju zZ+NYao<7-f;6!I9qrojL__Bq_8Aj*eWf1F7{8|X34B%t=asDn|xC&TATCb3(MCf@z zY=oEx>VvvLNQtPVfiW{NJR}A##?S^Y!$CDKC6m$U<_pLlsDAL0jQ$uOrKXOAFNa3L z#GAk=R7ip&kcAya4!%wZ7k;#@kQLbh}{sg1%6;SK(64> zkBlKd;!srUm*+5!EHD!GRt&5K}0 zLttpAabQr6Kt~va7zD!XI0K{(?PTb*o}d#&s-1jj5L1flR04(Uak-!2D-)}pW!_7cW8>k4AaL(k)a@>1Z0l?;=$ub#m2pRpnPrg68w7h z_a4~^nkV*L2fG;~b&_Ff=;bJQ0WblriHth5euB)x$PbtXiI9X%3KP2s5gLjk@d3uR znCQ#6S<4}x!)r)617o1AOdp7KdyT0a0uWTOSS)41@Er*blF>pjTnJ`dn{fL`#KnZ* z7tR~Ee*sD~o>dB`43`X&K{84v$=nJR*=zX)nHy4;tAUa6P|5_FlClCKU?(Z_#Ykl2 zmVgRaF&;+)@fa0RlNxNj6afJbqF;io`^SQEBpf^)9}k8j5YJu$qQ_bfqKPNRdM||z zM_Rcs7Hb7^1~0ZgBPyp{0|QE{LjwacG<31k`4ycte`xyG5^0E$C2hOu&P6u7{ZGV12W8v3HfdjWC-lwtl2 z$TvL&nQ9Ha9DuAGy0A+d9EzfIk?}lAcEJ$|28o`b0@p5a*$JNw#(8e^n9I;iG#QS8 z^ef8y1xSa)r{!@_Kd$9D0?{bg$3ZT85qmoV+D=4TwFzv7O~8>dQ^^Snom+tm@; z6GA_3`O!n8aK!?S>Pi&Q7!lll1ViK?lKG*#*O2q0nI=b zHTg|F)XEiyUr-SprZY4B+l+eBfuUiHS^5bM?C7DfxZR7WvY@y(CJ4U&=Om8M4P{G-Zs$K&XOH5XVv$W_3t;1`ZuP*n9M3_kQX50V3EbD}>&c z!|{||4h=(?3CSt<0MurP-Y@9%cizYZ(hm~ z7jwj!gUfd3YdJS^rZz0v*M4Xc9M$iaRwT_uAJ_$p^NS}xck-(zXL{c?Z}>15ukg*s zF~Z6p6zp%e{J7nDz?AbZA^dC*$ka8molb-m@)|ll4fS6^A`SJ;hTMk&PSfij(9cXQ zyIYO=1U!cNH0~#ail-uf21*KqNNLo!3PH>&5ZzvjL9$*53mlDZ8M?hDxe6r^7o%=X z*-oDBkmZPsu}p2%(ImMJ_w2+PRLko1z-Y4M-nTnm%Xu|ts`l61|8(D3@PS~kYfi6e*5EZ|bOMb@)=Zw=poRb2tj*-iTJoZ8)XO zLGY;fY(?$?~R|AWn1J?rvUm_DmdFcDi5N zcVpjlVBT3ZQ}peMw<>0X-@Sb6@{;eldEaw)O7D6Xe7#G)7v_C0Ecp79&iNm5kc>X_V*zQBJMb|e zc}VWWh1bZyDc9gyK8(`}D3&NtB+*i*_{!Z#${MjA7ZZX4&&B$!x6R42J+}eX!%u-K zQ5J@`)O}CVl|Ze{l2x*`LZb27SKx2TrkE2_CfH+A)@x7&BiA@XZAZpW?TN1-QOwJ# z@FAF}l~|~idICRl2ieb^LiUq9dV#&<2!g6>2hG2;rucD@iPZgCI=?{`^CW}rsix&f z#x##5713u;_aPc?Nf=wM;P0Az0)?=?!`%?55)B6~Mo=f>0UO*zd}IkHg-R*9Knd4@ z2UhlHFH$m4wBiE&=`2{z$^F&X)V43ZGTr;F_L-~S+%;!j+}e59d^ec1pI)}RA6ZN; zD|na1`Y#+Y-mEl?$WhGT6WMtLAO{Q{S`(y zS4^8J59N?*ci@Wa>@<}P#nqJY-pnS(kwYb%8rXYl#(SIYy`8H0>@^1fR4eGTfk9z&--aRBBJ@x;*+9enzn*#nr>9ap26 z7`qCi5eOa4utU(0#fAnlDvytm2g2|e%#A_c&hB3j`Rz{RXw!6fL`1ARM&}!t;LmlG zJJk2lv0=QY(cjHR;Qw0wUmrX8r!PfA^v=xQ(OEj$*|}3D;rtgQ;C3?_+}ClSC!u_C zEp#!uvzw8=?iiu(ffH&u7e}KkIT@i6E?&UPEn%`zQ`U7ohdL8(SQWToI5O4>WA0A* zMZ97K^0LTw&yrt>i{i)b71sBUbNB;T(pjQwCp+5`jv?46TCYWhhT7DJorxUnA$xYR zxA$bi&Ynf$L&{Y=KEz%Mf!Jumqp#UDJduk~Gp{j5u!h^j_H6;}a!0^x;$LFevI|U2 zg!n~#a7d-}ME!J{2Bf#K@X(MvfX9ql`~xl~g!gTElk&g$RI;Rb(boK)yL?%_->_)g zkiMUaEZXYcb5|&@r-m18YiAl}Yk#|CZrftbPTiZ(qOBtRU9!ZxX!E}3UaSA`J-0}= zv48W*)U`$1y7z0FW)J=LiMhVT+TBUFn6#g`(}Yvz7a1j|^Jf)#&t0Y@PL}LhwCzdX zPrbBg6W?=Jzvs+NlfQqKIFp??f9zzs5>tboE-C(idSo2~`-g(Ta2^^Eok>=t-UPMHcAX4(sd;$RqOGtioFj5MSnqGvO0TBDT_>=Ln+e;3?R;fN6vJ2 z4IFsBx8pR#E5gT=AjF;{Wsg$+PE@=5rt2bN0r_-S6wA`;F#7 zTxbWHK!@ScyEsn>=Y+UH65_`5|0!;&MZ@A|$#~8pnd1~ zG1()PLrDk71SOwB)l)Wh9}cFRBk1gLOfn7N#;r{Z!MI8foMaIjfvpQSjGCof`Xni* zZZ_qXA>~%4$p(gCGLFdC6Xk4ZafJ*bEqSf+)46l0T*^tUYb@npHDaSM!w{kfIEF`+ zsQ2SU>sH|NEfW`~bC8ZTQYE6qhE|$hQ~3m8pMG-S|E6SQvLhYPU^zx38|Udk{=0fa z6M|wy=R6`K2Gxwj)RjkiY)e8Zr7^EeAqXDLtg1@Y`tb~eFffRyA5sqOGz2JB4aW0? z&?|I8ZYMRf3W}!R__8ozsS>)-a-src+1g?X8)W)`L&cOb`Zmf(ai21baj9ChN9OE? z3jt=IycaAW;e|Se*4tI z)*~M=p7yzdA{e;jaVpOQw-`B&DDV?N$^k-BUsRrPB?!wnTbK^7L zDx9x9Iu~5pdgR{LBMY@h7pjlG<35_SA5C|d$}FB4mE>zE;YsKtzl;R;s4i)Q>0k)c zp&(3$AzePwu|p|8=h1oGJ`Q2Xx<*G3<_MBovmw=aK`-Z?H|X!y3+FKyU|ArtdCR~c zEl=P&1I*0*Za}ex^Rb|RG*9WdF8h8{>r&Gm{7*LRnT#*FtLNR-Gu7|7YiCZq<8DaW z58c^>Q@XLA7}&rID~e-kZi3q_K_#Bdi3$9~0Ak>CfFzCAeKbt>O?%l-x;l8z4u$>Z zJR)WPaj+*Adj|6yjLG5)ZnwEo0EP;x!WgP43YMxcPj(#^_^|r7$QModEPmLylF6Ku zIkM~=-l143mk%s96%mTPLn(=e0I3Ir0QtfU45SY1FVNf;2bm-!E4*V z0H#uLfj7E|Va$?7sSQ`?WX36FCL+%9hz(#njE>|zcU}G*rTHSAzC;O&2OtGtdUyCzN~3tJYAEW())e9rc^%~K-xi&z+rgw1)OzqM=Fcl zK@y0W`ctwfk_D4J&eBWgpCr|5K(PQOm4 zH|RtVl)pkJQeEWVqSKpn`Z}F{gH9_j<#l>SrxgBThj1cH*~Br$<^Fuva$)hr;mOkr z=HmCtt6?gC z{YC_VclMIGnD-!Mn;|J$@ck=e;n@P&qMp8PC^-s^u;kWjaJi(51z5FH}#I@Q=&TWKmI6#VB ziIazz%d6s<$6R-qYiLkMel^!jilo7eE4glwa{%}9XYQ~R0Yly`K91sBkbTO6>AQ&h zaVwv^AO{>wU%+vQ%>}_@a11s+vVe219?ehrmXVKn8tEzXVrENkXgh6T+x(}rdW6w4!@X#gj|zJu9(Y^5Lc?khHatLK}LE;!) zk?6(fIpaUfip6COG8$n1`Yn3@f8s=}3Ct$QOFcr5+vu_RGG;_#gvXevq(kwbP#hx% z*f{+_!zkK6d$wK+|8(otJepXm~zqER$_S?RX*GQd|)!H={LaM zlizRnCx{S*Ws|m_J3UjSU)noy=x0v%eRtV(?KgaH_-5;Gmn^Sup6y*~**o8|H`%-| zxo;#{J&FuG5y~JZV)(!+xJssiuaDgvo4It`{FDBq^!!4{3(ITQf2V!++P~k8w5}aU zYT3a_2Mu_YK+u3=1aBj2>Eq|}h~P;!{Mn@($ssvgt%S5(>}EhGafd;kQ@$R$Wo$#j zRmx4iv;*Oxf!HWH#L6CFi5tjQ^z4P+xDieZ6f8-cbD-M&I8iiB742jPQO#K>nO2CW&1wo zH>vhbrfhvWnx}&}=C?DrQTF@^E;;@@m(XU+fR0UTXRg{#)>EHJmk>l6wC10-UxUK^ zY^|Zv9Qtz7+%vd~)fbq^55HEf-@5&WD52?@+T&#Ka_KNt%LiO7*Kfr+ugxO4(;Voj zxZ~+B&Q_~zWuUL}(>Z$!0$;`O1$%>630M}~zC)uFprcy;$d^4MfKKB;T9^ro+d(?` zvmYdHW5iLBV?XqhAmSDf!3YB*7#R8oR5lzCP=g5b9ETZ$dpPH=qmpLzI9L^hePdN?ev&Pl`1_)Gh)Z zxT2=EI8F^5Vxd6@zGbs}#Ny!!Br<6LK=3&jHB8x*$RJqf(>(fH=xq77>GXfmiA4Sk z9!^X(+UZq@m5XyU44l+ewk?U2Agw=XFm;}R5iX5XlwKF;92S~>_=NB>l;M-gv zHsBHr7bi-o6>u2c8j z#frQ!JU5i=xVq@Rmb71kt7t>}+~%eBL-Xy2mg+n1)pvZgZaQbNpyqDdMAru{VU2Hg zXrXk=yt!!N@Xu^{Q`;5`8`4>YZ(Ec8z@qzN(th#hRa@r5ck36bPR*MuCk{*vQMED( zJ^W&_|6=mR!DL5p(H%&I_UBu<=b`q2$^jlE5x83G5P+zzU{nU-#Yg&20Ey|DEQAEhm@Ds=gINeHymkX(9>iB&D^S z)Y){=C+zrVE1eEZe~j@U){hI6xE9|*?HO8s9P zgGJVkf%&rhHTj3~e@{<;_Gqj=U1D0eiMsu6f&(4FGsJ@Z?rgs6=QXD$zj+6x-N({e zv-yZVH3wUNS~$yInx?#~aPzy`&FVEKZ>|f1PEEG^ZT=j;!|(Lx`t$rS7i|M1=;%9J zk8{@)hne3BIDz6^GsBJC6SYP2qb|RtM-%zjC+y_+sAWY9`~^ekvr3_>3JQ5aPeD4B zdX*IhHh(_qto1wU42V(Bp+rsGP;*ExM1mTMs9mAIFl$Mr=`YkP$%Mr`%&L(M<<#(U z`peW5PcD&`Yr*#lZfjHDCQN->6MxSFz=#f??l|6Yuva|j3nK*ippRxt=~ta0J$i`l zl=)8zXW2tFaq)!jDyULzw;m@qV54qeZv5*5d)1nHgADEzKG;K|yI zXE5PWtMT&|estB6{|oNr|BBN~x`H?V{@&l&%L?}9rfi{?LxY&mNZE%%h{b{)oN^sJ z+1-1z`^vs$%0|IZ^oTkzWrgF$ zFk(#{>?tdZ_vfF{2dv*KD4hi{{*WdaET}^>BoYOSxnSZ*_m%t?NV%V&M4*}wmYu~* z&b9N-wGWJzT+aiCkY7IeDKN{5oCi*8j_Z-h?y`R95p3?yUi$n?5Awme9~KE>?SiZJ zjrg=_D*rYAr2mchL`Tw9yS%n)qGNf>wuuve-G5=;es1b<33v%VGna*Rpd(^#}Pm6?u~#U+Q}3 z7S0(0WJ>n!nzwr=a}aH_b!O|br}*`pn>o|o1yAjyW!dRjbXG3A@|Rpy^RB8Tc(b_b zXRUKq2ty6~-gQ0q(1yPMcP-h==k4W7_NsaOuicrnSG{ZB_0S~b zSCK8KV5;u*=9|r5@y*xoUvwTwnh$*R%RNGF9dn17&F`o;|Et)%IkNpnLI~=AX8h(B zodDf}FVip~0e}sY8iohyptVs#01#my067}BkS}oylPvH%*Gn$OEw?YTr>n&Z8rNVq zhDXO32Kfyne6KtZ_F>2t%m(4X2=ReW(IO|Z8bsmAbfpcGw6L-$leCzcr4g?lQL`lU zXf_`8DJJL)I-7^ae4BMCH704v(6I?XD-Ox%w?GgwwVC{8)$#vH;UikkR==5ArTBIs z=PUJoqkc*sEmK4y_;bODmw9fPqG)57U*joQnK@->P8s$kS`&<+@187qk6I;~D2G6c%J{3rDAPkAavqZr04)6=6k!C5^ROS!-U2Sa4phHi{- zVp4ZjTWS%Z0;137SpF8@GCZOw79u3>*LL67J?Wc^FKs(|Z`;wuievNJj=uHE%+=W| zbJbt##Lc~mWA8YQO&nYv0cIAmHbjYA4A1qpYif7R19gE)=eL#)yt3hjoq z59ze*sd-?+@B5Ct2NwLou${Z+fsL;0g1hiR4qZ8f67hkPu3Un(_+c)t`1sb17z{yW zMynCffs|k#0`@}a&N|Ld7<$c73+$IjyYj$@GG@BE^qtD)KLrs~5$zCUYSKh~s%P zA}d5KetWw?2d!KvU+5k6h~{=xa~cZ8%jvfvw-{(Fpez3$erHxSB1h8lL#g(m4pztP zrCR~FEXoC_CX^f4Z%uellE{jq?{Dy&9IaLy?OK=~`rWmnC9bXkC^OmXVMLC0J=SzV z)Mbjx|1YrZ`!fmyV^rK6br3dydfO9+``8j^ij5U>{bN|ANLkg176Q}75R!aI%Ym>} zWMGY@Ti8;u97@pZ>rU)F({pHlZ-=TYK(Xo7IKduqpT-ur5(CX)pQ!N!-0YP9Ga%jF zO%gl{#!rZcpc)~=4An&cN#ZM-ZlUTwN4rmVNWI`Oy(hJbG;*CvuTd!+YDJx}V3np; zc~@oa-+%l}$7!(ldyF>!J-uT2zDHO84^D{I3SjCsC^NoD@-=6Uc7%yzQMR$+lwIww zluhX>HmMn-!MwsAs7T!`_?%B_{t}58+5UCVZWseSLc`X3^?N3ROL;Z-@@nqvn>f5| z&Ra5<%$rM=?MIe#+*5`xmrWi1M*kcAN&AKmEC_A-&><9*-`GDLU#i+TU$t?f<9=@b zSH07nGi_f#HXB+j*u0Pn|C#0OyO*{fySM$=JMOOe?Z@WVZ=TsYw`tn?>v_1FIC9V3 zwOm%7G?z~t`OqcUJyVC4it6Tz>Sps7i#9Bmtb09jGxBap!(`{Ot7OTwZr-&nS+)O8 z=baskt`o50OCGh36M4gbSp%FTONCRElus1*t;UT?kG`fkB` zlx#2HB6_l7&mHUS>x=fIh!C6F^kvtpHdMT@7WHryC7qQ?bLB@L95x7^;|3Pj@#9^2 zow@c5@mv#i8_|DEVSdaSD#{G4oy7&_JYFQxBOqc(fO8z-cidfF(A2_9P^0kWws9o|uA_+qo zG6cbY;$JM6rwNl|_wD}M-FK^#_T$SrYoPf~x8sC?Xt93u$UEpWhOt^v?-kn)KAlMzehty)pIQ*DdUt}+lm2l67<}(M3zlvQzbhxolC}J2DomR5+~Hf62_3HmuIpAQrPbPei9L$SxQTOkJ{>VI8laU;6w!e28j?Z-mPZ{4 zHHiDW4~bMX2@hCHieaB(8-rqk$wr2UAELSP-{8a@p|y}9mdOBPjWPLSyk~&sdQ_TL z+YjHZy<7UoWX!exoi&dckSv150mk9VpEh#}%)w58Re`I3-S9}n!za;PzHZe@T1(_2 zQBI04EuR57ge{eU@z0mYc)E7=L&BL#R(3iFJGpKnkGtM-QWtlvRvC!}ja3BMr;ouF zEj|pohJjf^xUS*gmJqhcK(*i0ZiK$!Qmi2$V>+jGs==N5GiKnMdp@HM1|877TyESNBK)l8 zwuFSBca}%ZI0J{Xa$6`oR5j(G6`8R9$xh5$;TuGAgHWX1J9G|CS@X(Xf{xD`vaOz? zD#UuOt#VTJUDS-cb@f#1>o+r%7Bf?*t1){TZcamFbk@=_1T1H)<0dwfh^}F#K!iPA z31JQqC9*k0ThxNt!^eEzc}^KNmslDMn3V@=VD9_89?qnuMsqMcvnb(`#Sb%y3W03u zvPR4uen*2+*f8s_t|L%aVHC(PKG=2unUwcu@g7RDWNm|~unKOcOkp*uxdbs^$c!w6 zq$>`n@jyuCV09KP&0hD%7&TBL)Ox0zTiSJNa&K7nWb;yr4yDb z(wa4D0a43Ibf3NyrnPgqs}iI&lFYO?L#$AVWxZp{TJcP~rFA@8uok|$LQjC4UDVil zA9J2b)MRFY*m~v>!=4V0N3gtcVLQpqxdn%?JmoMPlHfOl8j;U01T=jaiN-84q?&1lN}tGWQB8@^MO(Z%JZ$GH zsH02_hr7=(iSkQydznsuLZ^>#f{#sFtR$KUy8r<>1?`qrBvsYkqx8xI0JGqylobp4 zVb&{Up*1EiF_sbE)+e#;C^A^$g<7(#ZQ0s z)6@1vo4B0cK6mK%PrQ9%Zr`1jq`N0+?}0+s(6m&){a*d{MaPc$`t3JIriP{iGx@J> z#m&UQdB=|BHI++iHY~2$aKklenA)^lv__|h{`UD~?dD|u7U+(iaz0g+Tz~AY|L)NR zkAKNCFz*>y@B}6;KXZC+Y?(SUz3Dg3&YXU?xN*6pa_Vey?dfD;?>$%V!yKWkiFE2* z)TgF?sk(i!x*biQ?3~`ODZ0-xEljNz-O@#7zX|lxLK8u4=JX;)*^Jh3 zAw_OQ!^dE)h46-#_eC#c)%g}grIi(kZjr0XL2i0Z>m#Ho8=SK=VgVu(h6se zL|2bY`Y&6M`x-5;eN6^oVl;|9gI-1cn^j`IL+@0zC)aJB>%6_?cEg?gWWmAzw*IFZ z{&>TZ>&&AZp@PR7W(+Jd>H|dlWD3kH1XjxsChFtdG(_n9R9W=_V4y8IWHD5bCR;Iy z#Ygco6=Qd41q&InL#yAf;}$;mb_|5w7zl3#$bx5@z=L~Z^BScV_=yT#dleyse{E}nT^*}>8V;mP zhr9&x3GDeumC$WS_V-w z?RAi`+=b=R_-J5Eu0~o0y?zFSKlf~yG~WjbT=~-0sYAcIXZp}g$D1dXy)AQv->H}i zOq-|9PM?~NPY)#v>+ZP@-#&Fae*4Is1Gignvuw|q?EP}is|L6W)Imb5yf z3e-NqC(we7I+CdA%25kuv#f9IyWM{Q5 zVzoB*h^tdcIV=Tm7|N;x-(-bf4{&JZAvxhzsxW$~Uf;Vtz?GGU*0cF=t@M!c7kq)$ zDqm{Yfy6;hM?Df-T5b5a@-COv4m}E}dwNujP?x!sr}m!uau^~LQhzd7yZ!mV5fA!4 z3nYOfTGS9mY)-ab?E9(TgLzklk|9-A9PxV|k0X8$WF?{RyFHD%<|&sMGue*5AN6c_ zN*ZT}q3cJA=!Mb34}T5}3k7F9ieUlVixlT7AzDDV;46oCKkU(kJlt5N7qEl z(rs1@7+_caGO2*!D)=_2c)QxPJ{NFGMSk@mex*_@+Y7IUl?ROl{~BqH8dJu~ERR-SWx0G^;cv(DebxdxKRe|K*?)V3J%IP#@m{%dypBZU^3aIev$rFD_kY?r?JP{ zvnQQGy;4(3WpG3;R|S8-v5YGfaMiBV^<@QUQYCm`k-t(}n+sJ!#VOPfQ_F;Fy436X z;dadrze7!l9)W(=L5$qApAAL` z`cfIw=(AdX2}-Z?my!~seoqtuJbYa^>5>gs!$I6O1= z7^5!5_ss;@6=>H;1tX9*@z*fs(1Zn>sbfK5ut7X_PeY2=p`{Vo{D~6$tb|1)W=*vV zHS$?9Y{o?{xy?Kh3%^Wz#pZ?`6sHaC2eu)Krb65_g3$QxVr-`@+EUEHH&0TmJW!B= zkzG6k_8rz9fC|NT=-@G8O6V1B)W&9Su+B4AzkH5F!Hb((+s`Q*k8rmOHour?)CZt5 z5U>DoEEvZ3sze|Y-_?m}V{$I~PX0D9DBd{5Rb7^qI{mN3_cmmh@M2Q zCaioJ#Y7JFF2OWVQ1m5=*!Dj(#iWLmV^%8hToykNHklwcuAXx6iaOVh2DT@hmz2e^ zP@;@C@L+m3iI^xhVe=tTu0;J4g<_+A?t;GYCaSZ2#J30I-Wa0%_h=HNefnW89ZVI& z4LgW&15pi+$opsl3JF9Oop|Jhk|O^hUMY%J%6R$eO1JXVI@aab9e@yWwX6GcC?iH9 z=7ey^cxTgpv3&n36QJck!n2R9ya6d=TU)}ocXJ|_-}CTzF_A|Ic7W}GDYont6K=ZK zg4M*vl%a{)FPW1kI*1(%Ayj7+G2z5R#UVxf#s9*AHONgbMTZf?kF`Ak8XuuY15RF1 zE}-iWY6e{#{_OvguCS%TMKauc2RHKH(i7XzI4v6huTX@C%Z zGc)7Y=_*O5H|S$K(2GqPAw7#6g<2oMrk*nBjD8)H1-c~@Whw{eMQrtU4G!?Xq=#Rl zhkT?Bu(gY5N;_pmMW~ufl%$%PvzA#ku`>hK!Iz2C%GGqD1St#k7B+$ygd=M_mNqod z0_^lEm~!zIaj=O}JTQX2b!c52e#2avVEVpfjV^VcZ(p)of{4WQB3wfdWfnc)l#n(P{<5O?$ zo-O)L;`WAx)~*Hj@w>r$ZV4|Z4kz{?Z?9dpmmbtU(~$@%%E-oSisAbI*?vTShC z6@1_Bel71t-n1FUtkUL1dkaDtzft~1`Ap~R{)IB%T>d*{ZIjl?t2gqN?M2^kz2SPt zUOVYpwimwsshgjA$6oU#=ff7EaN`FX1$Xh=rHjQqZxLf9C4#&bxKAm8fg}hS}hyOf(4Kr*n7gsG6H_R6|EPJHnnHQ41{g}5e$(!uNOz+zDOO?&@ z=$D~n#g@sAsk$5R)y_{AwagAJ`S#BH_9k2Q-M(_C_>L`k_=RM_i|@Mn->+!UCEPL> zn(IsMKA9}&dDnI7VWCh^_Mk+lZ%CT!C%UK3{draWyuD)5Jmp`ms-5wF{nPXI%1PT) z#n0SDtm{jr(NE%iaRbcrh!1IgtNC{Mo%kPp=Et9z-{DVQihN))R_BqW{m9fXs+(Ie zed!%n{UoAJ%clF@an-(h0@1O#Yo;Bu4ez?z9##lq!-tiy(=$)8vUz*iqP^nhWmPlQ zZ@b=deKT)b92t* zi8t$JTnptb$ui&U*bmR#*|o6eWO8>8VrWho@_`W>ASy%#l)r=!#tIE3&{XcQ}7cE|^o-In2XJobqF{@GyCyR$20cDDTPA zYDBfFFyhg+X#<;N(nY`8R5(GgFzcRMba$?^EGSuYz?A3GTDqR~W=^&kS;Vns=~J}Y z>36L#!l9e}FfL^%x6ZCZ9LoN*Ux2E>W-$Y*Fd;(>l+kXgKWdQ0G}y06y=1ai@B+{L zj!)DIibc+nyOaxMH!ocVyLY!~y$iLsM`@Q+gCficP&;!ue4%!FbZzFUZ9Y`n0>G}& z@61$1`@ZjY=u}Dg4JbW=Jw+1KI{5j<+)!W-zb~f?{j=?*#zf>P4Yf~2D)e)*F`xXNYKPI3$&5M zniZSVYd#kr8{uxI(N^>$O&P!>WO&qx3vFRz;B6FBMLH_1#p&zK$I6n&3Z)~z5L)sW z>dQYMGlViQ*H7Gj3b$--M?b40e;Q9y)v7$q+o03c)D@A>5>KI_z}7l0-~M*V?EdNY zZxzm*`ew;Dc7JX6@0KX*9amKwpqxQ~SGTim*_0?*4IZ*8zrr= z9jAI(`LN>gtV>iJZK`z{i`dgbr`S33`q97T{5?7A$GLb=k9%-3K z@fWz55FR49pzvES%sFl!UaUK`Q263}-J$8;r6VuAbL55B4^17y?R?>j3$A{0TESX# z<@2`k>F}bh;k}xM1y9#obP?NDBAh<*4fQVP(O<6-`+Z3v~wYM*F@*f zY}jxPB47>Q;${Z15#Kv_+w!}|=5{Cd^)2TYzg~8;Y@U!7Yuhd4Z%=x5++GJ6(tXtM zkAflR7_Eeolb7_=W9gGE-%QMmB-?tEd1n^wew{++%AM%=b598-JKdGjFU*cFxVKE2 zvFP3Vt97tAa+>#+W|Fx$WsvX}y?9I#XI=zaz zvs_sE`rey+lLa-&n&<8m{ZZAAtCFXr;w9t{hsi1w~RgH z@-C{V9b|dJz2#8>&J&nKjhb|nHdEd;D^sOzNO^~{$-90tra&QYFj>bD*Ilev=ExHO_cKSCVC9mPQL?Zp4M376)FeIy3TVQ(ZjRdfZZIc@TDtGe z+Kygohpm@S z-ZNI^>M$c?azkXY0cAMZbe| zOO|7=gR^f&^N8G~XhC_17g*oqExb?JDDDvx6nue`Hu~;w=wv`zW+_iQk#bg@i)Hk} zL)R4(yF5VYh$PY~GH~oFuben?f8);EJMUau+<0;!uV;Sa$yxJjzjEVOrsMbA>v2am z$DaG1lBrKGdFsH6+$Ch6oZh_P78z%Hb@#HX=(UqKPLc=DOn0)febKdf9&)HOuX*>}y^XA{XCXOqvLUwZy>68|q=PM#e}R*o*Z#=s_*oE7uVis`G1 z&N_XqPRupVl_z&e=`ctsn5+yfx`x;{Yq3hpJ?C01WZUvq%l9__&~iJF;VqkdvHzX@ z{R`XAEwrCssyVOAIa#~scJN1|-ygj_aQ9@gvVX~S?qQXXU;jWv805sskAAscaG&CV zTQ`e4&6#T?pc^vX(arTCE z+Cg&$Aftvy{}pOypdP+Q+N*2~48|Ui6@KVa5?^0&NP;?x8>J}ooE5KeD;e09wFChi zAlunfvDL`2DFABu#B_Fqh~_*)IxO#-l>xeV{#b?!qk;~}^{gpDEOW7zcHUF#=$2xV zN6ME9wkc42Y^w{UxKtz+>r?h-hfhh`k1D-#Ef@*)EIYEXk!sD{vzncr*MbLia_9RQU{Ftkr!XNb8>Aiz-Q!s-K}pYIHf@mDWE)&b7d^x)pc^ zA#CNbS$PntKV6^Aq?3kw5Ise^jv<|9L`vtTiW3{4nQ=nx|ARHDnvjST$K0#Tc87+0&#;Pp5@bdcB%=HV-Pv zt5UTp;XHh4`vFsa^yqCZ%Cl@u7( z9}|M8BipC0xs*!C3Td@ZJ3-f0^efaQ?0MC4qJnx(@|`HH7Nlp$Nv)HbTX2%~#TK=0 zaQkmn1yME4F#O;$`fKASgYRSDvq`N>&o;I6^sm}-C{WfAjOtK2teIhBkF8z%C#>Cy zd=Zep8OHSbc7wE~R2Z?3V1Y>;u5Qha4Nxj{Dw;0+Horx!X%G_?fFsRZu%myMdX*0F zS78F9?)Jy#xkK8SZM7NvjW_UNTY2Ek9=v&O<#Sr7UT@CU zL1C}oph1Bvjr-8T=N?xlwfCz?RcF4;{Rh?3)OS%w{~`5Rk9MU#5ri~%#G3D*IQ-~2 zqF$wc*$B@9fqp3P{6JWqLE5iM2D=Oc*rRgXG@$X{s7pGK9s1PS!CC6}UTloi5A{pQ z(JUQQ#~CPUt}@BC6DEUPe{K&ZDOMgmr&c~mFE-k9diA^hUiF!Z0csB@oMAg?bS`K# zTCPxk%~v~`*Wagp-FN(Sec!sxWVXgZ-fZ%Bqn^IkGNxqgShm%Z@gzsrm&1gZn-==Q^mW*{jODnRL z-SL>R!L5QXGg_6#cJN5hE^`m*jVlhdH7DG7A6W4cx9WRl3>kdVdmY6Ku1?o8y!u%> z@BXm*w(o|DktorwVhazn#tLJG^TIe{R-`uir}h+!U|EUTooYTR=F#H2IxL5la^)|2 zUQ)|V|Egsw7)n#ggrShA2t%WV(cE-@NQZUwQ}aiboS0-gqI*-Q)&W%OsNTS&%juYo zCXcGMk-ESIP-_)ZYCh`k(`lD<{JbEYz+X51PM#N9&Cw#}UtAn5VZWu(HK1>0_$%*^ zsO4g(+%#X%+69b@swvXHYA?XtXzYahB7B1j{1yI+?5Xv_jX{W3`YU;#u*i@OMtNL4 zv2^b0RexN)^5^znQ?L6@sJ)ERvTzAJeNIiA{>4*_kd?<~D}Un8g=zJ$(%Wm%+iAL; zSosZ6c>h|pH<-cIhBmD8ul29fyDx&CuIu^q>aW#)jaK=K{8c@_vO2{sni?gBZpX4atMgkQ?Roh1GefqlfDv{_>tLu1@2x zQX$4l%F=4-Bs26DF}hXuv^={0BOs6!deU*qM;Xbmq^0?R@+<`p^&4sp`%WO2SJk`p zujG?@RPB}dzl+goc)N=k&@J_^@mFU&Sp!q2gX2-UT*b@%iL@AA?q60*f0l8SZLF%} zB3kpT5{loc_ePJd_pjIb60HFLsPV6FrU??|AX*%)_19{tKq0~Lvd|5GjXr08?W$br zbae2PzlPCIDO*nL&1rwF)C&rG#$R`Sw>~e)kEeZUI*L(IqAyv+I#5jqwE7n5tSU`a zfp&F)`!PLc1#i*S?Rhl^^iW!JeBo@7bX-5$sg{qC#9#CmNhh9^7Vo?D?=|RxUuEb& z{VP*4*qqb5&FJcQ9;r|2^Icg-9fmS1g6`XavxqN{E}UHhxapz{bSVO9n4c|HVe~lO zBwf^{F3FzSU#zto!fQsm2h(3azJI0Q`QXa8`en=f#Tj@L0^STghA$8Q^Ty8#%%l=7 z>HFqvxzo5wQF=t*j)9lm;^K9QFGhhg&v|0&Z(4D>%*Zx|bJBP>L5D9c!(TX~kNxtnEN zi9%D_l7(4=N+FCW__dIG=wFBN_{Y0UYX!J8jUzzAiNzKMv6~R>qLp&fmP*)W3h@WD z(n>64SJ!?^v|@{AEOI(V+cHPSe8Gz}C&;3au)bF#ZR3ocD`=lmg|UraIh)Ifw(*#O0;SlSh9slcd=b3X-g`23z4T8Ii0a}=z{WfxM+t-h>@4ahKrc6yyF) zN>UQ{E^+-D8T`$S|cOYTL5u=Fa+o!7|G z(t=~ufEnl^;_+beO=htz6tQ*0-Da|sT0vAD>i zp|mW-Mm2T=RX|4is3k9I!#?HTAiPREcu9_ohR6A~SmDdqvT-P;8%iilX zrEG(jBH=+mg=RTF?XxuT$7RtLNJk+A>vza(F5(->}mjQDaU<-(1E@+Bjg_!PfPi0Dj=1$ktCnI;*gKlo)D*>4IF zP{TBwxvvxV(!{DSY}|tu3d+in+)uM~6j09Mo%uFW#-N>X?@Y(xw?8u*N)~zHQTDr^ zQT#IV3k|)Usp5kQ#6X)4X>d$@K>-x&K9z)Gi(ah6O+#2)7%}~c$WV;AOakW+_=$c8 z=$W%E#;?*Sb$`TNaK_|Kf`^9UCrd*prcqLiyF|OPWEfRIT{J9P(YocgH+3hDeDb=J z+vw^_u3Cx)6IF&sBA`0)nCW|LscmFyX`1lTeIRv7w<0(`)zFnSWr#DdEw^l*kV} z(MF&0q=G0HZQ#j?8If}MF7Z*5@F=54`JB0!Q`oCVVMOWY6x6D+mISRIuds@QXO#9a z)g38YpL7)4I2}%`Rpb^CtJKHHD<1+76PJiolkngSV=*AO*M}1!nU-@8)B1KXM!*s+ zjq1)`Ae0RhKHVUml(g->#N$W1I{5R}x(2TjZj63Ol;Jy-fP);-*POdGz^A^QeCkyy zLX}R~RX&wiPa+S%Gm1sFXtQkMnxhb$!ict2whizm3V2JE6mSv~GhOZpq1$PIL%m6p zZ=mIs{A>e9A?DD<88HWH5;0dd*ZLgtP(ADk6<2xK%TcuW5z0(c>E=^^yyebjY>@qK z%icfTqEP8I0^@@Su1s#;70;JoX!P z43<%E>IofB_9i@xAfqzsk?>>&UZXQvKsI}-24Ev`!lU#Sk1y6g;UWuv!WyLJjjxPY zfzm`RHrAo=I2K?_zr<@Z!68u(d7k|G2|tzn=_!(cw(Vi@0DO6X#zX^T0^+@JbR2_Z zie2C_6c8<-#Vs>Y(YteJ_b-UwcFQhwEMG^!Lr865i?=0m&=s^O0&Oenkf0v;_a8iS z<^&BYEV;lJWB}NPWQ=#n)x#kO)uOT&rdSsf8+iI%4T)`zZiTHX&VjUCC#ez~_ zRExi$6XP}8DWM%BC`Ri^r1B3O?cOgvKhSZux1;+sHo-g1g2U0vQnr)j$V zEjqnPr#I;IQ#$>UPG7?boe&EYC8Fm_gWo-J>&W+BSZIE3xoXGl^*?I-e&bJeF6``1?l_q|IhtHMhIPU%YbXLb zYIt8O*Ol}gxZCx>Xjm6DC<%;~D+YviTb#UV$u%32n>&*|XO`E-wJfgcvT%N9{r6(G zH~mol(JMcA?G?-JisjtX4{QtuW{VXIzq$&R zoR#y=$_3}Ti9@t|@{t=ymR#PX%Zud^uo=u^nRlY&{kE;Y-}H9VQrp4#wu5&LF18(6 zD0yMN?Z|BLPrLrO>!&Ba`st~0S`XkiKaGd;B`-`I`ohWOn*Xn?^N(rcJmdHs9N0b^ zW9+ky!NwQ}m|unv0tpZj2Z)JV643%_Lx{((@Vf((9}}U2wUz4rkYdtxm?cw=wNfrx zq`1?hFfG!ER4G)N)Ysfy&LmW@HPW>IAc@6NY0{qeaM`9w*7xq-_q}`1=kt5_Ja^CM z$-iCp+cI9;vRD=}eq9tf`1-X=flG^J_~q#q$OtXR9yvMYp5=}uT+J8OB4^#w>nqH+ zmTxS=n_c|L^QhT}v_BCBhNXdt$iRfC^slAro~5PF+ppVM=C*Co#;29ddiU~0AoG5K zs(>%0H_m!D6!iO(5!w^Jcz@>iGd%72aPq^+_x`6uQg~Jsp7qqskQX_RN~yq+&!3$;8&gmaC`1#CF>1+`EX5H^F)HciEj^NDBw`uiEn@*~RBc&RJGiVR zQYx$w3TrT|Kz~l?y&(0DMS90X%{yxbCTh%O-J&r+K<`jslE`++OIl`TSm982Ep&xW zBf3POO4lhB(pc|L-E4=ztf?=K3bq5tgWt_Y^aX($*sTtA-0EEDgrjG_W9My$k+lZV zwE|Taua$qh&?&As+;@NUtI_cJ<(iegZ%4ivS?NM_r$C+CZFg_q!rSOz8?v?|x;R)1~i);AZl7B`_h1^G=|&yBlB!Qs z^`XI0{ObC}`I)&H09~wE?CXwV&WjEW2#&!QiHQ#Fnqx4;#N)IiLqWt)z#9rBLq)_; zAwU{&L(>lAwKN8)h+}XYw|_w-W~2tZfKxcsNG^L($60i)o3DN+qNxnDaG3z|5H&eC zcTifeQ`BS!ngHPdf7>+k%~FnDu6%Ul;SoNgRn)Zxli^4C)U&BJhL~IS1-qbjt`{F( zDq8MaX+k}NeDP33;|#b1)1l7)-)MYQ1~?z57fuW6;*DHqz`F~Y7BqHlSaP`|E;sLL z_@@kBwnj8oL6fzvNxkO%2-@u}>w{H-#<8O$EcUtmtb3=Hp!>;HRo`|!?D8+Cfc4-y z$(y|p?-xi~%PZvd#}=DHjZ*1RvGnMCWw3IwDWE|5W6yq!+Pm~c4cE=O!paARr89rB zE@{z;A*3A^lq2!LzJqrS5q!tYm$dSQt$gkqDCbS2Z4;F3(BMTg5fr^WO1_xmAd%4m80Z$CmR#m1Nd#D`76!2xAY#6{_dd>yU;%?=Bs#;hl}k zeUC;Tj-s}=q&8oq&4;@CrS40S?n^)fNz-o{%+He&($cqq3zD;?B2pNj&6>3@wM1r) zl;K*t{thh3$11>!ytPq3c7Rjj$fkR2=YZ>?;XYh6;XFA9e&9H>zrG8ba+i zGT1)RQpEK=POn%d11I4uj_NZ)l`I)@vW_nsxTG(<+yq-6PG8Ccx@;x@M4Cu*)SS)g zKc8kBk*Scg{-Ga6ei z&6+s1=%|D%Uv@RBej91e3d+vCwLQP_ajnQ&hUjvE+P@LUnPjtBD7ze$H!gQApN^mD z@^E^yovq_-<$JyDZU;(hd0QE>mLnQpJ2n)C-M+sxveV%avB1qcj-%{`<%;Fr6+oyA zh%E!+i9s}SMLcl@{p>1gnnu$0(JcF3ccJ%^@GHv*Z4!tsQM&2IgYdwg3=!~IoMpul#fjLffMk90YUn*WG;x9 z3y{glC2&tMJYS__)G zO?_5qQqt!loU{z~AgBuEOBqFxj3Q*gl7WR5-1A*Xa1XigAalB6l!gkTF*yX&x_jgqSCjCQ>kSFn@3RMo{+W~lBsRtJazGn zZRkw5N&@>4Rm!U zhso=3Bm1b-JYGeus2HpVG~>0@H&im{YE8~WY7%cfI6-IxQ!-ZT@lq&~Xm=h=qK8eO zYpI5bB;{H`GS<@LB~Y|(ajPe2`nx=L4J>WgapO(Ood}b^YLc*;4uk$p=f^^gx+%&vFmwTQ93Bo(E?})qC4u(i z$(8<|aVQ(={V8bA7@o(tJ<#z=td)3vyzc^T3KqdsvOKAP!iyL2CVvVFRo#UWImrCyMFvL*oMf3xhHxsCjmSLj_Hf8IKE9b40lb~1b@H(6WkWN zJN>dXYQuBapbQ>Hf~#77@(5-Qejwn-zDzk*-w)ncZ18~d91s)!zu@xs@pi6b$-#@V zjfvk|@w>5jv7ExvhQ)_v5X&%@c`UcE+{GfBhO&Js+lR8<7sC}lVEGponIa&Ym=4?} zrku&K{JT~vHfsF%wWcQ&Fc~k(j~{j;GU1&wuv+WDU3h7x zv170k=Skp&C#ZK4(#fM+1jvg-Slp5qtKN&$mG4Fg*?BpQ4-Px#u!L-7SqZ50GW$+nn|2gA*Kgjd`u^Ez+*iL t1!Q)tjfAN4m Date: Mon, 24 Aug 2026 14:28:48 -0700 Subject: [PATCH 07/13] ADFA-5153: Fix eight review findings, one of which made the output unreadable jatezzz, high: the script recompressed content against the dictionary and never declared a version, so the app -- which gates on the declared MAJOR, not on the presence of CompressionDictionary -- would refuse to attach the dictionary and every row just migrated would fail to decode. A database that looks migrated and serves nothing. The migrate phase now writes major 2 in the same transaction as the last batch, and says so; a dry run says what it would declare and why. Declared even after a partly failed run, since WebServer falls back to a plain decode for rows that did not migrate but cannot read the ones that did without it. jatezzz, medium: newly inserted continuation rows carried the base row's languageID, while WebServer looks continuations up with "languageId = 1" hardcoded. Any item whose base row is not language 1 would have had its continuations become invisible and its page truncate at 1 MiB -- the exact ADFA-5171 symptom this script exists to remove. Continuations are inserted as language 1 now. jatezzz, medium: retyping into a type whose own compression is not 'none' left the bytes compressed under a type they did not match, produced two spurious verifier errors per row, and could have served raw compressed bytes. Such a target is now refused with an actionable message and the row left alone. jatezzz, low: one missing or NULL-content row aborted the whole run from inside a worker, with earlier batches committed and no summary. Both phases now report it and continue. Testing that found a second path to the same crash: NULL content also made LENGTH() NULL, so the phase summary threw before any row was read. jatezzz, low: phase 1 submitted every candidate at once and held every decoded plaintext resident. It batches now, like phase 3. CodeRabbit, major: renumber_item parked rows under "{base}-renumbering-{n}", which a real row can already occupy -- failing on UNIQUE(path) after the collision checks had passed. A single ascending pass needs no temporary names: contiguity is verified first, so the lowest target is free and every later one was vacated by the move before it. My comment claiming otherwise was wrong. CodeRabbit, major: verify_retype reported valid -2 suffixes as errors in a retype-only run, so a successful run exited 1. That check now runs only when renumbering was requested. Co-Authored-By: Claude Opus 5 --- .../migrate_content_to_dictionary_brotli.py | 228 +++++++++++++----- 1 file changed, 171 insertions(+), 57 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index d46b80a9c0..9ff51ef870 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -84,6 +84,9 @@ CHUNK_BYTES = 1024 * 1024 +# WebServer looks continuations up with "languageId = 1" hardcoded, whatever the base row says. +CONTINUATION_LANGUAGE_ID = 1 + def is_text_type(value: str) -> bool: """Whether a ContentTypes.value is a text type, matched at the boundary. @@ -322,7 +325,9 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: rows = connection.execute( f""" SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, - LENGTH(C.content), CT.value, CT.compression + -- IFNULL: a row with NULL content has NULL length, which made the byte totals + -- (and so the phase summary) throw before read_blobs could report the row. + IFNULL(LENGTH(C.content), 0), CT.value, CT.compression FROM Content C JOIN ContentTypes CT ON CT.id = C.contentTypeID WHERE {predicate} @@ -362,11 +367,17 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: return sorted(items.values(), key=lambda item: item.base_path) -def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes]: +def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes] | None: + """An item's slices in order, or None when a row has vanished or holds NULL content. + + None rather than an exception: a single unreadable row used to abort the whole run from + inside a worker, leaving earlier batches committed and printing no summary at all. + """ ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] placeholders = ",".join("?" * len(ids)) found = dict(connection.execute(f"SELECT id, content FROM Content WHERE id IN ({placeholders})", ids).fetchall()) - return [found[row_id] for row_id in ids] + blobs = [found.get(row_id) for row_id in ids] + return None if any(blob is None for blob in blobs) else blobs def write_item( @@ -406,7 +417,12 @@ def write_item( INSERT INTO Content (path, languageID, content, contentTypeID, templateId) VALUES (?, ?, ?, ?, ?) """, - (f"{item.base_path}-{suffix}", item.language_id, payload, type_id, item.template_id), + # languageID 1, not the base row's: WebServer's continuation query is + # "WHERE path = ? AND languageId = 1" (documented in + # docs/documentation-database.md), so a continuation inserted under any other + # language is invisible and the page truncates at its first 1 MiB -- the exact + # ADFA-5171 symptom this script exists to remove. + (f"{item.base_path}-{suffix}", CONTINUATION_LANGUAGE_ID, payload, type_id, item.template_id), ) inserted += 1 else: @@ -433,6 +449,58 @@ def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int ) +# The MAJOR the app requires before it will attach the dictionary at all +# (DatabaseVersionResolver.MAJOR_VERSION_WITH_COMPRESSION_DICTIONARY). Migrating content without +# declaring this leaves a database whose every brotli row fails to decode: WebServer gates on the +# declared version, not on the presence of CompressionDictionary, so it never attaches the +# dictionary and the plain decode then throws "corrupted input" on every migrated row. +DICTIONARY_MAJOR_VERSION = 2 + +VERSION_TABLE_SQL = """ +CREATE TABLE IF NOT EXISTS DocumentationDatabaseVersion ( + major INT NOT NULL, + minor INT NOT NULL, + patch INT NOT NULL, + who TEXT NOT NULL, + comment TEXT NOT NULL, + changeTime TIMESTAMP DEFAULT CURRENT_TIMESTAMP +) +""" + + +def declared_major(connection: sqlite3.Connection) -> int | None: + """The MAJOR this database declares, or None when it declares none. + + Reads the row with the highest rowid: the table holds exactly one row by contract, and this is + the row both the app's DatabaseVersionResolver and docdb-studio read (ADFA-5220). + """ + if not table_exists(connection, "DocumentationDatabaseVersion"): + return None + row = connection.execute( + "SELECT major FROM DocumentationDatabaseVersion ORDER BY rowid DESC LIMIT 1" + ).fetchone() + return row[0] if row is not None and row[0] is not None else None + + +def declare_dictionary_version(connection: sqlite3.Connection) -> None: + """Records that this database's brotli content is dictionary-compressed. + + Written in the same transaction as the last batch of content, because the two facts have to + travel together: content compressed against the dictionary, and a version saying so. Exactly + one row, replaced rather than appended, matching populate_db.py. + """ + connection.execute(VERSION_TABLE_SQL) + connection.execute("DELETE FROM DocumentationDatabaseVersion") + connection.execute( + "INSERT INTO DocumentationDatabaseVersion (major, minor, patch, who, comment) VALUES (?, 0, 0, ?, ?)", + ( + DICTIONARY_MAJOR_VERSION, + "migrate_content_to_dictionary_brotli.py", + "Content rows compressed against CompressionDictionary", + ), + ) + + def table_exists(connection: sqlite3.Connection, name: str) -> bool: """Whether `name` is a table in this database -- checked the way WebServer does.""" found = connection.execute( @@ -468,15 +536,13 @@ def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> st return f"{target} already exists and belongs to another row; left alone" if write: - # Two passes: park every row under a name nothing can collide with, then - # settle them into their new slots. One ascending pass would be enough only - # if the target were always already free, and for a shift of 1 it never is. - for row_id, suffix, _ in item.continuations: - connection.execute( - "UPDATE Content SET path = ? WHERE id = ?", - (f"{item.base_path}-renumbering-{suffix}", row_id), - ) - for row_id, suffix, _ in item.continuations: + # One ascending pass, no temporary names. The suffixes were just verified + # contiguous, so the lowest target (first_suffix - shift) is free -- nothing + # occupies a suffix below first_suffix -- and every later target was vacated by + # the move before it. The parking pass this replaces invented + # "{base}-renumbering-{n}" paths that a real row could already hold, which would + # fail on UNIQUE(path) after the collision checks above had passed. + for row_id, suffix, _ in sorted(item.continuations, key=lambda entry: entry[1]): connection.execute( "UPDATE Content SET path = ? WHERE id = ?", (f"{item.base_path}-{suffix - shift}", row_id), @@ -518,30 +584,40 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s retyped: list[tuple[Item, Inspection, str]] = [] before_total = after_total = 0 - pending = {pool.submit(inspect_item, item, read_blobs(connection, item)): item for item in items} - for future in futures.as_completed(pending): - item = pending[future] - found = future.result() - if found.status == "error": - errors.append(f"{item.base_path}: {found.detail}") - continue - if found.status == "keep": - counts["left as text"] = counts.get("left as text", 0) + 1 - notes.append(f"{item.base_path}: {found.detail}") - continue - - target = found.sniffed - if target == "video/quicktime" and args.mov_type == "mp4": - target = "video/mp4" - extension = item.base_path.rsplit(".", 1)[-1].lower() - if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ - and not (extension == "mov" and target.startswith("video/")): - notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") - - retyped.append((item, found, target)) - counts[target] = counts.get(target, 0) + 1 - before_total += found.before - after_total += found.after + # Batched like phase 3, and for the same reason: this holds every candidate's decoded + # plaintext resident until the write loop, and one row in this database is 23 MB. + for offset in range(0, len(items), args.batch): + pending = {} + for item in items[offset : offset + args.batch]: + blobs = read_blobs(connection, item) + if blobs is None: + errors.append(f"{item.base_path}: a row is missing or holds NULL content; left alone") + continue + pending[pool.submit(inspect_item, item, blobs)] = item + + for future in futures.as_completed(pending): + item = pending[future] + found = future.result() + if found.status == "error": + errors.append(f"{item.base_path}: {found.detail}") + continue + if found.status == "keep": + counts["left as text"] = counts.get("left as text", 0) + 1 + notes.append(f"{item.base_path}: {found.detail}") + continue + + target = found.sniffed + if target == "video/quicktime" and args.mov_type == "mp4": + target = "video/mp4" + extension = item.base_path.rsplit(".", 1)[-1].lower() + if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ + and not (extension == "mov" and target.startswith("video/")): + notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") + + retyped.append((item, found, target)) + counts[target] = counts.get(target, 0) + 1 + before_total += found.before + after_total += found.after missing = sorted({target for _, _, target in retyped if target not in types}) for value in missing: @@ -556,20 +632,28 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s print(f" ContentTypes would insert {value} (compression none)") inserted_total = deleted_total = 0 + kept = [] for item, found, target in retyped: type_id, compression = types[target] + # A target type whose own compression is not 'none' cannot receive plaintext. Retyping + # into it used to leave the bytes compressed and the row declaring a type they are not, + # which the verifier below then reported twice -- and if WebServer does not handle that + # compression at all, the row would serve raw compressed bytes to a browser. + if compression != "none": + errors.append( + f"{item.base_path}: {target} is registered with compression '{compression}', not 'none'; " + f"left as {item.content_type}. Fix the ContentTypes row, then re-run" + ) + counts[target] = counts.get(target, 0) - 1 + continue + kept.append((item, found, target)) if not write: continue - if compression == "none": - inserted, deleted = write_item( - connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id - ) - inserted_total += inserted - deleted_total += deleted - else: - # The honest type is itself a compressed one, so the stored bytes stay - # as they are and phase 3 picks the row up. - retype_rows(connection, item, type_id) + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + inserted_total += inserted + deleted_total += deleted if write: connection.commit() @@ -579,7 +663,7 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") if write: print(f" rows inserted {inserted_total} deleted {deleted_total}") - return {item.base_path for item, _, _ in retyped}, errors, notes + return {item.base_path for item, _, _ in kept}, errors, notes def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> tuple[int, list[str], list[str]]: @@ -624,7 +708,7 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> return fixed, errors, notes -def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[str]: +def verify_retype(connection, retyped_paths: set[str], mov_type: str, expect_renumbered: bool) -> list[str]: """Re-read what phase 1 wrote and confirm the bytes match the declared type.""" problems: list[str] = [] written = {item.base_path: item for item in load_items(connection, "1 = 1")} @@ -642,7 +726,9 @@ def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[st problems.append(f"{path}: declared {expected} but the stored bytes sniff as {found or 'unknown'}") if item.compression != "none": problems.append(f"{path}: retyped to {expected}, whose compression is {item.compression}") - if item.continuations and item.suffixes != list(range(1, len(item.continuations) + 1)): + # Only when this run was asked to renumber: a retype-only run legitimately leaves + # -2-based numbering alone, and reporting it as an error made a successful run exit 1. + if expect_renumbered and item.continuations and item.suffixes != list(range(1, len(item.continuations) + 1)): problems.append(f"{path}: continuations numbered {item.suffixes}, expected 1..n") return problems @@ -729,7 +815,9 @@ def select(items: list[Item]) -> list[Item]: if write: # A verification failure means the bytes and their declared type disagree # after we wrote them -- the most serious thing this script can report. - errors += verify_retype(connection, retyped_paths, args.mov_type) + errors += verify_retype( + connection, retyped_paths, args.mov_type, expect_renumbered="renumber" in args.phase_list + ) print() if "renumber" in args.phase_list: @@ -765,12 +853,22 @@ def select(items: list[Item]) -> list[Item]: for offset in range(0, len(items), args.batch): batch = items[offset : offset + args.batch] - pending = { - pool.submit( - migrate_item, item, read_blobs(connection, item), args.quality, args.window, args.only_if_smaller - ): item - for item in batch - } + pending = {} + for item in batch: + blobs = read_blobs(connection, item) + if blobs is None: + # Reported, not raised: this used to surface as a TypeError inside a worker + # and abort the run with earlier batches already committed and no summary. + counts["error"] += 1 + failed_items.append( + Result(item.base_path, "error", detail="a row is missing or holds NULL content") + ) + continue + pending[ + pool.submit( + migrate_item, item, blobs, args.quality, args.window, args.only_if_smaller + ) + ] = item for future in futures.as_completed(pending): item = pending[future] @@ -831,10 +929,26 @@ def select(items: list[Item]) -> list[Item]: report(errors, notes) if write: + # The version goes in with the content, not after it: a database holding + # dictionary-compressed rows while declaring anything below + # DICTIONARY_MAJOR_VERSION is one the app refuses to attach the dictionary for, so every + # row just migrated fails to decode. Declared even on a partly failed run -- WebServer + # tries the dictionary first and falls back to a plain decode, so a row that did not + # migrate still serves, while the ones that did only serve with this row present. + before = declared_major(connection) + if before is None or before < DICTIONARY_MAJOR_VERSION: + declare_dictionary_version(connection) + print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " + f"(was {'none' if before is None else before})") connection.commit() print("\nRun VACUUM to reclaim the freed pages: sqlite3 %s 'VACUUM;'" % args.database) else: connection.rollback() + before = declared_major(connection) + if before is None or before < DICTIONARY_MAJOR_VERSION: + print(f"would declare database version {DICTIONARY_MAJOR_VERSION}.0.0 " + f"(currently {'none' if before is None else before}) -- without it the app will not " + f"attach the dictionary and every migrated row fails to decode") print("\nNothing written. Re-run with --yes on a copy to apply.") connection.close() From 0be5a445da3cd651404e59c1c9dbe8077da3e0d4 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 24 Aug 2026 15:22:47 -0700 Subject: [PATCH 08/13] ADFA-5153: Finish three fixes the review found half-done All three follow from the previous round and are the reviewer's, not mine. Batching phase 1's submissions bounded the workers, not the memory: every completed Inspection was appended to a list and written only after the last batch, so --batch did nothing about the thing that actually runs out. Each batch is now inspected, typed, written and committed before the next one starts, and each item's decoded payload is dropped as soon as it is written. The verifier gets a set of paths rather than a list holding slices. The continuation language fix only covered inserts. Reusing an existing continuation row updated its content and type but left its languageID, so a row that predates this script and carries the base row's language stayed invisible to WebServer's continuation query -- the same truncation, through the row this script chose not to replace. The update normalises it too. The version was declared after the last batch, so a run interrupted between two committed batches left dictionary-compressed rows in a database still declaring a version the app will not attach the dictionary for: every committed row would fail to decode. It is now declared in the same transaction as the first batch of migrated content, with the end-of-run declaration kept as the fallback for a run that migrates nothing but finds content already migrated. Verified: a 4-row database with a realistic dictionary declares 2.0.0 during the batch loop rather than after it; retype with --batch 1 still retypes and still refuses a compressed target type; and the earlier round's checks all still hold -- the parking-path squatter, the NULL-content row, language 1 on inserted continuations, no suffix complaints in a retype-only run, and a dry run that leaves the file byte-identical. Co-Authored-By: Claude Opus 5 --- .../migrate_content_to_dictionary_brotli.py | 109 +++++++++++------- 1 file changed, 68 insertions(+), 41 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 9ff51ef870..825d44c10c 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -427,8 +427,12 @@ def write_item( inserted += 1 else: connection.execute( - "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", - (payload, type_id, row_id), + # languageID too, not just the bytes: a continuation row that predates this script + # can carry the base row's language, and WebServer's continuation query filters on + # languageId = 1 -- so reusing the row without normalising it leaves the page + # truncated exactly as an unnumbered continuation would. + "UPDATE Content SET content = ?, contentTypeID = ?, languageID = ? WHERE id = ?", + (payload, type_id, CONTINUATION_LANGUAGE_ID, row_id), ) # Whatever is left over described slices the new stream no longer needs. @@ -586,6 +590,12 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s # Batched like phase 3, and for the same reason: this holds every candidate's decoded # plaintext resident until the write loop, and one row in this database is 23 MB. + inserted_total = deleted_total = 0 + kept: set[str] = set() + + # Each batch is inspected *and written* before the next one starts. Batching only the + # submissions still held every decoded plaintext until a write loop at the end, so --batch + # bounded the workers and not the memory, which is the thing that runs out. for offset in range(0, len(items), args.batch): pending = {} for item in items[offset : offset + args.batch]: @@ -614,48 +624,46 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s and not (extension == "mov" and target.startswith("video/")): notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") - retyped.append((item, found, target)) + if target not in types: + if write: + cursor = connection.execute( + "INSERT INTO ContentTypes (value, compression) VALUES (?, 'none')", (value := target,) + ) + types[value] = (cursor.lastrowid, "none") + print(f" ContentTypes + id {cursor.lastrowid} {value} (compression none)") + else: + types[target] = (-1, "none") + print(f" ContentTypes would insert {target} (compression none)") + + type_id, compression = types[target] + # A target type whose own compression is not 'none' cannot receive plaintext. Retyping + # into it would leave the bytes compressed under a type they are not, which the verifier + # then reports twice -- and if WebServer does not handle that compression at all, the row + # serves raw compressed bytes to a browser. + if compression != "none": + errors.append( + f"{item.base_path}: {target} is registered with compression '{compression}', not 'none'; " + f"left as {item.content_type}. Fix the ContentTypes row, then re-run" + ) + continue + counts[target] = counts.get(target, 0) + 1 before_total += found.before after_total += found.after + kept.add(item.base_path) - missing = sorted({target for _, _, target in retyped if target not in types}) - for value in missing: - if write: - cursor = connection.execute( - "INSERT INTO ContentTypes (value, compression) VALUES (?, 'none')", (value,) - ) - types[value] = (cursor.lastrowid, "none") - print(f" ContentTypes + id {cursor.lastrowid} {value} (compression none)") - else: - types[value] = (-1, "none") - print(f" ContentTypes would insert {value} (compression none)") + if write: + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + inserted_total += inserted + deleted_total += deleted + # found.slices is the only large thing here; dropping the reference lets this batch's + # plaintext be collected before the next batch decodes its own. + found.slices = [] - inserted_total = deleted_total = 0 - kept = [] - for item, found, target in retyped: - type_id, compression = types[target] - # A target type whose own compression is not 'none' cannot receive plaintext. Retyping - # into it used to leave the bytes compressed and the row declaring a type they are not, - # which the verifier below then reported twice -- and if WebServer does not handle that - # compression at all, the row would serve raw compressed bytes to a browser. - if compression != "none": - errors.append( - f"{item.base_path}: {target} is registered with compression '{compression}', not 'none'; " - f"left as {item.content_type}. Fix the ContentTypes row, then re-run" - ) - counts[target] = counts.get(target, 0) - 1 - continue - kept.append((item, found, target)) - if not write: - continue - inserted, deleted = write_item( - connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id - ) - inserted_total += inserted - deleted_total += deleted - if write: - connection.commit() + if write: + connection.commit() for target, count in sorted(counts.items(), key=lambda kv: -kv[1]): print(f" {target:22} {count:>4}") @@ -663,7 +671,7 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") if write: print(f" rows inserted {inserted_total} deleted {deleted_total}") - return {item.base_path for item, _, _ in kept}, errors, notes + return kept, errors, notes def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> tuple[int, list[str], list[str]]: @@ -844,6 +852,8 @@ def select(items: list[Item]) -> list[Item]: print() counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + wrote_migrated_content = False + version_declared = False before_total = after_total = 0 inserted_total = deleted_total = 0 # Not named `errors`: that name already holds this run's phase 1 and 2 failures, @@ -885,8 +895,23 @@ def select(items: list[Item]) -> list[Item]: inserted, deleted = write_item(connection, item, result.slices, renumber=False) inserted_total += inserted deleted_total += deleted + wrote_migrated_content = True if write: + # In the same transaction as the first batch of migrated content, not after the last + # one. A run interrupted between two committed batches would otherwise leave the + # database holding dictionary-compressed rows while declaring a version below the one + # the app requires -- so it would decline the dictionary and every committed row + # would fail to decode. Declaring first means the worst an interruption leaves is a + # partly migrated database that still serves, since the app falls back to a plain + # decode per row. + if wrote_migrated_content and not version_declared: + before = declared_major(connection) + if before is None or before < DICTIONARY_MAJOR_VERSION: + declare_dictionary_version(connection) + print(f"\ndeclared database version {DICTIONARY_MAJOR_VERSION}.0.0 " + f"(was {'none' if before is None else before})") + version_declared = True connection.commit() done = min(offset + args.batch, len(items)) @@ -935,8 +960,10 @@ def select(items: list[Item]) -> list[Item]: # row just migrated fails to decode. Declared even on a partly failed run -- WebServer # tries the dictionary first and falls back to a plain decode, so a row that did not # migrate still serves, while the ones that did only serve with this row present. + # Anything already dictionary-compressed still needs the declaration, even when this run + # migrated nothing itself (every row came back "already"). before = declared_major(connection) - if before is None or before < DICTIONARY_MAJOR_VERSION: + if not version_declared and (before is None or before < DICTIONARY_MAJOR_VERSION): declare_dictionary_version(connection) print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " f"(was {'none' if before is None else before})") From 45a8aa6068b775c9c519350650317078ce102852 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Tue, 25 Aug 2026 18:47:28 -0700 Subject: [PATCH 09/13] ADFA-5153: Only renumber items that are actually chunked, and refuse a taken path Two ways this script could damage content, both found in review. phase_renumber decided an item was chunked from the path suffix alone. A real page whose greedy base happens to be another real page -- k/kotlin-1-2 under k/kotlin-1 -- was therefore renamed to k/kotlin-1-1, which 404s every link to it and leaves the base looking like a two-slice item. The signal that would have prevented it, base row length == CHUNK_BYTES, was already computed twenty lines below as a note, i.e. after every rename had been made, and report() caps notes at 30. That test now selects the candidates instead: an item is chunked when its base row is exactly CHUNK_BYTES, which is what the app's own continuation query requires before it will reassemble anything. A numeric-suffixed sibling with a differently sized base is reported as independent content and left alone. write_item INSERTed continuation paths with no UNIQUE(path) check, though renumber_item already makes exactly that check before it moves anything. An occupied target -- a foreign row, or a continuation the phase predicate excludes -- surfaced as a bare sqlite3.IntegrityError from the middle of a phase, with earlier batches committed and no summary printed, which is the failure the batching was introduced to avoid. write_item raises PathClash before it writes anything now, and both call sites record it as an error for that item and carry on with the rest. Verified against synthetic databases holding each case: - k/kotlin-1 (4 bytes) + k/kotlin-1-2: left alone, reported as independent. Against the script as it stood, k/kotlin-1-2 is renamed to k/kotlin-1-1. - big/page (exactly CHUNK_BYTES) + -2 + -3: still renumbered to start at -1, so the repair this phase exists for is unaffected. - img.gif with continuations at -2/-3 while another content type owns img.gif-1: PathClash, named, instead of an IntegrityError mid-run. Found in review of PR #1724. --- .../migrate_content_to_dictionary_brotli.py | 73 +++++++++++++++---- 1 file changed, 59 insertions(+), 14 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 825d44c10c..f0c690413b 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -380,6 +380,22 @@ def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes] | None return None if any(blob is None for blob in blobs) else blobs +class PathClash(Exception): + """A continuation path this item needs is owned by some other row.""" + + +def continuation_clash(connection: sqlite3.Connection, item: Item, slices: int, renumber: bool) -> str: + """The first target path owned by a foreign row, described; '' if the write is safe.""" + start = 1 if renumber else item.first_suffix + own = {row_id for row_id, _, _ in item.continuations} + for suffix in range(start, start + slices - 1): + target = f"{item.base_path}-{suffix}" + row = connection.execute("SELECT id FROM Content WHERE path = ?", (target,)).fetchone() + if row and row[0] not in own: + return f"{target} already exists and belongs to another row" + return "" + + def write_item( connection: sqlite3.Connection, item: Item, @@ -387,7 +403,17 @@ def write_item( renumber: bool, content_type_id: int | None = None, ) -> tuple[int, int]: - """Write an item's new slices back. Returns (rows inserted, rows deleted).""" + """Write an item's new slices back. Returns (rows inserted, rows deleted). + + Raises PathClash if a continuation path is owned by a foreign row. renumber_item makes the + same check before it moves anything; this one did not, so an occupied path surfaced as a bare + sqlite3.IntegrityError out of the middle of a phase, with earlier batches already committed + and no summary printed -- the failure mode the batching was introduced to avoid. + """ + clash = continuation_clash(connection, item, len(slices), renumber) + if clash: + raise PathClash(f"{item.base_path}: {clash}; left alone") + type_id = item.content_type_id if content_type_id is None else content_type_id connection.execute( "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", @@ -653,9 +679,14 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s kept.add(item.base_path) if write: - inserted, deleted = write_item( - connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id - ) + try: + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + except PathClash as clash: + errors.append(str(clash)) + found.slices = [] + continue inserted_total += inserted deleted_total += deleted # found.slices is the only large thing here; dropping the reference lets this batch's @@ -684,14 +715,22 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> items = select([item for item in load_items(connection, "1 = 1") if item.continuations]) if args.renumber_scope == "retyped": items = [item for item in items if item.base_path in retyped_paths] - broken = [item for item in items if item.first_suffix != 1] + # A "-" sibling is a naming coincidence until the base row proves otherwise. The app + # decides an item is chunked by its base row being exactly CHUNK_BYTES (WebServer's continuation + # query), so that is the test here too. Without it this phase renamed real, independent pages: + # a page k/kotlin-1-2 whose greedy base is the real page k/kotlin-1 was renumbered to + # k/kotlin-1-1, which 404s every link to it and leaves the base looking like a 2-slice item. + # The check used to happen 20 lines below, as a note, after every rename had been made. + chunked = [item for item in items if item.base_bytes == CHUNK_BYTES] + coincidental = [item for item in items if item.base_bytes != CHUNK_BYTES] + broken = [item for item in chunked if item.first_suffix != 1] starts = sorted({item.first_suffix for item in broken}) if broken: - print(f"[2/3] renumber {len(broken)} of {len(items)} chunked items start at " + print(f"[2/3] renumber {len(broken)} of {len(chunked)} chunked items start at " f"{', '.join('-' + str(n) for n in starts)} instead of -1") else: - print(f"[2/3] renumber all {len(items)} chunked items already start at -1") + print(f"[2/3] renumber all {len(chunked)} chunked items already start at -1") errors: list[str] = [] notes: list[str] = [] fixed = 0 @@ -705,12 +744,12 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> if write: connection.commit() - for item in items: - if item.base_bytes != CHUNK_BYTES: - notes.append( - f"{item.base_path}: chunked but its base row is {item.base_bytes:,} bytes, not " - f"{CHUNK_BYTES:,} -- the app detects chunking by that exact length, so it will not reassemble" - ) + for item in coincidental: + notes.append( + f"{item.base_path}: has a numeric-suffixed sibling but its base row is " + f"{item.base_bytes:,} bytes, not {CHUNK_BYTES:,} -- treated as independent content and " + f"left alone, since the app only reassembles an item whose base row is exactly that long" + ) if fixed: print(f" {'renumbered' if write else 'would renumber'} to start at -1: {fixed}") return fixed, errors, notes @@ -892,7 +931,13 @@ def select(items: list[Item]) -> list[Item]: if result.status == "error": failed_items.append(result) elif result.status == "migrated" and write: - inserted, deleted = write_item(connection, item, result.slices, renumber=False) + try: + inserted, deleted = write_item(connection, item, result.slices, renumber=False) + except PathClash as clash: + failed_items.append(result) + errors.append(str(clash)) + result.slices = [] + continue inserted_total += inserted deleted_total += deleted wrote_migrated_content = True From 56d04c62c8b1342d6ccbcb12b316a770f6a097f4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 16:14:42 +0000 Subject: [PATCH 10/13] ADFA-5153: address review - append-only version log, gate continuation grouping, guard v2 declaration Three fixes from review: - declare_dictionary_version no longer DELETEs the version log. The table is append-only by contract (docs/documentation-database.md, DatabaseVersionResolver): the last-inserted row wins, so the INSERT alone declares version 2 and prior rows stay as history. Docstrings that asserted a one-row contract are corrected. - load_items only groups a "-" sibling as a continuation when its base row holds exactly CHUNK_BYTES, the app's own chunk-detection rule. Grouping on the name alone let phase 1's rewrite of a short base absorb an independent sibling page's bytes and delete its row. Every phase inherits the gate from this one choke point; phase_renumber's local copy of the test is now redundant and reduced to a comment. - Declaring version 2 over rows left plain is guarded two ways: --only-if-smaller is refused whenever the migrate phase runs (it deliberately leaves plain rows in a database that will declare 2, and a plain row can decode against the dictionary to wrong bytes without erroring), and a write run that declared 2 ends with an explicit WARNING when any brotli item did not migrate. --- .../migrate_content_to_dictionary_brotli.py | 76 +++++++++++++------ 1 file changed, 51 insertions(+), 25 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index f0c690413b..6c5cf624d8 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -334,15 +334,17 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: """ ).fetchall() - paths = {row[1] for row in rows} + base_bytes_by_path = {row[1]: row[5] for row in rows} items: dict[str, Item] = {} continuations: list[tuple[str, int, int, int]] = [] for row_id, path, language_id, type_id, template_id, length, type_value, compression in rows: match = CONTINUATION.match(path) - # A continuation only counts as one if its base is itself a row; a path - # that merely ends in - is ordinary content. - if match and match.group(1) in paths: + # A "-" sibling is a naming coincidence until the base row proves otherwise, and + # the proof is the base holding exactly CHUNK_BYTES -- the app's own chunk-detection rule. + # Grouping on the name alone let a rewrite of the greedy base (retype or migrate) absorb an + # independent page's bytes and delete its row; here every phase inherits the test. + if match and base_bytes_by_path.get(match.group(1)) == CHUNK_BYTES: continuations.append((match.group(1), row_id, int(match.group(2)), length)) else: items[path] = Item( @@ -501,8 +503,9 @@ def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int def declared_major(connection: sqlite3.Connection) -> int | None: """The MAJOR this database declares, or None when it declares none. - Reads the row with the highest rowid: the table holds exactly one row by contract, and this is - the row both the app's DatabaseVersionResolver and docdb-studio read (ADFA-5220). + Reads the row with the highest rowid: the table is append-only by contract + (docs/documentation-database.md), so the row inserted last is the current version -- the same + row the app's DatabaseVersionResolver reads (ADFA-5220). """ if not table_exists(connection, "DocumentationDatabaseVersion"): return None @@ -515,12 +518,13 @@ def declared_major(connection: sqlite3.Connection) -> int | None: def declare_dictionary_version(connection: sqlite3.Connection) -> None: """Records that this database's brotli content is dictionary-compressed. - Written in the same transaction as the last batch of content, because the two facts have to - travel together: content compressed against the dictionary, and a version saying so. Exactly - one row, replaced rather than appended, matching populate_db.py. + Written in the same transaction as the first batch of migrated content, because the two facts + have to travel together: content compressed against the dictionary, and a version saying so. + The log is append-only by contract (docs/documentation-database.md, DatabaseVersionResolver): + each change is another INSERT and the row inserted last is the current version, so prior + version rows are history to keep, never state to replace. """ connection.execute(VERSION_TABLE_SQL) - connection.execute("DELETE FROM DocumentationDatabaseVersion") connection.execute( "INSERT INTO DocumentationDatabaseVersion (major, minor, patch, who, comment) VALUES (?, 0, 0, ?, ?)", ( @@ -715,14 +719,11 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> items = select([item for item in load_items(connection, "1 = 1") if item.continuations]) if args.renumber_scope == "retyped": items = [item for item in items if item.base_path in retyped_paths] - # A "-" sibling is a naming coincidence until the base row proves otherwise. The app - # decides an item is chunked by its base row being exactly CHUNK_BYTES (WebServer's continuation - # query), so that is the test here too. Without it this phase renamed real, independent pages: - # a page k/kotlin-1-2 whose greedy base is the real page k/kotlin-1 was renumbered to - # k/kotlin-1-1, which 404s every link to it and leaves the base looking like a 2-slice item. - # The check used to happen 20 lines below, as a note, after every rename had been made. - chunked = [item for item in items if item.base_bytes == CHUNK_BYTES] - coincidental = [item for item in items if item.base_bytes != CHUNK_BYTES] + # load_items only groups a "-" sibling under a base holding exactly CHUNK_BYTES -- the + # app's own chunk-detection rule -- so every item here is genuinely chunked, and a + # coincidentally named independent page (e.g. k/kotlin-1-2 next to the real page k/kotlin-1) + # never reaches the renames below. + chunked = items broken = [item for item in chunked if item.first_suffix != 1] starts = sorted({item.first_suffix for item in broken}) @@ -744,12 +745,6 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> if write: connection.commit() - for item in coincidental: - notes.append( - f"{item.base_path}: has a numeric-suffixed sibling but its base row is " - f"{item.base_bytes:,} bytes, not {CHUNK_BYTES:,} -- treated as independent content and " - f"left alone, since the app only reassembles an item whose base row is exactly that long" - ) if fixed: print(f" {'renumbered' if write else 'would renumber'} to start at -1: {fixed}") return fixed, errors, notes @@ -801,7 +796,8 @@ def main() -> int: parser.add_argument( "--only-if-smaller", action="store_true", - help="leave a row alone when its dictionary-compressed form is not smaller", + help="leave a row alone when its dictionary-compressed form is not smaller " + "(refused when the migrate phase runs: see the error it prints)", ) args = parser.parse_args() write = args.yes and not args.dry_run @@ -812,6 +808,20 @@ def main() -> int: print(f"error: unknown phase(s) {', '.join(unknown)}; pick from {', '.join(ALL_PHASES)}", file=sys.stderr) return 2 + # A migrate run declares version DICTIONARY_MAJOR_VERSION, and a plain-brotli row left behind + # in a database declaring that version can decode against the dictionary to different bytes + # *without erroring* -- served as silently wrong content. --only-if-smaller deliberately + # leaves such rows, so the two cannot travel together. + if args.only_if_smaller and "migrate" in args.phase_list: + print( + "error: --only-if-smaller cannot be combined with the migrate phase: it deliberately " + "leaves rows plain-compressed in a database the run declares version " + f"{DICTIONARY_MAJOR_VERSION}, and a plain row in such a database can decode against " + "the dictionary to wrong bytes without erroring", + file=sys.stderr, + ) + return 2 + connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") @@ -1012,7 +1022,23 @@ def select(items: list[Item]) -> list[Item]: declare_dictionary_version(connection) print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " f"(was {'none' if before is None else before})") + version_declared = True connection.commit() + # Attaching the dictionary to a plain-compressed row usually throws (the app then falls + # back to a plain decode), but a small fraction decode without error to different bytes. + # So a declared database still holding plain rows is not merely incomplete: it can serve + # silently wrong content. counts["error"] misses items that failed at the write (PathClash) + # after counting as migrated; failed_items holds both, so it is the honest tally. + remaining = counts["unchanged"] + len(failed_items) + if version_declared and remaining: + print( + f"\nWARNING: this database declares version {DICTIONARY_MAJOR_VERSION}.x but " + f"{remaining} brotli item(s) did not migrate and are still stored as before. " + f"A plain-compressed row can decode against the dictionary to wrong bytes " + f"without erroring, so re-run this script to completion before shipping " + f"this database.", + file=sys.stderr, + ) print("\nRun VACUUM to reclaim the freed pages: sqlite3 %s 'VACUUM;'" % args.database) else: connection.rollback() From 9914db6a6c2a9fd1047df1b7b190e28314030828 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Wed, 26 Aug 2026 15:14:13 -0700 Subject: [PATCH 11/13] ADFA-5153: A 1 MiB base is not proof of chunking The guard I added last round -- base row exactly CHUNK_BYTES -- rules out a small base and nothing else. A real page that happens to be exactly 1 MiB, sitting next to independently named "-2"/"-3" pages, was still grouped with them, and phase 2 renamed those pages into its slice slots. Reproduced against the real schema: p/page (1,048,576 bytes) plus p/page-2 and p/page-3 at 11 bytes each came out as p/page, p/page-1 (11 bytes), p/page-2 (11 bytes) -- both original URLs 404, one page gone, and the app appends a foreign page's bytes when it reassembles. The comment asserting this could not happen was wrong. The signal was already loaded: a genuine slice set has every slice except the last at exactly CHUNK_BYTES, because that is how the writer splits. An 11-byte "-2" followed by a "-3" is provably not one. The test now runs in load_items, so all three phases inherit it rather than phase 2 alone, and a sibling set that fails it becomes independent items instead of being silently absorbed. Verified: the two coincidence cases are left untouched, and both genuine mis-numbered slice sets are still repaired. A continuation whose base never became an Item was dropped silently -- not migrated, not counted, not reported, in a database the run then declares version 2. It is reported now. The version declaration is refused on a --path or --limit run. It covers the whole database, so only a run that considered the whole database may make it; a scoped run left tens of thousands of rows plain while telling the app they were dictionary-compressed. Most such rows throw and fall back, but a fraction decode without error to different bytes, which the code's own comment says two lines further down. The run now prints why it withheld the declaration. Phase 3 releases result.slices after writing, which phase 1 already did with a comment explaining why. Found in review of PR #1724. --- .../migrate_content_to_dictionary_brotli.py | 70 ++++++++++++++++++- 1 file changed, 68 insertions(+), 2 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 6c5cf624d8..2cf7a3a3ba 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -358,14 +358,54 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: base_bytes=length, ) + orphans: list[tuple[str, int, int, int]] = [] for base_path, row_id, suffix, length in continuations: owner = items.get(base_path) if owner is not None: owner.continuations.append((row_id, suffix, length)) + else: + # The greedy base is itself a continuation, or the phase predicate excluded it, so this + # row has no owner to be a slice of. Silently dropping it meant a row that was never + # migrated, never counted and never reported -- in a database the run then declares + # version 2. + orphans.append((base_path, row_id, suffix, length)) for item in items.values(): item.continuations.sort(key=lambda entry: entry[1]) + # A base of exactly CHUNK_BYTES is not enough on its own. A genuine slice set has every slice + # except the last at exactly CHUNK_BYTES, because that is how the writer splits; an 11-byte "-2" + # followed by a "-3" is provably not one. Without this, a real page that happens to be exactly + # 1 MiB, sitting next to independently named "-2"/"-3" pages, was grouped with them and phase 2 + # renamed those pages into its slice slots -- both URLs 404, and the app appends a foreign page's + # bytes on reassembly. Verified against the real schema before and after this check. + for item in list(items.values()): + if not item.continuations: + continue + head = item.continuations[:-1] + if all(length == CHUNK_BYTES for _, _, length in head): + continue + for row_id, suffix, length in item.continuations: + path = f"{item.base_path}-{suffix}" + items[path] = Item( + base_path=path, + base_id=row_id, + language_id=item.language_id, + content_type_id=item.content_type_id, + template_id=item.template_id, + content_type=item.content_type, + compression=item.compression, + base_bytes=length, + ) + item.continuations.clear() + + if orphans: + for base_path, _, suffix, _ in orphans: + print( + f" note: {base_path}-{suffix} looks like a continuation of {base_path}, which is " + f"not itself a migratable row; left alone and not migrated" + ) + return sorted(items.values(), key=lambda item: item.base_path) @@ -515,6 +555,23 @@ def declared_major(connection: sqlite3.Connection) -> int | None: return row[0] if row is not None and row[0] is not None else None +def may_declare_version(args) -> str: + """'' if this run may declare MAJOR 2, else why it may not. + + The declaration tells the app every brotli row is dictionary-compressed, and it applies to the + whole database -- so only a run that considered the whole database may make it. A --path or + --limit run migrates a handful and would leave the rest plain while claiming otherwise; the app + then attaches the dictionary to those rows, and while most throw and fall back, a fraction + decode without error to *different bytes*. That is silent wrong content, which is worse than the + unmigrated state it replaces. + """ + if args.path: + return f"the run was scoped by --path {args.path!r}" + if args.limit: + return f"the run was scoped by --limit {args.limit}" + return "" + + def declare_dictionary_version(connection: sqlite3.Connection) -> None: """Records that this database's brotli content is dictionary-compressed. @@ -951,6 +1008,10 @@ def select(items: list[Item]) -> list[Item]: inserted_total += inserted deleted_total += deleted wrote_migrated_content = True + # Same reason as phase 1: a completed Future holds its Result, and pending keeps + # every Future in the batch, so without this a batch of recompressed payloads + # stays resident while the next batch reads its own. + result.slices = [] if write: # In the same transaction as the first batch of migrated content, not after the last @@ -960,7 +1021,7 @@ def select(items: list[Item]) -> list[Item]: # would fail to decode. Declaring first means the worst an interruption leaves is a # partly migrated database that still serves, since the app falls back to a plain # decode per row. - if wrote_migrated_content and not version_declared: + if wrote_migrated_content and not version_declared and not may_declare_version(args): before = declared_major(connection) if before is None or before < DICTIONARY_MAJOR_VERSION: declare_dictionary_version(connection) @@ -1018,7 +1079,12 @@ def select(items: list[Item]) -> list[Item]: # Anything already dictionary-compressed still needs the declaration, even when this run # migrated nothing itself (every row came back "already"). before = declared_major(connection) - if not version_declared and (before is None or before < DICTIONARY_MAJOR_VERSION): + withheld = may_declare_version(args) + if withheld and (before is None or before < DICTIONARY_MAJOR_VERSION): + print(f"\nWARNING did NOT declare database version {DICTIONARY_MAJOR_VERSION}.0.0: {withheld}.") + print(" The declaration covers the whole database, so only an unscoped run may make") + print(" it. Re-run without --path/--limit before shipping this database.") + if not version_declared and not withheld and (before is None or before < DICTIONARY_MAJOR_VERSION): declare_dictionary_version(connection) print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " f"(was {'none' if before is None else before})") From bbdb3ea5be9a59c12954a76a7b6a25e8a5398ae3 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 22:29:49 +0000 Subject: [PATCH 12/13] ADFA-5153: harden migration script against mid-run failures and bad flags Three findings from CodeRabbit's review of PR #1724, each verified against a scratch database before and after: - Guard future.result() in the retype and migrate as_completed loops. A worker exception (e.g. the brotli CLI vanishing mid-run) re-raised and aborted the phase with earlier batches already committed and no summary; it is now recorded as that item's error and the run completes. A shutil.which("brotli") preflight also refuses retype/migrate runs up front when the CLI is missing, next to the dictionary checks. - Validate --batch and --workers >= 1. Zero raised from range() or ProcessPoolExecutor after work may have started; a negative batch silently processed nothing. Both now exit 2 with a clear error, alongside the existing phase-name check. - verify_retype passed read_blobs() straight to b"".join(), so a row deleted or NULLed between the write and the verify aborted verification with a TypeError. It now records the problem and continues, matching the adjacent vanished-row branch. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0197g8vkUQ1d6oLNbi8EnAYe --- .../migrate_content_to_dictionary_brotli.py | 43 +++++++++++++++++-- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 2cf7a3a3ba..3e5bbdc7b8 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -75,6 +75,7 @@ import concurrent.futures as futures import os import re +import shutil import sqlite3 import subprocess import sys @@ -694,7 +695,13 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s for future in futures.as_completed(pending): item = pending[future] - found = future.result() + try: + found = future.result() + except Exception as exc: + # A worker crash used to re-raise here and abort the phase with earlier + # batches already committed and no summary. Report it like any other failure. + errors.append(f"{item.base_path}: {type(exc).__name__}: {exc}") + continue if found.status == "error": errors.append(f"{item.base_path}: {found.detail}") continue @@ -816,7 +823,11 @@ def verify_retype(connection, retyped_paths: set[str], mov_type: str, expect_ren if item is None: problems.append(f"{path}: row vanished") continue - payload = b"".join(read_blobs(connection, item)) + blobs = read_blobs(connection, item) + if blobs is None: + problems.append(f"{path}: a row is missing or holds NULL content") + continue + payload = b"".join(blobs) found = sniff(payload) expected = item.content_type if expected == "video/mp4" and mov_type == "mp4" and found == "video/quicktime": @@ -879,6 +890,15 @@ def main() -> int: ) return 2 + # range() raises on a zero batch, a negative one silently processes nothing, and + # ProcessPoolExecutor raises on zero workers -- all after work may have started. + if args.batch < 1: + print(f"error: --batch must be at least 1, got {args.batch}", file=sys.stderr) + return 2 + if args.workers < 1: + print(f"error: --workers must be at least 1, got {args.workers}", file=sys.stderr) + return 2 + connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") @@ -887,6 +907,14 @@ def main() -> int: # numbering most needs repairing. dictionary = b"" if any(phase in ("retype", "migrate") for phase in args.phase_list): + # Fail before any phase runs, not per item inside a worker mid-run. + if shutil.which("brotli") is None: + print( + "error: retype and migrate need the brotli CLI (>= 1.0) on PATH; " + "no Python binding exposes custom dictionaries", + file=sys.stderr, + ) + return 2 if not table_exists(connection, "CompressionDictionary"): print( "error: this database has no CompressionDictionary table, so there is nothing to " @@ -988,7 +1016,16 @@ def select(items: list[Item]) -> list[Item]: for future in futures.as_completed(pending): item = pending[future] - result = future.result() + try: + result = future.result() + except Exception as exc: + # Same reason as the read_blobs guard above: report, do not abort a run + # whose earlier batches are already committed. + counts["error"] += 1 + failed_items.append( + Result(item.base_path, "error", detail=f"{type(exc).__name__}: {exc}") + ) + continue counts[result.status] += 1 before_total += result.before after_total += result.after or result.before From ca5f7e3b16cd64b3592b4d73d1e22c4531c51166 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 22:48:42 +0000 Subject: [PATCH 13/13] ADFA-5153: ungrouped rows keep their own metadata; renumber sets languageID Three fixes in load_items/renumber_item: - A row split back out of a disproved continuation group was built from the base item's languageID/contentTypeID/templateId, so phase 3 relabelled an independent page with the base's MIME type. It now carries its own row's metadata. - Orphaned "-N" rows (whose would-be base is itself a continuation) were logged and dropped, so a plain-brotli row silently survived a run that declares version 2. They now migrate as standalone items; a genuine stray slice fails to decode and feeds the existing "did not migrate" warning instead of vanishing. - renumber_item's UPDATE moved only the path. WebServer loads continuations with "languageId = 1" hardcoded, so a renumbered row under another language stayed invisible and the page still truncated. The UPDATE now normalises languageID like write_item does. Verified on scratch databases: split rows keep type/language/template through a migrate run; a migratable orphan round-trips against the dictionary while an undecodable one exits 1 with the version warning; renumbered continuations end at languageID 1. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0197g8vkUQ1d6oLNbi8EnAYe --- .../migrate_content_to_dictionary_brotli.py | 78 ++++++++++--------- 1 file changed, 43 insertions(+), 35 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 3e5bbdc7b8..e7e750d5df 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -336,10 +336,28 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: ).fetchall() base_bytes_by_path = {row[1]: row[5] for row in rows} + rows_by_id = {row[0]: row for row in rows} items: dict[str, Item] = {} continuations: list[tuple[str, int, int, int]] = [] - for row_id, path, language_id, type_id, template_id, length, type_value, compression in rows: + def standalone(row: tuple) -> Item: + """An Item carrying the row's own metadata -- a row ungrouped later (a disproved + continuation, an orphan) is an independent page, and stamping the would-be base's + contentTypeID/languageID onto it would relabel a foreign page.""" + row_id, path, language_id, type_id, template_id, length, type_value, compression = row + return Item( + base_path=path, + base_id=row_id, + language_id=language_id, + content_type_id=type_id, + template_id=template_id, + content_type=type_value, + compression=compression, + base_bytes=length, + ) + + for row in rows: + row_id, path, length = row[0], row[1], row[5] match = CONTINUATION.match(path) # A "-" sibling is a naming coincidence until the base row proves otherwise, and # the proof is the base holding exactly CHUNK_BYTES -- the app's own chunk-detection rule. @@ -348,16 +366,7 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: if match and base_bytes_by_path.get(match.group(1)) == CHUNK_BYTES: continuations.append((match.group(1), row_id, int(match.group(2)), length)) else: - items[path] = Item( - base_path=path, - base_id=row_id, - language_id=language_id, - content_type_id=type_id, - template_id=template_id, - content_type=type_value, - compression=compression, - base_bytes=length, - ) + items[path] = standalone(row) orphans: list[tuple[str, int, int, int]] = [] for base_path, row_id, suffix, length in continuations: @@ -365,10 +374,10 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: if owner is not None: owner.continuations.append((row_id, suffix, length)) else: - # The greedy base is itself a continuation, or the phase predicate excluded it, so this - # row has no owner to be a slice of. Silently dropping it meant a row that was never - # migrated, never counted and never reported -- in a database the run then declares - # version 2. + # The greedy base is itself a continuation, so this row has no owner to be a slice + # of. Silently dropping it meant a row that was never migrated, never counted and + # never reported -- in a database the run then declares version 2 while the row is + # still plain brotli, which can decode against the dictionary to wrong bytes. orphans.append((base_path, row_id, suffix, length)) for item in items.values(): @@ -386,26 +395,22 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: head = item.continuations[:-1] if all(length == CHUNK_BYTES for _, _, length in head): continue - for row_id, suffix, length in item.continuations: - path = f"{item.base_path}-{suffix}" - items[path] = Item( - base_path=path, - base_id=row_id, - language_id=item.language_id, - content_type_id=item.content_type_id, - template_id=item.template_id, - content_type=item.content_type, - compression=item.compression, - base_bytes=length, - ) + for row_id, _, _ in item.continuations: + row = rows_by_id[row_id] + items[row[1]] = standalone(row) item.continuations.clear() - if orphans: - for base_path, _, suffix, _ in orphans: - print( - f" note: {base_path}-{suffix} looks like a continuation of {base_path}, which is " - f"not itself a migratable row; left alone and not migrated" - ) + # An orphan is still a row this phase selected, so it becomes its own item and migrates + # normally. If it really is a stray slice of some stream, its bytes decode neither plainly + # nor with the dictionary, and it surfaces as an error instead of silently surviving a run + # that declares version 2. + for base_path, row_id, suffix, _ in orphans: + row = rows_by_id[row_id] + items[row[1]] = standalone(row) + print( + f" note: {base_path}-{suffix} looks like a continuation of {base_path}, which is " + f"not itself a migratable row; treated as an independent page" + ) return sorted(items.values(), key=lambda item: item.base_path) @@ -636,8 +641,11 @@ def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> st # fail on UNIQUE(path) after the collision checks above had passed. for row_id, suffix, _ in sorted(item.continuations, key=lambda entry: entry[1]): connection.execute( - "UPDATE Content SET path = ? WHERE id = ?", - (f"{item.base_path}-{suffix - shift}", row_id), + # languageID too, for the same reason write_item normalises it: WebServer loads + # continuations with "languageId = 1" hardcoded, so a renumbered row left under + # another language is invisible and the page still truncates at its first 1 MiB. + "UPDATE Content SET path = ?, languageID = ? WHERE id = ?", + (f"{item.base_path}-{suffix - shift}", CONTINUATION_LANGUAGE_ID, row_id), ) return ""