ADFA-5153: Compress Content table with a trained Brotli dictionary

.github/workflows/build-kotlin-docs.yaml

@@ -143,7 +143,10 @@ jobs:
       - name: Install system dependencies
         run: |
           sudo apt-get update -y
-          sudo apt-get install -y pngquant unzip zip sqlite3
+          # brotli: the CLI, not the Python package. populate_db.py's
+          # DictionaryCompressor and sync_kdoc_json_to_db.py shell out to it because
+          # no Python binding exposes a custom dictionary (ADFA-5153).
+          sudo apt-get install -y pngquant unzip zip sqlite3 brotli
 
       - name: Install Python dependencies
         run: |

.github/workflows/docdb-regression-test.yaml

@@ -98,7 +98,9 @@ jobs:
         echo "Extracting database from zip file..."
 
         # Install unzip if not available
-        sudo apt-get update -qq && sudo apt-get install -y unzip
+        # brotli: docdb_studio reads dictionary-compressed Content rows through the
+        # CLI (ADFA-5153); the downloaded production database is one of those.
+        sudo apt-get update -qq && sudo apt-get install -y unzip brotli
 
         # Extract the zip file
         if ! unzip -o documentation.zip; then

ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md

@@ -13,6 +13,10 @@ and its media straight into a `documentation.db`-schema SQLite database.
 | [`build_nav.py`](build_nav.py) | Builds `nav.json`/`nav.html` sidebar navigation from `kr.tree`, resolving each `<toc-element topic="...">` against `md_to_json.py`'s output. |
 | [`find_missing_assets.py`](find_missing_assets.py) | QA pass: reports cross-page links, images, and `<include>` targets in the source tree that don't resolve to anything. Reuses `md_to_json.py`'s own resolution logic, so it flags exactly what would end up broken on the rendered site. |
 | [`populate_db.py`](populate_db.py) | The database path: converts the docs tree the same way `md_to_json.py` does, builds nav the same way `build_nav.py` does, and inserts pages + nav + images + CSS/JS directly into `documentation.db` (replacing everything under `k/html/` and `assets/`). Supports pruning whole `kr.tree` subtrees via `--blacklisted-element-titles`. |
+| [`migrate_content_to_dictionary_brotli.py`](migrate_content_to_dictionary_brotli.py) | One-off, resumable: recompresses every `brotli` Content row against the database's shared `CompressionDictionary`, training one first if there is none (ADFA-5153). Covers the rows `populate_db.py` never touches. |
+| [`renumber_misnumbered_fragments.py`](renumber_misnumbered_fragments.py) | One-off repair: chunked rows whose continuations start at `-2` (or `-0`) instead of `-1`, which `WebServer.kt` reassembles truncated (ADFA-5171). Moves paths only, never content. |
+| [`remint_dictionary.py`](remint_dictionary.py) | One-off, **destructive**: trains a *new* shared dictionary and recompresses every row against it, in one transaction. The only safe way to change a dictionary, since the stored one is otherwise permanent for that database's content. Pair with `verify_remint_dictionary.py` before putting the result in place. |
+| [`verify_remint_dictionary.py`](verify_remint_dictionary.py) | Read-only gate for the above: decodes every row out of both databases and requires the plaintexts to match, exiting non-zero otherwise. A mismatched dictionary decodes into wrong bytes without erroring, so this is what makes re-minting safe. |
 | [`optimize_media.py`](optimize_media.py) | Standalone media optimizer: downscales/recompresses a directory of images (pngquant, Pillow, Scour/cairosvg for SVG) into a mirrored output directory. |
 | [`insert_optimized_media.py`](insert_optimized_media.py) | Runs `optimize_media.py`'s pipeline over a directory of raw media, then replaces the corresponding `k/html/images/*` rows in an existing database, rewriting any page that referenced a renamed file and deleting anything left unreferenced. |
 
@@ -22,6 +26,7 @@ and its media straight into a `documentation.db`-schema SQLite database.
 - `pip install markdown-it-py Pillow scour brotli`
 - `cairosvg` (only needed if an optimized SVG exceeds `--svg-rasterize-threshold`): `pip install cairosvg`
 - `pngquant` on `PATH` (e.g. `apt install pngquant`) — required by `optimize_media.py`/`insert_optimized_media.py`, and by `populate_db.py` for the images it inserts directly from the Writerside export.
+- `brotli` on `PATH` (e.g. `apt install brotli`) — the **command-line tool**, which is a different artifact from the `brotli` Python package listed above. `populate_db.py`, `insert_optimized_media.py`, `migrate_content_to_dictionary_brotli.py` and `remint_dictionary.py` compress against the shared dictionary in `CompressionDictionary` (ADFA-5153), and no Python binding exposes a custom dictionary, so they shell out to this binary. Without it they fail at startup.
 
 `populate_db.py` also expects, relative to its own location, and already
 included in this directory:

ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/insert_optimized_media.py

@@ -62,15 +62,14 @@
 import tempfile
 from pathlib import Path
 
-import brotli
-
 from optimize_media import (
     BUILTIN_DEFAULTS, Logger, OPTION_SPECS, add_optimize_arguments, find_pngquant, optimize_directory,
     resolve_config,
 )
 from populate_db import (
-    CHUNK_SIZE, EXTENSION_TO_CONTENT_TYPE, IMAGES_DB_PATH_PREFIX, IMAGES_URL_PREFIX, LANGUAGE, PAGE_CONTENT_TYPE,
-    backup_database, get_content_type, get_id, insert_chunked_content,
+    CHUNK_SIZE, DictionaryCompressor, EXTENSION_TO_CONTENT_TYPE, IMAGES_DB_PATH_PREFIX, IMAGES_URL_PREFIX,
+    LANGUAGE, PAGE_CONTENT_TYPE, backup_database, fragment_chain, get_content_type, get_id,
+    insert_chunked_content, load_dictionary,
 )
 
 WEBP_CONTENT_TYPE = "image/webp"
@@ -104,12 +103,22 @@ def delete_content(conn, path: str) -> None:
     """Deletes a Content row and any chunked continuation fragments for it
     (see insert_chunked_content/CHUNK_SIZE) - safe to call even if nothing
     exists yet at that path. Content.path is UNIQUE, so this has to run
-    before any re-insert at the same path."""
-    conn.execute("DELETE FROM Content WHERE path = ? OR path LIKE ?", (path, f"{path}-%"))
+    before any re-insert at the same path.
+
+    Deletes by exact path rather than by a LIKE pattern. `_` is a single-
+    character wildcard in LIKE and the `-%` suffix does not restrict the tail to
+    digits, so "DELETE ... WHERE path LIKE '<path>-%'" also removes rows that
+    merely resemble a continuation - and those are never re-inserted, so the
+    loss is permanent. populate_db.fragment_chain does the over-matching query
+    once and re-checks every candidate's suffix, which is what makes the result
+    exact."""
+    conn.execute("DELETE FROM Content WHERE path = ?", (path,))
+    for _number, fragment_path in fragment_chain(conn, path):
+        conn.execute("DELETE FROM Content WHERE path = ?", (fragment_path,))
 
 
 def insert_optimized_file(conn, data: bytes, name: str, db_path: str, language_id: int, content_type_cache: dict,
-                           chunked_log: list) -> bool:
+                           chunked_log: list, compressor: DictionaryCompressor) -> bool:
     """Inserts one already-optimized file's bytes as-is. Unlike
     populate_db.py's own insert_file, this does not run pngquant itself -
     optimize_media.py already did, and running it again here would just
@@ -125,7 +134,7 @@ def insert_optimized_file(conn, data: bytes, name: str, db_path: str, language_i
     content_type_id, compress = content_type_cache[content_type_value]
 
     if compress:
-        data = brotli.compress(data)
+        data = compressor.compress(data)
     delete_content(conn, db_path)
     insert_chunked_content(conn, db_path, language_id, content_type_id, 0, data, chunked_log)
     return True
@@ -175,7 +184,7 @@ def reassemble_content(conn, path: str, first_content: bytes) -> bytes:
 
 
 def rewrite_pages(conn, rename_map: dict, language_id: int, page_content_type_id: int, logger: Logger,
-                   chunked_log: list) -> int:
+                   chunked_log: list, compressor: DictionaryCompressor) -> int:
     """Rewrites every k/html/*.html page (and the nav row) that references a
     renamed image, replacing "/k/html/images/<old-name>" with
     "/k/html/images/<new-name>" wherever it appears. Operates directly on
@@ -225,12 +234,12 @@ def rewrite_pages(conn, rename_map: dict, language_id: int, page_content_type_id
     changed = 0
     for path, first_content, template_id in rows:
         full = reassemble_content(conn, path, first_content)
-        text = brotli.decompress(full).decode("utf-8")
+        text = compressor.decompress(full).decode("utf-8")
         hits = len(old_ref_pattern.findall(text))
         if not hits:
             continue
         new_text = old_ref_pattern.sub(lambda m: replacements[m.group(0)], text)
-        blob = brotli.compress(new_text.encode("utf-8"))
+        blob = compressor.compress(new_text.encode("utf-8"))
         delete_content(conn, path)
         insert_chunked_content(conn, path, language_id, page_content_type_id, template_id, blob, chunked_log)
         changed += 1
@@ -246,7 +255,7 @@ def rewrite_pages(conn, rename_map: dict, language_id: int, page_content_type_id
 IMAGE_REF_RE = re.compile(re.escape(IMAGES_URL_PREFIX) + r'([^\\"]+)\\"')
 
 
-def collect_referenced_media(conn, page_content_type_id: int) -> set:
+def collect_referenced_media(conn, page_content_type_id: int, compressor: DictionaryCompressor) -> set:
     """Bare filenames (e.g. "mascot.png") referenced by at least one
     src="/k/html/images/<name>" anywhere across current k/html/*.html page
     content and the nav row - the same row selection/reassembly
@@ -259,7 +268,7 @@ def collect_referenced_media(conn, page_content_type_id: int) -> set:
     referenced = set()
     for path, first_content in rows:
         full = reassemble_content(conn, path, first_content)
-        text = brotli.decompress(full).decode("utf-8")
+        text = compressor.decompress(full).decode("utf-8")
         referenced.update(IMAGE_REF_RE.findall(text))
     return referenced
 
@@ -287,7 +296,8 @@ def is_fragment(path: str) -> bool:
     return {path[len(IMAGES_DB_PATH_PREFIX):]: path for path in paths if not is_fragment(path)}
 
 
-def delete_unreferenced_media(conn, page_content_type_id: int, logger: Logger) -> int:
+def delete_unreferenced_media(conn, page_content_type_id: int, logger: Logger,
+                               compressor: DictionaryCompressor) -> int:
     """Deletes every currently-stored k/html/images/<name> row (base row and
     any chunked fragments) that no page or the nav row references even once.
     Must run after insertion and rename-rewriting, so it sees the final,
@@ -296,7 +306,7 @@ def delete_unreferenced_media(conn, page_content_type_id: int, logger: Logger) -
     rewrite_pages will have already fixed up by the time this runs. Returns
     the number of images removed."""
     stored = list_stored_media(conn)
-    referenced = collect_referenced_media(conn, page_content_type_id)
+    referenced = collect_referenced_media(conn, page_content_type_id, compressor)
     removed = 0
     for name, path in sorted(stored.items()):
         if name in referenced:
@@ -384,44 +394,56 @@ def main() -> None:
             conn.execute("BEGIN")
             language_id = get_id(conn, "Languages", LANGUAGE)
             page_content_type_id = get_id(conn, "ContentTypes", PAGE_CONTENT_TYPE)
+            # This script only ever runs against a database populate_db.py
+            # already populated (see module docstring), so its
+            # CompressionDictionary must already exist - never train a new
+            # one here, since that would orphan every row already
+            # compressed against the existing one (see DictionaryCompressor).
+            compressor = DictionaryCompressor(load_dictionary(conn))
 
             content_type_cache = {}
             chunked_log = []
             inserted = 0
             seen_names = {}
-            for out_path in sorted(work_dir.rglob("*")):
-                if out_path.is_dir():
-                    continue
-                name = out_path.name
-                if name in seen_names:
-                    logger.error(
-                        f"warning: {out_path} has the same filename as {seen_names[name]}; keeping the first, "
-                        "skipping this one"
-                    )
-                    continue
-                seen_names[name] = out_path
-                db_path = f"{IMAGES_DB_PATH_PREFIX}{name}"
-                if insert_optimized_file(conn, out_path.read_bytes(), name, db_path, language_id, content_type_cache,
-                                          chunked_log):
-                    inserted += 1
+            try:
+                for out_path in sorted(work_dir.rglob("*")):
+                    if out_path.is_dir():
+                        continue
+                    name = out_path.name
+                    if name in seen_names:
+                        logger.error(
+                            f"warning: {out_path} has the same filename as {seen_names[name]}; keeping the first, "
+                            "skipping this one"
+                        )
+                        continue
+                    seen_names[name] = out_path
+                    db_path = f"{IMAGES_DB_PATH_PREFIX}{name}"
+                    if insert_optimized_file(conn, out_path.read_bytes(), name, db_path, language_id,
+                                              content_type_cache, chunked_log, compressor):
+                        inserted += 1
+                        if cfg["verbose"]:
+                            logger.info(f"[OK] {out_path} -> {db_path}")
+
+                # A renamed file's old basename no longer appears anywhere under
+                # work_dir (that's what makes it a rename), so the loop above
+                # never visits its old db_path to replace it - it'd otherwise
+                # linger forever as an orphaned, no-longer-referenced row.
+                removed = 0
+                for old_name in rename_map:
+                    old_db_path = f"{IMAGES_DB_PATH_PREFIX}{old_name}"
+                    delete_content(conn, old_db_path)
+                    removed += 1
                     if cfg["verbose"]:
-                        logger.info(f"[OK] {out_path} -> {db_path}")
-
-            # A renamed file's old basename no longer appears anywhere under
-            # work_dir (that's what makes it a rename), so the loop above
-            # never visits its old db_path to replace it - it'd otherwise
-            # linger forever as an orphaned, no-longer-referenced row.
-            removed = 0
-            for old_name in rename_map:
-                old_db_path = f"{IMAGES_DB_PATH_PREFIX}{old_name}"
-                delete_content(conn, old_db_path)
-                removed += 1
-                if cfg["verbose"]:
-                    logger.info(f"[REMOVED] {old_db_path} (renamed to {IMAGES_DB_PATH_PREFIX}{rename_map[old_name]})")
-
-            changed_pages = rewrite_pages(conn, rename_map, language_id, page_content_type_id, logger, chunked_log)
-
-            unreferenced_removed = delete_unreferenced_media(conn, page_content_type_id, logger)
+                        logger.info(
+                            f"[REMOVED] {old_db_path} (renamed to {IMAGES_DB_PATH_PREFIX}{rename_map[old_name]})"
+                        )
+
+                changed_pages = rewrite_pages(conn, rename_map, language_id, page_content_type_id, logger,
+                                               chunked_log, compressor)
+
+                unreferenced_removed = delete_unreferenced_media(conn, page_content_type_id, logger, compressor)
+            finally:
+                compressor.close()
 
             conn.commit()
         except Exception: