ADFA-4739: Kotlin docs DB pipeline + Build Kotlin Docs GitHub Action

[alexmmiller]

Summary

An end-to-end pipeline for loading Kotlin website content and
kotlin-stdlib/-reflect/-test JSON docs into documentation.db, plus a
GitHub Action (Build Kotlin Docs) that runs it against a Drive-hosted copy
of the database.

Scope

Split out of #21
into two ticket-scoped PRs:

• ADFA-5039 - producing the raw JSON data for the Kotlin website (md_to_json.py).
• ADFA-4739 (this PR) - the database-manipulation side and the GitHub Action.

This PR depends on ADFA-5039 merging first. populate_db.py,
build_nav.py, and find_missing_assets.py all import md_to_json.py
directly, and that file isn't included here. Please review this PR's
content, but hold off merging until #23 lands - README.md in particular
will very likely conflict with #23's own new README.md at the same path
(both add a new file there) and need a quick manual merge to combine the two
once both are in.

Changes

• ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/: build_nav.py
  (sidebar nav from kr.tree), find_missing_assets.py (source QA),
  populate_db.py (inserts pages/nav/media into documentation.db,
  supports pruning via --blacklisted-element-titles), optimize_media.py
  / insert_optimized_media.py (media optimization + DB update),
  templates/, assets/.
• scripts/sync_kotlin_stdlib_docs/sync_kdoc_json_to_db.py - syncs
  Dokka-generated kotlin-stdlib/-reflect/-test JSON into the database.
• ProcessDocs/ProcessKotlinDocs/run_e2e_pipeline_test.sh - local e2e test
  of the whole pipeline against a scratch copy of the database.
• .github/workflows/build-kotlin-docs.yaml - CI counterpart: pulls
  documentation.db (and Writerside's webHelpImages.zip) from Google
  Drive, runs the same pipeline, uploads the result back.
• Dokka-plugin-kdoc2json/scripts/kotlin/build-stdlib-json-docs.sh +
  build.gradle.kts tweak - builds the kdoc-to-json plugin and generates
  the stdlib JSON docs the sync script consumes.
• CLAUDE.md - repo-level orientation doc (schema drift between this repo's
  tooling and the live production database, repo tour, decisions log).

Test plan

• Once ADFA-5039 is merged and this branch is rebased, re-run
  run_e2e_pipeline_test.sh locally against a scratch database.
• Trigger Build Kotlin Docs with dry_run: true and confirm the
  summary/blacklist-verification steps pass.
• Reviewer confirms the scripts match their originals in ADFA-4739: Pipeline for producing template-based Kotlin documentation #21 (no edits
  made during the split, aside from README.md being trimmed to drop the
  md_to_json.py-only section now covered by ADFA-5039).

[hal-eisen-adfa]

Review scoped to the files that are NEW in this PR. I deliberately skipped populate_db.py, build_nav.py, insert_optimized_media.py, sync_kdoc_json_to_db.py and build-kotlin-docs.yaml — I verified by blob hash that those are byte-identical to #21, where I've already left 11 findings. Everything below is specific to this PR.

Blocking (2):

1. CLAUDE.md makes a factual claim about templateId that the same commit falsifies — and reasons from it to a scope conclusion.
2. nav.peb emits a nav-hidden class that nothing consumes, so entries meant to be hidden render visible.

Non-blocking (3) — real, but lower severity or arguably intentional; flagged for your judgment rather than as merge blockers.

Context worth stating plainly: this PR is a clean split from #21, and the split itself is well executed — README.md correctly documents the ADFA-5039 dependency, and I checked that this PR is not shipping a stale pre-fix version of anything. A number of things I went looking for came back clean: build-stdlib-json-docs.sh's restore trap genuinely fires on both failure and SIGINT (verified with timeout -s INT) and is registered before the file swap, so there's no race window; optimize_media.py handles pngquant failures and animated images with explicit fallback-and-log rather than silent data loss, and returns a non-zero exit on errors; docs.css/sidebar.js/tabs.js/nav.peb agree on breakpoints and class names; and the README's documented CLI invocations match the argparse definitions exactly.

I also considered and did not file the page.peb "lone <tab>" workaround here — the root cause is md_to_json.py's tag regex, which isn't in this PR, and I've already filed it as blocking on #23.

[hal-eisen-adfa]

Blocking — this claim is false as of the commit that adds it, and it's load-bearing.

The parenthetical asserts a repo-wide search for templateId "turns up zero matches outside WebServer.kt", and the surrounding sentence uses that to conclude the concern "is not a gap to fill; it's out of scope for this repo."

But this same commit adds populate_db.py, which references templateId at lines 87, 98, 145, 261, 266, 276 and 286 — including the actual insert:

INSERT INTO Content (path, languageID, content, contentTypeID, templateId) VALUES (...)

and insert_optimized_media.py, which filters on it at lines 214-215 and 256 (... WHERE ... AND templateId != 0). sync_kdoc_json_to_db.py:13 mentions it too.

So the grep this sentence invites the reader to trust returns numerous in-repo hits the moment this PR lands.

What makes this worth blocking rather than a doc nit: CLAUDE.md exists to orient future readers and agents, and this isn't a stale aside — the false premise is used to justify a scope boundary. An agent reading this will conclude the repo doesn't touch templateId and may "helpfully" strip it from exactly the INSERT that needs it.

Suggest either dropping the parenthetical or narrowing it to the Templates/Bookshelf/BookCategories/PUCC names, which I did not find outside WebServer.kt.

[alexmmiller]

Fixed in 66da59d - dropped templateId from the "zero matches outside WebServer.kt" list, keeping only Templates, Bookshelf, BookCategories, PUCC (the names that actually are absent), and added a clarifying note that populate_db.py/insert_optimized_media.py do read/write templateId directly since it's a plain column on Content they populate.

[hal-eisen-adfa]

Blocking — the nav-hidden class is inert, so hidden nav entries render fully visible.

This line emits nav-hidden for nodes the tree marks hidden, and the header comment at line 10 documents hidden as "true for entries Writerside hides from the primary tree." But nothing anywhere consumes the class. Verified:

grep -c 'nav-hidden' assets/docs.css            -> 0
grep -n  'hidden'     assets/sidebar.js          -> (no matches)
grep -c 'nav-hidden' templates/nav.html          -> 29

I read docs.css in full (227 lines): the only display: none rules are for .nav-item > .nav-subtree, .nav-toggle and .nav-backdrop — none for .nav-hidden. And page.peb links only /assets/docs.css, so there's no other stylesheet that could supply the rule, and nothing injects CSS elsewhere.

Concrete effect: the committed nav.html already carries 29 nav-hidden entries — the Kotlin tour steps, e.g. "Hello world" — and every one of them shows in the sidebar identically to a normal entry. Writerside hides those individual tour steps from the primary nav by design, so the shipped sidebar gets cluttered with dozens of entries that were never meant to be listed.

One .nav-hidden { display: none; } rule in docs.css closes it. Worth deciding deliberately though — if these should actually be visible, the class and its comment are the thing to remove.

[alexmmiller]

Fixed in 66da59d - added .nav-hidden { display: none; } to assets/docs.css, matching the documented intent ("entries Writerside hides from the primary tree"). Went with hiding them rather than removing the class.

[hal-eisen-adfa]

Non-blocking — this section describes a branch state that is already out of date.

The text says Dokka-plugin-kdoc2json/ on main is "just a README.md describing the intended design plus a flowchart image; there is no code here yet", that the implementation "exists only on the unmerged branch fix/ADFA-4514", and flags "rebase fix/ADFA-4514 onto current main before merging". Line 183 repeats the rebase item in the decisions log.

fix/ADFA-4514 is already merged — 4c6b8aef ("Merge pull request #18 from appdevforall/fix/ADFA-4514") is on main, and git ls-tree -r main shows JsonOutputPlugin.kt, JsonRenderer.kt, ModelMapper.kt, LinkPostProcessor.kt, the test suite and the kotlin-stdlib-docs build scripts all present on main today.

So a reader is told a whole plugin implementation is missing from main and that a rebase is still outstanding, when both are resolved.

I've marked this non-blocking because it's a point-in-time note that was presumably true when drafted, and unlike the templateId claim above nothing reasons from it. But it's the kind of staleness that a repo-orientation doc is specifically supposed to avoid, and it'll mislead the next agent that reads it.

[alexmmiller]

Fixed in 66da59d - rewrote the Dokka-plugin-kdoc2json/ bullet to describe it as merged (fix/ADFA-4514, 4c6b8aef) with the actual files present on main, and removed the now-resolved rebase item from the decisions log (was line 183).

[hal-eisen-adfa]

Non-blocking — bare pip install / python3, against the repo's own established pattern.

This is the same objection I left as blocking on #23's review_build_json.sh, so flagging it here for consistency rather than re-litigating it.

Our convention (~/.claude/CLAUDE.md) is explicit: "Always use uv for Python project management. Never use pip, pip3, or the uv pip compatibility shim", and "never invoke python/python3 directly outside a uv run context."

In this PR that shows up in several places:

• these two lines (pip install markdown-it-py Pillow scour brotli, pip install cairosvg)
• README lines 53, 56, 59, 72, 90, 109, 115 — every usage example is python3 <script>.py ...
• optimize_media.py:47-48 (docstring) and :302 (a runtime RuntimeError telling the user to pip install cairosvg)
• run_e2e_pipeline_test.sh lines 100, 105, 125, 141, 145, 175 — six bare python3 invocations

This isn't only a global-preference thing: docdb-studio/ and check-tools/ in this same repo each ship a pyproject.toml + uv.lock and document uv sync / uv run, and docdb-studio/CLAUDE.md says "Dependencies are managed with uv. Always use uv run."

Related and independently worth fixing: scour and cairosvg are not in requirements.txt (unchanged by this PR), so they exist only in these prose instructions.

I've left this non-blocking because these are developer-facing helper scripts and you may have deliberately kept them dependency-light — but if so, that's worth a line in the README saying so.

[alexmmiller]

Non-blocking, addressed in 66da59d rather than just documented as intentional - converted every pip install/bare python3 reference (README usage examples, optimize_media.py's docstring + RuntimeError message, and all 6 invocations in run_e2e_pipeline_test.sh) to uv run --with-requirements <repo-root>/requirements.txt, matching the review_build_json.sh fix on #23. Also added scour/cairosvg to requirements.txt since they were only ever in prose.

[hal-eisen-adfa]

Non-blocking — per-file scan failures are swallowed and the script still exits 0.

The except catches everything from converter.convert_file(...), prints to stderr, and continues — but nothing counts the failures, and main() has no sys.exit(1) path except the directory-validation check at line 82. So if every file in the corpus raised, this still exits 0 and prints a report claiming nothing is broken.

That matters because of how the script is positioned: it's step 1/5 in run_e2e_pipeline_test.sh:100, and the README describes it as the pre-flight gate where you "fix anything broken in the source before converting it." set -euo pipefail in the shell script can't help — exit 0 is exit 0.

Same shape I flagged as blocking on populate_db.py:483 (#21) and md_to_json.py:607 (#23). I've scored it lower here because this tool only writes an advisory markdown report and doesn't mutate the database, so a bad run misleads rather than corrupts.

Counting failures and exiting non-zero — or at minimum printing a N files failed to scan line into the report itself — would make a broken run distinguishable from a clean one.

[alexmmiller]

Fixed in 66da59d - added a failure counter and --allow-failures flag mirroring md_to_json.py's pattern from #23: sys.exit(1) if any file failed to scan (unless the flag is passed), and a "N file(s) failed to scan" line (with an explicit "this report is incomplete" marker when non-zero) in the emitted report itself. Covered by new tests in tests/test_find_missing_assets.py, which stub the not-yet-merged md_to_json import via PYTHONPATH since that module isn't on this branch until #23 lands.

[alexmmiller]

Pushed 66da59d addressing your review. Replied inline to each of the 5 line comments individually; summary here.

Blocking (2)

1. CLAUDE.md's templateId "out of scope" claim was false as of this same PR (line 88) - dropped templateId from the "zero matches outside WebServer.kt" parenthetical, keeping only Templates/Bookshelf/BookCategories/PUCC (the names that actually are absent), and added a note that populate_db.py/insert_optimized_media.py do read/write templateId directly.
2. nav-hidden was inert (templates/nav.peb:38) - added .nav-hidden { display: none; } to assets/docs.css, so Writerside-hidden nav entries (the 29 tour-step cases you found in the committed nav.html) are actually hidden.

Non-blocking (3)

3. Stale Dokka-plugin-kdoc2json branch state (CLAUDE.md:150-156, decisions log line 183) - rewrote the bullet to describe the plugin as merged (fix/ADFA-4514, 4c6b8aef) with the actual files present on main, and removed the resolved rebase item from the decisions log.
4. pip/bare python3 vs. the repo's uv convention - converted every usage example in README.md, optimize_media.py's docstring + RuntimeError message, and all 6 invocations in run_e2e_pipeline_test.sh to uv run --with-requirements <repo-root>/requirements.txt. Also added scour/cairosvg to requirements.txt, since they'd only ever existed in prose.
5. find_missing_assets.py swallowed per-file scan failures and exited 0 regardless (line 98) - added a failure counter, a --allow-failures opt-out flag, sys.exit(1) on any scan failure, and a "N file(s) failed to scan" line in the report itself (with an explicit "this report is incomplete" marker when non-zero) - mirroring the pattern md_to_json.py uses on ADFA-5039: Convert kotlin-web-site docs to JSON #23.

Tests

Added tests/test_find_missing_assets.py covering item 5's exit-code/report behavior. Since md_to_json.py isn't on this branch yet (lands with #23), the tests stub that import via PYTHONPATH and run the script as a subprocess rather than importing it directly. 3/3 pass.

Verified the uv run --with-requirements invocations actually work (installed uv locally and ran optimize_media.py --help plus the heredoc pattern used in the e2e script through it) and that run_e2e_pipeline_test.sh is still syntactically valid (bash -n). Full end-to-end execution of find_missing_assets.py/populate_db.py still isn't possible here until #23 merges, per the PR description.

Sent prematurely, apologies

[alexmmiller]

Review: database-insertion path

Focused pass over the files that write to documentation.db. Four things on this branch do:

File	What it writes
ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/populate_db.py	wipes and rebuilds k/html/* + assets/*
ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/insert_optimized_media.py	replaces k/html/images/*, rewrites pages, deletes unreferenced media
scripts/sync_kotlin_stdlib_docs/sync_kdoc_json_to_db.py	overwrites/deletes k/kotlin-{stdlib,reflect,test}/* + orphaned tooltips
.github/workflows/build-kotlin-docs.yaml:244	one inline INSERT OR IGNORE INTO ContentTypes

optimize_media.py, build_nav.py, and find_missing_assets.py never open a connection.

Two things that are right, up front

Content types are handled correctly throughout. Every type is resolved by value through get_content_type/get_id, which raise if the row is missing; extension mapping is an explicit allowlist (EXTENSION_TO_CONTENT_TYPE) that skips-and-warns on an unknown extension rather than falling back to text/plain; compression is read from ContentTypes.compression rather than assumed. There is not one hardcoded contentTypeID in the PR. This matters because ~34% of the .html rows in the current production DB are typed text/plain, caused by exactly the two anti-patterns this PR avoids (a text/plain fallback for unrecognized extensions, and hardcoded positional type IDs). The text/html-for-JSON-storage choice at populate_db.py:149 is deliberate and correct — templateId points the server at page.peb, so the served type is HTML.

The big DELETE is correctly scoped. populate_db.py:530's DELETE FROM Content WHERE path LIKE 'k/html/%' OR path LIKE 'assets/%' hits 489 rows against the current production DB (267 pages + nav, 222 images) plus the 3 assets — exactly what the script reinserts. The 3,264 k/ rows outside k/html/ (stdlib, disambiguation) are untouched.

Findings

1. HIGH — a conversion failure silently drops a live page and leaves a normal-looking nav link to it. populate_db.py:483

convert_file failures are caught, logged to stderr, and skipped, so the page never reaches pages and no Content row is inserted — but line 530 already deleted the old, working row in the same transaction, which then commits. Worse, topic_index_db still contains the stem (only blacklisted stems get popped at line 460), so build_node resolves it, sets no_link = False, and emits a normally-styled <a href="/k/html/<stem>.html"> with a humanize()-derived title. Net effect of one bad .md: a page vanishes from production and the sidebar links to a 404 in the same styling as every working page.

This is a regression against the standalone path — build_nav.py's load_page_index builds stem_to_id from generated JSON, so a failed page drops out of nav automatically. The in-process variant loses that property.

Minimum fix: topic_index_db.pop(md_path.stem, None) in the except, matching the blacklist path so links render styled-broken. Better: exit non-zero on any conversion failure, since CI uploads this DB to Drive and nothing currently checks that Converted {len(pages)}/{len(md_files)} is a clean ratio.

2. MEDIUM-HIGH — duplicate .md stems abort the whole run. populate_db.py:475-479

md_files iterates every *.md and derives db_id = f"k/html/{md_path.stem}". Content.path is UNIQUE (UNIQUE('path') in the live schema does enforce — SQLite resolves the quoted string as an identifier; verified), so two same-stem files in different topics/ subdirectories mean two inserts at one path → IntegrityError → rollback → raw traceback, after the full conversion pass has already run. md_to_json.build_topic_index handles exactly this case with keep-first-plus-warning, and topic_index_db inherits that dedupe — but the conversion loop doesn't. It fails safe (nothing uploaded), but it should dedupe by stem the same way rather than crash on an upstream rename.

3. MEDIUM — delete_unreferenced_media is an unguarded mass delete with no dry-run. insert_optimized_media.py:290

It deletes every k/html/images/* row not referenced by a k/html/%.html page with templateId != 0. In CI that's safe — steps 2 and 3 share if: !inputs.skip_website_docs, so pages always exist. Run standalone against a DB without those page rows and collect_referenced_media returns an empty set, so it deletes every image, including the ones inserted seconds earlier in the same transaction. There's no floor check (if not referenced: abort) and, unlike sync_kdoc_json_to_db.py, this script has no --dry-run — the most destructive of the three is the one without a preview mode.

Related blind spot: only page/nav rows are scanned, so an image referenced from assets/docs.css or a .peb template would be collected as garbage. Latent today (no url() in docs.css, no image refs in either template — checked), but it's a silent-data-loss trap for whoever adds a CSS background later.

4. MEDIUM — sync_kdoc_json_to_db.py ignores the chunking contract the rest of the PR treats as mandatory. sync_kdoc_json_to_db.py:164

It does UPDATE Content SET content = ? with the entire new blob and never chunks. Meanwhile any existing <path>-N fragment matches the prefix LIKE, fails relative_target_path's file lookup, and gets deleted. So a stdlib page crossing 1 MiB produces one oversized row that no longer matches WebServer.kt's reassembly protocol — the same protocol populate_db.py:145 argues has to be byte-exact on both sides. Latent right now (largest stdlib blob is 172 KB, and none of the 44 chunked rows in the DB are under these prefixes), but the two scripts disagree on an invariant the PR itself documents as non-negotiable.

5. MEDIUM — unknown contentTypeID silently writes uncompressed bytes and commits. sync_kdoc_json_to_db.py:154-157

compression_by_type.get(...) missing → compression = "none" → raw bytes stored in a row whose declared type says brotli. The warning goes to stderr, the transaction commits, and the server gets undecompressable content. Given that a mistyped content column already shipped to production undetected for months, this should be fatal, not a warning.

6. LOW-MEDIUM — delete-after-insert in the rename loop can remove a just-inserted row. insert_optimized_media.py:415

Inserts run first, then for old_name in rename_map: delete_content(...). If one rename's new name equals another's old name (a.png → b.webp alongside b.webp → c.webp), the delete pass removes the row the insert pass just wrote from a.png. This is precisely the chain-rename hazard rewrite_pages' docstring reasons about carefully for text substitution, and the same reasoning wasn't applied here. Unlikely with Writerside sources; the fix is just moving the delete loop above the insert loop.

7. LOW — unescaped LIKE wildcards. insert_optimized_media.py:103

delete_content does path LIKE ? with f"{path}-%" built from a filename. _ and % are wildcards. No current image name contains either, but NAV_CONTENT_PATH is k/html/_nav.html, so the pattern is already k/html/_nav.html-% with a live wildcard in it. Add ESCAPE '\' and escape the parameter. Every other LIKE in the PR uses literal prefixes and is fine.

8. LOW — image_index_db assumes a flat zip and would break silently otherwise. populate_db.py:467

{name: name for name in image_names} keys on the full zip entry, but Converter.resolve_image_src looks up src.rsplit("/", 1)[-1], and md_to_json.build_image_index maps bare-name → rel-path. A nested zip entry gives a key nothing can match, so images resolve as missing while the row lands at k/html/images/sub/foo.png. {Path(name).name: name} makes it match the documented contract at no cost.

9. LOW — the three scripts back up two different ways. populate_db.py:212 and insert_optimized_media.py use VACUUM INTO (documented as WAL-safe); sync_kdoc_json_to_db.py:39 uses shutil.copy2. The live DB is journal_mode=delete, so this isn't currently biting, but the PR's own docstring makes the argument for VACUUM INTO and then one script doesn't follow it. Also worth noting all three run in CI against the same ~300 MB file, so a run keeps three full backup copies on the runner.

Nits

• build-kotlin-docs.yaml:151 says scour/cairosvg are "not in the root requirements.txt" — this same PR adds them to requirements.txt, so they're installed twice and the comment is stale.
• .gitignore's *.db doesn't match documentation.db.backup-<ts> or documentation.db.bak.<ts>, so every local run leaves untracked noise in git status.

Merge order

The PR body's caveat holds — populate_db.py, build_nav.py, and find_missing_assets.py all import md_to_json, which isn't in this diff, so nothing here is runnable until #23 lands. This review was done against #23's copy of md_to_json.py.