diff --git a/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md index 47008e27..67cb27da 100644 --- a/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md +++ b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md @@ -1,69 +1,37 @@ # Process Kotlin Website JSON -Scripts for converting a `kotlin-web-site/docs` checkout (JetBrains -Writerside-flavored Markdown) into the JSON block schema this project's -templating engine renders, and for loading that JSON, its navigation tree, -and its media straight into a `documentation.db`-schema SQLite database. +Converts a `kotlin-web-site/docs` checkout (JetBrains Writerside-flavored +Markdown) into the JSON block schema this project's templating engine +renders. -## Scripts - -| Script | Purpose | -|---|---| -| [`md_to_json.py`](md_to_json.py) | Converts every `topics/**/*.md` page into one JSON file (see schema below). Writes `theme.json` and copies `images/` into the output directory. | -| [`build_nav.py`](build_nav.py) | Builds `nav.json`/`nav.html` sidebar navigation from `kr.tree`, resolving each `` against `md_to_json.py`'s output. | -| [`find_missing_assets.py`](find_missing_assets.py) | QA pass: reports cross-page links, images, and `` targets in the source tree that don't resolve to anything. Reuses `md_to_json.py`'s own resolution logic, so it flags exactly what would end up broken on the rendered site. | -| [`populate_db.py`](populate_db.py) | The database path: converts the docs tree the same way `md_to_json.py` does, builds nav the same way `build_nav.py` does, and inserts pages + nav + images + CSS/JS directly into `documentation.db` (replacing everything under `k/html/` and `assets/`). Supports pruning whole `kr.tree` subtrees via `--blacklisted-element-titles`. | -| [`migrate_content_to_dictionary_brotli.py`](migrate_content_to_dictionary_brotli.py) | One-off, resumable: recompresses every `brotli` Content row against the database's shared `CompressionDictionary`, training one first if there is none (ADFA-5153). Covers the rows `populate_db.py` never touches. | -| [`renumber_misnumbered_fragments.py`](renumber_misnumbered_fragments.py) | One-off repair: chunked rows whose continuations start at `-2` (or `-0`) instead of `-1`, which `WebServer.kt` reassembles truncated (ADFA-5171). Moves paths only, never content. | -| [`remint_dictionary.py`](remint_dictionary.py) | One-off, **destructive**: trains a *new* shared dictionary and recompresses every row against it, in one transaction. The only safe way to change a dictionary, since the stored one is otherwise permanent for that database's content. Pair with `verify_remint_dictionary.py` before putting the result in place. | -| [`verify_remint_dictionary.py`](verify_remint_dictionary.py) | Read-only gate for the above: decodes every row out of both databases and requires the plaintexts to match, exiting non-zero otherwise. A mismatched dictionary decodes into wrong bytes without erroring, so this is what makes re-minting safe. | -| [`optimize_media.py`](optimize_media.py) | Standalone media optimizer: downscales/recompresses a directory of images (pngquant, Pillow, Scour/cairosvg for SVG) into a mirrored output directory. | -| [`insert_optimized_media.py`](insert_optimized_media.py) | Runs `optimize_media.py`'s pipeline over a directory of raw media, then replaces the corresponding `k/html/images/*` rows in an existing database, rewriting any page that referenced a renamed file and deleting anything left unreferenced. | +This PR (ADFA-5039) covers only [`md_to_json.py`](md_to_json.py) — the +conversion step itself. Building the sidebar nav from `kr.tree` +(`build_nav.py`), QA-ing the source tree for broken links/images +(`find_missing_assets.py`), and loading any of this into `documentation.db` +(`populate_db.py`, `insert_optimized_media.py`) are a separate ticket +(ADFA-4739) and land in a later PR. ## Requirements - Python 3.10+ -- `pip install markdown-it-py Pillow scour brotli` -- `cairosvg` (only needed if an optimized SVG exceeds `--svg-rasterize-threshold`): `pip install cairosvg` -- `pngquant` on `PATH` (e.g. `apt install pngquant`) — required by `optimize_media.py`/`insert_optimized_media.py`, and by `populate_db.py` for the images it inserts directly from the Writerside export. -- `brotli` on `PATH` (e.g. `apt install brotli`) — the **command-line tool**, which is a different artifact from the `brotli` Python package listed above. `populate_db.py`, `insert_optimized_media.py`, `migrate_content_to_dictionary_brotli.py` and `remint_dictionary.py` compress against the shared dictionary in `CompressionDictionary` (ADFA-5153), and no Python binding exposes a custom dictionary, so they shell out to this binary. Without it they fail at startup. - -`populate_db.py` also expects, relative to its own location, and already -included in this directory: +- `markdown-it-py` (now in the repo's root `requirements.txt`) -- `templates/page.peb`, `templates/nav.peb` — Pebble templates upserted into the `Templates` table. -- `assets/docs.css`, `assets/tabs.js`, `assets/sidebar.js` — static assets inserted at `assets/`. +## Usage -## Inputs you need before starting +```bash +python3 md_to_json.py [--topics-subdir topics] [--images-subdir images] [--allow-failures] +``` -- A checkout of `kotlin-web-site/docs` (the `` argument below) — contains `topics/`, `images/`, `v.list`, and `kr.tree`. -- A config JSON with theming colors, e.g.: +- `` — a checkout of `kotlin-web-site/docs` (contains `v.list`, `topics/`, `images/`). +- `` — a JSON file with theming colors, e.g. [`config.json`](config.json): ```json {"broken-ext-link-color": "#cc0000", "menu-no-link-color": "#999999"} ``` -- Writerside's own image export zip (e.g. `webHelpImages.zip`, found next to `kr.tree`) if you're using `populate_db.py`. - -## Workflow: generate JSON + nav for a static/templated preview - -Use this to produce standalone JSON pages and nav data (not the database) -for local inspection or a different renderer. - -```bash -# 1. Convert every topic .md into JSON, one file per page -python3 md_to_json.py config.json - -# 2. Build the sidebar nav from kr.tree against that JSON output -python3 build_nav.py - -# 3. (optional) Check for broken links/images/includes in the source tree -python3 find_missing_assets.py missing-assets-report.md -``` `` ends up containing: - `topics/**/*.json` — one page per source `.md` file (schema below) -- `theme.json` — the two theming colors, carried from `config.json` +- `theme.json` — the two theming colors, carried through from `` - `images/` — copied straight from `/images/` -- `nav.json` / `nav.html` — sidebar tree and a pre-rendered static copy ### Page JSON schema @@ -77,77 +45,30 @@ python3 find_missing_assets.py missing-assets-report.md ``` Block types: `heading`, `paragraph`, `code`, `blockquote`, `list`, `table`, -`image`, `hr`, `tabs`, `note`/`tip`/`warning`, `html` (raw passthrough). See -the module docstring in [`md_to_json.py`](md_to_json.py) for full shapes and +`hr`, `tabs`, `note`/`tip`/`warning`, `html` (raw passthrough). See the +module docstring in [`md_to_json.py`](md_to_json.py) for full shapes and known limitations (nested tabs, `` resolution, variable -substitution). - -## Workflow: generate + insert directly into the documentation database - -This is the path that actually populates `documentation.db`. It performs -the same conversion as `md_to_json.py`/`build_nav.py` internally — you don't -run those scripts first. +substitution). There is no standalone `image` block type - an image is +always inline content inside whatever block contains it (typically +`paragraph`), rendered straight into that block's own `html` string. + +A heading's `id` is `slugify()`'d from its text, unless the source line has +an explicit `{id="..."}` (which overrides it directly). Cross-page links +that carry a source `#anchor` are passed through verbatim rather than +re-slugified, so a link and its target agree as long as both derive their id +the same way; a hand-written `#anchor` that doesn't match either path (e.g. +because Writerside's own anchor algorithm diverges from `slugify()` on +headings with inline code or punctuation) will resolve to the right page but +land on no anchor. Not currently detected - worth spot-checking if a page's +in-page anchors stop scrolling to the right place. + +## Trying it out + +[`review_build_json.sh`](review_build_json.sh) is a throwaway helper for +reviewers — it clones `kotlin-web-site` and runs `md_to_json.py` against it +via `uv run` so you can look at real output without any other setup. It's +not part of the actual pipeline (that's ADFA-4739): ```bash -python3 populate_db.py config.json [db-path] +./review_build_json.sh ``` - -- `db-path` defaults to `documentation.db` in the current directory, and must already exist with the expected schema (`Languages`, `ContentTypes`, `Templates` tables populated). -- A timestamped backup (`.backup-`) is written before any changes, via SQLite's `VACUUM INTO`. -- Everything under `k/html/` and `assets/` is deleted and re-inserted in a single transaction (rolled back on error), then the database is `VACUUM`ed. - -### Pruning documentation you don't want (ADFA-4737) - -To leave a whole `kr.tree` subtree out of the database entirely — nav -entry, converted pages, and all — pass `--blacklisted-element-titles` with -the full `toc-title` path from a top-level element down to the one you want -to drop. Levels are joined with `\/` (backslash-slash), not a bare `/`, -since a bare `/` commonly appears inside a real title. The example below is -illustrative only — open `/kr.tree` and copy the actual -`toc-title` chain for whatever section you're dropping (e.g. Kotlin/Wasm): - -```bash -python3 populate_db.py config.json documentation.db \ - --blacklisted-element-titles \ - "\/" -``` - -Any other page's in-content link to a pruned topic renders as a styled -"broken" link (via `broken-ext-link-color`) rather than a dead link with no -indication anything changed. Run with `--blacklisted-element-titles` first -against a scratch copy of the database and check the warnings on stderr for -any path that didn't match — that usually means the toc-title or ancestor -chain was copied wrong. - -## Workflow: optimizing and inserting media - -Two options, depending on whether the database already has pages loaded: - -**Standalone optimization only** (no database involved): - -```bash -python3 optimize_media.py [--max-width 500] [--webp] [...] -``` - -**Optimize and update an existing database's images in place:** - -```bash -python3 insert_optimized_media.py [work-dir] [options] -``` - -This re-runs `optimize_media.py`'s pipeline, backs up the database first, -replaces each `k/html/images/` row with the optimized bytes, rewrites -any page/nav reference to a file that got renamed during optimization (e.g. -`--webp` conversion or SVG rasterization), and deletes any image no page -references anymore. Both scripts share the same tuning flags -(`--max-width`, `--jpeg-quality`, `--webp`, `--webp-quality`, -`--pngquant-speed`, `--svg-precision`, `--svg-rasterize-threshold`, -`--verbose`, `--log-file`), settable via `--config ` instead of the -command line — see either script's module docstring for the full option -reference. - -## Recommended order for a full refresh - -1. `find_missing_assets.py` against the new `` — fix anything broken in the source before converting it. -2. `populate_db.py`, with `--blacklisted-element-titles` for anything you don't want documented (e.g. Kotlin/Wasm per ADFA-4737). -3. `insert_optimized_media.py` against the raw media directory, if you want optimized (resized/compressed) images rather than Writerside's own export as-is. diff --git a/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/config.json b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/config.json new file mode 100644 index 00000000..b69baed6 --- /dev/null +++ b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/config.json @@ -0,0 +1,4 @@ +{ + "broken-ext-link-color": "#cc0000", + "menu-no-link-color": "#999999" +} diff --git a/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/md_to_json.py b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/md_to_json.py new file mode 100644 index 00000000..3bb51193 --- /dev/null +++ b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/md_to_json.py @@ -0,0 +1,1117 @@ +#!/usr/bin/env python3 +""" +Converts JetBrains Writerside-flavored Markdown (as used by kotlin-web-site/docs) +into a simple JSON block schema suitable for a templating engine. + +Usage: + python3 md_to_json.py [--topics-subdir topics] + [--images-subdir images] [--allow-failures] + + is the checkout of kotlin-web-site/docs (contains v.list, topics/, ...). +One JSON file is written per input .md file, mirroring its relative path under +. + + is a JSON file with: + {"broken-ext-link-color": "#cc0000", "menu-no-link-color": "#999999"} +"broken-ext-link-color" colors tags in the rendered content that are +either off-site (any http(s)/mailto: link) or a same-tree ".md" reference +that doesn't resolve to a real page. "menu-no-link-color" isn't used here - +it's carried through to /theme.json for build_nav.py (which +builds the sidebar from kr.tree, a separate input this script doesn't read) +to pick up. + +Output schema (one object per page): +{ + "id": "enum-classes", + "sourceFile": "topics/enum-classes.md", + "title": "Enum classes", + "blocks": [ , ... ] +} + +Block shapes: + {"type": "heading", "level": 2, "id": "anonymous-classes", "html": "..."} + - "attrs" is present only if the heading's source line had a trailing + `{...}` attribute group (e.g. `## Title {id="custom-anchor"}`); an + explicit "id" in it overrides the auto-generated slug. + {"type": "paragraph", "html": "..."} + {"type": "code", "lang": "kotlin", "code": "...", "attrs": {"kotlin-runnable": "true"}} + {"type": "blockquote", "attrs": {"style": "note"}, "blocks": [...]} + {"type": "list", "ordered": false, "items": [{"blocks": [...]}]} + {"type": "table", "headers": ["a", "b"], "rows": [["1", "2"]]} + {"type": "hr"} + {"type": "tabs", "attrs": {"group": "build-system"}, + "tabs": [{"title": "Gradle", "attrs": {"group-key": "gradle"}, "blocks": [...]}]} + {"type": "note"|"tip"|"warning", "attrs": {...}, "blocks": [...]} + - Writerside's block-level admonition tags. Only the block form (the tag + alone on its own line) becomes one of these; the single-line + "text" form stays a raw "html" block - see the known + limitations below. Note that a `> quoted {style="note"}` blockquote is + a *different* shape: that stays "blockquote" with attrs.style. + {"type": "html", "html": ""} + +A "tab" normally never appears as a block type of its own: group_containers +nests it inside its parent "tabs" block's "tabs" list (above). The exception +is a written outside any - malformed source, which comes through +as a bare {"type": "tab", "attrs": ..., "blocks": [...]} block that a template +written only against the shapes above will not render. + +There is no standalone "image" block type - CommonMark only ever produces +"image" as an inline token nested inside a paragraph/heading/etc., so a +`![alt](foo.png)` in the source always ends up as an inside that +block's own "html" string (via render_inline), never as a top-level block +of its own. + +Cross-page links (`[text](other-page.md#anchor)`) and images (`![alt](foo.png)`) +use Writerside's bare-filename convention - the referenced file is looked up +by name anywhere under / (links) or images/ (images), the same +way kr.tree's topic="..." references are resolved. Rendered HTML rewrites +these to root-relative URLs that only resolve once this JSON has been passed +through templates/page.peb and rendered by RenderDocs: links become +"/.html#anchor", and images become "/images/". +The images/ directory itself is copied to /images/ so the +resolved paths have something to point at. + +Known limitations (fine for a first pass, worth revisiting before production use): + - / grouping and attribute-line merging (the "{...}" line after a + fence/blockquote) only happen at the top level of a page and inside list + items/blockquotes/table cells one level deep; deeply nested tabs-in-tabs + are not handled. + - // admonitions are recognized as block-level tags. The + inline, single-line form seen inside HTML tables (e.g. roadmap.md) is passed + through as raw "html" blocks instead of being unpacked, since those pages + are basically hand-written HTML tables rather than prose. + - elements are passed through as raw "html" blocks; resolving them + to the referenced snippet is not implemented. + - %variables% (defined in v.list) are substituted textually in rendered HTML + and code, using simple %name% -> value replacement. + - A handful of images/ filenames collide across subdirectories (leftover + duplicates in the source tree); resolution keeps the first match in sorted + order and prints a warning rather than guessing which one is "correct". + - A heading's id is slugify()'d from its text unless the source overrides + it with a trailing "{id=...}". A hand-written #anchor that was authored + against Writerside's own anchor algorithm rather than either of those + could still resolve to the right page but land on no anchor, if the two + algorithms diverge on a case not yet seen in the corpus (inline code or + punctuation in the heading, duplicate heading text) - not currently + detected. + - "html" blocks are raw CommonMark passthrough and are NOT guaranteed to + be individually well-formed - e.g. a hand-written whose rows + span multiple blank-line-separated chunks comes through as several + "html" blocks, each containing a structurally unbalanced fragment + (measured on the live corpus: 194 of 624 "html" blocks). Concatenating + consecutive "html" blocks verbatim reproduces the original markup + correctly; a template that wraps each block in its own element + (instead of concatenating verbatim) will shred tables like this one. +""" +import argparse +import json +import os +import re +import shutil +import sys +import xml.etree.ElementTree as ET +from pathlib import Path + +from markdown_it import MarkdownIt +from markdown_it.token import Token + +TITLE_RE = re.compile(r"^\[//\]:\s*#\s*\(title:\s*(.*?)\)\s*$", re.MULTILINE) +# A ``` or ~~~ fence opener/closer, matched against an already-lstrip()ped +# line. Used by fenced_spans to keep extract_title out of code samples. +FENCE_LINE_RE = re.compile(r"^(`{3,}|~{3,})") +ATTR_LINE_RE = re.compile(r"^\{(.*)\}$") +TRAILING_ATTR_GROUPS_RE = re.compile(r"\s*((?:\{[^{}]*\})+)\s*$") + +# No leading "^": matched via .match(content, pos) below, which already +# anchors at pos - unlike search(), match() never scans forward - but "^" +# itself always means the absolute start of the *string*, not of pos, so +# keeping it here would silently stop the pos-advancing loop after the +# first brace group on every second-and-later iteration. +IMAGE_ATTR_GROUP_RE = re.compile(r"\{([^{}]*)\}") +ATTR_PAIR_RE = re.compile(r'([\w-]+)\s*=\s*(?:"([^"]*)"|(\S+))') +VAR_RE = re.compile(r"%([\w.-]+)%") +# Leading "(?:[\w.-]+/)*" absorbs (and discards) any path prefix instead of +# just failing to match - a link like "tour/hello.md" is still resolved by +# its bare filename per Writerside's convention (see module docstring), the +# same way build_topic_index looks pages up; without it, a path-containing +# ".md" link neither resolved nor got classified as "broken" (classify_href +# uses this same pattern) - it shipped verbatim with no warning at all. +MD_LINK_RE = re.compile(r"^(?:[\w.-]+/)*([\w.-]+)\.md(#.*)?$") +EXTERNAL_HREF_RE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+.-]*:)?//|^mailto:", re.I) +LINK_TAG_RE = re.compile(r']*\bhref="([^"]*)"[^>]*>') +STYLE_ATTR_RE = re.compile(r'\bstyle="([^"]*)"') +IMG_TAG_RE = re.compile(r"]*>", re.I) +IMG_SRC_RE = re.compile(r'src="([^"]*)"') +# Only the hex lengths CSS actually defines (#rgb, #rgba, #rrggbb, #rrggbbaa) +# plus a bare color keyword. A flat {3,8} also admitted #12345 and #1234567, +# which no browser accepts - the validator is what keeps this value safe to +# interpolate into a style="" attribute (see load_config), so it should mean +# what it says rather than waving through lengths that only look plausible. +COLOR_RE = re.compile(r"^#(?:[0-9a-fA-F]{3,4}|[0-9a-fA-F]{6}|[0-9a-fA-F]{8})$|^[a-zA-Z]+$") + +# Fallback base for a heading whose text slugifies to nothing at all (e.g. +# "## ..."), so it still gets a linkable anchor instead of id="". +EMPTY_HEADING_ID_BASE = "section" + +# TAG_RE is built from this set (rather than hardcoding the same five names +# twice) so the two can't silently diverge. The (?![\w-]) after the +# alternation is load-bearing: without it, "tab" matches as a prefix of +# "table", consuming "
" as tag "tab" with attrs "le". +# +# Group 3 (attrs) is non-greedy and group 4 (self-closing "/") is anchored +# right before the final ">" - with a single greedy "([^>]*)/?>$" instead, +# the attrs group swallows a self-closing tag's trailing "/" before the +# optional "/?" ever gets a chance to match it, so "" came out +# indistinguishable from "": an opener with no matching closer, +# silently nesting the rest of the page inside it. +CONTAINER_TAGS = {"tabs", "tab", "note", "tip", "warning"} +TAG_RE = re.compile(r"^<(/?)(" + "|".join(CONTAINER_TAGS) + r")(?![\w-])([^>]*?)\s*(/?)>$", re.I) + + +def build_topic_index(topics_dir: Path) -> dict: + """Bare filename stem (e.g. "enum-classes") -> page id (e.g. "kotlin-tour/enum-classes"). + + Mirrors build_image_index's collision handling: a stem that exists more + than once under topics_dir keeps its first (sorted) page id and prints a + warning naming the others, rather than resolving to whichever sorts + first with no diagnostic at all.""" + index = {} + candidates = {} + for md_path in sorted(topics_dir.rglob("*.md")): + page_id = md_path.relative_to(topics_dir).with_suffix("").as_posix() + candidates.setdefault(md_path.stem, []).append(page_id) + index.setdefault(md_path.stem, page_id) + for stem, ids in sorted(candidates.items()): + if len(ids) > 1: + print(f"warning: ambiguous topic filename {stem!r}: " + f"using {ids[0]}.md, ignoring {', '.join(i + '.md' for i in ids[1:])}", file=sys.stderr) + return index + + +def build_image_index(images_dir: Path): + """Bare filename (e.g. "mascot-main.png") -> path relative to images_dir. + + Returns (index, collisions), where collisions is a list of + (filename, [candidate relative paths]) for filenames that exist in more + than one place under images_dir - index keeps the first (sorted) one.""" + index = {} + candidates = {} + if not images_dir.is_dir(): + return index, [] + for img_path in sorted(images_dir.rglob("*")): + if not img_path.is_file(): + continue + rel = img_path.relative_to(images_dir).as_posix() + candidates.setdefault(img_path.name, []).append(rel) + index.setdefault(img_path.name, rel) + collisions = [(name, rels) for name, rels in sorted(candidates.items()) if len(rels) > 1] + for name, rels in collisions: + print(f"warning: ambiguous image filename {name!r}: " + f"using images/{rels[0]}, ignoring {', '.join('images/' + r for r in rels[1:])}", file=sys.stderr) + return index, collisions + + +def load_variables(docs_root: Path) -> dict: + v_list = docs_root / "v.list" + if not v_list.exists(): + return {} + tree = ET.parse(v_list) + return {el.get("name"): el.get("value") for el in tree.getroot().findall("var")} + + +def substitute_vars(text: str, variables: dict) -> str: + if not text: + return text + return VAR_RE.sub(lambda m: variables.get(m.group(1), m.group(0)), text) + + +def parse_attrs(attr_str: str) -> dict: + """name="quoted value" or name=bare -> {"name": "quoted value"/"bare"}, + decided per pair. findall() coerces a non-participating group to "" (not + None), and exactly one of quoted/bare participates per match, so the + other is always "" - `quoted or bare` picks whichever one actually + matched, including a genuinely empty quoted value ("" or "" -> "").""" + attrs = {} + for name, quoted, bare in ATTR_PAIR_RE.findall(attr_str or ""): + attrs[name] = quoted or bare + return attrs + + +def fenced_spans(raw_text: str) -> list: + """(start, end) character ranges of every ``` / ~~~ fenced code block, + including the fence lines themselves. An unterminated fence runs to the + end of the document, which is what CommonMark does too.""" + spans = [] + pos = 0 + open_at = None + open_fence = None + for line in raw_text.splitlines(keepends=True): + stripped = line.lstrip() + m = FENCE_LINE_RE.match(stripped) + if m: + fence = m.group(1) + if open_at is None: + open_at, open_fence = pos, fence + # A closing fence must be the same character and at least as long + # as the opener - "````" inside a ``` block is content, not a + # close, and a shorter run never closes a longer one. + elif fence[0] == open_fence[0] and len(fence) >= len(open_fence): + spans.append((open_at, pos + len(line))) + open_at = open_fence = None + pos += len(line) + if open_at is not None: + spans.append((open_at, len(raw_text))) + return spans + + +def extract_title(raw_text: str): + """Pulls Writerside's `[//]: # (title: ...)` comment out of the source. + + Skips any match inside a fenced code block: a page documenting Writerside + syntax shows that comment as a code sample, and taking it would both set a + bogus title and delete the line out of the sample being shown.""" + spans = None + for m in TITLE_RE.finditer(raw_text): + if spans is None: # only pay for this if a match exists + spans = fenced_spans(raw_text) + if any(start <= m.start() < end for start, end in spans): + continue + title = m.group(1).strip() + return title, raw_text[: m.start()] + raw_text[m.end():] + return None, raw_text + + +def slugify(text: str) -> str: + """Deletes-vs-hyphenates matters: dropping punctuation entirely (the + previous behavior) merges "package.json" into "packagejson", diverging + from Writerside's own anchor algorithm (which hyphenates it) on every + heading with a "." "/" ":" etc. - measured at 209 dead #anchor links + across the real kotlin-web-site corpus. Hyphenating punctuation instead + keeps this aligned with those hand-written anchors in the common case.""" + slug = re.sub(r"[^\w\s-]", "-", text.lower()) + return re.sub(r"[\s_-]+", "-", slug).strip("-") + + +class Node: + """Generic open/close tree built from markdown-it's flat token stream.""" + + __slots__ = ("token", "children") + + def __init__(self, token: Token): + self.token = token + self.children = [] + + +def build_tree(tokens) -> list: + root = [] + stack = [root] + for tok in tokens: + if tok.nesting == 1: + node = Node(tok) + stack[-1].append(node) + stack.append(node.children) + elif tok.nesting == -1: + # markdown-it always emits balanced nesting, so this guard is + # defensive rather than a case seen in the corpus - but popping + # the root off an already-empty stack turns a malformed token + # stream (a plugin, or a future markdown-it change) into an + # "IndexError: pop from empty list" with nothing pointing at the + # cause. Ignoring the stray closer keeps the tree usable and + # names the problem instead. + if len(stack) > 1: + stack.pop() + else: + print(f"warning: unbalanced token stream: stray closing {tok.type!r} at top level", + file=sys.stderr) + else: + stack[-1].append(Node(tok)) + return root + + +class Converter: + def __init__(self, md: MarkdownIt, variables: dict, topic_index: dict = None, image_index: dict = None, + broken_ext_link_color: str = None, image_url_prefix: str = "/images/"): + self.md = md + self.variables = variables + self.topic_index = topic_index or {} + self.image_index = image_index or {} + self.broken_ext_link_color = broken_ext_link_color + # Overridable so a different deployment target (e.g. populate_db.py's + # database-backed site, which serves images from "/k/html/images/" + # rather than a bare "/images/") can retarget every image src without + # a separate rewrite pass - resolve_image_src just uses this prefix + # directly. + self.image_url_prefix = image_url_prefix + self.current_source = None + # Populated as a side effect of resolve_href/resolve_image_src failing + # to resolve a reference; find_missing_assets.py reuses this same + # resolution logic (rather than re-parsing links with regexes) by + # running convert_file over every page and reading this list back. + self.warnings = [] + # Reset per file in convert_file - two headings with the same text + # on the same page would otherwise share a slug, so an anchor to the + # second one lands on the first. + self.seen_heading_ids = set() + + def unique_heading_id(self, text: str) -> str: + """slugify(text), de-duped against every other heading id already + seen on this page - resolve_href points at these ids verbatim (via + the source's own #anchor), so two headings sharing a slug would + make any link to the second one land on the first instead.""" + base = slug = slugify(text) + if not base: + # A heading whose text is entirely punctuation ("## ...") slugifies + # to "", and id="" is both invalid HTML and impossible to link to. + # Fall back to a positional name so it still gets a usable anchor; + # the de-dup loop below numbers subsequent ones. + base = slug = EMPTY_HEADING_ID_BASE + n = 2 + while slug in self.seen_heading_ids: + slug = f"{base}-{n}" + n += 1 + self.seen_heading_ids.add(slug) + return slug + + def reserve_explicit_heading_ids(self, tokens) -> None: + """Registers every explicit `{id="..."}` on the page *before* any + heading is converted, so an auto-slugified heading can never take an + id that an explicit one further down the page also claims. + + Registering them lazily as each heading was reached only protected + explicit ids from *later* auto-slugs. In the other order - + `## Custom anchor` followed by `## Something {id="custom-anchor"}` - + the auto-slug got there first and both headings ended up with + id="custom-anchor", so any link to the explicit one landed on the + auto one instead. Explicit ids are the intentional, presumably-stable + anchors, so they win in both directions and the auto-slug moves. + + Read-only: this inspects `inline.content` rather than going through + extract_trailing_attrs, which mutates the token children it parses. + """ + previous = None + for tok in tokens: + # Only an inline directly inside a heading - markdown-it emits + # heading_open / inline / heading_close. A paragraph that merely + # ends in "{id=...}" is not a heading anchor and must not reserve + # one (nothing reads a trailing attr group off a paragraph). + is_heading_inline = tok.type == "inline" and previous == "heading_open" + previous = tok.type + if not is_heading_inline or not tok.content: + continue + m = TRAILING_ATTR_GROUPS_RE.search(tok.content) + if not m: + continue + attrs = {} + for group in re.findall(r"\{([^{}]*)\}", m.group(1)): + attrs.update(parse_attrs(group)) + explicit = attrs.get("id") + if explicit: + if explicit in self.seen_heading_ids: + print(f"warning: duplicate explicit heading id {explicit!r} in {self.current_source!r}", + file=sys.stderr) + self.warnings.append( + {"kind": "heading-id", "source": self.current_source, "reference": explicit} + ) + self.seen_heading_ids.add(explicit) + + def resolve_href(self, href: str): + """"other-page.md#anchor" -> "/.html#anchor", or None to leave href untouched.""" + m = MD_LINK_RE.match(href or "") + if not m: + return None + stem, anchor = m.groups() + page_id = self.topic_index.get(stem) + if page_id is None: + print(f"warning: link to unknown topic {href!r}", file=sys.stderr) + self.warnings.append({"kind": "link", "source": self.current_source, "reference": href}) + return None + return f"/{page_id}.html{anchor or ''}" + + def resolve_image_src(self, src: str): + """"foo.png", optionally with a relative path prefix (e.g. + "sub/foo.png" - resolved by bare filename anywhere under images/, + same Writerside convention as resolve_href for topics; a path + prefix used to bail out here with no warning, unlike the + bare-filename miss below) -> "", + or None to leave src untouched (already-absolute "/..." paths and + "scheme://..." URLs are left alone - "/" in src is redundant with + startswith("/") once a relative path prefix is allowed through, so + it's dropped rather than kept as dead code).""" + if not src or "://" in src or src.startswith("/"): + return None + name = src.rsplit("/", 1)[-1] + rel = self.image_index.get(name) + if rel is None: + print(f"warning: image not found: {src!r}", file=sys.stderr) + self.warnings.append({"kind": "image", "source": self.current_source, "reference": src}) + return None + return f"{self.image_url_prefix}{rel}" + + def rewrite_urls(self, html: str) -> str: + """Rewrites every href="...md" / src="foo.png" attribute found in a + blob of rendered/raw HTML. Applied to markdown-rendered HTML *and* to + Writerside's raw / passthrough HTML (which markdown-it never + tokenizes as links/images at all, so token-level rewriting alone + would miss it); resolve_href/resolve_image_src already leave anything + that isn't a bare same-tree ".md"/image reference untouched, so this + is safe to run unconditionally on any HTML string.""" + if not html: + return html + + def href_repl(m): + new_href = self.resolve_href(m.group(1)) + return f'href="{new_href}"' if new_href is not None else m.group(0) + + def src_repl(m): + new_src = self.resolve_image_src(m.group(1)) + return f'src="{new_src}"' if new_src is not None else m.group(0) + + html = re.sub(r'href="([^"]*)"', href_repl, html) + # Scoped to tags specifically - a bare src="..." regex + # would also rewrite ') + assert 'src="x.js">' in html + assert 'src="x.js">' in html + assert '/images/sub/x.js' in html + assert not any(w["kind"] == "image" for w in conv.warnings) + + +# --- resolve_href / resolve_image_src: bare-filename path prefixes ------ + +def test_resolve_href_resolves_a_link_with_a_path_prefix(): + """MD_LINK_RE's char class excluded "/", so a link like + "tour/hello.md" (a real, if unusual, way to reference a topic) neither + resolved nor got flagged "broken" by classify_href (same regex) - it + shipped verbatim with no warning at all, unlike the unknown-bare-stem + case just below it.""" + conv = make_converter(topic_index={"hello": "kotlin-tour/hello"}) + assert conv.resolve_href("tour/hello.md#section") == "/kotlin-tour/hello.html#section" + assert conv.warnings == [] + + +def test_resolve_href_path_prefixed_unknown_stem_still_warns(): + conv = make_converter(topic_index={}) + assert conv.resolve_href("tour/missing.md") is None + assert conv.warnings == [{"kind": "link", "source": None, "reference": "tour/missing.md"}] + + +def test_resolve_image_src_resolves_a_bare_filename_with_a_path_prefix(): + """A relative path prefix (as opposed to a bare filename) used to bail + out silently here with no warning either way - unlike the bare-filename + miss just below, which does warn. Writerside's own convention (see + module docstring) is to resolve images by bare filename anywhere under + images/, same as topics.""" + conv = make_converter(image_index={"pic.png": "sub/pic.png"}) + assert conv.resolve_image_src("sub/pic.png") == "/images/sub/pic.png" + assert conv.warnings == [] + + +def test_resolve_image_src_path_prefixed_unknown_name_warns(): + conv = make_converter(image_index={}) + assert conv.resolve_image_src("sub/missing.png") is None + assert conv.warnings == [{"kind": "image", "source": None, "reference": "sub/missing.png"}] + + +def test_resolve_image_src_still_leaves_absolute_and_external_untouched(): + conv = make_converter(image_index={"pic.png": "pic.png"}) + assert conv.resolve_image_src("/already/absolute.png") is None + assert conv.resolve_image_src("https://example.com/x.png") is None + assert conv.warnings == [] + + +# --- load_config: reject an unsafe/invalid color ------------------------ + +def test_load_config_rejects_markup_injection_payload(tmp_path): + """broken-ext-link-color is interpolated directly into a style="..." + HTML attribute; an unvalidated value is a markup-injection hole.""" + config_path = tmp_path / "config.json" + config_path.write_text(json.dumps({ + "broken-ext-link-color": 'red">', + "menu-no-link-color": "#999999", + })) + with pytest.raises(SystemExit) as exc_info: + m.load_config(config_path) + assert exc_info.value.code == 1 + + +def test_load_config_rejects_non_string_color_value(tmp_path): + """A JSON number (an unquoted hex-like value is a plausible hand-edit + slip, e.g. writing cc0000 instead of "#cc0000") used to raise a bare + TypeError from COLOR_RE.match(int) instead of the intended clean + "invalid ... value" error the line below is meant to produce.""" + config_path = tmp_path / "config.json" + config_path.write_text(json.dumps({ + "broken-ext-link-color": 123, + "menu-no-link-color": "#999999", + })) + with pytest.raises(SystemExit) as exc_info: + m.load_config(config_path) + assert exc_info.value.code == 1 + + +def test_load_config_accepts_hex_and_named_colors(tmp_path): + config_path = tmp_path / "config.json" + config_path.write_text(json.dumps({ + "broken-ext-link-color": "#cc0000", + "menu-no-link-color": "gray", + })) + config = m.load_config(config_path) + assert config["broken-ext-link-color"] == "#cc0000" + assert config["menu-no-link-color"] == "gray" + + +# --- heading ids: de-dup + explicit {id=...} override ------------------- + +def test_duplicate_heading_text_gets_deduped_ids(): + """Two headings with identical text used to share a slug, so an anchor + to the second one landed on the first.""" + conv = make_converter() + first = conv.unique_heading_id("Overview") + second = conv.unique_heading_id("Overview") + assert first == "overview" + assert second == "overview-2" + + +def test_heading_trailing_id_attr_overrides_slug_and_is_stripped_from_html(): + """## Checks with `is`/`!is` operators {id="is-and-is-operators"} used + to render the literal "{id="is-and-is-operators"}" as visible + heading text, with no id override and no attribute handling at all.""" + md = m.make_markdown_it() + conv = m.Converter(md, {}) + tokens = md.parse('## Checks with `is` {id="is-and-is-operators"}\n') + tree = m.build_tree(tokens) + block = conv.convert_node(tree[0]) + assert block["id"] == "is-and-is-operators" + assert "{id=" not in block["html"] + assert block["attrs"] == {"id": "is-and-is-operators"} + + +def test_heading_without_trailing_attrs_unaffected(): + md = m.make_markdown_it() + conv = m.Converter(md, {}) + tokens = md.parse("## Plain heading\n") + tree = m.build_tree(tokens) + block = conv.convert_node(tree[0]) + assert block["id"] == "plain-heading" + assert "attrs" not in block + + +# --- build_topic_index: collision warning mirrors build_image_index ----- + +def test_build_topic_index_warns_on_duplicate_stem(tmp_path, capsys): + """Two topics sharing a filename stem used to resolve first-wins with + no diagnostic at all, unlike the equivalent image-filename collision.""" + topics = tmp_path / "topics" + (topics / "native").mkdir(parents=True) + (topics / "js").mkdir(parents=True) + (topics / "native" / "basics.md").write_text("native") + (topics / "js" / "basics.md").write_text("js") + + index = m.build_topic_index(topics) + assert index["basics"] in ("native/basics", "js/basics") + assert "warning: ambiguous topic filename 'basics'" in capsys.readouterr().err + + +def test_build_topic_index_no_warning_without_collision(tmp_path, capsys): + topics = tmp_path / "topics" + topics.mkdir() + (topics / "a.md").write_text("a") + m.build_topic_index(topics) + assert capsys.readouterr().err == "" + + +# --- main(): exit code reflects partial failure ------------------------- + +def _write_minimal_docs_root(tmp_path): + docs_root = tmp_path / "docs" + (docs_root / "topics").mkdir(parents=True) + (docs_root / "topics" / "good.md").write_text("# Good\n\nHello.\n") + config = tmp_path / "config.json" + config.write_text(json.dumps({"broken-ext-link-color": "#cc0000", "menu-no-link-color": "#999999"})) + return docs_root, config + + +def _run_main(*args): + script = Path(__file__).resolve().parent.parent / "md_to_json.py" + return subprocess.run([sys.executable, str(script), *map(str, args)], capture_output=True, text=True) + + +def test_main_exits_zero_on_full_success(tmp_path): + docs_root, config = _write_minimal_docs_root(tmp_path) + out_dir = tmp_path / "out" + result = _run_main(docs_root, out_dir, config) + assert result.returncode == 0 + assert "Converted 1/1" in result.stdout + + +def test_main_exits_nonzero_when_a_file_fails(tmp_path, monkeypatch): + """A run that converts nothing (or partially fails) used to print + "Converted 0/N files" and still exit 0 - a CI step calling this + couldn't distinguish a complete run from a total failure.""" + docs_root, config = _write_minimal_docs_root(tmp_path) + # A .md file that isn't valid UTF-8 makes convert_file's read_text raise. + (docs_root / "topics" / "bad.md").write_bytes(b"\xff\xfe not utf-8") + out_dir = tmp_path / "out" + result = _run_main(docs_root, out_dir, config) + assert result.returncode == 1 + assert "Converted 1/2" in result.stdout + + +def test_main_allow_failures_exits_zero_despite_failure(tmp_path): + docs_root, config = _write_minimal_docs_root(tmp_path) + (docs_root / "topics" / "bad.md").write_bytes(b"\xff\xfe not utf-8") + out_dir = tmp_path / "out" + result = _run_main(docs_root, out_dir, config, "--allow-failures") + assert result.returncode == 0 + + +def test_main_uses_posix_separators_for_nested_page_ids(tmp_path): + """page_id/sourceFile used str(Path(...)) instead of .as_posix(), which + would disagree with build_topic_index's forward-slashed ids on Windows.""" + docs_root, config = _write_minimal_docs_root(tmp_path) + (docs_root / "topics" / "tour").mkdir() + (docs_root / "topics" / "tour" / "hello.md").write_text("# Hello\n") + out_dir = tmp_path / "out" + result = _run_main(docs_root, out_dir, config) + assert result.returncode == 0 + page = json.loads((out_dir / "topics" / "tour" / "hello.json").read_text()) + assert page["id"] == "tour/hello" + assert page["sourceFile"] == "topics/tour/hello.md" + + +def test_main_prunes_stale_topic_json(tmp_path): + """A topic removed upstream used to keep shipping its stale JSON + forever, since nothing ever cleared the output topics directory.""" + docs_root, config = _write_minimal_docs_root(tmp_path) + out_dir = tmp_path / "out" + assert _run_main(docs_root, out_dir, config).returncode == 0 + assert (out_dir / "topics" / "good.json").exists() + + (docs_root / "topics" / "good.md").unlink() + (docs_root / "topics" / "new.md").write_text("# New\n") + assert _run_main(docs_root, out_dir, config).returncode == 0 + assert not (out_dir / "topics" / "good.json").exists() + assert (out_dir / "topics" / "new.json").exists() + + +def test_main_prunes_stale_images(tmp_path): + """copytree(dirs_exist_ok=True) only ever adds/overwrites - an image + deleted upstream used to keep shipping in the output images/ directory + forever, unlike a removed topic's stale JSON (pruned by the rmtree + exercised just above).""" + docs_root, config = _write_minimal_docs_root(tmp_path) + (docs_root / "images").mkdir() + (docs_root / "images" / "keep.png").write_bytes(b"keep") + (docs_root / "images" / "stale.png").write_bytes(b"stale") + out_dir = tmp_path / "out" + assert _run_main(docs_root, out_dir, config).returncode == 0 + assert (out_dir / "images" / "keep.png").exists() + assert (out_dir / "images" / "stale.png").exists() + + (docs_root / "images" / "stale.png").unlink() + assert _run_main(docs_root, out_dir, config).returncode == 0 + assert (out_dir / "images" / "keep.png").exists() + assert not (out_dir / "images" / "stale.png").exists() + + +def test_main_prunes_now_empty_stale_image_subdirectories(tmp_path): + docs_root, config = _write_minimal_docs_root(tmp_path) + (docs_root / "images" / "sub").mkdir(parents=True) + (docs_root / "images" / "sub" / "only.png").write_bytes(b"only") + out_dir = tmp_path / "out" + assert _run_main(docs_root, out_dir, config).returncode == 0 + assert (out_dir / "images" / "sub" / "only.png").exists() + + (docs_root / "images" / "sub" / "only.png").unlink() + (docs_root / "images" / "sub").rmdir() + assert _run_main(docs_root, out_dir, config).returncode == 0 + assert not (out_dir / "images" / "sub").exists() + + +def test_main_refuses_when_output_dir_is_docs_root(tmp_path): + """output_dir == docs_root makes topics_out_dir the very same directory + as the source topics/ - the pruning rmtree used to delete it outright, + with every already-globbed source file then failing to convert.""" + docs_root, config = _write_minimal_docs_root(tmp_path) + result = _run_main(docs_root, docs_root, config) + assert result.returncode == 1 + assert (docs_root / "topics" / "good.md").exists() + + +def test_main_refuses_before_writing_theme_json_into_the_source(tmp_path): + """The output_dir-aliases-docs_root guard used to run only right + before the topics_out_dir rmtree, well after theme.json (and the + images/ copytree) had already written into the very directory the + refusal exists to protect.""" + docs_root, config = _write_minimal_docs_root(tmp_path) + result = _run_main(docs_root, docs_root, config) + assert result.returncode == 1 + assert not (docs_root / "theme.json").exists() + + +# --- heading ids: an explicit {id=...} must win in BOTH directions ------ + +def _heading_ids(src, conv=None): + """Runs the same reserve-then-convert sequence convert_file does.""" + md = m.make_markdown_it() + conv = conv or m.Converter(md, {}) + conv.current_source = "t.md" + tokens = md.parse(src) + conv.reserve_explicit_heading_ids(tokens) + blocks = conv.convert_nodes(m.build_tree(tokens)) + return [b["id"] for b in blocks if b["type"] == "heading"] + + +def test_explicit_heading_id_after_a_colliding_auto_slug_does_not_duplicate(): + """Explicit ids were registered lazily as each heading was reached, so + they were only protected from a *later* auto-slug. In this order the + auto-slug got there first and both headings came out with + id="custom-anchor" - any link to the explicit one landed on the auto + one instead.""" + ids = _heading_ids('## Custom anchor\n\n## Something else {id="custom-anchor"}\n') + assert ids == ["custom-anchor-2", "custom-anchor"] + assert len(set(ids)) == len(ids) + + +def test_explicit_heading_id_before_an_auto_slug_still_wins(): + ids = _heading_ids('## Something else {id="custom-anchor"}\n\n## Custom anchor\n') + assert ids == ["custom-anchor", "custom-anchor-2"] + + +def test_two_explicit_heading_ids_that_collide_are_reported(): + """Two hand-written ids that are genuinely the same is a source bug this + can't silently fix - both are kept as authored, and it warns.""" + conv = m.Converter(m.make_markdown_it(), {}) + ids = _heading_ids('## A {id="dup"}\n\n## B {id="dup"}\n', conv) + assert ids == ["dup", "dup"] + assert conv.warnings == [{"kind": "heading-id", "source": "t.md", "reference": "dup"}] + + +def test_paragraph_trailing_attr_group_does_not_reserve_a_heading_id(): + """Only an inline directly inside a heading reserves an id - a paragraph + that merely ends in "{id=...}" is not an anchor, and reserving from it + would push an unrelated heading's slug to "-2" for no reason.""" + assert _heading_ids('Text {id="para-thing"}\n\n## Para thing\n') == ["para-thing"] + + +def test_punctuation_only_heading_gets_a_usable_id(): + """slugify("...") is "", and id="" is invalid HTML and unlinkable.""" + assert _heading_ids("## ...\n\n## ???\n") == ["section", "section-2"] + + +# --- extract_title: must not match inside a fenced code block ----------- + +def test_extract_title_ignores_a_title_comment_inside_a_fence(): + """A page documenting Writerside syntax shows the title comment as a code + sample; matching it both set a bogus title and deleted the line out of + the sample being displayed.""" + src = "# Page\n\n```\n[//]: # (title: Not a title)\n```\n" + title, body = m.extract_title(src) + assert title is None + assert "[//]: # (title: Not a title)" in body + + +def test_extract_title_still_finds_a_real_title_alongside_a_fenced_one(): + src = '[//]: # (title: Real)\n\n```\n[//]: # (title: Nope)\n```\n' + title, body = m.extract_title(src) + assert title == "Real" + assert "[//]: # (title: Nope)" in body # the sample is left intact + assert "[//]: # (title: Real)" not in body # the real one is consumed + + +def test_extract_title_handles_tilde_and_nested_longer_fences(): + assert m.extract_title("~~~\n[//]: # (title: Nope)\n~~~\n")[0] is None + # A ``` run inside a ```` block is content, not a close. + assert m.extract_title("````\n```\n[//]: # (title: Nope)\n```\n````\n")[0] is None + + +def test_extract_title_unterminated_fence_runs_to_end_of_document(): + assert m.extract_title("```\n[//]: # (title: Nope)\n")[0] is None + + +def test_fenced_spans_reports_no_spans_for_fence_free_text(): + assert m.fenced_spans("just prose\n\nmore prose\n") == [] + + +# --- html_block: a blank-only raw run is not a block -------------------- + +def test_blank_only_raw_run_emits_no_empty_html_block(): + """The "\\n\\n" between "" and "" is a truthy list of empty + strings, so it produced an {"type": "html", "html": ""} block carrying + nothing at all.""" + node = m.Node(FakeToken("html_block", content="\n\n")) + result = make_converter().convert_node(node) + assert [b["type"] for b in result] == ["tag_marker", "tag_marker"] + + +def test_blank_lines_inside_a_real_raw_run_are_still_preserved(): + node = m.Node(FakeToken("html_block", content="
\na\n\nb\n
")) + assert make_converter().convert_node(node) == [{"type": "html", "html": "
\na\n\nb\n
"}] + + +# --- COLOR_RE: only the hex lengths CSS actually defines ---------------- + +@pytest.mark.parametrize("value", ["#cc0000", "#fff", "#ffff", "#ffffffff", "red", "rebeccapurple"]) +def test_color_re_accepts_valid_css_colors(value): + assert m.COLOR_RE.match(value) + + +@pytest.mark.parametrize("value", ["#12345", "#1234567", "#gg0000", "rgb(1,2,3)", "red;x", ""]) +def test_color_re_rejects_invalid_colors(value): + """A flat {3,8} waved through #12345 and #1234567, which no browser + accepts - this value is interpolated into a style="" attribute, so the + validator should mean what it says.""" + assert not m.COLOR_RE.match(value) + + +# --- build_tree: a stray closer must not pop the root ------------------- + +def test_build_tree_tolerates_a_stray_closing_token(capsys): + """Popping an already-empty stack turned a malformed token stream into + "IndexError: pop from empty list" with nothing naming the cause.""" + class Tok: + def __init__(self, nesting): + self.nesting, self.type, self.content, self.children = nesting, "stray", "", None + + tree = m.build_tree([Tok(-1), Tok(0)]) + assert len(tree) == 1 + assert "unbalanced token stream" in capsys.readouterr().err + + +# --- coverage gaps flagged in review: features with no test at all ------ + +def test_variable_substitution_replaces_known_names_and_leaves_others(tmp_path): + """%variables% substitution is a documented feature that reaches every + rendered string, and had no test.""" + assert m.substitute_vars("Kotlin %v% and %unknown%", {"v": "2.0"}) == "Kotlin 2.0 and %unknown%" + assert m.substitute_vars("", {"v": "2.0"}) == "" + + +def test_load_variables_reads_v_list(tmp_path): + (tmp_path / "v.list").write_text( + '\n\n \n\n' + ) + assert m.load_variables(tmp_path) == {"kv": "2.0.20"} + + +def test_load_variables_missing_v_list_is_empty(tmp_path): + assert m.load_variables(tmp_path) == {} + + +def test_variables_are_substituted_before_parsing(tmp_path): + """Substituting post-render would miss the title and would have to fight + markdown-it percent-encoding "%" inside link URLs.""" + src = tmp_path / "p.md" + src.write_text('[//]: # (title: Kotlin %kv%)\n\nUse %kv% today.\n') + conv = m.Converter(m.make_markdown_it(), {"kv": "2.0.20"}) + page = conv.convert_file(src, "p", "topics/p.md") + assert page["title"] == "Kotlin 2.0.20" + assert "2.0.20" in page["blocks"][0]["html"] + + +def test_block_level_note_becomes_a_note_block(): + """// block tags are a documented block type that no + test exercised end-to-end - and the live corpus only uses the + single-line form, so nothing covered this path at all.""" + src = '\n\nWatch out.\n\n\n' + md = m.make_markdown_it() + conv = m.Converter(md, {}) + conv.current_source = "t.md" + blocks = conv.convert_nodes(m.build_tree(md.parse(src))) + assert len(blocks) == 1 + assert blocks[0]["type"] == "note" + assert [b["type"] for b in blocks[0]["blocks"]] == ["paragraph"] + assert conv.warnings == [] + + +def test_single_line_note_stays_a_raw_html_block(): + """The inline form inside hand-written HTML tables (all 5 uses in the + live corpus) is documented to pass through untouched.""" + node = m.Node(FakeToken("html_block", content="inline text")) + result = make_converter().convert_node(node) + assert result == [{"type": "html", "html": "inline text"}] + + +def test_bare_tabs_of_adjacent_code_fences_synthesizes_tabs(): + """A wrapping only fenced code with no tags used to render a + tabs shell with no "tabs" key, silently dropping every code block.""" + src = '\n\n```kotlin\nval a = 1\n```\n\n```groovy\ndef a = 1\n```\n\n\n' + md = m.make_markdown_it() + conv = m.Converter(md, {}) + conv.current_source = "t.md" + blocks = conv.convert_nodes(m.build_tree(md.parse(src))) + assert len(blocks) == 1 and blocks[0]["type"] == "tabs" + assert [t["title"] for t in blocks[0]["tabs"]] == ["Kotlin", "Groovy"] + assert [t["attrs"]["group-key"] for t in blocks[0]["tabs"]] == ["kotlin", "groovy"] + assert [t["blocks"][0]["type"] for t in blocks[0]["tabs"]] == ["code", "code"] + + +def test_bare_tabs_with_nothing_tab_like_splices_content_in_place(): + """Dropping the wrapper beats emitting a "tabs" block with no "tabs" key, + which page.peb renders as an empty
with the content gone.""" + src = '\n\nJust prose.\n\n\n' + md = m.make_markdown_it() + conv = m.Converter(md, {}) + conv.current_source = "t.md" + blocks = conv.convert_nodes(m.build_tree(md.parse(src))) + assert [b["type"] for b in blocks] == ["paragraph"] + + +def test_blockquote_list_table_and_hr_shapes(): + """The blockquote/list/table/hr converters had no direct test.""" + md = m.make_markdown_it() + conv = m.Converter(md, {}) + conv.current_source = "t.md" + src = "> quoted\n\n- one\n- two\n\n1. first\n\n| a | b |\n|---|---|\n| 1 | 2 |\n\n---\n" + blocks = conv.convert_nodes(m.build_tree(md.parse(src))) + by_type = {b["type"]: b for b in blocks} + assert set(by_type) == {"blockquote", "list", "table", "hr"} + assert by_type["blockquote"]["blocks"][0]["type"] == "paragraph" + assert by_type["table"]["headers"] == ["a", "b"] + assert by_type["table"]["rows"] == [["1", "2"]] + # the source has a bullet list and an ordered list, in that order + assert [b["ordered"] for b in blocks if b["type"] == "list"] == [False, True] + + +def test_table_with_an_empty_cell_does_not_crash(): + md = m.make_markdown_it() + conv = m.Converter(md, {}) + blocks = conv.convert_nodes(m.build_tree(md.parse("| a | b |\n|---|---|\n| | y |\n"))) + assert blocks[0]["rows"] == [["", "y"]] + + +def test_no_raw_key_survives_into_output(): + """"_raw" is an internal marker merge_attr_lines consumes; it must never + reach the JSON.""" + md = m.make_markdown_it() + conv = m.Converter(md, {}) + conv.current_source = "t.md" + src = 'Para.\n\n{style="note"}\n\n> quoted\n\n- item\n\n nested\n' + assert '"_raw"' not in json.dumps(conv.convert_nodes(m.build_tree(md.parse(src)))) diff --git a/requirements.txt b/requirements.txt index 265d3c39..c5fc3aa0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,3 +5,4 @@ brotli Pillow openpyxl>=3.1.0 tqdm-loggable>=0.1.0 +markdown-it-py>=2.0