Add JavaTemplated pipeline for java.base API docs

JavaTemplated/.gitignore

@@ -0,0 +1,13 @@
+# Local checkout of https://github.com/openjdk/jdk17 - fetched fresh per the README, not committed.
+jdk17/
+
+# Gradle build output, both subprojects.
+*/build/
+*/.gradle/
+
+# Generated docs - these are build output, not source.
+dokka-java-base-docs/build/dokka-json-output/
+html-output/
+
+# Bisection tool run logs (not needed to build, only useful while actively debugging).
+*.log

JavaTemplated/README.md

@@ -0,0 +1,137 @@
+# JavaTemplated
+
+Generates HTML API documentation for OpenJDK 17's `java.base` module: Dokka (with the
+`kdoc-to-json` plugin from [`../Dokka-plugin-kdoc2json`](../Dokka-plugin-kdoc2json))
+produces JSON, then a small Pebble-based renderer turns that JSON into HTML.
+
+```
+jdk17 source  --[Dokka + kdoc-to-json]-->  JSON  --[Pebble template]-->  HTML
+```
+
+## Layout
+
+- **`dokka-java-base-docs/`** - the Dokka project. `build.gradle.kts` points Dokka's Java
+  source set at a `jdk17` checkout (see below) and applies `kdoc-to-json` so the
+  `dokkaGenerateHtml` task writes JSON instead of HTML. Also bakes in a required
+  workaround - see "The Dokka bug" below.
+  - `bisect_inheritdoc.py` - the tool that found every file in that workaround. Rerun it
+    if a JDK/Dokka/plugin version bump reintroduces the crash on a new file.
+- **`pebble-renderer/`** - a small Kotlin CLI (`RenderHtml.kt`) that walks a directory of
+  `*.json` files, evaluates each one through a Pebble template, and writes the result to
+  the same relative path with a `.html` extension - so the output tree mirrors the JSON
+  tree.
+- **`peb.peb.txt`** - the actual Pebble template consumed by `pebble-renderer`. Renders
+  Dokka's `kind`-discriminated JSON shapes (module / package / class / function /
+  property / ...) into styled HTML pages, with internal cross-references rewritten from
+  `.json` to `.html` so links between generated pages resolve correctly.
+
+Not checked in - fetched or generated locally (see `.gitignore`):
+- `jdk17/` - the OpenJDK source, cloned fresh as a sibling of `dokka-java-base-docs/`.
+- `dokka-java-base-docs/build/dokka-json-output/` - Dokka's JSON output.
+- `html-output/` (or wherever you point the renderer) - the final HTML.
+
+## Prerequisites
+
+A JDK 17 install, distinct from whatever this machine's default `java` is - the Kotlin
+Gradle plugin `kdoc-to-json` depends on doesn't support newer JDK targets, and
+`compileJava`/`compileKotlin` need to agree on one. `bisect_inheritdoc.py` finds one
+automatically (via macOS's `java_home -v 17`, overridable with `JDK17_HOME`); for the raw
+`./gradlew` invocations below, export it yourself, e.g.:
+
+```bash
+export JAVA_HOME=$(/usr/libexec/java_home -v 17)   # macOS
+# or, if you don't have one: brew install openjdk@17
+```
+
+## 1. Fetch the JDK 17 source
+
+```bash
+cd JavaTemplated
+git clone --no-checkout --depth 1 --filter=blob:none https://github.com/openjdk/jdk17.git jdk17
+cd jdk17
+git sparse-checkout init --cone
+git sparse-checkout set src/java.base
+git checkout
+```
+
+(A full checkout works too; `dokka-java-base-docs/build.gradle.kts` only reads
+`src/java.base/share/classes`. The sparse checkout above just avoids pulling ~30 other
+modules you don't need.)
+
+## 2. Build and publish the kdoc-to-json plugin
+
+```bash
+cd ../../Dokka-plugin-kdoc2json/kdoc-to-json
+./gradlew clean publishToMavenLocal
+```
+
+This publishes `org.appdevforall.dokka:kdoc-to-json:1.0.0-SNAPSHOT` to `~/.m2`, which
+`dokka-java-base-docs/build.gradle.kts` depends on.
+
+## 3. Generate the JSON
+
+```bash
+cd ../../JavaTemplated/dokka-java-base-docs
+export JAVA_TOOL_OPTIONS="-Xss256m"   # see "The Dokka bug" below
+./gradlew dokkaGenerateHtml
+```
+
+(The task is still named `dokkaGenerateHtml` - `kdoc-to-json` overrides Dokka's renderer
+to emit JSON instead, it doesn't rename the task.) Output lands in
+`build/dokka-json-output/` - 22,209 files, ~215MB for the full module.
+
+## 4. Render the JSON to HTML
+
+```bash
+cd ../pebble-renderer
+./gradlew run --args="../dokka-java-base-docs/build/dokka-json-output ../html-output"
+```
+
+The Pebble template defaults to `../peb.peb.txt` (this project's sibling above); pass a
+third arg to `--args` to use a different one. See `pebble-renderer`'s own usage text
+(`./gradlew run` with no args) for details.
+
+## The Dokka bug
+
+A full `java.base` run crashes with `java.lang.StackOverflowError` deep in Dokka's own
+Java `{@inheritDoc}` resolver
+(`org.jetbrains.dokka.analysis.java.parsers.doctag.PsiElementToHtmlConverter`) - a real,
+still-open upstream bug ([kotlin/dokka#2171](https://github.com/Kotlin/dokka/issues/2171)),
+not something introduced by `kdoc-to-json` or this project's config. Confirmed independent
+of stack size (crashes identically at a 256MB thread stack, 512x the JVM default) and
+**not** fixed by Dokka 2.2.0 GA (`kdoc-to-json` normally targets 2.2.0-Beta).
+
+`bisect_inheritdoc.py` binary-searches subsets of a package's source files (via a staging
+directory of symlinks passed as Dokka's sole `sourceRoots` entry - the only mechanism that
+actually excludes a file from analysis; Dokka re-walks each `sourceRoots` directory from
+disk rather than respecting Gradle-level `FileTree`/`suppressedFiles` filters) to find the
+minimal file or file-pair that reproduces the crash.
+
+`dokka-java-base-docs/build.gradle.kts` hard-excludes the 9 pairs (18 files) found this
+way - each a class alongside its immediate super/interface, both carrying heavy
+`{@inheritDoc}` javadoc:
+
+| Pair | Package |
+| --- | --- |
+| `Executable` / `Constructor` | `java.lang.reflect` |
+| `AccessibleObject` / `Field` | `java.lang.reflect` |
+| `BufferedReader` / `LineNumberReader` | `java.io` |
+| `AbstractList` / `AbstractSequentialList` | `java.util` |
+| `NavigableMap` / `TreeMap` | `java.util` |
+| `NavigableSet` / `TreeSet` | `java.util` |
+| `ConcurrentNavigableMap` / `ConcurrentSkipListMap` | `java.util.concurrent` |
+| `ScheduledThreadPoolExecutor` / `ThreadPoolExecutor` | `java.util.concurrent` |
+| `BlockingDeque` / `LinkedBlockingDeque` | `java.util.concurrent` |
+
+That's 18 of ~2,750 `java.base` source files (0.65%) missing their own page; everything
+else - including the *other* half of each pair above - documents normally. If this needs
+revisiting, rerun `bisect_inheritdoc.py <package>` for any package that fails and fold the
+newly found files into the exclude list in `build.gradle.kts`.
+
+## Not carried over from the original investigation
+
+A one-off fork of `kdoc-to-json` rebuilt against Dokka 2.2.0 GA (to test whether a newer
+Dokka fixed the bug above - it didn't) isn't included here; it added nothing to the
+working pipeline. Raw bisection/build logs from that investigation also aren't included -
+none of it is needed to build the docs, only `bisect_inheritdoc.py` itself is kept, for
+future maintenance.

JavaTemplated/dokka-java-base-docs/bisect_inheritdoc.py

@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+"""Binary-search bisection to find the minimal set of .java files in a java.base package
+that trigger Dokka's StackOverflowError (github.com/Kotlin/dokka/issues/2171 - an
+unresolvable {@inheritDoc}/related tag during Java analysis).
+
+IMPORTANT: uses TRUE inclusion via a staging directory of symlinks passed as the sole
+sourceRoots entry (-PabsSourceRoot). An earlier version of this script used the
+`suppressedFiles` Dokka option to exclude candidates instead, which turned out to be a
+dead end: suppressedFiles only filters final output pages - the analysis phase where the
+crash happens still processes every file under sourceRoots regardless of it. Confirmed by
+direct experiment: suppressing 268/269 files still crashed identically, but truly
+restricting sourceRoots to that same 1 file (via a symlink staging dir) built successfully.
+
+Usage: python3 bisect_inheritdoc.py <package, e.g. java/lang> [java/io ...]
+"""
+import glob
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+PROJECT_DIR = os.path.dirname(os.path.abspath(__file__))
+JDK_BASE_CLASSES = os.path.abspath(
+    os.path.join(PROJECT_DIR, "..", "jdk17", "src", "java.base", "share", "classes")
+)
+GRADLEW = os.path.join(PROJECT_DIR, "gradlew")
+
+
+def find_jdk17_home():
+    """The Kotlin Gradle plugin kdoc-to-json depends on doesn't support this machine's
+    default JDK as a compile target if it's newer than 17, so JAVA_HOME must be pinned to
+    an actual JDK 17 install. Set JDK17_HOME to override; otherwise this tries macOS's
+    `java_home` locator (install one with e.g. `brew install openjdk@17` if it fails)."""
+    override = os.environ.get("JDK17_HOME")
+    if override:
+        return override
+    try:
+        result = subprocess.run(
+            ["/usr/libexec/java_home", "-v", "17"],
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+        return result.stdout.strip()
+    except Exception as e:
+        raise RuntimeError(
+            "Could not locate a JDK 17 installation via /usr/libexec/java_home. Install "
+            "one (e.g. `brew install openjdk@17`) or set the JDK17_HOME environment "
+            "variable to its home directory."
+        ) from e
+
+
+ENV = dict(os.environ)
+ENV["JAVA_HOME"] = find_jdk17_home()
+ENV["JAVA_TOOL_OPTIONS"] = "-Xss256m"
+
+test_count = 0
+
+# Files already confirmed (via prior bisection runs on java/lang, java/io, java/util in
+# isolation) to trigger github.com/Kotlin/dokka/issues/2171 - excluded up front so a
+# broader run (e.g. the whole "java" top-level dir, all 9 subpackages together) looks for
+# NEW combos instead of rediscovering these.
+KNOWN_BAD = {
+    os.path.join(JDK_BASE_CLASSES, rel)
+    for rel in [
+        "java/lang/reflect/Constructor.java",
+        "java/lang/reflect/Executable.java",
+        "java/lang/reflect/AccessibleObject.java",
+        "java/lang/reflect/Field.java",
+        "java/io/BufferedReader.java",
+        "java/io/LineNumberReader.java",
+        "java/util/AbstractList.java",
+        "java/util/AbstractSequentialList.java",
+        "java/util/NavigableMap.java",
+        "java/util/TreeMap.java",
+        "java/util/concurrent/ConcurrentNavigableMap.java",
+        "java/util/concurrent/ConcurrentSkipListMap.java",
+        "java/util/concurrent/ScheduledThreadPoolExecutor.java",
+        "java/util/concurrent/ThreadPoolExecutor.java",
+    ]
+}
+
+
+def all_java_files(pkg):
+    files = sorted(
+        glob.glob(os.path.join(JDK_BASE_CLASSES, pkg, "**", "*.java"), recursive=True)
+    )
+    return [f for f in files if f not in KNOWN_BAD]
+
+
+def make_staging_dir(included):
+    staging = tempfile.mkdtemp(prefix="dokka-stage-", dir="/tmp")
+    for f in included:
+        rel = os.path.relpath(f, JDK_BASE_CLASSES)
+        dest = os.path.join(staging, rel)
+        os.makedirs(os.path.dirname(dest), exist_ok=True)
+        os.symlink(f, dest)
+    return staging
+
+
+def test_fails(included):
+    """True if this exact set of files, as the ONLY sourceRoots content, reproduces the
+    StackOverflowError. False if it builds successfully. Raises on anything else."""
+    global test_count
+    test_count += 1
+    staging = make_staging_dir(included)
+    try:
+        proc = subprocess.run(
+            [GRADLEW, "dokkaGenerateHtml", f"-PabsSourceRoot={staging}"],
+            cwd=PROJECT_DIR,
+            env=ENV,
+            capture_output=True,
+            text=True,
+            timeout=300,
+        )
+    finally:
+        shutil.rmtree(staging, ignore_errors=True)
+
+    out = proc.stdout + proc.stderr
+    names = ", ".join(os.path.basename(f) for f in included) if len(included) <= 8 else ""
+    print(f"  [test {test_count}] {len(included)} files {names} -> ", end="", flush=True)
+    if "BUILD SUCCESSFUL" in out:
+        print("PASS")
+        return False
+    if "StackOverflowError" in out:
+        print("FAIL (StackOverflowError)")
+        return True
+    print("UNEXPECTED RESULT")
+    print(out[-4000:])
+    raise RuntimeError(f"Unexpected gradle result for {len(included)} files")
+
+
+def shrink_with_anchor(pool, anchor):
+    """pool + anchor is known to fail. Binary-search pool (keeping the full anchor fixed
+    and present in every test) down to a minimal subset that, combined with anchor, still
+    fails. Scales as O(log n) instead of the combinatorial blowup of trying every subset."""
+    if len(pool) <= 1:
+        return pool
+    mid = len(pool) // 2
+    left, right = pool[:mid], pool[mid:]
+    if test_fails(left + anchor):
+        return shrink_with_anchor(left, anchor)
+    if test_fails(right + anchor):
+        return shrink_with_anchor(right, anchor)
+    return pool  # both halves need each other too - can't shrink further this way
+
+
+def find_minimal_combo(left, right):
+    """left+right is confirmed to fail, but neither alone does - the crash needs files
+    from both sides. Alternately shrink each side while anchoring the other, converging
+    on a small (not always provably minimal, but tight) cross-cutting combo."""
+    print(f"  -- {len(left) + len(right)} files fail together but neither half alone does; "
+          f"anchored-shrinking to find the cross-cutting combo")
+    min_left = shrink_with_anchor(left, right)
+    min_right = shrink_with_anchor(right, min_left)
+    # One more pass: min_right may be small enough now to shrink min_left further.
+    min_left = shrink_with_anchor(min_left, min_right)
+    combo = min_left + min_right
+    assert test_fails(combo), "combo lost the failure during minimization - bug in shrink logic"
+    return combo
+
+
+def find_bad(candidates):
+    if not candidates:
+        return []
+    if not test_fails(candidates):
+        return []
+    if len(candidates) == 1:
+        print(f"  >>> BAD FILE: {candidates[0]}")
+        return candidates
+    mid = len(candidates) // 2
+    left, right = candidates[:mid], candidates[mid:]
+    bad_left = find_bad(left)
+    bad_right = find_bad(right)
+    if bad_left or bad_right:
+        return bad_left + bad_right
+    combo = find_minimal_combo(left, right)
+    print(f"  >>> MINIMAL FAILING COMBO ({len(combo)} files): {[os.path.basename(f) for f in combo]}")
+    # BUG FIX: finding one combo does NOT prove the rest of this branch is clean - a
+    # second, independent problem could be hiding in the same left+right split. Recurse
+    # into what's left after removing the combo to be sure. (This was missing in the
+    # first few runs, which is why re-testing "java" with all found files excluded still
+    # crashed - at least one more combo was hiding, unverified, in an already-resolved
+    # branch.)
+    combo_set = set(combo)
+    remaining = [f for f in candidates if f not in combo_set]
+    more_bad = find_bad(remaining)
+    return combo + more_bad
+
+
+def main():
+    packages = sys.argv[1:]
+    if not packages:
+        print(__doc__)
+        sys.exit(1)
+
+    results = {}
+    for pkg in packages:
+        print(f"=== Bisecting {pkg} ===", flush=True)
+        files = all_java_files(pkg)
+        print(f"  {len(files)} total .java files under {pkg}")
+        bad = find_bad(files)
+        results[pkg] = bad
+        print(f"  {pkg}: found {len(bad)} minimal-failing file(s) after {test_count} tests total so far")
+
+    print("\n=== SUMMARY ===")
+    for pkg, bad in results.items():
+        print(f"{pkg}: {len(bad)} file(s) in minimal failing set")
+        for f in bad:
+            print(f"  - {f}")
+
+
+if __name__ == "__main__":
+    main()