Update Highlights and intros

code_generator/config/programs.yaml

@@ -20,6 +20,7 @@ programs:
   language: 6502 Assembly
   description: The game that invented cinematic platformers — built by one developer over four years on Apple II hardware
     with 128K of RAM. The source code was recovered in 2012 from 22-year-old floppy disks.
+  wikipedia_url: https://en.wikipedia.org/wiki/Prince_of_Persia_(1989_video_game)
   github_url: https://github.com/jmechner/Prince-of-Persia-Apple-II
   github_repo: jmechner/Prince-of-Persia-Apple-II
   github_branch: master
@@ -182,8 +183,10 @@ programs:
   author: Anderson, Blank, Daniels, Lebling
   year: 1977
   language: MDL (Muddle)
-  description: The game that invented the text adventure — built organically at MIT on a PDP-10, recovered from a 1990 tape,
-    and released open-source in 2025. Its parser, its puzzles, and its sarcastic error messages changed computing forever.
+  description: One of the earliest and most influential text adventures — built organically at MIT on a PDP-10, inspired
+    by Colossal Cave Adventure, recovered from a 1990 tape, and released open-source in 2025. Its sophisticated parser,
+    intricate puzzles, and sarcastic error messages set the template for the genre.
+  wikipedia_url: https://en.wikipedia.org/wiki/Zork
   github_url: https://github.com/MITDDC/zork
   github_repo: MITDDC/zork
   github_branch: master
@@ -269,6 +272,7 @@ programs:
   language: 8086 Assembly
   description: Written in six weeks by one programmer, acquired by Microsoft for $25,000, and licensed to IBM — then to every
     IBM clone maker on earth. The source code of the PC era.
+  wikipedia_url: https://en.wikipedia.org/wiki/MS-DOS
   github_url: https://github.com/microsoft/MS-DOS
   github_repo: microsoft/MS-DOS
   github_branch: main
@@ -523,6 +527,7 @@ programs:
   year: 1977
   language: 6502 Assembly
   description: Microsoft's implementation of BASIC for the 6502 microprocessor, a pivotal moment in personal computing history.
+  wikipedia_url: https://en.wikipedia.org/wiki/Microsoft_BASIC
   context: Microsoft BASIC for the 6502 was a key enabler for early personal computers, including the Apple II. Written in
     assembly language, it was designed to fit within the tight memory constraints of the era, often just a few kilobytes.
     This version showcases the ingenuity required to deliver a functional programming language on limited hardware.
@@ -543,7 +548,9 @@ programs:
   author: John Carmack, John Romero, Tom Hall
   year: 1992
   language: C, x86 Assembly
-  description: The groundbreaking source code for Wolfenstein 3D, a seminal first-person shooter that defined the genre.
+  description: The groundbreaking source code for Wolfenstein 3D, a seminal first-person shooter that popularized the genre
+    and established the template for games like DOOM and Quake.
+  wikipedia_url: https://en.wikipedia.org/wiki/Wolfenstein_3D
   context: Wolfenstein 3D was developed by id Software and released in 1992 for MS-DOS. It introduced fast-paced, first-person
     action with smooth scrolling and immersive gameplay, pushing the limits of hardware at the time. The source code, later
     released as open source, showcases the ingenuity behind one of gaming's most influential titles.
@@ -697,6 +704,7 @@ programs:
   language: C
   description: The source code for DOOM, one of the most influential video games of all time, showcasing early 3D graphics
     and multiplayer networking.
+  wikipedia_url: https://en.wikipedia.org/wiki/Doom_(1993_video_game)
   context: DOOM revolutionized gaming with its fast-paced action, immersive 3D environments, and multiplayer capabilities.
     Originally released in 1993, its source code was later made available, allowing developers and enthusiasts to study its
     groundbreaking techniques. The game ran on modest hardware of the era, pushing the limits of what was possible on consumer
@@ -921,7 +929,9 @@ programs:
   author: John Carmack, Michael Abrash, John Cash
   year: 1996
   language: C, x86 Assembly
-  description: The groundbreaking source code for Quake, a seminal first-person shooter that defined 3D gaming.
+  description: The groundbreaking source code for Quake, a seminal first-person shooter with a fully 3D engine that
+    influenced an entire generation of game developers.
+  wikipedia_url: https://en.wikipedia.org/wiki/Quake_(video_game)
   context: Quake was developed by id Software and released in 1996, introducing true 3D environments and multiplayer capabilities
     that revolutionized the gaming industry. The codebase showcases advanced rendering techniques and optimization for hardware
     constraints of the era, such as x86 processors and limited memory. The release of the source code under the GPL in 1999

code_generator/find_images.py

@@ -133,6 +133,40 @@ def _is_relevant(topic: str, image_name: str, caption: str, client) -> bool:
         return True  # fail open
 
 
+# ---------------------------------------------------------------------------
+# Wikipedia text extract
+# ---------------------------------------------------------------------------
+
+def fetch_wiki_extract(wikipedia_url: str, max_chars: int = 3000) -> str | None:
+    """Return the plain-text extract for a Wikipedia article URL, or None.
+
+    Fetches the full article extract (intro + body sections) so the caller
+    can use it as a factual grounding source when generating LLM content.
+    Truncates to *max_chars* characters to keep prompts manageable.
+    """
+    if not wikipedia_url:
+        return None
+    m = re.search(r"/wiki/(.+)$", wikipedia_url)
+    if not m:
+        return None
+    title = m.group(1)
+    try:
+        data = _post(WIKIPEDIA_API, {
+            "action": "query",
+            "titles": title,
+            "prop": "extracts",
+            "explaintext": "true",
+            "exsectionformat": "plain",
+        })
+        for page in data.get("query", {}).get("pages", {}).values():
+            extract = page.get("extract", "")
+            if extract:
+                return extract[:max_chars]
+    except Exception:
+        pass
+    return None
+
+
 # ---------------------------------------------------------------------------
 # Public API
 # ---------------------------------------------------------------------------

code_generator/formatter.py

@@ -125,12 +125,29 @@ def _format_from_data(
     code_lines: list[str],
     data: dict,
     is_excerpt: bool,
+    preserved_ranges: dict | None = None,
 ) -> str:
-    """Write YAML front matter + fenced code block from an already-parsed data dict."""
+    """Write YAML front matter + fenced code block from an already-parsed data dict.
+
+    preserved_ranges: optional mapping of enhancement id → (line_start, line_end).
+    When provided, any enhancement whose id appears in the mapping will have its
+    line_start/line_end replaced with the preserved values, protecting range work
+    from being overwritten by fresh LLM output.
+    """
     description = data.get("description", file_cfg.get("description", ""))
     summary = data.get("summary", [])
     enhancements = _clean_enhancements(data.get("enhancements", []), code_lines)
 
+    if preserved_ranges:
+        patched = []
+        for enh in enhancements:
+            eid = enh.get("id", "")
+            if eid in preserved_ranges:
+                enh = dict(enh)
+                enh["line_start"], enh["line_end"] = preserved_ranges[eid]
+            patched.append(enh)
+        enhancements = patched
+
     # Build a mapping from original 1-based line numbers to filtered 1-based line
     # numbers, removing lines that would break GitHub's markdown renderer (e.g.
     # Emacs file-mode comments like "// Emacs style mode select   -*- C++ -*-"
@@ -219,9 +236,11 @@ def format_file(
     code_lines: list[str],
     raw_json: str,
     is_excerpt: bool,
+    preserved_ranges: dict | None = None,
 ) -> str:
     data = _parse_json(raw_json)
-    return _format_from_data(program, file_cfg, code_lines, data, is_excerpt)
+    return _format_from_data(program, file_cfg, code_lines, data, is_excerpt,
+                             preserved_ranges=preserved_ranges)
 
 
 def format_file_from_dict(
@@ -230,6 +249,8 @@ def format_file_from_dict(
     code_lines: list[str],
     data: dict,
     is_excerpt: bool,
+    preserved_ranges: dict | None = None,
 ) -> str:
     """Format a file from an already-parsed data dict (skips JSON parsing)."""
-    return _format_from_data(program, file_cfg, code_lines, data, is_excerpt)
+    return _format_from_data(program, file_cfg, code_lines, data, is_excerpt,
+                             preserved_ranges=preserved_ranges)

code_generator/generator.py

@@ -15,7 +15,7 @@
 from prompts import build_prompt, build_chunk_prompt, CHUNK_THRESHOLD, _asm_landmarks
 from intro_prompts import build_intro_prompt
 from formatter import format_file, format_file_from_dict, parse_response_json
-from find_images import fill_file_images, find_program_image
+from find_images import fill_file_images, find_program_image, fetch_wiki_extract
 from range_fixer import fix_ranges
 from highlights_prompts import build_highlights_prompt
 import catalog_sync
@@ -26,6 +26,33 @@
 _CATALOG_PATH = Path(__file__).parent.parent / "public" / "catalog.json"
 
 
+def _load_existing_ranges(path: Path) -> dict:
+    """Read an existing .md file and return {enhancement_id: (line_start, line_end)}.
+
+    Used by --preserve-ranges to protect manually-corrected ranges from being
+    overwritten when regenerating enhancement content.
+    """
+    if not path.exists():
+        return {}
+    try:
+        text = path.read_text(encoding="utf-8")
+        # Extract the YAML frontmatter between the first pair of --- delimiters
+        m = re.match(r"^---\n(.*?)\n---", text, re.DOTALL)
+        if not m:
+            return {}
+        meta = yaml.safe_load(m.group(1))
+        result = {}
+        for enh in meta.get("enhancements") or []:
+            eid = enh.get("id")
+            ls = enh.get("line_start")
+            le = enh.get("line_end")
+            if eid and ls is not None and le is not None:
+                result[eid] = (int(ls), int(le))
+        return result
+    except Exception:
+        return {}
+
+
 def save_introduction(slug: str, intro_text: str) -> None:
     if _CATALOG_PATH.exists():
         catalog = json.loads(_CATALOG_PATH.read_text(encoding="utf-8"))
@@ -108,7 +135,9 @@ def generate_highlights(program: dict, client, gen_cfg: dict, force: bool,
         return False
 
     console.print(f"[cyan]gen   {slug}/highlights[/cyan]")
-    messages = build_highlights_prompt(program, files_with_enhancements)
+    wiki_url = program.get("wikipedia_url")
+    wiki_text = fetch_wiki_extract(wiki_url) if wiki_url else None
+    messages = build_highlights_prompt(program, files_with_enhancements, wiki_text=wiki_text)
 
     try:
         raw = client.complete(
@@ -267,7 +296,17 @@ def generate_intro(program: dict, client, gen_cfg: dict, force: bool, fetch_imag
 
     console.print(f"[cyan]gen   {slug}/introduction[/cyan]")
 
-    messages = build_intro_prompt(program)
+    wiki_url = program.get("wikipedia_url")
+    wiki_text = None
+    if wiki_url:
+        console.print("  fetching Wikipedia article…")
+        wiki_text = fetch_wiki_extract(wiki_url)
+        if wiki_text:
+            console.print(f"  [dim]-> {len(wiki_text)} chars fetched[/dim]")
+        else:
+            console.print("  [dim]Wikipedia fetch returned nothing[/dim]")
+
+    messages = build_intro_prompt(program, wiki_text=wiki_text)
 
     console.print("  calling model (intro)…")
     try:
@@ -335,6 +374,7 @@ def main() -> None:
     parser.add_argument("--replace-images", action="store_true", help="Clear and re-fetch all images (replaces bad ones); implies --find-images")
     parser.add_argument("--program-image", action="store_true", help="Fetch (or replace) the program intro image only; skip file images")
     parser.add_argument("--fix-ranges", action="store_true", help="Post-process existing .md files to correct enhancement line ranges and exit")
+    parser.add_argument("--preserve-ranges", action="store_true", help="When regenerating, keep existing line_start/line_end values (matched by enhancement id) instead of using the LLM's new ranges")
     parser.add_argument("--config", default="config/programs.yaml", help="Path to programs.yaml")
     args = parser.parse_args()
 
@@ -485,6 +525,16 @@ def main() -> None:
 
             console.print(f"[cyan]gen   {prog_slug}/{file_slug}[/cyan]")
 
+            # Load existing ranges before regeneration so --preserve-ranges can
+            # patch them back in after the LLM produces new content.
+            existing_path = output_dir / prog_slug / f"{file_slug}.md"
+            preserved_ranges = (
+                _load_existing_ranges(existing_path)
+                if args.preserve_ranges else None
+            )
+            if preserved_ranges:
+                console.print(f"  [dim]preserving {len(preserved_ranges)} existing range(s)[/dim]")
+
             max_lines = file_cfg.get("max_lines") or gen_cfg.get("default_max_lines") or None
             try:
                 code_lines, is_excerpt = fetch_source(
@@ -558,7 +608,8 @@ def main() -> None:
                     continue
 
                 merged = _merge_chunk_responses(parsed_chunks)
-                content = format_file_from_dict(program, file_cfg, code_lines, merged, is_excerpt)
+                content = format_file_from_dict(program, file_cfg, code_lines, merged, is_excerpt,
+                                                preserved_ranges=preserved_ranges)
 
             else:
                 # --- Standard path for small files ---
@@ -582,7 +633,8 @@ def main() -> None:
                     console.print(f"  [red]{e}[/red]")
                     continue
 
-                content = format_file(program, file_cfg, code_lines, raw, is_excerpt)
+                content = format_file(program, file_cfg, code_lines, raw, is_excerpt,
+                                     preserved_ranges=preserved_ranges)
 
             path = ckpt.save(prog_slug, file_slug, content)
             console.print(f"  [green]-> {path}[/green]")

code_generator/highlights_prompts.py

@@ -5,7 +5,8 @@
 They are the things that made it legendary: the innovations that changed how software or games were built, \
 the tricks that made developers' jaws drop, the moments that defined a genre or an industry.
 
-You will receive a program's metadata and a list of its annotated source files with their section titles. \
+You will receive a program's metadata, a list of its annotated source files with their section titles, \
+and when available, the Wikipedia article about the program. \
 Use this to identify which files contain the code behind each highlight.
 
 Output valid JSON only — no markdown fences and no extra text:
@@ -41,10 +42,18 @@
 - Write past tense for history; present tense for what the code does
 - Do not end any paragraph with: "This underscores", "This highlights", "This reflects", \
   "This reinforces", "This exemplifies", "It is worth noting", "It is important to note"
+- FACTUAL ACCURACY: When a Wikipedia article is provided, treat it as the authoritative \
+  source for all historical claims. Do not assert facts that contradict or are absent from \
+  the Wikipedia article. Only use "first" or superlative claims if the Wikipedia article \
+  supports them explicitly.
 """
 
 
-def build_highlights_prompt(program: dict, files_with_enhancements: list[dict]) -> list[dict]:
+def build_highlights_prompt(
+    program: dict,
+    files_with_enhancements: list[dict],
+    wiki_text: str | None = None,
+) -> list[dict]:
     """Build a prompt for generating program highlights.
 
     files_with_enhancements: list of dicts with:
@@ -62,6 +71,13 @@ def build_highlights_prompt(program: dict, files_with_enhancements: list[dict])
     if program.get("context"):
         parts.append(f"Historical context: {program['context'].strip()}")
 
+    if wiki_text:
+        parts.append(
+            "\n--- Wikipedia article (authoritative factual source) ---\n"
+            + wiki_text.strip()
+            + "\n--- End Wikipedia article ---"
+        )
+
     parts.append("\nAnnotated source files (slug — title):")
     for f in files_with_enhancements:
         parts.append(f"\n  slug: \"{f['slug']}\"  —  {f['title']}")

code_generator/intro_prompts.py

@@ -5,8 +5,8 @@
 reader inside the moment this program was created, helping them feel the constraints, \
 the urgency, and the ingenuity of the people who wrote it.
 
-You will receive metadata about a historically important program. Write a 4–6 paragraph \
-historical narrative introduction about it.
+You will receive metadata about a historically important program, and when available, \
+the Wikipedia article about it. Write a 4–6 paragraph historical narrative introduction.
 
 Output valid JSON with no markdown fences and no extra text:
 {"introduction": "paragraph1\n\nparagraph2\n\n..."}
@@ -26,10 +26,15 @@
 - Be specific: name the people, the machines, the years, the dollar amounts, the deadlines
 - Do not end any paragraph with: "This underscores", "This highlights", "This reflects", \
   "This reinforces", "This exemplifies", "It is worth noting", "It is important to note"
+- FACTUAL ACCURACY: When a Wikipedia article is provided, treat it as the authoritative \
+  source for all historical claims. Do not assert facts that contradict or are absent from \
+  the Wikipedia article. Pay special attention to superlatives and "first" claims — only \
+  make them if the Wikipedia article supports them. If Wikipedia mentions notable \
+  predecessors or context that complicates a claim, reflect that nuance.
 """
 
 
-def build_intro_prompt(program: dict) -> list[dict]:
+def build_intro_prompt(program: dict, wiki_text: str | None = None) -> list[dict]:
     parts = [
         f"Title: {program['title']}",
         f"Year: {program['year']}",
@@ -53,6 +58,13 @@ def build_intro_prompt(program: dict) -> list[dict]:
             desc = f.get("description", "")
             parts.append(f"  - {f['title']}: {desc}")
 
+    if wiki_text:
+        parts.append(
+            "\n--- Wikipedia article (authoritative factual source) ---\n"
+            + wiki_text.strip()
+            + "\n--- End Wikipedia article ---"
+        )
+
     user_content = "\n".join(parts)
 
     return [