Skip to content

docs: tighten atlas–guide routing and fix starter-API/eval-budget claims - #192

Merged
ethancjackson merged 1 commit into
mainfrom
docs/atlas-guides-iteration
Aug 20, 2026
Merged

docs: tighten atlas–guide routing and fix starter-API/eval-budget claims#192
ethancjackson merged 1 commit into
mainfrom
docs/atlas-guides-iteration

Conversation

@ethancjackson

Copy link
Copy Markdown
Collaborator

One quality-iteration pass over the architecture atlas and the four strategy guides before they ship to participants. Same participant narrative, two file types, one review. Follow-up to #190/#191.

What changed

Routing (the two-path vocabulary). guides/README.md now opens with the atlas as the map and routes by path instead of a numbered ladder: Path A — extend a reference implementation (its 99_starter_agent.ipynb + guide 3, close with guide 4) — vs Path B — bring your own dataset (guides 1 → 2 → optional 3 → 4). The atlas mirrors this: §06 gains a five-row Where do you land? chooser (replacing the energy-only CTA), §07 is reframed as Path A vs Path B, and §02/§04/§05 refs deep-link real guide headings. Each guide links back to its atlas section.

Starter-API honesty (the structural docs bug). Guide 3 previously claimed the sibling implementations "share the same template, so everything transfers" — but only energy composes via a ToolSpec toolbelt (starter_agent/tools.py); the sp500/food/BoC starters expose enable_search/enable_code_exec toggles and have no tools.py. A new opening section, Which starter are you on?, maps the two styles onto the shared AgentConfig contract, and the inventory-first search example gains a verified model_copy variant for toggle starters (construction executed against the real sp500 package).

Eval-budget accuracy. Guide 2 implied evaluate() always enforces max_runs. Per evaluation/eval.py, the budget only applies when an EvalTracker is attached — the guide now says so (with the two-line tracker usage) and agrees with atlas §04.

Sample-dataset re-theme (diesel → lumber). The committed synthetic series read as a knockoff of the energy/WTI track (diesel spot, USD/bbl). It is now the Harbourview lumber spot price (USD/m³) — a pure rename: the CSV is byte-identical (git shows a 100% rename), so every printed number in guides 1/2/4 remains valid. This also sharpens guide 4's payload-audit beat: the borrowed WTI builder's last_close_usd_bbl field is now a genuinely wrong-units mislabel you catch by reading. Guide 4 additionally boxes the borrowed energy artifacts (Audit 2) as a second, optional lab so they can't read as part of the Harbourview experiment.

Verification

  • Every runnable snippet in guides 1, 2, and 4 was re-executed from the repo root after the edits. All deterministic outputs (row counts, naive-baseline CRPS, coverage tables, paired stats) match the printed text exactly.
  • DartsAutoARIMAPredictor is not run-to-run deterministic (four fresh runs: CRPS 1.928–1.952 vs printed 1.922). Printed numbers were left as-is; guides 2 and 4 now carry a one-line heads-up so participants don't mistake the wobble for a mistake.
  • Guide 3's new toggle snippet constructs successfully against the real sp500 starter package (no credentials needed).
  • Atlas: all new hrefs verified against the repo tree; anchor fragments checked against the actual guide headings; visually QA'd in light + dark themes and at mobile width (chooser stacks cleanly, no horizontal overflow).

Out of scope (unchanged from the plan)

No starter-API unification in code, no guide rewrites, no implementation README/notebook changes, no AutoARIMA re-runs to refresh printed tables.

🤖 Generated with Claude Code

One quality pass over the atlas and the four strategy guides before shipping
to participants — wayfinding, honesty about the two starter-agent APIs, and
one accuracy fix, with every guide snippet re-executed as written.

- Guides now route by path: Path A (extend a reference implementation via its
  99_starter_agent + guide 3) vs Path B (bring your own dataset, guides 1→2→4).
  guides/README opens with the atlas as the map; each guide links back to its
  atlas section.
- Guide 3 no longer claims the sibling starters share energy's template: a new
  'Which starter are you on?' section maps energy's ToolSpec toolbelt to the
  sp500/food/BoC enable_* toggles (same AgentConfig fields), with a verified
  model_copy variant of the inventory-first search example for toggle starters.
- Guide 2 now matches the code (and atlas §04): evaluate() enforces max_runs
  only when an EvalTracker is attached; shown with the two-line tracker usage.
- Atlas: §06 gains a five-row 'Where do you land?' chooser replacing the
  energy-only CTA; §07 is reframed as Path A vs Path B; §02/§04/§05 refs deep-
  link real guide headings; hero names guides/ as the worksheet layer.
- The guides' committed sample series is re-themed diesel → lumber so it can't
  be confused with the energy/WTI track (pure rename, CSV values untouched —
  all deterministic expected outputs verified bit-identical; AutoARIMA's known
  run-to-run wobble is now called out in guides 2 and 4).
- Guide 4 labels the borrowed energy artifacts (WTI prompt builder, committed
  eval YAML) as a second, optional lab distinct from the Harbourview run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ethancjackson
ethancjackson merged commit 1ca028d into main Aug 20, 2026
1 check passed
@ethancjackson
ethancjackson deleted the docs/atlas-guides-iteration branch August 20, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant