docs: tighten atlas–guide routing and fix starter-API/eval-budget claims - #192
Merged
Conversation
One quality pass over the atlas and the four strategy guides before shipping to participants — wayfinding, honesty about the two starter-agent APIs, and one accuracy fix, with every guide snippet re-executed as written. - Guides now route by path: Path A (extend a reference implementation via its 99_starter_agent + guide 3) vs Path B (bring your own dataset, guides 1→2→4). guides/README opens with the atlas as the map; each guide links back to its atlas section. - Guide 3 no longer claims the sibling starters share energy's template: a new 'Which starter are you on?' section maps energy's ToolSpec toolbelt to the sp500/food/BoC enable_* toggles (same AgentConfig fields), with a verified model_copy variant of the inventory-first search example for toggle starters. - Guide 2 now matches the code (and atlas §04): evaluate() enforces max_runs only when an EvalTracker is attached; shown with the two-line tracker usage. - Atlas: §06 gains a five-row 'Where do you land?' chooser replacing the energy-only CTA; §07 is reframed as Path A vs Path B; §02/§04/§05 refs deep- link real guide headings; hero names guides/ as the worksheet layer. - The guides' committed sample series is re-themed diesel → lumber so it can't be confused with the energy/WTI track (pure rename, CSV values untouched — all deterministic expected outputs verified bit-identical; AutoARIMA's known run-to-run wobble is now called out in guides 2 and 4). - Guide 4 labels the borrowed energy artifacts (WTI prompt builder, committed eval YAML) as a second, optional lab distinct from the Harbourview run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One quality-iteration pass over the architecture atlas and the four strategy guides before they ship to participants. Same participant narrative, two file types, one review. Follow-up to #190/#191.
What changed
Routing (the two-path vocabulary).
guides/README.mdnow opens with the atlas as the map and routes by path instead of a numbered ladder: Path A — extend a reference implementation (its99_starter_agent.ipynb+ guide 3, close with guide 4) — vs Path B — bring your own dataset (guides 1 → 2 → optional 3 → 4). The atlas mirrors this: §06 gains a five-row Where do you land? chooser (replacing the energy-only CTA), §07 is reframed as Path A vs Path B, and §02/§04/§05 refs deep-link real guide headings. Each guide links back to its atlas section.Starter-API honesty (the structural docs bug). Guide 3 previously claimed the sibling implementations "share the same template, so everything transfers" — but only energy composes via a
ToolSpectoolbelt (starter_agent/tools.py); the sp500/food/BoC starters exposeenable_search/enable_code_exectoggles and have notools.py. A new opening section, Which starter are you on?, maps the two styles onto the sharedAgentConfigcontract, and the inventory-first search example gains a verifiedmodel_copyvariant for toggle starters (construction executed against the real sp500 package).Eval-budget accuracy. Guide 2 implied
evaluate()always enforcesmax_runs. Perevaluation/eval.py, the budget only applies when anEvalTrackeris attached — the guide now says so (with the two-line tracker usage) and agrees with atlas §04.Sample-dataset re-theme (diesel → lumber). The committed synthetic series read as a knockoff of the energy/WTI track (diesel spot, USD/bbl). It is now the Harbourview lumber spot price (USD/m³) — a pure rename: the CSV is byte-identical (git shows a 100% rename), so every printed number in guides 1/2/4 remains valid. This also sharpens guide 4's payload-audit beat: the borrowed WTI builder's
last_close_usd_bblfield is now a genuinely wrong-units mislabel you catch by reading. Guide 4 additionally boxes the borrowed energy artifacts (Audit 2) as a second, optional lab so they can't read as part of the Harbourview experiment.Verification
DartsAutoARIMAPredictoris not run-to-run deterministic (four fresh runs: CRPS 1.928–1.952 vs printed 1.922). Printed numbers were left as-is; guides 2 and 4 now carry a one-line heads-up so participants don't mistake the wobble for a mistake.Out of scope (unchanged from the plan)
No starter-API unification in code, no guide rewrites, no implementation README/notebook changes, no AutoARIMA re-runs to refresh printed tables.
🤖 Generated with Claude Code