run_evolution checkpoint/target_score params; propagate llm.provider (#472); background_blur example#457
Merged
codelion merged 4 commits intoJul 14, 2026
Conversation
…ality gate A self-contained example of the "optimise a real hot function" pattern, where the score is a speedup but fidelity is a hard gate: a fast-but-wrong candidate scores zero. Deterministic and dependency-light (numpy only) - the video sequence is synthesised from a fixed seed, so there is no webcam, GPU, model or dataset. The seed is a correct-but-slow O(k^2) 2D Gaussian convolution. Over 100 iterations the search reached 62x over that seed (2.6x over a hand-written separable+fp32 implementation), stacking separable convolution, a half-resolution blur pipeline, float32, batching across the sequence, and a lerp composite. 28 of 100 candidates were rejected outright by the quality gate. Two things this example exists to demonstrate: - The obvious quality gate is exploitable. Grading mean SSIM and worst-FRAME SSIM lets a "blur frame 0's background and reuse it forever" candidate score 47x while leaving a visible person-shaped ghost: it damages one region, and the frame average hides it. Grading the worst 16x16 REGION catches it (0.74 vs >0.97 for legitimate approximations). test_gaming.py encodes the cheats as tests so they must lose. - Timing-as-fitness needs care. Measuring the baseline once and caching it lets a transiently loaded machine inflate every speedup that follows (this reported 82x for a program that is really 62x). The evaluator now interleaves baseline and candidate in the same call, warms up, and takes best-of-N. Uses the cascade (cheap smoke -> quality gate -> benchmark) so timing is only spent on candidates that are already correct, artifacts to tell the model WHY it was rejected, and a (complexity, ssim) MAP-Elites grid to keep exact-and-fast and approximate-and-faster strategies both alive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ence#472); add background_blur example; bump 0.3.1 Provider propagation fix: - LLMConfig pushes shared settings down to each LLMModelConfig via a `shared_config` dict, but that dict omitted `provider`. Every model therefore kept provider=None, and LLMEnsemble._create_model (which routes on `provider`) sent them all to the OpenAI backend regardless of the config. A config with `provider: claude_code` would crash with a missing-OPENAI_API_KEY error, making the examples/claude_code_quickstart example unusable. - Add `provider` to both shared_config dicts (__post_init__ and rebuild_models). Propagation uses overwrite=False, so an explicit per-model provider still wins. - Add tests/test_provider_propagation.py: covers propagation to evolution and evaluator models, per-model override precedence, the unchanged None default, survival across rebuild_models(), and end-to-end that LLMEnsemble now actually constructs ClaudeCodeLLM. All four propagation assertions fail without the fix. Independently reimplemented; supersedes algorithmicsuperintelligence#472. New example - examples/background_blur: Evolving a hot function behind a hard quality gate. The score is a speedup, but fidelity is pass/fail, so a fast-but-wrong candidate scores zero. Deterministic and numpy-only (the video is synthesised from a fixed seed - no webcam, GPU or dataset). Demonstrates the cascade (cheap smoke -> quality gate -> benchmark, so timing is only spent on candidates that are already correct), artifacts that tell the model WHY it was rejected, and a (complexity, ssim) MAP-Elites grid. Two traps it documents, both found the hard way: - The obvious gate is exploitable. Grading mean and worst-FRAME SSIM lets a "blur frame 0's background and reuse it forever" candidate score 47x while leaving a visible person-shaped ghost - it damages one region and the frame average hides it. Grading the worst 16x16 REGION catches it. test_gaming.py encodes the cheats as tests so they must lose. - Timing-as-fitness needs care: caching a single baseline measurement lets a transiently loaded machine inflate every later speedup. Baseline and candidate are now measured interleaved, warmed up, best-of-N. Bump version to 0.3.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bundles three things for the 0.3.1 release.
1.
run_evolution()checkpoint + target_score (original PR)Plumbs
target_scoreandcheckpoint_paththrough therun_evolution()library API. Both already existed oncontroller.run()but were not exposed, so library users could not resume from a checkpoint or stop at a target score.2. Propagate
llm.providerto per-model configsLLMConfigpushes shared settings down to eachLLMModelConfigvia ashared_configdict — but that dict omittedprovider. Every model therefore keptprovider=None, andLLMEnsemble._create_model(which routes onprovider) sent them all to the OpenAI backend regardless of the config. A config withprovider: claude_codecrashed with a missing-OPENAI_API_KEYerror, makingexamples/claude_code_quickstartunusable.Fixed by adding
providerto bothshared_configdicts (__post_init__andrebuild_models). Propagation goes throughupdate_model_params(..., overwrite=False), so an explicit per-modelproviderstill takes precedence.New
tests/test_provider_propagation.pycovers propagation to evolution and evaluator models, per-model override precedence, the unchangedNonedefault, survival acrossrebuild_models(), and end-to-end thatLLMEnsemblenow actually constructsClaudeCodeLLM. The propagation assertions fail without the fix.Supersedes #472 (independently reimplemented).
3. New example:
examples/background_blurEvolving a hot function behind a hard quality gate — the score is a speedup, but fidelity is pass/fail, so a fast-but-wrong candidate scores zero. Deterministic and numpy-only (the video sequence is synthesised from a fixed seed: no webcam, GPU, model or dataset).
Demonstrates the cascade (cheap smoke → quality gate → benchmark, so timing is only spent on candidates that are already correct), artifacts that tell the model why it was rejected, and a
(complexity, ssim)MAP-Elites grid.It also documents two traps, both found the hard way:
test_gaming.pyencodes the cheats as tests so they must lose.Bumps version to 0.3.1.