Skip to content

run_evolution checkpoint/target_score params; propagate llm.provider (#472); background_blur example#457

Merged
codelion merged 4 commits into
algorithmicsuperintelligence:mainfrom
PetrichorHlacyon:run_evolution_with_checkpoint
Jul 14, 2026
Merged

run_evolution checkpoint/target_score params; propagate llm.provider (#472); background_blur example#457
codelion merged 4 commits into
algorithmicsuperintelligence:mainfrom
PetrichorHlacyon:run_evolution_with_checkpoint

Conversation

@PetrichorHlacyon

@PetrichorHlacyon PetrichorHlacyon commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Bundles three things for the 0.3.1 release.

1. run_evolution() checkpoint + target_score (original PR)

Plumbs target_score and checkpoint_path through the run_evolution() library API. Both already existed on controller.run() but were not exposed, so library users could not resume from a checkpoint or stop at a target score.

2. Propagate llm.provider to per-model configs

LLMConfig pushes shared settings down to each LLMModelConfig via a shared_config dict — but that dict omitted provider. Every model therefore kept provider=None, and LLMEnsemble._create_model (which routes on provider) sent them all to the OpenAI backend regardless of the config. A config with provider: claude_code crashed with a missing-OPENAI_API_KEY error, making examples/claude_code_quickstart unusable.

>>> c = Config.from_dict({"llm": {"provider": "claude_code", "models": [{"name": "a"}]}})
>>> c.llm.provider
'claude_code'
>>> [m.provider for m in c.llm.models]
[None]          # <-- never propagated

Fixed by adding provider to both shared_config dicts (__post_init__ and rebuild_models). Propagation goes through update_model_params(..., overwrite=False), so an explicit per-model provider still takes precedence.

New tests/test_provider_propagation.py covers propagation to evolution and evaluator models, per-model override precedence, the unchanged None default, survival across rebuild_models(), and end-to-end that LLMEnsemble now actually constructs ClaudeCodeLLM. The propagation assertions fail without the fix.

Supersedes #472 (independently reimplemented).

3. New example: examples/background_blur

Evolving a hot function behind a hard quality gate — the score is a speedup, but fidelity is pass/fail, so a fast-but-wrong candidate scores zero. Deterministic and numpy-only (the video sequence is synthesised from a fixed seed: no webcam, GPU, model or dataset).

Demonstrates the cascade (cheap smoke → quality gate → benchmark, so timing is only spent on candidates that are already correct), artifacts that tell the model why it was rejected, and a (complexity, ssim) MAP-Elites grid.

It also documents two traps, both found the hard way:

  • The obvious quality gate is exploitable. Grading mean SSIM and worst-frame SSIM lets a "blur frame 0's background and reuse it forever" candidate score 47x while leaving a visible person-shaped ghost — it damages one region, and the frame average hides it. Grading the worst 16×16 region catches it (0.74 vs >0.97 for legitimate approximations). test_gaming.py encodes the cheats as tests so they must lose.
  • Timing-as-fitness needs care. Caching a single baseline measurement lets a transiently loaded machine inflate every later speedup (it reported 82x for a program that is really 62x). Baseline and candidate are now measured interleaved, warmed up, best-of-N.

Bumps version to 0.3.1.

@CLAassistant

CLAassistant commented Apr 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

codelion and others added 3 commits July 14, 2026 08:02
…ality gate

A self-contained example of the "optimise a real hot function" pattern, where the
score is a speedup but fidelity is a hard gate: a fast-but-wrong candidate scores
zero. Deterministic and dependency-light (numpy only) - the video sequence is
synthesised from a fixed seed, so there is no webcam, GPU, model or dataset.

The seed is a correct-but-slow O(k^2) 2D Gaussian convolution. Over 100 iterations
the search reached 62x over that seed (2.6x over a hand-written separable+fp32
implementation), stacking separable convolution, a half-resolution blur pipeline,
float32, batching across the sequence, and a lerp composite. 28 of 100 candidates
were rejected outright by the quality gate.

Two things this example exists to demonstrate:

- The obvious quality gate is exploitable. Grading mean SSIM and worst-FRAME SSIM
  lets a "blur frame 0's background and reuse it forever" candidate score 47x while
  leaving a visible person-shaped ghost: it damages one region, and the frame
  average hides it. Grading the worst 16x16 REGION catches it (0.74 vs >0.97 for
  legitimate approximations). test_gaming.py encodes the cheats as tests so they
  must lose.

- Timing-as-fitness needs care. Measuring the baseline once and caching it lets a
  transiently loaded machine inflate every speedup that follows (this reported 82x
  for a program that is really 62x). The evaluator now interleaves baseline and
  candidate in the same call, warms up, and takes best-of-N.

Uses the cascade (cheap smoke -> quality gate -> benchmark) so timing is only spent
on candidates that are already correct, artifacts to tell the model WHY it was
rejected, and a (complexity, ssim) MAP-Elites grid to keep exact-and-fast and
approximate-and-faster strategies both alive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ence#472); add background_blur example; bump 0.3.1

Provider propagation fix:
- LLMConfig pushes shared settings down to each LLMModelConfig via a `shared_config`
  dict, but that dict omitted `provider`. Every model therefore kept provider=None,
  and LLMEnsemble._create_model (which routes on `provider`) sent them all to the
  OpenAI backend regardless of the config. A config with `provider: claude_code`
  would crash with a missing-OPENAI_API_KEY error, making the
  examples/claude_code_quickstart example unusable.
- Add `provider` to both shared_config dicts (__post_init__ and rebuild_models).
  Propagation uses overwrite=False, so an explicit per-model provider still wins.
- Add tests/test_provider_propagation.py: covers propagation to evolution and
  evaluator models, per-model override precedence, the unchanged None default,
  survival across rebuild_models(), and end-to-end that LLMEnsemble now actually
  constructs ClaudeCodeLLM. All four propagation assertions fail without the fix.

Independently reimplemented; supersedes algorithmicsuperintelligence#472.

New example - examples/background_blur:
  Evolving a hot function behind a hard quality gate. The score is a speedup, but
  fidelity is pass/fail, so a fast-but-wrong candidate scores zero. Deterministic and
  numpy-only (the video is synthesised from a fixed seed - no webcam, GPU or dataset).
  Demonstrates the cascade (cheap smoke -> quality gate -> benchmark, so timing is
  only spent on candidates that are already correct), artifacts that tell the model
  WHY it was rejected, and a (complexity, ssim) MAP-Elites grid.

  Two traps it documents, both found the hard way:
  - The obvious gate is exploitable. Grading mean and worst-FRAME SSIM lets a
    "blur frame 0's background and reuse it forever" candidate score 47x while
    leaving a visible person-shaped ghost - it damages one region and the frame
    average hides it. Grading the worst 16x16 REGION catches it. test_gaming.py
    encodes the cheats as tests so they must lose.
  - Timing-as-fitness needs care: caching a single baseline measurement lets a
    transiently loaded machine inflate every later speedup. Baseline and candidate
    are now measured interleaved, warmed up, best-of-N.

Bump version to 0.3.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codelion codelion changed the title run_evolution function with checkpoint parameter run_evolution checkpoint/target_score params; propagate llm.provider (#472); background_blur example Jul 14, 2026
@codelion
codelion merged commit 8bd33ad into algorithmicsuperintelligence:main Jul 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants