Skip to content

Fix: propagate top-level llm.provider to per-model configs#472

Closed
froucoux wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
froucoux:fix/llm-provider-propagation
Closed

Fix: propagate top-level llm.provider to per-model configs#472
froucoux wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
froucoux:fix/llm-provider-propagation

Conversation

@froucoux

Copy link
Copy Markdown

Summary

A top-level llm.provider (e.g. claude_code) set in a config was silently dropped and never reached the individual models, so the ensemble always fell back to the OpenAI backend. This makes the examples/claude_code_quickstart example crash immediately with:

openai.OpenAIError: Missing credentials. Please pass an `api_key` ... or set the `OPENAI_API_KEY` ... environment variable.

even though config.yaml correctly sets provider: "claude_code".

Root cause

LLMConfig propagates shared settings down to each per-model LLMModelConfig via a shared_config dict (in both __post_init__ and rebuild_models). That dict included api_base, api_key, temperature, etc. — but not provider. So each model kept provider=None, and LLMEnsemble._create_model routed it to OpenAILLM instead of ClaudeCodeLLM.

>>> c = Config.from_yaml("examples/claude_code_quickstart/config.yaml")
>>> c.llm.provider
'claude_code'
>>> [m.provider for m in c.llm.models]
[None, None]          # <-- provider never propagated

Fix

Add "provider": self.provider to both shared_config dicts. Propagation goes through update_model_params(..., overwrite=False), so an explicit per-model provider still takes precedence over the top-level default.

Tests

Added TestProviderPropagation to tests/test_claude_code_llm.py:

  • top-level provider reaches every evolution and evaluator model,
  • an explicit per-model provider overrides the top-level one,
  • the default provider stays None (unchanged OpenAI-default behavior),
  • LLMEnsemble built from such a config yields ClaudeCodeLLM instances end to end.

The three propagation tests fail on main (None != 'claude_code') and pass with this change. Full suite: 417 passed (unchanged), plus the new cases.

How it was found

Running the documented examples/claude_code_quickstart example with the Claude Code CLI backend on Python 3.14. After the fix, a full 50-iteration run completed successfully, evolving the seed random-search into a simulated-annealing optimizer (combined_score 1.2148 → 1.4995).

The `provider` field (e.g. "claude_code") set at the top `llm:` level was
omitted from the shared config that LLMConfig propagates down to each model,
so it never reached the individual LLMModelConfig entries. As a result the
ensemble ignored `provider: "claude_code"` and fell back to the OpenAI
backend, crashing with "Missing credentials ... OPENAI_API_KEY" — which
made the examples/claude_code_quickstart example unusable out of the box.

Add `provider` to the shared_config dicts in both LLMConfig.__post_init__
and rebuild_models. Propagation uses update_model_params (overwrite=False),
so an explicit per-model `provider` still takes precedence.

Add regression tests asserting that a top-level provider reaches every
evolution and evaluator model, that per-model overrides win, that the
default remains None, and that the ensemble builds ClaudeCodeLLM end to end.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

codelion added a commit to PetrichorHlacyon/openevolve that referenced this pull request Jul 14, 2026
…ence#472); add background_blur example; bump 0.3.1

Provider propagation fix:
- LLMConfig pushes shared settings down to each LLMModelConfig via a `shared_config`
  dict, but that dict omitted `provider`. Every model therefore kept provider=None,
  and LLMEnsemble._create_model (which routes on `provider`) sent them all to the
  OpenAI backend regardless of the config. A config with `provider: claude_code`
  would crash with a missing-OPENAI_API_KEY error, making the
  examples/claude_code_quickstart example unusable.
- Add `provider` to both shared_config dicts (__post_init__ and rebuild_models).
  Propagation uses overwrite=False, so an explicit per-model provider still wins.
- Add tests/test_provider_propagation.py: covers propagation to evolution and
  evaluator models, per-model override precedence, the unchanged None default,
  survival across rebuild_models(), and end-to-end that LLMEnsemble now actually
  constructs ClaudeCodeLLM. All four propagation assertions fail without the fix.

Independently reimplemented; supersedes algorithmicsuperintelligence#472.

New example - examples/background_blur:
  Evolving a hot function behind a hard quality gate. The score is a speedup, but
  fidelity is pass/fail, so a fast-but-wrong candidate scores zero. Deterministic and
  numpy-only (the video is synthesised from a fixed seed - no webcam, GPU or dataset).
  Demonstrates the cascade (cheap smoke -> quality gate -> benchmark, so timing is
  only spent on candidates that are already correct), artifacts that tell the model
  WHY it was rejected, and a (complexity, ssim) MAP-Elites grid.

  Two traps it documents, both found the hard way:
  - The obvious gate is exploitable. Grading mean and worst-FRAME SSIM lets a
    "blur frame 0's background and reuse it forever" candidate score 47x while
    leaving a visible person-shaped ghost - it damages one region and the frame
    average hides it. Grading the worst 16x16 REGION catches it. test_gaming.py
    encodes the cheats as tests so they must lose.
  - Timing-as-fitness needs care: caching a single baseline measurement lets a
    transiently loaded machine inflate every later speedup. Baseline and candidate
    are now measured interleaved, warmed up, best-of-N.

Bump version to 0.3.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
codelion pushed a commit that referenced this pull request Jul 14, 2026
…round_blur example (#457)

Plumbs target_score/checkpoint_path through run_evolution(); fixes llm.provider not propagating to per-model configs (supersedes #472); adds the background_blur example (hot function behind a hard quality gate). Bumps version to 0.3.1.
@codelion

Copy link
Copy Markdown
Member

Thanks for this — the diagnosis was spot on, and the reproduction made it easy to confirm:

>>> c = Config.from_dict({"llm": {"provider": "claude_code", "models": [{"name": "a"}]}})
>>> [m.provider for m in c.llm.models]
[None]          # provider never propagated

Since the CLA is unsigned we could not merge this branch directly, so the fix was reimplemented independently and has shipped in v0.3.1 via #457 (merged as 8bd33ad):

  • provider added to both shared_config dicts (__post_init__ and rebuild_models), so a top-level llm.provider now reaches every evolution and evaluator model. Propagation uses overwrite=False, so an explicit per-model provider still wins.
  • New tests/test_provider_propagation.py covering propagation to both model lists, per-model override precedence, the unchanged None default, survival across rebuild_models(), and end-to-end that LLMEnsemble now actually constructs ClaudeCodeLLM rather than silently falling back to OpenAILLM.

Closing as superseded — but the credit for finding and diagnosing this is yours. If you do sign the CLA at some point, we would be glad to take future contributions directly.

@codelion codelion closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants