Fix: propagate top-level llm.provider to per-model configs#472
Closed
froucoux wants to merge 1 commit into
Closed
Conversation
The `provider` field (e.g. "claude_code") set at the top `llm:` level was omitted from the shared config that LLMConfig propagates down to each model, so it never reached the individual LLMModelConfig entries. As a result the ensemble ignored `provider: "claude_code"` and fell back to the OpenAI backend, crashing with "Missing credentials ... OPENAI_API_KEY" — which made the examples/claude_code_quickstart example unusable out of the box. Add `provider` to the shared_config dicts in both LLMConfig.__post_init__ and rebuild_models. Propagation uses update_model_params (overwrite=False), so an explicit per-model `provider` still takes precedence. Add regression tests asserting that a top-level provider reaches every evolution and evaluator model, that per-model overrides win, that the default remains None, and that the ensemble builds ClaudeCodeLLM end to end.
|
|
codelion
added a commit
to PetrichorHlacyon/openevolve
that referenced
this pull request
Jul 14, 2026
…ence#472); add background_blur example; bump 0.3.1 Provider propagation fix: - LLMConfig pushes shared settings down to each LLMModelConfig via a `shared_config` dict, but that dict omitted `provider`. Every model therefore kept provider=None, and LLMEnsemble._create_model (which routes on `provider`) sent them all to the OpenAI backend regardless of the config. A config with `provider: claude_code` would crash with a missing-OPENAI_API_KEY error, making the examples/claude_code_quickstart example unusable. - Add `provider` to both shared_config dicts (__post_init__ and rebuild_models). Propagation uses overwrite=False, so an explicit per-model provider still wins. - Add tests/test_provider_propagation.py: covers propagation to evolution and evaluator models, per-model override precedence, the unchanged None default, survival across rebuild_models(), and end-to-end that LLMEnsemble now actually constructs ClaudeCodeLLM. All four propagation assertions fail without the fix. Independently reimplemented; supersedes algorithmicsuperintelligence#472. New example - examples/background_blur: Evolving a hot function behind a hard quality gate. The score is a speedup, but fidelity is pass/fail, so a fast-but-wrong candidate scores zero. Deterministic and numpy-only (the video is synthesised from a fixed seed - no webcam, GPU or dataset). Demonstrates the cascade (cheap smoke -> quality gate -> benchmark, so timing is only spent on candidates that are already correct), artifacts that tell the model WHY it was rejected, and a (complexity, ssim) MAP-Elites grid. Two traps it documents, both found the hard way: - The obvious gate is exploitable. Grading mean and worst-FRAME SSIM lets a "blur frame 0's background and reuse it forever" candidate score 47x while leaving a visible person-shaped ghost - it damages one region and the frame average hides it. Grading the worst 16x16 REGION catches it. test_gaming.py encodes the cheats as tests so they must lose. - Timing-as-fitness needs care: caching a single baseline measurement lets a transiently loaded machine inflate every later speedup. Baseline and candidate are now measured interleaved, warmed up, best-of-N. Bump version to 0.3.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
|
Thanks for this — the diagnosis was spot on, and the reproduction made it easy to confirm: >>> c = Config.from_dict({"llm": {"provider": "claude_code", "models": [{"name": "a"}]}})
>>> [m.provider for m in c.llm.models]
[None] # provider never propagatedSince the CLA is unsigned we could not merge this branch directly, so the fix was reimplemented independently and has shipped in v0.3.1 via #457 (merged as 8bd33ad):
Closing as superseded — but the credit for finding and diagnosing this is yours. If you do sign the CLA at some point, we would be glad to take future contributions directly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A top-level
llm.provider(e.g.claude_code) set in a config was silently dropped and never reached the individual models, so the ensemble always fell back to the OpenAI backend. This makes theexamples/claude_code_quickstartexample crash immediately with:even though
config.yamlcorrectly setsprovider: "claude_code".Root cause
LLMConfigpropagates shared settings down to each per-modelLLMModelConfigvia ashared_configdict (in both__post_init__andrebuild_models). That dict includedapi_base,api_key,temperature, etc. — but notprovider. So each model keptprovider=None, andLLMEnsemble._create_modelrouted it toOpenAILLMinstead ofClaudeCodeLLM.Fix
Add
"provider": self.providerto bothshared_configdicts. Propagation goes throughupdate_model_params(..., overwrite=False), so an explicit per-modelproviderstill takes precedence over the top-level default.Tests
Added
TestProviderPropagationtotests/test_claude_code_llm.py:None(unchanged OpenAI-default behavior),LLMEnsemblebuilt from such a config yieldsClaudeCodeLLMinstances end to end.The three propagation tests fail on
main(None != 'claude_code') and pass with this change. Full suite:417 passed(unchanged), plus the new cases.How it was found
Running the documented
examples/claude_code_quickstartexample with the Claude Code CLI backend on Python 3.14. After the fix, a full 50-iteration run completed successfully, evolving the seed random-search into a simulated-annealing optimizer (combined_score1.2148 → 1.4995).