feat(evals): add code-as-policy evaluation runtime

[TomCC7]

Contribution path

• Small, safe change that does not need a tracking issue
• Stacked on feat(evals): passive/interactive agent eval framework over memory2 #3411

Problem

Interactive robot evaluations need to resemble real operation: a policy should see the complete running DimOS blueprint, while benchmark-owned privileged simulator state remains isolated in the scorer. Agent generation cannot sit inside the realtime task horizon.

Solution

Add a code-policy evaluation layer alongside #3411's existing passive and interactive runtimes:

• explore with a fixed Pi agent using gpt-5.6-luna, medium reasoning, and one persistent Python REPL tool
• accept a strictly typed policy callable with the signature def policy(app: Dimos) -> None
• let Pi inspect up to five complete fresh debug trials through stopped TrialRun snapshots containing outcomes, errors, logs, read-only Memory2, and artifacts
• freeze the last accepted callable as the task-level artifact
• execute held-out cases without an agent, in a clean process against a fresh policy-only DimOS blueprint
• keep native environment lifecycle and privileged scoring entirely evaluation-owned
• discover third-party evaluations through the dimos.evaluations plugin entry point and expose them through dimos eval run

This PR adds only the generic runtime and plugin interface; a concrete LIBERO adapter remains separate.

How to Test

./bin/pytest-fast -q
uv run --group lint mypy dimos/agents/code_policy_core.py dimos/agents/code_policy_server.py dimos/benchmark/evaluation dimos/cli/eval.py
npm test --prefix packages/pi-code-policy-extension

Results: 3,997 passed, 31 expected skips. Focused evaluation and Memory2 suite: 139 passed.

AI assistance

Codex with GPT-5.6 assisted with architecture exploration, implementation, tests, documentation, and PR drafting. The design and implementation were reviewed interactively by the author.

Checklist

• I have read and approved the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 75.55402% with 353 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/agents/code_policy_core.py	56.48%	80 Missing and 14 partials ⚠️
dimos/cli/eval.py	19.35%	75 Missing ⚠️
dimos/benchmark/evaluation/runtime.py	69.14%	53 Missing and 5 partials ⚠️
dimos/benchmark/evaluation/pi_process.py	77.50%	18 Missing and 18 partials ⚠️
dimos/benchmark/evaluation/registry.py	68.42%	19 Missing and 5 partials ⚠️
dimos/agents/code_policy_server.py	80.80%	12 Missing and 7 partials ⚠️
dimos/benchmark/evaluation/runner.py	73.77%	13 Missing and 3 partials ⚠️
dimos/memory2/store/sqlite.py	53.84%	5 Missing and 7 partials ⚠️
dimos/benchmark/evaluation/models.py	92.30%	3 Missing and 3 partials ⚠️
dimos/benchmark/evaluation/protocol.py	92.75%	3 Missing and 2 partials ⚠️
... and 5 more

@@                   Coverage Diff                    @@
##           feat/evals-framework    #3434      +/-   ##
========================================================
- Coverage                 76.12%   76.09%   -0.03%     
========================================================
  Files                      1201     1218      +17     
  Lines                    115988   117470    +1482     
  Branches                  10415    10555     +140     
========================================================
+ Hits                      88294    89388    +1094     
- Misses                    24671    24995     +324     
- Partials                   3023     3087      +64     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	70.40% <75.34%> (+0.04%)	⬆️
OS-ubuntu-latest	72.22% <75.55%> (+0.02%)	⬆️
Py-3.10	72.21% <75.55%> (+0.01%)	⬆️
Py-3.11	72.21% <75.55%> (+0.01%)	⬆️
Py-3.12	72.21% <75.55%> (+0.01%)	⬆️
Py-3.13	72.21% <75.55%> (+0.02%)	⬆️
Py-3.14	72.21% <75.55%> (+0.01%)	⬆️
Py-3.14t	72.21% <75.55%> (+0.01%)	⬆️
SelfHosted-Large	29.73% <32.13%> (+0.04%)	⬆️
SelfHosted-Linux	35.87% <33.24%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/benchmark/evaluation/test_framework.py	100.00% <100.00%> (ø)
dimos/benchmark/evaluation/test_models.py	100.00% <100.00%> (ø)
dimos/benchmark/evaluation/test_pi_process.py	100.00% <100.00%> (ø)
dimos/benchmark/evaluation/test_policy_runtime.py	100.00% <100.00%> (ø)
dimos/cli/dimos.py	65.21% <100.00%> (+0.58%)	⬆️
dimos/memory2/blobstore/sqlite.py	96.77% <100.00%> (+0.05%)	⬆️
dimos/memory2/utils/sqlite.py	100.00% <100.00%> (ø)
dimos/memory2/vectorstore/sqlite.py	83.05% <100.00%> (+0.29%)	⬆️
dimos/agents/test_code_policy_core.py	97.22% <97.22%> (ø)
dimos/benchmark/evaluation/test_registry.py	97.67% <97.67%> (ø)
... and 13 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.