feat(evals): passive/interactive agent eval framework over memory2

[spomichter]

Problem

Eval framework, supports both InteractiveEvals and PassiveEvals, @paul-nechifor old dimsim spatial memory eval replicated via the new framework here pretty clean:

BED = Vector3(-3.567, -1.332, 0.0)

go_to_bed = InteractiveEval(
    id="go_to_bed",
    inputs="go to the bed",
    score=lambda s: ramp((BED - s.streams.odom.last().data.position).length(), band=2.0),
    aggregate=final,          # or floor ("never left the zone") / mean / auc later
    blueprint="unitree-go2-agentic go2-memory",
    simulator="dimsim",
    scene="apartment",
)

Uses langchain openevals for VQA formatting so its compatible with all public benchmarks we'd want to run. Scoring is generic and done via lambda one-liners. Agent loop runs LLM queries with a light agent_loop in the EvalRunner -- kept minimal and using langchain client since just doing dumb QA.

Closes DIM-1392, DIM-1390

Solution

• new dimos/evals/ package. one word (evals) everywhere, no benchmark vs eval split
• two case types: PassiveEval (frozen mem2 recording, any replay window, non-autoregressive) and InteractiveEval (live sim/robot, actions mutate state, scored by sampling teh live recorder store)
• memory2 the source of truth for all eval input — context selectors return real Streams, no copies, no parallel data structres
• scoring is jsut lambdas over typed msg arithmetic (within, ramp, Vector3 subtraction etc), graded [0,1] credit not only pass/fail
• we use the langchain/openevals standard for vqa formatting (inputs / reference_outputs, evaluators wrapped not subclassed) so external benchmarks map on natively and we can test against them
• generic typing: PassiveEval[T] ties expected/parse/score togehter so mypy catches a mismatched case at suite definition time, not mid-run
• runnable via cli (dimos evals run ...), mcp (EvalModule skills so coding agents can iterate), and pytest
• --blind ablation built in — same suite with observations withheld, alredy caught two guessable MCQs
• MoondreamChat adapter so evals run keyless on any gpu box

defining evals

passive one-off over a replay — the whole thing is one literal:

from dimos.evals.scorers import first_number, within, yes_no, exact
from dimos.evals.types import PassiveEval, InteractiveEval, Suite

# numeric, graded: 1.0 exact, linear to 0 at band
distance = PassiveEval(
    id="distance_10min",
    inputs="How far have you traveled in the last 10 minutes, in meters?",
    expected=142.0,
    parse=first_number,
    score=within(15.0),
    context=(lambda s: s.streams.odom.range_time(0, 600),),   # real mem2 stream
    dataset="go2_hongkong_office",
)

# vqa over a 10-image window
person = PassiveEval(
    id="person_visible",
    inputs="Is a person visible in any of these images?",
    expected="yes",
    parse=yes_no,
    score=exact,
    context=(lambda s: s.streams.color_image.range_time(58, 61).limit(10),),
    dataset="go2_short",
)

interactive — the case names its environment, score is sampled from the live mem2 store every interval_s and reduced by aggregate:

from dimos.evals.scorers import final, ramp
from dimos.msgs.geometry_msgs.Vector3 import Vector3

BED = Vector3(-3.567, -1.332, 0.0)

go_to_bed = InteractiveEval(
    id="go_to_bed",
    inputs="go to the bed",
    score=lambda s: ramp((BED - s.streams.odom.last().data.position).length(), band=2.0),
    aggregate=final,          # or floor ("never left the zone") / mean / auc later
    blueprint="unitree-go2-agentic go2-memory",
    simulator="dimsim",
    scene="apartment",
)

running

from dimos.evals.runner import EvalRunner, summarize

results = EvalRunner().run(SUITE)                    # prod model config (gpt-5.6-luna)
results = EvalRunner(blind=True).run(SUITE)          # guessing ablation
results = EvalRunner(chat_model=MoondreamChat()).run(SUITE)  # keyless local
print(summarize(results))

or dimos evals run dimos.evals.suites.go2_smoke --blind. every run writes results.jsonl + summary.json + per-case transcripts to ~/.local/state/dimos/evals/run-*/.

first real numbers

luna sighted: examples 1.00 / smoke 1.00 / vqa 0.86, blind = refusals. interactive rig scored 0.856 on a scripted go-to-bed in dimsim. the full agentic go-to-bed scores 0.0 right now — unitree-go2-agentic publishes no /odom in dimsim on current main (upstream test_dimsim_spatial_memory fails teh same way, suspect the control coordinator refactor). separate ticket coming.

Breaking Changes

None

How to Test

uv run pytest dimos/evals dimos/codebase_checks -q          # offline unit + wiring tests, no keys
uv run pytest dimos/evals/test_smoke.py -m self_hosted      # live model smoke (needs OPENAI_API_KEY + lfs data)
dimos evals run dimos.evals.suites.examples                 # 2 doc cases against go2_short
dimos evals run dimos.evals.suites.go2_smoke --blind        # guessing ablation

interactive (needs deno + display): dimos evals run dimos.evals.suites.dimsim_house --live-db recording_go2.db

Contributor License Agreement

• I have read and approved the CLA

[codecov]

Codecov Report

❌ Patch coverage is 80.95238% with 132 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/evals/runner.py	68.26%	65 Missing and 8 partials ⚠️
dimos/evals/suites/dimsim_house.py	40.00%	21 Missing ⚠️
dimos/evals/cli.py	33.33%	16 Missing ⚠️
dimos/evals/scorers.py	78.94%	8 Missing ⚠️
dimos/evals/types.py	91.76%	5 Missing and 2 partials ⚠️
dimos/evals/test_evals.py	97.12%	4 Missing ⚠️
dimos/evals/test_mem2_wiring.py	97.19%	1 Missing and 2 partials ⚠️

@@            Coverage Diff             @@
##             main    #3411      +/-   ##
==========================================
+ Coverage   76.09%   76.12%   +0.03%     
==========================================
  Files        1190     1201      +11     
  Lines      115295   115988     +693     
  Branches    10367    10415      +48     
==========================================
+ Hits        87729    88294     +565     
- Misses      24554    24671     +117     
- Partials     3012     3023      +11     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	70.35% <78.93%> (+0.19%)	⬆️
OS-ubuntu-latest	72.20% <78.93%> (+0.04%)	⬆️
Py-3.10	72.19% <78.93%> (+0.05%)	⬆️
Py-3.11	72.19% <78.93%> (+0.04%)	⬆️
Py-3.12	72.19% <78.93%> (+0.04%)	⬆️
Py-3.13	72.19% <78.93%> (+0.04%)	⬆️
Py-3.14	72.20% <78.93%> (+0.04%)	⬆️
Py-3.14t	72.19% <78.93%> (+0.04%)	⬆️
SelfHosted-Large	29.68% <34.11%> (+0.02%)	⬆️
SelfHosted-Linux	35.92% <48.52%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/cli/dimos.py	64.62% <100.00%> (+0.15%)	⬆️
dimos/evals/suites/examples.py	100.00% <100.00%> (ø)
dimos/evals/suites/go2_smoke.py	100.00% <100.00%> (ø)
dimos/evals/suites/go2_vqa.py	100.00% <100.00%> (ø)
dimos/evals/test_smoke.py	100.00% <100.00%> (ø)
dimos/robot/all_blueprints.py	100.00% <ø> (ø)
dimos/evals/test_mem2_wiring.py	97.19% <97.19%> (ø)
dimos/evals/test_evals.py	97.12% <97.12%> (ø)
dimos/evals/types.py	91.76% <91.76%> (ø)
dimos/evals/scorers.py	78.94% <78.94%> (ø)
... and 3 more

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[spomichter]

@greptile review

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[TomCC7]

We need to design runtime and scorer information access more carefully. The current scheme is wrong IMO because it evaluates against Memory2 state produced by the runtime itself. To simulate real operation as faithfully as possible, the agent should have access to all information exposed by the running blueprint, including RPCs, streams, and Memory2. Scoring should instead use independent, privileged simulator state, such as ground-truth robot and object poses and object types.