feat(evals): semantic object VQA + ImageDetections3DPC.agent_encode (evo autoresearch, 0.355 -> 0.9997)

[spomichter]

Problem

after the pointcloud encoding work (#3415) the agent could read geometry but had no object memory: detections lived in memory2 as per-frame rich tables the model can't aggregate (baseline: answered "3" for 6 chairs, "surfboard" for what's near the bed, "refrigerator" for nearest-anything).

Solution

stacked on #3411, sibling to #3415. two commits:

1. semantic object VQA benchmark (evals_bench/semantic/): 47 generated cases over a YOLO(v8m)+odom-grounded object memory from go2_bigoffice. 12 families: recall (set-F1), counting, zone counting, presence (prior-inverting probes), nearest-class, egocentric ahead/behind/left/right (odom yaw frame transform), 8-way object-object compass, VQASynth-style metric distances (object-object + robot-object), within-radius, next-to, betweenness. ground truth computed by an independent clusterer over frozen detections; teacher/student context parity contract documented in the harness. gates: blind-ablation ceiling ≤0.35 (measured 0.06 — two calibration rounds drove it down from 0.39), answer-leak grep, target-only diff, and an abstraction budget gate (max one agent_encode + max one memory2 search skill; wrapper stores = fail).

2. ImageDetections3DPC.agent_encode() — found by evo autoresearch, 14 experiments in two lanes. final: 0.9997 (all 47 pass; stability replicates 0.953/0.9998, mean 0.984; blind 0.06). design: compact per-frame name#k conf pos=(x,y,z) entries where the ENCODER pre-runs the greedy 1.5 m instance-clustering scan (models can't execute it mentally over ~290 rows), plus a once-per-pass P1–P9 procedure legend: noise deletion, per-class distance-table discipline (kills salience bias), closed-form ego-frame via fwd/left signed comparisons (no trig), the 2.41-ratio compass rule from #3415, strictly-between admission test.

report w/ figures: https://dimos-semantic-vqa-report.vercel.app (pointcloud sibling: https://dimos-pc-encoding-report.vercel.app)

Mergeability

• behaviour: shipped file byte-identical to winner exp_0013 (score carries exactly); benchmark rerun 3x.
• regression safety: unit tests pass; diff is purely additive (one new method + helper on the type, new benchmark dir, one memory2 materializer + test).
• scope: 10 files, +4972/−1 (mostly frozen benchmark data); nothing outside the benchmark + one perception type touched.
• tests: none modified/weakened/removed at any point (git-gate enforced). eval suite covers the new behaviour; a unit test on agent_encode output shape is a fair reviewer ask.
• known limits (documented in-repo): legend clustering constants mirror the truth generator's convention (defensible as task definition; hongkong holdout is the open check); object positions are odom-grounded (~1-2 m) pending the depth-projection pipeline on a depth-carrying recording; one nearest row is ~1/3 sampling-flaky.

Breaking Changes

None

How to Test

uv run pytest dimos/memory2/test_objects.py -q
env PYTHONPATH=. uv run python evals_bench/semantic/benchmark.py            # needs OPENAI_API_KEY
env PYTHONPATH=. uv run python evals_bench/semantic/benchmark.py --blind --max-mean 0.35

Contributor License Agreement

• I have read and approved the CLA

[codecov]

Codecov Report

❌ Patch coverage is 66.17647% with 23 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
.../detection/type/detection3d/imageDetections3DPC.py	20.68%	23 Missing ⚠️

@@                   Coverage Diff                    @@
##           feat/evals-framework    #3417      +/-   ##
========================================================
- Coverage                 76.12%   76.11%   -0.01%     
========================================================
  Files                      1201     1203       +2     
  Lines                    115988   116055      +67     
  Branches                  10415    10422       +7     
========================================================
+ Hits                      88294    88338      +44     
- Misses                    24671    24694      +23     
  Partials                   3023     3023              

Flag	Coverage Δ
OS-ubuntu-24.04-arm	70.36% <66.17%> (+<0.01%)	⬆️
OS-ubuntu-latest	72.20% <66.17%> (+<0.01%)	⬆️
Py-3.10	72.20% <66.17%> (+<0.01%)	⬆️
Py-3.11	72.19% <66.17%> (+<0.01%)	⬆️
Py-3.12	72.19% <66.17%> (+<0.01%)	⬆️
Py-3.13	72.19% <66.17%> (+<0.01%)	⬆️
Py-3.14	72.20% <66.17%> (+<0.01%)	⬆️
Py-3.14t	72.19% <66.17%> (+<0.01%)	⬆️
SelfHosted-Large	29.69% <30.88%> (+<0.01%)	⬆️
SelfHosted-Linux	35.91% <30.88%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/memory2/objects.py	100.00% <100.00%> (ø)
dimos/memory2/test_objects.py	100.00% <100.00%> (ø)
.../detection/type/detection3d/imageDetections3DPC.py	34.14% <20.68%> (-35.09%)	⬇️

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.