feat(memory2): Space raster backend + experimental memory2 agent

[Mgczacki]

An approach for #1913

What does it do?

Two things in one PR:

1. memory2 Space gets a cv2 raster backend. Space.to_bgr() / Space.to_png() produce the same world view a vision LLM gets, alongside the existing SVG and Rerun renderers. Plus the elements you need to draw on top of an occupancy map: Polygon, Wedge, RasterOverlay, and Point.shape / halo variants.

2. An experimental LangChain agent at dimos/memory2/experimental/memory2_agent/. Given a memory2 database, it introspects all streams to perform modality fusion (when the tools are compatible with the modality) in order to generate rich/useful temporal-spatial representations, domain-structure validation, and measurement capabilities on spatial data.

How do we achieve this?

(rendering)

1. Raster backend mirrors the SVG backend: walks space.elements, accumulates a world-frame Bounds, paints onto a BGR ndarray. The Bounds class is lifted into dimos/memory2/vis/space/bounds.py so both backends share it.
2. Three new element types — Polygon, Wedge, RasterOverlay — cover what the agent draws on top of occupancy maps: room boundaries, camera FOV cones, arbitrary world-frame masks (overclaim highlights, heatmaps, etc.).
3. Point grew shape ("dot" / "cross" / "x" / "square") and halo (black underlay), so markers stay readable on busy maps.
4. resolve_deferred now walks color, fill, AND stroke, so cmap-based colors work on Polygon's two-color split.
5. PointCloud2 only gets inflated once per render — the bounds pass and the draw pass share a per-call cache keyed by id(el).

(agent)

1. Provide tools to reason over memory2 types (listing streams, summarizing, getting recent observations).
2. Rendering of occupancy maps, and of placing points on it to have visual reference for coordinates.
3. Temporal indexing. As it can be a consistent method of joining between modalities, regardless of sampling rate and measurement error.
4. Searching for entities - Finding (through CLIP) the entities that exist in an embedded stream, and when they appear.
5. Searching for coordinates - Finding (through raytracing) whether a coordinate (2D) has been observed in any of the images the system captured.
6. Walkthrough tools: Allows the system to follow a heavily downsampled video stream's frames, to understand frame by frame what is happening.
7. Calculations - uses a minimized python REPL for mathematical operations.
8. Sizing rooms in the occupancy maps by proposing bounding polygon points, then upsizing them/downsizing them based on visual evidence (images), as well as out-of-room lidar measurements, out-of-room odom measurements, and room intersections in order to iteratively refine the border between rooms.
9. Skills - Any hard algorithms that can be achieved by a composition of the previous skills. The skill set covers room reasoning — counting, sizing, dwell time, and cross-room comparisons — by segmenting the space into verified room polygons (thinking_about_rooms), describing a specific room (describe_room), finding exploration frontiers (unexplored_spaces), counting unique instances of a kind of thing across many frames (count_unique_things), distances, object positions, and reasoning from another entity's viewpoint.

Full tool/skill inventory in dimos/memory2/experimental/memory2_agent/README.md (15 tools, 7 skills).

Examples (what the agent sees)

The screenshots below are real tool returns from a memory2 agent run.

show_map — top-down lidar map with the robot's pose pinned. The agent uses this to orient itself in world coordinates.

frames_facing — top-down view with viewing cones. Given a world (x, y), the tool overlays the cones of the camera frames whose field of view could contain that point. Used for finding which recorded frames "saw" a target location.

verify_room_partition — map with the agent's room polygons + per-room areas. The agent submits candidate polygons; the tool overlays them and flags issues (overlap, unpartitioned floor blobs, odometry outside any room).

walkthrough — annotated frame strip across a time range. Each tile is captioned with t, robot (x, y), and yaw. Used for summarising what was visible across a stretch of the walk in one call.

frames_facing (per-frame) — recorded camera frame with the query point reprojected as a red X. Used to verify whether a candidate (x, y) actually lands on the target object: if the red cross sits on the body of the thing across multiple views, the position is right.

Worked example: routing on a map it built itself

Prompt: "Is it possible to go from the room with the greeting robot to the room with the acoustic guitar without passing by any elevators?"

With no prior map, the agent locates the greeting robot, the acoustic guitar, and the elevators (CLIP search → frames → poses), plots the three on the top-down lidar map, and reasons from the visible corridor layout plus a walkthrough of the route. Answer: no — the greeting-robot area joins the rest of the office only through the lower corridor, and the elevator lobby sits squarely on it. (On this run it answered straight from the occupancy map + landmark positions; the heavier thinking_about_rooms segmentation isn't needed for a chokepoint this clear. We independently confirmed it with a free-space path search that can't avoid the elevator lobby.) This is the greeting_to_guitar_avoids_elevators_no test.

Worked example: the nearest robot to the guitar

Prompt: "How far is the acoustic guitar from the closest robot to it?"

This needs more than a lookup: the agent has to find every robot, place each in world coordinates, measure its distance to the guitar, and take the minimum. It locates ~7 robots across the office and picks the nearest — ~10–12 m.

The two endpoints, as the agent saw them — the acoustic guitar (t≈313 s) and the nearest robot, a white service unit in the retail aisle (t≈376 s); both sit near x = -2, ~10 m apart along the same aisle:

Honesty note: object positions come from bearing triangulation, so they're estimates (±a couple metres), and "nearest" is a near-tie between three robots clustered ~10–12 m away. The matching test (acoustic_guitar_closest_robot_distance) accepts 9–13 m and is a non-strict xfail for exactly that reason — the capability is real, the precision is bounded.

Worked example: a room it could see but not reach

Prompt: "Were you able to look clearly into a room that you didn't walk into? If yes, where (x, y)?"

There is no semantic map of this place — everything the agent knows comes from the recorded ego-frames + lidar + odometry. On this run it walks the corridor past a cluster of sealed glass-walled meeting pods, recognises one as a room it can see into but never entered, and returns a world coordinate: 15.5, -2.5. The robot passes within ~1.6 m of it in the corridor, but the pods are sealed glass, so it never goes in.

The agent's answer (15.5, -2.5) reprojected into two recorded frames — the marker lands on the glass-walled meeting room the robot passed but didn't enter.

This prompt is deliberately under-determined — the office has several rooms glimpsed-through-glass-but-not-entered, and the agent names a different valid one across runs. So the matching end-to-end test, looked_into_glass_room_not_entered, is a non-strict xfail: it pins this answer as the ground truth and documents the capability, without asserting the agent reproduces the exact same room every run.

How do we test the agent?

End-to-end tests live in dimos/memory2/experimental/test_memory2_agent_ask.py. They run the real LangChain agent against a recorded SqliteStore + a live OpenAI model, so they're gated behind the new experimental marker and excluded from the default pytest run.

Most prompts end with an explicit format directive ("Reply with only the number, nothing else") so the agent commits a clean, parseable final answer instead of dumping its full reasoning chain. A few open-ended cases (room descriptions, expected refusals) grade on content instead.

Two recordings, two env vars. Both ship with the repo as Git LFS tarballs under data/.lfs/ (go2_short.db.tar.gz, go2_hongkong_office.db.tar.gz) — git lfs pull them and extract (tar -xzf data/.lfs/go2_short.db.tar.gz -C data/, or just call dimos.utils.data.get_data(...), which pulls + decompresses), then point the env vars at the extracted .db.

Tool-coverage tests (go2_short.db)

These assert the agent picked the right kind of tool — they don't grade the answer's content.

• test_lists_streams — "How many streams does this memory store have?" → expects list_streams to be called and 4 in the answer (build_memory.py writes 4 streams).
• test_visual_question_uses_image_tool — "At t=22s show me what the robot saw directly forward and describe it in one sentence." → expects at least one of {show_image, recall_view, walkthrough, show_map, frames_facing} to be called and a non-empty answer afterwards. Confirms the langgraph Command path is end-to-end functional.

Content-grounded QA on go2_short.db — test_short_recording_qa (10 cases)

go2_short.db — a short go2 walk through an office with two rooms, two white robots, and a long meeting table. Path supplied via MEMORY2_AGENT_DB.

id	Question (verbatim)	Expected
rooms_count_2	"How many rooms are there? Reply with only the number, nothing else."	2
biggest_room_area_~80m2	"What's the area of the biggest room? Reply with only the number in m², nothing else."	~80 m², accept 64–96 (±20%)
start_equals_end_room	"Did you start in the same room as you ended? Reply with only yes or no, nothing else."	yes
closest_to_meeting_table_2m	"What's the closest distance in meters that you got to the long meeting table? Round to whole numbers no decimals. Reply with only the number, nothing else."	≤ 2
white_robots_count_2	"How many white robots did you pass by? Reply with only the number, nothing else."	2
white_robots_distance_apart	"What's the approximate straight-line distance in meters between the two white robots (not walking distance — the real distance, even across walls)? Round to a whole number. Reply with only the number, nothing else."	3–6 m (inclusive)
man_in_black_moved_hand	"What did the man in black move at the end? Reply with only the single body part, nothing else."	hand or finger
multi_choice_letter_B	4-option multi-choice: plants vs trashcans behind robots (answer is permuted to position B to dodge first-position bias)	letter B
exploration_waypoint_roi	"What's the highest-ROI waypoint to explore next to expand the map? Reply with only the coordinate in the format `x, y` (two numbers separated by a comma), nothing else."	~(+4.2, +9.0) — east-lobe frontier, ±1.5 m per axis
passed_through_doorway_top_left	"Where is the doorway you passed through that's at the top-left of your trajectory? Reply with only the coordinate in the format `x, y` (two numbers separated by a comma), nothing else."	~(−2.0, +9.1) — interior doorway at the upper-left bend, ±1.5 m per axis

Content-grounded QA on go2_hongkong_office.db — test_hongkong_recording_qa (16 cases)

A longer (~558 s) recording of the Hong Kong office — a mixed office and retail space: elevator lobby, glass meeting pods, lounges, a pantry, snack/drink aisles, an acoustic guitar, and several robots (wheeled service robots plus a quadruped robot dog on a dog bed). Path supplied via MEMORY2_AGENT_DB_HONGKONG. Cases marked (skip) have no pinned ground truth yet; (xfail) are known misses / noisy measurements (non-strict, so a lucky run XPASSes); (flaky) pass most of the time within a wide band.

id	Question (verbatim)	Expected
white_robots_count_4_hk	"How many white robots did you pass by? Reply with only the number, nothing else."	4, accept 3 (one is barely visible in two frames)
musical_instruments_last_5min_yes	"Did you see any musical instruments in the last five minutes? Reply with only yes or no, nothing else."	yes — an acoustic guitar is visible at ~t303–312s, inside the last five minutes (a four-minute window would exclude it)
acoustic_guitar_closest_robot_distance (xfail)	"What's the straight-line distance in meters between the acoustic guitar and the closest robot to it? Round to a whole number. Reply with only the number, nothing else."	~11 m, accept 9–13; xfail — noisy short-baseline depth triangulation, agent often lands just outside the band
elevator_room_center	"What's the center coordinate of the room with the elevators? Reply with only the coordinate in the format `x, y` (two numbers separated by a comma), nothing else."	~(+4.55, +2.22) — boundary between the lower central corridor and the right connector, ±1.5 m per axis
total_floor_area (flaky)	"What's the total floor area of the office, summed across all rooms, in square meters? Reply with only the number, nothing else."	~400 m² (eyeballed), accept 300–500 (±100) to absorb polygon-tightness variance
tennis_court_no_room_fits	"Could we have enough space for a standard singles tennis court in any of the rooms if we removed everything in it? If no room fits, say no. Otherwise, say which one."	no — a singles court needs ~24 m × 8 m; no single room is ~24 m long
desks_need_cable_management (skip)	"How many desks need cable management? Reply with only the number, nothing else."	any number (well-formed); skip — ground truth not pinned
table_fit_door_refuses	"For the table the man is sitting on, could we realistically fit it through the nearest door … measure the table's narrow dimension and the door clear width … give a yes/no with the margin in cm …"	a refusal — without a triangulation recipe the tools can't return object-bounded metric widths (lidar near yields point-cloud metadata, the map is a rendered image); the honest move is to decline rather than fabricate a cm margin
longest_time_room_is_a_lounge	"What room did you spend the longest time in? Reply with a short description of the room."	a lounge / sofa / seating / reception area (two lounges tie within sampling noise; accept either)
mixed_use_office_and_retail	"Is this place a regular office, a retail store, or both? Reply with one of: office, store, or both."	both — requires synthesizing offices + the stocked retail section across the whole walk
lounge_closer_to_elevators_than_store	"Which did you pass closer to the elevators — the lounge with the sofas, or the retail store with the snack aisles? Reply with only 'lounge' or 'store'."	lounge — ~6 m from the elevator lobby vs ~17 m for the retail aisles
greeting_to_guitar_avoids_elevators_no	"Is it possible to go from the room with the greeting robot to the room with the acoustic guitar without passing by any elevators? Reply with only yes or no, nothing else."	no — the elevator lobby is the chokepoint between the two
elevator_to_pantry_avoids_white_robots_yes	"Is it possible to go from the elevator room to the pantry/shelves room without passing by the room with the several white robots? Reply with only yes or no, nothing else."	yes — an alternate route to the pantry avoids the white-robots room
failed_nav_attempt_place (skip)	"Is there a place you tried to go to but couldn't? If no, say no. If yes, reply with only the coordinate in the format `x, y` …"	well-formed (a negative, or a parseable coordinate); skip — ground truth not pinned
nonstanding_robot_on_bed_or_dog (xfail)	"Did you see the robot that is not standing? What is it on?"	mentions "dog" / "bed" — a quadruped robot dog rests folded on a dog bed at ~(−2, +4.5), t≈197s; xfail — the agent fixates on the wheeled service robots and misses this small, edge-of-frame target
looked_into_glass_room_not_entered (xfail)	"Were you able to look clearly into a room that you didn't walk into? If no, say no. If yes, briefly name the room and give its approximate center coordinate in the format `x, y`."	yes + a coordinate within ±1.5 m of the glass-walled meeting/office room at ~(+15.5, −2.5) — sealed glass pods the robot passes alongside in the corridor (nearest approach ~1.6 m at ~t84s) but never enters. xfail (non-strict): the prompt is under-determined — across three runs the agent named three different seen-but-not-walked rooms ((6,−3.5), (13.2,−1.2), (15.5,−2.5)) — so this bound usually misses (XFAIL) but a matching run XPASSes; documents the capability without asserting reproducibility. See the worked example above.

How to run

(default suite — experimental excluded; should stay green for everyone)

```
pytest
```

(unit tests for the new memory2 plotting surface — no LLM, no recording needed)

```
pytest dimos/memory2/vis/space/test_space.py
```

(end-to-end agent tests — opt-in, needs OpenAI + the recording(s); skips cleanly if a recording env var is unset)

```
export OPENAI_API_KEY=...
export MEMORY2_AGENT_DB=/path/to/go2_short.db # required for test_short_recording_qa
export MEMORY2_AGENT_DB_HONGKONG=/path/to/go2_hongkong.db # required for test_hongkong_recording_qa
export MEMORY2_AGENT_MODEL=gpt-4.1-mini # optional, default gpt-5.5
pytest -m experimental dimos/memory2/experimental/ -v
```

(one-shot CLI for ad-hoc questions)

```
python -m dimos.memory2.experimental.memory2_agent.ask \
--db /path/to/recording.db \
--model gpt-4.1-mini \
"Where is the biggest room?"
```

(broader smoke run — 7 mixed questions, no assertions, just prints traces)

```
python -m dimos.memory2.experimental.memory2_agent.run_smoke --db /path/to/recording.db
```

Out of scope

• No new core dependencies — LangChain / `langchain_openai` / `langchain_google_genai` only get imported under `dimos/memory2/experimental/`.
• SVG renderer behaviour is preserved (same default padding, same Pose render shape).

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
1855	1	1854	56

View the top 1 failed test(s) by shortest run time

dimos.project.test_no_sections::test_no_section_markers

Stack Traces | 0.504s run time

def test_no_section_markers():
        """
        Fail if any file contains section-style comment markers.
    
        If a file is too complicated to be understood without sections, then the
        sections should be files. We don't need "subfiles".
        """
        violations = find_section_markers()
        if violations:
            report_lines = [
                f"Found {len(violations)} section marker(s). "
                "If a file is too complicated to be understood without sections, "
                'then the sections should be files. We don\'t need "subfiles".',
                "",
            ]
            for path, lineno, text in violations:
                report_lines.append(f"  {path}:{lineno}: {text.strip()}")
>           raise AssertionError("\n".join(report_lines))
E           AssertionError: Found 1 section marker(s). If a file is too complicated to be understood without sections, then the sections should be files. We don't need "subfiles".
E           
E             .../experimental/memory2_agent/map_view.py:462: # --- Begin Space construction ----------------------------------------

lineno     = 462
path       = '.../experimental/memory2_agent/map_view.py'
report_lines = ['Found 1 section marker(s). If a file is too complicated to be understood without sections, then the sections should ....../experimental/memory2_agent/map_view.py:462: # --- Begin Space construction ----------------------------------------']
text       = '    # --- Begin Space construction ----------------------------------------'
violations = [('.../experimental/memory2_agent/map_view.py', 462, '    # --- Begin Space construction ----------------------------------------')]

dimos/project/test_no_sections.py:145: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[greptile-apps]

Greptile Summary

This PR adds a cv2 raster backend for the Space visualization system and ships an experimental LangChain agent that reasons over memory2 databases using spatial, temporal, and semantic tools.

• Raster backend (raster.py): mirrors the existing SVG backend, walking space.elements to paint onto a BGR np.ndarray. Adds three new element types — Polygon, Wedge, RasterOverlay — and extends Point with shape / halo options. The Bounds class is extracted to a shared module so both backends share it.
• Experimental memory2 agent (memory2_agent/): a LangChain ReAct agent backed by 15 tools and 7 skills covering map rendering, CLIP semantic search, camera-FOV projection, walkthrough frame sampling, room partition verification, and a sandboxed Python REPL. All previously flagged issues from earlier review rounds are resolved in this HEAD.

Confidence Score: 5/5

Safe to merge — all blocking issues from previous review rounds are resolved, and the experimental path is properly isolated.

All previously identified defects are addressed. The raster renderer correctly handles Y-flipping, alpha blending, and PointCloud2 caching. The agent tool_lock correctly serializes SQLite access, dynamic duration injection is clean, and _fix_parallel_tool_batches reordering logic is sound.

No files require special attention.

Important Files Changed

Filename	Overview
dimos/memory2/vis/space/raster.py	New cv2 raster renderer mirroring the SVG backend. Alpha blending, RGBA to BGR conversion, and PointCloud2 bounds/draw caching are all correct.
dimos/memory2/experimental/memory2_agent/map_view.py	Largest new file covering map rendering, room-partition verification, FOV cone visualization, and frame recall. Pose-None guards and valid_rooms/polys_world index alignment are both correct.
dimos/memory2/experimental/memory2_agent/tools.py	Tool wrappers with dynamic known_streams, indexed show_image queries, and _tool_lock serialization for the shared SQLite connection.
dimos/memory2/experimental/memory2_agent/agent.py	ReAct agent wiring with dynamic duration injection and sound parallel-tool-response reordering.
dimos/memory2/vis/space/elements.py	Clean additions of Polygon, Wedge, RasterOverlay types and Point.shape/halo fields.
dimos/memory2/vis/space/bounds.py	Shared Bounds dataclass extracted from svg.py with no logic changes.
dimos/memory2/experimental/test_memory2_agent_ask.py	E2E tests properly gated behind the experimental marker with appropriate xfail markers for noisy measurements.

_{Reviews (10): Last reviewed commit: "build(memory2/agent): move experimental ..." | Re-trigger Greptile}

[greptile-apps]

Full image stream materialized on every show_image call

store.stream(stream).to_list() deserializes every observation in the image stream into memory before the single nearest-timestamp entry is picked with min(...). At 10 fps over 60 s, this is ~600 full-resolution images (~2.8 MB each uncompressed) loaded unnecessarily for each tool call. The store already supports ordered queries (see recent which uses .order_by("ts", desc=True).limit(n)). A narrower query or at minimum deferring data decoding would avoid the O(N) memory spike. The same pattern appears in frames_that_could_see_point (loads all color_image frames before filtering by FOV).

[Mgczacki]

Going around this afaik would need further changes to memory2 itself, so I don't think it's in scope for this PR.

[greptile-apps]

obs.pose is accessed without a None guard here. Every other function in this file that iterates color_image observations — frames_that_could_see_point (line 860) and recall_view (line 1215) — contains an explicit if obs.pose is None: continue check. If any sampled frame lacks a pose (e.g. frames recorded before odometry was established), obs.pose[0] raises TypeError, which propagates uncaught through the walkthrough tool in tools.py (no try/except wrapper there).

Suggested change

	for s_ts in sample_ts:
	obs = min(img_obs, key=lambda o: abs(o.ts - s_ts))
	x, y = obs.pose[0], obs.pose[1]
	qx, qy, qz, qw = obs.pose[3:7]
	for s_ts in sample_ts:
	obs = min(img_obs, key=lambda o: abs(o.ts - s_ts))
	if obs.pose is None:
	continue
	x, y = obs.pose[0], obs.pose[1]
	qx, qy, qz, qw = obs.pose[3:7]