Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions packages/mcp/examples/photo-to-scene.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,10 @@ Optional knobs:
`sampling_response_invalid` MCP errors.
3. A fresh `SceneGraph` is built using the core schema factories: a
`site` → `building` → `level 0` skeleton, then one `WallNode` per
vision wall and one `ZoneNode` per vision room. Each node is
vision wall and one `ZoneNode` per vision room. Detected doors and windows
are attached to the nearest wall, stairs get a stair plus segment node, and
detected furniture is matched to the built-in catalog and placed with
clearance checks. Each node is
re-parsed with `AnyNode.safeParse`; invalid ones are dropped with a
warning appended to `notes`.
4. `bridge.setScene(...)` swaps the live scene so any follow-up MCP call
Expand All @@ -69,6 +72,10 @@ Optional knobs:
"url": "/scene/scene_01hx8a...",
"walls": 4,
"rooms": 1,
"doors": 2,
"windows": 4,
"stairs": 1,
"furniture": 6,
"confidence": 0.82
}
```
Expand Down Expand Up @@ -116,4 +123,6 @@ wall, and issues `cut_opening` — no extra wiring needed.

- `photo_to_scene` is a one-shot primitive: one call, one scene.
- Vision confidence is surfaced so the agent can warn the user.
- v0.1 covers walls + zones; doors, windows, items are follow-up tools.
- The one-shot builder returns walls, rooms, openings, stairs, and matched
catalog furniture. Items that cannot be matched or placed are reported in
`notes` rather than silently dropped.
20 changes: 20 additions & 0 deletions packages/mcp/src/tools/photo-to-scene/photo-to-scene.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,13 @@ const VALID_VISION_JSON = {
approximateAreaSqM: 20,
},
],
doors: [{ position: [2.5, 0], widthM: 0.9, swingDirection: 'inward' }],
windows: [{ position: [5, 2], widthM: 1.2, heightM: 1.2, sillHeightM: 0.9 }],
stairs: [{ position: [1, 1], widthM: 1, runLengthM: 2.5, rotationDeg: 90 }],
furniture: [
{ type: 'sofa', position: [2.5, 2.8], rotationDeg: 0, widthM: 2.2, depthM: 0.9, confidence: 0.9 },
{ type: 'coffee-table', position: [2.5, 1], rotationDeg: 0, confidence: 0.86 },
],
approximateDimensions: { widthM: 5, depthM: 4 },
confidence: 0.82,
}
Expand Down Expand Up @@ -103,6 +110,10 @@ describe('photo_to_scene', () => {
url?: string
walls: number
rooms: number
doors: number
windows: number
stairs: number
furniture: number
confidence: number
}
expect(structured.walls).toBe(4)
Expand All @@ -124,6 +135,15 @@ describe('photo_to_scene', () => {
const zones = allNodes.filter((n) => n.type === 'zone')
expect(walls.length).toBe(4)
expect(zones.length).toBe(1)
expect(allNodes.filter((n) => n.type === 'door').length).toBe(1)
expect(allNodes.filter((n) => n.type === 'window').length).toBe(1)
expect(allNodes.filter((n) => n.type === 'stair').length).toBe(1)
expect(allNodes.filter((n) => n.type === 'stair-segment').length).toBe(1)
expect(allNodes.filter((n) => n.type === 'item').length).toBe(2)
expect(structured.doors).toBe(1)
expect(structured.windows).toBe(1)
expect(structured.stairs).toBe(1)
expect(structured.furniture).toBe(2)

// Scene was persisted in the store.
const saved = await store.load(structured.sceneId!)
Expand Down
Loading