Skip to content

feat: semantic gallery search over the embedding index - #43

Open
lstein wants to merge 1 commit into
feat/image-map-09-cluster-selectfrom
feat/image-map-10-semantic-search
Open

feat: semantic gallery search over the embedding index#43
lstein wants to merge 1 commit into
feat/image-map-09-cluster-selectfrom
feat/image-map-10-semantic-search

Conversation

@lstein

@lstein lstein commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

PR 10 of the Image Map stack — full-stack semantic search alongside the existing metadata search.

Backend

  • GET /v1/image_map/search ranks the caller's accessible images by similarity to a text query (q) or a reference image (image_name). Text queries embed through the configured model's lazily-loaded text tower — with per-model padding (SigLIP requires pad-to-max-length; CLIP pads to longest). Vision-only installs raise a typed error mapped to 409 with a clear message; image similarity still works there (stored embedding, guarded by assert_image_read_access).
  • Similarity is a dot product over L2-normalized embeddings with an argpartition top-k. The accessible embedding matrix sits in a 2-entry LRU keyed by the scope hash (self-invalidating on any accessible-set change; ~300MB/entry at 100k×768 is the deliberate capacity trade-off, documented in code). Encoding and search run off the event loop; admins search globally, users their accessible set.

Frontend

  • A "Semantic" checkbox beside the gallery search field switches the (300ms-debounced) text box to similarity ranking; results render in the normal grid in rank order, paginated by slicing the ranked window and bulk-hydrating DTOs. Names the hydration endpoint no longer knows are subtracted from the total so infinite scroll can't chase an unreachable count.
  • The search field is a drop target: dropping a gallery image shows a "Similar to " chip and ranks by visual similarity; unchecking Semantic clears the chip too (no dead-looking controls).
  • Search identity participates in the gallery's cache keys, so mode toggles interact correctly with the TTL cache and scroll window.

Two adversarial reviews covered this code; all confirmed findings (SigLIP padding, cache clear-all thrash, unreachable-total scroll loop, per-keystroke encode + 409-toast spam, chip/checkbox state hole, cross-hook click race) are fixed with tests. Noted latent divergence: per-user shared_boards scoping (no write API yet) is honored by search scope but not by the hydration endpoint's access check — flagged in code for when shares ship.

Testing

Backend: search ranking/scoping incl. a real per-user SQL scope test, 409 paths, reference-image access control. Frontend: query building, ranked-window hydration incl. dropped-name totals, debounce-covered view logic. Full suites green: 200 backend / 907 frontend / lint / build.

🤖 Generated with Claude Code

@lstein
lstein marked this pull request as draft August 2, 2026 22:24
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from 2b60df0 to fb7b7f8 Compare August 2, 2026 22:38
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from 3990b04 to 5784c6a Compare August 2, 2026 22:38
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from fb7b7f8 to 8742a79 Compare August 3, 2026 01:38
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from 5784c6a to 1966859 Compare August 3, 2026 01:38
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from 8742a79 to b9be000 Compare August 3, 2026 02:26
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from 1966859 to 238768f Compare August 3, 2026 02:26
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch 3 times, most recently from ff47c33 to f5fab5e Compare August 3, 2026 16:10
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from 238768f to 0fca217 Compare August 3, 2026 18:21
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from f5fab5e to 14ad377 Compare August 3, 2026 18:21
@lstein
lstein marked this pull request as ready for review August 3, 2026 18:28
@lstein
lstein requested a review from Pfannkuchensack as a code owner August 3, 2026 18:28
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from 14ad377 to 000ebea Compare August 4, 2026 00:24
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from 0fca217 to f204e11 Compare August 4, 2026 01:43
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from 000ebea to 986cd46 Compare August 4, 2026 01:43
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from f204e11 to d0c4e2e Compare August 4, 2026 05:37
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch 2 times, most recently from b9b9f00 to ff1ee4f Compare August 4, 2026 14:58
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from d0c4e2e to 6c2d8df Compare August 4, 2026 14:58
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from ff1ee4f to ff2a0ad Compare August 4, 2026 22:32
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from 6c2d8df to a423504 Compare August 4, 2026 22:32
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from ff2a0ad to b592992 Compare August 5, 2026 00:31
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch 2 times, most recently from 3374c16 to ce5c18d Compare August 5, 2026 00:38
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch 2 times, most recently from f46643e to e6b4634 Compare August 5, 2026 01:18
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from ce5c18d to 69fa638 Compare August 5, 2026 01:18
@lstein
feat: semantic gallery search over the embedding index
7362636 
Adds cosine-similarity search to the gallery, alongside the existing
metadata search.

Backend:
- `GET /v1/image_map/search` ranks the caller's accessible images by
  similarity to a text query (`q`) or a reference image (`image_name`).
  Text queries embed through the configured model's text tower, loaded
  lazily; vision-only installs raise a typed error mapped to 409 with a
  clear message — image similarity still works there, using the reference
  image's stored embedding (guarded by assert_image_read_access).
- Similarity is a dot product over L2-normalized embeddings with an
  argpartition top-k; the accessible embedding matrix is cached in memory
  keyed by scope hash (self-invalidating when the accessible set changes).
  Encoding and search run off the event loop. Admins search globally;
  users their accessible set.

Frontend:
- A "Semantic" checkbox beside the gallery search field switches the text
  box to similarity ranking; results render in the normal gallery grid in
  rank order, paginated by slicing the ranked window and bulk-hydrating
  DTOs (board/order/starred controls do not apply to ranked results).
- The search field is a drop target: dropping a gallery image replaces the
  text box with a "Similar to <image>" chip and ranks the gallery by
  visual similarity to it; clearing the chip restores text search.
- Search identity participates in the gallery's query cache keys, so
  toggling modes or editing the query interacts correctly with the
  existing TTL cache and infinite-scroll window.

Amended during the rebase onto the re-architected webv2:

The backend half applied cleanly (image_map router, image_index service,
and their tests); OpenAPI artifacts were regenerated the CI way. The
frontend half was rewritten against main's gallery architecture
(features/gallery core/data/ui + TanStack Query) instead of the old
workbench widget pipeline:

- The reference union and blob registry moved from workbench/gallery/
  {api,externalImageQuery}.ts into features/gallery/core/
  semanticImageQuery.ts (types, single-slot external-file registry,
  dataTransfer interpretation, persisted-value parse with legacy
  bare-string tolerance and dangling-file-key clearing).
- The transport (searchGallerySemantic, ranked item-name refs) lives in
  features/gallery/data/backend.ts; GalleryItemsFilter and
  CanonicalGalleryItemsFilter carry the reference, with file queries
  canonicalized to their registry id (label-free) so cache identity is
  stable across relabels. Semantic queries ride the date-board mechanism:
  one shared ranked name-list query (60s stale window) serves every page,
  both pagination modes, and range selection, and each page hydrates a
  slice of it in rank order — so ranks cannot drift between pages and a
  dropped-file reference uploads its blob once per search, not once per
  page fetch or cache invalidation. AbortSignals and account-scope checks
  compose exactly as in the neighboring branches.
- Relevance ordering survives the shared window merge:
  mergeGalleryItemWindow skips the date re-sort and the recent-image
  overlay while a semantic query is active, and pending queue
  placeholders are hidden in the state view (a ranked list has no
  chronological insertion point).
- The chip UI replaced the old toolbar checkbox row: GalleryItemSearch
  wraps main's token-chip search field with the drop target (dnd-kit
  droppable + native file/URL drop handlers) and renders the "Similar
  to …" chip in place of the input. In-app drops resolve through
  GalleryBoardDragMonitor (resolveGallerySemanticSearchDrop in
  galleryDnd.ts), matching how board drops are forwarded; drags carrying
  only videos resolve to nothing.
- Setting/clearing the reference goes through the gallery UI port's
  widgets.patchGalleryValues (persisted widget values), the same surface
  the parsed query is read back from; the state view exposes an
  identity-stable accessor so in-flight searches are not restarted by
  unrelated renders.
- Hardened after an adversarial review of the rewrite: optimistic move
  patches keep items in ranked windows (similarity membership is
  board-agnostic — previously a move would evict the item and it would
  flicker back after invalidation); semantic query failures surface
  through the notifications port instead of rendering as an empty "no
  images match" grid; registry file ids carry a random token so a
  persisted id from another JS realm cannot resolve the wrong blob; and
  plain-text drops fall through to the browser's default insertion into
  the search input instead of being swallowed by the drop target.
  Known, inherited-from-v7 limitation: the ranked result set ignores
  board/view/order/starred controls, but those controls stay interactive
  and each change re-keys (and re-runs) the same search.
- Dropped: the "Semantic" text-search checkbox — on this architecture
  text queries stay metadata search and only image references rank by
  similarity (the backend's `q=` endpoint remains available). Also
  dropped the v7 spring-loaded board-menu fix: main's boards UI is a
  docked panel (GalleryBoardsPanel + GalleryBoardDragMonitor disclosure),
  not an auto-opening dropdown covering the toolbar, so the bug the fix
  addressed does not exist here.
- Architecture performance baselines recaptured (build + browser, browser
  verified passing twice): core/semanticImageQuery.ts joins the initial
  editor source graph through data/backend.ts and galleryStateView.ts,
  which are already loaded at editor boot (editor initial +2,773 raw
  bytes; owned bytes unchanged; launchpad minifier drift only).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review fix: applying or clearing a semantic image query resets gallery
pagination and clears the text search term the chip replaces.
@lstein
lstein force-pushed the feat/image-map-09-cluster-select branch from 69fa638 to f717ca1 Compare August 5, 2026 01:21
@lstein
lstein force-pushed the feat/image-map-10-semantic-search branch from e6b4634 to 7362636 Compare August 5, 2026 01:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@blessedcoolant blessedcoolant Awaiting requested review from blessedcoolant blessedcoolant is a code owner
@JPPhoto JPPhoto Awaiting requested review from JPPhoto JPPhoto is a code owner
@Pfannkuchensack Pfannkuchensack Awaiting requested review from Pfannkuchensack Pfannkuchensack is a code owner

Assignees

No one assigned

Labels

api frontend python python-tests services

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lstein