feat: automatic cluster labels for the image map

[lstein]

Summary

PR 11 — the top of the Image Map stack. Ports PhotoMapAI's vocabulary-based cluster labeling.

• A bundled ~1700-phrase vocabulary (Places365 + curated) embeds through the index model's text tower with a prompt-template ensemble (encode each template, L2-normalize, mean-pool, re-normalize). First run per model takes tens of seconds, so phrase embeddings are disk-cached next to the database — keyed by model (filename sanitized for Windows) and a fingerprint covering vocabulary, templates, and the embedding recipe, so a tokenization fix can never serve stale cached vectors.
• GET /v1/image_map/cluster_labels labels each visible cluster with the phrase nearest its image-embedding centroid (top-k with alternates). Clustering mirrors /points exactly (same eps semantics over the same accessible set), image embeddings come from the same LRU matrix cache the search endpoint uses (no per-request full-BLOB read), and the response carries the projection's updated_at.
• The frontend fetches labels best-effort after each points refresh — skipped entirely when the map isn't ready — and renders them as plotly annotations at cluster centroids. Label responses are discarded unless they match the current projection and are the newest request, so a recompute or out-of-order response can never attach labels to renumbered clusters; identical label sets don't trigger a rebuild.

An adversarial review covered this diff; all its findings (stale-label attachment, per-request matrix materialization, Windows-illegal cache filename, cache poisoning across recipe changes, wasted 409 fetches) are fixed above.

---

Image Map setup walkthrough (whole stack, PRs 1–11)

Since this PR is the top of the stack, here is an end-to-end guide to running the feature.

1. Configure invokeai.yaml

image_index_enabled: true          # off by default — the whole feature is inert without it

# Optional (defaults shown):
image_index_model: clip-vit-large-patch14   # any installed CLIP Vision or SigLIP model, by name
image_index_device: cpu                     # only `cpu` is meaningful — see below
image_index_batch_size: 8

Restart the backend after changing these. image_index_device: cpu embeds with a service-local CPU copy of the model: no VRAM use, and indexing keeps running during generations. Left unset, embedding runs through the model cache on the GPU and politely pauses while a generation is in progress.

2. Install the embedding model

Model Manager → Starter Models → install "clip-vit-large-patch14" (CLIP ViT-L Image Encoder, type: CLIP Vision).

Gotcha: the starter catalog has two models named clip-vit-large-patch14. The index needs the CLIP Vision one. The other (type CLIPEmbed, "CLIP-L text encoder", used by FLUX) will not work — if you install only that one, the startup log tells you explicitly that a same-named model of the wrong type was found.

The CLIP Vision install ships the full dual-tower checkpoint, so it also powers text search and cluster labels. On first load, transformers prints a loud "LOAD REPORT" listing text_model.* keys as UNEXPECTED — this is benign (the vision tower is being loaded from a dual-tower checkpoint; the text tower loads separately when search/labels need it).

There is no auto-download: if the configured model isn't installed, the service idles and logs a warning.

SigLIP-family models also work (image_index_model: <name>), with different score characteristics. Changing the model discards all embeddings computed by the previous model and re-embeds from scratch.

3. Populating the index

On the next start, a backfill pass embeds every existing gallery image in batches; new images are embedded within a second or two of landing in the gallery. Admins see an "indexing N/M" counter in the Image Map widget's footer while work is pending.

The first map open after a backend restart takes ~30–60 s: UMAP's numba kernels JIT-compile once per process, then the projection computes. Later recomputes take seconds, and an unchanged map serves instantly from cache.

4. Using the map

Open the Image Map widget from the widget picker (docks left/right/center; the widget menu also offers Float window — drag, resize, shade, maximize, dock back).

• Points are colored by cluster; unclustered noise is dimmed. Cluster labels render at cluster centroids (toggle in the widget header).
• Click a point → selects that image in the gallery, Preview follows. The gold circle-dot marks the current image and tracks selection from anywhere in the app, recentering while preserving your zoom.
• Wheel zooms around the cursor, two-finger pinch works on trackpads, drag pans. Faint gridlines help orientation when few points are visible.
• Hover a point for a thumbnail popup.
• Cluster-select mode (header toggle): clicking a point selects its whole cluster in the gallery (ordered by proximity walk from the clicked image) and highlights it on the map; plain click clears.
• The footer shows the point count, an "updating…" hint while a recompute is in flight, and a manual refresh button.

The map refreshes itself: after a generation, the new point appears a few seconds after the embed completes (admins ride the status event; other users get a per-user counts-free poke for their own images). Deletions vanish automatically for admins; non-admin users must press the footer refresh after deleting (the backend can't resolve an owner for an already-deleted row).

5. Semantic and image-similarity search

In the gallery search field:

• Text search: type a query, then click the sparkles button at the field's right end. The query becomes a Semantic search: "…" chip and the grid re-ranks by similarity. Requires the model's text tower (the full CLIP Vision install has it; a vision-only install returns a clear "text search unavailable" error).
• Image similarity: drag a gallery thumbnail onto the search field → Similar to <image> chip, grid ranked by visual similarity. The field also accepts image files dropped from the OS and images dragged in from web pages (app-origin URLs resolve to the stored embedding; external URLs are fetched and embedded server-side for that one query).
• Clear the chip to return to normal metadata search. Inside a semantic query the text is taken verbatim — date tokens (from:, to:…) and starred-first ordering don't apply while a semantic query is active.

Gotchas

• Only gallery images are indexed: non-intermediate images in the general category — i.e. exactly what shows in the gallery grid. Canvas layers, masks, intermediates, and images saved as assets never appear on the map or in semantic search. In particular, a workflow must send its output to the gallery for it to be mapped.
• Two starter models share the name clip-vit-large-patch14; the index needs the CLIP Vision one (see §2).
• Switching image_index_model re-embeds the entire gallery.
• Multiuser: each user's map and search cover their own images plus shared/public-board images; admins see everything. The footer's index counts are admin-only by design (they aggregate all users' activity).
• Cluster labels embed their vocabulary on first use per model (tens of seconds), then serve from a disk cache next to the database.
• DBSCAN's eps is available on the API (GET /v1/image_map/points?eps=) but not yet exposed in the UI.

---

Testing

Labeling math (vocabulary load/dedup, fingerprint discrimination, template ensembling/pooling, per-cluster nearest-phrase with noise exclusion), router alignment test proving labels line up with /points cluster ids, text-encoder-unavailable 409, no-projection empty case. Full suites green: 200 backend / 907 frontend / lint / build at the stack top.

🤖 Generated with Claude Code