Skip to content

feat(api): background image embedding indexer service - #35

Open
lstein wants to merge 1 commit into
feat/image-map-01-schemafrom
feat/image-map-02-indexer
Open

feat(api): background image embedding indexer service#35
lstein wants to merge 1 commit into
feat/image-map-01-schemafrom
feat/image-map-02-indexer

Conversation

@lstein

@lstein lstein commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #34 (PR 2 of the Image Map stack). Adds the worker that keeps the embedding index current:

  • ImageIndexService: daemon worker thread fed by the image service's on_changed/on_deleted callbacks (enqueue-only — they fire synchronously on the generation thread) plus a startup backfill over unembedded gallery images.
  • Encoding: via the configured CLIP Vision or SigLIP model (image_index_model, default clip-vit-large-patch14). On GPU, the model loads through the shared model cache and indexing pauses while a generation is in progress. image_index_device: cpu keeps a service-local CPU copy instead — zero VRAM contention.
  • Resilience (from adversarial review): image processors fall back to library defaults when the model dir ships no preprocessor_config.json (true of InvokeAI-published CLIP Vision models — same convention as IP-Adapter/FLUX); inputs are converted to the cached model's dtype (fp16 on GPU); transient failures are retried up to 3× with backoff before an image is marked failed; the dedup set is released in a finally so no image can get stuck unindexable; stop() logs if the worker doesn't join; double-start() is guarded.
  • Config: image_index_enabled (default off), image_index_model, image_index_device, image_index_batch_size.
  • Events: image_index_status (aggregate counts) — routed to the admin room only, since the counts aggregate over all users' images.
  • Changing the configured model discards embeddings from the previous model.

An adversarial review of this diff was run; both critical findings (broken processor loading on stock models, GPU dtype mismatch) plus the retry/poisoning, _pending-leak, and event-privacy findings are fixed above. Remaining accepted trade-offs: CPU mode holds its model copy in RAM while the service lives; image_index_device values other than cpu defer to the model cache (documented in the config description).

Testing

10 service tests with an injected fake encoder (CI never loads models): backfill with eligibility filtering, callback-driven indexing, unloadable-image skip, transient-failure retry-to-success, bad-output batch failure, processor-fallback, status events, clean stop.

🤖 Generated with Claude Code

@lstein
lstein marked this pull request as draft August 2, 2026 22:23
@lstein
lstein force-pushed the feat/image-map-02-indexer branch from a970db9 to 1bc499a Compare August 2, 2026 22:38
@lstein
lstein force-pushed the feat/image-map-02-indexer branch from 1bc499a to c5cc29c Compare August 3, 2026 01:38
@lstein
lstein marked this pull request as ready for review August 3, 2026 18:28
@lstein
lstein requested a review from Pfannkuchensack as a code owner August 3, 2026 18:28
@lstein
lstein force-pushed the feat/image-map-02-indexer branch 2 times, most recently from 7ea889c to 5ca27a0 Compare August 4, 2026 05:37
@lstein
lstein force-pushed the feat/image-map-01-schema branch from b3f4618 to 712ed53 Compare August 4, 2026 05:37
@lstein
lstein force-pushed the feat/image-map-02-indexer branch 2 times, most recently from 4a7793d to 8412295 Compare August 4, 2026 22:32
Adds the worker that keeps the image embedding index current:

- New `ImageIndexService`: daemon worker thread fed by the image service's
  on_changed/on_deleted callbacks (enqueue-only on the caller's thread) plus
  a startup backfill pass over unembedded gallery images. Images that fail
  to load or encode are skipped and remembered so they cannot stall the
  backfill.
- Embeds via the configured CLIP Vision or SigLIP model. On GPU the model
  loads through the shared model cache and indexing pauses while a
  generation is in progress; `image_index_device: cpu` keeps a
  service-local CPU copy instead, avoiding the cache and VRAM entirely.
- New config: `image_index_enabled` (default off), `image_index_model`
  (default clip-vit-large-patch14), `image_index_device`,
  `image_index_batch_size`.
- New broadcast-safe `image_index_status` socket event with aggregate
  counts.
- Changing the configured model discards embeddings from the old model.

Tests use an injected fake encoder; no models or GPU are touched in CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lstein
lstein force-pushed the feat/image-map-02-indexer branch from 8412295 to ad8d3f1 Compare August 5, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant