Skip to content

feat(artwork): background self-heal scanner (phase 1.i.1) - #31

Merged
InstaZDLL merged 4 commits into
mainfrom
feat/1-i-1-artwork-background-scanner
Jun 6, 2026
Merged

feat(artwork): background self-heal scanner (phase 1.i.1)#31
InstaZDLL merged 4 commits into
mainfrom
feat/1-i-1-artwork-background-scanner

Conversation

@InstaZDLL

@InstaZDLL InstaZDLL commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

Sprint 1 / Phase 1.i.1 — closes Sprint 1 on artwork. Spawns a tokio task at boot that periodically scans metadata_artwork for parents whose variant cache is incomplete (a partial-write incident left a parent row + parent bytes but lost a variant insert), pulls the source bytes from object_store, re-runs the resize pipeline, and inserts only the variants still missing.

Why not apalis

The post-1.g sprint plan explicitly listed three candidates: `apalis` (Postgres queue + retries), `tokio-cron-scheduler` (simple cron), or rolling-our-own with `tokio::spawn` + the existing compaction-loop pattern.

The workload here is periodic catch-up, not queue-driven retries with priorities — the right primitive is a tokio interval loop. The surrounding infrastructure already has a sync compaction loop to generalise from, so the new module sits cleanly next to it.

`apalis` lands when a job type genuinely needs persistent queues + priorities + retries — RFC-004 community moderation (vote-based, needs durable queue) is the natural first user.

Wire surface

New env vars, opt-out by default once the artwork backend is set:

Env Default Effect
`WAVEFLOW_ARTWORK_SCANNER_DISABLED` unset Set to anything non-empty to skip the spawn
`WAVEFLOW_ARTWORK_SCANNER_INTERVAL_SECS` 300 Cycle cadence; floor 1 s applied
`WAVEFLOW_ARTWORK_SCANNER_BATCH_SIZE` 50 Max parents per cycle; floor 1 applied

Highlights

  • `db::artwork::list_partial_parents` — `LEFT JOIN ... GROUP BY parent_hash` + `COUNT(*) < expected` query, ordered `a.created_at ASC` so a long backlog drains in upload order.
  • `artwork_jobs::run_once` public for deterministic test drive (`cargo test` doesn't wait on the 5-minute ticker).
  • `ON CONFLICT (parent_hash, variant) DO NOTHING` on the repair inserts → race-safe with the upload-side repair from feat(artwork): synchronous resize pipeline + variant endpoint (phase 1.h.3) #30 + with multi-replica deploys.
  • Per-parent failures isolated — one broken parent (e.g. parent bytes lost from storage) logs + skips, doesn't abort the cycle.
  • `MissedTickBehavior::Skip` — a slow cycle doesn't queue up back-to-back catch-up runs.
  • First tick consumed — fresh boot doesn't compete with first wave of REST traffic for the pool.
  • `Debug` on `ArtworkScannerConfig` structurally redacted (no secrets, just numbers).

Test plan

  • `cargo fmt --all --check` ✅ locally
  • `cargo clippy --all-targets --all-features -- -D warnings` ✅ locally
  • `cargo test --lib` — 34/34 pass (no new unit tests here; the new module is exercised end-to-end)
  • `cargo test --test artwork_scanner` — 3 integration tests (real Postgres needed; CI will run):
    • `run_once_repairs_a_partial_cache` — seed parent + 1 variant, call run_once, assert both variants present + correct shape.
    • `run_once_is_a_noop_when_cache_is_complete` — empty cycle reports 0 repairs.
    • `run_once_skips_parents_with_missing_bytes` — phantom-hash parent doesn't abort the cycle.

What's not in scope

  • `apalis` infrastructure — deferred to a job type that actually benefits from it.
  • Multi-replica leader election — multiple scanner replicas are safe (race-safe via `ON CONFLICT`) but redundant. A leader-election scheme (advisory locks, etc.) lands when a deploy actually needs more than one replica.
  • Per-parent retry tracking — partial-write incidents are rare enough that "the next cycle picks it up" is the right semantics; per-parent attempt counters would buy us nothing today.

Sprint 1 ✅ COMPLETE

This closes the artwork track for Sprint 1:

Next: Sprint 2 — playlist_track materialization (Phase 1.j) + OAuth providers + RFC-004 draft.

Summary by CodeRabbit

  • Nouvelles fonctionnalités

    • Scanner d'arrière-plan « self‑heal » pour réparer automatiquement les caches d'artwork incomplets (exécution périodique, traitement par lots, idempotence et sécurité en cas de concurrence).
  • Configuration

    • Variables d'environnement pour activer/désactiver le scanner, régler l'intervalle d'exécution et la taille des lots.
  • Documentation

    • Documentation décrivant le comportement, la cadence, la gestion d'erreurs et le choix d'implémentation.
  • Base de données

    • Migration ajoutant un champ de backoff pour marquer les échecs de réparation.
  • Tests

    • Tests d'intégration couvrant plusieurs scénarios de réparation.

The 1.h.3 upload handler heals partial variant caches on the
re-upload path, but only when a client happens to re-POST the same
bytes. A partial cache nobody re-uploads stays partial forever and
the public read endpoints keep advertising an incomplete
`variants[]` to every reader.

Spawn a tokio task at boot — same shape as the sync compaction loop
— that periodically scans `metadata_artwork` for parents whose
variant row count is below `EXPECTED_VARIANT_COUNT`, fetches the
source bytes from object_store, re-runs the resize pipeline, and
inserts only the variants still missing. The repair writes go
through `ON CONFLICT (parent_hash, variant) DO NOTHING` so a
concurrent upload-side repair (or a peer scanner in a multi-
replica deploy) collapses cleanly.

Configurable via env: `WAVEFLOW_ARTWORK_SCANNER_DISABLED` to opt
out entirely, plus `_INTERVAL_SECS` (default 300, floor 1) and
`_BATCH_SIZE` (default 50, floor 1) for the cadence + batch knobs.
Floors are applied at boot so a misconfigured 0 doesn't busy-loop
the worker. The scanner is only resolved when an artwork backend is
also configured — a scanner with no storage would have nothing to
do.

We deliberately picked a tokio polling loop over `apalis` for
1.i.1: the workload is "periodic catch-up", not "queue-driven
retries with priorities", and the surrounding infra already has a
compaction loop to generalise from. The post-1.g sprint plan
explicitly lists this as one of the valid candidates (`apalis`,
`tokio-cron-scheduler`, or rolling-our-own). `apalis` lands when a
job type genuinely needs persistent queues + priorities (e.g.
RFC-004 community moderation).

Tests:
- `run_once_repairs_a_partial_cache` seeds a parent row + parent
  bytes + one variant, calls `run_once`, asserts both variants
  present afterwards with the missing one inserted.
- `run_once_is_a_noop_when_cache_is_complete` confirms a clean
  cycle reports 0 repairs.
- `run_once_skips_parents_with_missing_bytes` asserts a parent
  with a phantom hash (no bytes in storage) doesn't abort the
  cycle, leaving the broken row for the next pass.

Signed-off-by: InstaZDLL <github.105mh@8shield.net>
@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 4674ee90-3a4f-44e5-aec2-0ec8102a0b4e

📥 Commits

Reviewing files that changed from the base of the PR and between d99919f and b3a5235.

📒 Files selected for processing (1)
  • src/config.rs

📝 Walkthrough

Walkthrough

Cette PR ajoute un scanner Tokio de fond configurable via variables d'environnement qui, périodiquement, liste les parents d'artwork partiels, récupère leurs bytes, relance le pipeline de génération de variantes et insère idempotent les variantes manquantes.

Changes

Scanner d'artwork auto-réparant

Layer / File(s) Summary
Configuration et contrats publics
src/config.rs, src/artwork_jobs.rs
Constantes publiques et ArtworkScannerConfig; Config reçoit artwork_scanner: Option<...> et lit/valide WAVEFLOW_ARTWORK_SCANNER_* (interval ≥1s, batch ≥1).
Schéma DB et requêtes de sélection
src/db.rs, migrations/20260608000000_artwork_repair_backoff.sql
Ajout de la colonne last_repair_failure_at (epoch-ms) + index partiel ; fonctions list_partial_parents(...) et mark_repair_failure(...) pour sélectionner candidats et marquer échecs.
Implémentation du scanner et logique de réparation
src/artwork_jobs.rs
Module public artwork_jobs : documentation, constantes, spawn() lance la boucle périodique, run_once() liste parents à réparer et itère, repair_one() recharge bytes, génère variantes et écrit/injecte les variantes manquantes de façon idempotente.
Lancement et exposition du module
src/lib.rs, src/main.rs
Export pub mod artwork_jobs; le binaire démarre la tâche conditionnellement si artwork_storage et config.artwork_scanner sont présents, avec logs d'activation/désactivation.
Tests d'intégration et fixtures
tests/artwork_scanner.rs, tests/support.rs
Tests couvrant réparation d'un cache partiel, réparation depuis zéro, noop quand complet, gestion d'une entrée sans bytes (marquage backoff); fixtures initialisent artwork_scanner: None.
Documentation et .env
CLAUDE.md, .env.example
Doc du flux de scan (liste→récupère bytes→génère→insère), idempotence via ON CONFLICT DO NOTHING, choix du polling Tokio ; variables d'environnement documentées.
Dépendances dev
Cargo.toml
Mise à jour dev-dependency tokio-tungstenite 0.27 → 0.29.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • InstaZDLL/waveflow-server#28: Le PR principal étend le module src/db.rs::artwork et la table metadata_artwork introduite par le PR #28 (ajout de last_repair_failure_at + fonctions list_partial_parents/mark_repair_failure) afin d’alimenter le scanner “self-heal” qui réutilise aussi ArtworkStorage; c’est donc directement lié au socle cache artwork.
  • InstaZDLL/waveflow-server#30: Le PR principal ajoute un scanner “self-heal” en arrière-plan qui régénère et réinsère les variantes manquantes en réutilisant artwork_pipeline::generate_variants et la logique DB insert_variant_if_absent introduites dans le PR #30.

Poem

🖼️ Un scanner veille dans l'ombre,
Guérissant les caches pleins de trous,
Tokio tique, les variantes dansent,
INSERT CONFLICT — tout reste intact,
Les images reprennent leurs couleurs.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Le titre résume précisément le changement principal : implémentation d'un scanner self-heal en arrière-plan pour gérer le cache incomplet des variantes d'artwork.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/1-i-1-artwork-background-scanner

Comment @coderabbitai help to get the list of available commands and usage tips.

@InstaZDLL InstaZDLL self-assigned this Jun 6, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/db.rs`:
- Around line 655-656: La requête de sélection "ORDER BY a.created_at ASC LIMIT
..." fait revenir sans cesse les mêmes parents irrécupérables en tête et peut
provoquer la famine; changez la stratégie dans src/db.rs et la logique de
run_once (src/artwork_jobs.rs) pour ajouter et utiliser une métrique de
retry/backoff persistée (par ex. colonnes retry_count et next_retry_at ou
last_failed_at) : à l'échec incrémentez retry_count et mettez next_retry_at =
now() + backoff(retry_count) dans la transaction d'échec, et modifiez la requête
de sélection pour trier par next_retry_at ASC (ou COALESCE(next_retry_at,
a.created_at)) au lieu de created_at seul afin d'éviter de resservir
indéfiniment les mêmes échecs; ainsi run_once lira parents dont next_retry_at
est atteint et les parents irrécupérables restent pénalisés jusqu'à expiration
du backoff.

In `@tests/artwork_scanner.rs`:
- Around line 37-154: Add an extra test that mirrors the partial-cache happy
path but starts with a parent row and zero metadata_artwork_variant rows while
the parent bytes are present in storage: create a tempdir-backed ArtworkStorage
via ArtworkStorage::local, write the source bytes (use synth_jpeg and
blake3::hash to compute parent_hash), insert the metadata_artwork row, ensure no
variant rows exist for parent_hash, call artwork_jobs::run_once(&pool, &storage,
50) and assert it returns 1 (or 2 if the scanner reports both variants) and that
both 'thumb' and 'preview' rows now exist in metadata_artwork_variant; reference
the existing tests run_once_repairs_a_partial_cache and
run_once_skips_parents_with_missing_bytes for structure and reuse the same query
patterns against metadata_artwork and metadata_artwork_variant to validate pre-
and post-conditions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: dd4cea07-49a1-412e-81c8-996f28a149f3

📥 Commits

Reviewing files that changed from the base of the PR and between f177ab1 and 53ad0a0.

📒 Files selected for processing (9)
  • .env.example
  • CLAUDE.md
  • src/artwork_jobs.rs
  • src/config.rs
  • src/db.rs
  • src/lib.rs
  • src/main.rs
  • tests/artwork_scanner.rs
  • tests/support.rs

Comment thread src/db.rs Outdated
Comment thread tests/artwork_scanner.rs
Three updates in one round:

1. Starvation guard on the scanner (CR finding). Without it, an
   irrecoverable parent (e.g. one whose source bytes were lost from
   object_store) sat at the head of `ORDER BY created_at ASC` every
   cycle and dominated the batch. Add a `last_repair_failure_at`
   column on `metadata_artwork`, stamp it on every failed repair,
   and order the scanner query by `last_repair_failure_at NULLS
   FIRST, created_at ASC` with a 1-hour backoff filter. Untouched
   parents always lead; freshly-failed ones recede until the
   cooldown expires. Permanently broken rows stop blocking the
   queue.

2. New integration test `run_once_repairs_parent_with_no_variants`
   for the "parent landed but zero variants made it" edge case (CR
   finding). Mirrors the existing single-variant test structure;
   asserts the scanner regenerates the full set from scratch.

3. Extended `run_once_skips_parents_with_missing_bytes` to assert
   the backoff stamp lands on the failed row AND that a follow-up
   `list_partial_parents` call doesn't re-surface the same hash
   inside the 1-hour window — the property the starvation guard is
   supposed to deliver.

Bumps that fit cleanly without an API audit:
- uuid 1.23.1 → 1.23.2 (patch, error message tweaks)
- tokio-tungstenite 0.27 → 0.29 (dev-dep, sync WS tests only)

Skipped (production-critical, deserve dedicated dependabot PRs):
- rand 0.8 → 0.10 (Alphanumeric::sample_string API may have moved)
- hmac 0.12 → 0.13 (stream_token signing)
- sha2 0.10 → 0.11 (stream_token + sha-256 hashing)

Signed-off-by: InstaZDLL <github.105mh@8shield.net>
@InstaZDLL

Copy link
Copy Markdown
Owner Author

@coderabbitai trois corrections pushées (c1e9666) :

1. Starvation guard (finding 1) — addressed via migration 20260608000000_artwork_repair_backoff.sql qui ajoute metadata_artwork.last_repair_failure_at TIMESTAMPTZ + partial index sur les rows ayant un timestamp. La query list_partial_parents gagne :

  • WHERE last_repair_failure_at IS NULL OR < backoff_cutoff
  • ORDER BY last_repair_failure_at ASC NULLS FIRST, created_at ASC

Cooldown 1h. À chaque échec de repair, le scanner appelle db::artwork::mark_repair_failure(hash) qui UPDATE last_repair_failure_at = NOW(). Untouched parents always lead via NULLS FIRST ; freshly-failed parents recèdent jusqu'à expiration. La mark elle-même est best-effort (warn + continue si DB hiccup).

2. Zero-variant test (finding 2)run_once_repairs_parent_with_no_variants ajouté : seed parent row + parent bytes en storage, zéro variant row → run_once → assert 2 variants présentes (thumb + preview) après. Mirror du test single-variant pour l'edge case.

3. Backoff coverage — étendu run_once_skips_parents_with_missing_bytes pour asserter que la stamp landed ET qu'un second list_partial_parents call immédiat ne re-surface pas le hash (la property que la guarde donne).

Bumps appliqués pendant que j'y étais :

  • uuid 1.23.1 → 1.23.2 (patch, error msg)
  • tokio-tungstenite 0.27 → 0.29 (dev-dep, tests sync only)

Skipped pour PRs dedicated : rand 0.8→0.10 (Alphanumeric API), hmac 0.12→0.13, sha2 0.10→0.11 (tous prod-critical sur stream_token/share_token).

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Look at the new migration file
fd '20260608000000' --exec cat {}

# Look at the updated db.rs artwork module
rg -n 'list_partial_parents|mark_repair_failure|last_repair_failure_at|backoff' src/db.rs -A5 -B2

# Look at artwork_jobs.rs for backoff/mark_repair_failure usage
rg -n 'mark_repair_failure|backoff|last_repair_failure_at|run_once|repair_one' src/artwork_jobs.rs -A5 -B2

# Look at the test file
cat tests/artwork_scanner.rs

Length of output: 16966


🏁 Script executed:

#!/bin/bash
# Verify REPAIR_BACKOFF constant and any other new constants
rg -n 'REPAIR_BACKOFF|EXPECTED_VARIANT_COUNT|DEFAULT_SCAN_INTERVAL|DEFAULT_BATCH_SIZE' src/artwork_jobs.rs -A2

# Check the full artwork_jobs.rs for any subtle issues
cat src/artwork_jobs.rs

Length of output: 11014



@InstaZDLL les trois corrections sont correctes. Voici le verdict point par point :

Finding 1 — Starvation guard ✅

  • La migration, le filtre SQL et le tri NULLS FIRST, created_at ASC sont tous cohérents.
  • mark_repair_failure en best-effort (warn + continue) est le bon compromis — un DB hiccup sur le stamp coûte une retry immédiate, pas un crash.
  • chrono::Duration::from_std(REPAIR_BACKOFF).expect(...) : 3 600 s rentre largement dans un chrono::Duration, le .expect est safe.

Observation mineure (non bloquante) : le partial index WHERE last_repair_failure_at IS NOT NULL couvre bien le chemin de filtre/tri des rows en backoff. Les rows NULL (jamais tentées) tombent en dehors de l'index et sont triées par seq-scan — c'est le chemin chaud (elles sont prioritaires). À l'échelle attendue du service c'est négligeable, mais si le catalogue grandit significativement, un index complet sur la colonne (sans WHERE) serait plus efficace pour le ORDER BY NULLS FIRST.

Finding 2 — Zero-variant test ✅

Le test run_once_repairs_parent_with_no_variants est le mirror exact du test single-variant. Il couvre l'edge case proprement (spot-check des deux variants par nom après repair).

Finding 3 — Backoff coverage ✅

La vérification à deux temps dans run_once_skips_parents_with_missing_bytes est solide :

  1. Assert que last_repair_failure_at IS NOT NULL après le premier cycle.
  2. Appel direct à list_partial_parents avec Utc::now() - 3600s comme backoff_cutoff — le phantom_hash fraîchement stampé échoue la condition < $3 et est bien exclu. Le skew horloge app/DB est de l'ordre de la milliseconde face à une fenêtre d'1h, donc aucun risque de flakiness.

Bumps de dépendances : uuid patch et tokio-tungstenite dev-dep uniquement — aucun risque.

Tout est bon, le PR est prêt à merger.

[approve]


✏️ Learnings added
Learnt from: InstaZDLL
URL: https://github.com/InstaZDLL/waveflow-server/pull/31

Timestamp: 2026-06-06T19:20:48.123Z
Learning: Dans `src/artwork_jobs.rs` (waveflow-server, Rust/sqlx/Postgres), `EXPECTED_VARIANT_COUNT: i64 = 2` est maintenu en lockstep manuel avec `crate::artwork_pipeline::VariantKind` (`thumb` + `preview`). Toute addition d'un nouveau `VariantKind` dans le pipeline doit s'accompagner d'un bump de cette constante, sinon le scanner considère 2 variants = complet et n'héale pas les lignes partielles.

Learnt from: InstaZDLL
URL: https://github.com/InstaZDLL/waveflow-server/pull/31

Timestamp: 2026-06-06T19:20:48.123Z
Learning: Dans `src/artwork_jobs.rs` (waveflow-server), le scanner background artwork utilise `REPAIR_BACKOFF = Duration::from_secs(3600)` (1h) comme cooldown après échec de repair. La colonne `metadata_artwork.last_repair_failure_at TIMESTAMPTZ` (ajoutée par la migration `20260608000000_artwork_repair_backoff.sql`) est stampée via `db::artwork::mark_repair_failure(pool, hash)` en best-effort (warn + continue si DB hiccup). `list_partial_parents` filtre avec `IS NULL OR < backoff_cutoff` et ordonne `NULLS FIRST, created_at ASC`.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


🧠 Learnings used
Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 16
File: src/db.rs:144-146
Timestamp: 2026-05-31T00:40:18.480Z
Learning: Dans `src/db.rs` (waveflow-server, Rust/sqlx/Postgres), la fonction `find_or_provision_by_external_id` utilise un pattern "read-first, write-on-miss" plutôt qu'un CTE `INSERT ... ON CONFLICT DO NOTHING RETURNING id UNION ALL SELECT id`. Raison : sous READ COMMITTED, les deux branches d'un CTE partagent le même snapshot statement — si une tx concurrente insère la même ligne entre l'acquisition du snapshot et notre INSERT, DO NOTHING se déclenche sans RETURNING, et le SELECT du UNION ALL ne voit pas encore la row commitée → 0 lignes → erreur 500. Le pattern retenu (SELECT optionnel → return si trouvé; sinon INSERT ON CONFLICT DO UPDATE SET external_id = EXCLUDED.external_id RETURNING id) évite ce cas en garantissant que RETURNING se déclenche toujours sur le miss path, même en cas de race concurrente. Référence : Erwin Brandstetter sur la race condition du pattern CTE UPSERT.

Learnt from: CR
Repo: InstaZDLL/waveflow-server PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-06-06T18:53:32.905Z
Learning: Applies to src/{api/**/artwork,artwork_pipeline}.rs : Artwork cache (Phase 1.h). `POST /api/v1/artwork` accepts raw image bytes (content-type ∈ {image/jpeg, image/png, image/webp}, 4 MiB cap), hashes server-side, runs through `artwork_pipeline::generate_variants` to produce thumb (≤128px) + preview (≤480px) variants. Idempotent: re-uploads of same bytes skip storage write and pipeline run.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@migrations/20260608000000_artwork_repair_backoff.sql`:
- Around line 18-28: Change the new column last_repair_failure_at from
TIMESTAMPTZ to BIGINT (epoch-millis) and recreate the index
idx_metadata_artwork_repair_pending to target that BIGINT column (keeping the
WHERE last_repair_failure_at IS NOT NULL predicate); then update the Rust code
paths that consume/produce this value (list_partial_parents and
mark_repair_failure) to pass/expect epoch-millis BIGINT (not a timestamptz) so
the DB schema and the waveflow-core SQLite migrations remain parity.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 76aa6290-55d1-4f2c-ab7a-a6dc7b97369d

📥 Commits

Reviewing files that changed from the base of the PR and between 53ad0a0 and c1e9666.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock, !Cargo.lock, !*.lock
📒 Files selected for processing (5)
  • Cargo.toml
  • migrations/20260608000000_artwork_repair_backoff.sql
  • src/artwork_jobs.rs
  • src/db.rs
  • tests/artwork_scanner.rs

Comment thread migrations/20260608000000_artwork_repair_backoff.sql
CodeRabbit flagged the column type violates the repo's documented
schema convention. CLAUDE.md says: "epoch-millis BIGINT for
timestamps" + "Postgres tables must mirror shapes in waveflow-core's
SQLite migrations for repository trait parity" — TIMESTAMPTZ breaks
both.

Switch the new column to BIGINT and propagate the epoch-millis
contract through the API:
- Migration: `last_repair_failure_at BIGINT`, partial index
  unchanged.
- `db::artwork::list_partial_parents` takes `backoff_cutoff_ms: i64`
  instead of `chrono::DateTime<Utc>`.
- `db::artwork::mark_repair_failure` takes `now_ms: i64` instead of
  using SQL `NOW()` — caller mints the timestamp, which also keeps
  the scanner clock and the DB clock decoupled for a future
  distributed deploy.
- `artwork_jobs::run_once` mints `chrono::Utc::now().timestamp_millis()`
  once per cycle and passes it down.
- Integration test reads the stamp as `Option<i64>` and computes
  the cutoff via `timestamp_millis() - 3_600_000`.

Signed-off-by: InstaZDLL <github.105mh@8shield.net>
@InstaZDLL

Copy link
Copy Markdown
Owner Author

@coderabbitai bien vu, finding valide — le contrat CLAUDE.md est explicite : "epoch-millis BIGINT for timestamps" + "Postgres tables must mirror shapes in waveflow-core's SQLite migrations". TIMESTAMPTZ cassait les deux.

Switched (la migration n'était pas mergée donc je l'ai modifiée in-place) :

  • Migration : last_repair_failure_at BIGINT, partial index inchangé.
  • list_partial_parents : prend backoff_cutoff_ms: i64 au lieu de chrono::DateTime<Utc>.
  • mark_repair_failure : prend now_ms: i64 au lieu d'utiliser SQL NOW() — caller mint le timestamp, ce qui découple aussi clock scanner ↔ clock DB pour un futur deploy distribué.
  • run_once : mint chrono::Utc::now().timestamp_millis() une fois par cycle, passe le scalar partout.
  • Test : lit le stamp comme Option<i64>, cutoff via timestamp_millis() - 3_600_000.

Push : d99919f.

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the migration change
fd '20260608000000' --exec cat {}

# Check db.rs for updated signatures
rg -n 'list_partial_parents|mark_repair_failure|backoff_cutoff|now_ms|last_repair_failure_at' src/db.rs -A8 -B2

# Check artwork_jobs.rs for how run_once mints the timestamp
rg -n 'timestamp_millis|backoff_cutoff|mark_repair_failure|run_once|REPAIR_BACKOFF' src/artwork_jobs.rs -A5 -B2

# Check test assertions
rg -n 'last_repair_failure_at|backoff_cutoff|timestamp_millis|now_ms|Option<i64>' tests/artwork_scanner.rs -A5 -B2

Length of output: 9240



@InstaZDLL tout est cohérent sur d99919f. Vérifications :

  • Migration : BIGINT ✅, respecte le contrat CLAUDE.md epoch-millis.
  • list_partial_parents : backoff_cutoff_ms: i64, filtre SQL OR a.last_repair_failure_at < $3 ✅.
  • mark_repair_failure : now_ms: i64 passé par le caller ✅ — découplage clock scanner/DB propre.
  • run_once : mint unique en tête de cycle, saturating_sub sur i64 pour le cutoff ✅ — bonne défense même si en pratique un epoch-millis actuel ne peut pas underflower.
  • Tests : Option<i64> + timestamp_millis() - 3_600_000 ✅.

Rien à corriger. PR prêt à merger.

[approve]


✏️ Learnings added
Learnt from: InstaZDLL
URL: https://github.com/InstaZDLL/waveflow-server/pull/31

Timestamp: 2026-06-06T19:46:50.736Z
Learning: Dans `src/artwork_jobs.rs` et la migration `20260608000000_artwork_repair_backoff.sql` (waveflow-server), la colonne `metadata_artwork.last_repair_failure_at` est `BIGINT` epoch-millis (pas `TIMESTAMPTZ`), conforme au contrat CLAUDE.md "epoch-millis BIGINT for timestamps" + compatibilité SQLite mirror. `list_partial_parents` prend `backoff_cutoff_ms: i64`. `mark_repair_failure` prend `now_ms: i64` (caller-minted, clock DB découpée). `run_once` mint `chrono::Utc::now().timestamp_millis()` une fois par cycle et utilise `saturating_sub` pour le cutoff. `REPAIR_BACKOFF = 3600s` → 3_600_000 ms.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


⛔ Learnings removed
Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 0
File: :0-0
Timestamp: 2026-06-06T19:20:48.123Z
Learning: Dans `src/artwork_jobs.rs` (waveflow-server), le scanner background artwork utilise `REPAIR_BACKOFF = Duration::from_secs(3600)` (1h) comme cooldown après échec de repair. La colonne `metadata_artwork.last_repair_failure_at TIMESTAMPTZ` (ajoutée par la migration `20260608000000_artwork_repair_backoff.sql`) est stampée via `db::artwork::mark_repair_failure(pool, hash)` en best-effort (warn + continue si DB hiccup). `list_partial_parents` filtre avec `IS NULL OR < backoff_cutoff` et ordonne `NULLS FIRST, created_at ASC`.

🧠 Learnings used
Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 0
File: :0-0
Timestamp: 2026-06-06T19:20:48.123Z
Learning: Dans `src/artwork_jobs.rs` (waveflow-server), le scanner background artwork utilise `REPAIR_BACKOFF = Duration::from_secs(3600)` (1h) comme cooldown après échec de repair. La colonne `metadata_artwork.last_repair_failure_at TIMESTAMPTZ` (ajoutée par la migration `20260608000000_artwork_repair_backoff.sql`) est stampée via `db::artwork::mark_repair_failure(pool, hash)` en best-effort (warn + continue si DB hiccup). `list_partial_parents` filtre avec `IS NULL OR < backoff_cutoff` et ordonne `NULLS FIRST, created_at ASC`.

Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 16
File: src/db.rs:144-146
Timestamp: 2026-05-31T00:40:18.480Z
Learning: Dans `src/db.rs` (waveflow-server, Rust/sqlx/Postgres), la fonction `find_or_provision_by_external_id` utilise un pattern "read-first, write-on-miss" plutôt qu'un CTE `INSERT ... ON CONFLICT DO NOTHING RETURNING id UNION ALL SELECT id`. Raison : sous READ COMMITTED, les deux branches d'un CTE partagent le même snapshot statement — si une tx concurrente insère la même ligne entre l'acquisition du snapshot et notre INSERT, DO NOTHING se déclenche sans RETURNING, et le SELECT du UNION ALL ne voit pas encore la row commitée → 0 lignes → erreur 500. Le pattern retenu (SELECT optionnel → return si trouvé; sinon INSERT ON CONFLICT DO UPDATE SET external_id = EXCLUDED.external_id RETURNING id) évite ce cas en garantissant que RETURNING se déclenche toujours sur le miss path, même en cas de race concurrente. Référence : Erwin Brandstetter sur la race condition du pattern CTE UPSERT.

Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 0
File: :0-0
Timestamp: 2026-06-06T19:20:48.123Z
Learning: Dans `src/artwork_jobs.rs` (waveflow-server, Rust/sqlx/Postgres), `EXPECTED_VARIANT_COUNT: i64 = 2` est maintenu en lockstep manuel avec `crate::artwork_pipeline::VariantKind` (`thumb` + `preview`). Toute addition d'un nouveau `VariantKind` dans le pipeline doit s'accompagner d'un bump de cette constante, sinon le scanner considère 2 variants = complet et n'héale pas les lignes partielles.

Learnt from: CR
Repo: InstaZDLL/waveflow-server PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-06-06T18:53:32.905Z
Learning: Applies to migrations/*.sql : Schema parity with the desktop SQLite migrations. Postgres tables mirror the shapes in the desktop repo's `src-tauri/migrations/app/` so `PostgresProfileRepository` and `SqliteProfileRepository` satisfy the same trait against identical rows. Keep types compatible (e.g. `BIGSERIAL` ↔ SQLite `INTEGER PK`, epoch-millis `BIGINT` for timestamps).

Learnt from: CR
Repo: InstaZDLL/waveflow-server PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-06-06T18:53:32.905Z
Learning: Applies to src/{api/**/artwork,artwork_pipeline}.rs : Artwork cache (Phase 1.h). `POST /api/v1/artwork` accepts raw image bytes (content-type ∈ {image/jpeg, image/png, image/webp}, 4 MiB cap), hashes server-side, runs through `artwork_pipeline::generate_variants` to produce thumb (≤128px) + preview (≤480px) variants. Idempotent: re-uploads of same bytes skip storage write and pipeline run.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@migrations/20260608000000_artwork_repair_backoff.sql`:
- Around line 18-25: The migration file
migrations/20260608000000_artwork_repair_backoff.sql has a future timestamp
(2026-06-08) — rename the file so its leading timestamp uses the current date
(e.g. 20260606) to avoid reserving a future migration order; keep the rest of
the file (ALTER TABLE metadata_artwork ADD COLUMN last_repair_failure_at BIGINT)
unchanged, ensure the new filename is unique among migrations, and perform a git
mv/commit so references and history stay consistent.

In `@src/artwork_jobs.rs`:
- Around line 141-145: The scanner breaks if batch_size > i64::MAX because
run_once casts a usize batch_size to i64 and passes it to list_partial_parents
(used as SQL LIMIT); clamp or validate WAVEFLOW_ARTWORK_SCANNER_BATCH_SIZE to
i64::MAX when reading the config (or perform a checked conversion in run_once)
so the value never wraps negative; update the code around run_once where it
calls list_partial_parents (and/or the config reader that produces batch_size)
to use i64::try_from or clamp_to(i64::MAX) and return/log a clear error if the
configured batch is > i64::MAX.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 56207b6f-864a-4039-b50c-250dd9aa6ce1

📥 Commits

Reviewing files that changed from the base of the PR and between c1e9666 and d99919f.

📒 Files selected for processing (4)
  • migrations/20260608000000_artwork_repair_backoff.sql
  • src/artwork_jobs.rs
  • src/db.rs
  • tests/artwork_scanner.rs

Comment thread migrations/20260608000000_artwork_repair_backoff.sql
Comment thread src/artwork_jobs.rs
CodeRabbit flagged a `usize → i64` cast in run_once that could wrap
for a hostile `WAVEFLOW_ARTWORK_SCANNER_BATCH_SIZE` value above
`i64::MAX`. The hot path passes the value straight to a SQL `LIMIT`
binding (sqlx serialises as i64), so a wrap would either error
inside Postgres or skew the query semantics.

Clamp at boot via `.min(i64::MAX as usize)` so the hot loop stays a
plain `as i64` cast. Defaulted value (50) is unaffected; only a
deliberately-set astronomical env value triggers the clamp.

Skipping the second finding (rename migration to today's date,
2026-06-06): PR #30 already merged `20260607000000_metadata_artwork_variant.sql`,
so renaming this file's prefix to `20260606xxxxxx` would put it
BEFORE the previously-applied migration in sqlx's ordering. The
"future date" pattern is already established in the repo (the
20260607 itself); enforcing today's wall-clock here would only be
cosmetic and re-checksum a row that nobody's applied yet anyway.

Signed-off-by: InstaZDLL <github.105mh@8shield.net>
@InstaZDLL

Copy link
Copy Markdown
Owner Author

@coderabbitai deux findings vérifiés :

1. batch_size overflow → ✅ applied (b3a5235)

Valide. usize::MAX > i64::MAX sur 64-bit, et la valeur passe en LIMIT $ (sqlx bind i64). Clamp ajouté au boot dans Config::from_env :

.min(i64::MAX as usize)

Le hot path run_once reste un as i64 plain ; seul un env value délibérément astronomique trigger le clamp.

2. Migration filename "future date" → skipped (brief reason)

La file s'appelle 20260608000000 (2026-06-08, deux jours "dans le futur" vs today 2026-06-06). Renommer en 20260606xxxxxx la mettrait AVANT la 20260607000000_metadata_artwork_variant.sql déjà mergée dans PR #30 — sqlx ordonne par leading timestamp, donc ça inverserait l'ordre d'application pour les fresh installs. Le pattern "future date" est déjà établi par la 20260607 elle-même ; enforcer today's wall-clock ici serait cosmetique seulement (la migration n'a pas été appliquée par personne donc pas de checksum à protéger, mais la breaking change d'ordering est réelle).

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the batch_size clamp in Config::from_env
rg -n 'i64::MAX|batch_size' src/config.rs -A3 -B3

Length of output: 1421



@InstaZDLL les deux points sont corrects :

Finding 1 — batch_size clamp ✅
Clamp bien positionné à boot (ligne 257 de config.rs), avec commentaire inline qui documente exactement le wrap risk usize::MAX > i64::MAX. Le hot path run_once garde son as i64 sans guard — c'est la bonne partition.

Finding 2 — Migration filename ✅ (skipped, justification valide)
La dépendance 20260607000000_metadata_artwork_variant.sql20260608000000_artwork_repair_backoff.sql impose cet ordre. Renommer en 20260606... inverserait l'ordre d'application sur fresh install (ALTER sur table non encore créée). Le skip est correct.

PR prêt à merger.

[approve]


🧠 Learnings used
Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 0
File: :0-0
Timestamp: 2026-06-06T19:46:50.736Z
Learning: Dans `src/artwork_jobs.rs` et la migration `20260608000000_artwork_repair_backoff.sql` (waveflow-server), la colonne `metadata_artwork.last_repair_failure_at` est `BIGINT` epoch-millis (pas `TIMESTAMPTZ`), conforme au contrat CLAUDE.md "epoch-millis BIGINT for timestamps" + compatibilité SQLite mirror. `list_partial_parents` prend `backoff_cutoff_ms: i64`. `mark_repair_failure` prend `now_ms: i64` (caller-minted, clock DB découpée). `run_once` mint `chrono::Utc::now().timestamp_millis()` une fois par cycle et utilise `saturating_sub` pour le cutoff. `REPAIR_BACKOFF = 3600s` → 3_600_000 ms.

Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 16
File: src/db.rs:144-146
Timestamp: 2026-05-31T00:40:18.480Z
Learning: Dans `src/db.rs` (waveflow-server, Rust/sqlx/Postgres), la fonction `find_or_provision_by_external_id` utilise un pattern "read-first, write-on-miss" plutôt qu'un CTE `INSERT ... ON CONFLICT DO NOTHING RETURNING id UNION ALL SELECT id`. Raison : sous READ COMMITTED, les deux branches d'un CTE partagent le même snapshot statement — si une tx concurrente insère la même ligne entre l'acquisition du snapshot et notre INSERT, DO NOTHING se déclenche sans RETURNING, et le SELECT du UNION ALL ne voit pas encore la row commitée → 0 lignes → erreur 500. Le pattern retenu (SELECT optionnel → return si trouvé; sinon INSERT ON CONFLICT DO UPDATE SET external_id = EXCLUDED.external_id RETURNING id) évite ce cas en garantissant que RETURNING se déclenche toujours sur le miss path, même en cas de race concurrente. Référence : Erwin Brandstetter sur la race condition du pattern CTE UPSERT.

Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 0
File: :0-0
Timestamp: 2026-06-06T19:20:48.123Z
Learning: Dans `src/artwork_jobs.rs` (waveflow-server, Rust/sqlx/Postgres), `EXPECTED_VARIANT_COUNT: i64 = 2` est maintenu en lockstep manuel avec `crate::artwork_pipeline::VariantKind` (`thumb` + `preview`). Toute addition d'un nouveau `VariantKind` dans le pipeline doit s'accompagner d'un bump de cette constante, sinon le scanner considère 2 variants = complet et n'héale pas les lignes partielles.

Learnt from: InstaZDLL
Repo: InstaZDLL/waveflow-server PR: 6
File: src/api/profiles.rs:254-273
Timestamp: 2026-05-30T03:28:04.286Z
Learning: In `src/api/profiles.rs`, the non-atomic rename_then_read race (rename_for_user followed by get_for_user) is a known issue intentionally deferred. The proper fix is to add `UPDATE ... RETURNING *` to `rename_for_user` in waveflow-core, batched with the same refactor for `insert_for_user` and `touch_last_used`, rather than a piecemeal server-side workaround.

Learnt from: CR
Repo: InstaZDLL/waveflow-server PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-06-06T18:53:32.905Z
Learning: Applies to migrations/*.sql : Migrations are immutable once merged. They're embedded at compile time via `sqlx::migrate!("./migrations")`. Schema changes = a new dated migration file (`YYYYMMDDHHMMSS_name.sql`). Boot applies pending migrations *before* opening the listener.

Learnt from: CR
Repo: InstaZDLL/waveflow-server PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-06-06T18:53:32.905Z
Learning: Applies to src/{api/**/artwork,artwork_pipeline}.rs : Artwork cache (Phase 1.h). `POST /api/v1/artwork` accepts raw image bytes (content-type ∈ {image/jpeg, image/png, image/webp}, 4 MiB cap), hashes server-side, runs through `artwork_pipeline::generate_variants` to produce thumb (≤128px) + preview (≤480px) variants. Idempotent: re-uploads of same bytes skip storage write and pipeline run.

@InstaZDLL
InstaZDLL merged commit b2996f8 into main Jun 6, 2026
8 checks passed
@InstaZDLL
InstaZDLL deleted the feat/1-i-1-artwork-background-scanner branch June 6, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant