WS-ART-001-02C3: add artifact recovery attempt chain

[Abiorh001]

PR Trust Bundle: WS-ART-001-02C3

Chunk

WS-ART-001-02C3 — Recovery Attempt And Idempotency Chain (L1)

Goal And Human-Approved Intent

Add the durable, read-only provider-observation recovery envelope and exact
source-job to retry-job chain. The signed loop-memory start authorized this ART
chunk to run concurrently with AUTH; it does not activate Operator routes.

What Changed And Why

• added migration 0032 for recovery envelopes and immutable verification
  lineage;
• added nullable task context so the same chain supports guide and task-backed
  artifacts;
• added exact replay, lifetime source ownership, concurrent winner recovery,
  linear retry ancestry, and atomic terminal finalization;
• added a typed Operator authority seam whose production-safe implementation
  denies until the later AUTH-owned activation;
• required fresh actor and exact authority revalidation on creation and every
  replay, with bounded authorization evidence in the initiation audit;
• added focused migration, concurrency, guide, authorization, fencing, and
  terminal-outcome tests.

Design And Alternatives

The verification job remains the sole executable lease owner. Recovery is an
envelope, not a second worker lease. PostgreSQL uniqueness and custody triggers
enforce one lifetime source owner and immediate-parent chaining. Provider
mutation replay and a task-only recovery abstraction were rejected because
both violate the initiative contract.

Scope And Product Behavior

No routes, provider mutation, guide/task/submission lifecycle transitions,
review decisions, payment, reputation, dependencies, CI workflow changes, or
coverage reductions are included. Recovery remains infrastructure state and
does not introduce product decisions beyond accept, needs_revision, and
reject.

Acceptance Evidence

• exact and concurrent replay preserve one envelope, retry job, and initiation
  audit;
• changed/lifetime reuse conflicts without new recovery ownership;
• only exhausted terminal provider_unavailable work is recoverable;
• taskless guide recovery is representable and replayable;
• denied creation and denied replay return no identifiers and add no rows;
• success and every failed terminal outcome finalize envelope and audit under
  the verification transaction;
• terminal authority drift writes no terminal recovery facts;
• exhausted retries form only the next linear chain link.

Tests And CI Integrity

Local focused evidence:

• ruff check app tests alembic/versions/0032_artifact_recovery_attempts.py;
• focused recovery tests, including all original five, guide/deny replay,
  authority drift, and sampled failed outcome;
• migration upgrade/downgrade and recovery-schema tests;
• python3 scripts/check_stale_artifact_contracts.py;
• python3 scripts/test_agent_gates.py — 89 passed;
• git diff --check.

The expensive full backend suite, remaining parameter combinations, global
78% coverage, and cumulative artifact 90% coverage are intentionally delegated
to the existing sharded GitHub Actions. No CI or package-script file changed.
No tests were removed, skipped, or weakened.

Reviewer Results

• senior engineering: PASS WITH LOW RISKS;
• architecture: PASS WITH LOW RISKS;
• QA/test: PASS;
• security/auth: PASS WITH LOW RISKS;
• product/ops: PASS WITH LOW RISKS;
• reuse/dedup: PASS WITH LOW RISKS;
• CI integrity: PASS;
• test delta: PASS;
• docs: PASS.

External review and hosted CI are pending publication of the PR.

Remaining Risks And Follow-Up

Low risks: legacy terminal audit metadata describes the verifier imperfectly,
the audit builder has private cross-class ownership, and human-proof validation
has small domain-specific duplication. The later typed Operator audit surface
may clean these up. Successor WS-ART-001-02D owns Operator routes and real AUTH
activation; it must not start automatically before this chunk merges and the
signed successor event is approved.

Human Review Focus And Merge Ownership

Verify that no source can own two recoveries, no replay can bypass fresh
authority, and recovery stays separate from provider mutation and product
lifecycle state. A human owns the merge decision; Codex must not merge without
explicit approval for this PR.

Summary by CodeRabbit

• New Features

  • Added a controlled artifact verification recovery flow that supports safe retries after provider-unavailable failures.
  • Added idempotent replay handling to prevent duplicate recovery operations, including concurrent requests.
  • Added support for recovery of artifacts without an associated task.

• Bug Fixes

  • Improved handling of terminal verification outcomes, including integrity mismatches and failed retries.
  • Strengthened auditability and consistency of recovery results.

• Tests

  • Added comprehensive coverage for retries, authorization, concurrency, lineage, and database migration behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 27ba41c1-694f-4125-af68-4d2bd8e56fee

📥 Commits

Reviewing files that changed from the base of the PR and between 1473f7a and 7f8bb4f.

📒 Files selected for processing (15)

• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C3-external-review-response.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C3-internal-review-evidence.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C3-pr-trust-bundle.md
• .agent-loop/merge-intents/WS-ART-001-02C3.json
• backend/alembic/versions/0032_artifact_recovery_attempts.py
• backend/app/db/models.py
• backend/app/interfaces/artifact_operations.py
• backend/app/modules/artifacts/models.py
• backend/app/modules/artifacts/repository.py
• backend/app/modules/artifacts/schemas.py
• backend/app/modules/artifacts/service.py
• backend/tests/test_alembic.py
• backend/tests/test_artifact_recovery.py

---

📝 Walkthrough

Walkthrough

Introduces immutable recovery-attempt storage, verification retry lineage, authorization-bound recovery creation, idempotent replay, terminal recovery updates, audit events, migration safeguards, and PostgreSQL-backed integration tests.

Changes

Artifact recovery and retry lineage

Layer / File(s)	Summary
Recovery persistence and contracts backend/alembic/versions/0032_artifact_recovery_attempts.py, backend/app/modules/artifacts/models.py, backend/app/modules/artifacts/schemas.py, backend/app/interfaces/artifact_operations.py	Adds recovery-attempt tables, verification-job parent lineage, custody triggers, status and terminal invariants, nullable task context, and authorization/result types.
Recovery creation and terminalization backend/app/modules/artifacts/service.py, backend/app/modules/artifacts/repository.py, backend/app/db/models.py	Adds locked recovery lookups, idempotent creation and replay, authorization revalidation, initiation/completion audits, retry lineage, and terminal recovery finalization.
Schema and behavior validation backend/tests/test_artifact_recovery.py, backend/tests/test_alembic.py, .agent-loop/..., .agent-loop/merge-intents/...	Tests migration reversibility, denial and conflict paths, terminal outcomes, concurrent replay, linear retry chains, and records review and merge metadata.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant ArtifactRecoveryService
  participant ArtifactRepository
  participant PostgreSQL
  participant VerificationService
  Caller->>ArtifactRecoveryService: submit recovery request
  ArtifactRecoveryService->>ArtifactRepository: lock source and recovery rows
  ArtifactRepository->>PostgreSQL: create or replay recovery attempt
  ArtifactRecoveryService->>PostgreSQL: persist retry job and initiation audit
  VerificationService->>ArtifactRecoveryService: report retry terminal outcome
  ArtifactRecoveryService->>PostgreSQL: persist completion audit and terminal state

Loading

Possibly related PRs

• Flow-Research/workstream#141: Introduces the recovery request contract updated in this PR.
• Flow-Research/workstream#159: Introduces verification-job fencing and receipt structures extended here.

Suggested reviewers: abiorh-claw

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description covers the feature, but it omits required template sections like Commands Run, Result Summary, Acceptance Criteria Proof, Test Delta, and the reviewer table metadata.	Add the missing template sections and required metadata, especially Evidence, Acceptance Criteria Proof, Test Delta, External Review, CI And Gate Integrity, and the full reviewer-results table.
Docstring Coverage	⚠️ Warning	Docstring coverage is 31.48% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: adding the artifact recovery attempt chain.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/ws-art-001-02c3-recovery-attempt-idempotency

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}