WS-ART-001-02C2: verification publication and fencing

[Abiorh001]

PR Trust Bundle: WS-ART-001-02C2

Chunk

WS-ART-001-02C2 - Verification Publication And Fencing

Reviewed implementation SHA: e59a6dfc977fa63ad7177ab9adb8338333aa1daf

Trusted base: 42a89b2d

Merge intent: .agent-loop/merge-intents/WS-ART-001-02C2.json

Goal And Human-Approved Intent

Resolve committed artifact puts, publish bounded pending work, verify complete
provider reads, and fence every terminal transition without activating product
callers, recovery, Operator routes, or background write replay.

What Changed And Why

• Added caller-only provider write execution and read-only ambiguous-result
  observation so acknowledgement loss cannot cause a second write.
• Added durable verification jobs and separate operation, observation, and
  verification receipts for auditable outcomes.
• Added PostgreSQL executor UUID/generation leases and same-transaction
  authority revalidation so stale or revoked work cannot commit terminal facts.
• Added bounded post-commit publication scanning and total complete-read
  deadlines.
• Integrated current main and linearized migrations as outbox 0029 followed by
  artifact verification 0030, eliminating the duplicate-head collision.

Design And Alternatives

The orchestrator owns provider execution through ArtifactStore; product
services do not receive raw provider access. Background resolution observes
only and never retries writes. Production tasks remain deny-only with no Beat
schedule. Ad hoc provider factories, a second AUTH evaluator, background write
replay, and dual scanner/outbox publication were rejected.

Scope Control And Product Behavior

No task, submission, checker, review, revision, contribution, compensation,
reputation, deletion, recovery, Operator API, R2, Flow Node, or product-cutover
behavior changed. Review decisions remain accept, needs_revision, and
reject. The three ART internal actions remain planned and inactive.

Acceptance Criteria Proof

• One linear Alembic head after trusted main integration.
• Provider writes occur only in the caller-owned orchestrator path.
• Acknowledgement loss resolves through read-only observation.
• Missing, mismatch, conflict, unavailable, and verified outcomes are typed
  and fenced.
• Stale executor/generation or authority drift writes zero terminal facts.
• Verification reads have a total deadline and lease safety margin.
• Publication is bounded, duplicate-safe, post-commit, and not scheduled in
  production.
• Exactly one schema-v2 merge intent names inactive successor 02C3 and
  requires a separate explicit start.

Tests And Checks

Alembic heads: 0030_artifact_verification (single head)
Fresh isolated migration integration: 3 passed
Agent gates: 88 passed
Focused ART matrix: 342 passed, 1 transient failure, coverage 92.75% (floor 90%)
Exact failed parameter rerun: 1 passed
Paired denial matrix rerun: 2 passed
Reviewer repeated denial matrix: 12/12 passed
Verification + architecture smoke: 15 passed
Ruff: PASS
Stale wording/contracts: PASS
Markdown links: PASS
git diff --check: PASS

The transient failure is disclosed, not converted into a green aggregate. It
is non-reproduced and reviewers found no assertion weakening or fixture defect.

CI Integrity And Test Delta

No workflow, threshold, ignore, skip, xfail, retry, or failure-bypass change was
introduced. The 90 percent affected-subsystem floor and 78 percent repository
floor remain intact. Thirty-two named tests were added and none removed or
skipped.

Reviewer Results

All nine exact-SHA tracks completed: senior engineering PASS; security PASS;
product/ops PASS; reuse PASS; CI integrity PASS; docs PASS; architecture, QA,
and test delta PASS WITH LOW RISKS. No required fix remains and no reviewer
session is open.

External Review

Source	Status	Notes
GitHub Agent Gates	Pending	Must run on the published evidence head.
GitHub Backend	Pending	Must prove final isolated suite and coverage floors.
CodeRabbit	Pending	Fresh current-head review required after publication.
Human review	Pending	Only the user may approve this PR for merge.

Remaining Risks And Follow-Up

• Monitor the non-reproduced denial-test observation in hosted CI.
• AUTH-owned action activation, 02C3 recovery, 02D Operator/readiness work, and
  product cutover remain separate, explicitly started chunks.

Human Review Focus

• Can any stale generation, changed resource fact, or revoked identity persist
  terminal state?
• Can acknowledgement loss cause a second provider write?
• Does migration 0030 safely compose after main's outbox 0029?
• Are hidden tasks still deny-only, unscheduled, and unreachable by products?

Human Merge Ownership

• I can explain what changed and why.
• I understand the disclosed transient-test risk.
• GitHub CI and external review pass on the final head.
• I explicitly approve this PR for merge.

Summary by CodeRabbit

• New Features

  • Added durable artifact verification and observation tracking.
  • Improved handling of ambiguous uploads, provider failures, integrity mismatches, and verification outcomes.
  • Added safeguards against stale or conflicting background operations.
  • Added configuration options for verification timing, retries, leases, and pending-work scanning.

• Bug Fixes

  • Prevented inactive artifact actions from running or being scheduled prematurely.
  • Strengthened validation to prevent storage namespace and execution-state mismatches.

• Documentation

  • Updated artifact storage specifications and rollout status to reflect the new verification and fencing behavior.

[coderabbitai]

📝 Walkthrough

Walkthrough

This change advances ART to active chunk 02C2, adding verification jobs, immutable receipts, execution fencing, bounded scanning, deny-only internal authority, expanded orchestration and repository logic, migrations, tests, documentation, and updated AUTH/ART gate tracking.

Changes

Artifact verification and fencing

Layer / File(s)	Summary
Contracts, schema, and execution design .agent-loop/initiatives/..., backend/app/modules/artifacts/{schemas,models}.py, backend/alembic/versions/0030_artifact_verification_fencing.py, backend/app/core/config.py	Defines 02C2 behavior, authority contracts, verification models, versioned receipts, fencing constraints, migration logic, and lease/deadline settings.
Repository and orchestration flows backend/app/modules/artifacts/{repository,service}.py	Adds fenced put and verification claims, durable receipts, locking helpers, terminal transitions, namespace validation, bounded pending-work scanning, and admission integration.
Runtime task and error boundaries backend/app/workers/*, backend/app/modules/outbox/service.py, backend/tests/assertion_helpers.py	Registers deny-only artifact tasks, tightens outbox error cleanup, and inspects SQLAlchemy session state for retained secrets.
Migration, fencing, and authority validation backend/tests/test_artifact_*.py, backend/tests/test_alembic.py, backend/tests/test_assertion_helpers.py	Adds coverage for state fencing, leases, authority revalidation, provider outcomes, migration guards, architecture boundaries, deadlines, and session retention.
Initiative state and review evidence .agent-loop/**, docs/spec_artifact_storage_service.md, scripts/test_agent_gates.py	Updates 02C2 status, AUTH/ART custody tracking, review evidence, specification text, and trusted-main gate assertions.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• Flow-Research/workstream#101: Earlier immutable artifact domain foundations extended by the verification and fencing changes.
• Flow-Research/workstream#141: Introduced the ArtifactStore v2 foundations extended by this verification stage.
• Flow-Research/workstream#157: Related AUTH-09E and ART 02C2 gate-state updates.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.65% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately captures the main change: verification publication and fencing for WS-ART-001-02C2.
Description check	✅ Passed	The description covers the main template sections, intent, changes, tests, reviewer status, risks, and merge ownership, though some subsections are summarized.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/ws-art-001-02c2-verification-publication-fencing

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
@.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C2-internal-review-evidence.md:
- Around line 52-56: Update the reviewer result values in the evidence table to
use only the repository’s accepted outcome strings, replacing the invalid
conditional variants in the product/ops, CI integrity, and docs rows while
preserving their conclusions. Also refresh the Reviewed code SHA metadata to
match the latest code revision, including non-evidence changes such as
WORK_QUEUE.md.

In `@backend/app/modules/artifacts/service.py`:
- Around line 395-397: Roll back the active claim transaction before returning
"stale" when _matching_job_fence fails after lock_verification_job in the
verification-job flow. Update this post-claim mismatch branch to match the
rollback behavior in execute_committed_put and resolve_put_attempt, while
preserving the existing stale return.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 82105bf9-8e37-4f23-8c26-749168909f0f

📥 Commits

Reviewing files that changed from the base of the PR and between fe0e449 and 5f54838.

📒 Files selected for processing (29)

• .agent-loop/LOOP_STATE.md
• .agent-loop/WORK_QUEUE.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C2-external-review-response.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C2-internal-review-evidence.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C2-pr-trust-bundle.md
• .agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/CHUNK_MAP.md
• .agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md
• .agent-loop/merge-intents/WS-ART-001-02C2.json
• backend/alembic/versions/0030_artifact_verification_fencing.py
• backend/app/core/config.py
• backend/app/modules/artifacts/models.py
• backend/app/modules/artifacts/repository.py
• backend/app/modules/artifacts/schemas.py
• backend/app/modules/artifacts/service.py
• backend/app/modules/outbox/service.py
• backend/app/workers/artifacts.py
• backend/app/workers/celery_app.py
• backend/tests/assertion_helpers.py
• backend/tests/test_alembic.py
• backend/tests/test_artifact_admission.py
• backend/tests/test_artifact_architecture.py
• backend/tests/test_artifact_verification.py
• backend/tests/test_assertion_helpers.py
• docs/spec_artifact_storage_service.md
• scripts/test_agent_gates.py

[coderabbitai]

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Missing claim_transaction.rollback() on post-claim fence mismatch.

Unlike execute_committed_put (Lines 225-227) and resolve_put_attempt (Lines 305-307), this early return "stale" leaves the async with self._session.begin() block via normal exit, which commits the just-performed claim (generation+1, executor, running, lease). If this branch is ever reached, the job is orphaned in running and is only recovered on lease expiry rather than being immediately released. Roll back for consistency with the put paths.

🔒 Proposed fix

             claimed = await self._repo.lock_verification_job(str(job_id))
             if not _matching_job_fence(claimed, executor_id, candidate_generation + 1):
+                await claim_transaction.rollback()
                 return "stale"

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	claimed = await self._repo.lock_verification_job(str(job_id))
	if not _matching_job_fence(claimed, executor_id, candidate_generation + 1):
	return "stale"
	claimed = await self._repo.lock_verification_job(str(job_id))
	if not _matching_job_fence(claimed, executor_id, candidate_generation + 1):
	await claim_transaction.rollback()
	return "stale"

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/modules/artifacts/service.py` around lines 395 - 397, Roll back
the active claim transaction before returning "stale" when _matching_job_fence
fails after lock_verification_job in the verification-job flow. Update this
post-claim mismatch branch to match the rollback behavior in
execute_committed_put and resolve_put_attempt, while preserving the existing
stale return.