Add S3-compatible artifact storage with MinIO proof

[Abiorh001]

Workstream PR Trust Bundle

Chunk

WS-ART-001-02B1 - S3-Compatible MinIO And AWS

Merge intent: .agent-loop/merge-intents/WS-ART-001-02B1.json

Goal

Add one provider-neutral S3 protocol implementation, prove it against real
MinIO for local/CI, and validate isolated native AWS configuration without
making AWS runtime-eligible before the separately owned live-proof chunk.

Human-Approved Intent

• Intent: .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md
• Chunk contract: .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md

What Changed

• Added S3CompatibleArtifactStore behind the existing ArtifactStore v2 and typed external-service factory boundaries.
• Added real digest-pinned, loopback-bound MinIO proof for local/dev/test/CI.
• Added closed AWS workload-identity configuration while preserving the pre-I/O artifact_provider_live_proof_required guard.
• Added canonical S3 namespace validation, immutable object behavior, bounded streaming, and secret-safe failure paths.
• Addressed all four valid CodeRabbit findings: bounded source cursor, deferred credential materialization, IPv6 canonicalization, and deep byte/object secret scanning.

Why It Changed

Workstream needs provider-neutral immutable artifact storage before submission
admission, verification, publication, and recovery can safely bind product
records to exact bytes.

Design Chosen

One S3-protocol adapter serves MinIO proof and the inactive AWS profile. MinIO
is non-production proof; AWS is configuration-ready but unavailable until
Chunk 07 supplies release-bound live proof.

Alternatives Rejected

• Provider-specific MinIO and AWS implementations.
• R2 or Flow Node runtime in v0.1.
• Presigned, multipart, list, delete, copy, or public-access capabilities.

Scope Control

Allowed Files Changed

• ART adapter/interface/configuration, worker startup guards, focused tests, CI/Compose, ART docs, and loop evidence.

Files Outside Contract

• None.

Product Behavior

• No Workstream product behavior changed.
• Product behavior changed and is explained here.

Evidence

Commands Run

docker compose up -d --wait postgres redis minio
cd backend && .venv/bin/ruff check app tests
cd backend && .venv/bin/pip check
cd backend && WORKSTREAM_TEST_MINIO_ENDPOINT=http://127.0.0.1:9000 .venv/bin/python -m pytest <focused ART suite> --cov-fail-under=90
python3 scripts/test_agent_gates.py
python3 scripts/check_stale_artifact_contracts.py
python3 scripts/check_stale_authorization_docs.py
python3 scripts/check_stale_review_contracts.py
python3 scripts/check_stale_workstream_wording.py
python3 scripts/check_markdown_links.py

Result Summary

443 real-service focused tests PASS
S3CompatibleArtifactStore 91%; S3 validation 97%; combined scope 92.52%
Ruff, pip integrity, 88 agent gates, stale scans, links, Compose, diff PASS

Acceptance Criteria Proof

• Server-derived sealed commitments control provider object keys.
• Exact replay and conditional no-overwrite semantics are deterministic.
• Real LocalStorage and MinIO pass the same ArtifactStore v2 vectors.
• AWS credential sources are closed and AWS fails before any credential/provider side effect.
• Repository 78% and cumulative focused 90% CI gates remain fail closed.
• ART/AUTH/REV ownership remains separated and 02C1 remains inactive.

Test Delta

Tests Added

• Real MinIO conformance, AWS credential isolation, namespace/replay boundaries, security object-graph checks, and all external-review regressions.

Tests Modified

• ART architecture, configuration, cleanup wiring, LocalStorage parity, audit, and deterministic CI gate tests.

Tests Removed Or Skipped

• None.

Internal Reviewer Results

Reviewed code SHA: 9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a

Reviewed at: 2026-07-18T23:40:43Z

Reviewer run IDs are recorded in the committed internal-review evidence.

Reviewer	Result	Blocking Findings	Notes
Senior engineering	PASS WITH LOW RISKS	None	Exact SDK pins bound the reviewed inactive AWS dependency.
QA/test	PASS WITH LOW RISKS	None	All four repairs and real-service evidence covered.
Security/auth	PASS	None	Credential, secret, endpoint, and no-I/O boundaries hold.
Product/ops	PASS	None	No product lifecycle behavior activated.
Architecture	PASS WITH LOW RISKS	None	ART-only typed adapter path preserved.
CI integrity	PASS	None	No threshold or workflow weakening.
Docs	PASS	None	Contracts and eligibility wording align.
Reuse/dedup	PASS WITH LOW RISKS	None	Canonical helpers and factory path reused.
Test delta	PASS WITH LOW RISKS	None	No skip or assertion weakening.

External Review

External review response file:

• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md

Source	Status	Notes
CodeRabbit	Repairs verified; rerun pending	Four valid findings fixed and automatically resolved; immediate incremental rerun rate-limited.
GitHub checks	Pending	Agent Gates passed; Backend running on repaired candidate.

CI And Gate Integrity

• No workflow weakening.
• No lint/test/docstring gate weakening.
• No coverage threshold weakening.
• No package script weakening.
• No unpinned new GitHub Action.
• Checkout credential persistence disabled where checkout is used.

Remaining Risks

• Native AWS remains unavailable until Chunk 07 live proof.
• MinIO proves protocol behavior only for non-production environments.
• Admission, verification, publication, and recovery remain later ART chunks.

Follow-Up Work

WS-ART-001-02C1 is the same-initiative successor but requires a separate
explicit user start after this PR merges.

Human Review Focus

• Confirm there is one S3 protocol adapter, not provider forks.
• Confirm untrusted input cannot choose object identity, credential sources, proxies, or namespace.
• Confirm AWS fails before credential/provider side effects.
• Confirm no later ART capability or AUTH/REV/CON behavior is activated.

Human Merge Ownership

• I can explain what changed.
• I can explain why it changed.
• I know what could break.
• I accept the remaining risks.
• The user explicitly approved this specific PR for merge.

Summary by CodeRabbit

• New Features

  • Added S3-compatible immutable artifact storage using MinIO for local and CI environments.
  • Added strict S3 namespace, endpoint, bucket, prefix, and credential validation.
  • Added isolated AWS workload-identity configuration checks; native AWS remains disabled until live proof is available.
  • Added MinIO to local Docker Compose and backend CI workflows.

• Bug Fixes

  • Improved fail-closed startup and cleanup behavior for unsupported provider configurations.
  • Strengthened protection against credential and secret exposure in errors and logs.

• Documentation

  • Updated setup instructions, architecture guidance, and artifact-storage specifications for MinIO and AWS behavior.

[coderabbitai]

📝 Walkthrough

Walkthrough

WS-ART-001-02B1 adds an S3-compatible artifact adapter with MinIO integration, strict configuration and namespace validation, isolated AWS credential handling, runtime eligibility guards, expanded proof tests, CI coverage enforcement, documentation, and updated workstream state.

Changes

S3-compatible artifact storage

Layer / File(s)	Summary
Configuration and namespace contracts backend/app/core/*, backend/app/interfaces/artifacts.py, backend/app/adapters/artifacts/references.py, .agent-loop/initiatives/...	Adds S3 settings, secret scrubbing, canonical endpoint and namespace validation, provider-profile rules, and shared provider-object references.
Adapter implementation and runtime wiring backend/app/adapters/artifacts/*, backend/app/workers/*, backend/pyproject.toml	Adds immutable MinIO/S3 operations, replay verification, isolated workload-identity handling, factory registration, and early native-AWS eligibility failures.
Adapter and credential proof validation backend/tests/*	Adds MinIO integration, AWS isolation, namespace, replay, architecture, streaming, error-mapping, and secret-retention tests.
MinIO environment and CI integration docker-compose.yml, .github/workflows/backend.yml, README.md, docs/*	Adds pinned MinIO startup, readiness checks, coverage gates, local setup instructions, and eligibility documentation.
Coverage gates and workstream state .agent-loop/*, scripts/test_agent_gates.py	Updates activation state, review evidence, merge metadata, coverage requirements, and gate regression checks.

Estimated code review effort: 5 (Critical) | ~90+ minutes

Sequence Diagram(s)

sequenceDiagram
  participant Application
  participant ArtifactStoreFactory
  participant S3CompatibleArtifactStore
  participant MinIO
  Application->>ArtifactStoreFactory: create artifact store bootstrap
  ArtifactStoreFactory->>S3CompatibleArtifactStore: validate and initialize after namespace claim
  S3CompatibleArtifactStore->>MinIO: put, head, open, or verify immutable object
  MinIO-->>S3CompatibleArtifactStore: object response or provider error
  S3CompatibleArtifactStore-->>Application: artifact result or domain error

Loading

Possibly related PRs

• Flow-Research/workstream#101: Extends the immutable artifact adapter system and shared provider-reference handling.
• Flow-Research/workstream#141: Introduces the ArtifactStore v2 interfaces and composition layer extended here.
• Flow-Research/workstream#98: Updates the related WS-ART workflow state and chunk activation boundaries.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 44.55% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: adding S3-compatible artifact storage with MinIO proof.
Description check	✅ Passed	The description closely matches the trust-bundle template and includes the required sections, evidence, reviews, and risk details.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/ws-art-001-02b1-s3-compatible-minio-aws

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/app/adapters/artifacts/s3_compatible.py`:
- Around line 826-836: Update the pending-buffer fill loop around `_iterator`
and `_pending` to retain a memoryview cursor for each source chunk and extend
`_pending` only with the portion needed to satisfy the requested amount, rather
than copying the entire chunk first. Preserve the bytes-type validation and
ensure any unconsumed chunk data remains available for subsequent reads while
keeping `_pending` within the configured buffer threshold.
- Around line 1115-1127: Update the workload-identity credential resolution flow
around session.get_credentials() to materialize deferred credentials by awaiting
credentials.get_frozen_credentials() before returning. If materialization fails,
raise the same AWS workload identity configuration error used for resolution
failures, while preserving the existing method validation and successful return
behavior.

In `@backend/app/core/s3_validation.py`:
- Around line 90-96: Canonicalize IPv6 hostnames before constructing the
normalized endpoint in the scheme/hostname formatting flow. Parse IPv6 literals
with the standard IP address utility and use its compressed canonical
representation, then apply bracket and port formatting; leave domain and IPv4
handling unchanged so equivalent IPv6 spellings produce the same namespace
identity.

In `@backend/tests/assertion_helpers.py`:
- Around line 23-26: Update the assertion helper’s value-scanning logic to
handle byte buffers by checking the secret against bytes values, and recursively
inspect public object state stored in __dict__. Ensure nested values are scanned
before declaring the secret absent, while preserving the existing str and
SecretStr handling.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1c891da0-3df5-4f6c-9a0b-e73e917eb5f1

📥 Commits

Reviewing files that changed from the base of the PR and between 983b9e5 and 7dcce9f.

📒 Files selected for processing (47)

• .agent-loop/LOOP_STATE.md
• .agent-loop/WORK_QUEUE.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md
• .agent-loop/merge-intents/WS-ART-001-02B1.json
• .github/workflows/backend.yml
• README.md
• backend/app/adapters/artifacts/__init__.py
• backend/app/adapters/artifacts/local.py
• backend/app/adapters/artifacts/references.py
• backend/app/adapters/artifacts/s3_compatible.py
• backend/app/core/config.py
• backend/app/core/s3_validation.py
• backend/app/interfaces/artifacts.py
• backend/app/workers/artifacts.py
• backend/app/workers/celery_app.py
• backend/pyproject.toml
• backend/tests/assertion_helpers.py
• backend/tests/test_artifact_architecture.py
• backend/tests/test_artifact_cleanup_wiring.py
• backend/tests/test_artifact_store_conformance.py
• backend/tests/test_assertion_helpers.py
• backend/tests/test_audit.py
• backend/tests/test_aws_credential_isolation.py
• backend/tests/test_config.py
• backend/tests/test_local_artifact_store.py
• backend/tests/test_s3_artifact_store.py
• docker-compose.yml
• docs/architecture_lockdown.md
• docs/decision_0013_immutable_artifact_storage_boundary.md
• docs/spec_artifact_storage_service.md
• scripts/test_agent_gates.py

[Abiorh001]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
@.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md:
- Line 131: Update the test-count sentence in the review evidence to hyphenate
the compound modifier, changing “443 real-service focused tests” to “443
real-service-focused tests.”
- Line 56: Update the credential-materialization wording in the review evidence
text from “Required selected AWS workload credentials” to “Required the selected
AWS workload credentials,” preserving the rest of the statement unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 85332753-4e8d-4671-a9f8-af09fae9be14

📥 Commits

Reviewing files that changed from the base of the PR and between 7dcce9f and 1b923b0.

📒 Files selected for processing (11)

• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md
• backend/app/adapters/artifacts/s3_compatible.py
• backend/app/core/s3_validation.py
• backend/tests/assertion_helpers.py
• backend/tests/test_assertion_helpers.py
• backend/tests/test_aws_credential_isolation.py
• backend/tests/test_config.py
• backend/tests/test_s3_artifact_store.py

🚧 Files skipped from review as they are similar to previous changes (6)

• backend/tests/assertion_helpers.py
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md
• backend/app/core/s3_validation.py
• .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md
• backend/app/adapters/artifacts/s3_compatible.py
• backend/tests/test_aws_credential_isolation.py