From ca09cfd978be46905b502ee264be31ed642bba80 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 07:13:00 +0100 Subject: [PATCH 01/39] Implement S3-compatible artifact storage --- .agent-loop/LOOP_STATE.md | 12 +- .agent-loop/WORK_QUEUE.md | 5 +- .../CHUNK_MAP.md | 2 +- .../STATUS.md | 44 +- ...WS-ART-001-02B1-s3-compatible-minio-aws.md | 46 +- ...1-02C1-admission-put-attempt-foundation.md | 1 + ...1-02C2-verification-publication-fencing.md | 1 + ...T-001-02C3-recovery-attempt-idempotency.md | 1 + ...RT-001-02D-operator-artifact-operations.md | 1 + .../WS-ART-001-03-guide-source-cutover.md | 1 + ...S-ART-001-04A-upload-inspection-sealing.md | 1 + .../WS-ART-001-04B-pre-submit-admission.md | 1 + ...-ART-001-05-submission-artifact-cutover.md | 1 + ...T-001-06A-checker-input-materialization.md | 1 + .../WS-ART-001-06B-checker-output-routing.md | 1 + .../WS-ART-001-07-recovery-live-proof.md | 1 + .../merge-intents/WS-ART-001-02B1.json | 9 + .github/workflows/backend.yml | 26 + backend/app/adapters/artifacts/__init__.py | 21 + .../app/adapters/artifacts/s3_compatible.py | 763 ++++++++++++++++++ backend/app/core/config.py | 276 ++++++- backend/app/interfaces/artifacts.py | 23 + backend/pyproject.toml | 2 + backend/tests/test_artifact_architecture.py | 34 +- backend/tests/test_artifact_cleanup_wiring.py | 32 +- .../tests/test_artifact_store_conformance.py | 41 + .../tests/test_aws_credential_isolation.py | 324 ++++++++ backend/tests/test_config.py | 288 ++++++- backend/tests/test_s3_artifact_store.py | 538 ++++++++++++ docker-compose.yml | 16 + docs/spec_artifact_storage_service.md | 22 +- scripts/test_agent_gates.py | 61 +- 32 files changed, 2490 insertions(+), 106 deletions(-) create mode 100644 .agent-loop/merge-intents/WS-ART-001-02B1.json create mode 100644 backend/app/adapters/artifacts/s3_compatible.py create mode 100644 backend/tests/test_aws_credential_isolation.py create mode 100644 backend/tests/test_s3_artifact_store.py diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index 63acd88f..2bec2b0a 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -17,7 +17,11 @@ actor provisioning and leaves service runtime admission plus feature actions inactive. - PR #141 merged `WS-ART-001-02A3` into `main` as `a10d901` on - 2026-07-18; ART-02B1 remains inactive pending a separate explicit start. + 2026-07-18; the user then explicitly started ART-02B1. +- Active ART implementation chunk: `WS-ART-001-02B1` on + `codex/ws-art-001-02b1-s3-compatible-minio-aws`. +- The ART worktree consumes merged AUTH, REV, and CON contracts without + editing or activating their independently owned runtime behavior. - Active implementation chunk: `WS-AUTH-001-09C` on `codex/ws-auth-001-09c-actor-identity-admin-reads`. - AUTH-09C is bounded to two exact administrative read routes and activates @@ -35,6 +39,9 @@ PostgreSQL, and Agent Gates proof passes. All required exact integrated-head internal repair tracks pass at `c47d627`; push and replacement external checks remain. No service caller becomes executable before AUTH-09E. +- Current ART gate: integrate trusted `main`, complete deterministic 02B1 + proof, and pass all nine exact-SHA internal reviewer tracks before opening + the ART PR. No later ART chunk starts automatically. - Scope checkpoint: AWS S3 is the only v0.1 production provider; MinIO is local/CI S3 protocol proof; LocalStorage is focused development/test; R2 and Flow Node are deferred. Product modules receive narrow artifact capabilities, @@ -49,7 +56,8 @@ admission, prepared mutation authority, and exact AUTH-only activation chunks; neither reconciliation PR activates feature behavior. - Parallel artifact checkpoint: ART-02A1, ART-02A2, and ART-02A3 merged through - PRs #127, #129, and #141. ART-02B1 remains inactive. + PRs #127, #129, and #141. ART-02B1 is active and adds real MinIO protocol + proof plus a fail-closed, runtime-ineligible native AWS profile. - Authorization checkpoint: AUTH-07B through AUTH-09B merged through PRs #130, #131, #132, and #143. Signed memory stopped after 09B, and the user explicitly started AUTH-09C. diff --git a/.agent-loop/WORK_QUEUE.md b/.agent-loop/WORK_QUEUE.md index 9c438252..c012c428 100644 --- a/.agent-loop/WORK_QUEUE.md +++ b/.agent-loop/WORK_QUEUE.md @@ -5,6 +5,7 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| | `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | PR #146 open; all valid CodeRabbit findings repaired; integrated exact-head review passed; replacement external checks pending | +| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Active after explicit user start; implementation repair and exact-head internal review in progress | Live post-merge state remains read from signed `automation/loop-memory` output. This authored queue records the separately approved parallel chunks. @@ -17,7 +18,6 @@ output. This authored queue records the separately approved parallel chunks. | `WS-AUTH-001-09D` | Actor And Identity-Link Lifecycle Mutations | L1 | Inactive until 09C merge/memory and explicit user start | | `WS-QUAL-001-02` | Project Service Coverage | L1 | Inactive until 01B2 merge/memory plus explicit user start | | `WS-POL-002-04` | Locked Runtime Execution And Routing Hardening | L1 | Inactive pending relevant authorization proof and a separate explicit user start | -| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Inactive until 02A3 merge and explicit user start | | `WS-ART-001-02C1` | Admission And Put-Attempt Foundation | L1 | Inactive until 02B1 merge and explicit user start | | `WS-ART-001-02C2` | Verification Publication And Fencing | L1 | Inactive until 02C1 merge and explicit user start | | `WS-ART-001-02C3` | Recovery Attempt And Idempotency Chain | L1 | Inactive until 02C2 merge and explicit user start | @@ -97,7 +97,8 @@ coverage implementation chunk from this worktree. `WS-ART-001-01`, the AWS-first planning amendment, `02A1`, `02A2`, and `02A3` are merged; PR #141 merged `02A3` as `a10d901`. R2 and Flow Node are deferred. -`02B1` remains inactive until a separate explicit user start. +The user explicitly started `02B1` on 2026-07-18. `02C1` remains inactive +until `02B1` merges and receives a separate explicit start. Coverage work proceeds independently in its own worktree and is not owned by this AUTH queue update. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index e99f7c80..46488869 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -10,7 +10,7 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-02A1` | Install only ADR 0014's small typed external-service adapter/factory foundation without migrating a capability. | L1 | Merged through PR #127 as `f64a8e5` | | `WS-ART-001-02A2` | Add bounded committed-source preparation and inactive scratch-cleanup mechanics without changing the active v1 port. | L1 | Merged through PR #129 as `9a04434` on 2026-07-16 | | `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Internal review and deterministic evidence passed; external checks pending | -| `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Proposed after 02A3 | +| `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Active after 02A3 merged through PR #141 and explicit user start | | `WS-ART-001-02C1` | Add the generic durable-byte admission ledger and durable put-attempt state foundation without provider execution. | L1 | Proposed after 02B1 | | `WS-ART-001-02C2` | Add put resolution, verification publication, complete-object observation, immutable receipts, and PostgreSQL execution fencing without recovery attempts or routes. | L1 | Proposed after 02C1 | | `WS-ART-001-02C3` | Add the recovery-attempt model and exact idempotent source-job to retry-job chain without public or Operator routes. | L1 | Proposed after 02C2 | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index 6cda9a86..63e8197a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -5,9 +5,10 @@ Original planning merged through PR #97, artifact/LocalStorage foundation merged through PR #101, the AWS-first object-storage amendment merged through PR #120 as `4408256`, the external-service adapter foundation merged through -PR #127 as `f64a8e5`, and committed-source preparation merged through PR #129 -as `9a04434` on 2026-07-16. The user explicitly started -`WS-ART-001-02A3` on 2026-07-16. +PR #127 as `f64a8e5`, committed-source preparation merged through PR #129 as +`9a04434`, and the ArtifactStore v2 Local clean cut merged through PR #141 as +`a10d901` on 2026-07-18. The user explicitly started `WS-ART-001-02B1` on +2026-07-18. The planning-only cross-initiative boundary reconciliation merged through PR #139 as `5d353b6`, and AUTH's owner reconciliation merged through PR #140 as @@ -31,29 +32,26 @@ approval or reusable evidence. Its source remains on branch ## Current Work -`WS-ART-001-02A3` implementation and merged-main deterministic repair are -complete. PR #141 is open. Exact-SHA review found and repaired a residual -LocalStorage startup race, typed-factory startup mismatch, and vague Operator -resource vocabulary. Fresh deterministic coverage and all required internal -reviewer tracks passed; external checks are now pending. Its approved -boundary atomically replaces ArtifactStore v1 with byte-only v2, migrates -LocalStorage and the empty pre-production artifact schema, installs the -immutable storage-namespace fence, -removes dormant `flow_node` configuration, and activates startup plus periodic -scratch cleanup. It does not activate product ingest, durable admission, -put-attempt resolution, verification jobs, or recovery. +`WS-ART-001-02B1` is active. It adds one `S3CompatibleArtifactStore`, runs the +shared ArtifactStore v2 vectors against real digest-pinned MinIO, and validates +an isolated native-AWS workload-identity profile. MinIO is runtime-eligible +only in local/development/test after the PostgreSQL namespace claim. Native AWS +remains runtime-ineligible and fails with +`artifact_provider_live_proof_required` before factory construction, +credential resolution, namespace claim, or provider I/O. No product ingest, +durable admission, put-attempt resolution, verification job, recovery route, +or optional-provider runtime is activated. R2 and Flow Node remain deferred. ## Next Proposed Chunk -`02B1` owns the S3-compatible adapter, MinIO proof, and AWS S3 profile after -`02A3` merges and receives a separate explicit start. There is no active R2 or -Flow Node chunk. +`02C1` owns generic durable-byte admission and put-attempt state only after +`02B1` merges and receives a separate explicit start. Neither R2 nor Flow Node +has a v0.1 chunk. ## Gate -All reviews before the latest repairs are retained as history, not reused as -final provenance. Deterministic proof and exact-SHA internal review have passed. -The current gate is GitHub Actions, CodeRabbit, and explicit human review. -Durable admission, put attempts, verification publication, and recovery remain -in their later owning chunks. No -later artifact chunk starts automatically, and only the user may approve merge. +The current gate is deterministic 02B1 proof followed by all nine exact-SHA +internal reviewer tracks. GitHub Actions, CodeRabbit, and explicit human review +follow only after that evidence passes. Durable admission, put attempts, +verification publication, and recovery remain in later owning chunks. No later +artifact chunk starts automatically, and only the user may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index 756acf1c..d2534884 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-02B1 - S3-Compatible MinIO And AWS -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02A3 +Initiative: `WS-ART-001` | Risk: L1 | Status: Active after explicit human start Artifact contract phase: `artifact_store_cutover` @@ -14,9 +14,14 @@ credential-delivery contract. ## Allowed Files - `backend/app/adapters/artifacts/s3_compatible.py` and adapter registration; +- `backend/app/interfaces/artifacts.py` only to register the closed + `minio-v1` and `aws-s3-v1` namespace profiles, their exact descriptor keys, + and the stable AWS live-proof-required error; - AWS/MinIO settings in `backend/app/core/config.py`; -- backend dependency manifest/lock with exact `aiobotocore==3.7.0` and - `botocore==1.43.0` pins, `docker-compose.yml`, and +- `backend/pyproject.toml` with exact `aiobotocore==3.7.0` and + `botocore==1.43.0` pins; this repository has no backend dependency lockfile, + so dependency tests must inspect the exact manifest pins and installed + versions; `docker-compose.yml`, and `.github/workflows/backend.yml` for MinIO and exact coverage proof; - `scripts/test_agent_gates.py` only to assert the exact workflow command, source set, threshold, and cumulative retention; @@ -59,8 +64,12 @@ credential-delivery contract. - real digest-pinned MinIO and LocalStorage pass the same v2 conformance suite; - the application factory can instantiate only LocalStorage and MinIO in this chunk. Valid AWS configuration remains runtime-ineligible with the stable - `artifact_provider_live_proof_required` startup failure; only Chunk 07 may - add the immutable activation record and production composition guard; + `artifact_provider_live_proof_required` startup failure. The composition root + raises that typed error after settings validation but before factory + construction, namespace claim, credential resolver construction, credential + loading, or provider I/O. Tests prove no namespace row is persisted and no + credential source is touched. Only Chunk 07 may add the immutable activation + record and production composition guard; - startup and operation tests reject configured adapter/profile/namespace mismatch against persisted deployment or replica identity; no request is routed to a second namespace; @@ -73,6 +82,30 @@ credential-delivery contract. repository command below; `scripts/test_agent_gates.py` fails on workflow command, source-set, threshold, or cumulative-retention drift. +## Canonical Namespace Profiles + +The shared namespace value object accepts only these additional profiles: + +```text +minio-v1: + addressing_style + bucket + endpoint_identity + private_prefix + region + +aws-s3-v1: + addressing_style + bucket + private_prefix + region +``` + +`endpoint_identity` is a canonical non-secret hash of the normalized MinIO +endpoint, not credential material. Native AWS has no configured endpoint and +therefore no endpoint descriptor key. Descriptor key ordering remains +canonical and exact; unknown, missing, or extra keys fail closed. + ## Exact CI Coverage Gate ```bash @@ -81,6 +114,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 ``` ## Verification @@ -88,7 +122,7 @@ coverage report --include='app/main.py' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio (cd backend && .venv/bin/ruff check app tests) -(cd backend && .venv/bin/pytest tests/test_artifact_store_conformance.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) +(cd backend && .venv/bin/pytest tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_preparation.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) python3 scripts/check_stale_artifact_contracts.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md index 3306c578..72f71b00 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md @@ -71,6 +71,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md index 26db8e48..af7abb2e 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md @@ -86,6 +86,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md index b2901004..8ffa49f8 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md @@ -63,6 +63,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md index ee82beaf..b2f7566d 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md @@ -129,6 +129,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md index cc146e27..437771a4 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md @@ -109,6 +109,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md index 58d4ba62..5f2f8f18 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md @@ -89,6 +89,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md index ad8e193e..9a66a981 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md @@ -90,6 +90,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md index d3038977..c3840e2f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md @@ -95,6 +95,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md index fc02995c..5a94e64e 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md @@ -76,6 +76,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md index 20a634d1..5c43d4e7 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md @@ -84,6 +84,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md index 8b9574a3..7a8959e8 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md @@ -126,6 +126,7 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/merge-intents/WS-ART-001-02B1.json b/.agent-loop/merge-intents/WS-ART-001-02B1.json new file mode 100644 index 00000000..923a9476 --- /dev/null +++ b/.agent-loop/merge-intents/WS-ART-001-02B1.json @@ -0,0 +1,9 @@ +{ + "chunk_id": "WS-ART-001-02B1", + "chunk_title": "S3-Compatible MinIO And AWS", + "initiative_id": "WS-ART-001", + "next_chunk_id": "WS-ART-001-02C1", + "next_chunk_title": "Admission And Put-Attempt Foundation", + "next_requires_explicit_start": true, + "schema_version": 2 +} diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index 83b071e2..2c797a38 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -45,6 +45,23 @@ jobs: python -m pip install --upgrade pip python -m pip install -e ".[dev]" + - name: Start real MinIO artifact provider + run: | + image='quay.io/minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e' + docker run --detach --rm --name workstream-minio \ + --publish 9000:9000 \ + --env MINIO_ROOT_USER=workstream-minio \ + --env MINIO_ROOT_PASSWORD=workstream-minio-secret-key \ + "$image" server /data --address :9000 + for attempt in $(seq 1 60); do + if curl --fail --silent http://127.0.0.1:9000/minio/health/live >/dev/null; then + exit 0 + fi + sleep 1 + done + docker logs workstream-minio + exit 1 + - name: Lint working-directory: backend run: ruff check app tests scripts @@ -63,6 +80,7 @@ jobs: working-directory: backend env: WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres + WORKSTREAM_TEST_MINIO_ENDPOINT: http://127.0.0.1:9000 run: | metadata_dir="$(mktemp -d)" trap 'rm -rf "$metadata_dir"' EXIT @@ -108,6 +126,14 @@ jobs: --precision=2 --fail-under=90 + - name: S3-compatible artifact adapter coverage + working-directory: backend + run: >- + coverage report + --include='app/adapters/artifacts/s3_compatible.py' + --precision=2 + --fail-under=90 + - name: Actor subsystem coverage working-directory: backend run: >- diff --git a/backend/app/adapters/artifacts/__init__.py b/backend/app/adapters/artifacts/__init__.py index 1805b7d1..a15229c3 100644 --- a/backend/app/adapters/artifacts/__init__.py +++ b/backend/app/adapters/artifacts/__init__.py @@ -6,6 +6,7 @@ from app.interfaces.artifacts import ( ARTIFACT_STORE_CAPABILITY_KEY, ArtifactConfigurationError, + ArtifactProviderLiveProofRequiredError, ArtifactStoreBootstrap, ) from app.interfaces.external_services import ExternalServiceAdapterFactory @@ -27,7 +28,23 @@ def create_artifact_store_bootstrap(settings: Settings) -> ArtifactStoreBootstra Raises: ExternalServiceConfigurationError: If the provider is not registered. """ + if ( + settings.artifact_store_backend == "s3_compatible" + and settings.artifact_s3_provider_profile == "aws_s3" + ): + from app.adapters.artifacts.s3_compatible import ( + validate_aws_workload_identity_environment, + ) + + validate_aws_workload_identity_environment(settings) + raise ArtifactProviderLiveProofRequiredError( + "AWS artifact provider requires live deployment proof" + ) + from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap + from app.adapters.artifacts.s3_compatible import ( + create_minio_artifact_store_bootstrap, + ) factory = ExternalServiceAdapterFactory[ArtifactStoreBootstrap]( ARTIFACT_STORE_CAPABILITY_KEY @@ -46,6 +63,10 @@ def create_local_store() -> ArtifactStoreBootstrap: ) factory.register("local", create_local_store) + factory.register( + "s3_compatible", + lambda: create_minio_artifact_store_bootstrap(settings), + ) return factory.create(settings.artifact_store_backend) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py new file mode 100644 index 00000000..65447440 --- /dev/null +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -0,0 +1,763 @@ +"""Private S3-protocol implementation of the immutable ArtifactStore v2 port.""" + +from __future__ import annotations + +import asyncio +from collections.abc import AsyncIterator, Mapping +from contextlib import asynccontextmanager +import hashlib +import os +from pathlib import Path +import re +from typing import Any + +from aiobotocore.config import AioConfig +from aiobotocore.credentials import ( + AioAssumeRoleWithWebIdentityProvider, + AioContainerProvider, + AioCredentialResolver, + AioInstanceMetadataProvider, +) +from aiobotocore.session import AioSession +from aiobotocore.utils import AioInstanceMetadataFetcher +from botocore.exceptions import BotoCoreError, ClientError + +from app.core.config import Settings +from app.core.hashing import canonical_json_hash +from app.interfaces.artifacts import ( + ARTIFACT_STORE_CAPABILITY_KEY, + ArtifactByteRange, + ArtifactConfigurationError, + ArtifactInputMismatchError, + ArtifactIntegrityError, + ArtifactLimitExceededError, + ArtifactObjectHead, + ArtifactObjectMissingError, + ArtifactOperationConflictError, + ArtifactPutObservation, + ArtifactPutResult, + ArtifactRangeInvalidError, + ArtifactStoreError, + ArtifactStoreNamespaceClaim, + ArtifactStoreNamespaceIdentity, + ArtifactStoreUnavailableError, + artifact_store_namespace_material, +) +from app.interfaces.external_services import ExternalServiceAdapterIdentity +from app.modules.artifacts.preparation import HARD_MAXIMUM_ARTIFACT_BYTES +from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource + + +_IDENTITY = ExternalServiceAdapterIdentity(ARTIFACT_STORE_CAPABILITY_KEY, "s3_compatible") +_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") +_PRECONDITION_ERROR_CODES = frozenset( + {"409", "412", "ConditionalRequestConflict", "PreconditionFailed"} +) +_MISSING_ERROR_CODES = frozenset({"404", "NoSuchKey", "NotFound"}) +_FORBIDDEN_AWS_CREDENTIAL_ENVIRONMENT = frozenset( + { + "AWS_ACCESS_KEY", + "AWS_ACCESS_KEY_ID", + "AWS_CONFIG_FILE", + "AWS_CREDENTIAL_FILE", + "AWS_DEFAULT_PROFILE", + "AWS_LOGIN_CACHE_DIRECTORY", + "AWS_PROFILE", + "AWS_SECRET_ACCESS_KEY", + "AWS_SECRET_KEY", + "AWS_SECURITY_TOKEN", + "AWS_SESSION_TOKEN", + "AWS_SHARED_CREDENTIALS_FILE", + "BOTO_CONFIG", + } +) +_WEB_IDENTITY_ENVIRONMENT = frozenset( + {"AWS_ROLE_ARN", "AWS_ROLE_SESSION_NAME", "AWS_WEB_IDENTITY_TOKEN_FILE"} +) +_CONTAINER_ENVIRONMENT = frozenset( + { + "AWS_CONTAINER_AUTHORIZATION_TOKEN", + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", + "AWS_CONTAINER_CREDENTIALS_FULL_URI", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", + } +) +_IAM_ROLE_ENVIRONMENT = frozenset( + { + "AWS_EC2_METADATA_DISABLED", + "AWS_EC2_METADATA_SERVICE_ENDPOINT", + "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE", + "AWS_EC2_METADATA_V1_DISABLED", + } +) + + +class S3CompatibleArtifactStoreBootstrap: + """Composition-only namespace lifecycle for one pinned S3 namespace.""" + + def __init__(self, adapter: S3CompatibleArtifactStore) -> None: + """Own one configured adapter before PostgreSQL namespace admission.""" + if type(adapter) is not S3CompatibleArtifactStore: + raise ValueError("S3-compatible artifact bootstrap adapter is invalid") + self._adapter = adapter + + @property + def identity(self) -> ExternalServiceAdapterIdentity: + """Return the canonical artifact-store/S3 adapter identity.""" + return self._adapter.identity + + @property + def namespace_identity(self) -> ArtifactStoreNamespaceIdentity: + """Return the configured provider namespace identity.""" + return self._adapter._namespace_identity + + def initialize_after_namespace_claim( + self, + claim: ArtifactStoreNamespaceClaim, + ) -> S3CompatibleArtifactStore: + """Return the byte store only after exact namespace admission.""" + return self._adapter._initialize_after_namespace_claim(claim) + + def close(self) -> None: + """Release this bootstrap and invalidate its adapter.""" + self._adapter.close() + + +class S3CompatibleArtifactStore: + """Immutable S3-protocol byte provider for MinIO and later AWS activation.""" + + def __init__( + self, + *, + provider_profile: str, + region: str, + endpoint_url: str | None, + bucket: str, + private_prefix: str, + addressing_style: str, + session: AioSession, + buffer_bytes: int, + connect_timeout_seconds: float, + read_timeout_seconds: float, + write_timeout_seconds: float, + pool_timeout_seconds: float, + operation_total_timeout_seconds: float, + max_pool_connections: int, + ) -> None: + """Pin one validated namespace and one isolated SDK session.""" + if provider_profile not in {"minio", "aws_s3"}: + raise ArtifactConfigurationError("S3 provider profile is invalid") + if not isinstance(session, AioSession): + raise ArtifactConfigurationError("S3 credential session is invalid") + if type(buffer_bytes) is not int or not 1 <= buffer_bytes <= 1024 * 1024: + raise ArtifactConfigurationError("S3 artifact buffer is invalid") + if type(max_pool_connections) is not int or not 1 <= max_pool_connections <= 256: + raise ArtifactConfigurationError("S3 connection pool is invalid") + for value in ( + connect_timeout_seconds, + read_timeout_seconds, + write_timeout_seconds, + pool_timeout_seconds, + operation_total_timeout_seconds, + ): + if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0: + raise ArtifactConfigurationError("S3 operation timeout is invalid") + + self._provider_profile = provider_profile + self._region = region + self._endpoint_url = endpoint_url + self._bucket = bucket + self._private_prefix = private_prefix + self._addressing_style = addressing_style + self._session = session + self._buffer_bytes = buffer_bytes + self._write_timeout_seconds = float(write_timeout_seconds) + self._pool_timeout_seconds = float(pool_timeout_seconds) + self._operation_total_timeout_seconds = float(operation_total_timeout_seconds) + self._client_config = AioConfig( + connect_timeout=float(connect_timeout_seconds), + read_timeout=float(read_timeout_seconds), + max_pool_connections=max_pool_connections, + retries={"max_attempts": 1, "mode": "standard"}, + request_checksum_calculation="when_required", + response_checksum_validation="when_required", + s3={ + "addressing_style": addressing_style, + "payload_signing_enabled": False, + }, + ) + self._initialized = False + + @property + def identity(self) -> ExternalServiceAdapterIdentity: + """Return the canonical artifact-store/S3 adapter identity.""" + return _IDENTITY + + @property + def _namespace_identity(self) -> ArtifactStoreNamespaceIdentity: + """Return the canonical non-secret identity of this S3 namespace.""" + profile = "minio-v1" if self._provider_profile == "minio" else "aws-s3-v1" + items = { + "addressing_style": self._addressing_style, + "bucket": self._bucket, + "private_prefix": self._private_prefix, + "region": self._region, + } + if self._provider_profile == "minio": + items["endpoint_identity"] = canonical_json_hash( + {"endpoint_url": self._endpoint_url} + ) + return ArtifactStoreNamespaceIdentity( + provider_profile=profile, + descriptor_items=tuple(sorted(items.items())), + ) + + def _initialize_after_namespace_claim( + self, + claim: ArtifactStoreNamespaceClaim, + ) -> S3CompatibleArtifactStore: + """Enable provider operations only for the exact PostgreSQL claim.""" + if ( + type(claim) is not ArtifactStoreNamespaceClaim + or claim.adapter_identity != self.identity + or claim.namespace_identity != self._namespace_identity + or claim.namespace_fingerprint != self._expected_namespace_fingerprint() + ): + self.close() + raise ArtifactConfigurationError("artifact namespace claim does not match provider") + if self._initialized: + self.close() + raise ArtifactConfigurationError("S3-compatible artifact storage is already initialized") + self._initialized = True + return self + + def _expected_namespace_fingerprint(self) -> str: + """Return the shared fingerprint for this configured S3 namespace.""" + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=self.identity, + namespace_identity=self._namespace_identity, + ) + return fingerprint + + def close(self) -> None: + """Invalidate this configured adapter; clients are operation-scoped.""" + self._initialized = False + + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + """Conditionally publish one sealed source or verify an exact replay.""" + self._require_initialized() + if type(source) is not CommittedArtifactSource: + raise ArtifactInputMismatchError("artifact source is not sealed") + commitment = source.commitment + if commitment.byte_count > HARD_MAXIMUM_ARTIFACT_BYTES: + raise ArtifactLimitExceededError("artifact source exceeds provider limit") + provider_object_ref = self._provider_object_ref(commitment) + body = _CommittedSourceBody(source, self._buffer_bytes) + try: + if commitment.byte_count == 0: + await body.validate_empty() + request_body: object = b"" if commitment.byte_count == 0 else body + async with asyncio.timeout(self._operation_total_timeout_seconds): + async with asyncio.timeout(self._write_timeout_seconds): + async with self._client() as client: + await client.put_object( + Bucket=self._bucket, + Key=self._object_key(provider_object_ref), + Body=request_body, + ContentLength=commitment.byte_count, + ContentType=commitment.media_type, + IfNoneMatch="*", + ) + if not body.complete: + if await self._matches_commitment(provider_object_ref, commitment): + return ArtifactPutResult(provider_object_ref, replayed=True) + raise ArtifactStoreUnavailableError("S3 artifact operation failed") + return ArtifactPutResult(provider_object_ref, replayed=False) + except ClientError as error: + if _provider_error_code(error) in _PRECONDITION_ERROR_CODES: + await self._verify_exact(provider_object_ref, commitment) + return ArtifactPutResult(provider_object_ref, replayed=True) + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + except ArtifactStoreError: + raise + except (TimeoutError, BotoCoreError, OSError): + if await self._matches_commitment(provider_object_ref, commitment): + return ArtifactPutResult(provider_object_ref, replayed=True) + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + + async def observe_put_result( + self, + commitment: ArtifactCommitment, + ) -> ArtifactPutObservation: + """Read and validate the deterministic committed object when present.""" + self._require_initialized() + if type(commitment) is not ArtifactCommitment: + raise ArtifactOperationConflictError("artifact commitment is invalid") + provider_object_ref = self._provider_object_ref(commitment) + observed = await self.head(provider_object_ref) + if not observed.exists: + return ArtifactPutObservation(provider_object_ref, committed=False) + await self._verify_exact(provider_object_ref, commitment) + return ArtifactPutObservation(provider_object_ref, committed=True) + + def open( + self, + provider_object_ref: str, + byte_range: ArtifactByteRange | None = None, + ) -> AsyncIterator[bytes]: + """Stream a full object or one bounded byte range.""" + self._require_initialized() + self._parse_provider_object_ref(provider_object_ref) + if byte_range is not None and type(byte_range) is not ArtifactByteRange: + raise ArtifactOperationConflictError("artifact byte range is invalid") + selected_range = byte_range or ArtifactByteRange() + + async def iterate() -> AsyncIterator[bytes]: + """Hold the S3 response open while yielding exact bounded chunks.""" + head = await self.head(provider_object_ref) + if not head.exists or head.byte_count is None: + raise ArtifactObjectMissingError("artifact object is missing") + if selected_range.offset > head.byte_count: + raise ArtifactRangeInvalidError("artifact range starts past object end") + expected = head.byte_count - selected_range.offset + if selected_range.length is not None: + expected = min(expected, selected_range.length) + if expected == 0: + return + + end = selected_range.offset + expected - 1 + request: dict[str, object] = { + "Bucket": self._bucket, + "Key": self._object_key(provider_object_ref), + } + if selected_range.offset or expected != head.byte_count: + request["Range"] = f"bytes={selected_range.offset}-{end}" + try: + async with asyncio.timeout(self._operation_total_timeout_seconds): + async with self._client() as client: + response = await client.get_object(**request) + body = response["Body"] + observed = 0 + async with body: + while True: + chunk = await body.read( + min(self._buffer_bytes, expected - observed + 1) + ) + if not chunk: + break + if not isinstance(chunk, bytes): + raise ArtifactIntegrityError( + "S3 artifact read returned invalid bytes" + ) + observed += len(chunk) + if observed > expected: + raise ArtifactIntegrityError( + "S3 artifact read exceeded its bound" + ) + yield chunk + if observed != expected: + raise ArtifactIntegrityError("S3 artifact object was truncated") + except ClientError as error: + code = _provider_error_code(error) + if code in _MISSING_ERROR_CODES: + raise ArtifactObjectMissingError("artifact object is missing") from None + raise ArtifactStoreUnavailableError("S3 artifact read failed") from None + except ArtifactStoreError: + raise + except (TimeoutError, BotoCoreError, OSError): + raise ArtifactStoreUnavailableError("S3 artifact read failed") from None + + return iterate() + + async def head(self, provider_object_ref: str) -> ArtifactObjectHead: + """Return exact size or authoritative absence without provider details.""" + self._require_initialized() + self._parse_provider_object_ref(provider_object_ref) + try: + async with asyncio.timeout(self._operation_total_timeout_seconds): + async with self._client() as client: + response = await client.head_object( + Bucket=self._bucket, + Key=self._object_key(provider_object_ref), + ) + byte_count = response.get("ContentLength") + if type(byte_count) is not int or byte_count < 0: + raise ArtifactIntegrityError("S3 artifact head lacks exact size") + return ArtifactObjectHead( + provider_object_ref, + exists=True, + byte_count=byte_count, + ) + except ClientError as error: + code = _provider_error_code(error) + if code in _MISSING_ERROR_CODES: + return ArtifactObjectHead(provider_object_ref, exists=False) + raise ArtifactStoreUnavailableError("S3 artifact head failed") from None + except ArtifactStoreError: + raise + except (TimeoutError, BotoCoreError, OSError): + raise ArtifactStoreUnavailableError("S3 artifact head failed") from None + + async def _verify_exact( + self, + provider_object_ref: str, + commitment: ArtifactCommitment, + ) -> None: + """Independently hash and count one complete provider object.""" + digest = hashlib.sha256() + byte_count = 0 + async for chunk in self.open(provider_object_ref): + digest.update(chunk) + byte_count += len(chunk) + if ( + byte_count != commitment.byte_count + or f"sha256:{digest.hexdigest()}" != commitment.sha256 + ): + raise ArtifactIntegrityError("S3 artifact object violates commitment") + + async def _matches_commitment( + self, + provider_object_ref: str, + commitment: ArtifactCommitment, + ) -> bool: + """Resolve an uncertain write only from an independently verified object.""" + try: + observed = await self.head(provider_object_ref) + if not observed.exists: + return False + await self._verify_exact(provider_object_ref, commitment) + except ArtifactObjectMissingError: + return False + return True + + @asynccontextmanager + async def _client(self) -> AsyncIterator[Any]: + """Create one bounded S3 client from the already-isolated session.""" + async with asyncio.timeout(self._pool_timeout_seconds): + context = self._session.create_client( + "s3", + region_name=self._region, + endpoint_url=self._endpoint_url, + config=self._client_config, + ) + client = await context.__aenter__() + try: + yield client + finally: + await context.__aexit__(None, None, None) + + def _require_initialized(self) -> None: + """Reject all operations before exact namespace admission.""" + if not self._initialized: + raise ArtifactConfigurationError("artifact store namespace is not initialized") + + def _object_key(self, provider_object_ref: str) -> str: + """Apply only the configured private prefix to a digest-derived reference.""" + self._parse_provider_object_ref(provider_object_ref) + return f"{self._private_prefix}/{provider_object_ref}" + + @staticmethod + def _provider_object_ref(commitment: ArtifactCommitment) -> str: + """Derive one identity-free reference solely from the server commitment.""" + digest_hex = commitment.sha256[7:] + return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" + + @staticmethod + def _parse_provider_object_ref(provider_object_ref: str) -> None: + """Reject caller-selected keys outside the canonical digest grammar.""" + if ( + not isinstance(provider_object_ref, str) + or _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) is None + ): + raise ArtifactOperationConflictError("artifact provider reference is invalid") + + +class _CommittedSourceBody: + """Single-use async request body that revalidates the sealed commitment.""" + + def __init__(self, source: CommittedArtifactSource, buffer_bytes: int) -> None: + """Bind one sealed source to one bounded second-pass request body.""" + self._source = source + self._commitment = source.commitment + self._buffer_bytes = buffer_bytes + self._consumed = False + self._complete = False + self._iterator: AsyncIterator[bytes] | None = None + self._pending = bytearray() + self._digest = hashlib.sha256() + self._byte_count = 0 + + def __aiter__(self) -> AsyncIterator[bytes]: + """Yield bounded bytes while checking the exact digest and count.""" + if self._consumed: + raise ArtifactInputMismatchError("artifact source stream was already consumed") + self._consumed = True + + async def iterate() -> AsyncIterator[bytes]: + """Yield the source until its commitment and EOF are proven.""" + while chunk := await self.read(self._buffer_bytes): + yield chunk + + return iterate() + + @property + def complete(self) -> bool: + """Return whether the sealed second-pass stream was fully validated.""" + return self._complete + + async def read(self, amount: int | None = -1) -> bytes: + """Return one bounded async file-like chunk for the pinned SDK.""" + if not self._consumed: + self._consumed = True + if self._complete: + return b"" + requested = self._buffer_bytes if amount is None or amount < 0 else amount + if requested == 0: + return b"" + requested = min(requested, self._buffer_bytes) + if self._iterator is None: + self._iterator = self._source.stream() + while len(self._pending) < requested: + try: + source_chunk = await anext(self._iterator) + except StopAsyncIteration: + self._finish() + break + if not isinstance(source_chunk, bytes): + raise ArtifactInputMismatchError("artifact source must yield bytes") + self._pending.extend(source_chunk) + if len(self._pending) > self._buffer_bytes + requested: + break + chunk = bytes(self._pending[:requested]) + del self._pending[:requested] + self._byte_count += len(chunk) + if self._byte_count > self._commitment.byte_count: + raise ArtifactInputMismatchError("artifact source exceeds committed byte count") + self._digest.update(chunk) + if self._byte_count == self._commitment.byte_count and not self._pending: + await self._require_source_exhausted() + if not chunk and not self._complete: + self._finish() + return chunk + + async def validate_empty(self) -> None: + """Validate a zero-byte source that the SDK need not consume.""" + if self._commitment.byte_count != 0: + raise ArtifactInputMismatchError("artifact source is not empty") + if await self.read(1) != b"" or not self._complete: + raise ArtifactInputMismatchError("artifact source violates commitment") + + async def _require_source_exhausted(self) -> None: + """Prove EOF before returning the final committed byte to the SDK.""" + if self._iterator is None: + self._iterator = self._source.stream() + while True: + try: + extra = await anext(self._iterator) + except StopAsyncIteration: + self._finish() + return + if not isinstance(extra, bytes): + raise ArtifactInputMismatchError("artifact source must yield bytes") + if extra: + raise ArtifactInputMismatchError( + "artifact source exceeds committed byte count" + ) + + def _finish(self) -> None: + """Validate commitment only after every pending byte is consumed.""" + if self._pending: + return + if ( + self._byte_count != self._commitment.byte_count + or f"sha256:{self._digest.hexdigest()}" != self._commitment.sha256 + ): + raise ArtifactInputMismatchError("artifact source violates commitment") + self._complete = True + +def create_minio_artifact_store_bootstrap( + settings: Settings, +) -> S3CompatibleArtifactStoreBootstrap: + """Construct the only S3 runtime profile enabled by this chunk.""" + if settings.artifact_s3_provider_profile != "minio": + raise ArtifactConfigurationError("MinIO artifact profile is not configured") + access_key = settings.artifact_s3_access_key_id + secret_key = settings.artifact_s3_secret_access_key + if access_key is None or secret_key is None: + raise ArtifactConfigurationError("MinIO artifact credentials are unavailable") + session = AioSession() + session.set_credentials( + access_key.get_secret_value(), + secret_key.get_secret_value(), + settings.artifact_s3_session_token.get_secret_value() + if settings.artifact_s3_session_token is not None + else None, + ) + return S3CompatibleArtifactStoreBootstrap( + S3CompatibleArtifactStore( + provider_profile="minio", + region=_required(settings.artifact_s3_region), + endpoint_url=_required(settings.artifact_s3_endpoint_url), + bucket=_required(settings.artifact_s3_bucket), + private_prefix=settings.artifact_s3_private_prefix, + addressing_style=settings.artifact_s3_addressing_style, + session=session, + buffer_bytes=settings.artifact_stream_buffer_bytes, + connect_timeout_seconds=settings.artifact_s3_connect_timeout_seconds, + read_timeout_seconds=settings.artifact_s3_read_timeout_seconds, + write_timeout_seconds=settings.artifact_s3_write_timeout_seconds, + pool_timeout_seconds=settings.artifact_s3_pool_timeout_seconds, + operation_total_timeout_seconds=( + settings.artifact_s3_operation_total_timeout_seconds + ), + max_pool_connections=settings.artifact_s3_max_pool_connections, + ) + ) + + +def validate_aws_workload_identity_environment( + settings: Settings, + *, + environ: Mapping[str, str] | None = None, +) -> None: + """Reject every ambient or unselected AWS credential source without loading it.""" + if settings.artifact_s3_provider_profile != "aws_s3": + raise ArtifactConfigurationError("native AWS S3 profile is not configured") + selected = settings.artifact_s3_aws_workload_identity_method + if selected is None: + raise ArtifactConfigurationError("AWS workload identity method is unavailable") + environment = os.environ if environ is None else environ + if _FORBIDDEN_AWS_CREDENTIAL_ENVIRONMENT.intersection(environment): + raise ArtifactConfigurationError("ambient AWS credential source is forbidden") + _reject_default_credential_files(environment) + + allowed_by_method = { + "assume-role-with-web-identity": _WEB_IDENTITY_ENVIRONMENT, + "container-role": _CONTAINER_ENVIRONMENT, + "iam-role": _IAM_ROLE_ENVIRONMENT, + } + unselected = set().union( + *(values for method, values in allowed_by_method.items() if method != selected) + ) + if unselected.intersection(environment): + raise ArtifactConfigurationError("unselected AWS workload source is configured") + if selected == "assume-role-with-web-identity": + if not {"AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"}.issubset(environment): + raise ArtifactConfigurationError("AWS web identity configuration is incomplete") + token_path = Path(environment["AWS_WEB_IDENTITY_TOKEN_FILE"]) + if not token_path.is_absolute() or not token_path.is_file(): + raise ArtifactConfigurationError("AWS web identity token path is invalid") + elif selected == "container-role": + locations = _CONTAINER_ENVIRONMENT.intersection(environment) & { + "AWS_CONTAINER_CREDENTIALS_FULL_URI", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", + } + if len(locations) != 1: + raise ArtifactConfigurationError("AWS container identity location is invalid") + auth = _CONTAINER_ENVIRONMENT.intersection(environment) & { + "AWS_CONTAINER_AUTHORIZATION_TOKEN", + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", + } + if len(auth) > 1: + raise ArtifactConfigurationError("AWS container identity token is ambiguous") + elif environment.get("AWS_EC2_METADATA_DISABLED", "false").lower() == "true": + raise ArtifactConfigurationError("AWS instance identity metadata is disabled") + + +def create_isolated_aws_workload_identity_session( + settings: Settings, + *, + environ: Mapping[str, str] | None = None, +) -> AioSession: + """Build an inactive SDK session whose resolver contains one selected provider.""" + validate_aws_workload_identity_environment(settings, environ=environ) + environment = os.environ if environ is None else environ + selected = settings.artifact_s3_aws_workload_identity_method + session = AioSession() + if selected == "assume-role-with-web-identity": + profile = { + "role_arn": environment["AWS_ROLE_ARN"], + "web_identity_token_file": environment["AWS_WEB_IDENTITY_TOKEN_FILE"], + } + role_session_name = environment.get("AWS_ROLE_SESSION_NAME") + if role_session_name is not None: + profile["role_session_name"] = role_session_name + provider = AioAssumeRoleWithWebIdentityProvider( + load_config=lambda: {"profiles": {"workstream-isolated": profile}}, + client_creator=session.create_client, + profile_name="workstream-isolated", + cache={}, + disable_env_vars=True, + ) + elif selected == "container-role": + provider = AioContainerProvider(environ=environment) + elif selected == "iam-role": + provider = AioInstanceMetadataProvider( + iam_role_fetcher=AioInstanceMetadataFetcher( + timeout=settings.artifact_s3_connect_timeout_seconds, + num_attempts=1, + env=dict(environment), + user_agent=session.user_agent(), + config={ + "ec2_metadata_service_endpoint": environment.get( + "AWS_EC2_METADATA_SERVICE_ENDPOINT" + ), + "ec2_metadata_service_endpoint_mode": environment.get( + "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE" + ), + "ec2_metadata_v1_disabled": environment.get( + "AWS_EC2_METADATA_V1_DISABLED", "false" + ).lower() + == "true", + }, + ) + ) + else: + raise ArtifactConfigurationError("AWS workload identity method is unsupported") + session.register_component("credential_provider", AioCredentialResolver([provider])) + return session + + +async def resolve_isolated_aws_workload_credentials( + session: AioSession, + *, + expected_method: str, +) -> object: + """Resolve credentials and require the exact selected SDK method.""" + credentials = await session.get_credentials() + if credentials is None or getattr(credentials, "method", None) != expected_method: + raise ArtifactConfigurationError("AWS workload identity method did not match") + return credentials + + +def _reject_default_credential_files(environ: Mapping[str, str]) -> None: + """Fail on default shared/config/Boto files without reading their contents.""" + home = Path(environ.get("HOME", str(Path.home()))) + for path in ( + home / ".aws" / "credentials", + home / ".aws" / "config", + home / ".boto", + Path("/etc/boto.cfg"), + ): + if path.exists(): + raise ArtifactConfigurationError("ambient AWS credential file is forbidden") + + +def _provider_error_code(error: ClientError) -> str: + """Return one bounded provider status/code without retaining response details.""" + response = error.response if isinstance(error.response, dict) else {} + provider_error = response.get("Error") if isinstance(response, dict) else None + code = provider_error.get("Code") if isinstance(provider_error, dict) else None + if isinstance(code, str) and code: + return code + metadata = response.get("ResponseMetadata") if isinstance(response, dict) else None + status = metadata.get("HTTPStatusCode") if isinstance(metadata, dict) else None + return str(status) if type(status) is int else "unknown" + + +def _required(value: str | None) -> str: + """Return one settings value already guaranteed by validation.""" + if not isinstance(value, str) or not value: + raise ArtifactConfigurationError("S3-compatible artifact setting is unavailable") + return value diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 8685880b..ee90ce07 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -6,10 +6,12 @@ import binascii import json import os +import re from collections.abc import Mapping from functools import lru_cache from pathlib import Path from typing import Literal, Self +from urllib.parse import urlsplit from dotenv import dotenv_values from pydantic import Field, PrivateAttr, SecretStr, model_validator @@ -21,6 +23,18 @@ ) +_ARTIFACT_S3_SECRET_FIELDS = frozenset( + { + "artifact_s3_access_key_id", + "artifact_s3_secret_access_key", + "artifact_s3_session_token", + } +) +_S3_REGION = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$") +_S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") +_S3_PREFIX_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") + + class Settings(BaseSettings): """Runtime configuration for the Workstream API.""" @@ -93,6 +107,31 @@ class Settings(BaseSettings): api_admin_mutation_rate_window_seconds: int = Field(default=60, ge=1, le=3_600) artifact_store_backend: Literal["disabled", "local", "s3_compatible"] = "disabled" artifact_local_root: Path | None = None + artifact_s3_provider_profile: Literal["aws_s3", "minio"] | None = None + artifact_s3_region: str | None = None + artifact_s3_endpoint_url: str | None = None + artifact_s3_bucket: str | None = None + artifact_s3_private_prefix: str = "workstream/artifacts" + artifact_s3_addressing_style: Literal["path", "virtual"] = "virtual" + artifact_s3_credential_mode: Literal[ + "aws_workload_identity", + "local_static", + ] | None = None + artifact_s3_aws_workload_identity_method: Literal[ + "assume-role-with-web-identity", + "container-role", + "iam-role", + ] | None = None + artifact_s3_connect_timeout_seconds: float = Field(default=5.0, gt=0.0, le=60.0) + artifact_s3_read_timeout_seconds: float = Field(default=60.0, gt=0.0, le=1800.0) + artifact_s3_write_timeout_seconds: float = Field(default=1800.0, gt=0.0, le=3600.0) + artifact_s3_pool_timeout_seconds: float = Field(default=5.0, gt=0.0, le=60.0) + artifact_s3_operation_total_timeout_seconds: float = Field( + default=1800.0, + gt=0.0, + le=3600.0, + ) + artifact_s3_max_pool_connections: int = Field(default=16, ge=1, le=256) artifact_maximum_bytes: int = Field(default=512 * 1024 * 1024, gt=0) artifact_stream_buffer_bytes: int = Field(default=1024 * 1024, gt=0, le=1024 * 1024) artifact_operation_lock_timeout_seconds: float = Field( @@ -140,22 +179,43 @@ class Settings(BaseSettings): ) _api_rate_limit_key_secret: SecretStr | None = PrivateAttr(default=None) + _artifact_s3_access_key_id: SecretStr | None = PrivateAttr(default=None) + _artifact_s3_secret_access_key: SecretStr | None = PrivateAttr(default=None) + _artifact_s3_session_token: SecretStr | None = PrivateAttr(default=None) def __init__(self, **values: object) -> None: """Remove rate-control secret material before structured validation.""" secret = _extract_api_rate_limit_key_secret(values) + s3_secrets = _extract_artifact_s3_static_secrets(values) super().__init__(**values) self._api_rate_limit_key_secret = secret + self._set_artifact_s3_static_secrets(s3_secrets) + self._validate_artifact_s3_secret_contract() @classmethod def model_validate(cls, obj: object, **kwargs: object) -> Self: """Sanitize secret-bearing mappings before Pydantic retains input.""" - if isinstance(obj, Mapping) and "api_rate_limit_key_secret" in obj: + if isinstance(obj, Mapping) and ( + "api_rate_limit_key_secret" in obj + or _ARTIFACT_S3_SECRET_FIELDS.intersection(obj) + ): sanitized = dict(obj) - secret = _extract_api_rate_limit_key_secret(sanitized) - sanitized["api_rate_limit_key_secret"] = None + secret = ( + _extract_api_rate_limit_key_secret(sanitized) + if "api_rate_limit_key_secret" in sanitized + else None + ) + s3_secrets = _extract_artifact_s3_static_secrets(sanitized) + if secret is not None: + sanitized["api_rate_limit_key_secret"] = None + for field_name in _ARTIFACT_S3_SECRET_FIELDS: + if field_name in obj: + sanitized[field_name] = None settings = super().model_validate(sanitized, **kwargs) - settings._api_rate_limit_key_secret = secret + if secret is not None: + settings._api_rate_limit_key_secret = secret + settings._set_artifact_s3_static_secrets(s3_secrets) + settings._validate_artifact_s3_secret_contract() return settings return super().model_validate(obj, **kwargs) @@ -174,19 +234,37 @@ def model_validate_json( parsed = None if malformed: raise ValueError("invalid settings JSON") - if isinstance(parsed, Mapping) and "api_rate_limit_key_secret" in parsed: + if isinstance(parsed, Mapping) and ( + "api_rate_limit_key_secret" in parsed + or _ARTIFACT_S3_SECRET_FIELDS.intersection(parsed) + ): return cls.model_validate(parsed, **kwargs) return super().model_validate_json(json_data, **kwargs) @classmethod def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: """Sanitize string-mapping secret input before structured validation.""" - if isinstance(obj, Mapping) and "api_rate_limit_key_secret" in obj: + if isinstance(obj, Mapping) and ( + "api_rate_limit_key_secret" in obj + or _ARTIFACT_S3_SECRET_FIELDS.intersection(obj) + ): sanitized = dict(obj) - secret = _extract_api_rate_limit_key_secret(sanitized) - sanitized["api_rate_limit_key_secret"] = None + secret = ( + _extract_api_rate_limit_key_secret(sanitized) + if "api_rate_limit_key_secret" in sanitized + else None + ) + s3_secrets = _extract_artifact_s3_static_secrets(sanitized) + if secret is not None: + sanitized["api_rate_limit_key_secret"] = None + for field_name in _ARTIFACT_S3_SECRET_FIELDS: + if field_name in obj: + sanitized[field_name] = None settings = super().model_validate_strings(sanitized, **kwargs) - settings._api_rate_limit_key_secret = secret + if secret is not None: + settings._api_rate_limit_key_secret = secret + settings._set_artifact_s3_static_secrets(s3_secrets) + settings._validate_artifact_s3_secret_contract() return settings return super().model_validate_strings(obj, **kwargs) @@ -202,6 +280,8 @@ def settings_customise_sources( """Keep the manually resolved rate key out of dotenv validation input.""" if isinstance(dotenv_settings, DotEnvSettingsSource): dotenv_settings.env_vars.pop("workstream_api_rate_limit_key_secret", None) + for field_name in _ARTIFACT_S3_SECRET_FIELDS: + dotenv_settings.env_vars.pop(f"workstream_{field_name}", None) return init_settings, env_settings, dotenv_settings, file_secret_settings @property @@ -209,6 +289,53 @@ def api_rate_limit_key_secret(self) -> SecretStr | None: """Return the validated, redacted rate-control key setting.""" return self._api_rate_limit_key_secret + @property + def artifact_s3_access_key_id(self) -> SecretStr | None: + """Return the local/CI MinIO access-key identifier.""" + return self._artifact_s3_access_key_id + + @property + def artifact_s3_secret_access_key(self) -> SecretStr | None: + """Return the local/CI MinIO secret access key.""" + return self._artifact_s3_secret_access_key + + @property + def artifact_s3_session_token(self) -> SecretStr | None: + """Return the optional local/CI MinIO session token.""" + return self._artifact_s3_session_token + + def _set_artifact_s3_static_secrets( + self, + secrets: tuple[SecretStr | None, SecretStr | None, SecretStr | None], + ) -> None: + """Retain static MinIO credentials only as redacted private values.""" + ( + self._artifact_s3_access_key_id, + self._artifact_s3_secret_access_key, + self._artifact_s3_session_token, + ) = secrets + + def _validate_artifact_s3_secret_contract(self) -> None: + """Reject static credentials outside a complete local MinIO profile.""" + secrets = ( + self.artifact_s3_access_key_id, + self.artifact_s3_secret_access_key, + self.artifact_s3_session_token, + ) + if self.artifact_store_backend != "s3_compatible" and any( + value is not None for value in secrets + ): + raise ValueError("static artifact credentials require MinIO storage") + if self.artifact_s3_provider_profile == "minio" and ( + self.artifact_s3_access_key_id is None + or self.artifact_s3_secret_access_key is None + ): + raise ValueError("MinIO artifact storage requires complete static credentials") + if self.artifact_s3_provider_profile == "aws_s3" and any( + value is not None for value in secrets + ): + raise ValueError("native AWS S3 rejects static credentials") + @model_validator(mode="after") def validate_artifact_storage(self) -> Settings: """Reject unsafe or incomplete artifact-storage combinations. @@ -247,8 +374,46 @@ def validate_artifact_storage(self) -> Settings: raise ValueError("local artifact storage is restricted to development and test") if self.artifact_local_root is None: raise ValueError("local artifact storage requires an artifact root") + return self + self._validate_s3_compatible_storage() return self + def _validate_s3_compatible_storage(self) -> None: + """Require one closed MinIO or native AWS configuration profile.""" + if self.artifact_s3_provider_profile is None: + raise ValueError("S3-compatible artifact storage requires a provider profile") + if not _is_canonical_s3_region(self.artifact_s3_region): + raise ValueError("S3-compatible artifact storage requires a canonical region") + if not _is_canonical_s3_bucket(self.artifact_s3_bucket): + raise ValueError("S3-compatible artifact storage requires a canonical bucket") + if not _is_canonical_s3_prefix(self.artifact_s3_private_prefix): + raise ValueError("S3-compatible artifact storage private prefix is invalid") + if self.artifact_maximum_bytes > 512 * 1024 * 1024: + raise ValueError("S3-compatible artifact maximum exceeds 512 MiB") + if self.artifact_s3_provider_profile == "minio": + self._validate_minio_storage() + return + self._validate_native_aws_storage() + + def _validate_minio_storage(self) -> None: + """Keep static MinIO credentials and endpoints local to development/CI.""" + if self.environment not in {"local", "dev", "development", "test"}: + raise ValueError("MinIO artifact storage is restricted to development and test") + if self.artifact_s3_credential_mode != "local_static": + raise ValueError("MinIO artifact storage requires local static credentials") + if self.artifact_s3_aws_workload_identity_method is not None: + raise ValueError("MinIO artifact storage cannot select AWS workload identity") + _validate_minio_endpoint(self.artifact_s3_endpoint_url) + + def _validate_native_aws_storage(self) -> None: + """Validate native AWS configuration without activating its runtime.""" + if self.artifact_s3_endpoint_url is not None: + raise ValueError("native AWS S3 does not accept a configured endpoint") + if self.artifact_s3_credential_mode != "aws_workload_identity": + raise ValueError("native AWS S3 requires workload identity") + if self.artifact_s3_aws_workload_identity_method is None: + raise ValueError("native AWS S3 requires one workload identity method") + @lru_cache def get_settings() -> Settings: @@ -286,6 +451,99 @@ def _extract_api_rate_limit_key_secret(values: dict[str, object]) -> SecretStr | return secret +def _extract_artifact_s3_static_secrets( + values: dict[str, object], +) -> tuple[SecretStr | None, SecretStr | None, SecretStr | None]: + """Remove optional MinIO credentials from structured settings input.""" + return tuple( + _extract_optional_secret( + values, + field_name=field_name, + environment_name=f"WORKSTREAM_{field_name.upper()}", + ) + for field_name in ( + "artifact_s3_access_key_id", + "artifact_s3_secret_access_key", + "artifact_s3_session_token", + ) + ) # type: ignore[return-value] + + +def _extract_optional_secret( + values: dict[str, object], + *, + field_name: str, + environment_name: str, +) -> SecretStr | None: + """Resolve one bounded secret without retaining it in Pydantic input.""" + missing = object() + raw_value = values.pop(field_name, missing) + if raw_value is missing: + raw_value = os.environ.get(environment_name, missing) + if raw_value is missing: + env_file = values.get("_env_file", ".env") + if env_file is not None: + env_encoding = values.get("_env_file_encoding", "utf-8") + env_files = (env_file,) if isinstance(env_file, (str, os.PathLike)) else tuple(env_file) + for path in env_files: + dotenv = dotenv_values(path, encoding=env_encoding) + raw_value = dotenv.get(environment_name, raw_value) + if raw_value is missing or raw_value is None: + return None + if isinstance(raw_value, SecretStr): + secret = raw_value + elif isinstance(raw_value, str): + secret = SecretStr(raw_value) + else: + raise ValueError("invalid artifact storage secret") + revealed = secret.get_secret_value() + if ( + not revealed + or len(revealed) > 4096 + or any(ord(character) < 32 or ord(character) == 127 for character in revealed) + ): + raise ValueError("invalid artifact storage secret") + return secret + + +def _is_canonical_s3_region(value: str | None) -> bool: + """Return whether one region is bounded and canonical.""" + return isinstance(value, str) and _S3_REGION.fullmatch(value) is not None + + +def _is_canonical_s3_bucket(value: str | None) -> bool: + """Return whether one dedicated bucket name is DNS-compatible.""" + if not isinstance(value, str) or _S3_BUCKET.fullmatch(value) is None: + return False + return ".." not in value and not re.fullmatch(r"\d+\.\d+\.\d+\.\d+", value) + + +def _is_canonical_s3_prefix(value: str) -> bool: + """Return whether one private object prefix has canonical path segments.""" + if not isinstance(value, str) or not value or len(value) > 512: + return False + segments = value.split("/") + return all(_S3_PREFIX_SEGMENT.fullmatch(segment) is not None for segment in segments) + + +def _validate_minio_endpoint(value: str | None) -> None: + """Require one canonical noncredentialed HTTP(S) MinIO origin.""" + if not isinstance(value, str) or not value or len(value) > 2048: + raise ValueError("MinIO artifact storage requires an endpoint") + parsed = urlsplit(value) + if ( + parsed.scheme not in {"http", "https"} + or not parsed.hostname + or parsed.username is not None + or parsed.password is not None + or parsed.query + or parsed.fragment + or parsed.path not in {"", "/"} + or value.endswith("/") + ): + raise ValueError("MinIO artifact storage endpoint is invalid") + + def decode_api_rate_limit_key_secret(value: SecretStr) -> bytes: """Decode one canonical padded Base64 rate-control key.""" secret = value.get_secret_value() diff --git a/backend/app/interfaces/artifacts.py b/backend/app/interfaces/artifacts.py index 9fbd6aa6..3b8189a7 100644 --- a/backend/app/interfaces/artifacts.py +++ b/backend/app/interfaces/artifacts.py @@ -21,6 +21,23 @@ ) _NAMESPACE_DESCRIPTOR_KEYS_BY_PROFILE = { "local-v2": frozenset({"private_prefix", "private_root_identity"}), + "minio-v1": frozenset( + { + "addressing_style", + "bucket", + "endpoint_identity", + "private_prefix", + "region", + } + ), + "aws-s3-v1": frozenset( + { + "addressing_style", + "bucket", + "private_prefix", + "region", + } + ), } @@ -253,6 +270,12 @@ class ArtifactConfigurationError(ArtifactStoreError): category = "configuration" +class ArtifactProviderLiveProofRequiredError(ArtifactConfigurationError): + """Raised while an AWS profile lacks the later live activation proof.""" + + code = "artifact_provider_live_proof_required" + + class ArtifactStore(ExternalServiceAdapter, Protocol): """Immutable byte-provider capability used only by artifact orchestration.""" diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 505e2e56..9b33baf3 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -4,8 +4,10 @@ version = "0.1.0" description = "Workstream backend API" requires-python = ">=3.11" dependencies = [ + "aiobotocore==3.7.0", "alembic>=1.13,<2.0", "asyncpg>=0.29,<1.0", + "botocore==1.43.0", "celery[redis]>=5.4,<6.0", "fastapi>=0.115,<1.0", "httpx>=0.27,<1.0", diff --git a/backend/tests/test_artifact_architecture.py b/backend/tests/test_artifact_architecture.py index 2ad35c0d..65aed7da 100644 --- a/backend/tests/test_artifact_architecture.py +++ b/backend/tests/test_artifact_architecture.py @@ -12,6 +12,7 @@ APP_ROOT = BACKEND_ROOT / "app" ARTIFACT_OPERATIONS = APP_ROOT / "interfaces" / "artifact_operations.py" COMPOSITION_ROOT = APP_ROOT / "adapters" / "artifacts" / "__init__.py" +S3_ADAPTER_MODULE = APP_ROOT / "adapters" / "artifacts" / "s3_compatible.py" CLOSED_PORTS = { "GuideArtifactIngestPort", "ContributorArtifactUploadPort", @@ -133,8 +134,37 @@ def test_concrete_adapter_construction_has_one_composition_path() -> None: adapter_calls.append(path) assert factory_calls == [COMPOSITION_ROOT] - assert adapter_calls == [COMPOSITION_ROOT] - assert concrete_imports == [COMPOSITION_ROOT] + assert adapter_calls == [COMPOSITION_ROOT, S3_ADAPTER_MODULE] + assert set(concrete_imports) == {COMPOSITION_ROOT} + + +def test_s3_adapter_exposes_only_required_immutable_object_operations() -> None: + """Keep list, delete, copy, ACL, and multipart behavior out of the provider.""" + allowed = {"get_object", "head_object", "put_object"} + forbidden = { + "abort_multipart_upload", + "complete_multipart_upload", + "copy_object", + "create_multipart_upload", + "delete_object", + "delete_objects", + "list_buckets", + "list_objects", + "list_objects_v2", + "put_object_acl", + "upload_part", + "upload_part_copy", + } + called = { + node.func.attr + for node in ast.walk(_tree(S3_ADAPTER_MODULE)) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and isinstance(node.func.value, ast.Name) + and node.func.value.id == "client" + } + assert called & forbidden == set() + assert called & allowed == allowed def test_provider_methods_stay_inside_artifact_orchestration_and_adapters() -> None: diff --git a/backend/tests/test_artifact_cleanup_wiring.py b/backend/tests/test_artifact_cleanup_wiring.py index e13646a9..1557d1a4 100644 --- a/backend/tests/test_artifact_cleanup_wiring.py +++ b/backend/tests/test_artifact_cleanup_wiring.py @@ -11,7 +11,7 @@ from app.adapters.artifacts import artifact_preparation_limits from app.core.config import Settings, get_settings from app.main import create_app -from app.interfaces.external_services import UnknownExternalServiceProviderError +from app.interfaces.artifacts import ArtifactProviderLiveProofRequiredError def _enabled_settings(tmp_path: Path, **changes: object) -> Settings: @@ -84,6 +84,11 @@ async def test_production_auth_validation_precedes_artifact_startup( environment="production", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method="container-role", ) app = create_app(settings) store = _Store(events) @@ -144,24 +149,37 @@ def unexpected(*_args: object, **_kwargs: object) -> None: pass -async def test_unimplemented_s3_startup_fails_through_typed_factory( +async def test_aws_s3_startup_requires_live_provider_proof( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, ) -> None: - """Keep real startup on the one ADR 0014 unavailable-provider path.""" + """Keep native AWS runtime-ineligible until the owned activation chunk.""" + import app.main as main_module + + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setenv( + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", + "/v2/credentials/workstream-runtime", + ) + monkeypatch.setattr(main_module, "build_auth_verifier", lambda _settings: None) app = create_app( Settings( - environment="test", + environment="production", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method="container-role", ) ) - with pytest.raises(UnknownExternalServiceProviderError) as caught: + with pytest.raises(ArtifactProviderLiveProofRequiredError) as caught: async with app.router.lifespan_context(app): pass - assert caught.value.identity is not None - assert caught.value.identity.provider_key == "s3_compatible" + assert caught.value.code == "artifact_provider_live_proof_required" async def test_namespace_failure_is_fail_closed_and_skips_cleanup( diff --git a/backend/tests/test_artifact_store_conformance.py b/backend/tests/test_artifact_store_conformance.py index 27acb8c8..d1df90af 100644 --- a/backend/tests/test_artifact_store_conformance.py +++ b/backend/tests/test_artifact_store_conformance.py @@ -259,6 +259,28 @@ def test_namespace_startup_values_are_closed_and_immutable() -> None: "private_prefix": "objects/sha256", "private_root_identity": "sha256:" + "0" * 64, } + for profile, items in ( + ( + "minio-v1", + ( + ("addressing_style", "path"), + ("bucket", "workstream-artifacts"), + ("endpoint_identity", "sha256:" + "1" * 64), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ( + "aws-s3-v1", + ( + ("addressing_style", "virtual"), + ("bucket", "workstream-artifacts"), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ): + assert ArtifactStoreNamespaceIdentity(profile, items).as_dict() == dict(items) for profile, items in ( ("", (("private_prefix", "objects/sha256"),)), @@ -282,6 +304,25 @@ def test_namespace_startup_values_are_closed_and_immutable() -> None: ("private_root_identity", "sha256:" + "0" * 64), ), ), + ( + "minio-v1", + ( + ("addressing_style", "path"), + ("bucket", "workstream-artifacts"), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ( + "aws-s3-v1", + ( + ("addressing_style", "virtual"), + ("bucket", "workstream-artifacts"), + ("endpoint_identity", "sha256:" + "1" * 64), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), ): with pytest.raises(ValueError): ArtifactStoreNamespaceIdentity(profile, items) diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py new file mode 100644 index 00000000..ec6f0f3f --- /dev/null +++ b/backend/tests/test_aws_credential_isolation.py @@ -0,0 +1,324 @@ +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest +from aiobotocore.credentials import ( + AioAssumeRoleWithWebIdentityProvider, + AioContainerProvider, + AioInstanceMetadataProvider, +) + +from app.adapters.artifacts import s3_compatible +from app.core.config import Settings +from app.interfaces.artifacts import ArtifactConfigurationError + + +def _aws_settings(tmp_path: Path, method: str = "container-role") -> Settings: + return Settings( + environment="production", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_private_prefix="workstream/artifacts", + artifact_s3_addressing_style="virtual", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method=method, # type: ignore[arg-type] + ) + + +def _container_environment(tmp_path: Path, **overrides: str) -> dict[str, str]: + environment = { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": "/v2/credentials/workstream-runtime", + } + environment.update(overrides) + return environment + + +def _web_identity_environment(tmp_path: Path, **overrides: str) -> dict[str, str]: + token = tmp_path / "web-identity-token" + token.write_text("token", encoding="utf-8") + environment = { + "HOME": str(tmp_path), + "AWS_ROLE_ARN": "arn:aws:iam::123456789012:role/workstream-runtime", + "AWS_ROLE_SESSION_NAME": "workstream-artifact-runtime", + "AWS_WEB_IDENTITY_TOKEN_FILE": str(token), + } + environment.update(overrides) + return environment + + +def _iam_environment(tmp_path: Path, **overrides: str) -> dict[str, str]: + environment = {"HOME": str(tmp_path)} + environment.update(overrides) + return environment + + +def _assert_rejects_before_session_construction( + monkeypatch: pytest.MonkeyPatch, + settings: Settings, + environment: dict[str, str], + match: str, +) -> None: + events: list[str] = [] + + def session_must_not_construct(*_args: object, **_kwargs: object) -> object: + events.append("session") + raise AssertionError("SDK session constructed before isolation rejection") + + monkeypatch.setattr(s3_compatible, "AioSession", session_must_not_construct) + + with pytest.raises(ArtifactConfigurationError, match=match): + s3_compatible.create_isolated_aws_workload_identity_session( + settings, + environ=environment, + ) + + assert events == [] + + +@pytest.mark.parametrize( + ("name", "value"), + [ + ("AWS_ACCESS_KEY", "legacy-access"), + ("AWS_ACCESS_KEY_ID", "explicit-access"), + ("AWS_SECRET_ACCESS_KEY", "explicit-secret"), + ("AWS_SESSION_TOKEN", "explicit-session-token"), + ("AWS_SECRET_KEY", "legacy-secret"), + ("AWS_SECURITY_TOKEN", "legacy-security-token"), + ("AWS_PROFILE", "prod"), + ("AWS_DEFAULT_PROFILE", "prod"), + ("AWS_SHARED_CREDENTIALS_FILE", "/tmp/credentials"), + ("AWS_CONFIG_FILE", "/tmp/config"), + ("AWS_CREDENTIAL_FILE", "/tmp/legacy-credentials"), + ("AWS_LOGIN_CACHE_DIRECTORY", "/tmp/aws-login-cache"), + ("BOTO_CONFIG", "/tmp/boto.cfg"), + ], +) +def test_forbidden_ambient_aws_sources_fail_before_resolver_loading( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, + value: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment(tmp_path, **{name: value}), + "ambient AWS credential source is forbidden", + ) + + +@pytest.mark.parametrize( + ("relative_path", "contents"), + [ + (".aws/credentials", "[default]\naws_access_key_id = poison\n"), + (".aws/config", "[profile prod]\ncredential_process = /bin/false\n"), + (".aws/config", "[profile prod]\nsso_start_url = https://example.test/start\n"), + (".boto", "[Credentials]\naws_access_key_id = poison\n"), + ], +) +def test_default_credential_files_fail_before_resolver_loading( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + relative_path: str, + contents: str, +) -> None: + poison = tmp_path / relative_path + poison.parent.mkdir(parents=True, exist_ok=True) + poison.write_text(contents, encoding="utf-8") + + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment(tmp_path), + "ambient AWS credential file is forbidden", + ) + + +@pytest.mark.parametrize( + ("selected_method", "environment"), + [ + ( + "assume-role-with-web-identity", + lambda tmp_path: _web_identity_environment( + tmp_path, + AWS_CONTAINER_CREDENTIALS_RELATIVE_URI="/v2/credentials/poison", + ), + ), + ( + "container-role", + lambda tmp_path: _container_environment( + tmp_path, + AWS_ROLE_ARN="arn:aws:iam::123456789012:role/unselected", + AWS_WEB_IDENTITY_TOKEN_FILE=str(tmp_path / "unselected-token"), + ), + ), + ( + "iam-role", + lambda tmp_path: _iam_environment( + tmp_path, + AWS_CONTAINER_CREDENTIALS_FULL_URI="http://127.0.0.1:9/credentials", + ), + ), + ], +) +def test_unselected_workload_identity_sources_fail_before_resolver_loading( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + selected_method: str, + environment: Any, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, selected_method), + environment(tmp_path), + "unselected AWS workload source is configured", + ) + + +@pytest.mark.parametrize( + ("method", "environment", "provider_type", "sdk_method"), + [ + ( + "assume-role-with-web-identity", + _web_identity_environment, + AioAssumeRoleWithWebIdentityProvider, + "assume-role-with-web-identity", + ), + ("container-role", _container_environment, AioContainerProvider, "container-role"), + ("iam-role", _iam_environment, AioInstanceMetadataProvider, "iam-role"), + ], +) +def test_isolated_session_contains_exactly_one_selected_provider( + tmp_path: Path, + method: str, + environment: Any, + provider_type: type[object], + sdk_method: str, +) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, method), + environ=environment(tmp_path), + ) + + resolver = session.get_component("credential_provider") + + assert len(resolver.providers) == 1 + provider = resolver.providers[0] + assert type(provider) is provider_type + assert provider.METHOD == sdk_method + if method == "assume-role-with-web-identity": + assert provider._disable_env_vars is True + assert provider._load_config() == { + "profiles": { + "workstream-isolated": { + "role_arn": "arn:aws:iam::123456789012:role/workstream-runtime", + "role_session_name": "workstream-artifact-runtime", + "web_identity_token_file": str(tmp_path / "web-identity-token"), + } + } + } + + +@pytest.mark.parametrize( + ("method", "environment", "message"), + [ + ( + "assume-role-with-web-identity", + lambda tmp_path: {"HOME": str(tmp_path), "AWS_ROLE_ARN": "arn"}, + "web identity configuration is incomplete", + ), + ( + "assume-role-with-web-identity", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_ROLE_ARN": "arn", + "AWS_WEB_IDENTITY_TOKEN_FILE": "relative-token", + }, + "web identity token path is invalid", + ), + ( + "container-role", + lambda tmp_path: {"HOME": str(tmp_path)}, + "container identity location is invalid", + ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_FULL_URI": "http://127.0.0.1:9/full", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": "/relative", + }, + "container identity location is invalid", + ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": "/relative", + "AWS_CONTAINER_AUTHORIZATION_TOKEN": "token", + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE": str(tmp_path / "token"), + }, + "container identity token is ambiguous", + ), + ( + "iam-role", + lambda tmp_path: {"HOME": str(tmp_path), "AWS_EC2_METADATA_DISABLED": "true"}, + "instance identity metadata is disabled", + ), + ], +) +def test_selected_workload_identity_source_must_be_complete( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + method: str, + environment: Any, + message: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, method), + environment(tmp_path), + message, + ) + + +async def test_resolved_workload_identity_method_must_match() -> None: + class FakeSession: + def __init__(self, method: str | None) -> None: + self._method = method + + async def get_credentials(self) -> object | None: + if self._method is None: + return None + return SimpleNamespace(method=self._method) + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity method did not match", + ): + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession("iam-role"), # type: ignore[arg-type] + expected_method="container-role", + ) + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity method did not match", + ): + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(None), # type: ignore[arg-type] + expected_method="container-role", + ) + + resolved = await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession("container-role"), # type: ignore[arg-type] + expected_method="container-role", + ) + assert getattr(resolved, "method") == "container-role" diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 114e6740..4bf18f68 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -1,7 +1,9 @@ from __future__ import annotations import base64 +import importlib.metadata import json +import tomllib from collections.abc import Mapping from pathlib import Path from typing import get_args @@ -15,6 +17,7 @@ from app.api.deps.auth import get_application_auth_verifier from app.core.auth import clear_auth_verifier_cache, get_auth_verifier from app.core.config import Settings, get_settings +from app.interfaces.artifacts import ArtifactProviderLiveProofRequiredError from app.interfaces.external_services import ( ExternalServiceConfigurationError, UnknownExternalServiceProviderError, @@ -355,6 +358,42 @@ def _flow_settings(**overrides) -> Settings: return Settings(**values) +def _minio_settings(tmp_path: Path, **overrides: object) -> Settings: + values: dict[str, object] = { + "environment": "test", + "artifact_store_backend": "s3_compatible", + "artifact_scratch_root": tmp_path / "scratch", + "artifact_s3_provider_profile": "minio", + "artifact_s3_region": "us-east-1", + "artifact_s3_endpoint_url": "http://127.0.0.1:9000", + "artifact_s3_bucket": "workstream-artifacts-test", + "artifact_s3_private_prefix": "workstream/artifacts", + "artifact_s3_addressing_style": "path", + "artifact_s3_credential_mode": "local_static", + "artifact_s3_access_key_id": "minio-access-key", + "artifact_s3_secret_access_key": "minio-secret-key", + } + values.update(overrides) + return Settings(**values) + + +def _aws_settings(tmp_path: Path, **overrides: object) -> Settings: + values: dict[str, object] = { + "environment": "production", + "artifact_store_backend": "s3_compatible", + "artifact_scratch_root": tmp_path / "scratch", + "artifact_s3_provider_profile": "aws_s3", + "artifact_s3_region": "us-east-1", + "artifact_s3_bucket": "workstream-artifacts-prod", + "artifact_s3_private_prefix": "workstream/artifacts", + "artifact_s3_addressing_style": "virtual", + "artifact_s3_credential_mode": "aws_workload_identity", + "artifact_s3_aws_workload_identity_method": "container-role", + } + values.update(overrides) + return Settings(**values) + + @pytest.mark.parametrize( ("overrides", "message"), [ @@ -551,15 +590,246 @@ def test_artifact_scratch_settings_are_bounded_and_separate(tmp_path) -> None: ) -def test_s3_compatible_factory_fails_typed_until_adapter_chunk(tmp_path: Path) -> None: - """Keep valid future S3 configuration unavailable until its owned chunk.""" - settings = Settings( - environment="test", - artifact_store_backend="s3_compatible", - artifact_scratch_root=tmp_path / "scratch", +def test_minio_settings_and_factory_construct_closed_namespace(tmp_path: Path) -> None: + """MinIO is the active S3-compatible runtime profile for local and CI proof.""" + settings = _minio_settings(tmp_path) + + bootstrap = create_artifact_store_bootstrap(settings) + + try: + assert bootstrap.identity.capability_key == "artifact_store" + assert bootstrap.identity.provider_key == "s3_compatible" + assert bootstrap.namespace_identity.provider_profile == "minio-v1" + assert bootstrap.namespace_identity.descriptor_items == ( + ("addressing_style", "path"), + ("bucket", "workstream-artifacts-test"), + ("endpoint_identity", bootstrap.namespace_identity.as_dict()["endpoint_identity"]), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ) + endpoint_identity = bootstrap.namespace_identity.as_dict()["endpoint_identity"] + assert endpoint_identity.startswith("sha256:") + assert "127.0.0.1" not in endpoint_identity + assert "minio-secret-key" not in repr(bootstrap.namespace_identity) + finally: + bootstrap.close() + + +@pytest.mark.parametrize( + ("overrides", "message"), + [ + ({"artifact_s3_provider_profile": None}, "requires a provider profile"), + ({"artifact_s3_region": "US-east-1"}, "requires a canonical region"), + ({"artifact_s3_bucket": "192.168.0.1"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket..name"}, "requires a canonical bucket"), + ({"artifact_s3_private_prefix": "/bad"}, "private prefix is invalid"), + ({"artifact_maximum_bytes": 512 * 1024 * 1024 + 1}, "maximum exceeds"), + ({"environment": "production"}, "MinIO artifact storage is restricted"), + ({"artifact_s3_credential_mode": "aws_workload_identity"}, "local static"), + ( + {"artifact_s3_aws_workload_identity_method": "iam-role"}, + "cannot select AWS workload identity", + ), + ({"artifact_s3_endpoint_url": None}, "requires an endpoint"), + ({"artifact_s3_endpoint_url": "https://user:pass@minio.test"}, "endpoint is invalid"), + ({"artifact_s3_endpoint_url": "http://minio.test/path"}, "endpoint is invalid"), + ({"artifact_s3_endpoint_url": "http://minio.test/"}, "endpoint is invalid"), + ], +) +def test_minio_settings_fail_closed( + tmp_path: Path, + overrides: dict[str, object], + message: str, +) -> None: + with pytest.raises((ValidationError, ValueError), match=message): + _minio_settings(tmp_path, **overrides) + + +@pytest.mark.parametrize( + ("removed", "message"), + [ + ("artifact_s3_access_key_id", "complete static credentials"), + ("artifact_s3_secret_access_key", "complete static credentials"), + ], +) +def test_minio_static_credentials_are_required( + tmp_path: Path, + removed: str, + message: str, +) -> None: + with pytest.raises(ValueError, match=message): + _minio_settings(tmp_path, **{removed: None}) + + +def test_minio_secret_values_are_absent_from_repr_and_validation_errors( + tmp_path: Path, +) -> None: + secret = "minio-secret-value-that-must-not-leak" + settings = _minio_settings(tmp_path, artifact_s3_secret_access_key=secret) + + assert settings.artifact_s3_secret_access_key is not None + assert settings.artifact_s3_secret_access_key.get_secret_value() == secret + assert secret not in repr(settings) + assert secret not in repr(settings.model_dump()) + + with pytest.raises(ValidationError) as caught: + _minio_settings( + tmp_path, + artifact_s3_secret_access_key=secret, + artifact_s3_region="US-east-1", + ) + + assert secret not in repr(caught.value.errors()) + assert secret not in caught.value.json() + _assert_secret_not_retained(caught.value, secret) + + +def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + env_secret = "env-minio-secret-value" + dotenv_secret = "dotenv-minio-secret-value" + env_file = tmp_path / ".env" + env_file.write_text( + "\n".join( + [ + "WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID=dotenv-access", + f"WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY={dotenv_secret}", + ] + ), + encoding="utf-8", ) - with pytest.raises(UnknownExternalServiceProviderError) as caught: + monkeypatch.setenv("WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID", "env-access") + monkeypatch.setenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY", env_secret) + + with pytest.raises(ValidationError) as env_error: + Settings( + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="minio", + artifact_s3_region="US-east-1", + artifact_s3_endpoint_url="http://127.0.0.1:9000", + artifact_s3_bucket="workstream-artifacts-test", + artifact_s3_credential_mode="local_static", + _env_file=env_file, + ) + assert env_secret not in repr(env_error.value.errors()) + _assert_secret_not_retained(env_error.value, env_secret) + + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID") + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY") + with pytest.raises(ValidationError) as dotenv_error: + Settings( + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="minio", + artifact_s3_region="US-east-1", + artifact_s3_endpoint_url="http://127.0.0.1:9000", + artifact_s3_bucket="workstream-artifacts-test", + artifact_s3_credential_mode="local_static", + _env_file=env_file, + ) + assert dotenv_secret not in repr(dotenv_error.value.errors()) + _assert_secret_not_retained(dotenv_error.value, dotenv_secret) + + +@pytest.mark.parametrize( + ("overrides", "message"), + [ + ({"artifact_s3_endpoint_url": "https://s3.us-east-1.amazonaws.com"}, "endpoint"), + ({"artifact_s3_credential_mode": "local_static"}, "workload identity"), + ({"artifact_s3_aws_workload_identity_method": None}, "one workload identity"), + ( + { + "artifact_s3_access_key_id": "static-access", + "artifact_s3_secret_access_key": "static-secret", + }, + "rejects static credentials", + ), + ], +) +def test_aws_s3_settings_fail_closed( + tmp_path: Path, + overrides: dict[str, object], + message: str, +) -> None: + with pytest.raises((ValidationError, ValueError), match=message): + _aws_settings(tmp_path, **overrides) + + +def test_aws_factory_fails_with_live_proof_before_constructing_provider( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + from app.adapters import artifacts as artifacts_module + from app.adapters.artifacts import s3_compatible + + settings = _aws_settings(tmp_path) + events: list[str] = [] + + class FactoryMustNotConstruct: + def __init__(self, *_args: object, **_kwargs: object) -> None: + events.append("factory") + raise AssertionError("factory constructed before AWS live proof failure") + + def session_must_not_construct(*_args: object, **_kwargs: object) -> object: + events.append("resolver") + raise AssertionError("credential resolver constructed before live proof failure") + + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setenv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "/v2/credentials/runtime") + monkeypatch.setattr(artifacts_module, "ExternalServiceAdapterFactory", FactoryMustNotConstruct) + monkeypatch.setattr(s3_compatible, "AioSession", session_must_not_construct) + + with pytest.raises(ArtifactProviderLiveProofRequiredError) as caught: create_artifact_store_bootstrap(settings) - assert caught.value.identity is not None - assert caught.value.identity.provider_key == "s3_compatible" + assert caught.value.code == "artifact_provider_live_proof_required" + assert events == [] + + +async def test_aws_lifespan_fails_before_namespace_claim_and_provider_io( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + import app.main as main_module + + settings = _aws_settings(tmp_path, environment="test") + events: list[str] = [] + + async def namespace_must_not_run(*_args: object, **_kwargs: object) -> object: + events.append("namespace") + raise AssertionError("namespace claim ran before AWS live proof failure") + + async def cleanup_must_not_run(*_args: object, **_kwargs: object) -> object: + events.append("cleanup") + raise AssertionError("provider cleanup ran before AWS live proof failure") + + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setenv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "/v2/credentials/runtime") + monkeypatch.setattr( + main_module, + "_validate_artifact_storage_namespace_at_startup", + namespace_must_not_run, + ) + monkeypatch.setattr(main_module, "cleanup_stale_artifact_scratch", cleanup_must_not_run) + app = create_app(settings) + + with pytest.raises(ArtifactProviderLiveProofRequiredError) as caught: + async with app.router.lifespan_context(app): + pass + + assert caught.value.code == "artifact_provider_live_proof_required" + assert events == [] + + +def test_s3_dependency_manifest_and_installed_versions_are_exact() -> None: + manifest = tomllib.loads((Path(__file__).parents[1] / "pyproject.toml").read_text()) + + assert "aiobotocore==3.7.0" in manifest["project"]["dependencies"] + assert "botocore==1.43.0" in manifest["project"]["dependencies"] + assert importlib.metadata.version("aiobotocore") == "3.7.0" + assert importlib.metadata.version("botocore") == "1.43.0" diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py new file mode 100644 index 00000000..2af7b24e --- /dev/null +++ b/backend/tests/test_s3_artifact_store.py @@ -0,0 +1,538 @@ +"""Real MinIO and focused protocol tests for S3CompatibleArtifactStore.""" + +from __future__ import annotations + +from contextlib import asynccontextmanager +import hashlib +import os +from pathlib import Path +from typing import Any + +from aiobotocore.config import AioConfig +from aiobotocore.session import AioSession +from botocore.exceptions import ClientError +import pytest + +from app.adapters.artifacts import create_artifact_store_bootstrap +from app.adapters.artifacts.s3_compatible import ( + S3CompatibleArtifactStore, + S3CompatibleArtifactStoreBootstrap, +) +import app.adapters.artifacts.s3_compatible as s3_module +from app.core.config import Settings +from app.interfaces.artifacts import ( + ArtifactByteRange, + ArtifactConfigurationError, + ArtifactIntegrityError, + ArtifactInputMismatchError, + ArtifactObjectMissingError, + ArtifactOperationConflictError, + ArtifactStore, + ArtifactStoreNamespaceClaim, + ArtifactStoreUnavailableError, + artifact_store_namespace_material, +) +from app.interfaces.external_services import ExternalServiceAdapterIdentity +from tests.artifact_store_helpers import minted_source +from tests.test_artifact_store_conformance import ArtifactStoreConformanceTests + + +MINIO_ENDPOINT = os.environ.get("WORKSTREAM_TEST_MINIO_ENDPOINT", "http://localhost:9000") +MINIO_REGION = "us-east-1" +MINIO_BUCKET = "workstream-artifacts" +MINIO_ACCESS_KEY = "workstream-minio" +MINIO_SECRET_KEY = "workstream-minio-secret-key" + + +@pytest.fixture(scope="module", autouse=True) +async def provision_minio_bucket() -> None: + """Provision only the dedicated integration bucket with MinIO admin credentials.""" + session = AioSession() + session.set_credentials(MINIO_ACCESS_KEY, MINIO_SECRET_KEY) + async with session.create_client( + "s3", + endpoint_url=MINIO_ENDPOINT, + region_name=MINIO_REGION, + config=AioConfig(s3={"addressing_style": "path"}), + ) as client: + try: + await client.create_bucket(Bucket=MINIO_BUCKET) + except ClientError as error: + if error.response.get("Error", {}).get("Code") not in { + "BucketAlreadyExists", + "BucketAlreadyOwnedByYou", + }: + raise + + +class TestMinioArtifactStoreConformance(ArtifactStoreConformanceTests): + """Run the same ArtifactStore v2 vectors against real MinIO.""" + + expected_identity = ExternalServiceAdapterIdentity( + "artifact_store", + "s3_compatible", + ) + + def make_store(self, root: Path) -> ArtifactStore: + """Construct one namespace-isolated real MinIO adapter.""" + namespace = hashlib.sha256(os.fspath(root.parent).encode()).hexdigest()[:20] + return initialize_minio_store(private_prefix=f"conformance/{namespace}") + + +def minio_settings( + *, + private_prefix: str, + access_key: str = MINIO_ACCESS_KEY, + secret_key: str = MINIO_SECRET_KEY, +) -> Settings: + """Return one complete local-only MinIO configuration.""" + return Settings( + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=Path("/tmp/workstream-test-artifact-scratch"), + artifact_s3_provider_profile="minio", + artifact_s3_region=MINIO_REGION, + artifact_s3_endpoint_url=MINIO_ENDPOINT, + artifact_s3_bucket=MINIO_BUCKET, + artifact_s3_private_prefix=private_prefix, + artifact_s3_addressing_style="path", + artifact_s3_credential_mode="local_static", + artifact_s3_access_key_id=access_key, + artifact_s3_secret_access_key=secret_key, + artifact_stream_buffer_bytes=2, + artifact_s3_operation_total_timeout_seconds=20, + artifact_s3_write_timeout_seconds=20, + ) + + +def initialize_minio_store(*, private_prefix: str) -> S3CompatibleArtifactStore: + """Construct, claim, and initialize one MinIO namespace.""" + bootstrap = create_artifact_store_bootstrap( + minio_settings(private_prefix=private_prefix) + ) + assert type(bootstrap) is S3CompatibleArtifactStoreBootstrap + namespace_identity = bootstrap.namespace_identity + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=bootstrap.identity, + namespace_identity=namespace_identity, + ) + return bootstrap.initialize_after_namespace_claim( + ArtifactStoreNamespaceClaim( + adapter_identity=bootstrap.identity, + namespace_identity=namespace_identity, + namespace_fingerprint=fingerprint, + ) + ) + + +@pytest.mark.asyncio +async def test_wrong_minio_credentials_map_403_to_unavailable() -> None: + """Never misclassify forbidden provider access as authoritative absence.""" + settings = minio_settings( + private_prefix="negative/forbidden", + access_key="unknown-minio-user", + secret_key="unknown-minio-password", + ) + bootstrap = create_artifact_store_bootstrap(settings) + namespace = bootstrap.namespace_identity + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=bootstrap.identity, + namespace_identity=namespace, + ) + store = bootstrap.initialize_after_namespace_claim( + ArtifactStoreNamespaceClaim(bootstrap.identity, namespace, fingerprint) + ) + try: + with pytest.raises(ArtifactStoreUnavailableError): + await store.head("sha256/00/" + "0" * 62) + finally: + bootstrap.close() + + +@pytest.mark.asyncio +async def test_adversarial_first_writer_cannot_be_accepted_as_replay( + tmp_path: Path, +) -> None: + """Reject existing wrong bytes at the server-derived content key.""" + private_prefix = f"adversarial/{hashlib.sha256(os.urandom(16)).hexdigest()[:20]}" + content = b"expected exact bytes" + digest = hashlib.sha256(content).hexdigest() + key = f"{private_prefix}/sha256/{digest[:2]}/{digest[2:]}" + await raw_minio_put(key, b"attacker-selected wrong bytes") + store = initialize_minio_store(private_prefix=private_prefix) + try: + async with minted_source(tmp_path / "scratch", content) as source: + with pytest.raises(ArtifactIntegrityError): + await store.put(source) + finally: + store.close() + + +@pytest.mark.asyncio +async def test_unsealed_client_selected_commitment_fails_before_provider_io( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Accept only the preparation service's inseparable source/commitment pair.""" + store = initialize_minio_store(private_prefix="negative/unsealed") + + @asynccontextmanager + async def unexpected_client() -> Any: + raise AssertionError("provider I/O occurred for an unsealed source") + yield + + monkeypatch.setattr(store, "_client", unexpected_client) + try: + with pytest.raises(ArtifactInputMismatchError, match="not sealed"): + await store.put(object()) # type: ignore[arg-type] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_hard_size_limit_fails_before_provider_io( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Reject an over-limit commitment before opening an S3 client.""" + store = initialize_minio_store(private_prefix="negative/oversized") + + @asynccontextmanager + async def unexpected_client() -> Any: + raise AssertionError("provider I/O occurred for an oversized source") + yield + + monkeypatch.setattr(store, "_client", unexpected_client) + monkeypatch.setattr(s3_module, "HARD_MAXIMUM_ARTIFACT_BYTES", 1) + try: + async with minted_source(tmp_path / "scratch", b"12") as source: + with pytest.raises(s3_module.ArtifactLimitExceededError): + await store.put(source) + finally: + store.close() + + +def test_namespace_claim_mismatch_fails_before_provider_io() -> None: + """Reject a configured namespace mismatch without routing elsewhere.""" + bootstrap = create_artifact_store_bootstrap( + minio_settings(private_prefix="namespace/exact") + ) + namespace = bootstrap.namespace_identity + wrong = ArtifactStoreNamespaceClaim( + adapter_identity=bootstrap.identity, + namespace_identity=namespace, + namespace_fingerprint="sha256:" + "0" * 64, + ) + with pytest.raises(ArtifactConfigurationError, match="does not match"): + bootstrap.initialize_after_namespace_claim(wrong) + + +async def raw_minio_put(key: str, content: bytes) -> None: + """Publish adversarial integration bytes outside the Workstream adapter.""" + session = AioSession() + session.set_credentials(MINIO_ACCESS_KEY, MINIO_SECRET_KEY) + async with session.create_client( + "s3", + endpoint_url=MINIO_ENDPOINT, + region_name=MINIO_REGION, + config=AioConfig(s3={"addressing_style": "path"}), + ) as client: + await client.put_object(Bucket=MINIO_BUCKET, Key=key, Body=content) + + +def test_s3_constructor_and_bootstrap_reject_invalid_runtime_values() -> None: + """Fail closed on malformed direct construction outside validated settings.""" + session = AioSession() + values = { + "provider_profile": "minio", + "region": MINIO_REGION, + "endpoint_url": MINIO_ENDPOINT, + "bucket": MINIO_BUCKET, + "private_prefix": "negative/constructor", + "addressing_style": "path", + "session": session, + "buffer_bytes": 2, + "connect_timeout_seconds": 1, + "read_timeout_seconds": 1, + "write_timeout_seconds": 1, + "pool_timeout_seconds": 1, + "operation_total_timeout_seconds": 2, + "max_pool_connections": 1, + } + for override in ( + {"provider_profile": "unknown"}, + {"session": object()}, + {"buffer_bytes": 0}, + {"buffer_bytes": True}, + {"max_pool_connections": 0}, + {"max_pool_connections": True}, + {"connect_timeout_seconds": 0}, + {"read_timeout_seconds": True}, + ): + with pytest.raises(ArtifactConfigurationError): + S3CompatibleArtifactStore(**(values | override)) # type: ignore[arg-type] + with pytest.raises(ValueError): + S3CompatibleArtifactStoreBootstrap(object()) # type: ignore[arg-type] + + +def test_namespace_claim_is_single_use_and_close_disables_operations() -> None: + """One exact claim activates one adapter once and close revokes local use.""" + bootstrap = create_artifact_store_bootstrap( + minio_settings(private_prefix="negative/single-use") + ) + namespace = bootstrap.namespace_identity + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=bootstrap.identity, + namespace_identity=namespace, + ) + claim = ArtifactStoreNamespaceClaim(bootstrap.identity, namespace, fingerprint) + store = bootstrap.initialize_after_namespace_claim(claim) + with pytest.raises(ArtifactConfigurationError, match="already initialized"): + bootstrap.initialize_after_namespace_claim(claim) + with pytest.raises(ArtifactConfigurationError, match="not initialized"): + store.open("sha256/00/" + "0" * 62) + + +@pytest.mark.asyncio +async def test_invalid_provider_reference_and_range_fail_before_provider_io() -> None: + """Reject caller-selected keys and malformed ranges at the adapter boundary.""" + store = initialize_minio_store(private_prefix="negative/references") + try: + with pytest.raises(ArtifactOperationConflictError): + await store.head("customer/project/file") + with pytest.raises(ArtifactOperationConflictError): + store.open("sha256/00/" + "0" * 62, object()) # type: ignore[arg-type] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_head_maps_only_404_to_missing_and_rejects_invalid_size( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Keep denied access and malformed provider metadata distinct from absence.""" + store = initialize_minio_store(private_prefix="negative/head") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + response: object + + async def head_object(self, **_kwargs: object) -> object: + if isinstance(self.response, BaseException): + raise self.response + return self.response + + client = Client() + + @asynccontextmanager + async def fake_client() -> Any: + yield client + + monkeypatch.setattr(store, "_client", fake_client) + try: + client.response = _client_error("404", 404) + assert (await store.head(provider_ref)).exists is False + client.response = _client_error("AccessDenied", 403) + with pytest.raises(ArtifactStoreUnavailableError): + await store.head(provider_ref) + client.response = {} + with pytest.raises(ArtifactIntegrityError, match="exact size"): + await store.head(provider_ref) + finally: + store.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("chunks", "expected_error"), + [ + ([b"a", b""], "truncated"), + ([b"abc"], "exceeded"), + (["not-bytes"], "invalid bytes"), + ], +) +async def test_open_rejects_invalid_provider_streams( + monkeypatch: pytest.MonkeyPatch, + chunks: list[object], + expected_error: str, +) -> None: + """Reject truncation, overread, and non-byte provider responses.""" + store = initialize_minio_store(private_prefix="negative/open") + provider_ref = "sha256/00/" + "0" * 62 + + class Body: + def __init__(self) -> None: + self._chunks = iter(chunks) + + async def __aenter__(self) -> Body: + return self + + async def __aexit__(self, *_args: object) -> None: + return None + + async def read(self, _amount: int) -> object: + return next(self._chunks, b"") + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + return {"ContentLength": 2} + + async def get_object(self, **_kwargs: object) -> dict[str, object]: + return {"Body": Body()} + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactIntegrityError, match=expected_error): + _ = [chunk async for chunk in store.open(provider_ref)] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_open_maps_provider_404_to_missing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Map an object disappearing between HEAD and GET to stable missing.""" + store = initialize_minio_store(private_prefix="negative/disappeared") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + return {"ContentLength": 1} + + async def get_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", 404) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactObjectMissingError): + _ = [chunk async for chunk in store.open(provider_ref, ArtifactByteRange())] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_precondition_failure_requires_exact_replay_verification( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Never accept a conditional-write conflict without a complete read proof.""" + store = initialize_minio_store(private_prefix="negative/precondition") + verified: list[str] = [] + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("PreconditionFailed", 412) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def verify(provider_ref: str, _commitment: object) -> None: + verified.append(provider_ref) + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_verify_exact", verify) + try: + async with minted_source(tmp_path / "scratch", b"replay") as source: + result = await store.put(source) + assert result.replayed is True + assert verified == [result.provider_object_ref] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_nonprecondition_put_error_is_unavailable( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Keep denied writes out of replay and integrity classifications.""" + store = initialize_minio_store(private_prefix="negative/denied-put") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("AccessDenied", 403) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + async with minted_source(tmp_path / "scratch", b"denied") as source: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + finally: + store.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("committed", [False, True]) +async def test_uncertain_transport_put_requires_independent_observation( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + committed: bool, +) -> None: + """Resolve acknowledgement loss only from exact deterministic-key bytes.""" + store = initialize_minio_store(private_prefix=f"negative/uncertain/{committed}") + observed: list[str] = [] + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise OSError("connection closed") + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def matches(provider_ref: str, _commitment: object) -> bool: + observed.append(provider_ref) + return committed + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_matches_commitment", matches) + try: + async with minted_source(tmp_path / "scratch", b"uncertain") as source: + if committed: + result = await store.put(source) + assert result.replayed is True + assert observed == [result.provider_object_ref] + else: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + assert len(observed) == 1 + finally: + store.close() + + +@pytest.mark.asyncio +async def test_put_observation_requires_canonical_commitment() -> None: + """Reject caller-assembled observation requests before provider I/O.""" + store = initialize_minio_store(private_prefix="negative/observation") + try: + with pytest.raises(ArtifactOperationConflictError): + await store.observe_put_result(object()) # type: ignore[arg-type] + finally: + store.close() + + +def _client_error(code: str, status: int) -> ClientError: + """Build one sanitized provider error for adapter mapping tests.""" + return ClientError( + { + "Error": {"Code": code, "Message": "provider detail"}, + "ResponseMetadata": {"HTTPStatusCode": status}, + }, + "HeadObject", + ) diff --git a/docker-compose.yml b/docker-compose.yml index 8496f1e3..d215c295 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -24,6 +24,22 @@ services: interval: 5s timeout: 5s retries: 10 + minio: + image: quay.io/minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e + command: server /data --address :9000 + environment: + MINIO_ROOT_USER: workstream-minio + MINIO_ROOT_PASSWORD: workstream-minio-secret-key + ports: + - "9000:9000" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 5s + timeout: 5s + retries: 20 + volumes: + - workstream_minio_data:/data volumes: workstream_postgres_data: + workstream_minio_data: diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 7d9bd422..380e474c 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -434,8 +434,8 @@ Configuration includes: - dedicated private Workstream artifact bucket and private prefix; the bucket contains no other application's objects; - addressing style; -- credential mode, optional access-key ID/secret/session token, and no resolved - credential persistence; +- credential mode, local/CI MinIO static credentials held only as redacted + private settings, and no resolved credential persistence; - connect/read/write/pool timeouts; - total verification deadline and maximum buffered bytes. @@ -459,6 +459,24 @@ Chunk 02B1 pins the async S3 SDK pair to `aiobotocore==3.7.0` and `botocore==1.43.0`. An SDK upgrade is a reviewed contract change because credential-provider precedence and refresh windows are operational behavior. +### Runtime Eligibility After Chunk 02B1 + +`minio` is the only active `s3_compatible` profile in this chunk. It requires +an explicit HTTP(S) endpoint, region, dedicated bucket, private prefix, +`local_static` credentials, and a local/development/test environment. Startup +still claims the exact `minio-v1` namespace in PostgreSQL before any object +operation. Local and CI run the real digest-pinned MinIO server and provision +only the dedicated test bucket; tests do not emulate S3 calls. + +`aws_s3` accepts only a native endpoint-less AWS configuration, explicit +region, and exactly one allowlisted workload-identity method. The composition +root validates that environment but then raises +`artifact_provider_live_proof_required` before constructing the adapter +factory, building or loading the credential resolver, claiming a namespace, or +performing provider I/O. Chunk 07 alone may make native AWS runtime-eligible by +adding the immutable live activation proof defined below. MinIO conformance is +not AWS activation evidence. + Cloudflare R2 has no v0.1 runtime mode, credential issuer, sidecar, secret contract, configuration profile, or deployment proof. Any future provider initiative must perform current discovery and pass the same ArtifactStore v2 diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index 0fcfcb33..2946b4cb 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -4144,55 +4144,30 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: assert "| `WS-AUTH-001-09B` | Merged |" in auth_status assert "| `WS-AUTH-001-09C` | In review |" in auth_status assert "Merged through PR #129 as `9a04434`" in artifact_map - artifact_phases = ( - "Deterministic proof passed; exact-SHA internal review in progress", - "Internal review and deterministic evidence passed; external checks pending", - ) - selected_phases = [phase for phase in artifact_phases if phase in artifact_map] - assert len(selected_phases) == 1 - selected_phase = selected_phases[0] - assert f"Status: {selected_phase}" in artifact_contract + assert "Merged through PR #141 as `a10d901`" in artifact_map + assert "Active after 02A3 merged through PR #141" in artifact_map + assert "Status: Active after explicit human start" in artifact_contract assert ( "AUTH's owner reconciliation merged through PR #140 as\n" "`d541521`" in artifact_status ) - assert ( - "`WS-ART-001-02A3` implementation and merged-main deterministic repair are\n" - "complete" in artifact_status + assert "`WS-ART-001-02B1` is active" in artifact_status + assert "The current gate is deterministic 02B1 proof followed by all nine" in ( + artifact_status.replace("\n", " ") ) - artifact_02a3_merged = ( - "`WS-ART-001-02A3` | ArtifactStore v2 Local Clean Cut | L1 | " - "Merged through PR #141 as `a10d901`" in work_queue + assert "No later artifact chunk starts automatically" in artifact_status.replace( + "\n", " " ) - if artifact_02a3_merged: - normalized_work_queue = " ".join(work_queue.split()) - assert "PR #141 merged `WS-ART-001-02A3` into `main` as `a10d901`" in loop_state - assert ( - "`WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | " - "Inactive until 02A3 merge and explicit user start" in work_queue - ) - assert "ART-02B1 remains inactive" in loop_state - assert "Do not start AUTH-09C or POL-002-04 automatically." not in ( - normalized_work_queue - ) - assert ( - "Its merged-main deterministic proof and exact-SHA internal review are " - "complete; external checks remain pending and `02B1` must not start " - "automatically." not in normalized_work_queue - ) - elif selected_phase == artifact_phases[0]: - assert selected_phase in work_queue - assert "The current gate is all nine\nexact-SHA internal tracks" in artifact_status - assert "Current gate: complete all nine exact-SHA internal reviewer tracks" in loop_state - else: - assert selected_phase in work_queue - assert "The current gate is GitHub Actions, CodeRabbit, and explicit human review" in ( - artifact_status.replace("\n", " ") - ) - assert "Current gate: publish PR #141 for GitHub Actions, CodeRabbit, and explicit" in ( - loop_state.replace("\n", " ") - ) - assert "No\nlater artifact chunk starts automatically" in artifact_status + assert "| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 |" in ( + work_queue + ) + assert "| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Active" in ( + work_queue + ) + assert "Current ART gate: integrate trusted `main`, complete deterministic 02B1" in ( + loop_state + ) + assert "No later ART chunk starts automatically" in loop_state.replace("\n", " ") def test_stale_authorization_discovery_includes_new_untracked_docs() -> None: From 216889697d341d569ce03be485c404efdc62225f Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 07:41:15 +0100 Subject: [PATCH 02/39] Harden S3 artifact storage review findings --- .../CHUNK_MAP.md | 2 +- README.md | 13 +++- backend/app/adapters/artifacts/__init__.py | 5 -- .../app/adapters/artifacts/s3_compatible.py | 29 ++++++--- .../tests/test_aws_credential_isolation.py | 20 ++++++- backend/tests/test_config.py | 9 +++ backend/tests/test_s3_artifact_store.py | 59 +++++++++++++++++++ docs/spec_artifact_storage_service.md | 32 ++++++++++ 8 files changed, 152 insertions(+), 17 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index 46488869..087bd357 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -9,7 +9,7 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-OBJECT-STORAGE-AMENDMENT` | Make AWS S3 the v0.1 production provider with MinIO local/CI protocol proof; keep optional providers outside v0.1. | L1 | Merged through PR #120 as `4408256` | | `WS-ART-001-02A1` | Install only ADR 0014's small typed external-service adapter/factory foundation without migrating a capability. | L1 | Merged through PR #127 as `f64a8e5` | | `WS-ART-001-02A2` | Add bounded committed-source preparation and inactive scratch-cleanup mechanics without changing the active v1 port. | L1 | Merged through PR #129 as `9a04434` on 2026-07-16 | -| `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Internal review and deterministic evidence passed; external checks pending | +| `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Merged through PR #141 as `a10d901` on 2026-07-18 | | `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Active after 02A3 merged through PR #141 and explicit user start | | `WS-ART-001-02C1` | Add the generic durable-byte admission ledger and durable put-attempt state foundation without provider execution. | L1 | Proposed after 02B1 | | `WS-ART-001-02C2` | Add put resolution, verification publication, complete-object observation, immutable receipts, and PostgreSQL execution fencing without recovery attempts or routes. | L1 | Proposed after 02C1 | diff --git a/README.md b/README.md index c7f14122..a89e3a2e 100644 --- a/README.md +++ b/README.md @@ -186,13 +186,20 @@ the repository is changed; it does not define runtime task or review records. ## Local Backend Database Workstream uses Postgres locally and in CI. It uses Celery with Redis for -durable local project setup jobs and automatic pre-review checker gates. Start -local services with: +durable local project setup jobs and automatic pre-review checker gates. MinIO +provides the S3-compatible artifact protocol in local development and CI. Start +the local services with: ```bash -docker compose up -d postgres redis +docker compose up -d postgres redis minio ``` +MinIO uses the compose-only static credentials and the private +`workstream-artifacts` bucket. Native AWS S3 accepts workload-identity +configuration but remains runtime-ineligible until live deployment proof is +approved; startup fails with `artifact_provider_live_proof_required` before +credential probing or provider I/O. + The default local development URL is: ```text diff --git a/backend/app/adapters/artifacts/__init__.py b/backend/app/adapters/artifacts/__init__.py index a15229c3..d5b461f7 100644 --- a/backend/app/adapters/artifacts/__init__.py +++ b/backend/app/adapters/artifacts/__init__.py @@ -32,11 +32,6 @@ def create_artifact_store_bootstrap(settings: Settings) -> ArtifactStoreBootstra settings.artifact_store_backend == "s3_compatible" and settings.artifact_s3_provider_profile == "aws_s3" ): - from app.adapters.artifacts.s3_compatible import ( - validate_aws_workload_identity_environment, - ) - - validate_aws_workload_identity_environment(settings) raise ArtifactProviderLiveProofRequiredError( "AWS artifact provider requires live deployment proof" ) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index 65447440..b205f0d9 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -359,6 +359,8 @@ async def iterate() -> AsyncIterator[bytes]: if observed != expected: raise ArtifactIntegrityError("S3 artifact object was truncated") except ClientError as error: + if _provider_http_status(error) == 403: + raise ArtifactStoreUnavailableError("S3 artifact read failed") from None code = _provider_error_code(error) if code in _MISSING_ERROR_CODES: raise ArtifactObjectMissingError("artifact object is missing") from None @@ -390,6 +392,8 @@ async def head(self, provider_object_ref: str) -> ArtifactObjectHead: byte_count=byte_count, ) except ClientError as error: + if _provider_http_status(error) == 403: + raise ArtifactStoreUnavailableError("S3 artifact head failed") from None code = _provider_error_code(error) if code in _MISSING_ERROR_CODES: return ArtifactObjectHead(provider_object_ref, exists=False) @@ -649,11 +653,11 @@ def validate_aws_workload_identity_environment( if not token_path.is_absolute() or not token_path.is_file(): raise ArtifactConfigurationError("AWS web identity token path is invalid") elif selected == "container-role": - locations = _CONTAINER_ENVIRONMENT.intersection(environment) & { - "AWS_CONTAINER_CREDENTIALS_FULL_URI", - "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", - } - if len(locations) != 1: + if "AWS_CONTAINER_CREDENTIALS_FULL_URI" in environment: + raise ArtifactConfigurationError( + "AWS container full credential URI is forbidden" + ) + if "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" not in environment: raise ArtifactConfigurationError("AWS container identity location is invalid") auth = _CONTAINER_ENVIRONMENT.intersection(environment) & { "AWS_CONTAINER_AUTHORIZATION_TOKEN", @@ -661,8 +665,11 @@ def validate_aws_workload_identity_environment( } if len(auth) > 1: raise ArtifactConfigurationError("AWS container identity token is ambiguous") - elif environment.get("AWS_EC2_METADATA_DISABLED", "false").lower() == "true": - raise ArtifactConfigurationError("AWS instance identity metadata is disabled") + else: + if "AWS_EC2_METADATA_SERVICE_ENDPOINT" in environment: + raise ArtifactConfigurationError("custom AWS metadata endpoint is forbidden") + if environment.get("AWS_EC2_METADATA_DISABLED", "false").lower() == "true": + raise ArtifactConfigurationError("AWS instance identity metadata is disabled") def create_isolated_aws_workload_identity_session( @@ -756,6 +763,14 @@ def _provider_error_code(error: ClientError) -> str: return str(status) if type(status) is int else "unknown" +def _provider_http_status(error: ClientError) -> int | None: + """Return the provider HTTP status independently from its error code.""" + response = error.response if isinstance(error.response, dict) else {} + metadata = response.get("ResponseMetadata") if isinstance(response, dict) else None + status = metadata.get("HTTPStatusCode") if isinstance(metadata, dict) else None + return status if type(status) is int else None + + def _required(value: str | None) -> str: """Return one settings value already guaranteed by validation.""" if not isinstance(value, str) or not value: diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index ec6f0f3f..ed848b08 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -248,6 +248,14 @@ def test_isolated_session_contains_exactly_one_selected_provider( lambda tmp_path: {"HOME": str(tmp_path)}, "container identity location is invalid", ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_FULL_URI": "https://credentials.example.test", + }, + "container full credential URI is forbidden", + ), ( "container-role", lambda tmp_path: { @@ -255,7 +263,7 @@ def test_isolated_session_contains_exactly_one_selected_provider( "AWS_CONTAINER_CREDENTIALS_FULL_URI": "http://127.0.0.1:9/full", "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": "/relative", }, - "container identity location is invalid", + "container full credential URI is forbidden", ), ( "container-role", @@ -272,6 +280,16 @@ def test_isolated_session_contains_exactly_one_selected_provider( lambda tmp_path: {"HOME": str(tmp_path), "AWS_EC2_METADATA_DISABLED": "true"}, "instance identity metadata is disabled", ), + ( + "iam-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_SERVICE_ENDPOINT": ( + "https://metadata.example.test/latest" + ), + }, + "custom AWS metadata endpoint is forbidden", + ), ], ) def test_selected_workload_identity_source_must_be_complete( diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 4bf18f68..ef81e4bd 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -779,10 +779,19 @@ def session_must_not_construct(*_args: object, **_kwargs: object) -> object: events.append("resolver") raise AssertionError("credential resolver constructed before live proof failure") + def credentials_must_not_be_probed(*_args: object, **_kwargs: object) -> None: + events.append("credential-probe") + raise AssertionError("credential sources probed before live proof failure") + monkeypatch.setenv("HOME", str(tmp_path)) monkeypatch.setenv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "/v2/credentials/runtime") monkeypatch.setattr(artifacts_module, "ExternalServiceAdapterFactory", FactoryMustNotConstruct) monkeypatch.setattr(s3_compatible, "AioSession", session_must_not_construct) + monkeypatch.setattr( + s3_compatible, + "validate_aws_workload_identity_environment", + credentials_must_not_be_probed, + ) with pytest.raises(ArtifactProviderLiveProofRequiredError) as caught: create_artifact_store_bootstrap(settings) diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 2af7b24e..c5b858df 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -170,6 +170,35 @@ async def test_adversarial_first_writer_cannot_be_accepted_as_replay( store.close() +@pytest.mark.asyncio +async def test_cross_project_prefix_cannot_occupy_another_content_key( + tmp_path: Path, +) -> None: + """Keep identical digest references isolated by claimed project namespace.""" + nonce = hashlib.sha256(os.urandom(16)).hexdigest()[:20] + attacker_prefix = f"projects/attacker-{nonce}" + target_prefix = f"projects/target-{nonce}" + content = b"target project exact bytes" + digest = hashlib.sha256(content).hexdigest() + provider_ref = f"sha256/{digest[:2]}/{digest[2:]}" + await raw_minio_put(f"{attacker_prefix}/{provider_ref}", b"attacker bytes") + + attacker_store = initialize_minio_store(private_prefix=attacker_prefix) + target_store = initialize_minio_store(private_prefix=target_prefix) + try: + async with minted_source(tmp_path / "attacker-scratch", content) as source: + with pytest.raises(ArtifactIntegrityError): + await attacker_store.put(source) + async with minted_source(tmp_path / "target-scratch", content) as source: + result = await target_store.put(source) + assert result.provider_object_ref == provider_ref + assert result.replayed is False + assert b"".join([chunk async for chunk in target_store.open(provider_ref)]) == content + finally: + attacker_store.close() + target_store.close() + + @pytest.mark.asyncio async def test_unsealed_client_selected_commitment_fails_before_provider_io( monkeypatch: pytest.MonkeyPatch, @@ -335,6 +364,9 @@ async def fake_client() -> Any: client.response = _client_error("404", 404) assert (await store.head(provider_ref)).exists is False client.response = _client_error("AccessDenied", 403) + with pytest.raises(ArtifactStoreUnavailableError): + await store.head(provider_ref) + client.response = _client_error("NoSuchKey", 403) with pytest.raises(ArtifactStoreUnavailableError): await store.head(provider_ref) client.response = {} @@ -421,6 +453,33 @@ async def fake_client() -> Any: store.close() +@pytest.mark.asyncio +async def test_open_403_precedes_missing_like_provider_code( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Never turn denied reads into authoritative object absence.""" + store = initialize_minio_store(private_prefix="negative/denied-open") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + return {"ContentLength": 1} + + async def get_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", 403) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactStoreUnavailableError): + _ = [chunk async for chunk in store.open(provider_ref)] + finally: + store.close() + + @pytest.mark.asyncio async def test_precondition_failure_requires_exact_replay_verification( monkeypatch: pytest.MonkeyPatch, diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 380e474c..f9117f81 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -779,6 +779,38 @@ The preparation settings use the standard `WORKSTREAM_` environment prefix: | `WORKSTREAM_ARTIFACT_STREAM_BUFFER_BYTES` | `1048576` | Bounded streaming buffer, limited to at most 1 MiB. | | `WORKSTREAM_ARTIFACT_OPERATION_LOCK_TIMEOUT_SECONDS` | `1800` | Maximum wait for a private cross-process artifact-store operation lock before failing closed. | +The S3-compatible provider settings use the same prefix: + +| Environment variable | Default | Contract | +|---|---:|---| +| `WORKSTREAM_ARTIFACT_STORE_BACKEND` | `disabled` | Set to `s3_compatible` to select the S3 protocol adapter. | +| `WORKSTREAM_ARTIFACT_S3_PROVIDER_PROFILE` | unset | Closed value: `minio` for local/CI or `aws_s3` for the inactive native AWS profile. | +| `WORKSTREAM_ARTIFACT_S3_REGION` | unset | Required canonical S3 region. | +| `WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL` | unset | Required HTTP(S) endpoint for MinIO; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_BUCKET` | unset | Required private bucket name. | +| `WORKSTREAM_ARTIFACT_S3_PRIVATE_PREFIX` | `workstream/artifacts` | Canonical private key prefix; object references remain server-derived SHA-256 paths. | +| `WORKSTREAM_ARTIFACT_S3_ADDRESSING_STYLE` | `virtual` | Closed value: `path` or `virtual`; local compose uses `path`. | +| `WORKSTREAM_ARTIFACT_S3_CREDENTIAL_MODE` | unset | MinIO requires `local_static`; native AWS requires `aws_workload_identity`. | +| `WORKSTREAM_ARTIFACT_S3_AWS_WORKLOAD_IDENTITY_METHOD` | unset | Native AWS selection: `assume-role-with-web-identity`, `container-role`, or `iam-role`; never used by MinIO. | +| `WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID` | unset | MinIO local/CI secret only; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY` | unset | MinIO local/CI secret only; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_SESSION_TOKEN` | unset | Optional MinIO local/CI secret only; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_CONNECT_TIMEOUT_SECONDS` | `5` | Provider connection timeout, at most 60 seconds. | +| `WORKSTREAM_ARTIFACT_S3_READ_TIMEOUT_SECONDS` | `60` | Provider read timeout, at most 1800 seconds. | +| `WORKSTREAM_ARTIFACT_S3_WRITE_TIMEOUT_SECONDS` | `1800` | Provider write timeout, at most 3600 seconds. | +| `WORKSTREAM_ARTIFACT_S3_POOL_TIMEOUT_SECONDS` | `5` | Client-pool acquisition timeout, at most 60 seconds. | +| `WORKSTREAM_ARTIFACT_S3_OPERATION_TOTAL_TIMEOUT_SECONDS` | `1800` | Total provider-operation timeout, at most 3600 seconds. | +| `WORKSTREAM_ARTIFACT_S3_MAX_POOL_CONNECTIONS` | `16` | Bounded client connection pool, from 1 through 256. | +| `WORKSTREAM_ARTIFACT_MAXIMUM_BYTES` | `536870912` | Hard object maximum; configuration cannot exceed 512 MiB. | + +MinIO is the only runtime-eligible S3 profile in this chunk and is restricted +to local, development, and test environments. Native AWS configuration rejects +static credentials, configured endpoints, unselected or ambient credential +sources, arbitrary container credential URLs, and custom instance-metadata +endpoints. Even valid native AWS settings fail with +`artifact_provider_live_proof_required` before credential-source probing, +factory construction, namespace claim, resolver loading, or provider I/O. + Every process sharing a scratch root must use the identical complete setting set. The root marker binds a canonical fingerprint of those limits and startup fails closed on mismatch; changing limits therefore requires an empty, From bb8cfa8f95ab6fa14a2795a475e150b2fd351bd3 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 08:10:26 +0100 Subject: [PATCH 03/39] Integrate artifact storage with current main --- ...WS-ART-001-02B1-s3-compatible-minio-aws.md | 3 ++ .../app/adapters/artifacts/s3_compatible.py | 7 ++-- backend/app/core/config.py | 23 ++++++++--- .../tests/test_aws_credential_isolation.py | 26 +++++++++++- backend/tests/test_config.py | 25 +++++++++++- docs/spec_artifact_storage_service.md | 3 +- scripts/test_agent_gates.py | 40 +++++++++++++++++-- 7 files changed, 111 insertions(+), 16 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index d2534884..242e8be3 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -56,6 +56,9 @@ credential-delivery contract. - explicit credentials, environment access keys, shared credential/config files, credential processes, login/SSO, legacy EC2/Boto sources, and every unselected workload provider fail startup before credential loading; +- container role accepts only the relative metadata path, IAM role rejects a + custom metadata endpoint and requires IMDSv2, and the isolated IAM fetcher + disables IMDSv1; - isolation tests poison nonselected file, process, cache, metadata, and network sources and prove none is accessed; MinIO static credentials are local/CI only and never survive error/log object graphs; diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index b205f0d9..fb1420cd 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -670,6 +670,8 @@ def validate_aws_workload_identity_environment( raise ArtifactConfigurationError("custom AWS metadata endpoint is forbidden") if environment.get("AWS_EC2_METADATA_DISABLED", "false").lower() == "true": raise ArtifactConfigurationError("AWS instance identity metadata is disabled") + if environment.get("AWS_EC2_METADATA_V1_DISABLED", "false").lower() != "true": + raise ArtifactConfigurationError("AWS instance identity requires IMDSv2") def create_isolated_aws_workload_identity_session( @@ -713,10 +715,7 @@ def create_isolated_aws_workload_identity_session( "ec2_metadata_service_endpoint_mode": environment.get( "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE" ), - "ec2_metadata_v1_disabled": environment.get( - "AWS_EC2_METADATA_V1_DISABLED", "false" - ).lower() - == "true", + "ec2_metadata_v1_disabled": True, }, ) ) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index ee90ce07..8d1cb97f 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -403,7 +403,9 @@ def _validate_minio_storage(self) -> None: raise ValueError("MinIO artifact storage requires local static credentials") if self.artifact_s3_aws_workload_identity_method is not None: raise ValueError("MinIO artifact storage cannot select AWS workload identity") - _validate_minio_endpoint(self.artifact_s3_endpoint_url) + self.artifact_s3_endpoint_url = _canonical_minio_endpoint( + self.artifact_s3_endpoint_url + ) def _validate_native_aws_storage(self) -> None: """Validate native AWS configuration without activating its runtime.""" @@ -526,22 +528,33 @@ def _is_canonical_s3_prefix(value: str) -> bool: return all(_S3_PREFIX_SEGMENT.fullmatch(segment) is not None for segment in segments) -def _validate_minio_endpoint(value: str | None) -> None: - """Require one canonical noncredentialed HTTP(S) MinIO origin.""" +def _canonical_minio_endpoint(value: str | None) -> str: + """Return one normalized noncredentialed HTTP(S) MinIO origin.""" if not isinstance(value, str) or not value or len(value) > 2048: raise ValueError("MinIO artifact storage requires an endpoint") parsed = urlsplit(value) + try: + port = parsed.port + except ValueError as error: + raise ValueError("MinIO artifact storage endpoint is invalid") from error if ( - parsed.scheme not in {"http", "https"} + parsed.scheme.lower() not in {"http", "https"} or not parsed.hostname or parsed.username is not None or parsed.password is not None or parsed.query or parsed.fragment or parsed.path not in {"", "/"} - or value.endswith("/") + or parsed.hostname.endswith(".") ): raise ValueError("MinIO artifact storage endpoint is invalid") + scheme = parsed.scheme.lower() + hostname = parsed.hostname.lower() + if ":" in hostname: + hostname = f"[{hostname}]" + if port is not None and port != {"http": 80, "https": 443}[scheme]: + hostname = f"{hostname}:{port}" + return f"{scheme}://{hostname}" def decode_api_rate_limit_key_secret(value: SecretStr) -> bytes: diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index ed848b08..1acdab84 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -54,7 +54,10 @@ def _web_identity_environment(tmp_path: Path, **overrides: str) -> dict[str, str def _iam_environment(tmp_path: Path, **overrides: str) -> dict[str, str]: - environment = {"HOME": str(tmp_path)} + environment = { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_V1_DISABLED": "true", + } environment.update(overrides) return environment @@ -224,6 +227,8 @@ def test_isolated_session_contains_exactly_one_selected_provider( } } } + if method == "iam-role": + assert provider._role_fetcher._imds_v1_disabled is True @pytest.mark.parametrize( @@ -277,9 +282,26 @@ def test_isolated_session_contains_exactly_one_selected_provider( ), ( "iam-role", - lambda tmp_path: {"HOME": str(tmp_path), "AWS_EC2_METADATA_DISABLED": "true"}, + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_DISABLED": "true", + "AWS_EC2_METADATA_V1_DISABLED": "true", + }, "instance identity metadata is disabled", ), + ( + "iam-role", + lambda tmp_path: {"HOME": str(tmp_path)}, + "instance identity requires IMDSv2", + ), + ( + "iam-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_V1_DISABLED": "false", + }, + "instance identity requires IMDSv2", + ), ( "iam-role", lambda tmp_path: { diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index ef81e4bd..0db301f3 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -633,7 +633,6 @@ def test_minio_settings_and_factory_construct_closed_namespace(tmp_path: Path) - ({"artifact_s3_endpoint_url": None}, "requires an endpoint"), ({"artifact_s3_endpoint_url": "https://user:pass@minio.test"}, "endpoint is invalid"), ({"artifact_s3_endpoint_url": "http://minio.test/path"}, "endpoint is invalid"), - ({"artifact_s3_endpoint_url": "http://minio.test/"}, "endpoint is invalid"), ], ) def test_minio_settings_fail_closed( @@ -736,6 +735,30 @@ def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( _assert_secret_not_retained(dotenv_error.value, dotenv_secret) +def test_minio_endpoint_is_normalized_before_namespace_identity(tmp_path: Path) -> None: + """Equivalent MinIO origins must produce one endpoint and namespace identity.""" + common = { + "environment": "test", + "artifact_store_backend": "s3_compatible", + "artifact_scratch_root": tmp_path / "scratch", + "artifact_s3_provider_profile": "minio", + "artifact_s3_region": "us-east-1", + "artifact_s3_bucket": "workstream-artifacts-test", + "artifact_s3_credential_mode": "local_static", + "artifact_s3_access_key_id": "local-access", + "artifact_s3_secret_access_key": "local-secret", + } + canonical = Settings(**common, artifact_s3_endpoint_url="http://localhost") + equivalent = Settings(**common, artifact_s3_endpoint_url="HTTP://LOCALHOST:80/") + + assert canonical.artifact_s3_endpoint_url == "http://localhost" + assert equivalent.artifact_s3_endpoint_url == "http://localhost" + assert ( + create_artifact_store_bootstrap(canonical).namespace_identity + == create_artifact_store_bootstrap(equivalent).namespace_identity + ) + + @pytest.mark.parametrize( ("overrides", "message"), [ diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index f9117f81..1b497e7c 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -807,7 +807,8 @@ MinIO is the only runtime-eligible S3 profile in this chunk and is restricted to local, development, and test environments. Native AWS configuration rejects static credentials, configured endpoints, unselected or ambient credential sources, arbitrary container credential URLs, and custom instance-metadata -endpoints. Even valid native AWS settings fail with +endpoints. IAM-role validation requires IMDSv2 and the isolated fetcher disables +IMDSv1. Even valid native AWS settings fail with `artifact_provider_live_proof_required` before credential-source probing, factory construction, namespace claim, resolver loading, or provider I/O. diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index 2946b4cb..a438370a 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -62,7 +62,10 @@ "coverage report --include='app/workers/*' --precision=2 --fail-under=90", "coverage report --include='app/main.py' --precision=2 --fail-under=90", ), - "02B1": (), + "02B1": ( + "coverage report --include='app/adapters/artifacts/s3_compatible.py' " + "--precision=2 --fail-under=90", + ), "02C1": ( "coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90", ), @@ -112,6 +115,29 @@ "python scripts/api_contract_e2e.py", ) ) +MINIO_IMAGE = ( + "quay.io/minio/minio:latest@" + "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e" +) +BACKEND_MINIO_START_COMMAND = "\n".join( + ( + f"image='{MINIO_IMAGE}'", + "docker run --detach --rm --name workstream-minio \\", + " --publish 9000:9000 \\", + " --env MINIO_ROOT_USER=workstream-minio \\", + " --env MINIO_ROOT_PASSWORD=workstream-minio-secret-key \\", + ' "$image" server /data --address :9000', + "for attempt in $(seq 1 60); do", + " if curl --fail --silent " + "http://127.0.0.1:9000/minio/health/live >/dev/null; then", + " exit 0", + " fi", + " sleep 1", + "done", + "docker logs workstream-minio", + "exit 1", + ) +) AUTH_09B_COVERAGE_COMMANDS = ( "coverage report --include='app/modules/actors/*' --precision=2 --fail-under=90", "coverage report --include='app/modules/authorization/*' " @@ -4130,7 +4156,7 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: ).read_text(encoding="utf-8") artifact_contract = Path( ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/" - "WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md" + "WS-ART-001-02B1-s3-compatible-minio-aws.md" ).read_text(encoding="utf-8") work_queue = Path(".agent-loop/WORK_QUEUE.md").read_text(encoding="utf-8") loop_state = Path(".agent-loop/LOOP_STATE.md").read_text(encoding="utf-8") @@ -4445,6 +4471,12 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: test_job = parsed_workflow["jobs"]["test"] assert set(test_job) == {"runs-on", "timeout-minutes", "services", "steps"} steps = test_job["steps"] + minio_steps = [ + step for step in steps if step.get("name") == "Start real MinIO artifact provider" + ] + assert len(minio_steps) == 1 + assert set(minio_steps[0]) == {"name", "run"} + assert str(minio_steps[0]["run"]).strip() == BACKEND_MINIO_START_COMMAND full_suite_steps = [ step for step in steps if step.get("name") == "Backend full-suite coverage" ] @@ -4454,13 +4486,15 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: assert full_suite_steps[0].get("env") == { "WORKSTREAM_TEST_ADMIN_DATABASE_URL": ( "postgresql+asyncpg://workstream:workstream@localhost:5433/postgres" - ) + ), + "WORKSTREAM_TEST_MINIO_ENDPOINT": "http://127.0.0.1:9000", } for forbidden_key in ("if", "continue-on-error", "shell"): assert forbidden_key not in full_suite_steps[0] assert full_suite_run.strip() == BACKEND_FULL_SUITE_COVERAGE_COMMAND assert "/tmp/workstream-database.json" not in workflow full_suite_index = steps.index(full_suite_steps[0]) + assert steps.index(minio_steps[0]) < full_suite_index isolated_steps = [ step for step in steps if step.get("name") == "Isolated database runner test" ] From 29162a5f44bcd00b9e0714348c620ba200ae446e Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 08:44:34 +0100 Subject: [PATCH 04/39] Harden S3 artifact source and credential boundaries --- .agent-loop/LOOP_STATE.md | 20 ++---- .agent-loop/WORK_QUEUE.md | 9 ++- ...WS-ART-001-02B1-s3-compatible-minio-aws.md | 3 +- backend/app/adapters/artifacts/local.py | 36 ++++------ backend/app/adapters/artifacts/references.py | 29 ++++++++ .../app/adapters/artifacts/s3_compatible.py | 66 +++++++++-------- .../tests/test_aws_credential_isolation.py | 72 ++++++++++++++++++- backend/tests/test_local_artifact_store.py | 8 ++- backend/tests/test_s3_artifact_store.py | 32 +++++++++ docs/spec_artifact_storage_service.md | 14 ++-- scripts/test_agent_gates.py | 5 +- 11 files changed, 205 insertions(+), 89 deletions(-) create mode 100644 backend/app/adapters/artifacts/references.py diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index 2bec2b0a..396b2aed 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -22,23 +22,14 @@ `codex/ws-art-001-02b1-s3-compatible-minio-aws`. - The ART worktree consumes merged AUTH, REV, and CON contracts without editing or activating their independently owned runtime behavior. -- Active implementation chunk: `WS-AUTH-001-09C` on - `codex/ws-auth-001-09c-actor-identity-admin-reads`. -- AUTH-09C is bounded to two exact administrative read routes and activates - only `actor.profile.read` and `actor.identity_link.read`. AUTH-09D remains - inactive. -- Start basis: trusted `main` at `a10d901` after PR #141. +- AUTH-09C merged through PR #146 as `0ffdabf`; AUTH-09D remains inactive + until an explicit human start. +- ART integration basis: trusted `main` at `0ffdabf` after PR #146. - PR #119 merged `WS-AUTH-001-05B` as `ad71c7e`. - PR #120 merged `WS-ART-001-OBJECT-STORAGE-AMENDMENT` as `4408256`. - PR #122 merged the first automated post-merge memory implementation as `fc89fb6`; its schema-v1 cross-initiative next pointer is superseded by the schema-v2 initiative-local clean cut. -- Current gate: PR #146's original Backend, Agent Gates, and CodeRabbit checks - passed. All five valid CodeRabbit proof findings are repaired at `c64bcc7`, - trusted `main` through PR #141 is integrated, and focused unit, real - PostgreSQL, and Agent Gates proof passes. All required exact integrated-head - internal repair tracks pass at `c47d627`; push and replacement external - checks remain. No service caller becomes executable before AUTH-09E. - Current ART gate: integrate trusted `main`, complete deterministic 02B1 proof, and pass all nine exact-SHA internal reviewer tracks before opening the ART PR. No later ART chunk starts automatically. @@ -58,9 +49,8 @@ - Parallel artifact checkpoint: ART-02A1, ART-02A2, and ART-02A3 merged through PRs #127, #129, and #141. ART-02B1 is active and adds real MinIO protocol proof plus a fail-closed, runtime-ineligible native AWS profile. -- Authorization checkpoint: AUTH-07B through AUTH-09B merged through PRs #130, - #131, #132, and #143. Signed memory stopped after 09B, and the user explicitly - started AUTH-09C. +- Authorization checkpoint: AUTH-07B through AUTH-09C merged through PRs #130, + #131, #132, #143, and #146. AUTH-09D remains inactive. - Parallel coverage work: `WS-QUAL-001-01B2` remains paused. Its last official whole-app result is `6466/8159` statements (`79.249908%`); no replacement evidence exists. diff --git a/.agent-loop/WORK_QUEUE.md b/.agent-loop/WORK_QUEUE.md index c012c428..2fdfed99 100644 --- a/.agent-loop/WORK_QUEUE.md +++ b/.agent-loop/WORK_QUEUE.md @@ -4,7 +4,6 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| -| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | PR #146 open; all valid CodeRabbit findings repaired; integrated exact-head review passed; replacement external checks pending | | `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Active after explicit user start; implementation repair and exact-head internal review in progress | Live post-merge state remains read from signed `automation/loop-memory` @@ -15,7 +14,7 @@ output. This authored queue records the separately approved parallel chunks. | Chunk | Title | Risk | Status | |---|---|---:|---| | `WS-QUAL-001-01B2` | Baseline Evidence And CI Ratchet | L1 | Paused for AUTH priority; no valid replacement baseline yet | -| `WS-AUTH-001-09D` | Actor And Identity-Link Lifecycle Mutations | L1 | Inactive until 09C merge/memory and explicit user start | +| `WS-AUTH-001-09D` | Actor And Identity-Link Lifecycle Mutations | L1 | Inactive until explicit user start | | `WS-QUAL-001-02` | Project Service Coverage | L1 | Inactive until 01B2 merge/memory plus explicit user start | | `WS-POL-002-04` | Locked Runtime Execution And Routing Hardening | L1 | Inactive pending relevant authorization proof and a separate explicit user start | | `WS-ART-001-02C1` | Admission And Put-Attempt Foundation | L1 | Inactive until 02B1 merge and explicit user start | @@ -27,6 +26,7 @@ output. This authored queue records the separately approved parallel chunks. | Chunk | Title | Risk | Status | |---|---|---:|---| +| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | Merged through PR #146 as `0ffdabf` on 2026-07-18 | | `WS-ENG-001-01` | Codex-native zero-trust loop bootstrap | L1 | Merged through PR #23 on 2026-06-20 | | `EXAMPLE-TERMINAL-BENCHMARK` | Terminal Benchmark example drill | L3 | Merged through PR #25 on 2026-06-21 | | `WS-POL-001-PLAN` | Submission Artifact Policy Foundation planning | L1 | Merged through PR #26 on 2026-06-27 | @@ -88,9 +88,8 @@ AUTH-06 merged through PR #124 as `f599551`. AUTH-07A, AUTH-07B, and AUTH-08 merged through PRs #126, #130, and #131. WS-XINT planning merged through PR #139, and its AUTH owner reconciliation merged through PR #140 as `d541521`. AUTH-09A merged through PR #132 as `299363a`, and signed schema-v2 memory -stopped. AUTH-09B merged through PR #143 as `053242b`; the user then explicitly -started AUTH-09C, whose PR #146 is in external repair. Do not start AUTH-09D or -POL-002-04 automatically. +stopped. AUTH-09B merged through PR #143 as `053242b`, and AUTH-09C merged +through PR #146 as `0ffdabf`. Do not start AUTH-09D or POL-002-04 automatically. Coverage R10 merged through PR #108. Do not start 01B2, chunk 02, or another coverage implementation chunk from this worktree. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index 242e8be3..002177b9 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -13,7 +13,8 @@ credential-delivery contract. ## Allowed Files -- `backend/app/adapters/artifacts/s3_compatible.py` and adapter registration; +- `backend/app/adapters/artifacts/s3_compatible.py`, the shared canonical + provider-reference helper, and adapter registration; - `backend/app/interfaces/artifacts.py` only to register the closed `minio-v1` and `aws-s3-v1` namespace profiles, their exact descriptor keys, and the stable AWS live-proof-required error; diff --git a/backend/app/adapters/artifacts/local.py b/backend/app/adapters/artifacts/local.py index 638780ae..e4889147 100644 --- a/backend/app/adapters/artifacts/local.py +++ b/backend/app/adapters/artifacts/local.py @@ -20,6 +20,10 @@ await_cancellation_resistant, run_blocking_cancellation_resistant, ) +from app.adapters.artifacts.references import ( + artifact_provider_object_ref, + parse_artifact_provider_object_ref, +) from app.core.file_locks import acquire_exclusive_file_lock from app.interfaces.artifacts import ( ARTIFACT_STORE_CAPABILITY_KEY, @@ -53,7 +57,6 @@ _LAYOUT_ENTRY_NAMES = frozenset( {_LAYOUT_MARKER, _LAYOUT_MARKER_TEMPORARY, "locks", "objects", "tmp"} ) -_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") _DIGEST_PREFIX_NAME = re.compile(r"^[0-9a-f]{2}$") _DIGEST_OBJECT_NAME = re.compile(r"^[0-9a-f]{62}$") _TEMPORARY_NAME = re.compile(r"^\.put\.[0-9a-f]{32}\.tmp$") @@ -218,7 +221,7 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: if commitment.byte_count > HARD_MAXIMUM_ARTIFACT_BYTES: raise ArtifactLimitExceededError("artifact source exceeds provider limit") - provider_object_ref = self._provider_object_ref(commitment) + provider_object_ref = artifact_provider_object_ref(commitment) lock: tuple[Any, int] | None = None descriptor: int | None = None temporary_name: str | None = None @@ -304,7 +307,7 @@ async def observe_put_result( self._require_initialized() if type(commitment) is not ArtifactCommitment: raise ArtifactOperationConflictError("artifact commitment is invalid") - provider_object_ref = self._provider_object_ref(commitment) + provider_object_ref = artifact_provider_object_ref(commitment) observed = await self.head(provider_object_ref) if not observed.exists: return ArtifactPutObservation(provider_object_ref, committed=False) @@ -318,7 +321,7 @@ def open( ) -> AsyncIterator[bytes]: """Stream a full object or range through bounded off-loop reads.""" self._require_initialized() - self._parse_provider_object_ref(provider_object_ref) + parse_artifact_provider_object_ref(provider_object_ref) if byte_range is not None and type(byte_range) is not ArtifactByteRange: raise ArtifactOperationConflictError("artifact byte range is invalid") selected_range = byte_range or ArtifactByteRange() @@ -371,7 +374,7 @@ async def iterate() -> AsyncIterator[bytes]: async def head(self, provider_object_ref: str) -> ArtifactObjectHead: """Return an existing or missing observation without exposing paths.""" self._require_initialized() - self._parse_provider_object_ref(provider_object_ref) + parse_artifact_provider_object_ref(provider_object_ref) try: try: byte_count = await self._run_io( @@ -743,21 +746,6 @@ def _stat_layout_file( finally: os.close(descriptor) - def _provider_object_ref(self, commitment: ArtifactCommitment) -> str: - """Derive one identity-free object reference from canonical SHA-256.""" - digest_hex = commitment.sha256[7:] - return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" - - @staticmethod - def _parse_provider_object_ref(provider_object_ref: str) -> tuple[str, str]: - """Validate the exact opaque local provider-reference grammar.""" - if not isinstance(provider_object_ref, str): - raise ArtifactOperationConflictError("artifact provider reference is invalid") - match = _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) - if match is None: - raise ArtifactOperationConflictError("artifact provider reference is invalid") - return match.group(1), match.group(2) - def _open_prefix(self, prefix: str, *, create: bool) -> int: """Open one digest-prefix directory without following links.""" if create: @@ -802,7 +790,7 @@ def _seal_temporary(descriptor: int) -> None: def _publish_exclusive(self, temporary_name: str, provider_object_ref: str) -> bool: """Hard-link one complete object without ever replacing existing bytes.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) prefix_fd = self._open_prefix(prefix, create=True) try: try: @@ -832,19 +820,19 @@ def _head_object(self, provider_object_ref: str) -> int: def _head_object_with_recovery(self, provider_object_ref: str) -> int: """Recover an interrupted publication under its digest lock before head.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) with self._locked_file(prefix + filename): return self._head_object(provider_object_ref) def _open_object_with_recovery(self, provider_object_ref: str) -> tuple[int, int]: """Recover an interrupted publication under its digest lock before open.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) with self._locked_file(prefix + filename): return self._open_object(provider_object_ref) def _open_object(self, provider_object_ref: str) -> tuple[int, int]: """Open one immutable no-follow object and return its exact size.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) prefix_fd = self._open_prefix(prefix, create=False) descriptor = -1 try: diff --git a/backend/app/adapters/artifacts/references.py b/backend/app/adapters/artifacts/references.py new file mode 100644 index 00000000..3ac7cd5b --- /dev/null +++ b/backend/app/adapters/artifacts/references.py @@ -0,0 +1,29 @@ +"""Canonical provider-object references shared by artifact adapters.""" + +from __future__ import annotations + +import re + +from app.interfaces.artifacts import ArtifactOperationConflictError +from app.modules.artifacts.sources import ArtifactCommitment + + +_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") + + +def artifact_provider_object_ref(commitment: ArtifactCommitment) -> str: + """Derive one identity-free reference solely from a server commitment.""" + if type(commitment) is not ArtifactCommitment: + raise ArtifactOperationConflictError("artifact commitment is invalid") + digest_hex = commitment.sha256[7:] + return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" + + +def parse_artifact_provider_object_ref(provider_object_ref: str) -> tuple[str, str]: + """Return digest path parts after enforcing the canonical grammar.""" + if not isinstance(provider_object_ref, str): + raise ArtifactOperationConflictError("artifact provider reference is invalid") + matched = _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) + if matched is None: + raise ArtifactOperationConflictError("artifact provider reference is invalid") + return matched.group(1), matched.group(2) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index fb1420cd..e941ce12 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -19,9 +19,13 @@ AioInstanceMetadataProvider, ) from aiobotocore.session import AioSession -from aiobotocore.utils import AioInstanceMetadataFetcher +from aiobotocore.utils import AioInstanceMetadataFetcher, _RefCountedSession from botocore.exceptions import BotoCoreError, ClientError +from app.adapters.artifacts.references import ( + artifact_provider_object_ref, + parse_artifact_provider_object_ref, +) from app.core.config import Settings from app.core.hashing import canonical_json_hash from app.interfaces.artifacts import ( @@ -49,7 +53,9 @@ _IDENTITY = ExternalServiceAdapterIdentity(ARTIFACT_STORE_CAPABILITY_KEY, "s3_compatible") -_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") +_CONTAINER_RELATIVE_URI = re.compile( + r"^/v2/credentials/[A-Za-z0-9._-]{1,512}$" +) _PRECONDITION_ERROR_CODES = frozenset( {"409", "412", "ConditionalRequestConflict", "PreconditionFailed"} ) @@ -252,7 +258,7 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: commitment = source.commitment if commitment.byte_count > HARD_MAXIMUM_ARTIFACT_BYTES: raise ArtifactLimitExceededError("artifact source exceeds provider limit") - provider_object_ref = self._provider_object_ref(commitment) + provider_object_ref = artifact_provider_object_ref(commitment) body = _CommittedSourceBody(source, self._buffer_bytes) try: if commitment.byte_count == 0: @@ -270,9 +276,9 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: IfNoneMatch="*", ) if not body.complete: - if await self._matches_commitment(provider_object_ref, commitment): - return ArtifactPutResult(provider_object_ref, replayed=True) - raise ArtifactStoreUnavailableError("S3 artifact operation failed") + raise ArtifactInputMismatchError( + "S3 provider did not consume the sealed artifact source" + ) return ArtifactPutResult(provider_object_ref, replayed=False) except ClientError as error: if _provider_error_code(error) in _PRECONDITION_ERROR_CODES: @@ -294,7 +300,7 @@ async def observe_put_result( self._require_initialized() if type(commitment) is not ArtifactCommitment: raise ArtifactOperationConflictError("artifact commitment is invalid") - provider_object_ref = self._provider_object_ref(commitment) + provider_object_ref = artifact_provider_object_ref(commitment) observed = await self.head(provider_object_ref) if not observed.exists: return ArtifactPutObservation(provider_object_ref, committed=False) @@ -308,7 +314,7 @@ def open( ) -> AsyncIterator[bytes]: """Stream a full object or one bounded byte range.""" self._require_initialized() - self._parse_provider_object_ref(provider_object_ref) + parse_artifact_provider_object_ref(provider_object_ref) if byte_range is not None and type(byte_range) is not ArtifactByteRange: raise ArtifactOperationConflictError("artifact byte range is invalid") selected_range = byte_range or ArtifactByteRange() @@ -375,7 +381,7 @@ async def iterate() -> AsyncIterator[bytes]: async def head(self, provider_object_ref: str) -> ArtifactObjectHead: """Return exact size or authoritative absence without provider details.""" self._require_initialized() - self._parse_provider_object_ref(provider_object_ref) + parse_artifact_provider_object_ref(provider_object_ref) try: async with asyncio.timeout(self._operation_total_timeout_seconds): async with self._client() as client: @@ -458,24 +464,9 @@ def _require_initialized(self) -> None: def _object_key(self, provider_object_ref: str) -> str: """Apply only the configured private prefix to a digest-derived reference.""" - self._parse_provider_object_ref(provider_object_ref) + parse_artifact_provider_object_ref(provider_object_ref) return f"{self._private_prefix}/{provider_object_ref}" - @staticmethod - def _provider_object_ref(commitment: ArtifactCommitment) -> str: - """Derive one identity-free reference solely from the server commitment.""" - digest_hex = commitment.sha256[7:] - return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" - - @staticmethod - def _parse_provider_object_ref(provider_object_ref: str) -> None: - """Reject caller-selected keys outside the canonical digest grammar.""" - if ( - not isinstance(provider_object_ref, str) - or _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) is None - ): - raise ArtifactOperationConflictError("artifact provider reference is invalid") - class _CommittedSourceBody: """Single-use async request body that revalidates the sealed commitment.""" @@ -657,14 +648,17 @@ def validate_aws_workload_identity_environment( raise ArtifactConfigurationError( "AWS container full credential URI is forbidden" ) - if "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" not in environment: + relative_uri = environment.get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI") + if ( + not isinstance(relative_uri, str) + or _CONTAINER_RELATIVE_URI.fullmatch(relative_uri) is None + ): raise ArtifactConfigurationError("AWS container identity location is invalid") - auth = _CONTAINER_ENVIRONMENT.intersection(environment) & { + if { "AWS_CONTAINER_AUTHORIZATION_TOKEN", "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", - } - if len(auth) > 1: - raise ArtifactConfigurationError("AWS container identity token is ambiguous") + }.intersection(environment): + raise ArtifactConfigurationError("AWS container identity token is forbidden") else: if "AWS_EC2_METADATA_SERVICE_ENDPOINT" in environment: raise ArtifactConfigurationError("custom AWS metadata endpoint is forbidden") @@ -700,7 +694,13 @@ def create_isolated_aws_workload_identity_session( disable_env_vars=True, ) elif selected == "container-role": - provider = AioContainerProvider(environ=environment) + provider = AioContainerProvider( + environ={ + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": environment[ + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" + ] + } + ) elif selected == "iam-role": provider = AioInstanceMetadataProvider( iam_role_fetcher=AioInstanceMetadataFetcher( @@ -717,6 +717,10 @@ def create_isolated_aws_workload_identity_session( ), "ec2_metadata_v1_disabled": True, }, + session=_RefCountedSession( + timeout=settings.artifact_s3_connect_timeout_seconds, + proxies={}, + ), ) ) else: diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 1acdab84..a4338294 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -229,6 +229,12 @@ def test_isolated_session_contains_exactly_one_selected_provider( } if method == "iam-role": assert provider._role_fetcher._imds_v1_disabled is True + assert ( + provider._role_fetcher._session._proxy_config.proxy_url_for( + "http://169.254.169.254/latest/meta-data/" + ) + is None + ) @pytest.mark.parametrize( @@ -266,7 +272,9 @@ def test_isolated_session_contains_exactly_one_selected_provider( lambda tmp_path: { "HOME": str(tmp_path), "AWS_CONTAINER_CREDENTIALS_FULL_URI": "http://127.0.0.1:9/full", - "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": "/relative", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ), }, "container full credential URI is forbidden", ), @@ -274,11 +282,23 @@ def test_isolated_session_contains_exactly_one_selected_provider( "container-role", lambda tmp_path: { "HOME": str(tmp_path), - "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": "/relative", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ), "AWS_CONTAINER_AUTHORIZATION_TOKEN": "token", + }, + "container identity token is forbidden", + ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ), "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE": str(tmp_path / "token"), }, - "container identity token is ambiguous", + "container identity token is forbidden", ), ( "iam-role", @@ -329,6 +349,52 @@ def test_selected_workload_identity_source_must_be_complete( ) +@pytest.mark.parametrize( + "relative_uri", + [ + "/relative", + "/v2/credentials/", + "/v2/credentials/workstream/runtime", + "https://credentials.example.test/v2/credentials/runtime", + "/v2/credentials/" + "a" * 513, + ], +) +def test_container_role_rejects_noncanonical_relative_uri( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + relative_uri: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment( + tmp_path, + AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=relative_uri, + ), + "container identity location is invalid", + ) + + +def test_iam_metadata_session_ignores_ambient_proxy_environment(tmp_path: Path) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, "iam-role"), + environ=_iam_environment( + tmp_path, + HTTP_PROXY="http://127.0.0.1:9", + HTTPS_PROXY="http://127.0.0.1:9", + ALL_PROXY="http://127.0.0.1:9", + ), + ) + provider = session.get_component("credential_provider").providers[0] + + assert ( + provider._role_fetcher._session._proxy_config.proxy_url_for( + "http://169.254.169.254/latest/meta-data/" + ) + is None + ) + + async def test_resolved_workload_identity_method_must_match() -> None: class FakeSession: def __init__(self, method: str | None) -> None: diff --git a/backend/tests/test_local_artifact_store.py b/backend/tests/test_local_artifact_store.py index fa61bf42..72bacc12 100644 --- a/backend/tests/test_local_artifact_store.py +++ b/backend/tests/test_local_artifact_store.py @@ -14,6 +14,10 @@ import pytest from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap +from app.adapters.artifacts.references import ( + artifact_provider_object_ref, + parse_artifact_provider_object_ref, +) from app.interfaces.artifacts import ( ArtifactByteRange, ArtifactConfigurationError, @@ -579,12 +583,12 @@ async def test_head_recovers_crash_after_link_before_temporary_unlink( try: async with minted_source(tmp_path / "scratch", b"crash recovery") as source: commitment = source.commitment - provider_ref = adapter._provider_object_ref(commitment) + provider_ref = artifact_provider_object_ref(commitment) descriptor, temporary_name = adapter._create_temporary() try: adapter._write_all(descriptor, memoryview(b"crash recovery")) adapter._seal_temporary(descriptor) - prefix, filename = adapter._parse_provider_object_ref(provider_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_ref) prefix_fd = adapter._open_prefix(prefix, create=True) try: os.link( diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index c5b858df..1c8ee37b 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -511,6 +511,38 @@ async def verify(provider_ref: str, _commitment: object) -> None: store.close() +@pytest.mark.asyncio +async def test_success_without_full_source_consumption_is_not_replay( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Reject a success response when the SDK did not validate all source bytes.""" + store = initialize_minio_store(private_prefix="negative/unconsumed-success") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + return {} + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def must_not_classify_as_replay(*_args: object) -> bool: + raise AssertionError("unconsumed successful writes are not replay candidates") + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_matches_commitment", must_not_classify_as_replay) + try: + async with minted_source(tmp_path / "scratch", b"unconsumed") as source: + with pytest.raises( + ArtifactInputMismatchError, + match="did not consume the sealed artifact source", + ): + await store.put(source) + finally: + store.close() + + @pytest.mark.asyncio async def test_nonprecondition_put_error_is_unavailable( monkeypatch: pytest.MonkeyPatch, diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 1b497e7c..b5da59a5 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -470,12 +470,14 @@ only the dedicated test bucket; tests do not emulate S3 calls. `aws_s3` accepts only a native endpoint-less AWS configuration, explicit region, and exactly one allowlisted workload-identity method. The composition -root validates that environment but then raises -`artifact_provider_live_proof_required` before constructing the adapter -factory, building or loading the credential resolver, claiming a namespace, or -performing provider I/O. Chunk 07 alone may make native AWS runtime-eligible by -adding the immutable live activation proof defined below. MinIO conformance is -not AWS activation evidence. +root validates settings and then raises `artifact_provider_live_proof_required` +before validating credential environment, constructing the adapter factory, +building or loading the credential resolver, claiming a namespace, or +performing provider I/O. The credential-environment validator and isolated +resolver are inactive deployment-proof support in this chunk; startup does not +invoke them. Chunk 07 alone may make native AWS runtime-eligible by adding the +immutable live activation proof defined below. MinIO conformance is not AWS +activation evidence. Cloudflare R2 has no v0.1 runtime mode, credential issuer, sidecar, secret contract, configuration profile, or deployment proof. Any future provider diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index a438370a..abf0df23 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -4184,8 +4184,9 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: assert "No later artifact chunk starts automatically" in artifact_status.replace( "\n", " " ) - assert "| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 |" in ( - work_queue + assert ( + "| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | " + "Merged through PR #146 as `0ffdabf`" in work_queue ) assert "| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Active" in ( work_queue From 4a7d8b085a65c074a912a82c9bc2df6360089ba9 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 09:15:03 +0100 Subject: [PATCH 05/39] Prove S3 replay and metadata isolation --- .agent-loop/LOOP_STATE.md | 5 +- README.md | 12 +++-- .../app/adapters/artifacts/s3_compatible.py | 12 ++++- .../tests/test_aws_credential_isolation.py | 39 +++++++++++++++ backend/tests/test_s3_artifact_store.py | 49 +++++++++++++++++++ docs/spec_artifact_storage_service.md | 3 +- 6 files changed, 112 insertions(+), 8 deletions(-) diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index 396b2aed..5de0975d 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -38,8 +38,9 @@ Flow Node are deferred. Product modules receive narrow artifact capabilities, and AWS cannot instantiate in production without release-bound live proof. - Authorization checkpoint: merged main contains 74 PermissionIds and 65 - ActionIds, with the two actor-self actions, seven AUTH-08 administrative - actions, and AUTH-09B `actor.service.provision` active. Merged AUTH-09A defines + ActionIds, with 12 active actions: the two actor-self actions, seven AUTH-08 + administrative actions, AUTH-09B `actor.service.provision`, and AUTH-09C + `actor.profile.read` plus `actor.identity_link.read`. Merged AUTH-09A defines seven fixed artifact service identities and eleven exact planned static matrix memberships. ART feature chunks supply hidden canonical behavior/resource composition. Merged diff --git a/README.md b/README.md index a89e3a2e..1be50461 100644 --- a/README.md +++ b/README.md @@ -195,10 +195,14 @@ docker compose up -d postgres redis minio ``` MinIO uses the compose-only static credentials and the private -`workstream-artifacts` bucket. Native AWS S3 accepts workload-identity -configuration but remains runtime-ineligible until live deployment proof is -approved; startup fails with `artifact_provider_live_proof_required` before -credential probing or provider I/O. +`workstream-artifacts` bucket. The integration tests create that bucket +automatically. For local runtime use, create the private bucket with an S3 +client against `http://localhost:9000` after MinIO is healthy, using access key +`workstream-minio` and secret key `workstream-minio-secret-key`, before starting +Workstream. Native AWS S3 accepts workload-identity configuration but remains +runtime-ineligible until live deployment proof is approved; startup fails with +`artifact_provider_live_proof_required` before credential probing or provider +I/O. The default local development URL is: diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index e941ce12..0e43a286 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -19,7 +19,11 @@ AioInstanceMetadataProvider, ) from aiobotocore.session import AioSession -from aiobotocore.utils import AioInstanceMetadataFetcher, _RefCountedSession +from aiobotocore.utils import ( + AioContainerMetadataFetcher, + AioInstanceMetadataFetcher, + _RefCountedSession, +) from botocore.exceptions import BotoCoreError, ClientError from app.adapters.artifacts.references import ( @@ -701,6 +705,12 @@ def create_isolated_aws_workload_identity_session( ] } ) + provider._fetcher = AioContainerMetadataFetcher( + session=_RefCountedSession( + timeout=settings.artifact_s3_connect_timeout_seconds, + proxies={}, + ) + ) elif selected == "iam-role": provider = AioInstanceMetadataProvider( iam_role_fetcher=AioInstanceMetadataFetcher( diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index a4338294..c6dcb33e 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -227,6 +227,18 @@ def test_isolated_session_contains_exactly_one_selected_provider( } } } + if method == "container-role": + assert provider._environ == { + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ) + } + assert ( + provider._fetcher._session._proxy_config.proxy_url_for( + "http://169.254.170.2/v2/credentials/workstream-runtime" + ) + is None + ) if method == "iam-role": assert provider._role_fetcher._imds_v1_disabled is True assert ( @@ -395,6 +407,33 @@ def test_iam_metadata_session_ignores_ambient_proxy_environment(tmp_path: Path) ) +def test_container_metadata_session_ignores_ambient_proxy_environment( + tmp_path: Path, +) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, "container-role"), + environ=_container_environment( + tmp_path, + HTTP_PROXY="http://127.0.0.1:9", + HTTPS_PROXY="http://127.0.0.1:9", + ALL_PROXY="http://127.0.0.1:9", + ), + ) + provider = session.get_component("credential_provider").providers[0] + + assert provider._environ == { + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ) + } + assert ( + provider._fetcher._session._proxy_config.proxy_url_for( + "http://169.254.170.2/v2/credentials/workstream-runtime" + ) + is None + ) + + async def test_resolved_workload_identity_method_must_match() -> None: class FakeSession: def __init__(self, method: str | None) -> None: diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 1c8ee37b..c1a7d83a 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -607,6 +607,55 @@ async def matches(provider_ref: str, _commitment: object) -> bool: store.close() +@pytest.mark.asyncio +@pytest.mark.parametrize("stored_bytes_match", [False, True]) +async def test_uncertain_transport_put_uses_real_minio_verification( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + stored_bytes_match: bool, +) -> None: + """Classify acknowledgement loss only from the complete provider bytes.""" + nonce = hashlib.sha256(os.urandom(16)).hexdigest()[:20] + private_prefix = f"uncertain/real-observation/{nonce}" + expected = b"exact committed bytes" + digest = hashlib.sha256(expected).hexdigest() + provider_ref = f"sha256/{digest[:2]}/{digest[2:]}" + stored = expected if stored_bytes_match else b"wrong existing bytes" + await raw_minio_put(f"{private_prefix}/{provider_ref}", stored) + + store = initialize_minio_store(private_prefix=private_prefix) + original_client = store._client + client_calls = 0 + + class LostAcknowledgementClient: + async def put_object(self, **_kwargs: object) -> object: + raise OSError("connection closed after provider write") + + @asynccontextmanager + async def lost_acknowledgement_then_real_provider() -> Any: + nonlocal client_calls + client_calls += 1 + if client_calls == 1: + yield LostAcknowledgementClient() + return + async with original_client() as client: + yield client + + monkeypatch.setattr(store, "_client", lost_acknowledgement_then_real_provider) + try: + async with minted_source(tmp_path / "scratch", expected) as source: + if stored_bytes_match: + result = await store.put(source) + assert result.provider_object_ref == provider_ref + assert result.replayed is True + else: + with pytest.raises(ArtifactIntegrityError): + await store.put(source) + assert client_calls >= 3 + finally: + store.close() + + @pytest.mark.asyncio async def test_put_observation_requires_canonical_commitment() -> None: """Reject caller-assembled observation requests before provider I/O.""" diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index b5da59a5..3bbdced5 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -1107,7 +1107,8 @@ bytes in PostgreSQL. ## Verification Strategy - one conformance suite runs against LocalStorage and real MinIO; -- a secret-free live smoke profile proves AWS S3 configuration; +- inactive validation proves the closed AWS S3 configuration contract, while + Chunk 07 owns live deployment proof; - integration tests use real S3-compatible API calls, not monkeypatching; - concurrent puts cannot overwrite existing bytes; - acknowledgement loss, exact replay, oversized-object refusal, truncation, From 6af0a535f17211671d45648af3375ad728ce9ddb Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 09:38:04 +0100 Subject: [PATCH 06/39] Close AWS isolation and settings validation gaps --- .../app/adapters/artifacts/s3_compatible.py | 44 ++++++++++++- backend/app/core/config.py | 20 +++++- .../tests/test_aws_credential_isolation.py | 62 +++++++++++++++++++ backend/tests/test_config.py | 32 +++++++++- backend/tests/test_s3_artifact_store.py | 29 +++++++++ docs/spec_artifact_storage_service.md | 8 ++- 6 files changed, 185 insertions(+), 10 deletions(-) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index 0e43a286..f825e9a4 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -24,6 +24,7 @@ AioInstanceMetadataFetcher, _RefCountedSession, ) +from botocore import UNSIGNED from botocore.exceptions import BotoCoreError, ClientError from app.adapters.artifacts.references import ( @@ -81,6 +82,20 @@ "BOTO_CONFIG", } ) +_FORBIDDEN_AWS_NETWORK_ENVIRONMENT = frozenset( + { + "AWS_CA_BUNDLE", + "AWS_ENDPOINT_URL", + "AWS_ENDPOINT_URL_STS", + "AWS_STS_REGIONAL_ENDPOINTS", + "AWS_USE_DUALSTACK_ENDPOINT", + "AWS_USE_FIPS_ENDPOINT", + "CURL_CA_BUNDLE", + "REQUESTS_CA_BUNDLE", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + } +) _WEB_IDENTITY_ENVIRONMENT = frozenset( {"AWS_ROLE_ARN", "AWS_ROLE_SESSION_NAME", "AWS_WEB_IDENTITY_TOKEN_FILE"} ) @@ -285,7 +300,13 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: ) return ArtifactPutResult(provider_object_ref, replayed=False) except ClientError as error: - if _provider_error_code(error) in _PRECONDITION_ERROR_CODES: + status = _provider_http_status(error) + if status == 403: + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + if ( + status in {409, 412} + and _provider_error_code(error) in _PRECONDITION_ERROR_CODES + ): await self._verify_exact(provider_object_ref, commitment) return ArtifactPutResult(provider_object_ref, replayed=True) raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None @@ -629,6 +650,8 @@ def validate_aws_workload_identity_environment( environment = os.environ if environ is None else environ if _FORBIDDEN_AWS_CREDENTIAL_ENVIRONMENT.intersection(environment): raise ArtifactConfigurationError("ambient AWS credential source is forbidden") + if _FORBIDDEN_AWS_NETWORK_ENVIRONMENT.intersection(environment): + raise ArtifactConfigurationError("ambient AWS network configuration is forbidden") _reject_default_credential_files(environment) allowed_by_method = { @@ -690,9 +713,26 @@ def create_isolated_aws_workload_identity_session( role_session_name = environment.get("AWS_ROLE_SESSION_NAME") if role_session_name is not None: profile["role_session_name"] = role_session_name + + def create_isolated_sts_client(service_name: str, **_kwargs: object) -> object: + if service_name != "sts": + raise ArtifactConfigurationError("AWS web identity service is invalid") + return session.create_client( + "sts", + region_name=_required(settings.artifact_s3_region), + verify=True, + config=AioConfig( + signature_version=UNSIGNED, + connect_timeout=settings.artifact_s3_connect_timeout_seconds, + read_timeout=settings.artifact_s3_read_timeout_seconds, + retries={"max_attempts": 1, "mode": "standard"}, + proxies={}, + ), + ) + provider = AioAssumeRoleWithWebIdentityProvider( load_config=lambda: {"profiles": {"workstream-isolated": profile}}, - client_creator=session.create_client, + client_creator=create_isolated_sts_client, profile_name="workstream-isolated", cache={}, disable_env_vars=True, diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 8d1cb97f..09cadbb5 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -4,6 +4,7 @@ import base64 import binascii +from contextvars import ContextVar import json import os import re @@ -33,6 +34,10 @@ _S3_REGION = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$") _S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") _S3_PREFIX_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") +_ALTERNATE_VALIDATION_RESTORES_SECRETS: ContextVar[bool] = ContextVar( + "alternate_validation_restores_secrets", + default=False, +) class Settings(BaseSettings): @@ -190,7 +195,8 @@ def __init__(self, **values: object) -> None: super().__init__(**values) self._api_rate_limit_key_secret = secret self._set_artifact_s3_static_secrets(s3_secrets) - self._validate_artifact_s3_secret_contract() + if not _ALTERNATE_VALIDATION_RESTORES_SECRETS.get(): + self._validate_artifact_s3_secret_contract() @classmethod def model_validate(cls, obj: object, **kwargs: object) -> Self: @@ -211,7 +217,11 @@ def model_validate(cls, obj: object, **kwargs: object) -> Self: for field_name in _ARTIFACT_S3_SECRET_FIELDS: if field_name in obj: sanitized[field_name] = None - settings = super().model_validate(sanitized, **kwargs) + restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + try: + settings = super().model_validate(sanitized, **kwargs) + finally: + _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) if secret is not None: settings._api_rate_limit_key_secret = secret settings._set_artifact_s3_static_secrets(s3_secrets) @@ -260,7 +270,11 @@ def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: for field_name in _ARTIFACT_S3_SECRET_FIELDS: if field_name in obj: sanitized[field_name] = None - settings = super().model_validate_strings(sanitized, **kwargs) + restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + try: + settings = super().model_validate_strings(sanitized, **kwargs) + finally: + _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) if secret is not None: settings._api_rate_limit_key_secret = secret settings._set_artifact_s3_static_secrets(s3_secrets) diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index c6dcb33e..2e825114 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -117,6 +117,34 @@ def test_forbidden_ambient_aws_sources_fail_before_resolver_loading( ) +@pytest.mark.parametrize( + "name", + [ + "AWS_CA_BUNDLE", + "AWS_ENDPOINT_URL", + "AWS_ENDPOINT_URL_STS", + "AWS_STS_REGIONAL_ENDPOINTS", + "AWS_USE_DUALSTACK_ENDPOINT", + "AWS_USE_FIPS_ENDPOINT", + "CURL_CA_BUNDLE", + "REQUESTS_CA_BUNDLE", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + ], +) +def test_web_identity_rejects_ambient_network_configuration_before_session( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "assume-role-with-web-identity"), + _web_identity_environment(tmp_path, **{name: "/tmp/poison"}), + "ambient AWS network configuration is forbidden", + ) + + @pytest.mark.parametrize( ("relative_path", "contents"), [ @@ -434,6 +462,40 @@ def test_container_metadata_session_ignores_ambient_proxy_environment( ) +def test_web_identity_sts_client_ignores_ambient_proxy_environment( + tmp_path: Path, +) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, "assume-role-with-web-identity"), + environ=_web_identity_environment( + tmp_path, + HTTP_PROXY="http://127.0.0.1:9", + HTTPS_PROXY="http://127.0.0.1:9", + ALL_PROXY="http://127.0.0.1:9", + ), + ) + provider = session.get_component("credential_provider").providers[0] + observed: dict[str, object] = {} + marker = object() + + def capture_client(service_name: str, **kwargs: object) -> object: + observed["service_name"] = service_name + observed.update(kwargs) + return marker + + session.create_client = capture_client # type: ignore[method-assign] + + created = provider._client_creator("sts", config=object()) + + assert created is marker + assert observed["service_name"] == "sts" + assert observed["region_name"] == "us-east-1" + assert observed["verify"] is True + config = observed["config"] + assert config.signature_version == s3_compatible.UNSIGNED + assert config.proxies == {} + + async def test_resolved_workload_identity_method_must_match() -> None: class FakeSession: def __init__(self, method: str | None) -> None: diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 0db301f3..224f194d 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -358,7 +358,7 @@ def _flow_settings(**overrides) -> Settings: return Settings(**values) -def _minio_settings(tmp_path: Path, **overrides: object) -> Settings: +def _minio_setting_values(tmp_path: Path, **overrides: object) -> dict[str, object]: values: dict[str, object] = { "environment": "test", "artifact_store_backend": "s3_compatible", @@ -374,7 +374,11 @@ def _minio_settings(tmp_path: Path, **overrides: object) -> Settings: "artifact_s3_secret_access_key": "minio-secret-key", } values.update(overrides) - return Settings(**values) + return values + + +def _minio_settings(tmp_path: Path, **overrides: object) -> Settings: + return Settings(**_minio_setting_values(tmp_path, **overrides)) def _aws_settings(tmp_path: Path, **overrides: object) -> Settings: @@ -683,6 +687,30 @@ def test_minio_secret_values_are_absent_from_repr_and_validation_errors( _assert_secret_not_retained(caught.value, secret) +@pytest.mark.parametrize( + "method_name", + ["model_validate", "model_validate_json", "model_validate_strings"], +) +def test_alternate_validation_loads_minio_secrets_without_retaining_them( + tmp_path: Path, + method_name: str, +) -> None: + secret = "alternate-minio-secret-value" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + ) + method = getattr(Settings, method_name) + + settings = method(json.dumps(payload) if method_name.endswith("json") else payload) + + assert settings.artifact_s3_secret_access_key is not None + assert settings.artifact_s3_secret_access_key.get_secret_value() == secret + assert secret not in repr(settings) + assert secret not in repr(settings.model_dump()) + + def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index c1a7d83a..2dd8fe85 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -568,6 +568,35 @@ async def fake_client() -> Any: store.close() +@pytest.mark.asyncio +async def test_put_403_precedes_precondition_like_provider_code( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Never classify denied writes as replay candidates.""" + store = initialize_minio_store(private_prefix="negative/denied-precondition") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("PreconditionFailed", 403) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def verification_must_not_run(*_args: object) -> None: + raise AssertionError("denied writes cannot enter replay verification") + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_verify_exact", verification_must_not_run) + try: + async with minted_source(tmp_path / "scratch", b"denied") as source: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + finally: + store.close() + + @pytest.mark.asyncio @pytest.mark.parametrize("committed", [False, True]) async def test_uncertain_transport_put_requires_independent_observation( diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 3bbdced5..89eba578 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -451,9 +451,11 @@ pinned botocore credential resolver to that provider before any provider is loaded, and verifies the resolved method. Explicit credentials, environment access keys, shared credential/config files, credential processes, login/SSO, legacy EC2/Boto sources, and every unselected workload provider are rejected. -Tests poison every nonselected source and prove that it is never read, executed, -or contacted. This is an allowlisted workload-identity contract, not authority -to use the ambient SDK default chain. +Web-identity STS, ECS container metadata, and EC2 instance metadata use +explicit no-proxy sessions. Web identity also rejects ambient endpoint and CA +overrides. Tests poison every nonselected source and prove that it is never +read, executed, or contacted. This is an allowlisted workload-identity +contract, not authority to use the ambient SDK default chain. Chunk 02B1 pins the async S3 SDK pair to `aiobotocore==3.7.0` and `botocore==1.43.0`. An SDK upgrade is a reviewed contract change because From caa906281c87ef26f97dc3df9d092c2ea110279e Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 10:01:10 +0100 Subject: [PATCH 07/39] Fail closed on S3 endpoint and absence ambiguity --- README.md | 4 +- .../app/adapters/artifacts/s3_compatible.py | 14 +++--- .../tests/test_aws_credential_isolation.py | 14 ++++++ backend/tests/test_s3_artifact_store.py | 48 +++++++++++++++++-- 4 files changed, 66 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 1be50461..61aaaa16 100644 --- a/README.md +++ b/README.md @@ -199,7 +199,9 @@ MinIO uses the compose-only static credentials and the private automatically. For local runtime use, create the private bucket with an S3 client against `http://localhost:9000` after MinIO is healthy, using access key `workstream-minio` and secret key `workstream-minio-secret-key`, before starting -Workstream. Native AWS S3 accepts workload-identity configuration but remains +Workstream. Configure the runtime with the exact +[artifact storage settings](docs/spec_artifact_storage_service.md#s3-compatible-adapter). +Native AWS S3 accepts workload-identity configuration but remains runtime-ineligible until live deployment proof is approved; startup fails with `artifact_provider_live_proof_required` before credential probing or provider I/O. diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index f825e9a4..a3afe7a4 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -390,10 +390,8 @@ async def iterate() -> AsyncIterator[bytes]: if observed != expected: raise ArtifactIntegrityError("S3 artifact object was truncated") except ClientError as error: - if _provider_http_status(error) == 403: - raise ArtifactStoreUnavailableError("S3 artifact read failed") from None - code = _provider_error_code(error) - if code in _MISSING_ERROR_CODES: + status = _provider_http_status(error) + if status == 404 and _provider_error_code(error) in _MISSING_ERROR_CODES: raise ArtifactObjectMissingError("artifact object is missing") from None raise ArtifactStoreUnavailableError("S3 artifact read failed") from None except ArtifactStoreError: @@ -423,10 +421,8 @@ async def head(self, provider_object_ref: str) -> ArtifactObjectHead: byte_count=byte_count, ) except ClientError as error: - if _provider_http_status(error) == 403: - raise ArtifactStoreUnavailableError("S3 artifact head failed") from None - code = _provider_error_code(error) - if code in _MISSING_ERROR_CODES: + status = _provider_http_status(error) + if status == 404 and _provider_error_code(error) in _MISSING_ERROR_CODES: return ArtifactObjectHead(provider_object_ref, exists=False) raise ArtifactStoreUnavailableError("S3 artifact head failed") from None except ArtifactStoreError: @@ -652,6 +648,8 @@ def validate_aws_workload_identity_environment( raise ArtifactConfigurationError("ambient AWS credential source is forbidden") if _FORBIDDEN_AWS_NETWORK_ENVIRONMENT.intersection(environment): raise ArtifactConfigurationError("ambient AWS network configuration is forbidden") + if any(name.startswith("AWS_ENDPOINT_URL_") for name in environment): + raise ArtifactConfigurationError("ambient AWS network configuration is forbidden") _reject_default_credential_files(environment) allowed_by_method = { diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 2e825114..602e8a6b 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -145,6 +145,20 @@ def test_web_identity_rejects_ambient_network_configuration_before_session( ) +@pytest.mark.parametrize("name", ["AWS_ENDPOINT_URL_S3", "AWS_ENDPOINT_URL_DYNAMODB"]) +def test_service_specific_endpoint_override_fails_before_session_construction( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment(tmp_path, **{name: "http://127.0.0.1:9000"}), + "ambient AWS network configuration is forbidden", + ) + + @pytest.mark.parametrize( ("relative_path", "contents"), [ diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 2dd8fe85..703b0fb8 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -480,6 +480,42 @@ async def fake_client() -> Any: store.close() +@pytest.mark.asyncio +@pytest.mark.parametrize("operation", ["head", "open"]) +@pytest.mark.parametrize("status", [500, None]) +async def test_missing_like_code_requires_authoritative_404_status( + monkeypatch: pytest.MonkeyPatch, + operation: str, + status: int | None, +) -> None: + """Treat missing-like codes without HTTP 404 as provider failures.""" + store = initialize_minio_store(private_prefix=f"negative/ambiguous-{operation}") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + if operation == "head": + raise _client_error("NoSuchKey", status) + return {"ContentLength": 1} + + async def get_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", status) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactStoreUnavailableError): + if operation == "head": + await store.head(provider_ref) + else: + _ = [chunk async for chunk in store.open(provider_ref)] + finally: + store.close() + + @pytest.mark.asyncio async def test_precondition_failure_requires_exact_replay_verification( monkeypatch: pytest.MonkeyPatch, @@ -696,12 +732,14 @@ async def test_put_observation_requires_canonical_commitment() -> None: store.close() -def _client_error(code: str, status: int) -> ClientError: +def _client_error(code: str, status: int | None) -> ClientError: """Build one sanitized provider error for adapter mapping tests.""" + response: dict[str, object] = { + "Error": {"Code": code, "Message": "provider detail"}, + } + if status is not None: + response["ResponseMetadata"] = {"HTTPStatusCode": status} return ClientError( - { - "Error": {"Code": code, "Message": "provider detail"}, - "ResponseMetadata": {"HTTPStatusCode": status}, - }, + response, "HeadObject", ) From 9e8ccad254dc36ea2fb3250e5686aba7b289f14d Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 10:32:05 +0100 Subject: [PATCH 08/39] Harden AWS workload environment isolation --- .../PLAN.md | 5 +- ...WS-ART-001-02B1-s3-compatible-minio-aws.md | 3 + .../app/adapters/artifacts/s3_compatible.py | 78 ++++++++++++------- .../tests/test_aws_credential_isolation.py | 45 +++++++++++ docs/spec_artifact_storage_service.md | 6 +- 5 files changed, 106 insertions(+), 31 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index ce764b8d..6938958d 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -89,7 +89,10 @@ mode `aws_workload_identity` selects exactly one allowlisted method: constrains the pinned credential resolver to the selected provider before any provider is loaded, verifies the resolved method, and rejects explicit credentials, ambient access keys, file/process/login/SSO sources, legacy -EC2/Boto sources, and every unselected workload provider. Chunk 02B1 pins +EC2/Boto sources, and every unselected workload provider. For the selected +method, startup accepts only that method's exact closed `AWS_*` environment +allowlist and rejects every other `AWS_*` or `BOTOCORE_*` SDK control before +constructing a session. Chunk 02B1 pins `aiobotocore==3.7.0` and `botocore==1.43.0`; SDK upgrades require an explicit dependency and credential-behavior review. MinIO static credentials are local/CI only. The endpoint is omitted for native AWS S3 and explicit for diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index 002177b9..ce3ee83a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -57,6 +57,9 @@ credential-delivery contract. - explicit credentials, environment access keys, shared credential/config files, credential processes, login/SSO, legacy EC2/Boto sources, and every unselected workload provider fail startup before credential loading; +- each selected workload method accepts only its exact closed `AWS_*` + environment allowlist; every other `AWS_*` or `BOTOCORE_*` SDK behavior + control fails before SDK session construction; - container role accepts only the relative metadata path, IAM role rejects a custom metadata endpoint and requires IMDSv2, and the isolated IAM fetcher disables IMDSv1; diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index a3afe7a4..465a0483 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -96,10 +96,10 @@ "SSL_CERT_FILE", } ) -_WEB_IDENTITY_ENVIRONMENT = frozenset( +_WEB_IDENTITY_SOURCE_ENVIRONMENT = frozenset( {"AWS_ROLE_ARN", "AWS_ROLE_SESSION_NAME", "AWS_WEB_IDENTITY_TOKEN_FILE"} ) -_CONTAINER_ENVIRONMENT = frozenset( +_CONTAINER_SOURCE_ENVIRONMENT = frozenset( { "AWS_CONTAINER_AUTHORIZATION_TOKEN", "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", @@ -107,7 +107,7 @@ "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", } ) -_IAM_ROLE_ENVIRONMENT = frozenset( +_IAM_ROLE_SOURCE_ENVIRONMENT = frozenset( { "AWS_EC2_METADATA_DISABLED", "AWS_EC2_METADATA_SERVICE_ENDPOINT", @@ -115,6 +115,16 @@ "AWS_EC2_METADATA_V1_DISABLED", } ) +_ALLOWED_AWS_ENVIRONMENT_BY_METHOD = { + "assume-role-with-web-identity": _WEB_IDENTITY_SOURCE_ENVIRONMENT, + "container-role": frozenset({"AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"}), + "iam-role": frozenset({"AWS_EC2_METADATA_V1_DISABLED"}), +} +_AWS_SOURCE_ENVIRONMENT_BY_METHOD = { + "assume-role-with-web-identity": _WEB_IDENTITY_SOURCE_ENVIRONMENT, + "container-role": _CONTAINER_SOURCE_ENVIRONMENT, + "iam-role": _IAM_ROLE_SOURCE_ENVIRONMENT, +} class S3CompatibleArtifactStoreBootstrap: @@ -652,16 +662,47 @@ def validate_aws_workload_identity_environment( raise ArtifactConfigurationError("ambient AWS network configuration is forbidden") _reject_default_credential_files(environment) - allowed_by_method = { - "assume-role-with-web-identity": _WEB_IDENTITY_ENVIRONMENT, - "container-role": _CONTAINER_ENVIRONMENT, - "iam-role": _IAM_ROLE_ENVIRONMENT, - } unselected = set().union( - *(values for method, values in allowed_by_method.items() if method != selected) + *( + values + for method, values in _AWS_SOURCE_ENVIRONMENT_BY_METHOD.items() + if method != selected + ) ) if unselected.intersection(environment): raise ArtifactConfigurationError("unselected AWS workload source is configured") + if selected == "container-role": + if "AWS_CONTAINER_CREDENTIALS_FULL_URI" in environment: + raise ArtifactConfigurationError( + "AWS container full credential URI is forbidden" + ) + if { + "AWS_CONTAINER_AUTHORIZATION_TOKEN", + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", + }.intersection(environment): + raise ArtifactConfigurationError("AWS container identity token is forbidden") + elif selected == "iam-role": + if "AWS_EC2_METADATA_SERVICE_ENDPOINT" in environment: + raise ArtifactConfigurationError("custom AWS metadata endpoint is forbidden") + if "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE" in environment: + raise ArtifactConfigurationError( + "custom AWS metadata endpoint mode is forbidden" + ) + if environment.get("AWS_EC2_METADATA_DISABLED", "false").lower() == "true": + raise ArtifactConfigurationError("AWS instance identity metadata is disabled") + + allowed_environment = _ALLOWED_AWS_ENVIRONMENT_BY_METHOD[selected] + unsupported_sdk_environment = { + name + for name in environment + if (name.startswith("AWS_") or name.startswith("BOTOCORE_")) + and name not in allowed_environment + } + if unsupported_sdk_environment: + raise ArtifactConfigurationError( + "unsupported AWS SDK environment configuration is forbidden" + ) + if selected == "assume-role-with-web-identity": if not {"AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"}.issubset(environment): raise ArtifactConfigurationError("AWS web identity configuration is incomplete") @@ -669,26 +710,13 @@ def validate_aws_workload_identity_environment( if not token_path.is_absolute() or not token_path.is_file(): raise ArtifactConfigurationError("AWS web identity token path is invalid") elif selected == "container-role": - if "AWS_CONTAINER_CREDENTIALS_FULL_URI" in environment: - raise ArtifactConfigurationError( - "AWS container full credential URI is forbidden" - ) relative_uri = environment.get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI") if ( not isinstance(relative_uri, str) or _CONTAINER_RELATIVE_URI.fullmatch(relative_uri) is None ): raise ArtifactConfigurationError("AWS container identity location is invalid") - if { - "AWS_CONTAINER_AUTHORIZATION_TOKEN", - "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", - }.intersection(environment): - raise ArtifactConfigurationError("AWS container identity token is forbidden") else: - if "AWS_EC2_METADATA_SERVICE_ENDPOINT" in environment: - raise ArtifactConfigurationError("custom AWS metadata endpoint is forbidden") - if environment.get("AWS_EC2_METADATA_DISABLED", "false").lower() == "true": - raise ArtifactConfigurationError("AWS instance identity metadata is disabled") if environment.get("AWS_EC2_METADATA_V1_DISABLED", "false").lower() != "true": raise ArtifactConfigurationError("AWS instance identity requires IMDSv2") @@ -757,12 +785,6 @@ def create_isolated_sts_client(service_name: str, **_kwargs: object) -> object: env=dict(environment), user_agent=session.user_agent(), config={ - "ec2_metadata_service_endpoint": environment.get( - "AWS_EC2_METADATA_SERVICE_ENDPOINT" - ), - "ec2_metadata_service_endpoint_mode": environment.get( - "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE" - ), "ec2_metadata_v1_disabled": True, }, session=_RefCountedSession( diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 602e8a6b..5b1664dd 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -159,6 +159,42 @@ def test_service_specific_endpoint_override_fails_before_session_construction( ) +@pytest.mark.parametrize( + "name", + [ + "AWS_ACCOUNT_ID_ENDPOINT_MODE", + "AWS_DATA_PATH", + "AWS_DEFAULTS_MODE", + "AWS_ENDPOINT_DISCOVERY_ENABLED", + "AWS_MAX_ATTEMPTS", + "AWS_REGION", + "AWS_RETRY_MODE", + "BOTOCORE_CONFIG", + ], +) +@pytest.mark.parametrize( + ("method", "environment"), + [ + ("assume-role-with-web-identity", _web_identity_environment), + ("container-role", _container_environment), + ("iam-role", _iam_environment), + ], +) +def test_unapproved_sdk_environment_controls_fail_before_session_construction( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, + method: str, + environment: Any, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, method), + environment(tmp_path, **{name: "poison"}), + "unsupported AWS SDK environment configuration is forbidden", + ) + + @pytest.mark.parametrize( ("relative_path", "contents"), [ @@ -386,6 +422,15 @@ def test_isolated_session_contains_exactly_one_selected_provider( }, "custom AWS metadata endpoint is forbidden", ), + ( + "iam-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE": "ipv6", + "AWS_EC2_METADATA_V1_DISABLED": "true", + }, + "custom AWS metadata endpoint mode is forbidden", + ), ], ) def test_selected_workload_identity_source_must_be_complete( diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 89eba578..3007cd74 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -811,8 +811,10 @@ MinIO is the only runtime-eligible S3 profile in this chunk and is restricted to local, development, and test environments. Native AWS configuration rejects static credentials, configured endpoints, unselected or ambient credential sources, arbitrary container credential URLs, and custom instance-metadata -endpoints. IAM-role validation requires IMDSv2 and the isolated fetcher disables -IMDSv1. Even valid native AWS settings fail with +endpoints. The selected method accepts only its exact closed `AWS_*` +environment allowlist; every other `AWS_*` or `BOTOCORE_*` SDK behavior control +fails before SDK session construction. IAM-role validation requires IMDSv2 and +the isolated fetcher disables IMDSv1. Even valid native AWS settings fail with `artifact_provider_live_proof_required` before credential-source probing, factory construction, namespace claim, resolver loading, or provider I/O. From 6724dda8fe5b782cf2a5bce523072b782ea91b3c Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 11:15:27 +0100 Subject: [PATCH 09/39] Close artifact storage isolation gaps --- .../PLAN.md | 5 +- ...WS-ART-001-02B1-s3-compatible-minio-aws.md | 20 +++-- ...1-02C1-admission-put-attempt-foundation.md | 1 + ...1-02C2-verification-publication-fencing.md | 1 + ...T-001-02C3-recovery-attempt-idempotency.md | 1 + ...RT-001-02D-operator-artifact-operations.md | 1 + .../WS-ART-001-03-guide-source-cutover.md | 1 + ...S-ART-001-04A-upload-inspection-sealing.md | 1 + .../WS-ART-001-04B-pre-submit-admission.md | 1 + ...-ART-001-05-submission-artifact-cutover.md | 1 + ...T-001-06A-checker-input-materialization.md | 1 + .../WS-ART-001-06B-checker-output-routing.md | 1 + .../WS-ART-001-07-recovery-live-proof.md | 1 + .github/workflows/backend.yml | 8 ++ backend/app/adapters/artifacts/__init__.py | 20 +++-- .../app/adapters/artifacts/s3_compatible.py | 23 +++++ backend/app/core/config.py | 69 +++------------ backend/app/core/s3_validation.py | 88 +++++++++++++++++++ backend/app/interfaces/artifacts.py | 6 ++ backend/app/workers/artifacts.py | 6 +- backend/app/workers/celery_app.py | 2 + backend/tests/test_artifact_cleanup_wiring.py | 29 ++++++ .../tests/test_artifact_store_conformance.py | 19 ++++ backend/tests/test_config.py | 50 +++++++++++ backend/tests/test_s3_artifact_store.py | 23 +++++ docs/spec_artifact_storage_service.md | 11 ++- scripts/test_agent_gates.py | 2 + 27 files changed, 313 insertions(+), 79 deletions(-) create mode 100644 backend/app/core/s3_validation.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index 6938958d..60db8577 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -98,8 +98,9 @@ dependency and credential-behavior review. MinIO static credentials are local/CI only. The endpoint is omitted for native AWS S3 and explicit for MinIO. AWS requires an explicit region and production requires HTTPS, a non-local resolved endpoint, and backend `s3_compatible`. Secrets and resolved -credentials are never persisted or retained by errors. Cloudflare R2 has no -v0.1 runtime profile, credential service, or configuration path. +credentials are never persisted or retained by errors. Object and credential +metadata transports explicitly ignore ambient HTTP proxy variables. Cloudflare +R2 has no v0.1 runtime profile, credential service, or configuration path. ## Immutable Object Identity diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index ce3ee83a..3f16cc93 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -14,7 +14,8 @@ credential-delivery contract. ## Allowed Files - `backend/app/adapters/artifacts/s3_compatible.py`, the shared canonical - provider-reference helper, and adapter registration; + provider-reference and S3 namespace-validation helpers, and adapter + registration; - `backend/app/interfaces/artifacts.py` only to register the closed `minio-v1` and `aws-s3-v1` namespace profiles, their exact descriptor keys, and the stable AWS live-proof-required error; @@ -60,6 +61,9 @@ credential-delivery contract. - each selected workload method accepts only its exact closed `AWS_*` environment allowlist; every other `AWS_*` or `BOTOCORE_*` SDK behavior control fails before SDK session construction; +- S3 object operations and credential metadata operations ignore ambient HTTP + proxy variables so signed requests, credentials, and artifact bytes cannot + be redirected outside the configured provider transport; - container role accepts only the relative metadata path, IAM role rejects a custom metadata endpoint and requires IMDSv2, and the isolated IAM fetcher disables IMDSv1; @@ -72,11 +76,12 @@ credential-delivery contract. - the application factory can instantiate only LocalStorage and MinIO in this chunk. Valid AWS configuration remains runtime-ineligible with the stable `artifact_provider_live_proof_required` startup failure. The composition root - raises that typed error after settings validation but before factory - construction, namespace claim, credential resolver construction, credential - loading, or provider I/O. Tests prove no namespace row is persisted and no - credential source is touched. Only Chunk 07 may add the immutable activation - record and production composition guard; + and Celery scheduler/task entry points raise that typed error after settings + validation but before factory construction, scratch cleanup, namespace claim, + credential resolver construction, credential loading, or provider I/O. Tests + prove no namespace row is persisted and no credential source is touched. Only + Chunk 07 may add the immutable activation record and production composition + guard; - startup and operation tests reject configured adapter/profile/namespace mismatch against persisted deployment or replica identity; no request is routed to a second namespace; @@ -122,6 +127,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 ``` ## Verification @@ -129,7 +135,7 @@ coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision= ```bash docker compose up -d --wait postgres redis minio (cd backend && .venv/bin/ruff check app tests) -(cd backend && .venv/bin/pytest tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_preparation.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) +(cd backend && .venv/bin/pytest tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_preparation.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov=app.core.s3_validation --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) python3 scripts/check_stale_artifact_contracts.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md index 72f71b00..a60655b8 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md @@ -72,6 +72,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md index af7abb2e..9d434824 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md @@ -87,6 +87,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md index 8ffa49f8..6229a0f9 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md @@ -64,6 +64,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md index b2f7566d..e58f45d7 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md @@ -130,6 +130,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md index 437771a4..93000b35 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md @@ -110,6 +110,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md index 5f2f8f18..e07b2d90 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md @@ -90,6 +90,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md index 9a66a981..3da38010 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md @@ -91,6 +91,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md index c3840e2f..c66fc262 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md @@ -96,6 +96,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md index 5a94e64e..85147cf3 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md @@ -77,6 +77,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md index 5c43d4e7..e4139c55 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md @@ -85,6 +85,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md index 7a8959e8..adeb5ec1 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md @@ -127,6 +127,7 @@ coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index 2c797a38..a9a7b490 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -134,6 +134,14 @@ jobs: --precision=2 --fail-under=90 + - name: S3 namespace validation coverage + working-directory: backend + run: >- + coverage report + --include='app/core/s3_validation.py' + --precision=2 + --fail-under=90 + - name: Actor subsystem coverage working-directory: backend run: >- diff --git a/backend/app/adapters/artifacts/__init__.py b/backend/app/adapters/artifacts/__init__.py index d5b461f7..9baf5ef4 100644 --- a/backend/app/adapters/artifacts/__init__.py +++ b/backend/app/adapters/artifacts/__init__.py @@ -28,13 +28,7 @@ def create_artifact_store_bootstrap(settings: Settings) -> ArtifactStoreBootstra Raises: ExternalServiceConfigurationError: If the provider is not registered. """ - if ( - settings.artifact_store_backend == "s3_compatible" - and settings.artifact_s3_provider_profile == "aws_s3" - ): - raise ArtifactProviderLiveProofRequiredError( - "AWS artifact provider requires live deployment proof" - ) + require_artifact_runtime_eligible(settings) from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap from app.adapters.artifacts.s3_compatible import ( @@ -65,6 +59,17 @@ def create_local_store() -> ArtifactStoreBootstrap: return factory.create(settings.artifact_store_backend) +def require_artifact_runtime_eligible(settings: Settings) -> None: + """Reject configured providers that this chunk has not activated.""" + if ( + settings.artifact_store_backend == "s3_compatible" + and settings.artifact_s3_provider_profile == "aws_s3" + ): + raise ArtifactProviderLiveProofRequiredError( + "AWS artifact provider requires live deployment proof" + ) + + def artifact_preparation_limits(settings: Settings) -> ArtifactPreparationLimits: """Map validated settings to the one process-independent scratch contract.""" return ArtifactPreparationLimits( @@ -92,6 +97,7 @@ def create_artifact_scratch_manager(settings: Settings) -> ArtifactScratchManage async def cleanup_stale_artifact_scratch(settings: Settings) -> int: """Run one database-independent stale cleanup with shared construction.""" + require_artifact_runtime_eligible(settings) manager = create_artifact_scratch_manager(settings) try: return await manager.cleanup_stale() diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index 465a0483..b0abf75b 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -33,6 +33,12 @@ ) from app.core.config import Settings from app.core.hashing import canonical_json_hash +from app.core.s3_validation import ( + canonical_minio_endpoint, + is_canonical_s3_bucket, + is_canonical_s3_prefix, + is_canonical_s3_region, +) from app.interfaces.artifacts import ( ARTIFACT_STORE_CAPABILITY_KEY, ArtifactByteRange, @@ -182,6 +188,21 @@ def __init__( """Pin one validated namespace and one isolated SDK session.""" if provider_profile not in {"minio", "aws_s3"}: raise ArtifactConfigurationError("S3 provider profile is invalid") + if not is_canonical_s3_region(region): + raise ArtifactConfigurationError("S3 region is invalid") + if not is_canonical_s3_bucket(bucket): + raise ArtifactConfigurationError("S3 bucket is invalid") + if not is_canonical_s3_prefix(private_prefix): + raise ArtifactConfigurationError("S3 private prefix is invalid") + if addressing_style not in {"path", "virtual"}: + raise ArtifactConfigurationError("S3 addressing style is invalid") + if provider_profile == "minio": + try: + endpoint_url = canonical_minio_endpoint(endpoint_url) + except ValueError as error: + raise ArtifactConfigurationError(str(error)) from None + elif endpoint_url is not None: + raise ArtifactConfigurationError("native AWS S3 endpoint is invalid") if not isinstance(session, AioSession): raise ArtifactConfigurationError("S3 credential session is invalid") if type(buffer_bytes) is not int or not 1 <= buffer_bytes <= 1024 * 1024: @@ -213,6 +234,7 @@ def __init__( connect_timeout=float(connect_timeout_seconds), read_timeout=float(read_timeout_seconds), max_pool_connections=max_pool_connections, + proxies={}, retries={"max_attempts": 1, "mode": "standard"}, request_checksum_calculation="when_required", response_checksum_validation="when_required", @@ -741,6 +763,7 @@ def create_isolated_aws_workload_identity_session( profile["role_session_name"] = role_session_name def create_isolated_sts_client(service_name: str, **_kwargs: object) -> object: + """Create only the pinned no-proxy STS client for web identity.""" if service_name != "sts": raise ArtifactConfigurationError("AWS web identity service is invalid") return session.create_client( diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 09cadbb5..fca8e21e 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -7,12 +7,10 @@ from contextvars import ContextVar import json import os -import re from collections.abc import Mapping from functools import lru_cache from pathlib import Path from typing import Literal, Self -from urllib.parse import urlsplit from dotenv import dotenv_values from pydantic import Field, PrivateAttr, SecretStr, model_validator @@ -23,6 +21,13 @@ SettingsConfigDict, ) +from app.core.s3_validation import ( + canonical_minio_endpoint, + is_canonical_s3_bucket, + is_canonical_s3_prefix, + is_canonical_s3_region, +) + _ARTIFACT_S3_SECRET_FIELDS = frozenset( { @@ -31,9 +36,6 @@ "artifact_s3_session_token", } ) -_S3_REGION = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$") -_S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") -_S3_PREFIX_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") _ALTERNATE_VALIDATION_RESTORES_SECRETS: ContextVar[bool] = ContextVar( "alternate_validation_restores_secrets", default=False, @@ -396,11 +398,11 @@ def _validate_s3_compatible_storage(self) -> None: """Require one closed MinIO or native AWS configuration profile.""" if self.artifact_s3_provider_profile is None: raise ValueError("S3-compatible artifact storage requires a provider profile") - if not _is_canonical_s3_region(self.artifact_s3_region): + if not is_canonical_s3_region(self.artifact_s3_region): raise ValueError("S3-compatible artifact storage requires a canonical region") - if not _is_canonical_s3_bucket(self.artifact_s3_bucket): + if not is_canonical_s3_bucket(self.artifact_s3_bucket): raise ValueError("S3-compatible artifact storage requires a canonical bucket") - if not _is_canonical_s3_prefix(self.artifact_s3_private_prefix): + if not is_canonical_s3_prefix(self.artifact_s3_private_prefix): raise ValueError("S3-compatible artifact storage private prefix is invalid") if self.artifact_maximum_bytes > 512 * 1024 * 1024: raise ValueError("S3-compatible artifact maximum exceeds 512 MiB") @@ -417,7 +419,7 @@ def _validate_minio_storage(self) -> None: raise ValueError("MinIO artifact storage requires local static credentials") if self.artifact_s3_aws_workload_identity_method is not None: raise ValueError("MinIO artifact storage cannot select AWS workload identity") - self.artifact_s3_endpoint_url = _canonical_minio_endpoint( + self.artifact_s3_endpoint_url = canonical_minio_endpoint( self.artifact_s3_endpoint_url ) @@ -522,55 +524,6 @@ def _extract_optional_secret( return secret -def _is_canonical_s3_region(value: str | None) -> bool: - """Return whether one region is bounded and canonical.""" - return isinstance(value, str) and _S3_REGION.fullmatch(value) is not None - - -def _is_canonical_s3_bucket(value: str | None) -> bool: - """Return whether one dedicated bucket name is DNS-compatible.""" - if not isinstance(value, str) or _S3_BUCKET.fullmatch(value) is None: - return False - return ".." not in value and not re.fullmatch(r"\d+\.\d+\.\d+\.\d+", value) - - -def _is_canonical_s3_prefix(value: str) -> bool: - """Return whether one private object prefix has canonical path segments.""" - if not isinstance(value, str) or not value or len(value) > 512: - return False - segments = value.split("/") - return all(_S3_PREFIX_SEGMENT.fullmatch(segment) is not None for segment in segments) - - -def _canonical_minio_endpoint(value: str | None) -> str: - """Return one normalized noncredentialed HTTP(S) MinIO origin.""" - if not isinstance(value, str) or not value or len(value) > 2048: - raise ValueError("MinIO artifact storage requires an endpoint") - parsed = urlsplit(value) - try: - port = parsed.port - except ValueError as error: - raise ValueError("MinIO artifact storage endpoint is invalid") from error - if ( - parsed.scheme.lower() not in {"http", "https"} - or not parsed.hostname - or parsed.username is not None - or parsed.password is not None - or parsed.query - or parsed.fragment - or parsed.path not in {"", "/"} - or parsed.hostname.endswith(".") - ): - raise ValueError("MinIO artifact storage endpoint is invalid") - scheme = parsed.scheme.lower() - hostname = parsed.hostname.lower() - if ":" in hostname: - hostname = f"[{hostname}]" - if port is not None and port != {"http": 80, "https": 443}[scheme]: - hostname = f"{hostname}:{port}" - return f"{scheme}://{hostname}" - - def decode_api_rate_limit_key_secret(value: SecretStr) -> bytes: """Decode one canonical padded Base64 rate-control key.""" secret = value.get_secret_value() diff --git a/backend/app/core/s3_validation.py b/backend/app/core/s3_validation.py new file mode 100644 index 00000000..1b57d4a0 --- /dev/null +++ b/backend/app/core/s3_validation.py @@ -0,0 +1,88 @@ +"""Canonical non-secret S3 namespace validation shared by configuration layers.""" + +from __future__ import annotations + +from collections.abc import Mapping +import re +from urllib.parse import urlsplit + + +_S3_REGION = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$") +_S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") +_S3_PREFIX_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") +_SHA256 = re.compile(r"^sha256:[0-9a-f]{64}$") + + +def is_canonical_s3_region(value: object) -> bool: + """Return whether one region is bounded and canonical.""" + return isinstance(value, str) and _S3_REGION.fullmatch(value) is not None + + +def is_canonical_s3_bucket(value: object) -> bool: + """Return whether one dedicated bucket name is DNS-compatible.""" + if not isinstance(value, str) or _S3_BUCKET.fullmatch(value) is None: + return False + return ".." not in value and re.fullmatch(r"\d+\.\d+\.\d+\.\d+", value) is None + + +def is_canonical_s3_prefix(value: object) -> bool: + """Return whether one private object prefix has canonical path segments.""" + if not isinstance(value, str) or not value or len(value) > 512: + return False + return all( + _S3_PREFIX_SEGMENT.fullmatch(segment) is not None + for segment in value.split("/") + ) + + +def canonical_minio_endpoint(value: object) -> str: + """Return one normalized noncredentialed HTTP(S) MinIO origin.""" + if not isinstance(value, str) or not value or len(value) > 2048: + raise ValueError("MinIO artifact storage requires an endpoint") + parsed = urlsplit(value) + try: + port = parsed.port + except ValueError as error: + raise ValueError("MinIO artifact storage endpoint is invalid") from error + if ( + parsed.scheme.lower() not in {"http", "https"} + or not parsed.hostname + or parsed.username is not None + or parsed.password is not None + or parsed.query + or parsed.fragment + or parsed.path not in {"", "/"} + or parsed.hostname.endswith(".") + ): + raise ValueError("MinIO artifact storage endpoint is invalid") + scheme = parsed.scheme.lower() + hostname = parsed.hostname.lower() + if ":" in hostname: + hostname = f"[{hostname}]" + if port is not None and port != {"http": 80, "https": 443}[scheme]: + hostname = f"{hostname}:{port}" + return f"{scheme}://{hostname}" + + +def validate_s3_namespace_descriptor( + provider_profile: str, + descriptor: Mapping[str, str], +) -> None: + """Reject malformed values in one closed S3 namespace descriptor.""" + if descriptor.get("addressing_style") not in {"path", "virtual"}: + raise ValueError("artifact S3 addressing style is invalid") + if not is_canonical_s3_bucket(descriptor.get("bucket")): + raise ValueError("artifact S3 bucket is invalid") + if not is_canonical_s3_prefix(descriptor.get("private_prefix")): + raise ValueError("artifact S3 private prefix is invalid") + if not is_canonical_s3_region(descriptor.get("region")): + raise ValueError("artifact S3 region is invalid") + endpoint_identity = descriptor.get("endpoint_identity") + if provider_profile == "minio-v1": + if not isinstance(endpoint_identity, str) or _SHA256.fullmatch(endpoint_identity) is None: + raise ValueError("artifact MinIO endpoint identity is invalid") + elif provider_profile == "aws-s3-v1": + if endpoint_identity is not None: + raise ValueError("artifact AWS namespace endpoint identity is forbidden") + else: + raise ValueError("artifact S3 provider profile is invalid") diff --git a/backend/app/interfaces/artifacts.py b/backend/app/interfaces/artifacts.py index 3b8189a7..2dd8e11f 100644 --- a/backend/app/interfaces/artifacts.py +++ b/backend/app/interfaces/artifacts.py @@ -7,6 +7,7 @@ from typing import Protocol from app.core.hashing import canonical_json_hash +from app.core.s3_validation import validate_s3_namespace_descriptor from app.interfaces.external_services import ( ExternalServiceAdapter, ExternalServiceAdapterIdentity, @@ -150,6 +151,11 @@ def __post_init__(self) -> None: or set(keys) != expected_keys ): raise ValueError("artifact namespace descriptor is not canonical") + if self.provider_profile in {"minio-v1", "aws-s3-v1"}: + validate_s3_namespace_descriptor( + self.provider_profile, + dict(self.descriptor_items), + ) def as_dict(self) -> dict[str, str]: """Return a fresh JSON-compatible descriptor projection.""" diff --git a/backend/app/workers/artifacts.py b/backend/app/workers/artifacts.py index 0e376420..88373f2b 100644 --- a/backend/app/workers/artifacts.py +++ b/backend/app/workers/artifacts.py @@ -2,7 +2,10 @@ from __future__ import annotations -from app.adapters.artifacts import cleanup_stale_artifact_scratch +from app.adapters.artifacts import ( + cleanup_stale_artifact_scratch, + require_artifact_runtime_eligible, +) from app.core.config import get_settings from app.workers.async_runner import run_async_task from app.workers.celery_app import ARTIFACT_SCRATCH_CLEANUP_TASK, celery_app @@ -14,4 +17,5 @@ def cleanup_stale_scratch() -> int: settings = get_settings() if settings.artifact_store_backend == "disabled": return 0 + require_artifact_runtime_eligible(settings) return run_async_task(lambda: cleanup_stale_artifact_scratch(settings)) diff --git a/backend/app/workers/celery_app.py b/backend/app/workers/celery_app.py index d1f9b431..0518f6e8 100644 --- a/backend/app/workers/celery_app.py +++ b/backend/app/workers/celery_app.py @@ -4,6 +4,7 @@ from celery import Celery +from app.adapters.artifacts import require_artifact_runtime_eligible from app.core.config import get_settings from app.workers.errors import CeleryConfigurationError @@ -18,6 +19,7 @@ def create_celery_app() -> Celery: Configured Celery application for durable background jobs. """ settings = get_settings() + require_artifact_runtime_eligible(settings) broker_url = settings.celery_broker_url if broker_url is None: if settings.celery_task_always_eager: diff --git a/backend/tests/test_artifact_cleanup_wiring.py b/backend/tests/test_artifact_cleanup_wiring.py index 1557d1a4..1dd1c501 100644 --- a/backend/tests/test_artifact_cleanup_wiring.py +++ b/backend/tests/test_artifact_cleanup_wiring.py @@ -341,3 +341,32 @@ async def fail_cleanup(_candidate: Settings) -> int: monkeypatch.setattr(artifacts_module, "cleanup_stale_artifact_scratch", fail_cleanup) with pytest.raises(RuntimeError, match="scratch cleanup failed"): artifacts_module.cleanup_stale_scratch.run() + + +def test_aws_s3_is_runtime_ineligible_for_celery_and_cleanup_task( + monkeypatch: pytest.MonkeyPatch, + request: pytest.FixtureRequest, + tmp_path: Path, +) -> None: + """Apply the same inactive-provider guard to API and worker entry points.""" + celery_module, artifacts_module = _load_worker_modules(monkeypatch, request) + settings = Settings( + environment="production", + celery_task_always_eager=True, + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method="container-role", + ) + monkeypatch.setattr(celery_module, "get_settings", lambda: settings) + with pytest.raises(ArtifactProviderLiveProofRequiredError) as startup_error: + celery_module.create_celery_app() + assert startup_error.value.code == "artifact_provider_live_proof_required" + + monkeypatch.setattr(artifacts_module, "get_settings", lambda: settings) + with pytest.raises(ArtifactProviderLiveProofRequiredError) as task_error: + artifacts_module.cleanup_stale_scratch.run() + assert task_error.value.code == "artifact_provider_live_proof_required" diff --git a/backend/tests/test_artifact_store_conformance.py b/backend/tests/test_artifact_store_conformance.py index d1df90af..a753989c 100644 --- a/backend/tests/test_artifact_store_conformance.py +++ b/backend/tests/test_artifact_store_conformance.py @@ -323,6 +323,25 @@ def test_namespace_startup_values_are_closed_and_immutable() -> None: ("region", "us-east-1"), ), ), + ( + "minio-v1", + ( + ("addressing_style", "path"), + ("bucket", "Bad..Bucket"), + ("endpoint_identity", "sha256:" + "1" * 64), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ( + "aws-s3-v1", + ( + ("addressing_style", "virtual"), + ("bucket", "workstream-artifacts"), + ("private_prefix", "../bad//prefix"), + ("region", "us-east-1"), + ), + ), ): with pytest.raises(ValueError): ArtifactStoreNamespaceIdentity(profile, items) diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 224f194d..78326cc4 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -17,6 +17,11 @@ from app.api.deps.auth import get_application_auth_verifier from app.core.auth import clear_auth_verifier_cache, get_auth_verifier from app.core.config import Settings, get_settings +from app.core.s3_validation import ( + canonical_minio_endpoint, + is_canonical_s3_prefix, + validate_s3_namespace_descriptor, +) from app.interfaces.artifacts import ArtifactProviderLiveProofRequiredError from app.interfaces.external_services import ( ExternalServiceConfigurationError, @@ -893,3 +898,48 @@ def test_s3_dependency_manifest_and_installed_versions_are_exact() -> None: assert "botocore==1.43.0" in manifest["project"]["dependencies"] assert importlib.metadata.version("aiobotocore") == "3.7.0" assert importlib.metadata.version("botocore") == "1.43.0" + + +@pytest.mark.parametrize("value", [None, "", "x" * 513]) +def test_s3_prefix_validation_rejects_noncanonical_outer_bounds(value: object) -> None: + assert is_canonical_s3_prefix(value) is False + + +def test_minio_endpoint_canonicalization_handles_ipv6_and_invalid_ports() -> None: + assert canonical_minio_endpoint("HTTP://[::1]:9000/") == "http://[::1]:9000" + with pytest.raises(ValueError, match="endpoint is invalid"): + canonical_minio_endpoint("http://localhost:not-a-port") + + +@pytest.mark.parametrize( + ("profile", "changes", "message"), + [ + ("minio-v1", {"addressing_style": "auto"}, "addressing style"), + ("minio-v1", {"region": "US-east-1"}, "region"), + ("minio-v1", {"endpoint_identity": "sha256:bad"}, "endpoint identity"), + ( + "aws-s3-v1", + {"endpoint_identity": "sha256:" + "1" * 64}, + "endpoint identity is forbidden", + ), + ("unsupported-v1", {}, "provider profile"), + ], +) +def test_s3_namespace_value_validation_fails_closed( + profile: str, + changes: dict[str, str], + message: str, +) -> None: + descriptor = { + "addressing_style": "path", + "bucket": "workstream-artifacts", + "endpoint_identity": "sha256:" + "1" * 64, + "private_prefix": "workstream/artifacts", + "region": "us-east-1", + } + if profile == "aws-s3-v1": + descriptor.pop("endpoint_identity") + descriptor.update(changes) + + with pytest.raises(ValueError, match=message): + validate_s3_namespace_descriptor(profile, descriptor) diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 703b0fb8..4bf4faaf 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -291,6 +291,11 @@ def test_s3_constructor_and_bootstrap_reject_invalid_runtime_values() -> None: } for override in ( {"provider_profile": "unknown"}, + {"region": "US-east-1"}, + {"bucket": "Bad..Bucket"}, + {"private_prefix": "../bad//prefix"}, + {"addressing_style": "automatic"}, + {"endpoint_url": "http://user:secret@localhost:9000"}, {"session": object()}, {"buffer_bytes": 0}, {"buffer_bytes": True}, @@ -305,6 +310,24 @@ def test_s3_constructor_and_bootstrap_reject_invalid_runtime_values() -> None: S3CompatibleArtifactStoreBootstrap(object()) # type: ignore[arg-type] +async def test_s3_operation_client_ignores_ambient_proxy_environment( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Keep signed object requests and bytes off ambient process proxies.""" + monkeypatch.setenv("HTTP_PROXY", "http://127.0.0.1:9") + monkeypatch.setenv("HTTPS_PROXY", "http://127.0.0.1:9") + monkeypatch.setenv("ALL_PROXY", "http://127.0.0.1:9") + store = initialize_minio_store(private_prefix="negative/proxy-isolation") + try: + async with store._client() as client: + proxy = client._endpoint.http_session._proxy_config.proxy_url_for( + MINIO_ENDPOINT + ) + assert proxy is None + finally: + store.close() + + def test_namespace_claim_is_single_use_and_close_disables_operations() -> None: """One exact claim activates one adapter once and close revokes local use.""" bootstrap = create_artifact_store_bootstrap( diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 3007cd74..bdbcb6d9 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -813,10 +813,13 @@ static credentials, configured endpoints, unselected or ambient credential sources, arbitrary container credential URLs, and custom instance-metadata endpoints. The selected method accepts only its exact closed `AWS_*` environment allowlist; every other `AWS_*` or `BOTOCORE_*` SDK behavior control -fails before SDK session construction. IAM-role validation requires IMDSv2 and -the isolated fetcher disables IMDSv1. Even valid native AWS settings fail with -`artifact_provider_live_proof_required` before credential-source probing, -factory construction, namespace claim, resolver loading, or provider I/O. +fails before SDK session construction. S3 object operations and credential +metadata operations explicitly ignore ambient HTTP proxy variables. IAM-role +validation requires IMDSv2 and the isolated fetcher disables IMDSv1. Even valid +native AWS settings fail with `artifact_provider_live_proof_required` at API, +Celery scheduler/task, and direct-cleanup entry points before credential-source +probing, factory construction, scratch cleanup, namespace claim, resolver +loading, or provider I/O. Every process sharing a scratch root must use the identical complete setting set. The root marker binds a canonical fingerprint of those limits and startup diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index abf0df23..bc96515f 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -65,6 +65,8 @@ "02B1": ( "coverage report --include='app/adapters/artifacts/s3_compatible.py' " "--precision=2 --fail-under=90", + "coverage report --include='app/core/s3_validation.py' " + "--precision=2 --fail-under=90", ), "02C1": ( "coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90", From 5f3ae9ccdee38d57a860992d70d19eedd30579d1 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 12:09:38 +0100 Subject: [PATCH 10/39] Strengthen S3 isolation proof --- backend/tests/assertion_helpers.py | 38 +++++++++++++++++ backend/tests/test_config.py | 56 +++++++------------------ backend/tests/test_s3_artifact_store.py | 44 +++++++++++++++---- 3 files changed, 90 insertions(+), 48 deletions(-) create mode 100644 backend/tests/assertion_helpers.py diff --git a/backend/tests/assertion_helpers.py b/backend/tests/assertion_helpers.py new file mode 100644 index 00000000..46e01f4c --- /dev/null +++ b/backend/tests/assertion_helpers.py @@ -0,0 +1,38 @@ +"""Shared deep assertions for security-sensitive test object graphs.""" + +from __future__ import annotations + +from collections.abc import Mapping + +from pydantic import SecretStr, ValidationError + + +def assert_secret_not_retained( + value: object, + secret: str, + seen: set[int] | None = None, +) -> None: + """Assert a secret is unreachable through an error's public object graph.""" + if seen is None: + seen = set() + if id(value) in seen: + return + seen.add(id(value)) + if isinstance(value, str): + assert secret not in value + elif isinstance(value, SecretStr): + assert value.get_secret_value() != secret + elif isinstance(value, BaseException): + if isinstance(value, ValidationError): + assert_secret_not_retained(value.errors(), secret, seen) + assert_secret_not_retained(value.args, secret, seen) + assert_secret_not_retained(vars(value), secret, seen) + assert_secret_not_retained(value.__cause__, secret, seen) + assert_secret_not_retained(value.__context__, secret, seen) + elif isinstance(value, Mapping): + for key, item in value.items(): + assert_secret_not_retained(key, secret, seen) + assert_secret_not_retained(item, secret, seen) + elif isinstance(value, (list, tuple, set)): + for item in value: + assert_secret_not_retained(item, secret, seen) diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 78326cc4..f71ddfda 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -9,7 +9,7 @@ from typing import get_args import pytest -from pydantic import SecretStr, ValidationError +from pydantic import ValidationError from pydantic_settings import BaseSettings from app.adapters.artifacts import create_artifact_store_bootstrap @@ -28,33 +28,7 @@ UnknownExternalServiceProviderError, ) from app.main import create_app - - -def _assert_secret_not_retained(value: object, secret: str, seen: set[int] | None = None) -> None: - """Assert a secret is unreachable through an error's public object graph.""" - if seen is None: - seen = set() - if id(value) in seen: - return - seen.add(id(value)) - if isinstance(value, str): - assert secret not in value - elif isinstance(value, SecretStr): - assert value.get_secret_value() != secret - elif isinstance(value, BaseException): - if isinstance(value, ValidationError): - _assert_secret_not_retained(value.errors(), secret, seen) - _assert_secret_not_retained(value.args, secret, seen) - _assert_secret_not_retained(vars(value), secret, seen) - _assert_secret_not_retained(value.__cause__, secret, seen) - _assert_secret_not_retained(value.__context__, secret, seen) - elif isinstance(value, Mapping): - for key, item in value.items(): - _assert_secret_not_retained(key, secret, seen) - _assert_secret_not_retained(item, secret, seen) - elif isinstance(value, (list, tuple, set)): - for item in value: - _assert_secret_not_retained(item, secret, seen) +from tests.assertion_helpers import assert_secret_not_retained def test_default_settings_are_fail_closed(monkeypatch: pytest.MonkeyPatch) -> None: @@ -140,7 +114,7 @@ def test_rate_limit_secret_is_absent_from_unrelated_structured_errors() -> None: assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained(caught.value, encoded) def test_environment_rate_limit_secret_is_absent_from_unrelated_structured_errors( @@ -154,7 +128,7 @@ def test_environment_rate_limit_secret_is_absent_from_unrelated_structured_error assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained(caught.value, encoded) def test_dotenv_rate_limit_secret_is_absent_from_unrelated_structured_errors( @@ -172,7 +146,7 @@ def test_dotenv_rate_limit_secret_is_absent_from_unrelated_structured_errors( assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained(caught.value, encoded) def test_model_validate_rejects_rate_limit_secret_without_structured_echo() -> None: @@ -183,7 +157,7 @@ def test_model_validate_rejects_rate_limit_secret_without_structured_echo() -> N assert not isinstance(caught.value, ValidationError) assert invalid not in f"{caught.value!s} {caught.value!r}" - _assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained(caught.value, invalid) def test_model_validate_rate_limit_secret_is_absent_from_unrelated_errors() -> None: @@ -199,7 +173,7 @@ def test_model_validate_rate_limit_secret_is_absent_from_unrelated_errors() -> N assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained(caught.value, encoded) @pytest.mark.parametrize( @@ -238,7 +212,7 @@ def capture_input( assert isinstance(observed["input"], Mapping) assert observed["input"]["api_rate_limit_key_secret"] is None - _assert_secret_not_retained(observed, encoded) + assert_secret_not_retained(observed, encoded) @pytest.mark.parametrize("method_name", ["model_validate_json", "model_validate_strings"]) @@ -266,7 +240,7 @@ def test_alternate_validation_rejects_rate_limit_secret_without_echo( assert not isinstance(caught.value, ValidationError) assert invalid not in f"{caught.value!s} {caught.value!r}" - _assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained(caught.value, invalid) @pytest.mark.parametrize("method_name", ["model_validate_json", "model_validate_strings"]) @@ -285,7 +259,7 @@ def test_alternate_validation_secret_is_absent_from_unrelated_errors( assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained(caught.value, encoded) def test_model_validate_json_rejects_malformed_document_without_echo() -> None: @@ -299,7 +273,7 @@ def test_model_validate_json_rejects_malformed_document_without_echo() -> None: assert invalid not in f"{caught.value!s} {caught.value!r}" assert caught.value.__cause__ is None assert caught.value.__context__ is None - _assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained(caught.value, invalid) @pytest.mark.parametrize( @@ -323,7 +297,7 @@ def test_rate_limit_secret_rejects_invalid_values_without_echo(value: str) -> No assert not hasattr(caught.value, "json") if value.strip(): assert value not in rendered - _assert_secret_not_retained(caught.value, value) + assert_secret_not_retained(caught.value, value) @pytest.mark.parametrize( @@ -689,7 +663,7 @@ def test_minio_secret_values_are_absent_from_repr_and_validation_errors( assert secret not in repr(caught.value.errors()) assert secret not in caught.value.json() - _assert_secret_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) @pytest.mark.parametrize( @@ -748,7 +722,7 @@ def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( _env_file=env_file, ) assert env_secret not in repr(env_error.value.errors()) - _assert_secret_not_retained(env_error.value, env_secret) + assert_secret_not_retained(env_error.value, env_secret) monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID") monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY") @@ -765,7 +739,7 @@ def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( _env_file=env_file, ) assert dotenv_secret not in repr(dotenv_error.value.errors()) - _assert_secret_not_retained(dotenv_error.value, dotenv_secret) + assert_secret_not_retained(dotenv_error.value, dotenv_secret) def test_minio_endpoint_is_normalized_before_namespace_identity(tmp_path: Path) -> None: diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 4bf4faaf..731e03b8 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -33,6 +33,7 @@ artifact_store_namespace_material, ) from app.interfaces.external_services import ExternalServiceAdapterIdentity +from tests.assertion_helpers import assert_secret_not_retained from tests.artifact_store_helpers import minted_source from tests.test_artifact_store_conformance import ArtifactStoreConformanceTests @@ -129,10 +130,12 @@ def initialize_minio_store(*, private_prefix: str) -> S3CompatibleArtifactStore: @pytest.mark.asyncio async def test_wrong_minio_credentials_map_403_to_unavailable() -> None: """Never misclassify forbidden provider access as authoritative absence.""" + access_key = "unknown-minio-user" + secret_key = "unknown-minio-password" settings = minio_settings( private_prefix="negative/forbidden", - access_key="unknown-minio-user", - secret_key="unknown-minio-password", + access_key=access_key, + secret_key=secret_key, ) bootstrap = create_artifact_store_bootstrap(settings) namespace = bootstrap.namespace_identity @@ -145,8 +148,13 @@ async def test_wrong_minio_credentials_map_403_to_unavailable() -> None: ArtifactStoreNamespaceClaim(bootstrap.identity, namespace, fingerprint) ) try: - with pytest.raises(ArtifactStoreUnavailableError): + with pytest.raises(ArtifactStoreUnavailableError) as caught: await store.head("sha256/00/" + "0" * 62) + assert str(caught.value) == "S3 artifact head failed" + assert caught.value.code == "artifact_storage_unavailable" + assert caught.value.__cause__ is None + assert_secret_not_retained(caught.value, access_key) + assert_secret_not_retained(caught.value, secret_key) finally: bootstrap.close() @@ -312,18 +320,40 @@ def test_s3_constructor_and_bootstrap_reject_invalid_runtime_values() -> None: async def test_s3_operation_client_ignores_ambient_proxy_environment( monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, ) -> None: """Keep signed object requests and bytes off ambient process proxies.""" - monkeypatch.setenv("HTTP_PROXY", "http://127.0.0.1:9") - monkeypatch.setenv("HTTPS_PROXY", "http://127.0.0.1:9") - monkeypatch.setenv("ALL_PROXY", "http://127.0.0.1:9") - store = initialize_minio_store(private_prefix="negative/proxy-isolation") + for name in ( + "HTTP_PROXY", + "HTTPS_PROXY", + "ALL_PROXY", + "http_proxy", + "https_proxy", + "all_proxy", + ): + monkeypatch.setenv(name, "http://127.0.0.1:9") + monkeypatch.delenv("NO_PROXY", raising=False) + monkeypatch.delenv("no_proxy", raising=False) + namespace = hashlib.sha256(os.fspath(tmp_path).encode()).hexdigest()[:16] + store = initialize_minio_store( + private_prefix=f"negative/proxy-isolation/{namespace}" + ) try: async with store._client() as client: proxy = client._endpoint.http_session._proxy_config.proxy_url_for( MINIO_ENDPOINT ) assert proxy is None + payload = b"real object I/O bypasses poisoned ambient proxies" + async with minted_source(tmp_path / "scratch", payload) as source: + result = await store.put(source) + observed = await store.head(result.provider_object_ref) + received = b"".join( + [chunk async for chunk in store.open(result.provider_object_ref)] + ) + assert observed.exists is True + assert observed.byte_count == len(payload) + assert received == payload finally: store.close() From fd31840c892c1f48e55e734f315db78d4db2e5e8 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 12:32:40 +0100 Subject: [PATCH 11/39] Harden artifact review evidence assertions --- ...WS-ART-001-02B1-s3-compatible-minio-aws.md | 3 ++ backend/tests/assertion_helpers.py | 2 +- backend/tests/test_assertion_helpers.py | 12 ++++++ backend/tests/test_audit.py | 37 +++++-------------- 4 files changed, 25 insertions(+), 29 deletions(-) create mode 100644 backend/tests/test_assertion_helpers.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index 3f16cc93..903dec1d 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -20,6 +20,9 @@ credential-delivery contract. `minio-v1` and `aws-s3-v1` namespace profiles, their exact descriptor keys, and the stable AWS live-proof-required error; - AWS/MinIO settings in `backend/app/core/config.py`; +- `backend/app/workers/celery_app.py` and `backend/app/workers/artifacts.py` + only to enforce the shared AWS runtime-ineligible guard before Celery startup + or artifact scratch cleanup; - `backend/pyproject.toml` with exact `aiobotocore==3.7.0` and `botocore==1.43.0` pins; this repository has no backend dependency lockfile, so dependency tests must inspect the exact manifest pins and installed diff --git a/backend/tests/assertion_helpers.py b/backend/tests/assertion_helpers.py index 46e01f4c..253da4a9 100644 --- a/backend/tests/assertion_helpers.py +++ b/backend/tests/assertion_helpers.py @@ -21,7 +21,7 @@ def assert_secret_not_retained( if isinstance(value, str): assert secret not in value elif isinstance(value, SecretStr): - assert value.get_secret_value() != secret + assert secret not in value.get_secret_value() elif isinstance(value, BaseException): if isinstance(value, ValidationError): assert_secret_not_retained(value.errors(), secret, seen) diff --git a/backend/tests/test_assertion_helpers.py b/backend/tests/test_assertion_helpers.py new file mode 100644 index 00000000..77b0c107 --- /dev/null +++ b/backend/tests/test_assertion_helpers.py @@ -0,0 +1,12 @@ +"""Tests for shared security-sensitive assertion helpers.""" + +from pydantic import SecretStr +import pytest + +from tests.assertion_helpers import assert_secret_not_retained + + +def test_secret_str_rejects_embedded_forbidden_value() -> None: + """Reject a forbidden value even when wrapped by other secret text.""" + with pytest.raises(AssertionError): + assert_secret_not_retained(SecretStr("prefix-forbidden-suffix"), "forbidden") diff --git a/backend/tests/test_audit.py b/backend/tests/test_audit.py index c085d045..80d0707f 100644 --- a/backend/tests/test_audit.py +++ b/backend/tests/test_audit.py @@ -30,26 +30,7 @@ PermissionId, ) from app.modules.tasks.models import AuditEvent - - -def _assert_value_not_retained(value: object, forbidden: str, seen: set[int] | None = None) -> None: - """Traverse public exception state and prove rejected input is absent.""" - seen = seen or set() - if id(value) in seen: - return - seen.add(id(value)) - if isinstance(value, str): - assert forbidden not in value - elif isinstance(value, BaseException): - for item in (value.args, vars(value), value.__cause__, value.__context__): - _assert_value_not_retained(item, forbidden, seen) - elif isinstance(value, Mapping): - for key, item in value.items(): - _assert_value_not_retained(key, forbidden, seen) - _assert_value_not_retained(item, forbidden, seen) - elif isinstance(value, (list, tuple, set)): - for item in value: - _assert_value_not_retained(item, forbidden, seen) +from tests.assertion_helpers import assert_secret_not_retained @pytest.fixture @@ -542,13 +523,13 @@ def __len__(self): for mapping in mappings: with pytest.raises(TypeError, match="invalid authority audit input") as caught: constructor(mapping) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(json.dumps(payload)) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate(HostileMapping()) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) safe = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED).model_dump( mode="json" @@ -633,11 +614,11 @@ def __repr__(self): for document in (raw, raw.encode(), bytearray(raw.encode())): with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(document) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) for document in (HostileBytearray(b"{}"), LyingBytearray(duplicate.encode())): with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(document) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) for patch, forbidden in ( ( @@ -662,10 +643,10 @@ def __repr__(self): for constructor in constructors: with pytest.raises(TypeError, match="invalid authority audit input") as caught: constructor(candidate) - _assert_value_not_retained(caught.value, forbidden) + assert_secret_not_retained(caught.value, forbidden) with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(json.dumps(candidate)) - _assert_value_not_retained(caught.value, forbidden) + assert_secret_not_retained(caught.value, forbidden) with pytest.raises(ValidationError, match="resource ID requires"): _authority_input( AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED, @@ -894,7 +875,7 @@ def __repr__(self): with warnings.catch_warnings(record=True) as caught_warnings: with pytest.raises(TypeError, match="invalid authority audit input") as caught: await AuditService(session).add_authority_event(value) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) assert caught_warnings == [] assert await session.get(AuditEvent, str(value.event_id)) is None with warnings.catch_warnings(record=True) as caught_warnings: From ad5f1b10885958991ffe81411b5dcab445f77741 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 13:07:10 +0100 Subject: [PATCH 12/39] Remove rejected S3 secrets from tracebacks --- backend/app/core/config.py | 62 ++++++++++++++++++++---------- backend/tests/assertion_helpers.py | 54 +++++++++++++++++++++----- backend/tests/test_config.py | 31 +++++++++++++++ 3 files changed, 117 insertions(+), 30 deletions(-) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index fca8e21e..2e63f62a 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -208,6 +208,8 @@ def model_validate(cls, obj: object, **kwargs: object) -> Self: or _ARTIFACT_S3_SECRET_FIELDS.intersection(obj) ): sanitized = dict(obj) + supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) + obj = None secret = ( _extract_api_rate_limit_key_secret(sanitized) if "api_rate_limit_key_secret" in sanitized @@ -216,9 +218,8 @@ def model_validate(cls, obj: object, **kwargs: object) -> Self: s3_secrets = _extract_artifact_s3_static_secrets(sanitized) if secret is not None: sanitized["api_rate_limit_key_secret"] = None - for field_name in _ARTIFACT_S3_SECRET_FIELDS: - if field_name in obj: - sanitized[field_name] = None + for field_name in supplied_s3_fields: + sanitized[field_name] = None restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) try: settings = super().model_validate(sanitized, **kwargs) @@ -250,7 +251,12 @@ def model_validate_json( "api_rate_limit_key_secret" in parsed or _ARTIFACT_S3_SECRET_FIELDS.intersection(parsed) ): - return cls.model_validate(parsed, **kwargs) + try: + return cls.model_validate(parsed, **kwargs) + except ValueError: + parsed = None + json_data = b"" + raise return super().model_validate_json(json_data, **kwargs) @classmethod @@ -261,6 +267,8 @@ def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: or _ARTIFACT_S3_SECRET_FIELDS.intersection(obj) ): sanitized = dict(obj) + supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) + obj = None secret = ( _extract_api_rate_limit_key_secret(sanitized) if "api_rate_limit_key_secret" in sanitized @@ -269,9 +277,8 @@ def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: s3_secrets = _extract_artifact_s3_static_secrets(sanitized) if secret is not None: sanitized["api_rate_limit_key_secret"] = None - for field_name in _ARTIFACT_S3_SECRET_FIELDS: - if field_name in obj: - sanitized[field_name] = None + for field_name in supplied_s3_fields: + sanitized[field_name] = None restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) try: settings = super().model_validate_strings(sanitized, **kwargs) @@ -473,18 +480,27 @@ def _extract_artifact_s3_static_secrets( values: dict[str, object], ) -> tuple[SecretStr | None, SecretStr | None, SecretStr | None]: """Remove optional MinIO credentials from structured settings input.""" - return tuple( - _extract_optional_secret( - values, - field_name=field_name, - environment_name=f"WORKSTREAM_{field_name.upper()}", - ) - for field_name in ( - "artifact_s3_access_key_id", - "artifact_s3_secret_access_key", - "artifact_s3_session_token", - ) - ) # type: ignore[return-value] + field_names = ( + "artifact_s3_access_key_id", + "artifact_s3_secret_access_key", + "artifact_s3_session_token", + ) + secrets: list[SecretStr | None] = [] + try: + for field_name in field_names: + secrets.append( + _extract_optional_secret( + values, + field_name=field_name, + environment_name=f"WORKSTREAM_{field_name.upper()}", + ) + ) + except ValueError: + secrets.clear() + for field_name in field_names: + values.pop(field_name, None) + raise + return secrets[0], secrets[1], secrets[2] def _extract_optional_secret( @@ -504,8 +520,10 @@ def _extract_optional_secret( env_encoding = values.get("_env_file_encoding", "utf-8") env_files = (env_file,) if isinstance(env_file, (str, os.PathLike)) else tuple(env_file) for path in env_files: - dotenv = dotenv_values(path, encoding=env_encoding) - raw_value = dotenv.get(environment_name, raw_value) + raw_value = dotenv_values(path, encoding=env_encoding).get( + environment_name, + raw_value, + ) if raw_value is missing or raw_value is None: return None if isinstance(raw_value, SecretStr): @@ -513,6 +531,7 @@ def _extract_optional_secret( elif isinstance(raw_value, str): secret = SecretStr(raw_value) else: + del raw_value raise ValueError("invalid artifact storage secret") revealed = secret.get_secret_value() if ( @@ -520,6 +539,7 @@ def _extract_optional_secret( or len(revealed) > 4096 or any(ord(character) < 32 or ord(character) == 127 for character in revealed) ): + del raw_value, secret, revealed raise ValueError("invalid artifact storage secret") return secret diff --git a/backend/tests/assertion_helpers.py b/backend/tests/assertion_helpers.py index 253da4a9..13496b62 100644 --- a/backend/tests/assertion_helpers.py +++ b/backend/tests/assertion_helpers.py @@ -11,8 +11,10 @@ def assert_secret_not_retained( value: object, secret: str, seen: set[int] | None = None, + *, + traceback_module_prefixes: tuple[str, ...] = (), ) -> None: - """Assert a secret is unreachable through an error's public object graph.""" + """Assert a secret is unreachable through selected error state.""" if seen is None: seen = set() if id(value) in seen: @@ -24,15 +26,49 @@ def assert_secret_not_retained( assert secret not in value.get_secret_value() elif isinstance(value, BaseException): if isinstance(value, ValidationError): - assert_secret_not_retained(value.errors(), secret, seen) - assert_secret_not_retained(value.args, secret, seen) - assert_secret_not_retained(vars(value), secret, seen) - assert_secret_not_retained(value.__cause__, secret, seen) - assert_secret_not_retained(value.__context__, secret, seen) + assert_secret_not_retained( + value.errors(), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + for related in (value.args, vars(value), value.__cause__, value.__context__): + assert_secret_not_retained( + related, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + traceback = value.__traceback__ + while traceback is not None: + module_name = str(traceback.tb_frame.f_globals.get("__name__", "")) + if module_name.startswith(traceback_module_prefixes): + assert_secret_not_retained( + dict(traceback.tb_frame.f_locals), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + traceback = traceback.tb_next elif isinstance(value, Mapping): for key, item in value.items(): - assert_secret_not_retained(key, secret, seen) - assert_secret_not_retained(item, secret, seen) + assert_secret_not_retained( + key, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + assert_secret_not_retained( + item, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) elif isinstance(value, (list, tuple, set)): for item in value: - assert_secret_not_retained(item, secret, seen) + assert_secret_not_retained( + item, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index f71ddfda..9ce821c7 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -666,6 +666,37 @@ def test_minio_secret_values_are_absent_from_repr_and_validation_errors( assert_secret_not_retained(caught.value, secret) +@pytest.mark.parametrize( + "method_name", + ["constructor", "model_validate", "model_validate_json", "model_validate_strings"], +) +def test_invalid_minio_secret_is_absent_from_application_traceback( + tmp_path: Path, + method_name: str, +) -> None: + """Rejected credentials must not remain in Workstream traceback locals.""" + secret = "invalid-minio-secret\n" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + ) + + with pytest.raises(ValueError, match="invalid artifact storage secret") as caught: + if method_name == "constructor": + Settings(**payload) + elif method_name == "model_validate_json": + Settings.model_validate_json(json.dumps(payload)) + else: + getattr(Settings, method_name)(payload) + + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + @pytest.mark.parametrize( "method_name", ["model_validate", "model_validate_json", "model_validate_strings"], From 69f7c18d82707e0f1b7e40f5bf3074811d14d32e Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 13:25:50 +0100 Subject: [PATCH 13/39] Clear S3 secrets from configuration failures --- backend/app/core/config.py | 142 ++++++++++++++++++++--------- backend/tests/assertion_helpers.py | 7 ++ backend/tests/test_config.py | 34 ++++++- 3 files changed, 137 insertions(+), 46 deletions(-) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 2e63f62a..8df2cc42 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -36,6 +36,11 @@ "artifact_s3_session_token", } ) +_EMPTY_ARTIFACT_S3_SECRETS: tuple[SecretStr | None, SecretStr | None, SecretStr | None] = ( + None, + None, + None, +) _ALTERNATE_VALIDATION_RESTORES_SECRETS: ContextVar[bool] = ContextVar( "alternate_validation_restores_secrets", default=False, @@ -192,13 +197,22 @@ class Settings(BaseSettings): def __init__(self, **values: object) -> None: """Remove rate-control secret material before structured validation.""" - secret = _extract_api_rate_limit_key_secret(values) - s3_secrets = _extract_artifact_s3_static_secrets(values) - super().__init__(**values) - self._api_rate_limit_key_secret = secret - self._set_artifact_s3_static_secrets(s3_secrets) - if not _ALTERNATE_VALIDATION_RESTORES_SECRETS.get(): - self._validate_artifact_s3_secret_contract() + secret: SecretStr | None = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + try: + secret = _extract_api_rate_limit_key_secret(values) + s3_secrets = _extract_artifact_s3_static_secrets(values) + super().__init__(**values) + self._api_rate_limit_key_secret = secret + self._set_artifact_s3_static_secrets(s3_secrets) + if not _ALTERNATE_VALIDATION_RESTORES_SECRETS.get(): + self._validate_artifact_s3_secret_contract() + except ValueError: + _clear_settings_private_secrets(self) + values.clear() + secret = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + raise @classmethod def model_validate(cls, obj: object, **kwargs: object) -> Self: @@ -210,26 +224,38 @@ def model_validate(cls, obj: object, **kwargs: object) -> Self: sanitized = dict(obj) supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) obj = None - secret = ( - _extract_api_rate_limit_key_secret(sanitized) - if "api_rate_limit_key_secret" in sanitized - else None - ) - s3_secrets = _extract_artifact_s3_static_secrets(sanitized) - if secret is not None: - sanitized["api_rate_limit_key_secret"] = None - for field_name in supplied_s3_fields: - sanitized[field_name] = None - restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + secret: SecretStr | None = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings: Settings | None = None try: - settings = super().model_validate(sanitized, **kwargs) - finally: - _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) - if secret is not None: - settings._api_rate_limit_key_secret = secret - settings._set_artifact_s3_static_secrets(s3_secrets) - settings._validate_artifact_s3_secret_contract() - return settings + secret = ( + _extract_api_rate_limit_key_secret(sanitized) + if "api_rate_limit_key_secret" in sanitized + else None + ) + s3_secrets = _extract_artifact_s3_static_secrets(sanitized) + if secret is not None: + sanitized["api_rate_limit_key_secret"] = None + for field_name in supplied_s3_fields: + sanitized[field_name] = None + restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + try: + settings = super().model_validate(sanitized, **kwargs) + finally: + _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) + if secret is not None: + settings._api_rate_limit_key_secret = secret + settings._set_artifact_s3_static_secrets(s3_secrets) + settings._validate_artifact_s3_secret_contract() + return settings + except ValueError: + if settings is not None: + _clear_settings_private_secrets(settings) + sanitized.clear() + secret = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings = None + raise return super().model_validate(obj, **kwargs) @classmethod @@ -269,26 +295,38 @@ def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: sanitized = dict(obj) supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) obj = None - secret = ( - _extract_api_rate_limit_key_secret(sanitized) - if "api_rate_limit_key_secret" in sanitized - else None - ) - s3_secrets = _extract_artifact_s3_static_secrets(sanitized) - if secret is not None: - sanitized["api_rate_limit_key_secret"] = None - for field_name in supplied_s3_fields: - sanitized[field_name] = None - restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + secret: SecretStr | None = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings: Settings | None = None try: - settings = super().model_validate_strings(sanitized, **kwargs) - finally: - _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) - if secret is not None: - settings._api_rate_limit_key_secret = secret - settings._set_artifact_s3_static_secrets(s3_secrets) - settings._validate_artifact_s3_secret_contract() - return settings + secret = ( + _extract_api_rate_limit_key_secret(sanitized) + if "api_rate_limit_key_secret" in sanitized + else None + ) + s3_secrets = _extract_artifact_s3_static_secrets(sanitized) + if secret is not None: + sanitized["api_rate_limit_key_secret"] = None + for field_name in supplied_s3_fields: + sanitized[field_name] = None + restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + try: + settings = super().model_validate_strings(sanitized, **kwargs) + finally: + _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) + if secret is not None: + settings._api_rate_limit_key_secret = secret + settings._set_artifact_s3_static_secrets(s3_secrets) + settings._validate_artifact_s3_secret_contract() + return settings + except ValueError: + if settings is not None: + _clear_settings_private_secrets(settings) + sanitized.clear() + secret = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings = None + raise return super().model_validate_strings(obj, **kwargs) @classmethod @@ -348,15 +386,18 @@ def _validate_artifact_s3_secret_contract(self) -> None: if self.artifact_store_backend != "s3_compatible" and any( value is not None for value in secrets ): + secrets = _EMPTY_ARTIFACT_S3_SECRETS raise ValueError("static artifact credentials require MinIO storage") if self.artifact_s3_provider_profile == "minio" and ( self.artifact_s3_access_key_id is None or self.artifact_s3_secret_access_key is None ): + secrets = _EMPTY_ARTIFACT_S3_SECRETS raise ValueError("MinIO artifact storage requires complete static credentials") if self.artifact_s3_provider_profile == "aws_s3" and any( value is not None for value in secrets ): + secrets = _EMPTY_ARTIFACT_S3_SECRETS raise ValueError("native AWS S3 rejects static credentials") @model_validator(mode="after") @@ -450,6 +491,17 @@ def get_settings() -> Settings: return Settings() +def _clear_settings_private_secrets(settings: object) -> None: + """Remove secret values from a partially or fully initialized Settings object.""" + private_values = getattr(settings, "__pydantic_private__", None) + if not isinstance(private_values, dict): + return + private_values["_api_rate_limit_key_secret"] = None + private_values["_artifact_s3_access_key_id"] = None + private_values["_artifact_s3_secret_access_key"] = None + private_values["_artifact_s3_session_token"] = None + + def _extract_api_rate_limit_key_secret(values: dict[str, object]) -> SecretStr | None: """Resolve and validate the key without entering Pydantic's input graph.""" missing = object() diff --git a/backend/tests/assertion_helpers.py b/backend/tests/assertion_helpers.py index 13496b62..76dc8ab2 100644 --- a/backend/tests/assertion_helpers.py +++ b/backend/tests/assertion_helpers.py @@ -64,6 +64,13 @@ def assert_secret_not_retained( seen, traceback_module_prefixes=traceback_module_prefixes, ) + elif isinstance(getattr(value, "__pydantic_private__", None), Mapping): + assert_secret_not_retained( + getattr(value, "__pydantic_private__"), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) elif isinstance(value, (list, tuple, set)): for item in value: assert_secret_not_retained( diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 9ce821c7..41dcb4a4 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -663,7 +663,39 @@ def test_minio_secret_values_are_absent_from_repr_and_validation_errors( assert secret not in repr(caught.value.errors()) assert secret not in caught.value.json() - assert_secret_not_retained(caught.value, secret) + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +@pytest.mark.parametrize( + "method_name", + ["model_validate", "model_validate_json", "model_validate_strings"], +) +def test_valid_minio_secret_is_absent_from_unrelated_application_traceback( + tmp_path: Path, + method_name: str, +) -> None: + """Later configuration errors must not retain an accepted credential.""" + secret = "valid-minio-secret-for-traceback" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + artifact_s3_region="US-east-1", + ) + method = getattr(Settings, method_name) + + with pytest.raises(ValueError) as caught: + method(json.dumps(payload) if method_name.endswith("json") else payload) + + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( From 6e99d1940c19919c759e726116f7cfce4096bced Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 14:20:31 +0100 Subject: [PATCH 14/39] Clear malformed secret JSON from tracebacks --- backend/app/core/config.py | 1 + backend/tests/test_config.py | 16 +++++++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 8df2cc42..2eca3847 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -272,6 +272,7 @@ def model_validate_json( malformed = True parsed = None if malformed: + json_data = b"" raise ValueError("invalid settings JSON") if isinstance(parsed, Mapping) and ( "api_rate_limit_key_secret" in parsed diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 41dcb4a4..6aaedbec 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -262,9 +262,15 @@ def test_alternate_validation_secret_is_absent_from_unrelated_errors( assert_secret_not_retained(caught.value, encoded) -def test_model_validate_json_rejects_malformed_document_without_echo() -> None: +@pytest.mark.parametrize( + "field_name", + ["api_rate_limit_key_secret", "artifact_s3_secret_access_key"], +) +def test_model_validate_json_rejects_malformed_document_without_echo( + field_name: str, +) -> None: invalid = "not-a-canonical-secret" - payload = f'{{"api_rate_limit_key_secret":"{invalid}"' + payload = f'{{"{field_name}":"{invalid}"' with pytest.raises(ValueError, match="^invalid settings JSON$") as caught: Settings.model_validate_json(payload) @@ -273,7 +279,11 @@ def test_model_validate_json_rejects_malformed_document_without_echo() -> None: assert invalid not in f"{caught.value!s} {caught.value!r}" assert caught.value.__cause__ is None assert caught.value.__context__ is None - assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained( + caught.value, + invalid, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( From 708eca1514fb5dbe41a308128ae3035b62b28514 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 14:43:50 +0100 Subject: [PATCH 15/39] Harden secret cleanup and S3 replay races --- .../app/adapters/artifacts/s3_compatible.py | 5 +- backend/app/core/config.py | 74 +++++++++++-------- backend/tests/test_config.py | 48 ++++++++++-- backend/tests/test_s3_artifact_store.py | 31 ++++++++ 4 files changed, 119 insertions(+), 39 deletions(-) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index b0abf75b..abad64e4 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -339,8 +339,9 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: status in {409, 412} and _provider_error_code(error) in _PRECONDITION_ERROR_CODES ): - await self._verify_exact(provider_object_ref, commitment) - return ArtifactPutResult(provider_object_ref, replayed=True) + if await self._matches_commitment(provider_object_ref, commitment): + return ArtifactPutResult(provider_object_ref, replayed=True) + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None except ArtifactStoreError: raise diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 2eca3847..a3586121 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -41,6 +41,7 @@ None, None, ) +_MISSING_SECRET = object() _ALTERNATE_VALIDATION_RESTORES_SECRETS: ContextVar[bool] = ContextVar( "alternate_validation_restores_secrets", default=False, @@ -505,27 +506,25 @@ def _clear_settings_private_secrets(settings: object) -> None: def _extract_api_rate_limit_key_secret(values: dict[str, object]) -> SecretStr | None: """Resolve and validate the key without entering Pydantic's input graph.""" - missing = object() - raw_value = values.pop("api_rate_limit_key_secret", missing) - if raw_value is missing: - raw_value = os.environ.get("WORKSTREAM_API_RATE_LIMIT_KEY_SECRET", missing) - if raw_value is missing: - env_file = values.get("_env_file", ".env") - if env_file is not None: - env_encoding = values.get("_env_file_encoding", "utf-8") - env_files = (env_file,) if isinstance(env_file, (str, os.PathLike)) else tuple(env_file) - for path in env_files: - dotenv = dotenv_values(path, encoding=env_encoding) - raw_value = dotenv.get("WORKSTREAM_API_RATE_LIMIT_KEY_SECRET", raw_value) - if raw_value is missing or raw_value is None: + raw_value = _pop_optional_secret_source( + values, + field_name="api_rate_limit_key_secret", + environment_name="WORKSTREAM_API_RATE_LIMIT_KEY_SECRET", + ) + if raw_value is _MISSING_SECRET or raw_value is None: return None if isinstance(raw_value, SecretStr): secret = raw_value elif isinstance(raw_value, str): secret = SecretStr(raw_value) else: + del raw_value raise ValueError("invalid API rate limit key secret") - decode_api_rate_limit_key_secret(secret) + try: + decode_api_rate_limit_key_secret(secret) + except ValueError: + del raw_value, secret + raise return secret @@ -563,21 +562,12 @@ def _extract_optional_secret( environment_name: str, ) -> SecretStr | None: """Resolve one bounded secret without retaining it in Pydantic input.""" - missing = object() - raw_value = values.pop(field_name, missing) - if raw_value is missing: - raw_value = os.environ.get(environment_name, missing) - if raw_value is missing: - env_file = values.get("_env_file", ".env") - if env_file is not None: - env_encoding = values.get("_env_file_encoding", "utf-8") - env_files = (env_file,) if isinstance(env_file, (str, os.PathLike)) else tuple(env_file) - for path in env_files: - raw_value = dotenv_values(path, encoding=env_encoding).get( - environment_name, - raw_value, - ) - if raw_value is missing or raw_value is None: + raw_value = _pop_optional_secret_source( + values, + field_name=field_name, + environment_name=environment_name, + ) + if raw_value is _MISSING_SECRET or raw_value is None: return None if isinstance(raw_value, SecretStr): secret = raw_value @@ -597,10 +587,34 @@ def _extract_optional_secret( return secret +def _pop_optional_secret_source( + values: dict[str, object], + *, + field_name: str, + environment_name: str, +) -> object: + """Remove one optional secret from direct, environment, or dotenv input.""" + raw_value = values.pop(field_name, _MISSING_SECRET) + if raw_value is _MISSING_SECRET: + raw_value = os.environ.get(environment_name, _MISSING_SECRET) + if raw_value is _MISSING_SECRET: + env_file = values.get("_env_file", ".env") + if env_file is not None: + env_encoding = values.get("_env_file_encoding", "utf-8") + env_files = (env_file,) if isinstance(env_file, (str, os.PathLike)) else tuple(env_file) + for path in env_files: + raw_value = dotenv_values(path, encoding=env_encoding).get( + environment_name, + raw_value, + ) + return raw_value + + def decode_api_rate_limit_key_secret(value: SecretStr) -> bytes: """Decode one canonical padded Base64 rate-control key.""" secret = value.get_secret_value() if not secret.isascii(): + del value, secret raise ValueError("invalid API rate limit key secret") invalid = False try: @@ -611,7 +625,9 @@ def decode_api_rate_limit_key_secret(value: SecretStr) -> bytes: encoded = b"" decoded = b"" if invalid: + del value, secret, encoded, decoded raise ValueError("invalid API rate limit key secret") if not 32 <= len(decoded) <= 64 or base64.b64encode(decoded) != encoded: + del value, secret, encoded, decoded raise ValueError("invalid API rate limit key secret") return decoded diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 6aaedbec..529e53af 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -114,7 +114,11 @@ def test_rate_limit_secret_is_absent_from_unrelated_structured_errors() -> None: assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) def test_environment_rate_limit_secret_is_absent_from_unrelated_structured_errors( @@ -128,7 +132,11 @@ def test_environment_rate_limit_secret_is_absent_from_unrelated_structured_error assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) def test_dotenv_rate_limit_secret_is_absent_from_unrelated_structured_errors( @@ -146,7 +154,11 @@ def test_dotenv_rate_limit_secret_is_absent_from_unrelated_structured_errors( assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) def test_model_validate_rejects_rate_limit_secret_without_structured_echo() -> None: @@ -157,7 +169,11 @@ def test_model_validate_rejects_rate_limit_secret_without_structured_echo() -> N assert not isinstance(caught.value, ValidationError) assert invalid not in f"{caught.value!s} {caught.value!r}" - assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained( + caught.value, + invalid, + traceback_module_prefixes=("app.",), + ) def test_model_validate_rate_limit_secret_is_absent_from_unrelated_errors() -> None: @@ -173,7 +189,11 @@ def test_model_validate_rate_limit_secret_is_absent_from_unrelated_errors() -> N assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( @@ -240,7 +260,11 @@ def test_alternate_validation_rejects_rate_limit_secret_without_echo( assert not isinstance(caught.value, ValidationError) assert invalid not in f"{caught.value!s} {caught.value!r}" - assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained( + caught.value, + invalid, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize("method_name", ["model_validate_json", "model_validate_strings"]) @@ -259,7 +283,11 @@ def test_alternate_validation_secret_is_absent_from_unrelated_errors( assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( @@ -307,7 +335,11 @@ def test_rate_limit_secret_rejects_invalid_values_without_echo(value: str) -> No assert not hasattr(caught.value, "json") if value.strip(): assert value not in rendered - assert_secret_not_retained(caught.value, value) + assert_secret_not_retained( + caught.value, + value, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 731e03b8..44943e56 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -582,6 +582,9 @@ class Client: async def put_object(self, **_kwargs: object) -> object: raise _client_error("PreconditionFailed", 412) + async def head_object(self, **_kwargs: object) -> object: + return {"ContentLength": 6} + @asynccontextmanager async def fake_client() -> Any: yield Client() @@ -600,6 +603,34 @@ async def verify(provider_ref: str, _commitment: object) -> None: store.close() +@pytest.mark.asyncio +async def test_precondition_failure_with_missing_replay_is_retryable( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Treat a not-yet-visible conditional-write winner as replay-required.""" + store = initialize_minio_store(private_prefix="negative/precondition-missing") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("PreconditionFailed", 412) + + async def head_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", 404) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + async with minted_source(tmp_path / "scratch", b"unverified-replay") as source: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + finally: + store.close() + + @pytest.mark.asyncio async def test_success_without_full_source_consumption_is_not_replay( monkeypatch: pytest.MonkeyPatch, From a8ebca079070173289d00b504efd0ae6c9b484a0 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 15:21:10 +0100 Subject: [PATCH 16/39] Validate sealed sources before replay --- .../app/adapters/artifacts/s3_compatible.py | 17 ++++++ backend/tests/test_s3_artifact_store.py | 54 +++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index abad64e4..b7caa4a7 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -339,6 +339,7 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: status in {409, 412} and _provider_error_code(error) in _PRECONDITION_ERROR_CODES ): + await self._validate_replay_source(body) if await self._matches_commitment(provider_object_ref, commitment): return ArtifactPutResult(provider_object_ref, replayed=True) raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None @@ -346,6 +347,7 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: except ArtifactStoreError: raise except (TimeoutError, BotoCoreError, OSError): + await self._validate_replay_source(body) if await self._matches_commitment(provider_object_ref, commitment): return ArtifactPutResult(provider_object_ref, replayed=True) raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None @@ -495,6 +497,14 @@ async def _matches_commitment( return False return True + async def _validate_replay_source(self, body: _CommittedSourceBody) -> None: + """Finish validating the sealed source before accepting any replay.""" + try: + async with asyncio.timeout(self._operation_total_timeout_seconds): + await body.validate_complete() + except TimeoutError: + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + @asynccontextmanager async def _client(self) -> AsyncIterator[Any]: """Create one bounded S3 client from the already-isolated session.""" @@ -597,6 +607,13 @@ async def validate_empty(self) -> None: if await self.read(1) != b"" or not self._complete: raise ArtifactInputMismatchError("artifact source violates commitment") + async def validate_complete(self) -> None: + """Consume and validate every remaining sealed source byte.""" + while await self.read(self._buffer_bytes): + pass + if not self._complete: + raise ArtifactInputMismatchError("artifact source violates commitment") + async def _require_source_exhausted(self) -> None: """Prove EOF before returning the final committed byte to the SDK.""" if self._iterator is None: diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 44943e56..abed3f34 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -2,6 +2,7 @@ from __future__ import annotations +from collections.abc import AsyncIterator from contextlib import asynccontextmanager import hashlib import os @@ -33,6 +34,8 @@ artifact_store_namespace_material, ) from app.interfaces.external_services import ExternalServiceAdapterIdentity +from app.modules.artifacts.preparation import ArtifactPreparationService +from app.modules.artifacts.sources import ArtifactCommitment from tests.assertion_helpers import assert_secret_not_retained from tests.artifact_store_helpers import minted_source from tests.test_artifact_store_conformance import ArtifactStoreConformanceTests @@ -805,6 +808,57 @@ async def lost_acknowledgement_then_real_provider() -> Any: store.close() +@pytest.mark.asyncio +@pytest.mark.parametrize("failure_mode", ["precondition", "transport"]) +async def test_replay_paths_cannot_accept_malformed_source( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + failure_mode: str, +) -> None: + """Require source proof even when an exact provider object is visible.""" + store = initialize_minio_store(private_prefix="negative/malformed-replay") + provider_checked = False + + class Client: + async def put_object(self, **kwargs: object) -> object: + if failure_mode == "precondition": + raise _client_error("PreconditionFailed", 412) + body: Any = kwargs["Body"] + assert await body.read(4) == b"go" + raise OSError("connection closed after partial source consumption") + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def malformed_stream( + _self: ArtifactPreparationService, + _binding: object, + _commitment: ArtifactCommitment, + ) -> AsyncIterator[bytes]: + yield b"goodX" + + async def exact_provider_match(*_args: object) -> bool: + nonlocal provider_checked + provider_checked = True + return True + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_matches_commitment", exact_provider_match) + try: + async with minted_source(tmp_path / "scratch", b"good") as source: + monkeypatch.setattr( + ArtifactPreparationService, + "open_committed_stream", + malformed_stream, + ) + with pytest.raises(ArtifactInputMismatchError): + await store.put(source) + assert provider_checked is False + finally: + store.close() + + @pytest.mark.asyncio async def test_put_observation_requires_canonical_commitment() -> None: """Reject caller-assembled observation requests before provider I/O.""" From 6b0f0c6c9964d7fe1067b338cb66ab5c7e95cfe1 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 16:01:22 +0100 Subject: [PATCH 17/39] Sanitize AWS credential resolution failures --- .../app/adapters/artifacts/s3_compatible.py | 11 +++++++- .../tests/test_aws_credential_isolation.py | 26 +++++++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index b7caa4a7..1a02ddcd 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -846,7 +846,16 @@ async def resolve_isolated_aws_workload_credentials( expected_method: str, ) -> object: """Resolve credentials and require the exact selected SDK method.""" - credentials = await session.get_credentials() + resolution_failed = False + try: + credentials = await session.get_credentials() + except Exception: + resolution_failed = True + credentials = None + if resolution_failed: + raise ArtifactConfigurationError( + "AWS workload identity credentials could not be resolved" + ) from None if credentials is None or getattr(credentials, "method", None) != expected_method: raise ArtifactConfigurationError("AWS workload identity method did not match") return credentials diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 5b1664dd..0bdf2425 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -14,6 +14,7 @@ from app.adapters.artifacts import s3_compatible from app.core.config import Settings from app.interfaces.artifacts import ArtifactConfigurationError +from tests.assertion_helpers import assert_secret_not_retained def _aws_settings(tmp_path: Path, method: str = "container-role") -> Settings: @@ -588,3 +589,28 @@ async def get_credentials(self) -> object | None: expected_method="container-role", ) assert getattr(resolved, "method") == "container-role" + + +async def test_credential_resolution_failure_is_sanitized() -> None: + secret = "provider-response-secret" + + class FakeSession: + async def get_credentials(self) -> object: + raise RuntimeError(f"credential endpoint returned {secret}") + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity credentials could not be resolved", + ) as caught: + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(), # type: ignore[arg-type] + expected_method="container-role", + ) + + assert caught.value.__cause__ is None + assert caught.value.__context__ is None + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) From f2a4d8c1b9cf3d383e938009193530a7a632509e Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 17:05:32 +0100 Subject: [PATCH 18/39] Prevent AWS metadata credential logging --- .../app/adapters/artifacts/s3_compatible.py | 256 +++++++++++++++++- .../tests/test_aws_credential_isolation.py | 159 ++++++++++- 2 files changed, 406 insertions(+), 9 deletions(-) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index 1a02ddcd..3024e727 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -5,9 +5,13 @@ import asyncio from collections.abc import AsyncIterator, Mapping from contextlib import asynccontextmanager +import datetime import hashlib +import inspect +import json import os from pathlib import Path +import random import re from typing import Any @@ -17,6 +21,7 @@ AioContainerProvider, AioCredentialResolver, AioInstanceMetadataProvider, + AioRefreshableCredentials, ) from aiobotocore.session import AioSession from aiobotocore.utils import ( @@ -25,7 +30,21 @@ _RefCountedSession, ) from botocore import UNSIGNED -from botocore.exceptions import BotoCoreError, ClientError +from botocore.awsrequest import AWSRequest +from botocore.exceptions import ( + BotoCoreError, + ClientError, + CredentialRetrievalError, + HTTPClientError, + InvalidIMDSEndpointError, + MetadataRetrievalError, + ReadTimeoutError, +) +from botocore.utils import ( + BadIMDSRequestError, + RETRYABLE_HTTP_ERRORS, + get_current_datetime, +) from app.adapters.artifacts.references import ( artifact_provider_object_ref, @@ -88,6 +107,233 @@ "BOTO_CONFIG", } ) + + +class _SanitizedContainerMetadataFetcher(AioContainerMetadataFetcher): + """Fetch ECS credentials without SDK logging response or exception data.""" + + async def _retrieve_credentials( + self, + full_url: str, + extra_headers: Mapping[str, str] | None = None, + ) -> dict[str, Any]: + headers = {"Accept": "application/json"} + if extra_headers is not None: + headers.update(extra_headers) + failed = False + for attempt in range(self.RETRY_ATTEMPTS): + try: + return await self._get_response( + full_url, + headers, + self.TIMEOUT_SECONDS, + ) + except MetadataRetrievalError: + if attempt + 1 >= self.RETRY_ATTEMPTS: + failed = True + break + await self._sleep(self.SLEEP_TIME) + if failed: + raise MetadataRetrievalError( + error_msg="container metadata request failed" + ) from None + raise MetadataRetrievalError(error_msg="container metadata request failed") + + async def _get_response( + self, + full_url: str, + headers: Mapping[str, str], + timeout: float, + ) -> dict[str, Any]: + try: + async with self._session.acquire() as session: + request = AWSRequest(method="GET", url=full_url, headers=dict(headers)) + response = await session.send(request.prepare()) + response_text = (await response.content).decode("utf-8") + if response.status_code != 200: + raise MetadataRetrievalError( + error_msg="container metadata returned a non-200 response" + ) + try: + parsed = json.loads(response_text) + except ValueError: + raise MetadataRetrievalError( + error_msg="container metadata returned invalid JSON" + ) from None + if not isinstance(parsed, dict): + raise MetadataRetrievalError( + error_msg="container metadata returned an invalid document" + ) + return parsed + except RETRYABLE_HTTP_ERRORS: + raise MetadataRetrievalError( + error_msg="container metadata request failed" + ) from None + + +class _SanitizedInstanceMetadataFetcher(AioInstanceMetadataFetcher): + """Fetch IMDSv2 credentials without SDK logging response or exception data.""" + + async def _fetch_metadata_token(self) -> str | None: + self._assert_enabled() + url = self._construct_url(self._TOKEN_PATH) + headers = {"x-aws-ec2-metadata-token-ttl-seconds": self._TOKEN_TTL} + self._add_user_agent(headers) + request = AWSRequest(method="PUT", url=url, headers=headers) + async with self._session.acquire() as session: + for _ in range(self._num_attempts): + try: + response = await session.send(request.prepare()) + if response.status_code == 200: + return await response.text + if response.status_code in (403, 404, 405): + return None + if response.status_code == 400: + raise BadIMDSRequestError(request) + except ReadTimeoutError: + return None + except RETRYABLE_HTTP_ERRORS: + continue + except HTTPClientError as error: + nested_error = error.kwargs.get("error") + name_resolution_failed = ( + nested_error + and getattr(nested_error, "errno", None) == 8 + ) or str(getattr(nested_error, "os_error", None)) == ( + "Domain name not found" + ) + if name_resolution_failed: + raise InvalidIMDSEndpointError( + endpoint=url, + error=error, + ) from None + raise + return None + + async def _get_request( + self, + url_path: str, + retry_func: Any, + token: str | None = None, + ) -> object: + self._assert_enabled() + if not token: + self._assert_v1_enabled() + retry_func = retry_func or self._default_retry + url = self._construct_url(url_path) + headers: dict[str, str] = {} + if token is not None: + headers["x-aws-ec2-metadata-token"] = token + self._add_user_agent(headers) + async with self._session.acquire() as session: + for _ in range(self._num_attempts): + try: + request = AWSRequest(method="GET", url=url, headers=headers) + response = await session.send(request.prepare()) + should_retry = retry_func(response) + if inspect.isawaitable(should_retry): + should_retry = await should_retry + if not should_retry: + return response + except RETRYABLE_HTTP_ERRORS: + continue + raise self._RETRIES_EXCEEDED_ERROR_CLS() + + async def _log_imds_response( + self, + response: object, + reason_to_log: str, + log_body: bool = False, + ) -> None: + return None + + async def retrieve_iam_role_credentials(self) -> dict[str, Any]: + try: + token = await self._fetch_metadata_token() + role_name = await self._get_iam_role(token) + credentials = await self._get_credentials(role_name, token) + if not self._contains_all_credential_fields(credentials): + return {} + resolved = { + "role_name": role_name, + "access_key": credentials["AccessKeyId"], + "secret_key": credentials["SecretAccessKey"], + "token": credentials["Token"], + "expiry_time": credentials["Expiration"], + } + self._evaluate_expiration(resolved) + return resolved + except (self._RETRIES_EXCEEDED_ERROR_CLS, BadIMDSRequestError): + return {} + + def _evaluate_expiration(self, credentials: dict[str, Any]) -> None: + expiration = credentials.get("expiry_time") + if expiration is None: + return + try: + parsed_expiration = datetime.datetime.strptime( + expiration, + "%Y-%m-%dT%H:%M:%SZ", + ) + except (TypeError, ValueError): + return + refresh_interval = self._config.get( + "ec2_credential_refresh_window", + 60 * 10, + ) + refresh_offset = datetime.timedelta( + seconds=refresh_interval + random.randint(120, 600) + ) + current_time = get_current_datetime() + if current_time >= parsed_expiration - refresh_offset: + credentials["expiry_time"] = (current_time + refresh_offset).strftime( + "%Y-%m-%dT%H:%M:%SZ" + ) + + +class _SanitizedContainerProvider(AioContainerProvider): + """Build ECS credentials without SDK exception logging.""" + + def _create_fetcher(self, full_uri: str, *_args: object, **_kwargs: object) -> Any: + async def fetch_credentials() -> dict[str, Any]: + retrieval_failed = False + try: + headers = self._build_headers() + response = await self._fetcher.retrieve_full_uri( + full_uri, + headers=headers, + ) + except MetadataRetrievalError: + retrieval_failed = True + response = {} + if retrieval_failed: + raise CredentialRetrievalError( + provider=self.METHOD, + error_msg="container metadata request failed", + ) from None + return { + "access_key": response["AccessKeyId"], + "secret_key": response["SecretAccessKey"], + "token": response["Token"], + "expiry_time": response["Expiration"], + "account_id": response.get("AccountId"), + } + + return fetch_credentials + + +class _SanitizedInstanceMetadataProvider(AioInstanceMetadataProvider): + """Build IMDS credentials without SDK response-derived logging.""" + + async def load(self) -> object | None: + metadata = await self._role_fetcher.retrieve_iam_role_credentials() + if not metadata: + return None + return AioRefreshableCredentials.create_from_metadata( + metadata, + method=self.METHOD, + refresh_using=self._role_fetcher.retrieve_iam_role_credentials, + ) _FORBIDDEN_AWS_NETWORK_ENVIRONMENT = frozenset( { "AWS_CA_BUNDLE", @@ -805,22 +1051,22 @@ def create_isolated_sts_client(service_name: str, **_kwargs: object) -> object: disable_env_vars=True, ) elif selected == "container-role": - provider = AioContainerProvider( + provider = _SanitizedContainerProvider( environ={ "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": environment[ "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" ] } ) - provider._fetcher = AioContainerMetadataFetcher( + provider._fetcher = _SanitizedContainerMetadataFetcher( session=_RefCountedSession( timeout=settings.artifact_s3_connect_timeout_seconds, proxies={}, ) ) elif selected == "iam-role": - provider = AioInstanceMetadataProvider( - iam_role_fetcher=AioInstanceMetadataFetcher( + provider = _SanitizedInstanceMetadataProvider( + iam_role_fetcher=_SanitizedInstanceMetadataFetcher( timeout=settings.artifact_s3_connect_timeout_seconds, num_attempts=1, env=dict(environment), diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 0bdf2425..89b83a2b 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -7,8 +7,6 @@ import pytest from aiobotocore.credentials import ( AioAssumeRoleWithWebIdentityProvider, - AioContainerProvider, - AioInstanceMetadataProvider, ) from app.adapters.artifacts import s3_compatible @@ -273,8 +271,18 @@ def test_unselected_workload_identity_sources_fail_before_resolver_loading( AioAssumeRoleWithWebIdentityProvider, "assume-role-with-web-identity", ), - ("container-role", _container_environment, AioContainerProvider, "container-role"), - ("iam-role", _iam_environment, AioInstanceMetadataProvider, "iam-role"), + ( + "container-role", + _container_environment, + s3_compatible._SanitizedContainerProvider, # noqa: SLF001 + "container-role", + ), + ( + "iam-role", + _iam_environment, + s3_compatible._SanitizedInstanceMetadataProvider, # noqa: SLF001 + "iam-role", + ), ], ) def test_isolated_session_contains_exactly_one_selected_provider( @@ -614,3 +622,146 @@ async def get_credentials(self) -> object: secret, traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), ) + + +async def test_container_metadata_error_body_is_never_logged_or_retained( + caplog: pytest.LogCaptureFixture, +) -> None: + secret = "ecs-response-secret" + + class Transport: + async def send(self, _request: object) -> object: + async def content() -> bytes: + return f'{{"SecretAccessKey":"{secret}"'.encode() + + return SimpleNamespace(status_code=200, content=content()) + + class Acquisition: + async def __aenter__(self) -> Transport: + return Transport() + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def acquire(self) -> Acquisition: + return Acquisition() + + async def no_sleep(_delay: float) -> None: + return None + + fetcher = s3_compatible._SanitizedContainerMetadataFetcher( # noqa: SLF001 + session=Pool(), # type: ignore[arg-type] + sleep=no_sleep, + ) + provider = s3_compatible._SanitizedContainerProvider( # noqa: SLF001 + environ={ + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ) + } + ) + provider._fetcher = fetcher + + class FakeSession: + async def get_credentials(self) -> object: + return await provider.load() + + caplog.set_level("DEBUG") + with pytest.raises(ArtifactConfigurationError) as caught: + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(), # type: ignore[arg-type] + expected_method="container-role", + ) + + assert secret not in caplog.text + for record in caplog.records: + assert_secret_not_retained( + (record.msg, record.args, record.exc_info, record.exc_text), + secret, + traceback_module_prefixes=( + "aiobotocore.credentials", + "app.adapters.artifacts.s3_compatible", + ), + ) + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) + + +async def test_imds_error_responses_are_never_logged( + caplog: pytest.LogCaptureFixture, +) -> None: + secret = "imds-response-secret" + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + ) + + response = SimpleNamespace( + status_code=500, + url="http://169.254.169.254/latest/meta-data/iam/security-credentials/", + content=secret.encode(), + ) + caplog.set_level("DEBUG") + + assert await fetcher._is_non_ok_response(response) is True # noqa: SLF001 + + async def token() -> str: + return "imds-v2-token" + + async def role_name(_token: str) -> str: + return "workstream-role" + + async def credentials(_role_name: str, _token: str) -> dict[str, str]: + return {"Code": "Error", "Message": secret} + + fetcher._fetch_metadata_token = token # type: ignore[method-assign] + fetcher._get_iam_role = role_name # type: ignore[method-assign] + fetcher._get_credentials = credentials # type: ignore[method-assign] + assert await fetcher.retrieve_iam_role_credentials() == {} + + class FakeRoleFetcher: + async def retrieve_iam_role_credentials(self) -> dict[str, str]: + return { + "role_name": secret, + "access_key": "access-key", + "secret_key": "secret-key", + "token": "session-token", + "expiry_time": "2099-01-01T00:00:00Z", + } + + provider = s3_compatible._SanitizedInstanceMetadataProvider( # noqa: SLF001 + iam_role_fetcher=FakeRoleFetcher() + ) + assert await provider.load() is not None + assert secret not in caplog.text + for record in caplog.records: + assert_secret_not_retained( + (record.msg, record.args, record.exc_info, record.exc_text), + secret, + ) + + +def test_imds_invalid_expiration_is_never_logged( + caplog: pytest.LogCaptureFixture, +) -> None: + secret = "imds-expiration-secret" + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + ) + credentials = {"expiry_time": secret} + caplog.set_level("DEBUG") + + fetcher._evaluate_expiration(credentials) # noqa: SLF001 + + assert credentials == {"expiry_time": secret} + assert secret not in caplog.text + for record in caplog.records: + assert_secret_not_retained( + (record.msg, record.args, record.exc_info, record.exc_text), + secret, + ) From 8003a2e03f9ba37e9b5b7bc74b0bd52184d7b936 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 17:30:19 +0100 Subject: [PATCH 19/39] Prove sanitized AWS metadata paths --- .../tests/test_aws_credential_isolation.py | 273 ++++++++++++++++++ 1 file changed, 273 insertions(+) diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 89b83a2b..02310450 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -1,5 +1,6 @@ from __future__ import annotations +import datetime from pathlib import Path from types import SimpleNamespace from typing import Any @@ -8,6 +9,7 @@ from aiobotocore.credentials import ( AioAssumeRoleWithWebIdentityProvider, ) +from botocore.exceptions import MetadataRetrievalError, ReadTimeoutError from app.adapters.artifacts import s3_compatible from app.core.config import Settings @@ -765,3 +767,274 @@ def test_imds_invalid_expiration_is_never_logged( (record.msg, record.args, record.exc_info, record.exc_text), secret, ) + + +async def test_sanitized_imds_fetcher_success_paths() -> None: + async def ready(value: object) -> object: + return value + + token_response = SimpleNamespace( + status_code=200, + text=ready("imds-v2-token"), + ) + + class Transport: + def __init__(self, response: object) -> None: + self.response = response + + async def send(self, _request: object) -> object: + return self.response + + class Acquisition: + def __init__(self, transport: Transport) -> None: + self.transport = transport + + async def __aenter__(self) -> Transport: + return self.transport + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def __init__(self, response: object) -> None: + self.transport = Transport(response) + + def acquire(self) -> Acquisition: + return Acquisition(self.transport) + + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + session=Pool(token_response), # type: ignore[arg-type] + ) + assert await fetcher._fetch_metadata_token() == "imds-v2-token" # noqa: SLF001 + + metadata_response = SimpleNamespace(status_code=200) + fetcher._session = Pool(metadata_response) + + async def no_retry(_response: object) -> bool: + return False + + assert ( + await fetcher._get_request( # noqa: SLF001 + "latest/meta-data/iam/security-credentials/workstream-role", + no_retry, + token="imds-v2-token", + ) + is metadata_response + ) + + async def token() -> str: + return "imds-v2-token" + + async def role_name(_token: str) -> str: + return "workstream-role" + + async def credentials(_role_name: str, _token: str) -> dict[str, str]: + return { + "AccessKeyId": "access-key", + "SecretAccessKey": "secret-key", + "Token": "session-token", + "Expiration": "2099-01-01T00:00:00Z", + } + + fetcher._fetch_metadata_token = token # type: ignore[method-assign] + fetcher._get_iam_role = role_name # type: ignore[method-assign] + fetcher._get_credentials = credentials # type: ignore[method-assign] + assert await fetcher.retrieve_iam_role_credentials() == { + "role_name": "workstream-role", + "access_key": "access-key", + "secret_key": "secret-key", + "token": "session-token", + "expiry_time": "2099-01-01T00:00:00Z", + } + + +async def test_sanitized_container_provider_success_path() -> None: + class Fetcher: + async def retrieve_full_uri( + self, + _full_uri: str, + *, + headers: dict[str, str], + ) -> dict[str, str]: + assert headers == {"Authorization": "Bearer identity-token"} + return { + "AccessKeyId": "access-key", + "SecretAccessKey": "secret-key", + "Token": "session-token", + "Expiration": "2099-01-01T00:00:00Z", + "AccountId": "123456789012", + } + + provider = s3_compatible._SanitizedContainerProvider( # noqa: SLF001 + environ={ + "AWS_CONTAINER_CREDENTIALS_FULL_URI": ( + "http://169.254.170.2/v2/credentials/workstream-runtime" + ), + "AWS_CONTAINER_AUTHORIZATION_TOKEN": "Bearer identity-token", + } + ) + provider._fetcher = Fetcher() + fetch_credentials = provider._create_fetcher( # noqa: SLF001 + "http://169.254.170.2/v2/credentials/workstream-runtime" + ) + + assert await fetch_credentials() == { + "access_key": "access-key", + "secret_key": "secret-key", + "token": "session-token", + "expiry_time": "2099-01-01T00:00:00Z", + "account_id": "123456789012", + } + + +@pytest.mark.parametrize( + ("status_code", "body", "match"), + [ + (500, b'{"SecretAccessKey":"provider-secret"}', "non-200"), + (200, b"[]", "invalid document"), + ], +) +async def test_sanitized_container_fetcher_rejects_response_shapes( + status_code: int, + body: bytes, + match: str, +) -> None: + class Transport: + async def send(self, _request: object) -> object: + async def content() -> bytes: + return body + + return SimpleNamespace(status_code=status_code, content=content()) + + class Acquisition: + async def __aenter__(self) -> Transport: + return Transport() + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def acquire(self) -> Acquisition: + return Acquisition() + + fetcher = s3_compatible._SanitizedContainerMetadataFetcher( # noqa: SLF001 + session=Pool(), # type: ignore[arg-type] + ) + + with pytest.raises(MetadataRetrievalError, match=match): + await fetcher._get_response( # noqa: SLF001 + "http://169.254.170.2/v2/credentials/workstream-runtime", + {"Accept": "application/json"}, + 1.0, + ) + + +async def test_sanitized_container_fetcher_accepts_headers_and_document() -> None: + observed_headers: dict[str, str] = {} + + class Transport: + async def send(self, request: object) -> object: + observed_headers.update(dict(request.headers.items())) + + async def content() -> bytes: + return b'{"AccessKeyId":"access-key"}' + + return SimpleNamespace(status_code=200, content=content()) + + class Acquisition: + async def __aenter__(self) -> Transport: + return Transport() + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def acquire(self) -> Acquisition: + return Acquisition() + + fetcher = s3_compatible._SanitizedContainerMetadataFetcher( # noqa: SLF001 + session=Pool(), # type: ignore[arg-type] + ) + + assert await fetcher._retrieve_credentials( # noqa: SLF001 + "http://169.254.170.2/v2/credentials/workstream-runtime", + {"Authorization": "Bearer identity-token"}, + ) == {"AccessKeyId": "access-key"} + assert observed_headers["Authorization"] == "Bearer identity-token" + + +async def test_sanitized_imds_fetcher_failure_and_empty_paths( + monkeypatch: pytest.MonkeyPatch, +) -> None: + class Transport: + def __init__(self, response: object | None = None) -> None: + self.response = response + self.raise_timeout = False + + async def send(self, _request: object) -> object: + if self.raise_timeout: + raise ReadTimeoutError(endpoint_url="http://169.254.169.254") + return self.response + + class Acquisition: + def __init__(self, transport: Transport) -> None: + self.transport = transport + + async def __aenter__(self) -> Transport: + return self.transport + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def __init__(self, transport: Transport) -> None: + self.transport = transport + + def acquire(self) -> Acquisition: + return Acquisition(self.transport) + + forbidden = Transport(SimpleNamespace(status_code=403, text="forbidden")) + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + session=Pool(forbidden), # type: ignore[arg-type] + ) + assert await fetcher._fetch_metadata_token() is None # noqa: SLF001 + + timeout = Transport() + timeout.raise_timeout = True + fetcher._session = Pool(timeout) + with pytest.raises(fetcher._RETRIES_EXCEEDED_ERROR_CLS): # noqa: SLF001 + await fetcher._get_request( # noqa: SLF001 + "latest/meta-data/iam/security-credentials/workstream-role", + lambda _response: False, + token="imds-v2-token", + ) + + async def retries_exceeded() -> str: + raise fetcher._RETRIES_EXCEEDED_ERROR_CLS() # noqa: SLF001 + + fetcher._fetch_metadata_token = retries_exceeded # type: ignore[method-assign] + assert await fetcher.retrieve_iam_role_credentials() == {} + + fetcher._evaluate_expiration({}) # noqa: SLF001 + monkeypatch.setattr( + s3_compatible, + "get_current_datetime", + lambda: datetime.datetime(2026, 1, 1), + ) + monkeypatch.setattr(s3_compatible.random, "randint", lambda _start, _end: 120) + expiring = {"expiry_time": "2026-01-01T00:01:00Z"} + fetcher._evaluate_expiration(expiring) # noqa: SLF001 + assert expiring["expiry_time"] == "2026-01-01T00:12:00Z" + + class EmptyRoleFetcher: + async def retrieve_iam_role_credentials(self) -> dict[str, str]: + return {} + + provider = s3_compatible._SanitizedInstanceMetadataProvider( # noqa: SLF001 + iam_role_fetcher=EmptyRoleFetcher() + ) + assert await provider.load() is None From e86c3bf0f009c283acc282a3aa82f7818d653f0d Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 18:01:32 +0100 Subject: [PATCH 20/39] Validate every S3 bucket label --- backend/app/core/s3_validation.py | 7 ++++++- backend/tests/test_config.py | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/backend/app/core/s3_validation.py b/backend/app/core/s3_validation.py index 1b57d4a0..c19e0c28 100644 --- a/backend/app/core/s3_validation.py +++ b/backend/app/core/s3_validation.py @@ -9,6 +9,7 @@ _S3_REGION = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$") _S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") +_S3_BUCKET_LABEL = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$") _S3_PREFIX_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") _SHA256 = re.compile(r"^sha256:[0-9a-f]{64}$") @@ -22,7 +23,11 @@ def is_canonical_s3_bucket(value: object) -> bool: """Return whether one dedicated bucket name is DNS-compatible.""" if not isinstance(value, str) or _S3_BUCKET.fullmatch(value) is None: return False - return ".." not in value and re.fullmatch(r"\d+\.\d+\.\d+\.\d+", value) is None + return ( + ".." not in value + and re.fullmatch(r"\d+\.\d+\.\d+\.\d+", value) is None + and all(_S3_BUCKET_LABEL.fullmatch(label) is not None for label in value.split(".")) + ) def is_canonical_s3_prefix(value: object) -> bool: diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 529e53af..519fdcf8 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -647,6 +647,8 @@ def test_minio_settings_and_factory_construct_closed_namespace(tmp_path: Path) - ({"artifact_s3_region": "US-east-1"}, "requires a canonical region"), ({"artifact_s3_bucket": "192.168.0.1"}, "requires a canonical bucket"), ({"artifact_s3_bucket": "bucket..name"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket.-name"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket-.name"}, "requires a canonical bucket"), ({"artifact_s3_private_prefix": "/bad"}, "private prefix is invalid"), ({"artifact_maximum_bytes": 512 * 1024 * 1024 + 1}, "maximum exceeds"), ({"environment": "production"}, "MinIO artifact storage is restricted"), From 0a4832aec7dee5502375b7f3612f03c3ff1e586c Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 18:41:21 +0100 Subject: [PATCH 21/39] Harden MinIO errors and bucket names --- .../app/adapters/artifacts/s3_compatible.py | 78 ++++++++++++------- backend/app/core/s3_validation.py | 11 +++ backend/tests/test_config.py | 63 ++++++++++++++- 3 files changed, 122 insertions(+), 30 deletions(-) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index 3024e727..131078f5 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -892,40 +892,60 @@ def create_minio_artifact_store_bootstrap( settings: Settings, ) -> S3CompatibleArtifactStoreBootstrap: """Construct the only S3 runtime profile enabled by this chunk.""" + bootstrap, error_message = _try_create_minio_artifact_store_bootstrap(settings) + if error_message is not None: + del settings + raise ArtifactConfigurationError(error_message) from None + if bootstrap is None: + del settings + raise ArtifactConfigurationError("MinIO artifact configuration is invalid") from None + return bootstrap + + +def _try_create_minio_artifact_store_bootstrap( + settings: Settings, +) -> tuple[S3CompatibleArtifactStoreBootstrap | None, str | None]: + """Build MinIO state without propagating a secret-bearing failure frame.""" if settings.artifact_s3_provider_profile != "minio": - raise ArtifactConfigurationError("MinIO artifact profile is not configured") + return None, "MinIO artifact profile is not configured" access_key = settings.artifact_s3_access_key_id secret_key = settings.artifact_s3_secret_access_key if access_key is None or secret_key is None: - raise ArtifactConfigurationError("MinIO artifact credentials are unavailable") - session = AioSession() - session.set_credentials( - access_key.get_secret_value(), - secret_key.get_secret_value(), - settings.artifact_s3_session_token.get_secret_value() - if settings.artifact_s3_session_token is not None - else None, - ) - return S3CompatibleArtifactStoreBootstrap( - S3CompatibleArtifactStore( - provider_profile="minio", - region=_required(settings.artifact_s3_region), - endpoint_url=_required(settings.artifact_s3_endpoint_url), - bucket=_required(settings.artifact_s3_bucket), - private_prefix=settings.artifact_s3_private_prefix, - addressing_style=settings.artifact_s3_addressing_style, - session=session, - buffer_bytes=settings.artifact_stream_buffer_bytes, - connect_timeout_seconds=settings.artifact_s3_connect_timeout_seconds, - read_timeout_seconds=settings.artifact_s3_read_timeout_seconds, - write_timeout_seconds=settings.artifact_s3_write_timeout_seconds, - pool_timeout_seconds=settings.artifact_s3_pool_timeout_seconds, - operation_total_timeout_seconds=( - settings.artifact_s3_operation_total_timeout_seconds - ), - max_pool_connections=settings.artifact_s3_max_pool_connections, + return None, "MinIO artifact credentials are unavailable" + try: + session = AioSession() + session.set_credentials( + access_key.get_secret_value(), + secret_key.get_secret_value(), + settings.artifact_s3_session_token.get_secret_value() + if settings.artifact_s3_session_token is not None + else None, ) - ) + bootstrap = S3CompatibleArtifactStoreBootstrap( + S3CompatibleArtifactStore( + provider_profile="minio", + region=_required(settings.artifact_s3_region), + endpoint_url=_required(settings.artifact_s3_endpoint_url), + bucket=_required(settings.artifact_s3_bucket), + private_prefix=settings.artifact_s3_private_prefix, + addressing_style=settings.artifact_s3_addressing_style, + session=session, + buffer_bytes=settings.artifact_stream_buffer_bytes, + connect_timeout_seconds=settings.artifact_s3_connect_timeout_seconds, + read_timeout_seconds=settings.artifact_s3_read_timeout_seconds, + write_timeout_seconds=settings.artifact_s3_write_timeout_seconds, + pool_timeout_seconds=settings.artifact_s3_pool_timeout_seconds, + operation_total_timeout_seconds=( + settings.artifact_s3_operation_total_timeout_seconds + ), + max_pool_connections=settings.artifact_s3_max_pool_connections, + ) + ) + except ArtifactConfigurationError as error: + return None, str(error) + except Exception: + return None, "MinIO artifact configuration is invalid" + return bootstrap, None def validate_aws_workload_identity_environment( diff --git a/backend/app/core/s3_validation.py b/backend/app/core/s3_validation.py index c19e0c28..ed372721 100644 --- a/backend/app/core/s3_validation.py +++ b/backend/app/core/s3_validation.py @@ -10,6 +10,15 @@ _S3_REGION = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$") _S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") _S3_BUCKET_LABEL = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$") +_S3_RESERVED_PREFIXES = ("xn--", "sthree-", "amzn-s3-demo-") +_S3_RESERVED_SUFFIXES = ( + "-s3alias", + "--ol-s3", + ".mrap", + "--x-s3", + "--table-s3", + "-an", +) _S3_PREFIX_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") _SHA256 = re.compile(r"^sha256:[0-9a-f]{64}$") @@ -27,6 +36,8 @@ def is_canonical_s3_bucket(value: object) -> bool: ".." not in value and re.fullmatch(r"\d+\.\d+\.\d+\.\d+", value) is None and all(_S3_BUCKET_LABEL.fullmatch(label) is not None for label in value.split(".")) + and not value.startswith(_S3_RESERVED_PREFIXES) + and not value.endswith(_S3_RESERVED_SUFFIXES) ) diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 519fdcf8..53af450b 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -13,6 +13,7 @@ from pydantic_settings import BaseSettings from app.adapters.artifacts import create_artifact_store_bootstrap +from app.adapters.artifacts.s3_compatible import create_minio_artifact_store_bootstrap from app.adapters.auth.flow import FlowAuthVerifier from app.api.deps.auth import get_application_auth_verifier from app.core.auth import clear_auth_verifier_cache, get_auth_verifier @@ -22,7 +23,10 @@ is_canonical_s3_prefix, validate_s3_namespace_descriptor, ) -from app.interfaces.artifacts import ArtifactProviderLiveProofRequiredError +from app.interfaces.artifacts import ( + ArtifactConfigurationError, + ArtifactProviderLiveProofRequiredError, +) from app.interfaces.external_services import ( ExternalServiceConfigurationError, UnknownExternalServiceProviderError, @@ -649,6 +653,15 @@ def test_minio_settings_and_factory_construct_closed_namespace(tmp_path: Path) - ({"artifact_s3_bucket": "bucket..name"}, "requires a canonical bucket"), ({"artifact_s3_bucket": "bucket.-name"}, "requires a canonical bucket"), ({"artifact_s3_bucket": "bucket-.name"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "xn--bucket"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "sthree-bucket"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "amzn-s3-demo-bucket"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket-s3alias"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket--ol-s3"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket.mrap"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket--x-s3"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket--table-s3"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket-an"}, "requires a canonical bucket"), ({"artifact_s3_private_prefix": "/bad"}, "private prefix is invalid"), ({"artifact_maximum_bytes": 512 * 1024 * 1024 + 1}, "maximum exceeds"), ({"environment": "production"}, "MinIO artifact storage is restricted"), @@ -671,6 +684,54 @@ def test_minio_settings_fail_closed( _minio_settings(tmp_path, **overrides) +def test_mutated_minio_settings_cannot_retain_credentials_in_constructor_error( + tmp_path: Path, +) -> None: + access_key = "minio-mutated-access-key" + secret_key = "minio-mutated-secret-key" + settings = _minio_settings( + tmp_path, + artifact_s3_access_key_id=access_key, + artifact_s3_secret_access_key=secret_key, + ) + settings.artifact_s3_bucket = "bucket.-invalid" + + with pytest.raises(ArtifactConfigurationError) as caught: + create_minio_artifact_store_bootstrap(settings) + + assert caught.value.__cause__ is None + assert caught.value.__context__ is None + assert_secret_not_retained( + caught.value, + access_key, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) + assert_secret_not_retained( + caught.value, + secret_key, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) + + with pytest.raises(ExternalServiceConfigurationError) as public_error: + create_artifact_store_bootstrap(settings) + assert_secret_not_retained( + public_error.value, + access_key, + traceback_module_prefixes=( + "app.adapters.artifacts.s3_compatible", + "app.interfaces.external_services", + ), + ) + assert_secret_not_retained( + public_error.value, + secret_key, + traceback_module_prefixes=( + "app.adapters.artifacts.s3_compatible", + "app.interfaces.external_services", + ), + ) + + @pytest.mark.parametrize( ("removed", "message"), [ From 4d6bf155c1990604aa516830be575c882e7627cb Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 19:18:32 +0100 Subject: [PATCH 22/39] Sanitize endpoints and canonicalize regions --- ...WS-ART-001-02B1-s3-compatible-minio-aws.md | 3 + backend/app/core/config.py | 53 ++++++++++- backend/app/core/s3_validation.py | 32 +++++-- backend/tests/test_config.py | 91 +++++++++++++++++++ docs/spec_artifact_storage_service.md | 5 + 5 files changed, 172 insertions(+), 12 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index 903dec1d..2fd08355 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -54,6 +54,9 @@ credential-delivery contract. - ETag/provider checksums are not Workstream integrity facts; - native AWS endpoint omission with explicit region and MinIO endpoint/region validate; production rejects HTTP/local endpoints and static credentials; +- the v0.1 AWS profile accepts global general-purpose buckets only and rejects + account-regional bucket names ending in `-an` because Workstream does not + emit the account-regional namespace request header; - AWS production credential mode `aws_workload_identity` selects exactly one of `assume-role-with-web-identity`, `container-role`, or `iam-role`; the credential resolver contains only that provider and the resolved method must diff --git a/backend/app/core/config.py b/backend/app/core/config.py index a3586121..471ebad1 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -36,6 +36,9 @@ "artifact_s3_session_token", } ) +_ARTIFACT_S3_SENSITIVE_INPUT_FIELDS = _ARTIFACT_S3_SECRET_FIELDS | { + "artifact_s3_endpoint_url" +} _EMPTY_ARTIFACT_S3_SECRETS: tuple[SecretStr | None, SecretStr | None, SecretStr | None] = ( None, None, @@ -201,6 +204,7 @@ def __init__(self, **values: object) -> None: secret: SecretStr | None = None s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS try: + _normalize_artifact_s3_endpoint_input(values) secret = _extract_api_rate_limit_key_secret(values) s3_secrets = _extract_artifact_s3_static_secrets(values) super().__init__(**values) @@ -220,7 +224,7 @@ def model_validate(cls, obj: object, **kwargs: object) -> Self: """Sanitize secret-bearing mappings before Pydantic retains input.""" if isinstance(obj, Mapping) and ( "api_rate_limit_key_secret" in obj - or _ARTIFACT_S3_SECRET_FIELDS.intersection(obj) + or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(obj) ): sanitized = dict(obj) supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) @@ -229,6 +233,7 @@ def model_validate(cls, obj: object, **kwargs: object) -> Self: s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS settings: Settings | None = None try: + _normalize_artifact_s3_endpoint_input(sanitized) secret = ( _extract_api_rate_limit_key_secret(sanitized) if "api_rate_limit_key_secret" in sanitized @@ -277,7 +282,7 @@ def model_validate_json( raise ValueError("invalid settings JSON") if isinstance(parsed, Mapping) and ( "api_rate_limit_key_secret" in parsed - or _ARTIFACT_S3_SECRET_FIELDS.intersection(parsed) + or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(parsed) ): try: return cls.model_validate(parsed, **kwargs) @@ -292,7 +297,7 @@ def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: """Sanitize string-mapping secret input before structured validation.""" if isinstance(obj, Mapping) and ( "api_rate_limit_key_secret" in obj - or _ARTIFACT_S3_SECRET_FIELDS.intersection(obj) + or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(obj) ): sanitized = dict(obj) supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) @@ -301,6 +306,7 @@ def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS settings: Settings | None = None try: + _normalize_artifact_s3_endpoint_input(sanitized) secret = ( _extract_api_rate_limit_key_secret(sanitized) if "api_rate_limit_key_secret" in sanitized @@ -494,7 +500,10 @@ def get_settings() -> Settings: def _clear_settings_private_secrets(settings: object) -> None: - """Remove secret values from a partially or fully initialized Settings object.""" + """Remove sensitive values from a partially or fully initialized Settings object.""" + settings_values = getattr(settings, "__dict__", None) + if isinstance(settings_values, dict): + settings_values["artifact_s3_endpoint_url"] = None private_values = getattr(settings, "__pydantic_private__", None) if not isinstance(private_values, dict): return @@ -504,6 +513,42 @@ def _clear_settings_private_secrets(settings: object) -> None: private_values["_artifact_s3_session_token"] = None +def _normalize_artifact_s3_endpoint_input(values: dict[str, object]) -> None: + """Replace a raw endpoint source with its noncredentialed canonical origin.""" + raw_value = values.pop("artifact_s3_endpoint_url", _MISSING_SECRET) + if raw_value is _MISSING_SECRET: + raw_value = os.environ.get( + "WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL", + _MISSING_SECRET, + ) + if raw_value is _MISSING_SECRET: + env_file = values.get("_env_file", ".env") + if env_file is not None: + env_encoding = values.get("_env_file_encoding", "utf-8") + env_files = ( + (env_file,) + if isinstance(env_file, (str, os.PathLike)) + else tuple(env_file) + ) + for path in env_files: + raw_value = dotenv_values(path, encoding=env_encoding).get( + "WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL", + raw_value, + ) + if raw_value is _MISSING_SECRET: + return + if raw_value is None: + values["artifact_s3_endpoint_url"] = None + return + try: + endpoint = canonical_minio_endpoint(raw_value) + except ValueError: + raw_value = None + raise ValueError("MinIO artifact storage endpoint is invalid") from None + values["artifact_s3_endpoint_url"] = endpoint + raw_value = None + + def _extract_api_rate_limit_key_secret(values: dict[str, object]) -> SecretStr | None: """Resolve and validate the key without entering Pydantic's input graph.""" raw_value = _pop_optional_secret_source( diff --git a/backend/app/core/s3_validation.py b/backend/app/core/s3_validation.py index ed372721..60432991 100644 --- a/backend/app/core/s3_validation.py +++ b/backend/app/core/s3_validation.py @@ -7,7 +7,7 @@ from urllib.parse import urlsplit -_S3_REGION = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$") +_S3_REGION = re.compile(r"^[a-z]{2,4}(?:-[a-z0-9]+)+-[0-9]+$") _S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") _S3_BUCKET_LABEL = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$") _S3_RESERVED_PREFIXES = ("xn--", "sthree-", "amzn-s3-demo-") @@ -25,11 +25,15 @@ def is_canonical_s3_region(value: object) -> bool: """Return whether one region is bounded and canonical.""" - return isinstance(value, str) and _S3_REGION.fullmatch(value) is not None + return ( + isinstance(value, str) + and len(value) <= 63 + and _S3_REGION.fullmatch(value) is not None + ) def is_canonical_s3_bucket(value: object) -> bool: - """Return whether one dedicated bucket name is DNS-compatible.""" + """Return whether one global general-purpose bucket is DNS-compatible.""" if not isinstance(value, str) or _S3_BUCKET.fullmatch(value) is None: return False return ( @@ -53,13 +57,25 @@ def is_canonical_s3_prefix(value: object) -> bool: def canonical_minio_endpoint(value: object) -> str: """Return one normalized noncredentialed HTTP(S) MinIO origin.""" + missing = value is None or value == "" + endpoint = _try_canonical_minio_endpoint(value) + del value + if endpoint is None: + if missing: + raise ValueError("MinIO artifact storage requires an endpoint") from None + raise ValueError("MinIO artifact storage endpoint is invalid") from None + return endpoint + + +def _try_canonical_minio_endpoint(value: object) -> str | None: + """Normalize one endpoint without retaining invalid input in an exception.""" if not isinstance(value, str) or not value or len(value) > 2048: - raise ValueError("MinIO artifact storage requires an endpoint") - parsed = urlsplit(value) + return None try: + parsed = urlsplit(value) port = parsed.port - except ValueError as error: - raise ValueError("MinIO artifact storage endpoint is invalid") from error + except (UnicodeError, ValueError): + return None if ( parsed.scheme.lower() not in {"http", "https"} or not parsed.hostname @@ -70,7 +86,7 @@ def canonical_minio_endpoint(value: object) -> str: or parsed.path not in {"", "/"} or parsed.hostname.endswith(".") ): - raise ValueError("MinIO artifact storage endpoint is invalid") + return None scheme = parsed.scheme.lower() hostname = parsed.hostname.lower() if ":" in hostname: diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 53af450b..8440ee9f 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -20,6 +20,7 @@ from app.core.config import Settings, get_settings from app.core.s3_validation import ( canonical_minio_endpoint, + is_canonical_s3_region, is_canonical_s3_prefix, validate_s3_namespace_descriptor, ) @@ -649,6 +650,9 @@ def test_minio_settings_and_factory_construct_closed_namespace(tmp_path: Path) - [ ({"artifact_s3_provider_profile": None}, "requires a provider profile"), ({"artifact_s3_region": "US-east-1"}, "requires a canonical region"), + ({"artifact_s3_region": "us-east-1-"}, "requires a canonical region"), + ({"artifact_s3_region": "1-east-1"}, "requires a canonical region"), + ({"artifact_s3_region": "a-east-1"}, "requires a canonical region"), ({"artifact_s3_bucket": "192.168.0.1"}, "requires a canonical bucket"), ({"artifact_s3_bucket": "bucket..name"}, "requires a canonical bucket"), ({"artifact_s3_bucket": "bucket.-name"}, "requires a canonical bucket"), @@ -1053,6 +1057,93 @@ def test_minio_endpoint_canonicalization_handles_ipv6_and_invalid_ports() -> Non canonical_minio_endpoint("http://localhost:not-a-port") +@pytest.mark.parametrize( + "region", + ["us-east-1", "us-gov-west-1", "eusc-de-east-1"], +) +def test_s3_region_validation_accepts_canonical_region_shapes(region: str) -> None: + assert is_canonical_s3_region(region) is True + + +@pytest.mark.parametrize( + "endpoint", + [ + "http://user:endpoint-secret@localhost:9000", + "http://localhost:endpoint-secret", + ], +) +def test_invalid_minio_endpoint_secrets_do_not_survive_errors( + tmp_path: Path, + endpoint: str, +) -> None: + secret = "endpoint-secret" + + with pytest.raises(ValueError) as endpoint_error: + canonical_minio_endpoint(endpoint) + assert_secret_not_retained( + endpoint_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_endpoint_url=endpoint, + ) + for method_name in ( + "constructor", + "model_validate", + "model_validate_json", + "model_validate_strings", + ): + with pytest.raises((ValidationError, ValueError)) as settings_error: + if method_name == "constructor": + Settings(**payload) + elif method_name == "model_validate_json": + Settings.model_validate_json(json.dumps(payload)) + else: + getattr(Settings, method_name)(payload) + assert_secret_not_retained( + settings_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +def test_invalid_minio_endpoint_secrets_from_env_and_dotenv_do_not_survive_errors( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + secret = "endpoint-source-secret" + endpoint = f"http://user:{secret}@localhost:9000" + payload = _minio_setting_values(tmp_path) + payload.pop("artifact_s3_endpoint_url") + monkeypatch.setenv("WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL", endpoint) + + with pytest.raises(ValueError) as env_error: + Settings(**payload) + assert_secret_not_retained( + env_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL") + env_file = tmp_path / ".env" + env_file.write_text( + f"WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL={endpoint}\n", + encoding="utf-8", + ) + with pytest.raises(ValueError) as dotenv_error: + Settings(**payload, _env_file=env_file) + assert_secret_not_retained( + dotenv_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + @pytest.mark.parametrize( ("profile", "changes", "message"), [ diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index bdbcb6d9..11d724f4 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -426,6 +426,11 @@ its only v0.1 production provider. MinIO proves the same protocol contract in local integration tests and CI. Cloudflare R2 is deferred and has no active runtime profile or credential contract. +The v0.1 AWS profile supports global general-purpose buckets only. Workstream +does not emit the account-regional namespace request header, so account-regional +bucket names ending in `-an` fail the canonical bucket gate rather than being +accepted with incomplete request semantics. + Configuration includes: - provider profile `aws_s3` or `minio`; From 148510a76ca416895dfdf20420a58b2717f56aa5 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 19:46:49 +0100 Subject: [PATCH 23/39] Refresh artifact integration basis --- .agent-loop/LOOP_STATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index e7e204af..40977bca 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -24,7 +24,7 @@ editing or activating their independently owned runtime behavior. - AUTH-09C merged through PR #146 as `0ffdabf`; AUTH-09D remains inactive until an explicit human start. -- ART integration basis: trusted `main` at `b2b9016` after PR #145. +- ART integration basis: trusted `main` at `f18b620` after PR #147. - PR #119 merged `WS-AUTH-001-05B` as `ad71c7e`. - PR #120 merged `WS-ART-001-OBJECT-STORAGE-AMENDMENT` as `4408256`. - PR #122 merged the first automated post-merge memory implementation as From 626653e1222c482cf052ed2db25e94cb9ff6b4a7 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 20:56:17 +0100 Subject: [PATCH 24/39] Record S3 artifact storage review evidence --- .../STATUS.md | 13 +- ...S-ART-001-02B1-internal-review-evidence.md | 116 ++++++++++++++++++ .../WS-ART-001-02B1-pr-trust-bundle.md | 105 ++++++++++++++++ 3 files changed, 229 insertions(+), 5 deletions(-) create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index 63e8197a..c331dafb 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -14,6 +14,8 @@ The planning-only cross-initiative boundary reconciliation merged through PR #139 as `5d353b6`, and AUTH's owner reconciliation merged through PR #140 as `d541521`. ART now consumes AUTH's canonical activation-custody and prepared mutation contracts without editing or activating AUTH runtime behavior. +AUTH-09D-A merged through PR #148 as `99ae4c9` and is integrated into the ART +candidate; AUTH-09D-B remains inactive. The Flow Node-focused amendment candidate `6cc422d` passed deterministic checks but failed internal review on recovery/API completeness. Before repair, the user @@ -50,8 +52,9 @@ has a v0.1 chunk. ## Gate -The current gate is deterministic 02B1 proof followed by all nine exact-SHA -internal reviewer tracks. GitHub Actions, CodeRabbit, and explicit human review -follow only after that evidence passes. Durable admission, put attempts, -verification publication, and recovery remain in later owning chunks. No later -artifact chunk starts automatically, and only the user may approve merge. +The reviewed implementation recorded: "The current gate is deterministic 02B1 +proof followed by all nine exact-SHA internal reviewer tracks." That gate passed +at `9cd41ab`. The remaining gate is GitHub Actions, CodeRabbit, and explicit +human review. Durable admission, put attempts, verification publication, and +recovery remain in later owning chunks. No later artifact chunk starts +automatically, and only the user may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md new file mode 100644 index 00000000..a783651d --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md @@ -0,0 +1,116 @@ +# Internal Review Evidence: WS-ART-001-02B1 + +## Chunk + +`WS-ART-001-02B1`: S3-Compatible MinIO And AWS + +open sub-agent sessions: none + +valid findings addressed: yes + +## Reviewed Revision + +Reviewed code SHA: `9cd41ab907aba323f1fbd79dac8bbe1602bcc30f` + +Reviewed at: 2026-07-18T19:49:08Z + +Reviewer run IDs: senior-engineering=019f7698-ca9c-75a0-9ad9-5ddb553144ee; architecture=019f7698-d670-7b02-9336-93111bfb0905; QA/test=019f7698-ea1c-7632-8c76-3b2bddc1a709; security/auth=019f76b8-a76d-7060-95a2-e3296813464e; product/ops=019f76b8-acef-7cc1-a7ba-24f9262521de; reuse/dedup=019f76b8-b3b7-7961-85d9-5dfd9cddaaec; CI-integrity=019f76be-436c-7473-92da-50ee58380fd4; test-delta=019f76be-4869-7350-a737-fa75a6123fd0; docs=019f76be-52ed-7cf1-b589-aa9dc1ddd1bb + +The reviewed base is `origin/main` at +`99ae4c963e53f317175dcb308b9e47c93ccf19ed`, including AUTH PR #148. +Only review artifacts and initiative status may change after the reviewed SHA. +Any implementation, test, workflow, policy, or chunk-contract change invalidates +this evidence and requires a new exact-head review cycle. + +## Reviewed Change + +- Added one S3-protocol ArtifactStore implementation shared by MinIO and the + inactive native AWS profile. +- Proved the active local/CI path against real digest-pinned MinIO with the same + ArtifactStore v2 contract used by LocalStorage. +- Kept native AWS runtime-ineligible until Chunk 07 supplies an immutable live + activation proof. +- Isolated each AWS workload-identity method from ambient credential, proxy, + file, process, cache, and unselected metadata sources. +- Preserved server-derived content keys, sealed source validation, exact replay, + bounded object size, no multipart/list/delete/copy surface, and sanitized + provider failures. +- Added canonical global general-purpose bucket, region, prefix, and MinIO + endpoint validation, including rejection of secret-bearing endpoint input. +- Retained the full repository 78 percent CI floor and every cumulative focused + 90 percent gate while adding exact S3 adapter and validation gates. +- Integrated AUTH PR #148 without editing or activating AUTH-owned runtime + behavior. + +## Reviewer Results + +| Reviewer | Result | Blocking findings | Notes | +|---|---:|---|---| +| senior engineering | PASS | None | The shared adapter remains maintainable, AWS remains unavailable before live proof, and no operational issue remains. | +| qa/test | PASS | None | Real MinIO, credential isolation, namespace fencing, configuration, and CI evidence cover the chunk contract without weakened tests. | +| security/auth | PASS | None | Secret-bearing endpoint and metadata failures are sanitized, credential sources are closed, and ART/AUTH ownership remains intact. | +| product/ops | PASS | None | No operator recovery, contributor, reviewer, contribution, compensation, fulfillment, or reputation behavior is activated. | +| architecture | PASS | None | One typed adapter path remains, concrete S3 construction stays in ART composition, and product services receive no concrete store. | +| ci integrity | PASS | None | The 78 percent repository gate and all cumulative 90 percent gates remain fail closed with no bypass. | +| docs | PASS | None | README, artifact specification, settings, provider eligibility, and loop state match the implementation. | +| reuse/dedup | PASS | None | Provider references, S3 validation, runtime guards, and deep secret assertions reuse canonical helpers without parallel implementations. | +| test delta | PASS | None | Removed placeholder assertions were replaced by stronger active-MinIO and inactive-AWS behavior proof; no test was skipped or weakened. | + +## Valid Findings Addressed + +- Replaced permissive S3 region validation with a bounded canonical region + grammar and complete positive/negative tests. +- Made invalid MinIO endpoint errors discard userinfo, malformed ports, and raw + source values across constructor, Pydantic mapping/JSON/string validation, + environment, dotenv, traceback, and structured error paths. +- Documented the intentional v0.1 global general-purpose bucket boundary and + rejection of account-regional `-an` names because Workstream does not emit + the account-regional namespace header. +- Rejected every AWS reserved general-purpose bucket prefix and suffix and + validated every dotted bucket label. +- Sanitized ECS and IMDS credential failure paths so SDK exceptions cannot + retain response bodies, authorization tokens, endpoints, or credentials. +- Bound replay to the sealed committed source before any provider operation and + removed stale provider-reference divergence. +- Reconciled authored ART loop state and work queue with AUTH PR #148 while + keeping AUTH-09D-B and every later ART chunk inactive. + +## Commands Run + +```bash +docker compose up -d --wait postgres redis minio +cd backend && .venv/bin/ruff check app tests +cd backend && .venv/bin/pip check +cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test WORKSTREAM_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/python -m pytest tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_preparation.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py tests/test_assertion_helpers.py tests/test_audit.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov=app.core.s3_validation --cov-report=term-missing --cov-fail-under=90 +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_stale_authorization_docs.py +python3 scripts/check_stale_review_contracts.py +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +Results: 428 real-service focused tests passed after AUTH PR #148 integration. +`S3CompatibleArtifactStore` coverage is 92 percent, S3 validation coverage is +100 percent, and combined changed-subsystem coverage is 92.95 percent. Ruff, +dependency integrity, stale contract/authorization/review/wording scans, +Markdown links, 87 agent-gate tests, and diff checks passed. + +The isolated full repository suite and 78 percent whole-app floor remain +authoritative in GitHub Backend CI, as explicitly chosen because local execution +is prohibitively slow. The CI reviewer confirmed that exact gate is retained. + +## Remaining Risks + +- Native AWS operations remain deliberately unavailable until Chunk 07 adds the + release-bound live proof and production composition guard. +- MinIO is a local/CI protocol proof, not production activation evidence. +- Provider acknowledgement remains non-bindable pending later admission, + verification, publication, and recovery chunks. + +## Stop Condition + +Publish this evidence-bound candidate for GitHub CI, CodeRabbit, and explicit +human review. Do not merge without the user's approval and do not start +`WS-ART-001-02C1` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md new file mode 100644 index 00000000..ee484c63 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md @@ -0,0 +1,105 @@ +# PR Trust Bundle: WS-ART-001-02B1 + +## Chunk + +`WS-ART-001-02B1` - S3-Compatible MinIO And AWS + +Merge intent: `.agent-loop/merge-intents/WS-ART-001-02B1.json` + +## Goal + +Add one provider-neutral S3 protocol implementation, prove it against real +MinIO for local/CI use, and validate an isolated native AWS configuration +without making AWS runtime-eligible before its separately owned live-proof +chunk. + +## What Changed + +- Added `S3CompatibleArtifactStore` behind the existing ArtifactStore v2 and + typed external-service factory boundaries. +- Added real digest-pinned MinIO to local/CI composition and shared conformance. +- Added closed AWS workload-identity configuration and isolation proof while + preserving the stable pre-I/O `artifact_provider_live_proof_required` guard. +- Added canonical S3 namespace, region, bucket, prefix, and endpoint validation. +- Added sanitized credential/metadata/endpoint failure paths and deep + traceback/object-graph tests. +- Added exact async SDK pins and cumulative CI coverage guards. +- Integrated AUTH PR #148 while leaving all AUTH runtime activation with AUTH. + +## Scope Control + +The chunk adds no product route, upload admission, put-attempt state, +verification publication, recovery operation, presigned URL, multipart upload, +list/delete/copy capability, Flow Node path, R2 profile, or authorization +activation. `WS-ART-001-02C1` remains inactive until this chunk merges and the +user explicitly starts it. + +## Acceptance Proof + +- [x] Server-derived sealed commitment controls every provider object key. +- [x] Exact replay and conditional no-overwrite semantics are deterministic. +- [x] Object size fails before provider I/O above 512 MiB. +- [x] Real LocalStorage and MinIO pass the same ArtifactStore v2 vectors. +- [x] AWS credential resolution is limited to one explicitly selected workload + source with ambient providers, proxies, and behavior controls rejected. +- [x] Native AWS fails before factory, credentials, namespace, scratch, or I/O. +- [x] ART/AUTH ownership remains separated after AUTH PR #148 integration. +- [x] Exactly one schema-v2 merge intent names only the same-initiative 02C1 + successor and requires a separate explicit start. + +## Tests And Checks + +```text +428 real-service focused tests PASS +S3CompatibleArtifactStore coverage 92% +S3 validation coverage 100% +Combined changed-subsystem coverage 92.95% +Ruff PASS +pip check PASS +87 agent-gate tests PASS +Stale artifact/authorization/review/Workstream wording PASS +Markdown links PASS +git diff --check PASS +``` + +GitHub Backend CI remains authoritative for the isolated full suite, the 78 +percent repository floor, and all cumulative focused 90 percent gates. + +## Internal Review + +Reviewed code SHA: `9cd41ab907aba323f1fbd79dac8bbe1602bcc30f` + +All nine required reviewer tracks passed the exact integrated SHA with no +remaining finding. Every reviewer session is closed. Reviewer IDs and the +addressed repair history are recorded in the internal review evidence. + +## External Review + +| Source | Status | Notes | +|---|---:|---| +| GitHub Actions | Pending | Runs the isolated full suite, repository floor, cumulative focused gates, and real MinIO proof. | +| CodeRabbit | Pending | External review starts after publication. | +| Human review | Pending | Only the user may approve merge. | + +## Remaining Risks + +- AWS is configuration-ready but intentionally unavailable until Chunk 07. +- MinIO proves protocol behavior only for local and CI environments. +- Durable admission, verification, publication, and recovery are later chunks. + +## Human Review Focus + +- Is there truly one S3 protocol adapter rather than provider-specific forks? +- Can untrusted input influence object identity, credential sources, endpoints, + proxies, or namespace selection? +- Does native AWS fail before every credential and provider side effect? +- Are ART and AUTH responsibilities still separate after PR #148 integration? +- Are all later ART capabilities still inactive? + +## Human Merge Ownership + +- [ ] I can explain what changed and why. +- [ ] I know what could break. +- [ ] I accept the remaining risks. +- [ ] GitHub CI and external review pass. +- [ ] The user explicitly approved this PR for merge. From 38e81b26940d0a719c9fc9375932e525fd7752ee Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 21:02:35 +0100 Subject: [PATCH 25/39] Fix artifact successor contract heading --- .../chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md index a60655b8..75b25d2e 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md @@ -1,4 +1,4 @@ -# Chunk Contract: WS-ART-001-02C1 Admission And Put-Attempt Foundation +# Chunk Contract: WS-ART-001-02C1 - Admission And Put-Attempt Foundation Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02B1 From 799a84febf639a4b2a53065a2ce9c115946e29be Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 21:19:32 +0100 Subject: [PATCH 26/39] Deduplicate settings secret validation --- backend/app/core/config.py | 123 ++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 69 deletions(-) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 471ebad1..aa4f2440 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -7,7 +7,7 @@ from contextvars import ContextVar import json import os -from collections.abc import Mapping +from collections.abc import Callable, Mapping from functools import lru_cache from pathlib import Path from typing import Literal, Self @@ -227,41 +227,12 @@ def model_validate(cls, obj: object, **kwargs: object) -> Self: or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(obj) ): sanitized = dict(obj) - supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) obj = None - secret: SecretStr | None = None - s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS - settings: Settings | None = None - try: - _normalize_artifact_s3_endpoint_input(sanitized) - secret = ( - _extract_api_rate_limit_key_secret(sanitized) - if "api_rate_limit_key_secret" in sanitized - else None - ) - s3_secrets = _extract_artifact_s3_static_secrets(sanitized) - if secret is not None: - sanitized["api_rate_limit_key_secret"] = None - for field_name in supplied_s3_fields: - sanitized[field_name] = None - restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) - try: - settings = super().model_validate(sanitized, **kwargs) - finally: - _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) - if secret is not None: - settings._api_rate_limit_key_secret = secret - settings._set_artifact_s3_static_secrets(s3_secrets) - settings._validate_artifact_s3_secret_contract() - return settings - except ValueError: - if settings is not None: - _clear_settings_private_secrets(settings) - sanitized.clear() - secret = None - s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS - settings = None - raise + return cls._validate_secret_mapping( + sanitized, + super().model_validate, + kwargs, + ) return super().model_validate(obj, **kwargs) @classmethod @@ -300,43 +271,57 @@ def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(obj) ): sanitized = dict(obj) - supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) obj = None - secret: SecretStr | None = None - s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS - settings: Settings | None = None - try: - _normalize_artifact_s3_endpoint_input(sanitized) - secret = ( - _extract_api_rate_limit_key_secret(sanitized) - if "api_rate_limit_key_secret" in sanitized - else None - ) - s3_secrets = _extract_artifact_s3_static_secrets(sanitized) - if secret is not None: - sanitized["api_rate_limit_key_secret"] = None - for field_name in supplied_s3_fields: - sanitized[field_name] = None - restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) - try: - settings = super().model_validate_strings(sanitized, **kwargs) - finally: - _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) - if secret is not None: - settings._api_rate_limit_key_secret = secret - settings._set_artifact_s3_static_secrets(s3_secrets) - settings._validate_artifact_s3_secret_contract() - return settings - except ValueError: - if settings is not None: - _clear_settings_private_secrets(settings) - sanitized.clear() - secret = None - s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS - settings = None - raise + return cls._validate_secret_mapping( + sanitized, + super().model_validate_strings, + kwargs, + ) return super().model_validate_strings(obj, **kwargs) + @classmethod + def _validate_secret_mapping( + cls, + sanitized: dict[str, object], + validator: Callable[..., Self], + kwargs: Mapping[str, object], + ) -> Self: + """Validate a mapping without retaining supplied secret material.""" + supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) + secret: SecretStr | None = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings: Settings | None = None + try: + _normalize_artifact_s3_endpoint_input(sanitized) + secret = ( + _extract_api_rate_limit_key_secret(sanitized) + if "api_rate_limit_key_secret" in sanitized + else None + ) + s3_secrets = _extract_artifact_s3_static_secrets(sanitized) + if secret is not None: + sanitized["api_rate_limit_key_secret"] = None + for field_name in supplied_s3_fields: + sanitized[field_name] = None + restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + try: + settings = validator(sanitized, **kwargs) + finally: + _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) + if secret is not None: + settings._api_rate_limit_key_secret = secret + settings._set_artifact_s3_static_secrets(s3_secrets) + settings._validate_artifact_s3_secret_contract() + return settings + except ValueError: + if settings is not None: + _clear_settings_private_secrets(settings) + sanitized.clear() + secret = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings = None + raise + @classmethod def settings_customise_sources( cls, From 16fd792351a3ce7179b62539a6c004c51cb945ef Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 21:32:51 +0100 Subject: [PATCH 27/39] Clear settings secrets on unexpected errors --- backend/app/core/config.py | 6 ++-- backend/tests/test_config.py | 54 +++++++++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 4 deletions(-) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index aa4f2440..ea58dd72 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -212,7 +212,7 @@ def __init__(self, **values: object) -> None: self._set_artifact_s3_static_secrets(s3_secrets) if not _ALTERNATE_VALIDATION_RESTORES_SECRETS.get(): self._validate_artifact_s3_secret_contract() - except ValueError: + except Exception: _clear_settings_private_secrets(self) values.clear() secret = None @@ -257,7 +257,7 @@ def model_validate_json( ): try: return cls.model_validate(parsed, **kwargs) - except ValueError: + except Exception: parsed = None json_data = b"" raise @@ -313,7 +313,7 @@ def _validate_secret_mapping( settings._set_artifact_s3_static_secrets(s3_secrets) settings._validate_artifact_s3_secret_contract() return settings - except ValueError: + except Exception: if settings is not None: _clear_settings_private_secrets(settings) sanitized.clear() diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 8440ee9f..b6e42b3c 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -225,7 +225,7 @@ def capture_input( obj: object, **kwargs: object, ) -> BaseSettings: - observed["input"] = obj + observed["input"] = dict(obj) if isinstance(obj, Mapping) else obj raise PydanticBoundaryReached monkeypatch.setattr(BaseSettings, base_method_name, classmethod(capture_input)) @@ -838,6 +838,58 @@ def test_invalid_minio_secret_is_absent_from_application_traceback( ) +@pytest.mark.parametrize( + ("method_name", "delegated_method", "error_type"), + [ + ("constructor", "__init__", RuntimeError), + ("model_validate", "model_validate", RuntimeError), + ("model_validate_json", "model_validate", TypeError), + ("model_validate_strings", "model_validate_strings", TypeError), + ], +) +def test_unexpected_validation_errors_do_not_retain_minio_secrets( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + method_name: str, + delegated_method: str, + error_type: type[Exception], +) -> None: + """Unexpected delegated errors must still clear every credential local.""" + secret = "minio-secret-from-unexpected-validator-error" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + ) + + def fail_validation(*args: object, **kwargs: object) -> None: + del args, kwargs + raise error_type("forced validation failure") + + if delegated_method == "__init__": + monkeypatch.setattr(BaseSettings, delegated_method, fail_validation) + else: + monkeypatch.setattr( + BaseSettings, + delegated_method, + classmethod(fail_validation), + ) + + with pytest.raises(error_type, match="^forced validation failure$") as caught: + if method_name == "constructor": + Settings(**payload) + elif method_name == "model_validate_json": + Settings.model_validate_json(json.dumps(payload)) + else: + getattr(Settings, method_name)(payload) + + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + @pytest.mark.parametrize( "method_name", ["model_validate", "model_validate_json", "model_validate_strings"], From 126b685ebda225eac328f52fc87cbfb8e6307626 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 21:44:41 +0100 Subject: [PATCH 28/39] Reject secret-bearing invalid settings JSON --- backend/app/core/config.py | 11 ++++++++++- backend/tests/test_config.py | 31 +++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index ea58dd72..dfadbcf5 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -251,6 +251,10 @@ def model_validate_json( if malformed: json_data = b"" raise ValueError("invalid settings JSON") + if not isinstance(parsed, Mapping): + parsed = None + json_data = b"" + raise ValueError("invalid settings JSON") if isinstance(parsed, Mapping) and ( "api_rate_limit_key_secret" in parsed or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(parsed) @@ -261,7 +265,12 @@ def model_validate_json( parsed = None json_data = b"" raise - return super().model_validate_json(json_data, **kwargs) + try: + return super().model_validate_json(json_data, **kwargs) + except Exception: + parsed = None + json_data = b"" + raise @classmethod def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index b6e42b3c..8f806a93 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -890,6 +890,37 @@ def fail_validation(*args: object, **kwargs: object) -> None: ) +@pytest.mark.parametrize( + ("field_name", "field_value"), + [ + ("artifact_s3_access_key_id", "nested-json-secret"), + ("artifact_s3_secret_access_key", "nested-json-secret"), + ("artifact_s3_session_token", "nested-json-secret"), + ( + "artifact_s3_endpoint_url", + "http://user:nested-json-secret@localhost:9000", + ), + ], +) +def test_invalid_json_shapes_do_not_retain_minio_secrets( + field_name: str, + field_value: str, +) -> None: + """Rejected non-object JSON must not survive in Workstream error frames.""" + secret = "nested-json-secret" + payload = json.dumps([{field_name: field_value}]) + + with pytest.raises(ValueError, match="^invalid settings JSON$") as caught: + Settings.model_validate_json(payload) + + assert secret not in f"{caught.value!s} {caught.value!r}" + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + @pytest.mark.parametrize( "method_name", ["model_validate", "model_validate_json", "model_validate_strings"], From fc0bc80ae1e652291f9df8bc3ba5cbeb86588a71 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 21:55:24 +0100 Subject: [PATCH 29/39] Clean partial S3 secrets on source errors --- backend/app/core/config.py | 2 +- backend/tests/test_config.py | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index dfadbcf5..3fa52fc4 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -586,7 +586,7 @@ def _extract_artifact_s3_static_secrets( environment_name=f"WORKSTREAM_{field_name.upper()}", ) ) - except ValueError: + except Exception: secrets.clear() for field_name in field_names: values.pop(field_name, None) diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 8f806a93..c3928c19 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -921,6 +921,32 @@ def test_invalid_json_shapes_do_not_retain_minio_secrets( ) +def test_partial_s3_secret_extraction_cleans_unexpected_dotenv_errors( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """A later source error must not retain an already-extracted credential.""" + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY", raising=False) + rate_secret = base64.b64encode(bytes(range(32))).decode("ascii") + access_key = "partially-extracted-minio-access-key" + payload = _minio_setting_values( + tmp_path, + api_rate_limit_key_secret=rate_secret, + artifact_s3_access_key_id=access_key, + _env_file=object(), + ) + payload.pop("artifact_s3_secret_access_key") + + with pytest.raises(TypeError) as caught: + Settings(**payload) + + assert_secret_not_retained( + caught.value, + access_key, + traceback_module_prefixes=("app.core.config",), + ) + + @pytest.mark.parametrize( "method_name", ["model_validate", "model_validate_json", "model_validate_strings"], From 042cad81ba1e37dd0334494cf014870851bc97e5 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 22:11:12 +0100 Subject: [PATCH 30/39] Bind local MinIO to loopback --- .github/workflows/backend.yml | 2 +- docker-compose.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index a9a7b490..06988490 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -49,7 +49,7 @@ jobs: run: | image='quay.io/minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e' docker run --detach --rm --name workstream-minio \ - --publish 9000:9000 \ + --publish 127.0.0.1:9000:9000 \ --env MINIO_ROOT_USER=workstream-minio \ --env MINIO_ROOT_PASSWORD=workstream-minio-secret-key \ "$image" server /data --address :9000 diff --git a/docker-compose.yml b/docker-compose.yml index d215c295..08d45ad1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -31,7 +31,7 @@ services: MINIO_ROOT_USER: workstream-minio MINIO_ROOT_PASSWORD: workstream-minio-secret-key ports: - - "9000:9000" + - "127.0.0.1:9000:9000" healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 5s From 5f26348de49b57aada93ddffb583f93603ec361b Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 22:15:16 +0100 Subject: [PATCH 31/39] Enforce loopback MinIO CI binding --- scripts/test_agent_gates.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index d871bad3..5826a52a 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -125,7 +125,7 @@ ( f"image='{MINIO_IMAGE}'", "docker run --detach --rm --name workstream-minio \\", - " --publish 9000:9000 \\", + " --publish 127.0.0.1:9000:9000 \\", " --env MINIO_ROOT_USER=workstream-minio \\", " --env MINIO_ROOT_PASSWORD=workstream-minio-secret-key \\", ' "$image" server /data --address :9000', From 5af5f7186f27ff2fdd4aeaf05b0510bbc363ac36 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 22:31:34 +0100 Subject: [PATCH 32/39] Clarify nonproduction MinIO eligibility --- README.md | 4 ++++ docs/architecture_lockdown.md | 5 +++++ docs/decision_0013_immutable_artifact_storage_boundary.md | 6 ++++++ docs/spec_artifact_storage_service.md | 7 +++++++ 4 files changed, 22 insertions(+) diff --git a/README.md b/README.md index 33826eec..b882766b 100644 --- a/README.md +++ b/README.md @@ -203,6 +203,10 @@ client against `http://localhost:9000` after MinIO is healthy, using access key `workstream-minio` and secret key `workstream-minio-secret-key`, before starting Workstream. Configure the runtime with the exact [artifact storage settings](docs/spec_artifact_storage_service.md#s3-compatible-adapter). +The repository-managed MinIO port is bound to host loopback. A Workstream +process running on a separate non-production container network may instead use +an operator-controlled private MinIO endpoint; that remains development/test +protocol proof and never qualifies as hosted-provider activation evidence. Native AWS S3 accepts workload-identity configuration but remains runtime-ineligible until live deployment proof is approved; startup fails with `artifact_provider_live_proof_required` before credential probing or provider diff --git a/docs/architecture_lockdown.md b/docs/architecture_lockdown.md index 2fe0be76..d26ee583 100644 --- a/docs/architecture_lockdown.md +++ b/docs/architecture_lockdown.md @@ -182,6 +182,11 @@ S3CompatibleArtifactStore AWS S3 in v0.1 production MinIO local and CI S3-compatible integration proof ``` +Here, `local and CI` is the non-production eligibility boundary. The +repository-managed MinIO service is published on host loopback; a private +operator-controlled container-network endpoint is also valid in local, +development, or test environments. MinIO is never hosted-production evidence. + AWS S3 is the only v0.1 production provider. Cloudflare R2 and Flow Node are deferred adapter initiatives. No provider owns product identity, authorization, lifecycle, bindings, audit, or integrity truth. PostgreSQL owns those facts; diff --git a/docs/decision_0013_immutable_artifact_storage_boundary.md b/docs/decision_0013_immutable_artifact_storage_boundary.md index 3a8f1ae6..f62dd068 100644 --- a/docs/decision_0013_immutable_artifact_storage_boundary.md +++ b/docs/decision_0013_immutable_artifact_storage_boundary.md @@ -28,6 +28,12 @@ S3CompatibleArtifactStore integration and production MinIO local and CI integration provider ``` +The `local/CI` MinIO label is a non-production eligibility class, not a literal +endpoint-host restriction. Repository-owned Compose and CI instances publish +on host loopback. An operator-controlled private container-network endpoint may +be used only in local, development, or test environments and does not count as +production activation evidence. + AWS S3 is the only v0.1 production provider. Each replica persists immutable provider profile and storage-namespace identity. Switching a populated deployment requires a separately approved complete-copy, verification, and diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 11d724f4..ccf1fa61 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -475,6 +475,13 @@ still claims the exact `minio-v1` namespace in PostgreSQL before any object operation. Local and CI run the real digest-pinned MinIO server and provision only the dedicated test bucket; tests do not emulate S3 calls. +`local/CI MinIO` names a non-production provider-eligibility class, not a +literal endpoint-host rule. Repository-managed Compose and CI publish MinIO on +host loopback only. A separately containerized Workstream process may use an +operator-controlled private MinIO endpoint in a local, development, or test +environment. Publicly exposed MinIO and every production-environment MinIO +configuration are outside v0.1; neither can satisfy native AWS live proof. + `aws_s3` accepts only a native endpoint-less AWS configuration, explicit region, and exactly one allowlisted workload-identity method. The composition root validates settings and then raises `artifact_provider_live_proof_required` From 48eeb2762f82e71129c4f42243eabe61b44933d3 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 22:53:47 +0100 Subject: [PATCH 33/39] Lock MinIO compose and namespace contracts --- docs/spec_artifact_storage_service.md | 13 +++++++++++++ scripts/test_agent_gates.py | 22 ++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index ccf1fa61..13f9d3c0 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -204,6 +204,19 @@ MinIO, and AWS S3, including region, bucket or private-root identity, normalized endpoint identity where applicable, private prefix, and addressing style; it contains no credential or resolved secret. +Every descriptor also carries the reserved `adapter`, `backend`, and +`provider_profile` keys. S3-compatible profiles then require exactly these +additional keys; missing, unknown, or extra keys fail closed: + +| Provider profile | Exact additional descriptor keys | +|---|---| +| `minio-v1` | `addressing_style`, `bucket`, `endpoint_identity`, `private_prefix`, `region` | +| `aws-s3-v1` | `addressing_style`, `bucket`, `private_prefix`, `region` | + +`endpoint_identity` is the non-secret SHA-256 identity of the normalized MinIO +endpoint. It is required for `minio-v1` and forbidden for native `aws-s3-v1`, +which has no configured endpoint. + For LocalStorage, the durable root must already exist with owner-private permissions before application startup. Its descriptor stores only a hash over the normalized path plus filesystem device/inode identity. Replacing or diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index 5826a52a..b6ab772e 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -4809,6 +4809,27 @@ def test_agent_gate_dependencies_and_workflow_are_pinned() -> None: ) +def test_local_minio_compose_is_regression_protected() -> None: + """Keep the repository-managed MinIO proof pinned and loopback-only.""" + compose = yaml.safe_load((ROOT / "docker-compose.yml").read_text(encoding="utf-8")) + minio = compose["services"]["minio"] + assert set(minio) == { + "image", + "command", + "environment", + "ports", + "healthcheck", + "volumes", + } + assert minio["image"] == MINIO_IMAGE + assert minio["command"] == "server /data --address :9000" + assert minio["environment"] == { + "MINIO_ROOT_USER": "workstream-minio", + "MINIO_ROOT_PASSWORD": "workstream-minio-secret-key", + } + assert minio["ports"] == ["127.0.0.1:9000:9000"] + + def test_backend_coverage_thresholds_are_regression_protected() -> None: """Keep both the approved global floor and stricter artifact floor fail closed.""" workflow_path = ROOT / ".github/workflows/backend.yml" @@ -5867,6 +5888,7 @@ def main() -> int: test_agent_gates_runs_stale_authorization_docs_fail_closed, test_agent_gates_runs_stale_artifact_contracts_fail_closed, test_agent_gate_dependencies_and_workflow_are_pinned, + test_local_minio_compose_is_regression_protected, test_backend_coverage_thresholds_are_regression_protected, test_artifact_coverage_phase_is_derived_from_work_queue, test_stale_artifact_contracts_cutover_rejects_reached_terms_only, From dc29ccc82ee835414df298b07723fa139834c2df Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 23:25:27 +0100 Subject: [PATCH 34/39] Bind ART 02B1 review evidence --- .../STATUS.md | 2 +- ...S-ART-001-02B1-internal-review-evidence.md | 32 +++++++++++++++---- .../WS-ART-001-02B1-pr-trust-bundle.md | 11 ++++--- 3 files changed, 32 insertions(+), 13 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index c331dafb..b4b08e89 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -54,7 +54,7 @@ has a v0.1 chunk. The reviewed implementation recorded: "The current gate is deterministic 02B1 proof followed by all nine exact-SHA internal reviewer tracks." That gate passed -at `9cd41ab`. The remaining gate is GitHub Actions, CodeRabbit, and explicit +at `48eeb276`. The remaining gate is GitHub Actions, CodeRabbit, and explicit human review. Durable admission, put attempts, verification publication, and recovery remain in later owning chunks. No later artifact chunk starts automatically, and only the user may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md index a783651d..413a272d 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md @@ -10,11 +10,11 @@ valid findings addressed: yes ## Reviewed Revision -Reviewed code SHA: `9cd41ab907aba323f1fbd79dac8bbe1602bcc30f` +Reviewed code SHA: `48eeb2762f82e71129c4f42243eabe61b44933d3` -Reviewed at: 2026-07-18T19:49:08Z +Reviewed at: 2026-07-18T22:22:34Z -Reviewer run IDs: senior-engineering=019f7698-ca9c-75a0-9ad9-5ddb553144ee; architecture=019f7698-d670-7b02-9336-93111bfb0905; QA/test=019f7698-ea1c-7632-8c76-3b2bddc1a709; security/auth=019f76b8-a76d-7060-95a2-e3296813464e; product/ops=019f76b8-acef-7cc1-a7ba-24f9262521de; reuse/dedup=019f76b8-b3b7-7961-85d9-5dfd9cddaaec; CI-integrity=019f76be-436c-7473-92da-50ee58380fd4; test-delta=019f76be-4869-7350-a737-fa75a6123fd0; docs=019f76be-52ed-7cf1-b589-aa9dc1ddd1bb +Reviewer run IDs: senior-engineering=019f7739-0db0-7392-ae3a-45438727badd; architecture=019f7739-003d-7bd1-9704-d4c3cbb92318; QA/test=019f7739-187d-7613-bb54-6f772673554d; security/auth=019f7741-7f1c-7780-93e1-bd40571685bd; product/ops=019f7741-8a64-7e82-9d48-ff8237bb7f4d; reuse/dedup=019f7741-9510-75c0-bdce-07c5c12b8b0a; CI-integrity=019f7748-0d50-7ea1-b15f-76d68a60fb58; test-delta=019f7748-2918-76b3-a2f1-374f14ab179e; docs=019f7748-53f2-7990-8a5e-15bf1c09fccf The reviewed base is `origin/main` at `99ae4c963e53f317175dcb308b9e47c93ccf19ed`, including AUTH PR #148. @@ -41,6 +41,12 @@ this evidence and requires a new exact-head review cycle. 90 percent gate while adding exact S3 adapter and validation gates. - Integrated AUTH PR #148 without editing or activating AUTH-owned runtime behavior. +- Bound repository-managed MinIO to host loopback in Compose and CI and added + deterministic gates for both definitions. +- Defined `local/CI MinIO` as non-production eligibility, including private + operator-controlled container-network development/test endpoints only. +- Documented and enforced the exact `minio-v1` and `aws-s3-v1` namespace + descriptor schemas. ## Reviewer Results @@ -74,6 +80,17 @@ this evidence and requires a new exact-head review cycle. removed stale provider-reference divergence. - Reconciled authored ART loop state and work queue with AUTH PR #148 while keeping AUTH-09D-B and every later ART chunk inactive. +- Corrected the successor chunk heading so merge-intent validation resolves + `WS-ART-001-02C1` without changing explicit-start authority. +- Consolidated duplicate mapping validators while preserving distinct Pydantic + entry points and clearing credentials from every ordinary exception path. +- Rejected non-object settings JSON before Pydantic can retain secret-bearing + input in structured errors and cleared partial S3 secret extraction on later + source failures. +- Bound repository-managed MinIO to loopback in Compose and CI and made both + definitions regression-protected by the 88-test agent-gate runner. +- Clarified private non-production MinIO eligibility and added the exact closed + S3 namespace profile table to the canonical artifact specification. ## Commands Run @@ -91,11 +108,11 @@ python3 scripts/test_agent_gates.py git diff --check ``` -Results: 428 real-service focused tests passed after AUTH PR #148 integration. +Results: 437 real-service focused tests passed after AUTH PR #148 integration. `S3CompatibleArtifactStore` coverage is 92 percent, S3 validation coverage is -100 percent, and combined changed-subsystem coverage is 92.95 percent. Ruff, +100 percent, and combined changed-subsystem coverage is 92.69 percent. Ruff, dependency integrity, stale contract/authorization/review/wording scans, -Markdown links, 87 agent-gate tests, and diff checks passed. +Markdown links, 88 agent-gate tests, Compose validation, and diff checks passed. The isolated full repository suite and 78 percent whole-app floor remain authoritative in GitHub Backend CI, as explicitly chosen because local execution @@ -105,7 +122,8 @@ is prohibitively slow. The CI reviewer confirmed that exact gate is retained. - Native AWS operations remain deliberately unavailable until Chunk 07 adds the release-bound live proof and production composition guard. -- MinIO is a local/CI protocol proof, not production activation evidence. +- MinIO is non-production protocol proof, including private local/dev/test + container-network use; it is not production activation evidence. - Provider acknowledgement remains non-bindable pending later admission, verification, publication, and recovery chunks. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md index ee484c63..73b0117b 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md @@ -50,13 +50,13 @@ user explicitly starts it. ## Tests And Checks ```text -428 real-service focused tests PASS +437 real-service focused tests PASS S3CompatibleArtifactStore coverage 92% S3 validation coverage 100% -Combined changed-subsystem coverage 92.95% +Combined changed-subsystem coverage 92.69% Ruff PASS pip check PASS -87 agent-gate tests PASS +88 agent-gate tests PASS Stale artifact/authorization/review/Workstream wording PASS Markdown links PASS git diff --check PASS @@ -67,7 +67,7 @@ percent repository floor, and all cumulative focused 90 percent gates. ## Internal Review -Reviewed code SHA: `9cd41ab907aba323f1fbd79dac8bbe1602bcc30f` +Reviewed code SHA: `48eeb2762f82e71129c4f42243eabe61b44933d3` All nine required reviewer tracks passed the exact integrated SHA with no remaining finding. Every reviewer session is closed. Reviewer IDs and the @@ -84,7 +84,8 @@ addressed repair history are recorded in the internal review evidence. ## Remaining Risks - AWS is configuration-ready but intentionally unavailable until Chunk 07. -- MinIO proves protocol behavior only for local and CI environments. +- MinIO proves protocol behavior only for non-production local, development, + test, and CI environments; repository-managed instances are loopback-bound. - Durable admission, verification, publication, and recovery are later chunks. ## Human Review Focus From 7dcce9ff3c9467069b79c84253371038e027db07 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sat, 18 Jul 2026 23:55:51 +0100 Subject: [PATCH 35/39] Refresh ART 02B1 review evidence after main integration --- .../STATUS.md | 11 ++++++----- .../WS-ART-001-02B1-internal-review-evidence.md | 16 ++++++++++------ .../reviews/WS-ART-001-02B1-pr-trust-bundle.md | 4 +++- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index b4b08e89..eb26899f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -53,8 +53,9 @@ has a v0.1 chunk. ## Gate The reviewed implementation recorded: "The current gate is deterministic 02B1 -proof followed by all nine exact-SHA internal reviewer tracks." That gate passed -at `48eeb276`. The remaining gate is GitHub Actions, CodeRabbit, and explicit -human review. Durable admission, put attempts, verification publication, and -recovery remain in later owning chunks. No later artifact chunk starts -automatically, and only the user may approve merge. +proof followed by all nine exact-SHA internal reviewer tracks." After integrating +latest `main`, including merged REV planning, that gate passed again at +`fafd6338` with no ART runtime or ownership drift. The remaining gate is GitHub +Actions, CodeRabbit, and explicit human review. Durable admission, put attempts, +verification publication, and recovery remain in later owning chunks. No later +artifact chunk starts automatically, and only the user may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md index 413a272d..4ccdf31a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md @@ -10,14 +10,16 @@ valid findings addressed: yes ## Reviewed Revision -Reviewed code SHA: `48eeb2762f82e71129c4f42243eabe61b44933d3` +Reviewed code SHA: `fafd6338623ab82152ba9dbe88b5622c73f1ff91` -Reviewed at: 2026-07-18T22:22:34Z +Reviewed at: 2026-07-18T22:51:42Z -Reviewer run IDs: senior-engineering=019f7739-0db0-7392-ae3a-45438727badd; architecture=019f7739-003d-7bd1-9704-d4c3cbb92318; QA/test=019f7739-187d-7613-bb54-6f772673554d; security/auth=019f7741-7f1c-7780-93e1-bd40571685bd; product/ops=019f7741-8a64-7e82-9d48-ff8237bb7f4d; reuse/dedup=019f7741-9510-75c0-bdce-07c5c12b8b0a; CI-integrity=019f7748-0d50-7ea1-b15f-76d68a60fb58; test-delta=019f7748-2918-76b3-a2f1-374f14ab179e; docs=019f7748-53f2-7990-8a5e-15bf1c09fccf +Reviewer run IDs: senior-engineering=019f7757-5a85-78d2-a205-5e4daa95e0f6; architecture=019f7757-4ec9-7640-b511-a6206d78ae18; QA/test=019f775f-20e2-7b51-92b9-00c8c628b1b5; security/auth=019f775f-64e8-7472-8efa-f689fd04eb9e; product/ops=019f775f-755f-7860-be64-0d24f7b3636d; reuse/dedup=019f775f-82ba-7dd2-9338-e4fa4d1678ea; CI-integrity=019f7766-5851-7b32-8dc2-427deba32f83; test-delta=019f7766-74ef-7961-93f9-573eda2455bf; docs=019f7766-9599-7503-a835-8eddfe95e343 The reviewed base is `origin/main` at -`99ae4c963e53f317175dcb308b9e47c93ccf19ed`, including AUTH PR #148. +`983b9e534b84f1590fafecc0ce1355cf131257ce`, including AUTH PR #148 and the +latest merged REV planning. The merged REV planning changes introduce no ART +runtime or ownership drift in `origin/main...HEAD`. Only review artifacts and initiative status may change after the reviewed SHA. Any implementation, test, workflow, policy, or chunk-contract change invalidates this evidence and requires a new exact-head review cycle. @@ -41,6 +43,8 @@ this evidence and requires a new exact-head review cycle. 90 percent gate while adding exact S3 adapter and validation gates. - Integrated AUTH PR #148 without editing or activating AUTH-owned runtime behavior. +- Integrated the latest merged REV planning without editing or activating + REV-owned runtime behavior. - Bound repository-managed MinIO to host loopback in Compose and CI and added deterministic gates for both definitions. - Defined `local/CI MinIO` as non-production eligibility, including private @@ -52,10 +56,10 @@ this evidence and requires a new exact-head review cycle. | Reviewer | Result | Blocking findings | Notes | |---|---:|---|---| -| senior engineering | PASS | None | The shared adapter remains maintainable, AWS remains unavailable before live proof, and no operational issue remains. | +| senior engineering | PASS WITH LOW RISKS | None | The shared adapter remains maintainable and AWS remains unavailable before live proof. A non-blocking repository-wide Ruff version-range reproducibility note remains outside this ART behavior. | | qa/test | PASS | None | Real MinIO, credential isolation, namespace fencing, configuration, and CI evidence cover the chunk contract without weakened tests. | | security/auth | PASS | None | Secret-bearing endpoint and metadata failures are sanitized, credential sources are closed, and ART/AUTH ownership remains intact. | -| product/ops | PASS | None | No operator recovery, contributor, reviewer, contribution, compensation, fulfillment, or reputation behavior is activated. | +| product/ops | PASS WITH LOW RISKS | None | No operator recovery, contributor, reviewer, contribution, compensation, fulfillment, or reputation behavior is activated. Existing pre-cutover R2 wording remains owned by later product cutover work and adds no runtime eligibility here. | | architecture | PASS | None | One typed adapter path remains, concrete S3 construction stays in ART composition, and product services receive no concrete store. | | ci integrity | PASS | None | The 78 percent repository gate and all cumulative 90 percent gates remain fail closed with no bypass. | | docs | PASS | None | README, artifact specification, settings, provider eligibility, and loop state match the implementation. | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md index 73b0117b..f0352fc0 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md @@ -25,6 +25,8 @@ chunk. traceback/object-graph tests. - Added exact async SDK pins and cumulative CI coverage guards. - Integrated AUTH PR #148 while leaving all AUTH runtime activation with AUTH. +- Integrated the latest merged REV planning without changing ART or REV runtime + ownership. ## Scope Control @@ -67,7 +69,7 @@ percent repository floor, and all cumulative focused 90 percent gates. ## Internal Review -Reviewed code SHA: `48eeb2762f82e71129c4f42243eabe61b44933d3` +Reviewed code SHA: `fafd6338623ab82152ba9dbe88b5622c73f1ff91` All nine required reviewer tracks passed the exact integrated SHA with no remaining finding. Every reviewer session is closed. Reviewer IDs and the From 9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sun, 19 Jul 2026 00:23:52 +0100 Subject: [PATCH 36/39] Harden S3 artifact streaming and credential proof --- .../app/adapters/artifacts/s3_compatible.py | 41 ++++++++++++----- backend/app/core/s3_validation.py | 5 +++ backend/tests/assertion_helpers.py | 17 +++++++ backend/tests/test_assertion_helpers.py | 19 ++++++++ .../tests/test_aws_credential_isolation.py | 44 ++++++++++++++++++- backend/tests/test_config.py | 4 ++ backend/tests/test_s3_artifact_store.py | 31 +++++++++++++ 7 files changed, 150 insertions(+), 11 deletions(-) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index 131078f5..ff5262d3 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -789,6 +789,7 @@ def __init__(self, source: CommittedArtifactSource, buffer_bytes: int) -> None: self._consumed = False self._complete = False self._iterator: AsyncIterator[bytes] | None = None + self._source_cursor: memoryview | None = None self._pending = bytearray() self._digest = hashlib.sha256() self._byte_count = 0 @@ -824,16 +825,22 @@ async def read(self, amount: int | None = -1) -> bytes: if self._iterator is None: self._iterator = self._source.stream() while len(self._pending) < requested: - try: - source_chunk = await anext(self._iterator) - except StopAsyncIteration: - self._finish() - break - if not isinstance(source_chunk, bytes): - raise ArtifactInputMismatchError("artifact source must yield bytes") - self._pending.extend(source_chunk) - if len(self._pending) > self._buffer_bytes + requested: - break + if self._source_cursor is None: + try: + source_chunk = await anext(self._iterator) + except StopAsyncIteration: + self._finish() + break + if not isinstance(source_chunk, bytes): + raise ArtifactInputMismatchError("artifact source must yield bytes") + self._source_cursor = memoryview(source_chunk) + if not self._source_cursor: + self._source_cursor = None + continue + needed = requested - len(self._pending) + copied = min(needed, len(self._source_cursor)) + self._pending.extend(self._source_cursor[:copied]) + self._source_cursor = self._source_cursor[copied:] or None chunk = bytes(self._pending[:requested]) del self._pending[:requested] self._byte_count += len(chunk) @@ -862,6 +869,10 @@ async def validate_complete(self) -> None: async def _require_source_exhausted(self) -> None: """Prove EOF before returning the final committed byte to the SDK.""" + if self._source_cursor: + raise ArtifactInputMismatchError( + "artifact source exceeds committed byte count" + ) if self._iterator is None: self._iterator = self._source.stream() while True: @@ -1124,6 +1135,16 @@ async def resolve_isolated_aws_workload_credentials( ) from None if credentials is None or getattr(credentials, "method", None) != expected_method: raise ArtifactConfigurationError("AWS workload identity method did not match") + materialization_failed = False + try: + await credentials.get_frozen_credentials() + except Exception: + materialization_failed = True + if materialization_failed: + del credentials + raise ArtifactConfigurationError( + "AWS workload identity credentials could not be resolved" + ) from None return credentials diff --git a/backend/app/core/s3_validation.py b/backend/app/core/s3_validation.py index 60432991..bd8d7769 100644 --- a/backend/app/core/s3_validation.py +++ b/backend/app/core/s3_validation.py @@ -3,6 +3,7 @@ from __future__ import annotations from collections.abc import Mapping +import ipaddress import re from urllib.parse import urlsplit @@ -90,6 +91,10 @@ def _try_canonical_minio_endpoint(value: object) -> str | None: scheme = parsed.scheme.lower() hostname = parsed.hostname.lower() if ":" in hostname: + try: + hostname = ipaddress.IPv6Address(hostname).compressed + except ipaddress.AddressValueError: + return None hostname = f"[{hostname}]" if port is not None and port != {"http": 80, "https": 443}[scheme]: hostname = f"{hostname}:{port}" diff --git a/backend/tests/assertion_helpers.py b/backend/tests/assertion_helpers.py index 76dc8ab2..23cc2fbf 100644 --- a/backend/tests/assertion_helpers.py +++ b/backend/tests/assertion_helpers.py @@ -22,6 +22,8 @@ def assert_secret_not_retained( seen.add(id(value)) if isinstance(value, str): assert secret not in value + elif isinstance(value, (bytes, bytearray, memoryview)): + assert secret.encode("utf-8") not in bytes(value) elif isinstance(value, SecretStr): assert secret not in value.get_secret_value() elif isinstance(value, BaseException): @@ -64,6 +66,21 @@ def assert_secret_not_retained( seen, traceback_module_prefixes=traceback_module_prefixes, ) + elif isinstance(getattr(value, "__dict__", None), Mapping): + assert_secret_not_retained( + vars(value), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + private = getattr(value, "__pydantic_private__", None) + if isinstance(private, Mapping): + assert_secret_not_retained( + private, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) elif isinstance(getattr(value, "__pydantic_private__", None), Mapping): assert_secret_not_retained( getattr(value, "__pydantic_private__"), diff --git a/backend/tests/test_assertion_helpers.py b/backend/tests/test_assertion_helpers.py index 77b0c107..54f8fe25 100644 --- a/backend/tests/test_assertion_helpers.py +++ b/backend/tests/test_assertion_helpers.py @@ -2,6 +2,7 @@ from pydantic import SecretStr import pytest +from types import SimpleNamespace from tests.assertion_helpers import assert_secret_not_retained @@ -10,3 +11,21 @@ def test_secret_str_rejects_embedded_forbidden_value() -> None: """Reject a forbidden value even when wrapped by other secret text.""" with pytest.raises(AssertionError): assert_secret_not_retained(SecretStr("prefix-forbidden-suffix"), "forbidden") + + +@pytest.mark.parametrize( + "value", + [b"prefix-forbidden-suffix", bytearray(b"forbidden"), memoryview(b"forbidden")], +) +def test_byte_buffers_reject_embedded_forbidden_value(value: object) -> None: + """Reject a forbidden value retained in any supported byte buffer.""" + with pytest.raises(AssertionError): + assert_secret_not_retained(value, "forbidden") + + +def test_public_object_state_rejects_nested_forbidden_value() -> None: + """Reject a forbidden value reachable through public object state.""" + value = SimpleNamespace(payload={"body": b"prefix-forbidden-suffix"}) + + with pytest.raises(AssertionError): + assert_secret_not_retained(value, "forbidden") diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 02310450..40449d6c 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -567,6 +567,16 @@ def capture_client(service_name: str, **kwargs: object) -> object: async def test_resolved_workload_identity_method_must_match() -> None: + materialized: list[str] = [] + + class FakeCredentials: + def __init__(self, method: str) -> None: + self.method = method + + async def get_frozen_credentials(self) -> object: + materialized.append(self.method) + return SimpleNamespace(access_key="access", secret_key="secret", token=None) + class FakeSession: def __init__(self, method: str | None) -> None: self._method = method @@ -574,7 +584,7 @@ def __init__(self, method: str | None) -> None: async def get_credentials(self) -> object | None: if self._method is None: return None - return SimpleNamespace(method=self._method) + return FakeCredentials(self._method) with pytest.raises( ArtifactConfigurationError, @@ -599,6 +609,38 @@ async def get_credentials(self) -> object | None: expected_method="container-role", ) assert getattr(resolved, "method") == "container-role" + assert materialized == ["container-role"] + + +async def test_workload_identity_materialization_failure_is_sanitized() -> None: + secret = "deferred-credential-secret" + + class DeferredCredentials: + method = "container-role" + + async def get_frozen_credentials(self) -> object: + raise RuntimeError(f"credential refresh returned {secret}") + + class FakeSession: + async def get_credentials(self) -> object: + return DeferredCredentials() + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity credentials could not be resolved", + ) as caught: + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(), # type: ignore[arg-type] + expected_method="container-role", + ) + + assert caught.value.__cause__ is None + assert caught.value.__context__ is None + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) async def test_credential_resolution_failure_is_sanitized() -> None: diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index c3928c19..8aec4145 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -1162,6 +1162,10 @@ def test_s3_prefix_validation_rejects_noncanonical_outer_bounds(value: object) - def test_minio_endpoint_canonicalization_handles_ipv6_and_invalid_ports() -> None: assert canonical_minio_endpoint("HTTP://[::1]:9000/") == "http://[::1]:9000" + assert ( + canonical_minio_endpoint("http://[0:0:0:0:0:0:0:1]:9000") + == "http://[::1]:9000" + ) with pytest.raises(ValueError, match="endpoint is invalid"): canonical_minio_endpoint("http://localhost:not-a-port") diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index abed3f34..054f601c 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -268,6 +268,37 @@ def test_namespace_claim_mismatch_fails_before_provider_io() -> None: bootstrap.initialize_after_namespace_claim(wrong) +async def test_committed_body_keeps_large_source_chunks_out_of_pending_buffer() -> None: + """Retain unconsumed source bytes by view while bounding copied request bytes.""" + content = b"0123456789abcdef" + commitment = ArtifactCommitment( + sha256=f"sha256:{hashlib.sha256(content).hexdigest()}", + byte_count=len(content), + media_type="application/octet-stream", + ) + + class Source: + def __init__(self) -> None: + self.commitment = commitment + + def stream(self) -> AsyncIterator[bytes]: + async def iterate() -> AsyncIterator[bytes]: + yield content + + return iterate() + + body = s3_module._CommittedSourceBody(Source(), 4) # type: ignore[arg-type] # noqa: SLF001 + + assert await body.read(4) == b"0123" + assert len(body._pending) == 0 # noqa: SLF001 + assert body._source_cursor is not None # noqa: SLF001 + assert len(body._source_cursor) == 12 # noqa: SLF001 + assert await body.read(4) == b"4567" + assert await body.read(4) == b"89ab" + assert await body.read(4) == b"cdef" + assert body.complete is True + + async def raw_minio_put(key: str, content: bytes) -> None: """Publish adversarial integration bytes outside the Workstream adapter.""" session = AioSession() From 2ba3af979b599ca0104e3a658e12137996a0f039 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sun, 19 Jul 2026 00:43:38 +0100 Subject: [PATCH 37/39] Bind ART 02B1 evidence to external review repairs --- .../STATUS.md | 5 ++- ...S-ART-001-02B1-internal-review-evidence.md | 40 +++++++++++++------ .../WS-ART-001-02B1-pr-trust-bundle.md | 13 +++--- 3 files changed, 39 insertions(+), 19 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index eb26899f..af1a1509 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -55,7 +55,8 @@ has a v0.1 chunk. The reviewed implementation recorded: "The current gate is deterministic 02B1 proof followed by all nine exact-SHA internal reviewer tracks." After integrating latest `main`, including merged REV planning, that gate passed again at -`fafd6338` with no ART runtime or ownership drift. The remaining gate is GitHub -Actions, CodeRabbit, and explicit human review. Durable admission, put attempts, +`9cd5620e` after addressing all four valid CodeRabbit findings, with no ART +runtime or ownership drift. The remaining gate is the post-fix GitHub Actions +and CodeRabbit rerun plus explicit human review. Durable admission, put attempts, verification publication, and recovery remain in later owning chunks. No later artifact chunk starts automatically, and only the user may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md index 4ccdf31a..ea52f940 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md @@ -10,11 +10,11 @@ valid findings addressed: yes ## Reviewed Revision -Reviewed code SHA: `fafd6338623ab82152ba9dbe88b5622c73f1ff91` +Reviewed code SHA: `9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a` -Reviewed at: 2026-07-18T22:51:42Z +Reviewed at: 2026-07-18T23:40:43Z -Reviewer run IDs: senior-engineering=019f7757-5a85-78d2-a205-5e4daa95e0f6; architecture=019f7757-4ec9-7640-b511-a6206d78ae18; QA/test=019f775f-20e2-7b51-92b9-00c8c628b1b5; security/auth=019f775f-64e8-7472-8efa-f689fd04eb9e; product/ops=019f775f-755f-7860-be64-0d24f7b3636d; reuse/dedup=019f775f-82ba-7dd2-9338-e4fa4d1678ea; CI-integrity=019f7766-5851-7b32-8dc2-427deba32f83; test-delta=019f7766-74ef-7961-93f9-573eda2455bf; docs=019f7766-9599-7503-a835-8eddfe95e343 +Reviewer run IDs: senior-engineering=019f778b-6b07-7561-a718-e1da40a532a0; architecture=019f778b-6372-7c83-85ca-a61f63885c17; QA/test=019f778b-74eb-7250-b07c-573c979b2849; security/auth=019f778b-81f0-7b31-85d5-8955f9047b3e; product/ops=019f7791-3326-71e3-b838-54143b94a129; reuse/dedup=019f7791-3943-7de3-bdbf-c3643f45c43c; CI-integrity=019f7791-43a6-78e0-8f17-67861bf27455; test-delta=019f7791-4c52-7091-9076-f534bccdfad6; docs=019f7797-215a-7e02-b6e4-c33b584ec2b5 The reviewed base is `origin/main` at `983b9e534b84f1590fafecc0ce1355cf131257ce`, including AUTH PR #148 and the @@ -51,20 +51,27 @@ this evidence and requires a new exact-head review cycle. operator-controlled container-network development/test endpoints only. - Documented and enforced the exact `minio-v1` and `aws-s3-v1` namespace descriptor schemas. +- Bounded copied request-body bytes independently of source chunk size by + retaining unconsumed source bytes through a memoryview cursor. +- Required selected AWS workload credentials to materialize successfully before + explicit resolution can report success. +- Canonicalized equivalent IPv6 MinIO literals before deriving endpoint and + namespace identity. +- Extended secret-retention proof to byte buffers and public object state. ## Reviewer Results | Reviewer | Result | Blocking findings | Notes | |---|---:|---|---| -| senior engineering | PASS WITH LOW RISKS | None | The shared adapter remains maintainable and AWS remains unavailable before live proof. A non-blocking repository-wide Ruff version-range reproducibility note remains outside this ART behavior. | -| qa/test | PASS | None | Real MinIO, credential isolation, namespace fencing, configuration, and CI evidence cover the chunk contract without weakened tests. | +| senior engineering | PASS WITH LOW RISKS | None | The shared adapter and four external-review repairs are maintainable. Exact SDK pins bound the reviewed reliance on SDK credential internals. | +| qa/test | PASS WITH LOW RISKS | None | Real MinIO, credential isolation, bounded streaming, IPv6 identity, secret scanning, and CI evidence cover the chunk contract without weakened tests. | | security/auth | PASS | None | Secret-bearing endpoint and metadata failures are sanitized, credential sources are closed, and ART/AUTH ownership remains intact. | -| product/ops | PASS WITH LOW RISKS | None | No operator recovery, contributor, reviewer, contribution, compensation, fulfillment, or reputation behavior is activated. Existing pre-cutover R2 wording remains owned by later product cutover work and adds no runtime eligibility here. | -| architecture | PASS | None | One typed adapter path remains, concrete S3 construction stays in ART composition, and product services receive no concrete store. | +| product/ops | PASS | None | No operator recovery, contributor, reviewer, contribution, compensation, fulfillment, or reputation behavior is activated. | +| architecture | PASS WITH LOW RISKS | None | One typed adapter path remains, concrete S3 construction stays in ART composition, and exact SDK pins bound the inactive AWS helper dependency. | | ci integrity | PASS | None | The 78 percent repository gate and all cumulative 90 percent gates remain fail closed with no bypass. | | docs | PASS | None | README, artifact specification, settings, provider eligibility, and loop state match the implementation. | -| reuse/dedup | PASS | None | Provider references, S3 validation, runtime guards, and deep secret assertions reuse canonical helpers without parallel implementations. | -| test delta | PASS | None | Removed placeholder assertions were replaced by stronger active-MinIO and inactive-AWS behavior proof; no test was skipped or weakened. | +| reuse/dedup | PASS WITH LOW RISKS | None | Provider references, S3 validation, runtime guards, and deep secret assertions reuse canonical helpers without parallel implementations. The reviewer relied on the recorded real-MinIO proof rather than rerunning it. | +| test delta | PASS WITH LOW RISKS | None | Active-MinIO and inactive-AWS proof is stronger, including all four repairs; no test was skipped or weakened. AWS remains intentionally inactive pending live proof. | ## Valid Findings Addressed @@ -95,6 +102,15 @@ this evidence and requires a new exact-head review cycle. definitions regression-protected by the 88-test agent-gate runner. - Clarified private non-production MinIO eligibility and added the exact closed S3 namespace profile table to the canonical artifact specification. +- Replaced whole-source-chunk pending copies with a bounded memoryview cursor and + added direct proof that unconsumed bytes remain available without expanding + the copied request buffer. +- Materialized deferred AWS workload credentials before returning explicit + resolution success and sanitized refresh failures. +- Compressed equivalent IPv6 endpoint literals before namespace identity + derivation. +- Extended deep secret assertions across bytes, bytearray, memoryview, and + nested public object state. ## Commands Run @@ -112,9 +128,9 @@ python3 scripts/test_agent_gates.py git diff --check ``` -Results: 437 real-service focused tests passed after AUTH PR #148 integration. -`S3CompatibleArtifactStore` coverage is 92 percent, S3 validation coverage is -100 percent, and combined changed-subsystem coverage is 92.69 percent. Ruff, +Results: 443 real-service focused tests passed after the external-review repairs. +`S3CompatibleArtifactStore` coverage is 91 percent, S3 validation coverage is +97 percent, and combined changed-subsystem coverage is 92.52 percent. Ruff, dependency integrity, stale contract/authorization/review/wording scans, Markdown links, 88 agent-gate tests, Compose validation, and diff checks passed. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md index f0352fc0..e1c38ede 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md @@ -27,6 +27,9 @@ chunk. - Integrated AUTH PR #148 while leaving all AUTH runtime activation with AUTH. - Integrated the latest merged REV planning without changing ART or REV runtime ownership. +- Bounded source-chunk copying, materialized deferred AWS credentials, + canonicalized IPv6 endpoint identity, and deepened secret-retention proof in + response to the valid external review findings. ## Scope Control @@ -52,10 +55,10 @@ user explicitly starts it. ## Tests And Checks ```text -437 real-service focused tests PASS -S3CompatibleArtifactStore coverage 92% -S3 validation coverage 100% -Combined changed-subsystem coverage 92.69% +443 real-service focused tests PASS +S3CompatibleArtifactStore coverage 91% +S3 validation coverage 97% +Combined changed-subsystem coverage 92.52% Ruff PASS pip check PASS 88 agent-gate tests PASS @@ -69,7 +72,7 @@ percent repository floor, and all cumulative focused 90 percent gates. ## Internal Review -Reviewed code SHA: `fafd6338623ab82152ba9dbe88b5622c73f1ff91` +Reviewed code SHA: `9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a` All nine required reviewer tracks passed the exact integrated SHA with no remaining finding. Every reviewer session is closed. Reviewer IDs and the From 1b923b0e7abd3cea79824e2d6821fe7ae8d25a12 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sun, 19 Jul 2026 00:48:53 +0100 Subject: [PATCH 38/39] Record ART 02B1 external review response --- ...S-ART-001-02B1-external-review-response.md | 61 +++++++++++++++++++ .../WS-ART-001-02B1-pr-trust-bundle.md | 8 ++- 2 files changed, 67 insertions(+), 2 deletions(-) create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md new file mode 100644 index 00000000..9139cf85 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md @@ -0,0 +1,61 @@ +# External Review Response: WS-ART-001-02B1 + +## Current State + +- Pull request: #151, open and not draft +- Reviewed implementation SHA: `9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a` +- Published evidence-bound SHA: `2ba3af979b599ca0104e3a658e12137996a0f039` +- CodeRabbit's four actionable findings: fixed and automatically resolved +- Post-fix incremental CodeRabbit review: rate-limited; explicit rerun pending +- GitHub Agent Gates: passed on the repaired candidate +- GitHub Backend: pending on the repaired candidate +- Human merge approval: pending + +Internal sub-agent review is recorded only in the separate internal-review +evidence. This file records CodeRabbit, GitHub, and later human PR feedback. + +## CodeRabbit Findings Addressed + +| Finding | Status | Response | +|---|---:|---| +| A complete source chunk was copied into the pending request buffer before applying the configured bound. | Fixed | `_CommittedSourceBody` retains unconsumed bytes through a memoryview cursor and copies only the amount requested. Direct regression proof verifies the copied pending buffer remains bounded while later reads receive the remaining bytes. | +| Deferred AWS workload credentials could be reported as resolved before token or role materialization. | Fixed | Explicit credential resolution now awaits `get_frozen_credentials()` after exact method validation and maps refresh failure to the same sanitized configuration error. | +| Equivalent IPv6 MinIO endpoint spellings produced different namespace identities. | Fixed | IPv6 literals are parsed and compressed before normalized endpoint and namespace identity derivation. | +| Secret-retention assertions did not inspect byte buffers or public object state. | Fixed | The shared assertion now checks bytes, bytearray, memoryview, nested `__dict__` state, and Pydantic private state, with direct negative tests. | + +CodeRabbit marked all four review threads resolved after commits `9cd5620` through +`2ba3af9`. + +## Findings Not Applied + +CodeRabbit's generic pre-merge widget reported 45.14 percent docstring coverage +against its 80 percent default. The repository's authoritative configured +docstring gate is 92.0 percent and passed in GitHub before the repairs. No +duplicate or weaker docstring gate was added. + +The widget also reported that the PR description did not follow the repository +trust-bundle template. The PR body is being replaced with the current reviewed +chunk, evidence, reviewer, gate, and human-checkpoint data. + +## Validation After Repairs + +```text +443 real-service focused tests: PASS +S3CompatibleArtifactStore coverage: 91% +S3 validation coverage: 97% +Combined changed-subsystem coverage: 92.52% +Ruff: PASS +Dependency integrity: PASS +Agent gates: 88 PASS +Stale artifact/authorization/review/Workstream scans: PASS +Markdown links and Compose validation: PASS +All nine exact-SHA internal reviewer tracks: PASS +Open sub-agent sessions: none +``` + +## Remaining External Gate + +Wait for GitHub Backend on the final published head. When CodeRabbit review +capacity becomes available, request the post-fix incremental review and verify +that no unresolved actionable thread remains. Stop for the user's explicit +merge decision. Do not start `WS-ART-001-02C1` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md index e1c38ede..de3bde91 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md @@ -80,10 +80,14 @@ addressed repair history are recorded in the internal review evidence. ## External Review +External review response file: + +- `.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md` + | Source | Status | Notes | |---|---:|---| -| GitHub Actions | Pending | Runs the isolated full suite, repository floor, cumulative focused gates, and real MinIO proof. | -| CodeRabbit | Pending | External review starts after publication. | +| GitHub Actions | Pending | Agent Gates passed; Backend reruns the isolated full suite, repository floor, cumulative focused gates, and real MinIO proof on the repaired candidate. | +| CodeRabbit | Repairs verified; rerun pending | Four valid findings were fixed and automatically resolved. The immediate incremental rerun was rate-limited. | | Human review | Pending | Only the user may approve merge. | ## Remaining Risks From 1260353f7ed5b25a0e9d9585a1d8ed112c2bef3e Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Sun, 19 Jul 2026 01:05:24 +0100 Subject: [PATCH 39/39] Resolve ART 02B1 external review wording --- ...S-ART-001-02B1-external-review-response.md | 19 +++++++++++-------- ...S-ART-001-02B1-internal-review-evidence.md | 4 ++-- .../WS-ART-001-02B1-pr-trust-bundle.md | 2 +- 3 files changed, 14 insertions(+), 11 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md index 9139cf85..c3a624a3 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md @@ -5,8 +5,9 @@ - Pull request: #151, open and not draft - Reviewed implementation SHA: `9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a` - Published evidence-bound SHA: `2ba3af979b599ca0104e3a658e12137996a0f039` -- CodeRabbit's four actionable findings: fixed and automatically resolved -- Post-fix incremental CodeRabbit review: rate-limited; explicit rerun pending +- CodeRabbit's four implementation findings: fixed and automatically resolved +- Post-fix incremental CodeRabbit review: complete; two evidence-wording + findings fixed, final verification pending - GitHub Agent Gates: passed on the repaired candidate - GitHub Backend: pending on the repaired candidate - Human merge approval: pending @@ -22,9 +23,12 @@ evidence. This file records CodeRabbit, GitHub, and later human PR feedback. | Deferred AWS workload credentials could be reported as resolved before token or role materialization. | Fixed | Explicit credential resolution now awaits `get_frozen_credentials()` after exact method validation and maps refresh failure to the same sanitized configuration error. | | Equivalent IPv6 MinIO endpoint spellings produced different namespace identities. | Fixed | IPv6 literals are parsed and compressed before normalized endpoint and namespace identity derivation. | | Secret-retention assertions did not inspect byte buffers or public object state. | Fixed | The shared assertion now checks bytes, bytearray, memoryview, nested `__dict__` state, and Pydantic private state, with direct negative tests. | +| Credential-materialization evidence wording was grammatically ambiguous. | Fixed | The sentence now identifies "the selected AWS workload credentials" unambiguously. | +| The real-service-focused test-count modifier was not hyphenated. | Fixed | The compound modifier is now hyphenated. | -CodeRabbit marked all four review threads resolved after commits `9cd5620` through -`2ba3af9`. +CodeRabbit marked all four implementation review threads resolved after commits +`9cd5620` through `2ba3af9`. Its post-fix review found only the two evidence +wording corrections recorded above. ## Findings Not Applied @@ -55,7 +59,6 @@ Open sub-agent sessions: none ## Remaining External Gate -Wait for GitHub Backend on the final published head. When CodeRabbit review -capacity becomes available, request the post-fix incremental review and verify -that no unresolved actionable thread remains. Stop for the user's explicit -merge decision. Do not start `WS-ART-001-02C1` automatically. +Wait for GitHub Backend and final CodeRabbit verification on the published head, +then verify that no unresolved actionable thread remains. Stop for the user's +explicit merge decision. Do not start `WS-ART-001-02C1` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md index ea52f940..877385e2 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md @@ -53,7 +53,7 @@ this evidence and requires a new exact-head review cycle. descriptor schemas. - Bounded copied request-body bytes independently of source chunk size by retaining unconsumed source bytes through a memoryview cursor. -- Required selected AWS workload credentials to materialize successfully before +- Required the selected AWS workload credentials to materialize successfully before explicit resolution can report success. - Canonicalized equivalent IPv6 MinIO literals before deriving endpoint and namespace identity. @@ -128,7 +128,7 @@ python3 scripts/test_agent_gates.py git diff --check ``` -Results: 443 real-service focused tests passed after the external-review repairs. +Results: 443 real-service-focused tests passed after the external-review repairs. `S3CompatibleArtifactStore` coverage is 91 percent, S3 validation coverage is 97 percent, and combined changed-subsystem coverage is 92.52 percent. Ruff, dependency integrity, stale contract/authorization/review/wording scans, diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md index de3bde91..7e5ab24a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md @@ -87,7 +87,7 @@ External review response file: | Source | Status | Notes | |---|---:|---| | GitHub Actions | Pending | Agent Gates passed; Backend reruns the isolated full suite, repository floor, cumulative focused gates, and real MinIO proof on the repaired candidate. | -| CodeRabbit | Repairs verified; rerun pending | Four valid findings were fixed and automatically resolved. The immediate incremental rerun was rate-limited. | +| CodeRabbit | Final verification pending | Four implementation findings were fixed and resolved; the post-fix review found only two evidence-wording corrections, now fixed. | | Human review | Pending | Only the user may approve merge. | ## Remaining Risks