diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index 93319361..f798f4d1 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -19,33 +19,30 @@ - AUTH-09C merged through PR #146 as `0ffdabf`; signed schema-v2 memory at `eeb3dc2` recorded its two administrative reads and stopped. - PR #141 merged `WS-ART-001-02A3` into `main` as `a10d901` on - 2026-07-18; ART-02B1 remains inactive pending a separate explicit start. -- Active implementation chunk: `WS-AUTH-001-09D-A`. The explicitly started parent - `WS-AUTH-001-09D` failed required preimplementation review and was split - before runtime edits into `WS-AUTH-001-09D-A` and `WS-AUTH-001-09D-B`. -- Active branch: `WS-AUTH-001-09D-A` on - `codex/ws-auth-001-09d-actor-identity-lifecycle`. It owns lifecycle evidence - repair plus three exact profile lifecycle mutations. `WS-AUTH-001-09D-B` - remains inactive until 09D-A merge, signed memory, and explicit user start. -- Start basis: trusted `main` at `0ffdabf` after PR #146. + 2026-07-18; the user then explicitly started ART-02B1. +- Active ART implementation chunk: `WS-ART-001-02B1` on + `codex/ws-art-001-02b1-s3-compatible-minio-aws`. +- The ART worktree consumes merged AUTH, REV, and CON contracts without + editing or activating their independently owned runtime behavior. +- AUTH-09D-A merged through PR #148 as `99ae4c9`; AUTH-09D-B remains inactive + until signed memory and an explicit human start. +- ART integration basis: trusted `main` at `99ae4c9` after PR #148. - PR #119 merged `WS-AUTH-001-05B` as `ad71c7e`. - PR #120 merged `WS-ART-001-OBJECT-STORAGE-AMENDMENT` as `4408256`. - PR #122 merged the first automated post-merge memory implementation as `fc89fb6`; its schema-v1 cross-initiative next pointer is superseded by the schema-v2 initiative-local clean cut. -- Current gate: PR #148 external repair aligns database lifecycle-reason - whitespace with the API and repairs stale reviewer evidence. PostgreSQL proof - plus required exact-head internal review pass at `7c33e64`; canonical evidence - is refreshed against that SHA after repairing the Backend fixture cascade. - Replacement external checks and explicit human approval remain. No service - caller becomes executable before AUTH-09E. +- Current ART gate: integrate trusted `main`, complete deterministic 02B1 + proof, and pass all nine exact-SHA internal reviewer tracks before opening + the ART PR. No later ART chunk starts automatically. - Scope checkpoint: AWS S3 is the only v0.1 production provider; MinIO is local/CI S3 protocol proof; LocalStorage is focused development/test; R2 and Flow Node are deferred. Product modules receive narrow artifact capabilities, and AWS cannot instantiate in production without release-bound live proof. - Authorization checkpoint: merged main contains 74 PermissionIds and 65 - ActionIds, with the two actor-self actions, seven AUTH-08 administrative - actions, and AUTH-09B `actor.service.provision` active. Merged AUTH-09A defines + ActionIds, with 12 active actions: the two actor-self actions, seven AUTH-08 + administrative actions, AUTH-09B `actor.service.provision`, and AUTH-09C + `actor.profile.read` plus `actor.identity_link.read`. Merged AUTH-09A defines seven fixed artifact service identities and eleven exact planned static matrix memberships. ART feature chunks supply hidden canonical behavior/resource composition. Merged @@ -53,10 +50,10 @@ admission, prepared mutation authority, and exact AUTH-only activation chunks; neither reconciliation PR activates feature behavior. - Parallel artifact checkpoint: ART-02A1, ART-02A2, and ART-02A3 merged through - PRs #127, #129, and #141. ART-02B1 remains inactive. -- Authorization checkpoint: AUTH-07B through AUTH-09C merged through PRs #130, - #131, #132, #143, and #146. Signed memory stopped after 09C, and the user - explicitly started AUTH-09D; required review split it before runtime edits. + PRs #127, #129, and #141. ART-02B1 is active and adds real MinIO protocol + proof plus a fail-closed, runtime-ineligible native AWS profile. +- Authorization checkpoint: AUTH-07B through AUTH-09D-A merged through PRs + #130, #131, #132, #143, #146, and #148. AUTH-09D-B remains inactive. - Parallel coverage work: `WS-QUAL-001-01B2` remains paused. Its last official whole-app result is `6466/8159` statements (`79.249908%`); no replacement evidence exists. diff --git a/.agent-loop/WORK_QUEUE.md b/.agent-loop/WORK_QUEUE.md index 54e88279..fc318af2 100644 --- a/.agent-loop/WORK_QUEUE.md +++ b/.agent-loop/WORK_QUEUE.md @@ -4,7 +4,7 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| -| `WS-AUTH-001-09D-A` | Profile Lifecycle And Evidence Repair | L1 | PR #148 external repair and exact-head review pass at `efc4e6c`; canonical evidence refresh pending | +| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Active after explicit user start; implementation repair and exact-head internal review in progress | Live post-merge state remains read from signed `automation/loop-memory` output. This authored queue records the separately approved parallel chunks. @@ -17,7 +17,6 @@ output. This authored queue records the separately approved parallel chunks. | `WS-AUTH-001-09D-B` | Identity-Link Lifecycle And Race Closure | L1 | Inactive until 09D-A merge/memory and explicit user start | | `WS-QUAL-001-02` | Project Service Coverage | L1 | Inactive until 01B2 merge/memory plus explicit user start | | `WS-POL-002-04` | Locked Runtime Execution And Routing Hardening | L1 | Inactive pending relevant authorization proof and a separate explicit user start | -| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Inactive until 02A3 merge and explicit user start | | `WS-ART-001-02C1` | Admission And Put-Attempt Foundation | L1 | Inactive until 02B1 merge and explicit user start | | `WS-ART-001-02C2` | Verification Publication And Fencing | L1 | Inactive until 02C1 merge and explicit user start | | `WS-ART-001-02C3` | Recovery Attempt And Idempotency Chain | L1 | Inactive until 02C2 merge and explicit user start | @@ -27,6 +26,8 @@ output. This authored queue records the separately approved parallel chunks. | Chunk | Title | Risk | Status | |---|---|---:|---| +| `WS-AUTH-001-09D-A` | Profile Lifecycle And Evidence Repair | L1 | Merged through PR #148 as `99ae4c9` on 2026-07-18 | +| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | Merged through PR #146 as `0ffdabf` on 2026-07-18 | | `WS-ENG-001-01` | Codex-native zero-trust loop bootstrap | L1 | Merged through PR #23 on 2026-06-20 | | `EXAMPLE-TERMINAL-BENCHMARK` | Terminal Benchmark example drill | L3 | Merged through PR #25 on 2026-06-21 | | `WS-POL-001-PLAN` | Submission Artifact Policy Foundation planning | L1 | Merged through PR #26 on 2026-06-27 | @@ -90,17 +91,17 @@ and its AUTH owner reconciliation merged through PR #140 as `d541521`. AUTH-09A merged through PR #132 as `299363a`, and signed schema-v2 memory stopped. AUTH-09B merged through PR #143 as `053242b`; the user then explicitly started AUTH-09C. PR #146 merged it as `0ffdabf`; signed memory at `eeb3dc2` -stopped. The user explicitly started AUTH-09D, and required review split it -before runtime edits. Only 09D-A implementation is active after exact-SHA -preimplementation review passed. Do not start 09D-B, -09E, or POL-002-04 automatically. +stopped. Required review split AUTH-09D before runtime edits, and AUTH-09D-A +merged through PR #148 as `99ae4c9`. Do not start 09D-B, 09E, or POL-002-04 +automatically. Coverage R10 merged through PR #108. Do not start 01B2, chunk 02, or another coverage implementation chunk from this worktree. `WS-ART-001-01`, the AWS-first planning amendment, `02A1`, `02A2`, and `02A3` are merged; PR #141 merged `02A3` as `a10d901`. R2 and Flow Node are deferred. -`02B1` remains inactive until a separate explicit user start. +The user explicitly started `02B1` on 2026-07-18. `02C1` remains inactive +until `02B1` merges and receives a separate explicit start. Coverage work proceeds independently in its own worktree and is not owned by this AUTH queue update. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index e99f7c80..087bd357 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -9,8 +9,8 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-OBJECT-STORAGE-AMENDMENT` | Make AWS S3 the v0.1 production provider with MinIO local/CI protocol proof; keep optional providers outside v0.1. | L1 | Merged through PR #120 as `4408256` | | `WS-ART-001-02A1` | Install only ADR 0014's small typed external-service adapter/factory foundation without migrating a capability. | L1 | Merged through PR #127 as `f64a8e5` | | `WS-ART-001-02A2` | Add bounded committed-source preparation and inactive scratch-cleanup mechanics without changing the active v1 port. | L1 | Merged through PR #129 as `9a04434` on 2026-07-16 | -| `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Internal review and deterministic evidence passed; external checks pending | -| `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Proposed after 02A3 | +| `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Merged through PR #141 as `a10d901` on 2026-07-18 | +| `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Active after 02A3 merged through PR #141 and explicit user start | | `WS-ART-001-02C1` | Add the generic durable-byte admission ledger and durable put-attempt state foundation without provider execution. | L1 | Proposed after 02B1 | | `WS-ART-001-02C2` | Add put resolution, verification publication, complete-object observation, immutable receipts, and PostgreSQL execution fencing without recovery attempts or routes. | L1 | Proposed after 02C1 | | `WS-ART-001-02C3` | Add the recovery-attempt model and exact idempotent source-job to retry-job chain without public or Operator routes. | L1 | Proposed after 02C2 | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index ce764b8d..60db8577 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -89,14 +89,18 @@ mode `aws_workload_identity` selects exactly one allowlisted method: constrains the pinned credential resolver to the selected provider before any provider is loaded, verifies the resolved method, and rejects explicit credentials, ambient access keys, file/process/login/SSO sources, legacy -EC2/Boto sources, and every unselected workload provider. Chunk 02B1 pins +EC2/Boto sources, and every unselected workload provider. For the selected +method, startup accepts only that method's exact closed `AWS_*` environment +allowlist and rejects every other `AWS_*` or `BOTOCORE_*` SDK control before +constructing a session. Chunk 02B1 pins `aiobotocore==3.7.0` and `botocore==1.43.0`; SDK upgrades require an explicit dependency and credential-behavior review. MinIO static credentials are local/CI only. The endpoint is omitted for native AWS S3 and explicit for MinIO. AWS requires an explicit region and production requires HTTPS, a non-local resolved endpoint, and backend `s3_compatible`. Secrets and resolved -credentials are never persisted or retained by errors. Cloudflare R2 has no -v0.1 runtime profile, credential service, or configuration path. +credentials are never persisted or retained by errors. Object and credential +metadata transports explicitly ignore ambient HTTP proxy variables. Cloudflare +R2 has no v0.1 runtime profile, credential service, or configuration path. ## Immutable Object Identity diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index 6cda9a86..af1a1509 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -5,14 +5,17 @@ Original planning merged through PR #97, artifact/LocalStorage foundation merged through PR #101, the AWS-first object-storage amendment merged through PR #120 as `4408256`, the external-service adapter foundation merged through -PR #127 as `f64a8e5`, and committed-source preparation merged through PR #129 -as `9a04434` on 2026-07-16. The user explicitly started -`WS-ART-001-02A3` on 2026-07-16. +PR #127 as `f64a8e5`, committed-source preparation merged through PR #129 as +`9a04434`, and the ArtifactStore v2 Local clean cut merged through PR #141 as +`a10d901` on 2026-07-18. The user explicitly started `WS-ART-001-02B1` on +2026-07-18. The planning-only cross-initiative boundary reconciliation merged through PR #139 as `5d353b6`, and AUTH's owner reconciliation merged through PR #140 as `d541521`. ART now consumes AUTH's canonical activation-custody and prepared mutation contracts without editing or activating AUTH runtime behavior. +AUTH-09D-A merged through PR #148 as `99ae4c9` and is integrated into the ART +candidate; AUTH-09D-B remains inactive. The Flow Node-focused amendment candidate `6cc422d` passed deterministic checks but failed internal review on recovery/API completeness. Before repair, the user @@ -31,29 +34,29 @@ approval or reusable evidence. Its source remains on branch ## Current Work -`WS-ART-001-02A3` implementation and merged-main deterministic repair are -complete. PR #141 is open. Exact-SHA review found and repaired a residual -LocalStorage startup race, typed-factory startup mismatch, and vague Operator -resource vocabulary. Fresh deterministic coverage and all required internal -reviewer tracks passed; external checks are now pending. Its approved -boundary atomically replaces ArtifactStore v1 with byte-only v2, migrates -LocalStorage and the empty pre-production artifact schema, installs the -immutable storage-namespace fence, -removes dormant `flow_node` configuration, and activates startup plus periodic -scratch cleanup. It does not activate product ingest, durable admission, -put-attempt resolution, verification jobs, or recovery. +`WS-ART-001-02B1` is active. It adds one `S3CompatibleArtifactStore`, runs the +shared ArtifactStore v2 vectors against real digest-pinned MinIO, and validates +an isolated native-AWS workload-identity profile. MinIO is runtime-eligible +only in local/development/test after the PostgreSQL namespace claim. Native AWS +remains runtime-ineligible and fails with +`artifact_provider_live_proof_required` before factory construction, +credential resolution, namespace claim, or provider I/O. No product ingest, +durable admission, put-attempt resolution, verification job, recovery route, +or optional-provider runtime is activated. R2 and Flow Node remain deferred. ## Next Proposed Chunk -`02B1` owns the S3-compatible adapter, MinIO proof, and AWS S3 profile after -`02A3` merges and receives a separate explicit start. There is no active R2 or -Flow Node chunk. +`02C1` owns generic durable-byte admission and put-attempt state only after +`02B1` merges and receives a separate explicit start. Neither R2 nor Flow Node +has a v0.1 chunk. ## Gate -All reviews before the latest repairs are retained as history, not reused as -final provenance. Deterministic proof and exact-SHA internal review have passed. -The current gate is GitHub Actions, CodeRabbit, and explicit human review. -Durable admission, put attempts, verification publication, and recovery remain -in their later owning chunks. No -later artifact chunk starts automatically, and only the user may approve merge. +The reviewed implementation recorded: "The current gate is deterministic 02B1 +proof followed by all nine exact-SHA internal reviewer tracks." After integrating +latest `main`, including merged REV planning, that gate passed again at +`9cd5620e` after addressing all four valid CodeRabbit findings, with no ART +runtime or ownership drift. The remaining gate is the post-fix GitHub Actions +and CodeRabbit rerun plus explicit human review. Durable admission, put attempts, +verification publication, and recovery remain in later owning chunks. No later +artifact chunk starts automatically, and only the user may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index 756acf1c..2fd08355 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-02B1 - S3-Compatible MinIO And AWS -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02A3 +Initiative: `WS-ART-001` | Risk: L1 | Status: Active after explicit human start Artifact contract phase: `artifact_store_cutover` @@ -13,10 +13,20 @@ credential-delivery contract. ## Allowed Files -- `backend/app/adapters/artifacts/s3_compatible.py` and adapter registration; +- `backend/app/adapters/artifacts/s3_compatible.py`, the shared canonical + provider-reference and S3 namespace-validation helpers, and adapter + registration; +- `backend/app/interfaces/artifacts.py` only to register the closed + `minio-v1` and `aws-s3-v1` namespace profiles, their exact descriptor keys, + and the stable AWS live-proof-required error; - AWS/MinIO settings in `backend/app/core/config.py`; -- backend dependency manifest/lock with exact `aiobotocore==3.7.0` and - `botocore==1.43.0` pins, `docker-compose.yml`, and +- `backend/app/workers/celery_app.py` and `backend/app/workers/artifacts.py` + only to enforce the shared AWS runtime-ineligible guard before Celery startup + or artifact scratch cleanup; +- `backend/pyproject.toml` with exact `aiobotocore==3.7.0` and + `botocore==1.43.0` pins; this repository has no backend dependency lockfile, + so dependency tests must inspect the exact manifest pins and installed + versions; `docker-compose.yml`, and `.github/workflows/backend.yml` for MinIO and exact coverage proof; - `scripts/test_agent_gates.py` only to assert the exact workflow command, source set, threshold, and cumulative retention; @@ -44,6 +54,9 @@ credential-delivery contract. - ETag/provider checksums are not Workstream integrity facts; - native AWS endpoint omission with explicit region and MinIO endpoint/region validate; production rejects HTTP/local endpoints and static credentials; +- the v0.1 AWS profile accepts global general-purpose buckets only and rejects + account-regional bucket names ending in `-an` because Workstream does not + emit the account-regional namespace request header; - AWS production credential mode `aws_workload_identity` selects exactly one of `assume-role-with-web-identity`, `container-role`, or `iam-role`; the credential resolver contains only that provider and the resolved method must @@ -51,6 +64,15 @@ credential-delivery contract. - explicit credentials, environment access keys, shared credential/config files, credential processes, login/SSO, legacy EC2/Boto sources, and every unselected workload provider fail startup before credential loading; +- each selected workload method accepts only its exact closed `AWS_*` + environment allowlist; every other `AWS_*` or `BOTOCORE_*` SDK behavior + control fails before SDK session construction; +- S3 object operations and credential metadata operations ignore ambient HTTP + proxy variables so signed requests, credentials, and artifact bytes cannot + be redirected outside the configured provider transport; +- container role accepts only the relative metadata path, IAM role rejects a + custom metadata endpoint and requires IMDSv2, and the isolated IAM fetcher + disables IMDSv1; - isolation tests poison nonselected file, process, cache, metadata, and network sources and prove none is accessed; MinIO static credentials are local/CI only and never survive error/log object graphs; @@ -59,8 +81,13 @@ credential-delivery contract. - real digest-pinned MinIO and LocalStorage pass the same v2 conformance suite; - the application factory can instantiate only LocalStorage and MinIO in this chunk. Valid AWS configuration remains runtime-ineligible with the stable - `artifact_provider_live_proof_required` startup failure; only Chunk 07 may - add the immutable activation record and production composition guard; + `artifact_provider_live_proof_required` startup failure. The composition root + and Celery scheduler/task entry points raise that typed error after settings + validation but before factory construction, scratch cleanup, namespace claim, + credential resolver construction, credential loading, or provider I/O. Tests + prove no namespace row is persisted and no credential source is touched. Only + Chunk 07 may add the immutable activation record and production composition + guard; - startup and operation tests reject configured adapter/profile/namespace mismatch against persisted deployment or replica identity; no request is routed to a second namespace; @@ -73,6 +100,30 @@ credential-delivery contract. repository command below; `scripts/test_agent_gates.py` fails on workflow command, source-set, threshold, or cumulative-retention drift. +## Canonical Namespace Profiles + +The shared namespace value object accepts only these additional profiles: + +```text +minio-v1: + addressing_style + bucket + endpoint_identity + private_prefix + region + +aws-s3-v1: + addressing_style + bucket + private_prefix + region +``` + +`endpoint_identity` is a canonical non-secret hash of the normalized MinIO +endpoint, not credential material. Native AWS has no configured endpoint and +therefore no endpoint descriptor key. Descriptor key ordering remains +canonical and exact; unknown, missing, or extra keys fail closed. + ## Exact CI Coverage Gate ```bash @@ -81,6 +132,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 ``` ## Verification @@ -88,7 +141,7 @@ coverage report --include='app/main.py' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio (cd backend && .venv/bin/ruff check app tests) -(cd backend && .venv/bin/pytest tests/test_artifact_store_conformance.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) +(cd backend && .venv/bin/pytest tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_preparation.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov=app.core.s3_validation --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) python3 scripts/check_stale_artifact_contracts.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md index 3306c578..75b25d2e 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md @@ -1,4 +1,4 @@ -# Chunk Contract: WS-ART-001-02C1 Admission And Put-Attempt Foundation +# Chunk Contract: WS-ART-001-02C1 - Admission And Put-Attempt Foundation Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02B1 @@ -71,6 +71,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md index 26db8e48..9d434824 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md @@ -86,6 +86,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md index b2901004..6229a0f9 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md @@ -63,6 +63,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md index ee82beaf..e58f45d7 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md @@ -129,6 +129,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md index cc146e27..93000b35 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md @@ -109,6 +109,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md index 58d4ba62..e07b2d90 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md @@ -89,6 +89,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md index ad8e193e..3da38010 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md @@ -90,6 +90,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md index d3038977..c66fc262 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md @@ -95,6 +95,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md index fc02995c..85147cf3 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md @@ -76,6 +76,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md index 20a634d1..e4139c55 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md @@ -84,6 +84,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md index 8b9574a3..adeb5ec1 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md @@ -126,6 +126,8 @@ coverage report --include='app/interfaces/external_services.py' --precision=2 -- coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 coverage report --include='app/api/router.py' --precision=2 --fail-under=90 coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md new file mode 100644 index 00000000..c3a624a3 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md @@ -0,0 +1,64 @@ +# External Review Response: WS-ART-001-02B1 + +## Current State + +- Pull request: #151, open and not draft +- Reviewed implementation SHA: `9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a` +- Published evidence-bound SHA: `2ba3af979b599ca0104e3a658e12137996a0f039` +- CodeRabbit's four implementation findings: fixed and automatically resolved +- Post-fix incremental CodeRabbit review: complete; two evidence-wording + findings fixed, final verification pending +- GitHub Agent Gates: passed on the repaired candidate +- GitHub Backend: pending on the repaired candidate +- Human merge approval: pending + +Internal sub-agent review is recorded only in the separate internal-review +evidence. This file records CodeRabbit, GitHub, and later human PR feedback. + +## CodeRabbit Findings Addressed + +| Finding | Status | Response | +|---|---:|---| +| A complete source chunk was copied into the pending request buffer before applying the configured bound. | Fixed | `_CommittedSourceBody` retains unconsumed bytes through a memoryview cursor and copies only the amount requested. Direct regression proof verifies the copied pending buffer remains bounded while later reads receive the remaining bytes. | +| Deferred AWS workload credentials could be reported as resolved before token or role materialization. | Fixed | Explicit credential resolution now awaits `get_frozen_credentials()` after exact method validation and maps refresh failure to the same sanitized configuration error. | +| Equivalent IPv6 MinIO endpoint spellings produced different namespace identities. | Fixed | IPv6 literals are parsed and compressed before normalized endpoint and namespace identity derivation. | +| Secret-retention assertions did not inspect byte buffers or public object state. | Fixed | The shared assertion now checks bytes, bytearray, memoryview, nested `__dict__` state, and Pydantic private state, with direct negative tests. | +| Credential-materialization evidence wording was grammatically ambiguous. | Fixed | The sentence now identifies "the selected AWS workload credentials" unambiguously. | +| The real-service-focused test-count modifier was not hyphenated. | Fixed | The compound modifier is now hyphenated. | + +CodeRabbit marked all four implementation review threads resolved after commits +`9cd5620` through `2ba3af9`. Its post-fix review found only the two evidence +wording corrections recorded above. + +## Findings Not Applied + +CodeRabbit's generic pre-merge widget reported 45.14 percent docstring coverage +against its 80 percent default. The repository's authoritative configured +docstring gate is 92.0 percent and passed in GitHub before the repairs. No +duplicate or weaker docstring gate was added. + +The widget also reported that the PR description did not follow the repository +trust-bundle template. The PR body is being replaced with the current reviewed +chunk, evidence, reviewer, gate, and human-checkpoint data. + +## Validation After Repairs + +```text +443 real-service focused tests: PASS +S3CompatibleArtifactStore coverage: 91% +S3 validation coverage: 97% +Combined changed-subsystem coverage: 92.52% +Ruff: PASS +Dependency integrity: PASS +Agent gates: 88 PASS +Stale artifact/authorization/review/Workstream scans: PASS +Markdown links and Compose validation: PASS +All nine exact-SHA internal reviewer tracks: PASS +Open sub-agent sessions: none +``` + +## Remaining External Gate + +Wait for GitHub Backend and final CodeRabbit verification on the published head, +then verify that no unresolved actionable thread remains. Stop for the user's +explicit merge decision. Do not start `WS-ART-001-02C1` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md new file mode 100644 index 00000000..877385e2 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-internal-review-evidence.md @@ -0,0 +1,154 @@ +# Internal Review Evidence: WS-ART-001-02B1 + +## Chunk + +`WS-ART-001-02B1`: S3-Compatible MinIO And AWS + +open sub-agent sessions: none + +valid findings addressed: yes + +## Reviewed Revision + +Reviewed code SHA: `9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a` + +Reviewed at: 2026-07-18T23:40:43Z + +Reviewer run IDs: senior-engineering=019f778b-6b07-7561-a718-e1da40a532a0; architecture=019f778b-6372-7c83-85ca-a61f63885c17; QA/test=019f778b-74eb-7250-b07c-573c979b2849; security/auth=019f778b-81f0-7b31-85d5-8955f9047b3e; product/ops=019f7791-3326-71e3-b838-54143b94a129; reuse/dedup=019f7791-3943-7de3-bdbf-c3643f45c43c; CI-integrity=019f7791-43a6-78e0-8f17-67861bf27455; test-delta=019f7791-4c52-7091-9076-f534bccdfad6; docs=019f7797-215a-7e02-b6e4-c33b584ec2b5 + +The reviewed base is `origin/main` at +`983b9e534b84f1590fafecc0ce1355cf131257ce`, including AUTH PR #148 and the +latest merged REV planning. The merged REV planning changes introduce no ART +runtime or ownership drift in `origin/main...HEAD`. +Only review artifacts and initiative status may change after the reviewed SHA. +Any implementation, test, workflow, policy, or chunk-contract change invalidates +this evidence and requires a new exact-head review cycle. + +## Reviewed Change + +- Added one S3-protocol ArtifactStore implementation shared by MinIO and the + inactive native AWS profile. +- Proved the active local/CI path against real digest-pinned MinIO with the same + ArtifactStore v2 contract used by LocalStorage. +- Kept native AWS runtime-ineligible until Chunk 07 supplies an immutable live + activation proof. +- Isolated each AWS workload-identity method from ambient credential, proxy, + file, process, cache, and unselected metadata sources. +- Preserved server-derived content keys, sealed source validation, exact replay, + bounded object size, no multipart/list/delete/copy surface, and sanitized + provider failures. +- Added canonical global general-purpose bucket, region, prefix, and MinIO + endpoint validation, including rejection of secret-bearing endpoint input. +- Retained the full repository 78 percent CI floor and every cumulative focused + 90 percent gate while adding exact S3 adapter and validation gates. +- Integrated AUTH PR #148 without editing or activating AUTH-owned runtime + behavior. +- Integrated the latest merged REV planning without editing or activating + REV-owned runtime behavior. +- Bound repository-managed MinIO to host loopback in Compose and CI and added + deterministic gates for both definitions. +- Defined `local/CI MinIO` as non-production eligibility, including private + operator-controlled container-network development/test endpoints only. +- Documented and enforced the exact `minio-v1` and `aws-s3-v1` namespace + descriptor schemas. +- Bounded copied request-body bytes independently of source chunk size by + retaining unconsumed source bytes through a memoryview cursor. +- Required the selected AWS workload credentials to materialize successfully before + explicit resolution can report success. +- Canonicalized equivalent IPv6 MinIO literals before deriving endpoint and + namespace identity. +- Extended secret-retention proof to byte buffers and public object state. + +## Reviewer Results + +| Reviewer | Result | Blocking findings | Notes | +|---|---:|---|---| +| senior engineering | PASS WITH LOW RISKS | None | The shared adapter and four external-review repairs are maintainable. Exact SDK pins bound the reviewed reliance on SDK credential internals. | +| qa/test | PASS WITH LOW RISKS | None | Real MinIO, credential isolation, bounded streaming, IPv6 identity, secret scanning, and CI evidence cover the chunk contract without weakened tests. | +| security/auth | PASS | None | Secret-bearing endpoint and metadata failures are sanitized, credential sources are closed, and ART/AUTH ownership remains intact. | +| product/ops | PASS | None | No operator recovery, contributor, reviewer, contribution, compensation, fulfillment, or reputation behavior is activated. | +| architecture | PASS WITH LOW RISKS | None | One typed adapter path remains, concrete S3 construction stays in ART composition, and exact SDK pins bound the inactive AWS helper dependency. | +| ci integrity | PASS | None | The 78 percent repository gate and all cumulative 90 percent gates remain fail closed with no bypass. | +| docs | PASS | None | README, artifact specification, settings, provider eligibility, and loop state match the implementation. | +| reuse/dedup | PASS WITH LOW RISKS | None | Provider references, S3 validation, runtime guards, and deep secret assertions reuse canonical helpers without parallel implementations. The reviewer relied on the recorded real-MinIO proof rather than rerunning it. | +| test delta | PASS WITH LOW RISKS | None | Active-MinIO and inactive-AWS proof is stronger, including all four repairs; no test was skipped or weakened. AWS remains intentionally inactive pending live proof. | + +## Valid Findings Addressed + +- Replaced permissive S3 region validation with a bounded canonical region + grammar and complete positive/negative tests. +- Made invalid MinIO endpoint errors discard userinfo, malformed ports, and raw + source values across constructor, Pydantic mapping/JSON/string validation, + environment, dotenv, traceback, and structured error paths. +- Documented the intentional v0.1 global general-purpose bucket boundary and + rejection of account-regional `-an` names because Workstream does not emit + the account-regional namespace header. +- Rejected every AWS reserved general-purpose bucket prefix and suffix and + validated every dotted bucket label. +- Sanitized ECS and IMDS credential failure paths so SDK exceptions cannot + retain response bodies, authorization tokens, endpoints, or credentials. +- Bound replay to the sealed committed source before any provider operation and + removed stale provider-reference divergence. +- Reconciled authored ART loop state and work queue with AUTH PR #148 while + keeping AUTH-09D-B and every later ART chunk inactive. +- Corrected the successor chunk heading so merge-intent validation resolves + `WS-ART-001-02C1` without changing explicit-start authority. +- Consolidated duplicate mapping validators while preserving distinct Pydantic + entry points and clearing credentials from every ordinary exception path. +- Rejected non-object settings JSON before Pydantic can retain secret-bearing + input in structured errors and cleared partial S3 secret extraction on later + source failures. +- Bound repository-managed MinIO to loopback in Compose and CI and made both + definitions regression-protected by the 88-test agent-gate runner. +- Clarified private non-production MinIO eligibility and added the exact closed + S3 namespace profile table to the canonical artifact specification. +- Replaced whole-source-chunk pending copies with a bounded memoryview cursor and + added direct proof that unconsumed bytes remain available without expanding + the copied request buffer. +- Materialized deferred AWS workload credentials before returning explicit + resolution success and sanitized refresh failures. +- Compressed equivalent IPv6 endpoint literals before namespace identity + derivation. +- Extended deep secret assertions across bytes, bytearray, memoryview, and + nested public object state. + +## Commands Run + +```bash +docker compose up -d --wait postgres redis minio +cd backend && .venv/bin/ruff check app tests +cd backend && .venv/bin/pip check +cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test WORKSTREAM_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/python -m pytest tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_preparation.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py tests/test_assertion_helpers.py tests/test_audit.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov=app.core.s3_validation --cov-report=term-missing --cov-fail-under=90 +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_stale_authorization_docs.py +python3 scripts/check_stale_review_contracts.py +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +Results: 443 real-service-focused tests passed after the external-review repairs. +`S3CompatibleArtifactStore` coverage is 91 percent, S3 validation coverage is +97 percent, and combined changed-subsystem coverage is 92.52 percent. Ruff, +dependency integrity, stale contract/authorization/review/wording scans, +Markdown links, 88 agent-gate tests, Compose validation, and diff checks passed. + +The isolated full repository suite and 78 percent whole-app floor remain +authoritative in GitHub Backend CI, as explicitly chosen because local execution +is prohibitively slow. The CI reviewer confirmed that exact gate is retained. + +## Remaining Risks + +- Native AWS operations remain deliberately unavailable until Chunk 07 adds the + release-bound live proof and production composition guard. +- MinIO is non-production protocol proof, including private local/dev/test + container-network use; it is not production activation evidence. +- Provider acknowledgement remains non-bindable pending later admission, + verification, publication, and recovery chunks. + +## Stop Condition + +Publish this evidence-bound candidate for GitHub CI, CodeRabbit, and explicit +human review. Do not merge without the user's approval and do not start +`WS-ART-001-02C1` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md new file mode 100644 index 00000000..7e5ab24a --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-pr-trust-bundle.md @@ -0,0 +1,115 @@ +# PR Trust Bundle: WS-ART-001-02B1 + +## Chunk + +`WS-ART-001-02B1` - S3-Compatible MinIO And AWS + +Merge intent: `.agent-loop/merge-intents/WS-ART-001-02B1.json` + +## Goal + +Add one provider-neutral S3 protocol implementation, prove it against real +MinIO for local/CI use, and validate an isolated native AWS configuration +without making AWS runtime-eligible before its separately owned live-proof +chunk. + +## What Changed + +- Added `S3CompatibleArtifactStore` behind the existing ArtifactStore v2 and + typed external-service factory boundaries. +- Added real digest-pinned MinIO to local/CI composition and shared conformance. +- Added closed AWS workload-identity configuration and isolation proof while + preserving the stable pre-I/O `artifact_provider_live_proof_required` guard. +- Added canonical S3 namespace, region, bucket, prefix, and endpoint validation. +- Added sanitized credential/metadata/endpoint failure paths and deep + traceback/object-graph tests. +- Added exact async SDK pins and cumulative CI coverage guards. +- Integrated AUTH PR #148 while leaving all AUTH runtime activation with AUTH. +- Integrated the latest merged REV planning without changing ART or REV runtime + ownership. +- Bounded source-chunk copying, materialized deferred AWS credentials, + canonicalized IPv6 endpoint identity, and deepened secret-retention proof in + response to the valid external review findings. + +## Scope Control + +The chunk adds no product route, upload admission, put-attempt state, +verification publication, recovery operation, presigned URL, multipart upload, +list/delete/copy capability, Flow Node path, R2 profile, or authorization +activation. `WS-ART-001-02C1` remains inactive until this chunk merges and the +user explicitly starts it. + +## Acceptance Proof + +- [x] Server-derived sealed commitment controls every provider object key. +- [x] Exact replay and conditional no-overwrite semantics are deterministic. +- [x] Object size fails before provider I/O above 512 MiB. +- [x] Real LocalStorage and MinIO pass the same ArtifactStore v2 vectors. +- [x] AWS credential resolution is limited to one explicitly selected workload + source with ambient providers, proxies, and behavior controls rejected. +- [x] Native AWS fails before factory, credentials, namespace, scratch, or I/O. +- [x] ART/AUTH ownership remains separated after AUTH PR #148 integration. +- [x] Exactly one schema-v2 merge intent names only the same-initiative 02C1 + successor and requires a separate explicit start. + +## Tests And Checks + +```text +443 real-service focused tests PASS +S3CompatibleArtifactStore coverage 91% +S3 validation coverage 97% +Combined changed-subsystem coverage 92.52% +Ruff PASS +pip check PASS +88 agent-gate tests PASS +Stale artifact/authorization/review/Workstream wording PASS +Markdown links PASS +git diff --check PASS +``` + +GitHub Backend CI remains authoritative for the isolated full suite, the 78 +percent repository floor, and all cumulative focused 90 percent gates. + +## Internal Review + +Reviewed code SHA: `9cd5620ef5f72e7ba9abc75e9ac7b398996f0c8a` + +All nine required reviewer tracks passed the exact integrated SHA with no +remaining finding. Every reviewer session is closed. Reviewer IDs and the +addressed repair history are recorded in the internal review evidence. + +## External Review + +External review response file: + +- `.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02B1-external-review-response.md` + +| Source | Status | Notes | +|---|---:|---| +| GitHub Actions | Pending | Agent Gates passed; Backend reruns the isolated full suite, repository floor, cumulative focused gates, and real MinIO proof on the repaired candidate. | +| CodeRabbit | Final verification pending | Four implementation findings were fixed and resolved; the post-fix review found only two evidence-wording corrections, now fixed. | +| Human review | Pending | Only the user may approve merge. | + +## Remaining Risks + +- AWS is configuration-ready but intentionally unavailable until Chunk 07. +- MinIO proves protocol behavior only for non-production local, development, + test, and CI environments; repository-managed instances are loopback-bound. +- Durable admission, verification, publication, and recovery are later chunks. + +## Human Review Focus + +- Is there truly one S3 protocol adapter rather than provider-specific forks? +- Can untrusted input influence object identity, credential sources, endpoints, + proxies, or namespace selection? +- Does native AWS fail before every credential and provider side effect? +- Are ART and AUTH responsibilities still separate after PR #148 integration? +- Are all later ART capabilities still inactive? + +## Human Merge Ownership + +- [ ] I can explain what changed and why. +- [ ] I know what could break. +- [ ] I accept the remaining risks. +- [ ] GitHub CI and external review pass. +- [ ] The user explicitly approved this PR for merge. diff --git a/.agent-loop/merge-intents/WS-ART-001-02B1.json b/.agent-loop/merge-intents/WS-ART-001-02B1.json new file mode 100644 index 00000000..923a9476 --- /dev/null +++ b/.agent-loop/merge-intents/WS-ART-001-02B1.json @@ -0,0 +1,9 @@ +{ + "chunk_id": "WS-ART-001-02B1", + "chunk_title": "S3-Compatible MinIO And AWS", + "initiative_id": "WS-ART-001", + "next_chunk_id": "WS-ART-001-02C1", + "next_chunk_title": "Admission And Put-Attempt Foundation", + "next_requires_explicit_start": true, + "schema_version": 2 +} diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index 83b071e2..06988490 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -45,6 +45,23 @@ jobs: python -m pip install --upgrade pip python -m pip install -e ".[dev]" + - name: Start real MinIO artifact provider + run: | + image='quay.io/minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e' + docker run --detach --rm --name workstream-minio \ + --publish 127.0.0.1:9000:9000 \ + --env MINIO_ROOT_USER=workstream-minio \ + --env MINIO_ROOT_PASSWORD=workstream-minio-secret-key \ + "$image" server /data --address :9000 + for attempt in $(seq 1 60); do + if curl --fail --silent http://127.0.0.1:9000/minio/health/live >/dev/null; then + exit 0 + fi + sleep 1 + done + docker logs workstream-minio + exit 1 + - name: Lint working-directory: backend run: ruff check app tests scripts @@ -63,6 +80,7 @@ jobs: working-directory: backend env: WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres + WORKSTREAM_TEST_MINIO_ENDPOINT: http://127.0.0.1:9000 run: | metadata_dir="$(mktemp -d)" trap 'rm -rf "$metadata_dir"' EXIT @@ -108,6 +126,22 @@ jobs: --precision=2 --fail-under=90 + - name: S3-compatible artifact adapter coverage + working-directory: backend + run: >- + coverage report + --include='app/adapters/artifacts/s3_compatible.py' + --precision=2 + --fail-under=90 + + - name: S3 namespace validation coverage + working-directory: backend + run: >- + coverage report + --include='app/core/s3_validation.py' + --precision=2 + --fail-under=90 + - name: Actor subsystem coverage working-directory: backend run: >- diff --git a/README.md b/README.md index d174488a..b882766b 100644 --- a/README.md +++ b/README.md @@ -188,13 +188,30 @@ the repository is changed; it does not define runtime task or review records. ## Local Backend Database Workstream uses Postgres locally and in CI. It uses Celery with Redis for -durable local project setup jobs and automatic pre-review checker gates. Start -local services with: +durable local project setup jobs and automatic pre-review checker gates. MinIO +provides the S3-compatible artifact protocol in local development and CI. Start +the local services with: ```bash -docker compose up -d postgres redis +docker compose up -d postgres redis minio ``` +MinIO uses the compose-only static credentials and the private +`workstream-artifacts` bucket. The integration tests create that bucket +automatically. For local runtime use, create the private bucket with an S3 +client against `http://localhost:9000` after MinIO is healthy, using access key +`workstream-minio` and secret key `workstream-minio-secret-key`, before starting +Workstream. Configure the runtime with the exact +[artifact storage settings](docs/spec_artifact_storage_service.md#s3-compatible-adapter). +The repository-managed MinIO port is bound to host loopback. A Workstream +process running on a separate non-production container network may instead use +an operator-controlled private MinIO endpoint; that remains development/test +protocol proof and never qualifies as hosted-provider activation evidence. +Native AWS S3 accepts workload-identity configuration but remains +runtime-ineligible until live deployment proof is approved; startup fails with +`artifact_provider_live_proof_required` before credential probing or provider +I/O. + The default local development URL is: ```text diff --git a/backend/app/adapters/artifacts/__init__.py b/backend/app/adapters/artifacts/__init__.py index 1805b7d1..9baf5ef4 100644 --- a/backend/app/adapters/artifacts/__init__.py +++ b/backend/app/adapters/artifacts/__init__.py @@ -6,6 +6,7 @@ from app.interfaces.artifacts import ( ARTIFACT_STORE_CAPABILITY_KEY, ArtifactConfigurationError, + ArtifactProviderLiveProofRequiredError, ArtifactStoreBootstrap, ) from app.interfaces.external_services import ExternalServiceAdapterFactory @@ -27,7 +28,12 @@ def create_artifact_store_bootstrap(settings: Settings) -> ArtifactStoreBootstra Raises: ExternalServiceConfigurationError: If the provider is not registered. """ + require_artifact_runtime_eligible(settings) + from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap + from app.adapters.artifacts.s3_compatible import ( + create_minio_artifact_store_bootstrap, + ) factory = ExternalServiceAdapterFactory[ArtifactStoreBootstrap]( ARTIFACT_STORE_CAPABILITY_KEY @@ -46,9 +52,24 @@ def create_local_store() -> ArtifactStoreBootstrap: ) factory.register("local", create_local_store) + factory.register( + "s3_compatible", + lambda: create_minio_artifact_store_bootstrap(settings), + ) return factory.create(settings.artifact_store_backend) +def require_artifact_runtime_eligible(settings: Settings) -> None: + """Reject configured providers that this chunk has not activated.""" + if ( + settings.artifact_store_backend == "s3_compatible" + and settings.artifact_s3_provider_profile == "aws_s3" + ): + raise ArtifactProviderLiveProofRequiredError( + "AWS artifact provider requires live deployment proof" + ) + + def artifact_preparation_limits(settings: Settings) -> ArtifactPreparationLimits: """Map validated settings to the one process-independent scratch contract.""" return ArtifactPreparationLimits( @@ -76,6 +97,7 @@ def create_artifact_scratch_manager(settings: Settings) -> ArtifactScratchManage async def cleanup_stale_artifact_scratch(settings: Settings) -> int: """Run one database-independent stale cleanup with shared construction.""" + require_artifact_runtime_eligible(settings) manager = create_artifact_scratch_manager(settings) try: return await manager.cleanup_stale() diff --git a/backend/app/adapters/artifacts/local.py b/backend/app/adapters/artifacts/local.py index 638780ae..e4889147 100644 --- a/backend/app/adapters/artifacts/local.py +++ b/backend/app/adapters/artifacts/local.py @@ -20,6 +20,10 @@ await_cancellation_resistant, run_blocking_cancellation_resistant, ) +from app.adapters.artifacts.references import ( + artifact_provider_object_ref, + parse_artifact_provider_object_ref, +) from app.core.file_locks import acquire_exclusive_file_lock from app.interfaces.artifacts import ( ARTIFACT_STORE_CAPABILITY_KEY, @@ -53,7 +57,6 @@ _LAYOUT_ENTRY_NAMES = frozenset( {_LAYOUT_MARKER, _LAYOUT_MARKER_TEMPORARY, "locks", "objects", "tmp"} ) -_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") _DIGEST_PREFIX_NAME = re.compile(r"^[0-9a-f]{2}$") _DIGEST_OBJECT_NAME = re.compile(r"^[0-9a-f]{62}$") _TEMPORARY_NAME = re.compile(r"^\.put\.[0-9a-f]{32}\.tmp$") @@ -218,7 +221,7 @@ async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: if commitment.byte_count > HARD_MAXIMUM_ARTIFACT_BYTES: raise ArtifactLimitExceededError("artifact source exceeds provider limit") - provider_object_ref = self._provider_object_ref(commitment) + provider_object_ref = artifact_provider_object_ref(commitment) lock: tuple[Any, int] | None = None descriptor: int | None = None temporary_name: str | None = None @@ -304,7 +307,7 @@ async def observe_put_result( self._require_initialized() if type(commitment) is not ArtifactCommitment: raise ArtifactOperationConflictError("artifact commitment is invalid") - provider_object_ref = self._provider_object_ref(commitment) + provider_object_ref = artifact_provider_object_ref(commitment) observed = await self.head(provider_object_ref) if not observed.exists: return ArtifactPutObservation(provider_object_ref, committed=False) @@ -318,7 +321,7 @@ def open( ) -> AsyncIterator[bytes]: """Stream a full object or range through bounded off-loop reads.""" self._require_initialized() - self._parse_provider_object_ref(provider_object_ref) + parse_artifact_provider_object_ref(provider_object_ref) if byte_range is not None and type(byte_range) is not ArtifactByteRange: raise ArtifactOperationConflictError("artifact byte range is invalid") selected_range = byte_range or ArtifactByteRange() @@ -371,7 +374,7 @@ async def iterate() -> AsyncIterator[bytes]: async def head(self, provider_object_ref: str) -> ArtifactObjectHead: """Return an existing or missing observation without exposing paths.""" self._require_initialized() - self._parse_provider_object_ref(provider_object_ref) + parse_artifact_provider_object_ref(provider_object_ref) try: try: byte_count = await self._run_io( @@ -743,21 +746,6 @@ def _stat_layout_file( finally: os.close(descriptor) - def _provider_object_ref(self, commitment: ArtifactCommitment) -> str: - """Derive one identity-free object reference from canonical SHA-256.""" - digest_hex = commitment.sha256[7:] - return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" - - @staticmethod - def _parse_provider_object_ref(provider_object_ref: str) -> tuple[str, str]: - """Validate the exact opaque local provider-reference grammar.""" - if not isinstance(provider_object_ref, str): - raise ArtifactOperationConflictError("artifact provider reference is invalid") - match = _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) - if match is None: - raise ArtifactOperationConflictError("artifact provider reference is invalid") - return match.group(1), match.group(2) - def _open_prefix(self, prefix: str, *, create: bool) -> int: """Open one digest-prefix directory without following links.""" if create: @@ -802,7 +790,7 @@ def _seal_temporary(descriptor: int) -> None: def _publish_exclusive(self, temporary_name: str, provider_object_ref: str) -> bool: """Hard-link one complete object without ever replacing existing bytes.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) prefix_fd = self._open_prefix(prefix, create=True) try: try: @@ -832,19 +820,19 @@ def _head_object(self, provider_object_ref: str) -> int: def _head_object_with_recovery(self, provider_object_ref: str) -> int: """Recover an interrupted publication under its digest lock before head.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) with self._locked_file(prefix + filename): return self._head_object(provider_object_ref) def _open_object_with_recovery(self, provider_object_ref: str) -> tuple[int, int]: """Recover an interrupted publication under its digest lock before open.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) with self._locked_file(prefix + filename): return self._open_object(provider_object_ref) def _open_object(self, provider_object_ref: str) -> tuple[int, int]: """Open one immutable no-follow object and return its exact size.""" - prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_object_ref) prefix_fd = self._open_prefix(prefix, create=False) descriptor = -1 try: diff --git a/backend/app/adapters/artifacts/references.py b/backend/app/adapters/artifacts/references.py new file mode 100644 index 00000000..3ac7cd5b --- /dev/null +++ b/backend/app/adapters/artifacts/references.py @@ -0,0 +1,29 @@ +"""Canonical provider-object references shared by artifact adapters.""" + +from __future__ import annotations + +import re + +from app.interfaces.artifacts import ArtifactOperationConflictError +from app.modules.artifacts.sources import ArtifactCommitment + + +_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") + + +def artifact_provider_object_ref(commitment: ArtifactCommitment) -> str: + """Derive one identity-free reference solely from a server commitment.""" + if type(commitment) is not ArtifactCommitment: + raise ArtifactOperationConflictError("artifact commitment is invalid") + digest_hex = commitment.sha256[7:] + return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" + + +def parse_artifact_provider_object_ref(provider_object_ref: str) -> tuple[str, str]: + """Return digest path parts after enforcing the canonical grammar.""" + if not isinstance(provider_object_ref, str): + raise ArtifactOperationConflictError("artifact provider reference is invalid") + matched = _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) + if matched is None: + raise ArtifactOperationConflictError("artifact provider reference is invalid") + return matched.group(1), matched.group(2) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py new file mode 100644 index 00000000..ff5262d3 --- /dev/null +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -0,0 +1,1188 @@ +"""Private S3-protocol implementation of the immutable ArtifactStore v2 port.""" + +from __future__ import annotations + +import asyncio +from collections.abc import AsyncIterator, Mapping +from contextlib import asynccontextmanager +import datetime +import hashlib +import inspect +import json +import os +from pathlib import Path +import random +import re +from typing import Any + +from aiobotocore.config import AioConfig +from aiobotocore.credentials import ( + AioAssumeRoleWithWebIdentityProvider, + AioContainerProvider, + AioCredentialResolver, + AioInstanceMetadataProvider, + AioRefreshableCredentials, +) +from aiobotocore.session import AioSession +from aiobotocore.utils import ( + AioContainerMetadataFetcher, + AioInstanceMetadataFetcher, + _RefCountedSession, +) +from botocore import UNSIGNED +from botocore.awsrequest import AWSRequest +from botocore.exceptions import ( + BotoCoreError, + ClientError, + CredentialRetrievalError, + HTTPClientError, + InvalidIMDSEndpointError, + MetadataRetrievalError, + ReadTimeoutError, +) +from botocore.utils import ( + BadIMDSRequestError, + RETRYABLE_HTTP_ERRORS, + get_current_datetime, +) + +from app.adapters.artifacts.references import ( + artifact_provider_object_ref, + parse_artifact_provider_object_ref, +) +from app.core.config import Settings +from app.core.hashing import canonical_json_hash +from app.core.s3_validation import ( + canonical_minio_endpoint, + is_canonical_s3_bucket, + is_canonical_s3_prefix, + is_canonical_s3_region, +) +from app.interfaces.artifacts import ( + ARTIFACT_STORE_CAPABILITY_KEY, + ArtifactByteRange, + ArtifactConfigurationError, + ArtifactInputMismatchError, + ArtifactIntegrityError, + ArtifactLimitExceededError, + ArtifactObjectHead, + ArtifactObjectMissingError, + ArtifactOperationConflictError, + ArtifactPutObservation, + ArtifactPutResult, + ArtifactRangeInvalidError, + ArtifactStoreError, + ArtifactStoreNamespaceClaim, + ArtifactStoreNamespaceIdentity, + ArtifactStoreUnavailableError, + artifact_store_namespace_material, +) +from app.interfaces.external_services import ExternalServiceAdapterIdentity +from app.modules.artifacts.preparation import HARD_MAXIMUM_ARTIFACT_BYTES +from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource + + +_IDENTITY = ExternalServiceAdapterIdentity(ARTIFACT_STORE_CAPABILITY_KEY, "s3_compatible") +_CONTAINER_RELATIVE_URI = re.compile( + r"^/v2/credentials/[A-Za-z0-9._-]{1,512}$" +) +_PRECONDITION_ERROR_CODES = frozenset( + {"409", "412", "ConditionalRequestConflict", "PreconditionFailed"} +) +_MISSING_ERROR_CODES = frozenset({"404", "NoSuchKey", "NotFound"}) +_FORBIDDEN_AWS_CREDENTIAL_ENVIRONMENT = frozenset( + { + "AWS_ACCESS_KEY", + "AWS_ACCESS_KEY_ID", + "AWS_CONFIG_FILE", + "AWS_CREDENTIAL_FILE", + "AWS_DEFAULT_PROFILE", + "AWS_LOGIN_CACHE_DIRECTORY", + "AWS_PROFILE", + "AWS_SECRET_ACCESS_KEY", + "AWS_SECRET_KEY", + "AWS_SECURITY_TOKEN", + "AWS_SESSION_TOKEN", + "AWS_SHARED_CREDENTIALS_FILE", + "BOTO_CONFIG", + } +) + + +class _SanitizedContainerMetadataFetcher(AioContainerMetadataFetcher): + """Fetch ECS credentials without SDK logging response or exception data.""" + + async def _retrieve_credentials( + self, + full_url: str, + extra_headers: Mapping[str, str] | None = None, + ) -> dict[str, Any]: + headers = {"Accept": "application/json"} + if extra_headers is not None: + headers.update(extra_headers) + failed = False + for attempt in range(self.RETRY_ATTEMPTS): + try: + return await self._get_response( + full_url, + headers, + self.TIMEOUT_SECONDS, + ) + except MetadataRetrievalError: + if attempt + 1 >= self.RETRY_ATTEMPTS: + failed = True + break + await self._sleep(self.SLEEP_TIME) + if failed: + raise MetadataRetrievalError( + error_msg="container metadata request failed" + ) from None + raise MetadataRetrievalError(error_msg="container metadata request failed") + + async def _get_response( + self, + full_url: str, + headers: Mapping[str, str], + timeout: float, + ) -> dict[str, Any]: + try: + async with self._session.acquire() as session: + request = AWSRequest(method="GET", url=full_url, headers=dict(headers)) + response = await session.send(request.prepare()) + response_text = (await response.content).decode("utf-8") + if response.status_code != 200: + raise MetadataRetrievalError( + error_msg="container metadata returned a non-200 response" + ) + try: + parsed = json.loads(response_text) + except ValueError: + raise MetadataRetrievalError( + error_msg="container metadata returned invalid JSON" + ) from None + if not isinstance(parsed, dict): + raise MetadataRetrievalError( + error_msg="container metadata returned an invalid document" + ) + return parsed + except RETRYABLE_HTTP_ERRORS: + raise MetadataRetrievalError( + error_msg="container metadata request failed" + ) from None + + +class _SanitizedInstanceMetadataFetcher(AioInstanceMetadataFetcher): + """Fetch IMDSv2 credentials without SDK logging response or exception data.""" + + async def _fetch_metadata_token(self) -> str | None: + self._assert_enabled() + url = self._construct_url(self._TOKEN_PATH) + headers = {"x-aws-ec2-metadata-token-ttl-seconds": self._TOKEN_TTL} + self._add_user_agent(headers) + request = AWSRequest(method="PUT", url=url, headers=headers) + async with self._session.acquire() as session: + for _ in range(self._num_attempts): + try: + response = await session.send(request.prepare()) + if response.status_code == 200: + return await response.text + if response.status_code in (403, 404, 405): + return None + if response.status_code == 400: + raise BadIMDSRequestError(request) + except ReadTimeoutError: + return None + except RETRYABLE_HTTP_ERRORS: + continue + except HTTPClientError as error: + nested_error = error.kwargs.get("error") + name_resolution_failed = ( + nested_error + and getattr(nested_error, "errno", None) == 8 + ) or str(getattr(nested_error, "os_error", None)) == ( + "Domain name not found" + ) + if name_resolution_failed: + raise InvalidIMDSEndpointError( + endpoint=url, + error=error, + ) from None + raise + return None + + async def _get_request( + self, + url_path: str, + retry_func: Any, + token: str | None = None, + ) -> object: + self._assert_enabled() + if not token: + self._assert_v1_enabled() + retry_func = retry_func or self._default_retry + url = self._construct_url(url_path) + headers: dict[str, str] = {} + if token is not None: + headers["x-aws-ec2-metadata-token"] = token + self._add_user_agent(headers) + async with self._session.acquire() as session: + for _ in range(self._num_attempts): + try: + request = AWSRequest(method="GET", url=url, headers=headers) + response = await session.send(request.prepare()) + should_retry = retry_func(response) + if inspect.isawaitable(should_retry): + should_retry = await should_retry + if not should_retry: + return response + except RETRYABLE_HTTP_ERRORS: + continue + raise self._RETRIES_EXCEEDED_ERROR_CLS() + + async def _log_imds_response( + self, + response: object, + reason_to_log: str, + log_body: bool = False, + ) -> None: + return None + + async def retrieve_iam_role_credentials(self) -> dict[str, Any]: + try: + token = await self._fetch_metadata_token() + role_name = await self._get_iam_role(token) + credentials = await self._get_credentials(role_name, token) + if not self._contains_all_credential_fields(credentials): + return {} + resolved = { + "role_name": role_name, + "access_key": credentials["AccessKeyId"], + "secret_key": credentials["SecretAccessKey"], + "token": credentials["Token"], + "expiry_time": credentials["Expiration"], + } + self._evaluate_expiration(resolved) + return resolved + except (self._RETRIES_EXCEEDED_ERROR_CLS, BadIMDSRequestError): + return {} + + def _evaluate_expiration(self, credentials: dict[str, Any]) -> None: + expiration = credentials.get("expiry_time") + if expiration is None: + return + try: + parsed_expiration = datetime.datetime.strptime( + expiration, + "%Y-%m-%dT%H:%M:%SZ", + ) + except (TypeError, ValueError): + return + refresh_interval = self._config.get( + "ec2_credential_refresh_window", + 60 * 10, + ) + refresh_offset = datetime.timedelta( + seconds=refresh_interval + random.randint(120, 600) + ) + current_time = get_current_datetime() + if current_time >= parsed_expiration - refresh_offset: + credentials["expiry_time"] = (current_time + refresh_offset).strftime( + "%Y-%m-%dT%H:%M:%SZ" + ) + + +class _SanitizedContainerProvider(AioContainerProvider): + """Build ECS credentials without SDK exception logging.""" + + def _create_fetcher(self, full_uri: str, *_args: object, **_kwargs: object) -> Any: + async def fetch_credentials() -> dict[str, Any]: + retrieval_failed = False + try: + headers = self._build_headers() + response = await self._fetcher.retrieve_full_uri( + full_uri, + headers=headers, + ) + except MetadataRetrievalError: + retrieval_failed = True + response = {} + if retrieval_failed: + raise CredentialRetrievalError( + provider=self.METHOD, + error_msg="container metadata request failed", + ) from None + return { + "access_key": response["AccessKeyId"], + "secret_key": response["SecretAccessKey"], + "token": response["Token"], + "expiry_time": response["Expiration"], + "account_id": response.get("AccountId"), + } + + return fetch_credentials + + +class _SanitizedInstanceMetadataProvider(AioInstanceMetadataProvider): + """Build IMDS credentials without SDK response-derived logging.""" + + async def load(self) -> object | None: + metadata = await self._role_fetcher.retrieve_iam_role_credentials() + if not metadata: + return None + return AioRefreshableCredentials.create_from_metadata( + metadata, + method=self.METHOD, + refresh_using=self._role_fetcher.retrieve_iam_role_credentials, + ) +_FORBIDDEN_AWS_NETWORK_ENVIRONMENT = frozenset( + { + "AWS_CA_BUNDLE", + "AWS_ENDPOINT_URL", + "AWS_ENDPOINT_URL_STS", + "AWS_STS_REGIONAL_ENDPOINTS", + "AWS_USE_DUALSTACK_ENDPOINT", + "AWS_USE_FIPS_ENDPOINT", + "CURL_CA_BUNDLE", + "REQUESTS_CA_BUNDLE", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + } +) +_WEB_IDENTITY_SOURCE_ENVIRONMENT = frozenset( + {"AWS_ROLE_ARN", "AWS_ROLE_SESSION_NAME", "AWS_WEB_IDENTITY_TOKEN_FILE"} +) +_CONTAINER_SOURCE_ENVIRONMENT = frozenset( + { + "AWS_CONTAINER_AUTHORIZATION_TOKEN", + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", + "AWS_CONTAINER_CREDENTIALS_FULL_URI", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", + } +) +_IAM_ROLE_SOURCE_ENVIRONMENT = frozenset( + { + "AWS_EC2_METADATA_DISABLED", + "AWS_EC2_METADATA_SERVICE_ENDPOINT", + "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE", + "AWS_EC2_METADATA_V1_DISABLED", + } +) +_ALLOWED_AWS_ENVIRONMENT_BY_METHOD = { + "assume-role-with-web-identity": _WEB_IDENTITY_SOURCE_ENVIRONMENT, + "container-role": frozenset({"AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"}), + "iam-role": frozenset({"AWS_EC2_METADATA_V1_DISABLED"}), +} +_AWS_SOURCE_ENVIRONMENT_BY_METHOD = { + "assume-role-with-web-identity": _WEB_IDENTITY_SOURCE_ENVIRONMENT, + "container-role": _CONTAINER_SOURCE_ENVIRONMENT, + "iam-role": _IAM_ROLE_SOURCE_ENVIRONMENT, +} + + +class S3CompatibleArtifactStoreBootstrap: + """Composition-only namespace lifecycle for one pinned S3 namespace.""" + + def __init__(self, adapter: S3CompatibleArtifactStore) -> None: + """Own one configured adapter before PostgreSQL namespace admission.""" + if type(adapter) is not S3CompatibleArtifactStore: + raise ValueError("S3-compatible artifact bootstrap adapter is invalid") + self._adapter = adapter + + @property + def identity(self) -> ExternalServiceAdapterIdentity: + """Return the canonical artifact-store/S3 adapter identity.""" + return self._adapter.identity + + @property + def namespace_identity(self) -> ArtifactStoreNamespaceIdentity: + """Return the configured provider namespace identity.""" + return self._adapter._namespace_identity + + def initialize_after_namespace_claim( + self, + claim: ArtifactStoreNamespaceClaim, + ) -> S3CompatibleArtifactStore: + """Return the byte store only after exact namespace admission.""" + return self._adapter._initialize_after_namespace_claim(claim) + + def close(self) -> None: + """Release this bootstrap and invalidate its adapter.""" + self._adapter.close() + + +class S3CompatibleArtifactStore: + """Immutable S3-protocol byte provider for MinIO and later AWS activation.""" + + def __init__( + self, + *, + provider_profile: str, + region: str, + endpoint_url: str | None, + bucket: str, + private_prefix: str, + addressing_style: str, + session: AioSession, + buffer_bytes: int, + connect_timeout_seconds: float, + read_timeout_seconds: float, + write_timeout_seconds: float, + pool_timeout_seconds: float, + operation_total_timeout_seconds: float, + max_pool_connections: int, + ) -> None: + """Pin one validated namespace and one isolated SDK session.""" + if provider_profile not in {"minio", "aws_s3"}: + raise ArtifactConfigurationError("S3 provider profile is invalid") + if not is_canonical_s3_region(region): + raise ArtifactConfigurationError("S3 region is invalid") + if not is_canonical_s3_bucket(bucket): + raise ArtifactConfigurationError("S3 bucket is invalid") + if not is_canonical_s3_prefix(private_prefix): + raise ArtifactConfigurationError("S3 private prefix is invalid") + if addressing_style not in {"path", "virtual"}: + raise ArtifactConfigurationError("S3 addressing style is invalid") + if provider_profile == "minio": + try: + endpoint_url = canonical_minio_endpoint(endpoint_url) + except ValueError as error: + raise ArtifactConfigurationError(str(error)) from None + elif endpoint_url is not None: + raise ArtifactConfigurationError("native AWS S3 endpoint is invalid") + if not isinstance(session, AioSession): + raise ArtifactConfigurationError("S3 credential session is invalid") + if type(buffer_bytes) is not int or not 1 <= buffer_bytes <= 1024 * 1024: + raise ArtifactConfigurationError("S3 artifact buffer is invalid") + if type(max_pool_connections) is not int or not 1 <= max_pool_connections <= 256: + raise ArtifactConfigurationError("S3 connection pool is invalid") + for value in ( + connect_timeout_seconds, + read_timeout_seconds, + write_timeout_seconds, + pool_timeout_seconds, + operation_total_timeout_seconds, + ): + if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0: + raise ArtifactConfigurationError("S3 operation timeout is invalid") + + self._provider_profile = provider_profile + self._region = region + self._endpoint_url = endpoint_url + self._bucket = bucket + self._private_prefix = private_prefix + self._addressing_style = addressing_style + self._session = session + self._buffer_bytes = buffer_bytes + self._write_timeout_seconds = float(write_timeout_seconds) + self._pool_timeout_seconds = float(pool_timeout_seconds) + self._operation_total_timeout_seconds = float(operation_total_timeout_seconds) + self._client_config = AioConfig( + connect_timeout=float(connect_timeout_seconds), + read_timeout=float(read_timeout_seconds), + max_pool_connections=max_pool_connections, + proxies={}, + retries={"max_attempts": 1, "mode": "standard"}, + request_checksum_calculation="when_required", + response_checksum_validation="when_required", + s3={ + "addressing_style": addressing_style, + "payload_signing_enabled": False, + }, + ) + self._initialized = False + + @property + def identity(self) -> ExternalServiceAdapterIdentity: + """Return the canonical artifact-store/S3 adapter identity.""" + return _IDENTITY + + @property + def _namespace_identity(self) -> ArtifactStoreNamespaceIdentity: + """Return the canonical non-secret identity of this S3 namespace.""" + profile = "minio-v1" if self._provider_profile == "minio" else "aws-s3-v1" + items = { + "addressing_style": self._addressing_style, + "bucket": self._bucket, + "private_prefix": self._private_prefix, + "region": self._region, + } + if self._provider_profile == "minio": + items["endpoint_identity"] = canonical_json_hash( + {"endpoint_url": self._endpoint_url} + ) + return ArtifactStoreNamespaceIdentity( + provider_profile=profile, + descriptor_items=tuple(sorted(items.items())), + ) + + def _initialize_after_namespace_claim( + self, + claim: ArtifactStoreNamespaceClaim, + ) -> S3CompatibleArtifactStore: + """Enable provider operations only for the exact PostgreSQL claim.""" + if ( + type(claim) is not ArtifactStoreNamespaceClaim + or claim.adapter_identity != self.identity + or claim.namespace_identity != self._namespace_identity + or claim.namespace_fingerprint != self._expected_namespace_fingerprint() + ): + self.close() + raise ArtifactConfigurationError("artifact namespace claim does not match provider") + if self._initialized: + self.close() + raise ArtifactConfigurationError("S3-compatible artifact storage is already initialized") + self._initialized = True + return self + + def _expected_namespace_fingerprint(self) -> str: + """Return the shared fingerprint for this configured S3 namespace.""" + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=self.identity, + namespace_identity=self._namespace_identity, + ) + return fingerprint + + def close(self) -> None: + """Invalidate this configured adapter; clients are operation-scoped.""" + self._initialized = False + + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + """Conditionally publish one sealed source or verify an exact replay.""" + self._require_initialized() + if type(source) is not CommittedArtifactSource: + raise ArtifactInputMismatchError("artifact source is not sealed") + commitment = source.commitment + if commitment.byte_count > HARD_MAXIMUM_ARTIFACT_BYTES: + raise ArtifactLimitExceededError("artifact source exceeds provider limit") + provider_object_ref = artifact_provider_object_ref(commitment) + body = _CommittedSourceBody(source, self._buffer_bytes) + try: + if commitment.byte_count == 0: + await body.validate_empty() + request_body: object = b"" if commitment.byte_count == 0 else body + async with asyncio.timeout(self._operation_total_timeout_seconds): + async with asyncio.timeout(self._write_timeout_seconds): + async with self._client() as client: + await client.put_object( + Bucket=self._bucket, + Key=self._object_key(provider_object_ref), + Body=request_body, + ContentLength=commitment.byte_count, + ContentType=commitment.media_type, + IfNoneMatch="*", + ) + if not body.complete: + raise ArtifactInputMismatchError( + "S3 provider did not consume the sealed artifact source" + ) + return ArtifactPutResult(provider_object_ref, replayed=False) + except ClientError as error: + status = _provider_http_status(error) + if status == 403: + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + if ( + status in {409, 412} + and _provider_error_code(error) in _PRECONDITION_ERROR_CODES + ): + await self._validate_replay_source(body) + if await self._matches_commitment(provider_object_ref, commitment): + return ArtifactPutResult(provider_object_ref, replayed=True) + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + except ArtifactStoreError: + raise + except (TimeoutError, BotoCoreError, OSError): + await self._validate_replay_source(body) + if await self._matches_commitment(provider_object_ref, commitment): + return ArtifactPutResult(provider_object_ref, replayed=True) + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + + async def observe_put_result( + self, + commitment: ArtifactCommitment, + ) -> ArtifactPutObservation: + """Read and validate the deterministic committed object when present.""" + self._require_initialized() + if type(commitment) is not ArtifactCommitment: + raise ArtifactOperationConflictError("artifact commitment is invalid") + provider_object_ref = artifact_provider_object_ref(commitment) + observed = await self.head(provider_object_ref) + if not observed.exists: + return ArtifactPutObservation(provider_object_ref, committed=False) + await self._verify_exact(provider_object_ref, commitment) + return ArtifactPutObservation(provider_object_ref, committed=True) + + def open( + self, + provider_object_ref: str, + byte_range: ArtifactByteRange | None = None, + ) -> AsyncIterator[bytes]: + """Stream a full object or one bounded byte range.""" + self._require_initialized() + parse_artifact_provider_object_ref(provider_object_ref) + if byte_range is not None and type(byte_range) is not ArtifactByteRange: + raise ArtifactOperationConflictError("artifact byte range is invalid") + selected_range = byte_range or ArtifactByteRange() + + async def iterate() -> AsyncIterator[bytes]: + """Hold the S3 response open while yielding exact bounded chunks.""" + head = await self.head(provider_object_ref) + if not head.exists or head.byte_count is None: + raise ArtifactObjectMissingError("artifact object is missing") + if selected_range.offset > head.byte_count: + raise ArtifactRangeInvalidError("artifact range starts past object end") + expected = head.byte_count - selected_range.offset + if selected_range.length is not None: + expected = min(expected, selected_range.length) + if expected == 0: + return + + end = selected_range.offset + expected - 1 + request: dict[str, object] = { + "Bucket": self._bucket, + "Key": self._object_key(provider_object_ref), + } + if selected_range.offset or expected != head.byte_count: + request["Range"] = f"bytes={selected_range.offset}-{end}" + try: + async with asyncio.timeout(self._operation_total_timeout_seconds): + async with self._client() as client: + response = await client.get_object(**request) + body = response["Body"] + observed = 0 + async with body: + while True: + chunk = await body.read( + min(self._buffer_bytes, expected - observed + 1) + ) + if not chunk: + break + if not isinstance(chunk, bytes): + raise ArtifactIntegrityError( + "S3 artifact read returned invalid bytes" + ) + observed += len(chunk) + if observed > expected: + raise ArtifactIntegrityError( + "S3 artifact read exceeded its bound" + ) + yield chunk + if observed != expected: + raise ArtifactIntegrityError("S3 artifact object was truncated") + except ClientError as error: + status = _provider_http_status(error) + if status == 404 and _provider_error_code(error) in _MISSING_ERROR_CODES: + raise ArtifactObjectMissingError("artifact object is missing") from None + raise ArtifactStoreUnavailableError("S3 artifact read failed") from None + except ArtifactStoreError: + raise + except (TimeoutError, BotoCoreError, OSError): + raise ArtifactStoreUnavailableError("S3 artifact read failed") from None + + return iterate() + + async def head(self, provider_object_ref: str) -> ArtifactObjectHead: + """Return exact size or authoritative absence without provider details.""" + self._require_initialized() + parse_artifact_provider_object_ref(provider_object_ref) + try: + async with asyncio.timeout(self._operation_total_timeout_seconds): + async with self._client() as client: + response = await client.head_object( + Bucket=self._bucket, + Key=self._object_key(provider_object_ref), + ) + byte_count = response.get("ContentLength") + if type(byte_count) is not int or byte_count < 0: + raise ArtifactIntegrityError("S3 artifact head lacks exact size") + return ArtifactObjectHead( + provider_object_ref, + exists=True, + byte_count=byte_count, + ) + except ClientError as error: + status = _provider_http_status(error) + if status == 404 and _provider_error_code(error) in _MISSING_ERROR_CODES: + return ArtifactObjectHead(provider_object_ref, exists=False) + raise ArtifactStoreUnavailableError("S3 artifact head failed") from None + except ArtifactStoreError: + raise + except (TimeoutError, BotoCoreError, OSError): + raise ArtifactStoreUnavailableError("S3 artifact head failed") from None + + async def _verify_exact( + self, + provider_object_ref: str, + commitment: ArtifactCommitment, + ) -> None: + """Independently hash and count one complete provider object.""" + digest = hashlib.sha256() + byte_count = 0 + async for chunk in self.open(provider_object_ref): + digest.update(chunk) + byte_count += len(chunk) + if ( + byte_count != commitment.byte_count + or f"sha256:{digest.hexdigest()}" != commitment.sha256 + ): + raise ArtifactIntegrityError("S3 artifact object violates commitment") + + async def _matches_commitment( + self, + provider_object_ref: str, + commitment: ArtifactCommitment, + ) -> bool: + """Resolve an uncertain write only from an independently verified object.""" + try: + observed = await self.head(provider_object_ref) + if not observed.exists: + return False + await self._verify_exact(provider_object_ref, commitment) + except ArtifactObjectMissingError: + return False + return True + + async def _validate_replay_source(self, body: _CommittedSourceBody) -> None: + """Finish validating the sealed source before accepting any replay.""" + try: + async with asyncio.timeout(self._operation_total_timeout_seconds): + await body.validate_complete() + except TimeoutError: + raise ArtifactStoreUnavailableError("S3 artifact operation failed") from None + + @asynccontextmanager + async def _client(self) -> AsyncIterator[Any]: + """Create one bounded S3 client from the already-isolated session.""" + async with asyncio.timeout(self._pool_timeout_seconds): + context = self._session.create_client( + "s3", + region_name=self._region, + endpoint_url=self._endpoint_url, + config=self._client_config, + ) + client = await context.__aenter__() + try: + yield client + finally: + await context.__aexit__(None, None, None) + + def _require_initialized(self) -> None: + """Reject all operations before exact namespace admission.""" + if not self._initialized: + raise ArtifactConfigurationError("artifact store namespace is not initialized") + + def _object_key(self, provider_object_ref: str) -> str: + """Apply only the configured private prefix to a digest-derived reference.""" + parse_artifact_provider_object_ref(provider_object_ref) + return f"{self._private_prefix}/{provider_object_ref}" + + +class _CommittedSourceBody: + """Single-use async request body that revalidates the sealed commitment.""" + + def __init__(self, source: CommittedArtifactSource, buffer_bytes: int) -> None: + """Bind one sealed source to one bounded second-pass request body.""" + self._source = source + self._commitment = source.commitment + self._buffer_bytes = buffer_bytes + self._consumed = False + self._complete = False + self._iterator: AsyncIterator[bytes] | None = None + self._source_cursor: memoryview | None = None + self._pending = bytearray() + self._digest = hashlib.sha256() + self._byte_count = 0 + + def __aiter__(self) -> AsyncIterator[bytes]: + """Yield bounded bytes while checking the exact digest and count.""" + if self._consumed: + raise ArtifactInputMismatchError("artifact source stream was already consumed") + self._consumed = True + + async def iterate() -> AsyncIterator[bytes]: + """Yield the source until its commitment and EOF are proven.""" + while chunk := await self.read(self._buffer_bytes): + yield chunk + + return iterate() + + @property + def complete(self) -> bool: + """Return whether the sealed second-pass stream was fully validated.""" + return self._complete + + async def read(self, amount: int | None = -1) -> bytes: + """Return one bounded async file-like chunk for the pinned SDK.""" + if not self._consumed: + self._consumed = True + if self._complete: + return b"" + requested = self._buffer_bytes if amount is None or amount < 0 else amount + if requested == 0: + return b"" + requested = min(requested, self._buffer_bytes) + if self._iterator is None: + self._iterator = self._source.stream() + while len(self._pending) < requested: + if self._source_cursor is None: + try: + source_chunk = await anext(self._iterator) + except StopAsyncIteration: + self._finish() + break + if not isinstance(source_chunk, bytes): + raise ArtifactInputMismatchError("artifact source must yield bytes") + self._source_cursor = memoryview(source_chunk) + if not self._source_cursor: + self._source_cursor = None + continue + needed = requested - len(self._pending) + copied = min(needed, len(self._source_cursor)) + self._pending.extend(self._source_cursor[:copied]) + self._source_cursor = self._source_cursor[copied:] or None + chunk = bytes(self._pending[:requested]) + del self._pending[:requested] + self._byte_count += len(chunk) + if self._byte_count > self._commitment.byte_count: + raise ArtifactInputMismatchError("artifact source exceeds committed byte count") + self._digest.update(chunk) + if self._byte_count == self._commitment.byte_count and not self._pending: + await self._require_source_exhausted() + if not chunk and not self._complete: + self._finish() + return chunk + + async def validate_empty(self) -> None: + """Validate a zero-byte source that the SDK need not consume.""" + if self._commitment.byte_count != 0: + raise ArtifactInputMismatchError("artifact source is not empty") + if await self.read(1) != b"" or not self._complete: + raise ArtifactInputMismatchError("artifact source violates commitment") + + async def validate_complete(self) -> None: + """Consume and validate every remaining sealed source byte.""" + while await self.read(self._buffer_bytes): + pass + if not self._complete: + raise ArtifactInputMismatchError("artifact source violates commitment") + + async def _require_source_exhausted(self) -> None: + """Prove EOF before returning the final committed byte to the SDK.""" + if self._source_cursor: + raise ArtifactInputMismatchError( + "artifact source exceeds committed byte count" + ) + if self._iterator is None: + self._iterator = self._source.stream() + while True: + try: + extra = await anext(self._iterator) + except StopAsyncIteration: + self._finish() + return + if not isinstance(extra, bytes): + raise ArtifactInputMismatchError("artifact source must yield bytes") + if extra: + raise ArtifactInputMismatchError( + "artifact source exceeds committed byte count" + ) + + def _finish(self) -> None: + """Validate commitment only after every pending byte is consumed.""" + if self._pending: + return + if ( + self._byte_count != self._commitment.byte_count + or f"sha256:{self._digest.hexdigest()}" != self._commitment.sha256 + ): + raise ArtifactInputMismatchError("artifact source violates commitment") + self._complete = True + +def create_minio_artifact_store_bootstrap( + settings: Settings, +) -> S3CompatibleArtifactStoreBootstrap: + """Construct the only S3 runtime profile enabled by this chunk.""" + bootstrap, error_message = _try_create_minio_artifact_store_bootstrap(settings) + if error_message is not None: + del settings + raise ArtifactConfigurationError(error_message) from None + if bootstrap is None: + del settings + raise ArtifactConfigurationError("MinIO artifact configuration is invalid") from None + return bootstrap + + +def _try_create_minio_artifact_store_bootstrap( + settings: Settings, +) -> tuple[S3CompatibleArtifactStoreBootstrap | None, str | None]: + """Build MinIO state without propagating a secret-bearing failure frame.""" + if settings.artifact_s3_provider_profile != "minio": + return None, "MinIO artifact profile is not configured" + access_key = settings.artifact_s3_access_key_id + secret_key = settings.artifact_s3_secret_access_key + if access_key is None or secret_key is None: + return None, "MinIO artifact credentials are unavailable" + try: + session = AioSession() + session.set_credentials( + access_key.get_secret_value(), + secret_key.get_secret_value(), + settings.artifact_s3_session_token.get_secret_value() + if settings.artifact_s3_session_token is not None + else None, + ) + bootstrap = S3CompatibleArtifactStoreBootstrap( + S3CompatibleArtifactStore( + provider_profile="minio", + region=_required(settings.artifact_s3_region), + endpoint_url=_required(settings.artifact_s3_endpoint_url), + bucket=_required(settings.artifact_s3_bucket), + private_prefix=settings.artifact_s3_private_prefix, + addressing_style=settings.artifact_s3_addressing_style, + session=session, + buffer_bytes=settings.artifact_stream_buffer_bytes, + connect_timeout_seconds=settings.artifact_s3_connect_timeout_seconds, + read_timeout_seconds=settings.artifact_s3_read_timeout_seconds, + write_timeout_seconds=settings.artifact_s3_write_timeout_seconds, + pool_timeout_seconds=settings.artifact_s3_pool_timeout_seconds, + operation_total_timeout_seconds=( + settings.artifact_s3_operation_total_timeout_seconds + ), + max_pool_connections=settings.artifact_s3_max_pool_connections, + ) + ) + except ArtifactConfigurationError as error: + return None, str(error) + except Exception: + return None, "MinIO artifact configuration is invalid" + return bootstrap, None + + +def validate_aws_workload_identity_environment( + settings: Settings, + *, + environ: Mapping[str, str] | None = None, +) -> None: + """Reject every ambient or unselected AWS credential source without loading it.""" + if settings.artifact_s3_provider_profile != "aws_s3": + raise ArtifactConfigurationError("native AWS S3 profile is not configured") + selected = settings.artifact_s3_aws_workload_identity_method + if selected is None: + raise ArtifactConfigurationError("AWS workload identity method is unavailable") + environment = os.environ if environ is None else environ + if _FORBIDDEN_AWS_CREDENTIAL_ENVIRONMENT.intersection(environment): + raise ArtifactConfigurationError("ambient AWS credential source is forbidden") + if _FORBIDDEN_AWS_NETWORK_ENVIRONMENT.intersection(environment): + raise ArtifactConfigurationError("ambient AWS network configuration is forbidden") + if any(name.startswith("AWS_ENDPOINT_URL_") for name in environment): + raise ArtifactConfigurationError("ambient AWS network configuration is forbidden") + _reject_default_credential_files(environment) + + unselected = set().union( + *( + values + for method, values in _AWS_SOURCE_ENVIRONMENT_BY_METHOD.items() + if method != selected + ) + ) + if unselected.intersection(environment): + raise ArtifactConfigurationError("unselected AWS workload source is configured") + if selected == "container-role": + if "AWS_CONTAINER_CREDENTIALS_FULL_URI" in environment: + raise ArtifactConfigurationError( + "AWS container full credential URI is forbidden" + ) + if { + "AWS_CONTAINER_AUTHORIZATION_TOKEN", + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", + }.intersection(environment): + raise ArtifactConfigurationError("AWS container identity token is forbidden") + elif selected == "iam-role": + if "AWS_EC2_METADATA_SERVICE_ENDPOINT" in environment: + raise ArtifactConfigurationError("custom AWS metadata endpoint is forbidden") + if "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE" in environment: + raise ArtifactConfigurationError( + "custom AWS metadata endpoint mode is forbidden" + ) + if environment.get("AWS_EC2_METADATA_DISABLED", "false").lower() == "true": + raise ArtifactConfigurationError("AWS instance identity metadata is disabled") + + allowed_environment = _ALLOWED_AWS_ENVIRONMENT_BY_METHOD[selected] + unsupported_sdk_environment = { + name + for name in environment + if (name.startswith("AWS_") or name.startswith("BOTOCORE_")) + and name not in allowed_environment + } + if unsupported_sdk_environment: + raise ArtifactConfigurationError( + "unsupported AWS SDK environment configuration is forbidden" + ) + + if selected == "assume-role-with-web-identity": + if not {"AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"}.issubset(environment): + raise ArtifactConfigurationError("AWS web identity configuration is incomplete") + token_path = Path(environment["AWS_WEB_IDENTITY_TOKEN_FILE"]) + if not token_path.is_absolute() or not token_path.is_file(): + raise ArtifactConfigurationError("AWS web identity token path is invalid") + elif selected == "container-role": + relative_uri = environment.get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI") + if ( + not isinstance(relative_uri, str) + or _CONTAINER_RELATIVE_URI.fullmatch(relative_uri) is None + ): + raise ArtifactConfigurationError("AWS container identity location is invalid") + else: + if environment.get("AWS_EC2_METADATA_V1_DISABLED", "false").lower() != "true": + raise ArtifactConfigurationError("AWS instance identity requires IMDSv2") + + +def create_isolated_aws_workload_identity_session( + settings: Settings, + *, + environ: Mapping[str, str] | None = None, +) -> AioSession: + """Build an inactive SDK session whose resolver contains one selected provider.""" + validate_aws_workload_identity_environment(settings, environ=environ) + environment = os.environ if environ is None else environ + selected = settings.artifact_s3_aws_workload_identity_method + session = AioSession() + if selected == "assume-role-with-web-identity": + profile = { + "role_arn": environment["AWS_ROLE_ARN"], + "web_identity_token_file": environment["AWS_WEB_IDENTITY_TOKEN_FILE"], + } + role_session_name = environment.get("AWS_ROLE_SESSION_NAME") + if role_session_name is not None: + profile["role_session_name"] = role_session_name + + def create_isolated_sts_client(service_name: str, **_kwargs: object) -> object: + """Create only the pinned no-proxy STS client for web identity.""" + if service_name != "sts": + raise ArtifactConfigurationError("AWS web identity service is invalid") + return session.create_client( + "sts", + region_name=_required(settings.artifact_s3_region), + verify=True, + config=AioConfig( + signature_version=UNSIGNED, + connect_timeout=settings.artifact_s3_connect_timeout_seconds, + read_timeout=settings.artifact_s3_read_timeout_seconds, + retries={"max_attempts": 1, "mode": "standard"}, + proxies={}, + ), + ) + + provider = AioAssumeRoleWithWebIdentityProvider( + load_config=lambda: {"profiles": {"workstream-isolated": profile}}, + client_creator=create_isolated_sts_client, + profile_name="workstream-isolated", + cache={}, + disable_env_vars=True, + ) + elif selected == "container-role": + provider = _SanitizedContainerProvider( + environ={ + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": environment[ + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" + ] + } + ) + provider._fetcher = _SanitizedContainerMetadataFetcher( + session=_RefCountedSession( + timeout=settings.artifact_s3_connect_timeout_seconds, + proxies={}, + ) + ) + elif selected == "iam-role": + provider = _SanitizedInstanceMetadataProvider( + iam_role_fetcher=_SanitizedInstanceMetadataFetcher( + timeout=settings.artifact_s3_connect_timeout_seconds, + num_attempts=1, + env=dict(environment), + user_agent=session.user_agent(), + config={ + "ec2_metadata_v1_disabled": True, + }, + session=_RefCountedSession( + timeout=settings.artifact_s3_connect_timeout_seconds, + proxies={}, + ), + ) + ) + else: + raise ArtifactConfigurationError("AWS workload identity method is unsupported") + session.register_component("credential_provider", AioCredentialResolver([provider])) + return session + + +async def resolve_isolated_aws_workload_credentials( + session: AioSession, + *, + expected_method: str, +) -> object: + """Resolve credentials and require the exact selected SDK method.""" + resolution_failed = False + try: + credentials = await session.get_credentials() + except Exception: + resolution_failed = True + credentials = None + if resolution_failed: + raise ArtifactConfigurationError( + "AWS workload identity credentials could not be resolved" + ) from None + if credentials is None or getattr(credentials, "method", None) != expected_method: + raise ArtifactConfigurationError("AWS workload identity method did not match") + materialization_failed = False + try: + await credentials.get_frozen_credentials() + except Exception: + materialization_failed = True + if materialization_failed: + del credentials + raise ArtifactConfigurationError( + "AWS workload identity credentials could not be resolved" + ) from None + return credentials + + +def _reject_default_credential_files(environ: Mapping[str, str]) -> None: + """Fail on default shared/config/Boto files without reading their contents.""" + home = Path(environ.get("HOME", str(Path.home()))) + for path in ( + home / ".aws" / "credentials", + home / ".aws" / "config", + home / ".boto", + Path("/etc/boto.cfg"), + ): + if path.exists(): + raise ArtifactConfigurationError("ambient AWS credential file is forbidden") + + +def _provider_error_code(error: ClientError) -> str: + """Return one bounded provider status/code without retaining response details.""" + response = error.response if isinstance(error.response, dict) else {} + provider_error = response.get("Error") if isinstance(response, dict) else None + code = provider_error.get("Code") if isinstance(provider_error, dict) else None + if isinstance(code, str) and code: + return code + metadata = response.get("ResponseMetadata") if isinstance(response, dict) else None + status = metadata.get("HTTPStatusCode") if isinstance(metadata, dict) else None + return str(status) if type(status) is int else "unknown" + + +def _provider_http_status(error: ClientError) -> int | None: + """Return the provider HTTP status independently from its error code.""" + response = error.response if isinstance(error.response, dict) else {} + metadata = response.get("ResponseMetadata") if isinstance(response, dict) else None + status = metadata.get("HTTPStatusCode") if isinstance(metadata, dict) else None + return status if type(status) is int else None + + +def _required(value: str | None) -> str: + """Return one settings value already guaranteed by validation.""" + if not isinstance(value, str) or not value: + raise ArtifactConfigurationError("S3-compatible artifact setting is unavailable") + return value diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 8685880b..3fa52fc4 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -4,9 +4,10 @@ import base64 import binascii +from contextvars import ContextVar import json import os -from collections.abc import Mapping +from collections.abc import Callable, Mapping from functools import lru_cache from pathlib import Path from typing import Literal, Self @@ -20,6 +21,35 @@ SettingsConfigDict, ) +from app.core.s3_validation import ( + canonical_minio_endpoint, + is_canonical_s3_bucket, + is_canonical_s3_prefix, + is_canonical_s3_region, +) + + +_ARTIFACT_S3_SECRET_FIELDS = frozenset( + { + "artifact_s3_access_key_id", + "artifact_s3_secret_access_key", + "artifact_s3_session_token", + } +) +_ARTIFACT_S3_SENSITIVE_INPUT_FIELDS = _ARTIFACT_S3_SECRET_FIELDS | { + "artifact_s3_endpoint_url" +} +_EMPTY_ARTIFACT_S3_SECRETS: tuple[SecretStr | None, SecretStr | None, SecretStr | None] = ( + None, + None, + None, +) +_MISSING_SECRET = object() +_ALTERNATE_VALIDATION_RESTORES_SECRETS: ContextVar[bool] = ContextVar( + "alternate_validation_restores_secrets", + default=False, +) + class Settings(BaseSettings): """Runtime configuration for the Workstream API.""" @@ -93,6 +123,31 @@ class Settings(BaseSettings): api_admin_mutation_rate_window_seconds: int = Field(default=60, ge=1, le=3_600) artifact_store_backend: Literal["disabled", "local", "s3_compatible"] = "disabled" artifact_local_root: Path | None = None + artifact_s3_provider_profile: Literal["aws_s3", "minio"] | None = None + artifact_s3_region: str | None = None + artifact_s3_endpoint_url: str | None = None + artifact_s3_bucket: str | None = None + artifact_s3_private_prefix: str = "workstream/artifacts" + artifact_s3_addressing_style: Literal["path", "virtual"] = "virtual" + artifact_s3_credential_mode: Literal[ + "aws_workload_identity", + "local_static", + ] | None = None + artifact_s3_aws_workload_identity_method: Literal[ + "assume-role-with-web-identity", + "container-role", + "iam-role", + ] | None = None + artifact_s3_connect_timeout_seconds: float = Field(default=5.0, gt=0.0, le=60.0) + artifact_s3_read_timeout_seconds: float = Field(default=60.0, gt=0.0, le=1800.0) + artifact_s3_write_timeout_seconds: float = Field(default=1800.0, gt=0.0, le=3600.0) + artifact_s3_pool_timeout_seconds: float = Field(default=5.0, gt=0.0, le=60.0) + artifact_s3_operation_total_timeout_seconds: float = Field( + default=1800.0, + gt=0.0, + le=3600.0, + ) + artifact_s3_max_pool_connections: int = Field(default=16, ge=1, le=256) artifact_maximum_bytes: int = Field(default=512 * 1024 * 1024, gt=0) artifact_stream_buffer_bytes: int = Field(default=1024 * 1024, gt=0, le=1024 * 1024) artifact_operation_lock_timeout_seconds: float = Field( @@ -140,23 +195,44 @@ class Settings(BaseSettings): ) _api_rate_limit_key_secret: SecretStr | None = PrivateAttr(default=None) + _artifact_s3_access_key_id: SecretStr | None = PrivateAttr(default=None) + _artifact_s3_secret_access_key: SecretStr | None = PrivateAttr(default=None) + _artifact_s3_session_token: SecretStr | None = PrivateAttr(default=None) def __init__(self, **values: object) -> None: """Remove rate-control secret material before structured validation.""" - secret = _extract_api_rate_limit_key_secret(values) - super().__init__(**values) - self._api_rate_limit_key_secret = secret + secret: SecretStr | None = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + try: + _normalize_artifact_s3_endpoint_input(values) + secret = _extract_api_rate_limit_key_secret(values) + s3_secrets = _extract_artifact_s3_static_secrets(values) + super().__init__(**values) + self._api_rate_limit_key_secret = secret + self._set_artifact_s3_static_secrets(s3_secrets) + if not _ALTERNATE_VALIDATION_RESTORES_SECRETS.get(): + self._validate_artifact_s3_secret_contract() + except Exception: + _clear_settings_private_secrets(self) + values.clear() + secret = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + raise @classmethod def model_validate(cls, obj: object, **kwargs: object) -> Self: """Sanitize secret-bearing mappings before Pydantic retains input.""" - if isinstance(obj, Mapping) and "api_rate_limit_key_secret" in obj: + if isinstance(obj, Mapping) and ( + "api_rate_limit_key_secret" in obj + or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(obj) + ): sanitized = dict(obj) - secret = _extract_api_rate_limit_key_secret(sanitized) - sanitized["api_rate_limit_key_secret"] = None - settings = super().model_validate(sanitized, **kwargs) - settings._api_rate_limit_key_secret = secret - return settings + obj = None + return cls._validate_secret_mapping( + sanitized, + super().model_validate, + kwargs, + ) return super().model_validate(obj, **kwargs) @classmethod @@ -173,23 +249,88 @@ def model_validate_json( malformed = True parsed = None if malformed: + json_data = b"" + raise ValueError("invalid settings JSON") + if not isinstance(parsed, Mapping): + parsed = None + json_data = b"" raise ValueError("invalid settings JSON") - if isinstance(parsed, Mapping) and "api_rate_limit_key_secret" in parsed: - return cls.model_validate(parsed, **kwargs) - return super().model_validate_json(json_data, **kwargs) + if isinstance(parsed, Mapping) and ( + "api_rate_limit_key_secret" in parsed + or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(parsed) + ): + try: + return cls.model_validate(parsed, **kwargs) + except Exception: + parsed = None + json_data = b"" + raise + try: + return super().model_validate_json(json_data, **kwargs) + except Exception: + parsed = None + json_data = b"" + raise @classmethod def model_validate_strings(cls, obj: object, **kwargs: object) -> Self: """Sanitize string-mapping secret input before structured validation.""" - if isinstance(obj, Mapping) and "api_rate_limit_key_secret" in obj: + if isinstance(obj, Mapping) and ( + "api_rate_limit_key_secret" in obj + or _ARTIFACT_S3_SENSITIVE_INPUT_FIELDS.intersection(obj) + ): sanitized = dict(obj) - secret = _extract_api_rate_limit_key_secret(sanitized) - sanitized["api_rate_limit_key_secret"] = None - settings = super().model_validate_strings(sanitized, **kwargs) - settings._api_rate_limit_key_secret = secret - return settings + obj = None + return cls._validate_secret_mapping( + sanitized, + super().model_validate_strings, + kwargs, + ) return super().model_validate_strings(obj, **kwargs) + @classmethod + def _validate_secret_mapping( + cls, + sanitized: dict[str, object], + validator: Callable[..., Self], + kwargs: Mapping[str, object], + ) -> Self: + """Validate a mapping without retaining supplied secret material.""" + supplied_s3_fields = _ARTIFACT_S3_SECRET_FIELDS.intersection(sanitized) + secret: SecretStr | None = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings: Settings | None = None + try: + _normalize_artifact_s3_endpoint_input(sanitized) + secret = ( + _extract_api_rate_limit_key_secret(sanitized) + if "api_rate_limit_key_secret" in sanitized + else None + ) + s3_secrets = _extract_artifact_s3_static_secrets(sanitized) + if secret is not None: + sanitized["api_rate_limit_key_secret"] = None + for field_name in supplied_s3_fields: + sanitized[field_name] = None + restore_token = _ALTERNATE_VALIDATION_RESTORES_SECRETS.set(True) + try: + settings = validator(sanitized, **kwargs) + finally: + _ALTERNATE_VALIDATION_RESTORES_SECRETS.reset(restore_token) + if secret is not None: + settings._api_rate_limit_key_secret = secret + settings._set_artifact_s3_static_secrets(s3_secrets) + settings._validate_artifact_s3_secret_contract() + return settings + except Exception: + if settings is not None: + _clear_settings_private_secrets(settings) + sanitized.clear() + secret = None + s3_secrets = _EMPTY_ARTIFACT_S3_SECRETS + settings = None + raise + @classmethod def settings_customise_sources( cls, @@ -202,6 +343,8 @@ def settings_customise_sources( """Keep the manually resolved rate key out of dotenv validation input.""" if isinstance(dotenv_settings, DotEnvSettingsSource): dotenv_settings.env_vars.pop("workstream_api_rate_limit_key_secret", None) + for field_name in _ARTIFACT_S3_SECRET_FIELDS: + dotenv_settings.env_vars.pop(f"workstream_{field_name}", None) return init_settings, env_settings, dotenv_settings, file_secret_settings @property @@ -209,6 +352,56 @@ def api_rate_limit_key_secret(self) -> SecretStr | None: """Return the validated, redacted rate-control key setting.""" return self._api_rate_limit_key_secret + @property + def artifact_s3_access_key_id(self) -> SecretStr | None: + """Return the local/CI MinIO access-key identifier.""" + return self._artifact_s3_access_key_id + + @property + def artifact_s3_secret_access_key(self) -> SecretStr | None: + """Return the local/CI MinIO secret access key.""" + return self._artifact_s3_secret_access_key + + @property + def artifact_s3_session_token(self) -> SecretStr | None: + """Return the optional local/CI MinIO session token.""" + return self._artifact_s3_session_token + + def _set_artifact_s3_static_secrets( + self, + secrets: tuple[SecretStr | None, SecretStr | None, SecretStr | None], + ) -> None: + """Retain static MinIO credentials only as redacted private values.""" + ( + self._artifact_s3_access_key_id, + self._artifact_s3_secret_access_key, + self._artifact_s3_session_token, + ) = secrets + + def _validate_artifact_s3_secret_contract(self) -> None: + """Reject static credentials outside a complete local MinIO profile.""" + secrets = ( + self.artifact_s3_access_key_id, + self.artifact_s3_secret_access_key, + self.artifact_s3_session_token, + ) + if self.artifact_store_backend != "s3_compatible" and any( + value is not None for value in secrets + ): + secrets = _EMPTY_ARTIFACT_S3_SECRETS + raise ValueError("static artifact credentials require MinIO storage") + if self.artifact_s3_provider_profile == "minio" and ( + self.artifact_s3_access_key_id is None + or self.artifact_s3_secret_access_key is None + ): + secrets = _EMPTY_ARTIFACT_S3_SECRETS + raise ValueError("MinIO artifact storage requires complete static credentials") + if self.artifact_s3_provider_profile == "aws_s3" and any( + value is not None for value in secrets + ): + secrets = _EMPTY_ARTIFACT_S3_SECRETS + raise ValueError("native AWS S3 rejects static credentials") + @model_validator(mode="after") def validate_artifact_storage(self) -> Settings: """Reject unsafe or incomplete artifact-storage combinations. @@ -247,8 +440,48 @@ def validate_artifact_storage(self) -> Settings: raise ValueError("local artifact storage is restricted to development and test") if self.artifact_local_root is None: raise ValueError("local artifact storage requires an artifact root") + return self + self._validate_s3_compatible_storage() return self + def _validate_s3_compatible_storage(self) -> None: + """Require one closed MinIO or native AWS configuration profile.""" + if self.artifact_s3_provider_profile is None: + raise ValueError("S3-compatible artifact storage requires a provider profile") + if not is_canonical_s3_region(self.artifact_s3_region): + raise ValueError("S3-compatible artifact storage requires a canonical region") + if not is_canonical_s3_bucket(self.artifact_s3_bucket): + raise ValueError("S3-compatible artifact storage requires a canonical bucket") + if not is_canonical_s3_prefix(self.artifact_s3_private_prefix): + raise ValueError("S3-compatible artifact storage private prefix is invalid") + if self.artifact_maximum_bytes > 512 * 1024 * 1024: + raise ValueError("S3-compatible artifact maximum exceeds 512 MiB") + if self.artifact_s3_provider_profile == "minio": + self._validate_minio_storage() + return + self._validate_native_aws_storage() + + def _validate_minio_storage(self) -> None: + """Keep static MinIO credentials and endpoints local to development/CI.""" + if self.environment not in {"local", "dev", "development", "test"}: + raise ValueError("MinIO artifact storage is restricted to development and test") + if self.artifact_s3_credential_mode != "local_static": + raise ValueError("MinIO artifact storage requires local static credentials") + if self.artifact_s3_aws_workload_identity_method is not None: + raise ValueError("MinIO artifact storage cannot select AWS workload identity") + self.artifact_s3_endpoint_url = canonical_minio_endpoint( + self.artifact_s3_endpoint_url + ) + + def _validate_native_aws_storage(self) -> None: + """Validate native AWS configuration without activating its runtime.""" + if self.artifact_s3_endpoint_url is not None: + raise ValueError("native AWS S3 does not accept a configured endpoint") + if self.artifact_s3_credential_mode != "aws_workload_identity": + raise ValueError("native AWS S3 requires workload identity") + if self.artifact_s3_aws_workload_identity_method is None: + raise ValueError("native AWS S3 requires one workload identity method") + @lru_cache def get_settings() -> Settings: @@ -260,36 +493,167 @@ def get_settings() -> Settings: return Settings() -def _extract_api_rate_limit_key_secret(values: dict[str, object]) -> SecretStr | None: - """Resolve and validate the key without entering Pydantic's input graph.""" - missing = object() - raw_value = values.pop("api_rate_limit_key_secret", missing) - if raw_value is missing: - raw_value = os.environ.get("WORKSTREAM_API_RATE_LIMIT_KEY_SECRET", missing) - if raw_value is missing: +def _clear_settings_private_secrets(settings: object) -> None: + """Remove sensitive values from a partially or fully initialized Settings object.""" + settings_values = getattr(settings, "__dict__", None) + if isinstance(settings_values, dict): + settings_values["artifact_s3_endpoint_url"] = None + private_values = getattr(settings, "__pydantic_private__", None) + if not isinstance(private_values, dict): + return + private_values["_api_rate_limit_key_secret"] = None + private_values["_artifact_s3_access_key_id"] = None + private_values["_artifact_s3_secret_access_key"] = None + private_values["_artifact_s3_session_token"] = None + + +def _normalize_artifact_s3_endpoint_input(values: dict[str, object]) -> None: + """Replace a raw endpoint source with its noncredentialed canonical origin.""" + raw_value = values.pop("artifact_s3_endpoint_url", _MISSING_SECRET) + if raw_value is _MISSING_SECRET: + raw_value = os.environ.get( + "WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL", + _MISSING_SECRET, + ) + if raw_value is _MISSING_SECRET: env_file = values.get("_env_file", ".env") if env_file is not None: env_encoding = values.get("_env_file_encoding", "utf-8") - env_files = (env_file,) if isinstance(env_file, (str, os.PathLike)) else tuple(env_file) + env_files = ( + (env_file,) + if isinstance(env_file, (str, os.PathLike)) + else tuple(env_file) + ) for path in env_files: - dotenv = dotenv_values(path, encoding=env_encoding) - raw_value = dotenv.get("WORKSTREAM_API_RATE_LIMIT_KEY_SECRET", raw_value) - if raw_value is missing or raw_value is None: + raw_value = dotenv_values(path, encoding=env_encoding).get( + "WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL", + raw_value, + ) + if raw_value is _MISSING_SECRET: + return + if raw_value is None: + values["artifact_s3_endpoint_url"] = None + return + try: + endpoint = canonical_minio_endpoint(raw_value) + except ValueError: + raw_value = None + raise ValueError("MinIO artifact storage endpoint is invalid") from None + values["artifact_s3_endpoint_url"] = endpoint + raw_value = None + + +def _extract_api_rate_limit_key_secret(values: dict[str, object]) -> SecretStr | None: + """Resolve and validate the key without entering Pydantic's input graph.""" + raw_value = _pop_optional_secret_source( + values, + field_name="api_rate_limit_key_secret", + environment_name="WORKSTREAM_API_RATE_LIMIT_KEY_SECRET", + ) + if raw_value is _MISSING_SECRET or raw_value is None: return None if isinstance(raw_value, SecretStr): secret = raw_value elif isinstance(raw_value, str): secret = SecretStr(raw_value) else: + del raw_value raise ValueError("invalid API rate limit key secret") - decode_api_rate_limit_key_secret(secret) + try: + decode_api_rate_limit_key_secret(secret) + except ValueError: + del raw_value, secret + raise return secret +def _extract_artifact_s3_static_secrets( + values: dict[str, object], +) -> tuple[SecretStr | None, SecretStr | None, SecretStr | None]: + """Remove optional MinIO credentials from structured settings input.""" + field_names = ( + "artifact_s3_access_key_id", + "artifact_s3_secret_access_key", + "artifact_s3_session_token", + ) + secrets: list[SecretStr | None] = [] + try: + for field_name in field_names: + secrets.append( + _extract_optional_secret( + values, + field_name=field_name, + environment_name=f"WORKSTREAM_{field_name.upper()}", + ) + ) + except Exception: + secrets.clear() + for field_name in field_names: + values.pop(field_name, None) + raise + return secrets[0], secrets[1], secrets[2] + + +def _extract_optional_secret( + values: dict[str, object], + *, + field_name: str, + environment_name: str, +) -> SecretStr | None: + """Resolve one bounded secret without retaining it in Pydantic input.""" + raw_value = _pop_optional_secret_source( + values, + field_name=field_name, + environment_name=environment_name, + ) + if raw_value is _MISSING_SECRET or raw_value is None: + return None + if isinstance(raw_value, SecretStr): + secret = raw_value + elif isinstance(raw_value, str): + secret = SecretStr(raw_value) + else: + del raw_value + raise ValueError("invalid artifact storage secret") + revealed = secret.get_secret_value() + if ( + not revealed + or len(revealed) > 4096 + or any(ord(character) < 32 or ord(character) == 127 for character in revealed) + ): + del raw_value, secret, revealed + raise ValueError("invalid artifact storage secret") + return secret + + +def _pop_optional_secret_source( + values: dict[str, object], + *, + field_name: str, + environment_name: str, +) -> object: + """Remove one optional secret from direct, environment, or dotenv input.""" + raw_value = values.pop(field_name, _MISSING_SECRET) + if raw_value is _MISSING_SECRET: + raw_value = os.environ.get(environment_name, _MISSING_SECRET) + if raw_value is _MISSING_SECRET: + env_file = values.get("_env_file", ".env") + if env_file is not None: + env_encoding = values.get("_env_file_encoding", "utf-8") + env_files = (env_file,) if isinstance(env_file, (str, os.PathLike)) else tuple(env_file) + for path in env_files: + raw_value = dotenv_values(path, encoding=env_encoding).get( + environment_name, + raw_value, + ) + return raw_value + + def decode_api_rate_limit_key_secret(value: SecretStr) -> bytes: """Decode one canonical padded Base64 rate-control key.""" secret = value.get_secret_value() if not secret.isascii(): + del value, secret raise ValueError("invalid API rate limit key secret") invalid = False try: @@ -300,7 +664,9 @@ def decode_api_rate_limit_key_secret(value: SecretStr) -> bytes: encoded = b"" decoded = b"" if invalid: + del value, secret, encoded, decoded raise ValueError("invalid API rate limit key secret") if not 32 <= len(decoded) <= 64 or base64.b64encode(decoded) != encoded: + del value, secret, encoded, decoded raise ValueError("invalid API rate limit key secret") return decoded diff --git a/backend/app/core/s3_validation.py b/backend/app/core/s3_validation.py new file mode 100644 index 00000000..bd8d7769 --- /dev/null +++ b/backend/app/core/s3_validation.py @@ -0,0 +1,125 @@ +"""Canonical non-secret S3 namespace validation shared by configuration layers.""" + +from __future__ import annotations + +from collections.abc import Mapping +import ipaddress +import re +from urllib.parse import urlsplit + + +_S3_REGION = re.compile(r"^[a-z]{2,4}(?:-[a-z0-9]+)+-[0-9]+$") +_S3_BUCKET = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") +_S3_BUCKET_LABEL = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$") +_S3_RESERVED_PREFIXES = ("xn--", "sthree-", "amzn-s3-demo-") +_S3_RESERVED_SUFFIXES = ( + "-s3alias", + "--ol-s3", + ".mrap", + "--x-s3", + "--table-s3", + "-an", +) +_S3_PREFIX_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") +_SHA256 = re.compile(r"^sha256:[0-9a-f]{64}$") + + +def is_canonical_s3_region(value: object) -> bool: + """Return whether one region is bounded and canonical.""" + return ( + isinstance(value, str) + and len(value) <= 63 + and _S3_REGION.fullmatch(value) is not None + ) + + +def is_canonical_s3_bucket(value: object) -> bool: + """Return whether one global general-purpose bucket is DNS-compatible.""" + if not isinstance(value, str) or _S3_BUCKET.fullmatch(value) is None: + return False + return ( + ".." not in value + and re.fullmatch(r"\d+\.\d+\.\d+\.\d+", value) is None + and all(_S3_BUCKET_LABEL.fullmatch(label) is not None for label in value.split(".")) + and not value.startswith(_S3_RESERVED_PREFIXES) + and not value.endswith(_S3_RESERVED_SUFFIXES) + ) + + +def is_canonical_s3_prefix(value: object) -> bool: + """Return whether one private object prefix has canonical path segments.""" + if not isinstance(value, str) or not value or len(value) > 512: + return False + return all( + _S3_PREFIX_SEGMENT.fullmatch(segment) is not None + for segment in value.split("/") + ) + + +def canonical_minio_endpoint(value: object) -> str: + """Return one normalized noncredentialed HTTP(S) MinIO origin.""" + missing = value is None or value == "" + endpoint = _try_canonical_minio_endpoint(value) + del value + if endpoint is None: + if missing: + raise ValueError("MinIO artifact storage requires an endpoint") from None + raise ValueError("MinIO artifact storage endpoint is invalid") from None + return endpoint + + +def _try_canonical_minio_endpoint(value: object) -> str | None: + """Normalize one endpoint without retaining invalid input in an exception.""" + if not isinstance(value, str) or not value or len(value) > 2048: + return None + try: + parsed = urlsplit(value) + port = parsed.port + except (UnicodeError, ValueError): + return None + if ( + parsed.scheme.lower() not in {"http", "https"} + or not parsed.hostname + or parsed.username is not None + or parsed.password is not None + or parsed.query + or parsed.fragment + or parsed.path not in {"", "/"} + or parsed.hostname.endswith(".") + ): + return None + scheme = parsed.scheme.lower() + hostname = parsed.hostname.lower() + if ":" in hostname: + try: + hostname = ipaddress.IPv6Address(hostname).compressed + except ipaddress.AddressValueError: + return None + hostname = f"[{hostname}]" + if port is not None and port != {"http": 80, "https": 443}[scheme]: + hostname = f"{hostname}:{port}" + return f"{scheme}://{hostname}" + + +def validate_s3_namespace_descriptor( + provider_profile: str, + descriptor: Mapping[str, str], +) -> None: + """Reject malformed values in one closed S3 namespace descriptor.""" + if descriptor.get("addressing_style") not in {"path", "virtual"}: + raise ValueError("artifact S3 addressing style is invalid") + if not is_canonical_s3_bucket(descriptor.get("bucket")): + raise ValueError("artifact S3 bucket is invalid") + if not is_canonical_s3_prefix(descriptor.get("private_prefix")): + raise ValueError("artifact S3 private prefix is invalid") + if not is_canonical_s3_region(descriptor.get("region")): + raise ValueError("artifact S3 region is invalid") + endpoint_identity = descriptor.get("endpoint_identity") + if provider_profile == "minio-v1": + if not isinstance(endpoint_identity, str) or _SHA256.fullmatch(endpoint_identity) is None: + raise ValueError("artifact MinIO endpoint identity is invalid") + elif provider_profile == "aws-s3-v1": + if endpoint_identity is not None: + raise ValueError("artifact AWS namespace endpoint identity is forbidden") + else: + raise ValueError("artifact S3 provider profile is invalid") diff --git a/backend/app/interfaces/artifacts.py b/backend/app/interfaces/artifacts.py index 9fbd6aa6..2dd8e11f 100644 --- a/backend/app/interfaces/artifacts.py +++ b/backend/app/interfaces/artifacts.py @@ -7,6 +7,7 @@ from typing import Protocol from app.core.hashing import canonical_json_hash +from app.core.s3_validation import validate_s3_namespace_descriptor from app.interfaces.external_services import ( ExternalServiceAdapter, ExternalServiceAdapterIdentity, @@ -21,6 +22,23 @@ ) _NAMESPACE_DESCRIPTOR_KEYS_BY_PROFILE = { "local-v2": frozenset({"private_prefix", "private_root_identity"}), + "minio-v1": frozenset( + { + "addressing_style", + "bucket", + "endpoint_identity", + "private_prefix", + "region", + } + ), + "aws-s3-v1": frozenset( + { + "addressing_style", + "bucket", + "private_prefix", + "region", + } + ), } @@ -133,6 +151,11 @@ def __post_init__(self) -> None: or set(keys) != expected_keys ): raise ValueError("artifact namespace descriptor is not canonical") + if self.provider_profile in {"minio-v1", "aws-s3-v1"}: + validate_s3_namespace_descriptor( + self.provider_profile, + dict(self.descriptor_items), + ) def as_dict(self) -> dict[str, str]: """Return a fresh JSON-compatible descriptor projection.""" @@ -253,6 +276,12 @@ class ArtifactConfigurationError(ArtifactStoreError): category = "configuration" +class ArtifactProviderLiveProofRequiredError(ArtifactConfigurationError): + """Raised while an AWS profile lacks the later live activation proof.""" + + code = "artifact_provider_live_proof_required" + + class ArtifactStore(ExternalServiceAdapter, Protocol): """Immutable byte-provider capability used only by artifact orchestration.""" diff --git a/backend/app/workers/artifacts.py b/backend/app/workers/artifacts.py index 0e376420..88373f2b 100644 --- a/backend/app/workers/artifacts.py +++ b/backend/app/workers/artifacts.py @@ -2,7 +2,10 @@ from __future__ import annotations -from app.adapters.artifacts import cleanup_stale_artifact_scratch +from app.adapters.artifacts import ( + cleanup_stale_artifact_scratch, + require_artifact_runtime_eligible, +) from app.core.config import get_settings from app.workers.async_runner import run_async_task from app.workers.celery_app import ARTIFACT_SCRATCH_CLEANUP_TASK, celery_app @@ -14,4 +17,5 @@ def cleanup_stale_scratch() -> int: settings = get_settings() if settings.artifact_store_backend == "disabled": return 0 + require_artifact_runtime_eligible(settings) return run_async_task(lambda: cleanup_stale_artifact_scratch(settings)) diff --git a/backend/app/workers/celery_app.py b/backend/app/workers/celery_app.py index d1f9b431..0518f6e8 100644 --- a/backend/app/workers/celery_app.py +++ b/backend/app/workers/celery_app.py @@ -4,6 +4,7 @@ from celery import Celery +from app.adapters.artifacts import require_artifact_runtime_eligible from app.core.config import get_settings from app.workers.errors import CeleryConfigurationError @@ -18,6 +19,7 @@ def create_celery_app() -> Celery: Configured Celery application for durable background jobs. """ settings = get_settings() + require_artifact_runtime_eligible(settings) broker_url = settings.celery_broker_url if broker_url is None: if settings.celery_task_always_eager: diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 505e2e56..9b33baf3 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -4,8 +4,10 @@ version = "0.1.0" description = "Workstream backend API" requires-python = ">=3.11" dependencies = [ + "aiobotocore==3.7.0", "alembic>=1.13,<2.0", "asyncpg>=0.29,<1.0", + "botocore==1.43.0", "celery[redis]>=5.4,<6.0", "fastapi>=0.115,<1.0", "httpx>=0.27,<1.0", diff --git a/backend/tests/assertion_helpers.py b/backend/tests/assertion_helpers.py new file mode 100644 index 00000000..23cc2fbf --- /dev/null +++ b/backend/tests/assertion_helpers.py @@ -0,0 +1,98 @@ +"""Shared deep assertions for security-sensitive test object graphs.""" + +from __future__ import annotations + +from collections.abc import Mapping + +from pydantic import SecretStr, ValidationError + + +def assert_secret_not_retained( + value: object, + secret: str, + seen: set[int] | None = None, + *, + traceback_module_prefixes: tuple[str, ...] = (), +) -> None: + """Assert a secret is unreachable through selected error state.""" + if seen is None: + seen = set() + if id(value) in seen: + return + seen.add(id(value)) + if isinstance(value, str): + assert secret not in value + elif isinstance(value, (bytes, bytearray, memoryview)): + assert secret.encode("utf-8") not in bytes(value) + elif isinstance(value, SecretStr): + assert secret not in value.get_secret_value() + elif isinstance(value, BaseException): + if isinstance(value, ValidationError): + assert_secret_not_retained( + value.errors(), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + for related in (value.args, vars(value), value.__cause__, value.__context__): + assert_secret_not_retained( + related, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + traceback = value.__traceback__ + while traceback is not None: + module_name = str(traceback.tb_frame.f_globals.get("__name__", "")) + if module_name.startswith(traceback_module_prefixes): + assert_secret_not_retained( + dict(traceback.tb_frame.f_locals), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + traceback = traceback.tb_next + elif isinstance(value, Mapping): + for key, item in value.items(): + assert_secret_not_retained( + key, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + assert_secret_not_retained( + item, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + elif isinstance(getattr(value, "__dict__", None), Mapping): + assert_secret_not_retained( + vars(value), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + private = getattr(value, "__pydantic_private__", None) + if isinstance(private, Mapping): + assert_secret_not_retained( + private, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + elif isinstance(getattr(value, "__pydantic_private__", None), Mapping): + assert_secret_not_retained( + getattr(value, "__pydantic_private__"), + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) + elif isinstance(value, (list, tuple, set)): + for item in value: + assert_secret_not_retained( + item, + secret, + seen, + traceback_module_prefixes=traceback_module_prefixes, + ) diff --git a/backend/tests/test_artifact_architecture.py b/backend/tests/test_artifact_architecture.py index 2ad35c0d..65aed7da 100644 --- a/backend/tests/test_artifact_architecture.py +++ b/backend/tests/test_artifact_architecture.py @@ -12,6 +12,7 @@ APP_ROOT = BACKEND_ROOT / "app" ARTIFACT_OPERATIONS = APP_ROOT / "interfaces" / "artifact_operations.py" COMPOSITION_ROOT = APP_ROOT / "adapters" / "artifacts" / "__init__.py" +S3_ADAPTER_MODULE = APP_ROOT / "adapters" / "artifacts" / "s3_compatible.py" CLOSED_PORTS = { "GuideArtifactIngestPort", "ContributorArtifactUploadPort", @@ -133,8 +134,37 @@ def test_concrete_adapter_construction_has_one_composition_path() -> None: adapter_calls.append(path) assert factory_calls == [COMPOSITION_ROOT] - assert adapter_calls == [COMPOSITION_ROOT] - assert concrete_imports == [COMPOSITION_ROOT] + assert adapter_calls == [COMPOSITION_ROOT, S3_ADAPTER_MODULE] + assert set(concrete_imports) == {COMPOSITION_ROOT} + + +def test_s3_adapter_exposes_only_required_immutable_object_operations() -> None: + """Keep list, delete, copy, ACL, and multipart behavior out of the provider.""" + allowed = {"get_object", "head_object", "put_object"} + forbidden = { + "abort_multipart_upload", + "complete_multipart_upload", + "copy_object", + "create_multipart_upload", + "delete_object", + "delete_objects", + "list_buckets", + "list_objects", + "list_objects_v2", + "put_object_acl", + "upload_part", + "upload_part_copy", + } + called = { + node.func.attr + for node in ast.walk(_tree(S3_ADAPTER_MODULE)) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and isinstance(node.func.value, ast.Name) + and node.func.value.id == "client" + } + assert called & forbidden == set() + assert called & allowed == allowed def test_provider_methods_stay_inside_artifact_orchestration_and_adapters() -> None: diff --git a/backend/tests/test_artifact_cleanup_wiring.py b/backend/tests/test_artifact_cleanup_wiring.py index e13646a9..1dd1c501 100644 --- a/backend/tests/test_artifact_cleanup_wiring.py +++ b/backend/tests/test_artifact_cleanup_wiring.py @@ -11,7 +11,7 @@ from app.adapters.artifacts import artifact_preparation_limits from app.core.config import Settings, get_settings from app.main import create_app -from app.interfaces.external_services import UnknownExternalServiceProviderError +from app.interfaces.artifacts import ArtifactProviderLiveProofRequiredError def _enabled_settings(tmp_path: Path, **changes: object) -> Settings: @@ -84,6 +84,11 @@ async def test_production_auth_validation_precedes_artifact_startup( environment="production", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method="container-role", ) app = create_app(settings) store = _Store(events) @@ -144,24 +149,37 @@ def unexpected(*_args: object, **_kwargs: object) -> None: pass -async def test_unimplemented_s3_startup_fails_through_typed_factory( +async def test_aws_s3_startup_requires_live_provider_proof( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, ) -> None: - """Keep real startup on the one ADR 0014 unavailable-provider path.""" + """Keep native AWS runtime-ineligible until the owned activation chunk.""" + import app.main as main_module + + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setenv( + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", + "/v2/credentials/workstream-runtime", + ) + monkeypatch.setattr(main_module, "build_auth_verifier", lambda _settings: None) app = create_app( Settings( - environment="test", + environment="production", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method="container-role", ) ) - with pytest.raises(UnknownExternalServiceProviderError) as caught: + with pytest.raises(ArtifactProviderLiveProofRequiredError) as caught: async with app.router.lifespan_context(app): pass - assert caught.value.identity is not None - assert caught.value.identity.provider_key == "s3_compatible" + assert caught.value.code == "artifact_provider_live_proof_required" async def test_namespace_failure_is_fail_closed_and_skips_cleanup( @@ -323,3 +341,32 @@ async def fail_cleanup(_candidate: Settings) -> int: monkeypatch.setattr(artifacts_module, "cleanup_stale_artifact_scratch", fail_cleanup) with pytest.raises(RuntimeError, match="scratch cleanup failed"): artifacts_module.cleanup_stale_scratch.run() + + +def test_aws_s3_is_runtime_ineligible_for_celery_and_cleanup_task( + monkeypatch: pytest.MonkeyPatch, + request: pytest.FixtureRequest, + tmp_path: Path, +) -> None: + """Apply the same inactive-provider guard to API and worker entry points.""" + celery_module, artifacts_module = _load_worker_modules(monkeypatch, request) + settings = Settings( + environment="production", + celery_task_always_eager=True, + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method="container-role", + ) + monkeypatch.setattr(celery_module, "get_settings", lambda: settings) + with pytest.raises(ArtifactProviderLiveProofRequiredError) as startup_error: + celery_module.create_celery_app() + assert startup_error.value.code == "artifact_provider_live_proof_required" + + monkeypatch.setattr(artifacts_module, "get_settings", lambda: settings) + with pytest.raises(ArtifactProviderLiveProofRequiredError) as task_error: + artifacts_module.cleanup_stale_scratch.run() + assert task_error.value.code == "artifact_provider_live_proof_required" diff --git a/backend/tests/test_artifact_store_conformance.py b/backend/tests/test_artifact_store_conformance.py index 27acb8c8..a753989c 100644 --- a/backend/tests/test_artifact_store_conformance.py +++ b/backend/tests/test_artifact_store_conformance.py @@ -259,6 +259,28 @@ def test_namespace_startup_values_are_closed_and_immutable() -> None: "private_prefix": "objects/sha256", "private_root_identity": "sha256:" + "0" * 64, } + for profile, items in ( + ( + "minio-v1", + ( + ("addressing_style", "path"), + ("bucket", "workstream-artifacts"), + ("endpoint_identity", "sha256:" + "1" * 64), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ( + "aws-s3-v1", + ( + ("addressing_style", "virtual"), + ("bucket", "workstream-artifacts"), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ): + assert ArtifactStoreNamespaceIdentity(profile, items).as_dict() == dict(items) for profile, items in ( ("", (("private_prefix", "objects/sha256"),)), @@ -282,6 +304,44 @@ def test_namespace_startup_values_are_closed_and_immutable() -> None: ("private_root_identity", "sha256:" + "0" * 64), ), ), + ( + "minio-v1", + ( + ("addressing_style", "path"), + ("bucket", "workstream-artifacts"), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ( + "aws-s3-v1", + ( + ("addressing_style", "virtual"), + ("bucket", "workstream-artifacts"), + ("endpoint_identity", "sha256:" + "1" * 64), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ( + "minio-v1", + ( + ("addressing_style", "path"), + ("bucket", "Bad..Bucket"), + ("endpoint_identity", "sha256:" + "1" * 64), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ), + ), + ( + "aws-s3-v1", + ( + ("addressing_style", "virtual"), + ("bucket", "workstream-artifacts"), + ("private_prefix", "../bad//prefix"), + ("region", "us-east-1"), + ), + ), ): with pytest.raises(ValueError): ArtifactStoreNamespaceIdentity(profile, items) diff --git a/backend/tests/test_assertion_helpers.py b/backend/tests/test_assertion_helpers.py new file mode 100644 index 00000000..54f8fe25 --- /dev/null +++ b/backend/tests/test_assertion_helpers.py @@ -0,0 +1,31 @@ +"""Tests for shared security-sensitive assertion helpers.""" + +from pydantic import SecretStr +import pytest +from types import SimpleNamespace + +from tests.assertion_helpers import assert_secret_not_retained + + +def test_secret_str_rejects_embedded_forbidden_value() -> None: + """Reject a forbidden value even when wrapped by other secret text.""" + with pytest.raises(AssertionError): + assert_secret_not_retained(SecretStr("prefix-forbidden-suffix"), "forbidden") + + +@pytest.mark.parametrize( + "value", + [b"prefix-forbidden-suffix", bytearray(b"forbidden"), memoryview(b"forbidden")], +) +def test_byte_buffers_reject_embedded_forbidden_value(value: object) -> None: + """Reject a forbidden value retained in any supported byte buffer.""" + with pytest.raises(AssertionError): + assert_secret_not_retained(value, "forbidden") + + +def test_public_object_state_rejects_nested_forbidden_value() -> None: + """Reject a forbidden value reachable through public object state.""" + value = SimpleNamespace(payload={"body": b"prefix-forbidden-suffix"}) + + with pytest.raises(AssertionError): + assert_secret_not_retained(value, "forbidden") diff --git a/backend/tests/test_audit.py b/backend/tests/test_audit.py index 1f5e60ec..709c46ff 100644 --- a/backend/tests/test_audit.py +++ b/backend/tests/test_audit.py @@ -30,26 +30,7 @@ PermissionId, ) from app.modules.tasks.models import AuditEvent - - -def _assert_value_not_retained(value: object, forbidden: str, seen: set[int] | None = None) -> None: - """Traverse public exception state and prove rejected input is absent.""" - seen = seen or set() - if id(value) in seen: - return - seen.add(id(value)) - if isinstance(value, str): - assert forbidden not in value - elif isinstance(value, BaseException): - for item in (value.args, vars(value), value.__cause__, value.__context__): - _assert_value_not_retained(item, forbidden, seen) - elif isinstance(value, Mapping): - for key, item in value.items(): - _assert_value_not_retained(key, forbidden, seen) - _assert_value_not_retained(item, forbidden, seen) - elif isinstance(value, (list, tuple, set)): - for item in value: - _assert_value_not_retained(item, forbidden, seen) +from tests.assertion_helpers import assert_secret_not_retained @pytest.fixture @@ -545,13 +526,13 @@ def __len__(self): for mapping in mappings: with pytest.raises(TypeError, match="invalid authority audit input") as caught: constructor(mapping) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(json.dumps(payload)) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate(HostileMapping()) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) safe = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED).model_dump( mode="json" @@ -636,11 +617,11 @@ def __repr__(self): for document in (raw, raw.encode(), bytearray(raw.encode())): with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(document) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) for document in (HostileBytearray(b"{}"), LyingBytearray(duplicate.encode())): with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(document) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) for patch, forbidden in ( ( @@ -665,10 +646,10 @@ def __repr__(self): for constructor in constructors: with pytest.raises(TypeError, match="invalid authority audit input") as caught: constructor(candidate) - _assert_value_not_retained(caught.value, forbidden) + assert_secret_not_retained(caught.value, forbidden) with pytest.raises(TypeError, match="invalid authority audit input") as caught: AuthorityAuditEventInput.model_validate_json(json.dumps(candidate)) - _assert_value_not_retained(caught.value, forbidden) + assert_secret_not_retained(caught.value, forbidden) with pytest.raises(ValidationError, match="resource ID requires"): _authority_input( AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED, @@ -897,7 +878,7 @@ def __repr__(self): with warnings.catch_warnings(record=True) as caught_warnings: with pytest.raises(TypeError, match="invalid authority audit input") as caught: await AuditService(session).add_authority_event(value) - _assert_value_not_retained(caught.value, secret) + assert_secret_not_retained(caught.value, secret) assert caught_warnings == [] assert await session.get(AuditEvent, str(value.event_id)) is None with warnings.catch_warnings(record=True) as caught_warnings: diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py new file mode 100644 index 00000000..40449d6c --- /dev/null +++ b/backend/tests/test_aws_credential_isolation.py @@ -0,0 +1,1082 @@ +from __future__ import annotations + +import datetime +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest +from aiobotocore.credentials import ( + AioAssumeRoleWithWebIdentityProvider, +) +from botocore.exceptions import MetadataRetrievalError, ReadTimeoutError + +from app.adapters.artifacts import s3_compatible +from app.core.config import Settings +from app.interfaces.artifacts import ArtifactConfigurationError +from tests.assertion_helpers import assert_secret_not_retained + + +def _aws_settings(tmp_path: Path, method: str = "container-role") -> Settings: + return Settings( + environment="production", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="aws_s3", + artifact_s3_region="us-east-1", + artifact_s3_bucket="workstream-artifacts-prod", + artifact_s3_private_prefix="workstream/artifacts", + artifact_s3_addressing_style="virtual", + artifact_s3_credential_mode="aws_workload_identity", + artifact_s3_aws_workload_identity_method=method, # type: ignore[arg-type] + ) + + +def _container_environment(tmp_path: Path, **overrides: str) -> dict[str, str]: + environment = { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": "/v2/credentials/workstream-runtime", + } + environment.update(overrides) + return environment + + +def _web_identity_environment(tmp_path: Path, **overrides: str) -> dict[str, str]: + token = tmp_path / "web-identity-token" + token.write_text("token", encoding="utf-8") + environment = { + "HOME": str(tmp_path), + "AWS_ROLE_ARN": "arn:aws:iam::123456789012:role/workstream-runtime", + "AWS_ROLE_SESSION_NAME": "workstream-artifact-runtime", + "AWS_WEB_IDENTITY_TOKEN_FILE": str(token), + } + environment.update(overrides) + return environment + + +def _iam_environment(tmp_path: Path, **overrides: str) -> dict[str, str]: + environment = { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_V1_DISABLED": "true", + } + environment.update(overrides) + return environment + + +def _assert_rejects_before_session_construction( + monkeypatch: pytest.MonkeyPatch, + settings: Settings, + environment: dict[str, str], + match: str, +) -> None: + events: list[str] = [] + + def session_must_not_construct(*_args: object, **_kwargs: object) -> object: + events.append("session") + raise AssertionError("SDK session constructed before isolation rejection") + + monkeypatch.setattr(s3_compatible, "AioSession", session_must_not_construct) + + with pytest.raises(ArtifactConfigurationError, match=match): + s3_compatible.create_isolated_aws_workload_identity_session( + settings, + environ=environment, + ) + + assert events == [] + + +@pytest.mark.parametrize( + ("name", "value"), + [ + ("AWS_ACCESS_KEY", "legacy-access"), + ("AWS_ACCESS_KEY_ID", "explicit-access"), + ("AWS_SECRET_ACCESS_KEY", "explicit-secret"), + ("AWS_SESSION_TOKEN", "explicit-session-token"), + ("AWS_SECRET_KEY", "legacy-secret"), + ("AWS_SECURITY_TOKEN", "legacy-security-token"), + ("AWS_PROFILE", "prod"), + ("AWS_DEFAULT_PROFILE", "prod"), + ("AWS_SHARED_CREDENTIALS_FILE", "/tmp/credentials"), + ("AWS_CONFIG_FILE", "/tmp/config"), + ("AWS_CREDENTIAL_FILE", "/tmp/legacy-credentials"), + ("AWS_LOGIN_CACHE_DIRECTORY", "/tmp/aws-login-cache"), + ("BOTO_CONFIG", "/tmp/boto.cfg"), + ], +) +def test_forbidden_ambient_aws_sources_fail_before_resolver_loading( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, + value: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment(tmp_path, **{name: value}), + "ambient AWS credential source is forbidden", + ) + + +@pytest.mark.parametrize( + "name", + [ + "AWS_CA_BUNDLE", + "AWS_ENDPOINT_URL", + "AWS_ENDPOINT_URL_STS", + "AWS_STS_REGIONAL_ENDPOINTS", + "AWS_USE_DUALSTACK_ENDPOINT", + "AWS_USE_FIPS_ENDPOINT", + "CURL_CA_BUNDLE", + "REQUESTS_CA_BUNDLE", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + ], +) +def test_web_identity_rejects_ambient_network_configuration_before_session( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "assume-role-with-web-identity"), + _web_identity_environment(tmp_path, **{name: "/tmp/poison"}), + "ambient AWS network configuration is forbidden", + ) + + +@pytest.mark.parametrize("name", ["AWS_ENDPOINT_URL_S3", "AWS_ENDPOINT_URL_DYNAMODB"]) +def test_service_specific_endpoint_override_fails_before_session_construction( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment(tmp_path, **{name: "http://127.0.0.1:9000"}), + "ambient AWS network configuration is forbidden", + ) + + +@pytest.mark.parametrize( + "name", + [ + "AWS_ACCOUNT_ID_ENDPOINT_MODE", + "AWS_DATA_PATH", + "AWS_DEFAULTS_MODE", + "AWS_ENDPOINT_DISCOVERY_ENABLED", + "AWS_MAX_ATTEMPTS", + "AWS_REGION", + "AWS_RETRY_MODE", + "BOTOCORE_CONFIG", + ], +) +@pytest.mark.parametrize( + ("method", "environment"), + [ + ("assume-role-with-web-identity", _web_identity_environment), + ("container-role", _container_environment), + ("iam-role", _iam_environment), + ], +) +def test_unapproved_sdk_environment_controls_fail_before_session_construction( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + name: str, + method: str, + environment: Any, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, method), + environment(tmp_path, **{name: "poison"}), + "unsupported AWS SDK environment configuration is forbidden", + ) + + +@pytest.mark.parametrize( + ("relative_path", "contents"), + [ + (".aws/credentials", "[default]\naws_access_key_id = poison\n"), + (".aws/config", "[profile prod]\ncredential_process = /bin/false\n"), + (".aws/config", "[profile prod]\nsso_start_url = https://example.test/start\n"), + (".boto", "[Credentials]\naws_access_key_id = poison\n"), + ], +) +def test_default_credential_files_fail_before_resolver_loading( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + relative_path: str, + contents: str, +) -> None: + poison = tmp_path / relative_path + poison.parent.mkdir(parents=True, exist_ok=True) + poison.write_text(contents, encoding="utf-8") + + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment(tmp_path), + "ambient AWS credential file is forbidden", + ) + + +@pytest.mark.parametrize( + ("selected_method", "environment"), + [ + ( + "assume-role-with-web-identity", + lambda tmp_path: _web_identity_environment( + tmp_path, + AWS_CONTAINER_CREDENTIALS_RELATIVE_URI="/v2/credentials/poison", + ), + ), + ( + "container-role", + lambda tmp_path: _container_environment( + tmp_path, + AWS_ROLE_ARN="arn:aws:iam::123456789012:role/unselected", + AWS_WEB_IDENTITY_TOKEN_FILE=str(tmp_path / "unselected-token"), + ), + ), + ( + "iam-role", + lambda tmp_path: _iam_environment( + tmp_path, + AWS_CONTAINER_CREDENTIALS_FULL_URI="http://127.0.0.1:9/credentials", + ), + ), + ], +) +def test_unselected_workload_identity_sources_fail_before_resolver_loading( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + selected_method: str, + environment: Any, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, selected_method), + environment(tmp_path), + "unselected AWS workload source is configured", + ) + + +@pytest.mark.parametrize( + ("method", "environment", "provider_type", "sdk_method"), + [ + ( + "assume-role-with-web-identity", + _web_identity_environment, + AioAssumeRoleWithWebIdentityProvider, + "assume-role-with-web-identity", + ), + ( + "container-role", + _container_environment, + s3_compatible._SanitizedContainerProvider, # noqa: SLF001 + "container-role", + ), + ( + "iam-role", + _iam_environment, + s3_compatible._SanitizedInstanceMetadataProvider, # noqa: SLF001 + "iam-role", + ), + ], +) +def test_isolated_session_contains_exactly_one_selected_provider( + tmp_path: Path, + method: str, + environment: Any, + provider_type: type[object], + sdk_method: str, +) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, method), + environ=environment(tmp_path), + ) + + resolver = session.get_component("credential_provider") + + assert len(resolver.providers) == 1 + provider = resolver.providers[0] + assert type(provider) is provider_type + assert provider.METHOD == sdk_method + if method == "assume-role-with-web-identity": + assert provider._disable_env_vars is True + assert provider._load_config() == { + "profiles": { + "workstream-isolated": { + "role_arn": "arn:aws:iam::123456789012:role/workstream-runtime", + "role_session_name": "workstream-artifact-runtime", + "web_identity_token_file": str(tmp_path / "web-identity-token"), + } + } + } + if method == "container-role": + assert provider._environ == { + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ) + } + assert ( + provider._fetcher._session._proxy_config.proxy_url_for( + "http://169.254.170.2/v2/credentials/workstream-runtime" + ) + is None + ) + if method == "iam-role": + assert provider._role_fetcher._imds_v1_disabled is True + assert ( + provider._role_fetcher._session._proxy_config.proxy_url_for( + "http://169.254.169.254/latest/meta-data/" + ) + is None + ) + + +@pytest.mark.parametrize( + ("method", "environment", "message"), + [ + ( + "assume-role-with-web-identity", + lambda tmp_path: {"HOME": str(tmp_path), "AWS_ROLE_ARN": "arn"}, + "web identity configuration is incomplete", + ), + ( + "assume-role-with-web-identity", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_ROLE_ARN": "arn", + "AWS_WEB_IDENTITY_TOKEN_FILE": "relative-token", + }, + "web identity token path is invalid", + ), + ( + "container-role", + lambda tmp_path: {"HOME": str(tmp_path)}, + "container identity location is invalid", + ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_FULL_URI": "https://credentials.example.test", + }, + "container full credential URI is forbidden", + ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_FULL_URI": "http://127.0.0.1:9/full", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ), + }, + "container full credential URI is forbidden", + ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ), + "AWS_CONTAINER_AUTHORIZATION_TOKEN": "token", + }, + "container identity token is forbidden", + ), + ( + "container-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ), + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE": str(tmp_path / "token"), + }, + "container identity token is forbidden", + ), + ( + "iam-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_DISABLED": "true", + "AWS_EC2_METADATA_V1_DISABLED": "true", + }, + "instance identity metadata is disabled", + ), + ( + "iam-role", + lambda tmp_path: {"HOME": str(tmp_path)}, + "instance identity requires IMDSv2", + ), + ( + "iam-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_V1_DISABLED": "false", + }, + "instance identity requires IMDSv2", + ), + ( + "iam-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_SERVICE_ENDPOINT": ( + "https://metadata.example.test/latest" + ), + }, + "custom AWS metadata endpoint is forbidden", + ), + ( + "iam-role", + lambda tmp_path: { + "HOME": str(tmp_path), + "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE": "ipv6", + "AWS_EC2_METADATA_V1_DISABLED": "true", + }, + "custom AWS metadata endpoint mode is forbidden", + ), + ], +) +def test_selected_workload_identity_source_must_be_complete( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + method: str, + environment: Any, + message: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, method), + environment(tmp_path), + message, + ) + + +@pytest.mark.parametrize( + "relative_uri", + [ + "/relative", + "/v2/credentials/", + "/v2/credentials/workstream/runtime", + "https://credentials.example.test/v2/credentials/runtime", + "/v2/credentials/" + "a" * 513, + ], +) +def test_container_role_rejects_noncanonical_relative_uri( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + relative_uri: str, +) -> None: + _assert_rejects_before_session_construction( + monkeypatch, + _aws_settings(tmp_path, "container-role"), + _container_environment( + tmp_path, + AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=relative_uri, + ), + "container identity location is invalid", + ) + + +def test_iam_metadata_session_ignores_ambient_proxy_environment(tmp_path: Path) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, "iam-role"), + environ=_iam_environment( + tmp_path, + HTTP_PROXY="http://127.0.0.1:9", + HTTPS_PROXY="http://127.0.0.1:9", + ALL_PROXY="http://127.0.0.1:9", + ), + ) + provider = session.get_component("credential_provider").providers[0] + + assert ( + provider._role_fetcher._session._proxy_config.proxy_url_for( + "http://169.254.169.254/latest/meta-data/" + ) + is None + ) + + +def test_container_metadata_session_ignores_ambient_proxy_environment( + tmp_path: Path, +) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, "container-role"), + environ=_container_environment( + tmp_path, + HTTP_PROXY="http://127.0.0.1:9", + HTTPS_PROXY="http://127.0.0.1:9", + ALL_PROXY="http://127.0.0.1:9", + ), + ) + provider = session.get_component("credential_provider").providers[0] + + assert provider._environ == { + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ) + } + assert ( + provider._fetcher._session._proxy_config.proxy_url_for( + "http://169.254.170.2/v2/credentials/workstream-runtime" + ) + is None + ) + + +def test_web_identity_sts_client_ignores_ambient_proxy_environment( + tmp_path: Path, +) -> None: + session = s3_compatible.create_isolated_aws_workload_identity_session( + _aws_settings(tmp_path, "assume-role-with-web-identity"), + environ=_web_identity_environment( + tmp_path, + HTTP_PROXY="http://127.0.0.1:9", + HTTPS_PROXY="http://127.0.0.1:9", + ALL_PROXY="http://127.0.0.1:9", + ), + ) + provider = session.get_component("credential_provider").providers[0] + observed: dict[str, object] = {} + marker = object() + + def capture_client(service_name: str, **kwargs: object) -> object: + observed["service_name"] = service_name + observed.update(kwargs) + return marker + + session.create_client = capture_client # type: ignore[method-assign] + + created = provider._client_creator("sts", config=object()) + + assert created is marker + assert observed["service_name"] == "sts" + assert observed["region_name"] == "us-east-1" + assert observed["verify"] is True + config = observed["config"] + assert config.signature_version == s3_compatible.UNSIGNED + assert config.proxies == {} + + +async def test_resolved_workload_identity_method_must_match() -> None: + materialized: list[str] = [] + + class FakeCredentials: + def __init__(self, method: str) -> None: + self.method = method + + async def get_frozen_credentials(self) -> object: + materialized.append(self.method) + return SimpleNamespace(access_key="access", secret_key="secret", token=None) + + class FakeSession: + def __init__(self, method: str | None) -> None: + self._method = method + + async def get_credentials(self) -> object | None: + if self._method is None: + return None + return FakeCredentials(self._method) + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity method did not match", + ): + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession("iam-role"), # type: ignore[arg-type] + expected_method="container-role", + ) + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity method did not match", + ): + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(None), # type: ignore[arg-type] + expected_method="container-role", + ) + + resolved = await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession("container-role"), # type: ignore[arg-type] + expected_method="container-role", + ) + assert getattr(resolved, "method") == "container-role" + assert materialized == ["container-role"] + + +async def test_workload_identity_materialization_failure_is_sanitized() -> None: + secret = "deferred-credential-secret" + + class DeferredCredentials: + method = "container-role" + + async def get_frozen_credentials(self) -> object: + raise RuntimeError(f"credential refresh returned {secret}") + + class FakeSession: + async def get_credentials(self) -> object: + return DeferredCredentials() + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity credentials could not be resolved", + ) as caught: + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(), # type: ignore[arg-type] + expected_method="container-role", + ) + + assert caught.value.__cause__ is None + assert caught.value.__context__ is None + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) + + +async def test_credential_resolution_failure_is_sanitized() -> None: + secret = "provider-response-secret" + + class FakeSession: + async def get_credentials(self) -> object: + raise RuntimeError(f"credential endpoint returned {secret}") + + with pytest.raises( + ArtifactConfigurationError, + match="AWS workload identity credentials could not be resolved", + ) as caught: + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(), # type: ignore[arg-type] + expected_method="container-role", + ) + + assert caught.value.__cause__ is None + assert caught.value.__context__ is None + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) + + +async def test_container_metadata_error_body_is_never_logged_or_retained( + caplog: pytest.LogCaptureFixture, +) -> None: + secret = "ecs-response-secret" + + class Transport: + async def send(self, _request: object) -> object: + async def content() -> bytes: + return f'{{"SecretAccessKey":"{secret}"'.encode() + + return SimpleNamespace(status_code=200, content=content()) + + class Acquisition: + async def __aenter__(self) -> Transport: + return Transport() + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def acquire(self) -> Acquisition: + return Acquisition() + + async def no_sleep(_delay: float) -> None: + return None + + fetcher = s3_compatible._SanitizedContainerMetadataFetcher( # noqa: SLF001 + session=Pool(), # type: ignore[arg-type] + sleep=no_sleep, + ) + provider = s3_compatible._SanitizedContainerProvider( # noqa: SLF001 + environ={ + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI": ( + "/v2/credentials/workstream-runtime" + ) + } + ) + provider._fetcher = fetcher + + class FakeSession: + async def get_credentials(self) -> object: + return await provider.load() + + caplog.set_level("DEBUG") + with pytest.raises(ArtifactConfigurationError) as caught: + await s3_compatible.resolve_isolated_aws_workload_credentials( + FakeSession(), # type: ignore[arg-type] + expected_method="container-role", + ) + + assert secret not in caplog.text + for record in caplog.records: + assert_secret_not_retained( + (record.msg, record.args, record.exc_info, record.exc_text), + secret, + traceback_module_prefixes=( + "aiobotocore.credentials", + "app.adapters.artifacts.s3_compatible", + ), + ) + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) + + +async def test_imds_error_responses_are_never_logged( + caplog: pytest.LogCaptureFixture, +) -> None: + secret = "imds-response-secret" + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + ) + + response = SimpleNamespace( + status_code=500, + url="http://169.254.169.254/latest/meta-data/iam/security-credentials/", + content=secret.encode(), + ) + caplog.set_level("DEBUG") + + assert await fetcher._is_non_ok_response(response) is True # noqa: SLF001 + + async def token() -> str: + return "imds-v2-token" + + async def role_name(_token: str) -> str: + return "workstream-role" + + async def credentials(_role_name: str, _token: str) -> dict[str, str]: + return {"Code": "Error", "Message": secret} + + fetcher._fetch_metadata_token = token # type: ignore[method-assign] + fetcher._get_iam_role = role_name # type: ignore[method-assign] + fetcher._get_credentials = credentials # type: ignore[method-assign] + assert await fetcher.retrieve_iam_role_credentials() == {} + + class FakeRoleFetcher: + async def retrieve_iam_role_credentials(self) -> dict[str, str]: + return { + "role_name": secret, + "access_key": "access-key", + "secret_key": "secret-key", + "token": "session-token", + "expiry_time": "2099-01-01T00:00:00Z", + } + + provider = s3_compatible._SanitizedInstanceMetadataProvider( # noqa: SLF001 + iam_role_fetcher=FakeRoleFetcher() + ) + assert await provider.load() is not None + assert secret not in caplog.text + for record in caplog.records: + assert_secret_not_retained( + (record.msg, record.args, record.exc_info, record.exc_text), + secret, + ) + + +def test_imds_invalid_expiration_is_never_logged( + caplog: pytest.LogCaptureFixture, +) -> None: + secret = "imds-expiration-secret" + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + ) + credentials = {"expiry_time": secret} + caplog.set_level("DEBUG") + + fetcher._evaluate_expiration(credentials) # noqa: SLF001 + + assert credentials == {"expiry_time": secret} + assert secret not in caplog.text + for record in caplog.records: + assert_secret_not_retained( + (record.msg, record.args, record.exc_info, record.exc_text), + secret, + ) + + +async def test_sanitized_imds_fetcher_success_paths() -> None: + async def ready(value: object) -> object: + return value + + token_response = SimpleNamespace( + status_code=200, + text=ready("imds-v2-token"), + ) + + class Transport: + def __init__(self, response: object) -> None: + self.response = response + + async def send(self, _request: object) -> object: + return self.response + + class Acquisition: + def __init__(self, transport: Transport) -> None: + self.transport = transport + + async def __aenter__(self) -> Transport: + return self.transport + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def __init__(self, response: object) -> None: + self.transport = Transport(response) + + def acquire(self) -> Acquisition: + return Acquisition(self.transport) + + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + session=Pool(token_response), # type: ignore[arg-type] + ) + assert await fetcher._fetch_metadata_token() == "imds-v2-token" # noqa: SLF001 + + metadata_response = SimpleNamespace(status_code=200) + fetcher._session = Pool(metadata_response) + + async def no_retry(_response: object) -> bool: + return False + + assert ( + await fetcher._get_request( # noqa: SLF001 + "latest/meta-data/iam/security-credentials/workstream-role", + no_retry, + token="imds-v2-token", + ) + is metadata_response + ) + + async def token() -> str: + return "imds-v2-token" + + async def role_name(_token: str) -> str: + return "workstream-role" + + async def credentials(_role_name: str, _token: str) -> dict[str, str]: + return { + "AccessKeyId": "access-key", + "SecretAccessKey": "secret-key", + "Token": "session-token", + "Expiration": "2099-01-01T00:00:00Z", + } + + fetcher._fetch_metadata_token = token # type: ignore[method-assign] + fetcher._get_iam_role = role_name # type: ignore[method-assign] + fetcher._get_credentials = credentials # type: ignore[method-assign] + assert await fetcher.retrieve_iam_role_credentials() == { + "role_name": "workstream-role", + "access_key": "access-key", + "secret_key": "secret-key", + "token": "session-token", + "expiry_time": "2099-01-01T00:00:00Z", + } + + +async def test_sanitized_container_provider_success_path() -> None: + class Fetcher: + async def retrieve_full_uri( + self, + _full_uri: str, + *, + headers: dict[str, str], + ) -> dict[str, str]: + assert headers == {"Authorization": "Bearer identity-token"} + return { + "AccessKeyId": "access-key", + "SecretAccessKey": "secret-key", + "Token": "session-token", + "Expiration": "2099-01-01T00:00:00Z", + "AccountId": "123456789012", + } + + provider = s3_compatible._SanitizedContainerProvider( # noqa: SLF001 + environ={ + "AWS_CONTAINER_CREDENTIALS_FULL_URI": ( + "http://169.254.170.2/v2/credentials/workstream-runtime" + ), + "AWS_CONTAINER_AUTHORIZATION_TOKEN": "Bearer identity-token", + } + ) + provider._fetcher = Fetcher() + fetch_credentials = provider._create_fetcher( # noqa: SLF001 + "http://169.254.170.2/v2/credentials/workstream-runtime" + ) + + assert await fetch_credentials() == { + "access_key": "access-key", + "secret_key": "secret-key", + "token": "session-token", + "expiry_time": "2099-01-01T00:00:00Z", + "account_id": "123456789012", + } + + +@pytest.mark.parametrize( + ("status_code", "body", "match"), + [ + (500, b'{"SecretAccessKey":"provider-secret"}', "non-200"), + (200, b"[]", "invalid document"), + ], +) +async def test_sanitized_container_fetcher_rejects_response_shapes( + status_code: int, + body: bytes, + match: str, +) -> None: + class Transport: + async def send(self, _request: object) -> object: + async def content() -> bytes: + return body + + return SimpleNamespace(status_code=status_code, content=content()) + + class Acquisition: + async def __aenter__(self) -> Transport: + return Transport() + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def acquire(self) -> Acquisition: + return Acquisition() + + fetcher = s3_compatible._SanitizedContainerMetadataFetcher( # noqa: SLF001 + session=Pool(), # type: ignore[arg-type] + ) + + with pytest.raises(MetadataRetrievalError, match=match): + await fetcher._get_response( # noqa: SLF001 + "http://169.254.170.2/v2/credentials/workstream-runtime", + {"Accept": "application/json"}, + 1.0, + ) + + +async def test_sanitized_container_fetcher_accepts_headers_and_document() -> None: + observed_headers: dict[str, str] = {} + + class Transport: + async def send(self, request: object) -> object: + observed_headers.update(dict(request.headers.items())) + + async def content() -> bytes: + return b'{"AccessKeyId":"access-key"}' + + return SimpleNamespace(status_code=200, content=content()) + + class Acquisition: + async def __aenter__(self) -> Transport: + return Transport() + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def acquire(self) -> Acquisition: + return Acquisition() + + fetcher = s3_compatible._SanitizedContainerMetadataFetcher( # noqa: SLF001 + session=Pool(), # type: ignore[arg-type] + ) + + assert await fetcher._retrieve_credentials( # noqa: SLF001 + "http://169.254.170.2/v2/credentials/workstream-runtime", + {"Authorization": "Bearer identity-token"}, + ) == {"AccessKeyId": "access-key"} + assert observed_headers["Authorization"] == "Bearer identity-token" + + +async def test_sanitized_imds_fetcher_failure_and_empty_paths( + monkeypatch: pytest.MonkeyPatch, +) -> None: + class Transport: + def __init__(self, response: object | None = None) -> None: + self.response = response + self.raise_timeout = False + + async def send(self, _request: object) -> object: + if self.raise_timeout: + raise ReadTimeoutError(endpoint_url="http://169.254.169.254") + return self.response + + class Acquisition: + def __init__(self, transport: Transport) -> None: + self.transport = transport + + async def __aenter__(self) -> Transport: + return self.transport + + async def __aexit__(self, *_args: object) -> None: + return None + + class Pool: + def __init__(self, transport: Transport) -> None: + self.transport = transport + + def acquire(self) -> Acquisition: + return Acquisition(self.transport) + + forbidden = Transport(SimpleNamespace(status_code=403, text="forbidden")) + fetcher = s3_compatible._SanitizedInstanceMetadataFetcher( # noqa: SLF001 + env={"AWS_EC2_METADATA_V1_DISABLED": "true"}, + config={"ec2_metadata_v1_disabled": True}, + session=Pool(forbidden), # type: ignore[arg-type] + ) + assert await fetcher._fetch_metadata_token() is None # noqa: SLF001 + + timeout = Transport() + timeout.raise_timeout = True + fetcher._session = Pool(timeout) + with pytest.raises(fetcher._RETRIES_EXCEEDED_ERROR_CLS): # noqa: SLF001 + await fetcher._get_request( # noqa: SLF001 + "latest/meta-data/iam/security-credentials/workstream-role", + lambda _response: False, + token="imds-v2-token", + ) + + async def retries_exceeded() -> str: + raise fetcher._RETRIES_EXCEEDED_ERROR_CLS() # noqa: SLF001 + + fetcher._fetch_metadata_token = retries_exceeded # type: ignore[method-assign] + assert await fetcher.retrieve_iam_role_credentials() == {} + + fetcher._evaluate_expiration({}) # noqa: SLF001 + monkeypatch.setattr( + s3_compatible, + "get_current_datetime", + lambda: datetime.datetime(2026, 1, 1), + ) + monkeypatch.setattr(s3_compatible.random, "randint", lambda _start, _end: 120) + expiring = {"expiry_time": "2026-01-01T00:01:00Z"} + fetcher._evaluate_expiration(expiring) # noqa: SLF001 + assert expiring["expiry_time"] == "2026-01-01T00:12:00Z" + + class EmptyRoleFetcher: + async def retrieve_iam_role_credentials(self) -> dict[str, str]: + return {} + + provider = s3_compatible._SanitizedInstanceMetadataProvider( # noqa: SLF001 + iam_role_fetcher=EmptyRoleFetcher() + ) + assert await provider.load() is None diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 114e6740..8aec4145 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -1,52 +1,39 @@ from __future__ import annotations import base64 +import importlib.metadata import json +import tomllib from collections.abc import Mapping from pathlib import Path from typing import get_args import pytest -from pydantic import SecretStr, ValidationError +from pydantic import ValidationError from pydantic_settings import BaseSettings from app.adapters.artifacts import create_artifact_store_bootstrap +from app.adapters.artifacts.s3_compatible import create_minio_artifact_store_bootstrap from app.adapters.auth.flow import FlowAuthVerifier from app.api.deps.auth import get_application_auth_verifier from app.core.auth import clear_auth_verifier_cache, get_auth_verifier from app.core.config import Settings, get_settings +from app.core.s3_validation import ( + canonical_minio_endpoint, + is_canonical_s3_region, + is_canonical_s3_prefix, + validate_s3_namespace_descriptor, +) +from app.interfaces.artifacts import ( + ArtifactConfigurationError, + ArtifactProviderLiveProofRequiredError, +) from app.interfaces.external_services import ( ExternalServiceConfigurationError, UnknownExternalServiceProviderError, ) from app.main import create_app - - -def _assert_secret_not_retained(value: object, secret: str, seen: set[int] | None = None) -> None: - """Assert a secret is unreachable through an error's public object graph.""" - if seen is None: - seen = set() - if id(value) in seen: - return - seen.add(id(value)) - if isinstance(value, str): - assert secret not in value - elif isinstance(value, SecretStr): - assert value.get_secret_value() != secret - elif isinstance(value, BaseException): - if isinstance(value, ValidationError): - _assert_secret_not_retained(value.errors(), secret, seen) - _assert_secret_not_retained(value.args, secret, seen) - _assert_secret_not_retained(vars(value), secret, seen) - _assert_secret_not_retained(value.__cause__, secret, seen) - _assert_secret_not_retained(value.__context__, secret, seen) - elif isinstance(value, Mapping): - for key, item in value.items(): - _assert_secret_not_retained(key, secret, seen) - _assert_secret_not_retained(item, secret, seen) - elif isinstance(value, (list, tuple, set)): - for item in value: - _assert_secret_not_retained(item, secret, seen) +from tests.assertion_helpers import assert_secret_not_retained def test_default_settings_are_fail_closed(monkeypatch: pytest.MonkeyPatch) -> None: @@ -132,7 +119,11 @@ def test_rate_limit_secret_is_absent_from_unrelated_structured_errors() -> None: assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) def test_environment_rate_limit_secret_is_absent_from_unrelated_structured_errors( @@ -146,7 +137,11 @@ def test_environment_rate_limit_secret_is_absent_from_unrelated_structured_error assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) def test_dotenv_rate_limit_secret_is_absent_from_unrelated_structured_errors( @@ -164,7 +159,11 @@ def test_dotenv_rate_limit_secret_is_absent_from_unrelated_structured_errors( assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) def test_model_validate_rejects_rate_limit_secret_without_structured_echo() -> None: @@ -175,7 +174,11 @@ def test_model_validate_rejects_rate_limit_secret_without_structured_echo() -> N assert not isinstance(caught.value, ValidationError) assert invalid not in f"{caught.value!s} {caught.value!r}" - _assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained( + caught.value, + invalid, + traceback_module_prefixes=("app.",), + ) def test_model_validate_rate_limit_secret_is_absent_from_unrelated_errors() -> None: @@ -191,7 +194,11 @@ def test_model_validate_rate_limit_secret_is_absent_from_unrelated_errors() -> N assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( @@ -218,7 +225,7 @@ def capture_input( obj: object, **kwargs: object, ) -> BaseSettings: - observed["input"] = obj + observed["input"] = dict(obj) if isinstance(obj, Mapping) else obj raise PydanticBoundaryReached monkeypatch.setattr(BaseSettings, base_method_name, classmethod(capture_input)) @@ -230,7 +237,7 @@ def capture_input( assert isinstance(observed["input"], Mapping) assert observed["input"]["api_rate_limit_key_secret"] is None - _assert_secret_not_retained(observed, encoded) + assert_secret_not_retained(observed, encoded) @pytest.mark.parametrize("method_name", ["model_validate_json", "model_validate_strings"]) @@ -258,7 +265,11 @@ def test_alternate_validation_rejects_rate_limit_secret_without_echo( assert not isinstance(caught.value, ValidationError) assert invalid not in f"{caught.value!s} {caught.value!r}" - _assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained( + caught.value, + invalid, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize("method_name", ["model_validate_json", "model_validate_strings"]) @@ -277,12 +288,22 @@ def test_alternate_validation_secret_is_absent_from_unrelated_errors( assert encoded not in repr(caught.value.errors()) assert encoded not in caught.value.json() - _assert_secret_not_retained(caught.value, encoded) + assert_secret_not_retained( + caught.value, + encoded, + traceback_module_prefixes=("app.",), + ) -def test_model_validate_json_rejects_malformed_document_without_echo() -> None: +@pytest.mark.parametrize( + "field_name", + ["api_rate_limit_key_secret", "artifact_s3_secret_access_key"], +) +def test_model_validate_json_rejects_malformed_document_without_echo( + field_name: str, +) -> None: invalid = "not-a-canonical-secret" - payload = f'{{"api_rate_limit_key_secret":"{invalid}"' + payload = f'{{"{field_name}":"{invalid}"' with pytest.raises(ValueError, match="^invalid settings JSON$") as caught: Settings.model_validate_json(payload) @@ -291,7 +312,11 @@ def test_model_validate_json_rejects_malformed_document_without_echo() -> None: assert invalid not in f"{caught.value!s} {caught.value!r}" assert caught.value.__cause__ is None assert caught.value.__context__ is None - _assert_secret_not_retained(caught.value, invalid) + assert_secret_not_retained( + caught.value, + invalid, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( @@ -315,7 +340,11 @@ def test_rate_limit_secret_rejects_invalid_values_without_echo(value: str) -> No assert not hasattr(caught.value, "json") if value.strip(): assert value not in rendered - _assert_secret_not_retained(caught.value, value) + assert_secret_not_retained( + caught.value, + value, + traceback_module_prefixes=("app.",), + ) @pytest.mark.parametrize( @@ -355,6 +384,46 @@ def _flow_settings(**overrides) -> Settings: return Settings(**values) +def _minio_setting_values(tmp_path: Path, **overrides: object) -> dict[str, object]: + values: dict[str, object] = { + "environment": "test", + "artifact_store_backend": "s3_compatible", + "artifact_scratch_root": tmp_path / "scratch", + "artifact_s3_provider_profile": "minio", + "artifact_s3_region": "us-east-1", + "artifact_s3_endpoint_url": "http://127.0.0.1:9000", + "artifact_s3_bucket": "workstream-artifacts-test", + "artifact_s3_private_prefix": "workstream/artifacts", + "artifact_s3_addressing_style": "path", + "artifact_s3_credential_mode": "local_static", + "artifact_s3_access_key_id": "minio-access-key", + "artifact_s3_secret_access_key": "minio-secret-key", + } + values.update(overrides) + return values + + +def _minio_settings(tmp_path: Path, **overrides: object) -> Settings: + return Settings(**_minio_setting_values(tmp_path, **overrides)) + + +def _aws_settings(tmp_path: Path, **overrides: object) -> Settings: + values: dict[str, object] = { + "environment": "production", + "artifact_store_backend": "s3_compatible", + "artifact_scratch_root": tmp_path / "scratch", + "artifact_s3_provider_profile": "aws_s3", + "artifact_s3_region": "us-east-1", + "artifact_s3_bucket": "workstream-artifacts-prod", + "artifact_s3_private_prefix": "workstream/artifacts", + "artifact_s3_addressing_style": "virtual", + "artifact_s3_credential_mode": "aws_workload_identity", + "artifact_s3_aws_workload_identity_method": "container-role", + } + values.update(overrides) + return Settings(**values) + + @pytest.mark.parametrize( ("overrides", "message"), [ @@ -551,15 +620,672 @@ def test_artifact_scratch_settings_are_bounded_and_separate(tmp_path) -> None: ) -def test_s3_compatible_factory_fails_typed_until_adapter_chunk(tmp_path: Path) -> None: - """Keep valid future S3 configuration unavailable until its owned chunk.""" - settings = Settings( - environment="test", - artifact_store_backend="s3_compatible", - artifact_scratch_root=tmp_path / "scratch", +def test_minio_settings_and_factory_construct_closed_namespace(tmp_path: Path) -> None: + """MinIO is the active S3-compatible runtime profile for local and CI proof.""" + settings = _minio_settings(tmp_path) + + bootstrap = create_artifact_store_bootstrap(settings) + + try: + assert bootstrap.identity.capability_key == "artifact_store" + assert bootstrap.identity.provider_key == "s3_compatible" + assert bootstrap.namespace_identity.provider_profile == "minio-v1" + assert bootstrap.namespace_identity.descriptor_items == ( + ("addressing_style", "path"), + ("bucket", "workstream-artifacts-test"), + ("endpoint_identity", bootstrap.namespace_identity.as_dict()["endpoint_identity"]), + ("private_prefix", "workstream/artifacts"), + ("region", "us-east-1"), + ) + endpoint_identity = bootstrap.namespace_identity.as_dict()["endpoint_identity"] + assert endpoint_identity.startswith("sha256:") + assert "127.0.0.1" not in endpoint_identity + assert "minio-secret-key" not in repr(bootstrap.namespace_identity) + finally: + bootstrap.close() + + +@pytest.mark.parametrize( + ("overrides", "message"), + [ + ({"artifact_s3_provider_profile": None}, "requires a provider profile"), + ({"artifact_s3_region": "US-east-1"}, "requires a canonical region"), + ({"artifact_s3_region": "us-east-1-"}, "requires a canonical region"), + ({"artifact_s3_region": "1-east-1"}, "requires a canonical region"), + ({"artifact_s3_region": "a-east-1"}, "requires a canonical region"), + ({"artifact_s3_bucket": "192.168.0.1"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket..name"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket.-name"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket-.name"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "xn--bucket"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "sthree-bucket"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "amzn-s3-demo-bucket"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket-s3alias"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket--ol-s3"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket.mrap"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket--x-s3"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket--table-s3"}, "requires a canonical bucket"), + ({"artifact_s3_bucket": "bucket-an"}, "requires a canonical bucket"), + ({"artifact_s3_private_prefix": "/bad"}, "private prefix is invalid"), + ({"artifact_maximum_bytes": 512 * 1024 * 1024 + 1}, "maximum exceeds"), + ({"environment": "production"}, "MinIO artifact storage is restricted"), + ({"artifact_s3_credential_mode": "aws_workload_identity"}, "local static"), + ( + {"artifact_s3_aws_workload_identity_method": "iam-role"}, + "cannot select AWS workload identity", + ), + ({"artifact_s3_endpoint_url": None}, "requires an endpoint"), + ({"artifact_s3_endpoint_url": "https://user:pass@minio.test"}, "endpoint is invalid"), + ({"artifact_s3_endpoint_url": "http://minio.test/path"}, "endpoint is invalid"), + ], +) +def test_minio_settings_fail_closed( + tmp_path: Path, + overrides: dict[str, object], + message: str, +) -> None: + with pytest.raises((ValidationError, ValueError), match=message): + _minio_settings(tmp_path, **overrides) + + +def test_mutated_minio_settings_cannot_retain_credentials_in_constructor_error( + tmp_path: Path, +) -> None: + access_key = "minio-mutated-access-key" + secret_key = "minio-mutated-secret-key" + settings = _minio_settings( + tmp_path, + artifact_s3_access_key_id=access_key, + artifact_s3_secret_access_key=secret_key, + ) + settings.artifact_s3_bucket = "bucket.-invalid" + + with pytest.raises(ArtifactConfigurationError) as caught: + create_minio_artifact_store_bootstrap(settings) + + assert caught.value.__cause__ is None + assert caught.value.__context__ is None + assert_secret_not_retained( + caught.value, + access_key, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), ) - with pytest.raises(UnknownExternalServiceProviderError) as caught: + assert_secret_not_retained( + caught.value, + secret_key, + traceback_module_prefixes=("app.adapters.artifacts.s3_compatible",), + ) + + with pytest.raises(ExternalServiceConfigurationError) as public_error: create_artifact_store_bootstrap(settings) + assert_secret_not_retained( + public_error.value, + access_key, + traceback_module_prefixes=( + "app.adapters.artifacts.s3_compatible", + "app.interfaces.external_services", + ), + ) + assert_secret_not_retained( + public_error.value, + secret_key, + traceback_module_prefixes=( + "app.adapters.artifacts.s3_compatible", + "app.interfaces.external_services", + ), + ) + + +@pytest.mark.parametrize( + ("removed", "message"), + [ + ("artifact_s3_access_key_id", "complete static credentials"), + ("artifact_s3_secret_access_key", "complete static credentials"), + ], +) +def test_minio_static_credentials_are_required( + tmp_path: Path, + removed: str, + message: str, +) -> None: + with pytest.raises(ValueError, match=message): + _minio_settings(tmp_path, **{removed: None}) + + +def test_minio_secret_values_are_absent_from_repr_and_validation_errors( + tmp_path: Path, +) -> None: + secret = "minio-secret-value-that-must-not-leak" + settings = _minio_settings(tmp_path, artifact_s3_secret_access_key=secret) + + assert settings.artifact_s3_secret_access_key is not None + assert settings.artifact_s3_secret_access_key.get_secret_value() == secret + assert secret not in repr(settings) + assert secret not in repr(settings.model_dump()) + + with pytest.raises(ValidationError) as caught: + _minio_settings( + tmp_path, + artifact_s3_secret_access_key=secret, + artifact_s3_region="US-east-1", + ) + + assert secret not in repr(caught.value.errors()) + assert secret not in caught.value.json() + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +@pytest.mark.parametrize( + "method_name", + ["model_validate", "model_validate_json", "model_validate_strings"], +) +def test_valid_minio_secret_is_absent_from_unrelated_application_traceback( + tmp_path: Path, + method_name: str, +) -> None: + """Later configuration errors must not retain an accepted credential.""" + secret = "valid-minio-secret-for-traceback" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + artifact_s3_region="US-east-1", + ) + method = getattr(Settings, method_name) + + with pytest.raises(ValueError) as caught: + method(json.dumps(payload) if method_name.endswith("json") else payload) + + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +@pytest.mark.parametrize( + "method_name", + ["constructor", "model_validate", "model_validate_json", "model_validate_strings"], +) +def test_invalid_minio_secret_is_absent_from_application_traceback( + tmp_path: Path, + method_name: str, +) -> None: + """Rejected credentials must not remain in Workstream traceback locals.""" + secret = "invalid-minio-secret\n" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + ) + + with pytest.raises(ValueError, match="invalid artifact storage secret") as caught: + if method_name == "constructor": + Settings(**payload) + elif method_name == "model_validate_json": + Settings.model_validate_json(json.dumps(payload)) + else: + getattr(Settings, method_name)(payload) + + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +@pytest.mark.parametrize( + ("method_name", "delegated_method", "error_type"), + [ + ("constructor", "__init__", RuntimeError), + ("model_validate", "model_validate", RuntimeError), + ("model_validate_json", "model_validate", TypeError), + ("model_validate_strings", "model_validate_strings", TypeError), + ], +) +def test_unexpected_validation_errors_do_not_retain_minio_secrets( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + method_name: str, + delegated_method: str, + error_type: type[Exception], +) -> None: + """Unexpected delegated errors must still clear every credential local.""" + secret = "minio-secret-from-unexpected-validator-error" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + ) + + def fail_validation(*args: object, **kwargs: object) -> None: + del args, kwargs + raise error_type("forced validation failure") + + if delegated_method == "__init__": + monkeypatch.setattr(BaseSettings, delegated_method, fail_validation) + else: + monkeypatch.setattr( + BaseSettings, + delegated_method, + classmethod(fail_validation), + ) + + with pytest.raises(error_type, match="^forced validation failure$") as caught: + if method_name == "constructor": + Settings(**payload) + elif method_name == "model_validate_json": + Settings.model_validate_json(json.dumps(payload)) + else: + getattr(Settings, method_name)(payload) + + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +@pytest.mark.parametrize( + ("field_name", "field_value"), + [ + ("artifact_s3_access_key_id", "nested-json-secret"), + ("artifact_s3_secret_access_key", "nested-json-secret"), + ("artifact_s3_session_token", "nested-json-secret"), + ( + "artifact_s3_endpoint_url", + "http://user:nested-json-secret@localhost:9000", + ), + ], +) +def test_invalid_json_shapes_do_not_retain_minio_secrets( + field_name: str, + field_value: str, +) -> None: + """Rejected non-object JSON must not survive in Workstream error frames.""" + secret = "nested-json-secret" + payload = json.dumps([{field_name: field_value}]) + + with pytest.raises(ValueError, match="^invalid settings JSON$") as caught: + Settings.model_validate_json(payload) + + assert secret not in f"{caught.value!s} {caught.value!r}" + assert_secret_not_retained( + caught.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +def test_partial_s3_secret_extraction_cleans_unexpected_dotenv_errors( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """A later source error must not retain an already-extracted credential.""" + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY", raising=False) + rate_secret = base64.b64encode(bytes(range(32))).decode("ascii") + access_key = "partially-extracted-minio-access-key" + payload = _minio_setting_values( + tmp_path, + api_rate_limit_key_secret=rate_secret, + artifact_s3_access_key_id=access_key, + _env_file=object(), + ) + payload.pop("artifact_s3_secret_access_key") + + with pytest.raises(TypeError) as caught: + Settings(**payload) + + assert_secret_not_retained( + caught.value, + access_key, + traceback_module_prefixes=("app.core.config",), + ) + + +@pytest.mark.parametrize( + "method_name", + ["model_validate", "model_validate_json", "model_validate_strings"], +) +def test_alternate_validation_loads_minio_secrets_without_retaining_them( + tmp_path: Path, + method_name: str, +) -> None: + secret = "alternate-minio-secret-value" + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_secret_access_key=secret, + ) + method = getattr(Settings, method_name) + + settings = method(json.dumps(payload) if method_name.endswith("json") else payload) + + assert settings.artifact_s3_secret_access_key is not None + assert settings.artifact_s3_secret_access_key.get_secret_value() == secret + assert secret not in repr(settings) + assert secret not in repr(settings.model_dump()) + + +def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + env_secret = "env-minio-secret-value" + dotenv_secret = "dotenv-minio-secret-value" + env_file = tmp_path / ".env" + env_file.write_text( + "\n".join( + [ + "WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID=dotenv-access", + f"WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY={dotenv_secret}", + ] + ), + encoding="utf-8", + ) + monkeypatch.setenv("WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID", "env-access") + monkeypatch.setenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY", env_secret) + + with pytest.raises(ValidationError) as env_error: + Settings( + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="minio", + artifact_s3_region="US-east-1", + artifact_s3_endpoint_url="http://127.0.0.1:9000", + artifact_s3_bucket="workstream-artifacts-test", + artifact_s3_credential_mode="local_static", + _env_file=env_file, + ) + assert env_secret not in repr(env_error.value.errors()) + assert_secret_not_retained(env_error.value, env_secret) + + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID") + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY") + with pytest.raises(ValidationError) as dotenv_error: + Settings( + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + artifact_s3_provider_profile="minio", + artifact_s3_region="US-east-1", + artifact_s3_endpoint_url="http://127.0.0.1:9000", + artifact_s3_bucket="workstream-artifacts-test", + artifact_s3_credential_mode="local_static", + _env_file=env_file, + ) + assert dotenv_secret not in repr(dotenv_error.value.errors()) + assert_secret_not_retained(dotenv_error.value, dotenv_secret) + + +def test_minio_endpoint_is_normalized_before_namespace_identity(tmp_path: Path) -> None: + """Equivalent MinIO origins must produce one endpoint and namespace identity.""" + common = { + "environment": "test", + "artifact_store_backend": "s3_compatible", + "artifact_scratch_root": tmp_path / "scratch", + "artifact_s3_provider_profile": "minio", + "artifact_s3_region": "us-east-1", + "artifact_s3_bucket": "workstream-artifacts-test", + "artifact_s3_credential_mode": "local_static", + "artifact_s3_access_key_id": "local-access", + "artifact_s3_secret_access_key": "local-secret", + } + canonical = Settings(**common, artifact_s3_endpoint_url="http://localhost") + equivalent = Settings(**common, artifact_s3_endpoint_url="HTTP://LOCALHOST:80/") + + assert canonical.artifact_s3_endpoint_url == "http://localhost" + assert equivalent.artifact_s3_endpoint_url == "http://localhost" + assert ( + create_artifact_store_bootstrap(canonical).namespace_identity + == create_artifact_store_bootstrap(equivalent).namespace_identity + ) + + +@pytest.mark.parametrize( + ("overrides", "message"), + [ + ({"artifact_s3_endpoint_url": "https://s3.us-east-1.amazonaws.com"}, "endpoint"), + ({"artifact_s3_credential_mode": "local_static"}, "workload identity"), + ({"artifact_s3_aws_workload_identity_method": None}, "one workload identity"), + ( + { + "artifact_s3_access_key_id": "static-access", + "artifact_s3_secret_access_key": "static-secret", + }, + "rejects static credentials", + ), + ], +) +def test_aws_s3_settings_fail_closed( + tmp_path: Path, + overrides: dict[str, object], + message: str, +) -> None: + with pytest.raises((ValidationError, ValueError), match=message): + _aws_settings(tmp_path, **overrides) + + +def test_aws_factory_fails_with_live_proof_before_constructing_provider( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + from app.adapters import artifacts as artifacts_module + from app.adapters.artifacts import s3_compatible + + settings = _aws_settings(tmp_path) + events: list[str] = [] + + class FactoryMustNotConstruct: + def __init__(self, *_args: object, **_kwargs: object) -> None: + events.append("factory") + raise AssertionError("factory constructed before AWS live proof failure") + + def session_must_not_construct(*_args: object, **_kwargs: object) -> object: + events.append("resolver") + raise AssertionError("credential resolver constructed before live proof failure") + + def credentials_must_not_be_probed(*_args: object, **_kwargs: object) -> None: + events.append("credential-probe") + raise AssertionError("credential sources probed before live proof failure") + + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setenv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "/v2/credentials/runtime") + monkeypatch.setattr(artifacts_module, "ExternalServiceAdapterFactory", FactoryMustNotConstruct) + monkeypatch.setattr(s3_compatible, "AioSession", session_must_not_construct) + monkeypatch.setattr( + s3_compatible, + "validate_aws_workload_identity_environment", + credentials_must_not_be_probed, + ) + + with pytest.raises(ArtifactProviderLiveProofRequiredError) as caught: + create_artifact_store_bootstrap(settings) + + assert caught.value.code == "artifact_provider_live_proof_required" + assert events == [] + + +async def test_aws_lifespan_fails_before_namespace_claim_and_provider_io( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + import app.main as main_module + + settings = _aws_settings(tmp_path, environment="test") + events: list[str] = [] + + async def namespace_must_not_run(*_args: object, **_kwargs: object) -> object: + events.append("namespace") + raise AssertionError("namespace claim ran before AWS live proof failure") + + async def cleanup_must_not_run(*_args: object, **_kwargs: object) -> object: + events.append("cleanup") + raise AssertionError("provider cleanup ran before AWS live proof failure") + + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setenv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "/v2/credentials/runtime") + monkeypatch.setattr( + main_module, + "_validate_artifact_storage_namespace_at_startup", + namespace_must_not_run, + ) + monkeypatch.setattr(main_module, "cleanup_stale_artifact_scratch", cleanup_must_not_run) + app = create_app(settings) + + with pytest.raises(ArtifactProviderLiveProofRequiredError) as caught: + async with app.router.lifespan_context(app): + pass + + assert caught.value.code == "artifact_provider_live_proof_required" + assert events == [] + + +def test_s3_dependency_manifest_and_installed_versions_are_exact() -> None: + manifest = tomllib.loads((Path(__file__).parents[1] / "pyproject.toml").read_text()) + + assert "aiobotocore==3.7.0" in manifest["project"]["dependencies"] + assert "botocore==1.43.0" in manifest["project"]["dependencies"] + assert importlib.metadata.version("aiobotocore") == "3.7.0" + assert importlib.metadata.version("botocore") == "1.43.0" + + +@pytest.mark.parametrize("value", [None, "", "x" * 513]) +def test_s3_prefix_validation_rejects_noncanonical_outer_bounds(value: object) -> None: + assert is_canonical_s3_prefix(value) is False + + +def test_minio_endpoint_canonicalization_handles_ipv6_and_invalid_ports() -> None: + assert canonical_minio_endpoint("HTTP://[::1]:9000/") == "http://[::1]:9000" + assert ( + canonical_minio_endpoint("http://[0:0:0:0:0:0:0:1]:9000") + == "http://[::1]:9000" + ) + with pytest.raises(ValueError, match="endpoint is invalid"): + canonical_minio_endpoint("http://localhost:not-a-port") + + +@pytest.mark.parametrize( + "region", + ["us-east-1", "us-gov-west-1", "eusc-de-east-1"], +) +def test_s3_region_validation_accepts_canonical_region_shapes(region: str) -> None: + assert is_canonical_s3_region(region) is True + + +@pytest.mark.parametrize( + "endpoint", + [ + "http://user:endpoint-secret@localhost:9000", + "http://localhost:endpoint-secret", + ], +) +def test_invalid_minio_endpoint_secrets_do_not_survive_errors( + tmp_path: Path, + endpoint: str, +) -> None: + secret = "endpoint-secret" + + with pytest.raises(ValueError) as endpoint_error: + canonical_minio_endpoint(endpoint) + assert_secret_not_retained( + endpoint_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + payload = _minio_setting_values( + tmp_path, + artifact_scratch_root=str(tmp_path / "scratch"), + artifact_s3_endpoint_url=endpoint, + ) + for method_name in ( + "constructor", + "model_validate", + "model_validate_json", + "model_validate_strings", + ): + with pytest.raises((ValidationError, ValueError)) as settings_error: + if method_name == "constructor": + Settings(**payload) + elif method_name == "model_validate_json": + Settings.model_validate_json(json.dumps(payload)) + else: + getattr(Settings, method_name)(payload) + assert_secret_not_retained( + settings_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +def test_invalid_minio_endpoint_secrets_from_env_and_dotenv_do_not_survive_errors( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + secret = "endpoint-source-secret" + endpoint = f"http://user:{secret}@localhost:9000" + payload = _minio_setting_values(tmp_path) + payload.pop("artifact_s3_endpoint_url") + monkeypatch.setenv("WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL", endpoint) + + with pytest.raises(ValueError) as env_error: + Settings(**payload) + assert_secret_not_retained( + env_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL") + env_file = tmp_path / ".env" + env_file.write_text( + f"WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL={endpoint}\n", + encoding="utf-8", + ) + with pytest.raises(ValueError) as dotenv_error: + Settings(**payload, _env_file=env_file) + assert_secret_not_retained( + dotenv_error.value, + secret, + traceback_module_prefixes=("app.",), + ) + + +@pytest.mark.parametrize( + ("profile", "changes", "message"), + [ + ("minio-v1", {"addressing_style": "auto"}, "addressing style"), + ("minio-v1", {"region": "US-east-1"}, "region"), + ("minio-v1", {"endpoint_identity": "sha256:bad"}, "endpoint identity"), + ( + "aws-s3-v1", + {"endpoint_identity": "sha256:" + "1" * 64}, + "endpoint identity is forbidden", + ), + ("unsupported-v1", {}, "provider profile"), + ], +) +def test_s3_namespace_value_validation_fails_closed( + profile: str, + changes: dict[str, str], + message: str, +) -> None: + descriptor = { + "addressing_style": "path", + "bucket": "workstream-artifacts", + "endpoint_identity": "sha256:" + "1" * 64, + "private_prefix": "workstream/artifacts", + "region": "us-east-1", + } + if profile == "aws-s3-v1": + descriptor.pop("endpoint_identity") + descriptor.update(changes) - assert caught.value.identity is not None - assert caught.value.identity.provider_key == "s3_compatible" + with pytest.raises(ValueError, match=message): + validate_s3_namespace_descriptor(profile, descriptor) diff --git a/backend/tests/test_local_artifact_store.py b/backend/tests/test_local_artifact_store.py index fa61bf42..72bacc12 100644 --- a/backend/tests/test_local_artifact_store.py +++ b/backend/tests/test_local_artifact_store.py @@ -14,6 +14,10 @@ import pytest from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap +from app.adapters.artifacts.references import ( + artifact_provider_object_ref, + parse_artifact_provider_object_ref, +) from app.interfaces.artifacts import ( ArtifactByteRange, ArtifactConfigurationError, @@ -579,12 +583,12 @@ async def test_head_recovers_crash_after_link_before_temporary_unlink( try: async with minted_source(tmp_path / "scratch", b"crash recovery") as source: commitment = source.commitment - provider_ref = adapter._provider_object_ref(commitment) + provider_ref = artifact_provider_object_ref(commitment) descriptor, temporary_name = adapter._create_temporary() try: adapter._write_all(descriptor, memoryview(b"crash recovery")) adapter._seal_temporary(descriptor) - prefix, filename = adapter._parse_provider_object_ref(provider_ref) + prefix, filename = parse_artifact_provider_object_ref(provider_ref) prefix_fd = adapter._open_prefix(prefix, create=True) try: os.link( diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py new file mode 100644 index 00000000..054f601c --- /dev/null +++ b/backend/tests/test_s3_artifact_store.py @@ -0,0 +1,914 @@ +"""Real MinIO and focused protocol tests for S3CompatibleArtifactStore.""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +import hashlib +import os +from pathlib import Path +from typing import Any + +from aiobotocore.config import AioConfig +from aiobotocore.session import AioSession +from botocore.exceptions import ClientError +import pytest + +from app.adapters.artifacts import create_artifact_store_bootstrap +from app.adapters.artifacts.s3_compatible import ( + S3CompatibleArtifactStore, + S3CompatibleArtifactStoreBootstrap, +) +import app.adapters.artifacts.s3_compatible as s3_module +from app.core.config import Settings +from app.interfaces.artifacts import ( + ArtifactByteRange, + ArtifactConfigurationError, + ArtifactIntegrityError, + ArtifactInputMismatchError, + ArtifactObjectMissingError, + ArtifactOperationConflictError, + ArtifactStore, + ArtifactStoreNamespaceClaim, + ArtifactStoreUnavailableError, + artifact_store_namespace_material, +) +from app.interfaces.external_services import ExternalServiceAdapterIdentity +from app.modules.artifacts.preparation import ArtifactPreparationService +from app.modules.artifacts.sources import ArtifactCommitment +from tests.assertion_helpers import assert_secret_not_retained +from tests.artifact_store_helpers import minted_source +from tests.test_artifact_store_conformance import ArtifactStoreConformanceTests + + +MINIO_ENDPOINT = os.environ.get("WORKSTREAM_TEST_MINIO_ENDPOINT", "http://localhost:9000") +MINIO_REGION = "us-east-1" +MINIO_BUCKET = "workstream-artifacts" +MINIO_ACCESS_KEY = "workstream-minio" +MINIO_SECRET_KEY = "workstream-minio-secret-key" + + +@pytest.fixture(scope="module", autouse=True) +async def provision_minio_bucket() -> None: + """Provision only the dedicated integration bucket with MinIO admin credentials.""" + session = AioSession() + session.set_credentials(MINIO_ACCESS_KEY, MINIO_SECRET_KEY) + async with session.create_client( + "s3", + endpoint_url=MINIO_ENDPOINT, + region_name=MINIO_REGION, + config=AioConfig(s3={"addressing_style": "path"}), + ) as client: + try: + await client.create_bucket(Bucket=MINIO_BUCKET) + except ClientError as error: + if error.response.get("Error", {}).get("Code") not in { + "BucketAlreadyExists", + "BucketAlreadyOwnedByYou", + }: + raise + + +class TestMinioArtifactStoreConformance(ArtifactStoreConformanceTests): + """Run the same ArtifactStore v2 vectors against real MinIO.""" + + expected_identity = ExternalServiceAdapterIdentity( + "artifact_store", + "s3_compatible", + ) + + def make_store(self, root: Path) -> ArtifactStore: + """Construct one namespace-isolated real MinIO adapter.""" + namespace = hashlib.sha256(os.fspath(root.parent).encode()).hexdigest()[:20] + return initialize_minio_store(private_prefix=f"conformance/{namespace}") + + +def minio_settings( + *, + private_prefix: str, + access_key: str = MINIO_ACCESS_KEY, + secret_key: str = MINIO_SECRET_KEY, +) -> Settings: + """Return one complete local-only MinIO configuration.""" + return Settings( + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=Path("/tmp/workstream-test-artifact-scratch"), + artifact_s3_provider_profile="minio", + artifact_s3_region=MINIO_REGION, + artifact_s3_endpoint_url=MINIO_ENDPOINT, + artifact_s3_bucket=MINIO_BUCKET, + artifact_s3_private_prefix=private_prefix, + artifact_s3_addressing_style="path", + artifact_s3_credential_mode="local_static", + artifact_s3_access_key_id=access_key, + artifact_s3_secret_access_key=secret_key, + artifact_stream_buffer_bytes=2, + artifact_s3_operation_total_timeout_seconds=20, + artifact_s3_write_timeout_seconds=20, + ) + + +def initialize_minio_store(*, private_prefix: str) -> S3CompatibleArtifactStore: + """Construct, claim, and initialize one MinIO namespace.""" + bootstrap = create_artifact_store_bootstrap( + minio_settings(private_prefix=private_prefix) + ) + assert type(bootstrap) is S3CompatibleArtifactStoreBootstrap + namespace_identity = bootstrap.namespace_identity + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=bootstrap.identity, + namespace_identity=namespace_identity, + ) + return bootstrap.initialize_after_namespace_claim( + ArtifactStoreNamespaceClaim( + adapter_identity=bootstrap.identity, + namespace_identity=namespace_identity, + namespace_fingerprint=fingerprint, + ) + ) + + +@pytest.mark.asyncio +async def test_wrong_minio_credentials_map_403_to_unavailable() -> None: + """Never misclassify forbidden provider access as authoritative absence.""" + access_key = "unknown-minio-user" + secret_key = "unknown-minio-password" + settings = minio_settings( + private_prefix="negative/forbidden", + access_key=access_key, + secret_key=secret_key, + ) + bootstrap = create_artifact_store_bootstrap(settings) + namespace = bootstrap.namespace_identity + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=bootstrap.identity, + namespace_identity=namespace, + ) + store = bootstrap.initialize_after_namespace_claim( + ArtifactStoreNamespaceClaim(bootstrap.identity, namespace, fingerprint) + ) + try: + with pytest.raises(ArtifactStoreUnavailableError) as caught: + await store.head("sha256/00/" + "0" * 62) + assert str(caught.value) == "S3 artifact head failed" + assert caught.value.code == "artifact_storage_unavailable" + assert caught.value.__cause__ is None + assert_secret_not_retained(caught.value, access_key) + assert_secret_not_retained(caught.value, secret_key) + finally: + bootstrap.close() + + +@pytest.mark.asyncio +async def test_adversarial_first_writer_cannot_be_accepted_as_replay( + tmp_path: Path, +) -> None: + """Reject existing wrong bytes at the server-derived content key.""" + private_prefix = f"adversarial/{hashlib.sha256(os.urandom(16)).hexdigest()[:20]}" + content = b"expected exact bytes" + digest = hashlib.sha256(content).hexdigest() + key = f"{private_prefix}/sha256/{digest[:2]}/{digest[2:]}" + await raw_minio_put(key, b"attacker-selected wrong bytes") + store = initialize_minio_store(private_prefix=private_prefix) + try: + async with minted_source(tmp_path / "scratch", content) as source: + with pytest.raises(ArtifactIntegrityError): + await store.put(source) + finally: + store.close() + + +@pytest.mark.asyncio +async def test_cross_project_prefix_cannot_occupy_another_content_key( + tmp_path: Path, +) -> None: + """Keep identical digest references isolated by claimed project namespace.""" + nonce = hashlib.sha256(os.urandom(16)).hexdigest()[:20] + attacker_prefix = f"projects/attacker-{nonce}" + target_prefix = f"projects/target-{nonce}" + content = b"target project exact bytes" + digest = hashlib.sha256(content).hexdigest() + provider_ref = f"sha256/{digest[:2]}/{digest[2:]}" + await raw_minio_put(f"{attacker_prefix}/{provider_ref}", b"attacker bytes") + + attacker_store = initialize_minio_store(private_prefix=attacker_prefix) + target_store = initialize_minio_store(private_prefix=target_prefix) + try: + async with minted_source(tmp_path / "attacker-scratch", content) as source: + with pytest.raises(ArtifactIntegrityError): + await attacker_store.put(source) + async with minted_source(tmp_path / "target-scratch", content) as source: + result = await target_store.put(source) + assert result.provider_object_ref == provider_ref + assert result.replayed is False + assert b"".join([chunk async for chunk in target_store.open(provider_ref)]) == content + finally: + attacker_store.close() + target_store.close() + + +@pytest.mark.asyncio +async def test_unsealed_client_selected_commitment_fails_before_provider_io( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Accept only the preparation service's inseparable source/commitment pair.""" + store = initialize_minio_store(private_prefix="negative/unsealed") + + @asynccontextmanager + async def unexpected_client() -> Any: + raise AssertionError("provider I/O occurred for an unsealed source") + yield + + monkeypatch.setattr(store, "_client", unexpected_client) + try: + with pytest.raises(ArtifactInputMismatchError, match="not sealed"): + await store.put(object()) # type: ignore[arg-type] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_hard_size_limit_fails_before_provider_io( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Reject an over-limit commitment before opening an S3 client.""" + store = initialize_minio_store(private_prefix="negative/oversized") + + @asynccontextmanager + async def unexpected_client() -> Any: + raise AssertionError("provider I/O occurred for an oversized source") + yield + + monkeypatch.setattr(store, "_client", unexpected_client) + monkeypatch.setattr(s3_module, "HARD_MAXIMUM_ARTIFACT_BYTES", 1) + try: + async with minted_source(tmp_path / "scratch", b"12") as source: + with pytest.raises(s3_module.ArtifactLimitExceededError): + await store.put(source) + finally: + store.close() + + +def test_namespace_claim_mismatch_fails_before_provider_io() -> None: + """Reject a configured namespace mismatch without routing elsewhere.""" + bootstrap = create_artifact_store_bootstrap( + minio_settings(private_prefix="namespace/exact") + ) + namespace = bootstrap.namespace_identity + wrong = ArtifactStoreNamespaceClaim( + adapter_identity=bootstrap.identity, + namespace_identity=namespace, + namespace_fingerprint="sha256:" + "0" * 64, + ) + with pytest.raises(ArtifactConfigurationError, match="does not match"): + bootstrap.initialize_after_namespace_claim(wrong) + + +async def test_committed_body_keeps_large_source_chunks_out_of_pending_buffer() -> None: + """Retain unconsumed source bytes by view while bounding copied request bytes.""" + content = b"0123456789abcdef" + commitment = ArtifactCommitment( + sha256=f"sha256:{hashlib.sha256(content).hexdigest()}", + byte_count=len(content), + media_type="application/octet-stream", + ) + + class Source: + def __init__(self) -> None: + self.commitment = commitment + + def stream(self) -> AsyncIterator[bytes]: + async def iterate() -> AsyncIterator[bytes]: + yield content + + return iterate() + + body = s3_module._CommittedSourceBody(Source(), 4) # type: ignore[arg-type] # noqa: SLF001 + + assert await body.read(4) == b"0123" + assert len(body._pending) == 0 # noqa: SLF001 + assert body._source_cursor is not None # noqa: SLF001 + assert len(body._source_cursor) == 12 # noqa: SLF001 + assert await body.read(4) == b"4567" + assert await body.read(4) == b"89ab" + assert await body.read(4) == b"cdef" + assert body.complete is True + + +async def raw_minio_put(key: str, content: bytes) -> None: + """Publish adversarial integration bytes outside the Workstream adapter.""" + session = AioSession() + session.set_credentials(MINIO_ACCESS_KEY, MINIO_SECRET_KEY) + async with session.create_client( + "s3", + endpoint_url=MINIO_ENDPOINT, + region_name=MINIO_REGION, + config=AioConfig(s3={"addressing_style": "path"}), + ) as client: + await client.put_object(Bucket=MINIO_BUCKET, Key=key, Body=content) + + +def test_s3_constructor_and_bootstrap_reject_invalid_runtime_values() -> None: + """Fail closed on malformed direct construction outside validated settings.""" + session = AioSession() + values = { + "provider_profile": "minio", + "region": MINIO_REGION, + "endpoint_url": MINIO_ENDPOINT, + "bucket": MINIO_BUCKET, + "private_prefix": "negative/constructor", + "addressing_style": "path", + "session": session, + "buffer_bytes": 2, + "connect_timeout_seconds": 1, + "read_timeout_seconds": 1, + "write_timeout_seconds": 1, + "pool_timeout_seconds": 1, + "operation_total_timeout_seconds": 2, + "max_pool_connections": 1, + } + for override in ( + {"provider_profile": "unknown"}, + {"region": "US-east-1"}, + {"bucket": "Bad..Bucket"}, + {"private_prefix": "../bad//prefix"}, + {"addressing_style": "automatic"}, + {"endpoint_url": "http://user:secret@localhost:9000"}, + {"session": object()}, + {"buffer_bytes": 0}, + {"buffer_bytes": True}, + {"max_pool_connections": 0}, + {"max_pool_connections": True}, + {"connect_timeout_seconds": 0}, + {"read_timeout_seconds": True}, + ): + with pytest.raises(ArtifactConfigurationError): + S3CompatibleArtifactStore(**(values | override)) # type: ignore[arg-type] + with pytest.raises(ValueError): + S3CompatibleArtifactStoreBootstrap(object()) # type: ignore[arg-type] + + +async def test_s3_operation_client_ignores_ambient_proxy_environment( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Keep signed object requests and bytes off ambient process proxies.""" + for name in ( + "HTTP_PROXY", + "HTTPS_PROXY", + "ALL_PROXY", + "http_proxy", + "https_proxy", + "all_proxy", + ): + monkeypatch.setenv(name, "http://127.0.0.1:9") + monkeypatch.delenv("NO_PROXY", raising=False) + monkeypatch.delenv("no_proxy", raising=False) + namespace = hashlib.sha256(os.fspath(tmp_path).encode()).hexdigest()[:16] + store = initialize_minio_store( + private_prefix=f"negative/proxy-isolation/{namespace}" + ) + try: + async with store._client() as client: + proxy = client._endpoint.http_session._proxy_config.proxy_url_for( + MINIO_ENDPOINT + ) + assert proxy is None + payload = b"real object I/O bypasses poisoned ambient proxies" + async with minted_source(tmp_path / "scratch", payload) as source: + result = await store.put(source) + observed = await store.head(result.provider_object_ref) + received = b"".join( + [chunk async for chunk in store.open(result.provider_object_ref)] + ) + assert observed.exists is True + assert observed.byte_count == len(payload) + assert received == payload + finally: + store.close() + + +def test_namespace_claim_is_single_use_and_close_disables_operations() -> None: + """One exact claim activates one adapter once and close revokes local use.""" + bootstrap = create_artifact_store_bootstrap( + minio_settings(private_prefix="negative/single-use") + ) + namespace = bootstrap.namespace_identity + _, fingerprint = artifact_store_namespace_material( + backend="s3_compatible", + adapter_identity=bootstrap.identity, + namespace_identity=namespace, + ) + claim = ArtifactStoreNamespaceClaim(bootstrap.identity, namespace, fingerprint) + store = bootstrap.initialize_after_namespace_claim(claim) + with pytest.raises(ArtifactConfigurationError, match="already initialized"): + bootstrap.initialize_after_namespace_claim(claim) + with pytest.raises(ArtifactConfigurationError, match="not initialized"): + store.open("sha256/00/" + "0" * 62) + + +@pytest.mark.asyncio +async def test_invalid_provider_reference_and_range_fail_before_provider_io() -> None: + """Reject caller-selected keys and malformed ranges at the adapter boundary.""" + store = initialize_minio_store(private_prefix="negative/references") + try: + with pytest.raises(ArtifactOperationConflictError): + await store.head("customer/project/file") + with pytest.raises(ArtifactOperationConflictError): + store.open("sha256/00/" + "0" * 62, object()) # type: ignore[arg-type] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_head_maps_only_404_to_missing_and_rejects_invalid_size( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Keep denied access and malformed provider metadata distinct from absence.""" + store = initialize_minio_store(private_prefix="negative/head") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + response: object + + async def head_object(self, **_kwargs: object) -> object: + if isinstance(self.response, BaseException): + raise self.response + return self.response + + client = Client() + + @asynccontextmanager + async def fake_client() -> Any: + yield client + + monkeypatch.setattr(store, "_client", fake_client) + try: + client.response = _client_error("404", 404) + assert (await store.head(provider_ref)).exists is False + client.response = _client_error("AccessDenied", 403) + with pytest.raises(ArtifactStoreUnavailableError): + await store.head(provider_ref) + client.response = _client_error("NoSuchKey", 403) + with pytest.raises(ArtifactStoreUnavailableError): + await store.head(provider_ref) + client.response = {} + with pytest.raises(ArtifactIntegrityError, match="exact size"): + await store.head(provider_ref) + finally: + store.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("chunks", "expected_error"), + [ + ([b"a", b""], "truncated"), + ([b"abc"], "exceeded"), + (["not-bytes"], "invalid bytes"), + ], +) +async def test_open_rejects_invalid_provider_streams( + monkeypatch: pytest.MonkeyPatch, + chunks: list[object], + expected_error: str, +) -> None: + """Reject truncation, overread, and non-byte provider responses.""" + store = initialize_minio_store(private_prefix="negative/open") + provider_ref = "sha256/00/" + "0" * 62 + + class Body: + def __init__(self) -> None: + self._chunks = iter(chunks) + + async def __aenter__(self) -> Body: + return self + + async def __aexit__(self, *_args: object) -> None: + return None + + async def read(self, _amount: int) -> object: + return next(self._chunks, b"") + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + return {"ContentLength": 2} + + async def get_object(self, **_kwargs: object) -> dict[str, object]: + return {"Body": Body()} + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactIntegrityError, match=expected_error): + _ = [chunk async for chunk in store.open(provider_ref)] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_open_maps_provider_404_to_missing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Map an object disappearing between HEAD and GET to stable missing.""" + store = initialize_minio_store(private_prefix="negative/disappeared") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + return {"ContentLength": 1} + + async def get_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", 404) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactObjectMissingError): + _ = [chunk async for chunk in store.open(provider_ref, ArtifactByteRange())] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_open_403_precedes_missing_like_provider_code( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Never turn denied reads into authoritative object absence.""" + store = initialize_minio_store(private_prefix="negative/denied-open") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + return {"ContentLength": 1} + + async def get_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", 403) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactStoreUnavailableError): + _ = [chunk async for chunk in store.open(provider_ref)] + finally: + store.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("operation", ["head", "open"]) +@pytest.mark.parametrize("status", [500, None]) +async def test_missing_like_code_requires_authoritative_404_status( + monkeypatch: pytest.MonkeyPatch, + operation: str, + status: int | None, +) -> None: + """Treat missing-like codes without HTTP 404 as provider failures.""" + store = initialize_minio_store(private_prefix=f"negative/ambiguous-{operation}") + provider_ref = "sha256/00/" + "0" * 62 + + class Client: + async def head_object(self, **_kwargs: object) -> dict[str, int]: + if operation == "head": + raise _client_error("NoSuchKey", status) + return {"ContentLength": 1} + + async def get_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", status) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + with pytest.raises(ArtifactStoreUnavailableError): + if operation == "head": + await store.head(provider_ref) + else: + _ = [chunk async for chunk in store.open(provider_ref)] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_precondition_failure_requires_exact_replay_verification( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Never accept a conditional-write conflict without a complete read proof.""" + store = initialize_minio_store(private_prefix="negative/precondition") + verified: list[str] = [] + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("PreconditionFailed", 412) + + async def head_object(self, **_kwargs: object) -> object: + return {"ContentLength": 6} + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def verify(provider_ref: str, _commitment: object) -> None: + verified.append(provider_ref) + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_verify_exact", verify) + try: + async with minted_source(tmp_path / "scratch", b"replay") as source: + result = await store.put(source) + assert result.replayed is True + assert verified == [result.provider_object_ref] + finally: + store.close() + + +@pytest.mark.asyncio +async def test_precondition_failure_with_missing_replay_is_retryable( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Treat a not-yet-visible conditional-write winner as replay-required.""" + store = initialize_minio_store(private_prefix="negative/precondition-missing") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("PreconditionFailed", 412) + + async def head_object(self, **_kwargs: object) -> object: + raise _client_error("NoSuchKey", 404) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + async with minted_source(tmp_path / "scratch", b"unverified-replay") as source: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + finally: + store.close() + + +@pytest.mark.asyncio +async def test_success_without_full_source_consumption_is_not_replay( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Reject a success response when the SDK did not validate all source bytes.""" + store = initialize_minio_store(private_prefix="negative/unconsumed-success") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + return {} + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def must_not_classify_as_replay(*_args: object) -> bool: + raise AssertionError("unconsumed successful writes are not replay candidates") + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_matches_commitment", must_not_classify_as_replay) + try: + async with minted_source(tmp_path / "scratch", b"unconsumed") as source: + with pytest.raises( + ArtifactInputMismatchError, + match="did not consume the sealed artifact source", + ): + await store.put(source) + finally: + store.close() + + +@pytest.mark.asyncio +async def test_nonprecondition_put_error_is_unavailable( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Keep denied writes out of replay and integrity classifications.""" + store = initialize_minio_store(private_prefix="negative/denied-put") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("AccessDenied", 403) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + monkeypatch.setattr(store, "_client", fake_client) + try: + async with minted_source(tmp_path / "scratch", b"denied") as source: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + finally: + store.close() + + +@pytest.mark.asyncio +async def test_put_403_precedes_precondition_like_provider_code( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Never classify denied writes as replay candidates.""" + store = initialize_minio_store(private_prefix="negative/denied-precondition") + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise _client_error("PreconditionFailed", 403) + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def verification_must_not_run(*_args: object) -> None: + raise AssertionError("denied writes cannot enter replay verification") + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_verify_exact", verification_must_not_run) + try: + async with minted_source(tmp_path / "scratch", b"denied") as source: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + finally: + store.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("committed", [False, True]) +async def test_uncertain_transport_put_requires_independent_observation( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + committed: bool, +) -> None: + """Resolve acknowledgement loss only from exact deterministic-key bytes.""" + store = initialize_minio_store(private_prefix=f"negative/uncertain/{committed}") + observed: list[str] = [] + + class Client: + async def put_object(self, **_kwargs: object) -> object: + raise OSError("connection closed") + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def matches(provider_ref: str, _commitment: object) -> bool: + observed.append(provider_ref) + return committed + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_matches_commitment", matches) + try: + async with minted_source(tmp_path / "scratch", b"uncertain") as source: + if committed: + result = await store.put(source) + assert result.replayed is True + assert observed == [result.provider_object_ref] + else: + with pytest.raises(ArtifactStoreUnavailableError): + await store.put(source) + assert len(observed) == 1 + finally: + store.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("stored_bytes_match", [False, True]) +async def test_uncertain_transport_put_uses_real_minio_verification( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + stored_bytes_match: bool, +) -> None: + """Classify acknowledgement loss only from the complete provider bytes.""" + nonce = hashlib.sha256(os.urandom(16)).hexdigest()[:20] + private_prefix = f"uncertain/real-observation/{nonce}" + expected = b"exact committed bytes" + digest = hashlib.sha256(expected).hexdigest() + provider_ref = f"sha256/{digest[:2]}/{digest[2:]}" + stored = expected if stored_bytes_match else b"wrong existing bytes" + await raw_minio_put(f"{private_prefix}/{provider_ref}", stored) + + store = initialize_minio_store(private_prefix=private_prefix) + original_client = store._client + client_calls = 0 + + class LostAcknowledgementClient: + async def put_object(self, **_kwargs: object) -> object: + raise OSError("connection closed after provider write") + + @asynccontextmanager + async def lost_acknowledgement_then_real_provider() -> Any: + nonlocal client_calls + client_calls += 1 + if client_calls == 1: + yield LostAcknowledgementClient() + return + async with original_client() as client: + yield client + + monkeypatch.setattr(store, "_client", lost_acknowledgement_then_real_provider) + try: + async with minted_source(tmp_path / "scratch", expected) as source: + if stored_bytes_match: + result = await store.put(source) + assert result.provider_object_ref == provider_ref + assert result.replayed is True + else: + with pytest.raises(ArtifactIntegrityError): + await store.put(source) + assert client_calls >= 3 + finally: + store.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("failure_mode", ["precondition", "transport"]) +async def test_replay_paths_cannot_accept_malformed_source( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + failure_mode: str, +) -> None: + """Require source proof even when an exact provider object is visible.""" + store = initialize_minio_store(private_prefix="negative/malformed-replay") + provider_checked = False + + class Client: + async def put_object(self, **kwargs: object) -> object: + if failure_mode == "precondition": + raise _client_error("PreconditionFailed", 412) + body: Any = kwargs["Body"] + assert await body.read(4) == b"go" + raise OSError("connection closed after partial source consumption") + + @asynccontextmanager + async def fake_client() -> Any: + yield Client() + + async def malformed_stream( + _self: ArtifactPreparationService, + _binding: object, + _commitment: ArtifactCommitment, + ) -> AsyncIterator[bytes]: + yield b"goodX" + + async def exact_provider_match(*_args: object) -> bool: + nonlocal provider_checked + provider_checked = True + return True + + monkeypatch.setattr(store, "_client", fake_client) + monkeypatch.setattr(store, "_matches_commitment", exact_provider_match) + try: + async with minted_source(tmp_path / "scratch", b"good") as source: + monkeypatch.setattr( + ArtifactPreparationService, + "open_committed_stream", + malformed_stream, + ) + with pytest.raises(ArtifactInputMismatchError): + await store.put(source) + assert provider_checked is False + finally: + store.close() + + +@pytest.mark.asyncio +async def test_put_observation_requires_canonical_commitment() -> None: + """Reject caller-assembled observation requests before provider I/O.""" + store = initialize_minio_store(private_prefix="negative/observation") + try: + with pytest.raises(ArtifactOperationConflictError): + await store.observe_put_result(object()) # type: ignore[arg-type] + finally: + store.close() + + +def _client_error(code: str, status: int | None) -> ClientError: + """Build one sanitized provider error for adapter mapping tests.""" + response: dict[str, object] = { + "Error": {"Code": code, "Message": "provider detail"}, + } + if status is not None: + response["ResponseMetadata"] = {"HTTPStatusCode": status} + return ClientError( + response, + "HeadObject", + ) diff --git a/docker-compose.yml b/docker-compose.yml index 8496f1e3..08d45ad1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -24,6 +24,22 @@ services: interval: 5s timeout: 5s retries: 10 + minio: + image: quay.io/minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e + command: server /data --address :9000 + environment: + MINIO_ROOT_USER: workstream-minio + MINIO_ROOT_PASSWORD: workstream-minio-secret-key + ports: + - "127.0.0.1:9000:9000" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 5s + timeout: 5s + retries: 20 + volumes: + - workstream_minio_data:/data volumes: workstream_postgres_data: + workstream_minio_data: diff --git a/docs/architecture_lockdown.md b/docs/architecture_lockdown.md index 2fe0be76..d26ee583 100644 --- a/docs/architecture_lockdown.md +++ b/docs/architecture_lockdown.md @@ -182,6 +182,11 @@ S3CompatibleArtifactStore AWS S3 in v0.1 production MinIO local and CI S3-compatible integration proof ``` +Here, `local and CI` is the non-production eligibility boundary. The +repository-managed MinIO service is published on host loopback; a private +operator-controlled container-network endpoint is also valid in local, +development, or test environments. MinIO is never hosted-production evidence. + AWS S3 is the only v0.1 production provider. Cloudflare R2 and Flow Node are deferred adapter initiatives. No provider owns product identity, authorization, lifecycle, bindings, audit, or integrity truth. PostgreSQL owns those facts; diff --git a/docs/decision_0013_immutable_artifact_storage_boundary.md b/docs/decision_0013_immutable_artifact_storage_boundary.md index 3a8f1ae6..f62dd068 100644 --- a/docs/decision_0013_immutable_artifact_storage_boundary.md +++ b/docs/decision_0013_immutable_artifact_storage_boundary.md @@ -28,6 +28,12 @@ S3CompatibleArtifactStore integration and production MinIO local and CI integration provider ``` +The `local/CI` MinIO label is a non-production eligibility class, not a literal +endpoint-host restriction. Repository-owned Compose and CI instances publish +on host loopback. An operator-controlled private container-network endpoint may +be used only in local, development, or test environments and does not count as +production activation evidence. + AWS S3 is the only v0.1 production provider. Each replica persists immutable provider profile and storage-namespace identity. Switching a populated deployment requires a separately approved complete-copy, verification, and diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 7d9bd422..13f9d3c0 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -204,6 +204,19 @@ MinIO, and AWS S3, including region, bucket or private-root identity, normalized endpoint identity where applicable, private prefix, and addressing style; it contains no credential or resolved secret. +Every descriptor also carries the reserved `adapter`, `backend`, and +`provider_profile` keys. S3-compatible profiles then require exactly these +additional keys; missing, unknown, or extra keys fail closed: + +| Provider profile | Exact additional descriptor keys | +|---|---| +| `minio-v1` | `addressing_style`, `bucket`, `endpoint_identity`, `private_prefix`, `region` | +| `aws-s3-v1` | `addressing_style`, `bucket`, `private_prefix`, `region` | + +`endpoint_identity` is the non-secret SHA-256 identity of the normalized MinIO +endpoint. It is required for `minio-v1` and forbidden for native `aws-s3-v1`, +which has no configured endpoint. + For LocalStorage, the durable root must already exist with owner-private permissions before application startup. Its descriptor stores only a hash over the normalized path plus filesystem device/inode identity. Replacing or @@ -426,6 +439,11 @@ its only v0.1 production provider. MinIO proves the same protocol contract in local integration tests and CI. Cloudflare R2 is deferred and has no active runtime profile or credential contract. +The v0.1 AWS profile supports global general-purpose buckets only. Workstream +does not emit the account-regional namespace request header, so account-regional +bucket names ending in `-an` fail the canonical bucket gate rather than being +accepted with incomplete request semantics. + Configuration includes: - provider profile `aws_s3` or `minio`; @@ -434,8 +452,8 @@ Configuration includes: - dedicated private Workstream artifact bucket and private prefix; the bucket contains no other application's objects; - addressing style; -- credential mode, optional access-key ID/secret/session token, and no resolved - credential persistence; +- credential mode, local/CI MinIO static credentials held only as redacted + private settings, and no resolved credential persistence; - connect/read/write/pool timeouts; - total verification deadline and maximum buffered bytes. @@ -451,14 +469,43 @@ pinned botocore credential resolver to that provider before any provider is loaded, and verifies the resolved method. Explicit credentials, environment access keys, shared credential/config files, credential processes, login/SSO, legacy EC2/Boto sources, and every unselected workload provider are rejected. -Tests poison every nonselected source and prove that it is never read, executed, -or contacted. This is an allowlisted workload-identity contract, not authority -to use the ambient SDK default chain. +Web-identity STS, ECS container metadata, and EC2 instance metadata use +explicit no-proxy sessions. Web identity also rejects ambient endpoint and CA +overrides. Tests poison every nonselected source and prove that it is never +read, executed, or contacted. This is an allowlisted workload-identity +contract, not authority to use the ambient SDK default chain. Chunk 02B1 pins the async S3 SDK pair to `aiobotocore==3.7.0` and `botocore==1.43.0`. An SDK upgrade is a reviewed contract change because credential-provider precedence and refresh windows are operational behavior. +### Runtime Eligibility After Chunk 02B1 + +`minio` is the only active `s3_compatible` profile in this chunk. It requires +an explicit HTTP(S) endpoint, region, dedicated bucket, private prefix, +`local_static` credentials, and a local/development/test environment. Startup +still claims the exact `minio-v1` namespace in PostgreSQL before any object +operation. Local and CI run the real digest-pinned MinIO server and provision +only the dedicated test bucket; tests do not emulate S3 calls. + +`local/CI MinIO` names a non-production provider-eligibility class, not a +literal endpoint-host rule. Repository-managed Compose and CI publish MinIO on +host loopback only. A separately containerized Workstream process may use an +operator-controlled private MinIO endpoint in a local, development, or test +environment. Publicly exposed MinIO and every production-environment MinIO +configuration are outside v0.1; neither can satisfy native AWS live proof. + +`aws_s3` accepts only a native endpoint-less AWS configuration, explicit +region, and exactly one allowlisted workload-identity method. The composition +root validates settings and then raises `artifact_provider_live_proof_required` +before validating credential environment, constructing the adapter factory, +building or loading the credential resolver, claiming a namespace, or +performing provider I/O. The credential-environment validator and isolated +resolver are inactive deployment-proof support in this chunk; startup does not +invoke them. Chunk 07 alone may make native AWS runtime-eligible by adding the +immutable live activation proof defined below. MinIO conformance is not AWS +activation evidence. + Cloudflare R2 has no v0.1 runtime mode, credential issuer, sidecar, secret contract, configuration profile, or deployment proof. Any future provider initiative must perform current discovery and pass the same ArtifactStore v2 @@ -761,6 +808,44 @@ The preparation settings use the standard `WORKSTREAM_` environment prefix: | `WORKSTREAM_ARTIFACT_STREAM_BUFFER_BYTES` | `1048576` | Bounded streaming buffer, limited to at most 1 MiB. | | `WORKSTREAM_ARTIFACT_OPERATION_LOCK_TIMEOUT_SECONDS` | `1800` | Maximum wait for a private cross-process artifact-store operation lock before failing closed. | +The S3-compatible provider settings use the same prefix: + +| Environment variable | Default | Contract | +|---|---:|---| +| `WORKSTREAM_ARTIFACT_STORE_BACKEND` | `disabled` | Set to `s3_compatible` to select the S3 protocol adapter. | +| `WORKSTREAM_ARTIFACT_S3_PROVIDER_PROFILE` | unset | Closed value: `minio` for local/CI or `aws_s3` for the inactive native AWS profile. | +| `WORKSTREAM_ARTIFACT_S3_REGION` | unset | Required canonical S3 region. | +| `WORKSTREAM_ARTIFACT_S3_ENDPOINT_URL` | unset | Required HTTP(S) endpoint for MinIO; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_BUCKET` | unset | Required private bucket name. | +| `WORKSTREAM_ARTIFACT_S3_PRIVATE_PREFIX` | `workstream/artifacts` | Canonical private key prefix; object references remain server-derived SHA-256 paths. | +| `WORKSTREAM_ARTIFACT_S3_ADDRESSING_STYLE` | `virtual` | Closed value: `path` or `virtual`; local compose uses `path`. | +| `WORKSTREAM_ARTIFACT_S3_CREDENTIAL_MODE` | unset | MinIO requires `local_static`; native AWS requires `aws_workload_identity`. | +| `WORKSTREAM_ARTIFACT_S3_AWS_WORKLOAD_IDENTITY_METHOD` | unset | Native AWS selection: `assume-role-with-web-identity`, `container-role`, or `iam-role`; never used by MinIO. | +| `WORKSTREAM_ARTIFACT_S3_ACCESS_KEY_ID` | unset | MinIO local/CI secret only; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY` | unset | MinIO local/CI secret only; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_SESSION_TOKEN` | unset | Optional MinIO local/CI secret only; forbidden for native AWS. | +| `WORKSTREAM_ARTIFACT_S3_CONNECT_TIMEOUT_SECONDS` | `5` | Provider connection timeout, at most 60 seconds. | +| `WORKSTREAM_ARTIFACT_S3_READ_TIMEOUT_SECONDS` | `60` | Provider read timeout, at most 1800 seconds. | +| `WORKSTREAM_ARTIFACT_S3_WRITE_TIMEOUT_SECONDS` | `1800` | Provider write timeout, at most 3600 seconds. | +| `WORKSTREAM_ARTIFACT_S3_POOL_TIMEOUT_SECONDS` | `5` | Client-pool acquisition timeout, at most 60 seconds. | +| `WORKSTREAM_ARTIFACT_S3_OPERATION_TOTAL_TIMEOUT_SECONDS` | `1800` | Total provider-operation timeout, at most 3600 seconds. | +| `WORKSTREAM_ARTIFACT_S3_MAX_POOL_CONNECTIONS` | `16` | Bounded client connection pool, from 1 through 256. | +| `WORKSTREAM_ARTIFACT_MAXIMUM_BYTES` | `536870912` | Hard object maximum; configuration cannot exceed 512 MiB. | + +MinIO is the only runtime-eligible S3 profile in this chunk and is restricted +to local, development, and test environments. Native AWS configuration rejects +static credentials, configured endpoints, unselected or ambient credential +sources, arbitrary container credential URLs, and custom instance-metadata +endpoints. The selected method accepts only its exact closed `AWS_*` +environment allowlist; every other `AWS_*` or `BOTOCORE_*` SDK behavior control +fails before SDK session construction. S3 object operations and credential +metadata operations explicitly ignore ambient HTTP proxy variables. IAM-role +validation requires IMDSv2 and the isolated fetcher disables IMDSv1. Even valid +native AWS settings fail with `artifact_provider_live_proof_required` at API, +Celery scheduler/task, and direct-cleanup entry points before credential-source +probing, factory construction, scratch cleanup, namespace claim, resolver +loading, or provider I/O. + Every process sharing a scratch root must use the identical complete setting set. The root marker binds a canonical fingerprint of those limits and startup fails closed on mismatch; changing limits therefore requires an empty, @@ -1054,7 +1139,8 @@ bytes in PostgreSQL. ## Verification Strategy - one conformance suite runs against LocalStorage and real MinIO; -- a secret-free live smoke profile proves AWS S3 configuration; +- inactive validation proves the closed AWS S3 configuration contract, while + Chunk 07 owns live deployment proof; - integration tests use real S3-compatible API calls, not monkeypatching; - concurrent puts cannot overwrite existing bytes; - acknowledgement loss, exact replay, oversized-object refusal, truncation, diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index eb5b8f63..b6ab772e 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -62,7 +62,12 @@ "coverage report --include='app/workers/*' --precision=2 --fail-under=90", "coverage report --include='app/main.py' --precision=2 --fail-under=90", ), - "02B1": (), + "02B1": ( + "coverage report --include='app/adapters/artifacts/s3_compatible.py' " + "--precision=2 --fail-under=90", + "coverage report --include='app/core/s3_validation.py' " + "--precision=2 --fail-under=90", + ), "02C1": ( "coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90", ), @@ -112,6 +117,29 @@ "python scripts/api_contract_e2e.py", ) ) +MINIO_IMAGE = ( + "quay.io/minio/minio:latest@" + "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e" +) +BACKEND_MINIO_START_COMMAND = "\n".join( + ( + f"image='{MINIO_IMAGE}'", + "docker run --detach --rm --name workstream-minio \\", + " --publish 127.0.0.1:9000:9000 \\", + " --env MINIO_ROOT_USER=workstream-minio \\", + " --env MINIO_ROOT_PASSWORD=workstream-minio-secret-key \\", + ' "$image" server /data --address :9000', + "for attempt in $(seq 1 60); do", + " if curl --fail --silent " + "http://127.0.0.1:9000/minio/health/live >/dev/null; then", + " exit 0", + " fi", + " sleep 1", + "done", + "docker logs workstream-minio", + "exit 1", + ) +) AUTH_09B_COVERAGE_COMMANDS = ( "coverage report --include='app/modules/actors/*' --precision=2 --fail-under=90", "coverage report --include='app/modules/authorization/*' " @@ -4125,7 +4153,7 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: ).read_text(encoding="utf-8") artifact_contract = Path( ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/" - "WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md" + "WS-ART-001-02B1-s3-compatible-minio-aws.md" ).read_text(encoding="utf-8") work_queue = Path(".agent-loop/WORK_QUEUE.md").read_text(encoding="utf-8") loop_state = Path(".agent-loop/LOOP_STATE.md").read_text(encoding="utf-8") @@ -4142,62 +4170,31 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: assert "| `WS-AUTH-001-09D-A` | Active |" in auth_status assert "| `WS-AUTH-001-09D-B` | Inactive |" in auth_status assert "Merged through PR #129 as `9a04434`" in artifact_map - artifact_phases = ( - "Deterministic proof passed; exact-SHA internal review in progress", - "Internal review and deterministic evidence passed; external checks pending", - ) - selected_phases = [phase for phase in artifact_phases if phase in artifact_map] - assert len(selected_phases) == 1 - selected_phase = selected_phases[0] - assert f"Status: {selected_phase}" in artifact_contract + assert "Merged through PR #141 as `a10d901`" in artifact_map + assert "Active after 02A3 merged through PR #141" in artifact_map + assert "Status: Active after explicit human start" in artifact_contract assert ( "AUTH's owner reconciliation merged through PR #140 as\n" "`d541521`" in artifact_status ) + assert "`WS-ART-001-02B1` is active" in artifact_status + assert "The current gate is deterministic 02B1 proof followed by all nine" in ( + artifact_status.replace("\n", " ") + ) + assert "No later artifact chunk starts automatically" in artifact_status.replace( + "\n", " " + ) assert ( - "`WS-ART-001-02A3` implementation and merged-main deterministic repair are\n" - "complete" in artifact_status + "| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | " + "Merged through PR #146 as `0ffdabf`" in work_queue ) - artifact_02a3_merged = ( - "`WS-ART-001-02A3` | ArtifactStore v2 Local Clean Cut | L1 | " - "Merged through PR #141 as `a10d901`" in work_queue + assert "| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Active" in ( + work_queue ) - if artifact_02a3_merged: - normalized_work_queue = " ".join(work_queue.split()) - assert "PR #141 merged `WS-ART-001-02A3` into `main` as `a10d901`" in loop_state - assert ( - "`WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | " - "Inactive until 02A3 merge and explicit user start" in work_queue - ) - assert "ART-02B1 remains inactive" in loop_state - assert "Do not start AUTH-09C or POL-002-04 automatically." not in ( - normalized_work_queue - ) - assert ( - "Its merged-main deterministic proof and exact-SHA internal review are " - "complete; external checks remain pending and `02B1` must not start " - "automatically." not in normalized_work_queue - ) - elif selected_phase == artifact_phases[0]: - assert selected_phase in work_queue - assert ( - "The current gate is all nine\nexact-SHA internal tracks" in artifact_status - ) - assert ( - "Current gate: complete all nine exact-SHA internal reviewer tracks" - in loop_state - ) - else: - assert selected_phase in work_queue - assert ( - "The current gate is GitHub Actions, CodeRabbit, and explicit human review" - in (artifact_status.replace("\n", " ")) - ) - assert ( - "Current gate: publish PR #141 for GitHub Actions, CodeRabbit, and explicit" - in (loop_state.replace("\n", " ")) - ) - assert "No\nlater artifact chunk starts automatically" in artifact_status + assert "Current ART gate: integrate trusted `main`, complete deterministic 02B1" in ( + loop_state + ) + assert "No later ART chunk starts automatically" in loop_state.replace("\n", " ") def test_stale_authorization_discovery_includes_new_untracked_docs() -> None: @@ -4812,6 +4809,27 @@ def test_agent_gate_dependencies_and_workflow_are_pinned() -> None: ) +def test_local_minio_compose_is_regression_protected() -> None: + """Keep the repository-managed MinIO proof pinned and loopback-only.""" + compose = yaml.safe_load((ROOT / "docker-compose.yml").read_text(encoding="utf-8")) + minio = compose["services"]["minio"] + assert set(minio) == { + "image", + "command", + "environment", + "ports", + "healthcheck", + "volumes", + } + assert minio["image"] == MINIO_IMAGE + assert minio["command"] == "server /data --address :9000" + assert minio["environment"] == { + "MINIO_ROOT_USER": "workstream-minio", + "MINIO_ROOT_PASSWORD": "workstream-minio-secret-key", + } + assert minio["ports"] == ["127.0.0.1:9000:9000"] + + def test_backend_coverage_thresholds_are_regression_protected() -> None: """Keep both the approved global floor and stricter artifact floor fail closed.""" workflow_path = ROOT / ".github/workflows/backend.yml" @@ -4820,6 +4838,12 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: test_job = parsed_workflow["jobs"]["test"] assert set(test_job) == {"runs-on", "timeout-minutes", "services", "steps"} steps = test_job["steps"] + minio_steps = [ + step for step in steps if step.get("name") == "Start real MinIO artifact provider" + ] + assert len(minio_steps) == 1 + assert set(minio_steps[0]) == {"name", "run"} + assert str(minio_steps[0]["run"]).strip() == BACKEND_MINIO_START_COMMAND full_suite_steps = [ step for step in steps if step.get("name") == "Backend full-suite coverage" ] @@ -4829,13 +4853,15 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: assert full_suite_steps[0].get("env") == { "WORKSTREAM_TEST_ADMIN_DATABASE_URL": ( "postgresql+asyncpg://workstream:workstream@localhost:5433/postgres" - ) + ), + "WORKSTREAM_TEST_MINIO_ENDPOINT": "http://127.0.0.1:9000", } for forbidden_key in ("if", "continue-on-error", "shell"): assert forbidden_key not in full_suite_steps[0] assert full_suite_run.strip() == BACKEND_FULL_SUITE_COVERAGE_COMMAND assert "/tmp/workstream-database.json" not in workflow full_suite_index = steps.index(full_suite_steps[0]) + assert steps.index(minio_steps[0]) < full_suite_index isolated_steps = [ step for step in steps if step.get("name") == "Isolated database runner test" ] @@ -5862,6 +5888,7 @@ def main() -> int: test_agent_gates_runs_stale_authorization_docs_fail_closed, test_agent_gates_runs_stale_artifact_contracts_fail_closed, test_agent_gate_dependencies_and_workflow_are_pinned, + test_local_minio_compose_is_regression_protected, test_backend_coverage_thresholds_are_regression_protected, test_artifact_coverage_phase_is_derived_from_work_queue, test_stale_artifact_contracts_cutover_rejects_reached_terms_only,