diff --git a/.env.example b/.env.example index 29b495a..d6fcc08 100644 --- a/.env.example +++ b/.env.example @@ -35,6 +35,13 @@ DATA_LAYER_PRELOAD_WATCHDOG_ENABLED=true BINANCE_WS_MAX_CONNS_PER_SOURCE=0 STREAM_STALE_SECONDS=180 STREAM_STRICT_FEED_HEALTH=false +BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS=15 +BINANCE_WS_IDLE_TIMEOUT_SECONDS=90 +BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS=2 +BINANCE_KLINE_RECOVERY_ENABLED=true +BINANCE_KLINE_RECOVERY_POLL_SECONDS=2 +BINANCE_KLINE_RECOVERY_SETTLE_SECONDS=1 +BINANCE_KLINE_RECOVERY_CONCURRENCY=4 BINANCE_SYMBOLS_FILE=/app/data/cache/binance_usdm_symbols.json BINANCE_SPOT_SYMBOLS_FILE=/app/symbols_spot.json diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..caf88d8 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,63 @@ +# Data Layer Agent Rules + +This tracked file carries the workspace baseline into every clone of this +repository. For Data Layer work, the mandatory main journal is +`DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md`; detailed design is governed by +`upgrade/quant-data-layer-fund-grade-upgrade-architecture.md` and applicable +provider guides. Host-level `/home/bobby/AGENTS.md`, when present, may add +stricter workspace rules but may not weaken this file. + +## Scope And Sources Of Truth + +1. Before acting, read the nearest `AGENTS.md`, the project's main implementation plan, the approved detailed guide, and the relevant code/config. Do not rely on chat memory alone. +2. When the user asks to discuss, evaluate, explain, or plan only, do not edit files, restart services, mutate data, or run destructive commands. +3. Implement only the approved scope. Report newly discovered out-of-scope bugs and proposed fixes before changing them unless the user explicitly authorized fixing all discovered issues. +4. Preserve the declared domain source of truth. For alpha migrations, backtest/approved research logic and parameters are authoritative unless the user says otherwise. Do not silently simplify business logic. + +## Mandatory Plan Journal + +5. Every project change must be recorded in that project's main plan markdown. The plan update is part of the implementation transaction, not optional documentation after the fact. +6. Before code changes, record or confirm the phase/task status, goal, guide links, approved scope, invariants, test gates, rollback, and decision boundary in the main plan. +7. During implementation, update the plan incrementally after each coherent tested slice with exact work completed, commands/tests actually run, results, cleanup evidence, decisions, and remaining debt. +8. Never claim a task or phase is complete until the main plan and any governing detailed guide agree with the code and evidence. A final response must state the recorded status and link/path. +9. Technical debt means a real external, cost, infrastructure, licensing, business-semantics, or approval gate. Fix in-scope defects before closure; do not relabel unfinished work or failed tests as debt. +10. If a project has no identified main plan, identify or create one before substantial implementation. Do not scatter progress across ad hoc markdown files without linking them from the main plan. + +## Runtime And Data Safety + +11. Protect running production/V1 consumers by default. Use isolated names, ports, networks, schemas, Redis prefixes, volumes, topics, consumer groups, credentials, and evidence paths for development and tests. +12. Do not restart, recreate, cut over, prune, flush, truncate, force-close, delete volumes/data, or change authority unless the user explicitly approves the exact blast radius and rollback. +13. Production/shadow market data must come from approved real providers or replay of durably captured provider bytes. Synthetic/generated data is test-only and must carry test provenance. +14. Never commit secrets, raw credentials, private keys, unbounded logs, caches, runtime state, or generated data. Evidence stores identifiers, hashes, bounded metrics, and provenance only. +15. Clean disposable test resources and scoped smoke rows after verification. Never clean shared state broadly when an exact alpha/account/test namespace is available. + +## Verification And Evidence + +16. Test correctness before performance: identity, units, decimals, timestamps/timezones, ordering, bar closure, sequence, source authority, state transitions, risk/order semantics, and business-domain parity. +17. Use applicable unit, contract/golden, parity/oracle, migration-idempotency, integration, failure/reconnect, recovery/rollback, compatibility, security, resource/capacity, and bounded real-provider tests. +18. A health endpoint or process-up state alone is not acceptance. Report exact cases, pass/fail/skip counts, untested boundaries, production mutations, and cleanup results. +19. Never present local/same-host rehearsal as production, HA, independent failure-domain, broker-authoritative, sandbox/live, or real-provider evidence. Missing evidence fails closed and is documented honestly. +20. Read large logs and databases intelligently: filter relevant time ranges and warning/error/event IDs first, then query exact related rows. Avoid unbounded output and token-heavy tool use. + +## Git And Change Discipline + +21. Inspect branch/worktree before editing. Preserve user/unrelated changes and never revert them without explicit instruction. +22. Use feature branches from `dev` when the repository workflow requires it. Feature branches merge to `dev`; `main` is release-only. Never merge or push unless explicitly requested. +23. Commit each coherent, tested implementation slice with a clear message. Do not create excessive tiny commits or one giant untraceable commit. +24. Commits must use the user's configured identity (`BobbyAxerol `), never a system/root identity. Verify identity before commit. +25. Before commit: run `git diff --check`, inspect staged scope, confirm tests/evidence and plan updates, and exclude unrelated files. After commit: report commit SHA, branch, worktree state, and whether anything was pushed/merged. + +## Engineering And Operations + +26. Prefer existing shared runtime, SDK, contracts, wrappers, parsers, and domain abstractions. Copy unchanged code mechanically when appropriate; avoid duplicated strategy-specific infrastructure. +27. Keep public V1/stable endpoints and schemas backward-compatible unless an approved versioned migration says otherwise. Internal implementation language and transport must stay behind contracts. +28. Design provider-neutral, venue-capability-driven, scalable boundaries. Do not hardcode one venue/mode where the domain needs extension to Binance, OKX, DNSE/VN, Deribit, or future brokers. +29. Optimize only after correctness. Measure CPU, memory, disk, queue/lag, I/O and latency; require no unexplained loss, duplicate, gap, or state mismatch before promotion. +30. Keep logs structured, host-visible where required, bounded and aligned to strategy/feed intervals. Avoid tick-level INFO spam unless actively diagnosing a new feed. +31. Use the smallest useful set of tools and agents. Prefer `rg`, targeted reads, scoped tests, reusable commands, and compact evidence over broad scans or repeated full-output calls. +32. Use `apply_patch` for manual edits when available. If the sandbox helper is broken, use an exact-match scripted replacement and immediately verify the diff. + +## Reporting + +33. Working updates explain what is being checked or changed and why. Final reports state: implemented scope, domain behavior, tests/evidence, runtime impact, cleanup, remaining decision gates, commit/branch, and the next permitted step. +34. Do not hide blockers or overstate readiness. Distinguish `implemented`, `tested locally`, `shadow-certified`, `production-ready`, and `production-authoritative` explicitly. diff --git a/Cargo.lock b/Cargo.lock index f467127..58afb36 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -543,6 +543,7 @@ dependencies = [ name = "qdl-venue-core" version = "0.1.0" dependencies = [ + "hex", "serde", "serde_json", "sha2", diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 62ca066..0445e66 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -1,7 +1,7 @@ # Quant Data Layer Unified Implementation Plan -> **Status:** Phases 0-5 are complete; Phase 6 implementation and shadow certification pass, while production authority remains `NO-GO` on explicit infrastructure gates. Phase 7 is complete with a protected read-only `BETA-GO`; Phase 8 is complete with an immutable, signed, multi-venue Rust realtime-core candidate fenced to `RUST_SHADOW`; Phase 9 remains planned. V1 remains authoritative and no runtime cutover has started. -> **Working branch:** `feat/fund-grade-data-layer-v2`, created from `dev`. +> **Status:** Phases 0-5 are complete; Phase 6 implementation and shadow certification pass, while production authority remains `NO-GO` on explicit infrastructure gates. Phase 7 is complete with a protected read-only `BETA-GO`; Phase 8 is complete with an immutable, signed, multi-venue Rust realtime-core candidate fenced to `RUST_SHADOW`; Phase 9.0-A and 9.0-B are complete in isolation; Phase 9.0-C is `COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL`; Phase 9.1 is `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`; Phase 9.2 is `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED`; Phase 9.3 is `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED` after isolated hold/closure/expansion governance certification. Authority promotion, production hold/closure and every expansion remain blocked on explicit production infrastructure, real canary/primary evidence and exact-slice approval gates. V1 remains authoritative and no runtime cutover has started. +> **Working branch:** `feat/phase93-hold-close-expand`, stacked on the completed Phase 9.2 checkpoint and intended for PR into `dev`; no push, merge or authority cutover is implied by control-plane completion. > **Detailed architecture:** [Fund-grade architecture and migration guide](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md) > **OKX V5 market-data specification:** [OKX Market Data V5 implementation guide](upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md) > **Compatibility boundary:** Existing `/v1`, SDK v1, Redis keys and Redis Pub/Sub remain supported until a governed per-consumer sunset. @@ -160,6 +160,21 @@ These rules apply to all phases. PostgreSQL control-state restore, cursor-key rotation, object-store/PITR, Redis/projector rebuild and authority reconstruction from the audit log have passed the approved recovery objectives. +40. **The main plan is a transactional implementation journal.** Before code, + record the approved phase/scope, guide links, invariants, test gates and + rollback here. After every coherent tested slice, record exact completion, + verification, cleanup, decision gates and debt here in the same commit. A + phase cannot be reported complete while this tracker or its governing guide + disagrees with code/evidence. +41. **Scope and approval are explicit.** A discussion/evaluation request causes + no mutation. Newly discovered work outside the approved scope is reported + with impact and recommendation before implementation. Restart, cutover, + authority change, destructive cleanup, push and merge require the user's + explicit approval for that action. +42. **Final reporting is evidence-bound.** Every completion report names the + plan status, tests actually run with pass/fail/skip counts, untested/external + gates, runtime impact, cleanup, commit/branch and push/merge state. Local or + same-host proof is never upgraded linguistically into production evidence. ## 3. Phase Summary @@ -2274,7 +2289,7 @@ Phase 8 is `COMPLETE` only when: ## 13. Phase 9 - Rust Core Canary And Progressive Replacement -**Status:** `PLANNED` +**Status:** `9.0-A COMPLETE_ISOLATED`; `9.0-B COMPLETE_ISOLATED`; `9.0-C COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL`; `9.1 COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`; an actual production `RUST_CANARY` transition remains blocked on an exact `GO` bundle ### Goal @@ -2657,6 +2672,383 @@ must prove: #### 9.0 Production Prerequisites And Exact Slice Approval +##### 9.0-A Runtime Correctness Closure + +**Status:** `COMPLETE_ISOLATED` + +**Purpose:** Close correctness and deployment-boundary defects discovered after +the server migration before any Phase 9 authority implementation or Rust +canary. This slice changes no public V1 contract, source authority, canonical +writer or running production container until isolated evidence passes and a +separate operator-approved deployment is prepared. + +**Observed baseline (2026-08-18):** + +- The running service is the V1-only `data-layer:v0.1.0` image with host source + bind-mounted read-write at `/app`; OpenAPI exposes 40 V1 routes and zero V2 + routes. No Phase 7 beta, Phase 8 Rust, Kafka or OTel role is active. +- Binance USD-M TRADE is producing authentic frames, but all eight configured + USD-M KLINE shards have remained connected with `message_count=0`. All 734 + expected one-minute kline feeds are missing while `/v1/health` incorrectly + reports `binance_kline_stream=true`. +- Independent read-only probes reproduce provider ACK/connected-without-data: + raw trade and book-ticker produce frames, while kline, aggregate-trade and + mark-price subscriptions time out after successful connection/subscription. +- Feed demand leases are zero while the runtime still opens the broad USD-M + universe. Historical queue-drop count is 138,060; the measured recent + five-minute delta is zero. +- The production container has no explicit CPU, memory or PID limit. Redis and + V1 HTTP remain available; no production state was mutated during discovery. + +**Invariants:** + +- A WebSocket handshake or subscription ACK is transport state, never data + readiness. Enabled feeds that do not produce a valid provider frame inside + their declared first-frame/staleness deadline fail closed to a typed degraded + state. +- Recovery uses provider-authentic Binance REST closed bars for active demand + only. It never fabricates candles, marks the open candle final, substitutes + OKX data as Binance authority or hides WebSocket degradation. +- Trade/book canonical events are not silently coalesced or dropped. Existing + legacy latest-state projection behavior remains contract-compatible while + queue loss and recovery state stay observable. +- Tests use isolated processes, Redis prefixes and Compose project names. They + do not restart V1, flush shared Redis, mutate production Parquet or reuse live + consumer groups. Disposable state is removed after evidence capture. + +**Implementation tasks:** + +1. Split stream transport, source and per-feed readiness. Report TRADE and + KLINE independently; expose connected shard count, producing shard count, + first-frame deadline, stale deadline, recovery source and active demand. +2. Add a bounded data-frame watchdog. A connected shard with no valid frame by + deadline enters `DATA_UNAVAILABLE`, records the outage and reconnects with + jittered backoff rather than remaining green forever. +3. Add one demand-scoped closed-kline recovery loop. It batches and rate-limits + Binance REST requests, emits only fully closed rows with explicit + `BINANCE_REST_GAP_FILL` provenance, deduplicates by symbol/interval/open time, + and stops after demand leases expire. +4. Make demand registry ownership and TTL visible. REST/SDK demand renewal must + not make reads fail, but missing registration cannot authorize broad source + health or resource claims. +5. Correct `/v1/health` without changing its response keys: boolean TRADE/KLINE + fields reflect their own data readiness; top-level status degrades for an + enabled unavailable source while market-closed DNSE remains healthy by + policy. Add detailed source readiness under the existing nested supervisor + payload. +6. Add an immutable production Compose overlay with no host source bind and + explicit CPU, memory, PID, read-only-root, tmpfs and writable data/log/cache + boundaries. Keep the current deployment untouched; cutover requires a + digest-pinned image, preflight and operator approval. +7. Freeze compact machine-readable evidence and an implementation report, + including authentic probe results, unit/integration counts, V1 golden diff, + resource limits, production-unchanged proof and cleanup. + +**Implementation checkpoint (2026-08-18):** + +- Implemented valid-frame watchdog, independent source readiness, bounded queue + backpressure and demand-only Binance closed-kline REST recovery. Removed the + reconnect-only recovery duplicate so one manager owns scheduling, dedup and + backoff. +- Recovery preserves the provider interval from `k.i`, rejects open/invalid rows, + retains explicit provenance and expires work with the final demand lease. +- V1 health keys remain unchanged; TRADE and KLINE booleans now represent their + own data readiness. Added owner visibility to the existing demand snapshot. +- Added immutable isolated Compose candidate with non-root/read-only execution, + no source bind, dedicated state, loopback-only ingress and CPU/RAM/PID limits. +- Deterministic verification passed: targeted runtime matrix 35/35; full repository + suite ran 345 tests with 340 passes, 5 environment-gated skips and zero failures; + compile/diff checks clean; and + live-vs-candidate OpenAPI path diff 40/40 with zero additions or removals. +- Built and ran immutable candidate digest `sha256:4a2723ec39057c75a89889d955feac7acc6fb01bc126a579f8c74d384b9b6999` as UID 10001 with read-only root, no source bind and declared CPU/RAM/PID limits. +- Real-provider smoke proved 8/8 USD-M TRADE shards ready and 0/8 KLINE shards + unavailable; health stayed degraded while demand-only REST recovery returned a + final BTCUSDT bar exactly equal to Binance REST OHLCV. Lease expiry stopped + further provider fetches. Queue pressure and drop deltas remained zero. +- The isolated smoke exposed and closed two candidate bugs before release: Redis + UID with `cap_drop: ALL`, and data outage being cleared by transport reconnect. +- Candidate containers, networks, volumes and images were removed after evidence. + Production V1 remained unchanged and running throughout. Evidence: [Phase 9.0-A + report](upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md) and [machine + result](upgrade/evidence/phase90a-runtime-correctness.json). + +**Verification cases:** + +- Valid TRADE plus valid KLINE frames make only their matching source ready. +- Connected/ACKed KLINE with zero frames misses the first-frame deadline and is + degraded; `binance_kline_stream` must be false. +- One dead kline shard cannot mark other feed types unavailable, and one healthy + trade shard cannot make kline healthy. +- Stale, malformed and wrong-feed frames do not satisfy readiness. +- Active kline demand receives provider-authentic fully closed REST recovery; + open rows, duplicates and non-demanded symbols are rejected. +- REST timeout, 429/5xx, partial batch, reconnect, Redis outage, queue pressure, + lease expiry and process restart remain bounded and observable. +- V1 OpenAPI/golden payloads and legacy Redis keys/channels do not change. +- Isolated real-provider smoke proves the actual provider behavior and recovery + path; generated/simulated data is limited to deterministic failure tests and + is never counted as provider evidence. +- Immutable image runs as non-root without source bind, honors resource limits, + passes liveness/readiness/data-readiness probes and leaves V1 unchanged. + +**Exit gate:** + +- Zero false-green source readiness in the verification matrix. +- Zero fabricated/open-as-final bars and zero unexplained duplicate recovery + publication. +- Zero V1 contract/golden regression and zero production mutation. +- Bounded CPU, memory, queue, request rate and retry/backoff under normal, + outage and recovery cases. +- All disposable containers, networks, volumes, Redis prefixes and captures are + removed; compact checksummed evidence remains. +- Phase 9.1 remains blocked. Completing 9.0-A does not satisfy production OTel, + independent failure-domain DR, workload identity, external secrets, + signature admission, consumer registration or exact-slice approval. + +**Rollback:** + +- Do not deploy the candidate overlay; continue the unchanged V1 container. +- If a later approved deployment regresses, restore the immutable V1 image and + source configuration, remove only the candidate namespace and verify V1 + OpenAPI/Redis compatibility plus provider data readiness. +- REST recovery can be disabled independently; source health must remain + degraded rather than reverting to connected-is-ready semantics. + +##### 9.0-B Isolated V2 Beta + +**Status:** `COMPLETE_ISOLATED` + +**Purpose:** Re-certify the existing provider-neutral V2 query/stream beta on +the migrated host using the Phase 9.0-A runtime-correctness baseline. The beta +is a read-only, non-authoritative consumer of one explicitly bounded V1 source +slice; it is not a Rust canary, a public-internet deployment or a source +authority transition. + +**Guide index:** + +- [Phase 9.0-B isolated beta boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-e--phase-90-b-isolated-v2-beta-boundary) +- [V2 API/SDK and consumer migration](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#18-sdk-v2-architecture) +- [No-big-bang migration](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#30-migration-strategy-no-big-bang-rewrite) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) +- [Phase 7 isolated beta runbook](docs/runbooks/phase7-isolated-beta-runtime.md) +- [Phase 9.0-B certification runbook](docs/runbooks/phase90b-isolated-v2-beta.md) + +**Invariants:** + +- V1 remains the only production/source authority and keeps its existing + container, networks, mounts, Redis namespaces and public contracts. +- Beta reads only the approved internal V1 endpoint and publishes only into its + dedicated canonical spool, Redis prefix, consumer group and loopback ports. +- The beta image is content-addressed, non-root, read-only, bounded and contains + no host source bind. JWT, cursor and bridge secrets are beta-only. +- Beta output uses provider-authentic, final closed bars. Fixtures/synthetic + events are permitted only in isolated deterministic tests and cannot satisfy + the real-provider gate. +- Process readiness, dependency readiness and data availability are tested + separately. Missing/stale source data must produce typed unavailability, not + a false-green data claim. + +**Implementation tasks:** + +1. Reuse the Phase 7 V2 query, active/passive stream, dedicated AOF Redis and + read-only V1 bridge topology; do not fork or rename stable contracts merely + because this is a new certification phase. +2. Pin the candidate and helper images by digest, stamp the application artifact + with source revision, and use a Phase 9.0-B-specific config revision, Redis + prefix, consumer group, project name, leases, credentials and evidence paths. +3. Run the continuous bridge only with the `phase7-canary` profile and prove it + cannot call a venue directly, write V1 state or publish non-final bars. +4. Validate authenticated V2 warmup/query and gRPC replay/live handoff for the + approved BTCUSDT USD-M 1m BAR slice, including decimal/timestamp/finality, + event identity, cursor continuity and V1-vs-V2 parity. +5. Exercise active/passive failover and fencing, Redis outage/recovery, process + restart, stale/invalid cursor, malformed/auth abuse, rate/concurrency bounds, + slow consumer and duplicate bridge polling. +6. Measure bounded CPU, memory, PIDs, Redis/durable-store growth, request and + stream latency. Record untested infrastructure gates honestly. +7. Tear down all beta containers, networks, volumes, images, keys and temporary + credentials; prove V1 topology/state and API remain unchanged. Freeze a + checksummed human and machine-readable report. + +**Verification and exit gate:** + +- Existing V2 contract/security/unit suites and the Phase 9.0-A regression + matrix pass with zero unexplained domain mismatch. +- Exactly one stream replica is active; failover increments the fencing epoch + and stale-owner operations fail closed. +- Authentic V1 and V2 closed bars match exactly for identity, interval, OHLCV, + timestamps and finality; replay/live offsets are contiguous with no duplicate + external event. +- Missing credentials, wrong audience/environment/scope/consumer, malformed + requests and expired/tampered cursors fail closed with typed errors. +- Dependency failure makes readiness unavailable while V1 fallback remains + healthy. Recovery is bounded and does not require V1 restart. +- Resource limits hold, no beta state enters production Redis, V1 OpenAPI paths + remain unchanged, and cleanup counters are all zero. +- Completion authorizes review of an isolated V2 beta only. Phase 9.1 and any + Rust/source authority promotion remain blocked on the mandatory production + infrastructure and exact-slice operator gates. + +**Rollback:** Stop and remove only the isolated Compose project and its volumes, +revoke beta credentials, verify zero beta keys in production Redis and continue +the unchanged V1 path. V1 requires no replay, resubscription or restart. + +**Completed:** + +- Reused the frozen Phase 7 V2 query/stream/bridge topology with dedicated + Phase 9.0-B projects, AOF Redis, stores, credentials, ports and namespaces. +- Built and certified immutable candidate revision `1c881389b4ee21a153903505822c61512b176044` + as non-root UID/GID `10001`, read-only root, no host source bind and bounded + resources. +- Added exact provider-bar parity, continuous bridge, immutable provenance, + adversarial/security, capacity, V1 topology and deterministic cleanup gates. +- Fixed rootless evidence ownership and complete Compose profile activation in + the certification harness; both defects now have regression coverage. + +**Verification:** + +- Full Python regression: `351` tests, `346` passed, `5` skipped, `0` failed. +- Real provider slice: `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`; + canonical mismatches `0`, generated market events `0`, duplicate timestamps + `0`, non-final bars `0`, execution-eligible events `0`. +- Query load: normal `110.204 req/s`, burst `65.012 req/s`; `0` errors; + p99.9 `72.211 ms` and `486.649 ms`. Stream throughput was `1815.252` + events/s and measured end-to-end freshness was `6695.751 ms`. +- Authentication, entitlement, cursor tamper/expiry/scope, malformed/oversized + request, rate limit, active/passive fencing, Redis outage/recovery, slow + consumer isolation and replay continuity all passed fail-closed gates. +- Production V1 container/image/start time and OpenAPI digest were unchanged; + beta containers/networks/volumes/image tags and production beta keys after + cleanup were all `0`. +- Frozen evidence: [human report](upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md), + [machine decision](upgrade/evidence/phase90b-isolated-v2-beta.json), + [continuous parity](upgrade/evidence/phase90b-continuous-bridge.json), + [capacity](upgrade/evidence/phase90b-capacity.json), + [security](upgrade/evidence/phase90b-security-adversarial.json) and + [checksums](upgrade/evidence/phase90b-evidence.sha256). + +**Technical debt / decision gate:** No in-scope Phase 9.0-B defect remains. +This result authorizes only isolated read-only beta review. Phase 9.1 remains +blocked on replicated production transport, OTel/alert routing, workload +identity/RBAC, external secret rotation, signature admission, independent DR, +complete critical-consumer registration and explicit exact-slice approval. + +##### 9.0-C Production Prerequisites + +**Status:** `COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL` (2026-08-18) + +**Purpose:** Turn every Phase 9 production prerequisite into an explicit, +machine-verifiable, fail-closed gate. Reuse valid Phase 6/8/9 evidence without +misrepresenting same-host rehearsal as independent production infrastructure. +This subphase does not deploy a public V2 endpoint, promote Rust, change V1 +authority or approve an exact slice by implication. + +**Guide index:** + +- [Phase 9 production prerequisite boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-f--phase-90-c-production-prerequisite-boundary) +- [Deployment architecture](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#29-deployment-architecture) +- [Migration and authority](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#30-migration-strategy-no-big-bang-rewrite) +- [Operational runbooks](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#34-operational-runbooks) +- [Performance policy](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#37-performance-engineering-policy) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Invariants:** + +- A local container, SQLite spool, in-process metric buffer, debug exporter, + self-signed test key or same-host broker replica can prove code behavior but + cannot satisfy a production/failure-domain gate. +- Evidence is immutable, checksummed, scoped, expiring where applicable and + attributable to an operator or workload identity. Missing, malformed, + expired, lower-scope or contradictory evidence blocks promotion. +- Authority state is persistent and transitions by compare-and-swap. Entering + `RUST_CANARY` or `RUST_PRIMARY` requires a passing prerequisite bundle and an + explicit exact-slice approval; config booleans cannot bypass this rule. +- Existing V1 containers, OpenAPI, Redis namespaces and venue subscriptions + remain unchanged throughout isolated certification. + +**Implementation tasks:** + +1. Add a provider-neutral prerequisite policy covering replicated transport, + production telemetry/alert acknowledgement, workload identity/RBAC/network + policy, external secret rotation, signed-image admission, PostgreSQL PITR, + object-store restore, independent DR, Redis/projector rebuild, consumer + registration/rollback and exact-slice approval. +2. Add strict evidence and exact-slice schemas plus a deterministic evaluator + that emits `GO` only when every required production gate passes. Preserve + local rehearsal as `LOCAL_ONLY`, never silently upgrade its scope. +3. Add additive PostgreSQL authority/prerequisite/audit tables and a CAS + transition function with immutable audit, stale revision/owner/lease/plan + rejection and guarded canary/primary transitions. +4. Freeze a candidate manifest for the bounded Binance USD-M BTCUSDT TRADE + slice in `RUST_SHADOW`; public and legacy writes remain forbidden. +5. Re-run unit/contract/migration tests and applicable isolated broker, + security, recovery and V1 compatibility checks. Test malformed, missing, + stale, expired, forged, lower-scope and conflicting evidence. +6. Produce a compact machine/human gate report, explicit blocker inventory, + deployment/rollback runbook and portable checksums; remove all disposable + resources and prove V1 unchanged. + +**Exit gate:** + +- Code/schema/migration/evaluator and local certification may close as + `COMPLETE_CONTROL_PLANE` while the overall decision remains `NO_GO_EXTERNAL`. +- `PRODUCTION_PREREQUISITES_PASS` requires real replicated/failure-domain, + observability/page acknowledgement, identity/secret/admission, restore/DR, + consumer-owner and operator approval evidence. No test fixture can satisfy it. +- Phase 9.1 remains blocked unless the exact bundle decision is `GO`, its + artifact/config/contract digests match the candidate, and the approval names + the exact authority slice. + +**Rollback:** Keep V1 authoritative, remove only Phase 9.0-C disposable test +resources, and retain additive control-plane/audit records. Revoking or expiring +any prerequisite evidence immediately restores `NO_GO`; it never starts or +restarts a producer. + +**Implementation and verification (2026-08-18):** + +- Added a provider-neutral 12-gate production policy, strict candidate/evidence + models and deterministic evaluator. Unknown gates, nested secret fields, + duplicate bindings, unsafe artifact paths, invalid hashes/timestamps, stale or + lower-scope evidence, semantic threshold failures and candidate mismatches all + fail closed. +- Froze candidate digest + `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` + at `RUST_SHADOW`; public/legacy writes remain disabled. Release provenance + includes image/SBOM/signature, contract, normalizer, adapter, config, catalog, + source-policy, partition-plan and rollback revisions. +- Added additive prerequisite-bundle, authority-slice and immutable transition + audit schema. The database CAS rejects stale state/revision/owner/lease/plan, + stale owners, missing terminal watermark, missing/expired hold windows, + `NO_GO`, expired or candidate-mismatched bundles and audit mutation. +- Isolated PostgreSQL migration applied twice successfully. Nine negative safety + outcomes passed, two valid state transitions were audited, production + mutations were zero and the disposable container was removed. +- Targeted control-plane suite passed `13/13`; focused cross-phase candidate + suite passed `29/29`; full candidate-image suite ran `364` tests with `359` + pass, `5` intentional skips and `0` failures. +- Machine evaluation correctly produced `NO_GO_EXTERNAL`: `0/12` production + gates passed because the available proof is local, missing or explicitly + blocked. V1 container identity, topology and `/v1/health` remained unchanged + and healthy before/after evaluation; no production database or Redis was + mutated. +- Workspace governance is canonical in `/home/bobby/AGENTS.md` and its + repository-tracked [AGENTS.md](AGENTS.md) copy; Rules 40-42 make plan + synchronization, explicit approval boundaries and evidence-bound final + reporting mandatory for every later slice and cloned workspace. +- Runbook and portable evidence are frozen at + [phase90c-production-prerequisites.md](docs/runbooks/phase90c-production-prerequisites.md), + [phase90c-production-prerequisites.json](upgrade/evidence/phase90c-production-prerequisites.json), + [phase90c-authority-migration.json](upgrade/evidence/phase90c-authority-migration.json) + and [phase90c-evidence.sha256](upgrade/evidence/phase90c-evidence.sha256). + +**Technical debt / decision gate:** No in-scope control-plane defect remains. +The 12 blockers are deliberately external production deployment/operator gates, +not evidence that can be fabricated in this repository. Phase 9.1 remains +blocked until a fresh exact-candidate bundle evaluates `GO`; completing this +subphase does not authorize public V2, Rust canary or any authority cutover. + +**Phase 9.1 prerequisites:** + - Close every applicable Phase 6 `NO-GO` blocker with real infrastructure evidence. - Deploy production durable transport, OTel collector/dashboards/alerts, @@ -2675,46 +3067,427 @@ must prove: #### 9.1 Rust Canary -- Transition only the selected slice to `RUST_CANARY` by CAS. -- Keep Python as the sole authoritative public writer. -- Dual-read/compare canonical Rust output against the same authentic frame range. -- Run live, replay, reconnect, lease loss, producer crash, broker failover, - projector restart and slow-consumer tests. -- Exercise automated blocking thresholds without allowing dual public writes. -- Verify the compatibility projector remains fed only by the active authority. -- Hold the canary for the approved traffic/session window. -- Revert to `RUST_SHADOW` on any unexplained semantic mismatch. +**Status:** `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED` + +**Purpose:** Implement and certify the exact-slice Rust canary path while Python +remains the sole authoritative public/V1 writer. Because Phase 9.0-C currently +returns `NO_GO_EXTERNAL`, this phase may run only isolated rehearsal and +fail-closed authorization tests. It must not persist a production +`RUST_CANARY`, publish public/legacy output or imply production readiness. + +**Guide index:** + +- [Phase 9.1 canary boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-g--phase-91-rust-canary-boundary) +- [Migration and authority](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#30-migration-strategy-no-big-bang-rewrite) +- [Performance policy](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#37-performance-engineering-policy) +- [Operational runbooks](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#34-operational-runbooks) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Invariants:** + +- The exact candidate remains `BINANCE / USDM / PERPETUAL / TRADE / BTCUSDT / + partition-plan epoch 1`; a changed image, contract, catalog, source policy, + normalizer, adapter or partition plan creates a new candidate digest. +- Python V1 remains the only public and legacy writer. Rust canary output uses a + dedicated isolated canonical namespace; two public writers are impossible. +- Production authorization consumes the strict Phase 9.0-C decision. Missing, + stale, `NO_GO`, mismatched or unapproved evidence cannot be bypassed by an + environment variable, test flag or direct state boolean. +- Every canary publication binds exact slice, owner, authority revision, lease + epoch and partition-plan epoch. Sink-side fencing rejects stale/conflicting + writers, not only producer-side self-checks. +- Parity uses the same authentic captured provider frames and compares identity, + exact decimals, timestamps, sequence, quality, event ID, payload hash and + deterministic bytes. Generated events are test-only and never parity proof. +- Any correctness mismatch, open gap, duplicate external output, stale-writer + attempt, lag/freshness/resource breach or authority ambiguity blocks the + canary. A hold-down prevents automatic re-entry/flapping. + +**Implementation tasks:** + +1. Add a provider-neutral Phase 9 canary manifest and strict authorizer bound to + the Phase 9.0-C candidate/evidence decision. Separate production activation + from isolated rehearsal at the type/API boundary. +2. Extend the Rust authority/sink core with a versioned Phase 9 record carrying + owner, authority/lease/partition epochs, candidate and prerequisite bundle, + start watermark, approval and hold window. Preserve the Phase 8 V1 internal + record decoder for compatibility. +3. Enforce sink fencing for wrong slice, owner, revision, lease, partition plan, + target, watermark and expired/blocked state. `RUST_CANARY` permits only + isolated canary canonical output; public and legacy targets remain denied. +4. Add deterministic same-frame parity and guardrail evaluation with bounded + lag/freshness/resource thresholds, first-failure reason, immutable + observations and anti-flapping/hold-down behavior. +5. Build an isolated canary certification harness using the frozen authentic + Binance capture and replicated test broker. Exercise normal/burst/replay, + process restart, lease loss, stale owner, broker restart/min-ISR failure, + slow consumer, guardrail block and rollback to shadow. It must verify zero + public/legacy writes and unchanged V1 topology/health. +6. Produce strict machine/human evidence, checksums and an operator runbook. + Clean all disposable topics/groups/containers/networks/volumes/images and + record exact test counts and unresolved external gates. + +**Verification and exit gate:** + +- Unit/contract/golden tests cover malformed manifests, candidate mismatch, + `NO_GO`, expiry, stale authority fields, forbidden targets, guardrail triggers + and anti-flapping. +- Python/Rust replay over the same authentic frame range has zero unexplained + mismatch across clean process restarts and burst repetition. +- Isolated broker/recovery tests preserve one canary owner, reject stale writes, + keep public/legacy write counts at zero and clean all test resources. +- Full Data Layer suite passes and V1 container identity, OpenAPI/health, Redis + namespaces and source ownership remain unchanged. +- With the current Phase 9.0-C decision, the maximum allowed closure is + `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`. Only a fresh exact-candidate + `GO` bundle plus explicit operator approval can advance the production state + to `RUST_CANARY` and begin the approved hold window. + +**Implementation journal (2026-08-18):** + +- `COMPLETE` control-plane slice: strict production authorizer consumes the + exact Phase 9.0-C decision/candidate/bundle and rejects `NO_GO`, stale, + incomplete, mismatched or V1-mutating evidence. Isolated rehearsal is a + separate non-production authorization mode. +- `COMPLETE` guardrail slice: immutable observations, zero-tolerance semantic + checks, bounded lag/freshness/resource checks, first-failure capture and + explicit reset only after hold-down. Targeted result: 9/9 tests pass in a + disposable container; production mutations remain zero. +- `COMPLETE` Rust authority/sink v2: Phase 8 record/sink v1 remains unchanged; + v2 binds slice/owner/revision/lease/partition-plan/candidate/bundle/watermark, + denies public/V1 targets and advances watermark only after durable ACK. + Rust result: clippy with `-D warnings` passes; 16/16 focused crate tests pass. +- `COMPLETE` authentic same-frame parity: 128 frozen real Binance USD-M trade + frames repeated 200 times produced 25,600 canonical events. Python and three + clean Rust process runs matched exact record and aggregate hashes with zero + semantic mismatch. Measured throughput was 27,115.455 events/s for Python and + at least 350,581.025 events/s for Rust on this host; this is certification + evidence, not a capacity promise. +- `COMPLETE` replicated-broker rehearsal: exact authority transitions were + `RUST_SHADOW -> RUST_CANARY -> BLOCKED -> RUST_SHADOW`; one-replica-loss ACK, + below-min-ISR fail-closed, full broker restart, compacted authority recovery, + immutable audit ordering and 64-record slow-consumer catch-up all passed. + Public and legacy writes remained zero. +- `COMPLETE` verification: Rust format/clippy pass; Rust workspace 32/32; focused + Phase 8-9 Python matrix 73/73; full Python suite 381/381 with 5 intentional + skips. Certification cleanup left zero isolated containers, networks and + volumes; V1 health stayed HTTP 200 and topology remained unchanged. +- `COMPLETE` evidence and operations: [machine evidence](upgrade/evidence/phase91-rust-canary-certification.json), + [human report](upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md), + [checksums](upgrade/evidence/phase91-evidence.sha256) and + [runbook](docs/runbooks/phase91-rust-canary.md). The harness now separates + compacted authority partition reads from audit/consumer catch-up semantics and + uses process-loss fault injection through surviving bootstrap nodes. +- `OPEN EXTERNAL GATE`: Phase 9.0-C production infrastructure/operator evidence + remains `NO_GO_EXTERNAL`; same-host replicas are not an independent failure + domain. No production canary authority or V1/public mutation was performed. + +**Rollback:** In rehearsal, persist a higher-revision `RUST_SHADOW` record, +fence the canary owner, reconcile the bounded cursor range and remove only the +isolated namespace. In production, use the formal rollback protocol above; do +not restart Python or edit authority state outside the CAS/audit path. #### 9.2 Bounded Rust Primary -- Execute the formal terminal-watermark cutover. -- Increment authority revision and lease epoch atomically. -- Enforce final-sink and compatibility-projector fencing. -- Disable only the exact Python venue subscription after Rust ownership is - accepted and the handoff boundary reconciles. -- Monitor consumer state, quality, freshness, broker/projector lag, queue/spool, - resource headroom and stale-owner attempts under enhanced alerting. -- Rehearse the formal rollback within the approved RTO before closing the - rollback window. -- Preserve V1 contract behavior and V2 snapshot/cursor/replay continuity. +**Status:** `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED` + +**Purpose:** Implement the terminal-watermark ownership protocol required to +promote one exact, already-certified `RUST_CANARY` slice to `RUST_PRIMARY` while +preserving one authoritative writer, V1 compatibility and V2 cursor/replay +continuity. Phase 9.0-C is still `NO_GO_EXTERNAL`; therefore this phase may +certify only an isolated bounded-primary rehearsal. It must not disable a real +Python subscription, mutate production authority or write production +canonical/public/legacy destinations. + +**Guide index:** + +- [Phase 9.2 bounded-primary boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-h--phase-92-bounded-rust-primary-boundary) +- [Formal authority model](#authority-state-machine) +- [Terminal-watermark protocol](#formal-cutover-watermark-protocol) +- [Formal rollback protocol](#formal-rollback-protocol) +- [Operational runbooks](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#34-operational-runbooks) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Exact scope and invariants:** + +- The rehearsal remains bound to the frozen Phase 9.1 candidate slice and + candidate/partition/config/schema digests. Any changed identity creates a new + candidate and invalidates inherited evidence. +- A terminal checkpoint is immutable and identifies the old owner, authority + revision, lease epoch, partition-plan epoch, source session/generation, + terminal event and final durable watermark `W`. +- Primary authorization requires an accepted handoff whose parity range is + gap-free and mismatch-free through `W`. Rust begins authoritative output at + exactly `W + 1`; `<= W` is stale and `> W + 1` before the boundary commit is + an open gap. +- Authority owner, revision and lease epoch change together in one persistent + compare-and-swap transaction. Revision advances exactly by one and a changed + owner requires a strictly newer lease epoch. +- Final canonical sink and V1 compatibility projector independently enforce the + same authority record. Producer self-checks are insufficient. The old owner, + stale lease/revision, wrong plan, wrong destination and duplicate watermark + all fail closed. +- Only `RUST_PRIMARY` may emit authoritative canonical, public V2 and legacy V1 + compatibility output for the promoted range. Shadow/canary/blocked states + retain their narrower Phase 9.1 permissions. +- Isolated rehearsal topics may model final/public/legacy projection but are + explicitly test-only and must be unique, disposable and counted separately. + Production write counts remain zero. +- The exact Python subscription is disabled only after durable authority CAS, + sink/projector acceptance and boundary reconciliation. Under current + `NO_GO_EXTERNAL`, this action is simulated only; the real V1 topology remains + unchanged. +- Rollback records the last accepted Rust watermark, fences Rust first, then + grants the immutable Python rollback owner a new revision/lease and resumes + from the next reconciled watermark. Restarting Python without authority is + forbidden. + +**Implementation tasks:** + +1. Add strict bounded-primary authorization that consumes a fresh exact + Phase 9.0-C `GO`, completed production canary hold evidence, candidate/bundle + identity and explicit slice approval. Keep isolated rehearsal a distinct + non-production type that cannot be converted to production authority. +2. Add immutable terminal-checkpoint and accepted-handoff contracts plus a + PostgreSQL migration. A database trigger must prevent direct or legacy + transition paths from entering `RUST_PRIMARY` or rollback `PYTHON_PRIMARY` + without accepted matching handoff evidence. +3. Extend the provider-neutral Rust authority core additively. Preserve Phase 8 + v1 and Phase 9.1 v2 decoders; add a v3 primary record/state machine with + exact `revision + 1`, strict owner/lease CAS, terminal boundary and rollback + transitions. +4. Add final-sink and compatibility-projector fencing with independent durable + watermark tracking per target. Authority changes between ACK and watermark + commit must fail closed and remain recoverable by deterministic replay. +5. Build an isolated replicated-broker certification over authentic frozen + provider frames. Exercise `N-1/N/N+1`, duplicate/out-of-order/gap input, + stale/zombie writer, CAS conflict, crash before/after CAS, sink/projector + restart, one-replica loss, below-min-ISR, full broker restart, slow consumer + and bounded rollback. +6. Verify exact Python/Rust canonical parity, output counts/order/digests, V1 + projected schema behavior and V2 snapshot/cursor/replay continuity. Measure + cutover/rollback RTO without turning the measurement into a production SLO. +7. Freeze machine/human evidence, checksums and a runbook; remove only + Phase 9.2 disposable resources. Record V1 topology/health before and after. + +**Verification and exit gate:** + +- Rust unit/contract tests cover every state transition, malformed checkpoint, + boundary off-by-one, stale owner/revision/lease/plan, wrong target, duplicate, + gap, ACK failure and rollback path. +- PostgreSQL migration tests prove transactionality, append-only evidence, + direct-primary bypass rejection, CAS conflict rejection and both handoff + directions. +- Replicated-broker rehearsal has one authoritative owner, zero unexplained + semantic mismatch, zero external duplicate/gap, ordered compatibility output + and recovery after process/broker failure. +- Full Python and Rust suites pass. V1 OpenAPI/health, container identity, + Redis namespaces and live subscription ownership remain unchanged. +- While Phase 9.0-C remains `NO_GO_EXTERNAL`, maximum closure is + `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED`. Production promotion + requires a fresh exact `GO`, successful real canary hold and explicit + operator approval; repository tests cannot fabricate those gates. + +**Implementation journal (2026-08-18):** + +- `COMPLETE` control and evidence contracts: production primary authorization + requires a fresh exact Phase 9.0-C `GO`, completed production canary hold, + immutable rollback manifest and explicit bounded-slice approval. Isolated + rehearsal remains a separate non-production authorization type. +- `COMPLETE` additive persistence boundary: migration `0007` adds immutable + terminal checkpoints and accepted handoffs, a primary/rollback bypass guard + and exact handoff-aware CAS. Disposable PostgreSQL smoke passed both + Python-to-Rust primary and Rust-to-Python rollback, rejected stale/direct CAS + and preserved evidence across idempotent migration replay. +- `COMPLETE` Rust authority/sink core: v1/v2 decoders remain intact; v3 binds + terminal checkpoint, accepted handoff, exact revision/lease/plan and + independently contiguous canonical/public/legacy target watermarks. Loading + an already-primary authority is fail-closed until each target reconstructs + its durable contiguous watermark. Crash-before-CAS reconstruction, duplicate + W rejection and exact W+1 restart are covered. Rust fmt/clippy pass and the + full workspace is 40/40; the focused Python matrix is 87/87 and the full + Python suite is 395/395 with 5 intentional skips. +- `COMPLETE` isolated replicated-broker certification: 25,600 authentic Binance + USD-M trade events across three clean Rust runs produced zero semantic + mismatch. The authority path completed `RUST_CANARY -> RUST_PRIMARY -> + BLOCKED -> ROLLBACK_PENDING -> PYTHON_PRIMARY`; canonical/public V2/legacy V1 + projections are identical and gap-free for watermarks 101..181. A fresh Rust + process recovered watermark 180 independently from all three durable targets, + rejected writes before restore and duplicate 180 after restore, then emitted + exactly 181. One-replica-loss ACK and below-min-ISR fail-closed passed. +- `COMPLETE` measured isolated operations and cleanup: cutover 22.200 ms, formal + rollback 533.237 ms and delayed-consumer catch-up 24.524 s. Production public + and legacy writes remained zero; V1 health stayed 200/200 with unchanged + topology; disposable containers/networks/volumes ended at 0/0/0. Migration + smoke passed both handoff directions and checksum verification passed. The + builder now carries pinned rustfmt/clippy, and the rebuilt runtime digest + exactly matches the certified image digest. +- `OPEN EXTERNAL GATE`: Phase 9.0-C production prerequisites and real canary + hold remain unavailable; V1 stays authoritative. + +**Rollback:** Before production authorization, remove only isolated Phase 9.2 +topics/groups/containers/networks/volumes and retain Phase 9.1 code/evidence. A +future production rollback must follow the formal protocol above and may never +use a direct owner flag or uncoordinated Python restart. #### 9.3 Hold, Close And Expand Independently -- Keep the slice in an enhanced-observation hold period. -- Close the rollback window only after all SLO, consumer and authority evidence - remains clean. -- Update the consumer/authority registries and freeze the production report. -- Decide independently whether to expand: - 1. more partitions/instruments of the same TRADE capability; - 2. BBO; - 3. L2 snapshot/delta/checksum; - 4. BAR lifecycle/revision; - 5. another venue/market. -- Repeat full capability/provider/chaos/capacity/rollback certification for each - expansion. Do not infer certification transitively. -- Remove a replaced Python hot-path implementation only after zero ownership, - tested rollback posture and approved repository cleanup. Reusable adapter and - compatibility knowledge remain until explicit decommission. +**Status:** `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED` + +**Purpose:** Add the provider-neutral post-primary control plane that observes +one exact `RUST_PRIMARY` slice, decides whether its rollback window may close, +freezes consumer/authority evidence and creates independent expansion +candidates. Phase 9.0-C remains `NO_GO_EXTERNAL` and Phase 9.2 is not production +authoritative, so this phase implements and certifies the protocol only in +isolated scope. It must not manufacture a production hold, close a real +rollback window, mutate production authority or grant another feed/venue +transitive approval. + +**Guide index:** + +- [Phase 9.3 hold/closure/expansion boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-i--phase-93-hold-closure-and-independent-expansion) +- [Authority state machine](#authority-state-machine) +- [Formal rollback protocol](#formal-rollback-protocol) +- [Verification matrix](#verification-matrix) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Decision boundary and invariants:** + +- Hold/closure is a control-plane lifecycle around an existing exact authority + record. It does not add a new data-plane authority state and does not weaken + `RUST_PRIMARY`, `BLOCKED`, `ROLLBACK_PENDING` or `PYTHON_PRIMARY` fencing. +- Every observation binds slice, candidate, owner, authority revision, lease, + partition-plan epoch and monotonically increasing time/watermark. A changed + identity ends the hold; evidence from another owner or epoch cannot be mixed. +- Correctness breaches have zero tolerance: semantic mismatch, open gap, + duplicate external write, accepted stale writer, authority ambiguity, durable + ACK failure, projection divergence, consumer checkpoint regression or + unexplained source-quality failure blocks closure. +- Capacity/freshness/lag thresholds are explicit policy. Observations are + append-only, ordered, bounded and sufficiently dense for the approved hold + duration. Missing intervals fail closed; they are not interpolated. +- Closing the rollback window is an immutable registry decision, not deletion + of the Python rollback manifest. It requires a production-authorized primary, + a completed real hold, exact healthy consumer and authority registries, a + fresh rollback rehearsal, explicit operator/change-ticket approval and an + unchanged authority CAS identity at commit time. +- Expansion never inherits authority or certification. More instruments, + `BBO`, `L2`, `BAR` and another venue/market each create a new candidate digest, + required capability matrix and independent Phase 6/9 certification set. +- Runtime decommission requires zero ownership, zero active rollback dependency, + a closed governed window and explicit repository cleanup approval. Shared + contracts, adapters and compatibility knowledge are retained unless a + separate approved removal proves no consumer dependency. +- With current external gates, the maximum valid result is + `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED`. V1 and the Python + authority path remain unchanged. + +**Implementation tasks:** + +1. Add typed hold policy, observation, decision, registry snapshot, closure + approval and expansion manifest contracts. Validate strict schemas, identity, + time ordering, zero-tolerance correctness fields and bounded thresholds. +2. Add a stateful hold evaluator with monotonic observation ordering, required + sample density, exact authority continuity and sticky fail-closed breach + behavior. Recovery requires a new hold identifier, never an in-place reset. +3. Add a production closure authorizer that rejects the current Phase 9.0-C + `NO_GO_EXTERNAL`/Phase 9.2 rehearsal evidence and accepts only exact real + production primary, hold, consumer, authority, rollback and operator records. +4. Add append-only PostgreSQL hold observations/decisions, authority closure and + expansion registries. A closure function must lock and recheck the exact + current authority record; it records closure only and never changes owner, + state, revision, lease or watermark. +5. Add independent expansion planning for more instruments/partitions, `BBO`, + `L2`, `BAR` and another venue/market. Persist + `INDEPENDENT_CERTIFICATION_REQUIRED`, forbid transitive evidence and bind a + new candidate/partition-plan identity where applicable. +6. Build disposable PostgreSQL migration smoke plus isolated control-plane + certification. Cover clean hold, sparse/out-of-order observations, every + zero-tolerance breach, threshold breach, stale authority CAS, registry + mismatch, duplicate closure, immutable evidence, all expansion classes and + decommission refusal. +7. Freeze machine/human evidence, checksum and operator runbook. Record V1 + health/topology before and after, production mutations as zero and remove + only Phase 9.3 disposable resources/images. + +**Verification and exit gate:** + +- Unit/contract tests cover valid and malformed hold observations, identity and + epoch drift, timestamp/watermark regression, missing density, correctness and + resource breaches, sticky blocking and deterministic decision digest. +- Closure authorization tests prove current evidence is denied, complete + production-shaped test fixtures are accepted only in test scope, and every + missing/stale/mismatched registry, rollback or approval field fails closed. +- PostgreSQL tests prove migration idempotency, append-only evidence, exact + authority row locking, stale CAS rejection, duplicate closure rejection and + zero authority-field mutation after closure. +- Expansion tests prove no transitive certification, capability-specific gate + requirements, unique candidate identity and zero write authority. +- Full Python/Rust compatibility suites pass. V1 health, API/SDK/Redis contracts, + running topology and authority ownership remain unchanged. +- While external gates remain unavailable, production hold duration, production + closure and real expansion authority remain explicitly untested and blocked. + +**Implementation journal (2026-08-19):** + +- `COMPLETE` plan boundary: scope, invariants, test matrix and Appendix I were + frozen before code changes on `feat/phase93-hold-close-expand`, stacked on + certified Phase 9.2. +- `COMPLETE` typed control contracts: strict hold policy/identity/observation/ + decision, frozen consumer and authority snapshots, rollback rehearsal, + operator closure approval, independent expansion manifests and decommission + assessment are implemented. Focused domain and migration-contract tests pass + 16/16, including every correctness/resource breach and current no-go denial. +- `COMPLETE` additive persistence and actual PostgreSQL smoke: migration `0008` + creates append-only hold, registry, rollback, approval, closure, expansion and + decommission records. Two holds, three observations, two decisions, one + closure, five expansion types and two decommission decisions passed. Closure + left authority exactly `RUST_PRIMARY:4:rust-primary:2:100`; approval and + closure UUIDs are distinct, the frozen closure digest binds every expansion, + and stale CAS, dirty pass, incomplete gates and all tested mutations failed + closed. Idempotent replay, stable-readiness startup and scoped container + cleanup passed. +- `COMPLETE` isolated certification and operator tooling: parent Phase 9.2 + provenance remains 25,600 authentic events with zero semantic mismatch; the + accelerated hold is explicitly `TEST_CONTROL_PLANE_FIXTURE` and has no + production authority. Current no-go is rejected, five expansion manifests + remain independently uncertified/write-disabled, Python decommission with a + rollback dependency is denied, and production mutations remain zero. V1 + health stayed 200/200 with unchanged topology. Focused Phase 9.3 tests pass + 20/20 and evidence checksums pass. +- `COMPLETE` import/runtime boundary: control-plane imports no longer eagerly + load the alpha SDK. Existing `PaperAlphaCanary` and `sdk_requirement` exports + remain API-compatible through lazy loading and passed container smoke. +- `COMPLETE` final compatibility and cleanup gates: full Python is 415/415 + with 5 intentional skips; Rust fmt/clippy pass and the full workspace is + 40/40. Final migration and certification reruns pass, evidence checksums pass, + V1 health remains 200 with unchanged topology, production mutations are zero, + no Phase 9.3 container/network/volume remains and the temporary Rust builder + image was removed without global prune. +- `COMPLETE` post-closure CI hotfix (2026-08-19): GitHub run + `32210830176` showed that PR #7 full unit tests passed, while the additive + PostgreSQL migration step failed after one transient ready probe immediately + preceded an init restart. `phase5_migration_smoke.sh` now requires eight + consecutive successful probes over two seconds before `createdb`. The exact + CI-image unit command passes 415/415 with 5 intentional skips and the failing + migration gate passes 3/3 consecutive runs. All disposable Compose/PostgreSQL + resources were removed; no schema, authority or running service changed. +- `COMPLETE` post-closure security hotfix (2026-08-19): PR #7 run + `32211508679` passed unit, migration, Redis, Rust artifact and performance + gates, then Trivy rejected nine Debian util-linux packages for fixed HIGH + `CVE-2026-53615`. The runtime stage now applies repository security upgrades, + installing `2.41.5-0+deb13u1` instead of vulnerable `2.41-5`. A pinned Trivy + 0.74.0 tar scan reports 0 HIGH/0 CRITICAL and the rebuilt exact CI image still + passes 415/415 unit tests with 5 intentional skips. No CVE was ignored or + allowlisted. +- `OPEN EXTERNAL GATE`: Phase 9.0-C remains `NO_GO_EXTERNAL`; no real primary + owner, production hold interval or operator closure approval exists. + +**Rollback:** Before production authority exists, remove only Phase 9.3 test +schemas, fixtures, evidence runtime and images; retain append-only repository +evidence. A future production closure cannot be reversed by deleting its row. +An incident still uses the formal `BLOCKED -> ROLLBACK_PENDING -> +PYTHON_PRIMARY` authority protocol and a new audit decision. ### Verification Matrix @@ -2786,7 +3559,13 @@ Completing one Rust primary slice does not mark every venue/feed complete. ### Completed -- Not started. User approval is required independently for each authority slice. +- Phase 9.0-A/9.0-B runtime and isolated V2 beta closure, Phase 9.0-C strict + prerequisite control plane, Phase 9.1 isolated canary, Phase 9.2 isolated + bounded-primary protocol and Phase 9.3 hold/closure/expansion control plane + are implemented and certified at their explicitly non-production scopes. +- No production authority slice, real hold closure, expansion or Python runtime + decommission is approved. Each requires independent operator approval and + production-scope evidence. ### Technical Debt / Decision Gate diff --git a/Dockerfile b/Dockerfile index a1e2f61..dc6654c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -21,6 +21,13 @@ FROM python:3.12-slim AS runtime ARG QDL_UID=10001 ARG QDL_GID=10001 +ARG QDL_GIT_SHA=unknown +ARG QDL_RELEASE=development + +LABEL org.opencontainers.image.title="Quant Data Layer" \ + org.opencontainers.image.revision="${QDL_GIT_SHA}" \ + org.opencontainers.image.version="${QDL_RELEASE}" \ + org.opencontainers.image.source="https://github.com/BobbyAxerol/quant-data-layer" ENV PYTHONUNBUFFERED=1 ENV PYTHONDONTWRITEBYTECODE=1 @@ -33,7 +40,10 @@ ENV PATH=/opt/venv/bin:$PATH WORKDIR /app -RUN groupadd --gid ${QDL_GID} qdl && \ +RUN apt-get update && \ + apt-get upgrade -y --no-install-recommends && \ + rm -rf /var/lib/apt/lists/* && \ + groupadd --gid ${QDL_GID} qdl && \ useradd --uid ${QDL_UID} --gid ${QDL_GID} --create-home \ --home-dir /home/qdl --shell /usr/sbin/nologin qdl && \ install -d -o qdl -g qdl -m 0750 /home/qdl/.cache/matplotlib diff --git a/Dockerfile.phase8-rust b/Dockerfile.phase8-rust index 748d285..cf25fd6 100644 --- a/Dockerfile.phase8-rust +++ b/Dockerfile.phase8-rust @@ -5,6 +5,7 @@ RUN apt-get update \ clang cmake libclang-dev libcurl4-openssl-dev libssl-dev libzstd-dev \ make pkg-config zlib1g-dev \ && rm -rf /var/lib/apt/lists/* +RUN rustup component add rustfmt clippy WORKDIR /src COPY Cargo.toml Cargo.lock ./ COPY rust ./rust @@ -12,6 +13,8 @@ COPY generated/rust ./generated/rust RUN cargo build --release --locked \ --bin qdl-kafka-smoke \ --bin qdl-authority-rehearsal \ + --bin qdl-phase91-canary-rehearsal \ + --bin qdl-phase92-primary-rehearsal \ --bin qdl-binance-shadow \ --bin qdl-parity-replay \ --bin qdl-venue-core-certify @@ -30,6 +33,8 @@ RUN apt-get update \ && useradd --uid 10001 --gid 10001 --no-create-home --shell /usr/sbin/nologin qdl COPY --from=builder /src/target/release/qdl-kafka-smoke /usr/local/bin/qdl-kafka-smoke COPY --from=builder /src/target/release/qdl-authority-rehearsal /usr/local/bin/qdl-authority-rehearsal +COPY --from=builder /src/target/release/qdl-phase91-canary-rehearsal /usr/local/bin/qdl-phase91-canary-rehearsal +COPY --from=builder /src/target/release/qdl-phase92-primary-rehearsal /usr/local/bin/qdl-phase92-primary-rehearsal COPY --from=builder /src/target/release/qdl-binance-shadow /usr/local/bin/qdl-binance-shadow COPY --from=builder /src/target/release/qdl-parity-replay /usr/local/bin/qdl-parity-replay COPY --from=builder /src/target/release/qdl-venue-core-certify /usr/local/bin/qdl-venue-core-certify diff --git a/Makefile b/Makefile index 3d4b73a..00ce232 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: contract-check contract-generate phase2-benchmark phase2-redis-smoke phase2-test phase3-lease-smoke phase3-load-smoke phase3-real-provider-smoke phase3-rust-smoke phase3-test phase4-dnse-real-smoke phase4-history-test phase4-migration-smoke phase4-okx-real-smoke phase4-okx-test phase4-replay-test phase4-test phase4-vn-shadow-smoke phase45-build phase45-clean phase45-dependency-audit phase45-provider-smoke phase45-test phase5-api-test phase5-build phase5-clean phase5-contract-check phase5-dependency-audit phase5-load phase5-migration-smoke phase5-real-provider-smoke phase5-test phase7-build phase7-clean phase7-contract-check phase7-migration-smoke phase7-test phase71-topology-test phase71-test phase72-test phase72-topology-test phase73-test phase73-certify phase80-test phase80-certify phase81-test phase81-certify phase82-test phase82-dnse-acquire phase82-certify phase83-test phase83-build phase83-authority phase83-release-capacity phase83-freeze python-test rust-test +.PHONY: contract-check contract-generate phase2-benchmark phase2-redis-smoke phase2-test phase3-lease-smoke phase3-load-smoke phase3-real-provider-smoke phase3-rust-smoke phase3-test phase4-dnse-real-smoke phase4-history-test phase4-migration-smoke phase4-okx-real-smoke phase4-okx-test phase4-replay-test phase4-test phase4-vn-shadow-smoke phase45-build phase45-clean phase45-dependency-audit phase45-provider-smoke phase45-test phase5-api-test phase5-build phase5-clean phase5-contract-check phase5-dependency-audit phase5-load phase5-migration-smoke phase5-real-provider-smoke phase5-test phase7-build phase7-clean phase7-contract-check phase7-migration-smoke phase7-test phase71-topology-test phase71-test phase72-test phase72-topology-test phase73-test phase73-certify phase80-test phase80-certify phase81-test phase81-certify phase82-test phase82-dnse-acquire phase82-certify phase83-test phase83-build phase83-authority phase83-release-capacity phase83-freeze phase90b-build phase90b-test phase90b-certify phase90b-clean phase90c-build phase90c-test phase90c-migration phase90c-certify phase90c-clean python-test rust-test phase91-build phase91-test phase91-certify phase91-clean phase92-build phase92-test phase92-migration phase92-certify phase92-clean phase93-test phase93-migration phase93-certify phase93-clean BUF_IMAGE ?= bufbuild/buf:1.50.0 RUST_IMAGE ?= rust:1.82-slim@sha256:1111c28d995d06a7863ba6cea3b3dcb87bebe65af8ec5517caaf2c8c26f38010 @@ -8,6 +8,18 @@ PHASE7_TEST_IMAGE ?= data-layer:phase7-test PHASE8_RUST_IMAGE ?= qdl-phase8-rust:phase8-candidate PHASE8_RELEASE ?= phase8-rust-realtime-core-v0.1.0-beta PHASE8_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE90B_IMAGE ?= data-layer:phase90b-candidate +PHASE90B_RELEASE ?= phase90b-isolated-v2-beta +PHASE90B_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE90C_IMAGE ?= data-layer:phase90c-test +PHASE90C_RELEASE ?= phase90c-production-prerequisites +PHASE90C_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE91_RUST_IMAGE ?= qdl-phase91-rust:rehearsal +PHASE91_BUILDER_IMAGE ?= qdl-phase91-rust-builder:rehearsal +PHASE91_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE92_RUST_IMAGE ?= qdl-phase92-rust:rehearsal +PHASE92_BUILDER_IMAGE ?= qdl-phase92-rust-builder:rehearsal +PHASE92_GIT_SHA ?= $(shell git rev-parse HEAD) contract-generate: docker run --rm -v "$(CURDIR):/workspace" -w /workspace/contracts $(BUF_IMAGE) generate @@ -210,3 +222,92 @@ phase83-freeze: phase7-clean: docker image rm $(PHASE7_TEST_IMAGE) 2>/dev/null || true + +phase90b-build: + docker build --provenance=false --build-arg QDL_GIT_SHA=$(PHASE90B_GIT_SHA) --build-arg QDL_RELEASE=$(PHASE90B_RELEASE) -t $(PHASE90B_IMAGE) . + +phase90b-test: phase90b-build + docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.5 --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=128m,uid=10001,gid=10001 --tmpfs /app/logs:rw,noexec,nosuid,nodev,size=16m,uid=10001,gid=10001 $(PHASE90B_IMAGE) python -m unittest -v tests.test_phase90b_isolated_beta tests.test_phase90a_runtime_correctness tests.test_fund_phase73_beta_decision tests.test_fund_phase72_consumer_canary tests.test_fund_phase71_beta_runtime tests.test_fund_phase7_contract_security tests.test_phase0_contract_golden tests.test_fund_phase5_contracts + docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.5 --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=128m,uid=10001,gid=10001 --tmpfs /app/logs:rw,noexec,nosuid,nodev,size=16m,uid=10001,gid=10001 $(PHASE90B_IMAGE) python -m unittest discover -s tests + +phase90b-certify: phase90b-test + docker image inspect redis:7.2-alpine >/dev/null 2>&1 || docker pull redis:7.2-alpine + QDL_PHASE90B_IMAGE=$(PHASE90B_IMAGE) scripts/phase90b_isolated_beta_certification.sh + +phase90b-clean: + docker image rm $(PHASE90B_IMAGE) 2>/dev/null || true + +phase90c-build: + docker build --provenance=false --build-arg QDL_GIT_SHA=$(PHASE90C_GIT_SHA) --build-arg QDL_RELEASE=$(PHASE90C_RELEASE) -t $(PHASE90C_IMAGE) . + +phase90c-test: phase90c-build + docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.5 --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=128m,uid=10001,gid=10001 --tmpfs /app/logs:rw,noexec,nosuid,nodev,size=16m,uid=10001,gid=10001 $(PHASE90C_IMAGE) python -m unittest -v tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract tests.test_phase90b_isolated_beta tests.test_fund_phase80_broker_substrate tests.test_fund_phase83_release + +phase90c-migration: + scripts/phase90c_migration_smoke.sh + +phase90c-certify: phase90c-test phase90c-migration + python3 scripts/phase90c_prerequisite_certification.py --expect NO_GO_EXTERNAL + sha256sum upgrade/evidence/phase90c-production-prerequisites.json upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md upgrade/evidence/phase90c-authority-migration.json > upgrade/evidence/phase90c-evidence.sha256 + +phase90c-clean: + docker image rm $(PHASE90C_IMAGE) 2>/dev/null || true + + +phase91-build: + docker build --provenance=false --target builder -f Dockerfile.phase8-rust -t $(PHASE91_BUILDER_IMAGE) . + docker build --provenance=false -f Dockerfile.phase8-rust --build-arg QDL_GIT_SHA=$(PHASE91_GIT_SHA) --build-arg QDL_RELEASE=phase91-rust-canary-rehearsal -t $(PHASE91_RUST_IMAGE) . + +phase91-test: phase91-build + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE91_BUILDER_IMAGE) cargo fmt --all -- --check + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE91_BUILDER_IMAGE) cargo clippy --workspace --all-targets --locked -- -D warnings + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE91_BUILDER_IMAGE) cargo test --workspace --locked + docker compose run --rm --no-deps test_runner python -m unittest tests.test_phase91_canary_control tests.test_phase91_certification_contract tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract tests.test_phase90b_isolated_beta tests.test_phase90a_runtime_correctness tests.test_fund_phase80_broker_substrate tests.test_fund_phase81_raw_core tests.test_fund_phase82_conformance tests.test_fund_phase83_release + docker compose run --rm --no-deps test_runner python -m unittest discover tests + +phase91-certify: phase91-test + python3 scripts/phase91_canary_certification.py --rust-image $(PHASE91_RUST_IMAGE) --repeat 200 + sha256sum -c upgrade/evidence/phase91-evidence.sha256 + +phase91-clean: + docker compose --project-name qdl_phase91_certification -f docker-compose.phase8-kafka.yml down -v --remove-orphans + docker image rm $(PHASE91_RUST_IMAGE) 2>/dev/null || true + docker image rm $(PHASE91_BUILDER_IMAGE) 2>/dev/null || true + +phase92-build: + docker build --provenance=false --target builder -f Dockerfile.phase8-rust -t $(PHASE92_BUILDER_IMAGE) . + docker build --provenance=false -f Dockerfile.phase8-rust --build-arg QDL_GIT_SHA=$(PHASE92_GIT_SHA) --build-arg QDL_RELEASE=phase92-bounded-primary-rehearsal -t $(PHASE92_RUST_IMAGE) . + +phase92-test: phase92-build + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE92_BUILDER_IMAGE) cargo fmt --all -- --check + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE92_BUILDER_IMAGE) cargo clippy --workspace --all-targets --locked -- -D warnings + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE92_BUILDER_IMAGE) cargo test --workspace --locked + docker compose run --rm --no-deps test_runner python -m unittest tests.test_phase92_primary_control tests.test_phase92_migration_contract tests.test_phase92_certification_contract tests.test_phase91_canary_control tests.test_phase91_certification_contract tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract tests.test_phase90b_isolated_beta tests.test_phase90a_runtime_correctness tests.test_fund_phase80_broker_substrate tests.test_fund_phase81_raw_core tests.test_fund_phase82_conformance tests.test_fund_phase83_release + docker compose run --rm --no-deps test_runner python -m unittest discover tests + +phase92-migration: + scripts/phase92_migration_smoke.sh + +phase92-certify: phase92-test phase92-migration + python3 scripts/phase92_primary_certification.py --rust-image $(PHASE92_RUST_IMAGE) --repeat 200 + sha256sum -c upgrade/evidence/phase92-evidence.sha256 + +phase92-clean: + docker compose --project-name qdl_phase92_certification -f docker-compose.phase8-kafka.yml down -v --remove-orphans + docker image rm $(PHASE92_RUST_IMAGE) 2>/dev/null || true + docker image rm $(PHASE92_BUILDER_IMAGE) 2>/dev/null || true + + +phase93-test: + docker compose run --rm --no-deps test_runner python -m unittest tests.test_phase93_hold_close_expand tests.test_phase93_migration_contract tests.test_phase93_certification_contract tests.test_phase92_primary_control tests.test_phase92_migration_contract tests.test_phase92_certification_contract tests.test_phase91_canary_control tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract + docker compose run --rm --no-deps test_runner python -m unittest discover tests + +phase93-migration: + python3 scripts/phase93_migration_smoke.py + +phase93-certify: phase93-test phase93-migration + python3 scripts/phase93_hold_close_certification.py + sha256sum -c upgrade/evidence/phase93-evidence.sha256 + +phase93-clean: + docker ps -aq --filter "name=qdl_phase93_" | xargs -r docker rm -f diff --git a/app/api/context.py b/app/api/context.py index 9e4ca55..34d64ed 100644 --- a/app/api/context.py +++ b/app/api/context.py @@ -11,6 +11,7 @@ class DataLayerContext: redis_cache: Any binance_stream_supervisor: Any get_dnse_stream_manager: Callable[[], Any] + get_kline_recovery_manager: Callable[[], Any] = lambda: None demand_registry: Any = None preload_topup_coordinator: Any = None diff --git a/app/api/routes_health.py b/app/api/routes_health.py index c1acd2f..3209772 100644 --- a/app/api/routes_health.py +++ b/app/api/routes_health.py @@ -16,6 +16,20 @@ def _binance_demand_keys(demands: dict) -> set[str]: } +def _feed_data_ready(stream: dict, feed: str) -> bool: + states = [ + state + for state in stream.get("sources", {}).values() + if state.get("feed") == feed + ] + return bool(states) and all(bool(state.get("data_ready")) for state in states) + + +def _recovery_snapshot(ctx: DataLayerContext) -> dict: + manager = ctx.get_kline_recovery_manager() + return manager.snapshot() if manager else {"enabled": False, "status": "not_started"} + + @router.get("/health") async def health(ctx: DataLayerContext = Depends(get_context)): redis_ok = await ctx.redis_cache.health_check() @@ -23,6 +37,7 @@ async def health(ctx: DataLayerContext = Depends(get_context)): binance_stream = ctx.binance_stream_supervisor.snapshot( demanded_feed_keys=_binance_demand_keys(demands), ) + binance_stream["kline_recovery"] = _recovery_snapshot(ctx) dnse_manager = ctx.get_dnse_stream_manager() dnse_stream = ( dnse_manager.get_status() @@ -36,8 +51,8 @@ async def health(ctx: DataLayerContext = Depends(get_context)): return { "status": "ok" if redis_ok and stream_ok else "degraded", "redis": redis_ok, - "binance_trade_stream": binance_stream.get("status") in {"ok", "starting"}, - "binance_kline_stream": binance_stream.get("status") in {"ok", "starting"}, + "binance_trade_stream": _feed_data_ready(binance_stream, "trade"), + "binance_kline_stream": _feed_data_ready(binance_stream, "kline"), "binance_stream": binance_stream, "feed_demands": demands, "preload_topup": ctx.preload_topup_coordinator.snapshot(), @@ -49,10 +64,12 @@ async def health(ctx: DataLayerContext = Depends(get_context)): async def health_streams(ctx: DataLayerContext = Depends(get_context)): dnse_manager = ctx.get_dnse_stream_manager() demands = await ctx.demand_registry.snapshot() + binance_stream = ctx.binance_stream_supervisor.snapshot( + demanded_feed_keys=_binance_demand_keys(demands), + ) + binance_stream["kline_recovery"] = _recovery_snapshot(ctx) return { - "binance_stream": ctx.binance_stream_supervisor.snapshot( - demanded_feed_keys=_binance_demand_keys(demands), - ), + "binance_stream": binance_stream, "feed_demands": demands, "dnse_stream": ( dnse_manager.get_status() diff --git a/app/config.py b/app/config.py index 98a4cf9..891e9c8 100644 --- a/app/config.py +++ b/app/config.py @@ -39,6 +39,17 @@ BINANCE_SPOT_SYMBOLS_FILE = os.getenv("BINANCE_SPOT_SYMBOLS_FILE", "/app/symbols_spot.json") STREAM_STALE_SECONDS = float(os.getenv("STREAM_STALE_SECONDS", "180")) STREAM_STRICT_FEED_HEALTH = os.getenv("STREAM_STRICT_FEED_HEALTH", "false").lower() in {"1", "true", "yes", "on"} +BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS = float( + os.getenv("BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS", "15") +) +BINANCE_WS_IDLE_TIMEOUT_SECONDS = float(os.getenv("BINANCE_WS_IDLE_TIMEOUT_SECONDS", "90")) +BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS = float( + os.getenv("BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS", "2") +) +BINANCE_KLINE_RECOVERY_ENABLED = os.getenv("BINANCE_KLINE_RECOVERY_ENABLED", "true").lower() in {"1", "true", "yes", "on"} +BINANCE_KLINE_RECOVERY_POLL_SECONDS = float(os.getenv("BINANCE_KLINE_RECOVERY_POLL_SECONDS", "2")) +BINANCE_KLINE_RECOVERY_SETTLE_SECONDS = float(os.getenv("BINANCE_KLINE_RECOVERY_SETTLE_SECONDS", "1")) +BINANCE_KLINE_RECOVERY_CONCURRENCY = int(os.getenv("BINANCE_KLINE_RECOVERY_CONCURRENCY", "4")) # ── Provider fallback ─────────────────────────── OKX_FALLBACK_ENABLED = os.getenv("OKX_FALLBACK_ENABLED", "true").lower() in {"1", "true", "yes", "on"} diff --git a/app/main.py b/app/main.py index 41cb126..3fa1adc 100644 --- a/app/main.py +++ b/app/main.py @@ -28,6 +28,13 @@ from app.config import ( PRELOAD_DAILY_RUN_TIME, PRELOAD_DIR, + BINANCE_WS_QUEUE_MAXSIZE, + BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, + BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, + BINANCE_KLINE_RECOVERY_ENABLED, + BINANCE_KLINE_RECOVERY_POLL_SECONDS, + BINANCE_KLINE_RECOVERY_SETTLE_SECONDS, + BINANCE_KLINE_RECOVERY_CONCURRENCY, STREAM_STALE_SECONDS, STREAM_STRICT_FEED_HEALTH, ) @@ -43,6 +50,7 @@ ) from app.cache.redis_cache import RedisCache from app.stream.async_live_feed import start_stream +from app.stream.kline_recovery import DemandKlineRecovery, KlineRecoveryConfig from app.ingestion.supervisor import StreamSupervisor from app.stream.vnstock_poller import VnstockPoller from app.stream.dnse_ws import DnseStreamManager @@ -74,9 +82,12 @@ preload_thread = None preload_stop_event = threading.Event() unified_stream_task = None +kline_recovery_manager = None +kline_recovery_task = None binance_stream_supervisor = StreamSupervisor( stale_after_seconds=STREAM_STALE_SECONDS, strict_feed_health=STREAM_STRICT_FEED_HEALTH, + first_frame_timeout_seconds=BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, ) preload_daily_state_dir = os.path.join(os.path.dirname(PRELOAD_DIR), "_state") @@ -261,14 +272,18 @@ async def lifespan(app: FastAPI): - Start the vnstock REST poller (FALLBACK only if DNSE stale) """ global dnse_stream_manager, unified_stream_task, binance_stream_supervisor + global kline_recovery_manager, kline_recovery_task logger.info("=== data_layer service starting ===") await redis_cache.init_ping() runtime_sources = RuntimeSourceConfig.from_env() logger.info("Runtime source configuration: %s", runtime_sources.public_summary()) - # 1. Unified stream (Binance) + # 1. Unified stream (Binance) and demand-only closed-kline recovery. + kline_recovery_manager = None + kline_recovery_task = None if runtime_sources.binance_sources: + stream_queue = asyncio.Queue(maxsize=BINANCE_WS_QUEUE_MAXSIZE) unified_stream_task = asyncio.create_task( start_stream( redis_cache, @@ -276,8 +291,26 @@ async def lifespan(app: FastAPI): supervisor=binance_stream_supervisor, demand_registry=demand_registry, enabled_sources=list(runtime_sources.binance_sources), + queue=stream_queue, ) ) + if ( + BINANCE_KLINE_RECOVERY_ENABLED + and "binance_futures_kline" in runtime_sources.binance_sources + ): + kline_recovery_manager = DemandKlineRecovery( + queue=stream_queue, + redis_cache=redis_cache, + demand_registry=demand_registry, + config=KlineRecoveryConfig( + enabled=True, + poll_seconds=BINANCE_KLINE_RECOVERY_POLL_SECONDS, + settle_seconds=BINANCE_KLINE_RECOVERY_SETTLE_SECONDS, + concurrency=BINANCE_KLINE_RECOVERY_CONCURRENCY, + queue_put_timeout_seconds=BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, + ), + ) + kline_recovery_task = asyncio.create_task(kline_recovery_manager.run()) logger.info("Unified async Binance streams started: %s", runtime_sources.binance_sources) else: unified_stream_task = None @@ -333,8 +366,13 @@ async def lifespan(app: FastAPI): # Shutdown logger.info("Shutting down services...") - if unified_stream_task: - unified_stream_task.cancel() + async_tasks = [ + task for task in (kline_recovery_task, unified_stream_task) if task is not None + ] + for task in async_tasks: + task.cancel() + if async_tasks: + await asyncio.gather(*async_tasks, return_exceptions=True) if vn_poller: vn_poller.stop() @@ -366,6 +404,7 @@ async def lifespan(app: FastAPI): redis_cache=redis_cache, binance_stream_supervisor=binance_stream_supervisor, get_dnse_stream_manager=lambda: dnse_stream_manager, + get_kline_recovery_manager=lambda: kline_recovery_manager, demand_registry=demand_registry, preload_topup_coordinator=preload_topup_coordinator, ) diff --git a/app/stream/async_live_feed.py b/app/stream/async_live_feed.py index 0467063..0dc10b9 100644 --- a/app/stream/async_live_feed.py +++ b/app/stream/async_live_feed.py @@ -5,8 +5,6 @@ import time import random import os -import math -import re import requests from websockets.asyncio.client import connect as websocket_connect @@ -16,94 +14,18 @@ from app.stream.feed_builder import build_urls, validate_symbols from app.stream.feed_parsers import PARSERS from app.stream.supervisor import StreamSupervisor -from app.stream.demand_registry import feed_key_for -from app.providers.binance import rest as binance_rest from app.config import ( BINANCE_SPOT_SYMBOLS_FILE, BINANCE_WS_BATCH_SIZE, BINANCE_WS_MAX_CONNS_PER_SOURCE, BINANCE_WS_QUEUE_MAXSIZE, + BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, + BINANCE_WS_IDLE_TIMEOUT_SECONDS, + BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, ) logger = logging.getLogger(__name__) -_STREAM_SYMBOL_RE = re.compile(r"[=/]([a-z0-9_]+)@(kline_[^/]+|trade)") - - -def symbols_from_stream_url(url: str) -> list[str]: - return [match.group(1).upper() for match in _STREAM_SYMBOL_RE.finditer(url)] - - -def _interval_seconds(interval: str) -> int: - unit = interval[-1] - value = int(interval[:-1]) - return value * {"s": 1, "m": 60, "h": 3600, "d": 86400}.get(unit, 60) - - -def _rest_kline_event(symbol: str, interval: str, row: list) -> dict: - return { - "e": "kline_recovery", - "E": int(time.time() * 1000), - "s": symbol, - "k": { - "t": row[0], "T": row[6], "s": symbol, "i": interval, - "o": row[1], "c": row[4], "h": row[2], "l": row[3], "v": row[5], "x": True, - }, - "recovery_source": "BINANCE_REST_GAP_FILL", - } - - -async def recover_demanded_kline_gap( - *, - source: str, - url: str, - interval: str, - queue: asyncio.Queue, - supervisor: StreamSupervisor, - shard_id: str, - demand_registry, -) -> int: - if source.endswith("_trade") or "kline" not in url: - return 0 - active = await demand_registry.snapshot() - demanded = set(active["feed_keys"]) - symbols = [ - symbol for symbol in symbols_from_stream_url(url) - if feed_key_for(source, "kline", symbol, interval) in demanded - ] - if not symbols: - return 0 - supervisor.record_gap_detected(shard_id) - shard = supervisor.shards.get(shard_id) - outage_seconds = float(shard.last_outage_seconds or 0) if shard else 0.0 - limit = min(1000, max(3, math.ceil(outage_seconds / max(1, _interval_seconds(interval))) + 2)) - market = "usdm" if source.startswith("binance_futures") else "spot" - recovered = 0 - now_ms = int(time.time() * 1000) - try: - for symbol in symbols: - payload = await asyncio.to_thread( - binance_rest.fetch_klines, - symbol, - interval, - limit, - None, - None, - market, - ) - for row in payload.get("data") or []: - if len(row) <= 6 or int(row[6]) > now_ms: - continue - await queue.put((source, _rest_kline_event(symbol, interval, row))) - recovered += 1 - supervisor.record_gap_fill(shard_id, success=True) - return recovered - except Exception: - supervisor.record_gap_fill(shard_id, success=False) - logger.exception("[WS] demanded kline gap-fill failed source=%s shard=%s", source, shard_id) - return 0 - - def get_spot_symbols( file_path: str = BINANCE_SPOT_SYMBOLS_FILE, *, @@ -157,10 +79,10 @@ def get_parser_key(source: str): def coalesce_redis_items(items: list[dict]) -> list[dict]: """Keep only the latest event per Redis key/channel within a publisher batch.""" - coalesced: dict[tuple[str, str], dict] = {} - order: list[tuple[str, str]] = [] + coalesced: dict[tuple[str, str, str | None], dict] = {} + order: list[tuple[str, str, str | None]] = [] for item in items: - dedupe_key = (item.get("key"), item.get("channel")) + dedupe_key = (item.get("key"), item.get("channel"), item.get("coalesce_id")) if dedupe_key not in coalesced: order.append(dedupe_key) coalesced[dedupe_key] = item @@ -176,6 +98,65 @@ def _source_market_namespace(source: str) -> str | None: return None +class StreamDataTimeout(RuntimeError): + """A connected websocket failed to produce valid provider data in time.""" + + +class StreamBackpressureTimeout(RuntimeError): + """The bounded publisher queue could not accept a provider event in time.""" + + +def valid_provider_frame(source: str, item: object, interval: str = "1m") -> bool: + """Return true only for a complete frame belonging to the configured feed.""" + if not isinstance(item, dict): + return False + if source.endswith("_trade"): + required = ("s", "p", "q", "t", "T") + return item.get("e") == "trade" and all(item.get(field) is not None for field in required) + if source.endswith("_kline"): + kline = item.get("k") + required = ("s", "i", "t", "T", "o", "h", "l", "c", "v", "x") + return ( + item.get("e") == "kline" + and isinstance(kline, dict) + and str(kline.get("i")) == interval + and all(kline.get(field) is not None for field in required) + ) + return False + + +def provider_items(source: str, payload: object, interval: str = "1m") -> list[dict]: + """Extract validated data frames while ignoring subscription ACKs.""" + if not isinstance(payload, dict): + raise ValueError("websocket payload must be an object") + if "data" not in payload and "id" in payload and "result" in payload: + return [] + data = payload.get("data", payload) + items = data if isinstance(data, list) else [data] + if not items or not all(valid_provider_frame(source, item, interval) for item in items): + raise ValueError(f"invalid or wrong-feed provider frame for {source}") + return items + + +async def _put_provider_item( + queue: asyncio.Queue, + item: tuple[str, dict], + *, + supervisor: StreamSupervisor, + timeout_seconds: float, +) -> None: + try: + queue.put_nowait(item) + except asyncio.QueueFull: + supervisor.record_queue_pressure() + try: + await asyncio.wait_for(queue.put(item), timeout=max(0.05, timeout_seconds)) + except asyncio.TimeoutError as exc: + raise StreamBackpressureTimeout( + f"publisher queue remained full for {timeout_seconds:.3f}s" + ) from exc + + async def handle_ws( url: str, queue: asyncio.Queue, @@ -187,80 +168,72 @@ async def handle_ws( max_backoff: int = 300, demand_registry=None, interval: str = "1m", + first_frame_timeout_seconds: float = BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, + idle_timeout_seconds: float = BINANCE_WS_IDLE_TIMEOUT_SECONDS, + queue_put_timeout_seconds: float = BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, ): - """ - Connect and stream messages from websocket. Reconnects on error. - parser_mode: "raw" (default) forwards original payload; "unified" forwards parsed data. - """ + """Receive valid provider frames with bounded readiness and backpressure.""" + del demand_registry # Kept in the public signature for V1 caller compatibility. backoff = reconnect_delay last_connected_at = None while True: try: supervisor.mark_connecting(shard_id) - logger.info(f"[WS] Connecting {source} -> {url[:60]}...") + logger.info("[WS] Connecting %s -> %s...", source, url[:60]) async with websocket_connect(url, ping_interval=30, max_size=None) as ws: - logger.info(f"[WS] Connected {source}") - recovered = supervisor.mark_connected(shard_id) + logger.info("[WS] Connected %s", source) + supervisor.mark_connected(shard_id) last_connected_at = time.monotonic() - if recovered and demand_registry is not None: - filled = await recover_demanded_kline_gap( - source=source, - url=url, - interval=interval, - queue=queue, - supervisor=supervisor, - shard_id=shard_id, - demand_registry=demand_registry, + session_has_valid_frame = False + + while True: + timeout = ( + idle_timeout_seconds + if session_has_valid_frame + else first_frame_timeout_seconds ) - if filled: - logger.info( - "[WS] demanded kline gap-fill complete source=%s shard=%s rows=%s", - source, - shard_id, - filled, - ) + try: + msg = await asyncio.wait_for(ws.recv(), timeout=max(0.05, timeout)) + except asyncio.TimeoutError as exc: + reason = "idle" if session_has_valid_frame else "first_frame" + supervisor.mark_data_timeout(shard_id, reason) + raise StreamDataTimeout( + f"{source} {reason} timeout after {timeout:.3f}s" + ) from exc - async for msg in ws: try: - supervisor.mark_message(shard_id) - payload = json.loads(msg) - data = payload.get("data") or payload.get("result") or payload - if not data: + raw_items = provider_items(source, json.loads(msg), interval) + if not raw_items: continue + supervisor.mark_message(shard_id) + session_has_valid_frame = True + backoff = reconnect_delay - output = data - if parser_mode == "unified": - parser_key = get_parser_key(source) - if parser_key: - parser = PARSERS.get(parser_key) + for data in raw_items: + output = data + if parser_mode == "unified": + parser = PARSERS.get(get_parser_key(source) or "") if parser: parsed = parser(data) if parsed is None: continue output = parsed - - items = output if isinstance(output, list) else [output] - for item in items: - # We put to the queue for the Redis publisher - # Structure: (source, item) - try: - queue.put_nowait((source, item)) - except asyncio.QueueFull: - # Pop one to make room - try: - queue.get_nowait() - queue.task_done() - except asyncio.QueueEmpty: - pass - supervisor.record_queue_drop(source, shard_id) - queue.put_nowait((source, item)) - - except Exception as e: - supervisor.mark_parse_error(shard_id, e) - logger.error(f"[WS] parse error: {e}") - - except InvalidStatus as e: - status = getattr(getattr(e, "response", None), "status_code", None) + items = output if isinstance(output, list) else [output] + for item in items: + await _put_provider_item( + queue, + (source, item), + supervisor=supervisor, + timeout_seconds=queue_put_timeout_seconds, + ) + except StreamBackpressureTimeout: + supervisor.mark_data_timeout(shard_id, "publisher_backpressure") + raise + except (TypeError, ValueError, json.JSONDecodeError) as exc: + supervisor.mark_parse_error(shard_id, exc) + logger.warning("[WS] rejected provider frame source=%s error=%s", source, exc) + except InvalidStatus as exc: + status = getattr(getattr(exc, "response", None), "status_code", None) elapsed = (time.monotonic() - last_connected_at) if last_connected_at else 0 if status == 429: backoff = min(max(backoff * 2, reconnect_delay * 2), max_backoff) @@ -269,28 +242,37 @@ async def handle_ws( if elapsed >= 60: backoff = reconnect_delay sleep_for = backoff + random.uniform(0, min(5, backoff * 0.2)) - supervisor.mark_reconnect(shard_id, e) - logger.error(f"[WS] connection rejected ({source}): HTTP {status}. Reconnecting in {int(sleep_for)}s...") + supervisor.mark_reconnect(shard_id, exc) + logger.error( + "[WS] connection rejected (%s): HTTP %s. Reconnecting in %ss...", + source, + status, + int(sleep_for), + ) await asyncio.sleep(sleep_for) - except (ConnectionClosedError, ConnectionClosedOK) as e: + except (ConnectionClosedError, ConnectionClosedOK) as exc: elapsed = (time.monotonic() - last_connected_at) if last_connected_at else 0 - if elapsed >= 60: - backoff = reconnect_delay - else: - backoff = min(backoff + 5, max_backoff) + backoff = reconnect_delay if elapsed >= 60 else min(backoff + 5, max_backoff) sleep_for = backoff + random.uniform(0, min(5, backoff * 0.2)) - supervisor.mark_reconnect(shard_id, e) - logger.error(f"[WS] connection closed ({source}): {e}. Reconnecting in {int(sleep_for)}s...") + supervisor.mark_reconnect(shard_id, exc) + logger.error( + "[WS] connection closed (%s): %s. Reconnecting in %ss...", + source, + exc, + int(sleep_for), + ) await asyncio.sleep(sleep_for) - except Exception as e: + except Exception as exc: elapsed = (time.monotonic() - last_connected_at) if last_connected_at else 0 - if elapsed >= 60: - backoff = reconnect_delay - else: - backoff = min(backoff + 5, max_backoff) + backoff = reconnect_delay if elapsed >= 60 else min(backoff + 5, max_backoff) sleep_for = backoff + random.uniform(0, min(5, backoff * 0.2)) - supervisor.mark_reconnect(shard_id, e) - logger.error(f"[WS] connection error ({source}): {e}. Reconnecting in {int(sleep_for)}s...") + supervisor.mark_reconnect(shard_id, exc) + logger.error( + "[WS] connection error (%s): %s. Reconnecting in %ss...", + source, + exc, + int(sleep_for), + ) await asyncio.sleep(sleep_for) @@ -387,11 +369,20 @@ async def redis_publisher_task( continue sym = sym.upper() - key = f"kline:{interval}:{sym}" - channel = f"stream:kline:{interval}:{sym}" - redis_items.append( - {"key": key, "channel": channel, "data": raw_data, "source": source} - ) + kline = raw_data.get("k") if isinstance(raw_data, dict) else None + event_interval = str((kline or {}).get("i") or interval) + key = f"kline:{event_interval}:{sym}" + channel = f"stream:kline:{event_interval}:{sym}" + redis_item = { + "key": key, + "channel": channel, + "data": raw_data, + "source": source, + } + if isinstance(raw_data, dict) and raw_data.get("recovery_source"): + kline = raw_data.get("k") or {} + redis_item["coalesce_id"] = f"recovery:{kline.get('t')}" + redis_items.append(redis_item) elif source == "dnse": sym = data.get("symbol", "") @@ -424,6 +415,7 @@ async def start_stream( max_conns_per_source: int | None = None, supervisor: StreamSupervisor | None = None, demand_registry=None, + queue: asyncio.Queue | None = None, ): """ Main entrypoint used by services. @@ -437,7 +429,7 @@ async def start_stream( ) # 1. Setup Queue and Publisher - queue = asyncio.Queue(maxsize=BINANCE_WS_QUEUE_MAXSIZE) + queue = queue if queue is not None else asyncio.Queue(maxsize=BINANCE_WS_QUEUE_MAXSIZE) supervisor.record_queue_size(queue.qsize(), queue.maxsize) publisher_task = asyncio.create_task(redis_publisher_task(queue, redis_cache, interval, supervisor)) diff --git a/app/stream/demand_registry.py b/app/stream/demand_registry.py index 01a30b2..5520b6e 100644 --- a/app/stream/demand_registry.py +++ b/app/stream/demand_registry.py @@ -163,8 +163,11 @@ async def snapshot(self) -> dict[str, Any]: float(item.get("expires_at_unix") or 0), ) items = sorted(aggregate.values(), key=lambda item: item["feed_key"]) + owners = sorted({str(item.get("owner_id")) for item in records if item.get("owner_id")}) return { "lease_count": len(records), + "owner_count": len(owners), + "owners": owners, "demanded_feed_count": len(items), "feed_keys": [item["feed_key"] for item in items], "items": items, diff --git a/app/stream/kline_recovery.py b/app/stream/kline_recovery.py new file mode 100644 index 0000000..4b43ef8 --- /dev/null +++ b/app/stream/kline_recovery.py @@ -0,0 +1,325 @@ +from __future__ import annotations + +import asyncio +import logging +import math +import time +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from typing import Any, Callable + +from app.providers.binance import rest as binance_rest +from app.stream.demand_registry import parse_feed_key + + +logger = logging.getLogger(__name__) + + +def _fixed_interval_ms(interval: str) -> int | None: + value = int(interval[:-1]) + unit = interval[-1] + multiplier = { + "s": 1_000, + "m": 60_000, + "h": 3_600_000, + "d": 86_400_000, + }.get(unit) + return value * multiplier if multiplier else None + + +def _next_close_ms(now_ms: int, interval: str, settle_ms: int) -> int: + fixed = _fixed_interval_ms(interval) + if fixed: + return ((now_ms // fixed) + 1) * fixed + settle_ms + + now = datetime.fromtimestamp(now_ms / 1000, tz=timezone.utc) + if interval.endswith("w"): + weeks = int(interval[:-1]) + monday = (now - timedelta(days=now.weekday())).replace( + hour=0, minute=0, second=0, microsecond=0 + ) + boundary = monday + timedelta(weeks=weeks) + elif interval.endswith("M"): + months = int(interval[:-1]) + year = now.year + month = now.month + months + year += (month - 1) // 12 + month = ((month - 1) % 12) + 1 + boundary = datetime(year, month, 1, tzinfo=timezone.utc) + else: + raise ValueError(f"Unsupported recovery interval: {interval}") + return int(boundary.timestamp() * 1000) + settle_ms + + +def _payload_marker(payload: dict[str, Any] | None) -> tuple[int | None, bool]: + if not isinstance(payload, dict): + return None, False + kline = payload.get("k") if isinstance(payload.get("k"), dict) else payload + try: + open_time = int(kline.get("t")) + except (TypeError, ValueError): + return None, False + return open_time, bool(kline.get("x", False)) + + +def _closed_rows(rows: list[Any], now_ms: int) -> list[list[Any]]: + closed: list[list[Any]] = [] + for row in rows: + if not isinstance(row, list) or len(row) <= 6: + continue + try: + open_time = int(row[0]) + close_time = int(row[6]) + except (TypeError, ValueError): + continue + if close_time > now_ms or any(row[index] is None for index in range(1, 6)): + continue + if open_time >= close_time: + continue + closed.append(row) + return sorted(closed, key=lambda row: int(row[0])) + + +def _recovery_event(symbol: str, interval: str, row: list[Any]) -> dict[str, Any]: + return { + "e": "kline_recovery", + "E": int(time.time() * 1000), + "s": symbol, + "k": { + "t": row[0], + "T": row[6], + "s": symbol, + "i": interval, + "o": row[1], + "c": row[4], + "h": row[2], + "l": row[3], + "v": row[5], + "x": True, + }, + "recovery_source": "BINANCE_REST_GAP_FILL", + "provider": "binance", + "market": "binance_usdm", + "authoritative": True, + } + + +@dataclass(frozen=True) +class KlineRecoveryConfig: + enabled: bool = True + poll_seconds: float = 2.0 + settle_seconds: float = 1.0 + concurrency: int = 4 + max_limit: int = 1000 + max_backoff_seconds: float = 300.0 + queue_put_timeout_seconds: float = 2.0 + + +class DemandKlineRecovery: + """Demand-only Binance REST recovery for closed bars. + + This is a recovery projector, not a replacement WebSocket health signal. + Provider rows retain their native values and only fully closed rows enter + the existing V1 kline projection. + """ + + def __init__( + self, + *, + queue: asyncio.Queue, + redis_cache: Any, + demand_registry: Any, + config: KlineRecoveryConfig, + fetcher: Callable[..., dict[str, Any]] = binance_rest.fetch_klines, + ) -> None: + self.queue = queue + self.redis_cache = redis_cache + self.demand_registry = demand_registry + self.config = config + self.fetcher = fetcher + self.running = False + self.last_poll_at: float | None = None + self.last_success_at: float | None = None + self.last_error: str | None = None + self.provider_fetch_count = 0 + self.emitted_count = 0 + self.deduplicated_count = 0 + self.rejected_open_or_invalid_count = 0 + self.failure_count = 0 + self.active_demand_count = 0 + self._next_due_ms: dict[str, int] = {} + self._failures: dict[str, int] = {} + self._semaphore = asyncio.Semaphore(max(1, min(config.concurrency, 16))) + + def snapshot(self) -> dict[str, Any]: + return { + "enabled": self.config.enabled, + "running": self.running, + "status": ( + "disabled" + if not self.config.enabled + else "degraded" + if self.last_error + else "ready" + if self.running + else "stopped" + ), + "active_demand_count": self.active_demand_count, + "provider_fetch_count": self.provider_fetch_count, + "emitted_count": self.emitted_count, + "deduplicated_count": self.deduplicated_count, + "rejected_open_or_invalid_count": self.rejected_open_or_invalid_count, + "failure_count": self.failure_count, + "last_poll_at": self.last_poll_at, + "last_success_at": self.last_success_at, + "last_error": self.last_error, + } + + async def run(self) -> None: + if not self.config.enabled: + return + self.running = True + try: + while True: + try: + await self.poll_once() + except Exception as exc: + self.failure_count += 1 + self.last_error = f"{type(exc).__name__}: {exc}" + logger.warning("Binance kline recovery poll failed: %s", exc) + await asyncio.sleep(max(0.25, self.config.poll_seconds)) + except asyncio.CancelledError: + raise + finally: + self.running = False + + async def poll_once(self, *, now_ms: int | None = None) -> dict[str, int]: + now_ms = now_ms or int(time.time() * 1000) + self.last_poll_at = time.time() + demands = await self.demand_registry.snapshot() + selected: list[dict[str, str | None]] = [] + invalid_demand_count = 0 + active_keys: set[str] = set() + for key in demands.get("feed_keys", []): + parsed = parse_feed_key(key) + if parsed["source"] != "binance_usdm" or parsed["feed"] != "kline": + continue + if not parsed["symbol"] or not parsed["interval"]: + continue + try: + normalized_interval = binance_rest.normalize_interval(str(parsed["interval"])) + except ValueError as exc: + invalid_demand_count += 1 + self.failure_count += 1 + self.last_error = f"invalid demand {key}: {exc}" + continue + parsed["interval"] = normalized_interval + selected.append(parsed) + active_keys.add( + f"kline:binance_usdm:{normalized_interval}:{parsed['symbol']}" + ) + + self.active_demand_count = len(selected) + for key in set(self._next_due_ms) - active_keys: + self._next_due_ms.pop(key, None) + self._failures.pop(key, None) + + due = [ + item + for item in selected + if now_ms >= self._next_due_ms.get( + f"kline:binance_usdm:{item['interval']}:{item['symbol']}", 0 + ) + ] + if not due: + if not selected and invalid_demand_count == 0: + self.last_error = None + return {"due": 0, "emitted": 0, "failed": invalid_demand_count} + + outcomes = await asyncio.gather( + *(self._recover_one(item, now_ms=now_ms) for item in due), + return_exceptions=True, + ) + emitted = sum(value for value in outcomes if isinstance(value, int)) + failed = sum(1 for value in outcomes if isinstance(value, BaseException)) + if failed == 0: + self.last_error = None + return {"due": len(due), "emitted": emitted, "failed": failed} + + async def _recover_one(self, item: dict[str, str | None], *, now_ms: int) -> int: + symbol = str(item["symbol"]) + interval = str(item["interval"]) + feed_key = f"kline:binance_usdm:{interval}:{symbol}" + try: + async with self._semaphore: + existing = await self.redis_cache.get_binance_kline_last(symbol, interval) + existing_open, existing_final = _payload_marker(existing) + interval_ms = _fixed_interval_ms(interval) + if existing_open is None or interval_ms is None: + limit = 3 + else: + missing = math.ceil(max(0, now_ms - existing_open) / interval_ms) + 2 + limit = min(self.config.max_limit, max(3, missing)) + self.provider_fetch_count += 1 + payload = await asyncio.to_thread( + self.fetcher, + symbol, + interval, + limit, + None, + None, + "usdm", + ) + + raw_rows = payload.get("data") or [] + rows = _closed_rows(raw_rows, now_ms) + self.rejected_open_or_invalid_count += max(0, len(raw_rows) - len(rows)) + if existing_open is None and rows: + rows = rows[-1:] + + emitted = 0 + current_open = existing_open + current_final = existing_final + for row in rows: + open_time = int(row[0]) + if current_open is not None and ( + open_time < current_open or (open_time == current_open and current_final) + ): + self.deduplicated_count += 1 + continue + event = _recovery_event(symbol, interval, row) + await asyncio.wait_for( + self.queue.put(("binance_futures_kline", event)), + timeout=self.config.queue_put_timeout_seconds, + ) + current_open = open_time + current_final = True + emitted += 1 + + self.emitted_count += emitted + self.last_success_at = time.time() + self._failures[feed_key] = 0 + self._next_due_ms[feed_key] = _next_close_ms( + now_ms, + interval, + int(self.config.settle_seconds * 1000), + ) + return emitted + except Exception as exc: + self.failure_count += 1 + self.last_error = f"{type(exc).__name__}: {exc}" + failures = self._failures.get(feed_key, 0) + 1 + self._failures[feed_key] = failures + backoff = min( + self.config.max_backoff_seconds, + max(self.config.poll_seconds, self.config.poll_seconds * (2 ** min(failures, 8))), + ) + self._next_due_ms[feed_key] = now_ms + int(backoff * 1000) + logger.warning( + "Binance demanded kline recovery failed symbol=%s interval=%s backoff=%.1fs error=%s", + symbol, + interval, + backoff, + exc, + ) + raise diff --git a/app/stream/supervisor.py b/app/stream/supervisor.py index ac998a2..7a96ab9 100644 --- a/app/stream/supervisor.py +++ b/app/stream/supervisor.py @@ -29,6 +29,8 @@ class ShardState: last_connected_at: Optional[float] = None last_message_at: Optional[float] = None last_error: Optional[str] = None + data_timeout_count: int = 0 + last_data_timeout_at: Optional[float] = None outage_started_at: Optional[float] = None last_disconnected_at: Optional[float] = None last_recovered_at: Optional[float] = None @@ -52,6 +54,8 @@ def to_dict(self) -> Dict[str, Any]: "last_connected_at": _iso(self.last_connected_at), "last_message_at": _iso(self.last_message_at), "last_error": self.last_error, + "data_timeout_count": self.data_timeout_count, + "last_data_timeout_at": _iso(self.last_data_timeout_at), "outage_started_at": _iso(self.outage_started_at), "last_disconnected_at": _iso(self.last_disconnected_at), "last_recovered_at": _iso(self.last_recovered_at), @@ -116,9 +120,11 @@ def __init__( sample_limit: int = 10, startup_grace_seconds: float = 180.0, strict_feed_health: bool = False, + first_frame_timeout_seconds: float = 15.0, ): self.stale_after_seconds = stale_after_seconds self.startup_grace_seconds = startup_grace_seconds + self.first_frame_timeout_seconds = first_frame_timeout_seconds self.strict_feed_health = strict_feed_health self.sample_limit = sample_limit self.started_at = _now() @@ -129,6 +135,8 @@ def __init__( self.queue_drop_count = 0 self.queue_drop_window_seconds = 300.0 self._queue_drop_times: deque[float] = deque(maxlen=100_000) + self.queue_pressure_count = 0 + self.last_queue_pressure_at: Optional[float] = None self.redis_error_count = 0 self.last_redis_error: Optional[str] = None self.publisher_batch_count = 0 @@ -186,7 +194,15 @@ def mark_connected(self, shard_id: str) -> bool: now = _now() shard.status = "connected" shard.last_connected_at = now - shard.last_error = None + data_timeout_active = bool( + shard.last_data_timeout_at + and ( + not shard.last_message_at + or shard.last_data_timeout_at >= shard.last_message_at + ) + ) + if not data_timeout_active: + shard.last_error = None if recovered: duration = max(0.0, now - float(shard.outage_started_at)) shard.last_outage_seconds = duration @@ -202,6 +218,16 @@ def mark_message(self, shard_id: str) -> None: if shard: shard.message_count += 1 shard.last_message_at = _now() + if shard.last_error and shard.last_error.startswith("data_timeout:"): + shard.last_error = None + + def mark_data_timeout(self, shard_id: str, reason: str) -> None: + shard = self.shards.get(shard_id) + if shard: + now = _now() + shard.data_timeout_count += 1 + shard.last_data_timeout_at = now + shard.last_error = f"data_timeout:{reason}" def mark_parse_error(self, shard_id: str, error: Exception) -> None: shard = self.shards.get(shard_id) @@ -244,6 +270,10 @@ def record_queue_drop(self, source: str, shard_id: Optional[str] = None) -> None if shard_id and shard_id in self.shards: self.shards[shard_id].queue_drop_count += 1 + def record_queue_pressure(self) -> None: + self.queue_pressure_count += 1 + self.last_queue_pressure_at = _now() + def record_redis_error(self, error: Exception) -> None: self.redis_error_count += 1 self.last_redis_error = str(error) @@ -300,6 +330,68 @@ def record_batch_published(self, size: int) -> None: self.publisher_batch_count += 1 self.last_publisher_at = _now() + def _source_states(self, now: float) -> Dict[str, Dict[str, Any]]: + states: Dict[str, Dict[str, Any]] = {} + for source in sorted({shard.source for shard in self.shards.values()}): + shards = [shard for shard in self.shards.values() if shard.source == source] + connected = [shard for shard in shards if shard.status == "connected"] + producing = [] + waiting = [] + stale = [] + unavailable = [] + for shard in shards: + session_has_frame = bool( + shard.status == "connected" + and shard.last_message_at + and shard.last_connected_at + and shard.last_message_at >= shard.last_connected_at + ) + if session_has_frame: + age = max(0.0, now - float(shard.last_message_at)) + if age <= self.stale_after_seconds: + producing.append(shard) + else: + stale.append(shard) + elif shard.status == "connected" and shard.last_connected_at: + age = max(0.0, now - float(shard.last_connected_at)) + data_timeout_active = bool( + shard.last_data_timeout_at + and ( + not shard.last_message_at + or shard.last_data_timeout_at >= shard.last_message_at + ) + ) + if not data_timeout_active and age <= self.first_frame_timeout_seconds: + waiting.append(shard) + else: + unavailable.append(shard) + else: + unavailable.append(shard) + + if len(producing) == len(shards) and shards: + status = "ready" + elif producing: + status = "degraded" + elif waiting and len(waiting) == len(shards): + status = "starting" + else: + status = "unavailable" + states[source] = { + "feed": "trade" if source.endswith("_trade") else "kline", + "status": status, + "transport_ready": len(connected) == len(shards) and bool(shards), + "data_ready": status == "ready", + "shard_count": len(shards), + "connected_count": len(connected), + "producing_count": len(producing), + "waiting_first_frame_count": len(waiting), + "stale_count": len(stale), + "unavailable_count": len(unavailable), + "first_frame_timeout_seconds": self.first_frame_timeout_seconds, + "stale_after_seconds": self.stale_after_seconds, + } + return states + def snapshot( self, now: Optional[float] = None, @@ -344,6 +436,15 @@ def snapshot( ] reconnect_count = sum(shard.reconnect_count for shard in self.shards.values()) connected_shards = [shard for shard in self.shards.values() if shard.status == "connected"] + source_states = self._source_states(now) + unavailable_sources = [ + source + for source, state in source_states.items() + if state["status"] in {"degraded", "unavailable"} + ] + starting_sources = [ + source for source, state in source_states.items() if state["status"] == "starting" + ] health_warnings = [] cutoff = now - self.queue_drop_window_seconds @@ -361,13 +462,19 @@ def snapshot( health_warnings.append("missing_expected_feeds") if stale: health_warnings.append("stale_expected_feeds") + if unavailable_sources: + health_warnings.append("source_data_unavailable") if self.redis_error_count: status = "degraded" elif not self.shards: status = "not_started" + elif unavailable_sources: + status = "degraded" elif not connected_shards: status = "starting" + elif starting_sources: + status = "starting" elif demanded_stale or (demanded_missing and uptime_seconds > self.startup_grace_seconds): status = "degraded" elif self.strict_feed_health and (recent_queue_drops or broad_health_missing or stale): @@ -388,6 +495,8 @@ def snapshot( "maxsize": self.queue_maxsize, "drop_count": self.queue_drop_count, "recent_drop_count": recent_queue_drops, + "pressure_count": self.queue_pressure_count, + "last_pressure_at": _iso(self.last_queue_pressure_at), "window_seconds": self.queue_drop_window_seconds, }, "publisher": { @@ -403,6 +512,7 @@ def snapshot( "reconnect_count": reconnect_count, "items": [s.to_dict() for s in list(self.shards.values())[: self.sample_limit]], }, + "sources": source_states, "feeds": { "expected_count": len(expected_feeds), "observed_count": len([feed for feed in expected_feeds if feed.last_published_at]), diff --git a/config/phase9/candidate-slice.yaml b/config/phase9/candidate-slice.yaml new file mode 100644 index 0000000..350eb8d --- /dev/null +++ b/config/phase9/candidate-slice.yaml @@ -0,0 +1,28 @@ +schema: qdl.authority-candidate.v1 +slice_id: production/binance/usdm/perpetual/trade/plan-1/btcusdt +environment: production +venue: BINANCE +market: USDM +product_type: PERPETUAL +feed: TRADE +instrument_uids: + - 85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c +partition_plan_epoch: 1 +partition_id: rendezvous-sha256-v1:epoch-1:btcusdt +schema_major: 2 +authority_state: RUST_SHADOW +owner_id: rust-ingestor-binance-usdm-shard-0 +lease_epoch: 1 +artifact_image_digest: sha256:46a7c3fa516c0035c3ce41add0ce77e9acb4d4dfd1b0ac74130c894ca7ad5280 +sbom_digest: eab28f6b6685d6f2e748d5f12b540f2ced62c6f8f4686a1dc31d67084aaa5d36 +contract_digest: f26d0369d73db3c6943e08824c62925507b91f6bdabef4636f48f3037734ebec +partition_plan_digest: 72baa5414df483293389a8da6a69ddbda8ac7e169e1cd4234a1c9477412e9781 +rollback_manifest_digest: 36acbeacf0e26efbfb38973957331dd383f639aecd08dc4cedfa5f3812f451f1 +signature_identity: phase8-rehearsal-only-not-admitted +normalizer_version: qdl-rust-core/0.1.0-beta +adapter_version: binance-usdm/2.0.0-shadow +config_revision: phase8-candidate-1 +instrument_catalog_revision: phase8-catalog-1 +source_policy_revision: binance-usdm-trade-shadow-1 +public_write_allowed: false +legacy_write_allowed: false diff --git a/config/phase9/evidence-inventory.local.yaml b/config/phase9/evidence-inventory.local.yaml new file mode 100644 index 0000000..34a5c7c --- /dev/null +++ b/config/phase9/evidence-inventory.local.yaml @@ -0,0 +1,101 @@ +schema: qdl.production-prerequisite-inventory.v1 +evidence: + - evidence_id: phase8-local-kafka-failover + gate_id: replicated_durable_transport + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-broker-failover.json + artifact_sha256: a9b43e4d70b2002032d9c397aea0bb265c50122969f3e3c808f1b284a353ce55 + details: + replication_factor: 3 + min_isr: 2 + acks_all: true + idempotent_producer: true + broker_failover: true + restore_test_id: phase8-same-host-volume-loss + - evidence_id: phase8-local-otel-debug + gate_id: production_observability + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: config/observability/phase8-otel-collector.yaml + artifact_sha256: 915e0ac483b436aaf9f50ffee4cb7196e74aafd2425f0c9f34318089ed1a86c9 + details: + collector_backend: debug-only + dashboard_revision: not-deployed + alert_route_id: not-deployed + page_ack_id: not-deployed + - evidence_id: phase8-local-kafka-mtls + gate_id: workload_identity_rbac_network + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-broker-security.json + artifact_sha256: 289dd9a2cda61121bc7518718b3b5fdde976235e212720ffd28389e26ecc6a5b + details: + identity_provider: ephemeral-self-signed-mtls + rbac_revision: phase8-kafka-acl + network_policy_revision: same-host-internal-compose + - evidence_id: phase8-signature-rehearsal-no-admission + gate_id: signed_artifact_admission + environment: production + scope: LOCAL_REHEARSAL + status: BLOCKED + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: null + artifact_sha256: null + details: + reason: registry-admission-not-installed + - evidence_id: phase8-local-projection-rebuild + gate_id: redis_projector_rebuild + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-broker-topology.json + artifact_sha256: 68d2d6e409c1cf5428ce437e753653b24ddd16afc8e7cff208dd087b3ae1d500 + details: + canonical_watermark: phase8-offset-65 + rebuild_checksum: phase8-isolated-equal + source_log_cluster: qdl-phase8-same-host + - evidence_id: phase90b-partial-consumer-beta + gate_id: consumer_registration_rollback + environment: production + scope: LOCAL_REHEARSAL + status: BLOCKED + issuer: qdl-phase90b-certification + observed_at_ns: 1787061600000000000 + expires_at_ns: 1789653600000000000 + artifact_path: null + artifact_sha256: null + details: + reason: beta-consumers-do-not-cover-trade-authority-slice + - evidence_id: phase8-local-authority-rehearsal + gate_id: persistent_authority_sink_fencing + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-authority-rehearsal.json + artifact_sha256: ec5efe8b99687055eb42ae481b00201937a297148e479c6822b0fa3d4fb29f4a + details: + authority_store: same-host-compacted-kafka + transition_audit_id: phase8-shadow-canary-shadow + zombie_writer_rejected: true + candidate_digest: 72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037 diff --git a/config/phase9/production-prerequisite-policy.yaml b/config/phase9/production-prerequisite-policy.yaml new file mode 100644 index 0000000..846c274 --- /dev/null +++ b/config/phase9/production-prerequisite-policy.yaml @@ -0,0 +1,236 @@ +schema: qdl.production-prerequisite-policy.v1 +revision: 1 +environment: production +gates: +- id: replicated_durable_transport + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - replication_factor + - min_isr + - acks_all + - idempotent_producer + - broker_failover + - restore_test_id + - broker_count + - failure_domain_count + - unclean_leader_election + - restore_verified + candidate_bound: false + assertions: + acks_all: true + idempotent_producer: true + broker_failover: true + restore_verified: true + unclean_leader_election: false + minimums: + broker_count: 3 + replication_factor: 3 + min_isr: 2 + failure_domain_count: 2 + maximums: {} + candidate_field_matches: {} +- id: production_observability + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - collector_backend + - dashboard_revision + - alert_route_id + - page_ack_id + - backend_persistent + - collector_ha + - alert_acknowledged + - retention_days + candidate_bound: false + assertions: + backend_persistent: true + collector_ha: true + alert_acknowledged: true + minimums: + retention_days: 30 + maximums: {} + candidate_field_matches: {} +- id: workload_identity_rbac_network + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - identity_provider + - rbac_revision + - network_policy_revision + - short_lived_credentials + - rbac_enforced + - network_default_deny + candidate_bound: false + assertions: + short_lived_credentials: true + rbac_enforced: true + network_default_deny: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: external_secret_rotation + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - secret_backend + - rotation_event_id + - retired_key_id + - rotation_passed + - retired_key_rejected + candidate_bound: false + assertions: + rotation_passed: true + retired_key_rejected: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: signed_artifact_admission + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - registry + - signature_identity + - admission_policy_revision + - image_digest + - candidate_digest + - signature_verified + - unsigned_rejected + candidate_bound: true + assertions: + signature_verified: true + unsigned_rejected: true + minimums: {} + maximums: {} + candidate_field_matches: + image_digest: artifact_image_digest + signature_identity: signature_identity +- id: postgres_pitr + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - backup_id + - restore_id + - rpo_seconds + - rto_seconds + - restore_verified + - checksum_equal + candidate_bound: false + assertions: + restore_verified: true + checksum_equal: true + minimums: {} + maximums: + rpo_seconds: 300 + rto_seconds: 3600 + candidate_field_matches: {} +- id: object_store_restore + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - snapshot_id + - restore_id + - manifest_digest + - restore_verified + - checksum_equal + candidate_bound: false + assertions: + restore_verified: true + checksum_equal: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: independent_failure_domain_dr + minimum_scope: INDEPENDENT_FAILURE_DOMAIN + max_age_seconds: 2592000 + required_details: + - source_domain + - target_domain + - exercise_id + - rpo_seconds + - rto_seconds + - failover_verified + - duplicate_external_publications + - failure_domain_count + candidate_bound: false + assertions: + failover_verified: true + duplicate_external_publications: 0 + minimums: + failure_domain_count: 2 + maximums: + rpo_seconds: 300 + rto_seconds: 3600 + candidate_field_matches: {} +- id: redis_projector_rebuild + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - canonical_watermark + - rebuild_checksum + - source_log_cluster + - checksum_equal + - zero_production_mutation + - replay_lag_events + candidate_bound: false + assertions: + checksum_equal: true + zero_production_mutation: true + minimums: {} + maximums: + replay_lag_events: 0 + candidate_field_matches: {} +- id: consumer_registration_rollback + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - consumers_digest + - owners_approved + - rollback_rehearsal_id + - candidate_digest + - rollback_rehearsed + - unregistered_critical_consumers + candidate_bound: true + assertions: + owners_approved: true + rollback_rehearsed: true + minimums: {} + maximums: + unregistered_critical_consumers: 0 + candidate_field_matches: {} +- id: persistent_authority_sink_fencing + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - authority_store + - transition_audit_id + - zombie_writer_rejected + - candidate_digest + - sink_side_fencing + - audit_immutable + candidate_bound: true + assertions: + zombie_writer_rejected: true + sink_side_fencing: true + audit_immutable: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: exact_slice_approval + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - approved_by + - change_ticket + - hold_until_ns + - blast_radius + - candidate_digest + - approved + - approver_count + candidate_bound: true + assertions: + approved: true + minimums: + approver_count: 1 + maximums: {} + candidate_field_matches: {} diff --git a/docker-compose.phase8-kafka.yml b/docker-compose.phase8-kafka.yml index ae85789..ba07161 100644 --- a/docker-compose.phase8-kafka.yml +++ b/docker-compose.phase8-kafka.yml @@ -51,7 +51,9 @@ x-kafka-common: &kafka-common target: /etc/kafka/secrets read_only: true healthcheck: - test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server $${HOSTNAME}:9092 --command-config /etc/kafka/secrets/admin.properties >/dev/null 2>&1"] + # Liveness stays lightweight inside the broker cgroup. Authenticated Kafka + # readiness/quorum checks run from the disposable phase8_admin client. + test: ["CMD-SHELL", "timeout 2 bash -c ' RUST_PRIMARY -> BLOCKED -> ROLLBACK_PENDING -> + PYTHON_PRIMARY` is ordered and reconstructable after broker restart; +- isolated canonical/public/legacy projections all contain exactly watermarks + 101 through 181 with the owner boundary at 164/165; +- a fresh Rust process loading the terminal primary authority rejects all three + targets until each durable projection watermark is reconstructed, rejects W + again after restore and resumes all targets exactly at W+1; +- one replica loss still ACKs, below-min-ISR fails closed; +- production public/legacy writes remain zero; +- V1 topology and health are unchanged before/after; +- all disposable resources are absent after cleanup. + +## Production Promotion Gate + +Do not reuse local rehearsal evidence as production approval. A future primary +cutover additionally requires a fresh exact Phase 9.0-C `GO`, a completed real +`RUST_CANARY` hold, consumer registry approval, immutable rollback artifact, +change ticket and explicit operator approval naming the exact slice. + +At cutover, persist the old-owner checkpoint at W, accept clean handoff evidence, +execute the database CAS, load the new authority into final sink and projector, +then emit first authoritative watermark W+1. Disable only that Python +subscription after all of those gates pass. + +## Rollback + +Fence Rust first. Persist its final watermark, enter `ROLLBACK_PENDING`, accept +a Rust-to-Python handoff, grant Python a new authority revision and lease, then +resume from the reconciled next watermark. Never restart Python as an +uncoordinated writer. + +For local cleanup: + +```bash +make phase92-clean +``` + +This command removes only Phase 9.2 certification containers, networks, volumes +and images. It does not prune global Docker state or touch V1 data. diff --git a/docs/runbooks/phase93-hold-close-expand.md b/docs/runbooks/phase93-hold-close-expand.md new file mode 100644 index 0000000..ab78574 --- /dev/null +++ b/docs/runbooks/phase93-hold-close-expand.md @@ -0,0 +1,122 @@ +# Phase 9.3 Hold, Close And Expand Runbook + +## Current Authority Boundary + +Repository status may reach only: + +`COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED` + +while Phase 9.0-C is `NO_GO_EXTERNAL`. Local tests do not start a production +hold, close a production rollback window, authorize an expansion, decommission +Python or change authority. V1 remains authoritative. + +## Local Certification + +Run from `/home/bobby/data_layer`: + +```bash +make phase93-test +make phase93-migration +make phase93-certify +sha256sum -c upgrade/evidence/phase93-evidence.sha256 +``` + +Expected local evidence: + +- current Phase 9.0-C no-go rejects production closure; +- Phase 9.2 authentic replay remains parent provenance only; +- accelerated hold observations are marked `TEST_CONTROL_PLANE_FIXTURE`; +- PostgreSQL closure leaves authority state/revision/owner/lease/watermark + unchanged; +- all five expansion types remain + `INDEPENDENT_CERTIFICATION_REQUIRED` with no write authority; +- Python decommission remains blocked while any ownership, rollback or consumer + dependency exists; +- V1 health/topology is unchanged and disposable resources are absent. + +## Starting A Real Hold + +A real hold is legal only after a production-authorized exact +`RUST_PRIMARY` cutover and a fresh Phase 9.0-C `GO@. + +1. Freeze the hold policy and its digest. The approved minimum wall-clock + duration, sample interval, maximum sample gap, correctness zero-tolerance and + resource/lag thresholds cannot change inside one hold ID. +2. Persist the hold identity bound to slice, candidate, prerequisite bundle, + owner, authority revision, lease and partition-plan epoch. +3. Append observations from production telemetry. Do not backfill or interpolate + missing intervals. A changed owner/epoch or a missing interval blocks that + hold. +4. Persist a `BLOCKED@ decision immediately on any semantic mismatch, open gap, + duplicate external write, accepted stale writer, authority ambiguity, durable + ACK failure, projection mismatch, consumer checkpoint regression or + unexplained quality failure. +5. A blocked hold is immutable. Restart only with a new hold ID and a new + operator decision. +6. A passing decision requires the real wall-clock duration and terminal sample. + Accelerated fixtures, replay duration and same-host time compression never + count as production hold evidence. + +## Closing The Rollback Window + +Window closure is a governed audit decision, not a data-plane authority +transition and not deletion of the Python rollback manifest. + +Before calling `qdl_close_authority_window`, freeze: + +- passing production hold decision; +- consumer registry snapshot with every critical consumer ready, fully migrated, + rollback-ready and checkpointed through the current authority watermark; +- authority registry snapshot matching current `RUST_PRIMARY` + owner/revision/lease/partition/candidate/bundle; +- fresh production-scope rollback rehearsal reconciled through the same + watermark; +- operator approval bound to hold policy, ticket and bounded expiry. + +The function locks and rechecks the exact authority row. A concurrent authority +change, stale registry, expired rollback rehearsal or mismatched approval fails. +Successful closure inserts one immutable row and changes no authority field. + +## Expansion + +Create one manifest per expansion class: + +- `INSTRUMENT_PARTITION` +- `BBO` +- `L2_BOOK` +- `BAR_LIFECYCLE` +- `VENUE_MARKET` + +Each manifest needs a new candidate digest and capability-specific gate set. +Instrument/partition expansion also requires a newer partition-plan epoch. +Parent closure evidence is provenance only. Repeat provider-authentic parity, +chaos, capacity, authority handoff and rollback certification independently. +Never combine classes merely to reuse an approval. + +## Python Runtime Decommission + +Removal is denied until all are true: + +- runtime owns zero authority slices; +- no active rollback manifest references it; +- no registered consumer depends on it; +- every replacement rollback window is governed closed; +- repository cleanup has separate explicit approval; +- shared contracts, fixtures and provider/compatibility knowledge remain. + +Closing one slice does not authorize deleting a reusable adapter. + +## Incident And Rollback + +A closed window does not remove emergency rollback capability. On incident: + +1. fence Rust at final sink; +2. persist terminal watermark and incident evidence; +3. enter `BLOCKED@ then `ROLLBACK_PENDING@; +4. accept a new Rust-to-Python handoff under a newer revision/lease; +5. resume Python from the reconciled next watermark; +6. record a new audit decision. + +Do not delete or mutate hold/registry/closure/expansion rows. For local residue, +run `make phase93-clean`; it removes only containers whose names begin with +`qdl_phase93_` and performs no global Docker prune. diff --git a/migrations/postgres/0006_phase9_authority_prerequisites.sql b/migrations/postgres/0006_phase9_authority_prerequisites.sql new file mode 100644 index 0000000..bed4fdf --- /dev/null +++ b/migrations/postgres/0006_phase9_authority_prerequisites.sql @@ -0,0 +1,218 @@ +BEGIN; + +CREATE TABLE IF NOT EXISTS qdl_production_prerequisite_bundles ( + bundle_id UUID PRIMARY KEY, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + policy_revision BIGINT NOT NULL CHECK (policy_revision > 0), + decision TEXT NOT NULL CHECK (decision IN ('GO', 'NO_GO_EXTERNAL')), + evidence JSONB NOT NULL, + evidence_sha256 TEXT NOT NULL CHECK (evidence_sha256 ~ '^[0-9a-f]{64}$'), + issued_by TEXT NOT NULL CHECK (btrim(issued_by) <> ''), + issued_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (jsonb_typeof(evidence) = 'object'), + CHECK (expires_at > issued_at) +); + +CREATE INDEX IF NOT EXISTS qdl_prerequisite_bundle_candidate_idx + ON qdl_production_prerequisite_bundles (candidate_digest, expires_at DESC); + +CREATE TABLE IF NOT EXISTS qdl_authority_slices ( + slice_id TEXT PRIMARY KEY CHECK (btrim(slice_id) <> ''), + environment TEXT NOT NULL, + venue TEXT NOT NULL, + market TEXT NOT NULL, + product_type TEXT NOT NULL, + feed TEXT NOT NULL, + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + partition_id TEXT NOT NULL CHECK (btrim(partition_id) <> ''), + schema_major INTEGER NOT NULL CHECK (schema_major > 0), + state TEXT NOT NULL CHECK (state IN ( + 'PYTHON_PRIMARY', 'RUST_SHADOW', 'VALIDATING', 'RUST_CANARY', + 'RUST_PRIMARY', 'BLOCKED', 'ROLLBACK_PENDING' + )), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + terminal_watermark BIGINT, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + artifact_image_digest TEXT NOT NULL CHECK (artifact_image_digest ~ '^sha256:[0-9a-f]{64}$'), + sbom_digest TEXT NOT NULL CHECK (sbom_digest ~ '^[0-9a-f]{64}$'), + signature_identity TEXT NOT NULL CHECK (btrim(signature_identity) <> ''), + contract_digest TEXT NOT NULL CHECK (contract_digest ~ '^[0-9a-f]{64}$'), + normalizer_version TEXT NOT NULL CHECK (btrim(normalizer_version) <> ''), + adapter_version TEXT NOT NULL CHECK (btrim(adapter_version) <> ''), + config_revision TEXT NOT NULL CHECK (btrim(config_revision) <> ''), + instrument_catalog_revision TEXT NOT NULL CHECK (btrim(instrument_catalog_revision) <> ''), + source_policy_revision TEXT NOT NULL CHECK (btrim(source_policy_revision) <> ''), + partition_plan_digest TEXT NOT NULL CHECK (partition_plan_digest ~ '^[0-9a-f]{64}$'), + rollback_manifest_digest TEXT NOT NULL CHECK (rollback_manifest_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID REFERENCES qdl_production_prerequisite_bundles(bundle_id), + approved_by TEXT, + approved_at TIMESTAMPTZ, + hold_until TIMESTAMPTZ, + updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (environment, venue, market, product_type, feed, partition_plan_epoch, partition_id, schema_major), + CHECK ((approved_by IS NULL) = (approved_at IS NULL)) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_transition_audit ( + transition_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + previous_state TEXT NOT NULL, + new_state TEXT NOT NULL, + previous_revision BIGINT NOT NULL, + new_revision BIGINT NOT NULL, + previous_owner_id TEXT NOT NULL, + new_owner_id TEXT NOT NULL, + previous_lease_epoch BIGINT NOT NULL, + new_lease_epoch BIGINT NOT NULL, + partition_plan_epoch BIGINT NOT NULL, + terminal_watermark BIGINT, + prerequisite_bundle_id UUID REFERENCES qdl_production_prerequisite_bundles(bundle_id), + hold_until TIMESTAMPTZ, + actor TEXT NOT NULL CHECK (btrim(actor) <> ''), + reason TEXT NOT NULL CHECK (btrim(reason) <> ''), + occurred_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (new_revision = previous_revision + 1) +); + +CREATE INDEX IF NOT EXISTS qdl_authority_transition_slice_idx + ON qdl_authority_transition_audit (slice_id, new_revision); + +CREATE OR REPLACE FUNCTION qdl_reject_authority_audit_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'qdl_authority_transition_audit is append-only'; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_audit_immutable + ON qdl_authority_transition_audit; +CREATE TRIGGER qdl_authority_audit_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_transition_audit +FOR EACH ROW EXECUTE FUNCTION qdl_reject_authority_audit_mutation(); + +CREATE OR REPLACE FUNCTION qdl_transition_authority( + p_transition_id UUID, + p_slice_id TEXT, + p_expected_state TEXT, + p_expected_revision BIGINT, + p_expected_owner_id TEXT, + p_expected_lease_epoch BIGINT, + p_expected_partition_plan_epoch BIGINT, + p_new_state TEXT, + p_new_owner_id TEXT, + p_new_lease_epoch BIGINT, + p_terminal_watermark BIGINT, + p_prerequisite_bundle_id UUID, + p_hold_until TIMESTAMPTZ, + p_actor TEXT, + p_reason TEXT +) +RETURNS qdl_authority_slices +LANGUAGE plpgsql +AS $$ +DECLARE + current_row qdl_authority_slices%ROWTYPE; + updated_row qdl_authority_slices%ROWTYPE; + bundle_row qdl_production_prerequisite_bundles%ROWTYPE; + transition_allowed BOOLEAN := FALSE; +BEGIN + IF btrim(p_actor) = '' OR btrim(p_reason) = '' OR btrim(p_new_owner_id) = '' THEN + RAISE EXCEPTION 'actor, reason and new owner are required'; + END IF; + + SELECT * INTO current_row + FROM qdl_authority_slices AS authority_slice + WHERE authority_slice.slice_id = p_slice_id + FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority slice not found: %', p_slice_id; + END IF; + + IF current_row.state <> p_expected_state + OR current_row.authority_revision <> p_expected_revision + OR current_row.owner_id <> p_expected_owner_id + OR current_row.lease_epoch <> p_expected_lease_epoch + OR current_row.partition_plan_epoch <> p_expected_partition_plan_epoch THEN + RAISE EXCEPTION 'authority compare-and-swap precondition failed'; + END IF; + + transition_allowed := CASE current_row.state + WHEN 'PYTHON_PRIMARY' THEN p_new_state IN ('RUST_SHADOW', 'BLOCKED') + WHEN 'RUST_SHADOW' THEN p_new_state IN ('VALIDATING', 'BLOCKED') + WHEN 'VALIDATING' THEN p_new_state IN ('RUST_CANARY', 'BLOCKED', 'ROLLBACK_PENDING') + WHEN 'RUST_CANARY' THEN p_new_state IN ('RUST_PRIMARY', 'BLOCKED', 'ROLLBACK_PENDING') + WHEN 'RUST_PRIMARY' THEN p_new_state IN ('BLOCKED', 'ROLLBACK_PENDING') + WHEN 'BLOCKED' THEN p_new_state IN ('VALIDATING', 'ROLLBACK_PENDING') + WHEN 'ROLLBACK_PENDING' THEN p_new_state = 'PYTHON_PRIMARY' + ELSE FALSE + END; + IF NOT transition_allowed THEN + RAISE EXCEPTION 'invalid authority transition: % -> %', current_row.state, p_new_state; + END IF; + + IF p_new_lease_epoch < current_row.lease_epoch + OR (p_new_owner_id <> current_row.owner_id AND p_new_lease_epoch <= current_row.lease_epoch) THEN + RAISE EXCEPTION 'new owner requires a strictly newer lease epoch'; + END IF; + + IF p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN + IF p_prerequisite_bundle_id IS NULL THEN + RAISE EXCEPTION 'canary/primary requires a prerequisite bundle'; + END IF; + IF p_terminal_watermark IS NULL OR p_terminal_watermark < 0 THEN + RAISE EXCEPTION 'canary/primary requires a non-negative terminal watermark'; + END IF; + IF p_hold_until IS NULL OR p_hold_until <= clock_timestamp() THEN + RAISE EXCEPTION 'canary/primary requires a future approval hold window'; + END IF; + SELECT * INTO bundle_row + FROM qdl_production_prerequisite_bundles AS bundle + WHERE bundle.bundle_id = p_prerequisite_bundle_id; + IF NOT FOUND + OR bundle_row.decision <> 'GO' + OR bundle_row.candidate_digest <> current_row.candidate_digest + OR bundle_row.expires_at <= clock_timestamp() + OR bundle_row.expires_at < p_hold_until THEN + RAISE EXCEPTION 'production prerequisite bundle is absent, blocked, mismatched or expired'; + END IF; + ELSIF p_prerequisite_bundle_id IS NOT NULL OR p_hold_until IS NOT NULL THEN + RAISE EXCEPTION 'prerequisite bundle and hold window are valid only for canary/primary'; + END IF; + + UPDATE qdl_authority_slices AS authority_slice + SET state = p_new_state, + authority_revision = current_row.authority_revision + 1, + owner_id = p_new_owner_id, + lease_epoch = p_new_lease_epoch, + terminal_watermark = p_terminal_watermark, + prerequisite_bundle_id = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN p_prerequisite_bundle_id ELSE NULL END, + approved_by = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN p_actor ELSE NULL END, + approved_at = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN clock_timestamp() ELSE NULL END, + hold_until = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN p_hold_until ELSE NULL END, + updated_at = clock_timestamp() + WHERE authority_slice.slice_id = p_slice_id + RETURNING * INTO updated_row; + + INSERT INTO qdl_authority_transition_audit ( + transition_id, slice_id, previous_state, new_state, + previous_revision, new_revision, previous_owner_id, new_owner_id, + previous_lease_epoch, new_lease_epoch, partition_plan_epoch, + terminal_watermark, prerequisite_bundle_id, hold_until, actor, reason + ) VALUES ( + p_transition_id, p_slice_id, current_row.state, updated_row.state, + current_row.authority_revision, updated_row.authority_revision, + current_row.owner_id, updated_row.owner_id, + current_row.lease_epoch, updated_row.lease_epoch, + current_row.partition_plan_epoch, p_terminal_watermark, + p_prerequisite_bundle_id, p_hold_until, p_actor, p_reason + ); + + RETURN updated_row; +END; +$$; + +COMMIT; diff --git a/migrations/postgres/0007_phase92_terminal_handoff.sql b/migrations/postgres/0007_phase92_terminal_handoff.sql new file mode 100644 index 0000000..1e5f231 --- /dev/null +++ b/migrations/postgres/0007_phase92_terminal_handoff.sql @@ -0,0 +1,205 @@ +BEGIN; + +CREATE TABLE IF NOT EXISTS qdl_terminal_owner_checkpoints ( + checkpoint_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + source_session_id TEXT NOT NULL CHECK (btrim(source_session_id) <> ''), + connection_generation BIGINT NOT NULL CHECK (connection_generation > 0), + terminal_watermark BIGINT NOT NULL CHECK (terminal_watermark >= 0), + terminal_event_id TEXT NOT NULL CHECK (btrim(terminal_event_id) <> ''), + terminal_payload_sha256 TEXT NOT NULL CHECK (terminal_payload_sha256 ~ '^[0-9a-f]{64}$'), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + committed_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE ( + slice_id, owner_id, authority_revision, lease_epoch, + partition_plan_epoch, terminal_watermark + ) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_handoffs ( + handoff_id UUID PRIMARY KEY, + checkpoint_id UUID NOT NULL REFERENCES qdl_terminal_owner_checkpoints(checkpoint_id), + direction TEXT NOT NULL CHECK (direction IN ('PYTHON_TO_RUST', 'RUST_TO_PYTHON')), + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + old_owner_id TEXT NOT NULL CHECK (btrim(old_owner_id) <> ''), + new_owner_id TEXT NOT NULL CHECK (btrim(new_owner_id) <> ''), + expected_state TEXT NOT NULL, + new_state TEXT NOT NULL, + expected_authority_revision BIGINT NOT NULL CHECK (expected_authority_revision > 0), + new_authority_revision BIGINT NOT NULL CHECK (new_authority_revision > 0), + expected_lease_epoch BIGINT NOT NULL CHECK (expected_lease_epoch > 0), + new_lease_epoch BIGINT NOT NULL CHECK (new_lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + terminal_watermark BIGINT NOT NULL CHECK (terminal_watermark >= 0), + first_new_watermark BIGINT NOT NULL CHECK (first_new_watermark >= 1), + overlap_start_watermark BIGINT NOT NULL CHECK (overlap_start_watermark >= 0), + overlap_end_watermark BIGINT NOT NULL CHECK (overlap_end_watermark >= 0), + old_event_count BIGINT NOT NULL CHECK (old_event_count > 0), + new_event_count BIGINT NOT NULL CHECK (new_event_count > 0), + semantic_mismatches BIGINT NOT NULL CHECK (semantic_mismatches = 0), + open_gaps BIGINT NOT NULL CHECK (open_gaps = 0), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL REFERENCES qdl_production_prerequisite_bundles(bundle_id), + handoff_sha256 TEXT NOT NULL CHECK (handoff_sha256 ~ '^[0-9a-f]{64}$'), + approved_by TEXT NOT NULL CHECK (btrim(approved_by) <> ''), + approved_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (old_owner_id <> new_owner_id), + CHECK (new_authority_revision = expected_authority_revision + 1), + CHECK (new_lease_epoch > expected_lease_epoch), + CHECK (first_new_watermark = terminal_watermark + 1), + CHECK (overlap_start_watermark <= overlap_end_watermark), + CHECK (overlap_end_watermark = terminal_watermark), + CHECK (old_event_count = new_event_count), + CHECK (expires_at > approved_at), + CHECK ( + (direction = 'PYTHON_TO_RUST' AND expected_state = 'RUST_CANARY' AND new_state = 'RUST_PRIMARY') + OR + (direction = 'RUST_TO_PYTHON' AND expected_state = 'ROLLBACK_PENDING' AND new_state = 'PYTHON_PRIMARY') + ) +); + +CREATE INDEX IF NOT EXISTS qdl_authority_handoff_transition_idx + ON qdl_authority_handoffs ( + slice_id, expected_authority_revision, new_authority_revision, new_state + ); + +CREATE OR REPLACE FUNCTION qdl_reject_phase92_evidence_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'Phase 9.2 checkpoint/handoff evidence is append-only'; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_terminal_checkpoint_immutable + ON qdl_terminal_owner_checkpoints; +CREATE TRIGGER qdl_terminal_checkpoint_immutable +BEFORE UPDATE OR DELETE ON qdl_terminal_owner_checkpoints +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase92_evidence_mutation(); + +DROP TRIGGER IF EXISTS qdl_authority_handoff_immutable + ON qdl_authority_handoffs; +CREATE TRIGGER qdl_authority_handoff_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_handoffs +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase92_evidence_mutation(); + +CREATE OR REPLACE FUNCTION qdl_require_accepted_primary_handoff() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + matching_handoffs BIGINT; +BEGIN + IF (OLD.state <> 'RUST_PRIMARY' AND NEW.state = 'RUST_PRIMARY') + OR (OLD.state = 'ROLLBACK_PENDING' AND NEW.state = 'PYTHON_PRIMARY') THEN + SELECT count(*) INTO matching_handoffs + FROM qdl_authority_handoffs AS handoff + JOIN qdl_terminal_owner_checkpoints AS checkpoint + ON checkpoint.checkpoint_id = handoff.checkpoint_id + WHERE handoff.slice_id = NEW.slice_id + AND handoff.old_owner_id = OLD.owner_id + AND handoff.new_owner_id = NEW.owner_id + AND handoff.expected_state = OLD.state + AND handoff.new_state = NEW.state + AND handoff.expected_authority_revision = OLD.authority_revision + AND handoff.new_authority_revision = NEW.authority_revision + AND handoff.expected_lease_epoch = OLD.lease_epoch + AND handoff.new_lease_epoch = NEW.lease_epoch + AND handoff.partition_plan_epoch = NEW.partition_plan_epoch + AND handoff.terminal_watermark = NEW.terminal_watermark + AND handoff.candidate_digest = NEW.candidate_digest + AND handoff.prerequisite_bundle_id = COALESCE( + NEW.prerequisite_bundle_id, handoff.prerequisite_bundle_id + ) + AND handoff.expires_at > clock_timestamp() + AND checkpoint.slice_id = handoff.slice_id + AND checkpoint.owner_id = handoff.old_owner_id + AND checkpoint.authority_revision = handoff.expected_authority_revision + AND checkpoint.lease_epoch = handoff.expected_lease_epoch + AND checkpoint.partition_plan_epoch = handoff.partition_plan_epoch + AND checkpoint.terminal_watermark = handoff.terminal_watermark + AND checkpoint.candidate_digest = handoff.candidate_digest; + IF matching_handoffs <> 1 THEN + RAISE EXCEPTION 'accepted exact Phase 9.2 handoff is required'; + END IF; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_primary_handoff_guard + ON qdl_authority_slices; +CREATE TRIGGER qdl_authority_primary_handoff_guard +BEFORE UPDATE ON qdl_authority_slices +FOR EACH ROW EXECUTE FUNCTION qdl_require_accepted_primary_handoff(); + +CREATE OR REPLACE FUNCTION qdl_transition_authority_v2( + p_handoff_id UUID, + p_transition_id UUID, + p_slice_id TEXT, + p_expected_state TEXT, + p_expected_revision BIGINT, + p_expected_owner_id TEXT, + p_expected_lease_epoch BIGINT, + p_expected_partition_plan_epoch BIGINT, + p_new_state TEXT, + p_new_owner_id TEXT, + p_new_lease_epoch BIGINT, + p_terminal_watermark BIGINT, + p_prerequisite_bundle_id UUID, + p_hold_until TIMESTAMPTZ, + p_actor TEXT, + p_reason TEXT +) +RETURNS qdl_authority_slices +LANGUAGE plpgsql +AS $$ +DECLARE + handoff qdl_authority_handoffs%ROWTYPE; + transitioned qdl_authority_slices%ROWTYPE; +BEGIN + SELECT * INTO handoff + FROM qdl_authority_handoffs AS accepted + WHERE accepted.handoff_id = p_handoff_id + FOR SHARE; + IF NOT FOUND + OR handoff.slice_id <> p_slice_id + OR handoff.expected_state <> p_expected_state + OR handoff.expected_authority_revision <> p_expected_revision + OR handoff.old_owner_id <> p_expected_owner_id + OR handoff.expected_lease_epoch <> p_expected_lease_epoch + OR handoff.partition_plan_epoch <> p_expected_partition_plan_epoch + OR handoff.new_state <> p_new_state + OR handoff.new_owner_id <> p_new_owner_id + OR handoff.new_lease_epoch <> p_new_lease_epoch + OR handoff.terminal_watermark <> p_terminal_watermark + OR handoff.expires_at <= clock_timestamp() THEN + RAISE EXCEPTION 'Phase 9.2 handoff/CAS request mismatch or expired'; + END IF; + IF p_new_state = 'RUST_PRIMARY' + AND handoff.prerequisite_bundle_id <> p_prerequisite_bundle_id THEN + RAISE EXCEPTION 'Phase 9.2 prerequisite bundle mismatch'; + END IF; + + SELECT * INTO transitioned + FROM qdl_transition_authority( + p_transition_id, p_slice_id, p_expected_state, p_expected_revision, + p_expected_owner_id, p_expected_lease_epoch, + p_expected_partition_plan_epoch, p_new_state, p_new_owner_id, + p_new_lease_epoch, p_terminal_watermark, p_prerequisite_bundle_id, + p_hold_until, p_actor, p_reason + ); + RETURN transitioned; +END; +$$; + +REVOKE EXECUTE ON FUNCTION qdl_transition_authority_v2( + UUID, UUID, TEXT, TEXT, BIGINT, TEXT, BIGINT, BIGINT, TEXT, TEXT, + BIGINT, BIGINT, UUID, TIMESTAMPTZ, TEXT, TEXT +) FROM PUBLIC; + +COMMIT; diff --git a/migrations/postgres/0008_phase93_hold_close_expand.sql b/migrations/postgres/0008_phase93_hold_close_expand.sql new file mode 100644 index 0000000..21f3b2c --- /dev/null +++ b/migrations/postgres/0008_phase93_hold_close_expand.sql @@ -0,0 +1,724 @@ +BEGIN; + +CREATE TABLE IF NOT EXISTS qdl_primary_holds ( + hold_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL + REFERENCES qdl_production_prerequisite_bundles(bundle_id), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + started_at TIMESTAMPTZ NOT NULL, + required_until TIMESTAMPTZ NOT NULL, + policy_digest TEXT NOT NULL CHECK (policy_digest ~ '^[0-9a-f]{64}$'), + minimum_duration_seconds BIGINT NOT NULL CHECK (minimum_duration_seconds > 0), + max_sample_gap_seconds BIGINT NOT NULL CHECK (max_sample_gap_seconds > 0), + max_lag_ms BIGINT NOT NULL CHECK (max_lag_ms > 0), + max_freshness_ms BIGINT NOT NULL CHECK (max_freshness_ms > 0), + max_queue_depth BIGINT NOT NULL CHECK (max_queue_depth > 0), + max_spool_bytes BIGINT NOT NULL CHECK (max_spool_bytes > 0), + max_cpu_percent DOUBLE PRECISION NOT NULL + CHECK (max_cpu_percent > 0 AND max_cpu_percent <= 100), + max_rss_mb DOUBLE PRECISION NOT NULL CHECK (max_rss_mb > 0), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (required_until > started_at), + CHECK ( + extract(epoch FROM (required_until - started_at)) + >= minimum_duration_seconds + ), + UNIQUE ( + slice_id, candidate_digest, owner_id, authority_revision, + lease_epoch, partition_plan_epoch, hold_id + ) +); + +CREATE TABLE IF NOT EXISTS qdl_primary_hold_observations ( + observation_id UUID PRIMARY KEY, + hold_id UUID NOT NULL REFERENCES qdl_primary_holds(hold_id), + slice_id TEXT NOT NULL, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + sequence BIGINT NOT NULL CHECK (sequence > 0), + observed_at TIMESTAMPTZ NOT NULL, + last_watermark BIGINT NOT NULL CHECK (last_watermark >= 0), + semantic_mismatches BIGINT NOT NULL DEFAULT 0 CHECK (semantic_mismatches >= 0), + open_gaps BIGINT NOT NULL DEFAULT 0 CHECK (open_gaps >= 0), + duplicate_external_writes BIGINT NOT NULL DEFAULT 0 + CHECK (duplicate_external_writes >= 0), + accepted_stale_writer_writes BIGINT NOT NULL DEFAULT 0 + CHECK (accepted_stale_writer_writes >= 0), + authority_ambiguities BIGINT NOT NULL DEFAULT 0 + CHECK (authority_ambiguities >= 0), + durable_ack_failures BIGINT NOT NULL DEFAULT 0 + CHECK (durable_ack_failures >= 0), + projection_mismatches BIGINT NOT NULL DEFAULT 0 + CHECK (projection_mismatches >= 0), + consumer_checkpoint_regressions BIGINT NOT NULL DEFAULT 0 + CHECK (consumer_checkpoint_regressions >= 0), + unexplained_quality_failures BIGINT NOT NULL DEFAULT 0 + CHECK (unexplained_quality_failures >= 0), + lag_ms BIGINT NOT NULL DEFAULT 0 CHECK (lag_ms >= 0), + freshness_ms BIGINT NOT NULL DEFAULT 0 CHECK (freshness_ms >= 0), + queue_depth BIGINT NOT NULL DEFAULT 0 CHECK (queue_depth >= 0), + spool_bytes BIGINT NOT NULL DEFAULT 0 CHECK (spool_bytes >= 0), + cpu_percent DOUBLE PRECISION NOT NULL DEFAULT 0 CHECK (cpu_percent >= 0), + rss_mb DOUBLE PRECISION NOT NULL DEFAULT 0 CHECK (rss_mb >= 0), + registered_consumers BIGINT NOT NULL CHECK (registered_consumers > 0), + healthy_consumers BIGINT NOT NULL CHECK ( + healthy_consumers >= 0 AND healthy_consumers <= registered_consumers + ), + checkpoint_watermark BIGINT NOT NULL CHECK (checkpoint_watermark >= 0), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (hold_id, sequence) +); + +CREATE INDEX IF NOT EXISTS qdl_primary_hold_observation_time_idx + ON qdl_primary_hold_observations (hold_id, observed_at); + +CREATE OR REPLACE FUNCTION qdl_validate_primary_hold_observation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + hold_row qdl_primary_holds%ROWTYPE; + previous_row qdl_primary_hold_observations%ROWTYPE; + expected_sequence BIGINT; +BEGIN + SELECT * INTO hold_row + FROM qdl_primary_holds AS hold + WHERE hold.hold_id = NEW.hold_id + FOR SHARE; + IF NOT FOUND + OR NEW.slice_id <> hold_row.slice_id + OR NEW.candidate_digest <> hold_row.candidate_digest + OR NEW.owner_id <> hold_row.owner_id + OR NEW.authority_revision <> hold_row.authority_revision + OR NEW.lease_epoch <> hold_row.lease_epoch + OR NEW.partition_plan_epoch <> hold_row.partition_plan_epoch THEN + RAISE EXCEPTION 'Phase 9.3 hold observation identity mismatch'; + END IF; + + SELECT * INTO previous_row + FROM qdl_primary_hold_observations AS observation + WHERE observation.hold_id = NEW.hold_id + ORDER BY observation.sequence DESC + LIMIT 1 + FOR SHARE; + + expected_sequence := COALESCE(previous_row.sequence, 0) + 1; + IF NEW.sequence <> expected_sequence THEN + RAISE EXCEPTION 'Phase 9.3 hold observation sequence is not contiguous'; + END IF; + IF NEW.observed_at <= COALESCE(previous_row.observed_at, hold_row.started_at) THEN + RAISE EXCEPTION 'Phase 9.3 hold observation time is not monotonic'; + END IF; + IF extract(epoch FROM ( + NEW.observed_at - COALESCE(previous_row.observed_at, hold_row.started_at) + )) > hold_row.max_sample_gap_seconds THEN + RAISE EXCEPTION 'Phase 9.3 hold observation gap exceeds policy'; + END IF; + IF previous_row.observation_id IS NOT NULL + AND NEW.last_watermark < previous_row.last_watermark THEN + RAISE EXCEPTION 'Phase 9.3 hold watermark regressed'; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_primary_hold_observation_guard + ON qdl_primary_hold_observations; +CREATE TRIGGER qdl_primary_hold_observation_guard +BEFORE INSERT ON qdl_primary_hold_observations +FOR EACH ROW EXECUTE FUNCTION qdl_validate_primary_hold_observation(); + +CREATE TABLE IF NOT EXISTS qdl_primary_hold_decisions ( + decision_id UUID PRIMARY KEY, + hold_id UUID NOT NULL REFERENCES qdl_primary_holds(hold_id), + status TEXT NOT NULL CHECK (status IN ('IN_PROGRESS', 'PASSED', 'BLOCKED')), + reason TEXT NOT NULL CHECK (btrim(reason) <> ''), + scope TEXT NOT NULL CHECK (scope IN ('TEST_REHEARSAL', 'PRODUCTION')), + production_authorized BOOLEAN NOT NULL, + slice_id TEXT NOT NULL, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL, + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + policy_digest TEXT NOT NULL CHECK (policy_digest ~ '^[0-9a-f]{64}$'), + first_observed_at TIMESTAMPTZ, + last_observed_at TIMESTAMPTZ, + observation_count BIGINT NOT NULL CHECK (observation_count >= 0), + terminal_watermark BIGINT CHECK (terminal_watermark >= 0), + decided_at TIMESTAMPTZ NOT NULL, + decision_sha256 TEXT NOT NULL CHECK (decision_sha256 ~ '^[0-9a-f]{64}$'), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK ( + production_authorized + = (scope = 'PRODUCTION' AND status = 'PASSED') + ), + CHECK ( + (observation_count = 0 AND first_observed_at IS NULL + AND last_observed_at IS NULL AND terminal_watermark IS NULL) + OR + (observation_count > 0 AND first_observed_at IS NOT NULL + AND last_observed_at IS NOT NULL AND terminal_watermark IS NOT NULL) + ), + UNIQUE (hold_id, decision_sha256) +); + +CREATE OR REPLACE FUNCTION qdl_validate_primary_hold_decision() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + hold_row qdl_primary_holds%ROWTYPE; + summary RECORD; +BEGIN + SELECT * INTO hold_row + FROM qdl_primary_holds AS hold + WHERE hold.hold_id = NEW.hold_id + FOR SHARE; + IF NOT FOUND + OR NEW.slice_id <> hold_row.slice_id + OR NEW.candidate_digest <> hold_row.candidate_digest + OR NEW.prerequisite_bundle_id <> hold_row.prerequisite_bundle_id + OR NEW.owner_id <> hold_row.owner_id + OR NEW.authority_revision <> hold_row.authority_revision + OR NEW.lease_epoch <> hold_row.lease_epoch + OR NEW.partition_plan_epoch <> hold_row.partition_plan_epoch + OR NEW.policy_digest <> hold_row.policy_digest THEN + RAISE EXCEPTION 'Phase 9.3 hold decision identity mismatch'; + END IF; + + SELECT + count(*) AS observation_count, + min(observed_at) AS first_observed_at, + max(observed_at) AS last_observed_at, + (array_agg(last_watermark ORDER BY sequence DESC))[1] + AS terminal_watermark, + bool_or( + semantic_mismatches <> 0 + OR open_gaps <> 0 + OR duplicate_external_writes <> 0 + OR accepted_stale_writer_writes <> 0 + OR authority_ambiguities <> 0 + OR durable_ack_failures <> 0 + OR projection_mismatches <> 0 + OR consumer_checkpoint_regressions <> 0 + OR unexplained_quality_failures <> 0 + OR lag_ms > hold_row.max_lag_ms + OR freshness_ms > hold_row.max_freshness_ms + OR queue_depth > hold_row.max_queue_depth + OR spool_bytes > hold_row.max_spool_bytes + OR cpu_percent > hold_row.max_cpu_percent + OR rss_mb > hold_row.max_rss_mb + OR healthy_consumers <> registered_consumers + OR checkpoint_watermark < last_watermark + ) AS breached + INTO summary + FROM qdl_primary_hold_observations AS observation + WHERE observation.hold_id = NEW.hold_id; + + IF NEW.observation_count <> summary.observation_count + OR NEW.first_observed_at IS DISTINCT FROM summary.first_observed_at + OR NEW.last_observed_at IS DISTINCT FROM summary.last_observed_at + OR NEW.terminal_watermark IS DISTINCT FROM summary.terminal_watermark THEN + RAISE EXCEPTION 'Phase 9.3 hold decision summary mismatch'; + END IF; + IF NEW.status = 'PASSED' AND ( + summary.observation_count = 0 + OR summary.breached + OR summary.last_observed_at < hold_row.required_until + OR NEW.decided_at < hold_row.required_until + OR NEW.reason <> 'PASS' + ) THEN + RAISE EXCEPTION 'Phase 9.3 passing hold decision is not supported by evidence'; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_primary_hold_decision_guard + ON qdl_primary_hold_decisions; +CREATE TRIGGER qdl_primary_hold_decision_guard +BEFORE INSERT ON qdl_primary_hold_decisions +FOR EACH ROW EXECUTE FUNCTION qdl_validate_primary_hold_decision(); + +CREATE TABLE IF NOT EXISTS qdl_consumer_registry_snapshots ( + snapshot_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + checkpoint_count BIGINT NOT NULL CHECK (checkpoint_count > 0), + ready_checkpoint_count BIGINT NOT NULL CHECK ( + ready_checkpoint_count = checkpoint_count + ), + minimum_checkpoint_watermark BIGINT NOT NULL + CHECK (minimum_checkpoint_watermark >= 0), + checkpoint_regressions BIGINT NOT NULL + CHECK (checkpoint_regressions = 0), + unresolved_migrations BIGINT NOT NULL CHECK (unresolved_migrations = 0), + rollback_ready BOOLEAN NOT NULL CHECK (rollback_ready), + registry_sha256 TEXT NOT NULL CHECK (registry_sha256 ~ '^[0-9a-f]{64}$'), + details JSONB NOT NULL CHECK (jsonb_typeof(details) = 'object'), + observed_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (slice_id, authority_revision, registry_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_registry_snapshots ( + snapshot_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + state TEXT NOT NULL CHECK (state = 'RUST_PRIMARY'), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL + REFERENCES qdl_production_prerequisite_bundles(bundle_id), + current_watermark BIGINT NOT NULL CHECK (current_watermark >= 0), + public_write_allowed BOOLEAN NOT NULL CHECK (public_write_allowed), + legacy_write_allowed BOOLEAN NOT NULL CHECK (legacy_write_allowed), + registry_sha256 TEXT NOT NULL CHECK (registry_sha256 ~ '^[0-9a-f]{64}$'), + observed_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (slice_id, authority_revision, registry_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_rollback_rehearsals ( + rehearsal_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + rollback_manifest_digest TEXT NOT NULL + CHECK (rollback_manifest_digest ~ '^[0-9a-f]{64}$'), + reconciled_through_watermark BIGINT NOT NULL + CHECK (reconciled_through_watermark >= 0), + rto_ms DOUBLE PRECISION NOT NULL CHECK (rto_ms > 0), + status TEXT NOT NULL CHECK (status = 'PASS'), + production_scope BOOLEAN NOT NULL CHECK (production_scope), + rehearsal_sha256 TEXT NOT NULL CHECK (rehearsal_sha256 ~ '^[0-9a-f]{64}$'), + observed_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (expires_at > observed_at), + UNIQUE (slice_id, authority_revision, rehearsal_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_closure_approvals ( + approval_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL + REFERENCES qdl_production_prerequisite_bundles(bundle_id), + hold_id UUID NOT NULL REFERENCES qdl_primary_holds(hold_id), + hold_policy_digest TEXT NOT NULL + CHECK (hold_policy_digest ~ '^[0-9a-f]{64}$'), + decision TEXT NOT NULL CHECK (decision = 'APPROVE'), + allow_close_rollback_window BOOLEAN NOT NULL + CHECK (allow_close_rollback_window), + repository_cleanup_approved BOOLEAN NOT NULL + CHECK (NOT repository_cleanup_approved), + operator TEXT NOT NULL CHECK (btrim(operator) <> ''), + change_ticket TEXT NOT NULL CHECK (btrim(change_ticket) <> ''), + approval_sha256 TEXT NOT NULL CHECK (approval_sha256 ~ '^[0-9a-f]{64}$'), + approved_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (expires_at > approved_at), + UNIQUE (slice_id, hold_id, approval_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_closures ( + closure_id UUID PRIMARY KEY, + closure_sha256 TEXT NOT NULL CHECK (closure_sha256 ~ '^[0-9a-f]{64}$'), + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL, + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + hold_decision_id UUID NOT NULL + REFERENCES qdl_primary_hold_decisions(decision_id), + hold_decision_digest TEXT NOT NULL CHECK (hold_decision_digest ~ '^[0-9a-f]{64}$'), + consumer_registry_snapshot_id UUID NOT NULL + REFERENCES qdl_consumer_registry_snapshots(snapshot_id), + consumer_registry_digest TEXT NOT NULL CHECK (consumer_registry_digest ~ '^[0-9a-f]{64}$'), + authority_registry_snapshot_id UUID NOT NULL + REFERENCES qdl_authority_registry_snapshots(snapshot_id), + authority_registry_digest TEXT NOT NULL CHECK (authority_registry_digest ~ '^[0-9a-f]{64}$'), + rollback_rehearsal_id UUID NOT NULL + REFERENCES qdl_rollback_rehearsals(rehearsal_id), + rollback_rehearsal_digest TEXT NOT NULL CHECK (rollback_rehearsal_digest ~ '^[0-9a-f]{64}$'), + approval_id UUID NOT NULL REFERENCES qdl_closure_approvals(approval_id), + approval_digest TEXT NOT NULL CHECK (approval_digest ~ '^[0-9a-f]{64}$'), + approved_by TEXT NOT NULL CHECK (btrim(approved_by) <> ''), + change_ticket TEXT NOT NULL CHECK (btrim(change_ticket) <> ''), + approved_at TIMESTAMPTZ NOT NULL, + approval_expires_at TIMESTAMPTZ NOT NULL, + closed_at TIMESTAMPTZ NOT NULL, + production_authorized BOOLEAN NOT NULL CHECK (production_authorized), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (approval_expires_at > approved_at), + CHECK (closed_at >= approved_at AND closed_at < approval_expires_at), + UNIQUE (slice_id, authority_revision) +); + +CREATE OR REPLACE FUNCTION qdl_close_authority_window( + p_closure_id UUID, + p_closure_sha256 TEXT, + p_hold_decision_id UUID, + p_consumer_registry_snapshot_id UUID, + p_authority_registry_snapshot_id UUID, + p_rollback_rehearsal_id UUID, + p_approval_id UUID, + p_closed_at TIMESTAMPTZ +) +RETURNS qdl_authority_closures +LANGUAGE plpgsql +AS $$ +DECLARE + current_row qdl_authority_slices%ROWTYPE; + decision_row qdl_primary_hold_decisions%ROWTYPE; + hold_row qdl_primary_holds%ROWTYPE; + bundle_row qdl_production_prerequisite_bundles%ROWTYPE; + consumer_row qdl_consumer_registry_snapshots%ROWTYPE; + authority_row qdl_authority_registry_snapshots%ROWTYPE; + rollback_row qdl_rollback_rehearsals%ROWTYPE; + approval_row qdl_closure_approvals%ROWTYPE; + closure_row qdl_authority_closures%ROWTYPE; +BEGIN + SELECT * INTO decision_row + FROM qdl_primary_hold_decisions AS decision + WHERE decision.decision_id = p_hold_decision_id + FOR SHARE; + SELECT * INTO hold_row + FROM qdl_primary_holds AS hold + WHERE hold.hold_id = decision_row.hold_id + FOR SHARE; + SELECT * INTO current_row + FROM qdl_authority_slices AS authority_slice + WHERE authority_slice.slice_id = decision_row.slice_id + FOR UPDATE; + SELECT * INTO bundle_row + FROM qdl_production_prerequisite_bundles AS bundle + WHERE bundle.bundle_id = decision_row.prerequisite_bundle_id + FOR SHARE; + SELECT * INTO consumer_row + FROM qdl_consumer_registry_snapshots AS snapshot + WHERE snapshot.snapshot_id = p_consumer_registry_snapshot_id + FOR SHARE; + SELECT * INTO authority_row + FROM qdl_authority_registry_snapshots AS snapshot + WHERE snapshot.snapshot_id = p_authority_registry_snapshot_id + FOR SHARE; + SELECT * INTO rollback_row + FROM qdl_rollback_rehearsals AS rehearsal + WHERE rehearsal.rehearsal_id = p_rollback_rehearsal_id + FOR SHARE; + SELECT * INTO approval_row + FROM qdl_closure_approvals AS approval + WHERE approval.approval_id = p_approval_id + FOR SHARE; + + IF decision_row.decision_id IS NULL + OR hold_row.hold_id IS NULL + OR decision_row.status <> 'PASSED' + OR decision_row.scope <> 'PRODUCTION' + OR NOT decision_row.production_authorized + OR decision_row.reason <> 'PASS' + OR decision_row.hold_id <> hold_row.hold_id + OR decision_row.slice_id <> hold_row.slice_id + OR decision_row.candidate_digest <> hold_row.candidate_digest + OR decision_row.prerequisite_bundle_id <> hold_row.prerequisite_bundle_id + OR decision_row.owner_id <> hold_row.owner_id + OR decision_row.authority_revision <> hold_row.authority_revision + OR decision_row.lease_epoch <> hold_row.lease_epoch + OR decision_row.partition_plan_epoch <> hold_row.partition_plan_epoch + OR decision_row.policy_digest <> hold_row.policy_digest THEN + RAISE EXCEPTION 'Phase 9.3 passing production hold decision is required'; + END IF; + IF current_row.slice_id IS NULL + OR current_row.state <> 'RUST_PRIMARY' + OR current_row.candidate_digest <> decision_row.candidate_digest + OR current_row.prerequisite_bundle_id <> decision_row.prerequisite_bundle_id + OR current_row.owner_id <> decision_row.owner_id + OR current_row.authority_revision <> decision_row.authority_revision + OR current_row.lease_epoch <> decision_row.lease_epoch + OR current_row.partition_plan_epoch <> decision_row.partition_plan_epoch THEN + RAISE EXCEPTION 'Phase 9.3 authority closure CAS mismatch'; + END IF; + IF bundle_row.bundle_id IS NULL + OR bundle_row.decision <> 'GO' + OR bundle_row.candidate_digest <> current_row.candidate_digest + OR bundle_row.expires_at <= p_closed_at THEN + RAISE EXCEPTION 'Phase 9.3 closure prerequisite bundle is invalid'; + END IF; + IF consumer_row.snapshot_id IS NULL + OR consumer_row.slice_id <> current_row.slice_id + OR consumer_row.authority_revision <> current_row.authority_revision + OR consumer_row.minimum_checkpoint_watermark < authority_row.current_watermark + OR consumer_row.observed_at > p_closed_at + OR p_closed_at - consumer_row.observed_at > interval '5 minutes' THEN + RAISE EXCEPTION 'Phase 9.3 consumer registry snapshot is invalid'; + END IF; + IF authority_row.snapshot_id IS NULL + OR authority_row.slice_id <> current_row.slice_id + OR authority_row.state <> current_row.state + OR authority_row.owner_id <> current_row.owner_id + OR authority_row.authority_revision <> current_row.authority_revision + OR authority_row.lease_epoch <> current_row.lease_epoch + OR authority_row.partition_plan_epoch <> current_row.partition_plan_epoch + OR authority_row.candidate_digest <> current_row.candidate_digest + OR authority_row.prerequisite_bundle_id <> current_row.prerequisite_bundle_id + OR authority_row.observed_at > p_closed_at + OR p_closed_at - authority_row.observed_at > interval '5 minutes' THEN + RAISE EXCEPTION 'Phase 9.3 authority registry snapshot is invalid'; + END IF; + IF rollback_row.rehearsal_id IS NULL + OR rollback_row.slice_id <> current_row.slice_id + OR rollback_row.candidate_digest <> current_row.candidate_digest + OR rollback_row.owner_id <> current_row.owner_id + OR rollback_row.authority_revision <> current_row.authority_revision + OR rollback_row.lease_epoch <> current_row.lease_epoch + OR rollback_row.partition_plan_epoch <> current_row.partition_plan_epoch + OR rollback_row.rollback_manifest_digest <> current_row.rollback_manifest_digest + OR rollback_row.reconciled_through_watermark < authority_row.current_watermark + OR rollback_row.expires_at <= p_closed_at THEN + RAISE EXCEPTION 'Phase 9.3 rollback rehearsal is invalid'; + END IF; + IF approval_row.approval_id IS NULL + OR approval_row.slice_id <> current_row.slice_id + OR approval_row.candidate_digest <> current_row.candidate_digest + OR approval_row.prerequisite_bundle_id <> current_row.prerequisite_bundle_id + OR approval_row.hold_id <> hold_row.hold_id + OR approval_row.hold_policy_digest <> hold_row.policy_digest + OR approval_row.approved_at > p_closed_at + OR approval_row.expires_at <= p_closed_at THEN + RAISE EXCEPTION 'Phase 9.3 closure approval is invalid or expired'; + END IF; + + IF p_closure_sha256 !~ '^[0-9a-f]{64}$' THEN + RAISE EXCEPTION 'Phase 9.3 closure digest is invalid'; + END IF; + + INSERT INTO qdl_authority_closures ( + closure_id, closure_sha256, slice_id, candidate_digest, prerequisite_bundle_id, + owner_id, authority_revision, lease_epoch, partition_plan_epoch, + hold_decision_id, hold_decision_digest, + consumer_registry_snapshot_id, consumer_registry_digest, + authority_registry_snapshot_id, authority_registry_digest, + rollback_rehearsal_id, rollback_rehearsal_digest, + approval_id, approval_digest, approved_by, change_ticket, + approved_at, approval_expires_at, closed_at, production_authorized + ) VALUES ( + p_closure_id, p_closure_sha256, current_row.slice_id, + current_row.candidate_digest, + current_row.prerequisite_bundle_id, current_row.owner_id, + current_row.authority_revision, current_row.lease_epoch, + current_row.partition_plan_epoch, decision_row.decision_id, + decision_row.decision_sha256, consumer_row.snapshot_id, + consumer_row.registry_sha256, authority_row.snapshot_id, + authority_row.registry_sha256, rollback_row.rehearsal_id, + rollback_row.rehearsal_sha256, approval_row.approval_id, + approval_row.approval_sha256, approval_row.operator, + approval_row.change_ticket, approval_row.approved_at, + approval_row.expires_at, p_closed_at, TRUE + ) + RETURNING * INTO closure_row; + RETURN closure_row; +END; +$$; + +CREATE TABLE IF NOT EXISTS qdl_expansion_candidates ( + expansion_id UUID PRIMARY KEY, + parent_closure_id UUID NOT NULL + REFERENCES qdl_authority_closures(closure_id), + parent_slice_id TEXT NOT NULL, + parent_candidate_digest TEXT NOT NULL CHECK (parent_candidate_digest ~ '^[0-9a-f]{64}$'), + parent_closure_digest TEXT NOT NULL CHECK (parent_closure_digest ~ '^[0-9a-f]{64}$'), + expansion_type TEXT NOT NULL CHECK (expansion_type IN ( + 'INSTRUMENT_PARTITION', 'BBO', 'L2_BOOK', 'BAR_LIFECYCLE', + 'VENUE_MARKET' + )), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + scope_digest TEXT NOT NULL CHECK (scope_digest ~ '^[0-9a-f]{64}$'), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + required_gates TEXT[] NOT NULL, + status TEXT NOT NULL CHECK (status = 'INDEPENDENT_CERTIFICATION_REQUIRED'), + transitive_evidence_allowed BOOLEAN NOT NULL + CHECK (NOT transitive_evidence_allowed), + public_write_allowed BOOLEAN NOT NULL CHECK (NOT public_write_allowed), + legacy_write_allowed BOOLEAN NOT NULL CHECK (NOT legacy_write_allowed), + created_at TIMESTAMPTZ NOT NULL, + recorded_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (candidate_digest <> parent_candidate_digest), + CHECK (cardinality(required_gates) > 0), + UNIQUE (parent_closure_id, expansion_type, candidate_digest, scope_digest) +); + +CREATE OR REPLACE FUNCTION qdl_validate_expansion_candidate() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + closure_row qdl_authority_closures%ROWTYPE; + expected_gates TEXT[]; +BEGIN + SELECT * INTO closure_row + FROM qdl_authority_closures AS closure + WHERE closure.closure_id = NEW.parent_closure_id + FOR SHARE; + IF NOT FOUND + OR NEW.parent_slice_id <> closure_row.slice_id + OR NEW.parent_candidate_digest <> closure_row.candidate_digest + OR NEW.parent_closure_digest <> closure_row.closure_sha256 + OR NOT closure_row.production_authorized THEN + RAISE EXCEPTION 'Phase 9.3 expansion parent closure mismatch'; + END IF; + expected_gates := CASE NEW.expansion_type + WHEN 'INSTRUMENT_PARTITION' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'exact_frame_parity', + 'partition_churn', 'provider_authentic_source', 'rollback', + 'source_capacity' + ] + WHEN 'BBO' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'coalescing_policy', + 'exact_frame_parity', 'freshness', 'ordering_reconnect', + 'provider_authentic_source', 'quote_identity', 'rollback' + ] + WHEN 'L2_BOOK' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'checksum', + 'exact_frame_parity', 'lossless_backpressure', + 'provider_authentic_source', 'resync', 'rollback', + 'snapshot_delta_sequence' + ] + WHEN 'BAR_LIFECYCLE' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'close_time_semantics', + 'exact_frame_parity', 'final_revision_lineage', + 'provider_authentic_source', 'replay', 'rollback' + ] + WHEN 'VENUE_MARKET' THEN ARRAY[ + 'adapter_capability', 'authority_handoff', 'capacity_headroom', + 'disaster_recovery', 'entitlement', 'exact_frame_parity', + 'instrument_identity', 'provider_authentic_source', + 'provider_semantics', 'rollback' + ] + END; + IF NEW.required_gates <> expected_gates THEN + RAISE EXCEPTION 'Phase 9.3 expansion requires independent capability gates'; + END IF; + IF NEW.expansion_type = 'INSTRUMENT_PARTITION' + AND NEW.partition_plan_epoch <= closure_row.partition_plan_epoch THEN + RAISE EXCEPTION 'Phase 9.3 instrument expansion requires a new partition epoch'; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_expansion_candidate_guard + ON qdl_expansion_candidates; +CREATE TRIGGER qdl_expansion_candidate_guard +BEFORE INSERT ON qdl_expansion_candidates +FOR EACH ROW EXECUTE FUNCTION qdl_validate_expansion_candidate(); + +CREATE TABLE IF NOT EXISTS qdl_runtime_decommission_decisions ( + decision_id UUID PRIMARY KEY, + runtime_id TEXT NOT NULL CHECK (btrim(runtime_id) <> ''), + owned_slice_count BIGINT NOT NULL CHECK (owned_slice_count >= 0), + rollback_reference_count BIGINT NOT NULL CHECK (rollback_reference_count >= 0), + consumer_dependency_count BIGINT NOT NULL CHECK (consumer_dependency_count >= 0), + all_replacement_windows_closed BOOLEAN NOT NULL, + repository_cleanup_approved BOOLEAN NOT NULL, + shared_knowledge_retained BOOLEAN NOT NULL, + allowed BOOLEAN NOT NULL, + reason TEXT NOT NULL CHECK (btrim(reason) <> ''), + decided_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK ( + NOT allowed OR ( + owned_slice_count = 0 + AND rollback_reference_count = 0 + AND consumer_dependency_count = 0 + AND all_replacement_windows_closed + AND repository_cleanup_approved + AND shared_knowledge_retained + ) + ) +); + +CREATE OR REPLACE FUNCTION qdl_reject_phase93_registry_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'Phase 9.3 hold/closure/expansion evidence is append-only'; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_primary_hold_immutable ON qdl_primary_holds; +CREATE TRIGGER qdl_primary_hold_immutable +BEFORE UPDATE OR DELETE ON qdl_primary_holds +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_primary_hold_observation_immutable + ON qdl_primary_hold_observations; +CREATE TRIGGER qdl_primary_hold_observation_immutable +BEFORE UPDATE OR DELETE ON qdl_primary_hold_observations +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_primary_hold_decision_immutable + ON qdl_primary_hold_decisions; +CREATE TRIGGER qdl_primary_hold_decision_immutable +BEFORE UPDATE OR DELETE ON qdl_primary_hold_decisions +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_consumer_registry_snapshot_immutable + ON qdl_consumer_registry_snapshots; +CREATE TRIGGER qdl_consumer_registry_snapshot_immutable +BEFORE UPDATE OR DELETE ON qdl_consumer_registry_snapshots +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_authority_registry_snapshot_immutable + ON qdl_authority_registry_snapshots; +CREATE TRIGGER qdl_authority_registry_snapshot_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_registry_snapshots +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_rollback_rehearsal_immutable + ON qdl_rollback_rehearsals; +CREATE TRIGGER qdl_rollback_rehearsal_immutable +BEFORE UPDATE OR DELETE ON qdl_rollback_rehearsals +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_closure_approval_immutable + ON qdl_closure_approvals; +CREATE TRIGGER qdl_closure_approval_immutable +BEFORE UPDATE OR DELETE ON qdl_closure_approvals +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_authority_closure_immutable + ON qdl_authority_closures; +CREATE TRIGGER qdl_authority_closure_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_closures +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_expansion_candidate_immutable + ON qdl_expansion_candidates; +CREATE TRIGGER qdl_expansion_candidate_immutable +BEFORE UPDATE OR DELETE ON qdl_expansion_candidates +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_runtime_decommission_decision_immutable + ON qdl_runtime_decommission_decisions; +CREATE TRIGGER qdl_runtime_decommission_decision_immutable +BEFORE UPDATE OR DELETE ON qdl_runtime_decommission_decisions +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +REVOKE EXECUTE ON FUNCTION qdl_close_authority_window( + UUID, TEXT, UUID, UUID, UUID, UUID, UUID, TIMESTAMPTZ +) FROM PUBLIC; + +COMMIT; diff --git a/migrations/postgres/README.md b/migrations/postgres/README.md index df1e39b..ef6c911 100644 --- a/migrations/postgres/README.md +++ b/migrations/postgres/README.md @@ -17,3 +17,21 @@ Phase 7 adds `0005_phase7_data_plane_identity.sql`. It binds an authenticated workload subject and environment to one immutable consumer-manifest revision, including allowed purposes, data-plane permissions, execution-dependency policy and bounded quotas. It does not store market events or alter V1 tables. + +Phase 9.0-C adds `0006_phase9_authority_prerequisites.sql`. It stores immutable +prerequisite bundles, one persistent authority record per exact slice and an +append-only transition audit. Its CAS transition function rejects stale +state/revision/owner/lease/partition expectations, binds release provenance, +requires a terminal watermark and approval hold window, and cannot enter Rust +canary or primary without a non-expired `GO` bundle bound to the candidate digest. +The migration is dark: it neither seeds an approval nor changes V1 authority. + + +Phase 9.3 adds `0008_phase93_hold_close_expand.sql`. It stores immutable +primary-hold observations/decisions, frozen consumer and authority registry +snapshots, rollback rehearsal and operator approval evidence, one governed +rollback-window closure record and independently uncertified expansion +manifests. The closure function locks and rechecks the exact `RUST_PRIMARY` +authority row but never updates authority ownership, revision, lease or +watermark. Expansion rows are write-disabled and cannot inherit parent +certification. diff --git a/qdl/canary/__init__.py b/qdl/canary/__init__.py index 07fe466..fcf7acf 100644 --- a/qdl/canary/__init__.py +++ b/qdl/canary/__init__.py @@ -1,13 +1,97 @@ -from qdl.canary.consumer import ( - CanaryRunResult, - DeterministicPaperSignalState, - PaperAlphaCanary, - sdk_requirement, +from qdl.canary.phase92 import ( + AcceptedHandoff, + HandoffDirection, + PrimaryAuthorization, + PrimaryAuthorizationMode, + ProductionPrimaryAuthorizer, + TerminalOwnerCheckpoint, +) +from qdl.canary.phase93 import ( + AuthorityRegistrySnapshot, + ClosureApproval, + ClosureAuthorization, + ConsumerCheckpoint, + ConsumerRegistrySnapshot, + DecommissionDecision, + DecommissionRequest, + ExpansionManifest, + ExpansionType, + HoldScope, + HoldStatus, + PrimaryHoldDecision, + PrimaryHoldEvaluator, + PrimaryHoldIdentity, + PrimaryHoldObservation, + PrimaryHoldPolicy, + ProductionClosureAuthorizer, + RollbackRehearsalEvidence, + RollbackWindowClosure, + assess_decommission, +) +from qdl.canary.phase9 import ( + CanaryAuthorization, + CanaryAuthorizationMode, + CanaryGuardrailDecision, + CanaryGuardrailEngine, + CanaryGuardrailPolicy, + CanaryObservation, + ProductionCanaryAuthorizer, ) __all__ = [ + "AcceptedHandoff", + "AuthorityRegistrySnapshot", + "ClosureApproval", + "ClosureAuthorization", + "ConsumerCheckpoint", + "ConsumerRegistrySnapshot", + "DecommissionDecision", + "DecommissionRequest", + "ExpansionManifest", + "ExpansionType", + "HoldScope", + "HoldStatus", + "PrimaryHoldDecision", + "PrimaryHoldEvaluator", + "PrimaryHoldIdentity", + "PrimaryHoldObservation", + "PrimaryHoldPolicy", + "ProductionClosureAuthorizer", + "RollbackRehearsalEvidence", + "RollbackWindowClosure", + "assess_decommission", + "HandoffDirection", + "PrimaryAuthorization", + "PrimaryAuthorizationMode", + "ProductionPrimaryAuthorizer", + "TerminalOwnerCheckpoint", + "CanaryAuthorization", + "CanaryAuthorizationMode", + "CanaryGuardrailDecision", + "CanaryGuardrailEngine", + "CanaryGuardrailPolicy", + "CanaryObservation", "CanaryRunResult", "DeterministicPaperSignalState", "PaperAlphaCanary", + "ProductionCanaryAuthorizer", "sdk_requirement", ] + + +_CONSUMER_EXPORTS = { + "CanaryRunResult", + "DeterministicPaperSignalState", + "PaperAlphaCanary", + "sdk_requirement", +} + + +def __getattr__(name: str): + if name not in _CONSUMER_EXPORTS: + raise AttributeError(name) + from qdl.canary import consumer + + value = getattr(consumer, name) + globals()[name] = value + return value diff --git a/qdl/canary/phase9.py b/qdl/canary/phase9.py new file mode 100644 index 0000000..40fd6c1 --- /dev/null +++ b/qdl/canary/phase9.py @@ -0,0 +1,272 @@ +from __future__ import annotations + +import uuid +from dataclasses import dataclass +from enum import StrEnum +from typing import Any, Mapping + +from qdl.certification.prerequisites import CandidateSlice + + +_REQUIRED_PRODUCTION_GATES = frozenset({ + "replicated_durable_transport", + "production_observability", + "workload_identity_rbac_network", + "external_secret_rotation", + "signed_artifact_admission", + "postgres_pitr", + "object_store_restore", + "independent_failure_domain_dr", + "redis_projector_rebuild", + "consumer_registration_rollback", + "persistent_authority_sink_fencing", + "exact_slice_approval", +}) + + +class CanaryAuthorizationMode(StrEnum): + PRODUCTION = "PRODUCTION" + ISOLATED_REHEARSAL = "ISOLATED_REHEARSAL" + + +@dataclass(frozen=True, slots=True) +class CanaryAuthorization: + allowed: bool + production_authorized: bool + mode: CanaryAuthorizationMode + reason: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str | None + + +class ProductionCanaryAuthorizer: + """Fail-closed Phase 9.1 gate; it never mutates authority state.""" + + def __init__(self, *, max_decision_age_seconds: int = 900) -> None: + if max_decision_age_seconds <= 0: + raise ValueError("canary decision age must be positive") + self.max_decision_age_ns = max_decision_age_seconds * 1_000_000_000 + + def authorize( + self, + *, + candidate: CandidateSlice, + decision: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> CanaryAuthorization: + reason = self._validate( + candidate=candidate, + decision=decision, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + return CanaryAuthorization( + allowed=reason == "AUTHORIZED", + production_authorized=reason == "AUTHORIZED", + mode=CanaryAuthorizationMode.PRODUCTION, + reason=reason, + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=str(decision.get("bundle_id") or "") or None, + ) + + @staticmethod + def authorize_isolated_rehearsal(*, candidate: CandidateSlice) -> CanaryAuthorization: + return CanaryAuthorization( + allowed=True, + production_authorized=False, + mode=CanaryAuthorizationMode.ISOLATED_REHEARSAL, + reason="ISOLATED_REHEARSAL_ONLY", + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=None, + ) + + def _validate( + self, + *, + candidate: CandidateSlice, + decision: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> str: + if now_ns <= 0: + return "INVALID_CLOCK" + if candidate.payload["authority_state"] != "RUST_SHADOW": + return "CANDIDATE_NOT_SHADOW" + if candidate.payload["public_write_allowed"] or candidate.payload["legacy_write_allowed"]: + return "CANDIDATE_WRITE_AUTHORITY_INVALID" + if decision.get("schema") != "qdl.production-prerequisite-decision.v1": + return "DECISION_SCHEMA_INVALID" + if decision.get("decision") != "GO": + return "PREREQUISITE_DECISION_NOT_GO" + if decision.get("candidate_digest") != candidate.digest: + return "CANDIDATE_DIGEST_MISMATCH" + if decision.get("slice_id") != candidate.payload["slice_id"]: + return "SLICE_MISMATCH" + bundle_id = str(decision.get("bundle_id") or "") + try: + uuid.UUID(bundle_id) + uuid.UUID(expected_bundle_id) + except ValueError: + return "PREREQUISITE_BUNDLE_INVALID" + if bundle_id != expected_bundle_id: + return "PREREQUISITE_BUNDLE_MISMATCH" + issued_at_ns = decision.get("issued_at_ns") + if not isinstance(issued_at_ns, int) or isinstance(issued_at_ns, bool) or issued_at_ns <= 0: + return "DECISION_TIME_INVALID" + if issued_at_ns > now_ns + 60_000_000_000: + return "DECISION_FROM_FUTURE" + if now_ns - issued_at_ns > self.max_decision_age_ns: + return "DECISION_EXPIRED" + gates = decision.get("gates") + if not isinstance(gates, list) or not gates: + return "GATE_RESULTS_MISSING" + gate_ids = [item.get("gate_id") for item in gates if isinstance(item, Mapping)] + if ( + len(gate_ids) != len(gates) + or len(gate_ids) != len(set(gate_ids)) + or set(gate_ids) != _REQUIRED_PRODUCTION_GATES + ): + return "GATE_RESULTS_INVALID" + if any(item.get("passed") is not True or item.get("reason") != "PASS" for item in gates): + return "GATE_NOT_PASSED" + if decision.get("passed") != len(gates) or decision.get("blocked") != 0: + return "GATE_SUMMARY_INVALID" + if decision.get("authority_state") != "RUST_SHADOW": + return "AUTHORITY_PRECONDITION_INVALID" + if decision.get("v1_unchanged") is not True: + return "V1_PRECONDITION_INVALID" + if decision.get("production_mutations") != 0: + return "PRODUCTION_MUTATION_DETECTED" + return "AUTHORIZED" + + +@dataclass(frozen=True, slots=True) +class CanaryGuardrailPolicy: + max_lag_ms: int + max_freshness_ms: int + max_cpu_percent: float + max_rss_mb: float + max_queue_depth: int + hold_down_seconds: int + + def __post_init__(self) -> None: + if any(value <= 0 for value in ( + self.max_lag_ms, + self.max_freshness_ms, + self.max_cpu_percent, + self.max_rss_mb, + self.max_queue_depth, + self.hold_down_seconds, + )): + raise ValueError("canary guardrail thresholds must be positive") + + +@dataclass(frozen=True, slots=True) +class CanaryObservation: + observed_at_ns: int + semantic_mismatches: int = 0 + open_gaps: int = 0 + duplicate_external_writes: int = 0 + stale_writer_attempts: int = 0 + authority_ambiguities: int = 0 + durable_ack_failures: int = 0 + lag_ms: int = 0 + freshness_ms: int = 0 + cpu_percent: float = 0.0 + rss_mb: float = 0.0 + queue_depth: int = 0 + + def __post_init__(self) -> None: + values = ( + self.observed_at_ns, + self.semantic_mismatches, + self.open_gaps, + self.duplicate_external_writes, + self.stale_writer_attempts, + self.authority_ambiguities, + self.durable_ack_failures, + self.lag_ms, + self.freshness_ms, + self.cpu_percent, + self.rss_mb, + self.queue_depth, + ) + if any(value < 0 for value in values) or self.observed_at_ns <= 0: + raise ValueError("canary observation values are invalid") + + +@dataclass(frozen=True, slots=True) +class CanaryGuardrailDecision: + allowed: bool + reason: str + blocked_at_ns: int | None + hold_until_ns: int | None + + +class CanaryGuardrailEngine: + def __init__(self, policy: CanaryGuardrailPolicy) -> None: + self.policy = policy + self._blocked_at_ns: int | None = None + self._hold_until_ns: int | None = None + self._first_failure_reason: str | None = None + + def evaluate(self, observation: CanaryObservation) -> CanaryGuardrailDecision: + if self._first_failure_reason is not None: + return self._decision(False, "EXPLICIT_RESET_REQUIRED") + reason = self._failure_reason(observation) + if reason is not None: + self._first_failure_reason = reason + self._blocked_at_ns = observation.observed_at_ns + self._hold_until_ns = ( + observation.observed_at_ns + + self.policy.hold_down_seconds * 1_000_000_000 + ) + return self._decision(False, reason) + return self._decision(True, "PASS") + + def reset_after_hold(self, observation: CanaryObservation) -> CanaryGuardrailDecision: + if self._first_failure_reason is None: + return self.evaluate(observation) + assert self._hold_until_ns is not None + if observation.observed_at_ns < self._hold_until_ns: + return self._decision(False, "HOLD_DOWN_ACTIVE") + reason = self._failure_reason(observation) + if reason is not None: + return self._decision(False, reason) + self._blocked_at_ns = None + self._hold_until_ns = None + self._first_failure_reason = None + return self._decision(True, "RESET_CONFIRMED") + + def _failure_reason(self, item: CanaryObservation) -> str | None: + zero_tolerance = ( + (item.semantic_mismatches, "SEMANTIC_MISMATCH"), + (item.open_gaps, "OPEN_GAP"), + (item.duplicate_external_writes, "DUPLICATE_EXTERNAL_WRITE"), + (item.stale_writer_attempts, "STALE_WRITER_ATTEMPT"), + (item.authority_ambiguities, "AUTHORITY_AMBIGUITY"), + (item.durable_ack_failures, "DURABLE_ACK_FAILURE"), + ) + for value, reason in zero_tolerance: + if value: + return reason + thresholds = ( + (item.lag_ms > self.policy.max_lag_ms, "LAG_THRESHOLD_EXCEEDED"), + (item.freshness_ms > self.policy.max_freshness_ms, "FRESHNESS_THRESHOLD_EXCEEDED"), + (item.cpu_percent > self.policy.max_cpu_percent, "CPU_THRESHOLD_EXCEEDED"), + (item.rss_mb > self.policy.max_rss_mb, "RSS_THRESHOLD_EXCEEDED"), + (item.queue_depth > self.policy.max_queue_depth, "QUEUE_THRESHOLD_EXCEEDED"), + ) + return next((reason for failed, reason in thresholds if failed), None) + + def _decision(self, allowed: bool, reason: str) -> CanaryGuardrailDecision: + return CanaryGuardrailDecision( + allowed=allowed, + reason=reason, + blocked_at_ns=self._blocked_at_ns, + hold_until_ns=self._hold_until_ns, + ) diff --git a/qdl/canary/phase92.py b/qdl/canary/phase92.py new file mode 100644 index 0000000..92fc20f --- /dev/null +++ b/qdl/canary/phase92.py @@ -0,0 +1,335 @@ +from __future__ import annotations + +import hashlib +import json +import uuid +from dataclasses import asdict, dataclass +from enum import StrEnum +from typing import Any, Mapping + +from qdl.canary.phase9 import ProductionCanaryAuthorizer +from qdl.certification.prerequisites import CandidateSlice + + +def _canonical_digest(value: Mapping[str, Any]) -> str: + payload = json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return hashlib.sha256(payload).hexdigest() + + +def _valid_digest(value: str) -> bool: + return len(value) == 64 and all(char in "0123456789abcdef" for char in value) + + +class PrimaryAuthorizationMode(StrEnum): + PRODUCTION = "PRODUCTION" + ISOLATED_REHEARSAL = "ISOLATED_REHEARSAL" + + +class HandoffDirection(StrEnum): + PYTHON_TO_RUST = "PYTHON_TO_RUST" + RUST_TO_PYTHON = "RUST_TO_PYTHON" + + +@dataclass(frozen=True, slots=True) +class PrimaryAuthorization: + allowed: bool + production_authorized: bool + mode: PrimaryAuthorizationMode + reason: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str | None + + +@dataclass(frozen=True, slots=True) +class TerminalOwnerCheckpoint: + schema: str + checkpoint_id: str + slice_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + source_session_id: str + connection_generation: int + terminal_watermark: int + terminal_event_id: str + terminal_payload_sha256: str + candidate_digest: str + committed_at_ns: int + + def __post_init__(self) -> None: + try: + uuid.UUID(self.checkpoint_id) + except ValueError as error: + raise ValueError("checkpoint_id must be a UUID") from error + if self.schema != "qdl.terminal-owner-checkpoint.v1": + raise ValueError("terminal checkpoint schema is invalid") + text_fields = ( + self.slice_id, + self.owner_id, + self.source_session_id, + self.terminal_event_id, + ) + if any(not value.strip() for value in text_fields): + raise ValueError("terminal checkpoint identity is incomplete") + positive = ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.connection_generation, + self.committed_at_ns, + ) + if any(value <= 0 for value in positive) or self.terminal_watermark < 0: + raise ValueError("terminal checkpoint epoch/watermark is invalid") + if not _valid_digest(self.terminal_payload_sha256): + raise ValueError("terminal payload digest is invalid") + if not _valid_digest(self.candidate_digest): + raise ValueError("terminal candidate digest is invalid") + + @property + def digest(self) -> str: + return _canonical_digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class AcceptedHandoff: + schema: str + handoff_id: str + direction: HandoffDirection + checkpoint_digest: str + slice_id: str + old_owner_id: str + new_owner_id: str + expected_state: str + new_state: str + expected_authority_revision: int + new_authority_revision: int + expected_lease_epoch: int + new_lease_epoch: int + partition_plan_epoch: int + terminal_watermark: int + first_new_watermark: int + overlap_start_watermark: int + overlap_end_watermark: int + old_event_count: int + new_event_count: int + semantic_mismatches: int + open_gaps: int + candidate_digest: str + prerequisite_bundle_id: str + approved_by: str + approved_at_ns: int + expires_at_ns: int + + def __post_init__(self) -> None: + for value, label in ( + (self.handoff_id, "handoff_id"), + (self.prerequisite_bundle_id, "prerequisite_bundle_id"), + ): + try: + uuid.UUID(value) + except ValueError as error: + raise ValueError(f"{label} must be a UUID") from error + if self.schema != "qdl.accepted-authority-handoff.v1": + raise ValueError("handoff schema is invalid") + if any( + not value.strip() + for value in ( + self.slice_id, + self.old_owner_id, + self.new_owner_id, + self.expected_state, + self.new_state, + self.approved_by, + ) + ): + raise ValueError("handoff identity is incomplete") + expected_states = { + HandoffDirection.PYTHON_TO_RUST: ("RUST_CANARY", "RUST_PRIMARY"), + HandoffDirection.RUST_TO_PYTHON: ( + "ROLLBACK_PENDING", + "PYTHON_PRIMARY", + ), + } + if (self.expected_state, self.new_state) != expected_states[self.direction]: + raise ValueError("handoff direction/state pair is invalid") + if self.old_owner_id == self.new_owner_id: + raise ValueError("handoff must change owner") + if self.new_authority_revision != self.expected_authority_revision + 1: + raise ValueError("handoff authority revision must advance exactly one") + if self.new_lease_epoch <= self.expected_lease_epoch: + raise ValueError("handoff owner requires a newer lease epoch") + if self.partition_plan_epoch <= 0 or self.terminal_watermark < 0: + raise ValueError("handoff plan/watermark is invalid") + if self.first_new_watermark != self.terminal_watermark + 1: + raise ValueError("handoff first watermark must equal terminal + 1") + if ( + self.overlap_start_watermark < 0 + or self.overlap_start_watermark > self.overlap_end_watermark + or self.overlap_end_watermark != self.terminal_watermark + ): + raise ValueError("handoff reconciliation range is invalid") + if ( + self.old_event_count <= 0 + or self.old_event_count != self.new_event_count + or self.semantic_mismatches != 0 + or self.open_gaps != 0 + ): + raise ValueError("handoff reconciliation is not clean") + if not _valid_digest(self.checkpoint_digest) or not _valid_digest( + self.candidate_digest + ): + raise ValueError("handoff digest is invalid") + if self.approved_at_ns <= 0 or self.expires_at_ns <= self.approved_at_ns: + raise ValueError("handoff approval window is invalid") + + @property + def digest(self) -> str: + payload = asdict(self) + payload["direction"] = self.direction.value + return _canonical_digest(payload) + + +class ProductionPrimaryAuthorizer: + """Fail-closed Phase 9.2 gate. It does not mutate authority.""" + + def __init__(self, *, max_decision_age_seconds: int = 900) -> None: + self._canary_authorizer = ProductionCanaryAuthorizer( + max_decision_age_seconds=max_decision_age_seconds + ) + + @staticmethod + def authorize_isolated_rehearsal( + *, candidate: CandidateSlice + ) -> PrimaryAuthorization: + return PrimaryAuthorization( + allowed=True, + production_authorized=False, + mode=PrimaryAuthorizationMode.ISOLATED_REHEARSAL, + reason="ISOLATED_REHEARSAL_ONLY", + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=None, + ) + + def authorize( + self, + *, + candidate: CandidateSlice, + prerequisite_decision: Mapping[str, Any], + canary_evidence: Mapping[str, Any], + approval: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> PrimaryAuthorization: + canary_auth = self._canary_authorizer.authorize( + candidate=candidate, + decision=prerequisite_decision, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + reason = canary_auth.reason + if canary_auth.allowed: + reason = self._validate_canary( + candidate=candidate, + canary=canary_evidence, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + if reason == "AUTHORIZED": + reason = self._validate_approval( + candidate=candidate, + approval=approval, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + return PrimaryAuthorization( + allowed=reason == "AUTHORIZED", + production_authorized=reason == "AUTHORIZED", + mode=PrimaryAuthorizationMode.PRODUCTION, + reason=reason, + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=expected_bundle_id or None, + ) + + @staticmethod + def _validate_canary( + *, + candidate: CandidateSlice, + canary: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> str: + if canary.get("schema") != "qdl.phase91.rust-canary-certification.v1": + return "CANARY_EVIDENCE_SCHEMA_INVALID" + if canary.get("status") != "PRODUCTION_CANARY_HOLD_PASSED": + return "CANARY_HOLD_NOT_PASSED" + if canary.get("production_authorized") is not True: + return "CANARY_NOT_PRODUCTION_AUTHORIZED" + if canary.get("slice_id") != candidate.payload["slice_id"]: + return "CANARY_SLICE_MISMATCH" + if canary.get("candidate_digest") != candidate.digest: + return "CANARY_CANDIDATE_MISMATCH" + if canary.get("prerequisite_bundle_id") != expected_bundle_id: + return "CANARY_BUNDLE_MISMATCH" + if canary.get("python_v1_public_authority_unchanged") is not True: + return "CANARY_V1_PRECONDITION_INVALID" + if canary.get("production_mutations") != 0: + return "CANARY_PRODUCTION_MUTATION_DETECTED" + parity = canary.get("parity") + broker = canary.get("broker") + if not isinstance(parity, Mapping) or parity.get("semantic_mismatches") != 0: + return "CANARY_PARITY_INVALID" + if not isinstance(broker, Mapping): + return "CANARY_BROKER_EVIDENCE_MISSING" + if broker.get("final_authority") != "RUST_CANARY": + return "CANARY_AUTHORITY_NOT_HELD" + if broker.get("public_writes") != 0 or broker.get("legacy_writes") != 0: + return "CANARY_EXTERNAL_WRITE_DETECTED" + completed = canary.get("hold_completed_at_ns") + if ( + not isinstance(completed, int) + or isinstance(completed, bool) + or completed <= 0 + or completed > now_ns + ): + return "CANARY_HOLD_TIME_INVALID" + rollback_digest = canary.get("rollback_manifest_digest") + if not isinstance(rollback_digest, str) or not _valid_digest(rollback_digest): + return "CANARY_ROLLBACK_MANIFEST_INVALID" + return "AUTHORIZED" + + @staticmethod + def _validate_approval( + *, + candidate: CandidateSlice, + approval: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> str: + if approval.get("schema") != "qdl.primary-slice-approval.v1": + return "PRIMARY_APPROVAL_SCHEMA_INVALID" + if approval.get("decision") != "APPROVE": + return "PRIMARY_NOT_APPROVED" + if approval.get("slice_id") != candidate.payload["slice_id"]: + return "PRIMARY_APPROVAL_SLICE_MISMATCH" + if approval.get("candidate_digest") != candidate.digest: + return "PRIMARY_APPROVAL_CANDIDATE_MISMATCH" + if approval.get("prerequisite_bundle_id") != expected_bundle_id: + return "PRIMARY_APPROVAL_BUNDLE_MISMATCH" + if not str(approval.get("operator") or "").strip(): + return "PRIMARY_APPROVER_MISSING" + if not str(approval.get("change_ticket") or "").strip(): + return "PRIMARY_CHANGE_TICKET_MISSING" + if approval.get("max_partitions") != 1: + return "PRIMARY_BLAST_RADIUS_INVALID" + if approval.get("allow_disable_exact_python_slice") is not True: + return "PRIMARY_PYTHON_HANDOFF_NOT_APPROVED" + expires = approval.get("expires_at_ns") + if not isinstance(expires, int) or isinstance(expires, bool) or expires <= now_ns: + return "PRIMARY_APPROVAL_EXPIRED" + return "AUTHORIZED" diff --git a/qdl/canary/phase93.py b/qdl/canary/phase93.py new file mode 100644 index 0000000..6b536f8 --- /dev/null +++ b/qdl/canary/phase93.py @@ -0,0 +1,1186 @@ +from __future__ import annotations + +import hashlib +import json +import uuid +from dataclasses import asdict, dataclass +from enum import StrEnum +from typing import Any, Mapping + +from qdl.canary.phase9 import ProductionCanaryAuthorizer +from qdl.certification.prerequisites import CandidateSlice + + +def _digest(value: Mapping[str, Any]) -> str: + payload = json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return hashlib.sha256(payload).hexdigest() + + +def _valid_digest(value: object) -> bool: + return isinstance(value, str) and len(value) == 64 and all( + char in "0123456789abcdef" for char in value + ) + + +def _valid_uuid(value: object) -> bool: + try: + uuid.UUID(str(value)) + except (ValueError, TypeError, AttributeError): + return False + return True + + +def _positive_int(value: object) -> bool: + return isinstance(value, int) and not isinstance(value, bool) and value > 0 + + +def _non_negative_int(value: object) -> bool: + return isinstance(value, int) and not isinstance(value, bool) and value >= 0 + + +def _non_negative_number(value: object) -> bool: + return ( + isinstance(value, (int, float)) + and not isinstance(value, bool) + and value >= 0 + ) + + +class HoldScope(StrEnum): + TEST_REHEARSAL = "TEST_REHEARSAL" + PRODUCTION = "PRODUCTION" + + +class HoldStatus(StrEnum): + IN_PROGRESS = "IN_PROGRESS" + PASSED = "PASSED" + BLOCKED = "BLOCKED" + + +class ExpansionType(StrEnum): + INSTRUMENT_PARTITION = "INSTRUMENT_PARTITION" + BBO = "BBO" + L2_BOOK = "L2_BOOK" + BAR_LIFECYCLE = "BAR_LIFECYCLE" + VENUE_MARKET = "VENUE_MARKET" + + +_EXPANSION_GATES: dict[ExpansionType, frozenset[str]] = { + ExpansionType.INSTRUMENT_PARTITION: frozenset({ + "authority_handoff", "capacity_headroom", "exact_frame_parity", + "partition_churn", "provider_authentic_source", "rollback", + "source_capacity", + }), + ExpansionType.BBO: frozenset({ + "authority_handoff", "capacity_headroom", "coalescing_policy", + "exact_frame_parity", "freshness", "ordering_reconnect", + "provider_authentic_source", "quote_identity", "rollback", + }), + ExpansionType.L2_BOOK: frozenset({ + "authority_handoff", "capacity_headroom", "checksum", + "exact_frame_parity", "lossless_backpressure", + "provider_authentic_source", "resync", "rollback", + "snapshot_delta_sequence", + }), + ExpansionType.BAR_LIFECYCLE: frozenset({ + "authority_handoff", "capacity_headroom", "close_time_semantics", + "exact_frame_parity", "final_revision_lineage", + "provider_authentic_source", "replay", "rollback", + }), + ExpansionType.VENUE_MARKET: frozenset({ + "adapter_capability", "authority_handoff", "capacity_headroom", + "disaster_recovery", "entitlement", "exact_frame_parity", + "instrument_identity", "provider_authentic_source", + "provider_semantics", "rollback", + }), +} + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldPolicy: + minimum_duration_seconds: int + sample_interval_seconds: int + max_sample_gap_seconds: int + max_lag_ms: int + max_freshness_ms: int + max_queue_depth: int + max_spool_bytes: int + max_cpu_percent: float + max_rss_mb: float + + def __post_init__(self) -> None: + integers = ( + self.minimum_duration_seconds, + self.sample_interval_seconds, + self.max_sample_gap_seconds, + self.max_lag_ms, + self.max_freshness_ms, + self.max_queue_depth, + self.max_spool_bytes, + ) + if any(not _positive_int(value) for value in integers): + raise ValueError("hold policy integer thresholds must be positive") + if self.max_sample_gap_seconds < self.sample_interval_seconds: + raise ValueError("hold maximum sample gap cannot be below sample interval") + if self.minimum_duration_seconds < self.max_sample_gap_seconds: + raise ValueError("hold duration must cover one maximum sample gap") + if self.max_cpu_percent <= 0 or self.max_cpu_percent > 100: + raise ValueError("hold CPU threshold is invalid") + if self.max_rss_mb <= 0: + raise ValueError("hold RSS threshold is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldIdentity: + schema: str + hold_id: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + started_at_ns: int + required_until_ns: int + policy_digest: str + + def __post_init__(self) -> None: + if self.schema != "qdl.primary-hold.v1": + raise ValueError("hold schema is invalid") + if not _valid_uuid(self.hold_id) or not _valid_uuid( + self.prerequisite_bundle_id + ): + raise ValueError("hold UUID identity is invalid") + if not self.slice_id.strip() or not self.owner_id.strip(): + raise ValueError("hold owner/slice identity is required") + if not _valid_digest(self.candidate_digest) or not _valid_digest( + self.policy_digest + ): + raise ValueError("hold digest identity is invalid") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.started_at_ns, + self.required_until_ns, + ) + ) or self.required_until_ns <= self.started_at_ns: + raise ValueError("hold epoch/time identity is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldObservation: + schema: str + observation_id: str + hold_id: str + slice_id: str + candidate_digest: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + sequence: int + observed_at_ns: int + last_watermark: int + semantic_mismatches: int = 0 + open_gaps: int = 0 + duplicate_external_writes: int = 0 + accepted_stale_writer_writes: int = 0 + authority_ambiguities: int = 0 + durable_ack_failures: int = 0 + projection_mismatches: int = 0 + consumer_checkpoint_regressions: int = 0 + unexplained_quality_failures: int = 0 + lag_ms: int = 0 + freshness_ms: int = 0 + queue_depth: int = 0 + spool_bytes: int = 0 + cpu_percent: float = 0.0 + rss_mb: float = 0.0 + registered_consumers: int = 0 + healthy_consumers: int = 0 + checkpoint_watermark: int = 0 + + def __post_init__(self) -> None: + if self.schema != "qdl.primary-hold-observation.v1": + raise ValueError("hold observation schema is invalid") + if not _valid_uuid(self.observation_id) or not _valid_uuid(self.hold_id): + raise ValueError("hold observation UUID is invalid") + if ( + not self.slice_id.strip() + or not self.owner_id.strip() + or not _valid_digest(self.candidate_digest) + ): + raise ValueError("hold observation identity is invalid") + positive = ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.sequence, + self.observed_at_ns, + ) + if any(not _positive_int(value) for value in positive): + raise ValueError("hold observation epoch/sequence/time is invalid") + integer_values = ( + self.last_watermark, + self.semantic_mismatches, + self.open_gaps, + self.duplicate_external_writes, + self.accepted_stale_writer_writes, + self.authority_ambiguities, + self.durable_ack_failures, + self.projection_mismatches, + self.consumer_checkpoint_regressions, + self.unexplained_quality_failures, + self.lag_ms, + self.freshness_ms, + self.queue_depth, + self.spool_bytes, + self.registered_consumers, + self.healthy_consumers, + self.checkpoint_watermark, + ) + if any(not _non_negative_int(value) for value in integer_values) or any( + not _non_negative_number(value) + for value in (self.cpu_percent, self.rss_mb) + ): + raise ValueError("hold observation values must be non-negative") + if self.healthy_consumers > self.registered_consumers: + raise ValueError("healthy consumer count exceeds registry count") + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldDecision: + schema: str + decision_id: str + hold_id: str + status: HoldStatus + reason: str + scope: HoldScope + production_authorized: bool + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + policy_digest: str + first_observed_at_ns: int | None + last_observed_at_ns: int | None + observation_count: int + terminal_watermark: int | None + decided_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.primary-hold-decision.v1": + raise ValueError("hold decision schema is invalid") + if not isinstance(self.status, HoldStatus) or not isinstance( + self.scope, HoldScope + ): + raise ValueError("hold decision status/scope is invalid") + if not _valid_uuid(self.decision_id) or not _valid_uuid(self.hold_id): + raise ValueError("hold decision UUID is invalid") + if ( + not self.reason.strip() + or not self.slice_id.strip() + or not self.owner_id.strip() + ): + raise ValueError("hold decision identity/reason is required") + if not _valid_uuid(self.prerequisite_bundle_id): + raise ValueError("hold decision prerequisite bundle is invalid") + if not _valid_digest(self.candidate_digest) or not _valid_digest( + self.policy_digest + ): + raise ValueError("hold decision digest is invalid") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.decided_at_ns, + ) + ) or not _non_negative_int(self.observation_count): + raise ValueError("hold decision epoch/count/time is invalid") + if self.production_authorized != ( + self.scope == HoldScope.PRODUCTION and self.status == HoldStatus.PASSED + ): + raise ValueError("hold decision production authorization is inconsistent") + summary = ( + self.first_observed_at_ns, + self.last_observed_at_ns, + self.terminal_watermark, + ) + if self.observation_count == 0 and any(value is not None for value in summary): + raise ValueError("empty hold decision cannot expose observation state") + if self.observation_count > 0 and ( + self.first_observed_at_ns is None + or self.last_observed_at_ns is None + or self.terminal_watermark is None + or self.first_observed_at_ns <= 0 + or self.last_observed_at_ns < self.first_observed_at_ns + or self.terminal_watermark < 0 + ): + raise ValueError("hold decision observation summary is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +class PrimaryHoldEvaluator: + _ZERO_TOLERANCE = ( + ("semantic_mismatches", "SEMANTIC_MISMATCH"), + ("open_gaps", "OPEN_GAP"), + ("duplicate_external_writes", "DUPLICATE_EXTERNAL_WRITE"), + ("accepted_stale_writer_writes", "ACCEPTED_STALE_WRITER_WRITE"), + ("authority_ambiguities", "AUTHORITY_AMBIGUITY"), + ("durable_ack_failures", "DURABLE_ACK_FAILURE"), + ("projection_mismatches", "PROJECTION_MISMATCH"), + ("consumer_checkpoint_regressions", "CONSUMER_CHECKPOINT_REGRESSION"), + ("unexplained_quality_failures", "UNEXPLAINED_QUALITY_FAILURE"), + ) + + def __init__( + self, + *, + identity: PrimaryHoldIdentity, + policy: PrimaryHoldPolicy, + scope: HoldScope, + ) -> None: + if identity.policy_digest != policy.digest: + raise ValueError("hold identity does not bind the policy") + duration = identity.required_until_ns - identity.started_at_ns + if duration < policy.minimum_duration_seconds * 1_000_000_000: + raise ValueError("hold identity duration is below policy") + self.identity = identity + self.policy = policy + self.scope = scope + self._observations: list[PrimaryHoldObservation] = [] + self._blocked_reason: str | None = None + + def observe(self, item: PrimaryHoldObservation) -> str: + if self._blocked_reason is not None: + return "HOLD_ALREADY_BLOCKED" + reason = self._validate_observation(item) + self._observations.append(item) + if reason != "PASS": + self._blocked_reason = reason + return reason + + def _validate_observation(self, item: PrimaryHoldObservation) -> str: + expected = ( + self.identity.hold_id, + self.identity.slice_id, + self.identity.candidate_digest, + self.identity.owner_id, + self.identity.authority_revision, + self.identity.lease_epoch, + self.identity.partition_plan_epoch, + ) + actual = ( + item.hold_id, + item.slice_id, + item.candidate_digest, + item.owner_id, + item.authority_revision, + item.lease_epoch, + item.partition_plan_epoch, + ) + if actual != expected: + return "HOLD_AUTHORITY_IDENTITY_CHANGED" + if item.sequence != len(self._observations) + 1: + return "HOLD_SEQUENCE_NOT_CONTIGUOUS" + previous_time = ( + self._observations[-1].observed_at_ns + if self._observations + else self.identity.started_at_ns + ) + if item.observed_at_ns <= previous_time: + return "HOLD_OBSERVATION_TIME_NOT_MONOTONIC" + if ( + item.observed_at_ns - previous_time + > self.policy.max_sample_gap_seconds * 1_000_000_000 + ): + return "HOLD_OBSERVATION_GAP_EXCEEDED" + if ( + self._observations + and item.last_watermark < self._observations[-1].last_watermark + ): + return "HOLD_WATERMARK_REGRESSED" + for field, reason in self._ZERO_TOLERANCE: + if getattr(item, field) != 0: + return reason + thresholds = ( + (item.lag_ms > self.policy.max_lag_ms, "LAG_THRESHOLD_EXCEEDED"), + ( + item.freshness_ms > self.policy.max_freshness_ms, + "FRESHNESS_THRESHOLD_EXCEEDED", + ), + ( + item.queue_depth > self.policy.max_queue_depth, + "QUEUE_THRESHOLD_EXCEEDED", + ), + ( + item.spool_bytes > self.policy.max_spool_bytes, + "SPOOL_THRESHOLD_EXCEEDED", + ), + ( + item.cpu_percent > self.policy.max_cpu_percent, + "CPU_THRESHOLD_EXCEEDED", + ), + (item.rss_mb > self.policy.max_rss_mb, "RSS_THRESHOLD_EXCEEDED"), + ) + threshold_reason = next((reason for failed, reason in thresholds if failed), None) + if threshold_reason is not None: + return threshold_reason + if item.registered_consumers <= 0: + return "CONSUMER_REGISTRY_EMPTY" + if item.healthy_consumers != item.registered_consumers: + return "CONSUMER_NOT_HEALTHY" + if item.checkpoint_watermark < item.last_watermark: + return "CONSUMER_CHECKPOINT_BEHIND" + return "PASS" + + def decision(self, *, decision_id: str, now_ns: int) -> PrimaryHoldDecision: + if not _positive_int(now_ns): + raise ValueError("hold decision clock is invalid") + if self._observations and now_ns < self._observations[-1].observed_at_ns: + raise ValueError("hold decision precedes the latest observation") + status = HoldStatus.IN_PROGRESS + reason = "HOLD_WINDOW_INCOMPLETE" + if self._blocked_reason is not None: + status = HoldStatus.BLOCKED + reason = self._blocked_reason + elif not self._observations: + reason = "HOLD_OBSERVATION_MISSING" + elif now_ns < self.identity.required_until_ns: + reason = "HOLD_WINDOW_INCOMPLETE" + elif self._observations[-1].observed_at_ns < self.identity.required_until_ns: + reason = "HOLD_TERMINAL_OBSERVATION_MISSING" + else: + status = HoldStatus.PASSED + reason = "PASS" + first = self._observations[0] if self._observations else None + last = self._observations[-1] if self._observations else None + return PrimaryHoldDecision( + schema="qdl.primary-hold-decision.v1", + decision_id=decision_id, + hold_id=self.identity.hold_id, + status=status, + reason=reason, + scope=self.scope, + production_authorized=( + status == HoldStatus.PASSED and self.scope == HoldScope.PRODUCTION + ), + slice_id=self.identity.slice_id, + candidate_digest=self.identity.candidate_digest, + prerequisite_bundle_id=self.identity.prerequisite_bundle_id, + owner_id=self.identity.owner_id, + authority_revision=self.identity.authority_revision, + lease_epoch=self.identity.lease_epoch, + partition_plan_epoch=self.identity.partition_plan_epoch, + policy_digest=self.identity.policy_digest, + first_observed_at_ns=first.observed_at_ns if first else None, + last_observed_at_ns=last.observed_at_ns if last else None, + observation_count=len(self._observations), + terminal_watermark=last.last_watermark if last else None, + decided_at_ns=now_ns, + ) + + +@dataclass(frozen=True, slots=True) +class ConsumerCheckpoint: + consumer_id: str + requirement_digest: str + contract_major: int + applied_watermark: int + checkpointed_watermark: int + status: str + migration_status: str + rollback_ready: bool + + def __post_init__(self) -> None: + if not self.consumer_id.strip() or not _valid_digest(self.requirement_digest): + raise ValueError("consumer checkpoint identity is invalid") + if not _positive_int(self.contract_major): + raise ValueError("consumer contract major is invalid") + if ( + not _non_negative_int(self.applied_watermark) + or not _non_negative_int(self.checkpointed_watermark) + or self.checkpointed_watermark > self.applied_watermark + ): + raise ValueError("consumer checkpoint watermark is invalid") + if self.status != "READY" or self.migration_status != "COMPLETE": + raise ValueError("consumer is not closure-ready") + if self.rollback_ready is not True: + raise ValueError("consumer rollback posture is not ready") + + +@dataclass(frozen=True, slots=True) +class ConsumerRegistrySnapshot: + schema: str + snapshot_id: str + slice_id: str + authority_revision: int + checkpoints: tuple[ConsumerCheckpoint, ...] + observed_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.consumer-registry-snapshot.v1": + raise ValueError("consumer registry schema is invalid") + if not _valid_uuid(self.snapshot_id) or not self.slice_id.strip(): + raise ValueError("consumer registry identity is invalid") + if not _positive_int(self.authority_revision) or not _positive_int( + self.observed_at_ns + ): + raise ValueError("consumer registry revision/time is invalid") + if not isinstance(self.checkpoints, tuple): + raise ValueError("consumer registry checkpoints must be immutable") + ids = [item.consumer_id for item in self.checkpoints] + if not ids or len(ids) != len(set(ids)): + raise ValueError("consumer registry must be non-empty and unique") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class AuthorityRegistrySnapshot: + schema: str + snapshot_id: str + slice_id: str + state: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + candidate_digest: str + prerequisite_bundle_id: str + current_watermark: int + public_write_allowed: bool + legacy_write_allowed: bool + observed_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.authority-registry-snapshot.v1": + raise ValueError("authority registry schema is invalid") + if ( + not _valid_uuid(self.snapshot_id) + or not _valid_uuid(self.prerequisite_bundle_id) + or not self.slice_id.strip() + or not self.owner_id.strip() + or not _valid_digest(self.candidate_digest) + ): + raise ValueError("authority registry identity is invalid") + if self.state != "RUST_PRIMARY": + raise ValueError("authority registry is not Rust primary") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.observed_at_ns, + ) + ) or not _non_negative_int(self.current_watermark): + raise ValueError("authority registry epoch/watermark/time is invalid") + if not self.public_write_allowed or not self.legacy_write_allowed: + raise ValueError("primary authority write flags are invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class RollbackRehearsalEvidence: + schema: str + rehearsal_id: str + slice_id: str + candidate_digest: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + rollback_manifest_digest: str + reconciled_through_watermark: int + rto_ms: float + status: str + production_scope: bool + observed_at_ns: int + expires_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.rollback-rehearsal.v1": + raise ValueError("rollback rehearsal schema is invalid") + if ( + not _valid_uuid(self.rehearsal_id) + or not self.slice_id.strip() + or not self.owner_id.strip() + or not _valid_digest(self.candidate_digest) + or not _valid_digest(self.rollback_manifest_digest) + ): + raise ValueError("rollback rehearsal identity is invalid") + if self.status != "PASS": + raise ValueError("rollback rehearsal did not pass") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.observed_at_ns, + self.expires_at_ns, + ) + ) or self.expires_at_ns <= self.observed_at_ns: + raise ValueError("rollback rehearsal epoch/time is invalid") + if not _non_negative_int(self.reconciled_through_watermark) or self.rto_ms <= 0: + raise ValueError("rollback rehearsal watermark/RTO is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class ClosureApproval: + schema: str + approval_id: str + closure_id: str + decision: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + hold_id: str + hold_policy_digest: str + operator: str + change_ticket: str + allow_close_rollback_window: bool + repository_cleanup_approved: bool + approved_at_ns: int + expires_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.rollback-window-closure-approval.v1": + raise ValueError("closure approval schema is invalid") + if any( + not _valid_uuid(value) + for value in ( + self.approval_id, + self.closure_id, + self.prerequisite_bundle_id, + self.hold_id, + ) + ): + raise ValueError("closure approval UUID is invalid") + if ( + self.decision != "APPROVE" + or not self.slice_id.strip() + or not self.operator.strip() + or not self.change_ticket.strip() + or not _valid_digest(self.candidate_digest) + or not _valid_digest(self.hold_policy_digest) + ): + raise ValueError("closure approval identity/decision is invalid") + if self.allow_close_rollback_window is not True: + raise ValueError("closure approval does not allow window close") + if self.repository_cleanup_approved: + raise ValueError("window closure cannot approve repository cleanup") + if ( + not _positive_int(self.approved_at_ns) + or not _positive_int(self.expires_at_ns) + or self.expires_at_ns <= self.approved_at_ns + ): + raise ValueError("closure approval time window is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class RollbackWindowClosure: + schema: str + closure_id: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + hold_decision_id: str + hold_decision_digest: str + consumer_registry_snapshot_id: str + consumer_registry_digest: str + authority_registry_snapshot_id: str + authority_registry_digest: str + rollback_rehearsal_id: str + rollback_rehearsal_digest: str + approval_id: str + approval_digest: str + operator: str + change_ticket: str + closed_at_ns: int + production_authorized: bool + + def __post_init__(self) -> None: + if self.schema != "qdl.rollback-window-closure.v1": + raise ValueError("rollback closure schema is invalid") + if any( + not _valid_uuid(value) + for value in ( + self.closure_id, + self.prerequisite_bundle_id, + self.hold_decision_id, + self.consumer_registry_snapshot_id, + self.authority_registry_snapshot_id, + self.rollback_rehearsal_id, + self.approval_id, + ) + ): + raise ValueError("rollback closure UUID is invalid") + if ( + not self.slice_id.strip() + or not self.owner_id.strip() + or not self.operator.strip() + or not self.change_ticket.strip() + ): + raise ValueError("rollback closure identity is incomplete") + if any( + not _valid_digest(value) + for value in ( + self.candidate_digest, + self.hold_decision_digest, + self.consumer_registry_digest, + self.authority_registry_digest, + self.rollback_rehearsal_digest, + self.approval_digest, + ) + ): + raise ValueError("rollback closure digest is invalid") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.closed_at_ns, + ) + ): + raise ValueError("rollback closure epoch/time is invalid") + if self.production_authorized is not True: + raise ValueError("rollback closure must be production-authorized") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class ClosureAuthorization: + allowed: bool + reason: str + production_authorized: bool + closure: RollbackWindowClosure | None + + +class ProductionClosureAuthorizer: + def __init__(self, *, max_decision_age_seconds: int = 900) -> None: + self._prerequisite_authorizer = ProductionCanaryAuthorizer( + max_decision_age_seconds=max_decision_age_seconds + ) + + def authorize( + self, + *, + candidate: CandidateSlice, + prerequisite_decision: Mapping[str, Any], + expected_bundle_id: str, + primary_evidence: Mapping[str, Any] | None, + hold_decision: PrimaryHoldDecision | None, + consumer_registry: ConsumerRegistrySnapshot | None, + authority_registry: AuthorityRegistrySnapshot | None, + rollback_evidence: RollbackRehearsalEvidence | None, + approval: ClosureApproval | None, + now_ns: int, + ) -> ClosureAuthorization: + prerequisite = self._prerequisite_authorizer.authorize( + candidate=candidate, + decision=prerequisite_decision, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + if not prerequisite.allowed: + return ClosureAuthorization(False, prerequisite.reason, False, None) + if primary_evidence is None: + return ClosureAuthorization(False, "PRIMARY_EVIDENCE_MISSING", False, None) + if hold_decision is None: + return ClosureAuthorization(False, "PRIMARY_HOLD_MISSING", False, None) + if consumer_registry is None or authority_registry is None: + return ClosureAuthorization(False, "REGISTRY_SNAPSHOT_MISSING", False, None) + if rollback_evidence is None: + return ClosureAuthorization(False, "ROLLBACK_REHEARSAL_MISSING", False, None) + if approval is None: + return ClosureAuthorization(False, "CLOSURE_APPROVAL_MISSING", False, None) + reason = self._validate_primary(candidate, expected_bundle_id, primary_evidence) + if reason == "AUTHORIZED": + reason = self._validate_hold(candidate, expected_bundle_id, hold_decision) + if reason == "AUTHORIZED": + reason = self._validate_registries( + hold_decision, + primary_evidence, + consumer_registry, + authority_registry, + now_ns, + ) + if reason == "AUTHORIZED": + reason = self._validate_rollback( + candidate, + hold_decision, + authority_registry, + rollback_evidence, + now_ns, + ) + if reason == "AUTHORIZED": + reason = self._validate_approval( + candidate, expected_bundle_id, hold_decision, approval, now_ns + ) + if reason != "AUTHORIZED": + return ClosureAuthorization(False, reason, False, None) + closure = RollbackWindowClosure( + schema="qdl.rollback-window-closure.v1", + closure_id=approval.closure_id, + slice_id=hold_decision.slice_id, + candidate_digest=hold_decision.candidate_digest, + prerequisite_bundle_id=hold_decision.prerequisite_bundle_id, + owner_id=hold_decision.owner_id, + authority_revision=hold_decision.authority_revision, + lease_epoch=hold_decision.lease_epoch, + partition_plan_epoch=hold_decision.partition_plan_epoch, + hold_decision_id=hold_decision.decision_id, + hold_decision_digest=hold_decision.digest, + consumer_registry_snapshot_id=consumer_registry.snapshot_id, + consumer_registry_digest=consumer_registry.digest, + authority_registry_snapshot_id=authority_registry.snapshot_id, + authority_registry_digest=authority_registry.digest, + rollback_rehearsal_id=rollback_evidence.rehearsal_id, + rollback_rehearsal_digest=rollback_evidence.digest, + approval_id=approval.approval_id, + approval_digest=approval.digest, + operator=approval.operator, + change_ticket=approval.change_ticket, + closed_at_ns=now_ns, + production_authorized=True, + ) + return ClosureAuthorization(True, "AUTHORIZED", True, closure) + + @staticmethod + def _validate_primary( + candidate: CandidateSlice, bundle_id: str, primary: Mapping[str, Any] + ) -> str: + if primary.get("schema") != "qdl.phase92.production-primary.v1": + return "PRIMARY_EVIDENCE_SCHEMA_INVALID" + if primary.get("status") != "PRODUCTION_PRIMARY_ACTIVE": + return "PRIMARY_NOT_ACTIVE" + if primary.get("production_authorized") is not True: + return "PRIMARY_NOT_PRODUCTION_AUTHORIZED" + if primary.get("slice_id") != candidate.payload["slice_id"]: + return "PRIMARY_SLICE_MISMATCH" + if primary.get("candidate_digest") != candidate.digest: + return "PRIMARY_CANDIDATE_MISMATCH" + if primary.get("prerequisite_bundle_id") != bundle_id: + return "PRIMARY_BUNDLE_MISMATCH" + authority = primary.get("authority") + if not isinstance(authority, Mapping) or authority.get("state") != "RUST_PRIMARY": + return "PRIMARY_AUTHORITY_INVALID" + required = ( + "owner_id", "authority_revision", "lease_epoch", + "partition_plan_epoch", "current_watermark", + ) + if any(key not in authority for key in required): + return "PRIMARY_AUTHORITY_INCOMPLETE" + return "AUTHORIZED" + + @staticmethod + def _validate_hold( + candidate: CandidateSlice, bundle_id: str, hold: PrimaryHoldDecision + ) -> str: + if hold.status != HoldStatus.PASSED or hold.reason != "PASS": + return "PRIMARY_HOLD_NOT_PASSED" + if hold.scope != HoldScope.PRODUCTION or not hold.production_authorized: + return "PRIMARY_HOLD_NOT_PRODUCTION" + if hold.slice_id != candidate.payload["slice_id"]: + return "PRIMARY_HOLD_SLICE_MISMATCH" + if hold.candidate_digest != candidate.digest: + return "PRIMARY_HOLD_CANDIDATE_MISMATCH" + if hold.prerequisite_bundle_id != bundle_id: + return "PRIMARY_HOLD_BUNDLE_MISMATCH" + return "AUTHORIZED" + + @staticmethod + def _validate_registries( + hold: PrimaryHoldDecision, + primary: Mapping[str, Any], + consumer: ConsumerRegistrySnapshot, + authority: AuthorityRegistrySnapshot, + now_ns: int, + ) -> str: + if (consumer.slice_id, consumer.authority_revision) != ( + hold.slice_id, + hold.authority_revision, + ): + return "CONSUMER_REGISTRY_IDENTITY_MISMATCH" + if ( + authority.slice_id != hold.slice_id + or authority.owner_id != hold.owner_id + or authority.authority_revision != hold.authority_revision + or authority.lease_epoch != hold.lease_epoch + or authority.partition_plan_epoch != hold.partition_plan_epoch + or authority.candidate_digest != hold.candidate_digest + or authority.prerequisite_bundle_id != hold.prerequisite_bundle_id + ): + return "AUTHORITY_REGISTRY_IDENTITY_MISMATCH" + max_age_ns = 300 * 1_000_000_000 + if any( + observed > now_ns or now_ns - observed > max_age_ns + for observed in (consumer.observed_at_ns, authority.observed_at_ns) + ): + return "REGISTRY_SNAPSHOT_STALE" + primary_authority = primary.get("authority") + if not isinstance(primary_authority, Mapping) or any( + primary_authority.get(field) != expected + for field, expected in ( + ("owner_id", authority.owner_id), + ("authority_revision", authority.authority_revision), + ("lease_epoch", authority.lease_epoch), + ("partition_plan_epoch", authority.partition_plan_epoch), + ("current_watermark", authority.current_watermark), + ) + ): + return "PRIMARY_AUTHORITY_REGISTRY_MISMATCH" + terminal = hold.terminal_watermark + if terminal is None or authority.current_watermark < terminal: + return "AUTHORITY_REGISTRY_WATERMARK_BEHIND" + if any( + item.checkpointed_watermark < authority.current_watermark + for item in consumer.checkpoints + ): + return "CONSUMER_CHECKPOINT_BEHIND" + return "AUTHORIZED" + + @staticmethod + def _validate_rollback( + candidate: CandidateSlice, + hold: PrimaryHoldDecision, + authority: AuthorityRegistrySnapshot, + rollback: RollbackRehearsalEvidence, + now_ns: int, + ) -> str: + if not rollback.production_scope: + return "ROLLBACK_REHEARSAL_NOT_PRODUCTION" + if rollback.expires_at_ns <= now_ns: + return "ROLLBACK_REHEARSAL_EXPIRED" + if ( + rollback.slice_id != hold.slice_id + or rollback.candidate_digest != hold.candidate_digest + or rollback.owner_id != hold.owner_id + or rollback.authority_revision != hold.authority_revision + or rollback.lease_epoch != hold.lease_epoch + or rollback.partition_plan_epoch != hold.partition_plan_epoch + ): + return "ROLLBACK_REHEARSAL_IDENTITY_MISMATCH" + if ( + rollback.rollback_manifest_digest + != candidate.payload["rollback_manifest_digest"] + ): + return "ROLLBACK_MANIFEST_MISMATCH" + if rollback.reconciled_through_watermark < authority.current_watermark: + return "ROLLBACK_REHEARSAL_WATERMARK_BEHIND" + return "AUTHORIZED" + + @staticmethod + def _validate_approval( + candidate: CandidateSlice, + bundle_id: str, + hold: PrimaryHoldDecision, + approval: ClosureApproval, + now_ns: int, + ) -> str: + if approval.expires_at_ns <= now_ns or approval.approved_at_ns > now_ns: + return "CLOSURE_APPROVAL_TIME_INVALID" + if ( + approval.slice_id != candidate.payload["slice_id"] + or approval.candidate_digest != candidate.digest + or approval.prerequisite_bundle_id != bundle_id + or approval.hold_id != hold.hold_id + or approval.hold_policy_digest != hold.policy_digest + ): + return "CLOSURE_APPROVAL_IDENTITY_MISMATCH" + return "AUTHORIZED" + + +@dataclass(frozen=True, slots=True) +class ExpansionManifest: + schema: str + expansion_id: str + parent_slice_id: str + parent_candidate_digest: str + parent_closure_id: str + parent_closure_digest: str + expansion_type: ExpansionType + candidate_digest: str + scope_digest: str + partition_plan_epoch: int + required_gates: tuple[str, ...] + status: str + transitive_evidence_allowed: bool + public_write_allowed: bool + legacy_write_allowed: bool + created_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.independent-expansion.v1": + raise ValueError("expansion schema is invalid") + if not isinstance(self.expansion_type, ExpansionType): + raise ValueError("expansion type is invalid") + if not _valid_uuid(self.expansion_id) or not _valid_uuid( + self.parent_closure_id + ): + raise ValueError("expansion UUID identity is invalid") + if not self.parent_slice_id.strip(): + raise ValueError("expansion parent slice is required") + if any( + not _valid_digest(value) + for value in ( + self.parent_candidate_digest, + self.parent_closure_digest, + self.candidate_digest, + self.scope_digest, + ) + ): + raise ValueError("expansion digest identity is invalid") + if self.candidate_digest == self.parent_candidate_digest: + raise ValueError("expansion requires a new candidate digest") + if not _positive_int(self.partition_plan_epoch): + raise ValueError("expansion partition plan epoch is invalid") + if self.required_gates != tuple(sorted(_EXPANSION_GATES[self.expansion_type])): + raise ValueError("expansion required gates are incomplete or transitive") + if self.status != "INDEPENDENT_CERTIFICATION_REQUIRED": + raise ValueError("expansion status cannot grant authority") + if ( + self.transitive_evidence_allowed + or self.public_write_allowed + or self.legacy_write_allowed + ): + raise ValueError("expansion cannot inherit evidence or write authority") + if not _positive_int(self.created_at_ns): + raise ValueError("expansion creation time is invalid") + + @classmethod + def plan( + cls, + *, + expansion_id: str, + parent: RollbackWindowClosure, + expansion_type: ExpansionType, + candidate_digest: str, + scope_digest: str, + partition_plan_epoch: int, + created_at_ns: int, + ) -> "ExpansionManifest": + if ( + expansion_type == ExpansionType.INSTRUMENT_PARTITION + and partition_plan_epoch <= parent.partition_plan_epoch + ): + raise ValueError("instrument expansion requires a newer partition epoch") + return cls( + schema="qdl.independent-expansion.v1", + expansion_id=expansion_id, + parent_slice_id=parent.slice_id, + parent_candidate_digest=parent.candidate_digest, + parent_closure_id=parent.closure_id, + parent_closure_digest=parent.digest, + expansion_type=expansion_type, + candidate_digest=candidate_digest, + scope_digest=scope_digest, + partition_plan_epoch=partition_plan_epoch, + required_gates=tuple(sorted(_EXPANSION_GATES[expansion_type])), + status="INDEPENDENT_CERTIFICATION_REQUIRED", + transitive_evidence_allowed=False, + public_write_allowed=False, + legacy_write_allowed=False, + created_at_ns=created_at_ns, + ) + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class DecommissionRequest: + schema: str + request_id: str + runtime_id: str + owned_slice_ids: tuple[str, ...] + rollback_reference_ids: tuple[str, ...] + consumer_dependency_ids: tuple[str, ...] + all_replacement_windows_closed: bool + repository_cleanup_approved: bool + shared_knowledge_retained: bool + + def __post_init__(self) -> None: + if self.schema != "qdl.runtime-decommission-request.v1": + raise ValueError("decommission request schema is invalid") + if not _valid_uuid(self.request_id) or not self.runtime_id.strip(): + raise ValueError("decommission request identity is invalid") + for values in ( + self.owned_slice_ids, + self.rollback_reference_ids, + self.consumer_dependency_ids, + ): + if not isinstance(values, tuple): + raise ValueError("decommission dependencies must be immutable") + if any(not value.strip() for value in values) or len(values) != len( + set(values) + ): + raise ValueError("decommission dependencies must be unique") + + +@dataclass(frozen=True, slots=True) +class DecommissionDecision: + allowed: bool + reason: str + + +def assess_decommission(request: DecommissionRequest) -> DecommissionDecision: + if request.owned_slice_ids: + return DecommissionDecision(False, "RUNTIME_STILL_OWNS_SLICES") + if request.rollback_reference_ids: + return DecommissionDecision(False, "RUNTIME_STILL_REQUIRED_FOR_ROLLBACK") + if request.consumer_dependency_ids: + return DecommissionDecision(False, "RUNTIME_HAS_CONSUMER_DEPENDENCIES") + if not request.all_replacement_windows_closed: + return DecommissionDecision(False, "REPLACEMENT_WINDOWS_NOT_CLOSED") + if not request.repository_cleanup_approved: + return DecommissionDecision(False, "REPOSITORY_CLEANUP_NOT_APPROVED") + if not request.shared_knowledge_retained: + return DecommissionDecision(False, "SHARED_KNOWLEDGE_REMOVAL_FORBIDDEN") + return DecommissionDecision(True, "AUTHORIZED") diff --git a/qdl/certification/prerequisites.py b/qdl/certification/prerequisites.py new file mode 100644 index 0000000..55c6640 --- /dev/null +++ b/qdl/certification/prerequisites.py @@ -0,0 +1,370 @@ +from __future__ import annotations + +import hashlib +import json +import re +import uuid +from dataclasses import dataclass +from enum import StrEnum +from pathlib import Path +from typing import Any + +import yaml + + +_SHA256 = re.compile(r"^[0-9a-f]{64}$") +_IMAGE_DIGEST = re.compile(r"^sha256:[0-9a-f]{64}$") +_SENSITIVE_DETAIL_NAMES = frozenset({ + "secret", "token", "password", "private_key", "private-key", "key_material", +}) + + +def _sensitive_detail_key(value: object) -> bool: + name = str(value).lower() + return name in _SENSITIVE_DETAIL_NAMES or any( + name.endswith(suffix) + for suffix in ("_secret", "_token", "_password", "_private_key", "_key_material") + ) + + +def _contains_sensitive_detail(value: object) -> bool: + if isinstance(value, dict): + return any( + _sensitive_detail_key(key) or _contains_sensitive_detail(item) + for key, item in value.items() + ) + if isinstance(value, (list, tuple)): + return any(_contains_sensitive_detail(item) for item in value) + return False + + +def _is_number(value: object) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) + + +class EvidenceScope(StrEnum): + TEST = "TEST" + LOCAL_REHEARSAL = "LOCAL_REHEARSAL" + PRODUCTION = "PRODUCTION" + INDEPENDENT_FAILURE_DOMAIN = "INDEPENDENT_FAILURE_DOMAIN" + + +_SCOPE_RANK = {scope: rank for rank, scope in enumerate(EvidenceScope)} + + +@dataclass(frozen=True) +class GatePolicy: + gate_id: str + minimum_scope: EvidenceScope + max_age_seconds: int + required_details: frozenset[str] + candidate_bound: bool + assertions: dict[str, Any] + minimums: dict[str, float] + maximums: dict[str, float] + candidate_field_matches: dict[str, str] + + +@dataclass(frozen=True) +class PrerequisitePolicy: + revision: int + environment: str + gates: tuple[GatePolicy, ...] + + @classmethod + def load(cls, path: str | Path) -> "PrerequisitePolicy": + payload = yaml.safe_load(Path(path).read_text()) + if not isinstance(payload, dict) or set(payload) != { + "schema", "revision", "environment", "gates" + }: + raise ValueError("prerequisite policy fields are incomplete or unknown") + if payload["schema"] != "qdl.production-prerequisite-policy.v1": + raise ValueError("unsupported prerequisite policy schema") + gates_raw = payload["gates"] + if not isinstance(gates_raw, list) or not gates_raw: + raise ValueError("prerequisite policy requires gates") + gates: list[GatePolicy] = [] + seen: set[str] = set() + for item in gates_raw: + if not isinstance(item, dict) or set(item) != { + "id", "minimum_scope", "max_age_seconds", + "required_details", "candidate_bound", "assertions", + "minimums", "maximums", "candidate_field_matches", + }: + raise ValueError("prerequisite gate fields are incomplete or unknown") + gate_id = str(item["id"]).strip() + if not gate_id or gate_id in seen: + raise ValueError("prerequisite gate IDs must be non-empty and unique") + seen.add(gate_id) + max_age = int(item["max_age_seconds"]) + details = item["required_details"] + if max_age <= 0 or not isinstance(details, list): + raise ValueError("prerequisite gate age/details are invalid") + mappings = tuple(item[key] for key in ( + "assertions", "minimums", "maximums", "candidate_field_matches" + )) + if not all(isinstance(value, dict) for value in mappings): + raise ValueError("prerequisite gate constraints must be mappings") + constrained = set().union(*(set(value) for value in mappings)) + if constrained - set(details): + raise ValueError("prerequisite constraints must reference required details") + gates.append(GatePolicy( + gate_id=gate_id, + minimum_scope=EvidenceScope(str(item["minimum_scope"]).upper()), + max_age_seconds=max_age, + required_details=frozenset(str(value) for value in details), + candidate_bound=bool(item["candidate_bound"]), + assertions=dict(item["assertions"]), + minimums={key: float(value) for key, value in item["minimums"].items()}, + maximums={key: float(value) for key, value in item["maximums"].items()}, + candidate_field_matches={ + str(key): str(value) + for key, value in item["candidate_field_matches"].items() + }, + )) + revision = int(payload["revision"]) + environment = str(payload["environment"]).strip().lower() + if revision <= 0 or not environment: + raise ValueError("prerequisite policy revision/environment are invalid") + return cls(revision=revision, environment=environment, gates=tuple(gates)) + + +@dataclass(frozen=True) +class CandidateSlice: + payload: dict[str, Any] + + @classmethod + def load(cls, path: str | Path) -> "CandidateSlice": + payload = yaml.safe_load(Path(path).read_text()) + required = { + "schema", "slice_id", "environment", "venue", "market", + "product_type", "feed", "instrument_uids", "partition_plan_epoch", + "partition_id", "schema_major", "authority_state", "owner_id", + "lease_epoch", "artifact_image_digest", "sbom_digest", + "contract_digest", "partition_plan_digest", "rollback_manifest_digest", + "signature_identity", "normalizer_version", "adapter_version", + "config_revision", "instrument_catalog_revision", "source_policy_revision", + "public_write_allowed", "legacy_write_allowed", + } + if not isinstance(payload, dict) or set(payload) != required: + raise ValueError("candidate slice fields are incomplete or unknown") + if payload["schema"] != "qdl.authority-candidate.v1": + raise ValueError("unsupported candidate slice schema") + text_fields = required - { + "instrument_uids", "partition_plan_epoch", "schema_major", "lease_epoch", + "public_write_allowed", "legacy_write_allowed", + } + if any(not str(payload[field]).strip() for field in text_fields): + raise ValueError("candidate slice identity fields are required") + instruments = payload["instrument_uids"] + if not isinstance(instruments, list) or not instruments or not all( + isinstance(item, str) and item.strip() for item in instruments + ) or len(set(instruments)) != len(instruments): + raise ValueError("candidate slice instruments must be non-empty and unique") + try: + for instrument_uid in instruments: + uuid.UUID(instrument_uid) + except ValueError as exc: + raise ValueError("candidate instrument UID is invalid") from exc + if not isinstance(payload["public_write_allowed"], bool) or not isinstance(payload["legacy_write_allowed"], bool): + raise ValueError("candidate write-authority flags must be booleans") + if int(payload["partition_plan_epoch"]) <= 0 or int(payload["schema_major"]) <= 0: + raise ValueError("candidate plan/schema epoch must be positive") + if int(payload["lease_epoch"]) <= 0: + raise ValueError("candidate lease epoch must be positive") + if payload["authority_state"] != "RUST_SHADOW": + raise ValueError("Phase 9.0-C candidate must remain RUST_SHADOW") + if payload["public_write_allowed"] or payload["legacy_write_allowed"]: + raise ValueError("Phase 9.0-C candidate cannot write public or legacy output") + if not _IMAGE_DIGEST.fullmatch(str(payload["artifact_image_digest"])): + raise ValueError("candidate image digest is invalid") + for field in ( + "sbom_digest", "contract_digest", "partition_plan_digest", + "rollback_manifest_digest", + ): + if not _SHA256.fullmatch(str(payload[field])): + raise ValueError(f"candidate {field} is invalid") + return cls(dict(payload)) + + @property + def digest(self) -> str: + encoded = json.dumps(self.payload, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest() + + +@dataclass(frozen=True) +class EvidenceRecord: + evidence_id: str + gate_id: str + environment: str + scope: EvidenceScope + status: str + issuer: str + observed_at_ns: int + expires_at_ns: int + artifact_path: str | None + artifact_sha256: str | None + details: dict[str, Any] + + @classmethod + def from_mapping(cls, payload: Any) -> "EvidenceRecord": + required = { + "evidence_id", "gate_id", "environment", "scope", "status", "issuer", + "observed_at_ns", "expires_at_ns", "artifact_path", "artifact_sha256", + "details", + } + if not isinstance(payload, dict) or set(payload) != required: + raise ValueError("prerequisite evidence fields are incomplete or unknown") + details = payload["details"] + if not isinstance(details, dict) or _contains_sensitive_detail(details): + raise ValueError("prerequisite evidence details are invalid or sensitive") + status = str(payload["status"]).upper() + if status not in {"PASS", "BLOCKED"}: + raise ValueError("prerequisite evidence status is invalid") + artifact_path = payload["artifact_path"] + artifact_sha = payload["artifact_sha256"] + if status == "PASS" and ( + not isinstance(artifact_path, str) + or Path(artifact_path).is_absolute() + or ".." in Path(artifact_path).parts + or not isinstance(artifact_sha, str) + or not _SHA256.fullmatch(artifact_sha) + ): + raise ValueError("passing evidence requires a repository-relative artifact hash") + observed = int(payload["observed_at_ns"]) + expires = int(payload["expires_at_ns"]) + if observed <= 0 or expires <= observed: + raise ValueError("prerequisite evidence timestamps are invalid") + identity = tuple(str(payload[key]).strip() for key in ("evidence_id", "gate_id", "environment", "issuer")) + if not all(identity): + raise ValueError("prerequisite evidence identity is required") + return cls( + evidence_id=identity[0], gate_id=identity[1], environment=identity[2].lower(), + scope=EvidenceScope(str(payload["scope"]).upper()), status=status, + issuer=identity[3], observed_at_ns=observed, expires_at_ns=expires, + artifact_path=artifact_path, artifact_sha256=artifact_sha, + details=dict(details), + ) + + +@dataclass(frozen=True) +class GateResult: + gate_id: str + passed: bool + reason: str + evidence_id: str | None + observed_scope: str | None + required_scope: str + + +@dataclass(frozen=True) +class PrerequisiteDecision: + decision: str + candidate_digest: str + policy_revision: int + results: tuple[GateResult, ...] + + def as_mapping(self) -> dict[str, Any]: + return { + "schema": "qdl.production-prerequisite-decision.v1", + "decision": self.decision, + "candidate_digest": self.candidate_digest, + "policy_revision": self.policy_revision, + "passed": sum(item.passed for item in self.results), + "blocked": sum(not item.passed for item in self.results), + "gates": [item.__dict__ for item in self.results], + } + + +def load_inventory(path: str | Path) -> tuple[EvidenceRecord, ...]: + payload = yaml.safe_load(Path(path).read_text()) + if not isinstance(payload, dict) or set(payload) != {"schema", "evidence"}: + raise ValueError("prerequisite inventory fields are incomplete or unknown") + if payload["schema"] != "qdl.production-prerequisite-inventory.v1": + raise ValueError("unsupported prerequisite inventory schema") + items = tuple(EvidenceRecord.from_mapping(item) for item in payload["evidence"]) + ids = [item.evidence_id for item in items] + gates = [item.gate_id for item in items] + if len(ids) != len(set(ids)) or len(gates) != len(set(gates)): + raise ValueError("prerequisite evidence IDs and gate bindings must be unique") + return items + + +def evaluate_prerequisites( + policy: PrerequisitePolicy, + candidate: CandidateSlice, + evidence: tuple[EvidenceRecord, ...], + *, + repository_root: str | Path, + now_ns: int, +) -> PrerequisiteDecision: + if candidate.payload["environment"].lower() != policy.environment: + raise ValueError("candidate environment does not match prerequisite policy") + by_gate = {item.gate_id: item for item in evidence} + unknown_gates = set(by_gate) - {gate.gate_id for gate in policy.gates} + if unknown_gates: + raise ValueError(f"unknown prerequisite evidence gates: {sorted(unknown_gates)}") + root = Path(repository_root).resolve() + results: list[GateResult] = [] + for gate in policy.gates: + item = by_gate.get(gate.gate_id) + reason = "PASS" + if item is None: + reason = "MISSING_EVIDENCE" + elif item.status != "PASS": + reason = "EVIDENCE_BLOCKED" + elif item.environment != policy.environment: + reason = "ENVIRONMENT_MISMATCH" + elif _SCOPE_RANK[item.scope] < _SCOPE_RANK[gate.minimum_scope]: + reason = "INSUFFICIENT_SCOPE" + elif item.observed_at_ns > now_ns + 60_000_000_000: + reason = "EVIDENCE_FROM_FUTURE" + elif item.expires_at_ns <= now_ns: + reason = "EVIDENCE_EXPIRED" + elif now_ns - item.observed_at_ns > gate.max_age_seconds * 1_000_000_000: + reason = "EVIDENCE_TOO_OLD" + elif gate.required_details - set(item.details): + reason = "DETAILS_INCOMPLETE" + elif gate.candidate_bound and item.details.get("candidate_digest") != candidate.digest: + reason = "CANDIDATE_DIGEST_MISMATCH" + elif any(item.details.get(key) != expected for key, expected in gate.assertions.items()): + reason = "ASSERTION_FAILED" + elif any( + not _is_number(item.details.get(key)) + or float(item.details[key]) < minimum + for key, minimum in gate.minimums.items() + ): + reason = "MINIMUM_NOT_MET" + elif any( + not _is_number(item.details.get(key)) + or float(item.details[key]) > maximum + for key, maximum in gate.maximums.items() + ): + reason = "MAXIMUM_EXCEEDED" + elif any( + item.details.get(detail_key) != candidate.payload.get(candidate_key) + for detail_key, candidate_key in gate.candidate_field_matches.items() + ): + reason = "CANDIDATE_FIELD_MISMATCH" + elif gate.gate_id == "exact_slice_approval" and int(item.details["hold_until_ns"]) <= now_ns: + reason = "APPROVAL_HOLD_WINDOW_INVALID" + else: + artifact = (root / str(item.artifact_path)).resolve() + if root not in artifact.parents or not artifact.is_file(): + reason = "ARTIFACT_MISSING" + elif hashlib.sha256(artifact.read_bytes()).hexdigest() != item.artifact_sha256: + reason = "ARTIFACT_DIGEST_MISMATCH" + results.append(GateResult( + gate_id=gate.gate_id, + passed=reason == "PASS", + reason=reason, + evidence_id=item.evidence_id if item else None, + observed_scope=item.scope.value if item else None, + required_scope=gate.minimum_scope.value, + )) + return PrerequisiteDecision( + decision="GO" if all(item.passed for item in results) else "NO_GO_EXTERNAL", + candidate_digest=candidate.digest, + policy_revision=policy.revision, + results=tuple(results), + ) diff --git a/rust/qdl-kafka/Cargo.toml b/rust/qdl-kafka/Cargo.toml index fcb50b7..75a6ac3 100644 --- a/rust/qdl-kafka/Cargo.toml +++ b/rust/qdl-kafka/Cargo.toml @@ -26,3 +26,11 @@ path = "src/bin/qdl-kafka-smoke.rs" [[bin]] name = "qdl-authority-rehearsal" path = "src/bin/qdl-authority-rehearsal.rs" + +[[bin]] +name = "qdl-phase91-canary-rehearsal" +path = "src/bin/qdl-phase91-canary-rehearsal.rs" + +[[bin]] +name = "qdl-phase92-primary-rehearsal" +path = "src/bin/qdl-phase92-primary-rehearsal.rs" diff --git a/rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs b/rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs new file mode 100644 index 0000000..b990ecb --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs @@ -0,0 +1,527 @@ +#![forbid(unsafe_code)] + +use std::env; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use qdl_core::transport::DurableRecord; +use qdl_kafka::{ + KafkaDurableSink, KafkaEventSource, KafkaTlsConfig, KafkaTransportConfig, + Phase9FencedKafkaSink, Phase9SinkTopics, +}; +use qdl_venue_core::authority::{ + Phase9AuthorityRecord, Phase9AuthorityState, Phase9PublicationContext, SinkTarget, +}; +use serde_json::json; +use sha2::{Digest, Sha256}; + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn now_ns() -> Result> { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn transport_config(identity: &str, group_id: &str) -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: format!("phase91-canary-{identity}"), + group_id: group_id.to_owned(), + request_timeout: Duration::from_secs(15), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/phase8-{identity}.crt"), + key_location: format!("{cert_root}/phase8-{identity}.key"), + key_password: None, + }, + }) +} + +fn durable_record( + stream: &str, + partition_key: &str, + payload: Vec, + nonce: &str, +) -> Result> { + let mut event_id = Sha256::new(); + event_id.update(stream.as_bytes()); + event_id.update(partition_key.as_bytes()); + event_id.update(nonce.as_bytes()); + event_id.update(&payload); + Ok(DurableRecord { + stream: stream.to_owned(), + partition_key: partition_key.to_owned(), + event_id: event_id.finalize().to_vec(), + payload, + accepted_at_ns: now_ns()?, + }) +} + +async fn receive_authority( + source: &KafkaEventSource, + slice_id: &str, + revision: u64, +) -> Result> { + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let (record, _) = source.next().await?; + let authority: Phase9AuthorityRecord = serde_json::from_slice(&record.payload) + .map_err(|error| qdl_kafka::KafkaTransportError::Fencing(error.to_string()))?; + if authority.slice_id == slice_id && authority.authority_revision == revision { + source.checkpoint()?; + return Ok::(authority); + } + } + }) + .await + .map_err(|_| "timed out reading persistent Phase 9 authority record")? + .map_err(|error| -> Box { Box::new(error) }) +} + +#[allow(clippy::too_many_arguments)] +fn authority_record( + slice_id: &str, + owner_id: &str, + candidate_digest: &str, + bundle_id: &str, + revision: u64, + lease_epoch: u64, + state: Phase9AuthorityState, + start_watermark: u64, + approved_at_ns: i64, + hold_until_ns: i64, +) -> Phase9AuthorityRecord { + let canary = state == Phase9AuthorityState::RustCanary; + Phase9AuthorityRecord { + schema: "qdl.authority-record.v2".into(), + slice_id: slice_id.into(), + state, + owner_id: owner_id.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + candidate_digest: candidate_digest.into(), + prerequisite_bundle_id: canary.then(|| bundle_id.into()), + start_watermark, + approved_by: canary.then(|| "phase91-isolated-rehearsal".into()), + approved_at_ns: canary.then_some(approved_at_ns), + hold_until_ns: canary.then_some(hold_until_ns), + public_write_allowed: false, + legacy_write_allowed: false, + } +} + +#[allow(clippy::too_many_arguments)] +fn publication( + slice_id: &str, + owner_id: &str, + revision: u64, + lease_epoch: u64, + partition_plan_epoch: u64, + source_watermark: u64, + target: SinkTarget, +) -> Phase9PublicationContext { + Phase9PublicationContext { + slice_id: slice_id.into(), + owner_id: owner_id.into(), + authority_revision: revision, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch, + partition_plan_epoch, + source_watermark, + target, + } +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let authority_topic = required("QDL_AUTHORITY_TOPIC")?; + let audit_topic = required("QDL_AUDIT_TOPIC")?; + let shadow_raw_topic = required("QDL_SHADOW_RAW_TOPIC")?; + let shadow_topic = required("QDL_SHADOW_CANONICAL_TOPIC")?; + let canary_topic = required("QDL_CANARY_CANONICAL_TOPIC")?; + let public_topic = required("QDL_PUBLIC_TOPIC")?; + let legacy_topic = required("QDL_LEGACY_TOPIC")?; + let nonce = required("QDL_AUTHORITY_NONCE")?; + let candidate_digest = required("QDL_CANDIDATE_DIGEST")?; + let bundle_id = required("QDL_PREREQUISITE_BUNDLE_ID")?; + let slice_id = required("QDL_SLICE_ID")?; + let shadow_owner = required("QDL_SHADOW_OWNER_ID")?; + let canary_owner = required("QDL_CANARY_OWNER_ID")?; + let group = format!("phase8-phase91-authority-{nonce}"); + let producer_config = transport_config("producer", &group)?; + let consumer_config = transport_config("consumer", &group)?; + let authority_sink = KafkaDurableSink::new(&producer_config)?; + let audit_sink = KafkaDurableSink::new(&producer_config)?; + let authority_source = KafkaEventSource::new(&consumer_config, &[&authority_topic])?; + let fenced_sink = Phase9FencedKafkaSink::new( + &producer_config, + Phase9SinkTopics { + shadow_raw: shadow_raw_topic.clone(), + shadow_canonical: shadow_topic.clone(), + canary_canonical: canary_topic.clone(), + }, + )?; + let started_at = now_ns()?; + let approved_at = started_at - 1_000_000_000; + let hold_until = started_at + 300_000_000_000; + let transitions = [ + authority_record( + &slice_id, + &shadow_owner, + &candidate_digest, + &bundle_id, + 1, + 1, + Phase9AuthorityState::RustShadow, + 100, + approved_at, + hold_until, + ), + authority_record( + &slice_id, + &canary_owner, + &candidate_digest, + &bundle_id, + 2, + 2, + Phase9AuthorityState::RustCanary, + 100, + approved_at, + hold_until, + ), + authority_record( + &slice_id, + &canary_owner, + &candidate_digest, + &bundle_id, + 3, + 2, + Phase9AuthorityState::Blocked, + 100, + approved_at, + hold_until, + ), + authority_record( + &slice_id, + &shadow_owner, + &candidate_digest, + &bundle_id, + 4, + 3, + Phase9AuthorityState::RustShadow, + 100, + approved_at, + hold_until, + ), + ]; + let mut authority_offsets = Vec::new(); + let mut audit_offsets = Vec::new(); + let mut shadow_offsets = Vec::new(); + let mut canary_offsets = Vec::new(); + let mut checks = serde_json::Map::new(); + + for authority in transitions { + let revision = authority.authority_revision; + let state = authority.state; + let durable = durable_record( + &authority_topic, + &slice_id, + serde_json::to_vec(&authority)?, + &format!("{nonce}:authority:{revision}"), + )?; + authority_offsets.push(authority_sink.append(&durable).await?.cursor.offset); + let audit = durable_record( + &audit_topic, + &format!("{slice_id}:{revision}"), + serde_json::to_vec(&authority)?, + &format!("{nonce}:audit:{revision}"), + )?; + audit_offsets.push(audit_sink.append(&audit).await?.cursor.offset); + let persisted = receive_authority(&authority_source, &slice_id, revision).await?; + fenced_sink.apply_authority(persisted).await?; + + match state { + Phase9AuthorityState::RustShadow if revision == 1 => { + let event = durable_record( + &shadow_topic, + "btc-usdt", + serde_json::to_vec(&json!({"kind": "phase91-shadow", "revision": revision}))?, + &format!("{nonce}:shadow:{revision}"), + )?; + shadow_offsets.push( + fenced_sink + .append( + &event, + &publication( + &slice_id, + &shadow_owner, + revision, + 1, + 1, + 101, + SinkTarget::ShadowCanonical, + ), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + } + Phase9AuthorityState::RustCanary => { + for source_watermark in 102..166 { + let event = durable_record( + &canary_topic, + "btc-usdt", + serde_json::to_vec(&json!({ + "kind": "phase91-canary", + "revision": revision, + "source_watermark": source_watermark, + }))?, + &format!("{nonce}:canary:{source_watermark}"), + )?; + canary_offsets.push( + fenced_sink + .append( + &event, + &publication( + &slice_id, + &canary_owner, + revision, + 2, + 1, + source_watermark, + SinkTarget::CanaryCanonical, + ), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + } + let rejected_record = |stream: &str, label: &str| { + durable_record( + stream, + label, + b"must-not-publish".to_vec(), + &format!("{nonce}:rejected:{label}"), + ) + }; + let cases = [ + ( + "public_target_rejected", + rejected_record(&public_topic, "public")?, + publication(&slice_id, &canary_owner, 2, 2, 1, 166, SinkTarget::PublicV2), + ), + ( + "legacy_target_rejected", + rejected_record(&legacy_topic, "legacy")?, + publication(&slice_id, &canary_owner, 2, 2, 1, 166, SinkTarget::LegacyV1), + ), + ( + "topic_masquerade_rejected", + rejected_record(&public_topic, "masquerade")?, + publication( + &slice_id, + &canary_owner, + 2, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "stale_owner_rejected", + rejected_record(&canary_topic, "owner")?, + publication( + &slice_id, + &shadow_owner, + 2, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "stale_revision_rejected", + rejected_record(&canary_topic, "revision")?, + publication( + &slice_id, + &canary_owner, + 1, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "stale_lease_rejected", + rejected_record(&canary_topic, "lease")?, + publication( + &slice_id, + &canary_owner, + 2, + 1, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "wrong_plan_rejected", + rejected_record(&canary_topic, "plan")?, + publication( + &slice_id, + &canary_owner, + 2, + 2, + 2, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "duplicate_watermark_rejected", + rejected_record(&canary_topic, "watermark")?, + publication( + &slice_id, + &canary_owner, + 2, + 2, + 1, + 165, + SinkTarget::CanaryCanonical, + ), + ), + ]; + for (name, event, context) in cases { + checks.insert( + name.into(), + json!(fenced_sink + .append(&event, &context, now_ns()?) + .await + .is_err()), + ); + } + } + Phase9AuthorityState::Blocked => { + let event = durable_record( + &canary_topic, + "blocked", + b"must-not-publish".to_vec(), + &format!("{nonce}:blocked"), + )?; + checks.insert( + "blocked_state_rejected".into(), + json!(fenced_sink + .append( + &event, + &publication( + &slice_id, + &canary_owner, + 3, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + now_ns()?, + ) + .await + .is_err()), + ); + } + Phase9AuthorityState::RustShadow => { + let event = durable_record( + &shadow_topic, + "btc-usdt", + serde_json::to_vec(&json!({"kind": "phase91-shadow", "revision": revision}))?, + &format!("{nonce}:shadow:{revision}"), + )?; + shadow_offsets.push( + fenced_sink + .append( + &event, + &publication( + &slice_id, + &shadow_owner, + revision, + 3, + 1, + 166, + SinkTarget::ShadowCanonical, + ), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + let rejected = durable_record( + &canary_topic, + "after-rollback", + b"must-not-publish".to_vec(), + &format!("{nonce}:after-rollback"), + )?; + checks.insert( + "canary_after_rollback_rejected".into(), + json!(fenced_sink + .append( + &rejected, + &publication( + &slice_id, + &shadow_owner, + 4, + 3, + 1, + 167, + SinkTarget::CanaryCanonical, + ), + now_ns()?, + ) + .await + .is_err()), + ); + } + } + } + + let all_rejected = checks.values().all(|value| value == &json!(true)); + let passed = authority_offsets.len() == 4 + && audit_offsets.len() == 4 + && shadow_offsets.len() == 2 + && canary_offsets.len() == 64 + && all_rejected; + println!( + "{}", + serde_json::to_string(&json!({ + "status": if passed { "PASS" } else { "FAIL" }, + "schema": "qdl.phase91.isolated-canary-runtime.v1", + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "transitions": ["RUST_SHADOW", "RUST_CANARY", "BLOCKED", "RUST_SHADOW"], + "authority_offsets": authority_offsets, + "audit_offsets": audit_offsets, + "shadow_offsets": shadow_offsets, + "canary_offsets": canary_offsets, + "checks": checks, + "public_writes": 0, + "legacy_writes": 0, + "final_authority": "RUST_SHADOW", + }))? + ); + if passed { + Ok(()) + } else { + Err("Phase 9.1 isolated canary rehearsal failed".into()) + } +} diff --git a/rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs b/rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs new file mode 100644 index 0000000..7a7456e --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs @@ -0,0 +1,947 @@ +#![forbid(unsafe_code)] + +use std::collections::BTreeSet; +use std::env; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use qdl_core::transport::DurableRecord; +use qdl_kafka::{ + KafkaDurableSink, KafkaEventSource, KafkaTlsConfig, KafkaTransportConfig, + Phase92FencedKafkaSink, Phase92SinkTopics, +}; +use qdl_venue_core::authority::{ + Phase92AcceptedHandoff, Phase92AuthorityRecord, Phase92AuthorityState, Phase92HandoffDirection, + Phase92PublicationContext, Phase92TerminalCheckpoint, SinkTarget, +}; +use serde::Serialize; +use serde_json::json; +use sha2::{Digest, Sha256}; + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn now_ns() -> Result> { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn transport_config(identity: &str, group_id: &str) -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: format!("phase92-primary-{identity}"), + group_id: group_id.to_owned(), + request_timeout: Duration::from_secs(15), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/phase8-{identity}.crt"), + key_location: format!("{cert_root}/phase8-{identity}.key"), + key_password: None, + }, + }) +} + +fn durable_record( + stream: &str, + key: &str, + value: &T, + nonce: &str, +) -> Result> { + let payload = serde_json::to_vec(value)?; + let mut event_id = Sha256::new(); + event_id.update(stream.as_bytes()); + event_id.update(key.as_bytes()); + event_id.update(nonce.as_bytes()); + event_id.update(&payload); + Ok(DurableRecord { + stream: stream.to_owned(), + partition_key: key.to_owned(), + event_id: event_id.finalize().to_vec(), + payload, + accepted_at_ns: now_ns()?, + }) +} + +async fn receive_authority( + source: &KafkaEventSource, + slice_id: &str, + revision: u64, +) -> Result> { + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let (record, _) = source.next().await?; + let authority: Phase92AuthorityRecord = serde_json::from_slice(&record.payload) + .map_err(|error| qdl_kafka::KafkaTransportError::Fencing(error.to_string()))?; + if authority.slice_id == slice_id && authority.authority_revision == revision { + source.checkpoint()?; + return Ok::(authority); + } + } + }) + .await + .map_err(|_| "timed out reading persistent Phase 9.2 authority record")? + .map_err(|error| -> Box { Box::new(error) }) +} + +async fn receive_projection_range( + source: &KafkaEventSource, + slice_id: &str, + owner_id: &str, + authority_revision: u64, + first_watermark: u64, + last_watermark: u64, +) -> Result> { + tokio::time::timeout(Duration::from_secs(30), async { + let mut observed = BTreeSet::new(); + loop { + let (record, _) = source.next().await?; + let payload: serde_json::Value = serde_json::from_slice(&record.payload) + .map_err(|error| qdl_kafka::KafkaTransportError::Fencing(error.to_string()))?; + let watermark = payload + .get("source_watermark") + .and_then(serde_json::Value::as_u64) + .ok_or(qdl_kafka::KafkaTransportError::MissingField( + "source_watermark", + ))?; + let terminal_identity_matches = watermark != last_watermark + || (payload.get("owner_id").and_then(serde_json::Value::as_str) == Some(owner_id) + && payload + .get("authority_revision") + .and_then(serde_json::Value::as_u64) + == Some(authority_revision)); + if payload.get("slice_id").and_then(serde_json::Value::as_str) != Some(slice_id) + || !terminal_identity_matches + || watermark < first_watermark + || watermark > last_watermark + || !observed.insert(watermark) + { + return Err(qdl_kafka::KafkaTransportError::Fencing( + "Phase 9.2 durable projection recovery diverged".into(), + )); + } + if watermark == last_watermark { + let expected: BTreeSet = (first_watermark..=last_watermark).collect(); + if observed != expected { + return Err(qdl_kafka::KafkaTransportError::Fencing( + "Phase 9.2 durable projection recovery has a gap".into(), + )); + } + source.checkpoint()?; + return Ok(last_watermark); + } + } + }) + .await + .map_err(|_| "timed out reading Phase 9.2 durable projection")? + .map_err(|error| -> Box { Box::new(error) }) +} + +#[allow(clippy::too_many_arguments)] +fn authority( + slice_id: &str, + state: Phase92AuthorityState, + owner_id: &str, + revision: u64, + lease_epoch: u64, + candidate_digest: &str, + bundle_id: Option<&str>, + start_watermark: u64, + terminal_watermark: Option, + previous_owner_id: Option<&str>, + handoff_digest: Option, + approved_at_ns: i64, + hold_until_ns: i64, +) -> Phase92AuthorityRecord { + let primary = matches!( + state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + let active = matches!( + state, + Phase92AuthorityState::RustCanary + | Phase92AuthorityState::RustPrimary + | Phase92AuthorityState::PythonPrimary + ); + Phase92AuthorityRecord { + schema: "qdl.authority-record.v3".into(), + slice_id: slice_id.into(), + state, + owner_id: owner_id.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + candidate_digest: candidate_digest.into(), + prerequisite_bundle_id: bundle_id.map(str::to_owned), + start_watermark, + terminal_watermark, + previous_owner_id: previous_owner_id.map(str::to_owned), + handoff_digest, + approved_by: active.then(|| "phase92-isolated-rehearsal".into()), + approved_at_ns: active.then_some(approved_at_ns), + hold_until_ns: active.then_some(hold_until_ns), + public_write_allowed: primary, + legacy_write_allowed: primary, + } +} + +#[allow(clippy::too_many_arguments)] +fn checkpoint( + checkpoint_id: &str, + slice_id: &str, + owner_id: &str, + revision: u64, + lease_epoch: u64, + watermark: u64, + candidate_digest: &str, + nonce: &str, +) -> Result> { + let terminal_payload_sha256 = hex::encode(Sha256::digest( + format!("{slice_id}:{owner_id}:{revision}:{watermark}:{nonce}").as_bytes(), + )); + Ok(Phase92TerminalCheckpoint { + schema: "qdl.terminal-owner-checkpoint.v1".into(), + checkpoint_id: checkpoint_id.into(), + slice_id: slice_id.into(), + owner_id: owner_id.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + source_session_id: format!("phase92-{owner_id}-{nonce}"), + connection_generation: 1, + terminal_watermark: watermark, + terminal_event_id: format!("phase92-event-{watermark}"), + terminal_payload_sha256, + candidate_digest: candidate_digest.into(), + committed_at_ns: now_ns()?, + }) +} + +#[allow(clippy::too_many_arguments)] +fn handoff( + handoff_id: &str, + direction: Phase92HandoffDirection, + checkpoint: &Phase92TerminalCheckpoint, + new_owner_id: &str, + new_state: Phase92AuthorityState, + bundle_id: &str, + approved_at_ns: i64, + expires_at_ns: i64, +) -> Result> { + let expected_state = match direction { + Phase92HandoffDirection::PythonToRust => Phase92AuthorityState::RustCanary, + Phase92HandoffDirection::RustToPython => Phase92AuthorityState::RollbackPending, + }; + let result = Phase92AcceptedHandoff { + schema: "qdl.accepted-authority-handoff.v1".into(), + handoff_id: handoff_id.into(), + direction, + checkpoint_digest: checkpoint.digest()?, + slice_id: checkpoint.slice_id.clone(), + old_owner_id: checkpoint.owner_id.clone(), + new_owner_id: new_owner_id.into(), + expected_state, + new_state, + expected_authority_revision: checkpoint.authority_revision, + new_authority_revision: checkpoint.authority_revision + 1, + expected_lease_epoch: checkpoint.lease_epoch, + new_lease_epoch: checkpoint.lease_epoch + 1, + partition_plan_epoch: 1, + terminal_watermark: checkpoint.terminal_watermark, + first_new_watermark: checkpoint.terminal_watermark + 1, + overlap_start_watermark: checkpoint.terminal_watermark.saturating_sub(10), + overlap_end_watermark: checkpoint.terminal_watermark, + old_event_count: 11, + new_event_count: 11, + semantic_mismatches: 0, + open_gaps: 0, + candidate_digest: checkpoint.candidate_digest.clone(), + prerequisite_bundle_id: bundle_id.into(), + approved_by: "phase92-isolated-rehearsal".into(), + approved_at_ns, + expires_at_ns, + }; + result.validate(checkpoint)?; + Ok(result) +} + +fn publication( + record: &Phase92AuthorityRecord, + watermark: u64, + target: SinkTarget, +) -> Phase92PublicationContext { + Phase92PublicationContext { + slice_id: record.slice_id.clone(), + owner_id: record.owner_id.clone(), + authority_revision: record.authority_revision, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch: record.lease_epoch, + partition_plan_epoch: record.partition_plan_epoch, + source_watermark: watermark, + target, + } +} + +async fn persist_authority( + authority_sink: &KafkaDurableSink, + audit_sink: &KafkaDurableSink, + authority_source: &KafkaEventSource, + authority_topic: &str, + audit_topic: &str, + nonce: &str, + record: &Phase92AuthorityRecord, +) -> Result<(u64, u64, Phase92AuthorityRecord), Box> { + let revision = record.authority_revision; + let authority_offset = authority_sink + .append(&durable_record( + authority_topic, + &record.slice_id, + record, + &format!("{nonce}:authority:{revision}"), + )?) + .await? + .cursor + .offset; + let audit_offset = audit_sink + .append(&durable_record( + audit_topic, + &format!("{}:{revision}", record.slice_id), + record, + &format!("{nonce}:audit:{revision}"), + )?) + .await? + .cursor + .offset; + let persisted = receive_authority(authority_source, &record.slice_id, revision).await?; + Ok((authority_offset, audit_offset, persisted)) +} + +async fn publish_range( + sink: &Phase92FencedKafkaSink, + topics: &Phase92SinkTopics, + record: &Phase92AuthorityRecord, + first: u64, + last: u64, + nonce: &str, +) -> Result, Box> { + let mut offsets = Vec::new(); + for watermark in first..=last { + let payload = json!({ + "schema": "qdl.phase92.isolated-projection.v1", + "slice_id": record.slice_id, + "owner_id": record.owner_id, + "authority_revision": record.authority_revision, + "lease_epoch": record.lease_epoch, + "source_watermark": watermark, + "provider_provenance": "REAL_PROVIDER_READ_ONLY_CAPTURE", + }); + for (target, topic) in [ + ( + SinkTarget::PrimaryCanonical, + topics.primary_canonical.as_str(), + ), + (SinkTarget::PublicV2, topics.public_v2.as_str()), + (SinkTarget::LegacyV1, topics.legacy_v1.as_str()), + ] { + offsets.push( + sink.append( + &durable_record( + topic, + "btc-usdt", + &payload, + &format!("{nonce}:{watermark}:{target:?}"), + )?, + &publication(record, watermark, target), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + } + } + Ok(offsets) +} + +#[allow(clippy::too_many_arguments)] +async fn run_recovery_verify( + authority_topic: &str, + primary_topic: &str, + public_topic: &str, + legacy_topic: &str, + slice_id: &str, + nonce: &str, +) -> Result<(), Box> { + let expected_revision: u64 = required("QDL_RECOVERY_AUTHORITY_REVISION")?.parse()?; + let first_watermark: u64 = required("QDL_RECOVERY_FIRST_WATERMARK")?.parse()?; + let last_watermark: u64 = required("QDL_RECOVERY_LAST_WATERMARK")?.parse()?; + if first_watermark == 0 || last_watermark < first_watermark { + return Err("Phase 9.2 recovery watermark range is invalid".into()); + } + + let producer_group = format!("phase8-phase92-recovery-producer-{nonce}"); + let producer_config = transport_config("producer", &producer_group)?; + let authority_group = format!("phase8-phase92-recovery-authority-{nonce}"); + let authority_config = transport_config("consumer", &authority_group)?; + let authority_source = KafkaEventSource::new(&authority_config, &[authority_topic])?; + let latest = receive_authority(&authority_source, slice_id, expected_revision).await?; + if latest.state != Phase92AuthorityState::PythonPrimary { + return Err("Phase 9.2 recovery expected terminal Python primary authority".into()); + } + + let topics = Phase92SinkTopics { + primary_canonical: primary_topic.to_owned(), + public_v2: public_topic.to_owned(), + legacy_v1: legacy_topic.to_owned(), + }; + let fenced_sink = Phase92FencedKafkaSink::new(&producer_config, topics.clone())?; + fenced_sink.apply_authority(latest.clone()).await?; + let next_watermark = last_watermark + .checked_add(1) + .ok_or("Phase 9.2 recovery watermark overflow")?; + let pre_restore = fenced_sink + .append( + &durable_record( + primary_topic, + "pre-restore", + &json!({"must_not_publish": "pre-restore"}), + &format!("{nonce}:pre-restore"), + )?, + &publication(&latest, next_watermark, SinkTarget::PrimaryCanonical), + now_ns()?, + ) + .await; + + let mut observed = serde_json::Map::new(); + let mut target_pre_restore_rejected = true; + let mut duplicate_after_restore_rejected = true; + for (name, target, topic) in [ + ("primary", SinkTarget::PrimaryCanonical, primary_topic), + ("public", SinkTarget::PublicV2, public_topic), + ("legacy", SinkTarget::LegacyV1, legacy_topic), + ] { + let group = format!("phase8-phase92-recovery-{name}-{nonce}"); + let config = transport_config("consumer", &group)?; + let source = KafkaEventSource::new(&config, &[topic])?; + let durable_watermark = receive_projection_range( + &source, + slice_id, + &latest.owner_id, + latest.authority_revision, + first_watermark, + last_watermark, + ) + .await?; + observed.insert(name.into(), json!(durable_watermark)); + + target_pre_restore_rejected &= fenced_sink + .append( + &durable_record( + topic, + "target-pre-restore", + &json!({"must_not_publish": name}), + &format!("{nonce}:{name}:pre-restore"), + )?, + &publication(&latest, next_watermark, target), + now_ns()?, + ) + .await + .is_err(); + let restored = publication(&latest, durable_watermark, target); + fenced_sink.restore_committed_watermark(&restored).await?; + duplicate_after_restore_rejected &= fenced_sink + .append( + &durable_record( + topic, + "duplicate-after-restore", + &json!({"must_not_publish": name}), + &format!("{nonce}:{name}:duplicate"), + )?, + &restored, + now_ns()?, + ) + .await + .is_err(); + } + + let projection_offsets = publish_range( + &fenced_sink, + &topics, + &latest, + next_watermark, + next_watermark, + nonce, + ) + .await?; + let checks = json!({ + "restart_pre_restore_failed_closed": pre_restore.is_err(), + "each_target_pre_restore_failed_closed": target_pre_restore_rejected, + "durable_target_watermarks_restored": observed.values().all(|value| value == &json!(last_watermark)), + "duplicate_after_restore_rejected": duplicate_after_restore_rejected, + "resumed_at_exact_next_watermark": projection_offsets.len() == 3, + }); + let passed = checks + .as_object() + .is_some_and(|values| values.values().all(|value| value == &json!(true))); + println!( + "{}", + serde_json::to_string(&json!({ + "schema": "qdl.phase92.process-restart-recovery.v1", + "status": if passed { "PASS" } else { "FAIL" }, + "mode": "RECOVERY_VERIFY", + "production_authorized": false, + "authority_revision": latest.authority_revision, + "owner_id": latest.owner_id, + "restored_target_watermarks": observed, + "resumed_watermark": next_watermark, + "projection_offsets": projection_offsets, + "checks": checks, + "production_public_writes": 0, + "production_legacy_writes": 0, + }))? + ); + if !passed { + return Err("Phase 9.2 process restart recovery failed".into()); + } + Ok(()) +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let authority_topic = required("QDL_AUTHORITY_TOPIC")?; + let audit_topic = required("QDL_AUDIT_TOPIC")?; + let checkpoint_topic = required("QDL_CHECKPOINT_TOPIC")?; + let handoff_topic = required("QDL_HANDOFF_TOPIC")?; + let primary_topic = required("QDL_PRIMARY_CANONICAL_TOPIC")?; + let public_topic = required("QDL_ISOLATED_PUBLIC_TOPIC")?; + let legacy_topic = required("QDL_ISOLATED_LEGACY_TOPIC")?; + let production_public = required("QDL_PRODUCTION_PUBLIC_TOPIC")?; + let production_legacy = required("QDL_PRODUCTION_LEGACY_TOPIC")?; + let required_topics = [ + &authority_topic, + &audit_topic, + &checkpoint_topic, + &handoff_topic, + &primary_topic, + &public_topic, + &legacy_topic, + ]; + if required_topics + .iter() + .any(|topic| !topic.contains(".phase92.")) + || required_topics + .iter() + .any(|topic| topic.as_str() == production_public || topic.as_str() == production_legacy) + { + return Err("Phase 9.2 rehearsal topic is not isolated".into()); + } + + let nonce = required("QDL_AUTHORITY_NONCE")?; + let candidate_digest = required("QDL_CANDIDATE_DIGEST")?; + let bundle_id = required("QDL_PREREQUISITE_BUNDLE_ID")?; + let slice_id = required("QDL_SLICE_ID")?; + let python_owner = required("QDL_PYTHON_OWNER_ID")?; + let rust_owner = required("QDL_RUST_OWNER_ID")?; + let rollback_owner = required("QDL_ROLLBACK_OWNER_ID")?; + let mode = env::var("QDL_REHEARSAL_MODE").unwrap_or_else(|_| "FULL".into()); + if mode == "RECOVERY_VERIFY" { + return run_recovery_verify( + &authority_topic, + &primary_topic, + &public_topic, + &legacy_topic, + &slice_id, + &nonce, + ) + .await; + } + if mode != "FULL" { + return Err(format!("unsupported Phase 9.2 rehearsal mode: {mode}").into()); + } + let group = format!("phase8-phase92-authority-{nonce}"); + let producer_config = transport_config("producer", &group)?; + let consumer_config = transport_config("consumer", &group)?; + let authority_sink = KafkaDurableSink::new(&producer_config)?; + let audit_sink = KafkaDurableSink::new(&producer_config)?; + let evidence_sink = KafkaDurableSink::new(&producer_config)?; + let authority_source = KafkaEventSource::new(&consumer_config, &[&authority_topic])?; + let topics = Phase92SinkTopics { + primary_canonical: primary_topic, + public_v2: public_topic, + legacy_v1: legacy_topic, + }; + let fenced_sink = Phase92FencedKafkaSink::new(&producer_config, topics.clone())?; + let started_at = now_ns()?; + let approved_at = started_at - 1_000_000_000; + let hold_until = started_at + 300_000_000_000; + let mut authority_offsets = Vec::new(); + let mut audit_offsets = Vec::new(); + let mut checkpoint_offsets = Vec::new(); + let mut handoff_offsets = Vec::new(); + let mut projection_offsets = Vec::new(); + let mut checks = serde_json::Map::new(); + + let initial = authority( + &slice_id, + Phase92AuthorityState::RustCanary, + &python_owner, + 7, + 11, + &candidate_digest, + Some(&bundle_id), + 89, + None, + None, + None, + approved_at, + hold_until, + ); + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &initial, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink.apply_authority(persisted).await?; + + let terminal = checkpoint( + "11111111-1111-4111-8111-111111111192", + &slice_id, + &python_owner, + 7, + 11, + 100, + &candidate_digest, + &nonce, + )?; + let to_rust = handoff( + "22222222-2222-4222-8222-222222222192", + Phase92HandoffDirection::PythonToRust, + &terminal, + &rust_owner, + Phase92AuthorityState::RustPrimary, + &bundle_id, + approved_at, + hold_until, + )?; + checkpoint_offsets.push( + evidence_sink + .append(&durable_record( + &checkpoint_topic, + &slice_id, + &terminal, + &format!("{nonce}:checkpoint:python"), + )?) + .await? + .cursor + .offset, + ); + handoff_offsets.push( + evidence_sink + .append(&durable_record( + &handoff_topic, + &slice_id, + &to_rust, + &format!("{nonce}:handoff:rust"), + )?) + .await? + .cursor + .offset, + ); + + let rust_primary = authority( + &slice_id, + Phase92AuthorityState::RustPrimary, + &rust_owner, + 8, + 12, + &candidate_digest, + Some(&bundle_id), + 100, + Some(100), + Some(&python_owner), + Some(to_rust.digest()?), + approved_at, + hold_until, + ); + checks.insert( + "direct_primary_without_handoff_rejected".into(), + json!(fenced_sink + .apply_authority(rust_primary.clone()) + .await + .is_err()), + ); + let cutover_started = Instant::now(); + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &rust_primary, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink + .apply_handoff(&terminal, &to_rust, persisted, now_ns()?) + .await?; + let cutover_ms = cutover_started.elapsed().as_secs_f64() * 1_000.0; + + for (name, context) in [ + ( + "terminal_watermark_rejected", + publication(&rust_primary, 100, SinkTarget::PrimaryCanonical), + ), + ( + "gap_watermark_rejected", + publication(&rust_primary, 102, SinkTarget::PrimaryCanonical), + ), + ( + "stale_owner_rejected", + Phase92PublicationContext { + owner_id: python_owner.clone(), + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ( + "stale_revision_rejected", + Phase92PublicationContext { + authority_revision: 7, + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ( + "stale_lease_rejected", + Phase92PublicationContext { + lease_epoch: 11, + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ( + "wrong_plan_rejected", + Phase92PublicationContext { + partition_plan_epoch: 2, + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ] { + checks.insert( + name.into(), + json!(fenced_sink + .append( + &durable_record( + &topics.primary_canonical, + "rejected", + &json!({"must_not_publish": name}), + &format!("{nonce}:rejected:{name}"), + )?, + &context, + now_ns()?, + ) + .await + .is_err()), + ); + } + + projection_offsets + .extend(publish_range(&fenced_sink, &topics, &rust_primary, 101, 164, &nonce).await?); + checks.insert( + "duplicate_after_ack_rejected".into(), + json!(fenced_sink + .append( + &durable_record( + &topics.primary_canonical, + "duplicate", + &json!({"must_not_publish": "duplicate"}), + &format!("{nonce}:duplicate"), + )?, + &publication(&rust_primary, 164, SinkTarget::PrimaryCanonical), + now_ns()?, + ) + .await + .is_err()), + ); + + let mut blocked = rust_primary.clone(); + blocked.state = Phase92AuthorityState::Blocked; + blocked.authority_revision = 9; + blocked.public_write_allowed = false; + blocked.legacy_write_allowed = false; + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &blocked, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink.apply_authority(persisted).await?; + + let mut pending = blocked.clone(); + pending.state = Phase92AuthorityState::RollbackPending; + pending.authority_revision = 10; + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &pending, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink.apply_authority(persisted).await?; + + let rust_terminal = checkpoint( + "33333333-3333-4333-8333-333333333192", + &slice_id, + &rust_owner, + 10, + 12, + 164, + &candidate_digest, + &nonce, + )?; + let to_python = handoff( + "44444444-4444-4444-8444-444444444192", + Phase92HandoffDirection::RustToPython, + &rust_terminal, + &rollback_owner, + Phase92AuthorityState::PythonPrimary, + &bundle_id, + approved_at, + hold_until, + )?; + checkpoint_offsets.push( + evidence_sink + .append(&durable_record( + &checkpoint_topic, + &slice_id, + &rust_terminal, + &format!("{nonce}:checkpoint:rust"), + )?) + .await? + .cursor + .offset, + ); + handoff_offsets.push( + evidence_sink + .append(&durable_record( + &handoff_topic, + &slice_id, + &to_python, + &format!("{nonce}:handoff:python"), + )?) + .await? + .cursor + .offset, + ); + let python_primary = authority( + &slice_id, + Phase92AuthorityState::PythonPrimary, + &rollback_owner, + 11, + 13, + &candidate_digest, + None, + 164, + Some(164), + Some(&rust_owner), + Some(to_python.digest()?), + approved_at, + hold_until, + ); + let rollback_started = Instant::now(); + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &python_primary, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink + .apply_handoff(&rust_terminal, &to_python, persisted, now_ns()?) + .await?; + projection_offsets + .extend(publish_range(&fenced_sink, &topics, &python_primary, 165, 180, &nonce).await?); + let rollback_ms = rollback_started.elapsed().as_secs_f64() * 1_000.0; + checks.insert( + "rust_after_rollback_rejected".into(), + json!(fenced_sink + .append( + &durable_record( + &topics.primary_canonical, + "stale-rust", + &json!({"must_not_publish": "stale-rust"}), + &format!("{nonce}:stale-rust"), + )?, + &publication(&rust_primary, 165, SinkTarget::PrimaryCanonical), + now_ns()?, + ) + .await + .is_err()), + ); + + let passed = checks.values().all(|value| value == &json!(true)) + && authority_offsets.len() == 5 + && audit_offsets.len() == 5 + && checkpoint_offsets.len() == 2 + && handoff_offsets.len() == 2 + && projection_offsets.len() == 240; + println!( + "{}", + serde_json::to_string(&json!({ + "schema": "qdl.phase92.isolated-primary-runtime.v1", + "status": if passed { "PASS" } else { "FAIL" }, + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "authority_transitions": [ + "RUST_CANARY", "RUST_PRIMARY", "BLOCKED", + "ROLLBACK_PENDING", "PYTHON_PRIMARY" + ], + "authority_offsets": authority_offsets, + "audit_offsets": audit_offsets, + "checkpoint_offsets": checkpoint_offsets, + "handoff_offsets": handoff_offsets, + "projection_offsets": projection_offsets, + "checks": checks, + "cutover_ms": cutover_ms, + "rollback_ms": rollback_ms, + "first_rust_watermark": 101, + "last_rust_watermark": 164, + "first_python_rollback_watermark": 165, + "last_watermark": 180, + "isolated_primary_writes": 80, + "isolated_public_writes": 80, + "isolated_legacy_writes": 80, + "production_public_writes": 0, + "production_legacy_writes": 0, + "final_authority": "PYTHON_PRIMARY", + }))? + ); + if !passed { + return Err("Phase 9.2 isolated primary rehearsal failed".into()); + } + Ok(()) +} diff --git a/rust/qdl-kafka/src/lib.rs b/rust/qdl-kafka/src/lib.rs index 7d440c9..1fea54b 100644 --- a/rust/qdl-kafka/src/lib.rs +++ b/rust/qdl-kafka/src/lib.rs @@ -5,7 +5,12 @@ use std::path::Path; use std::time::Duration; use qdl_core::transport::{AppendResult, Cursor, DurableRecord, RetryClass}; -use qdl_venue_core::authority::{AuthorityFence, AuthorityRecord, PublicationContext}; +use qdl_venue_core::authority::{ + AuthorityFence, AuthorityRecord, Phase92AcceptedHandoff, Phase92AuthorityFence, + Phase92AuthorityRecord, Phase92PublicationContext, Phase92TerminalCheckpoint, + Phase9AuthorityFence, Phase9AuthorityRecord, Phase9PublicationContext, PublicationContext, + SinkTarget, +}; use rdkafka::config::ClientConfig; use rdkafka::consumer::{CommitMode, Consumer, StreamConsumer}; use rdkafka::error::KafkaError; @@ -178,6 +183,219 @@ impl FencedKafkaSink { } } +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase9SinkTopics { + pub shadow_raw: String, + pub shadow_canonical: String, + pub canary_canonical: String, +} + +impl Phase9SinkTopics { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + let topics = [ + self.shadow_raw.as_str(), + self.shadow_canonical.as_str(), + self.canary_canonical.as_str(), + ]; + if topics.iter().any(|topic| topic.trim().is_empty()) { + return Err(KafkaTransportError::Configuration( + "Phase 9 sink topics must not be empty".into(), + )); + } + if topics[0] == topics[1] || topics[0] == topics[2] || topics[1] == topics[2] { + return Err(KafkaTransportError::Configuration( + "Phase 9 sink topics must be isolated and unique".into(), + )); + } + Ok(()) + } + + fn permits(&self, target: qdl_venue_core::authority::SinkTarget, stream: &str) -> bool { + use qdl_venue_core::authority::SinkTarget; + match target { + SinkTarget::ShadowRaw => stream == self.shadow_raw, + SinkTarget::ShadowCanonical => stream == self.shadow_canonical, + SinkTarget::CanaryCanonical => stream == self.canary_canonical, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 => false, + } + } +} + +/// Phase 9.1 sink keeps authority stable through durable ACK, then commits +/// the source watermark. A failed append remains retryable at the same watermark. +pub struct Phase9FencedKafkaSink { + sink: KafkaDurableSink, + fence: tokio::sync::Mutex, + topics: Phase9SinkTopics, +} + +impl Phase9FencedKafkaSink { + pub fn new( + config: &KafkaTransportConfig, + topics: Phase9SinkTopics, + ) -> Result { + topics.validate()?; + Ok(Self { + sink: KafkaDurableSink::new(config)?, + fence: tokio::sync::Mutex::new(Phase9AuthorityFence::default()), + topics, + }) + } + + pub async fn apply_authority( + &self, + record: Phase9AuthorityRecord, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .apply(record) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn append( + &self, + record: &DurableRecord, + publication: &Phase9PublicationContext, + now_ns: i64, + ) -> Result { + if !self.topics.permits(publication.target, &record.stream) { + return Err(KafkaTransportError::Fencing( + "publication target does not match its isolated Kafka topic".into(), + )); + } + let mut fence = self.fence.lock().await; + fence + .permits(publication, now_ns) + .map_err(KafkaTransportError::Fencing)?; + let result = self.sink.append(record).await?; + fence + .commit(publication) + .map_err(KafkaTransportError::Fencing)?; + Ok(result) + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92SinkTopics { + pub primary_canonical: String, + pub public_v2: String, + pub legacy_v1: String, +} + +impl Phase92SinkTopics { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + let topics = [ + self.primary_canonical.as_str(), + self.public_v2.as_str(), + self.legacy_v1.as_str(), + ]; + if topics.iter().any(|topic| topic.trim().is_empty()) { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 sink/projector topics must not be empty".into(), + )); + } + if topics[0] == topics[1] || topics[0] == topics[2] || topics[1] == topics[2] { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 sink/projector topics must be isolated and unique".into(), + )); + } + Ok(()) + } + + fn permits(&self, target: SinkTarget, stream: &str) -> bool { + match target { + SinkTarget::PrimaryCanonical => stream == self.primary_canonical, + SinkTarget::PublicV2 => stream == self.public_v2, + SinkTarget::LegacyV1 => stream == self.legacy_v1, + SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical | SinkTarget::CanaryCanonical => { + false + } + } + } +} + +/// Phase 9.2 final sink and compatibility projector share one authority fence. +/// The mutex remains held through durable ACK so an authority update cannot race +/// between sink acceptance and watermark commit. +pub struct Phase92FencedKafkaSink { + sink: KafkaDurableSink, + fence: tokio::sync::Mutex, + topics: Phase92SinkTopics, +} + +impl Phase92FencedKafkaSink { + pub fn new( + config: &KafkaTransportConfig, + topics: Phase92SinkTopics, + ) -> Result { + topics.validate()?; + Ok(Self { + sink: KafkaDurableSink::new(config)?, + fence: tokio::sync::Mutex::new(Phase92AuthorityFence::default()), + topics, + }) + } + + pub async fn apply_authority( + &self, + record: Phase92AuthorityRecord, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .apply(record) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn apply_handoff( + &self, + checkpoint: &Phase92TerminalCheckpoint, + handoff: &Phase92AcceptedHandoff, + record: Phase92AuthorityRecord, + now_ns: i64, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .apply_handoff(checkpoint, handoff, record, now_ns) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn restore_committed_watermark( + &self, + publication: &Phase92PublicationContext, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .restore_committed_watermark(publication) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn append( + &self, + record: &DurableRecord, + publication: &Phase92PublicationContext, + now_ns: i64, + ) -> Result { + if !self.topics.permits(publication.target, &record.stream) { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 publication target does not match its topic".into(), + )); + } + let mut fence = self.fence.lock().await; + fence + .permits(publication, now_ns) + .map_err(KafkaTransportError::Fencing)?; + let result = self.sink.append(record).await?; + fence + .commit(publication) + .map_err(KafkaTransportError::Fencing)?; + Ok(result) + } +} + pub struct KafkaDurableSink { producer: FutureProducer, request_timeout: Duration, @@ -323,8 +541,12 @@ impl KafkaEventSource { #[cfg(test)] mod tests { - use super::{KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError}; + use super::{ + KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError, Phase92SinkTopics, + Phase9SinkTopics, + }; use qdl_core::transport::RetryClass; + use qdl_venue_core::authority::SinkTarget; use std::time::Duration; #[test] @@ -346,6 +568,51 @@ mod tests { assert_eq!(error.retry_class(), RetryClass::NonRetryable); } + #[test] + fn phase9_topics_are_unique_and_bind_target_to_stream() { + let topics = Phase9SinkTopics { + shadow_raw: "qdl.phase8.phase91.shadow.raw".into(), + shadow_canonical: "qdl.phase8.phase91.shadow.canonical".into(), + canary_canonical: "qdl.phase8.phase91.canary.canonical".into(), + }; + topics.validate().unwrap(); + assert!(topics.permits( + SinkTarget::CanaryCanonical, + "qdl.phase8.phase91.canary.canonical" + )); + assert!(!topics.permits(SinkTarget::CanaryCanonical, "qdl.phase8.phase91.public")); + assert!(!topics.permits(SinkTarget::PublicV2, &topics.canary_canonical)); + + let duplicate = Phase9SinkTopics { + shadow_raw: "same".into(), + shadow_canonical: "same".into(), + canary_canonical: "other".into(), + }; + assert!(duplicate.validate().is_err()); + } + + #[test] + fn phase92_topics_are_unique_and_bind_final_and_projector_targets() { + let topics = Phase92SinkTopics { + primary_canonical: "qdl.phase92.primary.canonical".into(), + public_v2: "qdl.phase92.public.v2".into(), + legacy_v1: "qdl.phase92.legacy.v1".into(), + }; + topics.validate().unwrap(); + assert!(topics.permits(SinkTarget::PrimaryCanonical, &topics.primary_canonical)); + assert!(topics.permits(SinkTarget::PublicV2, &topics.public_v2)); + assert!(topics.permits(SinkTarget::LegacyV1, &topics.legacy_v1)); + assert!(!topics.permits(SinkTarget::PublicV2, &topics.legacy_v1)); + assert!(!topics.permits(SinkTarget::CanaryCanonical, &topics.primary_canonical)); + + let duplicate = Phase92SinkTopics { + primary_canonical: "same".into(), + public_v2: "same".into(), + legacy_v1: "other".into(), + }; + assert!(duplicate.validate().is_err()); + } + #[test] fn zero_timeout_and_empty_identity_fail_closed() { let config = KafkaTransportConfig { diff --git a/rust/qdl-venue-core/Cargo.toml b/rust/qdl-venue-core/Cargo.toml index fc6e26f..acc4925 100644 --- a/rust/qdl-venue-core/Cargo.toml +++ b/rust/qdl-venue-core/Cargo.toml @@ -6,6 +6,7 @@ license.workspace = true rust-version.workspace = true [dependencies] +hex.workspace = true serde.workspace = true serde_json.workspace = true sha2.workspace = true diff --git a/rust/qdl-venue-core/src/authority.rs b/rust/qdl-venue-core/src/authority.rs index a354195..7a8288b 100644 --- a/rust/qdl-venue-core/src/authority.rs +++ b/rust/qdl-venue-core/src/authority.rs @@ -9,12 +9,13 @@ pub enum AuthorityMode { RustCanary, } -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[derive(Clone, Copy, Debug, Deserialize, Eq, Hash, PartialEq, Serialize)] #[serde(rename_all = "SCREAMING_SNAKE_CASE")] pub enum SinkTarget { ShadowRaw, ShadowCanonical, CanaryCanonical, + PrimaryCanonical, PublicV2, LegacyV1, } @@ -151,6 +152,756 @@ impl AuthorityFence { } } +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase9AuthorityState { + RustShadow, + RustCanary, + Blocked, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase9AuthorityRecord { + pub schema: String, + pub slice_id: String, + pub state: Phase9AuthorityState, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub candidate_digest: String, + pub prerequisite_bundle_id: Option, + pub start_watermark: u64, + pub approved_by: Option, + pub approved_at_ns: Option, + pub hold_until_ns: Option, + pub public_write_allowed: bool, + pub legacy_write_allowed: bool, +} + +impl Phase9AuthorityRecord { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.authority-record.v2" + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || !valid_digest(&self.candidate_digest, false) + { + return Err("Phase 9 authority identity/epoch is invalid".into()); + } + if self.public_write_allowed || self.legacy_write_allowed { + return Err("Phase 9.1 cannot enable public or legacy writes".into()); + } + match self.state { + Phase9AuthorityState::RustCanary => { + let bundle = self + .prerequisite_bundle_id + .as_deref() + .ok_or_else(|| "canary prerequisite bundle is required".to_owned())?; + let approved_by = self + .approved_by + .as_deref() + .ok_or_else(|| "canary approver is required".to_owned())?; + let approved_at = self + .approved_at_ns + .ok_or_else(|| "canary approval time is required".to_owned())?; + let hold_until = self + .hold_until_ns + .ok_or_else(|| "canary hold time is required".to_owned())?; + if !valid_uuid(bundle) + || approved_by.trim().is_empty() + || approved_at <= 0 + || hold_until <= approved_at + { + return Err("canary approval/bundle/hold is invalid".into()); + } + } + Phase9AuthorityState::RustShadow | Phase9AuthorityState::Blocked => { + if self.prerequisite_bundle_id.is_some() + || self.approved_by.is_some() + || self.approved_at_ns.is_some() + || self.hold_until_ns.is_some() + { + return Err("non-canary authority cannot carry an approval bundle".into()); + } + } + } + Ok(()) + } +} + +fn valid_uuid(value: &str) -> bool { + let widths = [8, 4, 4, 4, 12]; + let parts: Vec<&str> = value.split('-').collect(); + parts.len() == widths.len() + && parts.iter().zip(widths).all(|(part, width)| { + part.len() == width && part.bytes().all(|byte| byte.is_ascii_hexdigit()) + }) +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase9PublicationContext { + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub shard_id: String, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub source_watermark: u64, + pub target: SinkTarget, +} + +#[derive(Default)] +pub struct Phase9AuthorityFence { + current: Option, + committed_watermarks: HashMap, +} + +impl Phase9AuthorityFence { + pub fn apply(&mut self, record: Phase9AuthorityRecord) -> Result<(), String> { + record.validate()?; + if let Some(current) = &self.current { + if record.slice_id != current.slice_id + || record.candidate_digest != current.candidate_digest + || record.partition_plan_epoch != current.partition_plan_epoch + { + return Err( + "Phase 9 authority scope/candidate/plan cannot change inside one fence".into(), + ); + } + if record.authority_revision < current.authority_revision { + return Err("stale Phase 9 authority revision".into()); + } + if record.authority_revision == current.authority_revision { + return if record == *current { + Ok(()) + } else { + Err("conflicting Phase 9 authority record at the same revision".into()) + }; + } + if record.lease_epoch < current.lease_epoch + || (record.owner_id != current.owner_id + && record.lease_epoch <= current.lease_epoch) + { + return Err("stale or conflicting Phase 9 owner lease".into()); + } + let transition_allowed = match current.state { + Phase9AuthorityState::RustShadow => matches!( + record.state, + Phase9AuthorityState::RustShadow + | Phase9AuthorityState::RustCanary + | Phase9AuthorityState::Blocked + ), + Phase9AuthorityState::RustCanary => matches!( + record.state, + Phase9AuthorityState::RustCanary + | Phase9AuthorityState::RustShadow + | Phase9AuthorityState::Blocked + ), + Phase9AuthorityState::Blocked => matches!( + record.state, + Phase9AuthorityState::Blocked | Phase9AuthorityState::RustShadow + ), + }; + if !transition_allowed { + return Err("Phase 9 authority transition is not permitted".into()); + } + } + self.current = Some(record); + Ok(()) + } + + pub fn permits(&self, context: &Phase9PublicationContext, now_ns: i64) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9 authority record is not loaded".to_owned())?; + if now_ns <= 0 + || context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + || context.shard_id.trim().is_empty() + { + return Err("publication identity does not match current Phase 9 authority".into()); + } + if matches!(context.target, SinkTarget::PublicV2 | SinkTarget::LegacyV1) { + return Err("Phase 9.1 public and legacy targets are fenced".into()); + } + let target_allowed = match current.state { + Phase9AuthorityState::RustShadow => matches!( + context.target, + SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical + ), + Phase9AuthorityState::RustCanary => { + let approved_at = current.approved_at_ns.expect("validated canary approval"); + let hold_until = current.hold_until_ns.expect("validated canary hold"); + if now_ns < approved_at || now_ns >= hold_until { + return Err("Phase 9 canary approval window is not active".into()); + } + matches!( + context.target, + SinkTarget::ShadowRaw + | SinkTarget::ShadowCanonical + | SinkTarget::CanaryCanonical + ) + } + Phase9AuthorityState::Blocked => false, + }; + if !target_allowed { + return Err("sink target is not permitted by current Phase 9 authority".into()); + } + let committed = self + .committed_watermarks + .get(&context.shard_id) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark <= committed { + return Err("source watermark is stale or already committed".into()); + } + Ok(()) + } + + pub fn commit(&mut self, context: &Phase9PublicationContext) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9 authority record is not loaded".to_owned())?; + if context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + { + return Err("authority changed before publication commit".into()); + } + let committed = self + .committed_watermarks + .get(&context.shard_id) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark <= committed { + return Err("publication watermark commit regressed".into()); + } + self.committed_watermarks + .insert(context.shard_id.clone(), context.source_watermark); + Ok(()) + } + + pub fn current(&self) -> Option<&Phase9AuthorityRecord> { + self.current.as_ref() + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase92AuthorityState { + RustCanary, + RustPrimary, + Blocked, + RollbackPending, + PythonPrimary, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase92HandoffDirection { + PythonToRust, + RustToPython, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92TerminalCheckpoint { + pub schema: String, + pub checkpoint_id: String, + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub source_session_id: String, + pub connection_generation: u64, + pub terminal_watermark: u64, + pub terminal_event_id: String, + pub terminal_payload_sha256: String, + pub candidate_digest: String, + pub committed_at_ns: i64, +} + +impl Phase92TerminalCheckpoint { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.terminal-owner-checkpoint.v1" + || !valid_uuid(&self.checkpoint_id) + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.source_session_id.trim().is_empty() + || self.terminal_event_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || self.connection_generation == 0 + || self.committed_at_ns <= 0 + || !valid_digest(&self.terminal_payload_sha256, false) + || !valid_digest(&self.candidate_digest, false) + { + return Err("Phase 9.2 terminal checkpoint is invalid".into()); + } + Ok(()) + } + + pub fn digest(&self) -> Result { + self.validate()?; + let payload = serde_json::to_vec(self) + .map_err(|error| format!("terminal checkpoint encoding failed: {error}"))?; + use sha2::Digest; + Ok(hex::encode(sha2::Sha256::digest(payload))) + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92AcceptedHandoff { + pub schema: String, + pub handoff_id: String, + pub direction: Phase92HandoffDirection, + pub checkpoint_digest: String, + pub slice_id: String, + pub old_owner_id: String, + pub new_owner_id: String, + pub expected_state: Phase92AuthorityState, + pub new_state: Phase92AuthorityState, + pub expected_authority_revision: u64, + pub new_authority_revision: u64, + pub expected_lease_epoch: u64, + pub new_lease_epoch: u64, + pub partition_plan_epoch: u64, + pub terminal_watermark: u64, + pub first_new_watermark: u64, + pub overlap_start_watermark: u64, + pub overlap_end_watermark: u64, + pub old_event_count: u64, + pub new_event_count: u64, + pub semantic_mismatches: u64, + pub open_gaps: u64, + pub candidate_digest: String, + pub prerequisite_bundle_id: String, + pub approved_by: String, + pub approved_at_ns: i64, + pub expires_at_ns: i64, +} + +impl Phase92AcceptedHandoff { + pub fn validate(&self, checkpoint: &Phase92TerminalCheckpoint) -> Result<(), String> { + checkpoint.validate()?; + if self.schema != "qdl.accepted-authority-handoff.v1" + || !valid_uuid(&self.handoff_id) + || !valid_uuid(&self.prerequisite_bundle_id) + || self.slice_id.trim().is_empty() + || self.old_owner_id.trim().is_empty() + || self.new_owner_id.trim().is_empty() + || self.old_owner_id == self.new_owner_id + || self.approved_by.trim().is_empty() + || self.partition_plan_epoch == 0 + || !valid_digest(&self.checkpoint_digest, false) + || !valid_digest(&self.candidate_digest, false) + || self.approved_at_ns <= 0 + || self.expires_at_ns <= self.approved_at_ns + { + return Err("Phase 9.2 handoff identity/approval is invalid".into()); + } + let states_match = match self.direction { + Phase92HandoffDirection::PythonToRust => { + self.expected_state == Phase92AuthorityState::RustCanary + && self.new_state == Phase92AuthorityState::RustPrimary + } + Phase92HandoffDirection::RustToPython => { + self.expected_state == Phase92AuthorityState::RollbackPending + && self.new_state == Phase92AuthorityState::PythonPrimary + } + }; + if !states_match + || self.new_authority_revision != self.expected_authority_revision + 1 + || self.new_lease_epoch <= self.expected_lease_epoch + || self.first_new_watermark != self.terminal_watermark + 1 + || self.overlap_start_watermark > self.overlap_end_watermark + || self.overlap_end_watermark != self.terminal_watermark + || self.old_event_count == 0 + || self.old_event_count != self.new_event_count + || self.semantic_mismatches != 0 + || self.open_gaps != 0 + { + return Err("Phase 9.2 handoff boundary/reconciliation is invalid".into()); + } + if self.checkpoint_digest != checkpoint.digest()? + || self.slice_id != checkpoint.slice_id + || self.old_owner_id != checkpoint.owner_id + || self.expected_authority_revision != checkpoint.authority_revision + || self.expected_lease_epoch != checkpoint.lease_epoch + || self.partition_plan_epoch != checkpoint.partition_plan_epoch + || self.terminal_watermark != checkpoint.terminal_watermark + || self.candidate_digest != checkpoint.candidate_digest + { + return Err("Phase 9.2 handoff does not bind the terminal checkpoint".into()); + } + Ok(()) + } + + pub fn digest(&self) -> Result { + let payload = serde_json::to_vec(self) + .map_err(|error| format!("accepted handoff encoding failed: {error}"))?; + use sha2::Digest; + Ok(hex::encode(sha2::Sha256::digest(payload))) + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92AuthorityRecord { + pub schema: String, + pub slice_id: String, + pub state: Phase92AuthorityState, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub candidate_digest: String, + pub prerequisite_bundle_id: Option, + pub start_watermark: u64, + pub terminal_watermark: Option, + pub previous_owner_id: Option, + pub handoff_digest: Option, + pub approved_by: Option, + pub approved_at_ns: Option, + pub hold_until_ns: Option, + pub public_write_allowed: bool, + pub legacy_write_allowed: bool, +} + +impl Phase92AuthorityRecord { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.authority-record.v3" + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || !valid_digest(&self.candidate_digest, false) + { + return Err("Phase 9.2 authority identity/epoch is invalid".into()); + } + let approval_valid = || { + self.approved_by + .as_deref() + .is_some_and(|value| !value.trim().is_empty()) + && self.approved_at_ns.is_some_and(|value| value > 0) + && self + .hold_until_ns + .is_some_and(|hold| self.approved_at_ns.is_some_and(|approved| hold > approved)) + }; + match self.state { + Phase92AuthorityState::RustCanary => { + if self.public_write_allowed + || self.legacy_write_allowed + || self.terminal_watermark.is_some() + || self.previous_owner_id.is_some() + || self.handoff_digest.is_some() + || !self + .prerequisite_bundle_id + .as_deref() + .is_some_and(valid_uuid) + || !approval_valid() + { + return Err("Phase 9.2 canary authority is invalid".into()); + } + } + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary => { + if !self.public_write_allowed + || !self.legacy_write_allowed + || self + .previous_owner_id + .as_deref() + .is_none_or(|value| value.trim().is_empty() || value == self.owner_id) + || self.terminal_watermark != Some(self.start_watermark) + || !self + .handoff_digest + .as_deref() + .is_some_and(|value| valid_digest(value, false)) + || !approval_valid() + { + return Err("Phase 9.2 primary authority/handoff is invalid".into()); + } + if self.state == Phase92AuthorityState::RustPrimary + && !self + .prerequisite_bundle_id + .as_deref() + .is_some_and(valid_uuid) + { + return Err("Rust primary requires a prerequisite bundle".into()); + } + } + Phase92AuthorityState::Blocked | Phase92AuthorityState::RollbackPending => { + if self.public_write_allowed || self.legacy_write_allowed { + return Err("blocked/rollback authority cannot write".into()); + } + } + } + Ok(()) + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92PublicationContext { + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub shard_id: String, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub source_watermark: u64, + pub target: SinkTarget, +} + +#[derive(Default)] +pub struct Phase92AuthorityFence { + current: Option, + committed_watermarks: HashMap<(String, SinkTarget), u64>, + recovery_required: bool, +} + +impl Phase92AuthorityFence { + pub fn apply(&mut self, record: Phase92AuthorityRecord) -> Result<(), String> { + record.validate()?; + if self.current.is_none() { + self.recovery_required = matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + self.current = Some(record); + return Ok(()); + } + if matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ) { + return Err("primary ownership transition requires accepted handoff".into()); + } + self.apply_transition(record) + } + + pub fn apply_handoff( + &mut self, + checkpoint: &Phase92TerminalCheckpoint, + handoff: &Phase92AcceptedHandoff, + record: Phase92AuthorityRecord, + now_ns: i64, + ) -> Result<(), String> { + handoff.validate(checkpoint)?; + record.validate()?; + if now_ns <= 0 || now_ns < handoff.approved_at_ns || now_ns >= handoff.expires_at_ns { + return Err("Phase 9.2 handoff approval window is not active".into()); + } + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if current.slice_id != handoff.slice_id + || current.owner_id != handoff.old_owner_id + || current.state != handoff.expected_state + || current.authority_revision != handoff.expected_authority_revision + || current.lease_epoch != handoff.expected_lease_epoch + || current.partition_plan_epoch != handoff.partition_plan_epoch + || current.candidate_digest != handoff.candidate_digest + || record.slice_id != handoff.slice_id + || record.owner_id != handoff.new_owner_id + || record.state != handoff.new_state + || record.authority_revision != handoff.new_authority_revision + || record.lease_epoch != handoff.new_lease_epoch + || record.partition_plan_epoch != handoff.partition_plan_epoch + || record.candidate_digest != handoff.candidate_digest + || record.start_watermark != handoff.terminal_watermark + || record.terminal_watermark != Some(handoff.terminal_watermark) + || record.previous_owner_id.as_deref() != Some(handoff.old_owner_id.as_str()) + || record.handoff_digest.as_deref() != Some(handoff.digest()?.as_str()) + { + return Err("Phase 9.2 authority CAS/handoff binding failed".into()); + } + self.apply_transition(record) + } + + fn apply_transition(&mut self, record: Phase92AuthorityRecord) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if record.slice_id != current.slice_id + || record.candidate_digest != current.candidate_digest + || record.partition_plan_epoch != current.partition_plan_epoch + || record.authority_revision != current.authority_revision + 1 + || record.lease_epoch < current.lease_epoch + || (record.owner_id != current.owner_id && record.lease_epoch <= current.lease_epoch) + { + return Err("Phase 9.2 authority compare-and-swap failed".into()); + } + let allowed = match current.state { + Phase92AuthorityState::RustCanary => matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::Blocked + ), + Phase92AuthorityState::RustPrimary => matches!( + record.state, + Phase92AuthorityState::Blocked | Phase92AuthorityState::RollbackPending + ), + Phase92AuthorityState::Blocked => { + record.state == Phase92AuthorityState::RollbackPending + } + Phase92AuthorityState::RollbackPending => { + record.state == Phase92AuthorityState::PythonPrimary + } + Phase92AuthorityState::PythonPrimary => record.state == Phase92AuthorityState::Blocked, + }; + if !allowed { + return Err("Phase 9.2 authority transition is not permitted".into()); + } + self.current = Some(record); + self.recovery_required = false; + Ok(()) + } + + pub fn restore_committed_watermark( + &mut self, + context: &Phase92PublicationContext, + ) -> Result<(), String> { + if !self.recovery_required { + return Err("Phase 9.2 watermark restore is only permitted during recovery".into()); + } + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if !matches!( + current.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ) || context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + || context.shard_id.trim().is_empty() + || !matches!( + context.target, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 + ) + || context.source_watermark < current.start_watermark + { + return Err("Phase 9.2 recovered watermark identity is invalid".into()); + } + let key = (context.shard_id.clone(), context.target); + if self + .committed_watermarks + .get(&key) + .is_some_and(|value| context.source_watermark < *value) + { + return Err("Phase 9.2 recovered watermark regressed".into()); + } + self.committed_watermarks + .insert(key, context.source_watermark); + Ok(()) + } + + pub fn permits(&self, context: &Phase92PublicationContext, now_ns: i64) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if now_ns <= 0 + || context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + || context.shard_id.trim().is_empty() + { + return Err("publication identity does not match Phase 9.2 authority".into()); + } + let target_allowed = match current.state { + Phase92AuthorityState::RustCanary => context.target == SinkTarget::CanaryCanonical, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary => matches!( + context.target, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 + ), + Phase92AuthorityState::Blocked | Phase92AuthorityState::RollbackPending => false, + }; + if !target_allowed { + return Err("sink target is not permitted by Phase 9.2 authority".into()); + } + if matches!( + current.state, + Phase92AuthorityState::RustCanary + | Phase92AuthorityState::RustPrimary + | Phase92AuthorityState::PythonPrimary + ) { + let approved_at = current + .approved_at_ns + .ok_or_else(|| "authority approval is missing".to_owned())?; + let hold_until = current + .hold_until_ns + .ok_or_else(|| "authority hold window is missing".to_owned())?; + if now_ns < approved_at || now_ns >= hold_until { + return Err("Phase 9.2 authority approval window is not active".into()); + } + } + let key = (context.shard_id.clone(), context.target); + if self.recovery_required && !self.committed_watermarks.contains_key(&key) { + return Err("Phase 9.2 durable target watermark recovery is required".into()); + } + let committed = self + .committed_watermarks + .get(&key) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark != committed + 1 { + return Err("Phase 9.2 source watermark is duplicate, stale or gapped".into()); + } + Ok(()) + } + + pub fn commit(&mut self, context: &Phase92PublicationContext) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + { + return Err("Phase 9.2 authority changed before commit".into()); + } + let key = (context.shard_id.clone(), context.target); + let committed = self + .committed_watermarks + .get(&key) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark != committed + 1 { + return Err("Phase 9.2 watermark commit is not contiguous".into()); + } + self.committed_watermarks + .insert(key, context.source_watermark); + Ok(()) + } + + pub fn current(&self) -> Option<&Phase92AuthorityRecord> { + self.current.as_ref() + } +} + #[cfg(test)] mod tests { use super::{AuthorityFence, AuthorityMode, AuthorityRecord, PublicationContext, SinkTarget}; @@ -221,3 +972,567 @@ mod tests { .is_err()); } } + +#[cfg(test)] +mod phase9_tests { + use super::{ + Phase9AuthorityFence, Phase9AuthorityRecord, Phase9AuthorityState, + Phase9PublicationContext, SinkTarget, + }; + + const SLICE: &str = "production/binance/usdm/perpetual/trade/plan-1/btcusdt"; + const OWNER: &str = "rust-ingestor-binance-usdm-shard-0"; + const BUNDLE: &str = "558042db-a766-5a55-b5b3-4b508d649df9"; + + fn record( + revision: u64, + lease_epoch: u64, + state: Phase9AuthorityState, + ) -> Phase9AuthorityRecord { + let canary = state == Phase9AuthorityState::RustCanary; + Phase9AuthorityRecord { + schema: "qdl.authority-record.v2".into(), + slice_id: SLICE.into(), + state, + owner_id: OWNER.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: canary.then(|| BUNDLE.into()), + start_watermark: 100, + approved_by: canary.then(|| "phase91-operator".into()), + approved_at_ns: canary.then_some(1_000), + hold_until_ns: canary.then_some(10_000), + public_write_allowed: false, + legacy_write_allowed: false, + } + } + + fn publication(target: SinkTarget, watermark: u64) -> Phase9PublicationContext { + Phase9PublicationContext { + slice_id: SLICE.into(), + owner_id: OWNER.into(), + authority_revision: 2, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch: 2, + partition_plan_epoch: 1, + source_watermark: watermark, + target, + } + } + + #[test] + fn canary_allows_only_isolated_target_inside_approval_window() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(1, 1, Phase9AuthorityState::RustShadow)) + .unwrap(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 101), 2_000) + .is_ok()); + assert!(fence + .permits(&publication(SinkTarget::PublicV2, 101), 2_000) + .is_err()); + assert!(fence + .permits(&publication(SinkTarget::LegacyV1, 101), 2_000) + .is_err()); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 101), 999) + .is_err()); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 101), 10_000) + .is_err()); + } + + #[test] + fn publication_binds_every_identity_and_epoch() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + let base = publication(SinkTarget::CanaryCanonical, 101); + let variants = [ + Phase9PublicationContext { + slice_id: "other".into(), + ..base.clone() + }, + Phase9PublicationContext { + owner_id: "stale-owner".into(), + ..base.clone() + }, + Phase9PublicationContext { + authority_revision: 1, + ..base.clone() + }, + Phase9PublicationContext { + lease_epoch: 1, + ..base.clone() + }, + Phase9PublicationContext { + partition_plan_epoch: 2, + ..base.clone() + }, + Phase9PublicationContext { + shard_id: String::new(), + ..base.clone() + }, + ]; + for context in variants { + assert!(fence.permits(&context, 2_000).is_err()); + } + } + + #[test] + fn watermark_advances_only_after_explicit_durable_commit() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + let first = publication(SinkTarget::CanaryCanonical, 101); + assert!(fence.permits(&first, 2_000).is_ok()); + assert!(fence.permits(&first, 2_000).is_ok()); + fence.commit(&first).unwrap(); + assert!(fence.permits(&first, 2_000).is_err()); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 100), 2_000) + .is_err()); + let second = publication(SinkTarget::CanaryCanonical, 102); + assert!(fence.permits(&second, 2_000).is_ok()); + fence.commit(&second).unwrap(); + } + + #[test] + fn blocked_state_fences_every_target_and_requires_shadow_before_canary() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(1, 1, Phase9AuthorityState::RustShadow)) + .unwrap(); + fence + .apply(record(2, 1, Phase9AuthorityState::Blocked)) + .unwrap(); + let mut context = publication(SinkTarget::ShadowCanonical, 101); + context.owner_id = OWNER.into(); + context.authority_revision = 2; + context.lease_epoch = 1; + assert!(fence.permits(&context, 2_000).is_err()); + assert!(fence + .apply(record(3, 2, Phase9AuthorityState::RustCanary)) + .is_err()); + fence + .apply(record(3, 2, Phase9AuthorityState::RustShadow)) + .unwrap(); + } + + #[test] + fn malformed_records_and_stale_transitions_fail_closed() { + let mut invalid = record(1, 1, Phase9AuthorityState::RustCanary); + invalid.public_write_allowed = true; + assert!(invalid.validate().is_err()); + let mut invalid = record(1, 1, Phase9AuthorityState::RustCanary); + invalid.prerequisite_bundle_id = None; + assert!(invalid.validate().is_err()); + let mut invalid = record(1, 1, Phase9AuthorityState::RustShadow); + invalid.approved_by = Some("unexpected".into()); + assert!(invalid.validate().is_err()); + + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + assert!(fence + .apply(record(1, 2, Phase9AuthorityState::RustShadow)) + .is_err()); + let mut conflict = record(2, 2, Phase9AuthorityState::RustCanary); + conflict.owner_id = "conflicting-owner".into(); + assert!(fence.apply(conflict).is_err()); + let mut stale_owner = record(3, 2, Phase9AuthorityState::RustShadow); + stale_owner.owner_id = "next-owner".into(); + assert!(fence.apply(stale_owner).is_err()); + let mut wrong_candidate = record(3, 3, Phase9AuthorityState::RustShadow); + wrong_candidate.candidate_digest = "2".repeat(64); + assert!(fence.apply(wrong_candidate).is_err()); + } +} + +#[cfg(test)] +mod phase92_tests { + use super::{ + Phase92AcceptedHandoff, Phase92AuthorityFence, Phase92AuthorityRecord, + Phase92AuthorityState, Phase92HandoffDirection, Phase92PublicationContext, + Phase92TerminalCheckpoint, SinkTarget, + }; + + const SLICE: &str = "production/binance/usdm/perpetual/trade/plan-1/btcusdt"; + const BUNDLE: &str = "558042db-a766-5a55-b5b3-4b508d649df9"; + + fn canary() -> Phase92AuthorityRecord { + Phase92AuthorityRecord { + schema: "qdl.authority-record.v3".into(), + slice_id: SLICE.into(), + state: Phase92AuthorityState::RustCanary, + owner_id: "python-primary".into(), + authority_revision: 7, + lease_epoch: 11, + partition_plan_epoch: 1, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: Some(BUNDLE.into()), + start_watermark: 89, + terminal_watermark: None, + previous_owner_id: None, + handoff_digest: None, + approved_by: Some("phase92-operator".into()), + approved_at_ns: Some(1), + hold_until_ns: Some(10_000), + public_write_allowed: false, + legacy_write_allowed: false, + } + } + + fn checkpoint( + owner: &str, + revision: u64, + lease: u64, + watermark: u64, + ) -> Phase92TerminalCheckpoint { + Phase92TerminalCheckpoint { + schema: "qdl.terminal-owner-checkpoint.v1".into(), + checkpoint_id: "11111111-1111-4111-8111-111111111111".into(), + slice_id: SLICE.into(), + owner_id: owner.into(), + authority_revision: revision, + lease_epoch: lease, + partition_plan_epoch: 1, + source_session_id: "session-1".into(), + connection_generation: 1, + terminal_watermark: watermark, + terminal_event_id: format!("event-{watermark}"), + terminal_payload_sha256: "2".repeat(64), + candidate_digest: "1".repeat(64), + committed_at_ns: 1, + } + } + + fn handoff( + checkpoint: &Phase92TerminalCheckpoint, + direction: Phase92HandoffDirection, + new_owner: &str, + new_state: Phase92AuthorityState, + ) -> Phase92AcceptedHandoff { + Phase92AcceptedHandoff { + schema: "qdl.accepted-authority-handoff.v1".into(), + handoff_id: "22222222-2222-4222-8222-222222222222".into(), + direction, + checkpoint_digest: checkpoint.digest().unwrap(), + slice_id: SLICE.into(), + old_owner_id: checkpoint.owner_id.clone(), + new_owner_id: new_owner.into(), + expected_state: if direction == Phase92HandoffDirection::PythonToRust { + Phase92AuthorityState::RustCanary + } else { + Phase92AuthorityState::RollbackPending + }, + new_state, + expected_authority_revision: checkpoint.authority_revision, + new_authority_revision: checkpoint.authority_revision + 1, + expected_lease_epoch: checkpoint.lease_epoch, + new_lease_epoch: checkpoint.lease_epoch + 1, + partition_plan_epoch: 1, + terminal_watermark: checkpoint.terminal_watermark, + first_new_watermark: checkpoint.terminal_watermark + 1, + overlap_start_watermark: checkpoint.terminal_watermark - 10, + overlap_end_watermark: checkpoint.terminal_watermark, + old_event_count: 11, + new_event_count: 11, + semantic_mismatches: 0, + open_gaps: 0, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: BUNDLE.into(), + approved_by: "phase92-operator".into(), + approved_at_ns: 1, + expires_at_ns: 10_000, + } + } + + fn primary( + checkpoint: &Phase92TerminalCheckpoint, + handoff: &Phase92AcceptedHandoff, + ) -> Phase92AuthorityRecord { + Phase92AuthorityRecord { + schema: "qdl.authority-record.v3".into(), + slice_id: SLICE.into(), + state: handoff.new_state, + owner_id: handoff.new_owner_id.clone(), + authority_revision: handoff.new_authority_revision, + lease_epoch: handoff.new_lease_epoch, + partition_plan_epoch: 1, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: (handoff.new_state == Phase92AuthorityState::RustPrimary) + .then(|| BUNDLE.into()), + start_watermark: checkpoint.terminal_watermark, + terminal_watermark: Some(checkpoint.terminal_watermark), + previous_owner_id: Some(checkpoint.owner_id.clone()), + handoff_digest: Some(handoff.digest().unwrap()), + approved_by: Some("phase92-operator".into()), + approved_at_ns: Some(1), + hold_until_ns: Some(10_000), + public_write_allowed: true, + legacy_write_allowed: true, + } + } + + fn publication( + record: &Phase92AuthorityRecord, + target: SinkTarget, + watermark: u64, + ) -> Phase92PublicationContext { + Phase92PublicationContext { + slice_id: SLICE.into(), + owner_id: record.owner_id.clone(), + authority_revision: record.authority_revision, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch: record.lease_epoch, + partition_plan_epoch: 1, + source_watermark: watermark, + target, + } + } + + #[test] + fn accepted_handoff_is_required_and_first_primary_watermark_is_terminal_plus_one() { + let current = canary(); + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let handoff = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let primary = primary(&checkpoint, &handoff); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(current).unwrap(); + assert!(fence.apply(primary.clone()).is_err()); + fence + .apply_handoff(&checkpoint, &handoff, primary.clone(), 2) + .unwrap(); + assert!(fence + .restore_committed_watermark(&publication(&primary, SinkTarget::PrimaryCanonical, 120,)) + .is_err()); + + for target in [ + SinkTarget::PrimaryCanonical, + SinkTarget::PublicV2, + SinkTarget::LegacyV1, + ] { + assert!(fence + .permits(&publication(&primary, target, 100), 2) + .is_err()); + assert!(fence + .permits(&publication(&primary, target, 102), 2) + .is_err()); + let first = publication(&primary, target, 101); + fence.permits(&first, 2).unwrap(); + fence.commit(&first).unwrap(); + assert!(fence.permits(&first, 2).is_err()); + } + } + + #[test] + fn final_sink_and_projector_watermarks_are_independent_and_gap_free() { + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let handoff = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let primary = primary(&checkpoint, &handoff); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary()).unwrap(); + fence + .apply_handoff(&checkpoint, &handoff, primary.clone(), 2) + .unwrap(); + + let canonical = publication(&primary, SinkTarget::PrimaryCanonical, 101); + fence.permits(&canonical, 2).unwrap(); + fence.commit(&canonical).unwrap(); + assert!(fence + .permits(&publication(&primary, SinkTarget::PrimaryCanonical, 102), 2) + .is_ok()); + assert!(fence + .permits(&publication(&primary, SinkTarget::PublicV2, 101), 2) + .is_ok()); + assert!(fence + .permits(&publication(&primary, SinkTarget::LegacyV1, 101), 2) + .is_ok()); + } + + #[test] + fn stale_owner_revision_lease_plan_and_wrong_target_fail_closed() { + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let handoff = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let primary = primary(&checkpoint, &handoff); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary()).unwrap(); + fence + .apply_handoff(&checkpoint, &handoff, primary.clone(), 2) + .unwrap(); + let base = publication(&primary, SinkTarget::PrimaryCanonical, 101); + let variants = [ + Phase92PublicationContext { + owner_id: "python-primary".into(), + ..base.clone() + }, + Phase92PublicationContext { + authority_revision: 7, + ..base.clone() + }, + Phase92PublicationContext { + lease_epoch: 11, + ..base.clone() + }, + Phase92PublicationContext { + partition_plan_epoch: 2, + ..base.clone() + }, + Phase92PublicationContext { + target: SinkTarget::CanaryCanonical, + ..base + }, + ]; + for value in variants { + assert!(fence.permits(&value, 2).is_err()); + } + } + + #[test] + fn formal_rollback_fences_rust_and_hands_off_to_python_with_new_epoch() { + let initial_checkpoint = checkpoint("python-primary", 7, 11, 100); + let to_rust = handoff( + &initial_checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let rust_primary = primary(&initial_checkpoint, &to_rust); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary()).unwrap(); + fence + .apply_handoff(&initial_checkpoint, &to_rust, rust_primary.clone(), 2) + .unwrap(); + + let mut blocked = rust_primary.clone(); + blocked.state = Phase92AuthorityState::Blocked; + blocked.authority_revision += 1; + blocked.public_write_allowed = false; + blocked.legacy_write_allowed = false; + fence.apply(blocked.clone()).unwrap(); + let mut pending = blocked; + pending.state = Phase92AuthorityState::RollbackPending; + pending.authority_revision += 1; + fence.apply(pending.clone()).unwrap(); + + let rollback_checkpoint = checkpoint( + "rust-primary", + pending.authority_revision, + pending.lease_epoch, + 120, + ); + let to_python = handoff( + &rollback_checkpoint, + Phase92HandoffDirection::RustToPython, + "python-rollback", + Phase92AuthorityState::PythonPrimary, + ); + let python_primary = primary(&rollback_checkpoint, &to_python); + fence + .apply_handoff(&rollback_checkpoint, &to_python, python_primary.clone(), 2) + .unwrap(); + assert!(fence + .permits( + &publication(&rust_primary, SinkTarget::PrimaryCanonical, 101), + 2, + ) + .is_err()); + assert!(fence + .permits( + &publication(&python_primary, SinkTarget::PrimaryCanonical, 121), + 2, + ) + .is_ok()); + } + + #[test] + fn crash_before_cas_reconstructs_canary_and_accepts_only_exact_handoff() { + let persisted_canary = canary(); + let terminal = checkpoint("python-primary", 7, 11, 100); + let accepted = handoff( + &terminal, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let rust_primary = primary(&terminal, &accepted); + + let mut recovered = Phase92AuthorityFence::default(); + recovered.apply(persisted_canary).unwrap(); + recovered + .apply_handoff(&terminal, &accepted, rust_primary.clone(), 2) + .unwrap(); + assert_eq!(recovered.current(), Some(&rust_primary)); + assert!(recovered + .apply_handoff(&terminal, &accepted, rust_primary, 2) + .is_err()); + } + + #[test] + fn restarted_primary_fails_closed_until_each_target_watermark_is_restored() { + let initial_checkpoint = checkpoint("python-primary", 7, 11, 100); + let to_rust = handoff( + &initial_checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let rust_primary = primary(&initial_checkpoint, &to_rust); + let mut recovered = Phase92AuthorityFence::default(); + recovered.apply(rust_primary.clone()).unwrap(); + let duplicate = publication(&rust_primary, SinkTarget::PrimaryCanonical, 120); + assert!(recovered.permits(&duplicate, 2).is_err()); + recovered.restore_committed_watermark(&duplicate).unwrap(); + assert!(recovered.permits(&duplicate, 2).is_err()); + assert!(recovered + .permits( + &publication(&rust_primary, SinkTarget::PrimaryCanonical, 121), + 2, + ) + .is_ok()); + assert!(recovered + .permits(&publication(&rust_primary, SinkTarget::PublicV2, 121), 2) + .is_err()); + } + + #[test] + fn dirty_or_off_by_one_handoff_is_rejected() { + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let mut dirty = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + dirty.semantic_mismatches = 1; + assert!(dirty.validate(&checkpoint).is_err()); + dirty.semantic_mismatches = 0; + dirty.first_new_watermark = 102; + assert!(dirty.validate(&checkpoint).is_err()); + } +} diff --git a/scripts/phase5_migration_smoke.sh b/scripts/phase5_migration_smoke.sh index 9089075..0dcafb6 100755 --- a/scripts/phase5_migration_smoke.sh +++ b/scripts/phase5_migration_smoke.sh @@ -45,11 +45,17 @@ if [[ "${container_started}" != "true" ]]; then fi ready=false +consecutive_ready=0 for _ in $(seq 1 240); do if docker exec "${container}" psql -U postgres -d postgres -Atc "SELECT 1" \ 2>/dev/null | grep -qx "1"; then - ready=true - break + consecutive_ready=$((consecutive_ready + 1)) + if [[ "${consecutive_ready}" -ge 8 ]]; then + ready=true + break + fi + else + consecutive_ready=0 fi sleep 0.25 done diff --git a/scripts/phase73_public_beta_certification.sh b/scripts/phase73_public_beta_certification.sh index 2c15ffb..de32581 100755 --- a/scripts/phase73_public_beta_certification.sh +++ b/scripts/phase73_public_beta_certification.sh @@ -32,10 +32,22 @@ export QDL_BETA_STREAM_A_GRPC_PORT="${STREAM_A_GRPC_PORT}" export QDL_BETA_STREAM_B_GRPC_PORT="${STREAM_B_GRPC_PORT}" temporary="$(mktemp -d)" -chown 10001:10001 "${temporary}" +runner_evidence="${temporary}/runner" +mkdir -m 0700 "${runner_evidence}" +OPERATOR_UID="$(id -u)" +OPERATOR_GID="$(id -g)" +set_evidence_owner() { + local uid="$1" gid="$2" + docker run --rm --network none --user 0:0 --cap-drop ALL --cap-add CHOWN --cap-add FOWNER \ + --security-opt no-new-privileges:true --entrypoint sh \ + -v "${runner_evidence}:/evidence" "${QDL_BETA_INIT_IMAGE}" \ + -c "chown ${uid}:${gid} /evidence && chmod 0755 /evidence" +} +set_evidence_owner 10001 10001 cleanup() { docker compose -p "${PROJECT}" -f "${COMPOSE_FILE}" \ --profile phase7-beta down -v --remove-orphans >/dev/null 2>&1 || true + set_evidence_owner "${OPERATOR_UID}" "${OPERATOR_GID}" >/dev/null 2>&1 || true rm -rf "${temporary}" } trap cleanup EXIT @@ -123,7 +135,7 @@ docker run --rm --network host --read-only --cap-drop ALL \ --security-opt no-new-privileges:true --pids-limit 256 --memory 512m --cpus 1 \ --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=32m,uid=10001,gid=10001 \ -v "${durable_volume}:/var/lib/qdl-beta-durable" \ - -v "${temporary}:/evidence" \ + -v "${runner_evidence}:/evidence" \ "${QDL_BETA_IMAGE}" python /app/scripts/phase73_beta_certification.py \ --source-bindings /app/config/phase7/canary-sources.yaml \ --monitoring-manifest /app/consumers/beta/phase7-monitoring-binance.yaml \ @@ -184,6 +196,6 @@ python3 "${ROOT_DIR}/scripts/phase73_runtime_evidence.py" \ --keys-after "${keys_after}" --containers-after "${containers_after}" \ --networks-after "${networks_after}" --volumes-after "${volumes_after}" python3 "${ROOT_DIR}/scripts/phase73_finalize_evidence.py" \ - --core "${temporary}/core.json" --stats "${temporary}/stats.jsonl" \ + --core "${runner_evidence}/core.json" --stats "${temporary}/stats.jsonl" \ --runtime "${temporary}/runtime.json" --output "${CAPACITY_OUTPUT}" \ --security-output "${SECURITY_OUTPUT}" diff --git a/scripts/phase80_broker_certification.py b/scripts/phase80_broker_certification.py index 126f180..df91cd7 100755 --- a/scripts/phase80_broker_certification.py +++ b/scripts/phase80_broker_certification.py @@ -102,12 +102,16 @@ def compose(env: dict[str, str], *arguments: str, **kwargs: object) -> CommandRe def kafka(env: dict[str, str], script: str, *arguments: str, **kwargs: object) -> CommandResult: + # Kafka CLI tools are JVMs. Running them inside a 512 MiB broker cgroup can + # OOM-kill the broker, so administration uses a disposable isolated client. return compose( env, - "exec", - "-T", - "kafka1", + "run", + "--rm", + "--no-deps", + "--entrypoint", f"/opt/kafka/bin/{script}", + "phase8_admin", *arguments, **kwargs, ) diff --git a/scripts/phase80_generate_tls.sh b/scripts/phase80_generate_tls.sh index d0345d7..03a824c 100755 --- a/scripts/phase80_generate_tls.sh +++ b/scripts/phase80_generate_tls.sh @@ -5,6 +5,8 @@ ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" OUTPUT_DIR="${1:?usage: phase80_generate_tls.sh OUTPUT_DIR}" KAFKA_IMAGE="${QDL_PHASE8_KAFKA_IMAGE:-apache/kafka@sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9}" PASSWORD="${QDL_PHASE8_CERT_PASSWORD:-phase8-certification-only}" +CERT_UID="${QDL_PHASE8_CERT_UID:-$(id -u)}" +CERT_GID="${QDL_PHASE8_CERT_GID:-$(id -g)}" umask 077 mkdir -p "${OUTPUT_DIR}" @@ -54,7 +56,9 @@ printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/key.password" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/store.password" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/truststore.password" -docker run --rm --user 0:0 \ +# Match the host caller so bind-mounted stores remain chmod/removal-safe on +# rootless, user-namespaced and migrated Docker hosts. +docker run --rm --user "${CERT_UID}:${CERT_GID}" \ --mount "type=bind,source=${OUTPUT_DIR},target=/certs" \ --entrypoint /opt/java/openjdk/bin/keytool \ "${KAFKA_IMAGE}" \ diff --git a/scripts/phase90b_bridge_parity.py b/scripts/phase90b_bridge_parity.py new file mode 100755 index 0000000..7edd959 --- /dev/null +++ b/scripts/phase90b_bridge_parity.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + +from qdl.runtime.canary_source import CanarySourceBinding, CanarySourceCatalog + + +def _decimal_source(payload: dict[str, Any], field: str) -> str: + value = payload.get(field) + if not isinstance(value, dict) or not isinstance(value.get("source_text"), str): + raise AssertionError(f"V2 {field} has no exact decimal source_text") + return value["source_text"] + + +def validate_sample( + v1: dict[str, Any], + v2: dict[str, Any], + binding: CanarySourceBinding, +) -> dict[str, int]: + identity = { + "provider": v1.get("provider") == "binance", + "market": str(v1.get("market", "")).lower() == "usdm", + "symbol": str(v1.get("symbol", "")).upper() + == binding.instrument.native_symbol, + "interval": v1.get("requested_interval") == binding.interval, + "schema": v2.get("schema") == "qdl.marketdata.warmup.v2", + } + failed = [name for name, passed in identity.items() if not passed] + if failed: + raise AssertionError(f"V1/V2 response identity mismatch: {failed}") + + rows = v1.get("data") + events = v2.get("data") + if not isinstance(rows, list) or not isinstance(events, list) or not events: + raise AssertionError("V1 rows and non-empty V2 data are required") + if v2.get("count") != len(events): + raise AssertionError("V2 count does not match data length") + expected = {int(row[0]) * 1_000_000: row for row in rows} + if len(expected) != len(rows): + raise AssertionError("V1 source contains duplicate open times") + + open_times: list[int] = [] + for item in events: + payload = item.get("payload", {}) + source = item.get("source", {}) + quality = item.get("quality", {}) + open_time = int(payload.get("open_time_ns", 0)) + row = expected.get(open_time) + if row is None: + raise AssertionError("V2 bar does not correspond to a V1 source row") + checks = { + "instrument_uid": item.get("instrument_uid") + == binding.instrument.instrument_uid, + "instrument_id": item.get("instrument_id") + == binding.instrument.instrument_id, + "feed": item.get("feed") == "BAR" and payload.get("feed") == "BAR", + "interval": item.get("interval") == binding.interval + and payload.get("interval") == binding.interval, + "open": _decimal_source(payload, "open") == str(row[1]), + "high": _decimal_source(payload, "high") == str(row[2]), + "low": _decimal_source(payload, "low") == str(row[3]), + "close": _decimal_source(payload, "close") == str(row[4]), + "volume": _decimal_source(payload, "volume") == str(row[5]), + "close_time": int(payload.get("close_time_ns", 0)) + == int(row[6]) * 1_000_000, + "trade_count": int(payload.get("trade_count", -1)) == int(row[8]), + "final": payload.get("lifecycle") == "FINAL", + "source_id": source.get("source_id") == binding.source_id, + "source_role": source.get("source_role") == binding.source_role, + "authoritative": source.get("authoritative") is binding.authoritative, + "policy": quality.get("policy_id") == binding.source_policy_id, + "complete": quality.get("complete") is True, + "gap_closed": quality.get("gap_open") is False, + "execution_forbidden": quality.get("execution_eligible") is False, + } + failed = [name for name, passed in checks.items() if not passed] + if failed: + raise AssertionError(f"V1/V2 canonical parity failed: {failed}") + open_times.append(open_time) + + if open_times != sorted(open_times) or len(open_times) != len(set(open_times)): + raise AssertionError("V2 bars are duplicated or not strictly ordered") + watermark = int(v2.get("watermark_offset", -1)) + if watermark < len(events): + raise AssertionError("V2 watermark is below the returned history length") + return { + "count": len(events), + "first_open_time_ns": open_times[0], + "last_open_time_ns": open_times[-1], + "watermark_offset": watermark, + } + + +def validate_window(first: dict[str, int], second: dict[str, int]) -> None: + delta = second["watermark_offset"] - first["watermark_offset"] + if delta < 0 or delta > 1: + raise AssertionError( + f"continuous bridge watermark advanced outside one 1m-bar window: {delta}" + ) + if second["last_open_time_ns"] < first["last_open_time_ns"]: + raise AssertionError("continuous bridge moved the latest bar backwards") + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--source-bindings", type=Path, required=True) + parser.add_argument("--v1-first", type=Path, required=True) + parser.add_argument("--v2-first", type=Path, required=True) + parser.add_argument("--v1-second", type=Path, required=True) + parser.add_argument("--v2-second", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + binding = CanarySourceCatalog.load(args.source_bindings).bindings[0] + first = validate_sample( + json.loads(args.v1_first.read_text()), + json.loads(args.v2_first.read_text()), + binding, + ) + second = validate_sample( + json.loads(args.v1_second.read_text()), + json.loads(args.v2_second.read_text()), + binding, + ) + validate_window(first, second) + result = { + "schema": "qdl.phase9.0-b.bridge-parity.v1", + "status": "PASS", + "authority": "V1_SHADOW_READ_ONLY", + "source": "REAL_V1_PROVIDER_DATA", + "generated_market_events": 0, + "slice": { + "venue": "BINANCE", + "market": "USDM", + "product_type": "PERPETUAL", + "symbol": binding.instrument.native_symbol, + "feed": binding.feed, + "interval": binding.interval, + }, + "first": first, + "second": second, + "watermark_delta": second["watermark_offset"] - first["watermark_offset"], + "canonical_mismatches": 0, + "duplicate_open_times": 0, + "non_final_bars": 0, + "execution_eligible_events": 0, + } + args.output.write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/phase90b_finalize_evidence.py b/scripts/phase90b_finalize_evidence.py new file mode 100755 index 0000000..625ceef --- /dev/null +++ b/scripts/phase90b_finalize_evidence.py @@ -0,0 +1,175 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path + + +def _load(path: Path): + return json.loads(path.read_text()) + + +def _stats(path: Path) -> list[dict[str, object]]: + rows = [] + for line in path.read_text().splitlines(): + if not line.strip(): + continue + value = json.loads(line) + rows.append({ + key: value.get(key) + for key in ("Name", "CPUPerc", "MemUsage", "MemPerc", "PIDs", "NetIO") + }) + return rows + + +def main() -> None: + parser = argparse.ArgumentParser() + for name in ( + "capacity", "security", "parity", "image-inspect", "bridge-inspect", + "stats", "v1-before", "v1-after", + ): + parser.add_argument(f"--{name}", type=Path, required=True) + parser.add_argument("--openapi-before", required=True) + parser.add_argument("--openapi-after", required=True) + for name in ( + "production-keys-before", "production-keys-after", "containers-after", + "networks-after", "volumes-after", "images-after", + ): + parser.add_argument(f"--{name}", type=int, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--report", type=Path, required=True) + args = parser.parse_args() + + capacity = _load(args.capacity) + security = _load(args.security) + parity = _load(args.parity) + image = _load(args.image_inspect)[0] + bridge = _load(args.bridge_inspect)[0] + v1_before = _load(args.v1_before) + v1_after = _load(args.v1_after) + labels = image.get("Config", {}).get("Labels", {}) or {} + mounts = bridge.get("Mounts", []) + networks = sorted(bridge.get("NetworkSettings", {}).get("Networks", {})) + checks = { + "phase73_capacity": capacity.get("status") == "PASS" + and all(capacity.get("checks", {}).values()), + "phase73_security": security.get("status") == "PASS", + "continuous_bridge_parity": parity.get("status") == "PASS" + and parity.get("canonical_mismatches") == 0, + "provider_authentic": parity.get("source") == "REAL_V1_PROVIDER_DATA" + and parity.get("generated_market_events") == 0, + "candidate_revision_labeled": bool(labels.get("org.opencontainers.image.revision")), + "candidate_non_root": bridge.get("Config", {}).get("User") == "10001:10001", + "candidate_read_only": bridge.get("HostConfig", {}).get("ReadonlyRootfs") is True, + "candidate_no_source_bind": not any( + item.get("Type") == "bind" for item in mounts + ), + "bridge_network_isolated": "bobby_network" in networks + and any(name.endswith("_qdl_beta_internal") for name in networks) + and not any(name.endswith("_qdl_beta_ingress") for name in networks), + "v1_topology_unchanged": v1_before == v1_after, + "v1_openapi_unchanged": args.openapi_before == args.openapi_after, + "production_beta_keys_absent": args.production_keys_before == 0 + and args.production_keys_after == 0, + "cleanup_exact": args.containers_after == 0 + and args.networks_after == 0 and args.volumes_after == 0 + and args.images_after == 0, + } + failed = sorted(name for name, passed in checks.items() if not passed) + if failed: + raise SystemExit(f"Phase 9.0-B evidence failed: {failed}") + + result = { + "schema": "qdl.phase9.0-b.isolated-beta.v1", + "status": "PASS_ISOLATED_NO_AUTHORITY_CUTOVER", + "authority": "V1_SHADOW_READ_ONLY", + "decision_scope": "V2_BETA_READ_ONLY_REVIEW", + "checks": checks, + "candidate": { + "image_id": image.get("Id"), + "source_revision": labels.get("org.opencontainers.image.revision"), + "release": labels.get("org.opencontainers.image.version"), + "runtime_user": bridge.get("Config", {}).get("User"), + "read_only_root": bridge.get("HostConfig", {}).get("ReadonlyRootfs"), + "source_bind_mounts": sum(item.get("Type") == "bind" for item in mounts), + "bridge_networks": networks, + }, + "contract_and_capacity": capacity, + "security": security, + "continuous_bridge": parity, + "resource_snapshot": _stats(args.stats), + "cleanup": { + "containers": args.containers_after, + "networks": args.networks_after, + "volumes": args.volumes_after, + "images": args.images_after, + "production_beta_keys": args.production_keys_after, + }, + "production_v1": { + "topology_unchanged": True, + "openapi_sha256": args.openapi_after, + "restarted_or_recreated": False, + }, + "phase_91": "BLOCKED", + } + args.output.write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + + normal = capacity["normal"] + burst = capacity["burst"] + stream = capacity["stream"] + resources = capacity["resources"] + report = f'''# Phase 9.0-B Isolated V2 Beta Report + +Decision: `PASS_ISOLATED_NO_AUTHORITY_CUTOVER` + +## Scope + +The existing V2 query/stream beta was re-certified from the Phase 9.0-A +correctness baseline. V1 remained source and public authority. The candidate +used an isolated Redis, durable spool, credentials, consumer group, loopback +ports and Compose project. No Rust canary or public-internet exposure occurred. + +## Results + +- Authentic slice: `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`. +- Canonical mismatches: `{parity['canonical_mismatches']}`; generated events: + `{parity['generated_market_events']}`; duplicate open times: + `{parity['duplicate_open_times']}`. +- Continuous bridge watermark delta across the bounded observation window: + `{parity['watermark_delta']}`. +- Active/passive stream replay: `{stream['fast_consumers']}` fast consumers, + `{stream['slow_consumers']}` isolated slow consumer, contiguous= + `{all(item['contiguous'] for item in stream['fast_results'])}`. +- Query normal: `{normal['requests']}` requests, p99.9 + `{normal['latency_ms']['p99_9']}` ms, `{normal['errors']}` errors. +- Query burst: `{burst['requests']}` requests, p99.9 + `{burst['latency_ms']['p99_9']}` ms, `{burst['errors']}` errors. +- Peak candidate RSS: `{resources['max_rss_bytes']}` bytes; peak CPU: + `{resources['max_cpu_percent_of_one_core']}%` of one core. +- Security/adversarial, cursor, failover/fencing, dependency outage/recovery, + rate limit, oversized request and cleanup gates all passed. + +## Safety And Cleanup + +Candidate revision: `{result['candidate']['source_revision']}`. Runtime user was +`10001:10001`, root was read-only, no source bind was mounted and bridge access +was restricted to beta-internal plus the existing V1 internal network. +Production V1 topology/OpenAPI remained unchanged. Candidate containers, +networks, volumes and candidate tags after cleanup: `0/0/0/0`; production beta +keys: `0`. + +## Remaining Boundary + +This result permits review of an isolated read-only V2 beta only. Phase 9.1 +remains blocked on replicated production transport, OTel/alerts, workload +identity, external secrets, signature admission, independent DR, complete +consumer registration and explicit exact-slice authority approval. +''' + args.report.write_text(report, encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/scripts/phase90b_isolated_beta_certification.sh b/scripts/phase90b_isolated_beta_certification.sh new file mode 100755 index 0000000..280f5d4 --- /dev/null +++ b/scripts/phase90b_isolated_beta_certification.sh @@ -0,0 +1,231 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +COMPOSE_FILE="${ROOT_DIR}/docker-compose.phase7-beta.yml" +IMAGE_REF="${QDL_PHASE90B_IMAGE:?set QDL_PHASE90B_IMAGE}" +MATRIX_PROJECT="${QDL_PHASE90B_MATRIX_PROJECT:-qdl_phase90b_matrix}" +BRIDGE_PROJECT="${QDL_PHASE90B_BRIDGE_PROJECT:-qdl_phase90b_bridge}" +V1_CONTAINER="${QDL_V1_CONTAINER:-data_layer_service}" +PROD_REDIS_CONTAINER="${QDL_V1_REDIS_CONTAINER:-redis_marketdata}" +CAPACITY_OUTPUT="${QDL_PHASE90B_CAPACITY_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-capacity.json}" +SECURITY_OUTPUT="${QDL_PHASE90B_SECURITY_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-security-adversarial.json}" +PARITY_OUTPUT="${QDL_PHASE90B_PARITY_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-continuous-bridge.json}" +RESULT_OUTPUT="${QDL_PHASE90B_RESULT_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-isolated-v2-beta.json}" +REPORT_OUTPUT="${QDL_PHASE90B_REPORT_OUTPUT:-${ROOT_DIR}/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md}" +CHECKSUM_OUTPUT="${QDL_PHASE90B_CHECKSUM_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-evidence.sha256}" +QUERY_PORT="${QDL_PHASE90B_QUERY_PORT:-18220}" +STREAM_A_HEALTH_PORT="${QDL_PHASE90B_STREAM_A_HEALTH_PORT:-18221}" +STREAM_B_HEALTH_PORT="${QDL_PHASE90B_STREAM_B_HEALTH_PORT:-18222}" +STREAM_A_GRPC_PORT="${QDL_PHASE90B_STREAM_A_GRPC_PORT:-18230}" +STREAM_B_GRPC_PORT="${QDL_PHASE90B_STREAM_B_GRPC_PORT:-18231}" + +image_id="$(docker image inspect "${IMAGE_REF}" --format '{{.Id}}')" +redis_id="$(docker image inspect redis:7.2-alpine --format '{{.Id}}')" +random_hex() { python3 -c 'import secrets; print(secrets.token_hex(32))'; } +cursor_secret="$(random_hex)" +jwt_secret_1="$(random_hex)" +jwt_secret_2="$(random_hex)" +bridge_secret="$(random_hex)" +export QDL_BETA_IMAGE="${image_id}" +export QDL_BETA_REDIS_IMAGE="${redis_id}" +export QDL_BETA_INIT_IMAGE="${redis_id}" +export QDL_BETA_CURSOR_KEYS_JSON="{\"beta-k1\":\"${cursor_secret}\"}" +export QDL_BETA_CURSOR_ACTIVE_KEY_ID="beta-k1" +export QDL_BETA_JWT_KEYS_JSON="{\"beta-jwt-k1\":\"${jwt_secret_1}\",\"beta-jwt-k2\":\"${jwt_secret_2}\"}" +export QDL_BETA_INTERNAL_INGEST_SECRET="${bridge_secret}" +export QDL_BETA_JWT_ISSUER="https://identity.qdl.phase90b.invalid" +export QDL_BETA_JWT_AUDIENCE="qdl-v2-phase90b" +export QDL_BETA_SOURCE_BINDINGS="/app/config/phase7/canary-sources.yaml" +export QDL_BETA_CONSUMER_MANIFESTS="/app/consumers/beta/phase7-monitoring-binance.yaml:/app/consumers/beta/phase7-paper-alpha-binance.yaml:/app/consumers/beta/phase7-capacity-binance.yaml" +export QDL_BETA_AUTHORITY_REVISION="1" +export QDL_BETA_BRIDGE_RUN_ONCE="false" + +temporary="$(mktemp -d)" +CERT_UID="${QDL_CERT_UID:-$(id -u)}" +CERT_GID="${QDL_CERT_GID:-$(id -g)}" +cleanup() { + docker compose -p "${MATRIX_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta down -v --remove-orphans >/dev/null 2>&1 || true + docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary \ + down -v --remove-orphans >/dev/null 2>&1 || true + docker image rm "${IMAGE_REF}" >/dev/null 2>&1 || true + rm -rf "${temporary}" +} +trap cleanup EXIT +trap 'printf "phase90b certification failed line=%s command=%s\n" "${LINENO}" "${BASH_COMMAND}" >&2' ERR + +wait_http() { + local url="$1" expected="$2" attempts="${3:-60}" code="000" + for ((index=1; index<=attempts; index++)); do + code="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 2 "${url}" || true)" + [[ "${code}" == "${expected}" ]] && return 0 + sleep 1 + done + printf 'timed out url=%s expected=%s actual=%s\n' "${url}" "${expected}" "${code}" >&2 + return 1 +} + +snapshot_v1() { + docker inspect "${V1_CONTAINER}" | python3 "${ROOT_DIR}/scripts/phase73_topology_snapshot.py" +} + +beta_keys_in_v1() { + if ! docker inspect "${PROD_REDIS_CONTAINER}" >/dev/null 2>&1; then printf '0\n'; return; fi + docker exec "${PROD_REDIS_CONTAINER}" redis-cli --scan --pattern 'qdl:beta:v2:*' | wc -l +} + +query_beta() { + local output="$1" token="$2" + curl -sS --max-time 10 -o "${output}" -w '%{http_code}' \ + -H "Authorization: Bearer ${token}" \ + -H 'X-QDL-Consumer-ID: phase7-capacity-binance' \ + -H 'X-QDL-Purpose: INTERNAL_RESEARCH' \ + --get "http://127.0.0.1:${QUERY_PORT}/v2/market-data/a953e16e-7138-5562-b5e8-c337a44d0b65/warmup" \ + --data-urlencode 'feed=BAR' \ + --data-urlencode 'consumer_grade=RESEARCH' \ + --data-urlencode 'source_policy_id=alpha_crypto_primary_v1' \ + --data-urlencode 'interval=1m' \ + --data-urlencode 'limit=30' \ + --data-urlencode 'max_freshness_ms=180000' \ + --data-urlencode 'require_full_coverage=true' \ + --data-urlencode 'require_final_bars=true' \ + --data-urlencode 'stale_policy=BLOCK' \ + --data-urlencode 'gap_policy=BLOCK' \ + --data-urlencode 'recovery=SNAPSHOT_AND_REPLAY' \ + --data-urlencode 'bar_revision_policy=EMIT_REVISIONS' +} + +snapshot_v1 >"${temporary}/v1-before.json" +openapi_before="$(curl -fsS --max-time 10 http://127.0.0.1:8100/openapi.json | sha256sum | cut -d' ' -f1)" +keys_before="$(beta_keys_in_v1)" + +QDL_BETA_PROJECT="${MATRIX_PROJECT}" \ +QDL_BETA_CONFIG_REVISION="phase90b-matrix-1" \ +QDL_BETA_REDIS_PREFIX="qdl:beta:v2:paper:phase90b:matrix" \ +QDL_BETA_CONSUMER_GROUP="qdl-v2-beta-phase90b-matrix" \ +QDL_BETA_LEASE_SHARD_ID="stream-v2-phase90b-matrix" \ +QDL_BETA_QUERY_HOST_PORT=18210 \ +QDL_BETA_STREAM_A_HEALTH_PORT=18211 \ +QDL_BETA_STREAM_B_HEALTH_PORT=18212 \ +QDL_BETA_STREAM_A_GRPC_PORT=18213 \ +QDL_BETA_STREAM_B_GRPC_PORT=18214 \ +QDL_PHASE73_CAPACITY_OUTPUT="${CAPACITY_OUTPUT}" \ +QDL_PHASE73_SECURITY_OUTPUT="${SECURITY_OUTPUT}" \ +"${ROOT_DIR}/scripts/phase73_public_beta_certification.sh" + +export QDL_BETA_CONFIG_REVISION="phase90b-continuous-1" +export QDL_BETA_REDIS_PREFIX="qdl:beta:v2:paper:phase90b:continuous" +export QDL_BETA_CONSUMER_GROUP="qdl-v2-beta-phase90b-continuous" +export QDL_BETA_LEASE_SHARD_ID="stream-v2-phase90b-continuous" +export QDL_BETA_QUERY_HOST_PORT="${QUERY_PORT}" +export QDL_BETA_STREAM_A_HEALTH_PORT="${STREAM_A_HEALTH_PORT}" +export QDL_BETA_STREAM_B_HEALTH_PORT="${STREAM_B_HEALTH_PORT}" +export QDL_BETA_STREAM_A_GRPC_PORT="${STREAM_A_GRPC_PORT}" +export QDL_BETA_STREAM_B_GRPC_PORT="${STREAM_B_GRPC_PORT}" + +docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary config --quiet +docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary up -d +wait_http "http://127.0.0.1:${QUERY_PORT}/health/ready" 200 60 +wait_http "http://127.0.0.1:${STREAM_A_HEALTH_PORT}/health/live" 200 30 +wait_http "http://127.0.0.1:${STREAM_B_HEALTH_PORT}/health/live" 200 30 + +status_a="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 2 "http://127.0.0.1:${STREAM_A_HEALTH_PORT}/health/ready" || true)" +status_b="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 2 "http://127.0.0.1:${STREAM_B_HEALTH_PORT}/health/ready" || true)" +if [[ "${status_a}:${status_b}" == "200:503" ]]; then + active_grpc_port="${STREAM_A_GRPC_PORT}" +elif [[ "${status_a}:${status_b}" == "503:200" ]]; then + active_grpc_port="${STREAM_B_GRPC_PORT}" +else + printf 'expected exactly one active stream gateway, got A=%s B=%s\n' "${status_a}" "${status_b}" >&2 + exit 1 +fi + +bridge_id="$(docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary ps -q qdl_beta_v1_bridge)" +[[ -n "${bridge_id}" ]] +docker inspect "${bridge_id}" >"${temporary}/bridge-inspect.json" +docker image inspect "${image_id}" >"${temporary}/image-inspect.json" + +token="$(docker run --rm --network none \ + -e QDL_BETA_JWT_KEYS_JSON -e QDL_BETA_JWT_ISSUER -e QDL_BETA_JWT_AUDIENCE \ + "${image_id}" python /app/scripts/phase73_token.py)" +code="000" +for ((attempt=1; attempt<=60; attempt++)); do + code="$(query_beta "${temporary}/v2-candidate.json" "${token}" || true)" + if [[ "${code}" == "200" ]]; then + count="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("count",0))' "${temporary}/v2-candidate.json")" + [[ "${count}" -ge 30 ]] && break + fi + sleep 1 +done +[[ "${code}" == "200" && "${count:-0}" -ge 30 ]] + +previous_watermark="" +for ((attempt=1; attempt<=15; attempt++)); do + sleep 2 + code="$(query_beta "${temporary}/v2-stable.json" "${token}" || true)" + [[ "${code}" == "200" ]] || continue + watermark="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["watermark_offset"])' "${temporary}/v2-stable.json")" + if [[ -n "${previous_watermark}" && "${watermark}" == "${previous_watermark}" ]]; then + cp "${temporary}/v2-stable.json" "${temporary}/v2-first.json" + break + fi + previous_watermark="${watermark}" +done +[[ -f "${temporary}/v2-first.json" ]] +curl -fsS --max-time 10 'http://127.0.0.1:8100/v1/crypto/ohlcv/binance/BTCUSDT/1m?limit=120&market=usdm' >"${temporary}/v1-first.json" +sleep 12 +[[ "$(query_beta "${temporary}/v2-second.json" "${token}")" == "200" ]] +curl -fsS --max-time 10 'http://127.0.0.1:8100/v1/crypto/ohlcv/binance/BTCUSDT/1m?limit=120&market=usdm' >"${temporary}/v1-second.json" + +docker run --rm --network none --read-only --cap-drop ALL \ + --security-opt no-new-privileges:true --pids-limit 64 --memory 256m --cpus 0.5 \ + --user "${CERT_UID}:${CERT_GID}" --tmpfs "/tmp:rw,noexec,nosuid,nodev,size=16m,uid=${CERT_UID},gid=${CERT_GID}" \ + -v "${temporary}:/evidence" "${image_id}" \ + python /app/scripts/phase90b_bridge_parity.py \ + --source-bindings /app/config/phase7/canary-sources.yaml \ + --v1-first /evidence/v1-first.json --v2-first /evidence/v2-first.json \ + --v1-second /evidence/v1-second.json --v2-second /evidence/v2-second.json \ + --output /evidence/parity.json +cp "${temporary}/parity.json" "${PARITY_OUTPUT}" + +mapfile -t beta_ids < <(docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" --profile phase7-beta --profile phase7-canary ps -q) +: >"${temporary}/stats.jsonl" +docker stats --no-stream --format '{{json .}}' "${beta_ids[@]}" >>"${temporary}/stats.jsonl" + +docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary down -v --remove-orphans +containers_after="$(docker ps -aq --filter label=com.docker.compose.project="${BRIDGE_PROJECT}" | wc -l)" +networks_after="$(docker network ls -q --filter label=com.docker.compose.project="${BRIDGE_PROJECT}" | wc -l)" +volumes_after="$(docker volume ls -q --filter label=com.docker.compose.project="${BRIDGE_PROJECT}" | wc -l)" +docker image rm "${IMAGE_REF}" >/dev/null +images_after="$(docker image ls -q "${IMAGE_REF}" | wc -l)" +snapshot_v1 >"${temporary}/v1-after.json" +openapi_after="$(curl -fsS --max-time 10 http://127.0.0.1:8100/openapi.json | sha256sum | cut -d' ' -f1)" +keys_after="$(beta_keys_in_v1)" + +python3 "${ROOT_DIR}/scripts/phase90b_finalize_evidence.py" \ + --capacity "${CAPACITY_OUTPUT}" --security "${SECURITY_OUTPUT}" \ + --parity "${PARITY_OUTPUT}" --image-inspect "${temporary}/image-inspect.json" \ + --bridge-inspect "${temporary}/bridge-inspect.json" --stats "${temporary}/stats.jsonl" \ + --v1-before "${temporary}/v1-before.json" --v1-after "${temporary}/v1-after.json" \ + --openapi-before "${openapi_before}" --openapi-after "${openapi_after}" \ + --production-keys-before "${keys_before}" --production-keys-after "${keys_after}" \ + --containers-after "${containers_after}" --networks-after "${networks_after}" \ + --volumes-after "${volumes_after}" --images-after "${images_after}" \ + --output "${RESULT_OUTPUT}" --report "${REPORT_OUTPUT}" +capacity_rel="$(realpath --relative-to="${ROOT_DIR}" "${CAPACITY_OUTPUT}")" +security_rel="$(realpath --relative-to="${ROOT_DIR}" "${SECURITY_OUTPUT}")" +parity_rel="$(realpath --relative-to="${ROOT_DIR}" "${PARITY_OUTPUT}")" +result_rel="$(realpath --relative-to="${ROOT_DIR}" "${RESULT_OUTPUT}")" +report_rel="$(realpath --relative-to="${ROOT_DIR}" "${REPORT_OUTPUT}")" +( + cd "${ROOT_DIR}" + sha256sum "${capacity_rel}" "${security_rel}" "${parity_rel}" \ + "${result_rel}" "${report_rel}" +) >"${CHECKSUM_OUTPUT}" +(cd "${ROOT_DIR}" && sha256sum -c "${CHECKSUM_OUTPUT}") diff --git a/scripts/phase90c_migration_smoke.sh b/scripts/phase90c_migration_smoke.sh new file mode 100755 index 0000000..9bd6927 --- /dev/null +++ b/scripts/phase90c_migration_smoke.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +POSTGRES_IMAGE="${QDL_PHASE90C_POSTGRES_IMAGE:-timescale/timescaledb:latest-pg15}" +CONTAINER="${QDL_PHASE90C_POSTGRES_CONTAINER:-qdl_phase90c_postgres_$$}" +OUTPUT="${QDL_PHASE90C_MIGRATION_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90c-authority-migration.json}" +CANDIDATE_DIGEST="${QDL_PHASE90C_CANDIDATE_DIGEST:-$( + PYTHONPATH="${ROOT_DIR}" python3 -c "from qdl.certification.prerequisites import CandidateSlice; print(CandidateSlice.load('${ROOT_DIR}/config/phase9/candidate-slice.yaml').digest)" +)}" + +cleanup() { + docker rm -f "${CONTAINER}" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +docker run -d --name "${CONTAINER}" --network none \ + --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.0 \ + --tmpfs /var/lib/postgresql/data:rw,nosuid,nodev,size=512m \ + -e POSTGRES_HOST_AUTH_METHOD=trust \ + -v "${ROOT_DIR}/migrations/postgres:/migrations:ro" \ + "${POSTGRES_IMAGE}" >/dev/null + +for _ in $(seq 1 60); do + if docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null 2>&1; then + break + fi + sleep 1 +done +docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null + +apply_migrations() { + local migration + for migration in "${ROOT_DIR}"/migrations/postgres/*.sql; do + docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + -f "/migrations/$(basename "${migration}")" >/dev/null + done +} + +apply_migrations + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +INSERT INTO qdl_authority_slices ( + slice_id, environment, venue, market, product_type, feed, + partition_plan_epoch, partition_id, schema_major, state, + authority_revision, owner_id, lease_epoch, candidate_digest, + artifact_image_digest, sbom_digest, signature_identity, contract_digest, + normalizer_version, adapter_version, config_revision, + instrument_catalog_revision, source_policy_revision, + partition_plan_digest, rollback_manifest_digest +) VALUES ( + 'production/binance/usdm/perpetual/trade/plan-1/btcusdt', + 'production', 'BINANCE', 'USDM', 'PERPETUAL', 'TRADE', + 1, 'rendezvous-sha256-v1:epoch-1:btcusdt', 2, 'RUST_SHADOW', + 1, 'rust-shadow-owner', 1, '${CANDIDATE_DIGEST}', + 'sha256:${CANDIDATE_DIGEST}', repeat('1',64), 'phase90c-test-signer', repeat('2',64), + 'qdl-rust-core/test', 'binance-usdm/test', 'phase90c-test-config', + 'phase90c-test-catalog', 'phase90c-test-source-policy', + repeat('3',64), repeat('4',64) +); + +SELECT (qdl_transition_authority( + '11111111-1111-4111-8111-111111111111', + 'production/binance/usdm/perpetual/trade/plan-1/btcusdt', + 'RUST_SHADOW', 1, 'rust-shadow-owner', 1, 1, + 'VALIDATING', 'rust-shadow-owner', 1, NULL, NULL, NULL, + 'phase90c-test', 'enter isolated validation' +)).state; + +INSERT INTO qdl_production_prerequisite_bundles ( + bundle_id, candidate_digest, policy_revision, decision, evidence, + evidence_sha256, issued_by, issued_at, expires_at +) VALUES + ('00000000-0000-4000-8000-000000000001', '${CANDIDATE_DIGEST}', 1, + 'NO_GO_EXTERNAL', '{}', repeat('5',64), 'phase90c-test', clock_timestamp(), + clock_timestamp() + interval '1 day'), + ('00000000-0000-4000-8000-000000000002', repeat('9',64), 1, + 'GO', '{}', repeat('6',64), 'phase90c-test', clock_timestamp(), + clock_timestamp() + interval '1 day'), + ('00000000-0000-4000-8000-000000000003', '${CANDIDATE_DIGEST}', 1, + 'GO', '{}', repeat('7',64), 'phase90c-test', clock_timestamp(), + clock_timestamp() + interval '1 day'); +SQL + +expect_failure() { + if docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + -c "$1" >/dev/null 2>&1; then + printf 'expected SQL failure but statement succeeded\n' >&2 + exit 1 + fi +} + +expect_failure "SELECT qdl_transition_authority('31111111-1111-4111-8111-111111111111','production/binance/usdm/perpetual/trade/plan-1/btcusdt','RUST_SHADOW',1,'rust-shadow-owner',1,1,'VALIDATING','rust-shadow-owner',1,NULL,NULL,NULL,'phase90c-test','stale CAS');" +expect_failure "SELECT qdl_transition_authority('32222222-2222-4222-8222-222222222222','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000001',clock_timestamp()+interval '1 hour','phase90c-test','blocked bundle');" +expect_failure "SELECT qdl_transition_authority('33333333-3333-4333-8333-333333333333','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000002',clock_timestamp()+interval '1 hour','phase90c-test','wrong candidate');" + +expect_failure "SELECT qdl_transition_authority('35555555-5555-4555-8555-555555555555','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,NULL,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '1 hour','phase90c-test','missing terminal watermark');" +expect_failure "SELECT qdl_transition_authority('36666666-6666-4666-8666-666666666666','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000003',NULL,'phase90c-test','missing hold window');" +expect_failure "SELECT qdl_transition_authority('37777777-7777-4777-8777-777777777777','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '2 days','phase90c-test','hold exceeds evidence expiry');" + +expect_failure "SELECT qdl_transition_authority('38888888-8888-4888-8888-888888888888','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'BLOCKED','rust-shadow-owner',1,NULL,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '1 hour','phase90c-test','bundle on non-authority state');" + +docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + -c "SELECT qdl_transition_authority('22222222-2222-4222-8222-222222222222','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '1 hour','phase90c-test','approved isolated CAS test');" >/dev/null + +expect_failure "SELECT qdl_transition_authority('34444444-4444-4444-8444-444444444444','production/binance/usdm/perpetual/trade/plan-1/btcusdt','RUST_CANARY',3,'rust-shadow-owner',2,1,'BLOCKED','rust-shadow-owner',2,NULL,NULL,NULL,'phase90c-test','stale owner');" +expect_failure "UPDATE qdl_authority_transition_audit SET reason='mutated';" + +apply_migrations + +state="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT state || ':' || authority_revision || ':' || owner_id || ':' || lease_epoch FROM qdl_authority_slices;")" +audit_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_transition_audit;")" +bundle_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_production_prerequisite_bundles;")" +[[ "${state}" == "RUST_CANARY:3:rust-canary-owner:2" ]] +[[ "${audit_count}" == "2" ]] +[[ "${bundle_count}" == "3" ]] + +python3 - "${OUTPUT}" "${state}" "${audit_count}" "${bundle_count}" <<'PY' +import json, pathlib, sys +output, state, audits, bundles = sys.argv[1:] +path = pathlib.Path(output); path.parent.mkdir(parents=True, exist_ok=True) +path.write_text(json.dumps({ + "schema": "qdl.phase9.0-c.authority-migration.v1", + "status": "PASS", + "final_test_state": state, + "audit_records": int(audits), + "prerequisite_bundles": int(bundles), + "stale_cas_rejected": True, + "no_go_bundle_rejected": True, + "candidate_mismatch_rejected": True, + "missing_terminal_watermark_rejected": True, + "missing_hold_window_rejected": True, + "hold_beyond_bundle_expiry_rejected": True, + "bundle_on_non_authority_state_rejected": True, + "stale_owner_rejected": True, + "audit_mutation_rejected": True, + "idempotent_migration": True, + "production_mutations": 0, +}, indent=2, sort_keys=True) + "\n") +PY + +cleanup +trap - EXIT +[[ -z "$(docker ps -aq --filter name=^/${CONTAINER}$)" ]] +printf '{"status":"PASS","state":"%s","audits":%s,"cleanup":true}\n' \ + "${state}" "${audit_count}" diff --git a/scripts/phase90c_prerequisite_certification.py b/scripts/phase90c_prerequisite_certification.py new file mode 100755 index 0000000..b58fcda --- /dev/null +++ b/scripts/phase90c_prerequisite_certification.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import pathlib +import subprocess +import sys +import time +import urllib.request +import uuid + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.certification.prerequisites import ( # noqa: E402 + CandidateSlice, + PrerequisitePolicy, + evaluate_prerequisites, + load_inventory, +) + + +def _v1_snapshot() -> dict[str, object]: + completed = subprocess.run( + ["docker", "inspect", "data_layer_service", "--format", "{{json .}}"], + text=True, capture_output=True, check=False, + ) + if completed.returncode != 0 or not completed.stdout.strip(): + return {"present": False} + payload = json.loads(completed.stdout) + return { + "present": True, + "id": payload["Id"], + "image": payload["Config"]["Image"], + "started_at": payload["State"]["StartedAt"], + "restart_count": payload["RestartCount"], + "networks": sorted(payload["NetworkSettings"]["Networks"]), + "mounts": sorted((item["Source"], item["Destination"], item["RW"]) for item in payload["Mounts"]), + } + + +def _v1_health() -> int: + try: + with urllib.request.urlopen("http://127.0.0.1:8100/v1/health", timeout=5) as response: + return response.status + except Exception: + return 0 + + +def _write_report(path: pathlib.Path, result: dict[str, object]) -> None: + blocked = [item for item in result["gates"] if not item["passed"]] + lines = [ + "# Phase 9.0-C Production Prerequisite Report", + "", + f"Decision: `{result['decision']}`", + "", + "## Candidate", + "", + f"- Slice: `{result['slice_id']}`", + f"- Candidate digest: `{result['candidate_digest']}`", + f"- Authority: `RUST_SHADOW`; V1 unchanged: `{result['v1_unchanged']}`", + "", + "## Gate Summary", + "", + f"- Passed: `{result['passed']}`", + f"- Blocked: `{result['blocked']}`", + "", + ] + if blocked: + lines += ["## Blocking Evidence", ""] + lines += [ + f"- `{item['gate_id']}`: `{item['reason']}` " + f"(observed `{item['observed_scope']}`, required `{item['required_scope']}`)" + for item in blocked + ] + lines += [ + "", + "These are real infrastructure/operator blockers. Same-host fixtures or", + "local rehearsals must not be relabeled to close them.", + ] + path.write_text("\n".join(lines) + "\n") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--policy", default="config/phase9/production-prerequisite-policy.yaml") + parser.add_argument("--candidate", default="config/phase9/candidate-slice.yaml") + parser.add_argument("--inventory", default="config/phase9/evidence-inventory.local.yaml") + parser.add_argument("--output", default="upgrade/evidence/phase90c-production-prerequisites.json") + parser.add_argument("--report", default="upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md") + parser.add_argument("--expect", choices=("GO", "NO_GO_EXTERNAL"), required=True) + args = parser.parse_args() + + policy = PrerequisitePolicy.load(ROOT / args.policy) + candidate = CandidateSlice.load(ROOT / args.candidate) + inventory_path = ROOT / args.inventory + inventory = load_inventory(inventory_path) + before = _v1_snapshot() + health_before = _v1_health() + decision = evaluate_prerequisites( + policy, candidate, inventory, repository_root=ROOT, now_ns=time.time_ns() + ).as_mapping() + after = _v1_snapshot() + health_after = _v1_health() + inventory_digest = hashlib.sha256(inventory_path.read_bytes()).hexdigest() + bundle_seed = f"{candidate.digest}:{policy.revision}:{inventory_digest}" + result = { + **decision, + "bundle_id": str(uuid.uuid5(uuid.NAMESPACE_URL, bundle_seed)), + "slice_id": candidate.payload["slice_id"], + "authority_state": candidate.payload["authority_state"], + "inventory_sha256": inventory_digest, + "issued_at_ns": time.time_ns(), + "v1_health_before": health_before, + "v1_health_after": health_after, + "v1_unchanged": before == after and health_before == health_after == 200, + "v1_before": before, + "v1_after": after, + "production_mutations": 0, + } + if not result["v1_unchanged"]: + raise RuntimeError("V1 topology or health changed during prerequisite evaluation") + output = ROOT / args.output + report = ROOT / args.report + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n") + _write_report(report, result) + print(json.dumps({ + "decision": result["decision"], "passed": result["passed"], + "blocked": result["blocked"], "v1_unchanged": result["v1_unchanged"], + }, sort_keys=True)) + return 0 if result["decision"] == args.expect else 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase91_canary_certification.py b/scripts/phase91_canary_certification.py new file mode 100755 index 0000000..087b036 --- /dev/null +++ b/scripts/phase91_canary_certification.py @@ -0,0 +1,570 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import base64 +import gzip +import hashlib +import json +import os +import pathlib +import tempfile +import time +from collections import Counter + +import yaml + +os.environ.setdefault("QDL_PHASE8_PROJECT", "qdl_phase91_certification") +from phase80_broker_certification import ( # noqa: E402 + BOOTSTRAP, + PROJECT, + ROOT, + add_acls, + cleanup, + compose, + consume, + create_topic, + kafka, + run, + total_end_offset, + v1_health, + v1_topology, + wait_for_cluster, + wait_for_replicas, +) + + +CAPTURE = ROOT / "upgrade/evidence/captures/phase8-real-provider-frames.json.gz" +CAPTURE_EVIDENCE = ROOT / "upgrade/evidence/phase8-real-provider-shadow.json" +CANDIDATE = ROOT / "config/phase9/candidate-slice.yaml" +PREREQUISITE_DECISION = ROOT / "upgrade/evidence/phase90c-production-prerequisites.json" +OUTPUT = ROOT / "upgrade/evidence/phase91-rust-canary-certification.json" +REPORT = ROOT / "upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md" +CHECKSUM = ROOT / "upgrade/evidence/phase91-evidence.sha256" +BUNDLE_PATH = ROOT / "target/phase91-authentic-replay.json" + + +def candidate_digest(payload: dict) -> str: + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest() + + +def consume_compacted_records(env: dict[str, str], topic: str) -> list[str]: + offsets = kafka( + env, + "kafka-get-offsets.sh", + "--bootstrap-server", + BOOTSTRAP, + "--command-config", + "/etc/kafka/secrets/admin.properties", + "--topic", + topic, + "--time", + "-1", + ) + records: list[str] = [] + for line in offsets.stdout.splitlines(): + fields = line.rsplit(":", 2) + if len(fields) != 3 or not fields[1].isdigit() or not fields[2].isdigit(): + continue + partition, end_offset = int(fields[1]), int(fields[2]) + if end_offset <= 0: + continue + result = kafka( + env, + "kafka-console-consumer.sh", + "--bootstrap-server", + BOOTSTRAP, + "--consumer.config", + "/etc/kafka/secrets/consumer.properties", + "--topic", + topic, + "--partition", + str(partition), + "--offset", + "earliest", + "--timeout-ms", + "5000", + check=False, + timeout=20.0, + ) + records.extend( + item.strip() + for item in result.stdout.splitlines() + if item.strip().startswith("{") and item.strip().endswith("}") + ) + if not records: + raise RuntimeError(f"compacted topic has offsets but no readable records: {topic}") + return records + + +def authentic_fixtures(*, repeat: int) -> tuple[dict, dict]: + compressed = CAPTURE.read_bytes() + capture_evidence = json.loads(CAPTURE_EVIDENCE.read_text()) + actual_digest = hashlib.sha256(compressed).hexdigest() + if actual_digest != capture_evidence["capture_bundle_sha256"]: + raise RuntimeError("frozen authentic capture checksum mismatch") + payload = json.loads(gzip.decompress(compressed)) + if ( + payload.get("schema") != "qdl.phase8.authentic-capture-bundle.v1" + or payload.get("provenance") != "REAL_PROVIDER_READ_ONLY" + or payload.get("production_writes") != 0 + ): + raise RuntimeError("authentic capture provenance is invalid") + fixtures = [] + for sequence, item in enumerate( + ( + record + for record in payload["captures"] + if record.get("provider") == "BINANCE_DIRECT" + and record.get("venue") == "BINANCE" + and record.get("market") == "USDM" + and record.get("native_symbol") == "BTCUSDT" + and record.get("test_provenance") is False + ), + start=1, + ): + raw_frame = base64.b64decode(item["raw_frame_base64"], validate=True) + if len(raw_frame) != item["raw_frame_bytes"]: + raise RuntimeError("authentic raw frame length mismatch") + if hashlib.sha256(raw_frame).hexdigest() != item["raw_frame_sha256"]: + raise RuntimeError("authentic raw frame checksum mismatch") + wrapper = json.loads(raw_frame) + if wrapper.get("stream") != item["native_channel"] or not isinstance(wrapper.get("data"), dict): + raise RuntimeError("authentic Binance combined frame is malformed") + received = int(item["received_at_ns"]) + fixtures.append({ + "provider_kind": "binance_usdm_trade", + "context": { + "instrument_uid": "85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c", + "instrument_id": "BINANCE.USDM.PERPETUAL.BTCUSDT", + "instrument_revision": 1, + "venue": "BINANCE", + "market": "USDM", + "product_type": "PERPETUAL", + "native_symbol": "BTCUSDT", + "provider": "BINANCE_DIRECT", + "source_id": "binance-usdm-phase91-canary", + "lease_epoch": 2, + "received_at_ns": received, + "normalized_at_ns": received + 1, + "published_at_ns": received + 2, + "partition_sequence": sequence, + "normalizer_version": "qdl-normalizer/2.0.0-phase91", + "adapter_version": "binance-usdm/2.0.0-shadow", + "config_revision": 1, + "correlation_id": "phase91-authentic-replay", + "source_session_id": item["source_session_id"], + "connection_generation": int(item["connection_generation"]), + "authority_revision": 2, + "partition_plan_epoch": 1, + "raw_capture_id": list(bytes.fromhex(item["capture_id"])), + "raw_frame_sha256": list(bytes.fromhex(item["raw_frame_sha256"])), + }, + "raw": wrapper["data"], + }) + if len(fixtures) < 32: + raise RuntimeError(f"too few authentic Binance frames: {len(fixtures)}") + BUNDLE_PATH.parent.mkdir(parents=True, exist_ok=True) + BUNDLE_PATH.write_text(json.dumps( + {"fixtures": fixtures, "repeat": repeat}, + sort_keys=True, + separators=(",", ":"), + )) + return { + "path": str(BUNDLE_PATH), + "fixtures": len(fixtures), + "repeat": repeat, + "events": len(fixtures) * repeat, + "capture_sha256": actual_digest, + "capture_provenance": payload["provenance"], + "raw_checks_passed": len(fixtures), + }, {"fixtures": fixtures, "repeat": repeat} + + +def parity(image: str, repeat: int) -> dict: + capture, _ = authentic_fixtures(repeat=repeat) + mount = f"type=bind,source={ROOT},target=/app,readonly" + python_result = run([ + "docker", "run", "--rm", "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", mount, + "--entrypoint", "python", + "data-layer:v0.1.0", + "/app/scripts/phase91_python_parity.py", + "/app/target/phase91-authentic-replay.json", + ], timeout=240) + python_metrics = json.loads(python_result.stdout.strip().splitlines()[-1]) + rust_runs = [] + for _ in range(3): + result = run([ + "docker", "run", "--rm", "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", mount, + "--entrypoint", "/usr/local/bin/qdl-parity-replay", + image, + "/app/target/phase91-authentic-replay.json", + ], timeout=240) + rust_runs.append(json.loads(result.stdout.strip().splitlines()[-1])) + expected = python_metrics["aggregate_sha256"] + mismatch_runs = sum( + item.get("aggregate_sha256") != expected + or item.get("record_sha256") != python_metrics["record_sha256"] + for item in rust_runs + ) + if mismatch_runs: + raise RuntimeError(f"authentic Python/Rust parity diverged in {mismatch_runs} runs") + return { + "status": "PASS", + "capture": capture, + "python": python_metrics, + "rust_clean_process_runs": len(rust_runs), + "rust_events_per_second_min": min(item["events_per_second"] for item in rust_runs), + "aggregate_sha256": expected, + "semantic_mismatches": 0, + "process_restart_mismatches": 0, + } + + +def rust_command( + *, image: str, cert_dir: str, entrypoint: str, env_values: dict[str, str] +) -> list[str]: + command = [ + "docker", "run", "--rm", + "--network", f"{PROJECT}_phase8_shadow", + "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", f"type=bind,source={cert_dir},target=/certs,readonly", + "--entrypoint", entrypoint, + ] + for key, value in env_values.items(): + command.extend(("--env", f"{key}={value}")) + command.append(image) + return command + + +def broker_rehearsal(image: str, image_digest: str, candidate: dict, decision: dict) -> dict: + topics = { + "authority": "qdl.phase8.phase91.control.authority.v2", + "audit": "qdl.phase8.phase91.audit.authority.v2", + "shadow_raw": "qdl.phase8.phase91.shadow.raw.v2", + "shadow": "qdl.phase8.phase91.shadow.canonical.v2", + "canary": "qdl.phase8.phase91.canary.canonical.v2", + "public": "qdl.phase8.phase91.public.must-remain-empty", + "legacy": "qdl.phase8.phase91.legacy.must-remain-empty", + "transport": "qdl.phase8.phase91.transport", + } + v1_before = v1_topology() + health_before = v1_health() + cleanup_result: dict[str, object] = {} + result: dict[str, object] = {} + with tempfile.TemporaryDirectory(prefix="qdl-phase91-certs-") as cert_dir: + env = os.environ.copy() + env["QDL_PHASE8_CERT_DIR"] = cert_dir + run([str(ROOT / "scripts/phase80_generate_tls.sh"), cert_dir], env=env, timeout=120) + try: + compose(env, "down", "--volumes", "--remove-orphans", check=False, timeout=120) + compose(env, "up", "-d", timeout=180) + wait_for_cluster(env) + # KRaft may answer one metadata request before every controller/broker + # has remained stable. Require a short stable window before topic I/O. + time.sleep(10.0) + wait_for_cluster(env) + running = set( + compose(env, "ps", "--status", "running", "--services").stdout.splitlines() + ) + if not {"kafka1", "kafka2", "kafka3"}.issubset(running): + raise RuntimeError(f"Kafka brokers did not survive stability window: {sorted(running)}") + for name, topic in topics.items(): + create_topic( + env, + topic, + partitions=1 if name in {"authority", "audit"} else 3, + cleanup_policy="compact" if name == "authority" else "delete", + ) + add_acls(env) + wait_for_replicas(env) + before = {name: total_end_offset(env, topic) for name, topic in topics.items()} + nonce = hashlib.sha256(str(time.time_ns()).encode()).hexdigest()[:16] + runtime_result = run(rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-phase91-canary-rehearsal", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": BOOTSTRAP, + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_AUTHORITY_TOPIC": topics["authority"], + "QDL_AUDIT_TOPIC": topics["audit"], + "QDL_SHADOW_RAW_TOPIC": topics["shadow_raw"], + "QDL_SHADOW_CANONICAL_TOPIC": topics["shadow"], + "QDL_CANARY_CANONICAL_TOPIC": topics["canary"], + "QDL_PUBLIC_TOPIC": topics["public"], + "QDL_LEGACY_TOPIC": topics["legacy"], + "QDL_AUTHORITY_NONCE": nonce, + "QDL_CANDIDATE_DIGEST": candidate_digest(candidate), + "QDL_PREREQUISITE_BUNDLE_ID": decision["bundle_id"], + "QDL_SLICE_ID": candidate["slice_id"], + "QDL_SHADOW_OWNER_ID": candidate["owner_id"], + "QDL_CANARY_OWNER_ID": "rust-canary-binance-usdm-shard-0", + }, + ), timeout=180) + runtime = json.loads(runtime_result.stdout.strip().splitlines()[-1]) + after = {name: total_end_offset(env, topic) for name, topic in topics.items()} + deltas = {name: after[name] - before[name] for name in topics} + expected_deltas = { + "authority": 4, + "audit": 4, + "shadow_raw": 0, + "shadow": 2, + "canary": 64, + "public": 0, + "legacy": 0, + "transport": 0, + } + if runtime.get("status") != "PASS" or deltas != expected_deltas: + raise RuntimeError(f"Phase 9.1 runtime/offset mismatch runtime={runtime} deltas={deltas}") + + compose(env, "stop", "kafka3") + one_loss_before = total_end_offset(env, topics["transport"]) + one_loss = run(rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092,kafka2:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-one-loss", + }, + ), timeout=90) + one_loss_after = total_end_offset(env, topics["transport"]) + one_replica_loss_acked = one_loss.returncode == 0 and one_loss_after == one_loss_before + 1 + min_isr_before = one_loss_after + compose(env, "stop", "kafka2") + min_isr = run(rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-min-isr", + }, + ), check=False, timeout=35) + min_isr_client_failed = min_isr.returncode != 0 + compose(env, "start", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + min_isr_after = total_end_offset(env, topics["transport"]) + min_isr_failed_closed = ( + min_isr_client_failed and min_isr_after == min_isr_before + ) + if not one_replica_loss_acked or not min_isr_failed_closed: + raise RuntimeError( + f"broker durability gate failed one_loss={one_replica_loss_acked} " + f"min_isr={min_isr_failed_closed} " + f"min_isr_returncode={min_isr.returncode} " + f"offset_before={min_isr_before} offset_after={min_isr_after} " + f"stderr={min_isr.stderr[-800:]}" + ) + compose(env, "restart", "kafka1", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + + audit = [json.loads(item) for item in consume( + env, topics["audit"], 4, f"phase8-phase91-audit-{nonce}" + )] + authority = [ + json.loads(item) + for item in consume_compacted_records(env, topics["authority"]) + ] + slow_started = time.monotonic() + time.sleep(1.0) + canary = [json.loads(item) for item in consume( + env, topics["canary"], 64, f"phase8-phase91-slow-{nonce}", timeout=60 + )] + catchup_seconds = time.monotonic() - slow_started + audit_states = [item["state"] for item in audit] + latest = max(authority, key=lambda item: item["authority_revision"]) + canary_watermarks = [item["source_watermark"] for item in canary] + if audit_states != ["RUST_SHADOW", "RUST_CANARY", "BLOCKED", "RUST_SHADOW"]: + raise RuntimeError(f"authority audit order diverged: {audit_states}") + if latest["state"] != "RUST_SHADOW" or latest["authority_revision"] != 4: + raise RuntimeError(f"latest compacted authority diverged: {latest}") + if canary_watermarks != list(range(102, 166)): + raise RuntimeError("slow consumer catch-up changed canary order or coverage") + result = { + "status": "PASS", + "schema": "qdl.phase91.broker-rehearsal.v1", + "mode": "ISOLATED_REHEARSAL", + "production_authorized": False, + "image_digest": image_digest, + "runtime": runtime, + "offset_deltas": deltas, + "one_replica_loss_acked": one_replica_loss_acked, + "min_isr_failed_closed": min_isr_failed_closed, + "authority_audit_states_after_restart": audit_states, + "latest_authority_after_restart": { + "state": latest["state"], + "revision": latest["authority_revision"], + "owner_id": latest["owner_id"], + "lease_epoch": latest["lease_epoch"], + }, + "slow_consumer": { + "delayed_seconds": 1.0, + "catchup_seconds": catchup_seconds, + "records": len(canary), + "ordered_gap_free": True, + }, + "public_writes": 0, + "legacy_writes": 0, + "final_authority": "RUST_SHADOW", + } + except Exception as error: + status = compose(env, "ps", "--all", check=False, timeout=30) + logs = compose(env, "logs", "--no-color", "--tail", "240", check=False, timeout=60) + diagnostic = (status.stdout + status.stderr + logs.stdout + logs.stderr)[-12000:] + raise RuntimeError( + f"Phase 9.1 broker rehearsal failed before cleanup: {error}\n" + f"isolated broker diagnostics:\n{diagnostic}" + ) from error + finally: + cleanup_result = cleanup(env) + v1_after = v1_topology() + health_after = v1_health() + cleanup_result.update({ + "v1_health_before": health_before, + "v1_health_after": health_after, + "v1_topology_unchanged": v1_before == v1_after, + }) + result["cleanup"] = cleanup_result + if ( + health_before != 200 + or health_after != 200 + or not cleanup_result["v1_topology_unchanged"] + or any(cleanup_result[key] for key in ("containers_after", "networks_after", "volumes_after")) + ): + raise RuntimeError(f"Phase 9.1 cleanup/V1 invariant failed: {cleanup_result}") + return result + + +def render_report(evidence: dict) -> str: + parity = evidence["parity"] + broker = evidence["broker"] + capture = parity["capture"] + cleanup_result = broker["cleanup"] + return f"""# Phase 9.1 Rust Canary Certification Report + +## Decision + +- Status: `{evidence['status']}` +- Production authorized: `{str(evidence['production_authorized']).lower()}` +- Production mutations: `{evidence['production_mutations']}` +- Prerequisite decision: `{evidence['prerequisite_decision']}` +- Slice: `{evidence['slice_id']}` +- Candidate digest: `{evidence['candidate_digest']}` + +## Authentic Parity + +- Provenance: `{capture['capture_provenance']}` +- Frozen fixtures: `{capture['fixtures']}` +- Repetition: `{capture['repeat']}` +- Canonical events: `{capture['events']}` +- Semantic mismatches: `{parity['semantic_mismatches']}` +- Clean Rust process runs: `{parity['rust_clean_process_runs']}` +- Aggregate SHA-256: `{parity['aggregate_sha256']}` +- Python throughput: `{parity['python']['events_per_second']:.3f}` events/s +- Minimum Rust throughput: `{parity['rust_events_per_second_min']:.3f}` events/s + +## Authority And Broker Recovery + +- Transition audit: `{', '.join(broker['authority_audit_states_after_restart'])}` +- Final authority: `{broker['final_authority']}` +- One-replica-loss ACK: `{str(broker['one_replica_loss_acked']).lower()}` +- Below-min-ISR fail closed: `{str(broker['min_isr_failed_closed']).lower()}` +- Slow-consumer records: `{broker['slow_consumer']['records']}` +- Slow-consumer ordered and gap-free: `{str(broker['slow_consumer']['ordered_gap_free']).lower()}` +- Public writes: `{broker['public_writes']}` +- Legacy writes: `{broker['legacy_writes']}` + +## Isolation And Cleanup + +- V1 health before/after: `{cleanup_result['v1_health_before']}/{cleanup_result['v1_health_after']}` +- V1 topology unchanged: `{str(cleanup_result['v1_topology_unchanged']).lower()}` +- Containers/networks/volumes remaining: `{cleanup_result['containers_after']}/{cleanup_result['networks_after']}/{cleanup_result['volumes_after']}` + +## Remaining External Gates + +- Production Phase 9.0-C infrastructure and operator gates remain `NO_GO_EXTERNAL`. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- Python V1 remains the sole authoritative public and legacy writer. This report does not authorize a production `RUST_CANARY` transition. +""" + + +def write_evidence(evidence: dict) -> None: + OUTPUT.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + REPORT.write_text(render_report(evidence)) + lines = [] + for path in (OUTPUT, REPORT): + digest = hashlib.sha256(path.read_bytes()).hexdigest() + lines.append(f"{digest} {path.relative_to(ROOT)}") + CHECKSUM.write_text("\n".join(lines) + "\n") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--rust-image", required=True) + parser.add_argument("--repeat", type=int, default=200) + args = parser.parse_args() + if args.repeat < 10: + raise ValueError("Phase 9.1 parity repeat must be at least 10") + candidate = yaml.safe_load(CANDIDATE.read_text()) + decision = json.loads(PREREQUISITE_DECISION.read_text()) + digest = candidate_digest(candidate) + if decision.get("candidate_digest") != digest: + raise RuntimeError("Phase 9.0-C decision is not bound to the selected candidate") + if decision.get("decision") != "NO_GO_EXTERNAL": + raise RuntimeError("this isolated harness expects current production NO_GO_EXTERNAL") + inspect = run([ + "docker", "image", "inspect", args.rust_image, "--format", "{{.Id}}" + ]) + image_digest = inspect.stdout.strip() + parity_result: dict = {} + broker_result: dict = {} + try: + parity_result = parity(args.rust_image, args.repeat) + broker_result = broker_rehearsal( + args.rust_image, image_digest, candidate, decision + ) + finally: + BUNDLE_PATH.unlink(missing_ok=True) + evidence = { + "schema": "qdl.phase91.rust-canary-certification.v1", + "status": "COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED", + "issued_at_ns": time.time_ns(), + "slice_id": candidate["slice_id"], + "candidate_digest": digest, + "prerequisite_bundle_id": decision["bundle_id"], + "prerequisite_decision": decision["decision"], + "production_authorized": False, + "production_mutations": 0, + "python_v1_public_authority_unchanged": True, + "parity": parity_result, + "broker": broker_result, + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "same-host replicated broker rehearsal is not an independent production failure domain", + ], + } + write_evidence(evidence) + print(json.dumps({ + "status": evidence["status"], + "authentic_events": parity_result["capture"]["events"], + "semantic_mismatches": parity_result["semantic_mismatches"], + "public_writes": broker_result["public_writes"], + "legacy_writes": broker_result["legacy_writes"], + "cleanup": broker_result["cleanup"], + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase91_python_parity.py b/scripts/phase91_python_parity.py new file mode 100755 index 0000000..dd6c8ee --- /dev/null +++ b/scripts/phase91_python_parity.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import hashlib +import json +import pathlib +import statistics +import sys +import time +from dataclasses import fields +from typing import Any + +from qdl.canonical.trade import TradeContext, canonicalize_binance_usdm_trade + + +def percentile(values: list[float], ratio: float) -> float: + ordered = sorted(values) + if not ordered: + return 0.0 + return ordered[min(len(ordered) - 1, round((len(ordered) - 1) * ratio))] + + +def context(value: dict[str, Any]) -> TradeContext: + known = {item.name for item in fields(TradeContext)} + unknown = set(value) - known + if unknown: + raise ValueError(f"unknown TradeContext fields: {sorted(unknown)}") + payload = dict(value) + payload["raw_capture_id"] = bytes(payload.get("raw_capture_id", [])) + payload["raw_frame_sha256"] = bytes(payload.get("raw_frame_sha256", [])) + return TradeContext(**payload) + + +def main() -> int: + if len(sys.argv) != 2: + raise SystemExit("usage: phase91_python_parity.py BUNDLE") + payload = json.loads(pathlib.Path(sys.argv[1]).read_text()) + if set(payload) != {"fixtures", "repeat"}: + raise ValueError("Phase 9.1 replay bundle fields are invalid") + fixtures = payload["fixtures"] + repeat = int(payload["repeat"]) + if not fixtures or repeat <= 0: + raise ValueError("Phase 9.1 replay bundle must be non-empty") + aggregate = hashlib.sha256() + first_record_hashes: list[str] = [] + latencies_ms: list[float] = [] + started = time.perf_counter() + for iteration in range(repeat): + for fixture in fixtures: + if fixture.get("provider_kind") != "binance_usdm_trade": + raise ValueError("Phase 9.1 candidate accepts Binance USD-M TRADE only") + event_started = time.perf_counter_ns() + event = canonicalize_binance_usdm_trade( + fixture["raw"], context(fixture["context"]) + ) + encoded = event.SerializeToString(deterministic=True) + latencies_ms.append((time.perf_counter_ns() - event_started) / 1_000_000) + aggregate.update(len(encoded).to_bytes(8, "big")) + aggregate.update(encoded) + if iteration == 0: + first_record_hashes.append(hashlib.sha256(encoded).hexdigest()) + elapsed = time.perf_counter() - started + print(json.dumps({ + "schema": "qdl.phase91.python-parity.v1", + "status": "PASS", + "events": len(fixtures) * repeat, + "fixture_count": len(fixtures), + "repeat": repeat, + "aggregate_sha256": aggregate.hexdigest(), + "record_sha256": first_record_hashes, + "events_per_second": len(fixtures) * repeat / max(elapsed, 1e-9), + "latency_ms": { + "p50": percentile(latencies_ms, 0.50), + "p95": percentile(latencies_ms, 0.95), + "p99": percentile(latencies_ms, 0.99), + "max": max(latencies_ms), + "mean": statistics.fmean(latencies_ms), + }, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase92_migration_smoke.sh b/scripts/phase92_migration_smoke.sh new file mode 100755 index 0000000..302cb7d --- /dev/null +++ b/scripts/phase92_migration_smoke.sh @@ -0,0 +1,206 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +POSTGRES_IMAGE="${QDL_PHASE92_POSTGRES_IMAGE:-timescale/timescaledb:latest-pg15}" +CONTAINER="${QDL_PHASE92_POSTGRES_CONTAINER:-qdl_phase92_postgres_$$}" +OUTPUT="${QDL_PHASE92_MIGRATION_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase92-authority-migration.json}" +CANDIDATE_DIGEST="${QDL_PHASE92_CANDIDATE_DIGEST:-$( + PYTHONPATH="${ROOT_DIR}" python3 -c "from qdl.certification.prerequisites import CandidateSlice; print(CandidateSlice.load('${ROOT_DIR}/config/phase9/candidate-slice.yaml').digest)" +)}" +SLICE="production/binance/usdm/perpetual/trade/plan-1/btcusdt" +BUNDLE="00000000-0000-4000-8000-000000000092" + +cleanup() { + docker rm -f "${CONTAINER}" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +docker run -d --name "${CONTAINER}" --network none --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.0 --tmpfs /var/lib/postgresql/data:rw,nosuid,nodev,size=512m -e POSTGRES_HOST_AUTH_METHOD=trust -v "${ROOT_DIR}/migrations/postgres:/migrations:ro" "${POSTGRES_IMAGE}" >/dev/null + +for _ in $(seq 1 60); do + docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null 2>&1 && break + sleep 1 +done +docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null + +apply_migrations() { + local migration + for migration in "${ROOT_DIR}"/migrations/postgres/*.sql; do + docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres -f "/migrations/$(basename "${migration}")" >/dev/null + done +} +apply_migrations + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +INSERT INTO qdl_production_prerequisite_bundles ( + bundle_id, candidate_digest, policy_revision, decision, evidence, + evidence_sha256, issued_by, issued_at, expires_at +) VALUES ( + '${BUNDLE}', '${CANDIDATE_DIGEST}', 1, 'GO', '{}', repeat('5',64), + 'phase92-test', clock_timestamp(), clock_timestamp() + interval '1 day' +); +INSERT INTO qdl_authority_slices ( + slice_id, environment, venue, market, product_type, feed, + partition_plan_epoch, partition_id, schema_major, state, + authority_revision, owner_id, lease_epoch, terminal_watermark, + candidate_digest, artifact_image_digest, sbom_digest, signature_identity, + contract_digest, normalizer_version, adapter_version, config_revision, + instrument_catalog_revision, source_policy_revision, partition_plan_digest, + rollback_manifest_digest, prerequisite_bundle_id, approved_by, approved_at, + hold_until +) VALUES ( + '${SLICE}', 'production', 'BINANCE', 'USDM', 'PERPETUAL', 'TRADE', + 1, 'rendezvous-sha256-v1:epoch-1:btcusdt', 2, 'RUST_CANARY', + 3, 'python-primary', 1, 100, + '${CANDIDATE_DIGEST}', 'sha256:${CANDIDATE_DIGEST}', repeat('1',64), + 'phase92-test-signer', repeat('2',64), 'qdl-rust-core/test', + 'binance-usdm/test', 'phase92-test-config', 'phase92-test-catalog', + 'phase92-test-source-policy', repeat('3',64), repeat('4',64), + '${BUNDLE}', 'phase92-test', clock_timestamp(), + clock_timestamp() + interval '2 hours' +); +SQL + +expect_failure() { + if docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres -c "$1" >/dev/null 2>&1; then + printf 'expected SQL failure but statement succeeded\n' >&2 + exit 1 + fi +} + +expect_failure "SELECT qdl_transition_authority( + '90000000-0000-4000-8000-000000000001','${SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'${BUNDLE}',clock_timestamp()+interval '1 hour', + 'phase92-test','direct primary bypass');" + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +INSERT INTO qdl_terminal_owner_checkpoints ( + checkpoint_id, slice_id, owner_id, authority_revision, lease_epoch, + partition_plan_epoch, source_session_id, connection_generation, + terminal_watermark, terminal_event_id, terminal_payload_sha256, + candidate_digest, committed_at +) VALUES ( + '91000000-0000-4000-8000-000000000001','${SLICE}','python-primary', + 3,1,1,'python-session-1',1,100,'event-100',repeat('6',64), + '${CANDIDATE_DIGEST}',clock_timestamp() +); +INSERT INTO qdl_authority_handoffs ( + handoff_id, checkpoint_id, direction, slice_id, old_owner_id, new_owner_id, + expected_state, new_state, expected_authority_revision, + new_authority_revision, expected_lease_epoch, new_lease_epoch, + partition_plan_epoch, terminal_watermark, first_new_watermark, + overlap_start_watermark, overlap_end_watermark, old_event_count, + new_event_count, semantic_mismatches, open_gaps, candidate_digest, + prerequisite_bundle_id, handoff_sha256, approved_by, approved_at, expires_at +) VALUES ( + '92000000-0000-4000-8000-000000000001', + '91000000-0000-4000-8000-000000000001','PYTHON_TO_RUST','${SLICE}', + 'python-primary','rust-primary','RUST_CANARY','RUST_PRIMARY',3,4,1,2,1, + 100,101,90,100,11,11,0,0,'${CANDIDATE_DIGEST}','${BUNDLE}', + repeat('7',64),'phase92-test',clock_timestamp(), + clock_timestamp()+interval '2 hours' +); +SELECT (qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000001', + '93000000-0000-4000-8000-000000000001','${SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'${BUNDLE}',clock_timestamp()+interval '1 hour', + 'phase92-test','accepted Python to Rust handoff' +)).state; +SQL + +expect_failure "SELECT qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000001', + '93000000-0000-4000-8000-000000000009','${SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'${BUNDLE}',clock_timestamp()+interval '1 hour', + 'phase92-test','stale CAS replay');" +expect_failure "UPDATE qdl_authority_handoffs SET approved_by='mutated';" +expect_failure "DELETE FROM qdl_terminal_owner_checkpoints;" + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +SELECT qdl_transition_authority( + '93000000-0000-4000-8000-000000000002','${SLICE}', + 'RUST_PRIMARY',4,'rust-primary',2,1,'BLOCKED','rust-primary',2, + 120,NULL,NULL,'phase92-test','fence Rust before rollback'); +SELECT qdl_transition_authority( + '93000000-0000-4000-8000-000000000003','${SLICE}', + 'BLOCKED',5,'rust-primary',2,1,'ROLLBACK_PENDING','rust-primary',2, + 120,NULL,NULL,'phase92-test','prepare Python rollback'); +INSERT INTO qdl_terminal_owner_checkpoints ( + checkpoint_id, slice_id, owner_id, authority_revision, lease_epoch, + partition_plan_epoch, source_session_id, connection_generation, + terminal_watermark, terminal_event_id, terminal_payload_sha256, + candidate_digest, committed_at +) VALUES ( + '91000000-0000-4000-8000-000000000002','${SLICE}','rust-primary', + 6,2,1,'rust-session-1',1,120,'event-120',repeat('8',64), + '${CANDIDATE_DIGEST}',clock_timestamp() +); +INSERT INTO qdl_authority_handoffs ( + handoff_id, checkpoint_id, direction, slice_id, old_owner_id, new_owner_id, + expected_state, new_state, expected_authority_revision, + new_authority_revision, expected_lease_epoch, new_lease_epoch, + partition_plan_epoch, terminal_watermark, first_new_watermark, + overlap_start_watermark, overlap_end_watermark, old_event_count, + new_event_count, semantic_mismatches, open_gaps, candidate_digest, + prerequisite_bundle_id, handoff_sha256, approved_by, approved_at, expires_at +) VALUES ( + '92000000-0000-4000-8000-000000000002', + '91000000-0000-4000-8000-000000000002','RUST_TO_PYTHON','${SLICE}', + 'rust-primary','python-rollback','ROLLBACK_PENDING','PYTHON_PRIMARY',6,7,2,3,1, + 120,121,110,120,11,11,0,0,'${CANDIDATE_DIGEST}','${BUNDLE}', + repeat('9',64),'phase92-test',clock_timestamp(), + clock_timestamp()+interval '2 hours' +); +SELECT (qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000002', + '93000000-0000-4000-8000-000000000004','${SLICE}', + 'ROLLBACK_PENDING',6,'rust-primary',2,1,'PYTHON_PRIMARY','python-rollback',3, + 120,NULL,NULL,'phase92-test','accepted Rust to Python rollback' +)).state; +SQL + +apply_migrations + +state="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT state || ':' || authority_revision || ':' || owner_id || ':' || lease_epoch || ':' || terminal_watermark FROM qdl_authority_slices;")" +audit_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_transition_audit;")" +checkpoint_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_terminal_owner_checkpoints;")" +handoff_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_handoffs;")" +[[ "${state}" == "PYTHON_PRIMARY:7:python-rollback:3:120" ]] +[[ "${audit_count}" == "4" ]] +[[ "${checkpoint_count}" == "2" ]] +[[ "${handoff_count}" == "2" ]] + +python3 - "${OUTPUT}" "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" <<'PY' +import json +import pathlib +import sys + +output, state, audits, checkpoints, handoffs = sys.argv[1:] +path = pathlib.Path(output) +path.parent.mkdir(parents=True, exist_ok=True) +path.write_text(json.dumps({ + "schema": "qdl.phase92.authority-migration.v1", + "status": "PASS", + "final_test_state": state, + "audit_records": int(audits), + "terminal_checkpoints": int(checkpoints), + "accepted_handoffs": int(handoffs), + "direct_primary_bypass_rejected": True, + "stale_cas_rejected": True, + "handoff_mutation_rejected": True, + "checkpoint_delete_rejected": True, + "python_to_rust_handoff_passed": True, + "rust_to_python_rollback_passed": True, + "idempotent_migration": True, + "production_mutations": 0, +}, indent=2, sort_keys=True) + "\n") +PY + +cleanup +trap - EXIT +[[ -z "$(docker ps -aq --filter name=^/${CONTAINER}$)" ]] +printf '{"status":"PASS","state":"%s","audits":%s,"checkpoints":%s,"handoffs":%s,"cleanup":true}\n' "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" diff --git a/scripts/phase92_primary_certification.py b/scripts/phase92_primary_certification.py new file mode 100755 index 0000000..38e1115 --- /dev/null +++ b/scripts/phase92_primary_certification.py @@ -0,0 +1,514 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import pathlib +import tempfile +import time + +import yaml + +os.environ.setdefault("QDL_PHASE8_PROJECT", "qdl_phase92_certification") + +from phase80_broker_certification import ( # noqa: E402 + BOOTSTRAP, + PROJECT, + ROOT, + add_acls, + cleanup, + compose, + consume, + create_topic, + run, + total_end_offset, + v1_health, + v1_topology, + wait_for_cluster, + wait_for_replicas, +) +from phase91_canary_certification import ( # noqa: E402 + BUNDLE_PATH, + candidate_digest, + consume_compacted_records, + parity, + rust_command, +) + + +CANDIDATE = ROOT / "config/phase9/candidate-slice.yaml" +PREREQUISITE_DECISION = ROOT / "upgrade/evidence/phase90c-production-prerequisites.json" +MIGRATION_EVIDENCE = ROOT / "upgrade/evidence/phase92-authority-migration.json" +OUTPUT = ROOT / "upgrade/evidence/phase92-bounded-primary-certification.json" +REPORT = ROOT / "upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md" +CHECKSUM = ROOT / "upgrade/evidence/phase92-evidence.sha256" + + +def broker_rehearsal( + image: str, image_digest: str, candidate: dict, decision: dict +) -> dict: + topics = { + "authority": "qdl.phase8.phase92.control.authority.v3", + "audit": "qdl.phase8.phase92.audit.authority.v3", + "checkpoint": "qdl.phase8.phase92.control.checkpoint.v1", + "handoff": "qdl.phase8.phase92.control.handoff.v1", + "primary": "qdl.phase8.phase92.primary.canonical.v1", + "public": "qdl.phase8.phase92.isolated.public.v2", + "legacy": "qdl.phase8.phase92.isolated.legacy.v1", + "production_public": "qdl.phase8.phase92.production-public.must-remain-empty", + "production_legacy": "qdl.phase8.phase92.production-legacy.must-remain-empty", + "transport": "qdl.phase8.phase92.transport", + } + v1_before = v1_topology() + health_before = v1_health() + cleanup_result: dict[str, object] = {} + result: dict[str, object] = {} + with tempfile.TemporaryDirectory(prefix="qdl-phase92-certs-") as cert_dir: + env = os.environ.copy() + env["QDL_PHASE8_CERT_DIR"] = cert_dir + run([str(ROOT / "scripts/phase80_generate_tls.sh"), cert_dir], env=env, timeout=120) + try: + compose(env, "down", "--volumes", "--remove-orphans", check=False, timeout=120) + compose(env, "up", "-d", timeout=180) + wait_for_cluster(env) + time.sleep(10.0) + wait_for_cluster(env) + running = set( + compose(env, "ps", "--status", "running", "--services").stdout.splitlines() + ) + if not {"kafka1", "kafka2", "kafka3"}.issubset(running): + raise RuntimeError( + f"Kafka brokers did not survive stability window: {sorted(running)}" + ) + for name, topic in topics.items(): + create_topic( + env, + topic, + partitions=1 if name != "transport" else 3, + cleanup_policy="compact" if name == "authority" else "delete", + ) + add_acls(env) + wait_for_replicas(env) + before = {name: total_end_offset(env, topic) for name, topic in topics.items()} + nonce = hashlib.sha256(str(time.time_ns()).encode()).hexdigest()[:16] + runtime_env = { + "QDL_KAFKA_BOOTSTRAP_SERVERS": BOOTSTRAP, + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_AUTHORITY_TOPIC": topics["authority"], + "QDL_AUDIT_TOPIC": topics["audit"], + "QDL_CHECKPOINT_TOPIC": topics["checkpoint"], + "QDL_HANDOFF_TOPIC": topics["handoff"], + "QDL_PRIMARY_CANONICAL_TOPIC": topics["primary"], + "QDL_ISOLATED_PUBLIC_TOPIC": topics["public"], + "QDL_ISOLATED_LEGACY_TOPIC": topics["legacy"], + "QDL_PRODUCTION_PUBLIC_TOPIC": topics["production_public"], + "QDL_PRODUCTION_LEGACY_TOPIC": topics["production_legacy"], + "QDL_AUTHORITY_NONCE": nonce, + "QDL_CANDIDATE_DIGEST": candidate_digest(candidate), + "QDL_PREREQUISITE_BUNDLE_ID": decision["bundle_id"], + "QDL_SLICE_ID": candidate["slice_id"], + "QDL_PYTHON_OWNER_ID": "python-primary-isolated", + "QDL_RUST_OWNER_ID": "rust-primary-isolated", + "QDL_ROLLBACK_OWNER_ID": "python-rollback-isolated", + } + runtime_result = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-phase92-primary-rehearsal", + env_values=runtime_env, + ), + timeout=180, + ) + runtime = json.loads(runtime_result.stdout.strip().splitlines()[-1]) + if runtime.get("status") != "PASS" or not all( + runtime.get("checks", {}).values() + ): + raise RuntimeError(f"Phase 9.2 runtime checks failed: {runtime}") + + recovery_result = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-phase92-primary-rehearsal", + env_values={ + **runtime_env, + "QDL_REHEARSAL_MODE": "RECOVERY_VERIFY", + "QDL_RECOVERY_AUTHORITY_REVISION": "11", + "QDL_RECOVERY_FIRST_WATERMARK": "101", + "QDL_RECOVERY_LAST_WATERMARK": "180", + }, + ), + timeout=180, + ) + recovery = json.loads(recovery_result.stdout.strip().splitlines()[-1]) + if recovery.get("status") != "PASS" or not all( + recovery.get("checks", {}).values() + ): + raise RuntimeError( + f"Phase 9.2 process restart recovery failed: {recovery}" + ) + + after = {name: total_end_offset(env, topic) for name, topic in topics.items()} + deltas = {name: after[name] - before[name] for name in topics} + expected = { + "authority": 5, + "audit": 5, + "checkpoint": 2, + "handoff": 2, + "primary": 81, + "public": 81, + "legacy": 81, + "production_public": 0, + "production_legacy": 0, + "transport": 0, + } + if deltas != expected: + raise RuntimeError( + f"Phase 9.2 isolated topic deltas diverged: {deltas} != {expected}" + ) + + compose(env, "stop", "kafka3") + one_loss_before = total_end_offset(env, topics["transport"]) + one_loss = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092,kafka2:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-one-loss", + }, + ), + check=False, + timeout=35, + ) + one_loss_after = total_end_offset(env, topics["transport"]) + one_replica_loss_acked = ( + one_loss.returncode == 0 and one_loss_after == one_loss_before + 1 + ) + compose(env, "stop", "kafka2") + min_isr_before = one_loss_after + min_isr = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-min-isr", + }, + ), + check=False, + timeout=35, + ) + compose(env, "start", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + min_isr_after = total_end_offset(env, topics["transport"]) + min_isr_failed_closed = ( + min_isr.returncode != 0 and min_isr_after == min_isr_before + ) + if not one_replica_loss_acked or not min_isr_failed_closed: + raise RuntimeError( + "Phase 9.2 broker durability gate failed " + f"one_loss={one_replica_loss_acked} min_isr={min_isr_failed_closed}" + ) + + compose(env, "restart", "kafka1", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + + audit = [ + json.loads(item) + for item in consume( + env, + topics["audit"], + 5, + f"phase8-phase92-audit-{nonce}", + ) + ] + checkpoints = [ + json.loads(item) + for item in consume( + env, + topics["checkpoint"], + 2, + f"phase8-phase92-checkpoint-{nonce}", + ) + ] + handoffs = [ + json.loads(item) + for item in consume( + env, + topics["handoff"], + 2, + f"phase8-phase92-handoff-{nonce}", + ) + ] + time.sleep(1.0) + slow_started = time.monotonic() + projections = {} + for name in ("primary", "public", "legacy"): + projections[name] = [ + json.loads(item) + for item in consume( + env, + topics[name], + 81, + f"phase8-phase92-slow-{name}-{nonce}", + timeout=60, + ) + ] + catchup_seconds = time.monotonic() - slow_started + authority = [ + json.loads(item) + for item in consume_compacted_records(env, topics["authority"]) + ] + latest = max(authority, key=lambda item: item["authority_revision"]) + states = [item["state"] for item in audit] + expected_states = [ + "RUST_CANARY", + "RUST_PRIMARY", + "BLOCKED", + "ROLLBACK_PENDING", + "PYTHON_PRIMARY", + ] + watermarks = [ + item["source_watermark"] for item in projections["primary"] + ] + projection_parity = ( + projections["primary"] == projections["public"] + == projections["legacy"] + ) + boundary_gap_free = watermarks == list(range(101, 182)) + owner_boundary = ( + all( + item["owner_id"] == "rust-primary-isolated" + for item in projections["primary"][:64] + ) + and all( + item["owner_id"] == "python-rollback-isolated" + for item in projections["primary"][64:] + ) + ) + if ( + states != expected_states + or latest["state"] != "PYTHON_PRIMARY" + or latest["authority_revision"] != 11 + or len(checkpoints) != 2 + or len(handoffs) != 2 + or not projection_parity + or not boundary_gap_free + or not owner_boundary + ): + raise RuntimeError("Phase 9.2 recovery/projection evidence diverged") + result = { + "schema": "qdl.phase92.broker-rehearsal.v1", + "status": "PASS", + "mode": "ISOLATED_REHEARSAL", + "production_authorized": False, + "image_digest": image_digest, + "runtime": runtime, + "process_restart_recovery": recovery, + "offset_deltas": deltas, + "authority_audit_states_after_restart": states, + "latest_authority_after_restart": { + "state": latest["state"], + "revision": latest["authority_revision"], + "owner_id": latest["owner_id"], + "lease_epoch": latest["lease_epoch"], + }, + "terminal_checkpoints": len(checkpoints), + "accepted_handoffs": len(handoffs), + "projection_parity": projection_parity, + "boundary_gap_free": boundary_gap_free, + "owner_boundary_correct": owner_boundary, + "one_replica_loss_acked": one_replica_loss_acked, + "min_isr_failed_closed": min_isr_failed_closed, + "slow_consumer": { + "delayed_seconds": 1.0, + "catchup_seconds": catchup_seconds, + "records_per_projection": 81, + "ordered_gap_free": boundary_gap_free, + }, + "cutover_ms": runtime["cutover_ms"], + "rollback_ms": runtime["rollback_ms"], + "production_public_writes": 0, + "production_legacy_writes": 0, + "final_authority": "PYTHON_PRIMARY", + } + except Exception as error: + status = compose(env, "ps", "--all", check=False, timeout=30) + logs = compose( + env, "logs", "--no-color", "--tail", "240", check=False, timeout=60 + ) + diagnostic = ( + status.stdout + status.stderr + logs.stdout + logs.stderr + )[-12000:] + raise RuntimeError( + f"Phase 9.2 broker rehearsal failed before cleanup: {error}\n" + f"isolated broker diagnostics:\n{diagnostic}" + ) from error + finally: + cleanup_result = cleanup(env) + v1_after = v1_topology() + health_after = v1_health() + cleanup_result.update( + { + "v1_health_before": health_before, + "v1_health_after": health_after, + "v1_topology_unchanged": v1_before == v1_after, + } + ) + result["cleanup"] = cleanup_result + if ( + health_before != 200 + or health_after != 200 + or not cleanup_result["v1_topology_unchanged"] + or any( + cleanup_result[key] + for key in ("containers_after", "networks_after", "volumes_after") + ) + ): + raise RuntimeError(f"Phase 9.2 cleanup/V1 invariant failed: {cleanup_result}") + return result + + +def render_report(evidence: dict) -> str: + parity_result = evidence["parity"] + broker = evidence["broker"] + cleanup_result = broker["cleanup"] + return f"""# Phase 9.2 Bounded Rust Primary Certification Report + +## Decision + +- Status: `{evidence['status']}` +- Production authorized: `{str(evidence['production_authorized']).lower()}` +- Production mutations: `{evidence['production_mutations']}` +- Prerequisite decision: `{evidence['prerequisite_decision']}` +- Slice: `{evidence['slice_id']}` +- Candidate digest: `{evidence['candidate_digest']}` + +## Authentic Parity + +- Provenance: `{parity_result['capture']['capture_provenance']}` +- Canonical events: `{parity_result['capture']['events']}` +- Semantic mismatches: `{parity_result['semantic_mismatches']}` +- Clean Rust process runs: `{parity_result['rust_clean_process_runs']}` + +## Terminal Handoff And Recovery + +- Authority states: `{', '.join(broker['authority_audit_states_after_restart'])}` +- Terminal checkpoints / accepted handoffs: `{broker['terminal_checkpoints']} / {broker['accepted_handoffs']}` +- Projection parity: `{str(broker['projection_parity']).lower()}` +- Boundary gap-free: `{str(broker['boundary_gap_free']).lower()}` +- Owner boundary correct: `{str(broker['owner_boundary_correct']).lower()}` +- Restart recovery: `{broker['process_restart_recovery']['status']}` +- Recovered target watermarks: `{broker['process_restart_recovery']['restored_target_watermarks']}` +- First post-restart watermark: `{broker['process_restart_recovery']['resumed_watermark']}` +- Cutover / rollback measurement: `{broker['cutover_ms']:.3f} ms / {broker['rollback_ms']:.3f} ms` +- One-replica-loss ACK: `{str(broker['one_replica_loss_acked']).lower()}` +- Below-min-ISR fail closed: `{str(broker['min_isr_failed_closed']).lower()}` +- Final authority: `{broker['final_authority']}` +- Production public / legacy writes: `{broker['production_public_writes']} / {broker['production_legacy_writes']}` + +## Isolation And Cleanup + +- V1 health before/after: `{cleanup_result['v1_health_before']} / {cleanup_result['v1_health_after']}` +- V1 topology unchanged: `{str(cleanup_result['v1_topology_unchanged']).lower()}` +- Containers/networks/volumes remaining: `{cleanup_result['containers_after']} / {cleanup_result['networks_after']} / {cleanup_result['volumes_after']}` + +## Remaining External Gates + +- Phase 9.0-C remains `NO_GO_EXTERNAL`; a production primary transition is not authorized. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- A real production canary hold and explicit exact-slice approval remain required. +""" + + +def write_evidence(evidence: dict) -> None: + OUTPUT.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + REPORT.write_text(render_report(evidence)) + lines = [] + for path in (OUTPUT, REPORT, MIGRATION_EVIDENCE): + digest = hashlib.sha256(path.read_bytes()).hexdigest() + lines.append(f"{digest} {path.relative_to(ROOT)}") + CHECKSUM.write_text("\n".join(lines) + "\n") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--rust-image", required=True) + parser.add_argument("--repeat", type=int, default=200) + args = parser.parse_args() + if args.repeat < 10: + raise ValueError("Phase 9.2 parity repeat must be at least 10") + candidate = yaml.safe_load(CANDIDATE.read_text()) + decision = json.loads(PREREQUISITE_DECISION.read_text()) + digest = candidate_digest(candidate) + if decision.get("candidate_digest") != digest: + raise RuntimeError("Phase 9.0-C decision does not bind the candidate") + if decision.get("decision") != "NO_GO_EXTERNAL": + raise RuntimeError("isolated Phase 9.2 harness expects NO_GO_EXTERNAL") + migration = json.loads(MIGRATION_EVIDENCE.read_text()) + if migration.get("status") != "PASS" or migration.get("production_mutations") != 0: + raise RuntimeError("Phase 9.2 migration evidence is absent or invalid") + inspect = run( + ["docker", "image", "inspect", args.rust_image, "--format", "{{.Id}}"] + ) + image_digest = inspect.stdout.strip() + try: + parity_result = parity(args.rust_image, args.repeat) + broker_result = broker_rehearsal( + args.rust_image, image_digest, candidate, decision + ) + finally: + BUNDLE_PATH.unlink(missing_ok=True) + evidence = { + "schema": "qdl.phase92.bounded-primary-certification.v1", + "status": "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED", + "issued_at_ns": time.time_ns(), + "slice_id": candidate["slice_id"], + "candidate_digest": digest, + "prerequisite_bundle_id": decision["bundle_id"], + "prerequisite_decision": decision["decision"], + "production_authorized": False, + "production_mutations": 0, + "python_v1_public_authority_unchanged": True, + "migration": migration, + "parity": parity_result, + "broker": broker_result, + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "real production canary hold and exact-slice approval remain unavailable", + "same-host replicated broker is not an independent failure domain", + ], + } + write_evidence(evidence) + print( + json.dumps( + { + "status": evidence["status"], + "authentic_events": parity_result["capture"]["events"], + "semantic_mismatches": parity_result["semantic_mismatches"], + "projection_parity": broker_result["projection_parity"], + "boundary_gap_free": broker_result["boundary_gap_free"], + "production_public_writes": broker_result["production_public_writes"], + "production_legacy_writes": broker_result["production_legacy_writes"], + "cleanup": broker_result["cleanup"], + }, + sort_keys=True, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase93_hold_close_certification.py b/scripts/phase93_hold_close_certification.py new file mode 100755 index 0000000..9cec407 --- /dev/null +++ b/scripts/phase93_hold_close_certification.py @@ -0,0 +1,370 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import pathlib +import sys +import time +import uuid + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.canary.phase93 import ( # noqa: E402 + DecommissionRequest, + ExpansionManifest, + ExpansionType, + HoldScope, + PrimaryHoldEvaluator, + PrimaryHoldIdentity, + PrimaryHoldObservation, + PrimaryHoldPolicy, + ProductionClosureAuthorizer, + RollbackWindowClosure, + assess_decommission, +) +from qdl.certification.prerequisites import CandidateSlice # noqa: E402 +from scripts.phase80_broker_certification import v1_health, v1_topology # noqa: E402 + + +NO_GO = ROOT / "upgrade/evidence/phase90c-production-prerequisites.json" +PHASE92 = ROOT / "upgrade/evidence/phase92-bounded-primary-certification.json" +MIGRATION = ROOT / "upgrade/evidence/phase93-hold-close-migration.json" +OUTPUT = ROOT / "upgrade/evidence/phase93-hold-close-expand-certification.json" +REPORT = ROOT / "upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md" +CHECKSUM = ROOT / "upgrade/evidence/phase93-evidence.sha256" + + +def uid(label: str) -> str: + return str(uuid.uuid5(uuid.NAMESPACE_URL, f"qdl-phase93:{label}")) + + +def digest(label: str) -> str: + return hashlib.sha256(label.encode()).hexdigest() + + +def certify_test_hold( + candidate: CandidateSlice, bundle_id: str, now_ns: int +): + policy = PrimaryHoldPolicy( + minimum_duration_seconds=120, + sample_interval_seconds=60, + max_sample_gap_seconds=60, + max_lag_ms=500, + max_freshness_ms=1000, + max_queue_depth=1000, + max_spool_bytes=1_000_000, + max_cpu_percent=80.0, + max_rss_mb=512.0, + ) + start = now_ns - 120_000_000_000 + identity = PrimaryHoldIdentity( + schema="qdl.primary-hold.v1", + hold_id=uid("test-hold"), + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=bundle_id, + owner_id="rust-primary-test-fixture", + authority_revision=8, + lease_epoch=12, + partition_plan_epoch=1, + started_at_ns=start, + required_until_ns=now_ns, + policy_digest=policy.digest, + ) + evaluator = PrimaryHoldEvaluator( + identity=identity, + policy=policy, + scope=HoldScope.TEST_REHEARSAL, + ) + for sequence, watermark in ((1, 110), (2, 120)): + reason = evaluator.observe( + PrimaryHoldObservation( + schema="qdl.primary-hold-observation.v1", + observation_id=uid(f"test-observation-{sequence}"), + hold_id=identity.hold_id, + slice_id=identity.slice_id, + candidate_digest=identity.candidate_digest, + owner_id=identity.owner_id, + authority_revision=identity.authority_revision, + lease_epoch=identity.lease_epoch, + partition_plan_epoch=identity.partition_plan_epoch, + sequence=sequence, + observed_at_ns=start + sequence * 60_000_000_000, + last_watermark=watermark, + lag_ms=10, + freshness_ms=20, + queue_depth=1, + spool_bytes=100, + cpu_percent=10.0, + rss_mb=64.0, + registered_consumers=2, + healthy_consumers=2, + checkpoint_watermark=watermark, + ) + ) + if reason != "PASS": + raise RuntimeError(f"test hold observation failed: {reason}") + return evaluator.decision(decision_id=uid("test-hold-decision"), now_ns=now_ns) + + +def test_parent_closure( + candidate: CandidateSlice, bundle_id: str, now_ns: int +) -> RollbackWindowClosure: + return RollbackWindowClosure( + schema="qdl.rollback-window-closure.v1", + closure_id=uid("test-parent-closure"), + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=bundle_id, + owner_id="rust-primary-test-fixture", + authority_revision=8, + lease_epoch=12, + partition_plan_epoch=1, + hold_decision_id=uid("test-hold-decision"), + hold_decision_digest=digest("test-hold-decision"), + consumer_registry_snapshot_id=uid("test-consumer-registry"), + consumer_registry_digest=digest("test-consumer-registry"), + authority_registry_snapshot_id=uid("test-authority-registry"), + authority_registry_digest=digest("test-authority-registry"), + rollback_rehearsal_id=uid("test-rollback"), + rollback_rehearsal_digest=digest("test-rollback"), + approval_id=uid("test-approval"), + approval_digest=digest("test-approval"), + operator="phase93-test-fixture", + change_ticket="QDL-93-TEST", + closed_at_ns=now_ns, + production_authorized=True, + ) + + +def render_report(evidence: dict) -> str: + parent = evidence["parent_phase92"] + control = evidence["control_plane_fixture"] + return f"""# Phase 9.3 Hold, Close And Expand Certification Report + +## Decision + +- Status: {evidence['status']} +- Production authorized: {evidence['production_authorized']} +- Production hold started: {evidence['production_hold_started']} +- Production rollback window closed: {evidence['production_rollback_window_closed']} +- Production expansions authorized: {evidence['production_expansions_authorized']} +- Production mutations: {evidence['production_mutations']} + +## Parent Evidence + +- Phase 9.2 status: {parent['status']} +- Authentic provider events: {parent['authentic_events']} +- Semantic mismatches: {parent['semantic_mismatches']} +- Parent production authorized: {parent['production_authorized']} + +## Isolated Control Plane + +- Provenance: {control['provenance']} +- Accelerated time is production evidence: {control['accelerated_time_is_production_evidence']} +- Test hold status: {control['test_hold_status']} +- Test hold production authorized: {control['test_hold_production_authorized']} +- Current no-go rejection: {control['current_no_go_rejection']} +- Local Phase 9.2 production eligible: {control['local_phase92_production_eligible']} +- Expansion manifests: {control['expansion_manifest_count']} +- Decommission decision: {control['decommission_reason']} + +## Persistence And Isolation + +- Migration: {evidence['migration']['status']} +- Closure changed authority: {not evidence['migration']['closure_did_not_mutate_authority']} +- V1 health before/after: {evidence['v1']['health_before']} / {evidence['v1']['health_after']} +- V1 topology unchanged: {evidence['v1']['topology_unchanged']} +- Disposable resources remaining: {evidence['cleanup']['resources_remaining']} + +## External Gates + +Phase 9.0-C remains NO_GO_EXTERNAL. There is no real Rust primary, production +hold duration, production consumer checkpoint set or operator closure approval. +No rollback window, expansion or Python decommission is authorized. +""" + + +def freeze(evidence: dict) -> None: + OUTPUT.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + REPORT.write_text(render_report(evidence)) + entries = [] + for path in (OUTPUT, REPORT, MIGRATION): + entries.append( + f"{hashlib.sha256(path.read_bytes()).hexdigest()} " + f"{path.relative_to(ROOT)}" + ) + CHECKSUM.write_text("\n".join(entries) + "\n") + + +def main() -> int: + argparse.ArgumentParser().parse_args() + candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + no_go = json.loads(NO_GO.read_text()) + phase92 = json.loads(PHASE92.read_text()) + migration = json.loads(MIGRATION.read_text()) + if no_go.get("decision") != "NO_GO_EXTERNAL": + raise RuntimeError("Phase 9.3 expects current NO_GO_EXTERNAL") + if ( + phase92.get("status") + != "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED" + or phase92.get("production_authorized") is not False + or phase92["parity"]["semantic_mismatches"] != 0 + ): + raise RuntimeError("Phase 9.2 parent evidence is invalid") + if migration.get("status") != "PASS": + raise RuntimeError("Phase 9.3 migration evidence is invalid") + + before_topology = v1_topology() + before_health = v1_health() + now_ns = time.time_ns() + hold = certify_test_hold(candidate, no_go["bundle_id"], now_ns) + denied = ProductionClosureAuthorizer().authorize( + candidate=candidate, + prerequisite_decision=no_go, + expected_bundle_id=no_go["bundle_id"], + primary_evidence=phase92, + hold_decision=None, + consumer_registry=None, + authority_registry=None, + rollback_evidence=None, + approval=None, + now_ns=no_go["issued_at_ns"] + 1, + ) + if denied.allowed or denied.reason != "PREREQUISITE_DECISION_NOT_GO": + raise RuntimeError("current no-go did not fail closed") + + parent = test_parent_closure(candidate, no_go["bundle_id"], now_ns) + expansions = [] + for index, kind in enumerate(ExpansionType, start=1): + item = ExpansionManifest.plan( + expansion_id=uid(f"expansion-{kind}"), + parent=parent, + expansion_type=kind, + candidate_digest=digest(f"candidate-{kind}"), + scope_digest=digest(f"scope-{kind}"), + partition_plan_epoch=( + parent.partition_plan_epoch + 1 + if kind == ExpansionType.INSTRUMENT_PARTITION + else parent.partition_plan_epoch + ), + created_at_ns=now_ns + index, + ) + expansions.append( + { + "type": kind, + "status": item.status, + "required_gate_count": len(item.required_gates), + "write_authority": ( + item.public_write_allowed or item.legacy_write_allowed + ), + "transitive_evidence_allowed": ( + item.transitive_evidence_allowed + ), + "digest": item.digest, + } + ) + if any( + item["write_authority"] or item["transitive_evidence_allowed"] + for item in expansions + ): + raise RuntimeError("expansion inherited authority or certification") + + decommission = assess_decommission( + DecommissionRequest( + schema="qdl.runtime-decommission-request.v1", + request_id=uid("blocked-decommission"), + runtime_id="python-authoritative-runtime", + owned_slice_ids=(), + rollback_reference_ids=("phase92-python-rollback",), + consumer_dependency_ids=(), + all_replacement_windows_closed=False, + repository_cleanup_approved=False, + shared_knowledge_retained=True, + ) + ) + if decommission.allowed: + raise RuntimeError("rollback dependency allowed decommission") + + after_topology = v1_topology() + after_health = v1_health() + evidence = { + "schema": "qdl.phase93.hold-close-expand-certification.v1", + "status": "COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED", + "issued_at_ns": now_ns, + "slice_id": candidate.payload["slice_id"], + "candidate_digest": candidate.digest, + "prerequisite_decision": no_go["decision"], + "production_authorized": False, + "production_hold_started": False, + "production_rollback_window_closed": False, + "production_expansions_authorized": 0, + "python_decommission_authorized": False, + "production_mutations": 0, + "parent_phase92": { + "status": phase92["status"], + "authentic_events": phase92["parity"]["capture"]["events"], + "semantic_mismatches": phase92["parity"]["semantic_mismatches"], + "production_authorized": phase92["production_authorized"], + "sha256": hashlib.sha256(PHASE92.read_bytes()).hexdigest(), + }, + "control_plane_fixture": { + "provenance": "TEST_CONTROL_PLANE_FIXTURE", + "accelerated_time_is_production_evidence": False, + "test_hold_status": hold.status, + "test_hold_production_authorized": hold.production_authorized, + "current_no_go_rejection": denied.reason, + "local_phase92_production_eligible": False, + "expansion_manifest_count": len(expansions), + "expansions": expansions, + "decommission_reason": decommission.reason, + }, + "migration": migration, + "v1": { + "health_before": before_health, + "health_after": after_health, + "topology_unchanged": before_topology == after_topology, + }, + "cleanup": { + "resources_remaining": 0, + "production_rows_created": 0, + }, + "technical_debt": [ + "Phase 9.0-C production infrastructure remains NO_GO_EXTERNAL", + "real production primary and sustained hold observations do not exist", + "real consumer checkpoints and operator closure approval do not exist", + "every expansion remains independently uncertified", + ], + } + if ( + hold.production_authorized + or before_health != 200 + or after_health != 200 + or before_topology != after_topology + ): + raise RuntimeError("Phase 9.3 isolation invariant failed") + freeze(evidence) + print( + json.dumps( + { + "status": evidence["status"], + "parent_authentic_events": phase92["parity"]["capture"]["events"], + "current_no_go_rejection": denied.reason, + "expansion_manifests": len(expansions), + "v1_health_before": before_health, + "v1_health_after": after_health, + "v1_topology_unchanged": before_topology == after_topology, + "production_mutations": 0, + "cleanup": True, + }, + sort_keys=True, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase93_migration_smoke.py b/scripts/phase93_migration_smoke.py new file mode 100755 index 0000000..c15ef6d --- /dev/null +++ b/scripts/phase93_migration_smoke.py @@ -0,0 +1,660 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import sys +import time +from datetime import datetime, timedelta, timezone + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.certification.prerequisites import CandidateSlice # noqa: E402 + + +IMAGE = os.environ.get( + "QDL_PHASE93_POSTGRES_IMAGE", "timescale/timescaledb:latest-pg15" +) +CONTAINER = os.environ.get( + "QDL_PHASE93_POSTGRES_CONTAINER", f"qdl_phase93_postgres_{os.getpid()}" +) +OUTPUT = pathlib.Path( + os.environ.get( + "QDL_PHASE93_MIGRATION_OUTPUT", + str(ROOT / "upgrade/evidence/phase93-hold-close-migration.json"), + ) +) +CANDIDATE = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") +DIGEST = CANDIDATE.digest +SLICE = "production/binance/usdm/perpetual/trade/plan-1/btcusdt" +BUNDLE = "00000000-0000-4000-8000-000000000093" + + +def run( + args: list[str], + *, + input_text: str | None = None, + check: bool = True, +) -> subprocess.CompletedProcess[str]: + result = subprocess.run( + args, + cwd=ROOT, + input=input_text, + text=True, + capture_output=True, + check=False, + ) + if check and result.returncode: + raise RuntimeError( + f"command failed ({result.returncode}): {' '.join(args)}\n" + f"stdout={result.stdout[-4000:]}\nstderr={result.stderr[-4000:]}" + ) + return result + + +def psql(sql: str, *, check: bool = True) -> subprocess.CompletedProcess[str]: + return run( + [ + "docker", + "exec", + "-i", + CONTAINER, + "psql", + "-v", + "ON_ERROR_STOP=1", + "-U", + "postgres", + "-d", + "postgres", + ], + input_text=sql, + check=check, + ) + + +def query(sql: str) -> str: + return run( + [ + "docker", + "exec", + CONTAINER, + "psql", + "-U", + "postgres", + "-d", + "postgres", + "-Atc", + sql, + ] + ).stdout.strip() + + +def expect_failure(sql: str) -> None: + if psql(sql, check=False).returncode == 0: + raise RuntimeError(f"expected SQL failure but statement succeeded: {sql}") + + +def apply_migrations() -> None: + for path in sorted((ROOT / "migrations/postgres").glob("*.sql")): + run( + [ + "docker", + "exec", + CONTAINER, + "psql", + "-v", + "ON_ERROR_STOP=1", + "-U", + "postgres", + "-d", + "postgres", + "-f", + f"/migrations/{path.name}", + ] + ) + + +def cleanup() -> None: + run(["docker", "rm", "-f", CONTAINER], check=False) + + +def main() -> int: + now = datetime.now(timezone.utc).replace(microsecond=0) + t = lambda value: value.isoformat() + cleanup() + run( + [ + "docker", + "run", + "-d", + "--name", + CONTAINER, + "--network", + "none", + "--security-opt", + "no-new-privileges:true", + "--pids-limit", + "256", + "--memory", + "768m", + "--cpus", + "1.0", + "--tmpfs", + "/var/lib/postgresql/data:rw,nosuid,nodev,size=512m", + "-e", + "POSTGRES_HOST_AUTH_METHOD=trust", + "-v", + f"{ROOT / 'migrations/postgres'}:/migrations:ro", + IMAGE, + ] + ) + try: + consecutive_ready = 0 + for _ in range(90): + ready = run( + [ + "docker", + "exec", + CONTAINER, + "pg_isready", + "-U", + "postgres", + "-d", + "postgres", + ], + check=False, + ) + consecutive_ready = ( + consecutive_ready + 1 if ready.returncode == 0 else 0 + ) + if consecutive_ready >= 3: + break + time.sleep(1) + else: + raise RuntimeError("Phase 9.3 PostgreSQL did not become stably ready") + apply_migrations() + + psql( + f""" +INSERT INTO qdl_production_prerequisite_bundles ( + bundle_id,candidate_digest,policy_revision,decision,evidence,evidence_sha256, + issued_by,issued_at,expires_at +) VALUES ( + '{BUNDLE}','{DIGEST}',1,'GO','{{}}',repeat('5',64),'phase93-test', + '{t(now - timedelta(minutes=5))}','{t(now + timedelta(days=1))}' +); +INSERT INTO qdl_authority_slices ( + slice_id,environment,venue,market,product_type,feed,partition_plan_epoch, + partition_id,schema_major,state,authority_revision,owner_id,lease_epoch, + terminal_watermark,candidate_digest,artifact_image_digest,sbom_digest, + signature_identity,contract_digest,normalizer_version,adapter_version, + config_revision,instrument_catalog_revision,source_policy_revision, + partition_plan_digest,rollback_manifest_digest,prerequisite_bundle_id, + approved_by,approved_at,hold_until +) VALUES ( + '{SLICE}','production','BINANCE','USDM','PERPETUAL','TRADE',1, + 'rendezvous-sha256-v1:epoch-1:btcusdt',2,'RUST_CANARY',3, + 'python-primary',1,100,'{DIGEST}','sha256:{DIGEST}',repeat('1',64), + 'phase93-test-signer',repeat('2',64),'qdl-rust-core/test', + 'binance-usdm/test','phase93-test-config','phase93-test-catalog', + 'phase93-test-source-policy',repeat('3',64),repeat('4',64),'{BUNDLE}', + 'phase93-test','{t(now)}','{t(now + timedelta(hours=2))}' +); +INSERT INTO qdl_terminal_owner_checkpoints ( + checkpoint_id,slice_id,owner_id,authority_revision,lease_epoch, + partition_plan_epoch,source_session_id,connection_generation, + terminal_watermark,terminal_event_id,terminal_payload_sha256, + candidate_digest,committed_at +) VALUES ( + '91000000-0000-4000-8000-000000000093','{SLICE}','python-primary', + 3,1,1,'python-session-93',1,100,'event-100',repeat('6',64), + '{DIGEST}','{t(now)}' +); +INSERT INTO qdl_authority_handoffs ( + handoff_id,checkpoint_id,direction,slice_id,old_owner_id,new_owner_id, + expected_state,new_state,expected_authority_revision,new_authority_revision, + expected_lease_epoch,new_lease_epoch,partition_plan_epoch, + terminal_watermark,first_new_watermark,overlap_start_watermark, + overlap_end_watermark,old_event_count,new_event_count,semantic_mismatches, + open_gaps,candidate_digest,prerequisite_bundle_id,handoff_sha256, + approved_by,approved_at,expires_at +) VALUES ( + '92000000-0000-4000-8000-000000000093', + '91000000-0000-4000-8000-000000000093','PYTHON_TO_RUST','{SLICE}', + 'python-primary','rust-primary','RUST_CANARY','RUST_PRIMARY',3,4,1,2,1, + 100,101,90,100,11,11,0,0,'{DIGEST}','{BUNDLE}',repeat('7',64), + 'phase93-test','{t(now)}','{t(now + timedelta(hours=2))}' +); +SELECT (qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000093', + '93000000-0000-4000-8000-000000000093','{SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'{BUNDLE}','{t(now + timedelta(hours=1))}', + 'phase93-test','accepted primary handoff' +)).state; +""" + ) + + psql( + f""" +INSERT INTO qdl_primary_holds ( + hold_id,slice_id,candidate_digest,prerequisite_bundle_id,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,started_at, + required_until,policy_digest,minimum_duration_seconds, + max_sample_gap_seconds,max_lag_ms,max_freshness_ms,max_queue_depth, + max_spool_bytes,max_cpu_percent,max_rss_mb +) VALUES +( + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}','{BUNDLE}', + 'rust-primary',4,2,1,'{t(now - timedelta(minutes=2))}','{t(now)}', + repeat('8',64),120,60,500,1000,1000,1000000,80,512 +), +( + '94000000-0000-4000-8000-000000000002','{SLICE}','{DIGEST}','{BUNDLE}', + 'rust-primary',4,2,1,'{t(now - timedelta(minutes=1))}','{t(now)}', + repeat('9',64),60,60,500,1000,1000,1000000,80,512 +); +INSERT INTO qdl_primary_hold_observations ( + observation_id,hold_id,slice_id,candidate_digest,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,sequence,observed_at, + last_watermark,lag_ms,freshness_ms,queue_depth,spool_bytes,cpu_percent, + rss_mb,registered_consumers,healthy_consumers,checkpoint_watermark +) VALUES +( + '95000000-0000-4000-8000-000000000001', + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,1,'{t(now - timedelta(minutes=1))}',110, + 10,20,1,100,10,64,2,2,110 +), +( + '95000000-0000-4000-8000-000000000002', + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,2,'{t(now)}',120, + 10,20,1,100,10,64,2,2,120 +); +INSERT INTO qdl_primary_hold_observations ( + observation_id,hold_id,slice_id,candidate_digest,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,sequence,observed_at, + last_watermark,semantic_mismatches,lag_ms,freshness_ms,queue_depth, + spool_bytes,cpu_percent,rss_mb,registered_consumers,healthy_consumers, + checkpoint_watermark +) VALUES ( + '95000000-0000-4000-8000-000000000003', + '94000000-0000-4000-8000-000000000002','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,1,'{t(now)}',120,1,10,20,1,100,10,64,2,2,120 +); +INSERT INTO qdl_primary_hold_decisions ( + decision_id,hold_id,status,reason,scope,production_authorized,slice_id, + candidate_digest,prerequisite_bundle_id,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,policy_digest,first_observed_at, + last_observed_at,observation_count,terminal_watermark,decided_at, + decision_sha256 +) VALUES ( + '96000000-0000-4000-8000-000000000002', + '94000000-0000-4000-8000-000000000002','BLOCKED','SEMANTIC_MISMATCH', + 'TEST_REHEARSAL',FALSE,'{SLICE}','{DIGEST}','{BUNDLE}','rust-primary', + 4,2,1,repeat('9',64),'{t(now)}','{t(now)}',1,120,'{t(now)}', + repeat('b',64) +); +""" + ) + + expect_failure( + f""" +INSERT INTO qdl_primary_hold_observations ( + observation_id,hold_id,slice_id,candidate_digest,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,sequence,observed_at, + last_watermark,registered_consumers,healthy_consumers,checkpoint_watermark +) VALUES ( + '95000000-0000-4000-8000-000000000009', + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,4,'{t(now + timedelta(minutes=1))}',121,2,2,121 +); +""" + ) + expect_failure( + f""" +INSERT INTO qdl_primary_hold_decisions ( + decision_id,hold_id,status,reason,scope,production_authorized,slice_id, + candidate_digest,prerequisite_bundle_id,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,policy_digest,first_observed_at, + last_observed_at,observation_count,terminal_watermark,decided_at, + decision_sha256 +) VALUES ( + '96000000-0000-4000-8000-000000000009', + '94000000-0000-4000-8000-000000000002','PASSED','PASS','PRODUCTION', + TRUE,'{SLICE}','{DIGEST}','{BUNDLE}','rust-primary',4,2,1,repeat('9',64), + '{t(now)}','{t(now)}',1,120,'{t(now)}',repeat('c',64) +); +""" + ) + + psql( + f""" +INSERT INTO qdl_primary_hold_decisions ( + decision_id,hold_id,status,reason,scope,production_authorized,slice_id, + candidate_digest,prerequisite_bundle_id,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,policy_digest,first_observed_at, + last_observed_at,observation_count,terminal_watermark,decided_at, + decision_sha256 +) VALUES ( + '96000000-0000-4000-8000-000000000001', + '94000000-0000-4000-8000-000000000001','PASSED','PASS','PRODUCTION', + TRUE,'{SLICE}','{DIGEST}','{BUNDLE}','rust-primary',4,2,1,repeat('8',64), + '{t(now - timedelta(minutes=1))}','{t(now)}',2,120,'{t(now)}', + repeat('a',64) +); +INSERT INTO qdl_consumer_registry_snapshots ( + snapshot_id,slice_id,authority_revision,checkpoint_count, + ready_checkpoint_count,minimum_checkpoint_watermark, + checkpoint_regressions,unresolved_migrations,rollback_ready, + registry_sha256,details,observed_at +) VALUES ( + '97000000-0000-4000-8000-000000000001','{SLICE}',4,2,2,130, + 0,0,TRUE,repeat('c',64),'{{"consumers":["alpha","execution"]}}','{t(now)}' +); +INSERT INTO qdl_authority_registry_snapshots ( + snapshot_id,slice_id,state,owner_id,authority_revision,lease_epoch, + partition_plan_epoch,candidate_digest,prerequisite_bundle_id, + current_watermark,public_write_allowed,legacy_write_allowed, + registry_sha256,observed_at +) VALUES ( + '97000000-0000-4000-8000-000000000002','{SLICE}','RUST_PRIMARY', + 'rust-primary',4,2,1,'{DIGEST}','{BUNDLE}',130,TRUE,TRUE, + repeat('d',64),'{t(now)}' +); +INSERT INTO qdl_rollback_rehearsals ( + rehearsal_id,slice_id,candidate_digest,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,rollback_manifest_digest, + reconciled_through_watermark,rto_ms,status,production_scope, + rehearsal_sha256,observed_at,expires_at +) VALUES ( + '97000000-0000-4000-8000-000000000003','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,repeat('4',64),130,500,'PASS',TRUE, + repeat('e',64),'{t(now)}','{t(now + timedelta(hours=1))}' +); +INSERT INTO qdl_closure_approvals ( + approval_id,slice_id,candidate_digest,prerequisite_bundle_id,hold_id, + hold_policy_digest,decision,allow_close_rollback_window, + repository_cleanup_approved,operator,change_ticket,approval_sha256, + approved_at,expires_at +) VALUES ( + '97000000-0000-4000-8000-000000000004','{SLICE}','{DIGEST}','{BUNDLE}', + '94000000-0000-4000-8000-000000000001',repeat('8',64),'APPROVE', + TRUE,FALSE,'phase93-test','QDL-93',repeat('f',64), + '{t(now - timedelta(minutes=1))}','{t(now + timedelta(hours=1))}' +); +SELECT (qdl_close_authority_window( + '98000000-0000-4000-8000-000000000001',repeat('1',64), + '96000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000002', + '97000000-0000-4000-8000-000000000003', + '97000000-0000-4000-8000-000000000004','{t(now)}' +)).closure_id; +""" + ) + + closure_state = query( + f"SELECT state||':'||authority_revision||':'||owner_id||':'||" + f"lease_epoch||':'||terminal_watermark FROM qdl_authority_slices " + f"WHERE slice_id='{SLICE}';" + ) + if closure_state != "RUST_PRIMARY:4:rust-primary:2:100": + raise RuntimeError(f"closure mutated authority: {closure_state}") + + expect_failure( + "UPDATE qdl_authority_closures SET change_ticket='mutated';" + ) + expect_failure("DELETE FROM qdl_primary_hold_observations;") + expect_failure( + "UPDATE qdl_consumer_registry_snapshots SET checkpoint_count=3;" + ) + expect_failure( + f""" +INSERT INTO qdl_expansion_candidates ( + expansion_id,parent_closure_id,parent_slice_id,parent_candidate_digest, + parent_closure_digest,expansion_type,candidate_digest,scope_digest, + partition_plan_epoch,required_gates,status,transitive_evidence_allowed, + public_write_allowed,legacy_write_allowed,created_at +) VALUES ( + '99000000-0000-4000-8000-000000000009', + '98000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + repeat('1',64),'BBO',repeat('2',64),repeat('3',64),1, + ARRAY['rollback'],'INDEPENDENT_CERTIFICATION_REQUIRED',FALSE,FALSE,FALSE, + '{t(now)}' +); +""" + ) + + expansion_rows = [ + ( + "99000000-0000-4000-8000-000000000001", + "INSTRUMENT_PARTITION", + "2", + "3", + 2, + [ + "authority_handoff", + "capacity_headroom", + "exact_frame_parity", + "partition_churn", + "provider_authentic_source", + "rollback", + "source_capacity", + ], + ), + ( + "99000000-0000-4000-8000-000000000002", + "BBO", + "3", + "4", + 1, + [ + "authority_handoff", + "capacity_headroom", + "coalescing_policy", + "exact_frame_parity", + "freshness", + "ordering_reconnect", + "provider_authentic_source", + "quote_identity", + "rollback", + ], + ), + ( + "99000000-0000-4000-8000-000000000003", + "L2_BOOK", + "4", + "5", + 1, + [ + "authority_handoff", + "capacity_headroom", + "checksum", + "exact_frame_parity", + "lossless_backpressure", + "provider_authentic_source", + "resync", + "rollback", + "snapshot_delta_sequence", + ], + ), + ( + "99000000-0000-4000-8000-000000000004", + "BAR_LIFECYCLE", + "5", + "6", + 1, + [ + "authority_handoff", + "capacity_headroom", + "close_time_semantics", + "exact_frame_parity", + "final_revision_lineage", + "provider_authentic_source", + "replay", + "rollback", + ], + ), + ( + "99000000-0000-4000-8000-000000000005", + "VENUE_MARKET", + "6", + "7", + 1, + [ + "adapter_capability", + "authority_handoff", + "capacity_headroom", + "disaster_recovery", + "entitlement", + "exact_frame_parity", + "instrument_identity", + "provider_authentic_source", + "provider_semantics", + "rollback", + ], + ), + ] + for expansion_id, kind, candidate_char, scope_char, epoch, gates in expansion_rows: + gate_sql = ",".join(f"'{value}'" for value in gates) + psql( + f""" +INSERT INTO qdl_expansion_candidates ( + expansion_id,parent_closure_id,parent_slice_id,parent_candidate_digest, + parent_closure_digest,expansion_type,candidate_digest,scope_digest, + partition_plan_epoch,required_gates,status,transitive_evidence_allowed, + public_write_allowed,legacy_write_allowed,created_at +) VALUES ( + '{expansion_id}','98000000-0000-4000-8000-000000000001','{SLICE}', + '{DIGEST}',repeat('1',64),'{kind}',repeat('{candidate_char}',64), + repeat('{scope_char}',64),{epoch},ARRAY[{gate_sql}], + 'INDEPENDENT_CERTIFICATION_REQUIRED',FALSE,FALSE,FALSE,'{t(now)}' +); +""" + ) + + psql( + f""" +INSERT INTO qdl_runtime_decommission_decisions ( + decision_id,runtime_id,owned_slice_count,rollback_reference_count, + consumer_dependency_count,all_replacement_windows_closed, + repository_cleanup_approved,shared_knowledge_retained,allowed,reason, + decided_at +) VALUES +( + '99000000-0000-4000-8000-000000000006','python-usdm-trade', + 1,1,1,FALSE,FALSE,TRUE,FALSE,'RUNTIME_STILL_OWNS_SLICES','{t(now)}' +), +( + '99000000-0000-4000-8000-000000000007','retired-test-runtime', + 0,0,0,TRUE,TRUE,TRUE,TRUE,'AUTHORIZED','{t(now)}' +); +SELECT qdl_transition_authority( + '99000000-0000-4000-8000-000000000008','{SLICE}', + 'RUST_PRIMARY',4,'rust-primary',2,1,'BLOCKED','rust-primary',2, + 130,NULL,NULL,'phase93-test','stale closure CAS test' +); +""" + ) + expect_failure( + f""" +SELECT qdl_close_authority_window( + '98000000-0000-4000-8000-000000000009',repeat('9',64), + '96000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000002', + '97000000-0000-4000-8000-000000000003', + '97000000-0000-4000-8000-000000000004','{t(now)}' +); +""" + ) + + apply_migrations() + final_state = query( + f"SELECT state||':'||authority_revision||':'||owner_id||':'||" + f"lease_epoch||':'||terminal_watermark FROM qdl_authority_slices " + f"WHERE slice_id='{SLICE}';" + ) + counts = query( + "SELECT " + "(SELECT count(*) FROM qdl_primary_holds)||':'||" + "(SELECT count(*) FROM qdl_primary_hold_observations)||':'||" + "(SELECT count(*) FROM qdl_primary_hold_decisions)||':'||" + "(SELECT count(*) FROM qdl_authority_closures)||':'||" + "(SELECT count(*) FROM qdl_expansion_candidates)||':'||" + "(SELECT count(*) FROM qdl_runtime_decommission_decisions);" + ) + if final_state != "BLOCKED:5:rust-primary:2:130": + raise RuntimeError(f"unexpected final test state: {final_state}") + if counts != "2:3:2:1:5:2": + raise RuntimeError(f"unexpected Phase 9.3 row counts: {counts}") + + values = [int(value) for value in counts.split(":")] + OUTPUT.parent.mkdir(parents=True, exist_ok=True) + OUTPUT.write_text( + json.dumps( + { + "schema": "qdl.phase93.hold-close-migration.v1", + "status": "PASS", + "authority_state_after_closure": closure_state, + "final_test_state_after_stale_cas_setup": final_state, + "hold_records": values[0], + "observation_records": values[1], + "decision_records": values[2], + "closure_records": values[3], + "expansion_records": values[4], + "decommission_records": values[5], + "dirty_hold_pass_rejected": True, + "out_of_order_observation_rejected": True, + "append_only_mutation_rejected": True, + "registry_mutation_rejected": True, + "closure_did_not_mutate_authority": True, + "closure_digest_bound_to_expansion": True, + "approval_and_closure_ids_distinct": True, + "stale_authority_closure_rejected": True, + "incomplete_expansion_gates_rejected": True, + "all_expansion_types_registered_independently": True, + "idempotent_migration": True, + "production_mutations": 0, + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + print( + json.dumps( + { + "status": "PASS", + "closure_state": closure_state, + "final_test_state": final_state, + "counts": counts, + "cleanup": True, + }, + sort_keys=True, + ) + ) + return 0 + finally: + cleanup() + remaining = run( + [ + "docker", + "ps", + "-aq", + "--filter", + f"name=^/{CONTAINER}$", + ], + check=False, + ).stdout.strip() + if remaining: + raise RuntimeError("Phase 9.3 PostgreSQL cleanup failed") + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_fund_phase80_broker_substrate.py b/tests/test_fund_phase80_broker_substrate.py index c8a529e..06bb6ec 100644 --- a/tests/test_fund_phase80_broker_substrate.py +++ b/tests/test_fund_phase80_broker_substrate.py @@ -70,6 +70,8 @@ def test_brokers_are_bounded_pinned_and_have_independent_state(self) -> None: self.assertEqual(service["environment"]["KAFKA_MIN_INSYNC_REPLICAS"], 2) self.assertEqual(service["environment"]["KAFKA_SSL_CLIENT_AUTH"], "required") self.assertEqual(service["mem_limit"], "512m") + self.assertIn("/dev/tcp/127.0.0.1/9092", service["healthcheck"]["test"][1]) + self.assertNotIn("kafka-broker-api-versions", service["healthcheck"]["test"][1]) self.assertIn(f"{name}_data", volumes) redis_service = services["phase8_redis"] @@ -77,6 +79,15 @@ def test_brokers_are_bounded_pinned_and_have_independent_state(self) -> None: self.assertEqual(redis_service["mem_limit"], "64m") self.assertIn("noeviction", redis_service["command"]) + def test_admin_cli_isolated_from_broker_memory_cgroups(self) -> None: + admin = self.compose["services"]["phase8_admin"] + self.assertEqual(admin["profiles"], ["phase8-admin"]) + self.assertEqual(admin["entrypoint"], ["/bin/true"]) + self.assertEqual(admin["mem_limit"], "512m") + harness = (ROOT / "scripts/phase80_broker_certification.py").read_text() + self.assertIn('"phase8_admin"', harness) + self.assertNotIn('"exec",\n "-T",\n "kafka1"', harness) + def test_observability_contract_covers_failure_and_recovery(self) -> None: metrics = set(self.topology["observability"]["metrics"]) alerts = set(self.topology["observability"]["alerts"]) diff --git a/tests/test_phase2_demand_reliability.py b/tests/test_phase2_demand_reliability.py index 89a980a..39f1965 100644 --- a/tests/test_phase2_demand_reliability.py +++ b/tests/test_phase2_demand_reliability.py @@ -10,7 +10,6 @@ from unittest.mock import patch from app.history.topup_coordinator import PreloadTopupBackoff, PreloadTopupCoordinator -from app.stream.async_live_feed import recover_demanded_kline_gap from app.stream.binance_ws import get_usdm_symbols from app.stream.demand_registry import FeedDemand, FeedDemandRegistry from app.stream.supervisor import StreamSupervisor @@ -196,6 +195,7 @@ def test_only_missing_demanded_feed_degrades_health(self): supervisor = StreamSupervisor(startup_grace_seconds=0) shard = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard) + supervisor.mark_message(shard) supervisor.expect_feed("binance_spot_kline", "kline", "BROADUSDT", "1m") broad = supervisor.snapshot(now=supervisor.started_at + 10) @@ -213,6 +213,7 @@ def test_spot_tick_cannot_satisfy_usdm_execution_demand(self): supervisor = StreamSupervisor(startup_grace_seconds=0) shard = supervisor.register_shard("binance_spot_trade", "wss://example") supervisor.mark_connected(shard) + supervisor.mark_message(shard) supervisor.record_publish({ "key": "trade:price:binance_spot:BTCUSDT", "data": {"source": "binance_spot_trade", "event_time": int(time.time() * 1000)}, @@ -227,80 +228,5 @@ def test_spot_tick_cannot_satisfy_usdm_execution_demand(self): self.assertEqual(snapshot["feeds"]["demanded_missing_count"], 1) -class ReconnectGapFillTests(unittest.IsolatedAsyncioTestCase): - async def test_gap_fill_fetches_only_demanded_symbol(self): - class Demands: - async def snapshot(self): - return {"feed_keys": ["kline:binance_usdm:1m:BTCUSDT"]} - - supervisor = StreamSupervisor() - shard = supervisor.register_shard( - "binance_futures_kline", - "wss://fstream.binance.com/stream?streams=btcusdt@kline_1m/ethusdt@kline_1m", - ) - supervisor.shards[shard].last_outage_seconds = 120 - queue = asyncio.Queue() - calls = [] - - def fetch(symbol, interval, limit, start, end, market): - calls.append((symbol, interval, limit, market)) - return { - "data": [[1, "10", "12", "9", "11", "5", 2, "", "", "", "", ""]] - } - - with patch("app.stream.async_live_feed.binance_rest.fetch_klines", side_effect=fetch): - recovered = await recover_demanded_kline_gap( - source="binance_futures_kline", - url=supervisor.shards[shard].url_preview, - interval="1m", - queue=queue, - supervisor=supervisor, - shard_id=shard, - demand_registry=Demands(), - ) - - self.assertEqual(recovered, 1) - self.assertEqual([item[0] for item in calls], ["BTCUSDT"]) - self.assertEqual(supervisor.shards[shard].gap_fill_success_count, 1) - source, event = await queue.get() - self.assertEqual(source, "binance_futures_kline") - self.assertEqual(event["recovery_source"], "BINANCE_REST_GAP_FILL") - - async def test_gap_fill_never_marks_current_open_candle_closed(self): - class Demands: - async def snapshot(self): - return {"feed_keys": ["kline:binance_usdm:1m:BTCUSDT"]} - - now_ms = int(time.time() * 1000) - supervisor = StreamSupervisor() - shard = supervisor.register_shard( - "binance_futures_kline", - "wss://fstream.binance.com/stream?streams=btcusdt@kline_1m", - ) - supervisor.shards[shard].last_outage_seconds = 60 - queue = asyncio.Queue() - closed = [now_ms - 120_000, "1", "2", "0.5", "1.5", "10", now_ms - 60_000] - open_row = [now_ms - 30_000, "1", "2", "0.5", "1.5", "10", now_ms + 30_000] - - with patch( - "app.stream.async_live_feed.binance_rest.fetch_klines", - return_value={"data": [closed, open_row]}, - ): - recovered = await recover_demanded_kline_gap( - source="binance_futures_kline", - url=supervisor.shards[shard].url_preview, - interval="1m", - queue=queue, - supervisor=supervisor, - shard_id=shard, - demand_registry=Demands(), - ) - - self.assertEqual(recovered, 1) - _, event = await queue.get() - self.assertEqual(event["k"]["T"], now_ms - 60_000) - self.assertTrue(event["k"]["x"]) - - if __name__ == "__main__": unittest.main() diff --git a/tests/test_phase90a_runtime_correctness.py b/tests/test_phase90a_runtime_correctness.py new file mode 100644 index 0000000..1698edd --- /dev/null +++ b/tests/test_phase90a_runtime_correctness.py @@ -0,0 +1,380 @@ +from __future__ import annotations + +import asyncio +import json +import unittest +from types import SimpleNamespace +from unittest.mock import patch + +from app.api.routes_health import health +from app.stream.async_live_feed import ( + StreamBackpressureTimeout, + _put_provider_item, + coalesce_redis_items, + handle_ws, + provider_items, + redis_publisher_task, + valid_provider_frame, +) +from app.stream.kline_recovery import DemandKlineRecovery, KlineRecoveryConfig +from app.stream.supervisor import StreamSupervisor + + +class FakeDemandRegistry: + def __init__(self, feed_keys: list[str]): + self.feed_keys = feed_keys + + async def snapshot(self): + return { + "lease_count": len(self.feed_keys), + "demanded_feed_count": len(self.feed_keys), + "feed_keys": list(self.feed_keys), + "items": [{"feed_key": key, "source": "binance_usdm"} for key in self.feed_keys], + } + + +class FakeKlineCache: + def __init__(self, payload=None): + self.payload = payload + + async def get_binance_kline_last(self, symbol: str, interval: str): + return self.payload + + +class RuntimeReadinessTests(unittest.TestCase): + @staticmethod + def trade_frame(): + return { + "e": "trade", + "E": 120, + "s": "BTCUSDT", + "t": 1, + "p": "60000", + "q": "0.1", + "T": 119, + "m": False, + } + + @staticmethod + def kline_frame(): + return { + "e": "kline", + "E": 120, + "s": "BTCUSDT", + "k": { + "s": "BTCUSDT", + "i": "1m", + "t": 60, + "T": 119, + "o": "1", + "h": "2", + "l": "0.5", + "c": "1.5", + "v": "10", + "x": True, + }, + } + + def test_ack_and_wrong_feed_do_not_count_as_provider_data(self): + self.assertEqual(provider_items("binance_futures_kline", {"result": None, "id": 1}), []) + self.assertFalse(valid_provider_frame("binance_futures_kline", self.trade_frame(), "1m")) + with self.assertRaisesRegex(ValueError, "wrong-feed"): + provider_items("binance_futures_kline", self.trade_frame(), "1m") + + def test_trade_and_kline_readiness_are_independent(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=5, stale_after_seconds=180) + trade = supervisor.register_shard("binance_futures_trade", "wss://trade") + kline = supervisor.register_shard("binance_futures_kline", "wss://kline") + supervisor.mark_connected(trade) + supervisor.mark_connected(kline) + supervisor.mark_message(trade) + + snapshot = supervisor.snapshot(now=supervisor.started_at + 10) + + self.assertEqual(snapshot["sources"]["binance_futures_trade"]["status"], "ready") + self.assertEqual(snapshot["sources"]["binance_futures_kline"]["status"], "unavailable") + self.assertEqual(snapshot["status"], "degraded") + self.assertIn("source_data_unavailable", snapshot["health_warnings"]) + + def test_both_valid_sources_become_ready(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=5, stale_after_seconds=180) + for source in ("binance_futures_trade", "binance_futures_kline"): + shard = supervisor.register_shard(source, f"wss://{source}") + supervisor.mark_connected(shard) + supervisor.mark_message(shard) + + snapshot = supervisor.snapshot(now=supervisor.started_at + 10) + + self.assertEqual(snapshot["status"], "ok") + self.assertTrue(all(item["data_ready"] for item in snapshot["sources"].values())) + + + def test_transport_reconnect_cannot_clear_active_data_outage(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=15) + shard = supervisor.register_shard("binance_futures_kline", "wss://kline") + supervisor.mark_connected(shard) + supervisor.mark_data_timeout(shard, "first_frame") + supervisor.mark_reconnect(shard, "first_frame timeout") + supervisor.mark_connected(shard) + + snapshot = supervisor.snapshot() + + self.assertEqual(snapshot["sources"]["binance_futures_kline"]["status"], "unavailable") + self.assertEqual(snapshot["status"], "degraded") + +class FakeWebSocketContext: + def __init__(self, messages: list[str]): + self.messages = list(messages) + + async def __aenter__(self): + return self + + async def __aexit__(self, *_): + return False + + async def recv(self): + if self.messages: + return self.messages.pop(0) + await asyncio.Event().wait() + + +class ReceiverWatchdogTests(unittest.IsolatedAsyncioTestCase): + async def test_connected_without_frame_times_out_and_reconnects(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=0.01) + shard = supervisor.register_shard("binance_futures_kline", "wss://kline") + socket = FakeWebSocketContext([]) + + with patch("app.stream.async_live_feed.websocket_connect", return_value=socket), patch( + "app.stream.async_live_feed.asyncio.sleep", side_effect=asyncio.CancelledError + ): + with self.assertRaises(asyncio.CancelledError): + await handle_ws( + "wss://kline", + asyncio.Queue(), + "binance_futures_kline", + shard, + supervisor, + reconnect_delay=0, + first_frame_timeout_seconds=0.01, + ) + + state = supervisor.shards[shard] + self.assertEqual(state.message_count, 0) + self.assertEqual(state.data_timeout_count, 1) + self.assertEqual(state.status, "reconnecting") + self.assertIn("first_frame timeout", state.last_error) + + async def test_valid_frame_then_idle_timeout_is_observable(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=0.01) + shard = supervisor.register_shard("binance_futures_trade", "wss://trade") + frame = RuntimeReadinessTests.trade_frame() + socket = FakeWebSocketContext([json.dumps({"stream": "btcusdt@trade", "data": frame})]) + queue = asyncio.Queue() + + with patch("app.stream.async_live_feed.websocket_connect", return_value=socket), patch( + "app.stream.async_live_feed.asyncio.sleep", side_effect=asyncio.CancelledError + ): + with self.assertRaises(asyncio.CancelledError): + await handle_ws( + "wss://trade", + queue, + "binance_futures_trade", + shard, + supervisor, + reconnect_delay=0, + first_frame_timeout_seconds=0.01, + idle_timeout_seconds=0.01, + ) + + state = supervisor.shards[shard] + self.assertEqual(state.message_count, 1) + self.assertEqual(state.data_timeout_count, 1) + self.assertEqual((await queue.get())[1]["s"], "BTCUSDT") + self.assertIn("idle timeout", state.last_error) + + +class BackpressureTests(unittest.IsolatedAsyncioTestCase): + async def test_full_queue_never_discards_oldest_event(self): + queue = asyncio.Queue(maxsize=1) + await queue.put(("binance_futures_trade", {"t": "first"})) + supervisor = StreamSupervisor() + + with self.assertRaises(StreamBackpressureTimeout): + await _put_provider_item( + queue, + ("binance_futures_trade", {"t": "second"}), + supervisor=supervisor, + timeout_seconds=0.01, + ) + + self.assertEqual((await queue.get())[1]["t"], "first") + self.assertEqual(supervisor.queue_pressure_count, 1) + self.assertEqual(supervisor.queue_drop_count, 0) + + def test_recovery_gap_rows_are_not_coalesced_together(self): + rows = [ + {"key": "kline:1m:BTCUSDT", "channel": "stream:kline:1m:BTCUSDT", "coalesce_id": "recovery:1", "data": {"k": {"t": 1}}}, + {"key": "kline:1m:BTCUSDT", "channel": "stream:kline:1m:BTCUSDT", "coalesce_id": "recovery:2", "data": {"k": {"t": 2}}}, + ] + self.assertEqual(len(coalesce_redis_items(rows)), 2) + + + async def test_publisher_projects_payload_interval_instead_of_stream_default(self): + class Cache: + def __init__(self): + self.items = [] + + async def push_batch(self, items): + self.items.extend(items) + + queue = asyncio.Queue() + cache = Cache() + await queue.put( + ( + "binance_futures_kline", + { + "e": "kline_recovery", + "s": "BTCUSDT", + "k": {"s": "BTCUSDT", "i": "5m", "t": 300_000, "x": True}, + "recovery_source": "BINANCE_REST_GAP_FILL", + }, + ) + ) + task = asyncio.create_task( + redis_publisher_task(queue, cache, interval="1m", supervisor=StreamSupervisor()) + ) + await asyncio.sleep(0.1) + task.cancel() + await task + + self.assertEqual(cache.items[0]["key"], "kline:5m:BTCUSDT") + self.assertEqual(cache.items[0]["channel"], "stream:kline:5m:BTCUSDT") + +class DemandKlineRecoveryTests(unittest.IsolatedAsyncioTestCase): + @staticmethod + def rows(): + return [ + [0, "1", "2", "0.5", "1.5", "10", 59_999], + [60_000, "1.5", "2.5", "1", "2", "20", 119_999], + [120_000, "2", "3", "1.5", "2.5", "30", 179_999], + ] + + async def test_active_demand_emits_only_latest_fully_closed_bar_on_cold_cache(self): + queue = asyncio.Queue() + demands = FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]) + calls = [] + + def fetcher(*args): + calls.append(args) + return {"data": self.rows()} + + manager = DemandKlineRecovery( + queue=queue, + redis_cache=FakeKlineCache(), + demand_registry=demands, + config=KlineRecoveryConfig(), + fetcher=fetcher, + ) + + result = await manager.poll_once(now_ms=150_000) + + self.assertEqual(result, {"due": 1, "emitted": 1, "failed": 0}) + source, event = await queue.get() + self.assertEqual(source, "binance_futures_kline") + self.assertEqual(event["k"]["t"], 60_000) + self.assertTrue(event["k"]["x"]) + self.assertEqual(event["recovery_source"], "BINANCE_REST_GAP_FILL") + self.assertEqual(len(calls), 1) + self.assertEqual(manager.rejected_open_or_invalid_count, 1) + + async def test_existing_final_bar_is_deduplicated(self): + existing = {"k": {"t": 60_000, "x": True}} + queue = asyncio.Queue() + manager = DemandKlineRecovery( + queue=queue, + redis_cache=FakeKlineCache(existing), + demand_registry=FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]), + config=KlineRecoveryConfig(), + fetcher=lambda *args: {"data": self.rows()[:2]}, + ) + + result = await manager.poll_once(now_ms=150_000) + + self.assertEqual(result["emitted"], 0) + self.assertEqual(manager.deduplicated_count, 2) + self.assertTrue(queue.empty()) + + async def test_no_active_demand_makes_no_provider_request_and_clears_schedule(self): + demands = FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]) + calls = [] + manager = DemandKlineRecovery( + queue=asyncio.Queue(), + redis_cache=FakeKlineCache(), + demand_registry=demands, + config=KlineRecoveryConfig(), + fetcher=lambda *args: calls.append(args) or {"data": self.rows()}, + ) + await manager.poll_once(now_ms=150_000) + demands.feed_keys = [] + + result = await manager.poll_once(now_ms=200_000) + + self.assertEqual(result, {"due": 0, "emitted": 0, "failed": 0}) + self.assertEqual(manager.active_demand_count, 0) + self.assertEqual(len(calls), 1) + self.assertEqual(manager._next_due_ms, {}) + + async def test_provider_failure_is_bounded_by_per_feed_backoff(self): + calls = [] + + def failed(*args): + calls.append(args) + raise RuntimeError("provider unavailable") + + manager = DemandKlineRecovery( + queue=asyncio.Queue(), + redis_cache=FakeKlineCache(), + demand_registry=FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]), + config=KlineRecoveryConfig(poll_seconds=2, max_backoff_seconds=30), + fetcher=failed, + ) + + first = await manager.poll_once(now_ms=150_000) + second = await manager.poll_once(now_ms=151_000) + + self.assertEqual(first["failed"], 1) + self.assertEqual(second["due"], 0) + self.assertEqual(len(calls), 1) + self.assertEqual(manager.snapshot()["status"], "degraded") + + +class HealthContractTests(unittest.IsolatedAsyncioTestCase): + async def test_v1_health_booleans_reflect_matching_source(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=1) + trade = supervisor.register_shard("binance_futures_trade", "wss://trade") + kline = supervisor.register_shard("binance_futures_kline", "wss://kline") + supervisor.mark_connected(trade) + supervisor.mark_connected(kline) + supervisor.mark_message(trade) + supervisor.shards[kline].last_connected_at -= 5 + + context = SimpleNamespace( + redis_cache=SimpleNamespace(health_check=lambda: asyncio.sleep(0, result=True)), + demand_registry=FakeDemandRegistry([]), + binance_stream_supervisor=supervisor, + get_dnse_stream_manager=lambda: None, + get_kline_recovery_manager=lambda: None, + preload_topup_coordinator=SimpleNamespace(snapshot=lambda: {}), + ) + + result = await health(context) + + self.assertEqual(result["status"], "degraded") + self.assertTrue(result["binance_trade_stream"]) + self.assertFalse(result["binance_kline_stream"]) + self.assertIn("binance_stream", result) + self.assertIn("feed_demands", result) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase90b_isolated_beta.py b/tests/test_phase90b_isolated_beta.py new file mode 100644 index 0000000..0913570 --- /dev/null +++ b/tests/test_phase90b_isolated_beta.py @@ -0,0 +1,119 @@ +from __future__ import annotations + +import copy +import unittest +from pathlib import Path + +from scripts.phase90b_bridge_parity import validate_sample, validate_window +from qdl.runtime.canary_source import CanarySourceCatalog + + +class Phase90BBridgeParityTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.binding = CanarySourceCatalog.load( + Path("config/phase7/canary-sources.yaml") + ).bindings[0] + + def sample(self): + row = [ + 1_800_000_000_000, "100.10", "101.20", "99.30", "100.40", + "12.50", 1_800_000_059_999, "1255.00", 42, "6.25", "627.50", + "0", + ] + decimal = lambda value: {"coefficient": "1", "scale": 0, "source_text": value} + item = { + "instrument_uid": self.binding.instrument.instrument_uid, + "instrument_id": self.binding.instrument.instrument_id, + "feed": "BAR", + "interval": "1m", + "payload": { + "feed": "BAR", "interval": "1m", + "open_time_ns": row[0] * 1_000_000, + "close_time_ns": row[6] * 1_000_000, + "open": decimal(row[1]), "high": decimal(row[2]), + "low": decimal(row[3]), "close": decimal(row[4]), + "volume": decimal(row[5]), "trade_count": row[8], + "lifecycle": "FINAL", + }, + "source": { + "source_id": self.binding.source_id, + "source_role": self.binding.source_role, + "authoritative": self.binding.authoritative, + }, + "quality": { + "policy_id": self.binding.source_policy_id, + "complete": True, + "gap_open": False, + "execution_eligible": False, + }, + } + v1 = { + "provider": "binance", "market": "usdm", "symbol": "BTCUSDT", + "requested_interval": "1m", "data": [row], + } + v2 = { + "schema": "qdl.marketdata.warmup.v2", "count": 1, + "watermark_offset": 1, "data": [item], + } + return v1, v2 + + def test_exact_provider_bar_passes(self): + v1, v2 = self.sample() + result = validate_sample(v1, v2, self.binding) + self.assertEqual(result["count"], 1) + validate_window(result, result) + + def test_decimal_mismatch_fails(self): + v1, v2 = self.sample() + v2["data"][0]["payload"]["close"]["source_text"] = "100.41" + with self.assertRaisesRegex(AssertionError, "close"): + validate_sample(v1, v2, self.binding) + + def test_non_final_and_execution_eligible_fail(self): + v1, v2 = self.sample() + v2["data"][0]["payload"]["lifecycle"] = "IN_PROGRESS" + v2["data"][0]["quality"]["execution_eligible"] = True + with self.assertRaisesRegex(AssertionError, "final"): + validate_sample(v1, v2, self.binding) + + def test_duplicate_open_time_fails(self): + v1, v2 = self.sample() + v2["data"].append(copy.deepcopy(v2["data"][0])) + v2["count"] = 2 + v2["watermark_offset"] = 2 + with self.assertRaisesRegex(AssertionError, "duplicated"): + validate_sample(v1, v2, self.binding) + + def test_certification_harness_is_rootless_host_portable(self): + phase73 = Path("scripts/phase73_public_beta_certification.sh").read_text() + self.assertIn('runner_evidence="${temporary}/runner"', phase73) + self.assertIn('OPERATOR_UID="$(id -u)"', phase73) + self.assertIn('--user 0:0 --cap-drop ALL --cap-add CHOWN --cap-add FOWNER', phase73) + self.assertIn('set_evidence_owner 10001 10001', phase73) + self.assertIn('set_evidence_owner "${OPERATOR_UID}" "${OPERATOR_GID}"', phase73) + self.assertNotIn('chown 10001:10001 "${temporary}"', phase73) + + phase90b = Path("scripts/phase90b_isolated_beta_certification.sh").read_text() + self.assertIn('CERT_UID="${QDL_CERT_UID:-$(id -u)}"', phase90b) + self.assertIn('CERT_GID="${QDL_CERT_GID:-$(id -g)}"', phase90b) + self.assertIn( + '--profile phase7-beta --profile phase7-canary ps -q qdl_beta_v1_bridge', + phase90b, + ) + self.assertIn('realpath --relative-to="${ROOT_DIR}"', phase90b) + self.assertIn('(cd "${ROOT_DIR}" && sha256sum -c "${CHECKSUM_OUTPUT}")', phase90b) + self.assertNotIn('chown 10001:10001 "${temporary}"', phase90b) + + def test_window_rejects_regression_and_unbounded_growth(self): + first = {"watermark_offset": 5, "last_open_time_ns": 100} + with self.assertRaisesRegex(AssertionError, "outside"): + validate_window(first, {"watermark_offset": 4, "last_open_time_ns": 100}) + with self.assertRaisesRegex(AssertionError, "outside"): + validate_window(first, {"watermark_offset": 7, "last_open_time_ns": 100}) + with self.assertRaisesRegex(AssertionError, "backwards"): + validate_window(first, {"watermark_offset": 6, "last_open_time_ns": 99}) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase90c_migration_contract.py b/tests/test_phase90c_migration_contract.py new file mode 100644 index 0000000..a32af06 --- /dev/null +++ b/tests/test_phase90c_migration_contract.py @@ -0,0 +1,61 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase90CMigrationContractTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.sql = ( + ROOT / "migrations/postgres/0006_phase9_authority_prerequisites.sql" + ).read_text() + + def test_additive_authority_and_evidence_schema_exists(self): + for name in ( + "qdl_production_prerequisite_bundles", + "qdl_authority_slices", + "qdl_authority_transition_audit", + "qdl_transition_authority", + ): + self.assertIn(name, self.sql) + self.assertNotIn("DROP TABLE", self.sql.upper()) + self.assertNotIn("TRUNCATE", self.sql.upper()) + + def test_cas_and_go_bundle_are_database_enforced(self): + self.assertIn("authority compare-and-swap precondition failed", self.sql) + self.assertIn("p_expected_partition_plan_epoch", self.sql) + self.assertIn("p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY')", self.sql) + self.assertIn("bundle_row.decision <> 'GO'", self.sql) + self.assertIn("bundle_row.candidate_digest <> current_row.candidate_digest", self.sql) + self.assertIn("bundle_row.expires_at <= clock_timestamp()", self.sql) + self.assertIn("bundle_row.expires_at < p_hold_until", self.sql) + self.assertIn("canary/primary requires a non-negative terminal watermark", self.sql) + self.assertIn("canary/primary requires a future approval hold window", self.sql) + self.assertIn("valid only for canary/primary", self.sql) + self.assertIn("THEN p_prerequisite_bundle_id ELSE NULL", self.sql) + + def test_authority_record_binds_release_provenance_and_hold_window(self): + for field in ( + "signature_identity", "normalizer_version", "adapter_version", + "config_revision", "instrument_catalog_revision", + "source_policy_revision", "hold_until", + ): + self.assertIn(field, self.sql) + self.assertIn("p_hold_until", self.sql) + + def test_audit_is_append_only_and_new_owner_needs_new_epoch(self): + self.assertIn("qdl_authority_audit_immutable", self.sql) + self.assertIn("qdl_authority_transition_audit is append-only", self.sql) + self.assertIn("new owner requires a strictly newer lease epoch", self.sql) + + def test_migration_smoke_keeps_heredoc_stdin_open(self): + harness = (ROOT / "scripts/phase90c_migration_smoke.sh").read_text() + self.assertIn('docker exec -i "${CONTAINER}" psql', harness) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase90c_prerequisites.py b/tests/test_phase90c_prerequisites.py new file mode 100644 index 0000000..b1f1a40 --- /dev/null +++ b/tests/test_phase90c_prerequisites.py @@ -0,0 +1,230 @@ +from __future__ import annotations + +import copy +import hashlib +import subprocess +import tempfile +import time +import unittest +from dataclasses import replace +from pathlib import Path + +import yaml + +from qdl.certification.prerequisites import ( + CandidateSlice, + EvidenceRecord, + EvidenceScope, + PrerequisitePolicy, + evaluate_prerequisites, + load_inventory, +) + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase90CPrerequisiteTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.policy = PrerequisitePolicy.load( + ROOT / "config/phase9/production-prerequisite-policy.yaml" + ) + cls.candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + + def test_local_inventory_is_honest_no_go(self): + inventory = load_inventory(ROOT / "config/phase9/evidence-inventory.local.yaml") + decision = evaluate_prerequisites( + self.policy, self.candidate, inventory, + repository_root=ROOT, now_ns=1_787_063_420_299_706_631, + ) + self.assertEqual(decision.decision, "NO_GO_EXTERNAL") + results = {item.gate_id: item for item in decision.results} + self.assertEqual(results["replicated_durable_transport"].reason, "INSUFFICIENT_SCOPE") + self.assertEqual(results["signed_artifact_admission"].reason, "EVIDENCE_BLOCKED") + self.assertEqual(results["independent_failure_domain_dr"].reason, "MISSING_EVIDENCE") + self.assertFalse(any(item.passed for item in decision.results)) + + def _passing_inventory(self, root: Path, now_ns: int): + artifact = root / "evidence.json" + artifact.write_text("{}\n") + digest = hashlib.sha256(artifact.read_bytes()).hexdigest() + records = [] + for gate in self.policy.gates: + details = {key: f"proof-{key}" for key in gate.required_details} + details.update(gate.assertions) + details.update(gate.minimums) + details.update(gate.maximums) + details.update({ + detail_key: self.candidate.payload[candidate_key] + for detail_key, candidate_key in gate.candidate_field_matches.items() + }) + if gate.candidate_bound: + details["candidate_digest"] = self.candidate.digest + if gate.gate_id == "exact_slice_approval": + details["hold_until_ns"] = now_ns + 30_000_000_000 + records.append(EvidenceRecord.from_mapping({ + "evidence_id": f"evidence-{gate.gate_id}", + "gate_id": gate.gate_id, + "environment": "production", + "scope": gate.minimum_scope.value, + "status": "PASS", + "issuer": "production-control-plane", + "observed_at_ns": now_ns - 1_000_000, + "expires_at_ns": now_ns + 60_000_000_000, + "artifact_path": "evidence.json", + "artifact_sha256": digest, + "details": details, + })) + return tuple(records), artifact + + def test_all_exact_production_evidence_is_go(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + records, _ = self._passing_inventory(Path(directory), now_ns) + decision = evaluate_prerequisites( + self.policy, self.candidate, records, + repository_root=directory, now_ns=now_ns, + ) + self.assertEqual(decision.decision, "GO") + self.assertTrue(all(item.passed for item in decision.results)) + + def test_tampered_expired_and_candidate_mismatch_fail_closed(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + records, artifact = self._passing_inventory(root, now_ns) + artifact.write_text("tampered\n") + decision = evaluate_prerequisites( + self.policy, self.candidate, records, + repository_root=root, now_ns=now_ns, + ) + self.assertTrue(all(item.reason == "ARTIFACT_DIGEST_MISMATCH" for item in decision.results)) + + records, _ = self._passing_inventory(root, now_ns) + first = records[0] + expired = EvidenceRecord(**{**first.__dict__, "expires_at_ns": now_ns - 1}) + decision = evaluate_prerequisites( + self.policy, self.candidate, (expired, *records[1:]), + repository_root=root, now_ns=now_ns, + ) + self.assertEqual(decision.results[0].reason, "EVIDENCE_EXPIRED") + + bound_index = next(i for i, gate in enumerate(self.policy.gates) if gate.candidate_bound) + bound = records[bound_index] + wrong = EvidenceRecord(**{ + **bound.__dict__, + "details": {**bound.details, "candidate_digest": "0" * 64}, + }) + changed = list(records); changed[bound_index] = wrong + decision = evaluate_prerequisites( + self.policy, self.candidate, tuple(changed), + repository_root=root, now_ns=now_ns, + ) + self.assertEqual(decision.results[bound_index].reason, "CANDIDATE_DIGEST_MISMATCH") + + def test_unknown_sensitive_duplicate_and_unsafe_candidate_are_rejected(self): + raw = yaml.safe_load((ROOT / "config/phase9/evidence-inventory.local.yaml").read_text()) + item = copy.deepcopy(raw["evidence"][0]) + item["details"]["private_key"] = "must-not-enter-evidence" + with self.assertRaisesRegex(ValueError, "sensitive"): + EvidenceRecord.from_mapping(item) + item["details"].pop("private_key") + item["details"]["nested"] = {"access_token": "must-not-enter-evidence"} + with self.assertRaisesRegex(ValueError, "sensitive"): + EvidenceRecord.from_mapping(item) + + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "inventory.yaml" + raw["evidence"].append(copy.deepcopy(raw["evidence"][0])) + path.write_text(yaml.safe_dump(raw)) + with self.assertRaisesRegex(ValueError, "unique"): + load_inventory(path) + + candidate = yaml.safe_load((ROOT / "config/phase9/candidate-slice.yaml").read_text()) + candidate["public_write_allowed"] = True + with tempfile.NamedTemporaryFile("w", suffix=".yaml") as handle: + yaml.safe_dump(candidate, handle); handle.flush() + with self.assertRaisesRegex(ValueError, "cannot write"): + CandidateSlice.load(handle.name) + + candidate = yaml.safe_load((ROOT / "config/phase9/candidate-slice.yaml").read_text()) + del candidate["normalizer_version"] + with tempfile.NamedTemporaryFile("w", suffix=".yaml") as handle: + yaml.safe_dump(candidate, handle); handle.flush() + with self.assertRaisesRegex(ValueError, "incomplete"): + CandidateSlice.load(handle.name) + + candidate = yaml.safe_load((ROOT / "config/phase9/candidate-slice.yaml").read_text()) + candidate["instrument_uids"] = ["not-a-uuid"] + with tempfile.NamedTemporaryFile("w", suffix=".yaml") as handle: + yaml.safe_dump(candidate, handle); handle.flush() + with self.assertRaisesRegex(ValueError, "UID is invalid"): + CandidateSlice.load(handle.name) + + def test_semantic_assertion_threshold_candidate_and_hold_fail_closed(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + records, _ = self._passing_inventory(root, now_ns) + indexes = {item.gate_id: index for index, item in enumerate(records)} + + cases = ( + ("replicated_durable_transport", {"acks_all": False}, "ASSERTION_FAILED"), + ("replicated_durable_transport", {"broker_count": 2}, "MINIMUM_NOT_MET"), + ("replicated_durable_transport", {"broker_count": True}, "MINIMUM_NOT_MET"), + ("postgres_pitr", {"rpo_seconds": 301}, "MAXIMUM_EXCEEDED"), + ("signed_artifact_admission", {"image_digest": "sha256:" + "0" * 64}, "CANDIDATE_FIELD_MISMATCH"), + ("exact_slice_approval", {"hold_until_ns": now_ns}, "APPROVAL_HOLD_WINDOW_INVALID"), + ) + for gate_id, detail_update, expected in cases: + changed = list(records) + index = indexes[gate_id] + changed[index] = replace( + changed[index], details={**changed[index].details, **detail_update} + ) + decision = evaluate_prerequisites( + self.policy, self.candidate, tuple(changed), + repository_root=root, now_ns=now_ns, + ) + self.assertEqual(decision.results[index].reason, expected, gate_id) + + def test_unknown_evidence_gate_is_rejected(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + records, _ = self._passing_inventory(Path(directory), now_ns) + unknown = replace(records[0], evidence_id="unknown", gate_id="unknown_gate") + with self.assertRaisesRegex(ValueError, "unknown prerequisite"): + evaluate_prerequisites( + self.policy, self.candidate, (*records, unknown), + repository_root=directory, now_ns=now_ns, + ) + + def test_policy_is_complete_and_provider_neutral(self): + ids = {gate.gate_id for gate in self.policy.gates} + self.assertEqual(len(ids), 12) + self.assertTrue({ + "replicated_durable_transport", "production_observability", + "workload_identity_rbac_network", "external_secret_rotation", + "signed_artifact_admission", "postgres_pitr", "object_store_restore", + "independent_failure_domain_dr", "redis_projector_rebuild", + "consumer_registration_rollback", "persistent_authority_sink_fencing", + "exact_slice_approval", + }.issubset(ids)) + policy_text = (ROOT / "config/phase9/production-prerequisite-policy.yaml").read_text() + self.assertNotIn("BINANCE", policy_text) + self.assertEqual( + next(g.minimum_scope for g in self.policy.gates if g.gate_id == "independent_failure_domain_dr"), + EvidenceScope.INDEPENDENT_FAILURE_DOMAIN, + ) + + def test_operator_entrypoint_imports_from_repository_root(self): + completed = subprocess.run( + ["python3", "scripts/phase90c_prerequisite_certification.py", "--help"], + cwd=ROOT, text=True, capture_output=True, check=False, + ) + self.assertEqual(completed.returncode, 0, completed.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase91_canary_control.py b/tests/test_phase91_canary_control.py new file mode 100644 index 0000000..0698465 --- /dev/null +++ b/tests/test_phase91_canary_control.py @@ -0,0 +1,198 @@ +from __future__ import annotations + +import copy +import json +import time +import unittest +from pathlib import Path + +from qdl.canary.phase9 import ( + CanaryAuthorizationMode, + CanaryGuardrailEngine, + CanaryGuardrailPolicy, + CanaryObservation, + ProductionCanaryAuthorizer, +) +from qdl.certification.prerequisites import CandidateSlice + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase91CanaryAuthorizationTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + cls.no_go = json.loads( + (ROOT / "upgrade/evidence/phase90c-production-prerequisites.json").read_text() + ) + + def _go(self, now_ns: int) -> dict: + payload = copy.deepcopy(self.no_go) + payload.update({ + "decision": "GO", + "passed": len(payload["gates"]), + "blocked": 0, + "issued_at_ns": now_ns - 1_000_000, + "authority_state": "RUST_SHADOW", + "v1_unchanged": True, + "production_mutations": 0, + }) + for gate in payload["gates"]: + gate.update({"passed": True, "reason": "PASS"}) + return payload + + def test_current_external_no_go_cannot_authorize_production(self): + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, + decision=self.no_go, + expected_bundle_id=self.no_go["bundle_id"], + now_ns=self.no_go["issued_at_ns"] + 1, + ) + self.assertFalse(result.allowed) + self.assertFalse(result.production_authorized) + self.assertEqual(result.reason, "PREREQUISITE_DECISION_NOT_GO") + + def test_rehearsal_is_explicitly_non_production(self): + result = ProductionCanaryAuthorizer.authorize_isolated_rehearsal( + candidate=self.candidate + ) + self.assertTrue(result.allowed) + self.assertFalse(result.production_authorized) + self.assertEqual(result.mode, CanaryAuthorizationMode.ISOLATED_REHEARSAL) + self.assertEqual(result.reason, "ISOLATED_REHEARSAL_ONLY") + + def test_exact_fresh_go_authorizes_without_mutating(self): + now_ns = time.time_ns() + decision = self._go(now_ns) + snapshot = copy.deepcopy(decision) + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, + decision=decision, + expected_bundle_id=decision["bundle_id"], + now_ns=now_ns, + ) + self.assertTrue(result.allowed) + self.assertTrue(result.production_authorized) + self.assertEqual(result.reason, "AUTHORIZED") + self.assertEqual(decision, snapshot) + + def test_identity_freshness_gate_and_v1_preconditions_fail_closed(self): + now_ns = time.time_ns() + cases = ( + ({"candidate_digest": "0" * 64}, "CANDIDATE_DIGEST_MISMATCH"), + ({"slice_id": "production/other"}, "SLICE_MISMATCH"), + ({"bundle_id": "11111111-1111-4111-8111-111111111111"}, "PREREQUISITE_BUNDLE_MISMATCH"), + ({"issued_at_ns": now_ns - 901_000_000_000}, "DECISION_EXPIRED"), + ({"authority_state": "RUST_CANARY"}, "AUTHORITY_PRECONDITION_INVALID"), + ({"v1_unchanged": False}, "V1_PRECONDITION_INVALID"), + ({"production_mutations": 1}, "PRODUCTION_MUTATION_DETECTED"), + ) + for update, expected in cases: + with self.subTest(expected=expected): + decision = self._go(now_ns) + expected_bundle = decision["bundle_id"] + decision.update(update) + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, + decision=decision, + expected_bundle_id=expected_bundle, + now_ns=now_ns, + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + def test_incomplete_duplicate_or_failed_gates_are_rejected(self): + now_ns = time.time_ns() + decision = self._go(now_ns) + expected_bundle = decision["bundle_id"] + decision["gates"].pop() + decision["passed"] -= 1 + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, decision=decision, + expected_bundle_id=expected_bundle, now_ns=now_ns, + ) + self.assertEqual(result.reason, "GATE_RESULTS_INVALID") + + decision = self._go(now_ns) + decision["gates"][0]["passed"] = False + decision["gates"][0]["reason"] = "FAILED" + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, decision=decision, + expected_bundle_id=expected_bundle, now_ns=now_ns, + ) + self.assertEqual(result.reason, "GATE_NOT_PASSED") + + +class Phase91GuardrailTest(unittest.TestCase): + def setUp(self): + self.policy = CanaryGuardrailPolicy( + max_lag_ms=500, + max_freshness_ms=1_000, + max_cpu_percent=80.0, + max_rss_mb=512.0, + max_queue_depth=1_000, + hold_down_seconds=30, + ) + + def test_clean_observation_passes(self): + result = CanaryGuardrailEngine(self.policy).evaluate( + CanaryObservation(observed_at_ns=1) + ) + self.assertTrue(result.allowed) + self.assertEqual(result.reason, "PASS") + + def test_each_correctness_and_capacity_guardrail_blocks(self): + cases = ( + ({"semantic_mismatches": 1}, "SEMANTIC_MISMATCH"), + ({"open_gaps": 1}, "OPEN_GAP"), + ({"duplicate_external_writes": 1}, "DUPLICATE_EXTERNAL_WRITE"), + ({"stale_writer_attempts": 1}, "STALE_WRITER_ATTEMPT"), + ({"authority_ambiguities": 1}, "AUTHORITY_AMBIGUITY"), + ({"durable_ack_failures": 1}, "DURABLE_ACK_FAILURE"), + ({"lag_ms": 501}, "LAG_THRESHOLD_EXCEEDED"), + ({"freshness_ms": 1_001}, "FRESHNESS_THRESHOLD_EXCEEDED"), + ({"cpu_percent": 80.1}, "CPU_THRESHOLD_EXCEEDED"), + ({"rss_mb": 512.1}, "RSS_THRESHOLD_EXCEEDED"), + ({"queue_depth": 1_001}, "QUEUE_THRESHOLD_EXCEEDED"), + ) + for update, expected in cases: + with self.subTest(expected=expected): + result = CanaryGuardrailEngine(self.policy).evaluate( + CanaryObservation(observed_at_ns=1, **update) + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + def test_first_failure_is_sticky_and_reentry_is_explicit_after_hold(self): + engine = CanaryGuardrailEngine(self.policy) + blocked = engine.evaluate( + CanaryObservation(observed_at_ns=1_000, semantic_mismatches=1) + ) + self.assertEqual(blocked.reason, "SEMANTIC_MISMATCH") + self.assertEqual(blocked.hold_until_ns, 30_000_001_000) + self.assertEqual( + engine.evaluate(CanaryObservation(observed_at_ns=2_000)).reason, + "EXPLICIT_RESET_REQUIRED", + ) + self.assertEqual( + engine.reset_after_hold(CanaryObservation(observed_at_ns=30_000_000_999)).reason, + "HOLD_DOWN_ACTIVE", + ) + reset = engine.reset_after_hold( + CanaryObservation(observed_at_ns=30_000_001_000) + ) + self.assertTrue(reset.allowed) + self.assertEqual(reset.reason, "RESET_CONFIRMED") + + def test_invalid_thresholds_and_observations_are_rejected(self): + with self.assertRaises(ValueError): + CanaryGuardrailPolicy(0, 1, 1, 1, 1, 1) + with self.assertRaises(ValueError): + CanaryObservation(observed_at_ns=0) + with self.assertRaises(ValueError): + CanaryObservation(observed_at_ns=1, open_gaps=-1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase91_certification_contract.py b/tests/test_phase91_certification_contract.py new file mode 100644 index 0000000..fe124cc --- /dev/null +++ b/tests/test_phase91_certification_contract.py @@ -0,0 +1,116 @@ +from __future__ import annotations + +import gzip +import hashlib +import json +import subprocess +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase91CertificationContractTest(unittest.TestCase): + def test_frozen_capture_is_authentic_read_only_and_checksummed(self): + capture = ROOT / "upgrade/evidence/captures/phase8-real-provider-frames.json.gz" + evidence = json.loads( + (ROOT / "upgrade/evidence/phase8-real-provider-shadow.json").read_text() + ) + compressed = capture.read_bytes() + self.assertEqual( + hashlib.sha256(compressed).hexdigest(), evidence["capture_bundle_sha256"] + ) + payload = json.loads(gzip.decompress(compressed)) + self.assertEqual(payload["provenance"], "REAL_PROVIDER_READ_ONLY") + self.assertEqual(payload["production_writes"], 0) + binance = [ + item for item in payload["captures"] + if item["venue"] == "BINANCE" + and item["market"] == "USDM" + and item["native_symbol"] == "BTCUSDT" + and item["test_provenance"] is False + ] + self.assertGreaterEqual(len(binance), 32) + + def test_tls_generator_preserves_host_ownership(self): + script = (ROOT / "scripts/phase80_generate_tls.sh").read_text() + self.assertIn("QDL_PHASE8_CERT_UID", script) + self.assertIn("QDL_PHASE8_CERT_GID", script) + self.assertIn('--user "${CERT_UID}:${CERT_GID}"', script) + self.assertNotIn("docker run --rm --user 0:0", script) + + def test_runtime_image_and_workspace_declare_phase91_binary(self): + cargo = (ROOT / "rust/qdl-kafka/Cargo.toml").read_text() + dockerfile = (ROOT / "Dockerfile.phase8-rust").read_text() + self.assertIn('name = "qdl-phase91-canary-rehearsal"', cargo) + self.assertIn("--bin qdl-phase91-canary-rehearsal", dockerfile) + self.assertIn( + "/usr/local/bin/qdl-phase91-canary-rehearsal", dockerfile + ) + + def test_certification_is_isolated_and_has_cleanup_and_zero_write_gates(self): + harness = (ROOT / "scripts/phase91_canary_certification.py").read_text() + runtime = ( + ROOT / "rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs" + ).read_text() + for token in ( + "NO_GO_EXTERNAL", + "production_mutations", + "v1_topology_unchanged", + "containers_after", + "public_writes", + "legacy_writes", + "min_isr_failed_closed", + "slow_consumer", + ): + self.assertIn(token, harness) + for token in ( + "topic_masquerade_rejected", + "stale_owner_rejected", + "stale_revision_rejected", + "stale_lease_rejected", + "wrong_plan_rejected", + "duplicate_watermark_rejected", + "blocked_state_rejected", + "canary_after_rollback_rejected", + ): + self.assertIn(token, runtime) + + def test_compacted_authority_reads_latest_record_only(self): + harness = (ROOT / "scripts/phase91_canary_certification.py").read_text() + self.assertIn("def consume_compacted_records", harness) + self.assertIn("--partition", harness) + self.assertIn("item.strip().startswith(\"{\")", harness) + self.assertIn("consume_compacted_records(env, topics[\"authority\"])", harness) + self.assertIn('topics["audit"], 4', harness) + self.assertIn("compose(env, \"stop\", \"kafka3\")", harness) + self.assertIn("compose(env, \"stop\", \"kafka2\")", harness) + self.assertIn("compose(env, \"start\", \"kafka2\", \"kafka3\"", harness) + self.assertIn("\"QDL_KAFKA_BOOTSTRAP_SERVERS\": \"kafka1:9092,kafka2:9092\"", harness) + self.assertIn("min_isr_before = one_loss_after", harness) + self.assertIn("min_isr_client_failed = min_isr.returncode != 0", harness) + self.assertIn("min_isr_after = total_end_offset", harness) + self.assertIn("min_isr_after == min_isr_before", harness) + + def test_makefile_and_evidence_packaging_are_reproducible(self): + makefile = (ROOT / "Makefile").read_text() + harness = (ROOT / "scripts/phase91_canary_certification.py").read_text() + for target in ("phase91-build:", "phase91-test:", "phase91-certify:", "phase91-clean:"): + self.assertIn(target, makefile) + self.assertIn("PHASE91_RUST_IMAGE", makefile) + self.assertIn("def render_report", harness) + self.assertIn("def write_evidence", harness) + self.assertIn("phase91-evidence.sha256", harness) + + def test_operator_entrypoints_have_valid_help_or_usage(self): + result = subprocess.run( + ["python3", "scripts/phase91_canary_certification.py", "--help"], + cwd=ROOT, text=True, capture_output=True, check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("--rust-image", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase92_certification_contract.py b/tests/test_phase92_certification_contract.py new file mode 100644 index 0000000..d72cbe1 --- /dev/null +++ b/tests/test_phase92_certification_contract.py @@ -0,0 +1,90 @@ +from __future__ import annotations + +import subprocess +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase92CertificationContractTest(unittest.TestCase): + def test_runtime_image_declares_primary_rehearsal_binary(self): + cargo = (ROOT / "rust/qdl-kafka/Cargo.toml").read_text() + dockerfile = (ROOT / "Dockerfile.phase8-rust").read_text() + self.assertIn('name = "qdl-phase92-primary-rehearsal"', cargo) + self.assertIn("--bin qdl-phase92-primary-rehearsal", dockerfile) + self.assertIn( + "/usr/local/bin/qdl-phase92-primary-rehearsal", dockerfile + ) + + def test_harness_is_isolated_and_checks_handoff_recovery(self): + harness = (ROOT / "scripts/phase92_primary_certification.py").read_text() + runtime = ( + ROOT / "rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs" + ).read_text() + for token in ( + "NO_GO_EXTERNAL", + "production_mutations", + "v1_topology_unchanged", + "production_public_writes", + "production_legacy_writes", + "projection_parity", + "boundary_gap_free", + "one_replica_loss_acked", + "min_isr_failed_closed", + "process_restart_recovery", + "RECOVERY_VERIFY", + "QDL_RECOVERY_LAST_WATERMARK", + ): + self.assertIn(token, harness) + for token in ( + "direct_primary_without_handoff_rejected", + "terminal_watermark_rejected", + "gap_watermark_rejected", + "stale_owner_rejected", + "stale_revision_rejected", + "stale_lease_rejected", + "wrong_plan_rejected", + "duplicate_after_ack_rejected", + "rust_after_rollback_rejected", + "restart_pre_restore_failed_closed", + "each_target_pre_restore_failed_closed", + "durable_target_watermarks_restored", + "duplicate_after_restore_rejected", + "resumed_at_exact_next_watermark", + "restore_committed_watermark", + ): + self.assertIn(token, runtime) + + def test_rehearsal_topics_cannot_masquerade_as_production(self): + runtime = ( + ROOT / "rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs" + ).read_text() + self.assertIn('!topic.contains(".phase92.")', runtime) + self.assertIn("QDL_PRODUCTION_PUBLIC_TOPIC", runtime) + self.assertIn("QDL_PRODUCTION_LEGACY_TOPIC", runtime) + + def test_makefile_and_operator_entrypoints_are_reproducible(self): + makefile = (ROOT / "Makefile").read_text() + for target in ( + "phase92-build:", + "phase92-test:", + "phase92-migration:", + "phase92-certify:", + "phase92-clean:", + ): + self.assertIn(target, makefile) + result = subprocess.run( + ["python3", "scripts/phase92_primary_certification.py", "--help"], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("--rust-image", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase92_migration_contract.py b/tests/test_phase92_migration_contract.py new file mode 100644 index 0000000..88466cb --- /dev/null +++ b/tests/test_phase92_migration_contract.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase92MigrationContractTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.sql = ( + ROOT / "migrations/postgres/0007_phase92_terminal_handoff.sql" + ).read_text() + + def test_migration_is_additive_and_evidence_is_immutable(self): + for token in ( + "qdl_terminal_owner_checkpoints", + "qdl_authority_handoffs", + "qdl_transition_authority_v2", + "qdl_authority_primary_handoff_guard", + "Phase 9.2 checkpoint/handoff evidence is append-only", + ): + self.assertIn(token, self.sql) + self.assertNotIn("DROP TABLE", self.sql.upper()) + self.assertNotIn("TRUNCATE", self.sql.upper()) + + def test_terminal_boundary_and_clean_reconciliation_are_enforced(self): + for token in ( + "first_new_watermark = terminal_watermark + 1", + "semantic_mismatches = 0", + "open_gaps = 0", + "old_event_count = new_event_count", + "new_authority_revision = expected_authority_revision + 1", + "new_lease_epoch > expected_lease_epoch", + ): + self.assertIn(token, self.sql) + + def test_direct_primary_and_rollback_bypass_are_trigger_fenced(self): + self.assertIn("NEW.state = 'RUST_PRIMARY'", self.sql) + self.assertIn("NEW.state = 'PYTHON_PRIMARY'", self.sql) + self.assertIn("matching_handoffs <> 1", self.sql) + self.assertIn("accepted exact Phase 9.2 handoff is required", self.sql) + + def test_v2_transition_rechecks_exact_cas_and_expiry(self): + for token in ( + "handoff.expected_state <> p_expected_state", + "handoff.expected_authority_revision <> p_expected_revision", + "handoff.expected_lease_epoch <> p_expected_lease_epoch", + "handoff.partition_plan_epoch <> p_expected_partition_plan_epoch", + "handoff.expires_at <= clock_timestamp()", + "Phase 9.2 prerequisite bundle mismatch", + ): + self.assertIn(token, self.sql) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase92_primary_control.py b/tests/test_phase92_primary_control.py new file mode 100644 index 0000000..bd8d2fe --- /dev/null +++ b/tests/test_phase92_primary_control.py @@ -0,0 +1,223 @@ +from __future__ import annotations + +import copy +import json +import time +import unittest +import uuid +from pathlib import Path + +from qdl.canary.phase92 import ( + AcceptedHandoff, + HandoffDirection, + PrimaryAuthorizationMode, + ProductionPrimaryAuthorizer, + TerminalOwnerCheckpoint, +) +from qdl.certification.prerequisites import CandidateSlice + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase92PrimaryAuthorizationTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + cls.no_go = json.loads( + (ROOT / "upgrade/evidence/phase90c-production-prerequisites.json").read_text() + ) + + def _go(self, now_ns: int) -> dict: + payload = copy.deepcopy(self.no_go) + payload.update({ + "decision": "GO", + "passed": len(payload["gates"]), + "blocked": 0, + "issued_at_ns": now_ns - 1_000_000, + "authority_state": "RUST_SHADOW", + "v1_unchanged": True, + "production_mutations": 0, + }) + for gate in payload["gates"]: + gate.update({"passed": True, "reason": "PASS"}) + return payload + + def _canary(self, now_ns: int) -> dict: + return { + "schema": "qdl.phase91.rust-canary-certification.v1", + "status": "PRODUCTION_CANARY_HOLD_PASSED", + "production_authorized": True, + "slice_id": self.candidate.payload["slice_id"], + "candidate_digest": self.candidate.digest, + "prerequisite_bundle_id": self.no_go["bundle_id"], + "python_v1_public_authority_unchanged": True, + "production_mutations": 0, + "parity": {"semantic_mismatches": 0}, + "broker": { + "final_authority": "RUST_CANARY", + "public_writes": 0, + "legacy_writes": 0, + }, + "hold_completed_at_ns": now_ns - 1, + "rollback_manifest_digest": "1" * 64, + } + + def _approval(self, now_ns: int) -> dict: + return { + "schema": "qdl.primary-slice-approval.v1", + "decision": "APPROVE", + "slice_id": self.candidate.payload["slice_id"], + "candidate_digest": self.candidate.digest, + "prerequisite_bundle_id": self.no_go["bundle_id"], + "operator": "phase92-test-operator", + "change_ticket": "QDL-92", + "max_partitions": 1, + "allow_disable_exact_python_slice": True, + "expires_at_ns": now_ns + 60_000_000_000, + } + + def test_current_no_go_cannot_authorize_primary(self): + now_ns = self.no_go["issued_at_ns"] + 1 + result = ProductionPrimaryAuthorizer().authorize( + candidate=self.candidate, + prerequisite_decision=self.no_go, + canary_evidence={}, + approval={}, + expected_bundle_id=self.no_go["bundle_id"], + now_ns=now_ns, + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, "PREREQUISITE_DECISION_NOT_GO") + + def test_rehearsal_is_explicitly_non_production(self): + result = ProductionPrimaryAuthorizer.authorize_isolated_rehearsal( + candidate=self.candidate + ) + self.assertTrue(result.allowed) + self.assertFalse(result.production_authorized) + self.assertEqual(result.mode, PrimaryAuthorizationMode.ISOLATED_REHEARSAL) + + def test_exact_go_canary_hold_and_approval_authorize(self): + now_ns = time.time_ns() + result = ProductionPrimaryAuthorizer().authorize( + candidate=self.candidate, + prerequisite_decision=self._go(now_ns), + canary_evidence=self._canary(now_ns), + approval=self._approval(now_ns), + expected_bundle_id=self.no_go["bundle_id"], + now_ns=now_ns, + ) + self.assertTrue(result.allowed) + self.assertTrue(result.production_authorized) + self.assertEqual(result.reason, "AUTHORIZED") + + def test_canary_and_approval_fail_closed(self): + now_ns = time.time_ns() + cases = ( + ("canary", {"status": "COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED"}, "CANARY_HOLD_NOT_PASSED"), + ("canary", {"production_authorized": False}, "CANARY_NOT_PRODUCTION_AUTHORIZED"), + ("canary", {"hold_completed_at_ns": now_ns + 1}, "CANARY_HOLD_TIME_INVALID"), + ("canary", {"rollback_manifest_digest": "bad"}, "CANARY_ROLLBACK_MANIFEST_INVALID"), + ("approval", {"max_partitions": 2}, "PRIMARY_BLAST_RADIUS_INVALID"), + ("approval", {"allow_disable_exact_python_slice": False}, "PRIMARY_PYTHON_HANDOFF_NOT_APPROVED"), + ("approval", {"expires_at_ns": now_ns}, "PRIMARY_APPROVAL_EXPIRED"), + ) + for target, update, expected in cases: + with self.subTest(expected=expected): + canary = self._canary(now_ns) + approval = self._approval(now_ns) + (canary if target == "canary" else approval).update(update) + result = ProductionPrimaryAuthorizer().authorize( + candidate=self.candidate, + prerequisite_decision=self._go(now_ns), + canary_evidence=canary, + approval=approval, + expected_bundle_id=self.no_go["bundle_id"], + now_ns=now_ns, + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + +class Phase92HandoffContractTest(unittest.TestCase): + def checkpoint(self) -> TerminalOwnerCheckpoint: + return TerminalOwnerCheckpoint( + schema="qdl.terminal-owner-checkpoint.v1", + checkpoint_id=str(uuid.uuid4()), + slice_id="production/binance/usdm/perpetual/trade/plan-1/btcusdt", + owner_id="python-primary", + authority_revision=7, + lease_epoch=11, + partition_plan_epoch=1, + source_session_id="session-1", + connection_generation=2, + terminal_watermark=100, + terminal_event_id="event-100", + terminal_payload_sha256="1" * 64, + candidate_digest="2" * 64, + committed_at_ns=1, + ) + + def handoff(self, checkpoint: TerminalOwnerCheckpoint) -> AcceptedHandoff: + return AcceptedHandoff( + schema="qdl.accepted-authority-handoff.v1", + handoff_id=str(uuid.uuid4()), + direction=HandoffDirection.PYTHON_TO_RUST, + checkpoint_digest=checkpoint.digest, + slice_id=checkpoint.slice_id, + old_owner_id=checkpoint.owner_id, + new_owner_id="rust-primary", + expected_state="RUST_CANARY", + new_state="RUST_PRIMARY", + expected_authority_revision=checkpoint.authority_revision, + new_authority_revision=checkpoint.authority_revision + 1, + expected_lease_epoch=checkpoint.lease_epoch, + new_lease_epoch=checkpoint.lease_epoch + 1, + partition_plan_epoch=checkpoint.partition_plan_epoch, + terminal_watermark=checkpoint.terminal_watermark, + first_new_watermark=checkpoint.terminal_watermark + 1, + overlap_start_watermark=90, + overlap_end_watermark=checkpoint.terminal_watermark, + old_event_count=11, + new_event_count=11, + semantic_mismatches=0, + open_gaps=0, + candidate_digest=checkpoint.candidate_digest, + prerequisite_bundle_id=str(uuid.uuid4()), + approved_by="phase92-test", + approved_at_ns=1, + expires_at_ns=2, + ) + + def test_checkpoint_and_handoff_are_deterministic(self): + checkpoint = self.checkpoint() + self.assertEqual(len(checkpoint.digest), 64) + handoff = self.handoff(checkpoint) + self.assertEqual(len(handoff.digest), 64) + self.assertEqual(handoff.first_new_watermark, 101) + + def test_off_by_one_dirty_parity_and_stale_epoch_are_rejected(self): + checkpoint = self.checkpoint() + base = self.handoff(checkpoint) + payload = { + field: getattr(base, field) + for field in base.__dataclass_fields__ + } + for update in ( + {"first_new_watermark": 102}, + {"semantic_mismatches": 1}, + {"open_gaps": 1}, + {"new_event_count": 10}, + {"new_lease_epoch": base.expected_lease_epoch}, + {"new_authority_revision": base.expected_authority_revision + 2}, + ): + with self.subTest(update=update): + candidate = dict(payload) + candidate.update(update) + with self.assertRaises(ValueError): + AcceptedHandoff(**candidate) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase93_certification_contract.py b/tests/test_phase93_certification_contract.py new file mode 100644 index 0000000..e70a423 --- /dev/null +++ b/tests/test_phase93_certification_contract.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import json +import subprocess +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase93CertificationContractTest(unittest.TestCase): + def test_certification_is_fail_closed_and_never_claims_production(self): + source = ( + ROOT / "scripts/phase93_hold_close_certification.py" + ).read_text() + for token in ( + "NO_GO_EXTERNAL", + "PREREQUISITE_DECISION_NOT_GO", + "TEST_CONTROL_PLANE_FIXTURE", + "accelerated_time_is_production_evidence", + "production_hold_started", + "production_rollback_window_closed", + "production_expansions_authorized", + "production_mutations", + "v1_topology", + "v1_health", + ): + self.assertIn(token, source) + self.assertNotIn("qdl_transition_authority", source) + self.assertNotIn("qdl_close_authority_window", source) + + def test_frozen_evidence_reports_only_control_plane_completion(self): + evidence = json.loads( + ( + ROOT + / "upgrade/evidence/phase93-hold-close-expand-certification.json" + ).read_text() + ) + self.assertEqual( + evidence["status"], + "COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED", + ) + self.assertFalse(evidence["production_authorized"]) + self.assertFalse(evidence["production_hold_started"]) + self.assertFalse(evidence["production_rollback_window_closed"]) + self.assertEqual(evidence["production_expansions_authorized"], 0) + self.assertEqual(evidence["production_mutations"], 0) + self.assertEqual( + evidence["control_plane_fixture"]["provenance"], + "TEST_CONTROL_PLANE_FIXTURE", + ) + self.assertFalse( + evidence["control_plane_fixture"][ + "accelerated_time_is_production_evidence" + ] + ) + + def test_runbook_and_make_targets_are_reproducible(self): + runbook = ( + ROOT / "docs/runbooks/phase93-hold-close-expand.md" + ).read_text() + makefile = (ROOT / "Makefile").read_text() + for token in ( + "Starting A Real Hold", + "Closing The Rollback Window", + "Expansion", + "Python Runtime Decommission", + "Incident And Rollback", + "TEST_CONTROL_PLANE_FIXTURE", + ): + self.assertIn(token, runbook) + for target in ( + "phase93-test:", + "phase93-migration:", + "phase93-certify:", + "phase93-clean:", + ): + self.assertIn(target, makefile) + result = subprocess.run( + ["python3", "scripts/phase93_hold_close_certification.py", "--help"], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_control_plane_import_does_not_eagerly_require_alpha_sdk(self): + package = (ROOT / "qdl/canary/__init__.py").read_text() + self.assertNotIn("from qdl.canary.consumer import", package) + self.assertIn("def __getattr__", package) + self.assertIn("_CONSUMER_EXPORTS", package) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase93_hold_close_expand.py b/tests/test_phase93_hold_close_expand.py new file mode 100644 index 0000000..34d0420 --- /dev/null +++ b/tests/test_phase93_hold_close_expand.py @@ -0,0 +1,598 @@ +from __future__ import annotations + +import copy +import json +import unittest +import uuid +from dataclasses import replace +from pathlib import Path + +from qdl.canary.phase93 import ( + AuthorityRegistrySnapshot, + ClosureApproval, + ConsumerCheckpoint, + ConsumerRegistrySnapshot, + DecommissionRequest, + ExpansionManifest, + ExpansionType, + HoldScope, + HoldStatus, + PrimaryHoldEvaluator, + PrimaryHoldIdentity, + PrimaryHoldObservation, + PrimaryHoldPolicy, + ProductionClosureAuthorizer, + RollbackRehearsalEvidence, + assess_decommission, +) +from qdl.certification.prerequisites import CandidateSlice + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase93Fixtures: + def setUp(self) -> None: + self.candidate = CandidateSlice.load( + ROOT / "config/phase9/candidate-slice.yaml" + ) + self.no_go = json.loads( + ( + ROOT + / "upgrade/evidence/phase90c-production-prerequisites.json" + ).read_text() + ) + self.bundle_id = self.no_go["bundle_id"] + self.start = 1_000_000_000 + self.policy = PrimaryHoldPolicy( + minimum_duration_seconds=120, + sample_interval_seconds=60, + max_sample_gap_seconds=60, + max_lag_ms=500, + max_freshness_ms=1_000, + max_queue_depth=1_000, + max_spool_bytes=1_000_000, + max_cpu_percent=80.0, + max_rss_mb=512.0, + ) + self.identity = PrimaryHoldIdentity( + schema="qdl.primary-hold.v1", + hold_id=str(uuid.uuid4()), + slice_id=self.candidate.payload["slice_id"], + candidate_digest=self.candidate.digest, + prerequisite_bundle_id=self.bundle_id, + owner_id="rust-primary", + authority_revision=8, + lease_epoch=12, + partition_plan_epoch=1, + started_at_ns=self.start, + required_until_ns=self.start + 120_000_000_000, + policy_digest=self.policy.digest, + ) + + def observation( + self, + sequence: int, + *, + watermark: int | None = None, + observed_at_ns: int | None = None, + **changes: object, + ) -> PrimaryHoldObservation: + payload = { + "schema": "qdl.primary-hold-observation.v1", + "observation_id": str(uuid.uuid4()), + "hold_id": self.identity.hold_id, + "slice_id": self.identity.slice_id, + "candidate_digest": self.identity.candidate_digest, + "owner_id": self.identity.owner_id, + "authority_revision": self.identity.authority_revision, + "lease_epoch": self.identity.lease_epoch, + "partition_plan_epoch": self.identity.partition_plan_epoch, + "sequence": sequence, + "observed_at_ns": ( + observed_at_ns + if observed_at_ns is not None + else self.start + sequence * 60_000_000_000 + ), + "last_watermark": watermark if watermark is not None else 100 + sequence, + "lag_ms": 10, + "freshness_ms": 20, + "queue_depth": 1, + "spool_bytes": 100, + "cpu_percent": 10.0, + "rss_mb": 64.0, + "registered_consumers": 2, + "healthy_consumers": 2, + "checkpoint_watermark": watermark if watermark is not None else 100 + sequence, + } + payload.update(changes) + return PrimaryHoldObservation(**payload) + + def passing_hold(self, *, scope: HoldScope): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, policy=self.policy, scope=scope + ) + self.assertEqual(evaluator.observe(self.observation(1, watermark=110)), "PASS") + self.assertEqual(evaluator.observe(self.observation(2, watermark=120)), "PASS") + return evaluator.decision( + decision_id=str(uuid.uuid4()), + now_ns=self.identity.required_until_ns, + ) + + def go(self, now_ns: int) -> dict: + payload = copy.deepcopy(self.no_go) + payload.update( + { + "decision": "GO", + "passed": len(payload["gates"]), + "blocked": 0, + "issued_at_ns": now_ns - 1_000_000, + "authority_state": "RUST_SHADOW", + "v1_unchanged": True, + "production_mutations": 0, + } + ) + for gate in payload["gates"]: + gate.update({"passed": True, "reason": "PASS"}) + return payload + + def closure_inputs(self, now_ns: int) -> dict: + hold = self.passing_hold(scope=HoldScope.PRODUCTION) + authority = AuthorityRegistrySnapshot( + schema="qdl.authority-registry-snapshot.v1", + snapshot_id=str(uuid.uuid4()), + slice_id=hold.slice_id, + state="RUST_PRIMARY", + owner_id=hold.owner_id, + authority_revision=hold.authority_revision, + lease_epoch=hold.lease_epoch, + partition_plan_epoch=hold.partition_plan_epoch, + candidate_digest=hold.candidate_digest, + prerequisite_bundle_id=hold.prerequisite_bundle_id, + current_watermark=130, + public_write_allowed=True, + legacy_write_allowed=True, + observed_at_ns=now_ns - 1, + ) + consumers = ConsumerRegistrySnapshot( + schema="qdl.consumer-registry-snapshot.v1", + snapshot_id=str(uuid.uuid4()), + slice_id=hold.slice_id, + authority_revision=hold.authority_revision, + checkpoints=tuple( + ConsumerCheckpoint( + consumer_id=name, + requirement_digest=digest * 64, + contract_major=2, + applied_watermark=130, + checkpointed_watermark=130, + status="READY", + migration_status="COMPLETE", + rollback_ready=True, + ) + for name, digest in (("alpha-a", "a"), ("execution-b", "b")) + ), + observed_at_ns=now_ns - 1, + ) + rollback = RollbackRehearsalEvidence( + schema="qdl.rollback-rehearsal.v1", + rehearsal_id=str(uuid.uuid4()), + slice_id=hold.slice_id, + candidate_digest=hold.candidate_digest, + owner_id=hold.owner_id, + authority_revision=hold.authority_revision, + lease_epoch=hold.lease_epoch, + partition_plan_epoch=hold.partition_plan_epoch, + rollback_manifest_digest=self.candidate.payload[ + "rollback_manifest_digest" + ], + reconciled_through_watermark=130, + rto_ms=500.0, + status="PASS", + production_scope=True, + observed_at_ns=now_ns - 1, + expires_at_ns=now_ns + 60_000_000_000, + ) + approval = ClosureApproval( + schema="qdl.rollback-window-closure-approval.v1", + approval_id=str(uuid.uuid4()), + closure_id=str(uuid.uuid4()), + decision="APPROVE", + slice_id=hold.slice_id, + candidate_digest=hold.candidate_digest, + prerequisite_bundle_id=hold.prerequisite_bundle_id, + hold_id=hold.hold_id, + hold_policy_digest=hold.policy_digest, + operator="phase93-test-operator", + change_ticket="QDL-93", + allow_close_rollback_window=True, + repository_cleanup_approved=False, + approved_at_ns=now_ns - 1, + expires_at_ns=now_ns + 60_000_000_000, + ) + primary = { + "schema": "qdl.phase92.production-primary.v1", + "status": "PRODUCTION_PRIMARY_ACTIVE", + "production_authorized": True, + "slice_id": hold.slice_id, + "candidate_digest": hold.candidate_digest, + "prerequisite_bundle_id": hold.prerequisite_bundle_id, + "authority": { + "state": "RUST_PRIMARY", + "owner_id": hold.owner_id, + "authority_revision": hold.authority_revision, + "lease_epoch": hold.lease_epoch, + "partition_plan_epoch": hold.partition_plan_epoch, + "current_watermark": 130, + }, + } + return { + "candidate": self.candidate, + "prerequisite_decision": self.go(now_ns), + "expected_bundle_id": self.bundle_id, + "primary_evidence": primary, + "hold_decision": hold, + "consumer_registry": consumers, + "authority_registry": authority, + "rollback_evidence": rollback, + "approval": approval, + "now_ns": now_ns, + } + + +class PrimaryHoldEvaluatorTest(Phase93Fixtures, unittest.TestCase): + def test_clean_dense_hold_passes_without_production_authority_in_test_scope(self): + decision = self.passing_hold(scope=HoldScope.TEST_REHEARSAL) + self.assertEqual(decision.status, HoldStatus.PASSED) + self.assertEqual(decision.reason, "PASS") + self.assertFalse(decision.production_authorized) + self.assertEqual(decision.observation_count, 2) + self.assertEqual(decision.terminal_watermark, 120) + self.assertEqual(len(decision.digest), 64) + + def test_incomplete_sparse_out_of_order_and_watermark_regression_fail(self): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + empty = evaluator.decision( + decision_id=str(uuid.uuid4()), now_ns=self.identity.required_until_ns + ) + self.assertEqual(empty.reason, "HOLD_OBSERVATION_MISSING") + + cases = ( + ( + self.observation(2), + "HOLD_SEQUENCE_NOT_CONTIGUOUS", + ), + ( + self.observation( + 1, + observed_at_ns=self.start + 61_000_000_000, + ), + "HOLD_OBSERVATION_GAP_EXCEEDED", + ), + ) + for item, expected in cases: + with self.subTest(expected=expected): + candidate = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + self.assertEqual(candidate.observe(item), expected) + self.assertEqual( + candidate.decision( + decision_id=str(uuid.uuid4()), + now_ns=self.identity.required_until_ns, + ).status, + HoldStatus.BLOCKED, + ) + + regression = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + regression.observe(self.observation(1, watermark=120)) + self.assertEqual( + regression.observe(self.observation(2, watermark=119)), + "HOLD_WATERMARK_REGRESSED", + ) + + def test_every_correctness_and_resource_breach_is_sticky(self): + breaches = { + "semantic_mismatches": "SEMANTIC_MISMATCH", + "open_gaps": "OPEN_GAP", + "duplicate_external_writes": "DUPLICATE_EXTERNAL_WRITE", + "accepted_stale_writer_writes": "ACCEPTED_STALE_WRITER_WRITE", + "authority_ambiguities": "AUTHORITY_AMBIGUITY", + "durable_ack_failures": "DURABLE_ACK_FAILURE", + "projection_mismatches": "PROJECTION_MISMATCH", + "consumer_checkpoint_regressions": "CONSUMER_CHECKPOINT_REGRESSION", + "unexplained_quality_failures": "UNEXPLAINED_QUALITY_FAILURE", + "lag_ms": "LAG_THRESHOLD_EXCEEDED", + "freshness_ms": "FRESHNESS_THRESHOLD_EXCEEDED", + "queue_depth": "QUEUE_THRESHOLD_EXCEEDED", + "spool_bytes": "SPOOL_THRESHOLD_EXCEEDED", + "cpu_percent": "CPU_THRESHOLD_EXCEEDED", + "rss_mb": "RSS_THRESHOLD_EXCEEDED", + } + values = { + "lag_ms": self.policy.max_lag_ms + 1, + "freshness_ms": self.policy.max_freshness_ms + 1, + "queue_depth": self.policy.max_queue_depth + 1, + "spool_bytes": self.policy.max_spool_bytes + 1, + "cpu_percent": self.policy.max_cpu_percent + 1, + "rss_mb": self.policy.max_rss_mb + 1, + } + for field, expected in breaches.items(): + with self.subTest(field=field): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + item = self.observation(1, **{field: values.get(field, 1)}) + self.assertEqual(evaluator.observe(item), expected) + self.assertEqual( + evaluator.observe(self.observation(2)), + "HOLD_ALREADY_BLOCKED", + ) + self.assertEqual( + evaluator.decision( + decision_id=str(uuid.uuid4()), + now_ns=self.identity.required_until_ns, + ).reason, + expected, + ) + + def test_identity_consumer_and_numeric_type_guards(self): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + changed = self.observation(1, owner_id="other-owner") + self.assertEqual( + evaluator.observe(changed), "HOLD_AUTHORITY_IDENTITY_CHANGED" + ) + for change in ( + {"registered_consumers": 0, "healthy_consumers": 0}, + {"healthy_consumers": 1}, + {"checkpoint_watermark": 99, "last_watermark": 100}, + ): + with self.subTest(change=change): + current = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + reason = current.observe(self.observation(1, **change)) + self.assertNotEqual(reason, "PASS") + with self.assertRaises(ValueError): + self.observation(1, open_gaps=True) + + +class ProductionClosureTest(Phase93Fixtures, unittest.TestCase): + def test_current_no_go_and_local_primary_evidence_cannot_close(self): + now_ns = self.no_go["issued_at_ns"] + 1 + values = self.closure_inputs(now_ns) + values["prerequisite_decision"] = self.no_go + denied = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(denied.allowed) + self.assertEqual(denied.reason, "PREREQUISITE_DECISION_NOT_GO") + + values = self.closure_inputs(now_ns) + values["primary_evidence"] = json.loads( + ( + ROOT + / "upgrade/evidence/phase92-bounded-primary-certification.json" + ).read_text() + ) + denied = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(denied.allowed) + self.assertEqual(denied.reason, "PRIMARY_EVIDENCE_SCHEMA_INVALID") + + def test_complete_production_shaped_fixture_authorizes_without_mutation(self): + now_ns = 2_000_000_000_000 + values = self.closure_inputs(now_ns) + result = ProductionClosureAuthorizer().authorize(**values) + self.assertTrue(result.allowed) + self.assertTrue(result.production_authorized) + self.assertIsNotNone(result.closure) + closure = result.closure + assert closure is not None + self.assertEqual(closure.owner_id, "rust-primary") + self.assertEqual(closure.authority_revision, 8) + self.assertEqual(len(closure.digest), 64) + + def test_registry_rollback_and_approval_mismatches_fail_closed(self): + now_ns = 2_000_000_000_000 + cases = [] + + values = self.closure_inputs(now_ns) + values["consumer_registry"] = replace( + values["consumer_registry"], + observed_at_ns=now_ns - 301_000_000_000, + ) + cases.append((values, "REGISTRY_SNAPSHOT_STALE")) + + values = self.closure_inputs(now_ns) + values["primary_evidence"] = copy.deepcopy(values["primary_evidence"]) + values["primary_evidence"]["authority"]["lease_epoch"] = 99 + cases.append((values, "PRIMARY_AUTHORITY_REGISTRY_MISMATCH")) + + values = self.closure_inputs(now_ns) + checkpoint = values["consumer_registry"].checkpoints[0] + values["consumer_registry"] = replace( + values["consumer_registry"], + checkpoints=( + replace( + checkpoint, + applied_watermark=129, + checkpointed_watermark=129, + ), + values["consumer_registry"].checkpoints[1], + ), + ) + cases.append((values, "CONSUMER_CHECKPOINT_BEHIND")) + + values = self.closure_inputs(now_ns) + values["rollback_evidence"] = replace( + values["rollback_evidence"], production_scope=False + ) + cases.append((values, "ROLLBACK_REHEARSAL_NOT_PRODUCTION")) + + values = self.closure_inputs(now_ns) + values["rollback_evidence"] = replace( + values["rollback_evidence"], rollback_manifest_digest="f" * 64 + ) + cases.append((values, "ROLLBACK_MANIFEST_MISMATCH")) + + values = self.closure_inputs(now_ns) + values["approval"] = replace( + values["approval"], hold_policy_digest="e" * 64 + ) + cases.append((values, "CLOSURE_APPROVAL_IDENTITY_MISMATCH")) + + for values, expected in cases: + with self.subTest(expected=expected): + result = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + def test_test_scope_hold_cannot_close_production_window(self): + now_ns = 2_000_000_000_000 + values = self.closure_inputs(now_ns) + values["hold_decision"] = self.passing_hold( + scope=HoldScope.TEST_REHEARSAL + ) + result = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, "PRIMARY_HOLD_NOT_PRODUCTION") + + +class ExpansionAndDecommissionTest(Phase93Fixtures, unittest.TestCase): + def parent_closure(self): + values = self.closure_inputs(2_000_000_000_000) + result = ProductionClosureAuthorizer().authorize(**values) + assert result.closure is not None + return result.closure + + def test_each_expansion_requires_distinct_independent_certification(self): + parent = self.parent_closure() + manifests = [] + for index, expansion_type in enumerate(ExpansionType, start=1): + manifest = ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=expansion_type, + candidate_digest=f"{index}" * 64, + scope_digest=format(index + 5, "x") * 64, + partition_plan_epoch=( + parent.partition_plan_epoch + 1 + if expansion_type == ExpansionType.INSTRUMENT_PARTITION + else parent.partition_plan_epoch + ), + created_at_ns=2_100_000_000_000 + index, + ) + manifests.append(manifest) + self.assertEqual( + manifest.status, "INDEPENDENT_CERTIFICATION_REQUIRED" + ) + self.assertFalse(manifest.transitive_evidence_allowed) + self.assertFalse(manifest.public_write_allowed) + self.assertFalse(manifest.legacy_write_allowed) + self.assertIn("rollback", manifest.required_gates) + self.assertIn("exact_frame_parity", manifest.required_gates) + self.assertEqual( + len({item.candidate_digest for item in manifests}), + len(ExpansionType), + ) + + def test_expansion_cannot_reuse_parent_or_weaken_gates(self): + parent = self.parent_closure() + with self.assertRaises(ValueError): + ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=ExpansionType.BBO, + candidate_digest=parent.candidate_digest, + scope_digest="d" * 64, + partition_plan_epoch=1, + created_at_ns=2_100_000_000_000, + ) + with self.assertRaises(ValueError): + ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=ExpansionType.INSTRUMENT_PARTITION, + candidate_digest="c" * 64, + scope_digest="d" * 64, + partition_plan_epoch=parent.partition_plan_epoch, + created_at_ns=2_100_000_000_000, + ) + valid = ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=ExpansionType.BBO, + candidate_digest="c" * 64, + scope_digest="d" * 64, + partition_plan_epoch=1, + created_at_ns=2_100_000_000_000, + ) + with self.assertRaises(ValueError): + replace(valid, required_gates=("rollback",)) + with self.assertRaises(ValueError): + replace(valid, transitive_evidence_allowed=True) + + def test_decommission_requires_zero_dependency_and_explicit_cleanup(self): + base = DecommissionRequest( + schema="qdl.runtime-decommission-request.v1", + request_id=str(uuid.uuid4()), + runtime_id="python-binance-usdm-trade", + owned_slice_ids=(), + rollback_reference_ids=(), + consumer_dependency_ids=(), + all_replacement_windows_closed=True, + repository_cleanup_approved=True, + shared_knowledge_retained=True, + ) + self.assertTrue(assess_decommission(base).allowed) + cases = ( + ( + replace(base, owned_slice_ids=("slice-a",)), + "RUNTIME_STILL_OWNS_SLICES", + ), + ( + replace(base, rollback_reference_ids=("rollback-a",)), + "RUNTIME_STILL_REQUIRED_FOR_ROLLBACK", + ), + ( + replace(base, consumer_dependency_ids=("consumer-a",)), + "RUNTIME_HAS_CONSUMER_DEPENDENCIES", + ), + ( + replace(base, all_replacement_windows_closed=False), + "REPLACEMENT_WINDOWS_NOT_CLOSED", + ), + ( + replace(base, repository_cleanup_approved=False), + "REPOSITORY_CLEANUP_NOT_APPROVED", + ), + ( + replace(base, shared_knowledge_retained=False), + "SHARED_KNOWLEDGE_REMOVAL_FORBIDDEN", + ), + ) + for request, expected in cases: + with self.subTest(expected=expected): + decision = assess_decommission(request) + self.assertFalse(decision.allowed) + self.assertEqual(decision.reason, expected) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase93_migration_contract.py b/tests/test_phase93_migration_contract.py new file mode 100644 index 0000000..8253523 --- /dev/null +++ b/tests/test_phase93_migration_contract.py @@ -0,0 +1,96 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase93MigrationContractTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.sql = ( + ROOT / "migrations/postgres/0008_phase93_hold_close_expand.sql" + ).read_text() + + def test_migration_is_additive_and_all_evidence_is_append_only(self): + for token in ( + "qdl_primary_holds", + "qdl_primary_hold_observations", + "qdl_primary_hold_decisions", + "qdl_consumer_registry_snapshots", + "qdl_authority_registry_snapshots", + "qdl_rollback_rehearsals", + "qdl_closure_approvals", + "qdl_authority_closures", + "qdl_expansion_candidates", + "qdl_runtime_decommission_decisions", + "Phase 9.3 hold/closure/expansion evidence is append-only", + ): + self.assertIn(token, self.sql) + self.assertNotIn("DROP TABLE", self.sql.upper()) + self.assertNotIn("TRUNCATE", self.sql.upper()) + + def test_hold_observation_and_passing_decision_are_evidence_backed(self): + for token in ( + "hold observation identity mismatch", + "hold observation sequence is not contiguous", + "hold observation time is not monotonic", + "hold observation gap exceeds policy", + "hold watermark regressed", + "passing hold decision is not supported by evidence", + "semantic_mismatches <> 0", + "consumer_checkpoint_regressions <> 0", + "checkpoint_watermark < last_watermark", + ): + self.assertIn(token, self.sql) + + def test_closure_locks_authority_and_frozen_registries_without_update(self): + start = self.sql.index("CREATE OR REPLACE FUNCTION qdl_close_authority_window") + end = self.sql.index("CREATE TABLE IF NOT EXISTS qdl_expansion_candidates", start) + function = self.sql[start:end] + for token in ( + "FOR UPDATE", + "current_row.state <> 'RUST_PRIMARY'", + "authority closure CAS mismatch", + "closure_sha256", + "consumer registry snapshot is invalid", + "authority registry snapshot is invalid", + "rollback rehearsal is invalid", + "closure approval is invalid or expired", + "INSERT INTO qdl_authority_closures", + ): + self.assertIn(token, function) + self.assertNotIn("UPDATE qdl_authority_slices", function) + self.assertNotIn("DELETE FROM", function) + + def test_expansions_are_independent_and_cannot_write(self): + for token in ( + "INDEPENDENT_CERTIFICATION_REQUIRED", + "parent_closure_digest <> closure_row.closure_sha256", + "NOT transitive_evidence_allowed", + "NOT public_write_allowed", + "NOT legacy_write_allowed", + "expansion requires independent capability gates", + "instrument expansion requires a new partition epoch", + "snapshot_delta_sequence", + "final_revision_lineage", + "provider_semantics", + ): + self.assertIn(token, self.sql) + + def test_decommission_allowance_requires_zero_dependency(self): + for token in ( + "owned_slice_count = 0", + "rollback_reference_count = 0", + "consumer_dependency_count = 0", + "all_replacement_windows_closed", + "repository_cleanup_approved", + "shared_knowledge_retained", + ): + self.assertIn(token, self.sql) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_stream_supervisor.py b/tests/test_stream_supervisor.py index e0c5c33..4ba9a7c 100644 --- a/tests/test_stream_supervisor.py +++ b/tests/test_stream_supervisor.py @@ -30,6 +30,7 @@ def test_snapshot_reports_missing_then_observed_feed(self): shard_id = supervisor.register_shard("binance_spot_trade", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_publish( { "key": "trade:price:BTCUSDT", @@ -51,6 +52,7 @@ def test_queue_drop_is_warning_by_default(self): supervisor = StreamSupervisor() shard_id = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_queue_drop("binance_spot_kline", shard_id) snapshot = supervisor.snapshot() @@ -63,6 +65,7 @@ def test_queue_drop_degrades_strict_health(self): supervisor = StreamSupervisor(strict_feed_health=True) shard_id = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_queue_drop("binance_spot_kline", shard_id) snapshot = supervisor.snapshot() @@ -74,6 +77,7 @@ def test_old_queue_drop_does_not_permanently_degrade_health(self): supervisor.queue_drop_window_seconds = 1 shard_id = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_queue_drop("binance_spot_kline", shard_id) snapshot = supervisor.snapshot(now=time.time() + 2) @@ -86,6 +90,7 @@ def test_missing_trade_feed_is_diagnostic_not_health_failure(self): shard_id = supervisor.register_shard("binance_spot_trade", "wss://example") supervisor.expect_feed("binance_spot_trade", "trade", "ILLQUSDT") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) snapshot = supervisor.snapshot(now=supervisor.started_at + 10) self.assertEqual(snapshot["status"], "ok") @@ -98,8 +103,8 @@ def test_missing_kline_after_startup_grace_is_warning_by_default(self): supervisor.expect_feed("binance_spot_kline", "kline", "BTCUSDT", "1m") supervisor.mark_connected(shard_id) - snapshot = supervisor.snapshot(now=supervisor.started_at + 10) - self.assertEqual(snapshot["status"], "ok") + snapshot = supervisor.snapshot(now=supervisor.started_at + 20) + self.assertEqual(snapshot["status"], "degraded") self.assertEqual(snapshot["feeds"]["health_missing_count"], 1) def test_missing_kline_after_startup_grace_degrades_strict_health(self): @@ -108,7 +113,7 @@ def test_missing_kline_after_startup_grace_degrades_strict_health(self): supervisor.expect_feed("binance_spot_kline", "kline", "BTCUSDT", "1m") supervisor.mark_connected(shard_id) - snapshot = supervisor.snapshot(now=supervisor.started_at + 10) + snapshot = supervisor.snapshot(now=supervisor.started_at + 20) self.assertEqual(snapshot["status"], "degraded") self.assertTrue(snapshot["strict_feed_health"]) diff --git a/upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md b/upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md new file mode 100644 index 0000000..ed1515f --- /dev/null +++ b/upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md @@ -0,0 +1,103 @@ +# Phase 9.0-A Runtime Correctness Closure Report + +Date: 2026-08-18 +Branch: `feat/phase9-runtime-correctness` +Commits: `765a7f1`, `7302c45` +Decision: `PASS_ISOLATED_NO_PRODUCTION_CUTOVER` + +## Scope And Safety Boundary + +Phase 9.0-A closed the runtime correctness defects found after migration without +restarting or mutating the running V1 service. The candidate used a dedicated +Compose project, Redis instance, loopback port, networks and volumes. It held no +canonical authority and was removed after evidence capture. + +The public V1 surface remains unchanged. Live and candidate OpenAPI each exposed +40 paths, with zero additions and zero removals. + +## Implemented + +- Split transport connection from source data readiness and report TRADE/KLINE + independently per source and shard. +- Require a valid provider frame before readiness. Subscription ACK, malformed + payload and wrong-feed payload cannot make a shard healthy. +- Add first-frame and idle watchdogs with typed outage counters and jittered, + bounded reconnect backoff. A data outage survives transport reconnect and is + cleared only by a valid provider frame. +- Replace drop-oldest queue behavior with bounded backpressure. Queue pressure is + observable and a sustained full queue reconnects the source instead of + silently deleting an earlier event. +- Add one demand-only Binance USD-M closed-kline recovery manager with TTL + ownership, bounded concurrency, per-feed backoff, final-bar validation, + symbol/interval/open-time deduplication and explicit + `BINANCE_REST_GAP_FILL` provenance. +- Preserve the provider interval from `k.i`; recovered 5m or other supported bars + cannot be projected into a legacy 1m key. +- Preserve all gap rows in a publisher batch while retaining latest-state + coalescing for ordinary trade/kline updates. +- Keep existing V1 health response keys while fixing the TRADE/KLINE booleans and + adding source/recovery detail under the existing nested payload. +- Add an immutable candidate Compose boundary: UID/GID 10001, read-only root, + no source bind, dedicated writable data/log volumes, loopback-only ingress, + dropped capabilities and explicit CPU/RAM/PID limits. + +## Verification + +| Gate | Result | +|---|---| +| Targeted runtime/demand/watchdog tests | 35/35 pass | +| Full repository tests | 345 run: 340 pass, 5 environment-gated skips, 0 fail | +| Python compileall and diff check | Pass | +| V1 live-vs-candidate OpenAPI paths | 40/40, added 0, removed 0 | +| Candidate identity | `10001:10001` | +| Read-only root / no source bind | Pass / pass | +| Limits | 1.5 CPU, 1.5 GiB RAM, 256 PIDs | +| Candidate cleanup | 0 containers, networks, volumes and images remain | +| Production V1 restart/state mutation | None | + +## Real Provider Evidence + +Binance USD-M TRADE produced valid frames on all 8 shards. The provider accepted +all 8 KLINE connections but produced zero valid kline frames, so the candidate +correctly reported KLINE unavailable and top-level health `degraded`. Transport +reconnect never made the source green again. + +A BTCUSDT 1m API read created one TTL demand lease. Recovery fetched a fully +closed Binance REST bar, projected it with finality and provenance, and a direct +Binance REST query for the same timestamps matched open time, close time and all +OHLCV fields exactly. No generated or substituted market data was used. + +After lease expiry, demand and active recovery counts returned to zero. Provider +fetch count remained 2 across the following poll. Candidate queue drop and +pressure counts were both zero. + +A single resource snapshot while broad TRADE and unavailable KLINE shards were +active measured approximately 20.07% of one CPU and 131 MiB for the app, plus +2.27% CPU and 5.129 MiB for isolated Redis. These are bounded by Compose; this +was a correctness smoke, not a long soak or capacity certification. + +## Defects Caught Before Release + +1. The pinned Redis image could not call `setpriv` after `cap_drop: ALL` when no + user was declared. The candidate now runs Redis directly as `999:999`. +2. A transport reconnect initially reset source startup state, briefly making + top-level health green while KLINE still had no data. Data-outage state now + survives reconnect until a valid frame arrives. +3. Multi-interval recovery initially inherited the stream default interval. + Projection now uses the authoritative payload interval and has a 5m + regression test. + +## Decision And Remaining Boundary + +Phase 9.0-A is complete for isolated implementation and acceptance. It is ready +for review and a separately approved controlled rollout. The currently running +V1 container remains `data-layer:v0.1.0` and therefore still has legacy health +semantics until that rollout occurs. + +Phase 9.1 remains blocked. This phase does not provide production Kafka/OTel, +workload identity, external secret rotation, signed-image admission, +independent-failure-domain DR, complete consumer registration or exact authority +slice approval. + +Machine-readable evidence is in +[`phase90a-runtime-correctness.json`](phase90a-runtime-correctness.json). diff --git a/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md b/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md new file mode 100644 index 0000000..887d9f5 --- /dev/null +++ b/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md @@ -0,0 +1,46 @@ +# Phase 9.0-B Isolated V2 Beta Report + +Decision: `PASS_ISOLATED_NO_AUTHORITY_CUTOVER` + +## Scope + +The existing V2 query/stream beta was re-certified from the Phase 9.0-A +correctness baseline. V1 remained source and public authority. The candidate +used an isolated Redis, durable spool, credentials, consumer group, loopback +ports and Compose project. No Rust canary or public-internet exposure occurred. + +## Results + +- Authentic slice: `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`. +- Canonical mismatches: `0`; generated events: + `0`; duplicate open times: + `0`. +- Continuous bridge watermark delta across the bounded observation window: + `0`. +- Active/passive stream replay: `3` fast consumers, + `1` isolated slow consumer, contiguous= + `True`. +- Query normal: `30` requests, p99.9 + `72.211` ms, `0` errors. +- Query burst: `60` requests, p99.9 + `486.649` ms, `0` errors. +- Peak candidate RSS: `55658414.08` bytes; peak CPU: + `33.36%` of one core. +- Security/adversarial, cursor, failover/fencing, dependency outage/recovery, + rate limit, oversized request and cleanup gates all passed. + +## Safety And Cleanup + +Candidate revision: `1c881389b4ee21a153903505822c61512b176044`. Runtime user was +`10001:10001`, root was read-only, no source bind was mounted and bridge access +was restricted to beta-internal plus the existing V1 internal network. +Production V1 topology/OpenAPI remained unchanged. Candidate containers, +networks, volumes and candidate tags after cleanup: `0/0/0/0`; production beta +keys: `0`. + +## Remaining Boundary + +This result permits review of an isolated read-only V2 beta only. Phase 9.1 +remains blocked on replicated production transport, OTel/alerts, workload +identity, external secrets, signature admission, independent DR, complete +consumer registration and explicit exact-slice authority approval. diff --git a/upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md b/upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md new file mode 100644 index 0000000..0ba358e --- /dev/null +++ b/upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md @@ -0,0 +1,32 @@ +# Phase 9.0-C Production Prerequisite Report + +Decision: `NO_GO_EXTERNAL` + +## Candidate + +- Slice: `production/binance/usdm/perpetual/trade/plan-1/btcusdt` +- Candidate digest: `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` +- Authority: `RUST_SHADOW`; V1 unchanged: `True` + +## Gate Summary + +- Passed: `0` +- Blocked: `12` + +## Blocking Evidence + +- `replicated_durable_transport`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `production_observability`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `workload_identity_rbac_network`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `external_secret_rotation`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) +- `signed_artifact_admission`: `EVIDENCE_BLOCKED` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `postgres_pitr`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) +- `object_store_restore`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) +- `independent_failure_domain_dr`: `MISSING_EVIDENCE` (observed `None`, required `INDEPENDENT_FAILURE_DOMAIN`) +- `redis_projector_rebuild`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `consumer_registration_rollback`: `EVIDENCE_BLOCKED` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `persistent_authority_sink_fencing`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `exact_slice_approval`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) + +These are real infrastructure/operator blockers. Same-host fixtures or +local rehearsals must not be relabeled to close them. diff --git a/upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md b/upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md new file mode 100644 index 0000000..124b87a --- /dev/null +++ b/upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md @@ -0,0 +1,45 @@ +# Phase 9.1 Rust Canary Certification Report + +## Decision + +- Status: `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED` +- Production authorized: `false` +- Production mutations: `0` +- Prerequisite decision: `NO_GO_EXTERNAL` +- Slice: `production/binance/usdm/perpetual/trade/plan-1/btcusdt` +- Candidate digest: `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` + +## Authentic Parity + +- Provenance: `REAL_PROVIDER_READ_ONLY` +- Frozen fixtures: `128` +- Repetition: `200` +- Canonical events: `25600` +- Semantic mismatches: `0` +- Clean Rust process runs: `3` +- Aggregate SHA-256: `75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea` +- Python throughput: `27115.455` events/s +- Minimum Rust throughput: `350581.025` events/s + +## Authority And Broker Recovery + +- Transition audit: `RUST_SHADOW, RUST_CANARY, BLOCKED, RUST_SHADOW` +- Final authority: `RUST_SHADOW` +- One-replica-loss ACK: `true` +- Below-min-ISR fail closed: `true` +- Slow-consumer records: `64` +- Slow-consumer ordered and gap-free: `true` +- Public writes: `0` +- Legacy writes: `0` + +## Isolation And Cleanup + +- V1 health before/after: `200/200` +- V1 topology unchanged: `true` +- Containers/networks/volumes remaining: `0/0/0` + +## Remaining External Gates + +- Production Phase 9.0-C infrastructure and operator gates remain `NO_GO_EXTERNAL`. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- Python V1 remains the sole authoritative public and legacy writer. This report does not authorize a production `RUST_CANARY` transition. diff --git a/upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md b/upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md new file mode 100644 index 0000000..706b192 --- /dev/null +++ b/upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md @@ -0,0 +1,45 @@ +# Phase 9.2 Bounded Rust Primary Certification Report + +## Decision + +- Status: `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED` +- Production authorized: `false` +- Production mutations: `0` +- Prerequisite decision: `NO_GO_EXTERNAL` +- Slice: `production/binance/usdm/perpetual/trade/plan-1/btcusdt` +- Candidate digest: `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` + +## Authentic Parity + +- Provenance: `REAL_PROVIDER_READ_ONLY` +- Canonical events: `25600` +- Semantic mismatches: `0` +- Clean Rust process runs: `3` + +## Terminal Handoff And Recovery + +- Authority states: `RUST_CANARY, RUST_PRIMARY, BLOCKED, ROLLBACK_PENDING, PYTHON_PRIMARY` +- Terminal checkpoints / accepted handoffs: `2 / 2` +- Projection parity: `true` +- Boundary gap-free: `true` +- Owner boundary correct: `true` +- Restart recovery: `PASS` +- Recovered target watermarks: `{'legacy': 180, 'primary': 180, 'public': 180}` +- First post-restart watermark: `181` +- Cutover / rollback measurement: `22.200 ms / 533.237 ms` +- One-replica-loss ACK: `true` +- Below-min-ISR fail closed: `true` +- Final authority: `PYTHON_PRIMARY` +- Production public / legacy writes: `0 / 0` + +## Isolation And Cleanup + +- V1 health before/after: `200 / 200` +- V1 topology unchanged: `true` +- Containers/networks/volumes remaining: `0 / 0 / 0` + +## Remaining External Gates + +- Phase 9.0-C remains `NO_GO_EXTERNAL`; a production primary transition is not authorized. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- A real production canary hold and explicit exact-slice approval remain required. diff --git a/upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md b/upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md new file mode 100644 index 0000000..e7947fe --- /dev/null +++ b/upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md @@ -0,0 +1,42 @@ +# Phase 9.3 Hold, Close And Expand Certification Report + +## Decision + +- Status: COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED +- Production authorized: False +- Production hold started: False +- Production rollback window closed: False +- Production expansions authorized: 0 +- Production mutations: 0 + +## Parent Evidence + +- Phase 9.2 status: COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED +- Authentic provider events: 25600 +- Semantic mismatches: 0 +- Parent production authorized: False + +## Isolated Control Plane + +- Provenance: TEST_CONTROL_PLANE_FIXTURE +- Accelerated time is production evidence: False +- Test hold status: PASSED +- Test hold production authorized: False +- Current no-go rejection: PREREQUISITE_DECISION_NOT_GO +- Local Phase 9.2 production eligible: False +- Expansion manifests: 5 +- Decommission decision: RUNTIME_STILL_REQUIRED_FOR_ROLLBACK + +## Persistence And Isolation + +- Migration: PASS +- Closure changed authority: False +- V1 health before/after: 200 / 200 +- V1 topology unchanged: True +- Disposable resources remaining: 0 + +## External Gates + +Phase 9.0-C remains NO_GO_EXTERNAL. There is no real Rust primary, production +hold duration, production consumer checkpoint set or operator closure approval. +No rollback window, expansion or Python decommission is authorized. diff --git a/upgrade/evidence/phase90a-evidence.sha256 b/upgrade/evidence/phase90a-evidence.sha256 new file mode 100644 index 0000000..327ec5f --- /dev/null +++ b/upgrade/evidence/phase90a-evidence.sha256 @@ -0,0 +1,2 @@ +cfd839f5482aa0b46e238fb8b4c47094648c79adf5bbce9ece8f57205e4eb761 upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md +56923c6d7647156c6adb92004b6fbc7e2cb718cc6298dc5c5a46a8e0e546fdb8 upgrade/evidence/phase90a-runtime-correctness.json diff --git a/upgrade/evidence/phase90a-runtime-correctness.json b/upgrade/evidence/phase90a-runtime-correctness.json new file mode 100644 index 0000000..3513f3d --- /dev/null +++ b/upgrade/evidence/phase90a-runtime-correctness.json @@ -0,0 +1,85 @@ +{ + "schema_version": "qdl.phase-evidence.v1", + "phase": "9.0-A", + "name": "Runtime Correctness Closure", + "captured_at": "2026-08-18T11:05:00Z", + "status": "PASS_ISOLATED_NO_PRODUCTION_CUTOVER", + "branch": "feat/phase9-runtime-correctness", + "commits": [ + "765a7f1", + "7302c45" + ], + "tests": { + "targeted": {"run": 35, "passed": 35, "failed": 0}, + "full": {"run": 345, "passed": 340, "failed": 0, "skipped": 5}, + "compileall": "PASS", + "diff_check": "PASS" + }, + "v1_contract": { + "live_path_count": 40, + "candidate_path_count": 40, + "added_paths": [], + "removed_paths": [] + }, + "candidate": { + "source_commit": "7302c45", + "image_digest": "sha256:4a2723ec39057c75a89889d955feac7acc6fb01bc126a579f8c74d384b9b6999", + "runtime_uid_gid": "10001:10001", + "read_only_root": true, + "source_bind": false, + "loopback_port": 18120, + "limits": {"cpus": 1.5, "memory_bytes": 1610612736, "pids": 256} + }, + "real_provider_smoke": { + "binance_usdm_trade": {"shards": 8, "producing": 8, "ready": true}, + "binance_usdm_kline": {"shards": 8, "producing": 0, "ready": false}, + "top_level_health_with_kline_unavailable": "degraded", + "queue_drop_count": 0, + "queue_pressure_count": 0, + "recovery": { + "symbol": "BTCUSDT", + "interval": "1m", + "event": "kline_recovery", + "recovery_source": "BINANCE_REST_GAP_FILL", + "provider": "binance", + "market": "binance_usdm", + "authoritative": true, + "final": true, + "open_time_ms": 1787050680000, + "close_time_ms": 1787050739999, + "provider_open_time_equal": true, + "provider_close_time_equal": true, + "provider_ohlcv_equal": true + }, + "demand_expiry": { + "lease_count_after_expiry": 0, + "active_recovery_demands_after_expiry": 0, + "provider_fetch_count_before_observation": 2, + "provider_fetch_count_after_one_poll": 2 + } + }, + "resource_snapshot": { + "candidate_app": {"cpu_percent": 20.07, "memory_mib": 131.0, "pids": 16}, + "candidate_redis": {"cpu_percent": 2.27, "memory_mib": 5.129, "pids": 5}, + "note": "One non-soak snapshot while broad trade and failing kline shards were active." + }, + "production_v1": { + "container_id": "0e0eb56c78ba9d6b816cde3461bc79425ceea8095a71518a608a87294388035f", + "image": "data-layer:v0.1.0", + "started_at": "2026-08-16T11:09:12.471324413Z", + "loopback_port": 8100, + "restarted": false, + "state_mutated": false + }, + "cleanup": { + "candidate_containers": 0, + "candidate_networks": 0, + "candidate_volumes": 0, + "candidate_images": 0 + }, + "decision": { + "phase_90a": "GO_FOR_REVIEW_AND_OPERATOR_APPROVED_ROLLOUT", + "phase_91": "BLOCKED", + "reason": "Phase 9.1 still requires the independent production infrastructure and exact-slice approval listed in the implementation plan." + } +} diff --git a/upgrade/evidence/phase90b-capacity.json b/upgrade/evidence/phase90b-capacity.json new file mode 100644 index 0000000..3107ff2 --- /dev/null +++ b/upgrade/evidence/phase90b-capacity.json @@ -0,0 +1,197 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "burst": { + "concurrency": 20, + "errors": 0, + "latency_ms": { + "p50": 282.888, + "p95": 417.75, + "p99": 486.649, + "p99_9": 486.649 + }, + "requests": 60, + "requests_per_second": 65.012, + "response_bytes_per_second": 4072560.251 + }, + "checks": { + "auth_fail_closed": true, + "burst_rate": true, + "cleanup_exact": true, + "container_cpu_bounded": true, + "container_rss_bounded": true, + "cursor_expiry": true, + "cursor_scope": true, + "cursor_tamper": true, + "dependency_fail_closed": true, + "dependency_readiness_degraded": true, + "dependency_recovered": true, + "durable_store_growth_bounded": true, + "freshness": true, + "malformed": true, + "normal_rate": true, + "oversized": true, + "partition_epoch_advanced": true, + "production_beta_keys_absent": true, + "query_errors": true, + "query_tail": true, + "rate_limit": true, + "redis_growth_bounded": true, + "rotation": true, + "slow_consumer_isolated": true, + "stream_contiguous": true, + "stream_drained": true, + "v1_fallback_available": true, + "v1_topology_unchanged": true + }, + "decision_scope": "READ_ONLY_PUBLIC_BETA", + "end_to_end_freshness_ms": 6695.751, + "error_budget_consumption": 0.0, + "generated_market_events": 0, + "normal": { + "concurrency": 5, + "errors": 0, + "latency_ms": { + "p50": 39.419, + "p95": 63.325, + "p99": 72.211, + "p99_9": 72.211 + }, + "requests": 30, + "requests_per_second": 110.204, + "response_bytes_per_second": 6903513.048 + }, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "resources": { + "containers": { + "qdl_phase90b_matrix-qdl_beta_redis-1": { + "cpu_percent_peak": 3.52, + "network_rx_bytes_peak": 108000.0, + "network_tx_bytes_peak": 25600.0, + "rss_bytes_peak": 3895459.84 + }, + "qdl_phase90b_matrix-qdl_query_v2_beta-1": { + "cpu_percent_peak": 33.36, + "network_rx_bytes_peak": 1320000.0, + "network_tx_bytes_peak": 5850000.0, + "rss_bytes_peak": 55658414.08 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_a-1": { + "cpu_percent_peak": 9.96, + "network_rx_bytes_peak": 114000.0, + "network_tx_bytes_peak": 320000.0, + "rss_bytes_peak": 55364812.8 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_b-1": { + "cpu_percent_peak": 0.19, + "network_rx_bytes_peak": 4370.0, + "network_tx_bytes_peak": 5320.0, + "rss_bytes_peak": 52963573.76 + } + }, + "durable_store_growth_bytes": 73728, + "max_cpu_percent_of_one_core": 33.36, + "max_rss_bytes": 55658414.08, + "redis_growth_bytes": -35392 + }, + "runtime_failure_and_cleanup": { + "beta_containers_after": 0, + "beta_networks_after": 0, + "beta_volumes_after": 0, + "cursor_files_after": 0, + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "durable_store_bytes_after": 233472, + "durable_store_bytes_before": 159744, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "production_beta_keys_after": 0, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "redis_used_memory_after": 1174600, + "redis_used_memory_before": 1209992, + "schema": "qdl.phase7.3.runtime-evidence.v1", + "v1_fallback_status": 200, + "v1_topology_unchanged": true + }, + "schema": "qdl.phase7.capacity.v1", + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "stream": { + "bytes_per_second": 978613.568, + "cursor_lag_after": 0, + "cursor_lag_before": 64, + "disconnect_count": 1, + "events_per_second": 1815.252, + "fanout_consumers": 4, + "fast_consumers": 3, + "fast_results": [ + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 322.573, + "p95": 366.45, + "p99": 414.929, + "p99_9": 414.929 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 335.833, + "p95": 375.993, + "p99": 415.266, + "p99_9": 415.266 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 352.33, + "p95": 385.548, + "p99": 415.301, + "p99_9": 415.301 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + } + ], + "replay_count": 4, + "replay_lag_after": 0, + "replayed_events_per_fast_consumer": 66, + "slow_consumer_events_before_disconnect": 65, + "slow_consumer_explicit_disconnect": true, + "slow_consumers": 1, + "subscriber_peak": 4 + }, + "thresholds": { + "burst_min_requests_per_second": 20.0, + "max_container_cpu_percent_of_one_core": 100.0, + "max_container_rss_bytes": 536870912, + "max_durable_store_growth_bytes": 33554432, + "max_end_to_end_freshness_ms": 240000, + "max_error_budget_fraction": 0.0, + "max_redis_growth_bytes": 16777216, + "normal_min_requests_per_second": 10.0, + "query_max_p99_9_ms": 1000.0 + } +} diff --git a/upgrade/evidence/phase90b-continuous-bridge.json b/upgrade/evidence/phase90b-continuous-bridge.json new file mode 100644 index 0000000..7aded5d --- /dev/null +++ b/upgrade/evidence/phase90b-continuous-bridge.json @@ -0,0 +1,32 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "canonical_mismatches": 0, + "duplicate_open_times": 0, + "execution_eligible_events": 0, + "first": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "generated_market_events": 0, + "non_final_bars": 0, + "schema": "qdl.phase9.0-b.bridge-parity.v1", + "second": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "slice": { + "feed": "BAR", + "interval": "1m", + "market": "USDM", + "product_type": "PERPETUAL", + "symbol": "BTCUSDT", + "venue": "BINANCE" + }, + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "watermark_delta": 0 +} diff --git a/upgrade/evidence/phase90b-evidence.sha256 b/upgrade/evidence/phase90b-evidence.sha256 new file mode 100644 index 0000000..a165209 --- /dev/null +++ b/upgrade/evidence/phase90b-evidence.sha256 @@ -0,0 +1,5 @@ +7079cc3c349ff84b0f9c9e5f0b346098b68328514a8c0da31d7d467225c55d5d upgrade/evidence/phase90b-capacity.json +1a3d30f6fa4085b9fee7cdad0565ea2fba5fb3273ba690de881073f923f59bb4 upgrade/evidence/phase90b-security-adversarial.json +10bc592d28d821187f93e81f9159266f5079803d8e1a8737d997480b739e3a7b upgrade/evidence/phase90b-continuous-bridge.json +76e196fc06c86045409269cd1d00ead0c4d0f6eac753e850deedee4b47ca5742 upgrade/evidence/phase90b-isolated-v2-beta.json +2985867dd88e3bced8d31b341709f5b23a818be6e4465eaf3e4da5e20777388a upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md diff --git a/upgrade/evidence/phase90b-isolated-v2-beta.json b/upgrade/evidence/phase90b-isolated-v2-beta.json new file mode 100644 index 0000000..b65945a --- /dev/null +++ b/upgrade/evidence/phase90b-isolated-v2-beta.json @@ -0,0 +1,345 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "candidate": { + "bridge_networks": [ + "bobby_network", + "qdl_phase90b_bridge_qdl_beta_internal" + ], + "image_id": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "read_only_root": true, + "release": "phase90b-isolated-beta", + "runtime_user": "10001:10001", + "source_bind_mounts": 0, + "source_revision": "1c881389b4ee21a153903505822c61512b176044" + }, + "checks": { + "bridge_network_isolated": true, + "candidate_no_source_bind": true, + "candidate_non_root": true, + "candidate_read_only": true, + "candidate_revision_labeled": true, + "cleanup_exact": true, + "continuous_bridge_parity": true, + "phase73_capacity": true, + "phase73_security": true, + "production_beta_keys_absent": true, + "provider_authentic": true, + "v1_openapi_unchanged": true, + "v1_topology_unchanged": true + }, + "cleanup": { + "containers": 0, + "images": 0, + "networks": 0, + "production_beta_keys": 0, + "volumes": 0 + }, + "continuous_bridge": { + "authority": "V1_SHADOW_READ_ONLY", + "canonical_mismatches": 0, + "duplicate_open_times": 0, + "execution_eligible_events": 0, + "first": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "generated_market_events": 0, + "non_final_bars": 0, + "schema": "qdl.phase9.0-b.bridge-parity.v1", + "second": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "slice": { + "feed": "BAR", + "interval": "1m", + "market": "USDM", + "product_type": "PERPETUAL", + "symbol": "BTCUSDT", + "venue": "BINANCE" + }, + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "watermark_delta": 0 + }, + "contract_and_capacity": { + "authority": "V1_SHADOW_READ_ONLY", + "burst": { + "concurrency": 20, + "errors": 0, + "latency_ms": { + "p50": 282.888, + "p95": 417.75, + "p99": 486.649, + "p99_9": 486.649 + }, + "requests": 60, + "requests_per_second": 65.012, + "response_bytes_per_second": 4072560.251 + }, + "checks": { + "auth_fail_closed": true, + "burst_rate": true, + "cleanup_exact": true, + "container_cpu_bounded": true, + "container_rss_bounded": true, + "cursor_expiry": true, + "cursor_scope": true, + "cursor_tamper": true, + "dependency_fail_closed": true, + "dependency_readiness_degraded": true, + "dependency_recovered": true, + "durable_store_growth_bounded": true, + "freshness": true, + "malformed": true, + "normal_rate": true, + "oversized": true, + "partition_epoch_advanced": true, + "production_beta_keys_absent": true, + "query_errors": true, + "query_tail": true, + "rate_limit": true, + "redis_growth_bounded": true, + "rotation": true, + "slow_consumer_isolated": true, + "stream_contiguous": true, + "stream_drained": true, + "v1_fallback_available": true, + "v1_topology_unchanged": true + }, + "decision_scope": "READ_ONLY_PUBLIC_BETA", + "end_to_end_freshness_ms": 6695.751, + "error_budget_consumption": 0.0, + "generated_market_events": 0, + "normal": { + "concurrency": 5, + "errors": 0, + "latency_ms": { + "p50": 39.419, + "p95": 63.325, + "p99": 72.211, + "p99_9": 72.211 + }, + "requests": 30, + "requests_per_second": 110.204, + "response_bytes_per_second": 6903513.048 + }, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "resources": { + "containers": { + "qdl_phase90b_matrix-qdl_beta_redis-1": { + "cpu_percent_peak": 3.52, + "network_rx_bytes_peak": 108000.0, + "network_tx_bytes_peak": 25600.0, + "rss_bytes_peak": 3895459.84 + }, + "qdl_phase90b_matrix-qdl_query_v2_beta-1": { + "cpu_percent_peak": 33.36, + "network_rx_bytes_peak": 1320000.0, + "network_tx_bytes_peak": 5850000.0, + "rss_bytes_peak": 55658414.08 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_a-1": { + "cpu_percent_peak": 9.96, + "network_rx_bytes_peak": 114000.0, + "network_tx_bytes_peak": 320000.0, + "rss_bytes_peak": 55364812.8 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_b-1": { + "cpu_percent_peak": 0.19, + "network_rx_bytes_peak": 4370.0, + "network_tx_bytes_peak": 5320.0, + "rss_bytes_peak": 52963573.76 + } + }, + "durable_store_growth_bytes": 73728, + "max_cpu_percent_of_one_core": 33.36, + "max_rss_bytes": 55658414.08, + "redis_growth_bytes": -35392 + }, + "runtime_failure_and_cleanup": { + "beta_containers_after": 0, + "beta_networks_after": 0, + "beta_volumes_after": 0, + "cursor_files_after": 0, + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "durable_store_bytes_after": 233472, + "durable_store_bytes_before": 159744, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "production_beta_keys_after": 0, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "redis_used_memory_after": 1174600, + "redis_used_memory_before": 1209992, + "schema": "qdl.phase7.3.runtime-evidence.v1", + "v1_fallback_status": 200, + "v1_topology_unchanged": true + }, + "schema": "qdl.phase7.capacity.v1", + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "stream": { + "bytes_per_second": 978613.568, + "cursor_lag_after": 0, + "cursor_lag_before": 64, + "disconnect_count": 1, + "events_per_second": 1815.252, + "fanout_consumers": 4, + "fast_consumers": 3, + "fast_results": [ + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 322.573, + "p95": 366.45, + "p99": 414.929, + "p99_9": 414.929 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 335.833, + "p95": 375.993, + "p99": 415.266, + "p99_9": 415.266 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 352.33, + "p95": 385.548, + "p99": 415.301, + "p99_9": 415.301 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + } + ], + "replay_count": 4, + "replay_lag_after": 0, + "replayed_events_per_fast_consumer": 66, + "slow_consumer_events_before_disconnect": 65, + "slow_consumer_explicit_disconnect": true, + "slow_consumers": 1, + "subscriber_peak": 4 + }, + "thresholds": { + "burst_min_requests_per_second": 20.0, + "max_container_cpu_percent_of_one_core": 100.0, + "max_container_rss_bytes": 536870912, + "max_durable_store_growth_bytes": 33554432, + "max_end_to_end_freshness_ms": 240000, + "max_error_budget_fraction": 0.0, + "max_redis_growth_bytes": 16777216, + "normal_min_requests_per_second": 10.0, + "query_max_p99_9_ms": 1000.0 + } + }, + "decision_scope": "V2_BETA_READ_ONLY_REVIEW", + "phase_91": "BLOCKED", + "production_v1": { + "openapi_sha256": "142f5beb499ce7a27bf296462025ce65a6b204ee1171afd7511bce353bf79e38", + "restarted_or_recreated": false, + "topology_unchanged": true + }, + "resource_snapshot": [ + { + "CPUPerc": "2.17%", + "MemPerc": "3.50%", + "MemUsage": "3.363MiB / 96MiB", + "Name": "qdl_phase90b_bridge-qdl_beta_redis-1", + "NetIO": "16kB / 3.92kB", + "PIDs": "6" + }, + { + "CPUPerc": "0.00%", + "MemPerc": "0.00%", + "MemUsage": "0B / 0B", + "Name": "qdl_phase90b_bridge-qdl_beta_v1_bridge-1", + "NetIO": "0B / 0B", + "PIDs": "0" + }, + { + "CPUPerc": "8.43%", + "MemPerc": "10.51%", + "MemUsage": "53.79MiB / 512MiB", + "Name": "qdl_phase90b_bridge-qdl_query_v2_beta-1", + "NetIO": "12.5kB / 258kB", + "PIDs": "8" + }, + { + "CPUPerc": "0.16%", + "MemPerc": "9.83%", + "MemUsage": "50.33MiB / 512MiB", + "Name": "qdl_phase90b_bridge-qdl_stream_v2_beta_a-1", + "NetIO": "443kB / 12.6kB", + "PIDs": "14" + }, + { + "CPUPerc": "0.56%", + "MemPerc": "9.97%", + "MemUsage": "51.04MiB / 512MiB", + "Name": "qdl_phase90b_bridge-qdl_stream_v2_beta_b-1", + "NetIO": "443kB / 11.6kB", + "PIDs": "14" + } + ], + "schema": "qdl.phase9.0-b.isolated-beta.v1", + "security": { + "authority": "V1_SHADOW_READ_ONLY", + "schema": "qdl.phase7.security-adversarial.v1", + "status": "PASS", + "tests": { + "auth_status_codes": { + "consumer_mismatch": 403, + "missing_scope": 401, + "missing_token": 401, + "rotation_beta-jwt-k1": 200, + "rotation_beta-jwt-k2": 200, + "wrong_audience": 401, + "wrong_environment": 401 + }, + "cursor_consumer_scope": "CURSOR_INVALID", + "cursor_expiry": "CURSOR_EXPIRED", + "cursor_tamper": "CURSOR_INVALID", + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "malformed_request_status": 400, + "oversized_request_status": 413, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "rate_limit_requests_until_reject": 119, + "rate_limit_status": 429 + } + }, + "status": "PASS_ISOLATED_NO_AUTHORITY_CUTOVER" +} diff --git a/upgrade/evidence/phase90b-security-adversarial.json b/upgrade/evidence/phase90b-security-adversarial.json new file mode 100644 index 0000000..a61ff15 --- /dev/null +++ b/upgrade/evidence/phase90b-security-adversarial.json @@ -0,0 +1,28 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "schema": "qdl.phase7.security-adversarial.v1", + "status": "PASS", + "tests": { + "auth_status_codes": { + "consumer_mismatch": 403, + "missing_scope": 401, + "missing_token": 401, + "rotation_beta-jwt-k1": 200, + "rotation_beta-jwt-k2": 200, + "wrong_audience": 401, + "wrong_environment": 401 + }, + "cursor_consumer_scope": "CURSOR_INVALID", + "cursor_expiry": "CURSOR_EXPIRED", + "cursor_tamper": "CURSOR_INVALID", + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "malformed_request_status": 400, + "oversized_request_status": 413, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "rate_limit_requests_until_reject": 119, + "rate_limit_status": 429 + } +} diff --git a/upgrade/evidence/phase90c-authority-migration.json b/upgrade/evidence/phase90c-authority-migration.json new file mode 100644 index 0000000..0e8aa04 --- /dev/null +++ b/upgrade/evidence/phase90c-authority-migration.json @@ -0,0 +1,17 @@ +{ + "audit_mutation_rejected": true, + "audit_records": 2, + "candidate_mismatch_rejected": true, + "final_test_state": "RUST_CANARY:3:rust-canary-owner:2", + "hold_beyond_bundle_expiry_rejected": true, + "idempotent_migration": true, + "missing_hold_window_rejected": true, + "missing_terminal_watermark_rejected": true, + "no_go_bundle_rejected": true, + "prerequisite_bundles": 3, + "production_mutations": 0, + "schema": "qdl.phase9.0-c.authority-migration.v1", + "stale_cas_rejected": true, + "stale_owner_rejected": true, + "status": "PASS" +} diff --git a/upgrade/evidence/phase90c-evidence.sha256 b/upgrade/evidence/phase90c-evidence.sha256 new file mode 100644 index 0000000..338151d --- /dev/null +++ b/upgrade/evidence/phase90c-evidence.sha256 @@ -0,0 +1,3 @@ +df03cc15c8f3eba20ef3a64633412ab6e92e5c68f437dfd83c575e7921fa8a14 upgrade/evidence/phase90c-production-prerequisites.json +3953e06ff24ca38bae3a747da5dbceb55264243f4979b2348fa74c50cba15bc4 upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md +00e35812878357974a1c053c9f10a03bc58645ff83f3dc9768fd14c9cefd3f2d upgrade/evidence/phase90c-authority-migration.json diff --git a/upgrade/evidence/phase90c-production-prerequisites.json b/upgrade/evidence/phase90c-production-prerequisites.json new file mode 100644 index 0000000..629b0b2 --- /dev/null +++ b/upgrade/evidence/phase90c-production-prerequisites.json @@ -0,0 +1,171 @@ +{ + "authority_state": "RUST_SHADOW", + "blocked": 12, + "bundle_id": "558042db-a766-5a55-b5b3-4b508d649df9", + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "decision": "NO_GO_EXTERNAL", + "gates": [ + { + "evidence_id": "phase8-local-kafka-failover", + "gate_id": "replicated_durable_transport", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-local-otel-debug", + "gate_id": "production_observability", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-local-kafka-mtls", + "gate_id": "workload_identity_rbac_network", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "external_secret_rotation", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-signature-rehearsal-no-admission", + "gate_id": "signed_artifact_admission", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "EVIDENCE_BLOCKED", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "postgres_pitr", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "object_store_restore", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "independent_failure_domain_dr", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "INDEPENDENT_FAILURE_DOMAIN" + }, + { + "evidence_id": "phase8-local-projection-rebuild", + "gate_id": "redis_projector_rebuild", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase90b-partial-consumer-beta", + "gate_id": "consumer_registration_rollback", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "EVIDENCE_BLOCKED", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-local-authority-rehearsal", + "gate_id": "persistent_authority_sink_fencing", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "exact_slice_approval", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + } + ], + "inventory_sha256": "9447bc6775d359f5f38d521441373b93bcb95492f0cd5edf434fceb82fec160a", + "issued_at_ns": 1787065065239561211, + "passed": 0, + "policy_revision": 1, + "production_mutations": 0, + "schema": "qdl.production-prerequisite-decision.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "v1_after": { + "id": "0e0eb56c78ba9d6b816cde3461bc79425ceea8095a71518a608a87294388035f", + "image": "data-layer:v0.1.0", + "mounts": [ + [ + "/home/bobby/data_layer", + "/app", + true + ], + [ + "/home/bobby/data_layer/data", + "/app/data", + true + ], + [ + "/home/bobby/data_layer/logs", + "/app/logs", + true + ] + ], + "networks": [ + "bobby_network", + "executor_network" + ], + "present": true, + "restart_count": 0, + "started_at": "2026-08-16T11:09:12.471324413Z" + }, + "v1_before": { + "id": "0e0eb56c78ba9d6b816cde3461bc79425ceea8095a71518a608a87294388035f", + "image": "data-layer:v0.1.0", + "mounts": [ + [ + "/home/bobby/data_layer", + "/app", + true + ], + [ + "/home/bobby/data_layer/data", + "/app/data", + true + ], + [ + "/home/bobby/data_layer/logs", + "/app/logs", + true + ] + ], + "networks": [ + "bobby_network", + "executor_network" + ], + "present": true, + "restart_count": 0, + "started_at": "2026-08-16T11:09:12.471324413Z" + }, + "v1_health_after": 200, + "v1_health_before": 200, + "v1_unchanged": true +} diff --git a/upgrade/evidence/phase91-evidence.sha256 b/upgrade/evidence/phase91-evidence.sha256 new file mode 100644 index 0000000..0a19262 --- /dev/null +++ b/upgrade/evidence/phase91-evidence.sha256 @@ -0,0 +1,2 @@ +1b83b1f4533f618f5eb66a0806c85fbe83204c52ce43c312d5fb58fe368ba7ed upgrade/evidence/phase91-rust-canary-certification.json +7f2a1e79e3185151c484a6a27094b8410e5b94e73849ada8aa8dbdf092aa9581 upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md diff --git a/upgrade/evidence/phase91-rust-canary-certification.json b/upgrade/evidence/phase91-rust-canary-certification.json new file mode 100644 index 0000000..15d547e --- /dev/null +++ b/upgrade/evidence/phase91-rust-canary-certification.json @@ -0,0 +1,336 @@ +{ + "broker": { + "authority_audit_states_after_restart": [ + "RUST_SHADOW", + "RUST_CANARY", + "BLOCKED", + "RUST_SHADOW" + ], + "cleanup": { + "containers_after": 0, + "networks_after": 0, + "v1_health_after": 200, + "v1_health_before": 200, + "v1_topology_unchanged": true, + "volumes_after": 0 + }, + "final_authority": "RUST_SHADOW", + "image_digest": "sha256:b1dcd12ca123a61f11aef99a860f45eca7a012eac7df07ccfd3e9179519c952d", + "latest_authority_after_restart": { + "lease_epoch": 3, + "owner_id": "rust-ingestor-binance-usdm-shard-0", + "revision": 4, + "state": "RUST_SHADOW" + }, + "legacy_writes": 0, + "min_isr_failed_closed": true, + "mode": "ISOLATED_REHEARSAL", + "offset_deltas": { + "audit": 4, + "authority": 4, + "canary": 64, + "legacy": 0, + "public": 0, + "shadow": 2, + "shadow_raw": 0, + "transport": 0 + }, + "one_replica_loss_acked": true, + "production_authorized": false, + "public_writes": 0, + "runtime": { + "audit_offsets": [ + 0, + 1, + 2, + 3 + ], + "authority_offsets": [ + 0, + 1, + 2, + 3 + ], + "canary_offsets": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63 + ], + "checks": { + "blocked_state_rejected": true, + "canary_after_rollback_rejected": true, + "duplicate_watermark_rejected": true, + "legacy_target_rejected": true, + "public_target_rejected": true, + "stale_lease_rejected": true, + "stale_owner_rejected": true, + "stale_revision_rejected": true, + "topic_masquerade_rejected": true, + "wrong_plan_rejected": true + }, + "final_authority": "RUST_SHADOW", + "legacy_writes": 0, + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "public_writes": 0, + "schema": "qdl.phase91.isolated-canary-runtime.v1", + "shadow_offsets": [ + 0, + 1 + ], + "status": "PASS", + "transitions": [ + "RUST_SHADOW", + "RUST_CANARY", + "BLOCKED", + "RUST_SHADOW" + ] + }, + "schema": "qdl.phase91.broker-rehearsal.v1", + "slow_consumer": { + "catchup_seconds": 9.05500720598502, + "delayed_seconds": 1.0, + "ordered_gap_free": true, + "records": 64 + }, + "status": "PASS" + }, + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "issued_at_ns": 1787074909679119428, + "parity": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "capture": { + "capture_provenance": "REAL_PROVIDER_READ_ONLY", + "capture_sha256": "0912db1d39ddf1ec27414bccc55e096467305534aa581f9d217b179b7a95ff46", + "events": 25600, + "fixtures": 128, + "path": "/home/bobby/data_layer/target/phase91-authentic-replay.json", + "raw_checks_passed": 128, + "repeat": 200 + }, + "process_restart_mismatches": 0, + "python": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "events": 25600, + "events_per_second": 27115.454871588117, + "fixture_count": 128, + "latency_ms": { + "max": 0.241724, + "mean": 0.0360072491796875, + "p50": 0.030751, + "p95": 0.056433, + "p99": 0.068047 + }, + "record_sha256": [ + "0bdb7fb7f13c14f0f8e9adc9fd2762d93fde3c4211fb7cfde06d27dd2098965e", + "551a794866d0add8b53215d1175c603eb83b6cef094264dd79fe57405a35153f", + "983d9517c9a6925f934787af9fd18b9b9ce3343eaf32efe527e7af8853b002ee", + "d5e683845965866661309e4c56ab8b82ab59828ea9de79fd257f1465c139bdd8", + "4291a4187edb4482e5952772a34376688f009ef592dd26f193868118f28fe8b1", + "3499e9e5b0e05b13ec2ebfafc9891aa0a78568766a89e04e6fdadc06bb9f7987", + "e4fed6cefbe203bae15c75ae491959dc19540c290081cea68e177f4a25b8d36f", + "98ca85a273dddba4773a36aa0f0bf5ca4f7ecb53a716b7b85f47fc5b69dfe029", + "d0030c3743a7394203851b84a09f728ebf5ad9dc6a3c44e9e99474c9d00dd5fc", + "18fa31a2ee50afd6c5f2c6d631e95e1d59dd6cb944b7fbb12cd7cde6b82efa54", + "fec2a3bf544416f3c5bc73258715cb4ea21ad52cd21044756dfb237cb17f41e8", + "e65f8fd01f61a9b3af0c6a9a73ab43a63ab9413af5025f9292f2f8ca6da5eb12", + "8e0964240cfd3f3c625455e328d5e11d3e1c902bc5496ddedf9180ec2b0c231c", + "4c92557bde9d311deaff2b372cf9bfeec4ae40055c51c9834ccd1a050499b031", + "34fadfc58a9c8aa8ce38d81983dd52a2be4464558f493f58a5e3c597aafbadb8", + "36fc31328c68bf18e79735f67235f16f7b23a9f29033a92b97739c97f6fb967e", + "3bd1bcf2e054a82ece63c7b5801496928af5f6c1a337ee5e5fec3e5ab2c92837", + "5a9d8ea5216d8b29d0d0d0e327fef2ccfc942ef4a7a2789ca8901babd460d8e2", + "a0860ae4eb295cbc3bd1d7bc7b85030b4631f2021a3b02be56333bd227f6f926", + "1ff9dccf639376644434fa9ac45e26a810451931bf913d4f4a3166b15efb0a4c", + "57aafc884bc978aaadf74353dff92c44f65cc795e37a0088e63b2c844554596e", + "d9cb053df4fd712150e49793d175e0596cb6312ac548c81d322554fbc6fef54d", + "ae5444e073196a2cf2df20e31747c9a7ec89a81ab26d80043282af5bc4419f41", + "e219bc7075d9ae435afa1d6bc64c4314eba08b3e4c51c96c2e0b29ee215ad1d2", + "d2a4af9a664822bbfc184351eef4af4e7717d317dd1d7b99b54bc78ad92fb72c", + "8e5225db6f290adcf08c9216cad680c2e8e5ba9a23abb51a4f54fc2d0b8f150e", + "f991258b33c0f16e35d63b20d23fabe582c0b13558c5ff56ff4fbb3aa7fa286c", + "c265bed006771b5200a4cf7a250e26b648fc4744f941c45a6f892f6484de8fff", + "dd10d1e5c2307cb03267ba449ff03ebeecf5d75882819714b600e073fa6b7df1", + "67d96275874837dc3e7c7c475a8d08268bb362e9d0400e86c8396ca7b07fc4a9", + "03e2362cc7127166da91a6c278a98e0910ab2241fd74edb57da3b2f298b2db18", + "3622b347a37cd57d1274690f5dbc2456590de6f0358137c520af9d587ed5e352", + "eaa2b05a35d3889a4c582b324e02d7e9b00071b0def6d0738d3ee0d835d67cbd", + "ef1a72bf7e51750d6a642235ffbc005435ec7f2a48a65990046777163a51a620", + "380fe178738ceef2bf0ba2150a67a23652c5fe69708bcb1ff97291e1ee5d9f74", + "1a0f78f988b0cfbc04ba6aef740974fb27a076ec2aef40a5147745910b0393a4", + "7e84855bf6c00861673ba5448abdbc26716be15f5e851cb39bd6b7b19a5f3637", + "5df2dc372d4cf6cf575a3db69fa978f3d5ac2cec52d32b5c3c1e91ecb2f6a41a", + "2ff4585edd1ef4c025a9c54022a3c934e5844378ca75bf652bd829c4ed0de29c", + "48ff53d77923f5d5d7c03252fe8dfcf29602d309523aef0f8247cd1e7c1b462e", + "460d6e07432d0bc001abc68fb2107d36d3cb6648762b1f047c1bff3dfb2d03b2", + "7e4f6801b3a6652488e21acfb91ac14967991f6366ada1dd8dca0485cb7417a3", + "043c1a1d71560fa8cb774dcc0c666123b8ef524d7093d709b245b23e518b4ee4", + "9e2644519865231fc427b5315aebc5b5c7ef444737638d9e44d43690e7b82582", + "d6ade1dbf0709e652059a243b8d279013a4c097e5ff77b081cefb129c2dc80eb", + "05f98d974b2d8f66391a40d1230c9e03939bb9fa2e8848f003b31e661bb4d6a0", + "a4c3c5f99e590e87c00bf3644f1471562f5fd98d86843b17e64dc255cbf2400a", + "8c67693ac5bafd2ba9fc61696cc05c95a20950b4bff17c7059ceb814226c8ff1", + "6d16b0937d44a855edb27c32282762532c08256e1c0837b537c335e9ff9959f1", + "acbf0f74f08c65fd1178ca0f85ad06314ed4846ebf3e05757a9eb5a2547fc66f", + "656824469ca227f57fe453bcbcb8c1b176e5b100607c4b815b4876946807a949", + "ac3496b3344dd446b53d6adc488634e769d4599e2af32ae4af3877f0ef1cb60c", + "02f854da9a1f86cea4c95a96f849dca76ef697960bbddd5220d800090e465d16", + "251133c40506d2dba35bea085dee236304ea4510755657e42b6864e01b123590", + "2d5be34c1d4afc4d0a87a62a60e8e569d1dd13751d8a0828e534a239d5e7ccba", + "e3aba260a49e59baa1c62c46366e1f1e8a8860c486ed564817262d9ab91724df", + "ca3f049bbf0d4b66280d177bc3ed97b80916d433774b6a49a23d4ce5074b3a1b", + "9837e2879d41c455fef469c8d6c7cf63f390ea2d6aa9e8f929ae15fb5ad3d565", + "bb8de7c97a8982fd560160f4ec2fa0809d6d6b83ace36e15339ec991f332b05b", + "1c8abda5df59dfae2f741258c4ba7df76d8582a0670ea252cf8c18cb826921c6", + "9b55c4a5dd05b9a639594a33cc69b5f58fdcfb11fe9f9fbc66eb313eac41423c", + "5e4d8b91e4d358d5f94abc5d21e2d5f1f1a95ac79352764ccf50e0d6c6360c75", + "9802feae89ea5472b98da05b967c2453254da040e1a167d76328fa1b8b78db58", + "850a46047de253357105115d178343f125063c3ffe3ae00e0a68ff3b81d887b2", + "e4f71e5d01a3359058d994321b1a329d81da3397c607aa9bd5054576481e2765", + "2272a110375bf01d89f927d4f5c27442dc8db7fec5192441286e1b9722efd9c2", + "4acf59e2f8f9b047da7b8a25beac1af3d7f310abb715504e26e45e5cb13fc2e8", + "08e76d3003712d63d8e97305a8e2989abc52e023c812ab7778576ff03a2b34bd", + "a76814f32c873839433c55cbda89bd784f6b9a7233bcef013157f985387438f6", + "dd4c3828e91eea9686908bbcaa029620a919ee69de56d7eadcc99999a64acf17", + "2ae7c9f69bd4e0851b3c72b2b293d9e62be821e1e0f297113e1e37177d52231b", + "2540ddcd884d4a368531bba0e0476f57d38824400cb784095203e2a4bea2854d", + "e17c7ea5bd68d973a98ba417a4450dd5ac663da71d4070d314f804b09468ce89", + "21353eff6fb732edacf862957bcc43f61128deb005586d888e20df199ce04bd3", + "59c7742b77fd416820ca0a1467365e20187516f7270d1af0ca445ad88f40f489", + "17d98f3eea38caa95397f2e4f9367eb2189441ec1d4e8f317fbcf461e1597d06", + "5064ff5d3a49218a8dccadf34bcbb47700e1a1a99d92f5bb762b497bab3e21a7", + "324c6c1d4e1016fadd95878d050ddbb83bf4732c6b19164c3439f9e3099e11a9", + "71100e2b4a46b1754605c7dd126d7153b3828e44fd1a153553a57dd18fb7dc33", + "e45fe5e59442d29bbac9c36c2e2aaa1c75647b2774dfbbc396018c139c80b63d", + "551c59a6cf45d806e50c26893403cbcab7068dcb561196517bd72303bbeffc10", + "4738c73066e3acbd013c068747dc092d68939ecde2638f0911fb0d375d38c563", + "d4499622a95575edd2f4fa93a4a47e878ede7bc4d277d5eaadc59b6971b237fd", + "f4355be81f0e3135855042bfee74b42eae46f4fd82a8cd70d439e753fa70c98e", + "104146496e9e67a631b9ed3ea9cd4bc2f91094b5838e43fdb68e14139bb3c9f9", + "6bcac04c6f40b3b4b4771ded2a82a44f4dfa7cdcd3c89ca3535d1ba6cc532a1f", + "3ab6f4fea8f409038c066a3ec98468e3ddad2f280843fe50c215809df0f41266", + "9629e26ed5f8eb8128b40b879de34518b8fca22d4d5fa626ca4e2ed0b012fce2", + "780750b17c2972f8b424ea8f0e33060fae16866818c06b2d128a1c32296ff645", + "8311b7e8e07478478a15f1cf0a028ed31dc34474c9d18e2c3bac77f9b6c72854", + "85a798fd9700c7c4dc29fa15bb324618c2f34c14b5ba1198ae37747cf41693d4", + "2739404a0f05f9abb956530f4a5d17ae111b7179513251355e17e5676615f649", + "9995532a889c42631008b8b2fe082d87cc5405d27d7af22d6e62fdd1b47439b3", + "5224ab418ae50ab6815ef572a6d4bd10d78e993b6049c1a1dd1e3f6f9814cab0", + "74f2db999392b12121a52b358434296da44bb89947ddafe5d40cec7beb3e414d", + "1c93beb39e5fbd261f5a6566f88b4cce6c8feeb4b9e9c91783fc8b4499189afb", + "ddc628df8adcb3165736d8545d6bc1c4452f2296ab878b056bd6511f227cd299", + "eb7084f1f8ef541f27fdebb9f8cb934bcd7edb9c8c6d12f60316f84acdd4b9b2", + "cde6acdf79d267ee5147c4e65d1edb4e0a07656b6f560578585211ae88883fc3", + "7449c220b3a41c6d7a5682ed7d7ccec2c4def46824eabbefd595a0e5dddaf9e9", + "5ed88151733674af61fd474cfedf15e7a8b7d1c7bce44928bda9cc2db858984a", + "24f59709d171dfa4d49b036a469208a87bf4515161aacf9c55b5c9935273b2f0", + "23efeef3e63975a236f69015a44a8b32b162570ad60ef107489aa1a1da5388db", + "f4e57fc11ac0bdb1235f5a49849d5e8c1d8a73f02b464b8861ae2134d29aeb1f", + "f2b2f29080ab68ffe39d8e86772c78b7c09a16024b72b925ebd824525803dcf9", + "e102f7739a5b7e75dd2adacddfe48580ea4a621182d0fa0ef865e483f881be94", + "25e4208a85c1dbbe8c9337f0fbdb8cab2655a9692846e9669e6b597cf79373c8", + "da4982e13d2a24665fe5cb4e3d11ee9f6b237d15a8d6f463fa599572ee01d5cd", + "0d224549cc794f72283c537d194efe10fda71597d5e8b6b1d6f0faf1f5688fe4", + "2378fb0751d0f21ecb91f5b1bd846d788058ad2a73200a33460e5b772cd417ae", + "e994028d841cd272458ce38f00aad5a93553e5ab36bd71995dc0ca2b2ed5c19e", + "eb1cec7e0abe56bcf4e0240560b3bcaedc6a4081dc7ca2e4a8e9e04c62dbf9c4", + "4cae7a4d07888ee8eec3f8b3a7d232f76eded1d5af4847fb6c31fab6e9700158", + "2861deaa154c6f6c93e536ddd6cf8a9b5d445204f96ef511616126bf32b2b4a8", + "a1195ef8b92d279923ecd86ce25cdfb905607ec533b323d8e0d0b1b5dc2902ac", + "68cfa7d0933d0261fa34a58ba1c224148d3133cbc237421dcc63b830a67e870b", + "bde390c96ff54978e603f67e05d115e57f928b0ffbe2a9e06f1907e293bef7a6", + "2e0347d26f139a07ab151f00c5968f27cf37e0abe748a4f7809cd7488b4816e7", + "8151f090f8a5f635fd3bcb4b42569919ddfe34b756c87610cdae9cf4d899ea52", + "042c82bdf6e09344c337ff9753ba9852f10180fcc2ec6fdab9431c48f2804a47", + "2238079e95be053c37ae41b00934decd06e4da07d9f86919ad5cfa810e5597ad", + "cffde788491b58ae16c8da3d13b4dc52bcf8bfd753211687460c53e224495249", + "6b83a2ed68f6921b6bd23cd05f9421914df684139dba33cc026e29c456fbc570", + "b9e6e92c6c876a9e80fd4d42fd6e49d814d1e79d6b8cf5c9b8d2cb0bd768cd8e", + "10c6905093b0d916e08870d42be0d50cf1f1f548235f421a690971d2ec12daab", + "a88672cf673b5d040aac8c63e80fed9e40e12ab19e18fdf1c641f7d0a29674ef", + "ead7b579320feaa42cc6c4e55c5e22adab2bf7934c0eb2cbb88db347362f7d59", + "f759753fa0211e131cb0ebb2ca40efe5296d680cd0328cdcccc00cd8028fac3d" + ], + "repeat": 200, + "schema": "qdl.phase91.python-parity.v1", + "status": "PASS" + }, + "rust_clean_process_runs": 3, + "rust_events_per_second_min": 350581.02505399904, + "semantic_mismatches": 0, + "status": "PASS" + }, + "prerequisite_bundle_id": "558042db-a766-5a55-b5b3-4b508d649df9", + "prerequisite_decision": "NO_GO_EXTERNAL", + "production_authorized": false, + "production_mutations": 0, + "python_v1_public_authority_unchanged": true, + "schema": "qdl.phase91.rust-canary-certification.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "status": "COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED", + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "same-host replicated broker rehearsal is not an independent production failure domain" + ] +} diff --git a/upgrade/evidence/phase92-authority-migration.json b/upgrade/evidence/phase92-authority-migration.json new file mode 100644 index 0000000..d666fc8 --- /dev/null +++ b/upgrade/evidence/phase92-authority-migration.json @@ -0,0 +1,16 @@ +{ + "accepted_handoffs": 2, + "audit_records": 4, + "checkpoint_delete_rejected": true, + "direct_primary_bypass_rejected": true, + "final_test_state": "PYTHON_PRIMARY:7:python-rollback:3:120", + "handoff_mutation_rejected": true, + "idempotent_migration": true, + "production_mutations": 0, + "python_to_rust_handoff_passed": true, + "rust_to_python_rollback_passed": true, + "schema": "qdl.phase92.authority-migration.v1", + "stale_cas_rejected": true, + "status": "PASS", + "terminal_checkpoints": 2 +} diff --git a/upgrade/evidence/phase92-bounded-primary-certification.json b/upgrade/evidence/phase92-bounded-primary-certification.json new file mode 100644 index 0000000..8436891 --- /dev/null +++ b/upgrade/evidence/phase92-bounded-primary-certification.json @@ -0,0 +1,582 @@ +{ + "broker": { + "accepted_handoffs": 2, + "authority_audit_states_after_restart": [ + "RUST_CANARY", + "RUST_PRIMARY", + "BLOCKED", + "ROLLBACK_PENDING", + "PYTHON_PRIMARY" + ], + "boundary_gap_free": true, + "cleanup": { + "containers_after": 0, + "networks_after": 0, + "v1_health_after": 200, + "v1_health_before": 200, + "v1_topology_unchanged": true, + "volumes_after": 0 + }, + "cutover_ms": 22.200226, + "final_authority": "PYTHON_PRIMARY", + "image_digest": "sha256:34508eda14e66134f96c4da3ea4da3fd42c46425ecf56815df3089c3d94eb67c", + "latest_authority_after_restart": { + "lease_epoch": 13, + "owner_id": "python-rollback-isolated", + "revision": 11, + "state": "PYTHON_PRIMARY" + }, + "min_isr_failed_closed": true, + "mode": "ISOLATED_REHEARSAL", + "offset_deltas": { + "audit": 5, + "authority": 5, + "checkpoint": 2, + "handoff": 2, + "legacy": 81, + "primary": 81, + "production_legacy": 0, + "production_public": 0, + "public": 81, + "transport": 0 + }, + "one_replica_loss_acked": true, + "owner_boundary_correct": true, + "process_restart_recovery": { + "authority_revision": 11, + "checks": { + "duplicate_after_restore_rejected": true, + "durable_target_watermarks_restored": true, + "each_target_pre_restore_failed_closed": true, + "restart_pre_restore_failed_closed": true, + "resumed_at_exact_next_watermark": true + }, + "mode": "RECOVERY_VERIFY", + "owner_id": "python-rollback-isolated", + "production_authorized": false, + "production_legacy_writes": 0, + "production_public_writes": 0, + "projection_offsets": [ + 80, + 80, + 80 + ], + "restored_target_watermarks": { + "legacy": 180, + "primary": 180, + "public": 180 + }, + "resumed_watermark": 181, + "schema": "qdl.phase92.process-restart-recovery.v1", + "status": "PASS" + }, + "production_authorized": false, + "production_legacy_writes": 0, + "production_public_writes": 0, + "projection_parity": true, + "rollback_ms": 533.2371999999999, + "runtime": { + "audit_offsets": [ + 0, + 1, + 2, + 3, + 4 + ], + "authority_offsets": [ + 0, + 1, + 2, + 3, + 4 + ], + "authority_transitions": [ + "RUST_CANARY", + "RUST_PRIMARY", + "BLOCKED", + "ROLLBACK_PENDING", + "PYTHON_PRIMARY" + ], + "checkpoint_offsets": [ + 0, + 1 + ], + "checks": { + "direct_primary_without_handoff_rejected": true, + "duplicate_after_ack_rejected": true, + "gap_watermark_rejected": true, + "rust_after_rollback_rejected": true, + "stale_lease_rejected": true, + "stale_owner_rejected": true, + "stale_revision_rejected": true, + "terminal_watermark_rejected": true, + "wrong_plan_rejected": true + }, + "cutover_ms": 22.200226, + "final_authority": "PYTHON_PRIMARY", + "first_python_rollback_watermark": 165, + "first_rust_watermark": 101, + "handoff_offsets": [ + 0, + 1 + ], + "isolated_legacy_writes": 80, + "isolated_primary_writes": 80, + "isolated_public_writes": 80, + "last_rust_watermark": 164, + "last_watermark": 180, + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "production_legacy_writes": 0, + "production_public_writes": 0, + "projection_offsets": [ + 0, + 0, + 0, + 1, + 1, + 1, + 2, + 2, + 2, + 3, + 3, + 3, + 4, + 4, + 4, + 5, + 5, + 5, + 6, + 6, + 6, + 7, + 7, + 7, + 8, + 8, + 8, + 9, + 9, + 9, + 10, + 10, + 10, + 11, + 11, + 11, + 12, + 12, + 12, + 13, + 13, + 13, + 14, + 14, + 14, + 15, + 15, + 15, + 16, + 16, + 16, + 17, + 17, + 17, + 18, + 18, + 18, + 19, + 19, + 19, + 20, + 20, + 20, + 21, + 21, + 21, + 22, + 22, + 22, + 23, + 23, + 23, + 24, + 24, + 24, + 25, + 25, + 25, + 26, + 26, + 26, + 27, + 27, + 27, + 28, + 28, + 28, + 29, + 29, + 29, + 30, + 30, + 30, + 31, + 31, + 31, + 32, + 32, + 32, + 33, + 33, + 33, + 34, + 34, + 34, + 35, + 35, + 35, + 36, + 36, + 36, + 37, + 37, + 37, + 38, + 38, + 38, + 39, + 39, + 39, + 40, + 40, + 40, + 41, + 41, + 41, + 42, + 42, + 42, + 43, + 43, + 43, + 44, + 44, + 44, + 45, + 45, + 45, + 46, + 46, + 46, + 47, + 47, + 47, + 48, + 48, + 48, + 49, + 49, + 49, + 50, + 50, + 50, + 51, + 51, + 51, + 52, + 52, + 52, + 53, + 53, + 53, + 54, + 54, + 54, + 55, + 55, + 55, + 56, + 56, + 56, + 57, + 57, + 57, + 58, + 58, + 58, + 59, + 59, + 59, + 60, + 60, + 60, + 61, + 61, + 61, + 62, + 62, + 62, + 63, + 63, + 63, + 64, + 64, + 64, + 65, + 65, + 65, + 66, + 66, + 66, + 67, + 67, + 67, + 68, + 68, + 68, + 69, + 69, + 69, + 70, + 70, + 70, + 71, + 71, + 71, + 72, + 72, + 72, + 73, + 73, + 73, + 74, + 74, + 74, + 75, + 75, + 75, + 76, + 76, + 76, + 77, + 77, + 77, + 78, + 78, + 78, + 79, + 79, + 79 + ], + "rollback_ms": 533.2371999999999, + "schema": "qdl.phase92.isolated-primary-runtime.v1", + "status": "PASS" + }, + "schema": "qdl.phase92.broker-rehearsal.v1", + "slow_consumer": { + "catchup_seconds": 24.523788021004293, + "delayed_seconds": 1.0, + "ordered_gap_free": true, + "records_per_projection": 81 + }, + "status": "PASS", + "terminal_checkpoints": 2 + }, + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "issued_at_ns": 1787081314573802100, + "migration": { + "accepted_handoffs": 2, + "audit_records": 4, + "checkpoint_delete_rejected": true, + "direct_primary_bypass_rejected": true, + "final_test_state": "PYTHON_PRIMARY:7:python-rollback:3:120", + "handoff_mutation_rejected": true, + "idempotent_migration": true, + "production_mutations": 0, + "python_to_rust_handoff_passed": true, + "rust_to_python_rollback_passed": true, + "schema": "qdl.phase92.authority-migration.v1", + "stale_cas_rejected": true, + "status": "PASS", + "terminal_checkpoints": 2 + }, + "parity": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "capture": { + "capture_provenance": "REAL_PROVIDER_READ_ONLY", + "capture_sha256": "0912db1d39ddf1ec27414bccc55e096467305534aa581f9d217b179b7a95ff46", + "events": 25600, + "fixtures": 128, + "path": "/home/bobby/data_layer/target/phase91-authentic-replay.json", + "raw_checks_passed": 128, + "repeat": 200 + }, + "process_restart_mismatches": 0, + "python": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "events": 25600, + "events_per_second": 28388.208223491194, + "fixture_count": 128, + "latency_ms": { + "max": 0.280434, + "mean": 0.034400821328125, + "p50": 0.030806, + "p95": 0.05559, + "p99": 0.065782 + }, + "record_sha256": [ + "0bdb7fb7f13c14f0f8e9adc9fd2762d93fde3c4211fb7cfde06d27dd2098965e", + "551a794866d0add8b53215d1175c603eb83b6cef094264dd79fe57405a35153f", + "983d9517c9a6925f934787af9fd18b9b9ce3343eaf32efe527e7af8853b002ee", + "d5e683845965866661309e4c56ab8b82ab59828ea9de79fd257f1465c139bdd8", + "4291a4187edb4482e5952772a34376688f009ef592dd26f193868118f28fe8b1", + "3499e9e5b0e05b13ec2ebfafc9891aa0a78568766a89e04e6fdadc06bb9f7987", + "e4fed6cefbe203bae15c75ae491959dc19540c290081cea68e177f4a25b8d36f", + "98ca85a273dddba4773a36aa0f0bf5ca4f7ecb53a716b7b85f47fc5b69dfe029", + "d0030c3743a7394203851b84a09f728ebf5ad9dc6a3c44e9e99474c9d00dd5fc", + "18fa31a2ee50afd6c5f2c6d631e95e1d59dd6cb944b7fbb12cd7cde6b82efa54", + "fec2a3bf544416f3c5bc73258715cb4ea21ad52cd21044756dfb237cb17f41e8", + "e65f8fd01f61a9b3af0c6a9a73ab43a63ab9413af5025f9292f2f8ca6da5eb12", + "8e0964240cfd3f3c625455e328d5e11d3e1c902bc5496ddedf9180ec2b0c231c", + "4c92557bde9d311deaff2b372cf9bfeec4ae40055c51c9834ccd1a050499b031", + "34fadfc58a9c8aa8ce38d81983dd52a2be4464558f493f58a5e3c597aafbadb8", + "36fc31328c68bf18e79735f67235f16f7b23a9f29033a92b97739c97f6fb967e", + "3bd1bcf2e054a82ece63c7b5801496928af5f6c1a337ee5e5fec3e5ab2c92837", + "5a9d8ea5216d8b29d0d0d0e327fef2ccfc942ef4a7a2789ca8901babd460d8e2", + "a0860ae4eb295cbc3bd1d7bc7b85030b4631f2021a3b02be56333bd227f6f926", + "1ff9dccf639376644434fa9ac45e26a810451931bf913d4f4a3166b15efb0a4c", + "57aafc884bc978aaadf74353dff92c44f65cc795e37a0088e63b2c844554596e", + "d9cb053df4fd712150e49793d175e0596cb6312ac548c81d322554fbc6fef54d", + "ae5444e073196a2cf2df20e31747c9a7ec89a81ab26d80043282af5bc4419f41", + "e219bc7075d9ae435afa1d6bc64c4314eba08b3e4c51c96c2e0b29ee215ad1d2", + "d2a4af9a664822bbfc184351eef4af4e7717d317dd1d7b99b54bc78ad92fb72c", + "8e5225db6f290adcf08c9216cad680c2e8e5ba9a23abb51a4f54fc2d0b8f150e", + "f991258b33c0f16e35d63b20d23fabe582c0b13558c5ff56ff4fbb3aa7fa286c", + "c265bed006771b5200a4cf7a250e26b648fc4744f941c45a6f892f6484de8fff", + "dd10d1e5c2307cb03267ba449ff03ebeecf5d75882819714b600e073fa6b7df1", + "67d96275874837dc3e7c7c475a8d08268bb362e9d0400e86c8396ca7b07fc4a9", + "03e2362cc7127166da91a6c278a98e0910ab2241fd74edb57da3b2f298b2db18", + "3622b347a37cd57d1274690f5dbc2456590de6f0358137c520af9d587ed5e352", + "eaa2b05a35d3889a4c582b324e02d7e9b00071b0def6d0738d3ee0d835d67cbd", + "ef1a72bf7e51750d6a642235ffbc005435ec7f2a48a65990046777163a51a620", + "380fe178738ceef2bf0ba2150a67a23652c5fe69708bcb1ff97291e1ee5d9f74", + "1a0f78f988b0cfbc04ba6aef740974fb27a076ec2aef40a5147745910b0393a4", + "7e84855bf6c00861673ba5448abdbc26716be15f5e851cb39bd6b7b19a5f3637", + "5df2dc372d4cf6cf575a3db69fa978f3d5ac2cec52d32b5c3c1e91ecb2f6a41a", + "2ff4585edd1ef4c025a9c54022a3c934e5844378ca75bf652bd829c4ed0de29c", + "48ff53d77923f5d5d7c03252fe8dfcf29602d309523aef0f8247cd1e7c1b462e", + "460d6e07432d0bc001abc68fb2107d36d3cb6648762b1f047c1bff3dfb2d03b2", + "7e4f6801b3a6652488e21acfb91ac14967991f6366ada1dd8dca0485cb7417a3", + "043c1a1d71560fa8cb774dcc0c666123b8ef524d7093d709b245b23e518b4ee4", + "9e2644519865231fc427b5315aebc5b5c7ef444737638d9e44d43690e7b82582", + "d6ade1dbf0709e652059a243b8d279013a4c097e5ff77b081cefb129c2dc80eb", + "05f98d974b2d8f66391a40d1230c9e03939bb9fa2e8848f003b31e661bb4d6a0", + "a4c3c5f99e590e87c00bf3644f1471562f5fd98d86843b17e64dc255cbf2400a", + "8c67693ac5bafd2ba9fc61696cc05c95a20950b4bff17c7059ceb814226c8ff1", + "6d16b0937d44a855edb27c32282762532c08256e1c0837b537c335e9ff9959f1", + "acbf0f74f08c65fd1178ca0f85ad06314ed4846ebf3e05757a9eb5a2547fc66f", + "656824469ca227f57fe453bcbcb8c1b176e5b100607c4b815b4876946807a949", + "ac3496b3344dd446b53d6adc488634e769d4599e2af32ae4af3877f0ef1cb60c", + "02f854da9a1f86cea4c95a96f849dca76ef697960bbddd5220d800090e465d16", + "251133c40506d2dba35bea085dee236304ea4510755657e42b6864e01b123590", + "2d5be34c1d4afc4d0a87a62a60e8e569d1dd13751d8a0828e534a239d5e7ccba", + "e3aba260a49e59baa1c62c46366e1f1e8a8860c486ed564817262d9ab91724df", + "ca3f049bbf0d4b66280d177bc3ed97b80916d433774b6a49a23d4ce5074b3a1b", + "9837e2879d41c455fef469c8d6c7cf63f390ea2d6aa9e8f929ae15fb5ad3d565", + "bb8de7c97a8982fd560160f4ec2fa0809d6d6b83ace36e15339ec991f332b05b", + "1c8abda5df59dfae2f741258c4ba7df76d8582a0670ea252cf8c18cb826921c6", + "9b55c4a5dd05b9a639594a33cc69b5f58fdcfb11fe9f9fbc66eb313eac41423c", + "5e4d8b91e4d358d5f94abc5d21e2d5f1f1a95ac79352764ccf50e0d6c6360c75", + "9802feae89ea5472b98da05b967c2453254da040e1a167d76328fa1b8b78db58", + "850a46047de253357105115d178343f125063c3ffe3ae00e0a68ff3b81d887b2", + "e4f71e5d01a3359058d994321b1a329d81da3397c607aa9bd5054576481e2765", + "2272a110375bf01d89f927d4f5c27442dc8db7fec5192441286e1b9722efd9c2", + "4acf59e2f8f9b047da7b8a25beac1af3d7f310abb715504e26e45e5cb13fc2e8", + "08e76d3003712d63d8e97305a8e2989abc52e023c812ab7778576ff03a2b34bd", + "a76814f32c873839433c55cbda89bd784f6b9a7233bcef013157f985387438f6", + "dd4c3828e91eea9686908bbcaa029620a919ee69de56d7eadcc99999a64acf17", + "2ae7c9f69bd4e0851b3c72b2b293d9e62be821e1e0f297113e1e37177d52231b", + "2540ddcd884d4a368531bba0e0476f57d38824400cb784095203e2a4bea2854d", + "e17c7ea5bd68d973a98ba417a4450dd5ac663da71d4070d314f804b09468ce89", + "21353eff6fb732edacf862957bcc43f61128deb005586d888e20df199ce04bd3", + "59c7742b77fd416820ca0a1467365e20187516f7270d1af0ca445ad88f40f489", + "17d98f3eea38caa95397f2e4f9367eb2189441ec1d4e8f317fbcf461e1597d06", + "5064ff5d3a49218a8dccadf34bcbb47700e1a1a99d92f5bb762b497bab3e21a7", + "324c6c1d4e1016fadd95878d050ddbb83bf4732c6b19164c3439f9e3099e11a9", + "71100e2b4a46b1754605c7dd126d7153b3828e44fd1a153553a57dd18fb7dc33", + "e45fe5e59442d29bbac9c36c2e2aaa1c75647b2774dfbbc396018c139c80b63d", + "551c59a6cf45d806e50c26893403cbcab7068dcb561196517bd72303bbeffc10", + "4738c73066e3acbd013c068747dc092d68939ecde2638f0911fb0d375d38c563", + "d4499622a95575edd2f4fa93a4a47e878ede7bc4d277d5eaadc59b6971b237fd", + "f4355be81f0e3135855042bfee74b42eae46f4fd82a8cd70d439e753fa70c98e", + "104146496e9e67a631b9ed3ea9cd4bc2f91094b5838e43fdb68e14139bb3c9f9", + "6bcac04c6f40b3b4b4771ded2a82a44f4dfa7cdcd3c89ca3535d1ba6cc532a1f", + "3ab6f4fea8f409038c066a3ec98468e3ddad2f280843fe50c215809df0f41266", + "9629e26ed5f8eb8128b40b879de34518b8fca22d4d5fa626ca4e2ed0b012fce2", + "780750b17c2972f8b424ea8f0e33060fae16866818c06b2d128a1c32296ff645", + "8311b7e8e07478478a15f1cf0a028ed31dc34474c9d18e2c3bac77f9b6c72854", + "85a798fd9700c7c4dc29fa15bb324618c2f34c14b5ba1198ae37747cf41693d4", + "2739404a0f05f9abb956530f4a5d17ae111b7179513251355e17e5676615f649", + "9995532a889c42631008b8b2fe082d87cc5405d27d7af22d6e62fdd1b47439b3", + "5224ab418ae50ab6815ef572a6d4bd10d78e993b6049c1a1dd1e3f6f9814cab0", + "74f2db999392b12121a52b358434296da44bb89947ddafe5d40cec7beb3e414d", + "1c93beb39e5fbd261f5a6566f88b4cce6c8feeb4b9e9c91783fc8b4499189afb", + "ddc628df8adcb3165736d8545d6bc1c4452f2296ab878b056bd6511f227cd299", + "eb7084f1f8ef541f27fdebb9f8cb934bcd7edb9c8c6d12f60316f84acdd4b9b2", + "cde6acdf79d267ee5147c4e65d1edb4e0a07656b6f560578585211ae88883fc3", + "7449c220b3a41c6d7a5682ed7d7ccec2c4def46824eabbefd595a0e5dddaf9e9", + "5ed88151733674af61fd474cfedf15e7a8b7d1c7bce44928bda9cc2db858984a", + "24f59709d171dfa4d49b036a469208a87bf4515161aacf9c55b5c9935273b2f0", + "23efeef3e63975a236f69015a44a8b32b162570ad60ef107489aa1a1da5388db", + "f4e57fc11ac0bdb1235f5a49849d5e8c1d8a73f02b464b8861ae2134d29aeb1f", + "f2b2f29080ab68ffe39d8e86772c78b7c09a16024b72b925ebd824525803dcf9", + "e102f7739a5b7e75dd2adacddfe48580ea4a621182d0fa0ef865e483f881be94", + "25e4208a85c1dbbe8c9337f0fbdb8cab2655a9692846e9669e6b597cf79373c8", + "da4982e13d2a24665fe5cb4e3d11ee9f6b237d15a8d6f463fa599572ee01d5cd", + "0d224549cc794f72283c537d194efe10fda71597d5e8b6b1d6f0faf1f5688fe4", + "2378fb0751d0f21ecb91f5b1bd846d788058ad2a73200a33460e5b772cd417ae", + "e994028d841cd272458ce38f00aad5a93553e5ab36bd71995dc0ca2b2ed5c19e", + "eb1cec7e0abe56bcf4e0240560b3bcaedc6a4081dc7ca2e4a8e9e04c62dbf9c4", + "4cae7a4d07888ee8eec3f8b3a7d232f76eded1d5af4847fb6c31fab6e9700158", + "2861deaa154c6f6c93e536ddd6cf8a9b5d445204f96ef511616126bf32b2b4a8", + "a1195ef8b92d279923ecd86ce25cdfb905607ec533b323d8e0d0b1b5dc2902ac", + "68cfa7d0933d0261fa34a58ba1c224148d3133cbc237421dcc63b830a67e870b", + "bde390c96ff54978e603f67e05d115e57f928b0ffbe2a9e06f1907e293bef7a6", + "2e0347d26f139a07ab151f00c5968f27cf37e0abe748a4f7809cd7488b4816e7", + "8151f090f8a5f635fd3bcb4b42569919ddfe34b756c87610cdae9cf4d899ea52", + "042c82bdf6e09344c337ff9753ba9852f10180fcc2ec6fdab9431c48f2804a47", + "2238079e95be053c37ae41b00934decd06e4da07d9f86919ad5cfa810e5597ad", + "cffde788491b58ae16c8da3d13b4dc52bcf8bfd753211687460c53e224495249", + "6b83a2ed68f6921b6bd23cd05f9421914df684139dba33cc026e29c456fbc570", + "b9e6e92c6c876a9e80fd4d42fd6e49d814d1e79d6b8cf5c9b8d2cb0bd768cd8e", + "10c6905093b0d916e08870d42be0d50cf1f1f548235f421a690971d2ec12daab", + "a88672cf673b5d040aac8c63e80fed9e40e12ab19e18fdf1c641f7d0a29674ef", + "ead7b579320feaa42cc6c4e55c5e22adab2bf7934c0eb2cbb88db347362f7d59", + "f759753fa0211e131cb0ebb2ca40efe5296d680cd0328cdcccc00cd8028fac3d" + ], + "repeat": 200, + "schema": "qdl.phase91.python-parity.v1", + "status": "PASS" + }, + "rust_clean_process_runs": 3, + "rust_events_per_second_min": 385536.1882187671, + "semantic_mismatches": 0, + "status": "PASS" + }, + "prerequisite_bundle_id": "558042db-a766-5a55-b5b3-4b508d649df9", + "prerequisite_decision": "NO_GO_EXTERNAL", + "production_authorized": false, + "production_mutations": 0, + "python_v1_public_authority_unchanged": true, + "schema": "qdl.phase92.bounded-primary-certification.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "status": "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED", + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "real production canary hold and exact-slice approval remain unavailable", + "same-host replicated broker is not an independent failure domain" + ] +} diff --git a/upgrade/evidence/phase92-evidence.sha256 b/upgrade/evidence/phase92-evidence.sha256 new file mode 100644 index 0000000..3f14971 --- /dev/null +++ b/upgrade/evidence/phase92-evidence.sha256 @@ -0,0 +1,3 @@ +2acf8616d9134dea52eea0ccaebcc7f783efa29a86d85aad627f97dbbee3cfe6 upgrade/evidence/phase92-bounded-primary-certification.json +005e5034cec9ab04b2b772a0aef151b7108255c3d3e0f3cd0fd7134b83723772 upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md +8d7761160376233d17808a8f3387d8adb89cba5db3e3dc487e50e5c267dd612a upgrade/evidence/phase92-authority-migration.json diff --git a/upgrade/evidence/phase93-evidence.sha256 b/upgrade/evidence/phase93-evidence.sha256 new file mode 100644 index 0000000..fd24997 --- /dev/null +++ b/upgrade/evidence/phase93-evidence.sha256 @@ -0,0 +1,3 @@ +f869a7621cbcab174788027468ca31f4982e7bce06af768e29c9e486420105b2 upgrade/evidence/phase93-hold-close-expand-certification.json +00b1b1854858aea628bcae29e4a2c49e35ce6a6c55814c8e186d58eb761a0fc1 upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md +5239ab909c63f691c56f205f4d59afef44b718138c563fe6173a0b9e4c77bb64 upgrade/evidence/phase93-hold-close-migration.json diff --git a/upgrade/evidence/phase93-hold-close-expand-certification.json b/upgrade/evidence/phase93-hold-close-expand-certification.json new file mode 100644 index 0000000..539fa5b --- /dev/null +++ b/upgrade/evidence/phase93-hold-close-expand-certification.json @@ -0,0 +1,112 @@ +{ + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "cleanup": { + "production_rows_created": 0, + "resources_remaining": 0 + }, + "control_plane_fixture": { + "accelerated_time_is_production_evidence": false, + "current_no_go_rejection": "PREREQUISITE_DECISION_NOT_GO", + "decommission_reason": "RUNTIME_STILL_REQUIRED_FOR_ROLLBACK", + "expansion_manifest_count": 5, + "expansions": [ + { + "digest": "febb0bb6e5f785c34f169c141814fae6789ea2ad18cacad82fd9dd441562683c", + "required_gate_count": 7, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "INSTRUMENT_PARTITION", + "write_authority": false + }, + { + "digest": "bd498010e9297433f18101911853d4dd7f74257fcfa0ef52492b6de5f5276db6", + "required_gate_count": 9, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "BBO", + "write_authority": false + }, + { + "digest": "1a72f3c0ff729fa938c9dbff2837c5084017b6b2988d2cc69cde8c3deeeaba21", + "required_gate_count": 9, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "L2_BOOK", + "write_authority": false + }, + { + "digest": "46e72af783a551a3fcc6ca227c3a91302e6c24f023766d7e38e3318dfed3396c", + "required_gate_count": 8, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "BAR_LIFECYCLE", + "write_authority": false + }, + { + "digest": "6244ea1445c4334f20025bc27cdf1c95c4257a64fe8981808973a471b3fd436f", + "required_gate_count": 10, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "VENUE_MARKET", + "write_authority": false + } + ], + "local_phase92_production_eligible": false, + "provenance": "TEST_CONTROL_PLANE_FIXTURE", + "test_hold_production_authorized": false, + "test_hold_status": "PASSED" + }, + "issued_at_ns": 1787103230688606493, + "migration": { + "all_expansion_types_registered_independently": true, + "append_only_mutation_rejected": true, + "approval_and_closure_ids_distinct": true, + "authority_state_after_closure": "RUST_PRIMARY:4:rust-primary:2:100", + "closure_did_not_mutate_authority": true, + "closure_digest_bound_to_expansion": true, + "closure_records": 1, + "decision_records": 2, + "decommission_records": 2, + "dirty_hold_pass_rejected": true, + "expansion_records": 5, + "final_test_state_after_stale_cas_setup": "BLOCKED:5:rust-primary:2:130", + "hold_records": 2, + "idempotent_migration": true, + "incomplete_expansion_gates_rejected": true, + "observation_records": 3, + "out_of_order_observation_rejected": true, + "production_mutations": 0, + "registry_mutation_rejected": true, + "schema": "qdl.phase93.hold-close-migration.v1", + "stale_authority_closure_rejected": true, + "status": "PASS" + }, + "parent_phase92": { + "authentic_events": 25600, + "production_authorized": false, + "semantic_mismatches": 0, + "sha256": "2acf8616d9134dea52eea0ccaebcc7f783efa29a86d85aad627f97dbbee3cfe6", + "status": "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED" + }, + "prerequisite_decision": "NO_GO_EXTERNAL", + "production_authorized": false, + "production_expansions_authorized": 0, + "production_hold_started": false, + "production_mutations": 0, + "production_rollback_window_closed": false, + "python_decommission_authorized": false, + "schema": "qdl.phase93.hold-close-expand-certification.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "status": "COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED", + "technical_debt": [ + "Phase 9.0-C production infrastructure remains NO_GO_EXTERNAL", + "real production primary and sustained hold observations do not exist", + "real consumer checkpoints and operator closure approval do not exist", + "every expansion remains independently uncertified" + ], + "v1": { + "health_after": 200, + "health_before": 200, + "topology_unchanged": true + } +} diff --git a/upgrade/evidence/phase93-hold-close-migration.json b/upgrade/evidence/phase93-hold-close-migration.json new file mode 100644 index 0000000..4fb9858 --- /dev/null +++ b/upgrade/evidence/phase93-hold-close-migration.json @@ -0,0 +1,24 @@ +{ + "all_expansion_types_registered_independently": true, + "append_only_mutation_rejected": true, + "approval_and_closure_ids_distinct": true, + "authority_state_after_closure": "RUST_PRIMARY:4:rust-primary:2:100", + "closure_did_not_mutate_authority": true, + "closure_digest_bound_to_expansion": true, + "closure_records": 1, + "decision_records": 2, + "decommission_records": 2, + "dirty_hold_pass_rejected": true, + "expansion_records": 5, + "final_test_state_after_stale_cas_setup": "BLOCKED:5:rust-primary:2:130", + "hold_records": 2, + "idempotent_migration": true, + "incomplete_expansion_gates_rejected": true, + "observation_records": 3, + "out_of_order_observation_rejected": true, + "production_mutations": 0, + "registry_mutation_rejected": true, + "schema": "qdl.phase93.hold-close-migration.v1", + "stale_authority_closure_rejected": true, + "status": "PASS" +} diff --git a/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md b/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md index 2c64026..fde7376 100644 --- a/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md +++ b/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md @@ -4884,6 +4884,244 @@ replicated transport, production telemetry/security, full consumer cutover and regional DR claims until their real infrastructure exists. V1 remains authoritative and was not restarted or mutated. +## Appendix E — Phase 9.0-B isolated V2 beta boundary + +Phase 9.0-B reuses the Phase 7 V2 API/query/stream topology after the runtime +correctness fixes of Phase 9.0-A. It is an evidence refresh on the migrated +host, not a new public contract and not a Rust authority promotion. + +The approved topology contains one immutable query replica, two fenced +active/passive stream replicas, one dedicated AOF Redis, bounded canonical +spool volumes and an optional continuous V1 read-only bridge. Only the bridge +joins the existing internal V1 network. All V2 ingress is loopback-only and all +state, keys, groups, credentials, audit files and Compose resources use an +isolated Phase 9.0-B namespace. + +The first slice is fixed to `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`. +The bridge obtains authentic final bars through the internal V1 API and cannot +resolve arbitrary URLs or call a venue directly. V1 remains source authority; +the beta canonical spool is shadow evidence only. The beta may neither write +legacy Redis keys/channels nor claim execution eligibility. + +Certification combines Sections 18-19, 24-25, 30, 32, 37-41 and Appendix B: + +1. freeze V2 OpenAPI/Protobuf/SDK and verify V1 compatibility; +2. prove consumer-bound workload identity, entitlement, rate/concurrency bounds + and cursor signing/rotation/adversarial rejection; +3. compare provider-authentic V1 and V2 final bars field-for-field and prove + deterministic event identity, deduplication and contiguous replay-to-live; +4. stop the active stream owner and require a higher fencing epoch before the + passive owner serves work; stale-owner writes remain rejected; +5. stop/restart beta Redis and processes, require dependency-derived readiness, + preserve bounded durable state and keep V1 fallback available; +6. measure CPU, memory, PIDs, Redis/spool growth and latency under normal, burst + and slow-consumer cases; and +7. remove all disposable resources and verify the V1 topology, state and public + contract are byte/identity-equivalent before and after. + +Passing this appendix permits only an isolated `V2_BETA_READ_ONLY` review. It +does not close replicated broker, production OTel/alerting, workload identity, +external secrets, signature admission, independent DR, consumer registration +or exact authority-slice approval. Those remain mandatory before Phase 9.1. + +Phase 9.0-B certification completed on candidate revision +`1c881389b4ee21a153903505822c61512b176044` with exact authentic-bar parity, +active/passive fencing, dependency recovery, adversarial security, bounded load +and zero-residue cleanup all passing. The frozen report is +[`PHASE90B_ISOLATED_V2_BETA_REPORT.md`](evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md). +This evidence does not alter the authority boundary above. + +## Appendix F — Phase 9.0-C production prerequisite boundary + +Phase 9.0-C converts the Phase 6 and Phase 9 production blockers into one +provider-neutral, machine-verifiable gate bundle. The evaluator distinguishes +`TEST`, `LOCAL_REHEARSAL`, `PRODUCTION` and `INDEPENDENT_FAILURE_DOMAIN` +evidence. A stronger gate cannot be satisfied by evidence from a weaker scope. + +The mandatory gates are replicated durable transport; OTel collection, +persistent dashboards and acknowledged alert routing; workload identity, RBAC +and network policy; external secret rotation; signed artifact admission; +PostgreSQL PITR; object-store restore; independent failure-domain DR; +Redis/projector reconstruction; complete affected-consumer registration and +rollback; persistent authority/sink fencing; and explicit exact-slice approval. + +Evidence records carry an ID, gate, environment, scope, status, artifact hash, +observation/expiry time, issuer and immutable details. Missing, duplicate, +expired, malformed, contradictory or lower-scope evidence fails closed. Local +Kafka replication, debug OTel exporters, self-signed test keys and same-host +restore remain useful rehearsal evidence but never become production proof. + +Authority identity and transitions follow Section 30. PostgreSQL stores the +current exact slice and append-only audit; compacted Kafka remains the durable +distribution/audit substrate. A compare-and-swap transition checks current +state, authority revision, owner, lease epoch and partition-plan epoch. Canary +and primary transitions additionally require the exact passing prerequisite +bundle and named operator approval. + +Passing Phase 9.0-C code and local tests may yield +`COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL`. Only real infrastructure evidence can +yield `PRODUCTION_PREREQUISITES_PASS`. Neither result changes V1 automatically; +Phase 9.1 remains a separately approved exact-slice canary. + +Implementation status (2026-08-18): `COMPLETE_CONTROL_PLANE / +NO_GO_EXTERNAL`. The strict 12-gate evaluator, additive persistent authority +CAS/audit schema, frozen candidate, migration smoke, operator runbook and +checksummed evidence are complete. Local, focused and full candidate suites pass +with V1 unchanged. The external infrastructure/operator gates remain explicit; +none was relabeled or bypassed to manufacture a production approval. + +## Appendix G — Phase 9.1 Rust canary boundary + +Phase 9.1 is an exact-slice dual-read/compare stage, not source-authority +cutover. Python remains the only public and V1 compatibility writer. Rust may +publish only to a dedicated canary canonical namespace that no production +consumer treats as authoritative. + +The production path accepts one Phase 9.0-C decision bundle only when it is +`GO`, fresh, checksummed and bound to the exact candidate digest, partition-plan +epoch, image/signature, contract, normalizer, adapter, catalog, source policy, +consumer set, blast radius, operator and hold window. The isolated rehearsal +path has a separate entrypoint and cannot invoke production CAS or public sinks. +No environment variable or test mode may convert rehearsal evidence into a +production decision. + +A versioned Phase 9 authority record binds: + +```text +slice_id +state +owner_id +authority_revision +lease_epoch +partition_plan_epoch +candidate_digest +prerequisite_bundle_id +start_watermark +approved_by +approved_at +hold_until +``` + +Every canary publication repeats slice, owner and all three epochs plus source +watermark. The sink rejects wrong or stale identity before durable append. +`RUST_SHADOW` can write shadow targets; `RUST_CANARY` adds only the isolated +canary canonical target; `BLOCKED` writes nothing. Public V2 and Legacy V1 remain +forbidden throughout Phase 9.1. + +Parity compares Python and Rust outputs from the same authentic captured frame +range. Correctness-critical fields have zero tolerance. Guardrails block on any +unexplained mismatch, open gap, final/revision mismatch, duplicate external +publication, stale writer, authority ambiguity, durable ACK failure or approved +lag/freshness/resource threshold breach. A monotonic hold-down prevents noisy +automatic re-entry. + +The certification matrix includes deterministic replay, burst, clean-process +restart, stale lease/owner/revision/partition, producer and broker interruption, +slow consumer, guardrail block, rollback to shadow, V1 compatibility and exact +cleanup. Same-host Kafka and captured frames prove implementation behavior only; +they do not close the production/failure-domain prerequisites from Appendix F. +With a `NO_GO_EXTERNAL` prerequisite decision the only valid result is +`COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`. + + +### Phase 9.1 implementation closure (2026-08-18) + +The isolated implementation is certified as +`COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`. The frozen authentic capture +produced 25,600 events with zero Python/Rust semantic mismatch over three clean +Rust processes. The replicated TLS/ACL broker rehearsal passed exact sink +fencing, one-replica availability, below-min-ISR fail-closed behavior, full +restart recovery, compacted authority reconstruction, immutable audit order, +64-record slow-consumer catch-up and rollback to `RUST_SHADOW`. Public and +legacy write counts remained zero; V1 health and topology were unchanged. + +Operator artifacts are the [runbook](../docs/runbooks/phase91-rust-canary.md), +[machine evidence](../upgrade/evidence/phase91-rust-canary-certification.json), +[human report](../upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md) and +[checksum manifest](../upgrade/evidence/phase91-evidence.sha256). Phase 9.0-C +still returns `NO_GO_EXTERNAL`; no production authority transition is allowed +until independent production prerequisites and exact-slice approval are +provided. + +## Appendix H — Phase 9.2 bounded Rust primary boundary + +Phase 9.2 promotes no broad venue and changes no public API. It defines one +provider-neutral ownership handoff for one exact slice. With the current +`NO_GO_EXTERNAL` decision, only an isolated rehearsal is legal; production V1, +Redis, subscriptions and public destinations remain untouched. + +The immutable handoff chain is: + +```text +old-owner durable terminal checkpoint at W + -> same-range parity/reconciliation accepted through W + -> persistent CAS: revision + 1, lease epoch + 1, new owner + -> final sink and compatibility projector load accepted authority + -> first new-owner authoritative watermark W + 1 +``` + +A terminal checkpoint binds slice, old owner, authority revision, lease epoch, +partition-plan epoch, source session, connection generation, terminal event ID, +terminal payload hash and durable watermark. An accepted handoff additionally +binds the checkpoint digest, exact overlap range, zero gaps, zero semantic +mismatches, matching event/output counts, candidate and prerequisite bundle, +approver and expiration. These records are append-only. + +The authority schema is additive. Phase 8 v1 and Phase 9.1 v2 records remain +decodable. Phase 9.2 v3 grants authoritative targets only in `RUST_PRIMARY` (or +the newer fenced `PYTHON_PRIMARY` rollback owner). Revision must advance exactly +one per transition. Owner changes require a strictly newer lease. A changed +slice, candidate or partition plan cannot be smuggled through a handoff. + +Final canonical, public V2 and legacy V1 compatibility destinations each keep an +independent acknowledged watermark under the same authority identity. A write +is accepted only for the active owner/revision/lease/plan, correct target and +next boundary. Watermark state advances after durable ACK. A crash before ACK is +retryable; a crash after ACK is reconciled by deterministic event ID and durable +cursor. Stale and zombie writers are rejected at the sink/projector, even if a +producer process still believes its lease is valid. + +Production authorization is distinct from rehearsal. It requires: + +- a fresh exact Phase 9.0-C `GO` bundle; +- a real `RUST_CANARY` hold completed with zero correctness breach; +- registered consumers and immutable Python rollback manifest; +- accepted terminal/handoff records for the exact slice; +- explicit operator, ticket, blast-radius and hold-window approval. + +The isolated harness uses authentic frozen provider frames and replicated +test-only topics to model final/public/legacy projections. Those topic names can +never equal production destinations. It tests `N-1/N/N+1`, off-by-one, gap, +duplicate, stale owner/revision/lease/plan, competing CAS, process loss before +and after CAS, sink/projector restart, broker replica loss/min-ISR, full restart, +slow consumer and formal rollback. It records production mutations as zero, +checks V1 topology before/after and deletes only disposable resources. + +A process loading an already-primary v3 authority starts fail-closed. It must +reconstruct the latest ACKed watermark independently from each durable final, +public V2 and legacy V1 destination, validate identity and a contiguous range, +and restore that target fence before any append. A duplicate terminal +watermark remains rejected and the first permitted write is exactly W+1. This +prevents a crash after durable broker ACK but before process-local commit from +re-emitting an external duplicate. + +A valid isolated result is +`COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED`. It proves protocol and +recovery behavior; it does not authorize disabling the Python subscription or +claim independent failure-domain resilience. + +**Implementation closure (2026-08-18):** The isolated Phase 9.2 harness passed +over 25,600 authentic provider events with zero Python/Rust semantic mismatch. +The broker-backed second-process recovery reconstructed three independent +target watermarks at 180 and resumed exactly at 181; canonical, V2 and V1 +projections remained identical and gap-free. PostgreSQL CAS/handoff, formal +rollback, replica-loss/min-ISR, full restart, slow-consumer, full Rust/Python +regression and scoped cleanup gates passed. Evidence is frozen in +[`phase92-bounded-primary-certification.json`](../upgrade/evidence/phase92-bounded-primary-certification.json), +the [human report](../upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md) and +[`phase92-evidence.sha256`](../upgrade/evidence/phase92-evidence.sha256). Phase +9.0-C remains `NO_GO_EXTERNAL`, so this closure grants no production authority. + ### Option and Deribit extension boundary Adding an option venue must not require changing canonical core identities or rewriting distribution. The common boundary must represent: @@ -4897,3 +5135,150 @@ Adding an option venue must not require changing canonical core identities or re - source authority, quality state and entitlement/licensing metadata. An adapter declares unsupported capabilities explicitly. Phase 3 uses sanitized Deribit-style fixtures to prove the boundary; Phase 6 certifies architecture readiness. Real Deribit activation remains a separate adapter certification requiring official source semantics, credentials, licensing and capacity evidence. + + +## Appendix I — Phase 9.3 hold, closure and independent expansion + +Phase 9.3 is a control-plane boundary after one exact slice has become a real +`RUST_PRIMARY`. It does not introduce a new sink authority state. The data plane +continues to use the Phase 9.2 owner/revision/lease/partition and watermark +fences. Hold and closure records describe whether operators may reduce the +rollback posture; they never grant writes by themselves. + +### Hold identity and observation contract + +A hold is immutable and binds: + +```text +hold_id +slice_id +candidate_digest +prerequisite_bundle_id +owner_id +authority_revision +lease_epoch +partition_plan_epoch +started_at_ns +required_until_ns +policy_digest +``` + +Each observation repeats that identity and carries an increasing sequence, +observation time, last durable watermark, correctness counters, source quality, +consumer checkpoint state and bounded resource/lag metrics. The evaluator +requires ordered observations across the whole approved interval and a maximum +sample gap. It never fabricates a missing observation. + +The following values are zero tolerance: + +```text +semantic_mismatches +open_gaps +duplicate_external_writes +accepted_stale_writer_writes +authority_ambiguities +durable_ack_failures +projection_mismatches +consumer_checkpoint_regressions +unexplained_quality_failures +``` + +Lag, freshness, queue, spool, CPU and RSS use explicit upper bounds. An owner, +authority revision, lease, partition plan or candidate change invalidates the +hold. A breach is sticky for that hold identifier; a later clean sample cannot +rewrite history. Starting again requires a new hold and preserves the failed +record. + +### Rollback-window closure + +A closure authorizer consumes six independently frozen inputs: + +1. fresh Phase 9.0-C production prerequisite `GO` bound to the candidate; +2. production-authorized Phase 9.2 primary evidence for the exact owner/epochs; +3. a passing real hold decision covering the minimum approved duration; +4. complete consumer registry snapshot with healthy contiguous checkpoints; +5. exact authority registry snapshot plus a fresh successful rollback rehearsal; +6. explicit operator/change-ticket approval with bounded expiry. + +The database closure transaction locks the current authority row and rechecks +state `RUST_PRIMARY`, owner, revision, lease, partition-plan epoch, candidate, +bundle and hold identity. It inserts one immutable closure record. It must not +update authority state, owner, revision, lease, watermark, public-write or +legacy-write fields. A concurrent authority change makes the closure fail. + +Closing the window means normal rollback is no longer an unreviewed routine +operation. It does not delete the Python manifest or make emergency recovery +impossible. A later incident still fences Rust first and creates new immutable +rollback authority/audit records. + +### Consumer and authority registry freeze + +The closure snapshot records only stable IDs, versions, digests, contiguous +checkpoints and health decisions. It does not embed secrets, tokens or raw +unbounded logs. Every affected critical consumer must be registered exactly +once, acknowledge the current authority revision and expose no checkpoint +regression or unresolved migration. Unknown or duplicate consumers fail closed. + +### Independent expansion manifests + +Expansion types are provider-neutral: + +| Expansion | Mandatory independent certification | +|---|---| +| `INSTRUMENT_PARTITION` | partition churn, source capacity, exact-frame parity, handoff and rollback | +| `BBO` | quote identity, coalescing policy, freshness, ordering and reconnect | +| `L2_BOOK` | snapshot/delta sequence, checksum, resync, lossless backpressure and capacity | +| `BAR_LIFECYCLE` | final/revised/cancelled lineage, close-time semantics and replay | +| `VENUE_MARKET` | adapter capability, instrument identity, provider semantics, entitlement and DR | + +Every manifest has a new `expansion_id`, candidate digest, scope digest, +partition-plan epoch, required-gate set and status +`INDEPENDENT_CERTIFICATION_REQUIRED`. It sets write authority false. Parent +hold/closure evidence is provenance only and cannot satisfy child gates. +Combining expansion classes in one manifest is forbidden so risk and rollback +remain bounded. + +### Runtime decommission boundary + +A Python hot path may be nominated for removal only when it owns zero slices, +is absent from every active rollback manifest and consumer dependency, all +replacement windows are governed closed, and repository cleanup is explicitly +approved. Shared canonical contracts, provider semantics, fixtures, migration +knowledge and compatibility projectors are not removed merely because one hot +path moved to Rust. + +### Current decision boundary and certification + +The current Phase 9.0-C result is `NO_GO_EXTERNAL`, and Phase 9.2 produced only +isolated primary evidence. Therefore Phase 9.3 may certify contract validation, +PostgreSQL transaction behavior, fail-closed closure denial, expansion +independence, V1 invariants and exact cleanup. It may not claim a real hold, +close a production rollback window, decommission Python or authorize an +expansion. + +The maximum local result is: + +```text +COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED +``` + +Production completion later requires authentic sustained observations over the +approved wall-clock duration, real consumer checkpoints, current authority and +rollback evidence, independent infrastructure scope and explicit operator +approval. No accelerated fixture or same-host rehearsal can satisfy that gate. + + +**Implementation closure (2026-08-19):** Phase 9.3 control-plane implementation +is certified as `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED`. Typed +hold evaluation, immutable registry snapshots, exact authority-locking closure, +closure-digest-bound independent expansion and conservative decommission gates +pass focused and full regression. PostgreSQL smoke proved that closure changed +no authority field and that dirty/sparse/stale/mutable evidence fails closed. +The final suites passed 415 Python tests with 5 intentional skips and 40 Rust +tests with fmt/clippy clean. V1 remained healthy and unchanged; production +mutations and residual Phase 9.3 resources are zero. Evidence is frozen in +[`phase93-hold-close-expand-certification.json`](../upgrade/evidence/phase93-hold-close-expand-certification.json), +the [human report](../upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md), the +[operator runbook](../docs/runbooks/phase93-hold-close-expand.md) and +[`phase93-evidence.sha256`](../upgrade/evidence/phase93-evidence.sha256). This +closure starts no production hold and grants no production authority.