diff --git a/.env.example b/.env.example index 29b495a..908af1f 100644 --- a/.env.example +++ b/.env.example @@ -1,6 +1,15 @@ # ── DNSE OpenAPI ──────────────────────────────── DNSE_API_KEY= DNSE_API_SECRET_KEY= +DNSE_API_VERSION=2026-07-23 +DNSE_REST_BASE=https://openapi.dnse.com.vn +DNSE_WS_BASE=wss://ws-openapi.dnse.com.vn +# Keep false unless an approved egress proxy is required and monitored. +DNSE_REST_USE_ENV_PROXY=false +DNSE_REST_CONNECT_TIMEOUT_SECONDS=5 +DNSE_REST_READ_TIMEOUT_SECONDS=30 +DNSE_REST_ATTEMPTS=4 +DNSE_REST_MAX_BACKOFF_SECONDS=30 # ── vnstock ───────────────────────────────────── VNSTOCK_API_KEY= @@ -35,6 +44,13 @@ DATA_LAYER_PRELOAD_WATCHDOG_ENABLED=true BINANCE_WS_MAX_CONNS_PER_SOURCE=0 STREAM_STALE_SECONDS=180 STREAM_STRICT_FEED_HEALTH=false +BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS=15 +BINANCE_WS_IDLE_TIMEOUT_SECONDS=90 +BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS=2 +BINANCE_KLINE_RECOVERY_ENABLED=true +BINANCE_KLINE_RECOVERY_POLL_SECONDS=2 +BINANCE_KLINE_RECOVERY_SETTLE_SECONDS=1 +BINANCE_KLINE_RECOVERY_CONCURRENCY=4 BINANCE_SYMBOLS_FILE=/app/data/cache/binance_usdm_symbols.json BINANCE_SPOT_SYMBOLS_FILE=/app/symbols_spot.json diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..caf88d8 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,63 @@ +# Data Layer Agent Rules + +This tracked file carries the workspace baseline into every clone of this +repository. For Data Layer work, the mandatory main journal is +`DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md`; detailed design is governed by +`upgrade/quant-data-layer-fund-grade-upgrade-architecture.md` and applicable +provider guides. Host-level `/home/bobby/AGENTS.md`, when present, may add +stricter workspace rules but may not weaken this file. + +## Scope And Sources Of Truth + +1. Before acting, read the nearest `AGENTS.md`, the project's main implementation plan, the approved detailed guide, and the relevant code/config. Do not rely on chat memory alone. +2. When the user asks to discuss, evaluate, explain, or plan only, do not edit files, restart services, mutate data, or run destructive commands. +3. Implement only the approved scope. Report newly discovered out-of-scope bugs and proposed fixes before changing them unless the user explicitly authorized fixing all discovered issues. +4. Preserve the declared domain source of truth. For alpha migrations, backtest/approved research logic and parameters are authoritative unless the user says otherwise. Do not silently simplify business logic. + +## Mandatory Plan Journal + +5. Every project change must be recorded in that project's main plan markdown. The plan update is part of the implementation transaction, not optional documentation after the fact. +6. Before code changes, record or confirm the phase/task status, goal, guide links, approved scope, invariants, test gates, rollback, and decision boundary in the main plan. +7. During implementation, update the plan incrementally after each coherent tested slice with exact work completed, commands/tests actually run, results, cleanup evidence, decisions, and remaining debt. +8. Never claim a task or phase is complete until the main plan and any governing detailed guide agree with the code and evidence. A final response must state the recorded status and link/path. +9. Technical debt means a real external, cost, infrastructure, licensing, business-semantics, or approval gate. Fix in-scope defects before closure; do not relabel unfinished work or failed tests as debt. +10. If a project has no identified main plan, identify or create one before substantial implementation. Do not scatter progress across ad hoc markdown files without linking them from the main plan. + +## Runtime And Data Safety + +11. Protect running production/V1 consumers by default. Use isolated names, ports, networks, schemas, Redis prefixes, volumes, topics, consumer groups, credentials, and evidence paths for development and tests. +12. Do not restart, recreate, cut over, prune, flush, truncate, force-close, delete volumes/data, or change authority unless the user explicitly approves the exact blast radius and rollback. +13. Production/shadow market data must come from approved real providers or replay of durably captured provider bytes. Synthetic/generated data is test-only and must carry test provenance. +14. Never commit secrets, raw credentials, private keys, unbounded logs, caches, runtime state, or generated data. Evidence stores identifiers, hashes, bounded metrics, and provenance only. +15. Clean disposable test resources and scoped smoke rows after verification. Never clean shared state broadly when an exact alpha/account/test namespace is available. + +## Verification And Evidence + +16. Test correctness before performance: identity, units, decimals, timestamps/timezones, ordering, bar closure, sequence, source authority, state transitions, risk/order semantics, and business-domain parity. +17. Use applicable unit, contract/golden, parity/oracle, migration-idempotency, integration, failure/reconnect, recovery/rollback, compatibility, security, resource/capacity, and bounded real-provider tests. +18. A health endpoint or process-up state alone is not acceptance. Report exact cases, pass/fail/skip counts, untested boundaries, production mutations, and cleanup results. +19. Never present local/same-host rehearsal as production, HA, independent failure-domain, broker-authoritative, sandbox/live, or real-provider evidence. Missing evidence fails closed and is documented honestly. +20. Read large logs and databases intelligently: filter relevant time ranges and warning/error/event IDs first, then query exact related rows. Avoid unbounded output and token-heavy tool use. + +## Git And Change Discipline + +21. Inspect branch/worktree before editing. Preserve user/unrelated changes and never revert them without explicit instruction. +22. Use feature branches from `dev` when the repository workflow requires it. Feature branches merge to `dev`; `main` is release-only. Never merge or push unless explicitly requested. +23. Commit each coherent, tested implementation slice with a clear message. Do not create excessive tiny commits or one giant untraceable commit. +24. Commits must use the user's configured identity (`BobbyAxerol `), never a system/root identity. Verify identity before commit. +25. Before commit: run `git diff --check`, inspect staged scope, confirm tests/evidence and plan updates, and exclude unrelated files. After commit: report commit SHA, branch, worktree state, and whether anything was pushed/merged. + +## Engineering And Operations + +26. Prefer existing shared runtime, SDK, contracts, wrappers, parsers, and domain abstractions. Copy unchanged code mechanically when appropriate; avoid duplicated strategy-specific infrastructure. +27. Keep public V1/stable endpoints and schemas backward-compatible unless an approved versioned migration says otherwise. Internal implementation language and transport must stay behind contracts. +28. Design provider-neutral, venue-capability-driven, scalable boundaries. Do not hardcode one venue/mode where the domain needs extension to Binance, OKX, DNSE/VN, Deribit, or future brokers. +29. Optimize only after correctness. Measure CPU, memory, disk, queue/lag, I/O and latency; require no unexplained loss, duplicate, gap, or state mismatch before promotion. +30. Keep logs structured, host-visible where required, bounded and aligned to strategy/feed intervals. Avoid tick-level INFO spam unless actively diagnosing a new feed. +31. Use the smallest useful set of tools and agents. Prefer `rg`, targeted reads, scoped tests, reusable commands, and compact evidence over broad scans or repeated full-output calls. +32. Use `apply_patch` for manual edits when available. If the sandbox helper is broken, use an exact-match scripted replacement and immediately verify the diff. + +## Reporting + +33. Working updates explain what is being checked or changed and why. Final reports state: implemented scope, domain behavior, tests/evidence, runtime impact, cleanup, remaining decision gates, commit/branch, and the next permitted step. +34. Do not hide blockers or overstate readiness. Distinguish `implemented`, `tested locally`, `shadow-certified`, `production-ready`, and `production-authoritative` explicitly. diff --git a/Cargo.lock b/Cargo.lock index f467127..64a1eab 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -156,6 +156,16 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.52.0", +] + [[package]] name = "find-msvc-tools" version = "0.1.11" @@ -521,13 +531,18 @@ dependencies = [ "indexmap", "jobserver", "proc-macro-crate", + "prost", + "qdl-contracts", "qdl-core", + "qdl-realtime-core", "qdl-venue-core", "rdkafka", + "rustls", "serde", "serde_json", "sha2", "tokio", + "tokio-tungstenite", ] [[package]] @@ -539,10 +554,25 @@ dependencies = [ "sha2", ] +[[package]] +name = "qdl-realtime-core" +version = "0.1.0" +dependencies = [ + "prost", + "qdl-contracts", + "qdl-core", + "qdl-provider-envelope", + "qdl-venue-core", + "serde", + "serde_json", + "sha2", +] + [[package]] name = "qdl-venue-core" version = "0.1.0" dependencies = [ + "hex", "serde", "serde_json", "sha2", @@ -792,6 +822,16 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "slab" version = "0.4.12" @@ -855,6 +895,7 @@ dependencies = [ "libc", "mio", "pin-project-lite", + "signal-hook-registry", "socket2", "tokio-macros", "windows-sys 0.61.2", diff --git a/Cargo.toml b/Cargo.toml index 98b7b50..28805d7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,6 +4,7 @@ members = [ "rust/qdl-core", "rust/qdl-kafka", "rust/qdl-provider-envelope", + "rust/qdl-realtime-core", "rust/qdl-venue-core", ] resolver = "2" @@ -27,6 +28,6 @@ rustls = { version = "=0.23.43", default-features = false, features = ["ring", " serde = { version = "=1.0.228", features = ["derive"] } serde_json = "=1.0.145" sha2 = "0.10.9" -tokio = { version = "=1.48.0", features = ["fs", "io-util", "macros", "rt-multi-thread", "sync", "time"] } +tokio = { version = "=1.48.0", features = ["fs", "io-util", "macros", "rt-multi-thread", "signal", "sync", "time"] } tokio-tungstenite = { version = "=0.24.0", features = ["rustls-tls-webpki-roots"] } zeroize = "=1.8.1" diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 62ca066..e1fc820 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -1,7 +1,7 @@ # Quant Data Layer Unified Implementation Plan -> **Status:** Phases 0-5 are complete; Phase 6 implementation and shadow certification pass, while production authority remains `NO-GO` on explicit infrastructure gates. Phase 7 is complete with a protected read-only `BETA-GO`; Phase 8 is complete with an immutable, signed, multi-venue Rust realtime-core candidate fenced to `RUST_SHADOW`; Phase 9 remains planned. V1 remains authoritative and no runtime cutover has started. -> **Working branch:** `feat/fund-grade-data-layer-v2`, created from `dev`. +> **Status:** Phases 0-5 are complete; Phase 6 implementation and shadow certification pass, while production authority remains `NO-GO` on explicit infrastructure gates. Phase 7 is complete with a protected read-only `BETA-GO`; Phase 8 is complete with an immutable, signed, multi-venue Rust realtime-core candidate fenced to `RUST_SHADOW`; Phase 9.0-A and 9.0-B are complete in isolation; Phase 9.0-C is `COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL`; Phase 9.1 is `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`; Phase 9.2 is `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED`; Phase 9.3 is `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED` after isolated hold/closure/expansion governance certification. Authority promotion, production hold/closure and every expansion remain blocked on explicit production infrastructure, real canary/primary evidence and exact-slice approval gates. V1 remains authoritative and no runtime cutover has started. +> **Working branch:** `feat/v2-stable-rust-binance-okx`, based on `dev`; Phase B artifact certification is complete while the overall multi-venue conclusion remains `PARTIAL_EXTERNAL` for DNSE. No push, merge or authority cutover is implied. > **Detailed architecture:** [Fund-grade architecture and migration guide](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md) > **OKX V5 market-data specification:** [OKX Market Data V5 implementation guide](upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md) > **Compatibility boundary:** Existing `/v1`, SDK v1, Redis keys and Redis Pub/Sub remain supported until a governed per-consumer sunset. @@ -160,6 +160,21 @@ These rules apply to all phases. PostgreSQL control-state restore, cursor-key rotation, object-store/PITR, Redis/projector rebuild and authority reconstruction from the audit log have passed the approved recovery objectives. +40. **The main plan is a transactional implementation journal.** Before code, + record the approved phase/scope, guide links, invariants, test gates and + rollback here. After every coherent tested slice, record exact completion, + verification, cleanup, decision gates and debt here in the same commit. A + phase cannot be reported complete while this tracker or its governing guide + disagrees with code/evidence. +41. **Scope and approval are explicit.** A discussion/evaluation request causes + no mutation. Newly discovered work outside the approved scope is reported + with impact and recommendation before implementation. Restart, cutover, + authority change, destructive cleanup, push and merge require the user's + explicit approval for that action. +42. **Final reporting is evidence-bound.** Every completion report names the + plan status, tests actually run with pass/fail/skip counts, untested/external + gates, runtime impact, cleanup, commit/branch and push/merge state. Local or + same-host proof is never upgraded linguistically into production evidence. ## 3. Phase Summary @@ -2274,7 +2289,7 @@ Phase 8 is `COMPLETE` only when: ## 13. Phase 9 - Rust Core Canary And Progressive Replacement -**Status:** `PLANNED` +**Status:** `9.0-A COMPLETE_ISOLATED`; `9.0-B COMPLETE_ISOLATED`; `9.0-C COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL`; `9.1 COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`; an actual production `RUST_CANARY` transition remains blocked on an exact `GO` bundle ### Goal @@ -2657,6 +2672,383 @@ must prove: #### 9.0 Production Prerequisites And Exact Slice Approval +##### 9.0-A Runtime Correctness Closure + +**Status:** `COMPLETE_ISOLATED` + +**Purpose:** Close correctness and deployment-boundary defects discovered after +the server migration before any Phase 9 authority implementation or Rust +canary. This slice changes no public V1 contract, source authority, canonical +writer or running production container until isolated evidence passes and a +separate operator-approved deployment is prepared. + +**Observed baseline (2026-08-18):** + +- The running service is the V1-only `data-layer:v0.1.0` image with host source + bind-mounted read-write at `/app`; OpenAPI exposes 40 V1 routes and zero V2 + routes. No Phase 7 beta, Phase 8 Rust, Kafka or OTel role is active. +- Binance USD-M TRADE is producing authentic frames, but all eight configured + USD-M KLINE shards have remained connected with `message_count=0`. All 734 + expected one-minute kline feeds are missing while `/v1/health` incorrectly + reports `binance_kline_stream=true`. +- Independent read-only probes reproduce provider ACK/connected-without-data: + raw trade and book-ticker produce frames, while kline, aggregate-trade and + mark-price subscriptions time out after successful connection/subscription. +- Feed demand leases are zero while the runtime still opens the broad USD-M + universe. Historical queue-drop count is 138,060; the measured recent + five-minute delta is zero. +- The production container has no explicit CPU, memory or PID limit. Redis and + V1 HTTP remain available; no production state was mutated during discovery. + +**Invariants:** + +- A WebSocket handshake or subscription ACK is transport state, never data + readiness. Enabled feeds that do not produce a valid provider frame inside + their declared first-frame/staleness deadline fail closed to a typed degraded + state. +- Recovery uses provider-authentic Binance REST closed bars for active demand + only. It never fabricates candles, marks the open candle final, substitutes + OKX data as Binance authority or hides WebSocket degradation. +- Trade/book canonical events are not silently coalesced or dropped. Existing + legacy latest-state projection behavior remains contract-compatible while + queue loss and recovery state stay observable. +- Tests use isolated processes, Redis prefixes and Compose project names. They + do not restart V1, flush shared Redis, mutate production Parquet or reuse live + consumer groups. Disposable state is removed after evidence capture. + +**Implementation tasks:** + +1. Split stream transport, source and per-feed readiness. Report TRADE and + KLINE independently; expose connected shard count, producing shard count, + first-frame deadline, stale deadline, recovery source and active demand. +2. Add a bounded data-frame watchdog. A connected shard with no valid frame by + deadline enters `DATA_UNAVAILABLE`, records the outage and reconnects with + jittered backoff rather than remaining green forever. +3. Add one demand-scoped closed-kline recovery loop. It batches and rate-limits + Binance REST requests, emits only fully closed rows with explicit + `BINANCE_REST_GAP_FILL` provenance, deduplicates by symbol/interval/open time, + and stops after demand leases expire. +4. Make demand registry ownership and TTL visible. REST/SDK demand renewal must + not make reads fail, but missing registration cannot authorize broad source + health or resource claims. +5. Correct `/v1/health` without changing its response keys: boolean TRADE/KLINE + fields reflect their own data readiness; top-level status degrades for an + enabled unavailable source while market-closed DNSE remains healthy by + policy. Add detailed source readiness under the existing nested supervisor + payload. +6. Add an immutable production Compose overlay with no host source bind and + explicit CPU, memory, PID, read-only-root, tmpfs and writable data/log/cache + boundaries. Keep the current deployment untouched; cutover requires a + digest-pinned image, preflight and operator approval. +7. Freeze compact machine-readable evidence and an implementation report, + including authentic probe results, unit/integration counts, V1 golden diff, + resource limits, production-unchanged proof and cleanup. + +**Implementation checkpoint (2026-08-18):** + +- Implemented valid-frame watchdog, independent source readiness, bounded queue + backpressure and demand-only Binance closed-kline REST recovery. Removed the + reconnect-only recovery duplicate so one manager owns scheduling, dedup and + backoff. +- Recovery preserves the provider interval from `k.i`, rejects open/invalid rows, + retains explicit provenance and expires work with the final demand lease. +- V1 health keys remain unchanged; TRADE and KLINE booleans now represent their + own data readiness. Added owner visibility to the existing demand snapshot. +- Added immutable isolated Compose candidate with non-root/read-only execution, + no source bind, dedicated state, loopback-only ingress and CPU/RAM/PID limits. +- Deterministic verification passed: targeted runtime matrix 35/35; full repository + suite ran 345 tests with 340 passes, 5 environment-gated skips and zero failures; + compile/diff checks clean; and + live-vs-candidate OpenAPI path diff 40/40 with zero additions or removals. +- Built and ran immutable candidate digest `sha256:4a2723ec39057c75a89889d955feac7acc6fb01bc126a579f8c74d384b9b6999` as UID 10001 with read-only root, no source bind and declared CPU/RAM/PID limits. +- Real-provider smoke proved 8/8 USD-M TRADE shards ready and 0/8 KLINE shards + unavailable; health stayed degraded while demand-only REST recovery returned a + final BTCUSDT bar exactly equal to Binance REST OHLCV. Lease expiry stopped + further provider fetches. Queue pressure and drop deltas remained zero. +- The isolated smoke exposed and closed two candidate bugs before release: Redis + UID with `cap_drop: ALL`, and data outage being cleared by transport reconnect. +- Candidate containers, networks, volumes and images were removed after evidence. + Production V1 remained unchanged and running throughout. Evidence: [Phase 9.0-A + report](upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md) and [machine + result](upgrade/evidence/phase90a-runtime-correctness.json). + +**Verification cases:** + +- Valid TRADE plus valid KLINE frames make only their matching source ready. +- Connected/ACKed KLINE with zero frames misses the first-frame deadline and is + degraded; `binance_kline_stream` must be false. +- One dead kline shard cannot mark other feed types unavailable, and one healthy + trade shard cannot make kline healthy. +- Stale, malformed and wrong-feed frames do not satisfy readiness. +- Active kline demand receives provider-authentic fully closed REST recovery; + open rows, duplicates and non-demanded symbols are rejected. +- REST timeout, 429/5xx, partial batch, reconnect, Redis outage, queue pressure, + lease expiry and process restart remain bounded and observable. +- V1 OpenAPI/golden payloads and legacy Redis keys/channels do not change. +- Isolated real-provider smoke proves the actual provider behavior and recovery + path; generated/simulated data is limited to deterministic failure tests and + is never counted as provider evidence. +- Immutable image runs as non-root without source bind, honors resource limits, + passes liveness/readiness/data-readiness probes and leaves V1 unchanged. + +**Exit gate:** + +- Zero false-green source readiness in the verification matrix. +- Zero fabricated/open-as-final bars and zero unexplained duplicate recovery + publication. +- Zero V1 contract/golden regression and zero production mutation. +- Bounded CPU, memory, queue, request rate and retry/backoff under normal, + outage and recovery cases. +- All disposable containers, networks, volumes, Redis prefixes and captures are + removed; compact checksummed evidence remains. +- Phase 9.1 remains blocked. Completing 9.0-A does not satisfy production OTel, + independent failure-domain DR, workload identity, external secrets, + signature admission, consumer registration or exact-slice approval. + +**Rollback:** + +- Do not deploy the candidate overlay; continue the unchanged V1 container. +- If a later approved deployment regresses, restore the immutable V1 image and + source configuration, remove only the candidate namespace and verify V1 + OpenAPI/Redis compatibility plus provider data readiness. +- REST recovery can be disabled independently; source health must remain + degraded rather than reverting to connected-is-ready semantics. + +##### 9.0-B Isolated V2 Beta + +**Status:** `COMPLETE_ISOLATED` + +**Purpose:** Re-certify the existing provider-neutral V2 query/stream beta on +the migrated host using the Phase 9.0-A runtime-correctness baseline. The beta +is a read-only, non-authoritative consumer of one explicitly bounded V1 source +slice; it is not a Rust canary, a public-internet deployment or a source +authority transition. + +**Guide index:** + +- [Phase 9.0-B isolated beta boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-e--phase-90-b-isolated-v2-beta-boundary) +- [V2 API/SDK and consumer migration](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#18-sdk-v2-architecture) +- [No-big-bang migration](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#30-migration-strategy-no-big-bang-rewrite) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) +- [Phase 7 isolated beta runbook](docs/runbooks/phase7-isolated-beta-runtime.md) +- [Phase 9.0-B certification runbook](docs/runbooks/phase90b-isolated-v2-beta.md) + +**Invariants:** + +- V1 remains the only production/source authority and keeps its existing + container, networks, mounts, Redis namespaces and public contracts. +- Beta reads only the approved internal V1 endpoint and publishes only into its + dedicated canonical spool, Redis prefix, consumer group and loopback ports. +- The beta image is content-addressed, non-root, read-only, bounded and contains + no host source bind. JWT, cursor and bridge secrets are beta-only. +- Beta output uses provider-authentic, final closed bars. Fixtures/synthetic + events are permitted only in isolated deterministic tests and cannot satisfy + the real-provider gate. +- Process readiness, dependency readiness and data availability are tested + separately. Missing/stale source data must produce typed unavailability, not + a false-green data claim. + +**Implementation tasks:** + +1. Reuse the Phase 7 V2 query, active/passive stream, dedicated AOF Redis and + read-only V1 bridge topology; do not fork or rename stable contracts merely + because this is a new certification phase. +2. Pin the candidate and helper images by digest, stamp the application artifact + with source revision, and use a Phase 9.0-B-specific config revision, Redis + prefix, consumer group, project name, leases, credentials and evidence paths. +3. Run the continuous bridge only with the `phase7-canary` profile and prove it + cannot call a venue directly, write V1 state or publish non-final bars. +4. Validate authenticated V2 warmup/query and gRPC replay/live handoff for the + approved BTCUSDT USD-M 1m BAR slice, including decimal/timestamp/finality, + event identity, cursor continuity and V1-vs-V2 parity. +5. Exercise active/passive failover and fencing, Redis outage/recovery, process + restart, stale/invalid cursor, malformed/auth abuse, rate/concurrency bounds, + slow consumer and duplicate bridge polling. +6. Measure bounded CPU, memory, PIDs, Redis/durable-store growth, request and + stream latency. Record untested infrastructure gates honestly. +7. Tear down all beta containers, networks, volumes, images, keys and temporary + credentials; prove V1 topology/state and API remain unchanged. Freeze a + checksummed human and machine-readable report. + +**Verification and exit gate:** + +- Existing V2 contract/security/unit suites and the Phase 9.0-A regression + matrix pass with zero unexplained domain mismatch. +- Exactly one stream replica is active; failover increments the fencing epoch + and stale-owner operations fail closed. +- Authentic V1 and V2 closed bars match exactly for identity, interval, OHLCV, + timestamps and finality; replay/live offsets are contiguous with no duplicate + external event. +- Missing credentials, wrong audience/environment/scope/consumer, malformed + requests and expired/tampered cursors fail closed with typed errors. +- Dependency failure makes readiness unavailable while V1 fallback remains + healthy. Recovery is bounded and does not require V1 restart. +- Resource limits hold, no beta state enters production Redis, V1 OpenAPI paths + remain unchanged, and cleanup counters are all zero. +- Completion authorizes review of an isolated V2 beta only. Phase 9.1 and any + Rust/source authority promotion remain blocked on the mandatory production + infrastructure and exact-slice operator gates. + +**Rollback:** Stop and remove only the isolated Compose project and its volumes, +revoke beta credentials, verify zero beta keys in production Redis and continue +the unchanged V1 path. V1 requires no replay, resubscription or restart. + +**Completed:** + +- Reused the frozen Phase 7 V2 query/stream/bridge topology with dedicated + Phase 9.0-B projects, AOF Redis, stores, credentials, ports and namespaces. +- Built and certified immutable candidate revision `1c881389b4ee21a153903505822c61512b176044` + as non-root UID/GID `10001`, read-only root, no host source bind and bounded + resources. +- Added exact provider-bar parity, continuous bridge, immutable provenance, + adversarial/security, capacity, V1 topology and deterministic cleanup gates. +- Fixed rootless evidence ownership and complete Compose profile activation in + the certification harness; both defects now have regression coverage. + +**Verification:** + +- Full Python regression: `351` tests, `346` passed, `5` skipped, `0` failed. +- Real provider slice: `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`; + canonical mismatches `0`, generated market events `0`, duplicate timestamps + `0`, non-final bars `0`, execution-eligible events `0`. +- Query load: normal `110.204 req/s`, burst `65.012 req/s`; `0` errors; + p99.9 `72.211 ms` and `486.649 ms`. Stream throughput was `1815.252` + events/s and measured end-to-end freshness was `6695.751 ms`. +- Authentication, entitlement, cursor tamper/expiry/scope, malformed/oversized + request, rate limit, active/passive fencing, Redis outage/recovery, slow + consumer isolation and replay continuity all passed fail-closed gates. +- Production V1 container/image/start time and OpenAPI digest were unchanged; + beta containers/networks/volumes/image tags and production beta keys after + cleanup were all `0`. +- Frozen evidence: [human report](upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md), + [machine decision](upgrade/evidence/phase90b-isolated-v2-beta.json), + [continuous parity](upgrade/evidence/phase90b-continuous-bridge.json), + [capacity](upgrade/evidence/phase90b-capacity.json), + [security](upgrade/evidence/phase90b-security-adversarial.json) and + [checksums](upgrade/evidence/phase90b-evidence.sha256). + +**Technical debt / decision gate:** No in-scope Phase 9.0-B defect remains. +This result authorizes only isolated read-only beta review. Phase 9.1 remains +blocked on replicated production transport, OTel/alert routing, workload +identity/RBAC, external secret rotation, signature admission, independent DR, +complete critical-consumer registration and explicit exact-slice approval. + +##### 9.0-C Production Prerequisites + +**Status:** `COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL` (2026-08-18) + +**Purpose:** Turn every Phase 9 production prerequisite into an explicit, +machine-verifiable, fail-closed gate. Reuse valid Phase 6/8/9 evidence without +misrepresenting same-host rehearsal as independent production infrastructure. +This subphase does not deploy a public V2 endpoint, promote Rust, change V1 +authority or approve an exact slice by implication. + +**Guide index:** + +- [Phase 9 production prerequisite boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-f--phase-90-c-production-prerequisite-boundary) +- [Deployment architecture](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#29-deployment-architecture) +- [Migration and authority](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#30-migration-strategy-no-big-bang-rewrite) +- [Operational runbooks](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#34-operational-runbooks) +- [Performance policy](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#37-performance-engineering-policy) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Invariants:** + +- A local container, SQLite spool, in-process metric buffer, debug exporter, + self-signed test key or same-host broker replica can prove code behavior but + cannot satisfy a production/failure-domain gate. +- Evidence is immutable, checksummed, scoped, expiring where applicable and + attributable to an operator or workload identity. Missing, malformed, + expired, lower-scope or contradictory evidence blocks promotion. +- Authority state is persistent and transitions by compare-and-swap. Entering + `RUST_CANARY` or `RUST_PRIMARY` requires a passing prerequisite bundle and an + explicit exact-slice approval; config booleans cannot bypass this rule. +- Existing V1 containers, OpenAPI, Redis namespaces and venue subscriptions + remain unchanged throughout isolated certification. + +**Implementation tasks:** + +1. Add a provider-neutral prerequisite policy covering replicated transport, + production telemetry/alert acknowledgement, workload identity/RBAC/network + policy, external secret rotation, signed-image admission, PostgreSQL PITR, + object-store restore, independent DR, Redis/projector rebuild, consumer + registration/rollback and exact-slice approval. +2. Add strict evidence and exact-slice schemas plus a deterministic evaluator + that emits `GO` only when every required production gate passes. Preserve + local rehearsal as `LOCAL_ONLY`, never silently upgrade its scope. +3. Add additive PostgreSQL authority/prerequisite/audit tables and a CAS + transition function with immutable audit, stale revision/owner/lease/plan + rejection and guarded canary/primary transitions. +4. Freeze a candidate manifest for the bounded Binance USD-M BTCUSDT TRADE + slice in `RUST_SHADOW`; public and legacy writes remain forbidden. +5. Re-run unit/contract/migration tests and applicable isolated broker, + security, recovery and V1 compatibility checks. Test malformed, missing, + stale, expired, forged, lower-scope and conflicting evidence. +6. Produce a compact machine/human gate report, explicit blocker inventory, + deployment/rollback runbook and portable checksums; remove all disposable + resources and prove V1 unchanged. + +**Exit gate:** + +- Code/schema/migration/evaluator and local certification may close as + `COMPLETE_CONTROL_PLANE` while the overall decision remains `NO_GO_EXTERNAL`. +- `PRODUCTION_PREREQUISITES_PASS` requires real replicated/failure-domain, + observability/page acknowledgement, identity/secret/admission, restore/DR, + consumer-owner and operator approval evidence. No test fixture can satisfy it. +- Phase 9.1 remains blocked unless the exact bundle decision is `GO`, its + artifact/config/contract digests match the candidate, and the approval names + the exact authority slice. + +**Rollback:** Keep V1 authoritative, remove only Phase 9.0-C disposable test +resources, and retain additive control-plane/audit records. Revoking or expiring +any prerequisite evidence immediately restores `NO_GO`; it never starts or +restarts a producer. + +**Implementation and verification (2026-08-18):** + +- Added a provider-neutral 12-gate production policy, strict candidate/evidence + models and deterministic evaluator. Unknown gates, nested secret fields, + duplicate bindings, unsafe artifact paths, invalid hashes/timestamps, stale or + lower-scope evidence, semantic threshold failures and candidate mismatches all + fail closed. +- Froze candidate digest + `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` + at `RUST_SHADOW`; public/legacy writes remain disabled. Release provenance + includes image/SBOM/signature, contract, normalizer, adapter, config, catalog, + source-policy, partition-plan and rollback revisions. +- Added additive prerequisite-bundle, authority-slice and immutable transition + audit schema. The database CAS rejects stale state/revision/owner/lease/plan, + stale owners, missing terminal watermark, missing/expired hold windows, + `NO_GO`, expired or candidate-mismatched bundles and audit mutation. +- Isolated PostgreSQL migration applied twice successfully. Nine negative safety + outcomes passed, two valid state transitions were audited, production + mutations were zero and the disposable container was removed. +- Targeted control-plane suite passed `13/13`; focused cross-phase candidate + suite passed `29/29`; full candidate-image suite ran `364` tests with `359` + pass, `5` intentional skips and `0` failures. +- Machine evaluation correctly produced `NO_GO_EXTERNAL`: `0/12` production + gates passed because the available proof is local, missing or explicitly + blocked. V1 container identity, topology and `/v1/health` remained unchanged + and healthy before/after evaluation; no production database or Redis was + mutated. +- Workspace governance is canonical in `/home/bobby/AGENTS.md` and its + repository-tracked [AGENTS.md](AGENTS.md) copy; Rules 40-42 make plan + synchronization, explicit approval boundaries and evidence-bound final + reporting mandatory for every later slice and cloned workspace. +- Runbook and portable evidence are frozen at + [phase90c-production-prerequisites.md](docs/runbooks/phase90c-production-prerequisites.md), + [phase90c-production-prerequisites.json](upgrade/evidence/phase90c-production-prerequisites.json), + [phase90c-authority-migration.json](upgrade/evidence/phase90c-authority-migration.json) + and [phase90c-evidence.sha256](upgrade/evidence/phase90c-evidence.sha256). + +**Technical debt / decision gate:** No in-scope control-plane defect remains. +The 12 blockers are deliberately external production deployment/operator gates, +not evidence that can be fabricated in this repository. Phase 9.1 remains +blocked until a fresh exact-candidate bundle evaluates `GO`; completing this +subphase does not authorize public V2, Rust canary or any authority cutover. + +**Phase 9.1 prerequisites:** + - Close every applicable Phase 6 `NO-GO` blocker with real infrastructure evidence. - Deploy production durable transport, OTel collector/dashboards/alerts, @@ -2675,46 +3067,427 @@ must prove: #### 9.1 Rust Canary -- Transition only the selected slice to `RUST_CANARY` by CAS. -- Keep Python as the sole authoritative public writer. -- Dual-read/compare canonical Rust output against the same authentic frame range. -- Run live, replay, reconnect, lease loss, producer crash, broker failover, - projector restart and slow-consumer tests. -- Exercise automated blocking thresholds without allowing dual public writes. -- Verify the compatibility projector remains fed only by the active authority. -- Hold the canary for the approved traffic/session window. -- Revert to `RUST_SHADOW` on any unexplained semantic mismatch. +**Status:** `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED` + +**Purpose:** Implement and certify the exact-slice Rust canary path while Python +remains the sole authoritative public/V1 writer. Because Phase 9.0-C currently +returns `NO_GO_EXTERNAL`, this phase may run only isolated rehearsal and +fail-closed authorization tests. It must not persist a production +`RUST_CANARY`, publish public/legacy output or imply production readiness. + +**Guide index:** + +- [Phase 9.1 canary boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-g--phase-91-rust-canary-boundary) +- [Migration and authority](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#30-migration-strategy-no-big-bang-rewrite) +- [Performance policy](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#37-performance-engineering-policy) +- [Operational runbooks](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#34-operational-runbooks) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Invariants:** + +- The exact candidate remains `BINANCE / USDM / PERPETUAL / TRADE / BTCUSDT / + partition-plan epoch 1`; a changed image, contract, catalog, source policy, + normalizer, adapter or partition plan creates a new candidate digest. +- Python V1 remains the only public and legacy writer. Rust canary output uses a + dedicated isolated canonical namespace; two public writers are impossible. +- Production authorization consumes the strict Phase 9.0-C decision. Missing, + stale, `NO_GO`, mismatched or unapproved evidence cannot be bypassed by an + environment variable, test flag or direct state boolean. +- Every canary publication binds exact slice, owner, authority revision, lease + epoch and partition-plan epoch. Sink-side fencing rejects stale/conflicting + writers, not only producer-side self-checks. +- Parity uses the same authentic captured provider frames and compares identity, + exact decimals, timestamps, sequence, quality, event ID, payload hash and + deterministic bytes. Generated events are test-only and never parity proof. +- Any correctness mismatch, open gap, duplicate external output, stale-writer + attempt, lag/freshness/resource breach or authority ambiguity blocks the + canary. A hold-down prevents automatic re-entry/flapping. + +**Implementation tasks:** + +1. Add a provider-neutral Phase 9 canary manifest and strict authorizer bound to + the Phase 9.0-C candidate/evidence decision. Separate production activation + from isolated rehearsal at the type/API boundary. +2. Extend the Rust authority/sink core with a versioned Phase 9 record carrying + owner, authority/lease/partition epochs, candidate and prerequisite bundle, + start watermark, approval and hold window. Preserve the Phase 8 V1 internal + record decoder for compatibility. +3. Enforce sink fencing for wrong slice, owner, revision, lease, partition plan, + target, watermark and expired/blocked state. `RUST_CANARY` permits only + isolated canary canonical output; public and legacy targets remain denied. +4. Add deterministic same-frame parity and guardrail evaluation with bounded + lag/freshness/resource thresholds, first-failure reason, immutable + observations and anti-flapping/hold-down behavior. +5. Build an isolated canary certification harness using the frozen authentic + Binance capture and replicated test broker. Exercise normal/burst/replay, + process restart, lease loss, stale owner, broker restart/min-ISR failure, + slow consumer, guardrail block and rollback to shadow. It must verify zero + public/legacy writes and unchanged V1 topology/health. +6. Produce strict machine/human evidence, checksums and an operator runbook. + Clean all disposable topics/groups/containers/networks/volumes/images and + record exact test counts and unresolved external gates. + +**Verification and exit gate:** + +- Unit/contract/golden tests cover malformed manifests, candidate mismatch, + `NO_GO`, expiry, stale authority fields, forbidden targets, guardrail triggers + and anti-flapping. +- Python/Rust replay over the same authentic frame range has zero unexplained + mismatch across clean process restarts and burst repetition. +- Isolated broker/recovery tests preserve one canary owner, reject stale writes, + keep public/legacy write counts at zero and clean all test resources. +- Full Data Layer suite passes and V1 container identity, OpenAPI/health, Redis + namespaces and source ownership remain unchanged. +- With the current Phase 9.0-C decision, the maximum allowed closure is + `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`. Only a fresh exact-candidate + `GO` bundle plus explicit operator approval can advance the production state + to `RUST_CANARY` and begin the approved hold window. + +**Implementation journal (2026-08-18):** + +- `COMPLETE` control-plane slice: strict production authorizer consumes the + exact Phase 9.0-C decision/candidate/bundle and rejects `NO_GO`, stale, + incomplete, mismatched or V1-mutating evidence. Isolated rehearsal is a + separate non-production authorization mode. +- `COMPLETE` guardrail slice: immutable observations, zero-tolerance semantic + checks, bounded lag/freshness/resource checks, first-failure capture and + explicit reset only after hold-down. Targeted result: 9/9 tests pass in a + disposable container; production mutations remain zero. +- `COMPLETE` Rust authority/sink v2: Phase 8 record/sink v1 remains unchanged; + v2 binds slice/owner/revision/lease/partition-plan/candidate/bundle/watermark, + denies public/V1 targets and advances watermark only after durable ACK. + Rust result: clippy with `-D warnings` passes; 16/16 focused crate tests pass. +- `COMPLETE` authentic same-frame parity: 128 frozen real Binance USD-M trade + frames repeated 200 times produced 25,600 canonical events. Python and three + clean Rust process runs matched exact record and aggregate hashes with zero + semantic mismatch. Measured throughput was 27,115.455 events/s for Python and + at least 350,581.025 events/s for Rust on this host; this is certification + evidence, not a capacity promise. +- `COMPLETE` replicated-broker rehearsal: exact authority transitions were + `RUST_SHADOW -> RUST_CANARY -> BLOCKED -> RUST_SHADOW`; one-replica-loss ACK, + below-min-ISR fail-closed, full broker restart, compacted authority recovery, + immutable audit ordering and 64-record slow-consumer catch-up all passed. + Public and legacy writes remained zero. +- `COMPLETE` verification: Rust format/clippy pass; Rust workspace 32/32; focused + Phase 8-9 Python matrix 73/73; full Python suite 381/381 with 5 intentional + skips. Certification cleanup left zero isolated containers, networks and + volumes; V1 health stayed HTTP 200 and topology remained unchanged. +- `COMPLETE` evidence and operations: [machine evidence](upgrade/evidence/phase91-rust-canary-certification.json), + [human report](upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md), + [checksums](upgrade/evidence/phase91-evidence.sha256) and + [runbook](docs/runbooks/phase91-rust-canary.md). The harness now separates + compacted authority partition reads from audit/consumer catch-up semantics and + uses process-loss fault injection through surviving bootstrap nodes. +- `OPEN EXTERNAL GATE`: Phase 9.0-C production infrastructure/operator evidence + remains `NO_GO_EXTERNAL`; same-host replicas are not an independent failure + domain. No production canary authority or V1/public mutation was performed. + +**Rollback:** In rehearsal, persist a higher-revision `RUST_SHADOW` record, +fence the canary owner, reconcile the bounded cursor range and remove only the +isolated namespace. In production, use the formal rollback protocol above; do +not restart Python or edit authority state outside the CAS/audit path. #### 9.2 Bounded Rust Primary -- Execute the formal terminal-watermark cutover. -- Increment authority revision and lease epoch atomically. -- Enforce final-sink and compatibility-projector fencing. -- Disable only the exact Python venue subscription after Rust ownership is - accepted and the handoff boundary reconciles. -- Monitor consumer state, quality, freshness, broker/projector lag, queue/spool, - resource headroom and stale-owner attempts under enhanced alerting. -- Rehearse the formal rollback within the approved RTO before closing the - rollback window. -- Preserve V1 contract behavior and V2 snapshot/cursor/replay continuity. +**Status:** `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED` + +**Purpose:** Implement the terminal-watermark ownership protocol required to +promote one exact, already-certified `RUST_CANARY` slice to `RUST_PRIMARY` while +preserving one authoritative writer, V1 compatibility and V2 cursor/replay +continuity. Phase 9.0-C is still `NO_GO_EXTERNAL`; therefore this phase may +certify only an isolated bounded-primary rehearsal. It must not disable a real +Python subscription, mutate production authority or write production +canonical/public/legacy destinations. + +**Guide index:** + +- [Phase 9.2 bounded-primary boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-h--phase-92-bounded-rust-primary-boundary) +- [Formal authority model](#authority-state-machine) +- [Terminal-watermark protocol](#formal-cutover-watermark-protocol) +- [Formal rollback protocol](#formal-rollback-protocol) +- [Operational runbooks](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#34-operational-runbooks) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Exact scope and invariants:** + +- The rehearsal remains bound to the frozen Phase 9.1 candidate slice and + candidate/partition/config/schema digests. Any changed identity creates a new + candidate and invalidates inherited evidence. +- A terminal checkpoint is immutable and identifies the old owner, authority + revision, lease epoch, partition-plan epoch, source session/generation, + terminal event and final durable watermark `W`. +- Primary authorization requires an accepted handoff whose parity range is + gap-free and mismatch-free through `W`. Rust begins authoritative output at + exactly `W + 1`; `<= W` is stale and `> W + 1` before the boundary commit is + an open gap. +- Authority owner, revision and lease epoch change together in one persistent + compare-and-swap transaction. Revision advances exactly by one and a changed + owner requires a strictly newer lease epoch. +- Final canonical sink and V1 compatibility projector independently enforce the + same authority record. Producer self-checks are insufficient. The old owner, + stale lease/revision, wrong plan, wrong destination and duplicate watermark + all fail closed. +- Only `RUST_PRIMARY` may emit authoritative canonical, public V2 and legacy V1 + compatibility output for the promoted range. Shadow/canary/blocked states + retain their narrower Phase 9.1 permissions. +- Isolated rehearsal topics may model final/public/legacy projection but are + explicitly test-only and must be unique, disposable and counted separately. + Production write counts remain zero. +- The exact Python subscription is disabled only after durable authority CAS, + sink/projector acceptance and boundary reconciliation. Under current + `NO_GO_EXTERNAL`, this action is simulated only; the real V1 topology remains + unchanged. +- Rollback records the last accepted Rust watermark, fences Rust first, then + grants the immutable Python rollback owner a new revision/lease and resumes + from the next reconciled watermark. Restarting Python without authority is + forbidden. + +**Implementation tasks:** + +1. Add strict bounded-primary authorization that consumes a fresh exact + Phase 9.0-C `GO`, completed production canary hold evidence, candidate/bundle + identity and explicit slice approval. Keep isolated rehearsal a distinct + non-production type that cannot be converted to production authority. +2. Add immutable terminal-checkpoint and accepted-handoff contracts plus a + PostgreSQL migration. A database trigger must prevent direct or legacy + transition paths from entering `RUST_PRIMARY` or rollback `PYTHON_PRIMARY` + without accepted matching handoff evidence. +3. Extend the provider-neutral Rust authority core additively. Preserve Phase 8 + v1 and Phase 9.1 v2 decoders; add a v3 primary record/state machine with + exact `revision + 1`, strict owner/lease CAS, terminal boundary and rollback + transitions. +4. Add final-sink and compatibility-projector fencing with independent durable + watermark tracking per target. Authority changes between ACK and watermark + commit must fail closed and remain recoverable by deterministic replay. +5. Build an isolated replicated-broker certification over authentic frozen + provider frames. Exercise `N-1/N/N+1`, duplicate/out-of-order/gap input, + stale/zombie writer, CAS conflict, crash before/after CAS, sink/projector + restart, one-replica loss, below-min-ISR, full broker restart, slow consumer + and bounded rollback. +6. Verify exact Python/Rust canonical parity, output counts/order/digests, V1 + projected schema behavior and V2 snapshot/cursor/replay continuity. Measure + cutover/rollback RTO without turning the measurement into a production SLO. +7. Freeze machine/human evidence, checksums and a runbook; remove only + Phase 9.2 disposable resources. Record V1 topology/health before and after. + +**Verification and exit gate:** + +- Rust unit/contract tests cover every state transition, malformed checkpoint, + boundary off-by-one, stale owner/revision/lease/plan, wrong target, duplicate, + gap, ACK failure and rollback path. +- PostgreSQL migration tests prove transactionality, append-only evidence, + direct-primary bypass rejection, CAS conflict rejection and both handoff + directions. +- Replicated-broker rehearsal has one authoritative owner, zero unexplained + semantic mismatch, zero external duplicate/gap, ordered compatibility output + and recovery after process/broker failure. +- Full Python and Rust suites pass. V1 OpenAPI/health, container identity, + Redis namespaces and live subscription ownership remain unchanged. +- While Phase 9.0-C remains `NO_GO_EXTERNAL`, maximum closure is + `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED`. Production promotion + requires a fresh exact `GO`, successful real canary hold and explicit + operator approval; repository tests cannot fabricate those gates. + +**Implementation journal (2026-08-18):** + +- `COMPLETE` control and evidence contracts: production primary authorization + requires a fresh exact Phase 9.0-C `GO`, completed production canary hold, + immutable rollback manifest and explicit bounded-slice approval. Isolated + rehearsal remains a separate non-production authorization type. +- `COMPLETE` additive persistence boundary: migration `0007` adds immutable + terminal checkpoints and accepted handoffs, a primary/rollback bypass guard + and exact handoff-aware CAS. Disposable PostgreSQL smoke passed both + Python-to-Rust primary and Rust-to-Python rollback, rejected stale/direct CAS + and preserved evidence across idempotent migration replay. +- `COMPLETE` Rust authority/sink core: v1/v2 decoders remain intact; v3 binds + terminal checkpoint, accepted handoff, exact revision/lease/plan and + independently contiguous canonical/public/legacy target watermarks. Loading + an already-primary authority is fail-closed until each target reconstructs + its durable contiguous watermark. Crash-before-CAS reconstruction, duplicate + W rejection and exact W+1 restart are covered. Rust fmt/clippy pass and the + full workspace is 40/40; the focused Python matrix is 87/87 and the full + Python suite is 395/395 with 5 intentional skips. +- `COMPLETE` isolated replicated-broker certification: 25,600 authentic Binance + USD-M trade events across three clean Rust runs produced zero semantic + mismatch. The authority path completed `RUST_CANARY -> RUST_PRIMARY -> + BLOCKED -> ROLLBACK_PENDING -> PYTHON_PRIMARY`; canonical/public V2/legacy V1 + projections are identical and gap-free for watermarks 101..181. A fresh Rust + process recovered watermark 180 independently from all three durable targets, + rejected writes before restore and duplicate 180 after restore, then emitted + exactly 181. One-replica-loss ACK and below-min-ISR fail-closed passed. +- `COMPLETE` measured isolated operations and cleanup: cutover 22.200 ms, formal + rollback 533.237 ms and delayed-consumer catch-up 24.524 s. Production public + and legacy writes remained zero; V1 health stayed 200/200 with unchanged + topology; disposable containers/networks/volumes ended at 0/0/0. Migration + smoke passed both handoff directions and checksum verification passed. The + builder now carries pinned rustfmt/clippy, and the rebuilt runtime digest + exactly matches the certified image digest. +- `OPEN EXTERNAL GATE`: Phase 9.0-C production prerequisites and real canary + hold remain unavailable; V1 stays authoritative. + +**Rollback:** Before production authorization, remove only isolated Phase 9.2 +topics/groups/containers/networks/volumes and retain Phase 9.1 code/evidence. A +future production rollback must follow the formal protocol above and may never +use a direct owner flag or uncoordinated Python restart. #### 9.3 Hold, Close And Expand Independently -- Keep the slice in an enhanced-observation hold period. -- Close the rollback window only after all SLO, consumer and authority evidence - remains clean. -- Update the consumer/authority registries and freeze the production report. -- Decide independently whether to expand: - 1. more partitions/instruments of the same TRADE capability; - 2. BBO; - 3. L2 snapshot/delta/checksum; - 4. BAR lifecycle/revision; - 5. another venue/market. -- Repeat full capability/provider/chaos/capacity/rollback certification for each - expansion. Do not infer certification transitively. -- Remove a replaced Python hot-path implementation only after zero ownership, - tested rollback posture and approved repository cleanup. Reusable adapter and - compatibility knowledge remain until explicit decommission. +**Status:** `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED` + +**Purpose:** Add the provider-neutral post-primary control plane that observes +one exact `RUST_PRIMARY` slice, decides whether its rollback window may close, +freezes consumer/authority evidence and creates independent expansion +candidates. Phase 9.0-C remains `NO_GO_EXTERNAL` and Phase 9.2 is not production +authoritative, so this phase implements and certifies the protocol only in +isolated scope. It must not manufacture a production hold, close a real +rollback window, mutate production authority or grant another feed/venue +transitive approval. + +**Guide index:** + +- [Phase 9.3 hold/closure/expansion boundary](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#appendix-i--phase-93-hold-closure-and-independent-expansion) +- [Authority state machine](#authority-state-machine) +- [Formal rollback protocol](#formal-rollback-protocol) +- [Verification matrix](#verification-matrix) +- [Production acceptance checklist](upgrade/quant-data-layer-fund-grade-upgrade-architecture.md#41-production-acceptance-checklist) + +**Decision boundary and invariants:** + +- Hold/closure is a control-plane lifecycle around an existing exact authority + record. It does not add a new data-plane authority state and does not weaken + `RUST_PRIMARY`, `BLOCKED`, `ROLLBACK_PENDING` or `PYTHON_PRIMARY` fencing. +- Every observation binds slice, candidate, owner, authority revision, lease, + partition-plan epoch and monotonically increasing time/watermark. A changed + identity ends the hold; evidence from another owner or epoch cannot be mixed. +- Correctness breaches have zero tolerance: semantic mismatch, open gap, + duplicate external write, accepted stale writer, authority ambiguity, durable + ACK failure, projection divergence, consumer checkpoint regression or + unexplained source-quality failure blocks closure. +- Capacity/freshness/lag thresholds are explicit policy. Observations are + append-only, ordered, bounded and sufficiently dense for the approved hold + duration. Missing intervals fail closed; they are not interpolated. +- Closing the rollback window is an immutable registry decision, not deletion + of the Python rollback manifest. It requires a production-authorized primary, + a completed real hold, exact healthy consumer and authority registries, a + fresh rollback rehearsal, explicit operator/change-ticket approval and an + unchanged authority CAS identity at commit time. +- Expansion never inherits authority or certification. More instruments, + `BBO`, `L2`, `BAR` and another venue/market each create a new candidate digest, + required capability matrix and independent Phase 6/9 certification set. +- Runtime decommission requires zero ownership, zero active rollback dependency, + a closed governed window and explicit repository cleanup approval. Shared + contracts, adapters and compatibility knowledge are retained unless a + separate approved removal proves no consumer dependency. +- With current external gates, the maximum valid result is + `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED`. V1 and the Python + authority path remain unchanged. + +**Implementation tasks:** + +1. Add typed hold policy, observation, decision, registry snapshot, closure + approval and expansion manifest contracts. Validate strict schemas, identity, + time ordering, zero-tolerance correctness fields and bounded thresholds. +2. Add a stateful hold evaluator with monotonic observation ordering, required + sample density, exact authority continuity and sticky fail-closed breach + behavior. Recovery requires a new hold identifier, never an in-place reset. +3. Add a production closure authorizer that rejects the current Phase 9.0-C + `NO_GO_EXTERNAL`/Phase 9.2 rehearsal evidence and accepts only exact real + production primary, hold, consumer, authority, rollback and operator records. +4. Add append-only PostgreSQL hold observations/decisions, authority closure and + expansion registries. A closure function must lock and recheck the exact + current authority record; it records closure only and never changes owner, + state, revision, lease or watermark. +5. Add independent expansion planning for more instruments/partitions, `BBO`, + `L2`, `BAR` and another venue/market. Persist + `INDEPENDENT_CERTIFICATION_REQUIRED`, forbid transitive evidence and bind a + new candidate/partition-plan identity where applicable. +6. Build disposable PostgreSQL migration smoke plus isolated control-plane + certification. Cover clean hold, sparse/out-of-order observations, every + zero-tolerance breach, threshold breach, stale authority CAS, registry + mismatch, duplicate closure, immutable evidence, all expansion classes and + decommission refusal. +7. Freeze machine/human evidence, checksum and operator runbook. Record V1 + health/topology before and after, production mutations as zero and remove + only Phase 9.3 disposable resources/images. + +**Verification and exit gate:** + +- Unit/contract tests cover valid and malformed hold observations, identity and + epoch drift, timestamp/watermark regression, missing density, correctness and + resource breaches, sticky blocking and deterministic decision digest. +- Closure authorization tests prove current evidence is denied, complete + production-shaped test fixtures are accepted only in test scope, and every + missing/stale/mismatched registry, rollback or approval field fails closed. +- PostgreSQL tests prove migration idempotency, append-only evidence, exact + authority row locking, stale CAS rejection, duplicate closure rejection and + zero authority-field mutation after closure. +- Expansion tests prove no transitive certification, capability-specific gate + requirements, unique candidate identity and zero write authority. +- Full Python/Rust compatibility suites pass. V1 health, API/SDK/Redis contracts, + running topology and authority ownership remain unchanged. +- While external gates remain unavailable, production hold duration, production + closure and real expansion authority remain explicitly untested and blocked. + +**Implementation journal (2026-08-19):** + +- `COMPLETE` plan boundary: scope, invariants, test matrix and Appendix I were + frozen before code changes on `feat/phase93-hold-close-expand`, stacked on + certified Phase 9.2. +- `COMPLETE` typed control contracts: strict hold policy/identity/observation/ + decision, frozen consumer and authority snapshots, rollback rehearsal, + operator closure approval, independent expansion manifests and decommission + assessment are implemented. Focused domain and migration-contract tests pass + 16/16, including every correctness/resource breach and current no-go denial. +- `COMPLETE` additive persistence and actual PostgreSQL smoke: migration `0008` + creates append-only hold, registry, rollback, approval, closure, expansion and + decommission records. Two holds, three observations, two decisions, one + closure, five expansion types and two decommission decisions passed. Closure + left authority exactly `RUST_PRIMARY:4:rust-primary:2:100`; approval and + closure UUIDs are distinct, the frozen closure digest binds every expansion, + and stale CAS, dirty pass, incomplete gates and all tested mutations failed + closed. Idempotent replay, stable-readiness startup and scoped container + cleanup passed. +- `COMPLETE` isolated certification and operator tooling: parent Phase 9.2 + provenance remains 25,600 authentic events with zero semantic mismatch; the + accelerated hold is explicitly `TEST_CONTROL_PLANE_FIXTURE` and has no + production authority. Current no-go is rejected, five expansion manifests + remain independently uncertified/write-disabled, Python decommission with a + rollback dependency is denied, and production mutations remain zero. V1 + health stayed 200/200 with unchanged topology. Focused Phase 9.3 tests pass + 20/20 and evidence checksums pass. +- `COMPLETE` import/runtime boundary: control-plane imports no longer eagerly + load the alpha SDK. Existing `PaperAlphaCanary` and `sdk_requirement` exports + remain API-compatible through lazy loading and passed container smoke. +- `COMPLETE` final compatibility and cleanup gates: full Python is 415/415 + with 5 intentional skips; Rust fmt/clippy pass and the full workspace is + 40/40. Final migration and certification reruns pass, evidence checksums pass, + V1 health remains 200 with unchanged topology, production mutations are zero, + no Phase 9.3 container/network/volume remains and the temporary Rust builder + image was removed without global prune. +- `COMPLETE` post-closure CI hotfix (2026-08-19): GitHub run + `32210830176` showed that PR #7 full unit tests passed, while the additive + PostgreSQL migration step failed after one transient ready probe immediately + preceded an init restart. `phase5_migration_smoke.sh` now requires eight + consecutive successful probes over two seconds before `createdb`. The exact + CI-image unit command passes 415/415 with 5 intentional skips and the failing + migration gate passes 3/3 consecutive runs. All disposable Compose/PostgreSQL + resources were removed; no schema, authority or running service changed. +- `COMPLETE` post-closure security hotfix (2026-08-19): PR #7 run + `32211508679` passed unit, migration, Redis, Rust artifact and performance + gates, then Trivy rejected nine Debian util-linux packages for fixed HIGH + `CVE-2026-53615`. The runtime stage now applies repository security upgrades, + installing `2.41.5-0+deb13u1` instead of vulnerable `2.41-5`. A pinned Trivy + 0.74.0 tar scan reports 0 HIGH/0 CRITICAL and the rebuilt exact CI image still + passes 415/415 unit tests with 5 intentional skips. No CVE was ignored or + allowlisted. +- `OPEN EXTERNAL GATE`: Phase 9.0-C remains `NO_GO_EXTERNAL`; no real primary + owner, production hold interval or operator closure approval exists. + +**Rollback:** Before production authority exists, remove only Phase 9.3 test +schemas, fixtures, evidence runtime and images; retain append-only repository +evidence. A future production closure cannot be reversed by deleting its row. +An incident still uses the formal `BLOCKED -> ROLLBACK_PENDING -> +PYTHON_PRIMARY` authority protocol and a new audit decision. ### Verification Matrix @@ -2786,7 +3559,13 @@ Completing one Rust primary slice does not mark every venue/feed complete. ### Completed -- Not started. User approval is required independently for each authority slice. +- Phase 9.0-A/9.0-B runtime and isolated V2 beta closure, Phase 9.0-C strict + prerequisite control plane, Phase 9.1 isolated canary, Phase 9.2 isolated + bounded-primary protocol and Phase 9.3 hold/closure/expansion control plane + are implemented and certified at their explicitly non-production scopes. +- No production authority slice, real hold closure, expansion or Python runtime + decommission is approved. Each requires independent operator approval and + production-scope evidence. ### Technical Debt / Decision Gate @@ -3044,3 +3823,2193 @@ The Phase 7-9 program achieves its target when: state within approved objectives. 10. Further venue/feed expansion uses the same contracts, capability manifests and evidence gates without redesigning the core. + + +## 19. V2 Stable Rust Core And Binance/OKX/VN Equal-Source Closure + +**Status:** `PHASE A COMPLETE / PHASE B IN PROGRESS (ISOLATED STABLE) / RUNTIME CUTOVER NOT AUTHORIZED` + +### Purpose + +Close exactly two product gaps without creating another open-ended architecture +program: + +1. make Rust the authoritative realtime data core while Python remains the + stable REST/gRPC/SDK/history/control and compatibility edge; +2. make Binance, OKX and VN markets first-class sources under the same domain, + quality, durability and authority rules instead of treating OKX/VN as + reference-only or second-class paths. + +Stable V2 consumer contracts are independent from implementation language. +Alphas and Trading System must use the same V2 API/SDK while individual +venue/feed slices can move between Python rollback and Rust authority without a +consumer rewrite. + +### Frozen Product Scope + +The equal-source GA baseline is capability-equivalent rather than falsely +feed-identical. A venue is first-class when every feed it actually exposes uses +the same identity, exact-decimal/unit, lineage, quality, durability, recovery, +authority and consumer-readiness gates: + +| Venue/product | TRADE | BBO | BAR | L2 | +|---|---:|---:|---:|---:| +| Binance USD-M perpetual | required | required | required | capability-gated | +| Binance Spot | required | required | required | capability-gated | +| OKX SWAP perpetual | required | required | required | capability-gated | +| OKX Spot | required | required | required | capability-gated | + +| VN venue/product | TRADE | QUOTE | BAR | Provider edge | Rust core | +|---|---:|---:|---:|---|---| +| VN derivatives (DNSE primary) | required | capability-gated | required | Python/vendor SDK | required | +| VN equities (DNSE/vnstock policy) | required | capability-gated | required | Python/vendor SDK | required | + +`capability-gated` is not an optional quality class: the +capability registry records the provider's real product/feed support and each +advertised feed passes the same strict gates. The platform does not fabricate +crypto-style BBO or book semantics for a VN provider/feed that has not exposed +and certified them. DNSE and vnstock remain separate providers with immutable provenance and +audited source switching; `MARKET_CLOSED` is not stale or offline. + +L2 remains in the common canonical model and Rust state machine, but is not +advertised stable for either crypto venue until both Binance snapshot+delta and +OKX snapshot+`seqId/prevSeqId` resync pass the same production gate. OKX SBE, +VIP-only 10 ms books and Deribit options remain independent capabilities. + +### Non-Negotiable Invariants + +- Rust owns the common realtime core: raw-envelope validation, exact + canonicalization, quantity-unit enforcement, source session/generation, + ordering, deduplication, gap detection, backpressure, deterministic event + identity, quality transitions and durable publication. Rust also owns native + realtime transport/decoding for high-volume Binance and OKX feeds. +- Python owns V2 REST/gRPC and SDK facades, historical/warmup orchestration, + consumer manifests, workload identity, readiness, observability/control, + compatibility projection and low-rate/vendor-SDK acquisition adapters. A + Python DNSE/vnstock edge may emit only authenticated versioned raw provider + envelopes; it may not bypass the Rust canonical/quality/durable core. +- Python may not reimplement Binance/OKX/VN realtime domain decisions after + promotion. It may validate/project canonical events and provide a bounded + rollback adapter. +- Kafka-compatible durable records are the handoff between Rust and Python. + Redis remains latest-state/compatibility projection, not lossless authority. +- Binance, OKX and VN use one canonical identity/quantity-unit/quality/authority + contract and one durable publication boundary. Provider-native channel names + and provider-specific missing-value conventions never leak into alpha code. +- V1 remains available during migration. No running producer, Redis namespace, + alpha or Trading System adapter changes before isolated parity and an explicit + cutover approval naming the blast radius. +- Public V2 version `2.0.0` freezes URI, schema, units, decimal, timestamp, + bar lifecycle, error and cursor semantics. Internal source promotion cannot + change that contract. + +### Phase A - Unified Rust Realtime Core And Binance/OKX/VN Parity + +**Goal:** Deliver one Rust realtime core with first-class Binance, OKX and VN +sources, capability-truthful feeds and a durable canonical output contract. + +**Implementation:** + +1. Add provider-neutral Rust subscription/session contracts and dedicated + Binance/OKX JSON adapters behind the existing canonical envelope. +2. Implement OKX public `trades`, `bbo-tbt` and business candle channels, + strict ACK correlation, public/business socket separation, less-than-30- + second ping/pong, 480 control requests/hour guard, maintenance-notice + reconnect and bounded jittered backoff. +3. Implement OKX trade aggregation identity, BBO replace-only semantics and + candle `confirm=0/1` lifecycle with exact decimals/timestamps. +4. Add canonical quantity-unit semantics before the V2 stable freeze. Preserve + venue-native raw quantity plus explicit unit; BAR distinguishes base, quote + and contract volume where supplied. Never map OKX derivative contracts, + Binance base quantity or VN shares/contracts into an ambiguous unitless + field. Unknown/missing unit fails closed at execution-grade boundaries. +5. Route DNSE/vnstock authenticated raw envelopes through the same Rust + canonical/quality/durable core. Preserve VN timezone/calendar, sparse/no- + trade semantics, derivative multiplier/unit, market-closed state and provider + switch provenance. No synthetic bar or zero-default repair is permitted. +6. Keep L2 sequencing in the shared core but fail capability activation unless + snapshot/delta continuity and entitlement are certified. +7. Publish raw/canonical records through the existing TLS/ACL, + idempotent-ACK Kafka client under authority fencing. Final sink and projector + advance watermarks only after durable ACK. +8. Add Python/Rust golden parity for Binance and OKX TRADE/BBO/BAR and VN + TRADE/BAR plus capability-gated QUOTE, including units, session/calendar + state, malformed and missing + fields, duplicate/out-of-order input, reconnect, stale generation, queue + saturation and broker interruption. +9. Run bounded authentic Binance/OKX WebSocket and DNSE/VN-provider smoke with + production writes zero; freeze provenance, counts, hashes, latency and + resource evidence. Market closure is tested separately and never replaced by + generated provider data. + +**Exit gate:** + +- zero correctness-critical Python/Rust mismatch; +- no unexplained loss, duplicate or gap; +- capability-truthful first-class matrices for Binance, OKX and VN; +- every canonical quantity has an unambiguous unit and native lineage; +- VN calendar/session/provider-fallback semantics pass strict tests; +- Rust fmt/clippy/workspace and full Python compatibility pass; +- no V1 runtime change and all disposable resources removed. + +### Phase B - Stable Python Edge, Consumer Migration And Release + +**Goal:** Publish Data Layer `2.0.0 Internal Stable` with Rust realtime core +behind stable Python V2 endpoints and migrate real consumers without a public +contract change. + +### Phase B Execution Boundary + +**Approved scope:** implement, deploy and certify an isolated `2.0.0` stable +candidate. This approval does not change V1/Rust production authority, restart +port `8100`, write current Redis namespaces, or authorize production cutover. +Cutover remains a separate transaction after the exact topology, service/image +digests, ports, credentials, volumes, affected consumers and rollback command +are presented to and approved by the operator. + +**Stable topology:** + +```text +Rust/Python acquisition edges + -> Kafka raw (RF3/minISR2, mTLS/ACL) + -> transactional Rust canonical core + -> Kafka canonical/quarantine (replay authority) + -> Python stable projector + -> active fenced stream_v2 gateway -> shared bounded SQLite query cache + -> dedicated Redis V2 latest + V1 compatibility projection + -> Kafka consumer checkpoint only after cache + Redis success + -> query_v2 replicas read the bounded cache + -> stream_v2 active/passive serves snapshot + signed cursor + replay + live +``` + +- The stable projector never calls a venue and never creates canonical market + semantics. It validates generated Protobuf and projects only Rust output. +- SQLite is a rebuildable bounded cache, never the replay authority. Kafka raw + and canonical topics remain authoritative for recovery. +- Active stream gateway commits a canonical event before fan-out. Projector + writes Redis and advances Kafka offsets only after that commit. Duplicate + replay is idempotent across gateway, SQLite and Redis checkpoints. +- Binance, OKX and VN consumers use the same V2 API/SDK and stable catalog. + Provider/feed capability remains truthful; unavailable feeds fail readiness. +- Dedicated stable Redis may contain root-shaped V1 compatibility keys only + inside the isolated deployment. Current `redis_marketdata` is never touched. +- Stable runtime roles, health and data-readiness are separate. API replicas do + not open provider sockets and one slow consumer cannot block projector or + unrelated stream partitions. + +**Phase B acceptance gates:** + +1. Stable catalog/query models cover TRADE/QUOTE/BAR units, source role, + lifecycle/finality, quality/freshness and market-session semantics. +2. Canonical-before-raw, duplicate replay, crash before/after cache/Redis/Kafka + checkpoint, active gateway failover, Redis loss/rebuild and Kafka restart + recover without acknowledged loss or duplicate external publication. +3. V1 OpenAPI/SDK/Redis golden compatibility remains unchanged; V2 OpenAPI and + SDK freeze as `2.0.0` with generated artifacts current. +4. Registered monitoring, Binance alpha, OKX alpha, VN alpha and Trading System + paper manifests complete warmup -> cursor -> replay -> live and restart with + no direct venue connection or domain mismatch. +5. Real-provider lineage from Phase A is replayed through stable query/stream/ + compatibility projection; generated data is limited to explicit failure and + capacity tests. +6. Full Python/Rust/Buf/OpenAPI/security/capacity suites pass, immutable images + and compact evidence are frozen, all disposable resources are removed, and + V1 health/topology remain unchanged. + +**Rollback before cutover:** stop only the isolated stable project, revoke its +credentials and remove its dedicated topics/groups/Redis/cache volumes. V1 +continues unchanged. Production rollback after a later approved cutover must +use a newer authority revision and the frozen V1 manifest; it is not exercised +by implication in Phase B implementation. + +**Implementation:** + +1. Replace beta/shadow product labels with stable `2.0.0` labels while + retaining authority state as runtime metadata. +2. Add a canonical V2 deployment manifest containing Rust Binance/OKX ingestion, + Python DNSE/vnstock acquisition edges, the shared Rust core, durable broker, + Python projector/query/stream roles, dedicated Redis and immutable + volumes/images. +3. Add the Python durable-broker-to-query/projector adapter. SQLite remains only + a bounded query cache; Kafka is the replay authority. +4. Package `qdl_sdk` V2 with a durable checkpoint adapter and V1 facade. Update + the service access guide to V2-first with a capability-truthful matrix. +5. Register and test at least one real alpha per Binance, OKX and VN, the + Trading System market-data adapter and monitoring consumer. Paper consumers + move one manifest at a time and retain V1 rollback. +6. Certify warmup, snapshot, stream, reconnect, replay, cursor restore, + freshness/gap/session blocking, Redis rebuild, process/broker restart and + bounded load on the new server. +7. Build immutable `2.0.0` artifacts and release evidence. Runtime cutover is a + separate operator action after exact topology/rollback approval. + +**Exit gate:** + +- V2 API/SDK contract and generated artifacts are frozen and CI-green; +- Binance, OKX and VN real consumer cycles pass with no direct venue connection; +- Rust is the selected canonical realtime core for approved slices and Python + remains the outer/vendor-acquisition edge; +- V1 rollback remains tested; +- V2 runtime health/readiness is dependency-derived and host-visible; +- production cutover remains blocked until the operator approves exact services, + ports, volumes, credentials and rollback. + +### Phase B Subphase Control Board + +Phase B is executed as five independently reviewable subphases. A defect found +inside one subphase is recorded as a bounded repair slice under that subphase; +it does not silently open a new architecture scope. No later subphase may be +declared complete while an earlier required gate remains open. + +| Subphase | Scope and exit gate | Current status | Frozen evidence/conclusion | +|---|---|---|---| +| `B.0 Contract And Stable Edge` | Catalog identity, V2 query/stream/projector contracts, consumer manifests, isolated topology and V1 compatibility | `COMPLETE` | B1-B4: 6, 32, 26 and 34 targeted tests passed; the one conditional Redis case was run separately against disposable Redis and passed. Public V1 and production runtime were unchanged. | +| `B.1 Runtime Correctness And Capacity` | Authentic acquisition, Rust canonical core, bounded projector/cache, final BAR lifecycle, lossless-vs-latest delivery and resource convergence | `COMPLETE` | B5-B8: final full Python discovery passed 478 tests with 6 explicit skips; Rust fmt/Clippy/workspace passed. A clean candidate loaded 2,000 authentic closed BARs, converged core/projector lag to 50/29, retained canonical-only bounded cache, zero quarantine and bounded Redis/app memory. Intermediate failed candidates are diagnostic evidence, not accepted releases. | +| `B.2 Controlled Consumer Acceptance` | Registered Binance, OKX, VN, Trading System and monitoring warmup -> signed cursor -> replay -> live, including session/freshness semantics | `PARTIAL_EXTERNAL` | B9-B12: crypto alpha, monitoring and Trading System paper consumers passed on immutable `df88de0`; 500 rows per crypto binding, replica-equal results and 129-779 ms live freshness. B.2-D now passes local contract/recovery closure, but DNSE production bootstrap remains blocked by official REST TCP/443 egress and the current host credential is rejected by WebSocket authentication. Synthetic or lineage-incomplete V1 data is not accepted as a substitute. | +| `B.3 Durability And Recovery` | Process generation, active/passive handoff, broker quorum loss, Redis/projection-cache rebuild, exact cursor continuity and fail-closed recovery | `COMPLETE` | B13-B19 plus the approved clean-log closure passed. Fresh RF3/minISR2 Kafka accepted real Binance/OKX data; atomic cache rebuild converged at lag 19 with observed bound 46; 12 retained partitions had zero gaps/duplicates/quarantine; signed SDK consumers reached `REPLAYING -> LIVE`; Trading System paper snapshots were fresh/execution-eligible; V1 was unchanged. Conclusion: `PASS`. | +| `B.4 Release Certification And Cleanup` | Full Python/Rust/Buf/OpenAPI/security/capacity suites, immutable one-SHA images, compact evidence, docs/runbook, exact candidate cleanup and V1 invariant | `COMPLETE` | Final code SHA `2412572` produced the non-root Python/Rust image pair; full source, contract, security, capacity, real-provider restart and signed consumer gates passed. Exact candidate resources were removed and V1 stayed unchanged. Conclusion: `PASS`; Phase B overall remains `PARTIAL_EXTERNAL` only for the pre-existing DNSE provider gate. | + +#### B.2-D DNSE Production Provider Closure (`PASS_LOCAL_EXTERNAL_GATE`) + +**Goal:** close the remaining Vietnam-market provider gap without weakening the +first-class Binance/OKX/VN contract. DNSE remains a Python vendor acquisition +edge, while every authenticated raw TRADE/BAR envelope continues through the +shared Rust canonical, quality and durable-publication core. + +**Approved references:** the provider-neutral rules in +`upgrade/quant-data-layer-fund-grade-upgrade-architecture.md` sections 4, 14.4, +Phase A item 5 and Phase B, with operator procedure in +`docs/runbooks/dnse-production-provider-edge.md`; plus the operator-supplied local DNSE OpenAPI SDK +snapshot under `dnse_provider/`. The snapshot is protocol reference only: it has +no discovered redistribution license and contains credential-bearing examples, +so no source, secret or generated artifact from that directory may be committed. + +**Scope and invariants:** + +1. Add a versioned, TLS-verifying DNSE REST history transport with explicit + proxy policy, bounded thread-safe provider quota, bounded retry/backoff with + `Retry-After`, strict response/pagination validation and secret-redacted + errors. The default API contract revision is `2026-07-23`, configurable by + `DNSE_API_VERSION`; insecure SDK `CERT_NONE` behavior is forbidden. +2. Keep REST exclusively for cold bootstrap and bounded gap repair. Use the + provider-native authenticated `ohlc_closed.1` WebSocket for live final BARs; + TRADE and BAR enter one bounded lossless queue and are acknowledged only + after Kafka durable publication. No synthetic/no-trade BAR is fabricated. +3. Persist an atomic, mode/catalog/acquisition/authority-bound DNSE BAR + watermark only after complete Kafka ACK. A matching complete checkpoint + avoids repeated cold REST bootstrap; unreadable, conflicting, partial or + stale-authority state fails closed. Reconnect may replay equal final BARs but + changed values at one timestamp are quarantined by the existing core. +4. Preserve VN identity, `Asia/Ho_Chi_Minh` calendar/session semantics, + derivative contract/share units and immutable `DNSE_DIRECT` lineage. Market + closure is healthy non-execution state, not provider outage. +5. V1 port `8100`, current Redis/Kafka namespaces, production authority and all + running consumers remain unchanged. No service restart, runtime cutover, + provider relabeling, production write or broad cleanup is approved here. + +**Test and acceptance gates:** + +- unit/contract tests for version/signature headers, TLS/proxy policy, + timeout/429/5xx retry, malformed JSON, unequal OHLC arrays, timestamp bounds, + duplicate/conflicting rows, non-monotonic pagination and quota concurrency; +- closed-BAR WebSocket mapping tests for exact decimal text, final 1m timestamps, + wrong symbol/resolution, queue saturation, batched durable ACK and ACK failure; +- restart/checkpoint tests for atomic persistence, exact restore, partial ACK, + corruption, authority/catalog/acquisition mismatch and duplicate replay; +- existing Python/Rust VN golden parity, calendar/session, stable-edge, + deployment and compatibility suites remain green; +- bounded real-provider smoke must use authentic DNSE bytes and zero public V1 + writes. If this host still cannot reach official REST TCP/443, implementation + can become `PASS_LOCAL_EXTERNAL_GATE`, but DNSE production bootstrap remains + fail-closed until either DNSE permits this host egress or an approved isolated + DNSE acquisition edge publishes authenticated raw envelopes over mTLS/ACL. + +**Implementation journal:** + +- `2026-08-20 DNSE REST TRANSPORT SLICE PASSED`: added a dedicated provider + wrapper with API revision `2026-07-23`, HMAC nonce/signature, mandatory TLS and + hostname verification, explicit opt-in environment proxy, thread-safe bounded + quota, redacted status errors, bounded timeout/retry/`Retry-After`, response + byte/page/row limits, strict parallel-array/OHLC/timestamp validation and + monotonic pagination with exact duplicate/conflict handling. The legacy + `_fetch_ohlc_raw` signature now delegates to this wrapper; its date chunk loop + no longer skips one day between exclusive boundaries. Eight isolated, + network-disabled unit cases passed. No service, provider, Redis, Kafka or V1 + runtime was contacted or changed. +- `2026-08-20 DNSE DURABLE CLOSED-BAR SLICE PASSED LOCALLY`: the stable VN edge + now uses REST only for cold history and native authenticated + `ohlc_closed.1` for live final 1m bars. TRADE and BAR share one bounded + lossless queue; queue pressure, malformed/future/wrong-resolution BARs and + missing durable ACK fence the source. A complete binding-wide watermark is + persisted by fsync plus atomic rename only after all Kafka acknowledgements, + and is bound to slice, authority, catalog and acquisition revisions. Matching + state skips repeated bootstrap; corrupt, partial or mismatched state fails + closed. REST and WebSocket captures keep distinct transport provenance while + resolving to one canonical BAR binding and semantic identity in Rust. +- Compatibility remained explicit: the public V1 fallback signature and + derivative-symbol export were preserved, its historical chunks are now + contiguous and it raises rather than returning partial rows after terminal + failure. Vendor SDK dispatch queues remain unbounded by default for V1; only + the isolated V2 edge opts into bounded backpressure. Catalog/acquisition/ + capability revisions and adapter version were advanced together, and the + compose topology mounts the existing stable-state volume without changing + public authority. +- Verification on the exact source passed 513 Python tests with 6 declared + skips; focused DNSE suites passed 28/28, the broader compatibility slice + passed 78 with one conditional skip, and Rust fmt, Clippy `-D warnings` and + the full workspace passed 63 tests with zero failures/skips. The new Rust + oracle proves REST history and a repeated native closed-BAR callback produce + one deterministic canonical final BAR. Final compose rendering passed with + isolated dummy secrets and no container start. One attempted targeted rerun + in the immutable runtime image stopped before test collection because release + images deliberately exclude pytest; it performed no provider or data write. +- Authentic read-only preflight remains externally blocked. At + `2026-08-20T05:25:35Z`, direct official REST TCP/443 timed out after 5.002 + seconds before TLS. WebSocket TCP/TLS succeeded, but the current host key was + rejected as invalid before subscription. The local official SDK snapshot and + tracked SDK use the same HMAC authentication protocol, so no insecure or + synthetic workaround was introduced. V1 remained running and unchanged, + reporting `MARKET_CLOSED`, a previously authenticated healthy session and + zero queue drops; restarting it before credential rotation is unsafe. +- Cleanup removed the disposable Rust test image and temporary pytest data. + Release/rollback images, V1, production data, Redis/Kafka and volumes were + retained. Shared BuildKit cache was not broadly pruned because the host is + shared; exact disposable runtime artifacts are gone. Conclusion: + `PASS_LOCAL_EXTERNAL_GATE`, not DNSE production-authoritative. Promotion + requires a valid dedicated key plus official REST reachability, or an + operator-approved isolated egress edge, followed by the bounded 500-row, + closed-BAR, restart/checkpoint and signed-consumer gates in the runbook. + Implementation commits: `583ae82` (strict versioned history) and `5d37976` + (durable closed-BAR acquisition, checkpoint, compatibility and runbook). + +**Rollback:** revert only this bounded provider/edge commit and use the retained +`2.0.0-2412572` artifacts. Existing V1 remains the authority throughout. The +decision to deploy an egress-capable DNSE edge or promote the revised V2 slice +requires a separate operator-approved topology and blast radius. + +Every subphase closure records: approved boundary, invariant, exact commands and +pass/fail/skip counts, real-provider or test provenance, resource/latency data, +runtime mutations, cleanup, V1 impact, commit SHA, remaining external gate and +one explicit conclusion (`PASS`, `FAIL`, `PARTIAL_EXTERNAL` or `NOT_STARTED`). +A health endpoint or process-up result alone cannot close a subphase. + +**Phase B artifact hygiene:** after each coherent tested slice, remove its +disposable containers/networks and exact unreferenced image tags. Retain only +the running V1 artifacts, the active isolated candidate and one explicitly +named rollback generation. Build cache is cleaned at subphase boundaries using +the Data Layer builder/cache scope; broad host-wide prune is forbidden because +other repositories share this host. Every cleanup records the exact removed +objects and reclaimed bytes. B.4 performs the final one-SHA image replacement +and removes the superseded candidate/rollback artifacts after verification. + +### Implementation Journal + +- `2026-08-19 PLAN COMPLETE`: the original two-phase scope was frozen on + `feat/v2-stable-rust-binance-okx`. Official OKX V5 docs were rechecked: + public/business sockets are separate, connections with no subscription/data + over 30 seconds need ping/pong/reconnect, control requests are bounded to 480 + per connection/hour, `trades-all` is atomic, `bbo-tbt` is replace-only, and + stateful books rely on `seqId/prevSeqId`. +- `2026-08-19 SCOPE REFINEMENT APPROVED`: Phase A includes Binance, OKX and VN + as equal first-class sources. High-volume crypto transport is native Rust; + DNSE/vnstock may remain Python acquisition edges but must publish authenticated + raw envelopes into the same Rust canonical/quality/durable core. Feed support + is capability-truthful, not fabricated for visual symmetry. +- `2026-08-19 DOMAIN CORRECTION`: the current DNSE runtime subscribes native + trades even though its legacy Redis channel is named `stream:vn:*` and often + described as a quote. V2 records these as TRADE. DNSE/vnstock QUOTE is a + separate capability and cannot be advertised until a real bid/ask feed is + certified; legacy naming does not define the canonical domain. +- `2026-08-19 PHASE A SLICE A1 COMPLETE`: froze additive quantity-unit and + trade-identity semantics before V2 stable release. TRADE/QUOTE/BAR/BOOK/OI/ + TICKER quantities now carry explicit base/quote/contract/share units; BAR can + preserve native, base, quote and contract volume simultaneously. DNSE trades + without native IDs use `DERIVED_RAW_CAPTURE`, unknown aggressor side and + explicit missing-field/source-time quality flags. Canonical source role is no + longer hardcoded primary, so vnstock secondary provenance remains visible. +- `2026-08-19 SLICE A1 EVIDENCE`: Buf format/lint and breaking checks passed + against both Phase 1 and Phase 7 beta baselines. The OpenAPI semantic diff + passed with 10 operations, 42 schemas and zero hard break. Seventeen + Binance-USD-M/Spot, OKX-SWAP/Spot, DNSE-derivative/equity and vnstock-equity + fixtures match exact Python/Rust golden bytes; 56 cross-phase Python tests, + 9 frozen-contract tests and the full 43-test Rust workspace passed with + Clippy warnings denied. V1 runtime/topology was not changed. +- `2026-08-19 PHASE A SLICE A2 COMPLETE`: added the provider-neutral + `qdl-realtime-core`, native Rust Binance/OKX raw ingestion, Python DNSE/vnstock + and Binance REST BAR raw edges, fenced Kafka raw publication, atomic Kafka + consume-transform-produce transactions and separate canonical/quarantine + topics. Rust is now the only canonical realtime semantics core; Python is the + external/vendor-acquisition edge. +- `2026-08-19 PROVIDER DEFECT CLOSED`: Binance Kline WebSocket produced zero + frames through legacy combined, official public combined and official public + raw probes, matching the running V1 zero-message telemetry. Binance BAR now + uses the existing low-rate REST history boundary to fetch the latest closed + native row, then passes through Rust. It is never fabricated or silently + relabeled. Binance Spot BBO missing provider time uses receive time with an + explicit `SOURCE_TIME_MISSING` flag. +- `2026-08-19 PHASE A FINAL EVIDENCE`: real-provider certification committed 26 + raw -> 26 canonical records with zero quarantine across Binance USDM/Spot, + OKX SWAP/Spot and VN equity/derivative products. Transactional certification + committed under one Kafka replica loss, suppressed one duplicate and routed + one intentional sequence gap to quarantine with exact read-committed counts. + The release core processed 100,000 events at approximately 154,139 events/s + with p99 about 10.8 microseconds and zero loss/duplicate/quarantine. Full + Python regression passed 433 tests with 5 conditional skips; Rust fmt/Clippy + and all 50 tests passed, including atomic rollback for partially invalid + multi-row provider frames; Buf format/lint/breaking and code generation passed. + Every disposable container/network/volume/image was removed; final scoped + cleanup also removed 2.8 GiB of Cargo target artifacts and the 2.91 GB + `qdl-v2-rust-builder:test` image. V1 health/topology remained unchanged. See + [Phase A report](upgrade/evidence/PHASE_A_RUST_MULTIVENUE_CORE_REPORT.md). +- `2026-08-19 PHASE B STARTED`: operator approved isolated stable + implementation, consumer-manifest migration and immutable `2.0.0` packaging. + Production authority/cutover was not approved. Phase B follows the execution + boundary and acceptance gates above; every tested slice is journaled here. +- `2026-08-19 PHASE B SLICE B1 COMPLETE`: introduced the strict stable source + catalog and provider-neutral TRADE/QUOTE/BAR query backend. Instrument + metadata is declared once and feed bindings reference deterministic UIDs, + preventing per-feed metadata drift. Stable admission now requires exact raw + capture lineage, source session/generation, monotonic receive/normalize/ + publish timestamps, quantity units, trade identity, source/adapter/ + normalizer/authority revisions and final/revised BAR lifecycle where required. + The catalog baseline contains 16 bindings across Binance USD-M/Spot, OKX + SWAP/Spot, HNX VN derivatives and HOSE equity with truthful feed capability. +- `2026-08-19 PHASE B IDENTITY DEFECT CLOSED`: Phase A golden fixtures retained + older arbitrary crypto UIDs and therefore cannot define the stable registry. + Stable B1 keeps the Phase 1 deterministic `InstrumentIdentity` authority and + refuses those legacy IDs. New Phase B provider evidence must be captured with + the stable catalog identities before release; validation was not weakened. + VN30F1M is modeled explicitly as a continuous FUTURE series with roll-policy + metadata, while dated futures still fail closed without an expiry. +- `2026-08-19 PHASE B SLICE B1 VERIFICATION`: six isolated, network-disabled + tests passed for deterministic identity, unknown-field/provenance rejection, + continuous-vs-dated future semantics, exact TRADE/QUOTE/BAR units, crypto gap + blocking, sparse VN/market-closed behavior and consumer-bound signed replay + cursors. The test found and closed a query defect where `latest()` could miss + a gap immediately before its one-row window; latest/history now evaluate gap + state across all retained cache rows. No runtime container, Redis namespace, + consumer or production data was changed; the disposable test container was + removed by `--rm`. +- `2026-08-19 PHASE B SLICE B2 COMPLETE`: implemented the Kafka-authoritative + Python stable edge around the Rust core. The read-committed consumer disables + auto commit/store; the projector durably stores raw envelopes, holds bounded + canonical-before-raw records per broker partition, commits canonical data to + the active fenced stream gateway/shared cache, atomically projects dedicated + Redis latest/V1 compatibility state and Pub/Sub, and only then advances the + Kafka checkpoint. Assignment changes discard only local uncheckpointed queues + for broker replay. Stale projection epochs raise a hard fence rather than + masquerading as an idempotent duplicate. +- `2026-08-19 PHASE B COMPATIBILITY DECISION`: V1 Redis aliases are an explicit + per-binding policy. Binance USD-M owns the generic `BTCUSDT` trade/bar aliases; + Binance Spot gets market-qualified trade aliases only, so the stable candidate + cannot recreate the former mixed-writer race. OKX has no fabricated V1 Redis + alias and migrates through V2. DNSE TRADE may project the frozen `vn:quote:*` + compatibility shape only in the dedicated stable Redis; canonical semantics + remain TRADE. Current production Redis is never addressed. +- `2026-08-19 PHASE B SLICE B2 VERIFICATION`: 32 isolated tests passed with one + conditional Redis test skipped in the network-disabled run, covering B1/B2, + V2 API compatibility and beta-runtime regression. A separately named, + disposable Redis 7.2 container passed the real Lua/TTL/PubSub/idempotency/ + fencing test; exact test keys were removed, then the container and network + were deleted. Failure injection passed canonical-before-raw, Kafka checkpoint + failure/replay, Redis loss/rebuild, source alias isolation, HMAC rejection, + duplicate HTTP ingest and stale-writer fencing. No test used provider data as + generated production evidence and no live V1 process/state was mutated. +- `2026-08-19 PHASE B SLICE B3 STARTED`: freeze the real-consumer migration + registry and public package metadata for `2.0.0`. The bounded slice covers + monitoring, one Binance paper alpha, one OKX paper alpha, one VN paper alpha + and the Trading System paper adapter. Every manifest must resolve only + catalogued instrument/feed pairs, use SDK major 2, retain an explicit V1 + rollback contract and remain `SHADOW` with `cutover_authorized=false`. + Package, SDK and generated OpenAPI versions must agree exactly; V1 OpenAPI, + SDK and Redis compatibility goldens must remain byte/semantic compatible. + This slice does not deploy a runtime, migrate a consumer, change authority, + restart port `8100` or write any current Redis namespace. +- `2026-08-19 PHASE B SLICE B3 COMPLETE`: added a strict migration-plan + loader and five governed paper manifests for monitoring, Binance alpha, OKX + alpha, VN alpha and Trading System. Every data requirement resolves to the + deterministic stable catalog; only Trading System may declare + `PAPER_ONLY` execution dependency and all other consumers remain + `FORBIDDEN`. Unknown fields, unknown instrument/feed bindings, active + routes, weak rollback policy and implicit cutover fail closed. Package, + `qdl_sdk` and generated OpenAPI metadata now agree on `2.0.0`. +- `2026-08-19 PHASE B SLICE B3 VERIFICATION`: 26 isolated contract, + consumer-state, stable-edge and V1/V2 golden tests passed; one real-Redis + integration case was intentionally skipped in this network-disabled suite + because the same Lua/TTL/PubSub/fencing path passed against disposable Redis + in Slice B2. The source was mounted read-only with a tmpfs log path. No V1 + service, authority, consumer route, Redis state or provider connection was + changed, and no disposable file/container remained. +- `2026-08-19 PHASE B SLICE B4 STARTED`: build the isolated stable deployment + manifest and immutable runtime configuration around the committed B1-B3 + contracts. Broker topology reuses the certified RF3/minISR2 mTLS/ACL + substrate; Rust native acquisition and transactional canonical core consume + deterministic stable catalog identities; Python query, active/passive stream + and projector remain provider-free outer roles. Native channel mapping is a + separate, strictly validated acquisition contract and cannot redefine + instrument identity or canonical semantics. All names, ports, topics, + credentials, volumes, Redis prefixes and consumer groups are isolated. + Readiness must expose Kafka, dedicated Redis, cache, lease, catalog and + manifest dependencies. Deployment/certification may create only disposable + `qdl_v2_stable_candidate` resources; current V1 port `8100`, Redis and + consumers remain outside the topology and production cutover is not approved. +- `2026-08-19 PHASE B SLICE B4 COMPLETE`: added the self-contained isolated + stable topology, deterministic acquisition-to-core deployment contract, + immutable image-ID candidate bundle generator and idempotent RF3/minISR2 + TLS/ACL broker bootstrap. Sixteen catalog bindings map one-to-one across + Rust-native Binance/OKX, Python latest-closed Binance BAR and Python DNSE + SDK/REST acquisition; only Rust creates canonical semantics. Python roles are + non-root/read-only/bounded, state volume ownership has an isolated preflight, + query has two replicas, stream is active/passive, projector health is + dependency-derived and V1/current Redis names are absent. +- `2026-08-19 PHASE B B4 DEFECTS CLOSED`: stable Redis quota namespaces were + incorrectly rejected by the beta-only prefix guard and are now admitted only + under explicit `qdl:stable:v2:` isolation. Multi-replica projector recovery + could also stall when one replica persisted raw while another held canonical; + bounded idle polling now rechecks the shared cache without advancing Kafka + before downstream ACK. DNSE acquisition never evicts queue entries: pressure + or Kafka ACK failure fences the source for supervised recovery. +- `2026-08-19 PHASE B SLICE B4 VERIFICATION`: 34 isolated acquisition, + identity, authority, topology, failure, V1/V2 edge and release contract tests + passed with one separately-certified Redis integration skip. Docker Compose + rendered successfully with required secrets supplied only as disposable + validation values. Tests covered one-to-one source mapping, deterministic + bundle hashes, no secret disclosure, closed-BAR dedup, zero-volume VN BAR, + queue-pressure fencing, cross-replica wake-up and rejection of primary/public + authority. No runtime was deployed and no disposable resource remained. +- `2026-08-19 PHASE B SLICE B5 STARTED`: certify the committed `467bbf3` + candidate with immutable Python/Rust image IDs on the migrated 8-core host. + The isolated run must exercise RF3/minISR2 broker policy, transactional Rust + raw-to-canonical processing, Python projector/cache/dedicated Redis, active/ + passive stream handoff, both query replicas and all five SHADOW manifests. + Real-provider records or replay of Phase A durable provider captures are + mandatory for market-data evidence; generated fixtures are restricted to + failure/capacity injection. Broker loss, Redis rebuild, process restart, + cursor recovery and rollback cleanup must leave V1 topology/health unchanged. + Any runtime defect is fixed and retested before release evidence; cutover + remains explicitly unauthorized. +- `2026-08-19 PHASE B B5 RUNTIME DEFECT SLICE VERIFIED`: the first isolated + immutable-candidate boot exposed three fail-closed deployment defects before + any V1 or current-Redis write. Kafka brokers mounted `/tmp` with `noexec`, so + Zstd JNI could not load and raw producers received opaque `UNKNOWN` delivery + failures; only Kafka now receives a bounded `exec,nosuid,nodev` tmpfs. The + projector still mounted its old host identity path while its contract pointed + at the role-scoped `stable_tls` volume; all runtime identities now use the + root-copied, UID/GID `10001`, mode `0440`, read-only role volume. Finally, the + internal sink allowlist knew the retired single `stream_v2` name but not the + fixed active/passive roles; it now admits exactly `stream_v2_active` and + `stream_v2_passive` while continuing to reject HTTPS, redirects and external + hosts. Targeted deployment/edge verification passed 22 tests with one + separately gated disposable-Redis integration skip. A new immutable candidate + must be built from the committed fix SHA before runtime certification resumes. +- `2026-08-19 PHASE B B5 FINAL-BAR DOMAIN DEFECT CLOSED`: real OKX + `candle1m` traffic carries both provisional `confirm=0` and closed `confirm=1` + rows. The catalog already required final bars, but this policy was not included + in the generated Rust binding, so provisional canonical output reached and + correctly failed the Python projector gate. `require_final_bar` is now carried + catalog-to-core; Rust transactionally consumes raw provisional updates, counts + them as `filtered`, publishes no canonical/public event and does not mislabel + expected venue lifecycle as quarantine. Only final/revised BARs proceed; a + final-only policy attached to a non-BAR payload remains quarantined. Rust fmt + and warnings-as-errors Clippy passed, the full Rust workspace passed 51 tests, + and 22/22 targeted Python deployment/edge tests passed with one separately + gated disposable-Redis integration skip. Runtime retest requires images built + from the next committed SHA. +- `2026-08-19 PHASE B B5 INGRESS DEFECT CLOSED`: application readiness was + `READY` container-locally, but services attached only to Docker + `stable_internal`; Docker retained declared bindings without creating host + forwarding. The approved Phase 7 separation is restored: query and active/ + passive stream roles join a dedicated `stable_ingress` bridge and publish only + on `127.0.0.1`; Kafka, Redis, Rust core and projector remain unexposed, with + projector internal-only. Provider roles alone retain `stable_egress`. Targeted + topology/deployment/edge verification again passed 22 tests with the one + separately gated Redis integration skip. +- `2026-08-19 PHASE B B5 ACQUISITION RELIABILITY DEFECT FIXED, RUNTIME + RETEST PENDING`: authenticated V2 smoke correctly returned `DATA_STALE` for + Binance USD-M after the Rust provider process had exited on a real WebSocket + `Connection reset by peer`. Native Binance/OKX sessions now reconnect with + capped exponential jittered backoff, reset failure streak only after durable + raw ACK, and a top-level supervisor restarts transient decode/subscription/ + transport failures without weakening configuration or authority fail-closed + gates. OKX public/business futures are fail-fast joined so one dead channel + cannot be masked by the other. Every raw Kafka append retains identical + provider bytes, capture/event identity and receive time across retry; retryable + and capacity failures backpressure indefinitely until ACK or stop, while + fencing/configuration failures remain non-retryable. Bounded-test reservations + are released on non-retryable append failure. Rust fmt, warnings-as-errors + Clippy and all 51 workspace tests passed. A forced disconnect/reconnect smoke + on an immutable image is still required before this defect gate closes. +- `2026-08-19 PHASE B B5 PROJECTOR THROUGHPUT DEFECT IDENTIFIED, FIX IN + PROGRESS`: the immutable `5f32238` candidate keeps all four native providers, + the transactional Rust core and the Python projector alive, and both raw and + canonical Kafka offsets advance from real provider traffic. Authenticated + Binance USD-M V2 snapshot nevertheless fails closed as `DATA_STALE` because + `stable-projector-v1` accumulates thousands of records of lag. The root cause + is one synchronous Kafka commit round-trip after every individually durable + raw and canonical ACK. The correctness invariant remains unchanged: no Kafka + offset may be eligible for commit before local raw durability or complete + canonical stream/cache/Redis projection. The bounded fix is confined to the + Confluent adapter: coalesce only already-ACKed offsets per topic-partition, + submit asynchronous commits at a bounded count/time interval, surface commit + callback errors fail-closed, discard uncommitted batches on rebalance for + idempotent replay, and synchronously flush on orderly close. Unit failure + gates and a real-provider lag-delta retest are required before this slice can + close. V1, current Redis, public contracts and authority remain untouched. +- `2026-08-19 PHASE B B5 PROJECTOR CHECKPOINT FIX UNIT-VERIFIED, RUNTIME + RETEST PENDING`: the Confluent adapter now batches at most 128 downstream-ACKed + offsets or 100 milliseconds per topic-partition. Async callback failures stop + subsequent poll/checkpoint operations, assignment changes discard only the + uncommitted local batch for idempotent broker replay, and orderly close uses a + synchronous final flush. The engine still calls `checkpoint` only after raw + spool durability or complete canonical stream/cache/Redis projection. Thirty + deployment, stable-edge and release tests passed with one separately gated + real-Redis integration skip, including no premature checkpoint, coalesced + offsets, rebalance replay and callback fail-closed. An immutable Python image + plus real-provider lag-delta smoke remains required before closure. +- `2026-08-19 PHASE B B5 TRANSACTIONAL CORE RECOVERY DEFECT IDENTIFIED, FIX + IN PROGRESS`: a deliberately isolated broker-loss gate exposed a second + recovery boundary after the projector throughput fix. RF3/minISR2 preserved + the topics and Rust acquisition continued appending raw provider bytes, but + the transactional core stopped advancing canonical/quarantine offsets after + its transaction coordinator connection was interrupted. Public V2 correctly + returned `DATA_STALE`; there was no fabricated freshness, data loss or V1 + effect. The accidental Kafka1 exit code 137 was caused by running a separate + Java admin CLI inside the broker 512 MiB cgroup, not by broker steady-state + usage; all subsequent admin probes must use the isolated `stable_admin` role. + The bounded runtime fix must reconstruct the complete transactional bridge and + in-memory normalization state under the same authority/transactional identity + only for retryable Kafka transport errors, with capped jittered backoff and + explicit generation telemetry. Configuration, fencing, schema, decode and + domain errors remain non-retryable. Kafka exactly-once offsets then replay + only the uncommitted batch. A real one-broker loss/rejoin gate must prove raw, + canonical and projector offsets resume and authenticated snapshots return + `LIVE` before closure. +- `2026-08-19 PHASE B B5 TRANSACTIONAL CORE SUPERVISOR UNIT-VERIFIED, RUNTIME + RETEST PENDING`: `qdl-realtime-core` now owns a generation supervisor around + the complete transactional bridge and normalization state. Retryable/capacity + Kafka errors rebuild the producer, consumer and core under the unchanged + authority record and transactional ID after capped jittered backoff; Kafka + transaction recovery resolves any indeterminate batch before the consumer + resumes from committed offsets. Non-retryable transport and all non-Kafka + domain/schema/decode failures still terminate. Generation is emitted in + bounded progress telemetry. The pinned Rust build, rustfmt, Clippy with all + warnings denied and the retry-class unit test passed. Immutable-image broker + loss/rejoin evidence remains the closure gate. +- `2026-08-19 PHASE B B5 TRANSACTION OUTPUT THROUGHPUT DEFECT IDENTIFIED, FIX + IN PROGRESS`: after transaction-state recovery, the shared core resumed from + the committed watermark but could not catch live raw traffic. Measurement + showed the cause inside the transaction boundary: a batch of up to 256 + canonical/quarantine outputs was delivered with one awaited Kafka network + future at a time. The fix retains the same bounded batch, idempotent producer, + per-partition order, output validation, atomic offset commit and all-or-nothing + abort, but enqueues the batch delivery futures concurrently before awaiting + them. No new dependency or public behavior is introduced. Atomic rollback, + ordering, duplicate and real-provider lag-delta gates must pass before close. +- `2026-08-19 PHASE B B5 CONCURRENT TRANSACTION DELIVERY UNIT-VERIFIED, + RUNTIME RETEST PENDING`: the Rust bridge now polls all bounded delivery futures + for one transaction concurrently, while Kafka idempotence and max in-flight + preserve partition ordering and the existing transaction still atomically + commits output plus source offsets or aborts the whole batch. Pinned release + compile, rustfmt, all-target Clippy with warnings denied and all six qdl-kafka + tests passed. No dependency, topic, schema or authority changed. A new + immutable image must prove catch-up under real provider traffic and repeat + broker-loss recovery before this gate closes. +- `2026-08-19 PHASE B B5 OKX BBO DOMAIN DEFECT IDENTIFIED, FIX IN PROGRESS`: + bounded quarantine inspection found repeated `SEQUENCE_GAP` only on OKX + `bbo-tbt`. The approved OKX V5 guide defines this channel as a replace-only + best-bid/offer snapshot and explicitly forbids requiring `prevSeqId`; strict + continuity belongs to incremental book channels. Stable acquisition had + incorrectly assigned `CONTIGUOUS`, effectively requiring `seqId + 1` and + creating false gaps. Both OKX SWAP and Spot BBO bindings will use provider- + neutral partition/receive ordering (`NONE`) while retaining native `seqId` in + canonical provenance and deterministic event identity. Incremental book + policies and generic contiguous-order tests remain strict. Real OKX BBO must + then show canonical output without sequence-gap quarantine. +- `2026-08-19 PHASE B B5 OKX BBO POLICY UNIT-VERIFIED, RUNTIME RETEST + PENDING`: both stable OKX SWAP/Spot `bbo-tbt` bindings now use replace-only + partition ordering, and acquisition validation rejects any future attempt to + attach `MONOTONIC` or `CONTIGUOUS` native-sequence policy to `okx_bbo`. + Generic contiguous-order behavior remains unchanged for incremental channels. + Thirty deployment, stable-edge and release tests passed with one separately + gated real-Redis integration skip. Real-provider quarantine/canonical deltas + remain required after rebuilding the runtime bundle. +- `2026-08-19 PHASE B B5 PROJECTOR DURABILITY THROUGHPUT DEFECT IDENTIFIED, + FIX IN PROGRESS`: the clean immutable `3dc58cf` candidate proves real Binance + and OKX snapshots with correct canonical provenance and zero quarantine, but + `stable-projector-v1` lag is not bounded under the subscribed real-provider + burst. Measurement shows the per-event path performs one fsync-backed raw + cache transaction, one fsync-backed canonical cache transaction through the + active stream gateway, one signed HTTP request and one Redis Lua projection + before checkpoint eligibility. Kafka remains the replay authority and SQLite + durability must not be weakened. The bounded fix therefore batches existing + domain operations rather than changing semantics: poll a small time/count + window, append raw and canonical records with `append_many`, preserve order + per partition, send one signed internal ingest batch, project atomically per + event, and coalesce only fully downstream-ACKed Kafka offsets. Any partial + sink/projection/checkpoint failure leaves the batch replayable and idempotent; + stale leases, lineage mismatch, collision, capacity and unknown schema still + fail closed. Unit ordering/failure/replay tests plus real-provider lag, + quarantine, resource and recovery gates are mandatory before closure. +- `2026-08-19 PHASE B B5 PROJECTOR DURABLE MICRO-BATCH UNIT-VERIFIED, + RUNTIME RETEST PENDING`: the projector now polls at most 128 records over a + 25 ms bounded window, persists raw records with one `append_many` transaction, + joins canonical records without changing per-partition order, sends one + signed internal ingest batch to the fenced active gateway, persists canonical + records with one fsync-backed transaction and pipelines the unchanged atomic + Redis Lua projection. Kafka checkpoints remain after the complete downstream + ACK chain. Assignment changes discard only uncheckpointed memory; collision, + stale lease, unknown lineage and capacity gates remain fail-closed. Existing + single-event methods delegate to the batch path, preserving callers. Focused + tests prove two raw plus two canonical records use exactly two durable writes, + one projection batch and ordered raw-before-canonical checkpoints; injected + batch projection failure produces no canonical checkpoint and succeeds on + idempotent replay. Thirty-two Phase B edge/deployment/release tests passed with + one separately gated real-Redis integration skip; Python compile and + `git diff --check` passed. Immutable-image real-provider lag/I/O, Redis, + failover and broker recovery evidence remains mandatory. +- `2026-08-19 PHASE B B5 RUST CORE CAPACITY DEFECT IDENTIFIED, FIX IN + PROGRESS`: after projector batching, real raw ingress measured about 392 + events/second while one ordered transactional core sustained about 150 + events/second. Its final batches were near the 256-event bound and the process + saturated one CPU; raising its cgroup from 0.75 to 1.5 CPUs did not remove lag, + proving a single-loop topology bottleneck rather than insufficient host + headroom. Increasing batch size enough to hide the gap would add seconds of + market-data latency and is rejected. The stable topology will instead run + three explicit Rust core workers over the six Kafka partitions. Workers share + only the consumer group and authority, while each has a deterministic client, + shard and transactional ID so Kafka preserves one owner per partition and + exactly-once output/source-offset transactions without fencing peers. Runtime + bundle and Compose tests must reject duplicate identities or a worker count + exceeding topic partitions. Real traffic must show bounded/decreasing total + group lag, zero quarantine and no duplicate canonical/public writes before + this gate closes. +- `2026-08-19 PHASE B B5 THREE-CORE TOPOLOGY UNIT-VERIFIED, RUNTIME RETEST + PENDING`: the stable bundle now emits three explicit core configs over six + Kafka partitions. Worker `001` preserves `core.json`; workers `002/003` use + dedicated files. All share `qdl-v2-stable-core-v1` but have unique static + client, shard and transactional IDs, the same pinned authority and identical + provider-neutral normalization bindings. Compose defines three non-root, + read-only, bounded Rust services without public ports. Contract tests reject + worker indices outside the topology and verify identity uniqueness, common + group/authority and worker-count-to-partition bounds. The rendered Compose is + valid and 32 Phase B deployment/edge/release tests passed with one separately + gated real-Redis integration skip. An immutable runtime must still prove lag + convergence, exactly-once broker recovery and no quarantine/duplicates. +- `2026-08-19 PHASE B B5 RUST CORE HOT-PATH DEFECT IDENTIFIED, FIX IN + PROGRESS`: partition-level evidence showed one hot ordered raw partition can + still miss the 2-3 second execution freshness policy. Code inspection found + an algorithmic defect: every raw event clones the complete global dedup set, + dedup eviction queue, ordering tracker and partition-sequence map solely to + roll back a multi-row provider frame. With a one-million-event dedup bound, + processing cost grows with history and eventually becomes quadratic. The fix + will stage only mutations for the current provider frame: scalar partition + sequence, per-partition ordering transition and new event IDs. Those stages + commit only after every row passes canonicalization/finality/ordering; a row + failure discards them and emits the same atomic quarantine. Existing global + dedup, bounded eviction, session reset, duplicate, monotonic/contiguous gap + and stale-generation semantics must remain byte/domain equivalent. Rust unit, + atomic multi-row rollback, parity, warnings-as-errors, benchmark and real hot- + partition freshness gates are required before closure. +- `2026-08-19 PHASE B B5 RUST CORE STAGED HOT PATH UNIT/CAPACITY VERIFIED, + IMMUTABLE RUNTIME RETEST PENDING`: realtime-core no longer clones global + partition sequences, ordering partitions or the bounded dedup set/queue for + every provider event. It stages only current-frame partition sequences, + ordering transitions and new event IDs, then commits after every expanded row + passes; a failed row discards the stage and preserves atomic quarantine/replay + behavior. Two direct staged-ordering tests prove discarded state is invisible + and committed batches preserve duplicate/gap semantics. All 30 targeted + `qdl-venue-core`/`qdl-realtime-core` tests passed, including aggregated-frame + rollback, reconnect dedup, stale generation, sequence-gap, final-bar and + provider identity cases. The first unchanged 100,000-event/50,000 events/s + benchmark gate correctly failed at 6,177 events/s and exposed one residual + clone of the partition recent-ID set inside `stage()`. Removing that clone, + without lowering the gate, produced 100,000 canonical events, zero duplicate, + zero quarantine, 115,107 events/s, p50 6.19 microseconds and p99 18.921 + microseconds. Targeted Clippy, full workspace tests and an immutable three-core + real-provider freshness/recovery run remain mandatory before this gate closes. +- `2026-08-19 PHASE B B5 STABLE QUERY TAIL DEFECT IDENTIFIED, FIX IN + PROGRESS`: the clean `c177bb0` three-core candidate drained core lag to 11 + records and projector lag to 57 records with zero quarantine, and authenticated + Binance/OKX monitoring, alpha trade/bar and OKX execution snapshots passed. + Binance execution QUOTE nevertheless failed strict 2-second freshness 20/20. + Its Redis canonical envelope was current (receive age about 148 ms), while the + query backend reported about 230 seconds stale. The shared spool held more than + 36,000 events for that partition; `StableSpoolQueryBackend` called the replay + API `read(limit=10000)`, whose intentionally ascending semantics return the + oldest retained page, then treated the page tail as latest. The additive fix + must introduce an ordered `read_tail` primitive for latest-state/query use and + leave cursor/replay `read` semantics untouched. Transport ordering, tail bound, + stable query beyond 10,000 records, strict authenticated freshness and replay + regression tests are mandatory before runtime recovery certification resumes. +- `2026-08-19 PHASE B B5 STABLE QUERY TAIL UNIT-VERIFIED, IMMUTABLE + RUNTIME RETEST PENDING`: `SQLiteDurableSpool.read_tail` is an additive bounded + latest-window primitive that selects newest offsets efficiently and returns + them in ascending logical order. Existing `read(after=...)` replay/cursor + semantics are unchanged. Stable latest/history/status/gap views now use the + tail window. Regression tests prove replay limit two still returns offsets + 1-2 while tail limit two returns 4-5, and a real SQLite partition with 10,001 + canonical trade records returns offset 10,001 as LIVE instead of the end of + the oldest page. The two transport/stable-edge suites passed 32 tests with one + separately gated real-Redis skip. The immutable candidate must still prove the + Binance execution QUOTE 2-second policy, lag, replay and failure recovery. +- `2026-08-19 PHASE B B5 QUERY TAIL CORRECTNESS PASSED, READ + AMPLIFICATION FIX IN PROGRESS`: on the immutable `f120173` candidate, the + Binance USDM quote canonical partition exceeded 10,000 records and strict + 2-second authenticated execution snapshots passed 20/20, proving newest-tail + correctness. But latest/status latency remained 729-1,115 ms (813 ms average) + because every request decoded the full 10,000-row tail. Feed-aware bounds will + read one event for TRADE/QUOTE latest/status, retain the bounded BAR continuity + window needed for missing-candle detection, and read only the requested history + window for warmup/replay snapshot creation. Public contracts, durability, gap + policy and replay cursor semantics remain unchanged. Unit call-bound tests and + immutable post-10k latency/freshness evidence are required. +- `2026-08-19 PHASE B B5 QUERY READ AMPLIFICATION UNIT-VERIFIED, + IMMUTABLE LATENCY RETEST PENDING`: stable latest/status now reads one newest + event for TRADE/QUOTE; BAR latest retains the 10,000-event continuity window, + and history/snapshot reads exactly the requested warmup bound. A call-bound + test locks limits `1, 1, 10000` for trade latest, one-row history and bar latest + respectively. The transport/stable-edge suites now pass 33 tests with one + separately gated real-Redis skip, including the 10,001-record newest-tail + regression. The public API, freshness thresholds, gap policy and replay reads + were not changed. Immutable post-10k p95/p99 and strict snapshot evidence still + gates acceptance. +- `2026-08-19 PHASE B B5 POST-10K QUERY LATENCY VERIFIED, CLEAN + CANDIDATE/RECOVERY GATES PENDING`: two isolated query replicas were rolling- + recreated with immutable `d08c424` while preserving the `f120173` test spool + and credentials solely to measure the >10,000-record boundary. After a brief + real-provider catch-up interval correctly failed closed because the source + timestamp was about 16 seconds old, the feed recovered without mutation. One + hundred consecutive Binance execution QUOTE requests then passed the unchanged + 2-second policy: p50 2.94 ms, p95 3.74 ms, p99 32.44 ms, max 89.8 ms and 4.31 + ms average, versus 813 ms average before feed-aware bounds. This mixed rolling + rehearsal is not final certification; a clean all-`d08c424` candidate and + broker/provider/Redis/process recovery gates remain mandatory. +- `2026-08-19 PHASE B B5 TWO-BROKER FAIL-CLOSED PASSED, PROJECTOR + RECOVERY DEFECT IDENTIFIED`: with one isolated RF3 broker stopped, the durable + spool advanced from 5,063 to 12,423 events and quarantine stayed zero. With two + brokers stopped under minISR=2, queued work drained and the spool then remained + exactly constant at 27,403 for five seconds, proving no false acceptance. After + both brokers returned healthy, Rust producers/cores remained alive but the + Python projector did not recover because an asynchronous checkpoint error + terminated its process and Compose intentionally had `restart: no`. The fix is + a bounded in-process projector supervisor: recreate the poisoned Kafka consumer, + keep readiness NOT_READY while disconnected, retain the durable spool/Redis + target, replay only uncheckpointed Kafka records through idempotent sinks, and + use capped exponential backoff. Closing a poisoned generation must be logged but + must not mask the original error or prevent a new generation. Unit retry/close + tests and the same real two-broker outage/recovery gate are mandatory. +- `2026-08-19 PHASE B B5 PROJECTOR SUPERVISOR UNIT-VERIFIED, REAL + OUTAGE RETEST PENDING`: the stable projector now recreates a failed Kafka + consumer/engine generation in process with capped 250 ms-to-5 s exponential + backoff. Readiness reports Kafka NOT_READY while no generation is active. Shared + durable spool, HTTP sink and Redis target survive generation replacement, so + only uncheckpointed broker records replay through existing idempotency gates. A + poisoned generation close is bounded/logged and cannot mask the original fault + or prevent recovery; cancellation still propagates. Stable edge/release suites + passed 28 tests with one separately gated real-Redis skip, including injected + async-checkpoint failure, poisoned close, exact one-backoff retry and successful + second generation. The real two-broker recovery test must now be repeated. +- `2026-08-19 PHASE B B5 TWO-BROKER RECOVERY PARTIAL PASS, JOIN + BACKPRESSURE DEFECT IDENTIFIED`: after Kafka1/Kafka2 returned healthy, the same + projector process (`restart=0`) resumed the durable spool from 101,585 to + 104,614 records in five seconds with zero quarantine. This proves the new + supervisor reconnects autonomously. However, independent Kafka raw/canonical + topic scheduling let the canonical backlog outrun its correlated raw backlog; + the 10,000-record in-memory join bound then failed closed and repeatedly + recreated the consumer generation. No canonical record was checkpointed + before raw durability and no bad public data escaped, but repeated rebalances + leave lag unbounded and are not acceptable for stable release. The bounded + fix is Kafka-native high/low-water backpressure: pause only assigned canonical + partitions before the hard join limit, continue consuming and durably ACKing + raw partitions, drain correlated canonical records in partition order, then + resume canonical partitions below the low watermark. Increasing the RAM bound, + checkpointing canonical early or weakening lineage validation is forbidden. + Unit pause/resume, assignment, overflow, ordering and replay tests plus the + same real two-broker recovery/lag/quarantine gate are mandatory. +- `2026-08-19 PHASE B B5 PROJECTOR JOIN BACKPRESSURE UNIT-VERIFIED, + IMMUTABLE OUTAGE RETEST PENDING`: the Confluent adapter now pauses/resumes only + assigned canonical partitions and reapplies requested flow control after an + assignment epoch changes. The engine reserves one bounded poll batch before + its hard record limit, also applies byte high/low watermarks, continues raw + durability/checkpoint progress while canonical is paused, and resumes only + after both pending records and bytes drain below low water. The original hard + record/byte fence remains active if a broker violates pause. Thirty targeted + projector tests passed with one separately gated Redis integration skip; the + broader stable edge/release/deployment, V2 multi-venue contract and Phase 8.3 + release regression passed 50 tests with the same one conditional skip. Source + was mounted read-only, network was disabled and no V1/runtime state changed. + A fresh immutable image and repeated real two-broker recovery gate still block + candidate certification. +- `2026-08-19 PHASE B B5 CANONICAL REPLAY DETERMINISM DEFECT IDENTIFIED`: + the immutable `9afc21b` projector applied backpressure, reached previously + committed canonical replay and correctly stopped on `EventIdCollision` rather + than overwrite immutable data. A separately authorized read-only diagnostic + group scanned the isolated canonical log without joining/rebalancing the + projector. It found repeated records with the same raw capture, native source + sequence, business payload, partition key and event ID, but a different + `normalized_at_ns` generated from the Rust process wall clock. Thus a retry of + identical raw bytes was not byte-deterministic. The runtime core also rebuilt + partition sequence from process-local zero, which would regress continuity + when a process resumes from a committed Kafka offset. The fix must materialize + immutable normalized/published/accepted timestamps from the durable raw + capture time; processing latency remains operational telemetry. Runtime + `partition_sequence` must derive monotonically and deterministically from the + raw Kafka transport offset plus bounded expanded-row index under the existing + partition-plan epoch. Exact replay across fresh core instances, mid-log restart + continuity, multi-row ordering, overflow, quarantine and transactional tests + plus a clean isolated candidate are mandatory. Event identity, collision + detection, public contracts and V1 remain unchanged. +- `2026-08-19 PHASE B B5 CANONICAL REPLAY DETERMINISM UNIT/CAPACITY + VERIFIED, CLEAN RUNTIME RETEST PENDING`: Rust canonical and quarantine output + timestamps are now materialized from the immutable raw receive boundary; + wall-clock processing latency remains runtime telemetry and cannot alter event + bytes. The Kafka runtime derives `partition_sequence` from raw transport offset + with a fixed one-million-row stride and checked overflow, preserving monotonic + order across fresh processes and bounded expanded frames without changing the + public event-ID contract. Tests prove identical raw/offset input yields exact + bytes across fresh cores despite different processing clocks; quarantine is + likewise deterministic; a mid-log restart advances sequence; two rows retain + order; and offset overflow fails closed. Rust fmt and full workspace tests + passed 57 tests; warnings-as-errors Clippy passed. The unchanged release gate + processed 100,000 canonical events with zero duplicate/quarantine at 127,732 + events/s, p50 5.7 microseconds and p99 15.2 microseconds against the 50,000/s + minimum. No V1/current-Redis state changed. A clean immutable all-one-revision + candidate, duplicate-log scan and repeated recovery gates remain mandatory. +- `2026-08-19 PHASE B B5 REAL-CONSUMER GATE DEFECT IDENTIFIED`: authenticated + reads through both isolated query replicas proved OKX SWAP TRADE/QUOTE and + Binance/OKX final BAR projections, but Binance USD-M TRADE/QUOTE correctly + failed closed as `DATA_STALE`. A bounded provider probe proved both documented + Binance combined-stream paths were delivering authentic frames on this host. + The Rust Binance session had split the socket and discarded its write half, + so it could not explicitly answer provider WebSocket `PING` control frames; + repeated protocol resets eventually made the projected feed stale. The + approved in-scope repair is to keep the session full duplex, answer `PING` + with matching `PONG`, preserve bounded reconnect/backoff, then rebuild the + immutable candidate and repeat authenticated manifest, reconnect, lag, + replay and resource gates. V1 and current Redis remain untouched. +- `2026-08-19 PHASE B B5 BINANCE CONTROL-FRAME REPAIR UNIT-VERIFIED, + REAL-PROVIDER HOLD PENDING`: the native Binance loop now retains the full-duplex + socket and replies to every provider `PING` with the matching `PONG`; data-frame + parsing, authority fencing, durable ACK, bounded exponential backoff and event + identity are unchanged. The current source passed Rust format, workspace + Clippy with warnings denied and all 57 workspace tests. The first test run + failed only because the builder image intentionally omitted repository golden + fixtures; after copying the exact `contracts/` and `tests/` fixture trees into + the disposable test container, the unchanged source passed. A newly labeled + immutable image must now remain fresh beyond the Binance heartbeat boundary + and pass both-replica manifest smoke before this defect is closed. +- `2026-08-19 PHASE B B5 NATIVE-INGEST RESTART/THROUGHPUT DEFECTS + IDENTIFIED`: the c1701c1 real-manifest retest still found Binance USD-M stale. + A unique admin read-only consumer sampled 11,016 committed quarantine records; + every record was `STALE_GENERATION`, primarily Binance BBO/trade frames whose + process-local generation reset after container recreation. The raw ingestor + also waits for one RF3/all-ISR Kafka delivery ACK before reading the next frame; + authentic BTCUSDT USD-M burst traffic can therefore outrun the reader and reset + the provider socket even when Ping/Pong is correct. The in-scope closure is a + durable, fsync-backed per-ingestor/per-service connection-generation counter + mounted only in isolated stable state, plus a strictly bounded concurrent + durable-publish window that preserves per-key enqueue order, backpressures at + its configured limit and never drops/fabricates frames. Restart must advance + generation, stale old generations must still quarantine, and broker failure + must fail closed. Unit, clean restart, authentic provider, lag and quarantine + gates are mandatory before immutable candidate acceptance. +- `2026-08-19 PHASE B B5 NATIVE-INGEST CLOSURE UNIT-VERIFIED, CLEAN + RUNTIME PENDING`: every Rust-native config now owns a distinct absolute + generation-state path and a 512-record hard in-flight bound. Generation is + atomically persisted and fsynced before each connect/reconnect; OKX public and + business services use independent counters. Binance reads control/data frames + while Kafka `acks=all` deliveries are in flight, preserves enqueue order per + Kafka key, stops reading at the hard bound, retries retryable delivery failures + and drains every already-enqueued delivery before failing closed. Four native + services mount only the isolated stable-state volume; current V1 state is not + reachable. Rust format, full workspace Clippy and 59 workspace tests passed; a + final drain-safety edit additionally passed targeted Clippy and both ingestor + tests. Seven deployment tests and 62 broader Phase B/V2/security/release tests + passed with one separately gated Redis skip. A clean immutable candidate must + still prove restart generation advance, zero new quarantine, authentic Binance + freshness beyond heartbeat, bounded lag and broker-outage recovery. +- `2026-08-19 PHASE B B5 CLEAN 0C97E29 CANDIDATE CAPACITY DEFECT + IDENTIFIED, FIX IN PROGRESS`: both immutable images were rebuilt and verified + at revision `0c97e29`; a fresh RF3/minISR2 Kafka, dedicated Redis, stable + state and query cache were bootstrapped under the isolated + `qdl_v2_stable_candidate` project. Broker topic/ACL bootstrap passed and all + app roles used the same image revision. Authentic Binance/OKX traffic produced + zero quarantine/collision/startup errors, but authenticated real-consumer + readiness correctly failed closed because the single Python projector lag grew + monotonically. Within a few minutes its rebuildable SQLite cache reached + 751,783 records / 392,154,317 payload bytes (486,651 raw plus 265,132 + canonical) against the one-million-record hard bound, while projector I/O + reached about 5.18 GB and Kafka canonical/raw lag continued increasing. + Therefore process-up/healthy is explicitly rejected as acceptance and the + isolated acquisition/core/projector roles were stopped before capacity + exhaustion; V1 remained untouched. + + Root cause is the compatibility path, not provider correctness or Rust + normalization: Kafka is already the replay authority, yet the Python + projector serially performs per-record raw lookup, post-ACK canonical lookup + and checkpoint calls around otherwise batched fsync/HTTP/Redis operations, and + the transitional cache retains every high-frequency raw and canonical event + for 24 hours despite a capacity that cannot hold that horizon. The bounded + repair may not sample, drop, fabricate or weaken Kafka durability. It must: + (1) add bounded batch lookup/checkpoint primitives and remove per-event SQLite + rereads/thread hops while preserving downstream-before-checkpoint ordering; + (2) make the SQLite cache explicitly replay-window bounded per partition, with + deterministic cursor-expiry/snapshot recovery while Kafka remains the durable + replay source; (3) preserve sufficient raw lineage for every uncheckpointed + canonical record and fail closed under overflow; and (4) prove sustained + real-provider lag convergence, bounded cache/disk/I/O, exact replay/cursor + behavior, zero quarantine/collision and broker/restart/Redis recovery before + Phase B can close. Public V1/V2 contracts, event identity, freshness policy and + authority remain unchanged. +- `2026-08-19 PHASE B B5 BOUNDED PROJECTOR CACHE/HOT-PATH + UNIT-VERIFIED, IMMUTABLE RUNTIME RETEST PENDING`: the stable SQLite bridge now + supports a disabled-by-default per-partition replay window and the stable role + explicitly selects 10,000 records per partition, matching the public bounded + replay limit. Logical offsets remain monotonic; a cursor older than the retained + window deterministically raises `CursorExpired` and requires the existing + snapshot-and-replay recovery. Kafka remains the only replay authority. SQLite + keeps `synchronous=FULL` but checkpoints its WAL at 1,000 pages instead of + 100 to reduce checkpoint write amplification without weakening transaction + durability. + + The projector now resolves raw lineage and signed-ingest ACKs through bounded + batch queries, checkpoints each fully ACKed raw/canonical micro-batch in one + broker call, and uses a 512-record/10 ms hard micro-batch in the stable role. + Internal HMAC schema, canonical bytes, event IDs, per-partition ordering, + Redis atomic Lua projection and downstream-before-Kafka-checkpoint ordering + are unchanged. The generic spool default remains untrimmed, so existing V1 + and non-stable callers do not inherit the new window implicitly. Targeted + transport/projector/release tests passed 44 tests with one separately gated + Redis integration skip. The broader V2 API/SDK, security, Phase 8 and Phase B + regression passed 102 tests with the same one conditional skip, using the + pinned dependency image, read-only source and no network. A new immutable + image and clean authentic-provider lag/cache/I/O/restart/outage test still + gate acceptance. +- `2026-08-19 PHASE B B5 FIRST BOUNDED-CACHE RUNTIME RETEST FAILED, + TRANSPORT-LINEAGE REPAIR IN PROGRESS`: the clean all-`3d0ff9c` candidate + confirmed that batch lookup/checkpoint and partition windows prevent unbounded + cache growth, but they cannot make the prior topology sustainable. The + projector still consumed and fsync-persisted every raw frame solely to recover + raw bytes needed by the compatibility projection. After one minute, canonical + lag remained increasing (for example 65,351 records on hot canonical partition + 0), projector CPU was about 78%, block writes were already about 1.47 GB and + three raw partitions were continuously trimming at 10,000 rows. This also + risks evicting lineage before a lagged canonical record arrives. The isolated + producer/core/projector roles were stopped; V1 remained unchanged. + + The corrected internal boundary is additive and provider-neutral: the + transactional Rust core must copy the already authenticated + `RawProviderEnvelope` bytes into a private Kafka canonical-record header in + the same exactly-once transaction. The Python projector consumes only + read-committed canonical records, validates embedded capture ID/hash/provider/ + session/authority against the canonical envelope, and passes those bytes only + to the internal HMAC compatibility projector. SQLite then stores canonical + replay/query cache rows only; Kafka raw/canonical remain the authorities. + Public Protobuf, REST/gRPC, event ID, partition key, freshness, Redis V1 shape + and venue semantics may not change. The prior split raw-topic join remains a + test/rolling-compatibility path but is not selected by stable `2.0.0`. + Mandatory gates are exact Rust transaction-header replay, missing/tampered + lineage fail-closed, no raw SQLite rows in stable runtime, V1 compatibility + golden parity, sustained provider lag convergence, bounded canonical cache, + process/broker/Redis recovery and zero quarantine/collision. +- `2026-08-19 PHASE B B5 PRIVATE TRANSPORT LINEAGE UNIT-VERIFIED, + IMMUTABLE RUNTIME RETEST PENDING`: the Rust transactional canonical output + now carries the exact consumed `RawProviderEnvelope` in a private + `qdl-raw-provider-envelope` Kafka header in the same output/source-offset + transaction. The stable Python Kafka adapter selects only the read-committed + canonical topic, requires that header, validates capture ID and all raw + provenance again at the projector and signed-ingest boundary, and persists + only canonical events in the bounded SQLite cache. The previous raw-topic + join remains available only when explicitly configured for rolling/test + compatibility; missing inline lineage with canonical-only topology fails + closed. + + The internal HMAC endpoint accepts the lineage field additively, maps malformed + protobuf to bounded 422, and never writes the private raw bytes into cache + headers or public payloads. Public Protobuf/OpenAPI/SDK/event identity and + exact Redis compatibility projection remain unchanged. Rust fmt, full + workspace Clippy with warnings denied and all 60 workspace tests passed, + including exact present/absent private-header assertions. Python compileall + passed; targeted tests passed 46 with one conditional Redis skip and the broad + V2/security/Phase 8/Phase B suite passed 104 with the same skip. Tests cover + canonical-only subscription, valid embedded lineage without raw cache rows, + missing lineage, signed inline ingest, malformed lineage, legacy fallback, + replay/idempotency and checkpoint ordering. Clean immutable real-provider + lag/cache/I/O and recovery evidence remains mandatory. +- `2026-08-19 PHASE B B6 AUTHENTIC BAR WARMUP CLOSURE IN PROGRESS`: the clean + all-`2b38f95` candidate proved the private-lineage topology is sustainable: + projector lag converged from 1,112 to 62 while provider traffic continued, + SQLite contained canonical rows only, every partition stayed at or below + 10,000 rows, and no quarantine/collision was observed. Authenticated Binance + and OKX TRADE snapshots returned the typed V2 contract and a mismatched + consumer identity failed closed. The same real-runtime test exposed a release + blocker: both registered alpha manifests require 500 final 1m BARs, while the + clean latest-only BAR edges returned `409 PARTIAL_RESULT`. + + The approved repair remains inside the isolated Phase B topology. Add bounded + real-provider historical BAR bootstrap for Binance Spot/USD-M and OKX + Spot/SWAP, publish every native row through raw Kafka and the transactional + Rust core, and never write history directly into SQLite or Redis. Final BAR + identity must be idempotent across REST bootstrap, WebSocket delivery and + process restart; conflicting content for the same immutable provider bar must + fail closed. No synthetic/generated runtime market data is allowed. Required + gates are exact 500-row closed/contiguous coverage per crypto alpha manifest, + Python/Rust canonical-byte parity, duplicate/restart idempotency, malformed/ + incomplete provider fail-closed behavior, bounded resource/lag evidence and + warmup -> signed cursor -> replay -> live through the released SDK. Rollback + stops/removes only `qdl_v2_stable_candidate`; V1 port 8100, current Redis and + current consumers remain untouched. +- `2026-08-19 PHASE B B6 AUTHENTIC BAR WARMUP IMPLEMENTED AND UNIT-VERIFIED, + IMMUTABLE RUNTIME ACCEPTANCE PENDING`: Binance Spot/USD-M and OKX Spot/SWAP + now fetch a bounded 500-row final 1m history from the approved public REST + APIs before the live loop. Strict adapters reject partial coverage, malformed + native rows, provisional bars, time gaps and conflicting duplicate timestamps. + Every accepted row is wrapped with HTTP/provider provenance and enters raw + Kafka; the Python edge never writes SQLite or Redis directly. OKX final BAR + source/event identity no longer includes transport partition sequence, so the + same provider bar is idempotent across REST bootstrap, WebSocket delivery and + restart while a same-identity payload conflict remains fail-closed. The + provider-owning DNSE stable edge moved under `qdl.adapters.vn`; its old runtime + import remains a compatibility facade, restoring the static role boundary + without changing vendor behavior. + + Targeted BAR/deployment/golden tests passed 17/17. The official 19-case + multi-venue golden generator/check passed; only the intended OKX Spot/SWAP BAR + bytes and manifest hashes changed. Full Python discovery passed 478 tests with + six explicit dependency/infrastructure skips. Full Rust fmt, workspace Clippy + with warnings denied and workspace tests passed. A bounded real-provider probe + loaded exactly 500 contiguous closed rows for Binance USD-M, Binance Spot, OKX + SWAP and OKX Spot over the same window, with test provenance false. No V1 + service, current Redis namespace or consumer was mutated. A new all-one-SHA + immutable candidate must still prove the registered warmup -> cursor -> replay + -> live and recovery/resource gates before Phase B closes. +- `2026-08-19 PHASE B B7 NATIVE INGESTOR STARTUP-ORDER DEFECT CLOSED, + IMMUTABLE REHEARSAL PENDING`: the fresh all-`ad4338d` isolated candidate + bootstrapped exactly 500 contiguous final provider BARs for Binance USD-M, + Binance Spot, OKX SWAP and OKX Spot (2,000 raw records total). Runtime + inspection then found that the Spot native ingestors raced `stable_tls_init` + and exited before `/stable-certs/producer/ca.crt` existed; USD-M/SWAP happened + to win the same race. The compose contract now requires completed TLS and + state initialization plus healthy Kafka1/Kafka2/Kafka3 for all four native + ingestors. Compose validation passed and all seven stable deployment contract + tests passed, including exact dependency regression assertions for every + native ingestor. The next all-one-SHA immutable candidate must demonstrate all + four processes start cleanly before consumer/recovery certification continues. + Rollback remains removal of only project `qdl_v2_stable_candidate`; V1 is + unchanged. +- `2026-08-19 PHASE B B8 LATEST-STATE CAPACITY DEFECT CONFIRMED, FIX IN + PROGRESS`: after B7, all four native ingestors started cleanly from immutable + `6a9a19e` images and authentic 500-row BAR bootstrap again completed for all + four crypto bindings. Under current provider traffic the single Python + projector backlog nevertheless grew from about 35.8k to 244.6k records in + 60 seconds. Raising its quota from 0.75 to 2 CPUs still grew backlog by about + 63k/minute, and three projector replicas still could not drain the hottest + ordered partition. A read-committed one-record tail inspection identified + partition 0 as Binance USD-M `bookTicker`, receiving roughly 3.0-3.5k BBO + updates/second. Trade/final-BAR partitions were not the dominant source. + + CPU scaling and batch-boundary canonical coalescing are rejected: one cannot + parallelize a single ordered key, while transaction-batch coalescing would + make replay output depend on non-deterministic batch boundaries. The bounded + repair follows the approved lifecycle policy: acquisition bindings declare + `LOSSLESS` or `LATEST_STATE`; only `LATEST_STATE` BBO frames may replace the + same pending key inside a fixed bounded flush window before raw Kafka ACK. + The last authentic provider frame in each window is durably published with + unchanged bytes/provenance, a pending value is flushed on timer and orderly + disconnect, and coalescing counters are explicit. TRADE, final/revised BAR, + book and quality/authority transitions remain lossless and backpressured. + Rust unit/replay/ordering/failure tests plus clean real-provider lag, + freshness, resource and recovery evidence gate closure. The overloaded + isolated candidate and its volumes were removed before disk bounds; V1 was + not touched. +- `2026-08-19 PHASE B B8 LIFECYCLE-AWARE ACQUISITION COALESCING + UNIT-VERIFIED, IMMUTABLE RUNTIME RETEST PENDING`: generated native acquisition + configs now carry provider-neutral `feed` and `delivery_class` fields plus a + bounded 50 ms latest-state flush window. Rust rejects every mismatched policy: + only QUOTE may be `LATEST_STATE`, while TRADE and BAR must be `LOSSLESS`. + Binance and OKX keep at most one pending authentic raw frame per quote binding, + replace it only before Kafka acceptance, flush the last frame on the bounded + timer and orderly session exit, retain the existing ACK/retry/backpressure + path, and expose accepted/coalesced counts. Lossless frames bypass this buffer. + + Eleven targeted Python deployment/history tests passed. The pinned Rust + release build completed; four native-ingestor tests passed, targeted Clippy + passed with warnings denied, and rustfmt is clean. Unit evidence covers policy + mismatch rejection, last-frame byte/timestamp retention and durable generation + behavior. A new one-SHA candidate must still prove authentic quote freshness, + trade losslessness, decreasing/near-zero core and projector lag, bounded + cache/resource use, restart/failover and no quarantine/collision before B8 or + Phase B can close. +- `2026-08-19 PHASE B B8 IMMUTABLE REAL-PROVIDER CAPACITY GATE PASSED, + CONSUMER/RECOVERY CERTIFICATION IN PROGRESS`: a clean all-`0a25407` candidate + started all four Binance USD-M/Spot and OKX SWAP/Spot native ingestors only + after TLS/state initialization and healthy RF3/minISR2 brokers. The bounded + REST bootstrap published exactly 500 contiguous closed 1m provider bars per + binding (2,000 total, `test_provenance=false`) through raw Kafka and the Rust + core. Under continuing authentic provider traffic, Rust-core lag was 50 + records and projector lag was 29 records at the bounded observation point; + the prior projector growth reversed and an earlier 60-second sample decreased + from 265 to 84. SQLite contained only `md.canonical.v2`, zero quarantine rows, + 76,795 records, and no partition exceeded the configured 10,000-record replay + window. Query cache utilization was about 6.9%; all query dependencies were + READY; exactly one stream gateway held epoch-1 lease while its peer reported + STANDBY. The isolated Redis used about 3.3 MiB and app-role memory remained + bounded; no application warning/error was observed in the sampled startup and + runtime window. + + Current V1 remained container `0e0eb56c78ba`, image `8f2a5a3f1ff9`, Up and + HTTP 200 on port 8100. No current Redis namespace, V1 consumer or authority was + changed. B8 code/capacity is accepted, but Phase B is not closed: authenticated + Binance/OKX/VN/Trading-System consumer flows, active/passive restart, + generation restart, broker outage, Redis rebuild, full regression/release + checks, compact evidence and exact candidate cleanup remain mandatory. DNSE + testing must use the configured real provider credentials/session and fail + closed if 500-bar coverage is unavailable; generated VN data is forbidden. +- `2026-08-19 PHASE B B9 CONSUMER ACCEPTANCE DEFECTS CONFIRMED, FIX IN + PROGRESS`: authenticated SDK acceptance correctly failed closed with + `DATA_STALE`. Bounded inspection proved Binance BARs continued to append every + minute, but OKX Spot/SWAP BARs stopped at the one-time bootstrap because the + stable BAR cycle polled only Binance bindings. The real DNSE WebSocket + authenticated and subscribed successfully with configured credentials, while + its REST BAR poll timed out; the VN edge also has no 500-row historical + bootstrap and therefore cannot satisfy its registered alpha manifest after a + clean start. Finally, both query service and SDK compare wall-clock freshness + before honoring `MARKET_CLOSED`, contradicting the approved rule that a closed + session is neither stale nor offline. + + The in-scope repair is bounded and contract-preserving: poll the latest closed + OKX BAR through the existing strict history adapter; bootstrap exactly 500 + authentic closed DNSE 1m rows through raw Kafka/Rust with bounded retry and + duplicate/conflict validation; centralize VN session-calendar lookup and skip + live BAR polling while the session is closed; treat `MARKET_CLOSED` as + available historical state for ALPHA/RESEARCH while preserving + `execution_eligible=false` and execution-grade SDK fail-closed behavior. Tests + must cover incomplete/conflicting history, retry exhaustion, session closure, + OKX append, server/SDK policy and the real registered consumer flow. No public + schema, V1 route, current Redis or authority mode may change. +- `2026-08-19 PHASE B B9 MULTI-VENUE BAR/SESSION REPAIR IMPLEMENTED AND + UNIT-VERIFIED, IMMUTABLE RUNTIME RETEST PENDING`: the stable crypto BAR cycle + now appends the latest strictly closed native BAR for Binance USD-M/Spot and + OKX SWAP/Spot through one lossless raw-Kafka batch; OKX uses its existing + strict V5 history adapter with `limit=1`, and all Kafka ACKs are cardinality + checked. DNSE now bootstraps exactly 500 closed provider 1m rows per configured + FPT/VN30F1M binding over a bounded 30-day lookback, retries at most four times, + deduplicates identical native timestamps, rejects conflicting duplicates or + partial coverage, and publishes only authentic rows through raw Kafka/Rust. + Live DNSE BAR polling is retry-bounded, ACK-checked and skipped outside the + governed calendar session. + + The VN calendar moved to a shared domain resolver keyed by canonical + `session_calendar_id`. Query and SDK semantics now preserve the approved rule: + `MARKET_CLOSED` history is readable by ALPHA/RESEARCH despite wall-clock age, + while execution-grade access fails closed as `DATA_NOT_READY` and no item is + execution eligible. Public schema, route and event identity are unchanged. + Targeted tests passed 31/31; the broader stable projector/query/SDK/security/ + release regression passed 85/85 with one explicit infrastructure-gated skip; + compose validation including `stable-vn` passed. Test cases include OKX live + append/idempotence, DNSE transient retry, partial/conflicting history, exact + units/provenance, closed-session no-REST behavior, server/SDK policy and + execution blocking. The next gate is a clean immutable all-one-SHA candidate + with real Binance/OKX/DNSE consumer and recovery evidence. +- `2026-08-19 PHASE B B9 OKX PROVISIONAL-CLOSE RETRY CLOSED`: the first + immutable `0228036` runtime attempt showed OKX could still report the just-ended + candle as provisional at boundary `+1s`; the outer cycle recovered at `+3.5s` + and ACKed all four crypto BARs, but emitted a misleading ERROR. The OKX adapter + now retries this provider transition internally at bounded 0.5/1/2-second + delays (four attempts maximum) and raises only after exhaustion. No candle is + accepted before `confirm=1`, and event identity remains native-time based. + Fourteen focused BAR/deployment tests passed, including provisional recovery + and exhaustion. A final immutable image/runtime replay remains required. +- `2026-08-19 PHASE B B10 WARMUP FRESHNESS DEFECT CONFIRMED, FIX IN + PROGRESS`: real registered alpha warmup still failed `DATA_STALE` even though + Binance and OKX latest closed BARs were aligned and only 43 seconds old. The + SDK was applying realtime freshness/state and execution eligibility to every + row in a 500-row historical warmup. Historical context is necessarily older + than the realtime freshness threshold, so this made any non-trivial warmup + impossible. The correction keeps identity, source policy, coverage, + completeness, ordering and BAR finality checks on every row, but evaluates + stale/gap/realtime execution eligibility only on the tail watermark row. A + stale or non-authoritative tail must still fail closed. Public schema and + server query semantics remain unchanged; targeted SDK tests and the authentic + warmup -> cursor -> replay -> live flow gate closure. +- `2026-08-19 PHASE B B10 SDK WARMUP QUALITY REPAIR UNIT-VERIFIED, + AUTHENTIC FLOW RETEST PENDING`: SDK warmup validation now checks instrument, + feed, interval, source policy, completeness, history gap and final BAR semantics + for every row, while applying wall-clock freshness/state and execution + eligibility only to the tail watermark. A gap anywhere in the requested history + still blocks. A stale, unavailable or non-authoritative tail still blocks. This + preserves strict execution safety without rejecting valid historical context. + Twenty-two SDK/API/end-to-end tests passed, including an execution-grade + two-row warmup with an old non-executable context row and a fresh executable + tail, plus stale-tail rejection. Public V2 schema and server routes are + unchanged. +- `2026-08-19 PHASE B B11 UNIFIED-STREAM CURSOR SCOPE DEFECT CONFIRMED, + FIX IN PROGRESS`: the authenticated stable SDK flow passed query/history + validation after B10, then gRPC subscribe failed closed with + `CURSOR_INVALID: cursor stream does not match the data requirement`. The + signed cursor is correct: stable topology intentionally uses one canonical + transport stream, `md.canonical.v2`, and carries instrument/feed/source scope + in the partition key. The generic gRPC service still hardcodes the earlier + beta convention `md.canonical.v2.`, so it rejects its own stable + snapshot cursor. + + The bounded repair must not weaken cursor authorization or change any public + Protobuf/API/token. Introduce an injectable, transport-neutral cursor-scope + validator: the existing feed-scoped validator remains the default for beta + and existing tests, while stable runtime validates the exact canonical stream + and exact catalog binding partition for the requested instrument, feed, + interval and source policy. Wrong consumer, stream, instrument, feed, source + or policy must continue to fail closed. Required gates are targeted positive + and adversarial contract tests followed by the authentic registered + warmup -> signed cursor -> replay -> live flow. Rollback is code-only inside + the isolated candidate; V1 port 8100, current Redis and current consumers + remain unchanged. +- `2026-08-19 PHASE B B11 CATALOG-AUTHORITATIVE CURSOR SCOPE UNIT-VERIFIED, + AUTHENTIC FLOW RETEST PENDING`: gRPC stream scope validation is now an + injectable transport-neutral contract. The default validator preserves the + existing feed-specific beta convention without changing its call sites. The + stable runtime explicitly installs a catalog-backed validator that resolves + the full requirement, including interval and source policy, then requires the + exact catalog canonical stream and instrument/feed/source partition. This + permits the intentional unified `md.canonical.v2` stream without accepting an + arbitrary same-prefix stream or partition. Public protobuf, signed cursor + format, SDK and replay gateway are unchanged. + + The targeted Phase B edge and Phase 5 stream/SDK suites passed 39 tests with + one pre-existing dependency-gated skip. Positive unified-stream scope and + wrong stream, partition, feed and source-policy failures are covered; the + existing signed consumer/scope mismatch regression remains green. The final + immutable candidate still must pass the authenticated real-provider consumer + and recovery gates before Phase B can close. +- `2026-08-19 PHASE B B12 IMMUTABLE CRYPTO CONSUMER ACCEPTANCE PASSED, + RECOVERY CERTIFICATION IN PROGRESS`: immutable Python and Rust images were + built from `df88de0` with OCI revision/version labels and pinned into a fresh + secret bundle; the manifest records image SHA-256 IDs and no secret values. A + clean isolated RF3/minISR2 Kafka topology with mTLS/ACLs started four native + Binance USD-M/Spot and OKX SWAP/Spot ingestors, three Rust core workers, one + Python projector, two query replicas and one active plus one fenced standby + stream role. The real-provider BAR edge ACKed exactly 500 contiguous final 1m + rows for each of four crypto bindings, 2,000 total, through raw Kafka/Rust. + + Authenticated registered consumers then passed without source mounts or test + data. Binance and OKX alpha manifests each returned 500 rows with `FULL` + coverage and completed signed cursor `REPLAYING -> LIVE` handoff to the next + correct TRADE event. Both query replicas returned identical canonical 500-row + data and watermark. Monitoring received live Binance and OKX TRADE. The + Trading System paper manifest received live TRADE and QUOTE for both venues; + all four snapshots were `LIVE`, execution eligible and measured 129-779 ms + fresh against their strict limits. No V1/current Redis/consumer was changed. + + VN cannot be certified on this host yet: the official DNSE WebSocket endpoint + is reachable and authenticated in the prior bounded attempt, but the official + `openapi.dnse.com.vn` REST history endpoint is not reachable at TCP 443 from + this new host. Existing V1 Parquet lacks the exact raw provider lineage needed + to claim `DNSE_DIRECT`, so it is deliberately not relabeled or injected. This + is an external provider/egress gate, not permission to fabricate runtime data. + Phase B still requires restart/failover, two-broker outage, Redis rebuild, + full regression/release evidence and exact candidate cleanup. +- `2026-08-19 PHASE B B13 RECOVERY DEFECT CONFIRMED, FIX IN PROGRESS`: + restarting only the Binance USD-M native ingestor advanced its persisted + connection generation from 1 to 2 while all other sources remained at 1; the + three Rust workers continued processing with zero quarantine, so generation + restart/fencing behavior passed. Active/passive testing then stopped the exact + lease owner. The standby acquired epoch 2 and its readiness changed from 503 + `STANDBY` to 200 `READY`, but the first resume attempt overlapped takeover and + received retryable `GATEWAY_FENCED`; the continuity gate must be rerun after a + stable owner is observed. + + Bounded logs also exposed a real interceptor defect whenever an SDK stream is + closed: the unary-stream wrapper keeps a `ContextVar` token across `yield`, so + async-generator finalization may reset that token from a different context and + emit `Task exception was never retrieved`. The repair must scope authorization + around each awaited iterator step and explicit iterator close, never across a + yielded response. Authentication/authorization semantics, public gRPC schema + and cursor behavior cannot change. Tests must prove clean client cancellation + with no unhandled loop exception, retain all authorization failures, then rerun + checkpoint -> owner stop -> epoch takeover -> contiguous replay/live. V1 and + all non-candidate state remain untouched. +- `2026-08-19 PHASE B B13 STREAM FINALIZER REPAIR UNIT-VERIFIED, FAILOVER + RETEST PENDING`: the gRPC authorization interceptor now establishes and resets + request identity around each awaited source-iterator step. It holds no + `ContextVar` token across a yielded response, and explicit iterator close is + also authorized/reset within one context. Authentication, entitlement, quota, + cursor and protobuf behavior are unchanged. A new real-gRPC cancellation test + captures the event-loop exception channel and proves client `aclose()` emits no + cross-context finalizer error. Phase 5 stream/SDK, Phase 7 security and Phase B + edge suites passed 51 tests with one pre-existing dependency-gated skip. A new + immutable Python image and stable-owner checkpoint resume retest remain + mandatory before B13 closes. +- `2026-08-19 PHASE B B13 IMMUTABLE FAILOVER GATE PASSED`: a clean all-`cfc0246` + candidate repeated the owner transition with readiness gating. The SDK ACKed + offset 2,271 on epoch-1 owner `stable-stream-active`; after that exact owner + stopped, `stable-stream-passive` acquired epoch 2 and the SDK resumed from the + persisted signed checkpoint at exactly offset 2,272. The event was supplied by + durable replay before the `LIVE` control, as required for a gap-free handoff. + No cross-context finalizer, unhandled task, cursor or authorization error was + emitted. B13 is closed. +- `2026-08-19 PHASE B B14 KAFKA RECOVERY CAPACITY DEFECT CONFIRMED, FIX IN + PROGRESS`: the two-broker outage correctly failed closed: native producers + emitted `MessageTimedOut` retry events and no successful ACK, while Rust core + workers restarted their generations after broker transport failure. During + quorum restoration, however, Kafka3 was cgroup OOM-killed at its 512 MiB hard + limit (exit 137, `OOMKilled=true`) while reloading coordinator/transaction + metadata. Host memory remained healthy with about 9 GiB available, proving the + failure is an undersized broker cgroup rather than host exhaustion. + + Raise only the stable candidate broker limit to 768 MiB while retaining a 256 + MiB JVM heap, pinned image, bounded CPU and separate volumes. Add a deployment + regression for the measured headroom, roll the three candidate brokers one at + a time, and repeat two-broker loss/restore. Acceptance requires all 3/3 healthy, + no OOM, no false ACK, automatic producer/core/projector recovery, increasing + durable offsets and fresh query/stream service. This does not change V1 or the + older Phase 8 rehearsal compose contract. +- `2026-08-19 PHASE B B14 KAFKA RECOVERY HEADROOM UNIT-VERIFIED, RUNTIME + RETEST PENDING`: `docker-compose.v2-stable.yml` now bounds each broker at 768 + MiB while keeping `-Xms256m -Xmx256m`, 0.75 CPU, pinned image, RF3/minISR2 and + independent state unchanged. This supplies measured coordinator/native-buffer + recovery headroom rather than increasing the Java workload. Compose validation + passed and 16 stable deployment/release contract tests passed, including exact + resource assertions. A rolling 3/3 broker recreation and repeated two-broker + outage/restore still gate runtime acceptance. +- `2026-08-19 PHASE B B14 TWO-BROKER OUTAGE/RECOVERY GATE PASSED`: all three + candidate brokers were rolling-recreated with preserved independent volumes + and the new 768 MiB bound. The repeated loss of Kafka1 and Kafka2 left Kafka3 + running without OOM; native OKX/Binance publishers emitted retryable delivery + timeouts and no success ACK under minISR failure. After restoring the two + brokers, all six raw partitions reported ISR `1,2,3`, all brokers remained + `OOMKilled=false`, and durable raw offsets advanced from + `[27775,3876,26500,0,30715,14959]` to + `[33157,4851,29594,0,34752,16487]`. Rust generations recovered, all workers + resumed progress with quarantine zero, and a strict Trading System QUOTE + snapshot/live handoff returned `LIVE`, execution eligible and 239 ms fresh. + B14 is closed. +- `2026-08-19 PHASE B B15 REDIS REBUILD CERTIFICATION IN PROGRESS`: Redis is a + rebuildable stable projection and lease dependency, while Kafka plus canonical + SQLite remain durable authorities. Before recreating only the isolated stable + Redis, stop the projector and both stream processes so an ephemeral lease epoch + reset cannot overlap a locally unexpired old owner. Reset only the inactive + `stable-projector-v1` canonical Kafka group to earliest, restart one active plus + one standby stream, then restart the projector and require idempotent replay, + repopulated bounded Redis keys, query readiness, signed cursor replay/live, no + collision/quarantine and unchanged V1. Production promotion still requires a + governed external HA lease store or the same all-owner fencing runbook; this + local rebuild does not claim an independent failure domain. +- `2026-08-19 PHASE B B16 PROJECTION CACHE GENERATION DEFECT CONFIRMED, REPAIR + APPROVED`: the B15 rehearsal rebuilt Redis while retaining a bounded SQLite + cache whose oldest event rows had already been trimmed. Resetting the Kafka + projector group to earliest therefore reintroduced older canonical event IDs + after their SQLite dedup rows had expired; the cache assigned new logical + offsets and correctly failed strict BAR continuity with + `OPEN_SEQUENCE_GAP`. Kafka/canonical records were not lost and V1 was not + touched, but B15 is not accepted. + + Redis latest state and the SQLite query/stream spool are now one rebuildable + **projection cache unit** behind Kafka authority. Persist a random cache + identity in SQLite, bind the dedicated Redis namespace atomically to that + identity, and verify the binding in every Redis projection transaction. + Missing or mismatched identity with a non-empty spool fails closed before + consuming Kafka; a Redis flush during operation also fences the next write. + Do not retain an unbounded event-ID tombstone table merely to make a bounded + cache mimic the durable log. Recovery stops projector, query and every stream + lease owner, recreates only isolated stable Redis plus SQLite cache files, + resets only `stable-projector-v1` to earliest, then starts stream, projector + and query in that order. Existing signed cursors expire by design and clients + must perform a fresh warmup/handoff. + + Acceptance requires unit/real-Redis tests for stable identity across restart, + mismatch/missing/flush fencing and atomic empty-cache initialization; a full + canonical replay into a fresh bounded cache; zero gap/collision/quarantine; + replica-equal query results; SDK replay/live; fresh Trading System paper + data; and unchanged V1. Rollback removes only the isolated candidate and + restores the previous immutable image; no production authority or consumer + route changes are authorized. +- `2026-08-19 PHASE B B16 CACHE-GENERATION FENCING TARGETED TEST PASSED, + REAL-REDIS/REPLAY GATE PENDING`: the SQLite spool now persists one random + 128-bit lowercase-hex cache identity across process restart and generates a + new identity only when the cache file is rebuilt. The stable Redis target + atomically binds its isolated namespace to that identity; every projection + Lua transaction verifies the binding, so a missing/mismatched identity or a + Redis flush fences writes rather than creating a partial latest-state view. + Projector startup permits first binding only when the spool is empty and + readiness exposes a hashed cache identity, never the raw identifier. + + `python -m unittest -v tests.test_fund_phase2_transport + tests.test_phaseb_stable_edge` ran inside immutable Python image + `qdl-v2-python:2.0.0-cfc0246` with read-only source, no network and bounded + tmpfs: 42 cases ran, 41 passed and only the explicitly environment-gated + real-Redis integration case skipped. Python compile and `git diff --check` + passed. An interrupted attempt created only disposable network + `qdl-phaseb-cache-test`; it was verified and removed before continuing. + B16 remains `IN_PROGRESS` until the named real-Redis test and fresh atomic + Kafka replay acceptance pass. +- `2026-08-19 PHASE B B16 REAL-REDIS GENERATION GATE PASSED`: a named + disposable Redis 7.2 instance with persistence disabled, 16 MiB no-eviction + bound and isolated Docker network passed the previously skipped integration + test in 0.210 seconds. The test proved first atomic bind, TTL/non-TTL latest + writes, one Pub/Sub publication, duplicate suppression, stale lease fencing, + conflicting cache-ID rejection, live identity loss detection and projection + rejection after deleting the identity key. Combined with the preceding + network-disabled run, the B16 targeted gate is 42/42 passed with zero skips. + Container and network were removed immediately; no candidate, V1 or current + Redis state was addressed. B16 code is ready for a coherent commit, while + B.3 still requires the fresh full Kafka -> cache-unit replay runtime gate. +- `2026-08-19 PHASE B B17 ATOMIC CACHE-UNIT REBUILD STARTED`: add one guarded + operator command for the exact isolated stable project. It must stop the + projector, both query replicas and every stream lease owner; delete only the + three SQLite cache files in `stable_state`; flush only `stable_redis`; reset + only consumer group `stable-projector-v1` on `md.canonical.v2` to earliest; + start stream owners, then projector; wait for zero bounded lag and bound + cache identity; finally start query replicas. Kafka brokers/topics/raw data, + Rust core/acquisition, TLS, manifests and V1 remain untouched. + + The command defaults to plan-only and requires both an explicit apply flag + and exact confirmation token. A failure leaves readers/projector stopped or + NOT_READY and is safely rerunnable from Kafka; it never rolls forward a + partial cache as healthy. Tests must cover the destructive guard, exact + service/file/topic/group allowlist, lag parsing and fail-closed command + failure before the command is used on `qdl_v2_stable_candidate`. +- `2026-08-19 PHASE B B17 GUARDED REBUILD COMMAND UNIT-VERIFIED, RUNTIME + REHEARSAL PENDING`: `scripts/rebuild_v2_stable_projection_cache.py` is + plan-only by default and requires exact token + `REBUILD_QDL_V2_STABLE_PROJECTION_CACHE` for apply. Its constants pin the + stable Compose manifest/project, five cache users, three SQLite files, + `stable_redis`, `stable-projector-v1` and `md.canonical.v2`; callers cannot + inject an arbitrary service, volume, topic or consumer group. It starts + stream -> projector -> query, requires two consecutive zero-lag samples, + projector/query readiness and non-empty rebuilt Redis. + + Six command-policy tests passed for exact allowlists, confirmation, canonical + lag parsing, wrong-project rejection and abort-before-delete/flush while a + cache user remains running. The combined B16/B17 targeted suite ran 48 + cases: 47 passed and only the separately passed real-Redis case skipped in + the network-disabled invocation. Python compile and diff checks remain clean. + No runtime was mutated by these tests. B17 now needs a new immutable Python + image and one execution against the isolated cfc0246 candidate state. +- `2026-08-19 PHASE B B17 FIRST RUNTIME REHEARSAL FAILED CLOSED AT AN + OVER-STRICT LAG GATE`: the guarded command stopped all five cache users, + recreated only SQLite cache files plus isolated Redis, reset only the + canonical projector group and replayed the authentic Kafka log with image + `e002da6`. Lag converged from about 448,000 to 32 records within the + 900-second bound, but continuous live provider input prevented two exact-zero + samples. The command raised `TimeoutError` and left both query replicas + stopped; no partial cache was served and V1 remained unchanged. + + Exact zero is not a valid steady-state requirement for an actively written + topic. The bounded correction requires three consecutive samples at or below + 250 total records across exactly six canonical partitions, then projector + cache-generation readiness. Query starts only afterward, and strict + warmup/gap/freshness plus SDK/Trading-System checks remain the authoritative + data acceptance. Increasing the bound at runtime, stopping acquisition to + manufacture zero, or weakening freshness/gap policy is forbidden. +- `2026-08-19 PHASE B B17 CORRECTED ATOMIC REBUILD PASSED`: the same guarded + command completed against `qdl_v2_stable_candidate` after the fixed bounded + live-lag policy was applied. It observed all six canonical partitions, three + consecutive samples at or below the immutable 250-record bound, a maximum + accepted sample of 232 and final observed lag of 63. It rebuilt the isolated + Redis namespace to 47 keys, started stream -> projector -> query in the + declared order, and did not address V1, Kafka data, Rust acquisition/core or + production consumers. +- `2026-08-19 PHASE B B18 STRICT CONSUMER ACCEPTANCE FAILED CLOSED ON A REAL + BAR GAP`: immediately after the fresh Redis-plus-SQLite rebuild, the + authenticated SDK warmup for the registered Binance/OKX BAR consumers raised + `required feed has an unresolved sequence gap`. This proves the earlier + retained-cache hypothesis was incomplete: the gap is reproducible from the + canonical Kafka log into a new cache. The acceptance stopped before Trading + System execution data was exposed. B.3 remains `IN_PROGRESS`; the next + bounded repair must identify the exact venue/partition/open-time discontinuity + and fix provider bootstrap/canonical ordering or revision handling from real + data. Synthetic gap filling and policy relaxation are forbidden. +- `2026-08-19 PHASE B B18 ROOT CAUSE CONFIRMED FROM REAL PROVIDER DATA`: + read-only inspection of all four continuous BAR partitions found exactly one + discontinuity in each Binance Spot/USD-M partition: `16:52 -> 16:54 UTC`; both + OKX partitions were continuous. At 16:54 the BAR edge fetched four closed + provider bars but Kafka durable ACK failed. `run_cycle()` advanced its + in-memory `_last_open_ms` before receiving the ACK and, on recovery, requested + only the newest closed bar. It therefore skipped the unacknowledged Binance + 16:53 bar permanently. No explicit provider gap flag, duplicate open time or + generated row was present. + + The approved B18 repair is provider-neutral BAR catch-up inside the existing + edge: derive the missing interval count from the last ACKed open time, fetch a + bounded closed-history window when more than one bar is pending, verify exact + continuity/finality, publish the complete ordered batch, and advance each + watermark only after every Kafka ACK. Publish failure must preserve the old + watermark so the next cycle retries the same range. Tests must cover one-bar + fast path, multi-bar catch-up, publish failure/retry, incomplete provider + history fail-closed and Binance/OKX parity. Then rebuild the isolated cache + and rerun strict authenticated consumer acceptance. +- `2026-08-19 PHASE B B18 ACK-AUTHORITATIVE BAR CATCH-UP IMPLEMENTED AND + UNIT-VERIFIED`: the shared Binance/OKX BAR edge now freezes one observation + boundary per cycle, uses the one-bar latest path normally, and fetches up to + 1,000 real closed provider bars only when the last ACKed watermark proves a + backlog. It validates exact interval boundaries and complete ordered coverage, + publishes the whole batch, and advances per-binding watermarks only after all + Kafka acknowledgements. Provider incompleteness, non-integral boundaries, + excessive backlog and partial/failed ACKs fail closed without watermark + mutation. + + Focused deployment/history tests ran 16/16 passing with network disabled. New + cases prove an eight-row Binance/OKX catch-up is retried identically after an + injected Kafka ACK failure, and incomplete history reaches neither publisher + nor watermark. Python compile and diff checks passed. B18 is not runtime- + accepted until an immutable image heals the authentic 16:53 gaps via provider + history, a fresh cache rebuild reports zero open gaps, and strict SDK/Trading + System acceptance passes. +- `2026-08-19 PHASE B B18 RUNTIME HEAL EXPOSED CANONICAL DUPLICATE COLLISION`: + immutable image `2041f18` bootstrapped 2,000 real provider BAR rows with full + Kafka ACK, but the projector rejected repeated historical BARs because their + semantic `event_id`/`canonical_payload_hash` matched existing rows while + capture/session/timing provenance correctly differed. Generic SQLite collision + enforcement is therefore not weakened: the repair belongs in the stable + projector, which understands canonical semantics. It may classify only an + identical 32-byte canonical payload hash under the same event ID and partition + as a semantic duplicate; a changed market payload remains a hard collision. + + The same repair must keep late historical BARs in the bounded query cache + without replacing a newer Redis/latest projection. Exact Kafka replay after a + crash still reapplies an existing record before checkpoint; a later semantic + duplicate is checkpointed without fan-out. Tests must prove changed semantics + fail closed, provenance-only duplicate recovery is idempotent, late BAR repair + closes history without latest regression, and full fresh replay remains + deterministic. Runtime quarantine inspection found 417 records, all old OKX + stale-generation rows and none for the missing Binance 16:53 bars; no fake or + provider-invalid row is being accepted. +- `2026-08-19 PHASE B B18 SEMANTIC DUPLICATE AND LATE-BAR REPAIR UNIT-PASSED`: + the stable projector now keeps generic SQLite byte-collision fencing strict + while handling one narrower canonical case: the same event ID, partition and + independently recomputed 32-byte market-payload hash may be checkpointed as + an idempotent provenance-only duplicate without a second Redis/stream fan-out. + A changed payload, missing/invalid hash or changed partition remains a hard + `EventIdCollision`. Late historical BARs are durably admitted to the bounded + query history but are not projected over a newer latest BAR. + + The focused stable-edge/deployment/history run executed 46 cases in the + immutable `2041f18` Python test image with network disabled: 45 passed and the + separately proven real-Redis integration case was the sole conditional skip. + Tests explicitly prove provenance-only idempotence, changed-semantics + rejection, late-history repair without latest regression, ACK retry, + incomplete-history fencing, Binance/OKX BAR parity and ordered checkpointing. + No runtime, provider, Kafka, Redis, V1 route or durable volume was mutated by + this unit gate. B.3 remains `IN_PROGRESS` until the committed image passes the + isolated atomic rebuild and strict real-provider consumer acceptance. +- `2026-08-19 PHASE B B18 RUNTIME RETEST FAILED CLOSED AND B19 BOUNDED + REPAIR STARTED`: immutable projector image `8851166` correctly passed the + provenance-only duplicates but the guarded earliest replay exceeded its + 900-second operator timeout at 733,158 records of lag and then exposed real + changed-semantics BAR collisions. Query replicas remained stopped, projector + was stopped after diagnosis, V1 port 8100 remained healthy and no consumer + received candidate data. + + Read-only Kafka/SQLite inspection at the frozen offsets found two distinct + causes. OKX REST/WS BARs were numerically identical but differed only in exact + decimal spelling/scale and acquisition origin; byte/hash equality is therefore + stricter than market-semantic equality. Binance REST BARs differed in actual + close, volume, quote volume and trade count: the earlier `VENUE_NATIVE` row was + observed too close to the minute boundary and a later settled backfill changed + it. The next bounded B19 repair may normalize only BAR decimal values and + acquisition origin for semantic duplicate comparison; every non-BAR payload, + identity, timestamp, interval, lifecycle, unit and actual numeric value remains + strict. The provider edge also gains an aligned close-settlement grace so it + never labels a just-closed mutable row as accepted final data. Changed numeric + BARs continue to fail closed; no revision is fabricated in the projector. + Unit/parity tests precede any new runtime image. Existing isolated Kafka/data + volumes are retained pending an explicit clean-candidate decision. +- `2026-08-19 PHASE B B19 BAR SEMANTICS AND SETTLEMENT UNIT-PASSED`: + the projector now validates each exact Decimal audit spelling against its + coefficient/scale, compares BAR numbers with exact `Decimal` arithmetic and + ignores only acquisition origin plus equivalent trailing-zero spelling for + duplicate classification. Interval, times, OHLCV/base/quote/contract values, + trade count, finality, revision, lifecycle, supersession and quantity unit + remain strict; all non-BAR feeds still require equal canonical payload hashes. + The generic durable spool remains byte-immutable. + + The shared real-provider BAR edge now uses an explicit two-second settlement + grace for bootstrap and latest/catch-up observation and aligns its minute loop + to that delay. The value is frozen in the isolated compose and bounded to + 1-10 seconds. The stable edge/deployment/history suite ran 48 network-disabled + cases: 47 passed and the separately proven real-Redis conditional case was + the sole skip. New cases prove trailing-zero/origin equivalence, changed BAR + numeric rejection and equal settled observation for Binance/OKX. No runtime + was restarted by this gate. A clean isolated runtime data-log decision remains + required because the retained Kafka test log contains previously captured + materially wrong early-final Binance rows and must not be accepted by policy + relaxation. +- `2026-08-19 PHASE B B19 REAL-CAPTURE CLASSIFIER GATE PASSED, RUNTIME + REMAINS FAIL-CLOSED`: immutable image `c61fa39` classified two retained Kafka + records against the read-only SQLite cache exactly as required: the OKX + REST/WS BAR with equivalent numeric Decimals returned `same_market_semantics=true`; + the Binance BAR with changed close/volume/trade count returned `false`. The + diagnostic used the authorized projector identity with manual assignment and + no offset commit. It validates the classifier against durable provider bytes, + not generated market data. + + The old isolated Kafka log is not releasable evidence because it already + contains early-final Binance values from the superseded +1-second edge. No + policy was weakened to ingest them. Python candidate roles were consolidated + onto immutable image ID `b0560895...` and left stopped; Kafka/Rust/Redis and + all volumes were retained for audit. V1 health on port 8100 remained OK. +- `2026-08-19 PHASE B B19 INCREMENTAL ARTIFACT CLEANUP PASSED`: after exact + container-reference checks, cleanup removed only unreferenced Python tags + `e002da6`, `2041f18` and `8851166`. It then removed four QDL BuildKit records + older than one hour (608.7 MB). Image storage fell from 11.1 GB to 9.278 GB; + build-cache storage fell from 9.097 GB to 8.488 GB, for about 2.43 GB total + recovery. The retained artifacts are V1, active candidate `c61fa39`, one + Python/Rust rollback `cfc0246`, Kafka/Redis and all 17 volumes. No broad prune, + volume deletion, topic reset or production mutation occurred. +- `2026-08-20 PHASE B.3 CLEAN RUNTIME CLOSURE RESUMED`: the operator resumed + completion of the remaining B.3 durability/recovery gates before any B.4 + work. The operator explicitly approved stopping only + `qdl_v2_stable_candidate` and deleting its exact four test volumes + `kafka1_data`, `kafka2_data`, `kafka3_data` and `stable_state`; `stable_tls`, + every V1/production container and every production volume remain protected. The rehearsal reuses only the isolated Docker project + `qdl_v2_stable_candidate`; it does not create another release topology or + address V1. The exact reset scope is limited to candidate volumes + `kafka1_data`, `kafka2_data`, `kafka3_data` and `stable_state`, whose retained + records are already classified as invalid release evidence because they + contain superseded early-final Binance BARs. Candidate TLS/credentials are + preserved. V1 containers, Redis, Parquet, provider state, ports, routes and + consumer authority are immutable. + + The closure gate is: fresh real-provider bootstrap under immutable + `c61fa39`; ACK-authoritative BAR catch-up with the fixed two-second settlement + boundary; zero canonical gap/collision/quarantine; atomic Redis-plus-SQLite + rebuild; replica-equal query watermarks; signed SDK warmup -> replay -> LIVE + for registered Binance/OKX bindings; fresh Trading System paper snapshots; + bounded lag/resources; and unchanged V1 health. Any failure stops candidate + readers and leaves V1 authoritative. B.3 remains `IN_PROGRESS` until all + gates and exact cleanup evidence are recorded; B.4 remains forbidden. + + The first reset command was rejected by the safety gate before execution + because exact deletion approval was not yet explicit. Candidate containers, + all five candidate volumes and V1 remain unchanged; no workaround was used. +- `2026-08-20 PHASE B.3 CLEAN BOOTSTRAP PASSED, RECOVERY SAFETY REPAIR IN + PROGRESS`: after exact approval, only the candidate project was stopped and + the three Kafka plus one state volume were deleted; `stable_tls` and V1 were + verified preserved. Fresh RF3/minISR2 Kafka, Redis and state started; broker + bootstrap passed three topics, six partitions, mTLS and ACLs. Read-only TLS + validation found all nine expected files. Real-provider acquisition ACKed 500 + settled BARs for each Binance USD-M, Binance Spot, OKX SWAP and OKX Spot + binding (2,000 total), then ACKed the next closed cycle; Rust workers reported + zero quarantine/collision/error and canonical offsets advanced. + + Before the atomic rebuild, the safety gate identified that Compose dependency + traversal could rerun `stable_tls_init` when starting stream/query roles. It + was blocked before TLS mutation. The bounded repair makes recovery role start + explicit with `up --no-deps` after the project, Kafka, Redis, state and TLS + invariants are validated, and adds a unit test for the exact command. No + public contract, provider semantics, V1 route or authority changes. +- `2026-08-20 PHASE B.3 CLEAN RUNTIME CLOSURE PASSED`: the safety repair passed + 8/8 focused unit tests. The guarded atomic rebuild then deleted only the three + candidate SQLite cache files, flushed only candidate Redis and reset only + `stable-projector-v1` on `md.canonical.v2`. It observed all six partitions + for three consecutive samples within the fixed 250-record gate, with observed + bound 46 and final lag 19, then opened query replicas only after projector and + Redis readiness; Redis rebuilt to 47 keys. + + Signed public SDK acceptance used real-provider data with + `test_provenance=false`. Binance and OKX each returned 500 final 1-minute BARs + with full coverage, no open gap and replica-equal market semantics/watermark + 512. The only per-request replica difference was the expected clock-derived + `quality.freshness_ms`; all identity, timestamps, payload, source, contract + and other quality fields were equal. Both alpha streams emitted + `REPLAYING -> LIVE`, then two strictly contiguous events + (Binance 35752-35753; OKX 13155-13156) were ACKed. Trading System paper read + Binance/OKX TRADE and QUOTE snapshots at 146-316 ms freshness; all four were + execution eligible. + + Read-only cache verification found 73,456 canonical records across 12 bounded + partitions, maximum 10,000 per partition, zero retained offset gaps, zero + duplicate event IDs and zero quarantine rows. Kafka quarantine offsets were + zero on all six partitions; observed projector/core lag totals were 34/12. + Exactly one stream owner was READY and its peer STANDBY. Candidate TLS matched + the preserved source bundle SHA-256, V1 health remained `ok`, Redis used + 3.23 MiB/160 MiB and the largest app role used 69.87 MiB/512 MiB; Kafka + brokers stayed within 461.4 MiB/768 MiB. Logs contained no application + collision, unresolved gap, panic or quarantine; Kafka startup emitted only + benign internal-topic-already-exists warnings. + + The final five-module Phase B regression ran 63 cases: 62 passed and one + separately proven real-Redis conditional case skipped in network-disabled + mode. The acceptance harness itself failed closed before the final pass on + least-privilege env access, an invalid test-only `event_id` assumption, + request-time freshness comparison and expected control-event handling; none + mutated provider/Kafka/Redis data or exposed candidate output to production. + B.3 conclusion is `PASS`/`COMPLETE`. B.4 remains `NOT_STARTED`; no + production cutover or consumer authority migration is authorized. +- `2026-08-20 PHASE B.4 RELEASE CERTIFICATION STARTED`: B.3 is complete and + the operator approved B.4 only. Commit `5054e1e` is the provisional common + source SHA. Required gates are full Python discovery; Rust format, locked + workspace Clippy and tests; Buf format/lint/two-baseline breaking/generation; + OpenAPI semantic compatibility; package/release/security/capacity suites; + immutable Python and Rust images carrying the same source SHA; isolated + candidate rolling recreation; compact evidence; exact unreferenced + image/cache cleanup; and unchanged V1 health/OpenAPI/topology. + + Rollback is to stop only recreated candidate roles and restore pinned + `c61fa39` Python plus `cfc0246` Rust images against preserved candidate + Kafka/state/TLS. Public V1, provider data, production Redis/Parquet, routes and + consumer authority are immutable. A failed gate leaves B.4 in progress and + cannot be relabeled as technical debt. Push, merge, release publication and + production cutover remain outside scope. +- `2026-08-20 PHASE B.4 PYTHON CERTIFICATION PASSED`: immutable + `c61fa39` dependencies with source bind at `5054e1e` ran full unittest + discovery in a network-disabled, read-only, cap-dropped container bounded to + 768 MiB, 1.5 CPU and 256 PIDs. It executed 503 tests: 497 passed and six + explicit conditional/infrastructure cases skipped. Failure-path ERROR and + CRITICAL logs were injected assertions and their tests passed. No candidate, + V1, provider, Kafka, Redis or volume state was addressed. Rust and contract + gates remain pending; B.4 stays `IN_PROGRESS`. +- `2026-08-20 PHASE B.4 RUST PACKAGING DEFECT FOUND, BOUNDED FIX STARTED`: + the exact-SHA Rust builder compiled all release binaries, but full locked + workspace tests failed at compile time because `Dockerfile.phase8-rust` + copied Rust sources/generated bindings without the immutable + `contracts/golden` and `tests/fixtures/phase2` oracle files referenced by + Rust tests. No runtime or domain assertion failed, and no candidate was + recreated. The in-scope fix copies only those two bounded test inputs into + the builder stage and adds release-packaging assertions. Runtime stage, + binaries, contracts and provider behavior remain unchanged. Rust fmt/Clippy/ + tests must be rerun from a rebuilt builder before this gate can pass. +- `2026-08-20 PHASE B.4 RUST CERTIFICATION PASSED`: the rebuilt bounded + builder included only the two missing immutable oracle directories. + `cargo fmt --all -- --check`, locked workspace Clippy with + `-D warnings`, and the full workspace test gate passed under no-network, + cap-dropped, no-new-privileges execution bounded to 3 CPU, 3 GiB and 512 + PIDs. All 62 Rust tests passed with zero failures/skips, covering exact + Python/Rust golden bytes, Binance/OKX provider parsing, quantity/decimal + identity, generation fencing, replay/dedup/gap/quarantine, final BAR + semantics, Kafka TLS/transaction headers, authority handoff/rollback, + backpressure classes and VN source semantics. The targeted Python packaging + regression also passed 6/6 and `git diff --check` was clean. No candidate + or V1 runtime was mutated. Contract/security/package/capacity gates remain. +- `2026-08-20 PHASE B.4 CONTRACT CERTIFICATION PASSED`: Buf 1.50.0 + format, lint, breaking checks against both frozen Phase 1 and Phase 7 + baselines, and generation all passed; regenerated Python/Rust artifacts had + zero Git diff. Seven generated-contract/golden tests passed in a read-only, + no-network Python container. OpenAPI semantic comparison against `dev` + reported `PASS_PRE_BETA_FREEZE`, 10 operations, 42 schemas and zero removed + operation/response/schema/enum or security/required-parameter change. + Candidate/V1 runtime and durable state were not addressed. Security, + package, capacity and final one-SHA image gates remain. +- `2026-08-20 PHASE B.4 CAPACITY DIAGNOSTIC RECORDED`: the approved + Phase 2 durability gate passed at 3,042 append events/s, p99 14.4 ms, + 22,688 replay events/s and 6.03x disk amplification; the eight-replica V2 + API gate passed at 706 requests/s and p99 181 ms with zero venue connection. + A separate exploratory Phase 6 run requested 10,000 normal / 40,000 burst + events/s while cgroup-limited to 2 CPU and failed closed at 5,656 events/s. + Those rates are not the script's approved 500/1,500 certification defaults, + so this is retained as non-release diagnostic evidence rather than a code + defect or a lowered threshold. The approved bounded profile and final Rust + release benchmark must still pass. No runtime/data was mutated. +- `2026-08-20 PHASE B.4 RUST SUPPLY-CHAIN POLICY DEFECT FOUND`: + checksum-verified cargo-deny 0.20.2 fetched the current RustSec database and + reported advisories, bans and sources clean, but license validation failed + because the Rust builder did not copy the repository's existing `deny.toml`. + Cargo-deny inside that artifact therefore used its default deny-all license + policy and rejected normal MIT/Apache/BSD dependencies; host CI still had + the tracked policy. The bounded repair copies the reviewed policy into the + builder, makes its Linux target/advisory/license/source boundaries explicit, + and adds release regression assertions. No dependency, runtime or market + semantics change. +- `2026-08-20 PHASE B.4 CAPACITY AND SECURITY GATES PASSED`: the + approved 80-partition Phase 6 profile passed two normal windows at + 503.62/503.70 events/s and burst at 1,503.07 events/s, p99.9 128.055 ms, + zero queue rejection/replay mismatch and negative measured memory growth. + The reviewed `deny.toml` targets Linux production, has no advisory/license + exception, denies wildcard dependencies and unknown registry/Git sources, + and permits only the encountered permissive licenses. Checksum-pinned + cargo-deny 0.20.2 passed advisories, bans, licenses and sources; duplicate + transitive versions remained non-blocking graph warnings. Pip-audit reported + no known Python vulnerability. Pinned Trivy 0.73.0 source/config scanning, + excluding only runtime data/logs and compiled target artifacts, found zero + HIGH/CRITICAL misconfiguration and zero secret across 15 analyzed targets. + Seven release-policy/package tests passed and `git diff --check` remained + clean. Final full regression, one-SHA images, image scans and candidate + recreation remain. +- `2026-08-20 PHASE B.4 FINAL SOURCE REGRESSION PASSED`: an initial + read-only rerun omitted the required `/app/logs` tmpfs and stopped four + import modules at `RotatingFileHandler`; 495 other tests ran and no domain + assertion failed. With the documented non-root writable log tmpfs restored, + full network-disabled/read-only discovery passed 504 tests: 498 passed and + six explicit conditional/infrastructure cases skipped. Test-injected timeout, + stale-source, queue-fence and recovery logs remained expected assertions. + This closes source-level regression after the Docker packaging and supply- + chain policy repairs. The next gate is freezing the source commit and building + both immutable images from that exact SHA. +- `2026-08-20 PHASE B.4 IMMUTABLE ARTIFACT GATE PASSED`: source + commit `ea84a21be71572674cc5b160788d8edd0f870738` produced Python image + `sha256:00ffbd5b...` and Rust image `sha256:b464f342...`; both carry + exact OCI revision/version and run non-root. Trivy found zero + HIGH/CRITICAL vulnerability and zero embedded secret in either image. + Final-SHA Rust processed 100,000 events at 129,256 events/s, p99 12.906 + microseconds, zero duplicate and zero quarantine against the 50,000/s gate. + The final builder contained its policy and cargo-deny again passed all four + checks. +- `2026-08-20 PHASE B.4 CANDIDATE ROLLING GATE FAILED CLOSED`: only 13 + isolated application roles were recreated one at a time with `--no-deps`; + Kafka, Redis, TLS/state volumes and V1 were preserved. Both query replicas + became READY and exactly one stream peer was READY while the other was + STANDBY, but the projector then repeatedly rejected + `canonical event ID maps to different market semantics`. Two Rust workers + reported 497 quarantines after restart. This violates the zero-collision/ + zero-quarantine release invariant even though V1 remained `ok`. B.4 stays + `IN_PROGRESS`; candidate readers/workers must be stopped fail-closed, + durable evidence inspected and the root cause fixed or the pinned + `c61fa39`/`cfc0246` rollback restored before any acceptance claim. +- `2026-08-20 PHASE B.4 REAL-PROVIDER ROOT CAUSE AND REPAIR BOUNDARY`: + read-only Kafka/projector inspection decoded 994 committed quarantine records + as OKX `candle1m` `STALE_GENERATION`; the REST BAR edge reused generation + 1 while the native WebSocket owner had advanced the same canonical partition. + At the projector checkpoint, repeated Binance REST warmup also reused an + immutable revision-0 event ID for materially changed close/volume/trade-count + values. The projector and Rust fencing behaved correctly and remain strict. + The bounded repair makes closed 1m BAR acquisition single-owner REST for both + Binance and OKX while Rust remains the sole canonical core, persists an + authority/catalog-bound last-ACKed BAR watermark atomically in + `stable_state`, skips overlapping bootstrap after restart, and uses the + approved 10-second settlement ceiling. Corrupt/mismatched state, incomplete + history, changed immutable BAR semantics, partial Kafka ACK and stale + authority all continue to fail closed. Public V1/V2 contracts, event identity, + query semantics, V1 port 8100 and production authority do not change. Gates: + restart/corrupt-state/ACK-loss tests, native subscription manifest proof, + Python/Rust full regression, then a clean isolated real-provider rehearsal + with zero gap/collision/quarantine and restart continuity. +- `2026-08-20 PHASE B.4 BAR OWNERSHIP/CHECKPOINT REPAIR UNIT-PASSED`: + acquisition manifest revision 2 assigns all four Binance/OKX final 1m BAR + bindings to the single Python REST edge; the Rust native ingestors retain + eight lossless/latest-state TRADE/QUOTE bindings and Rust remains the only + canonical core. The edge now persists each last fully Kafka-ACKed open time + by atomic write/fsync/rename in isolated `stable_state`, restores only an + exact slice/authority/catalog/acquisition match, resumes partial bootstrap, + rejects corrupt/future watermarks and never advances on partial ACK. Compose + mounts the state volume, waits for its initializer and uses the bounded + 10-second settlement ceiling. Public APIs/event IDs/projector fencing are + unchanged. Syntax and compose validation passed; 18 targeted tests passed, + then all five Phase B modules ran 65 cases: 64 passed and the separately + proven real-Redis integration case was the sole skip. No app role, Kafka, + Redis, volume, V1 route or provider state was mutated by this test slice. + Full Python/Rust regression, immutable same-SHA artifacts and clean isolated + real-provider restart acceptance remain. +- `2026-08-20 PHASE B.4 REPAIR FULL SOURCE GATE PASSED`: two initial full + Python invocations reached 497 tests but four import modules could not open + `/app/logs/app.log` because the child tmpfs was root-owned; no domain + assertion failed. With the same source and a bounded log tmpfs owned by + non-root UID/GID 10001, full discovery ran 506 tests: 500 passed and six + explicit conditional/infrastructure cases skipped. Rust format, locked + workspace Clippy with warnings denied and all 62 workspace tests passed in + the exact builder with network disabled. Compose rendering and + `git diff --check` also passed. Test-injected timeout, queue-fence, + checkpoint and stale-source logs were expected assertions. No candidate/V1 + process or durable state was mutated. Freeze a final journal commit, build + both images from that one SHA, rescan and run isolated real-provider restart + acceptance next. +- `2026-08-20 PHASE B.4 FINAL ARTIFACT/RUNTIME GATES PASSED`: final code + commit `2412572eaa89864ce74910b0f2e5f8b50833fb15` produced Python image + `sha256:fec269ec555624baa68ee15fdd0281d72996e55f847b7347856be6b2fa51ea25` + and Rust image + `sha256:fbff0ed3c4390831a2aebf12f57c266eb6f01dde258b3cffacacbcbaa30d6c97`. + Both are non-root and carry the exact OCI revision/version. Pinned Trivy + 0.73.0 under the repository CI policy found zero fixable HIGH/CRITICAL + vulnerability and zero secret in both. The stricter no-ignore diagnostic + exposed only currently unfixed Debian findings and was retained as diagnostic + rather than hidden. The final Rust benchmark processed 100,000 events at + 133,477.5 events/s, p99 14,124 ns, zero duplicate/quarantine against the + 50,000/s gate; 25/25 final package/deployment tests passed. + + A fresh isolated RF3/minISR2 candidate bootstrapped 500 authentic closed 1m + BARs for each Binance USD-M, Binance Spot, OKX SWAP and OKX Spot binding. + Restart restored the revision-2 ACK checkpoint, skipped overlapping history + and caught up the exact closed-bar backlog. Cache inspection found 75,187 + canonical records across 12 bounded partitions, maximum 10,000 each, zero + offset gap, duplicate event ID or quarantine; Kafka quarantine offsets were + zero and projector lag was 35 under the 250 gate. Redis used 1.22 MiB/128 + MiB with 51 keys; app and broker roles stayed inside all configured bounds; + application logs had no warning/error/collision/gap during acceptance. +- `2026-08-20 PHASE B.4 FINAL CONSUMER ACCEPTANCE PASSED`: signed released + SDK clients read 500 final real-provider Binance and OKX BARs with replica- + equal market semantics. Both alpha streams reached `REPLAYING -> LIVE`, + ACKed contiguous events and resumed from durable cursor at exactly the prior + offset + 1. Trading System paper read authoritative/execution-eligible + Binance and OKX TRADE/QUOTE snapshots at 132-158 ms freshness; monitoring + reads were authoritative at 100-180 ms. No order, synthetic data or + production mutation occurred. DNSE remains the already recorded official- + provider external gate, so Phase B overall is `PARTIAL_EXTERNAL`; it does not + invalidate B.4 artifact certification or permit cutover. +- `2026-08-20 PHASE B.4 CLEANUP AND CLOSURE PASSED`: removed the fresh + `qdl_v2_b4_candidate` project and all five disposable volumes; removed the + stopped old candidate containers/networks and only its four approved Kafka/ + state test volumes while preserving `qdl_v2_stable_candidate_stable_tls`. + Removed three builder tags, superseded `ea84a21` Python/Rust tags and the + unused Python `cfc0246` tag. Retained final `2412572`, V1 and the tested + Python `c61fa39`/Rust `cfc0246` rollback pair. Exact-ID pruning of 41 B.4 + BuildKit records reduced cache from 168/12.94 GB to 154/10.94 GB; no broad + prune ran. The exact `/tmp` secret bundle, scan output and SDK harness were + deleted after bounded evidence was recorded. V1 was never restarted and remained `status=ok`, Redis true, + recent queue drops zero and DNSE `OPEN_HEALTHY`. Full evidence is frozen in + `upgrade/evidence/PHASE_B4_RELEASE_CERTIFICATION_REPORT.md`. B.4 conclusion + is `PASS`/`COMPLETE`; push, merge, release publication and authority/consumer + cutover remain unapproved. +- `2026-08-19 PHASE B ARTIFACT CLEANUP POLICY RECORDED`: Phase B ends at B.4; + B17/B18 are repair slices inside B.3, not new subphases. Exact cleanup retains + V1, active `e002da6`, active/rollback `cfc0246`, Kafka/Redis and all durable + volumes. Obsolete unreferenced QDL image tags and the two named test-builder + containers are removed after reference validation; shared host-wide prune is + forbidden. Final BuildKit cleanup is scoped to Data Layer build records and + recorded with before/after bytes. +- `2026-08-19 PHASE B INCREMENTAL ARTIFACT CLEANUP PASSED`: reference-aware + cleanup removed exactly two disposable test-builder containers and 47 obsolete + unreferenced QDL image tags. It retained `data-layer:v0.1.0`, Python candidate + `2.0.0-e002da6`, Python/Rust runtime rollback `2.0.0-cfc0246`, every running + Kafka/Redis image and all 17 volumes. Image cleanup reduced root filesystem + use from 91 GiB to 73 GiB. BuildKit cleanup was limited to records matching + `description~=qdl` and older than one hour; cache fell from 50.2 GB to 6.484 + GB and root filesystem use fell again to 47 GiB. Total host space recovered + was about 44 GiB. No host-wide prune ran, candidate/V1 containers remained + running and no data volume was deleted. Post-cleanup B16/B17 regression + ran 49 tests in the immutable `e002da6` Python image with network disabled: + 48 passed and the separately proven real-Redis conditional case was the sole + skip; Python compile and `git diff --check` passed. +- `RUNTIME UNCHANGED`: port 8100 still serves V1 from the existing container; + no restart, authority mutation or consumer migration has occurred. + +### Phase C - Production V2 And Rust Authority Cutover + +**Status:** `C.0 RELEASE/MERGE PREPARATION IN PROGRESS / PRODUCTION CUTOVER NOT AUTHORIZED` + +**Purpose:** move approved Binance and OKX feed slices from the current V1 +authority to the stable V2 contract with Rust as the actual canonical realtime +authority and Python retained as the API/SDK/history/control/projector edge. +DNSE remains a declared external debt and is disabled from initial production +promotion until its provider gates pass. + +**2026-08-20 pre-cutover audit facts:** + +- public port `8100` currently serves `data-layer:v0.1.0`; OpenAPI exposes 40 + V1 paths and zero V2 paths; +- no V2 stable container is running; only the retained + `qdl_v2_stable_candidate_stable_tls` volume remains; +- the feature branch is 81 commits ahead of `dev`; the latest released + `2.0.0-2412572` images predate the bounded DNSE closure commits; +- the stable compose and realtime binaries deliberately accept only + `RUST_SHADOW`; Phase 9.2 proves the CAS/handoff/fencing behavior in an + isolated rehearsal but is not wired into the long-running stable runtime; +- therefore changing an environment variable or routing consumers directly to + the current candidate would be an invalid cutover and could create ambiguous + writer authority. + +The operator procedure and merge/cutover commands are frozen in +`docs/runbooks/v2-production-rust-authority-cutover.md`. + +#### C.0 Release Branch Closure And Production-Authority Design + +**Goal:** merge the already-certified V2 implementation through `dev`, then +implement the missing long-running authority wiring on a new feature branch +without mutating V1. + +**Required work:** + +1. Fix plan/document drift, run CI-equivalent source/contract/security suites, + push the current feature branch and merge it to `dev` only after CI passes. + Do not merge `main` and do not deploy from an unmerged worktree. +2. Create `feat/v2-production-authority-cutover` from updated `dev`. +3. Connect the Phase 9 persistent PostgreSQL CAS and immutable audit/handoff + records to a transactional authority outbox and compacted Kafka authority + topic. A database transition and its outbox record are one transaction; + retries are idempotent and stale revision/owner/lease/plan fail closed. +4. Make the long-running Rust canonical sink consume and reconstruct that + authority stream, use the existing Phase 9.2 fence at every durable target, + and refuse canonical/public/compatibility writes until its exact authority + and target watermarks are reconstructed. +5. Keep acquisition separate from publication authority. Binance/OKX native + acquisition may publish authenticated raw events only under the current + slice lease; Python adapters may never bypass the Rust canonical core. +6. Add an operator CLI for preflight, transition, fence, rollback and status. + The CLI prints identities/revisions/watermarks/digests only, never secrets. +7. Build one immutable Python/Rust image pair from the merged SHA, generate + SBOM/provenance, and retain exactly one tested V1 rollback generation. + +**C.0 gates:** migration idempotency, transactional outbox replay, compacted +authority recovery, stale-writer rejection, restart recovery, exact Python/Rust +parity, public V1 contract compatibility, full source/Clippy/security tests and +zero production mutation. Conclusion must be either `PASS` or `FAIL`; missing +authority wiring cannot be deferred as operational debt. + +**C.0 implementation journal:** + +- `2026-08-20 RELEASE/CUTOVER PREPARATION RECORDED`: corrected the malformed + `RUNTIME UNCHANGED` journal line and added the production cutover boundary + plus `docs/runbooks/v2-production-rust-authority-cutover.md`. Read-only + runtime inspection proved V1 `0.1.0` still owns port `8100` with 40 V1 + paths and zero V2 paths, no V2 containers are running, and the current stable + binaries/config are intentionally shadow-only. The branch is 81 commits ahead + of `dev`; it must merge through CI before a new authority feature branch is + created. +- Documentation whitespace and secret scans passed; stable compose rendered + successfully with isolated dummy values and no container start. Host + preflight observed 11 GiB available RAM, 108 GiB free disk and eight CPUs. + No image build, provider call, service restart, authority mutation, consumer + route change, topic/Redis write, volume deletion, push or merge occurred. + C.0 remains `IN_PROGRESS` until the current PR is CI-green and merged to + `dev`; production authority wiring starts only on the new branch named in + this phase. Preparation commit: `130da39`. + +#### C.1 Isolated Stable V2 Deployment + +Deploy the immutable pair under project `qdl_v2_stable_candidate`, loopback +ports `18201/18202/18210/18211/18220/18221`, dedicated RF3/minISR2 Kafka, +dedicated Redis prefix/cache/state and unique credentials. Start in +`RUST_SHADOW`; exclude the `stable-vn` profile. V1 port `8100`, current +Redis, provider sockets and consumer routes remain unchanged. + +Require real Binance/OKX warmup and stream data, zero unexplained gap/duplicate/ +quarantine, bounded lag/resources, broker and process restart recovery, exact +cursor continuation, V1 health unchanged and exact disposable cleanup on +failure. + +#### C.2 Controlled Consumer Canary + +Migrate in order: monitoring, one paper alpha, Trading System paper adapter, +then remaining approved paper consumers. Each manifest performs +warmup -> signed cursor -> replay -> live and has an exact V1 rollback route. +No sandbox/live order consumer is included. A stale, gapped, non-authoritative +or session-invalid read blocks execution. + +#### C.3 Exact-Slice Rust Authority Promotion + +Promotion is one slice at a time, initially one Binance or OKX TRADE slice. +The approval packet must name image IDs, slice/binding, old/new owner, +authority revision, lease/plan epoch, terminal watermark `W`, topics/groups, +ports, volumes, credentials by secret reference, affected consumers, hold +duration and rollback command. + +The only allowed sequence is: + +`PYTHON_PRIMARY -> RUST_SHADOW -> RUST_CANARY -> RUST_PRIMARY`. + +Fence the old writer, persist its terminal checkpoint, accept the handoff, +execute the CAS/outbox transition, reconstruct every target through `W`, and +publish first as Rust at `W+1`. Any ambiguity, missing ACK, parity mismatch, +lag/gap, stale CAS or consumer failure enters `BLOCKED` and rolls back under a +newer revision; never restart V1 as an uncoordinated writer. + +#### C.4 Hold, Expand And Release + +Hold the first primary slice for the approved observation window with zero +authority ambiguity or unexplained market-data mismatch. Expand independently +by venue/feed manifest; no slice inherits certification. Only after all +registered consumers use V2 may the operator approve routing the stable public +endpoint and opening a V1 sunset window. DNSE remains disabled until its +separate provider gate passes. + +**Decision boundary:** C.0 code/release preparation and C.1 isolated deployment +are non-production-authority work. C.2 changes only explicitly named paper +consumer routes. C.3 and C.4 require a separate operator approval containing +the exact packet above. No command in this plan implicitly authorizes a restart, +authority mutation, consumer cutover, volume deletion or V1 shutdown. + +### Rollback + +Before runtime cutover, remove only isolated Rust/Kafka/V2 test resources. +After an approved cutover, fence the selected Rust slice, restore the matching +Python rollback manifest under a newer authority revision/lease, replay from the +last common durable watermark and leave all unrelated venue/feed slices +untouched. diff --git a/Dockerfile b/Dockerfile index a1e2f61..dc6654c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -21,6 +21,13 @@ FROM python:3.12-slim AS runtime ARG QDL_UID=10001 ARG QDL_GID=10001 +ARG QDL_GIT_SHA=unknown +ARG QDL_RELEASE=development + +LABEL org.opencontainers.image.title="Quant Data Layer" \ + org.opencontainers.image.revision="${QDL_GIT_SHA}" \ + org.opencontainers.image.version="${QDL_RELEASE}" \ + org.opencontainers.image.source="https://github.com/BobbyAxerol/quant-data-layer" ENV PYTHONUNBUFFERED=1 ENV PYTHONDONTWRITEBYTECODE=1 @@ -33,7 +40,10 @@ ENV PATH=/opt/venv/bin:$PATH WORKDIR /app -RUN groupadd --gid ${QDL_GID} qdl && \ +RUN apt-get update && \ + apt-get upgrade -y --no-install-recommends && \ + rm -rf /var/lib/apt/lists/* && \ + groupadd --gid ${QDL_GID} qdl && \ useradd --uid ${QDL_UID} --gid ${QDL_GID} --create-home \ --home-dir /home/qdl --shell /usr/sbin/nologin qdl && \ install -d -o qdl -g qdl -m 0750 /home/qdl/.cache/matplotlib diff --git a/Dockerfile.phase8-rust b/Dockerfile.phase8-rust index 748d285..25b8bad 100644 --- a/Dockerfile.phase8-rust +++ b/Dockerfile.phase8-rust @@ -5,14 +5,23 @@ RUN apt-get update \ clang cmake libclang-dev libcurl4-openssl-dev libssl-dev libzstd-dev \ make pkg-config zlib1g-dev \ && rm -rf /var/lib/apt/lists/* +RUN rustup component add rustfmt clippy WORKDIR /src COPY Cargo.toml Cargo.lock ./ +COPY deny.toml ./deny.toml COPY rust ./rust COPY generated/rust ./generated/rust +# Rust unit/parity tests compile against immutable provider and contract oracles. +COPY contracts/golden ./contracts/golden +COPY tests/fixtures/phase2 ./tests/fixtures/phase2 RUN cargo build --release --locked \ --bin qdl-kafka-smoke \ --bin qdl-authority-rehearsal \ + --bin qdl-phase91-canary-rehearsal \ + --bin qdl-phase92-primary-rehearsal \ --bin qdl-binance-shadow \ + --bin qdl-native-raw-ingestor \ + --bin qdl-realtime-core \ --bin qdl-parity-replay \ --bin qdl-venue-core-certify @@ -30,7 +39,11 @@ RUN apt-get update \ && useradd --uid 10001 --gid 10001 --no-create-home --shell /usr/sbin/nologin qdl COPY --from=builder /src/target/release/qdl-kafka-smoke /usr/local/bin/qdl-kafka-smoke COPY --from=builder /src/target/release/qdl-authority-rehearsal /usr/local/bin/qdl-authority-rehearsal +COPY --from=builder /src/target/release/qdl-phase91-canary-rehearsal /usr/local/bin/qdl-phase91-canary-rehearsal +COPY --from=builder /src/target/release/qdl-phase92-primary-rehearsal /usr/local/bin/qdl-phase92-primary-rehearsal COPY --from=builder /src/target/release/qdl-binance-shadow /usr/local/bin/qdl-binance-shadow +COPY --from=builder /src/target/release/qdl-native-raw-ingestor /usr/local/bin/qdl-native-raw-ingestor +COPY --from=builder /src/target/release/qdl-realtime-core /usr/local/bin/qdl-realtime-core COPY --from=builder /src/target/release/qdl-parity-replay /usr/local/bin/qdl-parity-replay COPY --from=builder /src/target/release/qdl-venue-core-certify /usr/local/bin/qdl-venue-core-certify USER 10001:10001 diff --git a/Makefile b/Makefile index 3d4b73a..00ce232 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: contract-check contract-generate phase2-benchmark phase2-redis-smoke phase2-test phase3-lease-smoke phase3-load-smoke phase3-real-provider-smoke phase3-rust-smoke phase3-test phase4-dnse-real-smoke phase4-history-test phase4-migration-smoke phase4-okx-real-smoke phase4-okx-test phase4-replay-test phase4-test phase4-vn-shadow-smoke phase45-build phase45-clean phase45-dependency-audit phase45-provider-smoke phase45-test phase5-api-test phase5-build phase5-clean phase5-contract-check phase5-dependency-audit phase5-load phase5-migration-smoke phase5-real-provider-smoke phase5-test phase7-build phase7-clean phase7-contract-check phase7-migration-smoke phase7-test phase71-topology-test phase71-test phase72-test phase72-topology-test phase73-test phase73-certify phase80-test phase80-certify phase81-test phase81-certify phase82-test phase82-dnse-acquire phase82-certify phase83-test phase83-build phase83-authority phase83-release-capacity phase83-freeze python-test rust-test +.PHONY: contract-check contract-generate phase2-benchmark phase2-redis-smoke phase2-test phase3-lease-smoke phase3-load-smoke phase3-real-provider-smoke phase3-rust-smoke phase3-test phase4-dnse-real-smoke phase4-history-test phase4-migration-smoke phase4-okx-real-smoke phase4-okx-test phase4-replay-test phase4-test phase4-vn-shadow-smoke phase45-build phase45-clean phase45-dependency-audit phase45-provider-smoke phase45-test phase5-api-test phase5-build phase5-clean phase5-contract-check phase5-dependency-audit phase5-load phase5-migration-smoke phase5-real-provider-smoke phase5-test phase7-build phase7-clean phase7-contract-check phase7-migration-smoke phase7-test phase71-topology-test phase71-test phase72-test phase72-topology-test phase73-test phase73-certify phase80-test phase80-certify phase81-test phase81-certify phase82-test phase82-dnse-acquire phase82-certify phase83-test phase83-build phase83-authority phase83-release-capacity phase83-freeze phase90b-build phase90b-test phase90b-certify phase90b-clean phase90c-build phase90c-test phase90c-migration phase90c-certify phase90c-clean python-test rust-test phase91-build phase91-test phase91-certify phase91-clean phase92-build phase92-test phase92-migration phase92-certify phase92-clean phase93-test phase93-migration phase93-certify phase93-clean BUF_IMAGE ?= bufbuild/buf:1.50.0 RUST_IMAGE ?= rust:1.82-slim@sha256:1111c28d995d06a7863ba6cea3b3dcb87bebe65af8ec5517caaf2c8c26f38010 @@ -8,6 +8,18 @@ PHASE7_TEST_IMAGE ?= data-layer:phase7-test PHASE8_RUST_IMAGE ?= qdl-phase8-rust:phase8-candidate PHASE8_RELEASE ?= phase8-rust-realtime-core-v0.1.0-beta PHASE8_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE90B_IMAGE ?= data-layer:phase90b-candidate +PHASE90B_RELEASE ?= phase90b-isolated-v2-beta +PHASE90B_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE90C_IMAGE ?= data-layer:phase90c-test +PHASE90C_RELEASE ?= phase90c-production-prerequisites +PHASE90C_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE91_RUST_IMAGE ?= qdl-phase91-rust:rehearsal +PHASE91_BUILDER_IMAGE ?= qdl-phase91-rust-builder:rehearsal +PHASE91_GIT_SHA ?= $(shell git rev-parse HEAD) +PHASE92_RUST_IMAGE ?= qdl-phase92-rust:rehearsal +PHASE92_BUILDER_IMAGE ?= qdl-phase92-rust-builder:rehearsal +PHASE92_GIT_SHA ?= $(shell git rev-parse HEAD) contract-generate: docker run --rm -v "$(CURDIR):/workspace" -w /workspace/contracts $(BUF_IMAGE) generate @@ -210,3 +222,92 @@ phase83-freeze: phase7-clean: docker image rm $(PHASE7_TEST_IMAGE) 2>/dev/null || true + +phase90b-build: + docker build --provenance=false --build-arg QDL_GIT_SHA=$(PHASE90B_GIT_SHA) --build-arg QDL_RELEASE=$(PHASE90B_RELEASE) -t $(PHASE90B_IMAGE) . + +phase90b-test: phase90b-build + docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.5 --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=128m,uid=10001,gid=10001 --tmpfs /app/logs:rw,noexec,nosuid,nodev,size=16m,uid=10001,gid=10001 $(PHASE90B_IMAGE) python -m unittest -v tests.test_phase90b_isolated_beta tests.test_phase90a_runtime_correctness tests.test_fund_phase73_beta_decision tests.test_fund_phase72_consumer_canary tests.test_fund_phase71_beta_runtime tests.test_fund_phase7_contract_security tests.test_phase0_contract_golden tests.test_fund_phase5_contracts + docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.5 --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=128m,uid=10001,gid=10001 --tmpfs /app/logs:rw,noexec,nosuid,nodev,size=16m,uid=10001,gid=10001 $(PHASE90B_IMAGE) python -m unittest discover -s tests + +phase90b-certify: phase90b-test + docker image inspect redis:7.2-alpine >/dev/null 2>&1 || docker pull redis:7.2-alpine + QDL_PHASE90B_IMAGE=$(PHASE90B_IMAGE) scripts/phase90b_isolated_beta_certification.sh + +phase90b-clean: + docker image rm $(PHASE90B_IMAGE) 2>/dev/null || true + +phase90c-build: + docker build --provenance=false --build-arg QDL_GIT_SHA=$(PHASE90C_GIT_SHA) --build-arg QDL_RELEASE=$(PHASE90C_RELEASE) -t $(PHASE90C_IMAGE) . + +phase90c-test: phase90c-build + docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.5 --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=128m,uid=10001,gid=10001 --tmpfs /app/logs:rw,noexec,nosuid,nodev,size=16m,uid=10001,gid=10001 $(PHASE90C_IMAGE) python -m unittest -v tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract tests.test_phase90b_isolated_beta tests.test_fund_phase80_broker_substrate tests.test_fund_phase83_release + +phase90c-migration: + scripts/phase90c_migration_smoke.sh + +phase90c-certify: phase90c-test phase90c-migration + python3 scripts/phase90c_prerequisite_certification.py --expect NO_GO_EXTERNAL + sha256sum upgrade/evidence/phase90c-production-prerequisites.json upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md upgrade/evidence/phase90c-authority-migration.json > upgrade/evidence/phase90c-evidence.sha256 + +phase90c-clean: + docker image rm $(PHASE90C_IMAGE) 2>/dev/null || true + + +phase91-build: + docker build --provenance=false --target builder -f Dockerfile.phase8-rust -t $(PHASE91_BUILDER_IMAGE) . + docker build --provenance=false -f Dockerfile.phase8-rust --build-arg QDL_GIT_SHA=$(PHASE91_GIT_SHA) --build-arg QDL_RELEASE=phase91-rust-canary-rehearsal -t $(PHASE91_RUST_IMAGE) . + +phase91-test: phase91-build + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE91_BUILDER_IMAGE) cargo fmt --all -- --check + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE91_BUILDER_IMAGE) cargo clippy --workspace --all-targets --locked -- -D warnings + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE91_BUILDER_IMAGE) cargo test --workspace --locked + docker compose run --rm --no-deps test_runner python -m unittest tests.test_phase91_canary_control tests.test_phase91_certification_contract tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract tests.test_phase90b_isolated_beta tests.test_phase90a_runtime_correctness tests.test_fund_phase80_broker_substrate tests.test_fund_phase81_raw_core tests.test_fund_phase82_conformance tests.test_fund_phase83_release + docker compose run --rm --no-deps test_runner python -m unittest discover tests + +phase91-certify: phase91-test + python3 scripts/phase91_canary_certification.py --rust-image $(PHASE91_RUST_IMAGE) --repeat 200 + sha256sum -c upgrade/evidence/phase91-evidence.sha256 + +phase91-clean: + docker compose --project-name qdl_phase91_certification -f docker-compose.phase8-kafka.yml down -v --remove-orphans + docker image rm $(PHASE91_RUST_IMAGE) 2>/dev/null || true + docker image rm $(PHASE91_BUILDER_IMAGE) 2>/dev/null || true + +phase92-build: + docker build --provenance=false --target builder -f Dockerfile.phase8-rust -t $(PHASE92_BUILDER_IMAGE) . + docker build --provenance=false -f Dockerfile.phase8-rust --build-arg QDL_GIT_SHA=$(PHASE92_GIT_SHA) --build-arg QDL_RELEASE=phase92-bounded-primary-rehearsal -t $(PHASE92_RUST_IMAGE) . + +phase92-test: phase92-build + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE92_BUILDER_IMAGE) cargo fmt --all -- --check + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE92_BUILDER_IMAGE) cargo clippy --workspace --all-targets --locked -- -D warnings + docker run --rm --mount type=bind,source=$(CURDIR),target=/src --workdir /src/rust $(PHASE92_BUILDER_IMAGE) cargo test --workspace --locked + docker compose run --rm --no-deps test_runner python -m unittest tests.test_phase92_primary_control tests.test_phase92_migration_contract tests.test_phase92_certification_contract tests.test_phase91_canary_control tests.test_phase91_certification_contract tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract tests.test_phase90b_isolated_beta tests.test_phase90a_runtime_correctness tests.test_fund_phase80_broker_substrate tests.test_fund_phase81_raw_core tests.test_fund_phase82_conformance tests.test_fund_phase83_release + docker compose run --rm --no-deps test_runner python -m unittest discover tests + +phase92-migration: + scripts/phase92_migration_smoke.sh + +phase92-certify: phase92-test phase92-migration + python3 scripts/phase92_primary_certification.py --rust-image $(PHASE92_RUST_IMAGE) --repeat 200 + sha256sum -c upgrade/evidence/phase92-evidence.sha256 + +phase92-clean: + docker compose --project-name qdl_phase92_certification -f docker-compose.phase8-kafka.yml down -v --remove-orphans + docker image rm $(PHASE92_RUST_IMAGE) 2>/dev/null || true + docker image rm $(PHASE92_BUILDER_IMAGE) 2>/dev/null || true + + +phase93-test: + docker compose run --rm --no-deps test_runner python -m unittest tests.test_phase93_hold_close_expand tests.test_phase93_migration_contract tests.test_phase93_certification_contract tests.test_phase92_primary_control tests.test_phase92_migration_contract tests.test_phase92_certification_contract tests.test_phase91_canary_control tests.test_phase90c_prerequisites tests.test_phase90c_migration_contract + docker compose run --rm --no-deps test_runner python -m unittest discover tests + +phase93-migration: + python3 scripts/phase93_migration_smoke.py + +phase93-certify: phase93-test phase93-migration + python3 scripts/phase93_hold_close_certification.py + sha256sum -c upgrade/evidence/phase93-evidence.sha256 + +phase93-clean: + docker ps -aq --filter "name=qdl_phase93_" | xargs -r docker rm -f diff --git a/README.md b/README.md index e05aa5c..433b067 100644 --- a/README.md +++ b/README.md @@ -178,6 +178,9 @@ Edit `.env` and fill in your credentials: # Required for VN stock live data (DNSE) DNSE_API_KEY=your_dnse_api_key DNSE_API_SECRET_KEY=your_dnse_secret +DNSE_API_VERSION=2026-07-23 +# Explicit opt-in only when an approved egress proxy is required. +DNSE_REST_USE_ENV_PROXY=false # Optional: vnstock API key VNSTOCK_API_KEY=your_vnstock_key @@ -305,6 +308,8 @@ All configuration is via environment variables. See [`.env.example`](./.env.exam | `REDIS_DB` | `2` | Redis database index | | `DNSE_API_KEY` | — | DNSE OpenAPI key | | `DNSE_API_SECRET_KEY` | — | DNSE OpenAPI secret | +| `DNSE_API_VERSION` | `2026-07-23` | Explicit DNSE REST contract revision | +| `DNSE_REST_USE_ENV_PROXY` | `false` | Opt in to an approved environment proxy; TLS verification remains mandatory | | `VNSTOCK_API_KEY` | — | vnstock API key | | `VNSTOCK_SOURCE` | `KBS` | vnstock data source | | `VNSTOCK_POLL_INTERVAL` | `3.0` | Polling interval (seconds) | diff --git a/app/api/context.py b/app/api/context.py index 9e4ca55..34d64ed 100644 --- a/app/api/context.py +++ b/app/api/context.py @@ -11,6 +11,7 @@ class DataLayerContext: redis_cache: Any binance_stream_supervisor: Any get_dnse_stream_manager: Callable[[], Any] + get_kline_recovery_manager: Callable[[], Any] = lambda: None demand_registry: Any = None preload_topup_coordinator: Any = None diff --git a/app/api/routes_health.py b/app/api/routes_health.py index c1acd2f..3209772 100644 --- a/app/api/routes_health.py +++ b/app/api/routes_health.py @@ -16,6 +16,20 @@ def _binance_demand_keys(demands: dict) -> set[str]: } +def _feed_data_ready(stream: dict, feed: str) -> bool: + states = [ + state + for state in stream.get("sources", {}).values() + if state.get("feed") == feed + ] + return bool(states) and all(bool(state.get("data_ready")) for state in states) + + +def _recovery_snapshot(ctx: DataLayerContext) -> dict: + manager = ctx.get_kline_recovery_manager() + return manager.snapshot() if manager else {"enabled": False, "status": "not_started"} + + @router.get("/health") async def health(ctx: DataLayerContext = Depends(get_context)): redis_ok = await ctx.redis_cache.health_check() @@ -23,6 +37,7 @@ async def health(ctx: DataLayerContext = Depends(get_context)): binance_stream = ctx.binance_stream_supervisor.snapshot( demanded_feed_keys=_binance_demand_keys(demands), ) + binance_stream["kline_recovery"] = _recovery_snapshot(ctx) dnse_manager = ctx.get_dnse_stream_manager() dnse_stream = ( dnse_manager.get_status() @@ -36,8 +51,8 @@ async def health(ctx: DataLayerContext = Depends(get_context)): return { "status": "ok" if redis_ok and stream_ok else "degraded", "redis": redis_ok, - "binance_trade_stream": binance_stream.get("status") in {"ok", "starting"}, - "binance_kline_stream": binance_stream.get("status") in {"ok", "starting"}, + "binance_trade_stream": _feed_data_ready(binance_stream, "trade"), + "binance_kline_stream": _feed_data_ready(binance_stream, "kline"), "binance_stream": binance_stream, "feed_demands": demands, "preload_topup": ctx.preload_topup_coordinator.snapshot(), @@ -49,10 +64,12 @@ async def health(ctx: DataLayerContext = Depends(get_context)): async def health_streams(ctx: DataLayerContext = Depends(get_context)): dnse_manager = ctx.get_dnse_stream_manager() demands = await ctx.demand_registry.snapshot() + binance_stream = ctx.binance_stream_supervisor.snapshot( + demanded_feed_keys=_binance_demand_keys(demands), + ) + binance_stream["kline_recovery"] = _recovery_snapshot(ctx) return { - "binance_stream": ctx.binance_stream_supervisor.snapshot( - demanded_feed_keys=_binance_demand_keys(demands), - ), + "binance_stream": binance_stream, "feed_demands": demands, "dnse_stream": ( dnse_manager.get_status() diff --git a/app/config.py b/app/config.py index 98a4cf9..4363529 100644 --- a/app/config.py +++ b/app/config.py @@ -8,6 +8,10 @@ DNSE_API_SECRET_KEY = os.getenv("DNSE_API_SECRET_KEY", "") DNSE_REST_BASE = os.getenv("DNSE_REST_BASE", "https://openapi.dnse.com.vn") DNSE_WS_BASE = os.getenv("DNSE_WS_BASE", "wss://ws-openapi.dnse.com.vn") +DNSE_API_VERSION = os.getenv("DNSE_API_VERSION", "2026-07-23") +DNSE_REST_USE_ENV_PROXY = os.getenv( + "DNSE_REST_USE_ENV_PROXY", "false" +).lower() in {"1", "true", "yes", "on"} # ── vnstock ───────────────────────────────────── VNSTOCK_API_KEY = os.getenv("VNSTOCK_API_KEY", "") @@ -39,6 +43,17 @@ BINANCE_SPOT_SYMBOLS_FILE = os.getenv("BINANCE_SPOT_SYMBOLS_FILE", "/app/symbols_spot.json") STREAM_STALE_SECONDS = float(os.getenv("STREAM_STALE_SECONDS", "180")) STREAM_STRICT_FEED_HEALTH = os.getenv("STREAM_STRICT_FEED_HEALTH", "false").lower() in {"1", "true", "yes", "on"} +BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS = float( + os.getenv("BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS", "15") +) +BINANCE_WS_IDLE_TIMEOUT_SECONDS = float(os.getenv("BINANCE_WS_IDLE_TIMEOUT_SECONDS", "90")) +BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS = float( + os.getenv("BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS", "2") +) +BINANCE_KLINE_RECOVERY_ENABLED = os.getenv("BINANCE_KLINE_RECOVERY_ENABLED", "true").lower() in {"1", "true", "yes", "on"} +BINANCE_KLINE_RECOVERY_POLL_SECONDS = float(os.getenv("BINANCE_KLINE_RECOVERY_POLL_SECONDS", "2")) +BINANCE_KLINE_RECOVERY_SETTLE_SECONDS = float(os.getenv("BINANCE_KLINE_RECOVERY_SETTLE_SECONDS", "1")) +BINANCE_KLINE_RECOVERY_CONCURRENCY = int(os.getenv("BINANCE_KLINE_RECOVERY_CONCURRENCY", "4")) # ── Provider fallback ─────────────────────────── OKX_FALLBACK_ENABLED = os.getenv("OKX_FALLBACK_ENABLED", "true").lower() in {"1", "true", "yes", "on"} diff --git a/app/database/dnse_fallback.py b/app/database/dnse_fallback.py index dfa6f58..d64df2f 100644 --- a/app/database/dnse_fallback.py +++ b/app/database/dnse_fallback.py @@ -1,180 +1,34 @@ -""" -DNSE OHLC fallback fetcher for preloading. +"""Bounded DNSE OHLC read-through fallback for legacy preload consumers.""" -Used ONLY when vnstock preload/storage fails with unknown issues. -This fetches data DIRECTLY from DNSE REST API — no disk storage. -Returns a DataFrame in memory for immediate consumption. +from __future__ import annotations -Limitations: DNSE only offers ~87 days lookback for 1m data. -Rate limits: 1,000 req/hour, 10,000 req/day. -""" -import os -import sys -import json -import time -import uuid -import random import logging -import pandas as pd -from datetime import datetime, timezone, timedelta -from typing import Optional -from collections import deque +import random +import time +from datetime import datetime, timedelta, timezone -# Add openapi_sdk to path -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "openapi_sdk", "python")) +import pandas as pd -from dnse import DNSEClient -from dnse.common import build_signature +from app.config import DNSE_API_KEY, DNSE_API_SECRET_KEY +from app.providers.dnse import fetch_dnse_ohlc_raw -from app.config import DNSE_API_KEY, DNSE_API_SECRET_KEY, DNSE_REST_BASE logger = logging.getLogger(__name__) - -# ── Rate Limiter (sliding window) ─────────────────────────────── - -class OHLCRateLimiter: - """ - Token bucket rate limiter for DNSE /price/ohlc endpoint. - 1,000 req/hour, 10,000 req/day. - Uses 90% safety margin. - """ - SAFE_HOURLY = 900 - SAFE_DAILY = 9000 - MIN_DELAY = 3600 / 900 # ~4.0s between requests - - def __init__(self): - self._hourly_window: deque = deque() - self._daily_window: deque = deque() - self._last_request_time: float = 0.0 - - def _cleanup(self): - now = time.time() - while self._hourly_window and self._hourly_window[0] < now - 3600: - self._hourly_window.popleft() - while self._daily_window and self._daily_window[0] < now - 86400: - self._daily_window.popleft() - - def wait_if_needed(self): - self._cleanup() - now = time.time() - - # Minimum delay - elapsed = now - self._last_request_time - if elapsed < self.MIN_DELAY: - time.sleep(self.MIN_DELAY - elapsed) - - # Daily quota - if len(self._daily_window) >= self.SAFE_DAILY: - oldest = self._daily_window[0] - wait = max(0, oldest + 86400 - time.time()) - logger.warning(f"DNSE daily quota reached. Waiting {wait/3600:.1f}h...") - time.sleep(wait + 1) - self._cleanup() - - # Hourly limit - if len(self._hourly_window) >= self.SAFE_HOURLY: - oldest = self._hourly_window[0] - wait = max(0, oldest + 3600 - time.time()) - logger.warning(f"DNSE hourly limit reached. Waiting {wait:.1f}s...") - time.sleep(wait + 1) - self._cleanup() - - def record(self): - now = time.time() - self._hourly_window.append(now) - self._daily_window.append(now) - self._last_request_time = now - - -# ── Singleton rate limiter ────────────────────────────────────── -_rate_limiter = OHLCRateLimiter() - -# Symbols that require bar_type='DERIVATIVE' +# Compatibility export used by the V1 preload/materialization policy. DERIVATIVE_SYMBOLS = {"VN30F1M", "VN30F2M", "VN30F1Q", "VN30F2Q"} -def _get_bar_type(symbol: str) -> str: - return "DERIVATIVE" if symbol.upper() in DERIVATIVE_SYMBOLS else "STOCK" - - def _to_unix(date_str: str) -> int: - """Convert 'YYYY-MM-DD' to Unix timestamp (seconds, UTC).""" - dt = datetime.strptime(date_str, "%Y-%m-%d").replace(tzinfo=timezone.utc) - return int(dt.timestamp()) - - -def _fetch_ohlc_raw(symbol: str, resolution: str, from_ts: int, to_ts: int) -> list: - """ - Fetch OHLC from DNSE REST using X-Aux-Date header + pagination. - Returns list of dicts [{t, o, h, l, c, v}, ...]. - Uses raw requests because DNSEClient.get_ohlc() uses Date header - but /price/ohlc requires X-Aux-Date. - """ - import requests as req - - bar_type = _get_bar_type(symbol) - path = "/price/ohlc" - rows = [] - current_from = from_ts - - while current_from < to_ts: - _rate_limiter.wait_if_needed() - - date_value = datetime.now(timezone.utc).strftime("%a, %d %b %Y %H:%M:%S %z") - nonce = uuid.uuid4().hex - headers_list, sig = build_signature( - DNSE_API_SECRET_KEY, "GET", path, date_value, - algorithm="hmac-sha256", nonce=nonce, header_name="X-Aux-Date", - ) - r = req.get( - f"{DNSE_REST_BASE}{path}", - params={ - "symbol": symbol, "type": bar_type, - "resolution": resolution, - "from": str(current_from), "to": str(to_ts), - }, - headers={ - "X-API-Key": DNSE_API_KEY, - "X-Aux-Date": date_value, - "X-Signature": ( - f'Signature keyId="{DNSE_API_KEY}",algorithm="hmac-sha256",' - f'headers="{headers_list}",signature="{sig}",nonce="{nonce}"' - ), - "Accept": "application/json", - }, - timeout=30, - ) - _rate_limiter.record() - - if r.status_code == 429: - wait = 60 + random.uniform(0, 30) - logger.warning(f"DNSE 429 for {symbol}. Backoff {wait:.0f}s...") - time.sleep(wait) - continue - - if r.status_code != 200: - logger.warning(f"DNSE HTTP {r.status_code} for {symbol}: {r.text[:200]}") - break + value = datetime.strptime(date_str, "%Y-%m-%d").replace(tzinfo=timezone.utc) + return int(value.timestamp()) - data = r.json() - t_arr = data.get("t") or [] - for i, t in enumerate(t_arr): - rows.append({ - "t": t, - "o": data["o"][i], - "h": data["h"][i], - "l": data["l"][i], - "c": data["c"][i], - "v": data["v"][i], - }) - next_time = data.get("nextTime", 0) - if next_time == 0 or next_time >= to_ts: - break - current_from = next_time - - return rows +def _fetch_ohlc_raw( + symbol: str, resolution: str, from_ts: int, to_ts: int +) -> list[dict]: + """Compatibility signature backed by the strict versioned DNSE provider.""" + return fetch_dnse_ohlc_raw(symbol, resolution, from_ts, to_ts) def fetch_dnse_ohlcv_direct( @@ -185,74 +39,72 @@ def fetch_dnse_ohlcv_direct( chunk_days: int = 7, max_retries: int = 3, ) -> pd.DataFrame: - """ - Fetch OHLCV directly from DNSE REST — NO disk storage. - Used as fallback when vnstock preload fails. - - Args: - symbol: Stock symbol (e.g., 'VCB') or derivative ('VN30F1M') - start: Start date 'YYYY-MM-DD' - end: End date 'YYYY-MM-DD' - resolution: '1' for 1m, '5' for 5m, etc. (1 3 5 15 30 1H 1D 1W) - chunk_days: Days per chunk to avoid timeout - max_retries: Retry count per chunk - - Returns: - DataFrame with columns [time, open, high, low, close, volume, symbol] - This is returned in-memory, NOT written to disk. - """ + """Fetch bounded DNSE OHLCV in memory without writing provider data to disk.""" if not DNSE_API_KEY or not DNSE_API_SECRET_KEY: - logger.error("DNSE credentials not set. Cannot use DNSE fallback.") + logger.error("DNSE credentials not set; direct OHLC fallback unavailable") return pd.DataFrame() + if chunk_days < 1 or not 1 <= max_retries <= 8: + raise ValueError("DNSE fallback chunk/retry bounds are invalid") start_dt = datetime.strptime(start, "%Y-%m-%d") - # Treat `end` as an inclusive trading date. The DNSE API uses epoch - # boundaries, so querying only to YYYY-MM-DD 00:00 would miss all intraday - # bars for that date during read-through top-up. + # End is an inclusive trading date; provider epochs use an exclusive upper day. end_dt = datetime.strptime(end, "%Y-%m-%d") + timedelta(days=1) - all_rows = [] + all_rows: list[dict] = [] current_start = start_dt while current_start < end_dt: current_end = min(current_start + timedelta(days=chunk_days), end_dt) from_ts = _to_unix(current_start.strftime("%Y-%m-%d")) to_ts = _to_unix(current_end.strftime("%Y-%m-%d")) - for attempt in range(max_retries): try: rows = _fetch_ohlc_raw(symbol, resolution, from_ts, to_ts) - if rows: - all_rows.extend(rows) - logger.info( - f"[DNSE-fallback] {symbol} {current_start.date()}->{current_end.date()}: " - f"{len(rows)} bars" - ) + all_rows.extend(rows) + logger.info( + "DNSE fallback chunk complete symbol=%s resolution=%s start=%s end=%s rows=%s", + symbol, + resolution, + current_start.date(), + current_end.date(), + len(rows), + ) break - except Exception as e: - if attempt == max_retries - 1: - logger.warning(f"[DNSE-fallback] {symbol} chunk failed: {e}") - else: - time.sleep(2 ** attempt + random.uniform(0, 1)) - - current_start = current_end + timedelta(days=1) + except Exception: + if attempt + 1 == max_retries: + logger.exception( + "DNSE fallback chunk exhausted symbol=%s resolution=%s start=%s end=%s", + symbol, + resolution, + current_start.date(), + current_end.date(), + ) + raise + time.sleep(2 ** attempt + random.uniform(0, 0.25)) + # current_end is already the next exclusive boundary. Do not skip a day. + current_start = current_end if not all_rows: return pd.DataFrame() - df = pd.DataFrame(all_rows) - # Rename columns to standard format - df = df.rename(columns={"t": "time", "o": "open", "h": "high", "l": "low", "c": "close", "v": "volume"}) - # Convert Unix timestamp to the canonical VN preload timezone. The preload - # parquet stores naive Asia/Ho_Chi_Minh timestamps; API responses later - # localize those timestamps and convert to UTC for clients. - if "time" in df.columns: - df["time"] = ( - pd.to_datetime(df["time"], unit="s", utc=True) - .dt.tz_convert("Asia/Ho_Chi_Minh") - .dt.tz_localize(None) - ) - df["symbol"] = symbol - df = df.drop_duplicates(subset=["time"]).sort_values("time").reset_index(drop=True) - - logger.info(f"[DNSE-fallback] {symbol}: {len(df)} total bars fetched (in-memory, no storage)") - return df + frame = pd.DataFrame(all_rows).rename(columns={ + "t": "time", + "o": "open", + "h": "high", + "l": "low", + "c": "close", + "v": "volume", + }) + frame["time"] = ( + pd.to_datetime(frame["time"], unit="s", utc=True) + .dt.tz_convert("Asia/Ho_Chi_Minh") + .dt.tz_localize(None) + ) + frame["symbol"] = symbol.strip().upper() + frame = frame.drop_duplicates(subset=["time"]).sort_values("time").reset_index(drop=True) + logger.info( + "DNSE fallback complete symbol=%s resolution=%s rows=%s storage=memory_only", + symbol, + resolution, + len(frame), + ) + return frame diff --git a/app/entrypoints/binance_bar_v2_stable.py b/app/entrypoints/binance_bar_v2_stable.py new file mode 100644 index 0000000..b49356e --- /dev/null +++ b/app/entrypoints/binance_bar_v2_stable.py @@ -0,0 +1,7 @@ +"""Stable Binance latest-closed BAR acquisition edge.""" + +from qdl.runtime.stable_bar_edge import main + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/app/entrypoints/projector_v2_stable.py b/app/entrypoints/projector_v2_stable.py new file mode 100644 index 0000000..ced8e87 --- /dev/null +++ b/app/entrypoints/projector_v2_stable.py @@ -0,0 +1,9 @@ +"""Kafka-authoritative Data Layer 2.0.0 stable projector.""" + +import asyncio + +from qdl.runtime.stable import serve_stable_projector + + +if __name__ == "__main__": + asyncio.run(serve_stable_projector()) diff --git a/app/entrypoints/query_v2_stable.py b/app/entrypoints/query_v2_stable.py new file mode 100644 index 0000000..f20fb2e --- /dev/null +++ b/app/entrypoints/query_v2_stable.py @@ -0,0 +1,6 @@ +"""Isolated Data Layer 2.0.0 stable query edge; V1 remains authoritative.""" + +from qdl.runtime.stable import create_stable_query_app + + +app = create_stable_query_app() diff --git a/app/entrypoints/stream_v2_stable.py b/app/entrypoints/stream_v2_stable.py new file mode 100644 index 0000000..847c2e2 --- /dev/null +++ b/app/entrypoints/stream_v2_stable.py @@ -0,0 +1,9 @@ +"""Isolated Data Layer 2.0.0 active/passive stream edge.""" + +import asyncio + +from qdl.runtime.stable import serve_stable_stream + + +if __name__ == "__main__": + asyncio.run(serve_stable_stream()) diff --git a/app/entrypoints/vn_edge_v2_stable.py b/app/entrypoints/vn_edge_v2_stable.py new file mode 100644 index 0000000..1d0e83e --- /dev/null +++ b/app/entrypoints/vn_edge_v2_stable.py @@ -0,0 +1,9 @@ +"""Stable DNSE vendor acquisition edge.""" + +import asyncio + +from qdl.adapters.vn.stable_edge import main + + +if __name__ == "__main__": + raise SystemExit(asyncio.run(main())) diff --git a/app/main.py b/app/main.py index 41cb126..3fa1adc 100644 --- a/app/main.py +++ b/app/main.py @@ -28,6 +28,13 @@ from app.config import ( PRELOAD_DAILY_RUN_TIME, PRELOAD_DIR, + BINANCE_WS_QUEUE_MAXSIZE, + BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, + BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, + BINANCE_KLINE_RECOVERY_ENABLED, + BINANCE_KLINE_RECOVERY_POLL_SECONDS, + BINANCE_KLINE_RECOVERY_SETTLE_SECONDS, + BINANCE_KLINE_RECOVERY_CONCURRENCY, STREAM_STALE_SECONDS, STREAM_STRICT_FEED_HEALTH, ) @@ -43,6 +50,7 @@ ) from app.cache.redis_cache import RedisCache from app.stream.async_live_feed import start_stream +from app.stream.kline_recovery import DemandKlineRecovery, KlineRecoveryConfig from app.ingestion.supervisor import StreamSupervisor from app.stream.vnstock_poller import VnstockPoller from app.stream.dnse_ws import DnseStreamManager @@ -74,9 +82,12 @@ preload_thread = None preload_stop_event = threading.Event() unified_stream_task = None +kline_recovery_manager = None +kline_recovery_task = None binance_stream_supervisor = StreamSupervisor( stale_after_seconds=STREAM_STALE_SECONDS, strict_feed_health=STREAM_STRICT_FEED_HEALTH, + first_frame_timeout_seconds=BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, ) preload_daily_state_dir = os.path.join(os.path.dirname(PRELOAD_DIR), "_state") @@ -261,14 +272,18 @@ async def lifespan(app: FastAPI): - Start the vnstock REST poller (FALLBACK only if DNSE stale) """ global dnse_stream_manager, unified_stream_task, binance_stream_supervisor + global kline_recovery_manager, kline_recovery_task logger.info("=== data_layer service starting ===") await redis_cache.init_ping() runtime_sources = RuntimeSourceConfig.from_env() logger.info("Runtime source configuration: %s", runtime_sources.public_summary()) - # 1. Unified stream (Binance) + # 1. Unified stream (Binance) and demand-only closed-kline recovery. + kline_recovery_manager = None + kline_recovery_task = None if runtime_sources.binance_sources: + stream_queue = asyncio.Queue(maxsize=BINANCE_WS_QUEUE_MAXSIZE) unified_stream_task = asyncio.create_task( start_stream( redis_cache, @@ -276,8 +291,26 @@ async def lifespan(app: FastAPI): supervisor=binance_stream_supervisor, demand_registry=demand_registry, enabled_sources=list(runtime_sources.binance_sources), + queue=stream_queue, ) ) + if ( + BINANCE_KLINE_RECOVERY_ENABLED + and "binance_futures_kline" in runtime_sources.binance_sources + ): + kline_recovery_manager = DemandKlineRecovery( + queue=stream_queue, + redis_cache=redis_cache, + demand_registry=demand_registry, + config=KlineRecoveryConfig( + enabled=True, + poll_seconds=BINANCE_KLINE_RECOVERY_POLL_SECONDS, + settle_seconds=BINANCE_KLINE_RECOVERY_SETTLE_SECONDS, + concurrency=BINANCE_KLINE_RECOVERY_CONCURRENCY, + queue_put_timeout_seconds=BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, + ), + ) + kline_recovery_task = asyncio.create_task(kline_recovery_manager.run()) logger.info("Unified async Binance streams started: %s", runtime_sources.binance_sources) else: unified_stream_task = None @@ -333,8 +366,13 @@ async def lifespan(app: FastAPI): # Shutdown logger.info("Shutting down services...") - if unified_stream_task: - unified_stream_task.cancel() + async_tasks = [ + task for task in (kline_recovery_task, unified_stream_task) if task is not None + ] + for task in async_tasks: + task.cancel() + if async_tasks: + await asyncio.gather(*async_tasks, return_exceptions=True) if vn_poller: vn_poller.stop() @@ -366,6 +404,7 @@ async def lifespan(app: FastAPI): redis_cache=redis_cache, binance_stream_supervisor=binance_stream_supervisor, get_dnse_stream_manager=lambda: dnse_stream_manager, + get_kline_recovery_manager=lambda: kline_recovery_manager, demand_registry=demand_registry, preload_topup_coordinator=preload_topup_coordinator, ) diff --git a/app/openapi_sdk/python/websocket_marketdata/trading_websocket/client.py b/app/openapi_sdk/python/websocket_marketdata/trading_websocket/client.py index 638d3ef..5f8c32f 100644 --- a/app/openapi_sdk/python/websocket_marketdata/trading_websocket/client.py +++ b/app/openapi_sdk/python/websocket_marketdata/trading_websocket/client.py @@ -78,6 +78,7 @@ def __init__( max_retries: int = 10, heartbeat_interval: float = 25.0, timeout: float = 60.0, + dispatch_queue_capacity: int = 0, ): """ Initialize trading client. @@ -100,6 +101,9 @@ def __init__( self.max_retries = max_retries self.heartbeat_interval = heartbeat_interval self.timeout = timeout + if dispatch_queue_capacity < 0 or dispatch_queue_capacity > 100_000: + raise ValueError("dispatch_queue_capacity must be between 0 and 100000") + self.dispatch_queue_capacity = dispatch_queue_capacity # Internal state self._connection: Optional[WebSocketConnection] = None @@ -162,7 +166,10 @@ async def connect(self) -> None: self._last_pong_time = time.time() # Init 1 queue per worker slot - self._dispatch_queues = [asyncio.Queue() for _ in range(self._num_workers)] + self._dispatch_queues = [ + asyncio.Queue(maxsize=self.dispatch_queue_capacity) + for _ in range(self._num_workers) + ] # Start dispatch workers (1 per slot, same symbol → same worker → ordered) self._dispatch_worker_tasks = [ diff --git a/app/providers/dnse/__init__.py b/app/providers/dnse/__init__.py index a0f4996..e64227a 100644 --- a/app/providers/dnse/__init__.py +++ b/app/providers/dnse/__init__.py @@ -1,2 +1,19 @@ """DNSE market-data provider integration.""" +from app.providers.dnse.history import ( + DEFAULT_DNSE_API_VERSION, + DnseHistoryClient, + DnseHistoryConfig, + DnseHistoryError, + DnseQuotaLimiter, + fetch_dnse_ohlc_raw, +) + +__all__ = [ + "DEFAULT_DNSE_API_VERSION", + "DnseHistoryClient", + "DnseHistoryConfig", + "DnseHistoryError", + "DnseQuotaLimiter", + "fetch_dnse_ohlc_raw", +] diff --git a/app/providers/dnse/history.py b/app/providers/dnse/history.py new file mode 100644 index 0000000..95ae5ca --- /dev/null +++ b/app/providers/dnse/history.py @@ -0,0 +1,389 @@ +from __future__ import annotations + +import json +import logging +import os +import random +import threading +import time +import uuid +from collections import deque +from dataclasses import dataclass +from datetime import datetime, timezone +from decimal import Decimal, InvalidOperation +from email.utils import parsedate_to_datetime +from typing import Any, Callable, Mapping + +import requests + +from app.openapi_sdk.python.dnse.common import build_signature + + +logger = logging.getLogger(__name__) + +DEFAULT_DNSE_API_VERSION = "2026-07-23" +_ALLOWED_RESOLUTIONS = frozenset({"1", "3", "5", "15", "30", "1H", "1D", "1W"}) +_DERIVATIVE_SYMBOLS = frozenset({"VN30F1M", "VN30F2M", "VN30F1Q", "VN30F2Q"}) + + +class DnseHistoryError(RuntimeError): + """A bounded, redacted DNSE history acquisition failure.""" + + def __init__(self, message: str, *, status_code: int | None = None) -> None: + super().__init__(message) + self.status_code = status_code + + +@dataclass(frozen=True) +class DnseHistoryConfig: + api_key: str + api_secret: str + base_url: str = "https://openapi.dnse.com.vn" + api_version: str = DEFAULT_DNSE_API_VERSION + connect_timeout_seconds: float = 5.0 + read_timeout_seconds: float = 30.0 + attempts: int = 4 + max_backoff_seconds: float = 30.0 + max_pages: int = 256 + max_rows: int = 100_000 + max_response_bytes: int = 8 * 1024 * 1024 + use_environment_proxy: bool = False + + def __post_init__(self) -> None: + if not self.api_key or not self.api_secret: + raise ValueError("DNSE history credentials are required") + if not self.base_url.startswith("https://"): + raise ValueError("DNSE history requires HTTPS") + if not self.api_version.strip(): + raise ValueError("DNSE API version is required") + if min(self.connect_timeout_seconds, self.read_timeout_seconds) <= 0: + raise ValueError("DNSE history timeouts must be positive") + if not 1 <= self.attempts <= 8: + raise ValueError("DNSE history attempts must be between 1 and 8") + if not 0.1 <= self.max_backoff_seconds <= 300: + raise ValueError("DNSE history max backoff is invalid") + if not 1 <= self.max_pages <= 1024 or not 1 <= self.max_rows <= 1_000_000: + raise ValueError("DNSE history page/row bound is invalid") + if not 1024 <= self.max_response_bytes <= 64 * 1024 * 1024: + raise ValueError("DNSE history response bound is invalid") + + @classmethod + def from_environment(cls) -> "DnseHistoryConfig": + return cls( + api_key=os.getenv("DNSE_API_KEY", ""), + api_secret=os.getenv("DNSE_API_SECRET_KEY", ""), + base_url=os.getenv("DNSE_REST_BASE", "https://openapi.dnse.com.vn"), + api_version=os.getenv("DNSE_API_VERSION", DEFAULT_DNSE_API_VERSION), + connect_timeout_seconds=float( + os.getenv("DNSE_REST_CONNECT_TIMEOUT_SECONDS", "5") + ), + read_timeout_seconds=float( + os.getenv("DNSE_REST_READ_TIMEOUT_SECONDS", "30") + ), + attempts=int(os.getenv("DNSE_REST_ATTEMPTS", "4")), + max_backoff_seconds=float( + os.getenv("DNSE_REST_MAX_BACKOFF_SECONDS", "30") + ), + use_environment_proxy=os.getenv( + "DNSE_REST_USE_ENV_PROXY", "false" + ).lower() in {"1", "true", "yes", "on"}, + ) + + +class DnseQuotaLimiter: + """Process-local thread-safe safety margin around documented DNSE quotas.""" + + def __init__( + self, + *, + hourly_limit: int = 900, + daily_limit: int = 9000, + min_interval_seconds: float = 4.0, + clock: Callable[[], float] = time.monotonic, + sleep: Callable[[float], None] = time.sleep, + ) -> None: + if min(hourly_limit, daily_limit) <= 0 or min_interval_seconds < 0: + raise ValueError("DNSE quota limiter bounds are invalid") + self.hourly_limit = hourly_limit + self.daily_limit = daily_limit + self.min_interval_seconds = min_interval_seconds + self.clock = clock + self.sleep = sleep + self._hourly: deque[float] = deque() + self._daily: deque[float] = deque() + self._last_request = float("-inf") + self._lock = threading.Lock() + + def acquire(self) -> None: + while True: + with self._lock: + now = self.clock() + while self._hourly and self._hourly[0] <= now - 3600: + self._hourly.popleft() + while self._daily and self._daily[0] <= now - 86400: + self._daily.popleft() + delays = [max(0.0, self._last_request + self.min_interval_seconds - now)] + if len(self._hourly) >= self.hourly_limit: + delays.append(max(0.0, self._hourly[0] + 3600 - now)) + if len(self._daily) >= self.daily_limit: + delays.append(max(0.0, self._daily[0] + 86400 - now)) + delay = max(delays) + if delay <= 0: + self._hourly.append(now) + self._daily.append(now) + self._last_request = now + return + self.sleep(max(delay, 0.001)) + + +class DnseHistoryClient: + """Strict DNSE `/price/ohlc` transport for bootstrap and bounded repair.""" + + def __init__( + self, + config: DnseHistoryConfig, + *, + session: requests.Session | None = None, + limiter: DnseQuotaLimiter | None = None, + sleep: Callable[[float], None] = time.sleep, + random_uniform: Callable[[float, float], float] = random.uniform, + ) -> None: + self.config = config + self.session = session or requests.Session() + self.session.trust_env = config.use_environment_proxy + self.limiter = limiter or DnseQuotaLimiter() + self.sleep = sleep + self.random_uniform = random_uniform + + @staticmethod + def _bar_type(symbol: str) -> str: + return "DERIVATIVE" if symbol in _DERIVATIVE_SYMBOLS else "STOCK" + + def _headers(self, path: str) -> dict[str, str]: + date_value = datetime.now(timezone.utc).strftime("%a, %d %b %Y %H:%M:%S %z") + nonce = uuid.uuid4().hex + signed_headers, signature = build_signature( + self.config.api_secret, + "GET", + path, + date_value, + algorithm="hmac-sha256", + nonce=nonce, + header_name="X-Aux-Date", + ) + return { + "Accept": "application/json", + "X-API-Key": self.config.api_key, + "X-Aux-Date": date_value, + "X-Signature": ( + f'Signature keyId="{self.config.api_key}",algorithm="hmac-sha256",' + f'headers="{signed_headers}",signature="{signature}",nonce="{nonce}"' + ), + "version": self.config.api_version, + } + + @staticmethod + def _retry_after_seconds(value: str | None, now: datetime | None = None) -> float | None: + if not value: + return None + try: + return max(0.0, float(value)) + except ValueError: + try: + parsed = parsedate_to_datetime(value) + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + current = now or datetime.now(timezone.utc) + return max(0.0, (parsed - current).total_seconds()) + except (TypeError, ValueError, OverflowError): + return None + + def _backoff(self, attempt: int, retry_after: str | None) -> float: + provider_delay = self._retry_after_seconds(retry_after) + exponential = min(2 ** attempt, self.config.max_backoff_seconds) + delay = provider_delay if provider_delay is not None else exponential + return min( + self.config.max_backoff_seconds, + max(0.0, delay) + self.random_uniform(0.0, min(0.25, delay / 10)), + ) + + def _request_page(self, params: Mapping[str, str]) -> Mapping[str, Any]: + path = "/price/ohlc" + last_error: Exception | None = None + for attempt in range(self.config.attempts): + self.limiter.acquire() + try: + response = self.session.get( + f"{self.config.base_url.rstrip('/')}{path}", + params=dict(params), + headers=self._headers(path), + timeout=( + self.config.connect_timeout_seconds, + self.config.read_timeout_seconds, + ), + verify=True, + ) + except requests.RequestException as error: + last_error = error + if attempt + 1 == self.config.attempts: + break + self.sleep(self._backoff(attempt, None)) + continue + + if len(response.content) > self.config.max_response_bytes: + raise DnseHistoryError("DNSE history response exceeds byte bound") + if response.status_code == 200: + try: + payload = response.json() + except (ValueError, json.JSONDecodeError) as error: + raise DnseHistoryError("DNSE history response is not valid JSON") from error + if not isinstance(payload, Mapping): + raise DnseHistoryError("DNSE history response root is invalid") + return payload + + last_error = DnseHistoryError( + f"DNSE history HTTP status={response.status_code}", + status_code=response.status_code, + ) + retryable = response.status_code in {408, 425, 429} or response.status_code >= 500 + if not retryable or attempt + 1 == self.config.attempts: + raise last_error + self.sleep(self._backoff(attempt, response.headers.get("Retry-After"))) + + raise DnseHistoryError( + f"DNSE history transport exhausted attempts={self.config.attempts}" + ) from last_error + + @staticmethod + def _decimal(value: Any, field: str, *, allow_zero: bool) -> Decimal: + if value is None or isinstance(value, bool): + raise DnseHistoryError(f"DNSE OHLC {field} is missing") + try: + parsed = Decimal(str(value)) + except (InvalidOperation, ValueError) as error: + raise DnseHistoryError(f"DNSE OHLC {field} is invalid") from error + if not parsed.is_finite() or parsed < 0 or (not allow_zero and parsed == 0): + raise DnseHistoryError(f"DNSE OHLC {field} is outside domain") + return parsed + + def _rows_from_page( + self, + payload: Mapping[str, Any], + *, + from_ts: int, + to_ts: int, + ) -> tuple[list[dict[str, Any]], int]: + fields = ("t", "o", "h", "l", "c", "v") + arrays = {field: payload.get(field) for field in fields} + if any(not isinstance(value, list) for value in arrays.values()): + raise DnseHistoryError("DNSE OHLC parallel arrays are missing") + lengths = {len(value) for value in arrays.values()} + if len(lengths) != 1: + raise DnseHistoryError("DNSE OHLC parallel array lengths differ") + + rows: list[dict[str, Any]] = [] + previous_time = -1 + for index in range(len(arrays["t"])): + timestamp = arrays["t"][index] + if isinstance(timestamp, bool): + raise DnseHistoryError("DNSE OHLC timestamp is invalid") + try: + timestamp = int(timestamp) + except (TypeError, ValueError) as error: + raise DnseHistoryError("DNSE OHLC timestamp is invalid") from error + if timestamp < from_ts or timestamp > to_ts or timestamp < previous_time: + raise DnseHistoryError("DNSE OHLC timestamp ordering/bound is invalid") + previous_time = timestamp + row = {field: arrays[field][index] for field in fields} + row["t"] = timestamp + prices = { + field: self._decimal(row[field], field, allow_zero=False) + for field in ("o", "h", "l", "c") + } + self._decimal(row["v"], "v", allow_zero=True) + if ( + prices["h"] < max(prices["o"], prices["c"], prices["l"]) + or prices["l"] > min(prices["o"], prices["c"], prices["h"]) + ): + raise DnseHistoryError("DNSE OHLC price invariants failed") + rows.append(row) + + next_time = payload.get("nextTime", 0) + if next_time in (None, ""): + next_time = 0 + if isinstance(next_time, bool): + raise DnseHistoryError("DNSE OHLC nextTime is invalid") + try: + next_time = int(next_time) + except (TypeError, ValueError) as error: + raise DnseHistoryError("DNSE OHLC nextTime is invalid") from error + return rows, next_time + + def fetch_ohlc( + self, + symbol: str, + resolution: str, + from_ts: int, + to_ts: int, + ) -> list[dict[str, Any]]: + normalized_symbol = symbol.strip().upper() + if not normalized_symbol or not normalized_symbol.replace("_", "").isalnum(): + raise ValueError("DNSE symbol is invalid") + if resolution not in _ALLOWED_RESOLUTIONS: + raise ValueError("DNSE resolution is unsupported") + if isinstance(from_ts, bool) or isinstance(to_ts, bool) or from_ts <= 0 or to_ts <= from_ts: + raise ValueError("DNSE history range is invalid") + + current_from = int(from_ts) + by_time: dict[int, tuple[str, dict[str, Any]]] = {} + for _page in range(self.config.max_pages): + payload = self._request_page({ + "symbol": normalized_symbol, + "type": self._bar_type(normalized_symbol), + "resolution": resolution, + "from": str(current_from), + "to": str(int(to_ts)), + }) + rows, next_time = self._rows_from_page( + payload, from_ts=current_from, to_ts=int(to_ts) + ) + for row in rows: + digest = json.dumps( + row, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ) + previous = by_time.get(row["t"]) + if previous is not None and previous[0] != digest: + raise DnseHistoryError( + f"DNSE OHLC conflicting timestamp={row['t']}" + ) + by_time[row["t"]] = digest, row + if len(by_time) > self.config.max_rows: + raise DnseHistoryError("DNSE history exceeds row bound") + + if next_time == 0 or next_time >= to_ts: + return [by_time[key][1] for key in sorted(by_time)] + if next_time <= current_from: + raise DnseHistoryError("DNSE OHLC pagination did not advance") + current_from = next_time + + raise DnseHistoryError("DNSE OHLC pagination exceeds page bound") + + +_default_client: DnseHistoryClient | None = None +_default_client_lock = threading.Lock() + + +def default_dnse_history_client() -> DnseHistoryClient: + global _default_client + with _default_client_lock: + if _default_client is None: + _default_client = DnseHistoryClient(DnseHistoryConfig.from_environment()) + return _default_client + + +def fetch_dnse_ohlc_raw( + symbol: str, resolution: str, from_ts: int, to_ts: int +) -> list[dict[str, Any]]: + return default_dnse_history_client().fetch_ohlc( + symbol, resolution, from_ts, to_ts + ) diff --git a/app/stream/async_live_feed.py b/app/stream/async_live_feed.py index 0467063..0dc10b9 100644 --- a/app/stream/async_live_feed.py +++ b/app/stream/async_live_feed.py @@ -5,8 +5,6 @@ import time import random import os -import math -import re import requests from websockets.asyncio.client import connect as websocket_connect @@ -16,94 +14,18 @@ from app.stream.feed_builder import build_urls, validate_symbols from app.stream.feed_parsers import PARSERS from app.stream.supervisor import StreamSupervisor -from app.stream.demand_registry import feed_key_for -from app.providers.binance import rest as binance_rest from app.config import ( BINANCE_SPOT_SYMBOLS_FILE, BINANCE_WS_BATCH_SIZE, BINANCE_WS_MAX_CONNS_PER_SOURCE, BINANCE_WS_QUEUE_MAXSIZE, + BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, + BINANCE_WS_IDLE_TIMEOUT_SECONDS, + BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, ) logger = logging.getLogger(__name__) -_STREAM_SYMBOL_RE = re.compile(r"[=/]([a-z0-9_]+)@(kline_[^/]+|trade)") - - -def symbols_from_stream_url(url: str) -> list[str]: - return [match.group(1).upper() for match in _STREAM_SYMBOL_RE.finditer(url)] - - -def _interval_seconds(interval: str) -> int: - unit = interval[-1] - value = int(interval[:-1]) - return value * {"s": 1, "m": 60, "h": 3600, "d": 86400}.get(unit, 60) - - -def _rest_kline_event(symbol: str, interval: str, row: list) -> dict: - return { - "e": "kline_recovery", - "E": int(time.time() * 1000), - "s": symbol, - "k": { - "t": row[0], "T": row[6], "s": symbol, "i": interval, - "o": row[1], "c": row[4], "h": row[2], "l": row[3], "v": row[5], "x": True, - }, - "recovery_source": "BINANCE_REST_GAP_FILL", - } - - -async def recover_demanded_kline_gap( - *, - source: str, - url: str, - interval: str, - queue: asyncio.Queue, - supervisor: StreamSupervisor, - shard_id: str, - demand_registry, -) -> int: - if source.endswith("_trade") or "kline" not in url: - return 0 - active = await demand_registry.snapshot() - demanded = set(active["feed_keys"]) - symbols = [ - symbol for symbol in symbols_from_stream_url(url) - if feed_key_for(source, "kline", symbol, interval) in demanded - ] - if not symbols: - return 0 - supervisor.record_gap_detected(shard_id) - shard = supervisor.shards.get(shard_id) - outage_seconds = float(shard.last_outage_seconds or 0) if shard else 0.0 - limit = min(1000, max(3, math.ceil(outage_seconds / max(1, _interval_seconds(interval))) + 2)) - market = "usdm" if source.startswith("binance_futures") else "spot" - recovered = 0 - now_ms = int(time.time() * 1000) - try: - for symbol in symbols: - payload = await asyncio.to_thread( - binance_rest.fetch_klines, - symbol, - interval, - limit, - None, - None, - market, - ) - for row in payload.get("data") or []: - if len(row) <= 6 or int(row[6]) > now_ms: - continue - await queue.put((source, _rest_kline_event(symbol, interval, row))) - recovered += 1 - supervisor.record_gap_fill(shard_id, success=True) - return recovered - except Exception: - supervisor.record_gap_fill(shard_id, success=False) - logger.exception("[WS] demanded kline gap-fill failed source=%s shard=%s", source, shard_id) - return 0 - - def get_spot_symbols( file_path: str = BINANCE_SPOT_SYMBOLS_FILE, *, @@ -157,10 +79,10 @@ def get_parser_key(source: str): def coalesce_redis_items(items: list[dict]) -> list[dict]: """Keep only the latest event per Redis key/channel within a publisher batch.""" - coalesced: dict[tuple[str, str], dict] = {} - order: list[tuple[str, str]] = [] + coalesced: dict[tuple[str, str, str | None], dict] = {} + order: list[tuple[str, str, str | None]] = [] for item in items: - dedupe_key = (item.get("key"), item.get("channel")) + dedupe_key = (item.get("key"), item.get("channel"), item.get("coalesce_id")) if dedupe_key not in coalesced: order.append(dedupe_key) coalesced[dedupe_key] = item @@ -176,6 +98,65 @@ def _source_market_namespace(source: str) -> str | None: return None +class StreamDataTimeout(RuntimeError): + """A connected websocket failed to produce valid provider data in time.""" + + +class StreamBackpressureTimeout(RuntimeError): + """The bounded publisher queue could not accept a provider event in time.""" + + +def valid_provider_frame(source: str, item: object, interval: str = "1m") -> bool: + """Return true only for a complete frame belonging to the configured feed.""" + if not isinstance(item, dict): + return False + if source.endswith("_trade"): + required = ("s", "p", "q", "t", "T") + return item.get("e") == "trade" and all(item.get(field) is not None for field in required) + if source.endswith("_kline"): + kline = item.get("k") + required = ("s", "i", "t", "T", "o", "h", "l", "c", "v", "x") + return ( + item.get("e") == "kline" + and isinstance(kline, dict) + and str(kline.get("i")) == interval + and all(kline.get(field) is not None for field in required) + ) + return False + + +def provider_items(source: str, payload: object, interval: str = "1m") -> list[dict]: + """Extract validated data frames while ignoring subscription ACKs.""" + if not isinstance(payload, dict): + raise ValueError("websocket payload must be an object") + if "data" not in payload and "id" in payload and "result" in payload: + return [] + data = payload.get("data", payload) + items = data if isinstance(data, list) else [data] + if not items or not all(valid_provider_frame(source, item, interval) for item in items): + raise ValueError(f"invalid or wrong-feed provider frame for {source}") + return items + + +async def _put_provider_item( + queue: asyncio.Queue, + item: tuple[str, dict], + *, + supervisor: StreamSupervisor, + timeout_seconds: float, +) -> None: + try: + queue.put_nowait(item) + except asyncio.QueueFull: + supervisor.record_queue_pressure() + try: + await asyncio.wait_for(queue.put(item), timeout=max(0.05, timeout_seconds)) + except asyncio.TimeoutError as exc: + raise StreamBackpressureTimeout( + f"publisher queue remained full for {timeout_seconds:.3f}s" + ) from exc + + async def handle_ws( url: str, queue: asyncio.Queue, @@ -187,80 +168,72 @@ async def handle_ws( max_backoff: int = 300, demand_registry=None, interval: str = "1m", + first_frame_timeout_seconds: float = BINANCE_WS_FIRST_FRAME_TIMEOUT_SECONDS, + idle_timeout_seconds: float = BINANCE_WS_IDLE_TIMEOUT_SECONDS, + queue_put_timeout_seconds: float = BINANCE_WS_QUEUE_PUT_TIMEOUT_SECONDS, ): - """ - Connect and stream messages from websocket. Reconnects on error. - parser_mode: "raw" (default) forwards original payload; "unified" forwards parsed data. - """ + """Receive valid provider frames with bounded readiness and backpressure.""" + del demand_registry # Kept in the public signature for V1 caller compatibility. backoff = reconnect_delay last_connected_at = None while True: try: supervisor.mark_connecting(shard_id) - logger.info(f"[WS] Connecting {source} -> {url[:60]}...") + logger.info("[WS] Connecting %s -> %s...", source, url[:60]) async with websocket_connect(url, ping_interval=30, max_size=None) as ws: - logger.info(f"[WS] Connected {source}") - recovered = supervisor.mark_connected(shard_id) + logger.info("[WS] Connected %s", source) + supervisor.mark_connected(shard_id) last_connected_at = time.monotonic() - if recovered and demand_registry is not None: - filled = await recover_demanded_kline_gap( - source=source, - url=url, - interval=interval, - queue=queue, - supervisor=supervisor, - shard_id=shard_id, - demand_registry=demand_registry, + session_has_valid_frame = False + + while True: + timeout = ( + idle_timeout_seconds + if session_has_valid_frame + else first_frame_timeout_seconds ) - if filled: - logger.info( - "[WS] demanded kline gap-fill complete source=%s shard=%s rows=%s", - source, - shard_id, - filled, - ) + try: + msg = await asyncio.wait_for(ws.recv(), timeout=max(0.05, timeout)) + except asyncio.TimeoutError as exc: + reason = "idle" if session_has_valid_frame else "first_frame" + supervisor.mark_data_timeout(shard_id, reason) + raise StreamDataTimeout( + f"{source} {reason} timeout after {timeout:.3f}s" + ) from exc - async for msg in ws: try: - supervisor.mark_message(shard_id) - payload = json.loads(msg) - data = payload.get("data") or payload.get("result") or payload - if not data: + raw_items = provider_items(source, json.loads(msg), interval) + if not raw_items: continue + supervisor.mark_message(shard_id) + session_has_valid_frame = True + backoff = reconnect_delay - output = data - if parser_mode == "unified": - parser_key = get_parser_key(source) - if parser_key: - parser = PARSERS.get(parser_key) + for data in raw_items: + output = data + if parser_mode == "unified": + parser = PARSERS.get(get_parser_key(source) or "") if parser: parsed = parser(data) if parsed is None: continue output = parsed - - items = output if isinstance(output, list) else [output] - for item in items: - # We put to the queue for the Redis publisher - # Structure: (source, item) - try: - queue.put_nowait((source, item)) - except asyncio.QueueFull: - # Pop one to make room - try: - queue.get_nowait() - queue.task_done() - except asyncio.QueueEmpty: - pass - supervisor.record_queue_drop(source, shard_id) - queue.put_nowait((source, item)) - - except Exception as e: - supervisor.mark_parse_error(shard_id, e) - logger.error(f"[WS] parse error: {e}") - - except InvalidStatus as e: - status = getattr(getattr(e, "response", None), "status_code", None) + items = output if isinstance(output, list) else [output] + for item in items: + await _put_provider_item( + queue, + (source, item), + supervisor=supervisor, + timeout_seconds=queue_put_timeout_seconds, + ) + except StreamBackpressureTimeout: + supervisor.mark_data_timeout(shard_id, "publisher_backpressure") + raise + except (TypeError, ValueError, json.JSONDecodeError) as exc: + supervisor.mark_parse_error(shard_id, exc) + logger.warning("[WS] rejected provider frame source=%s error=%s", source, exc) + except InvalidStatus as exc: + status = getattr(getattr(exc, "response", None), "status_code", None) elapsed = (time.monotonic() - last_connected_at) if last_connected_at else 0 if status == 429: backoff = min(max(backoff * 2, reconnect_delay * 2), max_backoff) @@ -269,28 +242,37 @@ async def handle_ws( if elapsed >= 60: backoff = reconnect_delay sleep_for = backoff + random.uniform(0, min(5, backoff * 0.2)) - supervisor.mark_reconnect(shard_id, e) - logger.error(f"[WS] connection rejected ({source}): HTTP {status}. Reconnecting in {int(sleep_for)}s...") + supervisor.mark_reconnect(shard_id, exc) + logger.error( + "[WS] connection rejected (%s): HTTP %s. Reconnecting in %ss...", + source, + status, + int(sleep_for), + ) await asyncio.sleep(sleep_for) - except (ConnectionClosedError, ConnectionClosedOK) as e: + except (ConnectionClosedError, ConnectionClosedOK) as exc: elapsed = (time.monotonic() - last_connected_at) if last_connected_at else 0 - if elapsed >= 60: - backoff = reconnect_delay - else: - backoff = min(backoff + 5, max_backoff) + backoff = reconnect_delay if elapsed >= 60 else min(backoff + 5, max_backoff) sleep_for = backoff + random.uniform(0, min(5, backoff * 0.2)) - supervisor.mark_reconnect(shard_id, e) - logger.error(f"[WS] connection closed ({source}): {e}. Reconnecting in {int(sleep_for)}s...") + supervisor.mark_reconnect(shard_id, exc) + logger.error( + "[WS] connection closed (%s): %s. Reconnecting in %ss...", + source, + exc, + int(sleep_for), + ) await asyncio.sleep(sleep_for) - except Exception as e: + except Exception as exc: elapsed = (time.monotonic() - last_connected_at) if last_connected_at else 0 - if elapsed >= 60: - backoff = reconnect_delay - else: - backoff = min(backoff + 5, max_backoff) + backoff = reconnect_delay if elapsed >= 60 else min(backoff + 5, max_backoff) sleep_for = backoff + random.uniform(0, min(5, backoff * 0.2)) - supervisor.mark_reconnect(shard_id, e) - logger.error(f"[WS] connection error ({source}): {e}. Reconnecting in {int(sleep_for)}s...") + supervisor.mark_reconnect(shard_id, exc) + logger.error( + "[WS] connection error (%s): %s. Reconnecting in %ss...", + source, + exc, + int(sleep_for), + ) await asyncio.sleep(sleep_for) @@ -387,11 +369,20 @@ async def redis_publisher_task( continue sym = sym.upper() - key = f"kline:{interval}:{sym}" - channel = f"stream:kline:{interval}:{sym}" - redis_items.append( - {"key": key, "channel": channel, "data": raw_data, "source": source} - ) + kline = raw_data.get("k") if isinstance(raw_data, dict) else None + event_interval = str((kline or {}).get("i") or interval) + key = f"kline:{event_interval}:{sym}" + channel = f"stream:kline:{event_interval}:{sym}" + redis_item = { + "key": key, + "channel": channel, + "data": raw_data, + "source": source, + } + if isinstance(raw_data, dict) and raw_data.get("recovery_source"): + kline = raw_data.get("k") or {} + redis_item["coalesce_id"] = f"recovery:{kline.get('t')}" + redis_items.append(redis_item) elif source == "dnse": sym = data.get("symbol", "") @@ -424,6 +415,7 @@ async def start_stream( max_conns_per_source: int | None = None, supervisor: StreamSupervisor | None = None, demand_registry=None, + queue: asyncio.Queue | None = None, ): """ Main entrypoint used by services. @@ -437,7 +429,7 @@ async def start_stream( ) # 1. Setup Queue and Publisher - queue = asyncio.Queue(maxsize=BINANCE_WS_QUEUE_MAXSIZE) + queue = queue if queue is not None else asyncio.Queue(maxsize=BINANCE_WS_QUEUE_MAXSIZE) supervisor.record_queue_size(queue.qsize(), queue.maxsize) publisher_task = asyncio.create_task(redis_publisher_task(queue, redis_cache, interval, supervisor)) diff --git a/app/stream/demand_registry.py b/app/stream/demand_registry.py index 01a30b2..5520b6e 100644 --- a/app/stream/demand_registry.py +++ b/app/stream/demand_registry.py @@ -163,8 +163,11 @@ async def snapshot(self) -> dict[str, Any]: float(item.get("expires_at_unix") or 0), ) items = sorted(aggregate.values(), key=lambda item: item["feed_key"]) + owners = sorted({str(item.get("owner_id")) for item in records if item.get("owner_id")}) return { "lease_count": len(records), + "owner_count": len(owners), + "owners": owners, "demanded_feed_count": len(items), "feed_keys": [item["feed_key"] for item in items], "items": items, diff --git a/app/stream/feed_builder.py b/app/stream/feed_builder.py index a00540d..563e6ac 100644 --- a/app/stream/feed_builder.py +++ b/app/stream/feed_builder.py @@ -40,7 +40,7 @@ def build_urls(symbols_by_source: Dict[str, List[str]], interval: str = "1m", ba continue if source in {"binance_futures", "binance_futures_kline"}: - base = "wss://fstream.binance.com/stream?streams=" + base = "wss://fstream.binance.com/public/stream?streams=" lower_syms = [s.lower() for s in clean] src_urls = [] for i in range(0, len(lower_syms), batch_size): @@ -60,7 +60,7 @@ def build_urls(symbols_by_source: Dict[str, List[str]], interval: str = "1m", ba urls[source] = src_urls elif source == "binance_futures_trade": - base = "wss://fstream.binance.com/stream?streams=" + base = "wss://fstream.binance.com/public/stream?streams=" lower_syms = [s.lower() for s in clean] src_urls = [] for i in range(0, len(lower_syms), batch_size): diff --git a/app/stream/kline_recovery.py b/app/stream/kline_recovery.py new file mode 100644 index 0000000..4b43ef8 --- /dev/null +++ b/app/stream/kline_recovery.py @@ -0,0 +1,325 @@ +from __future__ import annotations + +import asyncio +import logging +import math +import time +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from typing import Any, Callable + +from app.providers.binance import rest as binance_rest +from app.stream.demand_registry import parse_feed_key + + +logger = logging.getLogger(__name__) + + +def _fixed_interval_ms(interval: str) -> int | None: + value = int(interval[:-1]) + unit = interval[-1] + multiplier = { + "s": 1_000, + "m": 60_000, + "h": 3_600_000, + "d": 86_400_000, + }.get(unit) + return value * multiplier if multiplier else None + + +def _next_close_ms(now_ms: int, interval: str, settle_ms: int) -> int: + fixed = _fixed_interval_ms(interval) + if fixed: + return ((now_ms // fixed) + 1) * fixed + settle_ms + + now = datetime.fromtimestamp(now_ms / 1000, tz=timezone.utc) + if interval.endswith("w"): + weeks = int(interval[:-1]) + monday = (now - timedelta(days=now.weekday())).replace( + hour=0, minute=0, second=0, microsecond=0 + ) + boundary = monday + timedelta(weeks=weeks) + elif interval.endswith("M"): + months = int(interval[:-1]) + year = now.year + month = now.month + months + year += (month - 1) // 12 + month = ((month - 1) % 12) + 1 + boundary = datetime(year, month, 1, tzinfo=timezone.utc) + else: + raise ValueError(f"Unsupported recovery interval: {interval}") + return int(boundary.timestamp() * 1000) + settle_ms + + +def _payload_marker(payload: dict[str, Any] | None) -> tuple[int | None, bool]: + if not isinstance(payload, dict): + return None, False + kline = payload.get("k") if isinstance(payload.get("k"), dict) else payload + try: + open_time = int(kline.get("t")) + except (TypeError, ValueError): + return None, False + return open_time, bool(kline.get("x", False)) + + +def _closed_rows(rows: list[Any], now_ms: int) -> list[list[Any]]: + closed: list[list[Any]] = [] + for row in rows: + if not isinstance(row, list) or len(row) <= 6: + continue + try: + open_time = int(row[0]) + close_time = int(row[6]) + except (TypeError, ValueError): + continue + if close_time > now_ms or any(row[index] is None for index in range(1, 6)): + continue + if open_time >= close_time: + continue + closed.append(row) + return sorted(closed, key=lambda row: int(row[0])) + + +def _recovery_event(symbol: str, interval: str, row: list[Any]) -> dict[str, Any]: + return { + "e": "kline_recovery", + "E": int(time.time() * 1000), + "s": symbol, + "k": { + "t": row[0], + "T": row[6], + "s": symbol, + "i": interval, + "o": row[1], + "c": row[4], + "h": row[2], + "l": row[3], + "v": row[5], + "x": True, + }, + "recovery_source": "BINANCE_REST_GAP_FILL", + "provider": "binance", + "market": "binance_usdm", + "authoritative": True, + } + + +@dataclass(frozen=True) +class KlineRecoveryConfig: + enabled: bool = True + poll_seconds: float = 2.0 + settle_seconds: float = 1.0 + concurrency: int = 4 + max_limit: int = 1000 + max_backoff_seconds: float = 300.0 + queue_put_timeout_seconds: float = 2.0 + + +class DemandKlineRecovery: + """Demand-only Binance REST recovery for closed bars. + + This is a recovery projector, not a replacement WebSocket health signal. + Provider rows retain their native values and only fully closed rows enter + the existing V1 kline projection. + """ + + def __init__( + self, + *, + queue: asyncio.Queue, + redis_cache: Any, + demand_registry: Any, + config: KlineRecoveryConfig, + fetcher: Callable[..., dict[str, Any]] = binance_rest.fetch_klines, + ) -> None: + self.queue = queue + self.redis_cache = redis_cache + self.demand_registry = demand_registry + self.config = config + self.fetcher = fetcher + self.running = False + self.last_poll_at: float | None = None + self.last_success_at: float | None = None + self.last_error: str | None = None + self.provider_fetch_count = 0 + self.emitted_count = 0 + self.deduplicated_count = 0 + self.rejected_open_or_invalid_count = 0 + self.failure_count = 0 + self.active_demand_count = 0 + self._next_due_ms: dict[str, int] = {} + self._failures: dict[str, int] = {} + self._semaphore = asyncio.Semaphore(max(1, min(config.concurrency, 16))) + + def snapshot(self) -> dict[str, Any]: + return { + "enabled": self.config.enabled, + "running": self.running, + "status": ( + "disabled" + if not self.config.enabled + else "degraded" + if self.last_error + else "ready" + if self.running + else "stopped" + ), + "active_demand_count": self.active_demand_count, + "provider_fetch_count": self.provider_fetch_count, + "emitted_count": self.emitted_count, + "deduplicated_count": self.deduplicated_count, + "rejected_open_or_invalid_count": self.rejected_open_or_invalid_count, + "failure_count": self.failure_count, + "last_poll_at": self.last_poll_at, + "last_success_at": self.last_success_at, + "last_error": self.last_error, + } + + async def run(self) -> None: + if not self.config.enabled: + return + self.running = True + try: + while True: + try: + await self.poll_once() + except Exception as exc: + self.failure_count += 1 + self.last_error = f"{type(exc).__name__}: {exc}" + logger.warning("Binance kline recovery poll failed: %s", exc) + await asyncio.sleep(max(0.25, self.config.poll_seconds)) + except asyncio.CancelledError: + raise + finally: + self.running = False + + async def poll_once(self, *, now_ms: int | None = None) -> dict[str, int]: + now_ms = now_ms or int(time.time() * 1000) + self.last_poll_at = time.time() + demands = await self.demand_registry.snapshot() + selected: list[dict[str, str | None]] = [] + invalid_demand_count = 0 + active_keys: set[str] = set() + for key in demands.get("feed_keys", []): + parsed = parse_feed_key(key) + if parsed["source"] != "binance_usdm" or parsed["feed"] != "kline": + continue + if not parsed["symbol"] or not parsed["interval"]: + continue + try: + normalized_interval = binance_rest.normalize_interval(str(parsed["interval"])) + except ValueError as exc: + invalid_demand_count += 1 + self.failure_count += 1 + self.last_error = f"invalid demand {key}: {exc}" + continue + parsed["interval"] = normalized_interval + selected.append(parsed) + active_keys.add( + f"kline:binance_usdm:{normalized_interval}:{parsed['symbol']}" + ) + + self.active_demand_count = len(selected) + for key in set(self._next_due_ms) - active_keys: + self._next_due_ms.pop(key, None) + self._failures.pop(key, None) + + due = [ + item + for item in selected + if now_ms >= self._next_due_ms.get( + f"kline:binance_usdm:{item['interval']}:{item['symbol']}", 0 + ) + ] + if not due: + if not selected and invalid_demand_count == 0: + self.last_error = None + return {"due": 0, "emitted": 0, "failed": invalid_demand_count} + + outcomes = await asyncio.gather( + *(self._recover_one(item, now_ms=now_ms) for item in due), + return_exceptions=True, + ) + emitted = sum(value for value in outcomes if isinstance(value, int)) + failed = sum(1 for value in outcomes if isinstance(value, BaseException)) + if failed == 0: + self.last_error = None + return {"due": len(due), "emitted": emitted, "failed": failed} + + async def _recover_one(self, item: dict[str, str | None], *, now_ms: int) -> int: + symbol = str(item["symbol"]) + interval = str(item["interval"]) + feed_key = f"kline:binance_usdm:{interval}:{symbol}" + try: + async with self._semaphore: + existing = await self.redis_cache.get_binance_kline_last(symbol, interval) + existing_open, existing_final = _payload_marker(existing) + interval_ms = _fixed_interval_ms(interval) + if existing_open is None or interval_ms is None: + limit = 3 + else: + missing = math.ceil(max(0, now_ms - existing_open) / interval_ms) + 2 + limit = min(self.config.max_limit, max(3, missing)) + self.provider_fetch_count += 1 + payload = await asyncio.to_thread( + self.fetcher, + symbol, + interval, + limit, + None, + None, + "usdm", + ) + + raw_rows = payload.get("data") or [] + rows = _closed_rows(raw_rows, now_ms) + self.rejected_open_or_invalid_count += max(0, len(raw_rows) - len(rows)) + if existing_open is None and rows: + rows = rows[-1:] + + emitted = 0 + current_open = existing_open + current_final = existing_final + for row in rows: + open_time = int(row[0]) + if current_open is not None and ( + open_time < current_open or (open_time == current_open and current_final) + ): + self.deduplicated_count += 1 + continue + event = _recovery_event(symbol, interval, row) + await asyncio.wait_for( + self.queue.put(("binance_futures_kline", event)), + timeout=self.config.queue_put_timeout_seconds, + ) + current_open = open_time + current_final = True + emitted += 1 + + self.emitted_count += emitted + self.last_success_at = time.time() + self._failures[feed_key] = 0 + self._next_due_ms[feed_key] = _next_close_ms( + now_ms, + interval, + int(self.config.settle_seconds * 1000), + ) + return emitted + except Exception as exc: + self.failure_count += 1 + self.last_error = f"{type(exc).__name__}: {exc}" + failures = self._failures.get(feed_key, 0) + 1 + self._failures[feed_key] = failures + backoff = min( + self.config.max_backoff_seconds, + max(self.config.poll_seconds, self.config.poll_seconds * (2 ** min(failures, 8))), + ) + self._next_due_ms[feed_key] = now_ms + int(backoff * 1000) + logger.warning( + "Binance demanded kline recovery failed symbol=%s interval=%s backoff=%.1fs error=%s", + symbol, + interval, + backoff, + exc, + ) + raise diff --git a/app/stream/supervisor.py b/app/stream/supervisor.py index ac998a2..7a96ab9 100644 --- a/app/stream/supervisor.py +++ b/app/stream/supervisor.py @@ -29,6 +29,8 @@ class ShardState: last_connected_at: Optional[float] = None last_message_at: Optional[float] = None last_error: Optional[str] = None + data_timeout_count: int = 0 + last_data_timeout_at: Optional[float] = None outage_started_at: Optional[float] = None last_disconnected_at: Optional[float] = None last_recovered_at: Optional[float] = None @@ -52,6 +54,8 @@ def to_dict(self) -> Dict[str, Any]: "last_connected_at": _iso(self.last_connected_at), "last_message_at": _iso(self.last_message_at), "last_error": self.last_error, + "data_timeout_count": self.data_timeout_count, + "last_data_timeout_at": _iso(self.last_data_timeout_at), "outage_started_at": _iso(self.outage_started_at), "last_disconnected_at": _iso(self.last_disconnected_at), "last_recovered_at": _iso(self.last_recovered_at), @@ -116,9 +120,11 @@ def __init__( sample_limit: int = 10, startup_grace_seconds: float = 180.0, strict_feed_health: bool = False, + first_frame_timeout_seconds: float = 15.0, ): self.stale_after_seconds = stale_after_seconds self.startup_grace_seconds = startup_grace_seconds + self.first_frame_timeout_seconds = first_frame_timeout_seconds self.strict_feed_health = strict_feed_health self.sample_limit = sample_limit self.started_at = _now() @@ -129,6 +135,8 @@ def __init__( self.queue_drop_count = 0 self.queue_drop_window_seconds = 300.0 self._queue_drop_times: deque[float] = deque(maxlen=100_000) + self.queue_pressure_count = 0 + self.last_queue_pressure_at: Optional[float] = None self.redis_error_count = 0 self.last_redis_error: Optional[str] = None self.publisher_batch_count = 0 @@ -186,7 +194,15 @@ def mark_connected(self, shard_id: str) -> bool: now = _now() shard.status = "connected" shard.last_connected_at = now - shard.last_error = None + data_timeout_active = bool( + shard.last_data_timeout_at + and ( + not shard.last_message_at + or shard.last_data_timeout_at >= shard.last_message_at + ) + ) + if not data_timeout_active: + shard.last_error = None if recovered: duration = max(0.0, now - float(shard.outage_started_at)) shard.last_outage_seconds = duration @@ -202,6 +218,16 @@ def mark_message(self, shard_id: str) -> None: if shard: shard.message_count += 1 shard.last_message_at = _now() + if shard.last_error and shard.last_error.startswith("data_timeout:"): + shard.last_error = None + + def mark_data_timeout(self, shard_id: str, reason: str) -> None: + shard = self.shards.get(shard_id) + if shard: + now = _now() + shard.data_timeout_count += 1 + shard.last_data_timeout_at = now + shard.last_error = f"data_timeout:{reason}" def mark_parse_error(self, shard_id: str, error: Exception) -> None: shard = self.shards.get(shard_id) @@ -244,6 +270,10 @@ def record_queue_drop(self, source: str, shard_id: Optional[str] = None) -> None if shard_id and shard_id in self.shards: self.shards[shard_id].queue_drop_count += 1 + def record_queue_pressure(self) -> None: + self.queue_pressure_count += 1 + self.last_queue_pressure_at = _now() + def record_redis_error(self, error: Exception) -> None: self.redis_error_count += 1 self.last_redis_error = str(error) @@ -300,6 +330,68 @@ def record_batch_published(self, size: int) -> None: self.publisher_batch_count += 1 self.last_publisher_at = _now() + def _source_states(self, now: float) -> Dict[str, Dict[str, Any]]: + states: Dict[str, Dict[str, Any]] = {} + for source in sorted({shard.source for shard in self.shards.values()}): + shards = [shard for shard in self.shards.values() if shard.source == source] + connected = [shard for shard in shards if shard.status == "connected"] + producing = [] + waiting = [] + stale = [] + unavailable = [] + for shard in shards: + session_has_frame = bool( + shard.status == "connected" + and shard.last_message_at + and shard.last_connected_at + and shard.last_message_at >= shard.last_connected_at + ) + if session_has_frame: + age = max(0.0, now - float(shard.last_message_at)) + if age <= self.stale_after_seconds: + producing.append(shard) + else: + stale.append(shard) + elif shard.status == "connected" and shard.last_connected_at: + age = max(0.0, now - float(shard.last_connected_at)) + data_timeout_active = bool( + shard.last_data_timeout_at + and ( + not shard.last_message_at + or shard.last_data_timeout_at >= shard.last_message_at + ) + ) + if not data_timeout_active and age <= self.first_frame_timeout_seconds: + waiting.append(shard) + else: + unavailable.append(shard) + else: + unavailable.append(shard) + + if len(producing) == len(shards) and shards: + status = "ready" + elif producing: + status = "degraded" + elif waiting and len(waiting) == len(shards): + status = "starting" + else: + status = "unavailable" + states[source] = { + "feed": "trade" if source.endswith("_trade") else "kline", + "status": status, + "transport_ready": len(connected) == len(shards) and bool(shards), + "data_ready": status == "ready", + "shard_count": len(shards), + "connected_count": len(connected), + "producing_count": len(producing), + "waiting_first_frame_count": len(waiting), + "stale_count": len(stale), + "unavailable_count": len(unavailable), + "first_frame_timeout_seconds": self.first_frame_timeout_seconds, + "stale_after_seconds": self.stale_after_seconds, + } + return states + def snapshot( self, now: Optional[float] = None, @@ -344,6 +436,15 @@ def snapshot( ] reconnect_count = sum(shard.reconnect_count for shard in self.shards.values()) connected_shards = [shard for shard in self.shards.values() if shard.status == "connected"] + source_states = self._source_states(now) + unavailable_sources = [ + source + for source, state in source_states.items() + if state["status"] in {"degraded", "unavailable"} + ] + starting_sources = [ + source for source, state in source_states.items() if state["status"] == "starting" + ] health_warnings = [] cutoff = now - self.queue_drop_window_seconds @@ -361,13 +462,19 @@ def snapshot( health_warnings.append("missing_expected_feeds") if stale: health_warnings.append("stale_expected_feeds") + if unavailable_sources: + health_warnings.append("source_data_unavailable") if self.redis_error_count: status = "degraded" elif not self.shards: status = "not_started" + elif unavailable_sources: + status = "degraded" elif not connected_shards: status = "starting" + elif starting_sources: + status = "starting" elif demanded_stale or (demanded_missing and uptime_seconds > self.startup_grace_seconds): status = "degraded" elif self.strict_feed_health and (recent_queue_drops or broad_health_missing or stale): @@ -388,6 +495,8 @@ def snapshot( "maxsize": self.queue_maxsize, "drop_count": self.queue_drop_count, "recent_drop_count": recent_queue_drops, + "pressure_count": self.queue_pressure_count, + "last_pressure_at": _iso(self.last_queue_pressure_at), "window_seconds": self.queue_drop_window_seconds, }, "publisher": { @@ -403,6 +512,7 @@ def snapshot( "reconnect_count": reconnect_count, "items": [s.to_dict() for s in list(self.shards.values())[: self.sample_limit]], }, + "sources": source_states, "feeds": { "expected_count": len(expected_feeds), "observed_count": len([feed for feed in expected_feeds if feed.last_published_at]), diff --git a/config/phase9/candidate-slice.yaml b/config/phase9/candidate-slice.yaml new file mode 100644 index 0000000..350eb8d --- /dev/null +++ b/config/phase9/candidate-slice.yaml @@ -0,0 +1,28 @@ +schema: qdl.authority-candidate.v1 +slice_id: production/binance/usdm/perpetual/trade/plan-1/btcusdt +environment: production +venue: BINANCE +market: USDM +product_type: PERPETUAL +feed: TRADE +instrument_uids: + - 85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c +partition_plan_epoch: 1 +partition_id: rendezvous-sha256-v1:epoch-1:btcusdt +schema_major: 2 +authority_state: RUST_SHADOW +owner_id: rust-ingestor-binance-usdm-shard-0 +lease_epoch: 1 +artifact_image_digest: sha256:46a7c3fa516c0035c3ce41add0ce77e9acb4d4dfd1b0ac74130c894ca7ad5280 +sbom_digest: eab28f6b6685d6f2e748d5f12b540f2ced62c6f8f4686a1dc31d67084aaa5d36 +contract_digest: f26d0369d73db3c6943e08824c62925507b91f6bdabef4636f48f3037734ebec +partition_plan_digest: 72baa5414df483293389a8da6a69ddbda8ac7e169e1cd4234a1c9477412e9781 +rollback_manifest_digest: 36acbeacf0e26efbfb38973957331dd383f639aecd08dc4cedfa5f3812f451f1 +signature_identity: phase8-rehearsal-only-not-admitted +normalizer_version: qdl-rust-core/0.1.0-beta +adapter_version: binance-usdm/2.0.0-shadow +config_revision: phase8-candidate-1 +instrument_catalog_revision: phase8-catalog-1 +source_policy_revision: binance-usdm-trade-shadow-1 +public_write_allowed: false +legacy_write_allowed: false diff --git a/config/phase9/evidence-inventory.local.yaml b/config/phase9/evidence-inventory.local.yaml new file mode 100644 index 0000000..34a5c7c --- /dev/null +++ b/config/phase9/evidence-inventory.local.yaml @@ -0,0 +1,101 @@ +schema: qdl.production-prerequisite-inventory.v1 +evidence: + - evidence_id: phase8-local-kafka-failover + gate_id: replicated_durable_transport + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-broker-failover.json + artifact_sha256: a9b43e4d70b2002032d9c397aea0bb265c50122969f3e3c808f1b284a353ce55 + details: + replication_factor: 3 + min_isr: 2 + acks_all: true + idempotent_producer: true + broker_failover: true + restore_test_id: phase8-same-host-volume-loss + - evidence_id: phase8-local-otel-debug + gate_id: production_observability + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: config/observability/phase8-otel-collector.yaml + artifact_sha256: 915e0ac483b436aaf9f50ffee4cb7196e74aafd2425f0c9f34318089ed1a86c9 + details: + collector_backend: debug-only + dashboard_revision: not-deployed + alert_route_id: not-deployed + page_ack_id: not-deployed + - evidence_id: phase8-local-kafka-mtls + gate_id: workload_identity_rbac_network + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-broker-security.json + artifact_sha256: 289dd9a2cda61121bc7518718b3b5fdde976235e212720ffd28389e26ecc6a5b + details: + identity_provider: ephemeral-self-signed-mtls + rbac_revision: phase8-kafka-acl + network_policy_revision: same-host-internal-compose + - evidence_id: phase8-signature-rehearsal-no-admission + gate_id: signed_artifact_admission + environment: production + scope: LOCAL_REHEARSAL + status: BLOCKED + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: null + artifact_sha256: null + details: + reason: registry-admission-not-installed + - evidence_id: phase8-local-projection-rebuild + gate_id: redis_projector_rebuild + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-broker-topology.json + artifact_sha256: 68d2d6e409c1cf5428ce437e753653b24ddd16afc8e7cff208dd087b3ae1d500 + details: + canonical_watermark: phase8-offset-65 + rebuild_checksum: phase8-isolated-equal + source_log_cluster: qdl-phase8-same-host + - evidence_id: phase90b-partial-consumer-beta + gate_id: consumer_registration_rollback + environment: production + scope: LOCAL_REHEARSAL + status: BLOCKED + issuer: qdl-phase90b-certification + observed_at_ns: 1787061600000000000 + expires_at_ns: 1789653600000000000 + artifact_path: null + artifact_sha256: null + details: + reason: beta-consumers-do-not-cover-trade-authority-slice + - evidence_id: phase8-local-authority-rehearsal + gate_id: persistent_authority_sink_fencing + environment: production + scope: LOCAL_REHEARSAL + status: PASS + issuer: qdl-phase8-certification + observed_at_ns: 1786806000000000000 + expires_at_ns: 1789398000000000000 + artifact_path: upgrade/evidence/phase8-authority-rehearsal.json + artifact_sha256: ec5efe8b99687055eb42ae481b00201937a297148e479c6822b0fa3d4fb29f4a + details: + authority_store: same-host-compacted-kafka + transition_audit_id: phase8-shadow-canary-shadow + zombie_writer_rejected: true + candidate_digest: 72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037 diff --git a/config/phase9/production-prerequisite-policy.yaml b/config/phase9/production-prerequisite-policy.yaml new file mode 100644 index 0000000..846c274 --- /dev/null +++ b/config/phase9/production-prerequisite-policy.yaml @@ -0,0 +1,236 @@ +schema: qdl.production-prerequisite-policy.v1 +revision: 1 +environment: production +gates: +- id: replicated_durable_transport + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - replication_factor + - min_isr + - acks_all + - idempotent_producer + - broker_failover + - restore_test_id + - broker_count + - failure_domain_count + - unclean_leader_election + - restore_verified + candidate_bound: false + assertions: + acks_all: true + idempotent_producer: true + broker_failover: true + restore_verified: true + unclean_leader_election: false + minimums: + broker_count: 3 + replication_factor: 3 + min_isr: 2 + failure_domain_count: 2 + maximums: {} + candidate_field_matches: {} +- id: production_observability + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - collector_backend + - dashboard_revision + - alert_route_id + - page_ack_id + - backend_persistent + - collector_ha + - alert_acknowledged + - retention_days + candidate_bound: false + assertions: + backend_persistent: true + collector_ha: true + alert_acknowledged: true + minimums: + retention_days: 30 + maximums: {} + candidate_field_matches: {} +- id: workload_identity_rbac_network + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - identity_provider + - rbac_revision + - network_policy_revision + - short_lived_credentials + - rbac_enforced + - network_default_deny + candidate_bound: false + assertions: + short_lived_credentials: true + rbac_enforced: true + network_default_deny: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: external_secret_rotation + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - secret_backend + - rotation_event_id + - retired_key_id + - rotation_passed + - retired_key_rejected + candidate_bound: false + assertions: + rotation_passed: true + retired_key_rejected: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: signed_artifact_admission + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - registry + - signature_identity + - admission_policy_revision + - image_digest + - candidate_digest + - signature_verified + - unsigned_rejected + candidate_bound: true + assertions: + signature_verified: true + unsigned_rejected: true + minimums: {} + maximums: {} + candidate_field_matches: + image_digest: artifact_image_digest + signature_identity: signature_identity +- id: postgres_pitr + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - backup_id + - restore_id + - rpo_seconds + - rto_seconds + - restore_verified + - checksum_equal + candidate_bound: false + assertions: + restore_verified: true + checksum_equal: true + minimums: {} + maximums: + rpo_seconds: 300 + rto_seconds: 3600 + candidate_field_matches: {} +- id: object_store_restore + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - snapshot_id + - restore_id + - manifest_digest + - restore_verified + - checksum_equal + candidate_bound: false + assertions: + restore_verified: true + checksum_equal: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: independent_failure_domain_dr + minimum_scope: INDEPENDENT_FAILURE_DOMAIN + max_age_seconds: 2592000 + required_details: + - source_domain + - target_domain + - exercise_id + - rpo_seconds + - rto_seconds + - failover_verified + - duplicate_external_publications + - failure_domain_count + candidate_bound: false + assertions: + failover_verified: true + duplicate_external_publications: 0 + minimums: + failure_domain_count: 2 + maximums: + rpo_seconds: 300 + rto_seconds: 3600 + candidate_field_matches: {} +- id: redis_projector_rebuild + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - canonical_watermark + - rebuild_checksum + - source_log_cluster + - checksum_equal + - zero_production_mutation + - replay_lag_events + candidate_bound: false + assertions: + checksum_equal: true + zero_production_mutation: true + minimums: {} + maximums: + replay_lag_events: 0 + candidate_field_matches: {} +- id: consumer_registration_rollback + minimum_scope: PRODUCTION + max_age_seconds: 604800 + required_details: + - consumers_digest + - owners_approved + - rollback_rehearsal_id + - candidate_digest + - rollback_rehearsed + - unregistered_critical_consumers + candidate_bound: true + assertions: + owners_approved: true + rollback_rehearsed: true + minimums: {} + maximums: + unregistered_critical_consumers: 0 + candidate_field_matches: {} +- id: persistent_authority_sink_fencing + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - authority_store + - transition_audit_id + - zombie_writer_rejected + - candidate_digest + - sink_side_fencing + - audit_immutable + candidate_bound: true + assertions: + zombie_writer_rejected: true + sink_side_fencing: true + audit_immutable: true + minimums: {} + maximums: {} + candidate_field_matches: {} +- id: exact_slice_approval + minimum_scope: PRODUCTION + max_age_seconds: 86400 + required_details: + - approved_by + - change_ticket + - hold_until_ns + - blast_radius + - candidate_digest + - approved + - approver_count + candidate_bound: true + assertions: + approved: true + minimums: + approver_count: 1 + maximums: {} + candidate_field_matches: {} diff --git a/config/v2/stable-acquisition-bindings.yaml b/config/v2/stable-acquisition-bindings.yaml new file mode 100644 index 0000000..bf26075 --- /dev/null +++ b/config/v2/stable-acquisition-bindings.yaml @@ -0,0 +1,23 @@ +schema: qdl.v2.stable-acquisition-bindings.v1 +revision: 3 +topics: + raw: md.raw.stable.v1 + canonical: md.canonical.v2 + quarantine: md.quarantine.stable.v1 +bindings: +- {binding_id: binance-usdm-btcusdt-trade, mode: RUST_NATIVE, runtime: BINANCE, provider_kind: binance_usdm_trade, native_channel: "btcusdt@trade", sequence_policy: MONOTONIC, websocket_url: "wss://fstream.binance.com/public/stream", business_websocket_url: null} +- {binding_id: binance-usdm-btcusdt-quote, mode: RUST_NATIVE, runtime: BINANCE, provider_kind: binance_usdm_bbo, native_channel: "btcusdt@bookTicker", sequence_policy: MONOTONIC, websocket_url: "wss://fstream.binance.com/public/stream", business_websocket_url: null} +- {binding_id: binance-usdm-btcusdt-bar-1m, mode: PYTHON_REST, runtime: BINANCE, provider_kind: binance_usdm_rest_bar, native_channel: "rest-klines/1m", sequence_policy: NONE, websocket_url: null, business_websocket_url: null} +- {binding_id: binance-spot-btcusdt-trade, mode: RUST_NATIVE, runtime: BINANCE, provider_kind: binance_spot_trade, native_channel: "btcusdt@trade", sequence_policy: MONOTONIC, websocket_url: "wss://stream.binance.com:9443/stream", business_websocket_url: null} +- {binding_id: binance-spot-btcusdt-quote, mode: RUST_NATIVE, runtime: BINANCE, provider_kind: binance_spot_bbo, native_channel: "btcusdt@bookTicker", sequence_policy: MONOTONIC, websocket_url: "wss://stream.binance.com:9443/stream", business_websocket_url: null} +- {binding_id: binance-spot-btcusdt-bar-1m, mode: PYTHON_REST, runtime: BINANCE, provider_kind: binance_spot_rest_bar, native_channel: "rest-klines/1m", sequence_policy: NONE, websocket_url: null, business_websocket_url: null} +- {binding_id: okx-swap-btcusdt-trade, mode: RUST_NATIVE, runtime: OKX, provider_kind: okx_trade, native_channel: trades, sequence_policy: MONOTONIC, websocket_url: "wss://ws.okx.com:8443/ws/v5/public", business_websocket_url: "wss://ws.okx.com:8443/ws/v5/business"} +- {binding_id: okx-swap-btcusdt-quote, mode: RUST_NATIVE, runtime: OKX, provider_kind: okx_bbo, native_channel: bbo-tbt, sequence_policy: NONE, websocket_url: "wss://ws.okx.com:8443/ws/v5/public", business_websocket_url: "wss://ws.okx.com:8443/ws/v5/business"} +- {binding_id: okx-swap-btcusdt-bar-1m, mode: PYTHON_REST, runtime: OKX, provider_kind: okx_bar, native_channel: candle1m, sequence_policy: NONE, websocket_url: null, business_websocket_url: null} +- {binding_id: okx-spot-btcusdt-trade, mode: RUST_NATIVE, runtime: OKX, provider_kind: okx_trade, native_channel: trades, sequence_policy: MONOTONIC, websocket_url: "wss://ws.okx.com:8443/ws/v5/public", business_websocket_url: "wss://ws.okx.com:8443/ws/v5/business"} +- {binding_id: okx-spot-btcusdt-quote, mode: RUST_NATIVE, runtime: OKX, provider_kind: okx_bbo, native_channel: bbo-tbt, sequence_policy: NONE, websocket_url: "wss://ws.okx.com:8443/ws/v5/public", business_websocket_url: "wss://ws.okx.com:8443/ws/v5/business"} +- {binding_id: okx-spot-btcusdt-bar-1m, mode: PYTHON_REST, runtime: OKX, provider_kind: okx_bar, native_channel: candle1m, sequence_policy: NONE, websocket_url: null, business_websocket_url: null} +- {binding_id: dnse-vn30f1m-trade, mode: PYTHON_VENDOR_SDK, runtime: DNSE, provider_kind: dnse_trade, native_channel: trades, sequence_policy: NONE, websocket_url: null, business_websocket_url: null} +- {binding_id: dnse-vn30f1m-bar-1m, mode: PYTHON_VENDOR_SDK, runtime: DNSE, provider_kind: dnse_bar, native_channel: "ohlcv/1m", sequence_policy: NONE, websocket_url: null, business_websocket_url: null} +- {binding_id: dnse-fpt-trade, mode: PYTHON_VENDOR_SDK, runtime: DNSE, provider_kind: dnse_trade, native_channel: trades, sequence_policy: NONE, websocket_url: null, business_websocket_url: null} +- {binding_id: dnse-fpt-bar-1m, mode: PYTHON_VENDOR_SDK, runtime: DNSE, provider_kind: dnse_bar, native_channel: "ohlcv/1m", sequence_policy: NONE, websocket_url: null, business_websocket_url: null} diff --git a/config/v2/stable-capabilities.yaml b/config/v2/stable-capabilities.yaml new file mode 100644 index 0000000..83992ae --- /dev/null +++ b/config/v2/stable-capabilities.yaml @@ -0,0 +1,72 @@ +schema: qdl.v2.stable-capability-matrix.v2 +revision: 3 +public_contract_version: 2.0.0 +realtime_core_target: RUST +python_role: OUTER_AND_VENDOR_ACQUISITION_EDGE +runtime_authority: RUST_SHADOW +authority_eligible: false +equal_source_contract: true +capabilities: + - venue: BINANCE + market: USDM + product_type: PERPETUAL + acquisition_adapter: qdl-capability-routed-binance-edge + feed_adapters: + TRADE: qdl-rust-binance-json + BBO: qdl-rust-binance-json + BAR: qdl-python-binance-rest-latest-closed + canonical_core: qdl-rust-realtime-core + feeds: [TRADE, BBO, BAR] + certification: BOUNDED_REAL_PROVIDER_SHADOW + - venue: BINANCE + market: SPOT + product_type: SPOT + acquisition_adapter: qdl-capability-routed-binance-edge + feed_adapters: + TRADE: qdl-rust-binance-json + BBO: qdl-rust-binance-json + BAR: qdl-python-binance-rest-latest-closed + canonical_core: qdl-rust-realtime-core + feeds: [TRADE, BBO, BAR] + certification: BOUNDED_REAL_PROVIDER_SHADOW + - venue: OKX + market: SWAP + product_type: PERPETUAL + acquisition_adapter: qdl-rust-okx-json + canonical_core: qdl-rust-realtime-core + feeds: [TRADE, BBO, BAR] + certification: BOUNDED_REAL_PROVIDER_SHADOW + - venue: OKX + market: SPOT + product_type: SPOT + acquisition_adapter: qdl-rust-okx-json + canonical_core: qdl-rust-realtime-core + feeds: [TRADE, BBO, BAR] + certification: BOUNDED_REAL_PROVIDER_SHADOW + - venue: HNX + market: VN_DERIVATIVES + product_type: FUTURE + acquisition_adapter: qdl-python-dnse-vendor-edge@2.0.1 + canonical_core: qdl-rust-realtime-core + feeds: [TRADE, BAR] + certification: PROVIDER_DERIVED_DURABLE_REPLAY + - venue: HOSE + market: EQUITIES + product_type: COMMON_STOCK + acquisition_adapter: qdl-python-dnse-vnstock-policy-edge@2.0.1 + canonical_core: qdl-rust-realtime-core + feeds: [TRADE, BAR] + certification: PROVIDER_DERIVED_DURABLE_REPLAY +capability_gates: + CRYPTO_L2: + stable: false + reason: requires equal Binance/OKX snapshot-delta resync certification + VN_QUOTE: + stable: false + reason: requires a real DNSE/vnstock bid-ask subscription and source-switch certification + OKX_SBE: + stable: false + reason: entitlement and JSON parity required + OKX_TRADES_ALL: + stable: false + reason: atomic trade-count capability must be activated explicitly diff --git a/config/v2/stable-consumer-migration.yaml b/config/v2/stable-consumer-migration.yaml new file mode 100644 index 0000000..adb8e9d --- /dev/null +++ b/config/v2/stable-consumer-migration.yaml @@ -0,0 +1,31 @@ +schema: qdl.v2.stable-consumer-migration.v1 +revision: 1 +contract_version: 2.0.0 +authority: V1 +target_route: V1_WITH_V2_SHADOW +consumers: +- manifest: /app/consumers/stable/monitoring-multivenue.yaml + consumer_id: monitoring.multivenue.stable + state: SHADOW + rollback_route: V1 + cutover_authorized: false +- manifest: /app/consumers/stable/alpha-binance-paper.yaml + consumer_id: alpha.binance.paper.stable + state: SHADOW + rollback_route: V1 + cutover_authorized: false +- manifest: /app/consumers/stable/alpha-okx-paper.yaml + consumer_id: alpha.okx.paper.stable + state: SHADOW + rollback_route: V1 + cutover_authorized: false +- manifest: /app/consumers/stable/alpha-vn-paper.yaml + consumer_id: alpha.vn.paper.stable + state: SHADOW + rollback_route: V1 + cutover_authorized: false +- manifest: /app/consumers/stable/trading-system-paper.yaml + consumer_id: trading-system.paper.stable + state: SHADOW + rollback_route: V1 + cutover_authorized: false diff --git a/config/v2/stable-source-bindings.yaml b/config/v2/stable-source-bindings.yaml new file mode 100644 index 0000000..04cdcb1 --- /dev/null +++ b/config/v2/stable-source-bindings.yaml @@ -0,0 +1,387 @@ +schema: qdl.v2.stable-source-bindings.v1 +canonical_stream: md.canonical.v2 +catalog_revision: 2 +source_policy_revision: 1 +authority_revision: 1 +instruments: +- instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + instrument_id: BINANCE.USDM.PERPETUAL.BTC-USDT + metadata_revision: 1 + venue: BINANCE + market: USDM + product_type: PERPETUAL + canonical_symbol: BTC-USDT + native_symbol: BTCUSDT + asset_class: DERIVATIVE + base_asset: BTC + quote_asset: USDT + settlement_asset: USDT + price_tick: '0.10' + quantity_step: '0.001' + contract_multiplier: '1' + session_calendar_id: CRYPTO_24X7 + attributes: + contract_type: PERPETUAL +- instrument_uid: 26edfffd-6824-5e75-a620-5a122b3e1086 + instrument_id: BINANCE.SPOT.SPOT.BTC-USDT + metadata_revision: 1 + venue: BINANCE + market: SPOT + product_type: SPOT + canonical_symbol: BTC-USDT + native_symbol: BTCUSDT + asset_class: CRYPTO + base_asset: BTC + quote_asset: USDT + settlement_asset: USDT + price_tick: '0.10' + quantity_step: '0.000001' + contract_multiplier: '1' + session_calendar_id: CRYPTO_24X7 + attributes: {} +- instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + instrument_id: OKX.SWAP.PERPETUAL.BTC-USDT + metadata_revision: 1 + venue: OKX + market: SWAP + product_type: PERPETUAL + canonical_symbol: BTC-USDT + native_symbol: BTC-USDT-SWAP + asset_class: DERIVATIVE + base_asset: BTC + quote_asset: USDT + settlement_asset: USDT + price_tick: '0.10' + quantity_step: '1' + contract_multiplier: '0.01' + session_calendar_id: CRYPTO_24X7 + attributes: + instrument_type: SWAP + contract_type: linear +- instrument_uid: 9e996914-aa98-5ea0-b0a0-a78dc51e1f8d + instrument_id: OKX.SPOT.SPOT.BTC-USDT + metadata_revision: 1 + venue: OKX + market: SPOT + product_type: SPOT + canonical_symbol: BTC-USDT + native_symbol: BTC-USDT + asset_class: CRYPTO + base_asset: BTC + quote_asset: USDT + settlement_asset: USDT + price_tick: '0.10' + quantity_step: '0.000001' + contract_multiplier: '1' + session_calendar_id: CRYPTO_24X7 + attributes: + instrument_type: SPOT +- instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 + instrument_id: HNX.VN_DERIVATIVES.FUTURE.VN30F1M + metadata_revision: 1 + venue: HNX + market: VN_DERIVATIVES + product_type: FUTURE + canonical_symbol: VN30F1M + native_symbol: VN30F1M + asset_class: DERIVATIVE + base_asset: VN30 + quote_asset: VND + settlement_asset: VND + price_tick: '0.1' + quantity_step: '1' + contract_multiplier: '100000' + session_calendar_id: VN_DERIVATIVES + attributes: + continuous_series: 'true' + roll_policy: front_month_provider_alias +- instrument_uid: 6f025ab7-c0b4-5e08-866d-34ce62aef52f + instrument_id: HOSE.EQUITIES.COMMON_STOCK.FPT + metadata_revision: 1 + venue: HOSE + market: EQUITIES + product_type: COMMON_STOCK + canonical_symbol: FPT + native_symbol: FPT + asset_class: EQUITY + base_asset: FPT + quote_asset: VND + settlement_asset: VND + price_tick: '100' + quantity_step: '100' + contract_multiplier: '1' + session_calendar_id: VN_EQUITIES + attributes: {} +bindings: +- binding_id: binance-usdm-btcusdt-trade + instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: TRADE + interval: null + source: + provider: BINANCE_DIRECT + source_id: binance-usdm-btcusdt-trade-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: binance-usdm/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 15000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: BINANCE_TRADE_MARKET_AND_GENERIC +- binding_id: binance-usdm-btcusdt-quote + instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: QUOTE + interval: null + source: + provider: BINANCE_DIRECT + source_id: binance-usdm-btcusdt-quote-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: binance-usdm/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 5000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: NONE +- binding_id: binance-usdm-btcusdt-bar-1m + instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: BAR + interval: 1m + source: + provider: BINANCE_DIRECT + source_id: binance-usdm-btcusdt-bar-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: binance-rest/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 180000 + require_final_bar: true + continuous_calendar: true + v1_compatibility: BINANCE_BAR_GENERIC +- binding_id: binance-spot-btcusdt-trade + instrument_uid: 26edfffd-6824-5e75-a620-5a122b3e1086 + feed: TRADE + interval: null + source: + provider: BINANCE_DIRECT + source_id: binance-spot-btcusdt-trade-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: binance-spot/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 15000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: BINANCE_TRADE_MARKET_ONLY +- binding_id: binance-spot-btcusdt-quote + instrument_uid: 26edfffd-6824-5e75-a620-5a122b3e1086 + feed: QUOTE + interval: null + source: + provider: BINANCE_DIRECT + source_id: binance-spot-btcusdt-quote-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: binance-spot/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 5000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: NONE +- binding_id: binance-spot-btcusdt-bar-1m + instrument_uid: 26edfffd-6824-5e75-a620-5a122b3e1086 + feed: BAR + interval: 1m + source: + provider: BINANCE_DIRECT + source_id: binance-spot-btcusdt-bar-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: binance-rest/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 180000 + require_final_bar: true + continuous_calendar: true + v1_compatibility: NONE +- binding_id: okx-swap-btcusdt-trade + instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: TRADE + interval: null + source: + provider: OKX_DIRECT + source_id: okx-swap-btcusdt-trade-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: okx-v5/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 15000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: NONE +- binding_id: okx-swap-btcusdt-quote + instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: QUOTE + interval: null + source: + provider: OKX_DIRECT + source_id: okx-swap-btcusdt-quote-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: okx-v5/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 5000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: NONE +- binding_id: okx-swap-btcusdt-bar-1m + instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: BAR + interval: 1m + source: + provider: OKX_DIRECT + source_id: okx-swap-btcusdt-bar-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: okx-v5/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 180000 + require_final_bar: true + continuous_calendar: true + v1_compatibility: NONE +- binding_id: okx-spot-btcusdt-trade + instrument_uid: 9e996914-aa98-5ea0-b0a0-a78dc51e1f8d + feed: TRADE + interval: null + source: + provider: OKX_DIRECT + source_id: okx-spot-btcusdt-trade-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: okx-v5/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 15000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: NONE +- binding_id: okx-spot-btcusdt-quote + instrument_uid: 9e996914-aa98-5ea0-b0a0-a78dc51e1f8d + feed: QUOTE + interval: null + source: + provider: OKX_DIRECT + source_id: okx-spot-btcusdt-quote-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: okx-v5/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 5000 + require_final_bar: false + continuous_calendar: true + v1_compatibility: NONE +- binding_id: okx-spot-btcusdt-bar-1m + instrument_uid: 9e996914-aa98-5ea0-b0a0-a78dc51e1f8d + feed: BAR + interval: 1m + source: + provider: OKX_DIRECT + source_id: okx-spot-btcusdt-bar-stable-001 + source_role: PRIMARY + source_policy_id: crypto_primary_v2 + authoritative: true + adapter_version: okx-v5/2.0.0 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 180000 + require_final_bar: true + continuous_calendar: true + v1_compatibility: NONE +- binding_id: dnse-vn30f1m-trade + instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 + feed: TRADE + interval: null + source: + provider: DNSE_DIRECT + source_id: dnse-vn30f1m-trade-stable-001 + source_role: PRIMARY + source_policy_id: vn_primary_v2 + authoritative: true + adapter_version: dnse-openapi/2.0.1 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 15000 + require_final_bar: false + continuous_calendar: false + v1_compatibility: VN_TRADE_GENERIC +- binding_id: dnse-vn30f1m-bar-1m + instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 + feed: BAR + interval: 1m + source: + provider: DNSE_DIRECT + source_id: dnse-vn30f1m-bar-stable-001 + source_role: PRIMARY + source_policy_id: vn_primary_v2 + authoritative: true + adapter_version: dnse-openapi/2.0.1 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 180000 + require_final_bar: true + continuous_calendar: false + v1_compatibility: NONE +- binding_id: dnse-fpt-trade + instrument_uid: 6f025ab7-c0b4-5e08-866d-34ce62aef52f + feed: TRADE + interval: null + source: + provider: DNSE_DIRECT + source_id: dnse-fpt-trade-stable-001 + source_role: PRIMARY + source_policy_id: vn_primary_v2 + authoritative: true + adapter_version: dnse-openapi/2.0.1 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 15000 + require_final_bar: false + continuous_calendar: false + v1_compatibility: VN_TRADE_GENERIC +- binding_id: dnse-fpt-bar-1m + instrument_uid: 6f025ab7-c0b4-5e08-866d-34ce62aef52f + feed: BAR + interval: 1m + source: + provider: DNSE_DIRECT + source_id: dnse-fpt-bar-stable-001 + source_role: PRIMARY + source_policy_id: vn_primary_v2 + authoritative: true + adapter_version: dnse-openapi/2.0.1 + normalizer_version: qdl-rust-core/2.0.0 + quality: + stale_after_ms: 180000 + require_final_bar: true + continuous_calendar: false + v1_compatibility: NONE diff --git a/consumers/stable/alpha-binance-paper.yaml b/consumers/stable/alpha-binance-paper.yaml new file mode 100644 index 0000000..1d2136f --- /dev/null +++ b/consumers/stable/alpha-binance-paper.yaml @@ -0,0 +1,53 @@ +apiVersion: qdl/v2 +kind: DataRequirement +metadata: + id: alpha.binance.paper.stable + owner: alpha-platform + subject: spiffe://qdl/paper/alpha-binance-stable + environment: paper + revision: 1 +spec: + sdk_major: 2 + rollback_contract: V1 + execution_dependency: FORBIDDEN + permissions: + - instruments:read + - snapshot:read + - history:read + - status:read + - quality:read + - stream:read + purposes: + - INTERNAL_ALPHA + quotas: + requests_per_minute: 180 + max_batch_items: 50 + max_warmup_rows: 2000 + max_streams: 20 + max_buffer_events: 2000 + requirements: + - instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: BAR + consumer_grade: ALPHA + source_policy_id: crypto_primary_v2 + warmup_limit: 500 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: EMIT_REVISIONS + interval: 1m + - instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: TRADE + consumer_grade: ALPHA + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 15000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST diff --git a/consumers/stable/alpha-okx-paper.yaml b/consumers/stable/alpha-okx-paper.yaml new file mode 100644 index 0000000..5014393 --- /dev/null +++ b/consumers/stable/alpha-okx-paper.yaml @@ -0,0 +1,53 @@ +apiVersion: qdl/v2 +kind: DataRequirement +metadata: + id: alpha.okx.paper.stable + owner: alpha-platform + subject: spiffe://qdl/paper/alpha-okx-stable + environment: paper + revision: 1 +spec: + sdk_major: 2 + rollback_contract: V1 + execution_dependency: FORBIDDEN + permissions: + - instruments:read + - snapshot:read + - history:read + - status:read + - quality:read + - stream:read + purposes: + - INTERNAL_ALPHA + quotas: + requests_per_minute: 180 + max_batch_items: 50 + max_warmup_rows: 2000 + max_streams: 20 + max_buffer_events: 2000 + requirements: + - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: BAR + consumer_grade: ALPHA + source_policy_id: crypto_primary_v2 + warmup_limit: 500 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: EMIT_REVISIONS + interval: 1m + - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: TRADE + consumer_grade: ALPHA + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 15000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST diff --git a/consumers/stable/alpha-vn-paper.yaml b/consumers/stable/alpha-vn-paper.yaml new file mode 100644 index 0000000..8f69e69 --- /dev/null +++ b/consumers/stable/alpha-vn-paper.yaml @@ -0,0 +1,53 @@ +apiVersion: qdl/v2 +kind: DataRequirement +metadata: + id: alpha.vn.paper.stable + owner: alpha-platform + subject: spiffe://qdl/paper/alpha-vn-stable + environment: paper + revision: 1 +spec: + sdk_major: 2 + rollback_contract: V1 + execution_dependency: FORBIDDEN + permissions: + - instruments:read + - snapshot:read + - history:read + - status:read + - quality:read + - stream:read + purposes: + - INTERNAL_ALPHA + quotas: + requests_per_minute: 180 + max_batch_items: 50 + max_warmup_rows: 2000 + max_streams: 20 + max_buffer_events: 2000 + requirements: + - instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 + feed: BAR + consumer_grade: ALPHA + source_policy_id: vn_primary_v2 + warmup_limit: 500 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: EMIT_REVISIONS + interval: 1m + - instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 + feed: TRADE + consumer_grade: ALPHA + source_policy_id: vn_primary_v2 + warmup_limit: 0 + max_freshness_ms: 30000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST diff --git a/consumers/stable/monitoring-multivenue.yaml b/consumers/stable/monitoring-multivenue.yaml new file mode 100644 index 0000000..88d2040 --- /dev/null +++ b/consumers/stable/monitoring-multivenue.yaml @@ -0,0 +1,65 @@ +apiVersion: qdl/v2 +kind: DataRequirement +metadata: + id: monitoring.multivenue.stable + owner: data-platform + subject: spiffe://qdl/paper/monitoring-multivenue-stable + environment: paper + revision: 1 +spec: + sdk_major: 2 + rollback_contract: V1 + execution_dependency: FORBIDDEN + permissions: + - instruments:read + - snapshot:read + - history:read + - status:read + - quality:read + - stream:read + purposes: + - INTERNAL_RESEARCH + quotas: + requests_per_minute: 180 + max_batch_items: 50 + max_warmup_rows: 2000 + max_streams: 20 + max_buffer_events: 2000 + requirements: + - instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: TRADE + consumer_grade: RESEARCH + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 15000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST + - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: TRADE + consumer_grade: RESEARCH + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 15000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST + - instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 + feed: BAR + consumer_grade: RESEARCH + source_policy_id: vn_primary_v2 + warmup_limit: 30 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: EMIT_REVISIONS + interval: 1m diff --git a/consumers/stable/trading-system-paper.yaml b/consumers/stable/trading-system-paper.yaml new file mode 100644 index 0000000..31169b9 --- /dev/null +++ b/consumers/stable/trading-system-paper.yaml @@ -0,0 +1,88 @@ +apiVersion: qdl/v2 +kind: DataRequirement +metadata: + id: trading-system.paper.stable + owner: trading-platform + subject: spiffe://qdl/paper/trading-system-stable + environment: paper + revision: 1 +spec: + sdk_major: 2 + rollback_contract: V1 + execution_dependency: PAPER_ONLY + permissions: + - instruments:read + - snapshot:read + - history:read + - status:read + - quality:read + - stream:read + purposes: + - INTERNAL_EXECUTION + quotas: + requests_per_minute: 600 + max_batch_items: 50 + max_warmup_rows: 2000 + max_streams: 20 + max_buffer_events: 2000 + requirements: + - instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: TRADE + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 3000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST + - instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: QUOTE + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 2000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST + - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: TRADE + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 3000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST + - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: QUOTE + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + warmup_limit: 0 + max_freshness_ms: 2000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST + - instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 + feed: TRADE + consumer_grade: EXECUTION + source_policy_id: vn_primary_v2 + warmup_limit: 0 + max_freshness_ms: 30000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST diff --git a/contracts/golden/canonical/trade-envelope.bin b/contracts/golden/canonical/trade-envelope.bin index 222dcc1..1a19133 100644 Binary files a/contracts/golden/canonical/trade-envelope.bin and b/contracts/golden/canonical/trade-envelope.bin differ diff --git a/contracts/golden/canonical/trade-envelope.json b/contracts/golden/canonical/trade-envelope.json index 358f47e..e24f5df 100644 --- a/contracts/golden/canonical/trade-envelope.json +++ b/contracts/golden/canonical/trade-envelope.json @@ -3,6 +3,6 @@ "rust_consumer": "qdl-contracts prost 0.13.5", "schema": "qdl.marketdata.trade", "schema_major": 2, - "sha256": "671935121f18ac43f6a2bbfa23d5f940c0028cc176c8dcfde657a70ae91ea48c", - "size_bytes": 424 + "sha256": "e4eef5e19722e2e7d0b8a9c34d36ddbb62d6ee274a6c89c3d98644e3947560ba", + "size_bytes": 428 } diff --git a/contracts/golden/phase2/binance-spot-bar.bin b/contracts/golden/phase2/binance-spot-bar.bin new file mode 100644 index 0000000..dec391a --- /dev/null +++ b/contracts/golden/phase2/binance-spot-bar.bin @@ -0,0 +1,3 @@ + +qdl.marketdata.bar"JƗ";[^*$26edfffd-6824-5e75-a620-5a122b3e10862BINANCE.SPOT.SPOT.BTC-USDT8BBINANCEJSPOTRSPOTZBTCUSDTbBINANCE_DIRECTjbinance-spot-bar-001px*ԙ૜ƨ1786352340000:110:1786352400123 qdl-normalizer/2.0.0binance-json/1.0.0 {mGԵkgׂh8C|,3Dqphase-a-binance_spot_bar  +1mѿ""61200.00*"61240.002"61190.00:"61234.10B"12.500H P`hx"12.500 \ No newline at end of file diff --git a/contracts/golden/phase2/binance-spot-bbo.bin b/contracts/golden/phase2/binance-spot-bbo.bin new file mode 100644 index 0000000..1126ca7 --- /dev/null +++ b/contracts/golden/phase2/binance-spot-bbo.bin @@ -0,0 +1,3 @@ + +qdl.marketdata.quote"R݃@[oapf*$26edfffd-6824-5e75-a620-5a122b3e10862BINANCE.SPOT.SPOT.BTC-USDT8BBINANCEJSPOTRSPOTZBTCUSDTbBINANCE_DIRECTjbinance-spot-bbo-001px*ԙ૜ƨ 400900217 qdl-normalizer/2.0.0binance-json/1.0.0 %4de=2(bU{Hphase-a-binance_spot_bbo F +"61234.10 "1.250"61234.20" "0.500(0 \ No newline at end of file diff --git a/contracts/golden/phase2/binance-spot-rest-bar.bin b/contracts/golden/phase2/binance-spot-rest-bar.bin new file mode 100644 index 0000000..fb89c05 --- /dev/null +++ b/contracts/golden/phase2/binance-spot-rest-bar.bin @@ -0,0 +1,3 @@ + +qdl.marketdata.bar"o+z-֙g.T*$26edfffd-6824-5e75-a620-5a122b3e10862BINANCE.SPOT.SPOT.BTC-USDT8BBINANCEJSPOTRSPOTZBTCUSDTbBINANCE_DIRECTjbinance-spot-bar-001px*ԙ૜ƨ1786352340000:1786352399999 qdl-normalizer/2.0.0binance-rest/2.0.0 ŋ)xaI[:HF{qDphase-a-binance_spot_rest_bar  +1mѿ""61200.00*"61240.002"61190.00:"61234.10B"12.500H P`hx"12.500H" 765200.00 \ No newline at end of file diff --git a/contracts/golden/phase2/binance-spot-trade.bin b/contracts/golden/phase2/binance-spot-trade.bin new file mode 100644 index 0000000..134c77f Binary files /dev/null and b/contracts/golden/phase2/binance-spot-trade.bin differ diff --git a/contracts/golden/phase2/binance-usdm-bar.bin b/contracts/golden/phase2/binance-usdm-bar.bin index d1457b6..7ca7be0 100644 --- a/contracts/golden/phase2/binance-usdm-bar.bin +++ b/contracts/golden/phase2/binance-usdm-bar.bin @@ -1,3 +1,3 @@ -qdl.marketdata.bar"4Bf0y*$85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c2BINANCE.USDM.PERPETUAL.BTCUSDT8BBINANCEJUSDMR PERPETUALZBTCUSDTbBINANCE_DIRECTjbinance-usdm-bar-shadow-001px*ԙ૜ƨ1786352340000:110:1786352400123 qdl-normalizer/2.0.0binance-json/1.0.0 {mGԵkgׂh8C|,3Dqphase3-binance-bar | -1mѿ""61200.00*"61240.002"61190.00:"61234.10B"12.500H P`h \ No newline at end of file +qdl.marketdata.bar"4Bf0y*$85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c2BINANCE.USDM.PERPETUAL.BTCUSDT8BBINANCEJUSDMR PERPETUALZBTCUSDTbBINANCE_DIRECTjbinance-usdm-bar-shadow-001px*ԙ૜ƨ1786352340000:110:1786352400123 qdl-normalizer/2.0.0binance-json/1.0.0 {mGԵkgׂh8C|,3Dqphase3-binance-bar  +1mѿ""61200.00*"61240.002"61190.00:"61234.10B"12.500H P`hx"12.500 \ No newline at end of file diff --git a/contracts/golden/phase2/binance-usdm-bbo.bin b/contracts/golden/phase2/binance-usdm-bbo.bin index 4cb3116..caaa902 100644 --- a/contracts/golden/phase2/binance-usdm-bbo.bin +++ b/contracts/golden/phase2/binance-usdm-bbo.bin @@ -1,3 +1,3 @@ -qdl.marketdata.quote"̾g|ƚ4*$85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c2BINANCE.USDM.PERPETUAL.BTCUSDT8BBINANCEJUSDMR PERPETUALZBTCUSDTbBINANCE_DIRECTjbinance-usdm-bbo-shadow-001px*ԙ૜ƨ 400900217 qdl-normalizer/2.0.0binance-json/1.0.0 CQiB5)ށ]"LSPphase3-binance-bbo D -"61234.10 "1.250"61234.20" "0.500( \ No newline at end of file +qdl.marketdata.quote"̾g|ƚ4*$85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c2BINANCE.USDM.PERPETUAL.BTCUSDT8BBINANCEJUSDMR PERPETUALZBTCUSDTbBINANCE_DIRECTjbinance-usdm-bbo-shadow-001px*ԙ૜ƨ 400900217 qdl-normalizer/2.0.0binance-json/1.0.0 CQiB5)ށ]"LSPphase3-binance-bbo F +"61234.10 "1.250"61234.20" "0.500(0 \ No newline at end of file diff --git a/contracts/golden/phase2/binance-usdm-rest-bar.bin b/contracts/golden/phase2/binance-usdm-rest-bar.bin new file mode 100644 index 0000000..f3c972a --- /dev/null +++ b/contracts/golden/phase2/binance-usdm-rest-bar.bin @@ -0,0 +1,3 @@ + +qdl.marketdata.bar"F/+O&&*$85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c2BINANCE.USDM.PERPETUAL.BTCUSDT8BBINANCEJUSDMR PERPETUALZBTCUSDTbBINANCE_DIRECTjbinance-usdm-rest-bar-001px*ԙ૜ƨ1786352340000:1786352399999 qdl-normalizer/2.0.0binance-rest/2.0.0 ŋ)xaI[:HF{qDphase-a-binance_usdm_rest_bar  +1mѿ""61200.00*"61240.002"61190.00:"61234.10B"12.500H P`hx"12.500H" 765200.00 \ No newline at end of file diff --git a/contracts/golden/phase2/binance-usdm-trade.bin b/contracts/golden/phase2/binance-usdm-trade.bin index 9de228a..61a0e7a 100644 --- a/contracts/golden/phase2/binance-usdm-trade.bin +++ b/contracts/golden/phase2/binance-usdm-trade.bin @@ -1,3 +1,3 @@ -qdl.marketdata.trade"v@ş뉆wq*$85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c2BINANCE.USDM.PERPETUAL.BTC-USDT8BBINANCEJUSDMR PERPETUALZBTCUSDTbBINANCE_DIRECTjbinance-usdm-trade-shadow-001px*ԙ૜ƨ9876543210123456789 qdl-normalizer/2.0.0binance-json/1.0.0 s&÷V846bh M\ -phase2-binance-trade 8 -9876543210123456789"61234.10 "0.125  \ No newline at end of file +qdl.marketdata.trade"v@ş뉆wq*$85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c2BINANCE.USDM.PERPETUAL.BTC-USDT8BBINANCEJUSDMR PERPETUALZBTCUSDTbBINANCE_DIRECTjbinance-usdm-trade-shadow-001px*ԙ૜ƨ9876543210123456789 qdl-normalizer/2.0.0binance-json/1.0.0 s&÷V846bh M\ -phase2-binance-trade < +9876543210123456789"61234.10 "0.125 8@ \ No newline at end of file diff --git a/contracts/golden/phase2/dnse-derivative-bar.bin b/contracts/golden/phase2/dnse-derivative-bar.bin new file mode 100644 index 0000000..1e5a216 Binary files /dev/null and b/contracts/golden/phase2/dnse-derivative-bar.bin differ diff --git a/contracts/golden/phase2/dnse-derivative-trade.bin b/contracts/golden/phase2/dnse-derivative-trade.bin new file mode 100644 index 0000000..7923ff3 --- /dev/null +++ b/contracts/golden/phase2/dnse-derivative-trade.bin @@ -0,0 +1,3 @@ + +qdl.marketdata.trade"(嗣rs #R.*$d0ac3d4c-2374-5690-8af5-b970448f91d32!HNX.VN_DERIVATIVES.FUTURE.VN30F1M8BHNXJVN_DERIVATIVESRFUTUREZVN30F1Mb DNSE_DIRECTjdnse-derivative-trade-001px ԙ(derived:f5c2cd1b7260356b6e1b9f6718e826c2qdl-normalizer/2.0.0dnse_direct/2.0.0  3AN1eʲwqYBTK52.!phase-a-dnse-derivative-trade-001#dnse-derivative-trade-001-session-1 =҃V܍W'W1Fբw r`5kng&F +(derived:f5c2cd1b7260356b6e1b9f6718e826c2"1820.7"128@ \ No newline at end of file diff --git a/contracts/golden/phase2/dnse-equity-bar.bin b/contracts/golden/phase2/dnse-equity-bar.bin new file mode 100644 index 0000000..1ebd1b2 --- /dev/null +++ b/contracts/golden/phase2/dnse-equity-bar.bin @@ -0,0 +1,4 @@ + +qdl.marketdata.bar"s\"oL)*$6f025ab7-c0b4-5e08-866d-34ce62aef52f2HOSE.EQUITIES.COMMON_STOCK.FPT8BHOSEJEQUITIESR COMMON_STOCKZFPTb DNSE_DIRECTjdnse-equity-bar-001px ԙ1786352340000:1786352399999qdl-normalizer/2.0.0dnse_direct/2.0.0  4xЌ$f +g-c(UIf"phase-a-dnse-equity-bar-001dnse-equity-bar-001-session-1 Z ^7&-9}mlRŋ2,u E'X7if!5k>A0"l +1mѿ" "125000*ژ"125100.02л"124900.0: "125000B"100P`hx \ No newline at end of file diff --git a/contracts/golden/phase2/dnse-equity-trade.bin b/contracts/golden/phase2/dnse-equity-trade.bin new file mode 100644 index 0000000..9ab2e53 --- /dev/null +++ b/contracts/golden/phase2/dnse-equity-trade.bin @@ -0,0 +1,3 @@ + +qdl.marketdata.trade"%F;,*$6f025ab7-c0b4-5e08-866d-34ce62aef52f2HOSE.EQUITIES.COMMON_STOCK.FPT8BHOSEJEQUITIESR COMMON_STOCKZFPTb DNSE_DIRECTjdnse-equity-trade-001px ԙ(derived:6bcc4450a82a5d7755679978331d3dcaqdl-normalizer/2.0.0dnse_direct/2.0.0  6l1CG "=92߻e'>Lphase19-okx-spot-bar  +1mѿ""61200.00*"61240.002"61190.00:"61234.10B"12.500P`hx"12.500H" 765200.00 \ No newline at end of file diff --git a/contracts/golden/phase2/okx-spot-bbo.bin b/contracts/golden/phase2/okx-spot-bbo.bin new file mode 100644 index 0000000..88391a1 Binary files /dev/null and b/contracts/golden/phase2/okx-spot-bbo.bin differ diff --git a/contracts/golden/phase2/okx-spot-trade.bin b/contracts/golden/phase2/okx-spot-trade.bin new file mode 100644 index 0000000..404fdc1 --- /dev/null +++ b/contracts/golden/phase2/okx-spot-trade.bin @@ -0,0 +1,4 @@ + +qdl.marketdata.trade"6)QB6Q*$9e996914-aa98-5ea0-b0a0-a78dc51e1f8d2OKX.SPOT.SPOT.BTC-USDT8BOKXJSPOTRSPOTZBTC-USDTb +OKX_DIRECTjokx-public-trades-001px998877665544332211Xqdl-normalizer/2.0.0okx-json/1.0.0 B1]0°.k> "phase2-okx-spot-trade : +998877665544332211"61235.20 "1.50 8@ \ No newline at end of file diff --git a/contracts/golden/phase2/okx-swap-bar.bin b/contracts/golden/phase2/okx-swap-bar.bin new file mode 100644 index 0000000..a0b0b52 --- /dev/null +++ b/contracts/golden/phase2/okx-swap-bar.bin @@ -0,0 +1,4 @@ + +qdl.marketdata.bar"):VmCW*$23533297-11a8-5290-a800-c7f584706cd62OKX.SWAP.PERPETUAL.BTC-USDT8BOKXJSWAPR PERPETUALZ BTC-USDT-SWAPb +OKX_DIRECTjokx-business-candle-001pxѿ1786352340000:1Zqdl-normalizer/2.0.0okx-json/2.0.0  7mImy a#=WnNjYerphase19-okx-bar  +1mѿ""61200.00*"61240.002"61190.00:"61234.10B"12.500P`hx"12.500H" 765200.00"12.500 \ No newline at end of file diff --git a/contracts/golden/phase2/okx-swap-bbo.bin b/contracts/golden/phase2/okx-swap-bbo.bin new file mode 100644 index 0000000..afef81b --- /dev/null +++ b/contracts/golden/phase2/okx-swap-bbo.bin @@ -0,0 +1,4 @@ + +qdl.marketdata.quote" o){E *$23533297-11a8-5290-a800-c7f584706cd62OKX.SWAP.PERPETUAL.BTC-USDT8BOKXJSWAPR PERPETUALZ BTC-USDT-SWAPb +OKX_DIRECTjokx-public-bbo-001px817263Yqdl-normalizer/2.0.0okx-json/2.0.0 r,B#F3Wx͆Պ,<phase19-okx-bbo F +"61234.10 "1.250"61234.20" "0.500(0 \ No newline at end of file diff --git a/contracts/golden/phase2/okx-swap-trade.bin b/contracts/golden/phase2/okx-swap-trade.bin index 938f8c7..1636cfb 100644 --- a/contracts/golden/phase2/okx-swap-trade.bin +++ b/contracts/golden/phase2/okx-swap-trade.bin @@ -1,5 +1,5 @@ qdl.marketdata.trade"4UZI<r,3**$23533297-11a8-5290-a800-c7f584706cd62OKX.SWAP.PERPETUAL.BTC-USDT8BOKXJSWAPR PERPETUALZ BTC-USDT-SWAPb OKX_DIRECTjokx-public-trades-001px998877665544332211Xqdl-normalizer/2.0.0okx-json/1.0.0 } ]W'lE^Q? -ugphase2-okx-trade 6 -998877665544332211"61235.20 "1.50  \ No newline at end of file +ugphase2-okx-trade : +998877665544332211"61235.20 "1.50 8@ \ No newline at end of file diff --git a/contracts/golden/phase2/vnstock-equity-bar.bin b/contracts/golden/phase2/vnstock-equity-bar.bin new file mode 100644 index 0000000..80427dc Binary files /dev/null and b/contracts/golden/phase2/vnstock-equity-bar.bin differ diff --git a/contracts/proto/qdl/common/v1/common.proto b/contracts/proto/qdl/common/v1/common.proto index 11d2d2b..622e677 100644 --- a/contracts/proto/qdl/common/v1/common.proto +++ b/contracts/proto/qdl/common/v1/common.proto @@ -13,6 +13,17 @@ message DecimalValue { string source_text = 4; } +// QuantityUnit makes native quantity semantics explicit across spot, +// derivatives and equity markets. Asset symbols and contract multipliers are +// resolved from the versioned Instrument referenced by instrument_uid. +enum QuantityUnit { + QUANTITY_UNIT_UNSPECIFIED = 0; + QUANTITY_UNIT_BASE_ASSET = 1; + QUANTITY_UNIT_QUOTE_ASSET = 2; + QUANTITY_UNIT_CONTRACT = 3; + QUANTITY_UNIT_SHARE = 4; +} + enum SourceRole { SOURCE_ROLE_UNSPECIFIED = 0; SOURCE_ROLE_PRIMARY = 1; diff --git a/contracts/proto/qdl/marketdata/v2/market_data.proto b/contracts/proto/qdl/marketdata/v2/market_data.proto index 4aa8fa3..eb71061 100644 --- a/contracts/proto/qdl/marketdata/v2/market_data.proto +++ b/contracts/proto/qdl/marketdata/v2/market_data.proto @@ -5,6 +5,12 @@ package qdl.marketdata.v2; import "qdl/common/v1/common.proto"; import "qdl/quality/v1/quality.proto"; +enum TradeIdentityKind { + TRADE_IDENTITY_KIND_UNSPECIFIED = 0; + TRADE_IDENTITY_KIND_NATIVE = 1; + TRADE_IDENTITY_KIND_DERIVED_RAW_CAPTURE = 2; +} + enum BarLifecycle { BAR_LIFECYCLE_UNSPECIFIED = 0; BAR_LIFECYCLE_IN_PROGRESS = 1; @@ -20,6 +26,8 @@ message Trade { qdl.common.v1.AggressorSide aggressor_side = 4; bool is_block_trade = 5; bool is_buyer_maker = 6; + qdl.common.v1.QuantityUnit quantity_unit = 7; + TradeIdentityKind identity_kind = 8; } message Quote { @@ -28,6 +36,7 @@ message Quote { qdl.common.v1.DecimalValue ask_price = 3; qdl.common.v1.DecimalValue ask_quantity = 4; uint32 level = 5; + qdl.common.v1.QuantityUnit quantity_unit = 6; } message Bar { @@ -45,6 +54,10 @@ message Bar { qdl.common.v1.BarOrigin origin = 12; BarLifecycle lifecycle = 13; optional bytes supersedes_event_id = 14; + qdl.common.v1.QuantityUnit volume_unit = 15; + optional qdl.common.v1.DecimalValue base_volume = 16; + optional qdl.common.v1.DecimalValue quote_volume = 17; + optional qdl.common.v1.DecimalValue contract_volume = 18; } message BookLevel { @@ -52,6 +65,7 @@ message BookLevel { qdl.common.v1.DecimalValue price = 2; qdl.common.v1.DecimalValue quantity = 3; uint32 order_count = 4; + qdl.common.v1.QuantityUnit quantity_unit = 5; } message OrderBookSnapshot { @@ -79,6 +93,7 @@ message FundingRate { message OpenInterest { qdl.common.v1.DecimalValue quantity = 1; optional qdl.common.v1.DecimalValue notional = 2; + qdl.common.v1.QuantityUnit quantity_unit = 3; } message MarkIndexPrice { @@ -93,6 +108,8 @@ message Ticker { optional qdl.common.v1.DecimalValue high_24h = 4; optional qdl.common.v1.DecimalValue low_24h = 5; optional qdl.common.v1.DecimalValue volume_24h = 6; + qdl.common.v1.QuantityUnit last_quantity_unit = 7; + qdl.common.v1.QuantityUnit volume_24h_unit = 8; } message EventEnvelope { diff --git a/contracts/v2/openapi.snapshot.json b/contracts/v2/openapi.snapshot.json index 38484f4..151ddbd 100644 --- a/contracts/v2/openapi.snapshot.json +++ b/contracts/v2/openapi.snapshot.json @@ -15,6 +15,16 @@ "BarPayload": { "additionalProperties": false, "properties": { + "base_volume": { + "anyOf": [ + { + "$ref": "#/components/schemas/DecimalValue" + }, + { + "type": "null" + } + ] + }, "close": { "$ref": "#/components/schemas/DecimalValue" }, @@ -23,6 +33,16 @@ "title": "Close Time Ns", "type": "integer" }, + "contract_volume": { + "anyOf": [ + { + "$ref": "#/components/schemas/DecimalValue" + }, + { + "type": "null" + } + ] + }, "feed": { "const": "BAR", "default": "BAR", @@ -62,6 +82,16 @@ "title": "Origin", "type": "string" }, + "quote_volume": { + "anyOf": [ + { + "$ref": "#/components/schemas/DecimalValue" + }, + { + "type": "null" + } + ] + }, "revision": { "minimum": 0.0, "title": "Revision", @@ -86,6 +116,9 @@ }, "volume": { "$ref": "#/components/schemas/DecimalValue" + }, + "volume_unit": { + "$ref": "#/components/schemas/QuantityUnit" } }, "required": [ @@ -97,6 +130,7 @@ "low", "close", "volume", + "volume_unit", "lifecycle", "revision", "origin" @@ -302,6 +336,9 @@ "quantity": { "$ref": "#/components/schemas/DecimalValue" }, + "quantity_unit": { + "$ref": "#/components/schemas/QuantityUnit" + }, "side": { "enum": [ "BID", @@ -314,7 +351,8 @@ "required": [ "side", "price", - "quantity" + "quantity", + "quantity_unit" ], "title": "BookLevel", "type": "object" @@ -983,10 +1021,14 @@ }, "quantity": { "$ref": "#/components/schemas/DecimalValue" + }, + "quantity_unit": { + "$ref": "#/components/schemas/QuantityUnit" } }, "required": [ - "quantity" + "quantity", + "quantity_unit" ], "title": "OpenInterestPayload", "type": "object" @@ -1133,6 +1175,16 @@ "title": "QualityView", "type": "object" }, + "QuantityUnit": { + "enum": [ + "BASE_ASSET", + "QUOTE_ASSET", + "CONTRACT", + "SHARE" + ], + "title": "QuantityUnit", + "type": "string" + }, "QuotePayload": { "additionalProperties": false, "properties": { @@ -1159,13 +1211,17 @@ "minimum": 1.0, "title": "Level", "type": "integer" + }, + "quantity_unit": { + "$ref": "#/components/schemas/QuantityUnit" } }, "required": [ "bid_price", "bid_quantity", "ask_price", - "ask_quantity" + "ask_quantity", + "quantity_unit" ], "title": "QuotePayload", "type": "object" @@ -1485,6 +1541,16 @@ } ] }, + "last_quantity_unit": { + "anyOf": [ + { + "$ref": "#/components/schemas/QuantityUnit" + }, + { + "type": "null" + } + ] + }, "low_24h": { "anyOf": [ { @@ -1514,6 +1580,16 @@ "type": "null" } ] + }, + "volume_24h_unit": { + "anyOf": [ + { + "$ref": "#/components/schemas/QuantityUnit" + }, + { + "type": "null" + } + ] } }, "required": [ @@ -1522,13 +1598,22 @@ "title": "TickerPayload", "type": "object" }, + "TradeIdentityKind": { + "enum": [ + "NATIVE", + "DERIVED_RAW_CAPTURE" + ], + "title": "TradeIdentityKind", + "type": "string" + }, "TradePayload": { "additionalProperties": false, "properties": { "aggressor_side": { "enum": [ "BUY", - "SELL" + "SELL", + "UNKNOWN" ], "title": "Aggressor Side", "type": "string" @@ -1539,6 +1624,9 @@ "title": "Feed", "type": "string" }, + "identity_kind": { + "$ref": "#/components/schemas/TradeIdentityKind" + }, "is_block_trade": { "default": false, "title": "Is Block Trade", @@ -1560,13 +1648,18 @@ }, "quantity": { "$ref": "#/components/schemas/DecimalValue" + }, + "quantity_unit": { + "$ref": "#/components/schemas/QuantityUnit" } }, "required": [ "native_trade_id", "price", "quantity", - "aggressor_side" + "quantity_unit", + "aggressor_side", + "identity_kind" ], "title": "TradePayload", "type": "object" @@ -1687,7 +1780,7 @@ }, "info": { "title": "Quant Data Layer V2", - "version": "2.0.0-shadow" + "version": "2.0.0" }, "openapi": "3.1.0", "paths": { diff --git a/deny.toml b/deny.toml index 6ea2aeb..10cc3ac 100644 --- a/deny.toml +++ b/deny.toml @@ -1,23 +1,53 @@ +# Rust supply-chain policy for the Linux production workspace. +# Keep exceptions empty unless an advisory/license is reviewed and documented. +[graph] +targets = [ + { triple = "x86_64-unknown-linux-gnu" }, +] +all-features = false +no-default-features = false + [advisories] -version = 2 -yanked = "deny" +ignore = [] +git-fetch-with-cli = true [licenses] -version = 2 -confidence-threshold = 0.8 allow = [ - "Apache-2.0", - "BSD-3-Clause", - "CDLA-Permissive-2.0", - "ISC", - "MIT", - "Unicode-3.0", + "Apache-2.0", + "BSD-2-Clause", + "BSD-3-Clause", + "CDLA-Permissive-2.0", + "ISC", + "MIT", + "Unicode-3.0", + "Unlicense", ] +confidence-threshold = 0.8 +exceptions = [] + +[licenses.private] +ignore = false +registries = [] [bans] multiple-versions = "warn" wildcards = "deny" +highlight = "all" +workspace-default-features = "allow" +external-default-features = "allow" +allow = [] +allow-workspace = true +deny = [] +skip = [] +skip-tree = [] [sources] unknown-registry = "deny" unknown-git = "deny" +allow-registry = ["https://github.com/rust-lang/crates.io-index"] +allow-git = [] + +[sources.allow-org] +github = [] +gitlab = [] +bitbucket = [] diff --git a/docker-compose.phase8-kafka.yml b/docker-compose.phase8-kafka.yml index ae85789..ba07161 100644 --- a/docker-compose.phase8-kafka.yml +++ b/docker-compose.phase8-kafka.yml @@ -51,7 +51,9 @@ x-kafka-common: &kafka-common target: /etc/kafka/secrets read_only: true healthcheck: - test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server $${HOSTNAME}:9092 --command-config /etc/kafka/secrets/admin.properties >/dev/null 2>&1"] + # Liveness stays lightweight inside the broker cgroup. Authenticated Kafka + # readiness/quorum checks run from the disposable phase8_admin client. + test: ["CMD-SHELL", "timeout 2 bash -c '- + install -d -o 10001 -g 10001 -m 0750 + /stable-certs/producer /stable-certs/core /stable-certs/projector && + cp /source/producer/* /stable-certs/producer/ && + cp /source/core/* /stable-certs/core/ && + cp /source/projector/* /stable-certs/projector/ && + chown -R 10001:10001 /stable-certs && + find /stable-certs -type f -exec chmod 0440 {} + + restart: "no" + read_only: true + security_opt: [no-new-privileges:true] + cap_drop: [ALL] + cap_add: [CHOWN, DAC_OVERRIDE, FOWNER] + networks: [stable_internal] + volumes: + - ${QDL_STABLE_PRODUCER_CERT_DIR:?set QDL_STABLE_PRODUCER_CERT_DIR}:/source/producer:ro + - ${QDL_STABLE_CORE_CERT_DIR:?set QDL_STABLE_CORE_CERT_DIR}:/source/core:ro + - ${QDL_STABLE_PROJECTOR_CERT_DIR:?set QDL_STABLE_PROJECTOR_CERT_DIR}:/source/projector:ro + - stable_tls:/stable-certs + + query_v2_1: + <<: *python + command: [uvicorn, app.entrypoints.query_v2_stable:app, --host, 0.0.0.0, --port, "8200", --no-access-log] + networks: [stable_internal, stable_ingress] + ports: ["127.0.0.1:18201:8200"] + environment: + <<: *stable-env + QDL_STABLE_INSTANCE_ID: stable-query-1 + QDL_STABLE_CONSUMER_GROUP: stable-query-1 + QDL_STABLE_HTTP_PORT: "8200" + QDL_STABLE_GRPC_PORT: "8210" + QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-query-1-audit.jsonl + depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} + healthcheck: {test: [CMD, python, -c, "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8200/health/ready', timeout=2)"], interval: 5s, timeout: 3s, retries: 20} + + query_v2_2: + <<: *python + command: [uvicorn, app.entrypoints.query_v2_stable:app, --host, 0.0.0.0, --port, "8200", --no-access-log] + networks: [stable_internal, stable_ingress] + ports: ["127.0.0.1:18202:8200"] + environment: + <<: *stable-env + QDL_STABLE_INSTANCE_ID: stable-query-2 + QDL_STABLE_CONSUMER_GROUP: stable-query-2 + QDL_STABLE_HTTP_PORT: "8200" + QDL_STABLE_GRPC_PORT: "8210" + QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-query-2-audit.jsonl + depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} + + stream_v2_active: + <<: *python + command: [python, -m, app.entrypoints.stream_v2_stable] + networks: [stable_internal, stable_ingress] + ports: ["127.0.0.1:18210:8200", "127.0.0.1:18220:8210"] + environment: + <<: *stable-env + QDL_STABLE_INSTANCE_ID: stable-stream-active + QDL_STABLE_CONSUMER_GROUP: stable-stream-active + QDL_STABLE_HTTP_PORT: "8200" + QDL_STABLE_GRPC_PORT: "8210" + QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-stream-active-audit.jsonl + depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} + + stream_v2_passive: + <<: *python + command: [python, -m, app.entrypoints.stream_v2_stable] + networks: [stable_internal, stable_ingress] + ports: ["127.0.0.1:18211:8200", "127.0.0.1:18221:8210"] + environment: + <<: *stable-env + QDL_STABLE_INSTANCE_ID: stable-stream-passive + QDL_STABLE_CONSUMER_GROUP: stable-stream-passive + QDL_STABLE_HTTP_PORT: "8200" + QDL_STABLE_GRPC_PORT: "8210" + QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-stream-passive-audit.jsonl + depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} + + projector_v2: + <<: *python + command: [python, -m, app.entrypoints.projector_v2_stable] + environment: + <<: *stable-env + QDL_STABLE_INSTANCE_ID: stable-projector-1 + QDL_STABLE_CONSUMER_GROUP: stable-projector-v1 + QDL_STABLE_HTTP_PORT: "8230" + QDL_STABLE_GRPC_PORT: "8231" + QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-projector-1-audit.jsonl + QDL_STABLE_KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092 + QDL_STABLE_KAFKA_CLIENT_ID: stable-projector-1 + QDL_STABLE_KAFKA_RAW_TOPICS: md.raw.stable.v1 + QDL_STABLE_KAFKA_CANONICAL_TOPIC: md.canonical.v2 + QDL_STABLE_KAFKA_CERT_ROOT: /stable-certs/projector + QDL_STABLE_STREAM_INGEST_URLS_JSON: '["http://stream_v2_active:8200","http://stream_v2_passive:8200"]' + volumes: [stable_state:/var/lib/qdl-stable, stable_tls:/stable-certs:ro] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + stable_state_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + stable_redis: {condition: service_healthy} + stream_v2_active: {condition: service_started} + stream_v2_passive: {condition: service_started} + + rust_core: + <<: *rust + entrypoint: [/usr/local/bin/qdl-realtime-core] + command: [/runtime/core.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-core-001 + QDL_KAFKA_GROUP_ID: qdl-v2-stable-core-v1 + volumes: [stable_tls:/stable-certs:ro, "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/core.json:/runtime/core.json:ro"] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + rust_core_2: + <<: *rust + entrypoint: [/usr/local/bin/qdl-realtime-core] + command: [/runtime/core-002.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-core-002 + QDL_KAFKA_GROUP_ID: qdl-v2-stable-core-v1 + volumes: [stable_tls:/stable-certs:ro, "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/core-002.json:/runtime/core-002.json:ro"] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + rust_core_3: + <<: *rust + entrypoint: [/usr/local/bin/qdl-realtime-core] + command: [/runtime/core-003.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-core-003 + QDL_KAFKA_GROUP_ID: qdl-v2-stable-core-v1 + volumes: [stable_tls:/stable-certs:ro, "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/core-003.json:/runtime/core-003.json:ro"] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + ingestor_binance_usdm: + <<: *rust + entrypoint: [/usr/local/bin/qdl-native-raw-ingestor] + command: [/runtime/config.json] + networks: [stable_internal, stable_egress] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/producer + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-producer + QDL_KAFKA_GROUP_ID: qdl-v2-stable-producer + volumes: [stable_tls:/stable-certs:ro, stable_state:/var/lib/qdl-stable, "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/ingestor-binance-usdm.json:/runtime/config.json:ro"] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + stable_state_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + ingestor_binance_spot: + <<: *rust + entrypoint: [/usr/local/bin/qdl-native-raw-ingestor] + command: [/runtime/config.json] + networks: [stable_internal, stable_egress] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/producer + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-producer + QDL_KAFKA_GROUP_ID: qdl-v2-stable-producer + volumes: [stable_tls:/stable-certs:ro, stable_state:/var/lib/qdl-stable, "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/ingestor-binance-spot.json:/runtime/config.json:ro"] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + stable_state_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + ingestor_okx_swap: + <<: *rust + entrypoint: [/usr/local/bin/qdl-native-raw-ingestor] + command: [/runtime/config.json] + networks: [stable_internal, stable_egress] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/producer + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-producer + QDL_KAFKA_GROUP_ID: qdl-v2-stable-producer + volumes: [stable_tls:/stable-certs:ro, stable_state:/var/lib/qdl-stable, "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/ingestor-okx-swap.json:/runtime/config.json:ro"] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + stable_state_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + ingestor_okx_spot: + <<: *rust + entrypoint: [/usr/local/bin/qdl-native-raw-ingestor] + command: [/runtime/config.json] + networks: [stable_internal, stable_egress] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/producer + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-producer + QDL_KAFKA_GROUP_ID: qdl-v2-stable-producer + volumes: [stable_tls:/stable-certs:ro, stable_state:/var/lib/qdl-stable, "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/ingestor-okx-spot.json:/runtime/config.json:ro"] + depends_on: + stable_tls_init: {condition: service_completed_successfully} + stable_state_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + binance_bar_edge: + <<: *python + command: [python, -m, app.entrypoints.binance_bar_v2_stable] + networks: [stable_internal, stable_egress] + environment: + <<: *stable-env + QDL_STABLE_INSTANCE_ID: stable-binance-bar-edge + QDL_STABLE_CONSUMER_GROUP: stable-binance-bar-edge + QDL_STABLE_ACQUISITION_BINDINGS: /app/config/v2/stable-acquisition-bindings.yaml + QDL_STABLE_RUNTIME_DIR: /runtime + QDL_KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092 + QDL_KAFKA_CERT_ROOT: /stable-certs/producer + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-binance-bars + QDL_STABLE_BAR_WARMUP_ROWS: "500" + QDL_STABLE_BAR_SETTLEMENT_DELAY_SECONDS: "10" + QDL_STABLE_BAR_STATE_PATH: /var/lib/qdl-stable/runtime/stable-crypto-bar-edge.json + volumes: + - stable_state:/var/lib/qdl-stable + - stable_tls:/stable-certs:ro + - ${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}:/runtime:ro + depends_on: + stable_state_init: {condition: service_completed_successfully} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + vn_edge_v2: + <<: *python + profiles: [stable-vn] + command: [python, -m, app.entrypoints.vn_edge_v2_stable] + networks: [stable_internal, stable_egress] + environment: + <<: *stable-env + QDL_STABLE_INSTANCE_ID: stable-vn-edge + QDL_STABLE_CONSUMER_GROUP: stable-vn-edge + QDL_STABLE_ACQUISITION_BINDINGS: /app/config/v2/stable-acquisition-bindings.yaml + QDL_STABLE_RUNTIME_DIR: /runtime + QDL_STABLE_DNSE_QUEUE_CAPACITY: "5000" + QDL_STABLE_VN_WARMUP_ROWS: "500" + QDL_STABLE_VN_HISTORY_LOOKBACK_DAYS: "30" + QDL_STABLE_VN_HISTORY_ATTEMPTS: "1" + QDL_STABLE_DNSE_STATE_PATH: /var/lib/qdl-stable/runtime/stable-dnse-edge.json + QDL_KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092 + QDL_KAFKA_CERT_ROOT: /stable-certs/producer + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-vn + DNSE_API_KEY: ${DNSE_API_KEY:-} + DNSE_API_SECRET_KEY: ${DNSE_API_SECRET_KEY:-} + DNSE_API_VERSION: ${DNSE_API_VERSION:-2026-07-23} + DNSE_REST_BASE: ${DNSE_REST_BASE:-https://openapi.dnse.com.vn} + DNSE_REST_USE_ENV_PROXY: ${DNSE_REST_USE_ENV_PROXY:-false} + DNSE_REST_CONNECT_TIMEOUT_SECONDS: ${DNSE_REST_CONNECT_TIMEOUT_SECONDS:-5} + DNSE_REST_READ_TIMEOUT_SECONDS: ${DNSE_REST_READ_TIMEOUT_SECONDS:-30} + DNSE_REST_ATTEMPTS: ${DNSE_REST_ATTEMPTS:-4} + DNSE_REST_MAX_BACKOFF_SECONDS: ${DNSE_REST_MAX_BACKOFF_SECONDS:-30} + DNSE_WS_BASE: ${DNSE_WS_BASE:-wss://ws-openapi.dnse.com.vn} + volumes: + - stable_state:/var/lib/qdl-stable + - stable_tls:/stable-certs:ro + - ${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}:/runtime:ro + depends_on: + stable_state_init: {condition: service_completed_successfully} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + +networks: + stable_internal: {internal: true, labels: {qdl.scope: v2-stable-candidate}} + stable_ingress: {labels: {qdl.scope: v2-stable-candidate-ingress}} + stable_egress: {labels: {qdl.scope: v2-stable-candidate-egress}} +volumes: + kafka1_data: {labels: {qdl.scope: v2-stable-candidate}} + kafka2_data: {labels: {qdl.scope: v2-stable-candidate}} + kafka3_data: {labels: {qdl.scope: v2-stable-candidate}} + stable_state: {labels: {qdl.scope: v2-stable-candidate}} + stable_tls: {labels: {qdl.scope: v2-stable-candidate-secrets}} diff --git a/docs/runbooks/dnse-production-provider-edge.md b/docs/runbooks/dnse-production-provider-edge.md new file mode 100644 index 0000000..2997b8b --- /dev/null +++ b/docs/runbooks/dnse-production-provider-edge.md @@ -0,0 +1,125 @@ +# DNSE Production Provider Edge Runbook + +## Purpose And Authority + +This runbook closes the operational boundary for DNSE as the primary Vietnam- +market provider. It implements the architecture in +`upgrade/quant-data-layer-fund-grade-upgrade-architecture.md` section 14.4 and +is tracked by `DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md` B.2-D. + +DNSE is a Python vendor acquisition edge only. It publishes authenticated raw +TRADE and final BAR envelopes to the private Kafka-compatible raw topic. The +shared Rust core remains responsible for canonicalization, units, ordering, +deduplication, quality and durable publication. Redis and public APIs are never +written by the acquisition edge. + +## Security Invariants + +- Never use credentials embedded in examples or an unlicensed SDK snapshot. +- Provision a dedicated market-data key with the least privileges DNSE permits. +- Store the key only in the workload secret store; do not place it in Git, + evidence, command history, Docker labels or compose-rendered artifacts. +- TLS and hostname verification remain mandatory. `CERT_NONE` is forbidden. +- `DNSE_REST_USE_ENV_PROXY=false` is the default. Enabling it requires an + approved monitored proxy; it does not disable TLS verification. +- The edge receives only DNSE market-data credentials and Kafka producer mTLS + material scoped to `md.raw.stable.v1`. It receives no Trading System, Redis, + query, portfolio or broker-order credential. + +## Data Paths + +```text +cold start / bounded repair: +DNSE versioned REST /price/ohlc + -> strict pagination/schema/OHLC validation + -> authenticated raw REST_HISTORY envelope + +live: +DNSE authenticated WebSocket tick.G1/tick.G3 + ohlc_closed.1 + -> bounded vendor dispatch queues + -> bounded lossless edge queue + -> authenticated raw SDK_CALLBACK envelope + +both: +raw Kafka ACK -> atomic DNSE BAR checkpoint + -> Rust canonical/quality core + -> canonical Kafka -> stable Python projector/query/stream +``` + +REST is never polled once per minute. A matching complete checkpoint avoids +cold REST bootstrap after restart. The checkpoint is bound to slice, authority, +catalog revision, acquisition revision and all DNSE BAR binding IDs; any +corruption, partial state or mismatch fails closed. + +## Preflight + +Run preflight from the intended acquisition host without printing credentials: + +1. Confirm `DNSE_API_KEY` and `DNSE_API_SECRET_KEY` are non-empty by boolean and + length only. +2. Confirm direct TCP/TLS reachability to `openapi.dnse.com.vn:443` and + `ws-openapi.dnse.com.vn:443` with bounded connect timeouts. +3. Authenticate one read-only WebSocket session and subscribe to + `ohlc_closed.1` for one approved instrument. Reject any auth error. +4. Call one bounded historical window with `version: 2026-07-23`; require valid + parallel OHLC arrays, monotonic pagination and a final closed row. +5. Confirm system UTC/NTP health and the `Asia/Ho_Chi_Minh` trading calendar. + +A process-up or socket-connect result alone is not acceptance. REST, auth, +subscription, provider bytes and freshness must all pass. + +## Egress-Capable Edge Option + +If the primary Data Layer host cannot reach official DNSE REST, deploy only +`vn_edge_v2` in an approved independent egress domain: + +- outbound allowlist: official DNSE REST/WebSocket and the private Kafka + endpoints only; +- inbound: none except supervised administration; +- Kafka: mTLS, ACL limited to the raw topic, idempotent producer and `acks=all`; +- state: encrypted durable volume containing only the atomic edge checkpoint; +- identity: unique producer/client ID and source session/generation; +- observability: auth/reconnect, queue depth, ACK latency, last TRADE/BAR, + checkpoint age and fatal-fence alerts without payload/credential logging. + +Do not SSH-tunnel a long-lived production feed as an implicit architecture. +A temporary tunnel may certify reachability only. Do not relabel vnstock or +legacy Parquet as `DNSE_DIRECT`; fallback keeps its own provider identity and +requires an audited source-policy transition. + +## Promotion Gates + +Promotion requires all of the following: + +- exactly 500 authentic final 1m warmup rows per approved DNSE BAR binding; +- native closed-BAR callback observed and durably ACKed; +- restart restores the exact checkpoint and performs no repeated cold bootstrap; +- zero unexplained gap, duplicate, conflict, quarantine or queue drop; +- Python/Rust golden parity for DNSE TRADE/BAR, exact units and VN calendar; +- signed consumer warmup -> replay -> live, freshness/session semantics and + unchanged V1 consumers; +- explicit operator approval naming image SHA, host, service, topic ACL, + credential secret, state volume, affected consumers and rollback command. + +## Failure And Rollback + +Auth rejection, REST exhaustion, malformed data, incomplete history, queue +pressure, missing Kafka ACK, checkpoint mismatch or canonical conflict fences the +edge. It must not publish synthetic repair or advance the checkpoint. + +Rollback stops only the isolated DNSE V2 edge and returns its slice to the +retained `2.0.0-2412572` artifacts. V1 remains authoritative until a separately +approved cutover. Preserve bounded hashes/offsets/metrics for diagnosis; remove +all disposable smoke containers and test credentials. + +## Current Host Evidence (2026-08-20) + +- Official DNSE REST TCP/443 timed out before TLS negotiation. +- Official production WebSocket TCP/TLS connected, but the key currently loaded + from host `.env` was rejected as invalid before subscription. +- The running V1 process still reports an older authenticated DNSE session and + `MARKET_CLOSED`; it has zero queue drops. Restarting V1 before credential + rotation is therefore unsafe. + +These are external infrastructure/credential gates, not permission to weaken +provider provenance or claim DNSE production certification. diff --git a/docs/runbooks/phase90b-isolated-v2-beta.md b/docs/runbooks/phase90b-isolated-v2-beta.md new file mode 100644 index 0000000..7032782 --- /dev/null +++ b/docs/runbooks/phase90b-isolated-v2-beta.md @@ -0,0 +1,56 @@ +# Phase 9.0-B Isolated V2 Beta Runbook + +## Scope + +This runbook re-certifies the read-only V2 query/stream beta from a bounded V1 +source slice. V1 remains authoritative. The procedure must not recreate V1, +write production Redis, expose V2 publicly or promote Rust/source authority. + +## Run + +From the repository root on the approved feature revision: + +```bash +make phase90b-test +make phase90b-certify +``` + +The certification uses isolated Compose projects, beta-only credentials, +loopback ports, dedicated AOF Redis and bounded durable stores. It validates the +real `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m` bridge, exact V1/V2 +closed-bar parity, active/passive fencing, outage recovery, security abuse, +capacity and cleanup. + +## Required Evidence + +- `upgrade/evidence/phase90b-isolated-v2-beta.json` +- `upgrade/evidence/phase90b-continuous-bridge.json` +- `upgrade/evidence/phase90b-capacity.json` +- `upgrade/evidence/phase90b-security-adversarial.json` +- `upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md` +- `upgrade/evidence/phase90b-evidence.sha256` + +Verify the bundle with: + +```bash +sha256sum -c upgrade/evidence/phase90b-evidence.sha256 +``` + +The decision must be `PASS_ISOLATED_NO_AUTHORITY_CUTOVER`. Generated market +events, canonical mismatches, duplicate timestamps, non-final bars, +execution-eligible events, production beta keys and cleanup counters must all +be zero. + +## Rollback + +The script removes its candidate projects, volumes and image tag automatically. +On interruption, remove only `qdl_phase90b_matrix` and `qdl_phase90b_bridge` +with both `phase7-beta` and `phase7-canary` profiles. Revoke beta credentials, +then verify V1 container identity/OpenAPI and zero `qdl:beta:v2:*` keys in +production Redis. Never restart or repair V1 to make this beta gate pass. + +## Promotion Boundary + +Passing this runbook permits isolated read-only beta review only. It does not +authorize Phase 9.1, public V2 exposure, critical-consumer sole dependency or +Rust/source authority promotion. diff --git a/docs/runbooks/phase90c-production-prerequisites.md b/docs/runbooks/phase90c-production-prerequisites.md new file mode 100644 index 0000000..332c0ba --- /dev/null +++ b/docs/runbooks/phase90c-production-prerequisites.md @@ -0,0 +1,98 @@ +# Phase 9.0-C Production Prerequisite Runbook + +## Scope + +This runbook certifies the infrastructure, security, recovery and exact-slice +approval prerequisites for a future Phase 9.1 Rust canary. It does not promote +Rust, expose V2 publicly, alter V1 Redis namespaces or grant write authority. +The frozen candidate remains `RUST_SHADOW` with public and legacy writes off. + +## Evidence Trust Boundary + +Evidence scope is part of the contract: + +- `TEST` proves a deterministic unit or contract behavior only. +- `LOCAL_REHEARSAL` proves an isolated same-host integration behavior only. +- `PRODUCTION` must originate from the real production control plane. +- `INDEPENDENT_FAILURE_DOMAIN` must cross a real independent failure domain. + +Never relabel local containers, debug telemetry, self-signed test credentials or +same-host replicas as production evidence. Evidence files contain identifiers, +checksums and measurements only; never put credentials, tokens or private keys +in an inventory. + +## Local Control-Plane Certification + +From the repository root on the approved feature revision: + +```bash +make phase90c-certify +sha256sum -c upgrade/evidence/phase90c-evidence.sha256 +``` + +Expected local decision is `NO_GO_EXTERNAL`. The command validates strict +schema/evidence handling, authority CAS and immutable audit behavior in an +isolated PostgreSQL instance, applicable Phase 8/9 contracts, and V1 identity +and health before/after evaluation. It removes the disposable database +container. It never writes production PostgreSQL or Redis. + +Required local evidence: + +- `upgrade/evidence/phase90c-production-prerequisites.json` +- `upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md` +- `upgrade/evidence/phase90c-authority-migration.json` +- `upgrade/evidence/phase90c-evidence.sha256` + +## Production Evidence Intake + +1. Freeze the exact candidate manifest and verify all digests and revisions. +2. Collect each gate from its owning production system. Use repository-relative, + immutable evidence artifacts and SHA-256 hashes; keep secrets external. +3. Use `PRODUCTION` only for real production evidence. DR evidence must use + `INDEPENDENT_FAILURE_DOMAIN`. +4. Validate without mutation: + +```bash +python3 scripts/phase90c_prerequisite_certification.py --inventory path/to/approved-production-inventory.yaml --expect GO --output upgrade/evidence/phase90c-production-go.json --report upgrade/evidence/PHASE90C_PRODUCTION_GO_REPORT.md +``` + +5. Independently verify evidence ownership, expiry, candidate digest, image + signature identity, consumer approvals, blast radius and hold window. +6. Apply `0006_phase9_authority_prerequisites.sql` only through the normal + reviewed database migration lane with backup/PITR confirmed. Register the + checksummed decision bundle through the restricted control-plane database + identity; ingestion/projector identities must not have insert/update access. +7. Obtain a separate operator approval for the exact slice. A general Phase 9 + approval is insufficient. + +A `GO` report is a prerequisite artifact, not an authority transition. Phase +9.1 still requires its own approved canary procedure. + +## Authority Safety + +`qdl_transition_authority` rejects stale state, revision, owner, lease and +partition-plan expectations. Entering `RUST_CANARY` or `RUST_PRIMARY` also +requires a non-negative terminal watermark, unexpired matching `GO` bundle and +a future hold window that does not outlive the bundle. Every successful +transition is appended to immutable audit history. + +The authority tables are additive. Do not mutate transition audit rows, reuse a +bundle for another candidate digest, or bypass the function with direct updates. +Sink-side fencing remains mandatory; the database record alone cannot prevent a +misconfigured producer from attempting a write. + +## Rollback And Revocation + +Before Phase 9.1, rollback is simply to keep V1 authoritative and remove only +disposable certification resources. Revoke, expire or supersede compromised +evidence and issue a new bundle; do not edit an existing bundle or audit row. +Any missing, expired, malformed, lower-scope or mismatched evidence returns the +decision to `NO_GO_EXTERNAL` and must not restart or promote a producer. + +## Current Decision + +Phase 9.0-C may close as `COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL`. Real +replicated transport, production observability, workload identity/RBAC/network, +external secret rotation, signed registry admission, PITR/object restore, +independent DR, production projector rebuild, complete consumer ownership and +exact operator approval remain external deployment gates. diff --git a/docs/runbooks/phase91-rust-canary.md b/docs/runbooks/phase91-rust-canary.md new file mode 100644 index 0000000..ee1a12d --- /dev/null +++ b/docs/runbooks/phase91-rust-canary.md @@ -0,0 +1,100 @@ +# Phase 9.1 Rust Canary Runbook + +## Scope + +Phase 9.1 certifies a provider-neutral, exact-slice Rust canary path while +Python V1 remains the sole authoritative public and legacy writer. The frozen +rehearsal slice is `BINANCE / USDM / PERPETUAL / TRADE / BTCUSDT / +partition-plan epoch 1`. + +The current Phase 9.0-C decision is `NO_GO_EXTERNAL`. Therefore this runbook +permits only isolated rehearsal. It cannot persist production +`RUST_CANARY`, publish into public/V1 namespaces, alter V1 Redis ownership or +authorize a cutover. + +## Build And Test + +Run from the repository root on the reviewed Phase 9.1 revision: + +```bash +make phase91-test +``` + +This builds the Rust builder and non-root runtime images, runs Rust format, +clippy with warnings denied, the complete Rust workspace, focused Phase 8-9 +regressions and the full Python suite. + +## Isolated Certification + +```bash +make phase91-certify +``` + +The certification: + +1. Replays the frozen authentic Binance provider capture through Python and + three clean Rust processes. +2. Requires exact canonical bytes, decimal/timestamp/identity/event hashes and + aggregate SHA-256 parity. +3. Uses a separate three-node TLS/ACL Kafka project and isolated Redis. +4. Exercises exact authority fencing, one-replica loss, below-min-ISR failure, + full broker restart, compacted authority recovery, audit ordering, + slow-consumer catch-up and rollback to `RUST_SHADOW`. +5. Requires zero public and legacy writes and unchanged V1 identity/health. +6. Cleans its containers, networks and volumes in `finally`. + +Expected decision on the current prerequisite bundle: + +```text +COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED +``` + +Required evidence: + +- `upgrade/evidence/phase91-rust-canary-certification.json` +- `upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md` +- `upgrade/evidence/phase91-evidence.sha256` + +Verify independently: + +```bash +sha256sum -c upgrade/evidence/phase91-evidence.sha256 +``` + +## Failure Interpretation + +Any semantic mismatch, stale owner/revision/lease/partition plan, wrong target, +duplicate watermark, guardrail block, broker durability failure, replay gap, +public/legacy write or V1 topology change fails the certification. Do not +weaken a gate or relabel local evidence as production evidence. + +A compacted authority topic is read directly by partition after restart; +complete state-transition history is verified separately on the non-compacted +audit topic. Fault tests stop broker processes while clients bootstrap only +through surviving broker names, avoiding Docker DNS artifacts. + +## Cleanup + +```bash +make phase91-clean +``` + +The target removes only the isolated Phase 9.1 project and its two disposable +images. It must not prune Docker globally or remove V1 volumes/images. + +After cleanup verify: + +```bash +docker ps -a --filter label=com.docker.compose.project=qdl_phase91_certification +curl -fsS http://127.0.0.1:8100/v1/health +``` + +## Production Promotion + +Production promotion remains forbidden until Phase 9.0-C yields a fresh, +unexpired exact-candidate `GO` bundle from production/independent failure +domains and an operator separately approves the exact slice, blast radius, +hold window and rollback manifest. At that point, rerun this certification +against the immutable candidate, verify all consumers and execute the formal +authority CAS/audit procedure. Never promote through an environment flag or +direct database edit. diff --git a/docs/runbooks/phase92-bounded-rust-primary.md b/docs/runbooks/phase92-bounded-rust-primary.md new file mode 100644 index 0000000..9c4d072 --- /dev/null +++ b/docs/runbooks/phase92-bounded-rust-primary.md @@ -0,0 +1,80 @@ +# Phase 9.2 Bounded Rust Primary Runbook + +## Authority Boundary + +Phase 9.2 is additive and exact-slice only. Current repository evidence is an +isolated rehearsal because Phase 9.0-C is `NO_GO_EXTERNAL`. Do not disable a +production Python subscription, write production authority, or point isolated +projection topics at real V1/V2 destinations. + +The only valid local close state is: + +```text +COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED +``` + +## Preconditions + +1. Worktree is on the approved Phase 9.2 branch and candidate digest matches + `config/phase9/candidate-slice.yaml`. +2. V1 health is HTTP 200 and its container identity/restart count are recorded. +3. The Phase 9.0-C decision remains checksummed and is not edited for testing. +4. Docker has enough bounded capacity for three disposable KRaft replicas. +5. No production topic, Redis namespace, credential, volume or subscription is + mounted into the rehearsal. + +## Certification + +```bash +cd /home/bobby/data_layer +make phase92-test +make phase92-migration +make phase92-certify +sha256sum -c upgrade/evidence/phase92-evidence.sha256 +``` + +The certification must prove: + +- authentic Python/Rust canonical parity has zero mismatch; +- direct primary bypass, stale owner/revision/lease/plan, N, N+2, duplicate and + wrong-target writes fail closed; +- terminal checkpoint and accepted handoff records are immutable; +- `RUST_CANARY -> RUST_PRIMARY -> BLOCKED -> ROLLBACK_PENDING -> + PYTHON_PRIMARY` is ordered and reconstructable after broker restart; +- isolated canonical/public/legacy projections all contain exactly watermarks + 101 through 181 with the owner boundary at 164/165; +- a fresh Rust process loading the terminal primary authority rejects all three + targets until each durable projection watermark is reconstructed, rejects W + again after restore and resumes all targets exactly at W+1; +- one replica loss still ACKs, below-min-ISR fails closed; +- production public/legacy writes remain zero; +- V1 topology and health are unchanged before/after; +- all disposable resources are absent after cleanup. + +## Production Promotion Gate + +Do not reuse local rehearsal evidence as production approval. A future primary +cutover additionally requires a fresh exact Phase 9.0-C `GO`, a completed real +`RUST_CANARY` hold, consumer registry approval, immutable rollback artifact, +change ticket and explicit operator approval naming the exact slice. + +At cutover, persist the old-owner checkpoint at W, accept clean handoff evidence, +execute the database CAS, load the new authority into final sink and projector, +then emit first authoritative watermark W+1. Disable only that Python +subscription after all of those gates pass. + +## Rollback + +Fence Rust first. Persist its final watermark, enter `ROLLBACK_PENDING`, accept +a Rust-to-Python handoff, grant Python a new authority revision and lease, then +resume from the reconciled next watermark. Never restart Python as an +uncoordinated writer. + +For local cleanup: + +```bash +make phase92-clean +``` + +This command removes only Phase 9.2 certification containers, networks, volumes +and images. It does not prune global Docker state or touch V1 data. diff --git a/docs/runbooks/phase93-hold-close-expand.md b/docs/runbooks/phase93-hold-close-expand.md new file mode 100644 index 0000000..ab78574 --- /dev/null +++ b/docs/runbooks/phase93-hold-close-expand.md @@ -0,0 +1,122 @@ +# Phase 9.3 Hold, Close And Expand Runbook + +## Current Authority Boundary + +Repository status may reach only: + +`COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED` + +while Phase 9.0-C is `NO_GO_EXTERNAL`. Local tests do not start a production +hold, close a production rollback window, authorize an expansion, decommission +Python or change authority. V1 remains authoritative. + +## Local Certification + +Run from `/home/bobby/data_layer`: + +```bash +make phase93-test +make phase93-migration +make phase93-certify +sha256sum -c upgrade/evidence/phase93-evidence.sha256 +``` + +Expected local evidence: + +- current Phase 9.0-C no-go rejects production closure; +- Phase 9.2 authentic replay remains parent provenance only; +- accelerated hold observations are marked `TEST_CONTROL_PLANE_FIXTURE`; +- PostgreSQL closure leaves authority state/revision/owner/lease/watermark + unchanged; +- all five expansion types remain + `INDEPENDENT_CERTIFICATION_REQUIRED` with no write authority; +- Python decommission remains blocked while any ownership, rollback or consumer + dependency exists; +- V1 health/topology is unchanged and disposable resources are absent. + +## Starting A Real Hold + +A real hold is legal only after a production-authorized exact +`RUST_PRIMARY` cutover and a fresh Phase 9.0-C `GO@. + +1. Freeze the hold policy and its digest. The approved minimum wall-clock + duration, sample interval, maximum sample gap, correctness zero-tolerance and + resource/lag thresholds cannot change inside one hold ID. +2. Persist the hold identity bound to slice, candidate, prerequisite bundle, + owner, authority revision, lease and partition-plan epoch. +3. Append observations from production telemetry. Do not backfill or interpolate + missing intervals. A changed owner/epoch or a missing interval blocks that + hold. +4. Persist a `BLOCKED@ decision immediately on any semantic mismatch, open gap, + duplicate external write, accepted stale writer, authority ambiguity, durable + ACK failure, projection mismatch, consumer checkpoint regression or + unexplained quality failure. +5. A blocked hold is immutable. Restart only with a new hold ID and a new + operator decision. +6. A passing decision requires the real wall-clock duration and terminal sample. + Accelerated fixtures, replay duration and same-host time compression never + count as production hold evidence. + +## Closing The Rollback Window + +Window closure is a governed audit decision, not a data-plane authority +transition and not deletion of the Python rollback manifest. + +Before calling `qdl_close_authority_window`, freeze: + +- passing production hold decision; +- consumer registry snapshot with every critical consumer ready, fully migrated, + rollback-ready and checkpointed through the current authority watermark; +- authority registry snapshot matching current `RUST_PRIMARY` + owner/revision/lease/partition/candidate/bundle; +- fresh production-scope rollback rehearsal reconciled through the same + watermark; +- operator approval bound to hold policy, ticket and bounded expiry. + +The function locks and rechecks the exact authority row. A concurrent authority +change, stale registry, expired rollback rehearsal or mismatched approval fails. +Successful closure inserts one immutable row and changes no authority field. + +## Expansion + +Create one manifest per expansion class: + +- `INSTRUMENT_PARTITION` +- `BBO` +- `L2_BOOK` +- `BAR_LIFECYCLE` +- `VENUE_MARKET` + +Each manifest needs a new candidate digest and capability-specific gate set. +Instrument/partition expansion also requires a newer partition-plan epoch. +Parent closure evidence is provenance only. Repeat provider-authentic parity, +chaos, capacity, authority handoff and rollback certification independently. +Never combine classes merely to reuse an approval. + +## Python Runtime Decommission + +Removal is denied until all are true: + +- runtime owns zero authority slices; +- no active rollback manifest references it; +- no registered consumer depends on it; +- every replacement rollback window is governed closed; +- repository cleanup has separate explicit approval; +- shared contracts, fixtures and provider/compatibility knowledge remain. + +Closing one slice does not authorize deleting a reusable adapter. + +## Incident And Rollback + +A closed window does not remove emergency rollback capability. On incident: + +1. fence Rust at final sink; +2. persist terminal watermark and incident evidence; +3. enter `BLOCKED@ then `ROLLBACK_PENDING@; +4. accept a new Rust-to-Python handoff under a newer revision/lease; +5. resume Python from the reconciled next watermark; +6. record a new audit decision. + +Do not delete or mutate hold/registry/closure/expansion rows. For local residue, +run `make phase93-clean`; it removes only containers whose names begin with +`qdl_phase93_` and performs no global Docker prune. diff --git a/docs/runbooks/v2-production-rust-authority-cutover.md b/docs/runbooks/v2-production-rust-authority-cutover.md new file mode 100644 index 0000000..1b11ba9 --- /dev/null +++ b/docs/runbooks/v2-production-rust-authority-cutover.md @@ -0,0 +1,262 @@ +# V2 Production And Rust Authority Cutover Runbook + +## Purpose + +This runbook moves Data Layer consumers from the current V1 service to stable +V2 and promotes Rust as canonical realtime authority without creating two +writers. It implements Phase C in +`DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md`. + +This is a gated procedure, not one shell script. Stop at every approval point. +DNSE is excluded until its separate provider gate passes. + +## Current Baseline + +- V1: `data-layer:v0.1.0`, loopback port `8100`, production authority. +- Stable V2 query ports: `18201`, `18202`. +- Stable V2 stream HTTP ports: `18210`, `18211`. +- Stable V2 stream gRPC ports: `18220`, `18221`. +- Stable project: `qdl_v2_stable_candidate`. +- Initial V2 authority: `RUST_SHADOW`. +- Initial venues: Binance and OKX only. +- V1, current Redis and current provider processes are not restarted by the + merge or isolated-deploy steps. + +## Gate 0 - Merge The Certified Feature Branch Into Dev + +The assistant prepares and tests the feature branch. The operator reviews and +merges it; the assistant never merges `dev` or `main` without explicit +approval. + +```bash +cd /home/bobby/data_layer +git status --short +git diff --check +git log --oneline dev..feat/v2-stable-rust-binance-okx +git push -u origin feat/v2-stable-rust-binance-okx +``` + +Open and verify the pull request: + +```bash +gh pr create \ + --base dev \ + --head feat/v2-stable-rust-binance-okx \ + --title "feat(data-layer): release stable V2 Rust multivenue core" \ + --body-file docs/runbooks/v2-production-rust-authority-cutover.md +gh pr checks --watch +``` + +Review gates: + +- all required CI checks green; +- no secret, runtime state, provider SDK snapshot or generated cache tracked; +- the PR contains Phase B evidence and this Phase C plan; +- V1 compatibility tests remain green; +- no merge targets `main`. + +After the operator merges the PR in GitHub: + +```bash +cd /home/bobby/data_layer +git switch dev +git fetch origin +git pull --ff-only origin dev +git status --short +git switch -c feat/v2-production-authority-cutover +``` + +Stop and record the new branch SHA. Do not build production artifacts from the +old unmerged feature worktree. + +## Gate 1 - Production Authority Wiring + +The stable runtime currently accepts only `RUST_SHADOW`. Before any production +route changes, the new feature branch must implement and test: + +1. PostgreSQL authority CAS plus immutable handoff/audit records. +2. A transactional authority outbox written in the same database transaction. +3. An idempotent dispatcher to the compacted Kafka authority topic. +4. Rust startup/restart reconstruction from authority plus target watermarks. +5. Phase 9.2 sink fencing on every canonical/public/compatibility durable write. +6. Operator CLI commands for status, preflight, canary, promote, block and + rollback. +7. Fail-closed handling for stale owner, revision, lease, partition plan, + missing handoff and incomplete target recovery. + +Mandatory tests include migration idempotency, outbox retry/crash recovery, +compacted-topic rebuild, stale-writer races, broker loss, process restart, +W/W+1 handoff, rollback, exact market-data parity and V1 compatibility. + +Do not continue while the stable binary still rejects `RUST_CANARY` or +`RUST_PRIMARY`, or while authority can be changed by environment variable +alone. + +## Gate 2 - Build Immutable Artifacts + +After Gate 1 is committed and CI-green: + +```bash +cd /home/bobby/data_layer +export RELEASE_SHA="$(git rev-parse --short=12 HEAD)" +export PYTHON_IMAGE="qdl-v2-python:2.0.0-$RELEASE_SHA" +export RUST_IMAGE="qdl-v2-rust:2.0.0-$RELEASE_SHA" + +docker build --pull \ + --label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \ + -t "$PYTHON_IMAGE" . +docker build --pull -f Dockerfile.phase8-rust \ + --label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \ + -t "$RUST_IMAGE" . + +docker image inspect "$PYTHON_IMAGE" "$RUST_IMAGE" \ + --format '{{.RepoTags}} {{.Id}}' +``` + +Run the full Python/Rust/contract/security suites and generate SBOM, provenance +and image-digest evidence. Retain V1 and one tested rollback image pair. Remove +only exact failed-build tags and project-scoped test resources. + +## Gate 3 - Prepare The Isolated Stable Bundle + +Use a fresh private runtime directory and existing approved Kafka client CA. +Never commit `stable.env`, private keys or the generated bundle. + +```bash +export QDL_RELEASE_ROOT="/home/bobby/.local/state/qdl-v2/$RELEASE_SHA" +install -d -m 0700 "$QDL_RELEASE_ROOT" + +python scripts/phaseb_prepare_stable_candidate.py \ + --python-image "$PYTHON_IMAGE" \ + --rust-image "$RUST_IMAGE" \ + --cert-dir /path/to/approved/phase8-certificates \ + --output-dir "$QDL_RELEASE_ROOT" +``` + +The manifest must report contract `2.0.0`, authority `RUST_SHADOW`, +`cutover_authorized=false`, immutable image IDs, five consumer manifests and +no recorded secret values. + +## Gate 4 - Start Isolated V2 + +Start only dedicated infrastructure first: + +```bash +docker compose \ + --env-file "$QDL_RELEASE_ROOT/stable.env" \ + -f docker-compose.v2-stable.yml \ + up -d kafka1 kafka2 kafka3 stable_redis stable_state_init stable_tls_init + +python scripts/phaseb_bootstrap_stable_broker.py \ + --env-file "$QDL_RELEASE_ROOT/stable.env" +``` + +Then start Binance/OKX acquisition, Rust core and Python projection/API roles. +Do not enable profile `stable-vn`. + +```bash +docker compose \ + --env-file "$QDL_RELEASE_ROOT/stable.env" \ + -f docker-compose.v2-stable.yml \ + up -d \ + rust_core rust_core_2 rust_core_3 \ + ingestor_binance_usdm ingestor_binance_spot \ + ingestor_okx_swap ingestor_okx_spot \ + binance_bar_edge \ + stream_v2_active stream_v2_passive projector_v2 \ + query_v2_1 query_v2_2 +``` + +Acceptance: + +```bash +curl --fail --silent http://127.0.0.1:18201/health/ready +curl --fail --silent http://127.0.0.1:18202/health/ready +curl --fail --silent http://127.0.0.1:18210/health/ready +curl --fail --silent http://127.0.0.1:18211/health/ready +curl --fail --silent http://127.0.0.1:8100/v1/health +``` + +Require authentic Binance/OKX data, zero unexplained gap/duplicate/quarantine, +bounded broker/projector lag, exact replica results and V1 unchanged. + +Rollback before consumer migration: + +```bash +docker compose \ + --env-file "$QDL_RELEASE_ROOT/stable.env" \ + -f docker-compose.v2-stable.yml \ + down +``` + +Do not add `-v`; preserve evidence until the failed gate is understood. + +## Gate 5 - Paper Consumer Canary + +Move only explicitly named manifests, in this order: + +1. monitoring; +2. one paper alpha; +3. Trading System paper market-data adapter; +4. remaining approved paper consumers. + +For each consumer, record V1 cursor/watermark, V2 warmup result, first live +cursor, restart cursor, freshness, gaps, duplicates, source/session status and +rollback route. No sandbox/live order path is included. + +Rollback changes only that consumer endpoint/SDK config back to V1. + +## Gate 6 - Exact-Slice Authority Approval + +Before any `RUST_CANARY` or `RUST_PRIMARY` transition, produce this packet: + +```text +release_sha: +python_image_id: +rust_image_id: +slice_id: +binding_ids: +old_owner: +new_owner: +authority_revision: +lease_epoch: +partition_plan_epoch: +terminal_watermark_W: +authority_topic: +raw/canonical/public/compatibility topics: +consumer_groups: +V2_ports: +volumes: +secret_references: +affected_consumers: +canary_hold: +primary_hold: +rollback_command: +operator: +change_ticket: +``` + +The operator must explicitly approve this exact packet. + +The transition follows only: + +```text +PYTHON_PRIMARY + -> RUST_SHADOW + -> RUST_CANARY + -> RUST_PRIMARY +``` + +The old writer is fenced at `W`; Rust reconstructs every required target +through `W` and first publishes at `W+1`. Any failed gate enters `BLOCKED` +and rolls back using a newer authority revision and accepted reverse handoff. + +## Gate 7 - Expand And Release + +Hold the first primary slice for the approved window. Expand one venue/feed +slice at a time; no inherited certification. Keep V1 available until every +registered consumer has migrated and rollback has been exercised. + +Only then may the operator approve stable V2 public routing, a V1 sunset date, +a release PR `dev -> main`, and tag/release publication. DNSE remains disabled +until its provider-specific external gates pass. diff --git a/generated/python/qdl/common/v1/common_pb2.py b/generated/python/qdl/common/v1/common_pb2.py index a600543..017dc18 100644 --- a/generated/python/qdl/common/v1/common_pb2.py +++ b/generated/python/qdl/common/v1/common_pb2.py @@ -24,27 +24,29 @@ -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1aqdl/common/v1/common.proto\x12\rqdl.common.v1\"\x99\x01\n\x0c\x44\x65\x63imalValue\x12\x1c\n\x08mantissa\x18\x01 \x01(\x12H\x00R\x08mantissa\x12%\n\rmantissa_text\x18\x03 \x01(\tH\x00R\x0cmantissaText\x12\x14\n\x05scale\x18\x02 \x01(\x11R\x05scale\x12\x1f\n\x0bsource_text\x18\x04 \x01(\tR\nsourceTextB\r\n\x0b\x63oefficient*\x92\x01\n\nSourceRole\x12\x1b\n\x17SOURCE_ROLE_UNSPECIFIED\x10\x00\x12\x17\n\x13SOURCE_ROLE_PRIMARY\x10\x01\x12\x19\n\x15SOURCE_ROLE_SECONDARY\x10\x02\x12\x19\n\x15SOURCE_ROLE_REFERENCE\x10\x03\x12\x18\n\x14SOURCE_ROLE_BACKFILL\x10\x04*\x8d\x01\n\x0fSourceAuthority\x12 \n\x1cSOURCE_AUTHORITY_UNSPECIFIED\x10\x00\x12\x1d\n\x19SOURCE_AUTHORITY_DISABLED\x10\x01\x12\x1b\n\x17SOURCE_AUTHORITY_SHADOW\x10\x02\x12\x1c\n\x18SOURCE_AUTHORITY_PRIMARY\x10\x03*\xd6\x04\n\x0bQualityFlag\x12\x1c\n\x18QUALITY_FLAG_UNSPECIFIED\x10\x00\x12$\n QUALITY_FLAG_SOURCE_TIME_MISSING\x10\x01\x12!\n\x1dQUALITY_FLAG_SEQUENCE_MISSING\x10\x02\x12$\n QUALITY_FLAG_SEQUENCE_GAP_BEFORE\x10\x03\x12\x1a\n\x16QUALITY_FLAG_DUPLICATE\x10\x04\x12\x1d\n\x19QUALITY_FLAG_OUT_OF_ORDER\x10\x05\x12\x15\n\x11QUALITY_FLAG_LATE\x10\x06\x12\x16\n\x12QUALITY_FLAG_STALE\x10\x07\x12\x1f\n\x1bQUALITY_FLAG_PARSER_PARTIAL\x10\x08\x12\x1e\n\x1aQUALITY_FLAG_FIELD_MISSING\x10\t\x12#\n\x1fQUALITY_FLAG_PRECISION_ADJUSTED\x10\n\x12 \n\x1cQUALITY_FLAG_SOURCE_FALLBACK\x10\x0b\x12&\n\"QUALITY_FLAG_SOURCE_REFERENCE_ONLY\x10\x0c\x12\x1b\n\x17QUALITY_FLAG_BACKFILLED\x10\r\x12\x18\n\x14QUALITY_FLAG_REVISED\x10\x0e\x12 \n\x1cQUALITY_FLAG_CHECKSUM_FAILED\x10\x0f\x12 \n\x1cQUALITY_FLAG_RESYNC_REQUIRED\x10\x10\x12%\n!QUALITY_FLAG_CLOCK_SKEW_SUSPECTED\x10\x11*\xff\x03\n\x12\x46\x65\x65\x64LifecycleState\x12$\n FEED_LIFECYCLE_STATE_UNSPECIFIED\x10\x00\x12!\n\x1d\x46\x45\x45\x44_LIFECYCLE_STATE_DISABLED\x10\x01\x12!\n\x1d\x46\x45\x45\x44_LIFECYCLE_STATE_STARTING\x10\x02\x12#\n\x1f\x46\x45\x45\x44_LIFECYCLE_STATE_CONNECTING\x10\x03\x12$\n FEED_LIFECYCLE_STATE_SUBSCRIBING\x10\x04\x12 \n\x1c\x46\x45\x45\x44_LIFECYCLE_STATE_SYNCING\x10\x05\x12\x1d\n\x19\x46\x45\x45\x44_LIFECYCLE_STATE_LIVE\x10\x06\x12!\n\x1d\x46\x45\x45\x44_LIFECYCLE_STATE_DEGRADED\x10\x07\x12\x1f\n\x1b\x46\x45\x45\x44_LIFECYCLE_STATE_GAPPED\x10\x08\x12\"\n\x1e\x46\x45\x45\x44_LIFECYCLE_STATE_RESYNCING\x10\t\x12\x1e\n\x1a\x46\x45\x45\x44_LIFECYCLE_STATE_STALE\x10\n\x12 \n\x1c\x46\x45\x45\x44_LIFECYCLE_STATE_OFFLINE\x10\x0b\x12\x1f\n\x1b\x46\x45\x45\x44_LIFECYCLE_STATE_HALTED\x10\x0c\x12&\n\"FEED_LIFECYCLE_STATE_MARKET_CLOSED\x10\r*`\n\rAggressorSide\x12\x1e\n\x1a\x41GGRESSOR_SIDE_UNSPECIFIED\x10\x00\x12\x16\n\x12\x41GGRESSOR_SIDE_BUY\x10\x01\x12\x17\n\x13\x41GGRESSOR_SIDE_SELL\x10\x02*K\n\x08\x42ookSide\x12\x19\n\x15\x42OOK_SIDE_UNSPECIFIED\x10\x00\x12\x11\n\rBOOK_SIDE_BID\x10\x01\x12\x11\n\rBOOK_SIDE_ASK\x10\x02*\x95\x01\n\tBarOrigin\x12\x1a\n\x16\x42\x41R_ORIGIN_UNSPECIFIED\x10\x00\x12\x1b\n\x17\x42\x41R_ORIGIN_VENUE_NATIVE\x10\x01\x12\x19\n\x15\x42\x41R_ORIGIN_AGGREGATED\x10\x02\x12\x19\n\x15\x42\x41R_ORIGIN_BACKFILLED\x10\x03\x12\x19\n\x15\x42\x41R_ORIGIN_RECONCILED\x10\x04\x62\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1aqdl/common/v1/common.proto\x12\rqdl.common.v1\"\x99\x01\n\x0c\x44\x65\x63imalValue\x12\x1c\n\x08mantissa\x18\x01 \x01(\x12H\x00R\x08mantissa\x12%\n\rmantissa_text\x18\x03 \x01(\tH\x00R\x0cmantissaText\x12\x14\n\x05scale\x18\x02 \x01(\x11R\x05scale\x12\x1f\n\x0bsource_text\x18\x04 \x01(\tR\nsourceTextB\r\n\x0b\x63oefficient*\x9f\x01\n\x0cQuantityUnit\x12\x1d\n\x19QUANTITY_UNIT_UNSPECIFIED\x10\x00\x12\x1c\n\x18QUANTITY_UNIT_BASE_ASSET\x10\x01\x12\x1d\n\x19QUANTITY_UNIT_QUOTE_ASSET\x10\x02\x12\x1a\n\x16QUANTITY_UNIT_CONTRACT\x10\x03\x12\x17\n\x13QUANTITY_UNIT_SHARE\x10\x04*\x92\x01\n\nSourceRole\x12\x1b\n\x17SOURCE_ROLE_UNSPECIFIED\x10\x00\x12\x17\n\x13SOURCE_ROLE_PRIMARY\x10\x01\x12\x19\n\x15SOURCE_ROLE_SECONDARY\x10\x02\x12\x19\n\x15SOURCE_ROLE_REFERENCE\x10\x03\x12\x18\n\x14SOURCE_ROLE_BACKFILL\x10\x04*\x8d\x01\n\x0fSourceAuthority\x12 \n\x1cSOURCE_AUTHORITY_UNSPECIFIED\x10\x00\x12\x1d\n\x19SOURCE_AUTHORITY_DISABLED\x10\x01\x12\x1b\n\x17SOURCE_AUTHORITY_SHADOW\x10\x02\x12\x1c\n\x18SOURCE_AUTHORITY_PRIMARY\x10\x03*\xd6\x04\n\x0bQualityFlag\x12\x1c\n\x18QUALITY_FLAG_UNSPECIFIED\x10\x00\x12$\n QUALITY_FLAG_SOURCE_TIME_MISSING\x10\x01\x12!\n\x1dQUALITY_FLAG_SEQUENCE_MISSING\x10\x02\x12$\n QUALITY_FLAG_SEQUENCE_GAP_BEFORE\x10\x03\x12\x1a\n\x16QUALITY_FLAG_DUPLICATE\x10\x04\x12\x1d\n\x19QUALITY_FLAG_OUT_OF_ORDER\x10\x05\x12\x15\n\x11QUALITY_FLAG_LATE\x10\x06\x12\x16\n\x12QUALITY_FLAG_STALE\x10\x07\x12\x1f\n\x1bQUALITY_FLAG_PARSER_PARTIAL\x10\x08\x12\x1e\n\x1aQUALITY_FLAG_FIELD_MISSING\x10\t\x12#\n\x1fQUALITY_FLAG_PRECISION_ADJUSTED\x10\n\x12 \n\x1cQUALITY_FLAG_SOURCE_FALLBACK\x10\x0b\x12&\n\"QUALITY_FLAG_SOURCE_REFERENCE_ONLY\x10\x0c\x12\x1b\n\x17QUALITY_FLAG_BACKFILLED\x10\r\x12\x18\n\x14QUALITY_FLAG_REVISED\x10\x0e\x12 \n\x1cQUALITY_FLAG_CHECKSUM_FAILED\x10\x0f\x12 \n\x1cQUALITY_FLAG_RESYNC_REQUIRED\x10\x10\x12%\n!QUALITY_FLAG_CLOCK_SKEW_SUSPECTED\x10\x11*\xff\x03\n\x12\x46\x65\x65\x64LifecycleState\x12$\n FEED_LIFECYCLE_STATE_UNSPECIFIED\x10\x00\x12!\n\x1d\x46\x45\x45\x44_LIFECYCLE_STATE_DISABLED\x10\x01\x12!\n\x1d\x46\x45\x45\x44_LIFECYCLE_STATE_STARTING\x10\x02\x12#\n\x1f\x46\x45\x45\x44_LIFECYCLE_STATE_CONNECTING\x10\x03\x12$\n FEED_LIFECYCLE_STATE_SUBSCRIBING\x10\x04\x12 \n\x1c\x46\x45\x45\x44_LIFECYCLE_STATE_SYNCING\x10\x05\x12\x1d\n\x19\x46\x45\x45\x44_LIFECYCLE_STATE_LIVE\x10\x06\x12!\n\x1d\x46\x45\x45\x44_LIFECYCLE_STATE_DEGRADED\x10\x07\x12\x1f\n\x1b\x46\x45\x45\x44_LIFECYCLE_STATE_GAPPED\x10\x08\x12\"\n\x1e\x46\x45\x45\x44_LIFECYCLE_STATE_RESYNCING\x10\t\x12\x1e\n\x1a\x46\x45\x45\x44_LIFECYCLE_STATE_STALE\x10\n\x12 \n\x1c\x46\x45\x45\x44_LIFECYCLE_STATE_OFFLINE\x10\x0b\x12\x1f\n\x1b\x46\x45\x45\x44_LIFECYCLE_STATE_HALTED\x10\x0c\x12&\n\"FEED_LIFECYCLE_STATE_MARKET_CLOSED\x10\r*`\n\rAggressorSide\x12\x1e\n\x1a\x41GGRESSOR_SIDE_UNSPECIFIED\x10\x00\x12\x16\n\x12\x41GGRESSOR_SIDE_BUY\x10\x01\x12\x17\n\x13\x41GGRESSOR_SIDE_SELL\x10\x02*K\n\x08\x42ookSide\x12\x19\n\x15\x42OOK_SIDE_UNSPECIFIED\x10\x00\x12\x11\n\rBOOK_SIDE_BID\x10\x01\x12\x11\n\rBOOK_SIDE_ASK\x10\x02*\x95\x01\n\tBarOrigin\x12\x1a\n\x16\x42\x41R_ORIGIN_UNSPECIFIED\x10\x00\x12\x1b\n\x17\x42\x41R_ORIGIN_VENUE_NATIVE\x10\x01\x12\x19\n\x15\x42\x41R_ORIGIN_AGGREGATED\x10\x02\x12\x19\n\x15\x42\x41R_ORIGIN_BACKFILLED\x10\x03\x12\x19\n\x15\x42\x41R_ORIGIN_RECONCILED\x10\x04\x62\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) _builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'qdl.common.v1.common_pb2', _globals) if not _descriptor._USE_C_DESCRIPTORS: DESCRIPTOR._loaded_options = None - _globals['_SOURCEROLE']._serialized_start=202 - _globals['_SOURCEROLE']._serialized_end=348 - _globals['_SOURCEAUTHORITY']._serialized_start=351 - _globals['_SOURCEAUTHORITY']._serialized_end=492 - _globals['_QUALITYFLAG']._serialized_start=495 - _globals['_QUALITYFLAG']._serialized_end=1093 - _globals['_FEEDLIFECYCLESTATE']._serialized_start=1096 - _globals['_FEEDLIFECYCLESTATE']._serialized_end=1607 - _globals['_AGGRESSORSIDE']._serialized_start=1609 - _globals['_AGGRESSORSIDE']._serialized_end=1705 - _globals['_BOOKSIDE']._serialized_start=1707 - _globals['_BOOKSIDE']._serialized_end=1782 - _globals['_BARORIGIN']._serialized_start=1785 - _globals['_BARORIGIN']._serialized_end=1934 + _globals['_QUANTITYUNIT']._serialized_start=202 + _globals['_QUANTITYUNIT']._serialized_end=361 + _globals['_SOURCEROLE']._serialized_start=364 + _globals['_SOURCEROLE']._serialized_end=510 + _globals['_SOURCEAUTHORITY']._serialized_start=513 + _globals['_SOURCEAUTHORITY']._serialized_end=654 + _globals['_QUALITYFLAG']._serialized_start=657 + _globals['_QUALITYFLAG']._serialized_end=1255 + _globals['_FEEDLIFECYCLESTATE']._serialized_start=1258 + _globals['_FEEDLIFECYCLESTATE']._serialized_end=1769 + _globals['_AGGRESSORSIDE']._serialized_start=1771 + _globals['_AGGRESSORSIDE']._serialized_end=1867 + _globals['_BOOKSIDE']._serialized_start=1869 + _globals['_BOOKSIDE']._serialized_end=1944 + _globals['_BARORIGIN']._serialized_start=1947 + _globals['_BARORIGIN']._serialized_end=2096 _globals['_DECIMALVALUE']._serialized_start=46 _globals['_DECIMALVALUE']._serialized_end=199 # @@protoc_insertion_point(module_scope) diff --git a/generated/python/qdl/marketdata/v2/market_data_pb2.py b/generated/python/qdl/marketdata/v2/market_data_pb2.py index a3231de..c52d54e 100644 --- a/generated/python/qdl/marketdata/v2/market_data_pb2.py +++ b/generated/python/qdl/marketdata/v2/market_data_pb2.py @@ -26,35 +26,37 @@ from qdl.quality.v1 import quality_pb2 as qdl_dot_quality_dot_v1_dot_quality__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n#qdl/marketdata/v2/market_data.proto\x12\x11qdl.marketdata.v2\x1a\x1aqdl/common/v1/common.proto\x1a\x1cqdl/quality/v1/quality.proto\"\xac\x02\n\x05Trade\x12&\n\x0fnative_trade_id\x18\x01 \x01(\tR\rnativeTradeId\x12\x31\n\x05price\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x05price\x12\x37\n\x08quantity\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08quantity\x12\x43\n\x0e\x61ggressor_side\x18\x04 \x01(\x0e\x32\x1c.qdl.common.v1.AggressorSideR\raggressorSide\x12$\n\x0eis_block_trade\x18\x05 \x01(\x08R\x0cisBlockTrade\x12$\n\x0eis_buyer_maker\x18\x06 \x01(\x08R\x0cisBuyerMaker\"\x91\x02\n\x05Quote\x12\x38\n\tbid_price\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08\x62idPrice\x12>\n\x0c\x62id_quantity\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x0b\x62idQuantity\x12\x38\n\task_price\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08\x61skPrice\x12>\n\x0c\x61sk_quantity\x18\x04 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x0b\x61skQuantity\x12\x14\n\x05level\x18\x05 \x01(\rR\x05level\"\xf6\x04\n\x03\x42\x61r\x12\x1a\n\x08interval\x18\x01 \x01(\tR\x08interval\x12 \n\x0copen_time_ns\x18\x02 \x01(\x03R\nopenTimeNs\x12\"\n\rclose_time_ns\x18\x03 \x01(\x03R\x0b\x63loseTimeNs\x12/\n\x04open\x18\x04 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x04open\x12/\n\x04high\x18\x05 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x04high\x12-\n\x03low\x18\x06 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x03low\x12\x31\n\x05\x63lose\x18\x07 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x05\x63lose\x12\x33\n\x06volume\x18\x08 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x06volume\x12\x1f\n\x0btrade_count\x18\t \x01(\x04R\ntradeCount\x12\x19\n\x08is_final\x18\n \x01(\x08R\x07isFinal\x12\x1a\n\x08revision\x18\x0b \x01(\rR\x08revision\x12\x30\n\x06origin\x18\x0c \x01(\x0e\x32\x18.qdl.common.v1.BarOriginR\x06origin\x12=\n\tlifecycle\x18\r \x01(\x0e\x32\x1f.qdl.marketdata.v2.BarLifecycleR\tlifecycle\x12\x33\n\x13supersedes_event_id\x18\x0e \x01(\x0cH\x00R\x11supersedesEventId\x88\x01\x01\x42\x16\n\x14_supersedes_event_id\"\xc5\x01\n\tBookLevel\x12+\n\x04side\x18\x01 \x01(\x0e\x32\x17.qdl.common.v1.BookSideR\x04side\x12\x31\n\x05price\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x05price\x12\x37\n\x08quantity\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08quantity\x12\x1f\n\x0border_count\x18\x04 \x01(\rR\norderCount\"\xa4\x01\n\x11OrderBookSnapshot\x12\'\n\x0fnative_sequence\x18\x01 \x01(\tR\x0enativeSequence\x12\x1a\n\x08\x63hecksum\x18\x02 \x01(\tR\x08\x63hecksum\x12\x34\n\x06levels\x18\x03 \x03(\x0b\x32\x1c.qdl.marketdata.v2.BookLevelR\x06levels\x12\x14\n\x05\x64\x65pth\x18\x04 \x01(\rR\x05\x64\x65pth\"\x8b\x02\n\x0eOrderBookDelta\x12\x32\n\x15native_sequence_start\x18\x01 \x01(\tR\x13nativeSequenceStart\x12.\n\x13native_sequence_end\x18\x02 \x01(\tR\x11nativeSequenceEnd\x12+\n\x11snapshot_sequence\x18\x03 \x01(\tR\x10snapshotSequence\x12\x1a\n\x08\x63hecksum\x18\x04 \x01(\tR\x08\x63hecksum\x12\x36\n\x07updates\x18\x05 \x03(\x0b\x32\x1c.qdl.marketdata.v2.BookLevelR\x07updates\x12\x14\n\x05reset\x18\x06 \x01(\x08R\x05reset\"\xb5\x01\n\x0b\x46undingRate\x12/\n\x04rate\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x04rate\x12&\n\x0f\x66unding_time_ns\x18\x02 \x01(\x03R\rfundingTimeNs\x12\x34\n\x14next_funding_time_ns\x18\x03 \x01(\x03H\x00R\x11nextFundingTimeNs\x88\x01\x01\x42\x17\n\x15_next_funding_time_ns\"\x92\x01\n\x0cOpenInterest\x12\x37\n\x08quantity\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08quantity\x12<\n\x08notional\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x00R\x08notional\x88\x01\x01\x42\x0b\n\t_notional\"\x8a\x01\n\x0eMarkIndexPrice\x12:\n\nmark_price\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\tmarkPrice\x12<\n\x0bindex_price\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\nindexPrice\"\xc8\x03\n\x06Ticker\x12:\n\nlast_price\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\tlastPrice\x12\x45\n\rlast_quantity\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x00R\x0clastQuantity\x88\x01\x01\x12;\n\x08open_24h\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x01R\x07open24h\x88\x01\x01\x12;\n\x08high_24h\x18\x04 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x02R\x07high24h\x88\x01\x01\x12\x39\n\x07low_24h\x18\x05 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x03R\x06low24h\x88\x01\x01\x12?\n\nvolume_24h\x18\x06 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x04R\tvolume24h\x88\x01\x01\x42\x10\n\x0e_last_quantityB\x0b\n\t_open_24hB\x0b\n\t_high_24hB\n\n\x08_low_24hB\r\n\x0b_volume_24h\"\x93\x10\n\rEventEnvelope\x12\x1f\n\x0bschema_name\x18\x01 \x01(\tR\nschemaName\x12!\n\x0cschema_major\x18\x02 \x01(\rR\x0bschemaMajor\x12!\n\x0cschema_minor\x18\x03 \x01(\rR\x0bschemaMinor\x12\x19\n\x08\x65vent_id\x18\x04 \x01(\x0cR\x07\x65ventId\x12%\n\x0einstrument_uid\x18\x05 \x01(\tR\rinstrumentUid\x12#\n\rinstrument_id\x18\x06 \x01(\tR\x0cinstrumentId\x12/\n\x13instrument_revision\x18\x07 \x01(\x04R\x12instrumentRevision\x12\x14\n\x05venue\x18\x08 \x01(\tR\x05venue\x12\x16\n\x06market\x18\t \x01(\tR\x06market\x12!\n\x0cproduct_type\x18\n \x01(\tR\x0bproductType\x12#\n\rnative_symbol\x18\x0b \x01(\tR\x0cnativeSymbol\x12\x1a\n\x08provider\x18\x0c \x01(\tR\x08provider\x12\x1b\n\tsource_id\x18\r \x01(\tR\x08sourceId\x12:\n\x0bsource_role\x18\x0e \x01(\x0e\x32\x19.qdl.common.v1.SourceRoleR\nsourceRole\x12\x1f\n\x0blease_epoch\x18\x0f \x01(\x04R\nleaseEpoch\x12/\n\x14source_event_time_ns\x18\x10 \x01(\x03R\x11sourceEventTimeNs\x12$\n\x0ereceived_at_ns\x18\x11 \x01(\x03R\x0creceivedAtNs\x12(\n\x10normalized_at_ns\x18\x12 \x01(\x03R\x0enormalizedAtNs\x12&\n\x0fpublished_at_ns\x18\x13 \x01(\x03R\rpublishedAtNs\x12\'\n\x0fsource_sequence\x18\x14 \x01(\tR\x0esourceSequence\x12-\n\x12partition_sequence\x18\x15 \x01(\x04R\x11partitionSequence\x12-\n\x12normalizer_version\x18\x16 \x01(\tR\x11normalizerVersion\x12\'\n\x0f\x61\x64\x61pter_version\x18\x17 \x01(\tR\x0e\x61\x64\x61pterVersion\x12?\n\rquality_flags\x18\x18 \x03(\x0e\x32\x1a.qdl.common.v1.QualityFlagR\x0cqualityFlags\x12(\n\x10raw_payload_hash\x18\x19 \x01(\x0cR\x0erawPayloadHash\x12%\n\x0e\x63orrelation_id\x18\x1a \x01(\tR\rcorrelationId\x12\'\n\x0f\x63onfig_revision\x18\x1b \x01(\x04R\x0e\x63onfigRevision\x12*\n\x11source_session_id\x18\x1c \x01(\tR\x0fsourceSessionId\x12\x33\n\x15\x63onnection_generation\x18\x1d \x01(\x04R\x14\x63onnectionGeneration\x12-\n\x12\x61uthority_revision\x18\x1e \x01(\x04R\x11\x61uthorityRevision\x12\x30\n\x14partition_plan_epoch\x18\x1f \x01(\x04R\x12partitionPlanEpoch\x12\x34\n\x16\x63\x61nonical_payload_hash\x18 \x01(\x0cR\x14\x63\x61nonicalPayloadHash\x12$\n\x0eraw_capture_id\x18! \x01(\x0cR\x0crawCaptureId\x12\x30\n\x05trade\x18( \x01(\x0b\x32\x18.qdl.marketdata.v2.TradeH\x00R\x05trade\x12\x30\n\x05quote\x18) \x01(\x0b\x32\x18.qdl.marketdata.v2.QuoteH\x00R\x05quote\x12*\n\x03\x62\x61r\x18* \x01(\x0b\x32\x16.qdl.marketdata.v2.BarH\x00R\x03\x62\x61r\x12K\n\rbook_snapshot\x18+ \x01(\x0b\x32$.qdl.marketdata.v2.OrderBookSnapshotH\x00R\x0c\x62ookSnapshot\x12\x42\n\nbook_delta\x18, \x01(\x0b\x32!.qdl.marketdata.v2.OrderBookDeltaH\x00R\tbookDelta\x12\x43\n\x0c\x66unding_rate\x18- \x01(\x0b\x32\x1e.qdl.marketdata.v2.FundingRateH\x00R\x0b\x66undingRate\x12\x46\n\ropen_interest\x18. \x01(\x0b\x32\x1f.qdl.marketdata.v2.OpenInterestH\x00R\x0copenInterest\x12M\n\x10mark_index_price\x18/ \x01(\x0b\x32!.qdl.marketdata.v2.MarkIndexPriceH\x00R\x0emarkIndexPrice\x12\x33\n\x06ticker\x18\x30 \x01(\x0b\x32\x19.qdl.marketdata.v2.TickerH\x00R\x06ticker\x12?\n\nfeed_state\x18\x31 \x01(\x0b\x32\x1e.qdl.quality.v1.FeedStateEventH\x00R\tfeedState\x12G\n\rquality_event\x18\x32 \x01(\x0b\x32 .qdl.quality.v1.DataQualityEventH\x00R\x0cqualityEventB\t\n\x07payload*\x9d\x01\n\x0c\x42\x61rLifecycle\x12\x1d\n\x19\x42\x41R_LIFECYCLE_UNSPECIFIED\x10\x00\x12\x1d\n\x19\x42\x41R_LIFECYCLE_IN_PROGRESS\x10\x01\x12\x17\n\x13\x42\x41R_LIFECYCLE_FINAL\x10\x02\x12\x19\n\x15\x42\x41R_LIFECYCLE_REVISED\x10\x03\x12\x1b\n\x17\x42\x41R_LIFECYCLE_CANCELLED\x10\x04\x62\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n#qdl/marketdata/v2/market_data.proto\x12\x11qdl.marketdata.v2\x1a\x1aqdl/common/v1/common.proto\x1a\x1cqdl/quality/v1/quality.proto\"\xb9\x03\n\x05Trade\x12&\n\x0fnative_trade_id\x18\x01 \x01(\tR\rnativeTradeId\x12\x31\n\x05price\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x05price\x12\x37\n\x08quantity\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08quantity\x12\x43\n\x0e\x61ggressor_side\x18\x04 \x01(\x0e\x32\x1c.qdl.common.v1.AggressorSideR\raggressorSide\x12$\n\x0eis_block_trade\x18\x05 \x01(\x08R\x0cisBlockTrade\x12$\n\x0eis_buyer_maker\x18\x06 \x01(\x08R\x0cisBuyerMaker\x12@\n\rquantity_unit\x18\x07 \x01(\x0e\x32\x1b.qdl.common.v1.QuantityUnitR\x0cquantityUnit\x12I\n\ridentity_kind\x18\x08 \x01(\x0e\x32$.qdl.marketdata.v2.TradeIdentityKindR\x0cidentityKind\"\xd3\x02\n\x05Quote\x12\x38\n\tbid_price\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08\x62idPrice\x12>\n\x0c\x62id_quantity\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x0b\x62idQuantity\x12\x38\n\task_price\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08\x61skPrice\x12>\n\x0c\x61sk_quantity\x18\x04 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x0b\x61skQuantity\x12\x14\n\x05level\x18\x05 \x01(\rR\x05level\x12@\n\rquantity_unit\x18\x06 \x01(\x0e\x32\x1b.qdl.common.v1.QuantityUnitR\x0cquantityUnit\"\xbc\x07\n\x03\x42\x61r\x12\x1a\n\x08interval\x18\x01 \x01(\tR\x08interval\x12 \n\x0copen_time_ns\x18\x02 \x01(\x03R\nopenTimeNs\x12\"\n\rclose_time_ns\x18\x03 \x01(\x03R\x0b\x63loseTimeNs\x12/\n\x04open\x18\x04 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x04open\x12/\n\x04high\x18\x05 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x04high\x12-\n\x03low\x18\x06 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x03low\x12\x31\n\x05\x63lose\x18\x07 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x05\x63lose\x12\x33\n\x06volume\x18\x08 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x06volume\x12\x1f\n\x0btrade_count\x18\t \x01(\x04R\ntradeCount\x12\x19\n\x08is_final\x18\n \x01(\x08R\x07isFinal\x12\x1a\n\x08revision\x18\x0b \x01(\rR\x08revision\x12\x30\n\x06origin\x18\x0c \x01(\x0e\x32\x18.qdl.common.v1.BarOriginR\x06origin\x12=\n\tlifecycle\x18\r \x01(\x0e\x32\x1f.qdl.marketdata.v2.BarLifecycleR\tlifecycle\x12\x33\n\x13supersedes_event_id\x18\x0e \x01(\x0cH\x00R\x11supersedesEventId\x88\x01\x01\x12<\n\x0bvolume_unit\x18\x0f \x01(\x0e\x32\x1b.qdl.common.v1.QuantityUnitR\nvolumeUnit\x12\x41\n\x0b\x62\x61se_volume\x18\x10 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x01R\nbaseVolume\x88\x01\x01\x12\x43\n\x0cquote_volume\x18\x11 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x02R\x0bquoteVolume\x88\x01\x01\x12I\n\x0f\x63ontract_volume\x18\x12 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x03R\x0e\x63ontractVolume\x88\x01\x01\x42\x16\n\x14_supersedes_event_idB\x0e\n\x0c_base_volumeB\x0f\n\r_quote_volumeB\x12\n\x10_contract_volume\"\x87\x02\n\tBookLevel\x12+\n\x04side\x18\x01 \x01(\x0e\x32\x17.qdl.common.v1.BookSideR\x04side\x12\x31\n\x05price\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x05price\x12\x37\n\x08quantity\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08quantity\x12\x1f\n\x0border_count\x18\x04 \x01(\rR\norderCount\x12@\n\rquantity_unit\x18\x05 \x01(\x0e\x32\x1b.qdl.common.v1.QuantityUnitR\x0cquantityUnit\"\xa4\x01\n\x11OrderBookSnapshot\x12\'\n\x0fnative_sequence\x18\x01 \x01(\tR\x0enativeSequence\x12\x1a\n\x08\x63hecksum\x18\x02 \x01(\tR\x08\x63hecksum\x12\x34\n\x06levels\x18\x03 \x03(\x0b\x32\x1c.qdl.marketdata.v2.BookLevelR\x06levels\x12\x14\n\x05\x64\x65pth\x18\x04 \x01(\rR\x05\x64\x65pth\"\x8b\x02\n\x0eOrderBookDelta\x12\x32\n\x15native_sequence_start\x18\x01 \x01(\tR\x13nativeSequenceStart\x12.\n\x13native_sequence_end\x18\x02 \x01(\tR\x11nativeSequenceEnd\x12+\n\x11snapshot_sequence\x18\x03 \x01(\tR\x10snapshotSequence\x12\x1a\n\x08\x63hecksum\x18\x04 \x01(\tR\x08\x63hecksum\x12\x36\n\x07updates\x18\x05 \x03(\x0b\x32\x1c.qdl.marketdata.v2.BookLevelR\x07updates\x12\x14\n\x05reset\x18\x06 \x01(\x08R\x05reset\"\xb5\x01\n\x0b\x46undingRate\x12/\n\x04rate\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x04rate\x12&\n\x0f\x66unding_time_ns\x18\x02 \x01(\x03R\rfundingTimeNs\x12\x34\n\x14next_funding_time_ns\x18\x03 \x01(\x03H\x00R\x11nextFundingTimeNs\x88\x01\x01\x42\x17\n\x15_next_funding_time_ns\"\xd4\x01\n\x0cOpenInterest\x12\x37\n\x08quantity\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\x08quantity\x12<\n\x08notional\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x00R\x08notional\x88\x01\x01\x12@\n\rquantity_unit\x18\x03 \x01(\x0e\x32\x1b.qdl.common.v1.QuantityUnitR\x0cquantityUnitB\x0b\n\t_notional\"\x8a\x01\n\x0eMarkIndexPrice\x12:\n\nmark_price\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\tmarkPrice\x12<\n\x0bindex_price\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\nindexPrice\"\xd8\x04\n\x06Ticker\x12:\n\nlast_price\x18\x01 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueR\tlastPrice\x12\x45\n\rlast_quantity\x18\x02 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x00R\x0clastQuantity\x88\x01\x01\x12;\n\x08open_24h\x18\x03 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x01R\x07open24h\x88\x01\x01\x12;\n\x08high_24h\x18\x04 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x02R\x07high24h\x88\x01\x01\x12\x39\n\x07low_24h\x18\x05 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x03R\x06low24h\x88\x01\x01\x12?\n\nvolume_24h\x18\x06 \x01(\x0b\x32\x1b.qdl.common.v1.DecimalValueH\x04R\tvolume24h\x88\x01\x01\x12I\n\x12last_quantity_unit\x18\x07 \x01(\x0e\x32\x1b.qdl.common.v1.QuantityUnitR\x10lastQuantityUnit\x12\x43\n\x0fvolume_24h_unit\x18\x08 \x01(\x0e\x32\x1b.qdl.common.v1.QuantityUnitR\rvolume24hUnitB\x10\n\x0e_last_quantityB\x0b\n\t_open_24hB\x0b\n\t_high_24hB\n\n\x08_low_24hB\r\n\x0b_volume_24h\"\x93\x10\n\rEventEnvelope\x12\x1f\n\x0bschema_name\x18\x01 \x01(\tR\nschemaName\x12!\n\x0cschema_major\x18\x02 \x01(\rR\x0bschemaMajor\x12!\n\x0cschema_minor\x18\x03 \x01(\rR\x0bschemaMinor\x12\x19\n\x08\x65vent_id\x18\x04 \x01(\x0cR\x07\x65ventId\x12%\n\x0einstrument_uid\x18\x05 \x01(\tR\rinstrumentUid\x12#\n\rinstrument_id\x18\x06 \x01(\tR\x0cinstrumentId\x12/\n\x13instrument_revision\x18\x07 \x01(\x04R\x12instrumentRevision\x12\x14\n\x05venue\x18\x08 \x01(\tR\x05venue\x12\x16\n\x06market\x18\t \x01(\tR\x06market\x12!\n\x0cproduct_type\x18\n \x01(\tR\x0bproductType\x12#\n\rnative_symbol\x18\x0b \x01(\tR\x0cnativeSymbol\x12\x1a\n\x08provider\x18\x0c \x01(\tR\x08provider\x12\x1b\n\tsource_id\x18\r \x01(\tR\x08sourceId\x12:\n\x0bsource_role\x18\x0e \x01(\x0e\x32\x19.qdl.common.v1.SourceRoleR\nsourceRole\x12\x1f\n\x0blease_epoch\x18\x0f \x01(\x04R\nleaseEpoch\x12/\n\x14source_event_time_ns\x18\x10 \x01(\x03R\x11sourceEventTimeNs\x12$\n\x0ereceived_at_ns\x18\x11 \x01(\x03R\x0creceivedAtNs\x12(\n\x10normalized_at_ns\x18\x12 \x01(\x03R\x0enormalizedAtNs\x12&\n\x0fpublished_at_ns\x18\x13 \x01(\x03R\rpublishedAtNs\x12\'\n\x0fsource_sequence\x18\x14 \x01(\tR\x0esourceSequence\x12-\n\x12partition_sequence\x18\x15 \x01(\x04R\x11partitionSequence\x12-\n\x12normalizer_version\x18\x16 \x01(\tR\x11normalizerVersion\x12\'\n\x0f\x61\x64\x61pter_version\x18\x17 \x01(\tR\x0e\x61\x64\x61pterVersion\x12?\n\rquality_flags\x18\x18 \x03(\x0e\x32\x1a.qdl.common.v1.QualityFlagR\x0cqualityFlags\x12(\n\x10raw_payload_hash\x18\x19 \x01(\x0cR\x0erawPayloadHash\x12%\n\x0e\x63orrelation_id\x18\x1a \x01(\tR\rcorrelationId\x12\'\n\x0f\x63onfig_revision\x18\x1b \x01(\x04R\x0e\x63onfigRevision\x12*\n\x11source_session_id\x18\x1c \x01(\tR\x0fsourceSessionId\x12\x33\n\x15\x63onnection_generation\x18\x1d \x01(\x04R\x14\x63onnectionGeneration\x12-\n\x12\x61uthority_revision\x18\x1e \x01(\x04R\x11\x61uthorityRevision\x12\x30\n\x14partition_plan_epoch\x18\x1f \x01(\x04R\x12partitionPlanEpoch\x12\x34\n\x16\x63\x61nonical_payload_hash\x18 \x01(\x0cR\x14\x63\x61nonicalPayloadHash\x12$\n\x0eraw_capture_id\x18! \x01(\x0cR\x0crawCaptureId\x12\x30\n\x05trade\x18( \x01(\x0b\x32\x18.qdl.marketdata.v2.TradeH\x00R\x05trade\x12\x30\n\x05quote\x18) \x01(\x0b\x32\x18.qdl.marketdata.v2.QuoteH\x00R\x05quote\x12*\n\x03\x62\x61r\x18* \x01(\x0b\x32\x16.qdl.marketdata.v2.BarH\x00R\x03\x62\x61r\x12K\n\rbook_snapshot\x18+ \x01(\x0b\x32$.qdl.marketdata.v2.OrderBookSnapshotH\x00R\x0c\x62ookSnapshot\x12\x42\n\nbook_delta\x18, \x01(\x0b\x32!.qdl.marketdata.v2.OrderBookDeltaH\x00R\tbookDelta\x12\x43\n\x0c\x66unding_rate\x18- \x01(\x0b\x32\x1e.qdl.marketdata.v2.FundingRateH\x00R\x0b\x66undingRate\x12\x46\n\ropen_interest\x18. \x01(\x0b\x32\x1f.qdl.marketdata.v2.OpenInterestH\x00R\x0copenInterest\x12M\n\x10mark_index_price\x18/ \x01(\x0b\x32!.qdl.marketdata.v2.MarkIndexPriceH\x00R\x0emarkIndexPrice\x12\x33\n\x06ticker\x18\x30 \x01(\x0b\x32\x19.qdl.marketdata.v2.TickerH\x00R\x06ticker\x12?\n\nfeed_state\x18\x31 \x01(\x0b\x32\x1e.qdl.quality.v1.FeedStateEventH\x00R\tfeedState\x12G\n\rquality_event\x18\x32 \x01(\x0b\x32 .qdl.quality.v1.DataQualityEventH\x00R\x0cqualityEventB\t\n\x07payload*\x85\x01\n\x11TradeIdentityKind\x12#\n\x1fTRADE_IDENTITY_KIND_UNSPECIFIED\x10\x00\x12\x1e\n\x1aTRADE_IDENTITY_KIND_NATIVE\x10\x01\x12+\n\'TRADE_IDENTITY_KIND_DERIVED_RAW_CAPTURE\x10\x02*\x9d\x01\n\x0c\x42\x61rLifecycle\x12\x1d\n\x19\x42\x41R_LIFECYCLE_UNSPECIFIED\x10\x00\x12\x1d\n\x19\x42\x41R_LIFECYCLE_IN_PROGRESS\x10\x01\x12\x17\n\x13\x42\x41R_LIFECYCLE_FINAL\x10\x02\x12\x19\n\x15\x42\x41R_LIFECYCLE_REVISED\x10\x03\x12\x1b\n\x17\x42\x41R_LIFECYCLE_CANCELLED\x10\x04\x62\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) _builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'qdl.marketdata.v2.market_data_pb2', _globals) if not _descriptor._USE_C_DESCRIPTORS: DESCRIPTOR._loaded_options = None - _globals['_BARLIFECYCLE']._serialized_start=4969 - _globals['_BARLIFECYCLE']._serialized_end=5126 + _globals['_TRADEIDENTITYKIND']._serialized_start=5778 + _globals['_TRADEIDENTITYKIND']._serialized_end=5911 + _globals['_BARLIFECYCLE']._serialized_start=5914 + _globals['_BARLIFECYCLE']._serialized_end=6071 _globals['_TRADE']._serialized_start=117 - _globals['_TRADE']._serialized_end=417 - _globals['_QUOTE']._serialized_start=420 - _globals['_QUOTE']._serialized_end=693 - _globals['_BAR']._serialized_start=696 - _globals['_BAR']._serialized_end=1326 - _globals['_BOOKLEVEL']._serialized_start=1329 - _globals['_BOOKLEVEL']._serialized_end=1526 - _globals['_ORDERBOOKSNAPSHOT']._serialized_start=1529 - _globals['_ORDERBOOKSNAPSHOT']._serialized_end=1693 - _globals['_ORDERBOOKDELTA']._serialized_start=1696 - _globals['_ORDERBOOKDELTA']._serialized_end=1963 - _globals['_FUNDINGRATE']._serialized_start=1966 - _globals['_FUNDINGRATE']._serialized_end=2147 - _globals['_OPENINTEREST']._serialized_start=2150 - _globals['_OPENINTEREST']._serialized_end=2296 - _globals['_MARKINDEXPRICE']._serialized_start=2299 - _globals['_MARKINDEXPRICE']._serialized_end=2437 - _globals['_TICKER']._serialized_start=2440 - _globals['_TICKER']._serialized_end=2896 - _globals['_EVENTENVELOPE']._serialized_start=2899 - _globals['_EVENTENVELOPE']._serialized_end=4966 + _globals['_TRADE']._serialized_end=558 + _globals['_QUOTE']._serialized_start=561 + _globals['_QUOTE']._serialized_end=900 + _globals['_BAR']._serialized_start=903 + _globals['_BAR']._serialized_end=1859 + _globals['_BOOKLEVEL']._serialized_start=1862 + _globals['_BOOKLEVEL']._serialized_end=2125 + _globals['_ORDERBOOKSNAPSHOT']._serialized_start=2128 + _globals['_ORDERBOOKSNAPSHOT']._serialized_end=2292 + _globals['_ORDERBOOKDELTA']._serialized_start=2295 + _globals['_ORDERBOOKDELTA']._serialized_end=2562 + _globals['_FUNDINGRATE']._serialized_start=2565 + _globals['_FUNDINGRATE']._serialized_end=2746 + _globals['_OPENINTEREST']._serialized_start=2749 + _globals['_OPENINTEREST']._serialized_end=2961 + _globals['_MARKINDEXPRICE']._serialized_start=2964 + _globals['_MARKINDEXPRICE']._serialized_end=3102 + _globals['_TICKER']._serialized_start=3105 + _globals['_TICKER']._serialized_end=3705 + _globals['_EVENTENVELOPE']._serialized_start=3708 + _globals['_EVENTENVELOPE']._serialized_end=5775 # @@protoc_insertion_point(module_scope) diff --git a/generated/rust/qdl/common/v1/qdl.common.v1.rs b/generated/rust/qdl/common/v1/qdl.common.v1.rs index 41524cd..1b25f4e 100644 --- a/generated/rust/qdl/common/v1/qdl.common.v1.rs +++ b/generated/rust/qdl/common/v1/qdl.common.v1.rs @@ -21,6 +21,44 @@ pub mod decimal_value { MantissaText(::prost::alloc::string::String), } } +/// QuantityUnit makes native quantity semantics explicit across spot, +/// derivatives and equity markets. Asset symbols and contract multipliers are +/// resolved from the versioned Instrument referenced by instrument_uid. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] +#[repr(i32)] +pub enum QuantityUnit { + Unspecified = 0, + BaseAsset = 1, + QuoteAsset = 2, + Contract = 3, + Share = 4, +} +impl QuantityUnit { + /// String value of the enum field names used in the ProtoBuf definition. + /// + /// The values are not transformed in any way and thus are considered stable + /// (if the ProtoBuf definition does not change) and safe for programmatic use. + pub fn as_str_name(&self) -> &'static str { + match self { + Self::Unspecified => "QUANTITY_UNIT_UNSPECIFIED", + Self::BaseAsset => "QUANTITY_UNIT_BASE_ASSET", + Self::QuoteAsset => "QUANTITY_UNIT_QUOTE_ASSET", + Self::Contract => "QUANTITY_UNIT_CONTRACT", + Self::Share => "QUANTITY_UNIT_SHARE", + } + } + /// Creates an enum from field names used in the ProtoBuf definition. + pub fn from_str_name(value: &str) -> ::core::option::Option { + match value { + "QUANTITY_UNIT_UNSPECIFIED" => Some(Self::Unspecified), + "QUANTITY_UNIT_BASE_ASSET" => Some(Self::BaseAsset), + "QUANTITY_UNIT_QUOTE_ASSET" => Some(Self::QuoteAsset), + "QUANTITY_UNIT_CONTRACT" => Some(Self::Contract), + "QUANTITY_UNIT_SHARE" => Some(Self::Share), + _ => None, + } + } +} #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] #[repr(i32)] pub enum SourceRole { diff --git a/generated/rust/qdl/marketdata/v2/qdl.marketdata.v2.rs b/generated/rust/qdl/marketdata/v2/qdl.marketdata.v2.rs index fe077e2..b3a0805 100644 --- a/generated/rust/qdl/marketdata/v2/qdl.marketdata.v2.rs +++ b/generated/rust/qdl/marketdata/v2/qdl.marketdata.v2.rs @@ -14,6 +14,10 @@ pub struct Trade { pub is_block_trade: bool, #[prost(bool, tag="6")] pub is_buyer_maker: bool, + #[prost(enumeration="super::super::common::v1::QuantityUnit", tag="7")] + pub quantity_unit: i32, + #[prost(enumeration="TradeIdentityKind", tag="8")] + pub identity_kind: i32, } #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] pub struct Quote { @@ -27,6 +31,8 @@ pub struct Quote { pub ask_quantity: ::core::option::Option, #[prost(uint32, tag="5")] pub level: u32, + #[prost(enumeration="super::super::common::v1::QuantityUnit", tag="6")] + pub quantity_unit: i32, } #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] pub struct Bar { @@ -58,6 +64,14 @@ pub struct Bar { pub lifecycle: i32, #[prost(bytes="vec", optional, tag="14")] pub supersedes_event_id: ::core::option::Option<::prost::alloc::vec::Vec>, + #[prost(enumeration="super::super::common::v1::QuantityUnit", tag="15")] + pub volume_unit: i32, + #[prost(message, optional, tag="16")] + pub base_volume: ::core::option::Option, + #[prost(message, optional, tag="17")] + pub quote_volume: ::core::option::Option, + #[prost(message, optional, tag="18")] + pub contract_volume: ::core::option::Option, } #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] pub struct BookLevel { @@ -69,6 +83,8 @@ pub struct BookLevel { pub quantity: ::core::option::Option, #[prost(uint32, tag="4")] pub order_count: u32, + #[prost(enumeration="super::super::common::v1::QuantityUnit", tag="5")] + pub quantity_unit: i32, } #[derive(Clone, PartialEq, ::prost::Message)] pub struct OrderBookSnapshot { @@ -111,6 +127,8 @@ pub struct OpenInterest { pub quantity: ::core::option::Option, #[prost(message, optional, tag="2")] pub notional: ::core::option::Option, + #[prost(enumeration="super::super::common::v1::QuantityUnit", tag="3")] + pub quantity_unit: i32, } #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] pub struct MarkIndexPrice { @@ -133,6 +151,10 @@ pub struct Ticker { pub low_24h: ::core::option::Option, #[prost(message, optional, tag="6")] pub volume_24h: ::core::option::Option, + #[prost(enumeration="super::super::common::v1::QuantityUnit", tag="7")] + pub last_quantity_unit: i32, + #[prost(enumeration="super::super::common::v1::QuantityUnit", tag="8")] + pub volume_24h_unit: i32, } #[derive(Clone, PartialEq, ::prost::Message)] pub struct EventEnvelope { @@ -235,6 +257,35 @@ pub mod event_envelope { } #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] #[repr(i32)] +pub enum TradeIdentityKind { + Unspecified = 0, + Native = 1, + DerivedRawCapture = 2, +} +impl TradeIdentityKind { + /// String value of the enum field names used in the ProtoBuf definition. + /// + /// The values are not transformed in any way and thus are considered stable + /// (if the ProtoBuf definition does not change) and safe for programmatic use. + pub fn as_str_name(&self) -> &'static str { + match self { + Self::Unspecified => "TRADE_IDENTITY_KIND_UNSPECIFIED", + Self::Native => "TRADE_IDENTITY_KIND_NATIVE", + Self::DerivedRawCapture => "TRADE_IDENTITY_KIND_DERIVED_RAW_CAPTURE", + } + } + /// Creates an enum from field names used in the ProtoBuf definition. + pub fn from_str_name(value: &str) -> ::core::option::Option { + match value { + "TRADE_IDENTITY_KIND_UNSPECIFIED" => Some(Self::Unspecified), + "TRADE_IDENTITY_KIND_NATIVE" => Some(Self::Native), + "TRADE_IDENTITY_KIND_DERIVED_RAW_CAPTURE" => Some(Self::DerivedRawCapture), + _ => None, + } + } +} +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] +#[repr(i32)] pub enum BarLifecycle { Unspecified = 0, InProgress = 1, diff --git a/migrations/postgres/0006_phase9_authority_prerequisites.sql b/migrations/postgres/0006_phase9_authority_prerequisites.sql new file mode 100644 index 0000000..bed4fdf --- /dev/null +++ b/migrations/postgres/0006_phase9_authority_prerequisites.sql @@ -0,0 +1,218 @@ +BEGIN; + +CREATE TABLE IF NOT EXISTS qdl_production_prerequisite_bundles ( + bundle_id UUID PRIMARY KEY, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + policy_revision BIGINT NOT NULL CHECK (policy_revision > 0), + decision TEXT NOT NULL CHECK (decision IN ('GO', 'NO_GO_EXTERNAL')), + evidence JSONB NOT NULL, + evidence_sha256 TEXT NOT NULL CHECK (evidence_sha256 ~ '^[0-9a-f]{64}$'), + issued_by TEXT NOT NULL CHECK (btrim(issued_by) <> ''), + issued_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (jsonb_typeof(evidence) = 'object'), + CHECK (expires_at > issued_at) +); + +CREATE INDEX IF NOT EXISTS qdl_prerequisite_bundle_candidate_idx + ON qdl_production_prerequisite_bundles (candidate_digest, expires_at DESC); + +CREATE TABLE IF NOT EXISTS qdl_authority_slices ( + slice_id TEXT PRIMARY KEY CHECK (btrim(slice_id) <> ''), + environment TEXT NOT NULL, + venue TEXT NOT NULL, + market TEXT NOT NULL, + product_type TEXT NOT NULL, + feed TEXT NOT NULL, + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + partition_id TEXT NOT NULL CHECK (btrim(partition_id) <> ''), + schema_major INTEGER NOT NULL CHECK (schema_major > 0), + state TEXT NOT NULL CHECK (state IN ( + 'PYTHON_PRIMARY', 'RUST_SHADOW', 'VALIDATING', 'RUST_CANARY', + 'RUST_PRIMARY', 'BLOCKED', 'ROLLBACK_PENDING' + )), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + terminal_watermark BIGINT, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + artifact_image_digest TEXT NOT NULL CHECK (artifact_image_digest ~ '^sha256:[0-9a-f]{64}$'), + sbom_digest TEXT NOT NULL CHECK (sbom_digest ~ '^[0-9a-f]{64}$'), + signature_identity TEXT NOT NULL CHECK (btrim(signature_identity) <> ''), + contract_digest TEXT NOT NULL CHECK (contract_digest ~ '^[0-9a-f]{64}$'), + normalizer_version TEXT NOT NULL CHECK (btrim(normalizer_version) <> ''), + adapter_version TEXT NOT NULL CHECK (btrim(adapter_version) <> ''), + config_revision TEXT NOT NULL CHECK (btrim(config_revision) <> ''), + instrument_catalog_revision TEXT NOT NULL CHECK (btrim(instrument_catalog_revision) <> ''), + source_policy_revision TEXT NOT NULL CHECK (btrim(source_policy_revision) <> ''), + partition_plan_digest TEXT NOT NULL CHECK (partition_plan_digest ~ '^[0-9a-f]{64}$'), + rollback_manifest_digest TEXT NOT NULL CHECK (rollback_manifest_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID REFERENCES qdl_production_prerequisite_bundles(bundle_id), + approved_by TEXT, + approved_at TIMESTAMPTZ, + hold_until TIMESTAMPTZ, + updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (environment, venue, market, product_type, feed, partition_plan_epoch, partition_id, schema_major), + CHECK ((approved_by IS NULL) = (approved_at IS NULL)) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_transition_audit ( + transition_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + previous_state TEXT NOT NULL, + new_state TEXT NOT NULL, + previous_revision BIGINT NOT NULL, + new_revision BIGINT NOT NULL, + previous_owner_id TEXT NOT NULL, + new_owner_id TEXT NOT NULL, + previous_lease_epoch BIGINT NOT NULL, + new_lease_epoch BIGINT NOT NULL, + partition_plan_epoch BIGINT NOT NULL, + terminal_watermark BIGINT, + prerequisite_bundle_id UUID REFERENCES qdl_production_prerequisite_bundles(bundle_id), + hold_until TIMESTAMPTZ, + actor TEXT NOT NULL CHECK (btrim(actor) <> ''), + reason TEXT NOT NULL CHECK (btrim(reason) <> ''), + occurred_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (new_revision = previous_revision + 1) +); + +CREATE INDEX IF NOT EXISTS qdl_authority_transition_slice_idx + ON qdl_authority_transition_audit (slice_id, new_revision); + +CREATE OR REPLACE FUNCTION qdl_reject_authority_audit_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'qdl_authority_transition_audit is append-only'; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_audit_immutable + ON qdl_authority_transition_audit; +CREATE TRIGGER qdl_authority_audit_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_transition_audit +FOR EACH ROW EXECUTE FUNCTION qdl_reject_authority_audit_mutation(); + +CREATE OR REPLACE FUNCTION qdl_transition_authority( + p_transition_id UUID, + p_slice_id TEXT, + p_expected_state TEXT, + p_expected_revision BIGINT, + p_expected_owner_id TEXT, + p_expected_lease_epoch BIGINT, + p_expected_partition_plan_epoch BIGINT, + p_new_state TEXT, + p_new_owner_id TEXT, + p_new_lease_epoch BIGINT, + p_terminal_watermark BIGINT, + p_prerequisite_bundle_id UUID, + p_hold_until TIMESTAMPTZ, + p_actor TEXT, + p_reason TEXT +) +RETURNS qdl_authority_slices +LANGUAGE plpgsql +AS $$ +DECLARE + current_row qdl_authority_slices%ROWTYPE; + updated_row qdl_authority_slices%ROWTYPE; + bundle_row qdl_production_prerequisite_bundles%ROWTYPE; + transition_allowed BOOLEAN := FALSE; +BEGIN + IF btrim(p_actor) = '' OR btrim(p_reason) = '' OR btrim(p_new_owner_id) = '' THEN + RAISE EXCEPTION 'actor, reason and new owner are required'; + END IF; + + SELECT * INTO current_row + FROM qdl_authority_slices AS authority_slice + WHERE authority_slice.slice_id = p_slice_id + FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority slice not found: %', p_slice_id; + END IF; + + IF current_row.state <> p_expected_state + OR current_row.authority_revision <> p_expected_revision + OR current_row.owner_id <> p_expected_owner_id + OR current_row.lease_epoch <> p_expected_lease_epoch + OR current_row.partition_plan_epoch <> p_expected_partition_plan_epoch THEN + RAISE EXCEPTION 'authority compare-and-swap precondition failed'; + END IF; + + transition_allowed := CASE current_row.state + WHEN 'PYTHON_PRIMARY' THEN p_new_state IN ('RUST_SHADOW', 'BLOCKED') + WHEN 'RUST_SHADOW' THEN p_new_state IN ('VALIDATING', 'BLOCKED') + WHEN 'VALIDATING' THEN p_new_state IN ('RUST_CANARY', 'BLOCKED', 'ROLLBACK_PENDING') + WHEN 'RUST_CANARY' THEN p_new_state IN ('RUST_PRIMARY', 'BLOCKED', 'ROLLBACK_PENDING') + WHEN 'RUST_PRIMARY' THEN p_new_state IN ('BLOCKED', 'ROLLBACK_PENDING') + WHEN 'BLOCKED' THEN p_new_state IN ('VALIDATING', 'ROLLBACK_PENDING') + WHEN 'ROLLBACK_PENDING' THEN p_new_state = 'PYTHON_PRIMARY' + ELSE FALSE + END; + IF NOT transition_allowed THEN + RAISE EXCEPTION 'invalid authority transition: % -> %', current_row.state, p_new_state; + END IF; + + IF p_new_lease_epoch < current_row.lease_epoch + OR (p_new_owner_id <> current_row.owner_id AND p_new_lease_epoch <= current_row.lease_epoch) THEN + RAISE EXCEPTION 'new owner requires a strictly newer lease epoch'; + END IF; + + IF p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN + IF p_prerequisite_bundle_id IS NULL THEN + RAISE EXCEPTION 'canary/primary requires a prerequisite bundle'; + END IF; + IF p_terminal_watermark IS NULL OR p_terminal_watermark < 0 THEN + RAISE EXCEPTION 'canary/primary requires a non-negative terminal watermark'; + END IF; + IF p_hold_until IS NULL OR p_hold_until <= clock_timestamp() THEN + RAISE EXCEPTION 'canary/primary requires a future approval hold window'; + END IF; + SELECT * INTO bundle_row + FROM qdl_production_prerequisite_bundles AS bundle + WHERE bundle.bundle_id = p_prerequisite_bundle_id; + IF NOT FOUND + OR bundle_row.decision <> 'GO' + OR bundle_row.candidate_digest <> current_row.candidate_digest + OR bundle_row.expires_at <= clock_timestamp() + OR bundle_row.expires_at < p_hold_until THEN + RAISE EXCEPTION 'production prerequisite bundle is absent, blocked, mismatched or expired'; + END IF; + ELSIF p_prerequisite_bundle_id IS NOT NULL OR p_hold_until IS NOT NULL THEN + RAISE EXCEPTION 'prerequisite bundle and hold window are valid only for canary/primary'; + END IF; + + UPDATE qdl_authority_slices AS authority_slice + SET state = p_new_state, + authority_revision = current_row.authority_revision + 1, + owner_id = p_new_owner_id, + lease_epoch = p_new_lease_epoch, + terminal_watermark = p_terminal_watermark, + prerequisite_bundle_id = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN p_prerequisite_bundle_id ELSE NULL END, + approved_by = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN p_actor ELSE NULL END, + approved_at = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN clock_timestamp() ELSE NULL END, + hold_until = CASE WHEN p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY') THEN p_hold_until ELSE NULL END, + updated_at = clock_timestamp() + WHERE authority_slice.slice_id = p_slice_id + RETURNING * INTO updated_row; + + INSERT INTO qdl_authority_transition_audit ( + transition_id, slice_id, previous_state, new_state, + previous_revision, new_revision, previous_owner_id, new_owner_id, + previous_lease_epoch, new_lease_epoch, partition_plan_epoch, + terminal_watermark, prerequisite_bundle_id, hold_until, actor, reason + ) VALUES ( + p_transition_id, p_slice_id, current_row.state, updated_row.state, + current_row.authority_revision, updated_row.authority_revision, + current_row.owner_id, updated_row.owner_id, + current_row.lease_epoch, updated_row.lease_epoch, + current_row.partition_plan_epoch, p_terminal_watermark, + p_prerequisite_bundle_id, p_hold_until, p_actor, p_reason + ); + + RETURN updated_row; +END; +$$; + +COMMIT; diff --git a/migrations/postgres/0007_phase92_terminal_handoff.sql b/migrations/postgres/0007_phase92_terminal_handoff.sql new file mode 100644 index 0000000..1e5f231 --- /dev/null +++ b/migrations/postgres/0007_phase92_terminal_handoff.sql @@ -0,0 +1,205 @@ +BEGIN; + +CREATE TABLE IF NOT EXISTS qdl_terminal_owner_checkpoints ( + checkpoint_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + source_session_id TEXT NOT NULL CHECK (btrim(source_session_id) <> ''), + connection_generation BIGINT NOT NULL CHECK (connection_generation > 0), + terminal_watermark BIGINT NOT NULL CHECK (terminal_watermark >= 0), + terminal_event_id TEXT NOT NULL CHECK (btrim(terminal_event_id) <> ''), + terminal_payload_sha256 TEXT NOT NULL CHECK (terminal_payload_sha256 ~ '^[0-9a-f]{64}$'), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + committed_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE ( + slice_id, owner_id, authority_revision, lease_epoch, + partition_plan_epoch, terminal_watermark + ) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_handoffs ( + handoff_id UUID PRIMARY KEY, + checkpoint_id UUID NOT NULL REFERENCES qdl_terminal_owner_checkpoints(checkpoint_id), + direction TEXT NOT NULL CHECK (direction IN ('PYTHON_TO_RUST', 'RUST_TO_PYTHON')), + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + old_owner_id TEXT NOT NULL CHECK (btrim(old_owner_id) <> ''), + new_owner_id TEXT NOT NULL CHECK (btrim(new_owner_id) <> ''), + expected_state TEXT NOT NULL, + new_state TEXT NOT NULL, + expected_authority_revision BIGINT NOT NULL CHECK (expected_authority_revision > 0), + new_authority_revision BIGINT NOT NULL CHECK (new_authority_revision > 0), + expected_lease_epoch BIGINT NOT NULL CHECK (expected_lease_epoch > 0), + new_lease_epoch BIGINT NOT NULL CHECK (new_lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + terminal_watermark BIGINT NOT NULL CHECK (terminal_watermark >= 0), + first_new_watermark BIGINT NOT NULL CHECK (first_new_watermark >= 1), + overlap_start_watermark BIGINT NOT NULL CHECK (overlap_start_watermark >= 0), + overlap_end_watermark BIGINT NOT NULL CHECK (overlap_end_watermark >= 0), + old_event_count BIGINT NOT NULL CHECK (old_event_count > 0), + new_event_count BIGINT NOT NULL CHECK (new_event_count > 0), + semantic_mismatches BIGINT NOT NULL CHECK (semantic_mismatches = 0), + open_gaps BIGINT NOT NULL CHECK (open_gaps = 0), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL REFERENCES qdl_production_prerequisite_bundles(bundle_id), + handoff_sha256 TEXT NOT NULL CHECK (handoff_sha256 ~ '^[0-9a-f]{64}$'), + approved_by TEXT NOT NULL CHECK (btrim(approved_by) <> ''), + approved_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (old_owner_id <> new_owner_id), + CHECK (new_authority_revision = expected_authority_revision + 1), + CHECK (new_lease_epoch > expected_lease_epoch), + CHECK (first_new_watermark = terminal_watermark + 1), + CHECK (overlap_start_watermark <= overlap_end_watermark), + CHECK (overlap_end_watermark = terminal_watermark), + CHECK (old_event_count = new_event_count), + CHECK (expires_at > approved_at), + CHECK ( + (direction = 'PYTHON_TO_RUST' AND expected_state = 'RUST_CANARY' AND new_state = 'RUST_PRIMARY') + OR + (direction = 'RUST_TO_PYTHON' AND expected_state = 'ROLLBACK_PENDING' AND new_state = 'PYTHON_PRIMARY') + ) +); + +CREATE INDEX IF NOT EXISTS qdl_authority_handoff_transition_idx + ON qdl_authority_handoffs ( + slice_id, expected_authority_revision, new_authority_revision, new_state + ); + +CREATE OR REPLACE FUNCTION qdl_reject_phase92_evidence_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'Phase 9.2 checkpoint/handoff evidence is append-only'; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_terminal_checkpoint_immutable + ON qdl_terminal_owner_checkpoints; +CREATE TRIGGER qdl_terminal_checkpoint_immutable +BEFORE UPDATE OR DELETE ON qdl_terminal_owner_checkpoints +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase92_evidence_mutation(); + +DROP TRIGGER IF EXISTS qdl_authority_handoff_immutable + ON qdl_authority_handoffs; +CREATE TRIGGER qdl_authority_handoff_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_handoffs +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase92_evidence_mutation(); + +CREATE OR REPLACE FUNCTION qdl_require_accepted_primary_handoff() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + matching_handoffs BIGINT; +BEGIN + IF (OLD.state <> 'RUST_PRIMARY' AND NEW.state = 'RUST_PRIMARY') + OR (OLD.state = 'ROLLBACK_PENDING' AND NEW.state = 'PYTHON_PRIMARY') THEN + SELECT count(*) INTO matching_handoffs + FROM qdl_authority_handoffs AS handoff + JOIN qdl_terminal_owner_checkpoints AS checkpoint + ON checkpoint.checkpoint_id = handoff.checkpoint_id + WHERE handoff.slice_id = NEW.slice_id + AND handoff.old_owner_id = OLD.owner_id + AND handoff.new_owner_id = NEW.owner_id + AND handoff.expected_state = OLD.state + AND handoff.new_state = NEW.state + AND handoff.expected_authority_revision = OLD.authority_revision + AND handoff.new_authority_revision = NEW.authority_revision + AND handoff.expected_lease_epoch = OLD.lease_epoch + AND handoff.new_lease_epoch = NEW.lease_epoch + AND handoff.partition_plan_epoch = NEW.partition_plan_epoch + AND handoff.terminal_watermark = NEW.terminal_watermark + AND handoff.candidate_digest = NEW.candidate_digest + AND handoff.prerequisite_bundle_id = COALESCE( + NEW.prerequisite_bundle_id, handoff.prerequisite_bundle_id + ) + AND handoff.expires_at > clock_timestamp() + AND checkpoint.slice_id = handoff.slice_id + AND checkpoint.owner_id = handoff.old_owner_id + AND checkpoint.authority_revision = handoff.expected_authority_revision + AND checkpoint.lease_epoch = handoff.expected_lease_epoch + AND checkpoint.partition_plan_epoch = handoff.partition_plan_epoch + AND checkpoint.terminal_watermark = handoff.terminal_watermark + AND checkpoint.candidate_digest = handoff.candidate_digest; + IF matching_handoffs <> 1 THEN + RAISE EXCEPTION 'accepted exact Phase 9.2 handoff is required'; + END IF; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_primary_handoff_guard + ON qdl_authority_slices; +CREATE TRIGGER qdl_authority_primary_handoff_guard +BEFORE UPDATE ON qdl_authority_slices +FOR EACH ROW EXECUTE FUNCTION qdl_require_accepted_primary_handoff(); + +CREATE OR REPLACE FUNCTION qdl_transition_authority_v2( + p_handoff_id UUID, + p_transition_id UUID, + p_slice_id TEXT, + p_expected_state TEXT, + p_expected_revision BIGINT, + p_expected_owner_id TEXT, + p_expected_lease_epoch BIGINT, + p_expected_partition_plan_epoch BIGINT, + p_new_state TEXT, + p_new_owner_id TEXT, + p_new_lease_epoch BIGINT, + p_terminal_watermark BIGINT, + p_prerequisite_bundle_id UUID, + p_hold_until TIMESTAMPTZ, + p_actor TEXT, + p_reason TEXT +) +RETURNS qdl_authority_slices +LANGUAGE plpgsql +AS $$ +DECLARE + handoff qdl_authority_handoffs%ROWTYPE; + transitioned qdl_authority_slices%ROWTYPE; +BEGIN + SELECT * INTO handoff + FROM qdl_authority_handoffs AS accepted + WHERE accepted.handoff_id = p_handoff_id + FOR SHARE; + IF NOT FOUND + OR handoff.slice_id <> p_slice_id + OR handoff.expected_state <> p_expected_state + OR handoff.expected_authority_revision <> p_expected_revision + OR handoff.old_owner_id <> p_expected_owner_id + OR handoff.expected_lease_epoch <> p_expected_lease_epoch + OR handoff.partition_plan_epoch <> p_expected_partition_plan_epoch + OR handoff.new_state <> p_new_state + OR handoff.new_owner_id <> p_new_owner_id + OR handoff.new_lease_epoch <> p_new_lease_epoch + OR handoff.terminal_watermark <> p_terminal_watermark + OR handoff.expires_at <= clock_timestamp() THEN + RAISE EXCEPTION 'Phase 9.2 handoff/CAS request mismatch or expired'; + END IF; + IF p_new_state = 'RUST_PRIMARY' + AND handoff.prerequisite_bundle_id <> p_prerequisite_bundle_id THEN + RAISE EXCEPTION 'Phase 9.2 prerequisite bundle mismatch'; + END IF; + + SELECT * INTO transitioned + FROM qdl_transition_authority( + p_transition_id, p_slice_id, p_expected_state, p_expected_revision, + p_expected_owner_id, p_expected_lease_epoch, + p_expected_partition_plan_epoch, p_new_state, p_new_owner_id, + p_new_lease_epoch, p_terminal_watermark, p_prerequisite_bundle_id, + p_hold_until, p_actor, p_reason + ); + RETURN transitioned; +END; +$$; + +REVOKE EXECUTE ON FUNCTION qdl_transition_authority_v2( + UUID, UUID, TEXT, TEXT, BIGINT, TEXT, BIGINT, BIGINT, TEXT, TEXT, + BIGINT, BIGINT, UUID, TIMESTAMPTZ, TEXT, TEXT +) FROM PUBLIC; + +COMMIT; diff --git a/migrations/postgres/0008_phase93_hold_close_expand.sql b/migrations/postgres/0008_phase93_hold_close_expand.sql new file mode 100644 index 0000000..21f3b2c --- /dev/null +++ b/migrations/postgres/0008_phase93_hold_close_expand.sql @@ -0,0 +1,724 @@ +BEGIN; + +CREATE TABLE IF NOT EXISTS qdl_primary_holds ( + hold_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL + REFERENCES qdl_production_prerequisite_bundles(bundle_id), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + started_at TIMESTAMPTZ NOT NULL, + required_until TIMESTAMPTZ NOT NULL, + policy_digest TEXT NOT NULL CHECK (policy_digest ~ '^[0-9a-f]{64}$'), + minimum_duration_seconds BIGINT NOT NULL CHECK (minimum_duration_seconds > 0), + max_sample_gap_seconds BIGINT NOT NULL CHECK (max_sample_gap_seconds > 0), + max_lag_ms BIGINT NOT NULL CHECK (max_lag_ms > 0), + max_freshness_ms BIGINT NOT NULL CHECK (max_freshness_ms > 0), + max_queue_depth BIGINT NOT NULL CHECK (max_queue_depth > 0), + max_spool_bytes BIGINT NOT NULL CHECK (max_spool_bytes > 0), + max_cpu_percent DOUBLE PRECISION NOT NULL + CHECK (max_cpu_percent > 0 AND max_cpu_percent <= 100), + max_rss_mb DOUBLE PRECISION NOT NULL CHECK (max_rss_mb > 0), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (required_until > started_at), + CHECK ( + extract(epoch FROM (required_until - started_at)) + >= minimum_duration_seconds + ), + UNIQUE ( + slice_id, candidate_digest, owner_id, authority_revision, + lease_epoch, partition_plan_epoch, hold_id + ) +); + +CREATE TABLE IF NOT EXISTS qdl_primary_hold_observations ( + observation_id UUID PRIMARY KEY, + hold_id UUID NOT NULL REFERENCES qdl_primary_holds(hold_id), + slice_id TEXT NOT NULL, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + sequence BIGINT NOT NULL CHECK (sequence > 0), + observed_at TIMESTAMPTZ NOT NULL, + last_watermark BIGINT NOT NULL CHECK (last_watermark >= 0), + semantic_mismatches BIGINT NOT NULL DEFAULT 0 CHECK (semantic_mismatches >= 0), + open_gaps BIGINT NOT NULL DEFAULT 0 CHECK (open_gaps >= 0), + duplicate_external_writes BIGINT NOT NULL DEFAULT 0 + CHECK (duplicate_external_writes >= 0), + accepted_stale_writer_writes BIGINT NOT NULL DEFAULT 0 + CHECK (accepted_stale_writer_writes >= 0), + authority_ambiguities BIGINT NOT NULL DEFAULT 0 + CHECK (authority_ambiguities >= 0), + durable_ack_failures BIGINT NOT NULL DEFAULT 0 + CHECK (durable_ack_failures >= 0), + projection_mismatches BIGINT NOT NULL DEFAULT 0 + CHECK (projection_mismatches >= 0), + consumer_checkpoint_regressions BIGINT NOT NULL DEFAULT 0 + CHECK (consumer_checkpoint_regressions >= 0), + unexplained_quality_failures BIGINT NOT NULL DEFAULT 0 + CHECK (unexplained_quality_failures >= 0), + lag_ms BIGINT NOT NULL DEFAULT 0 CHECK (lag_ms >= 0), + freshness_ms BIGINT NOT NULL DEFAULT 0 CHECK (freshness_ms >= 0), + queue_depth BIGINT NOT NULL DEFAULT 0 CHECK (queue_depth >= 0), + spool_bytes BIGINT NOT NULL DEFAULT 0 CHECK (spool_bytes >= 0), + cpu_percent DOUBLE PRECISION NOT NULL DEFAULT 0 CHECK (cpu_percent >= 0), + rss_mb DOUBLE PRECISION NOT NULL DEFAULT 0 CHECK (rss_mb >= 0), + registered_consumers BIGINT NOT NULL CHECK (registered_consumers > 0), + healthy_consumers BIGINT NOT NULL CHECK ( + healthy_consumers >= 0 AND healthy_consumers <= registered_consumers + ), + checkpoint_watermark BIGINT NOT NULL CHECK (checkpoint_watermark >= 0), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (hold_id, sequence) +); + +CREATE INDEX IF NOT EXISTS qdl_primary_hold_observation_time_idx + ON qdl_primary_hold_observations (hold_id, observed_at); + +CREATE OR REPLACE FUNCTION qdl_validate_primary_hold_observation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + hold_row qdl_primary_holds%ROWTYPE; + previous_row qdl_primary_hold_observations%ROWTYPE; + expected_sequence BIGINT; +BEGIN + SELECT * INTO hold_row + FROM qdl_primary_holds AS hold + WHERE hold.hold_id = NEW.hold_id + FOR SHARE; + IF NOT FOUND + OR NEW.slice_id <> hold_row.slice_id + OR NEW.candidate_digest <> hold_row.candidate_digest + OR NEW.owner_id <> hold_row.owner_id + OR NEW.authority_revision <> hold_row.authority_revision + OR NEW.lease_epoch <> hold_row.lease_epoch + OR NEW.partition_plan_epoch <> hold_row.partition_plan_epoch THEN + RAISE EXCEPTION 'Phase 9.3 hold observation identity mismatch'; + END IF; + + SELECT * INTO previous_row + FROM qdl_primary_hold_observations AS observation + WHERE observation.hold_id = NEW.hold_id + ORDER BY observation.sequence DESC + LIMIT 1 + FOR SHARE; + + expected_sequence := COALESCE(previous_row.sequence, 0) + 1; + IF NEW.sequence <> expected_sequence THEN + RAISE EXCEPTION 'Phase 9.3 hold observation sequence is not contiguous'; + END IF; + IF NEW.observed_at <= COALESCE(previous_row.observed_at, hold_row.started_at) THEN + RAISE EXCEPTION 'Phase 9.3 hold observation time is not monotonic'; + END IF; + IF extract(epoch FROM ( + NEW.observed_at - COALESCE(previous_row.observed_at, hold_row.started_at) + )) > hold_row.max_sample_gap_seconds THEN + RAISE EXCEPTION 'Phase 9.3 hold observation gap exceeds policy'; + END IF; + IF previous_row.observation_id IS NOT NULL + AND NEW.last_watermark < previous_row.last_watermark THEN + RAISE EXCEPTION 'Phase 9.3 hold watermark regressed'; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_primary_hold_observation_guard + ON qdl_primary_hold_observations; +CREATE TRIGGER qdl_primary_hold_observation_guard +BEFORE INSERT ON qdl_primary_hold_observations +FOR EACH ROW EXECUTE FUNCTION qdl_validate_primary_hold_observation(); + +CREATE TABLE IF NOT EXISTS qdl_primary_hold_decisions ( + decision_id UUID PRIMARY KEY, + hold_id UUID NOT NULL REFERENCES qdl_primary_holds(hold_id), + status TEXT NOT NULL CHECK (status IN ('IN_PROGRESS', 'PASSED', 'BLOCKED')), + reason TEXT NOT NULL CHECK (btrim(reason) <> ''), + scope TEXT NOT NULL CHECK (scope IN ('TEST_REHEARSAL', 'PRODUCTION')), + production_authorized BOOLEAN NOT NULL, + slice_id TEXT NOT NULL, + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL, + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + policy_digest TEXT NOT NULL CHECK (policy_digest ~ '^[0-9a-f]{64}$'), + first_observed_at TIMESTAMPTZ, + last_observed_at TIMESTAMPTZ, + observation_count BIGINT NOT NULL CHECK (observation_count >= 0), + terminal_watermark BIGINT CHECK (terminal_watermark >= 0), + decided_at TIMESTAMPTZ NOT NULL, + decision_sha256 TEXT NOT NULL CHECK (decision_sha256 ~ '^[0-9a-f]{64}$'), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK ( + production_authorized + = (scope = 'PRODUCTION' AND status = 'PASSED') + ), + CHECK ( + (observation_count = 0 AND first_observed_at IS NULL + AND last_observed_at IS NULL AND terminal_watermark IS NULL) + OR + (observation_count > 0 AND first_observed_at IS NOT NULL + AND last_observed_at IS NOT NULL AND terminal_watermark IS NOT NULL) + ), + UNIQUE (hold_id, decision_sha256) +); + +CREATE OR REPLACE FUNCTION qdl_validate_primary_hold_decision() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + hold_row qdl_primary_holds%ROWTYPE; + summary RECORD; +BEGIN + SELECT * INTO hold_row + FROM qdl_primary_holds AS hold + WHERE hold.hold_id = NEW.hold_id + FOR SHARE; + IF NOT FOUND + OR NEW.slice_id <> hold_row.slice_id + OR NEW.candidate_digest <> hold_row.candidate_digest + OR NEW.prerequisite_bundle_id <> hold_row.prerequisite_bundle_id + OR NEW.owner_id <> hold_row.owner_id + OR NEW.authority_revision <> hold_row.authority_revision + OR NEW.lease_epoch <> hold_row.lease_epoch + OR NEW.partition_plan_epoch <> hold_row.partition_plan_epoch + OR NEW.policy_digest <> hold_row.policy_digest THEN + RAISE EXCEPTION 'Phase 9.3 hold decision identity mismatch'; + END IF; + + SELECT + count(*) AS observation_count, + min(observed_at) AS first_observed_at, + max(observed_at) AS last_observed_at, + (array_agg(last_watermark ORDER BY sequence DESC))[1] + AS terminal_watermark, + bool_or( + semantic_mismatches <> 0 + OR open_gaps <> 0 + OR duplicate_external_writes <> 0 + OR accepted_stale_writer_writes <> 0 + OR authority_ambiguities <> 0 + OR durable_ack_failures <> 0 + OR projection_mismatches <> 0 + OR consumer_checkpoint_regressions <> 0 + OR unexplained_quality_failures <> 0 + OR lag_ms > hold_row.max_lag_ms + OR freshness_ms > hold_row.max_freshness_ms + OR queue_depth > hold_row.max_queue_depth + OR spool_bytes > hold_row.max_spool_bytes + OR cpu_percent > hold_row.max_cpu_percent + OR rss_mb > hold_row.max_rss_mb + OR healthy_consumers <> registered_consumers + OR checkpoint_watermark < last_watermark + ) AS breached + INTO summary + FROM qdl_primary_hold_observations AS observation + WHERE observation.hold_id = NEW.hold_id; + + IF NEW.observation_count <> summary.observation_count + OR NEW.first_observed_at IS DISTINCT FROM summary.first_observed_at + OR NEW.last_observed_at IS DISTINCT FROM summary.last_observed_at + OR NEW.terminal_watermark IS DISTINCT FROM summary.terminal_watermark THEN + RAISE EXCEPTION 'Phase 9.3 hold decision summary mismatch'; + END IF; + IF NEW.status = 'PASSED' AND ( + summary.observation_count = 0 + OR summary.breached + OR summary.last_observed_at < hold_row.required_until + OR NEW.decided_at < hold_row.required_until + OR NEW.reason <> 'PASS' + ) THEN + RAISE EXCEPTION 'Phase 9.3 passing hold decision is not supported by evidence'; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_primary_hold_decision_guard + ON qdl_primary_hold_decisions; +CREATE TRIGGER qdl_primary_hold_decision_guard +BEFORE INSERT ON qdl_primary_hold_decisions +FOR EACH ROW EXECUTE FUNCTION qdl_validate_primary_hold_decision(); + +CREATE TABLE IF NOT EXISTS qdl_consumer_registry_snapshots ( + snapshot_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + checkpoint_count BIGINT NOT NULL CHECK (checkpoint_count > 0), + ready_checkpoint_count BIGINT NOT NULL CHECK ( + ready_checkpoint_count = checkpoint_count + ), + minimum_checkpoint_watermark BIGINT NOT NULL + CHECK (minimum_checkpoint_watermark >= 0), + checkpoint_regressions BIGINT NOT NULL + CHECK (checkpoint_regressions = 0), + unresolved_migrations BIGINT NOT NULL CHECK (unresolved_migrations = 0), + rollback_ready BOOLEAN NOT NULL CHECK (rollback_ready), + registry_sha256 TEXT NOT NULL CHECK (registry_sha256 ~ '^[0-9a-f]{64}$'), + details JSONB NOT NULL CHECK (jsonb_typeof(details) = 'object'), + observed_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (slice_id, authority_revision, registry_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_registry_snapshots ( + snapshot_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + state TEXT NOT NULL CHECK (state = 'RUST_PRIMARY'), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL + REFERENCES qdl_production_prerequisite_bundles(bundle_id), + current_watermark BIGINT NOT NULL CHECK (current_watermark >= 0), + public_write_allowed BOOLEAN NOT NULL CHECK (public_write_allowed), + legacy_write_allowed BOOLEAN NOT NULL CHECK (legacy_write_allowed), + registry_sha256 TEXT NOT NULL CHECK (registry_sha256 ~ '^[0-9a-f]{64}$'), + observed_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (slice_id, authority_revision, registry_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_rollback_rehearsals ( + rehearsal_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + rollback_manifest_digest TEXT NOT NULL + CHECK (rollback_manifest_digest ~ '^[0-9a-f]{64}$'), + reconciled_through_watermark BIGINT NOT NULL + CHECK (reconciled_through_watermark >= 0), + rto_ms DOUBLE PRECISION NOT NULL CHECK (rto_ms > 0), + status TEXT NOT NULL CHECK (status = 'PASS'), + production_scope BOOLEAN NOT NULL CHECK (production_scope), + rehearsal_sha256 TEXT NOT NULL CHECK (rehearsal_sha256 ~ '^[0-9a-f]{64}$'), + observed_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (expires_at > observed_at), + UNIQUE (slice_id, authority_revision, rehearsal_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_closure_approvals ( + approval_id UUID PRIMARY KEY, + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL + REFERENCES qdl_production_prerequisite_bundles(bundle_id), + hold_id UUID NOT NULL REFERENCES qdl_primary_holds(hold_id), + hold_policy_digest TEXT NOT NULL + CHECK (hold_policy_digest ~ '^[0-9a-f]{64}$'), + decision TEXT NOT NULL CHECK (decision = 'APPROVE'), + allow_close_rollback_window BOOLEAN NOT NULL + CHECK (allow_close_rollback_window), + repository_cleanup_approved BOOLEAN NOT NULL + CHECK (NOT repository_cleanup_approved), + operator TEXT NOT NULL CHECK (btrim(operator) <> ''), + change_ticket TEXT NOT NULL CHECK (btrim(change_ticket) <> ''), + approval_sha256 TEXT NOT NULL CHECK (approval_sha256 ~ '^[0-9a-f]{64}$'), + approved_at TIMESTAMPTZ NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (expires_at > approved_at), + UNIQUE (slice_id, hold_id, approval_sha256) +); + +CREATE TABLE IF NOT EXISTS qdl_authority_closures ( + closure_id UUID PRIMARY KEY, + closure_sha256 TEXT NOT NULL CHECK (closure_sha256 ~ '^[0-9a-f]{64}$'), + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + prerequisite_bundle_id UUID NOT NULL, + owner_id TEXT NOT NULL CHECK (btrim(owner_id) <> ''), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + lease_epoch BIGINT NOT NULL CHECK (lease_epoch > 0), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + hold_decision_id UUID NOT NULL + REFERENCES qdl_primary_hold_decisions(decision_id), + hold_decision_digest TEXT NOT NULL CHECK (hold_decision_digest ~ '^[0-9a-f]{64}$'), + consumer_registry_snapshot_id UUID NOT NULL + REFERENCES qdl_consumer_registry_snapshots(snapshot_id), + consumer_registry_digest TEXT NOT NULL CHECK (consumer_registry_digest ~ '^[0-9a-f]{64}$'), + authority_registry_snapshot_id UUID NOT NULL + REFERENCES qdl_authority_registry_snapshots(snapshot_id), + authority_registry_digest TEXT NOT NULL CHECK (authority_registry_digest ~ '^[0-9a-f]{64}$'), + rollback_rehearsal_id UUID NOT NULL + REFERENCES qdl_rollback_rehearsals(rehearsal_id), + rollback_rehearsal_digest TEXT NOT NULL CHECK (rollback_rehearsal_digest ~ '^[0-9a-f]{64}$'), + approval_id UUID NOT NULL REFERENCES qdl_closure_approvals(approval_id), + approval_digest TEXT NOT NULL CHECK (approval_digest ~ '^[0-9a-f]{64}$'), + approved_by TEXT NOT NULL CHECK (btrim(approved_by) <> ''), + change_ticket TEXT NOT NULL CHECK (btrim(change_ticket) <> ''), + approved_at TIMESTAMPTZ NOT NULL, + approval_expires_at TIMESTAMPTZ NOT NULL, + closed_at TIMESTAMPTZ NOT NULL, + production_authorized BOOLEAN NOT NULL CHECK (production_authorized), + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (approval_expires_at > approved_at), + CHECK (closed_at >= approved_at AND closed_at < approval_expires_at), + UNIQUE (slice_id, authority_revision) +); + +CREATE OR REPLACE FUNCTION qdl_close_authority_window( + p_closure_id UUID, + p_closure_sha256 TEXT, + p_hold_decision_id UUID, + p_consumer_registry_snapshot_id UUID, + p_authority_registry_snapshot_id UUID, + p_rollback_rehearsal_id UUID, + p_approval_id UUID, + p_closed_at TIMESTAMPTZ +) +RETURNS qdl_authority_closures +LANGUAGE plpgsql +AS $$ +DECLARE + current_row qdl_authority_slices%ROWTYPE; + decision_row qdl_primary_hold_decisions%ROWTYPE; + hold_row qdl_primary_holds%ROWTYPE; + bundle_row qdl_production_prerequisite_bundles%ROWTYPE; + consumer_row qdl_consumer_registry_snapshots%ROWTYPE; + authority_row qdl_authority_registry_snapshots%ROWTYPE; + rollback_row qdl_rollback_rehearsals%ROWTYPE; + approval_row qdl_closure_approvals%ROWTYPE; + closure_row qdl_authority_closures%ROWTYPE; +BEGIN + SELECT * INTO decision_row + FROM qdl_primary_hold_decisions AS decision + WHERE decision.decision_id = p_hold_decision_id + FOR SHARE; + SELECT * INTO hold_row + FROM qdl_primary_holds AS hold + WHERE hold.hold_id = decision_row.hold_id + FOR SHARE; + SELECT * INTO current_row + FROM qdl_authority_slices AS authority_slice + WHERE authority_slice.slice_id = decision_row.slice_id + FOR UPDATE; + SELECT * INTO bundle_row + FROM qdl_production_prerequisite_bundles AS bundle + WHERE bundle.bundle_id = decision_row.prerequisite_bundle_id + FOR SHARE; + SELECT * INTO consumer_row + FROM qdl_consumer_registry_snapshots AS snapshot + WHERE snapshot.snapshot_id = p_consumer_registry_snapshot_id + FOR SHARE; + SELECT * INTO authority_row + FROM qdl_authority_registry_snapshots AS snapshot + WHERE snapshot.snapshot_id = p_authority_registry_snapshot_id + FOR SHARE; + SELECT * INTO rollback_row + FROM qdl_rollback_rehearsals AS rehearsal + WHERE rehearsal.rehearsal_id = p_rollback_rehearsal_id + FOR SHARE; + SELECT * INTO approval_row + FROM qdl_closure_approvals AS approval + WHERE approval.approval_id = p_approval_id + FOR SHARE; + + IF decision_row.decision_id IS NULL + OR hold_row.hold_id IS NULL + OR decision_row.status <> 'PASSED' + OR decision_row.scope <> 'PRODUCTION' + OR NOT decision_row.production_authorized + OR decision_row.reason <> 'PASS' + OR decision_row.hold_id <> hold_row.hold_id + OR decision_row.slice_id <> hold_row.slice_id + OR decision_row.candidate_digest <> hold_row.candidate_digest + OR decision_row.prerequisite_bundle_id <> hold_row.prerequisite_bundle_id + OR decision_row.owner_id <> hold_row.owner_id + OR decision_row.authority_revision <> hold_row.authority_revision + OR decision_row.lease_epoch <> hold_row.lease_epoch + OR decision_row.partition_plan_epoch <> hold_row.partition_plan_epoch + OR decision_row.policy_digest <> hold_row.policy_digest THEN + RAISE EXCEPTION 'Phase 9.3 passing production hold decision is required'; + END IF; + IF current_row.slice_id IS NULL + OR current_row.state <> 'RUST_PRIMARY' + OR current_row.candidate_digest <> decision_row.candidate_digest + OR current_row.prerequisite_bundle_id <> decision_row.prerequisite_bundle_id + OR current_row.owner_id <> decision_row.owner_id + OR current_row.authority_revision <> decision_row.authority_revision + OR current_row.lease_epoch <> decision_row.lease_epoch + OR current_row.partition_plan_epoch <> decision_row.partition_plan_epoch THEN + RAISE EXCEPTION 'Phase 9.3 authority closure CAS mismatch'; + END IF; + IF bundle_row.bundle_id IS NULL + OR bundle_row.decision <> 'GO' + OR bundle_row.candidate_digest <> current_row.candidate_digest + OR bundle_row.expires_at <= p_closed_at THEN + RAISE EXCEPTION 'Phase 9.3 closure prerequisite bundle is invalid'; + END IF; + IF consumer_row.snapshot_id IS NULL + OR consumer_row.slice_id <> current_row.slice_id + OR consumer_row.authority_revision <> current_row.authority_revision + OR consumer_row.minimum_checkpoint_watermark < authority_row.current_watermark + OR consumer_row.observed_at > p_closed_at + OR p_closed_at - consumer_row.observed_at > interval '5 minutes' THEN + RAISE EXCEPTION 'Phase 9.3 consumer registry snapshot is invalid'; + END IF; + IF authority_row.snapshot_id IS NULL + OR authority_row.slice_id <> current_row.slice_id + OR authority_row.state <> current_row.state + OR authority_row.owner_id <> current_row.owner_id + OR authority_row.authority_revision <> current_row.authority_revision + OR authority_row.lease_epoch <> current_row.lease_epoch + OR authority_row.partition_plan_epoch <> current_row.partition_plan_epoch + OR authority_row.candidate_digest <> current_row.candidate_digest + OR authority_row.prerequisite_bundle_id <> current_row.prerequisite_bundle_id + OR authority_row.observed_at > p_closed_at + OR p_closed_at - authority_row.observed_at > interval '5 minutes' THEN + RAISE EXCEPTION 'Phase 9.3 authority registry snapshot is invalid'; + END IF; + IF rollback_row.rehearsal_id IS NULL + OR rollback_row.slice_id <> current_row.slice_id + OR rollback_row.candidate_digest <> current_row.candidate_digest + OR rollback_row.owner_id <> current_row.owner_id + OR rollback_row.authority_revision <> current_row.authority_revision + OR rollback_row.lease_epoch <> current_row.lease_epoch + OR rollback_row.partition_plan_epoch <> current_row.partition_plan_epoch + OR rollback_row.rollback_manifest_digest <> current_row.rollback_manifest_digest + OR rollback_row.reconciled_through_watermark < authority_row.current_watermark + OR rollback_row.expires_at <= p_closed_at THEN + RAISE EXCEPTION 'Phase 9.3 rollback rehearsal is invalid'; + END IF; + IF approval_row.approval_id IS NULL + OR approval_row.slice_id <> current_row.slice_id + OR approval_row.candidate_digest <> current_row.candidate_digest + OR approval_row.prerequisite_bundle_id <> current_row.prerequisite_bundle_id + OR approval_row.hold_id <> hold_row.hold_id + OR approval_row.hold_policy_digest <> hold_row.policy_digest + OR approval_row.approved_at > p_closed_at + OR approval_row.expires_at <= p_closed_at THEN + RAISE EXCEPTION 'Phase 9.3 closure approval is invalid or expired'; + END IF; + + IF p_closure_sha256 !~ '^[0-9a-f]{64}$' THEN + RAISE EXCEPTION 'Phase 9.3 closure digest is invalid'; + END IF; + + INSERT INTO qdl_authority_closures ( + closure_id, closure_sha256, slice_id, candidate_digest, prerequisite_bundle_id, + owner_id, authority_revision, lease_epoch, partition_plan_epoch, + hold_decision_id, hold_decision_digest, + consumer_registry_snapshot_id, consumer_registry_digest, + authority_registry_snapshot_id, authority_registry_digest, + rollback_rehearsal_id, rollback_rehearsal_digest, + approval_id, approval_digest, approved_by, change_ticket, + approved_at, approval_expires_at, closed_at, production_authorized + ) VALUES ( + p_closure_id, p_closure_sha256, current_row.slice_id, + current_row.candidate_digest, + current_row.prerequisite_bundle_id, current_row.owner_id, + current_row.authority_revision, current_row.lease_epoch, + current_row.partition_plan_epoch, decision_row.decision_id, + decision_row.decision_sha256, consumer_row.snapshot_id, + consumer_row.registry_sha256, authority_row.snapshot_id, + authority_row.registry_sha256, rollback_row.rehearsal_id, + rollback_row.rehearsal_sha256, approval_row.approval_id, + approval_row.approval_sha256, approval_row.operator, + approval_row.change_ticket, approval_row.approved_at, + approval_row.expires_at, p_closed_at, TRUE + ) + RETURNING * INTO closure_row; + RETURN closure_row; +END; +$$; + +CREATE TABLE IF NOT EXISTS qdl_expansion_candidates ( + expansion_id UUID PRIMARY KEY, + parent_closure_id UUID NOT NULL + REFERENCES qdl_authority_closures(closure_id), + parent_slice_id TEXT NOT NULL, + parent_candidate_digest TEXT NOT NULL CHECK (parent_candidate_digest ~ '^[0-9a-f]{64}$'), + parent_closure_digest TEXT NOT NULL CHECK (parent_closure_digest ~ '^[0-9a-f]{64}$'), + expansion_type TEXT NOT NULL CHECK (expansion_type IN ( + 'INSTRUMENT_PARTITION', 'BBO', 'L2_BOOK', 'BAR_LIFECYCLE', + 'VENUE_MARKET' + )), + candidate_digest TEXT NOT NULL CHECK (candidate_digest ~ '^[0-9a-f]{64}$'), + scope_digest TEXT NOT NULL CHECK (scope_digest ~ '^[0-9a-f]{64}$'), + partition_plan_epoch BIGINT NOT NULL CHECK (partition_plan_epoch > 0), + required_gates TEXT[] NOT NULL, + status TEXT NOT NULL CHECK (status = 'INDEPENDENT_CERTIFICATION_REQUIRED'), + transitive_evidence_allowed BOOLEAN NOT NULL + CHECK (NOT transitive_evidence_allowed), + public_write_allowed BOOLEAN NOT NULL CHECK (NOT public_write_allowed), + legacy_write_allowed BOOLEAN NOT NULL CHECK (NOT legacy_write_allowed), + created_at TIMESTAMPTZ NOT NULL, + recorded_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK (candidate_digest <> parent_candidate_digest), + CHECK (cardinality(required_gates) > 0), + UNIQUE (parent_closure_id, expansion_type, candidate_digest, scope_digest) +); + +CREATE OR REPLACE FUNCTION qdl_validate_expansion_candidate() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + closure_row qdl_authority_closures%ROWTYPE; + expected_gates TEXT[]; +BEGIN + SELECT * INTO closure_row + FROM qdl_authority_closures AS closure + WHERE closure.closure_id = NEW.parent_closure_id + FOR SHARE; + IF NOT FOUND + OR NEW.parent_slice_id <> closure_row.slice_id + OR NEW.parent_candidate_digest <> closure_row.candidate_digest + OR NEW.parent_closure_digest <> closure_row.closure_sha256 + OR NOT closure_row.production_authorized THEN + RAISE EXCEPTION 'Phase 9.3 expansion parent closure mismatch'; + END IF; + expected_gates := CASE NEW.expansion_type + WHEN 'INSTRUMENT_PARTITION' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'exact_frame_parity', + 'partition_churn', 'provider_authentic_source', 'rollback', + 'source_capacity' + ] + WHEN 'BBO' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'coalescing_policy', + 'exact_frame_parity', 'freshness', 'ordering_reconnect', + 'provider_authentic_source', 'quote_identity', 'rollback' + ] + WHEN 'L2_BOOK' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'checksum', + 'exact_frame_parity', 'lossless_backpressure', + 'provider_authentic_source', 'resync', 'rollback', + 'snapshot_delta_sequence' + ] + WHEN 'BAR_LIFECYCLE' THEN ARRAY[ + 'authority_handoff', 'capacity_headroom', 'close_time_semantics', + 'exact_frame_parity', 'final_revision_lineage', + 'provider_authentic_source', 'replay', 'rollback' + ] + WHEN 'VENUE_MARKET' THEN ARRAY[ + 'adapter_capability', 'authority_handoff', 'capacity_headroom', + 'disaster_recovery', 'entitlement', 'exact_frame_parity', + 'instrument_identity', 'provider_authentic_source', + 'provider_semantics', 'rollback' + ] + END; + IF NEW.required_gates <> expected_gates THEN + RAISE EXCEPTION 'Phase 9.3 expansion requires independent capability gates'; + END IF; + IF NEW.expansion_type = 'INSTRUMENT_PARTITION' + AND NEW.partition_plan_epoch <= closure_row.partition_plan_epoch THEN + RAISE EXCEPTION 'Phase 9.3 instrument expansion requires a new partition epoch'; + END IF; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_expansion_candidate_guard + ON qdl_expansion_candidates; +CREATE TRIGGER qdl_expansion_candidate_guard +BEFORE INSERT ON qdl_expansion_candidates +FOR EACH ROW EXECUTE FUNCTION qdl_validate_expansion_candidate(); + +CREATE TABLE IF NOT EXISTS qdl_runtime_decommission_decisions ( + decision_id UUID PRIMARY KEY, + runtime_id TEXT NOT NULL CHECK (btrim(runtime_id) <> ''), + owned_slice_count BIGINT NOT NULL CHECK (owned_slice_count >= 0), + rollback_reference_count BIGINT NOT NULL CHECK (rollback_reference_count >= 0), + consumer_dependency_count BIGINT NOT NULL CHECK (consumer_dependency_count >= 0), + all_replacement_windows_closed BOOLEAN NOT NULL, + repository_cleanup_approved BOOLEAN NOT NULL, + shared_knowledge_retained BOOLEAN NOT NULL, + allowed BOOLEAN NOT NULL, + reason TEXT NOT NULL CHECK (btrim(reason) <> ''), + decided_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + CHECK ( + NOT allowed OR ( + owned_slice_count = 0 + AND rollback_reference_count = 0 + AND consumer_dependency_count = 0 + AND all_replacement_windows_closed + AND repository_cleanup_approved + AND shared_knowledge_retained + ) + ) +); + +CREATE OR REPLACE FUNCTION qdl_reject_phase93_registry_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'Phase 9.3 hold/closure/expansion evidence is append-only'; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_primary_hold_immutable ON qdl_primary_holds; +CREATE TRIGGER qdl_primary_hold_immutable +BEFORE UPDATE OR DELETE ON qdl_primary_holds +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_primary_hold_observation_immutable + ON qdl_primary_hold_observations; +CREATE TRIGGER qdl_primary_hold_observation_immutable +BEFORE UPDATE OR DELETE ON qdl_primary_hold_observations +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_primary_hold_decision_immutable + ON qdl_primary_hold_decisions; +CREATE TRIGGER qdl_primary_hold_decision_immutable +BEFORE UPDATE OR DELETE ON qdl_primary_hold_decisions +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_consumer_registry_snapshot_immutable + ON qdl_consumer_registry_snapshots; +CREATE TRIGGER qdl_consumer_registry_snapshot_immutable +BEFORE UPDATE OR DELETE ON qdl_consumer_registry_snapshots +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_authority_registry_snapshot_immutable + ON qdl_authority_registry_snapshots; +CREATE TRIGGER qdl_authority_registry_snapshot_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_registry_snapshots +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_rollback_rehearsal_immutable + ON qdl_rollback_rehearsals; +CREATE TRIGGER qdl_rollback_rehearsal_immutable +BEFORE UPDATE OR DELETE ON qdl_rollback_rehearsals +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_closure_approval_immutable + ON qdl_closure_approvals; +CREATE TRIGGER qdl_closure_approval_immutable +BEFORE UPDATE OR DELETE ON qdl_closure_approvals +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_authority_closure_immutable + ON qdl_authority_closures; +CREATE TRIGGER qdl_authority_closure_immutable +BEFORE UPDATE OR DELETE ON qdl_authority_closures +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_expansion_candidate_immutable + ON qdl_expansion_candidates; +CREATE TRIGGER qdl_expansion_candidate_immutable +BEFORE UPDATE OR DELETE ON qdl_expansion_candidates +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +DROP TRIGGER IF EXISTS qdl_runtime_decommission_decision_immutable + ON qdl_runtime_decommission_decisions; +CREATE TRIGGER qdl_runtime_decommission_decision_immutable +BEFORE UPDATE OR DELETE ON qdl_runtime_decommission_decisions +FOR EACH ROW EXECUTE FUNCTION qdl_reject_phase93_registry_mutation(); + +REVOKE EXECUTE ON FUNCTION qdl_close_authority_window( + UUID, TEXT, UUID, UUID, UUID, UUID, UUID, TIMESTAMPTZ +) FROM PUBLIC; + +COMMIT; diff --git a/migrations/postgres/README.md b/migrations/postgres/README.md index df1e39b..ef6c911 100644 --- a/migrations/postgres/README.md +++ b/migrations/postgres/README.md @@ -17,3 +17,21 @@ Phase 7 adds `0005_phase7_data_plane_identity.sql`. It binds an authenticated workload subject and environment to one immutable consumer-manifest revision, including allowed purposes, data-plane permissions, execution-dependency policy and bounded quotas. It does not store market events or alter V1 tables. + +Phase 9.0-C adds `0006_phase9_authority_prerequisites.sql`. It stores immutable +prerequisite bundles, one persistent authority record per exact slice and an +append-only transition audit. Its CAS transition function rejects stale +state/revision/owner/lease/partition expectations, binds release provenance, +requires a terminal watermark and approval hold window, and cannot enter Rust +canary or primary without a non-expired `GO` bundle bound to the candidate digest. +The migration is dark: it neither seeds an approval nor changes V1 authority. + + +Phase 9.3 adds `0008_phase93_hold_close_expand.sql`. It stores immutable +primary-hold observations/decisions, frozen consumer and authority registry +snapshots, rollback rehearsal and operator approval evidence, one governed +rollback-window closure record and independently uncertified expansion +manifests. The closure function locks and rechecks the exact `RUST_PRIMARY` +authority row but never updates authority ownership, revision, lease or +watermark. Expansion rows are write-disabled and cannot inherit parent +certification. diff --git a/poetry.lock b/poetry.lock index d032b17..734fce4 100644 --- a/poetry.lock +++ b/poetry.lock @@ -356,6 +356,71 @@ files = [ {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, ] +[[package]] +name = "confluent-kafka" +version = "2.15.0" +description = "Confluent's Python client for Apache Kafka" +optional = false +python-versions = ">=3.8" +groups = ["main"] +files = [ + {file = "confluent_kafka-2.15.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:f8ac4addda1cff47adc316146ee74aa2a83eeef14b176252f6f1abd29ae1d5a2"}, + {file = "confluent_kafka-2.15.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:64c5a9dbe563c964e21a83f3cb4ff1de6536e484a56289fe9aff69fc33302c73"}, + {file = "confluent_kafka-2.15.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:f4a8f02dbc93f17e125a201c5e5ce2143794b71bb5c91836ef87959fc5dcf6a4"}, + {file = "confluent_kafka-2.15.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:d9c9d240b681427568d9e341cf3deb3b7350b69931e6fbc636c5986432fb03f4"}, + {file = "confluent_kafka-2.15.0-cp310-cp310-win_amd64.whl", hash = "sha256:039444ea430aaef415487df406070f364d956d76bbd980337455c2a7b5c5789b"}, + {file = "confluent_kafka-2.15.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3c4493caf1f1d2e295f5be8400d3a9854016456241f7062d0fb22dd01e2632e6"}, + {file = "confluent_kafka-2.15.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c311695a10cefc1579372bd2f3808d28fd0c3d8cc7f578c94a381f351018912e"}, + {file = "confluent_kafka-2.15.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:7868fe2c2e1f6434903e3f1ea0ba6976d9382f5a9ef46f85d205ca7eb6f28689"}, + {file = "confluent_kafka-2.15.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:e31bbb5775da23f14264b3e6efd3839c1611a639f3d8c15516350798fe6f2750"}, + {file = "confluent_kafka-2.15.0-cp311-cp311-win_amd64.whl", hash = "sha256:c041b784f25d2b0b8b2abab7390020951229b694a5c037402c493b0eeca12020"}, + {file = "confluent_kafka-2.15.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:6e5ea2933c0b07782a99dfdfc9d52e92526e7a61839a6994feb994dcf2a78a74"}, + {file = "confluent_kafka-2.15.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e4465a060ab215d5d514b181f3e7502c6fd7887529381c4af940a0b60f9d65cc"}, + {file = "confluent_kafka-2.15.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:b68e6f441866542c9150e6fdf106131eee5a51db591937888c866d799fb354ec"}, + {file = "confluent_kafka-2.15.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:fd833dab1392f456b9dd52b7ed9ac65b2cc36871ed5e3ebe8d53486c965b6453"}, + {file = "confluent_kafka-2.15.0-cp312-cp312-win_amd64.whl", hash = "sha256:2cc8d01a77c534669bb896f731c8917cd63b787618081fdd0d1420d58cd6815b"}, + {file = "confluent_kafka-2.15.0-cp313-cp313-macosx_13_0_arm64.whl", hash = "sha256:ff43508f8e929d83545272ef4f17e27fdaa9cc8397f77cf2236b22d67de4edc4"}, + {file = "confluent_kafka-2.15.0-cp313-cp313-macosx_13_0_x86_64.whl", hash = "sha256:41d4c7360c51202785e8e9aa56241dd392b9dbcc311eae7c5e6985b239efc12b"}, + {file = "confluent_kafka-2.15.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:b0ac3831aac47ea23699a6ca5b9836288a4b9d201015dbfc4c3ec9f20592fdb5"}, + {file = "confluent_kafka-2.15.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:0455754e8294e1e76cb5acc083c300b6bd8d88f2d2c551c583cbf4ab55889a57"}, + {file = "confluent_kafka-2.15.0-cp313-cp313-win_amd64.whl", hash = "sha256:7c59ea36606e0e946e6b353203136b4d71be6d95c0c8ae5113b81850ff5b7b41"}, + {file = "confluent_kafka-2.15.0-cp314-cp314-macosx_13_0_arm64.whl", hash = "sha256:e781d39ff31d5d10d0502471f92f95aff4f5be1eaae9a1f57aeac164bc9f9029"}, + {file = "confluent_kafka-2.15.0-cp314-cp314-macosx_13_0_x86_64.whl", hash = "sha256:000463c822a7adc3293369b63688b68d17c03a1a4a64f86182efc08f8b150676"}, + {file = "confluent_kafka-2.15.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:9ddf4cf4647e5d633ef64e3f5a349d5288edfedf974203993e9d86548c2695be"}, + {file = "confluent_kafka-2.15.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:d8ed33f623ff2a104fb76f99a67e9917f0170fddb4e28380fcdc83347b1646b2"}, + {file = "confluent_kafka-2.15.0-cp314-cp314-win_amd64.whl", hash = "sha256:2e80bd96f61aae2ffba951754a769e6d3c5ebb5a5e778ed0ae8ad899ea91556b"}, + {file = "confluent_kafka-2.15.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:5d80667060bac18eaa0a4821a7f162e96e0515d0b4b42e9d3725abc5782780d7"}, + {file = "confluent_kafka-2.15.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:59e88b759f256751fca7d5702fd6f892d9f5ee7710ecc37a4e1b38f6336b0383"}, + {file = "confluent_kafka-2.15.0-cp38-cp38-manylinux_2_28_aarch64.whl", hash = "sha256:c620043e21a7f7c0f21e034f9785eb4b827ea1f6236740b7a7d8e829c14e58c4"}, + {file = "confluent_kafka-2.15.0-cp38-cp38-manylinux_2_28_x86_64.whl", hash = "sha256:143ebab2df504775edd220406ab0e9476357d71fb679a049dd3b33badbc0872e"}, + {file = "confluent_kafka-2.15.0-cp38-cp38-win_amd64.whl", hash = "sha256:3dd735d1a148e1f566097910d23293ee0898bf4853ad472d10c65751239c7e91"}, + {file = "confluent_kafka-2.15.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:aadfa1f49fd770fe665662a8ae9670ae8c62ff2c360fed5aff0ddb54b4d39638"}, + {file = "confluent_kafka-2.15.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:ff6d0041e7d5318af548d90d67bb3a72410a77fe616a05564c3fa1ea13f23453"}, + {file = "confluent_kafka-2.15.0-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:537e6322f91e5077a6aa275ab1a9ceb4eaf0f85e1a2a711aa71e81a56dd95de7"}, + {file = "confluent_kafka-2.15.0-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:9edccce72df3c13d1dbdba0e1f95b9332eab08d4715b1bc9cfa5d45a92f9d11c"}, + {file = "confluent_kafka-2.15.0-cp39-cp39-win_amd64.whl", hash = "sha256:31b8cc57794c0f048a6875280397bf4970744afe743d2d2208247a2f63ec8c7f"}, + {file = "confluent_kafka-2.15.0.tar.gz", hash = "sha256:7ad9bad1cbabf6713ec039b8204b48d322024fd11397eec88d912e048c732ba7"}, +] + +[package.dependencies] +typing-extensions = {version = "*", markers = "python_version < \"3.11\""} + +[package.extras] +all = ["async-timeout", "attrs", "attrs", "attrs (>=21.2.0)", "authlib (>=1.0.0)", "authlib (>=1.0.0)", "avro (>=1.11.1,<2)", "avro (>=1.11.1,<2)", "azure-identity", "azure-identity", "azure-keyvault-keys", "azure-keyvault-keys", "black (>=24.0.0)", "boto3", "boto3 (>=1.35)", "boto3 (>=1.42.25)", "cachetools", "cachetools (>=5.5.0)", "cel-python (>=0.4.0)", "cel-python (>=0.4.0)", "certifi", "confluent-kafka", "fastapi", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "flake8", "google-api-core", "google-api-core", "google-auth", "google-auth", "google-cloud-kms", "google-cloud-kms", "google-re2 (<1.1.20251105)", "googleapis-common-protos", "googleapis-common-protos", "hkdf (==0.0.3)", "hkdf (==0.0.3)", "httpx (>=0.26)", "httpx (>=0.26)", "hvac", "hvac", "isort (>=5.13.0)", "jsonata-python", "jsonata-python", "jsonschema (>=4.18.0)", "jsonschema (>=4.18.0)", "mypy", "opentelemetry-distro", "opentelemetry-exporter-otlp", "pandoc", "pluggy (<1.6.0)", "protobuf", "protobuf", "psutil", "pydantic", "pytest", "pytest-asyncio", "pytest-timeout", "pytest_cov", "pyyaml (>=6.0.0)", "pyyaml (>=6.0.0)", "requests", "requests", "requests-mock", "respx", "six", "sphinx", "sphinx-rtd-theme", "tink", "tink", "tomli ; python_version < \"3.11\"", "types-cachetools", "types-requests", "urllib3 (<3)", "uvicorn"] +avro = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "avro (>=1.11.1,<2)", "cachetools (>=5.5.0)", "certifi", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "httpx (>=0.26)", "requests"] +dev = ["async-timeout", "attrs", "attrs", "attrs (>=21.2.0)", "authlib (>=1.0.0)", "authlib (>=1.0.0)", "avro (>=1.11.1,<2)", "avro (>=1.11.1,<2)", "azure-identity", "azure-identity", "azure-keyvault-keys", "azure-keyvault-keys", "black (>=24.0.0)", "boto3", "boto3 (>=1.35)", "boto3 (>=1.42.25)", "cachetools", "cachetools (>=5.5.0)", "cel-python (>=0.4.0)", "cel-python (>=0.4.0)", "certifi", "confluent-kafka", "fastapi", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "flake8", "google-api-core", "google-api-core", "google-auth", "google-auth", "google-cloud-kms", "google-cloud-kms", "google-re2 (<1.1.20251105)", "googleapis-common-protos", "googleapis-common-protos", "hkdf (==0.0.3)", "hkdf (==0.0.3)", "httpx (>=0.26)", "httpx (>=0.26)", "hvac", "hvac", "isort (>=5.13.0)", "jsonata-python", "jsonata-python", "jsonschema (>=4.18.0)", "jsonschema (>=4.18.0)", "mypy", "pandoc", "pluggy (<1.6.0)", "protobuf", "protobuf", "pydantic", "pytest", "pytest-asyncio", "pytest-timeout", "pytest_cov", "pyyaml (>=6.0.0)", "pyyaml (>=6.0.0)", "requests", "requests", "requests-mock", "respx", "six", "sphinx", "sphinx-rtd-theme", "tink", "tink", "tomli ; python_version < \"3.11\"", "types-cachetools", "types-requests", "urllib3 (<3)", "uvicorn"] +docs = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "avro (>=1.11.1,<2)", "azure-identity", "azure-keyvault-keys", "boto3 (>=1.35)", "cachetools (>=5.5.0)", "cel-python (>=0.4.0)", "certifi", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "google-api-core", "google-auth", "google-cloud-kms", "google-re2 (<1.1.20251105)", "googleapis-common-protos", "hkdf (==0.0.3)", "httpx (>=0.26)", "hvac", "jsonata-python", "jsonschema (>=4.18.0)", "pandoc", "protobuf", "pyyaml (>=6.0.0)", "requests", "sphinx", "sphinx-rtd-theme", "tink", "tomli ; python_version < \"3.11\""] +examples = ["attrs", "authlib (>=1.0.0)", "avro (>=1.11.1,<2)", "azure-identity", "azure-keyvault-keys", "boto3", "cachetools", "cel-python (>=0.4.0)", "confluent-kafka", "fastapi", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "google-api-core", "google-auth", "google-cloud-kms", "googleapis-common-protos", "hkdf (==0.0.3)", "httpx (>=0.26)", "hvac", "jsonata-python", "jsonschema (>=4.18.0)", "protobuf", "pydantic", "pyyaml (>=6.0.0)", "requests", "six", "tink", "uvicorn"] +json = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "cachetools (>=5.5.0)", "certifi", "httpx (>=0.26)", "jsonschema (>=4.18.0)"] +json-fast = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "cachetools (>=5.5.0)", "certifi", "httpx (>=0.26)", "jsonschema (>=4.18.0)", "orjson (>=3.10)"] +oauthbearer-aws = ["boto3 (>=1.42.25)"] +protobuf = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "cachetools (>=5.5.0)", "certifi", "googleapis-common-protos", "httpx (>=0.26)", "protobuf"] +rules = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "azure-identity", "azure-keyvault-keys", "boto3 (>=1.35)", "cachetools (>=5.5.0)", "cel-python (>=0.4.0)", "certifi", "google-api-core", "google-auth", "google-cloud-kms", "google-re2 (<1.1.20251105)", "hkdf (==0.0.3)", "httpx (>=0.26)", "hvac", "jsonata-python", "pyyaml (>=6.0.0)", "tink"] +schema-registry = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "cachetools (>=5.5.0)", "certifi", "httpx (>=0.26)"] +schemaregistry = ["attrs (>=21.2.0)", "authlib (>=1.0.0)", "cachetools (>=5.5.0)", "certifi", "httpx (>=0.26)"] +soaktest = ["opentelemetry-distro", "opentelemetry-exporter-otlp", "psutil"] +tests = ["async-timeout", "attrs", "attrs (>=21.2.0)", "authlib (>=1.0.0)", "avro (>=1.11.1,<2)", "azure-identity", "azure-keyvault-keys", "black (>=24.0.0)", "boto3 (>=1.35)", "boto3 (>=1.42.25)", "cachetools (>=5.5.0)", "cel-python (>=0.4.0)", "certifi", "fastavro (>=1.5.4,<1.8.0) ; python_version == \"3.7\"", "fastavro (>=1.5.4,<2) ; python_version > \"3.7\"", "flake8", "google-api-core", "google-auth", "google-cloud-kms", "google-re2 (<1.1.20251105)", "googleapis-common-protos", "hkdf (==0.0.3)", "httpx (>=0.26)", "hvac", "isort (>=5.13.0)", "jsonata-python", "jsonschema (>=4.18.0)", "mypy", "pluggy (<1.6.0)", "protobuf", "pytest", "pytest-asyncio", "pytest-timeout", "pytest_cov", "pyyaml (>=6.0.0)", "requests", "requests-mock", "respx", "tink", "types-cachetools", "types-requests", "urllib3 (<3)"] + [[package]] name = "contourpy" version = "1.3.2" @@ -2736,4 +2801,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.1" python-versions = "^3.10" -content-hash = "2791ff869427cfef745fea532d5032ab23d0a2754a87627f20617bfeb3a32a2a" +content-hash = "07ffa76dde42c0a3c35b4af86a3c9c0a2bb7f65552c87c7dc4ba78401246fda0" diff --git a/pyproject.toml b/pyproject.toml index 809fac5..baf23de 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,9 +1,9 @@ [project] name = "data-layer" -version = "0.1.0" +version = "2.0.0" description = "Centralized data layer for Binance crypto and VN stock market data" authors = [ - {name = "Bobby", email = "bobbyax.work@example.com"} + {name = "BobbyAxerol", email = "vugioan11022002@gmail.com"} ] readme = "README.md" requires-python = "^3.10" @@ -25,6 +25,7 @@ dependencies = [ "msgpack (>=1.1.0,<2.0.0)", "protobuf (>=6.31.1,<7.0.0)", "grpcio (>=1.70.0,<2.0.0)", + "confluent-kafka (>=2.15.0,<3.0.0)", "httpx (>=0.28.0,<1.0.0)", "idna (>=3.15,<4.0)", "pillow (>=12.3.0,<13.0.0)", diff --git a/qdl/adapters/binance/__init__.py b/qdl/adapters/binance/__init__.py new file mode 100644 index 0000000..571d307 --- /dev/null +++ b/qdl/adapters/binance/__init__.py @@ -0,0 +1,13 @@ +"""Binance V2 acquisition edges.""" + +from qdl.adapters.binance.bar_edge import ( + BinanceBarRawBinding, + fetch_closed_bar_history_raw_envelopes, + fetch_latest_closed_bar_raw_envelope, +) + +__all__ = [ + "BinanceBarRawBinding", + "fetch_closed_bar_history_raw_envelopes", + "fetch_latest_closed_bar_raw_envelope", +] diff --git a/qdl/adapters/binance/bar_edge.py b/qdl/adapters/binance/bar_edge.py new file mode 100644 index 0000000..beb3239 --- /dev/null +++ b/qdl/adapters/binance/bar_edge.py @@ -0,0 +1,241 @@ +from __future__ import annotations + +import json +import random +import time +from dataclasses import dataclass +from typing import Callable + +from app.providers.binance.rest import BinanceProviderError, fetch_klines +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.capture import capture_exact_frame + + +@dataclass(frozen=True) +class BinanceBarRawBinding: + market: str + product_type: str + native_symbol: str + interval: str + subscription_id: str + source_session_id: str + connection_generation: int + lease_epoch: int + authority_revision: int + partition_plan_epoch: int + adapter_version: str + config_revision: int + instrument_catalog_revision: int + + def __post_init__(self) -> None: + if self.market not in {"USDM", "SPOT"}: + raise ValueError("Binance bar market must be USDM or SPOT") + if self.product_type not in {"PERPETUAL", "SPOT"}: + raise ValueError("Binance bar product type is invalid") + if self.market == "USDM" and self.product_type != "PERPETUAL": + raise ValueError("Binance USDM bar requires PERPETUAL product") + if self.market == "SPOT" and self.product_type != "SPOT": + raise ValueError("Binance Spot bar requires SPOT product") + strings = ( + self.native_symbol, self.interval, self.subscription_id, + self.source_session_id, self.adapter_version, + ) + if any(not value.strip() for value in strings): + raise ValueError("Binance bar binding identity is incomplete") + if min( + self.connection_generation, + self.lease_epoch, + self.authority_revision, + self.partition_plan_epoch, + self.config_revision, + self.instrument_catalog_revision, + ) <= 0: + raise ValueError("Binance bar binding revisions/epochs must be positive") + + +def _interval_ms(interval: str) -> int: + units = {"m": 60_000, "h": 3_600_000, "d": 86_400_000} + if not interval or interval[-1] not in units: + raise ValueError("Binance history interval must have a fixed m/h/d duration") + count = int(interval[:-1]) + if count <= 0: + raise ValueError("Binance history interval must be positive") + return count * units[interval[-1]] + + +def _fetch_rows( + binding: BinanceBarRawBinding, + *, + observed_ms: int, + limit: int, + attempts: int, + fetcher: Callable, + sleep: Callable[[float], None], +) -> list[list]: + if attempts < 1 or attempts > 10: + raise ValueError("Binance bar attempts must be between 1 and 10") + last_error: BaseException | None = None + response = None + for attempt in range(attempts): + try: + response = fetcher( + binding.native_symbol, + interval=binding.interval, + limit=limit, + end_time=observed_ms, + market=binding.market.lower(), + ) + break + except (BinanceProviderError, OSError, ValueError) as error: + last_error = error + if attempt + 1 < attempts: + sleep(min(2**attempt, 4) + random.random() * 0.25) + if response is None: + raise RuntimeError("Binance closed-bar fetch exhausted retries") from last_error + rows = response.get("data") + if not isinstance(rows, list): + raise ValueError("Binance kline response data must be a list") + return rows + + +def _capture_row( + binding: BinanceBarRawBinding, + row: list, + *, + origin: str, + received_at_ns: int, + test_provenance: bool, +) -> raw_provider_pb2.RawProviderEnvelope: + raw = { + "symbol": binding.native_symbol.upper(), + "interval": binding.interval, + "bar_origin": origin, + "row": row, + } + raw_bytes = json.dumps( + raw, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return capture_exact_frame( + provider="BINANCE_DIRECT", + venue="BINANCE", + market=binding.market, + product_type=binding.product_type, + native_symbol=binding.native_symbol.upper(), + native_channel=f"rest-klines/{binding.interval}", + subscription_id=binding.subscription_id, + source_session_id=binding.source_session_id, + connection_generation=binding.connection_generation, + lease_epoch=binding.lease_epoch, + authority_revision=binding.authority_revision, + partition_plan_epoch=binding.partition_plan_epoch, + received_at_ns=received_at_ns, + raw_frame_bytes=raw_bytes, + adapter_version=binding.adapter_version, + config_revision=binding.config_revision, + instrument_catalog_revision=binding.instrument_catalog_revision, + correlation_id=( + f"binance:{binding.market}:{binding.native_symbol}:" + f"{binding.interval}:{row[0]}" + ), + transport_protocol=raw_provider_pb2.TRANSPORT_PROTOCOL_HTTP, + capture_boundary=raw_provider_pb2.CAPTURE_BOUNDARY_POST_DECOMPRESSION, + test_provenance=test_provenance, + ) + + +def _closed_rows( + rows: list[list], + *, + observed_ms: int, + interval_ms: int, +) -> tuple[list, ...]: + by_open: dict[int, list] = {} + for row in rows: + if not isinstance(row, list) or len(row) < 11: + raise ValueError("Binance kline row has invalid native shape") + open_time = int(row[0]) + close_time = int(row[6]) + if close_time >= observed_ms: + continue + if close_time != open_time + interval_ms - 1: + raise ValueError("Binance kline boundary differs from requested interval") + previous = by_open.get(open_time) + if previous is not None and previous != row: + raise ValueError("Binance returned conflicting rows for one open time") + by_open[open_time] = row + return tuple(by_open[key] for key in sorted(by_open)) + + +def fetch_closed_bar_history_raw_envelopes( + binding: BinanceBarRawBinding, + *, + limit: int, + now_ms: int | None = None, + attempts: int = 4, + fetcher: Callable = fetch_klines, + sleep: Callable[[float], None] = time.sleep, + test_provenance: bool = False, +) -> tuple[raw_provider_pb2.RawProviderEnvelope, ...]: + if limit < 1 or limit > 1000: + raise ValueError("Binance history limit must be between 1 and 1000") + observed_ms = int(now_ms if now_ms is not None else time.time() * 1000) + interval_ms = _interval_ms(binding.interval) + rows = _fetch_rows( + binding, + observed_ms=observed_ms, + limit=min(1000, limit + 2), + attempts=attempts, + fetcher=fetcher, + sleep=sleep, + ) + closed = _closed_rows(rows, observed_ms=observed_ms, interval_ms=interval_ms) + selected = closed[-limit:] + if len(selected) != limit: + raise RuntimeError( + f"Binance closed-bar history is incomplete requested={limit} observed={len(selected)}" + ) + opens = [int(row[0]) for row in selected] + if any(current - previous != interval_ms for previous, current in zip(opens, opens[1:])): + raise RuntimeError("Binance closed-bar history contains a time gap") + received_at_ns = time.time_ns() + return tuple( + _capture_row( + binding, + row, + origin="BACKFILLED", + received_at_ns=received_at_ns + index, + test_provenance=test_provenance, + ) + for index, row in enumerate(selected) + ) + + +def fetch_latest_closed_bar_raw_envelope( + binding: BinanceBarRawBinding, + *, + now_ms: int | None = None, + attempts: int = 4, + fetcher: Callable = fetch_klines, + sleep: Callable[[float], None] = time.sleep, + test_provenance: bool = False, +) -> raw_provider_pb2.RawProviderEnvelope: + observed_ms = int(now_ms if now_ms is not None else time.time() * 1000) + interval_ms = _interval_ms(binding.interval) + rows = _fetch_rows( + binding, + observed_ms=observed_ms, + limit=3, + attempts=attempts, + fetcher=fetcher, + sleep=sleep, + ) + closed = _closed_rows(rows, observed_ms=observed_ms, interval_ms=interval_ms) + if not closed: + raise RuntimeError("Binance returned no closed bar before the observation time") + return _capture_row( + binding, + closed[-1], + origin="VENUE_NATIVE", + received_at_ns=time.time_ns(), + test_provenance=test_provenance, + ) diff --git a/qdl/adapters/binance_usdm.py b/qdl/adapters/binance_usdm.py index 1993c62..15c33f6 100644 --- a/qdl/adapters/binance_usdm.py +++ b/qdl/adapters/binance_usdm.py @@ -24,7 +24,7 @@ BINANCE_USDM_EXCHANGE_INFO = "https://fapi.binance.com/fapi/v1/exchangeInfo" BINANCE_USDM_KLINES = "https://fapi.binance.com/fapi/v1/klines" -BINANCE_USDM_WS_BASE = "wss://fstream.binance.com/stream?streams=" +BINANCE_USDM_WS_BASE = "wss://fstream.binance.com/public/stream?streams=" async def _receive_until_stop(socket: Any, stop: asyncio.Event, timeout: float) -> Any | None: diff --git a/qdl/adapters/okx/__init__.py b/qdl/adapters/okx/__init__.py index 0c1b767..2ce0b8e 100644 --- a/qdl/adapters/okx/__init__.py +++ b/qdl/adapters/okx/__init__.py @@ -39,3 +39,14 @@ "OkxOpenInterestSnapshot", "PaginationStalled", ] +from qdl.adapters.okx.bar_edge import ( + OkxBarRawBinding, + fetch_closed_bar_history_raw_envelopes, + fetch_latest_closed_bar_raw_envelope, +) + +__all__ += [ + "OkxBarRawBinding", + "fetch_closed_bar_history_raw_envelopes", + "fetch_latest_closed_bar_raw_envelope", +] diff --git a/qdl/adapters/okx/bar_edge.py b/qdl/adapters/okx/bar_edge.py new file mode 100644 index 0000000..f19df1a --- /dev/null +++ b/qdl/adapters/okx/bar_edge.py @@ -0,0 +1,188 @@ +from __future__ import annotations + +import asyncio +import json +import time +from dataclasses import dataclass + +from qdl.adapters.okx.client import OkxRestClient +from qdl.adapters.okx.history import OkxHistoricalClient +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.capture import capture_exact_frame + + +@dataclass(frozen=True) +class OkxBarRawBinding: + market: str + product_type: str + native_symbol: str + interval: str + subscription_id: str + source_session_id: str + connection_generation: int + lease_epoch: int + authority_revision: int + partition_plan_epoch: int + adapter_version: str + config_revision: int + instrument_catalog_revision: int + + def __post_init__(self) -> None: + if self.market not in {"SWAP", "SPOT"}: + raise ValueError("OKX bar market must be SWAP or SPOT") + if self.product_type not in {"PERPETUAL", "SPOT"}: + raise ValueError("OKX bar product type is invalid") + if self.market == "SWAP" and self.product_type != "PERPETUAL": + raise ValueError("OKX SWAP bar requires PERPETUAL product") + if self.market == "SPOT" and self.product_type != "SPOT": + raise ValueError("OKX Spot bar requires SPOT product") + if self.interval != "1m": + raise ValueError("stable OKX history edge currently certifies 1m BAR only") + if any( + not value.strip() + for value in ( + self.native_symbol, + self.subscription_id, + self.source_session_id, + self.adapter_version, + ) + ): + raise ValueError("OKX bar binding identity is incomplete") + if min( + self.connection_generation, + self.lease_epoch, + self.authority_revision, + self.partition_plan_epoch, + self.config_revision, + self.instrument_catalog_revision, + ) <= 0: + raise ValueError("OKX bar binding revisions/epochs must be positive") + + +async def fetch_closed_bar_history_raw_envelopes( + binding: OkxBarRawBinding, + *, + limit: int, + now_ms: int | None = None, + history_client: OkxHistoricalClient | None = None, + test_provenance: bool = False, +) -> tuple[raw_provider_pb2.RawProviderEnvelope, ...]: + if limit < 1 or limit > 10_000: + raise ValueError("OKX history limit must be between 1 and 10000") + observed_ms = int(now_ms if now_ms is not None else time.time() * 1000) + closed_boundary_ms = observed_ms // 60_000 * 60_000 + start_ms = closed_boundary_ms - limit * 60_000 + end_ms = closed_boundary_ms - 1 + client = history_client or OkxHistoricalClient(OkxRestClient()) + history = await client.candles( + inst_id=binding.native_symbol, + bar=binding.interval, + start_ms=start_ms, + end_ms=end_ms, + price_type="TRADE", + max_records=limit, + max_pages=max(2, (limit + 299) // 300 + 1), + ) + records = history.records + if ( + history.coverage.status != "FULL" + or len(records) != limit + or history.coverage.observed_min_ts_ms != start_ms + ): + raise RuntimeError( + "OKX closed-bar history is incomplete " + f"requested={limit} observed={len(records)} coverage={history.coverage.status}" + ) + opens = [item.open_ts_ms for item in records] + if any(current - previous != 60_000 for previous, current in zip(opens, opens[1:])): + raise RuntimeError("OKX closed-bar history contains a time gap") + if any(not item.confirmed for item in records): + raise RuntimeError("OKX history returned a provisional candle in the closed window") + + received_at_ns = time.time_ns() + envelopes = [] + for index, item in enumerate(records): + row = [ + str(item.open_ts_ms), + item.open, + item.high, + item.low, + item.close, + item.volume_raw, + item.volume_ccy_raw, + item.volume_quote_raw, + "1", + ] + if any(value is None for value in row): + raise RuntimeError("OKX trade candle is missing a native volume field") + raw = { + "arg": { + "channel": "candle1m", + "instId": binding.native_symbol, + }, + "data": [row], + } + raw_bytes = json.dumps( + raw, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + envelopes.append(capture_exact_frame( + provider="OKX_DIRECT", + venue="OKX", + market=binding.market, + product_type=binding.product_type, + native_symbol=binding.native_symbol, + native_channel="candle1m", + subscription_id=binding.subscription_id, + source_session_id=binding.source_session_id, + connection_generation=binding.connection_generation, + lease_epoch=binding.lease_epoch, + authority_revision=binding.authority_revision, + partition_plan_epoch=binding.partition_plan_epoch, + received_at_ns=received_at_ns + index, + raw_frame_bytes=raw_bytes, + adapter_version=binding.adapter_version, + config_revision=binding.config_revision, + instrument_catalog_revision=binding.instrument_catalog_revision, + correlation_id=( + f"okx:{binding.market}:{binding.native_symbol}:" + f"{binding.interval}:{item.open_ts_ms}" + ), + transport_protocol=raw_provider_pb2.TRANSPORT_PROTOCOL_HTTP, + capture_boundary=raw_provider_pb2.CAPTURE_BOUNDARY_POST_DECOMPRESSION, + test_provenance=test_provenance, + )) + return tuple(envelopes) + +async def fetch_latest_closed_bar_raw_envelope( + binding: OkxBarRawBinding, + *, + now_ms: int | None = None, + history_client: OkxHistoricalClient | None = None, + attempts: int = 4, + sleep=asyncio.sleep, + test_provenance: bool = False, +) -> raw_provider_pb2.RawProviderEnvelope: + if not 1 <= attempts <= 8: + raise ValueError("OKX latest-closed BAR attempts must be between 1 and 8") + last_error: Exception | None = None + for attempt in range(attempts): + try: + values = await fetch_closed_bar_history_raw_envelopes( + binding, + limit=1, + now_ms=now_ms, + history_client=history_client, + test_provenance=test_provenance, + ) + if len(values) != 1: + raise RuntimeError( + "OKX latest-closed BAR lookup returned invalid cardinality" + ) + return values[0] + except Exception as error: + last_error = error + if attempt + 1 < attempts: + await sleep(min(0.5 * (2 ** attempt), 2.0)) + raise RuntimeError( + f"OKX latest-closed BAR lookup exhausted attempts={attempts}" + ) from last_error diff --git a/qdl/adapters/vn/__init__.py b/qdl/adapters/vn/__init__.py new file mode 100644 index 0000000..b802d2f --- /dev/null +++ b/qdl/adapters/vn/__init__.py @@ -0,0 +1,13 @@ +"""Vietnam-market provider acquisition edges for the V2 raw contract.""" + +from qdl.adapters.vn.raw_edge import ( + VnRawBinding, + build_dnse_bar_raw_envelope, + build_dnse_trade_raw_envelope, +) + +__all__ = [ + "VnRawBinding", + "build_dnse_bar_raw_envelope", + "build_dnse_trade_raw_envelope", +] diff --git a/qdl/adapters/vn/raw_edge.py b/qdl/adapters/vn/raw_edge.py new file mode 100644 index 0000000..63d3b1e --- /dev/null +++ b/qdl/adapters/vn/raw_edge.py @@ -0,0 +1,186 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from decimal import Decimal, InvalidOperation +from typing import Any, Literal, Mapping + +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.capture import capture_exact_frame + + +@dataclass(frozen=True) +class VnRawBinding: + venue: str + market: str + product_type: str + native_symbol: str + subscription_id: str + source_session_id: str + connection_generation: int + lease_epoch: int + authority_revision: int + partition_plan_epoch: int + adapter_version: str + config_revision: int + instrument_catalog_revision: int + + def __post_init__(self) -> None: + strings = ( + self.venue, self.market, self.product_type, self.native_symbol, + self.subscription_id, self.source_session_id, self.adapter_version, + ) + if any(not value.strip() for value in strings): + raise ValueError("VN raw binding identity is incomplete") + revisions = ( + self.connection_generation, self.lease_epoch, self.authority_revision, + self.partition_plan_epoch, self.config_revision, + self.instrument_catalog_revision, + ) + if min(revisions) <= 0: + raise ValueError("VN raw binding revisions/epochs must be positive") + + +def _decimal_text(value: Any, field: str) -> str: + if value is None or isinstance(value, bool): + raise ValueError(f"DNSE {field} is missing or invalid") + try: + parsed = Decimal(str(value)) + except (InvalidOperation, ValueError) as error: + raise ValueError(f"DNSE {field} is invalid") from error + if not parsed.is_finite() or parsed <= 0: + raise ValueError(f"DNSE {field} must be positive") + return format(parsed, "f") + + +def build_dnse_trade_raw_envelope( + delivery: Mapping[str, Any], + binding: VnRawBinding, + *, + received_at_ns: int, + test_provenance: bool = False, +) -> raw_provider_pb2.RawProviderEnvelope: + symbol = str(delivery.get("symbol") or "").upper() + if symbol != binding.native_symbol.upper(): + raise ValueError("DNSE delivery symbol does not match approved binding") + raw = { + "symbol": symbol, + "price": _decimal_text(delivery.get("price"), "price"), + "quantity": _decimal_text(delivery.get("quantity"), "quantity"), + "market_id": str(delivery.get("market_id") or delivery.get("marketId") or ""), + "board_id": str(delivery.get("board_id") or delivery.get("boardId") or ""), + "trading_session_id": str( + delivery.get("trading_session_id") + or delivery.get("tradingSessionId") + or "" + ), + "total_volume_traded": _decimal_text( + delivery.get("total_volume_traded") + or delivery.get("totalVolumeTraded") + or delivery.get("total_volume"), + "total_volume_traded", + ), + } + raw_bytes = json.dumps( + raw, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return capture_exact_frame( + provider="DNSE_DIRECT", + venue=binding.venue, + market=binding.market, + product_type=binding.product_type, + native_symbol=binding.native_symbol, + native_channel="trades", + subscription_id=binding.subscription_id, + source_session_id=binding.source_session_id, + connection_generation=binding.connection_generation, + lease_epoch=binding.lease_epoch, + authority_revision=binding.authority_revision, + partition_plan_epoch=binding.partition_plan_epoch, + received_at_ns=received_at_ns, + raw_frame_bytes=raw_bytes, + adapter_version=binding.adapter_version, + config_revision=binding.config_revision, + instrument_catalog_revision=binding.instrument_catalog_revision, + correlation_id=f"dnse:{binding.native_symbol}:{received_at_ns}", + transport_protocol=raw_provider_pb2.TRANSPORT_PROTOCOL_SDK_CALLBACK, + capture_boundary=raw_provider_pb2.CAPTURE_BOUNDARY_SDK_DELIVERY, + test_provenance=test_provenance, + ) + + +def build_dnse_bar_raw_envelope( + row: Mapping[str, Any], + binding: VnRawBinding, + *, + received_at_ns: int, + interval: str = "1m", + acquisition_origin: Literal["REST_HISTORY", "WEBSOCKET_CLOSED"] = "REST_HISTORY", + test_provenance: bool = False, +) -> raw_provider_pb2.RawProviderEnvelope: + if interval != "1m": + raise ValueError("stable DNSE BAR edge currently supports native 1m only") + if acquisition_origin not in {"REST_HISTORY", "WEBSOCKET_CLOSED"}: + raise ValueError("stable DNSE BAR acquisition origin is invalid") + try: + open_time_ms = int(row["t"]) * 1000 + except (KeyError, TypeError, ValueError) as error: + raise ValueError("DNSE BAR timestamp is invalid") from error + if open_time_ms <= 0: + raise ValueError("DNSE BAR timestamp must be positive") + values = {} + for field in ("o", "h", "l", "c"): + values[field] = _decimal_text(row.get(field), field) + try: + volume = Decimal(str(row.get("v"))) + except (InvalidOperation, ValueError, TypeError) as error: + raise ValueError("DNSE BAR volume is invalid") from error + if not volume.is_finite() or volume < 0: + raise ValueError("DNSE BAR volume must be non-negative") + raw = { + "symbol": binding.native_symbol.upper(), + "interval": interval, + "open_time_ms": open_time_ms, + "close_time_ms": open_time_ms + 59_999, + **values, + "v": format(volume, "f"), + "is_final": True, + "revision": 0, + "trade_count_available": False, + } + raw_bytes = json.dumps( + raw, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return capture_exact_frame( + provider="DNSE_DIRECT", + venue=binding.venue, + market=binding.market, + product_type=binding.product_type, + native_symbol=binding.native_symbol, + # One provider BAR product binding enters Rust; transport/capture fields + # retain whether the frame came from REST history or the closed-BAR SDK. + native_channel="ohlcv/1m", + subscription_id=binding.subscription_id, + source_session_id=binding.source_session_id, + connection_generation=binding.connection_generation, + lease_epoch=binding.lease_epoch, + authority_revision=binding.authority_revision, + partition_plan_epoch=binding.partition_plan_epoch, + received_at_ns=received_at_ns, + raw_frame_bytes=raw_bytes, + adapter_version=binding.adapter_version, + config_revision=binding.config_revision, + instrument_catalog_revision=binding.instrument_catalog_revision, + correlation_id=f"dnse:{binding.native_symbol}:1m:{open_time_ms}", + transport_protocol=( + raw_provider_pb2.TRANSPORT_PROTOCOL_SDK_CALLBACK + if acquisition_origin == "WEBSOCKET_CLOSED" + else raw_provider_pb2.TRANSPORT_PROTOCOL_HTTP + ), + capture_boundary=( + raw_provider_pb2.CAPTURE_BOUNDARY_SDK_DELIVERY + if acquisition_origin == "WEBSOCKET_CLOSED" + else raw_provider_pb2.CAPTURE_BOUNDARY_POST_DECOMPRESSION + ), + test_provenance=test_provenance, + ) diff --git a/qdl/adapters/vn/stable_edge.py b/qdl/adapters/vn/stable_edge.py new file mode 100644 index 0000000..4638714 --- /dev/null +++ b/qdl/adapters/vn/stable_edge.py @@ -0,0 +1,606 @@ +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +import os +import queue +import threading +import time +import uuid +from decimal import Decimal, InvalidOperation +from pathlib import Path +from typing import Any + +from app.config import DNSE_API_KEY, DNSE_API_SECRET_KEY, DNSE_WS_BASE +from app.providers.dnse import fetch_dnse_ohlc_raw +from app.stream.dnse_ws import TradingClient +from qdl.adapters.vn import ( + VnRawBinding, + build_dnse_bar_raw_envelope, + build_dnse_trade_raw_envelope, +) +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import StableAcquisitionPlan +from qdl.transport.kafka_raw import KafkaRawPublisher, KafkaRawPublisherConfig + + +logger = logging.getLogger(__name__) + + +class StableDnseVendorEdge: + """DNSE vendor acquisition with lossless Kafka ACK and atomic BAR recovery.""" + + def __init__( + self, + *, + catalog: StableSourceCatalog, + acquisition: StableAcquisitionPlan, + authority: dict[str, Any], + publisher: KafkaRawPublisher, + queue_capacity: int = 5000, + warmup_rows: int = 500, + history_lookback_days: int = 30, + history_attempts: int = 1, + history_fetcher=fetch_dnse_ohlc_raw, + state_path: str | Path | None = None, + clock=time.time, + sleep=time.sleep, + ) -> None: + if not 1 <= queue_capacity <= 100_000: + raise ValueError("stable DNSE queue capacity is invalid") + if not 1 <= warmup_rows <= 2000: + raise ValueError("stable DNSE warmup rows must be between 1 and 2000") + if not 1 <= history_lookback_days <= 87: + raise ValueError("stable DNSE history lookback must be between 1 and 87 days") + if not 1 <= history_attempts <= 8: + raise ValueError("stable DNSE history attempts must be between 1 and 8") + if ( + authority.get("mode") != "RUST_SHADOW" + or authority.get("public_write_allowed") is not False + or authority.get("legacy_write_allowed") is not False + ): + raise ValueError("stable DNSE edge requires shadow authority") + self.catalog = catalog + self.acquisition = acquisition + self.authority = authority + self.publisher = publisher + self.warmup_rows = warmup_rows + self.history_lookback_days = history_lookback_days + self.history_attempts = history_attempts + self.history_fetcher = history_fetcher + self.state_path = Path(state_path) if state_path is not None else None + self.clock = clock + self.sleep = sleep + self.session_id = ( + f"qdl-v2-stable-dnse-r{int(authority['revision'])}-{uuid.uuid4()}" + ) + source_by_id = {item.binding_id: item for item in catalog.bindings} + selected = tuple( + source_by_id[item.binding_id] + for item in acquisition.bindings + if item.mode == "PYTHON_VENDOR_SDK" and item.runtime == "DNSE" + ) + self.trade_sources = { + item.instrument.native_symbol: item + for item in selected + if item.feed.value == "TRADE" + } + self.bar_sources = { + item.instrument.native_symbol: item + for item in selected + if item.feed.value == "BAR" + } + if set(self.trade_sources) != set(self.bar_sources) or not self.trade_sources: + raise ValueError("stable DNSE trade/BAR symbols are inconsistent") + if any(source.interval != "1m" for source in self.bar_sources.values()): + raise ValueError("stable DNSE edge requires native final 1m BAR bindings") + + self._queue: queue.Queue[tuple[str, dict[str, Any], int]] = queue.Queue( + maxsize=queue_capacity + ) + self._fatal = threading.Event() + self._stopped = threading.Event() + self._state_lock = threading.Lock() + self._last_bar: dict[str, tuple[int, str]] = {} + self._observed_bar: dict[str, tuple[int, str]] = {} + self._history_bootstrapped = False + self._worker: threading.Thread | None = None + self._restore_state() + + @property + def _bar_binding_ids(self) -> tuple[str, ...]: + return tuple(sorted(source.binding_id for source in self.bar_sources.values())) + + def _binding(self, source) -> VnRawBinding: + identity = source.instrument.identity + return VnRawBinding( + venue=identity.venue, + market=identity.market, + product_type=identity.product_type.value, + native_symbol=source.instrument.native_symbol, + subscription_id=source.source_id, + source_session_id=self.session_id, + connection_generation=1, + lease_epoch=1, + authority_revision=int(self.authority["revision"]), + partition_plan_epoch=1, + adapter_version=source.adapter_version, + config_revision=self.acquisition.revision, + instrument_catalog_revision=self.catalog.catalog_revision, + ) + + @staticmethod + def _decimal_identity(value: Any, field: str, *, allow_zero: bool) -> str: + if value is None or isinstance(value, bool): + raise ValueError(f"DNSE BAR {field} is missing") + try: + parsed = Decimal(str(value)) + except (InvalidOperation, ValueError) as error: + raise ValueError(f"DNSE BAR {field} is invalid") from error + if not parsed.is_finite() or parsed < 0 or (not allow_zero and parsed == 0): + raise ValueError(f"DNSE BAR {field} is outside domain") + return format(parsed.normalize(), "f") + + @classmethod + def _row_identity(cls, row: dict[str, Any]) -> tuple[int, str]: + try: + open_time = int(row["t"]) + except (KeyError, TypeError, ValueError) as error: + raise ValueError("DNSE BAR timestamp is malformed") from error + if open_time <= 0 or open_time % 60: + raise ValueError("DNSE BAR timestamp is not aligned to native 1m") + prices = { + key: cls._decimal_identity(row.get(key), key, allow_zero=False) + for key in ("o", "h", "l", "c") + } + decimals = {key: Decimal(value) for key, value in prices.items()} + if ( + decimals["h"] < max(decimals["o"], decimals["c"], decimals["l"]) + or decimals["l"] > min(decimals["o"], decimals["c"], decimals["h"]) + ): + raise ValueError("DNSE BAR price invariants failed") + payload = { + "t": open_time, + **prices, + "v": cls._decimal_identity(row.get("v"), "v", allow_zero=True), + } + encoded = json.dumps( + payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ) + return open_time, hashlib.sha256(encoded.encode("ascii")).hexdigest() + + def _state_payload(self) -> dict[str, Any]: + return { + "schema": "qdl.stable-dnse-edge-state.v1", + "slice_id": str(self.authority.get("slice_id", "")), + "authority_revision": int(self.authority["revision"]), + "catalog_revision": int(self.catalog.catalog_revision), + "acquisition_revision": int(self.acquisition.revision), + "binding_ids": list(self._bar_binding_ids), + "last_bar": { + binding_id: { + "open_time_ms": value[0], + "payload_sha256": value[1], + } + for binding_id, value in sorted(self._last_bar.items()) + }, + } + + def _restore_state(self) -> None: + if self.state_path is None or not self.state_path.exists(): + return + try: + payload = json.loads(self.state_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as error: + raise RuntimeError("stable DNSE checkpoint is unreadable") from error + if not isinstance(payload, dict) or set(payload) != { + "schema", + "slice_id", + "authority_revision", + "catalog_revision", + "acquisition_revision", + "binding_ids", + "last_bar", + }: + raise RuntimeError("stable DNSE checkpoint fields are invalid") + expected = self._state_payload() + for field in ( + "schema", + "slice_id", + "authority_revision", + "catalog_revision", + "acquisition_revision", + "binding_ids", + ): + if payload[field] != expected[field]: + raise RuntimeError( + f"stable DNSE checkpoint {field} differs from runtime authority" + ) + values = payload["last_bar"] + if not isinstance(values, dict) or set(values) != set(self._bar_binding_ids): + raise RuntimeError("stable DNSE checkpoint is partial") + restored: dict[str, tuple[int, str]] = {} + for binding_id, item in values.items(): + if not isinstance(item, dict) or set(item) != { + "open_time_ms", "payload_sha256" + }: + raise RuntimeError("stable DNSE checkpoint watermark is invalid") + open_time_ms = item["open_time_ms"] + digest = item["payload_sha256"] + if ( + isinstance(open_time_ms, bool) + or not isinstance(open_time_ms, int) + or open_time_ms <= 0 + or open_time_ms % 60_000 + or not isinstance(digest, str) + or len(digest) != 64 + or any(char not in "0123456789abcdef" for char in digest) + ): + raise RuntimeError("stable DNSE checkpoint watermark is invalid") + restored[binding_id] = open_time_ms, digest + self._last_bar = restored + self._observed_bar = dict(restored) + self._history_bootstrapped = True + logger.info("stable DNSE checkpoint restored bindings=%s", len(restored)) + + def _persist_state(self) -> None: + if self.state_path is None: + return + parent = self.state_path.parent + parent.mkdir(parents=True, exist_ok=True) + temporary = parent / f".{self.state_path.name}.{os.getpid()}.tmp" + encoded = ( + json.dumps( + self._state_payload(), + sort_keys=True, + separators=(",", ":"), + ensure_ascii=True, + ) + + "\n" + ).encode("ascii") + descriptor = None + try: + descriptor = os.open( + temporary, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600 + ) + pending = memoryview(encoded) + while pending: + written = os.write(descriptor, pending) + if written <= 0: + raise OSError("stable DNSE checkpoint write made no progress") + pending = pending[written:] + os.fsync(descriptor) + os.close(descriptor) + descriptor = None + os.replace(temporary, self.state_path) + directory = os.open(parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(directory) + finally: + os.close(directory) + finally: + if descriptor is not None: + os.close(descriptor) + temporary.unlink(missing_ok=True) + + def _closed_history(self, symbol: str) -> tuple[dict[str, Any], ...]: + now_s = int(self.clock()) + start_s = now_s - self.history_lookback_days * 86_400 + last_error: Exception | None = None + for attempt in range(self.history_attempts): + try: + rows = self.history_fetcher(symbol, "1", start_s, now_s) + by_open: dict[int, tuple[str, dict[str, Any]]] = {} + for value in rows: + row = dict(value) + open_time, digest = self._row_identity(row) + if open_time + 60 > now_s: + continue + previous = by_open.get(open_time) + if previous is not None and previous[0] != digest: + raise RuntimeError( + f"DNSE historical BAR conflict symbol={symbol} open={open_time}" + ) + by_open[open_time] = digest, row + closed = tuple(by_open[key][1] for key in sorted(by_open)) + if len(closed) < self.warmup_rows: + raise RuntimeError( + f"DNSE historical BAR coverage incomplete symbol={symbol} " + f"expected={self.warmup_rows} actual={len(closed)}" + ) + return closed[-self.warmup_rows :] + except Exception as error: + last_error = error + if attempt + 1 < self.history_attempts: + self.sleep(min(2 ** attempt, 8)) + raise RuntimeError( + f"DNSE historical BAR bootstrap exhausted symbol={symbol} " + f"attempts={self.history_attempts}" + ) from last_error + + def bootstrap_history(self) -> int: + if self._history_bootstrapped: + return 0 + pending: list[tuple[str, dict[str, Any], Any]] = [] + for symbol, source in sorted(self.bar_sources.items()): + for row in self._closed_history(symbol): + pending.append((symbol, row, source)) + received_at_ns = int(self.clock() * 1_000_000_000) + envelopes = tuple( + build_dnse_bar_raw_envelope( + row, + self._binding(source), + received_at_ns=received_at_ns + index, + acquisition_origin="REST_HISTORY", + test_provenance=False, + ) + for index, (_symbol, row, source) in enumerate(pending) + ) + acknowledgements = self.publisher.publish_many(envelopes) + if len(acknowledgements) != len(envelopes): + raise RuntimeError("stable DNSE BAR bootstrap missed a Kafka ACK") + last_bar: dict[str, tuple[int, str]] = {} + for _symbol, row, source in pending: + open_time, digest = self._row_identity(row) + last_bar[source.binding_id] = open_time * 1000, digest + if set(last_bar) != set(self._bar_binding_ids): + raise RuntimeError("stable DNSE BAR bootstrap did not cover every binding") + with self._state_lock: + self._last_bar = last_bar + self._observed_bar = dict(last_bar) + self._persist_state() + self._history_bootstrapped = True + logger.info( + "stable real-provider DNSE BAR bootstrap ACK bindings=%s rows=%s", + len(last_bar), + len(envelopes), + ) + return len(acknowledgements) + + def _enqueue(self, kind: str, delivery: dict[str, Any], received_at_ns: int) -> None: + try: + self._queue.put_nowait((kind, delivery, received_at_ns)) + except queue.Full: + self._fatal.set() + logger.critical( + "stable DNSE queue exhausted; source fenced capacity=%s", + self._queue.maxsize, + ) + + def on_trade(self, trade) -> None: + symbol = str(getattr(trade, "symbol", "") or "").upper() + if symbol not in self.trade_sources or self._fatal.is_set(): + return + self._enqueue( + "TRADE", + { + "symbol": symbol, + "price": getattr(trade, "price", None), + "quantity": getattr(trade, "quantity", None), + "market_id": getattr(trade, "marketId", ""), + "board_id": getattr(trade, "boardId", ""), + "trading_session_id": getattr(trade, "tradingSessionId", ""), + "total_volume_traded": getattr(trade, "totalVolumeTraded", None), + }, + int(self.clock() * 1_000_000_000), + ) + + @staticmethod + def _provider_seconds(value: Any) -> int: + if isinstance(value, bool): + raise ValueError("DNSE closed BAR timestamp is invalid") + try: + timestamp = int(value) + except (TypeError, ValueError) as error: + raise ValueError("DNSE closed BAR timestamp is invalid") from error + if timestamp >= 1_000_000_000_000: + if timestamp % 1000: + raise ValueError("DNSE closed BAR millisecond timestamp loses precision") + timestamp //= 1000 + if timestamp <= 0: + raise ValueError("DNSE closed BAR timestamp is invalid") + return timestamp + + def on_ohlc_closed(self, ohlc) -> None: + symbol = str(getattr(ohlc, "symbol", "") or "").upper() + if symbol not in self.bar_sources or self._fatal.is_set(): + return + try: + resolution = str(getattr(ohlc, "resolution", "") or "") + if resolution != "1": + raise ValueError("DNSE closed BAR resolution differs from 1m binding") + row = { + "t": self._provider_seconds(getattr(ohlc, "time", None)), + "o": getattr(ohlc, "open", None), + "h": getattr(ohlc, "high", None), + "l": getattr(ohlc, "low", None), + "c": getattr(ohlc, "close", None), + "v": getattr(ohlc, "volume", None), + } + open_time, digest = self._row_identity(row) + if open_time + 60 > int(self.clock()) + 2: + raise ValueError("DNSE closed BAR is not closed at receipt time") + source = self.bar_sources[symbol] + with self._state_lock: + previous = self._observed_bar.get(source.binding_id) + if previous is not None and open_time * 1000 < previous[0]: + return + if previous is not None and open_time * 1000 == previous[0]: + if digest != previous[1]: + raise RuntimeError( + f"DNSE closed BAR conflict symbol={symbol} open={open_time}" + ) + return + self._observed_bar[source.binding_id] = open_time * 1000, digest + self._enqueue( + "BAR", + {"symbol": symbol, "row": row, "digest": digest}, + int(self.clock() * 1_000_000_000), + ) + except Exception: + self._fatal.set() + logger.exception("stable DNSE closed BAR invalid; source fenced symbol=%s", symbol) + + def _publish_worker(self) -> None: + while not self._fatal.is_set() and ( + not self._stopped.is_set() or not self._queue.empty() + ): + try: + first = self._queue.get(timeout=0.25) + except queue.Empty: + continue + batch = [first] + while len(batch) < 100: + try: + batch.append(self._queue.get_nowait()) + except queue.Empty: + break + try: + envelopes = [] + bar_updates: dict[str, tuple[int, str]] = {} + for kind, delivery, received_at_ns in batch: + symbol = str(delivery["symbol"]).upper() + if kind == "TRADE": + envelopes.append(build_dnse_trade_raw_envelope( + delivery, + self._binding(self.trade_sources[symbol]), + received_at_ns=received_at_ns, + test_provenance=False, + )) + elif kind == "BAR": + source = self.bar_sources[symbol] + row = delivery["row"] + envelopes.append(build_dnse_bar_raw_envelope( + row, + self._binding(source), + received_at_ns=received_at_ns, + acquisition_origin="WEBSOCKET_CLOSED", + test_provenance=False, + )) + bar_updates[source.binding_id] = ( + int(row["t"]) * 1000, + str(delivery["digest"]), + ) + else: + raise RuntimeError("stable DNSE queue event kind is invalid") + acknowledgements = self.publisher.publish_many(tuple(envelopes)) + if len(acknowledgements) != len(envelopes): + raise RuntimeError("stable DNSE raw batch missed a Kafka ACK") + if bar_updates: + with self._state_lock: + self._last_bar.update(bar_updates) + if set(self._last_bar) == set(self._bar_binding_ids): + self._persist_state() + except Exception: + self._fatal.set() + logger.exception("stable DNSE durable raw ACK failed; source fenced") + finally: + for _item in batch: + self._queue.task_done() + + async def run(self) -> None: + if not DNSE_API_KEY or not DNSE_API_SECRET_KEY: + raise RuntimeError("stable DNSE credentials are unavailable") + await asyncio.to_thread(self.bootstrap_history) + self._worker = threading.Thread( + target=self._publish_worker, + name="qdl-stable-dnse-kafka", + daemon=True, + ) + self._worker.start() + client = TradingClient( + api_key=DNSE_API_KEY, + api_secret=DNSE_API_SECRET_KEY, + base_url=DNSE_WS_BASE, + encoding="msgpack", + auto_reconnect=True, + max_retries=10, + heartbeat_interval=25.0, + dispatch_queue_capacity=max(100, self._queue.maxsize // 6), + ) + await client.connect() + symbols = sorted(self.trade_sources) + await client.subscribe_trades( + symbols=symbols, + on_trade=self.on_trade, + encoding="msgpack", + board_id="G1", + ) + await client.subscribe_trades( + symbols=symbols, + on_trade=self.on_trade, + encoding="msgpack", + board_id="G3", + ) + await client.subscribe_ohlc_closed( + symbols=symbols, + resolution="1", + on_ohlc=self.on_ohlc_closed, + encoding="msgpack", + ) + try: + while not self._stopped.is_set() and not self._fatal.is_set(): + if not client.is_healthy: + raise RuntimeError("stable DNSE SDK session is unhealthy") + await asyncio.sleep(5) + if self._fatal.is_set(): + raise RuntimeError("stable DNSE source fenced after loss/ACK failure") + finally: + self._stopped.set() + await client.disconnect() + if self._worker is not None: + await asyncio.to_thread(self._worker.join, 2.0) + self.publisher.close() + + def stop(self) -> None: + self._stopped.set() + + +def build_from_environment() -> StableDnseVendorEdge: + catalog = StableSourceCatalog.load(os.environ["QDL_STABLE_SOURCE_BINDINGS"]) + acquisition = StableAcquisitionPlan.load( + os.environ["QDL_STABLE_ACQUISITION_BINDINGS"], catalog=catalog + ) + authority = json.loads( + (Path(os.environ["QDL_STABLE_RUNTIME_DIR"]) / "authority.json").read_text() + ) + cert_root = Path(os.environ["QDL_KAFKA_CERT_ROOT"]) + publisher = KafkaRawPublisher(KafkaRawPublisherConfig( + bootstrap_servers=os.environ["QDL_KAFKA_BOOTSTRAP_SERVERS"], + client_id=os.environ["QDL_KAFKA_CLIENT_ID"], + topic=acquisition.raw_topic, + ca_path=cert_root / "ca.crt", + certificate_path=cert_root / "client.crt", + key_path=cert_root / "client.key", + )) + return StableDnseVendorEdge( + catalog=catalog, + acquisition=acquisition, + authority=authority, + publisher=publisher, + queue_capacity=int(os.environ.get("QDL_STABLE_DNSE_QUEUE_CAPACITY", "5000")), + warmup_rows=int(os.environ.get("QDL_STABLE_VN_WARMUP_ROWS", "500")), + history_lookback_days=int( + os.environ.get("QDL_STABLE_VN_HISTORY_LOOKBACK_DAYS", "30") + ), + history_attempts=int(os.environ.get("QDL_STABLE_VN_HISTORY_ATTEMPTS", "1")), + state_path=os.environ.get( + "QDL_STABLE_DNSE_STATE_PATH", + "/var/lib/qdl-stable/runtime/stable-dnse-edge.json", + ), + ) + + +async def main() -> int: + logging.basicConfig( + level=os.environ.get("LOG_LEVEL", "INFO"), + format="%(asctime)s %(levelname)s %(name)s %(message)s", + ) + edge = build_from_environment() + loop = asyncio.get_running_loop() + for signal_name in (__import__("signal").SIGTERM, __import__("signal").SIGINT): + loop.add_signal_handler(signal_name, edge.stop) + await edge.run() + return 0 diff --git a/qdl/api_v2/models.py b/qdl/api_v2/models.py index d68a4ef..9391149 100644 --- a/qdl/api_v2/models.py +++ b/qdl/api_v2/models.py @@ -1,5 +1,6 @@ from __future__ import annotations +from enum import StrEnum from typing import Annotated, Literal from pydantic import BaseModel, ConfigDict, Field, model_validator @@ -60,6 +61,18 @@ class DecimalValue(ClosedModel): source_text: str = Field(min_length=1, max_length=128) +class TradeIdentityKind(StrEnum): + NATIVE = "NATIVE" + DERIVED_RAW_CAPTURE = "DERIVED_RAW_CAPTURE" + + +class QuantityUnit(StrEnum): + BASE_ASSET = "BASE_ASSET" + QUOTE_ASSET = "QUOTE_ASSET" + CONTRACT = "CONTRACT" + SHARE = "SHARE" + + class SourceView(ClosedModel): venue: str = Field(min_length=1, max_length=40) provider: str = Field(min_length=1, max_length=80) @@ -99,7 +112,9 @@ class TradePayload(ClosedModel): native_trade_id: str = Field(min_length=1, max_length=200) price: DecimalValue quantity: DecimalValue - aggressor_side: Literal["BUY", "SELL"] + quantity_unit: QuantityUnit + aggressor_side: Literal["BUY", "SELL", "UNKNOWN"] + identity_kind: TradeIdentityKind is_block_trade: bool = False is_buyer_maker: bool = False @@ -110,6 +125,7 @@ class QuotePayload(ClosedModel): bid_quantity: DecimalValue ask_price: DecimalValue ask_quantity: DecimalValue + quantity_unit: QuantityUnit level: int = Field(default=1, ge=1) @@ -123,6 +139,10 @@ class BarPayload(ClosedModel): low: DecimalValue close: DecimalValue volume: DecimalValue + volume_unit: QuantityUnit + base_volume: DecimalValue | None = None + quote_volume: DecimalValue | None = None + contract_volume: DecimalValue | None = None trade_count: int = Field(default=0, ge=0) lifecycle: BarLifecycle revision: int = Field(ge=0) @@ -144,6 +164,7 @@ class BookLevel(ClosedModel): side: Literal["BID", "ASK"] price: DecimalValue quantity: DecimalValue + quantity_unit: QuantityUnit order_count: int = Field(default=0, ge=0) @@ -175,6 +196,7 @@ class FundingRatePayload(ClosedModel): class OpenInterestPayload(ClosedModel): feed: Literal[FeedType.OPEN_INTEREST] = FeedType.OPEN_INTEREST quantity: DecimalValue + quantity_unit: QuantityUnit notional: DecimalValue | None = None @@ -192,6 +214,16 @@ class TickerPayload(ClosedModel): high_24h: DecimalValue | None = None low_24h: DecimalValue | None = None volume_24h: DecimalValue | None = None + last_quantity_unit: QuantityUnit | None = None + volume_24h_unit: QuantityUnit | None = None + + @model_validator(mode="after") + def quantity_units_match_optional_values(self): + if (self.last_quantity is None) != (self.last_quantity_unit is None): + raise ValueError("ticker last quantity and unit must be present together") + if (self.volume_24h is None) != (self.volume_24h_unit is None): + raise ValueError("ticker 24h volume and unit must be present together") + return self MarketPayload = Annotated[ diff --git a/qdl/api_v2/router.py b/qdl/api_v2/router.py index ee71be1..9b7a7c6 100644 --- a/qdl/api_v2/router.py +++ b/qdl/api_v2/router.py @@ -141,6 +141,7 @@ def _book_levels(values: object) -> list[dict]: "side": item["side"], "price": _decimal(item["price"]), "quantity": _decimal(item["quantity"]), + "quantity_unit": item["quantity_unit"], "order_count": int(item.get("order_count", 0)), } for item in values @@ -155,7 +156,9 @@ def _typed_payload(item) -> dict: "native_trade_id": value["native_trade_id"], "price": _decimal(value["price"]), "quantity": _decimal(value["quantity"]), + "quantity_unit": value["quantity_unit"], "aggressor_side": value["aggressor_side"], + "identity_kind": value["identity_kind"], "is_block_trade": bool(value.get("is_block_trade", False)), "is_buyer_maker": bool(value.get("is_buyer_maker", False)), } @@ -166,6 +169,7 @@ def _typed_payload(item) -> dict: "bid_quantity": _decimal(value["bid_quantity"]), "ask_price": _decimal(value["ask_price"]), "ask_quantity": _decimal(value["ask_quantity"]), + "quantity_unit": value["quantity_unit"], "level": int(value.get("level", 1)), } if item.feed is FeedType.BAR: @@ -179,6 +183,22 @@ def _typed_payload(item) -> dict: "low": _decimal(value["low"]), "close": _decimal(value["close"]), "volume": _decimal(value["volume"]), + "volume_unit": value["volume_unit"], + "base_volume": ( + _decimal(value["base_volume"]) + if value.get("base_volume") is not None + else None + ), + "quote_volume": ( + _decimal(value["quote_volume"]) + if value.get("quote_volume") is not None + else None + ), + "contract_volume": ( + _decimal(value["contract_volume"]) + if value.get("contract_volume") is not None + else None + ), "trade_count": int(value.get("trade_count", 0)), "lifecycle": item.bar_lifecycle, "revision": item.revision, @@ -214,6 +234,7 @@ def _typed_payload(item) -> dict: return { "feed": item.feed, "quantity": _decimal(value["quantity"]), + "quantity_unit": value["quantity_unit"], "notional": _decimal(value["notional"]) if value.get("notional") is not None else None, } if item.feed is FeedType.MARK_INDEX_PRICE: @@ -226,6 +247,8 @@ def _typed_payload(item) -> dict: result = {"feed": item.feed, "last_price": _decimal(value["last_price"])} for field in ("last_quantity", "open_24h", "high_24h", "low_24h", "volume_24h"): result[field] = _decimal(value[field]) if value.get(field) is not None else None + result["last_quantity_unit"] = value.get("last_quantity_unit") + result["volume_24h_unit"] = value.get("volume_24h_unit") return result raise ValueError(f"public typed payload is undefined for {item.feed.value}") @@ -674,8 +697,14 @@ def create_v2_app( readiness_service=None, request_bounds: RequestBounds | None = None, cursor_issuer=None, + contract_version: str = "2.0.0-shadow", + authority: str = "SHADOW", ) -> FastAPI: - app = FastAPI(title="Quant Data Layer V2", version="2.0.0-shadow") + if not contract_version.startswith("2.0.0") or authority not in { + "SHADOW", "INTERNAL_STABLE", "PRIMARY" + }: + raise ValueError("V2 app contract version/authority is invalid") + app = FastAPI(title="Quant Data Layer V2", version=contract_version) app.state.v2_query_service = service app.state.v2_identity_service = identity_service app.state.v2_runtime_readiness = readiness_service or FailClosedReadiness() @@ -684,7 +713,8 @@ def create_v2_app( "role": "api_v2", "owns_live_ingestion": False, "owns_venue_connections": False, - "authority": "SHADOW", + "authority": authority, + "contract_version": contract_version, } app.include_router(router) if request_bounds is not None: diff --git a/qdl/canary/__init__.py b/qdl/canary/__init__.py index 07fe466..fcf7acf 100644 --- a/qdl/canary/__init__.py +++ b/qdl/canary/__init__.py @@ -1,13 +1,97 @@ -from qdl.canary.consumer import ( - CanaryRunResult, - DeterministicPaperSignalState, - PaperAlphaCanary, - sdk_requirement, +from qdl.canary.phase92 import ( + AcceptedHandoff, + HandoffDirection, + PrimaryAuthorization, + PrimaryAuthorizationMode, + ProductionPrimaryAuthorizer, + TerminalOwnerCheckpoint, +) +from qdl.canary.phase93 import ( + AuthorityRegistrySnapshot, + ClosureApproval, + ClosureAuthorization, + ConsumerCheckpoint, + ConsumerRegistrySnapshot, + DecommissionDecision, + DecommissionRequest, + ExpansionManifest, + ExpansionType, + HoldScope, + HoldStatus, + PrimaryHoldDecision, + PrimaryHoldEvaluator, + PrimaryHoldIdentity, + PrimaryHoldObservation, + PrimaryHoldPolicy, + ProductionClosureAuthorizer, + RollbackRehearsalEvidence, + RollbackWindowClosure, + assess_decommission, +) +from qdl.canary.phase9 import ( + CanaryAuthorization, + CanaryAuthorizationMode, + CanaryGuardrailDecision, + CanaryGuardrailEngine, + CanaryGuardrailPolicy, + CanaryObservation, + ProductionCanaryAuthorizer, ) __all__ = [ + "AcceptedHandoff", + "AuthorityRegistrySnapshot", + "ClosureApproval", + "ClosureAuthorization", + "ConsumerCheckpoint", + "ConsumerRegistrySnapshot", + "DecommissionDecision", + "DecommissionRequest", + "ExpansionManifest", + "ExpansionType", + "HoldScope", + "HoldStatus", + "PrimaryHoldDecision", + "PrimaryHoldEvaluator", + "PrimaryHoldIdentity", + "PrimaryHoldObservation", + "PrimaryHoldPolicy", + "ProductionClosureAuthorizer", + "RollbackRehearsalEvidence", + "RollbackWindowClosure", + "assess_decommission", + "HandoffDirection", + "PrimaryAuthorization", + "PrimaryAuthorizationMode", + "ProductionPrimaryAuthorizer", + "TerminalOwnerCheckpoint", + "CanaryAuthorization", + "CanaryAuthorizationMode", + "CanaryGuardrailDecision", + "CanaryGuardrailEngine", + "CanaryGuardrailPolicy", + "CanaryObservation", "CanaryRunResult", "DeterministicPaperSignalState", "PaperAlphaCanary", + "ProductionCanaryAuthorizer", "sdk_requirement", ] + + +_CONSUMER_EXPORTS = { + "CanaryRunResult", + "DeterministicPaperSignalState", + "PaperAlphaCanary", + "sdk_requirement", +} + + +def __getattr__(name: str): + if name not in _CONSUMER_EXPORTS: + raise AttributeError(name) + from qdl.canary import consumer + + value = getattr(consumer, name) + globals()[name] = value + return value diff --git a/qdl/canary/phase9.py b/qdl/canary/phase9.py new file mode 100644 index 0000000..40fd6c1 --- /dev/null +++ b/qdl/canary/phase9.py @@ -0,0 +1,272 @@ +from __future__ import annotations + +import uuid +from dataclasses import dataclass +from enum import StrEnum +from typing import Any, Mapping + +from qdl.certification.prerequisites import CandidateSlice + + +_REQUIRED_PRODUCTION_GATES = frozenset({ + "replicated_durable_transport", + "production_observability", + "workload_identity_rbac_network", + "external_secret_rotation", + "signed_artifact_admission", + "postgres_pitr", + "object_store_restore", + "independent_failure_domain_dr", + "redis_projector_rebuild", + "consumer_registration_rollback", + "persistent_authority_sink_fencing", + "exact_slice_approval", +}) + + +class CanaryAuthorizationMode(StrEnum): + PRODUCTION = "PRODUCTION" + ISOLATED_REHEARSAL = "ISOLATED_REHEARSAL" + + +@dataclass(frozen=True, slots=True) +class CanaryAuthorization: + allowed: bool + production_authorized: bool + mode: CanaryAuthorizationMode + reason: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str | None + + +class ProductionCanaryAuthorizer: + """Fail-closed Phase 9.1 gate; it never mutates authority state.""" + + def __init__(self, *, max_decision_age_seconds: int = 900) -> None: + if max_decision_age_seconds <= 0: + raise ValueError("canary decision age must be positive") + self.max_decision_age_ns = max_decision_age_seconds * 1_000_000_000 + + def authorize( + self, + *, + candidate: CandidateSlice, + decision: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> CanaryAuthorization: + reason = self._validate( + candidate=candidate, + decision=decision, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + return CanaryAuthorization( + allowed=reason == "AUTHORIZED", + production_authorized=reason == "AUTHORIZED", + mode=CanaryAuthorizationMode.PRODUCTION, + reason=reason, + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=str(decision.get("bundle_id") or "") or None, + ) + + @staticmethod + def authorize_isolated_rehearsal(*, candidate: CandidateSlice) -> CanaryAuthorization: + return CanaryAuthorization( + allowed=True, + production_authorized=False, + mode=CanaryAuthorizationMode.ISOLATED_REHEARSAL, + reason="ISOLATED_REHEARSAL_ONLY", + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=None, + ) + + def _validate( + self, + *, + candidate: CandidateSlice, + decision: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> str: + if now_ns <= 0: + return "INVALID_CLOCK" + if candidate.payload["authority_state"] != "RUST_SHADOW": + return "CANDIDATE_NOT_SHADOW" + if candidate.payload["public_write_allowed"] or candidate.payload["legacy_write_allowed"]: + return "CANDIDATE_WRITE_AUTHORITY_INVALID" + if decision.get("schema") != "qdl.production-prerequisite-decision.v1": + return "DECISION_SCHEMA_INVALID" + if decision.get("decision") != "GO": + return "PREREQUISITE_DECISION_NOT_GO" + if decision.get("candidate_digest") != candidate.digest: + return "CANDIDATE_DIGEST_MISMATCH" + if decision.get("slice_id") != candidate.payload["slice_id"]: + return "SLICE_MISMATCH" + bundle_id = str(decision.get("bundle_id") or "") + try: + uuid.UUID(bundle_id) + uuid.UUID(expected_bundle_id) + except ValueError: + return "PREREQUISITE_BUNDLE_INVALID" + if bundle_id != expected_bundle_id: + return "PREREQUISITE_BUNDLE_MISMATCH" + issued_at_ns = decision.get("issued_at_ns") + if not isinstance(issued_at_ns, int) or isinstance(issued_at_ns, bool) or issued_at_ns <= 0: + return "DECISION_TIME_INVALID" + if issued_at_ns > now_ns + 60_000_000_000: + return "DECISION_FROM_FUTURE" + if now_ns - issued_at_ns > self.max_decision_age_ns: + return "DECISION_EXPIRED" + gates = decision.get("gates") + if not isinstance(gates, list) or not gates: + return "GATE_RESULTS_MISSING" + gate_ids = [item.get("gate_id") for item in gates if isinstance(item, Mapping)] + if ( + len(gate_ids) != len(gates) + or len(gate_ids) != len(set(gate_ids)) + or set(gate_ids) != _REQUIRED_PRODUCTION_GATES + ): + return "GATE_RESULTS_INVALID" + if any(item.get("passed") is not True or item.get("reason") != "PASS" for item in gates): + return "GATE_NOT_PASSED" + if decision.get("passed") != len(gates) or decision.get("blocked") != 0: + return "GATE_SUMMARY_INVALID" + if decision.get("authority_state") != "RUST_SHADOW": + return "AUTHORITY_PRECONDITION_INVALID" + if decision.get("v1_unchanged") is not True: + return "V1_PRECONDITION_INVALID" + if decision.get("production_mutations") != 0: + return "PRODUCTION_MUTATION_DETECTED" + return "AUTHORIZED" + + +@dataclass(frozen=True, slots=True) +class CanaryGuardrailPolicy: + max_lag_ms: int + max_freshness_ms: int + max_cpu_percent: float + max_rss_mb: float + max_queue_depth: int + hold_down_seconds: int + + def __post_init__(self) -> None: + if any(value <= 0 for value in ( + self.max_lag_ms, + self.max_freshness_ms, + self.max_cpu_percent, + self.max_rss_mb, + self.max_queue_depth, + self.hold_down_seconds, + )): + raise ValueError("canary guardrail thresholds must be positive") + + +@dataclass(frozen=True, slots=True) +class CanaryObservation: + observed_at_ns: int + semantic_mismatches: int = 0 + open_gaps: int = 0 + duplicate_external_writes: int = 0 + stale_writer_attempts: int = 0 + authority_ambiguities: int = 0 + durable_ack_failures: int = 0 + lag_ms: int = 0 + freshness_ms: int = 0 + cpu_percent: float = 0.0 + rss_mb: float = 0.0 + queue_depth: int = 0 + + def __post_init__(self) -> None: + values = ( + self.observed_at_ns, + self.semantic_mismatches, + self.open_gaps, + self.duplicate_external_writes, + self.stale_writer_attempts, + self.authority_ambiguities, + self.durable_ack_failures, + self.lag_ms, + self.freshness_ms, + self.cpu_percent, + self.rss_mb, + self.queue_depth, + ) + if any(value < 0 for value in values) or self.observed_at_ns <= 0: + raise ValueError("canary observation values are invalid") + + +@dataclass(frozen=True, slots=True) +class CanaryGuardrailDecision: + allowed: bool + reason: str + blocked_at_ns: int | None + hold_until_ns: int | None + + +class CanaryGuardrailEngine: + def __init__(self, policy: CanaryGuardrailPolicy) -> None: + self.policy = policy + self._blocked_at_ns: int | None = None + self._hold_until_ns: int | None = None + self._first_failure_reason: str | None = None + + def evaluate(self, observation: CanaryObservation) -> CanaryGuardrailDecision: + if self._first_failure_reason is not None: + return self._decision(False, "EXPLICIT_RESET_REQUIRED") + reason = self._failure_reason(observation) + if reason is not None: + self._first_failure_reason = reason + self._blocked_at_ns = observation.observed_at_ns + self._hold_until_ns = ( + observation.observed_at_ns + + self.policy.hold_down_seconds * 1_000_000_000 + ) + return self._decision(False, reason) + return self._decision(True, "PASS") + + def reset_after_hold(self, observation: CanaryObservation) -> CanaryGuardrailDecision: + if self._first_failure_reason is None: + return self.evaluate(observation) + assert self._hold_until_ns is not None + if observation.observed_at_ns < self._hold_until_ns: + return self._decision(False, "HOLD_DOWN_ACTIVE") + reason = self._failure_reason(observation) + if reason is not None: + return self._decision(False, reason) + self._blocked_at_ns = None + self._hold_until_ns = None + self._first_failure_reason = None + return self._decision(True, "RESET_CONFIRMED") + + def _failure_reason(self, item: CanaryObservation) -> str | None: + zero_tolerance = ( + (item.semantic_mismatches, "SEMANTIC_MISMATCH"), + (item.open_gaps, "OPEN_GAP"), + (item.duplicate_external_writes, "DUPLICATE_EXTERNAL_WRITE"), + (item.stale_writer_attempts, "STALE_WRITER_ATTEMPT"), + (item.authority_ambiguities, "AUTHORITY_AMBIGUITY"), + (item.durable_ack_failures, "DURABLE_ACK_FAILURE"), + ) + for value, reason in zero_tolerance: + if value: + return reason + thresholds = ( + (item.lag_ms > self.policy.max_lag_ms, "LAG_THRESHOLD_EXCEEDED"), + (item.freshness_ms > self.policy.max_freshness_ms, "FRESHNESS_THRESHOLD_EXCEEDED"), + (item.cpu_percent > self.policy.max_cpu_percent, "CPU_THRESHOLD_EXCEEDED"), + (item.rss_mb > self.policy.max_rss_mb, "RSS_THRESHOLD_EXCEEDED"), + (item.queue_depth > self.policy.max_queue_depth, "QUEUE_THRESHOLD_EXCEEDED"), + ) + return next((reason for failed, reason in thresholds if failed), None) + + def _decision(self, allowed: bool, reason: str) -> CanaryGuardrailDecision: + return CanaryGuardrailDecision( + allowed=allowed, + reason=reason, + blocked_at_ns=self._blocked_at_ns, + hold_until_ns=self._hold_until_ns, + ) diff --git a/qdl/canary/phase92.py b/qdl/canary/phase92.py new file mode 100644 index 0000000..92fc20f --- /dev/null +++ b/qdl/canary/phase92.py @@ -0,0 +1,335 @@ +from __future__ import annotations + +import hashlib +import json +import uuid +from dataclasses import asdict, dataclass +from enum import StrEnum +from typing import Any, Mapping + +from qdl.canary.phase9 import ProductionCanaryAuthorizer +from qdl.certification.prerequisites import CandidateSlice + + +def _canonical_digest(value: Mapping[str, Any]) -> str: + payload = json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return hashlib.sha256(payload).hexdigest() + + +def _valid_digest(value: str) -> bool: + return len(value) == 64 and all(char in "0123456789abcdef" for char in value) + + +class PrimaryAuthorizationMode(StrEnum): + PRODUCTION = "PRODUCTION" + ISOLATED_REHEARSAL = "ISOLATED_REHEARSAL" + + +class HandoffDirection(StrEnum): + PYTHON_TO_RUST = "PYTHON_TO_RUST" + RUST_TO_PYTHON = "RUST_TO_PYTHON" + + +@dataclass(frozen=True, slots=True) +class PrimaryAuthorization: + allowed: bool + production_authorized: bool + mode: PrimaryAuthorizationMode + reason: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str | None + + +@dataclass(frozen=True, slots=True) +class TerminalOwnerCheckpoint: + schema: str + checkpoint_id: str + slice_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + source_session_id: str + connection_generation: int + terminal_watermark: int + terminal_event_id: str + terminal_payload_sha256: str + candidate_digest: str + committed_at_ns: int + + def __post_init__(self) -> None: + try: + uuid.UUID(self.checkpoint_id) + except ValueError as error: + raise ValueError("checkpoint_id must be a UUID") from error + if self.schema != "qdl.terminal-owner-checkpoint.v1": + raise ValueError("terminal checkpoint schema is invalid") + text_fields = ( + self.slice_id, + self.owner_id, + self.source_session_id, + self.terminal_event_id, + ) + if any(not value.strip() for value in text_fields): + raise ValueError("terminal checkpoint identity is incomplete") + positive = ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.connection_generation, + self.committed_at_ns, + ) + if any(value <= 0 for value in positive) or self.terminal_watermark < 0: + raise ValueError("terminal checkpoint epoch/watermark is invalid") + if not _valid_digest(self.terminal_payload_sha256): + raise ValueError("terminal payload digest is invalid") + if not _valid_digest(self.candidate_digest): + raise ValueError("terminal candidate digest is invalid") + + @property + def digest(self) -> str: + return _canonical_digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class AcceptedHandoff: + schema: str + handoff_id: str + direction: HandoffDirection + checkpoint_digest: str + slice_id: str + old_owner_id: str + new_owner_id: str + expected_state: str + new_state: str + expected_authority_revision: int + new_authority_revision: int + expected_lease_epoch: int + new_lease_epoch: int + partition_plan_epoch: int + terminal_watermark: int + first_new_watermark: int + overlap_start_watermark: int + overlap_end_watermark: int + old_event_count: int + new_event_count: int + semantic_mismatches: int + open_gaps: int + candidate_digest: str + prerequisite_bundle_id: str + approved_by: str + approved_at_ns: int + expires_at_ns: int + + def __post_init__(self) -> None: + for value, label in ( + (self.handoff_id, "handoff_id"), + (self.prerequisite_bundle_id, "prerequisite_bundle_id"), + ): + try: + uuid.UUID(value) + except ValueError as error: + raise ValueError(f"{label} must be a UUID") from error + if self.schema != "qdl.accepted-authority-handoff.v1": + raise ValueError("handoff schema is invalid") + if any( + not value.strip() + for value in ( + self.slice_id, + self.old_owner_id, + self.new_owner_id, + self.expected_state, + self.new_state, + self.approved_by, + ) + ): + raise ValueError("handoff identity is incomplete") + expected_states = { + HandoffDirection.PYTHON_TO_RUST: ("RUST_CANARY", "RUST_PRIMARY"), + HandoffDirection.RUST_TO_PYTHON: ( + "ROLLBACK_PENDING", + "PYTHON_PRIMARY", + ), + } + if (self.expected_state, self.new_state) != expected_states[self.direction]: + raise ValueError("handoff direction/state pair is invalid") + if self.old_owner_id == self.new_owner_id: + raise ValueError("handoff must change owner") + if self.new_authority_revision != self.expected_authority_revision + 1: + raise ValueError("handoff authority revision must advance exactly one") + if self.new_lease_epoch <= self.expected_lease_epoch: + raise ValueError("handoff owner requires a newer lease epoch") + if self.partition_plan_epoch <= 0 or self.terminal_watermark < 0: + raise ValueError("handoff plan/watermark is invalid") + if self.first_new_watermark != self.terminal_watermark + 1: + raise ValueError("handoff first watermark must equal terminal + 1") + if ( + self.overlap_start_watermark < 0 + or self.overlap_start_watermark > self.overlap_end_watermark + or self.overlap_end_watermark != self.terminal_watermark + ): + raise ValueError("handoff reconciliation range is invalid") + if ( + self.old_event_count <= 0 + or self.old_event_count != self.new_event_count + or self.semantic_mismatches != 0 + or self.open_gaps != 0 + ): + raise ValueError("handoff reconciliation is not clean") + if not _valid_digest(self.checkpoint_digest) or not _valid_digest( + self.candidate_digest + ): + raise ValueError("handoff digest is invalid") + if self.approved_at_ns <= 0 or self.expires_at_ns <= self.approved_at_ns: + raise ValueError("handoff approval window is invalid") + + @property + def digest(self) -> str: + payload = asdict(self) + payload["direction"] = self.direction.value + return _canonical_digest(payload) + + +class ProductionPrimaryAuthorizer: + """Fail-closed Phase 9.2 gate. It does not mutate authority.""" + + def __init__(self, *, max_decision_age_seconds: int = 900) -> None: + self._canary_authorizer = ProductionCanaryAuthorizer( + max_decision_age_seconds=max_decision_age_seconds + ) + + @staticmethod + def authorize_isolated_rehearsal( + *, candidate: CandidateSlice + ) -> PrimaryAuthorization: + return PrimaryAuthorization( + allowed=True, + production_authorized=False, + mode=PrimaryAuthorizationMode.ISOLATED_REHEARSAL, + reason="ISOLATED_REHEARSAL_ONLY", + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=None, + ) + + def authorize( + self, + *, + candidate: CandidateSlice, + prerequisite_decision: Mapping[str, Any], + canary_evidence: Mapping[str, Any], + approval: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> PrimaryAuthorization: + canary_auth = self._canary_authorizer.authorize( + candidate=candidate, + decision=prerequisite_decision, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + reason = canary_auth.reason + if canary_auth.allowed: + reason = self._validate_canary( + candidate=candidate, + canary=canary_evidence, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + if reason == "AUTHORIZED": + reason = self._validate_approval( + candidate=candidate, + approval=approval, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + return PrimaryAuthorization( + allowed=reason == "AUTHORIZED", + production_authorized=reason == "AUTHORIZED", + mode=PrimaryAuthorizationMode.PRODUCTION, + reason=reason, + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=expected_bundle_id or None, + ) + + @staticmethod + def _validate_canary( + *, + candidate: CandidateSlice, + canary: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> str: + if canary.get("schema") != "qdl.phase91.rust-canary-certification.v1": + return "CANARY_EVIDENCE_SCHEMA_INVALID" + if canary.get("status") != "PRODUCTION_CANARY_HOLD_PASSED": + return "CANARY_HOLD_NOT_PASSED" + if canary.get("production_authorized") is not True: + return "CANARY_NOT_PRODUCTION_AUTHORIZED" + if canary.get("slice_id") != candidate.payload["slice_id"]: + return "CANARY_SLICE_MISMATCH" + if canary.get("candidate_digest") != candidate.digest: + return "CANARY_CANDIDATE_MISMATCH" + if canary.get("prerequisite_bundle_id") != expected_bundle_id: + return "CANARY_BUNDLE_MISMATCH" + if canary.get("python_v1_public_authority_unchanged") is not True: + return "CANARY_V1_PRECONDITION_INVALID" + if canary.get("production_mutations") != 0: + return "CANARY_PRODUCTION_MUTATION_DETECTED" + parity = canary.get("parity") + broker = canary.get("broker") + if not isinstance(parity, Mapping) or parity.get("semantic_mismatches") != 0: + return "CANARY_PARITY_INVALID" + if not isinstance(broker, Mapping): + return "CANARY_BROKER_EVIDENCE_MISSING" + if broker.get("final_authority") != "RUST_CANARY": + return "CANARY_AUTHORITY_NOT_HELD" + if broker.get("public_writes") != 0 or broker.get("legacy_writes") != 0: + return "CANARY_EXTERNAL_WRITE_DETECTED" + completed = canary.get("hold_completed_at_ns") + if ( + not isinstance(completed, int) + or isinstance(completed, bool) + or completed <= 0 + or completed > now_ns + ): + return "CANARY_HOLD_TIME_INVALID" + rollback_digest = canary.get("rollback_manifest_digest") + if not isinstance(rollback_digest, str) or not _valid_digest(rollback_digest): + return "CANARY_ROLLBACK_MANIFEST_INVALID" + return "AUTHORIZED" + + @staticmethod + def _validate_approval( + *, + candidate: CandidateSlice, + approval: Mapping[str, Any], + expected_bundle_id: str, + now_ns: int, + ) -> str: + if approval.get("schema") != "qdl.primary-slice-approval.v1": + return "PRIMARY_APPROVAL_SCHEMA_INVALID" + if approval.get("decision") != "APPROVE": + return "PRIMARY_NOT_APPROVED" + if approval.get("slice_id") != candidate.payload["slice_id"]: + return "PRIMARY_APPROVAL_SLICE_MISMATCH" + if approval.get("candidate_digest") != candidate.digest: + return "PRIMARY_APPROVAL_CANDIDATE_MISMATCH" + if approval.get("prerequisite_bundle_id") != expected_bundle_id: + return "PRIMARY_APPROVAL_BUNDLE_MISMATCH" + if not str(approval.get("operator") or "").strip(): + return "PRIMARY_APPROVER_MISSING" + if not str(approval.get("change_ticket") or "").strip(): + return "PRIMARY_CHANGE_TICKET_MISSING" + if approval.get("max_partitions") != 1: + return "PRIMARY_BLAST_RADIUS_INVALID" + if approval.get("allow_disable_exact_python_slice") is not True: + return "PRIMARY_PYTHON_HANDOFF_NOT_APPROVED" + expires = approval.get("expires_at_ns") + if not isinstance(expires, int) or isinstance(expires, bool) or expires <= now_ns: + return "PRIMARY_APPROVAL_EXPIRED" + return "AUTHORIZED" diff --git a/qdl/canary/phase93.py b/qdl/canary/phase93.py new file mode 100644 index 0000000..6b536f8 --- /dev/null +++ b/qdl/canary/phase93.py @@ -0,0 +1,1186 @@ +from __future__ import annotations + +import hashlib +import json +import uuid +from dataclasses import asdict, dataclass +from enum import StrEnum +from typing import Any, Mapping + +from qdl.canary.phase9 import ProductionCanaryAuthorizer +from qdl.certification.prerequisites import CandidateSlice + + +def _digest(value: Mapping[str, Any]) -> str: + payload = json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return hashlib.sha256(payload).hexdigest() + + +def _valid_digest(value: object) -> bool: + return isinstance(value, str) and len(value) == 64 and all( + char in "0123456789abcdef" for char in value + ) + + +def _valid_uuid(value: object) -> bool: + try: + uuid.UUID(str(value)) + except (ValueError, TypeError, AttributeError): + return False + return True + + +def _positive_int(value: object) -> bool: + return isinstance(value, int) and not isinstance(value, bool) and value > 0 + + +def _non_negative_int(value: object) -> bool: + return isinstance(value, int) and not isinstance(value, bool) and value >= 0 + + +def _non_negative_number(value: object) -> bool: + return ( + isinstance(value, (int, float)) + and not isinstance(value, bool) + and value >= 0 + ) + + +class HoldScope(StrEnum): + TEST_REHEARSAL = "TEST_REHEARSAL" + PRODUCTION = "PRODUCTION" + + +class HoldStatus(StrEnum): + IN_PROGRESS = "IN_PROGRESS" + PASSED = "PASSED" + BLOCKED = "BLOCKED" + + +class ExpansionType(StrEnum): + INSTRUMENT_PARTITION = "INSTRUMENT_PARTITION" + BBO = "BBO" + L2_BOOK = "L2_BOOK" + BAR_LIFECYCLE = "BAR_LIFECYCLE" + VENUE_MARKET = "VENUE_MARKET" + + +_EXPANSION_GATES: dict[ExpansionType, frozenset[str]] = { + ExpansionType.INSTRUMENT_PARTITION: frozenset({ + "authority_handoff", "capacity_headroom", "exact_frame_parity", + "partition_churn", "provider_authentic_source", "rollback", + "source_capacity", + }), + ExpansionType.BBO: frozenset({ + "authority_handoff", "capacity_headroom", "coalescing_policy", + "exact_frame_parity", "freshness", "ordering_reconnect", + "provider_authentic_source", "quote_identity", "rollback", + }), + ExpansionType.L2_BOOK: frozenset({ + "authority_handoff", "capacity_headroom", "checksum", + "exact_frame_parity", "lossless_backpressure", + "provider_authentic_source", "resync", "rollback", + "snapshot_delta_sequence", + }), + ExpansionType.BAR_LIFECYCLE: frozenset({ + "authority_handoff", "capacity_headroom", "close_time_semantics", + "exact_frame_parity", "final_revision_lineage", + "provider_authentic_source", "replay", "rollback", + }), + ExpansionType.VENUE_MARKET: frozenset({ + "adapter_capability", "authority_handoff", "capacity_headroom", + "disaster_recovery", "entitlement", "exact_frame_parity", + "instrument_identity", "provider_authentic_source", + "provider_semantics", "rollback", + }), +} + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldPolicy: + minimum_duration_seconds: int + sample_interval_seconds: int + max_sample_gap_seconds: int + max_lag_ms: int + max_freshness_ms: int + max_queue_depth: int + max_spool_bytes: int + max_cpu_percent: float + max_rss_mb: float + + def __post_init__(self) -> None: + integers = ( + self.minimum_duration_seconds, + self.sample_interval_seconds, + self.max_sample_gap_seconds, + self.max_lag_ms, + self.max_freshness_ms, + self.max_queue_depth, + self.max_spool_bytes, + ) + if any(not _positive_int(value) for value in integers): + raise ValueError("hold policy integer thresholds must be positive") + if self.max_sample_gap_seconds < self.sample_interval_seconds: + raise ValueError("hold maximum sample gap cannot be below sample interval") + if self.minimum_duration_seconds < self.max_sample_gap_seconds: + raise ValueError("hold duration must cover one maximum sample gap") + if self.max_cpu_percent <= 0 or self.max_cpu_percent > 100: + raise ValueError("hold CPU threshold is invalid") + if self.max_rss_mb <= 0: + raise ValueError("hold RSS threshold is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldIdentity: + schema: str + hold_id: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + started_at_ns: int + required_until_ns: int + policy_digest: str + + def __post_init__(self) -> None: + if self.schema != "qdl.primary-hold.v1": + raise ValueError("hold schema is invalid") + if not _valid_uuid(self.hold_id) or not _valid_uuid( + self.prerequisite_bundle_id + ): + raise ValueError("hold UUID identity is invalid") + if not self.slice_id.strip() or not self.owner_id.strip(): + raise ValueError("hold owner/slice identity is required") + if not _valid_digest(self.candidate_digest) or not _valid_digest( + self.policy_digest + ): + raise ValueError("hold digest identity is invalid") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.started_at_ns, + self.required_until_ns, + ) + ) or self.required_until_ns <= self.started_at_ns: + raise ValueError("hold epoch/time identity is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldObservation: + schema: str + observation_id: str + hold_id: str + slice_id: str + candidate_digest: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + sequence: int + observed_at_ns: int + last_watermark: int + semantic_mismatches: int = 0 + open_gaps: int = 0 + duplicate_external_writes: int = 0 + accepted_stale_writer_writes: int = 0 + authority_ambiguities: int = 0 + durable_ack_failures: int = 0 + projection_mismatches: int = 0 + consumer_checkpoint_regressions: int = 0 + unexplained_quality_failures: int = 0 + lag_ms: int = 0 + freshness_ms: int = 0 + queue_depth: int = 0 + spool_bytes: int = 0 + cpu_percent: float = 0.0 + rss_mb: float = 0.0 + registered_consumers: int = 0 + healthy_consumers: int = 0 + checkpoint_watermark: int = 0 + + def __post_init__(self) -> None: + if self.schema != "qdl.primary-hold-observation.v1": + raise ValueError("hold observation schema is invalid") + if not _valid_uuid(self.observation_id) or not _valid_uuid(self.hold_id): + raise ValueError("hold observation UUID is invalid") + if ( + not self.slice_id.strip() + or not self.owner_id.strip() + or not _valid_digest(self.candidate_digest) + ): + raise ValueError("hold observation identity is invalid") + positive = ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.sequence, + self.observed_at_ns, + ) + if any(not _positive_int(value) for value in positive): + raise ValueError("hold observation epoch/sequence/time is invalid") + integer_values = ( + self.last_watermark, + self.semantic_mismatches, + self.open_gaps, + self.duplicate_external_writes, + self.accepted_stale_writer_writes, + self.authority_ambiguities, + self.durable_ack_failures, + self.projection_mismatches, + self.consumer_checkpoint_regressions, + self.unexplained_quality_failures, + self.lag_ms, + self.freshness_ms, + self.queue_depth, + self.spool_bytes, + self.registered_consumers, + self.healthy_consumers, + self.checkpoint_watermark, + ) + if any(not _non_negative_int(value) for value in integer_values) or any( + not _non_negative_number(value) + for value in (self.cpu_percent, self.rss_mb) + ): + raise ValueError("hold observation values must be non-negative") + if self.healthy_consumers > self.registered_consumers: + raise ValueError("healthy consumer count exceeds registry count") + + +@dataclass(frozen=True, slots=True) +class PrimaryHoldDecision: + schema: str + decision_id: str + hold_id: str + status: HoldStatus + reason: str + scope: HoldScope + production_authorized: bool + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + policy_digest: str + first_observed_at_ns: int | None + last_observed_at_ns: int | None + observation_count: int + terminal_watermark: int | None + decided_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.primary-hold-decision.v1": + raise ValueError("hold decision schema is invalid") + if not isinstance(self.status, HoldStatus) or not isinstance( + self.scope, HoldScope + ): + raise ValueError("hold decision status/scope is invalid") + if not _valid_uuid(self.decision_id) or not _valid_uuid(self.hold_id): + raise ValueError("hold decision UUID is invalid") + if ( + not self.reason.strip() + or not self.slice_id.strip() + or not self.owner_id.strip() + ): + raise ValueError("hold decision identity/reason is required") + if not _valid_uuid(self.prerequisite_bundle_id): + raise ValueError("hold decision prerequisite bundle is invalid") + if not _valid_digest(self.candidate_digest) or not _valid_digest( + self.policy_digest + ): + raise ValueError("hold decision digest is invalid") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.decided_at_ns, + ) + ) or not _non_negative_int(self.observation_count): + raise ValueError("hold decision epoch/count/time is invalid") + if self.production_authorized != ( + self.scope == HoldScope.PRODUCTION and self.status == HoldStatus.PASSED + ): + raise ValueError("hold decision production authorization is inconsistent") + summary = ( + self.first_observed_at_ns, + self.last_observed_at_ns, + self.terminal_watermark, + ) + if self.observation_count == 0 and any(value is not None for value in summary): + raise ValueError("empty hold decision cannot expose observation state") + if self.observation_count > 0 and ( + self.first_observed_at_ns is None + or self.last_observed_at_ns is None + or self.terminal_watermark is None + or self.first_observed_at_ns <= 0 + or self.last_observed_at_ns < self.first_observed_at_ns + or self.terminal_watermark < 0 + ): + raise ValueError("hold decision observation summary is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +class PrimaryHoldEvaluator: + _ZERO_TOLERANCE = ( + ("semantic_mismatches", "SEMANTIC_MISMATCH"), + ("open_gaps", "OPEN_GAP"), + ("duplicate_external_writes", "DUPLICATE_EXTERNAL_WRITE"), + ("accepted_stale_writer_writes", "ACCEPTED_STALE_WRITER_WRITE"), + ("authority_ambiguities", "AUTHORITY_AMBIGUITY"), + ("durable_ack_failures", "DURABLE_ACK_FAILURE"), + ("projection_mismatches", "PROJECTION_MISMATCH"), + ("consumer_checkpoint_regressions", "CONSUMER_CHECKPOINT_REGRESSION"), + ("unexplained_quality_failures", "UNEXPLAINED_QUALITY_FAILURE"), + ) + + def __init__( + self, + *, + identity: PrimaryHoldIdentity, + policy: PrimaryHoldPolicy, + scope: HoldScope, + ) -> None: + if identity.policy_digest != policy.digest: + raise ValueError("hold identity does not bind the policy") + duration = identity.required_until_ns - identity.started_at_ns + if duration < policy.minimum_duration_seconds * 1_000_000_000: + raise ValueError("hold identity duration is below policy") + self.identity = identity + self.policy = policy + self.scope = scope + self._observations: list[PrimaryHoldObservation] = [] + self._blocked_reason: str | None = None + + def observe(self, item: PrimaryHoldObservation) -> str: + if self._blocked_reason is not None: + return "HOLD_ALREADY_BLOCKED" + reason = self._validate_observation(item) + self._observations.append(item) + if reason != "PASS": + self._blocked_reason = reason + return reason + + def _validate_observation(self, item: PrimaryHoldObservation) -> str: + expected = ( + self.identity.hold_id, + self.identity.slice_id, + self.identity.candidate_digest, + self.identity.owner_id, + self.identity.authority_revision, + self.identity.lease_epoch, + self.identity.partition_plan_epoch, + ) + actual = ( + item.hold_id, + item.slice_id, + item.candidate_digest, + item.owner_id, + item.authority_revision, + item.lease_epoch, + item.partition_plan_epoch, + ) + if actual != expected: + return "HOLD_AUTHORITY_IDENTITY_CHANGED" + if item.sequence != len(self._observations) + 1: + return "HOLD_SEQUENCE_NOT_CONTIGUOUS" + previous_time = ( + self._observations[-1].observed_at_ns + if self._observations + else self.identity.started_at_ns + ) + if item.observed_at_ns <= previous_time: + return "HOLD_OBSERVATION_TIME_NOT_MONOTONIC" + if ( + item.observed_at_ns - previous_time + > self.policy.max_sample_gap_seconds * 1_000_000_000 + ): + return "HOLD_OBSERVATION_GAP_EXCEEDED" + if ( + self._observations + and item.last_watermark < self._observations[-1].last_watermark + ): + return "HOLD_WATERMARK_REGRESSED" + for field, reason in self._ZERO_TOLERANCE: + if getattr(item, field) != 0: + return reason + thresholds = ( + (item.lag_ms > self.policy.max_lag_ms, "LAG_THRESHOLD_EXCEEDED"), + ( + item.freshness_ms > self.policy.max_freshness_ms, + "FRESHNESS_THRESHOLD_EXCEEDED", + ), + ( + item.queue_depth > self.policy.max_queue_depth, + "QUEUE_THRESHOLD_EXCEEDED", + ), + ( + item.spool_bytes > self.policy.max_spool_bytes, + "SPOOL_THRESHOLD_EXCEEDED", + ), + ( + item.cpu_percent > self.policy.max_cpu_percent, + "CPU_THRESHOLD_EXCEEDED", + ), + (item.rss_mb > self.policy.max_rss_mb, "RSS_THRESHOLD_EXCEEDED"), + ) + threshold_reason = next((reason for failed, reason in thresholds if failed), None) + if threshold_reason is not None: + return threshold_reason + if item.registered_consumers <= 0: + return "CONSUMER_REGISTRY_EMPTY" + if item.healthy_consumers != item.registered_consumers: + return "CONSUMER_NOT_HEALTHY" + if item.checkpoint_watermark < item.last_watermark: + return "CONSUMER_CHECKPOINT_BEHIND" + return "PASS" + + def decision(self, *, decision_id: str, now_ns: int) -> PrimaryHoldDecision: + if not _positive_int(now_ns): + raise ValueError("hold decision clock is invalid") + if self._observations and now_ns < self._observations[-1].observed_at_ns: + raise ValueError("hold decision precedes the latest observation") + status = HoldStatus.IN_PROGRESS + reason = "HOLD_WINDOW_INCOMPLETE" + if self._blocked_reason is not None: + status = HoldStatus.BLOCKED + reason = self._blocked_reason + elif not self._observations: + reason = "HOLD_OBSERVATION_MISSING" + elif now_ns < self.identity.required_until_ns: + reason = "HOLD_WINDOW_INCOMPLETE" + elif self._observations[-1].observed_at_ns < self.identity.required_until_ns: + reason = "HOLD_TERMINAL_OBSERVATION_MISSING" + else: + status = HoldStatus.PASSED + reason = "PASS" + first = self._observations[0] if self._observations else None + last = self._observations[-1] if self._observations else None + return PrimaryHoldDecision( + schema="qdl.primary-hold-decision.v1", + decision_id=decision_id, + hold_id=self.identity.hold_id, + status=status, + reason=reason, + scope=self.scope, + production_authorized=( + status == HoldStatus.PASSED and self.scope == HoldScope.PRODUCTION + ), + slice_id=self.identity.slice_id, + candidate_digest=self.identity.candidate_digest, + prerequisite_bundle_id=self.identity.prerequisite_bundle_id, + owner_id=self.identity.owner_id, + authority_revision=self.identity.authority_revision, + lease_epoch=self.identity.lease_epoch, + partition_plan_epoch=self.identity.partition_plan_epoch, + policy_digest=self.identity.policy_digest, + first_observed_at_ns=first.observed_at_ns if first else None, + last_observed_at_ns=last.observed_at_ns if last else None, + observation_count=len(self._observations), + terminal_watermark=last.last_watermark if last else None, + decided_at_ns=now_ns, + ) + + +@dataclass(frozen=True, slots=True) +class ConsumerCheckpoint: + consumer_id: str + requirement_digest: str + contract_major: int + applied_watermark: int + checkpointed_watermark: int + status: str + migration_status: str + rollback_ready: bool + + def __post_init__(self) -> None: + if not self.consumer_id.strip() or not _valid_digest(self.requirement_digest): + raise ValueError("consumer checkpoint identity is invalid") + if not _positive_int(self.contract_major): + raise ValueError("consumer contract major is invalid") + if ( + not _non_negative_int(self.applied_watermark) + or not _non_negative_int(self.checkpointed_watermark) + or self.checkpointed_watermark > self.applied_watermark + ): + raise ValueError("consumer checkpoint watermark is invalid") + if self.status != "READY" or self.migration_status != "COMPLETE": + raise ValueError("consumer is not closure-ready") + if self.rollback_ready is not True: + raise ValueError("consumer rollback posture is not ready") + + +@dataclass(frozen=True, slots=True) +class ConsumerRegistrySnapshot: + schema: str + snapshot_id: str + slice_id: str + authority_revision: int + checkpoints: tuple[ConsumerCheckpoint, ...] + observed_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.consumer-registry-snapshot.v1": + raise ValueError("consumer registry schema is invalid") + if not _valid_uuid(self.snapshot_id) or not self.slice_id.strip(): + raise ValueError("consumer registry identity is invalid") + if not _positive_int(self.authority_revision) or not _positive_int( + self.observed_at_ns + ): + raise ValueError("consumer registry revision/time is invalid") + if not isinstance(self.checkpoints, tuple): + raise ValueError("consumer registry checkpoints must be immutable") + ids = [item.consumer_id for item in self.checkpoints] + if not ids or len(ids) != len(set(ids)): + raise ValueError("consumer registry must be non-empty and unique") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class AuthorityRegistrySnapshot: + schema: str + snapshot_id: str + slice_id: str + state: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + candidate_digest: str + prerequisite_bundle_id: str + current_watermark: int + public_write_allowed: bool + legacy_write_allowed: bool + observed_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.authority-registry-snapshot.v1": + raise ValueError("authority registry schema is invalid") + if ( + not _valid_uuid(self.snapshot_id) + or not _valid_uuid(self.prerequisite_bundle_id) + or not self.slice_id.strip() + or not self.owner_id.strip() + or not _valid_digest(self.candidate_digest) + ): + raise ValueError("authority registry identity is invalid") + if self.state != "RUST_PRIMARY": + raise ValueError("authority registry is not Rust primary") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.observed_at_ns, + ) + ) or not _non_negative_int(self.current_watermark): + raise ValueError("authority registry epoch/watermark/time is invalid") + if not self.public_write_allowed or not self.legacy_write_allowed: + raise ValueError("primary authority write flags are invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class RollbackRehearsalEvidence: + schema: str + rehearsal_id: str + slice_id: str + candidate_digest: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + rollback_manifest_digest: str + reconciled_through_watermark: int + rto_ms: float + status: str + production_scope: bool + observed_at_ns: int + expires_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.rollback-rehearsal.v1": + raise ValueError("rollback rehearsal schema is invalid") + if ( + not _valid_uuid(self.rehearsal_id) + or not self.slice_id.strip() + or not self.owner_id.strip() + or not _valid_digest(self.candidate_digest) + or not _valid_digest(self.rollback_manifest_digest) + ): + raise ValueError("rollback rehearsal identity is invalid") + if self.status != "PASS": + raise ValueError("rollback rehearsal did not pass") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.observed_at_ns, + self.expires_at_ns, + ) + ) or self.expires_at_ns <= self.observed_at_ns: + raise ValueError("rollback rehearsal epoch/time is invalid") + if not _non_negative_int(self.reconciled_through_watermark) or self.rto_ms <= 0: + raise ValueError("rollback rehearsal watermark/RTO is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class ClosureApproval: + schema: str + approval_id: str + closure_id: str + decision: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + hold_id: str + hold_policy_digest: str + operator: str + change_ticket: str + allow_close_rollback_window: bool + repository_cleanup_approved: bool + approved_at_ns: int + expires_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.rollback-window-closure-approval.v1": + raise ValueError("closure approval schema is invalid") + if any( + not _valid_uuid(value) + for value in ( + self.approval_id, + self.closure_id, + self.prerequisite_bundle_id, + self.hold_id, + ) + ): + raise ValueError("closure approval UUID is invalid") + if ( + self.decision != "APPROVE" + or not self.slice_id.strip() + or not self.operator.strip() + or not self.change_ticket.strip() + or not _valid_digest(self.candidate_digest) + or not _valid_digest(self.hold_policy_digest) + ): + raise ValueError("closure approval identity/decision is invalid") + if self.allow_close_rollback_window is not True: + raise ValueError("closure approval does not allow window close") + if self.repository_cleanup_approved: + raise ValueError("window closure cannot approve repository cleanup") + if ( + not _positive_int(self.approved_at_ns) + or not _positive_int(self.expires_at_ns) + or self.expires_at_ns <= self.approved_at_ns + ): + raise ValueError("closure approval time window is invalid") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class RollbackWindowClosure: + schema: str + closure_id: str + slice_id: str + candidate_digest: str + prerequisite_bundle_id: str + owner_id: str + authority_revision: int + lease_epoch: int + partition_plan_epoch: int + hold_decision_id: str + hold_decision_digest: str + consumer_registry_snapshot_id: str + consumer_registry_digest: str + authority_registry_snapshot_id: str + authority_registry_digest: str + rollback_rehearsal_id: str + rollback_rehearsal_digest: str + approval_id: str + approval_digest: str + operator: str + change_ticket: str + closed_at_ns: int + production_authorized: bool + + def __post_init__(self) -> None: + if self.schema != "qdl.rollback-window-closure.v1": + raise ValueError("rollback closure schema is invalid") + if any( + not _valid_uuid(value) + for value in ( + self.closure_id, + self.prerequisite_bundle_id, + self.hold_decision_id, + self.consumer_registry_snapshot_id, + self.authority_registry_snapshot_id, + self.rollback_rehearsal_id, + self.approval_id, + ) + ): + raise ValueError("rollback closure UUID is invalid") + if ( + not self.slice_id.strip() + or not self.owner_id.strip() + or not self.operator.strip() + or not self.change_ticket.strip() + ): + raise ValueError("rollback closure identity is incomplete") + if any( + not _valid_digest(value) + for value in ( + self.candidate_digest, + self.hold_decision_digest, + self.consumer_registry_digest, + self.authority_registry_digest, + self.rollback_rehearsal_digest, + self.approval_digest, + ) + ): + raise ValueError("rollback closure digest is invalid") + if any( + not _positive_int(value) + for value in ( + self.authority_revision, + self.lease_epoch, + self.partition_plan_epoch, + self.closed_at_ns, + ) + ): + raise ValueError("rollback closure epoch/time is invalid") + if self.production_authorized is not True: + raise ValueError("rollback closure must be production-authorized") + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class ClosureAuthorization: + allowed: bool + reason: str + production_authorized: bool + closure: RollbackWindowClosure | None + + +class ProductionClosureAuthorizer: + def __init__(self, *, max_decision_age_seconds: int = 900) -> None: + self._prerequisite_authorizer = ProductionCanaryAuthorizer( + max_decision_age_seconds=max_decision_age_seconds + ) + + def authorize( + self, + *, + candidate: CandidateSlice, + prerequisite_decision: Mapping[str, Any], + expected_bundle_id: str, + primary_evidence: Mapping[str, Any] | None, + hold_decision: PrimaryHoldDecision | None, + consumer_registry: ConsumerRegistrySnapshot | None, + authority_registry: AuthorityRegistrySnapshot | None, + rollback_evidence: RollbackRehearsalEvidence | None, + approval: ClosureApproval | None, + now_ns: int, + ) -> ClosureAuthorization: + prerequisite = self._prerequisite_authorizer.authorize( + candidate=candidate, + decision=prerequisite_decision, + expected_bundle_id=expected_bundle_id, + now_ns=now_ns, + ) + if not prerequisite.allowed: + return ClosureAuthorization(False, prerequisite.reason, False, None) + if primary_evidence is None: + return ClosureAuthorization(False, "PRIMARY_EVIDENCE_MISSING", False, None) + if hold_decision is None: + return ClosureAuthorization(False, "PRIMARY_HOLD_MISSING", False, None) + if consumer_registry is None or authority_registry is None: + return ClosureAuthorization(False, "REGISTRY_SNAPSHOT_MISSING", False, None) + if rollback_evidence is None: + return ClosureAuthorization(False, "ROLLBACK_REHEARSAL_MISSING", False, None) + if approval is None: + return ClosureAuthorization(False, "CLOSURE_APPROVAL_MISSING", False, None) + reason = self._validate_primary(candidate, expected_bundle_id, primary_evidence) + if reason == "AUTHORIZED": + reason = self._validate_hold(candidate, expected_bundle_id, hold_decision) + if reason == "AUTHORIZED": + reason = self._validate_registries( + hold_decision, + primary_evidence, + consumer_registry, + authority_registry, + now_ns, + ) + if reason == "AUTHORIZED": + reason = self._validate_rollback( + candidate, + hold_decision, + authority_registry, + rollback_evidence, + now_ns, + ) + if reason == "AUTHORIZED": + reason = self._validate_approval( + candidate, expected_bundle_id, hold_decision, approval, now_ns + ) + if reason != "AUTHORIZED": + return ClosureAuthorization(False, reason, False, None) + closure = RollbackWindowClosure( + schema="qdl.rollback-window-closure.v1", + closure_id=approval.closure_id, + slice_id=hold_decision.slice_id, + candidate_digest=hold_decision.candidate_digest, + prerequisite_bundle_id=hold_decision.prerequisite_bundle_id, + owner_id=hold_decision.owner_id, + authority_revision=hold_decision.authority_revision, + lease_epoch=hold_decision.lease_epoch, + partition_plan_epoch=hold_decision.partition_plan_epoch, + hold_decision_id=hold_decision.decision_id, + hold_decision_digest=hold_decision.digest, + consumer_registry_snapshot_id=consumer_registry.snapshot_id, + consumer_registry_digest=consumer_registry.digest, + authority_registry_snapshot_id=authority_registry.snapshot_id, + authority_registry_digest=authority_registry.digest, + rollback_rehearsal_id=rollback_evidence.rehearsal_id, + rollback_rehearsal_digest=rollback_evidence.digest, + approval_id=approval.approval_id, + approval_digest=approval.digest, + operator=approval.operator, + change_ticket=approval.change_ticket, + closed_at_ns=now_ns, + production_authorized=True, + ) + return ClosureAuthorization(True, "AUTHORIZED", True, closure) + + @staticmethod + def _validate_primary( + candidate: CandidateSlice, bundle_id: str, primary: Mapping[str, Any] + ) -> str: + if primary.get("schema") != "qdl.phase92.production-primary.v1": + return "PRIMARY_EVIDENCE_SCHEMA_INVALID" + if primary.get("status") != "PRODUCTION_PRIMARY_ACTIVE": + return "PRIMARY_NOT_ACTIVE" + if primary.get("production_authorized") is not True: + return "PRIMARY_NOT_PRODUCTION_AUTHORIZED" + if primary.get("slice_id") != candidate.payload["slice_id"]: + return "PRIMARY_SLICE_MISMATCH" + if primary.get("candidate_digest") != candidate.digest: + return "PRIMARY_CANDIDATE_MISMATCH" + if primary.get("prerequisite_bundle_id") != bundle_id: + return "PRIMARY_BUNDLE_MISMATCH" + authority = primary.get("authority") + if not isinstance(authority, Mapping) or authority.get("state") != "RUST_PRIMARY": + return "PRIMARY_AUTHORITY_INVALID" + required = ( + "owner_id", "authority_revision", "lease_epoch", + "partition_plan_epoch", "current_watermark", + ) + if any(key not in authority for key in required): + return "PRIMARY_AUTHORITY_INCOMPLETE" + return "AUTHORIZED" + + @staticmethod + def _validate_hold( + candidate: CandidateSlice, bundle_id: str, hold: PrimaryHoldDecision + ) -> str: + if hold.status != HoldStatus.PASSED or hold.reason != "PASS": + return "PRIMARY_HOLD_NOT_PASSED" + if hold.scope != HoldScope.PRODUCTION or not hold.production_authorized: + return "PRIMARY_HOLD_NOT_PRODUCTION" + if hold.slice_id != candidate.payload["slice_id"]: + return "PRIMARY_HOLD_SLICE_MISMATCH" + if hold.candidate_digest != candidate.digest: + return "PRIMARY_HOLD_CANDIDATE_MISMATCH" + if hold.prerequisite_bundle_id != bundle_id: + return "PRIMARY_HOLD_BUNDLE_MISMATCH" + return "AUTHORIZED" + + @staticmethod + def _validate_registries( + hold: PrimaryHoldDecision, + primary: Mapping[str, Any], + consumer: ConsumerRegistrySnapshot, + authority: AuthorityRegistrySnapshot, + now_ns: int, + ) -> str: + if (consumer.slice_id, consumer.authority_revision) != ( + hold.slice_id, + hold.authority_revision, + ): + return "CONSUMER_REGISTRY_IDENTITY_MISMATCH" + if ( + authority.slice_id != hold.slice_id + or authority.owner_id != hold.owner_id + or authority.authority_revision != hold.authority_revision + or authority.lease_epoch != hold.lease_epoch + or authority.partition_plan_epoch != hold.partition_plan_epoch + or authority.candidate_digest != hold.candidate_digest + or authority.prerequisite_bundle_id != hold.prerequisite_bundle_id + ): + return "AUTHORITY_REGISTRY_IDENTITY_MISMATCH" + max_age_ns = 300 * 1_000_000_000 + if any( + observed > now_ns or now_ns - observed > max_age_ns + for observed in (consumer.observed_at_ns, authority.observed_at_ns) + ): + return "REGISTRY_SNAPSHOT_STALE" + primary_authority = primary.get("authority") + if not isinstance(primary_authority, Mapping) or any( + primary_authority.get(field) != expected + for field, expected in ( + ("owner_id", authority.owner_id), + ("authority_revision", authority.authority_revision), + ("lease_epoch", authority.lease_epoch), + ("partition_plan_epoch", authority.partition_plan_epoch), + ("current_watermark", authority.current_watermark), + ) + ): + return "PRIMARY_AUTHORITY_REGISTRY_MISMATCH" + terminal = hold.terminal_watermark + if terminal is None or authority.current_watermark < terminal: + return "AUTHORITY_REGISTRY_WATERMARK_BEHIND" + if any( + item.checkpointed_watermark < authority.current_watermark + for item in consumer.checkpoints + ): + return "CONSUMER_CHECKPOINT_BEHIND" + return "AUTHORIZED" + + @staticmethod + def _validate_rollback( + candidate: CandidateSlice, + hold: PrimaryHoldDecision, + authority: AuthorityRegistrySnapshot, + rollback: RollbackRehearsalEvidence, + now_ns: int, + ) -> str: + if not rollback.production_scope: + return "ROLLBACK_REHEARSAL_NOT_PRODUCTION" + if rollback.expires_at_ns <= now_ns: + return "ROLLBACK_REHEARSAL_EXPIRED" + if ( + rollback.slice_id != hold.slice_id + or rollback.candidate_digest != hold.candidate_digest + or rollback.owner_id != hold.owner_id + or rollback.authority_revision != hold.authority_revision + or rollback.lease_epoch != hold.lease_epoch + or rollback.partition_plan_epoch != hold.partition_plan_epoch + ): + return "ROLLBACK_REHEARSAL_IDENTITY_MISMATCH" + if ( + rollback.rollback_manifest_digest + != candidate.payload["rollback_manifest_digest"] + ): + return "ROLLBACK_MANIFEST_MISMATCH" + if rollback.reconciled_through_watermark < authority.current_watermark: + return "ROLLBACK_REHEARSAL_WATERMARK_BEHIND" + return "AUTHORIZED" + + @staticmethod + def _validate_approval( + candidate: CandidateSlice, + bundle_id: str, + hold: PrimaryHoldDecision, + approval: ClosureApproval, + now_ns: int, + ) -> str: + if approval.expires_at_ns <= now_ns or approval.approved_at_ns > now_ns: + return "CLOSURE_APPROVAL_TIME_INVALID" + if ( + approval.slice_id != candidate.payload["slice_id"] + or approval.candidate_digest != candidate.digest + or approval.prerequisite_bundle_id != bundle_id + or approval.hold_id != hold.hold_id + or approval.hold_policy_digest != hold.policy_digest + ): + return "CLOSURE_APPROVAL_IDENTITY_MISMATCH" + return "AUTHORIZED" + + +@dataclass(frozen=True, slots=True) +class ExpansionManifest: + schema: str + expansion_id: str + parent_slice_id: str + parent_candidate_digest: str + parent_closure_id: str + parent_closure_digest: str + expansion_type: ExpansionType + candidate_digest: str + scope_digest: str + partition_plan_epoch: int + required_gates: tuple[str, ...] + status: str + transitive_evidence_allowed: bool + public_write_allowed: bool + legacy_write_allowed: bool + created_at_ns: int + + def __post_init__(self) -> None: + if self.schema != "qdl.independent-expansion.v1": + raise ValueError("expansion schema is invalid") + if not isinstance(self.expansion_type, ExpansionType): + raise ValueError("expansion type is invalid") + if not _valid_uuid(self.expansion_id) or not _valid_uuid( + self.parent_closure_id + ): + raise ValueError("expansion UUID identity is invalid") + if not self.parent_slice_id.strip(): + raise ValueError("expansion parent slice is required") + if any( + not _valid_digest(value) + for value in ( + self.parent_candidate_digest, + self.parent_closure_digest, + self.candidate_digest, + self.scope_digest, + ) + ): + raise ValueError("expansion digest identity is invalid") + if self.candidate_digest == self.parent_candidate_digest: + raise ValueError("expansion requires a new candidate digest") + if not _positive_int(self.partition_plan_epoch): + raise ValueError("expansion partition plan epoch is invalid") + if self.required_gates != tuple(sorted(_EXPANSION_GATES[self.expansion_type])): + raise ValueError("expansion required gates are incomplete or transitive") + if self.status != "INDEPENDENT_CERTIFICATION_REQUIRED": + raise ValueError("expansion status cannot grant authority") + if ( + self.transitive_evidence_allowed + or self.public_write_allowed + or self.legacy_write_allowed + ): + raise ValueError("expansion cannot inherit evidence or write authority") + if not _positive_int(self.created_at_ns): + raise ValueError("expansion creation time is invalid") + + @classmethod + def plan( + cls, + *, + expansion_id: str, + parent: RollbackWindowClosure, + expansion_type: ExpansionType, + candidate_digest: str, + scope_digest: str, + partition_plan_epoch: int, + created_at_ns: int, + ) -> "ExpansionManifest": + if ( + expansion_type == ExpansionType.INSTRUMENT_PARTITION + and partition_plan_epoch <= parent.partition_plan_epoch + ): + raise ValueError("instrument expansion requires a newer partition epoch") + return cls( + schema="qdl.independent-expansion.v1", + expansion_id=expansion_id, + parent_slice_id=parent.slice_id, + parent_candidate_digest=parent.candidate_digest, + parent_closure_id=parent.closure_id, + parent_closure_digest=parent.digest, + expansion_type=expansion_type, + candidate_digest=candidate_digest, + scope_digest=scope_digest, + partition_plan_epoch=partition_plan_epoch, + required_gates=tuple(sorted(_EXPANSION_GATES[expansion_type])), + status="INDEPENDENT_CERTIFICATION_REQUIRED", + transitive_evidence_allowed=False, + public_write_allowed=False, + legacy_write_allowed=False, + created_at_ns=created_at_ns, + ) + + @property + def digest(self) -> str: + return _digest(asdict(self)) + + +@dataclass(frozen=True, slots=True) +class DecommissionRequest: + schema: str + request_id: str + runtime_id: str + owned_slice_ids: tuple[str, ...] + rollback_reference_ids: tuple[str, ...] + consumer_dependency_ids: tuple[str, ...] + all_replacement_windows_closed: bool + repository_cleanup_approved: bool + shared_knowledge_retained: bool + + def __post_init__(self) -> None: + if self.schema != "qdl.runtime-decommission-request.v1": + raise ValueError("decommission request schema is invalid") + if not _valid_uuid(self.request_id) or not self.runtime_id.strip(): + raise ValueError("decommission request identity is invalid") + for values in ( + self.owned_slice_ids, + self.rollback_reference_ids, + self.consumer_dependency_ids, + ): + if not isinstance(values, tuple): + raise ValueError("decommission dependencies must be immutable") + if any(not value.strip() for value in values) or len(values) != len( + set(values) + ): + raise ValueError("decommission dependencies must be unique") + + +@dataclass(frozen=True, slots=True) +class DecommissionDecision: + allowed: bool + reason: str + + +def assess_decommission(request: DecommissionRequest) -> DecommissionDecision: + if request.owned_slice_ids: + return DecommissionDecision(False, "RUNTIME_STILL_OWNS_SLICES") + if request.rollback_reference_ids: + return DecommissionDecision(False, "RUNTIME_STILL_REQUIRED_FOR_ROLLBACK") + if request.consumer_dependency_ids: + return DecommissionDecision(False, "RUNTIME_HAS_CONSUMER_DEPENDENCIES") + if not request.all_replacement_windows_closed: + return DecommissionDecision(False, "REPLACEMENT_WINDOWS_NOT_CLOSED") + if not request.repository_cleanup_approved: + return DecommissionDecision(False, "REPOSITORY_CLEANUP_NOT_APPROVED") + if not request.shared_knowledge_retained: + return DecommissionDecision(False, "SHARED_KNOWLEDGE_REMOVAL_FORBIDDEN") + return DecommissionDecision(True, "AUTHORIZED") diff --git a/qdl/canonical/book.py b/qdl/canonical/book.py index 45afd5d..0b2ba15 100644 --- a/qdl/canonical/book.py +++ b/qdl/canonical/book.py @@ -5,6 +5,7 @@ from qdl.common.v1 import common_pb2 from qdl.marketdata.v2 import market_data_pb2 +from qdl.domain.quantity import quantity_unit_proto from qdl.canonical.market import _envelope from qdl.canonical.trade import ( @@ -15,7 +16,9 @@ ) -def _levels(rows: list[list[str]], side: int) -> list[market_data_pb2.BookLevel]: +def _levels( + rows: list[list[str]], side: int, quantity_unit: int +) -> list[market_data_pb2.BookLevel]: output = [] for row in rows: if len(row) < 4: @@ -26,6 +29,7 @@ def _levels(rows: list[list[str]], side: int) -> list[market_data_pb2.BookLevel] price=_decimal(row[0]), quantity=_decimal(row[1]), order_count=int(_required({"order_count": row[3]}, "order_count")), + quantity_unit=quantity_unit, ) ) return output @@ -52,9 +56,14 @@ def canonicalize_okx_book( source_sequence=sequence, source_event_time_ms=int(_required(row, "ts")), ) + unit = quantity_unit_proto( + venue=context.venue, + market=context.market, + product_type=context.product_type, + ) levels = [ - *_levels(row.get("bids", []), common_pb2.BOOK_SIDE_BID), - *_levels(row.get("asks", []), common_pb2.BOOK_SIDE_ASK), + *_levels(row.get("bids", []), common_pb2.BOOK_SIDE_BID, unit), + *_levels(row.get("asks", []), common_pb2.BOOK_SIDE_ASK, unit), ] checksum = str(row.get("checksum") or "") if action == "snapshot": @@ -95,6 +104,11 @@ def canonicalize_deribit_option_book_fixture( source_sequence=sequence, source_event_time_ms=int(_required(frame, "timestamp")), ) + unit = quantity_unit_proto( + venue=context.venue, + market=context.market, + product_type=context.product_type, + ) levels = [] for side, key in ( (common_pb2.BOOK_SIDE_BID, "bids"), diff --git a/qdl/canonical/market.py b/qdl/canonical/market.py index 28fc9cf..5e04ecd 100644 --- a/qdl/canonical/market.py +++ b/qdl/canonical/market.py @@ -5,6 +5,7 @@ from qdl.common.v1 import common_pb2 from qdl.domain.event_id import deterministic_event_id +from qdl.domain.quantity import quantity_unit_proto from qdl.marketdata.v2 import market_data_pb2 from qdl.canonical.trade import ( @@ -14,6 +15,7 @@ _required_bool, _set_canonical_payload_hash, _validate_shadow_context, + source_role_proto, canonical_json_bytes, ) @@ -34,7 +36,7 @@ def _envelope( instrument_revision=context.instrument_revision, venue=context.venue, market=context.market, product_type=context.product_type, native_symbol=context.native_symbol, provider=context.provider, - source_id=context.source_id, source_role=common_pb2.SOURCE_ROLE_PRIMARY, + source_id=context.source_id, source_role=source_role_proto(context.source_role), lease_epoch=context.lease_epoch, source_event_time_ns=source_event_time_ms * 1_000_000, received_at_ns=context.received_at_ns, @@ -55,6 +57,33 @@ def _envelope( ) +def _quantity_unit(context: TradeContext) -> int: + return quantity_unit_proto( + venue=context.venue, + market=context.market, + product_type=context.product_type, + ) + + +def _attach_bar_volumes( + bar: market_data_pb2.Bar, + *, + unit: int, + native: Any, + base: Any | None = None, + quote: Any | None = None, + contracts: Any | None = None, +) -> None: + bar.volume.CopyFrom(_decimal(native)) + bar.volume_unit = unit + if base is not None and base != "": + bar.base_volume.CopyFrom(_decimal(base)) + if quote is not None and quote != "": + bar.quote_volume.CopyFrom(_decimal(quote)) + if contracts is not None and contracts != "": + bar.contract_volume.CopyFrom(_decimal(contracts)) + + def _verify_symbol(raw: Mapping[str, Any], context: TradeContext) -> None: if str(_required(raw, "s")).upper() != context.native_symbol.upper(): raise ValueError("provider symbol does not match resolved instrument") @@ -65,15 +94,27 @@ def canonicalize_binance_usdm_bbo( ) -> market_data_pb2.EventEnvelope: _verify_symbol(raw, context) sequence = str(_required(raw, "u")) + provider_time = raw.get("T") or raw.get("E") envelope = _envelope( - raw=raw, context=context, feed="quote", source_sequence=sequence, - source_event_time_ms=int(raw.get("T") or _required(raw, "E")), + raw=raw, + context=context, + feed="quote", + source_sequence=sequence, + source_event_time_ms=( + int(provider_time) + if provider_time is not None + else context.received_at_ns // 1_000_000 + ), ) + if provider_time is None: + envelope.quality_flags.append(common_pb2.QUALITY_FLAG_SOURCE_TIME_MISSING) envelope.quote.CopyFrom(market_data_pb2.Quote( bid_price=_decimal(_required(raw, "b")), bid_quantity=_decimal(_required(raw, "B")), ask_price=_decimal(_required(raw, "a")), - ask_quantity=_decimal(_required(raw, "A")), level=1, + ask_quantity=_decimal(_required(raw, "A")), + level=1, + quantity_unit=_quantity_unit(context), )) _set_canonical_payload_hash(envelope, enabled=bool(context.source_session_id)) return envelope @@ -91,28 +132,151 @@ def canonicalize_binance_usdm_bar( source_time = int(_required(raw, "E")) sequence = f"{_required(kline, 't')}:{_required(kline, 'L')}:{source_time}" envelope = _envelope( - raw=raw, context=context, feed="bar", source_sequence=sequence, + raw=raw, + context=context, + feed="bar", + source_sequence=sequence, source_event_time_ms=source_time, ) is_final = _required_bool(kline, "x") - envelope.bar.CopyFrom(market_data_pb2.Bar( + bar = market_data_pb2.Bar( interval=str(_required(kline, "i")), open_time_ns=int(_required(kline, "t")) * 1_000_000, close_time_ns=int(_required(kline, "T")) * 1_000_000, - open=_decimal(_required(kline, "o")), high=_decimal(_required(kline, "h")), - low=_decimal(_required(kline, "l")), close=_decimal(_required(kline, "c")), - volume=_decimal(_required(kline, "v")), trade_count=int(_required(kline, "n")), - is_final=is_final, revision=0, + open=_decimal(_required(kline, "o")), + high=_decimal(_required(kline, "h")), + low=_decimal(_required(kline, "l")), + close=_decimal(_required(kline, "c")), + trade_count=int(_required(kline, "n")), + is_final=is_final, + revision=0, origin=common_pb2.BAR_ORIGIN_VENUE_NATIVE, lifecycle=( market_data_pb2.BAR_LIFECYCLE_FINAL if is_final else market_data_pb2.BAR_LIFECYCLE_IN_PROGRESS ), - )) + ) + native_volume = _required(kline, "v") + _attach_bar_volumes( + bar, + unit=_quantity_unit(context), + native=native_volume, + base=native_volume, + quote=kline.get("q"), + ) + envelope.bar.CopyFrom(bar) _set_canonical_payload_hash(envelope, enabled=bool(context.source_session_id)) return envelope +def _okx_frame_row( + raw: Mapping[str, Any], context: TradeContext, *, channel: str +) -> tuple[Mapping[str, Any], list[Any]]: + argument = raw.get("arg") + rows = raw.get("data") + if ( + not isinstance(argument, Mapping) + or argument.get("channel") != channel + or argument.get("instId") != context.native_symbol + ): + raise ValueError("OKX frame channel/instrument mismatch") + if not isinstance(rows, list) or len(rows) != 1: + raise ValueError("OKX frame requires one data row") + return argument, rows[0] + + +def canonicalize_okx_bbo( + raw: Mapping[str, Any], context: TradeContext +) -> market_data_pb2.EventEnvelope: + _, row = _okx_frame_row(raw, context, channel="bbo-tbt") + if not isinstance(row, Mapping): + raise ValueError("OKX BBO row must be an object") + bids, asks = row.get("bids"), row.get("asks") + if ( + not isinstance(bids, list) + or not isinstance(asks, list) + or len(bids) != 1 + or len(asks) != 1 + or not isinstance(bids[0], list) + or not isinstance(asks[0], list) + or len(bids[0]) < 2 + or len(asks[0]) < 2 + ): + raise ValueError("OKX bbo-tbt requires one bid and one ask") + sequence = str(_required(row, "seqId")) + envelope = _envelope( + raw=raw, + context=context, + feed="quote", + source_sequence=sequence, + source_event_time_ms=int(_required(row, "ts")), + ) + envelope.quote.CopyFrom( + market_data_pb2.Quote( + bid_price=_decimal(bids[0][0]), + bid_quantity=_decimal(bids[0][1]), + ask_price=_decimal(asks[0][0]), + ask_quantity=_decimal(asks[0][1]), + level=1, + quantity_unit=_quantity_unit(context), + ) + ) + _set_canonical_payload_hash(envelope, enabled=bool(context.source_session_id)) + return envelope + + +def canonicalize_okx_bar( + raw: Mapping[str, Any], context: TradeContext +) -> market_data_pb2.EventEnvelope: + _, row = _okx_frame_row(raw, context, channel="candle1m") + if not isinstance(row, list) or len(row) < 9: + raise ValueError("OKX candle1m requires the native nine-field row") + open_time_ms = int(row[0]) + confirm = str(row[8]) + if confirm not in {"0", "1"}: + raise ValueError("OKX candle confirm must be 0 or 1") + is_final = confirm == "1" + # Final/provisional venue bars keep one identity across REST bootstrap, + # WebSocket delivery and process restart. Content conflicts then fail closed. + sequence = f"{open_time_ms}:{confirm}" + envelope = _envelope( + raw=raw, + context=context, + feed="bar", + source_sequence=sequence, + source_event_time_ms=open_time_ms, + ) + envelope.quality_flags.append(common_pb2.QUALITY_FLAG_FIELD_MISSING) + unit = _quantity_unit(context) + bar = market_data_pb2.Bar( + interval="1m", + open_time_ns=open_time_ms * 1_000_000, + close_time_ns=(open_time_ms + 60_000 - 1) * 1_000_000, + open=_decimal(row[1]), + high=_decimal(row[2]), + low=_decimal(row[3]), + close=_decimal(row[4]), + trade_count=0, + is_final=is_final, + revision=0, + origin=common_pb2.BAR_ORIGIN_VENUE_NATIVE, + lifecycle=( + market_data_pb2.BAR_LIFECYCLE_FINAL + if is_final + else market_data_pb2.BAR_LIFECYCLE_IN_PROGRESS + ), + ) + _attach_bar_volumes( + bar, + unit=unit, + native=row[5], + base=(row[5] if unit == common_pb2.QUANTITY_UNIT_BASE_ASSET else row[6]), + quote=row[7], + contracts=(row[5] if unit == common_pb2.QUANTITY_UNIT_CONTRACT else None), + ) + envelope.bar.CopyFrom(bar) + _set_canonical_payload_hash(envelope, enabled=bool(context.source_session_id)) + return envelope def canonicalize_binance_usdm_rest_bar( raw: Mapping[str, Any], context: TradeContext @@ -122,25 +286,49 @@ def canonicalize_binance_usdm_rest_bar( row = raw.get("row") if not isinstance(row, list) or len(row) < 11: raise ValueError("Binance REST kline requires the unmodified native row") + origin_name = str(raw.get("bar_origin") or "BACKFILLED").upper() + origins = { + "VENUE_NATIVE": common_pb2.BAR_ORIGIN_VENUE_NATIVE, + "BACKFILLED": common_pb2.BAR_ORIGIN_BACKFILLED, + "RECONCILED": common_pb2.BAR_ORIGIN_RECONCILED, + } + if origin_name not in origins: + raise ValueError("Binance REST bar origin is invalid") sequence = f"{row[0]}:{row[6]}" envelope = _envelope( - raw=raw, context=context, feed="bar", source_sequence=sequence, + raw=raw, + context=context, + feed="bar", + source_sequence=sequence, source_event_time_ms=int(row[6]), ) - envelope.quality_flags.append(common_pb2.QUALITY_FLAG_BACKFILLED) - envelope.bar.CopyFrom(market_data_pb2.Bar( + bar = market_data_pb2.Bar( interval=str(_required(raw, "interval")), open_time_ns=int(row[0]) * 1_000_000, close_time_ns=int(row[6]) * 1_000_000, - open=_decimal(row[1]), high=_decimal(row[2]), low=_decimal(row[3]), - close=_decimal(row[4]), volume=_decimal(row[5]), trade_count=int(row[8]), - is_final=True, revision=0, origin=common_pb2.BAR_ORIGIN_BACKFILLED, + open=_decimal(row[1]), + high=_decimal(row[2]), + low=_decimal(row[3]), + close=_decimal(row[4]), + trade_count=int(row[8]), + is_final=True, + revision=0, + origin=origins[origin_name], lifecycle=market_data_pb2.BAR_LIFECYCLE_FINAL, - )) + ) + _attach_bar_volumes( + bar, + unit=_quantity_unit(context), + native=row[5], + base=row[5], + quote=row[7], + ) + if origin_name == "BACKFILLED": + envelope.quality_flags.append(common_pb2.QUALITY_FLAG_BACKFILLED) + envelope.bar.CopyFrom(bar) _set_canonical_payload_hash(envelope, enabled=bool(context.source_session_id)) return envelope - def canonicalize_dnse_bar( raw: Mapping[str, Any], context: TradeContext ) -> market_data_pb2.EventEnvelope: @@ -161,26 +349,32 @@ def canonicalize_dnse_bar( ) if not trade_count_available: envelope.quality_flags.append(common_pb2.QUALITY_FLAG_FIELD_MISSING) - envelope.bar.CopyFrom( - market_data_pb2.Bar( - interval=str(_required(raw, "interval")), - open_time_ns=open_time_ms * 1_000_000, - close_time_ns=close_time_ms * 1_000_000, - open=_decimal(_required(raw, "o")), - high=_decimal(_required(raw, "h")), - low=_decimal(_required(raw, "l")), - close=_decimal(_required(raw, "c")), - volume=_decimal(_required(raw, "v")), - trade_count=trade_count, - is_final=is_final, - revision=int(_required(raw, "revision")), - origin=common_pb2.BAR_ORIGIN_VENUE_NATIVE, - lifecycle=( - market_data_pb2.BAR_LIFECYCLE_FINAL - if is_final - else market_data_pb2.BAR_LIFECYCLE_IN_PROGRESS - ), - ) + unit = _quantity_unit(context) + native_volume = _required(raw, "v") + bar = market_data_pb2.Bar( + interval=str(_required(raw, "interval")), + open_time_ns=open_time_ms * 1_000_000, + close_time_ns=close_time_ms * 1_000_000, + open=_decimal(_required(raw, "o")), + high=_decimal(_required(raw, "h")), + low=_decimal(_required(raw, "l")), + close=_decimal(_required(raw, "c")), + trade_count=trade_count, + is_final=is_final, + revision=int(_required(raw, "revision")), + origin=common_pb2.BAR_ORIGIN_VENUE_NATIVE, + lifecycle=( + market_data_pb2.BAR_LIFECYCLE_FINAL + if is_final + else market_data_pb2.BAR_LIFECYCLE_IN_PROGRESS + ), + ) + _attach_bar_volumes( + bar, + unit=unit, + native=native_volume, + contracts=(native_volume if unit == common_pb2.QUANTITY_UNIT_CONTRACT else None), ) + envelope.bar.CopyFrom(bar) _set_canonical_payload_hash(envelope, enabled=bool(context.source_session_id)) return envelope diff --git a/qdl/canonical/trade.py b/qdl/canonical/trade.py index 9b94d93..0b5c6e4 100644 --- a/qdl/canonical/trade.py +++ b/qdl/canonical/trade.py @@ -8,6 +8,7 @@ from qdl.common.v1 import common_pb2 from qdl.domain.decimal import CanonicalDecimal from qdl.domain.event_id import deterministic_event_id +from qdl.domain.quantity import quantity_unit_proto from qdl.marketdata.v2 import market_data_pb2 from qdl.transport.contracts import DurableEvent, partition_key @@ -38,6 +39,11 @@ class TradeContext: partition_plan_epoch: int = 0 raw_capture_id: bytes = b"" raw_frame_sha256: bytes = b"" + source_role: str = "PRIMARY" + + def __post_init__(self) -> None: + object.__setattr__(self, "raw_capture_id", bytes(self.raw_capture_id)) + object.__setattr__(self, "raw_frame_sha256", bytes(self.raw_frame_sha256)) def canonical_json_bytes(value: Mapping[str, Any]) -> bytes: @@ -73,6 +79,13 @@ def _required_bool(raw: Mapping[str, Any], field: str) -> bool: return value +def source_role_proto(value: str) -> int: + normalized = value.strip().upper() + if normalized not in {"PRIMARY", "SECONDARY", "REFERENCE", "BACKFILL"}: + raise ValueError("canonical source role is invalid") + return getattr(common_pb2, f"SOURCE_ROLE_{normalized}") + + def _set_canonical_payload_hash( envelope: market_data_pb2.EventEnvelope, *, enabled: bool ) -> None: @@ -102,6 +115,7 @@ def _trade_envelope( side: int, source_event_time_ms: int, is_buyer_maker: bool, + identity_kind: int = market_data_pb2.TRADE_IDENTITY_KIND_NATIVE, ) -> market_data_pb2.EventEnvelope: _validate_shadow_context(context) raw_bytes = canonical_json_bytes(raw) @@ -130,7 +144,7 @@ def _trade_envelope( native_symbol=context.native_symbol, provider=context.provider, source_id=context.source_id, - source_role=common_pb2.SOURCE_ROLE_PRIMARY, + source_role=source_role_proto(context.source_role), lease_epoch=context.lease_epoch, source_event_time_ns=source_event_time_ms * 1_000_000, received_at_ns=context.received_at_ns, @@ -155,6 +169,12 @@ def _trade_envelope( aggressor_side=side, is_block_trade=False, is_buyer_maker=is_buyer_maker, + quantity_unit=quantity_unit_proto( + venue=context.venue, + market=context.market, + product_type=context.product_type, + ), + identity_kind=identity_kind, ), ) _set_canonical_payload_hash(envelope, enabled=bool(context.source_session_id)) @@ -210,6 +230,35 @@ def canonicalize_okx_trade( ) +def canonicalize_dnse_trade( + raw: Mapping[str, Any], context: TradeContext +) -> market_data_pb2.EventEnvelope: + symbol = str(_required(raw, "symbol")).upper() + if symbol != context.native_symbol.upper(): + raise ValueError("provider symbol does not match resolved instrument") + if len(context.raw_capture_id) != 16: + raise ValueError("DNSE trade identity requires an exact raw capture id") + source_time_ms = context.received_at_ns // 1_000_000 + envelope = _trade_envelope( + raw=raw, + context=context, + native_trade_id=f"derived:{context.raw_capture_id.hex()}", + price=_required(raw, "price"), + quantity=_required(raw, "quantity"), + side=common_pb2.AGGRESSOR_SIDE_UNSPECIFIED, + source_event_time_ms=source_time_ms, + is_buyer_maker=False, + identity_kind=market_data_pb2.TRADE_IDENTITY_KIND_DERIVED_RAW_CAPTURE, + ) + envelope.quality_flags.extend( + ( + common_pb2.QUALITY_FLAG_SOURCE_TIME_MISSING, + common_pb2.QUALITY_FLAG_FIELD_MISSING, + ) + ) + return envelope + + def canonical_event( envelope: market_data_pb2.EventEnvelope, *, diff --git a/qdl/certification/prerequisites.py b/qdl/certification/prerequisites.py new file mode 100644 index 0000000..55c6640 --- /dev/null +++ b/qdl/certification/prerequisites.py @@ -0,0 +1,370 @@ +from __future__ import annotations + +import hashlib +import json +import re +import uuid +from dataclasses import dataclass +from enum import StrEnum +from pathlib import Path +from typing import Any + +import yaml + + +_SHA256 = re.compile(r"^[0-9a-f]{64}$") +_IMAGE_DIGEST = re.compile(r"^sha256:[0-9a-f]{64}$") +_SENSITIVE_DETAIL_NAMES = frozenset({ + "secret", "token", "password", "private_key", "private-key", "key_material", +}) + + +def _sensitive_detail_key(value: object) -> bool: + name = str(value).lower() + return name in _SENSITIVE_DETAIL_NAMES or any( + name.endswith(suffix) + for suffix in ("_secret", "_token", "_password", "_private_key", "_key_material") + ) + + +def _contains_sensitive_detail(value: object) -> bool: + if isinstance(value, dict): + return any( + _sensitive_detail_key(key) or _contains_sensitive_detail(item) + for key, item in value.items() + ) + if isinstance(value, (list, tuple)): + return any(_contains_sensitive_detail(item) for item in value) + return False + + +def _is_number(value: object) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) + + +class EvidenceScope(StrEnum): + TEST = "TEST" + LOCAL_REHEARSAL = "LOCAL_REHEARSAL" + PRODUCTION = "PRODUCTION" + INDEPENDENT_FAILURE_DOMAIN = "INDEPENDENT_FAILURE_DOMAIN" + + +_SCOPE_RANK = {scope: rank for rank, scope in enumerate(EvidenceScope)} + + +@dataclass(frozen=True) +class GatePolicy: + gate_id: str + minimum_scope: EvidenceScope + max_age_seconds: int + required_details: frozenset[str] + candidate_bound: bool + assertions: dict[str, Any] + minimums: dict[str, float] + maximums: dict[str, float] + candidate_field_matches: dict[str, str] + + +@dataclass(frozen=True) +class PrerequisitePolicy: + revision: int + environment: str + gates: tuple[GatePolicy, ...] + + @classmethod + def load(cls, path: str | Path) -> "PrerequisitePolicy": + payload = yaml.safe_load(Path(path).read_text()) + if not isinstance(payload, dict) or set(payload) != { + "schema", "revision", "environment", "gates" + }: + raise ValueError("prerequisite policy fields are incomplete or unknown") + if payload["schema"] != "qdl.production-prerequisite-policy.v1": + raise ValueError("unsupported prerequisite policy schema") + gates_raw = payload["gates"] + if not isinstance(gates_raw, list) or not gates_raw: + raise ValueError("prerequisite policy requires gates") + gates: list[GatePolicy] = [] + seen: set[str] = set() + for item in gates_raw: + if not isinstance(item, dict) or set(item) != { + "id", "minimum_scope", "max_age_seconds", + "required_details", "candidate_bound", "assertions", + "minimums", "maximums", "candidate_field_matches", + }: + raise ValueError("prerequisite gate fields are incomplete or unknown") + gate_id = str(item["id"]).strip() + if not gate_id or gate_id in seen: + raise ValueError("prerequisite gate IDs must be non-empty and unique") + seen.add(gate_id) + max_age = int(item["max_age_seconds"]) + details = item["required_details"] + if max_age <= 0 or not isinstance(details, list): + raise ValueError("prerequisite gate age/details are invalid") + mappings = tuple(item[key] for key in ( + "assertions", "minimums", "maximums", "candidate_field_matches" + )) + if not all(isinstance(value, dict) for value in mappings): + raise ValueError("prerequisite gate constraints must be mappings") + constrained = set().union(*(set(value) for value in mappings)) + if constrained - set(details): + raise ValueError("prerequisite constraints must reference required details") + gates.append(GatePolicy( + gate_id=gate_id, + minimum_scope=EvidenceScope(str(item["minimum_scope"]).upper()), + max_age_seconds=max_age, + required_details=frozenset(str(value) for value in details), + candidate_bound=bool(item["candidate_bound"]), + assertions=dict(item["assertions"]), + minimums={key: float(value) for key, value in item["minimums"].items()}, + maximums={key: float(value) for key, value in item["maximums"].items()}, + candidate_field_matches={ + str(key): str(value) + for key, value in item["candidate_field_matches"].items() + }, + )) + revision = int(payload["revision"]) + environment = str(payload["environment"]).strip().lower() + if revision <= 0 or not environment: + raise ValueError("prerequisite policy revision/environment are invalid") + return cls(revision=revision, environment=environment, gates=tuple(gates)) + + +@dataclass(frozen=True) +class CandidateSlice: + payload: dict[str, Any] + + @classmethod + def load(cls, path: str | Path) -> "CandidateSlice": + payload = yaml.safe_load(Path(path).read_text()) + required = { + "schema", "slice_id", "environment", "venue", "market", + "product_type", "feed", "instrument_uids", "partition_plan_epoch", + "partition_id", "schema_major", "authority_state", "owner_id", + "lease_epoch", "artifact_image_digest", "sbom_digest", + "contract_digest", "partition_plan_digest", "rollback_manifest_digest", + "signature_identity", "normalizer_version", "adapter_version", + "config_revision", "instrument_catalog_revision", "source_policy_revision", + "public_write_allowed", "legacy_write_allowed", + } + if not isinstance(payload, dict) or set(payload) != required: + raise ValueError("candidate slice fields are incomplete or unknown") + if payload["schema"] != "qdl.authority-candidate.v1": + raise ValueError("unsupported candidate slice schema") + text_fields = required - { + "instrument_uids", "partition_plan_epoch", "schema_major", "lease_epoch", + "public_write_allowed", "legacy_write_allowed", + } + if any(not str(payload[field]).strip() for field in text_fields): + raise ValueError("candidate slice identity fields are required") + instruments = payload["instrument_uids"] + if not isinstance(instruments, list) or not instruments or not all( + isinstance(item, str) and item.strip() for item in instruments + ) or len(set(instruments)) != len(instruments): + raise ValueError("candidate slice instruments must be non-empty and unique") + try: + for instrument_uid in instruments: + uuid.UUID(instrument_uid) + except ValueError as exc: + raise ValueError("candidate instrument UID is invalid") from exc + if not isinstance(payload["public_write_allowed"], bool) or not isinstance(payload["legacy_write_allowed"], bool): + raise ValueError("candidate write-authority flags must be booleans") + if int(payload["partition_plan_epoch"]) <= 0 or int(payload["schema_major"]) <= 0: + raise ValueError("candidate plan/schema epoch must be positive") + if int(payload["lease_epoch"]) <= 0: + raise ValueError("candidate lease epoch must be positive") + if payload["authority_state"] != "RUST_SHADOW": + raise ValueError("Phase 9.0-C candidate must remain RUST_SHADOW") + if payload["public_write_allowed"] or payload["legacy_write_allowed"]: + raise ValueError("Phase 9.0-C candidate cannot write public or legacy output") + if not _IMAGE_DIGEST.fullmatch(str(payload["artifact_image_digest"])): + raise ValueError("candidate image digest is invalid") + for field in ( + "sbom_digest", "contract_digest", "partition_plan_digest", + "rollback_manifest_digest", + ): + if not _SHA256.fullmatch(str(payload[field])): + raise ValueError(f"candidate {field} is invalid") + return cls(dict(payload)) + + @property + def digest(self) -> str: + encoded = json.dumps(self.payload, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest() + + +@dataclass(frozen=True) +class EvidenceRecord: + evidence_id: str + gate_id: str + environment: str + scope: EvidenceScope + status: str + issuer: str + observed_at_ns: int + expires_at_ns: int + artifact_path: str | None + artifact_sha256: str | None + details: dict[str, Any] + + @classmethod + def from_mapping(cls, payload: Any) -> "EvidenceRecord": + required = { + "evidence_id", "gate_id", "environment", "scope", "status", "issuer", + "observed_at_ns", "expires_at_ns", "artifact_path", "artifact_sha256", + "details", + } + if not isinstance(payload, dict) or set(payload) != required: + raise ValueError("prerequisite evidence fields are incomplete or unknown") + details = payload["details"] + if not isinstance(details, dict) or _contains_sensitive_detail(details): + raise ValueError("prerequisite evidence details are invalid or sensitive") + status = str(payload["status"]).upper() + if status not in {"PASS", "BLOCKED"}: + raise ValueError("prerequisite evidence status is invalid") + artifact_path = payload["artifact_path"] + artifact_sha = payload["artifact_sha256"] + if status == "PASS" and ( + not isinstance(artifact_path, str) + or Path(artifact_path).is_absolute() + or ".." in Path(artifact_path).parts + or not isinstance(artifact_sha, str) + or not _SHA256.fullmatch(artifact_sha) + ): + raise ValueError("passing evidence requires a repository-relative artifact hash") + observed = int(payload["observed_at_ns"]) + expires = int(payload["expires_at_ns"]) + if observed <= 0 or expires <= observed: + raise ValueError("prerequisite evidence timestamps are invalid") + identity = tuple(str(payload[key]).strip() for key in ("evidence_id", "gate_id", "environment", "issuer")) + if not all(identity): + raise ValueError("prerequisite evidence identity is required") + return cls( + evidence_id=identity[0], gate_id=identity[1], environment=identity[2].lower(), + scope=EvidenceScope(str(payload["scope"]).upper()), status=status, + issuer=identity[3], observed_at_ns=observed, expires_at_ns=expires, + artifact_path=artifact_path, artifact_sha256=artifact_sha, + details=dict(details), + ) + + +@dataclass(frozen=True) +class GateResult: + gate_id: str + passed: bool + reason: str + evidence_id: str | None + observed_scope: str | None + required_scope: str + + +@dataclass(frozen=True) +class PrerequisiteDecision: + decision: str + candidate_digest: str + policy_revision: int + results: tuple[GateResult, ...] + + def as_mapping(self) -> dict[str, Any]: + return { + "schema": "qdl.production-prerequisite-decision.v1", + "decision": self.decision, + "candidate_digest": self.candidate_digest, + "policy_revision": self.policy_revision, + "passed": sum(item.passed for item in self.results), + "blocked": sum(not item.passed for item in self.results), + "gates": [item.__dict__ for item in self.results], + } + + +def load_inventory(path: str | Path) -> tuple[EvidenceRecord, ...]: + payload = yaml.safe_load(Path(path).read_text()) + if not isinstance(payload, dict) or set(payload) != {"schema", "evidence"}: + raise ValueError("prerequisite inventory fields are incomplete or unknown") + if payload["schema"] != "qdl.production-prerequisite-inventory.v1": + raise ValueError("unsupported prerequisite inventory schema") + items = tuple(EvidenceRecord.from_mapping(item) for item in payload["evidence"]) + ids = [item.evidence_id for item in items] + gates = [item.gate_id for item in items] + if len(ids) != len(set(ids)) or len(gates) != len(set(gates)): + raise ValueError("prerequisite evidence IDs and gate bindings must be unique") + return items + + +def evaluate_prerequisites( + policy: PrerequisitePolicy, + candidate: CandidateSlice, + evidence: tuple[EvidenceRecord, ...], + *, + repository_root: str | Path, + now_ns: int, +) -> PrerequisiteDecision: + if candidate.payload["environment"].lower() != policy.environment: + raise ValueError("candidate environment does not match prerequisite policy") + by_gate = {item.gate_id: item for item in evidence} + unknown_gates = set(by_gate) - {gate.gate_id for gate in policy.gates} + if unknown_gates: + raise ValueError(f"unknown prerequisite evidence gates: {sorted(unknown_gates)}") + root = Path(repository_root).resolve() + results: list[GateResult] = [] + for gate in policy.gates: + item = by_gate.get(gate.gate_id) + reason = "PASS" + if item is None: + reason = "MISSING_EVIDENCE" + elif item.status != "PASS": + reason = "EVIDENCE_BLOCKED" + elif item.environment != policy.environment: + reason = "ENVIRONMENT_MISMATCH" + elif _SCOPE_RANK[item.scope] < _SCOPE_RANK[gate.minimum_scope]: + reason = "INSUFFICIENT_SCOPE" + elif item.observed_at_ns > now_ns + 60_000_000_000: + reason = "EVIDENCE_FROM_FUTURE" + elif item.expires_at_ns <= now_ns: + reason = "EVIDENCE_EXPIRED" + elif now_ns - item.observed_at_ns > gate.max_age_seconds * 1_000_000_000: + reason = "EVIDENCE_TOO_OLD" + elif gate.required_details - set(item.details): + reason = "DETAILS_INCOMPLETE" + elif gate.candidate_bound and item.details.get("candidate_digest") != candidate.digest: + reason = "CANDIDATE_DIGEST_MISMATCH" + elif any(item.details.get(key) != expected for key, expected in gate.assertions.items()): + reason = "ASSERTION_FAILED" + elif any( + not _is_number(item.details.get(key)) + or float(item.details[key]) < minimum + for key, minimum in gate.minimums.items() + ): + reason = "MINIMUM_NOT_MET" + elif any( + not _is_number(item.details.get(key)) + or float(item.details[key]) > maximum + for key, maximum in gate.maximums.items() + ): + reason = "MAXIMUM_EXCEEDED" + elif any( + item.details.get(detail_key) != candidate.payload.get(candidate_key) + for detail_key, candidate_key in gate.candidate_field_matches.items() + ): + reason = "CANDIDATE_FIELD_MISMATCH" + elif gate.gate_id == "exact_slice_approval" and int(item.details["hold_until_ns"]) <= now_ns: + reason = "APPROVAL_HOLD_WINDOW_INVALID" + else: + artifact = (root / str(item.artifact_path)).resolve() + if root not in artifact.parents or not artifact.is_file(): + reason = "ARTIFACT_MISSING" + elif hashlib.sha256(artifact.read_bytes()).hexdigest() != item.artifact_sha256: + reason = "ARTIFACT_DIGEST_MISMATCH" + results.append(GateResult( + gate_id=gate.gate_id, + passed=reason == "PASS", + reason=reason, + evidence_id=item.evidence_id if item else None, + observed_scope=item.scope.value if item else None, + required_scope=gate.minimum_scope.value, + )) + return PrerequisiteDecision( + decision="GO" if all(item.passed for item in results) else "NO_GO_EXTERNAL", + candidate_digest=candidate.digest, + policy_revision=policy.revision, + results=tuple(results), + ) diff --git a/qdl/consumer/__init__.py b/qdl/consumer/__init__.py index dc4b2e0..67b472d 100644 --- a/qdl/consumer/__init__.py +++ b/qdl/consumer/__init__.py @@ -2,6 +2,10 @@ from typing import TYPE_CHECKING, Any +from qdl.consumer.stable import ( + StableConsumerMigration, + StableConsumerMigrationPlan, +) from qdl.consumer.manifest import ( ConsumerManifest, ConsumerManifestLoader, @@ -26,6 +30,8 @@ "ConsumerQuotas", "ConsumerRoute", "MigrationState", + "StableConsumerMigration", + "StableConsumerMigrationPlan", "ManifestShadowConsumer", "ShadowObservation", "UsageTelemetry", diff --git a/qdl/consumer/stable.py b/qdl/consumer/stable.py new file mode 100644 index 0000000..3670de5 --- /dev/null +++ b/qdl/consumer/stable.py @@ -0,0 +1,130 @@ +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + +from qdl.consumer.manifest import ConsumerManifest, ConsumerManifestLoader +from qdl.runtime.stable_catalog import StableSourceCatalog + + +@dataclass(frozen=True, slots=True) +class StableConsumerMigration: + consumer_id: str + manifest_path: Path + manifest: ConsumerManifest + state: str + rollback_route: str + cutover_authorized: bool + + +@dataclass(frozen=True, slots=True) +class StableConsumerMigrationPlan: + schema: str + revision: int + contract_version: str + authority: str + target_route: str + consumers: tuple[StableConsumerMigration, ...] + + def __post_init__(self) -> None: + if self.schema != "qdl.v2.stable-consumer-migration.v1": + raise ValueError("unsupported stable consumer migration schema") + if self.revision < 1 or self.contract_version != "2.0.0": + raise ValueError("stable consumer migration revision/version is invalid") + if self.authority != "V1" or self.target_route != "V1_WITH_V2_SHADOW": + raise ValueError("stable consumer migration must preserve V1 authority") + if not self.consumers: + raise ValueError("stable consumer migration requires consumers") + identifiers = [item.consumer_id for item in self.consumers] + if len(identifiers) != len(set(identifiers)): + raise ValueError("stable consumer migration IDs must be unique") + for item in self.consumers: + if ( + item.state != "SHADOW" + or item.rollback_route != "V1" + or item.cutover_authorized + or item.manifest.rollback_contract != "V1" + or item.manifest.sdk_major != 2 + ): + raise ValueError("stable consumer migration is not fail-closed") + if item.consumer_id != item.manifest.consumer_id: + raise ValueError("stable consumer migration identity mismatch") + + @classmethod + def load( + cls, + path: str | Path, + *, + manifest_root: str | Path, + catalog: StableSourceCatalog, + ) -> "StableConsumerMigrationPlan": + payload = yaml.safe_load(Path(path).read_text(encoding="utf-8")) + return cls.from_mapping( + payload, + manifest_root=manifest_root, + catalog=catalog, + ) + + @classmethod + def from_mapping( + cls, + payload: Any, + *, + manifest_root: str | Path, + catalog: StableSourceCatalog, + ) -> "StableConsumerMigrationPlan": + if not isinstance(payload, dict): + raise ValueError("stable consumer migration plan must be a mapping") + expected = { + "schema", "revision", "contract_version", "authority", + "target_route", "consumers", + } + if set(payload) != expected: + raise ValueError("stable consumer migration fields are incomplete or unknown") + consumers_raw = payload["consumers"] + if not isinstance(consumers_raw, list) or not 1 <= len(consumers_raw) <= 10_000: + raise ValueError("stable consumer migration requires 1..10000 consumers") + root = Path(manifest_root).resolve() + consumers: list[StableConsumerMigration] = [] + for item in consumers_raw: + if not isinstance(item, dict) or set(item) != { + "manifest", "consumer_id", "state", "rollback_route", + "cutover_authorized", + }: + raise ValueError("stable consumer migration item is incomplete or unknown") + declared = Path(str(item["manifest"])) + relative = ( + Path(*declared.parts[2:]) + if declared.is_absolute() and len(declared.parts) > 1 + and declared.parts[1] == "app" + else declared + ) + manifest_path = (root / relative).resolve() + try: + manifest_path.relative_to(root) + except ValueError as error: + raise ValueError("stable consumer manifest escapes repository root") from error + manifest = ConsumerManifestLoader.load(manifest_path) + for requirement in manifest.requirements: + catalog.binding_for(requirement) + consumers.append(StableConsumerMigration( + consumer_id=str(item["consumer_id"]), + manifest_path=manifest_path, + manifest=manifest, + state=str(item["state"]).upper(), + rollback_route=str(item["rollback_route"]).upper(), + cutover_authorized=item["cutover_authorized"], + )) + if any(not isinstance(item.cutover_authorized, bool) for item in consumers): + raise ValueError("stable consumer cutover_authorized must be boolean") + return cls( + schema=str(payload["schema"]), + revision=int(payload["revision"]), + contract_version=str(payload["contract_version"]), + authority=str(payload["authority"]).upper(), + target_route=str(payload["target_route"]).upper(), + consumers=tuple(consumers), + ) diff --git a/qdl/domain/calendar.py b/qdl/domain/calendar.py index 2c18d29..2af3354 100644 --- a/qdl/domain/calendar.py +++ b/qdl/domain/calendar.py @@ -51,3 +51,18 @@ def session_at_ns(self, timestamp_ns: int) -> MarketSession | None: def is_open_ns(self, timestamp_ns: int) -> bool: return self.session_at_ns(timestamp_ns) is not None + + +def trading_calendar_for_id(calendar_id: str) -> TradingCalendar: + """Resolve a governed session calendar from canonical instrument metadata.""" + if calendar_id.startswith("VN_"): + return TradingCalendar( + calendar_id=calendar_id, + timezone_name="Asia/Ho_Chi_Minh", + weekly_sessions=( + MarketSession(time(9, 0), time(11, 30), "MORNING"), + MarketSession(time(13, 0), time(14, 30), "AFTERNOON"), + MarketSession(time(14, 45), time(14, 46), "ATC_CLOSE"), + ), + ) + raise ValueError(f"unknown governed trading calendar: {calendar_id}") diff --git a/qdl/domain/instrument.py b/qdl/domain/instrument.py index c13b9e9..7fe9944 100644 --- a/qdl/domain/instrument.py +++ b/qdl/domain/instrument.py @@ -113,8 +113,18 @@ def __post_init__(self) -> None: if self.identity.product_type is ProductType.OPTION: if self.expiry_time_ns is None or self.strike_price is None or self.option_type is None: raise ValueError("options require expiry, strike and option_type") - if self.identity.product_type is ProductType.FUTURE and self.expiry_time_ns is None: + continuous_future = ( + self.identity.product_type is ProductType.FUTURE + and self.attributes.get("continuous_series", "false").lower() == "true" + ) + if ( + self.identity.product_type is ProductType.FUTURE + and self.expiry_time_ns is None + and not continuous_future + ): raise ValueError("dated futures require expiry_time_ns") + if continuous_future and self.expiry_time_ns is not None: + raise ValueError("continuous futures cannot carry a fixed expiry_time_ns") @property def instrument_uid(self) -> str: diff --git a/qdl/domain/quantity.py b/qdl/domain/quantity.py new file mode 100644 index 0000000..9517dc2 --- /dev/null +++ b/qdl/domain/quantity.py @@ -0,0 +1,77 @@ +from __future__ import annotations + +from enum import StrEnum + +from qdl.common.v1 import common_pb2 + + +class QuantityUnit(StrEnum): + BASE_ASSET = "BASE_ASSET" + QUOTE_ASSET = "QUOTE_ASSET" + CONTRACT = "CONTRACT" + SHARE = "SHARE" + + @property + def proto(self) -> int: + return getattr(common_pb2, f"QUANTITY_UNIT_{self.value}") + + +def resolve_quantity_unit( + *, venue: str, market: str, product_type: str +) -> QuantityUnit: + """Resolve the venue-native order/volume unit without silent defaults.""" + + venue_name = venue.strip().upper() + market_name = market.strip().upper() + product = product_type.strip().upper() + if not venue_name or not market_name or not product: + raise ValueError("quantity-unit identity is incomplete") + + if product == "COMMON_STOCK": + if venue_name in {"DNSE", "HOSE", "HNX", "UPCOM", "VN_MARKETS"}: + return QuantityUnit.SHARE + raise ValueError("COMMON_STOCK quantity unit requires a VN venue identity") + + if product == "FUTURE" and ( + venue_name in {"DNSE", "HOSE", "HNX", "UPCOM", "VN_MARKETS"} + or market_name in {"VN_DERIVATIVES", "DERIVATIVES"} + ): + return QuantityUnit.CONTRACT + + if venue_name == "OKX": + if market_name == "SPOT" and product == "SPOT": + return QuantityUnit.BASE_ASSET + if market_name in {"SWAP", "FUTURES", "OPTIONS"} and product in { + "PERPETUAL", "FUTURE", "OPTION", + }: + return QuantityUnit.CONTRACT + raise ValueError("unsupported OKX quantity-unit identity") + + if venue_name == "DERIBIT" and product == "OPTION": + return QuantityUnit.CONTRACT + + if venue_name == "BINANCE": + if market_name == "SPOT" and product == "SPOT": + return QuantityUnit.BASE_ASSET + if market_name == "USDM" and product == "PERPETUAL": + # Binance USD-M trade/book/kline quantity is expressed in base-asset + # units even though the instrument is a perpetual contract. + return QuantityUnit.BASE_ASSET + raise ValueError("unsupported Binance quantity-unit identity") + + raise ValueError( + f"quantity unit is undefined for {venue_name}/{market_name}/{product}" + ) + + +def quantity_unit_proto(*, venue: str, market: str, product_type: str) -> int: + return resolve_quantity_unit( + venue=venue, market=market, product_type=product_type + ).proto + + +def quantity_unit_name(value: int) -> str: + name = common_pb2.QuantityUnit.Name(value) + if name == "QUANTITY_UNIT_UNSPECIFIED": + raise ValueError("canonical quantity unit cannot be UNSPECIFIED") + return name.removeprefix("QUANTITY_UNIT_") diff --git a/qdl/projection/stable.py b/qdl/projection/stable.py new file mode 100644 index 0000000..6e4a8c1 --- /dev/null +++ b/qdl/projection/stable.py @@ -0,0 +1,420 @@ +from __future__ import annotations + +import hashlib +import json +import re +from dataclasses import dataclass +from typing import Protocol + +import redis + +from qdl.common.v1 import common_pb2 +from qdl.marketdata.v2 import market_data_pb2 +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.envelope import validate_raw_envelope +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.transport import StoredEvent + + +_KEY_PATTERNS = tuple(re.compile(value) for value in ( + r"trade:price:(?:last:)?(?:binance_usdm|binance_spot):[A-Z0-9_-]+", + r"trade:price:(?:last:)?[A-Z0-9_-]+", + r"kline:(?:last:)?[0-9]+[smhd]:[A-Z0-9_-]+", + r"vn:quote:(?:last:)?[A-Z0-9_-]+", +)) +_CHANNEL_PATTERNS = tuple(re.compile(value) for value in ( + r"stream:trade:(?:binance_usdm|binance_spot):[A-Z0-9_-]+", + r"stream:trade:[A-Z0-9_-]+", + r"stream:kline:[0-9]+[smhd]:[A-Z0-9_-]+", + r"stream:vn:[A-Z0-9_-]+", +)) + + +@dataclass(frozen=True, slots=True) +class StableProjectionItem: + key: str + payload: bytes + ttl_seconds: int = 0 + + def __post_init__(self) -> None: + if not self.key or not self.payload or self.ttl_seconds < 0: + raise ValueError("stable projection item is invalid") + + +@dataclass(frozen=True, slots=True) +class StableProjectionRecord: + partition_key: str + offset: int + event_id_hex: str + shard_id: str + lease_epoch: int + items: tuple[StableProjectionItem, ...] + publications: tuple[tuple[str, bytes], ...] = () + + def __post_init__(self) -> None: + if ( + not self.partition_key + or self.offset < 1 + or len(self.event_id_hex) != 32 + or not self.shard_id + or self.lease_epoch < 1 + or not self.items + ): + raise ValueError("stable projection record identity is invalid") + + +class ProjectionFenced(RuntimeError): + """Projection lease epoch is older than the committed Redis writer epoch.""" + + +class ProjectionCacheMismatch(RuntimeError): + """Redis and SQLite do not belong to the same rebuildable cache unit.""" + + +class StableProjectionTarget(Protocol): + def apply(self, record: StableProjectionRecord) -> bool: ... + def apply_many( + self, records: tuple[StableProjectionRecord, ...] | list[StableProjectionRecord] + ) -> tuple[bool, ...]: ... + + +class InMemoryStableProjectionTarget: + def __init__(self) -> None: + self.latest: dict[str, bytes] = {} + self.checkpoints: dict[str, tuple[int, str]] = {} + self.lease_epochs: dict[str, int] = {} + self.publications: list[tuple[str, bytes]] = [] + + def apply(self, record: StableProjectionRecord) -> bool: + return self.apply_many((record,))[0] + + def apply_many( + self, records: tuple[StableProjectionRecord, ...] | list[StableProjectionRecord] + ) -> tuple[bool, ...]: + results = [] + for record in records: + observed_epoch = self.lease_epochs.get(record.shard_id, 0) + if record.lease_epoch < observed_epoch: + raise ProjectionFenced("stable projection lease epoch is stale") + current = self.checkpoints.get(record.partition_key) + if current is not None and record.offset <= current[0]: + results.append(False) + continue + self.lease_epochs[record.shard_id] = record.lease_epoch + self.latest.update({item.key: item.payload for item in record.items}) + self.publications.extend(record.publications) + self.checkpoints[record.partition_key] = ( + record.offset, record.event_id_hex + ) + results.append(True) + return tuple(results) + + +_STABLE_CACHE_BIND_LUA = r""" +local current = redis.call('GET', KEYS[1]) +if current then + if current == ARGV[1] then + return 1 + end + return -1 +end +if ARGV[2] ~= '1' then + return 0 +end +redis.call('SET', KEYS[1], ARGV[1]) +return 1 +""" + + +_STABLE_APPLY_LUA = r""" +local cache_id = redis.call('GET', KEYS[3]) +if not cache_id or cache_id ~= ARGV[4] then + return -2 +end +local current_epoch = redis.call('GET', KEYS[2]) +if current_epoch and tonumber(current_epoch) > tonumber(ARGV[3]) then + return -1 +end +local current = redis.call('GET', KEYS[1]) +if current then + local current_offset = string.sub(current, 1, 20) + if current_offset >= ARGV[1] then + return 0 + end +end +local item_count = tonumber(ARGV[5]) +for index = 1, item_count do + local argument = 6 + ((index - 1) * 2) + local ttl = tonumber(ARGV[argument + 1]) + if ttl > 0 then + redis.call('SETEX', KEYS[index + 3], ttl, ARGV[argument]) + else + redis.call('SET', KEYS[index + 3], ARGV[argument]) + end +end +local channel_count_index = 6 + (item_count * 2) +local channel_count = tonumber(ARGV[channel_count_index]) +for index = 1, channel_count do + local argument = channel_count_index + 1 + ((index - 1) * 2) + redis.call('PUBLISH', ARGV[argument], ARGV[argument + 1]) +end +redis.call('SET', KEYS[2], ARGV[3]) +redis.call('SET', KEYS[1], ARGV[1] .. ':' .. ARGV[2]) +return 1 +""" + + +class RedisStableProjectionTarget: + """Atomic stable latest/V1 projection for a dedicated Redis database.""" + + def __init__( + self, + client: redis.Redis, + *, + namespace: str = "qdl:stable:v2", + dedicated_database: bool, + ) -> None: + self._client = client + self._namespace = namespace.rstrip(":") + self._cache_id: str | None = None + if not self._namespace or not dedicated_database: + raise ValueError("stable Redis projection requires an isolated database") + + @property + def cache_identity_key(self) -> str: + return f"{self._namespace}:projection-cache-id" + + def bind_cache(self, cache_id: str, *, initialize_if_missing: bool) -> None: + if len(cache_id) != 32 or any( + character not in "0123456789abcdef" for character in cache_id + ): + raise ValueError("stable projection cache identity is invalid") + result = int(self._client.eval( + _STABLE_CACHE_BIND_LUA, + 1, + self.cache_identity_key, + cache_id, + "1" if initialize_if_missing else "0", + )) + if result < 0: + raise ProjectionCacheMismatch( + "stable Redis and SQLite projection cache identities differ" + ) + if result == 0: + raise ProjectionCacheMismatch( + "stable Redis cache identity is missing for a non-empty spool" + ) + self._cache_id = cache_id + + def cache_is_bound(self) -> bool: + if self._cache_id is None: + return False + value = self._client.get(self.cache_identity_key) + if isinstance(value, bytes): + value = value.decode("ascii") + return value == self._cache_id + + def apply(self, record: StableProjectionRecord) -> bool: + return self.apply_many((record,))[0] + + def apply_many( + self, records: tuple[StableProjectionRecord, ...] | list[StableProjectionRecord] + ) -> tuple[bool, ...]: + values = tuple(records) + if not values: + return () + if self._cache_id is None: + raise ProjectionCacheMismatch("stable projection cache is not bound") + commands = [self._command(record) for record in values] + pipeline = self._client.pipeline(transaction=False) + for keys, arguments in commands: + pipeline.eval(_STABLE_APPLY_LUA, len(keys), *keys, *arguments) + raw_results = pipeline.execute() + results = [] + for result in raw_results: + value = int(result) + if value == -2: + raise ProjectionCacheMismatch( + "stable projection cache identity changed during operation" + ) + if value < 0: + raise ProjectionFenced("stable projection lease epoch is stale") + results.append(value > 0) + return tuple(results) + + def _command( + self, record: StableProjectionRecord + ) -> tuple[list[str], list[str | bytes]]: + for item in record.items: + if not ( + item.key.startswith(f"{self._namespace}:") + or any(pattern.fullmatch(item.key) for pattern in _KEY_PATTERNS) + ): + raise ValueError("stable projection key escapes its allowlist") + for channel, payload in record.publications: + if not payload or not any( + pattern.fullmatch(channel) for pattern in _CHANNEL_PATTERNS + ): + raise ValueError("stable compatibility channel escapes its allowlist") + partition_digest = hashlib.sha256(record.partition_key.encode()).hexdigest() + shard_digest = hashlib.sha256(record.shard_id.encode()).hexdigest() + keys = [ + f"{self._namespace}:checkpoint:{partition_digest}", + f"{self._namespace}:lease-epoch:{shard_digest}", + self.cache_identity_key, + *(item.key for item in record.items), + ] + arguments: list[str | bytes] = [ + f"{record.offset:020d}", + record.event_id_hex, + str(record.lease_epoch), + self._cache_id or "", + str(len(record.items)), + ] + for item in record.items: + arguments.extend((item.payload, str(item.ttl_seconds))) + arguments.append(str(len(record.publications))) + for channel, payload in record.publications: + arguments.extend((channel, payload)) + return keys, arguments + + +class StableCompatibilityProjector: + def __init__( + self, + catalog: StableSourceCatalog, + *, + namespace: str = "qdl:stable:v2", + latest_ttl_seconds: int = 60, + ) -> None: + if latest_ttl_seconds <= 0: + raise ValueError("stable compatibility TTL must be positive") + self.catalog = catalog + self.namespace = namespace.rstrip(":") + self.latest_ttl_seconds = latest_ttl_seconds + + def build( + self, + stored: StoredEvent, + raw_envelope_bytes: bytes, + ) -> StableProjectionRecord: + envelope = market_data_pb2.EventEnvelope.FromString(stored.event.payload) + binding = self.catalog.binding_for_envelope(envelope) + raw = raw_provider_pb2.RawProviderEnvelope.FromString(raw_envelope_bytes) + validate_raw_envelope(raw) + if ( + bytes(raw.capture_id) != bytes(envelope.raw_capture_id) + or bytes(raw.raw_frame_sha256) != bytes(envelope.raw_payload_hash) + or raw.provider != envelope.provider + or raw.venue != envelope.venue + or raw.market != envelope.market + or raw.native_symbol != envelope.native_symbol + or raw.source_session_id != envelope.source_session_id + or raw.connection_generation != envelope.connection_generation + or raw.authority_revision != envelope.authority_revision + ): + raise ValueError("canonical/raw stable lineage mismatch") + raw_json = json.loads(bytes(raw.raw_frame_bytes)) + if not isinstance(raw_json, dict): + raise ValueError("stable compatibility raw frame must be a JSON object") + feed = envelope.WhichOneof("payload") + items = [StableProjectionItem( + key=( + f"{self.namespace}:latest:{feed}:{envelope.venue.lower()}:" + f"{envelope.market.lower()}:{envelope.instrument_uid}" + ), + payload=stored.event.payload, + )] + publications: list[tuple[str, bytes]] = [] + self._legacy(binding, envelope, raw_json, items, publications) + return StableProjectionRecord( + partition_key=stored.cursor.partition_key, + offset=stored.cursor.offset, + event_id_hex=stored.event.event_id.hex(), + shard_id=envelope.source_id, + lease_epoch=envelope.lease_epoch, + items=tuple(items), + publications=tuple(publications), + ) + + def _legacy(self, binding, envelope, raw, items, publications) -> None: + policy = binding.v1_compatibility + if policy == "NONE": + return + symbol = envelope.native_symbol.upper() + if policy.startswith("BINANCE_TRADE"): + market = { + "USDM": "binance_usdm", + "SPOT": "binance_spot", + }[envelope.market] + trade = envelope.trade + buyer_maker = trade.aggressor_side == common_pb2.AGGRESSOR_SIDE_SELL + payload = json.dumps({ + "authoritative": True, + "event_time": envelope.source_event_time_ns // 1_000_000, + "is_live": True, + "market": market, + "price": float(trade.price.source_text), + "provider": "binance", + "quantity": float(trade.quantity.source_text), + "raw": raw, + "side": "sell" if buyer_maker else "buy", + "source": f"{market}_trade", + "symbol": symbol, + "trade_id": int(trade.native_trade_id) if trade.native_trade_id.isdigit() else 0, + "trade_time": envelope.source_event_time_ns // 1_000_000, + }, sort_keys=True, separators=(",", ":")).encode() + self._current_and_last( + items, f"trade:price:{market}:{symbol}", + f"trade:price:last:{market}:{symbol}", payload, + ) + publications.append((f"stream:trade:{market}:{symbol}", payload)) + if policy == "BINANCE_TRADE_MARKET_AND_GENERIC": + self._current_and_last( + items, f"trade:price:{symbol}", f"trade:price:last:{symbol}", payload, + ) + publications.append((f"stream:trade:{symbol}", payload)) + return + if policy == "BINANCE_BAR_GENERIC": + bar = envelope.bar + payload = json.dumps({ + "e": "kline", + "E": envelope.source_event_time_ns // 1_000_000, + "s": symbol, + "k": { + "t": bar.open_time_ns // 1_000_000, + "T": bar.close_time_ns // 1_000_000, + "s": symbol, + "i": bar.interval, + "o": bar.open.source_text, + "h": bar.high.source_text, + "l": bar.low.source_text, + "c": bar.close.source_text, + "v": bar.volume.source_text, + "n": bar.trade_count, + "x": bool(bar.is_final), + "q": bar.quote_volume.source_text if bar.HasField("quote_volume") else None, + }, + }, sort_keys=True, separators=(",", ":")).encode() + self._current_and_last( + items, f"kline:{bar.interval}:{symbol}", + f"kline:last:{bar.interval}:{symbol}", payload, + ) + publications.append((f"stream:kline:{bar.interval}:{symbol}", payload)) + return + if policy == "VN_TRADE_GENERIC": + if str(raw.get("symbol", "")).upper() != symbol: + raise ValueError("VN raw symbol differs from canonical instrument") + payload = json.dumps(raw, sort_keys=True, separators=(",", ":")).encode() + self._current_and_last( + items, f"vn:quote:{symbol}", f"vn:quote:last:{symbol}", payload, + ) + publications.append((f"stream:vn:{symbol}", payload)) + return + raise ValueError("unsupported stable V1 compatibility policy") + + def _current_and_last(self, items, current, last, payload) -> None: + items.extend(( + StableProjectionItem(current, payload, self.latest_ttl_seconds), + StableProjectionItem(last, payload), + )) diff --git a/qdl/query/service.py b/qdl/query/service.py index 66a851f..787f52b 100644 --- a/qdl/query/service.py +++ b/qdl/query/service.py @@ -9,6 +9,7 @@ BarRevisionPolicy, CanonicalErrorCode, CoverageStatus, + ConsumerGrade, DataRequirement, FeedType, QueryProblem, @@ -295,12 +296,21 @@ def _enforce( problem = evaluate_requirement( requirement, entitled=entitlement.allowed, - available=quality.state not in {"OFFLINE", "UNAVAILABLE"}, + available=( + quality.state not in {"OFFLINE", "UNAVAILABLE"} + and not ( + requirement.consumer_grade is ConsumerGrade.EXECUTION + and quality.state == "MARKET_CLOSED" + ) + ), fresh=( - quality.state not in {"STALE", "OFFLINE"} - and ( - requirement.max_freshness_ms is None - or quality.freshness_ms <= requirement.max_freshness_ms + quality.state == "MARKET_CLOSED" + or ( + quality.state not in {"STALE", "OFFLINE"} + and ( + requirement.max_freshness_ms is None + or quality.freshness_ms <= requirement.max_freshness_ms + ) ) ), authoritative=( diff --git a/qdl/runtime/canary_source.py b/qdl/runtime/canary_source.py index 190fbcf..a9958a4 100644 --- a/qdl/runtime/canary_source.py +++ b/qdl/runtime/canary_source.py @@ -18,6 +18,7 @@ InstrumentRegistry, ProductType, ) +from qdl.domain.quantity import quantity_unit_name from qdl.marketdata.v2 import market_data_pb2 from qdl.query import ( AccessPurpose, @@ -533,6 +534,22 @@ def _item( "low": _decimal_text(bar.low), "close": _decimal_text(bar.close), "volume": _decimal_text(bar.volume), + "volume_unit": quantity_unit_name(bar.volume_unit), + "base_volume": ( + _decimal_text(bar.base_volume) + if bar.HasField("base_volume") + else None + ), + "quote_volume": ( + _decimal_text(bar.quote_volume) + if bar.HasField("quote_volume") + else None + ), + "contract_volume": ( + _decimal_text(bar.contract_volume) + if bar.HasField("contract_volume") + else None + ), "trade_count": int(bar.trade_count), "origin": origin_name, "is_final": bool(bar.is_final), diff --git a/qdl/runtime/stable.py b/qdl/runtime/stable.py new file mode 100644 index 0000000..730632c --- /dev/null +++ b/qdl/runtime/stable.py @@ -0,0 +1,656 @@ +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +import os +import time +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Mapping + +import grpc +import redis +import uvicorn +from fastapi import FastAPI +from fastapi.responses import JSONResponse +from redis.asyncio import Redis as AsyncRedis + +from qdl.api_v2 import create_v2_app +from qdl.consumer import ConsumerManifestLoader, ConsumerManifestRegistry +from qdl.projection.stable import RedisStableProjectionTarget, StableCompatibilityProjector +from qdl.replay import GapFreeHandoff, SignedHandoffCursorCodec +from qdl.runtime.bounds import BoundedRequestMiddleware, RequestBounds +from qdl.runtime.lease import ActivePassiveGatewayLease, RedisGatewayLeaseStore +from qdl.runtime.readiness import ( + CallableReadinessProbe, + ComponentReadiness, + ComponentState, + MeasuredRuntimeReadiness, +) +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_ingest import ( + StableHttpCanonicalSink, + install_stable_canonical_ingest, +) +from qdl.runtime.stable_projector import ( + StableProjectorEngine, + supervise_stable_projector, +) +from qdl.runtime.stable_source import ( + StableCatalogCursorScopeValidator, + StableGrpcSnapshotLoader, + build_stable_query_stack, +) +from qdl.security import ( + AuditChain, + DataPlaneIdentityService, + DataPlaneSecurityConfig, + RedisMinuteQuota, +) +from qdl.stream import DurableStreamGateway, GrpcMarketDataService, create_grpc_server +from qdl.transport import SQLiteDurableSpool, SpoolConfig +from qdl.transport.kafka_projector import ( + ConfluentProjectorBroker, + KafkaProjectorConfig, +) + + +logger = logging.getLogger(__name__) + + +@dataclass(frozen=True, slots=True) +class StableRuntimeConfig: + role: str + instance_id: str + environment: str + config_revision: str + authority_mode: str + authority_revision: int + schema_digest: str + state_dir: Path + durable_state_dir: Path + audit_path: Path + manifest_paths: tuple[Path, ...] + source_bindings_path: Path + internal_ingest_secret: bytes + redis_url: str + redis_prefix: str + consumer_group: str + cursor_keys: Mapping[str, bytes] + active_cursor_key_id: str + cursor_ttl_seconds: int + http_port: int + grpc_port: int + max_request_bytes: int + max_concurrent_requests: int + max_concurrent_rpcs: int + max_streams: int + max_buffer_events: int + max_replay_events: int + lease_shard_id: str + lease_ttl_seconds: int + lease_renew_seconds: float + kafka_bootstrap_servers: str | None = None + kafka_client_id: str | None = None + kafka_raw_topics: tuple[str, ...] = () + kafka_canonical_topic: str | None = None + kafka_cert_root: Path | None = None + stream_ingest_urls: tuple[str, ...] = () + max_pending_records: int = 10_000 + max_pending_bytes: int = 256 * 1024 * 1024 + + def __post_init__(self) -> None: + if self.role not in {"query_v2", "stream_v2", "projector_v2"}: + raise ValueError("stable role is invalid") + if self.authority_mode != "RUST_SHADOW": + raise ValueError("Phase B stable runtime must remain RUST_SHADOW") + if not all(( + self.instance_id, self.environment, self.config_revision, + self.redis_url, self.redis_prefix, self.consumer_group, + )): + raise ValueError("stable runtime identity/isolation fields are required") + if not self.redis_prefix.startswith("qdl:stable:v2:"): + raise ValueError("stable Redis prefix must start with qdl:stable:v2:") + if len(self.schema_digest) != 64 or any( + value not in "0123456789abcdef" for value in self.schema_digest + ): + raise ValueError("stable schema digest must be lowercase SHA-256") + if self.authority_revision < 1 or not self.manifest_paths: + raise ValueError("stable authority revision and consumer manifests are required") + if not self.source_bindings_path.is_file(): + raise ValueError("stable source binding catalog is unavailable") + if len(self.internal_ingest_secret) < 32: + raise ValueError("stable internal ingest secret must contain 256 bits") + if self.active_cursor_key_id not in self.cursor_keys or any( + len(value) < 32 for value in self.cursor_keys.values() + ): + raise ValueError("stable active cursor key is unavailable or weak") + if not 60 <= self.cursor_ttl_seconds <= 86_400: + raise ValueError("stable cursor TTL must be 60..86400 seconds") + if min( + self.http_port, self.grpc_port, self.max_request_bytes, + self.max_concurrent_requests, self.max_concurrent_rpcs, + self.max_streams, self.max_buffer_events, self.max_replay_events, + self.max_pending_records, self.max_pending_bytes, + ) <= 0: + raise ValueError("stable runtime ports/bounds must be positive") + if self.max_buffer_events > 10_000 or self.max_replay_events > 10_000: + raise ValueError("stable stream/replay bounds exceed contract maximum") + if not 5 <= self.lease_ttl_seconds <= 300 or not ( + 0 < self.lease_renew_seconds < self.lease_ttl_seconds + ): + raise ValueError("stable gateway lease policy is invalid") + if self.durable_state_dir.resolve() == Path("/app").resolve(): + raise ValueError("stable durable state cannot use application source") + try: + self.audit_path.resolve().relative_to(self.state_dir.resolve()) + except ValueError as error: + raise ValueError("stable audit path must remain inside stable state") from error + if self.role == "projector_v2": + if not all(( + self.kafka_bootstrap_servers, self.kafka_client_id, + self.kafka_canonical_topic, self.kafka_cert_root, + )) or not self.kafka_raw_topics or not self.stream_ingest_urls: + raise ValueError("stable projector Kafka/stream dependencies are required") + + @classmethod + def from_environment( + cls, role: str, values: Mapping[str, str] | None = None + ) -> "StableRuntimeConfig": + env = os.environ if values is None else values + cursor_raw = json.loads(env["QDL_STABLE_CURSOR_KEYS_JSON"]) + if not isinstance(cursor_raw, dict) or not cursor_raw: + raise ValueError("QDL_STABLE_CURSOR_KEYS_JSON must be a non-empty object") + manifests = tuple( + Path(value) for value in env["QDL_STABLE_CONSUMER_MANIFESTS"].split(":") + if value + ) + state_dir = Path(env.get("QDL_STABLE_STATE_DIR", "/var/lib/qdl-stable")) + cert_root_raw = env.get("QDL_STABLE_KAFKA_CERT_ROOT", "").strip() + urls_raw = json.loads(env.get("QDL_STABLE_STREAM_INGEST_URLS_JSON", "[]")) + if not isinstance(urls_raw, list): + raise ValueError("QDL_STABLE_STREAM_INGEST_URLS_JSON must be an array") + instance_id = env.get("QDL_STABLE_INSTANCE_ID", f"stable-{role}-local") + return cls( + role=role, + instance_id=instance_id, + environment=env.get("QDL_ENVIRONMENT", "paper").lower(), + config_revision=env["QDL_CONFIG_REVISION"], + authority_mode=env.get("QDL_STABLE_AUTHORITY_MODE", "RUST_SHADOW"), + authority_revision=int(env["QDL_STABLE_AUTHORITY_REVISION"]), + schema_digest=env["QDL_STABLE_SCHEMA_DIGEST"], + state_dir=state_dir, + durable_state_dir=Path(env.get( + "QDL_STABLE_DURABLE_STATE_DIR", str(state_dir / "shared") + )), + audit_path=Path(env.get( + "QDL_STABLE_AUDIT_PATH", str(state_dir / f"{instance_id}-audit.jsonl") + )), + manifest_paths=manifests, + source_bindings_path=Path(env["QDL_STABLE_SOURCE_BINDINGS"]), + internal_ingest_secret=env["QDL_STABLE_INTERNAL_INGEST_SECRET"].encode(), + redis_url=env["QDL_STABLE_REDIS_URL"], + redis_prefix=env["QDL_STABLE_REDIS_PREFIX"], + consumer_group=env["QDL_STABLE_CONSUMER_GROUP"], + cursor_keys={str(key): str(value).encode() for key, value in cursor_raw.items()}, + active_cursor_key_id=env["QDL_STABLE_CURSOR_ACTIVE_KEY_ID"], + cursor_ttl_seconds=int(env.get("QDL_STABLE_CURSOR_TTL_SECONDS", "3600")), + http_port=int(env.get("QDL_STABLE_HTTP_PORT", "18200")), + grpc_port=int(env.get("QDL_STABLE_GRPC_PORT", "18210")), + max_request_bytes=int(env.get("QDL_STABLE_MAX_REQUEST_BYTES", "1048576")), + max_concurrent_requests=int(env.get("QDL_STABLE_MAX_CONCURRENT_REQUESTS", "200")), + max_concurrent_rpcs=int(env.get("QDL_STABLE_MAX_CONCURRENT_RPCS", "200")), + max_streams=int(env.get("QDL_STABLE_MAX_STREAMS", "1000")), + max_buffer_events=int(env.get("QDL_STABLE_MAX_BUFFER_EVENTS", "2000")), + max_replay_events=int(env.get("QDL_STABLE_MAX_REPLAY_EVENTS", "5000")), + lease_shard_id=env.get("QDL_STABLE_LEASE_SHARD_ID", "stable-stream-v2"), + lease_ttl_seconds=int(env.get("QDL_STABLE_LEASE_TTL_SECONDS", "15")), + lease_renew_seconds=float(env.get("QDL_STABLE_LEASE_RENEW_SECONDS", "5")), + kafka_bootstrap_servers=env.get("QDL_STABLE_KAFKA_BOOTSTRAP_SERVERS"), + kafka_client_id=env.get("QDL_STABLE_KAFKA_CLIENT_ID"), + kafka_raw_topics=tuple( + value for value in env.get("QDL_STABLE_KAFKA_RAW_TOPICS", "").split(",") + if value + ), + kafka_canonical_topic=env.get("QDL_STABLE_KAFKA_CANONICAL_TOPIC"), + kafka_cert_root=Path(cert_root_raw) if cert_root_raw else None, + stream_ingest_urls=tuple(str(value) for value in urls_raw), + max_pending_records=int(env.get("QDL_STABLE_MAX_PENDING_RECORDS", "10000")), + max_pending_bytes=int(env.get("QDL_STABLE_MAX_PENDING_BYTES", "268435456")), + ) + + def public_manifest(self) -> dict[str, object]: + return { + "contract_version": "2.0.0", + "role": self.role, + "instance_id": self.instance_id, + "environment": self.environment, + "config_revision": self.config_revision, + "authority": self.authority_mode, + "authority_revision": self.authority_revision, + "schema_digest": self.schema_digest, + "redis_prefix": self.redis_prefix, + "consumer_group": self.consumer_group, + "owns_venue_connections": False, + "writes_current_v1_redis": False, + "compatibility_projection": "DEDICATED_REDIS_ONLY", + "replay_authority": "KAFKA", + "query_cache_authority": False, + } + + +def _ready(name: str, *, detail: str, revision: str | None = None) -> ComponentReadiness: + return ComponentReadiness( + name, ComponentState.READY, detail=detail, revision=revision, + checked_at_ns=time.time_ns(), + ) + + +def load_stable_manifests(config: StableRuntimeConfig) -> ConsumerManifestRegistry: + manifests = tuple(ConsumerManifestLoader.load(path) for path in config.manifest_paths) + if any(item.environment != config.environment for item in manifests): + raise ValueError("stable consumer manifest environment mismatch") + return ConsumerManifestRegistry(manifests) + + +def build_stable_identity( + config: StableRuntimeConfig, manifests: ConsumerManifestRegistry +) -> DataPlaneIdentityService: + security = DataPlaneSecurityConfig.from_environment() + if security.environment != config.environment: + raise ValueError("stable JWT environment mismatch") + quota = RedisMinuteQuota.from_url( + config.redis_url, prefix=f"{config.redis_prefix}:identity" + ) + return DataPlaneIdentityService(security, manifests, quota=quota) + + +def build_stable_spool(config: StableRuntimeConfig) -> SQLiteDurableSpool: + config.durable_state_dir.mkdir(parents=True, exist_ok=True) + return SQLiteDurableSpool(SpoolConfig( + path=config.durable_state_dir / "canonical-cache.sqlite3", + max_records=1_000_000, + max_payload_bytes=2 * 1024 * 1024 * 1024, + max_storage_bytes=3 * 1024 * 1024 * 1024, + max_partitions=100_000, + max_consumer_checkpoints=100_000, + min_free_disk_bytes=512 * 1024 * 1024, + consumer_ttl_seconds=config.cursor_ttl_seconds, + replay_retention_seconds=24 * 3600, + max_partition_records=10_000, + )) + + +def build_stable_handoff( + config: StableRuntimeConfig, spool: SQLiteDurableSpool +) -> GapFreeHandoff: + return GapFreeHandoff( + spool, + SignedHandoffCursorCodec( + config.cursor_keys, active_key_id=config.active_cursor_key_id + ), + checkpoint_ttl_seconds=config.cursor_ttl_seconds, + ) + + +def stable_readiness( + config: StableRuntimeConfig, + manifests: ConsumerManifestRegistry, + spool: SQLiteDurableSpool, + *, + quota: RedisMinuteQuota | None = None, + extra_probes=(), +) -> MeasuredRuntimeReadiness: + async def cache(): + stats = await asyncio.to_thread(spool.stats) + return _ready( + "query_cache", + detail=f"bounded rebuildable cache records={stats.records} utilization={stats.utilization:.6f}", + ) + + async def redis_probe(): + if quota is None or not await asyncio.to_thread(quota.ping): + return ComponentReadiness( + "dedicated_redis", ComponentState.NOT_READY, + detail="stable dedicated Redis is unavailable", checked_at_ns=time.time_ns(), + ) + return _ready("dedicated_redis", detail="stable isolated Redis reachable") + + probes = ( + CallableReadinessProbe("consumer_manifests", lambda: _ready( + "consumer_manifests", detail=f"registered={manifests.count}", + revision=str(max(item[1] for item in manifests.revisions)), + )), + CallableReadinessProbe("query_cache", cache), + CallableReadinessProbe("dedicated_redis", redis_probe), + CallableReadinessProbe("authority_manifest", lambda: _ready( + "authority_manifest", detail=config.authority_mode, + revision=str(config.authority_revision), + )), + *tuple(extra_probes), + ) + return MeasuredRuntimeReadiness( + role=config.role, + authority=config.authority_mode, + config_revision=config.config_revision, + probes=probes, + ) + + +def install_stable_health(app, readiness, manifest) -> None: + @app.get("/health/live", include_in_schema=False) + async def live(): + return {"status": "alive", "runtime": manifest} + + @app.get("/health/ready", include_in_schema=False) + async def ready(): + snapshot = await readiness.snapshot() + return JSONResponse( + status_code=200 if snapshot.ready else 503, + content={ + "status": snapshot.status, "ready": snapshot.ready, + "role": snapshot.role, "authority": snapshot.authority, + "config_revision": snapshot.config_revision, + }, + ) + + @app.get("/health/dependencies", include_in_schema=False) + async def dependencies(): + snapshot = await readiness.snapshot() + return { + "status": snapshot.status, "ready": snapshot.ready, + "components": [{**asdict(item), "state": item.state.value} for item in snapshot.components], + } + + +def create_stable_query_app(config: StableRuntimeConfig | None = None) -> FastAPI: + config = config or StableRuntimeConfig.from_environment("query_v2") + config.state_dir.mkdir(parents=True, exist_ok=True) + manifests = load_stable_manifests(config) + identity = build_stable_identity(config, manifests) + spool = build_stable_spool(config) + catalog = StableSourceCatalog.load(config.source_bindings_path) + handoff = build_stable_handoff(config, spool) + service, _backend, issuer = build_stable_query_stack( + spool=spool, catalog=catalog, schema_digest=config.schema_digest, + handoff=handoff, cursor_ttl_seconds=config.cursor_ttl_seconds, + ) + readiness = stable_readiness( + config, manifests, spool, quota=identity.quota, + extra_probes=(CallableReadinessProbe("instrument_catalog", lambda: _ready( + "instrument_catalog", detail=f"bindings={len(catalog.bindings)}", + revision=str(catalog.catalog_revision), + )),), + ) + app = create_v2_app( + service, identity_service=identity, readiness_service=readiness, + cursor_issuer=issuer, + request_bounds=RequestBounds( + max_request_bytes=config.max_request_bytes, + max_concurrent_requests=config.max_concurrent_requests, + ), + contract_version="2.0.0", authority="INTERNAL_STABLE", + ) + app.state.runtime_manifest = config.public_manifest() + app.state.stable_spool = spool + app.state.stable_audit = AuditChain(config.audit_path) + install_stable_health(app, readiness, config.public_manifest()) + + @app.on_event("shutdown") + async def close_stable_query(): + await asyncio.to_thread(spool.close) + await asyncio.to_thread(identity.quota.close) + + return app + + +@dataclass(slots=True) +class StableStreamRuntime: + config: StableRuntimeConfig + redis: AsyncRedis + spool: SQLiteDurableSpool + gateway: DurableStreamGateway + lease: ActivePassiveGatewayLease + grpc_server: grpc.aio.Server + health_app: FastAPI + quota: RedisMinuteQuota + + async def start(self) -> None: + await self.redis.ping() + await self.lease.start() + self.grpc_server.add_insecure_port(f"0.0.0.0:{self.config.grpc_port}") + await self.grpc_server.start() + + async def stop(self) -> None: + await self.grpc_server.stop(grace=2.0) + await self.lease.stop() + await self.redis.aclose() + await asyncio.to_thread(self.quota.close) + await asyncio.to_thread(self.spool.close) + + +def create_stable_stream_runtime( + config: StableRuntimeConfig | None = None, +) -> StableStreamRuntime: + config = config or StableRuntimeConfig.from_environment("stream_v2") + config.state_dir.mkdir(parents=True, exist_ok=True) + manifests = load_stable_manifests(config) + identity = build_stable_identity(config, manifests) + spool = build_stable_spool(config) + handoff = build_stable_handoff(config, spool) + catalog = StableSourceCatalog.load(config.source_bindings_path) + async_redis = AsyncRedis.from_url(config.redis_url, decode_responses=True) + lease = ActivePassiveGatewayLease( + RedisGatewayLeaseStore(async_redis, prefix=config.redis_prefix), + shard_id=config.lease_shard_id, owner_id=config.instance_id, + ttl_seconds=config.lease_ttl_seconds, + renew_interval_seconds=config.lease_renew_seconds, + ) + gateway = DurableStreamGateway( + handoff=handoff, sink=spool, max_subscribers=config.max_streams, + max_buffer_events=config.max_buffer_events, + max_replay_events=config.max_replay_events, + cursor_ttl_seconds=config.cursor_ttl_seconds, authority=lease, + ) + lease.on_fenced = gateway.fence_all + query_service, backend, issuer = build_stable_query_stack( + spool=spool, catalog=catalog, schema_digest=config.schema_digest, + handoff=handoff, cursor_ttl_seconds=config.cursor_ttl_seconds, + ) + grpc_service = GrpcMarketDataService( + gateway=gateway, query_service=query_service, + snapshot_loader=StableGrpcSnapshotLoader( + service=query_service, backend=backend, issuer=issuer + ), + cursor_scope_validator=StableCatalogCursorScopeValidator(catalog), + ) + grpc_server = create_grpc_server( + grpc_service, identity_service=identity, + maximum_concurrent_rpcs=config.max_concurrent_rpcs, + max_receive_message_bytes=config.max_request_bytes, + ) + readiness = stable_readiness( + config, manifests, spool, quota=identity.quota, + extra_probes=(CallableReadinessProbe("gateway_lease", lease.readiness),), + ) + app = FastAPI( + title="Quant Data Layer V2 Stable Stream Health", version="2.0.0", + docs_url=None, redoc_url=None, openapi_url=None, + ) + app.add_middleware( + BoundedRequestMiddleware, + bounds=RequestBounds( + max_request_bytes=config.max_request_bytes, + max_concurrent_requests=config.max_concurrent_requests, + ), + ) + install_stable_health(app, readiness, config.public_manifest()) + install_stable_canonical_ingest( + app, gateway=gateway, catalog=catalog, spool=spool, + secret=config.internal_ingest_secret, + ) + return StableStreamRuntime( + config, async_redis, spool, gateway, lease, grpc_server, app, identity.quota + ) + + +async def serve_stable_stream() -> None: + runtime = create_stable_stream_runtime() + await runtime.start() + server = uvicorn.Server(uvicorn.Config( + runtime.health_app, host="0.0.0.0", port=runtime.config.http_port, + log_level="info", access_log=False, + )) + try: + await server.serve() + finally: + await runtime.stop() + + +async def serve_stable_projector() -> None: + config = StableRuntimeConfig.from_environment("projector_v2") + config.state_dir.mkdir(parents=True, exist_ok=True) + manifests = load_stable_manifests(config) + spool = build_stable_spool(config) + catalog = StableSourceCatalog.load(config.source_bindings_path) + assert config.kafka_cert_root is not None + assert config.kafka_bootstrap_servers is not None + assert config.kafka_client_id is not None + assert config.kafka_canonical_topic is not None + broker_config = KafkaProjectorConfig( + bootstrap_servers=config.kafka_bootstrap_servers, + client_id=config.kafka_client_id, + group_id=config.consumer_group, + raw_topics=(), # Rust canonical records carry private raw lineage. + canonical_topic=config.kafka_canonical_topic, + ca_path=config.kafka_cert_root / "ca.crt", + certificate_path=config.kafka_cert_root / "client.crt", + key_path=config.kafka_cert_root / "client.key", + checkpoint_batch_size=512, + checkpoint_interval_ms=100, + ) + redis_client = redis.Redis.from_url(config.redis_url) + quota = RedisMinuteQuota.from_url( + config.redis_url, prefix=f"{config.redis_prefix}:projector" + ) + sink = StableHttpCanonicalSink( + config.stream_ingest_urls, config.internal_ingest_secret, spool + ) + projector = StableCompatibilityProjector( + catalog, namespace=config.redis_prefix.rstrip(":") + ) + target = RedisStableProjectionTarget( + redis_client, + namespace=config.redis_prefix.rstrip(":"), + dedicated_database=True, + ) + spool_stats = await asyncio.to_thread(spool.stats) + await asyncio.to_thread( + target.bind_cache, + spool.cache_id, + initialize_if_missing=spool_stats.records == 0, + ) + active_broker: list[ConfluentProjectorBroker | None] = [None] + + def broker_factory(): + broker = ConfluentProjectorBroker(broker_config) + return broker, StableProjectorEngine( + broker=broker, + spool=spool, + catalog=catalog, + canonical_topic=config.kafka_canonical_topic, + raw_topics=(), # Rust canonical records carry private raw lineage. + sink=sink, + projector=projector, + target=target, + max_pending_records=config.max_pending_records, + max_pending_bytes=config.max_pending_bytes, + max_batch_records=512, + batch_wait_seconds=0.01, + ) + + def on_broker(broker): + active_broker[0] = broker + + async def kafka_probe() -> ComponentReadiness: + broker = active_broker[0] + try: + available = broker is not None and await asyncio.to_thread( + broker.ping, 1.0 + ) + except Exception: # noqa: BLE001 - readiness must degrade, not crash + available = False + if not available: + return ComponentReadiness( + "kafka_read_committed", + ComponentState.NOT_READY, + detail="stable Kafka metadata unavailable", + checked_at_ns=time.time_ns(), + ) + return _ready( + "kafka_read_committed", + detail="read_committed/manual-checkpoint consumer reachable", + ) + + async def projection_cache_probe() -> ComponentReadiness: + bound = await asyncio.to_thread(target.cache_is_bound) + if not bound: + return ComponentReadiness( + "projection_cache_generation", + ComponentState.NOT_READY, + detail="Redis/SQLite projection cache identity is missing or mismatched", + checked_at_ns=time.time_ns(), + ) + return _ready( + "projection_cache_generation", + detail=f"cache_id_sha256={hashlib.sha256(spool.cache_id.encode()).hexdigest()}", + ) + + readiness = stable_readiness( + config, + manifests, + spool, + quota=quota, + extra_probes=( + CallableReadinessProbe("kafka_read_committed", kafka_probe), + CallableReadinessProbe("projection_cache_generation", projection_cache_probe), + CallableReadinessProbe("instrument_catalog", lambda: _ready( + "instrument_catalog", + detail=f"bindings={len(catalog.bindings)}", + revision=str(catalog.catalog_revision), + )), + ), + ) + app = FastAPI( + title="Quant Data Layer V2 Stable Projector Health", + version="2.0.0", + docs_url=None, + redoc_url=None, + openapi_url=None, + ) + install_stable_health(app, readiness, config.public_manifest()) + server = uvicorn.Server(uvicorn.Config( + app, + host="0.0.0.0", + port=config.http_port, + log_level="info", + access_log=False, + )) + health_task = asyncio.create_task(server.serve()) + try: + await supervise_stable_projector( + broker_factory=broker_factory, + should_stop=health_task.done, + on_broker=on_broker, + ) + await health_task + finally: + server.should_exit = True + if not health_task.done(): + await health_task + await sink.close() + await asyncio.to_thread(redis_client.close) + await asyncio.to_thread(quota.close) + await asyncio.to_thread(spool.close) diff --git a/qdl/runtime/stable_bar_edge.py b/qdl/runtime/stable_bar_edge.py new file mode 100644 index 0000000..75e22a9 --- /dev/null +++ b/qdl/runtime/stable_bar_edge.py @@ -0,0 +1,576 @@ +from __future__ import annotations + +import asyncio +import json +import logging +import os +import signal +import threading +import time +from pathlib import Path + +from qdl.adapters.binance import ( + BinanceBarRawBinding, + fetch_closed_bar_history_raw_envelopes as fetch_binance_history, + fetch_latest_closed_bar_raw_envelope, +) +from qdl.adapters.okx.bar_edge import ( + OkxBarRawBinding, + fetch_closed_bar_history_raw_envelopes as fetch_okx_history, + fetch_latest_closed_bar_raw_envelope as fetch_okx_latest, +) +from qdl.runtime.stable_catalog import StableSourceBinding, StableSourceCatalog +from qdl.runtime.stable_deployment import ( + StableAcquisitionBinding, + StableAcquisitionPlan, +) +from qdl.transport.kafka_raw import KafkaRawPublisher, KafkaRawPublisherConfig + + +logger = logging.getLogger(__name__) + + +def _bar_interval_ms(interval: str) -> int: + units = { + "s": 1_000, + "m": 60_000, + "h": 3_600_000, + "d": 86_400_000, + } + if not interval or interval[-1] not in units: + raise ValueError("stable BAR interval is unsupported") + count = int(interval[:-1]) + if count <= 0: + raise ValueError("stable BAR interval must be positive") + return count * units[interval[-1]] + + +class StableBinanceBarEdge: + """Bounded real-provider BAR bootstrap plus Binance closed-bar polling. + + The class name is retained for the existing runtime entrypoint. Historical + Binance and OKX rows always enter raw Kafka and the Rust canonical core; + this edge never writes query cache or Redis state directly. + """ + + def __init__( + self, + *, + catalog: StableSourceCatalog, + acquisition: StableAcquisitionPlan, + authority: dict, + publisher: KafkaRawPublisher, + warmup_rows: int = 500, + max_catchup_rows: int = 1000, + settlement_delay_seconds: float = 10.0, + state_path: str | Path | None = None, + clock=time.time, + ) -> None: + if not 1 <= warmup_rows <= 1000: + raise ValueError("stable BAR warmup rows must be between 1 and 1000") + if not 1 <= max_catchup_rows <= 1000: + raise ValueError("stable BAR catch-up rows must be between 1 and 1000") + if not 1.0 <= settlement_delay_seconds <= 10.0: + raise ValueError("stable BAR settlement delay must be between 1 and 10 seconds") + self.catalog = catalog + self.acquisition = acquisition + self.authority = authority + self.publisher = publisher + self.warmup_rows = warmup_rows + self.max_catchup_rows = max_catchup_rows + self.settlement_delay_seconds = settlement_delay_seconds + self.state_path = Path(state_path) if state_path is not None else None + self.clock = clock + authority_revision = int(authority.get("revision", 0)) + self.binance_session_id = ( + f"qdl-v2-stable-binance-rest-r{authority_revision}" + ) + self.okx_session_id = f"qdl-v2-stable-okx-rest-r{authority_revision}" + self._last_open_ms: dict[str, int] = {} + self._history_bootstrapped = False + self._stopped = threading.Event() + + source_by_id = {item.binding_id: item for item in catalog.bindings} + pairs = tuple( + (source_by_id[item.binding_id], item) + for item in acquisition.bindings + ) + self.bindings = tuple( + pair + for pair in pairs + if pair[1].mode == "PYTHON_REST" and pair[1].runtime == "BINANCE" + ) + self.okx_bindings = tuple( + pair + for pair in pairs + if ( + pair[1].mode == "PYTHON_REST" + and pair[1].runtime == "OKX" + and pair[0].feed.value == "BAR" + ) + ) + if len(self.bindings) != 2 or len(self.okx_bindings) != 2: + raise ValueError( + "stable crypto BAR edge requires Binance and OKX Spot/derivative bindings" + ) + if ( + authority.get("mode") != "RUST_SHADOW" + or authority.get("public_write_allowed") is not False + or authority.get("legacy_write_allowed") is not False + ): + raise ValueError("stable crypto BAR edge requires shadow authority") + self._restore_state() + + @property + def _binding_ids(self) -> tuple[str, ...]: + return tuple(sorted( + source.binding_id + for source, _acquisition in self.bindings + self.okx_bindings + )) + + def _state_payload(self) -> dict: + return { + "schema": "qdl.stable-bar-edge-state.v1", + "slice_id": str(self.authority.get("slice_id", "")), + "authority_revision": int(self.authority["revision"]), + "catalog_revision": int(self.catalog.catalog_revision), + "acquisition_revision": int(self.acquisition.revision), + "binding_ids": list(self._binding_ids), + "last_open_ms": { + key: self._last_open_ms[key] for key in sorted(self._last_open_ms) + }, + } + + def _restore_state(self) -> None: + if self.state_path is None or not self.state_path.exists(): + return + try: + payload = json.loads(self.state_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as error: + raise RuntimeError("stable BAR checkpoint is unreadable") from error + if not isinstance(payload, dict) or set(payload) != { + "schema", "slice_id", "authority_revision", "catalog_revision", + "acquisition_revision", "binding_ids", "last_open_ms", + }: + raise RuntimeError("stable BAR checkpoint fields are invalid") + expected = self._state_payload() + for field in ( + "schema", "slice_id", "authority_revision", "catalog_revision", + "acquisition_revision", "binding_ids", + ): + if payload[field] != expected[field]: + raise RuntimeError( + f"stable BAR checkpoint {field} differs from runtime authority" + ) + last_open_ms = payload["last_open_ms"] + if not isinstance(last_open_ms, dict) or not set(last_open_ms).issubset( + self._binding_ids + ): + raise RuntimeError("stable BAR checkpoint binding watermarks are invalid") + restored: dict[str, int] = {} + sources = { + source.binding_id: source + for source, _acquisition in self.bindings + self.okx_bindings + } + for binding_id, value in last_open_ms.items(): + if ( + isinstance(value, bool) + or not isinstance(value, int) + or value <= 0 + or value % _bar_interval_ms(sources[binding_id].interval or "") + ): + raise RuntimeError("stable BAR checkpoint watermark is invalid") + restored[binding_id] = value + self._last_open_ms = restored + self._history_bootstrapped = set(restored) == set(self._binding_ids) + logger.info( + "stable BAR checkpoint restored bindings=%s complete=%s", + len(restored), + self._history_bootstrapped, + ) + + def _persist_state(self) -> None: + if self.state_path is None: + return + parent = self.state_path.parent + parent.mkdir(parents=True, exist_ok=True) + temporary = parent / f".{self.state_path.name}.{os.getpid()}.tmp" + encoded = ( + json.dumps( + self._state_payload(), + sort_keys=True, + separators=(",", ":"), + ensure_ascii=True, + ) + + "\n" + ).encode("ascii") + descriptor = None + try: + descriptor = os.open( + temporary, + os.O_WRONLY | os.O_CREAT | os.O_TRUNC, + 0o600, + ) + pending = memoryview(encoded) + while pending: + written = os.write(descriptor, pending) + if written <= 0: + raise OSError("stable BAR checkpoint write made no progress") + pending = pending[written:] + os.fsync(descriptor) + os.close(descriptor) + descriptor = None + os.replace(temporary, self.state_path) + directory = os.open(parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(directory) + finally: + os.close(directory) + finally: + if descriptor is not None: + os.close(descriptor) + temporary.unlink(missing_ok=True) + + def _binance_binding( + self, + source: StableSourceBinding, + ) -> BinanceBarRawBinding: + identity = source.instrument.identity + return BinanceBarRawBinding( + market=identity.market, + product_type=identity.product_type.value, + native_symbol=source.instrument.native_symbol, + interval=source.interval or "", + subscription_id=source.source_id, + source_session_id=self.binance_session_id, + connection_generation=1, + lease_epoch=1, + authority_revision=int(self.authority["revision"]), + partition_plan_epoch=1, + adapter_version=source.adapter_version, + config_revision=self.acquisition.revision, + instrument_catalog_revision=self.catalog.catalog_revision, + ) + + def _okx_binding( + self, + source: StableSourceBinding, + ) -> OkxBarRawBinding: + identity = source.instrument.identity + return OkxBarRawBinding( + market=identity.market, + product_type=identity.product_type.value, + native_symbol=source.instrument.native_symbol, + interval=source.interval or "", + subscription_id=source.source_id, + source_session_id=self.okx_session_id, + connection_generation=1, + lease_epoch=1, + authority_revision=int(self.authority["revision"]), + partition_plan_epoch=1, + adapter_version=source.adapter_version, + config_revision=self.acquisition.revision, + instrument_catalog_revision=self.catalog.catalog_revision, + ) + + def _publish_history( + self, + source: StableSourceBinding, + acquisition: StableAcquisitionBinding, + envelopes, + ) -> int: + values = tuple(envelopes) + if len(values) != self.warmup_rows: + raise RuntimeError( + f"stable BAR bootstrap coverage mismatch binding={source.binding_id} " + f"expected={self.warmup_rows} actual={len(values)}" + ) + acknowledgements = self.publisher.publish_many(values) + if len(acknowledgements) != len(values): + raise RuntimeError("stable BAR bootstrap did not receive every Kafka ACK") + payloads = [json.loads(item.raw_frame_bytes) for item in values] + opens = [ + int( + payload["row"][0] + if acquisition.runtime == "BINANCE" + else payload["data"][0][0] + ) + for payload in payloads + ] + self._last_open_ms[source.binding_id] = max(opens) + self._persist_state() + logger.info( + "stable real-provider BAR bootstrap ACK binding=%s venue=%s rows=%s " + "first_open_ms=%s last_open_ms=%s", + source.binding_id, + acquisition.runtime, + len(values), + min(opens), + max(opens), + ) + return len(acknowledgements) + + def _settled_observed_ms(self) -> int: + return int( + self.clock() * 1000 - self.settlement_delay_seconds * 1000 + ) + + def bootstrap_history(self) -> int: + if self._history_bootstrapped: + return 0 + observed_ms = self._settled_observed_ms() + published = 0 + for source, acquisition in self.bindings: + if source.binding_id in self._last_open_ms: + continue + published += self._publish_history( + source, + acquisition, + fetch_binance_history( + self._binance_binding(source), + limit=self.warmup_rows, + now_ms=observed_ms, + attempts=4, + test_provenance=False, + ), + ) + for source, acquisition in self.okx_bindings: + if source.binding_id in self._last_open_ms: + continue + published += self._publish_history( + source, + acquisition, + asyncio.run(fetch_okx_history( + self._okx_binding(source), + limit=self.warmup_rows, + now_ms=observed_ms, + test_provenance=False, + )), + ) + self._history_bootstrapped = ( + set(self._last_open_ms) == set(self._binding_ids) + ) + if not self._history_bootstrapped: + raise RuntimeError("stable BAR bootstrap did not checkpoint every binding") + logger.info( + "stable multi-venue BAR bootstrap complete bindings=%s rows=%s", + len(self.bindings) + len(self.okx_bindings), + published, + ) + return published + + @staticmethod + def _open_time_ms( + acquisition: StableAcquisitionBinding, + envelope, + ) -> int: + payload = json.loads(envelope.raw_frame_bytes) + if acquisition.runtime == "BINANCE": + return int(payload["row"][0]) + if acquisition.runtime == "OKX": + return int(payload["data"][0][0]) + raise ValueError("stable crypto BAR runtime is unsupported") + + def _pending_for_binding( + self, + source: StableSourceBinding, + acquisition: StableAcquisitionBinding, + latest, + *, + observed_ms: int, + ) -> tuple[tuple[object, int], ...]: + latest_open_ms = self._open_time_ms(acquisition, latest) + previous_open_ms = self._last_open_ms.get(source.binding_id) + if previous_open_ms is None: + return ((latest, latest_open_ms),) + if latest_open_ms < previous_open_ms: + raise RuntimeError( + f"stable BAR provider latest precedes durable watermark " + f"binding={source.binding_id} previous={previous_open_ms} " + f"latest={latest_open_ms}" + ) + if latest_open_ms == previous_open_ms: + return () + + interval_ms = _bar_interval_ms(source.interval or "") + elapsed_ms = latest_open_ms - previous_open_ms + if elapsed_ms % interval_ms: + raise RuntimeError( + f"stable BAR provider boundary mismatch binding={source.binding_id} " + f"previous={previous_open_ms} latest={latest_open_ms}" + ) + pending_rows = elapsed_ms // interval_ms + if pending_rows > self.max_catchup_rows: + raise RuntimeError( + f"stable BAR catch-up exceeds bound binding={source.binding_id} " + f"required={pending_rows} max={self.max_catchup_rows}" + ) + if pending_rows == 1: + values = (latest,) + elif acquisition.runtime == "BINANCE": + values = fetch_binance_history( + self._binance_binding(source), + limit=pending_rows, + now_ms=observed_ms, + attempts=4, + test_provenance=False, + ) + else: + values = asyncio.run(fetch_okx_history( + self._okx_binding(source), + limit=pending_rows, + now_ms=observed_ms, + test_provenance=False, + )) + + opens = tuple(self._open_time_ms(acquisition, item) for item in values) + expected = tuple( + range(previous_open_ms + interval_ms, latest_open_ms + 1, interval_ms) + ) + if opens != expected: + raise RuntimeError( + f"stable BAR catch-up is not contiguous binding={source.binding_id} " + f"expected_rows={len(expected)} observed_rows={len(opens)}" + ) + return tuple(zip(values, opens, strict=True)) + + def run_cycle(self) -> int: + observed_ms = self._settled_observed_ms() + latest = [] + for source, acquisition in self.bindings: + envelope = fetch_latest_closed_bar_raw_envelope( + self._binance_binding(source), + now_ms=observed_ms, + attempts=4, + test_provenance=False, + ) + latest.append((source, acquisition, envelope)) + for source, acquisition in self.okx_bindings: + envelope = asyncio.run(fetch_okx_latest( + self._okx_binding(source), + now_ms=observed_ms, + test_provenance=False, + )) + latest.append((source, acquisition, envelope)) + + pending = [] + for source, acquisition, envelope in latest: + pending.extend( + (source, item, open_time_ms) + for item, open_time_ms in self._pending_for_binding( + source, acquisition, envelope, observed_ms=observed_ms + ) + ) + if not pending: + return 0 + + acknowledgements = self.publisher.publish_many( + item for _source, item, _open_time_ms in pending + ) + if len(acknowledgements) != len(pending): + raise RuntimeError("stable latest-closed BAR cycle missed a Kafka ACK") + + acknowledged_opens: dict[str, int] = {} + for source, _item, open_time_ms in pending: + acknowledged_opens[source.binding_id] = max( + open_time_ms, acknowledged_opens.get(source.binding_id, -1) + ) + self._last_open_ms.update(acknowledged_opens) + self._persist_state() + logger.info( + "stable multi-venue closed BAR ACK count=%s bindings=%s", + len(acknowledgements), + ",".join(sorted(acknowledged_opens)), + ) + return len(acknowledgements) + + def run_forever(self) -> None: + failures = 0 + while not self._stopped.is_set(): + now = self.clock() + ready_at = ( + (int(now) // 60) * 60 + self.settlement_delay_seconds + ) + if not self._history_bootstrapped and now < ready_at: + self._stopped.wait(max(0.01, ready_at - now)) + continue + try: + self.bootstrap_history() + self.run_cycle() + failures = 0 + except Exception: + failures += 1 + logger.exception( + "stable crypto BAR cycle failed consecutive_failures=%s", failures + ) + now = self.clock() + next_boundary = ( + (int(now) // 60 + 1) * 60 + self.settlement_delay_seconds + ) + delay = max(0.25, next_boundary - now) + if failures: + delay = min(delay, min(2 ** min(failures, 6), 30)) + self._stopped.wait(delay) + + def stop(self, *_args) -> None: + self._stopped.set() + self.publisher.close() + + +def build_from_environment() -> StableBinanceBarEdge: + catalog = StableSourceCatalog.load(os.environ["QDL_STABLE_SOURCE_BINDINGS"]) + acquisition = StableAcquisitionPlan.load( + os.environ["QDL_STABLE_ACQUISITION_BINDINGS"], catalog=catalog + ) + runtime_dir = Path(os.environ["QDL_STABLE_RUNTIME_DIR"]) + authority = json.loads((runtime_dir / "authority.json").read_text(encoding="utf-8")) + cert_root = Path(os.environ["QDL_KAFKA_CERT_ROOT"]) + publisher = KafkaRawPublisher(KafkaRawPublisherConfig( + bootstrap_servers=os.environ["QDL_KAFKA_BOOTSTRAP_SERVERS"], + client_id=os.environ["QDL_KAFKA_CLIENT_ID"], + topic=acquisition.raw_topic, + ca_path=cert_root / "ca.crt", + certificate_path=cert_root / "client.crt", + key_path=cert_root / "client.key", + )) + return StableBinanceBarEdge( + catalog=catalog, + acquisition=acquisition, + authority=authority, + publisher=publisher, + warmup_rows=int(os.environ.get("QDL_STABLE_BAR_WARMUP_ROWS", "500")), + max_catchup_rows=int( + os.environ.get("QDL_STABLE_BAR_MAX_CATCHUP_ROWS", "1000") + ), + settlement_delay_seconds=float( + os.environ.get("QDL_STABLE_BAR_SETTLEMENT_DELAY_SECONDS", "10") + ), + state_path=os.environ.get( + "QDL_STABLE_BAR_STATE_PATH", + str( + Path( + os.environ.get( + "QDL_STABLE_STATE_DIR", + "/var/lib/qdl-stable/runtime", + ) + ) + / "stable-crypto-bar-edge.json" + ), + ), + ) + + +def main() -> int: + logging.basicConfig( + level=os.environ.get("LOG_LEVEL", "INFO"), + format="%(asctime)s %(levelname)s %(name)s %(message)s", + ) + edge = build_from_environment() + signal.signal(signal.SIGTERM, edge.stop) + signal.signal(signal.SIGINT, edge.stop) + try: + edge.run_forever() + finally: + if not edge._stopped.is_set(): + edge.stop() + return 0 diff --git a/qdl/runtime/stable_catalog.py b/qdl/runtime/stable_catalog.py new file mode 100644 index 0000000..5d64148 --- /dev/null +++ b/qdl/runtime/stable_catalog.py @@ -0,0 +1,389 @@ +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + +from qdl.domain.decimal import CanonicalDecimal +from qdl.domain.instrument import ( + AssetClass, + InstrumentAlias, + InstrumentIdentity, + InstrumentRecord, + InstrumentRegistry, + ProductType, +) +from qdl.marketdata.v2 import market_data_pb2 +from qdl.common.v1 import common_pb2 +from qdl.query import ( + AccessPurpose, + ConsumerGrade, + DataProduct, + DataRequirement, + EntitlementGrant, + EntitlementPolicy, + FeedType, +) +from qdl.transport.contracts import partition_key + + +@dataclass(frozen=True, slots=True) +class StableSourceBinding: + binding_id: str + instrument: InstrumentRecord + provider: str + source_id: str + source_role: str + source_policy_id: str + authoritative: bool + adapter_version: str + normalizer_version: str + feed: FeedType + interval: str | None + stale_after_ms: int + require_final_bar: bool + continuous_calendar: bool + v1_compatibility: str + canonical_stream: str + + def __post_init__(self) -> None: + required = ( + self.binding_id, + self.provider, + self.source_id, + self.source_role, + self.source_policy_id, + self.adapter_version, + self.normalizer_version, + self.canonical_stream, + ) + if any(not value.strip() for value in required): + raise ValueError("stable source binding identity is incomplete") + if self.source_role not in {"PRIMARY", "SECONDARY", "REFERENCE", "BACKFILL"}: + raise ValueError("stable source role is invalid") + if self.feed is FeedType.BAR and not self.interval: + raise ValueError("stable BAR binding requires interval") + if self.feed is not FeedType.BAR and self.interval is not None: + raise ValueError("stable non-BAR binding cannot have interval") + if self.require_final_bar and self.feed is not FeedType.BAR: + raise ValueError("require_final_bar is valid only for BAR") + if self.stale_after_ms <= 0: + raise ValueError("stable source freshness bound must be positive") + policies = { + "NONE", + "BINANCE_TRADE_MARKET_AND_GENERIC", + "BINANCE_TRADE_MARKET_ONLY", + "BINANCE_BAR_GENERIC", + "VN_TRADE_GENERIC", + } + if self.v1_compatibility not in policies: + raise ValueError("stable V1 compatibility policy is invalid") + if self.v1_compatibility != "NONE" and ( + ("TRADE" in self.v1_compatibility and self.feed is not FeedType.TRADE) + or ("BAR" in self.v1_compatibility and self.feed is not FeedType.BAR) + ): + raise ValueError("stable V1 compatibility policy differs from feed") + + @property + def partition_key(self) -> str: + return partition_key( + instrument_uid=self.instrument.instrument_uid, + feed_type=self.feed.value, + source_id=self.source_id, + ) + + @property + def requirement_key(self) -> tuple[str, FeedType, str | None]: + return self.instrument.instrument_uid, self.feed, self.interval + + +class StableSourceCatalog: + def __init__( + self, + *, + canonical_stream: str, + bindings: tuple[StableSourceBinding, ...], + catalog_revision: int, + source_policy_revision: int, + authority_revision: int, + ) -> None: + if not canonical_stream.strip() or not bindings: + raise ValueError("stable catalog requires canonical stream and bindings") + if min(catalog_revision, source_policy_revision, authority_revision) < 1: + raise ValueError("stable catalog revisions must be positive") + requirement_keys = [item.requirement_key for item in bindings] + binding_ids = [item.binding_id for item in bindings] + envelope_keys = [ + (item.instrument.instrument_uid, item.feed.value.lower(), item.interval or "", item.source_id) + for item in bindings + ] + if ( + len(requirement_keys) != len(set(requirement_keys)) + or len(binding_ids) != len(set(binding_ids)) + or len(envelope_keys) != len(set(envelope_keys)) + ): + raise ValueError("stable source bindings must be unique") + if any(item.canonical_stream != canonical_stream for item in bindings): + raise ValueError("stable bindings must share the catalog canonical stream") + records: dict[str, InstrumentRecord] = {} + for binding in bindings: + current = records.get(binding.instrument.instrument_uid) + if current is not None and current != binding.instrument: + raise ValueError("stable bindings disagree on instrument metadata") + records[binding.instrument.instrument_uid] = binding.instrument + self.canonical_stream = canonical_stream + self.bindings = bindings + self.catalog_revision = catalog_revision + self.source_policy_revision = source_policy_revision + self.authority_revision = authority_revision + self._by_requirement = {item.requirement_key: item for item in bindings} + self._by_envelope = {key: item for key, item in zip(envelope_keys, bindings, strict=True)} + + @classmethod + def load(cls, path: str | Path) -> "StableSourceCatalog": + raw = yaml.safe_load(Path(path).read_text(encoding="utf-8")) + if not isinstance(raw, dict) or raw.get("schema") != "qdl.v2.stable-source-bindings.v1": + raise ValueError("unsupported stable source binding schema") + expected = { + "schema", "canonical_stream", "catalog_revision", + "source_policy_revision", "authority_revision", "instruments", "bindings", + } + if set(raw) != expected: + raise ValueError("stable source catalog fields are incomplete or unknown") + instruments_raw = raw.get("instruments") + if not isinstance(instruments_raw, list) or not 1 <= len(instruments_raw) <= 10_000: + raise ValueError("stable source catalog requires 1..10000 instruments") + instruments = tuple(cls._instrument(value) for value in instruments_raw) + by_uid = {item.instrument_uid: item for item in instruments} + if len(by_uid) != len(instruments): + raise ValueError("stable source instrument UIDs must be unique") + values = raw.get("bindings") + if not isinstance(values, list) or not 1 <= len(values) <= 100_000: + raise ValueError("stable source catalog requires 1..100000 bindings") + canonical_stream = str(raw["canonical_stream"]) + return cls( + canonical_stream=canonical_stream, + bindings=tuple( + cls._binding(value, canonical_stream, by_uid) for value in values + ), + catalog_revision=int(raw["catalog_revision"]), + source_policy_revision=int(raw["source_policy_revision"]), + authority_revision=int(raw["authority_revision"]), + ) + + @staticmethod + def _instrument(raw: Any) -> InstrumentRecord: + if not isinstance(raw, dict): + raise ValueError("stable instrument must be a mapping") + expected = { + "instrument_uid", "instrument_id", "metadata_revision", "venue", "market", + "product_type", "canonical_symbol", "native_symbol", "asset_class", + "base_asset", "quote_asset", "settlement_asset", "price_tick", + "quantity_step", "contract_multiplier", "session_calendar_id", "attributes", + } + if set(raw) != expected: + raise ValueError("stable instrument fields are incomplete or unknown") + identity = InstrumentIdentity.create( + venue=str(raw["venue"]), + market=str(raw["market"]), + product_type=ProductType(str(raw["product_type"]).upper()), + canonical_symbol=str(raw["canonical_symbol"]), + ) + if ( + identity.instrument_uid != str(raw["instrument_uid"]) + or identity.instrument_id != str(raw["instrument_id"]).upper() + ): + raise ValueError("stable instrument UID/ID is not deterministic") + attributes = raw["attributes"] + if not isinstance(attributes, dict): + raise ValueError("stable instrument attributes must be a mapping") + return InstrumentRecord( + identity=identity, + metadata_revision=int(raw["metadata_revision"]), + asset_class=AssetClass(str(raw["asset_class"]).upper()), + native_symbol=str(raw["native_symbol"]).upper(), + base_asset=str(raw["base_asset"]).upper(), + quote_asset=str(raw["quote_asset"]).upper(), + settlement_asset=str(raw["settlement_asset"]).upper(), + price_tick=CanonicalDecimal.from_text(str(raw["price_tick"])), + quantity_step=CanonicalDecimal.from_text(str(raw["quantity_step"])), + contract_multiplier=CanonicalDecimal.from_text(str(raw["contract_multiplier"])), + session_calendar_id=str(raw["session_calendar_id"]), + attributes={str(key): str(value) for key, value in attributes.items()}, + ) + + @staticmethod + def _binding( + raw: Any, + canonical_stream: str, + instruments: dict[str, InstrumentRecord], + ) -> StableSourceBinding: + if not isinstance(raw, dict) or set(raw) != { + "binding_id", "instrument_uid", "feed", "interval", "source", "quality", + "v1_compatibility", + }: + raise ValueError("stable source binding fields are incomplete or unknown") + source = raw["source"] + quality = raw["quality"] + if not all(isinstance(value, dict) for value in (source, quality)): + raise ValueError("stable binding sections must be mappings") + try: + instrument = instruments[str(raw["instrument_uid"])] + except KeyError as error: + raise ValueError("stable binding references an unknown instrument") from error + expected_source = { + "provider", "source_id", "source_role", "source_policy_id", + "authoritative", "adapter_version", "normalizer_version", + } + if set(source) != expected_source: + raise ValueError("stable source lineage fields are incomplete or unknown") + if set(quality) != {"stale_after_ms", "require_final_bar", "continuous_calendar"}: + raise ValueError("stable source quality fields are incomplete or unknown") + interval = raw["interval"] + return StableSourceBinding( + binding_id=str(raw["binding_id"]), + instrument=instrument, + provider=str(source["provider"]), + source_id=str(source["source_id"]), + source_role=str(source["source_role"]).upper(), + source_policy_id=str(source["source_policy_id"]), + authoritative=bool(source["authoritative"]), + adapter_version=str(source["adapter_version"]), + normalizer_version=str(source["normalizer_version"]), + feed=FeedType(str(raw["feed"]).upper()), + interval=str(interval) if interval is not None else None, + stale_after_ms=int(quality["stale_after_ms"]), + require_final_bar=bool(quality["require_final_bar"]), + continuous_calendar=bool(quality["continuous_calendar"]), + v1_compatibility=str(raw["v1_compatibility"]).upper(), + canonical_stream=canonical_stream, + ) + + def binding_for(self, requirement: DataRequirement) -> StableSourceBinding: + try: + binding = self._by_requirement[ + requirement.instrument_uid, requirement.feed, requirement.interval + ] + except KeyError as error: + raise KeyError("requirement has no stable source binding") from error + if binding.source_policy_id != requirement.source_policy_id: + raise KeyError("requirement source policy does not match stable binding") + return binding + + def binding_for_envelope( + self, envelope: market_data_pb2.EventEnvelope + ) -> StableSourceBinding: + payload_name = envelope.WhichOneof("payload") + interval = envelope.bar.interval if payload_name == "bar" else "" + key = (envelope.instrument_uid, payload_name or "", interval, envelope.source_id) + try: + binding = self._by_envelope[key] + except KeyError as error: + raise ValueError("canonical event is outside the stable catalog") from error + expected_role = getattr(common_pb2, f"SOURCE_ROLE_{binding.source_role}") + if ( + envelope.instrument_id != binding.instrument.instrument_id + or envelope.instrument_revision != binding.instrument.metadata_revision + or envelope.venue != binding.instrument.identity.venue + or envelope.market != binding.instrument.identity.market + or envelope.product_type != binding.instrument.identity.product_type.value + or envelope.native_symbol != binding.instrument.native_symbol + or envelope.provider != binding.provider + or envelope.source_role != expected_role + or envelope.authority_revision != self.authority_revision + or envelope.adapter_version != binding.adapter_version + or envelope.normalizer_version != binding.normalizer_version + ): + raise ValueError("canonical event identity/lineage differs from stable binding") + if ( + envelope.schema_major != 2 + or len(envelope.event_id) != 16 + or len(envelope.raw_capture_id) != 16 + or len(envelope.raw_payload_hash) != 32 + or not envelope.source_session_id + or envelope.connection_generation < 1 + or envelope.lease_epoch < 1 + or envelope.partition_sequence < 1 + or not ( + 0 < envelope.received_at_ns + <= envelope.normalized_at_ns + <= envelope.published_at_ns + ) + ): + raise ValueError("stable canonical event provenance is incomplete") + self._validate_payload(envelope, binding) + return binding + + @staticmethod + def _validate_payload( + envelope: market_data_pb2.EventEnvelope, + binding: StableSourceBinding, + ) -> None: + payload_name = envelope.WhichOneof("payload") + if payload_name != binding.feed.value.lower(): + raise ValueError("canonical payload feed differs from stable binding") + if payload_name == "trade": + if ( + envelope.trade.quantity_unit == common_pb2.QUANTITY_UNIT_UNSPECIFIED + or envelope.trade.identity_kind + == market_data_pb2.TRADE_IDENTITY_KIND_UNSPECIFIED + ): + raise ValueError("stable trade quantity/identity semantics are incomplete") + elif payload_name == "quote": + if envelope.quote.quantity_unit == common_pb2.QUANTITY_UNIT_UNSPECIFIED: + raise ValueError("stable quote quantity unit is incomplete") + elif payload_name == "bar": + if envelope.bar.volume_unit == common_pb2.QUANTITY_UNIT_UNSPECIFIED: + raise ValueError("stable BAR volume unit is incomplete") + if binding.require_final_bar and ( + not envelope.bar.is_final + or envelope.bar.lifecycle + not in { + market_data_pb2.BAR_LIFECYCLE_FINAL, + market_data_pb2.BAR_LIFECYCLE_REVISED, + } + ): + raise ValueError("stable binding requires a final/revised BAR") + else: + raise ValueError("stable baseline supports only TRADE/QUOTE/BAR") + + def instrument_registry(self) -> InstrumentRegistry: + registry = InstrumentRegistry() + registered: set[str] = set() + for binding in self.bindings: + record = binding.instrument + if record.instrument_uid in registered: + continue + registry.register(record, [InstrumentAlias( + provider=binding.provider, + market=record.identity.market, + native_symbol=record.native_symbol, + instrument_uid=record.instrument_uid, + instrument_revision=record.metadata_revision, + valid_from_ns=0, + )]) + registered.add(record.instrument_uid) + return registry + + def entitlements(self) -> EntitlementPolicy: + grants = [] + for binding in self.bindings: + purposes = { + AccessPurpose.INTERNAL_ALPHA, + AccessPurpose.INTERNAL_RESEARCH, + } + if binding.authoritative: + purposes.add(AccessPurpose.INTERNAL_EXECUTION) + grants.append(EntitlementGrant( + source_id=binding.source_id, + license_revision="internal-stable-v2", + purposes=frozenset(purposes), + products=frozenset({ + DataProduct.CANONICAL_SNAPSHOT, + DataProduct.CANONICAL_HISTORY, + }), + valid_from_ns=0, + )) + return EntitlementPolicy(tuple(grants)) diff --git a/qdl/runtime/stable_deployment.py b/qdl/runtime/stable_deployment.py new file mode 100644 index 0000000..f3ae1ed --- /dev/null +++ b/qdl/runtime/stable_deployment.py @@ -0,0 +1,366 @@ +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Mapping +from urllib.parse import urlsplit + +import yaml + +from qdl.query import FeedType +from qdl.runtime.stable_catalog import StableSourceBinding, StableSourceCatalog + + +_MODES = frozenset({"RUST_NATIVE", "PYTHON_REST", "PYTHON_VENDOR_SDK"}) +_SEQUENCE_POLICIES = frozenset({"NONE", "MONOTONIC", "CONTIGUOUS"}) +STABLE_TOPIC_PARTITIONS = 6 +STABLE_CORE_WORKER_COUNT = 3 +_PROVIDER_KINDS = { + ("BINANCE", "TRADE"): frozenset({"binance_usdm_trade", "binance_spot_trade"}), + ("BINANCE", "QUOTE"): frozenset({"binance_usdm_bbo", "binance_spot_bbo"}), + ("BINANCE", "BAR"): frozenset({"binance_usdm_rest_bar", "binance_spot_rest_bar"}), + ("OKX", "TRADE"): frozenset({"okx_trade"}), + ("OKX", "QUOTE"): frozenset({"okx_bbo"}), + ("OKX", "BAR"): frozenset({"okx_bar"}), + ("HNX", "TRADE"): frozenset({"dnse_trade"}), + ("HNX", "BAR"): frozenset({"dnse_bar"}), + ("HOSE", "TRADE"): frozenset({"dnse_trade"}), + ("HOSE", "BAR"): frozenset({"dnse_bar"}), +} + + +@dataclass(frozen=True, slots=True) +class StableAcquisitionBinding: + binding_id: str + mode: str + runtime: str + provider_kind: str + native_channel: str + sequence_policy: str + websocket_url: str | None + business_websocket_url: str | None + + def validate(self, source: StableSourceBinding) -> None: + if ( + not self.binding_id + or self.mode not in _MODES + or not self.runtime + or not self.provider_kind + or not self.native_channel + or self.sequence_policy not in _SEQUENCE_POLICIES + ): + raise ValueError("stable acquisition binding is incomplete or unsupported") + allowed = _PROVIDER_KINDS.get( + (source.instrument.identity.venue, source.feed.value), frozenset() + ) + if self.provider_kind not in allowed: + raise ValueError("stable acquisition provider kind differs from catalog feed") + if self.provider_kind == "okx_bbo" and self.sequence_policy != "NONE": + raise ValueError("OKX bbo-tbt is replace-only and cannot require sequence continuity") + if self.mode == "RUST_NATIVE": + if self.runtime not in {"BINANCE", "OKX"}: + raise ValueError("Rust native acquisition supports Binance/OKX only") + if self.runtime != source.instrument.identity.venue: + raise ValueError("Rust runtime differs from stable venue") + self._require_wss(self.websocket_url) + if self.runtime == "OKX": + self._require_wss(self.business_websocket_url) + elif self.mode == "PYTHON_REST": + if ( + source.feed is not FeedType.BAR + or self.runtime not in {"BINANCE", "OKX"} + or self.runtime != source.instrument.identity.venue + or self.websocket_url is not None + or self.business_websocket_url is not None + ): + raise ValueError( + "Python REST acquisition is reserved for venue-owned BAR " + "without WebSocket" + ) + elif ( + source.instrument.identity.venue not in {"HNX", "HOSE"} + or self.runtime != "DNSE" + or self.websocket_url is not None + ): + raise ValueError("Python vendor SDK acquisition is reserved for VN sources") + + @staticmethod + def _require_wss(value: str | None) -> None: + parsed = urlsplit(value or "") + if parsed.scheme != "wss" or not parsed.hostname: + raise ValueError("stable native WebSocket URL must use wss") + + +@dataclass(frozen=True, slots=True) +class StableAcquisitionPlan: + schema: str + revision: int + raw_topic: str + canonical_topic: str + quarantine_topic: str + bindings: tuple[StableAcquisitionBinding, ...] + + def __post_init__(self) -> None: + if self.schema != "qdl.v2.stable-acquisition-bindings.v1" or self.revision < 1: + raise ValueError("unsupported stable acquisition schema/revision") + topics = (self.raw_topic, self.canonical_topic, self.quarantine_topic) + if any(not value for value in topics) or len(set(topics)) != 3: + raise ValueError("stable acquisition topics must be non-empty and unique") + identities = [item.binding_id for item in self.bindings] + if not identities or len(identities) != len(set(identities)): + raise ValueError("stable acquisition binding IDs must be non-empty and unique") + + @classmethod + def load( + cls, path: str | Path, *, catalog: StableSourceCatalog + ) -> "StableAcquisitionPlan": + payload = yaml.safe_load(Path(path).read_text(encoding="utf-8")) + if not isinstance(payload, dict) or set(payload) != { + "schema", "revision", "topics", "bindings", + }: + raise ValueError("stable acquisition plan fields are incomplete or unknown") + topics = payload["topics"] + values = payload["bindings"] + if not isinstance(topics, dict) or set(topics) != { + "raw", "canonical", "quarantine", + }: + raise ValueError("stable acquisition topics are incomplete or unknown") + if not isinstance(values, list) or not 1 <= len(values) <= 100_000: + raise ValueError("stable acquisition requires 1..100000 bindings") + bindings = [] + for value in values: + if not isinstance(value, dict) or set(value) != { + "binding_id", "mode", "runtime", "provider_kind", "native_channel", + "sequence_policy", "websocket_url", "business_websocket_url", + }: + raise ValueError("stable acquisition binding fields are incomplete or unknown") + bindings.append(StableAcquisitionBinding( + binding_id=str(value["binding_id"]), + mode=str(value["mode"]).upper(), + runtime=str(value["runtime"]).upper(), + provider_kind=str(value["provider_kind"]), + native_channel=str(value["native_channel"]), + sequence_policy=str(value["sequence_policy"]).upper(), + websocket_url=( + str(value["websocket_url"]) if value["websocket_url"] is not None else None + ), + business_websocket_url=( + str(value["business_websocket_url"]) + if value["business_websocket_url"] is not None else None + ), + )) + result = cls( + schema=str(payload["schema"]), + revision=int(payload["revision"]), + raw_topic=str(topics["raw"]), + canonical_topic=str(topics["canonical"]), + quarantine_topic=str(topics["quarantine"]), + bindings=tuple(bindings), + ) + source_by_id = {item.binding_id: item for item in catalog.bindings} + if set(source_by_id) != {item.binding_id for item in result.bindings}: + raise ValueError("stable acquisition and source catalog binding sets differ") + for item in result.bindings: + item.validate(source_by_id[item.binding_id]) + if result.canonical_topic != catalog.canonical_stream: + raise ValueError("stable acquisition canonical topic differs from catalog") + return result + + def core_config( + self, + *, + catalog: StableSourceCatalog, + authority: Mapping[str, Any], + max_events: int = 0, + worker_index: int = 1, + ) -> dict[str, Any]: + self._validate_authority(authority) + if not 1 <= worker_index <= STABLE_CORE_WORKER_COUNT: + raise ValueError("stable core worker index is outside the topology bound") + source_by_id = {item.binding_id: item for item in catalog.bindings} + acquisitions = {item.binding_id: item for item in self.bindings} + bindings = [] + for binding_id in sorted(source_by_id): + source = source_by_id[binding_id] + acquisition = acquisitions[binding_id] + identity = source.instrument.identity + bindings.append({ + "provider": source.provider, + "venue": identity.venue, + "market": identity.market, + "product_type": identity.product_type.value, + "native_symbol": source.instrument.native_symbol, + "native_channel": acquisition.native_channel, + "provider_kind": acquisition.provider_kind, + "instrument_uid": source.instrument.instrument_uid, + "instrument_id": source.instrument.instrument_id, + "instrument_revision": source.instrument.metadata_revision, + "instrument_catalog_revision": catalog.catalog_revision, + "source_id": source.source_id, + "source_role": source.source_role, + "normalizer_version": source.normalizer_version, + "require_final_bar": source.require_final_bar, + "sequence_policy": acquisition.sequence_policy, + }) + return { + "core": { + "canonical_stream": self.canonical_topic, + "quarantine_stream": self.quarantine_topic, + "allow_test_provenance": False, + "dedup_capacity": 1_000_000, + "bindings": bindings, + }, + "raw_topics": [self.raw_topic], + "authority": dict(authority), + "shard_id": f"qdl-v2-stable-core-{worker_index:03d}", + "transactional_id": f"qdl-v2-stable-core-{worker_index:03d}", + "batch_size": 256, + "batch_wait_ms": 25, + "max_events": max_events, + "metrics_every_batches": 100, + } + + def native_ingestor_configs( + self, + *, + catalog: StableSourceCatalog, + authority: Mapping[str, Any], + max_events: int = 0, + max_runtime_seconds: int = 0, + ) -> dict[str, dict[str, Any]]: + self._validate_authority(authority) + source_by_id = {item.binding_id: item for item in catalog.bindings} + grouped: dict[tuple[str, str], list[StableAcquisitionBinding]] = {} + for acquisition in self.bindings: + if acquisition.mode == "RUST_NATIVE": + source = source_by_id[acquisition.binding_id] + grouped.setdefault( + (acquisition.runtime, source.instrument.identity.market), [] + ).append(acquisition) + result = {} + for (runtime, market), values in sorted(grouped.items()): + first = values[0] + bindings = [] + for acquisition in sorted(values, key=lambda item: item.binding_id): + source = source_by_id[acquisition.binding_id] + identity = source.instrument.identity + bindings.append({ + "provider": source.provider, + "venue": identity.venue, + "market": identity.market, + "product_type": identity.product_type.value, + "native_symbol": source.instrument.native_symbol, + "native_channel": acquisition.native_channel, + "subscription_id": source.source_id, + "adapter_version": source.adapter_version, + "instrument_catalog_revision": catalog.catalog_revision, + "feed": source.feed.value, + "delivery_class": ( + "LATEST_STATE" + if source.feed is FeedType.QUOTE + else "LOSSLESS" + ), + }) + key = f"{runtime.lower()}-{market.lower()}" + result[key] = { + "runtime": runtime, + "websocket_url": first.websocket_url, + "business_websocket_url": first.business_websocket_url, + "raw_stream": self.raw_topic, + "shard_id": f"qdl-v2-stable-{key}", + "lease_epoch": 1, + "partition_plan_epoch": 1, + "config_revision": self.revision, + "heartbeat_seconds": 15, + "max_events": max_events, + "max_runtime_seconds": max_runtime_seconds, + "metrics_every_events": 1000, + "generation_state_path": ( + f"/var/lib/qdl-stable/runtime/generations/{key}" + ), + "max_inflight_publishes": 512, + "latest_state_flush_ms": 50, + "authority": dict(authority), + "bindings": bindings, + } + return result + + @staticmethod + def _validate_authority(authority: Mapping[str, Any]) -> None: + digest = str(authority.get("candidate_image_digest", "")) + if ( + authority.get("mode") != "RUST_SHADOW" + or authority.get("public_write_allowed") is not False + or authority.get("legacy_write_allowed") is not False + or not digest.startswith("sha256:") + or len(digest) != 71 + ): + raise ValueError("stable authority is not an isolated Rust shadow record") + + +def stable_authority_record( + *, + rust_image_digest: str, + capability_manifest: Path, + contract: Path, + partition_plan: bytes, + effective_at_ns: int, +) -> dict[str, Any]: + digest = rust_image_digest.removeprefix("sha256:") + if len(digest) != 64 or any(value not in "0123456789abcdef" for value in digest): + raise ValueError("stable Rust image digest must be SHA-256") + if effective_at_ns <= 0: + raise ValueError("stable authority effective time must be positive") + return { + "schema": "qdl.authority-record.v1", + "slice_id": "qdl-v2-stable-multivenue-shadow", + "revision": 1, + "mode": "RUST_SHADOW", + "candidate_image_digest": f"sha256:{digest}", + "capability_manifest_digest": hashlib.sha256(capability_manifest.read_bytes()).hexdigest(), + "contract_digest": hashlib.sha256(contract.read_bytes()).hexdigest(), + "partition_plan_digest": hashlib.sha256(partition_plan).hexdigest(), + "public_write_allowed": False, + "legacy_write_allowed": False, + "approved_by": "phase-b-isolated-stable-candidate", + "effective_at_ns": effective_at_ns, + } + + +def write_stable_runtime_bundle( + destination: Path, + *, + catalog: StableSourceCatalog, + acquisition: StableAcquisitionPlan, + authority: Mapping[str, Any], +) -> dict[str, str]: + destination.mkdir(parents=True, exist_ok=True) + payloads = { + "authority.json": dict(authority), + **{ + "core.json" if worker_index == 1 else f"core-{worker_index:03d}.json": + acquisition.core_config( + catalog=catalog, + authority=authority, + worker_index=worker_index, + ) + for worker_index in range(1, STABLE_CORE_WORKER_COUNT + 1) + }, + **{ + f"ingestor-{name}.json": payload + for name, payload in acquisition.native_ingestor_configs( + catalog=catalog, authority=authority + ).items() + }, + } + digests = {} + for name, payload in sorted(payloads.items()): + encoded = ( + json.dumps(payload, indent=2, sort_keys=True, separators=(",", ": ")) + "\n" + ).encode() + path = destination / name + path.write_bytes(encoded) + digests[name] = hashlib.sha256(encoded).hexdigest() + return digests diff --git a/qdl/runtime/stable_ingest.py b/qdl/runtime/stable_ingest.py new file mode 100644 index 0000000..6a60742 --- /dev/null +++ b/qdl/runtime/stable_ingest.py @@ -0,0 +1,311 @@ +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import hmac +import ipaddress +import json +import uuid +from dataclasses import dataclass, field +from urllib.parse import urlsplit + +import httpx +from fastapi import FastAPI, Header, HTTPException, Request +from google.protobuf.message import DecodeError + +from qdl.marketdata.v2 import market_data_pb2 +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.envelope import validate_raw_envelope +from qdl.runtime.lease import GatewayFenced +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.stream import DurableStreamGateway +from qdl.transport import DurableEvent, SQLiteDurableSpool, StoredEvent + + +_INGEST_SCHEMA = "qdl.v2.stable-canonical-ingest.v1" +_RESULT_SCHEMA = "qdl.v2.stable-canonical-ingest-result.v1" + + +def _signature(secret: bytes, body: bytes) -> str: + return "sha256=" + hmac.new(secret, body, hashlib.sha256).hexdigest() + + +def _internal_url(value: str) -> bool: + parsed = urlsplit(value) + if parsed.scheme != "http" or not parsed.hostname: + return False + try: + return ipaddress.ip_address(parsed.hostname).is_loopback + except ValueError: + return parsed.hostname in { + "localhost", + "stream_v2", + "stream_v2_active", + "stream_v2_passive", + "qdl-stable-stream", + } or parsed.hostname.endswith(".internal") + + +def install_stable_canonical_ingest( + app: FastAPI, + *, + gateway: DurableStreamGateway, + catalog: StableSourceCatalog, + spool: SQLiteDurableSpool, + secret: bytes, +) -> None: + if len(secret) < 32: + raise ValueError("stable internal ingest secret must contain at least 256 bits") + + @app.post("/internal/v2/canonical/events", include_in_schema=False) + async def ingest( + request: Request, + signature: str | None = Header(None, alias="X-QDL-Stable-Signature"), + ): + body = await request.body() + if not signature or not hmac.compare_digest(signature, _signature(secret, body)): + raise HTTPException(status_code=401, detail="invalid stable ingest signature") + try: + payload = json.loads(body) + if set(payload) != {"schema", "batch_id", "events"}: + raise ValueError("stable ingest fields are incomplete or unknown") + if payload["schema"] != _INGEST_SCHEMA: + raise ValueError("stable ingest schema is invalid") + uuid.UUID(str(payload["batch_id"])) + values = payload["events"] + if not isinstance(values, list) or not 1 <= len(values) <= 1000: + raise ValueError("stable ingest batch must contain 1..1000 events") + except (ValueError, TypeError, json.JSONDecodeError) as error: + raise HTTPException(status_code=400, detail=str(error)) from error + try: + lease_epoch = gateway.assert_active() + except GatewayFenced as error: + raise HTTPException(status_code=409, detail="stable gateway is not active") from error + + references = [] + for value in values: + try: + required_fields = {"canonical", "raw_stream", "raw_event_id"} + allowed_fields = required_fields | {"raw_provider_envelope"} + if ( + not isinstance(value, dict) + or not required_fields.issubset(value) + or set(value) - allowed_fields + ): + raise ValueError("stable event reference fields are incomplete or unknown") + canonical = base64.b64decode(value["canonical"], validate=True) + raw_event_id = bytes.fromhex(str(value["raw_event_id"])) + raw_stream = str(value["raw_stream"]) + inline_raw = ( + base64.b64decode(value["raw_provider_envelope"], validate=True) + if "raw_provider_envelope" in value + else None + ) + envelope = market_data_pb2.EventEnvelope.FromString(canonical) + binding = catalog.binding_for_envelope(envelope) + if raw_event_id != bytes(envelope.raw_capture_id): + raise ValueError("stable canonical raw reference is unavailable") + if inline_raw is not None: + raw = raw_provider_pb2.RawProviderEnvelope.FromString(inline_raw) + validate_raw_envelope(raw) + if ( + bytes(raw.capture_id) != raw_event_id + or bytes(raw.raw_frame_sha256) != bytes(envelope.raw_payload_hash) + or raw.provider != envelope.provider + or raw.venue != envelope.venue + or raw.market != envelope.market + or raw.native_symbol != envelope.native_symbol + or raw.source_session_id != envelope.source_session_id + or raw.connection_generation != envelope.connection_generation + or raw.authority_revision != envelope.authority_revision + ): + raise ValueError("private Kafka raw lineage validation failed") + references.append(( + binding, envelope, canonical, raw_stream, raw_event_id, inline_raw + )) + except (ValueError, TypeError, DecodeError) as error: + raise HTTPException(status_code=422, detail=str(error)) from error + + raw_by_reference = {} + for raw_stream in sorted({ + item[3] for item in references if item[5] is None + }): + event_ids = [ + item[4] + for item in references + if item[3] == raw_stream and item[5] is None + ] + found = await asyncio.to_thread( + spool.find_events, stream=raw_stream, event_ids=event_ids + ) + raw_by_reference.update( + {(raw_stream, event_id): stored for event_id, stored in found.items()} + ) + prepared = [] + for ( + binding, envelope, canonical, raw_stream, raw_event_id, inline_raw + ) in references: + if ( + inline_raw is None + and (raw_stream, raw_event_id) not in raw_by_reference + ): + raise HTTPException( + status_code=422, + detail="stable canonical raw reference is unavailable", + ) + prepared.append((binding, envelope, DurableEvent( + stream=catalog.canonical_stream, + partition_key=binding.partition_key, + event_id=bytes(envelope.event_id), + payload=canonical, + accepted_at_ns=max(envelope.received_at_ns, 1), + headers={ + "raw_stream": raw_stream, + "raw_event_id": raw_event_id.hex(), + }, + ))) + + try: + stored_values = await gateway.publish_many( + [event for _binding, _envelope, event in prepared] + ) + except GatewayFenced as error: + raise HTTPException(status_code=409, detail="stable gateway was fenced") from error + duplicate_ids = [ + event.event_id + for (_binding, _envelope, event), stored in zip( + prepared, stored_values, strict=True + ) + if stored is None + ] + duplicates = await asyncio.to_thread( + spool.find_events, + stream=catalog.canonical_stream, + event_ids=duplicate_ids, + ) + results = [] + for (binding, envelope, event), stored in zip( + prepared, stored_values, strict=True + ): + duplicate = stored is None + stored = stored or duplicates.get(event.event_id) + if stored is None: + raise HTTPException(status_code=503, detail="stable cache ACK is unavailable") + results.append({ + "event_id": envelope.event_id.hex(), + "partition_key": binding.partition_key, + "offset": stored.cursor.offset, + "duplicate": duplicate, + }) + return { + "schema": _RESULT_SCHEMA, + "lease_epoch": lease_epoch, + "results": results, + } + + +@dataclass(slots=True) +class StableHttpCanonicalSink: + urls: tuple[str, ...] + secret: bytes + spool: SQLiteDurableSpool + timeout_seconds: float = 10.0 + client: httpx.AsyncClient | None = None + _owns_client: bool = field(init=False) + + def __post_init__(self) -> None: + if ( + not self.urls + or any(not _internal_url(value) for value in self.urls) + or len(self.secret) < 32 + or self.timeout_seconds <= 0 + ): + raise ValueError("stable HTTP sink configuration is invalid") + self._owns_client = self.client is None + if self.client is None: + self.client = httpx.AsyncClient( + follow_redirects=False, + limits=httpx.Limits(max_connections=10, max_keepalive_connections=5), + timeout=self.timeout_seconds, + ) + + async def publish(self, event: DurableEvent) -> StoredEvent: + return (await self.publish_many((event,)))[0] + + async def publish_many( + self, events: tuple[DurableEvent, ...] | list[DurableEvent] + ) -> tuple[StoredEvent, ...]: + values = tuple(events) + if not 1 <= len(values) <= 1000: + raise ValueError("stable HTTP sink batch must contain 1..1000 events") + encoded = [] + for event in values: + raw_stream = event.headers.get("raw_stream") + raw_event_id = event.headers.get("raw_event_id") + if not raw_stream or not raw_event_id: + raise ValueError("stable HTTP sink requires a durable raw reference") + item = { + "canonical": base64.b64encode(event.payload).decode(), + "raw_stream": raw_stream, + "raw_event_id": raw_event_id, + } + inline_raw = event.headers.get("raw_provider_envelope") + if inline_raw: + item["raw_provider_envelope"] = inline_raw + encoded.append(item) + body = json.dumps({ + "schema": _INGEST_SCHEMA, + "batch_id": str(uuid.uuid4()), + "events": encoded, + }, sort_keys=True, separators=(",", ":")).encode() + last_error: BaseException | None = None + assert self.client is not None + for url in self.urls: + try: + response = await self.client.post( + f"{url.rstrip('/')}/internal/v2/canonical/events", + content=body, + headers={ + "Content-Type": "application/json", + "X-QDL-Stable-Signature": _signature(self.secret, body), + }, + ) + if response.status_code in {409, 503}: + continue + response.raise_for_status() + result = response.json() + acknowledgements = result.get("results", ()) + if ( + result.get("schema") != _RESULT_SCHEMA + or len(acknowledgements) != len(values) + or [item.get("event_id") for item in acknowledgements] + != [event.event_id.hex() for event in values] + ): + raise ValueError("stable ingest ACK contract is invalid") + stored_by_id = await asyncio.to_thread( + self.spool.find_events, + stream=values[0].stream, + event_ids=[event.event_id for event in values], + ) + stored_values = [] + for event, acknowledgement in zip( + values, acknowledgements, strict=True + ): + stored = stored_by_id.get(event.event_id) + if ( + stored is None + or stored.cursor.offset != int(acknowledgement["offset"]) + ): + raise ValueError("stable ingest ACK differs from shared cache") + stored_values.append(stored) + return tuple(stored_values) + except (httpx.HTTPError, ValueError, TypeError) as error: + last_error = error + raise RuntimeError( + "no active stable stream gateway accepted canonical data" + ) from last_error + + async def close(self) -> None: + if self._owns_client and self.client is not None: + await self.client.aclose() diff --git a/qdl/runtime/stable_projector.py b/qdl/runtime/stable_projector.py new file mode 100644 index 0000000..6bf35f8 --- /dev/null +++ b/qdl/runtime/stable_projector.py @@ -0,0 +1,708 @@ +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import hmac +import logging +from collections import defaultdict, deque +from dataclasses import dataclass +from decimal import Decimal, InvalidOperation +from typing import Awaitable, Callable, Protocol + +from qdl.marketdata.v2 import market_data_pb2 +from qdl.projection.stable import StableCompatibilityProjector, StableProjectionTarget +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.envelope import validate_raw_envelope +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.stream import DurableStreamGateway +from qdl.transport import ( + DurableEvent, + EventIdCollision, + SQLiteDurableSpool, + StoredEvent, +) +from qdl.transport.kafka_projector import KafkaProjectorRecord, ProjectorBroker + + +logger = logging.getLogger(__name__) + + +class StableCanonicalSink(Protocol): + async def publish(self, event: DurableEvent) -> StoredEvent: ... + async def publish_many( + self, events: tuple[DurableEvent, ...] | list[DurableEvent] + ) -> tuple[StoredEvent, ...]: ... + + +class LocalStableCanonicalSink: + """In-process implementation used by isolated tests and single-node rehearsal.""" + + def __init__(self, gateway: DurableStreamGateway, spool: SQLiteDurableSpool) -> None: + self.gateway = gateway + self.spool = spool + + async def publish(self, event: DurableEvent) -> StoredEvent: + return (await self.publish_many((event,)))[0] + + async def publish_many( + self, events: tuple[DurableEvent, ...] | list[DurableEvent] + ) -> tuple[StoredEvent, ...]: + values = tuple(events) + stored_values = await self.gateway.publish_many(values) + duplicate_ids = [ + event.event_id + for event, stored in zip(values, stored_values, strict=True) + if stored is None + ] + duplicates = await asyncio.to_thread( + self.spool.find_events, + stream=values[0].stream, + event_ids=duplicate_ids, + ) + resolved = [] + for event, stored in zip(values, stored_values, strict=True): + stored = stored or duplicates.get(event.event_id) + if stored is None: + raise RuntimeError("duplicate canonical ACK has no shared cache record") + resolved.append(stored) + return tuple(resolved) + + +@dataclass(frozen=True, slots=True) +class StableProjectorStats: + raw_committed: int + canonical_committed: int + duplicate_projections: int + pending_canonical: int + pending_bytes: int + + +@dataclass(frozen=True, slots=True) +class _ReadyCanonical: + partition: tuple[str, int] + record: KafkaProjectorRecord + envelope: market_data_pb2.EventEnvelope + raw_envelope: bytes + raw_stream: str + raw_event_id: bytes + event: DurableEvent + existing: StoredEvent | None = None + semantic_duplicate: bool = False + project_latest: bool = True + + +class StableProjectorEngine: + """Kafka-authoritative raw/canonical join with downstream-before-checkpoint ordering.""" + + def __init__( + self, + *, + broker: ProjectorBroker, + spool: SQLiteDurableSpool, + catalog: StableSourceCatalog, + canonical_topic: str, + raw_topics: tuple[str, ...], + sink: StableCanonicalSink, + projector: StableCompatibilityProjector, + target: StableProjectionTarget, + max_pending_records: int = 10_000, + max_pending_bytes: int = 256 * 1024 * 1024, + max_batch_records: int = 128, + batch_wait_seconds: float = 0.025, + ) -> None: + if ( + not canonical_topic + or canonical_topic in raw_topics + or len(raw_topics) != len(set(raw_topics)) + ): + raise ValueError("stable projector topics are invalid") + if max_pending_records <= 0 or max_pending_bytes <= 0: + raise ValueError("stable projector pending bounds must be positive") + if not 1 <= max_batch_records <= 1000 or not 0 < batch_wait_seconds <= 1: + raise ValueError("stable projector batch policy is invalid") + self.broker = broker + self.spool = spool + self.catalog = catalog + self.canonical_topic = canonical_topic + self.raw_topics = raw_topics + self.sink = sink + self.projector = projector + self.target = target + self.max_pending_records = max_pending_records + self.max_pending_bytes = max_pending_bytes + self.max_batch_records = max_batch_records + self.batch_wait_seconds = batch_wait_seconds + poll_headroom = min(max_batch_records, max(1, max_pending_records // 4)) + self._canonical_pause_high_records = max( + 1, max_pending_records - poll_headroom + ) + self._canonical_resume_low_records = self._canonical_pause_high_records // 2 + self._canonical_pause_high_bytes = max(1, max_pending_bytes * 3 // 4) + self._canonical_resume_low_bytes = self._canonical_pause_high_bytes // 2 + self._canonical_paused = False + self._queues: dict[tuple[str, int], deque[KafkaProjectorRecord]] = defaultdict(deque) + self._waiting: dict[bytes, set[tuple[str, int]]] = defaultdict(set) + self._assignment_epoch: int | None = None + self._pending_records = 0 + self._pending_bytes = 0 + self._raw_committed = 0 + self._canonical_committed = 0 + self._duplicate_projections = 0 + + async def accept(self, record: KafkaProjectorRecord) -> None: + await self.accept_many((record,)) + + async def accept_many( + self, records: tuple[KafkaProjectorRecord, ...] | list[KafkaProjectorRecord] + ) -> None: + values = tuple(records) + if not values: + return + start = 0 + while start < len(values): + epoch = values[start].assignment_epoch + end = start + 1 + while end < len(values) and values[end].assignment_epoch == epoch: + end += 1 + await self._accept_assignment_batch(values[start:end], epoch) + start = end + + async def _accept_assignment_batch( + self, records: tuple[KafkaProjectorRecord, ...], epoch: int + ) -> None: + self._handle_assignment(epoch) + raw_values: list[tuple[KafkaProjectorRecord, bytes, DurableEvent]] = [] + canonical_values: list[KafkaProjectorRecord] = [] + for record in records: + if record.topic in self.raw_topics: + capture_id, event = self._raw_event(record) + raw_values.append((record, capture_id, event)) + elif record.topic == self.canonical_topic: + canonical_values.append(record) + else: + raise ValueError("stable projector received an unconfigured topic") + + if raw_values: + await asyncio.to_thread( + self.spool.append_many, [item[2] for item in raw_values] + ) + await asyncio.to_thread( + self._checkpoint_records, [item[0] for item in raw_values] + ) + self._raw_committed += len(raw_values) + await self._drain_ready() + + for record in canonical_values: + envelope = market_data_pb2.EventEnvelope.FromString(record.payload) + binding = self.catalog.binding_for_envelope(envelope) + if ( + bytes(envelope.event_id) != record.event_id + or binding.partition_key != record.key + ): + raise ValueError("Kafka canonical metadata differs from stable envelope") + partition = (record.topic, record.partition) + queue = self._queues[partition] + if queue and record.offset <= queue[-1].offset: + raise ValueError("Kafka canonical partition order regressed") + self._admit_pending(record) + queue.append(record) + await self._drain_ready() + + async def run_once(self, timeout_seconds: float = 1.0) -> bool: + record = await asyncio.to_thread(self.broker.poll, timeout_seconds) + if record is None: + # Another projector replica may have persisted the correlated raw + # envelope into the shared cache. Retry bounded local partitions so + # cross-replica raw/canonical ordering cannot stall indefinitely. + await self._drain_ready() + return False + records = [record] + deadline = asyncio.get_running_loop().time() + self.batch_wait_seconds + while len(records) < self.max_batch_records: + remaining = deadline - asyncio.get_running_loop().time() + if remaining <= 0: + break + item = await asyncio.to_thread(self.broker.poll, remaining) + if item is None: + break + records.append(item) + await self.accept_many(records) + return True + + def _raw_event(self, record: KafkaProjectorRecord) -> tuple[bytes, DurableEvent]: + raw = raw_provider_pb2.RawProviderEnvelope.FromString(record.payload) + validate_raw_envelope(raw) + capture_id = bytes(raw.capture_id) + if capture_id != record.event_id: + raise ValueError("Kafka raw event ID differs from capture ID") + return capture_id, DurableEvent( + stream=record.topic, + partition_key=record.key, + event_id=capture_id, + payload=record.payload, + accepted_at_ns=record.accepted_at_ns, + headers={ + "kafka_partition": str(record.partition), + "kafka_offset": str(record.offset), + "schema": f"{raw.raw_schema_name}/{raw.raw_schema_major}", + }, + ) + + async def _drain_ready(self) -> None: + while True: + ready = await self._ready_batch() + if not ready: + return + fresh = tuple(item for item in ready if not item.semantic_duplicate) + fresh_stored = ( + await self.sink.publish_many([item.event for item in fresh]) + if fresh + else () + ) + stored_iterator = iter(fresh_stored) + resolved: list[tuple[_ReadyCanonical, StoredEvent]] = [] + for item in ready: + stored = item.existing if item.semantic_duplicate else next(stored_iterator) + if stored is None: + raise RuntimeError("stable semantic duplicate has no cache record") + resolved.append((item, stored)) + + projected = tuple( + (item, stored) + for item, stored in resolved + if item.project_latest and not item.semantic_duplicate + ) + projections = [ + self.projector.build(stored, item.raw_envelope) + for item, stored in projected + ] + applied = ( + await asyncio.to_thread(self.target.apply_many, projections) + if projections + else () + ) + if len(applied) != len(projected): + raise RuntimeError( + "stable projection target returned an invalid result count" + ) + applied_by_event = { + item.record.event_id: was_applied + for (item, _stored), was_applied in zip( + projected, applied, strict=True + ) + } + await asyncio.to_thread( + self._checkpoint_records, [item.record for item in ready] + ) + for item in ready: + if item.semantic_duplicate or ( + item.record.event_id in applied_by_event + and not applied_by_event[item.record.event_id] + ): + self._duplicate_projections += 1 + self._canonical_committed += 1 + queue = self._queues[item.partition] + current = queue.popleft() + if current.offset != item.record.offset: + raise RuntimeError( + "stable canonical queue order changed during batch" + ) + self._pending_records -= 1 + self._pending_bytes -= len(item.record.payload) + capture_id = bytes(item.envelope.raw_capture_id) + waiting = self._waiting.get(capture_id) + if waiting is not None: + waiting.discard(item.partition) + if not waiting: + self._waiting.pop(capture_id, None) + if not queue: + self._queues.pop(item.partition, None) + self._update_canonical_backpressure() + + @staticmethod + def _verified_payload_hash( + envelope: market_data_pb2.EventEnvelope, + ) -> bytes: + payload_name = envelope.WhichOneof("payload") + if not payload_name: + raise EventIdCollision("canonical duplicate has no market payload") + declared = bytes(envelope.canonical_payload_hash) + observed = hashlib.sha256( + getattr(envelope, payload_name).SerializeToString(deterministic=True) + ).digest() + if len(declared) != 32 or not hmac.compare_digest(declared, observed): + raise EventIdCollision( + "canonical duplicate payload hash is missing or invalid" + ) + return declared + + @staticmethod + def _decimal_semantic(value) -> Decimal: + coefficient_name = value.WhichOneof("coefficient") + if coefficient_name == "mantissa": + coefficient = int(value.mantissa) + elif coefficient_name == "mantissa_text": + try: + coefficient = int(value.mantissa_text) + except ValueError as error: + raise EventIdCollision( + "canonical decimal coefficient is invalid" + ) from error + else: + raise EventIdCollision("canonical decimal coefficient is missing") + try: + observed = Decimal(coefficient).scaleb(-int(value.scale)) + declared = Decimal(value.source_text) + except (InvalidOperation, ValueError) as error: + raise EventIdCollision("canonical decimal text is invalid") from error + if not declared.is_finite() or declared != observed: + raise EventIdCollision( + "canonical decimal audit text differs from its exact value" + ) + return observed + + @classmethod + def _bar_semantics(cls, bar: market_data_pb2.Bar) -> tuple: + def optional_decimal(field: str): + return ( + cls._decimal_semantic(getattr(bar, field)) + if bar.HasField(field) + else None + ) + + return ( + bar.interval, + int(bar.open_time_ns), + int(bar.close_time_ns), + cls._decimal_semantic(bar.open), + cls._decimal_semantic(bar.high), + cls._decimal_semantic(bar.low), + cls._decimal_semantic(bar.close), + cls._decimal_semantic(bar.volume), + int(bar.trade_count), + bool(bar.is_final), + int(bar.revision), + int(bar.lifecycle), + ( + bytes(bar.supersedes_event_id) + if bar.HasField("supersedes_event_id") + else None + ), + int(bar.volume_unit), + optional_decimal("base_volume"), + optional_decimal("quote_volume"), + optional_decimal("contract_volume"), + ) + + @classmethod + def _same_market_semantics( + cls, + existing: market_data_pb2.EventEnvelope, + candidate: market_data_pb2.EventEnvelope, + ) -> bool: + existing_name = existing.WhichOneof("payload") + candidate_name = candidate.WhichOneof("payload") + if existing_name != candidate_name: + return False + existing_hash = cls._verified_payload_hash(existing) + candidate_hash = cls._verified_payload_hash(candidate) + if existing_name == "bar": + # Decimal spelling and acquisition origin are audit provenance, not + # a different OHLCV observation. Every actual BAR value remains + # strict and is compared above exact Decimal arithmetic. + return cls._bar_semantics(existing.bar) == cls._bar_semantics(candidate.bar) + return hmac.compare_digest(existing_hash, candidate_hash) + + @classmethod + def _semantic_duplicate( + cls, + existing: StoredEvent, + record: KafkaProjectorRecord, + envelope: market_data_pb2.EventEnvelope, + ) -> bool: + if existing.event.payload == record.payload: + return False + existing_envelope = market_data_pb2.EventEnvelope.FromString( + existing.event.payload + ) + if ( + existing.cursor.partition_key != record.key + or bytes(existing_envelope.event_id) != record.event_id + or not cls._same_market_semantics(existing_envelope, envelope) + ): + raise EventIdCollision( + "canonical event ID maps to different market semantics" + ) + return True + + def _latest_bar_close_ns(self, partition_key: str) -> int | None: + rows = self.spool.read_tail( + stream=self.catalog.canonical_stream, + partition_key=partition_key, + limit=10_000, + ) + closes = [] + for stored in rows: + envelope = market_data_pb2.EventEnvelope.FromString( + stored.event.payload + ) + if envelope.WhichOneof("payload") == "bar": + closes.append(int(envelope.bar.close_time_ns)) + return max(closes) if closes else None + + async def _ready_batch(self) -> tuple[_ReadyCanonical, ...]: + candidates = [] + for partition in sorted(self._queues): + for record in self._queues[partition]: + if len(candidates) >= self.max_batch_records: + break + envelope = market_data_pb2.EventEnvelope.FromString(record.payload) + candidates.append(( + partition, record, envelope, bytes(envelope.raw_capture_id) + )) + if len(candidates) >= self.max_batch_records: + break + + existing_by_id = await asyncio.to_thread( + self.spool.find_events, + stream=self.catalog.canonical_stream, + event_ids=tuple(record.event_id for _p, record, _e, _c in candidates), + ) + semantic_duplicates = { + record.event_id: existing + for _partition, record, envelope, _capture_id in candidates + if (existing := existing_by_id.get(record.event_id)) is not None + and self._semantic_duplicate(existing, record, envelope) + } + fallback_ids = tuple( + capture_id + for _partition, record, _envelope, capture_id in candidates + if record.event_id not in semantic_duplicates + and record.raw_provider_envelope is None + ) + raw_by_id = await asyncio.to_thread(self._find_raw_many, fallback_ids) + bar_high_watermarks: dict[str, int | None] = {} + ready = [] + blocked_partitions = set() + for partition, record, envelope, capture_id in candidates: + if partition in blocked_partitions: + continue + existing = semantic_duplicates.get(record.event_id) + if existing is not None: + ready.append(_ReadyCanonical( + partition=partition, + record=record, + envelope=envelope, + raw_envelope=b"", + raw_stream="semantic-duplicate", + raw_event_id=capture_id, + event=DurableEvent( + stream=self.catalog.canonical_stream, + partition_key=record.key, + event_id=record.event_id, + payload=record.payload, + accepted_at_ns=record.accepted_at_ns, + ), + existing=existing, + semantic_duplicate=True, + project_latest=False, + )) + continue + + if record.raw_provider_envelope is not None: + raw_envelope = record.raw_provider_envelope + raw = raw_provider_pb2.RawProviderEnvelope.FromString(raw_envelope) + validate_raw_envelope(raw) + if bytes(raw.capture_id) != capture_id: + raise ValueError( + "private Kafka raw lineage differs from canonical capture ID" + ) + raw_stream = "kafka-header:qdl-raw-provider-envelope" + raw_event_id = capture_id + else: + stored_raw = raw_by_id.get(capture_id) + if stored_raw is None: + if not self.raw_topics: + raise ValueError( + "canonical record is missing private Kafka raw lineage" + ) + self._waiting[capture_id].add(partition) + blocked_partitions.add(partition) + continue + raw_envelope = stored_raw.event.payload + raw_stream = stored_raw.event.stream + raw_event_id = stored_raw.event.event_id + + project_latest = True + if envelope.WhichOneof("payload") == "bar": + if record.key not in bar_high_watermarks: + bar_high_watermarks[record.key] = await asyncio.to_thread( + self._latest_bar_close_ns, record.key + ) + current = bar_high_watermarks[record.key] + close_ns = int(envelope.bar.close_time_ns) + project_latest = current is None or close_ns >= current + bar_high_watermarks[record.key] = ( + close_ns if current is None else max(current, close_ns) + ) + ready.append(_ReadyCanonical( + partition=partition, + record=record, + envelope=envelope, + raw_envelope=raw_envelope, + raw_stream=raw_stream, + raw_event_id=raw_event_id, + event=DurableEvent( + stream=self.catalog.canonical_stream, + partition_key=record.key, + event_id=record.event_id, + payload=record.payload, + accepted_at_ns=record.accepted_at_ns, + headers={ + "raw_stream": raw_stream, + "raw_event_id": raw_event_id.hex(), + "raw_provider_envelope": base64.b64encode( + raw_envelope + ).decode("ascii"), + "kafka_topic": record.topic, + "kafka_partition": str(record.partition), + "kafka_offset": str(record.offset), + }, + ), + project_latest=project_latest, + )) + return tuple(ready) + + def _find_raw_many( + self, capture_ids: tuple[bytes, ...] + ) -> dict[bytes, StoredEvent]: + missing = set(capture_ids) + resolved = {} + for stream in self.raw_topics: + if not missing: + break + found = self.spool.find_events( + stream=stream, event_ids=tuple(missing) + ) + resolved.update(found) + missing.difference_update(found) + return resolved + + def _checkpoint_records( + self, records: tuple[KafkaProjectorRecord, ...] | list[KafkaProjectorRecord] + ) -> None: + values = tuple(records) + checkpoint_many = getattr(self.broker, "checkpoint_many", None) + if callable(checkpoint_many): + checkpoint_many(values) + return + for record in values: + self.broker.checkpoint(record) + + def _admit_pending(self, record: KafkaProjectorRecord) -> None: + if ( + self._pending_records + 1 > self.max_pending_records + or self._pending_bytes + len(record.payload) > self.max_pending_bytes + ): + raise RuntimeError("stable projector canonical-before-raw buffer exhausted") + self._pending_records += 1 + self._pending_bytes += len(record.payload) + self._update_canonical_backpressure() + + def _update_canonical_backpressure(self) -> None: + if ( + not self._canonical_paused + and ( + self._pending_records >= self._canonical_pause_high_records + or self._pending_bytes >= self._canonical_pause_high_bytes + ) + ): + self.broker.pause_canonical() + self._canonical_paused = True + elif ( + self._canonical_paused + and self._pending_records <= self._canonical_resume_low_records + and self._pending_bytes <= self._canonical_resume_low_bytes + ): + self.broker.resume_canonical() + self._canonical_paused = False + + def _handle_assignment(self, epoch: int) -> None: + if self._assignment_epoch is None: + self._assignment_epoch = epoch + return + if epoch == self._assignment_epoch: + return + # Uncheckpointed broker records are discarded locally and replayed by + # the new assignment. Durable cache/projection duplicates are idempotent. + if self._canonical_paused: + self.broker.resume_canonical() + self._canonical_paused = False + self._queues.clear() + self._waiting.clear() + self._pending_records = 0 + self._pending_bytes = 0 + self._assignment_epoch = epoch + + @property + def stats(self) -> StableProjectorStats: + return StableProjectorStats( + raw_committed=self._raw_committed, + canonical_committed=self._canonical_committed, + duplicate_projections=self._duplicate_projections, + pending_canonical=self._pending_records, + pending_bytes=self._pending_bytes, + ) + + +async def supervise_stable_projector( + *, + broker_factory: Callable[[], tuple[ProjectorBroker, StableProjectorEngine]], + should_stop: Callable[[], bool], + on_broker: Callable[[ProjectorBroker | None], None], + sleep: Callable[[float], Awaitable[None]] = asyncio.sleep, + retry_initial_seconds: float = 0.25, + retry_max_seconds: float = 5.0, +) -> None: + """Recreate poisoned Kafka generations without weakening ACK ordering.""" + + if ( + retry_initial_seconds <= 0 + or retry_max_seconds < retry_initial_seconds + ): + raise ValueError("stable projector retry policy is invalid") + failures = 0 + while not should_stop(): + broker = None + try: + broker, engine = broker_factory() + on_broker(broker) + while not should_stop(): + if await engine.run_once(timeout_seconds=1.0): + failures = 0 + except asyncio.CancelledError: + raise + except Exception as error: # noqa: BLE001 - supervisor boundary + failures += 1 + logger.warning( + "stable projector generation failed; reconnecting attempt=%s error=%s", + failures, + error, + ) + finally: + on_broker(None) + if broker is not None: + try: + await asyncio.to_thread(broker.close) + except Exception as error: # noqa: BLE001 - poisoned generation cleanup + logger.warning( + "stable projector generation close failed during recovery error=%s", + error, + ) + if not should_stop(): + delay = min( + retry_max_seconds, + retry_initial_seconds * (2 ** min(max(failures - 1, 0), 8)), + ) + await sleep(delay) diff --git a/qdl/runtime/stable_source.py b/qdl/runtime/stable_source.py new file mode 100644 index 0000000..a1f3204 --- /dev/null +++ b/qdl/runtime/stable_source.py @@ -0,0 +1,567 @@ +from __future__ import annotations + +import hashlib +import time +from dataclasses import replace + +from qdl.common.v1 import common_pb2 +from qdl.domain.calendar import trading_calendar_for_id +from qdl.domain.decimal import CanonicalDecimal +from qdl.domain.quantity import quantity_unit_name +from qdl.marketdata.v2 import market_data_pb2 +from qdl.query import ( + AccessPurpose, + BarLifecycle, + ContractMetadata, + ConsumerGrade, + CoverageStatus, + DataProduct, + DataRequirement, + FeedType, + GapRecord, + HistoryResult, + InstrumentQuery, + MarketDataItem, + QualityMetadata, + SourceMetadata, + V2QueryService, +) +from qdl.replay import GapFreeHandoff +from qdl.runtime.stable_catalog import StableSourceBinding, StableSourceCatalog +from qdl.stream import GrpcSnapshot +from qdl.transport import Cursor, SQLiteDurableSpool, StoredEvent + + +def _decimal_text(value) -> str: + if value.source_text: + parsed = CanonicalDecimal.from_text(value.source_text) + coefficient = ( + value.mantissa_text + if value.WhichOneof("coefficient") == "mantissa_text" + else value.mantissa + ) + if str(parsed.coefficient) != str(coefficient) or parsed.scale != value.scale: + raise ValueError("canonical decimal text differs from coefficient/scale") + return value.source_text + coefficient = ( + value.mantissa_text + if value.WhichOneof("coefficient") == "mantissa_text" + else value.mantissa + ) + return format( + CanonicalDecimal(coefficient, value.scale, str(coefficient)).as_decimal(), + "f", + ) + + +def _interval_ns(interval: str) -> int: + units = { + "s": 1_000_000_000, + "m": 60 * 1_000_000_000, + "h": 60 * 60 * 1_000_000_000, + "d": 24 * 60 * 60 * 1_000_000_000, + } + if not interval or interval[-1] not in units: + raise ValueError("stable BAR interval is unsupported") + count = int(interval[:-1]) + if count <= 0: + raise ValueError("stable BAR interval must be positive") + return count * units[interval[-1]] + + +def _quality_flag_names(envelope: market_data_pb2.EventEnvelope) -> tuple[str, ...]: + return tuple( + common_pb2.QualityFlag.Name(value).removeprefix("QUALITY_FLAG_") + for value in envelope.quality_flags + ) + + +class StableSpoolQueryBackend: + """Provider-neutral stable query view over a Kafka-rebuildable SQLite cache.""" + + def __init__( + self, + spool: SQLiteDurableSpool, + catalog: StableSourceCatalog, + *, + schema_digest: str, + config_revision: int = 1, + clock_ns=time.time_ns, + ) -> None: + if len(schema_digest) != 64: + raise ValueError("stable query schema digest must be SHA-256") + self.spool = spool + self.catalog = catalog + self.schema_digest = schema_digest + self.config_revision = config_revision + self._clock_ns = clock_ns + + def latest(self, requirement: DataRequirement) -> MarketDataItem | None: + records = self._records( + requirement, limit=10_000 if requirement.feed is FeedType.BAR else 1 + ) + items = self._items(requirement, records) + return items[-1] if items else None + + def history(self, requirement: DataRequirement) -> HistoryResult | None: + requested = requirement.warmup_limit or 1 + all_records = self._records(requirement, limit=requested) + if not all_records: + return None + binding = self.catalog.binding_for(requirement) + gap_open = bool(self._gaps(binding, all_records)) + records = all_records[-requested:] + items = self._items(requirement, records, gap_open=gap_open) + last = records[-1] + snapshot_hash = hashlib.sha256( + f"{last.cursor.stream}|{last.cursor.partition_key}|{last.cursor.offset}|" + f"{last.event.event_id.hex()}".encode() + ).hexdigest() + return HistoryResult( + items=items, + coverage=( + CoverageStatus.FULL + if len(items) >= requested + else CoverageStatus.PARTIAL + ), + snapshot_id=f"qdl-v2-{snapshot_hash[:32]}", + stream_cursor="CONSUMER_CURSOR_PENDING", + watermark_offset=last.cursor.offset, + data_as_of_ns=items[-1].observed_at_ns, + ) + + def feed_status(self, requirement: DataRequirement) -> QualityMetadata | None: + item = self.latest(requirement) + return item.quality if item else None + + def open_gaps(self) -> tuple[GapRecord, ...]: + gaps = [] + for binding in self.catalog.bindings: + records = tuple(self.spool.read_tail( + stream=binding.canonical_stream, + partition_key=binding.partition_key, + limit=10_000, + )) + gaps.extend(self._gaps(binding, records)) + return tuple(sorted(gaps, key=lambda item: (item.detected_at_ns, item.gap_id))) + + def stored_events(self, requirement: DataRequirement) -> tuple[StoredEvent, ...]: + requested = requirement.warmup_limit or 1 + return self._records(requirement, limit=requested) + + def _records( + self, requirement: DataRequirement, *, limit: int + ) -> tuple[StoredEvent, ...]: + binding = self.catalog.binding_for(requirement) + rows = self.spool.read_tail( + stream=binding.canonical_stream, + partition_key=binding.partition_key, + limit=limit, + ) + selected = [] + for row in rows: + envelope = market_data_pb2.EventEnvelope.FromString(row.event.payload) + self.catalog.binding_for_envelope(envelope) + if binding.feed is not FeedType.BAR or envelope.bar.interval == binding.interval: + selected.append(row) + return tuple(selected) + + def _items( + self, + requirement: DataRequirement, + records: tuple[StoredEvent, ...], + *, + gap_open: bool | None = None, + ) -> tuple[MarketDataItem, ...]: + if not records: + return () + binding = self.catalog.binding_for(requirement) + effective_gap = ( + bool(self._gaps(binding, records)) if gap_open is None else gap_open + ) + return tuple( + self._item(binding, stored, market_data_pb2.EventEnvelope.FromString( + stored.event.payload + ), effective_gap) + for stored in records + ) + + def _quality( + self, + binding: StableSourceBinding, + envelope: market_data_pb2.EventEnvelope, + *, + gap_open: bool, + ) -> QualityMetadata: + observed_ns = ( + envelope.bar.close_time_ns + if envelope.WhichOneof("payload") == "bar" + else envelope.source_event_time_ns + ) + freshness_ms = max(0, (self._clock_ns() - observed_ns) // 1_000_000) + flags = _quality_flag_names(envelope) + explicit_gap = any( + value in {"SEQUENCE_GAP_BEFORE", "OUT_OF_ORDER", "RESYNC_REQUIRED"} + for value in flags + ) + gap_open = gap_open or explicit_gap + market_closed = False + if not binding.continuous_calendar: + market_closed = not trading_calendar_for_id( + binding.instrument.session_calendar_id + ).is_open_ns(self._clock_ns()) + stale = freshness_ms > binding.stale_after_ms + if market_closed: + state = "MARKET_CLOSED" + elif gap_open: + state = "GAPPED" + elif stale: + state = "STALE" + else: + state = "LIVE" + complete = not gap_open + execution_eligible = ( + binding.authoritative + and binding.source_role == "PRIMARY" + and state == "LIVE" + and complete + ) + return QualityMetadata( + state=state, + freshness_ms=int(freshness_ms), + gap_open=gap_open, + complete=complete, + execution_eligible=execution_eligible, + policy_id=binding.source_policy_id, + flags=flags + (("MARKET_CLOSED",) if market_closed else ()), + ) + + def _gaps( + self, + binding: StableSourceBinding, + records: tuple[StoredEvent, ...], + ) -> tuple[GapRecord, ...]: + detected_at_ns = self._clock_ns() + result = [] + for stored in records: + envelope = market_data_pb2.EventEnvelope.FromString(stored.event.payload) + if common_pb2.QUALITY_FLAG_SEQUENCE_GAP_BEFORE in envelope.quality_flags: + result.append(self._gap( + binding, + f"sequence:{envelope.source_sequence}", + envelope.source_sequence, + detected_at_ns, + )) + if binding.feed is not FeedType.BAR or not binding.continuous_calendar: + return tuple(result) + opens = sorted({ + market_data_pb2.EventEnvelope.FromString(item.event.payload).bar.open_time_ns + for item in records + }) + step = _interval_ns(binding.interval or "") + for previous, current in zip(opens, opens[1:], strict=False): + expected = previous + step + if current != expected: + result.append(self._gap( + binding, str(expected), str(current), detected_at_ns + )) + return tuple(result) + + @staticmethod + def _gap( + binding: StableSourceBinding, + expected: str, + observed: str, + detected_at_ns: int, + ) -> GapRecord: + digest = hashlib.sha256( + f"{binding.binding_id}|{expected}|{observed}".encode() + ).hexdigest()[:24] + return GapRecord( + gap_id=f"stable-{digest}", + instrument_uid=binding.instrument.instrument_uid, + feed=binding.feed, + source_id=binding.source_id, + expected_sequence=expected, + observed_sequence=observed, + detected_at_ns=detected_at_ns, + ) + + def _item( + self, + binding: StableSourceBinding, + stored: StoredEvent, + envelope: market_data_pb2.EventEnvelope, + gap_open: bool, + ) -> MarketDataItem: + payload_name = envelope.WhichOneof("payload") + quality = self._quality(binding, envelope, gap_open=gap_open) + source_role = common_pb2.SourceRole.Name(envelope.source_role).removeprefix( + "SOURCE_ROLE_" + ) + common = dict( + instrument_uid=envelope.instrument_uid, + instrument_id=envelope.instrument_id, + instrument_revision=int(envelope.instrument_revision), + observed_at_ns=int(envelope.source_event_time_ns), + source=SourceMetadata( + venue=envelope.venue, + provider=envelope.provider, + source_id=envelope.source_id, + source_role=source_role, + authoritative=binding.authoritative, + ), + quality=quality, + contract=ContractMetadata( + schema_digest=self.schema_digest, + contract_version="2.0.0", + normalizer_version=envelope.normalizer_version, + adapter_version=envelope.adapter_version, + instrument_catalog_revision=self.catalog.catalog_revision, + source_policy_revision=self.catalog.source_policy_revision, + authority_revision=self.catalog.authority_revision, + config_revision=max(1, int(envelope.config_revision or self.config_revision)), + correlation_id=envelope.correlation_id or stored.event.event_id.hex(), + ), + watermark_offset=stored.cursor.offset, + ) + if payload_name == "trade": + side = common_pb2.AggressorSide.Name( + envelope.trade.aggressor_side + ).removeprefix("AGGRESSOR_SIDE_") + if side == "UNSPECIFIED": + side = "UNKNOWN" + identity = market_data_pb2.TradeIdentityKind.Name( + envelope.trade.identity_kind + ).removeprefix("TRADE_IDENTITY_KIND_") + return MarketDataItem( + feed=FeedType.TRADE, + payload={ + "native_trade_id": envelope.trade.native_trade_id, + "price": _decimal_text(envelope.trade.price), + "quantity": _decimal_text(envelope.trade.quantity), + "quantity_unit": quantity_unit_name(envelope.trade.quantity_unit), + "aggressor_side": side, + "identity_kind": identity, + "is_block_trade": bool(envelope.trade.is_block_trade), + "is_buyer_maker": bool(envelope.trade.is_buyer_maker), + }, + **common, + ) + if payload_name == "quote": + return MarketDataItem( + feed=FeedType.QUOTE, + payload={ + "bid_price": _decimal_text(envelope.quote.bid_price), + "bid_quantity": _decimal_text(envelope.quote.bid_quantity), + "ask_price": _decimal_text(envelope.quote.ask_price), + "ask_quantity": _decimal_text(envelope.quote.ask_quantity), + "quantity_unit": quantity_unit_name(envelope.quote.quantity_unit), + "level": int(envelope.quote.level), + }, + **common, + ) + if payload_name == "bar": + lifecycle = market_data_pb2.BarLifecycle.Name( + envelope.bar.lifecycle + ).removeprefix("BAR_LIFECYCLE_") + origin = common_pb2.BarOrigin.Name(envelope.bar.origin).removeprefix( + "BAR_ORIGIN_" + ) + return MarketDataItem( + feed=FeedType.BAR, + interval=envelope.bar.interval, + revision=int(envelope.bar.revision), + bar_lifecycle=BarLifecycle(lifecycle), + supersedes_event_id=( + bytes(envelope.bar.supersedes_event_id).hex() + if envelope.bar.HasField("supersedes_event_id") + else None + ), + payload={ + "open_time_ns": int(envelope.bar.open_time_ns), + "close_time_ns": int(envelope.bar.close_time_ns), + "open": _decimal_text(envelope.bar.open), + "high": _decimal_text(envelope.bar.high), + "low": _decimal_text(envelope.bar.low), + "close": _decimal_text(envelope.bar.close), + "volume": _decimal_text(envelope.bar.volume), + "volume_unit": quantity_unit_name(envelope.bar.volume_unit), + "base_volume": ( + _decimal_text(envelope.bar.base_volume) + if envelope.bar.HasField("base_volume") + else None + ), + "quote_volume": ( + _decimal_text(envelope.bar.quote_volume) + if envelope.bar.HasField("quote_volume") + else None + ), + "contract_volume": ( + _decimal_text(envelope.bar.contract_volume) + if envelope.bar.HasField("contract_volume") + else None + ), + "trade_count": int(envelope.bar.trade_count), + "origin": origin, + "is_final": bool(envelope.bar.is_final), + }, + **common, + ) + raise ValueError("stable query backend supports only TRADE/QUOTE/BAR") + + +class StableCatalogCursorScopeValidator: + """Bind public cursor scope to the exact stable catalog requirement.""" + + def __init__(self, catalog: StableSourceCatalog) -> None: + self.catalog = catalog + + def validate( + self, + requirement: DataRequirement, + *, + stream: str, + partition_key: str, + ) -> None: + try: + binding = self.catalog.binding_for(requirement) + except KeyError as error: + raise ValueError( + "cursor requirement has no matching stable binding" + ) from error + if stream != binding.canonical_stream: + raise ValueError("cursor stream does not match the stable binding") + if partition_key != binding.partition_key: + raise ValueError("cursor partition does not match the stable binding") + + +class StableConsumerCursorIssuer: + def __init__( + self, + handoff: GapFreeHandoff, + catalog: StableSourceCatalog, + *, + ttl_seconds: int, + ) -> None: + self.handoff = handoff + self.catalog = catalog + self.ttl_seconds = ttl_seconds + + def bind_item( + self, + requirement: DataRequirement, + item: MarketDataItem, + *, + consumer_id: str, + ) -> MarketDataItem: + snapshot_id = self._snapshot_id(requirement, item.watermark_offset) + token = self._issue( + requirement, consumer_id, snapshot_id, item.watermark_offset + ) + return replace(item, snapshot_id=snapshot_id, cursor=token) + + def bind_history( + self, + requirement: DataRequirement, + history: HistoryResult, + *, + consumer_id: str, + ) -> HistoryResult: + token = self._issue( + requirement, + consumer_id, + history.snapshot_id, + history.watermark_offset, + ) + return replace( + history, + stream_cursor=token, + items=tuple( + replace(item, snapshot_id=history.snapshot_id, cursor=token) + for item in history.items + ), + ) + + def _issue( + self, + requirement: DataRequirement, + consumer_id: str, + snapshot_id: str, + offset: int, + ) -> str: + binding = self.catalog.binding_for(requirement) + return self.handoff.issue( + consumer_id=consumer_id, + snapshot_id=snapshot_id, + snapshot_watermark=Cursor( + binding.canonical_stream, + binding.partition_key, + offset, + ), + ttl_seconds=self.ttl_seconds, + ).token + + @staticmethod + def _snapshot_id(requirement: DataRequirement, offset: int) -> str: + digest = hashlib.sha256( + f"{requirement.instrument_uid}|{requirement.feed.value}|" + f"{requirement.interval or ''}|{offset}".encode() + ).hexdigest() + return f"qdl-v2-{digest[:32]}" + + +class StableGrpcSnapshotLoader: + def __init__( + self, + *, + service: V2QueryService, + backend: StableSpoolQueryBackend, + issuer: StableConsumerCursorIssuer, + ) -> None: + self.service = service + self.backend = backend + self.issuer = issuer + + def load(self, requirement: DataRequirement, *, consumer_id: str) -> GrpcSnapshot: + purpose = { + ConsumerGrade.ALPHA: AccessPurpose.INTERNAL_ALPHA, + ConsumerGrade.RESEARCH: AccessPurpose.INTERNAL_RESEARCH, + ConsumerGrade.EXECUTION: AccessPurpose.INTERNAL_EXECUTION, + }[requirement.consumer_grade] + result = self.service.warmup(requirement, purpose=purpose) + history = self.issuer.bind_history( + requirement, result.history, consumer_id=consumer_id + ) + events = tuple( + market_data_pb2.EventEnvelope.FromString(item.event.payload) + for item in self.backend.stored_events(requirement) + ) + return GrpcSnapshot( + request_id=result.request_id, + snapshot_id=history.snapshot_id, + stream_cursor=history.stream_cursor, + data_as_of_ns=history.data_as_of_ns, + watermark_offset=history.watermark_offset, + events=events, + ) + + +def build_stable_query_stack( + *, + spool: SQLiteDurableSpool, + catalog: StableSourceCatalog, + schema_digest: str, + handoff: GapFreeHandoff, + cursor_ttl_seconds: int, +) -> tuple[V2QueryService, StableSpoolQueryBackend, StableConsumerCursorIssuer]: + backend = StableSpoolQueryBackend(spool, catalog, schema_digest=schema_digest) + service = V2QueryService( + instruments=InstrumentQuery(catalog.instrument_registry()), + backend=backend, + entitlements=catalog.entitlements(), + ) + issuer = StableConsumerCursorIssuer( + handoff, catalog, ttl_seconds=cursor_ttl_seconds + ) + return service, backend, issuer diff --git a/qdl/runtime/stable_vn_edge.py b/qdl/runtime/stable_vn_edge.py new file mode 100644 index 0000000..d6b9ea3 --- /dev/null +++ b/qdl/runtime/stable_vn_edge.py @@ -0,0 +1,9 @@ +"""Compatibility import for the provider-owned stable VN acquisition edge.""" + +from qdl.adapters.vn.stable_edge import ( + StableDnseVendorEdge, + build_from_environment, + main, +) + +__all__ = ["StableDnseVendorEdge", "build_from_environment", "main"] diff --git a/qdl/security/data_plane.py b/qdl/security/data_plane.py index 896edbc..df81208 100644 --- a/qdl/security/data_plane.py +++ b/qdl/security/data_plane.py @@ -120,8 +120,10 @@ class RedisMinuteQuota: def __init__(self, redis: Redis, *, prefix: str) -> None: normalized = prefix.strip(": ") - if not normalized.startswith("qdl:beta:v2:"): - raise ValueError("shared quota requires a dedicated beta Redis prefix") + if not normalized.startswith(("qdl:beta:v2:", "qdl:stable:v2:")): + raise ValueError( + "shared quota requires a dedicated beta or stable Redis prefix" + ) self.redis = redis self.prefix = normalized diff --git a/qdl/security/grpc.py b/qdl/security/grpc.py index d498122..edc2808 100644 --- a/qdl/security/grpc.py +++ b/qdl/security/grpc.py @@ -100,12 +100,27 @@ async def unary_unary(request, context): async def unary_stream(request, context): request_access = await authorize(context) - token = _CURRENT_ACCESS.set(request_access) + iterator = behavior(request, context).__aiter__() try: - async for response in behavior(request, context): + while True: + token = _CURRENT_ACCESS.set(request_access) + try: + response = await iterator.__anext__() + except StopAsyncIteration: + return + finally: + _CURRENT_ACCESS.reset(token) + # Never retain a ContextVar token across a yielded + # response: gRPC may finalize the generator elsewhere. yield response finally: - _CURRENT_ACCESS.reset(token) + close = getattr(iterator, "aclose", None) + if close is not None: + token = _CURRENT_ACCESS.set(request_access) + try: + await close() + finally: + _CURRENT_ACCESS.reset(token) return grpc.unary_stream_rpc_method_handler( unary_stream, diff --git a/qdl/stream/__init__.py b/qdl/stream/__init__.py index 4654b4b..7a73ce5 100644 --- a/qdl/stream/__init__.py +++ b/qdl/stream/__init__.py @@ -8,6 +8,8 @@ StreamSubscription, ) from qdl.stream.grpc_service import ( + CursorScopeValidator, + FeedScopedCursorScopeValidator, GrpcMarketDataService, GrpcSnapshot, SnapshotLoader, @@ -22,6 +24,8 @@ "StreamCapacityExceeded", "StreamRecord", "StreamSubscription", + "CursorScopeValidator", + "FeedScopedCursorScopeValidator", "GrpcMarketDataService", "GrpcSnapshot", "SnapshotLoader", diff --git a/qdl/stream/gateway.py b/qdl/stream/gateway.py index 4c99cef..461332c 100644 --- a/qdl/stream/gateway.py +++ b/qdl/stream/gateway.py @@ -1,11 +1,18 @@ from __future__ import annotations import asyncio +from contextlib import AsyncExitStack from dataclasses import dataclass from typing import Protocol from qdl.replay import GapFreeHandoff -from qdl.transport import CursorExpired, DurableEvent, EventSink, StoredEvent +from qdl.transport import ( + BatchEventSink, + CursorExpired, + DurableEvent, + EventSink, + StoredEvent, +) class SlowConsumer(RuntimeError): @@ -204,21 +211,51 @@ async def open( async def publish(self, event: DurableEvent) -> StoredEvent | None: """Commit before delivery; duplicate durable events are not re-delivered.""" + return (await self.publish_many((event,)))[0] + + async def publish_many( + self, events: tuple[DurableEvent, ...] | list[DurableEvent] + ) -> tuple[StoredEvent | None, ...]: + """Durably append one bounded batch before ordered live fan-out.""" + + values = tuple(events) + if not values: + return () lease_epoch = self.assert_active() - partition_lock = self._partition_lock(event.stream, event.partition_key) - async with partition_lock: + partitions = sorted({(event.stream, event.partition_key) for event in values}) + async with AsyncExitStack() as stack: + for stream, partition_key in partitions: + await stack.enter_async_context(self._partition_lock(stream, partition_key)) self.assert_active(lease_epoch) - result = await asyncio.to_thread(self._sink.append, event) + if isinstance(self._sink, BatchEventSink): + results = await asyncio.to_thread(self._sink.append_many, list(values)) + else: + results = [ + await asyncio.to_thread(self._sink.append, event) for event in values + ] self.assert_active(lease_epoch) - if result.duplicate: - return None - stored = StoredEvent(event, result.cursor, result.committed_at_ns, result.payload_sha256) + if len(results) != len(values): + raise RuntimeError("durable batch sink returned an invalid result count") + stored_values = tuple( + None + if result.duplicate + else StoredEvent( + event, + result.cursor, + result.committed_at_ns, + result.payload_sha256, + ) + for event, result in zip(values, results, strict=True) + ) async with self._subscriptions_lock: subscriptions = tuple(self._subscriptions.values()) - for stream, partition_key, subscription in subscriptions: - if stream == event.stream and partition_key == event.partition_key: - subscription.push(stored) - return stored + for event, stored in zip(values, stored_values, strict=True): + if stored is None: + continue + for stream, partition_key, subscription in subscriptions: + if stream == event.stream and partition_key == event.partition_key: + subscription.push(stored) + return stored_values async def close(self, subscription_id: int) -> None: async with self._subscriptions_lock: diff --git a/qdl/stream/grpc_service.py b/qdl/stream/grpc_service.py index 4766a07..94ecd8e 100644 --- a/qdl/stream/grpc_service.py +++ b/qdl/stream/grpc_service.py @@ -38,6 +38,38 @@ class SnapshotLoader(Protocol): def load(self, requirement: DataRequirement, *, consumer_id: str) -> GrpcSnapshot: ... +class CursorScopeValidator(Protocol): + def validate( + self, + requirement: DataRequirement, + *, + stream: str, + partition_key: str, + ) -> None: ... + + +class FeedScopedCursorScopeValidator: + """Validate the original feed-specific canonical stream convention.""" + + def validate( + self, + requirement: DataRequirement, + *, + stream: str, + partition_key: str, + ) -> None: + parts = partition_key.split("/") + if ( + len(parts) != 3 + or not parts[2] + or parts[:2] + != [requirement.instrument_uid, requirement.feed.value.lower()] + ): + raise ValueError("cursor scope does not match the data requirement") + if stream != f"md.canonical.v2.{requirement.feed.value.lower()}": + raise ValueError("cursor stream does not match the data requirement") + + def requirement_from_proto(value: query_pb2.DataRequirement) -> DataRequirement: def enum_value(number: int, enum_wrapper, prefix: str) -> str: name = enum_wrapper.Name(number) @@ -81,10 +113,14 @@ def __init__( gateway: DurableStreamGateway, query_service: V2QueryService, snapshot_loader: SnapshotLoader, + cursor_scope_validator: CursorScopeValidator | None = None, ) -> None: self.gateway = gateway self.query_service = query_service self.snapshot_loader = snapshot_loader + self.cursor_scope_validator = ( + cursor_scope_validator or FeedScopedCursorScopeValidator() + ) @staticmethod def _event(stored: StoredEvent, token: str) -> query_pb2.StreamRecord: @@ -114,13 +150,9 @@ async def subscribe(self, request: query_pb2.SubscribeRequest, context): token=request.cursor_token, consumer_id=request.consumer_id ) stream, partition_key = scope.stream, scope.partition_key - parts = partition_key.split("/", 2) - if len(parts) != 3 or parts[:2] != [ - requirement.instrument_uid, requirement.feed.value.lower() - ]: - raise ValueError("cursor scope does not match the data requirement") - if stream != f"md.canonical.v2.{requirement.feed.value.lower()}": - raise ValueError("cursor stream does not match the data requirement") + self.cursor_scope_validator.validate( + requirement, stream=stream, partition_key=partition_key + ) subscription = await self.gateway.open( consumer_id=request.consumer_id, stream=stream, diff --git a/qdl/transport/__init__.py b/qdl/transport/__init__.py index dcc5fda..716df3e 100644 --- a/qdl/transport/__init__.py +++ b/qdl/transport/__init__.py @@ -3,6 +3,7 @@ from qdl.transport.contracts import ( AppendResult, BackpressureRequired, + BatchEventSink, CheckpointRegression, Cursor, CursorExpired, @@ -22,6 +23,7 @@ __all__ = [ "AppendResult", "BackpressureRequired", + "BatchEventSink", "CheckpointRegression", "Cursor", "CursorExpired", diff --git a/qdl/transport/kafka_projector.py b/qdl/transport/kafka_projector.py new file mode 100644 index 0000000..ebeec07 --- /dev/null +++ b/qdl/transport/kafka_projector.py @@ -0,0 +1,293 @@ +from __future__ import annotations + +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol + +try: + from confluent_kafka import Consumer, KafkaException, TopicPartition +except ImportError: # Construction fails closed when the production dependency is absent. + Consumer = None + KafkaException = RuntimeError + TopicPartition = None + + +_EVENT_ID_HEADER = "qdl-event-id" +_RAW_ENVELOPE_HEADER = "qdl-raw-provider-envelope" + + +@dataclass(frozen=True, slots=True) +class KafkaProjectorRecord: + topic: str + partition: int + offset: int + key: str + event_id: bytes + payload: bytes + accepted_at_ns: int + assignment_epoch: int = 1 + raw_provider_envelope: bytes | None = None + + def __post_init__(self) -> None: + if ( + not self.topic + or self.partition < 0 + or self.offset < 0 + or not self.key + or len(self.event_id) not in {16, 32} + or not self.payload + or self.accepted_at_ns <= 0 + or self.assignment_epoch < 1 + or self.raw_provider_envelope == b"" + ): + raise ValueError("Kafka projector record is invalid") + + +class ProjectorBroker(Protocol): + def poll(self, timeout_seconds: float) -> KafkaProjectorRecord | None: ... + def checkpoint(self, record: KafkaProjectorRecord) -> None: ... + def checkpoint_many( + self, records: tuple[KafkaProjectorRecord, ...] | list[KafkaProjectorRecord] + ) -> None: ... + def pause_canonical(self) -> None: ... + def resume_canonical(self) -> None: ... + def close(self) -> None: ... + + +@dataclass(frozen=True, slots=True) +class KafkaProjectorConfig: + bootstrap_servers: str + client_id: str + group_id: str + raw_topics: tuple[str, ...] + canonical_topic: str + ca_path: Path + certificate_path: Path + key_path: Path + session_timeout_ms: int = 45_000 + max_poll_interval_ms: int = 300_000 + checkpoint_batch_size: int = 128 + checkpoint_interval_ms: int = 100 + + def validate(self) -> None: + if not all(( + self.bootstrap_servers.strip(), self.client_id.strip(), self.group_id.strip(), + self.canonical_topic.strip(), + )): + raise ValueError("Kafka stable projector identity/topics are required") + topics = (*self.raw_topics, self.canonical_topic) + if any(not value.strip() for value in topics) or len(topics) != len(set(topics)): + raise ValueError("Kafka stable projector topics must be non-empty and unique") + if not 6_000 <= self.session_timeout_ms < self.max_poll_interval_ms: + raise ValueError("Kafka stable projector timeout policy is invalid") + if not 1 <= self.checkpoint_batch_size <= 10_000 or not ( + 10 <= self.checkpoint_interval_ms <= 5_000 + ): + raise ValueError("Kafka stable projector checkpoint policy is invalid") + for value in (self.ca_path, self.certificate_path, self.key_path): + if not value.is_file(): + raise ValueError(f"Kafka stable projector TLS file is unavailable: {value}") + + +class ConfluentProjectorBroker: + """Read-committed source with bounded post-ACK checkpoint coalescing.""" + + def __init__(self, config: KafkaProjectorConfig, *, consumer_factory=None) -> None: + config.validate() + factory = consumer_factory or Consumer + if factory is None: + raise RuntimeError("confluent-kafka runtime dependency is unavailable") + self.config = config + self._assignment_epoch = 1 + self._closed = False + self._pending_offsets: dict[tuple[str, int], int] = {} + self._pending_checkpoint_calls = 0 + self._last_checkpoint_flush = time.monotonic() + self._commit_error: BaseException | None = None + self._canonical_pause_requested = False + self._canonical_pause_applied = False + self._consumer = factory({ + "bootstrap.servers": config.bootstrap_servers, + "client.id": config.client_id, + "group.id": config.group_id, + "security.protocol": "ssl", + "ssl.ca.location": str(config.ca_path), + "ssl.certificate.location": str(config.certificate_path), + "ssl.key.location": str(config.key_path), + "enable.auto.commit": False, + "enable.auto.offset.store": False, + "auto.offset.reset": "earliest", + "isolation.level": "read_committed", + "session.timeout.ms": config.session_timeout_ms, + "max.poll.interval.ms": config.max_poll_interval_ms, + "on_commit": self._on_commit, + }) + self._consumer.subscribe( + [*config.raw_topics, config.canonical_topic], + on_assign=self._on_assignment, + on_revoke=self._on_assignment, + on_lost=self._on_assignment, + ) + + def _on_commit(self, error, _partitions) -> None: + if error is not None: + self._commit_error = KafkaException(error) + + def _raise_commit_error(self) -> None: + if self._commit_error is not None: + raise RuntimeError("asynchronous stable checkpoint failed") from self._commit_error + + def _on_assignment(self, _consumer, _partitions) -> None: + self._assignment_epoch += 1 + # These records were downstream-ACKed but not broker-checkpointed. The + # new owner must replay them through the idempotent spool/projection path. + self._pending_offsets.clear() + self._pending_checkpoint_calls = 0 + self._last_checkpoint_flush = time.monotonic() + self._canonical_pause_applied = False + + def _canonical_assignments(self): + assignment = getattr(self._consumer, "assignment", None) + if assignment is None: + raise RuntimeError("Kafka consumer does not expose assignment flow control") + return [ + item for item in assignment() + if getattr(item, "topic", None) == self.config.canonical_topic + ] + + def _apply_canonical_flow_control(self) -> None: + partitions = self._canonical_assignments() + if not partitions: + self._canonical_pause_applied = False + return + if self._canonical_pause_requested and not self._canonical_pause_applied: + self._consumer.pause(partitions) + self._canonical_pause_applied = True + elif not self._canonical_pause_requested and self._canonical_pause_applied: + self._consumer.resume(partitions) + self._canonical_pause_applied = False + + def pause_canonical(self) -> None: + if self._closed: + raise RuntimeError("Kafka stable projector consumer is closed") + self._canonical_pause_requested = True + self._apply_canonical_flow_control() + + def resume_canonical(self) -> None: + if self._closed: + raise RuntimeError("Kafka stable projector consumer is closed") + self._canonical_pause_requested = False + self._apply_canonical_flow_control() + + def _flush_checkpoints(self, *, asynchronous: bool) -> None: + if not self._pending_offsets: + return + self._raise_commit_error() + assert TopicPartition is not None + offsets = [ + TopicPartition(topic, partition, offset) + for (topic, partition), offset in sorted(self._pending_offsets.items()) + ] + result = self._consumer.commit(offsets=offsets, asynchronous=asynchronous) + if not asynchronous and result: + errors = [getattr(item, "error", None) for item in result] + if any(error is not None for error in errors): + raise RuntimeError("synchronous stable checkpoint failed") + self._pending_offsets.clear() + self._pending_checkpoint_calls = 0 + self._last_checkpoint_flush = time.monotonic() + + def poll(self, timeout_seconds: float) -> KafkaProjectorRecord | None: + if self._closed: + raise RuntimeError("Kafka stable projector consumer is closed") + if timeout_seconds <= 0: + raise ValueError("Kafka poll timeout must be positive") + self._raise_commit_error() + self._apply_canonical_flow_control() + elapsed_ms = (time.monotonic() - self._last_checkpoint_flush) * 1000 + if self._pending_offsets and elapsed_ms >= self.config.checkpoint_interval_ms: + self._flush_checkpoints(asynchronous=True) + message = self._consumer.poll(timeout_seconds) + self._raise_commit_error() + # Assignment callbacks run inside poll. Apply an already-requested pause + # immediately after a new assignment; the engine keeps one poll-batch of + # headroom for the record that may have triggered the callback. + self._apply_canonical_flow_control() + if message is None: + return None + if message.error(): + raise KafkaException(message.error()) + headers = dict(message.headers() or ()) + event_id = headers.get(_EVENT_ID_HEADER) + raw_provider_envelope = headers.get(_RAW_ENVELOPE_HEADER) + key = message.key() + payload = message.value() + timestamp_ms = message.timestamp()[1] + if event_id is None or key is None or payload is None or timestamp_ms is None: + raise ValueError("Kafka stable projector record is missing required metadata") + try: + decoded_key = bytes(key).decode("utf-8") + except UnicodeDecodeError as error: + raise ValueError("Kafka stable projector partition key is not UTF-8") from error + return KafkaProjectorRecord( + topic=message.topic(), + partition=int(message.partition()), + offset=int(message.offset()), + key=decoded_key, + event_id=bytes(event_id), + payload=bytes(payload), + accepted_at_ns=max(1, int(timestamp_ms) * 1_000_000), + assignment_epoch=self._assignment_epoch, + raw_provider_envelope=( + bytes(raw_provider_envelope) + if raw_provider_envelope is not None + else None + ), + ) + + def ping(self, timeout_seconds: float = 1.0) -> bool: + if self._closed: + return False + if timeout_seconds <= 0: + raise ValueError("Kafka metadata timeout must be positive") + self._raise_commit_error() + metadata = self._consumer.list_topics(timeout=timeout_seconds) + return metadata is not None + + def checkpoint(self, record: KafkaProjectorRecord) -> None: + self.checkpoint_many((record,)) + + def checkpoint_many( + self, records: tuple[KafkaProjectorRecord, ...] | list[KafkaProjectorRecord] + ) -> None: + values = tuple(records) + if not values: + return + if any(record.assignment_epoch != self._assignment_epoch for record in values): + raise RuntimeError("Kafka assignment changed before stable checkpoint") + self._raise_commit_error() + for record in values: + key = (record.topic, record.partition) + next_offset = record.offset + 1 + current = self._pending_offsets.get(key) + if current is not None and next_offset < current: + raise RuntimeError("Kafka stable projector checkpoint regressed") + self._pending_offsets[key] = max(current or 0, next_offset) + self._pending_checkpoint_calls += len(values) + elapsed_ms = (time.monotonic() - self._last_checkpoint_flush) * 1000 + if ( + self._pending_checkpoint_calls >= self.config.checkpoint_batch_size + or elapsed_ms >= self.config.checkpoint_interval_ms + ): + self._flush_checkpoints(asynchronous=True) + + def close(self) -> None: + if self._closed: + return + try: + self._raise_commit_error() + self._flush_checkpoints(asynchronous=False) + finally: + self._closed = True + self._consumer.close() diff --git a/qdl/transport/kafka_raw.py b/qdl/transport/kafka_raw.py new file mode 100644 index 0000000..9ce3d25 --- /dev/null +++ b/qdl/transport/kafka_raw.py @@ -0,0 +1,156 @@ +from __future__ import annotations + +import asyncio +from dataclasses import dataclass +from pathlib import Path +from threading import Lock +from time import monotonic +from typing import Iterable + +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.envelope import validate_raw_envelope + +try: + from confluent_kafka import Producer +except ImportError: # Dependency readiness must fail closed at construction. + Producer = None + + +@dataclass(frozen=True) +class KafkaRawPublisherConfig: + bootstrap_servers: str + client_id: str + topic: str + ca_path: Path + certificate_path: Path + key_path: Path + delivery_timeout_seconds: float = 30.0 + queue_max_messages: int = 100_000 + linger_ms: int = 5 + + def validate(self) -> None: + if not self.bootstrap_servers.strip() or not self.client_id.strip() or not self.topic.strip(): + raise ValueError("Kafka raw publisher identity/topic is required") + if self.delivery_timeout_seconds <= 0 or self.queue_max_messages <= 0: + raise ValueError("Kafka raw publisher bounds must be positive") + if not 0 <= self.linger_ms <= 1000: + raise ValueError("Kafka raw publisher linger_ms is invalid") + for value in (self.ca_path, self.certificate_path, self.key_path): + if not value.is_file(): + raise ValueError(f"Kafka TLS file is unavailable: {value}") + + +@dataclass(frozen=True) +class RawPublishAck: + capture_id: bytes + partition: int + offset: int + + +class KafkaRawPublisher: + """Idempotent TLS Kafka producer for Python vendor-SDK acquisition edges.""" + + def __init__(self, config: KafkaRawPublisherConfig, *, producer_factory=None): + config.validate() + factory = producer_factory or Producer + if factory is None: + raise RuntimeError("confluent-kafka runtime dependency is unavailable") + self.config = config + self._producer = factory({ + "bootstrap.servers": config.bootstrap_servers, + "client.id": config.client_id, + "security.protocol": "ssl", + "ssl.ca.location": str(config.ca_path), + "ssl.certificate.location": str(config.certificate_path), + "ssl.key.location": str(config.key_path), + "enable.idempotence": True, + "acks": "all", + "max.in.flight.requests.per.connection": 5, + "retries": 2_147_483_647, + "compression.type": "zstd", + "linger.ms": config.linger_ms, + "queue.buffering.max.messages": config.queue_max_messages, + "delivery.timeout.ms": int(config.delivery_timeout_seconds * 1000), + }) + self._lock = Lock() + self._closed = False + + @staticmethod + def _key(value: raw_provider_pb2.RawProviderEnvelope) -> str: + return "/".join(( + value.venue, + value.market, + value.native_symbol, + value.native_channel, + )) + + def publish_many( + self, values: Iterable[raw_provider_pb2.RawProviderEnvelope] + ) -> tuple[RawPublishAck, ...]: + envelopes = tuple(values) + if not envelopes: + raise ValueError("raw publish batch must not be empty") + for value in envelopes: + validate_raw_envelope(value) + with self._lock: + if self._closed: + raise RuntimeError("Kafka raw publisher is closed") + pending = {bytes(value.capture_id) for value in envelopes} + acknowledgements: list[RawPublishAck] = [] + failures: list[str] = [] + + def delivered(error, message, *, capture_id): + pending.discard(capture_id) + if error is not None: + failures.append(str(error)) + return + acknowledgements.append(RawPublishAck( + capture_id=capture_id, + partition=int(message.partition()), + offset=int(message.offset()), + )) + + deadline = monotonic() + self.config.delivery_timeout_seconds + for value in envelopes: + capture_id = bytes(value.capture_id) + while True: + try: + self._producer.produce( + self.config.topic, + key=self._key(value).encode(), + value=value.SerializeToString(deterministic=True), + headers=[("qdl-event-id", capture_id)], + on_delivery=lambda error, message, capture_id=capture_id: delivered( + error, message, capture_id=capture_id + ), + ) + break + except BufferError: + if monotonic() >= deadline: + raise TimeoutError("Kafka raw producer queue remained full") + self._producer.poll(0.05) + self._producer.poll(0) + remaining = max(0.0, deadline - monotonic()) + undelivered = self._producer.flush(remaining) + if undelivered or pending or failures: + raise RuntimeError( + "Kafka raw durable ACK failed " + f"undelivered={undelivered} pending={len(pending)} failures={failures[:3]}" + ) + acknowledgements.sort(key=lambda item: item.capture_id) + return tuple(acknowledgements) + + async def publish_many_async( + self, values: Iterable[raw_provider_pb2.RawProviderEnvelope] + ) -> tuple[RawPublishAck, ...]: + batch = tuple(values) + return await asyncio.to_thread(self.publish_many, batch) + + def close(self) -> None: + with self._lock: + if self._closed: + return + remaining = self._producer.flush(self.config.delivery_timeout_seconds) + self._closed = True + if remaining: + raise RuntimeError(f"Kafka raw publisher closed with {remaining} undelivered records") diff --git a/qdl/transport/sqlite_spool.py b/qdl/transport/sqlite_spool.py index 2ff2695..3febf82 100644 --- a/qdl/transport/sqlite_spool.py +++ b/qdl/transport/sqlite_spool.py @@ -6,6 +6,7 @@ import sqlite3 import threading import time +import uuid from dataclasses import dataclass from pathlib import Path @@ -37,6 +38,7 @@ class SpoolConfig: consumer_ttl_seconds: int = 3600 replay_retention_seconds: int = 24 * 3600 maintenance_interval_seconds: int = 30 + max_partition_records: int = 0 def __post_init__(self) -> None: if self.max_records <= 0 or self.max_payload_bytes <= 0: @@ -45,6 +47,8 @@ def __post_init__(self) -> None: raise ValueError("max_event_bytes must fit inside max_payload_bytes") if self.max_batch_events <= 0: raise ValueError("max_batch_events must be positive") + if self.max_partition_records < 0: + raise ValueError("max_partition_records cannot be negative") if self.max_storage_bytes <= self.max_event_bytes: raise ValueError("max_storage_bytes must exceed max_event_bytes") if min( @@ -106,8 +110,8 @@ def _configure(self) -> None: self._connection.execute("PRAGMA synchronous=FULL") self._connection.execute("PRAGMA foreign_keys=ON") self._connection.execute("PRAGMA busy_timeout=10000") - self._connection.execute("PRAGMA wal_autocheckpoint=100") - self._connection.execute("PRAGMA journal_size_limit=16777216") + self._connection.execute("PRAGMA wal_autocheckpoint=1000") + self._connection.execute("PRAGMA journal_size_limit=67108864") def _migrate(self) -> None: self._connection.executescript( @@ -171,9 +175,33 @@ def _migrate(self) -> None: singleton, event_records, payload_bytes, last_maintenance_ns ) SELECT 1, COUNT(*), COALESCE(SUM(LENGTH(payload)), 0), 0 FROM events; + + CREATE TABLE IF NOT EXISTS cache_identity ( + singleton INTEGER PRIMARY KEY CHECK (singleton = 1), + cache_id TEXT NOT NULL, + created_at_ns INTEGER NOT NULL + ); + """ + ) + self._connection.execute( """ + INSERT OR IGNORE INTO cache_identity(singleton, cache_id, created_at_ns) + VALUES (1, ?, ?) + """, + (uuid.uuid4().hex, self._clock_ns()), ) + @property + def cache_id(self) -> str: + with self._lock: + row = self._connection.execute( + "SELECT cache_id FROM cache_identity WHERE singleton = 1" + ).fetchone() + value = str(row["cache_id"]) if row is not None else "" + if len(value) != 32 or any(char not in "0123456789abcdef" for char in value): + raise PayloadCorruption("spool cache identity is invalid") + return value + def append(self, event: DurableEvent) -> AppendResult: return self.append_many([event])[0] @@ -308,6 +336,10 @@ def append_many(self, events: list[DurableEvent]) -> list[AppendResult]: """, (added_records, added_payload_bytes), ) + if self.config.max_partition_records: + self._trim_partition_windows_locked({ + (event.stream, event.partition_key) for event in events + }) self._connection.execute("COMMIT") return results except BaseException: @@ -347,6 +379,28 @@ def read( ).fetchall() return [self._stored_event(row) for row in rows] + def read_tail( + self, + *, + stream: str, + partition_key: str, + limit: int = 100, + ) -> list[StoredEvent]: + """Return the newest bounded partition window in logical order.""" + + if limit <= 0 or limit > 10_000: + raise ValueError("limit must be between 1 and 10000") + with self._lock: + rows = self._connection.execute( + """ + SELECT * FROM events + WHERE stream = ? AND partition_key = ? + ORDER BY logical_offset DESC LIMIT ? + """, + (stream, partition_key, limit), + ).fetchall() + return [self._stored_event(row) for row in reversed(rows)] + def find_event(self, *, stream: str, event_id: bytes) -> StoredEvent | None: with self._lock: row = self._connection.execute( @@ -355,6 +409,30 @@ def find_event(self, *, stream: str, event_id: bytes) -> StoredEvent | None: ).fetchone() return self._stored_event(row) if row is not None else None + def find_events( + self, *, stream: str, event_ids: list[bytes] | tuple[bytes, ...] + ) -> dict[bytes, StoredEvent]: + """Resolve one bounded immutable-event batch without per-event queries.""" + + values = tuple(dict.fromkeys(event_ids)) + if not values: + return {} + if len(values) > 10_000: + raise ValueError("event lookup batch exceeds the bounded query window") + rows = [] + with self._lock: + for start in range(0, len(values), 500): + chunk = values[start:start + 500] + placeholders = ",".join("?" for _ in chunk) + rows.extend(self._connection.execute( + f"SELECT * FROM events WHERE stream = ? AND event_id IN ({placeholders})", + (stream, *chunk), + ).fetchall()) + return { + bytes(row["event_id"]): self._stored_event(row) + for row in rows + } + def register_consumer( self, *, @@ -675,6 +753,44 @@ def _trim_aged_unowned_locked(self, now_ns: int) -> None: ) self._decrement_usage_locked(int(removed[0]), int(removed[1])) + def _trim_partition_windows_locked( + self, partitions: set[tuple[str, str]] + ) -> None: + """Keep the newest replay window; older cursors fail with CursorExpired.""" + + limit = self.config.max_partition_records + if limit <= 0: + return + for stream, partition_key in partitions: + threshold = self._connection.execute( + """ + SELECT logical_offset FROM events + WHERE stream = ? AND partition_key = ? + ORDER BY logical_offset DESC LIMIT 1 OFFSET ? + """, + (stream, partition_key, limit - 1), + ).fetchone() + if threshold is None: + continue + removed = self._connection.execute( + """ + SELECT COUNT(*), COALESCE(SUM(LENGTH(payload)), 0) + FROM events + WHERE stream = ? AND partition_key = ? AND logical_offset < ? + """, + (stream, partition_key, int(threshold[0])), + ).fetchone() + if int(removed[0]) == 0: + continue + self._connection.execute( + """ + DELETE FROM events + WHERE stream = ? AND partition_key = ? AND logical_offset < ? + """, + (stream, partition_key, int(threshold[0])), + ) + self._decrement_usage_locked(int(removed[0]), int(removed[1])) + @staticmethod def _stored_event(row: sqlite3.Row) -> StoredEvent: payload = bytes(row["payload"]) diff --git a/qdl_sdk/__init__.py b/qdl_sdk/__init__.py index 840e007..9d2a8e0 100644 --- a/qdl_sdk/__init__.py +++ b/qdl_sdk/__init__.py @@ -1,5 +1,7 @@ """Stable Python SDK V2 with typed public models and generated transports.""" +__version__ = "2.0.0" + from qdl_sdk.client import AsyncDataLayerClient, DataLayerClientV2, WarmupStreamSession from qdl_sdk.credentials import ( CallbackCredentialProvider, @@ -16,16 +18,19 @@ GapPolicy, Grade, MarketDataView, + QuantityUnit, RecoveryPolicy, SnapshotResponse, StalePolicy, StreamEvent, + TradeIdentityKind, WarmupResponse, ) from qdl_sdk.transport import GrpcStreamTransport, RestQueryTransport from qdl_sdk.v1_facade import V1CompatibilityFacade __all__ = [ + "__version__", "AsyncDataLayerClient", "BarRevisionPolicy", "CallbackCredentialProvider", @@ -42,12 +47,14 @@ "GrpcStreamTransport", "MemoryCursorStore", "MarketDataView", + "QuantityUnit", "RestQueryTransport", "RecoveryPolicy", "SnapshotResponse", "StalePolicy", "StaticBearerCredential", "StreamEvent", + "TradeIdentityKind", "WarmupResponse", "V1CompatibilityFacade", "WarmupStreamSession", diff --git a/qdl_sdk/client.py b/qdl_sdk/client.py index 828522a..944155b 100644 --- a/qdl_sdk/client.py +++ b/qdl_sdk/client.py @@ -65,7 +65,8 @@ def _validate_query_payload( raise ContinuityError("PARTIAL_RESULT", "warmup count does not match returned rows") if requirement.require_full_coverage and response.coverage != "FULL": raise ContinuityError("PARTIAL_RESULT", "warmup response is not full coverage") - for row in rows: + for index, row in enumerate(rows): + is_tail = index == len(rows) - 1 if row.instrument_uid != requirement.instrument_uid: raise ContinuityError("CONFLICT", "query response instrument does not match requirement") if row.feed.value != requirement.feed.value: @@ -80,18 +81,26 @@ def _validate_query_payload( state = quality.state.upper() freshness_ms = quality.freshness_ms if ( - requirement.max_freshness_ms is not None + is_tail + and state != "MARKET_CLOSED" + and requirement.max_freshness_ms is not None and freshness_ms > requirement.max_freshness_ms and requirement.stale_policy.value in {"BLOCK", "PAUSE"} ): raise ContinuityError("DATA_STALE", "query response exceeds freshness policy") if quality.gap_open and requirement.gap_policy.value in {"BLOCK", "PAUSE"}: raise ContinuityError("OPEN_SEQUENCE_GAP", "query response has an open gap") - if state in {"STALE", "OFFLINE", "UNAVAILABLE"} and requirement.stale_policy.value in { + if is_tail and state in { + "STALE", "OFFLINE", "UNAVAILABLE" + } and requirement.stale_policy.value in { "BLOCK", "PAUSE", }: raise ContinuityError("DATA_STALE", f"query response quality state is {state}") - if requirement.consumer_grade is Grade.EXECUTION and not quality.execution_eligible: + if ( + is_tail + and requirement.consumer_grade is Grade.EXECUTION + and not quality.execution_eligible + ): raise ContinuityError( "SOURCE_NON_AUTHORITATIVE", "execution-grade response is not execution eligible", diff --git a/qdl_sdk/models.py b/qdl_sdk/models.py index 782300d..f0fd5a1 100644 --- a/qdl_sdk/models.py +++ b/qdl_sdk/models.py @@ -4,7 +4,13 @@ from enum import StrEnum from typing import Any -from qdl.api_v2.models import MarketDataView, SnapshotResponse, WarmupResponse +from qdl.api_v2.models import ( + MarketDataView, + QuantityUnit, + SnapshotResponse, + TradeIdentityKind, + WarmupResponse, +) from qdl.query.v2 import query_pb2 diff --git a/rust/qdl-contracts/src/lib.rs b/rust/qdl-contracts/src/lib.rs index b132d68..e12c6c4 100644 --- a/rust/qdl-contracts/src/lib.rs +++ b/rust/qdl-contracts/src/lib.rs @@ -59,8 +59,10 @@ pub mod qdl { #[cfg(test)] mod tests { - use super::qdl::common::v1::{decimal_value, AggressorSide, DecimalValue, SourceRole}; - use super::qdl::marketdata::v2::{event_envelope, EventEnvelope, Trade}; + use super::qdl::common::v1::{ + decimal_value, AggressorSide, DecimalValue, QuantityUnit, SourceRole, + }; + use super::qdl::marketdata::v2::{event_envelope, EventEnvelope, Trade, TradeIdentityKind}; use prost::Message; fn decimal(mantissa: i64, scale: i32, source_text: &str) -> DecimalValue { @@ -113,6 +115,8 @@ mod tests { aggressor_side: AggressorSide::Buy as i32, is_block_trade: false, is_buyer_maker: false, + quantity_unit: QuantityUnit::BaseAsset as i32, + identity_kind: TradeIdentityKind::Native as i32, })), } } diff --git a/rust/qdl-core/src/binance.rs b/rust/qdl-core/src/binance.rs index 04d8489..ec1208e 100644 --- a/rust/qdl-core/src/binance.rs +++ b/rust/qdl-core/src/binance.rs @@ -3,7 +3,7 @@ use serde_json::Value; use crate::canonical::{canonical_bytes, TradeContext, TradeFixture}; -pub const WS_COMBINED_BASE: &str = "wss://fstream.binance.com/stream?streams="; +pub const WS_COMBINED_BASE: &str = "wss://fstream.binance.com/public/stream?streams="; #[derive(Clone, Debug, Deserialize)] pub struct ShadowConfig { diff --git a/rust/qdl-core/src/canonical.rs b/rust/qdl-core/src/canonical.rs index d81d580..bd68960 100644 --- a/rust/qdl-core/src/canonical.rs +++ b/rust/qdl-core/src/canonical.rs @@ -1,7 +1,10 @@ use prost::Message; -use qdl_contracts::qdl::common::v1::{AggressorSide, BarOrigin, BookSide, QualityFlag, SourceRole}; +use qdl_contracts::qdl::common::v1::{ + AggressorSide, BarOrigin, BookSide, QualityFlag, QuantityUnit, SourceRole, +}; use qdl_contracts::qdl::marketdata::v2::{ event_envelope, Bar, BarLifecycle, BookLevel, EventEnvelope, OrderBookSnapshot, Quote, Trade, + TradeIdentityKind, }; use serde::Deserialize; use serde_json::Value; @@ -43,6 +46,12 @@ pub struct TradeContext { pub raw_capture_id: Vec, #[serde(default)] pub raw_frame_sha256: Vec, + #[serde(default = "default_source_role")] + pub source_role: String, +} + +fn default_source_role() -> String { + "PRIMARY".into() } #[derive(Clone, Debug, Deserialize)] @@ -109,16 +118,32 @@ fn canonical_json(raw: &Value) -> Result, String> { pub fn canonicalize_trade(fixture: &TradeFixture) -> Result { match fixture.provider_kind.as_str() { - "binance_usdm_trade" | "binance_usdm_agg_trade" => canonicalize_binance(fixture), - "binance_usdm_bbo" => canonicalize_binance_bbo(fixture), - "binance_usdm_bar" => canonicalize_binance_bar(fixture), + "binance_usdm_trade" | "binance_usdm_agg_trade" | "binance_spot_trade" => { + canonicalize_binance(fixture) + } + "binance_usdm_bbo" | "binance_spot_bbo" => canonicalize_binance_bbo(fixture), + "binance_usdm_bar" | "binance_spot_bar" => canonicalize_binance_bar(fixture), + "binance_usdm_rest_bar" | "binance_spot_rest_bar" => canonicalize_binance_rest_bar(fixture), "okx_trade" => canonicalize_okx(fixture), - "dnse_bar" => canonicalize_dnse_bar(fixture), + "okx_bbo" => canonicalize_okx_bbo(fixture), + "okx_bar" => canonicalize_okx_bar(fixture), + "dnse_trade" => canonicalize_dnse_trade(fixture), + "dnse_bar" | "vnstock_bar" => canonicalize_dnse_bar(fixture), "deribit_option_book_fixture" => canonicalize_deribit_fixture(fixture), other => Err(format!("unsupported provider fixture: {other}")), } } +fn source_role(value: &str) -> Result { + match value.trim().to_ascii_uppercase().as_str() { + "PRIMARY" => Ok(SourceRole::Primary), + "SECONDARY" => Ok(SourceRole::Secondary), + "REFERENCE" => Ok(SourceRole::Reference), + "BACKFILL" => Ok(SourceRole::Backfill), + _ => Err("canonical source role is invalid".into()), + } +} + fn base_envelope( fixture: &TradeFixture, feed: &str, @@ -154,7 +179,7 @@ fn base_envelope( native_symbol: context.native_symbol.clone(), provider: context.provider.clone(), source_id: context.source_id.clone(), - source_role: SourceRole::Primary as i32, + source_role: source_role(&context.source_role)? as i32, lease_epoch: context.lease_epoch, source_event_time_ns: source_event_time_ms * 1_000_000, received_at_ns: context.received_at_ns, @@ -192,6 +217,54 @@ fn validate_shadow_context(context: &TradeContext) -> Result<(), String> { Ok(()) } +fn quantity_unit(context: &TradeContext) -> Result { + let venue = context.venue.trim().to_ascii_uppercase(); + let market = context.market.trim().to_ascii_uppercase(); + let product = context.product_type.trim().to_ascii_uppercase(); + if venue.is_empty() || market.is_empty() || product.is_empty() { + return Err("quantity-unit identity is incomplete".into()); + } + if product == "COMMON_STOCK" { + return if matches!( + venue.as_str(), + "DNSE" | "HOSE" | "HNX" | "UPCOM" | "VN_MARKETS" + ) { + Ok(QuantityUnit::Share) + } else { + Err("COMMON_STOCK quantity unit requires a VN venue identity".into()) + }; + } + if product == "FUTURE" + && (matches!( + venue.as_str(), + "DNSE" | "HOSE" | "HNX" | "UPCOM" | "VN_MARKETS" + ) || matches!(market.as_str(), "VN_DERIVATIVES" | "DERIVATIVES")) + { + return Ok(QuantityUnit::Contract); + } + if venue == "OKX" { + return match (market.as_str(), product.as_str()) { + ("SPOT", "SPOT") => Ok(QuantityUnit::BaseAsset), + ("SWAP", "PERPETUAL") | ("FUTURES", "FUTURE") | ("OPTIONS", "OPTION") => { + Ok(QuantityUnit::Contract) + } + _ => Err("unsupported OKX quantity-unit identity".into()), + }; + } + if venue == "DERIBIT" && product == "OPTION" { + return Ok(QuantityUnit::Contract); + } + if venue == "BINANCE" { + return match (market.as_str(), product.as_str()) { + ("SPOT", "SPOT") | ("USDM", "PERPETUAL") => Ok(QuantityUnit::BaseAsset), + _ => Err("unsupported Binance quantity-unit identity".into()), + }; + } + Err(format!( + "quantity unit is undefined for {venue}/{market}/{product}" + )) +} + fn verify_binance_symbol(fixture: &TradeFixture) -> Result<(), String> { if text(&fixture.raw, "s")?.to_uppercase() != fixture.context.native_symbol.to_uppercase() { return Err("provider symbol does not match resolved instrument".into()); @@ -202,19 +275,29 @@ fn verify_binance_symbol(fixture: &TradeFixture) -> Result<(), String> { fn canonicalize_binance_bbo(fixture: &TradeFixture) -> Result { verify_binance_symbol(fixture)?; let sequence = text(&fixture.raw, "u")?; - let source_time = fixture + let provider_time = fixture .raw .get("T") .and_then(Value::as_i64) - .or_else(|| fixture.raw.get("E").and_then(Value::as_i64)) - .ok_or_else(|| "required provider timestamp is missing".to_owned())?; - let mut envelope = base_envelope(fixture, "quote", sequence, source_time)?; + .or_else(|| fixture.raw.get("E").and_then(Value::as_i64)); + let mut envelope = base_envelope( + fixture, + "quote", + sequence, + provider_time.unwrap_or(fixture.context.received_at_ns / 1_000_000), + )?; + if provider_time.is_none() { + envelope + .quality_flags + .push(QualityFlag::SourceTimeMissing as i32); + } envelope.payload = Some(event_envelope::Payload::Quote(Quote { bid_price: Some(parse_decimal(&text(&fixture.raw, "b")?)?), bid_quantity: Some(parse_decimal(&text(&fixture.raw, "B")?)?), ask_price: Some(parse_decimal(&text(&fixture.raw, "a")?)?), ask_quantity: Some(parse_decimal(&text(&fixture.raw, "A")?)?), level: 1, + quantity_unit: quantity_unit(&fixture.context)? as i32, })); set_payload_hash(&mut envelope)?; Ok(envelope) @@ -257,6 +340,79 @@ fn canonicalize_binance_bar(fixture: &TradeFixture) -> Result Result { + if text(&fixture.raw, "symbol")?.to_uppercase() != fixture.context.native_symbol.to_uppercase() + { + return Err("provider symbol does not match resolved instrument".into()); + } + let row = fixture + .raw + .get("row") + .and_then(Value::as_array) + .filter(|row| row.len() >= 11) + .ok_or_else(|| "Binance REST kline requires the unmodified native row".to_owned())?; + let origin_name = fixture + .raw + .get("bar_origin") + .and_then(Value::as_str) + .unwrap_or("BACKFILLED") + .to_ascii_uppercase(); + let origin = match origin_name.as_str() { + "VENUE_NATIVE" => BarOrigin::VenueNative, + "BACKFILLED" => BarOrigin::Backfilled, + "RECONCILED" => BarOrigin::Reconciled, + _ => return Err("Binance REST bar origin is invalid".into()), + }; + let open_time_ms = scalar_text(&row[0])? + .parse::() + .map_err(|_| "Binance REST open time is invalid".to_owned())?; + let close_time_ms = scalar_text(&row[6])? + .parse::() + .map_err(|_| "Binance REST close time is invalid".to_owned())?; + let mut envelope = base_envelope( + fixture, + "bar", + format!("{open_time_ms}:{close_time_ms}"), + close_time_ms, + )?; + if origin == BarOrigin::Backfilled { + envelope.quality_flags.push(QualityFlag::Backfilled as i32); + } + envelope.payload = Some(event_envelope::Payload::Bar(Bar { + interval: text(&fixture.raw, "interval")?, + open_time_ns: open_time_ms * 1_000_000, + close_time_ns: close_time_ms * 1_000_000, + open: Some(parse_decimal(&scalar_text(&row[1])?)?), + high: Some(parse_decimal(&scalar_text(&row[2])?)?), + low: Some(parse_decimal(&scalar_text(&row[3])?)?), + close: Some(parse_decimal(&scalar_text(&row[4])?)?), + volume: Some(parse_decimal(&scalar_text(&row[5])?)?), + trade_count: scalar_text(&row[8])? + .parse::() + .map_err(|_| "Binance REST trade count is invalid".to_owned())?, + is_final: true, + revision: 0, + origin: origin as i32, + lifecycle: BarLifecycle::Final as i32, + supersedes_event_id: None, + volume_unit: quantity_unit(&fixture.context)? as i32, + base_volume: Some(parse_decimal(&scalar_text(&row[5])?)?), + quote_volume: Some(parse_decimal(&scalar_text(&row[7])?)?), + contract_volume: None, })); set_payload_hash(&mut envelope)?; Ok(envelope) @@ -270,21 +426,24 @@ fn canonicalize_binance(fixture: &TradeFixture) -> Result let buyer_maker = boolean(&fixture.raw, "m")?; build_trade( fixture, - native_trade_id, - text(&fixture.raw, "p")?, - text(&fixture.raw, "q")?, - if buyer_maker { - AggressorSide::Sell - } else { - AggressorSide::Buy + TradeInput { + native_trade_id, + price: text(&fixture.raw, "p")?, + quantity: text(&fixture.raw, "q")?, + side: if buyer_maker { + AggressorSide::Sell + } else { + AggressorSide::Buy + }, + source_event_time_ms: fixture + .raw + .get("T") + .and_then(Value::as_i64) + .or_else(|| fixture.raw.get("E").and_then(Value::as_i64)) + .ok_or_else(|| "required provider timestamp is missing".to_owned())?, + is_buyer_maker: buyer_maker, + identity_kind: TradeIdentityKind::Native, }, - fixture - .raw - .get("T") - .and_then(Value::as_i64) - .or_else(|| fixture.raw.get("E").and_then(Value::as_i64)) - .ok_or_else(|| "required provider timestamp is missing".to_owned())?, - buyer_maker, ) } @@ -299,15 +458,162 @@ fn canonicalize_okx(fixture: &TradeFixture) -> Result { }; build_trade( fixture, - text(&fixture.raw, "tradeId")?, - text(&fixture.raw, "px")?, - text(&fixture.raw, "sz")?, - side, - integer(&fixture.raw, "ts")?, - false, + TradeInput { + native_trade_id: text(&fixture.raw, "tradeId")?, + price: text(&fixture.raw, "px")?, + quantity: text(&fixture.raw, "sz")?, + side, + source_event_time_ms: integer(&fixture.raw, "ts")?, + is_buyer_maker: false, + identity_kind: TradeIdentityKind::Native, + }, ) } +fn scalar_text(value: &Value) -> Result { + match value { + Value::String(value) if !value.is_empty() => Ok(value.clone()), + Value::Number(value) => Ok(value.to_string()), + _ => Err("provider scalar is missing or invalid".into()), + } +} + +fn okx_frame_row<'a>(fixture: &'a TradeFixture, channel: &str) -> Result<&'a Value, String> { + let argument = fixture + .raw + .get("arg") + .and_then(Value::as_object) + .ok_or_else(|| "OKX frame arg must be an object".to_owned())?; + if argument.get("channel").and_then(Value::as_str) != Some(channel) + || argument.get("instId").and_then(Value::as_str) + != Some(fixture.context.native_symbol.as_str()) + { + return Err("OKX frame channel/instrument mismatch".into()); + } + let rows = fixture + .raw + .get("data") + .and_then(Value::as_array) + .filter(|rows| rows.len() == 1) + .ok_or_else(|| "OKX frame requires one data row".to_owned())?; + Ok(&rows[0]) +} + +fn okx_bbo_level<'a>(row: &'a Value, side: &str) -> Result<&'a Vec, String> { + row.get(side) + .and_then(Value::as_array) + .filter(|levels| levels.len() == 1) + .and_then(|levels| levels[0].as_array()) + .filter(|level| level.len() >= 2) + .ok_or_else(|| format!("OKX bbo-tbt requires one {side} level")) +} + +fn canonicalize_okx_bbo(fixture: &TradeFixture) -> Result { + let row = okx_frame_row(fixture, "bbo-tbt")?; + let sequence = text(row, "seqId")?; + let source_time = integer(row, "ts")?; + let bid = okx_bbo_level(row, "bids")?; + let ask = okx_bbo_level(row, "asks")?; + let mut envelope = base_envelope(fixture, "quote", sequence, source_time)?; + envelope.payload = Some(event_envelope::Payload::Quote(Quote { + bid_price: Some(parse_decimal(&scalar_text(&bid[0])?)?), + bid_quantity: Some(parse_decimal(&scalar_text(&bid[1])?)?), + ask_price: Some(parse_decimal(&scalar_text(&ask[0])?)?), + ask_quantity: Some(parse_decimal(&scalar_text(&ask[1])?)?), + level: 1, + quantity_unit: quantity_unit(&fixture.context)? as i32, + })); + set_payload_hash(&mut envelope)?; + Ok(envelope) +} + +fn canonicalize_okx_bar(fixture: &TradeFixture) -> Result { + let row = okx_frame_row(fixture, "candle1m")? + .as_array() + .filter(|values| values.len() >= 9) + .ok_or_else(|| "OKX candle1m requires the native nine-field row".to_owned())?; + let open_time_ms = scalar_text(&row[0])? + .parse::() + .map_err(|_| "OKX candle open timestamp is invalid".to_owned())?; + let confirm = scalar_text(&row[8])?; + let is_final = match confirm.as_str() { + "0" => false, + "1" => true, + _ => return Err("OKX candle confirm must be 0 or 1".into()), + }; + // Keep one provider event identity across REST bootstrap, WebSocket delivery, + // and process restart. A same-identity content conflict must fail closed. + let sequence = format!("{open_time_ms}:{confirm}"); + let mut envelope = base_envelope(fixture, "bar", sequence, open_time_ms)?; + envelope + .quality_flags + .push(QualityFlag::FieldMissing as i32); + envelope.payload = Some(event_envelope::Payload::Bar(Bar { + interval: "1m".into(), + open_time_ns: open_time_ms * 1_000_000, + close_time_ns: (open_time_ms + 60_000 - 1) * 1_000_000, + open: Some(parse_decimal(&scalar_text(&row[1])?)?), + high: Some(parse_decimal(&scalar_text(&row[2])?)?), + low: Some(parse_decimal(&scalar_text(&row[3])?)?), + close: Some(parse_decimal(&scalar_text(&row[4])?)?), + volume: Some(parse_decimal(&scalar_text(&row[5])?)?), + trade_count: 0, + is_final, + revision: 0, + origin: BarOrigin::VenueNative as i32, + lifecycle: if is_final { + BarLifecycle::Final as i32 + } else { + BarLifecycle::InProgress as i32 + }, + supersedes_event_id: None, + volume_unit: quantity_unit(&fixture.context)? as i32, + base_volume: Some(parse_decimal(&scalar_text( + if quantity_unit(&fixture.context)? == QuantityUnit::BaseAsset { + &row[5] + } else { + &row[6] + }, + )?)?), + quote_volume: Some(parse_decimal(&scalar_text(&row[7])?)?), + contract_volume: if quantity_unit(&fixture.context)? == QuantityUnit::Contract { + Some(parse_decimal(&scalar_text(&row[5])?)?) + } else { + None + }, + })); + set_payload_hash(&mut envelope)?; + Ok(envelope) +} + +fn canonicalize_dnse_trade(fixture: &TradeFixture) -> Result { + if text(&fixture.raw, "symbol")?.to_uppercase() != fixture.context.native_symbol.to_uppercase() + { + return Err("DNSE trade symbol does not match resolved instrument".into()); + } + if fixture.context.raw_capture_id.len() != 16 { + return Err("DNSE trade identity requires an exact raw capture id".into()); + } + let native_trade_id = format!("derived:{}", hex::encode(&fixture.context.raw_capture_id)); + let mut envelope = build_trade( + fixture, + TradeInput { + native_trade_id, + price: text(&fixture.raw, "price")?, + quantity: text(&fixture.raw, "quantity")?, + side: AggressorSide::Unspecified, + source_event_time_ms: fixture.context.received_at_ns / 1_000_000, + is_buyer_maker: false, + identity_kind: TradeIdentityKind::DerivedRawCapture, + }, + )?; + envelope.quality_flags.extend([ + QualityFlag::SourceTimeMissing as i32, + QualityFlag::FieldMissing as i32, + ]); + Ok(envelope) +} + fn canonicalize_dnse_bar(fixture: &TradeFixture) -> Result { if text(&fixture.raw, "symbol")?.to_uppercase() != fixture.context.native_symbol.to_uppercase() { @@ -350,12 +656,25 @@ fn canonicalize_dnse_bar(fixture: &TradeFixture) -> Result Result, String> { +fn deribit_levels( + raw: &Value, + field: &str, + side: BookSide, + quantity_unit: QuantityUnit, +) -> Result, String> { let rows = raw .get(field) .and_then(Value::as_array) @@ -379,6 +698,7 @@ fn deribit_levels(raw: &Value, field: &str, side: BookSide) -> Result Result Result Result { + identity_kind: TradeIdentityKind, +} + +fn build_trade(fixture: &TradeFixture, trade: TradeInput) -> Result { let context = &fixture.context; validate_shadow_context(context)?; let raw_bytes = canonical_json(&fixture.raw)?; @@ -435,7 +758,7 @@ fn build_trade( context.instrument_uid.as_bytes(), b"trade", context.source_id.as_bytes(), - native_trade_id.as_bytes(), + trade.native_trade_id.as_bytes(), ], 16, )?; @@ -453,13 +776,13 @@ fn build_trade( native_symbol: context.native_symbol.clone(), provider: context.provider.clone(), source_id: context.source_id.clone(), - source_role: SourceRole::Primary as i32, + source_role: source_role(&context.source_role)? as i32, lease_epoch: context.lease_epoch, - source_event_time_ns: source_event_time_ms * 1_000_000, + source_event_time_ns: trade.source_event_time_ms * 1_000_000, received_at_ns: context.received_at_ns, normalized_at_ns: context.normalized_at_ns, published_at_ns: context.published_at_ns, - source_sequence: native_trade_id.clone(), + source_sequence: trade.native_trade_id.clone(), partition_sequence: context.partition_sequence, normalizer_version: context.normalizer_version.clone(), adapter_version: context.adapter_version.clone(), @@ -478,12 +801,14 @@ fn build_trade( canonical_payload_hash: vec![], raw_capture_id: context.raw_capture_id.clone(), payload: Some(event_envelope::Payload::Trade(Trade { - native_trade_id, - price: Some(parse_decimal(&price)?), - quantity: Some(parse_decimal(&quantity)?), - aggressor_side: side as i32, + native_trade_id: trade.native_trade_id, + price: Some(parse_decimal(&trade.price)?), + quantity: Some(parse_decimal(&trade.quantity)?), + aggressor_side: trade.side as i32, is_block_trade: false, - is_buyer_maker, + is_buyer_maker: trade.is_buyer_maker, + quantity_unit: quantity_unit(context)? as i32, + identity_kind: trade.identity_kind as i32, })), }; set_payload_hash(&mut envelope)?; @@ -502,9 +827,24 @@ mod tests { fn provider_fixtures_match_python_golden_bytes() { for (fixture_name, golden_name) in [ ("binance_usdm_trade.json", "binance-usdm-trade.bin"), - ("okx_trade.json", "okx-swap-trade.bin"), ("binance_usdm_bbo.json", "binance-usdm-bbo.bin"), ("binance_usdm_bar.json", "binance-usdm-bar.bin"), + ("binance_usdm_rest_bar.json", "binance-usdm-rest-bar.bin"), + ("binance_spot_trade.json", "binance-spot-trade.bin"), + ("binance_spot_bbo.json", "binance-spot-bbo.bin"), + ("binance_spot_bar.json", "binance-spot-bar.bin"), + ("binance_spot_rest_bar.json", "binance-spot-rest-bar.bin"), + ("okx_trade.json", "okx-swap-trade.bin"), + ("okx_bbo.json", "okx-swap-bbo.bin"), + ("okx_bar.json", "okx-swap-bar.bin"), + ("okx_spot_trade.json", "okx-spot-trade.bin"), + ("okx_spot_bbo.json", "okx-spot-bbo.bin"), + ("okx_spot_bar.json", "okx-spot-bar.bin"), + ("dnse_derivative_trade.json", "dnse-derivative-trade.bin"), + ("dnse_derivative_bar.json", "dnse-derivative-bar.bin"), + ("dnse_equity_trade.json", "dnse-equity-trade.bin"), + ("dnse_equity_bar.json", "dnse-equity-bar.bin"), + ("vnstock_equity_bar.json", "vnstock-equity-bar.bin"), ] { let fixture_path = format!( "{}/../../tests/fixtures/phase2/{fixture_name}", diff --git a/rust/qdl-core/src/lib.rs b/rust/qdl-core/src/lib.rs index 01d9c82..7f3aa0d 100644 --- a/rust/qdl-core/src/lib.rs +++ b/rust/qdl-core/src/lib.rs @@ -6,6 +6,7 @@ pub mod binance_session; pub mod canonical; pub mod decimal; pub mod event_id; +pub mod okx; pub mod okx_simulator; pub mod rate_limit; pub mod supervisor; diff --git a/rust/qdl-core/src/okx.rs b/rust/qdl-core/src/okx.rs new file mode 100644 index 0000000..4221ec4 --- /dev/null +++ b/rust/qdl-core/src/okx.rs @@ -0,0 +1,314 @@ +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum OkxService { + Public, + Business, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub struct OkxSubscription { + pub channel: String, + #[serde(rename = "instId")] + pub inst_id: String, +} + +impl OkxSubscription { + pub fn service(&self) -> Result { + match self.channel.as_str() { + "trades" | "bbo-tbt" | "books" => Ok(OkxService::Public), + "candle1m" | "trades-all" => Ok(OkxService::Business), + _ => Err(format!("unsupported stable OKX channel: {}", self.channel)), + } + } + + pub fn provider_kind(&self) -> Result<&'static str, String> { + match self.channel.as_str() { + "trades" | "trades-all" => Ok("okx_trade"), + "bbo-tbt" => Ok("okx_bbo"), + "candle1m" => Ok("okx_bar"), + "books" => Ok("okx_book"), + _ => Err(format!("unsupported stable OKX channel: {}", self.channel)), + } + } + + fn validate(&self) -> Result<(), String> { + if self.inst_id.trim().is_empty() { + return Err("OKX subscription instrument is required".into()); + } + self.service().map(|_| ()) + } +} + +#[derive(Serialize)] +struct SubscribeCommand<'a> { + id: &'a str, + op: &'static str, + args: &'a [OkxSubscription], +} + +pub fn subscription_command(id: &str, subscriptions: &[OkxSubscription]) -> Result { + if id.trim().is_empty() || id.len() > 32 || subscriptions.is_empty() { + return Err("OKX subscription id/list is invalid".into()); + } + let mut service = None; + for subscription in subscriptions { + subscription.validate()?; + let current = subscription.service()?; + if service.is_some_and(|expected| expected != current) { + return Err("OKX public and business subscriptions require separate sockets".into()); + } + service = Some(current); + } + serde_json::to_string(&SubscribeCommand { + id, + op: "subscribe", + args: subscriptions, + }) + .map_err(|error| error.to_string()) +} + +pub fn parse_subscription_ack( + payload: &Value, + expected_id: &str, + pending: &mut Vec<(String, String)>, +) -> Result { + match payload.get("event").and_then(Value::as_str) { + Some("error") => { + return Err(format!( + "OKX subscription rejected code={} message={}", + payload + .get("code") + .and_then(Value::as_str) + .unwrap_or("missing"), + payload + .get("msg") + .and_then(Value::as_str) + .unwrap_or("missing") + )) + } + Some("subscribe") => {} + Some(other) => return Err(format!("unexpected OKX control event: {other}")), + None => return Ok(false), + } + if payload.get("id").and_then(Value::as_str) != Some(expected_id) { + return Err("OKX subscription ACK id mismatch".into()); + } + let argument = payload + .get("arg") + .and_then(Value::as_object) + .ok_or_else(|| "OKX subscription ACK arg is missing".to_owned())?; + let channel = argument + .get("channel") + .and_then(Value::as_str) + .ok_or_else(|| "OKX subscription ACK channel is missing".to_owned())?; + let inst_id = argument + .get("instId") + .and_then(Value::as_str) + .ok_or_else(|| "OKX subscription ACK instrument is missing".to_owned())?; + let index = pending + .iter() + .position(|item| item.0 == channel && item.1 == inst_id) + .ok_or_else(|| "OKX subscription ACK is duplicate or undeclared".to_owned())?; + pending.remove(index); + Ok(true) +} + +#[derive(Clone, Debug)] +pub struct NativeFrame { + pub provider_kind: &'static str, + pub instrument_id: String, + pub raw: Value, +} + +pub fn expand_data_frame(payload: &Value) -> Result, String> { + if payload.get("event").is_some() { + return Ok(vec![]); + } + let argument = payload + .get("arg") + .and_then(Value::as_object) + .ok_or_else(|| "OKX data frame arg must be an object".to_owned())?; + let subscription = OkxSubscription { + channel: argument + .get("channel") + .and_then(Value::as_str) + .ok_or_else(|| "OKX data frame channel is missing".to_owned())? + .to_owned(), + inst_id: argument + .get("instId") + .and_then(Value::as_str) + .ok_or_else(|| "OKX data frame instrument is missing".to_owned())? + .to_owned(), + }; + subscription.validate()?; + let rows = payload + .get("data") + .and_then(Value::as_array) + .filter(|rows| !rows.is_empty()) + .ok_or_else(|| "OKX data frame rows are missing".to_owned())?; + match subscription.channel.as_str() { + "trades" | "trades-all" => rows + .iter() + .map(|row| { + let object = row + .as_object() + .ok_or_else(|| "OKX trade row must be an object".to_owned())?; + if object.get("instId").and_then(Value::as_str) + != Some(subscription.inst_id.as_str()) + { + return Err("OKX trade row instrument mismatch".into()); + } + Ok(NativeFrame { + provider_kind: "okx_trade", + instrument_id: subscription.inst_id.clone(), + raw: row.clone(), + }) + }) + .collect(), + "bbo-tbt" | "candle1m" | "books" => { + if rows.len() != 1 { + return Err("OKX BBO/BAR/BOOK frame requires one data row".into()); + } + Ok(vec![NativeFrame { + provider_kind: subscription.provider_kind()?, + instrument_id: subscription.inst_id, + raw: payload.clone(), + }]) + } + _ => Err("unsupported OKX stable frame".into()), + } +} + +#[derive(Clone, Debug, Default)] +pub struct ControlRequestBudget { + window_started_ns: i64, + used: u16, +} + +impl ControlRequestBudget { + const WINDOW_NS: i64 = 3_600_000_000_000; + const MAX_REQUESTS: u16 = 480; + + pub fn permit(&mut self, now_ns: i64) -> Result<(), String> { + if now_ns <= 0 { + return Err("OKX control request clock is invalid".into()); + } + if self.window_started_ns == 0 + || now_ns.saturating_sub(self.window_started_ns) >= Self::WINDOW_NS + { + self.window_started_ns = now_ns; + self.used = 0; + } + if self.used >= Self::MAX_REQUESTS { + return Err("OKX control request budget exhausted".into()); + } + self.used += 1; + Ok(()) + } + + pub fn used(&self) -> u16 { + self.used + } +} + +#[cfg(test)] +mod tests { + use super::{ + expand_data_frame, parse_subscription_ack, subscription_command, ControlRequestBudget, + OkxService, OkxSubscription, + }; + use serde_json::json; + + fn subscription(channel: &str) -> OkxSubscription { + OkxSubscription { + channel: channel.into(), + inst_id: "BTC-USDT-SWAP".into(), + } + } + + #[test] + fn public_and_business_commands_are_separate_and_ack_correlated() { + let command = + subscription_command("7", &[subscription("trades"), subscription("bbo-tbt")]).unwrap(); + assert!(command.contains(r#""op":"subscribe""#)); + assert!( + subscription_command("8", &[subscription("trades"), subscription("candle1m")]).is_err() + ); + assert_eq!( + subscription("trades").service().unwrap(), + OkxService::Public + ); + assert_eq!( + subscription("candle1m").service().unwrap(), + OkxService::Business + ); + + let mut pending = vec![ + ("trades".into(), "BTC-USDT-SWAP".into()), + ("bbo-tbt".into(), "BTC-USDT-SWAP".into()), + ]; + assert!(parse_subscription_ack( + &json!({"id":"7","event":"subscribe","arg":{ + "channel":"trades","instId":"BTC-USDT-SWAP" + }}), + "7", + &mut pending + ) + .unwrap()); + assert_eq!(pending.len(), 1); + assert!(parse_subscription_ack( + &json!({"id":"8","event":"subscribe","arg":{ + "channel":"bbo-tbt","instId":"BTC-USDT-SWAP" + }}), + "7", + &mut pending + ) + .is_err()); + } + + #[test] + fn trade_rows_expand_atomically_but_bbo_and_bar_keep_native_frame() { + let trades = expand_data_frame(&json!({ + "arg":{"channel":"trades","instId":"BTC-USDT-SWAP"}, + "data":[ + {"instId":"BTC-USDT-SWAP","tradeId":"1"}, + {"instId":"BTC-USDT-SWAP","tradeId":"2"} + ] + })) + .unwrap(); + assert_eq!(trades.len(), 2); + assert!(trades.iter().all(|item| item.provider_kind == "okx_trade")); + + for (channel, kind, row) in [ + ("bbo-tbt", "okx_bbo", json!({"seqId":1,"bids":[],"asks":[]})), + ( + "candle1m", + "okx_bar", + json!(["1", "1", "1", "1", "1", "1", "1", "1", "0"]), + ), + ] { + let items = expand_data_frame(&json!({ + "arg":{"channel":channel,"instId":"BTC-USDT-SWAP"}, + "data":[row] + })) + .unwrap(); + assert_eq!(items.len(), 1); + assert_eq!(items[0].provider_kind, kind); + assert_eq!(items[0].raw["arg"]["channel"], channel); + } + } + + #[test] + fn control_budget_is_bounded_and_resets_after_one_hour() { + let mut budget = ControlRequestBudget::default(); + for _ in 0..480 { + budget.permit(1).unwrap(); + } + assert_eq!(budget.used(), 480); + assert!(budget.permit(2).is_err()); + budget.permit(3_600_000_000_001).unwrap(); + assert_eq!(budget.used(), 1); + } +} diff --git a/rust/qdl-kafka/Cargo.toml b/rust/qdl-kafka/Cargo.toml index fcb50b7..5bc21e5 100644 --- a/rust/qdl-kafka/Cargo.toml +++ b/rust/qdl-kafka/Cargo.toml @@ -11,13 +11,18 @@ hex.workspace = true indexmap.workspace = true jobserver.workspace = true proc-macro-crate.workspace = true +prost.workspace = true +qdl-contracts.workspace = true qdl-core.workspace = true +qdl-realtime-core = { path = "../qdl-realtime-core", version = "=0.1.0" } qdl-venue-core = { path = "../qdl-venue-core", version = "=0.1.0" } rdkafka.workspace = true serde.workspace = true +rustls.workspace = true serde_json.workspace = true sha2.workspace = true tokio.workspace = true +tokio-tungstenite.workspace = true [[bin]] name = "qdl-kafka-smoke" @@ -26,3 +31,11 @@ path = "src/bin/qdl-kafka-smoke.rs" [[bin]] name = "qdl-authority-rehearsal" path = "src/bin/qdl-authority-rehearsal.rs" + +[[bin]] +name = "qdl-phase91-canary-rehearsal" +path = "src/bin/qdl-phase91-canary-rehearsal.rs" + +[[bin]] +name = "qdl-phase92-primary-rehearsal" +path = "src/bin/qdl-phase92-primary-rehearsal.rs" diff --git a/rust/qdl-kafka/src/bin/qdl-native-raw-ingestor.rs b/rust/qdl-kafka/src/bin/qdl-native-raw-ingestor.rs new file mode 100644 index 0000000..ec70a2e --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-native-raw-ingestor.rs @@ -0,0 +1,1368 @@ +#![forbid(unsafe_code)] + +use std::collections::{BTreeMap, HashMap}; +use std::env; +use std::fs::{self, File, OpenOptions}; +use std::future::Future; +use std::io::{ErrorKind, Write}; +use std::path::{Component, Path}; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::Arc; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use futures_util::stream::FuturesUnordered; +use futures_util::{SinkExt, StreamExt}; +use prost::Message as ProstMessage; +use qdl_contracts::qdl::provider::v1::{ + CaptureBoundary, RawProviderEnvelope, TransportCompression, TransportProtocol, +}; +use qdl_core::backoff::BackoffPolicy; +use qdl_core::binance::decode_combined; +use qdl_core::okx::{ + parse_subscription_ack, subscription_command, ControlRequestBudget, OkxService, OkxSubscription, +}; +use qdl_core::transport::{DurableRecord, RetryClass}; +use qdl_kafka::{ + FencedKafkaSink, KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError, PendingKafkaAppend, +}; +use qdl_venue_core::authority::{AuthorityMode, AuthorityRecord, PublicationContext, SinkTarget}; +use qdl_venue_core::backpressure::DeliveryClass; +use serde::Deserialize; +use serde_json::{json, Value}; +use sha2::{Digest, Sha256}; +use tokio_tungstenite::connect_async; +use tokio_tungstenite::tungstenite::Message; + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum ProviderRuntime { + Binance, + Okx, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum RawFeed { + Trade, + Quote, + Bar, +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct RawBinding { + provider: String, + venue: String, + market: String, + product_type: String, + native_symbol: String, + native_channel: String, + subscription_id: String, + adapter_version: String, + instrument_catalog_revision: u64, + feed: RawFeed, + delivery_class: DeliveryClass, +} + +impl RawBinding { + fn key(&self) -> String { + format!("{}|{}", self.native_channel, self.native_symbol) + } + + fn validate(&self, runtime: ProviderRuntime) -> Result<(), String> { + for (name, value) in [ + ("provider", self.provider.as_str()), + ("venue", self.venue.as_str()), + ("market", self.market.as_str()), + ("product_type", self.product_type.as_str()), + ("native_symbol", self.native_symbol.as_str()), + ("native_channel", self.native_channel.as_str()), + ("subscription_id", self.subscription_id.as_str()), + ("adapter_version", self.adapter_version.as_str()), + ] { + if value.trim().is_empty() { + return Err(format!("raw binding {name} must not be empty")); + } + } + if self.instrument_catalog_revision == 0 { + return Err("instrument catalog revision must be positive".into()); + } + if !matches!( + (self.feed, self.delivery_class), + (RawFeed::Quote, DeliveryClass::LatestState) + | (RawFeed::Trade | RawFeed::Bar, DeliveryClass::Lossless) + ) { + return Err("raw binding feed/delivery class is invalid".into()); + } + match runtime { + ProviderRuntime::Binance + if self.venue != "BINANCE" || !matches!(self.market.as_str(), "USDM" | "SPOT") => + { + Err("Binance raw binding identity is invalid".into()) + } + ProviderRuntime::Okx + if self.venue != "OKX" || !matches!(self.market.as_str(), "SWAP" | "SPOT") => + { + Err("OKX raw binding identity is invalid".into()) + } + _ => Ok(()), + } + } +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct IngestorConfig { + runtime: ProviderRuntime, + websocket_url: String, + business_websocket_url: Option, + raw_stream: String, + shard_id: String, + lease_epoch: u64, + partition_plan_epoch: u64, + config_revision: u64, + heartbeat_seconds: u64, + max_events: u64, + max_runtime_seconds: u64, + metrics_every_events: u64, + generation_state_path: String, + max_inflight_publishes: usize, + latest_state_flush_ms: u64, + authority: AuthorityRecord, + bindings: Vec, +} + +impl IngestorConfig { + fn validate(&self) -> Result<(), String> { + self.authority.validate()?; + if self.authority.mode != AuthorityMode::RustShadow + || self.websocket_url.trim().is_empty() + || !self.websocket_url.starts_with("wss://") + || self.raw_stream.trim().is_empty() + || self.shard_id.trim().is_empty() + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || self.config_revision == 0 + || self.heartbeat_seconds == 0 + || self.heartbeat_seconds >= 30 + || self.metrics_every_events == 0 + || self.max_inflight_publishes == 0 + || self.max_inflight_publishes > 4_096 + || self.latest_state_flush_ms == 0 + || self.latest_state_flush_ms > 1_000 + || self.bindings.is_empty() + { + return Err("native raw ingestor config is invalid or not RUST_SHADOW".into()); + } + let generation_path = Path::new(&self.generation_state_path); + if !generation_path.is_absolute() + || generation_path + .components() + .any(|component| matches!(component, Component::ParentDir)) + { + return Err("generation state path must be absolute without parent traversal".into()); + } + if self.runtime == ProviderRuntime::Okx { + let business = self + .business_websocket_url + .as_deref() + .ok_or("OKX business WebSocket URL is required")?; + if !business.starts_with("wss://") { + return Err("OKX business WebSocket URL must use wss".into()); + } + } + let mut keys = std::collections::HashSet::new(); + for binding in &self.bindings { + binding.validate(self.runtime)?; + if !keys.insert(binding.key()) { + return Err("duplicate native raw binding".into()); + } + } + Ok(()) + } +} + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn kafka_config(identity: &str) -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: format!("{}-{identity}", required("QDL_KAFKA_CLIENT_ID")?), + group_id: required("QDL_KAFKA_GROUP_ID")?, + request_timeout: Duration::from_secs(30), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/client.crt"), + key_location: format!("{cert_root}/client.key"), + key_password: None, + }, + }) +} + +fn now_ns() -> Result> { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn next_connection_generation( + state_path: &Path, +) -> Result> { + let parent = state_path + .parent() + .ok_or("generation state path has no parent")?; + fs::create_dir_all(parent)?; + let previous = match fs::read_to_string(state_path) { + Ok(value) => value + .trim() + .parse::() + .map_err(|_| "generation state is corrupt")?, + Err(error) if error.kind() == ErrorKind::NotFound => 0, + Err(error) => return Err(error.into()), + }; + let generation = previous + .checked_add(1) + .ok_or("connection generation overflow")?; + let temporary = state_path.with_extension(format!("tmp-{}-{}", std::process::id(), now_ns()?)); + let mut file = OpenOptions::new() + .create_new(true) + .write(true) + .open(&temporary)?; + writeln!(file, "{generation}")?; + file.sync_all()?; + fs::rename(&temporary, state_path)?; + File::open(parent)?.sync_all()?; + Ok(generation) +} + +fn capture_id(session: &str, generation: u64, received_at_ns: i64, frame: &[u8]) -> Vec { + let mut digest = Sha256::new(); + digest.update(session.as_bytes()); + digest.update(generation.to_be_bytes()); + digest.update(received_at_ns.to_be_bytes()); + digest.update(frame); + digest.finalize()[..16].to_vec() +} + +struct RawPublisher { + sink: FencedKafkaSink, + authority: AuthorityRecord, + raw_stream: String, + shard_id: String, + lease_epoch: u64, + partition_plan_epoch: u64, + config_revision: u64, +} + +impl RawPublisher { + fn new( + config: &IngestorConfig, + identity: &str, + ) -> Result> { + let sink = FencedKafkaSink::new(&kafka_config(identity)?)?; + sink.apply_authority(config.authority.clone())?; + Ok(Self { + sink, + authority: config.authority.clone(), + raw_stream: config.raw_stream.clone(), + shard_id: config.shard_id.clone(), + lease_epoch: config.lease_epoch, + partition_plan_epoch: config.partition_plan_epoch, + config_revision: config.config_revision, + }) + } + + fn record( + &self, + binding: &RawBinding, + session_id: &str, + generation: u64, + raw_frame: &[u8], + received_at_ns: i64, + ) -> DurableRecord { + let capture_id = capture_id(session_id, generation, received_at_ns, raw_frame); + let raw = RawProviderEnvelope { + raw_schema_name: "qdl.provider.raw".into(), + raw_schema_major: 1, + raw_schema_minor: 0, + capture_id: capture_id.clone(), + provider: binding.provider.clone(), + venue: binding.venue.clone(), + market: binding.market.clone(), + product_type: binding.product_type.clone(), + native_symbol: binding.native_symbol.clone(), + native_channel: binding.native_channel.clone(), + subscription_id: binding.subscription_id.clone(), + source_session_id: session_id.into(), + connection_generation: generation, + lease_epoch: self.lease_epoch, + authority_revision: self.authority.revision, + partition_plan_epoch: self.partition_plan_epoch, + received_at_ns, + transport_protocol: TransportProtocol::Websocket as i32, + transport_compression: TransportCompression::None as i32, + capture_boundary: CaptureBoundary::PostDecompression as i32, + raw_frame_sha256: Sha256::digest(raw_frame).to_vec(), + raw_frame_bytes: raw_frame.to_vec(), + adapter_version: binding.adapter_version.clone(), + config_revision: self.config_revision, + instrument_catalog_revision: binding.instrument_catalog_revision, + correlation_id: hex::encode(&capture_id), + test_provenance: false, + }; + DurableRecord { + stream: self.raw_stream.clone(), + partition_key: format!( + "{}/{}/{}/{}", + binding.venue, binding.market, binding.native_symbol, binding.native_channel + ), + event_id: capture_id, + payload: raw.encode_to_vec(), + accepted_at_ns: received_at_ns, + } + } + + fn publication(&self) -> PublicationContext { + PublicationContext { + slice_id: self.authority.slice_id.clone(), + authority_revision: self.authority.revision, + shard_id: self.shard_id.clone(), + lease_epoch: self.lease_epoch, + target: SinkTarget::ShadowRaw, + } + } + + async fn publish( + &self, + binding: &RawBinding, + session_id: &str, + generation: u64, + raw_frame: &[u8], + received_at_ns: i64, + ) -> Result<(), KafkaTransportError> { + let record = self.record(binding, session_id, generation, raw_frame, received_at_ns); + self.sink.append(&record, &self.publication()).await?; + Ok(()) + } + + async fn enqueue_with_retry( + &self, + binding: &RawBinding, + session_id: &str, + generation: u64, + raw_frame: &[u8], + received_at_ns: i64, + stopped: &AtomicBool, + ) -> Result { + let record = self.record(binding, session_id, generation, raw_frame, received_at_ns); + let publication = self.publication(); + let backoff = BackoffPolicy { + initial_ms: 10, + maximum_ms: 1_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate() + .map_err(KafkaTransportError::Configuration)?; + let mut failures = 0_u32; + loop { + match self.sink.enqueue(&record, &publication) { + Ok(delivery) => return Ok(delivery), + Err(error) + if error.retry_class() != RetryClass::NonRetryable + && !stopped.load(Ordering::Acquire) => + { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_raw_enqueue_retry", + "runtime": binding.venue.as_str(), + "attempt": failures, + "retry_class": format!("{:?}", error.retry_class()).to_ascii_uppercase(), + "error": error.to_string(), + })) + .unwrap_or_else(|_| "{\"event\":\"qdl_native_raw_enqueue_retry\"}".into()) + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + Err(error) => return Err(error), + } + } + } + + async fn publish_with_retry( + &self, + binding: &RawBinding, + session_id: &str, + generation: u64, + raw_frame: &[u8], + received_at_ns: i64, + stopped: &AtomicBool, + ) -> Result<(), KafkaTransportError> { + let backoff = BackoffPolicy { + initial_ms: 100, + maximum_ms: 30_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate() + .map_err(KafkaTransportError::Configuration)?; + let mut failures = 0_u32; + loop { + match self + .publish(binding, session_id, generation, raw_frame, received_at_ns) + .await + { + Ok(()) => return Ok(()), + Err(error) + if error.retry_class() != RetryClass::NonRetryable + && !stopped.load(Ordering::Acquire) => + { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_raw_publish_retry", + "runtime": binding.venue.as_str(), + "attempt": failures, + "retry_class": format!("{:?}", error.retry_class()).to_ascii_uppercase(), + "error": error.to_string(), + })) + .unwrap_or_else(|_| "{\"event\":\"qdl_native_raw_publish_retry\"}".into()) + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + Err(error) => return Err(error), + } + } + } +} + +type RawPublishFuture = Pin> + Send>>; + +fn raw_publish_future(delivery: PendingKafkaAppend) -> RawPublishFuture { + Box::pin(async move { delivery.wait().await.map(|_| ()) }) +} + +#[derive(Clone, Debug)] +struct PendingRawFrame { + binding: RawBinding, + session_id: String, + generation: u64, + raw_frame: Vec, + received_at_ns: i64, +} + +#[derive(Default)] +struct LatestStateBuffer { + frames: BTreeMap, +} + +impl LatestStateBuffer { + fn push(&mut self, frame: PendingRawFrame) -> bool { + debug_assert_eq!(frame.binding.delivery_class, DeliveryClass::LatestState); + self.frames.insert(frame.binding.key(), frame).is_some() + } + + fn is_empty(&self) -> bool { + self.frames.is_empty() + } + + fn drain(&mut self) -> Vec { + std::mem::take(&mut self.frames).into_values().collect() + } +} + +async fn enqueue_binance_frame( + frame: PendingRawFrame, + publisher: &RawPublisher, + inflight: &mut FuturesUnordered, + accepted: &AtomicU64, + max_events: u64, + max_inflight: usize, + stopped: &AtomicBool, +) -> Result { + while inflight.len() >= max_inflight { + match inflight.next().await { + Some(result) => result?, + None => { + return Err(KafkaTransportError::Configuration( + "Binance publish window became inconsistent".into(), + )) + } + } + } + if !reserve(accepted, max_events).await { + return Ok(false); + } + let delivery = publisher + .enqueue_with_retry( + &frame.binding, + &frame.session_id, + frame.generation, + &frame.raw_frame, + frame.received_at_ns, + stopped, + ) + .await; + match delivery { + Ok(delivery) => { + inflight.push(raw_publish_future(delivery)); + Ok(true) + } + Err(error) => { + if max_events > 0 { + accepted.fetch_sub(1, Ordering::AcqRel); + } + Err(error) + } + } +} + +async fn flush_binance_latest( + buffer: &mut LatestStateBuffer, + publisher: &RawPublisher, + inflight: &mut FuturesUnordered, + accepted: &AtomicU64, + max_events: u64, + max_inflight: usize, + stopped: &AtomicBool, +) -> Result { + for frame in buffer.drain() { + if !enqueue_binance_frame( + frame, + publisher, + inflight, + accepted, + max_events, + max_inflight, + stopped, + ) + .await? + { + return Ok(false); + } + } + Ok(true) +} + +async fn publish_serial_frame( + frame: PendingRawFrame, + publisher: &RawPublisher, + accepted: &AtomicU64, + max_events: u64, + stopped: &AtomicBool, +) -> Result { + if !reserve(accepted, max_events).await { + return Ok(false); + } + if let Err(error) = publisher + .publish_with_retry( + &frame.binding, + &frame.session_id, + frame.generation, + &frame.raw_frame, + frame.received_at_ns, + stopped, + ) + .await + { + if max_events > 0 { + accepted.fetch_sub(1, Ordering::AcqRel); + } + return Err(error); + } + Ok(true) +} + +async fn flush_latest_serial( + buffer: &mut LatestStateBuffer, + publisher: &RawPublisher, + accepted: &AtomicU64, + max_events: u64, + stopped: &AtomicBool, +) -> Result { + for frame in buffer.drain() { + if !publish_serial_frame(frame, publisher, accepted, max_events, stopped).await? { + return Ok(false); + } + } + Ok(true) +} + +fn deadline(seconds: u64) -> Option { + (seconds > 0).then(|| tokio::time::Instant::now() + Duration::from_secs(seconds)) +} + +fn should_stop( + stopped: &AtomicBool, + accepted: &AtomicU64, + max_events: u64, + expires: Option, +) -> bool { + stopped.load(Ordering::Acquire) + || (max_events > 0 && accepted.load(Ordering::Acquire) >= max_events) + || expires.is_some_and(|value| tokio::time::Instant::now() >= value) +} + +async fn reserve(accepted: &AtomicU64, max_events: u64) -> bool { + if max_events == 0 { + accepted.fetch_add(1, Ordering::AcqRel); + return true; + } + loop { + let current = accepted.load(Ordering::Acquire); + if current >= max_events { + return false; + } + if accepted + .compare_exchange(current, current + 1, Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + return true; + } + tokio::task::yield_now().await; + } +} + +async fn run_binance( + config: Arc, + accepted: Arc, + coalesced_latest: Arc, + stopped: Arc, +) -> Result<(), Box> { + let bindings: HashMap = config + .bindings + .iter() + .cloned() + .map(|binding| (binding.native_channel.clone(), binding)) + .collect(); + let streams = bindings.keys().cloned().collect::>().join("/"); + let url = format!( + "{}?streams={streams}", + config.websocket_url.trim_end_matches('?') + ); + let publisher = RawPublisher::new(&config, "binance")?; + let backoff = BackoffPolicy { + initial_ms: 250, + maximum_ms: 30_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate()?; + let generation_path = format!("{}.binance", config.generation_state_path); + let expires = deadline(config.max_runtime_seconds); + let mut failures = 0_u32; + while !should_stop(&stopped, &accepted, config.max_events, expires) { + let generation = next_connection_generation(Path::new(&generation_path))?; + match connect_async(&url).await { + Ok((mut socket, _)) => { + let session_id = format!( + "binance-{}-{generation}-{}", + config.market_name(), + now_ns()? + ); + let mut inflight = FuturesUnordered::::new(); + let mut latest = LatestStateBuffer::default(); + let mut latest_tick = + tokio::time::interval(Duration::from_millis(config.latest_state_flush_ms)); + latest_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + latest_tick.tick().await; + let mut disconnected = false; + let mut publish_error = None; + while !should_stop(&stopped, &accepted, config.max_events, expires) { + if inflight.len() >= config.max_inflight_publishes { + match inflight.next().await { + Some(Ok(())) => { + failures = 0; + continue; + } + Some(Err(error)) => { + publish_error = Some(error); + break; + } + None => return Err("Binance publish window became inconsistent".into()), + } + } + let message = tokio::select! { + biased; + _ = latest_tick.tick(), if !latest.is_empty() => { + if !flush_binance_latest( + &mut latest, + &publisher, + &mut inflight, + &accepted, + config.max_events, + config.max_inflight_publishes, + &stopped, + ).await? { + break; + } + failures = 0; + continue; + } + message = socket.next() => message, + completed = inflight.next(), if !inflight.is_empty() => { + match completed { + Some(Ok(())) => { + failures = 0; + continue; + } + Some(Err(error)) => { + publish_error = Some(error); + break; + } + None => return Err("Binance publish window became inconsistent".into()), + } + } + }; + let message = match message { + Some(Ok(message)) => message, + Some(Err(error)) => { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_session_disconnected", + "runtime": "BINANCE", + "attempt": failures, + "generation": generation, + "error": error.to_string(), + }))? + ); + disconnected = true; + break; + } + None => { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_session_disconnected", + "runtime": "BINANCE", + "attempt": failures, + "generation": generation, + "error": "provider closed the WebSocket", + }))? + ); + disconnected = true; + break; + } + }; + if let Message::Ping(payload) = message { + socket.send(Message::Pong(payload)).await?; + continue; + } + if !message.is_text() { + continue; + } + let raw_text = message.into_text()?.to_string(); + let frame = decode_combined(raw_text.clone())?; + let binding = bindings + .get(&frame.stream) + .ok_or("Binance frame has no approved binding")? + .clone(); + let frame = PendingRawFrame { + binding, + session_id: session_id.clone(), + generation, + raw_frame: raw_text.into_bytes(), + received_at_ns: now_ns()?, + }; + if frame.binding.delivery_class == DeliveryClass::LatestState { + if latest.push(frame) { + coalesced_latest.fetch_add(1, Ordering::AcqRel); + } + continue; + } + if !enqueue_binance_frame( + frame, + &publisher, + &mut inflight, + &accepted, + config.max_events, + config.max_inflight_publishes, + &stopped, + ) + .await? + { + break; + } + } + if publish_error.is_none() { + if let Err(error) = flush_binance_latest( + &mut latest, + &publisher, + &mut inflight, + &accepted, + config.max_events, + config.max_inflight_publishes, + &stopped, + ) + .await + { + publish_error = Some(error); + } + } + while let Some(result) = inflight.next().await { + match result { + Ok(()) => failures = 0, + Err(error) if publish_error.is_none() => publish_error = Some(error), + Err(_) => {} + } + } + if let Some(error) = publish_error { + return Err(error.into()); + } + if disconnected && !should_stop(&stopped, &accepted, config.max_events, expires) { + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + } + Err(error) => { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_connect_failed", + "runtime": "BINANCE", + "attempt": failures, + "generation": generation, + "error": error.to_string(), + }))? + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + } + } + Ok(()) +} + +impl IngestorConfig { + fn market_name(&self) -> &str { + self.bindings + .first() + .map(|binding| binding.market.as_str()) + .unwrap_or("unknown") + } +} + +async fn run_okx_service( + service: OkxService, + url: String, + bindings: Vec, + config: Arc, + accepted: Arc, + coalesced_latest: Arc, + stopped: Arc, +) -> Result<(), Box> { + if bindings.is_empty() { + return Ok(()); + } + let publisher = RawPublisher::new( + &config, + match service { + OkxService::Public => "okx-public", + OkxService::Business => "okx-business", + }, + )?; + let subscriptions: Vec = bindings + .iter() + .map(|binding| OkxSubscription { + channel: binding.native_channel.clone(), + inst_id: binding.native_symbol.clone(), + }) + .collect(); + let binding_map: HashMap = bindings + .into_iter() + .map(|binding| (binding.key(), binding)) + .collect(); + let backoff = BackoffPolicy { + initial_ms: 250, + maximum_ms: 30_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate()?; + let expires = deadline(config.max_runtime_seconds); + let service_name = match service { + OkxService::Public => "public", + OkxService::Business => "business", + }; + let generation_path = format!("{}.okx-{service_name}", config.generation_state_path); + let mut failures = 0_u32; + let mut budget = ControlRequestBudget::default(); + while !should_stop(&stopped, &accepted, config.max_events, expires) { + let generation = next_connection_generation(Path::new(&generation_path))?; + match connect_async(&url).await { + Ok((socket, _)) => { + let (mut writer, mut reader) = socket.split(); + let command_id = generation.to_string(); + budget.permit(now_ns()?)?; + writer + .send(Message::Text(subscription_command( + &command_id, + &subscriptions, + )?)) + .await?; + let mut pending = subscriptions + .iter() + .map(|item| (item.channel.clone(), item.inst_id.clone())) + .collect::>(); + while !pending.is_empty() { + let message = tokio::time::timeout(Duration::from_secs(10), reader.next()) + .await + .map_err(|_| "OKX subscription ACK timed out")? + .ok_or("OKX socket closed before subscription ACK")??; + if message.is_text() { + let payload: Value = serde_json::from_str(message.to_text()?)?; + parse_subscription_ack(&payload, &command_id, &mut pending)?; + } + } + let session_id = format!( + "okx-{}-{generation}-{}", + match service { + OkxService::Public => "public", + OkxService::Business => "business", + }, + now_ns()? + ); + let mut latest = LatestStateBuffer::default(); + let mut latest_tick = + tokio::time::interval(Duration::from_millis(config.latest_state_flush_ms)); + latest_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + latest_tick.tick().await; + while !should_stop(&stopped, &accepted, config.max_events, expires) { + let read = tokio::time::timeout( + Duration::from_secs(config.heartbeat_seconds), + reader.next(), + ); + tokio::pin!(read); + let outcome = tokio::select! { + result = &mut read => Some(result), + _ = latest_tick.tick(), if !latest.is_empty() => None, + }; + if outcome.is_none() { + if !flush_latest_serial( + &mut latest, + &publisher, + &accepted, + config.max_events, + &stopped, + ) + .await? + { + break; + } + failures = 0; + continue; + } + let message = match outcome.expect("OKX read outcome is present") { + Ok(Some(Ok(message))) => message, + Ok(Some(Err(_))) | Ok(None) => break, + Err(_) => { + writer.send(Message::Text("ping".into())).await?; + match tokio::time::timeout( + Duration::from_secs(config.heartbeat_seconds), + reader.next(), + ) + .await + { + Ok(Some(Ok(message))) + if message.is_text() && message.to_text()? == "pong" => + { + continue + } + _ => break, + } + } + }; + if !message.is_text() { + continue; + } + let raw_text = message.to_text()?.to_owned(); + let payload: Value = serde_json::from_str(&raw_text)?; + if payload.get("event").and_then(Value::as_str) == Some("notice") { + break; + } + if payload.get("event").is_some() { + return Err("unexpected OKX control event after subscription".into()); + } + let argument = payload + .get("arg") + .and_then(Value::as_object) + .ok_or("OKX data arg is missing")?; + let channel = argument + .get("channel") + .and_then(Value::as_str) + .ok_or("OKX data channel is missing")?; + let instrument = argument + .get("instId") + .and_then(Value::as_str) + .ok_or("OKX data instrument is missing")?; + let binding = binding_map + .get(&format!("{channel}|{instrument}")) + .ok_or("OKX frame has no approved binding")?; + let frame = PendingRawFrame { + binding: binding.clone(), + session_id: session_id.clone(), + generation, + raw_frame: raw_text.into_bytes(), + received_at_ns: now_ns()?, + }; + if frame.binding.delivery_class == DeliveryClass::LatestState { + if latest.push(frame) { + coalesced_latest.fetch_add(1, Ordering::AcqRel); + } + continue; + } + if !publish_serial_frame( + frame, + &publisher, + &accepted, + config.max_events, + &stopped, + ) + .await? + { + stopped.store(true, Ordering::Release); + break; + } + failures = 0; + } + flush_latest_serial( + &mut latest, + &publisher, + &accepted, + config.max_events, + &stopped, + ) + .await?; + if !should_stop(&stopped, &accepted, config.max_events, expires) { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_session_disconnected", + "runtime": "OKX", + "service": format!("{:?}", service).to_ascii_uppercase(), + "attempt": failures, + "generation": generation, + }))? + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + } + Err(error) => { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_connect_failed", + "runtime": "OKX", + "service": format!("{:?}", service).to_ascii_uppercase(), + "attempt": failures, + "error": error.to_string(), + }))? + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + } + } + Ok(()) +} + +async fn run_okx( + config: Arc, + accepted: Arc, + coalesced_latest: Arc, + stopped: Arc, +) -> Result<(), Box> { + let public = config + .bindings + .iter() + .filter(|binding| { + OkxSubscription { + channel: binding.native_channel.clone(), + inst_id: binding.native_symbol.clone(), + } + .service() + == Ok(OkxService::Public) + }) + .cloned() + .collect(); + let business = config + .bindings + .iter() + .filter(|binding| { + OkxSubscription { + channel: binding.native_channel.clone(), + inst_id: binding.native_symbol.clone(), + } + .service() + == Ok(OkxService::Business) + }) + .cloned() + .collect(); + tokio::try_join!( + run_okx_service( + OkxService::Public, + config.websocket_url.clone(), + public, + config.clone(), + accepted.clone(), + coalesced_latest.clone(), + stopped.clone(), + ), + run_okx_service( + OkxService::Business, + config + .business_websocket_url + .clone() + .ok_or("OKX business WebSocket URL is required")?, + business, + config, + accepted, + coalesced_latest, + stopped, + ), + )?; + Ok(()) +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + rustls::crypto::ring::default_provider() + .install_default() + .map_err(|_| "failed to install rustls ring crypto provider")?; + let config_path = env::args() + .nth(1) + .ok_or("usage: qdl-native-raw-ingestor CONFIG.json")?; + let config: IngestorConfig = serde_json::from_slice(&tokio::fs::read(config_path).await?)?; + config.validate()?; + let config = Arc::new(config); + let accepted = Arc::new(AtomicU64::new(0)); + let coalesced_latest = Arc::new(AtomicU64::new(0)); + let stopped = Arc::new(AtomicBool::new(false)); + let stop_signal = stopped.clone(); + tokio::spawn(async move { + if tokio::signal::ctrl_c().await.is_ok() { + stop_signal.store(true, Ordering::Release); + } + }); + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_raw_ingestor_started", + "runtime": format!("{:?}", config.runtime).to_ascii_uppercase(), + "authority": "RUST_SHADOW", + "bindings": config.bindings.len(), + "latest_state_flush_ms": config.latest_state_flush_ms, + "production_public_writes": 0, + "production_legacy_writes": 0, + }))? + ); + let supervisor_backoff = BackoffPolicy { + initial_ms: 500, + maximum_ms: 30_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate()?; + let supervisor_deadline = deadline(config.max_runtime_seconds); + let mut supervisor_failures = 0_u32; + while !should_stop(&stopped, &accepted, config.max_events, supervisor_deadline) { + let accepted_before = accepted.load(Ordering::Acquire); + let result = match config.runtime { + ProviderRuntime::Binance => { + run_binance( + config.clone(), + accepted.clone(), + coalesced_latest.clone(), + stopped.clone(), + ) + .await + } + ProviderRuntime::Okx => { + run_okx( + config.clone(), + accepted.clone(), + coalesced_latest.clone(), + stopped.clone(), + ) + .await + } + }; + match result { + Ok(()) => break, + Err(error) => { + if error + .downcast_ref::() + .is_some_and(|value| value.retry_class() == RetryClass::NonRetryable) + { + return Err(error); + } + if should_stop(&stopped, &accepted, config.max_events, supervisor_deadline) { + break; + } + if accepted.load(Ordering::Acquire) > accepted_before { + supervisor_failures = 0; + } + supervisor_failures = supervisor_failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_runtime_retry", + "runtime": format!("{:?}", config.runtime).to_ascii_uppercase(), + "attempt": supervisor_failures, + "error": error.to_string(), + }))? + ); + tokio::time::sleep(Duration::from_millis( + supervisor_backoff + .delay_ms(supervisor_failures, supervisor_failures.min(10_000) as u16), + )) + .await; + } + } + } + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_native_raw_ingestor_stopped", + "accepted_raw_frames": accepted.load(Ordering::Acquire), + "coalesced_latest_state_frames": coalesced_latest.load(Ordering::Acquire), + }))? + ); + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::{ + next_connection_generation, DeliveryClass, LatestStateBuffer, PendingRawFrame, + ProviderRuntime, RawBinding, RawFeed, + }; + use std::fs; + use std::path::PathBuf; + + fn generation_path(label: &str) -> PathBuf { + std::env::temp_dir().join(format!( + "qdl-native-generation-{label}-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )) + } + + fn binding(feed: RawFeed, delivery_class: DeliveryClass) -> RawBinding { + RawBinding { + provider: "BINANCE_DIRECT".into(), + venue: "BINANCE".into(), + market: "USDM".into(), + product_type: "PERPETUAL".into(), + native_symbol: "BTCUSDT".into(), + native_channel: match feed { + RawFeed::Trade => "btcusdt@trade", + RawFeed::Quote => "btcusdt@bookTicker", + RawFeed::Bar => "candle1m", + } + .into(), + subscription_id: "test-source".into(), + adapter_version: "test/2.0.0".into(), + instrument_catalog_revision: 1, + feed, + delivery_class, + } + } + + #[test] + fn feed_delivery_contract_allows_only_latest_quote_and_lossless_trade_bar() { + assert!(binding(RawFeed::Quote, DeliveryClass::LatestState) + .validate(ProviderRuntime::Binance) + .is_ok()); + assert!(binding(RawFeed::Trade, DeliveryClass::Lossless) + .validate(ProviderRuntime::Binance) + .is_ok()); + assert!(binding(RawFeed::Bar, DeliveryClass::Lossless) + .validate(ProviderRuntime::Binance) + .is_ok()); + assert!(binding(RawFeed::Trade, DeliveryClass::LatestState) + .validate(ProviderRuntime::Binance) + .is_err()); + assert!(binding(RawFeed::Quote, DeliveryClass::Lossless) + .validate(ProviderRuntime::Binance) + .is_err()); + } + + #[test] + fn latest_state_buffer_keeps_last_authentic_frame_per_binding() { + let mut buffer = LatestStateBuffer::default(); + let first = PendingRawFrame { + binding: binding(RawFeed::Quote, DeliveryClass::LatestState), + session_id: "session-1".into(), + generation: 1, + raw_frame: br#"{"u":1}"#.to_vec(), + received_at_ns: 10, + }; + let mut last = first.clone(); + last.raw_frame = br#"{"u":2}"#.to_vec(); + last.received_at_ns = 20; + assert!(!buffer.push(first)); + assert!(buffer.push(last)); + let values = buffer.drain(); + assert_eq!(values.len(), 1); + assert_eq!(values[0].raw_frame, br#"{"u":2}"#); + assert_eq!(values[0].received_at_ns, 20); + assert!(buffer.is_empty()); + } + + #[test] + fn durable_generation_advances_across_reopens_and_corruption_fails_closed() { + let directory = generation_path("restart"); + let state = directory.join("binance-usdm"); + assert_eq!(next_connection_generation(&state).unwrap(), 1); + assert_eq!(next_connection_generation(&state).unwrap(), 2); + assert_eq!(fs::read_to_string(&state).unwrap(), "2\n"); + fs::write(&state, "not-a-generation\n").unwrap(); + assert!(next_connection_generation(&state).is_err()); + fs::remove_dir_all(directory).unwrap(); + } + + #[test] + fn venue_service_generation_files_are_independent() { + let directory = generation_path("services"); + let public = directory.join("okx-swap.okx-public"); + let business = directory.join("okx-swap.okx-business"); + assert_eq!(next_connection_generation(&public).unwrap(), 1); + assert_eq!(next_connection_generation(&business).unwrap(), 1); + assert_eq!(next_connection_generation(&public).unwrap(), 2); + assert_eq!(fs::read_to_string(&business).unwrap(), "1\n"); + fs::remove_dir_all(directory).unwrap(); + } +} diff --git a/rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs b/rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs new file mode 100644 index 0000000..1fe3886 --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs @@ -0,0 +1,528 @@ +#![forbid(unsafe_code)] + +use std::env; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use qdl_core::transport::DurableRecord; +use qdl_kafka::{ + KafkaDurableSink, KafkaEventSource, KafkaTlsConfig, KafkaTransportConfig, + Phase9FencedKafkaSink, Phase9SinkTopics, +}; +use qdl_venue_core::authority::{ + Phase9AuthorityRecord, Phase9AuthorityState, Phase9PublicationContext, SinkTarget, +}; +use serde_json::json; +use sha2::{Digest, Sha256}; + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn now_ns() -> Result> { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn transport_config(identity: &str, group_id: &str) -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: format!("phase91-canary-{identity}"), + group_id: group_id.to_owned(), + request_timeout: Duration::from_secs(15), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/phase8-{identity}.crt"), + key_location: format!("{cert_root}/phase8-{identity}.key"), + key_password: None, + }, + }) +} + +fn durable_record( + stream: &str, + partition_key: &str, + payload: Vec, + nonce: &str, +) -> Result> { + let mut event_id = Sha256::new(); + event_id.update(stream.as_bytes()); + event_id.update(partition_key.as_bytes()); + event_id.update(nonce.as_bytes()); + event_id.update(&payload); + Ok(DurableRecord { + stream: stream.to_owned(), + partition_key: partition_key.to_owned(), + event_id: event_id.finalize().to_vec(), + payload, + accepted_at_ns: now_ns()?, + }) +} + +async fn receive_authority( + source: &KafkaEventSource, + slice_id: &str, + revision: u64, +) -> Result> { + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let (record, _) = source.next().await?; + let authority: Phase9AuthorityRecord = serde_json::from_slice(&record.payload) + .map_err(|error| qdl_kafka::KafkaTransportError::Fencing(error.to_string()))?; + if authority.slice_id == slice_id && authority.authority_revision == revision { + source.checkpoint()?; + return Ok::(authority); + } + } + }) + .await + .map_err(|_| "timed out reading persistent Phase 9 authority record")? + .map_err(|error| -> Box { Box::new(error) }) +} + +#[allow(clippy::too_many_arguments)] +fn authority_record( + slice_id: &str, + owner_id: &str, + candidate_digest: &str, + bundle_id: &str, + revision: u64, + lease_epoch: u64, + state: Phase9AuthorityState, + start_watermark: u64, + approved_at_ns: i64, + hold_until_ns: i64, +) -> Phase9AuthorityRecord { + let canary = state == Phase9AuthorityState::RustCanary; + Phase9AuthorityRecord { + schema: "qdl.authority-record.v2".into(), + slice_id: slice_id.into(), + state, + owner_id: owner_id.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + candidate_digest: candidate_digest.into(), + prerequisite_bundle_id: canary.then(|| bundle_id.into()), + start_watermark, + approved_by: canary.then(|| "phase91-isolated-rehearsal".into()), + approved_at_ns: canary.then_some(approved_at_ns), + hold_until_ns: canary.then_some(hold_until_ns), + public_write_allowed: false, + legacy_write_allowed: false, + } +} + +#[allow(clippy::too_many_arguments)] +fn publication( + slice_id: &str, + owner_id: &str, + revision: u64, + lease_epoch: u64, + partition_plan_epoch: u64, + source_watermark: u64, + target: SinkTarget, +) -> Phase9PublicationContext { + Phase9PublicationContext { + slice_id: slice_id.into(), + owner_id: owner_id.into(), + authority_revision: revision, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch, + partition_plan_epoch, + source_watermark, + target, + } +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let authority_topic = required("QDL_AUTHORITY_TOPIC")?; + let audit_topic = required("QDL_AUDIT_TOPIC")?; + let shadow_raw_topic = required("QDL_SHADOW_RAW_TOPIC")?; + let shadow_topic = required("QDL_SHADOW_CANONICAL_TOPIC")?; + let canary_topic = required("QDL_CANARY_CANONICAL_TOPIC")?; + let public_topic = required("QDL_PUBLIC_TOPIC")?; + let legacy_topic = required("QDL_LEGACY_TOPIC")?; + let nonce = required("QDL_AUTHORITY_NONCE")?; + let candidate_digest = required("QDL_CANDIDATE_DIGEST")?; + let bundle_id = required("QDL_PREREQUISITE_BUNDLE_ID")?; + let slice_id = required("QDL_SLICE_ID")?; + let shadow_owner = required("QDL_SHADOW_OWNER_ID")?; + let canary_owner = required("QDL_CANARY_OWNER_ID")?; + let group = format!("phase8-phase91-authority-{nonce}"); + let producer_config = transport_config("producer", &group)?; + let consumer_config = transport_config("consumer", &group)?; + let authority_sink = KafkaDurableSink::new(&producer_config)?; + let audit_sink = KafkaDurableSink::new(&producer_config)?; + let authority_source = KafkaEventSource::new(&consumer_config, &[&authority_topic])?; + let fenced_sink = Phase9FencedKafkaSink::new( + &producer_config, + Phase9SinkTopics { + shadow_raw: shadow_raw_topic.clone(), + shadow_canonical: shadow_topic.clone(), + shadow_quarantine: format!("{shadow_topic}.quarantine"), + canary_canonical: canary_topic.clone(), + }, + )?; + let started_at = now_ns()?; + let approved_at = started_at - 1_000_000_000; + let hold_until = started_at + 300_000_000_000; + let transitions = [ + authority_record( + &slice_id, + &shadow_owner, + &candidate_digest, + &bundle_id, + 1, + 1, + Phase9AuthorityState::RustShadow, + 100, + approved_at, + hold_until, + ), + authority_record( + &slice_id, + &canary_owner, + &candidate_digest, + &bundle_id, + 2, + 2, + Phase9AuthorityState::RustCanary, + 100, + approved_at, + hold_until, + ), + authority_record( + &slice_id, + &canary_owner, + &candidate_digest, + &bundle_id, + 3, + 2, + Phase9AuthorityState::Blocked, + 100, + approved_at, + hold_until, + ), + authority_record( + &slice_id, + &shadow_owner, + &candidate_digest, + &bundle_id, + 4, + 3, + Phase9AuthorityState::RustShadow, + 100, + approved_at, + hold_until, + ), + ]; + let mut authority_offsets = Vec::new(); + let mut audit_offsets = Vec::new(); + let mut shadow_offsets = Vec::new(); + let mut canary_offsets = Vec::new(); + let mut checks = serde_json::Map::new(); + + for authority in transitions { + let revision = authority.authority_revision; + let state = authority.state; + let durable = durable_record( + &authority_topic, + &slice_id, + serde_json::to_vec(&authority)?, + &format!("{nonce}:authority:{revision}"), + )?; + authority_offsets.push(authority_sink.append(&durable).await?.cursor.offset); + let audit = durable_record( + &audit_topic, + &format!("{slice_id}:{revision}"), + serde_json::to_vec(&authority)?, + &format!("{nonce}:audit:{revision}"), + )?; + audit_offsets.push(audit_sink.append(&audit).await?.cursor.offset); + let persisted = receive_authority(&authority_source, &slice_id, revision).await?; + fenced_sink.apply_authority(persisted).await?; + + match state { + Phase9AuthorityState::RustShadow if revision == 1 => { + let event = durable_record( + &shadow_topic, + "btc-usdt", + serde_json::to_vec(&json!({"kind": "phase91-shadow", "revision": revision}))?, + &format!("{nonce}:shadow:{revision}"), + )?; + shadow_offsets.push( + fenced_sink + .append( + &event, + &publication( + &slice_id, + &shadow_owner, + revision, + 1, + 1, + 101, + SinkTarget::ShadowCanonical, + ), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + } + Phase9AuthorityState::RustCanary => { + for source_watermark in 102..166 { + let event = durable_record( + &canary_topic, + "btc-usdt", + serde_json::to_vec(&json!({ + "kind": "phase91-canary", + "revision": revision, + "source_watermark": source_watermark, + }))?, + &format!("{nonce}:canary:{source_watermark}"), + )?; + canary_offsets.push( + fenced_sink + .append( + &event, + &publication( + &slice_id, + &canary_owner, + revision, + 2, + 1, + source_watermark, + SinkTarget::CanaryCanonical, + ), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + } + let rejected_record = |stream: &str, label: &str| { + durable_record( + stream, + label, + b"must-not-publish".to_vec(), + &format!("{nonce}:rejected:{label}"), + ) + }; + let cases = [ + ( + "public_target_rejected", + rejected_record(&public_topic, "public")?, + publication(&slice_id, &canary_owner, 2, 2, 1, 166, SinkTarget::PublicV2), + ), + ( + "legacy_target_rejected", + rejected_record(&legacy_topic, "legacy")?, + publication(&slice_id, &canary_owner, 2, 2, 1, 166, SinkTarget::LegacyV1), + ), + ( + "topic_masquerade_rejected", + rejected_record(&public_topic, "masquerade")?, + publication( + &slice_id, + &canary_owner, + 2, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "stale_owner_rejected", + rejected_record(&canary_topic, "owner")?, + publication( + &slice_id, + &shadow_owner, + 2, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "stale_revision_rejected", + rejected_record(&canary_topic, "revision")?, + publication( + &slice_id, + &canary_owner, + 1, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "stale_lease_rejected", + rejected_record(&canary_topic, "lease")?, + publication( + &slice_id, + &canary_owner, + 2, + 1, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "wrong_plan_rejected", + rejected_record(&canary_topic, "plan")?, + publication( + &slice_id, + &canary_owner, + 2, + 2, + 2, + 166, + SinkTarget::CanaryCanonical, + ), + ), + ( + "duplicate_watermark_rejected", + rejected_record(&canary_topic, "watermark")?, + publication( + &slice_id, + &canary_owner, + 2, + 2, + 1, + 165, + SinkTarget::CanaryCanonical, + ), + ), + ]; + for (name, event, context) in cases { + checks.insert( + name.into(), + json!(fenced_sink + .append(&event, &context, now_ns()?) + .await + .is_err()), + ); + } + } + Phase9AuthorityState::Blocked => { + let event = durable_record( + &canary_topic, + "blocked", + b"must-not-publish".to_vec(), + &format!("{nonce}:blocked"), + )?; + checks.insert( + "blocked_state_rejected".into(), + json!(fenced_sink + .append( + &event, + &publication( + &slice_id, + &canary_owner, + 3, + 2, + 1, + 166, + SinkTarget::CanaryCanonical, + ), + now_ns()?, + ) + .await + .is_err()), + ); + } + Phase9AuthorityState::RustShadow => { + let event = durable_record( + &shadow_topic, + "btc-usdt", + serde_json::to_vec(&json!({"kind": "phase91-shadow", "revision": revision}))?, + &format!("{nonce}:shadow:{revision}"), + )?; + shadow_offsets.push( + fenced_sink + .append( + &event, + &publication( + &slice_id, + &shadow_owner, + revision, + 3, + 1, + 166, + SinkTarget::ShadowCanonical, + ), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + let rejected = durable_record( + &canary_topic, + "after-rollback", + b"must-not-publish".to_vec(), + &format!("{nonce}:after-rollback"), + )?; + checks.insert( + "canary_after_rollback_rejected".into(), + json!(fenced_sink + .append( + &rejected, + &publication( + &slice_id, + &shadow_owner, + 4, + 3, + 1, + 167, + SinkTarget::CanaryCanonical, + ), + now_ns()?, + ) + .await + .is_err()), + ); + } + } + } + + let all_rejected = checks.values().all(|value| value == &json!(true)); + let passed = authority_offsets.len() == 4 + && audit_offsets.len() == 4 + && shadow_offsets.len() == 2 + && canary_offsets.len() == 64 + && all_rejected; + println!( + "{}", + serde_json::to_string(&json!({ + "status": if passed { "PASS" } else { "FAIL" }, + "schema": "qdl.phase91.isolated-canary-runtime.v1", + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "transitions": ["RUST_SHADOW", "RUST_CANARY", "BLOCKED", "RUST_SHADOW"], + "authority_offsets": authority_offsets, + "audit_offsets": audit_offsets, + "shadow_offsets": shadow_offsets, + "canary_offsets": canary_offsets, + "checks": checks, + "public_writes": 0, + "legacy_writes": 0, + "final_authority": "RUST_SHADOW", + }))? + ); + if passed { + Ok(()) + } else { + Err("Phase 9.1 isolated canary rehearsal failed".into()) + } +} diff --git a/rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs b/rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs new file mode 100644 index 0000000..7a7456e --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs @@ -0,0 +1,947 @@ +#![forbid(unsafe_code)] + +use std::collections::BTreeSet; +use std::env; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use qdl_core::transport::DurableRecord; +use qdl_kafka::{ + KafkaDurableSink, KafkaEventSource, KafkaTlsConfig, KafkaTransportConfig, + Phase92FencedKafkaSink, Phase92SinkTopics, +}; +use qdl_venue_core::authority::{ + Phase92AcceptedHandoff, Phase92AuthorityRecord, Phase92AuthorityState, Phase92HandoffDirection, + Phase92PublicationContext, Phase92TerminalCheckpoint, SinkTarget, +}; +use serde::Serialize; +use serde_json::json; +use sha2::{Digest, Sha256}; + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn now_ns() -> Result> { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn transport_config(identity: &str, group_id: &str) -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: format!("phase92-primary-{identity}"), + group_id: group_id.to_owned(), + request_timeout: Duration::from_secs(15), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/phase8-{identity}.crt"), + key_location: format!("{cert_root}/phase8-{identity}.key"), + key_password: None, + }, + }) +} + +fn durable_record( + stream: &str, + key: &str, + value: &T, + nonce: &str, +) -> Result> { + let payload = serde_json::to_vec(value)?; + let mut event_id = Sha256::new(); + event_id.update(stream.as_bytes()); + event_id.update(key.as_bytes()); + event_id.update(nonce.as_bytes()); + event_id.update(&payload); + Ok(DurableRecord { + stream: stream.to_owned(), + partition_key: key.to_owned(), + event_id: event_id.finalize().to_vec(), + payload, + accepted_at_ns: now_ns()?, + }) +} + +async fn receive_authority( + source: &KafkaEventSource, + slice_id: &str, + revision: u64, +) -> Result> { + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let (record, _) = source.next().await?; + let authority: Phase92AuthorityRecord = serde_json::from_slice(&record.payload) + .map_err(|error| qdl_kafka::KafkaTransportError::Fencing(error.to_string()))?; + if authority.slice_id == slice_id && authority.authority_revision == revision { + source.checkpoint()?; + return Ok::(authority); + } + } + }) + .await + .map_err(|_| "timed out reading persistent Phase 9.2 authority record")? + .map_err(|error| -> Box { Box::new(error) }) +} + +async fn receive_projection_range( + source: &KafkaEventSource, + slice_id: &str, + owner_id: &str, + authority_revision: u64, + first_watermark: u64, + last_watermark: u64, +) -> Result> { + tokio::time::timeout(Duration::from_secs(30), async { + let mut observed = BTreeSet::new(); + loop { + let (record, _) = source.next().await?; + let payload: serde_json::Value = serde_json::from_slice(&record.payload) + .map_err(|error| qdl_kafka::KafkaTransportError::Fencing(error.to_string()))?; + let watermark = payload + .get("source_watermark") + .and_then(serde_json::Value::as_u64) + .ok_or(qdl_kafka::KafkaTransportError::MissingField( + "source_watermark", + ))?; + let terminal_identity_matches = watermark != last_watermark + || (payload.get("owner_id").and_then(serde_json::Value::as_str) == Some(owner_id) + && payload + .get("authority_revision") + .and_then(serde_json::Value::as_u64) + == Some(authority_revision)); + if payload.get("slice_id").and_then(serde_json::Value::as_str) != Some(slice_id) + || !terminal_identity_matches + || watermark < first_watermark + || watermark > last_watermark + || !observed.insert(watermark) + { + return Err(qdl_kafka::KafkaTransportError::Fencing( + "Phase 9.2 durable projection recovery diverged".into(), + )); + } + if watermark == last_watermark { + let expected: BTreeSet = (first_watermark..=last_watermark).collect(); + if observed != expected { + return Err(qdl_kafka::KafkaTransportError::Fencing( + "Phase 9.2 durable projection recovery has a gap".into(), + )); + } + source.checkpoint()?; + return Ok(last_watermark); + } + } + }) + .await + .map_err(|_| "timed out reading Phase 9.2 durable projection")? + .map_err(|error| -> Box { Box::new(error) }) +} + +#[allow(clippy::too_many_arguments)] +fn authority( + slice_id: &str, + state: Phase92AuthorityState, + owner_id: &str, + revision: u64, + lease_epoch: u64, + candidate_digest: &str, + bundle_id: Option<&str>, + start_watermark: u64, + terminal_watermark: Option, + previous_owner_id: Option<&str>, + handoff_digest: Option, + approved_at_ns: i64, + hold_until_ns: i64, +) -> Phase92AuthorityRecord { + let primary = matches!( + state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + let active = matches!( + state, + Phase92AuthorityState::RustCanary + | Phase92AuthorityState::RustPrimary + | Phase92AuthorityState::PythonPrimary + ); + Phase92AuthorityRecord { + schema: "qdl.authority-record.v3".into(), + slice_id: slice_id.into(), + state, + owner_id: owner_id.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + candidate_digest: candidate_digest.into(), + prerequisite_bundle_id: bundle_id.map(str::to_owned), + start_watermark, + terminal_watermark, + previous_owner_id: previous_owner_id.map(str::to_owned), + handoff_digest, + approved_by: active.then(|| "phase92-isolated-rehearsal".into()), + approved_at_ns: active.then_some(approved_at_ns), + hold_until_ns: active.then_some(hold_until_ns), + public_write_allowed: primary, + legacy_write_allowed: primary, + } +} + +#[allow(clippy::too_many_arguments)] +fn checkpoint( + checkpoint_id: &str, + slice_id: &str, + owner_id: &str, + revision: u64, + lease_epoch: u64, + watermark: u64, + candidate_digest: &str, + nonce: &str, +) -> Result> { + let terminal_payload_sha256 = hex::encode(Sha256::digest( + format!("{slice_id}:{owner_id}:{revision}:{watermark}:{nonce}").as_bytes(), + )); + Ok(Phase92TerminalCheckpoint { + schema: "qdl.terminal-owner-checkpoint.v1".into(), + checkpoint_id: checkpoint_id.into(), + slice_id: slice_id.into(), + owner_id: owner_id.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + source_session_id: format!("phase92-{owner_id}-{nonce}"), + connection_generation: 1, + terminal_watermark: watermark, + terminal_event_id: format!("phase92-event-{watermark}"), + terminal_payload_sha256, + candidate_digest: candidate_digest.into(), + committed_at_ns: now_ns()?, + }) +} + +#[allow(clippy::too_many_arguments)] +fn handoff( + handoff_id: &str, + direction: Phase92HandoffDirection, + checkpoint: &Phase92TerminalCheckpoint, + new_owner_id: &str, + new_state: Phase92AuthorityState, + bundle_id: &str, + approved_at_ns: i64, + expires_at_ns: i64, +) -> Result> { + let expected_state = match direction { + Phase92HandoffDirection::PythonToRust => Phase92AuthorityState::RustCanary, + Phase92HandoffDirection::RustToPython => Phase92AuthorityState::RollbackPending, + }; + let result = Phase92AcceptedHandoff { + schema: "qdl.accepted-authority-handoff.v1".into(), + handoff_id: handoff_id.into(), + direction, + checkpoint_digest: checkpoint.digest()?, + slice_id: checkpoint.slice_id.clone(), + old_owner_id: checkpoint.owner_id.clone(), + new_owner_id: new_owner_id.into(), + expected_state, + new_state, + expected_authority_revision: checkpoint.authority_revision, + new_authority_revision: checkpoint.authority_revision + 1, + expected_lease_epoch: checkpoint.lease_epoch, + new_lease_epoch: checkpoint.lease_epoch + 1, + partition_plan_epoch: 1, + terminal_watermark: checkpoint.terminal_watermark, + first_new_watermark: checkpoint.terminal_watermark + 1, + overlap_start_watermark: checkpoint.terminal_watermark.saturating_sub(10), + overlap_end_watermark: checkpoint.terminal_watermark, + old_event_count: 11, + new_event_count: 11, + semantic_mismatches: 0, + open_gaps: 0, + candidate_digest: checkpoint.candidate_digest.clone(), + prerequisite_bundle_id: bundle_id.into(), + approved_by: "phase92-isolated-rehearsal".into(), + approved_at_ns, + expires_at_ns, + }; + result.validate(checkpoint)?; + Ok(result) +} + +fn publication( + record: &Phase92AuthorityRecord, + watermark: u64, + target: SinkTarget, +) -> Phase92PublicationContext { + Phase92PublicationContext { + slice_id: record.slice_id.clone(), + owner_id: record.owner_id.clone(), + authority_revision: record.authority_revision, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch: record.lease_epoch, + partition_plan_epoch: record.partition_plan_epoch, + source_watermark: watermark, + target, + } +} + +async fn persist_authority( + authority_sink: &KafkaDurableSink, + audit_sink: &KafkaDurableSink, + authority_source: &KafkaEventSource, + authority_topic: &str, + audit_topic: &str, + nonce: &str, + record: &Phase92AuthorityRecord, +) -> Result<(u64, u64, Phase92AuthorityRecord), Box> { + let revision = record.authority_revision; + let authority_offset = authority_sink + .append(&durable_record( + authority_topic, + &record.slice_id, + record, + &format!("{nonce}:authority:{revision}"), + )?) + .await? + .cursor + .offset; + let audit_offset = audit_sink + .append(&durable_record( + audit_topic, + &format!("{}:{revision}", record.slice_id), + record, + &format!("{nonce}:audit:{revision}"), + )?) + .await? + .cursor + .offset; + let persisted = receive_authority(authority_source, &record.slice_id, revision).await?; + Ok((authority_offset, audit_offset, persisted)) +} + +async fn publish_range( + sink: &Phase92FencedKafkaSink, + topics: &Phase92SinkTopics, + record: &Phase92AuthorityRecord, + first: u64, + last: u64, + nonce: &str, +) -> Result, Box> { + let mut offsets = Vec::new(); + for watermark in first..=last { + let payload = json!({ + "schema": "qdl.phase92.isolated-projection.v1", + "slice_id": record.slice_id, + "owner_id": record.owner_id, + "authority_revision": record.authority_revision, + "lease_epoch": record.lease_epoch, + "source_watermark": watermark, + "provider_provenance": "REAL_PROVIDER_READ_ONLY_CAPTURE", + }); + for (target, topic) in [ + ( + SinkTarget::PrimaryCanonical, + topics.primary_canonical.as_str(), + ), + (SinkTarget::PublicV2, topics.public_v2.as_str()), + (SinkTarget::LegacyV1, topics.legacy_v1.as_str()), + ] { + offsets.push( + sink.append( + &durable_record( + topic, + "btc-usdt", + &payload, + &format!("{nonce}:{watermark}:{target:?}"), + )?, + &publication(record, watermark, target), + now_ns()?, + ) + .await? + .cursor + .offset, + ); + } + } + Ok(offsets) +} + +#[allow(clippy::too_many_arguments)] +async fn run_recovery_verify( + authority_topic: &str, + primary_topic: &str, + public_topic: &str, + legacy_topic: &str, + slice_id: &str, + nonce: &str, +) -> Result<(), Box> { + let expected_revision: u64 = required("QDL_RECOVERY_AUTHORITY_REVISION")?.parse()?; + let first_watermark: u64 = required("QDL_RECOVERY_FIRST_WATERMARK")?.parse()?; + let last_watermark: u64 = required("QDL_RECOVERY_LAST_WATERMARK")?.parse()?; + if first_watermark == 0 || last_watermark < first_watermark { + return Err("Phase 9.2 recovery watermark range is invalid".into()); + } + + let producer_group = format!("phase8-phase92-recovery-producer-{nonce}"); + let producer_config = transport_config("producer", &producer_group)?; + let authority_group = format!("phase8-phase92-recovery-authority-{nonce}"); + let authority_config = transport_config("consumer", &authority_group)?; + let authority_source = KafkaEventSource::new(&authority_config, &[authority_topic])?; + let latest = receive_authority(&authority_source, slice_id, expected_revision).await?; + if latest.state != Phase92AuthorityState::PythonPrimary { + return Err("Phase 9.2 recovery expected terminal Python primary authority".into()); + } + + let topics = Phase92SinkTopics { + primary_canonical: primary_topic.to_owned(), + public_v2: public_topic.to_owned(), + legacy_v1: legacy_topic.to_owned(), + }; + let fenced_sink = Phase92FencedKafkaSink::new(&producer_config, topics.clone())?; + fenced_sink.apply_authority(latest.clone()).await?; + let next_watermark = last_watermark + .checked_add(1) + .ok_or("Phase 9.2 recovery watermark overflow")?; + let pre_restore = fenced_sink + .append( + &durable_record( + primary_topic, + "pre-restore", + &json!({"must_not_publish": "pre-restore"}), + &format!("{nonce}:pre-restore"), + )?, + &publication(&latest, next_watermark, SinkTarget::PrimaryCanonical), + now_ns()?, + ) + .await; + + let mut observed = serde_json::Map::new(); + let mut target_pre_restore_rejected = true; + let mut duplicate_after_restore_rejected = true; + for (name, target, topic) in [ + ("primary", SinkTarget::PrimaryCanonical, primary_topic), + ("public", SinkTarget::PublicV2, public_topic), + ("legacy", SinkTarget::LegacyV1, legacy_topic), + ] { + let group = format!("phase8-phase92-recovery-{name}-{nonce}"); + let config = transport_config("consumer", &group)?; + let source = KafkaEventSource::new(&config, &[topic])?; + let durable_watermark = receive_projection_range( + &source, + slice_id, + &latest.owner_id, + latest.authority_revision, + first_watermark, + last_watermark, + ) + .await?; + observed.insert(name.into(), json!(durable_watermark)); + + target_pre_restore_rejected &= fenced_sink + .append( + &durable_record( + topic, + "target-pre-restore", + &json!({"must_not_publish": name}), + &format!("{nonce}:{name}:pre-restore"), + )?, + &publication(&latest, next_watermark, target), + now_ns()?, + ) + .await + .is_err(); + let restored = publication(&latest, durable_watermark, target); + fenced_sink.restore_committed_watermark(&restored).await?; + duplicate_after_restore_rejected &= fenced_sink + .append( + &durable_record( + topic, + "duplicate-after-restore", + &json!({"must_not_publish": name}), + &format!("{nonce}:{name}:duplicate"), + )?, + &restored, + now_ns()?, + ) + .await + .is_err(); + } + + let projection_offsets = publish_range( + &fenced_sink, + &topics, + &latest, + next_watermark, + next_watermark, + nonce, + ) + .await?; + let checks = json!({ + "restart_pre_restore_failed_closed": pre_restore.is_err(), + "each_target_pre_restore_failed_closed": target_pre_restore_rejected, + "durable_target_watermarks_restored": observed.values().all(|value| value == &json!(last_watermark)), + "duplicate_after_restore_rejected": duplicate_after_restore_rejected, + "resumed_at_exact_next_watermark": projection_offsets.len() == 3, + }); + let passed = checks + .as_object() + .is_some_and(|values| values.values().all(|value| value == &json!(true))); + println!( + "{}", + serde_json::to_string(&json!({ + "schema": "qdl.phase92.process-restart-recovery.v1", + "status": if passed { "PASS" } else { "FAIL" }, + "mode": "RECOVERY_VERIFY", + "production_authorized": false, + "authority_revision": latest.authority_revision, + "owner_id": latest.owner_id, + "restored_target_watermarks": observed, + "resumed_watermark": next_watermark, + "projection_offsets": projection_offsets, + "checks": checks, + "production_public_writes": 0, + "production_legacy_writes": 0, + }))? + ); + if !passed { + return Err("Phase 9.2 process restart recovery failed".into()); + } + Ok(()) +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let authority_topic = required("QDL_AUTHORITY_TOPIC")?; + let audit_topic = required("QDL_AUDIT_TOPIC")?; + let checkpoint_topic = required("QDL_CHECKPOINT_TOPIC")?; + let handoff_topic = required("QDL_HANDOFF_TOPIC")?; + let primary_topic = required("QDL_PRIMARY_CANONICAL_TOPIC")?; + let public_topic = required("QDL_ISOLATED_PUBLIC_TOPIC")?; + let legacy_topic = required("QDL_ISOLATED_LEGACY_TOPIC")?; + let production_public = required("QDL_PRODUCTION_PUBLIC_TOPIC")?; + let production_legacy = required("QDL_PRODUCTION_LEGACY_TOPIC")?; + let required_topics = [ + &authority_topic, + &audit_topic, + &checkpoint_topic, + &handoff_topic, + &primary_topic, + &public_topic, + &legacy_topic, + ]; + if required_topics + .iter() + .any(|topic| !topic.contains(".phase92.")) + || required_topics + .iter() + .any(|topic| topic.as_str() == production_public || topic.as_str() == production_legacy) + { + return Err("Phase 9.2 rehearsal topic is not isolated".into()); + } + + let nonce = required("QDL_AUTHORITY_NONCE")?; + let candidate_digest = required("QDL_CANDIDATE_DIGEST")?; + let bundle_id = required("QDL_PREREQUISITE_BUNDLE_ID")?; + let slice_id = required("QDL_SLICE_ID")?; + let python_owner = required("QDL_PYTHON_OWNER_ID")?; + let rust_owner = required("QDL_RUST_OWNER_ID")?; + let rollback_owner = required("QDL_ROLLBACK_OWNER_ID")?; + let mode = env::var("QDL_REHEARSAL_MODE").unwrap_or_else(|_| "FULL".into()); + if mode == "RECOVERY_VERIFY" { + return run_recovery_verify( + &authority_topic, + &primary_topic, + &public_topic, + &legacy_topic, + &slice_id, + &nonce, + ) + .await; + } + if mode != "FULL" { + return Err(format!("unsupported Phase 9.2 rehearsal mode: {mode}").into()); + } + let group = format!("phase8-phase92-authority-{nonce}"); + let producer_config = transport_config("producer", &group)?; + let consumer_config = transport_config("consumer", &group)?; + let authority_sink = KafkaDurableSink::new(&producer_config)?; + let audit_sink = KafkaDurableSink::new(&producer_config)?; + let evidence_sink = KafkaDurableSink::new(&producer_config)?; + let authority_source = KafkaEventSource::new(&consumer_config, &[&authority_topic])?; + let topics = Phase92SinkTopics { + primary_canonical: primary_topic, + public_v2: public_topic, + legacy_v1: legacy_topic, + }; + let fenced_sink = Phase92FencedKafkaSink::new(&producer_config, topics.clone())?; + let started_at = now_ns()?; + let approved_at = started_at - 1_000_000_000; + let hold_until = started_at + 300_000_000_000; + let mut authority_offsets = Vec::new(); + let mut audit_offsets = Vec::new(); + let mut checkpoint_offsets = Vec::new(); + let mut handoff_offsets = Vec::new(); + let mut projection_offsets = Vec::new(); + let mut checks = serde_json::Map::new(); + + let initial = authority( + &slice_id, + Phase92AuthorityState::RustCanary, + &python_owner, + 7, + 11, + &candidate_digest, + Some(&bundle_id), + 89, + None, + None, + None, + approved_at, + hold_until, + ); + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &initial, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink.apply_authority(persisted).await?; + + let terminal = checkpoint( + "11111111-1111-4111-8111-111111111192", + &slice_id, + &python_owner, + 7, + 11, + 100, + &candidate_digest, + &nonce, + )?; + let to_rust = handoff( + "22222222-2222-4222-8222-222222222192", + Phase92HandoffDirection::PythonToRust, + &terminal, + &rust_owner, + Phase92AuthorityState::RustPrimary, + &bundle_id, + approved_at, + hold_until, + )?; + checkpoint_offsets.push( + evidence_sink + .append(&durable_record( + &checkpoint_topic, + &slice_id, + &terminal, + &format!("{nonce}:checkpoint:python"), + )?) + .await? + .cursor + .offset, + ); + handoff_offsets.push( + evidence_sink + .append(&durable_record( + &handoff_topic, + &slice_id, + &to_rust, + &format!("{nonce}:handoff:rust"), + )?) + .await? + .cursor + .offset, + ); + + let rust_primary = authority( + &slice_id, + Phase92AuthorityState::RustPrimary, + &rust_owner, + 8, + 12, + &candidate_digest, + Some(&bundle_id), + 100, + Some(100), + Some(&python_owner), + Some(to_rust.digest()?), + approved_at, + hold_until, + ); + checks.insert( + "direct_primary_without_handoff_rejected".into(), + json!(fenced_sink + .apply_authority(rust_primary.clone()) + .await + .is_err()), + ); + let cutover_started = Instant::now(); + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &rust_primary, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink + .apply_handoff(&terminal, &to_rust, persisted, now_ns()?) + .await?; + let cutover_ms = cutover_started.elapsed().as_secs_f64() * 1_000.0; + + for (name, context) in [ + ( + "terminal_watermark_rejected", + publication(&rust_primary, 100, SinkTarget::PrimaryCanonical), + ), + ( + "gap_watermark_rejected", + publication(&rust_primary, 102, SinkTarget::PrimaryCanonical), + ), + ( + "stale_owner_rejected", + Phase92PublicationContext { + owner_id: python_owner.clone(), + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ( + "stale_revision_rejected", + Phase92PublicationContext { + authority_revision: 7, + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ( + "stale_lease_rejected", + Phase92PublicationContext { + lease_epoch: 11, + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ( + "wrong_plan_rejected", + Phase92PublicationContext { + partition_plan_epoch: 2, + ..publication(&rust_primary, 101, SinkTarget::PrimaryCanonical) + }, + ), + ] { + checks.insert( + name.into(), + json!(fenced_sink + .append( + &durable_record( + &topics.primary_canonical, + "rejected", + &json!({"must_not_publish": name}), + &format!("{nonce}:rejected:{name}"), + )?, + &context, + now_ns()?, + ) + .await + .is_err()), + ); + } + + projection_offsets + .extend(publish_range(&fenced_sink, &topics, &rust_primary, 101, 164, &nonce).await?); + checks.insert( + "duplicate_after_ack_rejected".into(), + json!(fenced_sink + .append( + &durable_record( + &topics.primary_canonical, + "duplicate", + &json!({"must_not_publish": "duplicate"}), + &format!("{nonce}:duplicate"), + )?, + &publication(&rust_primary, 164, SinkTarget::PrimaryCanonical), + now_ns()?, + ) + .await + .is_err()), + ); + + let mut blocked = rust_primary.clone(); + blocked.state = Phase92AuthorityState::Blocked; + blocked.authority_revision = 9; + blocked.public_write_allowed = false; + blocked.legacy_write_allowed = false; + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &blocked, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink.apply_authority(persisted).await?; + + let mut pending = blocked.clone(); + pending.state = Phase92AuthorityState::RollbackPending; + pending.authority_revision = 10; + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &pending, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink.apply_authority(persisted).await?; + + let rust_terminal = checkpoint( + "33333333-3333-4333-8333-333333333192", + &slice_id, + &rust_owner, + 10, + 12, + 164, + &candidate_digest, + &nonce, + )?; + let to_python = handoff( + "44444444-4444-4444-8444-444444444192", + Phase92HandoffDirection::RustToPython, + &rust_terminal, + &rollback_owner, + Phase92AuthorityState::PythonPrimary, + &bundle_id, + approved_at, + hold_until, + )?; + checkpoint_offsets.push( + evidence_sink + .append(&durable_record( + &checkpoint_topic, + &slice_id, + &rust_terminal, + &format!("{nonce}:checkpoint:rust"), + )?) + .await? + .cursor + .offset, + ); + handoff_offsets.push( + evidence_sink + .append(&durable_record( + &handoff_topic, + &slice_id, + &to_python, + &format!("{nonce}:handoff:python"), + )?) + .await? + .cursor + .offset, + ); + let python_primary = authority( + &slice_id, + Phase92AuthorityState::PythonPrimary, + &rollback_owner, + 11, + 13, + &candidate_digest, + None, + 164, + Some(164), + Some(&rust_owner), + Some(to_python.digest()?), + approved_at, + hold_until, + ); + let rollback_started = Instant::now(); + let (authority_offset, audit_offset, persisted) = persist_authority( + &authority_sink, + &audit_sink, + &authority_source, + &authority_topic, + &audit_topic, + &nonce, + &python_primary, + ) + .await?; + authority_offsets.push(authority_offset); + audit_offsets.push(audit_offset); + fenced_sink + .apply_handoff(&rust_terminal, &to_python, persisted, now_ns()?) + .await?; + projection_offsets + .extend(publish_range(&fenced_sink, &topics, &python_primary, 165, 180, &nonce).await?); + let rollback_ms = rollback_started.elapsed().as_secs_f64() * 1_000.0; + checks.insert( + "rust_after_rollback_rejected".into(), + json!(fenced_sink + .append( + &durable_record( + &topics.primary_canonical, + "stale-rust", + &json!({"must_not_publish": "stale-rust"}), + &format!("{nonce}:stale-rust"), + )?, + &publication(&rust_primary, 165, SinkTarget::PrimaryCanonical), + now_ns()?, + ) + .await + .is_err()), + ); + + let passed = checks.values().all(|value| value == &json!(true)) + && authority_offsets.len() == 5 + && audit_offsets.len() == 5 + && checkpoint_offsets.len() == 2 + && handoff_offsets.len() == 2 + && projection_offsets.len() == 240; + println!( + "{}", + serde_json::to_string(&json!({ + "schema": "qdl.phase92.isolated-primary-runtime.v1", + "status": if passed { "PASS" } else { "FAIL" }, + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "authority_transitions": [ + "RUST_CANARY", "RUST_PRIMARY", "BLOCKED", + "ROLLBACK_PENDING", "PYTHON_PRIMARY" + ], + "authority_offsets": authority_offsets, + "audit_offsets": audit_offsets, + "checkpoint_offsets": checkpoint_offsets, + "handoff_offsets": handoff_offsets, + "projection_offsets": projection_offsets, + "checks": checks, + "cutover_ms": cutover_ms, + "rollback_ms": rollback_ms, + "first_rust_watermark": 101, + "last_rust_watermark": 164, + "first_python_rollback_watermark": 165, + "last_watermark": 180, + "isolated_primary_writes": 80, + "isolated_public_writes": 80, + "isolated_legacy_writes": 80, + "production_public_writes": 0, + "production_legacy_writes": 0, + "final_authority": "PYTHON_PRIMARY", + }))? + ); + if !passed { + return Err("Phase 9.2 isolated primary rehearsal failed".into()); + } + Ok(()) +} diff --git a/rust/qdl-kafka/src/bin/qdl-realtime-core.rs b/rust/qdl-kafka/src/bin/qdl-realtime-core.rs new file mode 100644 index 0000000..b32770b --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-realtime-core.rs @@ -0,0 +1,281 @@ +#![forbid(unsafe_code)] + +use std::env; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use prost::Message; +use qdl_contracts::qdl::provider::v1::RawProviderEnvelope; +use qdl_core::backoff::BackoffPolicy; +use qdl_core::transport::RetryClass; +use qdl_kafka::{ + KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError, TransactionalKafkaBridge, + TransactionalKafkaOutput, TransactionalShadowTopics, +}; +use qdl_realtime_core::{RealtimeCore, RealtimeCoreConfig}; +use qdl_venue_core::authority::{AuthorityMode, AuthorityRecord, PublicationContext, SinkTarget}; +use serde::Deserialize; +use serde_json::json; + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct RuntimeConfig { + core: RealtimeCoreConfig, + raw_topics: Vec, + authority: AuthorityRecord, + shard_id: String, + transactional_id: String, + batch_size: usize, + batch_wait_ms: u64, + max_events: u64, + metrics_every_batches: u64, +} + +impl RuntimeConfig { + fn validate(&self) -> Result<(), String> { + self.core.validate().map_err(|error| error.to_string())?; + self.authority.validate()?; + if self.authority.mode != AuthorityMode::RustShadow + || self.raw_topics.is_empty() + || self.raw_topics.iter().any(|topic| topic.trim().is_empty()) + || self.shard_id.trim().is_empty() + || self.transactional_id.trim().is_empty() + || self.batch_size == 0 + || self.batch_size > 10_000 + || self.batch_wait_ms == 0 + || self.batch_wait_ms > 1_000 + || self.metrics_every_batches == 0 + { + return Err("realtime-core runtime config is invalid or not RUST_SHADOW".into()); + } + Ok(()) + } +} + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn kafka_config() -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + let timeout_seconds = env::var("QDL_KAFKA_REQUEST_TIMEOUT_SECONDS") + .unwrap_or_else(|_| "30".into()) + .parse::() + .map_err(|_| "QDL_KAFKA_REQUEST_TIMEOUT_SECONDS must be positive")?; + if timeout_seconds == 0 { + return Err("QDL_KAFKA_REQUEST_TIMEOUT_SECONDS must be positive".into()); + } + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: required("QDL_KAFKA_CLIENT_ID")?, + group_id: required("QDL_KAFKA_GROUP_ID")?, + request_timeout: Duration::from_secs(timeout_seconds), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/client.crt"), + key_location: format!("{cert_root}/client.key"), + key_password: None, + }, + }) +} + +type RuntimeError = Box; + +fn now_ns() -> Result { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn should_retry_transport(class: RetryClass) -> bool { + class != RetryClass::NonRetryable +} + +fn retryable_runtime_error(error: &RuntimeError) -> bool { + error + .downcast_ref::() + .is_some_and(|value| should_retry_transport(value.retry_class())) +} + +async fn run_generation(config: &RuntimeConfig, generation: u64) -> Result<(), RuntimeError> { + let mut core = RealtimeCore::new(config.core.clone())?; + let bridge = TransactionalKafkaBridge::new( + &kafka_config()?, + TransactionalShadowTopics { + raw_inputs: config.raw_topics.clone(), + canonical: config.core.canonical_stream.clone(), + quarantine: config.core.quarantine_stream.clone(), + }, + &config.transactional_id, + )?; + bridge.apply_authority(config.authority.clone()).await?; + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_realtime_core_started", + "authority": "RUST_SHADOW", + "generation": generation, + "bindings": config.core.bindings.len(), + "batch_size": config.batch_size, + "production_public_writes": 0, + "production_legacy_writes": 0, + }))? + ); + + let mut processed = 0_u64; + let mut canonical = 0_u64; + let mut quarantines = 0_u64; + let mut duplicates = 0_u64; + let mut filtered = 0_u64; + let mut batches = 0_u64; + 'service: loop { + if config.max_events > 0 && processed >= config.max_events { + break; + } + let first = tokio::select! { + result = bridge.next() => result?, + _ = tokio::signal::ctrl_c() => break 'service, + }; + let mut inputs = vec![first]; + while inputs.len() < config.batch_size + && (config.max_events == 0 || processed + (inputs.len() as u64) < config.max_events) + { + match tokio::time::timeout(Duration::from_millis(config.batch_wait_ms), bridge.next()) + .await + { + Ok(Ok(input)) => inputs.push(input), + Ok(Err(error)) => return Err(error.into()), + Err(_) => break, + } + } + + let normalized_at_ns = now_ns()?; + let mut outputs = vec![]; + for input in &inputs { + let raw = RawProviderEnvelope::decode(input.record.payload.as_slice())?; + if raw.authority_revision != config.authority.revision { + return Err("raw authority revision does not match runtime authority".into()); + } + let result = core.process_at_transport_offset( + raw.clone(), + normalized_at_ns, + input.cursor.offset, + )?; + canonical += result.canonical.len() as u64; + quarantines += result.quarantines.len() as u64; + duplicates += result.duplicates as u64; + filtered += result.filtered as u64; + for record in result.canonical { + outputs.push(TransactionalKafkaOutput { + record, + publication: PublicationContext { + slice_id: config.authority.slice_id.clone(), + authority_revision: raw.authority_revision, + shard_id: config.shard_id.clone(), + lease_epoch: raw.lease_epoch, + target: SinkTarget::ShadowCanonical, + }, + raw_provider_envelope: Some(input.record.payload.clone()), + }); + } + for record in result.quarantines { + outputs.push(TransactionalKafkaOutput { + record, + publication: PublicationContext { + slice_id: config.authority.slice_id.clone(), + authority_revision: raw.authority_revision, + shard_id: config.shard_id.clone(), + lease_epoch: raw.lease_epoch, + target: SinkTarget::ShadowQuarantine, + }, + raw_provider_envelope: None, + }); + } + } + bridge.commit(&inputs, &outputs).await?; + processed += inputs.len() as u64; + batches += 1; + if batches % config.metrics_every_batches == 0 { + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_realtime_core_progress", + "generation": generation, + "processed": processed, + "canonical": canonical, + "quarantines": quarantines, + "duplicates": duplicates, + "filtered": filtered, + "batches": batches, + }))? + ); + } + } + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_realtime_core_stopped", + "generation": generation, + "processed": processed, + "canonical": canonical, + "quarantines": quarantines, + "duplicates": duplicates, + "filtered": filtered, + "batches": batches, + }))? + ); + Ok(()) +} + +#[tokio::main] +async fn main() -> Result<(), RuntimeError> { + let config_path = env::args() + .nth(1) + .ok_or("usage: qdl-realtime-core CONFIG.json")?; + let config: RuntimeConfig = serde_json::from_slice(&tokio::fs::read(config_path).await?)?; + config.validate()?; + let backoff = BackoffPolicy { + initial_ms: 500, + maximum_ms: 30_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate()?; + let mut generation = 0_u64; + let mut failures = 0_u32; + loop { + generation = generation.saturating_add(1); + match run_generation(&config, generation).await { + Ok(()) => return Ok(()), + Err(error) if retryable_runtime_error(&error) => { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_realtime_core_retry", + "generation": generation, + "attempt": failures, + "error": error.to_string(), + }))? + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + Err(error) => return Err(error), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn runtime_retries_only_retryable_or_capacity_transport_errors() { + assert!(should_retry_transport(RetryClass::Retryable)); + assert!(should_retry_transport(RetryClass::Capacity)); + assert!(!should_retry_transport(RetryClass::NonRetryable)); + } +} diff --git a/rust/qdl-kafka/src/lib.rs b/rust/qdl-kafka/src/lib.rs index 7d440c9..d76e728 100644 --- a/rust/qdl-kafka/src/lib.rs +++ b/rust/qdl-kafka/src/lib.rs @@ -4,16 +4,24 @@ use std::fmt::{Display, Formatter}; use std::path::Path; use std::time::Duration; +use futures_util::future::try_join_all; use qdl_core::transport::{AppendResult, Cursor, DurableRecord, RetryClass}; -use qdl_venue_core::authority::{AuthorityFence, AuthorityRecord, PublicationContext}; +use qdl_venue_core::authority::{ + AuthorityFence, AuthorityRecord, Phase92AcceptedHandoff, Phase92AuthorityFence, + Phase92AuthorityRecord, Phase92PublicationContext, Phase92TerminalCheckpoint, + Phase9AuthorityFence, Phase9AuthorityRecord, Phase9PublicationContext, PublicationContext, + SinkTarget, +}; use rdkafka::config::ClientConfig; use rdkafka::consumer::{CommitMode, Consumer, StreamConsumer}; use rdkafka::error::KafkaError; use rdkafka::message::{Header, Headers, Message, OwnedHeaders}; -use rdkafka::producer::{FutureProducer, FutureRecord}; +use rdkafka::producer::{DeliveryFuture, FutureProducer, FutureRecord, Producer}; +use rdkafka::topic_partition_list::{Offset, TopicPartitionList}; use rdkafka::util::Timeout; const EVENT_ID_HEADER: &str = "qdl-event-id"; +const RAW_ENVELOPE_HEADER: &str = "qdl-raw-provider-envelope"; #[derive(Clone, Debug, Eq, PartialEq)] pub struct KafkaTlsConfig { @@ -164,23 +172,275 @@ impl FencedKafkaSink { .map_err(KafkaTransportError::Fencing) } - pub async fn append( + pub fn enqueue( &self, record: &DurableRecord, publication: &PublicationContext, - ) -> Result { + ) -> Result { self.fence .lock() .map_err(|_| KafkaTransportError::Fencing("authority lock poisoned".into()))? .permits(publication) .map_err(KafkaTransportError::Fencing)?; - self.sink.append(record).await + self.sink.enqueue(record) + } + + pub async fn append( + &self, + record: &DurableRecord, + publication: &PublicationContext, + ) -> Result { + self.enqueue(record, publication)?.wait().await + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase9SinkTopics { + pub shadow_raw: String, + pub shadow_canonical: String, + pub shadow_quarantine: String, + pub canary_canonical: String, +} + +impl Phase9SinkTopics { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + let topics = [ + self.shadow_raw.as_str(), + self.shadow_canonical.as_str(), + self.shadow_quarantine.as_str(), + self.canary_canonical.as_str(), + ]; + if topics.iter().any(|topic| topic.trim().is_empty()) { + return Err(KafkaTransportError::Configuration( + "Phase 9 sink topics must not be empty".into(), + )); + } + let unique: std::collections::HashSet<&str> = topics.into_iter().collect(); + if unique.len() != 4 { + return Err(KafkaTransportError::Configuration( + "Phase 9 sink topics must be isolated and unique".into(), + )); + } + Ok(()) + } + + fn permits(&self, target: qdl_venue_core::authority::SinkTarget, stream: &str) -> bool { + use qdl_venue_core::authority::SinkTarget; + match target { + SinkTarget::ShadowRaw => stream == self.shadow_raw, + SinkTarget::ShadowCanonical => stream == self.shadow_canonical, + SinkTarget::ShadowQuarantine => stream == self.shadow_quarantine, + SinkTarget::CanaryCanonical => stream == self.canary_canonical, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 => false, + } + } +} + +/// Phase 9.1 sink keeps authority stable through durable ACK, then commits +/// the source watermark. A failed append remains retryable at the same watermark. +pub struct Phase9FencedKafkaSink { + sink: KafkaDurableSink, + fence: tokio::sync::Mutex, + topics: Phase9SinkTopics, +} + +impl Phase9FencedKafkaSink { + pub fn new( + config: &KafkaTransportConfig, + topics: Phase9SinkTopics, + ) -> Result { + topics.validate()?; + Ok(Self { + sink: KafkaDurableSink::new(config)?, + fence: tokio::sync::Mutex::new(Phase9AuthorityFence::default()), + topics, + }) + } + + pub async fn apply_authority( + &self, + record: Phase9AuthorityRecord, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .apply(record) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn append( + &self, + record: &DurableRecord, + publication: &Phase9PublicationContext, + now_ns: i64, + ) -> Result { + if !self.topics.permits(publication.target, &record.stream) { + return Err(KafkaTransportError::Fencing( + "publication target does not match its isolated Kafka topic".into(), + )); + } + let mut fence = self.fence.lock().await; + fence + .permits(publication, now_ns) + .map_err(KafkaTransportError::Fencing)?; + let result = self.sink.append(record).await?; + fence + .commit(publication) + .map_err(KafkaTransportError::Fencing)?; + Ok(result) + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92SinkTopics { + pub primary_canonical: String, + pub public_v2: String, + pub legacy_v1: String, +} + +impl Phase92SinkTopics { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + let topics = [ + self.primary_canonical.as_str(), + self.public_v2.as_str(), + self.legacy_v1.as_str(), + ]; + if topics.iter().any(|topic| topic.trim().is_empty()) { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 sink/projector topics must not be empty".into(), + )); + } + if topics[0] == topics[1] || topics[0] == topics[2] || topics[1] == topics[2] { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 sink/projector topics must be isolated and unique".into(), + )); + } + Ok(()) + } + + fn permits(&self, target: SinkTarget, stream: &str) -> bool { + match target { + SinkTarget::PrimaryCanonical => stream == self.primary_canonical, + SinkTarget::PublicV2 => stream == self.public_v2, + SinkTarget::LegacyV1 => stream == self.legacy_v1, + SinkTarget::ShadowRaw + | SinkTarget::ShadowCanonical + | SinkTarget::ShadowQuarantine + | SinkTarget::CanaryCanonical => false, + } + } +} + +/// Phase 9.2 final sink and compatibility projector share one authority fence. +/// The mutex remains held through durable ACK so an authority update cannot race +/// between sink acceptance and watermark commit. +pub struct Phase92FencedKafkaSink { + sink: KafkaDurableSink, + fence: tokio::sync::Mutex, + topics: Phase92SinkTopics, +} + +impl Phase92FencedKafkaSink { + pub fn new( + config: &KafkaTransportConfig, + topics: Phase92SinkTopics, + ) -> Result { + topics.validate()?; + Ok(Self { + sink: KafkaDurableSink::new(config)?, + fence: tokio::sync::Mutex::new(Phase92AuthorityFence::default()), + topics, + }) + } + + pub async fn apply_authority( + &self, + record: Phase92AuthorityRecord, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .apply(record) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn apply_handoff( + &self, + checkpoint: &Phase92TerminalCheckpoint, + handoff: &Phase92AcceptedHandoff, + record: Phase92AuthorityRecord, + now_ns: i64, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .apply_handoff(checkpoint, handoff, record, now_ns) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn restore_committed_watermark( + &self, + publication: &Phase92PublicationContext, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() + .await + .restore_committed_watermark(publication) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn append( + &self, + record: &DurableRecord, + publication: &Phase92PublicationContext, + now_ns: i64, + ) -> Result { + if !self.topics.permits(publication.target, &record.stream) { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 publication target does not match its topic".into(), + )); + } + let mut fence = self.fence.lock().await; + fence + .permits(publication, now_ns) + .map_err(KafkaTransportError::Fencing)?; + let result = self.sink.append(record).await?; + fence + .commit(publication) + .map_err(KafkaTransportError::Fencing)?; + Ok(result) + } +} + +pub struct PendingKafkaAppend { + delivery: DeliveryFuture, + stream: String, + partition_key: String, +} + +impl PendingKafkaAppend { + pub async fn wait(self) -> Result { + let delivery = self + .delivery + .await + .map_err(|_| KafkaTransportError::Delivery(KafkaError::Canceled))? + .map_err(|(error, _)| KafkaTransportError::Delivery(error))?; + let offset = u64::try_from(delivery.offset) + .map_err(|_| KafkaTransportError::InvalidOffset(delivery.offset))?; + Ok(AppendResult { + cursor: Cursor { + stream: self.stream, + transport_partition: delivery.partition, + partition_key: self.partition_key, + offset, + }, + duplicate: false, + }) } } pub struct KafkaDurableSink { producer: FutureProducer, - request_timeout: Duration, } impl KafkaDurableSink { @@ -198,14 +458,13 @@ impl KafkaDurableSink { ); Ok(Self { producer: client.create()?, - request_timeout: config.request_timeout, }) } - pub async fn append( + pub fn enqueue( &self, record: &DurableRecord, - ) -> Result { + ) -> Result { if record.stream.trim().is_empty() { return Err(KafkaTransportError::MissingField("stream")); } @@ -221,27 +480,324 @@ impl KafkaDurableSink { }); let delivery = self .producer - .send( + .send_result( FutureRecord::to(&record.stream) .key(record.partition_key.as_bytes()) .payload(record.payload.as_slice()) .headers(headers), - Timeout::After(self.request_timeout), ) + .map_err(|(error, _)| KafkaTransportError::Kafka(error))?; + Ok(PendingKafkaAppend { + delivery, + stream: record.stream.clone(), + partition_key: record.partition_key.clone(), + }) + } + + pub async fn append( + &self, + record: &DurableRecord, + ) -> Result { + self.enqueue(record)?.wait().await + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct TransactionalShadowTopics { + pub raw_inputs: Vec, + pub canonical: String, + pub quarantine: String, +} + +impl TransactionalShadowTopics { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + if self.raw_inputs.is_empty() + || self.raw_inputs.iter().any(|topic| topic.trim().is_empty()) + || self.canonical.trim().is_empty() + || self.quarantine.trim().is_empty() + { + return Err(KafkaTransportError::Configuration( + "transactional shadow topics must not be empty".into(), + )); + } + let mut unique = std::collections::HashSet::new(); + for topic in self + .raw_inputs + .iter() + .chain([&self.canonical, &self.quarantine]) + { + if !unique.insert(topic.as_str()) { + return Err(KafkaTransportError::Configuration( + "transactional shadow topics must be isolated".into(), + )); + } + } + Ok(()) + } + + fn permits(&self, target: SinkTarget, stream: &str) -> bool { + match target { + SinkTarget::ShadowCanonical => stream == self.canonical, + SinkTarget::ShadowQuarantine => stream == self.quarantine, + SinkTarget::ShadowRaw + | SinkTarget::CanaryCanonical + | SinkTarget::PrimaryCanonical + | SinkTarget::PublicV2 + | SinkTarget::LegacyV1 => false, + } + } +} + +pub struct TransactionalKafkaInput { + pub record: DurableRecord, + pub cursor: Cursor, +} + +pub struct TransactionalKafkaOutput { + pub record: DurableRecord, + pub publication: PublicationContext, + pub raw_provider_envelope: Option>, +} + +fn transactional_output_headers( + event_id: &[u8], + raw_provider_envelope: Option<&[u8]>, +) -> OwnedHeaders { + let mut headers = OwnedHeaders::new().insert(Header { + key: EVENT_ID_HEADER, + value: Some(event_id), + }); + if let Some(raw) = raw_provider_envelope { + headers = headers.insert(Header { + key: RAW_ENVELOPE_HEADER, + value: Some(raw), + }); + } + headers +} + +/// Kafka consume-transform-produce boundary. Output records and the next raw +/// consumer offset commit atomically, so a process crash cannot acknowledge raw +/// input without its canonical/quarantine result or duplicate committed output. +pub struct TransactionalKafkaBridge { + producer: FutureProducer, + consumer: StreamConsumer, + fence: tokio::sync::Mutex, + topics: TransactionalShadowTopics, + request_timeout: Duration, +} + +impl TransactionalKafkaBridge { + pub fn new( + config: &KafkaTransportConfig, + topics: TransactionalShadowTopics, + transactional_id: &str, + ) -> Result { + topics.validate()?; + if transactional_id.trim().is_empty() { + return Err(KafkaTransportError::Configuration( + "transactional.id must not be empty".into(), + )); + } + let mut consumer_config = config.client_config()?; + consumer_config + .set("group.id", &config.group_id) + .set("enable.auto.commit", "false") + .set("enable.auto.offset.store", "false") + .set("auto.offset.reset", "earliest") + .set("isolation.level", "read_committed"); + let consumer: StreamConsumer = consumer_config.create()?; + let raw_topics: Vec<&str> = topics.raw_inputs.iter().map(String::as_str).collect(); + consumer.subscribe(&raw_topics)?; + + let mut producer_config = config.client_config()?; + producer_config + .set("transactional.id", transactional_id) + .set("enable.idempotence", "true") + .set("acks", "all") + .set("max.in.flight.requests.per.connection", "5") + .set("retries", "2147483647") + .set("compression.type", "zstd") + .set( + "transaction.timeout.ms", + config.request_timeout.as_millis().to_string(), + ) + .set( + "delivery.timeout.ms", + config.request_timeout.as_millis().to_string(), + ); + let producer: FutureProducer = producer_config.create()?; + producer.init_transactions(Timeout::After(config.request_timeout))?; + Ok(Self { + producer, + consumer, + fence: tokio::sync::Mutex::new(AuthorityFence::default()), + topics, + request_timeout: config.request_timeout, + }) + } + + pub async fn apply_authority( + &self, + record: AuthorityRecord, + ) -> Result<(), KafkaTransportError> { + self.fence + .lock() .await - .map_err(|(error, _)| KafkaTransportError::Delivery(error))?; - let offset = u64::try_from(delivery.offset) - .map_err(|_| KafkaTransportError::InvalidOffset(delivery.offset))?; - Ok(AppendResult { - cursor: Cursor { - stream: record.stream.clone(), - transport_partition: delivery.partition, - partition_key: record.partition_key.clone(), - offset, + .apply(record) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn next(&self) -> Result { + let message = self.consumer.recv().await?; + let payload = message + .payload() + .ok_or(KafkaTransportError::MissingField("payload"))? + .to_vec(); + let key = message + .key() + .ok_or(KafkaTransportError::MissingField("partition_key"))?; + let partition_key = std::str::from_utf8(key) + .map_err(|_| KafkaTransportError::InvalidUtf8("partition_key"))? + .to_owned(); + let event_id = message + .headers() + .and_then(|headers| { + headers + .iter() + .find(|header| header.key == EVENT_ID_HEADER) + .and_then(|header| header.value.map(ToOwned::to_owned)) + }) + .ok_or(KafkaTransportError::MissingField("event_id header"))?; + let offset = u64::try_from(message.offset()) + .map_err(|_| KafkaTransportError::InvalidOffset(message.offset()))?; + let cursor = Cursor { + stream: message.topic().to_owned(), + transport_partition: message.partition(), + partition_key: partition_key.clone(), + offset, + }; + let accepted_at_ns = message.timestamp().to_millis().unwrap_or_default() * 1_000_000; + Ok(TransactionalKafkaInput { + record: DurableRecord { + stream: message.topic().to_owned(), + partition_key, + event_id, + payload, + accepted_at_ns, }, - duplicate: false, + cursor, }) } + + pub async fn commit( + &self, + inputs: &[TransactionalKafkaInput], + outputs: &[TransactionalKafkaOutput], + ) -> Result, KafkaTransportError> { + if inputs.is_empty() { + return Err(KafkaTransportError::Configuration( + "transaction input batch must not be empty".into(), + )); + } + if inputs + .iter() + .any(|input| !self.topics.raw_inputs.contains(&input.cursor.stream)) + { + return Err(KafkaTransportError::Fencing( + "transaction input is outside configured raw topics".into(), + )); + } + let mut fence = self.fence.lock().await; + for output in outputs { + if !self + .topics + .permits(output.publication.target, &output.record.stream) + { + return Err(KafkaTransportError::Fencing( + "transaction output target does not match shadow topic".into(), + )); + } + fence + .permits(&output.publication) + .map_err(KafkaTransportError::Fencing)?; + } + + self.producer.begin_transaction()?; + let transaction = async { + let deliveries = outputs.iter().map(|output| async move { + let headers = transactional_output_headers( + output.record.event_id.as_slice(), + output.raw_provider_envelope.as_deref(), + ); + let delivery = self + .producer + .send( + FutureRecord::to(&output.record.stream) + .key(output.record.partition_key.as_bytes()) + .payload(output.record.payload.as_slice()) + .headers(headers), + Timeout::After(self.request_timeout), + ) + .await + .map_err(|(error, _)| KafkaTransportError::Delivery(error))?; + let offset = u64::try_from(delivery.offset) + .map_err(|_| KafkaTransportError::InvalidOffset(delivery.offset))?; + Ok::<_, KafkaTransportError>(AppendResult { + cursor: Cursor { + stream: output.record.stream.clone(), + transport_partition: delivery.partition, + partition_key: output.record.partition_key.clone(), + offset, + }, + duplicate: false, + }) + }); + let accepted = try_join_all(deliveries).await?; + let mut next_offsets: std::collections::BTreeMap<(String, i32), i64> = + std::collections::BTreeMap::new(); + for input in inputs { + let next_offset = input + .cursor + .offset + .checked_add(1) + .and_then(|value| i64::try_from(value).ok()) + .ok_or(KafkaTransportError::InvalidOffset(i64::MAX))?; + next_offsets + .entry(( + input.cursor.stream.clone(), + input.cursor.transport_partition, + )) + .and_modify(|current| *current = (*current).max(next_offset)) + .or_insert(next_offset); + } + let mut offsets = TopicPartitionList::new(); + for ((topic, partition), offset) in next_offsets { + offsets.add_partition_offset(&topic, partition, Offset::Offset(offset))?; + } + let group = self.consumer.group_metadata().ok_or_else(|| { + KafkaTransportError::Configuration("consumer group metadata is unavailable".into()) + })?; + self.producer.send_offsets_to_transaction( + &offsets, + &group, + Timeout::After(self.request_timeout), + )?; + self.producer + .commit_transaction(Timeout::After(self.request_timeout))?; + Ok::<_, KafkaTransportError>(accepted) + } + .await; + match transaction { + Ok(value) => Ok(value), + Err(error) => { + self.producer + .abort_transaction(Timeout::After(self.request_timeout)) + .map_err(KafkaTransportError::Kafka)?; + Err(error) + } + } + } } pub struct KafkaEventSource { @@ -323,10 +879,31 @@ impl KafkaEventSource { #[cfg(test)] mod tests { - use super::{KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError}; + use super::{ + transactional_output_headers, KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError, + Phase92SinkTopics, Phase9SinkTopics, TransactionalShadowTopics, EVENT_ID_HEADER, + RAW_ENVELOPE_HEADER, + }; use qdl_core::transport::RetryClass; + use qdl_venue_core::authority::SinkTarget; + use rdkafka::message::Headers; use std::time::Duration; + #[test] + fn transactional_headers_preserve_private_raw_lineage() { + let event_id = [7_u8; 16]; + let raw = b"raw-provider-envelope"; + let headers = transactional_output_headers(&event_id, Some(raw)); + let values = headers + .iter() + .map(|header| (header.key, header.value.unwrap_or_default().to_vec())) + .collect::>(); + assert_eq!(values[EVENT_ID_HEADER], event_id); + assert_eq!(values[RAW_ENVELOPE_HEADER], raw); + let without_raw = transactional_output_headers(&event_id, None); + assert_eq!(without_raw.count(), 1); + } + #[test] fn config_fails_closed_without_tls_files() { let config = KafkaTransportConfig { @@ -346,6 +923,72 @@ mod tests { assert_eq!(error.retry_class(), RetryClass::NonRetryable); } + #[test] + fn phase9_topics_are_unique_and_bind_target_to_stream() { + let topics = Phase9SinkTopics { + shadow_raw: "qdl.phase8.phase91.shadow.raw".into(), + shadow_canonical: "qdl.phase8.phase91.shadow.canonical".into(), + shadow_quarantine: "qdl.phase8.phase91.shadow.quarantine".into(), + canary_canonical: "qdl.phase8.phase91.canary.canonical".into(), + }; + topics.validate().unwrap(); + assert!(topics.permits( + SinkTarget::CanaryCanonical, + "qdl.phase8.phase91.canary.canonical" + )); + assert!(!topics.permits(SinkTarget::CanaryCanonical, "qdl.phase8.phase91.public")); + assert!(!topics.permits(SinkTarget::PublicV2, &topics.canary_canonical)); + + let duplicate = Phase9SinkTopics { + shadow_raw: "same".into(), + shadow_canonical: "same".into(), + shadow_quarantine: "quarantine".into(), + canary_canonical: "other".into(), + }; + assert!(duplicate.validate().is_err()); + } + + #[test] + fn phase92_topics_are_unique_and_bind_final_and_projector_targets() { + let topics = Phase92SinkTopics { + primary_canonical: "qdl.phase92.primary.canonical".into(), + public_v2: "qdl.phase92.public.v2".into(), + legacy_v1: "qdl.phase92.legacy.v1".into(), + }; + topics.validate().unwrap(); + assert!(topics.permits(SinkTarget::PrimaryCanonical, &topics.primary_canonical)); + assert!(topics.permits(SinkTarget::PublicV2, &topics.public_v2)); + assert!(topics.permits(SinkTarget::LegacyV1, &topics.legacy_v1)); + assert!(!topics.permits(SinkTarget::PublicV2, &topics.legacy_v1)); + assert!(!topics.permits(SinkTarget::CanaryCanonical, &topics.primary_canonical)); + + let duplicate = Phase92SinkTopics { + primary_canonical: "same".into(), + public_v2: "same".into(), + legacy_v1: "other".into(), + }; + assert!(duplicate.validate().is_err()); + } + + #[test] + fn transactional_shadow_topics_are_isolated_and_target_bound() { + let topics = TransactionalShadowTopics { + raw_inputs: vec!["qdl.raw.binance".into(), "qdl.raw.okx".into()], + canonical: "qdl.canonical.v2".into(), + quarantine: "qdl.quarantine.v1".into(), + }; + topics.validate().unwrap(); + assert!(topics.permits(SinkTarget::ShadowCanonical, &topics.canonical)); + assert!(topics.permits(SinkTarget::ShadowQuarantine, &topics.quarantine)); + assert!(!topics.permits(SinkTarget::PublicV2, &topics.canonical)); + let duplicate = TransactionalShadowTopics { + raw_inputs: vec!["same".into()], + canonical: "same".into(), + quarantine: "other".into(), + }; + assert!(duplicate.validate().is_err()); + } + #[test] fn zero_timeout_and_empty_identity_fail_closed() { let config = KafkaTransportConfig { diff --git a/rust/qdl-realtime-core/Cargo.toml b/rust/qdl-realtime-core/Cargo.toml new file mode 100644 index 0000000..13a5f08 --- /dev/null +++ b/rust/qdl-realtime-core/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "qdl-realtime-core" +version = "0.1.0" +edition.workspace = true +license.workspace = true +rust-version.workspace = true + +[dependencies] +prost.workspace = true +qdl-contracts.workspace = true +qdl-core.workspace = true +qdl-provider-envelope = { path = "../qdl-provider-envelope", version = "=0.1.0" } +qdl-venue-core = { path = "../qdl-venue-core", version = "=0.1.0" } +serde.workspace = true +serde_json.workspace = true +sha2.workspace = true diff --git a/rust/qdl-realtime-core/src/bin/qdl-realtime-core-benchmark.rs b/rust/qdl-realtime-core/src/bin/qdl-realtime-core-benchmark.rs new file mode 100644 index 0000000..c6377c5 --- /dev/null +++ b/rust/qdl-realtime-core/src/bin/qdl-realtime-core-benchmark.rs @@ -0,0 +1,135 @@ +#![forbid(unsafe_code)] + +use std::time::Instant; + +use prost::Message; +use qdl_contracts::qdl::provider::v1::{ + CaptureBoundary, RawProviderEnvelope, TransportCompression, TransportProtocol, +}; +use qdl_realtime_core::{CoreBinding, RealtimeCore, RealtimeCoreConfig}; +use qdl_venue_core::ordering::SequencePolicy; +use serde_json::json; +use sha2::{Digest, Sha256}; + +fn percentile(values: &mut [u128], percentile: f64) -> u128 { + values.sort_unstable(); + let index = ((values.len() - 1) as f64 * percentile).round() as usize; + values[index] +} + +fn main() -> Result<(), Box> { + let events = std::env::args() + .nth(1) + .unwrap_or_else(|| "100000".into()) + .parse::()?; + let minimum_events_per_second = std::env::args() + .nth(2) + .unwrap_or_else(|| "50000".into()) + .parse::()?; + if events < 1000 || minimum_events_per_second <= 0.0 { + return Err("benchmark bounds are invalid".into()); + } + let binding = CoreBinding { + provider: "BINANCE_DIRECT".into(), + venue: "BINANCE".into(), + market: "USDM".into(), + product_type: "PERPETUAL".into(), + native_symbol: "BTCUSDT".into(), + native_channel: "trade".into(), + provider_kind: "binance_usdm_trade".into(), + instrument_uid: "benchmark-btcusdt".into(), + instrument_id: "BINANCE.USDM.PERPETUAL.BTC-USDT".into(), + instrument_revision: 1, + instrument_catalog_revision: 1, + source_id: "benchmark-binance-trade".into(), + source_role: "PRIMARY".into(), + normalizer_version: "qdl-rust-core/2.0.0".into(), + require_final_bar: false, + sequence_policy: SequencePolicy::Monotonic, + }; + let mut core = RealtimeCore::new(RealtimeCoreConfig { + canonical_stream: "benchmark.canonical".into(), + quarantine_stream: "benchmark.quarantine".into(), + allow_test_provenance: true, + dedup_capacity: events as usize, + bindings: vec![binding], + })?; + let started = Instant::now(); + let mut latencies = Vec::with_capacity(events as usize); + let mut output_count = 0_u64; + let mut output_bytes = 0_u64; + for sequence in 1..=events { + let raw_frame = serde_json::to_vec(&json!({ + "s": "BTCUSDT", "t": sequence, "p": "60000.10", + "q": "0.001", "T": 1_786_352_400_000_u64 + sequence, "m": false, + }))?; + let capture_id = Sha256::digest(sequence.to_be_bytes())[..16].to_vec(); + let envelope = RawProviderEnvelope { + raw_schema_name: "qdl.provider.raw".into(), + raw_schema_major: 1, + raw_schema_minor: 0, + capture_id, + provider: "BINANCE_DIRECT".into(), + venue: "BINANCE".into(), + market: "USDM".into(), + product_type: "PERPETUAL".into(), + native_symbol: "BTCUSDT".into(), + native_channel: "trade".into(), + subscription_id: "benchmark".into(), + source_session_id: "benchmark-session".into(), + connection_generation: 1, + lease_epoch: 1, + authority_revision: 1, + partition_plan_epoch: 1, + received_at_ns: 1_786_352_400_000_000_000 + sequence as i64, + transport_protocol: TransportProtocol::FileReplay as i32, + transport_compression: TransportCompression::None as i32, + capture_boundary: CaptureBoundary::ReplayBytes as i32, + raw_frame_sha256: Sha256::digest(&raw_frame).to_vec(), + raw_frame_bytes: raw_frame, + adapter_version: "benchmark/1".into(), + config_revision: 1, + instrument_catalog_revision: 1, + correlation_id: format!("benchmark-{sequence}"), + test_provenance: true, + }; + let encoded = envelope.encode_to_vec(); + let event_started = Instant::now(); + let batch = core.process_bytes(&encoded, 1_786_352_400_100_000_000 + sequence as i64)?; + latencies.push(event_started.elapsed().as_nanos()); + if batch.canonical.len() != 1 || !batch.quarantines.is_empty() || batch.duplicates != 0 { + return Err("benchmark core produced a non-canonical decision".into()); + } + output_count += 1; + output_bytes += batch.canonical[0].payload.len() as u64; + } + let elapsed = started.elapsed().as_secs_f64(); + let throughput = events as f64 / elapsed; + let p50_ns = percentile(&mut latencies, 0.50); + let p99_ns = percentile(&mut latencies, 0.99); + let status = if throughput >= minimum_events_per_second { + "PASS" + } else { + "FAIL" + }; + println!( + "{}", + serde_json::to_string(&json!({ + "status": status, + "events": events, + "canonical": output_count, + "quarantines": 0, + "duplicates": 0, + "elapsed_seconds": elapsed, + "events_per_second": throughput, + "minimum_events_per_second": minimum_events_per_second, + "p50_ns": p50_ns, + "p99_ns": p99_ns, + "output_bytes": output_bytes, + }))? + ); + if status != "PASS" { + return Err("realtime core throughput gate failed".into()); + } + Ok(()) +} diff --git a/rust/qdl-realtime-core/src/lib.rs b/rust/qdl-realtime-core/src/lib.rs new file mode 100644 index 0000000..9ca861a --- /dev/null +++ b/rust/qdl-realtime-core/src/lib.rs @@ -0,0 +1,1003 @@ +#![forbid(unsafe_code)] + +use std::collections::{BTreeMap, HashSet, VecDeque}; +use std::fmt::{Display, Formatter}; + +use prost::Message; +use qdl_contracts::qdl::marketdata::v2::{event_envelope, BarLifecycle, EventEnvelope}; +use qdl_contracts::qdl::provider::v1::{QuarantineReason, QuarantineRecord, RawProviderEnvelope}; +use qdl_core::canonical::{canonicalize_trade, TradeContext, TradeFixture}; +use qdl_core::okx::expand_data_frame; +use qdl_core::transport::DurableRecord; +use qdl_provider_envelope::validate as validate_raw; +use qdl_venue_core::ordering::{OrderingStage, OrderingTracker, SequenceDecision, SequencePolicy}; +use serde::Deserialize; +use serde_json::Value; +use sha2::{Digest, Sha256}; + +const TRANSPORT_SEQUENCE_STRIDE: u64 = 1_000_000; + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CoreBinding { + pub provider: String, + pub venue: String, + pub market: String, + pub product_type: String, + pub native_symbol: String, + pub native_channel: String, + pub provider_kind: String, + pub instrument_uid: String, + pub instrument_id: String, + pub instrument_revision: u64, + pub instrument_catalog_revision: u64, + pub source_id: String, + pub source_role: String, + pub normalizer_version: String, + #[serde(default)] + pub require_final_bar: bool, + pub sequence_policy: SequencePolicy, +} + +impl CoreBinding { + fn key(&self) -> String { + binding_key( + &self.provider, + &self.venue, + &self.market, + &self.product_type, + &self.native_symbol, + &self.native_channel, + ) + } + + fn validate(&self) -> Result<(), CoreError> { + for (name, value) in [ + ("provider", self.provider.as_str()), + ("venue", self.venue.as_str()), + ("market", self.market.as_str()), + ("product_type", self.product_type.as_str()), + ("native_symbol", self.native_symbol.as_str()), + ("native_channel", self.native_channel.as_str()), + ("provider_kind", self.provider_kind.as_str()), + ("instrument_uid", self.instrument_uid.as_str()), + ("instrument_id", self.instrument_id.as_str()), + ("source_id", self.source_id.as_str()), + ("source_role", self.source_role.as_str()), + ("normalizer_version", self.normalizer_version.as_str()), + ] { + if value.trim().is_empty() { + return Err(CoreError::Configuration(format!( + "binding {name} must not be empty" + ))); + } + } + if self.instrument_revision == 0 || self.instrument_catalog_revision == 0 { + return Err(CoreError::Configuration( + "binding instrument revisions must be positive".into(), + )); + } + if !matches!( + self.source_role.as_str(), + "PRIMARY" | "SECONDARY" | "REFERENCE" | "BACKFILL" + ) { + return Err(CoreError::Configuration( + "binding source role is invalid".into(), + )); + } + Ok(()) + } +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RealtimeCoreConfig { + pub canonical_stream: String, + pub quarantine_stream: String, + pub allow_test_provenance: bool, + pub dedup_capacity: usize, + pub bindings: Vec, +} + +impl RealtimeCoreConfig { + pub fn validate(&self) -> Result<(), CoreError> { + if self.canonical_stream.trim().is_empty() + || self.quarantine_stream.trim().is_empty() + || self.canonical_stream == self.quarantine_stream + || self.dedup_capacity == 0 + || self.bindings.is_empty() + { + return Err(CoreError::Configuration( + "realtime core stream/bounds/bindings are invalid".into(), + )); + } + let mut keys = HashSet::new(); + for binding in &self.bindings { + binding.validate()?; + if !keys.insert(binding.key()) { + return Err(CoreError::Configuration( + "duplicate realtime core binding".into(), + )); + } + } + Ok(()) + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum CoreError { + Configuration(String), + Decode(String), + RawEnvelope(String), + UnknownBinding, + ProvenanceRejected, +} + +impl Display for CoreError { + fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result { + match self { + Self::Configuration(value) => write!(formatter, "core config error: {value}"), + Self::Decode(value) => write!(formatter, "core decode error: {value}"), + Self::RawEnvelope(value) => write!(formatter, "raw envelope error: {value}"), + Self::UnknownBinding => write!(formatter, "raw envelope has no approved binding"), + Self::ProvenanceRejected => write!(formatter, "test provenance is not allowed"), + } + } +} + +impl std::error::Error for CoreError {} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ProcessBatch { + pub canonical: Vec, + pub quarantines: Vec, + pub duplicates: usize, + pub filtered: usize, +} + +pub struct RealtimeCore { + config: RealtimeCoreConfig, + bindings: BTreeMap, + partition_sequences: BTreeMap, + ordering: OrderingTracker, + seen_ids: HashSet>, + seen_order: VecDeque>, +} + +impl RealtimeCore { + pub fn new(config: RealtimeCoreConfig) -> Result { + config.validate()?; + let bindings = config + .bindings + .iter() + .cloned() + .map(|binding| (binding.key(), binding)) + .collect(); + Ok(Self { + config, + bindings, + partition_sequences: BTreeMap::new(), + ordering: OrderingTracker::new(4096), + seen_ids: HashSet::new(), + seen_order: VecDeque::new(), + }) + } + + pub fn process_bytes( + &mut self, + raw_bytes: &[u8], + normalized_at_ns: i64, + ) -> Result { + if normalized_at_ns <= 0 { + return Err(CoreError::Decode( + "normalized_at_ns must be positive".into(), + )); + } + let raw = RawProviderEnvelope::decode(raw_bytes) + .map_err(|error| CoreError::Decode(error.to_string()))?; + self.process(raw, normalized_at_ns) + } + + pub fn process( + &mut self, + raw: RawProviderEnvelope, + processing_at_ns: i64, + ) -> Result { + self.process_internal(raw, processing_at_ns, None) + } + + pub fn process_at_transport_offset( + &mut self, + raw: RawProviderEnvelope, + processing_at_ns: i64, + transport_offset: u64, + ) -> Result { + self.process_internal(raw, processing_at_ns, Some(transport_offset)) + } + + fn process_internal( + &mut self, + raw: RawProviderEnvelope, + processing_at_ns: i64, + transport_offset: Option, + ) -> Result { + if processing_at_ns <= 0 { + return Err(CoreError::Decode( + "processing_at_ns must be positive".into(), + )); + } + validate_raw(&raw).map_err(|error| CoreError::RawEnvelope(error.to_string()))?; + let materialized_at_ns = raw.received_at_ns; + if raw.test_provenance && !self.config.allow_test_provenance { + return Err(CoreError::ProvenanceRejected); + } + let key = binding_key( + &raw.provider, + &raw.venue, + &raw.market, + &raw.product_type, + &raw.native_symbol, + &raw.native_channel, + ); + let binding = self + .bindings + .get(&key) + .cloned() + .ok_or(CoreError::UnknownBinding)?; + if raw.instrument_catalog_revision != binding.instrument_catalog_revision { + return Ok(self.quarantine( + &raw, + QuarantineReason::UnknownInstrument, + "instrument catalog revision mismatch", + materialized_at_ns, + )); + } + let payload: Value = serde_json::from_slice(&raw.raw_frame_bytes) + .map_err(|error| CoreError::Decode(error.to_string()))?; + let frames = expand_frames(&binding, payload).map_err(CoreError::Decode)?; + if transport_offset.is_some() && frames.len() as u64 >= TRANSPORT_SEQUENCE_STRIDE { + return Err(CoreError::Configuration( + "expanded provider frame exceeds transport sequence stride".into(), + )); + } + let mut staged_partition_sequences = BTreeMap::new(); + let mut staged_ordering: BTreeMap = BTreeMap::new(); + let mut staged_seen_ids = HashSet::new(); + let mut staged_seen_order = Vec::new(); + let mut batch = ProcessBatch { + canonical: Vec::with_capacity(frames.len()), + quarantines: vec![], + duplicates: 0, + filtered: 0, + }; + let mut failure: Option<(QuarantineReason, &'static str)> = None; + for (row_index, (provider_kind, frame)) in frames.into_iter().enumerate() { + let partition_key = format!( + "{}/{}/{}", + binding.instrument_uid, provider_kind, binding.source_id + ); + let partition_sequence = if let Some(offset) = transport_offset { + offset + .checked_mul(TRANSPORT_SEQUENCE_STRIDE) + .and_then(|base| base.checked_add(row_index as u64 + 1)) + .ok_or_else(|| { + CoreError::Configuration( + "transport-derived partition sequence overflow".into(), + ) + })? + } else { + staged_partition_sequences + .get(&partition_key) + .or_else(|| self.partition_sequences.get(&partition_key)) + .copied() + .unwrap_or(0_u64) + .saturating_add(1) + }; + let fixture = TradeFixture { + provider_kind, + context: TradeContext { + instrument_uid: binding.instrument_uid.clone(), + instrument_id: binding.instrument_id.clone(), + instrument_revision: binding.instrument_revision, + venue: binding.venue.clone(), + market: binding.market.clone(), + product_type: binding.product_type.clone(), + native_symbol: binding.native_symbol.clone(), + provider: binding.provider.clone(), + source_id: binding.source_id.clone(), + lease_epoch: raw.lease_epoch, + received_at_ns: raw.received_at_ns, + normalized_at_ns: materialized_at_ns, + published_at_ns: materialized_at_ns, + partition_sequence, + normalizer_version: binding.normalizer_version.clone(), + adapter_version: raw.adapter_version.clone(), + config_revision: raw.config_revision, + correlation_id: raw.correlation_id.clone(), + source_session_id: raw.source_session_id.clone(), + connection_generation: raw.connection_generation, + authority_revision: raw.authority_revision, + partition_plan_epoch: raw.partition_plan_epoch, + raw_capture_id: raw.capture_id.clone(), + raw_frame_sha256: raw.raw_frame_sha256.clone(), + source_role: binding.source_role.clone(), + }, + raw: frame, + }; + let canonical = match canonicalize_trade(&fixture) { + Ok(value) => value, + Err(_) => { + failure = Some((QuarantineReason::SemanticInvalid, "canonicalization failed")); + break; + } + }; + if binding.require_final_bar { + match canonical.payload.as_ref() { + Some(event_envelope::Payload::Bar(bar)) + if bar.is_final + && matches!( + BarLifecycle::try_from(bar.lifecycle), + Ok(BarLifecycle::Final | BarLifecycle::Revised) + ) => {} + Some(event_envelope::Payload::Bar(_)) => { + batch.filtered += 1; + continue; + } + _ => { + failure = Some(( + QuarantineReason::SemanticInvalid, + "final BAR policy applied to a non-BAR payload", + )); + break; + } + } + } + if self.seen_ids.contains(&canonical.event_id) + || staged_seen_ids.contains(&canonical.event_id) + { + batch.duplicates += 1; + continue; + } + let sequence = match binding.sequence_policy { + SequencePolicy::None => partition_sequence, + SequencePolicy::Monotonic | SequencePolicy::Contiguous => { + match canonical.source_sequence.parse::() { + Ok(value) => value, + Err(_) => { + failure = Some(( + QuarantineReason::SemanticInvalid, + "native sequence is not numeric", + )); + break; + } + } + } + }; + let ordering_stage = staged_ordering + .entry(partition_key.clone()) + .or_insert_with(|| self.ordering.stage(&partition_key)); + match self.ordering.observe_staged( + ordering_stage, + &raw.source_session_id, + raw.connection_generation, + sequence, + canonical.event_id.clone(), + binding.sequence_policy, + ) { + SequenceDecision::Duplicate => { + batch.duplicates += 1; + continue; + } + SequenceDecision::Gap { .. } => { + failure = Some(( + QuarantineReason::SequenceGap, + "native sequence gap requires recovery", + )); + break; + } + SequenceDecision::OutOfOrder => { + failure = Some(( + QuarantineReason::SemanticInvalid, + "native sequence is out of order", + )); + break; + } + SequenceDecision::StaleSession => { + failure = Some(( + QuarantineReason::StaleGeneration, + "connection generation is stale", + )); + break; + } + SequenceDecision::Accepted | SequenceDecision::SessionStarted => {} + } + staged_partition_sequences.insert(partition_key, partition_sequence); + staged_seen_ids.insert(canonical.event_id.clone()); + staged_seen_order.push(canonical.event_id.clone()); + batch.canonical.push(canonical_record( + &self.config.canonical_stream, + canonical, + materialized_at_ns, + )); + } + if let Some((reason, summary)) = failure { + return Ok(self.quarantine(&raw, reason, summary, materialized_at_ns)); + } + self.partition_sequences.extend(staged_partition_sequences); + for stage in staged_ordering.into_values() { + self.ordering.commit_stage(stage); + } + for event_id in staged_seen_order { + self.remember(event_id); + } + Ok(batch) + } + + fn remember(&mut self, event_id: Vec) { + self.seen_ids.insert(event_id.clone()); + self.seen_order.push_back(event_id); + while self.seen_order.len() > self.config.dedup_capacity { + if let Some(expired) = self.seen_order.pop_front() { + self.seen_ids.remove(&expired); + } + } + } + + fn quarantine( + &self, + raw: &RawProviderEnvelope, + reason: QuarantineReason, + summary: &str, + now_ns: i64, + ) -> ProcessBatch { + let mut evidence = Sha256::new(); + evidence.update(raw.encode_to_vec()); + evidence.update((reason as i32).to_be_bytes()); + let record = QuarantineRecord { + raw: Some(raw.clone()), + reason: reason as i32, + safe_summary: summary.chars().take(200).collect(), + quarantined_at_ns: now_ns, + retry_count: 0, + evidence_sha256: evidence.finalize().to_vec(), + }; + ProcessBatch { + canonical: vec![], + quarantines: vec![DurableRecord { + stream: self.config.quarantine_stream.clone(), + partition_key: format!( + "{}/{}/{}", + raw.provider, raw.native_symbol, raw.source_session_id + ), + event_id: raw.capture_id.clone(), + payload: record.encode_to_vec(), + accepted_at_ns: now_ns, + }], + duplicates: 0, + filtered: 0, + } + } +} + +fn binding_key( + provider: &str, + venue: &str, + market: &str, + product_type: &str, + native_symbol: &str, + native_channel: &str, +) -> String { + [ + provider, + venue, + market, + product_type, + native_symbol, + native_channel, + ] + .join("|") +} + +fn expand_frames(binding: &CoreBinding, payload: Value) -> Result, String> { + if binding.venue == "OKX" && payload.get("arg").is_some() { + return expand_data_frame(&payload).map(|frames| { + frames + .into_iter() + .map(|frame| (frame.provider_kind.to_owned(), frame.raw)) + .collect() + }); + } + let frame = if binding.venue == "BINANCE" { + payload.get("data").cloned().unwrap_or(payload) + } else { + payload + }; + Ok(vec![(binding.provider_kind.clone(), frame)]) +} + +fn canonical_record(stream: &str, envelope: EventEnvelope, now_ns: i64) -> DurableRecord { + let feed = match envelope.payload.as_ref() { + Some(event_envelope::Payload::Trade(_)) => "trade", + Some(event_envelope::Payload::Quote(_)) => "quote", + Some(event_envelope::Payload::Bar(_)) => "bar", + Some(event_envelope::Payload::BookSnapshot(_)) => "book_snapshot", + Some(event_envelope::Payload::BookDelta(_)) => "book_delta", + Some(event_envelope::Payload::FundingRate(_)) => "funding_rate", + Some(event_envelope::Payload::OpenInterest(_)) => "open_interest", + Some(event_envelope::Payload::MarkIndexPrice(_)) => "mark_index_price", + Some(event_envelope::Payload::Ticker(_)) => "ticker", + Some(event_envelope::Payload::FeedState(_)) => "feed_state", + Some(event_envelope::Payload::QualityEvent(_)) => "quality_event", + None => "unknown", + }; + DurableRecord { + stream: stream.into(), + partition_key: format!( + "{}/{}/{}", + envelope.instrument_uid, feed, envelope.source_id + ), + event_id: envelope.event_id.clone(), + payload: envelope.encode_to_vec(), + accepted_at_ns: now_ns, + } +} + +#[cfg(test)] +mod tests { + use super::{CoreBinding, CoreError, RealtimeCore, RealtimeCoreConfig}; + use prost::Message; + use qdl_contracts::qdl::common::v1::{QuantityUnit, SourceRole}; + use qdl_contracts::qdl::marketdata::v2::{event_envelope, EventEnvelope, TradeIdentityKind}; + use qdl_contracts::qdl::provider::v1::{ + CaptureBoundary, QuarantineRecord, RawProviderEnvelope, TransportCompression, + TransportProtocol, + }; + use qdl_venue_core::ordering::SequencePolicy; + use sha2::{Digest, Sha256}; + + type BindingSpec<'a> = ( + &'a str, + &'a str, + &'a str, + &'a str, + &'a str, + &'a str, + &'a str, + &'a str, + SequencePolicy, + ); + + fn binding( + ( + provider, + venue, + market, + product, + symbol, + channel, + provider_kind, + source_role, + policy, + ): BindingSpec<'_>, + ) -> CoreBinding { + CoreBinding { + provider: provider.into(), + venue: venue.into(), + market: market.into(), + product_type: product.into(), + native_symbol: symbol.into(), + native_channel: channel.into(), + provider_kind: provider_kind.into(), + instrument_uid: format!("uid-{venue}-{market}-{symbol}"), + instrument_id: format!("{venue}.{market}.{product}.{symbol}"), + instrument_revision: 1, + instrument_catalog_revision: 3, + source_id: format!("source-{provider}-{channel}"), + source_role: source_role.into(), + normalizer_version: "qdl-rust-core/2.0.0".into(), + require_final_bar: provider_kind.ends_with("_bar"), + sequence_policy: policy, + } + } + + fn core(binding: CoreBinding, allow_test: bool) -> RealtimeCore { + RealtimeCore::new(RealtimeCoreConfig { + canonical_stream: "qdl.test.canonical.v2".into(), + quarantine_stream: "qdl.test.quarantine.v1".into(), + allow_test_provenance: allow_test, + dedup_capacity: 16, + bindings: vec![binding], + }) + .unwrap() + } + + fn raw(binding: &CoreBinding, frame: &[u8], generation: u64) -> RawProviderEnvelope { + RawProviderEnvelope { + raw_schema_name: "qdl.provider.raw".into(), + raw_schema_major: 1, + raw_schema_minor: 0, + capture_id: Sha256::digest([frame, &generation.to_be_bytes()].concat())[..16].to_vec(), + provider: binding.provider.clone(), + venue: binding.venue.clone(), + market: binding.market.clone(), + product_type: binding.product_type.clone(), + native_symbol: binding.native_symbol.clone(), + native_channel: binding.native_channel.clone(), + subscription_id: "subscription-1".into(), + source_session_id: format!("session-{generation}"), + connection_generation: generation, + lease_epoch: 7, + authority_revision: 1, + partition_plan_epoch: 1, + received_at_ns: 1_786_352_400_123_456_000, + transport_protocol: TransportProtocol::Websocket as i32, + transport_compression: TransportCompression::None as i32, + capture_boundary: CaptureBoundary::PostDecompression as i32, + raw_frame_bytes: frame.to_vec(), + raw_frame_sha256: Sha256::digest(frame).to_vec(), + adapter_version: format!("{}/2.0.0", binding.provider.to_ascii_lowercase()), + config_revision: 1, + instrument_catalog_revision: binding.instrument_catalog_revision, + correlation_id: "phase-a-core-test".into(), + test_provenance: true, + } + } + + #[test] + fn exact_duplicate_across_reconnect_is_not_republished() { + let binding = binding(( + "BINANCE_DIRECT", + "BINANCE", + "USDM", + "PERPETUAL", + "BTCUSDT", + "trade", + "binance_usdm_trade", + "PRIMARY", + SequencePolicy::Monotonic, + )); + let frame = br#"{"s":"BTCUSDT","t":10,"p":"60000.1","q":"0.01","T":3,"m":false}"#; + let mut core = core(binding.clone(), true); + let first = core.process(raw(&binding, frame, 1), 10).unwrap(); + assert_eq!(first.canonical.len(), 1); + let repeated = core.process(raw(&binding, frame, 2), 11).unwrap(); + assert_eq!(repeated.canonical.len(), 0); + assert_eq!(repeated.duplicates, 1); + } + + #[test] + fn transport_replay_is_byte_deterministic_across_fresh_cores() { + let binding = binding(( + "BINANCE_DIRECT", + "BINANCE", + "USDM", + "PERPETUAL", + "BTCUSDT", + "trade", + "binance_usdm_trade", + "PRIMARY", + SequencePolicy::Monotonic, + )); + let frame = br#"{"s":"BTCUSDT","t":10,"p":"60000.1","q":"0.01","T":3,"m":false}"#; + let captured = raw(&binding, frame, 1); + let mut first_core = core(binding.clone(), true); + let first = first_core + .process_at_transport_offset(captured.clone(), 10, 42) + .unwrap(); + let mut recovered_core = core(binding, true); + let replay = recovered_core + .process_at_transport_offset(captured.clone(), 999, 42) + .unwrap(); + + assert_eq!(first, replay); + let envelope = EventEnvelope::decode(first.canonical[0].payload.as_slice()).unwrap(); + assert_eq!(envelope.received_at_ns, captured.received_at_ns); + assert_eq!(envelope.normalized_at_ns, captured.received_at_ns); + assert_eq!(envelope.published_at_ns, captured.received_at_ns); + assert_eq!(envelope.partition_sequence, 42_000_001); + assert_eq!(first.canonical[0].accepted_at_ns, captured.received_at_ns); + } + + #[test] + fn transport_offset_keeps_restart_and_expanded_row_sequences_monotonic() { + let binding = binding(( + "OKX_DIRECT", + "OKX", + "SWAP", + "PERPETUAL", + "BTC-USDT-SWAP", + "trades", + "okx_trade", + "PRIMARY", + SequencePolicy::Monotonic, + )); + let first_frame = br#"{"arg":{"channel":"trades","instId":"BTC-USDT-SWAP"},"data":[{"instId":"BTC-USDT-SWAP","tradeId":"1","px":"1","sz":"2","side":"buy","ts":"3"},{"instId":"BTC-USDT-SWAP","tradeId":"2","px":"2","sz":"2","side":"buy","ts":"4"}]}"#; + let next_frame = br#"{"arg":{"channel":"trades","instId":"BTC-USDT-SWAP"},"data":[{"instId":"BTC-USDT-SWAP","tradeId":"3","px":"3","sz":"2","side":"buy","ts":"5"}]}"#; + let mut first_core = core(binding.clone(), true); + let first = first_core + .process_at_transport_offset(raw(&binding, first_frame, 1), 10, 42) + .unwrap(); + let first_sequences: Vec = first + .canonical + .iter() + .map(|record| { + EventEnvelope::decode(record.payload.as_slice()) + .unwrap() + .partition_sequence + }) + .collect(); + assert_eq!(first_sequences, vec![42_000_001, 42_000_002]); + + let mut restarted_core = core(binding.clone(), true); + let next = restarted_core + .process_at_transport_offset(raw(&binding, next_frame, 1), 999, 43) + .unwrap(); + let next_envelope = EventEnvelope::decode(next.canonical[0].payload.as_slice()).unwrap(); + assert_eq!(next_envelope.partition_sequence, 43_000_001); + assert!(next_envelope.partition_sequence > first_sequences[1]); + + let mut overflow_core = core(binding.clone(), true); + let overflow = + overflow_core.process_at_transport_offset(raw(&binding, next_frame, 1), 1, u64::MAX); + assert!(matches!(overflow, Err(CoreError::Configuration(_)))); + } + + #[test] + fn quarantine_replay_is_deterministic_across_processing_clocks() { + let binding = binding(( + "BINANCE_DIRECT", + "BINANCE", + "USDM", + "PERPETUAL", + "BTCUSDT", + "trade", + "binance_usdm_trade", + "PRIMARY", + SequencePolicy::Monotonic, + )); + let frame = br#"{"s":"BTCUSDT","t":10,"p":"1","q":"1","T":3,"m":false}"#; + let mut captured = raw(&binding, frame, 1); + captured.instrument_catalog_revision += 1; + let mut first_core = core(binding.clone(), true); + let first = first_core + .process_at_transport_offset(captured.clone(), 10, 7) + .unwrap(); + let mut recovered_core = core(binding, true); + let replay = recovered_core + .process_at_transport_offset(captured, 999, 7) + .unwrap(); + assert_eq!(first, replay); + assert_eq!(first.quarantines.len(), 1); + } + + #[test] + fn final_only_okx_bar_filters_provisional_and_publishes_confirmed() { + let binding = binding(( + "OKX_DIRECT", + "OKX", + "SWAP", + "PERPETUAL", + "BTC-USDT-SWAP", + "candle1m", + "okx_bar", + "PRIMARY", + SequencePolicy::None, + )); + assert!(binding.require_final_bar); + let frame = |confirm: u8| { + format!( + r#"{{"arg":{{"channel":"candle1m","instId":"BTC-USDT-SWAP"}},"data":[["1786352340000","61200.00","61240.00","61190.00","61234.10","12.500","12.500","765200.00","{confirm}"]]}}"# + ) + .into_bytes() + }; + let mut core = core(binding.clone(), true); + let provisional = core.process(raw(&binding, &frame(0), 1), 10).unwrap(); + assert!(provisional.canonical.is_empty()); + assert!(provisional.quarantines.is_empty()); + assert_eq!(provisional.filtered, 1); + + let final_bar = core.process(raw(&binding, &frame(1), 1), 11).unwrap(); + assert_eq!(final_bar.canonical.len(), 1); + assert!(final_bar.quarantines.is_empty()); + assert_eq!(final_bar.filtered, 0); + let envelope = EventEnvelope::decode(final_bar.canonical[0].payload.as_slice()).unwrap(); + let event_envelope::Payload::Bar(bar) = envelope.payload.unwrap() else { + panic!("OKX final candle must be BAR") + }; + assert!(bar.is_final); + } + + #[test] + fn stale_generation_and_contiguous_gap_are_quarantined() { + let binding = binding(( + "OKX_DIRECT", + "OKX", + "SWAP", + "PERPETUAL", + "BTC-USDT-SWAP", + "bbo-tbt", + "okx_bbo", + "PRIMARY", + SequencePolicy::Contiguous, + )); + let frame = |sequence: u64| { + format!( + r#"{{"arg":{{"channel":"bbo-tbt","instId":"BTC-USDT-SWAP"}},"data":[{{"bids":[["1","2","0","1"]],"asks":[["2","3","0","1"]],"seqId":{sequence},"ts":"3"}}]}}"# + ) + .into_bytes() + }; + let mut core = core(binding.clone(), true); + assert_eq!( + core.process(raw(&binding, &frame(10), 2), 10) + .unwrap() + .canonical + .len(), + 1 + ); + let gap = core.process(raw(&binding, &frame(12), 2), 11).unwrap(); + assert_eq!(gap.quarantines.len(), 1); + let decoded = QuarantineRecord::decode(gap.quarantines[0].payload.as_slice()).unwrap(); + assert_eq!(decoded.reason, 7); + let stale = core.process(raw(&binding, &frame(11), 1), 12).unwrap(); + assert_eq!(stale.quarantines.len(), 1); + let decoded = QuarantineRecord::decode(stale.quarantines[0].payload.as_slice()).unwrap(); + assert_eq!(decoded.reason, 5); + } + + #[test] + fn dnse_and_vnstock_keep_units_identity_and_source_role() { + let dnse = binding(( + "DNSE_DIRECT", + "HNX", + "VN_DERIVATIVES", + "FUTURE", + "VN30F1M", + "trades", + "dnse_trade", + "PRIMARY", + SequencePolicy::None, + )); + let mut dnse_core = core(dnse.clone(), true); + let event = dnse_core + .process( + raw( + &dnse, + br#"{"symbol":"VN30F1M","price":"1820.7","quantity":"12"}"#, + 1, + ), + 10, + ) + .unwrap(); + let event = EventEnvelope::decode(event.canonical[0].payload.as_slice()).unwrap(); + let event_envelope::Payload::Trade(trade) = event.payload.unwrap() else { + panic!("DNSE canonical payload must be trade") + }; + assert_eq!(trade.quantity_unit, QuantityUnit::Contract as i32); + assert_eq!( + trade.identity_kind, + TradeIdentityKind::DerivedRawCapture as i32 + ); + + let vnstock = binding(( + "VNSTOCK", + "HOSE", + "EQUITIES", + "COMMON_STOCK", + "FPT", + "ohlcv/1m", + "vnstock_bar", + "SECONDARY", + SequencePolicy::None, + )); + let mut vnstock_core = core(vnstock.clone(), true); + let event = vnstock_core + .process( + raw( + &vnstock, + br#"{"symbol":"FPT","interval":"1m","open_time_ms":1,"close_time_ms":2,"o":"1","h":"2","l":"1","c":"2","v":"100","is_final":true,"trade_count_available":false,"revision":0}"#, + 1, + ), + 10, + ) + .unwrap(); + let event = EventEnvelope::decode(event.canonical[0].payload.as_slice()).unwrap(); + assert_eq!(event.source_role, SourceRole::Secondary as i32); + let event_envelope::Payload::Bar(bar) = event.payload.unwrap() else { + panic!("VNstock canonical payload must be bar") + }; + assert_eq!(bar.volume_unit, QuantityUnit::Share as i32); + } + + #[test] + fn dnse_rest_and_closed_bar_callback_are_one_deterministic_bar() { + let binding = binding(( + "DNSE_DIRECT", + "HNX", + "VN_DERIVATIVES", + "FUTURE", + "VN30F1M", + "ohlcv/1m", + "dnse_bar", + "PRIMARY", + SequencePolicy::None, + )); + let frame = br#"{"symbol":"VN30F1M","interval":"1m","open_time_ms":1786352340000,"close_time_ms":1786352399999,"o":"1820.7","h":"1821.2","l":"1820.2","c":"1820.9","v":"12","is_final":true,"trade_count_available":false,"revision":0}"#; + let mut rest = raw(&binding, frame, 1); + rest.transport_protocol = TransportProtocol::Http as i32; + rest.capture_boundary = CaptureBoundary::PostDecompression as i32; + let mut core = core(binding.clone(), true); + let first = core.process(rest, 10).unwrap(); + assert_eq!(first.canonical.len(), 1); + let event = EventEnvelope::decode(first.canonical[0].payload.as_slice()).unwrap(); + let event_envelope::Payload::Bar(bar) = event.payload.unwrap() else { + panic!("DNSE canonical payload must be bar") + }; + assert_eq!(bar.volume_unit, QuantityUnit::Contract as i32); + assert_eq!( + bar.lifecycle, + qdl_contracts::qdl::marketdata::v2::BarLifecycle::Final as i32 + ); + + let mut websocket = raw(&binding, frame, 2); + websocket.transport_protocol = TransportProtocol::SdkCallback as i32; + websocket.capture_boundary = CaptureBoundary::SdkDelivery as i32; + let repeated = core.process(websocket, 11).unwrap(); + assert!(repeated.canonical.is_empty()); + assert_eq!(repeated.duplicates, 1); + assert!(repeated.quarantines.is_empty()); + } + + #[test] + fn aggregated_provider_frame_is_atomic_on_row_failure() { + let binding = binding(( + "OKX_DIRECT", + "OKX", + "SWAP", + "PERPETUAL", + "BTC-USDT-SWAP", + "trades", + "okx_trade", + "PRIMARY", + SequencePolicy::Monotonic, + )); + let invalid = br#"{"arg":{"channel":"trades","instId":"BTC-USDT-SWAP"},"data":[{"instId":"BTC-USDT-SWAP","tradeId":"1","px":"1","sz":"2","side":"buy","ts":"3"},{"instId":"BTC-USDT-SWAP","tradeId":"2","sz":"2","side":"buy","ts":"4"}]}"#; + let mut core = core(binding.clone(), true); + let rejected = core.process(raw(&binding, invalid, 1), 10).unwrap(); + assert!(rejected.canonical.is_empty()); + assert_eq!(rejected.quarantines.len(), 1); + + let corrected = br#"{"arg":{"channel":"trades","instId":"BTC-USDT-SWAP"},"data":[{"instId":"BTC-USDT-SWAP","tradeId":"1","px":"1","sz":"2","side":"buy","ts":"3"},{"instId":"BTC-USDT-SWAP","tradeId":"2","px":"2","sz":"2","side":"buy","ts":"4"}]}"#; + let accepted = core.process(raw(&binding, corrected, 1), 11).unwrap(); + assert_eq!(accepted.canonical.len(), 2); + assert!(accepted.quarantines.is_empty()); + assert_eq!(accepted.duplicates, 0); + } + + #[test] + fn unknown_binding_and_forbidden_test_provenance_fail_closed() { + let binding = binding(( + "BINANCE_DIRECT", + "BINANCE", + "USDM", + "PERPETUAL", + "BTCUSDT", + "trade", + "binance_usdm_trade", + "PRIMARY", + SequencePolicy::Monotonic, + )); + let frame = br#"{"s":"BTCUSDT","t":10,"p":"1","q":"1","T":3,"m":false}"#; + let mut strict = core(binding.clone(), false); + assert_eq!( + strict.process(raw(&binding, frame, 1), 10), + Err(CoreError::ProvenanceRejected) + ); + let mut unknown = raw(&binding, frame, 1); + unknown.native_channel = "unknown".into(); + let mut permissive = core(binding, true); + assert_eq!( + permissive.process(unknown, 10), + Err(CoreError::UnknownBinding) + ); + } +} diff --git a/rust/qdl-venue-core/Cargo.toml b/rust/qdl-venue-core/Cargo.toml index fc6e26f..acc4925 100644 --- a/rust/qdl-venue-core/Cargo.toml +++ b/rust/qdl-venue-core/Cargo.toml @@ -6,6 +6,7 @@ license.workspace = true rust-version.workspace = true [dependencies] +hex.workspace = true serde.workspace = true serde_json.workspace = true sha2.workspace = true diff --git a/rust/qdl-venue-core/src/authority.rs b/rust/qdl-venue-core/src/authority.rs index a354195..0852c1f 100644 --- a/rust/qdl-venue-core/src/authority.rs +++ b/rust/qdl-venue-core/src/authority.rs @@ -9,12 +9,14 @@ pub enum AuthorityMode { RustCanary, } -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[derive(Clone, Copy, Debug, Deserialize, Eq, Hash, PartialEq, Serialize)] #[serde(rename_all = "SCREAMING_SNAKE_CASE")] pub enum SinkTarget { ShadowRaw, ShadowCanonical, + ShadowQuarantine, CanaryCanonical, + PrimaryCanonical, PublicV2, LegacyV1, } @@ -133,11 +135,14 @@ impl AuthorityFence { let allowed = match current.mode { AuthorityMode::RustShadow => matches!( context.target, - SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical + SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical | SinkTarget::ShadowQuarantine ), AuthorityMode::RustCanary => matches!( context.target, - SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical | SinkTarget::CanaryCanonical + SinkTarget::ShadowRaw + | SinkTarget::ShadowCanonical + | SinkTarget::ShadowQuarantine + | SinkTarget::CanaryCanonical ), }; if !allowed { @@ -151,6 +156,756 @@ impl AuthorityFence { } } +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase9AuthorityState { + RustShadow, + RustCanary, + Blocked, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase9AuthorityRecord { + pub schema: String, + pub slice_id: String, + pub state: Phase9AuthorityState, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub candidate_digest: String, + pub prerequisite_bundle_id: Option, + pub start_watermark: u64, + pub approved_by: Option, + pub approved_at_ns: Option, + pub hold_until_ns: Option, + pub public_write_allowed: bool, + pub legacy_write_allowed: bool, +} + +impl Phase9AuthorityRecord { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.authority-record.v2" + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || !valid_digest(&self.candidate_digest, false) + { + return Err("Phase 9 authority identity/epoch is invalid".into()); + } + if self.public_write_allowed || self.legacy_write_allowed { + return Err("Phase 9.1 cannot enable public or legacy writes".into()); + } + match self.state { + Phase9AuthorityState::RustCanary => { + let bundle = self + .prerequisite_bundle_id + .as_deref() + .ok_or_else(|| "canary prerequisite bundle is required".to_owned())?; + let approved_by = self + .approved_by + .as_deref() + .ok_or_else(|| "canary approver is required".to_owned())?; + let approved_at = self + .approved_at_ns + .ok_or_else(|| "canary approval time is required".to_owned())?; + let hold_until = self + .hold_until_ns + .ok_or_else(|| "canary hold time is required".to_owned())?; + if !valid_uuid(bundle) + || approved_by.trim().is_empty() + || approved_at <= 0 + || hold_until <= approved_at + { + return Err("canary approval/bundle/hold is invalid".into()); + } + } + Phase9AuthorityState::RustShadow | Phase9AuthorityState::Blocked => { + if self.prerequisite_bundle_id.is_some() + || self.approved_by.is_some() + || self.approved_at_ns.is_some() + || self.hold_until_ns.is_some() + { + return Err("non-canary authority cannot carry an approval bundle".into()); + } + } + } + Ok(()) + } +} + +fn valid_uuid(value: &str) -> bool { + let widths = [8, 4, 4, 4, 12]; + let parts: Vec<&str> = value.split('-').collect(); + parts.len() == widths.len() + && parts.iter().zip(widths).all(|(part, width)| { + part.len() == width && part.bytes().all(|byte| byte.is_ascii_hexdigit()) + }) +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase9PublicationContext { + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub shard_id: String, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub source_watermark: u64, + pub target: SinkTarget, +} + +#[derive(Default)] +pub struct Phase9AuthorityFence { + current: Option, + committed_watermarks: HashMap, +} + +impl Phase9AuthorityFence { + pub fn apply(&mut self, record: Phase9AuthorityRecord) -> Result<(), String> { + record.validate()?; + if let Some(current) = &self.current { + if record.slice_id != current.slice_id + || record.candidate_digest != current.candidate_digest + || record.partition_plan_epoch != current.partition_plan_epoch + { + return Err( + "Phase 9 authority scope/candidate/plan cannot change inside one fence".into(), + ); + } + if record.authority_revision < current.authority_revision { + return Err("stale Phase 9 authority revision".into()); + } + if record.authority_revision == current.authority_revision { + return if record == *current { + Ok(()) + } else { + Err("conflicting Phase 9 authority record at the same revision".into()) + }; + } + if record.lease_epoch < current.lease_epoch + || (record.owner_id != current.owner_id + && record.lease_epoch <= current.lease_epoch) + { + return Err("stale or conflicting Phase 9 owner lease".into()); + } + let transition_allowed = match current.state { + Phase9AuthorityState::RustShadow => matches!( + record.state, + Phase9AuthorityState::RustShadow + | Phase9AuthorityState::RustCanary + | Phase9AuthorityState::Blocked + ), + Phase9AuthorityState::RustCanary => matches!( + record.state, + Phase9AuthorityState::RustCanary + | Phase9AuthorityState::RustShadow + | Phase9AuthorityState::Blocked + ), + Phase9AuthorityState::Blocked => matches!( + record.state, + Phase9AuthorityState::Blocked | Phase9AuthorityState::RustShadow + ), + }; + if !transition_allowed { + return Err("Phase 9 authority transition is not permitted".into()); + } + } + self.current = Some(record); + Ok(()) + } + + pub fn permits(&self, context: &Phase9PublicationContext, now_ns: i64) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9 authority record is not loaded".to_owned())?; + if now_ns <= 0 + || context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + || context.shard_id.trim().is_empty() + { + return Err("publication identity does not match current Phase 9 authority".into()); + } + if matches!(context.target, SinkTarget::PublicV2 | SinkTarget::LegacyV1) { + return Err("Phase 9.1 public and legacy targets are fenced".into()); + } + let target_allowed = match current.state { + Phase9AuthorityState::RustShadow => matches!( + context.target, + SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical | SinkTarget::ShadowQuarantine + ), + Phase9AuthorityState::RustCanary => { + let approved_at = current.approved_at_ns.expect("validated canary approval"); + let hold_until = current.hold_until_ns.expect("validated canary hold"); + if now_ns < approved_at || now_ns >= hold_until { + return Err("Phase 9 canary approval window is not active".into()); + } + matches!( + context.target, + SinkTarget::ShadowRaw + | SinkTarget::ShadowCanonical + | SinkTarget::CanaryCanonical + ) + } + Phase9AuthorityState::Blocked => false, + }; + if !target_allowed { + return Err("sink target is not permitted by current Phase 9 authority".into()); + } + let committed = self + .committed_watermarks + .get(&context.shard_id) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark <= committed { + return Err("source watermark is stale or already committed".into()); + } + Ok(()) + } + + pub fn commit(&mut self, context: &Phase9PublicationContext) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9 authority record is not loaded".to_owned())?; + if context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + { + return Err("authority changed before publication commit".into()); + } + let committed = self + .committed_watermarks + .get(&context.shard_id) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark <= committed { + return Err("publication watermark commit regressed".into()); + } + self.committed_watermarks + .insert(context.shard_id.clone(), context.source_watermark); + Ok(()) + } + + pub fn current(&self) -> Option<&Phase9AuthorityRecord> { + self.current.as_ref() + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase92AuthorityState { + RustCanary, + RustPrimary, + Blocked, + RollbackPending, + PythonPrimary, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase92HandoffDirection { + PythonToRust, + RustToPython, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92TerminalCheckpoint { + pub schema: String, + pub checkpoint_id: String, + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub source_session_id: String, + pub connection_generation: u64, + pub terminal_watermark: u64, + pub terminal_event_id: String, + pub terminal_payload_sha256: String, + pub candidate_digest: String, + pub committed_at_ns: i64, +} + +impl Phase92TerminalCheckpoint { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.terminal-owner-checkpoint.v1" + || !valid_uuid(&self.checkpoint_id) + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.source_session_id.trim().is_empty() + || self.terminal_event_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || self.connection_generation == 0 + || self.committed_at_ns <= 0 + || !valid_digest(&self.terminal_payload_sha256, false) + || !valid_digest(&self.candidate_digest, false) + { + return Err("Phase 9.2 terminal checkpoint is invalid".into()); + } + Ok(()) + } + + pub fn digest(&self) -> Result { + self.validate()?; + let payload = serde_json::to_vec(self) + .map_err(|error| format!("terminal checkpoint encoding failed: {error}"))?; + use sha2::Digest; + Ok(hex::encode(sha2::Sha256::digest(payload))) + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92AcceptedHandoff { + pub schema: String, + pub handoff_id: String, + pub direction: Phase92HandoffDirection, + pub checkpoint_digest: String, + pub slice_id: String, + pub old_owner_id: String, + pub new_owner_id: String, + pub expected_state: Phase92AuthorityState, + pub new_state: Phase92AuthorityState, + pub expected_authority_revision: u64, + pub new_authority_revision: u64, + pub expected_lease_epoch: u64, + pub new_lease_epoch: u64, + pub partition_plan_epoch: u64, + pub terminal_watermark: u64, + pub first_new_watermark: u64, + pub overlap_start_watermark: u64, + pub overlap_end_watermark: u64, + pub old_event_count: u64, + pub new_event_count: u64, + pub semantic_mismatches: u64, + pub open_gaps: u64, + pub candidate_digest: String, + pub prerequisite_bundle_id: String, + pub approved_by: String, + pub approved_at_ns: i64, + pub expires_at_ns: i64, +} + +impl Phase92AcceptedHandoff { + pub fn validate(&self, checkpoint: &Phase92TerminalCheckpoint) -> Result<(), String> { + checkpoint.validate()?; + if self.schema != "qdl.accepted-authority-handoff.v1" + || !valid_uuid(&self.handoff_id) + || !valid_uuid(&self.prerequisite_bundle_id) + || self.slice_id.trim().is_empty() + || self.old_owner_id.trim().is_empty() + || self.new_owner_id.trim().is_empty() + || self.old_owner_id == self.new_owner_id + || self.approved_by.trim().is_empty() + || self.partition_plan_epoch == 0 + || !valid_digest(&self.checkpoint_digest, false) + || !valid_digest(&self.candidate_digest, false) + || self.approved_at_ns <= 0 + || self.expires_at_ns <= self.approved_at_ns + { + return Err("Phase 9.2 handoff identity/approval is invalid".into()); + } + let states_match = match self.direction { + Phase92HandoffDirection::PythonToRust => { + self.expected_state == Phase92AuthorityState::RustCanary + && self.new_state == Phase92AuthorityState::RustPrimary + } + Phase92HandoffDirection::RustToPython => { + self.expected_state == Phase92AuthorityState::RollbackPending + && self.new_state == Phase92AuthorityState::PythonPrimary + } + }; + if !states_match + || self.new_authority_revision != self.expected_authority_revision + 1 + || self.new_lease_epoch <= self.expected_lease_epoch + || self.first_new_watermark != self.terminal_watermark + 1 + || self.overlap_start_watermark > self.overlap_end_watermark + || self.overlap_end_watermark != self.terminal_watermark + || self.old_event_count == 0 + || self.old_event_count != self.new_event_count + || self.semantic_mismatches != 0 + || self.open_gaps != 0 + { + return Err("Phase 9.2 handoff boundary/reconciliation is invalid".into()); + } + if self.checkpoint_digest != checkpoint.digest()? + || self.slice_id != checkpoint.slice_id + || self.old_owner_id != checkpoint.owner_id + || self.expected_authority_revision != checkpoint.authority_revision + || self.expected_lease_epoch != checkpoint.lease_epoch + || self.partition_plan_epoch != checkpoint.partition_plan_epoch + || self.terminal_watermark != checkpoint.terminal_watermark + || self.candidate_digest != checkpoint.candidate_digest + { + return Err("Phase 9.2 handoff does not bind the terminal checkpoint".into()); + } + Ok(()) + } + + pub fn digest(&self) -> Result { + let payload = serde_json::to_vec(self) + .map_err(|error| format!("accepted handoff encoding failed: {error}"))?; + use sha2::Digest; + Ok(hex::encode(sha2::Sha256::digest(payload))) + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92AuthorityRecord { + pub schema: String, + pub slice_id: String, + pub state: Phase92AuthorityState, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub candidate_digest: String, + pub prerequisite_bundle_id: Option, + pub start_watermark: u64, + pub terminal_watermark: Option, + pub previous_owner_id: Option, + pub handoff_digest: Option, + pub approved_by: Option, + pub approved_at_ns: Option, + pub hold_until_ns: Option, + pub public_write_allowed: bool, + pub legacy_write_allowed: bool, +} + +impl Phase92AuthorityRecord { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.authority-record.v3" + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || !valid_digest(&self.candidate_digest, false) + { + return Err("Phase 9.2 authority identity/epoch is invalid".into()); + } + let approval_valid = || { + self.approved_by + .as_deref() + .is_some_and(|value| !value.trim().is_empty()) + && self.approved_at_ns.is_some_and(|value| value > 0) + && self + .hold_until_ns + .is_some_and(|hold| self.approved_at_ns.is_some_and(|approved| hold > approved)) + }; + match self.state { + Phase92AuthorityState::RustCanary => { + if self.public_write_allowed + || self.legacy_write_allowed + || self.terminal_watermark.is_some() + || self.previous_owner_id.is_some() + || self.handoff_digest.is_some() + || !self + .prerequisite_bundle_id + .as_deref() + .is_some_and(valid_uuid) + || !approval_valid() + { + return Err("Phase 9.2 canary authority is invalid".into()); + } + } + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary => { + if !self.public_write_allowed + || !self.legacy_write_allowed + || self + .previous_owner_id + .as_deref() + .is_none_or(|value| value.trim().is_empty() || value == self.owner_id) + || self.terminal_watermark != Some(self.start_watermark) + || !self + .handoff_digest + .as_deref() + .is_some_and(|value| valid_digest(value, false)) + || !approval_valid() + { + return Err("Phase 9.2 primary authority/handoff is invalid".into()); + } + if self.state == Phase92AuthorityState::RustPrimary + && !self + .prerequisite_bundle_id + .as_deref() + .is_some_and(valid_uuid) + { + return Err("Rust primary requires a prerequisite bundle".into()); + } + } + Phase92AuthorityState::Blocked | Phase92AuthorityState::RollbackPending => { + if self.public_write_allowed || self.legacy_write_allowed { + return Err("blocked/rollback authority cannot write".into()); + } + } + } + Ok(()) + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92PublicationContext { + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub shard_id: String, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub source_watermark: u64, + pub target: SinkTarget, +} + +#[derive(Default)] +pub struct Phase92AuthorityFence { + current: Option, + committed_watermarks: HashMap<(String, SinkTarget), u64>, + recovery_required: bool, +} + +impl Phase92AuthorityFence { + pub fn apply(&mut self, record: Phase92AuthorityRecord) -> Result<(), String> { + record.validate()?; + if self.current.is_none() { + self.recovery_required = matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + self.current = Some(record); + return Ok(()); + } + if matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ) { + return Err("primary ownership transition requires accepted handoff".into()); + } + self.apply_transition(record) + } + + pub fn apply_handoff( + &mut self, + checkpoint: &Phase92TerminalCheckpoint, + handoff: &Phase92AcceptedHandoff, + record: Phase92AuthorityRecord, + now_ns: i64, + ) -> Result<(), String> { + handoff.validate(checkpoint)?; + record.validate()?; + if now_ns <= 0 || now_ns < handoff.approved_at_ns || now_ns >= handoff.expires_at_ns { + return Err("Phase 9.2 handoff approval window is not active".into()); + } + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if current.slice_id != handoff.slice_id + || current.owner_id != handoff.old_owner_id + || current.state != handoff.expected_state + || current.authority_revision != handoff.expected_authority_revision + || current.lease_epoch != handoff.expected_lease_epoch + || current.partition_plan_epoch != handoff.partition_plan_epoch + || current.candidate_digest != handoff.candidate_digest + || record.slice_id != handoff.slice_id + || record.owner_id != handoff.new_owner_id + || record.state != handoff.new_state + || record.authority_revision != handoff.new_authority_revision + || record.lease_epoch != handoff.new_lease_epoch + || record.partition_plan_epoch != handoff.partition_plan_epoch + || record.candidate_digest != handoff.candidate_digest + || record.start_watermark != handoff.terminal_watermark + || record.terminal_watermark != Some(handoff.terminal_watermark) + || record.previous_owner_id.as_deref() != Some(handoff.old_owner_id.as_str()) + || record.handoff_digest.as_deref() != Some(handoff.digest()?.as_str()) + { + return Err("Phase 9.2 authority CAS/handoff binding failed".into()); + } + self.apply_transition(record) + } + + fn apply_transition(&mut self, record: Phase92AuthorityRecord) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if record.slice_id != current.slice_id + || record.candidate_digest != current.candidate_digest + || record.partition_plan_epoch != current.partition_plan_epoch + || record.authority_revision != current.authority_revision + 1 + || record.lease_epoch < current.lease_epoch + || (record.owner_id != current.owner_id && record.lease_epoch <= current.lease_epoch) + { + return Err("Phase 9.2 authority compare-and-swap failed".into()); + } + let allowed = match current.state { + Phase92AuthorityState::RustCanary => matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::Blocked + ), + Phase92AuthorityState::RustPrimary => matches!( + record.state, + Phase92AuthorityState::Blocked | Phase92AuthorityState::RollbackPending + ), + Phase92AuthorityState::Blocked => { + record.state == Phase92AuthorityState::RollbackPending + } + Phase92AuthorityState::RollbackPending => { + record.state == Phase92AuthorityState::PythonPrimary + } + Phase92AuthorityState::PythonPrimary => record.state == Phase92AuthorityState::Blocked, + }; + if !allowed { + return Err("Phase 9.2 authority transition is not permitted".into()); + } + self.current = Some(record); + self.recovery_required = false; + Ok(()) + } + + pub fn restore_committed_watermark( + &mut self, + context: &Phase92PublicationContext, + ) -> Result<(), String> { + if !self.recovery_required { + return Err("Phase 9.2 watermark restore is only permitted during recovery".into()); + } + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if !matches!( + current.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ) || context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + || context.shard_id.trim().is_empty() + || !matches!( + context.target, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 + ) + || context.source_watermark < current.start_watermark + { + return Err("Phase 9.2 recovered watermark identity is invalid".into()); + } + let key = (context.shard_id.clone(), context.target); + if self + .committed_watermarks + .get(&key) + .is_some_and(|value| context.source_watermark < *value) + { + return Err("Phase 9.2 recovered watermark regressed".into()); + } + self.committed_watermarks + .insert(key, context.source_watermark); + Ok(()) + } + + pub fn permits(&self, context: &Phase92PublicationContext, now_ns: i64) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if now_ns <= 0 + || context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + || context.shard_id.trim().is_empty() + { + return Err("publication identity does not match Phase 9.2 authority".into()); + } + let target_allowed = match current.state { + Phase92AuthorityState::RustCanary => context.target == SinkTarget::CanaryCanonical, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary => matches!( + context.target, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 + ), + Phase92AuthorityState::Blocked | Phase92AuthorityState::RollbackPending => false, + }; + if !target_allowed { + return Err("sink target is not permitted by Phase 9.2 authority".into()); + } + if matches!( + current.state, + Phase92AuthorityState::RustCanary + | Phase92AuthorityState::RustPrimary + | Phase92AuthorityState::PythonPrimary + ) { + let approved_at = current + .approved_at_ns + .ok_or_else(|| "authority approval is missing".to_owned())?; + let hold_until = current + .hold_until_ns + .ok_or_else(|| "authority hold window is missing".to_owned())?; + if now_ns < approved_at || now_ns >= hold_until { + return Err("Phase 9.2 authority approval window is not active".into()); + } + } + let key = (context.shard_id.clone(), context.target); + if self.recovery_required && !self.committed_watermarks.contains_key(&key) { + return Err("Phase 9.2 durable target watermark recovery is required".into()); + } + let committed = self + .committed_watermarks + .get(&key) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark != committed + 1 { + return Err("Phase 9.2 source watermark is duplicate, stale or gapped".into()); + } + Ok(()) + } + + pub fn commit(&mut self, context: &Phase92PublicationContext) -> Result<(), String> { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if context.slice_id != current.slice_id + || context.owner_id != current.owner_id + || context.authority_revision != current.authority_revision + || context.lease_epoch != current.lease_epoch + || context.partition_plan_epoch != current.partition_plan_epoch + { + return Err("Phase 9.2 authority changed before commit".into()); + } + let key = (context.shard_id.clone(), context.target); + let committed = self + .committed_watermarks + .get(&key) + .copied() + .unwrap_or(current.start_watermark); + if context.source_watermark != committed + 1 { + return Err("Phase 9.2 watermark commit is not contiguous".into()); + } + self.committed_watermarks + .insert(key, context.source_watermark); + Ok(()) + } + + pub fn current(&self) -> Option<&Phase92AuthorityRecord> { + self.current.as_ref() + } +} + #[cfg(test)] mod tests { use super::{AuthorityFence, AuthorityMode, AuthorityRecord, PublicationContext, SinkTarget}; @@ -221,3 +976,567 @@ mod tests { .is_err()); } } + +#[cfg(test)] +mod phase9_tests { + use super::{ + Phase9AuthorityFence, Phase9AuthorityRecord, Phase9AuthorityState, + Phase9PublicationContext, SinkTarget, + }; + + const SLICE: &str = "production/binance/usdm/perpetual/trade/plan-1/btcusdt"; + const OWNER: &str = "rust-ingestor-binance-usdm-shard-0"; + const BUNDLE: &str = "558042db-a766-5a55-b5b3-4b508d649df9"; + + fn record( + revision: u64, + lease_epoch: u64, + state: Phase9AuthorityState, + ) -> Phase9AuthorityRecord { + let canary = state == Phase9AuthorityState::RustCanary; + Phase9AuthorityRecord { + schema: "qdl.authority-record.v2".into(), + slice_id: SLICE.into(), + state, + owner_id: OWNER.into(), + authority_revision: revision, + lease_epoch, + partition_plan_epoch: 1, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: canary.then(|| BUNDLE.into()), + start_watermark: 100, + approved_by: canary.then(|| "phase91-operator".into()), + approved_at_ns: canary.then_some(1_000), + hold_until_ns: canary.then_some(10_000), + public_write_allowed: false, + legacy_write_allowed: false, + } + } + + fn publication(target: SinkTarget, watermark: u64) -> Phase9PublicationContext { + Phase9PublicationContext { + slice_id: SLICE.into(), + owner_id: OWNER.into(), + authority_revision: 2, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch: 2, + partition_plan_epoch: 1, + source_watermark: watermark, + target, + } + } + + #[test] + fn canary_allows_only_isolated_target_inside_approval_window() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(1, 1, Phase9AuthorityState::RustShadow)) + .unwrap(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 101), 2_000) + .is_ok()); + assert!(fence + .permits(&publication(SinkTarget::PublicV2, 101), 2_000) + .is_err()); + assert!(fence + .permits(&publication(SinkTarget::LegacyV1, 101), 2_000) + .is_err()); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 101), 999) + .is_err()); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 101), 10_000) + .is_err()); + } + + #[test] + fn publication_binds_every_identity_and_epoch() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + let base = publication(SinkTarget::CanaryCanonical, 101); + let variants = [ + Phase9PublicationContext { + slice_id: "other".into(), + ..base.clone() + }, + Phase9PublicationContext { + owner_id: "stale-owner".into(), + ..base.clone() + }, + Phase9PublicationContext { + authority_revision: 1, + ..base.clone() + }, + Phase9PublicationContext { + lease_epoch: 1, + ..base.clone() + }, + Phase9PublicationContext { + partition_plan_epoch: 2, + ..base.clone() + }, + Phase9PublicationContext { + shard_id: String::new(), + ..base.clone() + }, + ]; + for context in variants { + assert!(fence.permits(&context, 2_000).is_err()); + } + } + + #[test] + fn watermark_advances_only_after_explicit_durable_commit() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + let first = publication(SinkTarget::CanaryCanonical, 101); + assert!(fence.permits(&first, 2_000).is_ok()); + assert!(fence.permits(&first, 2_000).is_ok()); + fence.commit(&first).unwrap(); + assert!(fence.permits(&first, 2_000).is_err()); + assert!(fence + .permits(&publication(SinkTarget::CanaryCanonical, 100), 2_000) + .is_err()); + let second = publication(SinkTarget::CanaryCanonical, 102); + assert!(fence.permits(&second, 2_000).is_ok()); + fence.commit(&second).unwrap(); + } + + #[test] + fn blocked_state_fences_every_target_and_requires_shadow_before_canary() { + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(1, 1, Phase9AuthorityState::RustShadow)) + .unwrap(); + fence + .apply(record(2, 1, Phase9AuthorityState::Blocked)) + .unwrap(); + let mut context = publication(SinkTarget::ShadowCanonical, 101); + context.owner_id = OWNER.into(); + context.authority_revision = 2; + context.lease_epoch = 1; + assert!(fence.permits(&context, 2_000).is_err()); + assert!(fence + .apply(record(3, 2, Phase9AuthorityState::RustCanary)) + .is_err()); + fence + .apply(record(3, 2, Phase9AuthorityState::RustShadow)) + .unwrap(); + } + + #[test] + fn malformed_records_and_stale_transitions_fail_closed() { + let mut invalid = record(1, 1, Phase9AuthorityState::RustCanary); + invalid.public_write_allowed = true; + assert!(invalid.validate().is_err()); + let mut invalid = record(1, 1, Phase9AuthorityState::RustCanary); + invalid.prerequisite_bundle_id = None; + assert!(invalid.validate().is_err()); + let mut invalid = record(1, 1, Phase9AuthorityState::RustShadow); + invalid.approved_by = Some("unexpected".into()); + assert!(invalid.validate().is_err()); + + let mut fence = Phase9AuthorityFence::default(); + fence + .apply(record(2, 2, Phase9AuthorityState::RustCanary)) + .unwrap(); + assert!(fence + .apply(record(1, 2, Phase9AuthorityState::RustShadow)) + .is_err()); + let mut conflict = record(2, 2, Phase9AuthorityState::RustCanary); + conflict.owner_id = "conflicting-owner".into(); + assert!(fence.apply(conflict).is_err()); + let mut stale_owner = record(3, 2, Phase9AuthorityState::RustShadow); + stale_owner.owner_id = "next-owner".into(); + assert!(fence.apply(stale_owner).is_err()); + let mut wrong_candidate = record(3, 3, Phase9AuthorityState::RustShadow); + wrong_candidate.candidate_digest = "2".repeat(64); + assert!(fence.apply(wrong_candidate).is_err()); + } +} + +#[cfg(test)] +mod phase92_tests { + use super::{ + Phase92AcceptedHandoff, Phase92AuthorityFence, Phase92AuthorityRecord, + Phase92AuthorityState, Phase92HandoffDirection, Phase92PublicationContext, + Phase92TerminalCheckpoint, SinkTarget, + }; + + const SLICE: &str = "production/binance/usdm/perpetual/trade/plan-1/btcusdt"; + const BUNDLE: &str = "558042db-a766-5a55-b5b3-4b508d649df9"; + + fn canary() -> Phase92AuthorityRecord { + Phase92AuthorityRecord { + schema: "qdl.authority-record.v3".into(), + slice_id: SLICE.into(), + state: Phase92AuthorityState::RustCanary, + owner_id: "python-primary".into(), + authority_revision: 7, + lease_epoch: 11, + partition_plan_epoch: 1, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: Some(BUNDLE.into()), + start_watermark: 89, + terminal_watermark: None, + previous_owner_id: None, + handoff_digest: None, + approved_by: Some("phase92-operator".into()), + approved_at_ns: Some(1), + hold_until_ns: Some(10_000), + public_write_allowed: false, + legacy_write_allowed: false, + } + } + + fn checkpoint( + owner: &str, + revision: u64, + lease: u64, + watermark: u64, + ) -> Phase92TerminalCheckpoint { + Phase92TerminalCheckpoint { + schema: "qdl.terminal-owner-checkpoint.v1".into(), + checkpoint_id: "11111111-1111-4111-8111-111111111111".into(), + slice_id: SLICE.into(), + owner_id: owner.into(), + authority_revision: revision, + lease_epoch: lease, + partition_plan_epoch: 1, + source_session_id: "session-1".into(), + connection_generation: 1, + terminal_watermark: watermark, + terminal_event_id: format!("event-{watermark}"), + terminal_payload_sha256: "2".repeat(64), + candidate_digest: "1".repeat(64), + committed_at_ns: 1, + } + } + + fn handoff( + checkpoint: &Phase92TerminalCheckpoint, + direction: Phase92HandoffDirection, + new_owner: &str, + new_state: Phase92AuthorityState, + ) -> Phase92AcceptedHandoff { + Phase92AcceptedHandoff { + schema: "qdl.accepted-authority-handoff.v1".into(), + handoff_id: "22222222-2222-4222-8222-222222222222".into(), + direction, + checkpoint_digest: checkpoint.digest().unwrap(), + slice_id: SLICE.into(), + old_owner_id: checkpoint.owner_id.clone(), + new_owner_id: new_owner.into(), + expected_state: if direction == Phase92HandoffDirection::PythonToRust { + Phase92AuthorityState::RustCanary + } else { + Phase92AuthorityState::RollbackPending + }, + new_state, + expected_authority_revision: checkpoint.authority_revision, + new_authority_revision: checkpoint.authority_revision + 1, + expected_lease_epoch: checkpoint.lease_epoch, + new_lease_epoch: checkpoint.lease_epoch + 1, + partition_plan_epoch: 1, + terminal_watermark: checkpoint.terminal_watermark, + first_new_watermark: checkpoint.terminal_watermark + 1, + overlap_start_watermark: checkpoint.terminal_watermark - 10, + overlap_end_watermark: checkpoint.terminal_watermark, + old_event_count: 11, + new_event_count: 11, + semantic_mismatches: 0, + open_gaps: 0, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: BUNDLE.into(), + approved_by: "phase92-operator".into(), + approved_at_ns: 1, + expires_at_ns: 10_000, + } + } + + fn primary( + checkpoint: &Phase92TerminalCheckpoint, + handoff: &Phase92AcceptedHandoff, + ) -> Phase92AuthorityRecord { + Phase92AuthorityRecord { + schema: "qdl.authority-record.v3".into(), + slice_id: SLICE.into(), + state: handoff.new_state, + owner_id: handoff.new_owner_id.clone(), + authority_revision: handoff.new_authority_revision, + lease_epoch: handoff.new_lease_epoch, + partition_plan_epoch: 1, + candidate_digest: "1".repeat(64), + prerequisite_bundle_id: (handoff.new_state == Phase92AuthorityState::RustPrimary) + .then(|| BUNDLE.into()), + start_watermark: checkpoint.terminal_watermark, + terminal_watermark: Some(checkpoint.terminal_watermark), + previous_owner_id: Some(checkpoint.owner_id.clone()), + handoff_digest: Some(handoff.digest().unwrap()), + approved_by: Some("phase92-operator".into()), + approved_at_ns: Some(1), + hold_until_ns: Some(10_000), + public_write_allowed: true, + legacy_write_allowed: true, + } + } + + fn publication( + record: &Phase92AuthorityRecord, + target: SinkTarget, + watermark: u64, + ) -> Phase92PublicationContext { + Phase92PublicationContext { + slice_id: SLICE.into(), + owner_id: record.owner_id.clone(), + authority_revision: record.authority_revision, + shard_id: "binance-usdm-trade-0".into(), + lease_epoch: record.lease_epoch, + partition_plan_epoch: 1, + source_watermark: watermark, + target, + } + } + + #[test] + fn accepted_handoff_is_required_and_first_primary_watermark_is_terminal_plus_one() { + let current = canary(); + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let handoff = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let primary = primary(&checkpoint, &handoff); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(current).unwrap(); + assert!(fence.apply(primary.clone()).is_err()); + fence + .apply_handoff(&checkpoint, &handoff, primary.clone(), 2) + .unwrap(); + assert!(fence + .restore_committed_watermark(&publication(&primary, SinkTarget::PrimaryCanonical, 120,)) + .is_err()); + + for target in [ + SinkTarget::PrimaryCanonical, + SinkTarget::PublicV2, + SinkTarget::LegacyV1, + ] { + assert!(fence + .permits(&publication(&primary, target, 100), 2) + .is_err()); + assert!(fence + .permits(&publication(&primary, target, 102), 2) + .is_err()); + let first = publication(&primary, target, 101); + fence.permits(&first, 2).unwrap(); + fence.commit(&first).unwrap(); + assert!(fence.permits(&first, 2).is_err()); + } + } + + #[test] + fn final_sink_and_projector_watermarks_are_independent_and_gap_free() { + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let handoff = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let primary = primary(&checkpoint, &handoff); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary()).unwrap(); + fence + .apply_handoff(&checkpoint, &handoff, primary.clone(), 2) + .unwrap(); + + let canonical = publication(&primary, SinkTarget::PrimaryCanonical, 101); + fence.permits(&canonical, 2).unwrap(); + fence.commit(&canonical).unwrap(); + assert!(fence + .permits(&publication(&primary, SinkTarget::PrimaryCanonical, 102), 2) + .is_ok()); + assert!(fence + .permits(&publication(&primary, SinkTarget::PublicV2, 101), 2) + .is_ok()); + assert!(fence + .permits(&publication(&primary, SinkTarget::LegacyV1, 101), 2) + .is_ok()); + } + + #[test] + fn stale_owner_revision_lease_plan_and_wrong_target_fail_closed() { + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let handoff = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let primary = primary(&checkpoint, &handoff); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary()).unwrap(); + fence + .apply_handoff(&checkpoint, &handoff, primary.clone(), 2) + .unwrap(); + let base = publication(&primary, SinkTarget::PrimaryCanonical, 101); + let variants = [ + Phase92PublicationContext { + owner_id: "python-primary".into(), + ..base.clone() + }, + Phase92PublicationContext { + authority_revision: 7, + ..base.clone() + }, + Phase92PublicationContext { + lease_epoch: 11, + ..base.clone() + }, + Phase92PublicationContext { + partition_plan_epoch: 2, + ..base.clone() + }, + Phase92PublicationContext { + target: SinkTarget::CanaryCanonical, + ..base + }, + ]; + for value in variants { + assert!(fence.permits(&value, 2).is_err()); + } + } + + #[test] + fn formal_rollback_fences_rust_and_hands_off_to_python_with_new_epoch() { + let initial_checkpoint = checkpoint("python-primary", 7, 11, 100); + let to_rust = handoff( + &initial_checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let rust_primary = primary(&initial_checkpoint, &to_rust); + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary()).unwrap(); + fence + .apply_handoff(&initial_checkpoint, &to_rust, rust_primary.clone(), 2) + .unwrap(); + + let mut blocked = rust_primary.clone(); + blocked.state = Phase92AuthorityState::Blocked; + blocked.authority_revision += 1; + blocked.public_write_allowed = false; + blocked.legacy_write_allowed = false; + fence.apply(blocked.clone()).unwrap(); + let mut pending = blocked; + pending.state = Phase92AuthorityState::RollbackPending; + pending.authority_revision += 1; + fence.apply(pending.clone()).unwrap(); + + let rollback_checkpoint = checkpoint( + "rust-primary", + pending.authority_revision, + pending.lease_epoch, + 120, + ); + let to_python = handoff( + &rollback_checkpoint, + Phase92HandoffDirection::RustToPython, + "python-rollback", + Phase92AuthorityState::PythonPrimary, + ); + let python_primary = primary(&rollback_checkpoint, &to_python); + fence + .apply_handoff(&rollback_checkpoint, &to_python, python_primary.clone(), 2) + .unwrap(); + assert!(fence + .permits( + &publication(&rust_primary, SinkTarget::PrimaryCanonical, 101), + 2, + ) + .is_err()); + assert!(fence + .permits( + &publication(&python_primary, SinkTarget::PrimaryCanonical, 121), + 2, + ) + .is_ok()); + } + + #[test] + fn crash_before_cas_reconstructs_canary_and_accepts_only_exact_handoff() { + let persisted_canary = canary(); + let terminal = checkpoint("python-primary", 7, 11, 100); + let accepted = handoff( + &terminal, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let rust_primary = primary(&terminal, &accepted); + + let mut recovered = Phase92AuthorityFence::default(); + recovered.apply(persisted_canary).unwrap(); + recovered + .apply_handoff(&terminal, &accepted, rust_primary.clone(), 2) + .unwrap(); + assert_eq!(recovered.current(), Some(&rust_primary)); + assert!(recovered + .apply_handoff(&terminal, &accepted, rust_primary, 2) + .is_err()); + } + + #[test] + fn restarted_primary_fails_closed_until_each_target_watermark_is_restored() { + let initial_checkpoint = checkpoint("python-primary", 7, 11, 100); + let to_rust = handoff( + &initial_checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + let rust_primary = primary(&initial_checkpoint, &to_rust); + let mut recovered = Phase92AuthorityFence::default(); + recovered.apply(rust_primary.clone()).unwrap(); + let duplicate = publication(&rust_primary, SinkTarget::PrimaryCanonical, 120); + assert!(recovered.permits(&duplicate, 2).is_err()); + recovered.restore_committed_watermark(&duplicate).unwrap(); + assert!(recovered.permits(&duplicate, 2).is_err()); + assert!(recovered + .permits( + &publication(&rust_primary, SinkTarget::PrimaryCanonical, 121), + 2, + ) + .is_ok()); + assert!(recovered + .permits(&publication(&rust_primary, SinkTarget::PublicV2, 121), 2) + .is_err()); + } + + #[test] + fn dirty_or_off_by_one_handoff_is_rejected() { + let checkpoint = checkpoint("python-primary", 7, 11, 100); + let mut dirty = handoff( + &checkpoint, + Phase92HandoffDirection::PythonToRust, + "rust-primary", + Phase92AuthorityState::RustPrimary, + ); + dirty.semantic_mismatches = 1; + assert!(dirty.validate(&checkpoint).is_err()); + dirty.semantic_mismatches = 0; + dirty.first_new_watermark = 102; + assert!(dirty.validate(&checkpoint).is_err()); + } +} diff --git a/rust/qdl-venue-core/src/backpressure.rs b/rust/qdl-venue-core/src/backpressure.rs index 851daa5..6d0474a 100644 --- a/rust/qdl-venue-core/src/backpressure.rs +++ b/rust/qdl-venue-core/src/backpressure.rs @@ -1,6 +1,9 @@ use std::collections::{BTreeMap, VecDeque}; -#[derive(Clone, Copy, Debug, Eq, PartialEq)] +use serde::Deserialize; + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] pub enum DeliveryClass { Lossless, LatestState, diff --git a/rust/qdl-venue-core/src/ordering.rs b/rust/qdl-venue-core/src/ordering.rs index 1a9da79..ede1377 100644 --- a/rust/qdl-venue-core/src/ordering.rs +++ b/rust/qdl-venue-core/src/ordering.rs @@ -1,5 +1,15 @@ use std::collections::{BTreeMap, BTreeSet}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum SequencePolicy { + None, + Monotonic, + Contiguous, +} + #[derive(Clone, Debug, Eq, PartialEq)] pub enum SequenceDecision { Accepted, @@ -24,6 +34,16 @@ pub struct OrderingTracker { max_recent_ids: usize, } +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct OrderingStage { + partition_key: String, + session_id: String, + generation: u64, + last_sequence: Option, + pending_event_ids: BTreeSet>, + reset_recent: bool, +} + impl OrderingTracker { pub fn new(max_recent_ids: usize) -> Self { Self { @@ -40,45 +60,127 @@ impl OrderingTracker { sequence: u64, event_id: Vec, ) -> SequenceDecision { - let state = self.partitions.entry(partition_key.into()).or_default(); - if generation < state.generation { + self.observe_with_policy( + partition_key, + session_id, + generation, + sequence, + event_id, + SequencePolicy::Contiguous, + ) + } + + pub fn observe_with_policy( + &mut self, + partition_key: &str, + session_id: &str, + generation: u64, + sequence: u64, + event_id: Vec, + policy: SequencePolicy, + ) -> SequenceDecision { + let mut stage = self.stage(partition_key); + let decision = self.observe_staged( + &mut stage, session_id, generation, sequence, event_id, policy, + ); + if matches!( + decision, + SequenceDecision::Accepted | SequenceDecision::SessionStarted + ) { + self.commit_stage(stage); + } + decision + } + + pub fn stage(&self, partition_key: &str) -> OrderingStage { + let (session_id, generation, last_sequence) = self + .partitions + .get(partition_key) + .map(|state| { + ( + state.session_id.clone(), + state.generation, + state.last_sequence, + ) + }) + .unwrap_or_default(); + OrderingStage { + partition_key: partition_key.into(), + session_id, + generation, + last_sequence, + pending_event_ids: BTreeSet::new(), + reset_recent: false, + } + } + + pub fn observe_staged( + &self, + stage: &mut OrderingStage, + session_id: &str, + generation: u64, + sequence: u64, + event_id: Vec, + policy: SequencePolicy, + ) -> SequenceDecision { + if generation < stage.generation { return SequenceDecision::StaleSession; } - if generation > state.generation || state.session_id != session_id { - state.session_id = session_id.into(); - state.generation = generation; - state.last_sequence = Some(sequence); - state.recent_event_ids.clear(); - state.recent_event_ids.insert(event_id); + if generation > stage.generation || stage.session_id != session_id { + stage.session_id = session_id.into(); + stage.generation = generation; + stage.last_sequence = Some(sequence); + stage.pending_event_ids.clear(); + stage.pending_event_ids.insert(event_id); + stage.reset_recent = true; return SequenceDecision::SessionStarted; } - if state.recent_event_ids.contains(&event_id) { + let committed_duplicate = !stage.reset_recent + && self + .partitions + .get(&stage.partition_key) + .is_some_and(|state| state.recent_event_ids.contains(&event_id)); + if committed_duplicate || stage.pending_event_ids.contains(&event_id) { return SequenceDecision::Duplicate; } - let decision = match state.last_sequence { - Some(last) if sequence <= last => SequenceDecision::OutOfOrder, - Some(last) if sequence > last.saturating_add(1) => SequenceDecision::Gap { - expected: last.saturating_add(1), - actual: sequence, - }, + let decision = match (policy, stage.last_sequence) { + (SequencePolicy::None, _) => SequenceDecision::Accepted, + (_, Some(last)) if sequence <= last => SequenceDecision::OutOfOrder, + (SequencePolicy::Contiguous, Some(last)) if sequence > last.saturating_add(1) => { + SequenceDecision::Gap { + expected: last.saturating_add(1), + actual: sequence, + } + } _ => SequenceDecision::Accepted, }; if matches!(decision, SequenceDecision::Accepted) { - state.last_sequence = Some(sequence); - state.recent_event_ids.insert(event_id); - while state.recent_event_ids.len() > self.max_recent_ids { - if let Some(first) = state.recent_event_ids.iter().next().cloned() { - state.recent_event_ids.remove(&first); - } - } + stage.last_sequence = Some(sequence); + stage.pending_event_ids.insert(event_id); } decision } + + pub fn commit_stage(&mut self, stage: OrderingStage) { + let state = self.partitions.entry(stage.partition_key).or_default(); + state.session_id = stage.session_id; + state.generation = stage.generation; + state.last_sequence = stage.last_sequence; + if stage.reset_recent { + state.recent_event_ids.clear(); + } + state.recent_event_ids.extend(stage.pending_event_ids); + while state.recent_event_ids.len() > self.max_recent_ids { + if let Some(first) = state.recent_event_ids.iter().next().cloned() { + state.recent_event_ids.remove(&first); + } + } + } } #[cfg(test)] mod tests { - use super::{OrderingTracker, SequenceDecision}; + use super::{OrderingTracker, SequenceDecision, SequencePolicy}; #[test] fn duplicate_gap_out_of_order_and_session_reset_are_distinct() { @@ -115,4 +217,93 @@ mod tests { SequenceDecision::StaleSession ); } + #[test] + fn monotonic_allows_native_leaps_but_contiguous_detects_them() { + let mut monotonic = OrderingTracker::new(8); + assert_eq!( + monotonic + .observe_with_policy("trade", "s1", 1, 10, vec![1], SequencePolicy::Monotonic,), + SequenceDecision::SessionStarted + ); + assert_eq!( + monotonic + .observe_with_policy("trade", "s1", 1, 15, vec![2], SequencePolicy::Monotonic,), + SequenceDecision::Accepted + ); + let mut none = OrderingTracker::new(8); + none.observe_with_policy("bar", "s1", 1, 60, vec![1], SequencePolicy::None); + assert_eq!( + none.observe_with_policy("bar", "s1", 1, 1, vec![2], SequencePolicy::None), + SequenceDecision::Accepted + ); + } + + #[test] + fn discarded_stage_does_not_mutate_committed_ordering_state() { + let tracker = OrderingTracker::new(8); + let mut discarded = tracker.stage("btc"); + assert_eq!( + tracker.observe_staged( + &mut discarded, + "s1", + 1, + 10, + vec![1], + SequencePolicy::Contiguous, + ), + SequenceDecision::SessionStarted + ); + assert_eq!( + tracker.observe_staged( + &mut discarded, + "s1", + 1, + 11, + vec![2], + SequencePolicy::Contiguous, + ), + SequenceDecision::Accepted + ); + + let mut retry = tracker.stage("btc"); + assert_eq!( + tracker.observe_staged(&mut retry, "s1", 1, 10, vec![1], SequencePolicy::Contiguous,), + SequenceDecision::SessionStarted + ); + } + + #[test] + fn committed_stage_preserves_batch_duplicate_and_gap_semantics() { + let mut tracker = OrderingTracker::new(8); + let mut stage = tracker.stage("btc"); + assert_eq!( + tracker.observe_staged(&mut stage, "s1", 1, 10, vec![1], SequencePolicy::Contiguous,), + SequenceDecision::SessionStarted + ); + assert_eq!( + tracker.observe_staged(&mut stage, "s1", 1, 11, vec![2], SequencePolicy::Contiguous,), + SequenceDecision::Accepted + ); + assert_eq!( + tracker.observe_staged(&mut stage, "s1", 1, 11, vec![2], SequencePolicy::Contiguous,), + SequenceDecision::Duplicate + ); + tracker.commit_stage(stage); + + assert_eq!( + tracker.observe("btc", "s1", 1, 11, vec![2]), + SequenceDecision::Duplicate + ); + assert_eq!( + tracker.observe("btc", "s1", 1, 13, vec![3]), + SequenceDecision::Gap { + expected: 12, + actual: 13, + } + ); + assert_eq!( + tracker.observe("btc", "s1", 1, 12, vec![4]), + SequenceDecision::Accepted + ); + } } diff --git a/scripts/generate_phase1_contract_golden.py b/scripts/generate_phase1_contract_golden.py index 31dfd67..6d40266 100644 --- a/scripts/generate_phase1_contract_golden.py +++ b/scripts/generate_phase1_contract_golden.py @@ -56,6 +56,8 @@ def build_trade_envelope(): price=decimal(6_123_410, 2, "61234.10"), quantity=decimal(125, 3, "0.125"), aggressor_side=common_pb2.AGGRESSOR_SIDE_BUY, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + identity_kind=market_data_pb2.TRADE_IDENTITY_KIND_NATIVE, ), ) diff --git a/scripts/generate_phase5_openapi.py b/scripts/generate_phase5_openapi.py index 12dfbcd..12bda4a 100644 --- a/scripts/generate_phase5_openapi.py +++ b/scripts/generate_phase5_openapi.py @@ -9,7 +9,9 @@ def build_openapi() -> dict: # OpenAPI generation is pure and never invokes the injected query service. - return create_v2_app(None).openapi() # type: ignore[arg-type] + return create_v2_app( + None, contract_version="2.0.0", authority="INTERNAL_STABLE" + ).openapi() # type: ignore[arg-type] def main() -> None: diff --git a/scripts/generate_v2_stable_multivenue_golden.py b/scripts/generate_v2_stable_multivenue_golden.py new file mode 100755 index 0000000..4213a93 --- /dev/null +++ b/scripts/generate_v2_stable_multivenue_golden.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +from pathlib import Path +from typing import Callable + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.canonical.market import ( # noqa: E402 + canonicalize_binance_usdm_bar, + canonicalize_binance_usdm_bbo, + canonicalize_binance_usdm_rest_bar, + canonicalize_dnse_bar, + canonicalize_okx_bar, + canonicalize_okx_bbo, +) +from qdl.canonical.trade import ( # noqa: E402 + TradeContext, + canonicalize_binance_usdm_trade, + canonicalize_dnse_trade, + canonicalize_okx_trade, +) + + +FIXTURES = ROOT / "tests/fixtures/phase2" +OUTPUT = ROOT / "contracts/golden/phase2" +MANIFEST = OUTPUT / "manifest-v2-stable-multivenue.json" +Canonicalizer = Callable[[dict, TradeContext], object] +CASES: tuple[tuple[str, str, Canonicalizer], ...] = ( + ("binance_usdm_trade.json", "binance-usdm-trade.bin", canonicalize_binance_usdm_trade), + ("binance_usdm_bbo.json", "binance-usdm-bbo.bin", canonicalize_binance_usdm_bbo), + ("binance_usdm_bar.json", "binance-usdm-bar.bin", canonicalize_binance_usdm_bar), + ("binance_usdm_rest_bar.json", "binance-usdm-rest-bar.bin", canonicalize_binance_usdm_rest_bar), + ("binance_spot_trade.json", "binance-spot-trade.bin", canonicalize_binance_usdm_trade), + ("binance_spot_bbo.json", "binance-spot-bbo.bin", canonicalize_binance_usdm_bbo), + ("binance_spot_bar.json", "binance-spot-bar.bin", canonicalize_binance_usdm_bar), + ("binance_spot_rest_bar.json", "binance-spot-rest-bar.bin", canonicalize_binance_usdm_rest_bar), + ("okx_trade.json", "okx-swap-trade.bin", canonicalize_okx_trade), + ("okx_bbo.json", "okx-swap-bbo.bin", canonicalize_okx_bbo), + ("okx_bar.json", "okx-swap-bar.bin", canonicalize_okx_bar), + ("okx_spot_trade.json", "okx-spot-trade.bin", canonicalize_okx_trade), + ("okx_spot_bbo.json", "okx-spot-bbo.bin", canonicalize_okx_bbo), + ("okx_spot_bar.json", "okx-spot-bar.bin", canonicalize_okx_bar), + ("dnse_derivative_trade.json", "dnse-derivative-trade.bin", canonicalize_dnse_trade), + ("dnse_derivative_bar.json", "dnse-derivative-bar.bin", canonicalize_dnse_bar), + ("dnse_equity_trade.json", "dnse-equity-trade.bin", canonicalize_dnse_trade), + ("dnse_equity_bar.json", "dnse-equity-bar.bin", canonicalize_dnse_bar), + ("vnstock_equity_bar.json", "vnstock-equity-bar.bin", canonicalize_dnse_bar), +) + + +def render(fixture_name: str, canonicalize: Canonicalizer) -> bytes: + fixture = json.loads((FIXTURES / fixture_name).read_text()) + envelope = canonicalize(fixture["raw"], TradeContext(**fixture["context"])) + return envelope.SerializeToString(deterministic=True) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + manifest_cases = [] + for fixture_name, golden_name, canonicalize in CASES: + payload = render(fixture_name, canonicalize) + golden = OUTPUT / golden_name + if args.check: + if not golden.is_file() or golden.read_bytes() != payload: + raise RuntimeError(f"stale multivenue golden artifact: {golden_name}") + else: + golden.write_bytes(payload) + manifest_cases.append({ + "fixture": fixture_name, + "golden": golden_name, + "bytes": len(payload), + "sha256": hashlib.sha256(payload).hexdigest(), + }) + manifest = { + "schema": "qdl.v2.stable-multivenue-golden.v1", + "provenance": "DETERMINISTIC_PROVIDER_FIXTURE", + "cases": manifest_cases, + } + rendered = json.dumps(manifest, indent=2, sort_keys=True) + "\n" + if args.check: + if not MANIFEST.is_file() or MANIFEST.read_text() != rendered: + raise RuntimeError("stale multivenue golden manifest") + else: + MANIFEST.write_text(rendered) + print(json.dumps({"status": "PASS", "mode": "CHECK" if args.check else "WRITE", "cases": len(CASES)}, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase5_api_replica_load.py b/scripts/phase5_api_replica_load.py index 6d8ccc5..f9fbbd8 100644 --- a/scripts/phase5_api_replica_load.py +++ b/scripts/phase5_api_replica_load.py @@ -63,6 +63,8 @@ def _service(): "native_trade_id": "phase5-load-trade", "price": "60000.1", "quantity": "0.01", + "quantity_unit": "BASE_ASSET", + "identity_kind": "NATIVE", "aggressor_side": "BUY", "is_block_trade": False, "is_buyer_maker": False, diff --git a/scripts/phase5_migration_smoke.sh b/scripts/phase5_migration_smoke.sh index 9089075..0dcafb6 100755 --- a/scripts/phase5_migration_smoke.sh +++ b/scripts/phase5_migration_smoke.sh @@ -45,11 +45,17 @@ if [[ "${container_started}" != "true" ]]; then fi ready=false +consecutive_ready=0 for _ in $(seq 1 240); do if docker exec "${container}" psql -U postgres -d postgres -Atc "SELECT 1" \ 2>/dev/null | grep -qx "1"; then - ready=true - break + consecutive_ready=$((consecutive_ready + 1)) + if [[ "${consecutive_ready}" -ge 8 ]]; then + ready=true + break + fi + else + consecutive_ready=0 fi sleep 0.25 done diff --git a/scripts/phase73_public_beta_certification.sh b/scripts/phase73_public_beta_certification.sh index 2c15ffb..de32581 100755 --- a/scripts/phase73_public_beta_certification.sh +++ b/scripts/phase73_public_beta_certification.sh @@ -32,10 +32,22 @@ export QDL_BETA_STREAM_A_GRPC_PORT="${STREAM_A_GRPC_PORT}" export QDL_BETA_STREAM_B_GRPC_PORT="${STREAM_B_GRPC_PORT}" temporary="$(mktemp -d)" -chown 10001:10001 "${temporary}" +runner_evidence="${temporary}/runner" +mkdir -m 0700 "${runner_evidence}" +OPERATOR_UID="$(id -u)" +OPERATOR_GID="$(id -g)" +set_evidence_owner() { + local uid="$1" gid="$2" + docker run --rm --network none --user 0:0 --cap-drop ALL --cap-add CHOWN --cap-add FOWNER \ + --security-opt no-new-privileges:true --entrypoint sh \ + -v "${runner_evidence}:/evidence" "${QDL_BETA_INIT_IMAGE}" \ + -c "chown ${uid}:${gid} /evidence && chmod 0755 /evidence" +} +set_evidence_owner 10001 10001 cleanup() { docker compose -p "${PROJECT}" -f "${COMPOSE_FILE}" \ --profile phase7-beta down -v --remove-orphans >/dev/null 2>&1 || true + set_evidence_owner "${OPERATOR_UID}" "${OPERATOR_GID}" >/dev/null 2>&1 || true rm -rf "${temporary}" } trap cleanup EXIT @@ -123,7 +135,7 @@ docker run --rm --network host --read-only --cap-drop ALL \ --security-opt no-new-privileges:true --pids-limit 256 --memory 512m --cpus 1 \ --user 10001:10001 --tmpfs /tmp:rw,noexec,nosuid,nodev,size=32m,uid=10001,gid=10001 \ -v "${durable_volume}:/var/lib/qdl-beta-durable" \ - -v "${temporary}:/evidence" \ + -v "${runner_evidence}:/evidence" \ "${QDL_BETA_IMAGE}" python /app/scripts/phase73_beta_certification.py \ --source-bindings /app/config/phase7/canary-sources.yaml \ --monitoring-manifest /app/consumers/beta/phase7-monitoring-binance.yaml \ @@ -184,6 +196,6 @@ python3 "${ROOT_DIR}/scripts/phase73_runtime_evidence.py" \ --keys-after "${keys_after}" --containers-after "${containers_after}" \ --networks-after "${networks_after}" --volumes-after "${volumes_after}" python3 "${ROOT_DIR}/scripts/phase73_finalize_evidence.py" \ - --core "${temporary}/core.json" --stats "${temporary}/stats.jsonl" \ + --core "${runner_evidence}/core.json" --stats "${temporary}/stats.jsonl" \ --runtime "${temporary}/runtime.json" --output "${CAPACITY_OUTPUT}" \ --security-output "${SECURITY_OUTPUT}" diff --git a/scripts/phase80_broker_certification.py b/scripts/phase80_broker_certification.py index 126f180..df91cd7 100755 --- a/scripts/phase80_broker_certification.py +++ b/scripts/phase80_broker_certification.py @@ -102,12 +102,16 @@ def compose(env: dict[str, str], *arguments: str, **kwargs: object) -> CommandRe def kafka(env: dict[str, str], script: str, *arguments: str, **kwargs: object) -> CommandResult: + # Kafka CLI tools are JVMs. Running them inside a 512 MiB broker cgroup can + # OOM-kill the broker, so administration uses a disposable isolated client. return compose( env, - "exec", - "-T", - "kafka1", + "run", + "--rm", + "--no-deps", + "--entrypoint", f"/opt/kafka/bin/{script}", + "phase8_admin", *arguments, **kwargs, ) diff --git a/scripts/phase80_generate_tls.sh b/scripts/phase80_generate_tls.sh index d0345d7..cdb45c0 100755 --- a/scripts/phase80_generate_tls.sh +++ b/scripts/phase80_generate_tls.sh @@ -5,6 +5,8 @@ ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" OUTPUT_DIR="${1:?usage: phase80_generate_tls.sh OUTPUT_DIR}" KAFKA_IMAGE="${QDL_PHASE8_KAFKA_IMAGE:-apache/kafka@sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9}" PASSWORD="${QDL_PHASE8_CERT_PASSWORD:-phase8-certification-only}" +CERT_UID="${QDL_PHASE8_CERT_UID:-$(id -u)}" +CERT_GID="${QDL_PHASE8_CERT_GID:-$(id -g)}" umask 077 mkdir -p "${OUTPUT_DIR}" @@ -46,7 +48,7 @@ issue_certificate() { for broker in kafka1 kafka2 kafka3; do issue_certificate "${broker}" "${broker}" done -for client in phase8-admin phase8-producer phase8-consumer phase8-unauthorized; do +for client in phase8-admin phase8-producer phase8-consumer phase8-core phase8-unauthorized; do issue_certificate "${client}" "${client}" done @@ -54,7 +56,9 @@ printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/key.password" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/store.password" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/truststore.password" -docker run --rm --user 0:0 \ +# Match the host caller so bind-mounted stores remain chmod/removal-safe on +# rootless, user-namespaced and migrated Docker hosts. +docker run --rm --user "${CERT_UID}:${CERT_GID}" \ --mount "type=bind,source=${OUTPUT_DIR},target=/certs" \ --entrypoint /opt/java/openjdk/bin/keytool \ "${KAFKA_IMAGE}" \ @@ -103,6 +107,7 @@ find "${OUTPUT_DIR}" -type f -exec chmod 0644 {} + find "${OUTPUT_DIR}" -maxdepth 1 -name '*.key' \ ! -name 'phase8-producer.key' \ ! -name 'phase8-consumer.key' \ + ! -name 'phase8-core.key' \ -delete rm -f "${OUTPUT_DIR}"/*.csr "${OUTPUT_DIR}"/*.ext "${OUTPUT_DIR}"/*.srl diff --git a/scripts/phase90b_bridge_parity.py b/scripts/phase90b_bridge_parity.py new file mode 100755 index 0000000..7edd959 --- /dev/null +++ b/scripts/phase90b_bridge_parity.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + +from qdl.runtime.canary_source import CanarySourceBinding, CanarySourceCatalog + + +def _decimal_source(payload: dict[str, Any], field: str) -> str: + value = payload.get(field) + if not isinstance(value, dict) or not isinstance(value.get("source_text"), str): + raise AssertionError(f"V2 {field} has no exact decimal source_text") + return value["source_text"] + + +def validate_sample( + v1: dict[str, Any], + v2: dict[str, Any], + binding: CanarySourceBinding, +) -> dict[str, int]: + identity = { + "provider": v1.get("provider") == "binance", + "market": str(v1.get("market", "")).lower() == "usdm", + "symbol": str(v1.get("symbol", "")).upper() + == binding.instrument.native_symbol, + "interval": v1.get("requested_interval") == binding.interval, + "schema": v2.get("schema") == "qdl.marketdata.warmup.v2", + } + failed = [name for name, passed in identity.items() if not passed] + if failed: + raise AssertionError(f"V1/V2 response identity mismatch: {failed}") + + rows = v1.get("data") + events = v2.get("data") + if not isinstance(rows, list) or not isinstance(events, list) or not events: + raise AssertionError("V1 rows and non-empty V2 data are required") + if v2.get("count") != len(events): + raise AssertionError("V2 count does not match data length") + expected = {int(row[0]) * 1_000_000: row for row in rows} + if len(expected) != len(rows): + raise AssertionError("V1 source contains duplicate open times") + + open_times: list[int] = [] + for item in events: + payload = item.get("payload", {}) + source = item.get("source", {}) + quality = item.get("quality", {}) + open_time = int(payload.get("open_time_ns", 0)) + row = expected.get(open_time) + if row is None: + raise AssertionError("V2 bar does not correspond to a V1 source row") + checks = { + "instrument_uid": item.get("instrument_uid") + == binding.instrument.instrument_uid, + "instrument_id": item.get("instrument_id") + == binding.instrument.instrument_id, + "feed": item.get("feed") == "BAR" and payload.get("feed") == "BAR", + "interval": item.get("interval") == binding.interval + and payload.get("interval") == binding.interval, + "open": _decimal_source(payload, "open") == str(row[1]), + "high": _decimal_source(payload, "high") == str(row[2]), + "low": _decimal_source(payload, "low") == str(row[3]), + "close": _decimal_source(payload, "close") == str(row[4]), + "volume": _decimal_source(payload, "volume") == str(row[5]), + "close_time": int(payload.get("close_time_ns", 0)) + == int(row[6]) * 1_000_000, + "trade_count": int(payload.get("trade_count", -1)) == int(row[8]), + "final": payload.get("lifecycle") == "FINAL", + "source_id": source.get("source_id") == binding.source_id, + "source_role": source.get("source_role") == binding.source_role, + "authoritative": source.get("authoritative") is binding.authoritative, + "policy": quality.get("policy_id") == binding.source_policy_id, + "complete": quality.get("complete") is True, + "gap_closed": quality.get("gap_open") is False, + "execution_forbidden": quality.get("execution_eligible") is False, + } + failed = [name for name, passed in checks.items() if not passed] + if failed: + raise AssertionError(f"V1/V2 canonical parity failed: {failed}") + open_times.append(open_time) + + if open_times != sorted(open_times) or len(open_times) != len(set(open_times)): + raise AssertionError("V2 bars are duplicated or not strictly ordered") + watermark = int(v2.get("watermark_offset", -1)) + if watermark < len(events): + raise AssertionError("V2 watermark is below the returned history length") + return { + "count": len(events), + "first_open_time_ns": open_times[0], + "last_open_time_ns": open_times[-1], + "watermark_offset": watermark, + } + + +def validate_window(first: dict[str, int], second: dict[str, int]) -> None: + delta = second["watermark_offset"] - first["watermark_offset"] + if delta < 0 or delta > 1: + raise AssertionError( + f"continuous bridge watermark advanced outside one 1m-bar window: {delta}" + ) + if second["last_open_time_ns"] < first["last_open_time_ns"]: + raise AssertionError("continuous bridge moved the latest bar backwards") + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--source-bindings", type=Path, required=True) + parser.add_argument("--v1-first", type=Path, required=True) + parser.add_argument("--v2-first", type=Path, required=True) + parser.add_argument("--v1-second", type=Path, required=True) + parser.add_argument("--v2-second", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + binding = CanarySourceCatalog.load(args.source_bindings).bindings[0] + first = validate_sample( + json.loads(args.v1_first.read_text()), + json.loads(args.v2_first.read_text()), + binding, + ) + second = validate_sample( + json.loads(args.v1_second.read_text()), + json.loads(args.v2_second.read_text()), + binding, + ) + validate_window(first, second) + result = { + "schema": "qdl.phase9.0-b.bridge-parity.v1", + "status": "PASS", + "authority": "V1_SHADOW_READ_ONLY", + "source": "REAL_V1_PROVIDER_DATA", + "generated_market_events": 0, + "slice": { + "venue": "BINANCE", + "market": "USDM", + "product_type": "PERPETUAL", + "symbol": binding.instrument.native_symbol, + "feed": binding.feed, + "interval": binding.interval, + }, + "first": first, + "second": second, + "watermark_delta": second["watermark_offset"] - first["watermark_offset"], + "canonical_mismatches": 0, + "duplicate_open_times": 0, + "non_final_bars": 0, + "execution_eligible_events": 0, + } + args.output.write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/phase90b_finalize_evidence.py b/scripts/phase90b_finalize_evidence.py new file mode 100755 index 0000000..625ceef --- /dev/null +++ b/scripts/phase90b_finalize_evidence.py @@ -0,0 +1,175 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path + + +def _load(path: Path): + return json.loads(path.read_text()) + + +def _stats(path: Path) -> list[dict[str, object]]: + rows = [] + for line in path.read_text().splitlines(): + if not line.strip(): + continue + value = json.loads(line) + rows.append({ + key: value.get(key) + for key in ("Name", "CPUPerc", "MemUsage", "MemPerc", "PIDs", "NetIO") + }) + return rows + + +def main() -> None: + parser = argparse.ArgumentParser() + for name in ( + "capacity", "security", "parity", "image-inspect", "bridge-inspect", + "stats", "v1-before", "v1-after", + ): + parser.add_argument(f"--{name}", type=Path, required=True) + parser.add_argument("--openapi-before", required=True) + parser.add_argument("--openapi-after", required=True) + for name in ( + "production-keys-before", "production-keys-after", "containers-after", + "networks-after", "volumes-after", "images-after", + ): + parser.add_argument(f"--{name}", type=int, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--report", type=Path, required=True) + args = parser.parse_args() + + capacity = _load(args.capacity) + security = _load(args.security) + parity = _load(args.parity) + image = _load(args.image_inspect)[0] + bridge = _load(args.bridge_inspect)[0] + v1_before = _load(args.v1_before) + v1_after = _load(args.v1_after) + labels = image.get("Config", {}).get("Labels", {}) or {} + mounts = bridge.get("Mounts", []) + networks = sorted(bridge.get("NetworkSettings", {}).get("Networks", {})) + checks = { + "phase73_capacity": capacity.get("status") == "PASS" + and all(capacity.get("checks", {}).values()), + "phase73_security": security.get("status") == "PASS", + "continuous_bridge_parity": parity.get("status") == "PASS" + and parity.get("canonical_mismatches") == 0, + "provider_authentic": parity.get("source") == "REAL_V1_PROVIDER_DATA" + and parity.get("generated_market_events") == 0, + "candidate_revision_labeled": bool(labels.get("org.opencontainers.image.revision")), + "candidate_non_root": bridge.get("Config", {}).get("User") == "10001:10001", + "candidate_read_only": bridge.get("HostConfig", {}).get("ReadonlyRootfs") is True, + "candidate_no_source_bind": not any( + item.get("Type") == "bind" for item in mounts + ), + "bridge_network_isolated": "bobby_network" in networks + and any(name.endswith("_qdl_beta_internal") for name in networks) + and not any(name.endswith("_qdl_beta_ingress") for name in networks), + "v1_topology_unchanged": v1_before == v1_after, + "v1_openapi_unchanged": args.openapi_before == args.openapi_after, + "production_beta_keys_absent": args.production_keys_before == 0 + and args.production_keys_after == 0, + "cleanup_exact": args.containers_after == 0 + and args.networks_after == 0 and args.volumes_after == 0 + and args.images_after == 0, + } + failed = sorted(name for name, passed in checks.items() if not passed) + if failed: + raise SystemExit(f"Phase 9.0-B evidence failed: {failed}") + + result = { + "schema": "qdl.phase9.0-b.isolated-beta.v1", + "status": "PASS_ISOLATED_NO_AUTHORITY_CUTOVER", + "authority": "V1_SHADOW_READ_ONLY", + "decision_scope": "V2_BETA_READ_ONLY_REVIEW", + "checks": checks, + "candidate": { + "image_id": image.get("Id"), + "source_revision": labels.get("org.opencontainers.image.revision"), + "release": labels.get("org.opencontainers.image.version"), + "runtime_user": bridge.get("Config", {}).get("User"), + "read_only_root": bridge.get("HostConfig", {}).get("ReadonlyRootfs"), + "source_bind_mounts": sum(item.get("Type") == "bind" for item in mounts), + "bridge_networks": networks, + }, + "contract_and_capacity": capacity, + "security": security, + "continuous_bridge": parity, + "resource_snapshot": _stats(args.stats), + "cleanup": { + "containers": args.containers_after, + "networks": args.networks_after, + "volumes": args.volumes_after, + "images": args.images_after, + "production_beta_keys": args.production_keys_after, + }, + "production_v1": { + "topology_unchanged": True, + "openapi_sha256": args.openapi_after, + "restarted_or_recreated": False, + }, + "phase_91": "BLOCKED", + } + args.output.write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + + normal = capacity["normal"] + burst = capacity["burst"] + stream = capacity["stream"] + resources = capacity["resources"] + report = f'''# Phase 9.0-B Isolated V2 Beta Report + +Decision: `PASS_ISOLATED_NO_AUTHORITY_CUTOVER` + +## Scope + +The existing V2 query/stream beta was re-certified from the Phase 9.0-A +correctness baseline. V1 remained source and public authority. The candidate +used an isolated Redis, durable spool, credentials, consumer group, loopback +ports and Compose project. No Rust canary or public-internet exposure occurred. + +## Results + +- Authentic slice: `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`. +- Canonical mismatches: `{parity['canonical_mismatches']}`; generated events: + `{parity['generated_market_events']}`; duplicate open times: + `{parity['duplicate_open_times']}`. +- Continuous bridge watermark delta across the bounded observation window: + `{parity['watermark_delta']}`. +- Active/passive stream replay: `{stream['fast_consumers']}` fast consumers, + `{stream['slow_consumers']}` isolated slow consumer, contiguous= + `{all(item['contiguous'] for item in stream['fast_results'])}`. +- Query normal: `{normal['requests']}` requests, p99.9 + `{normal['latency_ms']['p99_9']}` ms, `{normal['errors']}` errors. +- Query burst: `{burst['requests']}` requests, p99.9 + `{burst['latency_ms']['p99_9']}` ms, `{burst['errors']}` errors. +- Peak candidate RSS: `{resources['max_rss_bytes']}` bytes; peak CPU: + `{resources['max_cpu_percent_of_one_core']}%` of one core. +- Security/adversarial, cursor, failover/fencing, dependency outage/recovery, + rate limit, oversized request and cleanup gates all passed. + +## Safety And Cleanup + +Candidate revision: `{result['candidate']['source_revision']}`. Runtime user was +`10001:10001`, root was read-only, no source bind was mounted and bridge access +was restricted to beta-internal plus the existing V1 internal network. +Production V1 topology/OpenAPI remained unchanged. Candidate containers, +networks, volumes and candidate tags after cleanup: `0/0/0/0`; production beta +keys: `0`. + +## Remaining Boundary + +This result permits review of an isolated read-only V2 beta only. Phase 9.1 +remains blocked on replicated production transport, OTel/alerts, workload +identity, external secrets, signature admission, independent DR, complete +consumer registration and explicit exact-slice authority approval. +''' + args.report.write_text(report, encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/scripts/phase90b_isolated_beta_certification.sh b/scripts/phase90b_isolated_beta_certification.sh new file mode 100755 index 0000000..280f5d4 --- /dev/null +++ b/scripts/phase90b_isolated_beta_certification.sh @@ -0,0 +1,231 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +COMPOSE_FILE="${ROOT_DIR}/docker-compose.phase7-beta.yml" +IMAGE_REF="${QDL_PHASE90B_IMAGE:?set QDL_PHASE90B_IMAGE}" +MATRIX_PROJECT="${QDL_PHASE90B_MATRIX_PROJECT:-qdl_phase90b_matrix}" +BRIDGE_PROJECT="${QDL_PHASE90B_BRIDGE_PROJECT:-qdl_phase90b_bridge}" +V1_CONTAINER="${QDL_V1_CONTAINER:-data_layer_service}" +PROD_REDIS_CONTAINER="${QDL_V1_REDIS_CONTAINER:-redis_marketdata}" +CAPACITY_OUTPUT="${QDL_PHASE90B_CAPACITY_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-capacity.json}" +SECURITY_OUTPUT="${QDL_PHASE90B_SECURITY_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-security-adversarial.json}" +PARITY_OUTPUT="${QDL_PHASE90B_PARITY_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-continuous-bridge.json}" +RESULT_OUTPUT="${QDL_PHASE90B_RESULT_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-isolated-v2-beta.json}" +REPORT_OUTPUT="${QDL_PHASE90B_REPORT_OUTPUT:-${ROOT_DIR}/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md}" +CHECKSUM_OUTPUT="${QDL_PHASE90B_CHECKSUM_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90b-evidence.sha256}" +QUERY_PORT="${QDL_PHASE90B_QUERY_PORT:-18220}" +STREAM_A_HEALTH_PORT="${QDL_PHASE90B_STREAM_A_HEALTH_PORT:-18221}" +STREAM_B_HEALTH_PORT="${QDL_PHASE90B_STREAM_B_HEALTH_PORT:-18222}" +STREAM_A_GRPC_PORT="${QDL_PHASE90B_STREAM_A_GRPC_PORT:-18230}" +STREAM_B_GRPC_PORT="${QDL_PHASE90B_STREAM_B_GRPC_PORT:-18231}" + +image_id="$(docker image inspect "${IMAGE_REF}" --format '{{.Id}}')" +redis_id="$(docker image inspect redis:7.2-alpine --format '{{.Id}}')" +random_hex() { python3 -c 'import secrets; print(secrets.token_hex(32))'; } +cursor_secret="$(random_hex)" +jwt_secret_1="$(random_hex)" +jwt_secret_2="$(random_hex)" +bridge_secret="$(random_hex)" +export QDL_BETA_IMAGE="${image_id}" +export QDL_BETA_REDIS_IMAGE="${redis_id}" +export QDL_BETA_INIT_IMAGE="${redis_id}" +export QDL_BETA_CURSOR_KEYS_JSON="{\"beta-k1\":\"${cursor_secret}\"}" +export QDL_BETA_CURSOR_ACTIVE_KEY_ID="beta-k1" +export QDL_BETA_JWT_KEYS_JSON="{\"beta-jwt-k1\":\"${jwt_secret_1}\",\"beta-jwt-k2\":\"${jwt_secret_2}\"}" +export QDL_BETA_INTERNAL_INGEST_SECRET="${bridge_secret}" +export QDL_BETA_JWT_ISSUER="https://identity.qdl.phase90b.invalid" +export QDL_BETA_JWT_AUDIENCE="qdl-v2-phase90b" +export QDL_BETA_SOURCE_BINDINGS="/app/config/phase7/canary-sources.yaml" +export QDL_BETA_CONSUMER_MANIFESTS="/app/consumers/beta/phase7-monitoring-binance.yaml:/app/consumers/beta/phase7-paper-alpha-binance.yaml:/app/consumers/beta/phase7-capacity-binance.yaml" +export QDL_BETA_AUTHORITY_REVISION="1" +export QDL_BETA_BRIDGE_RUN_ONCE="false" + +temporary="$(mktemp -d)" +CERT_UID="${QDL_CERT_UID:-$(id -u)}" +CERT_GID="${QDL_CERT_GID:-$(id -g)}" +cleanup() { + docker compose -p "${MATRIX_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta down -v --remove-orphans >/dev/null 2>&1 || true + docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary \ + down -v --remove-orphans >/dev/null 2>&1 || true + docker image rm "${IMAGE_REF}" >/dev/null 2>&1 || true + rm -rf "${temporary}" +} +trap cleanup EXIT +trap 'printf "phase90b certification failed line=%s command=%s\n" "${LINENO}" "${BASH_COMMAND}" >&2' ERR + +wait_http() { + local url="$1" expected="$2" attempts="${3:-60}" code="000" + for ((index=1; index<=attempts; index++)); do + code="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 2 "${url}" || true)" + [[ "${code}" == "${expected}" ]] && return 0 + sleep 1 + done + printf 'timed out url=%s expected=%s actual=%s\n' "${url}" "${expected}" "${code}" >&2 + return 1 +} + +snapshot_v1() { + docker inspect "${V1_CONTAINER}" | python3 "${ROOT_DIR}/scripts/phase73_topology_snapshot.py" +} + +beta_keys_in_v1() { + if ! docker inspect "${PROD_REDIS_CONTAINER}" >/dev/null 2>&1; then printf '0\n'; return; fi + docker exec "${PROD_REDIS_CONTAINER}" redis-cli --scan --pattern 'qdl:beta:v2:*' | wc -l +} + +query_beta() { + local output="$1" token="$2" + curl -sS --max-time 10 -o "${output}" -w '%{http_code}' \ + -H "Authorization: Bearer ${token}" \ + -H 'X-QDL-Consumer-ID: phase7-capacity-binance' \ + -H 'X-QDL-Purpose: INTERNAL_RESEARCH' \ + --get "http://127.0.0.1:${QUERY_PORT}/v2/market-data/a953e16e-7138-5562-b5e8-c337a44d0b65/warmup" \ + --data-urlencode 'feed=BAR' \ + --data-urlencode 'consumer_grade=RESEARCH' \ + --data-urlencode 'source_policy_id=alpha_crypto_primary_v1' \ + --data-urlencode 'interval=1m' \ + --data-urlencode 'limit=30' \ + --data-urlencode 'max_freshness_ms=180000' \ + --data-urlencode 'require_full_coverage=true' \ + --data-urlencode 'require_final_bars=true' \ + --data-urlencode 'stale_policy=BLOCK' \ + --data-urlencode 'gap_policy=BLOCK' \ + --data-urlencode 'recovery=SNAPSHOT_AND_REPLAY' \ + --data-urlencode 'bar_revision_policy=EMIT_REVISIONS' +} + +snapshot_v1 >"${temporary}/v1-before.json" +openapi_before="$(curl -fsS --max-time 10 http://127.0.0.1:8100/openapi.json | sha256sum | cut -d' ' -f1)" +keys_before="$(beta_keys_in_v1)" + +QDL_BETA_PROJECT="${MATRIX_PROJECT}" \ +QDL_BETA_CONFIG_REVISION="phase90b-matrix-1" \ +QDL_BETA_REDIS_PREFIX="qdl:beta:v2:paper:phase90b:matrix" \ +QDL_BETA_CONSUMER_GROUP="qdl-v2-beta-phase90b-matrix" \ +QDL_BETA_LEASE_SHARD_ID="stream-v2-phase90b-matrix" \ +QDL_BETA_QUERY_HOST_PORT=18210 \ +QDL_BETA_STREAM_A_HEALTH_PORT=18211 \ +QDL_BETA_STREAM_B_HEALTH_PORT=18212 \ +QDL_BETA_STREAM_A_GRPC_PORT=18213 \ +QDL_BETA_STREAM_B_GRPC_PORT=18214 \ +QDL_PHASE73_CAPACITY_OUTPUT="${CAPACITY_OUTPUT}" \ +QDL_PHASE73_SECURITY_OUTPUT="${SECURITY_OUTPUT}" \ +"${ROOT_DIR}/scripts/phase73_public_beta_certification.sh" + +export QDL_BETA_CONFIG_REVISION="phase90b-continuous-1" +export QDL_BETA_REDIS_PREFIX="qdl:beta:v2:paper:phase90b:continuous" +export QDL_BETA_CONSUMER_GROUP="qdl-v2-beta-phase90b-continuous" +export QDL_BETA_LEASE_SHARD_ID="stream-v2-phase90b-continuous" +export QDL_BETA_QUERY_HOST_PORT="${QUERY_PORT}" +export QDL_BETA_STREAM_A_HEALTH_PORT="${STREAM_A_HEALTH_PORT}" +export QDL_BETA_STREAM_B_HEALTH_PORT="${STREAM_B_HEALTH_PORT}" +export QDL_BETA_STREAM_A_GRPC_PORT="${STREAM_A_GRPC_PORT}" +export QDL_BETA_STREAM_B_GRPC_PORT="${STREAM_B_GRPC_PORT}" + +docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary config --quiet +docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary up -d +wait_http "http://127.0.0.1:${QUERY_PORT}/health/ready" 200 60 +wait_http "http://127.0.0.1:${STREAM_A_HEALTH_PORT}/health/live" 200 30 +wait_http "http://127.0.0.1:${STREAM_B_HEALTH_PORT}/health/live" 200 30 + +status_a="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 2 "http://127.0.0.1:${STREAM_A_HEALTH_PORT}/health/ready" || true)" +status_b="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 2 "http://127.0.0.1:${STREAM_B_HEALTH_PORT}/health/ready" || true)" +if [[ "${status_a}:${status_b}" == "200:503" ]]; then + active_grpc_port="${STREAM_A_GRPC_PORT}" +elif [[ "${status_a}:${status_b}" == "503:200" ]]; then + active_grpc_port="${STREAM_B_GRPC_PORT}" +else + printf 'expected exactly one active stream gateway, got A=%s B=%s\n' "${status_a}" "${status_b}" >&2 + exit 1 +fi + +bridge_id="$(docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary ps -q qdl_beta_v1_bridge)" +[[ -n "${bridge_id}" ]] +docker inspect "${bridge_id}" >"${temporary}/bridge-inspect.json" +docker image inspect "${image_id}" >"${temporary}/image-inspect.json" + +token="$(docker run --rm --network none \ + -e QDL_BETA_JWT_KEYS_JSON -e QDL_BETA_JWT_ISSUER -e QDL_BETA_JWT_AUDIENCE \ + "${image_id}" python /app/scripts/phase73_token.py)" +code="000" +for ((attempt=1; attempt<=60; attempt++)); do + code="$(query_beta "${temporary}/v2-candidate.json" "${token}" || true)" + if [[ "${code}" == "200" ]]; then + count="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("count",0))' "${temporary}/v2-candidate.json")" + [[ "${count}" -ge 30 ]] && break + fi + sleep 1 +done +[[ "${code}" == "200" && "${count:-0}" -ge 30 ]] + +previous_watermark="" +for ((attempt=1; attempt<=15; attempt++)); do + sleep 2 + code="$(query_beta "${temporary}/v2-stable.json" "${token}" || true)" + [[ "${code}" == "200" ]] || continue + watermark="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["watermark_offset"])' "${temporary}/v2-stable.json")" + if [[ -n "${previous_watermark}" && "${watermark}" == "${previous_watermark}" ]]; then + cp "${temporary}/v2-stable.json" "${temporary}/v2-first.json" + break + fi + previous_watermark="${watermark}" +done +[[ -f "${temporary}/v2-first.json" ]] +curl -fsS --max-time 10 'http://127.0.0.1:8100/v1/crypto/ohlcv/binance/BTCUSDT/1m?limit=120&market=usdm' >"${temporary}/v1-first.json" +sleep 12 +[[ "$(query_beta "${temporary}/v2-second.json" "${token}")" == "200" ]] +curl -fsS --max-time 10 'http://127.0.0.1:8100/v1/crypto/ohlcv/binance/BTCUSDT/1m?limit=120&market=usdm' >"${temporary}/v1-second.json" + +docker run --rm --network none --read-only --cap-drop ALL \ + --security-opt no-new-privileges:true --pids-limit 64 --memory 256m --cpus 0.5 \ + --user "${CERT_UID}:${CERT_GID}" --tmpfs "/tmp:rw,noexec,nosuid,nodev,size=16m,uid=${CERT_UID},gid=${CERT_GID}" \ + -v "${temporary}:/evidence" "${image_id}" \ + python /app/scripts/phase90b_bridge_parity.py \ + --source-bindings /app/config/phase7/canary-sources.yaml \ + --v1-first /evidence/v1-first.json --v2-first /evidence/v2-first.json \ + --v1-second /evidence/v1-second.json --v2-second /evidence/v2-second.json \ + --output /evidence/parity.json +cp "${temporary}/parity.json" "${PARITY_OUTPUT}" + +mapfile -t beta_ids < <(docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" --profile phase7-beta --profile phase7-canary ps -q) +: >"${temporary}/stats.jsonl" +docker stats --no-stream --format '{{json .}}' "${beta_ids[@]}" >>"${temporary}/stats.jsonl" + +docker compose -p "${BRIDGE_PROJECT}" -f "${COMPOSE_FILE}" \ + --profile phase7-beta --profile phase7-canary down -v --remove-orphans +containers_after="$(docker ps -aq --filter label=com.docker.compose.project="${BRIDGE_PROJECT}" | wc -l)" +networks_after="$(docker network ls -q --filter label=com.docker.compose.project="${BRIDGE_PROJECT}" | wc -l)" +volumes_after="$(docker volume ls -q --filter label=com.docker.compose.project="${BRIDGE_PROJECT}" | wc -l)" +docker image rm "${IMAGE_REF}" >/dev/null +images_after="$(docker image ls -q "${IMAGE_REF}" | wc -l)" +snapshot_v1 >"${temporary}/v1-after.json" +openapi_after="$(curl -fsS --max-time 10 http://127.0.0.1:8100/openapi.json | sha256sum | cut -d' ' -f1)" +keys_after="$(beta_keys_in_v1)" + +python3 "${ROOT_DIR}/scripts/phase90b_finalize_evidence.py" \ + --capacity "${CAPACITY_OUTPUT}" --security "${SECURITY_OUTPUT}" \ + --parity "${PARITY_OUTPUT}" --image-inspect "${temporary}/image-inspect.json" \ + --bridge-inspect "${temporary}/bridge-inspect.json" --stats "${temporary}/stats.jsonl" \ + --v1-before "${temporary}/v1-before.json" --v1-after "${temporary}/v1-after.json" \ + --openapi-before "${openapi_before}" --openapi-after "${openapi_after}" \ + --production-keys-before "${keys_before}" --production-keys-after "${keys_after}" \ + --containers-after "${containers_after}" --networks-after "${networks_after}" \ + --volumes-after "${volumes_after}" --images-after "${images_after}" \ + --output "${RESULT_OUTPUT}" --report "${REPORT_OUTPUT}" +capacity_rel="$(realpath --relative-to="${ROOT_DIR}" "${CAPACITY_OUTPUT}")" +security_rel="$(realpath --relative-to="${ROOT_DIR}" "${SECURITY_OUTPUT}")" +parity_rel="$(realpath --relative-to="${ROOT_DIR}" "${PARITY_OUTPUT}")" +result_rel="$(realpath --relative-to="${ROOT_DIR}" "${RESULT_OUTPUT}")" +report_rel="$(realpath --relative-to="${ROOT_DIR}" "${REPORT_OUTPUT}")" +( + cd "${ROOT_DIR}" + sha256sum "${capacity_rel}" "${security_rel}" "${parity_rel}" \ + "${result_rel}" "${report_rel}" +) >"${CHECKSUM_OUTPUT}" +(cd "${ROOT_DIR}" && sha256sum -c "${CHECKSUM_OUTPUT}") diff --git a/scripts/phase90c_migration_smoke.sh b/scripts/phase90c_migration_smoke.sh new file mode 100755 index 0000000..9bd6927 --- /dev/null +++ b/scripts/phase90c_migration_smoke.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +POSTGRES_IMAGE="${QDL_PHASE90C_POSTGRES_IMAGE:-timescale/timescaledb:latest-pg15}" +CONTAINER="${QDL_PHASE90C_POSTGRES_CONTAINER:-qdl_phase90c_postgres_$$}" +OUTPUT="${QDL_PHASE90C_MIGRATION_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase90c-authority-migration.json}" +CANDIDATE_DIGEST="${QDL_PHASE90C_CANDIDATE_DIGEST:-$( + PYTHONPATH="${ROOT_DIR}" python3 -c "from qdl.certification.prerequisites import CandidateSlice; print(CandidateSlice.load('${ROOT_DIR}/config/phase9/candidate-slice.yaml').digest)" +)}" + +cleanup() { + docker rm -f "${CONTAINER}" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +docker run -d --name "${CONTAINER}" --network none \ + --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.0 \ + --tmpfs /var/lib/postgresql/data:rw,nosuid,nodev,size=512m \ + -e POSTGRES_HOST_AUTH_METHOD=trust \ + -v "${ROOT_DIR}/migrations/postgres:/migrations:ro" \ + "${POSTGRES_IMAGE}" >/dev/null + +for _ in $(seq 1 60); do + if docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null 2>&1; then + break + fi + sleep 1 +done +docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null + +apply_migrations() { + local migration + for migration in "${ROOT_DIR}"/migrations/postgres/*.sql; do + docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + -f "/migrations/$(basename "${migration}")" >/dev/null + done +} + +apply_migrations + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +INSERT INTO qdl_authority_slices ( + slice_id, environment, venue, market, product_type, feed, + partition_plan_epoch, partition_id, schema_major, state, + authority_revision, owner_id, lease_epoch, candidate_digest, + artifact_image_digest, sbom_digest, signature_identity, contract_digest, + normalizer_version, adapter_version, config_revision, + instrument_catalog_revision, source_policy_revision, + partition_plan_digest, rollback_manifest_digest +) VALUES ( + 'production/binance/usdm/perpetual/trade/plan-1/btcusdt', + 'production', 'BINANCE', 'USDM', 'PERPETUAL', 'TRADE', + 1, 'rendezvous-sha256-v1:epoch-1:btcusdt', 2, 'RUST_SHADOW', + 1, 'rust-shadow-owner', 1, '${CANDIDATE_DIGEST}', + 'sha256:${CANDIDATE_DIGEST}', repeat('1',64), 'phase90c-test-signer', repeat('2',64), + 'qdl-rust-core/test', 'binance-usdm/test', 'phase90c-test-config', + 'phase90c-test-catalog', 'phase90c-test-source-policy', + repeat('3',64), repeat('4',64) +); + +SELECT (qdl_transition_authority( + '11111111-1111-4111-8111-111111111111', + 'production/binance/usdm/perpetual/trade/plan-1/btcusdt', + 'RUST_SHADOW', 1, 'rust-shadow-owner', 1, 1, + 'VALIDATING', 'rust-shadow-owner', 1, NULL, NULL, NULL, + 'phase90c-test', 'enter isolated validation' +)).state; + +INSERT INTO qdl_production_prerequisite_bundles ( + bundle_id, candidate_digest, policy_revision, decision, evidence, + evidence_sha256, issued_by, issued_at, expires_at +) VALUES + ('00000000-0000-4000-8000-000000000001', '${CANDIDATE_DIGEST}', 1, + 'NO_GO_EXTERNAL', '{}', repeat('5',64), 'phase90c-test', clock_timestamp(), + clock_timestamp() + interval '1 day'), + ('00000000-0000-4000-8000-000000000002', repeat('9',64), 1, + 'GO', '{}', repeat('6',64), 'phase90c-test', clock_timestamp(), + clock_timestamp() + interval '1 day'), + ('00000000-0000-4000-8000-000000000003', '${CANDIDATE_DIGEST}', 1, + 'GO', '{}', repeat('7',64), 'phase90c-test', clock_timestamp(), + clock_timestamp() + interval '1 day'); +SQL + +expect_failure() { + if docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + -c "$1" >/dev/null 2>&1; then + printf 'expected SQL failure but statement succeeded\n' >&2 + exit 1 + fi +} + +expect_failure "SELECT qdl_transition_authority('31111111-1111-4111-8111-111111111111','production/binance/usdm/perpetual/trade/plan-1/btcusdt','RUST_SHADOW',1,'rust-shadow-owner',1,1,'VALIDATING','rust-shadow-owner',1,NULL,NULL,NULL,'phase90c-test','stale CAS');" +expect_failure "SELECT qdl_transition_authority('32222222-2222-4222-8222-222222222222','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000001',clock_timestamp()+interval '1 hour','phase90c-test','blocked bundle');" +expect_failure "SELECT qdl_transition_authority('33333333-3333-4333-8333-333333333333','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000002',clock_timestamp()+interval '1 hour','phase90c-test','wrong candidate');" + +expect_failure "SELECT qdl_transition_authority('35555555-5555-4555-8555-555555555555','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,NULL,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '1 hour','phase90c-test','missing terminal watermark');" +expect_failure "SELECT qdl_transition_authority('36666666-6666-4666-8666-666666666666','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000003',NULL,'phase90c-test','missing hold window');" +expect_failure "SELECT qdl_transition_authority('37777777-7777-4777-8777-777777777777','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '2 days','phase90c-test','hold exceeds evidence expiry');" + +expect_failure "SELECT qdl_transition_authority('38888888-8888-4888-8888-888888888888','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'BLOCKED','rust-shadow-owner',1,NULL,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '1 hour','phase90c-test','bundle on non-authority state');" + +docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + -c "SELECT qdl_transition_authority('22222222-2222-4222-8222-222222222222','production/binance/usdm/perpetual/trade/plan-1/btcusdt','VALIDATING',2,'rust-shadow-owner',1,1,'RUST_CANARY','rust-canary-owner',2,123456,'00000000-0000-4000-8000-000000000003',clock_timestamp()+interval '1 hour','phase90c-test','approved isolated CAS test');" >/dev/null + +expect_failure "SELECT qdl_transition_authority('34444444-4444-4444-8444-444444444444','production/binance/usdm/perpetual/trade/plan-1/btcusdt','RUST_CANARY',3,'rust-shadow-owner',2,1,'BLOCKED','rust-shadow-owner',2,NULL,NULL,NULL,'phase90c-test','stale owner');" +expect_failure "UPDATE qdl_authority_transition_audit SET reason='mutated';" + +apply_migrations + +state="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT state || ':' || authority_revision || ':' || owner_id || ':' || lease_epoch FROM qdl_authority_slices;")" +audit_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_transition_audit;")" +bundle_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_production_prerequisite_bundles;")" +[[ "${state}" == "RUST_CANARY:3:rust-canary-owner:2" ]] +[[ "${audit_count}" == "2" ]] +[[ "${bundle_count}" == "3" ]] + +python3 - "${OUTPUT}" "${state}" "${audit_count}" "${bundle_count}" <<'PY' +import json, pathlib, sys +output, state, audits, bundles = sys.argv[1:] +path = pathlib.Path(output); path.parent.mkdir(parents=True, exist_ok=True) +path.write_text(json.dumps({ + "schema": "qdl.phase9.0-c.authority-migration.v1", + "status": "PASS", + "final_test_state": state, + "audit_records": int(audits), + "prerequisite_bundles": int(bundles), + "stale_cas_rejected": True, + "no_go_bundle_rejected": True, + "candidate_mismatch_rejected": True, + "missing_terminal_watermark_rejected": True, + "missing_hold_window_rejected": True, + "hold_beyond_bundle_expiry_rejected": True, + "bundle_on_non_authority_state_rejected": True, + "stale_owner_rejected": True, + "audit_mutation_rejected": True, + "idempotent_migration": True, + "production_mutations": 0, +}, indent=2, sort_keys=True) + "\n") +PY + +cleanup +trap - EXIT +[[ -z "$(docker ps -aq --filter name=^/${CONTAINER}$)" ]] +printf '{"status":"PASS","state":"%s","audits":%s,"cleanup":true}\n' \ + "${state}" "${audit_count}" diff --git a/scripts/phase90c_prerequisite_certification.py b/scripts/phase90c_prerequisite_certification.py new file mode 100755 index 0000000..b58fcda --- /dev/null +++ b/scripts/phase90c_prerequisite_certification.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import pathlib +import subprocess +import sys +import time +import urllib.request +import uuid + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.certification.prerequisites import ( # noqa: E402 + CandidateSlice, + PrerequisitePolicy, + evaluate_prerequisites, + load_inventory, +) + + +def _v1_snapshot() -> dict[str, object]: + completed = subprocess.run( + ["docker", "inspect", "data_layer_service", "--format", "{{json .}}"], + text=True, capture_output=True, check=False, + ) + if completed.returncode != 0 or not completed.stdout.strip(): + return {"present": False} + payload = json.loads(completed.stdout) + return { + "present": True, + "id": payload["Id"], + "image": payload["Config"]["Image"], + "started_at": payload["State"]["StartedAt"], + "restart_count": payload["RestartCount"], + "networks": sorted(payload["NetworkSettings"]["Networks"]), + "mounts": sorted((item["Source"], item["Destination"], item["RW"]) for item in payload["Mounts"]), + } + + +def _v1_health() -> int: + try: + with urllib.request.urlopen("http://127.0.0.1:8100/v1/health", timeout=5) as response: + return response.status + except Exception: + return 0 + + +def _write_report(path: pathlib.Path, result: dict[str, object]) -> None: + blocked = [item for item in result["gates"] if not item["passed"]] + lines = [ + "# Phase 9.0-C Production Prerequisite Report", + "", + f"Decision: `{result['decision']}`", + "", + "## Candidate", + "", + f"- Slice: `{result['slice_id']}`", + f"- Candidate digest: `{result['candidate_digest']}`", + f"- Authority: `RUST_SHADOW`; V1 unchanged: `{result['v1_unchanged']}`", + "", + "## Gate Summary", + "", + f"- Passed: `{result['passed']}`", + f"- Blocked: `{result['blocked']}`", + "", + ] + if blocked: + lines += ["## Blocking Evidence", ""] + lines += [ + f"- `{item['gate_id']}`: `{item['reason']}` " + f"(observed `{item['observed_scope']}`, required `{item['required_scope']}`)" + for item in blocked + ] + lines += [ + "", + "These are real infrastructure/operator blockers. Same-host fixtures or", + "local rehearsals must not be relabeled to close them.", + ] + path.write_text("\n".join(lines) + "\n") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--policy", default="config/phase9/production-prerequisite-policy.yaml") + parser.add_argument("--candidate", default="config/phase9/candidate-slice.yaml") + parser.add_argument("--inventory", default="config/phase9/evidence-inventory.local.yaml") + parser.add_argument("--output", default="upgrade/evidence/phase90c-production-prerequisites.json") + parser.add_argument("--report", default="upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md") + parser.add_argument("--expect", choices=("GO", "NO_GO_EXTERNAL"), required=True) + args = parser.parse_args() + + policy = PrerequisitePolicy.load(ROOT / args.policy) + candidate = CandidateSlice.load(ROOT / args.candidate) + inventory_path = ROOT / args.inventory + inventory = load_inventory(inventory_path) + before = _v1_snapshot() + health_before = _v1_health() + decision = evaluate_prerequisites( + policy, candidate, inventory, repository_root=ROOT, now_ns=time.time_ns() + ).as_mapping() + after = _v1_snapshot() + health_after = _v1_health() + inventory_digest = hashlib.sha256(inventory_path.read_bytes()).hexdigest() + bundle_seed = f"{candidate.digest}:{policy.revision}:{inventory_digest}" + result = { + **decision, + "bundle_id": str(uuid.uuid5(uuid.NAMESPACE_URL, bundle_seed)), + "slice_id": candidate.payload["slice_id"], + "authority_state": candidate.payload["authority_state"], + "inventory_sha256": inventory_digest, + "issued_at_ns": time.time_ns(), + "v1_health_before": health_before, + "v1_health_after": health_after, + "v1_unchanged": before == after and health_before == health_after == 200, + "v1_before": before, + "v1_after": after, + "production_mutations": 0, + } + if not result["v1_unchanged"]: + raise RuntimeError("V1 topology or health changed during prerequisite evaluation") + output = ROOT / args.output + report = ROOT / args.report + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n") + _write_report(report, result) + print(json.dumps({ + "decision": result["decision"], "passed": result["passed"], + "blocked": result["blocked"], "v1_unchanged": result["v1_unchanged"], + }, sort_keys=True)) + return 0 if result["decision"] == args.expect else 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase91_canary_certification.py b/scripts/phase91_canary_certification.py new file mode 100755 index 0000000..087b036 --- /dev/null +++ b/scripts/phase91_canary_certification.py @@ -0,0 +1,570 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import base64 +import gzip +import hashlib +import json +import os +import pathlib +import tempfile +import time +from collections import Counter + +import yaml + +os.environ.setdefault("QDL_PHASE8_PROJECT", "qdl_phase91_certification") +from phase80_broker_certification import ( # noqa: E402 + BOOTSTRAP, + PROJECT, + ROOT, + add_acls, + cleanup, + compose, + consume, + create_topic, + kafka, + run, + total_end_offset, + v1_health, + v1_topology, + wait_for_cluster, + wait_for_replicas, +) + + +CAPTURE = ROOT / "upgrade/evidence/captures/phase8-real-provider-frames.json.gz" +CAPTURE_EVIDENCE = ROOT / "upgrade/evidence/phase8-real-provider-shadow.json" +CANDIDATE = ROOT / "config/phase9/candidate-slice.yaml" +PREREQUISITE_DECISION = ROOT / "upgrade/evidence/phase90c-production-prerequisites.json" +OUTPUT = ROOT / "upgrade/evidence/phase91-rust-canary-certification.json" +REPORT = ROOT / "upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md" +CHECKSUM = ROOT / "upgrade/evidence/phase91-evidence.sha256" +BUNDLE_PATH = ROOT / "target/phase91-authentic-replay.json" + + +def candidate_digest(payload: dict) -> str: + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest() + + +def consume_compacted_records(env: dict[str, str], topic: str) -> list[str]: + offsets = kafka( + env, + "kafka-get-offsets.sh", + "--bootstrap-server", + BOOTSTRAP, + "--command-config", + "/etc/kafka/secrets/admin.properties", + "--topic", + topic, + "--time", + "-1", + ) + records: list[str] = [] + for line in offsets.stdout.splitlines(): + fields = line.rsplit(":", 2) + if len(fields) != 3 or not fields[1].isdigit() or not fields[2].isdigit(): + continue + partition, end_offset = int(fields[1]), int(fields[2]) + if end_offset <= 0: + continue + result = kafka( + env, + "kafka-console-consumer.sh", + "--bootstrap-server", + BOOTSTRAP, + "--consumer.config", + "/etc/kafka/secrets/consumer.properties", + "--topic", + topic, + "--partition", + str(partition), + "--offset", + "earliest", + "--timeout-ms", + "5000", + check=False, + timeout=20.0, + ) + records.extend( + item.strip() + for item in result.stdout.splitlines() + if item.strip().startswith("{") and item.strip().endswith("}") + ) + if not records: + raise RuntimeError(f"compacted topic has offsets but no readable records: {topic}") + return records + + +def authentic_fixtures(*, repeat: int) -> tuple[dict, dict]: + compressed = CAPTURE.read_bytes() + capture_evidence = json.loads(CAPTURE_EVIDENCE.read_text()) + actual_digest = hashlib.sha256(compressed).hexdigest() + if actual_digest != capture_evidence["capture_bundle_sha256"]: + raise RuntimeError("frozen authentic capture checksum mismatch") + payload = json.loads(gzip.decompress(compressed)) + if ( + payload.get("schema") != "qdl.phase8.authentic-capture-bundle.v1" + or payload.get("provenance") != "REAL_PROVIDER_READ_ONLY" + or payload.get("production_writes") != 0 + ): + raise RuntimeError("authentic capture provenance is invalid") + fixtures = [] + for sequence, item in enumerate( + ( + record + for record in payload["captures"] + if record.get("provider") == "BINANCE_DIRECT" + and record.get("venue") == "BINANCE" + and record.get("market") == "USDM" + and record.get("native_symbol") == "BTCUSDT" + and record.get("test_provenance") is False + ), + start=1, + ): + raw_frame = base64.b64decode(item["raw_frame_base64"], validate=True) + if len(raw_frame) != item["raw_frame_bytes"]: + raise RuntimeError("authentic raw frame length mismatch") + if hashlib.sha256(raw_frame).hexdigest() != item["raw_frame_sha256"]: + raise RuntimeError("authentic raw frame checksum mismatch") + wrapper = json.loads(raw_frame) + if wrapper.get("stream") != item["native_channel"] or not isinstance(wrapper.get("data"), dict): + raise RuntimeError("authentic Binance combined frame is malformed") + received = int(item["received_at_ns"]) + fixtures.append({ + "provider_kind": "binance_usdm_trade", + "context": { + "instrument_uid": "85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c", + "instrument_id": "BINANCE.USDM.PERPETUAL.BTCUSDT", + "instrument_revision": 1, + "venue": "BINANCE", + "market": "USDM", + "product_type": "PERPETUAL", + "native_symbol": "BTCUSDT", + "provider": "BINANCE_DIRECT", + "source_id": "binance-usdm-phase91-canary", + "lease_epoch": 2, + "received_at_ns": received, + "normalized_at_ns": received + 1, + "published_at_ns": received + 2, + "partition_sequence": sequence, + "normalizer_version": "qdl-normalizer/2.0.0-phase91", + "adapter_version": "binance-usdm/2.0.0-shadow", + "config_revision": 1, + "correlation_id": "phase91-authentic-replay", + "source_session_id": item["source_session_id"], + "connection_generation": int(item["connection_generation"]), + "authority_revision": 2, + "partition_plan_epoch": 1, + "raw_capture_id": list(bytes.fromhex(item["capture_id"])), + "raw_frame_sha256": list(bytes.fromhex(item["raw_frame_sha256"])), + }, + "raw": wrapper["data"], + }) + if len(fixtures) < 32: + raise RuntimeError(f"too few authentic Binance frames: {len(fixtures)}") + BUNDLE_PATH.parent.mkdir(parents=True, exist_ok=True) + BUNDLE_PATH.write_text(json.dumps( + {"fixtures": fixtures, "repeat": repeat}, + sort_keys=True, + separators=(",", ":"), + )) + return { + "path": str(BUNDLE_PATH), + "fixtures": len(fixtures), + "repeat": repeat, + "events": len(fixtures) * repeat, + "capture_sha256": actual_digest, + "capture_provenance": payload["provenance"], + "raw_checks_passed": len(fixtures), + }, {"fixtures": fixtures, "repeat": repeat} + + +def parity(image: str, repeat: int) -> dict: + capture, _ = authentic_fixtures(repeat=repeat) + mount = f"type=bind,source={ROOT},target=/app,readonly" + python_result = run([ + "docker", "run", "--rm", "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", mount, + "--entrypoint", "python", + "data-layer:v0.1.0", + "/app/scripts/phase91_python_parity.py", + "/app/target/phase91-authentic-replay.json", + ], timeout=240) + python_metrics = json.loads(python_result.stdout.strip().splitlines()[-1]) + rust_runs = [] + for _ in range(3): + result = run([ + "docker", "run", "--rm", "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", mount, + "--entrypoint", "/usr/local/bin/qdl-parity-replay", + image, + "/app/target/phase91-authentic-replay.json", + ], timeout=240) + rust_runs.append(json.loads(result.stdout.strip().splitlines()[-1])) + expected = python_metrics["aggregate_sha256"] + mismatch_runs = sum( + item.get("aggregate_sha256") != expected + or item.get("record_sha256") != python_metrics["record_sha256"] + for item in rust_runs + ) + if mismatch_runs: + raise RuntimeError(f"authentic Python/Rust parity diverged in {mismatch_runs} runs") + return { + "status": "PASS", + "capture": capture, + "python": python_metrics, + "rust_clean_process_runs": len(rust_runs), + "rust_events_per_second_min": min(item["events_per_second"] for item in rust_runs), + "aggregate_sha256": expected, + "semantic_mismatches": 0, + "process_restart_mismatches": 0, + } + + +def rust_command( + *, image: str, cert_dir: str, entrypoint: str, env_values: dict[str, str] +) -> list[str]: + command = [ + "docker", "run", "--rm", + "--network", f"{PROJECT}_phase8_shadow", + "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", f"type=bind,source={cert_dir},target=/certs,readonly", + "--entrypoint", entrypoint, + ] + for key, value in env_values.items(): + command.extend(("--env", f"{key}={value}")) + command.append(image) + return command + + +def broker_rehearsal(image: str, image_digest: str, candidate: dict, decision: dict) -> dict: + topics = { + "authority": "qdl.phase8.phase91.control.authority.v2", + "audit": "qdl.phase8.phase91.audit.authority.v2", + "shadow_raw": "qdl.phase8.phase91.shadow.raw.v2", + "shadow": "qdl.phase8.phase91.shadow.canonical.v2", + "canary": "qdl.phase8.phase91.canary.canonical.v2", + "public": "qdl.phase8.phase91.public.must-remain-empty", + "legacy": "qdl.phase8.phase91.legacy.must-remain-empty", + "transport": "qdl.phase8.phase91.transport", + } + v1_before = v1_topology() + health_before = v1_health() + cleanup_result: dict[str, object] = {} + result: dict[str, object] = {} + with tempfile.TemporaryDirectory(prefix="qdl-phase91-certs-") as cert_dir: + env = os.environ.copy() + env["QDL_PHASE8_CERT_DIR"] = cert_dir + run([str(ROOT / "scripts/phase80_generate_tls.sh"), cert_dir], env=env, timeout=120) + try: + compose(env, "down", "--volumes", "--remove-orphans", check=False, timeout=120) + compose(env, "up", "-d", timeout=180) + wait_for_cluster(env) + # KRaft may answer one metadata request before every controller/broker + # has remained stable. Require a short stable window before topic I/O. + time.sleep(10.0) + wait_for_cluster(env) + running = set( + compose(env, "ps", "--status", "running", "--services").stdout.splitlines() + ) + if not {"kafka1", "kafka2", "kafka3"}.issubset(running): + raise RuntimeError(f"Kafka brokers did not survive stability window: {sorted(running)}") + for name, topic in topics.items(): + create_topic( + env, + topic, + partitions=1 if name in {"authority", "audit"} else 3, + cleanup_policy="compact" if name == "authority" else "delete", + ) + add_acls(env) + wait_for_replicas(env) + before = {name: total_end_offset(env, topic) for name, topic in topics.items()} + nonce = hashlib.sha256(str(time.time_ns()).encode()).hexdigest()[:16] + runtime_result = run(rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-phase91-canary-rehearsal", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": BOOTSTRAP, + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_AUTHORITY_TOPIC": topics["authority"], + "QDL_AUDIT_TOPIC": topics["audit"], + "QDL_SHADOW_RAW_TOPIC": topics["shadow_raw"], + "QDL_SHADOW_CANONICAL_TOPIC": topics["shadow"], + "QDL_CANARY_CANONICAL_TOPIC": topics["canary"], + "QDL_PUBLIC_TOPIC": topics["public"], + "QDL_LEGACY_TOPIC": topics["legacy"], + "QDL_AUTHORITY_NONCE": nonce, + "QDL_CANDIDATE_DIGEST": candidate_digest(candidate), + "QDL_PREREQUISITE_BUNDLE_ID": decision["bundle_id"], + "QDL_SLICE_ID": candidate["slice_id"], + "QDL_SHADOW_OWNER_ID": candidate["owner_id"], + "QDL_CANARY_OWNER_ID": "rust-canary-binance-usdm-shard-0", + }, + ), timeout=180) + runtime = json.loads(runtime_result.stdout.strip().splitlines()[-1]) + after = {name: total_end_offset(env, topic) for name, topic in topics.items()} + deltas = {name: after[name] - before[name] for name in topics} + expected_deltas = { + "authority": 4, + "audit": 4, + "shadow_raw": 0, + "shadow": 2, + "canary": 64, + "public": 0, + "legacy": 0, + "transport": 0, + } + if runtime.get("status") != "PASS" or deltas != expected_deltas: + raise RuntimeError(f"Phase 9.1 runtime/offset mismatch runtime={runtime} deltas={deltas}") + + compose(env, "stop", "kafka3") + one_loss_before = total_end_offset(env, topics["transport"]) + one_loss = run(rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092,kafka2:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-one-loss", + }, + ), timeout=90) + one_loss_after = total_end_offset(env, topics["transport"]) + one_replica_loss_acked = one_loss.returncode == 0 and one_loss_after == one_loss_before + 1 + min_isr_before = one_loss_after + compose(env, "stop", "kafka2") + min_isr = run(rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-min-isr", + }, + ), check=False, timeout=35) + min_isr_client_failed = min_isr.returncode != 0 + compose(env, "start", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + min_isr_after = total_end_offset(env, topics["transport"]) + min_isr_failed_closed = ( + min_isr_client_failed and min_isr_after == min_isr_before + ) + if not one_replica_loss_acked or not min_isr_failed_closed: + raise RuntimeError( + f"broker durability gate failed one_loss={one_replica_loss_acked} " + f"min_isr={min_isr_failed_closed} " + f"min_isr_returncode={min_isr.returncode} " + f"offset_before={min_isr_before} offset_after={min_isr_after} " + f"stderr={min_isr.stderr[-800:]}" + ) + compose(env, "restart", "kafka1", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + + audit = [json.loads(item) for item in consume( + env, topics["audit"], 4, f"phase8-phase91-audit-{nonce}" + )] + authority = [ + json.loads(item) + for item in consume_compacted_records(env, topics["authority"]) + ] + slow_started = time.monotonic() + time.sleep(1.0) + canary = [json.loads(item) for item in consume( + env, topics["canary"], 64, f"phase8-phase91-slow-{nonce}", timeout=60 + )] + catchup_seconds = time.monotonic() - slow_started + audit_states = [item["state"] for item in audit] + latest = max(authority, key=lambda item: item["authority_revision"]) + canary_watermarks = [item["source_watermark"] for item in canary] + if audit_states != ["RUST_SHADOW", "RUST_CANARY", "BLOCKED", "RUST_SHADOW"]: + raise RuntimeError(f"authority audit order diverged: {audit_states}") + if latest["state"] != "RUST_SHADOW" or latest["authority_revision"] != 4: + raise RuntimeError(f"latest compacted authority diverged: {latest}") + if canary_watermarks != list(range(102, 166)): + raise RuntimeError("slow consumer catch-up changed canary order or coverage") + result = { + "status": "PASS", + "schema": "qdl.phase91.broker-rehearsal.v1", + "mode": "ISOLATED_REHEARSAL", + "production_authorized": False, + "image_digest": image_digest, + "runtime": runtime, + "offset_deltas": deltas, + "one_replica_loss_acked": one_replica_loss_acked, + "min_isr_failed_closed": min_isr_failed_closed, + "authority_audit_states_after_restart": audit_states, + "latest_authority_after_restart": { + "state": latest["state"], + "revision": latest["authority_revision"], + "owner_id": latest["owner_id"], + "lease_epoch": latest["lease_epoch"], + }, + "slow_consumer": { + "delayed_seconds": 1.0, + "catchup_seconds": catchup_seconds, + "records": len(canary), + "ordered_gap_free": True, + }, + "public_writes": 0, + "legacy_writes": 0, + "final_authority": "RUST_SHADOW", + } + except Exception as error: + status = compose(env, "ps", "--all", check=False, timeout=30) + logs = compose(env, "logs", "--no-color", "--tail", "240", check=False, timeout=60) + diagnostic = (status.stdout + status.stderr + logs.stdout + logs.stderr)[-12000:] + raise RuntimeError( + f"Phase 9.1 broker rehearsal failed before cleanup: {error}\n" + f"isolated broker diagnostics:\n{diagnostic}" + ) from error + finally: + cleanup_result = cleanup(env) + v1_after = v1_topology() + health_after = v1_health() + cleanup_result.update({ + "v1_health_before": health_before, + "v1_health_after": health_after, + "v1_topology_unchanged": v1_before == v1_after, + }) + result["cleanup"] = cleanup_result + if ( + health_before != 200 + or health_after != 200 + or not cleanup_result["v1_topology_unchanged"] + or any(cleanup_result[key] for key in ("containers_after", "networks_after", "volumes_after")) + ): + raise RuntimeError(f"Phase 9.1 cleanup/V1 invariant failed: {cleanup_result}") + return result + + +def render_report(evidence: dict) -> str: + parity = evidence["parity"] + broker = evidence["broker"] + capture = parity["capture"] + cleanup_result = broker["cleanup"] + return f"""# Phase 9.1 Rust Canary Certification Report + +## Decision + +- Status: `{evidence['status']}` +- Production authorized: `{str(evidence['production_authorized']).lower()}` +- Production mutations: `{evidence['production_mutations']}` +- Prerequisite decision: `{evidence['prerequisite_decision']}` +- Slice: `{evidence['slice_id']}` +- Candidate digest: `{evidence['candidate_digest']}` + +## Authentic Parity + +- Provenance: `{capture['capture_provenance']}` +- Frozen fixtures: `{capture['fixtures']}` +- Repetition: `{capture['repeat']}` +- Canonical events: `{capture['events']}` +- Semantic mismatches: `{parity['semantic_mismatches']}` +- Clean Rust process runs: `{parity['rust_clean_process_runs']}` +- Aggregate SHA-256: `{parity['aggregate_sha256']}` +- Python throughput: `{parity['python']['events_per_second']:.3f}` events/s +- Minimum Rust throughput: `{parity['rust_events_per_second_min']:.3f}` events/s + +## Authority And Broker Recovery + +- Transition audit: `{', '.join(broker['authority_audit_states_after_restart'])}` +- Final authority: `{broker['final_authority']}` +- One-replica-loss ACK: `{str(broker['one_replica_loss_acked']).lower()}` +- Below-min-ISR fail closed: `{str(broker['min_isr_failed_closed']).lower()}` +- Slow-consumer records: `{broker['slow_consumer']['records']}` +- Slow-consumer ordered and gap-free: `{str(broker['slow_consumer']['ordered_gap_free']).lower()}` +- Public writes: `{broker['public_writes']}` +- Legacy writes: `{broker['legacy_writes']}` + +## Isolation And Cleanup + +- V1 health before/after: `{cleanup_result['v1_health_before']}/{cleanup_result['v1_health_after']}` +- V1 topology unchanged: `{str(cleanup_result['v1_topology_unchanged']).lower()}` +- Containers/networks/volumes remaining: `{cleanup_result['containers_after']}/{cleanup_result['networks_after']}/{cleanup_result['volumes_after']}` + +## Remaining External Gates + +- Production Phase 9.0-C infrastructure and operator gates remain `NO_GO_EXTERNAL`. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- Python V1 remains the sole authoritative public and legacy writer. This report does not authorize a production `RUST_CANARY` transition. +""" + + +def write_evidence(evidence: dict) -> None: + OUTPUT.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + REPORT.write_text(render_report(evidence)) + lines = [] + for path in (OUTPUT, REPORT): + digest = hashlib.sha256(path.read_bytes()).hexdigest() + lines.append(f"{digest} {path.relative_to(ROOT)}") + CHECKSUM.write_text("\n".join(lines) + "\n") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--rust-image", required=True) + parser.add_argument("--repeat", type=int, default=200) + args = parser.parse_args() + if args.repeat < 10: + raise ValueError("Phase 9.1 parity repeat must be at least 10") + candidate = yaml.safe_load(CANDIDATE.read_text()) + decision = json.loads(PREREQUISITE_DECISION.read_text()) + digest = candidate_digest(candidate) + if decision.get("candidate_digest") != digest: + raise RuntimeError("Phase 9.0-C decision is not bound to the selected candidate") + if decision.get("decision") != "NO_GO_EXTERNAL": + raise RuntimeError("this isolated harness expects current production NO_GO_EXTERNAL") + inspect = run([ + "docker", "image", "inspect", args.rust_image, "--format", "{{.Id}}" + ]) + image_digest = inspect.stdout.strip() + parity_result: dict = {} + broker_result: dict = {} + try: + parity_result = parity(args.rust_image, args.repeat) + broker_result = broker_rehearsal( + args.rust_image, image_digest, candidate, decision + ) + finally: + BUNDLE_PATH.unlink(missing_ok=True) + evidence = { + "schema": "qdl.phase91.rust-canary-certification.v1", + "status": "COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED", + "issued_at_ns": time.time_ns(), + "slice_id": candidate["slice_id"], + "candidate_digest": digest, + "prerequisite_bundle_id": decision["bundle_id"], + "prerequisite_decision": decision["decision"], + "production_authorized": False, + "production_mutations": 0, + "python_v1_public_authority_unchanged": True, + "parity": parity_result, + "broker": broker_result, + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "same-host replicated broker rehearsal is not an independent production failure domain", + ], + } + write_evidence(evidence) + print(json.dumps({ + "status": evidence["status"], + "authentic_events": parity_result["capture"]["events"], + "semantic_mismatches": parity_result["semantic_mismatches"], + "public_writes": broker_result["public_writes"], + "legacy_writes": broker_result["legacy_writes"], + "cleanup": broker_result["cleanup"], + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase91_python_parity.py b/scripts/phase91_python_parity.py new file mode 100755 index 0000000..dd6c8ee --- /dev/null +++ b/scripts/phase91_python_parity.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import hashlib +import json +import pathlib +import statistics +import sys +import time +from dataclasses import fields +from typing import Any + +from qdl.canonical.trade import TradeContext, canonicalize_binance_usdm_trade + + +def percentile(values: list[float], ratio: float) -> float: + ordered = sorted(values) + if not ordered: + return 0.0 + return ordered[min(len(ordered) - 1, round((len(ordered) - 1) * ratio))] + + +def context(value: dict[str, Any]) -> TradeContext: + known = {item.name for item in fields(TradeContext)} + unknown = set(value) - known + if unknown: + raise ValueError(f"unknown TradeContext fields: {sorted(unknown)}") + payload = dict(value) + payload["raw_capture_id"] = bytes(payload.get("raw_capture_id", [])) + payload["raw_frame_sha256"] = bytes(payload.get("raw_frame_sha256", [])) + return TradeContext(**payload) + + +def main() -> int: + if len(sys.argv) != 2: + raise SystemExit("usage: phase91_python_parity.py BUNDLE") + payload = json.loads(pathlib.Path(sys.argv[1]).read_text()) + if set(payload) != {"fixtures", "repeat"}: + raise ValueError("Phase 9.1 replay bundle fields are invalid") + fixtures = payload["fixtures"] + repeat = int(payload["repeat"]) + if not fixtures or repeat <= 0: + raise ValueError("Phase 9.1 replay bundle must be non-empty") + aggregate = hashlib.sha256() + first_record_hashes: list[str] = [] + latencies_ms: list[float] = [] + started = time.perf_counter() + for iteration in range(repeat): + for fixture in fixtures: + if fixture.get("provider_kind") != "binance_usdm_trade": + raise ValueError("Phase 9.1 candidate accepts Binance USD-M TRADE only") + event_started = time.perf_counter_ns() + event = canonicalize_binance_usdm_trade( + fixture["raw"], context(fixture["context"]) + ) + encoded = event.SerializeToString(deterministic=True) + latencies_ms.append((time.perf_counter_ns() - event_started) / 1_000_000) + aggregate.update(len(encoded).to_bytes(8, "big")) + aggregate.update(encoded) + if iteration == 0: + first_record_hashes.append(hashlib.sha256(encoded).hexdigest()) + elapsed = time.perf_counter() - started + print(json.dumps({ + "schema": "qdl.phase91.python-parity.v1", + "status": "PASS", + "events": len(fixtures) * repeat, + "fixture_count": len(fixtures), + "repeat": repeat, + "aggregate_sha256": aggregate.hexdigest(), + "record_sha256": first_record_hashes, + "events_per_second": len(fixtures) * repeat / max(elapsed, 1e-9), + "latency_ms": { + "p50": percentile(latencies_ms, 0.50), + "p95": percentile(latencies_ms, 0.95), + "p99": percentile(latencies_ms, 0.99), + "max": max(latencies_ms), + "mean": statistics.fmean(latencies_ms), + }, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase92_migration_smoke.sh b/scripts/phase92_migration_smoke.sh new file mode 100755 index 0000000..302cb7d --- /dev/null +++ b/scripts/phase92_migration_smoke.sh @@ -0,0 +1,206 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +POSTGRES_IMAGE="${QDL_PHASE92_POSTGRES_IMAGE:-timescale/timescaledb:latest-pg15}" +CONTAINER="${QDL_PHASE92_POSTGRES_CONTAINER:-qdl_phase92_postgres_$$}" +OUTPUT="${QDL_PHASE92_MIGRATION_OUTPUT:-${ROOT_DIR}/upgrade/evidence/phase92-authority-migration.json}" +CANDIDATE_DIGEST="${QDL_PHASE92_CANDIDATE_DIGEST:-$( + PYTHONPATH="${ROOT_DIR}" python3 -c "from qdl.certification.prerequisites import CandidateSlice; print(CandidateSlice.load('${ROOT_DIR}/config/phase9/candidate-slice.yaml').digest)" +)}" +SLICE="production/binance/usdm/perpetual/trade/plan-1/btcusdt" +BUNDLE="00000000-0000-4000-8000-000000000092" + +cleanup() { + docker rm -f "${CONTAINER}" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +docker run -d --name "${CONTAINER}" --network none --security-opt no-new-privileges:true --pids-limit 256 --memory 768m --cpus 1.0 --tmpfs /var/lib/postgresql/data:rw,nosuid,nodev,size=512m -e POSTGRES_HOST_AUTH_METHOD=trust -v "${ROOT_DIR}/migrations/postgres:/migrations:ro" "${POSTGRES_IMAGE}" >/dev/null + +for _ in $(seq 1 60); do + docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null 2>&1 && break + sleep 1 +done +docker exec "${CONTAINER}" pg_isready -U postgres -d postgres >/dev/null + +apply_migrations() { + local migration + for migration in "${ROOT_DIR}"/migrations/postgres/*.sql; do + docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres -f "/migrations/$(basename "${migration}")" >/dev/null + done +} +apply_migrations + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +INSERT INTO qdl_production_prerequisite_bundles ( + bundle_id, candidate_digest, policy_revision, decision, evidence, + evidence_sha256, issued_by, issued_at, expires_at +) VALUES ( + '${BUNDLE}', '${CANDIDATE_DIGEST}', 1, 'GO', '{}', repeat('5',64), + 'phase92-test', clock_timestamp(), clock_timestamp() + interval '1 day' +); +INSERT INTO qdl_authority_slices ( + slice_id, environment, venue, market, product_type, feed, + partition_plan_epoch, partition_id, schema_major, state, + authority_revision, owner_id, lease_epoch, terminal_watermark, + candidate_digest, artifact_image_digest, sbom_digest, signature_identity, + contract_digest, normalizer_version, adapter_version, config_revision, + instrument_catalog_revision, source_policy_revision, partition_plan_digest, + rollback_manifest_digest, prerequisite_bundle_id, approved_by, approved_at, + hold_until +) VALUES ( + '${SLICE}', 'production', 'BINANCE', 'USDM', 'PERPETUAL', 'TRADE', + 1, 'rendezvous-sha256-v1:epoch-1:btcusdt', 2, 'RUST_CANARY', + 3, 'python-primary', 1, 100, + '${CANDIDATE_DIGEST}', 'sha256:${CANDIDATE_DIGEST}', repeat('1',64), + 'phase92-test-signer', repeat('2',64), 'qdl-rust-core/test', + 'binance-usdm/test', 'phase92-test-config', 'phase92-test-catalog', + 'phase92-test-source-policy', repeat('3',64), repeat('4',64), + '${BUNDLE}', 'phase92-test', clock_timestamp(), + clock_timestamp() + interval '2 hours' +); +SQL + +expect_failure() { + if docker exec "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres -c "$1" >/dev/null 2>&1; then + printf 'expected SQL failure but statement succeeded\n' >&2 + exit 1 + fi +} + +expect_failure "SELECT qdl_transition_authority( + '90000000-0000-4000-8000-000000000001','${SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'${BUNDLE}',clock_timestamp()+interval '1 hour', + 'phase92-test','direct primary bypass');" + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +INSERT INTO qdl_terminal_owner_checkpoints ( + checkpoint_id, slice_id, owner_id, authority_revision, lease_epoch, + partition_plan_epoch, source_session_id, connection_generation, + terminal_watermark, terminal_event_id, terminal_payload_sha256, + candidate_digest, committed_at +) VALUES ( + '91000000-0000-4000-8000-000000000001','${SLICE}','python-primary', + 3,1,1,'python-session-1',1,100,'event-100',repeat('6',64), + '${CANDIDATE_DIGEST}',clock_timestamp() +); +INSERT INTO qdl_authority_handoffs ( + handoff_id, checkpoint_id, direction, slice_id, old_owner_id, new_owner_id, + expected_state, new_state, expected_authority_revision, + new_authority_revision, expected_lease_epoch, new_lease_epoch, + partition_plan_epoch, terminal_watermark, first_new_watermark, + overlap_start_watermark, overlap_end_watermark, old_event_count, + new_event_count, semantic_mismatches, open_gaps, candidate_digest, + prerequisite_bundle_id, handoff_sha256, approved_by, approved_at, expires_at +) VALUES ( + '92000000-0000-4000-8000-000000000001', + '91000000-0000-4000-8000-000000000001','PYTHON_TO_RUST','${SLICE}', + 'python-primary','rust-primary','RUST_CANARY','RUST_PRIMARY',3,4,1,2,1, + 100,101,90,100,11,11,0,0,'${CANDIDATE_DIGEST}','${BUNDLE}', + repeat('7',64),'phase92-test',clock_timestamp(), + clock_timestamp()+interval '2 hours' +); +SELECT (qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000001', + '93000000-0000-4000-8000-000000000001','${SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'${BUNDLE}',clock_timestamp()+interval '1 hour', + 'phase92-test','accepted Python to Rust handoff' +)).state; +SQL + +expect_failure "SELECT qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000001', + '93000000-0000-4000-8000-000000000009','${SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'${BUNDLE}',clock_timestamp()+interval '1 hour', + 'phase92-test','stale CAS replay');" +expect_failure "UPDATE qdl_authority_handoffs SET approved_by='mutated';" +expect_failure "DELETE FROM qdl_terminal_owner_checkpoints;" + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres </dev/null +SELECT qdl_transition_authority( + '93000000-0000-4000-8000-000000000002','${SLICE}', + 'RUST_PRIMARY',4,'rust-primary',2,1,'BLOCKED','rust-primary',2, + 120,NULL,NULL,'phase92-test','fence Rust before rollback'); +SELECT qdl_transition_authority( + '93000000-0000-4000-8000-000000000003','${SLICE}', + 'BLOCKED',5,'rust-primary',2,1,'ROLLBACK_PENDING','rust-primary',2, + 120,NULL,NULL,'phase92-test','prepare Python rollback'); +INSERT INTO qdl_terminal_owner_checkpoints ( + checkpoint_id, slice_id, owner_id, authority_revision, lease_epoch, + partition_plan_epoch, source_session_id, connection_generation, + terminal_watermark, terminal_event_id, terminal_payload_sha256, + candidate_digest, committed_at +) VALUES ( + '91000000-0000-4000-8000-000000000002','${SLICE}','rust-primary', + 6,2,1,'rust-session-1',1,120,'event-120',repeat('8',64), + '${CANDIDATE_DIGEST}',clock_timestamp() +); +INSERT INTO qdl_authority_handoffs ( + handoff_id, checkpoint_id, direction, slice_id, old_owner_id, new_owner_id, + expected_state, new_state, expected_authority_revision, + new_authority_revision, expected_lease_epoch, new_lease_epoch, + partition_plan_epoch, terminal_watermark, first_new_watermark, + overlap_start_watermark, overlap_end_watermark, old_event_count, + new_event_count, semantic_mismatches, open_gaps, candidate_digest, + prerequisite_bundle_id, handoff_sha256, approved_by, approved_at, expires_at +) VALUES ( + '92000000-0000-4000-8000-000000000002', + '91000000-0000-4000-8000-000000000002','RUST_TO_PYTHON','${SLICE}', + 'rust-primary','python-rollback','ROLLBACK_PENDING','PYTHON_PRIMARY',6,7,2,3,1, + 120,121,110,120,11,11,0,0,'${CANDIDATE_DIGEST}','${BUNDLE}', + repeat('9',64),'phase92-test',clock_timestamp(), + clock_timestamp()+interval '2 hours' +); +SELECT (qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000002', + '93000000-0000-4000-8000-000000000004','${SLICE}', + 'ROLLBACK_PENDING',6,'rust-primary',2,1,'PYTHON_PRIMARY','python-rollback',3, + 120,NULL,NULL,'phase92-test','accepted Rust to Python rollback' +)).state; +SQL + +apply_migrations + +state="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT state || ':' || authority_revision || ':' || owner_id || ':' || lease_epoch || ':' || terminal_watermark FROM qdl_authority_slices;")" +audit_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_transition_audit;")" +checkpoint_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_terminal_owner_checkpoints;")" +handoff_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_handoffs;")" +[[ "${state}" == "PYTHON_PRIMARY:7:python-rollback:3:120" ]] +[[ "${audit_count}" == "4" ]] +[[ "${checkpoint_count}" == "2" ]] +[[ "${handoff_count}" == "2" ]] + +python3 - "${OUTPUT}" "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" <<'PY' +import json +import pathlib +import sys + +output, state, audits, checkpoints, handoffs = sys.argv[1:] +path = pathlib.Path(output) +path.parent.mkdir(parents=True, exist_ok=True) +path.write_text(json.dumps({ + "schema": "qdl.phase92.authority-migration.v1", + "status": "PASS", + "final_test_state": state, + "audit_records": int(audits), + "terminal_checkpoints": int(checkpoints), + "accepted_handoffs": int(handoffs), + "direct_primary_bypass_rejected": True, + "stale_cas_rejected": True, + "handoff_mutation_rejected": True, + "checkpoint_delete_rejected": True, + "python_to_rust_handoff_passed": True, + "rust_to_python_rollback_passed": True, + "idempotent_migration": True, + "production_mutations": 0, +}, indent=2, sort_keys=True) + "\n") +PY + +cleanup +trap - EXIT +[[ -z "$(docker ps -aq --filter name=^/${CONTAINER}$)" ]] +printf '{"status":"PASS","state":"%s","audits":%s,"checkpoints":%s,"handoffs":%s,"cleanup":true}\n' "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" diff --git a/scripts/phase92_primary_certification.py b/scripts/phase92_primary_certification.py new file mode 100755 index 0000000..38e1115 --- /dev/null +++ b/scripts/phase92_primary_certification.py @@ -0,0 +1,514 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import pathlib +import tempfile +import time + +import yaml + +os.environ.setdefault("QDL_PHASE8_PROJECT", "qdl_phase92_certification") + +from phase80_broker_certification import ( # noqa: E402 + BOOTSTRAP, + PROJECT, + ROOT, + add_acls, + cleanup, + compose, + consume, + create_topic, + run, + total_end_offset, + v1_health, + v1_topology, + wait_for_cluster, + wait_for_replicas, +) +from phase91_canary_certification import ( # noqa: E402 + BUNDLE_PATH, + candidate_digest, + consume_compacted_records, + parity, + rust_command, +) + + +CANDIDATE = ROOT / "config/phase9/candidate-slice.yaml" +PREREQUISITE_DECISION = ROOT / "upgrade/evidence/phase90c-production-prerequisites.json" +MIGRATION_EVIDENCE = ROOT / "upgrade/evidence/phase92-authority-migration.json" +OUTPUT = ROOT / "upgrade/evidence/phase92-bounded-primary-certification.json" +REPORT = ROOT / "upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md" +CHECKSUM = ROOT / "upgrade/evidence/phase92-evidence.sha256" + + +def broker_rehearsal( + image: str, image_digest: str, candidate: dict, decision: dict +) -> dict: + topics = { + "authority": "qdl.phase8.phase92.control.authority.v3", + "audit": "qdl.phase8.phase92.audit.authority.v3", + "checkpoint": "qdl.phase8.phase92.control.checkpoint.v1", + "handoff": "qdl.phase8.phase92.control.handoff.v1", + "primary": "qdl.phase8.phase92.primary.canonical.v1", + "public": "qdl.phase8.phase92.isolated.public.v2", + "legacy": "qdl.phase8.phase92.isolated.legacy.v1", + "production_public": "qdl.phase8.phase92.production-public.must-remain-empty", + "production_legacy": "qdl.phase8.phase92.production-legacy.must-remain-empty", + "transport": "qdl.phase8.phase92.transport", + } + v1_before = v1_topology() + health_before = v1_health() + cleanup_result: dict[str, object] = {} + result: dict[str, object] = {} + with tempfile.TemporaryDirectory(prefix="qdl-phase92-certs-") as cert_dir: + env = os.environ.copy() + env["QDL_PHASE8_CERT_DIR"] = cert_dir + run([str(ROOT / "scripts/phase80_generate_tls.sh"), cert_dir], env=env, timeout=120) + try: + compose(env, "down", "--volumes", "--remove-orphans", check=False, timeout=120) + compose(env, "up", "-d", timeout=180) + wait_for_cluster(env) + time.sleep(10.0) + wait_for_cluster(env) + running = set( + compose(env, "ps", "--status", "running", "--services").stdout.splitlines() + ) + if not {"kafka1", "kafka2", "kafka3"}.issubset(running): + raise RuntimeError( + f"Kafka brokers did not survive stability window: {sorted(running)}" + ) + for name, topic in topics.items(): + create_topic( + env, + topic, + partitions=1 if name != "transport" else 3, + cleanup_policy="compact" if name == "authority" else "delete", + ) + add_acls(env) + wait_for_replicas(env) + before = {name: total_end_offset(env, topic) for name, topic in topics.items()} + nonce = hashlib.sha256(str(time.time_ns()).encode()).hexdigest()[:16] + runtime_env = { + "QDL_KAFKA_BOOTSTRAP_SERVERS": BOOTSTRAP, + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_AUTHORITY_TOPIC": topics["authority"], + "QDL_AUDIT_TOPIC": topics["audit"], + "QDL_CHECKPOINT_TOPIC": topics["checkpoint"], + "QDL_HANDOFF_TOPIC": topics["handoff"], + "QDL_PRIMARY_CANONICAL_TOPIC": topics["primary"], + "QDL_ISOLATED_PUBLIC_TOPIC": topics["public"], + "QDL_ISOLATED_LEGACY_TOPIC": topics["legacy"], + "QDL_PRODUCTION_PUBLIC_TOPIC": topics["production_public"], + "QDL_PRODUCTION_LEGACY_TOPIC": topics["production_legacy"], + "QDL_AUTHORITY_NONCE": nonce, + "QDL_CANDIDATE_DIGEST": candidate_digest(candidate), + "QDL_PREREQUISITE_BUNDLE_ID": decision["bundle_id"], + "QDL_SLICE_ID": candidate["slice_id"], + "QDL_PYTHON_OWNER_ID": "python-primary-isolated", + "QDL_RUST_OWNER_ID": "rust-primary-isolated", + "QDL_ROLLBACK_OWNER_ID": "python-rollback-isolated", + } + runtime_result = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-phase92-primary-rehearsal", + env_values=runtime_env, + ), + timeout=180, + ) + runtime = json.loads(runtime_result.stdout.strip().splitlines()[-1]) + if runtime.get("status") != "PASS" or not all( + runtime.get("checks", {}).values() + ): + raise RuntimeError(f"Phase 9.2 runtime checks failed: {runtime}") + + recovery_result = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-phase92-primary-rehearsal", + env_values={ + **runtime_env, + "QDL_REHEARSAL_MODE": "RECOVERY_VERIFY", + "QDL_RECOVERY_AUTHORITY_REVISION": "11", + "QDL_RECOVERY_FIRST_WATERMARK": "101", + "QDL_RECOVERY_LAST_WATERMARK": "180", + }, + ), + timeout=180, + ) + recovery = json.loads(recovery_result.stdout.strip().splitlines()[-1]) + if recovery.get("status") != "PASS" or not all( + recovery.get("checks", {}).values() + ): + raise RuntimeError( + f"Phase 9.2 process restart recovery failed: {recovery}" + ) + + after = {name: total_end_offset(env, topic) for name, topic in topics.items()} + deltas = {name: after[name] - before[name] for name in topics} + expected = { + "authority": 5, + "audit": 5, + "checkpoint": 2, + "handoff": 2, + "primary": 81, + "public": 81, + "legacy": 81, + "production_public": 0, + "production_legacy": 0, + "transport": 0, + } + if deltas != expected: + raise RuntimeError( + f"Phase 9.2 isolated topic deltas diverged: {deltas} != {expected}" + ) + + compose(env, "stop", "kafka3") + one_loss_before = total_end_offset(env, topics["transport"]) + one_loss = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092,kafka2:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-one-loss", + }, + ), + check=False, + timeout=35, + ) + one_loss_after = total_end_offset(env, topics["transport"]) + one_replica_loss_acked = ( + one_loss.returncode == 0 and one_loss_after == one_loss_before + 1 + ) + compose(env, "stop", "kafka2") + min_isr_before = one_loss_after + min_isr = run( + rust_command( + image=image, + cert_dir=cert_dir, + entrypoint="/usr/local/bin/qdl-kafka-smoke", + env_values={ + "QDL_KAFKA_BOOTSTRAP_SERVERS": "kafka1:9092", + "QDL_KAFKA_CERT_ROOT": "/certs", + "QDL_KAFKA_SMOKE_TOPIC": topics["transport"], + "QDL_KAFKA_SMOKE_NONCE": f"{nonce}-min-isr", + }, + ), + check=False, + timeout=35, + ) + compose(env, "start", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + min_isr_after = total_end_offset(env, topics["transport"]) + min_isr_failed_closed = ( + min_isr.returncode != 0 and min_isr_after == min_isr_before + ) + if not one_replica_loss_acked or not min_isr_failed_closed: + raise RuntimeError( + "Phase 9.2 broker durability gate failed " + f"one_loss={one_replica_loss_acked} min_isr={min_isr_failed_closed}" + ) + + compose(env, "restart", "kafka1", "kafka2", "kafka3", timeout=120) + wait_for_cluster(env) + wait_for_replicas(env) + + audit = [ + json.loads(item) + for item in consume( + env, + topics["audit"], + 5, + f"phase8-phase92-audit-{nonce}", + ) + ] + checkpoints = [ + json.loads(item) + for item in consume( + env, + topics["checkpoint"], + 2, + f"phase8-phase92-checkpoint-{nonce}", + ) + ] + handoffs = [ + json.loads(item) + for item in consume( + env, + topics["handoff"], + 2, + f"phase8-phase92-handoff-{nonce}", + ) + ] + time.sleep(1.0) + slow_started = time.monotonic() + projections = {} + for name in ("primary", "public", "legacy"): + projections[name] = [ + json.loads(item) + for item in consume( + env, + topics[name], + 81, + f"phase8-phase92-slow-{name}-{nonce}", + timeout=60, + ) + ] + catchup_seconds = time.monotonic() - slow_started + authority = [ + json.loads(item) + for item in consume_compacted_records(env, topics["authority"]) + ] + latest = max(authority, key=lambda item: item["authority_revision"]) + states = [item["state"] for item in audit] + expected_states = [ + "RUST_CANARY", + "RUST_PRIMARY", + "BLOCKED", + "ROLLBACK_PENDING", + "PYTHON_PRIMARY", + ] + watermarks = [ + item["source_watermark"] for item in projections["primary"] + ] + projection_parity = ( + projections["primary"] == projections["public"] + == projections["legacy"] + ) + boundary_gap_free = watermarks == list(range(101, 182)) + owner_boundary = ( + all( + item["owner_id"] == "rust-primary-isolated" + for item in projections["primary"][:64] + ) + and all( + item["owner_id"] == "python-rollback-isolated" + for item in projections["primary"][64:] + ) + ) + if ( + states != expected_states + or latest["state"] != "PYTHON_PRIMARY" + or latest["authority_revision"] != 11 + or len(checkpoints) != 2 + or len(handoffs) != 2 + or not projection_parity + or not boundary_gap_free + or not owner_boundary + ): + raise RuntimeError("Phase 9.2 recovery/projection evidence diverged") + result = { + "schema": "qdl.phase92.broker-rehearsal.v1", + "status": "PASS", + "mode": "ISOLATED_REHEARSAL", + "production_authorized": False, + "image_digest": image_digest, + "runtime": runtime, + "process_restart_recovery": recovery, + "offset_deltas": deltas, + "authority_audit_states_after_restart": states, + "latest_authority_after_restart": { + "state": latest["state"], + "revision": latest["authority_revision"], + "owner_id": latest["owner_id"], + "lease_epoch": latest["lease_epoch"], + }, + "terminal_checkpoints": len(checkpoints), + "accepted_handoffs": len(handoffs), + "projection_parity": projection_parity, + "boundary_gap_free": boundary_gap_free, + "owner_boundary_correct": owner_boundary, + "one_replica_loss_acked": one_replica_loss_acked, + "min_isr_failed_closed": min_isr_failed_closed, + "slow_consumer": { + "delayed_seconds": 1.0, + "catchup_seconds": catchup_seconds, + "records_per_projection": 81, + "ordered_gap_free": boundary_gap_free, + }, + "cutover_ms": runtime["cutover_ms"], + "rollback_ms": runtime["rollback_ms"], + "production_public_writes": 0, + "production_legacy_writes": 0, + "final_authority": "PYTHON_PRIMARY", + } + except Exception as error: + status = compose(env, "ps", "--all", check=False, timeout=30) + logs = compose( + env, "logs", "--no-color", "--tail", "240", check=False, timeout=60 + ) + diagnostic = ( + status.stdout + status.stderr + logs.stdout + logs.stderr + )[-12000:] + raise RuntimeError( + f"Phase 9.2 broker rehearsal failed before cleanup: {error}\n" + f"isolated broker diagnostics:\n{diagnostic}" + ) from error + finally: + cleanup_result = cleanup(env) + v1_after = v1_topology() + health_after = v1_health() + cleanup_result.update( + { + "v1_health_before": health_before, + "v1_health_after": health_after, + "v1_topology_unchanged": v1_before == v1_after, + } + ) + result["cleanup"] = cleanup_result + if ( + health_before != 200 + or health_after != 200 + or not cleanup_result["v1_topology_unchanged"] + or any( + cleanup_result[key] + for key in ("containers_after", "networks_after", "volumes_after") + ) + ): + raise RuntimeError(f"Phase 9.2 cleanup/V1 invariant failed: {cleanup_result}") + return result + + +def render_report(evidence: dict) -> str: + parity_result = evidence["parity"] + broker = evidence["broker"] + cleanup_result = broker["cleanup"] + return f"""# Phase 9.2 Bounded Rust Primary Certification Report + +## Decision + +- Status: `{evidence['status']}` +- Production authorized: `{str(evidence['production_authorized']).lower()}` +- Production mutations: `{evidence['production_mutations']}` +- Prerequisite decision: `{evidence['prerequisite_decision']}` +- Slice: `{evidence['slice_id']}` +- Candidate digest: `{evidence['candidate_digest']}` + +## Authentic Parity + +- Provenance: `{parity_result['capture']['capture_provenance']}` +- Canonical events: `{parity_result['capture']['events']}` +- Semantic mismatches: `{parity_result['semantic_mismatches']}` +- Clean Rust process runs: `{parity_result['rust_clean_process_runs']}` + +## Terminal Handoff And Recovery + +- Authority states: `{', '.join(broker['authority_audit_states_after_restart'])}` +- Terminal checkpoints / accepted handoffs: `{broker['terminal_checkpoints']} / {broker['accepted_handoffs']}` +- Projection parity: `{str(broker['projection_parity']).lower()}` +- Boundary gap-free: `{str(broker['boundary_gap_free']).lower()}` +- Owner boundary correct: `{str(broker['owner_boundary_correct']).lower()}` +- Restart recovery: `{broker['process_restart_recovery']['status']}` +- Recovered target watermarks: `{broker['process_restart_recovery']['restored_target_watermarks']}` +- First post-restart watermark: `{broker['process_restart_recovery']['resumed_watermark']}` +- Cutover / rollback measurement: `{broker['cutover_ms']:.3f} ms / {broker['rollback_ms']:.3f} ms` +- One-replica-loss ACK: `{str(broker['one_replica_loss_acked']).lower()}` +- Below-min-ISR fail closed: `{str(broker['min_isr_failed_closed']).lower()}` +- Final authority: `{broker['final_authority']}` +- Production public / legacy writes: `{broker['production_public_writes']} / {broker['production_legacy_writes']}` + +## Isolation And Cleanup + +- V1 health before/after: `{cleanup_result['v1_health_before']} / {cleanup_result['v1_health_after']}` +- V1 topology unchanged: `{str(cleanup_result['v1_topology_unchanged']).lower()}` +- Containers/networks/volumes remaining: `{cleanup_result['containers_after']} / {cleanup_result['networks_after']} / {cleanup_result['volumes_after']}` + +## Remaining External Gates + +- Phase 9.0-C remains `NO_GO_EXTERNAL`; a production primary transition is not authorized. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- A real production canary hold and explicit exact-slice approval remain required. +""" + + +def write_evidence(evidence: dict) -> None: + OUTPUT.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + REPORT.write_text(render_report(evidence)) + lines = [] + for path in (OUTPUT, REPORT, MIGRATION_EVIDENCE): + digest = hashlib.sha256(path.read_bytes()).hexdigest() + lines.append(f"{digest} {path.relative_to(ROOT)}") + CHECKSUM.write_text("\n".join(lines) + "\n") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--rust-image", required=True) + parser.add_argument("--repeat", type=int, default=200) + args = parser.parse_args() + if args.repeat < 10: + raise ValueError("Phase 9.2 parity repeat must be at least 10") + candidate = yaml.safe_load(CANDIDATE.read_text()) + decision = json.loads(PREREQUISITE_DECISION.read_text()) + digest = candidate_digest(candidate) + if decision.get("candidate_digest") != digest: + raise RuntimeError("Phase 9.0-C decision does not bind the candidate") + if decision.get("decision") != "NO_GO_EXTERNAL": + raise RuntimeError("isolated Phase 9.2 harness expects NO_GO_EXTERNAL") + migration = json.loads(MIGRATION_EVIDENCE.read_text()) + if migration.get("status") != "PASS" or migration.get("production_mutations") != 0: + raise RuntimeError("Phase 9.2 migration evidence is absent or invalid") + inspect = run( + ["docker", "image", "inspect", args.rust_image, "--format", "{{.Id}}"] + ) + image_digest = inspect.stdout.strip() + try: + parity_result = parity(args.rust_image, args.repeat) + broker_result = broker_rehearsal( + args.rust_image, image_digest, candidate, decision + ) + finally: + BUNDLE_PATH.unlink(missing_ok=True) + evidence = { + "schema": "qdl.phase92.bounded-primary-certification.v1", + "status": "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED", + "issued_at_ns": time.time_ns(), + "slice_id": candidate["slice_id"], + "candidate_digest": digest, + "prerequisite_bundle_id": decision["bundle_id"], + "prerequisite_decision": decision["decision"], + "production_authorized": False, + "production_mutations": 0, + "python_v1_public_authority_unchanged": True, + "migration": migration, + "parity": parity_result, + "broker": broker_result, + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "real production canary hold and exact-slice approval remain unavailable", + "same-host replicated broker is not an independent failure domain", + ], + } + write_evidence(evidence) + print( + json.dumps( + { + "status": evidence["status"], + "authentic_events": parity_result["capture"]["events"], + "semantic_mismatches": parity_result["semantic_mismatches"], + "projection_parity": broker_result["projection_parity"], + "boundary_gap_free": broker_result["boundary_gap_free"], + "production_public_writes": broker_result["production_public_writes"], + "production_legacy_writes": broker_result["production_legacy_writes"], + "cleanup": broker_result["cleanup"], + }, + sort_keys=True, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase93_hold_close_certification.py b/scripts/phase93_hold_close_certification.py new file mode 100755 index 0000000..9cec407 --- /dev/null +++ b/scripts/phase93_hold_close_certification.py @@ -0,0 +1,370 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import pathlib +import sys +import time +import uuid + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.canary.phase93 import ( # noqa: E402 + DecommissionRequest, + ExpansionManifest, + ExpansionType, + HoldScope, + PrimaryHoldEvaluator, + PrimaryHoldIdentity, + PrimaryHoldObservation, + PrimaryHoldPolicy, + ProductionClosureAuthorizer, + RollbackWindowClosure, + assess_decommission, +) +from qdl.certification.prerequisites import CandidateSlice # noqa: E402 +from scripts.phase80_broker_certification import v1_health, v1_topology # noqa: E402 + + +NO_GO = ROOT / "upgrade/evidence/phase90c-production-prerequisites.json" +PHASE92 = ROOT / "upgrade/evidence/phase92-bounded-primary-certification.json" +MIGRATION = ROOT / "upgrade/evidence/phase93-hold-close-migration.json" +OUTPUT = ROOT / "upgrade/evidence/phase93-hold-close-expand-certification.json" +REPORT = ROOT / "upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md" +CHECKSUM = ROOT / "upgrade/evidence/phase93-evidence.sha256" + + +def uid(label: str) -> str: + return str(uuid.uuid5(uuid.NAMESPACE_URL, f"qdl-phase93:{label}")) + + +def digest(label: str) -> str: + return hashlib.sha256(label.encode()).hexdigest() + + +def certify_test_hold( + candidate: CandidateSlice, bundle_id: str, now_ns: int +): + policy = PrimaryHoldPolicy( + minimum_duration_seconds=120, + sample_interval_seconds=60, + max_sample_gap_seconds=60, + max_lag_ms=500, + max_freshness_ms=1000, + max_queue_depth=1000, + max_spool_bytes=1_000_000, + max_cpu_percent=80.0, + max_rss_mb=512.0, + ) + start = now_ns - 120_000_000_000 + identity = PrimaryHoldIdentity( + schema="qdl.primary-hold.v1", + hold_id=uid("test-hold"), + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=bundle_id, + owner_id="rust-primary-test-fixture", + authority_revision=8, + lease_epoch=12, + partition_plan_epoch=1, + started_at_ns=start, + required_until_ns=now_ns, + policy_digest=policy.digest, + ) + evaluator = PrimaryHoldEvaluator( + identity=identity, + policy=policy, + scope=HoldScope.TEST_REHEARSAL, + ) + for sequence, watermark in ((1, 110), (2, 120)): + reason = evaluator.observe( + PrimaryHoldObservation( + schema="qdl.primary-hold-observation.v1", + observation_id=uid(f"test-observation-{sequence}"), + hold_id=identity.hold_id, + slice_id=identity.slice_id, + candidate_digest=identity.candidate_digest, + owner_id=identity.owner_id, + authority_revision=identity.authority_revision, + lease_epoch=identity.lease_epoch, + partition_plan_epoch=identity.partition_plan_epoch, + sequence=sequence, + observed_at_ns=start + sequence * 60_000_000_000, + last_watermark=watermark, + lag_ms=10, + freshness_ms=20, + queue_depth=1, + spool_bytes=100, + cpu_percent=10.0, + rss_mb=64.0, + registered_consumers=2, + healthy_consumers=2, + checkpoint_watermark=watermark, + ) + ) + if reason != "PASS": + raise RuntimeError(f"test hold observation failed: {reason}") + return evaluator.decision(decision_id=uid("test-hold-decision"), now_ns=now_ns) + + +def test_parent_closure( + candidate: CandidateSlice, bundle_id: str, now_ns: int +) -> RollbackWindowClosure: + return RollbackWindowClosure( + schema="qdl.rollback-window-closure.v1", + closure_id=uid("test-parent-closure"), + slice_id=str(candidate.payload["slice_id"]), + candidate_digest=candidate.digest, + prerequisite_bundle_id=bundle_id, + owner_id="rust-primary-test-fixture", + authority_revision=8, + lease_epoch=12, + partition_plan_epoch=1, + hold_decision_id=uid("test-hold-decision"), + hold_decision_digest=digest("test-hold-decision"), + consumer_registry_snapshot_id=uid("test-consumer-registry"), + consumer_registry_digest=digest("test-consumer-registry"), + authority_registry_snapshot_id=uid("test-authority-registry"), + authority_registry_digest=digest("test-authority-registry"), + rollback_rehearsal_id=uid("test-rollback"), + rollback_rehearsal_digest=digest("test-rollback"), + approval_id=uid("test-approval"), + approval_digest=digest("test-approval"), + operator="phase93-test-fixture", + change_ticket="QDL-93-TEST", + closed_at_ns=now_ns, + production_authorized=True, + ) + + +def render_report(evidence: dict) -> str: + parent = evidence["parent_phase92"] + control = evidence["control_plane_fixture"] + return f"""# Phase 9.3 Hold, Close And Expand Certification Report + +## Decision + +- Status: {evidence['status']} +- Production authorized: {evidence['production_authorized']} +- Production hold started: {evidence['production_hold_started']} +- Production rollback window closed: {evidence['production_rollback_window_closed']} +- Production expansions authorized: {evidence['production_expansions_authorized']} +- Production mutations: {evidence['production_mutations']} + +## Parent Evidence + +- Phase 9.2 status: {parent['status']} +- Authentic provider events: {parent['authentic_events']} +- Semantic mismatches: {parent['semantic_mismatches']} +- Parent production authorized: {parent['production_authorized']} + +## Isolated Control Plane + +- Provenance: {control['provenance']} +- Accelerated time is production evidence: {control['accelerated_time_is_production_evidence']} +- Test hold status: {control['test_hold_status']} +- Test hold production authorized: {control['test_hold_production_authorized']} +- Current no-go rejection: {control['current_no_go_rejection']} +- Local Phase 9.2 production eligible: {control['local_phase92_production_eligible']} +- Expansion manifests: {control['expansion_manifest_count']} +- Decommission decision: {control['decommission_reason']} + +## Persistence And Isolation + +- Migration: {evidence['migration']['status']} +- Closure changed authority: {not evidence['migration']['closure_did_not_mutate_authority']} +- V1 health before/after: {evidence['v1']['health_before']} / {evidence['v1']['health_after']} +- V1 topology unchanged: {evidence['v1']['topology_unchanged']} +- Disposable resources remaining: {evidence['cleanup']['resources_remaining']} + +## External Gates + +Phase 9.0-C remains NO_GO_EXTERNAL. There is no real Rust primary, production +hold duration, production consumer checkpoint set or operator closure approval. +No rollback window, expansion or Python decommission is authorized. +""" + + +def freeze(evidence: dict) -> None: + OUTPUT.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + REPORT.write_text(render_report(evidence)) + entries = [] + for path in (OUTPUT, REPORT, MIGRATION): + entries.append( + f"{hashlib.sha256(path.read_bytes()).hexdigest()} " + f"{path.relative_to(ROOT)}" + ) + CHECKSUM.write_text("\n".join(entries) + "\n") + + +def main() -> int: + argparse.ArgumentParser().parse_args() + candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + no_go = json.loads(NO_GO.read_text()) + phase92 = json.loads(PHASE92.read_text()) + migration = json.loads(MIGRATION.read_text()) + if no_go.get("decision") != "NO_GO_EXTERNAL": + raise RuntimeError("Phase 9.3 expects current NO_GO_EXTERNAL") + if ( + phase92.get("status") + != "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED" + or phase92.get("production_authorized") is not False + or phase92["parity"]["semantic_mismatches"] != 0 + ): + raise RuntimeError("Phase 9.2 parent evidence is invalid") + if migration.get("status") != "PASS": + raise RuntimeError("Phase 9.3 migration evidence is invalid") + + before_topology = v1_topology() + before_health = v1_health() + now_ns = time.time_ns() + hold = certify_test_hold(candidate, no_go["bundle_id"], now_ns) + denied = ProductionClosureAuthorizer().authorize( + candidate=candidate, + prerequisite_decision=no_go, + expected_bundle_id=no_go["bundle_id"], + primary_evidence=phase92, + hold_decision=None, + consumer_registry=None, + authority_registry=None, + rollback_evidence=None, + approval=None, + now_ns=no_go["issued_at_ns"] + 1, + ) + if denied.allowed or denied.reason != "PREREQUISITE_DECISION_NOT_GO": + raise RuntimeError("current no-go did not fail closed") + + parent = test_parent_closure(candidate, no_go["bundle_id"], now_ns) + expansions = [] + for index, kind in enumerate(ExpansionType, start=1): + item = ExpansionManifest.plan( + expansion_id=uid(f"expansion-{kind}"), + parent=parent, + expansion_type=kind, + candidate_digest=digest(f"candidate-{kind}"), + scope_digest=digest(f"scope-{kind}"), + partition_plan_epoch=( + parent.partition_plan_epoch + 1 + if kind == ExpansionType.INSTRUMENT_PARTITION + else parent.partition_plan_epoch + ), + created_at_ns=now_ns + index, + ) + expansions.append( + { + "type": kind, + "status": item.status, + "required_gate_count": len(item.required_gates), + "write_authority": ( + item.public_write_allowed or item.legacy_write_allowed + ), + "transitive_evidence_allowed": ( + item.transitive_evidence_allowed + ), + "digest": item.digest, + } + ) + if any( + item["write_authority"] or item["transitive_evidence_allowed"] + for item in expansions + ): + raise RuntimeError("expansion inherited authority or certification") + + decommission = assess_decommission( + DecommissionRequest( + schema="qdl.runtime-decommission-request.v1", + request_id=uid("blocked-decommission"), + runtime_id="python-authoritative-runtime", + owned_slice_ids=(), + rollback_reference_ids=("phase92-python-rollback",), + consumer_dependency_ids=(), + all_replacement_windows_closed=False, + repository_cleanup_approved=False, + shared_knowledge_retained=True, + ) + ) + if decommission.allowed: + raise RuntimeError("rollback dependency allowed decommission") + + after_topology = v1_topology() + after_health = v1_health() + evidence = { + "schema": "qdl.phase93.hold-close-expand-certification.v1", + "status": "COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED", + "issued_at_ns": now_ns, + "slice_id": candidate.payload["slice_id"], + "candidate_digest": candidate.digest, + "prerequisite_decision": no_go["decision"], + "production_authorized": False, + "production_hold_started": False, + "production_rollback_window_closed": False, + "production_expansions_authorized": 0, + "python_decommission_authorized": False, + "production_mutations": 0, + "parent_phase92": { + "status": phase92["status"], + "authentic_events": phase92["parity"]["capture"]["events"], + "semantic_mismatches": phase92["parity"]["semantic_mismatches"], + "production_authorized": phase92["production_authorized"], + "sha256": hashlib.sha256(PHASE92.read_bytes()).hexdigest(), + }, + "control_plane_fixture": { + "provenance": "TEST_CONTROL_PLANE_FIXTURE", + "accelerated_time_is_production_evidence": False, + "test_hold_status": hold.status, + "test_hold_production_authorized": hold.production_authorized, + "current_no_go_rejection": denied.reason, + "local_phase92_production_eligible": False, + "expansion_manifest_count": len(expansions), + "expansions": expansions, + "decommission_reason": decommission.reason, + }, + "migration": migration, + "v1": { + "health_before": before_health, + "health_after": after_health, + "topology_unchanged": before_topology == after_topology, + }, + "cleanup": { + "resources_remaining": 0, + "production_rows_created": 0, + }, + "technical_debt": [ + "Phase 9.0-C production infrastructure remains NO_GO_EXTERNAL", + "real production primary and sustained hold observations do not exist", + "real consumer checkpoints and operator closure approval do not exist", + "every expansion remains independently uncertified", + ], + } + if ( + hold.production_authorized + or before_health != 200 + or after_health != 200 + or before_topology != after_topology + ): + raise RuntimeError("Phase 9.3 isolation invariant failed") + freeze(evidence) + print( + json.dumps( + { + "status": evidence["status"], + "parent_authentic_events": phase92["parity"]["capture"]["events"], + "current_no_go_rejection": denied.reason, + "expansion_manifests": len(expansions), + "v1_health_before": before_health, + "v1_health_after": after_health, + "v1_topology_unchanged": before_topology == after_topology, + "production_mutations": 0, + "cleanup": True, + }, + sort_keys=True, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase93_migration_smoke.py b/scripts/phase93_migration_smoke.py new file mode 100755 index 0000000..c15ef6d --- /dev/null +++ b/scripts/phase93_migration_smoke.py @@ -0,0 +1,660 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import sys +import time +from datetime import datetime, timedelta, timezone + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.certification.prerequisites import CandidateSlice # noqa: E402 + + +IMAGE = os.environ.get( + "QDL_PHASE93_POSTGRES_IMAGE", "timescale/timescaledb:latest-pg15" +) +CONTAINER = os.environ.get( + "QDL_PHASE93_POSTGRES_CONTAINER", f"qdl_phase93_postgres_{os.getpid()}" +) +OUTPUT = pathlib.Path( + os.environ.get( + "QDL_PHASE93_MIGRATION_OUTPUT", + str(ROOT / "upgrade/evidence/phase93-hold-close-migration.json"), + ) +) +CANDIDATE = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") +DIGEST = CANDIDATE.digest +SLICE = "production/binance/usdm/perpetual/trade/plan-1/btcusdt" +BUNDLE = "00000000-0000-4000-8000-000000000093" + + +def run( + args: list[str], + *, + input_text: str | None = None, + check: bool = True, +) -> subprocess.CompletedProcess[str]: + result = subprocess.run( + args, + cwd=ROOT, + input=input_text, + text=True, + capture_output=True, + check=False, + ) + if check and result.returncode: + raise RuntimeError( + f"command failed ({result.returncode}): {' '.join(args)}\n" + f"stdout={result.stdout[-4000:]}\nstderr={result.stderr[-4000:]}" + ) + return result + + +def psql(sql: str, *, check: bool = True) -> subprocess.CompletedProcess[str]: + return run( + [ + "docker", + "exec", + "-i", + CONTAINER, + "psql", + "-v", + "ON_ERROR_STOP=1", + "-U", + "postgres", + "-d", + "postgres", + ], + input_text=sql, + check=check, + ) + + +def query(sql: str) -> str: + return run( + [ + "docker", + "exec", + CONTAINER, + "psql", + "-U", + "postgres", + "-d", + "postgres", + "-Atc", + sql, + ] + ).stdout.strip() + + +def expect_failure(sql: str) -> None: + if psql(sql, check=False).returncode == 0: + raise RuntimeError(f"expected SQL failure but statement succeeded: {sql}") + + +def apply_migrations() -> None: + for path in sorted((ROOT / "migrations/postgres").glob("*.sql")): + run( + [ + "docker", + "exec", + CONTAINER, + "psql", + "-v", + "ON_ERROR_STOP=1", + "-U", + "postgres", + "-d", + "postgres", + "-f", + f"/migrations/{path.name}", + ] + ) + + +def cleanup() -> None: + run(["docker", "rm", "-f", CONTAINER], check=False) + + +def main() -> int: + now = datetime.now(timezone.utc).replace(microsecond=0) + t = lambda value: value.isoformat() + cleanup() + run( + [ + "docker", + "run", + "-d", + "--name", + CONTAINER, + "--network", + "none", + "--security-opt", + "no-new-privileges:true", + "--pids-limit", + "256", + "--memory", + "768m", + "--cpus", + "1.0", + "--tmpfs", + "/var/lib/postgresql/data:rw,nosuid,nodev,size=512m", + "-e", + "POSTGRES_HOST_AUTH_METHOD=trust", + "-v", + f"{ROOT / 'migrations/postgres'}:/migrations:ro", + IMAGE, + ] + ) + try: + consecutive_ready = 0 + for _ in range(90): + ready = run( + [ + "docker", + "exec", + CONTAINER, + "pg_isready", + "-U", + "postgres", + "-d", + "postgres", + ], + check=False, + ) + consecutive_ready = ( + consecutive_ready + 1 if ready.returncode == 0 else 0 + ) + if consecutive_ready >= 3: + break + time.sleep(1) + else: + raise RuntimeError("Phase 9.3 PostgreSQL did not become stably ready") + apply_migrations() + + psql( + f""" +INSERT INTO qdl_production_prerequisite_bundles ( + bundle_id,candidate_digest,policy_revision,decision,evidence,evidence_sha256, + issued_by,issued_at,expires_at +) VALUES ( + '{BUNDLE}','{DIGEST}',1,'GO','{{}}',repeat('5',64),'phase93-test', + '{t(now - timedelta(minutes=5))}','{t(now + timedelta(days=1))}' +); +INSERT INTO qdl_authority_slices ( + slice_id,environment,venue,market,product_type,feed,partition_plan_epoch, + partition_id,schema_major,state,authority_revision,owner_id,lease_epoch, + terminal_watermark,candidate_digest,artifact_image_digest,sbom_digest, + signature_identity,contract_digest,normalizer_version,adapter_version, + config_revision,instrument_catalog_revision,source_policy_revision, + partition_plan_digest,rollback_manifest_digest,prerequisite_bundle_id, + approved_by,approved_at,hold_until +) VALUES ( + '{SLICE}','production','BINANCE','USDM','PERPETUAL','TRADE',1, + 'rendezvous-sha256-v1:epoch-1:btcusdt',2,'RUST_CANARY',3, + 'python-primary',1,100,'{DIGEST}','sha256:{DIGEST}',repeat('1',64), + 'phase93-test-signer',repeat('2',64),'qdl-rust-core/test', + 'binance-usdm/test','phase93-test-config','phase93-test-catalog', + 'phase93-test-source-policy',repeat('3',64),repeat('4',64),'{BUNDLE}', + 'phase93-test','{t(now)}','{t(now + timedelta(hours=2))}' +); +INSERT INTO qdl_terminal_owner_checkpoints ( + checkpoint_id,slice_id,owner_id,authority_revision,lease_epoch, + partition_plan_epoch,source_session_id,connection_generation, + terminal_watermark,terminal_event_id,terminal_payload_sha256, + candidate_digest,committed_at +) VALUES ( + '91000000-0000-4000-8000-000000000093','{SLICE}','python-primary', + 3,1,1,'python-session-93',1,100,'event-100',repeat('6',64), + '{DIGEST}','{t(now)}' +); +INSERT INTO qdl_authority_handoffs ( + handoff_id,checkpoint_id,direction,slice_id,old_owner_id,new_owner_id, + expected_state,new_state,expected_authority_revision,new_authority_revision, + expected_lease_epoch,new_lease_epoch,partition_plan_epoch, + terminal_watermark,first_new_watermark,overlap_start_watermark, + overlap_end_watermark,old_event_count,new_event_count,semantic_mismatches, + open_gaps,candidate_digest,prerequisite_bundle_id,handoff_sha256, + approved_by,approved_at,expires_at +) VALUES ( + '92000000-0000-4000-8000-000000000093', + '91000000-0000-4000-8000-000000000093','PYTHON_TO_RUST','{SLICE}', + 'python-primary','rust-primary','RUST_CANARY','RUST_PRIMARY',3,4,1,2,1, + 100,101,90,100,11,11,0,0,'{DIGEST}','{BUNDLE}',repeat('7',64), + 'phase93-test','{t(now)}','{t(now + timedelta(hours=2))}' +); +SELECT (qdl_transition_authority_v2( + '92000000-0000-4000-8000-000000000093', + '93000000-0000-4000-8000-000000000093','{SLICE}', + 'RUST_CANARY',3,'python-primary',1,1,'RUST_PRIMARY','rust-primary',2, + 100,'{BUNDLE}','{t(now + timedelta(hours=1))}', + 'phase93-test','accepted primary handoff' +)).state; +""" + ) + + psql( + f""" +INSERT INTO qdl_primary_holds ( + hold_id,slice_id,candidate_digest,prerequisite_bundle_id,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,started_at, + required_until,policy_digest,minimum_duration_seconds, + max_sample_gap_seconds,max_lag_ms,max_freshness_ms,max_queue_depth, + max_spool_bytes,max_cpu_percent,max_rss_mb +) VALUES +( + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}','{BUNDLE}', + 'rust-primary',4,2,1,'{t(now - timedelta(minutes=2))}','{t(now)}', + repeat('8',64),120,60,500,1000,1000,1000000,80,512 +), +( + '94000000-0000-4000-8000-000000000002','{SLICE}','{DIGEST}','{BUNDLE}', + 'rust-primary',4,2,1,'{t(now - timedelta(minutes=1))}','{t(now)}', + repeat('9',64),60,60,500,1000,1000,1000000,80,512 +); +INSERT INTO qdl_primary_hold_observations ( + observation_id,hold_id,slice_id,candidate_digest,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,sequence,observed_at, + last_watermark,lag_ms,freshness_ms,queue_depth,spool_bytes,cpu_percent, + rss_mb,registered_consumers,healthy_consumers,checkpoint_watermark +) VALUES +( + '95000000-0000-4000-8000-000000000001', + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,1,'{t(now - timedelta(minutes=1))}',110, + 10,20,1,100,10,64,2,2,110 +), +( + '95000000-0000-4000-8000-000000000002', + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,2,'{t(now)}',120, + 10,20,1,100,10,64,2,2,120 +); +INSERT INTO qdl_primary_hold_observations ( + observation_id,hold_id,slice_id,candidate_digest,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,sequence,observed_at, + last_watermark,semantic_mismatches,lag_ms,freshness_ms,queue_depth, + spool_bytes,cpu_percent,rss_mb,registered_consumers,healthy_consumers, + checkpoint_watermark +) VALUES ( + '95000000-0000-4000-8000-000000000003', + '94000000-0000-4000-8000-000000000002','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,1,'{t(now)}',120,1,10,20,1,100,10,64,2,2,120 +); +INSERT INTO qdl_primary_hold_decisions ( + decision_id,hold_id,status,reason,scope,production_authorized,slice_id, + candidate_digest,prerequisite_bundle_id,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,policy_digest,first_observed_at, + last_observed_at,observation_count,terminal_watermark,decided_at, + decision_sha256 +) VALUES ( + '96000000-0000-4000-8000-000000000002', + '94000000-0000-4000-8000-000000000002','BLOCKED','SEMANTIC_MISMATCH', + 'TEST_REHEARSAL',FALSE,'{SLICE}','{DIGEST}','{BUNDLE}','rust-primary', + 4,2,1,repeat('9',64),'{t(now)}','{t(now)}',1,120,'{t(now)}', + repeat('b',64) +); +""" + ) + + expect_failure( + f""" +INSERT INTO qdl_primary_hold_observations ( + observation_id,hold_id,slice_id,candidate_digest,owner_id, + authority_revision,lease_epoch,partition_plan_epoch,sequence,observed_at, + last_watermark,registered_consumers,healthy_consumers,checkpoint_watermark +) VALUES ( + '95000000-0000-4000-8000-000000000009', + '94000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,4,'{t(now + timedelta(minutes=1))}',121,2,2,121 +); +""" + ) + expect_failure( + f""" +INSERT INTO qdl_primary_hold_decisions ( + decision_id,hold_id,status,reason,scope,production_authorized,slice_id, + candidate_digest,prerequisite_bundle_id,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,policy_digest,first_observed_at, + last_observed_at,observation_count,terminal_watermark,decided_at, + decision_sha256 +) VALUES ( + '96000000-0000-4000-8000-000000000009', + '94000000-0000-4000-8000-000000000002','PASSED','PASS','PRODUCTION', + TRUE,'{SLICE}','{DIGEST}','{BUNDLE}','rust-primary',4,2,1,repeat('9',64), + '{t(now)}','{t(now)}',1,120,'{t(now)}',repeat('c',64) +); +""" + ) + + psql( + f""" +INSERT INTO qdl_primary_hold_decisions ( + decision_id,hold_id,status,reason,scope,production_authorized,slice_id, + candidate_digest,prerequisite_bundle_id,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,policy_digest,first_observed_at, + last_observed_at,observation_count,terminal_watermark,decided_at, + decision_sha256 +) VALUES ( + '96000000-0000-4000-8000-000000000001', + '94000000-0000-4000-8000-000000000001','PASSED','PASS','PRODUCTION', + TRUE,'{SLICE}','{DIGEST}','{BUNDLE}','rust-primary',4,2,1,repeat('8',64), + '{t(now - timedelta(minutes=1))}','{t(now)}',2,120,'{t(now)}', + repeat('a',64) +); +INSERT INTO qdl_consumer_registry_snapshots ( + snapshot_id,slice_id,authority_revision,checkpoint_count, + ready_checkpoint_count,minimum_checkpoint_watermark, + checkpoint_regressions,unresolved_migrations,rollback_ready, + registry_sha256,details,observed_at +) VALUES ( + '97000000-0000-4000-8000-000000000001','{SLICE}',4,2,2,130, + 0,0,TRUE,repeat('c',64),'{{"consumers":["alpha","execution"]}}','{t(now)}' +); +INSERT INTO qdl_authority_registry_snapshots ( + snapshot_id,slice_id,state,owner_id,authority_revision,lease_epoch, + partition_plan_epoch,candidate_digest,prerequisite_bundle_id, + current_watermark,public_write_allowed,legacy_write_allowed, + registry_sha256,observed_at +) VALUES ( + '97000000-0000-4000-8000-000000000002','{SLICE}','RUST_PRIMARY', + 'rust-primary',4,2,1,'{DIGEST}','{BUNDLE}',130,TRUE,TRUE, + repeat('d',64),'{t(now)}' +); +INSERT INTO qdl_rollback_rehearsals ( + rehearsal_id,slice_id,candidate_digest,owner_id,authority_revision, + lease_epoch,partition_plan_epoch,rollback_manifest_digest, + reconciled_through_watermark,rto_ms,status,production_scope, + rehearsal_sha256,observed_at,expires_at +) VALUES ( + '97000000-0000-4000-8000-000000000003','{SLICE}','{DIGEST}', + 'rust-primary',4,2,1,repeat('4',64),130,500,'PASS',TRUE, + repeat('e',64),'{t(now)}','{t(now + timedelta(hours=1))}' +); +INSERT INTO qdl_closure_approvals ( + approval_id,slice_id,candidate_digest,prerequisite_bundle_id,hold_id, + hold_policy_digest,decision,allow_close_rollback_window, + repository_cleanup_approved,operator,change_ticket,approval_sha256, + approved_at,expires_at +) VALUES ( + '97000000-0000-4000-8000-000000000004','{SLICE}','{DIGEST}','{BUNDLE}', + '94000000-0000-4000-8000-000000000001',repeat('8',64),'APPROVE', + TRUE,FALSE,'phase93-test','QDL-93',repeat('f',64), + '{t(now - timedelta(minutes=1))}','{t(now + timedelta(hours=1))}' +); +SELECT (qdl_close_authority_window( + '98000000-0000-4000-8000-000000000001',repeat('1',64), + '96000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000002', + '97000000-0000-4000-8000-000000000003', + '97000000-0000-4000-8000-000000000004','{t(now)}' +)).closure_id; +""" + ) + + closure_state = query( + f"SELECT state||':'||authority_revision||':'||owner_id||':'||" + f"lease_epoch||':'||terminal_watermark FROM qdl_authority_slices " + f"WHERE slice_id='{SLICE}';" + ) + if closure_state != "RUST_PRIMARY:4:rust-primary:2:100": + raise RuntimeError(f"closure mutated authority: {closure_state}") + + expect_failure( + "UPDATE qdl_authority_closures SET change_ticket='mutated';" + ) + expect_failure("DELETE FROM qdl_primary_hold_observations;") + expect_failure( + "UPDATE qdl_consumer_registry_snapshots SET checkpoint_count=3;" + ) + expect_failure( + f""" +INSERT INTO qdl_expansion_candidates ( + expansion_id,parent_closure_id,parent_slice_id,parent_candidate_digest, + parent_closure_digest,expansion_type,candidate_digest,scope_digest, + partition_plan_epoch,required_gates,status,transitive_evidence_allowed, + public_write_allowed,legacy_write_allowed,created_at +) VALUES ( + '99000000-0000-4000-8000-000000000009', + '98000000-0000-4000-8000-000000000001','{SLICE}','{DIGEST}', + repeat('1',64),'BBO',repeat('2',64),repeat('3',64),1, + ARRAY['rollback'],'INDEPENDENT_CERTIFICATION_REQUIRED',FALSE,FALSE,FALSE, + '{t(now)}' +); +""" + ) + + expansion_rows = [ + ( + "99000000-0000-4000-8000-000000000001", + "INSTRUMENT_PARTITION", + "2", + "3", + 2, + [ + "authority_handoff", + "capacity_headroom", + "exact_frame_parity", + "partition_churn", + "provider_authentic_source", + "rollback", + "source_capacity", + ], + ), + ( + "99000000-0000-4000-8000-000000000002", + "BBO", + "3", + "4", + 1, + [ + "authority_handoff", + "capacity_headroom", + "coalescing_policy", + "exact_frame_parity", + "freshness", + "ordering_reconnect", + "provider_authentic_source", + "quote_identity", + "rollback", + ], + ), + ( + "99000000-0000-4000-8000-000000000003", + "L2_BOOK", + "4", + "5", + 1, + [ + "authority_handoff", + "capacity_headroom", + "checksum", + "exact_frame_parity", + "lossless_backpressure", + "provider_authentic_source", + "resync", + "rollback", + "snapshot_delta_sequence", + ], + ), + ( + "99000000-0000-4000-8000-000000000004", + "BAR_LIFECYCLE", + "5", + "6", + 1, + [ + "authority_handoff", + "capacity_headroom", + "close_time_semantics", + "exact_frame_parity", + "final_revision_lineage", + "provider_authentic_source", + "replay", + "rollback", + ], + ), + ( + "99000000-0000-4000-8000-000000000005", + "VENUE_MARKET", + "6", + "7", + 1, + [ + "adapter_capability", + "authority_handoff", + "capacity_headroom", + "disaster_recovery", + "entitlement", + "exact_frame_parity", + "instrument_identity", + "provider_authentic_source", + "provider_semantics", + "rollback", + ], + ), + ] + for expansion_id, kind, candidate_char, scope_char, epoch, gates in expansion_rows: + gate_sql = ",".join(f"'{value}'" for value in gates) + psql( + f""" +INSERT INTO qdl_expansion_candidates ( + expansion_id,parent_closure_id,parent_slice_id,parent_candidate_digest, + parent_closure_digest,expansion_type,candidate_digest,scope_digest, + partition_plan_epoch,required_gates,status,transitive_evidence_allowed, + public_write_allowed,legacy_write_allowed,created_at +) VALUES ( + '{expansion_id}','98000000-0000-4000-8000-000000000001','{SLICE}', + '{DIGEST}',repeat('1',64),'{kind}',repeat('{candidate_char}',64), + repeat('{scope_char}',64),{epoch},ARRAY[{gate_sql}], + 'INDEPENDENT_CERTIFICATION_REQUIRED',FALSE,FALSE,FALSE,'{t(now)}' +); +""" + ) + + psql( + f""" +INSERT INTO qdl_runtime_decommission_decisions ( + decision_id,runtime_id,owned_slice_count,rollback_reference_count, + consumer_dependency_count,all_replacement_windows_closed, + repository_cleanup_approved,shared_knowledge_retained,allowed,reason, + decided_at +) VALUES +( + '99000000-0000-4000-8000-000000000006','python-usdm-trade', + 1,1,1,FALSE,FALSE,TRUE,FALSE,'RUNTIME_STILL_OWNS_SLICES','{t(now)}' +), +( + '99000000-0000-4000-8000-000000000007','retired-test-runtime', + 0,0,0,TRUE,TRUE,TRUE,TRUE,'AUTHORIZED','{t(now)}' +); +SELECT qdl_transition_authority( + '99000000-0000-4000-8000-000000000008','{SLICE}', + 'RUST_PRIMARY',4,'rust-primary',2,1,'BLOCKED','rust-primary',2, + 130,NULL,NULL,'phase93-test','stale closure CAS test' +); +""" + ) + expect_failure( + f""" +SELECT qdl_close_authority_window( + '98000000-0000-4000-8000-000000000009',repeat('9',64), + '96000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000001', + '97000000-0000-4000-8000-000000000002', + '97000000-0000-4000-8000-000000000003', + '97000000-0000-4000-8000-000000000004','{t(now)}' +); +""" + ) + + apply_migrations() + final_state = query( + f"SELECT state||':'||authority_revision||':'||owner_id||':'||" + f"lease_epoch||':'||terminal_watermark FROM qdl_authority_slices " + f"WHERE slice_id='{SLICE}';" + ) + counts = query( + "SELECT " + "(SELECT count(*) FROM qdl_primary_holds)||':'||" + "(SELECT count(*) FROM qdl_primary_hold_observations)||':'||" + "(SELECT count(*) FROM qdl_primary_hold_decisions)||':'||" + "(SELECT count(*) FROM qdl_authority_closures)||':'||" + "(SELECT count(*) FROM qdl_expansion_candidates)||':'||" + "(SELECT count(*) FROM qdl_runtime_decommission_decisions);" + ) + if final_state != "BLOCKED:5:rust-primary:2:130": + raise RuntimeError(f"unexpected final test state: {final_state}") + if counts != "2:3:2:1:5:2": + raise RuntimeError(f"unexpected Phase 9.3 row counts: {counts}") + + values = [int(value) for value in counts.split(":")] + OUTPUT.parent.mkdir(parents=True, exist_ok=True) + OUTPUT.write_text( + json.dumps( + { + "schema": "qdl.phase93.hold-close-migration.v1", + "status": "PASS", + "authority_state_after_closure": closure_state, + "final_test_state_after_stale_cas_setup": final_state, + "hold_records": values[0], + "observation_records": values[1], + "decision_records": values[2], + "closure_records": values[3], + "expansion_records": values[4], + "decommission_records": values[5], + "dirty_hold_pass_rejected": True, + "out_of_order_observation_rejected": True, + "append_only_mutation_rejected": True, + "registry_mutation_rejected": True, + "closure_did_not_mutate_authority": True, + "closure_digest_bound_to_expansion": True, + "approval_and_closure_ids_distinct": True, + "stale_authority_closure_rejected": True, + "incomplete_expansion_gates_rejected": True, + "all_expansion_types_registered_independently": True, + "idempotent_migration": True, + "production_mutations": 0, + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + print( + json.dumps( + { + "status": "PASS", + "closure_state": closure_state, + "final_test_state": final_state, + "counts": counts, + "cleanup": True, + }, + sort_keys=True, + ) + ) + return 0 + finally: + cleanup() + remaining = run( + [ + "docker", + "ps", + "-aq", + "--filter", + f"name=^/{CONTAINER}$", + ], + check=False, + ).stdout.strip() + if remaining: + raise RuntimeError("Phase 9.3 PostgreSQL cleanup failed") + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_freeze_core_benchmark.py b/scripts/phasea_freeze_core_benchmark.py new file mode 100755 index 0000000..1b6e61c --- /dev/null +++ b/scripts/phasea_freeze_core_benchmark.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import subprocess +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--binary", type=Path, default=ROOT / "target/release/qdl-realtime-core-benchmark") + parser.add_argument("--events", type=int, default=100_000) + parser.add_argument("--minimum-events-per-second", type=int, default=50_000) + parser.add_argument("--output", type=Path, default=ROOT / "upgrade/evidence/phase-a-realtime-core-capacity.json") + args = parser.parse_args() + completed = subprocess.run( + [str(args.binary), str(args.events), str(args.minimum_events_per_second)], + cwd=ROOT, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=120, + check=False, + ) + if completed.returncode != 0: + raise RuntimeError(f"Rust core benchmark failed: {completed.stderr[-1000:]}") + payload = json.loads(completed.stdout.strip().splitlines()[-1]) + payload.update({ + "schema": "qdl.phase-a.realtime-core-capacity.v1", + "provenance": "SYNTHETIC_CAPACITY_ONLY", + "binary_sha256": hashlib.sha256(args.binary.read_bytes()).hexdigest(), + "production_writes": 0, + }) + args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + print(json.dumps({ + "status": payload["status"], + "events": payload["events"], + "events_per_second": payload["events_per_second"], + "p99_ns": payload["p99_ns"], + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_inspect_kafka.py b/scripts/phasea_inspect_kafka.py new file mode 100755 index 0000000..9cccb9e --- /dev/null +++ b/scripts/phasea_inspect_kafka.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import time +from pathlib import Path + +from confluent_kafka import Consumer + +from qdl.marketdata.v2 import market_data_pb2 +from qdl.provider.v1 import raw_provider_pb2 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--bootstrap", required=True) + parser.add_argument("--canonical-topic", required=True) + parser.add_argument("--quarantine-topic", required=True) + parser.add_argument("--cert-root", type=Path, required=True) + parser.add_argument("--expected-canonical", type=int, required=True) + parser.add_argument("--expected-quarantine", type=int, required=True) + parser.add_argument("--timeout-seconds", type=float, default=30) + parser.add_argument("--required-products", default="") + args = parser.parse_args() + consumer = Consumer({ + "bootstrap.servers": args.bootstrap, + "group.id": f"phase8-phase-a-inspect-{time.time_ns()}", + "security.protocol": "ssl", + "ssl.ca.location": str(args.cert_root / "ca.crt"), + "ssl.certificate.location": str(args.cert_root / "client.crt"), + "ssl.key.location": str(args.cert_root / "client.key"), + "enable.auto.commit": False, + "auto.offset.reset": "earliest", + "isolation.level": "read_committed", + }) + consumer.subscribe([args.canonical_topic, args.quarantine_topic]) + canonical = [] + quarantines = [] + deadline = time.monotonic() + args.timeout_seconds + while time.monotonic() < deadline and ( + len(canonical) < args.expected_canonical + or len(quarantines) < args.expected_quarantine + ): + message = consumer.poll(0.5) + if message is None: + continue + if message.error(): + raise RuntimeError(str(message.error())) + if message.topic() == args.canonical_topic: + envelope = market_data_pb2.EventEnvelope.FromString(message.value()) + canonical.append({ + "event_id": envelope.event_id.hex(), + "venue": envelope.venue, + "market": envelope.market, + "product_type": envelope.product_type, + "payload": envelope.WhichOneof("payload"), + "source_role": envelope.source_role, + }) + else: + value = raw_provider_pb2.QuarantineRecord.FromString(message.value()) + raw = value.raw + quarantines.append({ + "reason": value.reason, + "evidence": value.evidence_sha256.hex(), + "provider": raw.provider if raw else None, + "venue": raw.venue if raw else None, + "market": raw.market if raw else None, + "native_symbol": raw.native_symbol if raw else None, + "native_channel": raw.native_channel if raw else None, + "safe_summary": value.safe_summary, + }) + consumer.close() + if len(canonical) != args.expected_canonical or len(quarantines) != args.expected_quarantine: + raise RuntimeError( + f"unexpected visible counts canonical={len(canonical)} quarantine={len(quarantines)}" + ) + event_ids = [item["event_id"] for item in canonical] + if len(event_ids) != len(set(event_ids)): + raise RuntimeError("committed canonical topic contains duplicate event ids") + venues = sorted({item["venue"] for item in canonical}) + products = sorted({ + f"{item['venue']}/{item['market']}/{item['product_type']}" + for item in canonical + }) + required_products = { + value for value in args.required_products.split(",") if value + } + if required_products and not required_products.issubset(products): + raise RuntimeError( + f"multivenue canonical product coverage is incomplete: {products}" + ) + print(json.dumps({ + "status": "PASS", + "canonical": len(canonical), + "quarantines": len(quarantines), + "venues": venues, + "products": products, + "canonical_digest": hashlib.sha256( + json.dumps(canonical, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest(), + "quarantine_reasons": [item["reason"] for item in quarantines], + "quarantine_details": quarantines, + "production_writes": 0, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_real_provider_certification.py b/scripts/phasea_real_provider_certification.py new file mode 100755 index 0000000..37b3b53 --- /dev/null +++ b/scripts/phasea_real_provider_certification.py @@ -0,0 +1,480 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import shutil +import sys +import tempfile +import time + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +os.environ["QDL_PHASE8_PROJECT"] = "qdl_phasea_real_provider" + +from scripts.phase80_broker_certification import ( # noqa: E402 + BOOTSTRAP, + PROJECT, + add_acls, + cleanup, + compose, + create_topic, + kafka, + run, + v1_health, + v1_topology, + wait_for_cluster, + wait_for_replicas, +) + + +RUST_IMAGE = "qdl-phase-a-rust:real-provider" +PYTHON_IMAGE = "data-layer:phase-a-real-provider" +EVIDENCE = ROOT / "upgrade/evidence/phase-a-real-provider-core.json" +RAW_TOPIC = "qdl.phase8.phasea.real.raw.v1" +CANONICAL_TOPIC = "qdl.phase8.phasea.real.canonical.v2" +QUARANTINE_TOPIC = "qdl.phase8.phasea.real.quarantine.v1" +GROUP_ID = "phase8-phase-a-real-core" + + +def client_dir(root: pathlib.Path, principal: str) -> pathlib.Path: + target = root / principal + target.mkdir(mode=0o755) + for source, destination in ( + (root / "ca.crt", target / "ca.crt"), + (root / f"{principal}.crt", target / "client.crt"), + (root / f"{principal}.key", target / "client.key"), + ): + shutil.copyfile(source, destination) + destination.chmod(0o644) + return target + + +def add_core_acls(env: dict[str, str]) -> None: + common = ( + "--bootstrap-server", BOOTSTRAP, + "--command-config", "/etc/kafka/secrets/admin.properties", + "--add", "--allow-principal", "User:phase8-core", + ) + for topic, operations in ( + (RAW_TOPIC, ("READ", "DESCRIBE")), + (CANONICAL_TOPIC, ("WRITE", "DESCRIBE")), + (QUARANTINE_TOPIC, ("WRITE", "DESCRIBE")), + ): + args = list(common) + for operation in operations: + args.extend(("--operation", operation)) + args.extend(("--topic", topic)) + kafka(env, "kafka-acls.sh", *args) + kafka(env, "kafka-acls.sh", *common, "--operation", "READ", "--group", GROUP_ID) + kafka(env, "kafka-acls.sh", *common, "--operation", "IdempotentWrite", "--cluster") + kafka( + env, "kafka-acls.sh", *common, + "--operation", "WRITE", "--operation", "DESCRIBE", + "--transactional-id", "qdl-phase-a-real-", "--resource-pattern-type", "prefixed", + ) + + +def authority(image_id: str) -> dict: + digest = image_id.removeprefix("sha256:") + return { + "schema": "qdl.authority-record.v1", + "slice_id": "phase-a-real-provider-shadow", + "revision": 1, + "mode": "RUST_SHADOW", + "candidate_image_digest": f"sha256:{digest}", + "capability_manifest_digest": hashlib.sha256( + (ROOT / "config/v2/stable-capabilities.yaml").read_bytes() + ).hexdigest(), + "contract_digest": hashlib.sha256( + (ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto").read_bytes() + ).hexdigest(), + "partition_plan_digest": hashlib.sha256(b"phase-a-real-provider-plan").hexdigest(), + "public_write_allowed": False, + "legacy_write_allowed": False, + "approved_by": "phase-a-real-provider-certification", + "effective_at_ns": time.time_ns(), + } + + +def docker_run( + image: str, + certs: pathlib.Path, + command: list[str], + *, + entrypoint: str | None = None, + mounts: tuple[str, ...] = (), + timeout: float = 120.0, + egress: bool = False, +): + args = [ + "docker", "run", "--rm", "--network", f"{PROJECT}_phase8_shadow", + "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", f"type=bind,source={certs},target=/certs,readonly", + "--env", f"QDL_KAFKA_BOOTSTRAP_SERVERS={BOOTSTRAP}", + "--env", "QDL_KAFKA_CERT_ROOT=/certs", + "--env", "QDL_KAFKA_CLIENT_ID=qdl-phase-a-real", + "--env", f"QDL_KAFKA_GROUP_ID={GROUP_ID}", + ] + for mount in mounts: + args.extend(("--mount", mount)) + if entrypoint: + args.extend(("--entrypoint", entrypoint)) + args.append(image) + args.extend(command) + if not egress: + return run(args, timeout=timeout) + + name = f"qdl-phase-a-egress-{time.time_ns()}" + create_args = args.copy() + create_args[1:3] = ["create", "--name", name] + run(create_args, timeout=30.0) + try: + run(["docker", "network", "connect", "bridge", name], timeout=30.0) + return run(["docker", "start", "--attach", name], timeout=timeout) + finally: + run(["docker", "rm", "--force", name], check=False, timeout=30.0) + +def crypto_specs(authority_record: dict): + products = ( + ("BINANCE", "USDM", "PERPETUAL", "BTCUSDT", "BINANCE_DIRECT", + "wss://fstream.binance.com/public/stream", None, + (("trade", "binance_usdm_trade", 3), ("bookTicker", "binance_usdm_bbo", 2))), + ("BINANCE", "SPOT", "SPOT", "BTCUSDT", "BINANCE_DIRECT", + "wss://stream.binance.com:9443/stream", None, + (("trade", "binance_spot_trade", 3), ("bookTicker", "binance_spot_bbo", 2))), + ("OKX", "SWAP", "PERPETUAL", "BTC-USDT-SWAP", "OKX_DIRECT", + "wss://ws.okx.com:8443/ws/v5/public", "wss://ws.okx.com:8443/ws/v5/business", + (("trades", "okx_trade", 3), ("bbo-tbt", "okx_bbo", 2), ("candle1m", "okx_bar", 1))), + ("OKX", "SPOT", "SPOT", "BTC-USDT", "OKX_DIRECT", + "wss://ws.okx.com:8443/ws/v5/public", "wss://ws.okx.com:8443/ws/v5/business", + (("trades", "okx_trade", 3), ("bbo-tbt", "okx_bbo", 2), ("candle1m", "okx_bar", 1))), + ) + specs = [] + core_bindings = [] + for venue, market, product, symbol, provider, public_url, business_url, feeds in products: + runtime = "BINANCE" if venue == "BINANCE" else "OKX" + for channel, provider_kind, count in feeds: + native_channel = f"{symbol.lower()}@{channel}" if venue == "BINANCE" else channel + source_id = f"{venue.lower()}-{market.lower()}-{channel.lower()}" + binding = { + "provider": provider, + "venue": venue, + "market": market, + "product_type": product, + "native_symbol": symbol, + "native_channel": native_channel, + "subscription_id": f"phase-a-real-{source_id}", + "adapter_version": f"{venue.lower()}-json/2.0.0", + "instrument_catalog_revision": 3, + } + specs.append({ + "runtime": runtime, + "websocket_url": public_url, + "business_websocket_url": business_url, + "raw_stream": RAW_TOPIC, + "shard_id": f"phase-a-real-{source_id}", + "lease_epoch": 7, + "partition_plan_epoch": 1, + "config_revision": 1, + "heartbeat_seconds": 15, + "max_events": count, + "max_runtime_seconds": 45, + "metrics_every_events": 1, + "generation_state_path": f"/state/{source_id}", + "max_inflight_publishes": 64, + "authority": authority_record, + "bindings": [binding], + }) + core_bindings.append({ + "provider": provider, + "venue": venue, + "market": market, + "product_type": product, + "native_symbol": symbol, + "native_channel": native_channel, + "provider_kind": provider_kind, + "instrument_uid": f"uid-{venue.lower()}-{market.lower()}-{symbol.lower()}", + "instrument_id": f"{venue}.{market}.{product}.{symbol}", + "instrument_revision": 1, + "instrument_catalog_revision": 3, + "source_id": source_id, + "source_role": "PRIMARY", + "normalizer_version": "qdl-rust-core/2.0.0", + "sequence_policy": "MONOTONIC" if channel != "kline_1m" and channel != "candle1m" else "NONE", + }) + for market, product, provider_kind in ( + ("USDM", "PERPETUAL", "binance_usdm_rest_bar"), + ("SPOT", "SPOT", "binance_spot_rest_bar"), + ): + core_bindings.append({ + "provider": "BINANCE_DIRECT", + "venue": "BINANCE", + "market": market, + "product_type": product, + "native_symbol": "BTCUSDT", + "native_channel": "rest-klines/1m", + "provider_kind": provider_kind, + "instrument_uid": f"uid-binance-{market.lower()}-btcusdt", + "instrument_id": f"BINANCE.{market}.{product}.BTCUSDT", + "instrument_revision": 1, + "instrument_catalog_revision": 3, + "source_id": f"binance-{market.lower()}-rest-bar", + "source_role": "PRIMARY", + "normalizer_version": "qdl-rust-core/2.0.0", + "sequence_policy": "NONE", + }) + return specs, core_bindings + + +def get_dnse_payload() -> dict: + result = run([ + "docker", "exec", "redis_marketdata", "redis-cli", "GET", "vn:quote:last:GEX" + ]) + payload = json.loads(result.stdout) + if payload.get("source") != "dnse" or not payload.get("timestamp"): + raise RuntimeError(f"durable VN payload is not DNSE provider data: {payload}") + return payload + + +def main() -> int: + before_topology = v1_topology() + before_health = v1_health() + result = {} + cleanup_result = {} + with tempfile.TemporaryDirectory(prefix="qdl-phase-a-real-") as directory: + temp = pathlib.Path(directory) + temp.chmod(0o755) + env = os.environ.copy() + env["QDL_PHASE8_CERT_DIR"] = str(temp) + run([str(ROOT / "scripts/phase80_generate_tls.sh"), str(temp)], env=env, timeout=120.0) + producer_certs = client_dir(temp, "phase8-producer") + consumer_certs = client_dir(temp, "phase8-consumer") + core_certs = client_dir(temp, "phase8-core") + try: + run(["docker", "build", "--provenance=false", "-f", str(ROOT / "Dockerfile.phase8-rust"), + "-t", RUST_IMAGE, str(ROOT)], timeout=1200.0) + run(["docker", "build", "--provenance=false", "-t", PYTHON_IMAGE, str(ROOT)], timeout=1200.0) + rust_image_id = run(["docker", "image", "inspect", RUST_IMAGE, "--format", "{{.Id}}"]).stdout.strip() + authority_record = authority(rust_image_id) + specs, core_bindings = crypto_specs(authority_record) + dnse_payload = get_dnse_payload() + dnse_payload_path = temp / "dnse-payload.json" + dnse_payload_path.write_text(json.dumps(dnse_payload, sort_keys=True) + "\n") + dnse_payload_path.chmod(0o644) + symbol = str(dnse_payload["symbol"]).upper() + core_bindings.append({ + "provider": "DNSE_DIRECT", "venue": "HOSE", "market": "EQUITIES", + "product_type": "COMMON_STOCK", "native_symbol": symbol, + "native_channel": "trades", "provider_kind": "dnse_trade", + "instrument_uid": f"uid-hose-{symbol.lower()}", + "instrument_id": f"HOSE.EQUITIES.COMMON_STOCK.{symbol}", + "instrument_revision": 1, "instrument_catalog_revision": 3, + "source_id": f"dnse-{symbol.lower()}-trade", "source_role": "PRIMARY", + "normalizer_version": "qdl-rust-core/2.0.0", "sequence_policy": "NONE", + }) + core_bindings.append({ + "provider": "DNSE_DIRECT", "venue": "HNX", "market": "VN_DERIVATIVES", + "product_type": "FUTURE", "native_symbol": "VN30F1M", + "native_channel": "ohlcv/1m", "provider_kind": "dnse_bar", + "instrument_uid": "uid-hnx-vn30f1m", + "instrument_id": "HNX.VN_DERIVATIVES.FUTURE.VN30F1M", + "instrument_revision": 1, "instrument_catalog_revision": 3, + "source_id": "dnse-vn30f1m-history", "source_role": "BACKFILL", + "normalizer_version": "qdl-rust-core/2.0.0", "sequence_policy": "NONE", + }) + + compose(env, "up", "-d", timeout=180.0) + wait_for_cluster(env) + for topic in (RAW_TOPIC, CANONICAL_TOPIC, QUARANTINE_TOPIC): + create_topic(env, topic, partitions=3) + add_acls(env) + add_core_acls(env) + wait_for_replicas(env) + + native_results = [] + raw_total = 0 + for index, spec in enumerate(specs): + config_path = temp / f"native-{index}.json" + config_path.write_text(json.dumps(spec, indent=2, sort_keys=True) + "\n") + config_path.chmod(0o644) + execution = docker_run( + RUST_IMAGE, + producer_certs, + [f"/config/{config_path.name}"], + entrypoint="/usr/local/bin/qdl-native-raw-ingestor", + mounts=( + f"type=bind,source={config_path},target=/config/{config_path.name},readonly", + f"type=bind,source={temp},target=/state", + ), + timeout=90.0, + egress=True, + ) + summary = json.loads(execution.stdout.strip().splitlines()[-1]) + accepted = int(summary["accepted_raw_frames"]) + identity = { + "runtime": spec["runtime"], + "market": spec["bindings"][0]["market"], + "channel": spec["bindings"][0]["native_channel"], + } + if accepted != int(spec["max_events"]): + raise RuntimeError( + "native provider did not satisfy bounded count " + f"identity={identity}: {summary} stderr={execution.stderr[-2000:]}" + ) + print(json.dumps({ + "event": "phase_a_native_feed_pass", + **identity, + "accepted_raw_frames": accepted, + }, sort_keys=True), flush=True) + raw_total += accepted + native_results.append({ + "runtime": spec["runtime"], + "market": spec["bindings"][0]["market"], + "channel": spec["bindings"][0]["native_channel"], + "accepted_raw_frames": accepted, + }) + + bar_seed = docker_run( + PYTHON_IMAGE, + producer_certs, + ["python", "scripts/phasea_seed_binance_bars.py", "--bootstrap", BOOTSTRAP, + "--topic", RAW_TOPIC, "--cert-root", "/certs"], + timeout=120.0, + egress=True, + ) + bar_result = json.loads(bar_seed.stdout.strip().splitlines()[-1]) + if int(bar_result["raw_acked"]) != 2: + raise RuntimeError(f"Binance REST BAR seed failed: {bar_result}") + raw_total += int(bar_result["raw_acked"]) + + vn_seed = docker_run( + PYTHON_IMAGE, + producer_certs, + ["python", "scripts/phasea_seed_vn_replay.py", "--bootstrap", BOOTSTRAP, + "--topic", RAW_TOPIC, "--cert-root", "/certs", + "--payload", "/input/dnse-payload.json"], + mounts=(f"type=bind,source={dnse_payload_path},target=/input/dnse-payload.json,readonly",), + ) + vn_result = json.loads(vn_seed.stdout.strip().splitlines()[-1]) + raw_total += int(vn_result["raw_acked"]) + + vn_derivative_seed = docker_run( + PYTHON_IMAGE, + producer_certs, + ["python", "scripts/phasea_seed_vn_derivative_history.py", + "--bootstrap", BOOTSTRAP, "--topic", RAW_TOPIC, + "--cert-root", "/certs", "--parquet", "/input/VN30F1M.parquet"], + mounts=( + f"type=bind,source={ROOT / 'data/preload/1m/VN30F1M.parquet'},target=/input/VN30F1M.parquet,readonly", + ), + ) + vn_derivative_result = json.loads( + vn_derivative_seed.stdout.strip().splitlines()[-1] + ) + raw_total += int(vn_derivative_result["raw_acked"]) + + core_config = { + "core": { + "canonical_stream": CANONICAL_TOPIC, + "quarantine_stream": QUARANTINE_TOPIC, + "allow_test_provenance": False, + "dedup_capacity": 100000, + "bindings": core_bindings, + }, + "raw_topics": [RAW_TOPIC], + "authority": authority_record, + "shard_id": "phase-a-real-core-shard", + "transactional_id": "qdl-phase-a-real-core-shard", + "batch_size": 128, + "batch_wait_ms": 100, + "max_events": raw_total, + "metrics_every_batches": 1, + } + core_path = temp / "core-config.json" + core_path.write_text(json.dumps(core_config, indent=2, sort_keys=True) + "\n") + core_path.chmod(0o644) + core_execution = docker_run( + RUST_IMAGE, + core_certs, + ["/config/core-config.json"], + entrypoint="/usr/local/bin/qdl-realtime-core", + mounts=(f"type=bind,source={core_path},target=/config/core-config.json,readonly",), + timeout=120.0, + ) + core_summary = json.loads(core_execution.stdout.strip().splitlines()[-1]) + if core_summary.get("processed") != raw_total: + raise RuntimeError(f"real-provider core processed count failed: {core_summary}") + canonical_count = int(core_summary["canonical"]) + quarantine_count = int(core_summary["quarantines"]) + inspected = docker_run( + PYTHON_IMAGE, + consumer_certs, + ["python", "scripts/phasea_inspect_kafka.py", "--bootstrap", BOOTSTRAP, + "--canonical-topic", CANONICAL_TOPIC, "--quarantine-topic", QUARANTINE_TOPIC, + "--cert-root", "/certs", "--expected-canonical", str(canonical_count), + "--expected-quarantine", str(quarantine_count), "--required-products", + "BINANCE/USDM/PERPETUAL,BINANCE/SPOT/SPOT,OKX/SWAP/PERPETUAL,OKX/SPOT/SPOT,HOSE/EQUITIES/COMMON_STOCK,HNX/VN_DERIVATIVES/FUTURE"], + timeout=90.0, + ) + inspect_result = json.loads(inspected.stdout.strip().splitlines()[-1]) + if quarantine_count: + raise RuntimeError( + "real-provider records were quarantined " + f"summary={core_summary} details={inspect_result['quarantine_details']}" + ) + result = { + "schema": "qdl.phase-a.real-provider-core-certification.v1", + "status": "PASS", + "authority": "RUST_SHADOW", + "runtime_cutover": False, + "provenance": { + "crypto_trade_bbo": "REAL_PROVIDER_WEBSOCKET", + "binance_bar": bar_result["provenance"], + "okx_bar": "REAL_PROVIDER_WEBSOCKET", + "vn_equity": vn_result["provenance"], + "vn_equity_capture_boundary": vn_result["capture_boundary"], + "vn_derivative": vn_derivative_result["provenance"], + "vn_derivative_capture_boundary": vn_derivative_result["capture_boundary"], + "generated_market_data": 0, + }, + "native_runs": native_results, + "raw_acked": raw_total, + "canonical_read_committed": canonical_count, + "duplicates_suppressed": int(core_summary["duplicates"]), + "quarantines": 0, + "products": inspect_result["products"], + "canonical_digest": inspect_result["canonical_digest"], + "dnse_symbol": symbol, + "dnse_snapshot_age_seconds_at_test": round(time.time() - float(dnse_payload["timestamp"]), 3), + "rust_image_id": rust_image_id, + "v1_health_before": before_health, + } + finally: + cleanup_result = cleanup(env) + run(["docker", "image", "rm", RUST_IMAGE], check=False, timeout=120.0) + run(["docker", "image", "rm", PYTHON_IMAGE], check=False, timeout=120.0) + cleanup_result["v1_topology_unchanged"] = before_topology == v1_topology() + cleanup_result["v1_health_after"] = v1_health() + result["cleanup"] = cleanup_result + if not ( + result.get("status") == "PASS" + and cleanup_result.get("containers_after") == 0 + and cleanup_result.get("networks_after") == 0 + and cleanup_result.get("volumes_after") == 0 + and cleanup_result["v1_topology_unchanged"] + ): + raise RuntimeError(f"real-provider cleanup/invariant failed: {result}") + EVIDENCE.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n") + print(json.dumps({ + "status": result["status"], "raw": result["raw_acked"], + "canonical": result["canonical_read_committed"], + "products": result["products"], "cleanup": cleanup_result, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_seed_binance_bars.py b/scripts/phasea_seed_binance_bars.py new file mode 100755 index 0000000..1acf2fb --- /dev/null +++ b/scripts/phasea_seed_binance_bars.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +from qdl.adapters.binance import BinanceBarRawBinding, fetch_latest_closed_bar_raw_envelope +from qdl.transport.kafka_raw import KafkaRawPublisher, KafkaRawPublisherConfig + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--bootstrap", required=True) + parser.add_argument("--topic", required=True) + parser.add_argument("--cert-root", type=Path, required=True) + args = parser.parse_args() + now = time.time_ns() + envelopes = [] + rows = [] + for market, product in (("USDM", "PERPETUAL"), ("SPOT", "SPOT")): + envelope = fetch_latest_closed_bar_raw_envelope( + BinanceBarRawBinding( + market=market, + product_type=product, + native_symbol="BTCUSDT", + interval="1m", + subscription_id=f"phase-a-binance-{market.lower()}-rest-bar", + source_session_id=f"phase-a-binance-{market.lower()}-rest-{now}", + connection_generation=1, + lease_epoch=7, + authority_revision=1, + partition_plan_epoch=1, + adapter_version="binance-rest/2.0.0", + config_revision=1, + instrument_catalog_revision=3, + ), + attempts=4, + test_provenance=False, + ) + payload = json.loads(envelope.raw_frame_bytes) + rows.append({"market": market, "open_time_ms": payload["row"][0], "close_time_ms": payload["row"][6]}) + envelopes.append(envelope) + publisher = KafkaRawPublisher( + KafkaRawPublisherConfig( + bootstrap_servers=args.bootstrap, + client_id="qdl-phase-a-binance-rest-bars", + topic=args.topic, + ca_path=args.cert_root / "ca.crt", + certificate_path=args.cert_root / "client.crt", + key_path=args.cert_root / "client.key", + ) + ) + acknowledgements = publisher.publish_many(envelopes) + publisher.close() + print(json.dumps({ + "status": "PASS", "raw_acked": len(acknowledgements), + "rows": rows, "provenance": "REAL_BINANCE_REST_LATEST_CLOSED", + "production_writes": 0, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_seed_raw.py b/scripts/phasea_seed_raw.py new file mode 100755 index 0000000..b716805 --- /dev/null +++ b/scripts/phasea_seed_raw.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +from qdl.raw.capture import capture_exact_frame +from qdl.transport.kafka_raw import KafkaRawPublisher, KafkaRawPublisherConfig + + +def raw_envelope(case: dict, *, authority_revision: int = 1): + raw_bytes = json.dumps( + case["raw"], sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return capture_exact_frame( + provider=case["provider"], + venue=case["venue"], + market=case["market"], + product_type=case["product_type"], + native_symbol=case["native_symbol"], + native_channel=case["native_channel"], + subscription_id=f"phase-a-{case['native_channel']}", + source_session_id=case["source_session_id"], + connection_generation=case["generation"], + lease_epoch=7, + authority_revision=authority_revision, + partition_plan_epoch=1, + received_at_ns=case["received_at_ns"], + raw_frame_bytes=raw_bytes, + adapter_version=case["adapter_version"], + config_revision=1, + instrument_catalog_revision=3, + correlation_id=case["correlation_id"], + test_provenance=True, + ) + + +def cases(): + started = 1_786_352_400_123_456_000 + binance_10 = { + "s": "BTCUSDT", "t": 10, "p": "60000.1", "q": "0.01", "T": 3, "m": False, + } + common_binance = { + "provider": "BINANCE_DIRECT", "venue": "BINANCE", "market": "USDM", + "product_type": "PERPETUAL", "native_symbol": "BTCUSDT", + "native_channel": "trade", "source_session_id": "binance-phase-a-1", + "generation": 1, "adapter_version": "binance-json/2.0.0", + } + result = [ + {**common_binance, "raw": binance_10, "received_at_ns": started, "correlation_id": "binance-10"}, + {**common_binance, "raw": {**binance_10, "t": 11, "T": 4}, "received_at_ns": started + 1, "correlation_id": "binance-11"}, + {**common_binance, "raw": binance_10, "received_at_ns": started, "correlation_id": "binance-10-duplicate"}, + ] + result.append({ + "provider": "OKX_DIRECT", "venue": "OKX", "market": "SWAP", + "product_type": "PERPETUAL", "native_symbol": "BTC-USDT-SWAP", + "native_channel": "trades", "source_session_id": "okx-phase-a-1", + "generation": 1, "adapter_version": "okx-json/2.0.0", + "received_at_ns": started + 2, "correlation_id": "okx-trade", + "raw": {"arg": {"channel": "trades", "instId": "BTC-USDT-SWAP"}, "data": [ + {"instId": "BTC-USDT-SWAP", "tradeId": "20", "px": "60001", "sz": "2", "side": "buy", "ts": "5"} + ]}, + }) + result.append({ + "provider": "DNSE_DIRECT", "venue": "HNX", "market": "VN_DERIVATIVES", + "product_type": "FUTURE", "native_symbol": "VN30F1M", + "native_channel": "trades", "source_session_id": "dnse-phase-a-1", + "generation": 1, "adapter_version": "dnse-sdk/2.0.0", + "received_at_ns": started + 3, "correlation_id": "dnse-trade", + "raw": {"symbol": "VN30F1M", "price": "1820.7", "quantity": "12"}, + }) + result.append({ + "provider": "VNSTOCK", "venue": "HOSE", "market": "EQUITIES", + "product_type": "COMMON_STOCK", "native_symbol": "FPT", + "native_channel": "ohlcv/1m", "source_session_id": "vnstock-phase-a-1", + "generation": 1, "adapter_version": "vnstock/2.0.0", + "received_at_ns": started + 4, "correlation_id": "vnstock-bar", + "raw": {"symbol": "FPT", "interval": "1m", "open_time_ms": 1, + "close_time_ms": 2, "o": "1", "h": "2", "l": "1", "c": "2", + "v": "100", "is_final": True, "trade_count_available": False, "revision": 0}, + }) + bbo_common = { + "provider": "OKX_DIRECT", "venue": "OKX", "market": "SWAP", + "product_type": "PERPETUAL", "native_symbol": "BTC-USDT-SWAP", + "native_channel": "bbo-tbt", "source_session_id": "okx-bbo-phase-a-1", + "generation": 1, "adapter_version": "okx-json/2.0.0", + } + for index, sequence in enumerate((10, 12), start=5): + result.append({ + **bbo_common, + "received_at_ns": started + index, + "correlation_id": f"okx-bbo-{sequence}", + "raw": {"arg": {"channel": "bbo-tbt", "instId": "BTC-USDT-SWAP"}, + "data": [{"bids": [["1", "2", "0", "1"]], + "asks": [["2", "3", "0", "1"]], + "seqId": sequence, "ts": str(10 + index)}]}, + }) + return result + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--bootstrap", required=True) + parser.add_argument("--topic", required=True) + parser.add_argument("--cert-root", type=Path, required=True) + args = parser.parse_args() + publisher = KafkaRawPublisher( + KafkaRawPublisherConfig( + bootstrap_servers=args.bootstrap, + client_id="qdl-phase-a-seed", + topic=args.topic, + ca_path=args.cert_root / "ca.crt", + certificate_path=args.cert_root / "client.crt", + key_path=args.cert_root / "client.key", + ) + ) + values = tuple(raw_envelope(case) for case in cases()) + acknowledgements = publisher.publish_many(values) + publisher.close() + print(json.dumps({ + "status": "PASS", "raw_acked": len(acknowledgements), + "capture_ids": [item.capture_id.hex() for item in acknowledgements], + "production_writes": 0, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_seed_vn_derivative_history.py b/scripts/phasea_seed_vn_derivative_history.py new file mode 100755 index 0000000..697dd18 --- /dev/null +++ b/scripts/phasea_seed_vn_derivative_history.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import time +from pathlib import Path +from zoneinfo import ZoneInfo + +import pandas as pd + +from qdl.provider.v1 import raw_provider_pb2 +from qdl.raw.capture import capture_exact_frame +from qdl.transport.kafka_raw import KafkaRawPublisher, KafkaRawPublisherConfig + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--bootstrap", required=True) + parser.add_argument("--topic", required=True) + parser.add_argument("--cert-root", type=Path, required=True) + parser.add_argument("--parquet", type=Path, required=True) + args = parser.parse_args() + payload_bytes = args.parquet.read_bytes() + frame = pd.read_parquet(args.parquet) + if frame.empty: + raise RuntimeError("VN derivative canonical Parquet is empty") + row = frame.sort_values("time").iloc[-1] + local_time = pd.Timestamp(row["time"]).to_pydatetime().replace( + tzinfo=ZoneInfo("Asia/Ho_Chi_Minh") + ) + open_time_ms = int(local_time.timestamp() * 1000) + raw = { + "symbol": "VN30F1M", + "interval": "1m", + "open_time_ms": open_time_ms, + "close_time_ms": open_time_ms + 59_999, + "o": str(row["open"]), + "h": str(row["high"]), + "l": str(row["low"]), + "c": str(row["close"]), + "v": str(row["volume"]), + "is_final": True, + "trade_count_available": False, + "revision": 0, + } + raw_bytes = json.dumps(raw, sort_keys=True, separators=(",", ":")).encode() + now = time.time_ns() + envelope = capture_exact_frame( + provider="DNSE_DIRECT", + venue="HNX", + market="VN_DERIVATIVES", + product_type="FUTURE", + native_symbol="VN30F1M", + native_channel="ohlcv/1m", + subscription_id="phase-a-vn30f1m-history", + source_session_id=f"dnse-parquet-replay-{hashlib.sha256(payload_bytes).hexdigest()[:16]}", + connection_generation=1, + lease_epoch=7, + authority_revision=1, + partition_plan_epoch=1, + received_at_ns=now, + raw_frame_bytes=raw_bytes, + adapter_version="dnse-canonical-parquet-replay/2.0.0", + config_revision=1, + instrument_catalog_revision=3, + correlation_id=f"dnse:VN30F1M:1m:{open_time_ms}", + transport_protocol=raw_provider_pb2.TRANSPORT_PROTOCOL_FILE_REPLAY, + capture_boundary=raw_provider_pb2.CAPTURE_BOUNDARY_REPLAY_BYTES, + test_provenance=False, + ) + publisher = KafkaRawPublisher( + KafkaRawPublisherConfig( + bootstrap_servers=args.bootstrap, + client_id="qdl-phase-a-vn-derivative-history", + topic=args.topic, + ca_path=args.cert_root / "ca.crt", + certificate_path=args.cert_root / "client.crt", + key_path=args.cert_root / "client.key", + ) + ) + ack = publisher.publish_many((envelope,)) + publisher.close() + print(json.dumps({ + "status": "PASS", + "raw_acked": len(ack), + "symbol": "VN30F1M", + "open_time_local": local_time.isoformat(), + "parquet_sha256": hashlib.sha256(payload_bytes).hexdigest(), + "provenance": "REAL_DNSE_PROVIDER_DERIVED_CANONICAL_PARQUET_REPLAY", + "capture_boundary": "REPLAY_BYTES", + "production_writes": 0, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_seed_vn_replay.py b/scripts/phasea_seed_vn_replay.py new file mode 100755 index 0000000..94647de --- /dev/null +++ b/scripts/phasea_seed_vn_replay.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from qdl.adapters.vn import VnRawBinding, build_dnse_trade_raw_envelope +from qdl.transport.kafka_raw import KafkaRawPublisher, KafkaRawPublisherConfig + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--bootstrap", required=True) + parser.add_argument("--topic", required=True) + parser.add_argument("--cert-root", type=Path, required=True) + parser.add_argument("--payload", type=Path, required=True) + args = parser.parse_args() + payload = json.loads(args.payload.read_text()) + if payload.get("source") != "dnse": + raise RuntimeError("VN replay payload is not DNSE provider data") + symbol = str(payload["symbol"]).upper() + received_at_ns = int(float(payload["timestamp"]) * 1_000_000_000) + raw = build_dnse_trade_raw_envelope( + payload, + VnRawBinding( + venue="HOSE", + market="EQUITIES", + product_type="COMMON_STOCK", + native_symbol=symbol, + subscription_id=f"dnse-{symbol}-trades", + source_session_id=f"dnse-durable-replay-{received_at_ns}", + connection_generation=1, + lease_epoch=7, + authority_revision=1, + partition_plan_epoch=1, + adapter_version="dnse-sdk-v1-durable-replay/2.0.0", + config_revision=1, + instrument_catalog_revision=3, + ), + received_at_ns=received_at_ns, + test_provenance=False, + ) + publisher = KafkaRawPublisher( + KafkaRawPublisherConfig( + bootstrap_servers=args.bootstrap, + client_id="qdl-phase-a-vn-replay", + topic=args.topic, + ca_path=args.cert_root / "ca.crt", + certificate_path=args.cert_root / "client.crt", + key_path=args.cert_root / "client.key", + ) + ) + ack = publisher.publish_many((raw,)) + publisher.close() + print(json.dumps({ + "status": "PASS", "raw_acked": len(ack), "symbol": symbol, + "provenance": "REAL_DNSE_PROVIDER_DERIVED_DURABLE_REPLAY", + "capture_boundary": "SDK_DELIVERY", "production_writes": 0, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasea_transactional_core_certification.py b/scripts/phasea_transactional_core_certification.py new file mode 100755 index 0000000..c225eeb --- /dev/null +++ b/scripts/phasea_transactional_core_certification.py @@ -0,0 +1,290 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import shutil +import sys +import tempfile +import time + +ROOT = pathlib.Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +os.environ.setdefault("QDL_PHASE8_PROJECT", "qdl_phasea_transactional") + +from scripts.phase80_broker_certification import ( # noqa: E402 + BOOTSTRAP, + PROJECT, + add_acls, + cleanup, + compose, + create_topic, + kafka, + run, + v1_health, + v1_topology, + wait_for_cluster, + wait_for_replicas, +) + + +RUST_IMAGE = "qdl-phase-a-rust:test" +PYTHON_IMAGE = "data-layer:phase-a-test" +EVIDENCE = ROOT / "upgrade/evidence/phase-a-transactional-core.json" +RAW_TOPIC = "qdl.phase8.phasea.raw.v1" +CANONICAL_TOPIC = "qdl.phase8.phasea.canonical.v2" +QUARANTINE_TOPIC = "qdl.phase8.phasea.quarantine.v1" +GROUP_ID = "phase8-phase-a-core" + + +def core_acls(env: dict[str, str]) -> None: + common = ( + "--bootstrap-server", BOOTSTRAP, + "--command-config", "/etc/kafka/secrets/admin.properties", + "--add", "--allow-principal", "User:phase8-core", + ) + for topic, operations in ( + (RAW_TOPIC, ("READ", "DESCRIBE")), + (CANONICAL_TOPIC, ("WRITE", "DESCRIBE")), + (QUARANTINE_TOPIC, ("WRITE", "DESCRIBE")), + ): + arguments = list(common) + for operation in operations: + arguments.extend(("--operation", operation)) + arguments.extend(("--topic", topic)) + kafka(env, "kafka-acls.sh", *arguments) + kafka( + env, "kafka-acls.sh", *common, + "--operation", "READ", "--group", GROUP_ID, + ) + kafka( + env, "kafka-acls.sh", *common, + "--operation", "IdempotentWrite", "--cluster", + ) + kafka( + env, "kafka-acls.sh", *common, + "--operation", "WRITE", "--operation", "DESCRIBE", + "--transactional-id", "qdl-phase-a-", "--resource-pattern-type", "prefixed", + ) + + +def client_dir(root: pathlib.Path, principal: str) -> pathlib.Path: + target = root / principal + target.mkdir(mode=0o755) + for source, destination in ( + (root / "ca.crt", target / "ca.crt"), + (root / f"{principal}.crt", target / "client.crt"), + (root / f"{principal}.key", target / "client.key"), + ): + shutil.copyfile(source, destination) + destination.chmod(0o644) + return target + + +def authority(image_id: str) -> dict: + digest = image_id.removeprefix("sha256:") + static = hashlib.sha256(b"phase-a-multivenue").hexdigest() + return { + "schema": "qdl.authority-record.v1", + "slice_id": "phase-a-multivenue-shadow", + "revision": 1, + "mode": "RUST_SHADOW", + "candidate_image_digest": f"sha256:{digest}", + "capability_manifest_digest": hashlib.sha256( + (ROOT / "config/v2/stable-capabilities.yaml").read_bytes() + ).hexdigest(), + "contract_digest": hashlib.sha256( + (ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto").read_bytes() + ).hexdigest(), + "partition_plan_digest": static, + "public_write_allowed": False, + "legacy_write_allowed": False, + "approved_by": "phase-a-isolated-certification", + "effective_at_ns": time.time_ns(), + } + + +def bindings(): + common = {"instrument_revision": 1, "instrument_catalog_revision": 3, + "normalizer_version": "qdl-rust-core/2.0.0"} + return [ + {**common, "provider": "BINANCE_DIRECT", "venue": "BINANCE", "market": "USDM", + "product_type": "PERPETUAL", "native_symbol": "BTCUSDT", "native_channel": "trade", + "provider_kind": "binance_usdm_trade", "instrument_uid": "uid-binance-btcusdt", + "instrument_id": "BINANCE.USDM.PERPETUAL.BTC-USDT", "source_id": "binance-usdm-trade", + "source_role": "PRIMARY", "sequence_policy": "MONOTONIC"}, + {**common, "provider": "OKX_DIRECT", "venue": "OKX", "market": "SWAP", + "product_type": "PERPETUAL", "native_symbol": "BTC-USDT-SWAP", "native_channel": "trades", + "provider_kind": "okx_trade", "instrument_uid": "uid-okx-btc-swap", + "instrument_id": "OKX.SWAP.PERPETUAL.BTC-USDT", "source_id": "okx-swap-trade", + "source_role": "PRIMARY", "sequence_policy": "MONOTONIC"}, + {**common, "provider": "OKX_DIRECT", "venue": "OKX", "market": "SWAP", + "product_type": "PERPETUAL", "native_symbol": "BTC-USDT-SWAP", "native_channel": "bbo-tbt", + "provider_kind": "okx_bbo", "instrument_uid": "uid-okx-btc-swap", + "instrument_id": "OKX.SWAP.PERPETUAL.BTC-USDT", "source_id": "okx-swap-bbo", + "source_role": "PRIMARY", "sequence_policy": "CONTIGUOUS"}, + {**common, "provider": "DNSE_DIRECT", "venue": "HNX", "market": "VN_DERIVATIVES", + "product_type": "FUTURE", "native_symbol": "VN30F1M", "native_channel": "trades", + "provider_kind": "dnse_trade", "instrument_uid": "uid-hnx-vn30f1m", + "instrument_id": "HNX.VN_DERIVATIVES.FUTURE.VN30F1M", "source_id": "dnse-vn30-trade", + "source_role": "PRIMARY", "sequence_policy": "NONE"}, + {**common, "provider": "VNSTOCK", "venue": "HOSE", "market": "EQUITIES", + "product_type": "COMMON_STOCK", "native_symbol": "FPT", "native_channel": "ohlcv/1m", + "provider_kind": "vnstock_bar", "instrument_uid": "uid-hose-fpt", + "instrument_id": "HOSE.EQUITIES.COMMON_STOCK.FPT", "source_id": "vnstock-fpt-bar", + "source_role": "SECONDARY", "sequence_policy": "NONE"}, + ] + + +def docker_run(image: str, certs: pathlib.Path, command: list[str], *, entrypoint=None, mounts=()): + args = [ + "docker", "run", "--rm", "--network", f"{PROJECT}_phase8_shadow", + "--read-only", "--security-opt", "no-new-privileges:true", + "--mount", f"type=bind,source={certs},target=/certs,readonly", + "--env", f"QDL_KAFKA_BOOTSTRAP_SERVERS={BOOTSTRAP}", + "--env", "QDL_KAFKA_CERT_ROOT=/certs", + "--env", "QDL_KAFKA_CLIENT_ID=qdl-phase-a", + "--env", f"QDL_KAFKA_GROUP_ID={GROUP_ID}", + ] + for mount in mounts: + args.extend(("--mount", mount)) + if entrypoint: + args.extend(("--entrypoint", entrypoint)) + args.append(image) + args.extend(command) + return run(args, timeout=120.0) + + +def main() -> int: + before_topology = v1_topology() + before_health = v1_health() + result = {} + cleanup_result = {} + with tempfile.TemporaryDirectory(prefix="qdl-phase-a-") as directory: + temp = pathlib.Path(directory) + temp.chmod(0o755) + env = os.environ.copy() + env["QDL_PHASE8_CERT_DIR"] = str(temp) + run([str(ROOT / "scripts/phase80_generate_tls.sh"), str(temp)], env=env, timeout=120.0) + producer_certs = client_dir(temp, "phase8-producer") + consumer_certs = client_dir(temp, "phase8-consumer") + core_certs = client_dir(temp, "phase8-core") + try: + run(["docker", "build", "--provenance=false", "-f", str(ROOT / "Dockerfile.phase8-rust"), + "-t", RUST_IMAGE, str(ROOT)], timeout=1200.0) + run(["docker", "build", "--provenance=false", "-t", PYTHON_IMAGE, str(ROOT)], timeout=1200.0) + rust_image_id = run(["docker", "image", "inspect", RUST_IMAGE, "--format", "{{.Id}}"]).stdout.strip() + compose(env, "up", "-d", timeout=180.0) + wait_for_cluster(env) + for topic in (RAW_TOPIC, CANONICAL_TOPIC, QUARANTINE_TOPIC): + create_topic(env, topic, partitions=3) + add_acls(env) + core_acls(env) + wait_for_replicas(env) + + authority_record = authority(rust_image_id) + config = { + "core": { + "canonical_stream": CANONICAL_TOPIC, + "quarantine_stream": QUARANTINE_TOPIC, + "allow_test_provenance": True, + "dedup_capacity": 10000, + "bindings": bindings(), + }, + "raw_topics": [RAW_TOPIC], + "authority": authority_record, + "shard_id": "phase-a-core-shard-1", + "transactional_id": "qdl-phase-a-core-shard-1", + "batch_size": 8, + "batch_wait_ms": 100, + "max_events": 8, + "metrics_every_batches": 1, + } + config_path = temp / "core-config.json" + config_path.write_text(json.dumps(config, indent=2, sort_keys=True) + "\n") + config_path.chmod(0o644) + + seeded = docker_run( + PYTHON_IMAGE, producer_certs, + ["python", "scripts/phasea_seed_raw.py", "--bootstrap", BOOTSTRAP, + "--topic", RAW_TOPIC, "--cert-root", "/certs"], + ) + seed_result = json.loads(seeded.stdout.strip().splitlines()[-1]) + compose(env, "stop", "kafka3", timeout=60.0) + one_replica_loss_started_at = time.monotonic() + core_result = docker_run( + RUST_IMAGE, core_certs, ["/config/core-config.json"], + entrypoint="/usr/local/bin/qdl-realtime-core", + mounts=(f"type=bind,source={config_path},target=/config/core-config.json,readonly",), + ) + core_summary = json.loads(core_result.stdout.strip().splitlines()[-1]) + one_replica_loss_elapsed_seconds = round( + time.monotonic() - one_replica_loss_started_at, 6 + ) + compose(env, "start", "kafka3", timeout=90.0) + wait_for_cluster(env) + wait_for_replicas(env) + inspected = docker_run( + PYTHON_IMAGE, consumer_certs, + ["python", "scripts/phasea_inspect_kafka.py", "--bootstrap", BOOTSTRAP, + "--canonical-topic", CANONICAL_TOPIC, "--quarantine-topic", QUARANTINE_TOPIC, + "--cert-root", "/certs", "--expected-canonical", "6", + "--expected-quarantine", "1", "--required-products", + "BINANCE/USDM/PERPETUAL,OKX/SWAP/PERPETUAL,HNX/VN_DERIVATIVES/FUTURE,HOSE/EQUITIES/COMMON_STOCK"], + ) + inspect_result = json.loads(inspected.stdout.strip().splitlines()[-1]) + if seed_result.get("raw_acked") != 8: + raise RuntimeError(f"raw seed ACK count mismatch: {seed_result}") + if core_summary.get("processed") != 8 or core_summary.get("duplicates") != 1: + raise RuntimeError(f"transactional core summary mismatch: {core_summary}") + if inspect_result.get("canonical") != 6 or inspect_result.get("quarantines") != 1: + raise RuntimeError(f"read_committed output mismatch: {inspect_result}") + result = { + "schema": "qdl.phase-a.transactional-core-certification.v1", + "status": "PASS", + "authority": "RUST_SHADOW", + "runtime_cutover": False, + "raw_acked": 8, + "canonical_read_committed": 6, + "quarantine_read_committed": 1, + "duplicates_suppressed": 1, + "one_replica_loss_transaction_committed": True, + "one_replica_loss_elapsed_seconds": one_replica_loss_elapsed_seconds, + "replicas_restored_before_inspection": True, + "venues": inspect_result["venues"], + "quarantine_reasons": inspect_result["quarantine_reasons"], + "canonical_digest": inspect_result["canonical_digest"], + "rust_image_id": rust_image_id, + "python_image_id": run(["docker", "image", "inspect", PYTHON_IMAGE, "--format", "{{.Id}}"]).stdout.strip(), + "v1_health_before": before_health, + } + finally: + cleanup_result = cleanup(env) + run(["docker", "image", "rm", RUST_IMAGE], check=False, timeout=120.0) + run(["docker", "image", "rm", PYTHON_IMAGE], check=False, timeout=120.0) + after_topology = v1_topology() + after_health = v1_health() + cleanup_result["v1_topology_unchanged"] = before_topology == after_topology + cleanup_result["v1_health_after"] = after_health + result["cleanup"] = cleanup_result + if not ( + result.get("status") == "PASS" + and cleanup_result.get("containers_after") == 0 + and cleanup_result.get("networks_after") == 0 + and cleanup_result.get("volumes_after") == 0 + and cleanup_result["v1_topology_unchanged"] + ): + raise RuntimeError(f"Phase A transactional cleanup/invariant failed: {result}") + EVIDENCE.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n") + print(json.dumps({ + "status": result["status"], "canonical": result["canonical_read_committed"], + "quarantine": result["quarantine_read_committed"], + "cleanup": cleanup_result, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phaseb_bootstrap_stable_broker.py b/scripts/phaseb_bootstrap_stable_broker.py new file mode 100755 index 0000000..ce1830e --- /dev/null +++ b/scripts/phaseb_bootstrap_stable_broker.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +import subprocess +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +COMPOSE = ROOT / "docker-compose.v2-stable.yml" +BOOTSTRAP = "kafka1:9092,kafka2:9092,kafka3:9092" +ADMIN_CONFIG = "/etc/kafka/secrets/admin.properties" +TOPICS = ( + "md.raw.stable.v1", + "md.canonical.v2", + "md.quarantine.stable.v1", +) + + +def compose(env_file: Path, *arguments: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [ + "docker", "compose", "--env-file", str(env_file), + "-f", str(COMPOSE), "--profile", "stable-admin", *arguments, + ], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + timeout=120, + ) + + +def kafka(env_file: Path, executable: str, *arguments: str) -> str: + result = compose( + env_file, + "run", "--rm", "--no-deps", "--entrypoint", f"/opt/kafka/bin/{executable}", + "stable_admin", + "--bootstrap-server", BOOTSTRAP, + "--command-config", ADMIN_CONFIG, + *arguments, + ) + return result.stdout + + +def add_acl( + env_file: Path, + principal: str, + operations: tuple[str, ...], + resource_arguments: tuple[str, ...], +) -> None: + arguments = [ + "--add", "--allow-principal", f"User:{principal}", + ] + for operation in operations: + arguments.extend(("--operation", operation)) + arguments.extend(resource_arguments) + kafka(env_file, "kafka-acls.sh", *arguments) + + +def bootstrap(env_file: Path) -> dict[str, object]: + for topic in TOPICS: + kafka( + env_file, + "kafka-topics.sh", + "--create", "--if-not-exists", + "--topic", topic, + "--partitions", "6", + "--replication-factor", "3", + "--config", "min.insync.replicas=2", + "--config", "unclean.leader.election.enable=false", + "--config", "compression.type=producer", + ) + + add_acl( + env_file, "phase8-producer", ("WRITE", "DESCRIBE"), + ("--topic", TOPICS[0]), + ) + add_acl( + env_file, "phase8-producer", ("IdempotentWrite",), + ("--cluster",), + ) + for topic, operations in ( + (TOPICS[0], ("READ", "DESCRIBE")), + (TOPICS[1], ("WRITE", "DESCRIBE")), + (TOPICS[2], ("WRITE", "DESCRIBE")), + ): + add_acl(env_file, "phase8-core", operations, ("--topic", topic)) + add_acl( + env_file, "phase8-core", ("READ",), + ("--group", "qdl-v2-stable-core-v1"), + ) + add_acl(env_file, "phase8-core", ("IdempotentWrite",), ("--cluster",)) + add_acl( + env_file, "phase8-core", ("WRITE", "DESCRIBE"), + ( + "--transactional-id", "qdl-v2-stable-core-", + "--resource-pattern-type", "prefixed", + ), + ) + for topic in (TOPICS[0], TOPICS[1]): + add_acl( + env_file, "phase8-consumer", ("READ", "DESCRIBE"), + ("--topic", topic), + ) + add_acl( + env_file, "phase8-consumer", ("READ",), + ("--group", "stable-projector-v1"), + ) + + described = kafka( + env_file, "kafka-topics.sh", "--describe", + "--topic", "md.raw.stable.v1", + ) + if ( + "ReplicationFactor: 3" not in described + or "PartitionCount: 6" not in described + or "min.insync.replicas=2" not in described + ): + raise RuntimeError("stable Kafka topic policy verification failed") + return { + "schema": "qdl.v2.stable-broker-bootstrap.v1", + "status": "PASS", + "topics": list(TOPICS), + "partitions": 6, + "replication_factor": 3, + "min_insync_replicas": 2, + "tls_client_auth": "required", + "principals": ["phase8-producer", "phase8-core", "phase8-consumer"], + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--env-file", type=Path, required=True) + args = parser.parse_args() + result = bootstrap(args.env_file) + print(json.dumps(result, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phaseb_prepare_stable_candidate.py b/scripts/phaseb_prepare_stable_candidate.py new file mode 100755 index 0000000..3112846 --- /dev/null +++ b/scripts/phaseb_prepare_stable_candidate.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import secrets +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import ( + StableAcquisitionPlan, + stable_authority_record, + write_stable_runtime_bundle, +) + + +def image_id(reference: str) -> str: + completed = subprocess.run( + ["docker", "image", "inspect", reference, "--format", "{{.Id}}"], + check=True, + capture_output=True, + text=True, + timeout=30, + ) + value = completed.stdout.strip() + if not value.startswith("sha256:") or len(value) != 71: + raise RuntimeError("Docker image did not return an immutable SHA-256 ID") + return value + + +def copy_client_identity(source: Path, destination: Path, principal: str) -> None: + destination.mkdir(parents=True, exist_ok=False) + for source_name, target_name in ( + ("ca.crt", "ca.crt"), + (f"{principal}.crt", "client.crt"), + (f"{principal}.key", "client.key"), + ): + origin = source / source_name + if not origin.is_file(): + raise FileNotFoundError(f"stable TLS source is unavailable: {origin}") + shutil.copyfile(origin, destination / target_name) + for item in destination.iterdir(): + item.chmod(0o440) + + +def prepare_candidate( + *, + rust_image: str, + python_image: str, + cert_dir: Path, + output_dir: Path, + rust_image_id: str | None = None, + python_image_id: str | None = None, + host_cert_dir: Path | None = None, + host_output_dir: Path | None = None, +) -> dict[str, object]: + if output_dir.exists() and any(output_dir.iterdir()): + raise FileExistsError("stable candidate output directory must be empty") + output_dir.mkdir(parents=True, exist_ok=True) + runtime_dir = output_dir / "runtime" + identities_dir = output_dir / "identities" + identities_dir.mkdir() + rust_digest = rust_image_id or image_id(rust_image) + python_digest = python_image_id or image_id(python_image) + for value in (rust_digest, python_digest): + if not value.startswith("sha256:") or len(value) != 71: + raise ValueError("stable candidate image ID must be SHA-256") + catalog = StableSourceCatalog.load( + ROOT / "config/v2/stable-source-bindings.yaml" + ) + acquisition_path = ROOT / "config/v2/stable-acquisition-bindings.yaml" + acquisition = StableAcquisitionPlan.load(acquisition_path, catalog=catalog) + authority = stable_authority_record( + rust_image_digest=rust_digest, + capability_manifest=ROOT / "config/v2/stable-capabilities.yaml", + contract=ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto", + partition_plan=acquisition_path.read_bytes(), + effective_at_ns=__import__("time").time_ns(), + ) + bundle_digests = write_stable_runtime_bundle( + runtime_dir, + catalog=catalog, + acquisition=acquisition, + authority=authority, + ) + for role, principal in ( + ("producer", "phase8-producer"), + ("core", "phase8-core"), + ("projector", "phase8-consumer"), + ): + copy_client_identity(cert_dir, identities_dir / role, principal) + + schema_digest = hashlib.sha256( + (ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto").read_bytes() + ).hexdigest() + ingest_secret = secrets.token_urlsafe(48) + cursor_secret = secrets.token_urlsafe(48) + jwt_secret = secrets.token_urlsafe(48) + compose_cert_dir = (host_cert_dir or cert_dir).resolve() + compose_output_dir = (host_output_dir or output_dir).resolve() + values = { + "QDL_STABLE_SCHEMA_DIGEST": schema_digest, + "QDL_STABLE_INTERNAL_INGEST_SECRET": ingest_secret, + "QDL_STABLE_CURSOR_KEYS_JSON": json.dumps( + {"stable-k1": cursor_secret}, separators=(",", ":") + ), + "QDL_STABLE_JWT_KEYS_JSON": json.dumps( + {"stable-jwt-k1": jwt_secret}, separators=(",", ":") + ), + "QDL_STABLE_PYTHON_IMAGE": python_digest, + "QDL_STABLE_RUST_IMAGE": rust_digest, + "QDL_STABLE_CERT_DIR": str(compose_cert_dir), + "QDL_STABLE_PROJECTOR_CERT_DIR": str( + compose_output_dir / "identities/projector" + ), + "QDL_STABLE_CORE_CERT_DIR": str(compose_output_dir / "identities/core"), + "QDL_STABLE_PRODUCER_CERT_DIR": str( + compose_output_dir / "identities/producer" + ), + "QDL_STABLE_RUNTIME_DIR": str(compose_output_dir / "runtime"), + } + env_path = output_dir / "stable.env" + env_path.write_text( + "".join( + f"{key}='{value}'\n" if value.startswith("{") else f"{key}={value}\n" + for key, value in values.items() + ), + encoding="utf-8", + ) + env_path.chmod(0o600) + manifest = { + "schema": "qdl.v2.stable-candidate-bundle.v1", + "contract_version": "2.0.0", + "authority": "RUST_SHADOW", + "cutover_authorized": False, + "rust_image_id": rust_digest, + "python_image_id": python_digest, + "runtime_digests": bundle_digests, + "catalog_revision": catalog.catalog_revision, + "acquisition_revision": acquisition.revision, + "consumer_count": 5, + "secret_values_recorded": False, + } + manifest_path = output_dir / "candidate-manifest.json" + manifest_path.write_text( + json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + manifest_path.chmod(0o640) + return manifest + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--rust-image", required=True) + parser.add_argument("--python-image", required=True) + parser.add_argument("--cert-dir", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--rust-image-id") + parser.add_argument("--python-image-id") + parser.add_argument("--host-cert-dir", type=Path) + parser.add_argument("--host-output-dir", type=Path) + args = parser.parse_args() + manifest = prepare_candidate( + rust_image=args.rust_image, + python_image=args.python_image, + cert_dir=args.cert_dir, + output_dir=args.output_dir, + rust_image_id=args.rust_image_id, + python_image_id=args.python_image_id, + host_cert_dir=args.host_cert_dir, + host_output_dir=args.host_output_dir, + ) + print(json.dumps({ + "status": "PASS", + "contract_version": manifest["contract_version"], + "authority": manifest["authority"], + "cutover_authorized": manifest["cutover_authorized"], + "runtime_files": len(manifest["runtime_digests"]), + "secret_values_recorded": False, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/rebuild_v2_stable_projection_cache.py b/scripts/rebuild_v2_stable_projection_cache.py new file mode 100755 index 0000000..a544fa4 --- /dev/null +++ b/scripts/rebuild_v2_stable_projection_cache.py @@ -0,0 +1,357 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +import subprocess +import time +import urllib.request +from pathlib import Path +from typing import Callable, Sequence + + +ROOT = Path(__file__).resolve().parents[1] +COMPOSE_FILE = ROOT / "docker-compose.v2-stable.yml" +PROJECT_NAME = "qdl_v2_stable_candidate" +CONFIRM_TOKEN = "REBUILD_QDL_V2_STABLE_PROJECTION_CACHE" +CANONICAL_TOPIC = "md.canonical.v2" +PROJECTOR_GROUP = "stable-projector-v1" +KAFKA_BOOTSTRAP = "kafka1:9092,kafka2:9092,kafka3:9092" +KAFKA_ADMIN_CONFIG = "/etc/kafka/secrets/admin.properties" +EXPECTED_CANONICAL_PARTITIONS = 6 +MAX_ACCEPTED_LAG = 250 +REQUIRED_BOUNDED_LAG_SAMPLES = 3 +STOP_SERVICES = ( + "projector_v2", + "query_v2_1", + "query_v2_2", + "stream_v2_active", + "stream_v2_passive", +) +STREAM_SERVICES = ("stream_v2_active", "stream_v2_passive") +QUERY_SERVICES = ("query_v2_1", "query_v2_2") +CACHE_FILES = ( + "/var/lib/qdl-stable/shared/canonical-cache.sqlite3", + "/var/lib/qdl-stable/shared/canonical-cache.sqlite3-wal", + "/var/lib/qdl-stable/shared/canonical-cache.sqlite3-shm", +) +Run = Callable[..., subprocess.CompletedProcess[str]] + + +def compose_command(env_file: Path, *arguments: str) -> list[str]: + return [ + "docker", + "compose", + "--env-file", + str(env_file), + "-f", + str(COMPOSE_FILE), + "--profile", + "stable-admin", + *arguments, + ] + + +def rebuild_plan(env_file: Path) -> dict[str, object]: + return { + "schema": "qdl.v2.stable-projection-cache-rebuild.v1", + "project": PROJECT_NAME, + "env_file": str(env_file), + "authority": "Kafka canonical topic", + "stop_services": list(STOP_SERVICES), + "delete_files": list(CACHE_FILES), + "flush_service": "stable_redis", + "reset_group": PROJECTOR_GROUP, + "reset_topic": CANONICAL_TOPIC, + "lag_gate": { + "expected_partitions": EXPECTED_CANONICAL_PARTITIONS, + "max_total_records": MAX_ACCEPTED_LAG, + "consecutive_samples": REQUIRED_BOUNDED_LAG_SAMPLES, + }, + "start_order": [ + list(STREAM_SERVICES), + ["projector_v2"], + list(QUERY_SERVICES), + ], + "touches_v1": False, + "apply": False, + } + + +def require_authorization(*, apply: bool, confirm: str | None) -> None: + if not apply: + return + if confirm != CONFIRM_TOKEN: + raise ValueError(f"--confirm must equal {CONFIRM_TOKEN}") + + +def parse_canonical_lag(output: str) -> tuple[int, int]: + lags: list[int] = [] + for line in output.splitlines(): + fields = line.split() + if len(fields) < 6 or CANONICAL_TOPIC not in fields: + continue + topic_index = fields.index(CANONICAL_TOPIC) + if topic_index + 4 >= len(fields): + continue + partition = fields[topic_index + 1] + lag = fields[topic_index + 4] + if partition.isdigit() and lag.lstrip("-").isdigit(): + lags.append(max(0, int(lag))) + if not lags: + raise RuntimeError("canonical projector lag output has no partitions") + return sum(lags), len(lags) + + +def _run( + command: Sequence[str], + *, + timeout: float, + check: bool = True, +) -> subprocess.CompletedProcess[str]: + return subprocess.run( + list(command), + cwd=ROOT, + check=check, + capture_output=True, + text=True, + timeout=timeout, + ) + + +def _compose( + env_file: Path, + *arguments: str, + timeout: float = 120, + check: bool = True, +) -> subprocess.CompletedProcess[str]: + return _run( + compose_command(env_file, *arguments), + timeout=timeout, + check=check, + ) + + +def _start_services(env_file: Path, *services: str) -> None: + if not services: + raise ValueError("at least one stable service is required") + _compose(env_file, "up", "-d", "--no-deps", *services) + + +def _kafka_group(env_file: Path, *arguments: str) -> str: + result = _compose( + env_file, + "run", + "--rm", + "--no-deps", + "--entrypoint", + "/opt/kafka/bin/kafka-consumer-groups.sh", + "stable_admin", + "--bootstrap-server", + KAFKA_BOOTSTRAP, + "--command-config", + KAFKA_ADMIN_CONFIG, + *arguments, + timeout=120, + ) + return result.stdout + + +def _validate_project(env_file: Path) -> None: + if not env_file.is_file(): + raise FileNotFoundError(f"stable env file does not exist: {env_file}") + result = _compose(env_file, "config", "--format", "json") + document = json.loads(result.stdout) + if document.get("name") != PROJECT_NAME: + raise RuntimeError("compose project is not the isolated stable candidate") + + +def _wait_http(url: str, deadline: float) -> None: + last_error: BaseException | None = None + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(url, timeout=2) as response: + if response.status == 200: + return + except BaseException as error: + last_error = error + time.sleep(1) + raise TimeoutError(f"health endpoint did not become ready: {url}") from last_error + + +def _wait_projector_ready(env_file: Path, deadline: float) -> None: + probe = ( + "import urllib.request;" + "urllib.request.urlopen('http://127.0.0.1:8230/health/ready',timeout=2)" + ) + while time.monotonic() < deadline: + result = _compose( + env_file, + "exec", + "-T", + "projector_v2", + "python", + "-c", + probe, + timeout=15, + check=False, + ) + if result.returncode == 0: + return + time.sleep(2) + raise TimeoutError("stable projector did not become ready") + + +def lag_sample_acceptable(total_lag: int, partitions: int) -> bool: + return ( + partitions == EXPECTED_CANONICAL_PARTITIONS + and 0 <= total_lag <= MAX_ACCEPTED_LAG + ) + + +def _wait_bounded_lag(env_file: Path, deadline: float) -> dict[str, int]: + last_lag: int | None = None + partitions = 0 + consecutive = 0 + observed_bound = 0 + while time.monotonic() < deadline: + output = _kafka_group( + env_file, + "--group", + PROJECTOR_GROUP, + "--describe", + ) + last_lag, partitions = parse_canonical_lag(output) + if lag_sample_acceptable(last_lag, partitions): + consecutive += 1 + observed_bound = max(observed_bound, last_lag) + if consecutive >= REQUIRED_BOUNDED_LAG_SAMPLES: + return { + "lag": last_lag, + "partitions": partitions, + "observed_bound": observed_bound, + "configured_bound": MAX_ACCEPTED_LAG, + "consecutive_samples": consecutive, + } + else: + consecutive = 0 + observed_bound = 0 + time.sleep(2) + raise TimeoutError( + "stable projector did not enter its bounded live-lag window; " + f"last_lag={last_lag} partitions={partitions}" + ) + + +def execute_rebuild(env_file: Path, *, timeout_seconds: float) -> dict[str, object]: + _validate_project(env_file) + deadline = time.monotonic() + timeout_seconds + _compose(env_file, "stop", *STOP_SERVICES) + + running = set( + _compose( + env_file, "ps", "--services", "--status", "running" + ).stdout.split() + ) + unexpected = running.intersection(STOP_SERVICES) + if unexpected: + raise RuntimeError(f"cache users still running: {sorted(unexpected)}") + + cleanup = "rm -f " + " ".join(CACHE_FILES) + _compose( + env_file, + "run", + "--rm", + "--no-deps", + "--entrypoint", + "/bin/sh", + "stable_state_init", + "-ec", + cleanup, + ) + _compose( + env_file, + "exec", + "-T", + "stable_redis", + "redis-cli", + "FLUSHDB", + ) + dbsize = _compose( + env_file, + "exec", + "-T", + "stable_redis", + "redis-cli", + "DBSIZE", + ).stdout.strip() + if dbsize != "0": + raise RuntimeError("isolated stable Redis did not reset to zero keys") + + _kafka_group( + env_file, + "--group", + PROJECTOR_GROUP, + "--topic", + CANONICAL_TOPIC, + "--reset-offsets", + "--to-earliest", + "--execute", + ) + _start_services(env_file, *STREAM_SERVICES) + _wait_http("http://127.0.0.1:18210/health/live", deadline) + _wait_http("http://127.0.0.1:18211/health/live", deadline) + + _start_services(env_file, "projector_v2") + lag = _wait_bounded_lag(env_file, deadline) + _wait_projector_ready(env_file, deadline) + + _start_services(env_file, *QUERY_SERVICES) + _wait_http("http://127.0.0.1:18201/health/ready", deadline) + _wait_http("http://127.0.0.1:18202/health/ready", deadline) + + final_size = int( + _compose( + env_file, + "exec", + "-T", + "stable_redis", + "redis-cli", + "DBSIZE", + ).stdout.strip() + ) + if final_size <= 0: + raise RuntimeError("stable Redis projection remained empty after replay") + return { + **rebuild_plan(env_file), + "apply": True, + "status": "PASS", + "canonical_lag": lag, + "redis_keys": final_size, + } + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Atomically rebuild the isolated V2 stable projection cache." + ) + parser.add_argument("--env-file", type=Path, required=True) + parser.add_argument("--apply", action="store_true") + parser.add_argument("--confirm") + parser.add_argument("--timeout-seconds", type=float, default=900) + args = parser.parse_args() + if args.timeout_seconds <= 0: + parser.error("--timeout-seconds must be positive") + require_authorization(apply=args.apply, confirm=args.confirm) + if not args.apply: + print(json.dumps(rebuild_plan(args.env_file), indent=2, sort_keys=True)) + return 0 + result = execute_rebuild( + args.env_file.resolve(), timeout_seconds=args.timeout_seconds + ) + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/fixtures/phase2/binance_spot_bar.json b/tests/fixtures/phase2/binance_spot_bar.json new file mode 100644 index 0000000..d72a94d --- /dev/null +++ b/tests/fixtures/phase2/binance_spot_bar.json @@ -0,0 +1,43 @@ +{ + "context": { + "adapter_version": "binance-json/1.0.0", + "config_revision": 9, + "correlation_id": "phase-a-binance_spot_bar", + "instrument_id": "BINANCE.SPOT.SPOT.BTC-USDT", + "instrument_revision": 7, + "instrument_uid": "26edfffd-6824-5e75-a620-5a122b3e1086", + "lease_epoch": 42, + "market": "SPOT", + "native_symbol": "BTCUSDT", + "normalized_at_ns": 1786352400123500000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 1234, + "product_type": "SPOT", + "provider": "BINANCE_DIRECT", + "published_at_ns": 1786352400123700000, + "received_at_ns": 1786352400123456000, + "source_id": "binance-spot-bar-001", + "venue": "BINANCE" + }, + "provider_kind": "binance_spot_bar", + "raw": { + "E": 1786352400123, + "e": "kline", + "k": { + "L": 110, + "T": 1786352399999, + "c": "61234.10", + "f": 100, + "h": "61240.00", + "i": "1m", + "l": "61190.00", + "n": 11, + "o": "61200.00", + "s": "BTCUSDT", + "t": 1786352340000, + "v": "12.500", + "x": true + }, + "s": "BTCUSDT" + } +} diff --git a/tests/fixtures/phase2/binance_spot_bbo.json b/tests/fixtures/phase2/binance_spot_bbo.json new file mode 100644 index 0000000..980b112 --- /dev/null +++ b/tests/fixtures/phase2/binance_spot_bbo.json @@ -0,0 +1,32 @@ +{ + "context": { + "adapter_version": "binance-json/1.0.0", + "config_revision": 9, + "correlation_id": "phase-a-binance_spot_bbo", + "instrument_id": "BINANCE.SPOT.SPOT.BTC-USDT", + "instrument_revision": 7, + "instrument_uid": "26edfffd-6824-5e75-a620-5a122b3e1086", + "lease_epoch": 42, + "market": "SPOT", + "native_symbol": "BTCUSDT", + "normalized_at_ns": 1786352400123500000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 1234, + "product_type": "SPOT", + "provider": "BINANCE_DIRECT", + "published_at_ns": 1786352400123700000, + "received_at_ns": 1786352400123456000, + "source_id": "binance-spot-bbo-001", + "venue": "BINANCE" + }, + "provider_kind": "binance_spot_bbo", + "raw": { + "A": "0.500", + "B": "1.250", + "a": "61234.20", + "b": "61234.10", + "e": "bookTicker", + "s": "BTCUSDT", + "u": 400900217 + } +} diff --git a/tests/fixtures/phase2/binance_spot_rest_bar.json b/tests/fixtures/phase2/binance_spot_rest_bar.json new file mode 100644 index 0000000..f9ee2e7 --- /dev/null +++ b/tests/fixtures/phase2/binance_spot_rest_bar.json @@ -0,0 +1,42 @@ +{ + "context": { + "adapter_version": "binance-rest/2.0.0", + "config_revision": 9, + "correlation_id": "phase-a-binance_spot_rest_bar", + "instrument_id": "BINANCE.SPOT.SPOT.BTC-USDT", + "instrument_revision": 7, + "instrument_uid": "26edfffd-6824-5e75-a620-5a122b3e1086", + "lease_epoch": 42, + "market": "SPOT", + "native_symbol": "BTCUSDT", + "normalized_at_ns": 1786352400123500000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 1234, + "product_type": "SPOT", + "provider": "BINANCE_DIRECT", + "published_at_ns": 1786352400123700000, + "received_at_ns": 1786352400123456000, + "source_id": "binance-spot-bar-001", + "venue": "BINANCE" + }, + "provider_kind": "binance_spot_rest_bar", + "raw": { + "bar_origin": "VENUE_NATIVE", + "interval": "1m", + "row": [ + 1786352340000, + "61200.00", + "61240.00", + "61190.00", + "61234.10", + "12.500", + 1786352399999, + "765200.00", + 11, + "0", + "0", + "0" + ], + "symbol": "BTCUSDT" + } +} diff --git a/tests/fixtures/phase2/binance_spot_trade.json b/tests/fixtures/phase2/binance_spot_trade.json new file mode 100644 index 0000000..dcc86e7 --- /dev/null +++ b/tests/fixtures/phase2/binance_spot_trade.json @@ -0,0 +1,35 @@ +{ + "context": { + "adapter_version": "binance-json/1.0.0", + "config_revision": 9, + "correlation_id": "phase-a-binance_spot_trade", + "instrument_id": "BINANCE.SPOT.SPOT.BTC-USDT", + "instrument_revision": 7, + "instrument_uid": "26edfffd-6824-5e75-a620-5a122b3e1086", + "lease_epoch": 42, + "market": "SPOT", + "native_symbol": "BTCUSDT", + "normalized_at_ns": 1786352400123500000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 1234, + "product_type": "SPOT", + "provider": "BINANCE_DIRECT", + "published_at_ns": 1786352400123700000, + "received_at_ns": 1786352400123456000, + "source_id": "binance-spot-trade-001", + "venue": "BINANCE" + }, + "provider_kind": "binance_spot_trade", + "raw": { + "E": 1786352400124, + "T": 1786352400123, + "X": "MARKET", + "a": 9876543210123456789, + "e": "aggTrade", + "m": false, + "p": "61234.10", + "q": "0.125", + "s": "BTCUSDT", + "st": 1786352400122 + } +} diff --git a/tests/fixtures/phase2/binance_usdm_rest_bar.json b/tests/fixtures/phase2/binance_usdm_rest_bar.json new file mode 100644 index 0000000..b17c9f0 --- /dev/null +++ b/tests/fixtures/phase2/binance_usdm_rest_bar.json @@ -0,0 +1,42 @@ +{ + "context": { + "adapter_version": "binance-rest/2.0.0", + "config_revision": 9, + "correlation_id": "phase-a-binance_usdm_rest_bar", + "instrument_id": "BINANCE.USDM.PERPETUAL.BTCUSDT", + "instrument_revision": 7, + "instrument_uid": "85ad7cb6-7ebf-5c81-9d82-12c4c10ca85c", + "lease_epoch": 42, + "market": "USDM", + "native_symbol": "BTCUSDT", + "normalized_at_ns": 1786352400123500000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 1234, + "product_type": "PERPETUAL", + "provider": "BINANCE_DIRECT", + "published_at_ns": 1786352400123700000, + "received_at_ns": 1786352400123456000, + "source_id": "binance-usdm-rest-bar-001", + "venue": "BINANCE" + }, + "provider_kind": "binance_usdm_rest_bar", + "raw": { + "bar_origin": "VENUE_NATIVE", + "interval": "1m", + "row": [ + 1786352340000, + "61200.00", + "61240.00", + "61190.00", + "61234.10", + "12.500", + 1786352399999, + "765200.00", + 11, + "0", + "0", + "0" + ], + "symbol": "BTCUSDT" + } +} diff --git a/tests/fixtures/phase2/dnse_derivative_bar.json b/tests/fixtures/phase2/dnse_derivative_bar.json new file mode 100644 index 0000000..94907ec --- /dev/null +++ b/tests/fixtures/phase2/dnse_derivative_bar.json @@ -0,0 +1,94 @@ +{ + "context": { + "adapter_version": "dnse_direct/2.0.0", + "authority_revision": 1, + "config_revision": 19, + "connection_generation": 1, + "correlation_id": "phase-a-dnse-derivative-bar-001", + "instrument_id": "HNX.VN_DERIVATIVES.FUTURE.VN30F1M", + "instrument_revision": 1, + "instrument_uid": "d0ac3d4c-2374-5690-8af5-b970448f91d3", + "lease_epoch": 9, + "market": "VN_DERIVATIVES", + "native_symbol": "VN30F1M", + "normalized_at_ns": 1786352400123466000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_plan_epoch": 1, + "partition_sequence": 1, + "product_type": "FUTURE", + "provider": "DNSE_DIRECT", + "published_at_ns": 1786352400123476000, + "raw_capture_id": [ + 242, + 11, + 162, + 246, + 157, + 123, + 59, + 190, + 36, + 243, + 10, + 180, + 212, + 115, + 165, + 133 + ], + "raw_frame_sha256": [ + 52, + 50, + 139, + 48, + 99, + 42, + 150, + 19, + 54, + 59, + 7, + 24, + 150, + 222, + 189, + 99, + 22, + 128, + 96, + 109, + 81, + 235, + 132, + 233, + 120, + 166, + 54, + 166, + 86, + 203, + 65, + 236 + ], + "received_at_ns": 1786352400123456000, + "source_id": "dnse-derivative-bar-001", + "source_role": "PRIMARY", + "source_session_id": "dnse-derivative-bar-001-session-1", + "venue": "HNX" + }, + "provider_kind": "dnse_bar", + "raw": { + "c": "1820.7", + "close_time_ms": 1786352399999, + "h": "1821.2", + "interval": "1m", + "is_final": true, + "l": "1820.2", + "o": "1820.7", + "open_time_ms": 1786352340000, + "revision": 0, + "symbol": "VN30F1M", + "trade_count_available": false, + "v": "12" + } +} diff --git a/tests/fixtures/phase2/dnse_derivative_trade.json b/tests/fixtures/phase2/dnse_derivative_trade.json new file mode 100644 index 0000000..b0d6692 --- /dev/null +++ b/tests/fixtures/phase2/dnse_derivative_trade.json @@ -0,0 +1,88 @@ +{ + "context": { + "adapter_version": "dnse_direct/2.0.0", + "authority_revision": 1, + "config_revision": 19, + "connection_generation": 1, + "correlation_id": "phase-a-dnse-derivative-trade-001", + "instrument_id": "HNX.VN_DERIVATIVES.FUTURE.VN30F1M", + "instrument_revision": 1, + "instrument_uid": "d0ac3d4c-2374-5690-8af5-b970448f91d3", + "lease_epoch": 9, + "market": "VN_DERIVATIVES", + "native_symbol": "VN30F1M", + "normalized_at_ns": 1786352400123466000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_plan_epoch": 1, + "partition_sequence": 1, + "product_type": "FUTURE", + "provider": "DNSE_DIRECT", + "published_at_ns": 1786352400123476000, + "raw_capture_id": [ + 245, + 194, + 205, + 27, + 114, + 96, + 53, + 107, + 110, + 27, + 159, + 103, + 24, + 232, + 38, + 194 + ], + "raw_frame_sha256": [ + 183, + 166, + 51, + 65, + 143, + 78, + 49, + 101, + 247, + 202, + 178, + 119, + 30, + 159, + 142, + 140, + 23, + 113, + 89, + 131, + 66, + 84, + 168, + 238, + 165, + 75, + 53, + 50, + 222, + 210, + 211, + 46 + ], + "received_at_ns": 1786352400123456000, + "source_id": "dnse-derivative-trade-001", + "source_role": "PRIMARY", + "source_session_id": "dnse-derivative-trade-001-session-1", + "venue": "HNX" + }, + "provider_kind": "dnse_trade", + "raw": { + "market_id": "G3", + "price": "1820.7", + "quantity": "12", + "symbol": "VN30F1M", + "total_volume_traded": "12000", + "trading_session_id": "CONTINUOUS" + } +} diff --git a/tests/fixtures/phase2/dnse_equity_bar.json b/tests/fixtures/phase2/dnse_equity_bar.json new file mode 100644 index 0000000..3d9c3ce --- /dev/null +++ b/tests/fixtures/phase2/dnse_equity_bar.json @@ -0,0 +1,94 @@ +{ + "context": { + "adapter_version": "dnse_direct/2.0.0", + "authority_revision": 1, + "config_revision": 19, + "connection_generation": 1, + "correlation_id": "phase-a-dnse-equity-bar-001", + "instrument_id": "HOSE.EQUITIES.COMMON_STOCK.FPT", + "instrument_revision": 1, + "instrument_uid": "6f025ab7-c0b4-5e08-866d-34ce62aef52f", + "lease_epoch": 9, + "market": "EQUITIES", + "native_symbol": "FPT", + "normalized_at_ns": 1786352400123466000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_plan_epoch": 1, + "partition_sequence": 1, + "product_type": "COMMON_STOCK", + "provider": "DNSE_DIRECT", + "published_at_ns": 1786352400123476000, + "raw_capture_id": [ + 55, + 105, + 102, + 215, + 33, + 53, + 130, + 14, + 107, + 191, + 62, + 65, + 48, + 222, + 34, + 4 + ], + "raw_frame_sha256": [ + 52, + 4, + 120, + 208, + 140, + 36, + 102, + 132, + 10, + 240, + 184, + 132, + 103, + 45, + 99, + 221, + 230, + 21, + 40, + 85, + 73, + 185, + 165, + 20, + 5, + 130, + 149, + 144, + 102, + 133, + 34, + 156 + ], + "received_at_ns": 1786352400123456000, + "source_id": "dnse-equity-bar-001", + "source_role": "PRIMARY", + "source_session_id": "dnse-equity-bar-001-session-1", + "venue": "HOSE" + }, + "provider_kind": "dnse_bar", + "raw": { + "c": "125000", + "close_time_ms": 1786352399999, + "h": "125100.0", + "interval": "1m", + "is_final": true, + "l": "124900.0", + "o": "125000", + "open_time_ms": 1786352340000, + "revision": 0, + "symbol": "FPT", + "trade_count_available": false, + "v": "100" + } +} diff --git a/tests/fixtures/phase2/dnse_equity_trade.json b/tests/fixtures/phase2/dnse_equity_trade.json new file mode 100644 index 0000000..5ac8cdb --- /dev/null +++ b/tests/fixtures/phase2/dnse_equity_trade.json @@ -0,0 +1,88 @@ +{ + "context": { + "adapter_version": "dnse_direct/2.0.0", + "authority_revision": 1, + "config_revision": 19, + "connection_generation": 1, + "correlation_id": "phase-a-dnse-equity-trade-001", + "instrument_id": "HOSE.EQUITIES.COMMON_STOCK.FPT", + "instrument_revision": 1, + "instrument_uid": "6f025ab7-c0b4-5e08-866d-34ce62aef52f", + "lease_epoch": 9, + "market": "EQUITIES", + "native_symbol": "FPT", + "normalized_at_ns": 1786352400123466000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_plan_epoch": 1, + "partition_sequence": 1, + "product_type": "COMMON_STOCK", + "provider": "DNSE_DIRECT", + "published_at_ns": 1786352400123476000, + "raw_capture_id": [ + 107, + 204, + 68, + 80, + 168, + 42, + 93, + 119, + 85, + 103, + 153, + 120, + 51, + 29, + 61, + 202 + ], + "raw_frame_sha256": [ + 54, + 108, + 49, + 208, + 254, + 243, + 67, + 71, + 32, + 34, + 20, + 61, + 57, + 197, + 50, + 16, + 223, + 187, + 101, + 249, + 197, + 39, + 141, + 229, + 60, + 79, + 104, + 254, + 183, + 40, + 198, + 224 + ], + "received_at_ns": 1786352400123456000, + "source_id": "dnse-equity-trade-001", + "source_role": "PRIMARY", + "source_session_id": "dnse-equity-trade-001-session-1", + "venue": "HOSE" + }, + "provider_kind": "dnse_trade", + "raw": { + "market_id": "G1", + "price": "125000", + "quantity": "100", + "symbol": "FPT", + "total_volume_traded": "12000", + "trading_session_id": "CONTINUOUS" + } +} diff --git a/tests/fixtures/phase2/okx_bar.json b/tests/fixtures/phase2/okx_bar.json new file mode 100644 index 0000000..4822a6c --- /dev/null +++ b/tests/fixtures/phase2/okx_bar.json @@ -0,0 +1,42 @@ +{ + "context": { + "adapter_version": "okx-json/2.0.0", + "config_revision": 12, + "correlation_id": "phase19-okx-bar", + "instrument_id": "OKX.SWAP.PERPETUAL.BTC-USDT", + "instrument_revision": 2, + "instrument_uid": "23533297-11a8-5290-a800-c7f584706cd6", + "lease_epoch": 7, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "normalized_at_ns": 1786352400124100000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 90, + "product_type": "PERPETUAL", + "provider": "OKX_DIRECT", + "published_at_ns": 1786352400124200000, + "received_at_ns": 1786352400124000000, + "source_id": "okx-business-candle-001", + "venue": "OKX" + }, + "provider_kind": "okx_bar", + "raw": { + "arg": { + "channel": "candle1m", + "instId": "BTC-USDT-SWAP" + }, + "data": [ + [ + "1786352340000", + "61200.00", + "61240.00", + "61190.00", + "61234.10", + "12.500", + "12.500", + "765200.00", + "1" + ] + ] + } +} diff --git a/tests/fixtures/phase2/okx_bbo.json b/tests/fixtures/phase2/okx_bbo.json new file mode 100644 index 0000000..b98b047 --- /dev/null +++ b/tests/fixtures/phase2/okx_bbo.json @@ -0,0 +1,52 @@ +{ + "context": { + "adapter_version": "okx-json/2.0.0", + "config_revision": 12, + "correlation_id": "phase19-okx-bbo", + "instrument_id": "OKX.SWAP.PERPETUAL.BTC-USDT", + "instrument_revision": 2, + "instrument_uid": "23533297-11a8-5290-a800-c7f584706cd6", + "lease_epoch": 7, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "normalized_at_ns": 1786352400124100000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 89, + "product_type": "PERPETUAL", + "provider": "OKX_DIRECT", + "published_at_ns": 1786352400124200000, + "received_at_ns": 1786352400124000000, + "source_id": "okx-public-bbo-001", + "venue": "OKX" + }, + "provider_kind": "okx_bbo", + "raw": { + "action": "snapshot", + "arg": { + "channel": "bbo-tbt", + "instId": "BTC-USDT-SWAP" + }, + "data": [ + { + "asks": [ + [ + "61234.20", + "0.500", + "0", + "2" + ] + ], + "bids": [ + [ + "61234.10", + "1.250", + "0", + "3" + ] + ], + "seqId": 817263, + "ts": "1786352400125" + } + ] + } +} diff --git a/tests/fixtures/phase2/okx_spot_bar.json b/tests/fixtures/phase2/okx_spot_bar.json new file mode 100644 index 0000000..2448b1c --- /dev/null +++ b/tests/fixtures/phase2/okx_spot_bar.json @@ -0,0 +1,42 @@ +{ + "context": { + "adapter_version": "okx-json/2.0.0", + "config_revision": 12, + "correlation_id": "phase19-okx-spot-bar", + "instrument_id": "OKX.SPOT.SPOT.BTC-USDT", + "instrument_revision": 2, + "instrument_uid": "9e996914-aa98-5ea0-b0a0-a78dc51e1f8d", + "lease_epoch": 7, + "market": "SPOT", + "native_symbol": "BTC-USDT", + "normalized_at_ns": 1786352400124100000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 90, + "product_type": "SPOT", + "provider": "OKX_DIRECT", + "published_at_ns": 1786352400124200000, + "received_at_ns": 1786352400124000000, + "source_id": "okx-business-candle-001", + "venue": "OKX" + }, + "provider_kind": "okx_bar", + "raw": { + "arg": { + "channel": "candle1m", + "instId": "BTC-USDT" + }, + "data": [ + [ + "1786352340000", + "61200.00", + "61240.00", + "61190.00", + "61234.10", + "12.500", + "12.500", + "765200.00", + "1" + ] + ] + } +} diff --git a/tests/fixtures/phase2/okx_spot_bbo.json b/tests/fixtures/phase2/okx_spot_bbo.json new file mode 100644 index 0000000..8edb59f --- /dev/null +++ b/tests/fixtures/phase2/okx_spot_bbo.json @@ -0,0 +1,52 @@ +{ + "context": { + "adapter_version": "okx-json/2.0.0", + "config_revision": 12, + "correlation_id": "phase19-okx-spot-bbo", + "instrument_id": "OKX.SPOT.SPOT.BTC-USDT", + "instrument_revision": 2, + "instrument_uid": "9e996914-aa98-5ea0-b0a0-a78dc51e1f8d", + "lease_epoch": 7, + "market": "SPOT", + "native_symbol": "BTC-USDT", + "normalized_at_ns": 1786352400124100000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 89, + "product_type": "SPOT", + "provider": "OKX_DIRECT", + "published_at_ns": 1786352400124200000, + "received_at_ns": 1786352400124000000, + "source_id": "okx-public-bbo-001", + "venue": "OKX" + }, + "provider_kind": "okx_bbo", + "raw": { + "action": "snapshot", + "arg": { + "channel": "bbo-tbt", + "instId": "BTC-USDT" + }, + "data": [ + { + "asks": [ + [ + "61234.20", + "0.500", + "0", + "2" + ] + ], + "bids": [ + [ + "61234.10", + "1.250", + "0", + "3" + ] + ], + "seqId": 817263, + "ts": "1786352400125" + } + ] + } +} diff --git a/tests/fixtures/phase2/okx_spot_trade.json b/tests/fixtures/phase2/okx_spot_trade.json new file mode 100644 index 0000000..e963b78 --- /dev/null +++ b/tests/fixtures/phase2/okx_spot_trade.json @@ -0,0 +1,31 @@ +{ + "context": { + "adapter_version": "okx-json/1.0.0", + "config_revision": 11, + "correlation_id": "phase2-okx-spot-trade", + "instrument_id": "OKX.SPOT.SPOT.BTC-USDT", + "instrument_revision": 2, + "instrument_uid": "9e996914-aa98-5ea0-b0a0-a78dc51e1f8d", + "lease_epoch": 7, + "market": "SPOT", + "native_symbol": "BTC-USDT", + "normalized_at_ns": 1786352400124100000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_sequence": 88, + "product_type": "SPOT", + "provider": "OKX_DIRECT", + "published_at_ns": 1786352400124200000, + "received_at_ns": 1786352400124000000, + "source_id": "okx-public-trades-001", + "venue": "OKX" + }, + "provider_kind": "okx_trade", + "raw": { + "instId": "BTC-USDT", + "px": "61235.20", + "side": "sell", + "sz": "1.50", + "tradeId": "998877665544332211", + "ts": "1786352400125" + } +} diff --git a/tests/fixtures/phase2/vnstock_equity_bar.json b/tests/fixtures/phase2/vnstock_equity_bar.json new file mode 100644 index 0000000..5ac5dd6 --- /dev/null +++ b/tests/fixtures/phase2/vnstock_equity_bar.json @@ -0,0 +1,94 @@ +{ + "context": { + "adapter_version": "vnstock/2.0.0", + "authority_revision": 1, + "config_revision": 19, + "connection_generation": 1, + "correlation_id": "phase-a-vnstock-equity-bar-001", + "instrument_id": "HOSE.EQUITIES.COMMON_STOCK.FPT", + "instrument_revision": 1, + "instrument_uid": "6f025ab7-c0b4-5e08-866d-34ce62aef52f", + "lease_epoch": 9, + "market": "EQUITIES", + "native_symbol": "FPT", + "normalized_at_ns": 1786352400123466000, + "normalizer_version": "qdl-normalizer/2.0.0", + "partition_plan_epoch": 1, + "partition_sequence": 1, + "product_type": "COMMON_STOCK", + "provider": "VNSTOCK", + "published_at_ns": 1786352400123476000, + "raw_capture_id": [ + 198, + 109, + 167, + 232, + 109, + 44, + 9, + 142, + 123, + 58, + 103, + 231, + 142, + 162, + 48, + 41 + ], + "raw_frame_sha256": [ + 123, + 71, + 36, + 186, + 196, + 5, + 68, + 18, + 188, + 237, + 105, + 141, + 122, + 198, + 71, + 179, + 32, + 0, + 35, + 77, + 113, + 250, + 2, + 80, + 223, + 109, + 36, + 220, + 135, + 99, + 87, + 178 + ], + "received_at_ns": 1786352400123456000, + "source_id": "vnstock-equity-bar-001", + "source_role": "SECONDARY", + "source_session_id": "vnstock-equity-bar-001-session-1", + "venue": "HOSE" + }, + "provider_kind": "vnstock_bar", + "raw": { + "c": "125000", + "close_time_ms": 1786352399999, + "h": "125100", + "interval": "1m", + "is_final": true, + "l": "124900", + "o": "125000", + "open_time_ms": 1786352340000, + "revision": 0, + "symbol": "FPT", + "trade_count_available": false, + "v": "100" + } +} diff --git a/tests/test_dnse_provider_history.py b/tests/test_dnse_provider_history.py new file mode 100644 index 0000000..277fd7c --- /dev/null +++ b/tests/test_dnse_provider_history.py @@ -0,0 +1,229 @@ +from __future__ import annotations + +import threading +import unittest +from unittest.mock import patch + +from app.database import dnse_fallback +from app.openapi_sdk.python.websocket_marketdata.trading_websocket.client import TradingClient + +import requests + +from app.providers.dnse.history import ( + DnseHistoryClient, + DnseHistoryConfig, + DnseHistoryError, + DnseQuotaLimiter, +) + + +class _Response: + def __init__(self, status=200, payload=None, headers=None, content=None): + self.status_code = status + self._payload = payload + self.headers = headers or {} + self.content = content if content is not None else b"{}" + + def json(self): + if isinstance(self._payload, Exception): + raise self._payload + return self._payload + + +class _Session: + def __init__(self, outcomes): + self.outcomes = list(outcomes) + self.calls = [] + self.trust_env = None + + def get(self, url, **kwargs): + self.calls.append((url, kwargs)) + outcome = self.outcomes.pop(0) + if isinstance(outcome, Exception): + raise outcome + return outcome + + +class _Limiter: + def __init__(self): + self.calls = 0 + + def acquire(self): + self.calls += 1 + + +def _page(times=(100, 160), *, next_time=0, close=None): + close = close or ["100.5"] * len(times) + return { + "t": list(times), + "o": ["100"] * len(times), + "h": ["101"] * len(times), + "l": ["99"] * len(times), + "c": close, + "v": ["0"] * len(times), + "nextTime": next_time, + } + + +class DnseHistoryClientTests(unittest.TestCase): + def config(self, **overrides): + values = { + "api_key": "key-not-secret", + "api_secret": "secret-not-logged", + "attempts": 3, + "max_backoff_seconds": 5, + } + values.update(overrides) + return DnseHistoryConfig(**values) + + def client(self, outcomes, **config): + session = _Session(outcomes) + limiter = _Limiter() + sleeps = [] + client = DnseHistoryClient( + self.config(**config), + session=session, + limiter=limiter, + sleep=sleeps.append, + random_uniform=lambda _low, _high: 0, + ) + return client, session, limiter, sleeps + + def test_version_signature_tls_and_explicit_proxy_policy(self): + client, session, limiter, _sleeps = self.client([_Response(payload=_page())]) + rows = client.fetch_ohlc("vn30f1m", "1", 100, 220) + self.assertEqual([row["t"] for row in rows], [100, 160]) + self.assertFalse(session.trust_env) + self.assertEqual(limiter.calls, 1) + url, request = session.calls[0] + self.assertEqual(url, "https://openapi.dnse.com.vn/price/ohlc") + self.assertTrue(request["verify"]) + self.assertEqual(request["headers"]["version"], "2026-07-23") + self.assertEqual(request["headers"]["X-API-Key"], "key-not-secret") + self.assertIn('nonce="', request["headers"]["X-Signature"]) + self.assertEqual(request["params"]["type"], "DERIVATIVE") + self.assertEqual(request["timeout"], (5.0, 30.0)) + + def test_timeout_and_rate_limit_retry_are_bounded(self): + client, _session, limiter, sleeps = self.client([ + requests.Timeout("injected"), + _Response(status=429, payload={}, headers={"Retry-After": "2"}), + _Response(payload=_page()), + ]) + self.assertEqual(len(client.fetch_ohlc("FPT", "1", 100, 220)), 2) + self.assertEqual(limiter.calls, 3) + self.assertEqual(sleeps, [1, 2.0]) + + def test_non_retryable_status_is_redacted_and_fails_immediately(self): + client, _session, limiter, sleeps = self.client([ + _Response(status=401, payload={}, content=b"secret-provider-body"), + ]) + with self.assertRaisesRegex(DnseHistoryError, "status=401") as raised: + client.fetch_ohlc("FPT", "1", 100, 220) + self.assertNotIn("secret-provider-body", str(raised.exception)) + self.assertEqual(limiter.calls, 1) + self.assertEqual(sleeps, []) + + def test_malformed_parallel_arrays_and_price_domain_fail_closed(self): + malformed = _page() + malformed["v"] = ["1"] + client, *_ = self.client([_Response(payload=malformed)]) + with self.assertRaisesRegex(DnseHistoryError, "array lengths"): + client.fetch_ohlc("FPT", "1", 100, 220) + + invalid = _page(close=["102", "100"]) + client, *_ = self.client([_Response(payload=invalid)]) + with self.assertRaisesRegex(DnseHistoryError, "price invariants"): + client.fetch_ohlc("FPT", "1", 100, 220) + + def test_pagination_deduplicates_equal_overlap_and_rejects_conflict(self): + first = _page((100, 160), next_time=160) + equal = _page((160, 220), next_time=0) + client, *_ = self.client([_Response(payload=first), _Response(payload=equal)]) + rows = client.fetch_ohlc("FPT", "1", 100, 280) + self.assertEqual([row["t"] for row in rows], [100, 160, 220]) + + conflict = _page((160, 220), next_time=0, close=["100.6", "100.5"]) + client, *_ = self.client([_Response(payload=first), _Response(payload=conflict)]) + with self.assertRaisesRegex(DnseHistoryError, "conflicting timestamp"): + client.fetch_ohlc("FPT", "1", 100, 280) + + def test_non_advancing_pagination_and_response_bounds_fail_closed(self): + client, *_ = self.client([_Response(payload=_page((100,), next_time=100))]) + with self.assertRaisesRegex(DnseHistoryError, "did not advance"): + client.fetch_ohlc("FPT", "1", 100, 220) + + client, *_ = self.client( + [_Response(payload=_page(), content=b"x" * 2048)], + max_response_bytes=1024, + ) + with self.assertRaisesRegex(DnseHistoryError, "byte bound"): + client.fetch_ohlc("FPT", "1", 100, 220) + + def test_quota_limiter_serializes_concurrent_reservations(self): + limiter = DnseQuotaLimiter( + hourly_limit=100, + daily_limit=100, + min_interval_seconds=0, + ) + workers = [threading.Thread(target=limiter.acquire) for _ in range(20)] + for worker in workers: + worker.start() + for worker in workers: + worker.join(timeout=2) + self.assertTrue(all(not worker.is_alive() for worker in workers)) + self.assertEqual(len(limiter._hourly), 20) + self.assertEqual(len(limiter._daily), 20) + + def test_legacy_chunk_boundaries_are_contiguous_and_failure_is_not_partial(self): + calls = [] + + def fetcher(symbol, resolution, start, end): + calls.append((symbol, resolution, start, end)) + return [] + + with patch.object(dnse_fallback, "DNSE_API_KEY", "k"), patch.object( + dnse_fallback, "DNSE_API_SECRET_KEY", "s" + ), patch.object(dnse_fallback, "_fetch_ohlc_raw", side_effect=fetcher): + frame = dnse_fallback.fetch_dnse_ohlcv_direct( + "FPT", "2026-01-01", "2026-01-10", chunk_days=7, max_retries=1 + ) + self.assertTrue(frame.empty) + self.assertEqual(calls[0][3], calls[1][2]) + + with patch.object(dnse_fallback, "DNSE_API_KEY", "k"), patch.object( + dnse_fallback, "DNSE_API_SECRET_KEY", "s" + ), patch.object( + dnse_fallback, "_fetch_ohlc_raw", side_effect=TimeoutError("injected") + ): + with self.assertRaises(TimeoutError): + dnse_fallback.fetch_dnse_ohlcv_direct( + "FPT", "2026-01-01", "2026-01-01", max_retries=1 + ) + + def test_vendor_dispatch_queue_is_bounded_only_when_opted_in(self): + bounded = TradingClient("k", "s", dispatch_queue_capacity=77) + legacy = TradingClient("k", "s") + self.assertEqual(bounded.dispatch_queue_capacity, 77) + self.assertEqual(legacy.dispatch_queue_capacity, 0) + with self.assertRaisesRegex(ValueError, "dispatch_queue_capacity"): + TradingClient("k", "s", dispatch_queue_capacity=-1) + + def test_environment_version_and_proxy_are_explicit(self): + with patch.dict( + "os.environ", + { + "DNSE_API_KEY": "k", + "DNSE_API_SECRET_KEY": "s", + "DNSE_API_VERSION": "2026-08-01", + "DNSE_REST_USE_ENV_PROXY": "true", + }, + clear=True, + ): + config = DnseHistoryConfig.from_environment() + self.assertEqual(config.api_version, "2026-08-01") + self.assertTrue(config.use_environment_proxy) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_fund_phase2_transport.py b/tests/test_fund_phase2_transport.py index 8bdb16c..1cde242 100644 --- a/tests/test_fund_phase2_transport.py +++ b/tests/test_fund_phase2_transport.py @@ -76,6 +76,7 @@ def spool(self, **overrides) -> SQLiteDurableSpool: consumer_ttl_seconds=10, replay_retention_seconds=10, maintenance_interval_seconds=1, + max_partition_records=overrides.get("max_partition_records", 0), ) return SQLiteDurableSpool(config, clock_ns=self.clock) @@ -95,6 +96,65 @@ def test_commit_restart_replay_and_idempotent_retry(self): self.assertEqual([row.event.payload for row in rows], [b"event-1"]) self.assertEqual(rows[0].payload_sha256, first.payload_sha256) + def test_cache_identity_survives_restart_and_changes_after_atomic_rebuild(self): + with self.spool() as spool: + first_cache_id = spool.cache_id + spool.append(event(1)) + with self.spool() as recovered: + self.assertEqual(recovered.cache_id, first_cache_id) + self.path.unlink() + with self.spool() as rebuilt: + self.assertNotEqual(rebuilt.cache_id, first_cache_id) + self.assertEqual(rebuilt.stats().records, 0) + + def test_tail_returns_newest_window_without_changing_replay_order(self): + with self.spool(max_records=10) as spool: + spool.append_many([event(index) for index in range(1, 6)]) + replay = spool.read( + stream=event(1).stream, + partition_key=event(1).partition_key, + limit=2, + ) + latest = spool.read_tail( + stream=event(1).stream, + partition_key=event(1).partition_key, + limit=2, + ) + self.assertEqual([row.cursor.offset for row in replay], [1, 2]) + self.assertEqual([row.cursor.offset for row in latest], [4, 5]) + self.assertEqual( + [row.event.payload for row in latest], [b"event-4", b"event-5"] + ) + + def test_partition_window_is_bounded_and_old_cursor_expires(self): + with self.spool(max_records=10, max_partition_records=3) as spool: + spool.append_many([event(index) for index in range(1, 6)]) + self.assertEqual(spool.stats().records, 3) + self.assertEqual( + set(spool.find_events( + stream=event(1).stream, + event_ids=[event(index).event_id for index in range(1, 6)], + )), + {event(index).event_id for index in range(3, 6)}, + ) + self.assertIsNone( + spool.find_event(stream=event(1).stream, event_id=event(1).event_id) + ) + with self.assertRaises(CursorExpired): + spool.read( + stream=event(1).stream, + partition_key=event(1).partition_key, + after=Cursor(event(1).stream, event(1).partition_key, 1), + ) + retained = spool.read( + stream=event(1).stream, + partition_key=event(1).partition_key, + after=Cursor(event(1).stream, event(1).partition_key, 2), + ) + self.assertEqual( + [row.cursor.offset for row in retained], [3, 4, 5] + ) + def test_event_id_collision_fails_closed_without_partial_row(self): with self.spool() as spool: spool.append(event(1)) diff --git a/tests/test_fund_phase5_api.py b/tests/test_fund_phase5_api.py index 255925f..0ef8fff 100644 --- a/tests/test_fund_phase5_api.py +++ b/tests/test_fund_phase5_api.py @@ -111,6 +111,7 @@ def setUp(self): "low": str(59_999 + index), "close": str(60_000 + index), "volume": "12.5", + "volume_unit": "BASE_ASSET", "trade_count": 10, "origin": "VENUE_NATIVE", "is_final": True, @@ -330,6 +331,43 @@ def test_stale_and_unentitled_sources_return_stable_problem_details(self): self.assertEqual(denied.status_code, 403) self.assertEqual(denied.json()["code"], "SOURCE_NOT_ALLOWED") + def test_market_closed_history_is_available_but_not_execution_eligible(self): + current = self.backend.latest(self.requirement) + self.backend.put_latest( + self.requirement, + MarketDataItem( + **{ + **current.__dict__, + "quality": QualityMetadata( + "MARKET_CLOSED", 86_400_000, False, True, False, + "alpha_crypto_primary_v1", ("MARKET_CLOSED",), + ), + } + ), + ) + response = self.client.get( + f"/v2/market-data/{self.binance.instrument_uid}/snapshot", + params=self.params(max_freshness_ms=500), + ) + self.assertEqual(response.status_code, 200, response.text) + self.assertEqual(response.json()["data"]["quality"]["state"], "MARKET_CLOSED") + self.assertFalse(response.json()["data"]["quality"]["execution_eligible"]) + + execution_requirement = DataRequirement( + **{**self.requirement.__dict__, "consumer_grade": ConsumerGrade.EXECUTION} + ) + self.backend.put_latest( + execution_requirement, self.backend.latest(self.requirement) + ) + blocked = self.client.get( + f"/v2/market-data/{self.binance.instrument_uid}/snapshot", + headers=self.headers("INTERNAL_EXECUTION"), + params=self.params(consumer_grade="EXECUTION", max_freshness_ms=500), + ) + self.assertEqual(blocked.status_code, 503) + self.assertEqual(blocked.json()["code"], "DATA_NOT_READY") + self.assertEqual(blocked.json()["quality_state"], "MARKET_CLOSED") + def test_single_query_preserves_manifest_freshness_and_final_bar_policy(self): current = self.backend.latest(self.requirement) self.backend.put_latest( diff --git a/tests/test_fund_phase5_e2e.py b/tests/test_fund_phase5_e2e.py index c585f8c..9f66956 100644 --- a/tests/test_fund_phase5_e2e.py +++ b/tests/test_fund_phase5_e2e.py @@ -13,6 +13,7 @@ from qdl.api_v2 import create_v2_app from qdl.canonical.market import canonicalize_binance_usdm_bar from qdl.canonical.trade import TradeContext, canonical_event, raw_market_event +from qdl.common.v1 import common_pb2 from qdl.consumer import ( ConsumerManifestLoader, ConsumerMigrationRegistry, @@ -129,14 +130,17 @@ def _payload(feed: FeedType, now: int) -> dict: "open_time_ns": now - 60_000_000_000, "close_time_ns": now, "open": "60000", "high": "60100", "low": "59900", - "close": "60050", "volume": "10", "trade_count": 5, + "close": "60050", "volume": "10", "volume_unit": "BASE_ASSET", + "trade_count": 5, "origin": "VENUE_NATIVE", "is_final": True, } return { "native_trade_id": "trade-1", "price": "60050", "quantity": "0.01", + "quantity_unit": "BASE_ASSET", "aggressor_side": "BUY", + "identity_kind": "NATIVE", "is_block_trade": False, "is_buyer_maker": False, } @@ -328,6 +332,9 @@ async def test_provider_fixture_reaches_v1_projection_and_v2_stream_without_dive "low": canonical.bar.low.source_text, "close": canonical.bar.close.source_text, "volume": canonical.bar.volume.source_text, + "volume_unit": common_pb2.QuantityUnit.Name( + canonical.bar.volume_unit + ).removeprefix("QUANTITY_UNIT_"), "trade_count": canonical.bar.trade_count, "origin": "VENUE_NATIVE", "is_final": canonical.bar.is_final, diff --git a/tests/test_fund_phase5_stream_sdk.py b/tests/test_fund_phase5_stream_sdk.py index 5cd2c93..cb7051f 100644 --- a/tests/test_fund_phase5_stream_sdk.py +++ b/tests/test_fund_phase5_stream_sdk.py @@ -165,6 +165,7 @@ def row(self, requirement): "low": decimal, "close": decimal, "volume": decimal, + "volume_unit": "BASE_ASSET", "trade_count": 1, "lifecycle": "FINAL", "revision": 0, @@ -400,6 +401,47 @@ async def test_grpc_emits_backpressure_control_before_slow_consumer_disconnect(s await transport.close() await server.stop(grace=0) + async def test_grpc_stream_close_has_no_cross_context_finalizer_error(self): + service = GrpcMarketDataService( + gateway=self.gateway, + query_service=None, + snapshot_loader=SnapshotLoader(self.record, self.token), + ) + server = create_grpc_server(service, identity_service=self.identity) + port = server.add_insecure_port("127.0.0.1:0") + await server.start() + transport = GrpcStreamTransport( + f"127.0.0.1:{port}", + allow_insecure_loopback=True, + credential_provider=self.credential, + ) + requirement = DataRequirement( + self.record.instrument_uid, Feed.BAR, Grade.ALPHA, + "alpha_binance_v1", interval="1m", warmup_limit=1, + ) + loop = asyncio.get_running_loop() + previous_handler = loop.get_exception_handler() + unhandled = [] + loop.set_exception_handler(lambda _loop, context: unhandled.append(context)) + events = transport.subscribe( + requirement, consumer_id="alpha-shadow", cursor_token=self.token, + max_buffer_events=1, + ).__aiter__() + try: + self.assertEqual((await events.__anext__()).code, "REPLAYING") + await events.aclose() + await asyncio.sleep(0.05) + self.assertFalse( + [ + item for item in unhandled + if "different Context" in str(item.get("exception", "")) + ] + ) + finally: + loop.set_exception_handler(previous_handler) + await transport.close() + await server.stop(grace=0) + async def test_real_grpc_sdk_handoff_ack_restart_and_bar_revisions(self): registry = InstrumentRegistry() registry.register(self.record, []) @@ -416,7 +458,8 @@ async def test_real_grpc_sdk_handoff_ack_restart_and_bar_revisions(self): "open_time_ns": now - 60_000_000_000, "close_time_ns": now, "open": "60000", "high": "60100", "low": "59900", - "close": "60050", "volume": "10", "trade_count": 5, + "close": "60050", "volume": "10", "volume_unit": "BASE_ASSET", + "trade_count": 5, "origin": "VENUE_NATIVE", "is_final": True, }, SourceMetadata("BINANCE", "BINANCE_DIRECT", "BINANCE_DIRECT", "PRIMARY", True), @@ -578,6 +621,39 @@ async def test_cursor_expiration_rebuilds_snapshot_and_transient_error_reconnect ["snapshot-token", "snapshot-token", "token-1"], ) + async def test_warmup_applies_realtime_quality_only_to_tail_watermark(self): + requirement = DataRequirement( + self.record.instrument_uid, Feed.BAR, Grade.EXECUTION, + "execution_binance_v1", interval="1m", warmup_limit=2, + max_freshness_ms=500, + ) + query = FakeQueryTransport("snapshot-token") + original = query.warmup + + async def historical_context(*args, **kwargs): + payload = await original(*args, **kwargs) + old = {**payload["data"][0]} + old["quality"] = { + **payload["data"][0]["quality"], + "state": "STALE", + "freshness_ms": 86_400_000, + "execution_eligible": False, + } + payload["count"] = 2 + payload["data"] = [old, payload["data"][0]] + return payload + + query.warmup = historical_context + client = AsyncDataLayerClient( + query_transport=query, + stream_transport=ScriptedStreamTransport(()), + consumer_id="trading-system-shadow", + ) + response = await client.warmup(requirement) + self.assertEqual(response.count, 2) + self.assertEqual(response.data[0].quality.state, "STALE") + self.assertTrue(response.data[-1].quality.execution_eligible) + async def test_sdk_rejects_semantically_invalid_success_response(self): requirement = DataRequirement( self.record.instrument_uid, Feed.BAR, Grade.EXECUTION, @@ -605,6 +681,44 @@ async def stale(*args, **kwargs): ): pass + async def test_market_closed_is_readable_for_alpha_but_execution_fails_closed(self): + alpha = DataRequirement( + self.record.instrument_uid, Feed.BAR, Grade.ALPHA, + "alpha_binance_v1", interval="1m", warmup_limit=1, + max_freshness_ms=500, + ) + query = FakeQueryTransport(self.token) + original = query.warmup + + async def market_closed(*args, **kwargs): + payload = await original(*args, **kwargs) + payload["data"][0]["quality"].update({ + "state": "MARKET_CLOSED", + "freshness_ms": 86_400_000, + "execution_eligible": False, + "flags": ["MARKET_CLOSED"], + }) + return payload + + query.warmup = market_closed + alpha_client = AsyncDataLayerClient( + query_transport=query, + stream_transport=ScriptedStreamTransport(()), + consumer_id="alpha-shadow", + ) + response = await alpha_client.warmup(alpha) + self.assertEqual(response.data[0].quality.state, "MARKET_CLOSED") + self.assertFalse(response.data[0].quality.execution_eligible) + + execution = DataRequirement( + self.record.instrument_uid, Feed.BAR, Grade.EXECUTION, + "alpha_binance_v1", interval="1m", warmup_limit=1, + max_freshness_ms=500, + ) + with self.assertRaises(DataLayerError) as raised: + await alpha_client.warmup(execution) + self.assertEqual(raised.exception.code, "SOURCE_NON_AUTHORITATIVE") + async def test_public_query_wrappers_preserve_all_requirement_policies(self): requirement = DataRequirement( self.record.instrument_uid, diff --git a/tests/test_fund_phase7_contract_security.py b/tests/test_fund_phase7_contract_security.py index 8118450..ca56b9f 100644 --- a/tests/test_fund_phase7_contract_security.py +++ b/tests/test_fund_phase7_contract_security.py @@ -120,6 +120,7 @@ def __init__(self) -> None: "low": "59900.30", "close": "60050.40", "volume": "12.500", + "volume_unit": "BASE_ASSET", "trade_count": 42, "origin": "VENUE_NATIVE", "is_final": True, @@ -218,20 +219,26 @@ def test_typed_contract_rejects_provider_fallthrough_and_feed_mismatch(self): def test_every_public_feed_has_a_closed_discriminated_payload(self): decimal = {"coefficient": "1", "scale": 0, "source_text": "1"} - level = {"side": "BID", "price": decimal, "quantity": decimal} + level = { + "side": "BID", "price": decimal, "quantity": decimal, + "quantity_unit": "BASE_ASSET", + } payloads = { "TRADE": { "native_trade_id": "trade-1", "price": decimal, - "quantity": decimal, "aggressor_side": "BUY", + "quantity": decimal, "quantity_unit": "BASE_ASSET", + "aggressor_side": "BUY", "identity_kind": "NATIVE", }, "QUOTE": { "bid_price": decimal, "bid_quantity": decimal, "ask_price": decimal, "ask_quantity": decimal, + "quantity_unit": "BASE_ASSET", }, "BAR": { "interval": "1m", "open_time_ns": 1, "close_time_ns": 2, "open": decimal, "high": decimal, "low": decimal, - "close": decimal, "volume": decimal, "lifecycle": "FINAL", + "close": decimal, "volume": decimal, "volume_unit": "BASE_ASSET", + "lifecycle": "FINAL", "revision": 0, "origin": "VENUE_NATIVE", }, "BOOK_SNAPSHOT": { @@ -242,9 +249,15 @@ def test_every_public_feed_has_a_closed_discriminated_payload(self): "snapshot_sequence": "0", "updates": [level], }, "FUNDING_RATE": {"rate": decimal, "funding_time_ns": 1}, - "OPEN_INTEREST": {"quantity": decimal, "notional": decimal}, + "OPEN_INTEREST": { + "quantity": decimal, "quantity_unit": "CONTRACT", + "notional": decimal, + }, "MARK_INDEX_PRICE": {"mark_price": decimal, "index_price": decimal}, - "TICKER": {"last_price": decimal, "volume_24h": decimal}, + "TICKER": { + "last_price": decimal, "volume_24h": decimal, + "volume_24h_unit": "BASE_ASSET", + }, } base = { "instrument_uid": self.fixture.record.instrument_uid, diff --git a/tests/test_fund_phase80_broker_substrate.py b/tests/test_fund_phase80_broker_substrate.py index c8a529e..06bb6ec 100644 --- a/tests/test_fund_phase80_broker_substrate.py +++ b/tests/test_fund_phase80_broker_substrate.py @@ -70,6 +70,8 @@ def test_brokers_are_bounded_pinned_and_have_independent_state(self) -> None: self.assertEqual(service["environment"]["KAFKA_MIN_INSYNC_REPLICAS"], 2) self.assertEqual(service["environment"]["KAFKA_SSL_CLIENT_AUTH"], "required") self.assertEqual(service["mem_limit"], "512m") + self.assertIn("/dev/tcp/127.0.0.1/9092", service["healthcheck"]["test"][1]) + self.assertNotIn("kafka-broker-api-versions", service["healthcheck"]["test"][1]) self.assertIn(f"{name}_data", volumes) redis_service = services["phase8_redis"] @@ -77,6 +79,15 @@ def test_brokers_are_bounded_pinned_and_have_independent_state(self) -> None: self.assertEqual(redis_service["mem_limit"], "64m") self.assertIn("noeviction", redis_service["command"]) + def test_admin_cli_isolated_from_broker_memory_cgroups(self) -> None: + admin = self.compose["services"]["phase8_admin"] + self.assertEqual(admin["profiles"], ["phase8-admin"]) + self.assertEqual(admin["entrypoint"], ["/bin/true"]) + self.assertEqual(admin["mem_limit"], "512m") + harness = (ROOT / "scripts/phase80_broker_certification.py").read_text() + self.assertIn('"phase8_admin"', harness) + self.assertNotIn('"exec",\n "-T",\n "kafka1"', harness) + def test_observability_contract_covers_failure_and_recovery(self) -> None: metrics = set(self.topology["observability"]["metrics"]) alerts = set(self.topology["observability"]["alerts"]) diff --git a/tests/test_fund_phase83_release.py b/tests/test_fund_phase83_release.py index c280b40..78e44ab 100644 --- a/tests/test_fund_phase83_release.py +++ b/tests/test_fund_phase83_release.py @@ -2,6 +2,7 @@ import json import pathlib +import tomllib import unittest import yaml @@ -101,6 +102,26 @@ def test_authority_topic_is_compacted_and_candidate_image_is_non_root(self): ): self.assertIn(f"/usr/local/bin/{binary}", dockerfile) self.assertIn('io.qdl.authority.default="RUST_SHADOW"', dockerfile) + self.assertIn("COPY deny.toml ./deny.toml", dockerfile) + self.assertIn( + "COPY contracts/golden ./contracts/golden", + dockerfile, + ) + self.assertIn( + "COPY tests/fixtures/phase2 ./tests/fixtures/phase2", + dockerfile, + ) + + def test_rust_supply_chain_policy_is_explicit_and_fail_closed(self): + policy = tomllib.loads((ROOT / "deny.toml").read_text()) + self.assertEqual(policy["advisories"]["ignore"], []) + self.assertEqual(policy["bans"]["wildcards"], "deny") + self.assertEqual(policy["sources"]["unknown-registry"], "deny") + self.assertEqual(policy["sources"]["unknown-git"], "deny") + self.assertEqual(policy["sources"]["allow-git"], []) + self.assertIn("MIT", policy["licenses"]["allow"]) + self.assertIn("Apache-2.0", policy["licenses"]["allow"]) + self.assertEqual(policy["licenses"]["exceptions"], []) def test_sbom_includes_authority_transport_dependencies(self): packages = build_spdx(ROOT, release="phase8-test")["packages"] diff --git a/tests/test_phase2_demand_reliability.py b/tests/test_phase2_demand_reliability.py index 89a980a..39f1965 100644 --- a/tests/test_phase2_demand_reliability.py +++ b/tests/test_phase2_demand_reliability.py @@ -10,7 +10,6 @@ from unittest.mock import patch from app.history.topup_coordinator import PreloadTopupBackoff, PreloadTopupCoordinator -from app.stream.async_live_feed import recover_demanded_kline_gap from app.stream.binance_ws import get_usdm_symbols from app.stream.demand_registry import FeedDemand, FeedDemandRegistry from app.stream.supervisor import StreamSupervisor @@ -196,6 +195,7 @@ def test_only_missing_demanded_feed_degrades_health(self): supervisor = StreamSupervisor(startup_grace_seconds=0) shard = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard) + supervisor.mark_message(shard) supervisor.expect_feed("binance_spot_kline", "kline", "BROADUSDT", "1m") broad = supervisor.snapshot(now=supervisor.started_at + 10) @@ -213,6 +213,7 @@ def test_spot_tick_cannot_satisfy_usdm_execution_demand(self): supervisor = StreamSupervisor(startup_grace_seconds=0) shard = supervisor.register_shard("binance_spot_trade", "wss://example") supervisor.mark_connected(shard) + supervisor.mark_message(shard) supervisor.record_publish({ "key": "trade:price:binance_spot:BTCUSDT", "data": {"source": "binance_spot_trade", "event_time": int(time.time() * 1000)}, @@ -227,80 +228,5 @@ def test_spot_tick_cannot_satisfy_usdm_execution_demand(self): self.assertEqual(snapshot["feeds"]["demanded_missing_count"], 1) -class ReconnectGapFillTests(unittest.IsolatedAsyncioTestCase): - async def test_gap_fill_fetches_only_demanded_symbol(self): - class Demands: - async def snapshot(self): - return {"feed_keys": ["kline:binance_usdm:1m:BTCUSDT"]} - - supervisor = StreamSupervisor() - shard = supervisor.register_shard( - "binance_futures_kline", - "wss://fstream.binance.com/stream?streams=btcusdt@kline_1m/ethusdt@kline_1m", - ) - supervisor.shards[shard].last_outage_seconds = 120 - queue = asyncio.Queue() - calls = [] - - def fetch(symbol, interval, limit, start, end, market): - calls.append((symbol, interval, limit, market)) - return { - "data": [[1, "10", "12", "9", "11", "5", 2, "", "", "", "", ""]] - } - - with patch("app.stream.async_live_feed.binance_rest.fetch_klines", side_effect=fetch): - recovered = await recover_demanded_kline_gap( - source="binance_futures_kline", - url=supervisor.shards[shard].url_preview, - interval="1m", - queue=queue, - supervisor=supervisor, - shard_id=shard, - demand_registry=Demands(), - ) - - self.assertEqual(recovered, 1) - self.assertEqual([item[0] for item in calls], ["BTCUSDT"]) - self.assertEqual(supervisor.shards[shard].gap_fill_success_count, 1) - source, event = await queue.get() - self.assertEqual(source, "binance_futures_kline") - self.assertEqual(event["recovery_source"], "BINANCE_REST_GAP_FILL") - - async def test_gap_fill_never_marks_current_open_candle_closed(self): - class Demands: - async def snapshot(self): - return {"feed_keys": ["kline:binance_usdm:1m:BTCUSDT"]} - - now_ms = int(time.time() * 1000) - supervisor = StreamSupervisor() - shard = supervisor.register_shard( - "binance_futures_kline", - "wss://fstream.binance.com/stream?streams=btcusdt@kline_1m", - ) - supervisor.shards[shard].last_outage_seconds = 60 - queue = asyncio.Queue() - closed = [now_ms - 120_000, "1", "2", "0.5", "1.5", "10", now_ms - 60_000] - open_row = [now_ms - 30_000, "1", "2", "0.5", "1.5", "10", now_ms + 30_000] - - with patch( - "app.stream.async_live_feed.binance_rest.fetch_klines", - return_value={"data": [closed, open_row]}, - ): - recovered = await recover_demanded_kline_gap( - source="binance_futures_kline", - url=supervisor.shards[shard].url_preview, - interval="1m", - queue=queue, - supervisor=supervisor, - shard_id=shard, - demand_registry=Demands(), - ) - - self.assertEqual(recovered, 1) - _, event = await queue.get() - self.assertEqual(event["k"]["T"], now_ms - 60_000) - self.assertTrue(event["k"]["x"]) - - if __name__ == "__main__": unittest.main() diff --git a/tests/test_phase90a_runtime_correctness.py b/tests/test_phase90a_runtime_correctness.py new file mode 100644 index 0000000..1698edd --- /dev/null +++ b/tests/test_phase90a_runtime_correctness.py @@ -0,0 +1,380 @@ +from __future__ import annotations + +import asyncio +import json +import unittest +from types import SimpleNamespace +from unittest.mock import patch + +from app.api.routes_health import health +from app.stream.async_live_feed import ( + StreamBackpressureTimeout, + _put_provider_item, + coalesce_redis_items, + handle_ws, + provider_items, + redis_publisher_task, + valid_provider_frame, +) +from app.stream.kline_recovery import DemandKlineRecovery, KlineRecoveryConfig +from app.stream.supervisor import StreamSupervisor + + +class FakeDemandRegistry: + def __init__(self, feed_keys: list[str]): + self.feed_keys = feed_keys + + async def snapshot(self): + return { + "lease_count": len(self.feed_keys), + "demanded_feed_count": len(self.feed_keys), + "feed_keys": list(self.feed_keys), + "items": [{"feed_key": key, "source": "binance_usdm"} for key in self.feed_keys], + } + + +class FakeKlineCache: + def __init__(self, payload=None): + self.payload = payload + + async def get_binance_kline_last(self, symbol: str, interval: str): + return self.payload + + +class RuntimeReadinessTests(unittest.TestCase): + @staticmethod + def trade_frame(): + return { + "e": "trade", + "E": 120, + "s": "BTCUSDT", + "t": 1, + "p": "60000", + "q": "0.1", + "T": 119, + "m": False, + } + + @staticmethod + def kline_frame(): + return { + "e": "kline", + "E": 120, + "s": "BTCUSDT", + "k": { + "s": "BTCUSDT", + "i": "1m", + "t": 60, + "T": 119, + "o": "1", + "h": "2", + "l": "0.5", + "c": "1.5", + "v": "10", + "x": True, + }, + } + + def test_ack_and_wrong_feed_do_not_count_as_provider_data(self): + self.assertEqual(provider_items("binance_futures_kline", {"result": None, "id": 1}), []) + self.assertFalse(valid_provider_frame("binance_futures_kline", self.trade_frame(), "1m")) + with self.assertRaisesRegex(ValueError, "wrong-feed"): + provider_items("binance_futures_kline", self.trade_frame(), "1m") + + def test_trade_and_kline_readiness_are_independent(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=5, stale_after_seconds=180) + trade = supervisor.register_shard("binance_futures_trade", "wss://trade") + kline = supervisor.register_shard("binance_futures_kline", "wss://kline") + supervisor.mark_connected(trade) + supervisor.mark_connected(kline) + supervisor.mark_message(trade) + + snapshot = supervisor.snapshot(now=supervisor.started_at + 10) + + self.assertEqual(snapshot["sources"]["binance_futures_trade"]["status"], "ready") + self.assertEqual(snapshot["sources"]["binance_futures_kline"]["status"], "unavailable") + self.assertEqual(snapshot["status"], "degraded") + self.assertIn("source_data_unavailable", snapshot["health_warnings"]) + + def test_both_valid_sources_become_ready(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=5, stale_after_seconds=180) + for source in ("binance_futures_trade", "binance_futures_kline"): + shard = supervisor.register_shard(source, f"wss://{source}") + supervisor.mark_connected(shard) + supervisor.mark_message(shard) + + snapshot = supervisor.snapshot(now=supervisor.started_at + 10) + + self.assertEqual(snapshot["status"], "ok") + self.assertTrue(all(item["data_ready"] for item in snapshot["sources"].values())) + + + def test_transport_reconnect_cannot_clear_active_data_outage(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=15) + shard = supervisor.register_shard("binance_futures_kline", "wss://kline") + supervisor.mark_connected(shard) + supervisor.mark_data_timeout(shard, "first_frame") + supervisor.mark_reconnect(shard, "first_frame timeout") + supervisor.mark_connected(shard) + + snapshot = supervisor.snapshot() + + self.assertEqual(snapshot["sources"]["binance_futures_kline"]["status"], "unavailable") + self.assertEqual(snapshot["status"], "degraded") + +class FakeWebSocketContext: + def __init__(self, messages: list[str]): + self.messages = list(messages) + + async def __aenter__(self): + return self + + async def __aexit__(self, *_): + return False + + async def recv(self): + if self.messages: + return self.messages.pop(0) + await asyncio.Event().wait() + + +class ReceiverWatchdogTests(unittest.IsolatedAsyncioTestCase): + async def test_connected_without_frame_times_out_and_reconnects(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=0.01) + shard = supervisor.register_shard("binance_futures_kline", "wss://kline") + socket = FakeWebSocketContext([]) + + with patch("app.stream.async_live_feed.websocket_connect", return_value=socket), patch( + "app.stream.async_live_feed.asyncio.sleep", side_effect=asyncio.CancelledError + ): + with self.assertRaises(asyncio.CancelledError): + await handle_ws( + "wss://kline", + asyncio.Queue(), + "binance_futures_kline", + shard, + supervisor, + reconnect_delay=0, + first_frame_timeout_seconds=0.01, + ) + + state = supervisor.shards[shard] + self.assertEqual(state.message_count, 0) + self.assertEqual(state.data_timeout_count, 1) + self.assertEqual(state.status, "reconnecting") + self.assertIn("first_frame timeout", state.last_error) + + async def test_valid_frame_then_idle_timeout_is_observable(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=0.01) + shard = supervisor.register_shard("binance_futures_trade", "wss://trade") + frame = RuntimeReadinessTests.trade_frame() + socket = FakeWebSocketContext([json.dumps({"stream": "btcusdt@trade", "data": frame})]) + queue = asyncio.Queue() + + with patch("app.stream.async_live_feed.websocket_connect", return_value=socket), patch( + "app.stream.async_live_feed.asyncio.sleep", side_effect=asyncio.CancelledError + ): + with self.assertRaises(asyncio.CancelledError): + await handle_ws( + "wss://trade", + queue, + "binance_futures_trade", + shard, + supervisor, + reconnect_delay=0, + first_frame_timeout_seconds=0.01, + idle_timeout_seconds=0.01, + ) + + state = supervisor.shards[shard] + self.assertEqual(state.message_count, 1) + self.assertEqual(state.data_timeout_count, 1) + self.assertEqual((await queue.get())[1]["s"], "BTCUSDT") + self.assertIn("idle timeout", state.last_error) + + +class BackpressureTests(unittest.IsolatedAsyncioTestCase): + async def test_full_queue_never_discards_oldest_event(self): + queue = asyncio.Queue(maxsize=1) + await queue.put(("binance_futures_trade", {"t": "first"})) + supervisor = StreamSupervisor() + + with self.assertRaises(StreamBackpressureTimeout): + await _put_provider_item( + queue, + ("binance_futures_trade", {"t": "second"}), + supervisor=supervisor, + timeout_seconds=0.01, + ) + + self.assertEqual((await queue.get())[1]["t"], "first") + self.assertEqual(supervisor.queue_pressure_count, 1) + self.assertEqual(supervisor.queue_drop_count, 0) + + def test_recovery_gap_rows_are_not_coalesced_together(self): + rows = [ + {"key": "kline:1m:BTCUSDT", "channel": "stream:kline:1m:BTCUSDT", "coalesce_id": "recovery:1", "data": {"k": {"t": 1}}}, + {"key": "kline:1m:BTCUSDT", "channel": "stream:kline:1m:BTCUSDT", "coalesce_id": "recovery:2", "data": {"k": {"t": 2}}}, + ] + self.assertEqual(len(coalesce_redis_items(rows)), 2) + + + async def test_publisher_projects_payload_interval_instead_of_stream_default(self): + class Cache: + def __init__(self): + self.items = [] + + async def push_batch(self, items): + self.items.extend(items) + + queue = asyncio.Queue() + cache = Cache() + await queue.put( + ( + "binance_futures_kline", + { + "e": "kline_recovery", + "s": "BTCUSDT", + "k": {"s": "BTCUSDT", "i": "5m", "t": 300_000, "x": True}, + "recovery_source": "BINANCE_REST_GAP_FILL", + }, + ) + ) + task = asyncio.create_task( + redis_publisher_task(queue, cache, interval="1m", supervisor=StreamSupervisor()) + ) + await asyncio.sleep(0.1) + task.cancel() + await task + + self.assertEqual(cache.items[0]["key"], "kline:5m:BTCUSDT") + self.assertEqual(cache.items[0]["channel"], "stream:kline:5m:BTCUSDT") + +class DemandKlineRecoveryTests(unittest.IsolatedAsyncioTestCase): + @staticmethod + def rows(): + return [ + [0, "1", "2", "0.5", "1.5", "10", 59_999], + [60_000, "1.5", "2.5", "1", "2", "20", 119_999], + [120_000, "2", "3", "1.5", "2.5", "30", 179_999], + ] + + async def test_active_demand_emits_only_latest_fully_closed_bar_on_cold_cache(self): + queue = asyncio.Queue() + demands = FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]) + calls = [] + + def fetcher(*args): + calls.append(args) + return {"data": self.rows()} + + manager = DemandKlineRecovery( + queue=queue, + redis_cache=FakeKlineCache(), + demand_registry=demands, + config=KlineRecoveryConfig(), + fetcher=fetcher, + ) + + result = await manager.poll_once(now_ms=150_000) + + self.assertEqual(result, {"due": 1, "emitted": 1, "failed": 0}) + source, event = await queue.get() + self.assertEqual(source, "binance_futures_kline") + self.assertEqual(event["k"]["t"], 60_000) + self.assertTrue(event["k"]["x"]) + self.assertEqual(event["recovery_source"], "BINANCE_REST_GAP_FILL") + self.assertEqual(len(calls), 1) + self.assertEqual(manager.rejected_open_or_invalid_count, 1) + + async def test_existing_final_bar_is_deduplicated(self): + existing = {"k": {"t": 60_000, "x": True}} + queue = asyncio.Queue() + manager = DemandKlineRecovery( + queue=queue, + redis_cache=FakeKlineCache(existing), + demand_registry=FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]), + config=KlineRecoveryConfig(), + fetcher=lambda *args: {"data": self.rows()[:2]}, + ) + + result = await manager.poll_once(now_ms=150_000) + + self.assertEqual(result["emitted"], 0) + self.assertEqual(manager.deduplicated_count, 2) + self.assertTrue(queue.empty()) + + async def test_no_active_demand_makes_no_provider_request_and_clears_schedule(self): + demands = FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]) + calls = [] + manager = DemandKlineRecovery( + queue=asyncio.Queue(), + redis_cache=FakeKlineCache(), + demand_registry=demands, + config=KlineRecoveryConfig(), + fetcher=lambda *args: calls.append(args) or {"data": self.rows()}, + ) + await manager.poll_once(now_ms=150_000) + demands.feed_keys = [] + + result = await manager.poll_once(now_ms=200_000) + + self.assertEqual(result, {"due": 0, "emitted": 0, "failed": 0}) + self.assertEqual(manager.active_demand_count, 0) + self.assertEqual(len(calls), 1) + self.assertEqual(manager._next_due_ms, {}) + + async def test_provider_failure_is_bounded_by_per_feed_backoff(self): + calls = [] + + def failed(*args): + calls.append(args) + raise RuntimeError("provider unavailable") + + manager = DemandKlineRecovery( + queue=asyncio.Queue(), + redis_cache=FakeKlineCache(), + demand_registry=FakeDemandRegistry(["kline:binance_usdm:1m:BTCUSDT"]), + config=KlineRecoveryConfig(poll_seconds=2, max_backoff_seconds=30), + fetcher=failed, + ) + + first = await manager.poll_once(now_ms=150_000) + second = await manager.poll_once(now_ms=151_000) + + self.assertEqual(first["failed"], 1) + self.assertEqual(second["due"], 0) + self.assertEqual(len(calls), 1) + self.assertEqual(manager.snapshot()["status"], "degraded") + + +class HealthContractTests(unittest.IsolatedAsyncioTestCase): + async def test_v1_health_booleans_reflect_matching_source(self): + supervisor = StreamSupervisor(first_frame_timeout_seconds=1) + trade = supervisor.register_shard("binance_futures_trade", "wss://trade") + kline = supervisor.register_shard("binance_futures_kline", "wss://kline") + supervisor.mark_connected(trade) + supervisor.mark_connected(kline) + supervisor.mark_message(trade) + supervisor.shards[kline].last_connected_at -= 5 + + context = SimpleNamespace( + redis_cache=SimpleNamespace(health_check=lambda: asyncio.sleep(0, result=True)), + demand_registry=FakeDemandRegistry([]), + binance_stream_supervisor=supervisor, + get_dnse_stream_manager=lambda: None, + get_kline_recovery_manager=lambda: None, + preload_topup_coordinator=SimpleNamespace(snapshot=lambda: {}), + ) + + result = await health(context) + + self.assertEqual(result["status"], "degraded") + self.assertTrue(result["binance_trade_stream"]) + self.assertFalse(result["binance_kline_stream"]) + self.assertIn("binance_stream", result) + self.assertIn("feed_demands", result) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase90b_isolated_beta.py b/tests/test_phase90b_isolated_beta.py new file mode 100644 index 0000000..74fe215 --- /dev/null +++ b/tests/test_phase90b_isolated_beta.py @@ -0,0 +1,120 @@ +from __future__ import annotations + +import copy +import unittest +from pathlib import Path + +from scripts.phase90b_bridge_parity import validate_sample, validate_window +from qdl.runtime.canary_source import CanarySourceCatalog + + +class Phase90BBridgeParityTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.binding = CanarySourceCatalog.load( + Path("config/phase7/canary-sources.yaml") + ).bindings[0] + + def sample(self): + row = [ + 1_800_000_000_000, "100.10", "101.20", "99.30", "100.40", + "12.50", 1_800_000_059_999, "1255.00", 42, "6.25", "627.50", + "0", + ] + decimal = lambda value: {"coefficient": "1", "scale": 0, "source_text": value} + item = { + "instrument_uid": self.binding.instrument.instrument_uid, + "instrument_id": self.binding.instrument.instrument_id, + "feed": "BAR", + "interval": "1m", + "payload": { + "feed": "BAR", "interval": "1m", + "open_time_ns": row[0] * 1_000_000, + "close_time_ns": row[6] * 1_000_000, + "open": decimal(row[1]), "high": decimal(row[2]), + "low": decimal(row[3]), "close": decimal(row[4]), + "volume": decimal(row[5]), "volume_unit": "BASE_ASSET", + "trade_count": row[8], + "lifecycle": "FINAL", + }, + "source": { + "source_id": self.binding.source_id, + "source_role": self.binding.source_role, + "authoritative": self.binding.authoritative, + }, + "quality": { + "policy_id": self.binding.source_policy_id, + "complete": True, + "gap_open": False, + "execution_eligible": False, + }, + } + v1 = { + "provider": "binance", "market": "usdm", "symbol": "BTCUSDT", + "requested_interval": "1m", "data": [row], + } + v2 = { + "schema": "qdl.marketdata.warmup.v2", "count": 1, + "watermark_offset": 1, "data": [item], + } + return v1, v2 + + def test_exact_provider_bar_passes(self): + v1, v2 = self.sample() + result = validate_sample(v1, v2, self.binding) + self.assertEqual(result["count"], 1) + validate_window(result, result) + + def test_decimal_mismatch_fails(self): + v1, v2 = self.sample() + v2["data"][0]["payload"]["close"]["source_text"] = "100.41" + with self.assertRaisesRegex(AssertionError, "close"): + validate_sample(v1, v2, self.binding) + + def test_non_final_and_execution_eligible_fail(self): + v1, v2 = self.sample() + v2["data"][0]["payload"]["lifecycle"] = "IN_PROGRESS" + v2["data"][0]["quality"]["execution_eligible"] = True + with self.assertRaisesRegex(AssertionError, "final"): + validate_sample(v1, v2, self.binding) + + def test_duplicate_open_time_fails(self): + v1, v2 = self.sample() + v2["data"].append(copy.deepcopy(v2["data"][0])) + v2["count"] = 2 + v2["watermark_offset"] = 2 + with self.assertRaisesRegex(AssertionError, "duplicated"): + validate_sample(v1, v2, self.binding) + + def test_certification_harness_is_rootless_host_portable(self): + phase73 = Path("scripts/phase73_public_beta_certification.sh").read_text() + self.assertIn('runner_evidence="${temporary}/runner"', phase73) + self.assertIn('OPERATOR_UID="$(id -u)"', phase73) + self.assertIn('--user 0:0 --cap-drop ALL --cap-add CHOWN --cap-add FOWNER', phase73) + self.assertIn('set_evidence_owner 10001 10001', phase73) + self.assertIn('set_evidence_owner "${OPERATOR_UID}" "${OPERATOR_GID}"', phase73) + self.assertNotIn('chown 10001:10001 "${temporary}"', phase73) + + phase90b = Path("scripts/phase90b_isolated_beta_certification.sh").read_text() + self.assertIn('CERT_UID="${QDL_CERT_UID:-$(id -u)}"', phase90b) + self.assertIn('CERT_GID="${QDL_CERT_GID:-$(id -g)}"', phase90b) + self.assertIn( + '--profile phase7-beta --profile phase7-canary ps -q qdl_beta_v1_bridge', + phase90b, + ) + self.assertIn('realpath --relative-to="${ROOT_DIR}"', phase90b) + self.assertIn('(cd "${ROOT_DIR}" && sha256sum -c "${CHECKSUM_OUTPUT}")', phase90b) + self.assertNotIn('chown 10001:10001 "${temporary}"', phase90b) + + def test_window_rejects_regression_and_unbounded_growth(self): + first = {"watermark_offset": 5, "last_open_time_ns": 100} + with self.assertRaisesRegex(AssertionError, "outside"): + validate_window(first, {"watermark_offset": 4, "last_open_time_ns": 100}) + with self.assertRaisesRegex(AssertionError, "outside"): + validate_window(first, {"watermark_offset": 7, "last_open_time_ns": 100}) + with self.assertRaisesRegex(AssertionError, "backwards"): + validate_window(first, {"watermark_offset": 6, "last_open_time_ns": 99}) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase90c_migration_contract.py b/tests/test_phase90c_migration_contract.py new file mode 100644 index 0000000..a32af06 --- /dev/null +++ b/tests/test_phase90c_migration_contract.py @@ -0,0 +1,61 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase90CMigrationContractTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.sql = ( + ROOT / "migrations/postgres/0006_phase9_authority_prerequisites.sql" + ).read_text() + + def test_additive_authority_and_evidence_schema_exists(self): + for name in ( + "qdl_production_prerequisite_bundles", + "qdl_authority_slices", + "qdl_authority_transition_audit", + "qdl_transition_authority", + ): + self.assertIn(name, self.sql) + self.assertNotIn("DROP TABLE", self.sql.upper()) + self.assertNotIn("TRUNCATE", self.sql.upper()) + + def test_cas_and_go_bundle_are_database_enforced(self): + self.assertIn("authority compare-and-swap precondition failed", self.sql) + self.assertIn("p_expected_partition_plan_epoch", self.sql) + self.assertIn("p_new_state IN ('RUST_CANARY', 'RUST_PRIMARY')", self.sql) + self.assertIn("bundle_row.decision <> 'GO'", self.sql) + self.assertIn("bundle_row.candidate_digest <> current_row.candidate_digest", self.sql) + self.assertIn("bundle_row.expires_at <= clock_timestamp()", self.sql) + self.assertIn("bundle_row.expires_at < p_hold_until", self.sql) + self.assertIn("canary/primary requires a non-negative terminal watermark", self.sql) + self.assertIn("canary/primary requires a future approval hold window", self.sql) + self.assertIn("valid only for canary/primary", self.sql) + self.assertIn("THEN p_prerequisite_bundle_id ELSE NULL", self.sql) + + def test_authority_record_binds_release_provenance_and_hold_window(self): + for field in ( + "signature_identity", "normalizer_version", "adapter_version", + "config_revision", "instrument_catalog_revision", + "source_policy_revision", "hold_until", + ): + self.assertIn(field, self.sql) + self.assertIn("p_hold_until", self.sql) + + def test_audit_is_append_only_and_new_owner_needs_new_epoch(self): + self.assertIn("qdl_authority_audit_immutable", self.sql) + self.assertIn("qdl_authority_transition_audit is append-only", self.sql) + self.assertIn("new owner requires a strictly newer lease epoch", self.sql) + + def test_migration_smoke_keeps_heredoc_stdin_open(self): + harness = (ROOT / "scripts/phase90c_migration_smoke.sh").read_text() + self.assertIn('docker exec -i "${CONTAINER}" psql', harness) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase90c_prerequisites.py b/tests/test_phase90c_prerequisites.py new file mode 100644 index 0000000..b1f1a40 --- /dev/null +++ b/tests/test_phase90c_prerequisites.py @@ -0,0 +1,230 @@ +from __future__ import annotations + +import copy +import hashlib +import subprocess +import tempfile +import time +import unittest +from dataclasses import replace +from pathlib import Path + +import yaml + +from qdl.certification.prerequisites import ( + CandidateSlice, + EvidenceRecord, + EvidenceScope, + PrerequisitePolicy, + evaluate_prerequisites, + load_inventory, +) + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase90CPrerequisiteTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.policy = PrerequisitePolicy.load( + ROOT / "config/phase9/production-prerequisite-policy.yaml" + ) + cls.candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + + def test_local_inventory_is_honest_no_go(self): + inventory = load_inventory(ROOT / "config/phase9/evidence-inventory.local.yaml") + decision = evaluate_prerequisites( + self.policy, self.candidate, inventory, + repository_root=ROOT, now_ns=1_787_063_420_299_706_631, + ) + self.assertEqual(decision.decision, "NO_GO_EXTERNAL") + results = {item.gate_id: item for item in decision.results} + self.assertEqual(results["replicated_durable_transport"].reason, "INSUFFICIENT_SCOPE") + self.assertEqual(results["signed_artifact_admission"].reason, "EVIDENCE_BLOCKED") + self.assertEqual(results["independent_failure_domain_dr"].reason, "MISSING_EVIDENCE") + self.assertFalse(any(item.passed for item in decision.results)) + + def _passing_inventory(self, root: Path, now_ns: int): + artifact = root / "evidence.json" + artifact.write_text("{}\n") + digest = hashlib.sha256(artifact.read_bytes()).hexdigest() + records = [] + for gate in self.policy.gates: + details = {key: f"proof-{key}" for key in gate.required_details} + details.update(gate.assertions) + details.update(gate.minimums) + details.update(gate.maximums) + details.update({ + detail_key: self.candidate.payload[candidate_key] + for detail_key, candidate_key in gate.candidate_field_matches.items() + }) + if gate.candidate_bound: + details["candidate_digest"] = self.candidate.digest + if gate.gate_id == "exact_slice_approval": + details["hold_until_ns"] = now_ns + 30_000_000_000 + records.append(EvidenceRecord.from_mapping({ + "evidence_id": f"evidence-{gate.gate_id}", + "gate_id": gate.gate_id, + "environment": "production", + "scope": gate.minimum_scope.value, + "status": "PASS", + "issuer": "production-control-plane", + "observed_at_ns": now_ns - 1_000_000, + "expires_at_ns": now_ns + 60_000_000_000, + "artifact_path": "evidence.json", + "artifact_sha256": digest, + "details": details, + })) + return tuple(records), artifact + + def test_all_exact_production_evidence_is_go(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + records, _ = self._passing_inventory(Path(directory), now_ns) + decision = evaluate_prerequisites( + self.policy, self.candidate, records, + repository_root=directory, now_ns=now_ns, + ) + self.assertEqual(decision.decision, "GO") + self.assertTrue(all(item.passed for item in decision.results)) + + def test_tampered_expired_and_candidate_mismatch_fail_closed(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + records, artifact = self._passing_inventory(root, now_ns) + artifact.write_text("tampered\n") + decision = evaluate_prerequisites( + self.policy, self.candidate, records, + repository_root=root, now_ns=now_ns, + ) + self.assertTrue(all(item.reason == "ARTIFACT_DIGEST_MISMATCH" for item in decision.results)) + + records, _ = self._passing_inventory(root, now_ns) + first = records[0] + expired = EvidenceRecord(**{**first.__dict__, "expires_at_ns": now_ns - 1}) + decision = evaluate_prerequisites( + self.policy, self.candidate, (expired, *records[1:]), + repository_root=root, now_ns=now_ns, + ) + self.assertEqual(decision.results[0].reason, "EVIDENCE_EXPIRED") + + bound_index = next(i for i, gate in enumerate(self.policy.gates) if gate.candidate_bound) + bound = records[bound_index] + wrong = EvidenceRecord(**{ + **bound.__dict__, + "details": {**bound.details, "candidate_digest": "0" * 64}, + }) + changed = list(records); changed[bound_index] = wrong + decision = evaluate_prerequisites( + self.policy, self.candidate, tuple(changed), + repository_root=root, now_ns=now_ns, + ) + self.assertEqual(decision.results[bound_index].reason, "CANDIDATE_DIGEST_MISMATCH") + + def test_unknown_sensitive_duplicate_and_unsafe_candidate_are_rejected(self): + raw = yaml.safe_load((ROOT / "config/phase9/evidence-inventory.local.yaml").read_text()) + item = copy.deepcopy(raw["evidence"][0]) + item["details"]["private_key"] = "must-not-enter-evidence" + with self.assertRaisesRegex(ValueError, "sensitive"): + EvidenceRecord.from_mapping(item) + item["details"].pop("private_key") + item["details"]["nested"] = {"access_token": "must-not-enter-evidence"} + with self.assertRaisesRegex(ValueError, "sensitive"): + EvidenceRecord.from_mapping(item) + + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "inventory.yaml" + raw["evidence"].append(copy.deepcopy(raw["evidence"][0])) + path.write_text(yaml.safe_dump(raw)) + with self.assertRaisesRegex(ValueError, "unique"): + load_inventory(path) + + candidate = yaml.safe_load((ROOT / "config/phase9/candidate-slice.yaml").read_text()) + candidate["public_write_allowed"] = True + with tempfile.NamedTemporaryFile("w", suffix=".yaml") as handle: + yaml.safe_dump(candidate, handle); handle.flush() + with self.assertRaisesRegex(ValueError, "cannot write"): + CandidateSlice.load(handle.name) + + candidate = yaml.safe_load((ROOT / "config/phase9/candidate-slice.yaml").read_text()) + del candidate["normalizer_version"] + with tempfile.NamedTemporaryFile("w", suffix=".yaml") as handle: + yaml.safe_dump(candidate, handle); handle.flush() + with self.assertRaisesRegex(ValueError, "incomplete"): + CandidateSlice.load(handle.name) + + candidate = yaml.safe_load((ROOT / "config/phase9/candidate-slice.yaml").read_text()) + candidate["instrument_uids"] = ["not-a-uuid"] + with tempfile.NamedTemporaryFile("w", suffix=".yaml") as handle: + yaml.safe_dump(candidate, handle); handle.flush() + with self.assertRaisesRegex(ValueError, "UID is invalid"): + CandidateSlice.load(handle.name) + + def test_semantic_assertion_threshold_candidate_and_hold_fail_closed(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + records, _ = self._passing_inventory(root, now_ns) + indexes = {item.gate_id: index for index, item in enumerate(records)} + + cases = ( + ("replicated_durable_transport", {"acks_all": False}, "ASSERTION_FAILED"), + ("replicated_durable_transport", {"broker_count": 2}, "MINIMUM_NOT_MET"), + ("replicated_durable_transport", {"broker_count": True}, "MINIMUM_NOT_MET"), + ("postgres_pitr", {"rpo_seconds": 301}, "MAXIMUM_EXCEEDED"), + ("signed_artifact_admission", {"image_digest": "sha256:" + "0" * 64}, "CANDIDATE_FIELD_MISMATCH"), + ("exact_slice_approval", {"hold_until_ns": now_ns}, "APPROVAL_HOLD_WINDOW_INVALID"), + ) + for gate_id, detail_update, expected in cases: + changed = list(records) + index = indexes[gate_id] + changed[index] = replace( + changed[index], details={**changed[index].details, **detail_update} + ) + decision = evaluate_prerequisites( + self.policy, self.candidate, tuple(changed), + repository_root=root, now_ns=now_ns, + ) + self.assertEqual(decision.results[index].reason, expected, gate_id) + + def test_unknown_evidence_gate_is_rejected(self): + now_ns = time.time_ns() + with tempfile.TemporaryDirectory() as directory: + records, _ = self._passing_inventory(Path(directory), now_ns) + unknown = replace(records[0], evidence_id="unknown", gate_id="unknown_gate") + with self.assertRaisesRegex(ValueError, "unknown prerequisite"): + evaluate_prerequisites( + self.policy, self.candidate, (*records, unknown), + repository_root=directory, now_ns=now_ns, + ) + + def test_policy_is_complete_and_provider_neutral(self): + ids = {gate.gate_id for gate in self.policy.gates} + self.assertEqual(len(ids), 12) + self.assertTrue({ + "replicated_durable_transport", "production_observability", + "workload_identity_rbac_network", "external_secret_rotation", + "signed_artifact_admission", "postgres_pitr", "object_store_restore", + "independent_failure_domain_dr", "redis_projector_rebuild", + "consumer_registration_rollback", "persistent_authority_sink_fencing", + "exact_slice_approval", + }.issubset(ids)) + policy_text = (ROOT / "config/phase9/production-prerequisite-policy.yaml").read_text() + self.assertNotIn("BINANCE", policy_text) + self.assertEqual( + next(g.minimum_scope for g in self.policy.gates if g.gate_id == "independent_failure_domain_dr"), + EvidenceScope.INDEPENDENT_FAILURE_DOMAIN, + ) + + def test_operator_entrypoint_imports_from_repository_root(self): + completed = subprocess.run( + ["python3", "scripts/phase90c_prerequisite_certification.py", "--help"], + cwd=ROOT, text=True, capture_output=True, check=False, + ) + self.assertEqual(completed.returncode, 0, completed.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase91_canary_control.py b/tests/test_phase91_canary_control.py new file mode 100644 index 0000000..0698465 --- /dev/null +++ b/tests/test_phase91_canary_control.py @@ -0,0 +1,198 @@ +from __future__ import annotations + +import copy +import json +import time +import unittest +from pathlib import Path + +from qdl.canary.phase9 import ( + CanaryAuthorizationMode, + CanaryGuardrailEngine, + CanaryGuardrailPolicy, + CanaryObservation, + ProductionCanaryAuthorizer, +) +from qdl.certification.prerequisites import CandidateSlice + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase91CanaryAuthorizationTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + cls.no_go = json.loads( + (ROOT / "upgrade/evidence/phase90c-production-prerequisites.json").read_text() + ) + + def _go(self, now_ns: int) -> dict: + payload = copy.deepcopy(self.no_go) + payload.update({ + "decision": "GO", + "passed": len(payload["gates"]), + "blocked": 0, + "issued_at_ns": now_ns - 1_000_000, + "authority_state": "RUST_SHADOW", + "v1_unchanged": True, + "production_mutations": 0, + }) + for gate in payload["gates"]: + gate.update({"passed": True, "reason": "PASS"}) + return payload + + def test_current_external_no_go_cannot_authorize_production(self): + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, + decision=self.no_go, + expected_bundle_id=self.no_go["bundle_id"], + now_ns=self.no_go["issued_at_ns"] + 1, + ) + self.assertFalse(result.allowed) + self.assertFalse(result.production_authorized) + self.assertEqual(result.reason, "PREREQUISITE_DECISION_NOT_GO") + + def test_rehearsal_is_explicitly_non_production(self): + result = ProductionCanaryAuthorizer.authorize_isolated_rehearsal( + candidate=self.candidate + ) + self.assertTrue(result.allowed) + self.assertFalse(result.production_authorized) + self.assertEqual(result.mode, CanaryAuthorizationMode.ISOLATED_REHEARSAL) + self.assertEqual(result.reason, "ISOLATED_REHEARSAL_ONLY") + + def test_exact_fresh_go_authorizes_without_mutating(self): + now_ns = time.time_ns() + decision = self._go(now_ns) + snapshot = copy.deepcopy(decision) + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, + decision=decision, + expected_bundle_id=decision["bundle_id"], + now_ns=now_ns, + ) + self.assertTrue(result.allowed) + self.assertTrue(result.production_authorized) + self.assertEqual(result.reason, "AUTHORIZED") + self.assertEqual(decision, snapshot) + + def test_identity_freshness_gate_and_v1_preconditions_fail_closed(self): + now_ns = time.time_ns() + cases = ( + ({"candidate_digest": "0" * 64}, "CANDIDATE_DIGEST_MISMATCH"), + ({"slice_id": "production/other"}, "SLICE_MISMATCH"), + ({"bundle_id": "11111111-1111-4111-8111-111111111111"}, "PREREQUISITE_BUNDLE_MISMATCH"), + ({"issued_at_ns": now_ns - 901_000_000_000}, "DECISION_EXPIRED"), + ({"authority_state": "RUST_CANARY"}, "AUTHORITY_PRECONDITION_INVALID"), + ({"v1_unchanged": False}, "V1_PRECONDITION_INVALID"), + ({"production_mutations": 1}, "PRODUCTION_MUTATION_DETECTED"), + ) + for update, expected in cases: + with self.subTest(expected=expected): + decision = self._go(now_ns) + expected_bundle = decision["bundle_id"] + decision.update(update) + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, + decision=decision, + expected_bundle_id=expected_bundle, + now_ns=now_ns, + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + def test_incomplete_duplicate_or_failed_gates_are_rejected(self): + now_ns = time.time_ns() + decision = self._go(now_ns) + expected_bundle = decision["bundle_id"] + decision["gates"].pop() + decision["passed"] -= 1 + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, decision=decision, + expected_bundle_id=expected_bundle, now_ns=now_ns, + ) + self.assertEqual(result.reason, "GATE_RESULTS_INVALID") + + decision = self._go(now_ns) + decision["gates"][0]["passed"] = False + decision["gates"][0]["reason"] = "FAILED" + result = ProductionCanaryAuthorizer().authorize( + candidate=self.candidate, decision=decision, + expected_bundle_id=expected_bundle, now_ns=now_ns, + ) + self.assertEqual(result.reason, "GATE_NOT_PASSED") + + +class Phase91GuardrailTest(unittest.TestCase): + def setUp(self): + self.policy = CanaryGuardrailPolicy( + max_lag_ms=500, + max_freshness_ms=1_000, + max_cpu_percent=80.0, + max_rss_mb=512.0, + max_queue_depth=1_000, + hold_down_seconds=30, + ) + + def test_clean_observation_passes(self): + result = CanaryGuardrailEngine(self.policy).evaluate( + CanaryObservation(observed_at_ns=1) + ) + self.assertTrue(result.allowed) + self.assertEqual(result.reason, "PASS") + + def test_each_correctness_and_capacity_guardrail_blocks(self): + cases = ( + ({"semantic_mismatches": 1}, "SEMANTIC_MISMATCH"), + ({"open_gaps": 1}, "OPEN_GAP"), + ({"duplicate_external_writes": 1}, "DUPLICATE_EXTERNAL_WRITE"), + ({"stale_writer_attempts": 1}, "STALE_WRITER_ATTEMPT"), + ({"authority_ambiguities": 1}, "AUTHORITY_AMBIGUITY"), + ({"durable_ack_failures": 1}, "DURABLE_ACK_FAILURE"), + ({"lag_ms": 501}, "LAG_THRESHOLD_EXCEEDED"), + ({"freshness_ms": 1_001}, "FRESHNESS_THRESHOLD_EXCEEDED"), + ({"cpu_percent": 80.1}, "CPU_THRESHOLD_EXCEEDED"), + ({"rss_mb": 512.1}, "RSS_THRESHOLD_EXCEEDED"), + ({"queue_depth": 1_001}, "QUEUE_THRESHOLD_EXCEEDED"), + ) + for update, expected in cases: + with self.subTest(expected=expected): + result = CanaryGuardrailEngine(self.policy).evaluate( + CanaryObservation(observed_at_ns=1, **update) + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + def test_first_failure_is_sticky_and_reentry_is_explicit_after_hold(self): + engine = CanaryGuardrailEngine(self.policy) + blocked = engine.evaluate( + CanaryObservation(observed_at_ns=1_000, semantic_mismatches=1) + ) + self.assertEqual(blocked.reason, "SEMANTIC_MISMATCH") + self.assertEqual(blocked.hold_until_ns, 30_000_001_000) + self.assertEqual( + engine.evaluate(CanaryObservation(observed_at_ns=2_000)).reason, + "EXPLICIT_RESET_REQUIRED", + ) + self.assertEqual( + engine.reset_after_hold(CanaryObservation(observed_at_ns=30_000_000_999)).reason, + "HOLD_DOWN_ACTIVE", + ) + reset = engine.reset_after_hold( + CanaryObservation(observed_at_ns=30_000_001_000) + ) + self.assertTrue(reset.allowed) + self.assertEqual(reset.reason, "RESET_CONFIRMED") + + def test_invalid_thresholds_and_observations_are_rejected(self): + with self.assertRaises(ValueError): + CanaryGuardrailPolicy(0, 1, 1, 1, 1, 1) + with self.assertRaises(ValueError): + CanaryObservation(observed_at_ns=0) + with self.assertRaises(ValueError): + CanaryObservation(observed_at_ns=1, open_gaps=-1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase91_certification_contract.py b/tests/test_phase91_certification_contract.py new file mode 100644 index 0000000..fe124cc --- /dev/null +++ b/tests/test_phase91_certification_contract.py @@ -0,0 +1,116 @@ +from __future__ import annotations + +import gzip +import hashlib +import json +import subprocess +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase91CertificationContractTest(unittest.TestCase): + def test_frozen_capture_is_authentic_read_only_and_checksummed(self): + capture = ROOT / "upgrade/evidence/captures/phase8-real-provider-frames.json.gz" + evidence = json.loads( + (ROOT / "upgrade/evidence/phase8-real-provider-shadow.json").read_text() + ) + compressed = capture.read_bytes() + self.assertEqual( + hashlib.sha256(compressed).hexdigest(), evidence["capture_bundle_sha256"] + ) + payload = json.loads(gzip.decompress(compressed)) + self.assertEqual(payload["provenance"], "REAL_PROVIDER_READ_ONLY") + self.assertEqual(payload["production_writes"], 0) + binance = [ + item for item in payload["captures"] + if item["venue"] == "BINANCE" + and item["market"] == "USDM" + and item["native_symbol"] == "BTCUSDT" + and item["test_provenance"] is False + ] + self.assertGreaterEqual(len(binance), 32) + + def test_tls_generator_preserves_host_ownership(self): + script = (ROOT / "scripts/phase80_generate_tls.sh").read_text() + self.assertIn("QDL_PHASE8_CERT_UID", script) + self.assertIn("QDL_PHASE8_CERT_GID", script) + self.assertIn('--user "${CERT_UID}:${CERT_GID}"', script) + self.assertNotIn("docker run --rm --user 0:0", script) + + def test_runtime_image_and_workspace_declare_phase91_binary(self): + cargo = (ROOT / "rust/qdl-kafka/Cargo.toml").read_text() + dockerfile = (ROOT / "Dockerfile.phase8-rust").read_text() + self.assertIn('name = "qdl-phase91-canary-rehearsal"', cargo) + self.assertIn("--bin qdl-phase91-canary-rehearsal", dockerfile) + self.assertIn( + "/usr/local/bin/qdl-phase91-canary-rehearsal", dockerfile + ) + + def test_certification_is_isolated_and_has_cleanup_and_zero_write_gates(self): + harness = (ROOT / "scripts/phase91_canary_certification.py").read_text() + runtime = ( + ROOT / "rust/qdl-kafka/src/bin/qdl-phase91-canary-rehearsal.rs" + ).read_text() + for token in ( + "NO_GO_EXTERNAL", + "production_mutations", + "v1_topology_unchanged", + "containers_after", + "public_writes", + "legacy_writes", + "min_isr_failed_closed", + "slow_consumer", + ): + self.assertIn(token, harness) + for token in ( + "topic_masquerade_rejected", + "stale_owner_rejected", + "stale_revision_rejected", + "stale_lease_rejected", + "wrong_plan_rejected", + "duplicate_watermark_rejected", + "blocked_state_rejected", + "canary_after_rollback_rejected", + ): + self.assertIn(token, runtime) + + def test_compacted_authority_reads_latest_record_only(self): + harness = (ROOT / "scripts/phase91_canary_certification.py").read_text() + self.assertIn("def consume_compacted_records", harness) + self.assertIn("--partition", harness) + self.assertIn("item.strip().startswith(\"{\")", harness) + self.assertIn("consume_compacted_records(env, topics[\"authority\"])", harness) + self.assertIn('topics["audit"], 4', harness) + self.assertIn("compose(env, \"stop\", \"kafka3\")", harness) + self.assertIn("compose(env, \"stop\", \"kafka2\")", harness) + self.assertIn("compose(env, \"start\", \"kafka2\", \"kafka3\"", harness) + self.assertIn("\"QDL_KAFKA_BOOTSTRAP_SERVERS\": \"kafka1:9092,kafka2:9092\"", harness) + self.assertIn("min_isr_before = one_loss_after", harness) + self.assertIn("min_isr_client_failed = min_isr.returncode != 0", harness) + self.assertIn("min_isr_after = total_end_offset", harness) + self.assertIn("min_isr_after == min_isr_before", harness) + + def test_makefile_and_evidence_packaging_are_reproducible(self): + makefile = (ROOT / "Makefile").read_text() + harness = (ROOT / "scripts/phase91_canary_certification.py").read_text() + for target in ("phase91-build:", "phase91-test:", "phase91-certify:", "phase91-clean:"): + self.assertIn(target, makefile) + self.assertIn("PHASE91_RUST_IMAGE", makefile) + self.assertIn("def render_report", harness) + self.assertIn("def write_evidence", harness) + self.assertIn("phase91-evidence.sha256", harness) + + def test_operator_entrypoints_have_valid_help_or_usage(self): + result = subprocess.run( + ["python3", "scripts/phase91_canary_certification.py", "--help"], + cwd=ROOT, text=True, capture_output=True, check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("--rust-image", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase92_certification_contract.py b/tests/test_phase92_certification_contract.py new file mode 100644 index 0000000..d72cbe1 --- /dev/null +++ b/tests/test_phase92_certification_contract.py @@ -0,0 +1,90 @@ +from __future__ import annotations + +import subprocess +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase92CertificationContractTest(unittest.TestCase): + def test_runtime_image_declares_primary_rehearsal_binary(self): + cargo = (ROOT / "rust/qdl-kafka/Cargo.toml").read_text() + dockerfile = (ROOT / "Dockerfile.phase8-rust").read_text() + self.assertIn('name = "qdl-phase92-primary-rehearsal"', cargo) + self.assertIn("--bin qdl-phase92-primary-rehearsal", dockerfile) + self.assertIn( + "/usr/local/bin/qdl-phase92-primary-rehearsal", dockerfile + ) + + def test_harness_is_isolated_and_checks_handoff_recovery(self): + harness = (ROOT / "scripts/phase92_primary_certification.py").read_text() + runtime = ( + ROOT / "rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs" + ).read_text() + for token in ( + "NO_GO_EXTERNAL", + "production_mutations", + "v1_topology_unchanged", + "production_public_writes", + "production_legacy_writes", + "projection_parity", + "boundary_gap_free", + "one_replica_loss_acked", + "min_isr_failed_closed", + "process_restart_recovery", + "RECOVERY_VERIFY", + "QDL_RECOVERY_LAST_WATERMARK", + ): + self.assertIn(token, harness) + for token in ( + "direct_primary_without_handoff_rejected", + "terminal_watermark_rejected", + "gap_watermark_rejected", + "stale_owner_rejected", + "stale_revision_rejected", + "stale_lease_rejected", + "wrong_plan_rejected", + "duplicate_after_ack_rejected", + "rust_after_rollback_rejected", + "restart_pre_restore_failed_closed", + "each_target_pre_restore_failed_closed", + "durable_target_watermarks_restored", + "duplicate_after_restore_rejected", + "resumed_at_exact_next_watermark", + "restore_committed_watermark", + ): + self.assertIn(token, runtime) + + def test_rehearsal_topics_cannot_masquerade_as_production(self): + runtime = ( + ROOT / "rust/qdl-kafka/src/bin/qdl-phase92-primary-rehearsal.rs" + ).read_text() + self.assertIn('!topic.contains(".phase92.")', runtime) + self.assertIn("QDL_PRODUCTION_PUBLIC_TOPIC", runtime) + self.assertIn("QDL_PRODUCTION_LEGACY_TOPIC", runtime) + + def test_makefile_and_operator_entrypoints_are_reproducible(self): + makefile = (ROOT / "Makefile").read_text() + for target in ( + "phase92-build:", + "phase92-test:", + "phase92-migration:", + "phase92-certify:", + "phase92-clean:", + ): + self.assertIn(target, makefile) + result = subprocess.run( + ["python3", "scripts/phase92_primary_certification.py", "--help"], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("--rust-image", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase92_migration_contract.py b/tests/test_phase92_migration_contract.py new file mode 100644 index 0000000..88466cb --- /dev/null +++ b/tests/test_phase92_migration_contract.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase92MigrationContractTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.sql = ( + ROOT / "migrations/postgres/0007_phase92_terminal_handoff.sql" + ).read_text() + + def test_migration_is_additive_and_evidence_is_immutable(self): + for token in ( + "qdl_terminal_owner_checkpoints", + "qdl_authority_handoffs", + "qdl_transition_authority_v2", + "qdl_authority_primary_handoff_guard", + "Phase 9.2 checkpoint/handoff evidence is append-only", + ): + self.assertIn(token, self.sql) + self.assertNotIn("DROP TABLE", self.sql.upper()) + self.assertNotIn("TRUNCATE", self.sql.upper()) + + def test_terminal_boundary_and_clean_reconciliation_are_enforced(self): + for token in ( + "first_new_watermark = terminal_watermark + 1", + "semantic_mismatches = 0", + "open_gaps = 0", + "old_event_count = new_event_count", + "new_authority_revision = expected_authority_revision + 1", + "new_lease_epoch > expected_lease_epoch", + ): + self.assertIn(token, self.sql) + + def test_direct_primary_and_rollback_bypass_are_trigger_fenced(self): + self.assertIn("NEW.state = 'RUST_PRIMARY'", self.sql) + self.assertIn("NEW.state = 'PYTHON_PRIMARY'", self.sql) + self.assertIn("matching_handoffs <> 1", self.sql) + self.assertIn("accepted exact Phase 9.2 handoff is required", self.sql) + + def test_v2_transition_rechecks_exact_cas_and_expiry(self): + for token in ( + "handoff.expected_state <> p_expected_state", + "handoff.expected_authority_revision <> p_expected_revision", + "handoff.expected_lease_epoch <> p_expected_lease_epoch", + "handoff.partition_plan_epoch <> p_expected_partition_plan_epoch", + "handoff.expires_at <= clock_timestamp()", + "Phase 9.2 prerequisite bundle mismatch", + ): + self.assertIn(token, self.sql) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase92_primary_control.py b/tests/test_phase92_primary_control.py new file mode 100644 index 0000000..bd8d2fe --- /dev/null +++ b/tests/test_phase92_primary_control.py @@ -0,0 +1,223 @@ +from __future__ import annotations + +import copy +import json +import time +import unittest +import uuid +from pathlib import Path + +from qdl.canary.phase92 import ( + AcceptedHandoff, + HandoffDirection, + PrimaryAuthorizationMode, + ProductionPrimaryAuthorizer, + TerminalOwnerCheckpoint, +) +from qdl.certification.prerequisites import CandidateSlice + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase92PrimaryAuthorizationTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.candidate = CandidateSlice.load(ROOT / "config/phase9/candidate-slice.yaml") + cls.no_go = json.loads( + (ROOT / "upgrade/evidence/phase90c-production-prerequisites.json").read_text() + ) + + def _go(self, now_ns: int) -> dict: + payload = copy.deepcopy(self.no_go) + payload.update({ + "decision": "GO", + "passed": len(payload["gates"]), + "blocked": 0, + "issued_at_ns": now_ns - 1_000_000, + "authority_state": "RUST_SHADOW", + "v1_unchanged": True, + "production_mutations": 0, + }) + for gate in payload["gates"]: + gate.update({"passed": True, "reason": "PASS"}) + return payload + + def _canary(self, now_ns: int) -> dict: + return { + "schema": "qdl.phase91.rust-canary-certification.v1", + "status": "PRODUCTION_CANARY_HOLD_PASSED", + "production_authorized": True, + "slice_id": self.candidate.payload["slice_id"], + "candidate_digest": self.candidate.digest, + "prerequisite_bundle_id": self.no_go["bundle_id"], + "python_v1_public_authority_unchanged": True, + "production_mutations": 0, + "parity": {"semantic_mismatches": 0}, + "broker": { + "final_authority": "RUST_CANARY", + "public_writes": 0, + "legacy_writes": 0, + }, + "hold_completed_at_ns": now_ns - 1, + "rollback_manifest_digest": "1" * 64, + } + + def _approval(self, now_ns: int) -> dict: + return { + "schema": "qdl.primary-slice-approval.v1", + "decision": "APPROVE", + "slice_id": self.candidate.payload["slice_id"], + "candidate_digest": self.candidate.digest, + "prerequisite_bundle_id": self.no_go["bundle_id"], + "operator": "phase92-test-operator", + "change_ticket": "QDL-92", + "max_partitions": 1, + "allow_disable_exact_python_slice": True, + "expires_at_ns": now_ns + 60_000_000_000, + } + + def test_current_no_go_cannot_authorize_primary(self): + now_ns = self.no_go["issued_at_ns"] + 1 + result = ProductionPrimaryAuthorizer().authorize( + candidate=self.candidate, + prerequisite_decision=self.no_go, + canary_evidence={}, + approval={}, + expected_bundle_id=self.no_go["bundle_id"], + now_ns=now_ns, + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, "PREREQUISITE_DECISION_NOT_GO") + + def test_rehearsal_is_explicitly_non_production(self): + result = ProductionPrimaryAuthorizer.authorize_isolated_rehearsal( + candidate=self.candidate + ) + self.assertTrue(result.allowed) + self.assertFalse(result.production_authorized) + self.assertEqual(result.mode, PrimaryAuthorizationMode.ISOLATED_REHEARSAL) + + def test_exact_go_canary_hold_and_approval_authorize(self): + now_ns = time.time_ns() + result = ProductionPrimaryAuthorizer().authorize( + candidate=self.candidate, + prerequisite_decision=self._go(now_ns), + canary_evidence=self._canary(now_ns), + approval=self._approval(now_ns), + expected_bundle_id=self.no_go["bundle_id"], + now_ns=now_ns, + ) + self.assertTrue(result.allowed) + self.assertTrue(result.production_authorized) + self.assertEqual(result.reason, "AUTHORIZED") + + def test_canary_and_approval_fail_closed(self): + now_ns = time.time_ns() + cases = ( + ("canary", {"status": "COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED"}, "CANARY_HOLD_NOT_PASSED"), + ("canary", {"production_authorized": False}, "CANARY_NOT_PRODUCTION_AUTHORIZED"), + ("canary", {"hold_completed_at_ns": now_ns + 1}, "CANARY_HOLD_TIME_INVALID"), + ("canary", {"rollback_manifest_digest": "bad"}, "CANARY_ROLLBACK_MANIFEST_INVALID"), + ("approval", {"max_partitions": 2}, "PRIMARY_BLAST_RADIUS_INVALID"), + ("approval", {"allow_disable_exact_python_slice": False}, "PRIMARY_PYTHON_HANDOFF_NOT_APPROVED"), + ("approval", {"expires_at_ns": now_ns}, "PRIMARY_APPROVAL_EXPIRED"), + ) + for target, update, expected in cases: + with self.subTest(expected=expected): + canary = self._canary(now_ns) + approval = self._approval(now_ns) + (canary if target == "canary" else approval).update(update) + result = ProductionPrimaryAuthorizer().authorize( + candidate=self.candidate, + prerequisite_decision=self._go(now_ns), + canary_evidence=canary, + approval=approval, + expected_bundle_id=self.no_go["bundle_id"], + now_ns=now_ns, + ) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + +class Phase92HandoffContractTest(unittest.TestCase): + def checkpoint(self) -> TerminalOwnerCheckpoint: + return TerminalOwnerCheckpoint( + schema="qdl.terminal-owner-checkpoint.v1", + checkpoint_id=str(uuid.uuid4()), + slice_id="production/binance/usdm/perpetual/trade/plan-1/btcusdt", + owner_id="python-primary", + authority_revision=7, + lease_epoch=11, + partition_plan_epoch=1, + source_session_id="session-1", + connection_generation=2, + terminal_watermark=100, + terminal_event_id="event-100", + terminal_payload_sha256="1" * 64, + candidate_digest="2" * 64, + committed_at_ns=1, + ) + + def handoff(self, checkpoint: TerminalOwnerCheckpoint) -> AcceptedHandoff: + return AcceptedHandoff( + schema="qdl.accepted-authority-handoff.v1", + handoff_id=str(uuid.uuid4()), + direction=HandoffDirection.PYTHON_TO_RUST, + checkpoint_digest=checkpoint.digest, + slice_id=checkpoint.slice_id, + old_owner_id=checkpoint.owner_id, + new_owner_id="rust-primary", + expected_state="RUST_CANARY", + new_state="RUST_PRIMARY", + expected_authority_revision=checkpoint.authority_revision, + new_authority_revision=checkpoint.authority_revision + 1, + expected_lease_epoch=checkpoint.lease_epoch, + new_lease_epoch=checkpoint.lease_epoch + 1, + partition_plan_epoch=checkpoint.partition_plan_epoch, + terminal_watermark=checkpoint.terminal_watermark, + first_new_watermark=checkpoint.terminal_watermark + 1, + overlap_start_watermark=90, + overlap_end_watermark=checkpoint.terminal_watermark, + old_event_count=11, + new_event_count=11, + semantic_mismatches=0, + open_gaps=0, + candidate_digest=checkpoint.candidate_digest, + prerequisite_bundle_id=str(uuid.uuid4()), + approved_by="phase92-test", + approved_at_ns=1, + expires_at_ns=2, + ) + + def test_checkpoint_and_handoff_are_deterministic(self): + checkpoint = self.checkpoint() + self.assertEqual(len(checkpoint.digest), 64) + handoff = self.handoff(checkpoint) + self.assertEqual(len(handoff.digest), 64) + self.assertEqual(handoff.first_new_watermark, 101) + + def test_off_by_one_dirty_parity_and_stale_epoch_are_rejected(self): + checkpoint = self.checkpoint() + base = self.handoff(checkpoint) + payload = { + field: getattr(base, field) + for field in base.__dataclass_fields__ + } + for update in ( + {"first_new_watermark": 102}, + {"semantic_mismatches": 1}, + {"open_gaps": 1}, + {"new_event_count": 10}, + {"new_lease_epoch": base.expected_lease_epoch}, + {"new_authority_revision": base.expected_authority_revision + 2}, + ): + with self.subTest(update=update): + candidate = dict(payload) + candidate.update(update) + with self.assertRaises(ValueError): + AcceptedHandoff(**candidate) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase93_certification_contract.py b/tests/test_phase93_certification_contract.py new file mode 100644 index 0000000..e70a423 --- /dev/null +++ b/tests/test_phase93_certification_contract.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import json +import subprocess +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase93CertificationContractTest(unittest.TestCase): + def test_certification_is_fail_closed_and_never_claims_production(self): + source = ( + ROOT / "scripts/phase93_hold_close_certification.py" + ).read_text() + for token in ( + "NO_GO_EXTERNAL", + "PREREQUISITE_DECISION_NOT_GO", + "TEST_CONTROL_PLANE_FIXTURE", + "accelerated_time_is_production_evidence", + "production_hold_started", + "production_rollback_window_closed", + "production_expansions_authorized", + "production_mutations", + "v1_topology", + "v1_health", + ): + self.assertIn(token, source) + self.assertNotIn("qdl_transition_authority", source) + self.assertNotIn("qdl_close_authority_window", source) + + def test_frozen_evidence_reports_only_control_plane_completion(self): + evidence = json.loads( + ( + ROOT + / "upgrade/evidence/phase93-hold-close-expand-certification.json" + ).read_text() + ) + self.assertEqual( + evidence["status"], + "COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED", + ) + self.assertFalse(evidence["production_authorized"]) + self.assertFalse(evidence["production_hold_started"]) + self.assertFalse(evidence["production_rollback_window_closed"]) + self.assertEqual(evidence["production_expansions_authorized"], 0) + self.assertEqual(evidence["production_mutations"], 0) + self.assertEqual( + evidence["control_plane_fixture"]["provenance"], + "TEST_CONTROL_PLANE_FIXTURE", + ) + self.assertFalse( + evidence["control_plane_fixture"][ + "accelerated_time_is_production_evidence" + ] + ) + + def test_runbook_and_make_targets_are_reproducible(self): + runbook = ( + ROOT / "docs/runbooks/phase93-hold-close-expand.md" + ).read_text() + makefile = (ROOT / "Makefile").read_text() + for token in ( + "Starting A Real Hold", + "Closing The Rollback Window", + "Expansion", + "Python Runtime Decommission", + "Incident And Rollback", + "TEST_CONTROL_PLANE_FIXTURE", + ): + self.assertIn(token, runbook) + for target in ( + "phase93-test:", + "phase93-migration:", + "phase93-certify:", + "phase93-clean:", + ): + self.assertIn(target, makefile) + result = subprocess.run( + ["python3", "scripts/phase93_hold_close_certification.py", "--help"], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_control_plane_import_does_not_eagerly_require_alpha_sdk(self): + package = (ROOT / "qdl/canary/__init__.py").read_text() + self.assertNotIn("from qdl.canary.consumer import", package) + self.assertIn("def __getattr__", package) + self.assertIn("_CONSUMER_EXPORTS", package) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase93_hold_close_expand.py b/tests/test_phase93_hold_close_expand.py new file mode 100644 index 0000000..34d0420 --- /dev/null +++ b/tests/test_phase93_hold_close_expand.py @@ -0,0 +1,598 @@ +from __future__ import annotations + +import copy +import json +import unittest +import uuid +from dataclasses import replace +from pathlib import Path + +from qdl.canary.phase93 import ( + AuthorityRegistrySnapshot, + ClosureApproval, + ConsumerCheckpoint, + ConsumerRegistrySnapshot, + DecommissionRequest, + ExpansionManifest, + ExpansionType, + HoldScope, + HoldStatus, + PrimaryHoldEvaluator, + PrimaryHoldIdentity, + PrimaryHoldObservation, + PrimaryHoldPolicy, + ProductionClosureAuthorizer, + RollbackRehearsalEvidence, + assess_decommission, +) +from qdl.certification.prerequisites import CandidateSlice + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase93Fixtures: + def setUp(self) -> None: + self.candidate = CandidateSlice.load( + ROOT / "config/phase9/candidate-slice.yaml" + ) + self.no_go = json.loads( + ( + ROOT + / "upgrade/evidence/phase90c-production-prerequisites.json" + ).read_text() + ) + self.bundle_id = self.no_go["bundle_id"] + self.start = 1_000_000_000 + self.policy = PrimaryHoldPolicy( + minimum_duration_seconds=120, + sample_interval_seconds=60, + max_sample_gap_seconds=60, + max_lag_ms=500, + max_freshness_ms=1_000, + max_queue_depth=1_000, + max_spool_bytes=1_000_000, + max_cpu_percent=80.0, + max_rss_mb=512.0, + ) + self.identity = PrimaryHoldIdentity( + schema="qdl.primary-hold.v1", + hold_id=str(uuid.uuid4()), + slice_id=self.candidate.payload["slice_id"], + candidate_digest=self.candidate.digest, + prerequisite_bundle_id=self.bundle_id, + owner_id="rust-primary", + authority_revision=8, + lease_epoch=12, + partition_plan_epoch=1, + started_at_ns=self.start, + required_until_ns=self.start + 120_000_000_000, + policy_digest=self.policy.digest, + ) + + def observation( + self, + sequence: int, + *, + watermark: int | None = None, + observed_at_ns: int | None = None, + **changes: object, + ) -> PrimaryHoldObservation: + payload = { + "schema": "qdl.primary-hold-observation.v1", + "observation_id": str(uuid.uuid4()), + "hold_id": self.identity.hold_id, + "slice_id": self.identity.slice_id, + "candidate_digest": self.identity.candidate_digest, + "owner_id": self.identity.owner_id, + "authority_revision": self.identity.authority_revision, + "lease_epoch": self.identity.lease_epoch, + "partition_plan_epoch": self.identity.partition_plan_epoch, + "sequence": sequence, + "observed_at_ns": ( + observed_at_ns + if observed_at_ns is not None + else self.start + sequence * 60_000_000_000 + ), + "last_watermark": watermark if watermark is not None else 100 + sequence, + "lag_ms": 10, + "freshness_ms": 20, + "queue_depth": 1, + "spool_bytes": 100, + "cpu_percent": 10.0, + "rss_mb": 64.0, + "registered_consumers": 2, + "healthy_consumers": 2, + "checkpoint_watermark": watermark if watermark is not None else 100 + sequence, + } + payload.update(changes) + return PrimaryHoldObservation(**payload) + + def passing_hold(self, *, scope: HoldScope): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, policy=self.policy, scope=scope + ) + self.assertEqual(evaluator.observe(self.observation(1, watermark=110)), "PASS") + self.assertEqual(evaluator.observe(self.observation(2, watermark=120)), "PASS") + return evaluator.decision( + decision_id=str(uuid.uuid4()), + now_ns=self.identity.required_until_ns, + ) + + def go(self, now_ns: int) -> dict: + payload = copy.deepcopy(self.no_go) + payload.update( + { + "decision": "GO", + "passed": len(payload["gates"]), + "blocked": 0, + "issued_at_ns": now_ns - 1_000_000, + "authority_state": "RUST_SHADOW", + "v1_unchanged": True, + "production_mutations": 0, + } + ) + for gate in payload["gates"]: + gate.update({"passed": True, "reason": "PASS"}) + return payload + + def closure_inputs(self, now_ns: int) -> dict: + hold = self.passing_hold(scope=HoldScope.PRODUCTION) + authority = AuthorityRegistrySnapshot( + schema="qdl.authority-registry-snapshot.v1", + snapshot_id=str(uuid.uuid4()), + slice_id=hold.slice_id, + state="RUST_PRIMARY", + owner_id=hold.owner_id, + authority_revision=hold.authority_revision, + lease_epoch=hold.lease_epoch, + partition_plan_epoch=hold.partition_plan_epoch, + candidate_digest=hold.candidate_digest, + prerequisite_bundle_id=hold.prerequisite_bundle_id, + current_watermark=130, + public_write_allowed=True, + legacy_write_allowed=True, + observed_at_ns=now_ns - 1, + ) + consumers = ConsumerRegistrySnapshot( + schema="qdl.consumer-registry-snapshot.v1", + snapshot_id=str(uuid.uuid4()), + slice_id=hold.slice_id, + authority_revision=hold.authority_revision, + checkpoints=tuple( + ConsumerCheckpoint( + consumer_id=name, + requirement_digest=digest * 64, + contract_major=2, + applied_watermark=130, + checkpointed_watermark=130, + status="READY", + migration_status="COMPLETE", + rollback_ready=True, + ) + for name, digest in (("alpha-a", "a"), ("execution-b", "b")) + ), + observed_at_ns=now_ns - 1, + ) + rollback = RollbackRehearsalEvidence( + schema="qdl.rollback-rehearsal.v1", + rehearsal_id=str(uuid.uuid4()), + slice_id=hold.slice_id, + candidate_digest=hold.candidate_digest, + owner_id=hold.owner_id, + authority_revision=hold.authority_revision, + lease_epoch=hold.lease_epoch, + partition_plan_epoch=hold.partition_plan_epoch, + rollback_manifest_digest=self.candidate.payload[ + "rollback_manifest_digest" + ], + reconciled_through_watermark=130, + rto_ms=500.0, + status="PASS", + production_scope=True, + observed_at_ns=now_ns - 1, + expires_at_ns=now_ns + 60_000_000_000, + ) + approval = ClosureApproval( + schema="qdl.rollback-window-closure-approval.v1", + approval_id=str(uuid.uuid4()), + closure_id=str(uuid.uuid4()), + decision="APPROVE", + slice_id=hold.slice_id, + candidate_digest=hold.candidate_digest, + prerequisite_bundle_id=hold.prerequisite_bundle_id, + hold_id=hold.hold_id, + hold_policy_digest=hold.policy_digest, + operator="phase93-test-operator", + change_ticket="QDL-93", + allow_close_rollback_window=True, + repository_cleanup_approved=False, + approved_at_ns=now_ns - 1, + expires_at_ns=now_ns + 60_000_000_000, + ) + primary = { + "schema": "qdl.phase92.production-primary.v1", + "status": "PRODUCTION_PRIMARY_ACTIVE", + "production_authorized": True, + "slice_id": hold.slice_id, + "candidate_digest": hold.candidate_digest, + "prerequisite_bundle_id": hold.prerequisite_bundle_id, + "authority": { + "state": "RUST_PRIMARY", + "owner_id": hold.owner_id, + "authority_revision": hold.authority_revision, + "lease_epoch": hold.lease_epoch, + "partition_plan_epoch": hold.partition_plan_epoch, + "current_watermark": 130, + }, + } + return { + "candidate": self.candidate, + "prerequisite_decision": self.go(now_ns), + "expected_bundle_id": self.bundle_id, + "primary_evidence": primary, + "hold_decision": hold, + "consumer_registry": consumers, + "authority_registry": authority, + "rollback_evidence": rollback, + "approval": approval, + "now_ns": now_ns, + } + + +class PrimaryHoldEvaluatorTest(Phase93Fixtures, unittest.TestCase): + def test_clean_dense_hold_passes_without_production_authority_in_test_scope(self): + decision = self.passing_hold(scope=HoldScope.TEST_REHEARSAL) + self.assertEqual(decision.status, HoldStatus.PASSED) + self.assertEqual(decision.reason, "PASS") + self.assertFalse(decision.production_authorized) + self.assertEqual(decision.observation_count, 2) + self.assertEqual(decision.terminal_watermark, 120) + self.assertEqual(len(decision.digest), 64) + + def test_incomplete_sparse_out_of_order_and_watermark_regression_fail(self): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + empty = evaluator.decision( + decision_id=str(uuid.uuid4()), now_ns=self.identity.required_until_ns + ) + self.assertEqual(empty.reason, "HOLD_OBSERVATION_MISSING") + + cases = ( + ( + self.observation(2), + "HOLD_SEQUENCE_NOT_CONTIGUOUS", + ), + ( + self.observation( + 1, + observed_at_ns=self.start + 61_000_000_000, + ), + "HOLD_OBSERVATION_GAP_EXCEEDED", + ), + ) + for item, expected in cases: + with self.subTest(expected=expected): + candidate = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + self.assertEqual(candidate.observe(item), expected) + self.assertEqual( + candidate.decision( + decision_id=str(uuid.uuid4()), + now_ns=self.identity.required_until_ns, + ).status, + HoldStatus.BLOCKED, + ) + + regression = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + regression.observe(self.observation(1, watermark=120)) + self.assertEqual( + regression.observe(self.observation(2, watermark=119)), + "HOLD_WATERMARK_REGRESSED", + ) + + def test_every_correctness_and_resource_breach_is_sticky(self): + breaches = { + "semantic_mismatches": "SEMANTIC_MISMATCH", + "open_gaps": "OPEN_GAP", + "duplicate_external_writes": "DUPLICATE_EXTERNAL_WRITE", + "accepted_stale_writer_writes": "ACCEPTED_STALE_WRITER_WRITE", + "authority_ambiguities": "AUTHORITY_AMBIGUITY", + "durable_ack_failures": "DURABLE_ACK_FAILURE", + "projection_mismatches": "PROJECTION_MISMATCH", + "consumer_checkpoint_regressions": "CONSUMER_CHECKPOINT_REGRESSION", + "unexplained_quality_failures": "UNEXPLAINED_QUALITY_FAILURE", + "lag_ms": "LAG_THRESHOLD_EXCEEDED", + "freshness_ms": "FRESHNESS_THRESHOLD_EXCEEDED", + "queue_depth": "QUEUE_THRESHOLD_EXCEEDED", + "spool_bytes": "SPOOL_THRESHOLD_EXCEEDED", + "cpu_percent": "CPU_THRESHOLD_EXCEEDED", + "rss_mb": "RSS_THRESHOLD_EXCEEDED", + } + values = { + "lag_ms": self.policy.max_lag_ms + 1, + "freshness_ms": self.policy.max_freshness_ms + 1, + "queue_depth": self.policy.max_queue_depth + 1, + "spool_bytes": self.policy.max_spool_bytes + 1, + "cpu_percent": self.policy.max_cpu_percent + 1, + "rss_mb": self.policy.max_rss_mb + 1, + } + for field, expected in breaches.items(): + with self.subTest(field=field): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + item = self.observation(1, **{field: values.get(field, 1)}) + self.assertEqual(evaluator.observe(item), expected) + self.assertEqual( + evaluator.observe(self.observation(2)), + "HOLD_ALREADY_BLOCKED", + ) + self.assertEqual( + evaluator.decision( + decision_id=str(uuid.uuid4()), + now_ns=self.identity.required_until_ns, + ).reason, + expected, + ) + + def test_identity_consumer_and_numeric_type_guards(self): + evaluator = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + changed = self.observation(1, owner_id="other-owner") + self.assertEqual( + evaluator.observe(changed), "HOLD_AUTHORITY_IDENTITY_CHANGED" + ) + for change in ( + {"registered_consumers": 0, "healthy_consumers": 0}, + {"healthy_consumers": 1}, + {"checkpoint_watermark": 99, "last_watermark": 100}, + ): + with self.subTest(change=change): + current = PrimaryHoldEvaluator( + identity=self.identity, + policy=self.policy, + scope=HoldScope.TEST_REHEARSAL, + ) + reason = current.observe(self.observation(1, **change)) + self.assertNotEqual(reason, "PASS") + with self.assertRaises(ValueError): + self.observation(1, open_gaps=True) + + +class ProductionClosureTest(Phase93Fixtures, unittest.TestCase): + def test_current_no_go_and_local_primary_evidence_cannot_close(self): + now_ns = self.no_go["issued_at_ns"] + 1 + values = self.closure_inputs(now_ns) + values["prerequisite_decision"] = self.no_go + denied = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(denied.allowed) + self.assertEqual(denied.reason, "PREREQUISITE_DECISION_NOT_GO") + + values = self.closure_inputs(now_ns) + values["primary_evidence"] = json.loads( + ( + ROOT + / "upgrade/evidence/phase92-bounded-primary-certification.json" + ).read_text() + ) + denied = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(denied.allowed) + self.assertEqual(denied.reason, "PRIMARY_EVIDENCE_SCHEMA_INVALID") + + def test_complete_production_shaped_fixture_authorizes_without_mutation(self): + now_ns = 2_000_000_000_000 + values = self.closure_inputs(now_ns) + result = ProductionClosureAuthorizer().authorize(**values) + self.assertTrue(result.allowed) + self.assertTrue(result.production_authorized) + self.assertIsNotNone(result.closure) + closure = result.closure + assert closure is not None + self.assertEqual(closure.owner_id, "rust-primary") + self.assertEqual(closure.authority_revision, 8) + self.assertEqual(len(closure.digest), 64) + + def test_registry_rollback_and_approval_mismatches_fail_closed(self): + now_ns = 2_000_000_000_000 + cases = [] + + values = self.closure_inputs(now_ns) + values["consumer_registry"] = replace( + values["consumer_registry"], + observed_at_ns=now_ns - 301_000_000_000, + ) + cases.append((values, "REGISTRY_SNAPSHOT_STALE")) + + values = self.closure_inputs(now_ns) + values["primary_evidence"] = copy.deepcopy(values["primary_evidence"]) + values["primary_evidence"]["authority"]["lease_epoch"] = 99 + cases.append((values, "PRIMARY_AUTHORITY_REGISTRY_MISMATCH")) + + values = self.closure_inputs(now_ns) + checkpoint = values["consumer_registry"].checkpoints[0] + values["consumer_registry"] = replace( + values["consumer_registry"], + checkpoints=( + replace( + checkpoint, + applied_watermark=129, + checkpointed_watermark=129, + ), + values["consumer_registry"].checkpoints[1], + ), + ) + cases.append((values, "CONSUMER_CHECKPOINT_BEHIND")) + + values = self.closure_inputs(now_ns) + values["rollback_evidence"] = replace( + values["rollback_evidence"], production_scope=False + ) + cases.append((values, "ROLLBACK_REHEARSAL_NOT_PRODUCTION")) + + values = self.closure_inputs(now_ns) + values["rollback_evidence"] = replace( + values["rollback_evidence"], rollback_manifest_digest="f" * 64 + ) + cases.append((values, "ROLLBACK_MANIFEST_MISMATCH")) + + values = self.closure_inputs(now_ns) + values["approval"] = replace( + values["approval"], hold_policy_digest="e" * 64 + ) + cases.append((values, "CLOSURE_APPROVAL_IDENTITY_MISMATCH")) + + for values, expected in cases: + with self.subTest(expected=expected): + result = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, expected) + + def test_test_scope_hold_cannot_close_production_window(self): + now_ns = 2_000_000_000_000 + values = self.closure_inputs(now_ns) + values["hold_decision"] = self.passing_hold( + scope=HoldScope.TEST_REHEARSAL + ) + result = ProductionClosureAuthorizer().authorize(**values) + self.assertFalse(result.allowed) + self.assertEqual(result.reason, "PRIMARY_HOLD_NOT_PRODUCTION") + + +class ExpansionAndDecommissionTest(Phase93Fixtures, unittest.TestCase): + def parent_closure(self): + values = self.closure_inputs(2_000_000_000_000) + result = ProductionClosureAuthorizer().authorize(**values) + assert result.closure is not None + return result.closure + + def test_each_expansion_requires_distinct_independent_certification(self): + parent = self.parent_closure() + manifests = [] + for index, expansion_type in enumerate(ExpansionType, start=1): + manifest = ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=expansion_type, + candidate_digest=f"{index}" * 64, + scope_digest=format(index + 5, "x") * 64, + partition_plan_epoch=( + parent.partition_plan_epoch + 1 + if expansion_type == ExpansionType.INSTRUMENT_PARTITION + else parent.partition_plan_epoch + ), + created_at_ns=2_100_000_000_000 + index, + ) + manifests.append(manifest) + self.assertEqual( + manifest.status, "INDEPENDENT_CERTIFICATION_REQUIRED" + ) + self.assertFalse(manifest.transitive_evidence_allowed) + self.assertFalse(manifest.public_write_allowed) + self.assertFalse(manifest.legacy_write_allowed) + self.assertIn("rollback", manifest.required_gates) + self.assertIn("exact_frame_parity", manifest.required_gates) + self.assertEqual( + len({item.candidate_digest for item in manifests}), + len(ExpansionType), + ) + + def test_expansion_cannot_reuse_parent_or_weaken_gates(self): + parent = self.parent_closure() + with self.assertRaises(ValueError): + ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=ExpansionType.BBO, + candidate_digest=parent.candidate_digest, + scope_digest="d" * 64, + partition_plan_epoch=1, + created_at_ns=2_100_000_000_000, + ) + with self.assertRaises(ValueError): + ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=ExpansionType.INSTRUMENT_PARTITION, + candidate_digest="c" * 64, + scope_digest="d" * 64, + partition_plan_epoch=parent.partition_plan_epoch, + created_at_ns=2_100_000_000_000, + ) + valid = ExpansionManifest.plan( + expansion_id=str(uuid.uuid4()), + parent=parent, + expansion_type=ExpansionType.BBO, + candidate_digest="c" * 64, + scope_digest="d" * 64, + partition_plan_epoch=1, + created_at_ns=2_100_000_000_000, + ) + with self.assertRaises(ValueError): + replace(valid, required_gates=("rollback",)) + with self.assertRaises(ValueError): + replace(valid, transitive_evidence_allowed=True) + + def test_decommission_requires_zero_dependency_and_explicit_cleanup(self): + base = DecommissionRequest( + schema="qdl.runtime-decommission-request.v1", + request_id=str(uuid.uuid4()), + runtime_id="python-binance-usdm-trade", + owned_slice_ids=(), + rollback_reference_ids=(), + consumer_dependency_ids=(), + all_replacement_windows_closed=True, + repository_cleanup_approved=True, + shared_knowledge_retained=True, + ) + self.assertTrue(assess_decommission(base).allowed) + cases = ( + ( + replace(base, owned_slice_ids=("slice-a",)), + "RUNTIME_STILL_OWNS_SLICES", + ), + ( + replace(base, rollback_reference_ids=("rollback-a",)), + "RUNTIME_STILL_REQUIRED_FOR_ROLLBACK", + ), + ( + replace(base, consumer_dependency_ids=("consumer-a",)), + "RUNTIME_HAS_CONSUMER_DEPENDENCIES", + ), + ( + replace(base, all_replacement_windows_closed=False), + "REPLACEMENT_WINDOWS_NOT_CLOSED", + ), + ( + replace(base, repository_cleanup_approved=False), + "REPOSITORY_CLEANUP_NOT_APPROVED", + ), + ( + replace(base, shared_knowledge_retained=False), + "SHARED_KNOWLEDGE_REMOVAL_FORBIDDEN", + ), + ) + for request, expected in cases: + with self.subTest(expected=expected): + decision = assess_decommission(request) + self.assertFalse(decision.allowed) + self.assertEqual(decision.reason, expected) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phase93_migration_contract.py b/tests/test_phase93_migration_contract.py new file mode 100644 index 0000000..8253523 --- /dev/null +++ b/tests/test_phase93_migration_contract.py @@ -0,0 +1,96 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class Phase93MigrationContractTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.sql = ( + ROOT / "migrations/postgres/0008_phase93_hold_close_expand.sql" + ).read_text() + + def test_migration_is_additive_and_all_evidence_is_append_only(self): + for token in ( + "qdl_primary_holds", + "qdl_primary_hold_observations", + "qdl_primary_hold_decisions", + "qdl_consumer_registry_snapshots", + "qdl_authority_registry_snapshots", + "qdl_rollback_rehearsals", + "qdl_closure_approvals", + "qdl_authority_closures", + "qdl_expansion_candidates", + "qdl_runtime_decommission_decisions", + "Phase 9.3 hold/closure/expansion evidence is append-only", + ): + self.assertIn(token, self.sql) + self.assertNotIn("DROP TABLE", self.sql.upper()) + self.assertNotIn("TRUNCATE", self.sql.upper()) + + def test_hold_observation_and_passing_decision_are_evidence_backed(self): + for token in ( + "hold observation identity mismatch", + "hold observation sequence is not contiguous", + "hold observation time is not monotonic", + "hold observation gap exceeds policy", + "hold watermark regressed", + "passing hold decision is not supported by evidence", + "semantic_mismatches <> 0", + "consumer_checkpoint_regressions <> 0", + "checkpoint_watermark < last_watermark", + ): + self.assertIn(token, self.sql) + + def test_closure_locks_authority_and_frozen_registries_without_update(self): + start = self.sql.index("CREATE OR REPLACE FUNCTION qdl_close_authority_window") + end = self.sql.index("CREATE TABLE IF NOT EXISTS qdl_expansion_candidates", start) + function = self.sql[start:end] + for token in ( + "FOR UPDATE", + "current_row.state <> 'RUST_PRIMARY'", + "authority closure CAS mismatch", + "closure_sha256", + "consumer registry snapshot is invalid", + "authority registry snapshot is invalid", + "rollback rehearsal is invalid", + "closure approval is invalid or expired", + "INSERT INTO qdl_authority_closures", + ): + self.assertIn(token, function) + self.assertNotIn("UPDATE qdl_authority_slices", function) + self.assertNotIn("DELETE FROM", function) + + def test_expansions_are_independent_and_cannot_write(self): + for token in ( + "INDEPENDENT_CERTIFICATION_REQUIRED", + "parent_closure_digest <> closure_row.closure_sha256", + "NOT transitive_evidence_allowed", + "NOT public_write_allowed", + "NOT legacy_write_allowed", + "expansion requires independent capability gates", + "instrument expansion requires a new partition epoch", + "snapshot_delta_sequence", + "final_revision_lineage", + "provider_semantics", + ): + self.assertIn(token, self.sql) + + def test_decommission_allowance_requires_zero_dependency(self): + for token in ( + "owned_slice_count = 0", + "rollback_reference_count = 0", + "consumer_dependency_count = 0", + "all_replacement_windows_closed", + "repository_cleanup_approved", + "shared_knowledge_retained", + ): + self.assertIn(token, self.sql) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phaseb_bar_history_bootstrap.py b/tests/test_phaseb_bar_history_bootstrap.py new file mode 100644 index 0000000..13431c7 --- /dev/null +++ b/tests/test_phaseb_bar_history_bootstrap.py @@ -0,0 +1,406 @@ +from __future__ import annotations + +import asyncio +import json +import tempfile +import time +import unittest +from pathlib import Path +from unittest.mock import AsyncMock, patch + +from qdl.adapters.binance.bar_edge import ( + BinanceBarRawBinding, + fetch_closed_bar_history_raw_envelopes as fetch_binance_history, +) +from qdl.adapters.okx.bar_edge import ( + OkxBarRawBinding, + fetch_closed_bar_history_raw_envelopes as fetch_okx_history, + fetch_latest_closed_bar_raw_envelope as fetch_okx_latest, +) +from qdl.adapters.okx.history import ( + HistoryCoverage, + OkxCandle, + OkxCandleHistory, +) +from qdl.canonical.market import canonicalize_okx_bar +from qdl.canonical.trade import TradeContext +from qdl.runtime.stable_bar_edge import StableBinanceBarEdge +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import ( + StableAcquisitionPlan, + stable_authority_record, +) + + +ROOT = Path(__file__).resolve().parents[1] + + +def _binance_binding() -> BinanceBarRawBinding: + return BinanceBarRawBinding( + market="USDM", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + interval="1m", + subscription_id="binance-usdm-bar", + source_session_id="history-session", + connection_generation=1, + lease_epoch=1, + authority_revision=1, + partition_plan_epoch=1, + adapter_version="binance-rest/2.0.0", + config_revision=1, + instrument_catalog_revision=1, + ) + + +def _binance_row(open_time: int) -> list: + return [ + open_time, "100", "101", "99", "100.5", "10", + open_time + 59_999, "1005", 7, "4", "402", "0", + ] + + +def _okx_binding() -> OkxBarRawBinding: + return OkxBarRawBinding( + market="SWAP", + product_type="PERPETUAL", + native_symbol="BTC-USDT-SWAP", + interval="1m", + subscription_id="okx-swap-bar", + source_session_id="history-session", + connection_generation=1, + lease_epoch=1, + authority_revision=1, + partition_plan_epoch=1, + adapter_version="okx-v5/2.0.0", + config_revision=1, + instrument_catalog_revision=1, + ) + + +class BarHistoryAdapterTests(unittest.TestCase): + def test_binance_history_preserves_native_rows_and_strict_continuity(self): + rows = [_binance_row(value) for value in (60_000, 120_000, 180_000)] + calls = [] + + def fetcher(*args, **kwargs): + calls.append((args, kwargs)) + return {"data": rows} + + values = fetch_binance_history( + _binance_binding(), + limit=3, + now_ms=240_000, + fetcher=fetcher, + sleep=lambda _seconds: None, + ) + self.assertEqual(len(values), 3) + self.assertEqual(len(calls), 1) + payloads = [json.loads(item.raw_frame_bytes) for item in values] + self.assertEqual([item["row"] for item in payloads], rows) + self.assertEqual({item["bar_origin"] for item in payloads}, {"BACKFILLED"}) + self.assertTrue(all(not item.test_provenance for item in values)) + + with self.assertRaisesRegex(RuntimeError, "time gap"): + fetch_binance_history( + _binance_binding(), + limit=2, + now_ms=240_000, + fetcher=lambda *_args, **_kwargs: { + "data": [_binance_row(60_000), _binance_row(180_000)] + }, + sleep=lambda _seconds: None, + ) + + def test_okx_history_preserves_confirmed_native_rows_and_rejects_partial(self): + start = 60_000 + records = tuple( + OkxCandle( + inst_id="BTC-USDT-SWAP", + bar="1m", + price_type="TRADE", + open_ts_ms=start + index * 60_000, + open="100", + high="101", + low="99", + close="100.5", + volume_raw="10", + volume_ccy_raw="0.1", + volume_quote_raw="1005", + confirmed=True, + ) + for index in range(3) + ) + coverage = HistoryCoverage( + requested_start_ms=start, + requested_end_ms=239_999, + observed_min_ts_ms=start, + observed_max_ts_ms=180_000, + complete_left=True, + complete_right=True, + truncated=False, + terminal_reason="REACHED_REQUEST_START", + provider_endpoint="/api/v5/market/history-candles", + ) + + class Client: + async def candles(self, **_kwargs): + return OkxCandleHistory(records, coverage) + + values = asyncio.run(fetch_okx_history( + _okx_binding(), + limit=3, + now_ms=240_000, + history_client=Client(), + )) + payloads = [json.loads(item.raw_frame_bytes) for item in values] + self.assertEqual([int(item["data"][0][0]) for item in payloads], [60_000, 120_000, 180_000]) + self.assertEqual({item["arg"]["channel"] for item in payloads}, {"candle1m"}) + self.assertTrue(all(not item.test_provenance for item in values)) + + partial = HistoryCoverage( + requested_start_ms=start, + requested_end_ms=239_999, + observed_min_ts_ms=120_000, + observed_max_ts_ms=180_000, + complete_left=False, + complete_right=True, + truncated=False, + terminal_reason="PROVIDER_EXHAUSTED", + provider_endpoint="/api/v5/market/history-candles", + ) + + class Partial: + async def candles(self, **_kwargs): + return OkxCandleHistory(records[1:], partial) + + with self.assertRaisesRegex(RuntimeError, "incomplete"): + asyncio.run(fetch_okx_history( + _okx_binding(), + limit=3, + now_ms=240_000, + history_client=Partial(), + )) + + def test_okx_latest_closed_bar_retries_provisional_provider_state(self): + sentinel = object() + sleeps = [] + + async def no_wait(delay): + sleeps.append(delay) + + with patch( + "qdl.adapters.okx.bar_edge.fetch_closed_bar_history_raw_envelopes", + new_callable=AsyncMock, + side_effect=(RuntimeError("provisional"), (sentinel,)), + ) as fetch: + result = asyncio.run(fetch_okx_latest( + _okx_binding(), attempts=2, sleep=no_wait + )) + self.assertIs(result, sentinel) + self.assertEqual(fetch.await_count, 2) + self.assertEqual(sleeps, [0.5]) + + with patch( + "qdl.adapters.okx.bar_edge.fetch_closed_bar_history_raw_envelopes", + new_callable=AsyncMock, + side_effect=RuntimeError("still provisional"), + ): + with self.assertRaisesRegex(RuntimeError, "exhausted attempts=2"): + asyncio.run(fetch_okx_latest( + _okx_binding(), attempts=2, sleep=no_wait + )) + + def test_okx_final_bar_event_identity_is_transport_and_restart_independent(self): + fixture = json.loads( + (ROOT / "tests/fixtures/phase2/okx_bar.json").read_text(encoding="utf-8") + ) + first_context = TradeContext(**fixture["context"]) + second_values = dict(fixture["context"]) + second_values["partition_sequence"] += 99 + second_context = TradeContext(**second_values) + first = canonicalize_okx_bar(fixture["raw"], first_context) + second = canonicalize_okx_bar(fixture["raw"], second_context) + self.assertEqual(first.source_sequence, "1786352340000:1") + self.assertEqual(first.source_sequence, second.source_sequence) + self.assertEqual(first.event_id, second.event_id) + self.assertEqual(first.canonical_payload_hash, second.canonical_payload_hash) + + +class StableBarBootstrapTests(unittest.TestCase): + def setUp(self) -> None: + self.catalog_path = ROOT / "config/v2/stable-source-bindings.yaml" + self.acquisition_path = ROOT / "config/v2/stable-acquisition-bindings.yaml" + self.catalog = StableSourceCatalog.load(self.catalog_path) + self.acquisition = StableAcquisitionPlan.load( + self.acquisition_path, catalog=self.catalog + ) + self.authority = stable_authority_record( + rust_image_digest="a" * 64, + capability_manifest=ROOT / "config/v2/stable-capabilities.yaml", + contract=ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto", + partition_plan=self.acquisition_path.read_bytes(), + effective_at_ns=time.time_ns(), + ) + + def test_bootstrap_publishes_four_real_provider_batches_once(self): + class Envelope: + def __init__(self, venue: str, open_time: int): + payload = ( + {"row": _binance_row(open_time)} + if venue == "BINANCE" + else {"data": [[str(open_time), "1", "1", "1", "1", "1", "1", "1", "1"]]} + ) + self.raw_frame_bytes = json.dumps(payload).encode() + + class Publisher: + def __init__(self): + self.batches = [] + + def publish_many(self, values): + batch = tuple(values) + self.batches.append(batch) + return tuple(range(len(batch))) + + publisher = Publisher() + edge = StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=publisher, + warmup_rows=2, + clock=lambda: 180.0, + ) + with patch( + "qdl.runtime.stable_bar_edge.fetch_binance_history", + return_value=(Envelope("BINANCE", 60_000), Envelope("BINANCE", 120_000)), + ), patch( + "qdl.runtime.stable_bar_edge.fetch_okx_history", + new_callable=AsyncMock, + return_value=(Envelope("OKX", 60_000), Envelope("OKX", 120_000)), + ): + self.assertEqual(edge.bootstrap_history(), 8) + self.assertEqual(edge.bootstrap_history(), 0) + self.assertEqual([len(item) for item in publisher.batches], [2, 2, 2, 2]) + self.assertEqual(len(edge._last_open_ms), 4) + self.assertTrue(edge._history_bootstrapped) + + def test_durable_ack_watermark_skips_overlapping_restart_bootstrap(self): + class Envelope: + def __init__(self, venue: str, open_time: int): + payload = ( + {"row": _binance_row(open_time)} + if venue == "BINANCE" + else {"data": [[str(open_time), "1", "1", "1", "1", "1", "1", "1", "1"]]} + ) + self.raw_frame_bytes = json.dumps(payload).encode() + + class Publisher: + def __init__(self): + self.batches = [] + + def publish_many(self, values): + batch = tuple(values) + self.batches.append(batch) + return tuple(range(len(batch))) + + with tempfile.TemporaryDirectory(prefix="qdl-stable-bar-state-") as directory: + state_path = Path(directory) / "bar-edge.json" + first_publisher = Publisher() + first = StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=first_publisher, + warmup_rows=2, + state_path=state_path, + clock=lambda: 180.0, + ) + with patch( + "qdl.runtime.stable_bar_edge.fetch_binance_history", + return_value=( + Envelope("BINANCE", 60_000), + Envelope("BINANCE", 120_000), + ), + ), patch( + "qdl.runtime.stable_bar_edge.fetch_okx_history", + new_callable=AsyncMock, + return_value=( + Envelope("OKX", 60_000), + Envelope("OKX", 120_000), + ), + ): + self.assertEqual(first.bootstrap_history(), 8) + + persisted = json.loads(state_path.read_text(encoding="utf-8")) + self.assertEqual(persisted["schema"], "qdl.stable-bar-edge-state.v1") + self.assertEqual(set(persisted["last_open_ms"]), set(first._binding_ids)) + self.assertEqual(set(persisted["last_open_ms"].values()), {120_000}) + + restarted = StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=Publisher(), + warmup_rows=2, + state_path=state_path, + clock=lambda: 180.0, + ) + self.assertTrue(restarted._history_bootstrapped) + self.assertEqual(restarted._last_open_ms, first._last_open_ms) + self.assertEqual(restarted.binance_session_id, first.binance_session_id) + with patch( + "qdl.runtime.stable_bar_edge.fetch_binance_history", + side_effect=AssertionError("restart must not overlap bootstrap"), + ), patch( + "qdl.runtime.stable_bar_edge.fetch_okx_history", + new_callable=AsyncMock, + side_effect=AssertionError("restart must not overlap bootstrap"), + ): + self.assertEqual(restarted.bootstrap_history(), 0) + + def test_checkpoint_corruption_and_partial_ack_fail_closed(self): + class Envelope: + def __init__(self, open_time: int): + self.raw_frame_bytes = json.dumps( + {"row": _binance_row(open_time)} + ).encode() + + class MissingAckPublisher: + def publish_many(self, _values): + return () + + with tempfile.TemporaryDirectory(prefix="qdl-stable-bar-state-") as directory: + state_path = Path(directory) / "bar-edge.json" + state_path.write_text("{}", encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "fields are invalid"): + StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=MissingAckPublisher(), + state_path=state_path, + ) + + state_path.unlink() + edge = StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=MissingAckPublisher(), + warmup_rows=1, + state_path=state_path, + clock=lambda: 180.0, + ) + with patch( + "qdl.runtime.stable_bar_edge.fetch_binance_history", + return_value=(Envelope(120_000),), + ): + with self.assertRaisesRegex(RuntimeError, "every Kafka ACK"): + edge.bootstrap_history() + self.assertFalse(state_path.exists()) + self.assertEqual(edge._last_open_ms, {}) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phaseb_stable_deployment.py b/tests/test_phaseb_stable_deployment.py new file mode 100644 index 0000000..99ab11a --- /dev/null +++ b/tests/test_phaseb_stable_deployment.py @@ -0,0 +1,850 @@ +from __future__ import annotations + +import asyncio +import copy +import json +import tempfile +import time +import unittest +from datetime import datetime, timezone +from unittest.mock import AsyncMock, patch +from pathlib import Path + +import yaml + +from qdl.adapters.vn import build_dnse_bar_raw_envelope +from qdl.runtime.stable_bar_edge import StableBinanceBarEdge +from qdl.runtime.stable_vn_edge import StableDnseVendorEdge +from qdl.runtime.stable_catalog import StableSourceCatalog +from scripts.phaseb_prepare_stable_candidate import prepare_candidate + +from qdl.runtime.stable_deployment import ( + STABLE_CORE_WORKER_COUNT, + StableAcquisitionPlan, + stable_authority_record, + write_stable_runtime_bundle, +) + + +ROOT = Path(__file__).resolve().parents[1] +CATALOG_PATH = ROOT / "config/v2/stable-source-bindings.yaml" +ACQUISITION_PATH = ROOT / "config/v2/stable-acquisition-bindings.yaml" + + +class StableDeploymentContractTests(unittest.TestCase): + def setUp(self) -> None: + self.catalog = StableSourceCatalog.load(CATALOG_PATH) + self.acquisition = StableAcquisitionPlan.load( + ACQUISITION_PATH, catalog=self.catalog + ) + self.authority = stable_authority_record( + rust_image_digest="a" * 64, + capability_manifest=ROOT / "config/v2/stable-capabilities.yaml", + contract=ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto", + partition_plan=ACQUISITION_PATH.read_bytes(), + effective_at_ns=time.time_ns(), + ) + + def test_all_catalog_bindings_have_one_capability_truthful_acquisition(self): + self.assertEqual(len(self.catalog.bindings), 16) + self.assertEqual(len(self.acquisition.bindings), 16) + self.assertEqual(self.acquisition.revision, 3) + modes = {item.mode for item in self.acquisition.bindings} + self.assertEqual(modes, {"RUST_NATIVE", "PYTHON_REST", "PYTHON_VENDOR_SDK"}) + native = self.acquisition.native_ingestor_configs( + catalog=self.catalog, authority=self.authority + ) + self.assertEqual( + set(native), + {"binance-usdm", "binance-spot", "okx-swap", "okx-spot"}, + ) + self.assertEqual(sum(len(item["bindings"]) for item in native.values()), 8) + self.assertTrue(all(item["authority"]["mode"] == "RUST_SHADOW" for item in native.values())) + self.assertEqual( + { + item.binding_id + for item in self.acquisition.bindings + if item.mode == "PYTHON_REST" + }, + { + "binance-usdm-btcusdt-bar-1m", + "binance-spot-btcusdt-bar-1m", + "okx-swap-btcusdt-bar-1m", + "okx-spot-btcusdt-bar-1m", + }, + ) + self.assertEqual( + {item["max_inflight_publishes"] for item in native.values()}, {512} + ) + generation_paths = { + item["generation_state_path"] for item in native.values() + } + self.assertEqual(len(generation_paths), 4) + self.assertTrue(all( + value.startswith("/var/lib/qdl-stable/runtime/generations/") + for value in generation_paths + )) + self.assertEqual( + {item["latest_state_flush_ms"] for item in native.values()}, {50} + ) + delivery_by_feed = { + binding["feed"]: binding["delivery_class"] + for item in native.values() + for binding in item["bindings"] + } + self.assertEqual( + delivery_by_feed, + {"TRADE": "LOSSLESS", "QUOTE": "LATEST_STATE"}, + ) + okx_bbo = { + item.binding_id: item.sequence_policy + for item in self.acquisition.bindings + if item.provider_kind == "okx_bbo" + } + self.assertEqual( + okx_bbo, + { + "okx-spot-btcusdt-quote": "NONE", + "okx-swap-btcusdt-quote": "NONE", + }, + ) + + def test_core_bundle_uses_stable_identity_lineage_and_never_enables_public_writes(self): + core = self.acquisition.core_config( + catalog=self.catalog, authority=self.authority + ) + bindings = core["core"]["bindings"] + expected = {item.instrument.instrument_uid for item in self.catalog.bindings} + self.assertEqual({item["instrument_uid"] for item in bindings}, expected) + self.assertEqual( + {item["source_id"] for item in bindings}, + {item.source_id for item in self.catalog.bindings}, + ) + finality_by_source = { + item.source_id: item.require_final_bar for item in self.catalog.bindings + } + self.assertEqual( + {item["source_id"]: item["require_final_bar"] for item in bindings}, + finality_by_source, + ) + self.assertEqual(sum(finality_by_source.values()), 6) + self.assertFalse(core["authority"]["public_write_allowed"]) + self.assertFalse(core["authority"]["legacy_write_allowed"]) + self.assertFalse(core["core"]["allow_test_provenance"]) + self.assertEqual(core["raw_topics"], ["md.raw.stable.v1"]) + workers = [ + self.acquisition.core_config( + catalog=self.catalog, + authority=self.authority, + worker_index=index, + ) + for index in range(1, STABLE_CORE_WORKER_COUNT + 1) + ] + self.assertEqual( + {item["transactional_id"] for item in workers}, + {"qdl-v2-stable-core-001", "qdl-v2-stable-core-002", "qdl-v2-stable-core-003"}, + ) + self.assertEqual( + {item["shard_id"] for item in workers}, + {item["transactional_id"] for item in workers}, + ) + self.assertEqual({item["raw_topics"][0] for item in workers}, {"md.raw.stable.v1"}) + self.assertEqual({json.dumps(item["authority"], sort_keys=True) for item in workers}, { + json.dumps(self.authority, sort_keys=True) + }) + for invalid_index in (0, STABLE_CORE_WORKER_COUNT + 1): + with self.assertRaisesRegex(ValueError, "worker index"): + self.acquisition.core_config( + catalog=self.catalog, + authority=self.authority, + worker_index=invalid_index, + ) + + with tempfile.TemporaryDirectory(prefix="qdl-phaseb-bundle-") as directory: + first = write_stable_runtime_bundle( + Path(directory), + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + ) + second = write_stable_runtime_bundle( + Path(directory), + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + ) + self.assertEqual(first, second) + self.assertEqual( + set(first), + { + "authority.json", + "core.json", + "core-002.json", + "core-003.json", + "ingestor-binance-spot.json", + "ingestor-binance-usdm.json", + "ingestor-okx-spot.json", + "ingestor-okx-swap.json", + }, + ) + persisted = json.loads((Path(directory) / "core.json").read_text()) + self.assertEqual(persisted, core) + persisted_workers = [ + json.loads((Path(directory) / name).read_text()) + for name in ("core.json", "core-002.json", "core-003.json") + ] + self.assertEqual( + len({item["transactional_id"] for item in persisted_workers}), + STABLE_CORE_WORKER_COUNT, + ) + + def test_binance_bar_edge_publishes_each_closed_bar_once(self): + class Publisher: + def __init__(self): + self.batches = [] + + def publish_many(self, values): + batch = tuple(values) + self.batches.append(batch) + return tuple(range(len(batch))) + + class Envelope: + def __init__(self, venue, open_time_ms): + payload = ( + {"row": [open_time_ms, "1", "1", "1", "1", "1", open_time_ms + 59999]} + if venue == "BINANCE" + else {"data": [[str(open_time_ms), "1", "1", "1", "1", "1", "1", "1", "1"]]} + ) + self.raw_frame_bytes = json.dumps(payload).encode() + + publisher = Publisher() + edge = StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=publisher, + clock=lambda: 120.0, + ) + with patch( + "qdl.runtime.stable_bar_edge.fetch_latest_closed_bar_raw_envelope", + side_effect=( + Envelope("BINANCE", 60_000), Envelope("BINANCE", 60_000), + Envelope("BINANCE", 60_000), Envelope("BINANCE", 60_000), + ), + ) as binance_latest, patch( + "qdl.runtime.stable_bar_edge.fetch_okx_latest", + side_effect=( + Envelope("OKX", 60_000), Envelope("OKX", 60_000), + Envelope("OKX", 60_000), Envelope("OKX", 60_000), + ), + ) as okx_latest: + self.assertEqual(edge.run_cycle(), 4) + self.assertEqual(edge.run_cycle(), 0) + self.assertEqual([len(batch) for batch in publisher.batches], [4]) + self.assertEqual( + {call.kwargs["now_ms"] for call in binance_latest.call_args_list}, + {110_000}, + ) + self.assertEqual( + {call.kwargs["now_ms"] for call in okx_latest.call_args_list}, + {110_000}, + ) + + def test_bar_edge_retries_complete_catchup_after_kafka_ack_failure(self): + class Publisher: + def __init__(self): + self.calls = 0 + self.batches = [] + + def publish_many(self, values): + batch = tuple(values) + self.calls += 1 + self.batches.append(batch) + if self.calls == 1: + raise RuntimeError("injected Kafka ACK failure") + return tuple(range(len(batch))) + + class Envelope: + def __init__(self, venue, open_time_ms): + payload = ( + {"row": [ + open_time_ms, "1", "1", "1", "1", "1", + open_time_ms + 59_999, + ]} + if venue == "BINANCE" + else {"data": [[ + str(open_time_ms), "1", "1", "1", "1", + "1", "1", "1", "1", + ]]} + ) + self.raw_frame_bytes = json.dumps(payload).encode() + + publisher = Publisher() + edge = StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=publisher, + clock=lambda: 240.0, + ) + binding_ids = [ + source.binding_id + for source, _acquisition in edge.bindings + edge.okx_bindings + ] + edge._last_open_ms.update({binding_id: 60_000 for binding_id in binding_ids}) + binance_history = ( + Envelope("BINANCE", 120_000), + Envelope("BINANCE", 180_000), + ) + okx_history = ( + Envelope("OKX", 120_000), + Envelope("OKX", 180_000), + ) + with patch( + "qdl.runtime.stable_bar_edge.fetch_latest_closed_bar_raw_envelope", + side_effect=[Envelope("BINANCE", 180_000)] * 4, + ), patch( + "qdl.runtime.stable_bar_edge.fetch_okx_latest", + new_callable=AsyncMock, + side_effect=[Envelope("OKX", 180_000)] * 4, + ), patch( + "qdl.runtime.stable_bar_edge.fetch_binance_history", + side_effect=[binance_history] * 4, + ), patch( + "qdl.runtime.stable_bar_edge.fetch_okx_history", + new_callable=AsyncMock, + side_effect=[okx_history] * 4, + ): + with self.assertRaisesRegex(RuntimeError, "injected Kafka ACK failure"): + edge.run_cycle() + self.assertEqual( + edge._last_open_ms, + {binding_id: 60_000 for binding_id in binding_ids}, + ) + self.assertEqual(edge.run_cycle(), 8) + + self.assertEqual([len(batch) for batch in publisher.batches], [8, 8]) + self.assertEqual( + edge._last_open_ms, + {binding_id: 180_000 for binding_id in binding_ids}, + ) + + def test_bar_edge_rejects_incomplete_catchup_without_advancing_watermark(self): + class Publisher: + def __init__(self): + self.calls = 0 + + def publish_many(self, _values): + self.calls += 1 + return () + + class Envelope: + def __init__(self, venue, open_time_ms): + payload = ( + {"row": [ + open_time_ms, "1", "1", "1", "1", "1", + open_time_ms + 59_999, + ]} + if venue == "BINANCE" + else {"data": [[ + str(open_time_ms), "1", "1", "1", "1", + "1", "1", "1", "1", + ]]} + ) + self.raw_frame_bytes = json.dumps(payload).encode() + + publisher = Publisher() + edge = StableBinanceBarEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=publisher, + clock=lambda: 240.0, + ) + binding_ids = [ + source.binding_id + for source, _acquisition in edge.bindings + edge.okx_bindings + ] + edge._last_open_ms.update({binding_id: 60_000 for binding_id in binding_ids}) + with patch( + "qdl.runtime.stable_bar_edge.fetch_latest_closed_bar_raw_envelope", + side_effect=[Envelope("BINANCE", 180_000)] * 2, + ), patch( + "qdl.runtime.stable_bar_edge.fetch_okx_latest", + new_callable=AsyncMock, + side_effect=[Envelope("OKX", 180_000)] * 2, + ), patch( + "qdl.runtime.stable_bar_edge.fetch_binance_history", + return_value=(Envelope("BINANCE", 180_000),), + ): + with self.assertRaisesRegex(RuntimeError, "not contiguous"): + edge.run_cycle() + + self.assertEqual(publisher.calls, 0) + self.assertEqual( + edge._last_open_ms, + {binding_id: 60_000 for binding_id in binding_ids}, + ) + + def test_dnse_edge_fences_on_queue_pressure_and_bar_keeps_exact_units(self): + class Publisher: + pass + + edge = StableDnseVendorEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=Publisher(), + queue_capacity=1, + ) + trade = type("Trade", (), { + "symbol": "VN30F1M", + "price": "1820.7", + "quantity": "1", + "marketId": "VN30", + "boardId": "G3", + "tradingSessionId": "CONTINUOUS", + "totalVolumeTraded": "12", + })() + edge.on_trade(trade) + edge.on_trade(trade) + self.assertTrue(edge._fatal.is_set()) + self.assertEqual(edge._queue.qsize(), 1) + + source = edge.bar_sources["VN30F1M"] + row = {"t": 1_786_352_340, "o": "1820.7", "h": "1821.2", + "l": "1820.2", "c": "1820.7", "v": "0"} + envelope = build_dnse_bar_raw_envelope( + row, + edge._binding(source), + received_at_ns=1_786_352_400_000_000_000, + ) + websocket_envelope = build_dnse_bar_raw_envelope( + row, + edge._binding(source), + received_at_ns=1_786_352_400_000_000_001, + acquisition_origin="WEBSOCKET_CLOSED", + ) + payload = json.loads(envelope.raw_frame_bytes) + self.assertEqual(payload["interval"], "1m") + self.assertEqual(payload["v"], "0") + self.assertTrue(payload["is_final"]) + self.assertEqual(envelope.native_channel, websocket_envelope.native_channel) + self.assertNotEqual( + envelope.transport_protocol, websocket_envelope.transport_protocol + ) + self.assertNotEqual( + envelope.capture_boundary, websocket_envelope.capture_boundary + ) + self.assertFalse(envelope.test_provenance) + + def test_dnse_history_bootstrap_retries_checkpoints_and_restores(self): + class Publisher: + def __init__(self): + self.batches = [] + self.fail = False + + def publish_many(self, values): + batch = tuple(values) + self.batches.append(batch) + return () if self.fail else tuple(range(len(batch))) + + rows = [ + {"t": value, "o": "100", "h": "101", "l": "99", "c": "100.5", "v": "10"} + for value in (120, 180, 240) + ] + calls = [] + sleeps = [] + + def fetcher(symbol, resolution, start, end): + calls.append((symbol, resolution, start, end)) + if len(calls) == 1: + raise TimeoutError("injected transient DNSE timeout") + return rows + + with tempfile.TemporaryDirectory(prefix="qdl-dnse-state-") as directory: + state_path = Path(directory) / "dnse.json" + publisher = Publisher() + edge = StableDnseVendorEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=publisher, + warmup_rows=2, + history_lookback_days=1, + history_attempts=2, + history_fetcher=fetcher, + state_path=state_path, + clock=lambda: 400.0, + sleep=sleeps.append, + ) + self.assertEqual(edge.bootstrap_history(), 4) + self.assertEqual(edge.bootstrap_history(), 0) + self.assertEqual([len(batch) for batch in publisher.batches], [4]) + self.assertEqual(sleeps, [1]) + self.assertEqual(set(edge._last_bar), set(edge._bar_binding_ids)) + self.assertEqual(state_path.stat().st_mode & 0o777, 0o600) + self.assertTrue(all( + not item.test_provenance + for batch in publisher.batches + for item in batch + )) + + restored = StableDnseVendorEdge( + catalog=self.catalog, + acquisition=self.acquisition, + authority=self.authority, + publisher=publisher, + warmup_rows=2, + history_fetcher=lambda *_args: (_ for _ in ()).throw( + AssertionError("matching checkpoint must avoid REST bootstrap") + ), + state_path=state_path, + clock=lambda: 400.0, + ) + self.assertEqual(restored.bootstrap_history(), 0) + self.assertEqual(restored._last_bar, edge._last_bar) + + def test_dnse_closed_bar_uses_websocket_and_ack_advances_checkpoint(self): + class Publisher: + def __init__(self): + self.batches = [] + + def publish_many(self, values): + batch = tuple(values) + self.batches.append(batch) + return tuple(range(len(batch))) + + rows = [ + {"t": value, "o": "100", "h": "101", "l": "99", "c": "100.5", "v": "10"} + for value in (120, 180, 240) + ] + with tempfile.TemporaryDirectory(prefix="qdl-dnse-state-") as directory: + state_path = Path(directory) / "dnse.json" + publisher = Publisher() + edge = StableDnseVendorEdge( + catalog=self.catalog, acquisition=self.acquisition, + authority=self.authority, publisher=publisher, warmup_rows=2, + history_attempts=1, history_fetcher=lambda *_args: rows, + state_path=state_path, clock=lambda: 400.0, + ) + edge.bootstrap_history() + edge.history_fetcher = lambda *_args: (_ for _ in ()).throw( + AssertionError("live closed BAR must not poll REST") + ) + bar = type("Ohlc", (), { + "symbol": "VN30F1M", "resolution": "1", "time": 300, + "open": "100", "high": "102", "low": "99", "close": "101", + "volume": "12", + })() + edge.on_ohlc_closed(bar) + edge._stopped.set() + edge._publish_worker() + self.assertFalse(edge._fatal.is_set()) + self.assertEqual([len(batch) for batch in publisher.batches], [4, 1]) + live = publisher.batches[-1][0] + self.assertEqual(live.native_channel, "ohlcv/1m") + self.assertNotEqual( + live.transport_protocol, publisher.batches[0][0].transport_protocol + ) + binding_id = edge.bar_sources["VN30F1M"].binding_id + self.assertEqual(edge._last_bar[binding_id][0], 300_000) + restored = StableDnseVendorEdge( + catalog=self.catalog, acquisition=self.acquisition, + authority=self.authority, publisher=publisher, + state_path=state_path, clock=lambda: 400.0, + ) + self.assertEqual(restored._last_bar[binding_id][0], 300_000) + restored.on_ohlc_closed(bar) + self.assertTrue(restored._queue.empty()) + + def test_dnse_run_subscribes_native_closed_bar_without_live_rest_polling(self): + calls = [] + + class Publisher: + def publish_many(self, values): + batch = tuple(values) + return tuple(range(len(batch))) + + def close(self): + calls.append(("close",)) + + class Client: + is_healthy = True + + def __init__(self, **kwargs): + calls.append(( + "init", kwargs["base_url"], kwargs["dispatch_queue_capacity"] + )) + + async def connect(self): + calls.append(("connect",)) + + async def subscribe_trades(self, **kwargs): + calls.append(("trade", kwargs["board_id"], tuple(kwargs["symbols"]))) + + async def subscribe_ohlc_closed(self, **kwargs): + calls.append(("bar", kwargs["resolution"], tuple(kwargs["symbols"]))) + edge.stop() + + async def disconnect(self): + calls.append(("disconnect",)) + + edge = StableDnseVendorEdge( + catalog=self.catalog, acquisition=self.acquisition, + authority=self.authority, publisher=Publisher(), + history_fetcher=lambda *_args: (_ for _ in ()).throw( + AssertionError("restored runtime must not call REST") + ), + ) + edge._history_bootstrapped = True + with patch("qdl.adapters.vn.stable_edge.DNSE_API_KEY", "key"), patch( + "qdl.adapters.vn.stable_edge.DNSE_API_SECRET_KEY", "secret" + ), patch("qdl.adapters.vn.stable_edge.TradingClient", Client): + asyncio.run(edge.run()) + self.assertIn( + ("init", "wss://ws-openapi.dnse.com.vn", 833), calls + ) + self.assertIn(("trade", "G1", ("FPT", "VN30F1M")), calls) + self.assertIn(("trade", "G3", ("FPT", "VN30F1M")), calls) + self.assertIn(("bar", "1", ("FPT", "VN30F1M")), calls) + self.assertEqual(calls[-2:], [("disconnect",), ("close",)]) + + def test_dnse_history_conflict_partial_checkpoint_and_ack_failure_fail_closed(self): + base = {"t": 120, "o": "100", "h": "101", "l": "99", "c": "100", "v": "1"} + conflict = {**base, "c": "101"} + edge = StableDnseVendorEdge( + catalog=self.catalog, acquisition=self.acquisition, authority=self.authority, + publisher=object(), warmup_rows=2, history_attempts=1, + history_fetcher=lambda *_args: [base, conflict], + clock=lambda: 400.0, sleep=lambda _delay: None, + ) + with self.assertRaisesRegex(RuntimeError, "bootstrap exhausted"): + edge._closed_history("VN30F1M") + + partial = StableDnseVendorEdge( + catalog=self.catalog, acquisition=self.acquisition, authority=self.authority, + publisher=object(), warmup_rows=2, history_attempts=1, + history_fetcher=lambda *_args: [base], + clock=lambda: 400.0, sleep=lambda _delay: None, + ) + with self.assertRaisesRegex(RuntimeError, "bootstrap exhausted"): + partial._closed_history("VN30F1M") + + with tempfile.TemporaryDirectory(prefix="qdl-dnse-state-") as directory: + state_path = Path(directory) / "dnse.json" + state_path.write_text(json.dumps({ + "schema": "qdl.stable-dnse-edge-state.v1", + "slice_id": self.authority["slice_id"], + "authority_revision": self.authority["revision"], + "catalog_revision": self.catalog.catalog_revision, + "acquisition_revision": self.acquisition.revision, + "binding_ids": list(edge._bar_binding_ids), + "last_bar": {}, + })) + with self.assertRaisesRegex(RuntimeError, "partial"): + StableDnseVendorEdge( + catalog=self.catalog, acquisition=self.acquisition, + authority=self.authority, publisher=object(), state_path=state_path, + ) + + class MissingAckPublisher: + def publish_many(self, values): + tuple(values) + return () + + no_ack = StableDnseVendorEdge( + catalog=self.catalog, acquisition=self.acquisition, + authority=self.authority, publisher=MissingAckPublisher(), + warmup_rows=1, history_attempts=1, + history_fetcher=lambda *_args: [base], clock=lambda: 400.0, + ) + with self.assertRaisesRegex(RuntimeError, "missed a Kafka ACK"): + no_ack.bootstrap_history() + self.assertEqual(no_ack._last_bar, {}) + + def test_missing_binding_wrong_provider_kind_and_primary_authority_fail_closed(self): + payload = yaml.safe_load(ACQUISITION_PATH.read_text(encoding="utf-8")) + missing = copy.deepcopy(payload) + missing["bindings"].pop() + with tempfile.TemporaryDirectory(prefix="qdl-phaseb-acquisition-") as directory: + path = Path(directory) / "missing.yaml" + path.write_text(yaml.safe_dump(missing, sort_keys=False), encoding="utf-8") + with self.assertRaisesRegex(ValueError, "binding sets differ"): + StableAcquisitionPlan.load(path, catalog=self.catalog) + + wrong = copy.deepcopy(payload) + wrong["bindings"][0]["provider_kind"] = "okx_trade" + path.write_text(yaml.safe_dump(wrong, sort_keys=False), encoding="utf-8") + with self.assertRaisesRegex(ValueError, "provider kind differs"): + StableAcquisitionPlan.load(path, catalog=self.catalog) + + contiguous_bbo = copy.deepcopy(payload) + for item in contiguous_bbo["bindings"]: + if item["provider_kind"] == "okx_bbo": + item["sequence_policy"] = "CONTIGUOUS" + break + path.write_text( + yaml.safe_dump(contiguous_bbo, sort_keys=False), encoding="utf-8" + ) + with self.assertRaisesRegex(ValueError, "replace-only"): + StableAcquisitionPlan.load(path, catalog=self.catalog) + + primary = copy.deepcopy(self.authority) + primary["mode"] = "RUST_PRIMARY" + primary["public_write_allowed"] = True + with self.assertRaisesRegex(ValueError, "not an isolated Rust shadow"): + self.acquisition.core_config(catalog=self.catalog, authority=primary) + + +class StableComposeAndBundleTests(unittest.TestCase): + def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): + raw = (ROOT / "docker-compose.v2-stable.yml").read_text(encoding="utf-8") + compose = yaml.safe_load(raw) + services = compose["services"] + self.assertNotIn("8100", raw) + self.assertNotIn("redis_marketdata", raw) + self.assertTrue(compose["networks"]["stable_internal"]["internal"]) + self.assertFalse(compose["networks"]["stable_ingress"].get("internal", False)) + for name in ("query_v2_1", "query_v2_2", "stream_v2_active", "stream_v2_passive"): + self.assertEqual( + set(services[name]["networks"]), {"stable_internal", "stable_ingress"} + ) + self.assertTrue( + all(str(port).startswith("127.0.0.1:") for port in services[name]["ports"]) + ) + self.assertNotIn("ports", services["projector_v2"]) + self.assertEqual(services["projector_v2"]["networks"], ["stable_internal"]) + self.assertEqual( + compose["x-kafka-env"]["KAFKA_MIN_INSYNC_REPLICAS"], 2 + ) + self.assertEqual( + compose["x-kafka-env"]["KAFKA_DEFAULT_REPLICATION_FACTOR"], 3 + ) + self.assertEqual(compose["x-kafka"]["mem_limit"], "768m") + self.assertEqual( + compose["x-kafka-env"]["KAFKA_HEAP_OPTS"], "-Xms256m -Xmx256m" + ) + kafka_tmpfs = compose["x-kafka"]["tmpfs"] + self.assertEqual(kafka_tmpfs, ["/tmp:rw,nosuid,nodev,exec,size=32m"]) + self.assertNotIn("noexec", kafka_tmpfs[0]) + self.assertIn("stable_tls:/stable-certs:ro", services["projector_v2"]["volumes"]) + self.assertNotIn("/certs:ro", " ".join(services["projector_v2"]["volumes"])) + bar_edge = services["binance_bar_edge"] + self.assertEqual( + bar_edge["environment"]["QDL_STABLE_BAR_SETTLEMENT_DELAY_SECONDS"], + "10", + ) + self.assertEqual( + bar_edge["environment"]["QDL_STABLE_BAR_STATE_PATH"], + "/var/lib/qdl-stable/runtime/stable-crypto-bar-edge.json", + ) + self.assertIn("stable_state:/var/lib/qdl-stable", bar_edge["volumes"]) + self.assertEqual( + bar_edge["depends_on"]["stable_state_init"], + {"condition": "service_completed_successfully"}, + ) + for name in ( + "query_v2_1", "query_v2_2", "stream_v2_active", + "stream_v2_passive", "projector_v2", + ): + with self.subTest(service=name): + self.assertEqual(services[name]["user"], "10001:10001") + self.assertTrue(services[name]["read_only"]) + self.assertIn("ALL", services[name]["cap_drop"]) + self.assertEqual(services[name]["restart"], "no") + self.assertEqual( + set(services["ingestor_okx_swap"]["networks"]), + {"stable_internal", "stable_egress"}, + ) + for name in ( + "ingestor_binance_usdm", "ingestor_binance_spot", + "ingestor_okx_swap", "ingestor_okx_spot", + ): + with self.subTest(native_ingestor=name): + self.assertTrue(services[name]["read_only"]) + self.assertIn( + "stable_state:/var/lib/qdl-stable", services[name]["volumes"] + ) + self.assertEqual( + services[name]["depends_on"], + { + "stable_tls_init": { + "condition": "service_completed_successfully" + }, + "stable_state_init": { + "condition": "service_completed_successfully" + }, + "kafka1": {"condition": "service_healthy"}, + "kafka2": {"condition": "service_healthy"}, + "kafka3": {"condition": "service_healthy"}, + }, + ) + core_names = ("rust_core", "rust_core_2", "rust_core_3") + self.assertLessEqual( + len(core_names), compose["x-kafka-env"]["KAFKA_NUM_PARTITIONS"] + ) + self.assertEqual( + { + services[name]["environment"]["QDL_KAFKA_CLIENT_ID"] + for name in core_names + }, + { + "qdl-v2-stable-core-001", + "qdl-v2-stable-core-002", + "qdl-v2-stable-core-003", + }, + ) + self.assertEqual( + { + services[name]["environment"]["QDL_KAFKA_GROUP_ID"] + for name in core_names + }, + {"qdl-v2-stable-core-v1"}, + ) + for name in core_names: + with self.subTest(service=name): + self.assertEqual( + services[name]["entrypoint"], + ["/usr/local/bin/qdl-realtime-core"], + ) + self.assertIn("stable_tls:/stable-certs:ro", services[name]["volumes"]) + + def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): + with tempfile.TemporaryDirectory(prefix="qdl-phaseb-cert-") as cert_directory: + certs = Path(cert_directory) + (certs / "ca.crt").write_text("ca", encoding="ascii") + for principal in ("phase8-producer", "phase8-core", "phase8-consumer"): + (certs / f"{principal}.crt").write_text("crt", encoding="ascii") + (certs / f"{principal}.key").write_text("key", encoding="ascii") + with tempfile.TemporaryDirectory(prefix="qdl-phaseb-output-") as parent: + output = Path(parent) / "candidate" + with patch( + "scripts.phaseb_prepare_stable_candidate.image_id", + side_effect=("sha256:" + "a" * 64, "sha256:" + "b" * 64), + ): + manifest = prepare_candidate( + rust_image="qdl-rust:test", + python_image="qdl-python:test", + cert_dir=certs, + output_dir=output, + host_cert_dir=Path("/host/qdl/certs"), + host_output_dir=Path("/host/qdl/candidate"), + ) + self.assertFalse(manifest["cutover_authorized"]) + self.assertFalse(manifest["secret_values_recorded"]) + self.assertEqual((output / "stable.env").stat().st_mode & 0o777, 0o600) + env_text = (output / "stable.env").read_text() + self.assertIn("QDL_STABLE_CERT_DIR=/host/qdl/certs", env_text) + self.assertIn( + "QDL_STABLE_RUNTIME_DIR=/host/qdl/candidate/runtime", env_text + ) + public_manifest = (output / "candidate-manifest.json").read_text() + self.assertNotIn("QDL_STABLE_INTERNAL_INGEST_SECRET", public_manifest) + for name in ("core.json", "core-002.json", "core-003.json"): + self.assertTrue((output / f"runtime/{name}").is_file()) + self.assertTrue((output / "identities/projector/client.key").is_file()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phaseb_stable_edge.py b/tests/test_phaseb_stable_edge.py new file mode 100644 index 0000000..a17df9c --- /dev/null +++ b/tests/test_phaseb_stable_edge.py @@ -0,0 +1,1465 @@ +from __future__ import annotations + +import base64 +import hashlib +import json +import os +import tempfile +from urllib.parse import urlsplit +import unittest +from datetime import datetime, timezone +from pathlib import Path + +import httpx +import yaml +from fastapi import FastAPI + +from qdl.canonical.market import ( + canonicalize_binance_usdm_bbo, + canonicalize_binance_usdm_rest_bar, + canonicalize_dnse_bar, + canonicalize_okx_bbo, +) +from qdl.canonical.trade import ( + TradeContext, + canonicalize_binance_usdm_trade, + canonicalize_dnse_trade, +) +from qdl.common.v1 import common_pb2 +from qdl.marketdata.v2 import market_data_pb2 +from qdl.domain.instrument import ( + AssetClass, + InstrumentIdentity, + InstrumentRecord, + ProductType, +) +from qdl.domain.decimal import CanonicalDecimal +from qdl.query import ConsumerGrade, DataRequirement, FeedType +from qdl.projection.stable import ( + InMemoryStableProjectionTarget, + ProjectionCacheMismatch, + ProjectionFenced, + RedisStableProjectionTarget, + StableCompatibilityProjector, + StableProjectionItem, + StableProjectionRecord, +) +from qdl.raw.capture import bind_capture_context, capture_exact_frame +from qdl.replay import GapFreeHandoff, SignedHandoffCursorCodec +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable import StableRuntimeConfig +from qdl.runtime.stable_ingest import StableHttpCanonicalSink, install_stable_canonical_ingest +from qdl.runtime.stable_projector import ( + LocalStableCanonicalSink, + StableProjectorEngine, + supervise_stable_projector, +) +from qdl.runtime.stable_source import ( + StableCatalogCursorScopeValidator, + StableConsumerCursorIssuer, + StableSpoolQueryBackend, +) +from qdl.stream import DurableStreamGateway +from qdl.transport import DurableEvent, SQLiteDurableSpool, SpoolConfig +from qdl.transport.kafka_projector import KafkaProjectorRecord + + +ROOT = Path(__file__).resolve().parents[1] +CATALOG_PATH = ROOT / "config/v2/stable-source-bindings.yaml" +FIXTURE_PATH = ROOT / "tests/fixtures/phase2" + + +def _canonicalizer(name: str): + if "trade" in name: + return canonicalize_dnse_trade if name.startswith("dnse") else canonicalize_binance_usdm_trade + if "bbo" in name: + return canonicalize_okx_bbo if name.startswith("okx") else canonicalize_binance_usdm_bbo + if name.startswith("dnse"): + return canonicalize_dnse_bar + return canonicalize_binance_usdm_rest_bar + + +def _stable_event(catalog: StableSourceCatalog, fixture_name: str, binding_id: str): + fixture = json.loads((FIXTURE_PATH / fixture_name).read_text()) + binding = next(item for item in catalog.bindings if item.binding_id == binding_id) + raw_bytes = json.dumps( + fixture["raw"], sort_keys=True, separators=(",", ":") + ).encode() + context = dict(fixture["context"]) + context.update({ + "instrument_uid": binding.instrument.instrument_uid, + "instrument_id": binding.instrument.instrument_id, + "instrument_revision": binding.instrument.metadata_revision, + "venue": binding.instrument.identity.venue, + "market": binding.instrument.identity.market, + "product_type": binding.instrument.identity.product_type.value, + "native_symbol": binding.instrument.native_symbol, + "provider": binding.provider, + "source_id": binding.source_id, + "source_role": binding.source_role, + "adapter_version": binding.adapter_version, + "normalizer_version": binding.normalizer_version, + "authority_revision": catalog.authority_revision, + "source_session_id": f"phase-b-{binding.binding_id}-session-1", + "connection_generation": 1, + "partition_plan_epoch": 1, + "lease_epoch": 1, + "partition_sequence": 1, + "raw_capture_id": hashlib.sha256(raw_bytes).digest()[:16], + "raw_frame_sha256": hashlib.sha256(raw_bytes).digest(), + }) + context["normalized_at_ns"] = int(context["received_at_ns"]) + 1 + context["published_at_ns"] = int(context["received_at_ns"]) + 2 + return _canonicalizer(fixture_name)(fixture["raw"], TradeContext(**context)) + + +def _stable_pair(catalog: StableSourceCatalog, fixture_name: str, binding_id: str): + fixture = json.loads((FIXTURE_PATH / fixture_name).read_text()) + binding = next(item for item in catalog.bindings if item.binding_id == binding_id) + raw_bytes = json.dumps( + fixture["raw"], sort_keys=True, separators=(",", ":") + ).encode() + received_at_ns = int(fixture["context"]["received_at_ns"]) + raw = capture_exact_frame( + provider=binding.provider, + venue=binding.instrument.identity.venue, + market=binding.instrument.identity.market, + product_type=binding.instrument.identity.product_type.value, + native_symbol=binding.instrument.native_symbol, + native_channel=f"phase-b/{binding.feed.value.lower()}", + subscription_id=binding.binding_id, + source_session_id=f"phase-b-{binding.binding_id}-session-1", + connection_generation=1, + lease_epoch=1, + authority_revision=catalog.authority_revision, + partition_plan_epoch=1, + received_at_ns=received_at_ns, + raw_frame_bytes=raw_bytes, + adapter_version=binding.adapter_version, + config_revision=1, + instrument_catalog_revision=catalog.catalog_revision, + correlation_id=f"phase-b-{binding.binding_id}", + test_provenance=True, + ) + context = dict(fixture["context"]) + context.update({ + "instrument_uid": binding.instrument.instrument_uid, + "instrument_id": binding.instrument.instrument_id, + "instrument_revision": binding.instrument.metadata_revision, + "venue": binding.instrument.identity.venue, + "market": binding.instrument.identity.market, + "product_type": binding.instrument.identity.product_type.value, + "native_symbol": binding.instrument.native_symbol, + "provider": binding.provider, + "source_id": binding.source_id, + "source_role": binding.source_role, + "adapter_version": binding.adapter_version, + "normalizer_version": binding.normalizer_version, + "lease_epoch": 1, + "partition_sequence": 1, + "normalized_at_ns": received_at_ns + 1, + "published_at_ns": received_at_ns + 2, + }) + event = _canonicalizer(fixture_name)( + fixture["raw"], bind_capture_context(TradeContext(**context), raw) + ) + return binding, raw, event + + +class _Broker: + def __init__(self, *, fail_once_offset=None): + self.checkpoints = [] + self.fail_once_offset = fail_once_offset + self.flow_control = [] + self.checkpoint_batches = [] + + def poll(self, timeout_seconds): + del timeout_seconds + return None + + def checkpoint(self, record): + if self.fail_once_offset == (record.topic, record.partition, record.offset): + self.fail_once_offset = None + raise RuntimeError("injected checkpoint failure") + self.checkpoints.append((record.topic, record.partition, record.offset)) + + def checkpoint_many(self, records): + values = tuple(records) + self.checkpoint_batches.append(len(values)) + for record in values: + self.checkpoint(record) + + def pause_canonical(self): + self.flow_control.append("pause") + + def resume_canonical(self): + self.flow_control.append("resume") + + def close(self): + return None + + +def _broker_records(binding, raw, event, *, raw_offset=0, canonical_offset=0): + raw_topic = "qdl.stable.raw.phase-b.v1" + canonical_topic = "qdl.stable.canonical.phase-b.v2" + raw_record = KafkaProjectorRecord( + topic=raw_topic, partition=0, offset=raw_offset, + key=f"{raw.venue}/{raw.market}/{raw.native_symbol}/{raw.native_channel}", + event_id=bytes(raw.capture_id), + payload=raw.SerializeToString(deterministic=True), + accepted_at_ns=raw.received_at_ns, + ) + canonical_record = KafkaProjectorRecord( + topic=canonical_topic, partition=0, offset=canonical_offset, + key=binding.partition_key, event_id=bytes(event.event_id), + payload=event.SerializeToString(deterministic=True), + accepted_at_ns=event.received_at_ns, + ) + return raw_topic, canonical_topic, raw_record, canonical_record + + +def _append(spool, catalog, event): + binding = catalog.binding_for_envelope(event) + durable = DurableEvent( + stream=catalog.canonical_stream, + partition_key=binding.partition_key, + event_id=bytes(event.event_id), + payload=event.SerializeToString(deterministic=True), + accepted_at_ns=event.received_at_ns, + headers={ + "raw_stream": "md.raw.v1.phase-b", + "raw_event_id": event.raw_capture_id.hex(), + }, + ) + return spool.append(durable) + + +def _requirement(binding, *, grade=ConsumerGrade.ALPHA, warmup=1): + return DataRequirement( + instrument_uid=binding.instrument.instrument_uid, + feed=binding.feed, + interval=binding.interval, + consumer_grade=grade, + source_policy_id=binding.source_policy_id, + warmup_limit=warmup, + ) + + +class StableCatalogContractTests(unittest.TestCase): + def test_catalog_covers_equal_source_baseline_with_deterministic_identity(self): + catalog = StableSourceCatalog.load(CATALOG_PATH) + self.assertEqual(len(catalog.bindings), 16) + self.assertEqual( + {(item.instrument.identity.venue, item.feed.value) for item in catalog.bindings}, + { + ("BINANCE", "TRADE"), ("BINANCE", "QUOTE"), ("BINANCE", "BAR"), + ("OKX", "TRADE"), ("OKX", "QUOTE"), ("OKX", "BAR"), + ("HNX", "TRADE"), ("HNX", "BAR"), + ("HOSE", "TRADE"), ("HOSE", "BAR"), + }, + ) + for binding in catalog.bindings: + recreated = InstrumentIdentity.create( + venue=binding.instrument.identity.venue, + market=binding.instrument.identity.market, + product_type=binding.instrument.identity.product_type, + canonical_symbol=binding.instrument.identity.canonical_symbol, + ) + self.assertEqual(recreated.instrument_uid, binding.instrument.instrument_uid) + + def test_catalog_fails_closed_on_uid_unknown_fields_and_incomplete_lineage(self): + payload = yaml.safe_load(CATALOG_PATH.read_text()) + payload["instruments"][0]["instrument_uid"] = "not-deterministic" + with tempfile.TemporaryDirectory() as temp: + candidate = Path(temp) / "catalog.yaml" + candidate.write_text(yaml.safe_dump(payload, sort_keys=False)) + with self.assertRaisesRegex(ValueError, "not deterministic"): + StableSourceCatalog.load(candidate) + catalog = StableSourceCatalog.load(CATALOG_PATH) + event = _stable_event( + catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + event.raw_capture_id = b"" + with self.assertRaisesRegex(ValueError, "provenance is incomplete"): + catalog.binding_for_envelope(event) + + def test_continuous_future_is_explicit_and_dated_future_still_requires_expiry(self): + identity = InstrumentIdentity.create( + venue="HNX", market="VN_DERIVATIVES", + product_type=ProductType.FUTURE, canonical_symbol="VN30F1M", + ) + continuous = InstrumentRecord( + identity=identity, metadata_revision=1, asset_class=AssetClass.DERIVATIVE, + native_symbol="VN30F1M", base_asset="VN30", quote_asset="VND", + settlement_asset="VND", price_tick=CanonicalDecimal.from_text("0.1"), + quantity_step=CanonicalDecimal.from_text("1"), + contract_multiplier=CanonicalDecimal.from_text("100000"), + session_calendar_id="VN_DERIVATIVES", + attributes={"continuous_series": "true"}, + ) + self.assertEqual(continuous.attributes["continuous_series"], "true") + with self.assertRaisesRegex(ValueError, "dated futures require"): + InstrumentRecord( + identity=identity, metadata_revision=1, + asset_class=AssetClass.DERIVATIVE, native_symbol="VN30F2608", + base_asset="VN30", quote_asset="VND", settlement_asset="VND", + price_tick=CanonicalDecimal.from_text("0.1"), + quantity_step=CanonicalDecimal.from_text("1"), + contract_multiplier=CanonicalDecimal.from_text("100000"), + session_calendar_id="VN_DERIVATIVES", + ) + + +class StableQueryContractTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.catalog = StableSourceCatalog.load(CATALOG_PATH) + self.spool = SQLiteDurableSpool(SpoolConfig( + path=Path(self.temp.name) / "stable.sqlite3", + max_records=1000, + max_payload_bytes=8 * 1024 * 1024, + max_storage_bytes=16 * 1024 * 1024, + min_free_disk_bytes=0, + )) + + def tearDown(self): + self.spool.close() + self.temp.cleanup() + + def test_trade_quote_bar_units_and_quality_are_typed(self): + cases = ( + ("binance_usdm_trade.json", "binance-usdm-btcusdt-trade", "BASE_ASSET"), + ("okx_bbo.json", "okx-swap-btcusdt-quote", "CONTRACT"), + ("dnse_derivative_bar.json", "dnse-vn30f1m-bar-1m", "CONTRACT"), + ) + for fixture, binding_id, unit in cases: + with self.subTest(fixture=fixture): + binding = next(x for x in self.catalog.bindings if x.binding_id == binding_id) + event = _stable_event(self.catalog, fixture, binding_id) + _append(self.spool, self.catalog, event) + observed = ( + event.bar.close_time_ns + if event.WhichOneof("payload") == "bar" + else event.source_event_time_ns + ) + backend = StableSpoolQueryBackend( + self.spool, self.catalog, schema_digest="a" * 64, + clock_ns=lambda observed=observed: observed + 1_000_000, + ) + item = backend.latest(_requirement(binding)) + self.assertIsNotNone(item) + self.assertEqual(item.payload.get("quantity_unit") or item.payload.get("volume_unit"), unit) + self.assertEqual(item.contract.contract_version, "2.0.0") + expected_state = "MARKET_CLOSED" if binding.instrument.identity.venue == "HNX" else "LIVE" + self.assertEqual(item.quality.state, expected_state) + self.assertEqual(item.quality.execution_eligible, expected_state == "LIVE") + + def test_query_reads_are_bounded_by_feed_and_requested_history(self): + trade = next( + item + for item in self.catalog.bindings + if item.binding_id == "binance-usdm-btcusdt-trade" + ) + bar = next( + item + for item in self.catalog.bindings + if item.binding_id == "binance-usdm-btcusdt-bar-1m" + ) + trade_event = _stable_event( + self.catalog, "binance_usdm_trade.json", trade.binding_id + ) + bar_event = _stable_event( + self.catalog, "binance_usdm_rest_bar.json", bar.binding_id + ) + _append(self.spool, self.catalog, trade_event) + _append(self.spool, self.catalog, bar_event) + backend = StableSpoolQueryBackend( + self.spool, + self.catalog, + schema_digest="e" * 64, + clock_ns=lambda: max( + trade_event.source_event_time_ns, bar_event.bar.close_time_ns + ) + 1_000_000, + ) + observed_limits = [] + read_tail = self.spool.read_tail + + def tracked_read_tail(**kwargs): + observed_limits.append(kwargs["limit"]) + return read_tail(**kwargs) + + self.spool.read_tail = tracked_read_tail + self.assertIsNotNone(backend.latest(_requirement(trade))) + self.assertIsNotNone(backend.history(_requirement(trade, warmup=1))) + self.assertIsNotNone(backend.latest(_requirement(bar))) + self.assertEqual(observed_limits, [1, 1, 10_000]) + + def test_latest_uses_newest_tail_after_partition_exceeds_query_window(self): + binding = next( + item + for item in self.catalog.bindings + if item.binding_id == "binance-usdm-btcusdt-trade" + ) + template = _stable_event( + self.catalog, "binance_usdm_trade.json", binding.binding_id + ) + base_time = template.source_event_time_ns + tail_path = Path(self.temp.name) / "stable-tail.sqlite3" + with SQLiteDurableSpool(SpoolConfig( + path=tail_path, + max_records=10_010, + max_payload_bytes=32 * 1024 * 1024, + max_storage_bytes=48 * 1024 * 1024, + min_free_disk_bytes=0, + )) as spool: + pending = [] + for index in range(10_001): + event = type(template)() + event.CopyFrom(template) + event.event_id = hashlib.sha256( + f"phase-b-tail-{index}".encode() + ).digest()[:16] + event.source_sequence = str(index + 1) + event.partition_sequence = index + 1 + event.source_event_time_ns = base_time + index * 1_000_000 + event.received_at_ns = event.source_event_time_ns + 1 + event.normalized_at_ns = event.source_event_time_ns + 2 + event.published_at_ns = event.source_event_time_ns + 3 + pending.append(DurableEvent( + stream=self.catalog.canonical_stream, + partition_key=binding.partition_key, + event_id=bytes(event.event_id), + payload=event.SerializeToString(deterministic=True), + accepted_at_ns=event.received_at_ns, + headers={"schema": "qdl.marketdata.trade/2"}, + )) + if len(pending) == 1000: + spool.append_many(pending) + pending = [] + spool.append_many(pending) + expected_time = base_time + 10_000 * 1_000_000 + backend = StableSpoolQueryBackend( + spool, + self.catalog, + schema_digest="f" * 64, + clock_ns=lambda: expected_time + 1_000_000, + ) + latest = backend.latest(_requirement(binding)) + self.assertIsNotNone(latest) + self.assertEqual(latest.observed_at_ns, expected_time) + self.assertEqual(latest.watermark_offset, 10_001) + self.assertEqual(latest.quality.state, "LIVE") + + def test_crypto_gap_blocks_but_vn_sparse_minutes_are_not_fabricated(self): + crypto = next(x for x in self.catalog.bindings if x.binding_id == "binance-usdm-btcusdt-bar-1m") + first = _stable_event(self.catalog, "binance_usdm_rest_bar.json", crypto.binding_id) + second = type(first)() + second.CopyFrom(first) + second.bar.open_time_ns += 120 * 1_000_000_000 + second.bar.close_time_ns += 120 * 1_000_000_000 + second.source_event_time_ns += 120 * 1_000_000_000 + second.source_sequence = "gap-second" + second.event_id = hashlib.sha256(b"phase-b-gap-second").digest()[:16] + _append(self.spool, self.catalog, first) + _append(self.spool, self.catalog, second) + backend = StableSpoolQueryBackend( + self.spool, self.catalog, schema_digest="b" * 64, + clock_ns=lambda: second.bar.close_time_ns + 1_000_000, + ) + self.assertTrue(backend.latest(_requirement(crypto, warmup=2)).quality.gap_open) + + vn = next(x for x in self.catalog.bindings if x.binding_id == "dnse-vn30f1m-bar-1m") + vn_event = _stable_event(self.catalog, "dnse_derivative_bar.json", vn.binding_id) + _append(self.spool, self.catalog, vn_event) + closed_ns = int(datetime(2026, 8, 22, 3, tzinfo=timezone.utc).timestamp() * 1e9) + backend = StableSpoolQueryBackend( + self.spool, self.catalog, schema_digest="c" * 64, + clock_ns=lambda: closed_ns, + ) + item = backend.latest(_requirement(vn)) + self.assertEqual(item.quality.state, "MARKET_CLOSED") + self.assertFalse(item.quality.gap_open) + self.assertFalse(item.quality.execution_eligible) + + def test_history_cursor_is_signed_and_bound_to_consumer(self): + binding = next(x for x in self.catalog.bindings if x.binding_id == "binance-usdm-btcusdt-trade") + event = _stable_event(self.catalog, "binance_usdm_trade.json", binding.binding_id) + _append(self.spool, self.catalog, event) + backend = StableSpoolQueryBackend( + self.spool, self.catalog, schema_digest="d" * 64, + clock_ns=lambda: event.source_event_time_ns + 1_000_000, + ) + handoff = GapFreeHandoff( + self.spool, + SignedHandoffCursorCodec({"stable-k1": b"s" * 32}, active_key_id="stable-k1"), + checkpoint_ttl_seconds=3600, + ) + issuer = StableConsumerCursorIssuer(handoff, self.catalog, ttl_seconds=3600) + requirement = _requirement(binding) + history = issuer.bind_history( + requirement, backend.history(requirement), consumer_id="alpha-binance-paper", + ) + replay = handoff.replay( + token=history.stream_cursor, + consumer_id="alpha-binance-paper", + stream=binding.canonical_stream, + partition_key=binding.partition_key, + limit=10, + ) + self.assertEqual(replay, []) + with self.assertRaises(ValueError): + handoff.replay( + token=history.stream_cursor, + consumer_id="different-consumer", + stream=binding.canonical_stream, + partition_key=binding.partition_key, + limit=10, + ) + + +class StableCursorScopeValidatorTests(unittest.TestCase): + def setUp(self): + self.catalog = StableSourceCatalog.load(CATALOG_PATH) + self.binding = next( + item for item in self.catalog.bindings + if item.binding_id == "binance-usdm-btcusdt-trade" + ) + self.requirement = _requirement(self.binding) + self.validator = StableCatalogCursorScopeValidator(self.catalog) + + def test_unified_canonical_stream_matches_exact_catalog_binding(self): + self.validator.validate( + self.requirement, + stream=self.binding.canonical_stream, + partition_key=self.binding.partition_key, + ) + + def test_wrong_stream_partition_and_policy_fail_closed(self): + invalid_scopes = ( + (f"{self.binding.canonical_stream}.trade", self.binding.partition_key), + (self.binding.canonical_stream, self.binding.partition_key + "-other"), + ( + self.binding.canonical_stream, + self.binding.partition_key.replace("/trade/", "/quote/"), + ), + ) + for stream, partition_key in invalid_scopes: + with self.subTest(stream=stream, partition_key=partition_key): + with self.assertRaisesRegex(ValueError, "stable binding"): + self.validator.validate( + self.requirement, stream=stream, partition_key=partition_key + ) + + wrong_policy = DataRequirement( + instrument_uid=self.requirement.instrument_uid, + feed=self.requirement.feed, + interval=self.requirement.interval, + consumer_grade=self.requirement.consumer_grade, + source_policy_id="wrong-policy", + warmup_limit=self.requirement.warmup_limit, + ) + with self.assertRaisesRegex(ValueError, "no matching stable binding"): + self.validator.validate( + wrong_policy, + stream=self.binding.canonical_stream, + partition_key=self.binding.partition_key, + ) + + +class StableProjectorRecoveryTests(unittest.IsolatedAsyncioTestCase): + async def asyncSetUp(self): + self.temp = tempfile.TemporaryDirectory() + self.catalog = StableSourceCatalog.load(CATALOG_PATH) + self.spool = SQLiteDurableSpool(SpoolConfig( + path=Path(self.temp.name) / "projector.sqlite3", + max_records=1000, + max_payload_bytes=8 * 1024 * 1024, + max_storage_bytes=16 * 1024 * 1024, + min_free_disk_bytes=0, + )) + handoff = GapFreeHandoff( + self.spool, + SignedHandoffCursorCodec({"k": b"k" * 32}, active_key_id="k"), + checkpoint_ttl_seconds=3600, + ) + self.gateway = DurableStreamGateway(handoff=handoff, sink=self.spool) + + async def asyncTearDown(self): + self.spool.close() + self.temp.cleanup() + + def engine(self, broker, target, raw_topic, canonical_topic): + return StableProjectorEngine( + broker=broker, spool=self.spool, catalog=self.catalog, + canonical_topic=canonical_topic, raw_topics=(raw_topic,), + sink=LocalStableCanonicalSink(self.gateway, self.spool), + projector=StableCompatibilityProjector(self.catalog), target=target, + max_pending_records=10, max_pending_bytes=1024 * 1024, + ) + + async def test_supervisor_recreates_poisoned_generation_with_bounded_backoff(self): + stopped = [False] + sleeps = [] + active = [] + brokers = [] + generations = ["fail", "recover"] + + class Broker: + def __init__(self, fail_close): + self.fail_close = fail_close + self.closed = 0 + + def close(self): + self.closed += 1 + if self.fail_close: + raise RuntimeError("injected poisoned close") + + class Engine: + def __init__(self, outcome): + self.outcome = outcome + + async def run_once(self, timeout_seconds): + self.assert_timeout = timeout_seconds + if self.outcome == "fail": + raise RuntimeError("injected asynchronous checkpoint failure") + stopped[0] = True + return True + + def factory(): + outcome = generations.pop(0) + broker = Broker(outcome == "fail") + brokers.append(broker) + return broker, Engine(outcome) + + async def sleep(delay): + sleeps.append(delay) + + await supervise_stable_projector( + broker_factory=factory, + should_stop=lambda: stopped[0], + on_broker=active.append, + sleep=sleep, + ) + self.assertEqual(len(brokers), 2) + self.assertEqual([broker.closed for broker in brokers], [1, 1]) + self.assertEqual(sleeps, [0.25]) + self.assertEqual(active, [brokers[0], None, brokers[1], None]) + + async def test_canonical_before_raw_waits_and_checkpoints_after_all_downstreams(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept(canonical_record) + self.assertEqual(engine.stats.pending_canonical, 1) + self.assertEqual(broker.checkpoints, []) + await engine.accept(raw_record) + self.assertEqual(engine.stats.pending_canonical, 0) + self.assertEqual( + broker.checkpoints, + [(raw_topic, 0, 0), (canonical_topic, 0, 0)], + ) + self.assertIn("trade:price:BTCUSDT", target.latest) + self.assertIn("trade:price:binance_usdm:BTCUSDT", target.latest) + self.assertEqual(len(target.publications), 2) + + async def test_embedded_raw_lineage_projects_without_raw_cache_rows(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + inline = KafkaProjectorRecord( + topic=canonical_record.topic, + partition=canonical_record.partition, + offset=canonical_record.offset, + key=canonical_record.key, + event_id=canonical_record.event_id, + payload=canonical_record.payload, + accepted_at_ns=canonical_record.accepted_at_ns, + raw_provider_envelope=raw_record.payload, + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = StableProjectorEngine( + broker=broker, + spool=self.spool, + catalog=self.catalog, + canonical_topic=canonical_topic, + raw_topics=(), + sink=LocalStableCanonicalSink(self.gateway, self.spool), + projector=StableCompatibilityProjector(self.catalog), + target=target, + max_pending_records=10, + max_pending_bytes=1024 * 1024, + ) + await engine.accept(inline) + self.assertEqual(engine.stats.raw_committed, 0) + self.assertEqual(engine.stats.canonical_committed, 1) + self.assertIsNone( + self.spool.find_event(stream=raw_topic, event_id=raw_record.event_id) + ) + self.assertEqual( + broker.checkpoints, [(canonical_topic, 0, canonical_record.offset)] + ) + self.assertIn("trade:price:BTCUSDT", target.latest) + + async def test_missing_embedded_lineage_fails_closed_without_raw_subscription(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + _raw_topic, canonical_topic, _raw_record, canonical_record = _broker_records( + binding, raw, event + ) + broker = _Broker() + engine = StableProjectorEngine( + broker=broker, + spool=self.spool, + catalog=self.catalog, + canonical_topic=canonical_topic, + raw_topics=(), + sink=LocalStableCanonicalSink(self.gateway, self.spool), + projector=StableCompatibilityProjector(self.catalog), + target=InMemoryStableProjectionTarget(), + max_pending_records=10, + max_pending_bytes=1024 * 1024, + ) + with self.assertRaisesRegex(ValueError, "missing private Kafka raw lineage"): + await engine.accept(canonical_record) + self.assertEqual(broker.checkpoints, []) + + async def test_canonical_backpressure_pauses_before_hard_bound_and_resumes_after_raw(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + waiting = tuple( + KafkaProjectorRecord( + topic=canonical_record.topic, + partition=canonical_record.partition, + offset=offset, + key=canonical_record.key, + event_id=canonical_record.event_id, + payload=canonical_record.payload, + accepted_at_ns=canonical_record.accepted_at_ns + offset, + ) + for offset in range(8) + ) + + await engine.accept_many(waiting) + self.assertEqual(engine.stats.pending_canonical, 8) + self.assertEqual(broker.flow_control, ["pause"]) + self.assertEqual(broker.checkpoints, []) + + await engine.accept(raw_record) + self.assertEqual(engine.stats.pending_canonical, 0) + self.assertEqual(broker.flow_control, ["pause", "resume"]) + self.assertEqual( + broker.checkpoints, + [(raw_topic, 0, 0)] + + [(canonical_topic, 0, offset) for offset in range(8)], + ) + + async def test_canonical_backpressure_keeps_hard_bound_when_pause_is_violated(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, _raw_record, canonical_record = _broker_records( + binding, raw, event + ) + broker = _Broker() + engine = self.engine( + broker, InMemoryStableProjectionTarget(), raw_topic, canonical_topic + ) + records = tuple( + KafkaProjectorRecord( + topic=canonical_record.topic, + partition=canonical_record.partition, + offset=offset, + key=canonical_record.key, + event_id=canonical_record.event_id, + payload=canonical_record.payload, + accepted_at_ns=canonical_record.accepted_at_ns + offset, + ) + for offset in range(11) + ) + + await engine.accept_many(records[:10]) + with self.assertRaisesRegex(RuntimeError, "buffer exhausted"): + await engine.accept(records[10]) + self.assertEqual(engine.stats.pending_canonical, 10) + self.assertEqual(broker.flow_control, ["pause"]) + self.assertEqual(broker.checkpoints, []) + + async def test_projector_batches_durability_projection_and_checkpoints_in_order(self): + first_binding, first_raw, first_event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + second_binding, second_raw, second_event = _stable_pair( + self.catalog, "okx_bbo.json", "okx-swap-btcusdt-quote" + ) + raw_topic, canonical_topic, raw_first, canonical_first = _broker_records( + first_binding, first_raw, first_event, raw_offset=0, canonical_offset=0 + ) + _, _, raw_second, canonical_second = _broker_records( + second_binding, second_raw, second_event, raw_offset=1, canonical_offset=1 + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + append_calls = [] + original_append_many = self.spool.append_many + + def tracked_append_many(events): + append_calls.append(len(events)) + return original_append_many(events) + + projection_calls = [] + original_apply_many = target.apply_many + + def tracked_apply_many(records): + projection_calls.append(len(records)) + return original_apply_many(records) + + self.spool.append_many = tracked_append_many + target.apply_many = tracked_apply_many + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept_many( + (raw_first, raw_second, canonical_first, canonical_second) + ) + + self.assertEqual(append_calls, [2, 2]) + self.assertEqual(projection_calls, [2]) + self.assertEqual(broker.checkpoint_batches, [2, 2]) + self.assertEqual( + broker.checkpoints, + [ + (raw_topic, 0, 0), + (raw_topic, 0, 1), + (canonical_topic, 0, 0), + (canonical_topic, 0, 1), + ], + ) + self.assertEqual(engine.stats.raw_committed, 2) + self.assertEqual(engine.stats.canonical_committed, 2) + self.assertEqual(engine.stats.pending_canonical, 0) + + async def test_batch_projection_failure_replays_without_premature_checkpoint(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + + def fail_projection(_records): + raise RuntimeError("injected batch projection failure") + + target.apply_many = fail_projection + engine = self.engine(broker, target, raw_topic, canonical_topic) + with self.assertRaisesRegex(RuntimeError, "injected batch projection"): + await engine.accept_many((raw_record, canonical_record)) + self.assertEqual(broker.checkpoints, [(raw_topic, 0, 0)]) + + recovered_broker = _Broker() + recovered_target = InMemoryStableProjectionTarget() + recovered = self.engine( + recovered_broker, recovered_target, raw_topic, canonical_topic + ) + await recovered.accept(canonical_record) + self.assertEqual( + recovered_broker.checkpoints, [(canonical_topic, 0, 0)] + ) + self.assertEqual(recovered.stats.canonical_committed, 1) + self.assertEqual(recovered.stats.pending_canonical, 0) + + async def test_cross_replica_raw_cache_wakes_waiting_canonical_on_idle_poll(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept(canonical_record) + self.assertEqual(engine.stats.pending_canonical, 1) + + # Simulate another projector replica durably storing/checkpointing raw. + self.spool.append(DurableEvent( + stream=raw_topic, + partition_key=raw_record.key, + event_id=raw_record.event_id, + payload=raw_record.payload, + accepted_at_ns=raw_record.accepted_at_ns, + headers={"kafka_partition": "1", "kafka_offset": "7"}, + )) + processed = await engine.run_once(timeout_seconds=0.01) + self.assertFalse(processed) + self.assertEqual(engine.stats.pending_canonical, 0) + self.assertEqual(broker.checkpoints, [(canonical_topic, 0, 0)]) + self.assertEqual(len(target.publications), 2) + + async def test_provenance_only_semantic_duplicate_is_checkpointed_without_fanout(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, first = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept_many((raw_record, first)) + publication_count = len(target.publications) + + replayed = type(event)() + replayed.CopyFrom(event) + replayed.partition_sequence += 99 + replay = KafkaProjectorRecord( + topic=canonical_topic, + partition=0, + offset=1, + key=first.key, + event_id=first.event_id, + payload=replayed.SerializeToString(deterministic=True), + accepted_at_ns=first.accepted_at_ns + 1, + ) + await engine.accept(replay) + + self.assertEqual(len(target.publications), publication_count) + self.assertEqual(engine.stats.canonical_committed, 2) + self.assertEqual(engine.stats.duplicate_projections, 1) + self.assertEqual(broker.checkpoints[-1], (canonical_topic, 0, 1)) + + async def test_bar_decimal_spelling_and_origin_are_semantic_duplicates(self): + binding, raw, event = _stable_pair( + self.catalog, + "binance_usdm_rest_bar.json", + "binance-usdm-btcusdt-bar-1m", + ) + raw_topic, canonical_topic, raw_record, first = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept_many((raw_record, first)) + publication_count = len(target.publications) + + replayed = type(event)() + replayed.CopyFrom(event) + replayed.bar.low.mantissa *= 10 + replayed.bar.low.scale += 1 + replayed.bar.low.source_text += "0" + replayed.bar.origin = ( + common_pb2.BAR_ORIGIN_BACKFILLED + if event.bar.origin != common_pb2.BAR_ORIGIN_BACKFILLED + else common_pb2.BAR_ORIGIN_VENUE_NATIVE + ) + replayed.canonical_payload_hash = hashlib.sha256( + replayed.bar.SerializeToString(deterministic=True) + ).digest() + duplicate = KafkaProjectorRecord( + topic=canonical_topic, + partition=0, + offset=1, + key=first.key, + event_id=first.event_id, + payload=replayed.SerializeToString(deterministic=True), + accepted_at_ns=first.accepted_at_ns + 1, + ) + await engine.accept(duplicate) + + self.assertEqual(len(target.publications), publication_count) + self.assertEqual(engine.stats.duplicate_projections, 1) + self.assertEqual(broker.checkpoints[-1], (canonical_topic, 0, 1)) + + async def test_bar_numeric_change_remains_a_hard_collision(self): + binding, raw, event = _stable_pair( + self.catalog, + "binance_usdm_rest_bar.json", + "binance-usdm-btcusdt-bar-1m", + ) + raw_topic, canonical_topic, raw_record, first = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept_many((raw_record, first)) + + changed = type(event)() + changed.CopyFrom(event) + changed.bar.close.mantissa += 1 + changed.bar.close.source_text = str( + CanonicalDecimal( + changed.bar.close.mantissa, + changed.bar.close.scale, + changed.bar.close.source_text, + ).as_decimal() + ) + changed.canonical_payload_hash = hashlib.sha256( + changed.bar.SerializeToString(deterministic=True) + ).digest() + conflicting = KafkaProjectorRecord( + topic=canonical_topic, + partition=0, + offset=1, + key=first.key, + event_id=first.event_id, + payload=changed.SerializeToString(deterministic=True), + accepted_at_ns=first.accepted_at_ns + 1, + ) + with self.assertRaisesRegex(RuntimeError, "different market semantics"): + await engine.accept(conflicting) + + self.assertEqual(broker.checkpoints[-1], (canonical_topic, 0, 0)) + self.assertEqual(engine.stats.canonical_committed, 1) + + async def test_late_historical_bar_repairs_cache_without_latest_regression(self): + binding = next( + item + for item in self.catalog.bindings + if item.binding_id == "binance-usdm-btcusdt-bar-1m" + ) + fixture = json.loads( + (FIXTURE_PATH / "binance_usdm_rest_bar.json").read_text() + ) + + def pair(open_time_ms): + raw_payload = json.loads(json.dumps(fixture["raw"])) + raw_payload["row"][0] = open_time_ms + raw_payload["row"][6] = open_time_ms + 59_999 + raw_bytes = json.dumps( + raw_payload, sort_keys=True, separators=(",", ":") + ).encode() + received_at_ns = (open_time_ms + 60_001) * 1_000_000 + raw = capture_exact_frame( + provider=binding.provider, + venue=binding.instrument.identity.venue, + market=binding.instrument.identity.market, + product_type=binding.instrument.identity.product_type.value, + native_symbol=binding.instrument.native_symbol, + native_channel="rest-klines/1m", + subscription_id=binding.source_id, + source_session_id="phase-b-late-bar-session", + connection_generation=1, + lease_epoch=1, + authority_revision=self.catalog.authority_revision, + partition_plan_epoch=1, + received_at_ns=received_at_ns, + raw_frame_bytes=raw_bytes, + adapter_version=binding.adapter_version, + config_revision=1, + instrument_catalog_revision=self.catalog.catalog_revision, + correlation_id=f"phase-b-late-bar-{open_time_ms}", + test_provenance=True, + ) + context = dict(fixture["context"]) + context.update({ + "instrument_uid": binding.instrument.instrument_uid, + "instrument_id": binding.instrument.instrument_id, + "instrument_revision": binding.instrument.metadata_revision, + "venue": binding.instrument.identity.venue, + "market": binding.instrument.identity.market, + "product_type": binding.instrument.identity.product_type.value, + "native_symbol": binding.instrument.native_symbol, + "provider": binding.provider, + "source_id": binding.source_id, + "source_role": binding.source_role, + "adapter_version": binding.adapter_version, + "normalizer_version": binding.normalizer_version, + "lease_epoch": 1, + "partition_sequence": 1, + "normalized_at_ns": received_at_ns + 1, + "published_at_ns": received_at_ns + 2, + }) + event = canonicalize_binance_usdm_rest_bar( + raw_payload, bind_capture_context(TradeContext(**context), raw) + ) + return raw, event + + older_raw, older_event = pair(1786352340000) + later_raw, later_event = pair(1786352400000) + raw_topic, canonical_topic, later_raw_record, later_record = _broker_records( + binding, later_raw, later_event + ) + _raw_topic, _canonical_topic, older_raw_record, older_record = _broker_records( + binding, older_raw, older_event, raw_offset=1, canonical_offset=1 + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept_many((later_raw_record, later_record)) + latest_before = dict(target.latest) + + await engine.accept_many((older_raw_record, older_record)) + + self.assertEqual(target.latest, latest_before) + stored = self.spool.read_tail( + stream=self.catalog.canonical_stream, + partition_key=binding.partition_key, + limit=10, + ) + self.assertEqual(len(stored), 2) + opens = sorted( + market_data_pb2.EventEnvelope.FromString( + item.event.payload + ).bar.open_time_ns + for item in stored + ) + self.assertEqual( + opens, + [1786352340000 * 1_000_000, 1786352400000 * 1_000_000], + ) + self.assertEqual(broker.checkpoints[-1], (canonical_topic, 0, 1)) + + async def test_same_event_id_with_changed_market_semantics_fails_closed(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, first = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept_many((raw_record, first)) + + changed = type(event)() + changed.CopyFrom(event) + changed.trade.aggressor_side = ( + common_pb2.AGGRESSOR_SIDE_SELL + if event.trade.aggressor_side == common_pb2.AGGRESSOR_SIDE_BUY + else common_pb2.AGGRESSOR_SIDE_BUY + ) + changed.canonical_payload_hash = hashlib.sha256( + changed.trade.SerializeToString(deterministic=True) + ).digest() + conflicting = KafkaProjectorRecord( + topic=canonical_topic, + partition=0, + offset=1, + key=first.key, + event_id=first.event_id, + payload=changed.SerializeToString(deterministic=True), + accepted_at_ns=first.accepted_at_ns + 1, + ) + with self.assertRaisesRegex( + RuntimeError, "different market semantics" + ): + await engine.accept(conflicting) + + self.assertEqual(broker.checkpoints[-1], (canonical_topic, 0, 0)) + self.assertEqual(engine.stats.canonical_committed, 1) + + async def test_checkpoint_failure_replays_idempotently_without_duplicate_publication(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, first = _broker_records( + binding, raw, event + ) + broker = _Broker(fail_once_offset=(canonical_topic, 0, 0)) + target = InMemoryStableProjectionTarget() + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept(raw_record) + with self.assertRaisesRegex(RuntimeError, "injected checkpoint"): + await engine.accept(first) + self.assertEqual(len(target.publications), 2) + replay = KafkaProjectorRecord( + topic=canonical_topic, partition=0, offset=1, key=first.key, + event_id=first.event_id, payload=first.payload, + accepted_at_ns=first.accepted_at_ns, + ) + await engine.accept(replay) + self.assertEqual(len(target.publications), 2) + self.assertEqual(engine.stats.duplicate_projections, 2) + self.assertEqual( + broker.checkpoints[-2:], + [(canonical_topic, 0, 0), (canonical_topic, 0, 1)], + ) + + async def test_redis_loss_rebuilds_from_kafka_replay_and_spool_raw_lineage(self): + binding, raw, event = _stable_pair( + self.catalog, "okx_bbo.json", "okx-swap-btcusdt-quote" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + first_target = InMemoryStableProjectionTarget() + await self.engine(_Broker(), first_target, raw_topic, canonical_topic).accept(raw_record) + await self.engine(_Broker(), first_target, raw_topic, canonical_topic).accept(canonical_record) + rebuilt_target = InMemoryStableProjectionTarget() + rebuilt = self.engine(_Broker(), rebuilt_target, raw_topic, canonical_topic) + await rebuilt.accept(canonical_record) + canonical_keys = [key for key in rebuilt_target.latest if key.startswith("qdl:stable:v2:")] + self.assertEqual(len(canonical_keys), 1) + self.assertEqual(rebuilt_target.publications, []) + + async def test_alias_policy_prevents_spot_usdm_generic_mixed_writer(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_spot_trade.json", "binance-spot-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + target = InMemoryStableProjectionTarget() + engine = self.engine(_Broker(), target, raw_topic, canonical_topic) + await engine.accept(raw_record) + await engine.accept(canonical_record) + self.assertIn("trade:price:binance_spot:BTCUSDT", target.latest) + self.assertNotIn("trade:price:BTCUSDT", target.latest) + self.assertEqual( + [channel for channel, _ in target.publications], + ["stream:trade:binance_spot:BTCUSDT"], + ) + + async def test_signed_http_sink_commits_to_active_gateway_idempotently(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, _canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + await StableProjectorEngine( + broker=_Broker(), spool=self.spool, catalog=self.catalog, + canonical_topic="unused-canonical", raw_topics=(raw_topic,), + sink=LocalStableCanonicalSink(self.gateway, self.spool), + projector=StableCompatibilityProjector(self.catalog), + target=InMemoryStableProjectionTarget(), + ).accept(raw_record) + app = FastAPI() + secret = b"phase-b-stable-ingest-secret-32b" + install_stable_canonical_ingest( + app, gateway=self.gateway, catalog=self.catalog, + spool=self.spool, secret=secret, + ) + client = httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), base_url="http://localhost" + ) + sink = StableHttpCanonicalSink( + ("http://localhost",), secret, self.spool, client=client + ) + durable = DurableEvent( + stream=self.catalog.canonical_stream, + partition_key=binding.partition_key, + event_id=canonical_record.event_id, + payload=canonical_record.payload, + accepted_at_ns=canonical_record.accepted_at_ns, + headers={ + "raw_stream": raw_topic, + "raw_event_id": raw_record.event_id.hex(), + "raw_provider_envelope": base64.b64encode( + raw_record.payload + ).decode("ascii"), + }, + ) + try: + first = await sink.publish(durable) + second = await sink.publish(durable) + self.assertEqual(first.cursor, second.cursor) + tampered = DurableEvent( + stream=durable.stream, + partition_key=durable.partition_key, + event_id=durable.event_id, + payload=durable.payload, + accepted_at_ns=durable.accepted_at_ns, + headers={ + **durable.headers, + "raw_provider_envelope": base64.b64encode( + b"malformed-private-lineage" + ).decode("ascii"), + }, + ) + with self.assertRaisesRegex( + RuntimeError, "no active stable stream gateway" + ): + await sink.publish(tampered) + rejected = await client.post( + "/internal/v2/canonical/events", content=b"{}", + headers={"X-QDL-Stable-Signature": "sha256=bad"}, + ) + self.assertEqual(rejected.status_code, 401) + finally: + await sink.close() + await client.aclose() + + async def test_stale_projection_epoch_fails_without_broker_checkpoint(self): + binding, raw, event = _stable_pair( + self.catalog, "binance_usdm_trade.json", "binance-usdm-btcusdt-trade" + ) + raw_topic, canonical_topic, raw_record, canonical_record = _broker_records( + binding, raw, event + ) + broker = _Broker() + target = InMemoryStableProjectionTarget() + target.lease_epochs[binding.source_id] = event.lease_epoch + 1 + engine = self.engine(broker, target, raw_topic, canonical_topic) + await engine.accept(raw_record) + with self.assertRaises(ProjectionFenced): + await engine.accept(canonical_record) + self.assertEqual(broker.checkpoints, [(raw_topic, 0, 0)]) + + + +class StableRuntimeBoundaryTests(unittest.TestCase): + def environment(self, root): + return { + "QDL_ENVIRONMENT": "paper", + "QDL_CONFIG_REVISION": "phase-b-test-1", + "QDL_STABLE_AUTHORITY_MODE": "RUST_SHADOW", + "QDL_STABLE_AUTHORITY_REVISION": "1", + "QDL_STABLE_SCHEMA_DIGEST": "e" * 64, + "QDL_STABLE_STATE_DIR": str(root / "state"), + "QDL_STABLE_DURABLE_STATE_DIR": str(root / "durable"), + "QDL_STABLE_AUDIT_PATH": str(root / "state" / "audit.jsonl"), + "QDL_STABLE_CONSUMER_MANIFESTS": str(root / "consumer.yaml"), + "QDL_STABLE_SOURCE_BINDINGS": str(CATALOG_PATH), + "QDL_STABLE_INTERNAL_INGEST_SECRET": "i" * 32, + "QDL_STABLE_REDIS_URL": "redis://qdl-stable-redis:6379/0", + "QDL_STABLE_REDIS_PREFIX": "qdl:stable:v2:paper:test", + "QDL_STABLE_CONSUMER_GROUP": "qdl-stable-test", + "QDL_STABLE_CURSOR_KEYS_JSON": json.dumps({"stable-k1": "c" * 32}), + "QDL_STABLE_CURSOR_ACTIVE_KEY_ID": "stable-k1", + } + + def test_projector_sink_accepts_only_declared_internal_stream_roles(self): + secret = b"phase-b-stable-ingest-secret-32b" + client = object() + sink = StableHttpCanonicalSink( + ("http://stream_v2_active:8200", "http://stream_v2_passive:8200"), + secret, object(), client=client, + ) + self.assertIs(sink.client, client) + with self.assertRaisesRegex(ValueError, "configuration is invalid"): + StableHttpCanonicalSink( + ("https://stream_v2_active:8200",), secret, object(), client=client + ) + with self.assertRaisesRegex(ValueError, "configuration is invalid"): + StableHttpCanonicalSink( + ("http://external.example:8200",), secret, object(), client=client + ) + + def test_query_role_is_isolated_and_projector_dependencies_fail_closed(self): + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + values = self.environment(root) + config = StableRuntimeConfig.from_environment("query_v2", values) + manifest = config.public_manifest() + self.assertEqual(manifest["contract_version"], "2.0.0") + self.assertEqual(manifest["authority"], "RUST_SHADOW") + self.assertFalse(manifest["writes_current_v1_redis"]) + with self.assertRaisesRegex(ValueError, "Kafka/stream dependencies"): + StableRuntimeConfig.from_environment("projector_v2", values) + values["QDL_STABLE_AUTHORITY_MODE"] = "PRIMARY" + with self.assertRaisesRegex(ValueError, "must remain RUST_SHADOW"): + StableRuntimeConfig.from_environment("query_v2", values) + + +@unittest.skipUnless(os.getenv("QDL_PHASEB_REDIS_URL"), "isolated Redis is not configured") +class StableRedisProjectionIntegrationTests(unittest.TestCase): + namespace = "qdl:test:phaseb:stable:v2" + symbol = "PHASEBTESTBTCUSDT" + + def test_atomic_ttl_publication_duplicate_and_fencing(self): + import redis + + url = os.environ["QDL_PHASEB_REDIS_URL"] + if urlsplit(url).hostname != "qdl-phaseb-redis-test": + self.fail("Phase B Redis integration test requires its named disposable container") + client = redis.Redis.from_url(url) + current = f"trade:price:{self.symbol}" + last = f"trade:price:last:{self.symbol}" + channel = f"stream:trade:{self.symbol}" + scoped = [current, last, *client.scan_iter(match=f"{self.namespace}:*")] + if scoped: + client.delete(*scoped) + pubsub = client.pubsub(ignore_subscribe_messages=True) + pubsub.subscribe(channel) + pubsub.get_message(timeout=1) + try: + record = StableProjectionRecord( + partition_key="phase-b/test/partition", + offset=1, + event_id_hex="ab" * 16, + shard_id="phase-b-test-shard", + lease_epoch=2, + items=( + StableProjectionItem(f"{self.namespace}:latest:test", b"canonical"), + StableProjectionItem(current, b"legacy", 60), + StableProjectionItem(last, b"legacy"), + ), + publications=((channel, b"legacy"),), + ) + target = RedisStableProjectionTarget( + client, namespace=self.namespace, dedicated_database=True + ) + cache_id = "12" * 16 + target.bind_cache(cache_id, initialize_if_missing=True) + self.assertTrue(target.cache_is_bound()) + self.assertTrue(target.apply(record)) + self.assertGreater(client.ttl(current), 0) + self.assertEqual(client.ttl(last), -1) + message = pubsub.get_message(timeout=1) + self.assertIsNotNone(message) + self.assertEqual(message["channel"].decode(), channel) + self.assertFalse(target.apply(record)) + self.assertIsNone(pubsub.get_message(timeout=0.2)) + fenced = StableProjectionRecord( + partition_key=record.partition_key, + offset=2, + event_id_hex=record.event_id_hex, + shard_id=record.shard_id, + lease_epoch=1, + items=record.items, + publications=record.publications, + ) + with self.assertRaises(ProjectionFenced): + target.apply(fenced) + + mismatched = RedisStableProjectionTarget( + client, namespace=self.namespace, dedicated_database=True + ) + with self.assertRaises(ProjectionCacheMismatch): + mismatched.bind_cache("34" * 16, initialize_if_missing=False) + + client.delete(target.cache_identity_key) + self.assertFalse(target.cache_is_bound()) + changed = StableProjectionRecord( + partition_key=record.partition_key, + offset=3, + event_id_hex="cd" * 16, + shard_id=record.shard_id, + lease_epoch=2, + items=record.items, + publications=record.publications, + ) + with self.assertRaises(ProjectionCacheMismatch): + target.apply(changed) + with self.assertRaises(ProjectionCacheMismatch): + target.bind_cache(cache_id, initialize_if_missing=False) + finally: + pubsub.close() + scoped = [current, last, *client.scan_iter(match=f"{self.namespace}:*")] + if scoped: + client.delete(*scoped) + client.close() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phaseb_stable_rebuild.py b/tests/test_phaseb_stable_rebuild.py new file mode 100644 index 0000000..a086720 --- /dev/null +++ b/tests/test_phaseb_stable_rebuild.py @@ -0,0 +1,165 @@ +from __future__ import annotations + +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from scripts.rebuild_v2_stable_projection_cache import ( + CACHE_FILES, + CANONICAL_TOPIC, + CONFIRM_TOKEN, + EXPECTED_CANONICAL_PARTITIONS, + MAX_ACCEPTED_LAG, + PROJECT_NAME, + PROJECTOR_GROUP, + QUERY_SERVICES, + STOP_SERVICES, + STREAM_SERVICES, + _start_services, + _validate_project, + compose_command, + execute_rebuild, + lag_sample_acceptable, + parse_canonical_lag, + rebuild_plan, + require_authorization, +) + + +class StableProjectionCacheRebuildTests(unittest.TestCase): + def test_plan_is_exact_isolated_and_v1_safe(self): + env = Path("/tmp/stable.env") + plan = rebuild_plan(env) + self.assertEqual(plan["project"], PROJECT_NAME) + self.assertEqual(plan["stop_services"], list(STOP_SERVICES)) + self.assertEqual(plan["delete_files"], list(CACHE_FILES)) + self.assertEqual(plan["reset_group"], PROJECTOR_GROUP) + self.assertEqual(plan["reset_topic"], CANONICAL_TOPIC) + self.assertEqual( + plan["lag_gate"]["expected_partitions"], + EXPECTED_CANONICAL_PARTITIONS, + ) + self.assertEqual( + plan["lag_gate"]["max_total_records"], + MAX_ACCEPTED_LAG, + ) + self.assertEqual( + plan["start_order"], + [list(STREAM_SERVICES), ["projector_v2"], list(QUERY_SERVICES)], + ) + self.assertFalse(plan["touches_v1"]) + self.assertFalse(plan["apply"]) + + def test_apply_requires_exact_confirmation(self): + require_authorization(apply=False, confirm=None) + with self.assertRaisesRegex(ValueError, CONFIRM_TOKEN): + require_authorization(apply=True, confirm=None) + with self.assertRaisesRegex(ValueError, CONFIRM_TOKEN): + require_authorization(apply=True, confirm="WRONG") + require_authorization(apply=True, confirm=CONFIRM_TOKEN) + + def test_compose_command_is_pinned_to_stable_manifest(self): + command = compose_command(Path("/tmp/stable.env"), "config") + self.assertEqual(command[:2], ["docker", "compose"]) + self.assertIn("docker-compose.v2-stable.yml", command[5]) + self.assertEqual(command[-1], "config") + self.assertNotIn("docker-compose.yml", command[5]) + + def test_recovery_starts_roles_without_dependency_traversal(self): + env = Path("/tmp/stable.env") + with patch( + "scripts.rebuild_v2_stable_projection_cache._compose" + ) as compose: + _start_services(env, "stream_v2_active", "stream_v2_passive") + compose.assert_called_once_with( + env, + "up", + "-d", + "--no-deps", + "stream_v2_active", + "stream_v2_passive", + ) + with self.assertRaisesRegex(ValueError, "at least one"): + _start_services(env) + + def test_lag_parser_requires_real_canonical_partitions(self): + output = """GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID +stable-projector-v1 md.canonical.v2 0 10 12 2 - - - +stable-projector-v1 md.canonical.v2 1 20 20 0 - - - +stable-projector-v1 another.topic 2 0 99 99 - - - +""" + self.assertEqual(parse_canonical_lag(output), (2, 2)) + with self.assertRaisesRegex(RuntimeError, "no partitions"): + parse_canonical_lag("GROUP TOPIC PARTITION") + + def test_lag_gate_requires_all_partitions_and_fixed_bound(self): + self.assertTrue( + lag_sample_acceptable( + MAX_ACCEPTED_LAG, EXPECTED_CANONICAL_PARTITIONS + ) + ) + self.assertFalse( + lag_sample_acceptable( + MAX_ACCEPTED_LAG + 1, EXPECTED_CANONICAL_PARTITIONS + ) + ) + self.assertFalse( + lag_sample_acceptable( + 0, EXPECTED_CANONICAL_PARTITIONS - 1 + ) + ) + + def test_wrong_compose_project_fails_before_mutation(self): + with tempfile.TemporaryDirectory() as directory: + env = Path(directory) / "stable.env" + env.write_text("placeholder=true\n") + completed = subprocess.CompletedProcess( + args=[], returncode=0, stdout='{"name":"wrong"}', stderr="" + ) + with patch( + "scripts.rebuild_v2_stable_projection_cache._compose", + return_value=completed, + ): + with self.assertRaisesRegex(RuntimeError, "isolated stable candidate"): + _validate_project(env) + + def test_running_cache_user_blocks_before_delete_or_flush(self): + with tempfile.TemporaryDirectory() as directory: + env = Path(directory) / "stable.env" + env.write_text("placeholder=true\n") + calls = [] + + def fake_validate(_env): + return None + + def fake_compose(_env, *arguments, **_kwargs): + calls.append(arguments) + if arguments[:4] == ("ps", "--services", "--status", "running"): + return subprocess.CompletedProcess( + args=[], returncode=0, stdout="projector_v2\n", stderr="" + ) + return subprocess.CompletedProcess( + args=[], returncode=0, stdout="", stderr="" + ) + + with ( + patch( + "scripts.rebuild_v2_stable_projection_cache._validate_project", + fake_validate, + ), + patch( + "scripts.rebuild_v2_stable_projection_cache._compose", + fake_compose, + ), + ): + with self.assertRaisesRegex(RuntimeError, "still running"): + execute_rebuild(env, timeout_seconds=10) + flattened = " ".join(" ".join(call) for call in calls) + self.assertNotIn("FLUSHDB", flattened) + self.assertNotIn("canonical-cache.sqlite3", flattened) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phaseb_stable_release.py b/tests/test_phaseb_stable_release.py new file mode 100644 index 0000000..eb559e3 --- /dev/null +++ b/tests/test_phaseb_stable_release.py @@ -0,0 +1,306 @@ +from __future__ import annotations + +import copy +import json +import tempfile +import tomllib +import unittest +from dataclasses import replace +from pathlib import Path + +import yaml + +import qdl_sdk +from qdl.consumer.stable import StableConsumerMigrationPlan +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.security import RedisMinuteQuota +from qdl.transport.kafka_projector import ( + ConfluentProjectorBroker, + KafkaProjectorConfig, + KafkaProjectorRecord, +) +from scripts.generate_phase5_openapi import build_openapi + + +ROOT = Path(__file__).resolve().parents[1] +CATALOG_PATH = ROOT / "config/v2/stable-source-bindings.yaml" +MIGRATION_PATH = ROOT / "config/v2/stable-consumer-migration.yaml" + + +class StableConsumerMigrationContractTests(unittest.TestCase): + def setUp(self) -> None: + self.catalog = StableSourceCatalog.load(CATALOG_PATH) + + def load(self) -> StableConsumerMigrationPlan: + return StableConsumerMigrationPlan.load( + MIGRATION_PATH, + manifest_root=ROOT, + catalog=self.catalog, + ) + + def test_five_real_consumer_manifests_are_catalog_bound_and_fail_closed(self): + plan = self.load() + self.assertEqual(plan.contract_version, "2.0.0") + self.assertEqual(plan.authority, "V1") + self.assertEqual(plan.target_route, "V1_WITH_V2_SHADOW") + self.assertEqual(len(plan.consumers), 5) + self.assertEqual( + {item.consumer_id for item in plan.consumers}, + { + "monitoring.multivenue.stable", + "alpha.binance.paper.stable", + "alpha.okx.paper.stable", + "alpha.vn.paper.stable", + "trading-system.paper.stable", + }, + ) + for item in plan.consumers: + with self.subTest(consumer_id=item.consumer_id): + self.assertEqual(item.state, "SHADOW") + self.assertEqual(item.rollback_route, "V1") + self.assertFalse(item.cutover_authorized) + self.assertEqual(item.manifest.sdk_major, 2) + self.assertEqual(item.manifest.rollback_contract, "V1") + self.assertEqual(item.manifest.environment, "paper") + for requirement in item.manifest.requirements: + self.assertIsNotNone(self.catalog.binding_for(requirement)) + + def test_unknown_fields_active_route_and_unknown_binding_fail_closed(self): + payload = yaml.safe_load(MIGRATION_PATH.read_text(encoding="utf-8")) + unknown = copy.deepcopy(payload) + unknown["unexpected"] = True + with self.assertRaisesRegex(ValueError, "incomplete or unknown"): + StableConsumerMigrationPlan.from_mapping( + unknown, manifest_root=ROOT, catalog=self.catalog + ) + + active = copy.deepcopy(payload) + active["consumers"][0]["state"] = "ACTIVE" + active["consumers"][0]["cutover_authorized"] = True + with self.assertRaisesRegex(ValueError, "not fail-closed"): + StableConsumerMigrationPlan.from_mapping( + active, manifest_root=ROOT, catalog=self.catalog + ) + + manifest_path = ROOT / "consumers/stable/alpha-binance-paper.yaml" + manifest = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) + manifest["spec"]["requirements"][0]["instrument_uid"] = "unknown" + with tempfile.TemporaryDirectory(prefix="qdl-phaseb-manifest-") as directory: + root = Path(directory) + temporary = root / "consumers/stable/invalid.yaml" + temporary.parent.mkdir(parents=True) + temporary.write_text( + yaml.safe_dump(manifest, sort_keys=False), encoding="utf-8" + ) + invalid = copy.deepcopy(payload) + invalid["consumers"] = [invalid["consumers"][1]] + invalid["consumers"][0]["manifest"] = ( + "/app/consumers/stable/invalid.yaml" + ) + with self.assertRaisesRegex(KeyError, "no stable source binding"): + StableConsumerMigrationPlan.from_mapping( + invalid, manifest_root=root, catalog=self.catalog + ) + + def test_trading_system_is_the_only_paper_execution_dependency(self): + plan = self.load() + policies = { + item.consumer_id: item.manifest.execution_dependency + for item in plan.consumers + } + self.assertEqual(policies["trading-system.paper.stable"], "PAPER_ONLY") + self.assertEqual( + {value for key, value in policies.items() if key != "trading-system.paper.stable"}, + {"FORBIDDEN"}, + ) + + +class StableRuntimeDependencyTests(unittest.TestCase): + def test_stable_quota_namespace_is_isolated_and_current_namespace_is_rejected(self): + quota = RedisMinuteQuota(object(), prefix="qdl:stable:v2:paper:phaseb") + self.assertEqual(quota.prefix, "qdl:stable:v2:paper:phaseb") + with self.assertRaisesRegex(ValueError, "dedicated beta or stable"): + RedisMinuteQuota(object(), prefix="qdl:v2:current") + + def test_projector_kafka_readiness_uses_bounded_metadata_probe(self): + class FakeConsumer: + def __init__(self, config): + self.config = config + self.closed = False + + def subscribe(self, topics, **_callbacks): + self.topics = tuple(topics) + + def list_topics(self, *, timeout): + self.timeout = timeout + return {"cluster": "stable"} + + def close(self): + self.closed = True + + with tempfile.TemporaryDirectory(prefix="qdl-phaseb-kafka-") as directory: + root = Path(directory) + for name in ("ca.crt", "client.crt", "client.key"): + (root / name).write_text("test-only", encoding="ascii") + broker = ConfluentProjectorBroker( + KafkaProjectorConfig( + bootstrap_servers="kafka1:9092", + client_id="stable-projector", + group_id="stable-projector-v1", + raw_topics=(), + canonical_topic="md.canonical.stable.v2", + ca_path=root / "ca.crt", + certificate_path=root / "client.crt", + key_path=root / "client.key", + ), + consumer_factory=FakeConsumer, + ) + self.assertTrue(broker.ping(0.25)) + self.assertEqual( + broker._consumer.topics, ("md.canonical.stable.v2",) + ) + self.assertFalse(broker._consumer.config["enable.auto.commit"]) + self.assertEqual(broker._consumer.config["isolation.level"], "read_committed") + broker.close() + self.assertFalse(broker.ping()) + + def test_projector_coalesces_only_acked_offsets_and_replays_on_rebalance(self): + class FakeConsumer: + def __init__(self, config, *, commit_error=False): + self.config = config + self.commit_error = commit_error + self.commits = [] + self.closed = False + self.pause_calls = [] + self.resume_calls = [] + + def subscribe(self, topics, **callbacks): + self.topics = tuple(topics) + self.callbacks = callbacks + + def poll(self, _timeout): + return None + + def assignment(self): + class Partition: + def __init__(self, topic, partition): + self.topic = topic + self.partition = partition + + return [ + Partition("md.raw.stable.v1", 0), + Partition("md.canonical.v2", 0), + ] + + def pause(self, partitions): + self.paused = tuple((item.topic, item.partition) for item in partitions) + self.pause_calls.append(self.paused) + + def resume(self, partitions): + self.resumed = tuple((item.topic, item.partition) for item in partitions) + self.resume_calls.append(self.resumed) + + def list_topics(self, *, timeout): + del timeout + return {"cluster": "stable"} + + def commit(self, *, offsets, asynchronous): + self.commits.append((tuple(offsets), asynchronous)) + if asynchronous and self.commit_error: + self.config["on_commit"](RuntimeError("commit failed"), offsets) + return None if asynchronous else offsets + + def close(self): + self.closed = True + + def record(offset, *, partition=0, epoch=1): + return KafkaProjectorRecord( + topic="md.raw.stable.v1", + partition=partition, + offset=offset, + key="BINANCE/USDM/BTCUSDT/trade", + event_id=bytes([offset + 1]) * 16, + payload=b"raw", + accepted_at_ns=offset + 1, + assignment_epoch=epoch, + ) + + with tempfile.TemporaryDirectory(prefix="qdl-phaseb-kafka-") as directory: + root = Path(directory) + for name in ("ca.crt", "client.crt", "client.key"): + (root / name).write_text("test-only", encoding="ascii") + config = KafkaProjectorConfig( + bootstrap_servers="kafka1:9092", + client_id="stable-projector", + group_id="stable-projector-v1", + raw_topics=("md.raw.stable.v1",), + canonical_topic="md.canonical.v2", + ca_path=root / "ca.crt", + certificate_path=root / "client.crt", + key_path=root / "client.key", + checkpoint_batch_size=2, + checkpoint_interval_ms=5_000, + ) + broker = ConfluentProjectorBroker(config, consumer_factory=FakeConsumer) + broker.pause_canonical() + self.assertEqual(broker._consumer.paused, (("md.canonical.v2", 0),)) + broker.resume_canonical() + self.assertEqual(broker._consumer.resumed, (("md.canonical.v2", 0),)) + broker.checkpoint(record(0)) + self.assertEqual(broker._consumer.commits, []) + broker.checkpoint(record(1)) + offsets, asynchronous = broker._consumer.commits[0] + self.assertTrue(asynchronous) + self.assertEqual([(item.partition, item.offset) for item in offsets], [(0, 2)]) + + broker.checkpoint(record(0, partition=1)) + broker._consumer.callbacks["on_revoke"](broker._consumer, []) + self.assertEqual(broker._pending_offsets, {}) + broker.close() + self.assertEqual(len(broker._consumer.commits), 1) + + flow_broker = ConfluentProjectorBroker( + config, consumer_factory=FakeConsumer + ) + flow_broker.pause_canonical() + flow_broker._consumer.callbacks["on_revoke"]( + flow_broker._consumer, [] + ) + flow_broker.poll(0.1) + self.assertEqual(len(flow_broker._consumer.pause_calls), 2) + self.assertEqual( + set(flow_broker._consumer.pause_calls), + {(("md.canonical.v2", 0),)}, + ) + flow_broker.resume_canonical() + flow_broker.close() + + failed = ConfluentProjectorBroker( + replace(config, checkpoint_batch_size=1), + consumer_factory=lambda values: FakeConsumer(values, commit_error=True), + ) + failed.checkpoint(record(0)) + with self.assertRaisesRegex(RuntimeError, "asynchronous stable checkpoint"): + failed.poll(0.1) + with self.assertRaisesRegex(RuntimeError, "asynchronous stable checkpoint"): + failed.close() + self.assertTrue(failed._consumer.closed) + + +class StableReleaseVersionContractTests(unittest.TestCase): + def test_package_sdk_and_openapi_are_exactly_2_0_0(self): + package = tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8")) + snapshot = json.loads( + (ROOT / "contracts/v2/openapi.snapshot.json").read_text(encoding="utf-8") + ) + generated = build_openapi() + self.assertEqual(package["project"]["version"], "2.0.0") + self.assertEqual(qdl_sdk.__version__, "2.0.0") + self.assertEqual(generated["info"]["version"], "2.0.0") + self.assertEqual(snapshot, generated) + self.assertEqual(len(generated["paths"]), 10) + self.assertEqual(len(generated["components"]["schemas"]), 42) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_stream_supervisor.py b/tests/test_stream_supervisor.py index e0c5c33..4ba9a7c 100644 --- a/tests/test_stream_supervisor.py +++ b/tests/test_stream_supervisor.py @@ -30,6 +30,7 @@ def test_snapshot_reports_missing_then_observed_feed(self): shard_id = supervisor.register_shard("binance_spot_trade", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_publish( { "key": "trade:price:BTCUSDT", @@ -51,6 +52,7 @@ def test_queue_drop_is_warning_by_default(self): supervisor = StreamSupervisor() shard_id = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_queue_drop("binance_spot_kline", shard_id) snapshot = supervisor.snapshot() @@ -63,6 +65,7 @@ def test_queue_drop_degrades_strict_health(self): supervisor = StreamSupervisor(strict_feed_health=True) shard_id = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_queue_drop("binance_spot_kline", shard_id) snapshot = supervisor.snapshot() @@ -74,6 +77,7 @@ def test_old_queue_drop_does_not_permanently_degrade_health(self): supervisor.queue_drop_window_seconds = 1 shard_id = supervisor.register_shard("binance_spot_kline", "wss://example") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) supervisor.record_queue_drop("binance_spot_kline", shard_id) snapshot = supervisor.snapshot(now=time.time() + 2) @@ -86,6 +90,7 @@ def test_missing_trade_feed_is_diagnostic_not_health_failure(self): shard_id = supervisor.register_shard("binance_spot_trade", "wss://example") supervisor.expect_feed("binance_spot_trade", "trade", "ILLQUSDT") supervisor.mark_connected(shard_id) + supervisor.mark_message(shard_id) snapshot = supervisor.snapshot(now=supervisor.started_at + 10) self.assertEqual(snapshot["status"], "ok") @@ -98,8 +103,8 @@ def test_missing_kline_after_startup_grace_is_warning_by_default(self): supervisor.expect_feed("binance_spot_kline", "kline", "BTCUSDT", "1m") supervisor.mark_connected(shard_id) - snapshot = supervisor.snapshot(now=supervisor.started_at + 10) - self.assertEqual(snapshot["status"], "ok") + snapshot = supervisor.snapshot(now=supervisor.started_at + 20) + self.assertEqual(snapshot["status"], "degraded") self.assertEqual(snapshot["feeds"]["health_missing_count"], 1) def test_missing_kline_after_startup_grace_degrades_strict_health(self): @@ -108,7 +113,7 @@ def test_missing_kline_after_startup_grace_degrades_strict_health(self): supervisor.expect_feed("binance_spot_kline", "kline", "BTCUSDT", "1m") supervisor.mark_connected(shard_id) - snapshot = supervisor.snapshot(now=supervisor.started_at + 10) + snapshot = supervisor.snapshot(now=supervisor.started_at + 20) self.assertEqual(snapshot["status"], "degraded") self.assertTrue(snapshot["strict_feed_health"]) diff --git a/tests/test_v2_binance_bar_edge.py b/tests/test_v2_binance_bar_edge.py new file mode 100644 index 0000000..2ec55aa --- /dev/null +++ b/tests/test_v2_binance_bar_edge.py @@ -0,0 +1,81 @@ +from __future__ import annotations + +import json +import unittest + +from qdl.adapters.binance import ( + BinanceBarRawBinding, + fetch_latest_closed_bar_raw_envelope, +) +from qdl.canonical.market import canonicalize_binance_usdm_rest_bar +from qdl.canonical.trade import TradeContext +from qdl.common.v1 import common_pb2 +from qdl.raw.capture import bind_capture_context + + +class BinanceRestBarEdgeTests(unittest.TestCase): + def binding(self, market="USDM", product="PERPETUAL"): + return BinanceBarRawBinding( + market=market, + product_type=product, + native_symbol="BTCUSDT", + interval="1m", + subscription_id=f"binance-{market.lower()}-bar", + source_session_id="binance-rest-session-1", + connection_generation=1, + lease_epoch=7, + authority_revision=1, + partition_plan_epoch=1, + adapter_version="binance-rest/2.0.0", + config_revision=1, + instrument_catalog_revision=3, + ) + + @staticmethod + def response(*_args, **_kwargs): + return { + "data": [ + [0, "1", "2", "0.5", "1.5", "10", 59_999, "15", 2, "0", "0"], + [60_000, "1.5", "3", "1", "2", "20", 119_999, "40", 3, "0", "0"], + [120_000, "2", "4", "2", "3", "30", 179_999, "90", 4, "0", "0"], + ] + } + + def test_latest_closed_native_row_is_selected_and_canonicalized(self): + raw = fetch_latest_closed_bar_raw_envelope( + self.binding(), now_ms=150_000, fetcher=self.response, sleep=lambda _: None, + test_provenance=True, + ) + payload = json.loads(raw.raw_frame_bytes) + self.assertEqual(payload["row"][0], 60_000) + self.assertEqual(payload["bar_origin"], "VENUE_NATIVE") + context = bind_capture_context( + TradeContext( + instrument_uid="uid-binance", instrument_id="BINANCE.USDM.PERPETUAL.BTC-USDT", + instrument_revision=1, venue="BINANCE", market="USDM", + product_type="PERPETUAL", native_symbol="BTCUSDT", + provider="BINANCE_DIRECT", source_id="binance-rest-bar", source_role="PRIMARY", + lease_epoch=7, received_at_ns=1, normalized_at_ns=2, published_at_ns=3, + partition_sequence=1, normalizer_version="qdl-rust-core/2.0.0", + adapter_version="binance-rest/2.0.0", config_revision=1, + ), + raw, + ) + event = canonicalize_binance_usdm_rest_bar(payload, context) + self.assertEqual(event.bar.origin, common_pb2.BAR_ORIGIN_VENUE_NATIVE) + self.assertEqual(event.bar.volume_unit, common_pb2.QUANTITY_UNIT_BASE_ASSET) + self.assertEqual(event.bar.base_volume.source_text, "20") + self.assertEqual(event.bar.quote_volume.source_text, "40") + self.assertNotIn(common_pb2.QUALITY_FLAG_BACKFILLED, event.quality_flags) + + def test_no_closed_bar_and_bad_market_fail_closed(self): + with self.assertRaisesRegex(RuntimeError, "no closed bar"): + fetch_latest_closed_bar_raw_envelope( + self.binding(), now_ms=1, fetcher=self.response, sleep=lambda _: None, + ) + with self.assertRaisesRegex(ValueError, "market"): + self.binding("AUTO", "PERPETUAL") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_v2_binance_endpoint_contract.py b/tests/test_v2_binance_endpoint_contract.py new file mode 100644 index 0000000..b71118f --- /dev/null +++ b/tests/test_v2_binance_endpoint_contract.py @@ -0,0 +1,24 @@ +from __future__ import annotations + +import unittest + +from app.stream.feed_builder import build_urls +from qdl.adapters.binance_usdm import BINANCE_USDM_WS_BASE + + +class BinancePublicStreamEndpointTests(unittest.TestCase): + def test_usdm_kline_trade_and_bbo_use_official_public_stream_path(self): + urls = build_urls({"binance_futures": ["BTCUSDT"]}, interval="1m") + self.assertTrue(urls["binance_futures"]) + self.assertTrue( + all("fstream.binance.com/public/stream?streams=" in value for value in urls["binance_futures"]) + ) + self.assertEqual( + BINANCE_USDM_WS_BASE, + "wss://fstream.binance.com/public/stream?streams=", + ) + self.assertNotIn("fstream.binance.com/stream?streams=", BINANCE_USDM_WS_BASE) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_v2_rust_okx_core.py b/tests/test_v2_rust_okx_core.py new file mode 100644 index 0000000..c7dc462 --- /dev/null +++ b/tests/test_v2_rust_okx_core.py @@ -0,0 +1,118 @@ +from __future__ import annotations + +import json +import unittest +from pathlib import Path + +import yaml + +from qdl.canonical.market import canonicalize_okx_bar, canonicalize_okx_bbo +from qdl.canonical.trade import TradeContext +from qdl.common.v1 import common_pb2 +from qdl.marketdata.v2 import market_data_pb2 + + +ROOT = Path(__file__).resolve().parents[1] + + +class V2RustOkxCanonicalParityTest(unittest.TestCase): + def fixture(self, name: str): + payload = json.loads( + (ROOT / "tests/fixtures/phase2" / name).read_text() + ) + return payload, TradeContext(**payload["context"]) + + def test_python_oracle_matches_frozen_okx_bbo_and_bar_bytes(self): + for fixture_name, golden_name, canonicalize in ( + ("okx_bbo.json", "okx-swap-bbo.bin", canonicalize_okx_bbo), + ("okx_bar.json", "okx-swap-bar.bin", canonicalize_okx_bar), + ): + with self.subTest(fixture=fixture_name): + fixture, context = self.fixture(fixture_name) + actual = canonicalize( + fixture["raw"], context + ).SerializeToString(deterministic=True) + expected = ( + ROOT / "contracts/golden/phase2" / golden_name + ).read_bytes() + self.assertEqual(actual, expected) + + def test_bbo_is_replace_only_quote_and_bar_has_explicit_lifecycle(self): + fixture, context = self.fixture("okx_bbo.json") + bbo = canonicalize_okx_bbo(fixture["raw"], context) + self.assertEqual(bbo.WhichOneof("payload"), "quote") + self.assertEqual(bbo.quote.level, 1) + self.assertEqual(bbo.source_sequence, "817263") + + fixture, context = self.fixture("okx_bar.json") + bar = canonicalize_okx_bar(fixture["raw"], context) + self.assertEqual(bar.WhichOneof("payload"), "bar") + self.assertTrue(bar.bar.is_final) + self.assertEqual( + bar.bar.lifecycle, market_data_pb2.BAR_LIFECYCLE_FINAL + ) + self.assertEqual(bar.bar.interval, "1m") + self.assertIn( + common_pb2.QUALITY_FLAG_FIELD_MISSING, bar.quality_flags + ) + + def test_malformed_okx_identity_depth_and_confirmation_fail_closed(self): + fixture, context = self.fixture("okx_bbo.json") + malformed = json.loads(json.dumps(fixture["raw"])) + malformed["arg"]["instId"] = "ETH-USDT-SWAP" + with self.assertRaisesRegex(ValueError, "mismatch"): + canonicalize_okx_bbo(malformed, context) + malformed = json.loads(json.dumps(fixture["raw"])) + malformed["data"][0]["bids"].append(["1", "1", "0", "1"]) + with self.assertRaisesRegex(ValueError, "one bid"): + canonicalize_okx_bbo(malformed, context) + + fixture, context = self.fixture("okx_bar.json") + malformed = json.loads(json.dumps(fixture["raw"])) + malformed["data"][0][8] = "2" + with self.assertRaisesRegex(ValueError, "confirm"): + canonicalize_okx_bar(malformed, context) + + +class V2StableCapabilityMatrixTest(unittest.TestCase): + def test_binance_okx_and_vn_share_one_truthful_shadow_core(self): + matrix = yaml.safe_load( + (ROOT / "config/v2/stable-capabilities.yaml").read_text() + ) + self.assertEqual(matrix["public_contract_version"], "2.0.0") + self.assertEqual(matrix["runtime_authority"], "RUST_SHADOW") + self.assertFalse(matrix["authority_eligible"]) + expected = {"TRADE", "BBO", "BAR"} + rows = { + (item["venue"], item["market"]): item + for item in matrix["capabilities"] + } + self.assertEqual( + set(rows), + { + ("BINANCE", "USDM"), + ("BINANCE", "SPOT"), + ("OKX", "SWAP"), + ("OKX", "SPOT"), + ("HNX", "VN_DERIVATIVES"), + ("HOSE", "EQUITIES"), + }, + ) + for key in (("BINANCE", "USDM"), ("BINANCE", "SPOT"), ("OKX", "SWAP"), ("OKX", "SPOT")): + self.assertEqual(set(rows[key]["feeds"]), expected) + for key in (("HNX", "VN_DERIVATIVES"), ("HOSE", "EQUITIES")): + self.assertEqual(set(rows[key]["feeds"]), {"TRADE", "BAR"}) + self.assertTrue(matrix["equal_source_contract"]) + self.assertTrue( + all( + item["canonical_core"] == "qdl-rust-realtime-core" + for item in rows.values() + ) + ) + self.assertFalse(matrix["capability_gates"]["CRYPTO_L2"]["stable"]) + self.assertFalse(matrix["capability_gates"]["VN_QUOTE"]["stable"]) + + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_v2_stable_multivenue_contract.py b/tests/test_v2_stable_multivenue_contract.py new file mode 100644 index 0000000..0d3053f --- /dev/null +++ b/tests/test_v2_stable_multivenue_contract.py @@ -0,0 +1,183 @@ +from __future__ import annotations + +import json +import unittest +from pathlib import Path + +from pydantic import ValidationError + +from qdl.api_v2.models import BarPayload, DecimalValue, TradePayload +from qdl.canonical.market import ( + canonicalize_binance_usdm_bar, + canonicalize_binance_usdm_bbo, + canonicalize_dnse_bar, + canonicalize_okx_bar, + canonicalize_okx_bbo, +) +from qdl.canonical.trade import ( + TradeContext, + canonicalize_binance_usdm_trade, + canonicalize_dnse_trade, + canonicalize_okx_trade, +) +from qdl.common.v1 import common_pb2 +from qdl.domain.quantity import QuantityUnit, resolve_quantity_unit +from qdl.marketdata.v2 import market_data_pb2 + + +ROOT = Path(__file__).resolve().parents[1] +FIXTURES = ROOT / "tests/fixtures/phase2" +GOLDEN = ROOT / "contracts/golden/phase2" +MANIFEST = GOLDEN / "manifest-v2-stable-multivenue.json" + + +def canonicalizer(name: str): + if "trade" in name: + if name.startswith("binance"): + return canonicalize_binance_usdm_trade + if name.startswith("okx"): + return canonicalize_okx_trade + return canonicalize_dnse_trade + if "bbo" in name: + return canonicalize_binance_usdm_bbo if name.startswith("binance") else canonicalize_okx_bbo + if "rest_bar" in name: + from qdl.canonical.market import canonicalize_binance_usdm_rest_bar + return canonicalize_binance_usdm_rest_bar + return canonicalize_dnse_bar if name.startswith(("dnse", "vnstock")) else ( + canonicalize_binance_usdm_bar if name.startswith("binance") else canonicalize_okx_bar + ) + + +def load_event(fixture_name: str): + fixture = json.loads((FIXTURES / fixture_name).read_text()) + return canonicalizer(fixture_name)( + fixture["raw"], TradeContext(**fixture["context"]) + ) + + +def expected_unit(fixture_name: str) -> int: + if "spot" in fixture_name or fixture_name.startswith("binance"): + return common_pb2.QUANTITY_UNIT_BASE_ASSET + if "equity" in fixture_name: + return common_pb2.QUANTITY_UNIT_SHARE + return common_pb2.QUANTITY_UNIT_CONTRACT + + +class StableMultivenueCanonicalContractTests(unittest.TestCase): + def test_all_nineteen_python_events_match_frozen_bytes(self): + manifest = json.loads(MANIFEST.read_text()) + self.assertEqual(manifest["schema"], "qdl.v2.stable-multivenue-golden.v1") + self.assertEqual(len(manifest["cases"]), 19) + for case in manifest["cases"]: + with self.subTest(fixture=case["fixture"]): + event = load_event(case["fixture"]) + self.assertEqual( + event.SerializeToString(deterministic=True), + (GOLDEN / case["golden"]).read_bytes(), + ) + + def test_trade_quote_and_bar_units_are_never_unspecified(self): + manifest = json.loads(MANIFEST.read_text()) + for case in manifest["cases"]: + event = load_event(case["fixture"]) + expected = expected_unit(case["fixture"]) + payload = event.WhichOneof("payload") + with self.subTest(fixture=case["fixture"], payload=payload): + if payload == "trade": + self.assertEqual(event.trade.quantity_unit, expected) + elif payload == "quote": + self.assertEqual(event.quote.quantity_unit, expected) + elif payload == "bar": + self.assertEqual(event.bar.volume_unit, expected) + else: + self.fail(f"unexpected Phase A payload: {payload}") + + def test_spot_bbo_missing_provider_time_is_explicit(self): + event = load_event("binance_spot_bbo.json") + self.assertIn( + common_pb2.QUALITY_FLAG_SOURCE_TIME_MISSING, + event.quality_flags, + ) + self.assertEqual( + event.source_event_time_ns, + event.received_at_ns // 1_000_000 * 1_000_000, + ) + + def test_volume_components_preserve_venue_semantics(self): + swap = load_event("okx_bar.json").bar + self.assertEqual(swap.volume_unit, common_pb2.QUANTITY_UNIT_CONTRACT) + self.assertTrue(swap.HasField("contract_volume")) + self.assertTrue(swap.HasField("base_volume")) + self.assertTrue(swap.HasField("quote_volume")) + self.assertEqual(swap.volume.source_text, "12.500") + self.assertEqual(swap.base_volume.source_text, "12.500") + self.assertEqual(swap.quote_volume.source_text, "765200.00") + + spot = load_event("okx_spot_bar.json").bar + self.assertEqual(spot.volume_unit, common_pb2.QUANTITY_UNIT_BASE_ASSET) + self.assertFalse(spot.HasField("contract_volume")) + self.assertEqual(spot.base_volume.source_text, spot.volume.source_text) + + derivative = load_event("dnse_derivative_bar.json").bar + equity = load_event("dnse_equity_bar.json").bar + self.assertTrue(derivative.HasField("contract_volume")) + self.assertEqual(equity.volume_unit, common_pb2.QUANTITY_UNIT_SHARE) + self.assertFalse(equity.HasField("base_volume")) + + def test_dnse_missing_native_identity_is_explicit_not_fabricated(self): + event = load_event("dnse_derivative_trade.json") + self.assertEqual( + event.trade.identity_kind, + market_data_pb2.TRADE_IDENTITY_KIND_DERIVED_RAW_CAPTURE, + ) + self.assertTrue(event.trade.native_trade_id.startswith("derived:")) + self.assertEqual(event.trade.aggressor_side, common_pb2.AGGRESSOR_SIDE_UNSPECIFIED) + self.assertIn(common_pb2.QUALITY_FLAG_SOURCE_TIME_MISSING, event.quality_flags) + self.assertIn(common_pb2.QUALITY_FLAG_FIELD_MISSING, event.quality_flags) + + def test_provider_role_and_unknown_identity_fail_closed(self): + fallback = load_event("vnstock_equity_bar.json") + self.assertEqual(fallback.source_role, common_pb2.SOURCE_ROLE_SECONDARY) + with self.assertRaisesRegex(ValueError, "quantity unit is undefined"): + resolve_quantity_unit(venue="UNKNOWN", market="X", product_type="Y") + + +class StablePublicPayloadUnitTests(unittest.TestCase): + decimal = DecimalValue(coefficient="1", scale=0, source_text="1") + + def test_public_trade_and_bar_require_units_and_identity(self): + with self.assertRaises(ValidationError): + TradePayload( + native_trade_id="1", + price=self.decimal, + quantity=self.decimal, + aggressor_side="BUY", + identity_kind="NATIVE", + ) + with self.assertRaises(ValidationError): + BarPayload( + interval="1m", + open_time_ns=1, + close_time_ns=2, + open=self.decimal, + high=self.decimal, + low=self.decimal, + close=self.decimal, + volume=self.decimal, + lifecycle="FINAL", + revision=0, + origin="VENUE_NATIVE", + ) + trade = TradePayload( + native_trade_id="1", + price=self.decimal, + quantity=self.decimal, + quantity_unit=QuantityUnit.BASE_ASSET, + aggressor_side="BUY", + identity_kind="NATIVE", + ) + self.assertEqual(trade.quantity_unit, QuantityUnit.BASE_ASSET) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_v2_vn_raw_edge.py b/tests/test_v2_vn_raw_edge.py new file mode 100644 index 0000000..5848c4a --- /dev/null +++ b/tests/test_v2_vn_raw_edge.py @@ -0,0 +1,213 @@ +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +from qdl.adapters.vn import ( + VnRawBinding, + build_dnse_bar_raw_envelope, + build_dnse_trade_raw_envelope, +) +from qdl.canonical.trade import TradeContext, canonicalize_dnse_trade +from qdl.common.v1 import common_pb2 +from qdl.raw.capture import bind_capture_context +from qdl.raw.envelope import validate_raw_envelope +from qdl.transport.kafka_raw import KafkaRawPublisher, KafkaRawPublisherConfig + + +class FakeMessage: + def __init__(self, partition=1, offset=7): + self._partition = partition + self._offset = offset + + def partition(self): + return self._partition + + def offset(self): + return self._offset + + +class FakeProducer: + def __init__(self, config, *, fail=False): + self.config = config + self.fail = fail + self.pending = [] + self.records = [] + + def produce(self, topic, **kwargs): + self.records.append((topic, kwargs)) + self.pending.append(kwargs["on_delivery"]) + + def poll(self, _): + return 0 + + def flush(self, _): + callbacks, self.pending = self.pending, [] + for callback in callbacks: + callback(RuntimeError("delivery failed") if self.fail else None, FakeMessage()) + return 0 + + +class VnRawEdgeTests(unittest.TestCase): + def binding(self): + return VnRawBinding( + venue="HNX", + market="VN_DERIVATIVES", + product_type="FUTURE", + native_symbol="VN30F1M", + subscription_id="dnse-vn30-trades", + source_session_id="dnse-session-1", + connection_generation=1, + lease_epoch=7, + authority_revision=1, + partition_plan_epoch=1, + adapter_version="dnse-sdk/2.0.0", + config_revision=1, + instrument_catalog_revision=3, + ) + + def envelope(self): + return build_dnse_trade_raw_envelope( + { + "symbol": "VN30F1M", + "price": 1820.7, + "quantity": 12, + "totalVolumeTraded": 12000, + "marketId": "G3", + "boardId": "DERIVATIVES", + "tradingSessionId": 2, + }, + self.binding(), + received_at_ns=1_786_352_400_123_456_000, + test_provenance=True, + ) + + def test_dnse_sdk_delivery_preserves_honest_missing_trade_semantics(self): + raw = self.envelope() + validate_raw_envelope(raw) + self.assertEqual(raw.capture_boundary, 3) + payload = json.loads(raw.raw_frame_bytes) + self.assertEqual(payload["price"], "1820.7") + self.assertEqual(payload["quantity"], "12") + context = bind_capture_context( + TradeContext( + instrument_uid="uid-vn30f1m", + instrument_id="HNX.VN_DERIVATIVES.FUTURE.VN30F1M", + instrument_revision=1, + venue="HNX", + market="VN_DERIVATIVES", + product_type="FUTURE", + native_symbol="VN30F1M", + provider="DNSE_DIRECT", + source_id="dnse-vn30-trade", + source_role="PRIMARY", + lease_epoch=7, + received_at_ns=1, + normalized_at_ns=2, + published_at_ns=3, + partition_sequence=1, + normalizer_version="qdl-normalizer/2.0.0", + adapter_version="dnse-sdk/2.0.0", + config_revision=1, + ), + raw, + ) + event = canonicalize_dnse_trade(payload, context) + self.assertEqual(event.trade.quantity_unit, common_pb2.QUANTITY_UNIT_CONTRACT) + self.assertEqual(event.trade.aggressor_side, common_pb2.AGGRESSOR_SIDE_UNSPECIFIED) + self.assertTrue(event.trade.native_trade_id.startswith("derived:")) + + def test_rest_and_websocket_closed_bar_share_core_binding_with_honest_transport(self): + row = { + "t": 1_786_352_340, + "o": "1820.7", + "h": "1821.2", + "l": "1820.2", + "c": "1820.9", + "v": "0", + } + rest = build_dnse_bar_raw_envelope( + row, self.binding(), received_at_ns=1_786_352_400_000_000_000 + ) + websocket = build_dnse_bar_raw_envelope( + row, + self.binding(), + received_at_ns=1_786_352_400_000_000_001, + acquisition_origin="WEBSOCKET_CLOSED", + ) + validate_raw_envelope(rest) + validate_raw_envelope(websocket) + self.assertEqual(rest.native_channel, websocket.native_channel) + self.assertEqual(rest.raw_frame_bytes, websocket.raw_frame_bytes) + self.assertNotEqual(rest.transport_protocol, websocket.transport_protocol) + self.assertNotEqual(rest.capture_boundary, websocket.capture_boundary) + + def test_invalid_or_incomplete_dnse_delivery_fails_closed(self): + with self.assertRaisesRegex(ValueError, "symbol"): + build_dnse_trade_raw_envelope( + {"symbol": "FPT", "price": 1, "quantity": 1, "totalVolumeTraded": 1}, + self.binding(), + received_at_ns=1, + ) + with self.assertRaisesRegex(ValueError, "quantity"): + build_dnse_trade_raw_envelope( + {"symbol": "VN30F1M", "price": 1, "quantity": 0, "totalVolumeTraded": 1}, + self.binding(), + received_at_ns=1, + ) + + def test_python_vendor_edge_uses_idempotent_tls_durable_ack(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + for name in ("ca.crt", "client.crt", "client.key"): + (root / name).write_text("test") + created = [] + + def factory(config): + value = FakeProducer(config) + created.append(value) + return value + + publisher = KafkaRawPublisher( + KafkaRawPublisherConfig( + bootstrap_servers="kafka1:9092,kafka2:9092,kafka3:9092", + client_id="qdl-vn-edge", + topic="qdl.raw.vn.v1", + ca_path=root / "ca.crt", + certificate_path=root / "client.crt", + key_path=root / "client.key", + ), + producer_factory=factory, + ) + ack = publisher.publish_many((self.envelope(),)) + self.assertEqual(len(ack), 1) + self.assertEqual(ack[0].offset, 7) + self.assertTrue(created[0].config["enable.idempotence"]) + self.assertEqual(created[0].config["acks"], "all") + self.assertEqual(created[0].records[0][0], "qdl.raw.vn.v1") + publisher.close() + + def test_delivery_failure_is_not_reported_as_acknowledged(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + for name in ("ca.crt", "client.crt", "client.key"): + (root / name).write_text("test") + publisher = KafkaRawPublisher( + KafkaRawPublisherConfig( + bootstrap_servers="kafka:9092", + client_id="qdl-vn-edge", + topic="qdl.raw.vn.v1", + ca_path=root / "ca.crt", + certificate_path=root / "client.crt", + key_path=root / "client.key", + ), + producer_factory=lambda config: FakeProducer(config, fail=True), + ) + with self.assertRaisesRegex(RuntimeError, "durable ACK failed"): + publisher.publish_many((self.envelope(),)) + + +if __name__ == "__main__": + unittest.main() diff --git a/upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md b/upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md index 36ca5c0..4a83044 100644 --- a/upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md +++ b/upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md @@ -3,9 +3,9 @@ > **Trạng thái:** Implementation specification / agent guide > **Mục tiêu hệ thống:** [`BobbyAxerol/quant-data-layer`](https://github.com/BobbyAxerol/quant-data-layer) > **Nguồn chuẩn:** [OKX API v5](https://www.okx.com/docs-v5/en/) và [OKX API changelog](https://www.okx.com/docs-v5/log_en/) -> **Ngày đối chiếu:** 2026-08-13 +> **Ngày đối chiếu:** 2026-08-19 > **Phạm vi:** Market Data, Public Data, Status, WebSocket JSON, order-book state, lịch sử, normalization, Redis/REST contract, khả năng mở rộng SBE -> **Ngôn ngữ triển khai ưu tiên:** Python async; Rust/SBE là phase tối ưu riêng, không làm thay đổi contract phía consumer. +> **Ngôn ngữ triển khai ưu tiên:** Rust cho realtime JSON core; Python cho REST/gRPC/SDK/history/control/compatibility edge. SBE vẫn là capability riêng và không làm thay đổi contract consumer. > **Program tracker:** [`DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md`](../DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md) > **Kiến trúc nền:** [`quant-data-layer-fund-grade-upgrade-architecture.md`](quant-data-layer-fund-grade-upgrade-architecture.md) @@ -33,6 +33,16 @@ ## 0. Cách đọc tài liệu này +### 0.0 Quyết định V2 stable ngày 2026-08-19 + +OKX không còn là nguồn reference/fallback mặc định. Baseline stable yêu cầu OKX +SPOT/SWAP và Binance Spot/USD-M ngang hàng cho `TRADE`, `BBO` và `BAR`. Rust +sở hữu WebSocket transport, native decode, canonicalization, ordering/dedup/gap, +backpressure và durable publication. Python chỉ giữ outer API/SDK/history/control +và compatibility projection. Public/business OKX sockets phải tách riêng; +`trades-all`, VIP/SBE và L2 sâu vẫn capability-gated. Consumer contract V2 +không đổi khi authority nội bộ chuyển sang Rust. + Tài liệu này là **đặc tả triển khai**, không chỉ là danh sách endpoint. Agent triển khai phải tuân theo các từ khóa chuẩn sau: - **MUST / PHẢI**: yêu cầu bắt buộc để dữ liệu đúng hoặc hệ thống an toàn. diff --git a/upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md b/upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md new file mode 100644 index 0000000..ed1515f --- /dev/null +++ b/upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md @@ -0,0 +1,103 @@ +# Phase 9.0-A Runtime Correctness Closure Report + +Date: 2026-08-18 +Branch: `feat/phase9-runtime-correctness` +Commits: `765a7f1`, `7302c45` +Decision: `PASS_ISOLATED_NO_PRODUCTION_CUTOVER` + +## Scope And Safety Boundary + +Phase 9.0-A closed the runtime correctness defects found after migration without +restarting or mutating the running V1 service. The candidate used a dedicated +Compose project, Redis instance, loopback port, networks and volumes. It held no +canonical authority and was removed after evidence capture. + +The public V1 surface remains unchanged. Live and candidate OpenAPI each exposed +40 paths, with zero additions and zero removals. + +## Implemented + +- Split transport connection from source data readiness and report TRADE/KLINE + independently per source and shard. +- Require a valid provider frame before readiness. Subscription ACK, malformed + payload and wrong-feed payload cannot make a shard healthy. +- Add first-frame and idle watchdogs with typed outage counters and jittered, + bounded reconnect backoff. A data outage survives transport reconnect and is + cleared only by a valid provider frame. +- Replace drop-oldest queue behavior with bounded backpressure. Queue pressure is + observable and a sustained full queue reconnects the source instead of + silently deleting an earlier event. +- Add one demand-only Binance USD-M closed-kline recovery manager with TTL + ownership, bounded concurrency, per-feed backoff, final-bar validation, + symbol/interval/open-time deduplication and explicit + `BINANCE_REST_GAP_FILL` provenance. +- Preserve the provider interval from `k.i`; recovered 5m or other supported bars + cannot be projected into a legacy 1m key. +- Preserve all gap rows in a publisher batch while retaining latest-state + coalescing for ordinary trade/kline updates. +- Keep existing V1 health response keys while fixing the TRADE/KLINE booleans and + adding source/recovery detail under the existing nested payload. +- Add an immutable candidate Compose boundary: UID/GID 10001, read-only root, + no source bind, dedicated writable data/log volumes, loopback-only ingress, + dropped capabilities and explicit CPU/RAM/PID limits. + +## Verification + +| Gate | Result | +|---|---| +| Targeted runtime/demand/watchdog tests | 35/35 pass | +| Full repository tests | 345 run: 340 pass, 5 environment-gated skips, 0 fail | +| Python compileall and diff check | Pass | +| V1 live-vs-candidate OpenAPI paths | 40/40, added 0, removed 0 | +| Candidate identity | `10001:10001` | +| Read-only root / no source bind | Pass / pass | +| Limits | 1.5 CPU, 1.5 GiB RAM, 256 PIDs | +| Candidate cleanup | 0 containers, networks, volumes and images remain | +| Production V1 restart/state mutation | None | + +## Real Provider Evidence + +Binance USD-M TRADE produced valid frames on all 8 shards. The provider accepted +all 8 KLINE connections but produced zero valid kline frames, so the candidate +correctly reported KLINE unavailable and top-level health `degraded`. Transport +reconnect never made the source green again. + +A BTCUSDT 1m API read created one TTL demand lease. Recovery fetched a fully +closed Binance REST bar, projected it with finality and provenance, and a direct +Binance REST query for the same timestamps matched open time, close time and all +OHLCV fields exactly. No generated or substituted market data was used. + +After lease expiry, demand and active recovery counts returned to zero. Provider +fetch count remained 2 across the following poll. Candidate queue drop and +pressure counts were both zero. + +A single resource snapshot while broad TRADE and unavailable KLINE shards were +active measured approximately 20.07% of one CPU and 131 MiB for the app, plus +2.27% CPU and 5.129 MiB for isolated Redis. These are bounded by Compose; this +was a correctness smoke, not a long soak or capacity certification. + +## Defects Caught Before Release + +1. The pinned Redis image could not call `setpriv` after `cap_drop: ALL` when no + user was declared. The candidate now runs Redis directly as `999:999`. +2. A transport reconnect initially reset source startup state, briefly making + top-level health green while KLINE still had no data. Data-outage state now + survives reconnect until a valid frame arrives. +3. Multi-interval recovery initially inherited the stream default interval. + Projection now uses the authoritative payload interval and has a 5m + regression test. + +## Decision And Remaining Boundary + +Phase 9.0-A is complete for isolated implementation and acceptance. It is ready +for review and a separately approved controlled rollout. The currently running +V1 container remains `data-layer:v0.1.0` and therefore still has legacy health +semantics until that rollout occurs. + +Phase 9.1 remains blocked. This phase does not provide production Kafka/OTel, +workload identity, external secret rotation, signed-image admission, +independent-failure-domain DR, complete consumer registration or exact authority +slice approval. + +Machine-readable evidence is in +[`phase90a-runtime-correctness.json`](phase90a-runtime-correctness.json). diff --git a/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md b/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md new file mode 100644 index 0000000..887d9f5 --- /dev/null +++ b/upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md @@ -0,0 +1,46 @@ +# Phase 9.0-B Isolated V2 Beta Report + +Decision: `PASS_ISOLATED_NO_AUTHORITY_CUTOVER` + +## Scope + +The existing V2 query/stream beta was re-certified from the Phase 9.0-A +correctness baseline. V1 remained source and public authority. The candidate +used an isolated Redis, durable spool, credentials, consumer group, loopback +ports and Compose project. No Rust canary or public-internet exposure occurred. + +## Results + +- Authentic slice: `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`. +- Canonical mismatches: `0`; generated events: + `0`; duplicate open times: + `0`. +- Continuous bridge watermark delta across the bounded observation window: + `0`. +- Active/passive stream replay: `3` fast consumers, + `1` isolated slow consumer, contiguous= + `True`. +- Query normal: `30` requests, p99.9 + `72.211` ms, `0` errors. +- Query burst: `60` requests, p99.9 + `486.649` ms, `0` errors. +- Peak candidate RSS: `55658414.08` bytes; peak CPU: + `33.36%` of one core. +- Security/adversarial, cursor, failover/fencing, dependency outage/recovery, + rate limit, oversized request and cleanup gates all passed. + +## Safety And Cleanup + +Candidate revision: `1c881389b4ee21a153903505822c61512b176044`. Runtime user was +`10001:10001`, root was read-only, no source bind was mounted and bridge access +was restricted to beta-internal plus the existing V1 internal network. +Production V1 topology/OpenAPI remained unchanged. Candidate containers, +networks, volumes and candidate tags after cleanup: `0/0/0/0`; production beta +keys: `0`. + +## Remaining Boundary + +This result permits review of an isolated read-only V2 beta only. Phase 9.1 +remains blocked on replicated production transport, OTel/alerts, workload +identity, external secrets, signature admission, independent DR, complete +consumer registration and explicit exact-slice authority approval. diff --git a/upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md b/upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md new file mode 100644 index 0000000..0ba358e --- /dev/null +++ b/upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md @@ -0,0 +1,32 @@ +# Phase 9.0-C Production Prerequisite Report + +Decision: `NO_GO_EXTERNAL` + +## Candidate + +- Slice: `production/binance/usdm/perpetual/trade/plan-1/btcusdt` +- Candidate digest: `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` +- Authority: `RUST_SHADOW`; V1 unchanged: `True` + +## Gate Summary + +- Passed: `0` +- Blocked: `12` + +## Blocking Evidence + +- `replicated_durable_transport`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `production_observability`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `workload_identity_rbac_network`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `external_secret_rotation`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) +- `signed_artifact_admission`: `EVIDENCE_BLOCKED` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `postgres_pitr`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) +- `object_store_restore`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) +- `independent_failure_domain_dr`: `MISSING_EVIDENCE` (observed `None`, required `INDEPENDENT_FAILURE_DOMAIN`) +- `redis_projector_rebuild`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `consumer_registration_rollback`: `EVIDENCE_BLOCKED` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `persistent_authority_sink_fencing`: `INSUFFICIENT_SCOPE` (observed `LOCAL_REHEARSAL`, required `PRODUCTION`) +- `exact_slice_approval`: `MISSING_EVIDENCE` (observed `None`, required `PRODUCTION`) + +These are real infrastructure/operator blockers. Same-host fixtures or +local rehearsals must not be relabeled to close them. diff --git a/upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md b/upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md new file mode 100644 index 0000000..124b87a --- /dev/null +++ b/upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md @@ -0,0 +1,45 @@ +# Phase 9.1 Rust Canary Certification Report + +## Decision + +- Status: `COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED` +- Production authorized: `false` +- Production mutations: `0` +- Prerequisite decision: `NO_GO_EXTERNAL` +- Slice: `production/binance/usdm/perpetual/trade/plan-1/btcusdt` +- Candidate digest: `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` + +## Authentic Parity + +- Provenance: `REAL_PROVIDER_READ_ONLY` +- Frozen fixtures: `128` +- Repetition: `200` +- Canonical events: `25600` +- Semantic mismatches: `0` +- Clean Rust process runs: `3` +- Aggregate SHA-256: `75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea` +- Python throughput: `27115.455` events/s +- Minimum Rust throughput: `350581.025` events/s + +## Authority And Broker Recovery + +- Transition audit: `RUST_SHADOW, RUST_CANARY, BLOCKED, RUST_SHADOW` +- Final authority: `RUST_SHADOW` +- One-replica-loss ACK: `true` +- Below-min-ISR fail closed: `true` +- Slow-consumer records: `64` +- Slow-consumer ordered and gap-free: `true` +- Public writes: `0` +- Legacy writes: `0` + +## Isolation And Cleanup + +- V1 health before/after: `200/200` +- V1 topology unchanged: `true` +- Containers/networks/volumes remaining: `0/0/0` + +## Remaining External Gates + +- Production Phase 9.0-C infrastructure and operator gates remain `NO_GO_EXTERNAL`. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- Python V1 remains the sole authoritative public and legacy writer. This report does not authorize a production `RUST_CANARY` transition. diff --git a/upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md b/upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md new file mode 100644 index 0000000..706b192 --- /dev/null +++ b/upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md @@ -0,0 +1,45 @@ +# Phase 9.2 Bounded Rust Primary Certification Report + +## Decision + +- Status: `COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED` +- Production authorized: `false` +- Production mutations: `0` +- Prerequisite decision: `NO_GO_EXTERNAL` +- Slice: `production/binance/usdm/perpetual/trade/plan-1/btcusdt` +- Candidate digest: `72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037` + +## Authentic Parity + +- Provenance: `REAL_PROVIDER_READ_ONLY` +- Canonical events: `25600` +- Semantic mismatches: `0` +- Clean Rust process runs: `3` + +## Terminal Handoff And Recovery + +- Authority states: `RUST_CANARY, RUST_PRIMARY, BLOCKED, ROLLBACK_PENDING, PYTHON_PRIMARY` +- Terminal checkpoints / accepted handoffs: `2 / 2` +- Projection parity: `true` +- Boundary gap-free: `true` +- Owner boundary correct: `true` +- Restart recovery: `PASS` +- Recovered target watermarks: `{'legacy': 180, 'primary': 180, 'public': 180}` +- First post-restart watermark: `181` +- Cutover / rollback measurement: `22.200 ms / 533.237 ms` +- One-replica-loss ACK: `true` +- Below-min-ISR fail closed: `true` +- Final authority: `PYTHON_PRIMARY` +- Production public / legacy writes: `0 / 0` + +## Isolation And Cleanup + +- V1 health before/after: `200 / 200` +- V1 topology unchanged: `true` +- Containers/networks/volumes remaining: `0 / 0 / 0` + +## Remaining External Gates + +- Phase 9.0-C remains `NO_GO_EXTERNAL`; a production primary transition is not authorized. +- Same-host replicated broker rehearsal is not an independent production failure domain. +- A real production canary hold and explicit exact-slice approval remain required. diff --git a/upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md b/upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md new file mode 100644 index 0000000..e7947fe --- /dev/null +++ b/upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md @@ -0,0 +1,42 @@ +# Phase 9.3 Hold, Close And Expand Certification Report + +## Decision + +- Status: COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED +- Production authorized: False +- Production hold started: False +- Production rollback window closed: False +- Production expansions authorized: 0 +- Production mutations: 0 + +## Parent Evidence + +- Phase 9.2 status: COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED +- Authentic provider events: 25600 +- Semantic mismatches: 0 +- Parent production authorized: False + +## Isolated Control Plane + +- Provenance: TEST_CONTROL_PLANE_FIXTURE +- Accelerated time is production evidence: False +- Test hold status: PASSED +- Test hold production authorized: False +- Current no-go rejection: PREREQUISITE_DECISION_NOT_GO +- Local Phase 9.2 production eligible: False +- Expansion manifests: 5 +- Decommission decision: RUNTIME_STILL_REQUIRED_FOR_ROLLBACK + +## Persistence And Isolation + +- Migration: PASS +- Closure changed authority: False +- V1 health before/after: 200 / 200 +- V1 topology unchanged: True +- Disposable resources remaining: 0 + +## External Gates + +Phase 9.0-C remains NO_GO_EXTERNAL. There is no real Rust primary, production +hold duration, production consumer checkpoint set or operator closure approval. +No rollback window, expansion or Python decommission is authorized. diff --git a/upgrade/evidence/PHASE_A_RUST_MULTIVENUE_CORE_REPORT.md b/upgrade/evidence/PHASE_A_RUST_MULTIVENUE_CORE_REPORT.md new file mode 100644 index 0000000..9b254a0 --- /dev/null +++ b/upgrade/evidence/PHASE_A_RUST_MULTIVENUE_CORE_REPORT.md @@ -0,0 +1,105 @@ +# Phase A Rust Multi-Venue Realtime Core Report + +Date: 2026-08-19 + +## Decision + +- Status: `COMPLETE / ISOLATED CERTIFIED / CUTOVER NOT AUTHORIZED` +- Runtime authority: existing V1 remains authoritative +- Candidate authority used by certification: `RUST_SHADOW` +- Production/public/legacy writes: `0` +- Covered first-class products: Binance USD-M perpetual, Binance Spot, OKX + SWAP perpetual, OKX Spot, VN equities and VN derivatives. + +## Delivered Architecture + +- Added one provider-neutral `qdl-realtime-core` Rust crate. It validates the + versioned raw provider envelope, resolves an approved binding, enforces + source role and quantity units, canonicalizes, deduplicates across reconnect, + applies sequence policy and creates canonical/quarantine durable records. +- Added native Rust Binance/OKX acquisition with public/business socket + separation where required, heartbeat/reconnect/backoff and fenced raw Kafka + publication. Acquisition adapters never write Redis or public payloads. +- Added a Kafka consume-transform-produce transaction that atomically commits + canonical/quarantine outputs with the next raw consumer offsets. A crash + cannot acknowledge raw input without its corresponding output. +- Added a Python TLS/idempotent raw publisher for vendor-SDK/low-rate edges. + DNSE/vnstock remain Python acquisition adapters, but canonical semantics and + durable publication decisions belong to Rust. +- Added Binance latest-closed REST BAR acquisition. Binance TRADE/BBO remain + native Rust; BAR uses the low-rate Python history edge and then the same Rust + core. OKX TRADE/BBO/BAR remain native Rust. + +## Domain Corrections + +- Added explicit `BASE_ASSET`, `QUOTE_ASSET`, `CONTRACT` and `SHARE` quantity + units. BAR preserves native, base, quote and contract volume independently. +- Added native versus derived trade identity. DNSE trades without a native + trade ID use exact raw-capture identity; absent aggressor side/source time is + explicit in quality flags rather than fabricated. +- Removed hardcoded `SOURCE_ROLE_PRIMARY`; vnstock fallback remains secondary + and provider switches retain provenance. +- Binance Spot BBO may omit provider event time. The core uses receive time and + emits `SOURCE_TIME_MISSING` instead of rejecting or inventing a timestamp. +- DNSE legacy `stream:vn:*` naming no longer determines V2 semantics. Native + DNSE trade callbacks are canonical TRADE; QUOTE remains capability-gated. + +## Provider And Durability Evidence + +- Real provider shadow: 26 raw durable ACKs and 26 read-committed canonical + records, zero quarantine, across six product scopes. Binance/OKX crypto data + came from live WebSocket, Binance BAR from real latest-closed REST rows, VN + equity from a recent durable DNSE SDK-delivery snapshot and VN derivative BAR + from provider-derived canonical Parquet replay. Generated market events: 0. +- Transactional failure matrix: 8 raw records produced 6 canonical records, + one duplicate suppression and one intentional sequence-gap quarantine. The + transaction committed with one Kafka replica stopped, then full ISR restored; + read-committed visibility and counts remained exact. +- Capacity: 100,000 raw envelopes produced 100,000 canonical records with zero + duplicate/quarantine at approximately 154,139 events/s; p50 4.6 microseconds + and p99 10.8 microseconds on this host. +- All isolated containers, networks and volumes were removed. Scoped cleanup + also removed 2.8 GiB of Cargo target artifacts and the 2.91 GB disposable + Rust builder image. V1 topology was unchanged by inspection and health + remained HTTP 200 before and after every certification. + +Machine evidence: + +- `phase-a-real-provider-core.json` +- `phase-a-transactional-core.json` +- `phase-a-realtime-core-capacity.json` +- `contracts/golden/phase2/manifest-v2-stable-multivenue.json` + +## Verification + +- Python: 433 passed, 5 conditional integration skips. +- Rust: format and Clippy with warnings denied passed; 50 workspace tests plus + doc tests passed. +- Contracts: Buf format/lint and breaking checks passed against Phase 1 and + Phase 7 beta baselines; generated bindings are current. +- V2 OpenAPI semantic diff: 10 operations, 42 schemas, zero hard break. +- Python/Rust exact-byte corpus: 19 Binance/OKX/VN cases passed. + +## Defects Found And Closed + +1. Canonical quantities were unitless across products. +2. Source role was hardcoded primary. +3. DNSE missing trade identity/side could not be represented honestly. +4. The old Binance USD-M stream path was outdated after provider migration. +5. Binance Kline WebSocket emitted no frame on legacy, public-combined or + public-raw probes; BAR was moved to provider REST latest-closed acquisition, + not generated or silently omitted. +6. Binance Spot BBO omitted source timestamp and was initially quarantined. +7. Confluent Python headers required a list rather than a tuple. +8. Isolated Kafka network blocked venue DNS; only acquisition containers now + receive explicit egress during certification. +9. Multi-row provider frames could partially stage valid rows before a later row + failed. The core now rolls back ordering, dedup and partition state and emits + exactly one quarantine for the whole raw frame. + +## Remaining Boundary + +No in-scope Phase A defect remains. Phase B owns the stable Python +projector/query/stream deployment, registered consumer migration, immutable +`2.0.0` release and operator-approved per-slice cutover. This report does not +make Rust production-authoritative and does not change V1 runtime ownership. diff --git a/upgrade/evidence/PHASE_B4_RELEASE_CERTIFICATION_REPORT.md b/upgrade/evidence/PHASE_B4_RELEASE_CERTIFICATION_REPORT.md new file mode 100644 index 0000000..72ec51f --- /dev/null +++ b/upgrade/evidence/PHASE_B4_RELEASE_CERTIFICATION_REPORT.md @@ -0,0 +1,99 @@ +# Phase B.4 Release Certification And Cleanup + +Date: 2026-08-20 + +Conclusion: `PASS` for B.4. Data Layer `2.0.0 Internal Stable` artifacts are +ready for operator review. No push, merge, release publication, consumer +authority migration or production cutover was performed. Phase B overall +remains `PARTIAL_EXTERNAL` because the official DNSE provider path could not be +certified from this host; no synthetic substitute was accepted. + +## Frozen Artifacts + +- Code source: `2412572eaa89864ce74910b0f2e5f8b50833fb15`. +- Python: `qdl-v2-python:2.0.0-2412572`, image + `sha256:fec269ec555624baa68ee15fdd0281d72996e55f847b7347856be6b2fa51ea25`. +- Rust: `qdl-v2-rust:2.0.0-2412572`, image + `sha256:fbff0ed3c4390831a2aebf12f57c266eb6f01dde258b3cffacacbcbaa30d6c97`. +- Both images carry the exact OCI revision/version and run as non-root. +- Retained rollback: Python `2.0.0-c61fa39` and Rust `2.0.0-cfc0246`. + +## Correctness And Contract Gates + +- Final Python discovery: 506 cases, 500 passed and six explicit conditional + infrastructure skips. No domain assertion failed. +- Rust: format passed; locked Clippy with warnings denied passed; 62/62 + workspace tests passed. +- BAR ownership/checkpoint repair: 18/18 targeted tests passed. Five Phase B + modules ran 65 cases: 64 passed and one separately proven Redis integration + case skipped in network-disabled execution. +- Final package/deployment regression: 25/25 passed. +- Buf format/lint, two frozen-baseline breaking checks and generated-artifact + equality passed. V1 OpenAPI semantic compatibility had zero removals or + incompatible changes. +- Final Rust benchmark processed 100,000 events at 133,477.5 events/s with + p99 14,124 ns, zero duplicate and zero quarantine; gate was 50,000 events/s. + +## Security Gates + +- Cargo-deny passed advisories, bans, licenses and sources with the tracked + policy. Pip-audit found no known Python vulnerability. +- Pinned Trivy 0.73.0, using the repository CI policy `ignore-unfixed=true`, + found zero fixable HIGH/CRITICAL vulnerability and zero secret in both final + images. A stricter diagnostic without that policy reported only currently + unfixed Debian findings; it was not hidden or used to weaken the release + policy. + +## Real-Provider Runtime Acceptance + +- Fresh isolated RF3/minISR2 Kafka accepted 500 closed one-minute BARs for each + Binance USD-M, Binance Spot, OKX SWAP and OKX Spot binding: 2,000 total. +- Acquisition revision 2 used one REST owner for final BARs. Native Rust + ingestors retained TRADE/QUOTE; Rust remained the only canonical core. +- Restart restored the ACK-authoritative checkpoint, skipped overlapping + bootstrap and caught up the exact closed-bar backlog. Kafka quarantine stayed + zero across all six partitions. +- Cache contained 75,187 canonical records across 12 bounded partitions, + maximum 10,000 each, with zero internal offset gap, duplicate event ID or + quarantine row. Projector lag was 35 against the 250 bound. +- Redis contained 51 bounded projection keys and used 1.22 MiB of its 128 MiB + maxmemory. Largest Python role used 68.86 MiB/512 MiB; Rust roles used at + most 24.56 MiB/256 MiB; Kafka brokers remained below 439 MiB/768 MiB. +- Application log scan over the acceptance window found no warning, error, + panic, collision, stale generation or unresolved gap. + +## Consumer Acceptance + +- Signed SDK warmup returned 500 final real-provider BARs for both Binance and + OKX. Both query replicas had identical identity, payload, contract, source, + watermark and quality semantics after excluding request-clock freshness. +- Binance and OKX alpha streams emitted `REPLAYING -> LIVE`; acknowledged + offsets were contiguous and a new client resumed exactly at prior offset + 1. +- Trading System paper read Binance/OKX TRADE and QUOTE snapshots at 132-158 ms + freshness; all four were authoritative and execution eligible. +- Monitoring read authoritative Binance/OKX TRADE at 100-180 ms freshness. +- No order, synthetic market event or production state mutation occurred. + +## Cleanup And V1 Invariant + +- Removed the fresh `qdl_v2_b4_candidate` project, its networks and all five + disposable volumes. +- Removed all stopped containers/networks from `qdl_v2_stable_candidate`, then + deleted only its approved Kafka1/Kafka2/Kafka3/state test volumes. Preserved + `qdl_v2_stable_candidate_stable_tls`. +- Removed three B.4 builder image tags, superseded `ea84a21` Python/Rust tags + and the unused Python `cfc0246` tag. Final images and one tested rollback pair + remain. +- Pruned only 41 exact BuildKit IDs from the B.4 build window. Cache fell from + 168 records/12.94 GB to 154 records/10.94 GB. No broad prune ran. +- Deleted the exact temporary bundle containing test secrets, Trivy JSON and the + SDK acceptance harness after bounded evidence was recorded. +- V1 containers were not restarted; port 8100 remained `status=ok`, Redis true, + recent queue drops zero and DNSE `OPEN_HEALTHY` after cleanup. + +## Decision Boundary + +B.4 does not authorize V2 authority, route or consumer cutover. The next action +is review and PR into `dev`. Any later deployment requires the exact topology, +ports, image digests, credentials, volumes, affected consumer manifests and +rollback command to be approved as a separate transaction. diff --git a/upgrade/evidence/phase-a-real-provider-core.json b/upgrade/evidence/phase-a-real-provider-core.json new file mode 100644 index 0000000..b39e71b --- /dev/null +++ b/upgrade/evidence/phase-a-real-provider-core.json @@ -0,0 +1,102 @@ +{ + "authority": "RUST_SHADOW", + "canonical_digest": "88a0000ebdf2210dae33592f3759b65eee805172e55e23f000858bc0ab5ce8f6", + "canonical_read_committed": 26, + "cleanup": { + "containers_after": 0, + "networks_after": 0, + "v1_health_after": 200, + "v1_topology_unchanged": true, + "volumes_after": 0 + }, + "dnse_snapshot_age_seconds_at_test": 154.115, + "dnse_symbol": "GEX", + "duplicates_suppressed": 0, + "native_runs": [ + { + "accepted_raw_frames": 3, + "channel": "btcusdt@trade", + "market": "USDM", + "runtime": "BINANCE" + }, + { + "accepted_raw_frames": 2, + "channel": "btcusdt@bookTicker", + "market": "USDM", + "runtime": "BINANCE" + }, + { + "accepted_raw_frames": 3, + "channel": "btcusdt@trade", + "market": "SPOT", + "runtime": "BINANCE" + }, + { + "accepted_raw_frames": 2, + "channel": "btcusdt@bookTicker", + "market": "SPOT", + "runtime": "BINANCE" + }, + { + "accepted_raw_frames": 3, + "channel": "trades", + "market": "SWAP", + "runtime": "OKX" + }, + { + "accepted_raw_frames": 2, + "channel": "bbo-tbt", + "market": "SWAP", + "runtime": "OKX" + }, + { + "accepted_raw_frames": 1, + "channel": "candle1m", + "market": "SWAP", + "runtime": "OKX" + }, + { + "accepted_raw_frames": 3, + "channel": "trades", + "market": "SPOT", + "runtime": "OKX" + }, + { + "accepted_raw_frames": 2, + "channel": "bbo-tbt", + "market": "SPOT", + "runtime": "OKX" + }, + { + "accepted_raw_frames": 1, + "channel": "candle1m", + "market": "SPOT", + "runtime": "OKX" + } + ], + "products": [ + "BINANCE/SPOT/SPOT", + "BINANCE/USDM/PERPETUAL", + "HNX/VN_DERIVATIVES/FUTURE", + "HOSE/EQUITIES/COMMON_STOCK", + "OKX/SPOT/SPOT", + "OKX/SWAP/PERPETUAL" + ], + "provenance": { + "binance_bar": "REAL_BINANCE_REST_LATEST_CLOSED", + "crypto_trade_bbo": "REAL_PROVIDER_WEBSOCKET", + "generated_market_data": 0, + "okx_bar": "REAL_PROVIDER_WEBSOCKET", + "vn_derivative": "REAL_DNSE_PROVIDER_DERIVED_CANONICAL_PARQUET_REPLAY", + "vn_derivative_capture_boundary": "REPLAY_BYTES", + "vn_equity": "REAL_DNSE_PROVIDER_DERIVED_DURABLE_REPLAY", + "vn_equity_capture_boundary": "SDK_DELIVERY" + }, + "quarantines": 0, + "raw_acked": 26, + "runtime_cutover": false, + "rust_image_id": "sha256:7ff46a43188dd65cbb7c9ec16347c0bbd19e1e2c4a8bca6f403b728bf1b6e79c", + "schema": "qdl.phase-a.real-provider-core-certification.v1", + "status": "PASS", + "v1_health_before": 200 +} diff --git a/upgrade/evidence/phase-a-realtime-core-capacity.json b/upgrade/evidence/phase-a-realtime-core-capacity.json new file mode 100644 index 0000000..f1b1590 --- /dev/null +++ b/upgrade/evidence/phase-a-realtime-core-capacity.json @@ -0,0 +1,17 @@ +{ + "binary_sha256": "02972e3e09c9c40d6499ebf9e3fc3caf8d3cdd016ea75c200821c1e54d333321", + "canonical": 100000, + "duplicates": 0, + "elapsed_seconds": 0.648765929, + "events": 100000, + "events_per_second": 154138.79726103804, + "minimum_events_per_second": 50000.0, + "output_bytes": 45550175, + "p50_ns": 4539, + "p99_ns": 10760, + "production_writes": 0, + "provenance": "SYNTHETIC_CAPACITY_ONLY", + "quarantines": 0, + "schema": "qdl.phase-a.realtime-core-capacity.v1", + "status": "PASS" +} diff --git a/upgrade/evidence/phase-a-transactional-core.json b/upgrade/evidence/phase-a-transactional-core.json new file mode 100644 index 0000000..28bc877 --- /dev/null +++ b/upgrade/evidence/phase-a-transactional-core.json @@ -0,0 +1,33 @@ +{ + "authority": "RUST_SHADOW", + "canonical_digest": "0bea64d8c43f6124c2db5921eda008e00a82b3ef16b415a65df6e3178ca73c1b", + "canonical_read_committed": 6, + "cleanup": { + "containers_after": 0, + "networks_after": 0, + "v1_health_after": 200, + "v1_topology_unchanged": true, + "volumes_after": 0 + }, + "duplicates_suppressed": 1, + "one_replica_loss_elapsed_seconds": 7.889127, + "one_replica_loss_transaction_committed": true, + "python_image_id": "sha256:f1cd07c18b3816e6da57d1ae99ffe90c908be1fc99cd309392911f9fc9e179f1", + "quarantine_read_committed": 1, + "quarantine_reasons": [ + 7 + ], + "raw_acked": 8, + "replicas_restored_before_inspection": true, + "runtime_cutover": false, + "rust_image_id": "sha256:7ff46a43188dd65cbb7c9ec16347c0bbd19e1e2c4a8bca6f403b728bf1b6e79c", + "schema": "qdl.phase-a.transactional-core-certification.v1", + "status": "PASS", + "v1_health_before": 200, + "venues": [ + "BINANCE", + "HNX", + "HOSE", + "OKX" + ] +} diff --git a/upgrade/evidence/phase90a-evidence.sha256 b/upgrade/evidence/phase90a-evidence.sha256 new file mode 100644 index 0000000..327ec5f --- /dev/null +++ b/upgrade/evidence/phase90a-evidence.sha256 @@ -0,0 +1,2 @@ +cfd839f5482aa0b46e238fb8b4c47094648c79adf5bbce9ece8f57205e4eb761 upgrade/evidence/PHASE90A_RUNTIME_CORRECTNESS_REPORT.md +56923c6d7647156c6adb92004b6fbc7e2cb718cc6298dc5c5a46a8e0e546fdb8 upgrade/evidence/phase90a-runtime-correctness.json diff --git a/upgrade/evidence/phase90a-runtime-correctness.json b/upgrade/evidence/phase90a-runtime-correctness.json new file mode 100644 index 0000000..3513f3d --- /dev/null +++ b/upgrade/evidence/phase90a-runtime-correctness.json @@ -0,0 +1,85 @@ +{ + "schema_version": "qdl.phase-evidence.v1", + "phase": "9.0-A", + "name": "Runtime Correctness Closure", + "captured_at": "2026-08-18T11:05:00Z", + "status": "PASS_ISOLATED_NO_PRODUCTION_CUTOVER", + "branch": "feat/phase9-runtime-correctness", + "commits": [ + "765a7f1", + "7302c45" + ], + "tests": { + "targeted": {"run": 35, "passed": 35, "failed": 0}, + "full": {"run": 345, "passed": 340, "failed": 0, "skipped": 5}, + "compileall": "PASS", + "diff_check": "PASS" + }, + "v1_contract": { + "live_path_count": 40, + "candidate_path_count": 40, + "added_paths": [], + "removed_paths": [] + }, + "candidate": { + "source_commit": "7302c45", + "image_digest": "sha256:4a2723ec39057c75a89889d955feac7acc6fb01bc126a579f8c74d384b9b6999", + "runtime_uid_gid": "10001:10001", + "read_only_root": true, + "source_bind": false, + "loopback_port": 18120, + "limits": {"cpus": 1.5, "memory_bytes": 1610612736, "pids": 256} + }, + "real_provider_smoke": { + "binance_usdm_trade": {"shards": 8, "producing": 8, "ready": true}, + "binance_usdm_kline": {"shards": 8, "producing": 0, "ready": false}, + "top_level_health_with_kline_unavailable": "degraded", + "queue_drop_count": 0, + "queue_pressure_count": 0, + "recovery": { + "symbol": "BTCUSDT", + "interval": "1m", + "event": "kline_recovery", + "recovery_source": "BINANCE_REST_GAP_FILL", + "provider": "binance", + "market": "binance_usdm", + "authoritative": true, + "final": true, + "open_time_ms": 1787050680000, + "close_time_ms": 1787050739999, + "provider_open_time_equal": true, + "provider_close_time_equal": true, + "provider_ohlcv_equal": true + }, + "demand_expiry": { + "lease_count_after_expiry": 0, + "active_recovery_demands_after_expiry": 0, + "provider_fetch_count_before_observation": 2, + "provider_fetch_count_after_one_poll": 2 + } + }, + "resource_snapshot": { + "candidate_app": {"cpu_percent": 20.07, "memory_mib": 131.0, "pids": 16}, + "candidate_redis": {"cpu_percent": 2.27, "memory_mib": 5.129, "pids": 5}, + "note": "One non-soak snapshot while broad trade and failing kline shards were active." + }, + "production_v1": { + "container_id": "0e0eb56c78ba9d6b816cde3461bc79425ceea8095a71518a608a87294388035f", + "image": "data-layer:v0.1.0", + "started_at": "2026-08-16T11:09:12.471324413Z", + "loopback_port": 8100, + "restarted": false, + "state_mutated": false + }, + "cleanup": { + "candidate_containers": 0, + "candidate_networks": 0, + "candidate_volumes": 0, + "candidate_images": 0 + }, + "decision": { + "phase_90a": "GO_FOR_REVIEW_AND_OPERATOR_APPROVED_ROLLOUT", + "phase_91": "BLOCKED", + "reason": "Phase 9.1 still requires the independent production infrastructure and exact-slice approval listed in the implementation plan." + } +} diff --git a/upgrade/evidence/phase90b-capacity.json b/upgrade/evidence/phase90b-capacity.json new file mode 100644 index 0000000..3107ff2 --- /dev/null +++ b/upgrade/evidence/phase90b-capacity.json @@ -0,0 +1,197 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "burst": { + "concurrency": 20, + "errors": 0, + "latency_ms": { + "p50": 282.888, + "p95": 417.75, + "p99": 486.649, + "p99_9": 486.649 + }, + "requests": 60, + "requests_per_second": 65.012, + "response_bytes_per_second": 4072560.251 + }, + "checks": { + "auth_fail_closed": true, + "burst_rate": true, + "cleanup_exact": true, + "container_cpu_bounded": true, + "container_rss_bounded": true, + "cursor_expiry": true, + "cursor_scope": true, + "cursor_tamper": true, + "dependency_fail_closed": true, + "dependency_readiness_degraded": true, + "dependency_recovered": true, + "durable_store_growth_bounded": true, + "freshness": true, + "malformed": true, + "normal_rate": true, + "oversized": true, + "partition_epoch_advanced": true, + "production_beta_keys_absent": true, + "query_errors": true, + "query_tail": true, + "rate_limit": true, + "redis_growth_bounded": true, + "rotation": true, + "slow_consumer_isolated": true, + "stream_contiguous": true, + "stream_drained": true, + "v1_fallback_available": true, + "v1_topology_unchanged": true + }, + "decision_scope": "READ_ONLY_PUBLIC_BETA", + "end_to_end_freshness_ms": 6695.751, + "error_budget_consumption": 0.0, + "generated_market_events": 0, + "normal": { + "concurrency": 5, + "errors": 0, + "latency_ms": { + "p50": 39.419, + "p95": 63.325, + "p99": 72.211, + "p99_9": 72.211 + }, + "requests": 30, + "requests_per_second": 110.204, + "response_bytes_per_second": 6903513.048 + }, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "resources": { + "containers": { + "qdl_phase90b_matrix-qdl_beta_redis-1": { + "cpu_percent_peak": 3.52, + "network_rx_bytes_peak": 108000.0, + "network_tx_bytes_peak": 25600.0, + "rss_bytes_peak": 3895459.84 + }, + "qdl_phase90b_matrix-qdl_query_v2_beta-1": { + "cpu_percent_peak": 33.36, + "network_rx_bytes_peak": 1320000.0, + "network_tx_bytes_peak": 5850000.0, + "rss_bytes_peak": 55658414.08 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_a-1": { + "cpu_percent_peak": 9.96, + "network_rx_bytes_peak": 114000.0, + "network_tx_bytes_peak": 320000.0, + "rss_bytes_peak": 55364812.8 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_b-1": { + "cpu_percent_peak": 0.19, + "network_rx_bytes_peak": 4370.0, + "network_tx_bytes_peak": 5320.0, + "rss_bytes_peak": 52963573.76 + } + }, + "durable_store_growth_bytes": 73728, + "max_cpu_percent_of_one_core": 33.36, + "max_rss_bytes": 55658414.08, + "redis_growth_bytes": -35392 + }, + "runtime_failure_and_cleanup": { + "beta_containers_after": 0, + "beta_networks_after": 0, + "beta_volumes_after": 0, + "cursor_files_after": 0, + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "durable_store_bytes_after": 233472, + "durable_store_bytes_before": 159744, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "production_beta_keys_after": 0, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "redis_used_memory_after": 1174600, + "redis_used_memory_before": 1209992, + "schema": "qdl.phase7.3.runtime-evidence.v1", + "v1_fallback_status": 200, + "v1_topology_unchanged": true + }, + "schema": "qdl.phase7.capacity.v1", + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "stream": { + "bytes_per_second": 978613.568, + "cursor_lag_after": 0, + "cursor_lag_before": 64, + "disconnect_count": 1, + "events_per_second": 1815.252, + "fanout_consumers": 4, + "fast_consumers": 3, + "fast_results": [ + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 322.573, + "p95": 366.45, + "p99": 414.929, + "p99_9": 414.929 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 335.833, + "p95": 375.993, + "p99": 415.266, + "p99_9": 415.266 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 352.33, + "p95": 385.548, + "p99": 415.301, + "p99_9": 415.301 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + } + ], + "replay_count": 4, + "replay_lag_after": 0, + "replayed_events_per_fast_consumer": 66, + "slow_consumer_events_before_disconnect": 65, + "slow_consumer_explicit_disconnect": true, + "slow_consumers": 1, + "subscriber_peak": 4 + }, + "thresholds": { + "burst_min_requests_per_second": 20.0, + "max_container_cpu_percent_of_one_core": 100.0, + "max_container_rss_bytes": 536870912, + "max_durable_store_growth_bytes": 33554432, + "max_end_to_end_freshness_ms": 240000, + "max_error_budget_fraction": 0.0, + "max_redis_growth_bytes": 16777216, + "normal_min_requests_per_second": 10.0, + "query_max_p99_9_ms": 1000.0 + } +} diff --git a/upgrade/evidence/phase90b-continuous-bridge.json b/upgrade/evidence/phase90b-continuous-bridge.json new file mode 100644 index 0000000..7aded5d --- /dev/null +++ b/upgrade/evidence/phase90b-continuous-bridge.json @@ -0,0 +1,32 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "canonical_mismatches": 0, + "duplicate_open_times": 0, + "execution_eligible_events": 0, + "first": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "generated_market_events": 0, + "non_final_bars": 0, + "schema": "qdl.phase9.0-b.bridge-parity.v1", + "second": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "slice": { + "feed": "BAR", + "interval": "1m", + "market": "USDM", + "product_type": "PERPETUAL", + "symbol": "BTCUSDT", + "venue": "BINANCE" + }, + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "watermark_delta": 0 +} diff --git a/upgrade/evidence/phase90b-evidence.sha256 b/upgrade/evidence/phase90b-evidence.sha256 new file mode 100644 index 0000000..a165209 --- /dev/null +++ b/upgrade/evidence/phase90b-evidence.sha256 @@ -0,0 +1,5 @@ +7079cc3c349ff84b0f9c9e5f0b346098b68328514a8c0da31d7d467225c55d5d upgrade/evidence/phase90b-capacity.json +1a3d30f6fa4085b9fee7cdad0565ea2fba5fb3273ba690de881073f923f59bb4 upgrade/evidence/phase90b-security-adversarial.json +10bc592d28d821187f93e81f9159266f5079803d8e1a8737d997480b739e3a7b upgrade/evidence/phase90b-continuous-bridge.json +76e196fc06c86045409269cd1d00ead0c4d0f6eac753e850deedee4b47ca5742 upgrade/evidence/phase90b-isolated-v2-beta.json +2985867dd88e3bced8d31b341709f5b23a818be6e4465eaf3e4da5e20777388a upgrade/evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md diff --git a/upgrade/evidence/phase90b-isolated-v2-beta.json b/upgrade/evidence/phase90b-isolated-v2-beta.json new file mode 100644 index 0000000..b65945a --- /dev/null +++ b/upgrade/evidence/phase90b-isolated-v2-beta.json @@ -0,0 +1,345 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "candidate": { + "bridge_networks": [ + "bobby_network", + "qdl_phase90b_bridge_qdl_beta_internal" + ], + "image_id": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "read_only_root": true, + "release": "phase90b-isolated-beta", + "runtime_user": "10001:10001", + "source_bind_mounts": 0, + "source_revision": "1c881389b4ee21a153903505822c61512b176044" + }, + "checks": { + "bridge_network_isolated": true, + "candidate_no_source_bind": true, + "candidate_non_root": true, + "candidate_read_only": true, + "candidate_revision_labeled": true, + "cleanup_exact": true, + "continuous_bridge_parity": true, + "phase73_capacity": true, + "phase73_security": true, + "production_beta_keys_absent": true, + "provider_authentic": true, + "v1_openapi_unchanged": true, + "v1_topology_unchanged": true + }, + "cleanup": { + "containers": 0, + "images": 0, + "networks": 0, + "production_beta_keys": 0, + "volumes": 0 + }, + "continuous_bridge": { + "authority": "V1_SHADOW_READ_ONLY", + "canonical_mismatches": 0, + "duplicate_open_times": 0, + "execution_eligible_events": 0, + "first": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "generated_market_events": 0, + "non_final_bars": 0, + "schema": "qdl.phase9.0-b.bridge-parity.v1", + "second": { + "count": 30, + "first_open_time_ns": 1787053080000000000, + "last_open_time_ns": 1787054820000000000, + "watermark_offset": 119 + }, + "slice": { + "feed": "BAR", + "interval": "1m", + "market": "USDM", + "product_type": "PERPETUAL", + "symbol": "BTCUSDT", + "venue": "BINANCE" + }, + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "watermark_delta": 0 + }, + "contract_and_capacity": { + "authority": "V1_SHADOW_READ_ONLY", + "burst": { + "concurrency": 20, + "errors": 0, + "latency_ms": { + "p50": 282.888, + "p95": 417.75, + "p99": 486.649, + "p99_9": 486.649 + }, + "requests": 60, + "requests_per_second": 65.012, + "response_bytes_per_second": 4072560.251 + }, + "checks": { + "auth_fail_closed": true, + "burst_rate": true, + "cleanup_exact": true, + "container_cpu_bounded": true, + "container_rss_bounded": true, + "cursor_expiry": true, + "cursor_scope": true, + "cursor_tamper": true, + "dependency_fail_closed": true, + "dependency_readiness_degraded": true, + "dependency_recovered": true, + "durable_store_growth_bounded": true, + "freshness": true, + "malformed": true, + "normal_rate": true, + "oversized": true, + "partition_epoch_advanced": true, + "production_beta_keys_absent": true, + "query_errors": true, + "query_tail": true, + "rate_limit": true, + "redis_growth_bounded": true, + "rotation": true, + "slow_consumer_isolated": true, + "stream_contiguous": true, + "stream_drained": true, + "v1_fallback_available": true, + "v1_topology_unchanged": true + }, + "decision_scope": "READ_ONLY_PUBLIC_BETA", + "end_to_end_freshness_ms": 6695.751, + "error_budget_consumption": 0.0, + "generated_market_events": 0, + "normal": { + "concurrency": 5, + "errors": 0, + "latency_ms": { + "p50": 39.419, + "p95": 63.325, + "p99": 72.211, + "p99_9": 72.211 + }, + "requests": 30, + "requests_per_second": 110.204, + "response_bytes_per_second": 6903513.048 + }, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "resources": { + "containers": { + "qdl_phase90b_matrix-qdl_beta_redis-1": { + "cpu_percent_peak": 3.52, + "network_rx_bytes_peak": 108000.0, + "network_tx_bytes_peak": 25600.0, + "rss_bytes_peak": 3895459.84 + }, + "qdl_phase90b_matrix-qdl_query_v2_beta-1": { + "cpu_percent_peak": 33.36, + "network_rx_bytes_peak": 1320000.0, + "network_tx_bytes_peak": 5850000.0, + "rss_bytes_peak": 55658414.08 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_a-1": { + "cpu_percent_peak": 9.96, + "network_rx_bytes_peak": 114000.0, + "network_tx_bytes_peak": 320000.0, + "rss_bytes_peak": 55364812.8 + }, + "qdl_phase90b_matrix-qdl_stream_v2_beta_b-1": { + "cpu_percent_peak": 0.19, + "network_rx_bytes_peak": 4370.0, + "network_tx_bytes_peak": 5320.0, + "rss_bytes_peak": 52963573.76 + } + }, + "durable_store_growth_bytes": 73728, + "max_cpu_percent_of_one_core": 33.36, + "max_rss_bytes": 55658414.08, + "redis_growth_bytes": -35392 + }, + "runtime_failure_and_cleanup": { + "beta_containers_after": 0, + "beta_networks_after": 0, + "beta_volumes_after": 0, + "cursor_files_after": 0, + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "durable_store_bytes_after": 233472, + "durable_store_bytes_before": 159744, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "production_beta_keys_after": 0, + "provenance": { + "git_sha": "1c881389b4ee21a153903505822c61512b176044", + "image": "sha256:f8e3fba5e0f1bd2fc0b524a4eddcde906b952a1bbc3ac3591664c6c315475e26", + "machine": "x86_64", + "platform": "Linux-6.17.0-1017-aws-x86_64-with-glibc2.39" + }, + "redis_used_memory_after": 1174600, + "redis_used_memory_before": 1209992, + "schema": "qdl.phase7.3.runtime-evidence.v1", + "v1_fallback_status": 200, + "v1_topology_unchanged": true + }, + "schema": "qdl.phase7.capacity.v1", + "source": "REAL_V1_PROVIDER_DATA", + "status": "PASS", + "stream": { + "bytes_per_second": 978613.568, + "cursor_lag_after": 0, + "cursor_lag_before": 64, + "disconnect_count": 1, + "events_per_second": 1815.252, + "fanout_consumers": 4, + "fast_consumers": 3, + "fast_results": [ + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 322.573, + "p95": 366.45, + "p99": 414.929, + "p99_9": 414.929 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 335.833, + "p95": 375.993, + "p99": 415.266, + "p99_9": 415.266 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + }, + { + "contiguous": true, + "delivery_latency_ms": { + "p50": 352.33, + "p95": 385.548, + "p99": 415.301, + "p99_9": 415.301 + }, + "event_bytes": 35581, + "events": 66, + "first_offset": 54, + "last_offset": 119 + } + ], + "replay_count": 4, + "replay_lag_after": 0, + "replayed_events_per_fast_consumer": 66, + "slow_consumer_events_before_disconnect": 65, + "slow_consumer_explicit_disconnect": true, + "slow_consumers": 1, + "subscriber_peak": 4 + }, + "thresholds": { + "burst_min_requests_per_second": 20.0, + "max_container_cpu_percent_of_one_core": 100.0, + "max_container_rss_bytes": 536870912, + "max_durable_store_growth_bytes": 33554432, + "max_end_to_end_freshness_ms": 240000, + "max_error_budget_fraction": 0.0, + "max_redis_growth_bytes": 16777216, + "normal_min_requests_per_second": 10.0, + "query_max_p99_9_ms": 1000.0 + } + }, + "decision_scope": "V2_BETA_READ_ONLY_REVIEW", + "phase_91": "BLOCKED", + "production_v1": { + "openapi_sha256": "142f5beb499ce7a27bf296462025ce65a6b204ee1171afd7511bce353bf79e38", + "restarted_or_recreated": false, + "topology_unchanged": true + }, + "resource_snapshot": [ + { + "CPUPerc": "2.17%", + "MemPerc": "3.50%", + "MemUsage": "3.363MiB / 96MiB", + "Name": "qdl_phase90b_bridge-qdl_beta_redis-1", + "NetIO": "16kB / 3.92kB", + "PIDs": "6" + }, + { + "CPUPerc": "0.00%", + "MemPerc": "0.00%", + "MemUsage": "0B / 0B", + "Name": "qdl_phase90b_bridge-qdl_beta_v1_bridge-1", + "NetIO": "0B / 0B", + "PIDs": "0" + }, + { + "CPUPerc": "8.43%", + "MemPerc": "10.51%", + "MemUsage": "53.79MiB / 512MiB", + "Name": "qdl_phase90b_bridge-qdl_query_v2_beta-1", + "NetIO": "12.5kB / 258kB", + "PIDs": "8" + }, + { + "CPUPerc": "0.16%", + "MemPerc": "9.83%", + "MemUsage": "50.33MiB / 512MiB", + "Name": "qdl_phase90b_bridge-qdl_stream_v2_beta_a-1", + "NetIO": "443kB / 12.6kB", + "PIDs": "14" + }, + { + "CPUPerc": "0.56%", + "MemPerc": "9.97%", + "MemUsage": "51.04MiB / 512MiB", + "Name": "qdl_phase90b_bridge-qdl_stream_v2_beta_b-1", + "NetIO": "443kB / 11.6kB", + "PIDs": "14" + } + ], + "schema": "qdl.phase9.0-b.isolated-beta.v1", + "security": { + "authority": "V1_SHADOW_READ_ONLY", + "schema": "qdl.phase7.security-adversarial.v1", + "status": "PASS", + "tests": { + "auth_status_codes": { + "consumer_mismatch": 403, + "missing_scope": 401, + "missing_token": 401, + "rotation_beta-jwt-k1": 200, + "rotation_beta-jwt-k2": 200, + "wrong_audience": 401, + "wrong_environment": 401 + }, + "cursor_consumer_scope": "CURSOR_INVALID", + "cursor_expiry": "CURSOR_EXPIRED", + "cursor_tamper": "CURSOR_INVALID", + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "malformed_request_status": 400, + "oversized_request_status": 413, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "rate_limit_requests_until_reject": 119, + "rate_limit_status": 429 + } + }, + "status": "PASS_ISOLATED_NO_AUTHORITY_CUTOVER" +} diff --git a/upgrade/evidence/phase90b-security-adversarial.json b/upgrade/evidence/phase90b-security-adversarial.json new file mode 100644 index 0000000..a61ff15 --- /dev/null +++ b/upgrade/evidence/phase90b-security-adversarial.json @@ -0,0 +1,28 @@ +{ + "authority": "V1_SHADOW_READ_ONLY", + "schema": "qdl.phase7.security-adversarial.v1", + "status": "PASS", + "tests": { + "auth_status_codes": { + "consumer_mismatch": 403, + "missing_scope": 401, + "missing_token": 401, + "rotation_beta-jwt-k1": 200, + "rotation_beta-jwt-k2": 200, + "wrong_audience": 401, + "wrong_environment": 401 + }, + "cursor_consumer_scope": "CURSOR_INVALID", + "cursor_expiry": "CURSOR_EXPIRED", + "cursor_tamper": "CURSOR_INVALID", + "dependency_outage_query_status": 503, + "dependency_outage_ready_status": 503, + "dependency_recovery_ready_status": 200, + "malformed_request_status": 400, + "oversized_request_status": 413, + "partition_epoch_after": 2, + "partition_epoch_before": 1, + "rate_limit_requests_until_reject": 119, + "rate_limit_status": 429 + } +} diff --git a/upgrade/evidence/phase90c-authority-migration.json b/upgrade/evidence/phase90c-authority-migration.json new file mode 100644 index 0000000..0e8aa04 --- /dev/null +++ b/upgrade/evidence/phase90c-authority-migration.json @@ -0,0 +1,17 @@ +{ + "audit_mutation_rejected": true, + "audit_records": 2, + "candidate_mismatch_rejected": true, + "final_test_state": "RUST_CANARY:3:rust-canary-owner:2", + "hold_beyond_bundle_expiry_rejected": true, + "idempotent_migration": true, + "missing_hold_window_rejected": true, + "missing_terminal_watermark_rejected": true, + "no_go_bundle_rejected": true, + "prerequisite_bundles": 3, + "production_mutations": 0, + "schema": "qdl.phase9.0-c.authority-migration.v1", + "stale_cas_rejected": true, + "stale_owner_rejected": true, + "status": "PASS" +} diff --git a/upgrade/evidence/phase90c-evidence.sha256 b/upgrade/evidence/phase90c-evidence.sha256 new file mode 100644 index 0000000..338151d --- /dev/null +++ b/upgrade/evidence/phase90c-evidence.sha256 @@ -0,0 +1,3 @@ +df03cc15c8f3eba20ef3a64633412ab6e92e5c68f437dfd83c575e7921fa8a14 upgrade/evidence/phase90c-production-prerequisites.json +3953e06ff24ca38bae3a747da5dbceb55264243f4979b2348fa74c50cba15bc4 upgrade/evidence/PHASE90C_PRODUCTION_PREREQUISITES_REPORT.md +00e35812878357974a1c053c9f10a03bc58645ff83f3dc9768fd14c9cefd3f2d upgrade/evidence/phase90c-authority-migration.json diff --git a/upgrade/evidence/phase90c-production-prerequisites.json b/upgrade/evidence/phase90c-production-prerequisites.json new file mode 100644 index 0000000..629b0b2 --- /dev/null +++ b/upgrade/evidence/phase90c-production-prerequisites.json @@ -0,0 +1,171 @@ +{ + "authority_state": "RUST_SHADOW", + "blocked": 12, + "bundle_id": "558042db-a766-5a55-b5b3-4b508d649df9", + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "decision": "NO_GO_EXTERNAL", + "gates": [ + { + "evidence_id": "phase8-local-kafka-failover", + "gate_id": "replicated_durable_transport", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-local-otel-debug", + "gate_id": "production_observability", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-local-kafka-mtls", + "gate_id": "workload_identity_rbac_network", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "external_secret_rotation", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-signature-rehearsal-no-admission", + "gate_id": "signed_artifact_admission", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "EVIDENCE_BLOCKED", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "postgres_pitr", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "object_store_restore", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "independent_failure_domain_dr", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "INDEPENDENT_FAILURE_DOMAIN" + }, + { + "evidence_id": "phase8-local-projection-rebuild", + "gate_id": "redis_projector_rebuild", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase90b-partial-consumer-beta", + "gate_id": "consumer_registration_rollback", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "EVIDENCE_BLOCKED", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": "phase8-local-authority-rehearsal", + "gate_id": "persistent_authority_sink_fencing", + "observed_scope": "LOCAL_REHEARSAL", + "passed": false, + "reason": "INSUFFICIENT_SCOPE", + "required_scope": "PRODUCTION" + }, + { + "evidence_id": null, + "gate_id": "exact_slice_approval", + "observed_scope": null, + "passed": false, + "reason": "MISSING_EVIDENCE", + "required_scope": "PRODUCTION" + } + ], + "inventory_sha256": "9447bc6775d359f5f38d521441373b93bcb95492f0cd5edf434fceb82fec160a", + "issued_at_ns": 1787065065239561211, + "passed": 0, + "policy_revision": 1, + "production_mutations": 0, + "schema": "qdl.production-prerequisite-decision.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "v1_after": { + "id": "0e0eb56c78ba9d6b816cde3461bc79425ceea8095a71518a608a87294388035f", + "image": "data-layer:v0.1.0", + "mounts": [ + [ + "/home/bobby/data_layer", + "/app", + true + ], + [ + "/home/bobby/data_layer/data", + "/app/data", + true + ], + [ + "/home/bobby/data_layer/logs", + "/app/logs", + true + ] + ], + "networks": [ + "bobby_network", + "executor_network" + ], + "present": true, + "restart_count": 0, + "started_at": "2026-08-16T11:09:12.471324413Z" + }, + "v1_before": { + "id": "0e0eb56c78ba9d6b816cde3461bc79425ceea8095a71518a608a87294388035f", + "image": "data-layer:v0.1.0", + "mounts": [ + [ + "/home/bobby/data_layer", + "/app", + true + ], + [ + "/home/bobby/data_layer/data", + "/app/data", + true + ], + [ + "/home/bobby/data_layer/logs", + "/app/logs", + true + ] + ], + "networks": [ + "bobby_network", + "executor_network" + ], + "present": true, + "restart_count": 0, + "started_at": "2026-08-16T11:09:12.471324413Z" + }, + "v1_health_after": 200, + "v1_health_before": 200, + "v1_unchanged": true +} diff --git a/upgrade/evidence/phase91-evidence.sha256 b/upgrade/evidence/phase91-evidence.sha256 new file mode 100644 index 0000000..0a19262 --- /dev/null +++ b/upgrade/evidence/phase91-evidence.sha256 @@ -0,0 +1,2 @@ +1b83b1f4533f618f5eb66a0806c85fbe83204c52ce43c312d5fb58fe368ba7ed upgrade/evidence/phase91-rust-canary-certification.json +7f2a1e79e3185151c484a6a27094b8410e5b94e73849ada8aa8dbdf092aa9581 upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md diff --git a/upgrade/evidence/phase91-rust-canary-certification.json b/upgrade/evidence/phase91-rust-canary-certification.json new file mode 100644 index 0000000..15d547e --- /dev/null +++ b/upgrade/evidence/phase91-rust-canary-certification.json @@ -0,0 +1,336 @@ +{ + "broker": { + "authority_audit_states_after_restart": [ + "RUST_SHADOW", + "RUST_CANARY", + "BLOCKED", + "RUST_SHADOW" + ], + "cleanup": { + "containers_after": 0, + "networks_after": 0, + "v1_health_after": 200, + "v1_health_before": 200, + "v1_topology_unchanged": true, + "volumes_after": 0 + }, + "final_authority": "RUST_SHADOW", + "image_digest": "sha256:b1dcd12ca123a61f11aef99a860f45eca7a012eac7df07ccfd3e9179519c952d", + "latest_authority_after_restart": { + "lease_epoch": 3, + "owner_id": "rust-ingestor-binance-usdm-shard-0", + "revision": 4, + "state": "RUST_SHADOW" + }, + "legacy_writes": 0, + "min_isr_failed_closed": true, + "mode": "ISOLATED_REHEARSAL", + "offset_deltas": { + "audit": 4, + "authority": 4, + "canary": 64, + "legacy": 0, + "public": 0, + "shadow": 2, + "shadow_raw": 0, + "transport": 0 + }, + "one_replica_loss_acked": true, + "production_authorized": false, + "public_writes": 0, + "runtime": { + "audit_offsets": [ + 0, + 1, + 2, + 3 + ], + "authority_offsets": [ + 0, + 1, + 2, + 3 + ], + "canary_offsets": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63 + ], + "checks": { + "blocked_state_rejected": true, + "canary_after_rollback_rejected": true, + "duplicate_watermark_rejected": true, + "legacy_target_rejected": true, + "public_target_rejected": true, + "stale_lease_rejected": true, + "stale_owner_rejected": true, + "stale_revision_rejected": true, + "topic_masquerade_rejected": true, + "wrong_plan_rejected": true + }, + "final_authority": "RUST_SHADOW", + "legacy_writes": 0, + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "public_writes": 0, + "schema": "qdl.phase91.isolated-canary-runtime.v1", + "shadow_offsets": [ + 0, + 1 + ], + "status": "PASS", + "transitions": [ + "RUST_SHADOW", + "RUST_CANARY", + "BLOCKED", + "RUST_SHADOW" + ] + }, + "schema": "qdl.phase91.broker-rehearsal.v1", + "slow_consumer": { + "catchup_seconds": 9.05500720598502, + "delayed_seconds": 1.0, + "ordered_gap_free": true, + "records": 64 + }, + "status": "PASS" + }, + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "issued_at_ns": 1787074909679119428, + "parity": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "capture": { + "capture_provenance": "REAL_PROVIDER_READ_ONLY", + "capture_sha256": "0912db1d39ddf1ec27414bccc55e096467305534aa581f9d217b179b7a95ff46", + "events": 25600, + "fixtures": 128, + "path": "/home/bobby/data_layer/target/phase91-authentic-replay.json", + "raw_checks_passed": 128, + "repeat": 200 + }, + "process_restart_mismatches": 0, + "python": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "events": 25600, + "events_per_second": 27115.454871588117, + "fixture_count": 128, + "latency_ms": { + "max": 0.241724, + "mean": 0.0360072491796875, + "p50": 0.030751, + "p95": 0.056433, + "p99": 0.068047 + }, + "record_sha256": [ + "0bdb7fb7f13c14f0f8e9adc9fd2762d93fde3c4211fb7cfde06d27dd2098965e", + "551a794866d0add8b53215d1175c603eb83b6cef094264dd79fe57405a35153f", + "983d9517c9a6925f934787af9fd18b9b9ce3343eaf32efe527e7af8853b002ee", + "d5e683845965866661309e4c56ab8b82ab59828ea9de79fd257f1465c139bdd8", + "4291a4187edb4482e5952772a34376688f009ef592dd26f193868118f28fe8b1", + "3499e9e5b0e05b13ec2ebfafc9891aa0a78568766a89e04e6fdadc06bb9f7987", + "e4fed6cefbe203bae15c75ae491959dc19540c290081cea68e177f4a25b8d36f", + "98ca85a273dddba4773a36aa0f0bf5ca4f7ecb53a716b7b85f47fc5b69dfe029", + "d0030c3743a7394203851b84a09f728ebf5ad9dc6a3c44e9e99474c9d00dd5fc", + "18fa31a2ee50afd6c5f2c6d631e95e1d59dd6cb944b7fbb12cd7cde6b82efa54", + "fec2a3bf544416f3c5bc73258715cb4ea21ad52cd21044756dfb237cb17f41e8", + "e65f8fd01f61a9b3af0c6a9a73ab43a63ab9413af5025f9292f2f8ca6da5eb12", + "8e0964240cfd3f3c625455e328d5e11d3e1c902bc5496ddedf9180ec2b0c231c", + "4c92557bde9d311deaff2b372cf9bfeec4ae40055c51c9834ccd1a050499b031", + "34fadfc58a9c8aa8ce38d81983dd52a2be4464558f493f58a5e3c597aafbadb8", + "36fc31328c68bf18e79735f67235f16f7b23a9f29033a92b97739c97f6fb967e", + "3bd1bcf2e054a82ece63c7b5801496928af5f6c1a337ee5e5fec3e5ab2c92837", + "5a9d8ea5216d8b29d0d0d0e327fef2ccfc942ef4a7a2789ca8901babd460d8e2", + "a0860ae4eb295cbc3bd1d7bc7b85030b4631f2021a3b02be56333bd227f6f926", + "1ff9dccf639376644434fa9ac45e26a810451931bf913d4f4a3166b15efb0a4c", + "57aafc884bc978aaadf74353dff92c44f65cc795e37a0088e63b2c844554596e", + "d9cb053df4fd712150e49793d175e0596cb6312ac548c81d322554fbc6fef54d", + "ae5444e073196a2cf2df20e31747c9a7ec89a81ab26d80043282af5bc4419f41", + "e219bc7075d9ae435afa1d6bc64c4314eba08b3e4c51c96c2e0b29ee215ad1d2", + "d2a4af9a664822bbfc184351eef4af4e7717d317dd1d7b99b54bc78ad92fb72c", + "8e5225db6f290adcf08c9216cad680c2e8e5ba9a23abb51a4f54fc2d0b8f150e", + "f991258b33c0f16e35d63b20d23fabe582c0b13558c5ff56ff4fbb3aa7fa286c", + "c265bed006771b5200a4cf7a250e26b648fc4744f941c45a6f892f6484de8fff", + "dd10d1e5c2307cb03267ba449ff03ebeecf5d75882819714b600e073fa6b7df1", + "67d96275874837dc3e7c7c475a8d08268bb362e9d0400e86c8396ca7b07fc4a9", + "03e2362cc7127166da91a6c278a98e0910ab2241fd74edb57da3b2f298b2db18", + "3622b347a37cd57d1274690f5dbc2456590de6f0358137c520af9d587ed5e352", + "eaa2b05a35d3889a4c582b324e02d7e9b00071b0def6d0738d3ee0d835d67cbd", + "ef1a72bf7e51750d6a642235ffbc005435ec7f2a48a65990046777163a51a620", + "380fe178738ceef2bf0ba2150a67a23652c5fe69708bcb1ff97291e1ee5d9f74", + "1a0f78f988b0cfbc04ba6aef740974fb27a076ec2aef40a5147745910b0393a4", + "7e84855bf6c00861673ba5448abdbc26716be15f5e851cb39bd6b7b19a5f3637", + "5df2dc372d4cf6cf575a3db69fa978f3d5ac2cec52d32b5c3c1e91ecb2f6a41a", + "2ff4585edd1ef4c025a9c54022a3c934e5844378ca75bf652bd829c4ed0de29c", + "48ff53d77923f5d5d7c03252fe8dfcf29602d309523aef0f8247cd1e7c1b462e", + "460d6e07432d0bc001abc68fb2107d36d3cb6648762b1f047c1bff3dfb2d03b2", + "7e4f6801b3a6652488e21acfb91ac14967991f6366ada1dd8dca0485cb7417a3", + "043c1a1d71560fa8cb774dcc0c666123b8ef524d7093d709b245b23e518b4ee4", + "9e2644519865231fc427b5315aebc5b5c7ef444737638d9e44d43690e7b82582", + "d6ade1dbf0709e652059a243b8d279013a4c097e5ff77b081cefb129c2dc80eb", + "05f98d974b2d8f66391a40d1230c9e03939bb9fa2e8848f003b31e661bb4d6a0", + "a4c3c5f99e590e87c00bf3644f1471562f5fd98d86843b17e64dc255cbf2400a", + "8c67693ac5bafd2ba9fc61696cc05c95a20950b4bff17c7059ceb814226c8ff1", + "6d16b0937d44a855edb27c32282762532c08256e1c0837b537c335e9ff9959f1", + "acbf0f74f08c65fd1178ca0f85ad06314ed4846ebf3e05757a9eb5a2547fc66f", + "656824469ca227f57fe453bcbcb8c1b176e5b100607c4b815b4876946807a949", + "ac3496b3344dd446b53d6adc488634e769d4599e2af32ae4af3877f0ef1cb60c", + "02f854da9a1f86cea4c95a96f849dca76ef697960bbddd5220d800090e465d16", + "251133c40506d2dba35bea085dee236304ea4510755657e42b6864e01b123590", + "2d5be34c1d4afc4d0a87a62a60e8e569d1dd13751d8a0828e534a239d5e7ccba", + "e3aba260a49e59baa1c62c46366e1f1e8a8860c486ed564817262d9ab91724df", + "ca3f049bbf0d4b66280d177bc3ed97b80916d433774b6a49a23d4ce5074b3a1b", + "9837e2879d41c455fef469c8d6c7cf63f390ea2d6aa9e8f929ae15fb5ad3d565", + "bb8de7c97a8982fd560160f4ec2fa0809d6d6b83ace36e15339ec991f332b05b", + "1c8abda5df59dfae2f741258c4ba7df76d8582a0670ea252cf8c18cb826921c6", + "9b55c4a5dd05b9a639594a33cc69b5f58fdcfb11fe9f9fbc66eb313eac41423c", + "5e4d8b91e4d358d5f94abc5d21e2d5f1f1a95ac79352764ccf50e0d6c6360c75", + "9802feae89ea5472b98da05b967c2453254da040e1a167d76328fa1b8b78db58", + "850a46047de253357105115d178343f125063c3ffe3ae00e0a68ff3b81d887b2", + "e4f71e5d01a3359058d994321b1a329d81da3397c607aa9bd5054576481e2765", + "2272a110375bf01d89f927d4f5c27442dc8db7fec5192441286e1b9722efd9c2", + "4acf59e2f8f9b047da7b8a25beac1af3d7f310abb715504e26e45e5cb13fc2e8", + "08e76d3003712d63d8e97305a8e2989abc52e023c812ab7778576ff03a2b34bd", + "a76814f32c873839433c55cbda89bd784f6b9a7233bcef013157f985387438f6", + "dd4c3828e91eea9686908bbcaa029620a919ee69de56d7eadcc99999a64acf17", + "2ae7c9f69bd4e0851b3c72b2b293d9e62be821e1e0f297113e1e37177d52231b", + "2540ddcd884d4a368531bba0e0476f57d38824400cb784095203e2a4bea2854d", + "e17c7ea5bd68d973a98ba417a4450dd5ac663da71d4070d314f804b09468ce89", + "21353eff6fb732edacf862957bcc43f61128deb005586d888e20df199ce04bd3", + "59c7742b77fd416820ca0a1467365e20187516f7270d1af0ca445ad88f40f489", + "17d98f3eea38caa95397f2e4f9367eb2189441ec1d4e8f317fbcf461e1597d06", + "5064ff5d3a49218a8dccadf34bcbb47700e1a1a99d92f5bb762b497bab3e21a7", + "324c6c1d4e1016fadd95878d050ddbb83bf4732c6b19164c3439f9e3099e11a9", + "71100e2b4a46b1754605c7dd126d7153b3828e44fd1a153553a57dd18fb7dc33", + "e45fe5e59442d29bbac9c36c2e2aaa1c75647b2774dfbbc396018c139c80b63d", + "551c59a6cf45d806e50c26893403cbcab7068dcb561196517bd72303bbeffc10", + "4738c73066e3acbd013c068747dc092d68939ecde2638f0911fb0d375d38c563", + "d4499622a95575edd2f4fa93a4a47e878ede7bc4d277d5eaadc59b6971b237fd", + "f4355be81f0e3135855042bfee74b42eae46f4fd82a8cd70d439e753fa70c98e", + "104146496e9e67a631b9ed3ea9cd4bc2f91094b5838e43fdb68e14139bb3c9f9", + "6bcac04c6f40b3b4b4771ded2a82a44f4dfa7cdcd3c89ca3535d1ba6cc532a1f", + "3ab6f4fea8f409038c066a3ec98468e3ddad2f280843fe50c215809df0f41266", + "9629e26ed5f8eb8128b40b879de34518b8fca22d4d5fa626ca4e2ed0b012fce2", + "780750b17c2972f8b424ea8f0e33060fae16866818c06b2d128a1c32296ff645", + "8311b7e8e07478478a15f1cf0a028ed31dc34474c9d18e2c3bac77f9b6c72854", + "85a798fd9700c7c4dc29fa15bb324618c2f34c14b5ba1198ae37747cf41693d4", + "2739404a0f05f9abb956530f4a5d17ae111b7179513251355e17e5676615f649", + "9995532a889c42631008b8b2fe082d87cc5405d27d7af22d6e62fdd1b47439b3", + "5224ab418ae50ab6815ef572a6d4bd10d78e993b6049c1a1dd1e3f6f9814cab0", + "74f2db999392b12121a52b358434296da44bb89947ddafe5d40cec7beb3e414d", + "1c93beb39e5fbd261f5a6566f88b4cce6c8feeb4b9e9c91783fc8b4499189afb", + "ddc628df8adcb3165736d8545d6bc1c4452f2296ab878b056bd6511f227cd299", + "eb7084f1f8ef541f27fdebb9f8cb934bcd7edb9c8c6d12f60316f84acdd4b9b2", + "cde6acdf79d267ee5147c4e65d1edb4e0a07656b6f560578585211ae88883fc3", + "7449c220b3a41c6d7a5682ed7d7ccec2c4def46824eabbefd595a0e5dddaf9e9", + "5ed88151733674af61fd474cfedf15e7a8b7d1c7bce44928bda9cc2db858984a", + "24f59709d171dfa4d49b036a469208a87bf4515161aacf9c55b5c9935273b2f0", + "23efeef3e63975a236f69015a44a8b32b162570ad60ef107489aa1a1da5388db", + "f4e57fc11ac0bdb1235f5a49849d5e8c1d8a73f02b464b8861ae2134d29aeb1f", + "f2b2f29080ab68ffe39d8e86772c78b7c09a16024b72b925ebd824525803dcf9", + "e102f7739a5b7e75dd2adacddfe48580ea4a621182d0fa0ef865e483f881be94", + "25e4208a85c1dbbe8c9337f0fbdb8cab2655a9692846e9669e6b597cf79373c8", + "da4982e13d2a24665fe5cb4e3d11ee9f6b237d15a8d6f463fa599572ee01d5cd", + "0d224549cc794f72283c537d194efe10fda71597d5e8b6b1d6f0faf1f5688fe4", + "2378fb0751d0f21ecb91f5b1bd846d788058ad2a73200a33460e5b772cd417ae", + "e994028d841cd272458ce38f00aad5a93553e5ab36bd71995dc0ca2b2ed5c19e", + "eb1cec7e0abe56bcf4e0240560b3bcaedc6a4081dc7ca2e4a8e9e04c62dbf9c4", + "4cae7a4d07888ee8eec3f8b3a7d232f76eded1d5af4847fb6c31fab6e9700158", + "2861deaa154c6f6c93e536ddd6cf8a9b5d445204f96ef511616126bf32b2b4a8", + "a1195ef8b92d279923ecd86ce25cdfb905607ec533b323d8e0d0b1b5dc2902ac", + "68cfa7d0933d0261fa34a58ba1c224148d3133cbc237421dcc63b830a67e870b", + "bde390c96ff54978e603f67e05d115e57f928b0ffbe2a9e06f1907e293bef7a6", + "2e0347d26f139a07ab151f00c5968f27cf37e0abe748a4f7809cd7488b4816e7", + "8151f090f8a5f635fd3bcb4b42569919ddfe34b756c87610cdae9cf4d899ea52", + "042c82bdf6e09344c337ff9753ba9852f10180fcc2ec6fdab9431c48f2804a47", + "2238079e95be053c37ae41b00934decd06e4da07d9f86919ad5cfa810e5597ad", + "cffde788491b58ae16c8da3d13b4dc52bcf8bfd753211687460c53e224495249", + "6b83a2ed68f6921b6bd23cd05f9421914df684139dba33cc026e29c456fbc570", + "b9e6e92c6c876a9e80fd4d42fd6e49d814d1e79d6b8cf5c9b8d2cb0bd768cd8e", + "10c6905093b0d916e08870d42be0d50cf1f1f548235f421a690971d2ec12daab", + "a88672cf673b5d040aac8c63e80fed9e40e12ab19e18fdf1c641f7d0a29674ef", + "ead7b579320feaa42cc6c4e55c5e22adab2bf7934c0eb2cbb88db347362f7d59", + "f759753fa0211e131cb0ebb2ca40efe5296d680cd0328cdcccc00cd8028fac3d" + ], + "repeat": 200, + "schema": "qdl.phase91.python-parity.v1", + "status": "PASS" + }, + "rust_clean_process_runs": 3, + "rust_events_per_second_min": 350581.02505399904, + "semantic_mismatches": 0, + "status": "PASS" + }, + "prerequisite_bundle_id": "558042db-a766-5a55-b5b3-4b508d649df9", + "prerequisite_decision": "NO_GO_EXTERNAL", + "production_authorized": false, + "production_mutations": 0, + "python_v1_public_authority_unchanged": true, + "schema": "qdl.phase91.rust-canary-certification.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "status": "COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED", + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "same-host replicated broker rehearsal is not an independent production failure domain" + ] +} diff --git a/upgrade/evidence/phase92-authority-migration.json b/upgrade/evidence/phase92-authority-migration.json new file mode 100644 index 0000000..d666fc8 --- /dev/null +++ b/upgrade/evidence/phase92-authority-migration.json @@ -0,0 +1,16 @@ +{ + "accepted_handoffs": 2, + "audit_records": 4, + "checkpoint_delete_rejected": true, + "direct_primary_bypass_rejected": true, + "final_test_state": "PYTHON_PRIMARY:7:python-rollback:3:120", + "handoff_mutation_rejected": true, + "idempotent_migration": true, + "production_mutations": 0, + "python_to_rust_handoff_passed": true, + "rust_to_python_rollback_passed": true, + "schema": "qdl.phase92.authority-migration.v1", + "stale_cas_rejected": true, + "status": "PASS", + "terminal_checkpoints": 2 +} diff --git a/upgrade/evidence/phase92-bounded-primary-certification.json b/upgrade/evidence/phase92-bounded-primary-certification.json new file mode 100644 index 0000000..8436891 --- /dev/null +++ b/upgrade/evidence/phase92-bounded-primary-certification.json @@ -0,0 +1,582 @@ +{ + "broker": { + "accepted_handoffs": 2, + "authority_audit_states_after_restart": [ + "RUST_CANARY", + "RUST_PRIMARY", + "BLOCKED", + "ROLLBACK_PENDING", + "PYTHON_PRIMARY" + ], + "boundary_gap_free": true, + "cleanup": { + "containers_after": 0, + "networks_after": 0, + "v1_health_after": 200, + "v1_health_before": 200, + "v1_topology_unchanged": true, + "volumes_after": 0 + }, + "cutover_ms": 22.200226, + "final_authority": "PYTHON_PRIMARY", + "image_digest": "sha256:34508eda14e66134f96c4da3ea4da3fd42c46425ecf56815df3089c3d94eb67c", + "latest_authority_after_restart": { + "lease_epoch": 13, + "owner_id": "python-rollback-isolated", + "revision": 11, + "state": "PYTHON_PRIMARY" + }, + "min_isr_failed_closed": true, + "mode": "ISOLATED_REHEARSAL", + "offset_deltas": { + "audit": 5, + "authority": 5, + "checkpoint": 2, + "handoff": 2, + "legacy": 81, + "primary": 81, + "production_legacy": 0, + "production_public": 0, + "public": 81, + "transport": 0 + }, + "one_replica_loss_acked": true, + "owner_boundary_correct": true, + "process_restart_recovery": { + "authority_revision": 11, + "checks": { + "duplicate_after_restore_rejected": true, + "durable_target_watermarks_restored": true, + "each_target_pre_restore_failed_closed": true, + "restart_pre_restore_failed_closed": true, + "resumed_at_exact_next_watermark": true + }, + "mode": "RECOVERY_VERIFY", + "owner_id": "python-rollback-isolated", + "production_authorized": false, + "production_legacy_writes": 0, + "production_public_writes": 0, + "projection_offsets": [ + 80, + 80, + 80 + ], + "restored_target_watermarks": { + "legacy": 180, + "primary": 180, + "public": 180 + }, + "resumed_watermark": 181, + "schema": "qdl.phase92.process-restart-recovery.v1", + "status": "PASS" + }, + "production_authorized": false, + "production_legacy_writes": 0, + "production_public_writes": 0, + "projection_parity": true, + "rollback_ms": 533.2371999999999, + "runtime": { + "audit_offsets": [ + 0, + 1, + 2, + 3, + 4 + ], + "authority_offsets": [ + 0, + 1, + 2, + 3, + 4 + ], + "authority_transitions": [ + "RUST_CANARY", + "RUST_PRIMARY", + "BLOCKED", + "ROLLBACK_PENDING", + "PYTHON_PRIMARY" + ], + "checkpoint_offsets": [ + 0, + 1 + ], + "checks": { + "direct_primary_without_handoff_rejected": true, + "duplicate_after_ack_rejected": true, + "gap_watermark_rejected": true, + "rust_after_rollback_rejected": true, + "stale_lease_rejected": true, + "stale_owner_rejected": true, + "stale_revision_rejected": true, + "terminal_watermark_rejected": true, + "wrong_plan_rejected": true + }, + "cutover_ms": 22.200226, + "final_authority": "PYTHON_PRIMARY", + "first_python_rollback_watermark": 165, + "first_rust_watermark": 101, + "handoff_offsets": [ + 0, + 1 + ], + "isolated_legacy_writes": 80, + "isolated_primary_writes": 80, + "isolated_public_writes": 80, + "last_rust_watermark": 164, + "last_watermark": 180, + "mode": "ISOLATED_REHEARSAL", + "production_authorized": false, + "production_legacy_writes": 0, + "production_public_writes": 0, + "projection_offsets": [ + 0, + 0, + 0, + 1, + 1, + 1, + 2, + 2, + 2, + 3, + 3, + 3, + 4, + 4, + 4, + 5, + 5, + 5, + 6, + 6, + 6, + 7, + 7, + 7, + 8, + 8, + 8, + 9, + 9, + 9, + 10, + 10, + 10, + 11, + 11, + 11, + 12, + 12, + 12, + 13, + 13, + 13, + 14, + 14, + 14, + 15, + 15, + 15, + 16, + 16, + 16, + 17, + 17, + 17, + 18, + 18, + 18, + 19, + 19, + 19, + 20, + 20, + 20, + 21, + 21, + 21, + 22, + 22, + 22, + 23, + 23, + 23, + 24, + 24, + 24, + 25, + 25, + 25, + 26, + 26, + 26, + 27, + 27, + 27, + 28, + 28, + 28, + 29, + 29, + 29, + 30, + 30, + 30, + 31, + 31, + 31, + 32, + 32, + 32, + 33, + 33, + 33, + 34, + 34, + 34, + 35, + 35, + 35, + 36, + 36, + 36, + 37, + 37, + 37, + 38, + 38, + 38, + 39, + 39, + 39, + 40, + 40, + 40, + 41, + 41, + 41, + 42, + 42, + 42, + 43, + 43, + 43, + 44, + 44, + 44, + 45, + 45, + 45, + 46, + 46, + 46, + 47, + 47, + 47, + 48, + 48, + 48, + 49, + 49, + 49, + 50, + 50, + 50, + 51, + 51, + 51, + 52, + 52, + 52, + 53, + 53, + 53, + 54, + 54, + 54, + 55, + 55, + 55, + 56, + 56, + 56, + 57, + 57, + 57, + 58, + 58, + 58, + 59, + 59, + 59, + 60, + 60, + 60, + 61, + 61, + 61, + 62, + 62, + 62, + 63, + 63, + 63, + 64, + 64, + 64, + 65, + 65, + 65, + 66, + 66, + 66, + 67, + 67, + 67, + 68, + 68, + 68, + 69, + 69, + 69, + 70, + 70, + 70, + 71, + 71, + 71, + 72, + 72, + 72, + 73, + 73, + 73, + 74, + 74, + 74, + 75, + 75, + 75, + 76, + 76, + 76, + 77, + 77, + 77, + 78, + 78, + 78, + 79, + 79, + 79 + ], + "rollback_ms": 533.2371999999999, + "schema": "qdl.phase92.isolated-primary-runtime.v1", + "status": "PASS" + }, + "schema": "qdl.phase92.broker-rehearsal.v1", + "slow_consumer": { + "catchup_seconds": 24.523788021004293, + "delayed_seconds": 1.0, + "ordered_gap_free": true, + "records_per_projection": 81 + }, + "status": "PASS", + "terminal_checkpoints": 2 + }, + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "issued_at_ns": 1787081314573802100, + "migration": { + "accepted_handoffs": 2, + "audit_records": 4, + "checkpoint_delete_rejected": true, + "direct_primary_bypass_rejected": true, + "final_test_state": "PYTHON_PRIMARY:7:python-rollback:3:120", + "handoff_mutation_rejected": true, + "idempotent_migration": true, + "production_mutations": 0, + "python_to_rust_handoff_passed": true, + "rust_to_python_rollback_passed": true, + "schema": "qdl.phase92.authority-migration.v1", + "stale_cas_rejected": true, + "status": "PASS", + "terminal_checkpoints": 2 + }, + "parity": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "capture": { + "capture_provenance": "REAL_PROVIDER_READ_ONLY", + "capture_sha256": "0912db1d39ddf1ec27414bccc55e096467305534aa581f9d217b179b7a95ff46", + "events": 25600, + "fixtures": 128, + "path": "/home/bobby/data_layer/target/phase91-authentic-replay.json", + "raw_checks_passed": 128, + "repeat": 200 + }, + "process_restart_mismatches": 0, + "python": { + "aggregate_sha256": "75f2f97a0c2d9e9b7861e1ab192f66b85257ae85a69932f7f9c8e19a0c38a0ea", + "events": 25600, + "events_per_second": 28388.208223491194, + "fixture_count": 128, + "latency_ms": { + "max": 0.280434, + "mean": 0.034400821328125, + "p50": 0.030806, + "p95": 0.05559, + "p99": 0.065782 + }, + "record_sha256": [ + "0bdb7fb7f13c14f0f8e9adc9fd2762d93fde3c4211fb7cfde06d27dd2098965e", + "551a794866d0add8b53215d1175c603eb83b6cef094264dd79fe57405a35153f", + "983d9517c9a6925f934787af9fd18b9b9ce3343eaf32efe527e7af8853b002ee", + "d5e683845965866661309e4c56ab8b82ab59828ea9de79fd257f1465c139bdd8", + "4291a4187edb4482e5952772a34376688f009ef592dd26f193868118f28fe8b1", + "3499e9e5b0e05b13ec2ebfafc9891aa0a78568766a89e04e6fdadc06bb9f7987", + "e4fed6cefbe203bae15c75ae491959dc19540c290081cea68e177f4a25b8d36f", + "98ca85a273dddba4773a36aa0f0bf5ca4f7ecb53a716b7b85f47fc5b69dfe029", + "d0030c3743a7394203851b84a09f728ebf5ad9dc6a3c44e9e99474c9d00dd5fc", + "18fa31a2ee50afd6c5f2c6d631e95e1d59dd6cb944b7fbb12cd7cde6b82efa54", + "fec2a3bf544416f3c5bc73258715cb4ea21ad52cd21044756dfb237cb17f41e8", + "e65f8fd01f61a9b3af0c6a9a73ab43a63ab9413af5025f9292f2f8ca6da5eb12", + "8e0964240cfd3f3c625455e328d5e11d3e1c902bc5496ddedf9180ec2b0c231c", + "4c92557bde9d311deaff2b372cf9bfeec4ae40055c51c9834ccd1a050499b031", + "34fadfc58a9c8aa8ce38d81983dd52a2be4464558f493f58a5e3c597aafbadb8", + "36fc31328c68bf18e79735f67235f16f7b23a9f29033a92b97739c97f6fb967e", + "3bd1bcf2e054a82ece63c7b5801496928af5f6c1a337ee5e5fec3e5ab2c92837", + "5a9d8ea5216d8b29d0d0d0e327fef2ccfc942ef4a7a2789ca8901babd460d8e2", + "a0860ae4eb295cbc3bd1d7bc7b85030b4631f2021a3b02be56333bd227f6f926", + "1ff9dccf639376644434fa9ac45e26a810451931bf913d4f4a3166b15efb0a4c", + "57aafc884bc978aaadf74353dff92c44f65cc795e37a0088e63b2c844554596e", + "d9cb053df4fd712150e49793d175e0596cb6312ac548c81d322554fbc6fef54d", + "ae5444e073196a2cf2df20e31747c9a7ec89a81ab26d80043282af5bc4419f41", + "e219bc7075d9ae435afa1d6bc64c4314eba08b3e4c51c96c2e0b29ee215ad1d2", + "d2a4af9a664822bbfc184351eef4af4e7717d317dd1d7b99b54bc78ad92fb72c", + "8e5225db6f290adcf08c9216cad680c2e8e5ba9a23abb51a4f54fc2d0b8f150e", + "f991258b33c0f16e35d63b20d23fabe582c0b13558c5ff56ff4fbb3aa7fa286c", + "c265bed006771b5200a4cf7a250e26b648fc4744f941c45a6f892f6484de8fff", + "dd10d1e5c2307cb03267ba449ff03ebeecf5d75882819714b600e073fa6b7df1", + "67d96275874837dc3e7c7c475a8d08268bb362e9d0400e86c8396ca7b07fc4a9", + "03e2362cc7127166da91a6c278a98e0910ab2241fd74edb57da3b2f298b2db18", + "3622b347a37cd57d1274690f5dbc2456590de6f0358137c520af9d587ed5e352", + "eaa2b05a35d3889a4c582b324e02d7e9b00071b0def6d0738d3ee0d835d67cbd", + "ef1a72bf7e51750d6a642235ffbc005435ec7f2a48a65990046777163a51a620", + "380fe178738ceef2bf0ba2150a67a23652c5fe69708bcb1ff97291e1ee5d9f74", + "1a0f78f988b0cfbc04ba6aef740974fb27a076ec2aef40a5147745910b0393a4", + "7e84855bf6c00861673ba5448abdbc26716be15f5e851cb39bd6b7b19a5f3637", + "5df2dc372d4cf6cf575a3db69fa978f3d5ac2cec52d32b5c3c1e91ecb2f6a41a", + "2ff4585edd1ef4c025a9c54022a3c934e5844378ca75bf652bd829c4ed0de29c", + "48ff53d77923f5d5d7c03252fe8dfcf29602d309523aef0f8247cd1e7c1b462e", + "460d6e07432d0bc001abc68fb2107d36d3cb6648762b1f047c1bff3dfb2d03b2", + "7e4f6801b3a6652488e21acfb91ac14967991f6366ada1dd8dca0485cb7417a3", + "043c1a1d71560fa8cb774dcc0c666123b8ef524d7093d709b245b23e518b4ee4", + "9e2644519865231fc427b5315aebc5b5c7ef444737638d9e44d43690e7b82582", + "d6ade1dbf0709e652059a243b8d279013a4c097e5ff77b081cefb129c2dc80eb", + "05f98d974b2d8f66391a40d1230c9e03939bb9fa2e8848f003b31e661bb4d6a0", + "a4c3c5f99e590e87c00bf3644f1471562f5fd98d86843b17e64dc255cbf2400a", + "8c67693ac5bafd2ba9fc61696cc05c95a20950b4bff17c7059ceb814226c8ff1", + "6d16b0937d44a855edb27c32282762532c08256e1c0837b537c335e9ff9959f1", + "acbf0f74f08c65fd1178ca0f85ad06314ed4846ebf3e05757a9eb5a2547fc66f", + "656824469ca227f57fe453bcbcb8c1b176e5b100607c4b815b4876946807a949", + "ac3496b3344dd446b53d6adc488634e769d4599e2af32ae4af3877f0ef1cb60c", + "02f854da9a1f86cea4c95a96f849dca76ef697960bbddd5220d800090e465d16", + "251133c40506d2dba35bea085dee236304ea4510755657e42b6864e01b123590", + "2d5be34c1d4afc4d0a87a62a60e8e569d1dd13751d8a0828e534a239d5e7ccba", + "e3aba260a49e59baa1c62c46366e1f1e8a8860c486ed564817262d9ab91724df", + "ca3f049bbf0d4b66280d177bc3ed97b80916d433774b6a49a23d4ce5074b3a1b", + "9837e2879d41c455fef469c8d6c7cf63f390ea2d6aa9e8f929ae15fb5ad3d565", + "bb8de7c97a8982fd560160f4ec2fa0809d6d6b83ace36e15339ec991f332b05b", + "1c8abda5df59dfae2f741258c4ba7df76d8582a0670ea252cf8c18cb826921c6", + "9b55c4a5dd05b9a639594a33cc69b5f58fdcfb11fe9f9fbc66eb313eac41423c", + "5e4d8b91e4d358d5f94abc5d21e2d5f1f1a95ac79352764ccf50e0d6c6360c75", + "9802feae89ea5472b98da05b967c2453254da040e1a167d76328fa1b8b78db58", + "850a46047de253357105115d178343f125063c3ffe3ae00e0a68ff3b81d887b2", + "e4f71e5d01a3359058d994321b1a329d81da3397c607aa9bd5054576481e2765", + "2272a110375bf01d89f927d4f5c27442dc8db7fec5192441286e1b9722efd9c2", + "4acf59e2f8f9b047da7b8a25beac1af3d7f310abb715504e26e45e5cb13fc2e8", + "08e76d3003712d63d8e97305a8e2989abc52e023c812ab7778576ff03a2b34bd", + "a76814f32c873839433c55cbda89bd784f6b9a7233bcef013157f985387438f6", + "dd4c3828e91eea9686908bbcaa029620a919ee69de56d7eadcc99999a64acf17", + "2ae7c9f69bd4e0851b3c72b2b293d9e62be821e1e0f297113e1e37177d52231b", + "2540ddcd884d4a368531bba0e0476f57d38824400cb784095203e2a4bea2854d", + "e17c7ea5bd68d973a98ba417a4450dd5ac663da71d4070d314f804b09468ce89", + "21353eff6fb732edacf862957bcc43f61128deb005586d888e20df199ce04bd3", + "59c7742b77fd416820ca0a1467365e20187516f7270d1af0ca445ad88f40f489", + "17d98f3eea38caa95397f2e4f9367eb2189441ec1d4e8f317fbcf461e1597d06", + "5064ff5d3a49218a8dccadf34bcbb47700e1a1a99d92f5bb762b497bab3e21a7", + "324c6c1d4e1016fadd95878d050ddbb83bf4732c6b19164c3439f9e3099e11a9", + "71100e2b4a46b1754605c7dd126d7153b3828e44fd1a153553a57dd18fb7dc33", + "e45fe5e59442d29bbac9c36c2e2aaa1c75647b2774dfbbc396018c139c80b63d", + "551c59a6cf45d806e50c26893403cbcab7068dcb561196517bd72303bbeffc10", + "4738c73066e3acbd013c068747dc092d68939ecde2638f0911fb0d375d38c563", + "d4499622a95575edd2f4fa93a4a47e878ede7bc4d277d5eaadc59b6971b237fd", + "f4355be81f0e3135855042bfee74b42eae46f4fd82a8cd70d439e753fa70c98e", + "104146496e9e67a631b9ed3ea9cd4bc2f91094b5838e43fdb68e14139bb3c9f9", + "6bcac04c6f40b3b4b4771ded2a82a44f4dfa7cdcd3c89ca3535d1ba6cc532a1f", + "3ab6f4fea8f409038c066a3ec98468e3ddad2f280843fe50c215809df0f41266", + "9629e26ed5f8eb8128b40b879de34518b8fca22d4d5fa626ca4e2ed0b012fce2", + "780750b17c2972f8b424ea8f0e33060fae16866818c06b2d128a1c32296ff645", + "8311b7e8e07478478a15f1cf0a028ed31dc34474c9d18e2c3bac77f9b6c72854", + "85a798fd9700c7c4dc29fa15bb324618c2f34c14b5ba1198ae37747cf41693d4", + "2739404a0f05f9abb956530f4a5d17ae111b7179513251355e17e5676615f649", + "9995532a889c42631008b8b2fe082d87cc5405d27d7af22d6e62fdd1b47439b3", + "5224ab418ae50ab6815ef572a6d4bd10d78e993b6049c1a1dd1e3f6f9814cab0", + "74f2db999392b12121a52b358434296da44bb89947ddafe5d40cec7beb3e414d", + "1c93beb39e5fbd261f5a6566f88b4cce6c8feeb4b9e9c91783fc8b4499189afb", + "ddc628df8adcb3165736d8545d6bc1c4452f2296ab878b056bd6511f227cd299", + "eb7084f1f8ef541f27fdebb9f8cb934bcd7edb9c8c6d12f60316f84acdd4b9b2", + "cde6acdf79d267ee5147c4e65d1edb4e0a07656b6f560578585211ae88883fc3", + "7449c220b3a41c6d7a5682ed7d7ccec2c4def46824eabbefd595a0e5dddaf9e9", + "5ed88151733674af61fd474cfedf15e7a8b7d1c7bce44928bda9cc2db858984a", + "24f59709d171dfa4d49b036a469208a87bf4515161aacf9c55b5c9935273b2f0", + "23efeef3e63975a236f69015a44a8b32b162570ad60ef107489aa1a1da5388db", + "f4e57fc11ac0bdb1235f5a49849d5e8c1d8a73f02b464b8861ae2134d29aeb1f", + "f2b2f29080ab68ffe39d8e86772c78b7c09a16024b72b925ebd824525803dcf9", + "e102f7739a5b7e75dd2adacddfe48580ea4a621182d0fa0ef865e483f881be94", + "25e4208a85c1dbbe8c9337f0fbdb8cab2655a9692846e9669e6b597cf79373c8", + "da4982e13d2a24665fe5cb4e3d11ee9f6b237d15a8d6f463fa599572ee01d5cd", + "0d224549cc794f72283c537d194efe10fda71597d5e8b6b1d6f0faf1f5688fe4", + "2378fb0751d0f21ecb91f5b1bd846d788058ad2a73200a33460e5b772cd417ae", + "e994028d841cd272458ce38f00aad5a93553e5ab36bd71995dc0ca2b2ed5c19e", + "eb1cec7e0abe56bcf4e0240560b3bcaedc6a4081dc7ca2e4a8e9e04c62dbf9c4", + "4cae7a4d07888ee8eec3f8b3a7d232f76eded1d5af4847fb6c31fab6e9700158", + "2861deaa154c6f6c93e536ddd6cf8a9b5d445204f96ef511616126bf32b2b4a8", + "a1195ef8b92d279923ecd86ce25cdfb905607ec533b323d8e0d0b1b5dc2902ac", + "68cfa7d0933d0261fa34a58ba1c224148d3133cbc237421dcc63b830a67e870b", + "bde390c96ff54978e603f67e05d115e57f928b0ffbe2a9e06f1907e293bef7a6", + "2e0347d26f139a07ab151f00c5968f27cf37e0abe748a4f7809cd7488b4816e7", + "8151f090f8a5f635fd3bcb4b42569919ddfe34b756c87610cdae9cf4d899ea52", + "042c82bdf6e09344c337ff9753ba9852f10180fcc2ec6fdab9431c48f2804a47", + "2238079e95be053c37ae41b00934decd06e4da07d9f86919ad5cfa810e5597ad", + "cffde788491b58ae16c8da3d13b4dc52bcf8bfd753211687460c53e224495249", + "6b83a2ed68f6921b6bd23cd05f9421914df684139dba33cc026e29c456fbc570", + "b9e6e92c6c876a9e80fd4d42fd6e49d814d1e79d6b8cf5c9b8d2cb0bd768cd8e", + "10c6905093b0d916e08870d42be0d50cf1f1f548235f421a690971d2ec12daab", + "a88672cf673b5d040aac8c63e80fed9e40e12ab19e18fdf1c641f7d0a29674ef", + "ead7b579320feaa42cc6c4e55c5e22adab2bf7934c0eb2cbb88db347362f7d59", + "f759753fa0211e131cb0ebb2ca40efe5296d680cd0328cdcccc00cd8028fac3d" + ], + "repeat": 200, + "schema": "qdl.phase91.python-parity.v1", + "status": "PASS" + }, + "rust_clean_process_runs": 3, + "rust_events_per_second_min": 385536.1882187671, + "semantic_mismatches": 0, + "status": "PASS" + }, + "prerequisite_bundle_id": "558042db-a766-5a55-b5b3-4b508d649df9", + "prerequisite_decision": "NO_GO_EXTERNAL", + "production_authorized": false, + "production_mutations": 0, + "python_v1_public_authority_unchanged": true, + "schema": "qdl.phase92.bounded-primary-certification.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "status": "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED", + "technical_debt": [ + "production Phase 9.0-C infrastructure/operator gates remain NO_GO_EXTERNAL", + "real production canary hold and exact-slice approval remain unavailable", + "same-host replicated broker is not an independent failure domain" + ] +} diff --git a/upgrade/evidence/phase92-evidence.sha256 b/upgrade/evidence/phase92-evidence.sha256 new file mode 100644 index 0000000..3f14971 --- /dev/null +++ b/upgrade/evidence/phase92-evidence.sha256 @@ -0,0 +1,3 @@ +2acf8616d9134dea52eea0ccaebcc7f783efa29a86d85aad627f97dbbee3cfe6 upgrade/evidence/phase92-bounded-primary-certification.json +005e5034cec9ab04b2b772a0aef151b7108255c3d3e0f3cd0fd7134b83723772 upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md +8d7761160376233d17808a8f3387d8adb89cba5db3e3dc487e50e5c267dd612a upgrade/evidence/phase92-authority-migration.json diff --git a/upgrade/evidence/phase93-evidence.sha256 b/upgrade/evidence/phase93-evidence.sha256 new file mode 100644 index 0000000..fd24997 --- /dev/null +++ b/upgrade/evidence/phase93-evidence.sha256 @@ -0,0 +1,3 @@ +f869a7621cbcab174788027468ca31f4982e7bce06af768e29c9e486420105b2 upgrade/evidence/phase93-hold-close-expand-certification.json +00b1b1854858aea628bcae29e4a2c49e35ce6a6c55814c8e186d58eb761a0fc1 upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md +5239ab909c63f691c56f205f4d59afef44b718138c563fe6173a0b9e4c77bb64 upgrade/evidence/phase93-hold-close-migration.json diff --git a/upgrade/evidence/phase93-hold-close-expand-certification.json b/upgrade/evidence/phase93-hold-close-expand-certification.json new file mode 100644 index 0000000..539fa5b --- /dev/null +++ b/upgrade/evidence/phase93-hold-close-expand-certification.json @@ -0,0 +1,112 @@ +{ + "candidate_digest": "72eb1500e19a7e738373c85442c6fc42331cebd15aba86a8b746f62c2fedc037", + "cleanup": { + "production_rows_created": 0, + "resources_remaining": 0 + }, + "control_plane_fixture": { + "accelerated_time_is_production_evidence": false, + "current_no_go_rejection": "PREREQUISITE_DECISION_NOT_GO", + "decommission_reason": "RUNTIME_STILL_REQUIRED_FOR_ROLLBACK", + "expansion_manifest_count": 5, + "expansions": [ + { + "digest": "febb0bb6e5f785c34f169c141814fae6789ea2ad18cacad82fd9dd441562683c", + "required_gate_count": 7, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "INSTRUMENT_PARTITION", + "write_authority": false + }, + { + "digest": "bd498010e9297433f18101911853d4dd7f74257fcfa0ef52492b6de5f5276db6", + "required_gate_count": 9, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "BBO", + "write_authority": false + }, + { + "digest": "1a72f3c0ff729fa938c9dbff2837c5084017b6b2988d2cc69cde8c3deeeaba21", + "required_gate_count": 9, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "L2_BOOK", + "write_authority": false + }, + { + "digest": "46e72af783a551a3fcc6ca227c3a91302e6c24f023766d7e38e3318dfed3396c", + "required_gate_count": 8, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "BAR_LIFECYCLE", + "write_authority": false + }, + { + "digest": "6244ea1445c4334f20025bc27cdf1c95c4257a64fe8981808973a471b3fd436f", + "required_gate_count": 10, + "status": "INDEPENDENT_CERTIFICATION_REQUIRED", + "transitive_evidence_allowed": false, + "type": "VENUE_MARKET", + "write_authority": false + } + ], + "local_phase92_production_eligible": false, + "provenance": "TEST_CONTROL_PLANE_FIXTURE", + "test_hold_production_authorized": false, + "test_hold_status": "PASSED" + }, + "issued_at_ns": 1787103230688606493, + "migration": { + "all_expansion_types_registered_independently": true, + "append_only_mutation_rejected": true, + "approval_and_closure_ids_distinct": true, + "authority_state_after_closure": "RUST_PRIMARY:4:rust-primary:2:100", + "closure_did_not_mutate_authority": true, + "closure_digest_bound_to_expansion": true, + "closure_records": 1, + "decision_records": 2, + "decommission_records": 2, + "dirty_hold_pass_rejected": true, + "expansion_records": 5, + "final_test_state_after_stale_cas_setup": "BLOCKED:5:rust-primary:2:130", + "hold_records": 2, + "idempotent_migration": true, + "incomplete_expansion_gates_rejected": true, + "observation_records": 3, + "out_of_order_observation_rejected": true, + "production_mutations": 0, + "registry_mutation_rejected": true, + "schema": "qdl.phase93.hold-close-migration.v1", + "stale_authority_closure_rejected": true, + "status": "PASS" + }, + "parent_phase92": { + "authentic_events": 25600, + "production_authorized": false, + "semantic_mismatches": 0, + "sha256": "2acf8616d9134dea52eea0ccaebcc7f783efa29a86d85aad627f97dbbee3cfe6", + "status": "COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED" + }, + "prerequisite_decision": "NO_GO_EXTERNAL", + "production_authorized": false, + "production_expansions_authorized": 0, + "production_hold_started": false, + "production_mutations": 0, + "production_rollback_window_closed": false, + "python_decommission_authorized": false, + "schema": "qdl.phase93.hold-close-expand-certification.v1", + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "status": "COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED", + "technical_debt": [ + "Phase 9.0-C production infrastructure remains NO_GO_EXTERNAL", + "real production primary and sustained hold observations do not exist", + "real consumer checkpoints and operator closure approval do not exist", + "every expansion remains independently uncertified" + ], + "v1": { + "health_after": 200, + "health_before": 200, + "topology_unchanged": true + } +} diff --git a/upgrade/evidence/phase93-hold-close-migration.json b/upgrade/evidence/phase93-hold-close-migration.json new file mode 100644 index 0000000..4fb9858 --- /dev/null +++ b/upgrade/evidence/phase93-hold-close-migration.json @@ -0,0 +1,24 @@ +{ + "all_expansion_types_registered_independently": true, + "append_only_mutation_rejected": true, + "approval_and_closure_ids_distinct": true, + "authority_state_after_closure": "RUST_PRIMARY:4:rust-primary:2:100", + "closure_did_not_mutate_authority": true, + "closure_digest_bound_to_expansion": true, + "closure_records": 1, + "decision_records": 2, + "decommission_records": 2, + "dirty_hold_pass_rejected": true, + "expansion_records": 5, + "final_test_state_after_stale_cas_setup": "BLOCKED:5:rust-primary:2:130", + "hold_records": 2, + "idempotent_migration": true, + "incomplete_expansion_gates_rejected": true, + "observation_records": 3, + "out_of_order_observation_rejected": true, + "production_mutations": 0, + "registry_mutation_rejected": true, + "schema": "qdl.phase93.hold-close-migration.v1", + "stale_authority_closure_rejected": true, + "status": "PASS" +} diff --git a/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md b/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md index 2c64026..cb42263 100644 --- a/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md +++ b/upgrade/quant-data-layer-fund-grade-upgrade-architecture.md @@ -1438,6 +1438,26 @@ Với VN market data: - After-hours last snapshot không được đánh dấu live. - Trading calendar quyết định market closed, không dùng absence đơn thuần. +DNSE production acquisition boundary: + +- REST market-history transport sends an explicit version header, verifies TLS + and hostname, applies an explicit proxy policy, bounded quota/retry/backoff and + strict OHLC/pagination validation. REST is cold-bootstrap/gap-repair only. +- Live final 1m bars use authenticated native `ohlc_closed.1`; they do not poll + REST every minute. TRADE and BAR share the bounded lossless raw edge and reach + consumers only through durable broker ACK plus the Rust canonical core. +- The last final BAR watermark is atomic and bound to slice, authority, catalog, + acquisition revision and exact BAR bindings. It advances only after all + related durable ACKs; corrupt, partial or mismatched state fails closed. +- An official SDK snapshot without a redistribution license is protocol + reference, not vendored source. Insecure certificate disabling and examples + carrying credentials are never copied into release artifacts. +- If the primary host cannot reach official DNSE REST, run a separately governed + low-rate DNSE acquisition edge in an approved egress domain and publish + authenticated raw envelopes over mTLS/ACL. Never relabel vnstock/V1 Parquet as + `DNSE_DIRECT`; WebSocket-only collection may build durable future history but + is not an instant replacement for a fresh 500-row bootstrap. + ### 14.5 Failover state machine ```text @@ -4884,6 +4904,244 @@ replicated transport, production telemetry/security, full consumer cutover and regional DR claims until their real infrastructure exists. V1 remains authoritative and was not restarted or mutated. +## Appendix E — Phase 9.0-B isolated V2 beta boundary + +Phase 9.0-B reuses the Phase 7 V2 API/query/stream topology after the runtime +correctness fixes of Phase 9.0-A. It is an evidence refresh on the migrated +host, not a new public contract and not a Rust authority promotion. + +The approved topology contains one immutable query replica, two fenced +active/passive stream replicas, one dedicated AOF Redis, bounded canonical +spool volumes and an optional continuous V1 read-only bridge. Only the bridge +joins the existing internal V1 network. All V2 ingress is loopback-only and all +state, keys, groups, credentials, audit files and Compose resources use an +isolated Phase 9.0-B namespace. + +The first slice is fixed to `BINANCE / USDM / PERPETUAL / BTCUSDT / BAR / 1m`. +The bridge obtains authentic final bars through the internal V1 API and cannot +resolve arbitrary URLs or call a venue directly. V1 remains source authority; +the beta canonical spool is shadow evidence only. The beta may neither write +legacy Redis keys/channels nor claim execution eligibility. + +Certification combines Sections 18-19, 24-25, 30, 32, 37-41 and Appendix B: + +1. freeze V2 OpenAPI/Protobuf/SDK and verify V1 compatibility; +2. prove consumer-bound workload identity, entitlement, rate/concurrency bounds + and cursor signing/rotation/adversarial rejection; +3. compare provider-authentic V1 and V2 final bars field-for-field and prove + deterministic event identity, deduplication and contiguous replay-to-live; +4. stop the active stream owner and require a higher fencing epoch before the + passive owner serves work; stale-owner writes remain rejected; +5. stop/restart beta Redis and processes, require dependency-derived readiness, + preserve bounded durable state and keep V1 fallback available; +6. measure CPU, memory, PIDs, Redis/spool growth and latency under normal, burst + and slow-consumer cases; and +7. remove all disposable resources and verify the V1 topology, state and public + contract are byte/identity-equivalent before and after. + +Passing this appendix permits only an isolated `V2_BETA_READ_ONLY` review. It +does not close replicated broker, production OTel/alerting, workload identity, +external secrets, signature admission, independent DR, consumer registration +or exact authority-slice approval. Those remain mandatory before Phase 9.1. + +Phase 9.0-B certification completed on candidate revision +`1c881389b4ee21a153903505822c61512b176044` with exact authentic-bar parity, +active/passive fencing, dependency recovery, adversarial security, bounded load +and zero-residue cleanup all passing. The frozen report is +[`PHASE90B_ISOLATED_V2_BETA_REPORT.md`](evidence/PHASE90B_ISOLATED_V2_BETA_REPORT.md). +This evidence does not alter the authority boundary above. + +## Appendix F — Phase 9.0-C production prerequisite boundary + +Phase 9.0-C converts the Phase 6 and Phase 9 production blockers into one +provider-neutral, machine-verifiable gate bundle. The evaluator distinguishes +`TEST`, `LOCAL_REHEARSAL`, `PRODUCTION` and `INDEPENDENT_FAILURE_DOMAIN` +evidence. A stronger gate cannot be satisfied by evidence from a weaker scope. + +The mandatory gates are replicated durable transport; OTel collection, +persistent dashboards and acknowledged alert routing; workload identity, RBAC +and network policy; external secret rotation; signed artifact admission; +PostgreSQL PITR; object-store restore; independent failure-domain DR; +Redis/projector reconstruction; complete affected-consumer registration and +rollback; persistent authority/sink fencing; and explicit exact-slice approval. + +Evidence records carry an ID, gate, environment, scope, status, artifact hash, +observation/expiry time, issuer and immutable details. Missing, duplicate, +expired, malformed, contradictory or lower-scope evidence fails closed. Local +Kafka replication, debug OTel exporters, self-signed test keys and same-host +restore remain useful rehearsal evidence but never become production proof. + +Authority identity and transitions follow Section 30. PostgreSQL stores the +current exact slice and append-only audit; compacted Kafka remains the durable +distribution/audit substrate. A compare-and-swap transition checks current +state, authority revision, owner, lease epoch and partition-plan epoch. Canary +and primary transitions additionally require the exact passing prerequisite +bundle and named operator approval. + +Passing Phase 9.0-C code and local tests may yield +`COMPLETE_CONTROL_PLANE / NO_GO_EXTERNAL`. Only real infrastructure evidence can +yield `PRODUCTION_PREREQUISITES_PASS`. Neither result changes V1 automatically; +Phase 9.1 remains a separately approved exact-slice canary. + +Implementation status (2026-08-18): `COMPLETE_CONTROL_PLANE / +NO_GO_EXTERNAL`. The strict 12-gate evaluator, additive persistent authority +CAS/audit schema, frozen candidate, migration smoke, operator runbook and +checksummed evidence are complete. Local, focused and full candidate suites pass +with V1 unchanged. The external infrastructure/operator gates remain explicit; +none was relabeled or bypassed to manufacture a production approval. + +## Appendix G — Phase 9.1 Rust canary boundary + +Phase 9.1 is an exact-slice dual-read/compare stage, not source-authority +cutover. Python remains the only public and V1 compatibility writer. Rust may +publish only to a dedicated canary canonical namespace that no production +consumer treats as authoritative. + +The production path accepts one Phase 9.0-C decision bundle only when it is +`GO`, fresh, checksummed and bound to the exact candidate digest, partition-plan +epoch, image/signature, contract, normalizer, adapter, catalog, source policy, +consumer set, blast radius, operator and hold window. The isolated rehearsal +path has a separate entrypoint and cannot invoke production CAS or public sinks. +No environment variable or test mode may convert rehearsal evidence into a +production decision. + +A versioned Phase 9 authority record binds: + +```text +slice_id +state +owner_id +authority_revision +lease_epoch +partition_plan_epoch +candidate_digest +prerequisite_bundle_id +start_watermark +approved_by +approved_at +hold_until +``` + +Every canary publication repeats slice, owner and all three epochs plus source +watermark. The sink rejects wrong or stale identity before durable append. +`RUST_SHADOW` can write shadow targets; `RUST_CANARY` adds only the isolated +canary canonical target; `BLOCKED` writes nothing. Public V2 and Legacy V1 remain +forbidden throughout Phase 9.1. + +Parity compares Python and Rust outputs from the same authentic captured frame +range. Correctness-critical fields have zero tolerance. Guardrails block on any +unexplained mismatch, open gap, final/revision mismatch, duplicate external +publication, stale writer, authority ambiguity, durable ACK failure or approved +lag/freshness/resource threshold breach. A monotonic hold-down prevents noisy +automatic re-entry. + +The certification matrix includes deterministic replay, burst, clean-process +restart, stale lease/owner/revision/partition, producer and broker interruption, +slow consumer, guardrail block, rollback to shadow, V1 compatibility and exact +cleanup. Same-host Kafka and captured frames prove implementation behavior only; +they do not close the production/failure-domain prerequisites from Appendix F. +With a `NO_GO_EXTERNAL` prerequisite decision the only valid result is +`COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`. + + +### Phase 9.1 implementation closure (2026-08-18) + +The isolated implementation is certified as +`COMPLETE_IMPLEMENTATION / CANARY_NOT_AUTHORIZED`. The frozen authentic capture +produced 25,600 events with zero Python/Rust semantic mismatch over three clean +Rust processes. The replicated TLS/ACL broker rehearsal passed exact sink +fencing, one-replica availability, below-min-ISR fail-closed behavior, full +restart recovery, compacted authority reconstruction, immutable audit order, +64-record slow-consumer catch-up and rollback to `RUST_SHADOW`. Public and +legacy write counts remained zero; V1 health and topology were unchanged. + +Operator artifacts are the [runbook](../docs/runbooks/phase91-rust-canary.md), +[machine evidence](../upgrade/evidence/phase91-rust-canary-certification.json), +[human report](../upgrade/evidence/PHASE91_RUST_CANARY_REPORT.md) and +[checksum manifest](../upgrade/evidence/phase91-evidence.sha256). Phase 9.0-C +still returns `NO_GO_EXTERNAL`; no production authority transition is allowed +until independent production prerequisites and exact-slice approval are +provided. + +## Appendix H — Phase 9.2 bounded Rust primary boundary + +Phase 9.2 promotes no broad venue and changes no public API. It defines one +provider-neutral ownership handoff for one exact slice. With the current +`NO_GO_EXTERNAL` decision, only an isolated rehearsal is legal; production V1, +Redis, subscriptions and public destinations remain untouched. + +The immutable handoff chain is: + +```text +old-owner durable terminal checkpoint at W + -> same-range parity/reconciliation accepted through W + -> persistent CAS: revision + 1, lease epoch + 1, new owner + -> final sink and compatibility projector load accepted authority + -> first new-owner authoritative watermark W + 1 +``` + +A terminal checkpoint binds slice, old owner, authority revision, lease epoch, +partition-plan epoch, source session, connection generation, terminal event ID, +terminal payload hash and durable watermark. An accepted handoff additionally +binds the checkpoint digest, exact overlap range, zero gaps, zero semantic +mismatches, matching event/output counts, candidate and prerequisite bundle, +approver and expiration. These records are append-only. + +The authority schema is additive. Phase 8 v1 and Phase 9.1 v2 records remain +decodable. Phase 9.2 v3 grants authoritative targets only in `RUST_PRIMARY` (or +the newer fenced `PYTHON_PRIMARY` rollback owner). Revision must advance exactly +one per transition. Owner changes require a strictly newer lease. A changed +slice, candidate or partition plan cannot be smuggled through a handoff. + +Final canonical, public V2 and legacy V1 compatibility destinations each keep an +independent acknowledged watermark under the same authority identity. A write +is accepted only for the active owner/revision/lease/plan, correct target and +next boundary. Watermark state advances after durable ACK. A crash before ACK is +retryable; a crash after ACK is reconciled by deterministic event ID and durable +cursor. Stale and zombie writers are rejected at the sink/projector, even if a +producer process still believes its lease is valid. + +Production authorization is distinct from rehearsal. It requires: + +- a fresh exact Phase 9.0-C `GO` bundle; +- a real `RUST_CANARY` hold completed with zero correctness breach; +- registered consumers and immutable Python rollback manifest; +- accepted terminal/handoff records for the exact slice; +- explicit operator, ticket, blast-radius and hold-window approval. + +The isolated harness uses authentic frozen provider frames and replicated +test-only topics to model final/public/legacy projections. Those topic names can +never equal production destinations. It tests `N-1/N/N+1`, off-by-one, gap, +duplicate, stale owner/revision/lease/plan, competing CAS, process loss before +and after CAS, sink/projector restart, broker replica loss/min-ISR, full restart, +slow consumer and formal rollback. It records production mutations as zero, +checks V1 topology before/after and deletes only disposable resources. + +A process loading an already-primary v3 authority starts fail-closed. It must +reconstruct the latest ACKed watermark independently from each durable final, +public V2 and legacy V1 destination, validate identity and a contiguous range, +and restore that target fence before any append. A duplicate terminal +watermark remains rejected and the first permitted write is exactly W+1. This +prevents a crash after durable broker ACK but before process-local commit from +re-emitting an external duplicate. + +A valid isolated result is +`COMPLETE_IMPLEMENTATION / PRIMARY_NOT_AUTHORIZED`. It proves protocol and +recovery behavior; it does not authorize disabling the Python subscription or +claim independent failure-domain resilience. + +**Implementation closure (2026-08-18):** The isolated Phase 9.2 harness passed +over 25,600 authentic provider events with zero Python/Rust semantic mismatch. +The broker-backed second-process recovery reconstructed three independent +target watermarks at 180 and resumed exactly at 181; canonical, V2 and V1 +projections remained identical and gap-free. PostgreSQL CAS/handoff, formal +rollback, replica-loss/min-ISR, full restart, slow-consumer, full Rust/Python +regression and scoped cleanup gates passed. Evidence is frozen in +[`phase92-bounded-primary-certification.json`](../upgrade/evidence/phase92-bounded-primary-certification.json), +the [human report](../upgrade/evidence/PHASE92_BOUNDED_PRIMARY_REPORT.md) and +[`phase92-evidence.sha256`](../upgrade/evidence/phase92-evidence.sha256). Phase +9.0-C remains `NO_GO_EXTERNAL`, so this closure grants no production authority. + ### Option and Deribit extension boundary Adding an option venue must not require changing canonical core identities or rewriting distribution. The common boundary must represent: @@ -4897,3 +5155,559 @@ Adding an option venue must not require changing canonical core identities or re - source authority, quality state and entitlement/licensing metadata. An adapter declares unsupported capabilities explicitly. Phase 3 uses sanitized Deribit-style fixtures to prove the boundary; Phase 6 certifies architecture readiness. Real Deribit activation remains a separate adapter certification requiring official source semantics, credentials, licensing and capacity evidence. + + +## Appendix I — Phase 9.3 hold, closure and independent expansion + +Phase 9.3 is a control-plane boundary after one exact slice has become a real +`RUST_PRIMARY`. It does not introduce a new sink authority state. The data plane +continues to use the Phase 9.2 owner/revision/lease/partition and watermark +fences. Hold and closure records describe whether operators may reduce the +rollback posture; they never grant writes by themselves. + +### Hold identity and observation contract + +A hold is immutable and binds: + +```text +hold_id +slice_id +candidate_digest +prerequisite_bundle_id +owner_id +authority_revision +lease_epoch +partition_plan_epoch +started_at_ns +required_until_ns +policy_digest +``` + +Each observation repeats that identity and carries an increasing sequence, +observation time, last durable watermark, correctness counters, source quality, +consumer checkpoint state and bounded resource/lag metrics. The evaluator +requires ordered observations across the whole approved interval and a maximum +sample gap. It never fabricates a missing observation. + +The following values are zero tolerance: + +```text +semantic_mismatches +open_gaps +duplicate_external_writes +accepted_stale_writer_writes +authority_ambiguities +durable_ack_failures +projection_mismatches +consumer_checkpoint_regressions +unexplained_quality_failures +``` + +Lag, freshness, queue, spool, CPU and RSS use explicit upper bounds. An owner, +authority revision, lease, partition plan or candidate change invalidates the +hold. A breach is sticky for that hold identifier; a later clean sample cannot +rewrite history. Starting again requires a new hold and preserves the failed +record. + +### Rollback-window closure + +A closure authorizer consumes six independently frozen inputs: + +1. fresh Phase 9.0-C production prerequisite `GO` bound to the candidate; +2. production-authorized Phase 9.2 primary evidence for the exact owner/epochs; +3. a passing real hold decision covering the minimum approved duration; +4. complete consumer registry snapshot with healthy contiguous checkpoints; +5. exact authority registry snapshot plus a fresh successful rollback rehearsal; +6. explicit operator/change-ticket approval with bounded expiry. + +The database closure transaction locks the current authority row and rechecks +state `RUST_PRIMARY`, owner, revision, lease, partition-plan epoch, candidate, +bundle and hold identity. It inserts one immutable closure record. It must not +update authority state, owner, revision, lease, watermark, public-write or +legacy-write fields. A concurrent authority change makes the closure fail. + +Closing the window means normal rollback is no longer an unreviewed routine +operation. It does not delete the Python manifest or make emergency recovery +impossible. A later incident still fences Rust first and creates new immutable +rollback authority/audit records. + +### Consumer and authority registry freeze + +The closure snapshot records only stable IDs, versions, digests, contiguous +checkpoints and health decisions. It does not embed secrets, tokens or raw +unbounded logs. Every affected critical consumer must be registered exactly +once, acknowledge the current authority revision and expose no checkpoint +regression or unresolved migration. Unknown or duplicate consumers fail closed. + +### Independent expansion manifests + +Expansion types are provider-neutral: + +| Expansion | Mandatory independent certification | +|---|---| +| `INSTRUMENT_PARTITION` | partition churn, source capacity, exact-frame parity, handoff and rollback | +| `BBO` | quote identity, coalescing policy, freshness, ordering and reconnect | +| `L2_BOOK` | snapshot/delta sequence, checksum, resync, lossless backpressure and capacity | +| `BAR_LIFECYCLE` | final/revised/cancelled lineage, close-time semantics and replay | +| `VENUE_MARKET` | adapter capability, instrument identity, provider semantics, entitlement and DR | + +Every manifest has a new `expansion_id`, candidate digest, scope digest, +partition-plan epoch, required-gate set and status +`INDEPENDENT_CERTIFICATION_REQUIRED`. It sets write authority false. Parent +hold/closure evidence is provenance only and cannot satisfy child gates. +Combining expansion classes in one manifest is forbidden so risk and rollback +remain bounded. + +### Runtime decommission boundary + +A Python hot path may be nominated for removal only when it owns zero slices, +is absent from every active rollback manifest and consumer dependency, all +replacement windows are governed closed, and repository cleanup is explicitly +approved. Shared canonical contracts, provider semantics, fixtures, migration +knowledge and compatibility projectors are not removed merely because one hot +path moved to Rust. + +### Current decision boundary and certification + +The current Phase 9.0-C result is `NO_GO_EXTERNAL`, and Phase 9.2 produced only +isolated primary evidence. Therefore Phase 9.3 may certify contract validation, +PostgreSQL transaction behavior, fail-closed closure denial, expansion +independence, V1 invariants and exact cleanup. It may not claim a real hold, +close a production rollback window, decommission Python or authorize an +expansion. + +The maximum local result is: + +```text +COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED +``` + +Production completion later requires authentic sustained observations over the +approved wall-clock duration, real consumer checkpoints, current authority and +rollback evidence, independent infrastructure scope and explicit operator +approval. No accelerated fixture or same-host rehearsal can satisfy that gate. + + +**Implementation closure (2026-08-19):** Phase 9.3 control-plane implementation +is certified as `COMPLETE_CONTROL_PLANE / PRODUCTION_HOLD_NOT_STARTED`. Typed +hold evaluation, immutable registry snapshots, exact authority-locking closure, +closure-digest-bound independent expansion and conservative decommission gates +pass focused and full regression. PostgreSQL smoke proved that closure changed +no authority field and that dirty/sparse/stale/mutable evidence fails closed. +The final suites passed 415 Python tests with 5 intentional skips and 40 Rust +tests with fmt/clippy clean. V1 remained healthy and unchanged; production +mutations and residual Phase 9.3 resources are zero. Evidence is frozen in +[`phase93-hold-close-expand-certification.json`](../upgrade/evidence/phase93-hold-close-expand-certification.json), +the [human report](../upgrade/evidence/PHASE93_HOLD_CLOSE_EXPAND_REPORT.md), the +[operator runbook](../docs/runbooks/phase93-hold-close-expand.md) and +[`phase93-evidence.sha256`](../upgrade/evidence/phase93-evidence.sha256). This +closure starts no production hold and grants no production authority. + +## Appendix J — Phase B stable release execution ledger + +This appendix is the detailed execution index for Phase B in +`DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md`. It does not replace the immutable +journal there. The plan owns status and exact chronological evidence; this +guide owns the bounded work packages, decision rules and closure gates. + +### J.1 Subphase rule + +Phase B is split into `B.0` through `B.4`. Work on a discovered defect remains +inside the active subphase as a numbered repair slice. The agent must stop after +each coherent test gate, write the result to both this ledger and the main plan, +and commit that tested slice before continuing. A failed runtime candidate is +recorded as diagnostic evidence and cannot be cited as release acceptance. + +Each entry records the exact source/image SHA, tests actually run, pass/fail/skip +counts, provider/test provenance, correctness conclusion, resource evidence, +runtime impact, cleanup and remaining gate. Generated market data is permitted +only in explicit unit/failure/capacity tests. Real-provider or durably captured +provider bytes are mandatory for shadow/stable runtime claims. + +### J.2 B.0 — Contract and stable edge (`COMPLETE`) + +Scope: deterministic catalog identity, canonical V2 query/stream/projector +contracts, stable consumer manifests, isolated topology and exact V1 +compatibility. B1-B4 passed targeted suites of 6, 32, 26 and 34 tests. The +conditional Redis case was executed separately against named disposable Redis +and passed Lua TTL, Pub/Sub, idempotency and fencing. No V1 process, route, +authority or current Redis namespace changed. + +Conclusion: stable outer contracts and isolation boundary are accepted. + +### J.3 B.1 — Runtime correctness and capacity (`COMPLETE`) + +Scope: authentic provider acquisition, Rust canonicalization, final/revised BAR +lifecycle, deterministic replay, lossless versus latest-state delivery, bounded +SQLite/Redis state, Kafka lag and query latency. B5-B8 found and closed +deployment, provider-control, replay determinism, generation persistence, +projector batching/backpressure and hot-partition capacity defects. + +Final accepted evidence includes 478 Python tests with 6 explicit conditional +skips, clean Rust fmt/Clippy/workspace gates, 100,000-event Rust benchmarks +above the 50,000 events/s floor, exactly 2,000 authentic final closed crypto +BARs, core/projector lag of 50/29 at the observation point, canonical-only +SQLite with every partition at or below 10,000 records, zero quarantine and +bounded Redis/app memory. The 100-request execution QUOTE sample measured p50 +2.94 ms, p95 3.74 ms and p99 32.44 ms after the query-tail repair. + +Conclusion: the isolated runtime data path and measured capacity gate are +accepted; intermediate overloaded candidates are not release artifacts. + +### J.4 B.2 — Controlled consumer acceptance (`PARTIAL_EXTERNAL`) + +Scope: governed manifests for Binance alpha, OKX alpha, VN alpha, monitoring +and Trading System paper, each using warmup -> signed cursor -> replay -> live. +Targeted repairs passed 31 multi-venue/session tests, 85 broader tests with one +infrastructure skip, 14 OKX close-boundary tests, 22 SDK warmup tests and 39 +cursor/stream tests with one dependency skip. + +Immutable `df88de0` accepted Binance and OKX alpha flows with 500 authentic +rows per binding, replica-equal watermarks, live monitoring and Trading System +TRADE/QUOTE snapshots at 129-779 ms freshness. DNSE is not accepted on this +host because official REST TCP/443 is unreachable. Authenticated WebSocket +reachability or legacy Parquet without exact `DNSE_DIRECT` lineage cannot +substitute for the missing real history gate. + +Conclusion: crypto and paper adapter consumers pass; VN remains an explicit +external egress/provider gate and must fail closed. + +### J.5 B.3 — Durability and recovery (`COMPLETE`) + +B13 passed exact active/passive handoff: the SDK ACKed offset 2,271 under epoch +1 and resumed durable replay at offset 2,272 under epoch 2 before `LIVE`, with +no gRPC ContextVar/finalizer error. Its focused suites passed 51 tests with one +dependency skip. B14 passed two-broker minISR failure and recovery: no false +ACK, no broker OOM at the corrected 768 MiB bound, all raw partitions restored +ISR `1,2,3`, offsets advanced, quarantine remained zero and Trading System +QUOTE returned execution-eligible at 239 ms freshness. + +B15 failed because a Redis-only earliest replay retained a trimmed SQLite cache +and reintroduced old event IDs at new logical offsets, producing +`OPEN_SEQUENCE_GAP`. The accepted B16 repair treats Redis plus SQLite as one +rebuildable projection cache unit and fences them with a persistent cache +identity. Current source has passed 41 targeted tests with one explicit +real-Redis integration skip; this is unit-implemented, not runtime-accepted. + +The real-Redis generation gate then ran separately against a named Redis 7.2 +container with persistence disabled and a 16 MiB no-eviction bound. It passed +atomic first bind, exact-ID reuse, conflicting-ID rejection, TTL/latest writes, +single Pub/Sub publication, duplicate suppression, stale-lease fencing and +write rejection after live identity deletion in 0.210 seconds. Together with +the network-disabled run, B16 targeted evidence is 42/42 passed with zero +skips. The disposable Redis container and network were removed. + +`B17` now provides `scripts/rebuild_v2_stable_projection_cache.py`, plan-only +by default and apply-gated by the exact confirmation token. Its allowlist stops +only projector/query/stream roles, deletes only `canonical-cache.sqlite3` plus +its WAL/SHM sidecars, flushes only `stable_redis`, resets only +`stable-projector-v1` on `md.canonical.v2`, and starts stream -> projector -> +query after two zero-lag samples and readiness gates. Any failure leaves the +cache unavailable and rerunnable from Kafka; no partial cache may report ready. + +Six command-policy tests passed for authorization, exact scope, lag parsing, +wrong-project rejection and abort-before-mutation. Combined B16/B17 targeted +execution ran 48 tests with 47 passes and one real-Redis conditional skip; that +same Redis case passed separately. + +The first authentic replay converged from about 448,000 to 32 records but +timed out because two exact-zero samples cannot be guaranteed while producers +remain live. It failed closed with query replicas stopped. The corrected gate +uses three consecutive samples `<=250` across exactly six canonical partitions, +then projector identity readiness and strict downstream freshness/gap checks. +Acquisition remains live and no operator-configurable relaxation is allowed. + +The corrected guarded rebuild then passed with all six partitions, three +consecutive samples inside the fixed 250-record live-lag bound, observed bound +232 and final lag 63. Redis was rebuilt to 47 keys and readers were started only +after projector identity readiness. A strict authenticated consumer warmup +immediately afterward still failed closed on an unresolved BAR sequence gap. +Because this was a fresh cache built from canonical Kafka, B.3 now treats the +gap as a canonical/provider-bootstrap or revision-ordering defect rather than a +stale-cache artifact. The repair must identify the exact real-data +discontinuity; synthetic bars and relaxed gap policy are prohibited. + +Bounded B18 diagnosis then proved the exact discontinuity: Binance Spot and +USD-M both lacked 16:53 UTC after a four-record Kafka ACK failure at 16:54; OKX +remained continuous. The edge advanced `_last_open_ms` before ACK and only read +the latest closed bar on retry. The repair keeps the watermark ACK-authoritative +and performs bounded, continuity-validated provider-history catch-up whenever +more than one interval is pending. It must fail closed on incomplete history and +must not synthesize a candle. + +The B18 implementation keeps a per-cycle provider observation boundary and a +1,000-row hard catch-up bound. It advances each binding only after the complete +ordered batch receives Kafka ACKs. Sixteen focused deployment/history tests +passed, including multi-venue ACK failure/retry and incomplete-history fencing. +Runtime healing and strict consumer acceptance remain required. + +Runtime healing with `2041f18` then exposed a second bounded defect: repeated +historical BARs retain one semantic event ID/hash but carry new capture +provenance, so generic byte-level spool collision fencing stopped the projector. +The stable projector, not the generic spool, must recognize only equal 32-byte +canonical payload hashes as semantic duplicates and must keep late historical +repairs from regressing Redis/latest. Changed market semantics remain a hard +collision. + +The bounded projector repair is now unit-passed. It recomputes and validates +both canonical market-payload hashes, requires event ID and partition equality, +checkpoints provenance-only duplicates without a second fan-out, preserves the +generic spool collision policy and stores late BARs without replacing newer +latest state. The focused stable-edge/deployment/history suite ran 46 cases in +the immutable `2041f18` test image with network disabled: 45 passed and the one +conditional Redis case, already proven separately, was skipped. No runtime was +mutated. This is implementation evidence only; B.3 still requires a committed +immutable image, fresh atomic cache rebuild and real-provider consumer gate. + +A committed `8851166` runtime replay remained fail-closed. It exceeded the +900-second operator deadline with 733,158 records of lag and then exposed actual +BAR conflicts. Read-only inspection proved that OKX REST/WS rows had equal +numeric values but different exact decimal spelling and acquisition origin, +whereas early Binance rows had materially different close/volume/trade counts +from the later settled backfill. B19 therefore narrows semantic equality only +for BAR decimal normalization and origin provenance, keeps non-BAR/hash and all +actual BAR values strict, and adds aligned provider close-settlement grace. +Changed numeric bars remain rejected; the projector must not invent revisions. +The candidate remains unavailable and its volumes are retained until a clean +runtime rehearsal is explicitly authorized. + +B19 is unit-passed. BAR duplicate comparison now uses exact numeric Decimal +semantics while validating the preserved venue spelling, ignores only acquisition +origin and equivalent trailing zeros, and leaves every non-BAR payload plus all +actual BAR values strict. The generic spool remains byte-immutable. Binance and +OKX closed-history/latest observation now share an explicit two-second settlement +grace aligned to the poll boundary. The focused suite ran 48 cases: 47 passed and +the separately proven Redis conditional case was the sole skip. Runtime remains +fail-closed because the retained isolated Kafka log contains early-final Binance +rows with materially wrong values; relaxing changed-value policy is forbidden. + +Immutable `c61fa39` then passed a read-only durable-provider classifier gate: +OKX equivalent numeric Decimal/origin variants were semantic duplicates while a +Binance row with materially changed close/volume/trade count remained a hard +conflict. No Kafka offset was committed. The invalid retained candidate log was +not relaxed or rewritten; all Python roles were consolidated onto `c61fa39` and +left stopped while Kafka/Rust/Redis volumes were retained for audit. + +Incremental cleanup removed only three zero-reference Python tags (`e002da6`, +`2041f18`, `8851166`) plus four QDL BuildKit records older than one hour. Image +storage fell 11.1 -> 9.278 GB and build cache 9.097 -> 8.488 GB, about 2.43 GB +recovered. V1, `c61fa39`, rollback `cfc0246`, Kafka/Redis and all volumes remain. + +Remaining B.3 gate: an explicitly approved clean isolated Kafka candidate, +fresh atomic Redis-plus-SQLite rebuild, zero gap/collision/quarantine, replica +equality, signed SDK replay/live, fresh Trading System paper data and unchanged +V1. + +On 2026-08-20 the operator resumed this closure. The proposed reset boundary +is only `qdl_v2_stable_candidate` Kafka broker volumes 1-3 plus `stable_state`; +`stable_tls`, all V1/production state and every other Docker project are +preserved. The candidate must use real provider data, immutable `c61fa39` +Python code and the pinned Rust candidate, then pass settlement/catch-up, +zero-gap/collision/quarantine, atomic cache rebuild, replica equality, signed +SDK replay/live and Trading System paper-consumer checks. Failure leaves query +and stream readers unavailable and V1 authoritative. B.4 cannot start from a +partial result. The first reset attempt was safety-rejected before execution. +The operator subsequently approved deletion of exactly the three candidate +Kafka volumes and candidate `stable_state`; candidate `stable_tls`, all V1 and +all production state remain protected. + +The approved clean reset and broker bootstrap passed: RF3/minISR2, three topics, +six partitions, mTLS/ACLs and all nine preserved TLS files were verified. +Authentic bootstrap ACKed 2,000 settled Binance/OKX BARs and the next closed +cycle; Rust reported no quarantine/collision/error. A dependency-scope defect +was then blocked before mutation: the rebuild CLI could rerun `stable_tls_init` +when starting readers. Recovery starts must use `up --no-deps` after explicit +infra validation, with a unit test proving the exact command, so preserved TLS +and unrelated dependencies cannot be rewritten. + + +The clean B.3 closure passed. The atomic cache rebuild converged on all six +canonical partitions with observed lag bound 46 and final lag 19, restoring 47 +Redis keys before either query replica became ready. Signed SDK acceptance +returned 500 full-coverage final BARs for Binance and OKX from both replicas, +with equal market semantics and watermark 512; request-time +`quality.freshness_ms` was correctly treated as an observation rather than +replicated state. Both alpha consumers observed `REPLAYING -> LIVE` and ACKed +two contiguous records. Four Binance/OKX Trading System paper TRADE/QUOTE +snapshots were execution eligible at 146-316 ms freshness. + +The bounded SQLite cache held 73,456 records across 12 partitions, no retained +offset gap, no duplicate event ID, no quarantine and at most 10,000 rows per +partition. Kafka quarantine was empty; observed projector/core lag totals were +34/12. One stream owner was READY, the other STANDBY; TLS bundle hashes matched, +V1 health remained `ok`, and memory stayed within every container bound. +The five Phase B regression modules ran 63 tests: 62 passed and one separately +proven real-Redis conditional case skipped under network-disabled execution. +Conclusion: B.3 is `PASS`/`COMPLETE`; B.4 remains `NOT_STARTED`, and no +production authority or consumer cutover is implied. + +### J.6 B.4 — Release certification and cleanup (`IN_PROGRESS`) + + +B.4 started on 2026-08-20 with provisional common source SHA `5054e1e`. +Certification is correctness-first: full Python, Rust, Buf/OpenAPI, +security/package/capacity/compatibility gates must pass before either final +image is built. Both images then use that same source SHA and only the isolated +candidate may be recreated. Rollback pins `c61fa39` Python and `cfc0246` Rust +against preserved candidate Kafka/state/TLS. V1 and production authority are +immutable; release publication, push/merge and cutover are not authorized. + + +The full Python certification gate passed 503 tests with 497 passes and six +explicit conditional skips under network-disabled, read-only, cap-dropped +execution. Source was `5054e1e` with dependencies from the immutable Python +candidate. No runtime or durable state was mutated. Rust and contract gates are +still pending, so B.4 remains `IN_PROGRESS`. + +The first Rust gate found a builder-packaging defect before any runtime change: +release binaries compiled, but test compilation could not see immutable +`contracts/golden` and `tests/fixtures/phase2` inputs because the builder did +not copy them. The bounded repair adds only those test oracle paths plus a +packaging regression assertion, then rebuilds and reruns fmt/Clippy/tests. +Runtime-stage contents and market-data semantics remain unchanged. + +The bounded repair is accepted. The rebuilt no-network/cap-dropped builder +passed Rust format, locked workspace Clippy with warnings denied and all 62 +workspace tests with zero failures/skips. The six release-packaging regression +tests also passed. This covers exact Python/Rust golden bytes, multi-venue +provider semantics, deterministic replay, ordering/gap/quarantine, authority +handoff and rollback, Kafka security bindings, delivery classes and VN source +identity. No runtime was recreated; contract/security/package/capacity gates +remain open. + +The contract gate is also accepted: Buf 1.50.0 format/lint, two frozen-baseline +breaking checks and generation passed with no generated-code drift; seven +Python golden-contract tests passed. OpenAPI comparison against `dev` retained +10 operations and 42 schemas with zero hard break or security/required-parameter +change. Runtime remained untouched. Security/package/capacity and final +one-SHA artifact gates remain open. + +Capacity diagnostics passed the approved Phase 2 persistence/replay and Phase 5 +eight-replica API gates. An exploratory 10,000/40,000 events/s Python sustained +profile under 2 CPU failed closed at 5,656 events/s; these are above the +approved Phase 6 default 500/1,500 targets, so the result is diagnostic only and +does not lower any gate. The approved profile and final Rust release benchmark +remain mandatory. + +Cargo-deny then exposed a release-policy defect: advisories/bans/sources were +clean, but the Rust builder omitted the repository's tracked `deny.toml`, so its +containerized audit fell back to default deny-all and rejected every normal +permissive dependency. Host CI retained the policy. B.4 must copy and harden +that least-permissive license/source/advisory policy, add a packaging +regression check and rerun the checksum-pinned scanner. + +The repair is accepted. The explicit Linux policy has no advisory/license +exception, denies wildcard and unknown registry/Git sources, and allows only +the permissive licenses in the locked graph. Cargo-deny passed advisories, +bans, licenses and sources. Pip-audit found no known Python vulnerability; +Trivy found zero HIGH/CRITICAL source misconfiguration and zero secret. The +approved 80-partition Phase 6 profile also passed at 503.62/503.70 normal and +1,503.07 burst events/s with no queue reject, replay mismatch or memory growth. +Final full regression, one-SHA image/image-scan and candidate gates remain. + +The final source rerun first exposed only a harness omission: read-only +execution lacked the required non-root `/app/logs` tmpfs. The corrected +unchanged-source run passed all 504 tests with six explicit skips. Source-level +certification is closed; freeze/commit, same-SHA images, image scans and +isolated candidate recreation remain. + +The one-SHA artifact gate passed at source `ea84a21`: both non-root images +carried the exact revision, had zero HIGH/CRITICAL vulnerability/embedded +secret, and the final Rust core processed 100,000 events at 129,256 events/s +with p99 12.906 microseconds and zero duplicate/quarantine. + +The first isolated rolling recreation failed closed. All 13 app roles moved to +the new images without touching infra or V1, but the projector detected an +event-ID/market-semantics collision and two Rust workers reported 497 +quarantines. Query/stream readiness alone is rejected as acceptance. Stop +candidate app roles, preserve durable evidence and either repair the root cause +or restore the pinned rollback before B.4 can close. + +Read-only durable diagnosis narrowed the failure to two acquisition-lifecycle +defects, not weakened projector/core checks. Exactly 994 committed quarantine +records are OKX `candle1m` `STALE_GENERATION`: stateless REST used generation +1 after the WebSocket owner advanced the same BAR partition. Repeated Binance +REST bootstrap also emitted the same revision-0 BAR identity with materially +different close/volume/trade-count values. The bounded B.4 repair assigns one +REST owner to final 1m BAR for both Binance and OKX while retaining Rust as the +only canonical core; persists an atomic authority/catalog-bound last-ACKed +watermark in `stable_state`; skips overlap on restart; and uses the approved +10-second settlement ceiling. Corrupt or mismatched state, partial ACK, +incomplete history and changed immutable BAR semantics remain fail-closed. +V1/V2 public contracts, event identity and production authority are unchanged. +Restart/state-corruption/ACK-loss tests, native manifest proof, full regression +and a clean real-provider zero-gap/collision/quarantine rehearsal gate closure. + +The bounded repair is unit-accepted at acquisition revision 2. All four final +Binance/OKX 1m BAR bindings have one REST owner; native Rust ingestors retain +eight TRADE/QUOTE bindings and Rust remains the canonical core. Atomic +ACK-authoritative checkpoint restore is strict to slice/authority/catalog/ +acquisition identity, compose mounts initialized isolated state, and runtime +settlement is 10 seconds. Eighteen targeted tests passed; the five Phase B +modules ran 65 cases with 64 passes and one separately proven real-Redis skip. +No runtime or durable service was changed. Full regression, new same-SHA images +and clean real-provider restart acceptance remain. + +The repair full-source gate passed. Two initial Python harness attempts used a +root-owned child log tmpfs and stopped four imports without a domain failure; +the corrected non-root tmpfs run passed 500 of 506 tests with six explicit +conditional skips. Rust fmt, locked Clippy and all 62 workspace tests passed; +compose and diff validation were clean. Runtime remained untouched. Final +same-SHA artifacts, scans and isolated authentic-provider restart acceptance +remain. + +The final B.4 artifact and runtime gates passed at code source `2412572`. +The non-root Python/Rust images carry that exact revision; final source, +contract, package, security and capacity gates passed. The Rust benchmark +processed 100,000 events at 133,477.5 events/s with p99 14,124 ns and zero +duplicate/quarantine. The fresh isolated runtime bootstrapped 2,000 authentic +Binance/OKX closed BARs, restored its ACK-authoritative checkpoint across edge +restart, and retained zero Kafka/cache quarantine, zero offset gap and zero +event-ID duplicate. The bounded cache held 75,187 canonical records across 12 +partitions; projector lag was 35 under the 250-record gate. + +Released SDK acceptance returned replica-equal 500-row Binance/OKX warmups, +observed `REPLAYING -> LIVE`, and resumed each durable cursor at exactly the +previous offset + 1. Trading System paper received four authoritative, +execution-eligible Binance/OKX TRADE/QUOTE snapshots at 132-158 ms freshness. +No order or synthetic provider row was created. Exact candidate projects, +networks and disposable volumes were removed; final/rollback images and old +candidate TLS were retained. Forty-one exact B.4 BuildKit IDs were pruned, +reducing cache from 12.94 GB to 10.94 GB without a broad host prune. The exact +temporary secret bundle, scan JSON and SDK harness were deleted after evidence +was recorded. V1 port 8100 was never restarted and remained healthy. + +B.4 is therefore `PASS`/`COMPLETE` and `2.0.0 Internal Stable` artifacts are +ready for operator review. Phase B overall remains `PARTIAL_EXTERNAL` only for +the already recorded official DNSE provider gate. This is not authority or +consumer cutover approval; that remains a separate explicit transaction. The +compact ledger is +`upgrade/evidence/PHASE_B4_RELEASE_CERTIFICATION_REPORT.md`. + +Start only after B.3 is `PASS`. Run full Python discovery, Rust fmt/workspace +Clippy/tests, Buf/OpenAPI/package/security/capacity/compatibility gates; build +Python and Rust images from one final commit SHA; freeze compact evidence and +runbooks; remove only exact candidate containers, networks, volumes, bundles +and obsolete candidate images; verify V1 port 8100 and topology unchanged. + +Conclusion boundary: B.4 may declare `2.0.0 Internal Stable` artifacts ready +for operator review. It does not authorize production authority, route cutover +or consumer migration. + +Artifact hygiene is incremental, not deferred wholesale to B.4. Every tested +repair removes its named disposable containers/networks and unreferenced QDL +image tags. Keep only running V1, the active candidate and one named rollback +generation. Clean BuildKit only by the Data Layer builder/cache scope; never use +a broad host-wide prune on a shared server. Record exact objects and reclaimed +bytes in this ledger. B.4 replaces the surviving mixed candidate images with +one final SHA and performs the final bounded cleanup. + +The first incremental cleanup removed exactly two disposable Phase B builder +containers and 47 obsolete unreferenced QDL image tags while retaining V1, +`e002da6`, `cfc0246`, Kafka/Redis and all volumes. It then pruned only BuildKit +records matching `description~=qdl` and older than one hour. Image/cache cleanup +reduced root filesystem use from 91 GiB to 47 GiB; BuildKit cache fell from +50.2 GB to 6.484 GB. No broad Docker prune or runtime/data-volume mutation +occurred. Post-cleanup B16/B17 regression ran 49 cases: 48 passed and +the separately proven real-Redis conditional case was the sole skip; compile +and diff checks passed.