diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a030373..b364703 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -78,6 +78,24 @@ jobs: tar --no-same-owner -xzf "${archive}" "./${directory}/cargo-deny" check + sdk-python310: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + + - name: Set up declared minimum Python + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + with: + python-version: "3.10" + + - name: Build and import standalone SDK wheel + run: | + python scripts/build_qdl_sdk_release.py --output-dir /tmp/qdl-sdk + python -m pip install --disable-pip-version-check /tmp/qdl-sdk/qdl_sdk-2.0.0-py3-none-any.whl + cd "$(mktemp -d)" + PYTHONPATH= python -c 'import qdl_sdk; assert qdl_sdk.__version__ == "2.0.0"' + unit-tests: runs-on: ubuntu-latest steps: diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index e1fc820..32df41b 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5873,7 +5873,7 @@ and removes the superseded candidate/rollback artifacts after verification. ### Phase C - Production V2 And Rust Authority Cutover -**Status:** `C.0 RELEASE/MERGE PREPARATION IN PROGRESS / PRODUCTION CUTOVER NOT AUTHORIZED` +**Status:** `C.1 SHADOW-CERTIFIED / C.2 CONSUMER CLOSURE IN PROGRESS / PRODUCTION CUTOVER NOT AUTHORIZED` **Purpose:** move approved Binance and OKX feed slices from the current V1 authority to the stable V2 contract with Rust as the actual canonical realtime @@ -5887,8 +5887,8 @@ promotion until its provider gates pass. V1 paths and zero V2 paths; - no V2 stable container is running; only the retained `qdl_v2_stable_candidate_stable_tls` volume remains; -- the feature branch is 81 commits ahead of `dev`; the latest released - `2.0.0-2412572` images predate the bounded DNSE closure commits; +- the feature branch is more than 80 commits ahead of `dev`; the latest + released `2.0.0-2412572` images predate the bounded DNSE closure commits; - the stable compose and realtime binaries deliberately accept only `RUST_SHADOW`; Phase 9.2 proves the CAS/handoff/fencing behavior in an isolated rehearsal but is not wired into the long-running stable runtime; @@ -5926,23 +5926,98 @@ without mutating V1. The CLI prints identities/revisions/watermarks/digests only, never secrets. 7. Build one immutable Python/Rust image pair from the merged SHA, generate SBOM/provenance, and retain exactly one tested V1 rollback generation. +8. Publish `qdl_sdk==2.0.0` as a standalone immutable wheel with checksum, + SBOM and generated-contract digest. Trading System and execution-alpha base + images pin the same artifact; neither repository copies Data Layer service + internals or maintains an independent V2 schema parser. +9. Replace the bounded BTC-only certification catalog with a deterministic + production catalog/binding generator driven by approved venue metadata and + consumer manifests. Instrument UIDs remain stable across restart/rebuild; + arbitrary alpha symbols are resolved through `/v2/instruments`, never by + hardcoded UUIDs in consumers. Only demanded/approved Binance and OKX feeds + are acquired; disabled symbols fail readiness rather than creating data. +10. Freeze two versioned consumer classes: Trading System + `EXECUTION` grade and shared alpha runtime `ALPHA` grade. The execution + client requires authoritative/fresh/gap-free snapshots. Alpha warmup uses + final BAR snapshot/cursor/replay and the same SDK, while strategy/order + source remains untouched. V1 fallback is explicit and source-switch audited. **C.0 gates:** migration idempotency, transactional outbox replay, compacted authority recovery, stale-writer rejection, restart recovery, exact Python/Rust -parity, public V1 contract compatibility, full source/Clippy/security tests and -zero production mutation. Conclusion must be either `PASS` or `FAIL`; missing -authority wiring cannot be deferred as operational debt. +parity, deterministic multi-symbol catalog generation, SDK wheel reproducibility +and checksum verification, Trading System/alpha consumer contract tests, public +V1 compatibility, full source/Clippy/security tests and zero production +mutation. Conclusion must be either `PASS` or `FAIL`; missing authority, +catalog or SDK wiring cannot be deferred as operational debt. **C.0 implementation journal:** +- `2026-08-20 C.0 LONG-RUNNING PRIMARY BRIDGE ACTIVE`: implement the + production consume-transform-produce boundary as a separate Rust runtime, + leaving the certified shadow binary and V1 runtime unchanged. Every accepted + raw offset must transactionally commit its canonical/quarantine decision, + per-target projection progress and compacted target checkpoint. Startup must + reconstruct the latest compacted authority event and all applicable target + watermarks before any write; fresh accepted handoff may bootstrap exactly at + terminal W, and every normal/restart path resumes at W+1. Authority updates + race under the same fence held through transaction ACK. Missing, partial, + stale or conflicting recovery state fails closed. Tests must cover valid, + filtered, duplicate and quarantine decisions, crash/restart, compacted replay, + active authority change and rollback. This slice cannot change port 8100, + production routes, topics, consumers or authority. +- `2026-08-20 C.0 SDK PYTHON 3.10 BLOCKER PASS`: Trading System consumer + acceptance imported the prior immutable SDK wheel under its declared minimum + Python 3.10 runtime and found `enum.StrEnum` was Python 3.11-only. The fix is + owned by `qdl_sdk.models`, not patched in the consumer: Python 3.11+ uses the + standard enum and Python 3.10 uses an equivalent `str, Enum` compatibility + type. Added a dedicated CI job that builds and imports the standalone wheel + on Python 3.10 outside the source tree. Two release builds were byte-identical + at SHA-256 `3ea8f7e8b58f6c5ea1b2aa66ee94157f949d4cf6a71d708cb7508ed3b0abc600`; + an actual Python 3.10 wheel import passed and 17/17 SDK release/stream tests + passed on the Data Layer Python 3.12 runtime. Trading System updated its + vendor manifest/lock to that exact digest. V1 runtime, providers, authority, + Redis/PostgreSQL and consumer routes were unchanged. +- `2026-08-20 C.0 SDK STREAM PROJECTION PASS`: added one SDK-owned + canonical protobuf-to-typed-view decoder, so Trading System and alpha + consumers do not copy schema logic. It covers TRADE, QUOTE, BAR, book + snapshot/delta, funding, open interest, mark/index and ticker payloads with + exact coefficient/scale decimals, enums and optional bytes. The signed query + handoff remains the policy/catalog template; instrument/source transitions, + lower authority revision, stale execution data, open gap, incomplete + contract metadata and non-final execution bars fail closed. Freshness, + quality and execution eligibility are recomputed per event and the signed + cursor/watermark is preserved. All-feed projection, source/revision, gap and + stale tests plus existing SDK release/stream tests passed 20/20; isolated + lint and `git diff --check` passed. No runtime or provider was touched. +- `2026-08-20 C.0 LONG-RUNNING PRIMARY BRIDGE CODE PASS`: added a separate + multi-slice `qdl-production-core` binary and Phase 9.2 transactional bridge. + Authority is reconstructed per slice from the compacted control topic; raw + acquisition revision/lease is explicitly bound but separate from final + publication authority. Logical per-slice watermarks are independent of Kafka + partition offsets. Every raw decision commits its source offset, zero or more + canonical/quarantine records, progress for each permitted target and compacted + target checkpoints in one Kafka transaction. Filtered, duplicate and + quarantine decisions still advance projection progress without fabricating + market data. Expanded provider rows are hashed as one ordered checkpoint + payload set. Restart requires complete current-owner checkpoints, except the + first accepted W handoff may bootstrap exactly at terminal W; partial recovery + fails closed. Authority watcher updates share the transaction fence, so + BLOCKED/rollback cannot race a durable output ACK. Added deterministic + production-core configs generated from the approved provider metadata catalog, + plus immutable image packaging. Production catalog/runtime and outbox tests + passed 7/7; the complete Rust workspace passed 70/70 with strict Clippy and + formatting. No broker integration, image deployment, provider call, V1 route, + port 8100, production topic/database or consumer was mutated. RF3 transaction, + restart and rollback evidence remains mandatory in isolated C.1 before this + code can be called runtime-certified. - `2026-08-20 RELEASE/CUTOVER PREPARATION RECORDED`: corrected the malformed `RUNTIME UNCHANGED` journal line and added the production cutover boundary plus `docs/runbooks/v2-production-rust-authority-cutover.md`. Read-only runtime inspection proved V1 `0.1.0` still owns port `8100` with 40 V1 paths and zero V2 paths, no V2 containers are running, and the current stable - binaries/config are intentionally shadow-only. The branch is 81 commits ahead - of `dev`; it must merge through CI before a new authority feature branch is - created. + binaries/config are intentionally shadow-only. The branch is more than 80 + commits ahead of `dev`; it must merge through CI before a new authority + feature branch is created. - Documentation whitespace and secret scans passed; stable compose rendered successfully with isolated dummy values and no container start. Host preflight observed 11 GiB available RAM, 108 GiB free disk and eight CPUs. @@ -5951,6 +6026,85 @@ authority wiring cannot be deferred as operational debt. C.0 remains `IN_PROGRESS` until the current PR is CI-green and merged to `dev`; production authority wiring starts only on the new branch named in this phase. Preparation commit: `130da39`. +- `2026-08-20 CROSS-REPOSITORY V2 CONSUMER AUDIT RECORDED`: remote `dev` + merged the certified V2 branch at `468c951`; authority work continues on + `feat/v2-production-authority-cutover` from that merge, with the later + fast-track plan cherry-picked as `9e35b34` and `df94a51`. Trading System + currently has a V1 REST/Redis bridge and its `alpha_sdk` is primarily an + execution client; execution-alpha warmup/stream calls live in the shared + `alpha_runtime.orchestration.DataLayerGateway`. Therefore V2 is introduced + as one versioned `qdl_sdk` artifact used by both consumers. No strategy file, + signal rule or order endpoint is migrated for this data-plane change. + The audit also found the stable catalog is certification-bounded to BTC/VN + examples, so deterministic production symbol/catalog generation is a + mandatory C.0 gate before alpha consumers can be called V2-ready. +- `2026-08-20 C.0 SDK ARTIFACT/IDENTITY SLICE PASS`: moved every public V2 + response model into `qdl_sdk.models` and made `qdl.api_v2.models` reuse and + re-export that exact implementation. The SDK no longer imports Data Layer + service internals. Added bounded typed instrument catalog resolution by + venue/market/product/native symbol, including pagination-cycle, missing and + ambiguous-identity fail-closed behavior; consumers no longer need hardcoded + UUIDs. Added a deterministic standalone `qdl_sdk==2.0.0` wheel builder, + SHA-256 release manifest, generated-contract digest and CycloneDX SBOM. + Repeated builds produced an identical wheel digest and a network-off install + smoke imported exclusively from the installed wheel. Compile plus V1 golden, + API/SDK/stream/security/multi-venue tests passed 47/47. V1 runtime, provider sockets, + authority, Redis, Kafka, consumer routes and production data were untouched. + Production demand/catalog generation and long-running Rust authority wiring + remain the next C.0 slices; this slice alone does not authorize cutover. +- `2026-08-20 C.0 PRODUCTION CATALOG SLICE PASS`: added a strict + `qdl.v2.production-demand.v1` manifest and deterministic source/acquisition + catalog generator. It composes the existing authoritative Binance + `exchangeInfo` and OKX V5 `/public/instruments` parsers, preserves exact + price tick/quantity step/contract multiplier, derives stable UUIDv5 identity + from the approved canonical instrument ID, de-duplicates consumer demand and + fails closed on conflicting policies, missing/inactive metadata, ambiguous + identity or uncertified feeds/intervals. Binance canonical identity now uses + provider base/quote metadata (`ETH-USDT`) and includes the explicit contract + code for dated futures rather than treating native `ETHUSDT` as canonical. + Current production BAR acquisition is deliberately bounded to certified 1m; + higher alpha intervals must be resampled from final 1m bars or remain on + explicit V1 capability fallback until independently certified. Generated + source/acquisition YAML is reloaded through the runtime validators before it + is accepted, and provenance records metadata-capture hashes with + `fabricated_metadata=false`. Compile plus production catalog, identity, + Binance adapter and multi-venue contract tests passed 27/27. No real-provider + call or runtime/authority/consumer mutation occurred. +- `2026-08-20 C.0 TRANSACTIONAL AUTHORITY OUTBOX SLICE PASS`: added PostgreSQL + migration `0009_production_authority_outbox.sql`, which writes one immutable + authority-control outbox row in the same transaction as every Phase 9 CAS + transition. Bounded claim/ACK/retry operations bind worker ownership, recover + stale claims and never mutate event identity or payload. Added the Python + outbox dispatcher and idempotent Kafka publisher for the compacted authority + topic, plus a canonical `qdl.authority-control-event.v1` serializer that + validates exact Phase 9.2 checkpoint/handoff digests before exposing a + writable authority record. Rust now decodes that Python fixture, rejects + altered identity/conflicting duplicate/stale transition, remains fenced after + restart until every target watermark is restored, accepts only exact W/W+1 + handoff, and supports a newer-revision rollback to Python. Disposable + PostgreSQL migration smoke proved four ordered revisions, payload immutability, + bounded claim/ACK and scoped cleanup; no production database was touched. + Python authority/outbox/migration regressions passed 38/38. The complete Rust workspace + passed 66 tests, `cargo fmt --check` and strict `clippy -D warnings`. The + long-running transactional Rust consume-transform-produce bridge, independent + durable target-watermark restoration and operator CLI are still required C.0 + work; this slice does not authorize runtime authority or consumer cutover. +- `2026-08-20 OPERATOR CUTOVER SIMPLIFICATION RECORDED`: the operator reports + all alpha consumers are stopped and Trading System is the sole active + consumer. Phase C therefore removes staged alpha/monitoring migrations and + uses one bounded Trading System parity-and-route switch followed by a + preapproved Binance/OKX maintenance window. This reduces operations, not + correctness: persistent authority CAS/outbox, sink fencing, W/W+1 handoff, + durable audit and per-slice rollback remain mandatory. V1 stays hot on port + `8100`; DNSE stays V1-only. Fast-track planning commit: `e8167d4`. + +- `2026-08-20 C.0 SDK ALPHA STREAM POLICY CLOSURE STARTED`: downstream shared-runtime tests exposed a contract asymmetry: query validation enforces typed stale/gap policies for every consumer grade, while the stream projector currently blocks stale/gapped events only for `EXECUTION`. The source-owned SDK will enforce `stale_policy` and `gap_policy` identically for `ALPHA` and `EXECUTION`, retain the additional execution-eligibility gate for `EXECUTION`, and add explicit ALPHA stale/gap regression tests. A new deterministic wheel supersedes prior candidate digests only after Python 3.10 import, SDK release/stream tests, lint and byte-identical build pass. Consumers must update to that one digest; no downstream copy of projection logic is permitted. V1/runtime/provider/authority routes remain unchanged. + +- `2026-08-20 C.0 SDK ALPHA STREAM POLICY CLOSURE PASS`: the stream projector now applies typed `gap_policy` and `stale_policy` to ALPHA and EXECUTION consumers consistently; execution grade retains its additional authority/eligibility check. Added explicit ALPHA gap/stale regressions. SDK source projection/release/stream tests passed 21/21 on Python 3.12; the built wheel imported and passed 4/4 projection tests on the released Python 3.10 consumer runtime. Two independent builds were byte-identical at SHA-256 `3e1ce5e43d55ac4c04baf5b69354513f32090bd2e7060f1f4e659323470a27d0`; isolated Ruff lint and `git diff --check` passed. The repository legacy Poetry version syntax prevents modern Ruff from loading the root config and its existing files are not Ruff-format clean, so no unrelated format churn was introduced. No runtime/provider/authority route was touched. + +- `2026-08-20 C.0 FROZEN OPENAPI COMPATIBILITY BLOCKER STARTED`: the pre-build full suite passed 526 tests with 6 skips but failed both frozen OpenAPI assertions. Inspection found the earlier model-ownership move accidentally renamed response component `FeedType` to `Feed` and removed `BarLifecycle.UNSPECIFIED` from the published enum. Restore the frozen wire schema without reverting SDK ownership: declare `FeedType` as the concrete enum, export `Feed` as its SDK alias, retain `UNSPECIFIED` in OpenAPI and continue rejecting it in model validation. The unchanged frozen snapshot must pass; regenerating it to hide this break is forbidden. The SDK wheel and both downstream consumer pins must be rebuilt once more after the complete Python/Rust gates pass. + +- `2026-08-20 C.0 FROZEN OPENAPI COMPATIBILITY BLOCKER PASS`: the Data Layer service and SDK now share the exact public `qdl.query.FeedType`/`BarLifecycle` enum identity when that contract package is present; the standalone wheel supplies equivalent fallback enums and exports concise `Feed` as an alias. `UNSPECIFIED` remains published for wire compatibility and is rejected at the typed requirement/model boundary. The frozen OpenAPI snapshot was not modified and now matches exactly: 10 paths and 42 schemas. Targeted OpenAPI/SDK tests passed 24/24; the full Python suite passed 535/535 with 6 skips. The Rust workspace passed 70/70 plus `cargo fmt --check` and strict Clippy. Two release builds were byte-identical at final wheel SHA-256 `10f894604c543fc07499247b5c6fc38910b8e704bffe29683f100c519d6caa49`; the installed wheel passed 5/5 stream-projection tests on Python 3.10 and exposed `Feed.__name__ == FeedType`. Isolated Ruff and `git diff --check` passed. This supersedes all earlier candidate wheel digests; consumers must pin only this digest. V1/runtime/provider/authority routes remain unchanged. #### C.1 Isolated Stable V2 Deployment @@ -5965,46 +6119,377 @@ quarantine, bounded lag/resources, broker and process restart recovery, exact cursor continuation, V1 health unchanged and exact disposable cleanup on failure. -#### C.2 Controlled Consumer Canary - -Migrate in order: monitoring, one paper alpha, Trading System paper adapter, -then remaining approved paper consumers. Each manifest performs -warmup -> signed cursor -> replay -> live and has an exact V1 rollback route. -No sandbox/live order consumer is included. A stale, gapped, non-authoritative -or session-invalid read blocks execution. - -#### C.3 Exact-Slice Rust Authority Promotion - -Promotion is one slice at a time, initially one Binance or OKX TRADE slice. -The approval packet must name image IDs, slice/binding, old/new owner, -authority revision, lease/plan epoch, terminal watermark `W`, topics/groups, -ports, volumes, credentials by secret reference, affected consumers, hold -duration and rollback command. - -The only allowed sequence is: +**C.1 implementation journal:** + +- `2026-08-20 C.1 IMMUTABLE ISOLATED DEPLOYMENT STARTED`: build the Python + edge and Rust core from the same tested source revision + `f93b7f0e4d3381a01da48dafbb8263436b0315e1`. The immutable candidates are + `qdl-v2-python:2.0.0-f93b7f0e4d33` with image ID + `sha256:7a1b11097e4e85a51630068b2a619e34ce654b532d5ea750c6b8678882f2cc86` + and `qdl-v2-rust:2.0.0-f93b7f0e4d33` with image ID + `sha256:fc50dbf0a83323966ed6d8e76ae468a98edad6e34c7f0392a07924e94018348f`; + both carry the exact source revision label and run as UID/GID `10001` in + the stable compose. The bounded certification slice contains authentic + Binance USDM/Spot and OKX Swap/Spot BTC-USDT feeds only. It uses project + `qdl_v2_stable_candidate`, dedicated RF3/minISR2 Kafka, ephemeral Redis, + private state/TLS, loopback ports `18201/18202/18210/18211/18220/18221` + and `RUST_SHADOW`; `stable-vn` is excluded. Port `8100`, V1 containers, + V1 volumes and current consumer routes are immutable boundaries. Rollback + before consumer migration is project-scoped `docker compose down` without + `-v`; no authority CAS or Trading System route mutation is authorized by + this journal entry. + +- `2026-08-20 C.1 ISOLATED SHADOW CERTIFICATION PASS`: generated a private + `0600` environment bundle with short-lived test TLS material and + `cutover_authorized=false`, then started only project + `qdl_v2_stable_candidate`. Kafka created three six-partition topics at RF3, + minISR2 and full ISR. Authentic Binance USDM/Spot and OKX Swap/Spot trades + plus 500 closed 1m bars per binding entered the raw topic and passed through + the Rust canonical core. Query replicas returned exact BAR payload parity; + Binance/OKX five-row warmups were `FULL`, `FINAL`, authoritative, + complete and gap-free. The reusable + `scripts/phasec1_isolated_consumer_acceptance.py` used the released SDK to + prove signed snapshot handoff, `REPLAYING -> LIVE`, ACK, fsynced cursor + persistence, client recreation through the other query replica and exact + `N+1` resume for both venues. The quarantine topic remained zero on all + six partitions. +- Failure drills stayed project-scoped. Restarting one Rust worker made the + execution-grade query fail closed on freshness while the group rebalanced; + it recovered to lag 32 and the SDK acceptance passed again. Stopping the + current stream lease holder promoted the peer in five seconds; both replicas + remained live, exactly one was ready, and cursor/reconnect acceptance passed. + Restarting one Kafka broker restored full ISR with core lag 51 and projector + lag 57; post-restart SDK acceptance passed and quarantine remained zero. + Missing auth returned 401, mismatched consumer returned 403, and wrong + purpose on a market-data endpoint returned 403. At the bounded resource + sample Rust workers used 27-44 MiB each, Python roles 40-72 MiB, Redis 3 MiB + and Kafka brokers 427-433 MiB each. V1 remained healthy with 40 paths, image + `sha256:8f2a5a3f1ff97762feb1531c3787e714dfda60b0b64df5b7359b9e5f6740c980`, + original start time and restart count zero. +- `C.1 conclusion: PASS / SHADOW-CERTIFIED`. This is not production authority. + C.2 inspection exposed mandatory blockers before consumer cutover: stable + gRPC currently binds insecurely and is reachable only through loopback; + Trading System runtime IDs do not match the registered stable consumer ID; + and V2 routing is provider-wide although this certified catalog is a bounded + BTC slice. In addition, the stable runtime accepts only `RUST_SHADOW`; + production promotion must consume the durable authority CAS/outbox and fence + writers instead of changing an environment label. These are in-scope C.2/C.3 + correctness gates and must be fixed, not deferred as operational debt. + +#### C.2 Single-Consumer Trading System Cutover + +**C.2 implementation journal:** + +- `2026-08-20 C.2 CONSUMER INGRESS CLOSURE STARTED`: close the real + integration gaps found by C.1 before any consumer restart. Stable REST and + gRPC data-plane ingress must use server-authenticated TLS plus client + workload certificates; JWT issuer/audience/manifest authorization remains + mandatory at the application layer. Projector-to-stream ingest uses the same + authenticated transport. The source-owned `qdl_sdk` adds CA/client + certificate configuration once; Trading System and execution-alpha consume + it without custom transports. The Trading System manifest gains only the + final 1m BAR permissions its market-cache bridge actually uses. +- Trading System must route by a strict versioned slice manifest + `venue + market + product + native symbol + feed + interval`. In + `V2_PRIMARY`, only approved slices leave V1; all unmatched Binance symbols + remain on V1 and are audited as compatibility routes. OKX has no equivalent + V1 realtime endpoint and therefore fails closed on V2 outage rather than + being relabelled as a cross-venue fallback. One configured external consumer + identity must match the registered Data Layer manifest; unrelated Trading + System services remain V1 until separately manifested and credentialed. +- `2026-08-20 C.2 WORKLOAD TLS/SDK SLICE PASS`: added one source-owned + `WorkloadTlsConfig` for REST and gRPC client certificates, bounded + multi-target gRPC failover, mandatory stable query/stream server mTLS and + projector-to-stream HTTPS mTLS while retaining JWT/manifest authorization + and HMAC ingest signing. Stable bundles now carry separate query, stream, + projector and Trading System identities outside Git; the Trading System + manifest gained only Binance USDM and OKX SWAP final 1m BAR requirements. + Focused transport/security/stable-ingest tests passed 6/6; Python compile and + YAML parse gates passed. No V1 container, consumer route, authority state, + production data or order path was mutated. Real certificate handshake, + immutable rebuild and rotation/reconnect remain C.2 gates before closure. +- `2026-08-20 C.2 ISOLATED RESTART RECOVERY GATE FOUND`: the first secure + isolated restart proved the query mTLS positive handshake and rejected a + client without a workload certificate, while V1 remained HTTP 200 and was + not restarted. The projector correctly failed closed because retained + SQLite projection state was paired with a newly empty ephemeral Redis cache. + This is the designed B16 generation fence, not permission to bind an empty + cache over retained state. Before acceptance continues, update the existing + exact-scope B17 cache-unit rebuild command to use the new mTLS health probes, + then rebuild only the isolated Redis plus three SQLite cache files from the + Kafka canonical log. Kafka/provider data, V1, production Redis and authority + remain untouched. Acceptance requires bounded six-partition lag, a bound + cache identity, both secure query replicas ready and a fresh signed SDK + handoff after rebuild. +- `2026-08-20 C.2 RELEASE/REGRESSION SLICE PASS; RUNTIME REPLAY CONTINUES`: + the final source-owned `qdl_sdk==2.0.0` artifact was built twice + byte-identically at SHA-256 + `5891c0b99b29fd30ce008f6987a4ff9c9d4896259e415f72e5f9210460669951` + with Python >=3.10 and `PyJWT[crypto]` declared. The complete Python suite + passed 540/540 with six explicit environment skips; targeted secure + bundle/recovery/transport tests passed 14/14. Rust passed 70/70 plus + `cargo fmt --check` and strict Clippy. The real mTLS query handshake + returned 200 and a request without a client certificate failed the TLS + handshake; V1 remained HTTP 200. A non-destructive cache generation and + isolated projector group are replaying authentic retained Kafka records + because the exact destructive B17 rebuild was not approved. Therefore C.2 is + not yet closed and no consumer route or authority was promoted. +- `2026-08-20 C.2 ISOLATED CONSUMER ACCEPTANCE PASS`: the + non-destructive cache generation completed against authentic retained Kafka + bytes. At the final bounded snapshot the six-partition projector lag was + `144`, projector readiness was `READY`, both mTLS query replicas returned + 200 and no new projector error appeared in the last two minutes. The + source-owned `qdl_sdk==2.0.0` acceptance passed for Binance and OKX: + each venue returned five final 1m BARs with `FULL` coverage, identical query + replica fingerprints and authoritative provider identity; cursor resume was + contiguous `982448 -> 982449` for Binance and + `339821 -> 339822` for OKX. A request without a client certificate remained + rejected. Earlier ACL/stream errors were bounded startup/replay history and + were not active at acceptance. V1 stayed HTTP 200, no Trading System route, + order path or authority changed, and all alpha processes remained stopped. +- C.2 gates are mTLS positive/negative tests, certificate rotation/reconnect, + exact BAR/trade SDK projection, bounded route-manifest parser tests, V1 + unmatched-symbol compatibility, authenticated real Binance/OKX adapter + acceptance, no order submission and unchanged V1/runtime state. Authority + stays `RUST_SHADOW` until the separate C.3 CAS/outbox/fence packet passes. + +The operator confirms all alpha consumers are currently stopped and Trading +System is the only active Data Layer consumer. Do not create artificial alpha or +monitoring migration stages. Built-in V2 health/lag/authority telemetry remains +mandatory, but it is not a separate cutover consumer. + +Run one bounded Trading System dual-read parity window for Binance and OKX: +V1 remains the decision source while the same requested instruments, timestamps, +decimals, units, final BAR lifecycle and freshness are compared against V2. +After zero correctness mismatch and healthy cursor/replay evidence, switch the +Trading System market-data adapter to V2 in one controlled restart. Configure a +venue-aware rollback route: Binance/OKX primary V2 with explicit V1 fallback; +DNSE remains V1-only. Never splice providers silently--every route transition +records source, reason, watermark and operator/audit identity. + +A stale, gapped, non-authoritative, wrong-session or unit-mismatched V2 read +fails closed. Fallback to V1 is allowed only when V1 passes the same +freshness/session/contract checks and the source-switch audit is durable. + +#### C.3 Fast-Track Rust Authority Promotion + +**C.3 implementation journal:** + +- `2026-08-20 C.3 DURABLE AUTHORITY RUNTIME WIRING STARTED`: reuse, do not + fork, the accepted Phase 9.2 domain primitives: migrations + `0006/0007/0009`, transactional authority outbox, compacted control event, + Rust `qdl-production-core`, per-target sink fence and W/W+1 handoff. Add + only the missing deployable topology around them: a dedicated isolated + PostgreSQL authority-control database, one least-privilege authority + dispatcher identity, compacted authority/target-checkpoint topics, immutable + production-core configs and three bounded Rust workers behind an explicit + Compose profile. The existing shadow core remains the writer until a + separately approved operator packet fences it. +- Add one source-owned operator command that is plan-only by default and accepts + a versioned immutable packet. It must validate exact slices, candidate/image/ + contract/partition digests, expected state/revision/owner/lease, terminal + checkpoint, zero mismatch/gap canary evidence, hold expiry, Trading System + route and executable rollback before any SQL CAS. Apply requires an exact + confirmation token; transitions execute one slice at a time and stop on the + first failure. No environment-label-only promotion is valid. +- Gates are migration idempotency, DB transaction/outbox atomicity, broker ACK + retry/crash recovery, compacted control rebuild, Rust startup with missing/ + stale/partial authority failure, target fencing, real canary parity, + W/W+1 primary handoff, V1 fallback/return, bounded resources and full + Python/Rust/V1 contract regression. Code/test wiring cannot mutate production + authority; runtime promotion still requires the exact packet and explicit + operator approval named in this section. + +- `2026-08-20 C.3 TOPOLOGY/OPERATOR SLICE PASS`: added a dedicated + non-public PostgreSQL authority database, migration-owned least-privilege + dispatcher role, atomic dispatcher heartbeat, compacted authority/checkpoint + topics, per-principal Kafka ACLs and three bounded + `qdl-production-core` workers behind explicit control/primary profiles. + Stable bundle generation now emits production-core configs and separate + dispatcher/admin credentials outside Git. Added a strict, expiring, + digest-derived plan/apply packet command that validates real-data evidence, + exact route rollback, slice state/revision/owner/lease/digests and uses the + accepted SQL CAS functions one slice per transaction; it is plan-only unless + `--apply --confirm APPLY_C3_` matches the immutable packet. + Focused topology/outbox/packet tests passed 23/23. A network-none/tmpfs + PostgreSQL bootstrap proved all migrations, three SECURITY DEFINER functions, + direct-table UPDATE denial, dispatcher claim permission and migration + idempotency, then auto-removed the test container. Runbook: + [V2 production and Rust authority cutover](docs/runbooks/v2-production-rust-authority-cutover.md). + This code evidence does not authorize a production CAS or consumer restart. + +- `2026-08-20 C.3 FULL REGRESSION/BUNDLE GATE PASS`: full Python + regression passed 546/546 with six explicit environment skips; changed-file + Ruff passed; Rust passed 70/70, `cargo fmt --check` and strict Clippy. + TLS generation emitted the dedicated dispatcher identity, candidate bundle + generation passed with 12 runtime files and no secret values in the public + manifest, and Compose config parsed with both authority profiles. The first + Rust test attempt exhausted a 1 GiB disposable tmpfs during link; rerun with + debug symbols disabled passed in 1.5 GiB and left no build target on disk. + Repository-wide Ruff still reports 63 pre-existing findings outside this + slice; changed files have zero finding. No authority DB/volume, production + CAS, Trading System route, V1 service or provider ownership was mutated. + +- `2026-08-20 C.3 IMMUTABLE BUILD HYGIENE STARTED`: release preflight + found the Python builder/runtime base referenced a mutable tag while Rust + bases were digest-pinned. Pin both Python stages to the locally resolved + official image digest before building the commit-SHA release; verify both + stages use the same digest, rebuild, inspect OCI revision/version/non-root + identity, rerun image-level smoke and retain V1 unchanged. The source/test + slice pins both stages to the same official digest; focused contract tests + passed 14/14, changed-file Ruff and diff checks passed. + +- `2026-08-20 C.3 FINAL IMMUTABLE ARTIFACT GATE PASS`: commit + `5823d642027b7446aa72160aa2ec53c28fdd88f1` produced Python image + `sha256:1758b35646293eca717d269681b867fc485db896a70889bab53df47d8d87345f` + and Rust image + `sha256:1eda689c30484157092cc276a1487d36174acd1a97a353ed792642a6d5512211`. + Both images expose OCI version `2.0.0` and the exact full revision; Python + runs as `qdl:qdl`, Rust as UID/GID `10001:10001`. A network-none Python + image smoke imported the API and source-owned `qdl_sdk==2.0.0`; the Rust + image contains `qdl-production-core`, which failed closed with its usage + error when started without a config. A fresh private bundle generated from + those exact image IDs reported 12 runtime files, `RUST_SHADOW`, + `cutover_authorized=false` and no secret values in its public manifest. + Its manifest SHA-256 is + `a9d2835e86c0f6b2be7f90f7671d2f3d8dc9462da324703658991da774b4b1cb`; + Compose rendered successfully with both `stable-authority` and + `stable-authority-primary` profiles. No container, authority row, consumer + route, V1 service, provider ownership or persistent volume changed. The + next permitted operation is topology/packet preflight; a production CAS and + Trading System restart still require the exact packet approval below. + + +- `2026-08-20 C.3 PROMOTION-SCOPE BLOCKER FOUND; ARTIFACT REVOKED`: packet + preflight inspected the generated production-core configs and found all four + DNSE bindings present alongside the twelve approved Binance/OKX bindings. + This violates the explicit initial-cutover boundary that DNSE remains V1-only + and would make a production worker require DNSE authority/checkpoints even + when the `stable-vn` profile is disabled. The two image IDs above are valid + build evidence but are revoked as cutover artifacts. Fix the generator with + one strict, versioned, explicit authority-promotion binding manifest; filter + both canonical bindings and runtime slices from that manifest, reject empty, + duplicate or unknown bindings, and record its digest in the bundle. Add a + regression proving initial authority contains exactly twelve Binance/OKX + bindings and zero HNX/HOSE/DNSE binding. Re-run focused/full gates and rebuild + one new immutable image pair before topology deployment. V1 and the running + isolated shadow stack remain unchanged while this source-only repair runs. + + +- `2026-08-20 C.3 PROMOTION-SCOPE REPAIR PASS`: added strict manifest + `qdl.v2.authority-promotion-scope.v1`; production-core generation now filters + both canonical bindings and authority slices from its explicit binding IDs, + rejects empty/duplicate/unknown scope and records revision/digest/count in + the public bundle. The initial manifest selects exactly twelve Binance/OKX + trade/quote/final-1m-bar bindings and no DNSE/HNX/HOSE binding. Targeted + contract/bundle/authority tests passed 22/22; full Python passed 543 with six + environment skips; full Rust passed 70/70 with fmt and strict Clippy; isolated + changed-file Ruff passed. All three generated production workers contain + 12 slices, venues `BINANCE,OKX`, zero DNSE subscriptions and common scope + digest `06178202d7ec592c19c41a36c919a13a74971c3e39ed8e67ce9b5de3a978fcd2`. + Compose authority profiles render successfully. Tests used network-none + source mounts and disposable tmpfs/tooling; V1, the running isolated shadow, + Trading System routes, authority state and persistent volumes were unchanged. + + +- `2026-08-20 C.3 REBUILT RELEASE PAIR PASS`: tested repair commit + `3d3af1c530e1dd52b402294e0bb677eb334a15a2` produced Python image + `sha256:e61c7cb1372071daeb3f9753e616b073b514998845abc61ab168b2cb63617e90` + and Rust image + `sha256:676de79940ed83cc45a8c1490055c8fa69ddc5bcb032af4ab6a4851d25e921b6`. + Both carry exact revision/version labels and retain non-root users. Image-level + network-none smoke imported `qdl_sdk==2.0.0`; `qdl-production-core` remained + fail-closed without config. The fresh bundle binds those exact IDs, reports + `RUST_SHADOW`, `cutover_authorized=false`, scope digest + `06178202d7ec592c19c41a36c919a13a74971c3e39ed8e67ce9b5de3a978fcd2` + and twelve approved bindings; authority Compose profiles render cleanly. + This pair supersedes the revoked `5823d642` pair. No running container or + authority/consumer route changed. Merge/immutable deployment and the exact + operator packet remain the only gates before bounded runtime promotion. + Exact cleanup removed the two revoked `5823d642` image tags and three + disposable test/revoked-bundle paths only; no broad prune, active image, + final release bundle, V1 rollback artifact or volume was removed. + + +- `2026-08-20 C.2 CONSUMER-NETWORK BLOCKER FOUND`: final deployment + preflight compared Data Layer and Trading System Compose topology. Stable V2 + query/stream roles only join project-private networks and expose loopback host + ports, while Trading System resolves `qdl-v2-query` and + `qdl-v2-stream-a/b` from external `executor_network`; the container cannot + reach host loopback, so a real consumer cutover would fail despite valid SDK + and mTLS tests. Add one explicit generated external-consumer-network setting, + attach only the two query and two stream ingress roles with the frozen DNS + aliases, and keep Kafka/Redis/projector/Rust core off that network. Require + Compose contract tests for aliases/isolation plus existing full regressions. + Rebuild the same-SHA release pair after this bounded topology repair. No + running network/container is changed by the source fix. + + +- `2026-08-20 C.2 CONSUMER-NETWORK REPAIR PASS`: stable bundle generation now + requires a validated external consumer network and records it in private env + plus the non-secret manifest. Only query replicas join it as + `qdl-v2-query`; only active/passive stream roles join as + `qdl-v2-stream-a/b`. Kafka, Redis, projector, Rust shadow/primary cores and + ingestors remain absent from that network. Generated Compose validated + against existing external `executor_network`; focused tests passed 19/19, + full Python passed 543 with six environment skips, changed-file Ruff passed, + and the canonical cutover runbook now requires the network explicitly. + No container was attached, recreated or restarted; port 8100 and Trading + System remained unchanged. Commit and one final same-SHA image rebuild are + required before PR/cutover. + + +- `2026-08-20 C.2/C.3 FINAL RELEASE ARTIFACT PASS`: topology commit + `be35aa7389a37b31c21cc2689c25873dcfc7e73d` produced Python image + `sha256:89e359ecc731d68db7a1814885023e1ff9f0aea793e668b6298109eb463ff91c` + and Rust image + `sha256:ab57e015da2fb96ef6e4b2180676e0a41b2cc45b64080e820d6a8f29cdab180a`. + Machine-read OCI labels exactly match the Git SHA and version `2.0.0`; users + remain `qdl:qdl` and `10001:10001`. The final bundle manifest digest is + `6a3edff0fdaa690b1fc1237f5678bf8463355bbdc51afb59a018f3e629840425`, + binds `executor_network`, twelve Binance/OKX promotion bindings, zero DNSE, + `RUST_SHADOW` and `cutover_authorized=false`; complete authority Compose + rendering passes. One mistyped preflight revision image was detected by label + comparison and is explicitly not a release artifact or deployed runtime. + This is the only V2 pair eligible for the merge/cutover packet. + Scoped cleanup then removed the superseded `3d3af1c` pair, the mistyped + Python tag and disposable netfix/test bundles. No broad prune, active + candidate image, final bundle, V1 image or Docker volume was removed. + +Promote all approved Binance and OKX feed slices in one maintenance window, but +execute the CAS internally one slice at a time so a failure is isolated. One +operator packet may list the complete slice set, image IDs, old/new owners, +authority/lease/plan revisions, terminal watermarks, topics/groups, ports, +volumes, secret references, Trading System route and rollback command. + +Each slice still follows: `PYTHON_PRIMARY -> RUST_SHADOW -> RUST_CANARY -> RUST_PRIMARY`. -Fence the old writer, persist its terminal checkpoint, accept the handoff, -execute the CAS/outbox transition, reconstruct every target through `W`, and -publish first as Rust at `W+1`. Any ambiguity, missing ACK, parity mismatch, -lag/gap, stale CAS or consumer failure enters `BLOCKED` and rolls back under a -newer revision; never restart V1 as an uncoordinated writer. - -#### C.4 Hold, Expand And Release - -Hold the first primary slice for the approved observation window with zero -authority ambiguity or unexplained market-data mismatch. Expand independently -by venue/feed manifest; no slice inherits certification. Only after all -registered consumers use V2 may the operator approve routing the stable public -endpoint and opening a V1 sunset window. DNSE remains disabled until its +The canary is bounded by accepted real events and continuity evidence rather +than a long calendar wait. Fence the old writer at `W`, persist its terminal +checkpoint, accept the handoff, execute CAS/outbox, reconstruct every target +through `W`, and publish first as Rust at `W+1`. When one slice passes, the +same preapproved window proceeds to the next. Any ambiguity, missing ACK, +parity mismatch, lag/gap, stale CAS or Trading System failure enters `BLOCKED` +for that slice and restores V1 under a newer revision; unrelated promoted slices +remain governed independently. + +#### C.4 Close With V1 Hot Fallback + +After all approved Binance/OKX slices are `RUST_PRIMARY`, Trading System reads +V2 as its normal source and V1 stays running on port `8100` as the tested hot +fallback. There is no alpha-by-alpha migration while those alphas remain down +and no V1 sunset is part of this cutover. Publish the V2 release only after the +Trading System cycle, Rust authority audit, cursor/replay continuity and an +exercised V1 fallback/return-to-V2 drill pass. DNSE remains V1-only until its separate provider gate passes. **Decision boundary:** C.0 code/release preparation and C.1 isolated deployment -are non-production-authority work. C.2 changes only explicitly named paper -consumer routes. C.3 and C.4 require a separate operator approval containing -the exact packet above. No command in this plan implicitly authorizes a restart, -authority mutation, consumer cutover, volume deletion or V1 shutdown. +are non-production-authority work. C.2 changes only the Trading System +market-data route. C.3 requires one explicit operator packet for the approved +Binance/OKX slice set. No command implicitly authorizes deleting volumes, +stopping V1 or promoting DNSE. ### Rollback diff --git a/Dockerfile b/Dockerfile index dc6654c..ea152a3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.12-slim AS builder +FROM python:3.12-slim@sha256:2c941e860699f878900b0edc2403613c234d4b32eda3cc9fa7036991a2a63c4a AS builder ENV PYTHONUNBUFFERED=1 ENV PYTHONDONTWRITEBYTECODE=1 @@ -17,7 +17,7 @@ RUN python -m venv /opt/venv && \ poetry install --no-root --only main --no-ansi && \ /opt/venv/bin/python -m pip install --no-cache-dir --upgrade "setuptools>=78.1.1" -FROM python:3.12-slim AS runtime +FROM python:3.12-slim@sha256:2c941e860699f878900b0edc2403613c234d4b32eda3cc9fa7036991a2a63c4a AS runtime ARG QDL_UID=10001 ARG QDL_GID=10001 diff --git a/Dockerfile.phase8-rust b/Dockerfile.phase8-rust index 25b8bad..83e7fd6 100644 --- a/Dockerfile.phase8-rust +++ b/Dockerfile.phase8-rust @@ -14,6 +14,7 @@ COPY generated/rust ./generated/rust # Rust unit/parity tests compile against immutable provider and contract oracles. COPY contracts/golden ./contracts/golden COPY tests/fixtures/phase2 ./tests/fixtures/phase2 +COPY tests/fixtures/phase9 ./tests/fixtures/phase9 RUN cargo build --release --locked \ --bin qdl-kafka-smoke \ --bin qdl-authority-rehearsal \ @@ -22,6 +23,7 @@ RUN cargo build --release --locked \ --bin qdl-binance-shadow \ --bin qdl-native-raw-ingestor \ --bin qdl-realtime-core \ + --bin qdl-production-core \ --bin qdl-parity-replay \ --bin qdl-venue-core-certify @@ -44,6 +46,7 @@ COPY --from=builder /src/target/release/qdl-phase92-primary-rehearsal /usr/local COPY --from=builder /src/target/release/qdl-binance-shadow /usr/local/bin/qdl-binance-shadow COPY --from=builder /src/target/release/qdl-native-raw-ingestor /usr/local/bin/qdl-native-raw-ingestor COPY --from=builder /src/target/release/qdl-realtime-core /usr/local/bin/qdl-realtime-core +COPY --from=builder /src/target/release/qdl-production-core /usr/local/bin/qdl-production-core COPY --from=builder /src/target/release/qdl-parity-replay /usr/local/bin/qdl-parity-replay COPY --from=builder /src/target/release/qdl-venue-core-certify /usr/local/bin/qdl-venue-core-certify USER 10001:10001 diff --git a/app/entrypoints/query_v2_stable.py b/app/entrypoints/query_v2_stable.py index f20fb2e..b5dcf93 100644 --- a/app/entrypoints/query_v2_stable.py +++ b/app/entrypoints/query_v2_stable.py @@ -1,6 +1,9 @@ -"""Isolated Data Layer 2.0.0 stable query edge; V1 remains authoritative.""" +"""Stable Data Layer 2.0.0 query edge with mandatory workload mTLS.""" -from qdl.runtime.stable import create_stable_query_app +import asyncio +from qdl.runtime.stable import serve_stable_query -app = create_stable_query_app() + +if __name__ == "__main__": + asyncio.run(serve_stable_query()) diff --git a/config/v2/stable-authority-promotion-scope.yaml b/config/v2/stable-authority-promotion-scope.yaml new file mode 100644 index 0000000..0c90a42 --- /dev/null +++ b/config/v2/stable-authority-promotion-scope.yaml @@ -0,0 +1,15 @@ +schema: qdl.v2.authority-promotion-scope.v1 +revision: 1 +binding_ids: + - binance-spot-btcusdt-bar-1m + - binance-spot-btcusdt-quote + - binance-spot-btcusdt-trade + - binance-usdm-btcusdt-bar-1m + - binance-usdm-btcusdt-quote + - binance-usdm-btcusdt-trade + - okx-spot-btcusdt-bar-1m + - okx-spot-btcusdt-quote + - okx-spot-btcusdt-trade + - okx-swap-btcusdt-bar-1m + - okx-swap-btcusdt-quote + - okx-swap-btcusdt-trade diff --git a/consumers/stable/trading-system-paper.yaml b/consumers/stable/trading-system-paper.yaml index 31169b9..8d44e15 100644 --- a/consumers/stable/trading-system-paper.yaml +++ b/consumers/stable/trading-system-paper.yaml @@ -50,6 +50,19 @@ spec: gap_policy: BLOCK recovery: SNAPSHOT_AND_REPLAY bar_revision_policy: LATEST + - instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: BAR + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + interval: 1m + warmup_limit: 1000 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f feed: TRADE consumer_grade: EXECUTION @@ -74,6 +87,19 @@ spec: gap_policy: BLOCK recovery: SNAPSHOT_AND_REPLAY bar_revision_policy: LATEST + - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: BAR + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + interval: 1m + warmup_limit: 1000 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST - instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 feed: TRADE consumer_grade: EXECUTION diff --git a/docker-compose.v2-stable.yml b/docker-compose.v2-stable.yml index d80d614..a061bd3 100644 --- a/docker-compose.v2-stable.yml +++ b/docker-compose.v2-stable.yml @@ -82,7 +82,7 @@ x-stable-env: &stable-env QDL_DATA_JWT_KEYS_JSON: ${QDL_STABLE_JWT_KEYS_JSON:?set QDL_STABLE_JWT_KEYS_JSON} QDL_DATA_JWT_ISSUER: https://identity.qdl.stable.internal QDL_DATA_JWT_AUDIENCE: qdl-v2-stable - QDL_DATA_JWT_ALGORITHMS: HS256 + QDL_DATA_JWT_ALGORITHMS: RS256 QDL_DATA_JWT_MAX_LIFETIME_SECONDS: "900" x-python: &python @@ -94,7 +94,7 @@ x-python: &python cap_drop: [ALL] networks: [stable_internal] tmpfs: ["/tmp:rw,nosuid,nodev,noexec,size=64m"] - volumes: [stable_state:/var/lib/qdl-stable] + volumes: [stable_state:/var/lib/qdl-stable, stable_tls:/stable-certs:ro] environment: {<<: *stable-env} mem_limit: 512m cpus: 0.75 @@ -133,6 +133,33 @@ services: environment: {<<: *kafka-env, KAFKA_NODE_ID: 3, KAFKA_ADVERTISED_LISTENERS: "SSL://kafka3:9092", KAFKA_SSL_KEYSTORE_FILENAME: kafka3.keystore.p12} volumes: ["${QDL_STABLE_CERT_DIR:?set QDL_STABLE_CERT_DIR}:/etc/kafka/secrets:ro", kafka3_data:/var/lib/kafka/data] + stable_authority_db: + image: postgres@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685 + profiles: [stable-authority] + restart: "no" + security_opt: [no-new-privileges:true] + networks: [stable_internal] + environment: + POSTGRES_DB: qdl_authority + POSTGRES_USER: qdl_authority + POSTGRES_PASSWORD: ${QDL_STABLE_CONTROL_DB_PASSWORD:?set QDL_STABLE_CONTROL_DB_PASSWORD} + QDL_STABLE_DISPATCHER_DB_PASSWORD: ${QDL_STABLE_DISPATCHER_DB_PASSWORD:?set QDL_STABLE_DISPATCHER_DB_PASSWORD} + PGDATA: /var/lib/postgresql/data/pgdata + volumes: + - stable_authority_db:/var/lib/postgresql/data + - ./migrations/postgres:/docker-entrypoint-initdb.d:ro + tmpfs: + - /run/postgresql:rw,nosuid,nodev,noexec,size=8m + - /tmp:rw,nosuid,nodev,noexec,size=16m + healthcheck: + test: [CMD-SHELL, "pg_isready -U qdl_authority -d qdl_authority"] + interval: 3s + timeout: 3s + retries: 40 + start_period: 10s + mem_limit: 256m + cpus: 0.50 + stable_redis: image: redis@sha256:dfa18828cbc07b3ae6a95ec7343f6c214fdee2d836197b4be8e9904420762cd8 command: [redis-server, --appendonly, "no", --save, "", --maxmemory, 128mb, --maxmemory-policy, noeviction] @@ -177,10 +204,14 @@ services: - -c - >- install -d -o 10001 -g 10001 -m 0750 - /stable-certs/producer /stable-certs/core /stable-certs/projector && + /stable-certs/producer /stable-certs/core /stable-certs/projector + /stable-certs/query /stable-certs/stream /stable-certs/authority-dispatcher && cp /source/producer/* /stable-certs/producer/ && cp /source/core/* /stable-certs/core/ && cp /source/projector/* /stable-certs/projector/ && + cp /source/query/* /stable-certs/query/ && + cp /source/stream/* /stable-certs/stream/ && + cp /source/authority-dispatcher/* /stable-certs/authority-dispatcher/ && chown -R 10001:10001 /stable-certs && find /stable-certs -type f -exec chmod 0440 {} + restart: "no" @@ -193,12 +224,18 @@ services: - ${QDL_STABLE_PRODUCER_CERT_DIR:?set QDL_STABLE_PRODUCER_CERT_DIR}:/source/producer:ro - ${QDL_STABLE_CORE_CERT_DIR:?set QDL_STABLE_CORE_CERT_DIR}:/source/core:ro - ${QDL_STABLE_PROJECTOR_CERT_DIR:?set QDL_STABLE_PROJECTOR_CERT_DIR}:/source/projector:ro + - ${QDL_STABLE_QUERY_CERT_DIR:?set QDL_STABLE_QUERY_CERT_DIR}:/source/query:ro + - ${QDL_STABLE_STREAM_CERT_DIR:?set QDL_STABLE_STREAM_CERT_DIR}:/source/stream:ro + - ${QDL_STABLE_AUTHORITY_CERT_DIR:?set QDL_STABLE_AUTHORITY_CERT_DIR}:/source/authority-dispatcher:ro - stable_tls:/stable-certs query_v2_1: <<: *python - command: [uvicorn, app.entrypoints.query_v2_stable:app, --host, 0.0.0.0, --port, "8200", --no-access-log] - networks: [stable_internal, stable_ingress] + command: [python, -m, app.entrypoints.query_v2_stable] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-query]} ports: ["127.0.0.1:18201:8200"] environment: <<: *stable-env @@ -207,13 +244,19 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-query-1-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/query/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/query/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/query/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} - healthcheck: {test: [CMD, python, -c, "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8200/health/ready', timeout=2)"], interval: 5s, timeout: 3s, retries: 20} + healthcheck: {test: [CMD, python, -c, "import ssl,urllib.request; c=ssl.create_default_context(cafile='/stable-certs/query/ca.crt'); c.load_cert_chain('/stable-certs/query/server.crt','/stable-certs/query/server.key'); urllib.request.urlopen('https://localhost:8200/health/ready',context=c,timeout=2)"], interval: 5s, timeout: 3s, retries: 20} query_v2_2: <<: *python - command: [uvicorn, app.entrypoints.query_v2_stable:app, --host, 0.0.0.0, --port, "8200", --no-access-log] - networks: [stable_internal, stable_ingress] + command: [python, -m, app.entrypoints.query_v2_stable] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-query]} ports: ["127.0.0.1:18202:8200"] environment: <<: *stable-env @@ -222,12 +265,18 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-query-2-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/query/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/query/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/query/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} stream_v2_active: <<: *python command: [python, -m, app.entrypoints.stream_v2_stable] - networks: [stable_internal, stable_ingress] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-stream-a]} ports: ["127.0.0.1:18210:8200", "127.0.0.1:18220:8210"] environment: <<: *stable-env @@ -236,12 +285,18 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-stream-active-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/stream/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/stream/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/stream/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} stream_v2_passive: <<: *python command: [python, -m, app.entrypoints.stream_v2_stable] - networks: [stable_internal, stable_ingress] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-stream-b]} ports: ["127.0.0.1:18211:8200", "127.0.0.1:18221:8210"] environment: <<: *stable-env @@ -250,6 +305,9 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-stream-passive-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/stream/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/stream/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/stream/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} projector_v2: @@ -267,7 +325,10 @@ services: QDL_STABLE_KAFKA_RAW_TOPICS: md.raw.stable.v1 QDL_STABLE_KAFKA_CANONICAL_TOPIC: md.canonical.v2 QDL_STABLE_KAFKA_CERT_ROOT: /stable-certs/projector - QDL_STABLE_STREAM_INGEST_URLS_JSON: '["http://stream_v2_active:8200","http://stream_v2_passive:8200"]' + QDL_STABLE_TLS_CA_FILE: /stable-certs/projector/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/projector/client.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/projector/client.key + QDL_STABLE_STREAM_INGEST_URLS_JSON: '["https://stream_v2_active:8200","https://stream_v2_passive:8200"]' volumes: [stable_state:/var/lib/qdl-stable, stable_tls:/stable-certs:ro] depends_on: stable_tls_init: {condition: service_completed_successfully} @@ -279,6 +340,33 @@ services: stream_v2_active: {condition: service_started} stream_v2_passive: {condition: service_started} + authority_outbox_v2: + <<: *python + profiles: [stable-authority] + command: [python, /app/scripts/run_authority_outbox_dispatcher.py, --poll-seconds, "0.5", --batch-size, "20"] + environment: + <<: *stable-env + QDL_CONTROL_DB_DSN: ${QDL_STABLE_CONTROL_DB_DSN:?set QDL_STABLE_CONTROL_DB_DSN} + QDL_KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092 + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-authority-dispatcher + QDL_KAFKA_CA_LOCATION: /stable-certs/authority-dispatcher/ca.crt + QDL_KAFKA_CERT_LOCATION: /stable-certs/authority-dispatcher/client.crt + QDL_KAFKA_KEY_LOCATION: /stable-certs/authority-dispatcher/client.key + QDL_AUTHORITY_TOPIC: qdl.authority.v1 + QDL_AUTHORITY_HEALTH_FILE: /var/lib/qdl-stable/runtime/authority-dispatcher-health.json + depends_on: + stable_authority_db: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + stable_state_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + healthcheck: + test: [CMD, python, -c, "import json,time; p=json.load(open('/var/lib/qdl-stable/runtime/authority-dispatcher-health.json')); assert p['status']=='READY' and time.time_ns()-int(p['heartbeat_ns'])<5000000000"] + interval: 3s + timeout: 2s + retries: 20 + rust_core: <<: *rust entrypoint: [/usr/local/bin/qdl-realtime-core] @@ -327,6 +415,66 @@ services: kafka2: {condition: service_healthy} kafka3: {condition: service_healthy} + production_core_1: + <<: *rust + profiles: [stable-authority-primary] + entrypoint: [/usr/local/bin/qdl-production-core] + command: [/runtime/production-core-001.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-production-core-001 + QDL_KAFKA_GROUP_ID: qdl-v2-production-core-v1 + volumes: + - stable_tls:/stable-certs:ro + - "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/production-core-001.json:/runtime/production-core-001.json:ro" + depends_on: + authority_outbox_v2: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + production_core_2: + <<: *rust + profiles: [stable-authority-primary] + entrypoint: [/usr/local/bin/qdl-production-core] + command: [/runtime/production-core-002.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-production-core-002 + QDL_KAFKA_GROUP_ID: qdl-v2-production-core-v1 + volumes: + - stable_tls:/stable-certs:ro + - "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/production-core-002.json:/runtime/production-core-002.json:ro" + depends_on: + authority_outbox_v2: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + production_core_3: + <<: *rust + profiles: [stable-authority-primary] + entrypoint: [/usr/local/bin/qdl-production-core] + command: [/runtime/production-core-003.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-production-core-003 + QDL_KAFKA_GROUP_ID: qdl-v2-production-core-v1 + volumes: + - stable_tls:/stable-certs:ro + - "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/production-core-003.json:/runtime/production-core-003.json:ro" + depends_on: + authority_outbox_v2: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + ingestor_binance_usdm: <<: *rust entrypoint: [/usr/local/bin/qdl-native-raw-ingestor] @@ -467,9 +615,13 @@ networks: stable_internal: {internal: true, labels: {qdl.scope: v2-stable-candidate}} stable_ingress: {labels: {qdl.scope: v2-stable-candidate-ingress}} stable_egress: {labels: {qdl.scope: v2-stable-candidate-egress}} + stable_consumer: + external: true + name: ${QDL_STABLE_CONSUMER_NETWORK:?set QDL_STABLE_CONSUMER_NETWORK} volumes: kafka1_data: {labels: {qdl.scope: v2-stable-candidate}} kafka2_data: {labels: {qdl.scope: v2-stable-candidate}} kafka3_data: {labels: {qdl.scope: v2-stable-candidate}} stable_state: {labels: {qdl.scope: v2-stable-candidate}} stable_tls: {labels: {qdl.scope: v2-stable-candidate-secrets}} + stable_authority_db: {labels: {qdl.scope: v2-stable-authority-control}} diff --git a/docs/runbooks/v2-production-rust-authority-cutover.md b/docs/runbooks/v2-production-rust-authority-cutover.md index 1b11ba9..cd125eb 100644 --- a/docs/runbooks/v2-production-rust-authority-cutover.md +++ b/docs/runbooks/v2-production-rust-authority-cutover.md @@ -19,6 +19,7 @@ DNSE is excluded until its separate provider gate passes. - Stable project: `qdl_v2_stable_candidate`. - Initial V2 authority: `RUST_SHADOW`. - Initial venues: Binance and OKX only. +- Operator-declared active consumer: Trading System only; all alphas are down. - V1, current Redis and current provider processes are not restarted by the merge or isolated-deploy steps. @@ -92,6 +93,29 @@ Do not continue while the stable binary still rejects `RUST_CANARY` or `RUST_PRIMARY`, or while authority can be changed by environment variable alone. +Gate 1 is implemented on the feature branch, but no production CAS has +executed. The deployable topology reuses migrations 0006/0007/0009 and adds +migration 0010, a dedicated non-public PostgreSQL authority database, +function-scoped dispatcher role, transactional outbox dispatcher, compacted +authority/checkpoint topics, per-principal ACLs, and three bounded +qdl-production-core workers behind stable-authority and +stable-authority-primary profiles. + +The stable binary reconstructs authority and every target checkpoint before +reading raw input. Missing, partial, stale or wrong-owner state fails closed. +Only RUST_CANARY writes the canary topic; only RUST_PRIMARY writes canonical, +public V2 and legacy compatibility topics. + +Verification completed before immutable build: + +- focused topology/outbox/operator tests: 23/23 passed; +- disposable network-none PostgreSQL bootstrap and least-privilege smoke: pass; +- full Python suite: 546 passed, 6 environment skips; +- Rust workspace: 70 passed; fmt and strict Clippy passed; +- isolated real-provider SDK acceptance: Binance and OKX query/stream/cursor + parity passed while V1 remained HTTP 200. + + ## Gate 2 - Build Immutable Artifacts After Gate 1 is committed and CI-green: @@ -130,12 +154,15 @@ python scripts/phaseb_prepare_stable_candidate.py \ --python-image "$PYTHON_IMAGE" \ --rust-image "$RUST_IMAGE" \ --cert-dir /path/to/approved/phase8-certificates \ - --output-dir "$QDL_RELEASE_ROOT" + --output-dir "$QDL_RELEASE_ROOT" \ + --consumer-network executor_network ``` The manifest must report contract `2.0.0`, authority `RUST_SHADOW`, `cutover_authorized=false`, immutable image IDs, five consumer manifests and -no recorded secret values. +no recorded secret values. `--consumer-network` must name an already-created +external network shared with the sole approved consumer. Only V2 query/stream +ingress joins it; Kafka, Redis, projector and Rust cores remain private. ## Gate 4 - Start Isolated V2 @@ -170,13 +197,39 @@ docker compose \ Acceptance: ```bash -curl --fail --silent http://127.0.0.1:18201/health/ready -curl --fail --silent http://127.0.0.1:18202/health/ready -curl --fail --silent http://127.0.0.1:18210/health/ready -curl --fail --silent http://127.0.0.1:18211/health/ready +TLS_CA="$QDL_RELEASE_ROOT/identities/trading-system/ca.crt" +TLS_CERT="$QDL_RELEASE_ROOT/identities/trading-system/client.crt" +TLS_KEY="$QDL_RELEASE_ROOT/identities/trading-system/client.key" + +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18201/health/ready +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18202/health/ready +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18210/health/live +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18211/health/live curl --fail --silent http://127.0.0.1:8100/v1/health -``` +if curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18210/health/ready >/dev/null; then + GRPC_TARGET=localhost:18220 +else + curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18211/health/ready >/dev/null + GRPC_TARGET=localhost:18221 +fi + +docker run --rm --network host \ + -e QDL_STABLE_JWT_PRIVATE_KEY_FILE=/bundle/identities/trading-system-jwt/private.key \ + -e QDL_STABLE_JWT_KEY_ID=stable-trading-system-rs256-v1 \ + -v "$PWD:/workspace:ro" -v "$QDL_RELEASE_ROOT:/bundle:ro" \ + "$PYTHON_IMAGE" python /workspace/scripts/phasec1_isolated_consumer_acceptance.py \ + --grpc-target "$GRPC_TARGET" \ + --tls-ca-file /bundle/identities/trading-system/ca.crt \ + --tls-certificate-file /bundle/identities/trading-system/client.crt \ + --tls-private-key-file /bundle/identities/trading-system/client.key +``` Require authentic Binance/OKX data, zero unexplained gap/duplicate/quarantine, bounded broker/projector lag, exact replica results and V1 unchanged. @@ -191,20 +244,29 @@ docker compose \ Do not add `-v`; preserve evidence until the failed gate is understood. -## Gate 5 - Paper Consumer Canary +## Gate 5 - Trading System Dual-Read And Route Switch + +Do not start or migrate an alpha. Keep Trading System on V1 while its adapter +performs a bounded read-only comparison against V2 for the same Binance/OKX +instruments and closed-bar/event boundaries. -Move only explicitly named manifests, in this order: +Require exact identity, timestamp, decimal, unit, finality and session semantics; +zero unexplained gap/duplicate; bounded freshness/lag; and successful signed +cursor replay/restart. Built-in health and authority telemetry must be green. -1. monitoring; -2. one paper alpha; -3. Trading System paper market-data adapter; -4. remaining approved paper consumers. +Then perform one controlled Trading System adapter restart with venue-aware +routing: -For each consumer, record V1 cursor/watermark, V2 warmup result, first live -cursor, restart cursor, freshness, gaps, duplicates, source/session status and -rollback route. No sandbox/live order path is included. +```text +Binance/OKX: V2 primary -> governed V1 fallback +DNSE: V1 only +``` -Rollback changes only that consumer endpoint/SDK config back to V1. +The exact adapter config keys and restart command are populated from the current +Trading System deployment during the cutover preflight; do not invent or +hardcode them in advance. Every fallback/return transition records source, +reason, watermark and operator identity. V1 fallback is accepted only when its +freshness/session/contract checks pass; otherwise execution remains blocked. ## Gate 6 - Exact-Slice Authority Approval @@ -236,9 +298,11 @@ operator: change_ticket: ``` -The operator must explicitly approve this exact packet. +The operator must explicitly approve this exact packet. One packet may enumerate +all approved Binance/OKX slices for one maintenance window, but the controller +executes and audits each CAS independently. -The transition follows only: +Every slice follows only: ```text PYTHON_PRIMARY @@ -247,16 +311,53 @@ PYTHON_PRIMARY -> RUST_PRIMARY ``` -The old writer is fenced at `W`; Rust reconstructs every required target -through `W` and first publishes at `W+1`. Any failed gate enters `BLOCKED` -and rolls back using a newer authority revision and accepted reverse handoff. +The canary gate is bounded by accepted real events and continuity evidence, not +an arbitrary multi-day wait. The old writer is fenced at `W`; Rust reconstructs +all required targets through `W` and first publishes at `W+1`. A failed slice +enters `BLOCKED` and rolls back under a newer revision without undoing an +unrelated healthy slice. + +The source-owned command is plan-only by default. Every packet contains exactly +one state step for 1..32 unique slices, expires, binds image/contract/partition/ +route digests, requires clean real-provider evidence and includes an executable +Trading System V1 rollback command. + + python scripts/phasec3_authority_cutover.py \ + --packet /secure/qdl-v2/change/canary-packet.json + +Review the printed APPLY_C3_ token. Apply only the same packet bytes: + + QDL_CONTROL_ADMIN_DSN='postgresql://...' \ + python scripts/phasec3_authority_cutover.py \ + --packet /secure/qdl-v2/change/canary-packet.json \ + --apply --confirm APPLY_C3_ + +Start control services before canary. Start production workers only after the +RUST_CANARY authority event is durable: + + docker compose --env-file "$QDL_RELEASE_ROOT/stable.env" \ + -f docker-compose.v2-stable.yml --profile stable-authority \ + up -d stable_authority_db authority_outbox_v2 + + docker compose --env-file "$QDL_RELEASE_ROOT/stable.env" \ + -f docker-compose.v2-stable.yml \ + --profile stable-authority --profile stable-authority-primary \ + up -d production_core_1 production_core_2 production_core_3 + +The command checks the current DB row under lock and executes one transaction +per slice, stopping on the first stale CAS. Primary and Python restore use the +accepted qdl_transition_authority_v2 handoff; no environment label can promote +authority. + -## Gate 7 - Expand And Release +## Gate 7 - Close With V1 Hot Fallback -Hold the first primary slice for the approved window. Expand one venue/feed -slice at a time; no inherited certification. Keep V1 available until every -registered consumer has migrated and rollback has been exercised. +After approved Binance/OKX slices are `RUST_PRIMARY`, Trading System uses V2 +normally and V1 remains live at port `8100` as the tested fallback. Exercise +one V2 -> V1 -> V2 route drill with durable source-switch audit and no market +semantic mismatch. -Only then may the operator approve stable V2 public routing, a V1 sunset date, -a release PR `dev -> main`, and tag/release publication. DNSE remains disabled -until its provider-specific external gates pass. +There is no alpha migration and no V1 sunset in this cutover. Publish the V2 +release only after the Trading System cycle, authority audit, cursor/replay +continuity and fallback drill pass. DNSE remains V1-only until its provider +gate passes. diff --git a/migrations/postgres/0009_production_authority_outbox.sql b/migrations/postgres/0009_production_authority_outbox.sql new file mode 100644 index 0000000..4c00cf7 --- /dev/null +++ b/migrations/postgres/0009_production_authority_outbox.sql @@ -0,0 +1,230 @@ +BEGIN; + +CREATE EXTENSION IF NOT EXISTS pgcrypto; + +CREATE TABLE IF NOT EXISTS qdl_authority_event_outbox ( + event_id UUID PRIMARY KEY, + transition_id UUID NOT NULL UNIQUE + REFERENCES qdl_authority_transition_audit(transition_id), + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + event_kind TEXT NOT NULL CHECK (event_kind = 'AUTHORITY_TRANSITION'), + payload JSONB NOT NULL CHECK (jsonb_typeof(payload) = 'object'), + payload_sha256 TEXT NOT NULL CHECK (payload_sha256 ~ '^[0-9a-f]{64}$'), + status TEXT NOT NULL DEFAULT 'PENDING' + CHECK (status IN ('PENDING', 'DISPATCHING', 'PUBLISHED', 'BLOCKED')), + attempts INTEGER NOT NULL DEFAULT 0 CHECK (attempts >= 0), + available_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + locked_at TIMESTAMPTZ, + lock_owner TEXT, + last_error TEXT, + topic TEXT, + topic_partition INTEGER, + topic_offset BIGINT CHECK (topic_offset IS NULL OR topic_offset >= 0), + published_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (slice_id, authority_revision, event_kind), + CHECK ((locked_at IS NULL) = (lock_owner IS NULL)), + CHECK ( + (status = 'PUBLISHED' AND topic IS NOT NULL + AND topic_partition IS NOT NULL AND topic_offset IS NOT NULL + AND published_at IS NOT NULL) + OR status <> 'PUBLISHED' + ) +); + +CREATE INDEX IF NOT EXISTS qdl_authority_outbox_dispatch_idx + ON qdl_authority_event_outbox (status, available_at, created_at) + WHERE status IN ('PENDING', 'DISPATCHING'); + +CREATE OR REPLACE FUNCTION qdl_reject_authority_outbox_payload_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + IF NEW.event_id <> OLD.event_id + OR NEW.transition_id <> OLD.transition_id + OR NEW.slice_id <> OLD.slice_id + OR NEW.authority_revision <> OLD.authority_revision + OR NEW.event_kind <> OLD.event_kind + OR NEW.payload <> OLD.payload + OR NEW.payload_sha256 <> OLD.payload_sha256 + OR NEW.created_at <> OLD.created_at THEN + RAISE EXCEPTION 'authority outbox identity/payload is immutable'; + END IF; + NEW.updated_at := clock_timestamp(); + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_outbox_payload_immutable + ON qdl_authority_event_outbox; +CREATE TRIGGER qdl_authority_outbox_payload_immutable +BEFORE UPDATE ON qdl_authority_event_outbox +FOR EACH ROW EXECUTE FUNCTION qdl_reject_authority_outbox_payload_mutation(); + +CREATE OR REPLACE FUNCTION qdl_enqueue_authority_transition() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + authority_row qdl_authority_slices%ROWTYPE; + handoff_row qdl_authority_handoffs%ROWTYPE; + checkpoint_row qdl_terminal_owner_checkpoints%ROWTYPE; + event_payload JSONB; +BEGIN + SELECT * INTO authority_row + FROM qdl_authority_slices AS authority_slice + WHERE authority_slice.slice_id = NEW.slice_id; + IF NOT FOUND OR authority_row.authority_revision <> NEW.new_revision THEN + RAISE EXCEPTION 'authority outbox cannot snapshot a divergent authority row'; + END IF; + + SELECT * INTO handoff_row + FROM qdl_authority_handoffs AS handoff + WHERE handoff.slice_id = NEW.slice_id + AND handoff.expected_authority_revision = NEW.previous_revision + AND handoff.new_authority_revision = NEW.new_revision + AND handoff.old_owner_id = NEW.previous_owner_id + AND handoff.new_owner_id = NEW.new_owner_id + ORDER BY handoff.created_at DESC + LIMIT 1; + IF FOUND THEN + SELECT * INTO checkpoint_row + FROM qdl_terminal_owner_checkpoints AS checkpoint + WHERE checkpoint.checkpoint_id = handoff_row.checkpoint_id; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority handoff checkpoint is missing'; + END IF; + END IF; + + event_payload := jsonb_build_object( + 'schema', 'qdl.authority-outbox-event.v1', + 'event_id', NEW.transition_id, + 'transition', to_jsonb(NEW), + 'authority', to_jsonb(authority_row), + 'handoff', CASE WHEN handoff_row.handoff_id IS NULL THEN NULL ELSE to_jsonb(handoff_row) END, + 'checkpoint', CASE WHEN checkpoint_row.checkpoint_id IS NULL THEN NULL ELSE to_jsonb(checkpoint_row) END + ); + + INSERT INTO qdl_authority_event_outbox ( + event_id, transition_id, slice_id, authority_revision, + event_kind, payload, payload_sha256 + ) VALUES ( + NEW.transition_id, NEW.transition_id, NEW.slice_id, NEW.new_revision, + 'AUTHORITY_TRANSITION', event_payload, + encode(digest(convert_to(event_payload::text, 'UTF8'), 'sha256'), 'hex') + ) + ON CONFLICT (event_id) DO NOTHING; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_transition_outbox + ON qdl_authority_transition_audit; +CREATE TRIGGER qdl_authority_transition_outbox +AFTER INSERT ON qdl_authority_transition_audit +FOR EACH ROW EXECUTE FUNCTION qdl_enqueue_authority_transition(); + +CREATE OR REPLACE FUNCTION qdl_claim_authority_outbox( + p_lock_owner TEXT, + p_limit INTEGER, + p_lock_timeout INTERVAL DEFAULT INTERVAL '2 minutes' +) +RETURNS SETOF qdl_authority_event_outbox +LANGUAGE plpgsql +AS $$ +BEGIN + IF btrim(p_lock_owner) = '' OR p_limit < 1 OR p_limit > 100 + OR p_lock_timeout <= INTERVAL '0 seconds' THEN + RAISE EXCEPTION 'authority outbox claim bounds are invalid'; + END IF; + RETURN QUERY + WITH candidates AS ( + SELECT event_id + FROM qdl_authority_event_outbox + WHERE available_at <= clock_timestamp() + AND ( + status = 'PENDING' + OR ( + status = 'DISPATCHING' + AND locked_at < clock_timestamp() - p_lock_timeout + ) + ) + ORDER BY created_at, event_id + LIMIT p_limit + FOR UPDATE SKIP LOCKED + ) + UPDATE qdl_authority_event_outbox AS outbox + SET status = 'DISPATCHING', + attempts = outbox.attempts + 1, + locked_at = clock_timestamp(), + lock_owner = p_lock_owner, + last_error = NULL + FROM candidates + WHERE outbox.event_id = candidates.event_id + RETURNING outbox.*; +END; +$$; + +CREATE OR REPLACE FUNCTION qdl_complete_authority_outbox( + p_event_id UUID, + p_lock_owner TEXT, + p_topic TEXT, + p_partition INTEGER, + p_offset BIGINT +) +RETURNS VOID LANGUAGE plpgsql AS $$ +BEGIN + UPDATE qdl_authority_event_outbox + SET status = 'PUBLISHED', + topic = p_topic, + topic_partition = p_partition, + topic_offset = p_offset, + published_at = clock_timestamp(), + locked_at = NULL, + lock_owner = NULL, + last_error = NULL + WHERE event_id = p_event_id + AND status = 'DISPATCHING' + AND lock_owner = p_lock_owner + AND btrim(p_topic) <> '' + AND p_partition >= 0 + AND p_offset >= 0; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority outbox completion lost ownership or has invalid ACK'; + END IF; +END; +$$; + +CREATE OR REPLACE FUNCTION qdl_retry_authority_outbox( + p_event_id UUID, + p_lock_owner TEXT, + p_error TEXT, + p_retry_after INTERVAL +) +RETURNS VOID LANGUAGE plpgsql AS $$ +BEGIN + IF btrim(p_error) = '' OR p_retry_after <= INTERVAL '0 seconds' THEN + RAISE EXCEPTION 'authority outbox retry requires bounded error/delay'; + END IF; + UPDATE qdl_authority_event_outbox + SET status = CASE WHEN attempts >= 20 THEN 'BLOCKED' ELSE 'PENDING' END, + available_at = clock_timestamp() + p_retry_after, + locked_at = NULL, + lock_owner = NULL, + last_error = left(p_error, 2000) + WHERE event_id = p_event_id + AND status = 'DISPATCHING' + AND lock_owner = p_lock_owner; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority outbox retry lost ownership'; + END IF; +END; +$$; + +REVOKE EXECUTE ON FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + FROM PUBLIC; + +COMMIT; diff --git a/migrations/postgres/0010_authority_dispatcher_security.sql b/migrations/postgres/0010_authority_dispatcher_security.sql new file mode 100644 index 0000000..243eaa0 --- /dev/null +++ b/migrations/postgres/0010_authority_dispatcher_security.sql @@ -0,0 +1,20 @@ +BEGIN; + +ALTER FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + SECURITY DEFINER + SET search_path = pg_catalog, public; +ALTER FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + SECURITY DEFINER + SET search_path = pg_catalog, public; +ALTER FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + SECURITY DEFINER + SET search_path = pg_catalog, public; + +REVOKE ALL ON FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + FROM PUBLIC; +REVOKE ALL ON FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + FROM PUBLIC; +REVOKE ALL ON FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + FROM PUBLIC; + +COMMIT; diff --git a/migrations/postgres/9999_init_authority_dispatcher_role.sh b/migrations/postgres/9999_init_authority_dispatcher_role.sh new file mode 100755 index 0000000..cfe899f --- /dev/null +++ b/migrations/postgres/9999_init_authority_dispatcher_role.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${POSTGRES_USER:?POSTGRES_USER is required}" +: "${POSTGRES_DB:?POSTGRES_DB is required}" +: "${QDL_STABLE_DISPATCHER_DB_PASSWORD:?QDL_STABLE_DISPATCHER_DB_PASSWORD is required}" + +psql --set=ON_ERROR_STOP=1 \ + --username "${POSTGRES_USER}" \ + --dbname "${POSTGRES_DB}" \ + --set=dispatcher_password="${QDL_STABLE_DISPATCHER_DB_PASSWORD}" <<'SQL' +SELECT format( + 'CREATE ROLE qdl_authority_dispatcher LOGIN PASSWORD %L', + :'dispatcher_password' +) +WHERE NOT EXISTS ( + SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'qdl_authority_dispatcher' +) +\gexec +ALTER ROLE qdl_authority_dispatcher + NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOREPLICATION + PASSWORD :'dispatcher_password'; +GRANT CONNECT ON DATABASE qdl_authority TO qdl_authority_dispatcher; +GRANT USAGE ON SCHEMA public TO qdl_authority_dispatcher; +GRANT EXECUTE ON FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + TO qdl_authority_dispatcher; +GRANT EXECUTE ON FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + TO qdl_authority_dispatcher; +GRANT EXECUTE ON FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + TO qdl_authority_dispatcher; +SQL diff --git a/poetry.lock b/poetry.lock index 734fce4..96ba270 100644 --- a/poetry.lock +++ b/poetry.lock @@ -57,6 +57,79 @@ files = [ {file = "async_timeout-5.0.1.tar.gz", hash = "sha256:d9321a7a3d5a6a5e187e824d2fa0793ce379a202935782d555d6e9d2735677d3"}, ] +[[package]] +name = "asyncpg" +version = "0.31.0" +description = "An asyncio PostgreSQL driver" +optional = false +python-versions = ">=3.9.0" +groups = ["main"] +files = [ + {file = "asyncpg-0.31.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:831712dd3cf117eec68575a9b50da711893fd63ebe277fc155ecae1c6c9f0f61"}, + {file = "asyncpg-0.31.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0b17c89312c2f4ccea222a3a6571f7df65d4ba2c0e803339bfc7bed46a96d3be"}, + {file = "asyncpg-0.31.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3faa62f997db0c9add34504a68ac2c342cfee4d57a0c3062fcf0d86c7f9cb1e8"}, + {file = "asyncpg-0.31.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ea599d45c361dfbf398cb67da7fd052affa556a401482d3ff1ee99bd68808a1"}, + {file = "asyncpg-0.31.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:795416369c3d284e1837461909f58418ad22b305f955e625a4b3a2521d80a5f3"}, + {file = "asyncpg-0.31.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:a8d758dac9d2e723e173d286ef5e574f0b350ec00e9186fce84d0fc5f6a8e6b8"}, + {file = "asyncpg-0.31.0-cp310-cp310-win32.whl", hash = "sha256:2d076d42eb583601179efa246c5d7ae44614b4144bc1c7a683ad1222814ed095"}, + {file = "asyncpg-0.31.0-cp310-cp310-win_amd64.whl", hash = "sha256:9ea33213ac044171f4cac23740bed9a3805abae10e7025314cfbd725ec670540"}, + {file = "asyncpg-0.31.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:eee690960e8ab85063ba93af2ce128c0f52fd655fdff9fdb1a28df01329f031d"}, + {file = "asyncpg-0.31.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2657204552b75f8288de08ca60faf4a99a65deef3a71d1467454123205a88fab"}, + {file = "asyncpg-0.31.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a429e842a3a4b4ea240ea52d7fe3f82d5149853249306f7ff166cb9948faa46c"}, + {file = "asyncpg-0.31.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c0807be46c32c963ae40d329b3a686356e417f674c976c07fa49f1b30303f109"}, + {file = "asyncpg-0.31.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:e5d5098f63beeae93512ee513d4c0c53dc12e9aa2b7a1af5a81cddf93fe4e4da"}, + {file = "asyncpg-0.31.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37fc6c00a814e18eef51833545d1891cac9aa69140598bb076b4cd29b3e010b9"}, + {file = "asyncpg-0.31.0-cp311-cp311-win32.whl", hash = "sha256:5a4af56edf82a701aece93190cc4e094d2df7d33f6e915c222fb09efbb5afc24"}, + {file = "asyncpg-0.31.0-cp311-cp311-win_amd64.whl", hash = "sha256:480c4befbdf079c14c9ca43c8c5e1fe8b6296c96f1f927158d4f1e750aacc047"}, + {file = "asyncpg-0.31.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b44c31e1efc1c15188ef183f287c728e2046abb1d26af4d20858215d50d91fad"}, + {file = "asyncpg-0.31.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0c89ccf741c067614c9b5fc7f1fc6f3b61ab05ae4aaa966e6fd6b93097c7d20d"}, + {file = "asyncpg-0.31.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:12b3b2e39dc5470abd5e98c8d3373e4b1d1234d9fbdedf538798b2c13c64460a"}, + {file = "asyncpg-0.31.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:aad7a33913fb8bcb5454313377cc330fbb19a0cd5faa7272407d8a0c4257b671"}, + {file = "asyncpg-0.31.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3df118d94f46d85b2e434fd62c84cb66d5834d5a890725fe625f498e72e4d5ec"}, + {file = "asyncpg-0.31.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bd5b6efff3c17c3202d4b37189969acf8927438a238c6257f66be3c426beba20"}, + {file = "asyncpg-0.31.0-cp312-cp312-win32.whl", hash = "sha256:027eaa61361ec735926566f995d959ade4796f6a49d3bde17e5134b9964f9ba8"}, + {file = "asyncpg-0.31.0-cp312-cp312-win_amd64.whl", hash = "sha256:72d6bdcbc93d608a1158f17932de2321f68b1a967a13e014998db87a72ed3186"}, + {file = "asyncpg-0.31.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c204fab1b91e08b0f47e90a75d1b3c62174dab21f670ad6c5d0f243a228f015b"}, + {file = "asyncpg-0.31.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:54a64f91839ba59008eccf7aad2e93d6e3de688d796f35803235ea1c4898ae1e"}, + {file = "asyncpg-0.31.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0e0822b1038dc7253b337b0f3f676cadc4ac31b126c5d42691c39691962e403"}, + {file = "asyncpg-0.31.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bef056aa502ee34204c161c72ca1f3c274917596877f825968368b2c33f585f4"}, + {file = "asyncpg-0.31.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0bfbcc5b7ffcd9b75ab1558f00db2ae07db9c80637ad1b2469c43df79d7a5ae2"}, + {file = "asyncpg-0.31.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:22bc525ebbdc24d1261ecbf6f504998244d4e3be1721784b5f64664d61fbe602"}, + {file = "asyncpg-0.31.0-cp313-cp313-win32.whl", hash = "sha256:f890de5e1e4f7e14023619399a471ce4b71f5418cd67a51853b9910fdfa73696"}, + {file = "asyncpg-0.31.0-cp313-cp313-win_amd64.whl", hash = "sha256:dc5f2fa9916f292e5c5c8b2ac2813763bcd7f58e130055b4ad8a0531314201ab"}, + {file = "asyncpg-0.31.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:f6b56b91bb0ffc328c4e3ed113136cddd9deefdf5f79ab448598b9772831df44"}, + {file = "asyncpg-0.31.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:334dec28cf20d7f5bb9e45b39546ddf247f8042a690bff9b9573d00086e69cb5"}, + {file = "asyncpg-0.31.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:98cc158c53f46de7bb677fd20c417e264fc02b36d901cc2a43bd6cb0dc6dbfd2"}, + {file = "asyncpg-0.31.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9322b563e2661a52e3cdbc93eed3be7748b289f792e0011cb2720d278b366ce2"}, + {file = "asyncpg-0.31.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19857a358fc811d82227449b7ca40afb46e75b33eb8897240c3839dd8b744218"}, + {file = "asyncpg-0.31.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ba5f8886e850882ff2c2ace5732300e99193823e8107e2c53ef01c1ebfa1e85d"}, + {file = "asyncpg-0.31.0-cp314-cp314-win32.whl", hash = "sha256:cea3a0b2a14f95834cee29432e4ddc399b95700eb1d51bbc5bfee8f31fa07b2b"}, + {file = "asyncpg-0.31.0-cp314-cp314-win_amd64.whl", hash = "sha256:04d19392716af6b029411a0264d92093b6e5e8285ae97a39957b9a9c14ea72be"}, + {file = "asyncpg-0.31.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:bdb957706da132e982cc6856bb2f7b740603472b54c3ebc77fe60ea3e57e1bd2"}, + {file = "asyncpg-0.31.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6d11b198111a72f47154fa03b85799f9be63701e068b43f84ac25da0bda9cb31"}, + {file = "asyncpg-0.31.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:18c83b03bc0d1b23e6230f5bf8d4f217dc9bc08644ce0502a9d91dc9e634a9c7"}, + {file = "asyncpg-0.31.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e009abc333464ff18b8f6fd146addffd9aaf63e79aa3bb40ab7a4c332d0c5e9e"}, + {file = "asyncpg-0.31.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3b1fbcb0e396a5ca435a8826a87e5c2c2cc0c8c68eb6fadf82168056b0e53a8c"}, + {file = "asyncpg-0.31.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8df714dba348efcc162d2adf02d213e5fab1bd9f557e1305633e851a61814a7a"}, + {file = "asyncpg-0.31.0-cp314-cp314t-win32.whl", hash = "sha256:1b41f1afb1033f2b44f3234993b15096ddc9cd71b21a42dbd87fc6a57b43d65d"}, + {file = "asyncpg-0.31.0-cp314-cp314t-win_amd64.whl", hash = "sha256:bd4107bb7cdd0e9e65fae66a62afd3a249663b844fa34d479f6d5b3bef9c04c3"}, + {file = "asyncpg-0.31.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:ebb3cde58321a1f89ce41812be3f2a98dddedc1e76d0838aba1d724f1e4e1a95"}, + {file = "asyncpg-0.31.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e6974f36eb9a224d8fb428bcf66bd411aa12cf57c2967463178149e73d4de366"}, + {file = "asyncpg-0.31.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bc2b685f400ceae428f79f78b58110470d7b4466929a7f78d455964b17ad1008"}, + {file = "asyncpg-0.31.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bb223567dea5f47c45d347f2bde5486be8d9f40339f27217adb3fb1c3be51298"}, + {file = "asyncpg-0.31.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:22be6e02381bab3101cd502d9297ac71e2f966c86e20e78caead9934c98a8af6"}, + {file = "asyncpg-0.31.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:37a58919cfef2448a920df00d1b2f821762d17194d0dbf355d6dde8d952c04f9"}, + {file = "asyncpg-0.31.0-cp39-cp39-win32.whl", hash = "sha256:c1a9c5b71d2371a2290bc93336cd05ba4ec781683cab292adbddc084f89443c6"}, + {file = "asyncpg-0.31.0-cp39-cp39-win_amd64.whl", hash = "sha256:c1e1ab5bc65373d92dd749d7308c5b26fb2dc0fbe5d3bf68a32b676aa3bcd24a"}, + {file = "asyncpg-0.31.0.tar.gz", hash = "sha256:c989386c83940bfbd787180f2b1519415e2d3d6277a70d9d0f0145ac73500735"}, +] + +[package.dependencies] +async_timeout = {version = ">=4.0.3", markers = "python_version < \"3.11.0\""} + +[package.extras] +gssauth = ["gssapi ; platform_system != \"Windows\"", "sspilib ; platform_system == \"Windows\""] + [[package]] name = "beautifulsoup4" version = "4.14.3" @@ -592,6 +665,69 @@ mypy = ["bokeh", "contourpy[bokeh,docs]", "docutils-stubs", "mypy (==1.17.0)", " test = ["Pillow", "contourpy[test-no-images]", "matplotlib"] test-no-images = ["pytest", "pytest-cov", "pytest-rerunfailures", "pytest-xdist", "wurlitzer"] +[[package]] +name = "cryptography" +version = "50.0.0" +description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers." +optional = false +python-versions = "!=3.9.0,!=3.9.1,>=3.9" +groups = ["main"] +files = [ + {file = "cryptography-50.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:031e2d5dd4bb9caa3ca9c82e5a197fd8ae680232cee62603d1a813f3f07e3d03"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fd9192b7b70c573d7f214eb1ae35e00d359f6f5e4b27c7e21e30de1fc6204645"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:06a32a980526a6ab9a4b9bf8f7385800791e2bb960903cb6b530e4817509a3b7"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a1b30560f2acc95aa8b2e06e716a13dbfc97314747b80d9707e307f77b40d6b3"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:8d89f3976b10b4ce31118de72329025f70d2c6ead14a8217c5514dd2c6d5a78f"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:b42a28c1844fd9de8f3f7d540e36b66f3a9c83fceac7170ebc7a6a19edd9dcae"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:900131fafd8aead39ac7dd3a7e833be754c17a95cfd91221636949fe4eb0aa8a"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:07949c449a1abcf60d1ee6e88956d89404c7df3c8258f46589e912988e551987"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:f89831ef99dd7dd169ab06d63a831adb9e20a87aac6d380266bbda5823349169"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:82148ec5bddac30b51a5b3c1945075f896fa022cb93f8e4a01e9f6ee95292c5f"}, + {file = "cryptography-50.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1489e263a8048bb8b6a8bac662eb2d402ea5d2b7b4699b72f385f1e2772db105"}, + {file = "cryptography-50.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:7cec5b856506da6defb290f30c9ee687d5f5e8cb0bd3f6459dde43b0b4fa40ef"}, + {file = "cryptography-50.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30"}, + {file = "cryptography-50.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:49e7d93abdbd2990caced757e5fade25302f719c3c8fb6e6fff2dde98999fc41"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:19736989797678c6af1e55cd49055cdbcb55d8f6b5583ac5335f933aba9101dc"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:80b63928fa35083b33966ce1efb70e5b9607181e49dcd1c22c8c005e319f667f"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:d58c3db7cd6eed54e6c06744db55456b65ebd7492ddeae9c1e93cfca7aa857d3"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:df2a58a472f332225671c35b0a830208b86d004f82baa8530fa3782c85646533"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:11b74db56cdbe3cdee6e3f6982ecb70334fa10dce99ed58bf7894aaaa3b2a037"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f59e38625469987d7ef6d495323c55e7db6c212eaf6112267e0d3b565a2e9c9f"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:ecfed7367f965a0328cfbdd70da860f15441f002f613185668c6e6ebf5a0ac11"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:9aa87839c383bdbab6ef865787a1fb877af8dd03464c4400322726feaaadfc6d"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:6ba6a53445bd3cfa809ef3ef5f1589aa6ba08784a1d962bf47d0940e871dab1c"}, + {file = "cryptography-50.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3f5735ffe4996d28b809371756219f5354864902a3b9e7c0b9ee87041209fc9c"}, + {file = "cryptography-50.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1b4a266766514614f8aa60416e71f2fc6e575d36e7bdc90f644fadb2f4b75b95"}, + {file = "cryptography-50.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:12b9c6996425c76ea6c457ace4f3073e715b8c545add07cd1a8f3a4f90691269"}, + {file = "cryptography-50.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ccdc4a71a4dabae05de219404f9f4abc38e3b58422177ff93d0da05967dafa07"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:910e1d2668e7de9648f2bcee30e180db2a6b15c30f887d7c4c93ddf96e3992e3"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a91296cb61e8df6f86d0c19cc4068228da256bf59bf86049fbd821084565327f"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e722f16708d854fe924790e051061f6704a472c3bac347b6fd88033ea8dd0dc5"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:d764dcf130c428ef66786f866dd750f53182bc608813489915e9fc106bb0c82f"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:105110f43a471dbd0060b9c9516cb8a6a79233631a04cc2ba16f28323ac6e025"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:828743d939e9629bc267b8e2d08d8bb67cd4319c771a33d4b18b22dd8fb7440a"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:2a8183b489dc1f7f80f135780fadc1108f14b31b8a40411c7a5b17425f65f28b"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6e7d61120573a7f2cd94cc095f9e81f6967c61ccdf194285aa143ecec8e0b708"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:37fdb0d0111f1e2ff07139dfb79f1b49531f8e213c46f1163dd7642979b58c47"}, + {file = "cryptography-50.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c87f62a3d3b9888ed0fdde100ec06aa61ca9cd44bad9057d1dff9a516b5f5bb9"}, + {file = "cryptography-50.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:65c2c3add92b45fd0709db8594536aea39c2a67af0e27ffcf049c498501140b7"}, + {file = "cryptography-50.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:d24fead1d4d076e1bfb006dcec392074a3cd8d7b4fc8a595aa64073b2b7a96ba"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:5e34edd123674534acd70147f0ca331eaa2c74e6325fb2028c886aa26ba0b68c"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:8eb5e1172eb569ea8a872796576e6a67c276351728b6455d5beb01242b027c6a"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:910d11e1a385c654bf738bf3e6b8e6ed5de0f5610fcae2be9e5b398d8081d20e"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:62598a8a57f815db4c6259a4e97d857dab56697e7de8e8ab02352ab74da1995d"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:07479a1cb08219ab719147e742e76090c9c773321959bb94946fffdd397a6437"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:c99c003e088647b8a5b7c145d6f78c335f6348332b62e142d411c4b63d1460b9"}, + {file = "cryptography-50.0.0.tar.gz", hash = "sha256:eeac2acb5a20ed25e0ad6d1df9891a520b78b404266b6d11778f25d5d691a6c9"}, +] + +[package.dependencies] +cffi = {version = ">=2.0.0", markers = "platform_python_implementation != \"PyPy\""} +typing-extensions = {version = ">=4.13.2", markers = "python_full_version < \"3.11.0\""} + +[package.extras] +ssh = ["bcrypt (>=3.1.5)"] + [[package]] name = "cycler" version = "0.12.1" @@ -1998,6 +2134,7 @@ files = [ ] [package.dependencies] +cryptography = {version = ">=3.4.0", optional = true, markers = "extra == \"crypto\""} typing_extensions = {version = ">=4.0", markers = "python_version < \"3.11\""} [package.extras] @@ -2801,4 +2938,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.1" python-versions = "^3.10" -content-hash = "07ffa76dde42c0a3c35b4af86a3c9c0a2bb7f65552c87c7dc4ba78401246fda0" +content-hash = "f5d8c82d042f5dd6bd468543bb5d7f457532c2528daba5e77415d50d45c2bca2" diff --git a/pyproject.toml b/pyproject.toml index baf23de..f4c627c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,10 +26,11 @@ dependencies = [ "protobuf (>=6.31.1,<7.0.0)", "grpcio (>=1.70.0,<2.0.0)", "confluent-kafka (>=2.15.0,<3.0.0)", + "asyncpg (>=0.30.0,<1.0.0)", "httpx (>=0.28.0,<1.0.0)", "idna (>=3.15,<4.0)", "pillow (>=12.3.0,<13.0.0)", - "pyjwt (>=2.13.0,<3.0.0)", + "pyjwt[crypto] (>=2.13.0,<3.0.0)", "soupsieve (>=2.8.4,<3.0.0)", "starlette (>=1.3.1,<2.0.0)" ] diff --git a/qdl/adapters/binance_usdm.py b/qdl/adapters/binance_usdm.py index 15c33f6..35550ea 100644 --- a/qdl/adapters/binance_usdm.py +++ b/qdl/adapters/binance_usdm.py @@ -72,11 +72,21 @@ def parse_exchange_info(payload: Mapping[str, Any], *, valid_from_ns: int) -> Bi if not native_symbol: raise ValueError("Binance exchangeInfo contains an empty symbol") product_type = ProductType.PERPETUAL if contract_type == "PERPETUAL" else ProductType.FUTURE + base_asset = str(item.get("baseAsset") or "").upper() + quote_asset = str(item.get("quoteAsset") or "").upper() + if not base_asset or not quote_asset: + raise ValueError("Binance exchangeInfo is missing base/quote asset identity") + canonical_symbol = f"{base_asset}-{quote_asset}" + if product_type is ProductType.FUTURE: + _, separator, contract_code = native_symbol.rpartition("_") + if not separator or not contract_code.isdigit(): + raise ValueError("Binance dated future symbol has no contract code") + canonical_symbol = f"{canonical_symbol}-{contract_code}" identity = InstrumentIdentity.create( venue="BINANCE", market="USDM", product_type=product_type, - canonical_symbol=native_symbol, + canonical_symbol=canonical_symbol, ) expiry_ms = int(item.get("deliveryDate") or 0) record = InstrumentRecord( @@ -84,8 +94,8 @@ def parse_exchange_info(payload: Mapping[str, Any], *, valid_from_ns: int) -> Bi metadata_revision=1, asset_class=AssetClass.DERIVATIVE, native_symbol=native_symbol, - base_asset=str(item.get("baseAsset") or "").upper(), - quote_asset=str(item.get("quoteAsset") or "").upper(), + base_asset=base_asset, + quote_asset=quote_asset, settlement_asset=str(item.get("marginAsset") or "").upper(), price_tick=CanonicalDecimal.from_text(_filter_value(item, "PRICE_FILTER", "tickSize")), quantity_step=CanonicalDecimal.from_text(_filter_value(item, "LOT_SIZE", "stepSize")), diff --git a/qdl/api_v2/models.py b/qdl/api_v2/models.py index 9391149..ef1f9b1 100644 --- a/qdl/api_v2/models.py +++ b/qdl/api_v2/models.py @@ -1,12 +1,8 @@ from __future__ import annotations -from enum import StrEnum -from typing import Annotated, Literal - -from pydantic import BaseModel, ConfigDict, Field, model_validator +from pydantic import Field from qdl.query import ( - BarLifecycle, BarRevisionPolicy, ConsumerGrade, FeedType, @@ -14,10 +10,40 @@ RecoveryPolicy, StalePolicy, ) - - -class ClosedModel(BaseModel): - model_config = ConfigDict(extra="forbid", populate_by_name=True) +from qdl_sdk.models import ( + BarPayload, + BatchItemResponse, + BatchResponse, + BookDeltaPayload, + BookLevel, + BookSnapshotPayload, + ClosedModel, + ContractView, + DecimalValue, + FeedStatusResponse, + FundingRatePayload, + GapListResponse, + GapView, + InstrumentPageResponse, + InstrumentResponse, + InstrumentView, + MarkIndexPricePayload, + MarketDataView, + OpenInterestPayload, + ProblemDetails, + QualityView, + QuantityUnit, + QuotePayload, + ReadinessItemResponse, + ReadinessResponse, + SnapshotResponse, + SourceView, + SystemReadinessSummary, + TickerPayload, + TradeIdentityKind, + TradePayload, + WarmupResponse, +) class RequirementModel(ClosedModel): @@ -40,333 +66,3 @@ class BatchRequirementModel(ClosedModel): consumer_id: str = Field(min_length=1, max_length=200) requirements: list[RequirementModel] = Field(min_length=1, max_length=100) require_all: bool = True - - -class ProblemDetails(ClosedModel): - type: str - title: str - status: int - code: str - detail: str - request_id: str - retryable: bool - retry_after_ms: int | None = None - instrument_uid: str | None = None - quality_state: str | None = None - - -class DecimalValue(ClosedModel): - coefficient: str = Field(pattern=r"^-?(0|[1-9][0-9]*)$") - scale: int = Field(ge=-38, le=38) - source_text: str = Field(min_length=1, max_length=128) - - -class TradeIdentityKind(StrEnum): - NATIVE = "NATIVE" - DERIVED_RAW_CAPTURE = "DERIVED_RAW_CAPTURE" - - -class QuantityUnit(StrEnum): - BASE_ASSET = "BASE_ASSET" - QUOTE_ASSET = "QUOTE_ASSET" - CONTRACT = "CONTRACT" - SHARE = "SHARE" - - -class SourceView(ClosedModel): - venue: str = Field(min_length=1, max_length=40) - provider: str = Field(min_length=1, max_length=80) - source_id: str = Field(min_length=1, max_length=120) - source_role: Literal["PRIMARY", "SECONDARY", "REFERENCE", "BACKFILL"] - authoritative: bool - - -class QualityView(ClosedModel): - state: Literal[ - "DISABLED", "STARTING", "CONNECTING", "SUBSCRIBING", "SYNCING", - "LIVE", "DEGRADED", "GAPPED", "RESYNCING", "STALE", "OFFLINE", - "HALTED", "MARKET_CLOSED", - ] - freshness_ms: int = Field(ge=0) - gap_open: bool - complete: bool - execution_eligible: bool - policy_id: str = Field(min_length=1, max_length=200) - flags: list[str] - - -class ContractView(ClosedModel): - schema_digest: str = Field(pattern=r"^[0-9a-f]{64}$") - contract_version: str = Field(min_length=1, max_length=40) - normalizer_version: str = Field(min_length=1, max_length=80) - adapter_version: str = Field(min_length=1, max_length=80) - instrument_catalog_revision: int = Field(ge=1) - source_policy_revision: int = Field(ge=1) - authority_revision: int = Field(ge=1) - config_revision: int = Field(ge=1) - correlation_id: str = Field(min_length=1, max_length=200) - - -class TradePayload(ClosedModel): - feed: Literal[FeedType.TRADE] = FeedType.TRADE - native_trade_id: str = Field(min_length=1, max_length=200) - price: DecimalValue - quantity: DecimalValue - quantity_unit: QuantityUnit - aggressor_side: Literal["BUY", "SELL", "UNKNOWN"] - identity_kind: TradeIdentityKind - is_block_trade: bool = False - is_buyer_maker: bool = False - - -class QuotePayload(ClosedModel): - feed: Literal[FeedType.QUOTE] = FeedType.QUOTE - bid_price: DecimalValue - bid_quantity: DecimalValue - ask_price: DecimalValue - ask_quantity: DecimalValue - quantity_unit: QuantityUnit - level: int = Field(default=1, ge=1) - - -class BarPayload(ClosedModel): - feed: Literal[FeedType.BAR] = FeedType.BAR - interval: str = Field(min_length=1, max_length=20) - open_time_ns: int = Field(gt=0) - close_time_ns: int = Field(gt=0) - open: DecimalValue - high: DecimalValue - low: DecimalValue - close: DecimalValue - volume: DecimalValue - volume_unit: QuantityUnit - base_volume: DecimalValue | None = None - quote_volume: DecimalValue | None = None - contract_volume: DecimalValue | None = None - trade_count: int = Field(default=0, ge=0) - lifecycle: BarLifecycle - revision: int = Field(ge=0) - origin: Literal["VENUE_NATIVE", "AGGREGATED", "BACKFILLED", "RECONCILED"] - supersedes_event_id: str | None = None - - @model_validator(mode="after") - def validate_lifecycle(self): - if self.lifecycle is BarLifecycle.UNSPECIFIED: - raise ValueError("bar lifecycle cannot be UNSPECIFIED") - if self.lifecycle is BarLifecycle.REVISED and not self.supersedes_event_id: - raise ValueError("revised bar must identify the superseded event") - if self.close_time_ns <= self.open_time_ns: - raise ValueError("bar close time must be after open time") - return self - - -class BookLevel(ClosedModel): - side: Literal["BID", "ASK"] - price: DecimalValue - quantity: DecimalValue - quantity_unit: QuantityUnit - order_count: int = Field(default=0, ge=0) - - -class BookSnapshotPayload(ClosedModel): - feed: Literal[FeedType.BOOK_SNAPSHOT] = FeedType.BOOK_SNAPSHOT - native_sequence: str = Field(min_length=1, max_length=200) - checksum: str | None = Field(default=None, max_length=200) - levels: list[BookLevel] - depth: int = Field(ge=1) - - -class BookDeltaPayload(ClosedModel): - feed: Literal[FeedType.BOOK_DELTA] = FeedType.BOOK_DELTA - native_sequence_start: str = Field(min_length=1, max_length=200) - native_sequence_end: str = Field(min_length=1, max_length=200) - snapshot_sequence: str = Field(min_length=1, max_length=200) - checksum: str | None = Field(default=None, max_length=200) - updates: list[BookLevel] - reset: bool = False - - -class FundingRatePayload(ClosedModel): - feed: Literal[FeedType.FUNDING_RATE] = FeedType.FUNDING_RATE - rate: DecimalValue - funding_time_ns: int = Field(gt=0) - next_funding_time_ns: int | None = Field(default=None, gt=0) - - -class OpenInterestPayload(ClosedModel): - feed: Literal[FeedType.OPEN_INTEREST] = FeedType.OPEN_INTEREST - quantity: DecimalValue - quantity_unit: QuantityUnit - notional: DecimalValue | None = None - - -class MarkIndexPricePayload(ClosedModel): - feed: Literal[FeedType.MARK_INDEX_PRICE] = FeedType.MARK_INDEX_PRICE - mark_price: DecimalValue - index_price: DecimalValue - - -class TickerPayload(ClosedModel): - feed: Literal[FeedType.TICKER] = FeedType.TICKER - last_price: DecimalValue - last_quantity: DecimalValue | None = None - open_24h: DecimalValue | None = None - high_24h: DecimalValue | None = None - low_24h: DecimalValue | None = None - volume_24h: DecimalValue | None = None - last_quantity_unit: QuantityUnit | None = None - volume_24h_unit: QuantityUnit | None = None - - @model_validator(mode="after") - def quantity_units_match_optional_values(self): - if (self.last_quantity is None) != (self.last_quantity_unit is None): - raise ValueError("ticker last quantity and unit must be present together") - if (self.volume_24h is None) != (self.volume_24h_unit is None): - raise ValueError("ticker 24h volume and unit must be present together") - return self - - -MarketPayload = Annotated[ - TradePayload | QuotePayload | BarPayload | BookSnapshotPayload | BookDeltaPayload - | FundingRatePayload | OpenInterestPayload | MarkIndexPricePayload | TickerPayload, - Field(discriminator="feed"), -] - - -class MarketDataView(ClosedModel): - instrument_uid: str - instrument_id: str - instrument_revision: int = Field(ge=1) - feed: FeedType - interval: str | None - observed_at_ns: int = Field(gt=0) - revision: int = Field(ge=0) - payload: MarketPayload - source: SourceView - quality: QualityView - contract: ContractView - cursor: str | None = None - snapshot_id: str | None = None - watermark_offset: int = Field(default=0, ge=0) - - @model_validator(mode="after") - def feed_matches_payload(self): - if self.feed is FeedType.UNSPECIFIED or self.payload.feed is not self.feed: - raise ValueError("market-data envelope feed does not match its payload") - if self.feed is FeedType.BAR: - if not self.interval or self.payload.interval != self.interval: - raise ValueError("bar envelope and payload interval must match") - if self.payload.revision != self.revision: - raise ValueError("bar envelope and payload revision must match") - elif self.interval is not None: - raise ValueError("interval is valid only for bar market data") - return self - - -class SnapshotResponse(ClosedModel): - contract_schema: str = Field("qdl.marketdata.snapshot.v2", alias="schema") - request_id: str - data: MarketDataView - - -class InstrumentView(ClosedModel): - instrument_uid: str - instrument_id: str - venue: str - market: str - product_type: str - canonical_symbol: str - metadata_revision: int = Field(ge=1) - asset_class: str - native_symbol: str - status: str - - -class InstrumentPageResponse(ClosedModel): - contract_schema: str = Field("qdl.instruments.page.v2", alias="schema") - items: list[InstrumentView] - next_cursor: str | None = None - - -class InstrumentResponse(InstrumentView): - contract_schema: str = Field("qdl.instrument.v2", alias="schema") - - -class WarmupResponse(ClosedModel): - contract_schema: str = Field("qdl.marketdata.warmup.v2", alias="schema") - request_id: str - snapshot_id: str = Field(min_length=1) - data_as_of_ns: int = Field(gt=0) - stream_cursor: str = Field(min_length=1) - watermark_offset: int = Field(ge=0) - coverage: str - count: int - data: list[MarketDataView] - - -class BatchItemResponse(ClosedModel): - instrument_uid: str - status: str - data: WarmupResponse | None = None - problem: ProblemDetails | None = None - - -class BatchResponse(ClosedModel): - contract_schema: str = Field("qdl.marketdata.batch.v2", alias="schema") - request_id: str - partial: bool - success_count: int - error_count: int - results: list[BatchItemResponse] - - @model_validator(mode="after") - def counts_match(self): - if self.success_count + self.error_count != len(self.results): - raise ValueError("batch counts do not match results") - if self.partial != (self.error_count > 0): - raise ValueError("batch partial flag does not match errors") - return self - - -class ReadinessItemResponse(ClosedModel): - instrument_uid: str - status: str - quality: QualityView | None = None - problem: ProblemDetails | None = None - - -class ReadinessResponse(ClosedModel): - contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") - request_id: str - ready: bool - authority: str = "V1" - results: list[ReadinessItemResponse] - - -class FeedStatusResponse(ClosedModel): - contract_schema: str = Field("qdl.feed-status.v2", alias="schema") - instrument_uid: str - feed: FeedType - quality: QualityView - - -class GapView(ClosedModel): - gap_id: str - instrument_uid: str - feed: FeedType - source_id: str - expected_sequence: str - observed_sequence: str - detected_at_ns: int - - -class GapListResponse(ClosedModel): - contract_schema: str = Field("qdl.data-quality.gaps.v2", alias="schema") - items: list[GapView] - - -class SystemReadinessSummary(ClosedModel): - contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") - status: str - authority: str - v2_consumer_activation: str diff --git a/qdl/control/authority_outbox.py b/qdl/control/authority_outbox.py new file mode 100644 index 0000000..04a742f --- /dev/null +++ b/qdl/control/authority_outbox.py @@ -0,0 +1,399 @@ +from __future__ import annotations + +import asyncio +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import datetime, timezone +import hashlib +import json +from typing import Any, Protocol + + +_CONTROL_SCHEMA = "qdl.authority-control-event.v1" +_OUTBOX_SCHEMA = "qdl.authority-outbox-event.v1" +_PHASE92_STATES = { + "RUST_CANARY", "RUST_PRIMARY", "BLOCKED", + "ROLLBACK_PENDING", "PYTHON_PRIMARY", +} + + +def _timestamp_ns(value: Any) -> int: + if isinstance(value, datetime): + parsed = value + elif isinstance(value, str): + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + else: + raise ValueError("authority timestamp is missing or invalid") + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return int(parsed.timestamp() * 1_000_000_000) + + +def _uuid(value: Any, field: str) -> str: + text = str(value or "") + parts = text.split("-") + if [len(part) for part in parts] != [8, 4, 4, 4, 12] or any( + not all(character in "0123456789abcdefABCDEF" for character in part) + for part in parts + ): + raise ValueError(f"{field} is not a UUID") + return text.lower() + + +def _digest(payload: Mapping[str, Any]) -> str: + encoded = json.dumps( + payload, ensure_ascii=True, separators=(",", ":") + ).encode() + return hashlib.sha256(encoded).hexdigest() + + +def _checkpoint(raw: Mapping[str, Any] | None) -> dict[str, Any] | None: + if raw is None: + return None + result = { + "schema": "qdl.terminal-owner-checkpoint.v1", + "checkpoint_id": _uuid(raw.get("checkpoint_id"), "checkpoint_id"), + "slice_id": str(raw.get("slice_id") or ""), + "owner_id": str(raw.get("owner_id") or ""), + "authority_revision": int(raw.get("authority_revision") or 0), + "lease_epoch": int(raw.get("lease_epoch") or 0), + "partition_plan_epoch": int(raw.get("partition_plan_epoch") or 0), + "source_session_id": str(raw.get("source_session_id") or ""), + "connection_generation": int(raw.get("connection_generation") or 0), + "terminal_watermark": int(raw.get("terminal_watermark") or 0), + "terminal_event_id": str(raw.get("terminal_event_id") or ""), + "terminal_payload_sha256": str(raw.get("terminal_payload_sha256") or ""), + "candidate_digest": str(raw.get("candidate_digest") or ""), + "committed_at_ns": _timestamp_ns(raw.get("committed_at")), + } + if any( + not result[key] + for key in ("slice_id", "owner_id", "source_session_id", "terminal_event_id") + ) or any( + result[key] <= 0 + for key in ( + "authority_revision", "lease_epoch", "partition_plan_epoch", + "connection_generation", "committed_at_ns", + ) + ) or any( + len(result[key]) != 64 + for key in ("terminal_payload_sha256", "candidate_digest") + ): + raise ValueError("terminal checkpoint is incomplete") + return result + + +def _handoff( + raw: Mapping[str, Any] | None, + checkpoint: Mapping[str, Any] | None, +) -> dict[str, Any] | None: + if raw is None: + return None + if checkpoint is None: + raise ValueError("authority handoff has no terminal checkpoint") + result = { + "schema": "qdl.accepted-authority-handoff.v1", + "handoff_id": _uuid(raw.get("handoff_id"), "handoff_id"), + "direction": str(raw.get("direction") or ""), + "checkpoint_digest": _digest(checkpoint), + "slice_id": str(raw.get("slice_id") or ""), + "old_owner_id": str(raw.get("old_owner_id") or ""), + "new_owner_id": str(raw.get("new_owner_id") or ""), + "expected_state": str(raw.get("expected_state") or ""), + "new_state": str(raw.get("new_state") or ""), + "expected_authority_revision": int(raw.get("expected_authority_revision") or 0), + "new_authority_revision": int(raw.get("new_authority_revision") or 0), + "expected_lease_epoch": int(raw.get("expected_lease_epoch") or 0), + "new_lease_epoch": int(raw.get("new_lease_epoch") or 0), + "partition_plan_epoch": int(raw.get("partition_plan_epoch") or 0), + "terminal_watermark": int(raw.get("terminal_watermark") or 0), + "first_new_watermark": int(raw.get("first_new_watermark") or 0), + "overlap_start_watermark": int(raw.get("overlap_start_watermark") or 0), + "overlap_end_watermark": int(raw.get("overlap_end_watermark") or 0), + "old_event_count": int(raw.get("old_event_count") or 0), + "new_event_count": int(raw.get("new_event_count") or 0), + "semantic_mismatches": int(raw.get("semantic_mismatches") or 0), + "open_gaps": int(raw.get("open_gaps") or 0), + "candidate_digest": str(raw.get("candidate_digest") or ""), + "prerequisite_bundle_id": _uuid( + raw.get("prerequisite_bundle_id"), "prerequisite_bundle_id" + ), + "approved_by": str(raw.get("approved_by") or ""), + "approved_at_ns": _timestamp_ns(raw.get("approved_at")), + "expires_at_ns": _timestamp_ns(raw.get("expires_at")), + } + expected_digest = str(raw.get("handoff_sha256") or "") + actual_digest = _digest(result) + if len(expected_digest) != 64 or actual_digest != expected_digest: + raise ValueError("accepted handoff digest differs from immutable DB evidence") + if ( + result["new_authority_revision"] != result["expected_authority_revision"] + 1 + or result["new_lease_epoch"] <= result["expected_lease_epoch"] + or result["first_new_watermark"] != result["terminal_watermark"] + 1 + or result["semantic_mismatches"] != 0 + or result["open_gaps"] != 0 + or result["old_event_count"] != result["new_event_count"] + ): + raise ValueError("accepted handoff boundary is invalid") + return result + + +def _authority( + raw: Mapping[str, Any], + transition: Mapping[str, Any], + handoff: Mapping[str, Any] | None, +) -> dict[str, Any] | None: + state = str(raw.get("state") or "") + if state not in _PHASE92_STATES: + return None + terminal = raw.get("terminal_watermark") + start_watermark = int(terminal or 0) + primary = state in {"RUST_PRIMARY", "PYTHON_PRIMARY"} + if primary and handoff is None: + raise ValueError("primary authority requires accepted handoff evidence") + approved_by = raw.get("approved_by") + approved_at = raw.get("approved_at") + hold_until = raw.get("hold_until") + if state == "PYTHON_PRIMARY" and handoff is not None: + approved_by = handoff["approved_by"] + approved_at = handoff["approved_at_ns"] + hold_until = handoff["expires_at_ns"] + approved_at_ns = ( + int(approved_at) + if isinstance(approved_at, int) + else _timestamp_ns(approved_at) + if approved_at is not None + else None + ) + hold_until_ns = ( + int(hold_until) + if isinstance(hold_until, int) + else _timestamp_ns(hold_until) + if hold_until is not None + else None + ) + active = state in {"RUST_CANARY", "RUST_PRIMARY", "PYTHON_PRIMARY"} + result = { + "schema": "qdl.authority-record.v3", + "slice_id": str(raw.get("slice_id") or ""), + "state": state, + "owner_id": str(raw.get("owner_id") or ""), + "authority_revision": int(raw.get("authority_revision") or 0), + "lease_epoch": int(raw.get("lease_epoch") or 0), + "partition_plan_epoch": int(raw.get("partition_plan_epoch") or 0), + "candidate_digest": str(raw.get("candidate_digest") or ""), + "prerequisite_bundle_id": ( + str(raw["prerequisite_bundle_id"]) + if raw.get("prerequisite_bundle_id") is not None + else None + ), + "start_watermark": start_watermark, + "terminal_watermark": start_watermark if primary else None, + "previous_owner_id": handoff["old_owner_id"] if primary else None, + "handoff_digest": _digest(handoff) if primary else None, + "approved_by": str(approved_by) if active and approved_by is not None else None, + "approved_at_ns": approved_at_ns if active else None, + "hold_until_ns": hold_until_ns if active else None, + "public_write_allowed": primary, + "legacy_write_allowed": primary, + } + if result["authority_revision"] != int(transition.get("new_revision") or 0): + raise ValueError("authority row and transition revision differ") + if active and ( + not result["approved_by"] + or not result["approved_at_ns"] + or not result["hold_until_ns"] + or result["hold_until_ns"] <= result["approved_at_ns"] + ): + raise ValueError("active authority approval window is invalid") + if state in {"RUST_CANARY", "RUST_PRIMARY"} and not result["prerequisite_bundle_id"]: + raise ValueError("Rust canary/primary requires prerequisite bundle") + return result + + +def build_authority_control_event(payload: Mapping[str, Any]) -> dict[str, Any]: + if payload.get("schema") != _OUTBOX_SCHEMA: + raise ValueError("unsupported authority outbox payload schema") + event_id = _uuid(payload.get("event_id"), "event_id") + transition = payload.get("transition") + authority_row = payload.get("authority") + if not isinstance(transition, Mapping) or not isinstance(authority_row, Mapping): + raise ValueError("authority outbox snapshot is incomplete") + checkpoint = _checkpoint( + payload.get("checkpoint") if isinstance(payload.get("checkpoint"), Mapping) else None + ) + handoff = _handoff( + payload.get("handoff") if isinstance(payload.get("handoff"), Mapping) else None, + checkpoint, + ) + authority = _authority(authority_row, transition, handoff) + slice_id = str(authority_row.get("slice_id") or "") + revision = int(authority_row.get("authority_revision") or 0) + if ( + not slice_id + or revision <= 0 + or str(transition.get("transition_id") or "").lower() != event_id + or str(transition.get("slice_id") or "") != slice_id + ): + raise ValueError("authority outbox event identity is inconsistent") + return { + "schema": _CONTROL_SCHEMA, + "event_id": event_id, + "slice_id": slice_id, + "authority_revision": revision, + "database_state": str(authority_row.get("state") or ""), + "authority": authority, + "checkpoint": checkpoint, + "handoff": handoff, + } + + +@dataclass(frozen=True, slots=True) +class ClaimedAuthorityEvent: + event_id: str + payload: Mapping[str, Any] + attempts: int = 1 + + +@dataclass(frozen=True, slots=True) +class BrokerAck: + topic: str + partition: int + offset: int + + +class AuthorityOutboxRepository(Protocol): + async def claim(self, lock_owner: str, limit: int) -> list[ClaimedAuthorityEvent]: ... + async def complete(self, event_id: str, lock_owner: str, ack: BrokerAck) -> None: ... + async def retry(self, event_id: str, lock_owner: str, error: str, delay_seconds: float) -> None: ... + + +class AuthorityPublisher(Protocol): + async def publish(self, *, key: str, event_id: str, payload: bytes) -> BrokerAck: ... + + +class AuthorityOutboxDispatcher: + def __init__( + self, + *, + repository: AuthorityOutboxRepository, + publisher: AuthorityPublisher, + lock_owner: str, + batch_size: int = 20, + ) -> None: + if not lock_owner.strip() or not 1 <= batch_size <= 100: + raise ValueError("authority dispatcher identity/batch bounds are invalid") + self.repository = repository + self.publisher = publisher + self.lock_owner = lock_owner + self.batch_size = batch_size + + async def dispatch_once(self) -> int: + claimed = await self.repository.claim(self.lock_owner, self.batch_size) + published = 0 + for item in claimed: + try: + event = build_authority_control_event(item.payload) + encoded = json.dumps( + event, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + ack = await self.publisher.publish( + key=event["slice_id"], event_id=item.event_id, payload=encoded + ) + await self.repository.complete(item.event_id, self.lock_owner, ack) + published += 1 + except Exception as error: + delay = min(60.0, 0.5 * 2 ** min(7, max(0, item.attempts - 1))) + await self.repository.retry( + item.event_id, self.lock_owner, str(error), delay + ) + return published + + +class AsyncpgAuthorityOutboxRepository: + def __init__(self, pool: Any) -> None: + self._pool = pool + + @classmethod + async def connect(cls, dsn: str, *, min_size: int = 1, max_size: int = 4): + try: + import asyncpg + except ImportError as error: + raise RuntimeError("authority dispatcher requires asyncpg") from error + pool = await asyncpg.create_pool( + dsn=dsn, min_size=min_size, max_size=max_size, + command_timeout=15, server_settings={"application_name": "qdl-authority-outbox"}, + ) + return cls(pool) + + async def claim(self, lock_owner: str, limit: int) -> list[ClaimedAuthorityEvent]: + rows = await self._pool.fetch( + "SELECT event_id, payload FROM qdl_claim_authority_outbox($1, $2)", + lock_owner, limit, + ) + return [ + ClaimedAuthorityEvent( + str(row["event_id"]), row["payload"], int(row["attempts"]) + ) + for row in rows + ] + + async def complete(self, event_id: str, lock_owner: str, ack: BrokerAck) -> None: + await self._pool.execute( + "SELECT qdl_complete_authority_outbox($1::uuid, $2, $3, $4, $5)", + event_id, lock_owner, ack.topic, ack.partition, ack.offset, + ) + + async def retry(self, event_id: str, lock_owner: str, error: str, delay_seconds: float) -> None: + await self._pool.execute( + "SELECT qdl_retry_authority_outbox($1::uuid, $2, $3, make_interval(secs => $4))", + event_id, lock_owner, error, delay_seconds, + ) + + async def close(self) -> None: + await self._pool.close() + + +class KafkaAuthorityPublisher: + def __init__(self, config: Mapping[str, Any], *, topic: str, timeout_seconds: float = 15.0): + if not topic.strip() or timeout_seconds <= 0: + raise ValueError("authority Kafka topic/timeout is invalid") + from confluent_kafka import Producer + values = dict(config) + values.update({ + "acks": "all", + "enable.idempotence": True, + "compression.type": "zstd", + "max.in.flight.requests.per.connection": 5, + }) + self._producer = Producer(values) + self._topic = topic + self._timeout = timeout_seconds + + async def publish(self, *, key: str, event_id: str, payload: bytes) -> BrokerAck: + loop = asyncio.get_running_loop() + completed: asyncio.Future[BrokerAck] = loop.create_future() + + def delivery(error, message) -> None: + if error is not None: + loop.call_soon_threadsafe(completed.set_exception, RuntimeError(str(error))) + else: + loop.call_soon_threadsafe( + completed.set_result, + BrokerAck(message.topic(), message.partition(), message.offset()), + ) + + self._producer.produce( + self._topic, + key=key.encode(), + value=payload, + headers={"qdl-event-id": event_id.encode()}, + on_delivery=delivery, + ) + deadline = loop.time() + self._timeout + while not completed.done(): + self._producer.poll(0) + if loop.time() >= deadline: + raise TimeoutError("authority Kafka durable ACK timed out") + await asyncio.sleep(0.01) + return await completed diff --git a/qdl/control/cutover_packet.py b/qdl/control/cutover_packet.py new file mode 100644 index 0000000..9f04f5f --- /dev/null +++ b/qdl/control/cutover_packet.py @@ -0,0 +1,272 @@ +from __future__ import annotations + +import hashlib +import json +import time +import uuid +from dataclasses import dataclass +from typing import Any, Mapping, Sequence + + +_SCHEMA = "qdl.c3.authority-cutover-packet.v1" +_PAIRS = { + "SHADOW_VALIDATE": ("RUST_SHADOW", "VALIDATING"), + "CANARY": ("VALIDATING", "RUST_CANARY"), + "PRIMARY": ("RUST_CANARY", "RUST_PRIMARY"), + "BLOCK_CANARY": ("RUST_CANARY", "BLOCKED"), + "BLOCK_PRIMARY": ("RUST_PRIMARY", "BLOCKED"), + "ROLLBACK_PENDING": ("BLOCKED", "ROLLBACK_PENDING"), + "PYTHON_RESTORE": ("ROLLBACK_PENDING", "PYTHON_PRIMARY"), +} +_HANDOFF_STAGES = {"PRIMARY", "PYTHON_RESTORE"} +_ACTIVE_STAGES = {"CANARY", "PRIMARY"} + + +def _digest(value: Mapping[str, Any]) -> str: + encoded = json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return hashlib.sha256(encoded).hexdigest() + + +def _uuid(value: object, name: str) -> str: + try: + return str(uuid.UUID(str(value))) + except ValueError as error: + raise ValueError(f"{name} must be a UUID") from error + + +def _hex(value: object, name: str, *, image: bool = False) -> str: + text = str(value) + prefix = "sha256:" if image else "" + raw = text.removeprefix(prefix) if prefix else text + if (image and not text.startswith(prefix)) or len(raw) != 64 or any( + char not in "0123456789abcdef" for char in raw + ): + raise ValueError(f"{name} must be a SHA-256 digest") + return text + + +def _exact(value: Mapping[str, Any], fields: set[str], name: str) -> None: + if set(value) != fields: + raise ValueError(f"{name} fields are incomplete or unknown") + + +@dataclass(frozen=True, slots=True) +class CutoverSlice: + transition_id: str + handoff_id: str | None + slice_id: str + expected_state: str + expected_revision: int + expected_owner_id: str + expected_lease_epoch: int + partition_plan_epoch: int + new_state: str + new_owner_id: str + new_lease_epoch: int + terminal_watermark: int | None + prerequisite_bundle_id: str | None + hold_until: str | None + reason: str + + @classmethod + def parse(cls, raw: Mapping[str, Any], *, stage: str) -> "CutoverSlice": + _exact( + raw, + { + "transition_id", "handoff_id", "slice_id", "expected_state", + "expected_revision", "expected_owner_id", "expected_lease_epoch", + "partition_plan_epoch", "new_state", "new_owner_id", + "new_lease_epoch", "terminal_watermark", + "prerequisite_bundle_id", "hold_until", "reason", + }, + "cutover slice", + ) + expected_pair = _PAIRS[stage] + if (raw["expected_state"], raw["new_state"]) != expected_pair: + raise ValueError("cutover slice state pair differs from packet stage") + handoff_id = raw["handoff_id"] + prerequisite = raw["prerequisite_bundle_id"] + if stage in _HANDOFF_STAGES: + handoff_id = _uuid(handoff_id, "handoff_id") + elif handoff_id is not None: + raise ValueError("non-handoff stage cannot carry handoff_id") + if stage in _ACTIVE_STAGES: + prerequisite = _uuid(prerequisite, "prerequisite_bundle_id") + if raw["terminal_watermark"] is None or int(raw["terminal_watermark"]) < 0: + raise ValueError("active Rust stage requires terminal watermark") + if not str(raw["hold_until"] or "").strip(): + raise ValueError("active Rust stage requires hold_until") + elif prerequisite is not None or raw["hold_until"] is not None: + raise ValueError("inactive stage cannot carry prerequisite/hold") + positive = ( + int(raw["expected_revision"]), int(raw["expected_lease_epoch"]), + int(raw["partition_plan_epoch"]), int(raw["new_lease_epoch"]), + ) + if any(value <= 0 for value in positive): + raise ValueError("cutover slice revisions/epochs must be positive") + if ( + raw["expected_owner_id"] != raw["new_owner_id"] + and int(raw["new_lease_epoch"]) <= int(raw["expected_lease_epoch"]) + ): + raise ValueError("authority owner change requires newer lease") + if any( + not str(value).strip() + for value in ( + raw["slice_id"], raw["expected_owner_id"], + raw["new_owner_id"], raw["reason"], + ) + ): + raise ValueError("cutover slice identity/reason is incomplete") + return cls( + transition_id=_uuid(raw["transition_id"], "transition_id"), + handoff_id=handoff_id, + slice_id=str(raw["slice_id"]), + expected_state=str(raw["expected_state"]), + expected_revision=int(raw["expected_revision"]), + expected_owner_id=str(raw["expected_owner_id"]), + expected_lease_epoch=int(raw["expected_lease_epoch"]), + partition_plan_epoch=int(raw["partition_plan_epoch"]), + new_state=str(raw["new_state"]), + new_owner_id=str(raw["new_owner_id"]), + new_lease_epoch=int(raw["new_lease_epoch"]), + terminal_watermark=( + int(raw["terminal_watermark"]) + if raw["terminal_watermark"] is not None else None + ), + prerequisite_bundle_id=prerequisite, + hold_until=( + str(raw["hold_until"]) if raw["hold_until"] is not None else None + ), + reason=str(raw["reason"]), + ) + + +@dataclass(frozen=True, slots=True) +class AuthorityCutoverPacket: + raw: Mapping[str, Any] + packet_id: str + stage: str + actor: str + candidate_digest: str + artifact_image_digest: str + contract_digest: str + partition_plan_digest: str + route_manifest_digest: str + slices: tuple[CutoverSlice, ...] + + @classmethod + def parse( + cls, raw: Mapping[str, Any], *, now_ns: int | None = None + ) -> "AuthorityCutoverPacket": + _exact( + raw, + { + "schema", "packet_id", "stage", "issued_at_ns", "expires_at_ns", + "actor", "change_ticket", "candidate_digest", + "artifact_image_digest", "contract_digest", + "partition_plan_digest", "route_manifest_digest", + "consumer_route", "evidence", "slices", + }, + "cutover packet", + ) + if raw["schema"] != _SCHEMA or raw["stage"] not in _PAIRS: + raise ValueError("cutover packet schema/stage is unsupported") + current = time.time_ns() if now_ns is None else now_ns + issued = int(raw["issued_at_ns"]) + expires = int(raw["expires_at_ns"]) + if issued <= 0 or expires <= issued or not issued <= current < expires: + raise ValueError("cutover packet approval window is invalid") + if not str(raw["actor"]).strip() or not str(raw["change_ticket"]).strip(): + raise ValueError("cutover operator/change ticket is required") + route = raw["consumer_route"] + evidence = raw["evidence"] + if not isinstance(route, Mapping) or not isinstance(evidence, Mapping): + raise ValueError("cutover route/evidence must be objects") + _exact( + route, + { + "consumer_id", "expected_route", "new_route", "rollback_route", + "rollback_command", + }, + "consumer route", + ) + rollback = route["rollback_command"] + if ( + route["consumer_id"] != "trading-system" + or route["rollback_route"] != "V1" + or not isinstance(rollback, Sequence) + or isinstance(rollback, (str, bytes)) + or not rollback + or any(not isinstance(item, str) or not item.strip() for item in rollback) + ): + raise ValueError("cutover Trading System rollback route is invalid") + _exact( + evidence, + { + "provider_provenance", "semantic_mismatches", "open_gaps", + "duplicate_external_effects", "consumer_errors", + }, + "cutover evidence", + ) + if ( + evidence["provider_provenance"] != "REAL" + or any( + int(evidence[name]) != 0 + for name in ( + "semantic_mismatches", "open_gaps", + "duplicate_external_effects", "consumer_errors", + ) + ) + ): + raise ValueError("cutover evidence is not clean/authentic") + values = raw["slices"] + if not isinstance(values, list) or not 1 <= len(values) <= 32: + raise ValueError("cutover packet requires 1..32 slices") + slices = tuple( + CutoverSlice.parse(value, stage=str(raw["stage"])) for value in values + ) + identities = [item.slice_id for item in slices] + if len(identities) != len(set(identities)): + raise ValueError("cutover packet slice IDs must be unique") + packet = cls( + raw=dict(raw), + packet_id=_uuid(raw["packet_id"], "packet_id"), + stage=str(raw["stage"]), + actor=str(raw["actor"]), + candidate_digest=_hex(raw["candidate_digest"], "candidate_digest"), + artifact_image_digest=_hex( + raw["artifact_image_digest"], "artifact_image_digest", image=True + ), + contract_digest=_hex(raw["contract_digest"], "contract_digest"), + partition_plan_digest=_hex( + raw["partition_plan_digest"], "partition_plan_digest" + ), + route_manifest_digest=_hex( + raw["route_manifest_digest"], "route_manifest_digest" + ), + slices=slices, + ) + return packet + + @property + def digest(self) -> str: + return _digest(self.raw) + + @property + def confirmation_token(self) -> str: + return f"APPLY_C3_{self.digest[:16]}" + + def plan(self) -> dict[str, Any]: + return { + "schema": "qdl.c3.authority-cutover-plan.v1", + "packet_id": self.packet_id, + "packet_digest": self.digest, + "confirmation_token": self.confirmation_token, + "stage": self.stage, + "slice_count": len(self.slices), + "slice_ids": [item.slice_id for item in self.slices], + "apply_requested": False, + "production_mutations": 0, + } diff --git a/qdl/runtime/production_catalog.py b/qdl/runtime/production_catalog.py new file mode 100644 index 0000000..5cf3ad6 --- /dev/null +++ b/qdl/runtime/production_catalog.py @@ -0,0 +1,453 @@ +from __future__ import annotations + +from dataclasses import dataclass, replace +import hashlib +import json +from pathlib import Path +import re +from typing import Any, Iterable, Mapping + +import yaml + +from qdl.adapters.binance_usdm import BinanceDiscovery, parse_exchange_info +from qdl.adapters.okx.instruments import parse_public_instrument +from qdl.domain.instrument import InstrumentRecord, InstrumentStatus, ProductType +from qdl.query import ConsumerGrade, FeedType +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import StableAcquisitionPlan + + +_DEMAND_SCHEMA = "qdl.v2.production-demand.v1" +_SOURCE_SCHEMA = "qdl.v2.stable-source-bindings.v1" +_ACQUISITION_SCHEMA = "qdl.v2.stable-acquisition-bindings.v1" +_SUPPORTED_MARKETS = { + ("BINANCE", "USDM", "PERPETUAL"), + ("OKX", "SWAP", "PERPETUAL"), + ("OKX", "SPOT", "SPOT"), +} +_SUPPORTED_FEEDS = {FeedType.TRADE, FeedType.QUOTE, FeedType.BAR} + + +@dataclass(frozen=True, slots=True, order=True) +class ProductionDemand: + consumer_id: str + consumer_grade: ConsumerGrade + venue: str + market: str + product_type: str + native_symbol: str + feed: FeedType + interval: str | None + source_policy_id: str + + @property + def requirement_key(self) -> tuple[str, str, str, str, FeedType, str | None]: + return ( + self.venue, + self.market, + self.product_type, + self.native_symbol, + self.feed, + self.interval, + ) + + +@dataclass(frozen=True, slots=True) +class ProductionDemandManifest: + revision: int + demands: tuple[ProductionDemand, ...] + source_paths: tuple[str, ...] + + @classmethod + def load_many(cls, paths: Iterable[str | Path]) -> "ProductionDemandManifest": + demands: list[ProductionDemand] = [] + revisions: list[int] = [] + normalized_paths: list[str] = [] + for raw_path in paths: + path = Path(raw_path).resolve() + payload = yaml.safe_load(path.read_text(encoding="utf-8")) + if not isinstance(payload, dict) or set(payload) != { + "schema", "revision", "consumers" + }: + raise ValueError("production demand manifest fields are invalid") + if payload["schema"] != _DEMAND_SCHEMA or int(payload["revision"]) < 1: + raise ValueError("production demand manifest schema/revision is invalid") + consumers = payload["consumers"] + if not isinstance(consumers, list) or not 1 <= len(consumers) <= 10_000: + raise ValueError("production demand manifest needs bounded consumers") + revisions.append(int(payload["revision"])) + normalized_paths.append(str(path)) + for consumer in consumers: + if not isinstance(consumer, dict) or set(consumer) != { + "consumer_id", "consumer_grade", "requirements" + }: + raise ValueError("production demand consumer fields are invalid") + consumer_id = str(consumer["consumer_id"]).strip() + grade = ConsumerGrade(str(consumer["consumer_grade"]).upper()) + requirements = consumer["requirements"] + if not consumer_id or not isinstance(requirements, list) or not requirements: + raise ValueError("production demand consumer is empty") + for requirement in requirements: + demands.append(cls._requirement(consumer_id, grade, requirement)) + if not demands: + raise ValueError("production demand set cannot be empty") + deduped: dict[tuple[str, str, str, str, FeedType, str | None], ProductionDemand] = {} + consumers_by_key: dict[tuple[str, str, str, str, FeedType, str | None], set[str]] = {} + for item in sorted(demands): + key = item.requirement_key + current = deduped.get(key) + if current is not None and current.source_policy_id != item.source_policy_id: + raise ValueError("one feed requirement cannot use conflicting source policies") + deduped.setdefault(key, item) + consumers_by_key.setdefault(key, set()).add(item.consumer_id) + # Consumer IDs are audit inputs but do not alter a canonical source binding. + del consumers_by_key + return cls( + revision=max(revisions), + demands=tuple(sorted(deduped.values())), + source_paths=tuple(sorted(normalized_paths)), + ) + + @staticmethod + def _requirement( + consumer_id: str, + grade: ConsumerGrade, + raw: Any, + ) -> ProductionDemand: + if not isinstance(raw, dict) or set(raw) != { + "venue", "market", "product_type", "native_symbol", + "feed", "interval", "source_policy_id", + }: + raise ValueError("production demand requirement fields are invalid") + venue = str(raw["venue"]).strip().upper() + market = str(raw["market"]).strip().upper() + product = str(raw["product_type"]).strip().upper() + native_symbol = str(raw["native_symbol"]).strip().upper() + source_policy = str(raw["source_policy_id"]).strip() + feed = FeedType(str(raw["feed"]).strip().upper()) + interval = str(raw["interval"]).strip() if raw["interval"] is not None else None + if (venue, market, product) not in _SUPPORTED_MARKETS: + raise ValueError("production demand market/product is not certified") + if feed not in _SUPPORTED_FEEDS: + raise ValueError("production demand feed is not certified") + if feed is FeedType.BAR and interval != "1m": + raise ValueError("production V2 BAR acquisition is currently certified for 1m") + if feed is not FeedType.BAR and interval is not None: + raise ValueError("interval is valid only for BAR demand") + if not native_symbol or not source_policy: + raise ValueError("production demand identity/source policy is incomplete") + return ProductionDemand( + consumer_id=consumer_id, + consumer_grade=grade, + venue=venue, + market=market, + product_type=product, + native_symbol=native_symbol, + feed=feed, + interval=interval, + source_policy_id=source_policy, + ) + + +@dataclass(frozen=True, slots=True) +class ProductionCatalogBundle: + source_catalog: dict[str, Any] + acquisition_plan: dict[str, Any] + provenance: dict[str, Any] + + def write(self, output_dir: str | Path) -> dict[str, str]: + target = Path(output_dir) + target.mkdir(parents=True, exist_ok=True) + source_path = target / "production-source-bindings.yaml" + acquisition_path = target / "production-acquisition-bindings.yaml" + provenance_path = target / "production-catalog-provenance.json" + source_path.write_text(yaml.safe_dump(self.source_catalog, sort_keys=False)) + acquisition_path.write_text(yaml.safe_dump(self.acquisition_plan, sort_keys=False)) + provenance_path.write_text( + json.dumps(self.provenance, indent=2, sort_keys=True) + "\n" + ) + catalog = StableSourceCatalog.load(source_path) + StableAcquisitionPlan.load(acquisition_path, catalog=catalog) + return { + "source_catalog": str(source_path), + "acquisition_plan": str(acquisition_path), + "provenance": str(provenance_path), + } + + +class ProductionCatalogBuilder: + def __init__( + self, + *, + catalog_revision: int, + source_policy_revision: int, + authority_revision: int, + canonical_stream: str = "md.canonical.v2", + raw_topic: str = "md.raw.stable.v1", + quarantine_topic: str = "md.quarantine.stable.v1", + ) -> None: + if min(catalog_revision, source_policy_revision, authority_revision) < 1: + raise ValueError("production catalog revisions must be positive") + topics = (canonical_stream, raw_topic, quarantine_topic) + if any(not item.strip() for item in topics) or len(set(topics)) != 3: + raise ValueError("production catalog topics must be unique") + self.catalog_revision = catalog_revision + self.source_policy_revision = source_policy_revision + self.authority_revision = authority_revision + self.canonical_stream = canonical_stream + self.raw_topic = raw_topic + self.quarantine_topic = quarantine_topic + + def build( + self, + *, + demand: ProductionDemandManifest, + binance_usdm: BinanceDiscovery | None, + okx_rows: Iterable[Mapping[str, str]], + previous_catalog: StableSourceCatalog | None = None, + metadata_provenance: Mapping[str, str] | None = None, + ) -> ProductionCatalogBundle: + metadata = self._metadata(binance_usdm, okx_rows) + previous = self._previous_records(previous_catalog) + selected: dict[str, InstrumentRecord] = {} + bindings: list[dict[str, Any]] = [] + acquisitions: list[dict[str, Any]] = [] + for item in demand.demands: + key = (item.venue, item.market, item.native_symbol) + try: + discovered = metadata[key] + except KeyError as error: + raise ValueError(f"authoritative metadata missing demanded instrument: {key}") from error + if discovered.identity.product_type.value != item.product_type: + raise ValueError("demanded product type differs from authoritative metadata") + if discovered.status is not InstrumentStatus.ACTIVE: + raise ValueError("demanded instrument is not active") + record = self._revisioned(discovered, previous.get(discovered.instrument_id)) + selected[record.instrument_id] = record + binding_id = self._binding_id(item) + bindings.append(self._source_binding(binding_id, item, record)) + acquisitions.append(self._acquisition(binding_id, item)) + source = { + "schema": _SOURCE_SCHEMA, + "canonical_stream": self.canonical_stream, + "catalog_revision": self.catalog_revision, + "source_policy_revision": self.source_policy_revision, + "authority_revision": self.authority_revision, + "instruments": [ + self._instrument(record) for record in sorted(selected.values(), key=lambda value: value.instrument_id) + ], + "bindings": sorted(bindings, key=lambda value: value["binding_id"]), + } + acquisition = { + "schema": _ACQUISITION_SCHEMA, + "revision": demand.revision, + "topics": { + "raw": self.raw_topic, + "canonical": self.canonical_stream, + "quarantine": self.quarantine_topic, + }, + "bindings": sorted(acquisitions, key=lambda value: value["binding_id"]), + } + encoded_source = yaml.safe_dump(source, sort_keys=True).encode() + encoded_acquisition = yaml.safe_dump(acquisition, sort_keys=True).encode() + provenance = { + "schema": "qdl.v2.production-catalog-provenance.v1", + "catalog_revision": self.catalog_revision, + "demand_revision": demand.revision, + "demand_sources": list(demand.source_paths), + "instrument_count": len(selected), + "binding_count": len(bindings), + "instrument_ids": sorted(selected), + "source_catalog_sha256": hashlib.sha256(encoded_source).hexdigest(), + "acquisition_plan_sha256": hashlib.sha256(encoded_acquisition).hexdigest(), + "metadata": dict(sorted((metadata_provenance or {}).items())), + "fabricated_metadata": False, + } + return ProductionCatalogBundle(source, acquisition, provenance) + + @staticmethod + def _metadata( + binance_usdm: BinanceDiscovery | None, + okx_rows: Iterable[Mapping[str, str]], + ) -> dict[tuple[str, str, str], InstrumentRecord]: + values: list[InstrumentRecord] = [] + if binance_usdm is not None: + values.extend(binance_usdm.records) + for raw in okx_rows: + record, _ = parse_public_instrument( + raw, metadata_revision=1, valid_from_ns=0 + ) + values.append(record) + result: dict[tuple[str, str, str], InstrumentRecord] = {} + for record in values: + key = ( + record.identity.venue, + record.identity.market, + record.native_symbol, + ) + if key in result: + raise ValueError(f"duplicate authoritative instrument metadata: {key}") + result[key] = record + return result + + @staticmethod + def _previous_records( + catalog: StableSourceCatalog | None, + ) -> dict[str, InstrumentRecord]: + if catalog is None: + return {} + return { + binding.instrument.instrument_id: binding.instrument + for binding in catalog.bindings + } + + @classmethod + def _revisioned( + cls, current: InstrumentRecord, previous: InstrumentRecord | None + ) -> InstrumentRecord: + if previous is None: + return replace(current, metadata_revision=1) + previous_payload = cls._instrument(previous) | {"metadata_revision": 0} + current_payload = cls._instrument(current) | {"metadata_revision": 0} + revision = ( + previous.metadata_revision + if previous_payload == current_payload + else previous.metadata_revision + 1 + ) + return replace(current, metadata_revision=revision) + + @staticmethod + def _instrument(record: InstrumentRecord) -> dict[str, Any]: + identity = record.identity + return { + "instrument_uid": identity.instrument_uid, + "instrument_id": identity.instrument_id, + "metadata_revision": record.metadata_revision, + "venue": identity.venue, + "market": identity.market, + "product_type": identity.product_type.value, + "canonical_symbol": identity.canonical_symbol, + "native_symbol": record.native_symbol, + "asset_class": record.asset_class.value, + "base_asset": record.base_asset, + "quote_asset": record.quote_asset, + "settlement_asset": record.settlement_asset, + "price_tick": record.price_tick.source_text, + "quantity_step": record.quantity_step.source_text, + "contract_multiplier": record.contract_multiplier.source_text, + "session_calendar_id": record.session_calendar_id, + "attributes": dict(sorted(record.attributes.items())), + } + + @staticmethod + def _binding_id(item: ProductionDemand) -> str: + symbol = re.sub(r"[^a-z0-9]+", "-", item.native_symbol.lower()).strip("-") + suffix = f"-{item.interval}" if item.interval else "" + return f"{item.venue.lower()}-{item.market.lower()}-{symbol}-{item.feed.value.lower()}{suffix}" + + def _source_binding( + self, + binding_id: str, + item: ProductionDemand, + record: InstrumentRecord, + ) -> dict[str, Any]: + if item.feed is FeedType.TRADE: + stale_after_ms = 15_000 + elif item.feed is FeedType.QUOTE: + stale_after_ms = 5_000 + else: + stale_after_ms = 180_000 + adapter = ( + "binance-rest/2.0.0" + if item.venue == "BINANCE" and item.feed is FeedType.BAR + else "binance-usdm/2.0.0" + if item.venue == "BINANCE" + else "okx-v5/2.0.0" + ) + compatibility = "NONE" + if item.venue == "BINANCE" and item.feed is FeedType.TRADE: + compatibility = "BINANCE_TRADE_MARKET_AND_GENERIC" + elif item.venue == "BINANCE" and item.feed is FeedType.BAR: + compatibility = "BINANCE_BAR_GENERIC" + return { + "binding_id": binding_id, + "instrument_uid": record.instrument_uid, + "feed": item.feed.value, + "interval": item.interval, + "source": { + "provider": f"{item.venue}_DIRECT", + "source_id": f"{binding_id}-primary-v2", + "source_role": "PRIMARY", + "source_policy_id": item.source_policy_id, + "authoritative": True, + "adapter_version": adapter, + "normalizer_version": "qdl-rust-core/2.0.0", + }, + "quality": { + "stale_after_ms": stale_after_ms, + "require_final_bar": item.feed is FeedType.BAR, + "continuous_calendar": True, + }, + "v1_compatibility": compatibility, + } + + @staticmethod + def _acquisition(binding_id: str, item: ProductionDemand) -> dict[str, Any]: + if item.venue == "BINANCE": + if item.feed is FeedType.TRADE: + mode, kind, channel, sequence = ( + "RUST_NATIVE", "binance_usdm_trade", + f"{item.native_symbol.lower()}@trade", "MONOTONIC", + ) + elif item.feed is FeedType.QUOTE: + mode, kind, channel, sequence = ( + "RUST_NATIVE", "binance_usdm_bbo", + f"{item.native_symbol.lower()}@bookTicker", "MONOTONIC", + ) + else: + mode, kind, channel, sequence = ( + "PYTHON_REST", "binance_usdm_rest_bar", + f"rest-klines/{item.interval}", "NONE", + ) + websocket = ( + "wss://fstream.binance.com/public/stream" + if mode == "RUST_NATIVE" else None + ) + business = None + else: + if item.feed is FeedType.TRADE: + mode, kind, channel, sequence = "RUST_NATIVE", "okx_trade", "trades", "MONOTONIC" + elif item.feed is FeedType.QUOTE: + mode, kind, channel, sequence = "RUST_NATIVE", "okx_bbo", "bbo-tbt", "NONE" + else: + mode, kind, channel, sequence = "PYTHON_REST", "okx_bar", "candle1m", "NONE" + websocket = "wss://ws.okx.com:8443/ws/v5/public" if mode == "RUST_NATIVE" else None + business = "wss://ws.okx.com:8443/ws/v5/business" if mode == "RUST_NATIVE" else None + return { + "binding_id": binding_id, + "mode": mode, + "runtime": item.venue, + "provider_kind": kind, + "native_channel": channel, + "sequence_policy": sequence, + "websocket_url": websocket, + "business_websocket_url": business, + } + + +def load_binance_exchange_info(path: str | Path) -> BinanceDiscovery: + payload = json.loads(Path(path).read_text(encoding="utf-8")) + if not isinstance(payload, dict): + raise ValueError("Binance exchangeInfo capture must be an object") + return parse_exchange_info(payload, valid_from_ns=0) + + +def load_okx_instruments(path: str | Path) -> list[Mapping[str, str]]: + payload = json.loads(Path(path).read_text(encoding="utf-8")) + if isinstance(payload, dict): + if str(payload.get("code")) != "0" or not isinstance(payload.get("data"), list): + raise ValueError("OKX instruments capture is not a successful V5 response") + payload = payload["data"] + if not isinstance(payload, list) or not all(isinstance(item, dict) for item in payload): + raise ValueError("OKX instruments capture must contain a data list") + return payload diff --git a/qdl/runtime/stable.py b/qdl/runtime/stable.py index 730632c..2d14a3e 100644 --- a/qdl/runtime/stable.py +++ b/qdl/runtime/stable.py @@ -5,6 +5,7 @@ import json import logging import os +import ssl import time from dataclasses import asdict, dataclass from pathlib import Path @@ -74,6 +75,9 @@ class StableRuntimeConfig: audit_path: Path manifest_paths: tuple[Path, ...] source_bindings_path: Path + tls_ca_path: Path + tls_certificate_path: Path + tls_private_key_path: Path internal_ingest_secret: bytes redis_url: str redis_prefix: str @@ -121,6 +125,13 @@ def __post_init__(self) -> None: raise ValueError("stable authority revision and consumer manifests are required") if not self.source_bindings_path.is_file(): raise ValueError("stable source binding catalog is unavailable") + missing_tls = [ + path for path in ( + self.tls_ca_path, self.tls_certificate_path, self.tls_private_key_path + ) if not path.is_file() + ] + if missing_tls: + raise ValueError("stable workload TLS files are unavailable") if len(self.internal_ingest_secret) < 32: raise ValueError("stable internal ingest secret must contain 256 bits") if self.active_cursor_key_id not in self.cursor_keys or any( @@ -190,6 +201,9 @@ def from_environment( )), manifest_paths=manifests, source_bindings_path=Path(env["QDL_STABLE_SOURCE_BINDINGS"]), + tls_ca_path=Path(env["QDL_STABLE_TLS_CA_FILE"]), + tls_certificate_path=Path(env["QDL_STABLE_TLS_CERT_FILE"]), + tls_private_key_path=Path(env["QDL_STABLE_TLS_KEY_FILE"]), internal_ingest_secret=env["QDL_STABLE_INTERNAL_INGEST_SECRET"].encode(), redis_url=env["QDL_STABLE_REDIS_URL"], redis_prefix=env["QDL_STABLE_REDIS_PREFIX"], @@ -238,6 +252,7 @@ def public_manifest(self) -> dict[str, object]: "compatibility_projection": "DEDICATED_REDIS_ONLY", "replay_authority": "KAFKA", "query_cache_authority": False, + "transport_security": "MTLS_PLUS_JWT", } @@ -248,6 +263,35 @@ def _ready(name: str, *, detail: str, revision: str | None = None) -> ComponentR ) +def stable_client_ssl_context(config: StableRuntimeConfig) -> ssl.SSLContext: + context = ssl.create_default_context(cafile=str(config.tls_ca_path)) + context.minimum_version = ssl.TLSVersion.TLSv1_2 + context.load_cert_chain( + certfile=str(config.tls_certificate_path), + keyfile=str(config.tls_private_key_path), + ) + return context + + +def stable_uvicorn_tls(config: StableRuntimeConfig) -> dict[str, object]: + return { + "ssl_keyfile": str(config.tls_private_key_path), + "ssl_certfile": str(config.tls_certificate_path), + "ssl_ca_certs": str(config.tls_ca_path), + "ssl_cert_reqs": ssl.CERT_REQUIRED, + } + + +def stable_grpc_server_credentials( + config: StableRuntimeConfig, +) -> grpc.ServerCredentials: + return grpc.ssl_server_credentials( + ((config.tls_private_key_path.read_bytes(), config.tls_certificate_path.read_bytes()),), + root_certificates=config.tls_ca_path.read_bytes(), + require_client_auth=True, + ) + + def load_stable_manifests(config: StableRuntimeConfig) -> ConsumerManifestRegistry: manifests = tuple(ConsumerManifestLoader.load(path) for path in config.manifest_paths) if any(item.environment != config.environment for item in manifests): @@ -420,7 +464,12 @@ class StableStreamRuntime: async def start(self) -> None: await self.redis.ping() await self.lease.start() - self.grpc_server.add_insecure_port(f"0.0.0.0:{self.config.grpc_port}") + bound = self.grpc_server.add_secure_port( + f"0.0.0.0:{self.config.grpc_port}", + stable_grpc_server_credentials(self.config), + ) + if bound != self.config.grpc_port: + raise RuntimeError("stable gRPC mTLS port binding failed") await self.grpc_server.start() async def stop(self) -> None: @@ -496,12 +545,24 @@ def create_stable_stream_runtime( ) +async def serve_stable_query() -> None: + config = StableRuntimeConfig.from_environment("query_v2") + app = create_stable_query_app(config) + server = uvicorn.Server(uvicorn.Config( + app, host="0.0.0.0", port=config.http_port, + log_level="info", access_log=False, + **stable_uvicorn_tls(config), + )) + await server.serve() + + async def serve_stable_stream() -> None: runtime = create_stable_stream_runtime() await runtime.start() server = uvicorn.Server(uvicorn.Config( runtime.health_app, host="0.0.0.0", port=runtime.config.http_port, log_level="info", access_log=False, + **stable_uvicorn_tls(runtime.config), )) try: await server.serve() @@ -536,7 +597,8 @@ async def serve_stable_projector() -> None: config.redis_url, prefix=f"{config.redis_prefix}:projector" ) sink = StableHttpCanonicalSink( - config.stream_ingest_urls, config.internal_ingest_secret, spool + config.stream_ingest_urls, config.internal_ingest_secret, spool, + ssl_context=stable_client_ssl_context(config), ) projector = StableCompatibilityProjector( catalog, namespace=config.redis_prefix.rstrip(":") diff --git a/qdl/runtime/stable_deployment.py b/qdl/runtime/stable_deployment.py index f3ae1ed..350153c 100644 --- a/qdl/runtime/stable_deployment.py +++ b/qdl/runtime/stable_deployment.py @@ -31,6 +31,59 @@ } +@dataclass(frozen=True, slots=True) +class AuthorityPromotionScope: + schema: str + revision: int + binding_ids: tuple[str, ...] + + def __post_init__(self) -> None: + if ( + self.schema != "qdl.v2.authority-promotion-scope.v1" + or self.revision < 1 + or not self.binding_ids + or any(not value.strip() for value in self.binding_ids) + or len(self.binding_ids) != len(set(self.binding_ids)) + ): + raise ValueError("authority promotion scope is invalid") + + @classmethod + def load( + cls, path: str | Path, *, catalog: StableSourceCatalog + ) -> "AuthorityPromotionScope": + payload = yaml.safe_load(Path(path).read_text(encoding="utf-8")) + if not isinstance(payload, dict) or set(payload) != { + "schema", "revision", "binding_ids", + }: + raise ValueError("authority promotion scope fields are incomplete or unknown") + values = payload["binding_ids"] + if not isinstance(values, list) or not values: + raise ValueError("authority promotion scope requires binding IDs") + result = cls( + schema=str(payload["schema"]), + revision=int(payload["revision"]), + binding_ids=tuple(str(value) for value in values), + ) + catalog_ids = {item.binding_id for item in catalog.bindings} + unknown = set(result.binding_ids) - catalog_ids + if unknown: + raise ValueError( + "authority promotion scope contains unknown bindings: " + + ",".join(sorted(unknown)) + ) + return result + + def digest(self) -> str: + payload = { + "schema": self.schema, + "revision": self.revision, + "binding_ids": list(self.binding_ids), + } + return hashlib.sha256( + json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + + @dataclass(frozen=True, slots=True) class StableAcquisitionBinding: binding_id: str @@ -175,14 +228,18 @@ def core_config( authority: Mapping[str, Any], max_events: int = 0, worker_index: int = 1, + binding_ids: frozenset[str] | None = None, ) -> dict[str, Any]: self._validate_authority(authority) if not 1 <= worker_index <= STABLE_CORE_WORKER_COUNT: raise ValueError("stable core worker index is outside the topology bound") source_by_id = {item.binding_id: item for item in catalog.bindings} acquisitions = {item.binding_id: item for item in self.bindings} + selected_ids = frozenset(source_by_id) if binding_ids is None else binding_ids + if not selected_ids or not selected_ids.issubset(source_by_id): + raise ValueError("stable core binding selection is empty or unknown") bindings = [] - for binding_id in sorted(source_by_id): + for binding_id in sorted(selected_ids): source = source_by_id[binding_id] acquisition = acquisitions[binding_id] identity = source.instrument.identity @@ -222,6 +279,66 @@ def core_config( "metrics_every_batches": 100, } + def production_core_config( + self, + *, + catalog: StableSourceCatalog, + raw_authority: Mapping[str, Any], + promotion_scope: AuthorityPromotionScope, + worker_index: int, + partition_plan_epoch: int = 1, + ) -> dict[str, Any]: + self._validate_authority(raw_authority) + if partition_plan_epoch < 1: + raise ValueError("production partition plan epoch must be positive") + selected_ids = frozenset(promotion_scope.binding_ids) + shadow = self.core_config( + catalog=catalog, + authority=raw_authority, + worker_index=worker_index, + binding_ids=selected_ids, + ) + source_by_id = {item.binding_id: item for item in catalog.bindings} + slices = [] + for source in sorted( + (source_by_id[binding_id] for binding_id in selected_ids), + key=lambda value: value.binding_id, + ): + identity = source.instrument.identity + native = source.instrument.native_symbol.lower() + slice_id = ( + f"production/{identity.venue.lower()}/{identity.market.lower()}/" + f"{identity.product_type.value.lower()}/{source.feed.value.lower()}/" + f"plan-{partition_plan_epoch}/{native}" + ) + slices.append({ + "subscription_id": source.source_id, + "slice_id": slice_id, + "shard_id": source.binding_id, + "raw_authority_revision": int(raw_authority["revision"]), + "raw_lease_epoch": 1, + "raw_partition_plan_epoch": partition_plan_epoch, + }) + return { + "core": shadow["core"], + "topics": { + "raw_inputs": [self.raw_topic], + "authority_control": "qdl.authority.v1", + "target_checkpoints": "qdl.target-checkpoint.v1", + "canary_canonical": "md.canary.canonical.v2", + "primary_canonical": self.canonical_topic, + "public_v2": "md.projector.public.v2", + "legacy_v1": "md.projector.legacy.v1", + "quarantine": self.quarantine_topic, + }, + "slices": slices, + "transactional_id": f"qdl-v2-production-core-{worker_index:03d}", + "batch_size": 128, + "batch_wait_ms": 10, + "max_events": 0, + "metrics_every_batches": 100, + } + def native_ingestor_configs( self, *, @@ -300,6 +417,53 @@ def _validate_authority(authority: Mapping[str, Any]) -> None: raise ValueError("stable authority is not an isolated Rust shadow record") +def write_production_core_bundle( + destination: Path, + *, + catalog: StableSourceCatalog, + acquisition: StableAcquisitionPlan, + promotion_scope: AuthorityPromotionScope, + raw_authority: Mapping[str, Any], + partition_plan_epoch: int = 1, +) -> dict[str, str]: + destination.mkdir(parents=True, exist_ok=True) + payloads = { + f"production-core-{worker_index:03d}.json": + acquisition.production_core_config( + catalog=catalog, + raw_authority=raw_authority, + promotion_scope=promotion_scope, + worker_index=worker_index, + partition_plan_epoch=partition_plan_epoch, + ) + for worker_index in range(1, STABLE_CORE_WORKER_COUNT + 1) + } + digests = {} + for name, payload in sorted(payloads.items()): + encoded = ( + json.dumps(payload, indent=2, sort_keys=True, separators=(",", ": ")) + "\n" + ).encode() + path = destination / name + path.write_bytes(encoded) + digests[name] = hashlib.sha256(encoded).hexdigest() + manifest = { + "schema": "qdl.v2.production-core-bundle.v1", + "partition_plan_epoch": partition_plan_epoch, + "worker_count": STABLE_CORE_WORKER_COUNT, + "promotion_scope_revision": promotion_scope.revision, + "promotion_scope_digest": promotion_scope.digest(), + "promotion_binding_count": len(promotion_scope.binding_ids), + "files": digests, + } + encoded_manifest = ( + json.dumps(manifest, indent=2, sort_keys=True, separators=(",", ": ")) + "\n" + ).encode() + manifest_path = destination / "production-core-manifest.json" + manifest_path.write_bytes(encoded_manifest) + digests[manifest_path.name] = hashlib.sha256(encoded_manifest).hexdigest() + return digests + + def stable_authority_record( *, rust_image_digest: str, diff --git a/qdl/runtime/stable_ingest.py b/qdl/runtime/stable_ingest.py index 6a60742..b06a735 100644 --- a/qdl/runtime/stable_ingest.py +++ b/qdl/runtime/stable_ingest.py @@ -6,6 +6,7 @@ import hmac import ipaddress import json +import ssl import uuid from dataclasses import dataclass, field from urllib.parse import urlsplit @@ -33,7 +34,7 @@ def _signature(secret: bytes, body: bytes) -> str: def _internal_url(value: str) -> bool: parsed = urlsplit(value) - if parsed.scheme != "http" or not parsed.hostname: + if parsed.scheme not in {"http", "https"} or not parsed.hostname: return False try: return ipaddress.ip_address(parsed.hostname).is_loopback @@ -212,6 +213,7 @@ class StableHttpCanonicalSink: spool: SQLiteDurableSpool timeout_seconds: float = 10.0 client: httpx.AsyncClient | None = None + ssl_context: ssl.SSLContext | None = None _owns_client: bool = field(init=False) def __post_init__(self) -> None: @@ -222,12 +224,17 @@ def __post_init__(self) -> None: or self.timeout_seconds <= 0 ): raise ValueError("stable HTTP sink configuration is invalid") + if any(urlsplit(value).scheme == "https" for value in self.urls) and self.ssl_context is None and self.client is None: + raise ValueError("stable HTTPS sink requires a workload TLS context") + if self.client is not None and self.ssl_context is not None: + raise ValueError("stable HTTP sink client and TLS context are mutually exclusive") self._owns_client = self.client is None if self.client is None: self.client = httpx.AsyncClient( follow_redirects=False, limits=httpx.Limits(max_connections=10, max_keepalive_connections=5), timeout=self.timeout_seconds, + verify=self.ssl_context or True, ) async def publish(self, event: DurableEvent) -> StoredEvent: diff --git a/qdl_sdk/README.md b/qdl_sdk/README.md index 23e51bd..2ddc2d1 100644 --- a/qdl_sdk/README.md +++ b/qdl_sdk/README.md @@ -32,11 +32,18 @@ atomically restored the local state associated with that checkpoint. ```python from qdl_sdk import DataRequirement, Feed, Grade +instrument = await client.resolve_instrument( + venue="BINANCE", + market="USDM", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + consumer_grade=Grade.EXECUTION, +) requirement = DataRequirement( - instrument_uid="a953e16e-7138-5562-b5e8-c337a44d0b65", + instrument_uid=instrument.instrument_uid, feed=Feed.TRADE, consumer_grade=Grade.EXECUTION, - source_policy_id="execution_binance_usdm_v1", + source_policy_id="crypto_primary_v2", max_freshness_ms=1000, ) @@ -68,3 +75,18 @@ async with client.warmup_then_stream( handoff metadata is a hard continuity error. - The SDK never parses cursor internals and never silently accepts stale, gapped, partial or non-authoritative execution data. + +## Immutable consumer artifact + +Build the standalone artifact with: + +```bash +python scripts/build_qdl_sdk_release.py --output-dir dist/qdl-sdk +``` + +The output contains a reproducible `qdl_sdk-2.0.0-py3-none-any.whl`, a release +manifest with the wheel/source/generated-contract SHA-256 digests, and a +CycloneDX SBOM. The wheel contains only the public SDK plus generated Protobuf +contracts; it does not package `qdl.api_v2`, runtime adapters, provider code or +other Data Layer service internals. Trading System and the shared alpha runtime +must pin the same verified wheel digest. diff --git a/qdl_sdk/__init__.py b/qdl_sdk/__init__.py index 9d2a8e0..e66b085 100644 --- a/qdl_sdk/__init__.py +++ b/qdl_sdk/__init__.py @@ -6,6 +6,7 @@ from qdl_sdk.credentials import ( CallbackCredentialProvider, CredentialProvider, + RotatingJwtCredentialProvider, StaticBearerCredential, ) from qdl_sdk.cursor import CursorCheckpoint, FileCursorStore, MemoryCursorStore @@ -17,6 +18,9 @@ Feed, GapPolicy, Grade, + InstrumentPageResponse, + InstrumentResponse, + InstrumentView, MarketDataView, QuantityUnit, RecoveryPolicy, @@ -26,7 +30,9 @@ TradeIdentityKind, WarmupResponse, ) +from qdl_sdk.projection import market_data_view_from_stream from qdl_sdk.transport import GrpcStreamTransport, RestQueryTransport +from qdl_sdk.tls import WorkloadTlsConfig from qdl_sdk.v1_facade import V1CompatibilityFacade __all__ = [ @@ -45,17 +51,23 @@ "GapPolicy", "Grade", "GrpcStreamTransport", + "InstrumentPageResponse", + "InstrumentResponse", + "InstrumentView", "MemoryCursorStore", "MarketDataView", + "market_data_view_from_stream", "QuantityUnit", "RestQueryTransport", "RecoveryPolicy", + "RotatingJwtCredentialProvider", "SnapshotResponse", "StalePolicy", "StaticBearerCredential", "StreamEvent", "TradeIdentityKind", "WarmupResponse", + "WorkloadTlsConfig", "V1CompatibilityFacade", "WarmupStreamSession", ] diff --git a/qdl_sdk/client.py b/qdl_sdk/client.py index 944155b..ce34d73 100644 --- a/qdl_sdk/client.py +++ b/qdl_sdk/client.py @@ -13,6 +13,9 @@ DataRequirement, Feed, Grade, + InstrumentPageResponse, + InstrumentResponse, + InstrumentView, SnapshotResponse, StreamEvent, WarmupResponse, @@ -22,6 +25,17 @@ class QueryTransport(Protocol): async def warmup(self, requirement: DataRequirement, *, consumer_id: str) -> dict: ... async def snapshot(self, requirement: DataRequirement, *, consumer_id: str) -> dict: ... + async def instruments( + self, + *, + consumer_id: str, + consumer_grade: Grade, + cursor: str | None, + limit: int, + ) -> dict: ... + async def instrument( + self, identity: str, *, consumer_id: str, consumer_grade: Grade + ) -> dict: ... async def close(self) -> None: ... @@ -297,6 +311,98 @@ async def warmup(self, requirement: DataRequirement) -> WarmupResponse: self._record_query("/v2/market-data/warmup", response) return response + async def instrument( + self, identity: str, *, consumer_grade: Grade + ) -> InstrumentResponse: + payload = await self.query_transport.instrument( + identity, + consumer_id=self.consumer_id, + consumer_grade=consumer_grade, + ) + try: + return InstrumentResponse.model_validate(payload) + except ValidationError as error: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", + "instrument response violates the typed V2 contract", + ) from error + + async def resolve_instrument( + self, + *, + venue: str, + product_type: str, + native_symbol: str, + consumer_grade: Grade, + market: str | None = None, + page_limit: int = 500, + max_pages: int = 100, + ) -> InstrumentView: + if not isinstance(consumer_grade, Grade): + raise TypeError("consumer_grade must use the typed SDK enum") + if not 1 <= page_limit <= 500 or not 1 <= max_pages <= 100: + raise ValueError("instrument resolver page bounds are invalid") + expected = { + "venue": venue.strip().upper(), + "product_type": product_type.strip().upper(), + "native_symbol": native_symbol.strip().upper(), + "market": market.strip().upper() if market is not None else None, + } + if not all(expected[key] for key in ("venue", "product_type", "native_symbol")): + raise ValueError("venue, product_type and native_symbol are required") + cursor: str | None = None + matches: list[InstrumentView] = [] + seen_cursors: set[str] = set() + for _ in range(max_pages): + payload = await self.query_transport.instruments( + consumer_id=self.consumer_id, + consumer_grade=consumer_grade, + cursor=cursor, + limit=page_limit, + ) + try: + page = InstrumentPageResponse.model_validate(payload) + except ValidationError as error: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", + "instrument page violates the typed V2 contract", + ) from error + for item in page.items: + if ( + item.venue.upper() == expected["venue"] + and item.product_type.upper() == expected["product_type"] + and item.native_symbol.upper() == expected["native_symbol"] + and ( + expected["market"] is None + or item.market.upper() == expected["market"] + ) + ): + matches.append(item) + if page.next_cursor is None: + break + if page.next_cursor in seen_cursors: + raise ContinuityError( + "CONFLICT", "instrument catalog returned a cursor cycle" + ) + seen_cursors.add(page.next_cursor) + cursor = page.next_cursor + else: + raise ContinuityError( + "PARTIAL_RESULT", "instrument catalog exceeded bounded pagination" + ) + active = [item for item in matches if item.status.upper() == "ACTIVE"] + if len(active) == 1: + return active[0] + if not active: + raise DataLayerError( + "INSTRUMENT_NOT_FOUND", + "no active V2 instrument matches the venue/product/native symbol", + retryable=False, + ) + raise ContinuityError( + "CONFLICT", "instrument identity is ambiguous; specify market" + ) + @asynccontextmanager async def warmup_then_stream( self, diff --git a/qdl_sdk/credentials.py b/qdl_sdk/credentials.py index 4a459b4..609e749 100644 --- a/qdl_sdk/credentials.py +++ b/qdl_sdk/credentials.py @@ -1,7 +1,13 @@ from __future__ import annotations +import asyncio import inspect +import time +import uuid from dataclasses import dataclass +from pathlib import Path + +import jwt from typing import Awaitable, Callable, Protocol @@ -33,3 +39,84 @@ async def get_token(self) -> str: if not isinstance(token, str) or not token.strip(): raise RuntimeError("workload credential provider returned an empty token") return token + + +class RotatingJwtCredentialProvider: + """Signs bounded workload JWTs locally and reloads the private key on refresh.""" + + def __init__( + self, + *, + private_key_file: str | Path, + key_id: str, + algorithm: str, + issuer: str, + audience: str, + subject: str, + environment: str, + roles: tuple[str, ...], + venues: tuple[str, ...] = (), + consumer_manifest_revision: int = 1, + lifetime_seconds: int = 600, + refresh_before_seconds: int = 120, + clock=time.time, + ) -> None: + self.private_key_file = Path(private_key_file).expanduser().resolve() + self.key_id = key_id.strip() + self.algorithm = algorithm.strip().upper() + self.issuer = issuer.strip() + self.audience = audience.strip() + self.subject = subject.strip() + self.environment = environment.strip().lower() + self.roles = tuple(sorted(set(roles))) + self.venues = tuple(sorted({value.upper() for value in venues})) + self.consumer_manifest_revision = int(consumer_manifest_revision) + self.lifetime_seconds = int(lifetime_seconds) + self.refresh_before_seconds = int(refresh_before_seconds) + self._clock = clock + self._token = "" + self._expires_at = 0 + self._lock = asyncio.Lock() + if ( + not self.private_key_file.is_file() + or not all(( + self.key_id, self.issuer, self.audience, + self.subject, self.environment, + )) + or self.algorithm not in {"RS256", "ES256"} + or not self.roles + or self.consumer_manifest_revision < 1 + or not 60 <= self.lifetime_seconds <= 900 + or not 5 <= self.refresh_before_seconds < self.lifetime_seconds + ): + raise ValueError("rotating workload JWT configuration is invalid") + + async def get_token(self) -> str: + now = int(self._clock()) + if self._token and self._expires_at - now > self.refresh_before_seconds: + return self._token + async with self._lock: + now = int(self._clock()) + if self._token and self._expires_at - now > self.refresh_before_seconds: + return self._token + expires_at = now + self.lifetime_seconds + claims = { + "sub": self.subject, + "iss": self.issuer, + "aud": self.audience, + "iat": now, + "exp": expires_at, + "jti": str(uuid.uuid4()), + "environment": self.environment, + "roles": list(self.roles), + "venues": list(self.venues), + "consumer_manifest_revision": self.consumer_manifest_revision, + } + self._token = jwt.encode( + claims, + self.private_key_file.read_bytes(), + algorithm=self.algorithm, + headers={"kid": self.key_id}, + ) + self._expires_at = expires_at + return self._token diff --git a/qdl_sdk/models.py b/qdl_sdk/models.py index f0fd5a1..edc4dcf 100644 --- a/qdl_sdk/models.py +++ b/qdl_sdk/models.py @@ -1,29 +1,47 @@ from __future__ import annotations from dataclasses import dataclass -from enum import StrEnum -from typing import Any - -from qdl.api_v2.models import ( - MarketDataView, - QuantityUnit, - SnapshotResponse, - TradeIdentityKind, - WarmupResponse, -) +from enum import Enum +from typing import Annotated, Any, Literal + +try: # Python 3.11+; the released SDK contract supports Python 3.10. + from enum import StrEnum +except ImportError: # pragma: no cover - exercised by the Python 3.10 artifact gate. + class StrEnum(str, Enum): + def __str__(self) -> str: + return self.value + +from pydantic import BaseModel, ConfigDict, Field, model_validator + from qdl.query.v2 import query_pb2 -class Feed(StrEnum): - TRADE = "TRADE" - QUOTE = "QUOTE" - BAR = "BAR" - BOOK_SNAPSHOT = "BOOK_SNAPSHOT" - BOOK_DELTA = "BOOK_DELTA" - FUNDING_RATE = "FUNDING_RATE" - OPEN_INTEREST = "OPEN_INTEREST" - MARK_INDEX_PRICE = "MARK_INDEX_PRICE" - TICKER = "TICKER" +try: + # The service and SDK share enum identity when the public query package is present. + from qdl.query import BarLifecycle, FeedType +except ImportError: + class FeedType(StrEnum): + UNSPECIFIED = "UNSPECIFIED" + TRADE = "TRADE" + QUOTE = "QUOTE" + BAR = "BAR" + BOOK_SNAPSHOT = "BOOK_SNAPSHOT" + BOOK_DELTA = "BOOK_DELTA" + FUNDING_RATE = "FUNDING_RATE" + OPEN_INTEREST = "OPEN_INTEREST" + MARK_INDEX_PRICE = "MARK_INDEX_PRICE" + TICKER = "TICKER" + + class BarLifecycle(StrEnum): + UNSPECIFIED = "UNSPECIFIED" + IN_PROGRESS = "IN_PROGRESS" + FINAL = "FINAL" + REVISED = "REVISED" + CANCELLED = "CANCELLED" + + +# Concise public SDK spelling; the frozen wire component remains FeedType. +Feed = FeedType class Grade(StrEnum): @@ -56,6 +74,340 @@ class BarRevisionPolicy(StrEnum): EMIT_REVISIONS = "EMIT_REVISIONS" +class ClosedModel(BaseModel): + model_config = ConfigDict(extra="forbid", populate_by_name=True) + + +class ProblemDetails(ClosedModel): + type: str + title: str + status: int + code: str + detail: str + request_id: str + retryable: bool + retry_after_ms: int | None = None + instrument_uid: str | None = None + quality_state: str | None = None + + +class DecimalValue(ClosedModel): + coefficient: str = Field(pattern=r"^-?(0|[1-9][0-9]*)$") + scale: int = Field(ge=-38, le=38) + source_text: str = Field(min_length=1, max_length=128) + + +class TradeIdentityKind(StrEnum): + NATIVE = "NATIVE" + DERIVED_RAW_CAPTURE = "DERIVED_RAW_CAPTURE" + + +class QuantityUnit(StrEnum): + BASE_ASSET = "BASE_ASSET" + QUOTE_ASSET = "QUOTE_ASSET" + CONTRACT = "CONTRACT" + SHARE = "SHARE" + + +class SourceView(ClosedModel): + venue: str = Field(min_length=1, max_length=40) + provider: str = Field(min_length=1, max_length=80) + source_id: str = Field(min_length=1, max_length=120) + source_role: Literal["PRIMARY", "SECONDARY", "REFERENCE", "BACKFILL"] + authoritative: bool + + +class QualityView(ClosedModel): + state: Literal[ + "DISABLED", "STARTING", "CONNECTING", "SUBSCRIBING", "SYNCING", + "LIVE", "DEGRADED", "GAPPED", "RESYNCING", "STALE", "OFFLINE", + "HALTED", "MARKET_CLOSED", + ] + freshness_ms: int = Field(ge=0) + gap_open: bool + complete: bool + execution_eligible: bool + policy_id: str = Field(min_length=1, max_length=200) + flags: list[str] + + +class ContractView(ClosedModel): + schema_digest: str = Field(pattern=r"^[0-9a-f]{64}$") + contract_version: str = Field(min_length=1, max_length=40) + normalizer_version: str = Field(min_length=1, max_length=80) + adapter_version: str = Field(min_length=1, max_length=80) + instrument_catalog_revision: int = Field(ge=1) + source_policy_revision: int = Field(ge=1) + authority_revision: int = Field(ge=1) + config_revision: int = Field(ge=1) + correlation_id: str = Field(min_length=1, max_length=200) + + +class TradePayload(ClosedModel): + feed: Literal[Feed.TRADE] = Feed.TRADE + native_trade_id: str = Field(min_length=1, max_length=200) + price: DecimalValue + quantity: DecimalValue + quantity_unit: QuantityUnit + aggressor_side: Literal["BUY", "SELL", "UNKNOWN"] + identity_kind: TradeIdentityKind + is_block_trade: bool = False + is_buyer_maker: bool = False + + +class QuotePayload(ClosedModel): + feed: Literal[Feed.QUOTE] = Feed.QUOTE + bid_price: DecimalValue + bid_quantity: DecimalValue + ask_price: DecimalValue + ask_quantity: DecimalValue + quantity_unit: QuantityUnit + level: int = Field(default=1, ge=1) + + +class BarPayload(ClosedModel): + feed: Literal[Feed.BAR] = Feed.BAR + interval: str = Field(min_length=1, max_length=20) + open_time_ns: int = Field(gt=0) + close_time_ns: int = Field(gt=0) + open: DecimalValue + high: DecimalValue + low: DecimalValue + close: DecimalValue + volume: DecimalValue + volume_unit: QuantityUnit + base_volume: DecimalValue | None = None + quote_volume: DecimalValue | None = None + contract_volume: DecimalValue | None = None + trade_count: int = Field(default=0, ge=0) + lifecycle: BarLifecycle + revision: int = Field(ge=0) + origin: Literal["VENUE_NATIVE", "AGGREGATED", "BACKFILLED", "RECONCILED"] + supersedes_event_id: str | None = None + + @model_validator(mode="after") + def validate_lifecycle(self): + if self.lifecycle is BarLifecycle.UNSPECIFIED: + raise ValueError("bar lifecycle cannot be UNSPECIFIED") + if self.lifecycle is BarLifecycle.REVISED and not self.supersedes_event_id: + raise ValueError("revised bar must identify the superseded event") + if self.close_time_ns <= self.open_time_ns: + raise ValueError("bar close time must be after open time") + return self + + +class BookLevel(ClosedModel): + side: Literal["BID", "ASK"] + price: DecimalValue + quantity: DecimalValue + quantity_unit: QuantityUnit + order_count: int = Field(default=0, ge=0) + + +class BookSnapshotPayload(ClosedModel): + feed: Literal[Feed.BOOK_SNAPSHOT] = Feed.BOOK_SNAPSHOT + native_sequence: str = Field(min_length=1, max_length=200) + checksum: str | None = Field(default=None, max_length=200) + levels: list[BookLevel] + depth: int = Field(ge=1) + + +class BookDeltaPayload(ClosedModel): + feed: Literal[Feed.BOOK_DELTA] = Feed.BOOK_DELTA + native_sequence_start: str = Field(min_length=1, max_length=200) + native_sequence_end: str = Field(min_length=1, max_length=200) + snapshot_sequence: str = Field(min_length=1, max_length=200) + checksum: str | None = Field(default=None, max_length=200) + updates: list[BookLevel] + reset: bool = False + + +class FundingRatePayload(ClosedModel): + feed: Literal[Feed.FUNDING_RATE] = Feed.FUNDING_RATE + rate: DecimalValue + funding_time_ns: int = Field(gt=0) + next_funding_time_ns: int | None = Field(default=None, gt=0) + + +class OpenInterestPayload(ClosedModel): + feed: Literal[Feed.OPEN_INTEREST] = Feed.OPEN_INTEREST + quantity: DecimalValue + quantity_unit: QuantityUnit + notional: DecimalValue | None = None + + +class MarkIndexPricePayload(ClosedModel): + feed: Literal[Feed.MARK_INDEX_PRICE] = Feed.MARK_INDEX_PRICE + mark_price: DecimalValue + index_price: DecimalValue + + +class TickerPayload(ClosedModel): + feed: Literal[Feed.TICKER] = Feed.TICKER + last_price: DecimalValue + last_quantity: DecimalValue | None = None + open_24h: DecimalValue | None = None + high_24h: DecimalValue | None = None + low_24h: DecimalValue | None = None + volume_24h: DecimalValue | None = None + last_quantity_unit: QuantityUnit | None = None + volume_24h_unit: QuantityUnit | None = None + + @model_validator(mode="after") + def quantity_units_match_optional_values(self): + if (self.last_quantity is None) != (self.last_quantity_unit is None): + raise ValueError("ticker last quantity and unit must be present together") + if (self.volume_24h is None) != (self.volume_24h_unit is None): + raise ValueError("ticker 24h volume and unit must be present together") + return self + + +MarketPayload = Annotated[ + TradePayload | QuotePayload | BarPayload | BookSnapshotPayload | BookDeltaPayload + | FundingRatePayload | OpenInterestPayload | MarkIndexPricePayload | TickerPayload, + Field(discriminator="feed"), +] + + +class MarketDataView(ClosedModel): + instrument_uid: str + instrument_id: str + instrument_revision: int = Field(ge=1) + feed: Feed + interval: str | None + observed_at_ns: int = Field(gt=0) + revision: int = Field(ge=0) + payload: MarketPayload + source: SourceView + quality: QualityView + contract: ContractView + cursor: str | None = None + snapshot_id: str | None = None + watermark_offset: int = Field(default=0, ge=0) + + @model_validator(mode="after") + def feed_matches_payload(self): + if self.payload.feed is not self.feed: + raise ValueError("market-data envelope feed does not match its payload") + if self.feed is Feed.BAR: + if not self.interval or self.payload.interval != self.interval: + raise ValueError("bar envelope and payload interval must match") + if self.payload.revision != self.revision: + raise ValueError("bar envelope and payload revision must match") + elif self.interval is not None: + raise ValueError("interval is valid only for bar market data") + return self + + +class SnapshotResponse(ClosedModel): + contract_schema: str = Field("qdl.marketdata.snapshot.v2", alias="schema") + request_id: str + data: MarketDataView + + +class InstrumentView(ClosedModel): + instrument_uid: str + instrument_id: str + venue: str + market: str + product_type: str + canonical_symbol: str + metadata_revision: int = Field(ge=1) + asset_class: str + native_symbol: str + status: str + + +class InstrumentPageResponse(ClosedModel): + contract_schema: str = Field("qdl.instruments.page.v2", alias="schema") + items: list[InstrumentView] + next_cursor: str | None = None + + +class InstrumentResponse(InstrumentView): + contract_schema: str = Field("qdl.instrument.v2", alias="schema") + + +class WarmupResponse(ClosedModel): + contract_schema: str = Field("qdl.marketdata.warmup.v2", alias="schema") + request_id: str + snapshot_id: str = Field(min_length=1) + data_as_of_ns: int = Field(gt=0) + stream_cursor: str = Field(min_length=1) + watermark_offset: int = Field(ge=0) + coverage: str + count: int + data: list[MarketDataView] + + +class BatchItemResponse(ClosedModel): + instrument_uid: str + status: str + data: WarmupResponse | None = None + problem: ProblemDetails | None = None + + +class BatchResponse(ClosedModel): + contract_schema: str = Field("qdl.marketdata.batch.v2", alias="schema") + request_id: str + partial: bool + success_count: int + error_count: int + results: list[BatchItemResponse] + + @model_validator(mode="after") + def counts_match(self): + if self.success_count + self.error_count != len(self.results): + raise ValueError("batch counts do not match results") + if self.partial != (self.error_count > 0): + raise ValueError("batch partial flag does not match errors") + return self + + +class ReadinessItemResponse(ClosedModel): + instrument_uid: str + status: str + quality: QualityView | None = None + problem: ProblemDetails | None = None + + +class ReadinessResponse(ClosedModel): + contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") + request_id: str + ready: bool + authority: str = "V1" + results: list[ReadinessItemResponse] + + +class FeedStatusResponse(ClosedModel): + contract_schema: str = Field("qdl.feed-status.v2", alias="schema") + instrument_uid: str + feed: Feed + quality: QualityView + + +class GapView(ClosedModel): + gap_id: str + instrument_uid: str + feed: Feed + source_id: str + expected_sequence: str + observed_sequence: str + detected_at_ns: int + + +class GapListResponse(ClosedModel): + contract_schema: str = Field("qdl.data-quality.gaps.v2", alias="schema") + items: list[GapView] + + +class SystemReadinessSummary(ClosedModel): + contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") + status: str + authority: str + v2_consumer_activation: str + + @dataclass(frozen=True) class DataRequirement: instrument_uid: str @@ -75,6 +427,8 @@ class DataRequirement: def __post_init__(self) -> None: if not self.instrument_uid.strip() or not self.source_policy_id.strip(): raise ValueError("instrument_uid and source_policy_id are required") + if self.feed is Feed.UNSPECIFIED: + raise ValueError("UNSPECIFIED feed is invalid at the V2 boundary") enum_fields = ( (self.feed, Feed, "feed"), (self.consumer_grade, Grade, "consumer_grade"), @@ -157,7 +511,7 @@ def __post_init__(self) -> None: class ControlEvent: code: str detail: str - snapshot: WarmupResponse | None = None + snapshot: WarmupResponse | dict[str, Any] | None = None def __post_init__(self) -> None: if not self.code.strip() or not self.detail.strip(): diff --git a/qdl_sdk/projection.py b/qdl_sdk/projection.py new file mode 100644 index 0000000..a15d3fd --- /dev/null +++ b/qdl_sdk/projection.py @@ -0,0 +1,390 @@ +"""SDK-owned canonical protobuf to typed V2 view projection.""" + +from __future__ import annotations + +import time +from typing import Any + +from qdl.common.v1 import common_pb2 +from qdl.marketdata.v2 import market_data_pb2 +from qdl_sdk.errors import ContinuityError +from qdl_sdk.models import DataRequirement, Feed, MarketDataView, StreamEvent + +_GAP_FLAGS = {"SEQUENCE_GAP_BEFORE", "OUT_OF_ORDER", "RESYNC_REQUIRED"} + + +def _enum_name(enum_wrapper: Any, value: int, prefix: str) -> str: + try: + name = enum_wrapper.Name(value) + except ValueError as error: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", f"unknown canonical enum value {value}" + ) from error + result = name.removeprefix(prefix) + if result == "UNSPECIFIED": + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", f"canonical enum {prefix} is unspecified" + ) + return result + + +def _decimal(value) -> dict[str, str | int]: + selected = value.WhichOneof("coefficient") + if selected == "mantissa": + coefficient = str(value.mantissa) + elif selected == "mantissa_text": + coefficient = value.mantissa_text + else: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "canonical decimal coefficient is missing" + ) + return { + "coefficient": coefficient, + "scale": int(value.scale), + "source_text": value.source_text, + } + + +def _quantity_unit(value: int) -> str: + return _enum_name(common_pb2.QuantityUnit, value, "QUANTITY_UNIT_") + + +def _book_level(value) -> dict[str, Any]: + return { + "side": _enum_name(common_pb2.BookSide, value.side, "BOOK_SIDE_"), + "price": _decimal(value.price), + "quantity": _decimal(value.quantity), + "quantity_unit": _quantity_unit(value.quantity_unit), + "order_count": int(value.order_count), + } + + +def _payload( + envelope: market_data_pb2.EventEnvelope, +) -> tuple[Feed, str | None, int, dict[str, Any]]: + name = envelope.WhichOneof("payload") + if name == "trade": + side = common_pb2.AggressorSide.Name( + envelope.trade.aggressor_side + ).removeprefix("AGGRESSOR_SIDE_") + if side == "UNSPECIFIED": + side = "UNKNOWN" + identity = _enum_name( + market_data_pb2.TradeIdentityKind, + envelope.trade.identity_kind, + "TRADE_IDENTITY_KIND_", + ) + return ( + Feed.TRADE, + None, + 0, + { + "feed": "TRADE", + "native_trade_id": envelope.trade.native_trade_id, + "price": _decimal(envelope.trade.price), + "quantity": _decimal(envelope.trade.quantity), + "quantity_unit": _quantity_unit(envelope.trade.quantity_unit), + "aggressor_side": side, + "identity_kind": identity, + "is_block_trade": bool(envelope.trade.is_block_trade), + "is_buyer_maker": bool(envelope.trade.is_buyer_maker), + }, + ) + if name == "quote": + return ( + Feed.QUOTE, + None, + 0, + { + "feed": "QUOTE", + "bid_price": _decimal(envelope.quote.bid_price), + "bid_quantity": _decimal(envelope.quote.bid_quantity), + "ask_price": _decimal(envelope.quote.ask_price), + "ask_quantity": _decimal(envelope.quote.ask_quantity), + "quantity_unit": _quantity_unit(envelope.quote.quantity_unit), + "level": int(envelope.quote.level), + }, + ) + if name == "bar": + lifecycle = _enum_name( + market_data_pb2.BarLifecycle, + envelope.bar.lifecycle, + "BAR_LIFECYCLE_", + ) + payload: dict[str, Any] = { + "feed": "BAR", + "interval": envelope.bar.interval, + "open_time_ns": int(envelope.bar.open_time_ns), + "close_time_ns": int(envelope.bar.close_time_ns), + "open": _decimal(envelope.bar.open), + "high": _decimal(envelope.bar.high), + "low": _decimal(envelope.bar.low), + "close": _decimal(envelope.bar.close), + "volume": _decimal(envelope.bar.volume), + "volume_unit": _quantity_unit(envelope.bar.volume_unit), + "trade_count": int(envelope.bar.trade_count), + "lifecycle": lifecycle, + "revision": int(envelope.bar.revision), + "origin": _enum_name( + common_pb2.BarOrigin, envelope.bar.origin, "BAR_ORIGIN_" + ), + "supersedes_event_id": ( + bytes(envelope.bar.supersedes_event_id).hex() + if envelope.bar.HasField("supersedes_event_id") + else None + ), + } + for field in ("base_volume", "quote_volume", "contract_volume"): + payload[field] = ( + _decimal(getattr(envelope.bar, field)) + if envelope.bar.HasField(field) + else None + ) + return Feed.BAR, envelope.bar.interval, int(envelope.bar.revision), payload + if name == "book_snapshot": + return ( + Feed.BOOK_SNAPSHOT, + None, + 0, + { + "feed": "BOOK_SNAPSHOT", + "native_sequence": envelope.book_snapshot.native_sequence, + "checksum": envelope.book_snapshot.checksum or None, + "levels": [_book_level(item) for item in envelope.book_snapshot.levels], + "depth": int(envelope.book_snapshot.depth), + }, + ) + if name == "book_delta": + return ( + Feed.BOOK_DELTA, + None, + 0, + { + "feed": "BOOK_DELTA", + "native_sequence_start": envelope.book_delta.native_sequence_start, + "native_sequence_end": envelope.book_delta.native_sequence_end, + "snapshot_sequence": envelope.book_delta.snapshot_sequence, + "checksum": envelope.book_delta.checksum or None, + "updates": [_book_level(item) for item in envelope.book_delta.updates], + "reset": bool(envelope.book_delta.reset), + }, + ) + if name == "funding_rate": + return ( + Feed.FUNDING_RATE, + None, + 0, + { + "feed": "FUNDING_RATE", + "rate": _decimal(envelope.funding_rate.rate), + "funding_time_ns": int(envelope.funding_rate.funding_time_ns), + "next_funding_time_ns": ( + int(envelope.funding_rate.next_funding_time_ns) + if envelope.funding_rate.HasField("next_funding_time_ns") + else None + ), + }, + ) + if name == "open_interest": + return ( + Feed.OPEN_INTEREST, + None, + 0, + { + "feed": "OPEN_INTEREST", + "quantity": _decimal(envelope.open_interest.quantity), + "notional": ( + _decimal(envelope.open_interest.notional) + if envelope.open_interest.HasField("notional") + else None + ), + "quantity_unit": _quantity_unit(envelope.open_interest.quantity_unit), + }, + ) + if name == "mark_index_price": + return ( + Feed.MARK_INDEX_PRICE, + None, + 0, + { + "feed": "MARK_INDEX_PRICE", + "mark_price": _decimal(envelope.mark_index_price.mark_price), + "index_price": _decimal(envelope.mark_index_price.index_price), + }, + ) + if name == "ticker": + payload = {"feed": "TICKER", "last_price": _decimal(envelope.ticker.last_price)} + for field in ("last_quantity", "open_24h", "high_24h", "low_24h", "volume_24h"): + payload[field] = ( + _decimal(getattr(envelope.ticker, field)) + if envelope.ticker.HasField(field) + else None + ) + payload["last_quantity_unit"] = ( + _quantity_unit(envelope.ticker.last_quantity_unit) + if envelope.ticker.HasField("last_quantity") + else None + ) + payload["volume_24h_unit"] = ( + _quantity_unit(envelope.ticker.volume_24h_unit) + if envelope.ticker.HasField("volume_24h") + else None + ) + return Feed.TICKER, None, 0, payload + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", + f"unsupported canonical stream payload: {name or 'none'}", + ) + + +def market_data_view_from_stream( + event: StreamEvent, + *, + template: MarketDataView, + requirement: DataRequirement, + now_ns: int | None = None, +) -> MarketDataView: + """Project one SDK stream event through its authoritative query handoff. + + The query template owns source policy/catalog metadata. Any stream identity + or source transition requires a new snapshot and fails closed here. + """ + + envelope = event.event + if not isinstance(envelope, market_data_pb2.EventEnvelope): + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "stream event is not a canonical V2 envelope" + ) + if envelope.schema_major != 2 or not envelope.schema_name: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "stream envelope schema is not V2" + ) + if ( + envelope.instrument_uid != requirement.instrument_uid + or envelope.instrument_uid != template.instrument_uid + ): + raise ContinuityError("CONFLICT", "stream instrument UID differs from handoff") + if int(envelope.instrument_revision) != template.instrument_revision: + raise ContinuityError( + "CONFLICT", "stream instrument revision differs from handoff" + ) + if ( + envelope.venue.upper() != template.source.venue.upper() + or envelope.provider != template.source.provider + or envelope.source_id != template.source.source_id + ): + raise ContinuityError( + "SOURCE_NON_AUTHORITATIVE", + "stream source transition requires a new snapshot", + ) + feed, interval, revision, payload = _payload(envelope) + if feed is not requirement.feed or interval != requirement.interval: + raise ContinuityError( + "CONFLICT", "stream feed/interval differs from requirement" + ) + if requirement.source_policy_id != template.quality.policy_id: + raise ContinuityError( + "CONFLICT", "stream handoff source policy differs from requirement" + ) + if int(envelope.authority_revision) < template.contract.authority_revision: + raise ContinuityError( + "SOURCE_NON_AUTHORITATIVE", "stream authority revision moved backwards" + ) + if ( + not envelope.normalizer_version + or not envelope.adapter_version + or envelope.config_revision < 1 + ): + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "stream contract metadata is incomplete" + ) + + flags = [ + common_pb2.QualityFlag.Name(value).removeprefix("QUALITY_FLAG_") + for value in envelope.quality_flags + ] + gap_open = any(value in _GAP_FLAGS for value in flags) + observed_at_ns = int(envelope.source_event_time_ns) + if feed is Feed.BAR: + observed_for_freshness = int(envelope.bar.close_time_ns) + else: + observed_for_freshness = observed_at_ns + freshness_ms = max( + 0, ((now_ns or time.time_ns()) - observed_for_freshness) // 1_000_000 + ) + stale = ( + requirement.max_freshness_ms is not None + and freshness_ms > requirement.max_freshness_ms + ) + state = "GAPPED" if gap_open else "STALE" if stale else "LIVE" + if gap_open and requirement.gap_policy.value in {"BLOCK", "PAUSE"}: + raise ContinuityError( + "OPEN_SEQUENCE_GAP", "stream event violates the requested gap policy" + ) + if stale and requirement.stale_policy.value in {"BLOCK", "PAUSE"}: + raise ContinuityError( + "DATA_STALE", "stream event violates the requested freshness policy" + ) + + authoritative = ( + template.source.authoritative + and template.source.source_role == "PRIMARY" + and state == "LIVE" + ) + if requirement.consumer_grade.value == "EXECUTION" and not authoritative: + code = ( + "OPEN_SEQUENCE_GAP" + if gap_open + else "DATA_STALE" + if stale + else "SOURCE_NON_AUTHORITATIVE" + ) + raise ContinuityError(code, "stream event is not execution eligible") + if ( + feed is Feed.BAR + and requirement.require_final_bars + and payload["lifecycle"] not in {"FINAL", "REVISED"} + ): + raise ContinuityError("DATA_NOT_READY", "stream bar is not final") + + return MarketDataView.model_validate( + { + "instrument_uid": envelope.instrument_uid, + "instrument_id": envelope.instrument_id, + "instrument_revision": int(envelope.instrument_revision), + "feed": feed.value, + "interval": interval, + "observed_at_ns": observed_at_ns, + "revision": revision, + "payload": payload, + "source": { + "venue": envelope.venue, + "provider": envelope.provider, + "source_id": envelope.source_id, + "source_role": _enum_name( + common_pb2.SourceRole, envelope.source_role, "SOURCE_ROLE_" + ), + "authoritative": template.source.authoritative, + }, + "quality": { + "state": state, + "freshness_ms": int(freshness_ms), + "gap_open": gap_open, + "complete": not gap_open, + "execution_eligible": authoritative, + "policy_id": requirement.source_policy_id, + "flags": flags, + }, + "contract": { + **template.contract.model_dump(mode="json"), + "normalizer_version": envelope.normalizer_version, + "adapter_version": envelope.adapter_version, + "authority_revision": int(envelope.authority_revision), + "config_revision": int(envelope.config_revision), + "correlation_id": envelope.correlation_id + or bytes(envelope.event_id).hex(), + }, + "cursor": event.resume_token, + "watermark_offset": event.logical_offset, + } + ) diff --git a/qdl_sdk/tls.py b/qdl_sdk/tls.py new file mode 100644 index 0000000..8ba1a63 --- /dev/null +++ b/qdl_sdk/tls.py @@ -0,0 +1,50 @@ +from __future__ import annotations + +import ssl +from dataclasses import dataclass +from pathlib import Path + +import grpc + + +@dataclass(frozen=True, slots=True) +class WorkloadTlsConfig: + """Mutual-TLS identity shared by the REST and gRPC transports.""" + + ca_file: str | Path + certificate_file: str | Path + private_key_file: str | Path + + def __post_init__(self) -> None: + paths = self.paths() + missing = [name for name, path in paths.items() if not path.is_file()] + if missing: + raise ValueError( + "Data Layer workload TLS files are unavailable: " + + ",".join(sorted(missing)) + ) + + def paths(self) -> dict[str, Path]: + return { + "ca_file": Path(self.ca_file).expanduser().resolve(), + "certificate_file": Path(self.certificate_file).expanduser().resolve(), + "private_key_file": Path(self.private_key_file).expanduser().resolve(), + } + + def ssl_context(self) -> ssl.SSLContext: + paths = self.paths() + context = ssl.create_default_context(cafile=str(paths["ca_file"])) + context.minimum_version = ssl.TLSVersion.TLSv1_2 + context.load_cert_chain( + certfile=str(paths["certificate_file"]), + keyfile=str(paths["private_key_file"]), + ) + return context + + def grpc_credentials(self) -> grpc.ChannelCredentials: + paths = self.paths() + return grpc.ssl_channel_credentials( + root_certificates=paths["ca_file"].read_bytes(), + private_key=paths["private_key_file"].read_bytes(), + certificate_chain=paths["certificate_file"].read_bytes(), + ) diff --git a/qdl_sdk/transport.py b/qdl_sdk/transport.py index 911fc6b..92059f7 100644 --- a/qdl_sdk/transport.py +++ b/qdl_sdk/transport.py @@ -1,7 +1,7 @@ from __future__ import annotations import ipaddress -from collections.abc import AsyncIterator +from collections.abc import AsyncIterator, Sequence from urllib.parse import urlsplit import grpc @@ -11,6 +11,7 @@ from qdl_sdk.credentials import CredentialProvider from qdl_sdk.errors import CursorExpiredError, DataLayerError, SlowConsumerError from qdl_sdk.models import ControlEvent, DataRequirement, Grade, StreamEvent +from qdl_sdk.tls import WorkloadTlsConfig class RestQueryTransport: @@ -21,16 +22,20 @@ def __init__( timeout_seconds: float = 10.0, client: httpx.AsyncClient | None = None, credential_provider: CredentialProvider | None = None, + tls: WorkloadTlsConfig | None = None, ) -> None: if timeout_seconds <= 0: raise ValueError("query timeout must be positive") self.base_url = base_url.rstrip("/") + if client is not None and tls is not None: + raise ValueError("provide either REST client or workload TLS, not both") self._owns_client = client is None self._credential_provider = credential_provider self._client = client or httpx.AsyncClient( base_url=self.base_url, timeout=httpx.Timeout(timeout_seconds), limits=httpx.Limits(max_connections=100, max_keepalive_connections=20), + verify=tls.ssl_context() if tls is not None else True, ) async def warmup(self, requirement: DataRequirement, *, consumer_id: str) -> dict: @@ -53,31 +58,75 @@ async def snapshot(self, requirement: DataRequirement, *, consumer_id: str) -> d ) return self._decode(response) + async def instruments( + self, + *, + consumer_id: str, + consumer_grade: Grade, + cursor: str | None = None, + limit: int = 500, + ) -> dict: + headers = await self._identity_headers(consumer_grade, consumer_id) + params: dict[str, str | int] = {"limit": limit} + if cursor is not None: + params["cursor"] = cursor + response = await self._client.get( + "/v2/instruments", params=params, headers=headers + ) + return self._decode(response) + + async def instrument( + self, + identity: str, + *, + consumer_id: str, + consumer_grade: Grade, + ) -> dict: + if not identity.strip(): + raise ValueError("instrument identity is required") + headers = await self._identity_headers(consumer_grade, consumer_id) + response = await self._client.get( + f"/v2/instruments/{identity}", headers=headers + ) + return self._decode(response) + async def close(self) -> None: if self._owns_client: await self._client.aclose() async def _headers(self, requirement: DataRequirement, consumer_id: str) -> dict[str, str]: + return await self._identity_headers(requirement.consumer_grade, consumer_id) + + async def _identity_headers( + self, consumer_grade: Grade, consumer_id: str + ) -> dict[str, str]: if self._credential_provider is None: raise DataLayerError( "UNAUTHENTICATED", "V2 REST transport requires a workload credential provider", retryable=False, ) + if not isinstance(consumer_grade, Grade): + raise TypeError("consumer_grade must use the typed SDK enum") token = await self._credential_provider.get_token() return { "Authorization": f"Bearer {token}", "X-QDL-Consumer-ID": consumer_id, - "X-QDL-Purpose": self._purpose(requirement), + "X-QDL-Purpose": self._purpose(consumer_grade), } @staticmethod - def _purpose(requirement: DataRequirement) -> str: + def _purpose(consumer_grade: Grade | DataRequirement) -> str: + grade = ( + consumer_grade.consumer_grade + if isinstance(consumer_grade, DataRequirement) + else consumer_grade + ) return { Grade.EXECUTION: "INTERNAL_EXECUTION", Grade.ALPHA: "INTERNAL_ALPHA", Grade.RESEARCH: "INTERNAL_RESEARCH", - }[requirement.consumer_grade] + }[grade] @staticmethod def _decode(response: httpx.Response) -> dict: @@ -95,26 +144,46 @@ def _decode(response: httpx.Response) -> dict: class GrpcStreamTransport: def __init__( self, - target: str, + target: str | Sequence[str], *, credentials: grpc.ChannelCredentials | None = None, + tls: WorkloadTlsConfig | None = None, allow_insecure_loopback: bool = False, credential_provider: CredentialProvider | None = None, ) -> None: - if not target.strip(): - raise ValueError("gRPC stream target is required") - self.target = target + raw_targets = (target,) if isinstance(target, str) else tuple(target) + targets = tuple( + value.strip() + for item in raw_targets + for value in item.split(",") + if value.strip() + ) + if not targets or len(set(targets)) != len(targets): + raise ValueError("gRPC stream targets must be non-empty and unique") + self.targets = targets + self.target = targets[0] + self._target_index = 0 self._credential_provider = credential_provider + if credentials is not None and tls is not None: + raise ValueError("provide either credentials or workload TLS, not both") + credentials = credentials or (tls.grpc_credentials() if tls is not None else None) if credentials is None: - if not allow_insecure_loopback or not self._is_loopback(target): + if not allow_insecure_loopback or not all( + self._is_loopback(value) for value in targets + ): raise ValueError("insecure gRPC is allowed only for explicit loopback tests") - self._channel = grpc.aio.insecure_channel(target) + self._channels = tuple(grpc.aio.insecure_channel(value) for value in targets) else: - self._channel = grpc.aio.secure_channel(target, credentials) - self._subscribe = self._channel.unary_stream( - "/qdl.query.v2.MarketDataStreamService/Subscribe", - request_serializer=query_pb2.SubscribeRequest.SerializeToString, - response_deserializer=query_pb2.SubscribeResponse.FromString, + self._channels = tuple( + grpc.aio.secure_channel(value, credentials) for value in targets + ) + self._subscribes = tuple( + channel.unary_stream( + "/qdl.query.v2.MarketDataStreamService/Subscribe", + request_serializer=query_pb2.SubscribeRequest.SerializeToString, + response_deserializer=query_pb2.SubscribeResponse.FromString, + ) + for channel in self._channels ) async def subscribe( @@ -144,7 +213,8 @@ async def subscribe( ("x-qdl-purpose", RestQueryTransport._purpose(requirement)), ) try: - async for response in self._subscribe(request, metadata=metadata): + subscribe = self._subscribes[self._target_index] + async for response in subscribe(request, metadata=metadata): record = response.record payload = record.WhichOneof("payload") if payload == "control": @@ -158,6 +228,9 @@ async def subscribe( yield StreamEvent(record.logical_offset, record.resume_token, record.event) except grpc.aio.AioRpcError as error: detail = error.details() or "gRPC stream failed" + if error.code() is grpc.StatusCode.UNAVAILABLE and len(self.targets) > 1: + self._target_index = (self._target_index + 1) % len(self.targets) + self.target = self.targets[self._target_index] if error.code() is grpc.StatusCode.OUT_OF_RANGE: raise CursorExpiredError("CURSOR_EXPIRED", detail, retryable=False) from error if error.code() is grpc.StatusCode.RESOURCE_EXHAUSTED: @@ -172,7 +245,8 @@ async def subscribe( raise DataLayerError("DEPENDENCY_UNAVAILABLE", detail, retryable=True) from error async def close(self) -> None: - await self._channel.close() + for channel in self._channels: + await channel.close() @staticmethod def _is_loopback(target: str) -> bool: diff --git a/rust/qdl-kafka/src/bin/qdl-production-core.rs b/rust/qdl-kafka/src/bin/qdl-production-core.rs new file mode 100644 index 0000000..05de444 --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-production-core.rs @@ -0,0 +1,661 @@ +#![forbid(unsafe_code)] + +use std::collections::{HashMap, HashSet}; +use std::env; +use std::sync::Arc; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use prost::Message; +use qdl_contracts::qdl::provider::v1::RawProviderEnvelope; +use qdl_core::backoff::BackoffPolicy; +use qdl_core::transport::RetryClass; +use qdl_kafka::phase92_runtime::{ + KafkaCompactedSnapshotReader, Phase92Decision, Phase92Progress, Phase92TargetCheckpoint, + Phase92TransactionalKafkaBridge, Phase92TransactionalOutput, Phase92TransactionalTopics, +}; +use qdl_kafka::{KafkaEventSource, KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError}; +use qdl_realtime_core::{ProcessBatch, RealtimeCore, RealtimeCoreConfig}; +use qdl_venue_core::authority::{ + Phase92AuthorityControlEvent, Phase92AuthorityState, Phase92PublicationContext, SinkTarget, +}; +use serde::Deserialize; +use serde_json::json; + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct RuntimeSliceBinding { + subscription_id: String, + slice_id: String, + shard_id: String, + raw_authority_revision: u64, + raw_lease_epoch: u64, + raw_partition_plan_epoch: u64, +} + +impl RuntimeSliceBinding { + fn validate(&self) -> Result<(), String> { + if self.subscription_id.trim().is_empty() + || self.slice_id.trim().is_empty() + || self.shard_id.trim().is_empty() + || self.raw_authority_revision == 0 + || self.raw_lease_epoch == 0 + || self.raw_partition_plan_epoch == 0 + { + return Err("production runtime slice binding is invalid".into()); + } + Ok(()) + } +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct ProductionRuntimeConfig { + core: RealtimeCoreConfig, + topics: ProductionTopicConfig, + slices: Vec, + transactional_id: String, + batch_size: usize, + batch_wait_ms: u64, + max_events: u64, + metrics_every_batches: u64, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct ProductionTopicConfig { + raw_inputs: Vec, + authority_control: String, + target_checkpoints: String, + canary_canonical: String, + primary_canonical: String, + public_v2: String, + legacy_v1: String, + quarantine: String, +} + +impl ProductionRuntimeConfig { + fn topics(&self) -> Phase92TransactionalTopics { + Phase92TransactionalTopics { + raw_inputs: self.topics.raw_inputs.clone(), + canary_canonical: self.topics.canary_canonical.clone(), + primary_canonical: self.topics.primary_canonical.clone(), + public_v2: self.topics.public_v2.clone(), + legacy_v1: self.topics.legacy_v1.clone(), + quarantine: self.topics.quarantine.clone(), + target_checkpoints: self.topics.target_checkpoints.clone(), + authority_control: self.topics.authority_control.clone(), + } + } + + fn validate(&self) -> Result<(), String> { + self.core.validate().map_err(|error| error.to_string())?; + self.topics() + .validate() + .map_err(|error| error.to_string())?; + if self.transactional_id.trim().is_empty() + || self.slices.is_empty() + || self.batch_size == 0 + || self.batch_size > 1_000 + || self.batch_wait_ms == 0 + || self.batch_wait_ms > 1_000 + || self.metrics_every_batches == 0 + { + return Err("production realtime core config bounds are invalid".into()); + } + let mut subscriptions = HashSet::new(); + let mut slices = HashSet::new(); + for binding in &self.slices { + binding.validate()?; + if !subscriptions.insert(binding.subscription_id.clone()) + || !slices.insert(binding.slice_id.clone()) + { + return Err( + "production runtime subscription/slice identities must be unique".into(), + ); + } + } + Ok(()) + } + + fn bindings(&self) -> HashMap { + self.slices + .iter() + .cloned() + .map(|binding| (binding.subscription_id.clone(), binding)) + .collect() + } +} + +type RuntimeError = Box; + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn kafka_config(group_suffix: &str) -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + let timeout_seconds = env::var("QDL_KAFKA_REQUEST_TIMEOUT_SECONDS") + .unwrap_or_else(|_| "30".into()) + .parse::() + .map_err(|_| "QDL_KAFKA_REQUEST_TIMEOUT_SECONDS must be positive")?; + if timeout_seconds == 0 || group_suffix.trim().is_empty() { + return Err("Kafka timeout/group suffix must be positive and non-empty".into()); + } + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: format!("{}-{group_suffix}", required("QDL_KAFKA_CLIENT_ID")?), + group_id: format!("{}-{group_suffix}", required("QDL_KAFKA_GROUP_ID")?), + request_timeout: Duration::from_secs(timeout_seconds), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/client.crt"), + key_location: format!("{cert_root}/client.key"), + key_password: None, + }, + }) +} + +fn now_ns() -> Result { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn should_retry_transport(class: RetryClass) -> bool { + class != RetryClass::NonRetryable +} + +fn retryable_runtime_error(error: &RuntimeError) -> bool { + error + .downcast_ref::() + .is_some_and(|value| should_retry_transport(value.retry_class())) +} + +fn expected_targets(state: Phase92AuthorityState) -> Result, String> { + match state { + Phase92AuthorityState::RustCanary => Ok(vec![SinkTarget::CanaryCanonical]), + Phase92AuthorityState::RustPrimary => Ok(vec![ + SinkTarget::PrimaryCanonical, + SinkTarget::PublicV2, + SinkTarget::LegacyV1, + ]), + Phase92AuthorityState::PythonPrimary + | Phase92AuthorityState::Blocked + | Phase92AuthorityState::RollbackPending => { + Err("Rust production core does not own this authority state".into()) + } + } +} + +async fn restore_authority( + bridge: &Phase92TransactionalKafkaBridge, + config: &ProductionRuntimeConfig, +) -> Result<(), RuntimeError> { + let snapshot_config = kafka_config(&format!("phase92-recovery-{}", config.transactional_id))?; + let records = KafkaCompactedSnapshotReader::new(snapshot_config)?.read(&[ + &config.topics.authority_control, + &config.topics.target_checkpoints, + ])?; + let required_slices: HashSet<_> = config + .slices + .iter() + .map(|binding| binding.slice_id.as_str()) + .collect(); + let mut events = HashMap::new(); + let mut checkpoints = Vec::new(); + for record in records { + if record.topic == config.topics.authority_control { + let event: Phase92AuthorityControlEvent = serde_json::from_slice(&record.payload)?; + event.validate()?; + if record.key != event.slice_id { + return Err("authority compacted key differs from event slice".into()); + } + if required_slices.contains(event.slice_id.as_str()) { + if event.authority.is_none() { + return Err("production slice has no Phase 9.2 authority record".into()); + } + events.insert(event.slice_id.clone(), event); + } + } else if record.topic == config.topics.target_checkpoints { + let checkpoint: Phase92TargetCheckpoint = serde_json::from_slice(&record.payload)?; + checkpoint.validate()?; + if record.key != checkpoint.key() { + return Err("target checkpoint compacted key differs from payload".into()); + } + if required_slices.contains(checkpoint.slice_id.as_str()) { + checkpoints.push(checkpoint); + } + } + } + if events.len() != required_slices.len() { + return Err("one or more production slices have no authority event".into()); + } + let restore_time = now_ns()?; + for binding in &config.slices { + let event = events + .get(&binding.slice_id) + .ok_or("production authority event is missing")?; + bridge.apply_authority_event(event, restore_time).await?; + let authority = event + .authority + .as_ref() + .ok_or("production authority record is missing")?; + let targets = expected_targets(authority.state)?; + let exact: Vec<_> = checkpoints + .iter() + .filter(|checkpoint| { + checkpoint.slice_id == binding.slice_id + && checkpoint.shard_id == binding.shard_id + && checkpoint.owner_id == authority.owner_id + && checkpoint.authority_revision == authority.authority_revision + && checkpoint.lease_epoch == authority.lease_epoch + && checkpoint.partition_plan_epoch == authority.partition_plan_epoch + && checkpoint.candidate_digest == authority.candidate_digest + && targets.contains(&checkpoint.target) + }) + .collect(); + if exact.len() == targets.len() { + for checkpoint in exact { + bridge.restore_checkpoint(checkpoint).await?; + } + } else if exact.is_empty() && authority.state == Phase92AuthorityState::RustPrimary { + let handoff = event + .handoff + .as_ref() + .ok_or("fresh Rust primary bootstrap requires accepted handoff")?; + let terminal = event + .checkpoint + .as_ref() + .ok_or("fresh Rust primary bootstrap requires terminal checkpoint")?; + if handoff.terminal_watermark != authority.start_watermark + || terminal.terminal_watermark != authority.start_watermark + { + return Err("fresh Rust primary bootstrap W differs from handoff evidence".into()); + } + for target in targets { + let checkpoint = Phase92TargetCheckpoint { + schema: "qdl.target-watermark-checkpoint.v1".into(), + slice_id: authority.slice_id.clone(), + owner_id: authority.owner_id.clone(), + authority_revision: authority.authority_revision, + lease_epoch: authority.lease_epoch, + partition_plan_epoch: authority.partition_plan_epoch, + shard_id: binding.shard_id.clone(), + target, + source_watermark: authority.start_watermark, + source_event_id: format!("handoff-{}", handoff.handoff_id), + decision: Phase92Decision::Filtered, + output_payload_sha256: "0".repeat(64), + candidate_digest: authority.candidate_digest.clone(), + committed_at_ns: restore_time, + }; + bridge.restore_checkpoint(&checkpoint).await?; + } + } else if !exact.is_empty() || authority.state != Phase92AuthorityState::RustCanary { + return Err("target checkpoint recovery is partial or inconsistent".into()); + } + } + Ok(()) +} + +async fn watch_authority( + bridge: Arc, + config: KafkaTransportConfig, + topic: String, + allowed_slices: HashSet, +) -> Result<(), RuntimeError> { + let source = KafkaEventSource::new(&config, &[&topic])?; + loop { + let (record, _) = source.next().await?; + let event: Phase92AuthorityControlEvent = serde_json::from_slice(&record.payload)?; + event.validate()?; + if record.partition_key != event.slice_id { + return Err("authority stream key differs from event slice".into()); + } + if !allowed_slices.contains(&event.slice_id) { + source.checkpoint()?; + continue; + } + let current = bridge.current_authority(&event.slice_id).await; + if current + .as_ref() + .is_some_and(|value| event.authority_revision < value.authority_revision) + { + source.checkpoint()?; + continue; + } + if event.authority.is_none() { + return Err("active production slice received non-Phase92 authority event".into()); + } + bridge.apply_authority_event(&event, now_ns()?).await?; + source.checkpoint()?; + } +} + +fn validate_raw_authority( + raw: &RawProviderEnvelope, + binding: &RuntimeSliceBinding, +) -> Result<(), RuntimeError> { + if raw.subscription_id != binding.subscription_id + || raw.authority_revision != binding.raw_authority_revision + || raw.lease_epoch != binding.raw_lease_epoch + || raw.partition_plan_epoch != binding.raw_partition_plan_epoch + { + return Err("raw acquisition identity/lease differs from approved binding".into()); + } + Ok(()) +} + +fn decision(result: &ProcessBatch) -> Result { + let active = [ + !result.canonical.is_empty(), + !result.quarantines.is_empty(), + result.duplicates > 0, + result.filtered > 0, + ]; + if active.into_iter().filter(|value| *value).count() != 1 { + return Err("normalizer produced an ambiguous decision for one raw event".into()); + } + if !result.canonical.is_empty() { + Ok(Phase92Decision::Canonical) + } else if !result.quarantines.is_empty() { + Ok(Phase92Decision::Quarantine) + } else if result.duplicates > 0 { + Ok(Phase92Decision::Duplicate) + } else { + Ok(Phase92Decision::Filtered) + } +} + +fn output_stream(topics: &ProductionTopicConfig, target: SinkTarget) -> Result<&str, RuntimeError> { + match target { + SinkTarget::CanaryCanonical => Ok(&topics.canary_canonical), + SinkTarget::PrimaryCanonical => Ok(&topics.primary_canonical), + SinkTarget::PublicV2 => Ok(&topics.public_v2), + SinkTarget::LegacyV1 => Ok(&topics.legacy_v1), + _ => Err("production runtime received a non-production target".into()), + } +} + +async fn run_generation( + config: &ProductionRuntimeConfig, + generation: u64, +) -> Result<(), RuntimeError> { + let mut core = RealtimeCore::new(config.core.clone())?; + let bridge = Arc::new(Phase92TransactionalKafkaBridge::new( + &kafka_config("phase92-raw")?, + config.topics(), + &config.transactional_id, + )?); + restore_authority(&bridge, config).await?; + let binding_by_subscription = config.bindings(); + let allowed_slices: HashSet<_> = config + .slices + .iter() + .map(|binding| binding.slice_id.clone()) + .collect(); + let mut authority_task = tokio::spawn(watch_authority( + Arc::clone(&bridge), + kafka_config(&format!("phase92-authority-{}", config.transactional_id))?, + config.topics.authority_control.clone(), + allowed_slices, + )); + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_started", + "generation": generation, + "slices": config.slices.len(), + "bindings": config.core.bindings.len(), + "authority_reconstructed": true, + "target_watermarks_reconstructed": true, + }))? + ); + + let mut processed = 0_u64; + let mut canonical = 0_u64; + let mut quarantines = 0_u64; + let mut duplicates = 0_u64; + let mut filtered = 0_u64; + let mut batches = 0_u64; + 'service: loop { + if config.max_events > 0 && processed >= config.max_events { + break; + } + let first = tokio::select! { + result = bridge.next() => result?, + result = tokio::signal::ctrl_c() => { + result?; + break 'service; + } + result = &mut authority_task => { + return match result { + Ok(Ok(())) => Err("authority watcher stopped unexpectedly".into()), + Ok(Err(error)) => Err(error), + Err(error) => Err(error.into()), + }; + } + }; + let mut inputs = vec![first]; + while inputs.len() < config.batch_size + && (config.max_events == 0 || processed + (inputs.len() as u64) < config.max_events) + { + match tokio::time::timeout(Duration::from_millis(config.batch_wait_ms), bridge.next()) + .await + { + Ok(Ok(input)) => inputs.push(input), + Ok(Err(error)) => return Err(error.into()), + Err(_) => break, + } + } + + let normalized_at_ns = now_ns()?; + let mut outputs = Vec::new(); + let mut progress = Vec::new(); + let mut local_next: HashMap<(String, String, SinkTarget), u64> = HashMap::new(); + for input in &inputs { + let raw = RawProviderEnvelope::decode(input.record.payload.as_slice())?; + let binding = binding_by_subscription + .get(&raw.subscription_id) + .ok_or("raw event has no approved production slice binding")?; + validate_raw_authority(&raw, binding)?; + let authority = bridge + .current_authority(&binding.slice_id) + .await + .ok_or("production slice authority disappeared")?; + let targets = expected_targets(authority.state)?; + let result = + core.process_at_transport_offset(raw, normalized_at_ns, input.cursor.offset)?; + let item_decision = decision(&result)?; + canonical += result.canonical.len() as u64; + quarantines += result.quarantines.len() as u64; + duplicates += result.duplicates as u64; + filtered += result.filtered as u64; + + for target in targets { + let key = (binding.slice_id.clone(), binding.shard_id.clone(), target); + let source_watermark = if let Some(next) = local_next.get_mut(&key) { + let value = *next; + *next = next.checked_add(1).ok_or("logical watermark overflow")?; + value + } else { + let value = bridge + .next_watermark(&binding.slice_id, &binding.shard_id, target) + .await?; + local_next.insert( + key, + value.checked_add(1).ok_or("logical watermark overflow")?, + ); + value + }; + let publication = Phase92PublicationContext { + slice_id: authority.slice_id.clone(), + owner_id: authority.owner_id.clone(), + authority_revision: authority.authority_revision, + shard_id: binding.shard_id.clone(), + lease_epoch: authority.lease_epoch, + partition_plan_epoch: authority.partition_plan_epoch, + source_watermark, + target, + }; + progress.push(Phase92Progress { + publication: publication.clone(), + decision: item_decision, + source_cursor: input.cursor.clone(), + source_event_id: input.record.event_id.clone(), + }); + if item_decision == Phase92Decision::Canonical { + for record in &result.canonical { + let mut projected = record.clone(); + projected.stream = output_stream(&config.topics, target)?.into(); + outputs.push(Phase92TransactionalOutput { + record: projected, + publication: publication.clone(), + raw_provider_envelope: Some(input.record.payload.clone()), + }); + } + } else if item_decision == Phase92Decision::Quarantine + && target == SinkTarget::PrimaryCanonical + { + for record in &result.quarantines { + let mut quarantined = record.clone(); + quarantined.stream = config.topics.quarantine.clone(); + outputs.push(Phase92TransactionalOutput { + record: quarantined, + publication: publication.clone(), + raw_provider_envelope: Some(input.record.payload.clone()), + }); + } + } + } + } + bridge + .commit(&inputs, &outputs, &progress, normalized_at_ns) + .await?; + processed += inputs.len() as u64; + batches += 1; + if batches % config.metrics_every_batches == 0 { + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_progress", + "generation": generation, + "processed": processed, + "canonical": canonical, + "quarantines": quarantines, + "duplicates": duplicates, + "filtered": filtered, + "batches": batches, + }))? + ); + } + } + authority_task.abort(); + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_stopped", + "generation": generation, + "processed": processed, + "canonical": canonical, + "quarantines": quarantines, + "duplicates": duplicates, + "filtered": filtered, + "batches": batches, + }))? + ); + Ok(()) +} + +#[tokio::main] +async fn main() -> Result<(), RuntimeError> { + let config_path = env::args() + .nth(1) + .ok_or("usage: qdl-production-core CONFIG.json")?; + let config: ProductionRuntimeConfig = + serde_json::from_slice(&tokio::fs::read(config_path).await?)?; + config.validate()?; + let backoff = BackoffPolicy { + initial_ms: 500, + maximum_ms: 30_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate()?; + let mut generation = 0_u64; + let mut failures = 0_u32; + loop { + generation = generation.saturating_add(1); + match run_generation(&config, generation).await { + Ok(()) => return Ok(()), + Err(error) if retryable_runtime_error(&error) => { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_retry", + "generation": generation, + "attempt": failures, + "error": error.to_string(), + }))? + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + Err(error) => return Err(error), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn decision_is_exclusive_and_fail_closed() { + let canonical = ProcessBatch { + canonical: vec![qdl_core::transport::DurableRecord { + stream: "canonical".into(), + partition_key: "key".into(), + event_id: vec![1], + payload: vec![2], + accepted_at_ns: 1, + }], + quarantines: vec![], + duplicates: 0, + filtered: 0, + }; + assert_eq!(decision(&canonical).unwrap(), Phase92Decision::Canonical); + let ambiguous = ProcessBatch { + filtered: 1, + ..canonical + }; + assert!(decision(&ambiguous).is_err()); + } + + #[test] + fn raw_and_publication_authorities_are_explicitly_separate() { + let raw = RawProviderEnvelope { + subscription_id: "binance-btc-trade".into(), + authority_revision: 1, + lease_epoch: 2, + partition_plan_epoch: 3, + ..Default::default() + }; + let binding = RuntimeSliceBinding { + subscription_id: "binance-btc-trade".into(), + slice_id: "production/binance/usdm/perpetual/trade/plan-1/btcusdt".into(), + shard_id: "btcusdt-trade".into(), + raw_authority_revision: 1, + raw_lease_epoch: 2, + raw_partition_plan_epoch: 3, + }; + validate_raw_authority(&raw, &binding).unwrap(); + let mut stale = raw; + stale.lease_epoch = 1; + assert!(validate_raw_authority(&stale, &binding).is_err()); + } +} diff --git a/rust/qdl-kafka/src/lib.rs b/rust/qdl-kafka/src/lib.rs index d76e728..ca322a9 100644 --- a/rust/qdl-kafka/src/lib.rs +++ b/rust/qdl-kafka/src/lib.rs @@ -1,5 +1,7 @@ #![forbid(unsafe_code)] +pub mod phase92_runtime; + use std::fmt::{Display, Formatter}; use std::path::Path; use std::time::Duration; diff --git a/rust/qdl-kafka/src/phase92_runtime.rs b/rust/qdl-kafka/src/phase92_runtime.rs new file mode 100644 index 0000000..75267b2 --- /dev/null +++ b/rust/qdl-kafka/src/phase92_runtime.rs @@ -0,0 +1,792 @@ +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::time::{Duration, Instant}; + +use futures_util::future::try_join_all; +use qdl_core::transport::{AppendResult, Cursor, DurableRecord}; +use qdl_venue_core::authority::{ + Phase92AuthorityControlEvent, Phase92AuthorityFence, Phase92AuthorityRecord, + Phase92PublicationContext, SinkTarget, +}; +use rdkafka::consumer::{BaseConsumer, Consumer, StreamConsumer}; +use rdkafka::message::{Header, Headers, Message, OwnedHeaders}; +use rdkafka::producer::{FutureProducer, FutureRecord, Producer}; +use rdkafka::topic_partition_list::{Offset, TopicPartitionList}; +use rdkafka::util::Timeout; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +use super::{ + transactional_output_headers, KafkaTransportConfig, KafkaTransportError, + TransactionalKafkaInput, EVENT_ID_HEADER, +}; + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92TransactionalTopics { + pub raw_inputs: Vec, + pub canary_canonical: String, + pub primary_canonical: String, + pub public_v2: String, + pub legacy_v1: String, + pub quarantine: String, + pub target_checkpoints: String, + pub authority_control: String, +} + +impl Phase92TransactionalTopics { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + let outputs = [ + self.canary_canonical.as_str(), + self.primary_canonical.as_str(), + self.public_v2.as_str(), + self.legacy_v1.as_str(), + self.quarantine.as_str(), + self.target_checkpoints.as_str(), + self.authority_control.as_str(), + ]; + if self.raw_inputs.is_empty() + || self.raw_inputs.iter().any(|topic| topic.trim().is_empty()) + || outputs.iter().any(|topic| topic.trim().is_empty()) + { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transactional topics must not be empty".into(), + )); + } + let mut unique = HashSet::new(); + for topic in self.raw_inputs.iter().map(String::as_str).chain(outputs) { + if !unique.insert(topic) { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transactional topics must be isolated and unique".into(), + )); + } + } + Ok(()) + } + + fn permits_output(&self, target: SinkTarget, stream: &str) -> bool { + match target { + SinkTarget::CanaryCanonical => stream == self.canary_canonical, + SinkTarget::PrimaryCanonical => { + stream == self.primary_canonical || stream == self.quarantine + } + SinkTarget::PublicV2 => stream == self.public_v2, + SinkTarget::LegacyV1 => stream == self.legacy_v1, + SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical | SinkTarget::ShadowQuarantine => { + false + } + } + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase92Decision { + Canonical, + Quarantine, + Filtered, + Duplicate, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92Progress { + pub publication: Phase92PublicationContext, + pub decision: Phase92Decision, + pub source_cursor: Cursor, + pub source_event_id: Vec, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92TransactionalOutput { + pub record: DurableRecord, + pub publication: Phase92PublicationContext, + pub raw_provider_envelope: Option>, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92TargetCheckpoint { + pub schema: String, + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub shard_id: String, + pub target: SinkTarget, + pub source_watermark: u64, + pub source_event_id: String, + pub decision: Phase92Decision, + pub output_payload_sha256: String, + pub candidate_digest: String, + pub committed_at_ns: i64, +} + +impl Phase92TargetCheckpoint { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + if self.schema != "qdl.target-watermark-checkpoint.v1" + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || self.shard_id.trim().is_empty() + || self.source_event_id.len() < 2 + || self.output_payload_sha256.len() != 64 + || !self + .output_payload_sha256 + .bytes() + .all(|byte| byte.is_ascii_hexdigit()) + || self.candidate_digest.len() != 64 + || !self + .candidate_digest + .bytes() + .all(|byte| byte.is_ascii_hexdigit()) + || self.committed_at_ns <= 0 + || !matches!( + self.target, + SinkTarget::CanaryCanonical + | SinkTarget::PrimaryCanonical + | SinkTarget::PublicV2 + | SinkTarget::LegacyV1 + ) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 target checkpoint is invalid".into(), + )); + } + Ok(()) + } + + pub fn key(&self) -> String { + format!( + "{}|{}|{}", + self.slice_id, + self.shard_id, + target_name(self.target) + ) + } + + pub fn publication(&self) -> Phase92PublicationContext { + Phase92PublicationContext { + slice_id: self.slice_id.clone(), + owner_id: self.owner_id.clone(), + authority_revision: self.authority_revision, + shard_id: self.shard_id.clone(), + lease_epoch: self.lease_epoch, + partition_plan_epoch: self.partition_plan_epoch, + source_watermark: self.source_watermark, + target: self.target, + } + } +} + +fn target_name(target: SinkTarget) -> &'static str { + match target { + SinkTarget::ShadowRaw => "SHADOW_RAW", + SinkTarget::ShadowCanonical => "SHADOW_CANONICAL", + SinkTarget::ShadowQuarantine => "SHADOW_QUARANTINE", + SinkTarget::CanaryCanonical => "CANARY_CANONICAL", + SinkTarget::PrimaryCanonical => "PRIMARY_CANONICAL", + SinkTarget::PublicV2 => "PUBLIC_V2", + SinkTarget::LegacyV1 => "LEGACY_V1", + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct CompactedKafkaRecord { + pub topic: String, + pub key: String, + pub payload: Vec, + pub event_id: Option>, + pub partition: i32, + pub offset: i64, + pub accepted_at_ns: i64, +} + +pub struct KafkaCompactedSnapshotReader { + config: KafkaTransportConfig, +} + +impl KafkaCompactedSnapshotReader { + pub fn new(config: KafkaTransportConfig) -> Result { + config.validate()?; + Ok(Self { config }) + } + + pub fn read(&self, topics: &[&str]) -> Result, KafkaTransportError> { + if topics.is_empty() || topics.iter().any(|topic| topic.trim().is_empty()) { + return Err(KafkaTransportError::Configuration( + "compacted snapshot topics must not be empty".into(), + )); + } + let mut client = self.config.client_config()?; + client + .set("group.id", &self.config.group_id) + .set("enable.auto.commit", "false") + .set("enable.auto.offset.store", "false") + .set("auto.offset.reset", "earliest") + .set("isolation.level", "read_committed") + .set("enable.partition.eof", "true"); + let consumer: BaseConsumer = client.create()?; + let timeout = Timeout::After(self.config.request_timeout); + let mut assignment = TopicPartitionList::new(); + let mut remaining = BTreeMap::new(); + for topic in topics { + let metadata = consumer.fetch_metadata(Some(topic), timeout)?; + let metadata_topic = metadata + .topics() + .iter() + .find(|value| value.name() == *topic) + .ok_or(KafkaTransportError::MissingField( + "compacted topic metadata", + ))?; + if metadata_topic.partitions().is_empty() { + return Err(KafkaTransportError::Configuration(format!( + "compacted topic has no partitions: {topic}" + ))); + } + for partition in metadata_topic.partitions() { + let partition_id = partition.id(); + let (low, high) = consumer.fetch_watermarks(topic, partition_id, timeout)?; + assignment.add_partition_offset(topic, partition_id, Offset::Beginning)?; + if high > low { + remaining.insert(((*topic).to_owned(), partition_id), high - 1); + } + } + } + consumer.assign(&assignment)?; + + let deadline = Instant::now() + self.config.request_timeout; + let mut latest: HashMap<(String, String), CompactedKafkaRecord> = HashMap::new(); + while !remaining.is_empty() { + if Instant::now() >= deadline { + return Err(KafkaTransportError::Configuration( + "compacted snapshot did not reach captured high watermarks".into(), + )); + } + let Some(result) = consumer.poll(Duration::from_millis(100)) else { + continue; + }; + let message = match result { + Ok(message) => message, + Err(rdkafka::error::KafkaError::PartitionEOF(_)) => continue, + Err(error) => return Err(error.into()), + }; + let topic = message.topic().to_owned(); + let partition = message.partition(); + let key_bytes = message + .key() + .ok_or(KafkaTransportError::MissingField("compacted record key"))?; + let key = std::str::from_utf8(key_bytes) + .map_err(|_| KafkaTransportError::InvalidUtf8("compacted record key"))? + .to_owned(); + let identity = (topic.clone(), key.clone()); + if let Some(payload) = message.payload() { + let event_id = message.headers().and_then(|headers| { + headers + .iter() + .find(|header| header.key == EVENT_ID_HEADER) + .and_then(|header| header.value.map(ToOwned::to_owned)) + }); + latest.insert( + identity, + CompactedKafkaRecord { + topic: topic.clone(), + key, + payload: payload.to_vec(), + event_id, + partition, + offset: message.offset(), + accepted_at_ns: message.timestamp().to_millis().unwrap_or_default() + * 1_000_000, + }, + ); + } else { + latest.remove(&identity); + } + if remaining + .get(&(topic.clone(), partition)) + .is_some_and(|high| message.offset() >= *high) + { + remaining.remove(&(topic, partition)); + } + } + let mut records: Vec<_> = latest.into_values().collect(); + records.sort_by(|left, right| { + (&left.topic, &left.key, left.partition, left.offset).cmp(&( + &right.topic, + &right.key, + right.partition, + right.offset, + )) + }); + Ok(records) + } +} + +#[derive(Debug)] +pub struct Phase92CommitResult { + pub outputs: Vec, + pub checkpoints: Vec, +} + +pub struct Phase92TransactionalKafkaBridge { + producer: FutureProducer, + consumer: StreamConsumer, + fences: tokio::sync::Mutex>, + topics: Phase92TransactionalTopics, + request_timeout: Duration, +} + +impl Phase92TransactionalKafkaBridge { + pub fn new( + config: &KafkaTransportConfig, + topics: Phase92TransactionalTopics, + transactional_id: &str, + ) -> Result { + topics.validate()?; + if transactional_id.trim().is_empty() { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transactional.id must not be empty".into(), + )); + } + let mut consumer_config = config.client_config()?; + consumer_config + .set("group.id", &config.group_id) + .set("enable.auto.commit", "false") + .set("enable.auto.offset.store", "false") + .set("auto.offset.reset", "earliest") + .set("isolation.level", "read_committed"); + let consumer: StreamConsumer = consumer_config.create()?; + let raw_topics: Vec<&str> = topics.raw_inputs.iter().map(String::as_str).collect(); + consumer.subscribe(&raw_topics)?; + + let mut producer_config = config.client_config()?; + producer_config + .set("transactional.id", transactional_id) + .set("enable.idempotence", "true") + .set("acks", "all") + .set("max.in.flight.requests.per.connection", "5") + .set("retries", "2147483647") + .set("compression.type", "zstd") + .set( + "transaction.timeout.ms", + config.request_timeout.as_millis().to_string(), + ) + .set( + "delivery.timeout.ms", + config.request_timeout.as_millis().to_string(), + ); + let producer: FutureProducer = producer_config.create()?; + producer.init_transactions(Timeout::After(config.request_timeout))?; + Ok(Self { + producer, + consumer, + fences: tokio::sync::Mutex::new(HashMap::new()), + topics, + request_timeout: config.request_timeout, + }) + } + + pub async fn apply_authority_event( + &self, + event: &Phase92AuthorityControlEvent, + now_ns: i64, + ) -> Result<(), KafkaTransportError> { + let mut fences = self.fences.lock().await; + fences + .entry(event.slice_id.clone()) + .or_default() + .apply_control_event(event, now_ns) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn restore_checkpoint( + &self, + checkpoint: &Phase92TargetCheckpoint, + ) -> Result<(), KafkaTransportError> { + checkpoint.validate()?; + self.fences + .lock() + .await + .get_mut(&checkpoint.slice_id) + .ok_or_else(|| { + KafkaTransportError::Fencing("checkpoint slice authority is not loaded".into()) + })? + .restore_committed_watermark(&checkpoint.publication()) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn current_authority(&self, slice_id: &str) -> Option { + self.fences + .lock() + .await + .get(slice_id) + .and_then(|fence| fence.current().cloned()) + } + + pub async fn next_watermark( + &self, + slice_id: &str, + shard_id: &str, + target: SinkTarget, + ) -> Result { + self.fences + .lock() + .await + .get(slice_id) + .ok_or_else(|| { + KafkaTransportError::Fencing("publication slice authority is not loaded".into()) + })? + .next_watermark(shard_id, target) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn next(&self) -> Result { + let message = self.consumer.recv().await?; + let payload = message + .payload() + .ok_or(KafkaTransportError::MissingField("payload"))? + .to_vec(); + let key = message + .key() + .ok_or(KafkaTransportError::MissingField("partition_key"))?; + let partition_key = std::str::from_utf8(key) + .map_err(|_| KafkaTransportError::InvalidUtf8("partition_key"))? + .to_owned(); + let event_id = message + .headers() + .and_then(|headers| { + headers + .iter() + .find(|header| header.key == EVENT_ID_HEADER) + .and_then(|header| header.value.map(ToOwned::to_owned)) + }) + .ok_or(KafkaTransportError::MissingField("event_id header"))?; + let offset = u64::try_from(message.offset()) + .map_err(|_| KafkaTransportError::InvalidOffset(message.offset()))?; + Ok(TransactionalKafkaInput { + record: DurableRecord { + stream: message.topic().to_owned(), + partition_key: partition_key.clone(), + event_id, + payload, + accepted_at_ns: message.timestamp().to_millis().unwrap_or_default() * 1_000_000, + }, + cursor: Cursor { + stream: message.topic().to_owned(), + transport_partition: message.partition(), + partition_key, + offset, + }, + }) + } + + pub async fn commit( + &self, + inputs: &[TransactionalKafkaInput], + outputs: &[Phase92TransactionalOutput], + progress: &[Phase92Progress], + now_ns: i64, + ) -> Result { + if inputs.is_empty() || progress.is_empty() || now_ns <= 0 { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transaction requires input, progress and time".into(), + )); + } + if inputs + .iter() + .any(|input| !self.topics.raw_inputs.contains(&input.cursor.stream)) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 transaction input is outside raw topics".into(), + )); + } + let mut progress_identities = HashSet::new(); + for item in progress { + let identity = ( + item.publication.slice_id.clone(), + item.publication.shard_id.clone(), + item.publication.target, + item.publication.source_watermark, + ); + if !progress_identities.insert(identity) { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 transaction has duplicate target progress".into(), + )); + } + if item.source_event_id.is_empty() + || !inputs.iter().any(|input| { + input.cursor == item.source_cursor + && input.record.event_id == item.source_event_id + }) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 progress has no exact matching raw input".into(), + )); + } + } + for output in outputs { + if !self + .topics + .permits_output(output.publication.target, &output.record.stream) + || !progress + .iter() + .any(|item| item.publication == output.publication) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 output target/topic/progress binding failed".into(), + )); + } + } + + let mut fences = self.fences.lock().await; + let mut next_fences = fences.clone(); + for item in progress { + let next_fence = next_fences + .get_mut(&item.publication.slice_id) + .ok_or_else(|| { + KafkaTransportError::Fencing("progress slice authority is not loaded".into()) + })?; + next_fence + .permits(&item.publication, now_ns) + .map_err(KafkaTransportError::Fencing)?; + next_fence + .commit(&item.publication) + .map_err(KafkaTransportError::Fencing)?; + } + let checkpoints = progress + .iter() + .map(|item| { + let authority = next_fences + .get(&item.publication.slice_id) + .and_then(Phase92AuthorityFence::current) + .ok_or_else(|| { + KafkaTransportError::Fencing( + "checkpoint slice authority is not loaded".into(), + ) + })?; + let mut output_digest = Sha256::new(); + for output in outputs + .iter() + .filter(|output| output.publication == item.publication) + { + output_digest.update( + u64::try_from(output.record.payload.len()) + .map_err(|_| { + KafkaTransportError::Configuration( + "output payload length overflow".into(), + ) + })? + .to_be_bytes(), + ); + output_digest.update(&output.record.payload); + } + let output_payload_sha256 = hex::encode(output_digest.finalize()); + let checkpoint = Phase92TargetCheckpoint { + schema: "qdl.target-watermark-checkpoint.v1".into(), + slice_id: item.publication.slice_id.clone(), + owner_id: item.publication.owner_id.clone(), + authority_revision: item.publication.authority_revision, + lease_epoch: item.publication.lease_epoch, + partition_plan_epoch: item.publication.partition_plan_epoch, + shard_id: item.publication.shard_id.clone(), + target: item.publication.target, + source_watermark: item.publication.source_watermark, + source_event_id: hex::encode(&item.source_event_id), + decision: item.decision, + output_payload_sha256, + candidate_digest: authority.candidate_digest.clone(), + committed_at_ns: now_ns, + }; + checkpoint.validate()?; + let payload = serde_json::to_vec(&checkpoint).map_err(|error| { + KafkaTransportError::Configuration(format!( + "target checkpoint serialization failed: {error}" + )) + })?; + let event_id = Sha256::digest(&payload).to_vec(); + Ok::<_, KafkaTransportError>((checkpoint, payload, event_id)) + }) + .collect::, _>>()?; + + self.producer.begin_transaction()?; + let transaction = async { + let output_deliveries = outputs.iter().map(|output| async { + let headers = transactional_output_headers( + output.record.event_id.as_slice(), + output.raw_provider_envelope.as_deref(), + ); + deliver( + &self.producer, + &output.record.stream, + output.record.partition_key.as_bytes(), + output.record.payload.as_slice(), + headers, + self.request_timeout, + ) + .await + }); + let accepted_outputs = try_join_all(output_deliveries).await?; + + let checkpoint_deliveries = + checkpoints + .iter() + .map(|(checkpoint, payload, event_id)| async { + let headers = OwnedHeaders::new().insert(Header { + key: EVENT_ID_HEADER, + value: Some(event_id.as_slice()), + }); + deliver( + &self.producer, + &self.topics.target_checkpoints, + checkpoint.key().as_bytes(), + payload.as_slice(), + headers, + self.request_timeout, + ) + .await + }); + let accepted_checkpoints = try_join_all(checkpoint_deliveries).await?; + + let mut next_offsets: BTreeMap<(String, i32), i64> = BTreeMap::new(); + for input in inputs { + let next_offset = input + .cursor + .offset + .checked_add(1) + .and_then(|value| i64::try_from(value).ok()) + .ok_or(KafkaTransportError::InvalidOffset(i64::MAX))?; + next_offsets + .entry(( + input.cursor.stream.clone(), + input.cursor.transport_partition, + )) + .and_modify(|current| *current = (*current).max(next_offset)) + .or_insert(next_offset); + } + let mut offsets = TopicPartitionList::new(); + for ((topic, partition), offset) in next_offsets { + offsets.add_partition_offset(&topic, partition, Offset::Offset(offset))?; + } + let group = self.consumer.group_metadata().ok_or_else(|| { + KafkaTransportError::Configuration( + "Phase 9.2 consumer group metadata is unavailable".into(), + ) + })?; + self.producer.send_offsets_to_transaction( + &offsets, + &group, + Timeout::After(self.request_timeout), + )?; + self.producer + .commit_transaction(Timeout::After(self.request_timeout))?; + Ok::<_, KafkaTransportError>(Phase92CommitResult { + outputs: accepted_outputs, + checkpoints: accepted_checkpoints, + }) + } + .await; + match transaction { + Ok(result) => { + *fences = next_fences; + Ok(result) + } + Err(error) => { + self.producer + .abort_transaction(Timeout::After(self.request_timeout)) + .map_err(KafkaTransportError::Kafka)?; + Err(error) + } + } + } +} + +async fn deliver( + producer: &FutureProducer, + topic: &str, + key: &[u8], + payload: &[u8], + headers: OwnedHeaders, + timeout: Duration, +) -> Result { + let delivery = producer + .send( + FutureRecord::to(topic) + .key(key) + .payload(payload) + .headers(headers), + Timeout::After(timeout), + ) + .await + .map_err(|(error, _)| KafkaTransportError::Delivery(error))?; + let offset = u64::try_from(delivery.offset) + .map_err(|_| KafkaTransportError::InvalidOffset(delivery.offset))?; + Ok(AppendResult { + cursor: Cursor { + stream: topic.to_owned(), + transport_partition: delivery.partition, + partition_key: std::str::from_utf8(key) + .map_err(|_| KafkaTransportError::InvalidUtf8("output key"))? + .to_owned(), + offset, + }, + duplicate: false, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn topics() -> Phase92TransactionalTopics { + Phase92TransactionalTopics { + raw_inputs: vec!["raw".into()], + canary_canonical: "canary".into(), + primary_canonical: "canonical".into(), + public_v2: "public".into(), + legacy_v1: "legacy".into(), + quarantine: "quarantine".into(), + target_checkpoints: "checkpoints".into(), + authority_control: "authority".into(), + } + } + + #[test] + fn production_topics_are_unique_and_target_bound() { + let values = topics(); + values.validate().unwrap(); + assert!(values.permits_output(SinkTarget::PrimaryCanonical, "canonical")); + assert!(values.permits_output(SinkTarget::PrimaryCanonical, "quarantine")); + assert!(values.permits_output(SinkTarget::PublicV2, "public")); + assert!(values.permits_output(SinkTarget::LegacyV1, "legacy")); + assert!(values.permits_output(SinkTarget::CanaryCanonical, "canary")); + let mut duplicate = values; + duplicate.public_v2 = "canonical".into(); + assert!(duplicate.validate().is_err()); + } + + #[test] + fn checkpoint_roundtrip_preserves_exact_identity() { + let value = Phase92TargetCheckpoint { + schema: "qdl.target-watermark-checkpoint.v1".into(), + slice_id: "production/binance/usdm/perpetual/trade/plan-1/btcusdt".into(), + owner_id: "rust-primary".into(), + authority_revision: 4, + lease_epoch: 2, + partition_plan_epoch: 1, + shard_id: "core-1".into(), + target: SinkTarget::PublicV2, + source_watermark: 101, + source_event_id: "00".repeat(16), + decision: Phase92Decision::Canonical, + output_payload_sha256: "1".repeat(64), + candidate_digest: "2".repeat(64), + committed_at_ns: 1, + }; + value.validate().unwrap(); + let decoded: Phase92TargetCheckpoint = + serde_json::from_slice(&serde_json::to_vec(&value).unwrap()).unwrap(); + assert_eq!(decoded, value); + assert!(decoded.key().ends_with("|PUBLIC_V2")); + } +} diff --git a/rust/qdl-venue-core/src/authority.rs b/rust/qdl-venue-core/src/authority.rs index 0852c1f..512824e 100644 --- a/rust/qdl-venue-core/src/authority.rs +++ b/rust/qdl-venue-core/src/authority.rs @@ -658,6 +658,74 @@ impl Phase92AuthorityRecord { } } +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92AuthorityControlEvent { + pub schema: String, + pub event_id: String, + pub slice_id: String, + pub authority_revision: u64, + pub database_state: String, + pub authority: Option, + pub checkpoint: Option, + pub handoff: Option, +} + +impl Phase92AuthorityControlEvent { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.authority-control-event.v1" + || !valid_uuid(&self.event_id) + || self.slice_id.trim().is_empty() + || self.authority_revision == 0 + || self.database_state.trim().is_empty() + { + return Err("Phase 9.2 authority control event identity is invalid".into()); + } + let Some(authority) = &self.authority else { + if self.checkpoint.is_some() || self.handoff.is_some() { + return Err("non-writable authority event cannot carry handoff evidence".into()); + } + return Ok(()); + }; + authority.validate()?; + if authority.slice_id != self.slice_id + || authority.authority_revision != self.authority_revision + { + return Err("authority control event and authority record differ".into()); + } + let primary = matches!( + authority.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + if primary { + let checkpoint = self + .checkpoint + .as_ref() + .ok_or_else(|| "primary authority control event needs checkpoint".to_owned())?; + let handoff = self + .handoff + .as_ref() + .ok_or_else(|| "primary authority control event needs handoff".to_owned())?; + handoff.validate(checkpoint)?; + let handoff_digest = handoff.digest()?; + if authority.owner_id != handoff.new_owner_id + || authority.previous_owner_id.as_deref() != Some(handoff.old_owner_id.as_str()) + || authority.authority_revision != handoff.new_authority_revision + || authority.lease_epoch != handoff.new_lease_epoch + || authority.partition_plan_epoch != handoff.partition_plan_epoch + || authority.start_watermark != handoff.terminal_watermark + || authority.terminal_watermark != Some(handoff.terminal_watermark) + || authority.handoff_digest.as_deref() != Some(handoff_digest.as_str()) + { + return Err("primary authority record does not bind accepted handoff".into()); + } + } else if self.checkpoint.is_some() || self.handoff.is_some() { + return Err("non-primary authority event cannot carry handoff evidence".into()); + } + Ok(()) + } +} + #[derive(Clone, Debug, Eq, PartialEq)] pub struct Phase92PublicationContext { pub slice_id: String, @@ -670,7 +738,7 @@ pub struct Phase92PublicationContext { pub target: SinkTarget, } -#[derive(Default)] +#[derive(Clone, Default)] pub struct Phase92AuthorityFence { current: Option, committed_watermarks: HashMap<(String, SinkTarget), u64>, @@ -680,7 +748,15 @@ pub struct Phase92AuthorityFence { impl Phase92AuthorityFence { pub fn apply(&mut self, record: Phase92AuthorityRecord) -> Result<(), String> { record.validate()?; - if self.current.is_none() { + if let Some(current) = &self.current { + if record.authority_revision == current.authority_revision { + return if record == *current { + Ok(()) + } else { + Err("conflicting Phase 9.2 authority at the same revision".into()) + }; + } + } else { self.recovery_required = matches!( record.state, Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary @@ -697,6 +773,41 @@ impl Phase92AuthorityFence { self.apply_transition(record) } + pub fn apply_control_event( + &mut self, + event: &Phase92AuthorityControlEvent, + now_ns: i64, + ) -> Result<(), String> { + event.validate()?; + let Some(record) = event.authority.clone() else { + return Ok(()); + }; + if self + .current + .as_ref() + .is_some_and(|current| current == &record) + { + return Ok(()); + } + let primary = matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + if primary && self.current.is_some() { + let checkpoint = event + .checkpoint + .as_ref() + .ok_or_else(|| "primary authority event needs checkpoint".to_owned())?; + let handoff = event + .handoff + .as_ref() + .ok_or_else(|| "primary authority event needs handoff".to_owned())?; + self.apply_handoff(checkpoint, handoff, record, now_ns) + } else { + self.apply(record) + } + } + pub fn apply_handoff( &mut self, checkpoint: &Phase92TerminalCheckpoint, @@ -780,26 +891,26 @@ impl Phase92AuthorityFence { &mut self, context: &Phase92PublicationContext, ) -> Result<(), String> { - if !self.recovery_required { - return Err("Phase 9.2 watermark restore is only permitted during recovery".into()); - } let current = self .current .as_ref() .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; - if !matches!( + let restoring_canary = current.state == Phase92AuthorityState::RustCanary + && context.target == SinkTarget::CanaryCanonical; + let restoring_primary = matches!( current.state, Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary - ) || context.slice_id != current.slice_id + ) && matches!( + context.target, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 + ); + if !(restoring_canary || (self.recovery_required && restoring_primary)) + || context.slice_id != current.slice_id || context.owner_id != current.owner_id || context.authority_revision != current.authority_revision || context.lease_epoch != current.lease_epoch || context.partition_plan_epoch != current.partition_plan_epoch || context.shard_id.trim().is_empty() - || !matches!( - context.target, - SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 - ) || context.source_watermark < current.start_watermark { return Err("Phase 9.2 recovered watermark identity is invalid".into()); @@ -901,6 +1012,22 @@ impl Phase92AuthorityFence { Ok(()) } + pub fn next_watermark(&self, shard_id: &str, target: SinkTarget) -> Result { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if shard_id.trim().is_empty() { + return Err("Phase 9.2 shard identity is empty".into()); + } + self.committed_watermarks + .get(&(shard_id.to_owned(), target)) + .copied() + .unwrap_or(current.start_watermark) + .checked_add(1) + .ok_or_else(|| "Phase 9.2 watermark overflow".to_owned()) + } + pub fn current(&self) -> Option<&Phase92AuthorityRecord> { self.current.as_ref() } @@ -1540,3 +1667,108 @@ mod phase92_tests { assert!(dirty.validate(&checkpoint).is_err()); } } + +#[cfg(test)] +mod phase92_control_event_tests { + use super::{ + Phase92AuthorityControlEvent, Phase92AuthorityFence, Phase92AuthorityState, + Phase92PublicationContext, SinkTarget, + }; + + const EVENT_JSON: &str = + include_str!("../../../tests/fixtures/phase9/authority-control-primary.json"); + const ACTIVE_NOW_NS: i64 = 1_787_218_200_000_000_000; + + fn publication( + event: &Phase92AuthorityControlEvent, + target: SinkTarget, + watermark: u64, + ) -> Phase92PublicationContext { + let authority = event.authority.as_ref().expect("fixture authority"); + Phase92PublicationContext { + slice_id: authority.slice_id.clone(), + owner_id: authority.owner_id.clone(), + authority_revision: authority.authority_revision, + shard_id: "binance-usdm-trade-ethusdt-0".into(), + lease_epoch: authority.lease_epoch, + partition_plan_epoch: authority.partition_plan_epoch, + source_watermark: watermark, + target, + } + } + + #[test] + fn python_control_event_decodes_and_restart_stays_fenced_until_target_restore() { + let event: Phase92AuthorityControlEvent = + serde_json::from_str(EVENT_JSON).expect("control fixture decodes"); + event.validate().expect("control fixture validates"); + let mut fence = Phase92AuthorityFence::default(); + fence + .apply_control_event(&event, ACTIVE_NOW_NS) + .expect("primary snapshot loads in recovery mode"); + let first = publication(&event, SinkTarget::PrimaryCanonical, 501); + assert!(fence + .permits(&first, ACTIVE_NOW_NS) + .is_err_and(|error| error.contains("recovery is required"))); + for target in [ + SinkTarget::PrimaryCanonical, + SinkTarget::PublicV2, + SinkTarget::LegacyV1, + ] { + fence + .restore_committed_watermark(&publication(&event, target, 500)) + .expect("independent target watermark restores"); + } + fence + .permits(&first, ACTIVE_NOW_NS) + .expect("first post-handoff canonical watermark is W+1"); + fence.commit(&first).expect("W+1 commits"); + assert!(fence + .permits( + &publication(&event, SinkTarget::PrimaryCanonical, 501), + ACTIVE_NOW_NS, + ) + .is_err_and(|error| error.contains("duplicate, stale or gapped"))); + fence + .apply_control_event(&event, ACTIVE_NOW_NS) + .expect("identical compacted authority replay is idempotent"); + } + + #[test] + fn canary_to_primary_requires_and_applies_exact_handoff() { + let event: Phase92AuthorityControlEvent = + serde_json::from_str(EVENT_JSON).expect("control fixture decodes"); + let primary = event.authority.as_ref().expect("fixture authority"); + let handoff = event.handoff.as_ref().expect("fixture handoff"); + let mut canary = primary.clone(); + canary.state = Phase92AuthorityState::RustCanary; + canary.owner_id = handoff.old_owner_id.clone(); + canary.authority_revision = handoff.expected_authority_revision; + canary.lease_epoch = handoff.expected_lease_epoch; + canary.start_watermark = handoff.terminal_watermark; + canary.terminal_watermark = None; + canary.previous_owner_id = None; + canary.handoff_digest = None; + canary.public_write_allowed = false; + canary.legacy_write_allowed = false; + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary).expect("canary authority loads"); + fence + .apply_control_event(&event, ACTIVE_NOW_NS) + .expect("exact handoff promotes primary"); + assert_eq!( + fence.current().expect("primary authority").state, + Phase92AuthorityState::RustPrimary + ); + } + + #[test] + fn altered_outer_identity_fails_closed() { + let mut event: Phase92AuthorityControlEvent = + serde_json::from_str(EVENT_JSON).expect("control fixture decodes"); + event.authority_revision += 1; + assert!(event + .validate() + .is_err_and(|error| error.contains("differ"))); + } +} diff --git a/scripts/build_production_catalog.py b/scripts/build_production_catalog.py new file mode 100755 index 0000000..36f7a6a --- /dev/null +++ b/scripts/build_production_catalog.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path + +from qdl.runtime.production_catalog import ( + ProductionCatalogBuilder, + ProductionDemandManifest, + load_binance_exchange_info, + load_okx_instruments, +) +from qdl.runtime.stable_catalog import StableSourceCatalog + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--demand", action="append", required=True, type=Path) + parser.add_argument("--binance-usdm-exchange-info", type=Path) + parser.add_argument("--okx-instruments", type=Path) + parser.add_argument("--previous-catalog", type=Path) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--catalog-revision", type=int, required=True) + parser.add_argument("--source-policy-revision", type=int, required=True) + parser.add_argument("--authority-revision", type=int, required=True) + args = parser.parse_args() + demand = ProductionDemandManifest.load_many(args.demand) + needs_binance = any(item.venue == "BINANCE" for item in demand.demands) + needs_okx = any(item.venue == "OKX" for item in demand.demands) + if needs_binance != (args.binance_usdm_exchange_info is not None): + raise SystemExit("Binance metadata capture must be supplied exactly when demanded") + if needs_okx != (args.okx_instruments is not None): + raise SystemExit("OKX metadata capture must be supplied exactly when demanded") + binance = ( + load_binance_exchange_info(args.binance_usdm_exchange_info) + if args.binance_usdm_exchange_info else None + ) + okx = load_okx_instruments(args.okx_instruments) if args.okx_instruments else [] + previous = ( + StableSourceCatalog.load(args.previous_catalog) + if args.previous_catalog else None + ) + metadata = {} + if args.binance_usdm_exchange_info: + metadata["binance_exchange_info_sha256"] = sha256(args.binance_usdm_exchange_info) + if args.okx_instruments: + metadata["okx_instruments_sha256"] = sha256(args.okx_instruments) + bundle = ProductionCatalogBuilder( + catalog_revision=args.catalog_revision, + source_policy_revision=args.source_policy_revision, + authority_revision=args.authority_revision, + ).build( + demand=demand, + binance_usdm=binance, + okx_rows=okx, + previous_catalog=previous, + metadata_provenance=metadata, + ) + print(json.dumps(bundle.write(args.output_dir), indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_production_core_bundle.py b/scripts/build_production_core_bundle.py new file mode 100755 index 0000000..1c97e08 --- /dev/null +++ b/scripts/build_production_core_bundle.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import ( + StableAcquisitionPlan, + write_production_core_bundle, +) + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Build deterministic Phase C Rust production-core configs." + ) + parser.add_argument("--source-catalog", type=Path, required=True) + parser.add_argument("--acquisition-plan", type=Path, required=True) + parser.add_argument("--raw-authority", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--partition-plan-epoch", type=int, default=1) + args = parser.parse_args() + + catalog = StableSourceCatalog.load(args.source_catalog) + acquisition = StableAcquisitionPlan.load( + args.acquisition_plan, catalog=catalog + ) + authority = json.loads(args.raw_authority.read_text(encoding="utf-8")) + digests = write_production_core_bundle( + args.output_dir, + catalog=catalog, + acquisition=acquisition, + raw_authority=authority, + partition_plan_epoch=args.partition_plan_epoch, + ) + print(json.dumps({ + "schema": "qdl.v2.production-core-build-result.v1", + "status": "PASS", + "output_dir": str(args.output_dir.resolve()), + "digests": digests, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_qdl_sdk_release.py b/scripts/build_qdl_sdk_release.py new file mode 100755 index 0000000..2b3abd5 --- /dev/null +++ b/scripts/build_qdl_sdk_release.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import base64 +import csv +import hashlib +import io +import json +import os +from pathlib import Path +import zipfile + + +ROOT = Path(__file__).resolve().parents[1] +NAME = "qdl-sdk" +NORMALIZED_NAME = "qdl_sdk" +VERSION = "2.0.0" +DIST_INFO = f"{NORMALIZED_NAME}-{VERSION}.dist-info" +DEPENDENCIES = ( + "grpcio>=1.70.0,<2.0.0", + "httpx>=0.28.0,<1.0.0", + "protobuf>=6.31.1,<7.0.0", + "pydantic>=2.0.0,<3.0.0", + "PyJWT[crypto]>=2.13.0,<3.0.0", +) +FIXED_TIMESTAMP = (2020, 1, 1, 0, 0, 0) + + +def sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def record_digest(value: bytes) -> str: + digest = base64.urlsafe_b64encode(hashlib.sha256(value).digest()).rstrip(b"=") + return f"sha256={digest.decode('ascii')}" + + +def digest_paths(paths: list[Path]) -> str: + digest = hashlib.sha256() + for path in sorted(paths): + relative = path.relative_to(ROOT).as_posix().encode() + content = path.read_bytes() + digest.update(len(relative).to_bytes(4, "big")) + digest.update(relative) + digest.update(len(content).to_bytes(8, "big")) + digest.update(content) + return digest.hexdigest() + + +def release_files() -> dict[str, bytes]: + files: dict[str, bytes] = {} + for path in sorted((ROOT / "qdl_sdk").glob("*.py")): + files[path.relative_to(ROOT).as_posix()] = path.read_bytes() + readme = ROOT / "qdl_sdk/README.md" + files[readme.relative_to(ROOT).as_posix()] = readme.read_bytes() + for path in sorted((ROOT / "generated/python/qdl").rglob("*.py")): + archive = path.relative_to(ROOT / "generated/python").as_posix() + files[archive] = path.read_bytes() + forbidden = (b"qdl.api_v2", b"qdl.runtime", b"app.") + sdk_sources = b"\n".join( + value + for name, value in files.items() + if name.startswith("qdl_sdk/") and name.endswith(".py") + ) + found = [token.decode() for token in forbidden if token in sdk_sources] + if found: + raise RuntimeError(f"SDK imports service internals: {found}") + if not any(name == "qdl/query/v2/query_pb2.py" for name in files): + raise RuntimeError("generated query contract is missing from SDK artifact") + return files + + +def metadata() -> bytes: + requires = "".join(f"Requires-Dist: {dependency}\n" for dependency in DEPENDENCIES) + return ( + "Metadata-Version: 2.3\n" + f"Name: {NAME}\n" + f"Version: {VERSION}\n" + "Summary: Typed provider-neutral client for Quant Data Layer V2\n" + "Author-email: BobbyAxerol \n" + "License: MIT\n" + "Requires-Python: >=3.10\n" + f"{requires}" + "Description-Content-Type: text/markdown\n\n" + "# Quant Data Layer SDK\n" + ).encode() + + +def write_zip_entry(archive: zipfile.ZipFile, name: str, content: bytes) -> None: + info = zipfile.ZipInfo(name, FIXED_TIMESTAMP) + info.compress_type = zipfile.ZIP_DEFLATED + info.external_attr = 0o100644 << 16 + archive.writestr(info, content, compress_type=zipfile.ZIP_DEFLATED, compresslevel=9) + + +def build_wheel(output_dir: Path) -> dict[str, object]: + output_dir.mkdir(parents=True, exist_ok=True) + wheel_name = f"{NORMALIZED_NAME}-{VERSION}-py3-none-any.whl" + wheel_path = output_dir / wheel_name + files = release_files() + license_content = (ROOT / "LICENSE").read_bytes() + files[f"{DIST_INFO}/METADATA"] = metadata() + files[f"{DIST_INFO}/WHEEL"] = ( + "Wheel-Version: 1.0\n" + "Generator: qdl-sdk-release/1\n" + "Root-Is-Purelib: true\n" + "Tag: py3-none-any\n" + ).encode() + files[f"{DIST_INFO}/top_level.txt"] = b"qdl\nqdl_sdk\n" + files[f"{DIST_INFO}/licenses/LICENSE"] = license_content + + rows: list[list[str]] = [] + for name in sorted(files): + value = files[name] + rows.append([name, record_digest(value), str(len(value))]) + record_name = f"{DIST_INFO}/RECORD" + rows.append([record_name, "", ""]) + stream = io.StringIO(newline="") + csv.writer(stream, lineterminator="\n").writerows(rows) + files[record_name] = stream.getvalue().encode() + + temporary = wheel_path.with_suffix(".whl.tmp") + with zipfile.ZipFile(temporary, "w") as archive: + for name in sorted(files): + write_zip_entry(archive, name, files[name]) + os.replace(temporary, wheel_path) + + contract_paths = sorted((ROOT / "contracts/proto").rglob("*.proto")) + source_paths = sorted((ROOT / "qdl_sdk").glob("*.py")) + manifest = { + "schema": "qdl.sdk.release.v1", + "name": NAME, + "version": VERSION, + "wheel": wheel_path.name, + "wheel_sha256": sha256_bytes(wheel_path.read_bytes()), + "sdk_source_digest": digest_paths(source_paths), + "generated_contract_digest": digest_paths(contract_paths), + "python_requires": ">=3.10", + "dependencies": list(DEPENDENCIES), + "contains_service_internals": False, + "reproducible_timestamp": "2020-01-01T00:00:00Z", + } + manifest_path = output_dir / f"{NORMALIZED_NAME}-{VERSION}.release.json" + manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n") + sbom = { + "bomFormat": "CycloneDX", + "specVersion": "1.5", + "serialNumber": f"urn:uuid:{hashlib.sha256(wheel_path.read_bytes()).hexdigest()[:32]}", + "version": 1, + "metadata": { + "component": { + "type": "library", + "name": NAME, + "version": VERSION, + "hashes": [{"alg": "SHA-256", "content": manifest["wheel_sha256"]}], + } + }, + "components": [ + {"type": "library", "name": item.split(">=")[0], "version": item} + for item in DEPENDENCIES + ], + } + sbom_path = output_dir / f"{NORMALIZED_NAME}-{VERSION}.cdx.json" + sbom_path.write_text(json.dumps(sbom, indent=2, sort_keys=True) + "\n") + return {**manifest, "manifest": str(manifest_path), "sbom": str(sbom_path)} + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + print(json.dumps(build_wheel(args.output_dir.resolve()), indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phase80_generate_tls.sh b/scripts/phase80_generate_tls.sh index cdb45c0..b875ccb 100755 --- a/scripts/phase80_generate_tls.sh +++ b/scripts/phase80_generate_tls.sh @@ -22,13 +22,14 @@ issue_certificate() { local principal="$1" local dns_name="$2" local extension_file="${OUTPUT_DIR}/${principal}.ext" + local san_entries="${3:-DNS:${dns_name}}" openssl genrsa -out "${OUTPUT_DIR}/${principal}.key" 2048 >/dev/null 2>&1 openssl req -new -sha256 \ -key "${OUTPUT_DIR}/${principal}.key" \ -subj "/CN=${principal}" \ -out "${OUTPUT_DIR}/${principal}.csr" >/dev/null 2>&1 - printf 'subjectAltName=DNS:%s,DNS:localhost\nextendedKeyUsage=serverAuth,clientAuth\n' "${dns_name}" >"${extension_file}" + printf 'subjectAltName=%s,DNS:localhost\nextendedKeyUsage=serverAuth,clientAuth\n' "${san_entries}" >"${extension_file}" openssl x509 -req -sha256 -days 2 \ -in "${OUTPUT_DIR}/${principal}.csr" \ -CA "${OUTPUT_DIR}/ca.crt" \ @@ -48,9 +49,14 @@ issue_certificate() { for broker in kafka1 kafka2 kafka3; do issue_certificate "${broker}" "${broker}" done -for client in phase8-admin phase8-producer phase8-consumer phase8-core phase8-unauthorized; do +for client in phase8-admin phase8-producer phase8-consumer phase8-core phase8-unauthorized stable-authority-dispatcher stable-trading-system; do issue_certificate "${client}" "${client}" done +issue_certificate stable-trading-system-jwt stable-trading-system-jwt +openssl pkey -in "${OUTPUT_DIR}/stable-trading-system-jwt.key" -pubout \ + -out "${OUTPUT_DIR}/stable-trading-system-jwt.public.pem" >/dev/null 2>&1 +issue_certificate stable-query query_v2_1 "DNS:query_v2_1,DNS:query_v2_2,DNS:qdl-v2-query" +issue_certificate stable-stream stream_v2_active "DNS:stream_v2_active,DNS:stream_v2_passive,DNS:qdl-v2-stream" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/key.password" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/store.password" @@ -108,6 +114,11 @@ find "${OUTPUT_DIR}" -maxdepth 1 -name '*.key' \ ! -name 'phase8-producer.key' \ ! -name 'phase8-consumer.key' \ ! -name 'phase8-core.key' \ + ! -name 'stable-query.key' \ + ! -name 'stable-stream.key' \ + ! -name 'stable-authority-dispatcher.key' \ + ! -name 'stable-trading-system.key' \ + ! -name 'stable-trading-system-jwt.key' \ -delete rm -f "${OUTPUT_DIR}"/*.csr "${OUTPUT_DIR}"/*.ext "${OUTPUT_DIR}"/*.srl diff --git a/scripts/phase92_migration_smoke.sh b/scripts/phase92_migration_smoke.sh index 302cb7d..e1b960b 100755 --- a/scripts/phase92_migration_smoke.sh +++ b/scripts/phase92_migration_smoke.sh @@ -169,17 +169,39 @@ state="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT audit_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_transition_audit;")" checkpoint_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_terminal_owner_checkpoints;")" handoff_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_handoffs;")" +outbox_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_event_outbox;")" +outbox_revisions="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT string_agg(authority_revision::text, ',' ORDER BY authority_revision) FROM qdl_authority_event_outbox;")" [[ "${state}" == "PYTHON_PRIMARY:7:python-rollback:3:120" ]] [[ "${audit_count}" == "4" ]] [[ "${checkpoint_count}" == "2" ]] [[ "${handoff_count}" == "2" ]] +[[ "${outbox_count}" == "4" ]] +[[ "${outbox_revisions}" == "4,5,6,7" ]] +expect_failure "UPDATE qdl_authority_event_outbox SET payload = '{}'::jsonb;" + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres <<'SQL' >/dev/null +DO $$ +DECLARE + claimed qdl_authority_event_outbox%ROWTYPE; +BEGIN + SELECT * INTO claimed FROM qdl_claim_authority_outbox('phase92-smoke', 1); + IF claimed.event_id IS NULL THEN + RAISE EXCEPTION 'authority outbox claim returned no event'; + END IF; + PERFORM qdl_complete_authority_outbox( + claimed.event_id, 'phase92-smoke', 'qdl.authority.v1', 0, 12 + ); +END; +$$; +SQL +[[ "$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_event_outbox WHERE status='PUBLISHED' AND topic_offset=12;")" == "1" ]] -python3 - "${OUTPUT}" "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" <<'PY' +python3 - "${OUTPUT}" "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" "${outbox_count}" <<'PY' import json import pathlib import sys -output, state, audits, checkpoints, handoffs = sys.argv[1:] +output, state, audits, checkpoints, handoffs, outbox = sys.argv[1:] path = pathlib.Path(output) path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps({ @@ -189,6 +211,9 @@ path.write_text(json.dumps({ "audit_records": int(audits), "terminal_checkpoints": int(checkpoints), "accepted_handoffs": int(handoffs), + "authority_outbox_records": int(outbox), + "authority_outbox_payload_immutable": True, + "authority_outbox_claim_ack_passed": True, "direct_primary_bypass_rejected": True, "stale_cas_rejected": True, "handoff_mutation_rejected": True, @@ -203,4 +228,4 @@ PY cleanup trap - EXIT [[ -z "$(docker ps -aq --filter name=^/${CONTAINER}$)" ]] -printf '{"status":"PASS","state":"%s","audits":%s,"checkpoints":%s,"handoffs":%s,"cleanup":true}\n' "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" +printf '{"status":"PASS","state":"%s","audits":%s,"checkpoints":%s,"handoffs":%s,"outbox":%s,"cleanup":true}\n' "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" "${outbox_count}" diff --git a/scripts/phaseb_bootstrap_stable_broker.py b/scripts/phaseb_bootstrap_stable_broker.py index ce1830e..203d593 100755 --- a/scripts/phaseb_bootstrap_stable_broker.py +++ b/scripts/phaseb_bootstrap_stable_broker.py @@ -11,11 +11,17 @@ COMPOSE = ROOT / "docker-compose.v2-stable.yml" BOOTSTRAP = "kafka1:9092,kafka2:9092,kafka3:9092" ADMIN_CONFIG = "/etc/kafka/secrets/admin.properties" -TOPICS = ( - "md.raw.stable.v1", - "md.canonical.v2", - "md.quarantine.stable.v1", -) +TOPIC_POLICIES = { + "md.raw.stable.v1": "delete", + "md.canonical.v2": "delete", + "md.quarantine.stable.v1": "delete", + "qdl.authority.v1": "compact", + "qdl.target-checkpoint.v1": "compact", + "md.canary.canonical.v2": "delete", + "md.projector.public.v2": "delete", + "md.projector.legacy.v1": "delete", +} +TOPICS = tuple(TOPIC_POLICIES) def compose(env_file: Path, *arguments: str) -> subprocess.CompletedProcess[str]: @@ -60,7 +66,7 @@ def add_acl( def bootstrap(env_file: Path) -> dict[str, object]: - for topic in TOPICS: + for topic, cleanup_policy in TOPIC_POLICIES.items(): kafka( env_file, "kafka-topics.sh", @@ -71,6 +77,7 @@ def bootstrap(env_file: Path) -> dict[str, object]: "--config", "min.insync.replicas=2", "--config", "unclean.leader.election.enable=false", "--config", "compression.type=producer", + "--config", f"cleanup.policy={cleanup_policy}", ) add_acl( @@ -91,14 +98,47 @@ def bootstrap(env_file: Path) -> dict[str, object]: env_file, "phase8-core", ("READ",), ("--group", "qdl-v2-stable-core-v1"), ) - add_acl(env_file, "phase8-core", ("IdempotentWrite",), ("--cluster",)) + for topic in ("qdl.authority.v1", "qdl.target-checkpoint.v1"): + add_acl( + env_file, "phase8-core", ("READ", "DESCRIBE"), + ("--topic", topic), + ) + for topic in ( + "qdl.target-checkpoint.v1", + "md.canary.canonical.v2", + "md.projector.public.v2", + "md.projector.legacy.v1", + ): + add_acl( + env_file, "phase8-core", ("WRITE", "DESCRIBE"), + ("--topic", topic), + ) add_acl( - env_file, "phase8-core", ("WRITE", "DESCRIBE"), + env_file, "phase8-core", ("READ",), ( - "--transactional-id", "qdl-v2-stable-core-", + "--group", "qdl-v2-production-core-v1-", "--resource-pattern-type", "prefixed", ), ) + add_acl(env_file, "phase8-core", ("IdempotentWrite",), ("--cluster",)) + for transactional_prefix in ( + "qdl-v2-stable-core-", "qdl-v2-production-core-" + ): + add_acl( + env_file, "phase8-core", ("WRITE", "DESCRIBE"), + ( + "--transactional-id", transactional_prefix, + "--resource-pattern-type", "prefixed", + ), + ) + add_acl( + env_file, "stable-authority-dispatcher", ("WRITE", "DESCRIBE"), + ("--topic", "qdl.authority.v1"), + ) + add_acl( + env_file, "stable-authority-dispatcher", ("IdempotentWrite",), + ("--cluster",), + ) for topic in (TOPICS[0], TOPICS[1]): add_acl( env_file, "phase8-consumer", ("READ", "DESCRIBE"), @@ -127,7 +167,11 @@ def bootstrap(env_file: Path) -> dict[str, object]: "replication_factor": 3, "min_insync_replicas": 2, "tls_client_auth": "required", - "principals": ["phase8-producer", "phase8-core", "phase8-consumer"], + "topic_policies": dict(TOPIC_POLICIES), + "principals": [ + "phase8-producer", "phase8-core", "phase8-consumer", + "stable-authority-dispatcher", + ], } diff --git a/scripts/phaseb_prepare_stable_candidate.py b/scripts/phaseb_prepare_stable_candidate.py index 3112846..6b122e4 100755 --- a/scripts/phaseb_prepare_stable_candidate.py +++ b/scripts/phaseb_prepare_stable_candidate.py @@ -4,6 +4,7 @@ import argparse import hashlib import json +import re import secrets import shutil import subprocess @@ -15,8 +16,10 @@ from qdl.runtime.stable_catalog import StableSourceCatalog from qdl.runtime.stable_deployment import ( + AuthorityPromotionScope, StableAcquisitionPlan, stable_authority_record, + write_production_core_bundle, write_stable_runtime_bundle, ) @@ -50,12 +53,28 @@ def copy_client_identity(source: Path, destination: Path, principal: str) -> Non item.chmod(0o440) +def copy_server_identity(source: Path, destination: Path, principal: str) -> None: + destination.mkdir(parents=True, exist_ok=False) + for source_name, target_name in ( + ("ca.crt", "ca.crt"), + (f"{principal}.crt", "server.crt"), + (f"{principal}.key", "server.key"), + ): + origin = source / source_name + if not origin.is_file(): + raise FileNotFoundError(f"stable TLS source is unavailable: {origin}") + shutil.copyfile(origin, destination / target_name) + for item in destination.iterdir(): + item.chmod(0o440) + + def prepare_candidate( *, rust_image: str, python_image: str, cert_dir: Path, output_dir: Path, + consumer_network: str, rust_image_id: str | None = None, python_image_id: str | None = None, host_cert_dir: Path | None = None, @@ -63,6 +82,8 @@ def prepare_candidate( ) -> dict[str, object]: if output_dir.exists() and any(output_dir.iterdir()): raise FileExistsError("stable candidate output directory must be empty") + if re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}", consumer_network) is None: + raise ValueError("stable consumer network name is invalid") output_dir.mkdir(parents=True, exist_ok=True) runtime_dir = output_dir / "runtime" identities_dir = output_dir / "identities" @@ -77,6 +98,10 @@ def prepare_candidate( ) acquisition_path = ROOT / "config/v2/stable-acquisition-bindings.yaml" acquisition = StableAcquisitionPlan.load(acquisition_path, catalog=catalog) + promotion_scope = AuthorityPromotionScope.load( + ROOT / "config/v2/stable-authority-promotion-scope.yaml", + catalog=catalog, + ) authority = stable_authority_record( rust_image_digest=rust_digest, capability_manifest=ROOT / "config/v2/stable-capabilities.yaml", @@ -90,29 +115,57 @@ def prepare_candidate( acquisition=acquisition, authority=authority, ) + bundle_digests.update(write_production_core_bundle( + runtime_dir, + catalog=catalog, + acquisition=acquisition, + promotion_scope=promotion_scope, + raw_authority=authority, + partition_plan_epoch=1, + )) for role, principal in ( ("producer", "phase8-producer"), ("core", "phase8-core"), ("projector", "phase8-consumer"), + ("authority-dispatcher", "stable-authority-dispatcher"), + ("trading-system", "stable-trading-system"), ): copy_client_identity(cert_dir, identities_dir / role, principal) + copy_server_identity(cert_dir, identities_dir / "query", "stable-query") + copy_server_identity(cert_dir, identities_dir / "stream", "stable-stream") + jwt_identity_dir = identities_dir / "trading-system-jwt" + jwt_identity_dir.mkdir(parents=True, exist_ok=False) + for source_name, target_name in ( + ("stable-trading-system-jwt.key", "private.key"), + ("stable-trading-system-jwt.public.pem", "public.pem"), + ): + origin = cert_dir / source_name + if not origin.is_file(): + raise FileNotFoundError(f"stable JWT source is unavailable: {origin}") + shutil.copyfile(origin, jwt_identity_dir / target_name) + for item in jwt_identity_dir.iterdir(): + item.chmod(0o440) schema_digest = hashlib.sha256( (ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto").read_bytes() ).hexdigest() ingest_secret = secrets.token_urlsafe(48) cursor_secret = secrets.token_urlsafe(48) - jwt_secret = secrets.token_urlsafe(48) + control_db_password = secrets.token_urlsafe(32) + dispatcher_db_password = secrets.token_urlsafe(32) + jwt_public_key = (jwt_identity_dir / "public.pem").read_text(encoding="utf-8") compose_cert_dir = (host_cert_dir or cert_dir).resolve() compose_output_dir = (host_output_dir or output_dir).resolve() values = { "QDL_STABLE_SCHEMA_DIGEST": schema_digest, + "QDL_STABLE_CONSUMER_NETWORK": consumer_network, "QDL_STABLE_INTERNAL_INGEST_SECRET": ingest_secret, "QDL_STABLE_CURSOR_KEYS_JSON": json.dumps( {"stable-k1": cursor_secret}, separators=(",", ":") ), "QDL_STABLE_JWT_KEYS_JSON": json.dumps( - {"stable-jwt-k1": jwt_secret}, separators=(",", ":") + {"stable-trading-system-rs256-v1": jwt_public_key}, + separators=(",", ":"), ), "QDL_STABLE_PYTHON_IMAGE": python_digest, "QDL_STABLE_RUST_IMAGE": rust_digest, @@ -120,10 +173,31 @@ def prepare_candidate( "QDL_STABLE_PROJECTOR_CERT_DIR": str( compose_output_dir / "identities/projector" ), + "QDL_STABLE_AUTHORITY_CERT_DIR": str( + compose_output_dir / "identities/authority-dispatcher" + ), "QDL_STABLE_CORE_CERT_DIR": str(compose_output_dir / "identities/core"), "QDL_STABLE_PRODUCER_CERT_DIR": str( compose_output_dir / "identities/producer" ), + "QDL_STABLE_QUERY_CERT_DIR": str(compose_output_dir / "identities/query"), + "QDL_STABLE_STREAM_CERT_DIR": str(compose_output_dir / "identities/stream"), + "QDL_STABLE_TRADING_SYSTEM_CERT_DIR": str( + compose_output_dir / "identities/trading-system" + ), + "QDL_STABLE_TRADING_SYSTEM_JWT_PRIVATE_KEY": str( + compose_output_dir / "identities/trading-system-jwt/private.key" + ), + "QDL_STABLE_CONTROL_DB_PASSWORD": control_db_password, + "QDL_STABLE_DISPATCHER_DB_PASSWORD": dispatcher_db_password, + "QDL_STABLE_CONTROL_DB_DSN": ( + "postgresql://qdl_authority_dispatcher:" + f"{dispatcher_db_password}@stable_authority_db:5432/qdl_authority" + ), + "QDL_STABLE_CONTROL_ADMIN_DSN": ( + "postgresql://qdl_authority:" + f"{control_db_password}@stable_authority_db:5432/qdl_authority" + ), "QDL_STABLE_RUNTIME_DIR": str(compose_output_dir / "runtime"), } env_path = output_dir / "stable.env" @@ -145,7 +219,13 @@ def prepare_candidate( "runtime_digests": bundle_digests, "catalog_revision": catalog.catalog_revision, "acquisition_revision": acquisition.revision, + "authority_promotion_scope_revision": promotion_scope.revision, + "authority_promotion_scope_digest": promotion_scope.digest(), + "authority_promotion_binding_count": len(promotion_scope.binding_ids), + "consumer_network": consumer_network, "consumer_count": 5, + "workload_mtls": True, + "workload_identity_count": 4, "secret_values_recorded": False, } manifest_path = output_dir / "candidate-manifest.json" @@ -162,6 +242,7 @@ def main() -> int: parser.add_argument("--python-image", required=True) parser.add_argument("--cert-dir", type=Path, required=True) parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--consumer-network", required=True) parser.add_argument("--rust-image-id") parser.add_argument("--python-image-id") parser.add_argument("--host-cert-dir", type=Path) @@ -172,6 +253,7 @@ def main() -> int: python_image=args.python_image, cert_dir=args.cert_dir, output_dir=args.output_dir, + consumer_network=args.consumer_network, rust_image_id=args.rust_image_id, python_image_id=args.python_image_id, host_cert_dir=args.host_cert_dir, diff --git a/scripts/phasec1_isolated_consumer_acceptance.py b/scripts/phasec1_isolated_consumer_acceptance.py new file mode 100755 index 0000000..960dc91 --- /dev/null +++ b/scripts/phasec1_isolated_consumer_acceptance.py @@ -0,0 +1,357 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import tempfile +import time +import uuid +from dataclasses import dataclass +from pathlib import Path + +import jwt + +from qdl_sdk import ( + AsyncDataLayerClient, + ControlEvent, + DataRequirement, + Feed, + FileCursorStore, + GapPolicy, + Grade, + GrpcStreamTransport, + RecoveryPolicy, + RestQueryTransport, + StalePolicy, + StaticBearerCredential, + WorkloadTlsConfig, + StreamEvent, + market_data_view_from_stream, +) + + +@dataclass(frozen=True) +class VenueCase: + venue: str + instrument_uid: str + consumer_id: str + subject: str + provider: str + + +CASES = ( + VenueCase( + "BINANCE", + "a953e16e-7138-5562-b5e8-c337a44d0b65", + "trading-system.paper.stable", + "spiffe://qdl/paper/trading-system-stable", + "BINANCE_DIRECT", + ), + VenueCase( + "OKX", + "fb26214c-7b9b-5961-95b2-55154755af0f", + "trading-system.paper.stable", + "spiffe://qdl/paper/trading-system-stable", + "OKX_DIRECT", + ), +) + + +def token(subject: str, *, issuer: str, audience: str) -> str: + private_key_path = Path(os.environ["QDL_STABLE_JWT_PRIVATE_KEY_FILE"]) + key_id = os.environ["QDL_STABLE_JWT_KEY_ID"] + now = int(time.time()) + return jwt.encode( + { + "sub": subject, + "iss": issuer, + "aud": audience, + "iat": now, + "nbf": now - 1, + "exp": now + 300, + "jti": str(uuid.uuid4()), + "environment": "paper", + "roles": [ + "market_data_reader", + "historical_reader", + "stream_consumer", + ], + "consumer_manifest_revision": 1, + }, + private_key_path.read_bytes(), + algorithm="RS256", + headers={"kid": key_id}, + ) + + +def client( + *, + base_url: str, + grpc_target: str, + consumer_id: str, + bearer: str, + cursor_path: Path, + tls: WorkloadTlsConfig, +) -> AsyncDataLayerClient: + credential = StaticBearerCredential(bearer) + return AsyncDataLayerClient( + query_transport=RestQueryTransport( + base_url, + timeout_seconds=8, + credential_provider=credential, + tls=tls, + ), + stream_transport=GrpcStreamTransport( + grpc_target, + tls=tls, + credential_provider=credential, + ), + consumer_id=consumer_id, + cursor_store=FileCursorStore(cursor_path), + max_buffer_events=64, + max_reconnect_attempts=2, + ) + + +def bar_fingerprint(response) -> list[dict[str, object]]: + values = [] + for item in response.data: + if item.feed is not Feed.BAR: + raise AssertionError("warmup returned a non-BAR item") + if item.payload.lifecycle not in {"FINAL", "REVISED"}: + raise AssertionError("warmup returned a non-final BAR") + if ( + item.source.authoritative is not True + or item.quality.complete is not True + or item.quality.gap_open is not False + ): + raise AssertionError("warmup BAR failed authority/coverage gate") + values.append( + { + "instrument_uid": item.instrument_uid, + "instrument_id": item.instrument_id, + "observed_at_ns": item.observed_at_ns, + "payload": item.payload.model_dump(mode="json"), + "source": item.source.model_dump(mode="json"), + "contract": item.contract.model_dump(mode="json"), + } + ) + return values + + +async def next_data(session) -> tuple[StreamEvent, list[str]]: + controls: list[str] = [] + for _ in range(8): + item = await asyncio.wait_for(session.__anext__(), timeout=12) + if isinstance(item, ControlEvent): + controls.append(item.code) + continue + if not isinstance(item, StreamEvent): + raise AssertionError("stream returned an unknown SDK event") + return item, controls + raise AssertionError("stream did not reach a market-data event") + + +async def certify_case( + case: VenueCase, + *, + primary_url: str, + secondary_url: str, + grpc_target: str, + issuer: str, + audience: str, + state_dir: Path, + tls: WorkloadTlsConfig, +) -> dict[str, object]: + bearer = token(case.subject, issuer=issuer, audience=audience) + bar_requirement = DataRequirement( + case.instrument_uid, + Feed.BAR, + Grade.EXECUTION, + "crypto_primary_v2", + interval="1m", + warmup_limit=5, + max_freshness_ms=180_000, + stale_policy=StalePolicy.BLOCK, + gap_policy=GapPolicy.BLOCK, + recovery=RecoveryPolicy.SNAPSHOT_AND_REPLAY, + ) + primary = client( + base_url=primary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=state_dir / f"{case.venue.lower()}-query-primary.json", + tls=tls, + ) + secondary = client( + base_url=secondary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=state_dir / f"{case.venue.lower()}-query-secondary.json", + tls=tls, + ) + try: + first = await primary.warmup(bar_requirement) + second = await secondary.warmup(bar_requirement) + finally: + await primary.close() + await secondary.close() + first_rows = bar_fingerprint(first) + second_rows = bar_fingerprint(second) + if first_rows != second_rows: + raise AssertionError(f"{case.venue} query replicas diverged") + + trade_requirement = DataRequirement( + case.instrument_uid, + Feed.TRADE, + Grade.EXECUTION, + "crypto_primary_v2", + warmup_limit=0, + max_freshness_ms=15_000, + stale_policy=StalePolicy.BLOCK, + gap_policy=GapPolicy.BLOCK, + recovery=RecoveryPolicy.SNAPSHOT_AND_REPLAY, + ) + cursor_path = state_dir / f"{case.venue.lower()}-stream.json" + first_client = client( + base_url=primary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=cursor_path, + tls=tls, + ) + try: + async with first_client.warmup_then_stream(trade_requirement) as session: + first_event, first_controls = await next_data(session) + first_view = market_data_view_from_stream( + first_event, + template=session.warmup.data[-1], + requirement=trade_requirement, + ) + if ( + first_view.source.provider != case.provider + or not first_view.source.authoritative + or not first_view.quality.complete + or first_view.quality.gap_open + ): + raise AssertionError(f"{case.venue} first stream event failed quality") + session.acknowledge(first_event) + finally: + await first_client.close() + + resumed_client = client( + base_url=secondary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=cursor_path, + tls=tls, + ) + try: + async with resumed_client.warmup_then_stream( + trade_requirement, + resume_restored_state=True, + ) as session: + resumed_event, resumed_controls = await next_data(session) + resumed_view = market_data_view_from_stream( + resumed_event, + template=session.warmup.data[-1], + requirement=trade_requirement, + ) + if resumed_event.logical_offset != first_event.logical_offset + 1: + raise AssertionError( + f"{case.venue} cursor resume was not contiguous: " + f"{first_event.logical_offset}->{resumed_event.logical_offset}" + ) + if resumed_view.source.provider != case.provider: + raise AssertionError(f"{case.venue} resumed source changed") + session.acknowledge(resumed_event) + finally: + await resumed_client.close() + + persisted = json.loads(cursor_path.read_text(encoding="utf-8")) + offsets = [ + int(item["offset"]) + for item in persisted["items"].values() + ] + if offsets != [resumed_event.logical_offset]: + raise AssertionError(f"{case.venue} durable cursor was not ACKed") + + return { + "venue": case.venue, + "provider": case.provider, + "warmup_rows": len(first_rows), + "coverage": first.coverage, + "final_bar_close_time_ns": first_rows[-1]["payload"]["close_time_ns"], + "first_stream_offset": first_event.logical_offset, + "resumed_stream_offset": resumed_event.logical_offset, + "first_controls": first_controls, + "resumed_controls": resumed_controls, + "cursor_persisted": True, + "secret_values_recorded": False, + } + + +async def run(args: argparse.Namespace) -> dict[str, object]: + tls = WorkloadTlsConfig( + args.tls_ca_file, + args.tls_certificate_file, + args.tls_private_key_file, + ) + with tempfile.TemporaryDirectory(prefix="qdl-c1-sdk-") as temporary: + state_dir = Path(temporary) + results = [] + for case in CASES: + results.append( + await certify_case( + case, + primary_url=args.primary_url, + secondary_url=args.secondary_url, + grpc_target=args.grpc_target, + issuer=args.issuer, + audience=args.audience, + state_dir=state_dir, + tls=tls, + ) + ) + return { + "schema": "qdl.phase-c1.isolated-consumer-acceptance.v1", + "status": "PASS", + "contract_version": "2.0.0", + "authority_expected": "RUST_SHADOW", + "cases": results, + "secret_values_recorded": False, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--primary-url", default="https://localhost:18201") + parser.add_argument("--secondary-url", default="https://localhost:18202") + parser.add_argument("--grpc-target", required=True) + parser.add_argument( + "--issuer", default="https://identity.qdl.stable.internal" + ) + parser.add_argument("--audience", default="qdl-v2-stable") + parser.add_argument("--tls-ca-file", required=True) + parser.add_argument("--tls-certificate-file", required=True) + parser.add_argument("--tls-private-key-file", required=True) + args = parser.parse_args() + required_env = { + "QDL_STABLE_JWT_PRIVATE_KEY_FILE", + "QDL_STABLE_JWT_KEY_ID", + } + missing = sorted(required_env - os.environ.keys()) + if missing: + raise SystemExit(f"required JWT signer environment is missing: {missing}") + print(json.dumps(asyncio.run(run(args)), sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasec3_authority_cutover.py b/scripts/phasec3_authority_cutover.py new file mode 100755 index 0000000..d7376b9 --- /dev/null +++ b/scripts/phasec3_authority_cutover.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import asyncio +import json +import os +from pathlib import Path +from typing import Any + +from qdl.control.cutover_packet import AuthorityCutoverPacket, CutoverSlice + + +_CURRENT_SQL = """ +SELECT state, authority_revision, owner_id, lease_epoch, partition_plan_epoch, + candidate_digest, artifact_image_digest, contract_digest, + partition_plan_digest +FROM qdl_authority_slices +WHERE slice_id = $1 +FOR UPDATE +""" +_STANDARD_SQL = """ +SELECT * +FROM qdl_transition_authority( + $1::uuid, $2, $3, $4, $5, $6, $7, $8, $9, $10, + $11::uuid, $12::timestamptz, $13, $14 +) +""" +_HANDOFF_SQL = """ +SELECT * +FROM qdl_transition_authority_v2( + $1::uuid, $2::uuid, $3, $4, $5, $6, $7, $8, $9, $10, + $11, $12, $13::uuid, $14::timestamptz, $15, $16 +) +""" + + +def required(name: str) -> str: + value = os.environ.get(name, "").strip() + if not value: + raise RuntimeError(f"required environment variable is missing: {name}") + return value + + +def _assert_current( + row: Any, item: CutoverSlice, packet: AuthorityCutoverPacket +) -> None: + if row is None: + raise RuntimeError(f"authority slice is absent: {item.slice_id}") + expected = { + "state": item.expected_state, + "authority_revision": item.expected_revision, + "owner_id": item.expected_owner_id, + "lease_epoch": item.expected_lease_epoch, + "partition_plan_epoch": item.partition_plan_epoch, + "candidate_digest": packet.candidate_digest, + "artifact_image_digest": packet.artifact_image_digest, + "contract_digest": packet.contract_digest, + "partition_plan_digest": packet.partition_plan_digest, + } + actual = {name: row[name] for name in expected} + if actual != expected: + raise RuntimeError( + f"authority CAS precondition differs for {item.slice_id}: " + f"expected={expected!r} actual={actual!r}" + ) + + +async def _apply_one( + connection: Any, packet: AuthorityCutoverPacket, item: CutoverSlice +) -> dict[str, Any]: + async with connection.transaction(): + current = await connection.fetchrow(_CURRENT_SQL, item.slice_id) + _assert_current(current, item, packet) + if packet.stage in {"PRIMARY", "PYTHON_RESTORE"}: + updated = await connection.fetchrow( + _HANDOFF_SQL, + item.handoff_id, + item.transition_id, + item.slice_id, + item.expected_state, + item.expected_revision, + item.expected_owner_id, + item.expected_lease_epoch, + item.partition_plan_epoch, + item.new_state, + item.new_owner_id, + item.new_lease_epoch, + item.terminal_watermark, + item.prerequisite_bundle_id, + item.hold_until, + packet.actor, + item.reason, + ) + else: + updated = await connection.fetchrow( + _STANDARD_SQL, + item.transition_id, + item.slice_id, + item.expected_state, + item.expected_revision, + item.expected_owner_id, + item.expected_lease_epoch, + item.partition_plan_epoch, + item.new_state, + item.new_owner_id, + item.new_lease_epoch, + item.terminal_watermark, + item.prerequisite_bundle_id, + item.hold_until, + packet.actor, + item.reason, + ) + return { + "slice_id": item.slice_id, + "state": updated["state"], + "authority_revision": updated["authority_revision"], + "owner_id": updated["owner_id"], + "lease_epoch": updated["lease_epoch"], + } + + +async def apply_packet(packet: AuthorityCutoverPacket, dsn: str) -> list[dict[str, Any]]: + try: + import asyncpg + except ImportError as error: + raise RuntimeError("authority cutover apply requires asyncpg") from error + connection = await asyncpg.connect( + dsn=dsn, + command_timeout=15, + server_settings={"application_name": "qdl-c3-authority-cutover"}, + ) + results = [] + try: + for item in packet.slices: + results.append(await _apply_one(connection, packet, item)) + finally: + await connection.close() + return results + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--packet", type=Path, required=True) + parser.add_argument("--apply", action="store_true") + parser.add_argument("--confirm") + args = parser.parse_args() + raw = json.loads(args.packet.read_text(encoding="utf-8")) + packet = AuthorityCutoverPacket.parse(raw) + plan = packet.plan() + if not args.apply: + print(json.dumps(plan, indent=2, sort_keys=True)) + return 0 + if args.confirm != packet.confirmation_token: + raise RuntimeError( + "authority cutover confirmation token differs from immutable packet" + ) + results = asyncio.run( + apply_packet(packet, required("QDL_CONTROL_ADMIN_DSN")) + ) + print(json.dumps({ + **plan, + "apply_requested": True, + "production_mutations": len(results), + "results": results, + }, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasec3_authority_db_smoke.sh b/scripts/phasec3_authority_db_smoke.sh new file mode 100755 index 0000000..473d5cd --- /dev/null +++ b/scripts/phasec3_authority_db_smoke.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +NAME="qdl_c3_authority_db_smoke_$$" +IMAGE="postgres@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685" +OWNER_PASSWORD="c3-owner-test-only" +DISPATCHER_PASSWORD="c3-dispatcher-test-only" + +cleanup() { + docker stop "${NAME}" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +docker run --rm -d --name "${NAME}" --network none --tmpfs /var/lib/postgresql/data:rw,nosuid,nodev,size=256m --tmpfs /run/postgresql:rw,nosuid,nodev,size=8m -e POSTGRES_DB=qdl_authority -e POSTGRES_USER=qdl_authority -e POSTGRES_PASSWORD="${OWNER_PASSWORD}" -e QDL_STABLE_DISPATCHER_DB_PASSWORD="${DISPATCHER_PASSWORD}" -e PGDATA=/var/lib/postgresql/data/pgdata -v "${ROOT_DIR}/migrations/postgres:/docker-entrypoint-initdb.d:ro" "${IMAGE}" >/dev/null + +for _attempt in $(seq 1 60); do + if docker exec "${NAME}" pg_isready -U qdl_authority -d qdl_authority >/dev/null 2>&1; then + break + fi + sleep 1 +done +docker exec "${NAME}" pg_isready -U qdl_authority -d qdl_authority >/dev/null + +owner_query() { + docker exec -e PGPASSWORD="${OWNER_PASSWORD}" "${NAME}" psql --set=ON_ERROR_STOP=1 -At -U qdl_authority -d qdl_authority -c "$1" +} + +dispatcher_query() { + docker exec -e PGPASSWORD="${DISPATCHER_PASSWORD}" "${NAME}" psql --set=ON_ERROR_STOP=1 -At -U qdl_authority_dispatcher -d qdl_authority -c "$1" +} + +[[ "$(owner_query "SELECT count(*) FROM pg_proc WHERE proname IN ('qdl_claim_authority_outbox','qdl_complete_authority_outbox','qdl_retry_authority_outbox') AND prosecdef")" == "3" ]] +[[ "$(owner_query "SELECT count(*) FROM pg_proc WHERE proname IN ('qdl_claim_authority_outbox','qdl_complete_authority_outbox','qdl_retry_authority_outbox') AND 'search_path=pg_catalog, public'=ANY(proconfig)")" == "3" ]] +[[ "$(owner_query "SELECT rolsuper::int||':'||rolcreatedb::int||':'||rolcreaterole::int FROM pg_roles WHERE rolname='qdl_authority_dispatcher'")" == "0:0:0" ]] +[[ "$(owner_query "SELECT has_function_privilege('qdl_authority_dispatcher','qdl_claim_authority_outbox(text,integer,interval)','EXECUTE')::int")" == "1" ]] +[[ "$(owner_query "SELECT has_table_privilege('qdl_authority_dispatcher','qdl_authority_event_outbox','UPDATE')::int")" == "0" ]] +[[ "$(dispatcher_query "SELECT count(*) FROM qdl_claim_authority_outbox('c3-smoke',1)")" == "0" ]] + +if dispatcher_query "UPDATE qdl_authority_event_outbox SET status='BLOCKED' WHERE false" >/dev/null 2>&1; then + echo "dispatcher unexpectedly obtained direct table UPDATE" >&2 + exit 1 +fi + +owner_query "\\i /docker-entrypoint-initdb.d/0010_authority_dispatcher_security.sql" >/dev/null + +printf '%s\n' '{"schema":"qdl.c3.authority-db-smoke.v1","status":"PASS","production_mutations":0,"cleanup":"container-auto-remove"}' diff --git a/scripts/rebuild_v2_stable_projection_cache.py b/scripts/rebuild_v2_stable_projection_cache.py index a544fa4..988d327 100755 --- a/scripts/rebuild_v2_stable_projection_cache.py +++ b/scripts/rebuild_v2_stable_projection_cache.py @@ -3,6 +3,7 @@ import argparse import json +import ssl import subprocess import time import urllib.request @@ -166,11 +167,42 @@ def _validate_project(env_file: Path) -> None: raise RuntimeError("compose project is not the isolated stable candidate") -def _wait_http(url: str, deadline: float) -> None: +def _env_value(env_file: Path, key: str) -> str: + prefix = f"{key}=" + matches = [ + line[len(prefix):] + for line in env_file.read_text().splitlines() + if line.startswith(prefix) + ] + if len(matches) != 1 or not matches[0]: + raise ValueError(f"stable env must define exactly one {key}") + return matches[0] + + +def _stable_client_ssl_context(env_file: Path) -> ssl.SSLContext: + identity_root = Path( + _env_value(env_file, "QDL_STABLE_TRADING_SYSTEM_CERT_DIR") + ) + context = ssl.create_default_context(cafile=str(identity_root / "ca.crt")) + context.load_cert_chain( + certfile=str(identity_root / "client.crt"), + keyfile=str(identity_root / "client.key"), + ) + return context + + +def _wait_http( + url: str, + deadline: float, + *, + ssl_context: ssl.SSLContext | None = None, +) -> None: last_error: BaseException | None = None while time.monotonic() < deadline: try: - with urllib.request.urlopen(url, timeout=2) as response: + with urllib.request.urlopen( + url, timeout=2, context=ssl_context + ) as response: if response.status == 200: return except BaseException as error: @@ -298,17 +330,34 @@ def execute_rebuild(env_file: Path, *, timeout_seconds: float) -> dict[str, obje "--to-earliest", "--execute", ) + ssl_context = _stable_client_ssl_context(env_file) _start_services(env_file, *STREAM_SERVICES) - _wait_http("http://127.0.0.1:18210/health/live", deadline) - _wait_http("http://127.0.0.1:18211/health/live", deadline) + _wait_http( + "https://localhost:18210/health/live", + deadline, + ssl_context=ssl_context, + ) + _wait_http( + "https://localhost:18211/health/live", + deadline, + ssl_context=ssl_context, + ) _start_services(env_file, "projector_v2") lag = _wait_bounded_lag(env_file, deadline) _wait_projector_ready(env_file, deadline) _start_services(env_file, *QUERY_SERVICES) - _wait_http("http://127.0.0.1:18201/health/ready", deadline) - _wait_http("http://127.0.0.1:18202/health/ready", deadline) + _wait_http( + "https://localhost:18201/health/ready", + deadline, + ssl_context=ssl_context, + ) + _wait_http( + "https://localhost:18202/health/ready", + deadline, + ssl_context=ssl_context, + ) final_size = int( _compose( diff --git a/scripts/run_authority_outbox_dispatcher.py b/scripts/run_authority_outbox_dispatcher.py new file mode 100755 index 0000000..3e5da2c --- /dev/null +++ b/scripts/run_authority_outbox_dispatcher.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import socket +import time +from pathlib import Path + +from qdl.control.authority_outbox import ( + AsyncpgAuthorityOutboxRepository, + AuthorityOutboxDispatcher, + KafkaAuthorityPublisher, +) + + +def required(name: str) -> str: + value = os.environ.get(name, "").strip() + if not value: + raise RuntimeError(f"required environment variable is missing: {name}") + return value + + +def write_health( + path: Path, *, status: str, published: int, error: str | None = None +) -> None: + if status not in {"STARTING", "READY", "DEGRADED"} or published < 0: + raise ValueError("authority dispatcher health payload is invalid") + payload = { + "schema": "qdl.authority-dispatcher-health.v1", + "status": status, + "heartbeat_ns": time.time_ns(), + "published_last_cycle": published, + "error": error[:1000] if error else None, + } + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(path.suffix + ".tmp") + temporary.write_text( + json.dumps(payload, sort_keys=True, separators=(",", ":")) + "\n", + encoding="utf-8", + ) + temporary.replace(path) + + +async def run() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--once", action="store_true") + parser.add_argument("--poll-seconds", type=float, default=0.5) + parser.add_argument("--batch-size", type=int, default=20) + args = parser.parse_args() + if not 0.05 <= args.poll_seconds <= 60: + raise ValueError("poll interval is outside bounds") + health_path = Path(required("QDL_AUTHORITY_HEALTH_FILE")) + write_health(health_path, status="STARTING", published=0) + repository = await AsyncpgAuthorityOutboxRepository.connect(required("QDL_CONTROL_DB_DSN")) + publisher = KafkaAuthorityPublisher( + { + "bootstrap.servers": required("QDL_KAFKA_BOOTSTRAP_SERVERS"), + "client.id": required("QDL_KAFKA_CLIENT_ID"), + "security.protocol": "ssl", + "ssl.ca.location": required("QDL_KAFKA_CA_LOCATION"), + "ssl.certificate.location": required("QDL_KAFKA_CERT_LOCATION"), + "ssl.key.location": required("QDL_KAFKA_KEY_LOCATION"), + }, + topic=required("QDL_AUTHORITY_TOPIC"), + ) + dispatcher = AuthorityOutboxDispatcher( + repository=repository, + publisher=publisher, + lock_owner=f"{socket.gethostname()}:{os.getpid()}", + batch_size=args.batch_size, + ) + try: + while True: + count = await dispatcher.dispatch_once() + write_health(health_path, status="READY", published=count) + print(json.dumps({"event": "qdl_authority_outbox_dispatch", "published": count})) + if args.once: + return + await asyncio.sleep(args.poll_seconds if count == 0 else 0) + except Exception as error: + write_health( + health_path, status="DEGRADED", published=0, error=str(error) + ) + raise + finally: + await repository.close() + + +if __name__ == "__main__": + asyncio.run(run()) diff --git a/tests/fixtures/phase9/authority-control-primary.json b/tests/fixtures/phase9/authority-control-primary.json new file mode 100644 index 0000000..1403c99 --- /dev/null +++ b/tests/fixtures/phase9/authority-control-primary.json @@ -0,0 +1,72 @@ +{ + "authority": { + "approved_at_ns": 1787216400000000000, + "approved_by": "operator@example", + "authority_revision": 8, + "candidate_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "handoff_digest": "4fcd899c26ae7f129b50b0d6001bed77b613d4b24505fcec8ade9e063936e11e", + "hold_until_ns": 1787220000000000000, + "lease_epoch": 11, + "legacy_write_allowed": true, + "owner_id": "rust-primary-owner", + "partition_plan_epoch": 3, + "prerequisite_bundle_id": "40000000-0000-4000-8000-000000000004", + "previous_owner_id": "python-v1-owner", + "public_write_allowed": true, + "schema": "qdl.authority-record.v3", + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "start_watermark": 500, + "state": "RUST_PRIMARY", + "terminal_watermark": 500 + }, + "authority_revision": 8, + "checkpoint": { + "authority_revision": 7, + "candidate_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "checkpoint_id": "20000000-0000-4000-8000-000000000002", + "committed_at_ns": 1787216400000000000, + "connection_generation": 2, + "lease_epoch": 10, + "owner_id": "python-v1-owner", + "partition_plan_epoch": 3, + "schema": "qdl.terminal-owner-checkpoint.v1", + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "source_session_id": "python-session-7", + "terminal_event_id": "event-500", + "terminal_payload_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "terminal_watermark": 500 + }, + "database_state": "RUST_PRIMARY", + "event_id": "10000000-0000-4000-8000-000000000001", + "handoff": { + "approved_at_ns": 1787216400000000000, + "approved_by": "operator@example", + "candidate_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "checkpoint_digest": "ea1788a5d5f21969a70ccdad9c3ac56519d67a82149d8fec627d8cc1e3aa2c7b", + "direction": "PYTHON_TO_RUST", + "expected_authority_revision": 7, + "expected_lease_epoch": 10, + "expected_state": "RUST_CANARY", + "expires_at_ns": 1787220000000000000, + "first_new_watermark": 501, + "handoff_id": "30000000-0000-4000-8000-000000000003", + "new_authority_revision": 8, + "new_event_count": 11, + "new_lease_epoch": 11, + "new_owner_id": "rust-primary-owner", + "new_state": "RUST_PRIMARY", + "old_event_count": 11, + "old_owner_id": "python-v1-owner", + "open_gaps": 0, + "overlap_end_watermark": 500, + "overlap_start_watermark": 490, + "partition_plan_epoch": 3, + "prerequisite_bundle_id": "40000000-0000-4000-8000-000000000004", + "schema": "qdl.accepted-authority-handoff.v1", + "semantic_mismatches": 0, + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "terminal_watermark": 500 + }, + "schema": "qdl.authority-control-event.v1", + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT" +} diff --git a/tests/test_authority_outbox.py b/tests/test_authority_outbox.py new file mode 100644 index 0000000..459feb3 --- /dev/null +++ b/tests/test_authority_outbox.py @@ -0,0 +1,229 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from qdl.control.authority_outbox import ( + AuthorityOutboxDispatcher, + BrokerAck, + ClaimedAuthorityEvent, + _checkpoint, + _digest, + build_authority_control_event, +) + + +EVENT_ID = "10000000-0000-4000-8000-000000000001" +CHECKPOINT_ID = "20000000-0000-4000-8000-000000000002" +HANDOFF_ID = "30000000-0000-4000-8000-000000000003" +BUNDLE_ID = "40000000-0000-4000-8000-000000000004" +NOW = datetime(2026, 8, 20, 9, 0, tzinfo=timezone.utc) +LATER = datetime(2026, 8, 20, 10, 0, tzinfo=timezone.utc) +DIGEST = "a" * 64 + + +def checkpoint_raw(): + return { + "checkpoint_id": CHECKPOINT_ID, + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "owner_id": "python-v1-owner", + "authority_revision": 7, + "lease_epoch": 10, + "partition_plan_epoch": 3, + "source_session_id": "python-session-7", + "connection_generation": 2, + "terminal_watermark": 500, + "terminal_event_id": "event-500", + "terminal_payload_sha256": "b" * 64, + "candidate_digest": DIGEST, + "committed_at": NOW.isoformat(), + } + + +def handoff_raw(): + checkpoint = _checkpoint(checkpoint_raw()) + assert checkpoint is not None + expected = { + "schema": "qdl.accepted-authority-handoff.v1", + "handoff_id": HANDOFF_ID, + "direction": "PYTHON_TO_RUST", + "checkpoint_digest": _digest(checkpoint), + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "old_owner_id": "python-v1-owner", + "new_owner_id": "rust-primary-owner", + "expected_state": "RUST_CANARY", + "new_state": "RUST_PRIMARY", + "expected_authority_revision": 7, + "new_authority_revision": 8, + "expected_lease_epoch": 10, + "new_lease_epoch": 11, + "partition_plan_epoch": 3, + "terminal_watermark": 500, + "first_new_watermark": 501, + "overlap_start_watermark": 490, + "overlap_end_watermark": 500, + "old_event_count": 11, + "new_event_count": 11, + "semantic_mismatches": 0, + "open_gaps": 0, + "candidate_digest": DIGEST, + "prerequisite_bundle_id": BUNDLE_ID, + "approved_by": "operator@example", + "approved_at_ns": int(NOW.timestamp() * 1_000_000_000), + "expires_at_ns": int(LATER.timestamp() * 1_000_000_000), + } + return { + "handoff_id": HANDOFF_ID, + "checkpoint_id": CHECKPOINT_ID, + "direction": "PYTHON_TO_RUST", + "slice_id": expected["slice_id"], + "old_owner_id": expected["old_owner_id"], + "new_owner_id": expected["new_owner_id"], + "expected_state": expected["expected_state"], + "new_state": expected["new_state"], + "expected_authority_revision": 7, + "new_authority_revision": 8, + "expected_lease_epoch": 10, + "new_lease_epoch": 11, + "partition_plan_epoch": 3, + "terminal_watermark": 500, + "first_new_watermark": 501, + "overlap_start_watermark": 490, + "overlap_end_watermark": 500, + "old_event_count": 11, + "new_event_count": 11, + "semantic_mismatches": 0, + "open_gaps": 0, + "candidate_digest": DIGEST, + "prerequisite_bundle_id": BUNDLE_ID, + "handoff_sha256": _digest(expected), + "approved_by": "operator@example", + "approved_at": NOW.isoformat(), + "expires_at": LATER.isoformat(), + } + + +def outbox_payload(state="RUST_PRIMARY", *, handoff=True): + authority = { + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "state": state, + "owner_id": "rust-primary-owner", + "authority_revision": 8, + "lease_epoch": 11, + "partition_plan_epoch": 3, + "candidate_digest": DIGEST, + "prerequisite_bundle_id": BUNDLE_ID, + "terminal_watermark": 500, + "approved_by": "operator@example", + "approved_at": NOW.isoformat(), + "hold_until": LATER.isoformat(), + } + return { + "schema": "qdl.authority-outbox-event.v1", + "event_id": EVENT_ID, + "transition": { + "transition_id": EVENT_ID, + "slice_id": authority["slice_id"], + "previous_state": "RUST_CANARY", + "new_state": state, + "previous_revision": 7, + "new_revision": 8, + }, + "authority": authority, + "checkpoint": checkpoint_raw() if handoff else None, + "handoff": handoff_raw() if handoff else None, + } + + +class AuthorityControlEventTests(unittest.TestCase): + def test_primary_event_binds_exact_handoff_and_w_boundary(self): + event = build_authority_control_event(outbox_payload()) + authority = event["authority"] + self.assertEqual(authority["state"], "RUST_PRIMARY") + self.assertEqual(authority["start_watermark"], 500) + self.assertEqual(authority["terminal_watermark"], 500) + self.assertEqual(authority["previous_owner_id"], "python-v1-owner") + self.assertEqual(authority["handoff_digest"], _digest(event["handoff"])) + self.assertEqual(event["handoff"]["first_new_watermark"], 501) + + def test_shadow_transition_is_durable_but_not_phase92_writable(self): + payload = outbox_payload("RUST_SHADOW", handoff=False) + payload["authority"].update({ + "owner_id": "rust-shadow-owner", + "authority_revision": 2, + "lease_epoch": 2, + "terminal_watermark": None, + "prerequisite_bundle_id": None, + "approved_by": None, + "approved_at": None, + "hold_until": None, + }) + payload["transition"].update({"previous_revision": 1, "new_revision": 2}) + event = build_authority_control_event(payload) + self.assertIsNone(event["authority"]) + self.assertEqual(event["database_state"], "RUST_SHADOW") + + def test_tampered_handoff_and_primary_without_handoff_fail_closed(self): + tampered = outbox_payload() + tampered["handoff"]["handoff_sha256"] = "f" * 64 + with self.assertRaisesRegex(ValueError, "digest differs"): + build_authority_control_event(tampered) + with self.assertRaisesRegex(ValueError, "requires accepted handoff"): + build_authority_control_event(outbox_payload(handoff=False)) + + +class _Repository: + def __init__(self, payload): + self.claimed = [ClaimedAuthorityEvent(EVENT_ID, payload)] + self.completed = [] + self.retried = [] + + async def claim(self, lock_owner, limit): + del lock_owner, limit + values, self.claimed = self.claimed, [] + return values + + async def complete(self, event_id, lock_owner, ack): + self.completed.append((event_id, lock_owner, ack)) + + async def retry(self, event_id, lock_owner, error, delay_seconds): + self.retried.append((event_id, lock_owner, error, delay_seconds)) + + +class _Publisher: + def __init__(self, fail=False): + self.fail = fail + self.calls = [] + + async def publish(self, *, key, event_id, payload): + self.calls.append((key, event_id, payload)) + if self.fail: + raise RuntimeError("broker unavailable") + return BrokerAck("qdl.authority.v1", 0, 12) + + +class AuthorityOutboxDispatcherTests(unittest.IsolatedAsyncioTestCase): + async def test_ack_completes_and_failure_is_retried_without_false_publish(self): + repository = _Repository(outbox_payload()) + publisher = _Publisher() + dispatcher = AuthorityOutboxDispatcher( + repository=repository, publisher=publisher, + lock_owner="dispatcher-1", batch_size=1, + ) + self.assertEqual(await dispatcher.dispatch_once(), 1) + self.assertEqual(len(repository.completed), 1) + self.assertEqual(repository.retried, []) + self.assertEqual(publisher.calls[0][0], "BINANCE:USDM:TRADE:ETHUSDT") + + failed_repository = _Repository(outbox_payload()) + failed = AuthorityOutboxDispatcher( + repository=failed_repository, publisher=_Publisher(fail=True), + lock_owner="dispatcher-2", batch_size=1, + ) + self.assertEqual(await failed.dispatch_once(), 0) + self.assertEqual(failed_repository.completed, []) + self.assertEqual(len(failed_repository.retried), 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phaseb_stable_deployment.py b/tests/test_phaseb_stable_deployment.py index 99ab11a..a21cc30 100644 --- a/tests/test_phaseb_stable_deployment.py +++ b/tests/test_phaseb_stable_deployment.py @@ -6,7 +6,6 @@ import tempfile import time import unittest -from datetime import datetime, timezone from unittest.mock import AsyncMock, patch from pathlib import Path @@ -20,6 +19,7 @@ from qdl.runtime.stable_deployment import ( STABLE_CORE_WORKER_COUNT, + AuthorityPromotionScope, StableAcquisitionPlan, stable_authority_record, write_stable_runtime_bundle, @@ -29,6 +29,7 @@ ROOT = Path(__file__).resolve().parents[1] CATALOG_PATH = ROOT / "config/v2/stable-source-bindings.yaml" ACQUISITION_PATH = ROOT / "config/v2/stable-acquisition-bindings.yaml" +PROMOTION_SCOPE_PATH = ROOT / "config/v2/stable-authority-promotion-scope.yaml" class StableDeploymentContractTests(unittest.TestCase): @@ -37,6 +38,9 @@ def setUp(self) -> None: self.acquisition = StableAcquisitionPlan.load( ACQUISITION_PATH, catalog=self.catalog ) + self.promotion_scope = AuthorityPromotionScope.load( + PROMOTION_SCOPE_PATH, catalog=self.catalog + ) self.authority = stable_authority_record( rust_image_digest="a" * 64, capability_manifest=ROOT / "config/v2/stable-capabilities.yaml", @@ -45,6 +49,53 @@ def setUp(self) -> None: effective_at_ns=time.time_ns(), ) + def test_initial_authority_scope_is_explicit_and_excludes_dnse(self): + expected = { + item.binding_id + for item in self.catalog.bindings + if item.instrument.identity.venue in {"BINANCE", "OKX"} + } + self.assertEqual(set(self.promotion_scope.binding_ids), expected) + self.assertEqual(len(expected), 12) + runtime = self.acquisition.production_core_config( + catalog=self.catalog, + raw_authority=self.authority, + promotion_scope=self.promotion_scope, + worker_index=1, + ) + self.assertEqual(len(runtime["slices"]), 12) + self.assertEqual( + {item["subscription_id"] for item in runtime["slices"]}, + { + item.source_id + for item in self.catalog.bindings + if item.binding_id in expected + }, + ) + self.assertEqual( + {item["venue"] for item in runtime["core"]["bindings"]}, + {"BINANCE", "OKX"}, + ) + + def test_authority_scope_rejects_unknown_and_duplicate_bindings(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "scope.yaml" + path.write_text( + "schema: qdl.v2.authority-promotion-scope.v1\n" + "revision: 1\nbinding_ids: [missing-binding]\n", + encoding="utf-8", + ) + with self.assertRaisesRegex(ValueError, "unknown bindings"): + AuthorityPromotionScope.load(path, catalog=self.catalog) + path.write_text( + "schema: qdl.v2.authority-promotion-scope.v1\n" + "revision: 1\nbinding_ids: [binance-usdm-btcusdt-trade, " + "binance-usdm-btcusdt-trade]\n", + encoding="utf-8", + ) + with self.assertRaisesRegex(ValueError, "scope is invalid"): + AuthorityPromotionScope.load(path, catalog=self.catalog) + def test_all_catalog_bindings_have_one_capability_truthful_acquisition(self): self.assertEqual(len(self.catalog.bindings), 16) self.assertEqual(len(self.acquisition.bindings), 16) @@ -698,6 +749,16 @@ def test_missing_binding_wrong_provider_kind_and_primary_authority_fail_closed(s class StableComposeAndBundleTests(unittest.TestCase): + def test_python_release_base_is_digest_pinned_in_both_stages(self): + dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8") + pinned = ( + "python:3.12-slim@sha256:" + "2c941e860699f878900b0edc2403613c234d4b32" + "eda3cc9fa7036991a2a63c4a" + ) + self.assertEqual(dockerfile.count(pinned), 2) + self.assertNotIn("FROM python:3.12-slim AS", dockerfile) + def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): raw = (ROOT / "docker-compose.v2-stable.yml").read_text(encoding="utf-8") compose = yaml.safe_load(raw) @@ -708,7 +769,8 @@ def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): self.assertFalse(compose["networks"]["stable_ingress"].get("internal", False)) for name in ("query_v2_1", "query_v2_2", "stream_v2_active", "stream_v2_passive"): self.assertEqual( - set(services[name]["networks"]), {"stable_internal", "stable_ingress"} + set(services[name]["networks"]), + {"stable_internal", "stable_ingress", "stable_consumer"}, ) self.assertTrue( all(str(port).startswith("127.0.0.1:") for port in services[name]["ports"]) @@ -753,6 +815,30 @@ def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): self.assertTrue(services[name]["read_only"]) self.assertIn("ALL", services[name]["cap_drop"]) self.assertEqual(services[name]["restart"], "no") + ingress_aliases = { + "query_v2_1": "qdl-v2-query", + "query_v2_2": "qdl-v2-query", + "stream_v2_active": "qdl-v2-stream-a", + "stream_v2_passive": "qdl-v2-stream-b", + } + for name, alias in ingress_aliases.items(): + self.assertEqual( + services[name]["networks"]["stable_consumer"]["aliases"], + [alias], + ) + self.assertEqual( + compose["networks"]["stable_consumer"], + { + "external": True, + "name": "${QDL_STABLE_CONSUMER_NETWORK:" + "?set QDL_STABLE_CONSUMER_NETWORK}", + }, + ) + for name in ( + "kafka1", "kafka2", "kafka3", "stable_redis", "projector_v2", + "rust_core", "rust_core_2", "rust_core_3", + ): + self.assertNotIn("stable_consumer", services[name]["networks"]) self.assertEqual( set(services["ingestor_okx_swap"]["networks"]), {"stable_internal", "stable_egress"}, @@ -810,13 +896,76 @@ def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): ) self.assertIn("stable_tls:/stable-certs:ro", services[name]["volumes"]) + authority_db = services["stable_authority_db"] + self.assertEqual(authority_db["profiles"], ["stable-authority"]) + self.assertNotIn("ports", authority_db) + self.assertIn( + "./migrations/postgres:/docker-entrypoint-initdb.d:ro", + authority_db["volumes"], + ) + self.assertIn( + "stable_authority_db:/var/lib/postgresql/data", + authority_db["volumes"], + ) + dispatcher = services["authority_outbox_v2"] + self.assertEqual(dispatcher["profiles"], ["stable-authority"]) + self.assertEqual(dispatcher["networks"], ["stable_internal"]) + self.assertNotIn("ports", dispatcher) + self.assertEqual( + dispatcher["environment"]["QDL_AUTHORITY_TOPIC"], "qdl.authority.v1" + ) + self.assertIn( + "/stable-certs/authority-dispatcher/client.crt", + dispatcher["environment"]["QDL_KAFKA_CERT_LOCATION"], + ) + production_names = ( + "production_core_1", "production_core_2", "production_core_3" + ) + self.assertEqual( + { + services[name]["environment"]["QDL_KAFKA_CLIENT_ID"] + for name in production_names + }, + { + "qdl-v2-production-core-001", + "qdl-v2-production-core-002", + "qdl-v2-production-core-003", + }, + ) + for name in production_names: + with self.subTest(production_service=name): + self.assertEqual( + services[name]["profiles"], ["stable-authority-primary"] + ) + self.assertEqual( + services[name]["entrypoint"], + ["/usr/local/bin/qdl-production-core"], + ) + self.assertEqual(services[name]["user"], "10001:10001") + self.assertTrue(services[name]["read_only"]) + self.assertNotIn("ports", services[name]) + def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): with tempfile.TemporaryDirectory(prefix="qdl-phaseb-cert-") as cert_directory: certs = Path(cert_directory) (certs / "ca.crt").write_text("ca", encoding="ascii") - for principal in ("phase8-producer", "phase8-core", "phase8-consumer"): + for principal in ( + "phase8-producer", + "phase8-core", + "phase8-consumer", + "stable-authority-dispatcher", + "stable-trading-system", + "stable-query", + "stable-stream", + ): (certs / f"{principal}.crt").write_text("crt", encoding="ascii") (certs / f"{principal}.key").write_text("key", encoding="ascii") + (certs / "stable-trading-system-jwt.key").write_text( + "private", encoding="ascii" + ) + (certs / "stable-trading-system-jwt.public.pem").write_text( + "public", encoding="ascii" + ) with tempfile.TemporaryDirectory(prefix="qdl-phaseb-output-") as parent: output = Path(parent) / "candidate" with patch( @@ -828,22 +977,56 @@ def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): python_image="qdl-python:test", cert_dir=certs, output_dir=output, + consumer_network="executor_network", host_cert_dir=Path("/host/qdl/certs"), host_output_dir=Path("/host/qdl/candidate"), ) self.assertFalse(manifest["cutover_authorized"]) self.assertFalse(manifest["secret_values_recorded"]) + self.assertEqual(manifest["authority_promotion_binding_count"], 12) + self.assertEqual(manifest["consumer_network"], "executor_network") + self.assertEqual(len(manifest["authority_promotion_scope_digest"]), 64) + production = json.loads( + (output / "runtime/production-core-001.json").read_text() + ) + self.assertEqual(len(production["slices"]), 12) + self.assertEqual( + {item["venue"] for item in production["core"]["bindings"]}, + {"BINANCE", "OKX"}, + ) self.assertEqual((output / "stable.env").stat().st_mode & 0o777, 0o600) env_text = (output / "stable.env").read_text() self.assertIn("QDL_STABLE_CERT_DIR=/host/qdl/certs", env_text) + self.assertIn("QDL_STABLE_CONSUMER_NETWORK=executor_network", env_text) self.assertIn( "QDL_STABLE_RUNTIME_DIR=/host/qdl/candidate/runtime", env_text ) + self.assertIn( + "QDL_STABLE_AUTHORITY_CERT_DIR=" + "/host/qdl/candidate/identities/authority-dispatcher", + env_text, + ) + self.assertIn( + "postgresql://qdl_authority_dispatcher:", env_text + ) public_manifest = (output / "candidate-manifest.json").read_text() self.assertNotIn("QDL_STABLE_INTERNAL_INGEST_SECRET", public_manifest) for name in ("core.json", "core-002.json", "core-003.json"): self.assertTrue((output / f"runtime/{name}").is_file()) + for index in range(1, 4): + self.assertTrue( + (output / f"runtime/production-core-{index:03d}.json").is_file() + ) + self.assertTrue( + (output / "runtime/production-core-manifest.json").is_file() + ) self.assertTrue((output / "identities/projector/client.key").is_file()) + self.assertTrue( + ( + output + / "identities/authority-dispatcher/client.key" + ).is_file() + ) if __name__ == "__main__": diff --git a/tests/test_phaseb_stable_edge.py b/tests/test_phaseb_stable_edge.py index a17df9c..19d8f05 100644 --- a/tests/test_phaseb_stable_edge.py +++ b/tests/test_phaseb_stable_edge.py @@ -34,7 +34,7 @@ ProductType, ) from qdl.domain.decimal import CanonicalDecimal -from qdl.query import ConsumerGrade, DataRequirement, FeedType +from qdl.query import ConsumerGrade, DataRequirement from qdl.projection.stable import ( InMemoryStableProjectionTarget, ProjectionCacheMismatch, @@ -1319,6 +1319,8 @@ async def test_stale_projection_epoch_fails_without_broker_checkpoint(self): class StableRuntimeBoundaryTests(unittest.TestCase): def environment(self, root): + for name in ("ca.crt", "workload.crt", "workload.key"): + (root / name).write_text("test", encoding="utf-8") return { "QDL_ENVIRONMENT": "paper", "QDL_CONFIG_REVISION": "phase-b-test-1", @@ -1330,6 +1332,9 @@ def environment(self, root): "QDL_STABLE_AUDIT_PATH": str(root / "state" / "audit.jsonl"), "QDL_STABLE_CONSUMER_MANIFESTS": str(root / "consumer.yaml"), "QDL_STABLE_SOURCE_BINDINGS": str(CATALOG_PATH), + "QDL_STABLE_TLS_CA_FILE": str(root / "ca.crt"), + "QDL_STABLE_TLS_CERT_FILE": str(root / "workload.crt"), + "QDL_STABLE_TLS_KEY_FILE": str(root / "workload.key"), "QDL_STABLE_INTERNAL_INGEST_SECRET": "i" * 32, "QDL_STABLE_REDIS_URL": "redis://qdl-stable-redis:6379/0", "QDL_STABLE_REDIS_PREFIX": "qdl:stable:v2:paper:test", @@ -1346,9 +1351,13 @@ def test_projector_sink_accepts_only_declared_internal_stream_roles(self): secret, object(), client=client, ) self.assertIs(sink.client, client) - with self.assertRaisesRegex(ValueError, "configuration is invalid"): + secure = StableHttpCanonicalSink( + ("https://stream_v2_active:8200",), secret, object(), client=client + ) + self.assertIs(secure.client, client) + with self.assertRaisesRegex(ValueError, "workload TLS context"): StableHttpCanonicalSink( - ("https://stream_v2_active:8200",), secret, object(), client=client + ("https://stream_v2_active:8200",), secret, object() ) with self.assertRaisesRegex(ValueError, "configuration is invalid"): StableHttpCanonicalSink( diff --git a/tests/test_phaseb_stable_rebuild.py b/tests/test_phaseb_stable_rebuild.py index a086720..6e56e64 100644 --- a/tests/test_phaseb_stable_rebuild.py +++ b/tests/test_phaseb_stable_rebuild.py @@ -17,6 +17,8 @@ QUERY_SERVICES, STOP_SERVICES, STREAM_SERVICES, + _env_value, + _stable_client_ssl_context, _start_services, _validate_project, compose_command, @@ -60,6 +62,28 @@ def test_apply_requires_exact_confirmation(self): require_authorization(apply=True, confirm="WRONG") require_authorization(apply=True, confirm=CONFIRM_TOKEN) + def test_tls_identity_is_loaded_from_exact_env_binding(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + identity = root / "identity" + identity.mkdir() + env = root / "stable.env" + env.write_text( + f"QDL_STABLE_TRADING_SYSTEM_CERT_DIR={identity}\n" + ) + self.assertEqual( + _env_value(env, "QDL_STABLE_TRADING_SYSTEM_CERT_DIR"), + str(identity), + ) + with self.assertRaises(FileNotFoundError): + _stable_client_ssl_context(env) + env.write_text( + "QDL_STABLE_TRADING_SYSTEM_CERT_DIR=/first\n" + "QDL_STABLE_TRADING_SYSTEM_CERT_DIR=/second\n" + ) + with self.assertRaisesRegex(ValueError, "exactly one"): + _env_value(env, "QDL_STABLE_TRADING_SYSTEM_CERT_DIR") + def test_compose_command_is_pinned_to_stable_manifest(self): command = compose_command(Path("/tmp/stable.env"), "config") self.assertEqual(command[:2], ["docker", "compose"]) diff --git a/tests/test_phasec2_transport_security.py b/tests/test_phasec2_transport_security.py new file mode 100644 index 0000000..fa5d306 --- /dev/null +++ b/tests/test_phasec2_transport_security.py @@ -0,0 +1,141 @@ +from __future__ import annotations + +import ssl +import tempfile +import time +import unittest +from pathlib import Path +from unittest import mock + +import grpc +import httpx + +from qdl_sdk import GrpcStreamTransport, RestQueryTransport, WorkloadTlsConfig + + +class WorkloadTlsConfigTests(unittest.IsolatedAsyncioTestCase): + def identity(self, root: Path) -> WorkloadTlsConfig: + paths = [] + for name in ("ca.crt", "client.crt", "client.key"): + path = root / name + path.write_text("test-only", encoding="utf-8") + paths.append(path) + return WorkloadTlsConfig(*paths) + + async def test_missing_identity_fails_closed(self): + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + with self.assertRaisesRegex(ValueError, "TLS files are unavailable"): + WorkloadTlsConfig( + root / "missing-ca", + root / "missing-cert", + root / "missing-key", + ) + + async def test_rest_and_grpc_share_one_workload_identity(self): + with tempfile.TemporaryDirectory() as temp: + tls = self.identity(Path(temp)) + context = ssl.create_default_context() + credentials = grpc.ssl_channel_credentials() + with ( + mock.patch.object( + WorkloadTlsConfig, "ssl_context", return_value=context + ) as rest_tls, + mock.patch.object( + WorkloadTlsConfig, + "grpc_credentials", + return_value=credentials, + ) as grpc_tls, + ): + rest = RestQueryTransport("https://qdl-v2-query:8200", tls=tls) + stream = GrpcStreamTransport( + ("qdl-v2-stream-a:8210", "qdl-v2-stream-b:8210"), + tls=tls, + ) + try: + self.assertEqual( + stream.targets, + ("qdl-v2-stream-a:8210", "qdl-v2-stream-b:8210"), + ) + self.assertEqual(stream.target, "qdl-v2-stream-a:8210") + rest_tls.assert_called_once_with() + grpc_tls.assert_called_once_with() + finally: + await rest.close() + await stream.close() + + + async def test_rs256_token_is_cached_rotated_and_verified(self): + from cryptography.hazmat.primitives import serialization + from cryptography.hazmat.primitives.asymmetric import rsa + from qdl.security.policy import ServiceTokenVerifier + from qdl_sdk import RotatingJwtCredentialProvider + + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + private_path = root / "private.key" + private_path.write_bytes(key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.PKCS8, + serialization.NoEncryption(), + )) + public_key = key.public_key().public_bytes( + serialization.Encoding.PEM, + serialization.PublicFormat.SubjectPublicKeyInfo, + ) + now = [int(time.time())] + provider = RotatingJwtCredentialProvider( + private_key_file=private_path, + key_id="workload-rs256-v1", + algorithm="RS256", + issuer="https://identity.qdl", + audience="qdl-v2", + subject="spiffe://qdl/paper/trading-system-stable", + environment="paper", + roles=("market_data_reader", "stream_consumer"), + venues=("BINANCE", "OKX"), + consumer_manifest_revision=1, + lifetime_seconds=600, + refresh_before_seconds=120, + clock=lambda: now[0], + ) + first = await provider.get_token() + self.assertEqual(first, await provider.get_token()) + principal = ServiceTokenVerifier( + issuer="https://identity.qdl", + audience="qdl-v2", + keys_by_id={"workload-rs256-v1": public_key}, + algorithms=("RS256",), + max_lifetime_seconds=900, + ).verify(first, expected_environment="paper") + self.assertEqual( + principal.subject, + "spiffe://qdl/paper/trading-system-stable", + ) + self.assertEqual(principal.venues, frozenset({"BINANCE", "OKX"})) + now[0] += 500 + self.assertNotEqual(first, await provider.get_token()) + + async def test_ambiguous_tls_clients_and_duplicate_targets_are_rejected(self): + with tempfile.TemporaryDirectory() as temp: + tls = self.identity(Path(temp)) + async with httpx.AsyncClient() as client: + with self.assertRaisesRegex(ValueError, "either REST client"): + RestQueryTransport( + "https://qdl-v2-query:8200", client=client, tls=tls + ) + with self.assertRaisesRegex(ValueError, "non-empty and unique"): + GrpcStreamTransport( + ("127.0.0.1:18220", "127.0.0.1:18220"), + allow_insecure_loopback=True, + ) + with self.assertRaisesRegex(ValueError, "only for explicit loopback"): + GrpcStreamTransport( + ("stream-a:8210", "stream-b:8210"), + allow_insecure_loopback=True, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phasec3_authority_runtime.py b/tests/test_phasec3_authority_runtime.py new file mode 100644 index 0000000..15a53a5 --- /dev/null +++ b/tests/test_phasec3_authority_runtime.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import json +from pathlib import Path +import tempfile +import unittest + +from scripts.phaseb_bootstrap_stable_broker import TOPIC_POLICIES +from scripts.run_authority_outbox_dispatcher import write_health + + +ROOT = Path(__file__).resolve().parents[1] + + +class AuthorityRuntimeContractTests(unittest.TestCase): + def test_authority_and_checkpoint_topics_are_compacted(self): + self.assertEqual(TOPIC_POLICIES["qdl.authority.v1"], "compact") + self.assertEqual(TOPIC_POLICIES["qdl.target-checkpoint.v1"], "compact") + self.assertEqual(TOPIC_POLICIES["md.canonical.v2"], "delete") + source = ( + ROOT / "scripts/phaseb_bootstrap_stable_broker.py" + ).read_text(encoding="utf-8") + self.assertIn('"stable-authority-dispatcher"', source) + self.assertIn('"qdl-v2-production-core-"', source) + self.assertIn('"qdl-v2-production-core-v1-"', source) + + def test_dispatcher_role_is_function_scoped_and_not_table_writer(self): + migration = ( + ROOT / "migrations/postgres/0010_authority_dispatcher_security.sql" + ).read_text(encoding="utf-8") + init = ( + ROOT / "migrations/postgres/9999_init_authority_dispatcher_role.sh" + ).read_text(encoding="utf-8") + self.assertEqual(migration.count("SECURITY DEFINER"), 3) + self.assertEqual(migration.count("SET search_path = pg_catalog, public"), 3) + self.assertEqual(migration.count("FROM PUBLIC"), 3) + self.assertIn("NOSUPERUSER NOCREATEDB NOCREATEROLE", init) + self.assertEqual(init.count("GRANT EXECUTE ON FUNCTION"), 3) + self.assertNotIn("GRANT INSERT", init) + self.assertNotIn("GRANT UPDATE", init) + self.assertNotIn("GRANT DELETE", init) + + def test_dispatcher_health_is_atomic_bounded_and_fail_closed(self): + with tempfile.TemporaryDirectory(prefix="qdl-authority-health-") as directory: + path = Path(directory) / "runtime/health.json" + write_health(path, status="READY", published=3) + payload = json.loads(path.read_text(encoding="utf-8")) + self.assertEqual(payload["schema"], "qdl.authority-dispatcher-health.v1") + self.assertEqual(payload["status"], "READY") + self.assertEqual(payload["published_last_cycle"], 3) + self.assertGreater(payload["heartbeat_ns"], 0) + self.assertFalse(path.with_suffix(".json.tmp").exists()) + + write_health( + path, status="DEGRADED", published=0, error="x" * 2000 + ) + degraded = json.loads(path.read_text(encoding="utf-8")) + self.assertEqual(degraded["status"], "DEGRADED") + self.assertEqual(len(degraded["error"]), 1000) + with self.assertRaises(ValueError): + write_health(path, status="UNKNOWN", published=0) + with self.assertRaises(ValueError): + write_health(path, status="READY", published=-1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phasec3_cutover_packet.py b/tests/test_phasec3_cutover_packet.py new file mode 100644 index 0000000..e495fc8 --- /dev/null +++ b/tests/test_phasec3_cutover_packet.py @@ -0,0 +1,136 @@ +from __future__ import annotations + +import copy +import unittest + +from qdl.control.cutover_packet import AuthorityCutoverPacket +from scripts.phasec3_authority_cutover import _assert_current + + +NOW = 1_800_000_000_000_000_000 +DIGEST = "a" * 64 + + +def packet(stage: str = "CANARY") -> dict: + pairs = { + "SHADOW_VALIDATE": ("RUST_SHADOW", "VALIDATING"), + "CANARY": ("VALIDATING", "RUST_CANARY"), + "PRIMARY": ("RUST_CANARY", "RUST_PRIMARY"), + } + expected, new = pairs[stage] + active = stage in {"CANARY", "PRIMARY"} + return { + "schema": "qdl.c3.authority-cutover-packet.v1", + "packet_id": "10000000-0000-4000-8000-000000000001", + "stage": stage, + "issued_at_ns": NOW - 1, + "expires_at_ns": NOW + 1_000_000_000, + "actor": "operator@example", + "change_ticket": "QDL-C3-001", + "candidate_digest": DIGEST, + "artifact_image_digest": "sha256:" + "b" * 64, + "contract_digest": "c" * 64, + "partition_plan_digest": "d" * 64, + "route_manifest_digest": "e" * 64, + "consumer_route": { + "consumer_id": "trading-system", + "expected_route": "V1", + "new_route": "V2_PRIMARY", + "rollback_route": "V1", + "rollback_command": ["scripts/qdl_route.py", "--route", "V1"], + }, + "evidence": { + "provider_provenance": "REAL", + "semantic_mismatches": 0, + "open_gaps": 0, + "duplicate_external_effects": 0, + "consumer_errors": 0, + }, + "slices": [{ + "transition_id": "20000000-0000-4000-8000-000000000002", + "handoff_id": ( + "30000000-0000-4000-8000-000000000003" + if stage == "PRIMARY" else None + ), + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "expected_state": expected, + "expected_revision": 2, + "expected_owner_id": "rust-shadow", + "expected_lease_epoch": 1, + "partition_plan_epoch": 1, + "new_state": new, + "new_owner_id": "rust-canary", + "new_lease_epoch": 2, + "terminal_watermark": 500 if active else None, + "prerequisite_bundle_id": ( + "40000000-0000-4000-8000-000000000004" if active else None + ), + "hold_until": "2026-08-20T12:00:00Z" if active else None, + "reason": "bounded C.3 transition", + }], + } + + +class AuthorityCutoverPacketTests(unittest.TestCase): + def test_plan_is_immutable_and_has_no_mutation(self): + parsed = AuthorityCutoverPacket.parse(packet(), now_ns=NOW) + plan = parsed.plan() + self.assertEqual(plan["stage"], "CANARY") + self.assertEqual(plan["slice_count"], 1) + self.assertEqual(plan["production_mutations"], 0) + self.assertTrue(plan["confirmation_token"].startswith("APPLY_C3_")) + + changed = packet() + changed["change_ticket"] = "QDL-C3-002" + changed_packet = AuthorityCutoverPacket.parse(changed, now_ns=NOW) + self.assertNotEqual( + parsed.confirmation_token, changed_packet.confirmation_token + ) + + def test_unknown_dirty_or_invalid_transition_fails_closed(self): + cases = [] + unknown = packet() + unknown["unexpected"] = True + cases.append(unknown) + dirty = packet() + dirty["evidence"]["semantic_mismatches"] = 1 + cases.append(dirty) + wrong_pair = packet() + wrong_pair["slices"][0]["new_state"] = "RUST_PRIMARY" + cases.append(wrong_pair) + missing_rollback = packet() + missing_rollback["consumer_route"]["rollback_command"] = [] + cases.append(missing_rollback) + for value in cases: + with self.subTest(value=value): + with self.assertRaises(ValueError): + AuthorityCutoverPacket.parse(value, now_ns=NOW) + + def test_primary_requires_handoff_and_exact_database_preconditions(self): + parsed = AuthorityCutoverPacket.parse(packet("PRIMARY"), now_ns=NOW) + item = parsed.slices[0] + row = { + "state": item.expected_state, + "authority_revision": item.expected_revision, + "owner_id": item.expected_owner_id, + "lease_epoch": item.expected_lease_epoch, + "partition_plan_epoch": item.partition_plan_epoch, + "candidate_digest": parsed.candidate_digest, + "artifact_image_digest": parsed.artifact_image_digest, + "contract_digest": parsed.contract_digest, + "partition_plan_digest": parsed.partition_plan_digest, + } + _assert_current(row, item, parsed) + stale = copy.deepcopy(row) + stale["authority_revision"] += 1 + with self.assertRaises(RuntimeError): + _assert_current(stale, item, parsed) + + no_handoff = packet("PRIMARY") + no_handoff["slices"][0]["handoff_id"] = None + with self.assertRaises(ValueError): + AuthorityCutoverPacket.parse(no_handoff, now_ns=NOW) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_production_catalog.py b/tests/test_production_catalog.py new file mode 100644 index 0000000..dbdd01f --- /dev/null +++ b/tests/test_production_catalog.py @@ -0,0 +1,231 @@ +from __future__ import annotations + +import json +from pathlib import Path +import tempfile +import unittest + +from qdl.adapters.binance_usdm import parse_exchange_info +from qdl.runtime.production_catalog import ( + ProductionCatalogBuilder, + ProductionDemandManifest, +) +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import ( + AuthorityPromotionScope, + StableAcquisitionPlan, +) + + +BINANCE = { + "serverTime": 1000, + "symbols": [ + { + "symbol": "ETHUSDT", + "status": "TRADING", + "contractType": "PERPETUAL", + "baseAsset": "ETH", + "quoteAsset": "USDT", + "marginAsset": "USDT", + "deliveryDate": 0, + "filters": [ + {"filterType": "PRICE_FILTER", "tickSize": "0.01"}, + {"filterType": "LOT_SIZE", "stepSize": "0.001"}, + ], + }, + { + "symbol": "BTCUSDT_260925", + "status": "TRADING", + "contractType": "CURRENT_QUARTER", + "baseAsset": "BTC", + "quoteAsset": "USDT", + "marginAsset": "USDT", + "deliveryDate": 1789948800000, + "filters": [ + {"filterType": "PRICE_FILTER", "tickSize": "0.1"}, + {"filterType": "LOT_SIZE", "stepSize": "0.001"}, + ], + }, + ], +} +OKX = [ + { + "instType": "SWAP", + "instId": "ETH-USDT-SWAP", + "instFamily": "ETH-USDT", + "baseCcy": "ETH", + "quoteCcy": "USDT", + "settleCcy": "USDT", + "tickSz": "0.01", + "lotSz": "1", + "ctVal": "0.1", + "ctMult": "1", + "state": "live", + } +] + + +class ProductionCatalogTests(unittest.TestCase): + def _manifest(self, root: Path, *, conflicting=False) -> ProductionDemandManifest: + policy = "other_policy" if conflicting else "crypto_primary_v2" + payload = { + "schema": "qdl.v2.production-demand.v1", + "revision": 5, + "consumers": [ + { + "consumer_id": "trading-system.execution.v2", + "consumer_grade": "EXECUTION", + "requirements": [ + { + "venue": "BINANCE", "market": "USDM", + "product_type": "PERPETUAL", "native_symbol": "ETHUSDT", + "feed": "TRADE", "interval": None, + "source_policy_id": "crypto_primary_v2", + }, + { + "venue": "OKX", "market": "SWAP", + "product_type": "PERPETUAL", "native_symbol": "ETH-USDT-SWAP", + "feed": "BAR", "interval": "1m", + "source_policy_id": "crypto_primary_v2", + }, + ], + }, + { + "consumer_id": "alpha.shared.v2", + "consumer_grade": "ALPHA", + "requirements": [ + { + "venue": "BINANCE", "market": "USDM", + "product_type": "PERPETUAL", "native_symbol": "ETHUSDT", + "feed": "TRADE", "interval": None, + "source_policy_id": policy, + } + ], + }, + ], + } + path = root / ("conflict.yaml" if conflicting else "demand.yaml") + import yaml + path.write_text(yaml.safe_dump(payload, sort_keys=False)) + return ProductionDemandManifest.load_many([path]) + + def test_binance_identity_uses_canonical_pair_and_dated_contract_code(self): + discovery = parse_exchange_info(BINANCE, valid_from_ns=0) + by_native = {item.native_symbol: item for item in discovery.records} + self.assertEqual( + by_native["ETHUSDT"].instrument_id, + "BINANCE.USDM.PERPETUAL.ETH-USDT", + ) + self.assertEqual( + by_native["BTCUSDT_260925"].instrument_id, + "BINANCE.USDM.FUTURE.BTC-USDT-260925", + ) + + def test_generator_is_deterministic_validated_and_metadata_authentic(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + demand = self._manifest(root) + builder = ProductionCatalogBuilder( + catalog_revision=8, + source_policy_revision=3, + authority_revision=11, + ) + first = builder.build( + demand=demand, + binance_usdm=parse_exchange_info(BINANCE, valid_from_ns=99), + okx_rows=OKX, + metadata_provenance={"capture": "a" * 64}, + ) + second = builder.build( + demand=demand, + binance_usdm=parse_exchange_info(BINANCE, valid_from_ns=123), + okx_rows=list(reversed(OKX)), + metadata_provenance={"capture": "a" * 64}, + ) + self.assertEqual(first.source_catalog, second.source_catalog) + self.assertEqual(first.acquisition_plan, second.acquisition_plan) + self.assertEqual(first.provenance["fabricated_metadata"], False) + self.assertEqual(first.provenance["instrument_count"], 2) + paths = first.write(root / "out") + catalog = StableSourceCatalog.load(paths["source_catalog"]) + self.assertEqual(len(catalog.bindings), 2) + self.assertEqual( + {item.instrument.instrument_id for item in catalog.bindings}, + { + "BINANCE.USDM.PERPETUAL.ETH-USDT", + "OKX.SWAP.PERPETUAL.ETH-USDT", + }, + ) + self.assertEqual( + json.loads(Path(paths["provenance"]).read_text())["binding_count"], 2 + ) + acquisition = StableAcquisitionPlan.load( + paths["acquisition_plan"], catalog=catalog + ) + raw_authority = { + "schema": "qdl.authority-record.v1", + "slice_id": "qdl-v2-production-acquisition", + "revision": 11, + "mode": "RUST_SHADOW", + "candidate_image_digest": "sha256:" + "1" * 64, + "capability_manifest_digest": "2" * 64, + "contract_digest": "3" * 64, + "partition_plan_digest": "4" * 64, + "public_write_allowed": False, + "legacy_write_allowed": False, + "approved_by": "production-catalog-test", + "effective_at_ns": 1, + } + promotion_scope = AuthorityPromotionScope( + schema="qdl.v2.authority-promotion-scope.v1", + revision=1, + binding_ids=tuple(item.binding_id for item in catalog.bindings), + ) + runtime = acquisition.production_core_config( + catalog=catalog, + raw_authority=raw_authority, + promotion_scope=promotion_scope, + worker_index=1, + ) + self.assertEqual(len(runtime["slices"]), 2) + self.assertEqual( + {item["subscription_id"] for item in runtime["slices"]}, + {item.source_id for item in catalog.bindings}, + ) + self.assertEqual(runtime["topics"]["primary_canonical"], "md.canonical.v2") + self.assertEqual(runtime["topics"]["authority_control"], "qdl.authority.v1") + self.assertEqual(runtime["batch_size"], 128) + + def test_conflicts_missing_metadata_and_uncertified_interval_fail_closed(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + with self.assertRaisesRegex(ValueError, "conflicting source policies"): + self._manifest(root, conflicting=True) + demand = self._manifest(root) + with self.assertRaisesRegex(ValueError, "metadata missing"): + ProductionCatalogBuilder( + catalog_revision=1, + source_policy_revision=1, + authority_revision=1, + ).build(demand=demand, binance_usdm=None, okx_rows=[]) + payload = { + "schema": "qdl.v2.production-demand.v1", + "revision": 1, + "consumers": [{ + "consumer_id": "alpha", "consumer_grade": "ALPHA", + "requirements": [{ + "venue": "OKX", "market": "SWAP", "product_type": "PERPETUAL", + "native_symbol": "ETH-USDT-SWAP", "feed": "BAR", + "interval": "15m", "source_policy_id": "crypto_primary_v2", + }], + }], + } + import yaml + path = root / "bad-interval.yaml" + path.write_text(yaml.safe_dump(payload)) + with self.assertRaisesRegex(ValueError, "currently certified for 1m"): + ProductionDemandManifest.load_many([path]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_qdl_sdk_release.py b/tests/test_qdl_sdk_release.py new file mode 100644 index 0000000..5d5031d --- /dev/null +++ b/tests/test_qdl_sdk_release.py @@ -0,0 +1,163 @@ +from __future__ import annotations + +import csv +import hashlib +import io +import json +from pathlib import Path +import tempfile +import unittest +import zipfile + +from qdl_sdk import AsyncDataLayerClient, Grade +from qdl_sdk.errors import ContinuityError, DataLayerError +from scripts.build_qdl_sdk_release import build_wheel + + +class QdlSdkReleaseTests(unittest.TestCase): + def test_wheel_is_reproducible_self_contained_and_recorded(self): + with tempfile.TemporaryDirectory() as first, tempfile.TemporaryDirectory() as second: + first_result = build_wheel(Path(first)) + second_result = build_wheel(Path(second)) + self.assertEqual(first_result["wheel_sha256"], second_result["wheel_sha256"]) + wheel = Path(first) / str(first_result["wheel"]) + self.assertEqual(hashlib.sha256(wheel.read_bytes()).hexdigest(), first_result["wheel_sha256"]) + with zipfile.ZipFile(wheel) as archive: + names = set(archive.namelist()) + self.assertIn("qdl_sdk/__init__.py", names) + self.assertIn("qdl/query/v2/query_pb2.py", names) + self.assertIn("qdl/marketdata/v2/market_data_pb2.py", names) + self.assertFalse(any(name.startswith("qdl/api_v2/") for name in names)) + record_name = "qdl_sdk-2.0.0.dist-info/RECORD" + rows = list(csv.reader(io.StringIO(archive.read(record_name).decode()))) + self.assertEqual({row[0] for row in rows}, names) + release = json.loads(Path(first_result["manifest"]).read_text()) + self.assertFalse(release["contains_service_internals"]) + self.assertEqual(len(release["generated_contract_digest"]), 64) + sbom = json.loads(Path(first_result["sbom"]).read_text()) + self.assertEqual(sbom["bomFormat"], "CycloneDX") + + +class _CatalogTransport: + def __init__(self, pages): + self.pages = pages + + async def instruments(self, *, consumer_id, consumer_grade, cursor, limit): + del consumer_id, consumer_grade, limit + return self.pages[cursor] + + async def close(self): + return None + + +class _UnusedStreamTransport: + async def close(self): + return None + + +class QdlSdkInstrumentResolverTests(unittest.IsolatedAsyncioTestCase): + @staticmethod + def _item(uid, *, market, native_symbol="BTCUSDT"): + return { + "instrument_uid": uid, + "instrument_id": f"BINANCE.{market}.PERPETUAL.BTC-USDT", + "venue": "BINANCE", + "market": market, + "product_type": "PERPETUAL", + "canonical_symbol": "BTC-USDT", + "metadata_revision": 7, + "asset_class": "CRYPTO_DERIVATIVE", + "native_symbol": native_symbol, + "status": "ACTIVE", + } + + def _client(self, pages): + return AsyncDataLayerClient( + query_transport=_CatalogTransport(pages), + stream_transport=_UnusedStreamTransport(), + consumer_id="consumer-alpha-v2", + ) + + async def test_resolves_exact_active_identity_across_bounded_pages(self): + pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [self._item("uid-spot", market="SPOT", native_symbol="ETHUSDT")], + "next_cursor": "page-2", + }, + "page-2": { + "schema": "qdl.instruments.page.v2", + "items": [self._item("uid-usdm", market="USDM")], + "next_cursor": None, + }, + } + result = await self._client(pages).resolve_instrument( + venue="binance", + product_type="perpetual", + native_symbol="btcusdt", + market="usdm", + consumer_grade=Grade.ALPHA, + ) + self.assertEqual(result.instrument_uid, "uid-usdm") + + async def test_missing_and_ambiguous_identity_fail_closed(self): + missing_pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [], + "next_cursor": None, + } + } + with self.assertRaises(DataLayerError) as missing: + await self._client(missing_pages).resolve_instrument( + venue="BINANCE", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + consumer_grade=Grade.EXECUTION, + ) + self.assertEqual(missing.exception.code, "INSTRUMENT_NOT_FOUND") + + ambiguous_pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [ + self._item("uid-usdm", market="USDM"), + self._item("uid-coinm", market="COINM"), + ], + "next_cursor": None, + } + } + with self.assertRaises(ContinuityError) as ambiguous: + await self._client(ambiguous_pages).resolve_instrument( + venue="BINANCE", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + consumer_grade=Grade.EXECUTION, + ) + self.assertEqual(ambiguous.exception.code, "CONFLICT") + + async def test_catalog_cursor_cycle_fails_closed(self): + pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [], + "next_cursor": "cycle", + }, + "cycle": { + "schema": "qdl.instruments.page.v2", + "items": [], + "next_cursor": "cycle", + }, + } + with self.assertRaises(ContinuityError) as cycle: + await self._client(pages).resolve_instrument( + venue="OKX", + product_type="PERPETUAL", + native_symbol="BTC-USDT-SWAP", + consumer_grade=Grade.ALPHA, + ) + self.assertEqual(cycle.exception.code, "CONFLICT") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_qdl_sdk_stream_projection.py b/tests/test_qdl_sdk_stream_projection.py new file mode 100644 index 0000000..2967346 --- /dev/null +++ b/tests/test_qdl_sdk_stream_projection.py @@ -0,0 +1,364 @@ +from __future__ import annotations + +import unittest + +from qdl.common.v1 import common_pb2 +from qdl.marketdata.v2 import market_data_pb2 +from qdl_sdk import ( + DataRequirement, + Feed, + Grade, + StreamEvent, + market_data_view_from_stream, +) +from qdl_sdk.errors import ContinuityError +from qdl_sdk.models import MarketDataView + +NOW = 1_800_000_000_000_000_000 +DIGEST = "a" * 64 + + +def dec(value: int, scale: int = 2): + return common_pb2.DecimalValue(mantissa=value, scale=scale, source_text=str(value)) + + +def payload_fixture(feed: Feed) -> dict: + common = {"feed": feed.value} + return { + Feed.TRADE: { + **common, + "native_trade_id": "1", + "price": dv(), + "quantity": dv(), + "quantity_unit": "BASE_ASSET", + "aggressor_side": "BUY", + "identity_kind": "NATIVE", + }, + Feed.QUOTE: { + **common, + "bid_price": dv(), + "bid_quantity": dv(), + "ask_price": dv(101), + "ask_quantity": dv(), + "quantity_unit": "BASE_ASSET", + "level": 1, + }, + Feed.BAR: { + **common, + "interval": "1m", + "open_time_ns": NOW - 60_000_000_000, + "close_time_ns": NOW - 1, + "open": dv(), + "high": dv(102), + "low": dv(99), + "close": dv(101), + "volume": dv(), + "volume_unit": "BASE_ASSET", + "trade_count": 1, + "lifecycle": "FINAL", + "revision": 0, + "origin": "VENUE_NATIVE", + }, + Feed.BOOK_SNAPSHOT: { + **common, + "native_sequence": "1", + "levels": [level()], + "depth": 1, + }, + Feed.BOOK_DELTA: { + **common, + "native_sequence_start": "1", + "native_sequence_end": "2", + "snapshot_sequence": "1", + "updates": [level()], + "reset": False, + }, + Feed.FUNDING_RATE: {**common, "rate": dv(), "funding_time_ns": NOW}, + Feed.OPEN_INTEREST: {**common, "quantity": dv(), "quantity_unit": "CONTRACT"}, + Feed.MARK_INDEX_PRICE: {**common, "mark_price": dv(), "index_price": dv()}, + Feed.TICKER: {**common, "last_price": dv()}, + }[feed] + + +def dv(value: int = 100) -> dict: + return {"coefficient": str(value), "scale": 2, "source_text": str(value)} + + +def level() -> dict: + return { + "side": "BID", + "price": dv(), + "quantity": dv(), + "quantity_unit": "BASE_ASSET", + "order_count": 1, + } + + +def template(feed: Feed) -> MarketDataView: + return MarketDataView.model_validate( + { + "instrument_uid": "uid-1", + "instrument_id": "BINANCE.USDM.PERPETUAL.BTC-USDT", + "instrument_revision": 7, + "feed": feed.value, + "interval": "1m" if feed is Feed.BAR else None, + "observed_at_ns": NOW, + "revision": 0, + "payload": payload_fixture(feed), + "source": { + "venue": "BINANCE", + "provider": "BINANCE_DIRECT", + "source_id": "source-1", + "source_role": "PRIMARY", + "authoritative": True, + }, + "quality": { + "state": "LIVE", + "freshness_ms": 1, + "gap_open": False, + "complete": True, + "execution_eligible": True, + "policy_id": "crypto_primary_v2", + "flags": [], + }, + "contract": { + "schema_digest": DIGEST, + "contract_version": "2.0.0", + "normalizer_version": "old", + "adapter_version": "old", + "instrument_catalog_revision": 7, + "source_policy_revision": 2, + "authority_revision": 3, + "config_revision": 1, + "correlation_id": "snapshot", + }, + "watermark_offset": 10, + } + ) + + +def envelope(feed: Feed) -> market_data_pb2.EventEnvelope: + result = market_data_pb2.EventEnvelope( + schema_name="qdl.marketdata.v2.EventEnvelope", + schema_major=2, + schema_minor=0, + event_id=b"event-1", + instrument_uid="uid-1", + instrument_id="BINANCE.USDM.PERPETUAL.BTC-USDT", + instrument_revision=7, + venue="BINANCE", + market="USDM", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + provider="BINANCE_DIRECT", + source_id="source-1", + source_role=common_pb2.SOURCE_ROLE_PRIMARY, + source_event_time_ns=NOW, + normalizer_version="rust-v2", + adapter_version="binance-v2", + correlation_id="stream", + config_revision=2, + authority_revision=3, + ) + if feed is Feed.TRADE: + result.trade.CopyFrom( + market_data_pb2.Trade( + native_trade_id="1", + price=dec(100), + quantity=dec(2), + aggressor_side=common_pb2.AGGRESSOR_SIDE_BUY, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + identity_kind=market_data_pb2.TRADE_IDENTITY_KIND_NATIVE, + ) + ) + elif feed is Feed.QUOTE: + result.quote.CopyFrom( + market_data_pb2.Quote( + bid_price=dec(100), + bid_quantity=dec(2), + ask_price=dec(101), + ask_quantity=dec(3), + level=1, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ) + elif feed is Feed.BAR: + result.bar.CopyFrom( + market_data_pb2.Bar( + interval="1m", + open_time_ns=NOW - 60_000_000_000, + close_time_ns=NOW - 1, + open=dec(100), + high=dec(102), + low=dec(99), + close=dec(101), + volume=dec(4), + trade_count=2, + is_final=True, + revision=0, + origin=common_pb2.BAR_ORIGIN_VENUE_NATIVE, + lifecycle=market_data_pb2.BAR_LIFECYCLE_FINAL, + volume_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ) + elif feed is Feed.BOOK_SNAPSHOT: + result.book_snapshot.CopyFrom( + market_data_pb2.OrderBookSnapshot( + native_sequence="1", + levels=[ + market_data_pb2.BookLevel( + side=common_pb2.BOOK_SIDE_BID, + price=dec(100), + quantity=dec(2), + order_count=1, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ], + depth=1, + ) + ) + elif feed is Feed.BOOK_DELTA: + result.book_delta.CopyFrom( + market_data_pb2.OrderBookDelta( + native_sequence_start="1", + native_sequence_end="2", + snapshot_sequence="1", + updates=[ + market_data_pb2.BookLevel( + side=common_pb2.BOOK_SIDE_ASK, + price=dec(101), + quantity=dec(2), + order_count=1, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ], + ) + ) + elif feed is Feed.FUNDING_RATE: + result.funding_rate.CopyFrom( + market_data_pb2.FundingRate(rate=dec(1, 4), funding_time_ns=NOW) + ) + elif feed is Feed.OPEN_INTEREST: + result.open_interest.CopyFrom( + market_data_pb2.OpenInterest( + quantity=dec(10), quantity_unit=common_pb2.QUANTITY_UNIT_CONTRACT + ) + ) + elif feed is Feed.MARK_INDEX_PRICE: + result.mark_index_price.CopyFrom( + market_data_pb2.MarkIndexPrice(mark_price=dec(100), index_price=dec(99)) + ) + elif feed is Feed.TICKER: + result.ticker.CopyFrom(market_data_pb2.Ticker(last_price=dec(100))) + return result + + +class SdkStreamProjectionTests(unittest.TestCase): + def requirement(self, feed: Feed) -> DataRequirement: + return DataRequirement( + instrument_uid="uid-1", + feed=feed, + consumer_grade=Grade.EXECUTION, + source_policy_id="crypto_primary_v2", + interval="1m" if feed is Feed.BAR else None, + max_freshness_ms=1000, + ) + + def test_all_public_market_payloads_project_to_typed_view(self): + for feed in Feed: + if feed is Feed.UNSPECIFIED: + continue + with self.subTest(feed=feed): + result = market_data_view_from_stream( + StreamEvent(11, "signed", envelope(feed)), + template=template(feed), + requirement=self.requirement(feed), + now_ns=NOW + 100_000_000, + ) + self.assertIs(result.feed, feed) + self.assertEqual(result.watermark_offset, 11) + self.assertEqual(result.cursor, "signed") + self.assertTrue(result.quality.execution_eligible) + + def test_unspecified_feed_is_rejected_at_requirement_boundary(self): + with self.assertRaisesRegex(ValueError, "UNSPECIFIED"): + DataRequirement( + instrument_uid="uid-1", + feed=Feed.UNSPECIFIED, + consumer_grade=Grade.ALPHA, + source_policy_id="crypto_primary_v2", + ) + + def test_gap_and_stale_execution_events_fail_closed(self): + gapped = envelope(Feed.TRADE) + gapped.quality_flags.append(common_pb2.QUALITY_FLAG_SEQUENCE_GAP_BEFORE) + with self.assertRaises(ContinuityError) as gap: + market_data_view_from_stream( + StreamEvent(11, "signed", gapped), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW + 1, + ) + self.assertEqual(gap.exception.code, "OPEN_SEQUENCE_GAP") + with self.assertRaises(ContinuityError) as stale: + market_data_view_from_stream( + StreamEvent(11, "signed", envelope(Feed.TRADE)), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW + 2_000_000_000, + ) + self.assertEqual(stale.exception.code, "DATA_STALE") + + def test_gap_and_stale_alpha_events_obey_typed_policies(self): + requirement = DataRequirement( + instrument_uid="uid-1", + feed=Feed.TRADE, + consumer_grade=Grade.ALPHA, + source_policy_id="crypto_primary_v2", + max_freshness_ms=1000, + ) + gapped = envelope(Feed.TRADE) + gapped.quality_flags.append(common_pb2.QUALITY_FLAG_SEQUENCE_GAP_BEFORE) + with self.assertRaises(ContinuityError) as gap: + market_data_view_from_stream( + StreamEvent(11, "signed", gapped), + template=template(Feed.TRADE), + requirement=requirement, + now_ns=NOW + 1, + ) + self.assertEqual(gap.exception.code, "OPEN_SEQUENCE_GAP") + + with self.assertRaises(ContinuityError) as stale: + market_data_view_from_stream( + StreamEvent(11, "signed", envelope(Feed.TRADE)), + template=template(Feed.TRADE), + requirement=requirement, + now_ns=NOW + 2_000_000_000, + ) + self.assertEqual(stale.exception.code, "DATA_STALE") + + def test_source_transition_and_revision_regression_require_snapshot(self): + changed = envelope(Feed.TRADE) + changed.source_id = "source-2" + with self.assertRaises(ContinuityError) as source: + market_data_view_from_stream( + StreamEvent(11, "signed", changed), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW, + ) + self.assertEqual(source.exception.code, "SOURCE_NON_AUTHORITATIVE") + changed = envelope(Feed.TRADE) + changed.authority_revision = 2 + with self.assertRaises(ContinuityError) as authority: + market_data_view_from_stream( + StreamEvent(11, "signed", changed), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW, + ) + self.assertEqual(authority.exception.code, "SOURCE_NON_AUTHORITATIVE") + + +if __name__ == "__main__": + unittest.main()