From 9e35b34a263fec05af34fa0e1f32b25f316c7a23 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 08:27:23 +0000 Subject: [PATCH 01/21] docs(data-layer): fast-track Rust authority cutover --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 92 ++++++++++++------- .../v2-production-rust-authority-cutover.md | 60 +++++++----- 2 files changed, 96 insertions(+), 56 deletions(-) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index e1fc820..b3923fd 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5887,8 +5887,8 @@ promotion until its provider gates pass. V1 paths and zero V2 paths; - no V2 stable container is running; only the retained `qdl_v2_stable_candidate_stable_tls` volume remains; -- the feature branch is 81 commits ahead of `dev`; the latest released - `2.0.0-2412572` images predate the bounded DNSE closure commits; +- the feature branch is more than 80 commits ahead of `dev`; the latest + released `2.0.0-2412572` images predate the bounded DNSE closure commits; - the stable compose and realtime binaries deliberately accept only `RUST_SHADOW`; Phase 9.2 proves the CAS/handoff/fencing behavior in an isolated rehearsal but is not wired into the long-running stable runtime; @@ -5940,9 +5940,9 @@ authority wiring cannot be deferred as operational debt. plus `docs/runbooks/v2-production-rust-authority-cutover.md`. Read-only runtime inspection proved V1 `0.1.0` still owns port `8100` with 40 V1 paths and zero V2 paths, no V2 containers are running, and the current stable - binaries/config are intentionally shadow-only. The branch is 81 commits ahead - of `dev`; it must merge through CI before a new authority feature branch is - created. + binaries/config are intentionally shadow-only. The branch is more than 80 + commits ahead of `dev`; it must merge through CI before a new authority + feature branch is created. - Documentation whitespace and secret scans passed; stable compose rendered successfully with isolated dummy values and no container start. Host preflight observed 11 GiB available RAM, 108 GiB free disk and eight CPUs. @@ -5951,6 +5951,14 @@ authority wiring cannot be deferred as operational debt. C.0 remains `IN_PROGRESS` until the current PR is CI-green and merged to `dev`; production authority wiring starts only on the new branch named in this phase. Preparation commit: `130da39`. +- `2026-08-20 OPERATOR CUTOVER SIMPLIFICATION RECORDED`: the operator reports + all alpha consumers are stopped and Trading System is the sole active + consumer. Phase C therefore removes staged alpha/monitoring migrations and + uses one bounded Trading System parity-and-route switch followed by a + preapproved Binance/OKX maintenance window. This reduces operations, not + correctness: persistent authority CAS/outbox, sink fencing, W/W+1 handoff, + durable audit and per-slice rollback remain mandatory. V1 stays hot on port + `8100`; DNSE stays V1-only. #### C.1 Isolated Stable V2 Deployment @@ -5965,46 +5973,62 @@ quarantine, bounded lag/resources, broker and process restart recovery, exact cursor continuation, V1 health unchanged and exact disposable cleanup on failure. -#### C.2 Controlled Consumer Canary +#### C.2 Single-Consumer Trading System Cutover -Migrate in order: monitoring, one paper alpha, Trading System paper adapter, -then remaining approved paper consumers. Each manifest performs -warmup -> signed cursor -> replay -> live and has an exact V1 rollback route. -No sandbox/live order consumer is included. A stale, gapped, non-authoritative -or session-invalid read blocks execution. +The operator confirms all alpha consumers are currently stopped and Trading +System is the only active Data Layer consumer. Do not create artificial alpha or +monitoring migration stages. Built-in V2 health/lag/authority telemetry remains +mandatory, but it is not a separate cutover consumer. -#### C.3 Exact-Slice Rust Authority Promotion +Run one bounded Trading System dual-read parity window for Binance and OKX: +V1 remains the decision source while the same requested instruments, timestamps, +decimals, units, final BAR lifecycle and freshness are compared against V2. +After zero correctness mismatch and healthy cursor/replay evidence, switch the +Trading System market-data adapter to V2 in one controlled restart. Configure a +venue-aware rollback route: Binance/OKX primary V2 with explicit V1 fallback; +DNSE remains V1-only. Never splice providers silently--every route transition +records source, reason, watermark and operator/audit identity. -Promotion is one slice at a time, initially one Binance or OKX TRADE slice. -The approval packet must name image IDs, slice/binding, old/new owner, -authority revision, lease/plan epoch, terminal watermark `W`, topics/groups, -ports, volumes, credentials by secret reference, affected consumers, hold -duration and rollback command. +A stale, gapped, non-authoritative, wrong-session or unit-mismatched V2 read +fails closed. Fallback to V1 is allowed only when V1 passes the same +freshness/session/contract checks and the source-switch audit is durable. -The only allowed sequence is: +#### C.3 Fast-Track Rust Authority Promotion -`PYTHON_PRIMARY -> RUST_SHADOW -> RUST_CANARY -> RUST_PRIMARY`. +Promote all approved Binance and OKX feed slices in one maintenance window, but +execute the CAS internally one slice at a time so a failure is isolated. One +operator packet may list the complete slice set, image IDs, old/new owners, +authority/lease/plan revisions, terminal watermarks, topics/groups, ports, +volumes, secret references, Trading System route and rollback command. -Fence the old writer, persist its terminal checkpoint, accept the handoff, -execute the CAS/outbox transition, reconstruct every target through `W`, and -publish first as Rust at `W+1`. Any ambiguity, missing ACK, parity mismatch, -lag/gap, stale CAS or consumer failure enters `BLOCKED` and rolls back under a -newer revision; never restart V1 as an uncoordinated writer. +Each slice still follows: -#### C.4 Hold, Expand And Release +`PYTHON_PRIMARY -> RUST_SHADOW -> RUST_CANARY -> RUST_PRIMARY`. -Hold the first primary slice for the approved observation window with zero -authority ambiguity or unexplained market-data mismatch. Expand independently -by venue/feed manifest; no slice inherits certification. Only after all -registered consumers use V2 may the operator approve routing the stable public -endpoint and opening a V1 sunset window. DNSE remains disabled until its +The canary is bounded by accepted real events and continuity evidence rather +than a long calendar wait. Fence the old writer at `W`, persist its terminal +checkpoint, accept the handoff, execute CAS/outbox, reconstruct every target +through `W`, and publish first as Rust at `W+1`. When one slice passes, the +same preapproved window proceeds to the next. Any ambiguity, missing ACK, +parity mismatch, lag/gap, stale CAS or Trading System failure enters `BLOCKED` +for that slice and restores V1 under a newer revision; unrelated promoted slices +remain governed independently. + +#### C.4 Close With V1 Hot Fallback + +After all approved Binance/OKX slices are `RUST_PRIMARY`, Trading System reads +V2 as its normal source and V1 stays running on port `8100` as the tested hot +fallback. There is no alpha-by-alpha migration while those alphas remain down +and no V1 sunset is part of this cutover. Publish the V2 release only after the +Trading System cycle, Rust authority audit, cursor/replay continuity and an +exercised V1 fallback/return-to-V2 drill pass. DNSE remains V1-only until its separate provider gate passes. **Decision boundary:** C.0 code/release preparation and C.1 isolated deployment -are non-production-authority work. C.2 changes only explicitly named paper -consumer routes. C.3 and C.4 require a separate operator approval containing -the exact packet above. No command in this plan implicitly authorizes a restart, -authority mutation, consumer cutover, volume deletion or V1 shutdown. +are non-production-authority work. C.2 changes only the Trading System +market-data route. C.3 requires one explicit operator packet for the approved +Binance/OKX slice set. No command implicitly authorizes deleting volumes, +stopping V1 or promoting DNSE. ### Rollback diff --git a/docs/runbooks/v2-production-rust-authority-cutover.md b/docs/runbooks/v2-production-rust-authority-cutover.md index 1b11ba9..f6aa4d2 100644 --- a/docs/runbooks/v2-production-rust-authority-cutover.md +++ b/docs/runbooks/v2-production-rust-authority-cutover.md @@ -19,6 +19,7 @@ DNSE is excluded until its separate provider gate passes. - Stable project: `qdl_v2_stable_candidate`. - Initial V2 authority: `RUST_SHADOW`. - Initial venues: Binance and OKX only. +- Operator-declared active consumer: Trading System only; all alphas are down. - V1, current Redis and current provider processes are not restarted by the merge or isolated-deploy steps. @@ -191,20 +192,29 @@ docker compose \ Do not add `-v`; preserve evidence until the failed gate is understood. -## Gate 5 - Paper Consumer Canary +## Gate 5 - Trading System Dual-Read And Route Switch -Move only explicitly named manifests, in this order: +Do not start or migrate an alpha. Keep Trading System on V1 while its adapter +performs a bounded read-only comparison against V2 for the same Binance/OKX +instruments and closed-bar/event boundaries. -1. monitoring; -2. one paper alpha; -3. Trading System paper market-data adapter; -4. remaining approved paper consumers. +Require exact identity, timestamp, decimal, unit, finality and session semantics; +zero unexplained gap/duplicate; bounded freshness/lag; and successful signed +cursor replay/restart. Built-in health and authority telemetry must be green. -For each consumer, record V1 cursor/watermark, V2 warmup result, first live -cursor, restart cursor, freshness, gaps, duplicates, source/session status and -rollback route. No sandbox/live order path is included. +Then perform one controlled Trading System adapter restart with venue-aware +routing: -Rollback changes only that consumer endpoint/SDK config back to V1. +```text +Binance/OKX: V2 primary -> governed V1 fallback +DNSE: V1 only +``` + +The exact adapter config keys and restart command are populated from the current +Trading System deployment during the cutover preflight; do not invent or +hardcode them in advance. Every fallback/return transition records source, +reason, watermark and operator identity. V1 fallback is accepted only when its +freshness/session/contract checks pass; otherwise execution remains blocked. ## Gate 6 - Exact-Slice Authority Approval @@ -236,9 +246,11 @@ operator: change_ticket: ``` -The operator must explicitly approve this exact packet. +The operator must explicitly approve this exact packet. One packet may enumerate +all approved Binance/OKX slices for one maintenance window, but the controller +executes and audits each CAS independently. -The transition follows only: +Every slice follows only: ```text PYTHON_PRIMARY @@ -247,16 +259,20 @@ PYTHON_PRIMARY -> RUST_PRIMARY ``` -The old writer is fenced at `W`; Rust reconstructs every required target -through `W` and first publishes at `W+1`. Any failed gate enters `BLOCKED` -and rolls back using a newer authority revision and accepted reverse handoff. +The canary gate is bounded by accepted real events and continuity evidence, not +an arbitrary multi-day wait. The old writer is fenced at `W`; Rust reconstructs +all required targets through `W` and first publishes at `W+1`. A failed slice +enters `BLOCKED` and rolls back under a newer revision without undoing an +unrelated healthy slice. -## Gate 7 - Expand And Release +## Gate 7 - Close With V1 Hot Fallback -Hold the first primary slice for the approved window. Expand one venue/feed -slice at a time; no inherited certification. Keep V1 available until every -registered consumer has migrated and rollback has been exercised. +After approved Binance/OKX slices are `RUST_PRIMARY`, Trading System uses V2 +normally and V1 remains live at port `8100` as the tested fallback. Exercise +one V2 -> V1 -> V2 route drill with durable source-switch audit and no market +semantic mismatch. -Only then may the operator approve stable V2 public routing, a V1 sunset date, -a release PR `dev -> main`, and tag/release publication. DNSE remains disabled -until its provider-specific external gates pass. +There is no alpha migration and no V1 sunset in this cutover. Publish the V2 +release only after the Trading System cycle, authority audit, cursor/replay +continuity and fallback drill pass. DNSE remains V1-only until its provider +gate passes. From df94a51e8f61b5ef9b5c09ebe086158a2ae28fc0 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 08:27:46 +0000 Subject: [PATCH 02/21] docs(data-layer): record fast-track cutover SHA --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index b3923fd..22994e7 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5958,7 +5958,7 @@ authority wiring cannot be deferred as operational debt. preapproved Binance/OKX maintenance window. This reduces operations, not correctness: persistent authority CAS/outbox, sink fencing, W/W+1 handoff, durable audit and per-slice rollback remain mandatory. V1 stays hot on port - `8100`; DNSE stays V1-only. + `8100`; DNSE stays V1-only. Fast-track planning commit: `e8167d4`. #### C.1 Isolated Stable V2 Deployment From fb8b4f496930f25cfbee1765e41653dd39a35683 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 08:39:21 +0000 Subject: [PATCH 03/21] docs(data-layer): define production V2 consumer gates --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 35 +++++++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 22994e7..0d38e0b 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5926,12 +5926,29 @@ without mutating V1. The CLI prints identities/revisions/watermarks/digests only, never secrets. 7. Build one immutable Python/Rust image pair from the merged SHA, generate SBOM/provenance, and retain exactly one tested V1 rollback generation. +8. Publish `qdl_sdk==2.0.0` as a standalone immutable wheel with checksum, + SBOM and generated-contract digest. Trading System and execution-alpha base + images pin the same artifact; neither repository copies Data Layer service + internals or maintains an independent V2 schema parser. +9. Replace the bounded BTC-only certification catalog with a deterministic + production catalog/binding generator driven by approved venue metadata and + consumer manifests. Instrument UIDs remain stable across restart/rebuild; + arbitrary alpha symbols are resolved through `/v2/instruments`, never by + hardcoded UUIDs in consumers. Only demanded/approved Binance and OKX feeds + are acquired; disabled symbols fail readiness rather than creating data. +10. Freeze two versioned consumer classes: Trading System + `EXECUTION` grade and shared alpha runtime `ALPHA` grade. The execution + client requires authoritative/fresh/gap-free snapshots. Alpha warmup uses + final BAR snapshot/cursor/replay and the same SDK, while strategy/order + source remains untouched. V1 fallback is explicit and source-switch audited. **C.0 gates:** migration idempotency, transactional outbox replay, compacted authority recovery, stale-writer rejection, restart recovery, exact Python/Rust -parity, public V1 contract compatibility, full source/Clippy/security tests and -zero production mutation. Conclusion must be either `PASS` or `FAIL`; missing -authority wiring cannot be deferred as operational debt. +parity, deterministic multi-symbol catalog generation, SDK wheel reproducibility +and checksum verification, Trading System/alpha consumer contract tests, public +V1 compatibility, full source/Clippy/security tests and zero production +mutation. Conclusion must be either `PASS` or `FAIL`; missing authority, +catalog or SDK wiring cannot be deferred as operational debt. **C.0 implementation journal:** @@ -5951,6 +5968,18 @@ authority wiring cannot be deferred as operational debt. C.0 remains `IN_PROGRESS` until the current PR is CI-green and merged to `dev`; production authority wiring starts only on the new branch named in this phase. Preparation commit: `130da39`. +- `2026-08-20 CROSS-REPOSITORY V2 CONSUMER AUDIT RECORDED`: remote `dev` + merged the certified V2 branch at `468c951`; authority work continues on + `feat/v2-production-authority-cutover` from that merge, with the later + fast-track plan cherry-picked as `9e35b34` and `df94a51`. Trading System + currently has a V1 REST/Redis bridge and its `alpha_sdk` is primarily an + execution client; execution-alpha warmup/stream calls live in the shared + `alpha_runtime.orchestration.DataLayerGateway`. Therefore V2 is introduced + as one versioned `qdl_sdk` artifact used by both consumers. No strategy file, + signal rule or order endpoint is migrated for this data-plane change. + The audit also found the stable catalog is certification-bounded to BTC/VN + examples, so deterministic production symbol/catalog generation is a + mandatory C.0 gate before alpha consumers can be called V2-ready. - `2026-08-20 OPERATOR CUTOVER SIMPLIFICATION RECORDED`: the operator reports all alpha consumers are stopped and Trading System is the sole active consumer. Phase C therefore removes staged alpha/monitoring migrations and From ec80faff42ae5c8c46ef129d74bcd869f5b366bd Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 09:02:20 +0000 Subject: [PATCH 04/21] feat(sdk): publish standalone V2 consumer contract --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 14 + qdl/api_v2/models.py | 374 ++-------------------- qdl_sdk/README.md | 26 +- qdl_sdk/__init__.py | 6 + qdl_sdk/client.py | 106 ++++++ qdl_sdk/models.py | 354 +++++++++++++++++++- qdl_sdk/transport.py | 50 ++- scripts/build_qdl_sdk_release.py | 177 ++++++++++ tests/test_qdl_sdk_release.py | 163 ++++++++++ 9 files changed, 916 insertions(+), 354 deletions(-) create mode 100755 scripts/build_qdl_sdk_release.py create mode 100644 tests/test_qdl_sdk_release.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 0d38e0b..0d281f9 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5980,6 +5980,20 @@ catalog or SDK wiring cannot be deferred as operational debt. The audit also found the stable catalog is certification-bounded to BTC/VN examples, so deterministic production symbol/catalog generation is a mandatory C.0 gate before alpha consumers can be called V2-ready. +- `2026-08-20 C.0 SDK ARTIFACT/IDENTITY SLICE PASS`: moved every public V2 + response model into `qdl_sdk.models` and made `qdl.api_v2.models` reuse and + re-export that exact implementation. The SDK no longer imports Data Layer + service internals. Added bounded typed instrument catalog resolution by + venue/market/product/native symbol, including pagination-cycle, missing and + ambiguous-identity fail-closed behavior; consumers no longer need hardcoded + UUIDs. Added a deterministic standalone `qdl_sdk==2.0.0` wheel builder, + SHA-256 release manifest, generated-contract digest and CycloneDX SBOM. + Repeated builds produced an identical wheel digest and a network-off install + smoke imported exclusively from the installed wheel. Compile plus V1 golden, + API/SDK/stream/security/multi-venue tests passed 47/47. V1 runtime, provider sockets, + authority, Redis, Kafka, consumer routes and production data were untouched. + Production demand/catalog generation and long-running Rust authority wiring + remain the next C.0 slices; this slice alone does not authorize cutover. - `2026-08-20 OPERATOR CUTOVER SIMPLIFICATION RECORDED`: the operator reports all alpha consumers are stopped and Trading System is the sole active consumer. Phase C therefore removes staged alpha/monitoring migrations and diff --git a/qdl/api_v2/models.py b/qdl/api_v2/models.py index 9391149..ef1f9b1 100644 --- a/qdl/api_v2/models.py +++ b/qdl/api_v2/models.py @@ -1,12 +1,8 @@ from __future__ import annotations -from enum import StrEnum -from typing import Annotated, Literal - -from pydantic import BaseModel, ConfigDict, Field, model_validator +from pydantic import Field from qdl.query import ( - BarLifecycle, BarRevisionPolicy, ConsumerGrade, FeedType, @@ -14,10 +10,40 @@ RecoveryPolicy, StalePolicy, ) - - -class ClosedModel(BaseModel): - model_config = ConfigDict(extra="forbid", populate_by_name=True) +from qdl_sdk.models import ( + BarPayload, + BatchItemResponse, + BatchResponse, + BookDeltaPayload, + BookLevel, + BookSnapshotPayload, + ClosedModel, + ContractView, + DecimalValue, + FeedStatusResponse, + FundingRatePayload, + GapListResponse, + GapView, + InstrumentPageResponse, + InstrumentResponse, + InstrumentView, + MarkIndexPricePayload, + MarketDataView, + OpenInterestPayload, + ProblemDetails, + QualityView, + QuantityUnit, + QuotePayload, + ReadinessItemResponse, + ReadinessResponse, + SnapshotResponse, + SourceView, + SystemReadinessSummary, + TickerPayload, + TradeIdentityKind, + TradePayload, + WarmupResponse, +) class RequirementModel(ClosedModel): @@ -40,333 +66,3 @@ class BatchRequirementModel(ClosedModel): consumer_id: str = Field(min_length=1, max_length=200) requirements: list[RequirementModel] = Field(min_length=1, max_length=100) require_all: bool = True - - -class ProblemDetails(ClosedModel): - type: str - title: str - status: int - code: str - detail: str - request_id: str - retryable: bool - retry_after_ms: int | None = None - instrument_uid: str | None = None - quality_state: str | None = None - - -class DecimalValue(ClosedModel): - coefficient: str = Field(pattern=r"^-?(0|[1-9][0-9]*)$") - scale: int = Field(ge=-38, le=38) - source_text: str = Field(min_length=1, max_length=128) - - -class TradeIdentityKind(StrEnum): - NATIVE = "NATIVE" - DERIVED_RAW_CAPTURE = "DERIVED_RAW_CAPTURE" - - -class QuantityUnit(StrEnum): - BASE_ASSET = "BASE_ASSET" - QUOTE_ASSET = "QUOTE_ASSET" - CONTRACT = "CONTRACT" - SHARE = "SHARE" - - -class SourceView(ClosedModel): - venue: str = Field(min_length=1, max_length=40) - provider: str = Field(min_length=1, max_length=80) - source_id: str = Field(min_length=1, max_length=120) - source_role: Literal["PRIMARY", "SECONDARY", "REFERENCE", "BACKFILL"] - authoritative: bool - - -class QualityView(ClosedModel): - state: Literal[ - "DISABLED", "STARTING", "CONNECTING", "SUBSCRIBING", "SYNCING", - "LIVE", "DEGRADED", "GAPPED", "RESYNCING", "STALE", "OFFLINE", - "HALTED", "MARKET_CLOSED", - ] - freshness_ms: int = Field(ge=0) - gap_open: bool - complete: bool - execution_eligible: bool - policy_id: str = Field(min_length=1, max_length=200) - flags: list[str] - - -class ContractView(ClosedModel): - schema_digest: str = Field(pattern=r"^[0-9a-f]{64}$") - contract_version: str = Field(min_length=1, max_length=40) - normalizer_version: str = Field(min_length=1, max_length=80) - adapter_version: str = Field(min_length=1, max_length=80) - instrument_catalog_revision: int = Field(ge=1) - source_policy_revision: int = Field(ge=1) - authority_revision: int = Field(ge=1) - config_revision: int = Field(ge=1) - correlation_id: str = Field(min_length=1, max_length=200) - - -class TradePayload(ClosedModel): - feed: Literal[FeedType.TRADE] = FeedType.TRADE - native_trade_id: str = Field(min_length=1, max_length=200) - price: DecimalValue - quantity: DecimalValue - quantity_unit: QuantityUnit - aggressor_side: Literal["BUY", "SELL", "UNKNOWN"] - identity_kind: TradeIdentityKind - is_block_trade: bool = False - is_buyer_maker: bool = False - - -class QuotePayload(ClosedModel): - feed: Literal[FeedType.QUOTE] = FeedType.QUOTE - bid_price: DecimalValue - bid_quantity: DecimalValue - ask_price: DecimalValue - ask_quantity: DecimalValue - quantity_unit: QuantityUnit - level: int = Field(default=1, ge=1) - - -class BarPayload(ClosedModel): - feed: Literal[FeedType.BAR] = FeedType.BAR - interval: str = Field(min_length=1, max_length=20) - open_time_ns: int = Field(gt=0) - close_time_ns: int = Field(gt=0) - open: DecimalValue - high: DecimalValue - low: DecimalValue - close: DecimalValue - volume: DecimalValue - volume_unit: QuantityUnit - base_volume: DecimalValue | None = None - quote_volume: DecimalValue | None = None - contract_volume: DecimalValue | None = None - trade_count: int = Field(default=0, ge=0) - lifecycle: BarLifecycle - revision: int = Field(ge=0) - origin: Literal["VENUE_NATIVE", "AGGREGATED", "BACKFILLED", "RECONCILED"] - supersedes_event_id: str | None = None - - @model_validator(mode="after") - def validate_lifecycle(self): - if self.lifecycle is BarLifecycle.UNSPECIFIED: - raise ValueError("bar lifecycle cannot be UNSPECIFIED") - if self.lifecycle is BarLifecycle.REVISED and not self.supersedes_event_id: - raise ValueError("revised bar must identify the superseded event") - if self.close_time_ns <= self.open_time_ns: - raise ValueError("bar close time must be after open time") - return self - - -class BookLevel(ClosedModel): - side: Literal["BID", "ASK"] - price: DecimalValue - quantity: DecimalValue - quantity_unit: QuantityUnit - order_count: int = Field(default=0, ge=0) - - -class BookSnapshotPayload(ClosedModel): - feed: Literal[FeedType.BOOK_SNAPSHOT] = FeedType.BOOK_SNAPSHOT - native_sequence: str = Field(min_length=1, max_length=200) - checksum: str | None = Field(default=None, max_length=200) - levels: list[BookLevel] - depth: int = Field(ge=1) - - -class BookDeltaPayload(ClosedModel): - feed: Literal[FeedType.BOOK_DELTA] = FeedType.BOOK_DELTA - native_sequence_start: str = Field(min_length=1, max_length=200) - native_sequence_end: str = Field(min_length=1, max_length=200) - snapshot_sequence: str = Field(min_length=1, max_length=200) - checksum: str | None = Field(default=None, max_length=200) - updates: list[BookLevel] - reset: bool = False - - -class FundingRatePayload(ClosedModel): - feed: Literal[FeedType.FUNDING_RATE] = FeedType.FUNDING_RATE - rate: DecimalValue - funding_time_ns: int = Field(gt=0) - next_funding_time_ns: int | None = Field(default=None, gt=0) - - -class OpenInterestPayload(ClosedModel): - feed: Literal[FeedType.OPEN_INTEREST] = FeedType.OPEN_INTEREST - quantity: DecimalValue - quantity_unit: QuantityUnit - notional: DecimalValue | None = None - - -class MarkIndexPricePayload(ClosedModel): - feed: Literal[FeedType.MARK_INDEX_PRICE] = FeedType.MARK_INDEX_PRICE - mark_price: DecimalValue - index_price: DecimalValue - - -class TickerPayload(ClosedModel): - feed: Literal[FeedType.TICKER] = FeedType.TICKER - last_price: DecimalValue - last_quantity: DecimalValue | None = None - open_24h: DecimalValue | None = None - high_24h: DecimalValue | None = None - low_24h: DecimalValue | None = None - volume_24h: DecimalValue | None = None - last_quantity_unit: QuantityUnit | None = None - volume_24h_unit: QuantityUnit | None = None - - @model_validator(mode="after") - def quantity_units_match_optional_values(self): - if (self.last_quantity is None) != (self.last_quantity_unit is None): - raise ValueError("ticker last quantity and unit must be present together") - if (self.volume_24h is None) != (self.volume_24h_unit is None): - raise ValueError("ticker 24h volume and unit must be present together") - return self - - -MarketPayload = Annotated[ - TradePayload | QuotePayload | BarPayload | BookSnapshotPayload | BookDeltaPayload - | FundingRatePayload | OpenInterestPayload | MarkIndexPricePayload | TickerPayload, - Field(discriminator="feed"), -] - - -class MarketDataView(ClosedModel): - instrument_uid: str - instrument_id: str - instrument_revision: int = Field(ge=1) - feed: FeedType - interval: str | None - observed_at_ns: int = Field(gt=0) - revision: int = Field(ge=0) - payload: MarketPayload - source: SourceView - quality: QualityView - contract: ContractView - cursor: str | None = None - snapshot_id: str | None = None - watermark_offset: int = Field(default=0, ge=0) - - @model_validator(mode="after") - def feed_matches_payload(self): - if self.feed is FeedType.UNSPECIFIED or self.payload.feed is not self.feed: - raise ValueError("market-data envelope feed does not match its payload") - if self.feed is FeedType.BAR: - if not self.interval or self.payload.interval != self.interval: - raise ValueError("bar envelope and payload interval must match") - if self.payload.revision != self.revision: - raise ValueError("bar envelope and payload revision must match") - elif self.interval is not None: - raise ValueError("interval is valid only for bar market data") - return self - - -class SnapshotResponse(ClosedModel): - contract_schema: str = Field("qdl.marketdata.snapshot.v2", alias="schema") - request_id: str - data: MarketDataView - - -class InstrumentView(ClosedModel): - instrument_uid: str - instrument_id: str - venue: str - market: str - product_type: str - canonical_symbol: str - metadata_revision: int = Field(ge=1) - asset_class: str - native_symbol: str - status: str - - -class InstrumentPageResponse(ClosedModel): - contract_schema: str = Field("qdl.instruments.page.v2", alias="schema") - items: list[InstrumentView] - next_cursor: str | None = None - - -class InstrumentResponse(InstrumentView): - contract_schema: str = Field("qdl.instrument.v2", alias="schema") - - -class WarmupResponse(ClosedModel): - contract_schema: str = Field("qdl.marketdata.warmup.v2", alias="schema") - request_id: str - snapshot_id: str = Field(min_length=1) - data_as_of_ns: int = Field(gt=0) - stream_cursor: str = Field(min_length=1) - watermark_offset: int = Field(ge=0) - coverage: str - count: int - data: list[MarketDataView] - - -class BatchItemResponse(ClosedModel): - instrument_uid: str - status: str - data: WarmupResponse | None = None - problem: ProblemDetails | None = None - - -class BatchResponse(ClosedModel): - contract_schema: str = Field("qdl.marketdata.batch.v2", alias="schema") - request_id: str - partial: bool - success_count: int - error_count: int - results: list[BatchItemResponse] - - @model_validator(mode="after") - def counts_match(self): - if self.success_count + self.error_count != len(self.results): - raise ValueError("batch counts do not match results") - if self.partial != (self.error_count > 0): - raise ValueError("batch partial flag does not match errors") - return self - - -class ReadinessItemResponse(ClosedModel): - instrument_uid: str - status: str - quality: QualityView | None = None - problem: ProblemDetails | None = None - - -class ReadinessResponse(ClosedModel): - contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") - request_id: str - ready: bool - authority: str = "V1" - results: list[ReadinessItemResponse] - - -class FeedStatusResponse(ClosedModel): - contract_schema: str = Field("qdl.feed-status.v2", alias="schema") - instrument_uid: str - feed: FeedType - quality: QualityView - - -class GapView(ClosedModel): - gap_id: str - instrument_uid: str - feed: FeedType - source_id: str - expected_sequence: str - observed_sequence: str - detected_at_ns: int - - -class GapListResponse(ClosedModel): - contract_schema: str = Field("qdl.data-quality.gaps.v2", alias="schema") - items: list[GapView] - - -class SystemReadinessSummary(ClosedModel): - contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") - status: str - authority: str - v2_consumer_activation: str diff --git a/qdl_sdk/README.md b/qdl_sdk/README.md index 23e51bd..2ddc2d1 100644 --- a/qdl_sdk/README.md +++ b/qdl_sdk/README.md @@ -32,11 +32,18 @@ atomically restored the local state associated with that checkpoint. ```python from qdl_sdk import DataRequirement, Feed, Grade +instrument = await client.resolve_instrument( + venue="BINANCE", + market="USDM", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + consumer_grade=Grade.EXECUTION, +) requirement = DataRequirement( - instrument_uid="a953e16e-7138-5562-b5e8-c337a44d0b65", + instrument_uid=instrument.instrument_uid, feed=Feed.TRADE, consumer_grade=Grade.EXECUTION, - source_policy_id="execution_binance_usdm_v1", + source_policy_id="crypto_primary_v2", max_freshness_ms=1000, ) @@ -68,3 +75,18 @@ async with client.warmup_then_stream( handoff metadata is a hard continuity error. - The SDK never parses cursor internals and never silently accepts stale, gapped, partial or non-authoritative execution data. + +## Immutable consumer artifact + +Build the standalone artifact with: + +```bash +python scripts/build_qdl_sdk_release.py --output-dir dist/qdl-sdk +``` + +The output contains a reproducible `qdl_sdk-2.0.0-py3-none-any.whl`, a release +manifest with the wheel/source/generated-contract SHA-256 digests, and a +CycloneDX SBOM. The wheel contains only the public SDK plus generated Protobuf +contracts; it does not package `qdl.api_v2`, runtime adapters, provider code or +other Data Layer service internals. Trading System and the shared alpha runtime +must pin the same verified wheel digest. diff --git a/qdl_sdk/__init__.py b/qdl_sdk/__init__.py index 9d2a8e0..0c88b41 100644 --- a/qdl_sdk/__init__.py +++ b/qdl_sdk/__init__.py @@ -17,6 +17,9 @@ Feed, GapPolicy, Grade, + InstrumentPageResponse, + InstrumentResponse, + InstrumentView, MarketDataView, QuantityUnit, RecoveryPolicy, @@ -45,6 +48,9 @@ "GapPolicy", "Grade", "GrpcStreamTransport", + "InstrumentPageResponse", + "InstrumentResponse", + "InstrumentView", "MemoryCursorStore", "MarketDataView", "QuantityUnit", diff --git a/qdl_sdk/client.py b/qdl_sdk/client.py index 944155b..ce34d73 100644 --- a/qdl_sdk/client.py +++ b/qdl_sdk/client.py @@ -13,6 +13,9 @@ DataRequirement, Feed, Grade, + InstrumentPageResponse, + InstrumentResponse, + InstrumentView, SnapshotResponse, StreamEvent, WarmupResponse, @@ -22,6 +25,17 @@ class QueryTransport(Protocol): async def warmup(self, requirement: DataRequirement, *, consumer_id: str) -> dict: ... async def snapshot(self, requirement: DataRequirement, *, consumer_id: str) -> dict: ... + async def instruments( + self, + *, + consumer_id: str, + consumer_grade: Grade, + cursor: str | None, + limit: int, + ) -> dict: ... + async def instrument( + self, identity: str, *, consumer_id: str, consumer_grade: Grade + ) -> dict: ... async def close(self) -> None: ... @@ -297,6 +311,98 @@ async def warmup(self, requirement: DataRequirement) -> WarmupResponse: self._record_query("/v2/market-data/warmup", response) return response + async def instrument( + self, identity: str, *, consumer_grade: Grade + ) -> InstrumentResponse: + payload = await self.query_transport.instrument( + identity, + consumer_id=self.consumer_id, + consumer_grade=consumer_grade, + ) + try: + return InstrumentResponse.model_validate(payload) + except ValidationError as error: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", + "instrument response violates the typed V2 contract", + ) from error + + async def resolve_instrument( + self, + *, + venue: str, + product_type: str, + native_symbol: str, + consumer_grade: Grade, + market: str | None = None, + page_limit: int = 500, + max_pages: int = 100, + ) -> InstrumentView: + if not isinstance(consumer_grade, Grade): + raise TypeError("consumer_grade must use the typed SDK enum") + if not 1 <= page_limit <= 500 or not 1 <= max_pages <= 100: + raise ValueError("instrument resolver page bounds are invalid") + expected = { + "venue": venue.strip().upper(), + "product_type": product_type.strip().upper(), + "native_symbol": native_symbol.strip().upper(), + "market": market.strip().upper() if market is not None else None, + } + if not all(expected[key] for key in ("venue", "product_type", "native_symbol")): + raise ValueError("venue, product_type and native_symbol are required") + cursor: str | None = None + matches: list[InstrumentView] = [] + seen_cursors: set[str] = set() + for _ in range(max_pages): + payload = await self.query_transport.instruments( + consumer_id=self.consumer_id, + consumer_grade=consumer_grade, + cursor=cursor, + limit=page_limit, + ) + try: + page = InstrumentPageResponse.model_validate(payload) + except ValidationError as error: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", + "instrument page violates the typed V2 contract", + ) from error + for item in page.items: + if ( + item.venue.upper() == expected["venue"] + and item.product_type.upper() == expected["product_type"] + and item.native_symbol.upper() == expected["native_symbol"] + and ( + expected["market"] is None + or item.market.upper() == expected["market"] + ) + ): + matches.append(item) + if page.next_cursor is None: + break + if page.next_cursor in seen_cursors: + raise ContinuityError( + "CONFLICT", "instrument catalog returned a cursor cycle" + ) + seen_cursors.add(page.next_cursor) + cursor = page.next_cursor + else: + raise ContinuityError( + "PARTIAL_RESULT", "instrument catalog exceeded bounded pagination" + ) + active = [item for item in matches if item.status.upper() == "ACTIVE"] + if len(active) == 1: + return active[0] + if not active: + raise DataLayerError( + "INSTRUMENT_NOT_FOUND", + "no active V2 instrument matches the venue/product/native symbol", + retryable=False, + ) + raise ContinuityError( + "CONFLICT", "instrument identity is ambiguous; specify market" + ) + @asynccontextmanager async def warmup_then_stream( self, diff --git a/qdl_sdk/models.py b/qdl_sdk/models.py index f0fd5a1..130dd11 100644 --- a/qdl_sdk/models.py +++ b/qdl_sdk/models.py @@ -2,15 +2,10 @@ from dataclasses import dataclass from enum import StrEnum -from typing import Any - -from qdl.api_v2.models import ( - MarketDataView, - QuantityUnit, - SnapshotResponse, - TradeIdentityKind, - WarmupResponse, -) +from typing import Annotated, Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, model_validator + from qdl.query.v2 import query_pb2 @@ -56,6 +51,345 @@ class BarRevisionPolicy(StrEnum): EMIT_REVISIONS = "EMIT_REVISIONS" +class BarLifecycle(StrEnum): + IN_PROGRESS = "IN_PROGRESS" + FINAL = "FINAL" + REVISED = "REVISED" + CANCELLED = "CANCELLED" + + +class ClosedModel(BaseModel): + model_config = ConfigDict(extra="forbid", populate_by_name=True) + + +class ProblemDetails(ClosedModel): + type: str + title: str + status: int + code: str + detail: str + request_id: str + retryable: bool + retry_after_ms: int | None = None + instrument_uid: str | None = None + quality_state: str | None = None + + +class DecimalValue(ClosedModel): + coefficient: str = Field(pattern=r"^-?(0|[1-9][0-9]*)$") + scale: int = Field(ge=-38, le=38) + source_text: str = Field(min_length=1, max_length=128) + + +class TradeIdentityKind(StrEnum): + NATIVE = "NATIVE" + DERIVED_RAW_CAPTURE = "DERIVED_RAW_CAPTURE" + + +class QuantityUnit(StrEnum): + BASE_ASSET = "BASE_ASSET" + QUOTE_ASSET = "QUOTE_ASSET" + CONTRACT = "CONTRACT" + SHARE = "SHARE" + + +class SourceView(ClosedModel): + venue: str = Field(min_length=1, max_length=40) + provider: str = Field(min_length=1, max_length=80) + source_id: str = Field(min_length=1, max_length=120) + source_role: Literal["PRIMARY", "SECONDARY", "REFERENCE", "BACKFILL"] + authoritative: bool + + +class QualityView(ClosedModel): + state: Literal[ + "DISABLED", "STARTING", "CONNECTING", "SUBSCRIBING", "SYNCING", + "LIVE", "DEGRADED", "GAPPED", "RESYNCING", "STALE", "OFFLINE", + "HALTED", "MARKET_CLOSED", + ] + freshness_ms: int = Field(ge=0) + gap_open: bool + complete: bool + execution_eligible: bool + policy_id: str = Field(min_length=1, max_length=200) + flags: list[str] + + +class ContractView(ClosedModel): + schema_digest: str = Field(pattern=r"^[0-9a-f]{64}$") + contract_version: str = Field(min_length=1, max_length=40) + normalizer_version: str = Field(min_length=1, max_length=80) + adapter_version: str = Field(min_length=1, max_length=80) + instrument_catalog_revision: int = Field(ge=1) + source_policy_revision: int = Field(ge=1) + authority_revision: int = Field(ge=1) + config_revision: int = Field(ge=1) + correlation_id: str = Field(min_length=1, max_length=200) + + +class TradePayload(ClosedModel): + feed: Literal[Feed.TRADE] = Feed.TRADE + native_trade_id: str = Field(min_length=1, max_length=200) + price: DecimalValue + quantity: DecimalValue + quantity_unit: QuantityUnit + aggressor_side: Literal["BUY", "SELL", "UNKNOWN"] + identity_kind: TradeIdentityKind + is_block_trade: bool = False + is_buyer_maker: bool = False + + +class QuotePayload(ClosedModel): + feed: Literal[Feed.QUOTE] = Feed.QUOTE + bid_price: DecimalValue + bid_quantity: DecimalValue + ask_price: DecimalValue + ask_quantity: DecimalValue + quantity_unit: QuantityUnit + level: int = Field(default=1, ge=1) + + +class BarPayload(ClosedModel): + feed: Literal[Feed.BAR] = Feed.BAR + interval: str = Field(min_length=1, max_length=20) + open_time_ns: int = Field(gt=0) + close_time_ns: int = Field(gt=0) + open: DecimalValue + high: DecimalValue + low: DecimalValue + close: DecimalValue + volume: DecimalValue + volume_unit: QuantityUnit + base_volume: DecimalValue | None = None + quote_volume: DecimalValue | None = None + contract_volume: DecimalValue | None = None + trade_count: int = Field(default=0, ge=0) + lifecycle: BarLifecycle + revision: int = Field(ge=0) + origin: Literal["VENUE_NATIVE", "AGGREGATED", "BACKFILLED", "RECONCILED"] + supersedes_event_id: str | None = None + + @model_validator(mode="after") + def validate_lifecycle(self): + if self.lifecycle is BarLifecycle.REVISED and not self.supersedes_event_id: + raise ValueError("revised bar must identify the superseded event") + if self.close_time_ns <= self.open_time_ns: + raise ValueError("bar close time must be after open time") + return self + + +class BookLevel(ClosedModel): + side: Literal["BID", "ASK"] + price: DecimalValue + quantity: DecimalValue + quantity_unit: QuantityUnit + order_count: int = Field(default=0, ge=0) + + +class BookSnapshotPayload(ClosedModel): + feed: Literal[Feed.BOOK_SNAPSHOT] = Feed.BOOK_SNAPSHOT + native_sequence: str = Field(min_length=1, max_length=200) + checksum: str | None = Field(default=None, max_length=200) + levels: list[BookLevel] + depth: int = Field(ge=1) + + +class BookDeltaPayload(ClosedModel): + feed: Literal[Feed.BOOK_DELTA] = Feed.BOOK_DELTA + native_sequence_start: str = Field(min_length=1, max_length=200) + native_sequence_end: str = Field(min_length=1, max_length=200) + snapshot_sequence: str = Field(min_length=1, max_length=200) + checksum: str | None = Field(default=None, max_length=200) + updates: list[BookLevel] + reset: bool = False + + +class FundingRatePayload(ClosedModel): + feed: Literal[Feed.FUNDING_RATE] = Feed.FUNDING_RATE + rate: DecimalValue + funding_time_ns: int = Field(gt=0) + next_funding_time_ns: int | None = Field(default=None, gt=0) + + +class OpenInterestPayload(ClosedModel): + feed: Literal[Feed.OPEN_INTEREST] = Feed.OPEN_INTEREST + quantity: DecimalValue + quantity_unit: QuantityUnit + notional: DecimalValue | None = None + + +class MarkIndexPricePayload(ClosedModel): + feed: Literal[Feed.MARK_INDEX_PRICE] = Feed.MARK_INDEX_PRICE + mark_price: DecimalValue + index_price: DecimalValue + + +class TickerPayload(ClosedModel): + feed: Literal[Feed.TICKER] = Feed.TICKER + last_price: DecimalValue + last_quantity: DecimalValue | None = None + open_24h: DecimalValue | None = None + high_24h: DecimalValue | None = None + low_24h: DecimalValue | None = None + volume_24h: DecimalValue | None = None + last_quantity_unit: QuantityUnit | None = None + volume_24h_unit: QuantityUnit | None = None + + @model_validator(mode="after") + def quantity_units_match_optional_values(self): + if (self.last_quantity is None) != (self.last_quantity_unit is None): + raise ValueError("ticker last quantity and unit must be present together") + if (self.volume_24h is None) != (self.volume_24h_unit is None): + raise ValueError("ticker 24h volume and unit must be present together") + return self + + +MarketPayload = Annotated[ + TradePayload | QuotePayload | BarPayload | BookSnapshotPayload | BookDeltaPayload + | FundingRatePayload | OpenInterestPayload | MarkIndexPricePayload | TickerPayload, + Field(discriminator="feed"), +] + + +class MarketDataView(ClosedModel): + instrument_uid: str + instrument_id: str + instrument_revision: int = Field(ge=1) + feed: Feed + interval: str | None + observed_at_ns: int = Field(gt=0) + revision: int = Field(ge=0) + payload: MarketPayload + source: SourceView + quality: QualityView + contract: ContractView + cursor: str | None = None + snapshot_id: str | None = None + watermark_offset: int = Field(default=0, ge=0) + + @model_validator(mode="after") + def feed_matches_payload(self): + if self.payload.feed is not self.feed: + raise ValueError("market-data envelope feed does not match its payload") + if self.feed is Feed.BAR: + if not self.interval or self.payload.interval != self.interval: + raise ValueError("bar envelope and payload interval must match") + if self.payload.revision != self.revision: + raise ValueError("bar envelope and payload revision must match") + elif self.interval is not None: + raise ValueError("interval is valid only for bar market data") + return self + + +class SnapshotResponse(ClosedModel): + contract_schema: str = Field("qdl.marketdata.snapshot.v2", alias="schema") + request_id: str + data: MarketDataView + + +class InstrumentView(ClosedModel): + instrument_uid: str + instrument_id: str + venue: str + market: str + product_type: str + canonical_symbol: str + metadata_revision: int = Field(ge=1) + asset_class: str + native_symbol: str + status: str + + +class InstrumentPageResponse(ClosedModel): + contract_schema: str = Field("qdl.instruments.page.v2", alias="schema") + items: list[InstrumentView] + next_cursor: str | None = None + + +class InstrumentResponse(InstrumentView): + contract_schema: str = Field("qdl.instrument.v2", alias="schema") + + +class WarmupResponse(ClosedModel): + contract_schema: str = Field("qdl.marketdata.warmup.v2", alias="schema") + request_id: str + snapshot_id: str = Field(min_length=1) + data_as_of_ns: int = Field(gt=0) + stream_cursor: str = Field(min_length=1) + watermark_offset: int = Field(ge=0) + coverage: str + count: int + data: list[MarketDataView] + + +class BatchItemResponse(ClosedModel): + instrument_uid: str + status: str + data: WarmupResponse | None = None + problem: ProblemDetails | None = None + + +class BatchResponse(ClosedModel): + contract_schema: str = Field("qdl.marketdata.batch.v2", alias="schema") + request_id: str + partial: bool + success_count: int + error_count: int + results: list[BatchItemResponse] + + @model_validator(mode="after") + def counts_match(self): + if self.success_count + self.error_count != len(self.results): + raise ValueError("batch counts do not match results") + if self.partial != (self.error_count > 0): + raise ValueError("batch partial flag does not match errors") + return self + + +class ReadinessItemResponse(ClosedModel): + instrument_uid: str + status: str + quality: QualityView | None = None + problem: ProblemDetails | None = None + + +class ReadinessResponse(ClosedModel): + contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") + request_id: str + ready: bool + authority: str = "V1" + results: list[ReadinessItemResponse] + + +class FeedStatusResponse(ClosedModel): + contract_schema: str = Field("qdl.feed-status.v2", alias="schema") + instrument_uid: str + feed: Feed + quality: QualityView + + +class GapView(ClosedModel): + gap_id: str + instrument_uid: str + feed: Feed + source_id: str + expected_sequence: str + observed_sequence: str + detected_at_ns: int + + +class GapListResponse(ClosedModel): + contract_schema: str = Field("qdl.data-quality.gaps.v2", alias="schema") + items: list[GapView] + + +class SystemReadinessSummary(ClosedModel): + contract_schema: str = Field("qdl.system-readiness.v2", alias="schema") + status: str + authority: str + v2_consumer_activation: str + + @dataclass(frozen=True) class DataRequirement: instrument_uid: str @@ -157,7 +491,7 @@ def __post_init__(self) -> None: class ControlEvent: code: str detail: str - snapshot: WarmupResponse | None = None + snapshot: WarmupResponse | dict[str, Any] | None = None def __post_init__(self) -> None: if not self.code.strip() or not self.detail.strip(): diff --git a/qdl_sdk/transport.py b/qdl_sdk/transport.py index 911fc6b..ff8601a 100644 --- a/qdl_sdk/transport.py +++ b/qdl_sdk/transport.py @@ -53,31 +53,75 @@ async def snapshot(self, requirement: DataRequirement, *, consumer_id: str) -> d ) return self._decode(response) + async def instruments( + self, + *, + consumer_id: str, + consumer_grade: Grade, + cursor: str | None = None, + limit: int = 500, + ) -> dict: + headers = await self._identity_headers(consumer_grade, consumer_id) + params: dict[str, str | int] = {"limit": limit} + if cursor is not None: + params["cursor"] = cursor + response = await self._client.get( + "/v2/instruments", params=params, headers=headers + ) + return self._decode(response) + + async def instrument( + self, + identity: str, + *, + consumer_id: str, + consumer_grade: Grade, + ) -> dict: + if not identity.strip(): + raise ValueError("instrument identity is required") + headers = await self._identity_headers(consumer_grade, consumer_id) + response = await self._client.get( + f"/v2/instruments/{identity}", headers=headers + ) + return self._decode(response) + async def close(self) -> None: if self._owns_client: await self._client.aclose() async def _headers(self, requirement: DataRequirement, consumer_id: str) -> dict[str, str]: + return await self._identity_headers(requirement.consumer_grade, consumer_id) + + async def _identity_headers( + self, consumer_grade: Grade, consumer_id: str + ) -> dict[str, str]: if self._credential_provider is None: raise DataLayerError( "UNAUTHENTICATED", "V2 REST transport requires a workload credential provider", retryable=False, ) + if not isinstance(consumer_grade, Grade): + raise TypeError("consumer_grade must use the typed SDK enum") token = await self._credential_provider.get_token() return { "Authorization": f"Bearer {token}", "X-QDL-Consumer-ID": consumer_id, - "X-QDL-Purpose": self._purpose(requirement), + "X-QDL-Purpose": self._purpose(consumer_grade), } @staticmethod - def _purpose(requirement: DataRequirement) -> str: + def _purpose(consumer_grade: Grade | DataRequirement) -> str: + grade = ( + consumer_grade.consumer_grade + if isinstance(consumer_grade, DataRequirement) + else consumer_grade + ) return { Grade.EXECUTION: "INTERNAL_EXECUTION", Grade.ALPHA: "INTERNAL_ALPHA", Grade.RESEARCH: "INTERNAL_RESEARCH", - }[requirement.consumer_grade] + }[grade] @staticmethod def _decode(response: httpx.Response) -> dict: diff --git a/scripts/build_qdl_sdk_release.py b/scripts/build_qdl_sdk_release.py new file mode 100755 index 0000000..615fa86 --- /dev/null +++ b/scripts/build_qdl_sdk_release.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import base64 +import csv +import hashlib +import io +import json +import os +from pathlib import Path +import zipfile + + +ROOT = Path(__file__).resolve().parents[1] +NAME = "qdl-sdk" +NORMALIZED_NAME = "qdl_sdk" +VERSION = "2.0.0" +DIST_INFO = f"{NORMALIZED_NAME}-{VERSION}.dist-info" +DEPENDENCIES = ( + "grpcio>=1.70.0,<2.0.0", + "httpx>=0.28.0,<1.0.0", + "protobuf>=6.31.1,<7.0.0", + "pydantic>=2.0.0,<3.0.0", +) +FIXED_TIMESTAMP = (2020, 1, 1, 0, 0, 0) + + +def sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def record_digest(value: bytes) -> str: + digest = base64.urlsafe_b64encode(hashlib.sha256(value).digest()).rstrip(b"=") + return f"sha256={digest.decode('ascii')}" + + +def digest_paths(paths: list[Path]) -> str: + digest = hashlib.sha256() + for path in sorted(paths): + relative = path.relative_to(ROOT).as_posix().encode() + content = path.read_bytes() + digest.update(len(relative).to_bytes(4, "big")) + digest.update(relative) + digest.update(len(content).to_bytes(8, "big")) + digest.update(content) + return digest.hexdigest() + + +def release_files() -> dict[str, bytes]: + files: dict[str, bytes] = {} + for path in sorted((ROOT / "qdl_sdk").glob("*.py")): + files[path.relative_to(ROOT).as_posix()] = path.read_bytes() + readme = ROOT / "qdl_sdk/README.md" + files[readme.relative_to(ROOT).as_posix()] = readme.read_bytes() + for path in sorted((ROOT / "generated/python/qdl").rglob("*.py")): + archive = path.relative_to(ROOT / "generated/python").as_posix() + files[archive] = path.read_bytes() + forbidden = (b"qdl.api_v2", b"qdl.runtime", b"app.") + sdk_sources = b"\n".join( + value + for name, value in files.items() + if name.startswith("qdl_sdk/") and name.endswith(".py") + ) + found = [token.decode() for token in forbidden if token in sdk_sources] + if found: + raise RuntimeError(f"SDK imports service internals: {found}") + if not any(name == "qdl/query/v2/query_pb2.py" for name in files): + raise RuntimeError("generated query contract is missing from SDK artifact") + return files + + +def metadata() -> bytes: + requires = "".join(f"Requires-Dist: {dependency}\n" for dependency in DEPENDENCIES) + return ( + "Metadata-Version: 2.3\n" + f"Name: {NAME}\n" + f"Version: {VERSION}\n" + "Summary: Typed provider-neutral client for Quant Data Layer V2\n" + "Author-email: BobbyAxerol \n" + "License: MIT\n" + "Requires-Python: >=3.10\n" + f"{requires}" + "Description-Content-Type: text/markdown\n\n" + "# Quant Data Layer SDK\n" + ).encode() + + +def write_zip_entry(archive: zipfile.ZipFile, name: str, content: bytes) -> None: + info = zipfile.ZipInfo(name, FIXED_TIMESTAMP) + info.compress_type = zipfile.ZIP_DEFLATED + info.external_attr = 0o100644 << 16 + archive.writestr(info, content, compress_type=zipfile.ZIP_DEFLATED, compresslevel=9) + + +def build_wheel(output_dir: Path) -> dict[str, object]: + output_dir.mkdir(parents=True, exist_ok=True) + wheel_name = f"{NORMALIZED_NAME}-{VERSION}-py3-none-any.whl" + wheel_path = output_dir / wheel_name + files = release_files() + license_content = (ROOT / "LICENSE").read_bytes() + files[f"{DIST_INFO}/METADATA"] = metadata() + files[f"{DIST_INFO}/WHEEL"] = ( + "Wheel-Version: 1.0\n" + "Generator: qdl-sdk-release/1\n" + "Root-Is-Purelib: true\n" + "Tag: py3-none-any\n" + ).encode() + files[f"{DIST_INFO}/top_level.txt"] = b"qdl\nqdl_sdk\n" + files[f"{DIST_INFO}/licenses/LICENSE"] = license_content + + rows: list[list[str]] = [] + for name in sorted(files): + value = files[name] + rows.append([name, record_digest(value), str(len(value))]) + record_name = f"{DIST_INFO}/RECORD" + rows.append([record_name, "", ""]) + stream = io.StringIO(newline="") + csv.writer(stream, lineterminator="\n").writerows(rows) + files[record_name] = stream.getvalue().encode() + + temporary = wheel_path.with_suffix(".whl.tmp") + with zipfile.ZipFile(temporary, "w") as archive: + for name in sorted(files): + write_zip_entry(archive, name, files[name]) + os.replace(temporary, wheel_path) + + contract_paths = sorted((ROOT / "contracts/proto").rglob("*.proto")) + source_paths = sorted((ROOT / "qdl_sdk").glob("*.py")) + manifest = { + "schema": "qdl.sdk.release.v1", + "name": NAME, + "version": VERSION, + "wheel": wheel_path.name, + "wheel_sha256": sha256_bytes(wheel_path.read_bytes()), + "sdk_source_digest": digest_paths(source_paths), + "generated_contract_digest": digest_paths(contract_paths), + "python_requires": ">=3.10", + "dependencies": list(DEPENDENCIES), + "contains_service_internals": False, + "reproducible_timestamp": "2020-01-01T00:00:00Z", + } + manifest_path = output_dir / f"{NORMALIZED_NAME}-{VERSION}.release.json" + manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n") + sbom = { + "bomFormat": "CycloneDX", + "specVersion": "1.5", + "serialNumber": f"urn:uuid:{hashlib.sha256(wheel_path.read_bytes()).hexdigest()[:32]}", + "version": 1, + "metadata": { + "component": { + "type": "library", + "name": NAME, + "version": VERSION, + "hashes": [{"alg": "SHA-256", "content": manifest["wheel_sha256"]}], + } + }, + "components": [ + {"type": "library", "name": item.split(">=")[0], "version": item} + for item in DEPENDENCIES + ], + } + sbom_path = output_dir / f"{NORMALIZED_NAME}-{VERSION}.cdx.json" + sbom_path.write_text(json.dumps(sbom, indent=2, sort_keys=True) + "\n") + return {**manifest, "manifest": str(manifest_path), "sbom": str(sbom_path)} + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + print(json.dumps(build_wheel(args.output_dir.resolve()), indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_qdl_sdk_release.py b/tests/test_qdl_sdk_release.py new file mode 100644 index 0000000..5d5031d --- /dev/null +++ b/tests/test_qdl_sdk_release.py @@ -0,0 +1,163 @@ +from __future__ import annotations + +import csv +import hashlib +import io +import json +from pathlib import Path +import tempfile +import unittest +import zipfile + +from qdl_sdk import AsyncDataLayerClient, Grade +from qdl_sdk.errors import ContinuityError, DataLayerError +from scripts.build_qdl_sdk_release import build_wheel + + +class QdlSdkReleaseTests(unittest.TestCase): + def test_wheel_is_reproducible_self_contained_and_recorded(self): + with tempfile.TemporaryDirectory() as first, tempfile.TemporaryDirectory() as second: + first_result = build_wheel(Path(first)) + second_result = build_wheel(Path(second)) + self.assertEqual(first_result["wheel_sha256"], second_result["wheel_sha256"]) + wheel = Path(first) / str(first_result["wheel"]) + self.assertEqual(hashlib.sha256(wheel.read_bytes()).hexdigest(), first_result["wheel_sha256"]) + with zipfile.ZipFile(wheel) as archive: + names = set(archive.namelist()) + self.assertIn("qdl_sdk/__init__.py", names) + self.assertIn("qdl/query/v2/query_pb2.py", names) + self.assertIn("qdl/marketdata/v2/market_data_pb2.py", names) + self.assertFalse(any(name.startswith("qdl/api_v2/") for name in names)) + record_name = "qdl_sdk-2.0.0.dist-info/RECORD" + rows = list(csv.reader(io.StringIO(archive.read(record_name).decode()))) + self.assertEqual({row[0] for row in rows}, names) + release = json.loads(Path(first_result["manifest"]).read_text()) + self.assertFalse(release["contains_service_internals"]) + self.assertEqual(len(release["generated_contract_digest"]), 64) + sbom = json.loads(Path(first_result["sbom"]).read_text()) + self.assertEqual(sbom["bomFormat"], "CycloneDX") + + +class _CatalogTransport: + def __init__(self, pages): + self.pages = pages + + async def instruments(self, *, consumer_id, consumer_grade, cursor, limit): + del consumer_id, consumer_grade, limit + return self.pages[cursor] + + async def close(self): + return None + + +class _UnusedStreamTransport: + async def close(self): + return None + + +class QdlSdkInstrumentResolverTests(unittest.IsolatedAsyncioTestCase): + @staticmethod + def _item(uid, *, market, native_symbol="BTCUSDT"): + return { + "instrument_uid": uid, + "instrument_id": f"BINANCE.{market}.PERPETUAL.BTC-USDT", + "venue": "BINANCE", + "market": market, + "product_type": "PERPETUAL", + "canonical_symbol": "BTC-USDT", + "metadata_revision": 7, + "asset_class": "CRYPTO_DERIVATIVE", + "native_symbol": native_symbol, + "status": "ACTIVE", + } + + def _client(self, pages): + return AsyncDataLayerClient( + query_transport=_CatalogTransport(pages), + stream_transport=_UnusedStreamTransport(), + consumer_id="consumer-alpha-v2", + ) + + async def test_resolves_exact_active_identity_across_bounded_pages(self): + pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [self._item("uid-spot", market="SPOT", native_symbol="ETHUSDT")], + "next_cursor": "page-2", + }, + "page-2": { + "schema": "qdl.instruments.page.v2", + "items": [self._item("uid-usdm", market="USDM")], + "next_cursor": None, + }, + } + result = await self._client(pages).resolve_instrument( + venue="binance", + product_type="perpetual", + native_symbol="btcusdt", + market="usdm", + consumer_grade=Grade.ALPHA, + ) + self.assertEqual(result.instrument_uid, "uid-usdm") + + async def test_missing_and_ambiguous_identity_fail_closed(self): + missing_pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [], + "next_cursor": None, + } + } + with self.assertRaises(DataLayerError) as missing: + await self._client(missing_pages).resolve_instrument( + venue="BINANCE", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + consumer_grade=Grade.EXECUTION, + ) + self.assertEqual(missing.exception.code, "INSTRUMENT_NOT_FOUND") + + ambiguous_pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [ + self._item("uid-usdm", market="USDM"), + self._item("uid-coinm", market="COINM"), + ], + "next_cursor": None, + } + } + with self.assertRaises(ContinuityError) as ambiguous: + await self._client(ambiguous_pages).resolve_instrument( + venue="BINANCE", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + consumer_grade=Grade.EXECUTION, + ) + self.assertEqual(ambiguous.exception.code, "CONFLICT") + + async def test_catalog_cursor_cycle_fails_closed(self): + pages = { + None: { + "schema": "qdl.instruments.page.v2", + "items": [], + "next_cursor": "cycle", + }, + "cycle": { + "schema": "qdl.instruments.page.v2", + "items": [], + "next_cursor": "cycle", + }, + } + with self.assertRaises(ContinuityError) as cycle: + await self._client(pages).resolve_instrument( + venue="OKX", + product_type="PERPETUAL", + native_symbol="BTC-USDT-SWAP", + consumer_grade=Grade.ALPHA, + ) + self.assertEqual(cycle.exception.code, "CONFLICT") + + +if __name__ == "__main__": + unittest.main() From 364b7a55f16dabc2d6b52a9ccca9edab89d2004e Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 09:07:43 +0000 Subject: [PATCH 05/21] feat(catalog): generate production V2 bindings from venue metadata --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 18 + qdl/adapters/binance_usdm.py | 16 +- qdl/runtime/production_catalog.py | 453 ++++++++++++++++++++++ scripts/build_production_catalog.py | 70 ++++ tests/test_production_catalog.py | 191 +++++++++ 5 files changed, 745 insertions(+), 3 deletions(-) create mode 100644 qdl/runtime/production_catalog.py create mode 100755 scripts/build_production_catalog.py create mode 100644 tests/test_production_catalog.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 0d281f9..dd05dcb 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5994,6 +5994,24 @@ catalog or SDK wiring cannot be deferred as operational debt. authority, Redis, Kafka, consumer routes and production data were untouched. Production demand/catalog generation and long-running Rust authority wiring remain the next C.0 slices; this slice alone does not authorize cutover. +- `2026-08-20 C.0 PRODUCTION CATALOG SLICE PASS`: added a strict + `qdl.v2.production-demand.v1` manifest and deterministic source/acquisition + catalog generator. It composes the existing authoritative Binance + `exchangeInfo` and OKX V5 `/public/instruments` parsers, preserves exact + price tick/quantity step/contract multiplier, derives stable UUIDv5 identity + from the approved canonical instrument ID, de-duplicates consumer demand and + fails closed on conflicting policies, missing/inactive metadata, ambiguous + identity or uncertified feeds/intervals. Binance canonical identity now uses + provider base/quote metadata (`ETH-USDT`) and includes the explicit contract + code for dated futures rather than treating native `ETHUSDT` as canonical. + Current production BAR acquisition is deliberately bounded to certified 1m; + higher alpha intervals must be resampled from final 1m bars or remain on + explicit V1 capability fallback until independently certified. Generated + source/acquisition YAML is reloaded through the runtime validators before it + is accepted, and provenance records metadata-capture hashes with + `fabricated_metadata=false`. Compile plus production catalog, identity, + Binance adapter and multi-venue contract tests passed 27/27. No real-provider + call or runtime/authority/consumer mutation occurred. - `2026-08-20 OPERATOR CUTOVER SIMPLIFICATION RECORDED`: the operator reports all alpha consumers are stopped and Trading System is the sole active consumer. Phase C therefore removes staged alpha/monitoring migrations and diff --git a/qdl/adapters/binance_usdm.py b/qdl/adapters/binance_usdm.py index 15c33f6..35550ea 100644 --- a/qdl/adapters/binance_usdm.py +++ b/qdl/adapters/binance_usdm.py @@ -72,11 +72,21 @@ def parse_exchange_info(payload: Mapping[str, Any], *, valid_from_ns: int) -> Bi if not native_symbol: raise ValueError("Binance exchangeInfo contains an empty symbol") product_type = ProductType.PERPETUAL if contract_type == "PERPETUAL" else ProductType.FUTURE + base_asset = str(item.get("baseAsset") or "").upper() + quote_asset = str(item.get("quoteAsset") or "").upper() + if not base_asset or not quote_asset: + raise ValueError("Binance exchangeInfo is missing base/quote asset identity") + canonical_symbol = f"{base_asset}-{quote_asset}" + if product_type is ProductType.FUTURE: + _, separator, contract_code = native_symbol.rpartition("_") + if not separator or not contract_code.isdigit(): + raise ValueError("Binance dated future symbol has no contract code") + canonical_symbol = f"{canonical_symbol}-{contract_code}" identity = InstrumentIdentity.create( venue="BINANCE", market="USDM", product_type=product_type, - canonical_symbol=native_symbol, + canonical_symbol=canonical_symbol, ) expiry_ms = int(item.get("deliveryDate") or 0) record = InstrumentRecord( @@ -84,8 +94,8 @@ def parse_exchange_info(payload: Mapping[str, Any], *, valid_from_ns: int) -> Bi metadata_revision=1, asset_class=AssetClass.DERIVATIVE, native_symbol=native_symbol, - base_asset=str(item.get("baseAsset") or "").upper(), - quote_asset=str(item.get("quoteAsset") or "").upper(), + base_asset=base_asset, + quote_asset=quote_asset, settlement_asset=str(item.get("marginAsset") or "").upper(), price_tick=CanonicalDecimal.from_text(_filter_value(item, "PRICE_FILTER", "tickSize")), quantity_step=CanonicalDecimal.from_text(_filter_value(item, "LOT_SIZE", "stepSize")), diff --git a/qdl/runtime/production_catalog.py b/qdl/runtime/production_catalog.py new file mode 100644 index 0000000..5cf3ad6 --- /dev/null +++ b/qdl/runtime/production_catalog.py @@ -0,0 +1,453 @@ +from __future__ import annotations + +from dataclasses import dataclass, replace +import hashlib +import json +from pathlib import Path +import re +from typing import Any, Iterable, Mapping + +import yaml + +from qdl.adapters.binance_usdm import BinanceDiscovery, parse_exchange_info +from qdl.adapters.okx.instruments import parse_public_instrument +from qdl.domain.instrument import InstrumentRecord, InstrumentStatus, ProductType +from qdl.query import ConsumerGrade, FeedType +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import StableAcquisitionPlan + + +_DEMAND_SCHEMA = "qdl.v2.production-demand.v1" +_SOURCE_SCHEMA = "qdl.v2.stable-source-bindings.v1" +_ACQUISITION_SCHEMA = "qdl.v2.stable-acquisition-bindings.v1" +_SUPPORTED_MARKETS = { + ("BINANCE", "USDM", "PERPETUAL"), + ("OKX", "SWAP", "PERPETUAL"), + ("OKX", "SPOT", "SPOT"), +} +_SUPPORTED_FEEDS = {FeedType.TRADE, FeedType.QUOTE, FeedType.BAR} + + +@dataclass(frozen=True, slots=True, order=True) +class ProductionDemand: + consumer_id: str + consumer_grade: ConsumerGrade + venue: str + market: str + product_type: str + native_symbol: str + feed: FeedType + interval: str | None + source_policy_id: str + + @property + def requirement_key(self) -> tuple[str, str, str, str, FeedType, str | None]: + return ( + self.venue, + self.market, + self.product_type, + self.native_symbol, + self.feed, + self.interval, + ) + + +@dataclass(frozen=True, slots=True) +class ProductionDemandManifest: + revision: int + demands: tuple[ProductionDemand, ...] + source_paths: tuple[str, ...] + + @classmethod + def load_many(cls, paths: Iterable[str | Path]) -> "ProductionDemandManifest": + demands: list[ProductionDemand] = [] + revisions: list[int] = [] + normalized_paths: list[str] = [] + for raw_path in paths: + path = Path(raw_path).resolve() + payload = yaml.safe_load(path.read_text(encoding="utf-8")) + if not isinstance(payload, dict) or set(payload) != { + "schema", "revision", "consumers" + }: + raise ValueError("production demand manifest fields are invalid") + if payload["schema"] != _DEMAND_SCHEMA or int(payload["revision"]) < 1: + raise ValueError("production demand manifest schema/revision is invalid") + consumers = payload["consumers"] + if not isinstance(consumers, list) or not 1 <= len(consumers) <= 10_000: + raise ValueError("production demand manifest needs bounded consumers") + revisions.append(int(payload["revision"])) + normalized_paths.append(str(path)) + for consumer in consumers: + if not isinstance(consumer, dict) or set(consumer) != { + "consumer_id", "consumer_grade", "requirements" + }: + raise ValueError("production demand consumer fields are invalid") + consumer_id = str(consumer["consumer_id"]).strip() + grade = ConsumerGrade(str(consumer["consumer_grade"]).upper()) + requirements = consumer["requirements"] + if not consumer_id or not isinstance(requirements, list) or not requirements: + raise ValueError("production demand consumer is empty") + for requirement in requirements: + demands.append(cls._requirement(consumer_id, grade, requirement)) + if not demands: + raise ValueError("production demand set cannot be empty") + deduped: dict[tuple[str, str, str, str, FeedType, str | None], ProductionDemand] = {} + consumers_by_key: dict[tuple[str, str, str, str, FeedType, str | None], set[str]] = {} + for item in sorted(demands): + key = item.requirement_key + current = deduped.get(key) + if current is not None and current.source_policy_id != item.source_policy_id: + raise ValueError("one feed requirement cannot use conflicting source policies") + deduped.setdefault(key, item) + consumers_by_key.setdefault(key, set()).add(item.consumer_id) + # Consumer IDs are audit inputs but do not alter a canonical source binding. + del consumers_by_key + return cls( + revision=max(revisions), + demands=tuple(sorted(deduped.values())), + source_paths=tuple(sorted(normalized_paths)), + ) + + @staticmethod + def _requirement( + consumer_id: str, + grade: ConsumerGrade, + raw: Any, + ) -> ProductionDemand: + if not isinstance(raw, dict) or set(raw) != { + "venue", "market", "product_type", "native_symbol", + "feed", "interval", "source_policy_id", + }: + raise ValueError("production demand requirement fields are invalid") + venue = str(raw["venue"]).strip().upper() + market = str(raw["market"]).strip().upper() + product = str(raw["product_type"]).strip().upper() + native_symbol = str(raw["native_symbol"]).strip().upper() + source_policy = str(raw["source_policy_id"]).strip() + feed = FeedType(str(raw["feed"]).strip().upper()) + interval = str(raw["interval"]).strip() if raw["interval"] is not None else None + if (venue, market, product) not in _SUPPORTED_MARKETS: + raise ValueError("production demand market/product is not certified") + if feed not in _SUPPORTED_FEEDS: + raise ValueError("production demand feed is not certified") + if feed is FeedType.BAR and interval != "1m": + raise ValueError("production V2 BAR acquisition is currently certified for 1m") + if feed is not FeedType.BAR and interval is not None: + raise ValueError("interval is valid only for BAR demand") + if not native_symbol or not source_policy: + raise ValueError("production demand identity/source policy is incomplete") + return ProductionDemand( + consumer_id=consumer_id, + consumer_grade=grade, + venue=venue, + market=market, + product_type=product, + native_symbol=native_symbol, + feed=feed, + interval=interval, + source_policy_id=source_policy, + ) + + +@dataclass(frozen=True, slots=True) +class ProductionCatalogBundle: + source_catalog: dict[str, Any] + acquisition_plan: dict[str, Any] + provenance: dict[str, Any] + + def write(self, output_dir: str | Path) -> dict[str, str]: + target = Path(output_dir) + target.mkdir(parents=True, exist_ok=True) + source_path = target / "production-source-bindings.yaml" + acquisition_path = target / "production-acquisition-bindings.yaml" + provenance_path = target / "production-catalog-provenance.json" + source_path.write_text(yaml.safe_dump(self.source_catalog, sort_keys=False)) + acquisition_path.write_text(yaml.safe_dump(self.acquisition_plan, sort_keys=False)) + provenance_path.write_text( + json.dumps(self.provenance, indent=2, sort_keys=True) + "\n" + ) + catalog = StableSourceCatalog.load(source_path) + StableAcquisitionPlan.load(acquisition_path, catalog=catalog) + return { + "source_catalog": str(source_path), + "acquisition_plan": str(acquisition_path), + "provenance": str(provenance_path), + } + + +class ProductionCatalogBuilder: + def __init__( + self, + *, + catalog_revision: int, + source_policy_revision: int, + authority_revision: int, + canonical_stream: str = "md.canonical.v2", + raw_topic: str = "md.raw.stable.v1", + quarantine_topic: str = "md.quarantine.stable.v1", + ) -> None: + if min(catalog_revision, source_policy_revision, authority_revision) < 1: + raise ValueError("production catalog revisions must be positive") + topics = (canonical_stream, raw_topic, quarantine_topic) + if any(not item.strip() for item in topics) or len(set(topics)) != 3: + raise ValueError("production catalog topics must be unique") + self.catalog_revision = catalog_revision + self.source_policy_revision = source_policy_revision + self.authority_revision = authority_revision + self.canonical_stream = canonical_stream + self.raw_topic = raw_topic + self.quarantine_topic = quarantine_topic + + def build( + self, + *, + demand: ProductionDemandManifest, + binance_usdm: BinanceDiscovery | None, + okx_rows: Iterable[Mapping[str, str]], + previous_catalog: StableSourceCatalog | None = None, + metadata_provenance: Mapping[str, str] | None = None, + ) -> ProductionCatalogBundle: + metadata = self._metadata(binance_usdm, okx_rows) + previous = self._previous_records(previous_catalog) + selected: dict[str, InstrumentRecord] = {} + bindings: list[dict[str, Any]] = [] + acquisitions: list[dict[str, Any]] = [] + for item in demand.demands: + key = (item.venue, item.market, item.native_symbol) + try: + discovered = metadata[key] + except KeyError as error: + raise ValueError(f"authoritative metadata missing demanded instrument: {key}") from error + if discovered.identity.product_type.value != item.product_type: + raise ValueError("demanded product type differs from authoritative metadata") + if discovered.status is not InstrumentStatus.ACTIVE: + raise ValueError("demanded instrument is not active") + record = self._revisioned(discovered, previous.get(discovered.instrument_id)) + selected[record.instrument_id] = record + binding_id = self._binding_id(item) + bindings.append(self._source_binding(binding_id, item, record)) + acquisitions.append(self._acquisition(binding_id, item)) + source = { + "schema": _SOURCE_SCHEMA, + "canonical_stream": self.canonical_stream, + "catalog_revision": self.catalog_revision, + "source_policy_revision": self.source_policy_revision, + "authority_revision": self.authority_revision, + "instruments": [ + self._instrument(record) for record in sorted(selected.values(), key=lambda value: value.instrument_id) + ], + "bindings": sorted(bindings, key=lambda value: value["binding_id"]), + } + acquisition = { + "schema": _ACQUISITION_SCHEMA, + "revision": demand.revision, + "topics": { + "raw": self.raw_topic, + "canonical": self.canonical_stream, + "quarantine": self.quarantine_topic, + }, + "bindings": sorted(acquisitions, key=lambda value: value["binding_id"]), + } + encoded_source = yaml.safe_dump(source, sort_keys=True).encode() + encoded_acquisition = yaml.safe_dump(acquisition, sort_keys=True).encode() + provenance = { + "schema": "qdl.v2.production-catalog-provenance.v1", + "catalog_revision": self.catalog_revision, + "demand_revision": demand.revision, + "demand_sources": list(demand.source_paths), + "instrument_count": len(selected), + "binding_count": len(bindings), + "instrument_ids": sorted(selected), + "source_catalog_sha256": hashlib.sha256(encoded_source).hexdigest(), + "acquisition_plan_sha256": hashlib.sha256(encoded_acquisition).hexdigest(), + "metadata": dict(sorted((metadata_provenance or {}).items())), + "fabricated_metadata": False, + } + return ProductionCatalogBundle(source, acquisition, provenance) + + @staticmethod + def _metadata( + binance_usdm: BinanceDiscovery | None, + okx_rows: Iterable[Mapping[str, str]], + ) -> dict[tuple[str, str, str], InstrumentRecord]: + values: list[InstrumentRecord] = [] + if binance_usdm is not None: + values.extend(binance_usdm.records) + for raw in okx_rows: + record, _ = parse_public_instrument( + raw, metadata_revision=1, valid_from_ns=0 + ) + values.append(record) + result: dict[tuple[str, str, str], InstrumentRecord] = {} + for record in values: + key = ( + record.identity.venue, + record.identity.market, + record.native_symbol, + ) + if key in result: + raise ValueError(f"duplicate authoritative instrument metadata: {key}") + result[key] = record + return result + + @staticmethod + def _previous_records( + catalog: StableSourceCatalog | None, + ) -> dict[str, InstrumentRecord]: + if catalog is None: + return {} + return { + binding.instrument.instrument_id: binding.instrument + for binding in catalog.bindings + } + + @classmethod + def _revisioned( + cls, current: InstrumentRecord, previous: InstrumentRecord | None + ) -> InstrumentRecord: + if previous is None: + return replace(current, metadata_revision=1) + previous_payload = cls._instrument(previous) | {"metadata_revision": 0} + current_payload = cls._instrument(current) | {"metadata_revision": 0} + revision = ( + previous.metadata_revision + if previous_payload == current_payload + else previous.metadata_revision + 1 + ) + return replace(current, metadata_revision=revision) + + @staticmethod + def _instrument(record: InstrumentRecord) -> dict[str, Any]: + identity = record.identity + return { + "instrument_uid": identity.instrument_uid, + "instrument_id": identity.instrument_id, + "metadata_revision": record.metadata_revision, + "venue": identity.venue, + "market": identity.market, + "product_type": identity.product_type.value, + "canonical_symbol": identity.canonical_symbol, + "native_symbol": record.native_symbol, + "asset_class": record.asset_class.value, + "base_asset": record.base_asset, + "quote_asset": record.quote_asset, + "settlement_asset": record.settlement_asset, + "price_tick": record.price_tick.source_text, + "quantity_step": record.quantity_step.source_text, + "contract_multiplier": record.contract_multiplier.source_text, + "session_calendar_id": record.session_calendar_id, + "attributes": dict(sorted(record.attributes.items())), + } + + @staticmethod + def _binding_id(item: ProductionDemand) -> str: + symbol = re.sub(r"[^a-z0-9]+", "-", item.native_symbol.lower()).strip("-") + suffix = f"-{item.interval}" if item.interval else "" + return f"{item.venue.lower()}-{item.market.lower()}-{symbol}-{item.feed.value.lower()}{suffix}" + + def _source_binding( + self, + binding_id: str, + item: ProductionDemand, + record: InstrumentRecord, + ) -> dict[str, Any]: + if item.feed is FeedType.TRADE: + stale_after_ms = 15_000 + elif item.feed is FeedType.QUOTE: + stale_after_ms = 5_000 + else: + stale_after_ms = 180_000 + adapter = ( + "binance-rest/2.0.0" + if item.venue == "BINANCE" and item.feed is FeedType.BAR + else "binance-usdm/2.0.0" + if item.venue == "BINANCE" + else "okx-v5/2.0.0" + ) + compatibility = "NONE" + if item.venue == "BINANCE" and item.feed is FeedType.TRADE: + compatibility = "BINANCE_TRADE_MARKET_AND_GENERIC" + elif item.venue == "BINANCE" and item.feed is FeedType.BAR: + compatibility = "BINANCE_BAR_GENERIC" + return { + "binding_id": binding_id, + "instrument_uid": record.instrument_uid, + "feed": item.feed.value, + "interval": item.interval, + "source": { + "provider": f"{item.venue}_DIRECT", + "source_id": f"{binding_id}-primary-v2", + "source_role": "PRIMARY", + "source_policy_id": item.source_policy_id, + "authoritative": True, + "adapter_version": adapter, + "normalizer_version": "qdl-rust-core/2.0.0", + }, + "quality": { + "stale_after_ms": stale_after_ms, + "require_final_bar": item.feed is FeedType.BAR, + "continuous_calendar": True, + }, + "v1_compatibility": compatibility, + } + + @staticmethod + def _acquisition(binding_id: str, item: ProductionDemand) -> dict[str, Any]: + if item.venue == "BINANCE": + if item.feed is FeedType.TRADE: + mode, kind, channel, sequence = ( + "RUST_NATIVE", "binance_usdm_trade", + f"{item.native_symbol.lower()}@trade", "MONOTONIC", + ) + elif item.feed is FeedType.QUOTE: + mode, kind, channel, sequence = ( + "RUST_NATIVE", "binance_usdm_bbo", + f"{item.native_symbol.lower()}@bookTicker", "MONOTONIC", + ) + else: + mode, kind, channel, sequence = ( + "PYTHON_REST", "binance_usdm_rest_bar", + f"rest-klines/{item.interval}", "NONE", + ) + websocket = ( + "wss://fstream.binance.com/public/stream" + if mode == "RUST_NATIVE" else None + ) + business = None + else: + if item.feed is FeedType.TRADE: + mode, kind, channel, sequence = "RUST_NATIVE", "okx_trade", "trades", "MONOTONIC" + elif item.feed is FeedType.QUOTE: + mode, kind, channel, sequence = "RUST_NATIVE", "okx_bbo", "bbo-tbt", "NONE" + else: + mode, kind, channel, sequence = "PYTHON_REST", "okx_bar", "candle1m", "NONE" + websocket = "wss://ws.okx.com:8443/ws/v5/public" if mode == "RUST_NATIVE" else None + business = "wss://ws.okx.com:8443/ws/v5/business" if mode == "RUST_NATIVE" else None + return { + "binding_id": binding_id, + "mode": mode, + "runtime": item.venue, + "provider_kind": kind, + "native_channel": channel, + "sequence_policy": sequence, + "websocket_url": websocket, + "business_websocket_url": business, + } + + +def load_binance_exchange_info(path: str | Path) -> BinanceDiscovery: + payload = json.loads(Path(path).read_text(encoding="utf-8")) + if not isinstance(payload, dict): + raise ValueError("Binance exchangeInfo capture must be an object") + return parse_exchange_info(payload, valid_from_ns=0) + + +def load_okx_instruments(path: str | Path) -> list[Mapping[str, str]]: + payload = json.loads(Path(path).read_text(encoding="utf-8")) + if isinstance(payload, dict): + if str(payload.get("code")) != "0" or not isinstance(payload.get("data"), list): + raise ValueError("OKX instruments capture is not a successful V5 response") + payload = payload["data"] + if not isinstance(payload, list) or not all(isinstance(item, dict) for item in payload): + raise ValueError("OKX instruments capture must contain a data list") + return payload diff --git a/scripts/build_production_catalog.py b/scripts/build_production_catalog.py new file mode 100755 index 0000000..36f7a6a --- /dev/null +++ b/scripts/build_production_catalog.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path + +from qdl.runtime.production_catalog import ( + ProductionCatalogBuilder, + ProductionDemandManifest, + load_binance_exchange_info, + load_okx_instruments, +) +from qdl.runtime.stable_catalog import StableSourceCatalog + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--demand", action="append", required=True, type=Path) + parser.add_argument("--binance-usdm-exchange-info", type=Path) + parser.add_argument("--okx-instruments", type=Path) + parser.add_argument("--previous-catalog", type=Path) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--catalog-revision", type=int, required=True) + parser.add_argument("--source-policy-revision", type=int, required=True) + parser.add_argument("--authority-revision", type=int, required=True) + args = parser.parse_args() + demand = ProductionDemandManifest.load_many(args.demand) + needs_binance = any(item.venue == "BINANCE" for item in demand.demands) + needs_okx = any(item.venue == "OKX" for item in demand.demands) + if needs_binance != (args.binance_usdm_exchange_info is not None): + raise SystemExit("Binance metadata capture must be supplied exactly when demanded") + if needs_okx != (args.okx_instruments is not None): + raise SystemExit("OKX metadata capture must be supplied exactly when demanded") + binance = ( + load_binance_exchange_info(args.binance_usdm_exchange_info) + if args.binance_usdm_exchange_info else None + ) + okx = load_okx_instruments(args.okx_instruments) if args.okx_instruments else [] + previous = ( + StableSourceCatalog.load(args.previous_catalog) + if args.previous_catalog else None + ) + metadata = {} + if args.binance_usdm_exchange_info: + metadata["binance_exchange_info_sha256"] = sha256(args.binance_usdm_exchange_info) + if args.okx_instruments: + metadata["okx_instruments_sha256"] = sha256(args.okx_instruments) + bundle = ProductionCatalogBuilder( + catalog_revision=args.catalog_revision, + source_policy_revision=args.source_policy_revision, + authority_revision=args.authority_revision, + ).build( + demand=demand, + binance_usdm=binance, + okx_rows=okx, + previous_catalog=previous, + metadata_provenance=metadata, + ) + print(json.dumps(bundle.write(args.output_dir), indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_production_catalog.py b/tests/test_production_catalog.py new file mode 100644 index 0000000..0705998 --- /dev/null +++ b/tests/test_production_catalog.py @@ -0,0 +1,191 @@ +from __future__ import annotations + +import json +from pathlib import Path +import tempfile +import unittest + +from qdl.adapters.binance_usdm import parse_exchange_info +from qdl.runtime.production_catalog import ( + ProductionCatalogBuilder, + ProductionDemandManifest, +) +from qdl.runtime.stable_catalog import StableSourceCatalog + + +BINANCE = { + "serverTime": 1000, + "symbols": [ + { + "symbol": "ETHUSDT", + "status": "TRADING", + "contractType": "PERPETUAL", + "baseAsset": "ETH", + "quoteAsset": "USDT", + "marginAsset": "USDT", + "deliveryDate": 0, + "filters": [ + {"filterType": "PRICE_FILTER", "tickSize": "0.01"}, + {"filterType": "LOT_SIZE", "stepSize": "0.001"}, + ], + }, + { + "symbol": "BTCUSDT_260925", + "status": "TRADING", + "contractType": "CURRENT_QUARTER", + "baseAsset": "BTC", + "quoteAsset": "USDT", + "marginAsset": "USDT", + "deliveryDate": 1789948800000, + "filters": [ + {"filterType": "PRICE_FILTER", "tickSize": "0.1"}, + {"filterType": "LOT_SIZE", "stepSize": "0.001"}, + ], + }, + ], +} +OKX = [ + { + "instType": "SWAP", + "instId": "ETH-USDT-SWAP", + "instFamily": "ETH-USDT", + "baseCcy": "ETH", + "quoteCcy": "USDT", + "settleCcy": "USDT", + "tickSz": "0.01", + "lotSz": "1", + "ctVal": "0.1", + "ctMult": "1", + "state": "live", + } +] + + +class ProductionCatalogTests(unittest.TestCase): + def _manifest(self, root: Path, *, conflicting=False) -> ProductionDemandManifest: + policy = "other_policy" if conflicting else "crypto_primary_v2" + payload = { + "schema": "qdl.v2.production-demand.v1", + "revision": 5, + "consumers": [ + { + "consumer_id": "trading-system.execution.v2", + "consumer_grade": "EXECUTION", + "requirements": [ + { + "venue": "BINANCE", "market": "USDM", + "product_type": "PERPETUAL", "native_symbol": "ETHUSDT", + "feed": "TRADE", "interval": None, + "source_policy_id": "crypto_primary_v2", + }, + { + "venue": "OKX", "market": "SWAP", + "product_type": "PERPETUAL", "native_symbol": "ETH-USDT-SWAP", + "feed": "BAR", "interval": "1m", + "source_policy_id": "crypto_primary_v2", + }, + ], + }, + { + "consumer_id": "alpha.shared.v2", + "consumer_grade": "ALPHA", + "requirements": [ + { + "venue": "BINANCE", "market": "USDM", + "product_type": "PERPETUAL", "native_symbol": "ETHUSDT", + "feed": "TRADE", "interval": None, + "source_policy_id": policy, + } + ], + }, + ], + } + path = root / ("conflict.yaml" if conflicting else "demand.yaml") + import yaml + path.write_text(yaml.safe_dump(payload, sort_keys=False)) + return ProductionDemandManifest.load_many([path]) + + def test_binance_identity_uses_canonical_pair_and_dated_contract_code(self): + discovery = parse_exchange_info(BINANCE, valid_from_ns=0) + by_native = {item.native_symbol: item for item in discovery.records} + self.assertEqual( + by_native["ETHUSDT"].instrument_id, + "BINANCE.USDM.PERPETUAL.ETH-USDT", + ) + self.assertEqual( + by_native["BTCUSDT_260925"].instrument_id, + "BINANCE.USDM.FUTURE.BTC-USDT-260925", + ) + + def test_generator_is_deterministic_validated_and_metadata_authentic(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + demand = self._manifest(root) + builder = ProductionCatalogBuilder( + catalog_revision=8, + source_policy_revision=3, + authority_revision=11, + ) + first = builder.build( + demand=demand, + binance_usdm=parse_exchange_info(BINANCE, valid_from_ns=99), + okx_rows=OKX, + metadata_provenance={"capture": "a" * 64}, + ) + second = builder.build( + demand=demand, + binance_usdm=parse_exchange_info(BINANCE, valid_from_ns=123), + okx_rows=list(reversed(OKX)), + metadata_provenance={"capture": "a" * 64}, + ) + self.assertEqual(first.source_catalog, second.source_catalog) + self.assertEqual(first.acquisition_plan, second.acquisition_plan) + self.assertEqual(first.provenance["fabricated_metadata"], False) + self.assertEqual(first.provenance["instrument_count"], 2) + paths = first.write(root / "out") + catalog = StableSourceCatalog.load(paths["source_catalog"]) + self.assertEqual(len(catalog.bindings), 2) + self.assertEqual( + {item.instrument.instrument_id for item in catalog.bindings}, + { + "BINANCE.USDM.PERPETUAL.ETH-USDT", + "OKX.SWAP.PERPETUAL.ETH-USDT", + }, + ) + self.assertEqual( + json.loads(Path(paths["provenance"]).read_text())["binding_count"], 2 + ) + + def test_conflicts_missing_metadata_and_uncertified_interval_fail_closed(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + with self.assertRaisesRegex(ValueError, "conflicting source policies"): + self._manifest(root, conflicting=True) + demand = self._manifest(root) + with self.assertRaisesRegex(ValueError, "metadata missing"): + ProductionCatalogBuilder( + catalog_revision=1, + source_policy_revision=1, + authority_revision=1, + ).build(demand=demand, binance_usdm=None, okx_rows=[]) + payload = { + "schema": "qdl.v2.production-demand.v1", + "revision": 1, + "consumers": [{ + "consumer_id": "alpha", "consumer_grade": "ALPHA", + "requirements": [{ + "venue": "OKX", "market": "SWAP", "product_type": "PERPETUAL", + "native_symbol": "ETH-USDT-SWAP", "feed": "BAR", + "interval": "15m", "source_policy_id": "crypto_primary_v2", + }], + }], + } + import yaml + path = root / "bad-interval.yaml" + path.write_text(yaml.safe_dump(payload)) + with self.assertRaisesRegex(ValueError, "currently certified for 1m"): + ProductionDemandManifest.load_many([path]) + + +if __name__ == "__main__": + unittest.main() From 2f6cd10f7620098675913a3546e247a7f038e3ea Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 09:33:34 +0000 Subject: [PATCH 06/21] feat(authority): persist V2 authority transitions --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 19 + Dockerfile.phase8-rust | 1 + .../0009_production_authority_outbox.sql | 230 ++++++++++ poetry.lock | 75 +++- pyproject.toml | 1 + qdl/control/authority_outbox.py | 399 ++++++++++++++++++ rust/qdl-venue-core/src/authority.rs | 218 +++++++++- scripts/phase92_migration_smoke.sh | 31 +- scripts/run_authority_outbox_dispatcher.py | 62 +++ .../phase9/authority-control-primary.json | 72 ++++ tests/test_authority_outbox.py | 229 ++++++++++ 11 files changed, 1332 insertions(+), 5 deletions(-) create mode 100644 migrations/postgres/0009_production_authority_outbox.sql create mode 100644 qdl/control/authority_outbox.py create mode 100755 scripts/run_authority_outbox_dispatcher.py create mode 100644 tests/fixtures/phase9/authority-control-primary.json create mode 100644 tests/test_authority_outbox.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index dd05dcb..1c377fd 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6012,6 +6012,25 @@ catalog or SDK wiring cannot be deferred as operational debt. `fabricated_metadata=false`. Compile plus production catalog, identity, Binance adapter and multi-venue contract tests passed 27/27. No real-provider call or runtime/authority/consumer mutation occurred. +- `2026-08-20 C.0 TRANSACTIONAL AUTHORITY OUTBOX SLICE PASS`: added PostgreSQL + migration `0009_production_authority_outbox.sql`, which writes one immutable + authority-control outbox row in the same transaction as every Phase 9 CAS + transition. Bounded claim/ACK/retry operations bind worker ownership, recover + stale claims and never mutate event identity or payload. Added the Python + outbox dispatcher and idempotent Kafka publisher for the compacted authority + topic, plus a canonical `qdl.authority-control-event.v1` serializer that + validates exact Phase 9.2 checkpoint/handoff digests before exposing a + writable authority record. Rust now decodes that Python fixture, rejects + altered identity/conflicting duplicate/stale transition, remains fenced after + restart until every target watermark is restored, accepts only exact W/W+1 + handoff, and supports a newer-revision rollback to Python. Disposable + PostgreSQL migration smoke proved four ordered revisions, payload immutability, + bounded claim/ACK and scoped cleanup; no production database was touched. + Python authority/outbox/migration regressions passed 38/38. The complete Rust workspace + passed 66 tests, `cargo fmt --check` and strict `clippy -D warnings`. The + long-running transactional Rust consume-transform-produce bridge, independent + durable target-watermark restoration and operator CLI are still required C.0 + work; this slice does not authorize runtime authority or consumer cutover. - `2026-08-20 OPERATOR CUTOVER SIMPLIFICATION RECORDED`: the operator reports all alpha consumers are stopped and Trading System is the sole active consumer. Phase C therefore removes staged alpha/monitoring migrations and diff --git a/Dockerfile.phase8-rust b/Dockerfile.phase8-rust index 25b8bad..94cd756 100644 --- a/Dockerfile.phase8-rust +++ b/Dockerfile.phase8-rust @@ -14,6 +14,7 @@ COPY generated/rust ./generated/rust # Rust unit/parity tests compile against immutable provider and contract oracles. COPY contracts/golden ./contracts/golden COPY tests/fixtures/phase2 ./tests/fixtures/phase2 +COPY tests/fixtures/phase9 ./tests/fixtures/phase9 RUN cargo build --release --locked \ --bin qdl-kafka-smoke \ --bin qdl-authority-rehearsal \ diff --git a/migrations/postgres/0009_production_authority_outbox.sql b/migrations/postgres/0009_production_authority_outbox.sql new file mode 100644 index 0000000..4c00cf7 --- /dev/null +++ b/migrations/postgres/0009_production_authority_outbox.sql @@ -0,0 +1,230 @@ +BEGIN; + +CREATE EXTENSION IF NOT EXISTS pgcrypto; + +CREATE TABLE IF NOT EXISTS qdl_authority_event_outbox ( + event_id UUID PRIMARY KEY, + transition_id UUID NOT NULL UNIQUE + REFERENCES qdl_authority_transition_audit(transition_id), + slice_id TEXT NOT NULL REFERENCES qdl_authority_slices(slice_id), + authority_revision BIGINT NOT NULL CHECK (authority_revision > 0), + event_kind TEXT NOT NULL CHECK (event_kind = 'AUTHORITY_TRANSITION'), + payload JSONB NOT NULL CHECK (jsonb_typeof(payload) = 'object'), + payload_sha256 TEXT NOT NULL CHECK (payload_sha256 ~ '^[0-9a-f]{64}$'), + status TEXT NOT NULL DEFAULT 'PENDING' + CHECK (status IN ('PENDING', 'DISPATCHING', 'PUBLISHED', 'BLOCKED')), + attempts INTEGER NOT NULL DEFAULT 0 CHECK (attempts >= 0), + available_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + locked_at TIMESTAMPTZ, + lock_owner TEXT, + last_error TEXT, + topic TEXT, + topic_partition INTEGER, + topic_offset BIGINT CHECK (topic_offset IS NULL OR topic_offset >= 0), + published_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(), + UNIQUE (slice_id, authority_revision, event_kind), + CHECK ((locked_at IS NULL) = (lock_owner IS NULL)), + CHECK ( + (status = 'PUBLISHED' AND topic IS NOT NULL + AND topic_partition IS NOT NULL AND topic_offset IS NOT NULL + AND published_at IS NOT NULL) + OR status <> 'PUBLISHED' + ) +); + +CREATE INDEX IF NOT EXISTS qdl_authority_outbox_dispatch_idx + ON qdl_authority_event_outbox (status, available_at, created_at) + WHERE status IN ('PENDING', 'DISPATCHING'); + +CREATE OR REPLACE FUNCTION qdl_reject_authority_outbox_payload_mutation() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +BEGIN + IF NEW.event_id <> OLD.event_id + OR NEW.transition_id <> OLD.transition_id + OR NEW.slice_id <> OLD.slice_id + OR NEW.authority_revision <> OLD.authority_revision + OR NEW.event_kind <> OLD.event_kind + OR NEW.payload <> OLD.payload + OR NEW.payload_sha256 <> OLD.payload_sha256 + OR NEW.created_at <> OLD.created_at THEN + RAISE EXCEPTION 'authority outbox identity/payload is immutable'; + END IF; + NEW.updated_at := clock_timestamp(); + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_outbox_payload_immutable + ON qdl_authority_event_outbox; +CREATE TRIGGER qdl_authority_outbox_payload_immutable +BEFORE UPDATE ON qdl_authority_event_outbox +FOR EACH ROW EXECUTE FUNCTION qdl_reject_authority_outbox_payload_mutation(); + +CREATE OR REPLACE FUNCTION qdl_enqueue_authority_transition() +RETURNS TRIGGER LANGUAGE plpgsql AS $$ +DECLARE + authority_row qdl_authority_slices%ROWTYPE; + handoff_row qdl_authority_handoffs%ROWTYPE; + checkpoint_row qdl_terminal_owner_checkpoints%ROWTYPE; + event_payload JSONB; +BEGIN + SELECT * INTO authority_row + FROM qdl_authority_slices AS authority_slice + WHERE authority_slice.slice_id = NEW.slice_id; + IF NOT FOUND OR authority_row.authority_revision <> NEW.new_revision THEN + RAISE EXCEPTION 'authority outbox cannot snapshot a divergent authority row'; + END IF; + + SELECT * INTO handoff_row + FROM qdl_authority_handoffs AS handoff + WHERE handoff.slice_id = NEW.slice_id + AND handoff.expected_authority_revision = NEW.previous_revision + AND handoff.new_authority_revision = NEW.new_revision + AND handoff.old_owner_id = NEW.previous_owner_id + AND handoff.new_owner_id = NEW.new_owner_id + ORDER BY handoff.created_at DESC + LIMIT 1; + IF FOUND THEN + SELECT * INTO checkpoint_row + FROM qdl_terminal_owner_checkpoints AS checkpoint + WHERE checkpoint.checkpoint_id = handoff_row.checkpoint_id; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority handoff checkpoint is missing'; + END IF; + END IF; + + event_payload := jsonb_build_object( + 'schema', 'qdl.authority-outbox-event.v1', + 'event_id', NEW.transition_id, + 'transition', to_jsonb(NEW), + 'authority', to_jsonb(authority_row), + 'handoff', CASE WHEN handoff_row.handoff_id IS NULL THEN NULL ELSE to_jsonb(handoff_row) END, + 'checkpoint', CASE WHEN checkpoint_row.checkpoint_id IS NULL THEN NULL ELSE to_jsonb(checkpoint_row) END + ); + + INSERT INTO qdl_authority_event_outbox ( + event_id, transition_id, slice_id, authority_revision, + event_kind, payload, payload_sha256 + ) VALUES ( + NEW.transition_id, NEW.transition_id, NEW.slice_id, NEW.new_revision, + 'AUTHORITY_TRANSITION', event_payload, + encode(digest(convert_to(event_payload::text, 'UTF8'), 'sha256'), 'hex') + ) + ON CONFLICT (event_id) DO NOTHING; + RETURN NEW; +END; +$$; + +DROP TRIGGER IF EXISTS qdl_authority_transition_outbox + ON qdl_authority_transition_audit; +CREATE TRIGGER qdl_authority_transition_outbox +AFTER INSERT ON qdl_authority_transition_audit +FOR EACH ROW EXECUTE FUNCTION qdl_enqueue_authority_transition(); + +CREATE OR REPLACE FUNCTION qdl_claim_authority_outbox( + p_lock_owner TEXT, + p_limit INTEGER, + p_lock_timeout INTERVAL DEFAULT INTERVAL '2 minutes' +) +RETURNS SETOF qdl_authority_event_outbox +LANGUAGE plpgsql +AS $$ +BEGIN + IF btrim(p_lock_owner) = '' OR p_limit < 1 OR p_limit > 100 + OR p_lock_timeout <= INTERVAL '0 seconds' THEN + RAISE EXCEPTION 'authority outbox claim bounds are invalid'; + END IF; + RETURN QUERY + WITH candidates AS ( + SELECT event_id + FROM qdl_authority_event_outbox + WHERE available_at <= clock_timestamp() + AND ( + status = 'PENDING' + OR ( + status = 'DISPATCHING' + AND locked_at < clock_timestamp() - p_lock_timeout + ) + ) + ORDER BY created_at, event_id + LIMIT p_limit + FOR UPDATE SKIP LOCKED + ) + UPDATE qdl_authority_event_outbox AS outbox + SET status = 'DISPATCHING', + attempts = outbox.attempts + 1, + locked_at = clock_timestamp(), + lock_owner = p_lock_owner, + last_error = NULL + FROM candidates + WHERE outbox.event_id = candidates.event_id + RETURNING outbox.*; +END; +$$; + +CREATE OR REPLACE FUNCTION qdl_complete_authority_outbox( + p_event_id UUID, + p_lock_owner TEXT, + p_topic TEXT, + p_partition INTEGER, + p_offset BIGINT +) +RETURNS VOID LANGUAGE plpgsql AS $$ +BEGIN + UPDATE qdl_authority_event_outbox + SET status = 'PUBLISHED', + topic = p_topic, + topic_partition = p_partition, + topic_offset = p_offset, + published_at = clock_timestamp(), + locked_at = NULL, + lock_owner = NULL, + last_error = NULL + WHERE event_id = p_event_id + AND status = 'DISPATCHING' + AND lock_owner = p_lock_owner + AND btrim(p_topic) <> '' + AND p_partition >= 0 + AND p_offset >= 0; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority outbox completion lost ownership or has invalid ACK'; + END IF; +END; +$$; + +CREATE OR REPLACE FUNCTION qdl_retry_authority_outbox( + p_event_id UUID, + p_lock_owner TEXT, + p_error TEXT, + p_retry_after INTERVAL +) +RETURNS VOID LANGUAGE plpgsql AS $$ +BEGIN + IF btrim(p_error) = '' OR p_retry_after <= INTERVAL '0 seconds' THEN + RAISE EXCEPTION 'authority outbox retry requires bounded error/delay'; + END IF; + UPDATE qdl_authority_event_outbox + SET status = CASE WHEN attempts >= 20 THEN 'BLOCKED' ELSE 'PENDING' END, + available_at = clock_timestamp() + p_retry_after, + locked_at = NULL, + lock_owner = NULL, + last_error = left(p_error, 2000) + WHERE event_id = p_event_id + AND status = 'DISPATCHING' + AND lock_owner = p_lock_owner; + IF NOT FOUND THEN + RAISE EXCEPTION 'authority outbox retry lost ownership'; + END IF; +END; +$$; + +REVOKE EXECUTE ON FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + FROM PUBLIC; + +COMMIT; diff --git a/poetry.lock b/poetry.lock index 734fce4..ba46b21 100644 --- a/poetry.lock +++ b/poetry.lock @@ -57,6 +57,79 @@ files = [ {file = "async_timeout-5.0.1.tar.gz", hash = "sha256:d9321a7a3d5a6a5e187e824d2fa0793ce379a202935782d555d6e9d2735677d3"}, ] +[[package]] +name = "asyncpg" +version = "0.31.0" +description = "An asyncio PostgreSQL driver" +optional = false +python-versions = ">=3.9.0" +groups = ["main"] +files = [ + {file = "asyncpg-0.31.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:831712dd3cf117eec68575a9b50da711893fd63ebe277fc155ecae1c6c9f0f61"}, + {file = "asyncpg-0.31.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0b17c89312c2f4ccea222a3a6571f7df65d4ba2c0e803339bfc7bed46a96d3be"}, + {file = "asyncpg-0.31.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3faa62f997db0c9add34504a68ac2c342cfee4d57a0c3062fcf0d86c7f9cb1e8"}, + {file = "asyncpg-0.31.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ea599d45c361dfbf398cb67da7fd052affa556a401482d3ff1ee99bd68808a1"}, + {file = "asyncpg-0.31.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:795416369c3d284e1837461909f58418ad22b305f955e625a4b3a2521d80a5f3"}, + {file = "asyncpg-0.31.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:a8d758dac9d2e723e173d286ef5e574f0b350ec00e9186fce84d0fc5f6a8e6b8"}, + {file = "asyncpg-0.31.0-cp310-cp310-win32.whl", hash = "sha256:2d076d42eb583601179efa246c5d7ae44614b4144bc1c7a683ad1222814ed095"}, + {file = "asyncpg-0.31.0-cp310-cp310-win_amd64.whl", hash = "sha256:9ea33213ac044171f4cac23740bed9a3805abae10e7025314cfbd725ec670540"}, + {file = "asyncpg-0.31.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:eee690960e8ab85063ba93af2ce128c0f52fd655fdff9fdb1a28df01329f031d"}, + {file = "asyncpg-0.31.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2657204552b75f8288de08ca60faf4a99a65deef3a71d1467454123205a88fab"}, + {file = "asyncpg-0.31.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a429e842a3a4b4ea240ea52d7fe3f82d5149853249306f7ff166cb9948faa46c"}, + {file = "asyncpg-0.31.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c0807be46c32c963ae40d329b3a686356e417f674c976c07fa49f1b30303f109"}, + {file = "asyncpg-0.31.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:e5d5098f63beeae93512ee513d4c0c53dc12e9aa2b7a1af5a81cddf93fe4e4da"}, + {file = "asyncpg-0.31.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37fc6c00a814e18eef51833545d1891cac9aa69140598bb076b4cd29b3e010b9"}, + {file = "asyncpg-0.31.0-cp311-cp311-win32.whl", hash = "sha256:5a4af56edf82a701aece93190cc4e094d2df7d33f6e915c222fb09efbb5afc24"}, + {file = "asyncpg-0.31.0-cp311-cp311-win_amd64.whl", hash = "sha256:480c4befbdf079c14c9ca43c8c5e1fe8b6296c96f1f927158d4f1e750aacc047"}, + {file = "asyncpg-0.31.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b44c31e1efc1c15188ef183f287c728e2046abb1d26af4d20858215d50d91fad"}, + {file = "asyncpg-0.31.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0c89ccf741c067614c9b5fc7f1fc6f3b61ab05ae4aaa966e6fd6b93097c7d20d"}, + {file = "asyncpg-0.31.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:12b3b2e39dc5470abd5e98c8d3373e4b1d1234d9fbdedf538798b2c13c64460a"}, + {file = "asyncpg-0.31.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:aad7a33913fb8bcb5454313377cc330fbb19a0cd5faa7272407d8a0c4257b671"}, + {file = "asyncpg-0.31.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3df118d94f46d85b2e434fd62c84cb66d5834d5a890725fe625f498e72e4d5ec"}, + {file = "asyncpg-0.31.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bd5b6efff3c17c3202d4b37189969acf8927438a238c6257f66be3c426beba20"}, + {file = "asyncpg-0.31.0-cp312-cp312-win32.whl", hash = "sha256:027eaa61361ec735926566f995d959ade4796f6a49d3bde17e5134b9964f9ba8"}, + {file = "asyncpg-0.31.0-cp312-cp312-win_amd64.whl", hash = "sha256:72d6bdcbc93d608a1158f17932de2321f68b1a967a13e014998db87a72ed3186"}, + {file = "asyncpg-0.31.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c204fab1b91e08b0f47e90a75d1b3c62174dab21f670ad6c5d0f243a228f015b"}, + {file = "asyncpg-0.31.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:54a64f91839ba59008eccf7aad2e93d6e3de688d796f35803235ea1c4898ae1e"}, + {file = "asyncpg-0.31.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0e0822b1038dc7253b337b0f3f676cadc4ac31b126c5d42691c39691962e403"}, + {file = "asyncpg-0.31.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bef056aa502ee34204c161c72ca1f3c274917596877f825968368b2c33f585f4"}, + {file = "asyncpg-0.31.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0bfbcc5b7ffcd9b75ab1558f00db2ae07db9c80637ad1b2469c43df79d7a5ae2"}, + {file = "asyncpg-0.31.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:22bc525ebbdc24d1261ecbf6f504998244d4e3be1721784b5f64664d61fbe602"}, + {file = "asyncpg-0.31.0-cp313-cp313-win32.whl", hash = "sha256:f890de5e1e4f7e14023619399a471ce4b71f5418cd67a51853b9910fdfa73696"}, + {file = "asyncpg-0.31.0-cp313-cp313-win_amd64.whl", hash = "sha256:dc5f2fa9916f292e5c5c8b2ac2813763bcd7f58e130055b4ad8a0531314201ab"}, + {file = "asyncpg-0.31.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:f6b56b91bb0ffc328c4e3ed113136cddd9deefdf5f79ab448598b9772831df44"}, + {file = "asyncpg-0.31.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:334dec28cf20d7f5bb9e45b39546ddf247f8042a690bff9b9573d00086e69cb5"}, + {file = "asyncpg-0.31.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:98cc158c53f46de7bb677fd20c417e264fc02b36d901cc2a43bd6cb0dc6dbfd2"}, + {file = "asyncpg-0.31.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9322b563e2661a52e3cdbc93eed3be7748b289f792e0011cb2720d278b366ce2"}, + {file = "asyncpg-0.31.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19857a358fc811d82227449b7ca40afb46e75b33eb8897240c3839dd8b744218"}, + {file = "asyncpg-0.31.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ba5f8886e850882ff2c2ace5732300e99193823e8107e2c53ef01c1ebfa1e85d"}, + {file = "asyncpg-0.31.0-cp314-cp314-win32.whl", hash = "sha256:cea3a0b2a14f95834cee29432e4ddc399b95700eb1d51bbc5bfee8f31fa07b2b"}, + {file = "asyncpg-0.31.0-cp314-cp314-win_amd64.whl", hash = "sha256:04d19392716af6b029411a0264d92093b6e5e8285ae97a39957b9a9c14ea72be"}, + {file = "asyncpg-0.31.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:bdb957706da132e982cc6856bb2f7b740603472b54c3ebc77fe60ea3e57e1bd2"}, + {file = "asyncpg-0.31.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6d11b198111a72f47154fa03b85799f9be63701e068b43f84ac25da0bda9cb31"}, + {file = "asyncpg-0.31.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:18c83b03bc0d1b23e6230f5bf8d4f217dc9bc08644ce0502a9d91dc9e634a9c7"}, + {file = "asyncpg-0.31.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e009abc333464ff18b8f6fd146addffd9aaf63e79aa3bb40ab7a4c332d0c5e9e"}, + {file = "asyncpg-0.31.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3b1fbcb0e396a5ca435a8826a87e5c2c2cc0c8c68eb6fadf82168056b0e53a8c"}, + {file = "asyncpg-0.31.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8df714dba348efcc162d2adf02d213e5fab1bd9f557e1305633e851a61814a7a"}, + {file = "asyncpg-0.31.0-cp314-cp314t-win32.whl", hash = "sha256:1b41f1afb1033f2b44f3234993b15096ddc9cd71b21a42dbd87fc6a57b43d65d"}, + {file = "asyncpg-0.31.0-cp314-cp314t-win_amd64.whl", hash = "sha256:bd4107bb7cdd0e9e65fae66a62afd3a249663b844fa34d479f6d5b3bef9c04c3"}, + {file = "asyncpg-0.31.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:ebb3cde58321a1f89ce41812be3f2a98dddedc1e76d0838aba1d724f1e4e1a95"}, + {file = "asyncpg-0.31.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e6974f36eb9a224d8fb428bcf66bd411aa12cf57c2967463178149e73d4de366"}, + {file = "asyncpg-0.31.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bc2b685f400ceae428f79f78b58110470d7b4466929a7f78d455964b17ad1008"}, + {file = "asyncpg-0.31.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bb223567dea5f47c45d347f2bde5486be8d9f40339f27217adb3fb1c3be51298"}, + {file = "asyncpg-0.31.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:22be6e02381bab3101cd502d9297ac71e2f966c86e20e78caead9934c98a8af6"}, + {file = "asyncpg-0.31.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:37a58919cfef2448a920df00d1b2f821762d17194d0dbf355d6dde8d952c04f9"}, + {file = "asyncpg-0.31.0-cp39-cp39-win32.whl", hash = "sha256:c1a9c5b71d2371a2290bc93336cd05ba4ec781683cab292adbddc084f89443c6"}, + {file = "asyncpg-0.31.0-cp39-cp39-win_amd64.whl", hash = "sha256:c1e1ab5bc65373d92dd749d7308c5b26fb2dc0fbe5d3bf68a32b676aa3bcd24a"}, + {file = "asyncpg-0.31.0.tar.gz", hash = "sha256:c989386c83940bfbd787180f2b1519415e2d3d6277a70d9d0f0145ac73500735"}, +] + +[package.dependencies] +async_timeout = {version = ">=4.0.3", markers = "python_version < \"3.11.0\""} + +[package.extras] +gssauth = ["gssapi ; platform_system != \"Windows\"", "sspilib ; platform_system == \"Windows\""] + [[package]] name = "beautifulsoup4" version = "4.14.3" @@ -2801,4 +2874,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.1" python-versions = "^3.10" -content-hash = "07ffa76dde42c0a3c35b4af86a3c9c0a2bb7f65552c87c7dc4ba78401246fda0" +content-hash = "b5d9bb96e911f8d6124e04d78d461674b006d68bc5e66412936e29f77949e549" diff --git a/pyproject.toml b/pyproject.toml index baf23de..9a649bb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,6 +26,7 @@ dependencies = [ "protobuf (>=6.31.1,<7.0.0)", "grpcio (>=1.70.0,<2.0.0)", "confluent-kafka (>=2.15.0,<3.0.0)", + "asyncpg (>=0.30.0,<1.0.0)", "httpx (>=0.28.0,<1.0.0)", "idna (>=3.15,<4.0)", "pillow (>=12.3.0,<13.0.0)", diff --git a/qdl/control/authority_outbox.py b/qdl/control/authority_outbox.py new file mode 100644 index 0000000..04a742f --- /dev/null +++ b/qdl/control/authority_outbox.py @@ -0,0 +1,399 @@ +from __future__ import annotations + +import asyncio +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import datetime, timezone +import hashlib +import json +from typing import Any, Protocol + + +_CONTROL_SCHEMA = "qdl.authority-control-event.v1" +_OUTBOX_SCHEMA = "qdl.authority-outbox-event.v1" +_PHASE92_STATES = { + "RUST_CANARY", "RUST_PRIMARY", "BLOCKED", + "ROLLBACK_PENDING", "PYTHON_PRIMARY", +} + + +def _timestamp_ns(value: Any) -> int: + if isinstance(value, datetime): + parsed = value + elif isinstance(value, str): + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + else: + raise ValueError("authority timestamp is missing or invalid") + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return int(parsed.timestamp() * 1_000_000_000) + + +def _uuid(value: Any, field: str) -> str: + text = str(value or "") + parts = text.split("-") + if [len(part) for part in parts] != [8, 4, 4, 4, 12] or any( + not all(character in "0123456789abcdefABCDEF" for character in part) + for part in parts + ): + raise ValueError(f"{field} is not a UUID") + return text.lower() + + +def _digest(payload: Mapping[str, Any]) -> str: + encoded = json.dumps( + payload, ensure_ascii=True, separators=(",", ":") + ).encode() + return hashlib.sha256(encoded).hexdigest() + + +def _checkpoint(raw: Mapping[str, Any] | None) -> dict[str, Any] | None: + if raw is None: + return None + result = { + "schema": "qdl.terminal-owner-checkpoint.v1", + "checkpoint_id": _uuid(raw.get("checkpoint_id"), "checkpoint_id"), + "slice_id": str(raw.get("slice_id") or ""), + "owner_id": str(raw.get("owner_id") or ""), + "authority_revision": int(raw.get("authority_revision") or 0), + "lease_epoch": int(raw.get("lease_epoch") or 0), + "partition_plan_epoch": int(raw.get("partition_plan_epoch") or 0), + "source_session_id": str(raw.get("source_session_id") or ""), + "connection_generation": int(raw.get("connection_generation") or 0), + "terminal_watermark": int(raw.get("terminal_watermark") or 0), + "terminal_event_id": str(raw.get("terminal_event_id") or ""), + "terminal_payload_sha256": str(raw.get("terminal_payload_sha256") or ""), + "candidate_digest": str(raw.get("candidate_digest") or ""), + "committed_at_ns": _timestamp_ns(raw.get("committed_at")), + } + if any( + not result[key] + for key in ("slice_id", "owner_id", "source_session_id", "terminal_event_id") + ) or any( + result[key] <= 0 + for key in ( + "authority_revision", "lease_epoch", "partition_plan_epoch", + "connection_generation", "committed_at_ns", + ) + ) or any( + len(result[key]) != 64 + for key in ("terminal_payload_sha256", "candidate_digest") + ): + raise ValueError("terminal checkpoint is incomplete") + return result + + +def _handoff( + raw: Mapping[str, Any] | None, + checkpoint: Mapping[str, Any] | None, +) -> dict[str, Any] | None: + if raw is None: + return None + if checkpoint is None: + raise ValueError("authority handoff has no terminal checkpoint") + result = { + "schema": "qdl.accepted-authority-handoff.v1", + "handoff_id": _uuid(raw.get("handoff_id"), "handoff_id"), + "direction": str(raw.get("direction") or ""), + "checkpoint_digest": _digest(checkpoint), + "slice_id": str(raw.get("slice_id") or ""), + "old_owner_id": str(raw.get("old_owner_id") or ""), + "new_owner_id": str(raw.get("new_owner_id") or ""), + "expected_state": str(raw.get("expected_state") or ""), + "new_state": str(raw.get("new_state") or ""), + "expected_authority_revision": int(raw.get("expected_authority_revision") or 0), + "new_authority_revision": int(raw.get("new_authority_revision") or 0), + "expected_lease_epoch": int(raw.get("expected_lease_epoch") or 0), + "new_lease_epoch": int(raw.get("new_lease_epoch") or 0), + "partition_plan_epoch": int(raw.get("partition_plan_epoch") or 0), + "terminal_watermark": int(raw.get("terminal_watermark") or 0), + "first_new_watermark": int(raw.get("first_new_watermark") or 0), + "overlap_start_watermark": int(raw.get("overlap_start_watermark") or 0), + "overlap_end_watermark": int(raw.get("overlap_end_watermark") or 0), + "old_event_count": int(raw.get("old_event_count") or 0), + "new_event_count": int(raw.get("new_event_count") or 0), + "semantic_mismatches": int(raw.get("semantic_mismatches") or 0), + "open_gaps": int(raw.get("open_gaps") or 0), + "candidate_digest": str(raw.get("candidate_digest") or ""), + "prerequisite_bundle_id": _uuid( + raw.get("prerequisite_bundle_id"), "prerequisite_bundle_id" + ), + "approved_by": str(raw.get("approved_by") or ""), + "approved_at_ns": _timestamp_ns(raw.get("approved_at")), + "expires_at_ns": _timestamp_ns(raw.get("expires_at")), + } + expected_digest = str(raw.get("handoff_sha256") or "") + actual_digest = _digest(result) + if len(expected_digest) != 64 or actual_digest != expected_digest: + raise ValueError("accepted handoff digest differs from immutable DB evidence") + if ( + result["new_authority_revision"] != result["expected_authority_revision"] + 1 + or result["new_lease_epoch"] <= result["expected_lease_epoch"] + or result["first_new_watermark"] != result["terminal_watermark"] + 1 + or result["semantic_mismatches"] != 0 + or result["open_gaps"] != 0 + or result["old_event_count"] != result["new_event_count"] + ): + raise ValueError("accepted handoff boundary is invalid") + return result + + +def _authority( + raw: Mapping[str, Any], + transition: Mapping[str, Any], + handoff: Mapping[str, Any] | None, +) -> dict[str, Any] | None: + state = str(raw.get("state") or "") + if state not in _PHASE92_STATES: + return None + terminal = raw.get("terminal_watermark") + start_watermark = int(terminal or 0) + primary = state in {"RUST_PRIMARY", "PYTHON_PRIMARY"} + if primary and handoff is None: + raise ValueError("primary authority requires accepted handoff evidence") + approved_by = raw.get("approved_by") + approved_at = raw.get("approved_at") + hold_until = raw.get("hold_until") + if state == "PYTHON_PRIMARY" and handoff is not None: + approved_by = handoff["approved_by"] + approved_at = handoff["approved_at_ns"] + hold_until = handoff["expires_at_ns"] + approved_at_ns = ( + int(approved_at) + if isinstance(approved_at, int) + else _timestamp_ns(approved_at) + if approved_at is not None + else None + ) + hold_until_ns = ( + int(hold_until) + if isinstance(hold_until, int) + else _timestamp_ns(hold_until) + if hold_until is not None + else None + ) + active = state in {"RUST_CANARY", "RUST_PRIMARY", "PYTHON_PRIMARY"} + result = { + "schema": "qdl.authority-record.v3", + "slice_id": str(raw.get("slice_id") or ""), + "state": state, + "owner_id": str(raw.get("owner_id") or ""), + "authority_revision": int(raw.get("authority_revision") or 0), + "lease_epoch": int(raw.get("lease_epoch") or 0), + "partition_plan_epoch": int(raw.get("partition_plan_epoch") or 0), + "candidate_digest": str(raw.get("candidate_digest") or ""), + "prerequisite_bundle_id": ( + str(raw["prerequisite_bundle_id"]) + if raw.get("prerequisite_bundle_id") is not None + else None + ), + "start_watermark": start_watermark, + "terminal_watermark": start_watermark if primary else None, + "previous_owner_id": handoff["old_owner_id"] if primary else None, + "handoff_digest": _digest(handoff) if primary else None, + "approved_by": str(approved_by) if active and approved_by is not None else None, + "approved_at_ns": approved_at_ns if active else None, + "hold_until_ns": hold_until_ns if active else None, + "public_write_allowed": primary, + "legacy_write_allowed": primary, + } + if result["authority_revision"] != int(transition.get("new_revision") or 0): + raise ValueError("authority row and transition revision differ") + if active and ( + not result["approved_by"] + or not result["approved_at_ns"] + or not result["hold_until_ns"] + or result["hold_until_ns"] <= result["approved_at_ns"] + ): + raise ValueError("active authority approval window is invalid") + if state in {"RUST_CANARY", "RUST_PRIMARY"} and not result["prerequisite_bundle_id"]: + raise ValueError("Rust canary/primary requires prerequisite bundle") + return result + + +def build_authority_control_event(payload: Mapping[str, Any]) -> dict[str, Any]: + if payload.get("schema") != _OUTBOX_SCHEMA: + raise ValueError("unsupported authority outbox payload schema") + event_id = _uuid(payload.get("event_id"), "event_id") + transition = payload.get("transition") + authority_row = payload.get("authority") + if not isinstance(transition, Mapping) or not isinstance(authority_row, Mapping): + raise ValueError("authority outbox snapshot is incomplete") + checkpoint = _checkpoint( + payload.get("checkpoint") if isinstance(payload.get("checkpoint"), Mapping) else None + ) + handoff = _handoff( + payload.get("handoff") if isinstance(payload.get("handoff"), Mapping) else None, + checkpoint, + ) + authority = _authority(authority_row, transition, handoff) + slice_id = str(authority_row.get("slice_id") or "") + revision = int(authority_row.get("authority_revision") or 0) + if ( + not slice_id + or revision <= 0 + or str(transition.get("transition_id") or "").lower() != event_id + or str(transition.get("slice_id") or "") != slice_id + ): + raise ValueError("authority outbox event identity is inconsistent") + return { + "schema": _CONTROL_SCHEMA, + "event_id": event_id, + "slice_id": slice_id, + "authority_revision": revision, + "database_state": str(authority_row.get("state") or ""), + "authority": authority, + "checkpoint": checkpoint, + "handoff": handoff, + } + + +@dataclass(frozen=True, slots=True) +class ClaimedAuthorityEvent: + event_id: str + payload: Mapping[str, Any] + attempts: int = 1 + + +@dataclass(frozen=True, slots=True) +class BrokerAck: + topic: str + partition: int + offset: int + + +class AuthorityOutboxRepository(Protocol): + async def claim(self, lock_owner: str, limit: int) -> list[ClaimedAuthorityEvent]: ... + async def complete(self, event_id: str, lock_owner: str, ack: BrokerAck) -> None: ... + async def retry(self, event_id: str, lock_owner: str, error: str, delay_seconds: float) -> None: ... + + +class AuthorityPublisher(Protocol): + async def publish(self, *, key: str, event_id: str, payload: bytes) -> BrokerAck: ... + + +class AuthorityOutboxDispatcher: + def __init__( + self, + *, + repository: AuthorityOutboxRepository, + publisher: AuthorityPublisher, + lock_owner: str, + batch_size: int = 20, + ) -> None: + if not lock_owner.strip() or not 1 <= batch_size <= 100: + raise ValueError("authority dispatcher identity/batch bounds are invalid") + self.repository = repository + self.publisher = publisher + self.lock_owner = lock_owner + self.batch_size = batch_size + + async def dispatch_once(self) -> int: + claimed = await self.repository.claim(self.lock_owner, self.batch_size) + published = 0 + for item in claimed: + try: + event = build_authority_control_event(item.payload) + encoded = json.dumps( + event, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + ack = await self.publisher.publish( + key=event["slice_id"], event_id=item.event_id, payload=encoded + ) + await self.repository.complete(item.event_id, self.lock_owner, ack) + published += 1 + except Exception as error: + delay = min(60.0, 0.5 * 2 ** min(7, max(0, item.attempts - 1))) + await self.repository.retry( + item.event_id, self.lock_owner, str(error), delay + ) + return published + + +class AsyncpgAuthorityOutboxRepository: + def __init__(self, pool: Any) -> None: + self._pool = pool + + @classmethod + async def connect(cls, dsn: str, *, min_size: int = 1, max_size: int = 4): + try: + import asyncpg + except ImportError as error: + raise RuntimeError("authority dispatcher requires asyncpg") from error + pool = await asyncpg.create_pool( + dsn=dsn, min_size=min_size, max_size=max_size, + command_timeout=15, server_settings={"application_name": "qdl-authority-outbox"}, + ) + return cls(pool) + + async def claim(self, lock_owner: str, limit: int) -> list[ClaimedAuthorityEvent]: + rows = await self._pool.fetch( + "SELECT event_id, payload FROM qdl_claim_authority_outbox($1, $2)", + lock_owner, limit, + ) + return [ + ClaimedAuthorityEvent( + str(row["event_id"]), row["payload"], int(row["attempts"]) + ) + for row in rows + ] + + async def complete(self, event_id: str, lock_owner: str, ack: BrokerAck) -> None: + await self._pool.execute( + "SELECT qdl_complete_authority_outbox($1::uuid, $2, $3, $4, $5)", + event_id, lock_owner, ack.topic, ack.partition, ack.offset, + ) + + async def retry(self, event_id: str, lock_owner: str, error: str, delay_seconds: float) -> None: + await self._pool.execute( + "SELECT qdl_retry_authority_outbox($1::uuid, $2, $3, make_interval(secs => $4))", + event_id, lock_owner, error, delay_seconds, + ) + + async def close(self) -> None: + await self._pool.close() + + +class KafkaAuthorityPublisher: + def __init__(self, config: Mapping[str, Any], *, topic: str, timeout_seconds: float = 15.0): + if not topic.strip() or timeout_seconds <= 0: + raise ValueError("authority Kafka topic/timeout is invalid") + from confluent_kafka import Producer + values = dict(config) + values.update({ + "acks": "all", + "enable.idempotence": True, + "compression.type": "zstd", + "max.in.flight.requests.per.connection": 5, + }) + self._producer = Producer(values) + self._topic = topic + self._timeout = timeout_seconds + + async def publish(self, *, key: str, event_id: str, payload: bytes) -> BrokerAck: + loop = asyncio.get_running_loop() + completed: asyncio.Future[BrokerAck] = loop.create_future() + + def delivery(error, message) -> None: + if error is not None: + loop.call_soon_threadsafe(completed.set_exception, RuntimeError(str(error))) + else: + loop.call_soon_threadsafe( + completed.set_result, + BrokerAck(message.topic(), message.partition(), message.offset()), + ) + + self._producer.produce( + self._topic, + key=key.encode(), + value=payload, + headers={"qdl-event-id": event_id.encode()}, + on_delivery=delivery, + ) + deadline = loop.time() + self._timeout + while not completed.done(): + self._producer.poll(0) + if loop.time() >= deadline: + raise TimeoutError("authority Kafka durable ACK timed out") + await asyncio.sleep(0.01) + return await completed diff --git a/rust/qdl-venue-core/src/authority.rs b/rust/qdl-venue-core/src/authority.rs index 0852c1f..ef4484f 100644 --- a/rust/qdl-venue-core/src/authority.rs +++ b/rust/qdl-venue-core/src/authority.rs @@ -658,6 +658,74 @@ impl Phase92AuthorityRecord { } } +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92AuthorityControlEvent { + pub schema: String, + pub event_id: String, + pub slice_id: String, + pub authority_revision: u64, + pub database_state: String, + pub authority: Option, + pub checkpoint: Option, + pub handoff: Option, +} + +impl Phase92AuthorityControlEvent { + pub fn validate(&self) -> Result<(), String> { + if self.schema != "qdl.authority-control-event.v1" + || !valid_uuid(&self.event_id) + || self.slice_id.trim().is_empty() + || self.authority_revision == 0 + || self.database_state.trim().is_empty() + { + return Err("Phase 9.2 authority control event identity is invalid".into()); + } + let Some(authority) = &self.authority else { + if self.checkpoint.is_some() || self.handoff.is_some() { + return Err("non-writable authority event cannot carry handoff evidence".into()); + } + return Ok(()); + }; + authority.validate()?; + if authority.slice_id != self.slice_id + || authority.authority_revision != self.authority_revision + { + return Err("authority control event and authority record differ".into()); + } + let primary = matches!( + authority.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + if primary { + let checkpoint = self + .checkpoint + .as_ref() + .ok_or_else(|| "primary authority control event needs checkpoint".to_owned())?; + let handoff = self + .handoff + .as_ref() + .ok_or_else(|| "primary authority control event needs handoff".to_owned())?; + handoff.validate(checkpoint)?; + let handoff_digest = handoff.digest()?; + if authority.owner_id != handoff.new_owner_id + || authority.previous_owner_id.as_deref() != Some(handoff.old_owner_id.as_str()) + || authority.authority_revision != handoff.new_authority_revision + || authority.lease_epoch != handoff.new_lease_epoch + || authority.partition_plan_epoch != handoff.partition_plan_epoch + || authority.start_watermark != handoff.terminal_watermark + || authority.terminal_watermark != Some(handoff.terminal_watermark) + || authority.handoff_digest.as_deref() != Some(handoff_digest.as_str()) + { + return Err("primary authority record does not bind accepted handoff".into()); + } + } else if self.checkpoint.is_some() || self.handoff.is_some() { + return Err("non-primary authority event cannot carry handoff evidence".into()); + } + Ok(()) + } +} + #[derive(Clone, Debug, Eq, PartialEq)] pub struct Phase92PublicationContext { pub slice_id: String, @@ -680,7 +748,15 @@ pub struct Phase92AuthorityFence { impl Phase92AuthorityFence { pub fn apply(&mut self, record: Phase92AuthorityRecord) -> Result<(), String> { record.validate()?; - if self.current.is_none() { + if let Some(current) = &self.current { + if record.authority_revision == current.authority_revision { + return if record == *current { + Ok(()) + } else { + Err("conflicting Phase 9.2 authority at the same revision".into()) + }; + } + } else { self.recovery_required = matches!( record.state, Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary @@ -697,6 +773,41 @@ impl Phase92AuthorityFence { self.apply_transition(record) } + pub fn apply_control_event( + &mut self, + event: &Phase92AuthorityControlEvent, + now_ns: i64, + ) -> Result<(), String> { + event.validate()?; + let Some(record) = event.authority.clone() else { + return Ok(()); + }; + if self + .current + .as_ref() + .is_some_and(|current| current == &record) + { + return Ok(()); + } + let primary = matches!( + record.state, + Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary + ); + if primary && self.current.is_some() { + let checkpoint = event + .checkpoint + .as_ref() + .ok_or_else(|| "primary authority event needs checkpoint".to_owned())?; + let handoff = event + .handoff + .as_ref() + .ok_or_else(|| "primary authority event needs handoff".to_owned())?; + self.apply_handoff(checkpoint, handoff, record, now_ns) + } else { + self.apply(record) + } + } + pub fn apply_handoff( &mut self, checkpoint: &Phase92TerminalCheckpoint, @@ -1540,3 +1651,108 @@ mod phase92_tests { assert!(dirty.validate(&checkpoint).is_err()); } } + +#[cfg(test)] +mod phase92_control_event_tests { + use super::{ + Phase92AuthorityControlEvent, Phase92AuthorityFence, Phase92AuthorityState, + Phase92PublicationContext, SinkTarget, + }; + + const EVENT_JSON: &str = + include_str!("../../../tests/fixtures/phase9/authority-control-primary.json"); + const ACTIVE_NOW_NS: i64 = 1_787_218_200_000_000_000; + + fn publication( + event: &Phase92AuthorityControlEvent, + target: SinkTarget, + watermark: u64, + ) -> Phase92PublicationContext { + let authority = event.authority.as_ref().expect("fixture authority"); + Phase92PublicationContext { + slice_id: authority.slice_id.clone(), + owner_id: authority.owner_id.clone(), + authority_revision: authority.authority_revision, + shard_id: "binance-usdm-trade-ethusdt-0".into(), + lease_epoch: authority.lease_epoch, + partition_plan_epoch: authority.partition_plan_epoch, + source_watermark: watermark, + target, + } + } + + #[test] + fn python_control_event_decodes_and_restart_stays_fenced_until_target_restore() { + let event: Phase92AuthorityControlEvent = + serde_json::from_str(EVENT_JSON).expect("control fixture decodes"); + event.validate().expect("control fixture validates"); + let mut fence = Phase92AuthorityFence::default(); + fence + .apply_control_event(&event, ACTIVE_NOW_NS) + .expect("primary snapshot loads in recovery mode"); + let first = publication(&event, SinkTarget::PrimaryCanonical, 501); + assert!(fence + .permits(&first, ACTIVE_NOW_NS) + .is_err_and(|error| error.contains("recovery is required"))); + for target in [ + SinkTarget::PrimaryCanonical, + SinkTarget::PublicV2, + SinkTarget::LegacyV1, + ] { + fence + .restore_committed_watermark(&publication(&event, target, 500)) + .expect("independent target watermark restores"); + } + fence + .permits(&first, ACTIVE_NOW_NS) + .expect("first post-handoff canonical watermark is W+1"); + fence.commit(&first).expect("W+1 commits"); + assert!(fence + .permits( + &publication(&event, SinkTarget::PrimaryCanonical, 501), + ACTIVE_NOW_NS, + ) + .is_err_and(|error| error.contains("duplicate, stale or gapped"))); + fence + .apply_control_event(&event, ACTIVE_NOW_NS) + .expect("identical compacted authority replay is idempotent"); + } + + #[test] + fn canary_to_primary_requires_and_applies_exact_handoff() { + let event: Phase92AuthorityControlEvent = + serde_json::from_str(EVENT_JSON).expect("control fixture decodes"); + let primary = event.authority.as_ref().expect("fixture authority"); + let handoff = event.handoff.as_ref().expect("fixture handoff"); + let mut canary = primary.clone(); + canary.state = Phase92AuthorityState::RustCanary; + canary.owner_id = handoff.old_owner_id.clone(); + canary.authority_revision = handoff.expected_authority_revision; + canary.lease_epoch = handoff.expected_lease_epoch; + canary.start_watermark = handoff.terminal_watermark; + canary.terminal_watermark = None; + canary.previous_owner_id = None; + canary.handoff_digest = None; + canary.public_write_allowed = false; + canary.legacy_write_allowed = false; + let mut fence = Phase92AuthorityFence::default(); + fence.apply(canary).expect("canary authority loads"); + fence + .apply_control_event(&event, ACTIVE_NOW_NS) + .expect("exact handoff promotes primary"); + assert_eq!( + fence.current().expect("primary authority").state, + Phase92AuthorityState::RustPrimary + ); + } + + #[test] + fn altered_outer_identity_fails_closed() { + let mut event: Phase92AuthorityControlEvent = + serde_json::from_str(EVENT_JSON).expect("control fixture decodes"); + event.authority_revision += 1; + assert!(event + .validate() + .is_err_and(|error| error.contains("differ"))); + } +} diff --git a/scripts/phase92_migration_smoke.sh b/scripts/phase92_migration_smoke.sh index 302cb7d..e1b960b 100755 --- a/scripts/phase92_migration_smoke.sh +++ b/scripts/phase92_migration_smoke.sh @@ -169,17 +169,39 @@ state="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT audit_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_transition_audit;")" checkpoint_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_terminal_owner_checkpoints;")" handoff_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_handoffs;")" +outbox_count="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_event_outbox;")" +outbox_revisions="$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT string_agg(authority_revision::text, ',' ORDER BY authority_revision) FROM qdl_authority_event_outbox;")" [[ "${state}" == "PYTHON_PRIMARY:7:python-rollback:3:120" ]] [[ "${audit_count}" == "4" ]] [[ "${checkpoint_count}" == "2" ]] [[ "${handoff_count}" == "2" ]] +[[ "${outbox_count}" == "4" ]] +[[ "${outbox_revisions}" == "4,5,6,7" ]] +expect_failure "UPDATE qdl_authority_event_outbox SET payload = '{}'::jsonb;" + +docker exec -i "${CONTAINER}" psql -v ON_ERROR_STOP=1 -U postgres -d postgres <<'SQL' >/dev/null +DO $$ +DECLARE + claimed qdl_authority_event_outbox%ROWTYPE; +BEGIN + SELECT * INTO claimed FROM qdl_claim_authority_outbox('phase92-smoke', 1); + IF claimed.event_id IS NULL THEN + RAISE EXCEPTION 'authority outbox claim returned no event'; + END IF; + PERFORM qdl_complete_authority_outbox( + claimed.event_id, 'phase92-smoke', 'qdl.authority.v1', 0, 12 + ); +END; +$$; +SQL +[[ "$(docker exec "${CONTAINER}" psql -U postgres -d postgres -Atc "SELECT count(*) FROM qdl_authority_event_outbox WHERE status='PUBLISHED' AND topic_offset=12;")" == "1" ]] -python3 - "${OUTPUT}" "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" <<'PY' +python3 - "${OUTPUT}" "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" "${outbox_count}" <<'PY' import json import pathlib import sys -output, state, audits, checkpoints, handoffs = sys.argv[1:] +output, state, audits, checkpoints, handoffs, outbox = sys.argv[1:] path = pathlib.Path(output) path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps({ @@ -189,6 +211,9 @@ path.write_text(json.dumps({ "audit_records": int(audits), "terminal_checkpoints": int(checkpoints), "accepted_handoffs": int(handoffs), + "authority_outbox_records": int(outbox), + "authority_outbox_payload_immutable": True, + "authority_outbox_claim_ack_passed": True, "direct_primary_bypass_rejected": True, "stale_cas_rejected": True, "handoff_mutation_rejected": True, @@ -203,4 +228,4 @@ PY cleanup trap - EXIT [[ -z "$(docker ps -aq --filter name=^/${CONTAINER}$)" ]] -printf '{"status":"PASS","state":"%s","audits":%s,"checkpoints":%s,"handoffs":%s,"cleanup":true}\n' "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" +printf '{"status":"PASS","state":"%s","audits":%s,"checkpoints":%s,"handoffs":%s,"outbox":%s,"cleanup":true}\n' "${state}" "${audit_count}" "${checkpoint_count}" "${handoff_count}" "${outbox_count}" diff --git a/scripts/run_authority_outbox_dispatcher.py b/scripts/run_authority_outbox_dispatcher.py new file mode 100755 index 0000000..788c0f3 --- /dev/null +++ b/scripts/run_authority_outbox_dispatcher.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import socket + +from qdl.control.authority_outbox import ( + AsyncpgAuthorityOutboxRepository, + AuthorityOutboxDispatcher, + KafkaAuthorityPublisher, +) + + +def required(name: str) -> str: + value = os.environ.get(name, "").strip() + if not value: + raise RuntimeError(f"required environment variable is missing: {name}") + return value + + +async def run() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--once", action="store_true") + parser.add_argument("--poll-seconds", type=float, default=0.5) + parser.add_argument("--batch-size", type=int, default=20) + args = parser.parse_args() + if not 0.05 <= args.poll_seconds <= 60: + raise ValueError("poll interval is outside bounds") + repository = await AsyncpgAuthorityOutboxRepository.connect(required("QDL_CONTROL_DB_DSN")) + publisher = KafkaAuthorityPublisher( + { + "bootstrap.servers": required("QDL_KAFKA_BOOTSTRAP_SERVERS"), + "client.id": required("QDL_KAFKA_CLIENT_ID"), + "security.protocol": "ssl", + "ssl.ca.location": required("QDL_KAFKA_CA_LOCATION"), + "ssl.certificate.location": required("QDL_KAFKA_CERT_LOCATION"), + "ssl.key.location": required("QDL_KAFKA_KEY_LOCATION"), + }, + topic=required("QDL_AUTHORITY_TOPIC"), + ) + dispatcher = AuthorityOutboxDispatcher( + repository=repository, + publisher=publisher, + lock_owner=f"{socket.gethostname()}:{os.getpid()}", + batch_size=args.batch_size, + ) + try: + while True: + count = await dispatcher.dispatch_once() + print(json.dumps({"event": "qdl_authority_outbox_dispatch", "published": count})) + if args.once: + return + await asyncio.sleep(args.poll_seconds if count == 0 else 0) + finally: + await repository.close() + + +if __name__ == "__main__": + asyncio.run(run()) diff --git a/tests/fixtures/phase9/authority-control-primary.json b/tests/fixtures/phase9/authority-control-primary.json new file mode 100644 index 0000000..1403c99 --- /dev/null +++ b/tests/fixtures/phase9/authority-control-primary.json @@ -0,0 +1,72 @@ +{ + "authority": { + "approved_at_ns": 1787216400000000000, + "approved_by": "operator@example", + "authority_revision": 8, + "candidate_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "handoff_digest": "4fcd899c26ae7f129b50b0d6001bed77b613d4b24505fcec8ade9e063936e11e", + "hold_until_ns": 1787220000000000000, + "lease_epoch": 11, + "legacy_write_allowed": true, + "owner_id": "rust-primary-owner", + "partition_plan_epoch": 3, + "prerequisite_bundle_id": "40000000-0000-4000-8000-000000000004", + "previous_owner_id": "python-v1-owner", + "public_write_allowed": true, + "schema": "qdl.authority-record.v3", + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "start_watermark": 500, + "state": "RUST_PRIMARY", + "terminal_watermark": 500 + }, + "authority_revision": 8, + "checkpoint": { + "authority_revision": 7, + "candidate_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "checkpoint_id": "20000000-0000-4000-8000-000000000002", + "committed_at_ns": 1787216400000000000, + "connection_generation": 2, + "lease_epoch": 10, + "owner_id": "python-v1-owner", + "partition_plan_epoch": 3, + "schema": "qdl.terminal-owner-checkpoint.v1", + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "source_session_id": "python-session-7", + "terminal_event_id": "event-500", + "terminal_payload_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "terminal_watermark": 500 + }, + "database_state": "RUST_PRIMARY", + "event_id": "10000000-0000-4000-8000-000000000001", + "handoff": { + "approved_at_ns": 1787216400000000000, + "approved_by": "operator@example", + "candidate_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "checkpoint_digest": "ea1788a5d5f21969a70ccdad9c3ac56519d67a82149d8fec627d8cc1e3aa2c7b", + "direction": "PYTHON_TO_RUST", + "expected_authority_revision": 7, + "expected_lease_epoch": 10, + "expected_state": "RUST_CANARY", + "expires_at_ns": 1787220000000000000, + "first_new_watermark": 501, + "handoff_id": "30000000-0000-4000-8000-000000000003", + "new_authority_revision": 8, + "new_event_count": 11, + "new_lease_epoch": 11, + "new_owner_id": "rust-primary-owner", + "new_state": "RUST_PRIMARY", + "old_event_count": 11, + "old_owner_id": "python-v1-owner", + "open_gaps": 0, + "overlap_end_watermark": 500, + "overlap_start_watermark": 490, + "partition_plan_epoch": 3, + "prerequisite_bundle_id": "40000000-0000-4000-8000-000000000004", + "schema": "qdl.accepted-authority-handoff.v1", + "semantic_mismatches": 0, + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "terminal_watermark": 500 + }, + "schema": "qdl.authority-control-event.v1", + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT" +} diff --git a/tests/test_authority_outbox.py b/tests/test_authority_outbox.py new file mode 100644 index 0000000..459feb3 --- /dev/null +++ b/tests/test_authority_outbox.py @@ -0,0 +1,229 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from qdl.control.authority_outbox import ( + AuthorityOutboxDispatcher, + BrokerAck, + ClaimedAuthorityEvent, + _checkpoint, + _digest, + build_authority_control_event, +) + + +EVENT_ID = "10000000-0000-4000-8000-000000000001" +CHECKPOINT_ID = "20000000-0000-4000-8000-000000000002" +HANDOFF_ID = "30000000-0000-4000-8000-000000000003" +BUNDLE_ID = "40000000-0000-4000-8000-000000000004" +NOW = datetime(2026, 8, 20, 9, 0, tzinfo=timezone.utc) +LATER = datetime(2026, 8, 20, 10, 0, tzinfo=timezone.utc) +DIGEST = "a" * 64 + + +def checkpoint_raw(): + return { + "checkpoint_id": CHECKPOINT_ID, + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "owner_id": "python-v1-owner", + "authority_revision": 7, + "lease_epoch": 10, + "partition_plan_epoch": 3, + "source_session_id": "python-session-7", + "connection_generation": 2, + "terminal_watermark": 500, + "terminal_event_id": "event-500", + "terminal_payload_sha256": "b" * 64, + "candidate_digest": DIGEST, + "committed_at": NOW.isoformat(), + } + + +def handoff_raw(): + checkpoint = _checkpoint(checkpoint_raw()) + assert checkpoint is not None + expected = { + "schema": "qdl.accepted-authority-handoff.v1", + "handoff_id": HANDOFF_ID, + "direction": "PYTHON_TO_RUST", + "checkpoint_digest": _digest(checkpoint), + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "old_owner_id": "python-v1-owner", + "new_owner_id": "rust-primary-owner", + "expected_state": "RUST_CANARY", + "new_state": "RUST_PRIMARY", + "expected_authority_revision": 7, + "new_authority_revision": 8, + "expected_lease_epoch": 10, + "new_lease_epoch": 11, + "partition_plan_epoch": 3, + "terminal_watermark": 500, + "first_new_watermark": 501, + "overlap_start_watermark": 490, + "overlap_end_watermark": 500, + "old_event_count": 11, + "new_event_count": 11, + "semantic_mismatches": 0, + "open_gaps": 0, + "candidate_digest": DIGEST, + "prerequisite_bundle_id": BUNDLE_ID, + "approved_by": "operator@example", + "approved_at_ns": int(NOW.timestamp() * 1_000_000_000), + "expires_at_ns": int(LATER.timestamp() * 1_000_000_000), + } + return { + "handoff_id": HANDOFF_ID, + "checkpoint_id": CHECKPOINT_ID, + "direction": "PYTHON_TO_RUST", + "slice_id": expected["slice_id"], + "old_owner_id": expected["old_owner_id"], + "new_owner_id": expected["new_owner_id"], + "expected_state": expected["expected_state"], + "new_state": expected["new_state"], + "expected_authority_revision": 7, + "new_authority_revision": 8, + "expected_lease_epoch": 10, + "new_lease_epoch": 11, + "partition_plan_epoch": 3, + "terminal_watermark": 500, + "first_new_watermark": 501, + "overlap_start_watermark": 490, + "overlap_end_watermark": 500, + "old_event_count": 11, + "new_event_count": 11, + "semantic_mismatches": 0, + "open_gaps": 0, + "candidate_digest": DIGEST, + "prerequisite_bundle_id": BUNDLE_ID, + "handoff_sha256": _digest(expected), + "approved_by": "operator@example", + "approved_at": NOW.isoformat(), + "expires_at": LATER.isoformat(), + } + + +def outbox_payload(state="RUST_PRIMARY", *, handoff=True): + authority = { + "slice_id": "BINANCE:USDM:TRADE:ETHUSDT", + "state": state, + "owner_id": "rust-primary-owner", + "authority_revision": 8, + "lease_epoch": 11, + "partition_plan_epoch": 3, + "candidate_digest": DIGEST, + "prerequisite_bundle_id": BUNDLE_ID, + "terminal_watermark": 500, + "approved_by": "operator@example", + "approved_at": NOW.isoformat(), + "hold_until": LATER.isoformat(), + } + return { + "schema": "qdl.authority-outbox-event.v1", + "event_id": EVENT_ID, + "transition": { + "transition_id": EVENT_ID, + "slice_id": authority["slice_id"], + "previous_state": "RUST_CANARY", + "new_state": state, + "previous_revision": 7, + "new_revision": 8, + }, + "authority": authority, + "checkpoint": checkpoint_raw() if handoff else None, + "handoff": handoff_raw() if handoff else None, + } + + +class AuthorityControlEventTests(unittest.TestCase): + def test_primary_event_binds_exact_handoff_and_w_boundary(self): + event = build_authority_control_event(outbox_payload()) + authority = event["authority"] + self.assertEqual(authority["state"], "RUST_PRIMARY") + self.assertEqual(authority["start_watermark"], 500) + self.assertEqual(authority["terminal_watermark"], 500) + self.assertEqual(authority["previous_owner_id"], "python-v1-owner") + self.assertEqual(authority["handoff_digest"], _digest(event["handoff"])) + self.assertEqual(event["handoff"]["first_new_watermark"], 501) + + def test_shadow_transition_is_durable_but_not_phase92_writable(self): + payload = outbox_payload("RUST_SHADOW", handoff=False) + payload["authority"].update({ + "owner_id": "rust-shadow-owner", + "authority_revision": 2, + "lease_epoch": 2, + "terminal_watermark": None, + "prerequisite_bundle_id": None, + "approved_by": None, + "approved_at": None, + "hold_until": None, + }) + payload["transition"].update({"previous_revision": 1, "new_revision": 2}) + event = build_authority_control_event(payload) + self.assertIsNone(event["authority"]) + self.assertEqual(event["database_state"], "RUST_SHADOW") + + def test_tampered_handoff_and_primary_without_handoff_fail_closed(self): + tampered = outbox_payload() + tampered["handoff"]["handoff_sha256"] = "f" * 64 + with self.assertRaisesRegex(ValueError, "digest differs"): + build_authority_control_event(tampered) + with self.assertRaisesRegex(ValueError, "requires accepted handoff"): + build_authority_control_event(outbox_payload(handoff=False)) + + +class _Repository: + def __init__(self, payload): + self.claimed = [ClaimedAuthorityEvent(EVENT_ID, payload)] + self.completed = [] + self.retried = [] + + async def claim(self, lock_owner, limit): + del lock_owner, limit + values, self.claimed = self.claimed, [] + return values + + async def complete(self, event_id, lock_owner, ack): + self.completed.append((event_id, lock_owner, ack)) + + async def retry(self, event_id, lock_owner, error, delay_seconds): + self.retried.append((event_id, lock_owner, error, delay_seconds)) + + +class _Publisher: + def __init__(self, fail=False): + self.fail = fail + self.calls = [] + + async def publish(self, *, key, event_id, payload): + self.calls.append((key, event_id, payload)) + if self.fail: + raise RuntimeError("broker unavailable") + return BrokerAck("qdl.authority.v1", 0, 12) + + +class AuthorityOutboxDispatcherTests(unittest.IsolatedAsyncioTestCase): + async def test_ack_completes_and_failure_is_retried_without_false_publish(self): + repository = _Repository(outbox_payload()) + publisher = _Publisher() + dispatcher = AuthorityOutboxDispatcher( + repository=repository, publisher=publisher, + lock_owner="dispatcher-1", batch_size=1, + ) + self.assertEqual(await dispatcher.dispatch_once(), 1) + self.assertEqual(len(repository.completed), 1) + self.assertEqual(repository.retried, []) + self.assertEqual(publisher.calls[0][0], "BINANCE:USDM:TRADE:ETHUSDT") + + failed_repository = _Repository(outbox_payload()) + failed = AuthorityOutboxDispatcher( + repository=failed_repository, publisher=_Publisher(fail=True), + lock_owner="dispatcher-2", batch_size=1, + ) + self.assertEqual(await failed.dispatch_once(), 0) + self.assertEqual(failed_repository.completed, []) + self.assertEqual(len(failed_repository.retried), 1) + + +if __name__ == "__main__": + unittest.main() From 56ca153872d0680c555e7765b810eb48c00b1fe4 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 09:59:18 +0000 Subject: [PATCH 07/21] feat(core): add long-running Rust V2 authority runtime --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 34 + Dockerfile.phase8-rust | 2 + qdl/runtime/stable_deployment.py | 95 +++ rust/qdl-kafka/src/bin/qdl-production-core.rs | 661 +++++++++++++++ rust/qdl-kafka/src/lib.rs | 2 + rust/qdl-kafka/src/phase92_runtime.rs | 792 ++++++++++++++++++ rust/qdl-venue-core/src/authority.rs | 36 +- scripts/build_production_core_bundle.py | 48 ++ tests/test_production_catalog.py | 31 + 9 files changed, 1691 insertions(+), 10 deletions(-) create mode 100644 rust/qdl-kafka/src/bin/qdl-production-core.rs create mode 100644 rust/qdl-kafka/src/phase92_runtime.rs create mode 100755 scripts/build_production_core_bundle.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 1c377fd..0f0be5b 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5952,6 +5952,40 @@ catalog or SDK wiring cannot be deferred as operational debt. **C.0 implementation journal:** +- `2026-08-20 C.0 LONG-RUNNING PRIMARY BRIDGE ACTIVE`: implement the + production consume-transform-produce boundary as a separate Rust runtime, + leaving the certified shadow binary and V1 runtime unchanged. Every accepted + raw offset must transactionally commit its canonical/quarantine decision, + per-target projection progress and compacted target checkpoint. Startup must + reconstruct the latest compacted authority event and all applicable target + watermarks before any write; fresh accepted handoff may bootstrap exactly at + terminal W, and every normal/restart path resumes at W+1. Authority updates + race under the same fence held through transaction ACK. Missing, partial, + stale or conflicting recovery state fails closed. Tests must cover valid, + filtered, duplicate and quarantine decisions, crash/restart, compacted replay, + active authority change and rollback. This slice cannot change port 8100, + production routes, topics, consumers or authority. +- `2026-08-20 C.0 LONG-RUNNING PRIMARY BRIDGE CODE PASS`: added a separate + multi-slice `qdl-production-core` binary and Phase 9.2 transactional bridge. + Authority is reconstructed per slice from the compacted control topic; raw + acquisition revision/lease is explicitly bound but separate from final + publication authority. Logical per-slice watermarks are independent of Kafka + partition offsets. Every raw decision commits its source offset, zero or more + canonical/quarantine records, progress for each permitted target and compacted + target checkpoints in one Kafka transaction. Filtered, duplicate and + quarantine decisions still advance projection progress without fabricating + market data. Expanded provider rows are hashed as one ordered checkpoint + payload set. Restart requires complete current-owner checkpoints, except the + first accepted W handoff may bootstrap exactly at terminal W; partial recovery + fails closed. Authority watcher updates share the transaction fence, so + BLOCKED/rollback cannot race a durable output ACK. Added deterministic + production-core configs generated from the approved provider metadata catalog, + plus immutable image packaging. Production catalog/runtime and outbox tests + passed 7/7; the complete Rust workspace passed 70/70 with strict Clippy and + formatting. No broker integration, image deployment, provider call, V1 route, + port 8100, production topic/database or consumer was mutated. RF3 transaction, + restart and rollback evidence remains mandatory in isolated C.1 before this + code can be called runtime-certified. - `2026-08-20 RELEASE/CUTOVER PREPARATION RECORDED`: corrected the malformed `RUNTIME UNCHANGED` journal line and added the production cutover boundary plus `docs/runbooks/v2-production-rust-authority-cutover.md`. Read-only diff --git a/Dockerfile.phase8-rust b/Dockerfile.phase8-rust index 94cd756..83e7fd6 100644 --- a/Dockerfile.phase8-rust +++ b/Dockerfile.phase8-rust @@ -23,6 +23,7 @@ RUN cargo build --release --locked \ --bin qdl-binance-shadow \ --bin qdl-native-raw-ingestor \ --bin qdl-realtime-core \ + --bin qdl-production-core \ --bin qdl-parity-replay \ --bin qdl-venue-core-certify @@ -45,6 +46,7 @@ COPY --from=builder /src/target/release/qdl-phase92-primary-rehearsal /usr/local COPY --from=builder /src/target/release/qdl-binance-shadow /usr/local/bin/qdl-binance-shadow COPY --from=builder /src/target/release/qdl-native-raw-ingestor /usr/local/bin/qdl-native-raw-ingestor COPY --from=builder /src/target/release/qdl-realtime-core /usr/local/bin/qdl-realtime-core +COPY --from=builder /src/target/release/qdl-production-core /usr/local/bin/qdl-production-core COPY --from=builder /src/target/release/qdl-parity-replay /usr/local/bin/qdl-parity-replay COPY --from=builder /src/target/release/qdl-venue-core-certify /usr/local/bin/qdl-venue-core-certify USER 10001:10001 diff --git a/qdl/runtime/stable_deployment.py b/qdl/runtime/stable_deployment.py index f3ae1ed..1b6f5d2 100644 --- a/qdl/runtime/stable_deployment.py +++ b/qdl/runtime/stable_deployment.py @@ -222,6 +222,59 @@ def core_config( "metrics_every_batches": 100, } + def production_core_config( + self, + *, + catalog: StableSourceCatalog, + raw_authority: Mapping[str, Any], + worker_index: int, + partition_plan_epoch: int = 1, + ) -> dict[str, Any]: + self._validate_authority(raw_authority) + if partition_plan_epoch < 1: + raise ValueError("production partition plan epoch must be positive") + shadow = self.core_config( + catalog=catalog, + authority=raw_authority, + worker_index=worker_index, + ) + slices = [] + for source in sorted(catalog.bindings, key=lambda value: value.binding_id): + identity = source.instrument.identity + native = source.instrument.native_symbol.lower() + slice_id = ( + f"production/{identity.venue.lower()}/{identity.market.lower()}/" + f"{identity.product_type.value.lower()}/{source.feed.value.lower()}/" + f"plan-{partition_plan_epoch}/{native}" + ) + slices.append({ + "subscription_id": source.source_id, + "slice_id": slice_id, + "shard_id": source.binding_id, + "raw_authority_revision": int(raw_authority["revision"]), + "raw_lease_epoch": 1, + "raw_partition_plan_epoch": partition_plan_epoch, + }) + return { + "core": shadow["core"], + "topics": { + "raw_inputs": [self.raw_topic], + "authority_control": "qdl.authority.v1", + "target_checkpoints": "qdl.target-checkpoint.v1", + "canary_canonical": "md.canary.canonical.v2", + "primary_canonical": self.canonical_topic, + "public_v2": "md.projector.public.v2", + "legacy_v1": "md.projector.legacy.v1", + "quarantine": self.quarantine_topic, + }, + "slices": slices, + "transactional_id": f"qdl-v2-production-core-{worker_index:03d}", + "batch_size": 128, + "batch_wait_ms": 10, + "max_events": 0, + "metrics_every_batches": 100, + } + def native_ingestor_configs( self, *, @@ -300,6 +353,48 @@ def _validate_authority(authority: Mapping[str, Any]) -> None: raise ValueError("stable authority is not an isolated Rust shadow record") +def write_production_core_bundle( + destination: Path, + *, + catalog: StableSourceCatalog, + acquisition: StableAcquisitionPlan, + raw_authority: Mapping[str, Any], + partition_plan_epoch: int = 1, +) -> dict[str, str]: + destination.mkdir(parents=True, exist_ok=True) + payloads = { + f"production-core-{worker_index:03d}.json": + acquisition.production_core_config( + catalog=catalog, + raw_authority=raw_authority, + worker_index=worker_index, + partition_plan_epoch=partition_plan_epoch, + ) + for worker_index in range(1, STABLE_CORE_WORKER_COUNT + 1) + } + digests = {} + for name, payload in sorted(payloads.items()): + encoded = ( + json.dumps(payload, indent=2, sort_keys=True, separators=(",", ": ")) + "\n" + ).encode() + path = destination / name + path.write_bytes(encoded) + digests[name] = hashlib.sha256(encoded).hexdigest() + manifest = { + "schema": "qdl.v2.production-core-bundle.v1", + "partition_plan_epoch": partition_plan_epoch, + "worker_count": STABLE_CORE_WORKER_COUNT, + "files": digests, + } + encoded_manifest = ( + json.dumps(manifest, indent=2, sort_keys=True, separators=(",", ": ")) + "\n" + ).encode() + manifest_path = destination / "production-core-manifest.json" + manifest_path.write_bytes(encoded_manifest) + digests[manifest_path.name] = hashlib.sha256(encoded_manifest).hexdigest() + return digests + + def stable_authority_record( *, rust_image_digest: str, diff --git a/rust/qdl-kafka/src/bin/qdl-production-core.rs b/rust/qdl-kafka/src/bin/qdl-production-core.rs new file mode 100644 index 0000000..05de444 --- /dev/null +++ b/rust/qdl-kafka/src/bin/qdl-production-core.rs @@ -0,0 +1,661 @@ +#![forbid(unsafe_code)] + +use std::collections::{HashMap, HashSet}; +use std::env; +use std::sync::Arc; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use prost::Message; +use qdl_contracts::qdl::provider::v1::RawProviderEnvelope; +use qdl_core::backoff::BackoffPolicy; +use qdl_core::transport::RetryClass; +use qdl_kafka::phase92_runtime::{ + KafkaCompactedSnapshotReader, Phase92Decision, Phase92Progress, Phase92TargetCheckpoint, + Phase92TransactionalKafkaBridge, Phase92TransactionalOutput, Phase92TransactionalTopics, +}; +use qdl_kafka::{KafkaEventSource, KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError}; +use qdl_realtime_core::{ProcessBatch, RealtimeCore, RealtimeCoreConfig}; +use qdl_venue_core::authority::{ + Phase92AuthorityControlEvent, Phase92AuthorityState, Phase92PublicationContext, SinkTarget, +}; +use serde::Deserialize; +use serde_json::json; + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct RuntimeSliceBinding { + subscription_id: String, + slice_id: String, + shard_id: String, + raw_authority_revision: u64, + raw_lease_epoch: u64, + raw_partition_plan_epoch: u64, +} + +impl RuntimeSliceBinding { + fn validate(&self) -> Result<(), String> { + if self.subscription_id.trim().is_empty() + || self.slice_id.trim().is_empty() + || self.shard_id.trim().is_empty() + || self.raw_authority_revision == 0 + || self.raw_lease_epoch == 0 + || self.raw_partition_plan_epoch == 0 + { + return Err("production runtime slice binding is invalid".into()); + } + Ok(()) + } +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct ProductionRuntimeConfig { + core: RealtimeCoreConfig, + topics: ProductionTopicConfig, + slices: Vec, + transactional_id: String, + batch_size: usize, + batch_wait_ms: u64, + max_events: u64, + metrics_every_batches: u64, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct ProductionTopicConfig { + raw_inputs: Vec, + authority_control: String, + target_checkpoints: String, + canary_canonical: String, + primary_canonical: String, + public_v2: String, + legacy_v1: String, + quarantine: String, +} + +impl ProductionRuntimeConfig { + fn topics(&self) -> Phase92TransactionalTopics { + Phase92TransactionalTopics { + raw_inputs: self.topics.raw_inputs.clone(), + canary_canonical: self.topics.canary_canonical.clone(), + primary_canonical: self.topics.primary_canonical.clone(), + public_v2: self.topics.public_v2.clone(), + legacy_v1: self.topics.legacy_v1.clone(), + quarantine: self.topics.quarantine.clone(), + target_checkpoints: self.topics.target_checkpoints.clone(), + authority_control: self.topics.authority_control.clone(), + } + } + + fn validate(&self) -> Result<(), String> { + self.core.validate().map_err(|error| error.to_string())?; + self.topics() + .validate() + .map_err(|error| error.to_string())?; + if self.transactional_id.trim().is_empty() + || self.slices.is_empty() + || self.batch_size == 0 + || self.batch_size > 1_000 + || self.batch_wait_ms == 0 + || self.batch_wait_ms > 1_000 + || self.metrics_every_batches == 0 + { + return Err("production realtime core config bounds are invalid".into()); + } + let mut subscriptions = HashSet::new(); + let mut slices = HashSet::new(); + for binding in &self.slices { + binding.validate()?; + if !subscriptions.insert(binding.subscription_id.clone()) + || !slices.insert(binding.slice_id.clone()) + { + return Err( + "production runtime subscription/slice identities must be unique".into(), + ); + } + } + Ok(()) + } + + fn bindings(&self) -> HashMap { + self.slices + .iter() + .cloned() + .map(|binding| (binding.subscription_id.clone(), binding)) + .collect() + } +} + +type RuntimeError = Box; + +fn required(name: &str) -> Result { + env::var(name).map_err(|_| format!("required environment variable is missing: {name}")) +} + +fn kafka_config(group_suffix: &str) -> Result { + let cert_root = required("QDL_KAFKA_CERT_ROOT")?; + let timeout_seconds = env::var("QDL_KAFKA_REQUEST_TIMEOUT_SECONDS") + .unwrap_or_else(|_| "30".into()) + .parse::() + .map_err(|_| "QDL_KAFKA_REQUEST_TIMEOUT_SECONDS must be positive")?; + if timeout_seconds == 0 || group_suffix.trim().is_empty() { + return Err("Kafka timeout/group suffix must be positive and non-empty".into()); + } + Ok(KafkaTransportConfig { + bootstrap_servers: required("QDL_KAFKA_BOOTSTRAP_SERVERS")?, + client_id: format!("{}-{group_suffix}", required("QDL_KAFKA_CLIENT_ID")?), + group_id: format!("{}-{group_suffix}", required("QDL_KAFKA_GROUP_ID")?), + request_timeout: Duration::from_secs(timeout_seconds), + tls: KafkaTlsConfig { + ca_location: format!("{cert_root}/ca.crt"), + certificate_location: format!("{cert_root}/client.crt"), + key_location: format!("{cert_root}/client.key"), + key_password: None, + }, + }) +} + +fn now_ns() -> Result { + Ok(SystemTime::now() + .duration_since(UNIX_EPOCH)? + .as_nanos() + .try_into()?) +} + +fn should_retry_transport(class: RetryClass) -> bool { + class != RetryClass::NonRetryable +} + +fn retryable_runtime_error(error: &RuntimeError) -> bool { + error + .downcast_ref::() + .is_some_and(|value| should_retry_transport(value.retry_class())) +} + +fn expected_targets(state: Phase92AuthorityState) -> Result, String> { + match state { + Phase92AuthorityState::RustCanary => Ok(vec![SinkTarget::CanaryCanonical]), + Phase92AuthorityState::RustPrimary => Ok(vec![ + SinkTarget::PrimaryCanonical, + SinkTarget::PublicV2, + SinkTarget::LegacyV1, + ]), + Phase92AuthorityState::PythonPrimary + | Phase92AuthorityState::Blocked + | Phase92AuthorityState::RollbackPending => { + Err("Rust production core does not own this authority state".into()) + } + } +} + +async fn restore_authority( + bridge: &Phase92TransactionalKafkaBridge, + config: &ProductionRuntimeConfig, +) -> Result<(), RuntimeError> { + let snapshot_config = kafka_config(&format!("phase92-recovery-{}", config.transactional_id))?; + let records = KafkaCompactedSnapshotReader::new(snapshot_config)?.read(&[ + &config.topics.authority_control, + &config.topics.target_checkpoints, + ])?; + let required_slices: HashSet<_> = config + .slices + .iter() + .map(|binding| binding.slice_id.as_str()) + .collect(); + let mut events = HashMap::new(); + let mut checkpoints = Vec::new(); + for record in records { + if record.topic == config.topics.authority_control { + let event: Phase92AuthorityControlEvent = serde_json::from_slice(&record.payload)?; + event.validate()?; + if record.key != event.slice_id { + return Err("authority compacted key differs from event slice".into()); + } + if required_slices.contains(event.slice_id.as_str()) { + if event.authority.is_none() { + return Err("production slice has no Phase 9.2 authority record".into()); + } + events.insert(event.slice_id.clone(), event); + } + } else if record.topic == config.topics.target_checkpoints { + let checkpoint: Phase92TargetCheckpoint = serde_json::from_slice(&record.payload)?; + checkpoint.validate()?; + if record.key != checkpoint.key() { + return Err("target checkpoint compacted key differs from payload".into()); + } + if required_slices.contains(checkpoint.slice_id.as_str()) { + checkpoints.push(checkpoint); + } + } + } + if events.len() != required_slices.len() { + return Err("one or more production slices have no authority event".into()); + } + let restore_time = now_ns()?; + for binding in &config.slices { + let event = events + .get(&binding.slice_id) + .ok_or("production authority event is missing")?; + bridge.apply_authority_event(event, restore_time).await?; + let authority = event + .authority + .as_ref() + .ok_or("production authority record is missing")?; + let targets = expected_targets(authority.state)?; + let exact: Vec<_> = checkpoints + .iter() + .filter(|checkpoint| { + checkpoint.slice_id == binding.slice_id + && checkpoint.shard_id == binding.shard_id + && checkpoint.owner_id == authority.owner_id + && checkpoint.authority_revision == authority.authority_revision + && checkpoint.lease_epoch == authority.lease_epoch + && checkpoint.partition_plan_epoch == authority.partition_plan_epoch + && checkpoint.candidate_digest == authority.candidate_digest + && targets.contains(&checkpoint.target) + }) + .collect(); + if exact.len() == targets.len() { + for checkpoint in exact { + bridge.restore_checkpoint(checkpoint).await?; + } + } else if exact.is_empty() && authority.state == Phase92AuthorityState::RustPrimary { + let handoff = event + .handoff + .as_ref() + .ok_or("fresh Rust primary bootstrap requires accepted handoff")?; + let terminal = event + .checkpoint + .as_ref() + .ok_or("fresh Rust primary bootstrap requires terminal checkpoint")?; + if handoff.terminal_watermark != authority.start_watermark + || terminal.terminal_watermark != authority.start_watermark + { + return Err("fresh Rust primary bootstrap W differs from handoff evidence".into()); + } + for target in targets { + let checkpoint = Phase92TargetCheckpoint { + schema: "qdl.target-watermark-checkpoint.v1".into(), + slice_id: authority.slice_id.clone(), + owner_id: authority.owner_id.clone(), + authority_revision: authority.authority_revision, + lease_epoch: authority.lease_epoch, + partition_plan_epoch: authority.partition_plan_epoch, + shard_id: binding.shard_id.clone(), + target, + source_watermark: authority.start_watermark, + source_event_id: format!("handoff-{}", handoff.handoff_id), + decision: Phase92Decision::Filtered, + output_payload_sha256: "0".repeat(64), + candidate_digest: authority.candidate_digest.clone(), + committed_at_ns: restore_time, + }; + bridge.restore_checkpoint(&checkpoint).await?; + } + } else if !exact.is_empty() || authority.state != Phase92AuthorityState::RustCanary { + return Err("target checkpoint recovery is partial or inconsistent".into()); + } + } + Ok(()) +} + +async fn watch_authority( + bridge: Arc, + config: KafkaTransportConfig, + topic: String, + allowed_slices: HashSet, +) -> Result<(), RuntimeError> { + let source = KafkaEventSource::new(&config, &[&topic])?; + loop { + let (record, _) = source.next().await?; + let event: Phase92AuthorityControlEvent = serde_json::from_slice(&record.payload)?; + event.validate()?; + if record.partition_key != event.slice_id { + return Err("authority stream key differs from event slice".into()); + } + if !allowed_slices.contains(&event.slice_id) { + source.checkpoint()?; + continue; + } + let current = bridge.current_authority(&event.slice_id).await; + if current + .as_ref() + .is_some_and(|value| event.authority_revision < value.authority_revision) + { + source.checkpoint()?; + continue; + } + if event.authority.is_none() { + return Err("active production slice received non-Phase92 authority event".into()); + } + bridge.apply_authority_event(&event, now_ns()?).await?; + source.checkpoint()?; + } +} + +fn validate_raw_authority( + raw: &RawProviderEnvelope, + binding: &RuntimeSliceBinding, +) -> Result<(), RuntimeError> { + if raw.subscription_id != binding.subscription_id + || raw.authority_revision != binding.raw_authority_revision + || raw.lease_epoch != binding.raw_lease_epoch + || raw.partition_plan_epoch != binding.raw_partition_plan_epoch + { + return Err("raw acquisition identity/lease differs from approved binding".into()); + } + Ok(()) +} + +fn decision(result: &ProcessBatch) -> Result { + let active = [ + !result.canonical.is_empty(), + !result.quarantines.is_empty(), + result.duplicates > 0, + result.filtered > 0, + ]; + if active.into_iter().filter(|value| *value).count() != 1 { + return Err("normalizer produced an ambiguous decision for one raw event".into()); + } + if !result.canonical.is_empty() { + Ok(Phase92Decision::Canonical) + } else if !result.quarantines.is_empty() { + Ok(Phase92Decision::Quarantine) + } else if result.duplicates > 0 { + Ok(Phase92Decision::Duplicate) + } else { + Ok(Phase92Decision::Filtered) + } +} + +fn output_stream(topics: &ProductionTopicConfig, target: SinkTarget) -> Result<&str, RuntimeError> { + match target { + SinkTarget::CanaryCanonical => Ok(&topics.canary_canonical), + SinkTarget::PrimaryCanonical => Ok(&topics.primary_canonical), + SinkTarget::PublicV2 => Ok(&topics.public_v2), + SinkTarget::LegacyV1 => Ok(&topics.legacy_v1), + _ => Err("production runtime received a non-production target".into()), + } +} + +async fn run_generation( + config: &ProductionRuntimeConfig, + generation: u64, +) -> Result<(), RuntimeError> { + let mut core = RealtimeCore::new(config.core.clone())?; + let bridge = Arc::new(Phase92TransactionalKafkaBridge::new( + &kafka_config("phase92-raw")?, + config.topics(), + &config.transactional_id, + )?); + restore_authority(&bridge, config).await?; + let binding_by_subscription = config.bindings(); + let allowed_slices: HashSet<_> = config + .slices + .iter() + .map(|binding| binding.slice_id.clone()) + .collect(); + let mut authority_task = tokio::spawn(watch_authority( + Arc::clone(&bridge), + kafka_config(&format!("phase92-authority-{}", config.transactional_id))?, + config.topics.authority_control.clone(), + allowed_slices, + )); + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_started", + "generation": generation, + "slices": config.slices.len(), + "bindings": config.core.bindings.len(), + "authority_reconstructed": true, + "target_watermarks_reconstructed": true, + }))? + ); + + let mut processed = 0_u64; + let mut canonical = 0_u64; + let mut quarantines = 0_u64; + let mut duplicates = 0_u64; + let mut filtered = 0_u64; + let mut batches = 0_u64; + 'service: loop { + if config.max_events > 0 && processed >= config.max_events { + break; + } + let first = tokio::select! { + result = bridge.next() => result?, + result = tokio::signal::ctrl_c() => { + result?; + break 'service; + } + result = &mut authority_task => { + return match result { + Ok(Ok(())) => Err("authority watcher stopped unexpectedly".into()), + Ok(Err(error)) => Err(error), + Err(error) => Err(error.into()), + }; + } + }; + let mut inputs = vec![first]; + while inputs.len() < config.batch_size + && (config.max_events == 0 || processed + (inputs.len() as u64) < config.max_events) + { + match tokio::time::timeout(Duration::from_millis(config.batch_wait_ms), bridge.next()) + .await + { + Ok(Ok(input)) => inputs.push(input), + Ok(Err(error)) => return Err(error.into()), + Err(_) => break, + } + } + + let normalized_at_ns = now_ns()?; + let mut outputs = Vec::new(); + let mut progress = Vec::new(); + let mut local_next: HashMap<(String, String, SinkTarget), u64> = HashMap::new(); + for input in &inputs { + let raw = RawProviderEnvelope::decode(input.record.payload.as_slice())?; + let binding = binding_by_subscription + .get(&raw.subscription_id) + .ok_or("raw event has no approved production slice binding")?; + validate_raw_authority(&raw, binding)?; + let authority = bridge + .current_authority(&binding.slice_id) + .await + .ok_or("production slice authority disappeared")?; + let targets = expected_targets(authority.state)?; + let result = + core.process_at_transport_offset(raw, normalized_at_ns, input.cursor.offset)?; + let item_decision = decision(&result)?; + canonical += result.canonical.len() as u64; + quarantines += result.quarantines.len() as u64; + duplicates += result.duplicates as u64; + filtered += result.filtered as u64; + + for target in targets { + let key = (binding.slice_id.clone(), binding.shard_id.clone(), target); + let source_watermark = if let Some(next) = local_next.get_mut(&key) { + let value = *next; + *next = next.checked_add(1).ok_or("logical watermark overflow")?; + value + } else { + let value = bridge + .next_watermark(&binding.slice_id, &binding.shard_id, target) + .await?; + local_next.insert( + key, + value.checked_add(1).ok_or("logical watermark overflow")?, + ); + value + }; + let publication = Phase92PublicationContext { + slice_id: authority.slice_id.clone(), + owner_id: authority.owner_id.clone(), + authority_revision: authority.authority_revision, + shard_id: binding.shard_id.clone(), + lease_epoch: authority.lease_epoch, + partition_plan_epoch: authority.partition_plan_epoch, + source_watermark, + target, + }; + progress.push(Phase92Progress { + publication: publication.clone(), + decision: item_decision, + source_cursor: input.cursor.clone(), + source_event_id: input.record.event_id.clone(), + }); + if item_decision == Phase92Decision::Canonical { + for record in &result.canonical { + let mut projected = record.clone(); + projected.stream = output_stream(&config.topics, target)?.into(); + outputs.push(Phase92TransactionalOutput { + record: projected, + publication: publication.clone(), + raw_provider_envelope: Some(input.record.payload.clone()), + }); + } + } else if item_decision == Phase92Decision::Quarantine + && target == SinkTarget::PrimaryCanonical + { + for record in &result.quarantines { + let mut quarantined = record.clone(); + quarantined.stream = config.topics.quarantine.clone(); + outputs.push(Phase92TransactionalOutput { + record: quarantined, + publication: publication.clone(), + raw_provider_envelope: Some(input.record.payload.clone()), + }); + } + } + } + } + bridge + .commit(&inputs, &outputs, &progress, normalized_at_ns) + .await?; + processed += inputs.len() as u64; + batches += 1; + if batches % config.metrics_every_batches == 0 { + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_progress", + "generation": generation, + "processed": processed, + "canonical": canonical, + "quarantines": quarantines, + "duplicates": duplicates, + "filtered": filtered, + "batches": batches, + }))? + ); + } + } + authority_task.abort(); + println!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_stopped", + "generation": generation, + "processed": processed, + "canonical": canonical, + "quarantines": quarantines, + "duplicates": duplicates, + "filtered": filtered, + "batches": batches, + }))? + ); + Ok(()) +} + +#[tokio::main] +async fn main() -> Result<(), RuntimeError> { + let config_path = env::args() + .nth(1) + .ok_or("usage: qdl-production-core CONFIG.json")?; + let config: ProductionRuntimeConfig = + serde_json::from_slice(&tokio::fs::read(config_path).await?)?; + config.validate()?; + let backoff = BackoffPolicy { + initial_ms: 500, + maximum_ms: 30_000, + multiplier: 2, + jitter_bps: 2_000, + } + .validate()?; + let mut generation = 0_u64; + let mut failures = 0_u32; + loop { + generation = generation.saturating_add(1); + match run_generation(&config, generation).await { + Ok(()) => return Ok(()), + Err(error) if retryable_runtime_error(&error) => { + failures = failures.saturating_add(1); + eprintln!( + "{}", + serde_json::to_string(&json!({ + "event": "qdl_production_core_retry", + "generation": generation, + "attempt": failures, + "error": error.to_string(), + }))? + ); + tokio::time::sleep(Duration::from_millis( + backoff.delay_ms(failures, failures.min(10_000) as u16), + )) + .await; + } + Err(error) => return Err(error), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn decision_is_exclusive_and_fail_closed() { + let canonical = ProcessBatch { + canonical: vec![qdl_core::transport::DurableRecord { + stream: "canonical".into(), + partition_key: "key".into(), + event_id: vec![1], + payload: vec![2], + accepted_at_ns: 1, + }], + quarantines: vec![], + duplicates: 0, + filtered: 0, + }; + assert_eq!(decision(&canonical).unwrap(), Phase92Decision::Canonical); + let ambiguous = ProcessBatch { + filtered: 1, + ..canonical + }; + assert!(decision(&ambiguous).is_err()); + } + + #[test] + fn raw_and_publication_authorities_are_explicitly_separate() { + let raw = RawProviderEnvelope { + subscription_id: "binance-btc-trade".into(), + authority_revision: 1, + lease_epoch: 2, + partition_plan_epoch: 3, + ..Default::default() + }; + let binding = RuntimeSliceBinding { + subscription_id: "binance-btc-trade".into(), + slice_id: "production/binance/usdm/perpetual/trade/plan-1/btcusdt".into(), + shard_id: "btcusdt-trade".into(), + raw_authority_revision: 1, + raw_lease_epoch: 2, + raw_partition_plan_epoch: 3, + }; + validate_raw_authority(&raw, &binding).unwrap(); + let mut stale = raw; + stale.lease_epoch = 1; + assert!(validate_raw_authority(&stale, &binding).is_err()); + } +} diff --git a/rust/qdl-kafka/src/lib.rs b/rust/qdl-kafka/src/lib.rs index d76e728..ca322a9 100644 --- a/rust/qdl-kafka/src/lib.rs +++ b/rust/qdl-kafka/src/lib.rs @@ -1,5 +1,7 @@ #![forbid(unsafe_code)] +pub mod phase92_runtime; + use std::fmt::{Display, Formatter}; use std::path::Path; use std::time::Duration; diff --git a/rust/qdl-kafka/src/phase92_runtime.rs b/rust/qdl-kafka/src/phase92_runtime.rs new file mode 100644 index 0000000..75267b2 --- /dev/null +++ b/rust/qdl-kafka/src/phase92_runtime.rs @@ -0,0 +1,792 @@ +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::time::{Duration, Instant}; + +use futures_util::future::try_join_all; +use qdl_core::transport::{AppendResult, Cursor, DurableRecord}; +use qdl_venue_core::authority::{ + Phase92AuthorityControlEvent, Phase92AuthorityFence, Phase92AuthorityRecord, + Phase92PublicationContext, SinkTarget, +}; +use rdkafka::consumer::{BaseConsumer, Consumer, StreamConsumer}; +use rdkafka::message::{Header, Headers, Message, OwnedHeaders}; +use rdkafka::producer::{FutureProducer, FutureRecord, Producer}; +use rdkafka::topic_partition_list::{Offset, TopicPartitionList}; +use rdkafka::util::Timeout; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +use super::{ + transactional_output_headers, KafkaTransportConfig, KafkaTransportError, + TransactionalKafkaInput, EVENT_ID_HEADER, +}; + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92TransactionalTopics { + pub raw_inputs: Vec, + pub canary_canonical: String, + pub primary_canonical: String, + pub public_v2: String, + pub legacy_v1: String, + pub quarantine: String, + pub target_checkpoints: String, + pub authority_control: String, +} + +impl Phase92TransactionalTopics { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + let outputs = [ + self.canary_canonical.as_str(), + self.primary_canonical.as_str(), + self.public_v2.as_str(), + self.legacy_v1.as_str(), + self.quarantine.as_str(), + self.target_checkpoints.as_str(), + self.authority_control.as_str(), + ]; + if self.raw_inputs.is_empty() + || self.raw_inputs.iter().any(|topic| topic.trim().is_empty()) + || outputs.iter().any(|topic| topic.trim().is_empty()) + { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transactional topics must not be empty".into(), + )); + } + let mut unique = HashSet::new(); + for topic in self.raw_inputs.iter().map(String::as_str).chain(outputs) { + if !unique.insert(topic) { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transactional topics must be isolated and unique".into(), + )); + } + } + Ok(()) + } + + fn permits_output(&self, target: SinkTarget, stream: &str) -> bool { + match target { + SinkTarget::CanaryCanonical => stream == self.canary_canonical, + SinkTarget::PrimaryCanonical => { + stream == self.primary_canonical || stream == self.quarantine + } + SinkTarget::PublicV2 => stream == self.public_v2, + SinkTarget::LegacyV1 => stream == self.legacy_v1, + SinkTarget::ShadowRaw | SinkTarget::ShadowCanonical | SinkTarget::ShadowQuarantine => { + false + } + } + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Phase92Decision { + Canonical, + Quarantine, + Filtered, + Duplicate, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92Progress { + pub publication: Phase92PublicationContext, + pub decision: Phase92Decision, + pub source_cursor: Cursor, + pub source_event_id: Vec, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Phase92TransactionalOutput { + pub record: DurableRecord, + pub publication: Phase92PublicationContext, + pub raw_provider_envelope: Option>, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Phase92TargetCheckpoint { + pub schema: String, + pub slice_id: String, + pub owner_id: String, + pub authority_revision: u64, + pub lease_epoch: u64, + pub partition_plan_epoch: u64, + pub shard_id: String, + pub target: SinkTarget, + pub source_watermark: u64, + pub source_event_id: String, + pub decision: Phase92Decision, + pub output_payload_sha256: String, + pub candidate_digest: String, + pub committed_at_ns: i64, +} + +impl Phase92TargetCheckpoint { + pub fn validate(&self) -> Result<(), KafkaTransportError> { + if self.schema != "qdl.target-watermark-checkpoint.v1" + || self.slice_id.trim().is_empty() + || self.owner_id.trim().is_empty() + || self.authority_revision == 0 + || self.lease_epoch == 0 + || self.partition_plan_epoch == 0 + || self.shard_id.trim().is_empty() + || self.source_event_id.len() < 2 + || self.output_payload_sha256.len() != 64 + || !self + .output_payload_sha256 + .bytes() + .all(|byte| byte.is_ascii_hexdigit()) + || self.candidate_digest.len() != 64 + || !self + .candidate_digest + .bytes() + .all(|byte| byte.is_ascii_hexdigit()) + || self.committed_at_ns <= 0 + || !matches!( + self.target, + SinkTarget::CanaryCanonical + | SinkTarget::PrimaryCanonical + | SinkTarget::PublicV2 + | SinkTarget::LegacyV1 + ) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 target checkpoint is invalid".into(), + )); + } + Ok(()) + } + + pub fn key(&self) -> String { + format!( + "{}|{}|{}", + self.slice_id, + self.shard_id, + target_name(self.target) + ) + } + + pub fn publication(&self) -> Phase92PublicationContext { + Phase92PublicationContext { + slice_id: self.slice_id.clone(), + owner_id: self.owner_id.clone(), + authority_revision: self.authority_revision, + shard_id: self.shard_id.clone(), + lease_epoch: self.lease_epoch, + partition_plan_epoch: self.partition_plan_epoch, + source_watermark: self.source_watermark, + target: self.target, + } + } +} + +fn target_name(target: SinkTarget) -> &'static str { + match target { + SinkTarget::ShadowRaw => "SHADOW_RAW", + SinkTarget::ShadowCanonical => "SHADOW_CANONICAL", + SinkTarget::ShadowQuarantine => "SHADOW_QUARANTINE", + SinkTarget::CanaryCanonical => "CANARY_CANONICAL", + SinkTarget::PrimaryCanonical => "PRIMARY_CANONICAL", + SinkTarget::PublicV2 => "PUBLIC_V2", + SinkTarget::LegacyV1 => "LEGACY_V1", + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct CompactedKafkaRecord { + pub topic: String, + pub key: String, + pub payload: Vec, + pub event_id: Option>, + pub partition: i32, + pub offset: i64, + pub accepted_at_ns: i64, +} + +pub struct KafkaCompactedSnapshotReader { + config: KafkaTransportConfig, +} + +impl KafkaCompactedSnapshotReader { + pub fn new(config: KafkaTransportConfig) -> Result { + config.validate()?; + Ok(Self { config }) + } + + pub fn read(&self, topics: &[&str]) -> Result, KafkaTransportError> { + if topics.is_empty() || topics.iter().any(|topic| topic.trim().is_empty()) { + return Err(KafkaTransportError::Configuration( + "compacted snapshot topics must not be empty".into(), + )); + } + let mut client = self.config.client_config()?; + client + .set("group.id", &self.config.group_id) + .set("enable.auto.commit", "false") + .set("enable.auto.offset.store", "false") + .set("auto.offset.reset", "earliest") + .set("isolation.level", "read_committed") + .set("enable.partition.eof", "true"); + let consumer: BaseConsumer = client.create()?; + let timeout = Timeout::After(self.config.request_timeout); + let mut assignment = TopicPartitionList::new(); + let mut remaining = BTreeMap::new(); + for topic in topics { + let metadata = consumer.fetch_metadata(Some(topic), timeout)?; + let metadata_topic = metadata + .topics() + .iter() + .find(|value| value.name() == *topic) + .ok_or(KafkaTransportError::MissingField( + "compacted topic metadata", + ))?; + if metadata_topic.partitions().is_empty() { + return Err(KafkaTransportError::Configuration(format!( + "compacted topic has no partitions: {topic}" + ))); + } + for partition in metadata_topic.partitions() { + let partition_id = partition.id(); + let (low, high) = consumer.fetch_watermarks(topic, partition_id, timeout)?; + assignment.add_partition_offset(topic, partition_id, Offset::Beginning)?; + if high > low { + remaining.insert(((*topic).to_owned(), partition_id), high - 1); + } + } + } + consumer.assign(&assignment)?; + + let deadline = Instant::now() + self.config.request_timeout; + let mut latest: HashMap<(String, String), CompactedKafkaRecord> = HashMap::new(); + while !remaining.is_empty() { + if Instant::now() >= deadline { + return Err(KafkaTransportError::Configuration( + "compacted snapshot did not reach captured high watermarks".into(), + )); + } + let Some(result) = consumer.poll(Duration::from_millis(100)) else { + continue; + }; + let message = match result { + Ok(message) => message, + Err(rdkafka::error::KafkaError::PartitionEOF(_)) => continue, + Err(error) => return Err(error.into()), + }; + let topic = message.topic().to_owned(); + let partition = message.partition(); + let key_bytes = message + .key() + .ok_or(KafkaTransportError::MissingField("compacted record key"))?; + let key = std::str::from_utf8(key_bytes) + .map_err(|_| KafkaTransportError::InvalidUtf8("compacted record key"))? + .to_owned(); + let identity = (topic.clone(), key.clone()); + if let Some(payload) = message.payload() { + let event_id = message.headers().and_then(|headers| { + headers + .iter() + .find(|header| header.key == EVENT_ID_HEADER) + .and_then(|header| header.value.map(ToOwned::to_owned)) + }); + latest.insert( + identity, + CompactedKafkaRecord { + topic: topic.clone(), + key, + payload: payload.to_vec(), + event_id, + partition, + offset: message.offset(), + accepted_at_ns: message.timestamp().to_millis().unwrap_or_default() + * 1_000_000, + }, + ); + } else { + latest.remove(&identity); + } + if remaining + .get(&(topic.clone(), partition)) + .is_some_and(|high| message.offset() >= *high) + { + remaining.remove(&(topic, partition)); + } + } + let mut records: Vec<_> = latest.into_values().collect(); + records.sort_by(|left, right| { + (&left.topic, &left.key, left.partition, left.offset).cmp(&( + &right.topic, + &right.key, + right.partition, + right.offset, + )) + }); + Ok(records) + } +} + +#[derive(Debug)] +pub struct Phase92CommitResult { + pub outputs: Vec, + pub checkpoints: Vec, +} + +pub struct Phase92TransactionalKafkaBridge { + producer: FutureProducer, + consumer: StreamConsumer, + fences: tokio::sync::Mutex>, + topics: Phase92TransactionalTopics, + request_timeout: Duration, +} + +impl Phase92TransactionalKafkaBridge { + pub fn new( + config: &KafkaTransportConfig, + topics: Phase92TransactionalTopics, + transactional_id: &str, + ) -> Result { + topics.validate()?; + if transactional_id.trim().is_empty() { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transactional.id must not be empty".into(), + )); + } + let mut consumer_config = config.client_config()?; + consumer_config + .set("group.id", &config.group_id) + .set("enable.auto.commit", "false") + .set("enable.auto.offset.store", "false") + .set("auto.offset.reset", "earliest") + .set("isolation.level", "read_committed"); + let consumer: StreamConsumer = consumer_config.create()?; + let raw_topics: Vec<&str> = topics.raw_inputs.iter().map(String::as_str).collect(); + consumer.subscribe(&raw_topics)?; + + let mut producer_config = config.client_config()?; + producer_config + .set("transactional.id", transactional_id) + .set("enable.idempotence", "true") + .set("acks", "all") + .set("max.in.flight.requests.per.connection", "5") + .set("retries", "2147483647") + .set("compression.type", "zstd") + .set( + "transaction.timeout.ms", + config.request_timeout.as_millis().to_string(), + ) + .set( + "delivery.timeout.ms", + config.request_timeout.as_millis().to_string(), + ); + let producer: FutureProducer = producer_config.create()?; + producer.init_transactions(Timeout::After(config.request_timeout))?; + Ok(Self { + producer, + consumer, + fences: tokio::sync::Mutex::new(HashMap::new()), + topics, + request_timeout: config.request_timeout, + }) + } + + pub async fn apply_authority_event( + &self, + event: &Phase92AuthorityControlEvent, + now_ns: i64, + ) -> Result<(), KafkaTransportError> { + let mut fences = self.fences.lock().await; + fences + .entry(event.slice_id.clone()) + .or_default() + .apply_control_event(event, now_ns) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn restore_checkpoint( + &self, + checkpoint: &Phase92TargetCheckpoint, + ) -> Result<(), KafkaTransportError> { + checkpoint.validate()?; + self.fences + .lock() + .await + .get_mut(&checkpoint.slice_id) + .ok_or_else(|| { + KafkaTransportError::Fencing("checkpoint slice authority is not loaded".into()) + })? + .restore_committed_watermark(&checkpoint.publication()) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn current_authority(&self, slice_id: &str) -> Option { + self.fences + .lock() + .await + .get(slice_id) + .and_then(|fence| fence.current().cloned()) + } + + pub async fn next_watermark( + &self, + slice_id: &str, + shard_id: &str, + target: SinkTarget, + ) -> Result { + self.fences + .lock() + .await + .get(slice_id) + .ok_or_else(|| { + KafkaTransportError::Fencing("publication slice authority is not loaded".into()) + })? + .next_watermark(shard_id, target) + .map_err(KafkaTransportError::Fencing) + } + + pub async fn next(&self) -> Result { + let message = self.consumer.recv().await?; + let payload = message + .payload() + .ok_or(KafkaTransportError::MissingField("payload"))? + .to_vec(); + let key = message + .key() + .ok_or(KafkaTransportError::MissingField("partition_key"))?; + let partition_key = std::str::from_utf8(key) + .map_err(|_| KafkaTransportError::InvalidUtf8("partition_key"))? + .to_owned(); + let event_id = message + .headers() + .and_then(|headers| { + headers + .iter() + .find(|header| header.key == EVENT_ID_HEADER) + .and_then(|header| header.value.map(ToOwned::to_owned)) + }) + .ok_or(KafkaTransportError::MissingField("event_id header"))?; + let offset = u64::try_from(message.offset()) + .map_err(|_| KafkaTransportError::InvalidOffset(message.offset()))?; + Ok(TransactionalKafkaInput { + record: DurableRecord { + stream: message.topic().to_owned(), + partition_key: partition_key.clone(), + event_id, + payload, + accepted_at_ns: message.timestamp().to_millis().unwrap_or_default() * 1_000_000, + }, + cursor: Cursor { + stream: message.topic().to_owned(), + transport_partition: message.partition(), + partition_key, + offset, + }, + }) + } + + pub async fn commit( + &self, + inputs: &[TransactionalKafkaInput], + outputs: &[Phase92TransactionalOutput], + progress: &[Phase92Progress], + now_ns: i64, + ) -> Result { + if inputs.is_empty() || progress.is_empty() || now_ns <= 0 { + return Err(KafkaTransportError::Configuration( + "Phase 9.2 transaction requires input, progress and time".into(), + )); + } + if inputs + .iter() + .any(|input| !self.topics.raw_inputs.contains(&input.cursor.stream)) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 transaction input is outside raw topics".into(), + )); + } + let mut progress_identities = HashSet::new(); + for item in progress { + let identity = ( + item.publication.slice_id.clone(), + item.publication.shard_id.clone(), + item.publication.target, + item.publication.source_watermark, + ); + if !progress_identities.insert(identity) { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 transaction has duplicate target progress".into(), + )); + } + if item.source_event_id.is_empty() + || !inputs.iter().any(|input| { + input.cursor == item.source_cursor + && input.record.event_id == item.source_event_id + }) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 progress has no exact matching raw input".into(), + )); + } + } + for output in outputs { + if !self + .topics + .permits_output(output.publication.target, &output.record.stream) + || !progress + .iter() + .any(|item| item.publication == output.publication) + { + return Err(KafkaTransportError::Fencing( + "Phase 9.2 output target/topic/progress binding failed".into(), + )); + } + } + + let mut fences = self.fences.lock().await; + let mut next_fences = fences.clone(); + for item in progress { + let next_fence = next_fences + .get_mut(&item.publication.slice_id) + .ok_or_else(|| { + KafkaTransportError::Fencing("progress slice authority is not loaded".into()) + })?; + next_fence + .permits(&item.publication, now_ns) + .map_err(KafkaTransportError::Fencing)?; + next_fence + .commit(&item.publication) + .map_err(KafkaTransportError::Fencing)?; + } + let checkpoints = progress + .iter() + .map(|item| { + let authority = next_fences + .get(&item.publication.slice_id) + .and_then(Phase92AuthorityFence::current) + .ok_or_else(|| { + KafkaTransportError::Fencing( + "checkpoint slice authority is not loaded".into(), + ) + })?; + let mut output_digest = Sha256::new(); + for output in outputs + .iter() + .filter(|output| output.publication == item.publication) + { + output_digest.update( + u64::try_from(output.record.payload.len()) + .map_err(|_| { + KafkaTransportError::Configuration( + "output payload length overflow".into(), + ) + })? + .to_be_bytes(), + ); + output_digest.update(&output.record.payload); + } + let output_payload_sha256 = hex::encode(output_digest.finalize()); + let checkpoint = Phase92TargetCheckpoint { + schema: "qdl.target-watermark-checkpoint.v1".into(), + slice_id: item.publication.slice_id.clone(), + owner_id: item.publication.owner_id.clone(), + authority_revision: item.publication.authority_revision, + lease_epoch: item.publication.lease_epoch, + partition_plan_epoch: item.publication.partition_plan_epoch, + shard_id: item.publication.shard_id.clone(), + target: item.publication.target, + source_watermark: item.publication.source_watermark, + source_event_id: hex::encode(&item.source_event_id), + decision: item.decision, + output_payload_sha256, + candidate_digest: authority.candidate_digest.clone(), + committed_at_ns: now_ns, + }; + checkpoint.validate()?; + let payload = serde_json::to_vec(&checkpoint).map_err(|error| { + KafkaTransportError::Configuration(format!( + "target checkpoint serialization failed: {error}" + )) + })?; + let event_id = Sha256::digest(&payload).to_vec(); + Ok::<_, KafkaTransportError>((checkpoint, payload, event_id)) + }) + .collect::, _>>()?; + + self.producer.begin_transaction()?; + let transaction = async { + let output_deliveries = outputs.iter().map(|output| async { + let headers = transactional_output_headers( + output.record.event_id.as_slice(), + output.raw_provider_envelope.as_deref(), + ); + deliver( + &self.producer, + &output.record.stream, + output.record.partition_key.as_bytes(), + output.record.payload.as_slice(), + headers, + self.request_timeout, + ) + .await + }); + let accepted_outputs = try_join_all(output_deliveries).await?; + + let checkpoint_deliveries = + checkpoints + .iter() + .map(|(checkpoint, payload, event_id)| async { + let headers = OwnedHeaders::new().insert(Header { + key: EVENT_ID_HEADER, + value: Some(event_id.as_slice()), + }); + deliver( + &self.producer, + &self.topics.target_checkpoints, + checkpoint.key().as_bytes(), + payload.as_slice(), + headers, + self.request_timeout, + ) + .await + }); + let accepted_checkpoints = try_join_all(checkpoint_deliveries).await?; + + let mut next_offsets: BTreeMap<(String, i32), i64> = BTreeMap::new(); + for input in inputs { + let next_offset = input + .cursor + .offset + .checked_add(1) + .and_then(|value| i64::try_from(value).ok()) + .ok_or(KafkaTransportError::InvalidOffset(i64::MAX))?; + next_offsets + .entry(( + input.cursor.stream.clone(), + input.cursor.transport_partition, + )) + .and_modify(|current| *current = (*current).max(next_offset)) + .or_insert(next_offset); + } + let mut offsets = TopicPartitionList::new(); + for ((topic, partition), offset) in next_offsets { + offsets.add_partition_offset(&topic, partition, Offset::Offset(offset))?; + } + let group = self.consumer.group_metadata().ok_or_else(|| { + KafkaTransportError::Configuration( + "Phase 9.2 consumer group metadata is unavailable".into(), + ) + })?; + self.producer.send_offsets_to_transaction( + &offsets, + &group, + Timeout::After(self.request_timeout), + )?; + self.producer + .commit_transaction(Timeout::After(self.request_timeout))?; + Ok::<_, KafkaTransportError>(Phase92CommitResult { + outputs: accepted_outputs, + checkpoints: accepted_checkpoints, + }) + } + .await; + match transaction { + Ok(result) => { + *fences = next_fences; + Ok(result) + } + Err(error) => { + self.producer + .abort_transaction(Timeout::After(self.request_timeout)) + .map_err(KafkaTransportError::Kafka)?; + Err(error) + } + } + } +} + +async fn deliver( + producer: &FutureProducer, + topic: &str, + key: &[u8], + payload: &[u8], + headers: OwnedHeaders, + timeout: Duration, +) -> Result { + let delivery = producer + .send( + FutureRecord::to(topic) + .key(key) + .payload(payload) + .headers(headers), + Timeout::After(timeout), + ) + .await + .map_err(|(error, _)| KafkaTransportError::Delivery(error))?; + let offset = u64::try_from(delivery.offset) + .map_err(|_| KafkaTransportError::InvalidOffset(delivery.offset))?; + Ok(AppendResult { + cursor: Cursor { + stream: topic.to_owned(), + transport_partition: delivery.partition, + partition_key: std::str::from_utf8(key) + .map_err(|_| KafkaTransportError::InvalidUtf8("output key"))? + .to_owned(), + offset, + }, + duplicate: false, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn topics() -> Phase92TransactionalTopics { + Phase92TransactionalTopics { + raw_inputs: vec!["raw".into()], + canary_canonical: "canary".into(), + primary_canonical: "canonical".into(), + public_v2: "public".into(), + legacy_v1: "legacy".into(), + quarantine: "quarantine".into(), + target_checkpoints: "checkpoints".into(), + authority_control: "authority".into(), + } + } + + #[test] + fn production_topics_are_unique_and_target_bound() { + let values = topics(); + values.validate().unwrap(); + assert!(values.permits_output(SinkTarget::PrimaryCanonical, "canonical")); + assert!(values.permits_output(SinkTarget::PrimaryCanonical, "quarantine")); + assert!(values.permits_output(SinkTarget::PublicV2, "public")); + assert!(values.permits_output(SinkTarget::LegacyV1, "legacy")); + assert!(values.permits_output(SinkTarget::CanaryCanonical, "canary")); + let mut duplicate = values; + duplicate.public_v2 = "canonical".into(); + assert!(duplicate.validate().is_err()); + } + + #[test] + fn checkpoint_roundtrip_preserves_exact_identity() { + let value = Phase92TargetCheckpoint { + schema: "qdl.target-watermark-checkpoint.v1".into(), + slice_id: "production/binance/usdm/perpetual/trade/plan-1/btcusdt".into(), + owner_id: "rust-primary".into(), + authority_revision: 4, + lease_epoch: 2, + partition_plan_epoch: 1, + shard_id: "core-1".into(), + target: SinkTarget::PublicV2, + source_watermark: 101, + source_event_id: "00".repeat(16), + decision: Phase92Decision::Canonical, + output_payload_sha256: "1".repeat(64), + candidate_digest: "2".repeat(64), + committed_at_ns: 1, + }; + value.validate().unwrap(); + let decoded: Phase92TargetCheckpoint = + serde_json::from_slice(&serde_json::to_vec(&value).unwrap()).unwrap(); + assert_eq!(decoded, value); + assert!(decoded.key().ends_with("|PUBLIC_V2")); + } +} diff --git a/rust/qdl-venue-core/src/authority.rs b/rust/qdl-venue-core/src/authority.rs index ef4484f..512824e 100644 --- a/rust/qdl-venue-core/src/authority.rs +++ b/rust/qdl-venue-core/src/authority.rs @@ -738,7 +738,7 @@ pub struct Phase92PublicationContext { pub target: SinkTarget, } -#[derive(Default)] +#[derive(Clone, Default)] pub struct Phase92AuthorityFence { current: Option, committed_watermarks: HashMap<(String, SinkTarget), u64>, @@ -891,26 +891,26 @@ impl Phase92AuthorityFence { &mut self, context: &Phase92PublicationContext, ) -> Result<(), String> { - if !self.recovery_required { - return Err("Phase 9.2 watermark restore is only permitted during recovery".into()); - } let current = self .current .as_ref() .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; - if !matches!( + let restoring_canary = current.state == Phase92AuthorityState::RustCanary + && context.target == SinkTarget::CanaryCanonical; + let restoring_primary = matches!( current.state, Phase92AuthorityState::RustPrimary | Phase92AuthorityState::PythonPrimary - ) || context.slice_id != current.slice_id + ) && matches!( + context.target, + SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 + ); + if !(restoring_canary || (self.recovery_required && restoring_primary)) + || context.slice_id != current.slice_id || context.owner_id != current.owner_id || context.authority_revision != current.authority_revision || context.lease_epoch != current.lease_epoch || context.partition_plan_epoch != current.partition_plan_epoch || context.shard_id.trim().is_empty() - || !matches!( - context.target, - SinkTarget::PrimaryCanonical | SinkTarget::PublicV2 | SinkTarget::LegacyV1 - ) || context.source_watermark < current.start_watermark { return Err("Phase 9.2 recovered watermark identity is invalid".into()); @@ -1012,6 +1012,22 @@ impl Phase92AuthorityFence { Ok(()) } + pub fn next_watermark(&self, shard_id: &str, target: SinkTarget) -> Result { + let current = self + .current + .as_ref() + .ok_or_else(|| "Phase 9.2 authority record is not loaded".to_owned())?; + if shard_id.trim().is_empty() { + return Err("Phase 9.2 shard identity is empty".into()); + } + self.committed_watermarks + .get(&(shard_id.to_owned(), target)) + .copied() + .unwrap_or(current.start_watermark) + .checked_add(1) + .ok_or_else(|| "Phase 9.2 watermark overflow".to_owned()) + } + pub fn current(&self) -> Option<&Phase92AuthorityRecord> { self.current.as_ref() } diff --git a/scripts/build_production_core_bundle.py b/scripts/build_production_core_bundle.py new file mode 100755 index 0000000..1c97e08 --- /dev/null +++ b/scripts/build_production_core_bundle.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import ( + StableAcquisitionPlan, + write_production_core_bundle, +) + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Build deterministic Phase C Rust production-core configs." + ) + parser.add_argument("--source-catalog", type=Path, required=True) + parser.add_argument("--acquisition-plan", type=Path, required=True) + parser.add_argument("--raw-authority", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--partition-plan-epoch", type=int, default=1) + args = parser.parse_args() + + catalog = StableSourceCatalog.load(args.source_catalog) + acquisition = StableAcquisitionPlan.load( + args.acquisition_plan, catalog=catalog + ) + authority = json.loads(args.raw_authority.read_text(encoding="utf-8")) + digests = write_production_core_bundle( + args.output_dir, + catalog=catalog, + acquisition=acquisition, + raw_authority=authority, + partition_plan_epoch=args.partition_plan_epoch, + ) + print(json.dumps({ + "schema": "qdl.v2.production-core-build-result.v1", + "status": "PASS", + "output_dir": str(args.output_dir.resolve()), + "digests": digests, + }, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_production_catalog.py b/tests/test_production_catalog.py index 0705998..56525ed 100644 --- a/tests/test_production_catalog.py +++ b/tests/test_production_catalog.py @@ -11,6 +11,7 @@ ProductionDemandManifest, ) from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import StableAcquisitionPlan BINANCE = { @@ -155,6 +156,36 @@ def test_generator_is_deterministic_validated_and_metadata_authentic(self): self.assertEqual( json.loads(Path(paths["provenance"]).read_text())["binding_count"], 2 ) + acquisition = StableAcquisitionPlan.load( + paths["acquisition_plan"], catalog=catalog + ) + raw_authority = { + "schema": "qdl.authority-record.v1", + "slice_id": "qdl-v2-production-acquisition", + "revision": 11, + "mode": "RUST_SHADOW", + "candidate_image_digest": "sha256:" + "1" * 64, + "capability_manifest_digest": "2" * 64, + "contract_digest": "3" * 64, + "partition_plan_digest": "4" * 64, + "public_write_allowed": False, + "legacy_write_allowed": False, + "approved_by": "production-catalog-test", + "effective_at_ns": 1, + } + runtime = acquisition.production_core_config( + catalog=catalog, + raw_authority=raw_authority, + worker_index=1, + ) + self.assertEqual(len(runtime["slices"]), 2) + self.assertEqual( + {item["subscription_id"] for item in runtime["slices"]}, + {item.source_id for item in catalog.bindings}, + ) + self.assertEqual(runtime["topics"]["primary_canonical"], "md.canonical.v2") + self.assertEqual(runtime["topics"]["authority_control"], "qdl.authority.v1") + self.assertEqual(runtime["batch_size"], 128) def test_conflicts_missing_metadata_and_uncertified_interval_fail_closed(self): with tempfile.TemporaryDirectory() as directory: From e5d48530d75de7b906d2f28ab55d67117ef20fc4 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 10:20:19 +0000 Subject: [PATCH 08/21] fix(sdk): support declared Python 3.10 runtime --- .github/workflows/ci.yml | 18 ++++++++++++++++++ DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 12 ++++++++++++ qdl_sdk/models.py | 9 ++++++++- 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a030373..b364703 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -78,6 +78,24 @@ jobs: tar --no-same-owner -xzf "${archive}" "./${directory}/cargo-deny" check + sdk-python310: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + + - name: Set up declared minimum Python + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + with: + python-version: "3.10" + + - name: Build and import standalone SDK wheel + run: | + python scripts/build_qdl_sdk_release.py --output-dir /tmp/qdl-sdk + python -m pip install --disable-pip-version-check /tmp/qdl-sdk/qdl_sdk-2.0.0-py3-none-any.whl + cd "$(mktemp -d)" + PYTHONPATH= python -c 'import qdl_sdk; assert qdl_sdk.__version__ == "2.0.0"' + unit-tests: runs-on: ubuntu-latest steps: diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 0f0be5b..e860127 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5965,6 +5965,18 @@ catalog or SDK wiring cannot be deferred as operational debt. filtered, duplicate and quarantine decisions, crash/restart, compacted replay, active authority change and rollback. This slice cannot change port 8100, production routes, topics, consumers or authority. +- `2026-08-20 C.0 SDK PYTHON 3.10 BLOCKER PASS`: Trading System consumer + acceptance imported the prior immutable SDK wheel under its declared minimum + Python 3.10 runtime and found `enum.StrEnum` was Python 3.11-only. The fix is + owned by `qdl_sdk.models`, not patched in the consumer: Python 3.11+ uses the + standard enum and Python 3.10 uses an equivalent `str, Enum` compatibility + type. Added a dedicated CI job that builds and imports the standalone wheel + on Python 3.10 outside the source tree. Two release builds were byte-identical + at SHA-256 `3ea8f7e8b58f6c5ea1b2aa66ee94157f949d4cf6a71d708cb7508ed3b0abc600`; + an actual Python 3.10 wheel import passed and 17/17 SDK release/stream tests + passed on the Data Layer Python 3.12 runtime. Trading System updated its + vendor manifest/lock to that exact digest. V1 runtime, providers, authority, + Redis/PostgreSQL and consumer routes were unchanged. - `2026-08-20 C.0 LONG-RUNNING PRIMARY BRIDGE CODE PASS`: added a separate multi-slice `qdl-production-core` binary and Phase 9.2 transactional bridge. Authority is reconstructed per slice from the compacted control topic; raw diff --git a/qdl_sdk/models.py b/qdl_sdk/models.py index 130dd11..d32088b 100644 --- a/qdl_sdk/models.py +++ b/qdl_sdk/models.py @@ -1,9 +1,16 @@ from __future__ import annotations from dataclasses import dataclass -from enum import StrEnum +from enum import Enum from typing import Annotated, Any, Literal +try: # Python 3.11+; the released SDK contract supports Python 3.10. + from enum import StrEnum +except ImportError: # pragma: no cover - exercised by the Python 3.10 artifact gate. + class StrEnum(str, Enum): + def __str__(self) -> str: + return self.value + from pydantic import BaseModel, ConfigDict, Field, model_validator from qdl.query.v2 import query_pb2 From da79c408a8e7405bbfd902aeb88d07bb5734e072 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 10:28:55 +0000 Subject: [PATCH 09/21] feat(sdk): project canonical V2 stream events --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 12 + qdl_sdk/__init__.py | 2 + qdl_sdk/projection.py | 381 ++++++++++++++++++++++ tests/test_qdl_sdk_stream_projection.py | 325 ++++++++++++++++++ 4 files changed, 720 insertions(+) create mode 100644 qdl_sdk/projection.py create mode 100644 tests/test_qdl_sdk_stream_projection.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index e860127..244250c 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5977,6 +5977,18 @@ catalog or SDK wiring cannot be deferred as operational debt. passed on the Data Layer Python 3.12 runtime. Trading System updated its vendor manifest/lock to that exact digest. V1 runtime, providers, authority, Redis/PostgreSQL and consumer routes were unchanged. +- `2026-08-20 C.0 SDK STREAM PROJECTION PASS`: added one SDK-owned + canonical protobuf-to-typed-view decoder, so Trading System and alpha + consumers do not copy schema logic. It covers TRADE, QUOTE, BAR, book + snapshot/delta, funding, open interest, mark/index and ticker payloads with + exact coefficient/scale decimals, enums and optional bytes. The signed query + handoff remains the policy/catalog template; instrument/source transitions, + lower authority revision, stale execution data, open gap, incomplete + contract metadata and non-final execution bars fail closed. Freshness, + quality and execution eligibility are recomputed per event and the signed + cursor/watermark is preserved. All-feed projection, source/revision, gap and + stale tests plus existing SDK release/stream tests passed 20/20; isolated + lint and `git diff --check` passed. No runtime or provider was touched. - `2026-08-20 C.0 LONG-RUNNING PRIMARY BRIDGE CODE PASS`: added a separate multi-slice `qdl-production-core` binary and Phase 9.2 transactional bridge. Authority is reconstructed per slice from the compacted control topic; raw diff --git a/qdl_sdk/__init__.py b/qdl_sdk/__init__.py index 0c88b41..2270434 100644 --- a/qdl_sdk/__init__.py +++ b/qdl_sdk/__init__.py @@ -29,6 +29,7 @@ TradeIdentityKind, WarmupResponse, ) +from qdl_sdk.projection import market_data_view_from_stream from qdl_sdk.transport import GrpcStreamTransport, RestQueryTransport from qdl_sdk.v1_facade import V1CompatibilityFacade @@ -53,6 +54,7 @@ "InstrumentView", "MemoryCursorStore", "MarketDataView", + "market_data_view_from_stream", "QuantityUnit", "RestQueryTransport", "RecoveryPolicy", diff --git a/qdl_sdk/projection.py b/qdl_sdk/projection.py new file mode 100644 index 0000000..c2956aa --- /dev/null +++ b/qdl_sdk/projection.py @@ -0,0 +1,381 @@ +"""SDK-owned canonical protobuf to typed V2 view projection.""" + +from __future__ import annotations + +import time +from typing import Any + +from qdl.common.v1 import common_pb2 +from qdl.marketdata.v2 import market_data_pb2 +from qdl_sdk.errors import ContinuityError +from qdl_sdk.models import DataRequirement, Feed, MarketDataView, StreamEvent + +_GAP_FLAGS = {"SEQUENCE_GAP_BEFORE", "OUT_OF_ORDER", "RESYNC_REQUIRED"} + + +def _enum_name(enum_wrapper: Any, value: int, prefix: str) -> str: + try: + name = enum_wrapper.Name(value) + except ValueError as error: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", f"unknown canonical enum value {value}" + ) from error + result = name.removeprefix(prefix) + if result == "UNSPECIFIED": + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", f"canonical enum {prefix} is unspecified" + ) + return result + + +def _decimal(value) -> dict[str, str | int]: + selected = value.WhichOneof("coefficient") + if selected == "mantissa": + coefficient = str(value.mantissa) + elif selected == "mantissa_text": + coefficient = value.mantissa_text + else: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "canonical decimal coefficient is missing" + ) + return { + "coefficient": coefficient, + "scale": int(value.scale), + "source_text": value.source_text, + } + + +def _quantity_unit(value: int) -> str: + return _enum_name(common_pb2.QuantityUnit, value, "QUANTITY_UNIT_") + + +def _book_level(value) -> dict[str, Any]: + return { + "side": _enum_name(common_pb2.BookSide, value.side, "BOOK_SIDE_"), + "price": _decimal(value.price), + "quantity": _decimal(value.quantity), + "quantity_unit": _quantity_unit(value.quantity_unit), + "order_count": int(value.order_count), + } + + +def _payload( + envelope: market_data_pb2.EventEnvelope, +) -> tuple[Feed, str | None, int, dict[str, Any]]: + name = envelope.WhichOneof("payload") + if name == "trade": + side = common_pb2.AggressorSide.Name( + envelope.trade.aggressor_side + ).removeprefix("AGGRESSOR_SIDE_") + if side == "UNSPECIFIED": + side = "UNKNOWN" + identity = _enum_name( + market_data_pb2.TradeIdentityKind, + envelope.trade.identity_kind, + "TRADE_IDENTITY_KIND_", + ) + return ( + Feed.TRADE, + None, + 0, + { + "feed": "TRADE", + "native_trade_id": envelope.trade.native_trade_id, + "price": _decimal(envelope.trade.price), + "quantity": _decimal(envelope.trade.quantity), + "quantity_unit": _quantity_unit(envelope.trade.quantity_unit), + "aggressor_side": side, + "identity_kind": identity, + "is_block_trade": bool(envelope.trade.is_block_trade), + "is_buyer_maker": bool(envelope.trade.is_buyer_maker), + }, + ) + if name == "quote": + return ( + Feed.QUOTE, + None, + 0, + { + "feed": "QUOTE", + "bid_price": _decimal(envelope.quote.bid_price), + "bid_quantity": _decimal(envelope.quote.bid_quantity), + "ask_price": _decimal(envelope.quote.ask_price), + "ask_quantity": _decimal(envelope.quote.ask_quantity), + "quantity_unit": _quantity_unit(envelope.quote.quantity_unit), + "level": int(envelope.quote.level), + }, + ) + if name == "bar": + lifecycle = _enum_name( + market_data_pb2.BarLifecycle, + envelope.bar.lifecycle, + "BAR_LIFECYCLE_", + ) + payload: dict[str, Any] = { + "feed": "BAR", + "interval": envelope.bar.interval, + "open_time_ns": int(envelope.bar.open_time_ns), + "close_time_ns": int(envelope.bar.close_time_ns), + "open": _decimal(envelope.bar.open), + "high": _decimal(envelope.bar.high), + "low": _decimal(envelope.bar.low), + "close": _decimal(envelope.bar.close), + "volume": _decimal(envelope.bar.volume), + "volume_unit": _quantity_unit(envelope.bar.volume_unit), + "trade_count": int(envelope.bar.trade_count), + "lifecycle": lifecycle, + "revision": int(envelope.bar.revision), + "origin": _enum_name( + common_pb2.BarOrigin, envelope.bar.origin, "BAR_ORIGIN_" + ), + "supersedes_event_id": ( + bytes(envelope.bar.supersedes_event_id).hex() + if envelope.bar.HasField("supersedes_event_id") + else None + ), + } + for field in ("base_volume", "quote_volume", "contract_volume"): + payload[field] = ( + _decimal(getattr(envelope.bar, field)) + if envelope.bar.HasField(field) + else None + ) + return Feed.BAR, envelope.bar.interval, int(envelope.bar.revision), payload + if name == "book_snapshot": + return ( + Feed.BOOK_SNAPSHOT, + None, + 0, + { + "feed": "BOOK_SNAPSHOT", + "native_sequence": envelope.book_snapshot.native_sequence, + "checksum": envelope.book_snapshot.checksum or None, + "levels": [_book_level(item) for item in envelope.book_snapshot.levels], + "depth": int(envelope.book_snapshot.depth), + }, + ) + if name == "book_delta": + return ( + Feed.BOOK_DELTA, + None, + 0, + { + "feed": "BOOK_DELTA", + "native_sequence_start": envelope.book_delta.native_sequence_start, + "native_sequence_end": envelope.book_delta.native_sequence_end, + "snapshot_sequence": envelope.book_delta.snapshot_sequence, + "checksum": envelope.book_delta.checksum or None, + "updates": [_book_level(item) for item in envelope.book_delta.updates], + "reset": bool(envelope.book_delta.reset), + }, + ) + if name == "funding_rate": + return ( + Feed.FUNDING_RATE, + None, + 0, + { + "feed": "FUNDING_RATE", + "rate": _decimal(envelope.funding_rate.rate), + "funding_time_ns": int(envelope.funding_rate.funding_time_ns), + "next_funding_time_ns": ( + int(envelope.funding_rate.next_funding_time_ns) + if envelope.funding_rate.HasField("next_funding_time_ns") + else None + ), + }, + ) + if name == "open_interest": + return ( + Feed.OPEN_INTEREST, + None, + 0, + { + "feed": "OPEN_INTEREST", + "quantity": _decimal(envelope.open_interest.quantity), + "notional": ( + _decimal(envelope.open_interest.notional) + if envelope.open_interest.HasField("notional") + else None + ), + "quantity_unit": _quantity_unit(envelope.open_interest.quantity_unit), + }, + ) + if name == "mark_index_price": + return ( + Feed.MARK_INDEX_PRICE, + None, + 0, + { + "feed": "MARK_INDEX_PRICE", + "mark_price": _decimal(envelope.mark_index_price.mark_price), + "index_price": _decimal(envelope.mark_index_price.index_price), + }, + ) + if name == "ticker": + payload = {"feed": "TICKER", "last_price": _decimal(envelope.ticker.last_price)} + for field in ("last_quantity", "open_24h", "high_24h", "low_24h", "volume_24h"): + payload[field] = ( + _decimal(getattr(envelope.ticker, field)) + if envelope.ticker.HasField(field) + else None + ) + payload["last_quantity_unit"] = ( + _quantity_unit(envelope.ticker.last_quantity_unit) + if envelope.ticker.HasField("last_quantity") + else None + ) + payload["volume_24h_unit"] = ( + _quantity_unit(envelope.ticker.volume_24h_unit) + if envelope.ticker.HasField("volume_24h") + else None + ) + return Feed.TICKER, None, 0, payload + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", + f"unsupported canonical stream payload: {name or 'none'}", + ) + + +def market_data_view_from_stream( + event: StreamEvent, + *, + template: MarketDataView, + requirement: DataRequirement, + now_ns: int | None = None, +) -> MarketDataView: + """Project one SDK stream event through its authoritative query handoff. + + The query template owns source policy/catalog metadata. Any stream identity + or source transition requires a new snapshot and fails closed here. + """ + + envelope = event.event + if not isinstance(envelope, market_data_pb2.EventEnvelope): + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "stream event is not a canonical V2 envelope" + ) + if envelope.schema_major != 2 or not envelope.schema_name: + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "stream envelope schema is not V2" + ) + if ( + envelope.instrument_uid != requirement.instrument_uid + or envelope.instrument_uid != template.instrument_uid + ): + raise ContinuityError("CONFLICT", "stream instrument UID differs from handoff") + if int(envelope.instrument_revision) != template.instrument_revision: + raise ContinuityError( + "CONFLICT", "stream instrument revision differs from handoff" + ) + if ( + envelope.venue.upper() != template.source.venue.upper() + or envelope.provider != template.source.provider + or envelope.source_id != template.source.source_id + ): + raise ContinuityError( + "SOURCE_NON_AUTHORITATIVE", + "stream source transition requires a new snapshot", + ) + feed, interval, revision, payload = _payload(envelope) + if feed is not requirement.feed or interval != requirement.interval: + raise ContinuityError( + "CONFLICT", "stream feed/interval differs from requirement" + ) + if requirement.source_policy_id != template.quality.policy_id: + raise ContinuityError( + "CONFLICT", "stream handoff source policy differs from requirement" + ) + if int(envelope.authority_revision) < template.contract.authority_revision: + raise ContinuityError( + "SOURCE_NON_AUTHORITATIVE", "stream authority revision moved backwards" + ) + if ( + not envelope.normalizer_version + or not envelope.adapter_version + or envelope.config_revision < 1 + ): + raise ContinuityError( + "SCHEMA_NOT_SUPPORTED", "stream contract metadata is incomplete" + ) + + flags = [ + common_pb2.QualityFlag.Name(value).removeprefix("QUALITY_FLAG_") + for value in envelope.quality_flags + ] + gap_open = any(value in _GAP_FLAGS for value in flags) + observed_at_ns = int(envelope.source_event_time_ns) + if feed is Feed.BAR: + observed_for_freshness = int(envelope.bar.close_time_ns) + else: + observed_for_freshness = observed_at_ns + freshness_ms = max( + 0, ((now_ns or time.time_ns()) - observed_for_freshness) // 1_000_000 + ) + stale = ( + requirement.max_freshness_ms is not None + and freshness_ms > requirement.max_freshness_ms + ) + state = "GAPPED" if gap_open else "STALE" if stale else "LIVE" + authoritative = ( + template.source.authoritative + and template.source.source_role == "PRIMARY" + and state == "LIVE" + ) + if requirement.consumer_grade.value == "EXECUTION" and not authoritative: + code = ( + "OPEN_SEQUENCE_GAP" + if gap_open + else "DATA_STALE" + if stale + else "SOURCE_NON_AUTHORITATIVE" + ) + raise ContinuityError(code, "stream event is not execution eligible") + if ( + feed is Feed.BAR + and requirement.require_final_bars + and payload["lifecycle"] not in {"FINAL", "REVISED"} + ): + raise ContinuityError("DATA_NOT_READY", "stream bar is not final") + + return MarketDataView.model_validate( + { + "instrument_uid": envelope.instrument_uid, + "instrument_id": envelope.instrument_id, + "instrument_revision": int(envelope.instrument_revision), + "feed": feed.value, + "interval": interval, + "observed_at_ns": observed_at_ns, + "revision": revision, + "payload": payload, + "source": { + "venue": envelope.venue, + "provider": envelope.provider, + "source_id": envelope.source_id, + "source_role": _enum_name( + common_pb2.SourceRole, envelope.source_role, "SOURCE_ROLE_" + ), + "authoritative": template.source.authoritative, + }, + "quality": { + "state": state, + "freshness_ms": int(freshness_ms), + "gap_open": gap_open, + "complete": not gap_open, + "execution_eligible": authoritative, + "policy_id": requirement.source_policy_id, + "flags": flags, + }, + "contract": { + **template.contract.model_dump(mode="json"), + "normalizer_version": envelope.normalizer_version, + "adapter_version": envelope.adapter_version, + "authority_revision": int(envelope.authority_revision), + "config_revision": int(envelope.config_revision), + "correlation_id": envelope.correlation_id + or bytes(envelope.event_id).hex(), + }, + "cursor": event.resume_token, + "watermark_offset": event.logical_offset, + } + ) diff --git a/tests/test_qdl_sdk_stream_projection.py b/tests/test_qdl_sdk_stream_projection.py new file mode 100644 index 0000000..81e5902 --- /dev/null +++ b/tests/test_qdl_sdk_stream_projection.py @@ -0,0 +1,325 @@ +from __future__ import annotations + +import unittest + +from qdl.common.v1 import common_pb2 +from qdl.marketdata.v2 import market_data_pb2 +from qdl_sdk import ( + DataRequirement, + Feed, + Grade, + StreamEvent, + market_data_view_from_stream, +) +from qdl_sdk.errors import ContinuityError +from qdl_sdk.models import MarketDataView + +NOW = 1_800_000_000_000_000_000 +DIGEST = "a" * 64 + + +def dec(value: int, scale: int = 2): + return common_pb2.DecimalValue(mantissa=value, scale=scale, source_text=str(value)) + + +def payload_fixture(feed: Feed) -> dict: + common = {"feed": feed.value} + return { + Feed.TRADE: { + **common, + "native_trade_id": "1", + "price": dv(), + "quantity": dv(), + "quantity_unit": "BASE_ASSET", + "aggressor_side": "BUY", + "identity_kind": "NATIVE", + }, + Feed.QUOTE: { + **common, + "bid_price": dv(), + "bid_quantity": dv(), + "ask_price": dv(101), + "ask_quantity": dv(), + "quantity_unit": "BASE_ASSET", + "level": 1, + }, + Feed.BAR: { + **common, + "interval": "1m", + "open_time_ns": NOW - 60_000_000_000, + "close_time_ns": NOW - 1, + "open": dv(), + "high": dv(102), + "low": dv(99), + "close": dv(101), + "volume": dv(), + "volume_unit": "BASE_ASSET", + "trade_count": 1, + "lifecycle": "FINAL", + "revision": 0, + "origin": "VENUE_NATIVE", + }, + Feed.BOOK_SNAPSHOT: { + **common, + "native_sequence": "1", + "levels": [level()], + "depth": 1, + }, + Feed.BOOK_DELTA: { + **common, + "native_sequence_start": "1", + "native_sequence_end": "2", + "snapshot_sequence": "1", + "updates": [level()], + "reset": False, + }, + Feed.FUNDING_RATE: {**common, "rate": dv(), "funding_time_ns": NOW}, + Feed.OPEN_INTEREST: {**common, "quantity": dv(), "quantity_unit": "CONTRACT"}, + Feed.MARK_INDEX_PRICE: {**common, "mark_price": dv(), "index_price": dv()}, + Feed.TICKER: {**common, "last_price": dv()}, + }[feed] + + +def dv(value: int = 100) -> dict: + return {"coefficient": str(value), "scale": 2, "source_text": str(value)} + + +def level() -> dict: + return { + "side": "BID", + "price": dv(), + "quantity": dv(), + "quantity_unit": "BASE_ASSET", + "order_count": 1, + } + + +def template(feed: Feed) -> MarketDataView: + return MarketDataView.model_validate( + { + "instrument_uid": "uid-1", + "instrument_id": "BINANCE.USDM.PERPETUAL.BTC-USDT", + "instrument_revision": 7, + "feed": feed.value, + "interval": "1m" if feed is Feed.BAR else None, + "observed_at_ns": NOW, + "revision": 0, + "payload": payload_fixture(feed), + "source": { + "venue": "BINANCE", + "provider": "BINANCE_DIRECT", + "source_id": "source-1", + "source_role": "PRIMARY", + "authoritative": True, + }, + "quality": { + "state": "LIVE", + "freshness_ms": 1, + "gap_open": False, + "complete": True, + "execution_eligible": True, + "policy_id": "crypto_primary_v2", + "flags": [], + }, + "contract": { + "schema_digest": DIGEST, + "contract_version": "2.0.0", + "normalizer_version": "old", + "adapter_version": "old", + "instrument_catalog_revision": 7, + "source_policy_revision": 2, + "authority_revision": 3, + "config_revision": 1, + "correlation_id": "snapshot", + }, + "watermark_offset": 10, + } + ) + + +def envelope(feed: Feed) -> market_data_pb2.EventEnvelope: + result = market_data_pb2.EventEnvelope( + schema_name="qdl.marketdata.v2.EventEnvelope", + schema_major=2, + schema_minor=0, + event_id=b"event-1", + instrument_uid="uid-1", + instrument_id="BINANCE.USDM.PERPETUAL.BTC-USDT", + instrument_revision=7, + venue="BINANCE", + market="USDM", + product_type="PERPETUAL", + native_symbol="BTCUSDT", + provider="BINANCE_DIRECT", + source_id="source-1", + source_role=common_pb2.SOURCE_ROLE_PRIMARY, + source_event_time_ns=NOW, + normalizer_version="rust-v2", + adapter_version="binance-v2", + correlation_id="stream", + config_revision=2, + authority_revision=3, + ) + if feed is Feed.TRADE: + result.trade.CopyFrom( + market_data_pb2.Trade( + native_trade_id="1", + price=dec(100), + quantity=dec(2), + aggressor_side=common_pb2.AGGRESSOR_SIDE_BUY, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + identity_kind=market_data_pb2.TRADE_IDENTITY_KIND_NATIVE, + ) + ) + elif feed is Feed.QUOTE: + result.quote.CopyFrom( + market_data_pb2.Quote( + bid_price=dec(100), + bid_quantity=dec(2), + ask_price=dec(101), + ask_quantity=dec(3), + level=1, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ) + elif feed is Feed.BAR: + result.bar.CopyFrom( + market_data_pb2.Bar( + interval="1m", + open_time_ns=NOW - 60_000_000_000, + close_time_ns=NOW - 1, + open=dec(100), + high=dec(102), + low=dec(99), + close=dec(101), + volume=dec(4), + trade_count=2, + is_final=True, + revision=0, + origin=common_pb2.BAR_ORIGIN_VENUE_NATIVE, + lifecycle=market_data_pb2.BAR_LIFECYCLE_FINAL, + volume_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ) + elif feed is Feed.BOOK_SNAPSHOT: + result.book_snapshot.CopyFrom( + market_data_pb2.OrderBookSnapshot( + native_sequence="1", + levels=[ + market_data_pb2.BookLevel( + side=common_pb2.BOOK_SIDE_BID, + price=dec(100), + quantity=dec(2), + order_count=1, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ], + depth=1, + ) + ) + elif feed is Feed.BOOK_DELTA: + result.book_delta.CopyFrom( + market_data_pb2.OrderBookDelta( + native_sequence_start="1", + native_sequence_end="2", + snapshot_sequence="1", + updates=[ + market_data_pb2.BookLevel( + side=common_pb2.BOOK_SIDE_ASK, + price=dec(101), + quantity=dec(2), + order_count=1, + quantity_unit=common_pb2.QUANTITY_UNIT_BASE_ASSET, + ) + ], + ) + ) + elif feed is Feed.FUNDING_RATE: + result.funding_rate.CopyFrom( + market_data_pb2.FundingRate(rate=dec(1, 4), funding_time_ns=NOW) + ) + elif feed is Feed.OPEN_INTEREST: + result.open_interest.CopyFrom( + market_data_pb2.OpenInterest( + quantity=dec(10), quantity_unit=common_pb2.QUANTITY_UNIT_CONTRACT + ) + ) + elif feed is Feed.MARK_INDEX_PRICE: + result.mark_index_price.CopyFrom( + market_data_pb2.MarkIndexPrice(mark_price=dec(100), index_price=dec(99)) + ) + elif feed is Feed.TICKER: + result.ticker.CopyFrom(market_data_pb2.Ticker(last_price=dec(100))) + return result + + +class SdkStreamProjectionTests(unittest.TestCase): + def requirement(self, feed: Feed) -> DataRequirement: + return DataRequirement( + instrument_uid="uid-1", + feed=feed, + consumer_grade=Grade.EXECUTION, + source_policy_id="crypto_primary_v2", + interval="1m" if feed is Feed.BAR else None, + max_freshness_ms=1000, + ) + + def test_all_public_market_payloads_project_to_typed_view(self): + for feed in Feed: + with self.subTest(feed=feed): + result = market_data_view_from_stream( + StreamEvent(11, "signed", envelope(feed)), + template=template(feed), + requirement=self.requirement(feed), + now_ns=NOW + 100_000_000, + ) + self.assertIs(result.feed, feed) + self.assertEqual(result.watermark_offset, 11) + self.assertEqual(result.cursor, "signed") + self.assertTrue(result.quality.execution_eligible) + + def test_gap_and_stale_execution_events_fail_closed(self): + gapped = envelope(Feed.TRADE) + gapped.quality_flags.append(common_pb2.QUALITY_FLAG_SEQUENCE_GAP_BEFORE) + with self.assertRaises(ContinuityError) as gap: + market_data_view_from_stream( + StreamEvent(11, "signed", gapped), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW + 1, + ) + self.assertEqual(gap.exception.code, "OPEN_SEQUENCE_GAP") + with self.assertRaises(ContinuityError) as stale: + market_data_view_from_stream( + StreamEvent(11, "signed", envelope(Feed.TRADE)), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW + 2_000_000_000, + ) + self.assertEqual(stale.exception.code, "DATA_STALE") + + def test_source_transition_and_revision_regression_require_snapshot(self): + changed = envelope(Feed.TRADE) + changed.source_id = "source-2" + with self.assertRaises(ContinuityError) as source: + market_data_view_from_stream( + StreamEvent(11, "signed", changed), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW, + ) + self.assertEqual(source.exception.code, "SOURCE_NON_AUTHORITATIVE") + changed = envelope(Feed.TRADE) + changed.authority_revision = 2 + with self.assertRaises(ContinuityError) as authority: + market_data_view_from_stream( + StreamEvent(11, "signed", changed), + template=template(Feed.TRADE), + requirement=self.requirement(Feed.TRADE), + now_ns=NOW, + ) + self.assertEqual(authority.exception.code, "SOURCE_NON_AUTHORITATIVE") + + +if __name__ == "__main__": + unittest.main() From 26903bf3c7c367eee17e93cb6617bac7f5d934ec Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 11:10:29 +0000 Subject: [PATCH 10/21] fix(sdk): enforce alpha stream continuity policies --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 4 ++++ qdl_sdk/projection.py | 9 ++++++++ tests/test_qdl_sdk_stream_projection.py | 28 +++++++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 244250c..d0dea44 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6098,6 +6098,10 @@ catalog or SDK wiring cannot be deferred as operational debt. durable audit and per-slice rollback remain mandatory. V1 stays hot on port `8100`; DNSE stays V1-only. Fast-track planning commit: `e8167d4`. +- `2026-08-20 C.0 SDK ALPHA STREAM POLICY CLOSURE STARTED`: downstream shared-runtime tests exposed a contract asymmetry: query validation enforces typed stale/gap policies for every consumer grade, while the stream projector currently blocks stale/gapped events only for `EXECUTION`. The source-owned SDK will enforce `stale_policy` and `gap_policy` identically for `ALPHA` and `EXECUTION`, retain the additional execution-eligibility gate for `EXECUTION`, and add explicit ALPHA stale/gap regression tests. A new deterministic wheel supersedes prior candidate digests only after Python 3.10 import, SDK release/stream tests, lint and byte-identical build pass. Consumers must update to that one digest; no downstream copy of projection logic is permitted. V1/runtime/provider/authority routes remain unchanged. + +- `2026-08-20 C.0 SDK ALPHA STREAM POLICY CLOSURE PASS`: the stream projector now applies typed `gap_policy` and `stale_policy` to ALPHA and EXECUTION consumers consistently; execution grade retains its additional authority/eligibility check. Added explicit ALPHA gap/stale regressions. SDK source projection/release/stream tests passed 21/21 on Python 3.12; the built wheel imported and passed 4/4 projection tests on the released Python 3.10 consumer runtime. Two independent builds were byte-identical at SHA-256 `3e1ce5e43d55ac4c04baf5b69354513f32090bd2e7060f1f4e659323470a27d0`; isolated Ruff lint and `git diff --check` passed. The repository legacy Poetry version syntax prevents modern Ruff from loading the root config and its existing files are not Ruff-format clean, so no unrelated format churn was introduced. No runtime/provider/authority route was touched. + #### C.1 Isolated Stable V2 Deployment Deploy the immutable pair under project `qdl_v2_stable_candidate`, loopback diff --git a/qdl_sdk/projection.py b/qdl_sdk/projection.py index c2956aa..a15d3fd 100644 --- a/qdl_sdk/projection.py +++ b/qdl_sdk/projection.py @@ -317,6 +317,15 @@ def market_data_view_from_stream( and freshness_ms > requirement.max_freshness_ms ) state = "GAPPED" if gap_open else "STALE" if stale else "LIVE" + if gap_open and requirement.gap_policy.value in {"BLOCK", "PAUSE"}: + raise ContinuityError( + "OPEN_SEQUENCE_GAP", "stream event violates the requested gap policy" + ) + if stale and requirement.stale_policy.value in {"BLOCK", "PAUSE"}: + raise ContinuityError( + "DATA_STALE", "stream event violates the requested freshness policy" + ) + authoritative = ( template.source.authoritative and template.source.source_role == "PRIMARY" diff --git a/tests/test_qdl_sdk_stream_projection.py b/tests/test_qdl_sdk_stream_projection.py index 81e5902..18db0f1 100644 --- a/tests/test_qdl_sdk_stream_projection.py +++ b/tests/test_qdl_sdk_stream_projection.py @@ -298,6 +298,34 @@ def test_gap_and_stale_execution_events_fail_closed(self): ) self.assertEqual(stale.exception.code, "DATA_STALE") + def test_gap_and_stale_alpha_events_obey_typed_policies(self): + requirement = DataRequirement( + instrument_uid="uid-1", + feed=Feed.TRADE, + consumer_grade=Grade.ALPHA, + source_policy_id="crypto_primary_v2", + max_freshness_ms=1000, + ) + gapped = envelope(Feed.TRADE) + gapped.quality_flags.append(common_pb2.QUALITY_FLAG_SEQUENCE_GAP_BEFORE) + with self.assertRaises(ContinuityError) as gap: + market_data_view_from_stream( + StreamEvent(11, "signed", gapped), + template=template(Feed.TRADE), + requirement=requirement, + now_ns=NOW + 1, + ) + self.assertEqual(gap.exception.code, "OPEN_SEQUENCE_GAP") + + with self.assertRaises(ContinuityError) as stale: + market_data_view_from_stream( + StreamEvent(11, "signed", envelope(Feed.TRADE)), + template=template(Feed.TRADE), + requirement=requirement, + now_ns=NOW + 2_000_000_000, + ) + self.assertEqual(stale.exception.code, "DATA_STALE") + def test_source_transition_and_revision_regression_require_snapshot(self): changed = envelope(Feed.TRADE) changed.source_id = "source-2" From f93b7f0e4d3381a01da48dafbb8263436b0315e1 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 11:41:29 +0000 Subject: [PATCH 11/21] fix(contracts): preserve frozen V2 enum schema --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 4 ++ qdl_sdk/models.py | 47 +++++++++++++++-------- tests/test_qdl_sdk_stream_projection.py | 11 ++++++ 3 files changed, 45 insertions(+), 17 deletions(-) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index d0dea44..1d01166 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6102,6 +6102,10 @@ catalog or SDK wiring cannot be deferred as operational debt. - `2026-08-20 C.0 SDK ALPHA STREAM POLICY CLOSURE PASS`: the stream projector now applies typed `gap_policy` and `stale_policy` to ALPHA and EXECUTION consumers consistently; execution grade retains its additional authority/eligibility check. Added explicit ALPHA gap/stale regressions. SDK source projection/release/stream tests passed 21/21 on Python 3.12; the built wheel imported and passed 4/4 projection tests on the released Python 3.10 consumer runtime. Two independent builds were byte-identical at SHA-256 `3e1ce5e43d55ac4c04baf5b69354513f32090bd2e7060f1f4e659323470a27d0`; isolated Ruff lint and `git diff --check` passed. The repository legacy Poetry version syntax prevents modern Ruff from loading the root config and its existing files are not Ruff-format clean, so no unrelated format churn was introduced. No runtime/provider/authority route was touched. +- `2026-08-20 C.0 FROZEN OPENAPI COMPATIBILITY BLOCKER STARTED`: the pre-build full suite passed 526 tests with 6 skips but failed both frozen OpenAPI assertions. Inspection found the earlier model-ownership move accidentally renamed response component `FeedType` to `Feed` and removed `BarLifecycle.UNSPECIFIED` from the published enum. Restore the frozen wire schema without reverting SDK ownership: declare `FeedType` as the concrete enum, export `Feed` as its SDK alias, retain `UNSPECIFIED` in OpenAPI and continue rejecting it in model validation. The unchanged frozen snapshot must pass; regenerating it to hide this break is forbidden. The SDK wheel and both downstream consumer pins must be rebuilt once more after the complete Python/Rust gates pass. + +- `2026-08-20 C.0 FROZEN OPENAPI COMPATIBILITY BLOCKER PASS`: the Data Layer service and SDK now share the exact public `qdl.query.FeedType`/`BarLifecycle` enum identity when that contract package is present; the standalone wheel supplies equivalent fallback enums and exports concise `Feed` as an alias. `UNSPECIFIED` remains published for wire compatibility and is rejected at the typed requirement/model boundary. The frozen OpenAPI snapshot was not modified and now matches exactly: 10 paths and 42 schemas. Targeted OpenAPI/SDK tests passed 24/24; the full Python suite passed 535/535 with 6 skips. The Rust workspace passed 70/70 plus `cargo fmt --check` and strict Clippy. Two release builds were byte-identical at final wheel SHA-256 `10f894604c543fc07499247b5c6fc38910b8e704bffe29683f100c519d6caa49`; the installed wheel passed 5/5 stream-projection tests on Python 3.10 and exposed `Feed.__name__ == FeedType`. Isolated Ruff and `git diff --check` passed. This supersedes all earlier candidate wheel digests; consumers must pin only this digest. V1/runtime/provider/authority routes remain unchanged. + #### C.1 Isolated Stable V2 Deployment Deploy the immutable pair under project `qdl_v2_stable_candidate`, loopback diff --git a/qdl_sdk/models.py b/qdl_sdk/models.py index d32088b..edc4dcf 100644 --- a/qdl_sdk/models.py +++ b/qdl_sdk/models.py @@ -16,16 +16,32 @@ def __str__(self) -> str: from qdl.query.v2 import query_pb2 -class Feed(StrEnum): - TRADE = "TRADE" - QUOTE = "QUOTE" - BAR = "BAR" - BOOK_SNAPSHOT = "BOOK_SNAPSHOT" - BOOK_DELTA = "BOOK_DELTA" - FUNDING_RATE = "FUNDING_RATE" - OPEN_INTEREST = "OPEN_INTEREST" - MARK_INDEX_PRICE = "MARK_INDEX_PRICE" - TICKER = "TICKER" +try: + # The service and SDK share enum identity when the public query package is present. + from qdl.query import BarLifecycle, FeedType +except ImportError: + class FeedType(StrEnum): + UNSPECIFIED = "UNSPECIFIED" + TRADE = "TRADE" + QUOTE = "QUOTE" + BAR = "BAR" + BOOK_SNAPSHOT = "BOOK_SNAPSHOT" + BOOK_DELTA = "BOOK_DELTA" + FUNDING_RATE = "FUNDING_RATE" + OPEN_INTEREST = "OPEN_INTEREST" + MARK_INDEX_PRICE = "MARK_INDEX_PRICE" + TICKER = "TICKER" + + class BarLifecycle(StrEnum): + UNSPECIFIED = "UNSPECIFIED" + IN_PROGRESS = "IN_PROGRESS" + FINAL = "FINAL" + REVISED = "REVISED" + CANCELLED = "CANCELLED" + + +# Concise public SDK spelling; the frozen wire component remains FeedType. +Feed = FeedType class Grade(StrEnum): @@ -58,13 +74,6 @@ class BarRevisionPolicy(StrEnum): EMIT_REVISIONS = "EMIT_REVISIONS" -class BarLifecycle(StrEnum): - IN_PROGRESS = "IN_PROGRESS" - FINAL = "FINAL" - REVISED = "REVISED" - CANCELLED = "CANCELLED" - - class ClosedModel(BaseModel): model_config = ConfigDict(extra="forbid", populate_by_name=True) @@ -178,6 +187,8 @@ class BarPayload(ClosedModel): @model_validator(mode="after") def validate_lifecycle(self): + if self.lifecycle is BarLifecycle.UNSPECIFIED: + raise ValueError("bar lifecycle cannot be UNSPECIFIED") if self.lifecycle is BarLifecycle.REVISED and not self.supersedes_event_id: raise ValueError("revised bar must identify the superseded event") if self.close_time_ns <= self.open_time_ns: @@ -416,6 +427,8 @@ class DataRequirement: def __post_init__(self) -> None: if not self.instrument_uid.strip() or not self.source_policy_id.strip(): raise ValueError("instrument_uid and source_policy_id are required") + if self.feed is Feed.UNSPECIFIED: + raise ValueError("UNSPECIFIED feed is invalid at the V2 boundary") enum_fields = ( (self.feed, Feed, "feed"), (self.consumer_grade, Grade, "consumer_grade"), diff --git a/tests/test_qdl_sdk_stream_projection.py b/tests/test_qdl_sdk_stream_projection.py index 18db0f1..2967346 100644 --- a/tests/test_qdl_sdk_stream_projection.py +++ b/tests/test_qdl_sdk_stream_projection.py @@ -266,6 +266,8 @@ def requirement(self, feed: Feed) -> DataRequirement: def test_all_public_market_payloads_project_to_typed_view(self): for feed in Feed: + if feed is Feed.UNSPECIFIED: + continue with self.subTest(feed=feed): result = market_data_view_from_stream( StreamEvent(11, "signed", envelope(feed)), @@ -278,6 +280,15 @@ def test_all_public_market_payloads_project_to_typed_view(self): self.assertEqual(result.cursor, "signed") self.assertTrue(result.quality.execution_eligible) + def test_unspecified_feed_is_rejected_at_requirement_boundary(self): + with self.assertRaisesRegex(ValueError, "UNSPECIFIED"): + DataRequirement( + instrument_uid="uid-1", + feed=Feed.UNSPECIFIED, + consumer_grade=Grade.ALPHA, + source_policy_id="crypto_primary_v2", + ) + def test_gap_and_stale_execution_events_fail_closed(self): gapped = envelope(Feed.TRADE) gapped.quality_flags.append(common_pb2.QUALITY_FLAG_SEQUENCE_GAP_BEFORE) From 005fa66cbd4ea968eb42e6f84088f1feb357c88c Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 12:21:22 +0000 Subject: [PATCH 12/21] test(runtime): certify isolated V2 shadow deployment --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 59 ++- .../v2-production-rust-authority-cutover.md | 20 +- .../phasec1_isolated_consumer_acceptance.py | 335 ++++++++++++++++++ 3 files changed, 411 insertions(+), 3 deletions(-) create mode 100755 scripts/phasec1_isolated_consumer_acceptance.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 1d01166..90d3e60 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -5873,7 +5873,7 @@ and removes the superseded candidate/rollback artifacts after verification. ### Phase C - Production V2 And Rust Authority Cutover -**Status:** `C.0 RELEASE/MERGE PREPARATION IN PROGRESS / PRODUCTION CUTOVER NOT AUTHORIZED` +**Status:** `C.1 SHADOW-CERTIFIED / C.2 CONSUMER CLOSURE IN PROGRESS / PRODUCTION CUTOVER NOT AUTHORIZED` **Purpose:** move approved Binance and OKX feed slices from the current V1 authority to the stable V2 contract with Rust as the actual canonical realtime @@ -6119,6 +6119,63 @@ quarantine, bounded lag/resources, broker and process restart recovery, exact cursor continuation, V1 health unchanged and exact disposable cleanup on failure. +**C.1 implementation journal:** + +- `2026-08-20 C.1 IMMUTABLE ISOLATED DEPLOYMENT STARTED`: build the Python + edge and Rust core from the same tested source revision + `f93b7f0e4d3381a01da48dafbb8263436b0315e1`. The immutable candidates are + `qdl-v2-python:2.0.0-f93b7f0e4d33` with image ID + `sha256:7a1b11097e4e85a51630068b2a619e34ce654b532d5ea750c6b8678882f2cc86` + and `qdl-v2-rust:2.0.0-f93b7f0e4d33` with image ID + `sha256:fc50dbf0a83323966ed6d8e76ae468a98edad6e34c7f0392a07924e94018348f`; + both carry the exact source revision label and run as UID/GID `10001` in + the stable compose. The bounded certification slice contains authentic + Binance USDM/Spot and OKX Swap/Spot BTC-USDT feeds only. It uses project + `qdl_v2_stable_candidate`, dedicated RF3/minISR2 Kafka, ephemeral Redis, + private state/TLS, loopback ports `18201/18202/18210/18211/18220/18221` + and `RUST_SHADOW`; `stable-vn` is excluded. Port `8100`, V1 containers, + V1 volumes and current consumer routes are immutable boundaries. Rollback + before consumer migration is project-scoped `docker compose down` without + `-v`; no authority CAS or Trading System route mutation is authorized by + this journal entry. + +- `2026-08-20 C.1 ISOLATED SHADOW CERTIFICATION PASS`: generated a private + `0600` environment bundle with short-lived test TLS material and + `cutover_authorized=false`, then started only project + `qdl_v2_stable_candidate`. Kafka created three six-partition topics at RF3, + minISR2 and full ISR. Authentic Binance USDM/Spot and OKX Swap/Spot trades + plus 500 closed 1m bars per binding entered the raw topic and passed through + the Rust canonical core. Query replicas returned exact BAR payload parity; + Binance/OKX five-row warmups were `FULL`, `FINAL`, authoritative, + complete and gap-free. The reusable + `scripts/phasec1_isolated_consumer_acceptance.py` used the released SDK to + prove signed snapshot handoff, `REPLAYING -> LIVE`, ACK, fsynced cursor + persistence, client recreation through the other query replica and exact + `N+1` resume for both venues. The quarantine topic remained zero on all + six partitions. +- Failure drills stayed project-scoped. Restarting one Rust worker made the + execution-grade query fail closed on freshness while the group rebalanced; + it recovered to lag 32 and the SDK acceptance passed again. Stopping the + current stream lease holder promoted the peer in five seconds; both replicas + remained live, exactly one was ready, and cursor/reconnect acceptance passed. + Restarting one Kafka broker restored full ISR with core lag 51 and projector + lag 57; post-restart SDK acceptance passed and quarantine remained zero. + Missing auth returned 401, mismatched consumer returned 403, and wrong + purpose on a market-data endpoint returned 403. At the bounded resource + sample Rust workers used 27-44 MiB each, Python roles 40-72 MiB, Redis 3 MiB + and Kafka brokers 427-433 MiB each. V1 remained healthy with 40 paths, image + `sha256:8f2a5a3f1ff97762feb1531c3787e714dfda60b0b64df5b7359b9e5f6740c980`, + original start time and restart count zero. +- `C.1 conclusion: PASS / SHADOW-CERTIFIED`. This is not production authority. + C.2 inspection exposed mandatory blockers before consumer cutover: stable + gRPC currently binds insecurely and is reachable only through loopback; + Trading System runtime IDs do not match the registered stable consumer ID; + and V2 routing is provider-wide although this certified catalog is a bounded + BTC slice. In addition, the stable runtime accepts only `RUST_SHADOW`; + production promotion must consume the durable authority CAS/outbox and fence + writers instead of changing an environment label. These are in-scope C.2/C.3 + correctness gates and must be fixed, not deferred as operational debt. + #### C.2 Single-Consumer Trading System Cutover The operator confirms all alpha consumers are currently stopped and Trading diff --git a/docs/runbooks/v2-production-rust-authority-cutover.md b/docs/runbooks/v2-production-rust-authority-cutover.md index f6aa4d2..8495d13 100644 --- a/docs/runbooks/v2-production-rust-authority-cutover.md +++ b/docs/runbooks/v2-production-rust-authority-cutover.md @@ -173,9 +173,25 @@ Acceptance: ```bash curl --fail --silent http://127.0.0.1:18201/health/ready curl --fail --silent http://127.0.0.1:18202/health/ready -curl --fail --silent http://127.0.0.1:18210/health/ready -curl --fail --silent http://127.0.0.1:18211/health/ready +curl --fail --silent http://127.0.0.1:18210/health/live +curl --fail --silent http://127.0.0.1:18211/health/live curl --fail --silent http://127.0.0.1:8100/v1/health + +# Active/passive means both stream replicas are live and exactly one is ready. +# Select that replica's matching gRPC port for the SDK handoff/reconnect gate. +if curl --fail --silent http://127.0.0.1:18210/health/ready >/dev/null; then + GRPC_TARGET=127.0.0.1:18220 +else + curl --fail --silent http://127.0.0.1:18211/health/ready >/dev/null + GRPC_TARGET=127.0.0.1:18221 +fi +set -a +. "$QDL_RELEASE_ROOT/stable.env" +set +a +docker run --rm --network host \ + -e QDL_STABLE_JWT_KEYS_JSON="$QDL_STABLE_JWT_KEYS_JSON" \ + -v "$PWD/scripts/phasec1_isolated_consumer_acceptance.py:/acceptance.py:ro" \ + "$PYTHON_IMAGE" python /acceptance.py --grpc-target "$GRPC_TARGET" ``` Require authentic Binance/OKX data, zero unexplained gap/duplicate/quarantine, diff --git a/scripts/phasec1_isolated_consumer_acceptance.py b/scripts/phasec1_isolated_consumer_acceptance.py new file mode 100755 index 0000000..feb8457 --- /dev/null +++ b/scripts/phasec1_isolated_consumer_acceptance.py @@ -0,0 +1,335 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import tempfile +import time +import uuid +from dataclasses import dataclass +from pathlib import Path + +import jwt + +from qdl_sdk import ( + AsyncDataLayerClient, + ControlEvent, + DataRequirement, + Feed, + FileCursorStore, + GapPolicy, + Grade, + GrpcStreamTransport, + RecoveryPolicy, + RestQueryTransport, + StalePolicy, + StaticBearerCredential, + StreamEvent, + market_data_view_from_stream, +) + + +@dataclass(frozen=True) +class VenueCase: + venue: str + instrument_uid: str + consumer_id: str + subject: str + provider: str + + +CASES = ( + VenueCase( + "BINANCE", + "a953e16e-7138-5562-b5e8-c337a44d0b65", + "alpha.binance.paper.stable", + "spiffe://qdl/paper/alpha-binance-stable", + "BINANCE_DIRECT", + ), + VenueCase( + "OKX", + "fb26214c-7b9b-5961-95b2-55154755af0f", + "alpha.okx.paper.stable", + "spiffe://qdl/paper/alpha-okx-stable", + "OKX_DIRECT", + ), +) + + +def token(subject: str, *, issuer: str, audience: str) -> str: + keys = json.loads(os.environ["QDL_STABLE_JWT_KEYS_JSON"]) + key_id, secret = sorted(keys.items())[0] + now = int(time.time()) + return jwt.encode( + { + "sub": subject, + "iss": issuer, + "aud": audience, + "iat": now, + "nbf": now - 1, + "exp": now + 300, + "jti": str(uuid.uuid4()), + "environment": "paper", + "roles": [ + "market_data_reader", + "historical_reader", + "stream_consumer", + ], + "consumer_manifest_revision": 1, + }, + secret, + algorithm="HS256", + headers={"kid": key_id}, + ) + + +def client( + *, + base_url: str, + grpc_target: str, + consumer_id: str, + bearer: str, + cursor_path: Path, +) -> AsyncDataLayerClient: + credential = StaticBearerCredential(bearer) + return AsyncDataLayerClient( + query_transport=RestQueryTransport( + base_url, + timeout_seconds=8, + credential_provider=credential, + ), + stream_transport=GrpcStreamTransport( + grpc_target, + allow_insecure_loopback=True, + credential_provider=credential, + ), + consumer_id=consumer_id, + cursor_store=FileCursorStore(cursor_path), + max_buffer_events=64, + max_reconnect_attempts=2, + ) + + +def bar_fingerprint(response) -> list[dict[str, object]]: + values = [] + for item in response.data: + if item.feed is not Feed.BAR: + raise AssertionError("warmup returned a non-BAR item") + if item.payload.lifecycle not in {"FINAL", "REVISED"}: + raise AssertionError("warmup returned a non-final BAR") + if ( + item.source.authoritative is not True + or item.quality.complete is not True + or item.quality.gap_open is not False + ): + raise AssertionError("warmup BAR failed authority/coverage gate") + values.append( + { + "instrument_uid": item.instrument_uid, + "instrument_id": item.instrument_id, + "observed_at_ns": item.observed_at_ns, + "payload": item.payload.model_dump(mode="json"), + "source": item.source.model_dump(mode="json"), + "contract": item.contract.model_dump(mode="json"), + } + ) + return values + + +async def next_data(session) -> tuple[StreamEvent, list[str]]: + controls: list[str] = [] + for _ in range(8): + item = await asyncio.wait_for(session.__anext__(), timeout=12) + if isinstance(item, ControlEvent): + controls.append(item.code) + continue + if not isinstance(item, StreamEvent): + raise AssertionError("stream returned an unknown SDK event") + return item, controls + raise AssertionError("stream did not reach a market-data event") + + +async def certify_case( + case: VenueCase, + *, + primary_url: str, + secondary_url: str, + grpc_target: str, + issuer: str, + audience: str, + state_dir: Path, +) -> dict[str, object]: + bearer = token(case.subject, issuer=issuer, audience=audience) + bar_requirement = DataRequirement( + case.instrument_uid, + Feed.BAR, + Grade.ALPHA, + "crypto_primary_v2", + interval="1m", + warmup_limit=5, + max_freshness_ms=180_000, + stale_policy=StalePolicy.BLOCK, + gap_policy=GapPolicy.BLOCK, + recovery=RecoveryPolicy.SNAPSHOT_AND_REPLAY, + ) + primary = client( + base_url=primary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=state_dir / f"{case.venue.lower()}-query-primary.json", + ) + secondary = client( + base_url=secondary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=state_dir / f"{case.venue.lower()}-query-secondary.json", + ) + try: + first = await primary.warmup(bar_requirement) + second = await secondary.warmup(bar_requirement) + finally: + await primary.close() + await secondary.close() + first_rows = bar_fingerprint(first) + second_rows = bar_fingerprint(second) + if first_rows != second_rows: + raise AssertionError(f"{case.venue} query replicas diverged") + + trade_requirement = DataRequirement( + case.instrument_uid, + Feed.TRADE, + Grade.ALPHA, + "crypto_primary_v2", + warmup_limit=0, + max_freshness_ms=15_000, + stale_policy=StalePolicy.BLOCK, + gap_policy=GapPolicy.BLOCK, + recovery=RecoveryPolicy.SNAPSHOT_AND_REPLAY, + ) + cursor_path = state_dir / f"{case.venue.lower()}-stream.json" + first_client = client( + base_url=primary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=cursor_path, + ) + try: + async with first_client.warmup_then_stream(trade_requirement) as session: + first_event, first_controls = await next_data(session) + first_view = market_data_view_from_stream( + first_event, + template=session.warmup.data[-1], + requirement=trade_requirement, + ) + if ( + first_view.source.provider != case.provider + or not first_view.source.authoritative + or not first_view.quality.complete + or first_view.quality.gap_open + ): + raise AssertionError(f"{case.venue} first stream event failed quality") + session.acknowledge(first_event) + finally: + await first_client.close() + + resumed_client = client( + base_url=secondary_url, + grpc_target=grpc_target, + consumer_id=case.consumer_id, + bearer=bearer, + cursor_path=cursor_path, + ) + try: + async with resumed_client.warmup_then_stream( + trade_requirement, + resume_restored_state=True, + ) as session: + resumed_event, resumed_controls = await next_data(session) + resumed_view = market_data_view_from_stream( + resumed_event, + template=session.warmup.data[-1], + requirement=trade_requirement, + ) + if resumed_event.logical_offset != first_event.logical_offset + 1: + raise AssertionError( + f"{case.venue} cursor resume was not contiguous: " + f"{first_event.logical_offset}->{resumed_event.logical_offset}" + ) + if resumed_view.source.provider != case.provider: + raise AssertionError(f"{case.venue} resumed source changed") + session.acknowledge(resumed_event) + finally: + await resumed_client.close() + + persisted = json.loads(cursor_path.read_text(encoding="utf-8")) + offsets = [ + int(item["offset"]) + for item in persisted["items"].values() + ] + if offsets != [resumed_event.logical_offset]: + raise AssertionError(f"{case.venue} durable cursor was not ACKed") + + return { + "venue": case.venue, + "provider": case.provider, + "warmup_rows": len(first_rows), + "coverage": first.coverage, + "final_bar_close_time_ns": first_rows[-1]["payload"]["close_time_ns"], + "first_stream_offset": first_event.logical_offset, + "resumed_stream_offset": resumed_event.logical_offset, + "first_controls": first_controls, + "resumed_controls": resumed_controls, + "cursor_persisted": True, + "secret_values_recorded": False, + } + + +async def run(args: argparse.Namespace) -> dict[str, object]: + with tempfile.TemporaryDirectory(prefix="qdl-c1-sdk-") as temporary: + state_dir = Path(temporary) + results = [] + for case in CASES: + results.append( + await certify_case( + case, + primary_url=args.primary_url, + secondary_url=args.secondary_url, + grpc_target=args.grpc_target, + issuer=args.issuer, + audience=args.audience, + state_dir=state_dir, + ) + ) + return { + "schema": "qdl.phase-c1.isolated-consumer-acceptance.v1", + "status": "PASS", + "contract_version": "2.0.0", + "authority_expected": "RUST_SHADOW", + "cases": results, + "secret_values_recorded": False, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--primary-url", default="http://127.0.0.1:18201") + parser.add_argument("--secondary-url", default="http://127.0.0.1:18202") + parser.add_argument("--grpc-target", required=True) + parser.add_argument( + "--issuer", default="https://identity.qdl.stable.internal" + ) + parser.add_argument("--audience", default="qdl-v2-stable") + args = parser.parse_args() + if "QDL_STABLE_JWT_KEYS_JSON" not in os.environ: + raise SystemExit("QDL_STABLE_JWT_KEYS_JSON is required") + print(json.dumps(asyncio.run(run(args)), sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 8c8fa9dd7740b0ceba229854fac3c30061fbc338 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 13:46:40 +0000 Subject: [PATCH 13/21] feat(security): close V2 workload ingress --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 62 ++++++++ app/entrypoints/query_v2_stable.py | 9 +- consumers/stable/trading-system-paper.yaml | 26 ++++ docker-compose.v2-stable.yml | 34 ++++- poetry.lock | 66 +++++++- pyproject.toml | 2 +- qdl/runtime/stable.py | 66 +++++++- qdl/runtime/stable_ingest.py | 9 +- qdl_sdk/__init__.py | 4 + qdl_sdk/credentials.py | 87 +++++++++++ qdl_sdk/tls.py | 50 +++++++ qdl_sdk/transport.py | 58 +++++-- scripts/build_qdl_sdk_release.py | 1 + scripts/phase80_generate_tls.sh | 14 +- scripts/phaseb_prepare_stable_candidate.py | 45 +++++- .../phasec1_isolated_consumer_acceptance.py | 52 +++++-- scripts/rebuild_v2_stable_projection_cache.py | 61 +++++++- tests/test_phaseb_stable_deployment.py | 16 +- tests/test_phaseb_stable_edge.py | 15 +- tests/test_phaseb_stable_rebuild.py | 24 +++ tests/test_phasec2_transport_security.py | 141 ++++++++++++++++++ 21 files changed, 783 insertions(+), 59 deletions(-) create mode 100644 qdl_sdk/tls.py create mode 100644 tests/test_phasec2_transport_security.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 90d3e60..8059d5a 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6178,6 +6178,68 @@ failure. #### C.2 Single-Consumer Trading System Cutover +**C.2 implementation journal:** + +- `2026-08-20 C.2 CONSUMER INGRESS CLOSURE STARTED`: close the real + integration gaps found by C.1 before any consumer restart. Stable REST and + gRPC data-plane ingress must use server-authenticated TLS plus client + workload certificates; JWT issuer/audience/manifest authorization remains + mandatory at the application layer. Projector-to-stream ingest uses the same + authenticated transport. The source-owned `qdl_sdk` adds CA/client + certificate configuration once; Trading System and execution-alpha consume + it without custom transports. The Trading System manifest gains only the + final 1m BAR permissions its market-cache bridge actually uses. +- Trading System must route by a strict versioned slice manifest + `venue + market + product + native symbol + feed + interval`. In + `V2_PRIMARY`, only approved slices leave V1; all unmatched Binance symbols + remain on V1 and are audited as compatibility routes. OKX has no equivalent + V1 realtime endpoint and therefore fails closed on V2 outage rather than + being relabelled as a cross-venue fallback. One configured external consumer + identity must match the registered Data Layer manifest; unrelated Trading + System services remain V1 until separately manifested and credentialed. +- `2026-08-20 C.2 WORKLOAD TLS/SDK SLICE PASS`: added one source-owned + `WorkloadTlsConfig` for REST and gRPC client certificates, bounded + multi-target gRPC failover, mandatory stable query/stream server mTLS and + projector-to-stream HTTPS mTLS while retaining JWT/manifest authorization + and HMAC ingest signing. Stable bundles now carry separate query, stream, + projector and Trading System identities outside Git; the Trading System + manifest gained only Binance USDM and OKX SWAP final 1m BAR requirements. + Focused transport/security/stable-ingest tests passed 6/6; Python compile and + YAML parse gates passed. No V1 container, consumer route, authority state, + production data or order path was mutated. Real certificate handshake, + immutable rebuild and rotation/reconnect remain C.2 gates before closure. +- `2026-08-20 C.2 ISOLATED RESTART RECOVERY GATE FOUND`: the first secure + isolated restart proved the query mTLS positive handshake and rejected a + client without a workload certificate, while V1 remained HTTP 200 and was + not restarted. The projector correctly failed closed because retained + SQLite projection state was paired with a newly empty ephemeral Redis cache. + This is the designed B16 generation fence, not permission to bind an empty + cache over retained state. Before acceptance continues, update the existing + exact-scope B17 cache-unit rebuild command to use the new mTLS health probes, + then rebuild only the isolated Redis plus three SQLite cache files from the + Kafka canonical log. Kafka/provider data, V1, production Redis and authority + remain untouched. Acceptance requires bounded six-partition lag, a bound + cache identity, both secure query replicas ready and a fresh signed SDK + handoff after rebuild. +- `2026-08-20 C.2 RELEASE/REGRESSION SLICE PASS; RUNTIME REPLAY CONTINUES`: + the final source-owned `qdl_sdk==2.0.0` artifact was built twice + byte-identically at SHA-256 + `5891c0b99b29fd30ce008f6987a4ff9c9d4896259e415f72e5f9210460669951` + with Python >=3.10 and `PyJWT[crypto]` declared. The complete Python suite + passed 540/540 with six explicit environment skips; targeted secure + bundle/recovery/transport tests passed 14/14. Rust passed 70/70 plus + `cargo fmt --check` and strict Clippy. The real mTLS query handshake + returned 200 and a request without a client certificate failed the TLS + handshake; V1 remained HTTP 200. A non-destructive cache generation and + isolated projector group are replaying authentic retained Kafka records + because the exact destructive B17 rebuild was not approved. Therefore C.2 is + not yet closed and no consumer route or authority was promoted. +- C.2 gates are mTLS positive/negative tests, certificate rotation/reconnect, + exact BAR/trade SDK projection, bounded route-manifest parser tests, V1 + unmatched-symbol compatibility, authenticated real Binance/OKX adapter + acceptance, no order submission and unchanged V1/runtime state. Authority + stays `RUST_SHADOW` until the separate C.3 CAS/outbox/fence packet passes. + The operator confirms all alpha consumers are currently stopped and Trading System is the only active Data Layer consumer. Do not create artificial alpha or monitoring migration stages. Built-in V2 health/lag/authority telemetry remains diff --git a/app/entrypoints/query_v2_stable.py b/app/entrypoints/query_v2_stable.py index f20fb2e..b5dcf93 100644 --- a/app/entrypoints/query_v2_stable.py +++ b/app/entrypoints/query_v2_stable.py @@ -1,6 +1,9 @@ -"""Isolated Data Layer 2.0.0 stable query edge; V1 remains authoritative.""" +"""Stable Data Layer 2.0.0 query edge with mandatory workload mTLS.""" -from qdl.runtime.stable import create_stable_query_app +import asyncio +from qdl.runtime.stable import serve_stable_query -app = create_stable_query_app() + +if __name__ == "__main__": + asyncio.run(serve_stable_query()) diff --git a/consumers/stable/trading-system-paper.yaml b/consumers/stable/trading-system-paper.yaml index 31169b9..8d44e15 100644 --- a/consumers/stable/trading-system-paper.yaml +++ b/consumers/stable/trading-system-paper.yaml @@ -50,6 +50,19 @@ spec: gap_policy: BLOCK recovery: SNAPSHOT_AND_REPLAY bar_revision_policy: LATEST + - instrument_uid: a953e16e-7138-5562-b5e8-c337a44d0b65 + feed: BAR + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + interval: 1m + warmup_limit: 1000 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f feed: TRADE consumer_grade: EXECUTION @@ -74,6 +87,19 @@ spec: gap_policy: BLOCK recovery: SNAPSHOT_AND_REPLAY bar_revision_policy: LATEST + - instrument_uid: fb26214c-7b9b-5961-95b2-55154755af0f + feed: BAR + consumer_grade: EXECUTION + source_policy_id: crypto_primary_v2 + interval: 1m + warmup_limit: 1000 + max_freshness_ms: 180000 + require_full_coverage: true + require_final_bars: true + stale_policy: BLOCK + gap_policy: BLOCK + recovery: SNAPSHOT_AND_REPLAY + bar_revision_policy: LATEST - instrument_uid: d0ac3d4c-2374-5690-8af5-b970448f91d3 feed: TRADE consumer_grade: EXECUTION diff --git a/docker-compose.v2-stable.yml b/docker-compose.v2-stable.yml index d80d614..2045674 100644 --- a/docker-compose.v2-stable.yml +++ b/docker-compose.v2-stable.yml @@ -82,7 +82,7 @@ x-stable-env: &stable-env QDL_DATA_JWT_KEYS_JSON: ${QDL_STABLE_JWT_KEYS_JSON:?set QDL_STABLE_JWT_KEYS_JSON} QDL_DATA_JWT_ISSUER: https://identity.qdl.stable.internal QDL_DATA_JWT_AUDIENCE: qdl-v2-stable - QDL_DATA_JWT_ALGORITHMS: HS256 + QDL_DATA_JWT_ALGORITHMS: RS256 QDL_DATA_JWT_MAX_LIFETIME_SECONDS: "900" x-python: &python @@ -94,7 +94,7 @@ x-python: &python cap_drop: [ALL] networks: [stable_internal] tmpfs: ["/tmp:rw,nosuid,nodev,noexec,size=64m"] - volumes: [stable_state:/var/lib/qdl-stable] + volumes: [stable_state:/var/lib/qdl-stable, stable_tls:/stable-certs:ro] environment: {<<: *stable-env} mem_limit: 512m cpus: 0.75 @@ -177,10 +177,13 @@ services: - -c - >- install -d -o 10001 -g 10001 -m 0750 - /stable-certs/producer /stable-certs/core /stable-certs/projector && + /stable-certs/producer /stable-certs/core /stable-certs/projector + /stable-certs/query /stable-certs/stream && cp /source/producer/* /stable-certs/producer/ && cp /source/core/* /stable-certs/core/ && cp /source/projector/* /stable-certs/projector/ && + cp /source/query/* /stable-certs/query/ && + cp /source/stream/* /stable-certs/stream/ && chown -R 10001:10001 /stable-certs && find /stable-certs -type f -exec chmod 0440 {} + restart: "no" @@ -193,11 +196,13 @@ services: - ${QDL_STABLE_PRODUCER_CERT_DIR:?set QDL_STABLE_PRODUCER_CERT_DIR}:/source/producer:ro - ${QDL_STABLE_CORE_CERT_DIR:?set QDL_STABLE_CORE_CERT_DIR}:/source/core:ro - ${QDL_STABLE_PROJECTOR_CERT_DIR:?set QDL_STABLE_PROJECTOR_CERT_DIR}:/source/projector:ro + - ${QDL_STABLE_QUERY_CERT_DIR:?set QDL_STABLE_QUERY_CERT_DIR}:/source/query:ro + - ${QDL_STABLE_STREAM_CERT_DIR:?set QDL_STABLE_STREAM_CERT_DIR}:/source/stream:ro - stable_tls:/stable-certs query_v2_1: <<: *python - command: [uvicorn, app.entrypoints.query_v2_stable:app, --host, 0.0.0.0, --port, "8200", --no-access-log] + command: [python, -m, app.entrypoints.query_v2_stable] networks: [stable_internal, stable_ingress] ports: ["127.0.0.1:18201:8200"] environment: @@ -207,12 +212,15 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-query-1-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/query/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/query/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/query/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} - healthcheck: {test: [CMD, python, -c, "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8200/health/ready', timeout=2)"], interval: 5s, timeout: 3s, retries: 20} + healthcheck: {test: [CMD, python, -c, "import ssl,urllib.request; c=ssl.create_default_context(cafile='/stable-certs/query/ca.crt'); c.load_cert_chain('/stable-certs/query/server.crt','/stable-certs/query/server.key'); urllib.request.urlopen('https://localhost:8200/health/ready',context=c,timeout=2)"], interval: 5s, timeout: 3s, retries: 20} query_v2_2: <<: *python - command: [uvicorn, app.entrypoints.query_v2_stable:app, --host, 0.0.0.0, --port, "8200", --no-access-log] + command: [python, -m, app.entrypoints.query_v2_stable] networks: [stable_internal, stable_ingress] ports: ["127.0.0.1:18202:8200"] environment: @@ -222,6 +230,9 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-query-2-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/query/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/query/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/query/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} stream_v2_active: @@ -236,6 +247,9 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-stream-active-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/stream/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/stream/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/stream/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} stream_v2_passive: @@ -250,6 +264,9 @@ services: QDL_STABLE_HTTP_PORT: "8200" QDL_STABLE_GRPC_PORT: "8210" QDL_STABLE_AUDIT_PATH: /var/lib/qdl-stable/runtime/stable-stream-passive-audit.jsonl + QDL_STABLE_TLS_CA_FILE: /stable-certs/stream/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/stream/server.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/stream/server.key depends_on: {stable_redis: {condition: service_healthy}, stable_state_init: {condition: service_completed_successfully}, stable_tls_init: {condition: service_completed_successfully}} projector_v2: @@ -267,7 +284,10 @@ services: QDL_STABLE_KAFKA_RAW_TOPICS: md.raw.stable.v1 QDL_STABLE_KAFKA_CANONICAL_TOPIC: md.canonical.v2 QDL_STABLE_KAFKA_CERT_ROOT: /stable-certs/projector - QDL_STABLE_STREAM_INGEST_URLS_JSON: '["http://stream_v2_active:8200","http://stream_v2_passive:8200"]' + QDL_STABLE_TLS_CA_FILE: /stable-certs/projector/ca.crt + QDL_STABLE_TLS_CERT_FILE: /stable-certs/projector/client.crt + QDL_STABLE_TLS_KEY_FILE: /stable-certs/projector/client.key + QDL_STABLE_STREAM_INGEST_URLS_JSON: '["https://stream_v2_active:8200","https://stream_v2_passive:8200"]' volumes: [stable_state:/var/lib/qdl-stable, stable_tls:/stable-certs:ro] depends_on: stable_tls_init: {condition: service_completed_successfully} diff --git a/poetry.lock b/poetry.lock index ba46b21..96ba270 100644 --- a/poetry.lock +++ b/poetry.lock @@ -665,6 +665,69 @@ mypy = ["bokeh", "contourpy[bokeh,docs]", "docutils-stubs", "mypy (==1.17.0)", " test = ["Pillow", "contourpy[test-no-images]", "matplotlib"] test-no-images = ["pytest", "pytest-cov", "pytest-rerunfailures", "pytest-xdist", "wurlitzer"] +[[package]] +name = "cryptography" +version = "50.0.0" +description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers." +optional = false +python-versions = "!=3.9.0,!=3.9.1,>=3.9" +groups = ["main"] +files = [ + {file = "cryptography-50.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:031e2d5dd4bb9caa3ca9c82e5a197fd8ae680232cee62603d1a813f3f07e3d03"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fd9192b7b70c573d7f214eb1ae35e00d359f6f5e4b27c7e21e30de1fc6204645"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:06a32a980526a6ab9a4b9bf8f7385800791e2bb960903cb6b530e4817509a3b7"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a1b30560f2acc95aa8b2e06e716a13dbfc97314747b80d9707e307f77b40d6b3"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:8d89f3976b10b4ce31118de72329025f70d2c6ead14a8217c5514dd2c6d5a78f"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:b42a28c1844fd9de8f3f7d540e36b66f3a9c83fceac7170ebc7a6a19edd9dcae"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:900131fafd8aead39ac7dd3a7e833be754c17a95cfd91221636949fe4eb0aa8a"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:07949c449a1abcf60d1ee6e88956d89404c7df3c8258f46589e912988e551987"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:f89831ef99dd7dd169ab06d63a831adb9e20a87aac6d380266bbda5823349169"}, + {file = "cryptography-50.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:82148ec5bddac30b51a5b3c1945075f896fa022cb93f8e4a01e9f6ee95292c5f"}, + {file = "cryptography-50.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1489e263a8048bb8b6a8bac662eb2d402ea5d2b7b4699b72f385f1e2772db105"}, + {file = "cryptography-50.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:7cec5b856506da6defb290f30c9ee687d5f5e8cb0bd3f6459dde43b0b4fa40ef"}, + {file = "cryptography-50.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30"}, + {file = "cryptography-50.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:49e7d93abdbd2990caced757e5fade25302f719c3c8fb6e6fff2dde98999fc41"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:19736989797678c6af1e55cd49055cdbcb55d8f6b5583ac5335f933aba9101dc"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:80b63928fa35083b33966ce1efb70e5b9607181e49dcd1c22c8c005e319f667f"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:d58c3db7cd6eed54e6c06744db55456b65ebd7492ddeae9c1e93cfca7aa857d3"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:df2a58a472f332225671c35b0a830208b86d004f82baa8530fa3782c85646533"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:11b74db56cdbe3cdee6e3f6982ecb70334fa10dce99ed58bf7894aaaa3b2a037"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f59e38625469987d7ef6d495323c55e7db6c212eaf6112267e0d3b565a2e9c9f"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:ecfed7367f965a0328cfbdd70da860f15441f002f613185668c6e6ebf5a0ac11"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:9aa87839c383bdbab6ef865787a1fb877af8dd03464c4400322726feaaadfc6d"}, + {file = "cryptography-50.0.0-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:6ba6a53445bd3cfa809ef3ef5f1589aa6ba08784a1d962bf47d0940e871dab1c"}, + {file = "cryptography-50.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3f5735ffe4996d28b809371756219f5354864902a3b9e7c0b9ee87041209fc9c"}, + {file = "cryptography-50.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1b4a266766514614f8aa60416e71f2fc6e575d36e7bdc90f644fadb2f4b75b95"}, + {file = "cryptography-50.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:12b9c6996425c76ea6c457ace4f3073e715b8c545add07cd1a8f3a4f90691269"}, + {file = "cryptography-50.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ccdc4a71a4dabae05de219404f9f4abc38e3b58422177ff93d0da05967dafa07"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:910e1d2668e7de9648f2bcee30e180db2a6b15c30f887d7c4c93ddf96e3992e3"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a91296cb61e8df6f86d0c19cc4068228da256bf59bf86049fbd821084565327f"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e722f16708d854fe924790e051061f6704a472c3bac347b6fd88033ea8dd0dc5"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:d764dcf130c428ef66786f866dd750f53182bc608813489915e9fc106bb0c82f"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:105110f43a471dbd0060b9c9516cb8a6a79233631a04cc2ba16f28323ac6e025"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:828743d939e9629bc267b8e2d08d8bb67cd4319c771a33d4b18b22dd8fb7440a"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:2a8183b489dc1f7f80f135780fadc1108f14b31b8a40411c7a5b17425f65f28b"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6e7d61120573a7f2cd94cc095f9e81f6967c61ccdf194285aa143ecec8e0b708"}, + {file = "cryptography-50.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:37fdb0d0111f1e2ff07139dfb79f1b49531f8e213c46f1163dd7642979b58c47"}, + {file = "cryptography-50.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c87f62a3d3b9888ed0fdde100ec06aa61ca9cd44bad9057d1dff9a516b5f5bb9"}, + {file = "cryptography-50.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:65c2c3add92b45fd0709db8594536aea39c2a67af0e27ffcf049c498501140b7"}, + {file = "cryptography-50.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:d24fead1d4d076e1bfb006dcec392074a3cd8d7b4fc8a595aa64073b2b7a96ba"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:5e34edd123674534acd70147f0ca331eaa2c74e6325fb2028c886aa26ba0b68c"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:8eb5e1172eb569ea8a872796576e6a67c276351728b6455d5beb01242b027c6a"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:910d11e1a385c654bf738bf3e6b8e6ed5de0f5610fcae2be9e5b398d8081d20e"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:62598a8a57f815db4c6259a4e97d857dab56697e7de8e8ab02352ab74da1995d"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:07479a1cb08219ab719147e742e76090c9c773321959bb94946fffdd397a6437"}, + {file = "cryptography-50.0.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:c99c003e088647b8a5b7c145d6f78c335f6348332b62e142d411c4b63d1460b9"}, + {file = "cryptography-50.0.0.tar.gz", hash = "sha256:eeac2acb5a20ed25e0ad6d1df9891a520b78b404266b6d11778f25d5d691a6c9"}, +] + +[package.dependencies] +cffi = {version = ">=2.0.0", markers = "platform_python_implementation != \"PyPy\""} +typing-extensions = {version = ">=4.13.2", markers = "python_full_version < \"3.11.0\""} + +[package.extras] +ssh = ["bcrypt (>=3.1.5)"] + [[package]] name = "cycler" version = "0.12.1" @@ -2071,6 +2134,7 @@ files = [ ] [package.dependencies] +cryptography = {version = ">=3.4.0", optional = true, markers = "extra == \"crypto\""} typing_extensions = {version = ">=4.0", markers = "python_version < \"3.11\""} [package.extras] @@ -2874,4 +2938,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.1" python-versions = "^3.10" -content-hash = "b5d9bb96e911f8d6124e04d78d461674b006d68bc5e66412936e29f77949e549" +content-hash = "f5d8c82d042f5dd6bd468543bb5d7f457532c2528daba5e77415d50d45c2bca2" diff --git a/pyproject.toml b/pyproject.toml index 9a649bb..f4c627c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -30,7 +30,7 @@ dependencies = [ "httpx (>=0.28.0,<1.0.0)", "idna (>=3.15,<4.0)", "pillow (>=12.3.0,<13.0.0)", - "pyjwt (>=2.13.0,<3.0.0)", + "pyjwt[crypto] (>=2.13.0,<3.0.0)", "soupsieve (>=2.8.4,<3.0.0)", "starlette (>=1.3.1,<2.0.0)" ] diff --git a/qdl/runtime/stable.py b/qdl/runtime/stable.py index 730632c..2d14a3e 100644 --- a/qdl/runtime/stable.py +++ b/qdl/runtime/stable.py @@ -5,6 +5,7 @@ import json import logging import os +import ssl import time from dataclasses import asdict, dataclass from pathlib import Path @@ -74,6 +75,9 @@ class StableRuntimeConfig: audit_path: Path manifest_paths: tuple[Path, ...] source_bindings_path: Path + tls_ca_path: Path + tls_certificate_path: Path + tls_private_key_path: Path internal_ingest_secret: bytes redis_url: str redis_prefix: str @@ -121,6 +125,13 @@ def __post_init__(self) -> None: raise ValueError("stable authority revision and consumer manifests are required") if not self.source_bindings_path.is_file(): raise ValueError("stable source binding catalog is unavailable") + missing_tls = [ + path for path in ( + self.tls_ca_path, self.tls_certificate_path, self.tls_private_key_path + ) if not path.is_file() + ] + if missing_tls: + raise ValueError("stable workload TLS files are unavailable") if len(self.internal_ingest_secret) < 32: raise ValueError("stable internal ingest secret must contain 256 bits") if self.active_cursor_key_id not in self.cursor_keys or any( @@ -190,6 +201,9 @@ def from_environment( )), manifest_paths=manifests, source_bindings_path=Path(env["QDL_STABLE_SOURCE_BINDINGS"]), + tls_ca_path=Path(env["QDL_STABLE_TLS_CA_FILE"]), + tls_certificate_path=Path(env["QDL_STABLE_TLS_CERT_FILE"]), + tls_private_key_path=Path(env["QDL_STABLE_TLS_KEY_FILE"]), internal_ingest_secret=env["QDL_STABLE_INTERNAL_INGEST_SECRET"].encode(), redis_url=env["QDL_STABLE_REDIS_URL"], redis_prefix=env["QDL_STABLE_REDIS_PREFIX"], @@ -238,6 +252,7 @@ def public_manifest(self) -> dict[str, object]: "compatibility_projection": "DEDICATED_REDIS_ONLY", "replay_authority": "KAFKA", "query_cache_authority": False, + "transport_security": "MTLS_PLUS_JWT", } @@ -248,6 +263,35 @@ def _ready(name: str, *, detail: str, revision: str | None = None) -> ComponentR ) +def stable_client_ssl_context(config: StableRuntimeConfig) -> ssl.SSLContext: + context = ssl.create_default_context(cafile=str(config.tls_ca_path)) + context.minimum_version = ssl.TLSVersion.TLSv1_2 + context.load_cert_chain( + certfile=str(config.tls_certificate_path), + keyfile=str(config.tls_private_key_path), + ) + return context + + +def stable_uvicorn_tls(config: StableRuntimeConfig) -> dict[str, object]: + return { + "ssl_keyfile": str(config.tls_private_key_path), + "ssl_certfile": str(config.tls_certificate_path), + "ssl_ca_certs": str(config.tls_ca_path), + "ssl_cert_reqs": ssl.CERT_REQUIRED, + } + + +def stable_grpc_server_credentials( + config: StableRuntimeConfig, +) -> grpc.ServerCredentials: + return grpc.ssl_server_credentials( + ((config.tls_private_key_path.read_bytes(), config.tls_certificate_path.read_bytes()),), + root_certificates=config.tls_ca_path.read_bytes(), + require_client_auth=True, + ) + + def load_stable_manifests(config: StableRuntimeConfig) -> ConsumerManifestRegistry: manifests = tuple(ConsumerManifestLoader.load(path) for path in config.manifest_paths) if any(item.environment != config.environment for item in manifests): @@ -420,7 +464,12 @@ class StableStreamRuntime: async def start(self) -> None: await self.redis.ping() await self.lease.start() - self.grpc_server.add_insecure_port(f"0.0.0.0:{self.config.grpc_port}") + bound = self.grpc_server.add_secure_port( + f"0.0.0.0:{self.config.grpc_port}", + stable_grpc_server_credentials(self.config), + ) + if bound != self.config.grpc_port: + raise RuntimeError("stable gRPC mTLS port binding failed") await self.grpc_server.start() async def stop(self) -> None: @@ -496,12 +545,24 @@ def create_stable_stream_runtime( ) +async def serve_stable_query() -> None: + config = StableRuntimeConfig.from_environment("query_v2") + app = create_stable_query_app(config) + server = uvicorn.Server(uvicorn.Config( + app, host="0.0.0.0", port=config.http_port, + log_level="info", access_log=False, + **stable_uvicorn_tls(config), + )) + await server.serve() + + async def serve_stable_stream() -> None: runtime = create_stable_stream_runtime() await runtime.start() server = uvicorn.Server(uvicorn.Config( runtime.health_app, host="0.0.0.0", port=runtime.config.http_port, log_level="info", access_log=False, + **stable_uvicorn_tls(runtime.config), )) try: await server.serve() @@ -536,7 +597,8 @@ async def serve_stable_projector() -> None: config.redis_url, prefix=f"{config.redis_prefix}:projector" ) sink = StableHttpCanonicalSink( - config.stream_ingest_urls, config.internal_ingest_secret, spool + config.stream_ingest_urls, config.internal_ingest_secret, spool, + ssl_context=stable_client_ssl_context(config), ) projector = StableCompatibilityProjector( catalog, namespace=config.redis_prefix.rstrip(":") diff --git a/qdl/runtime/stable_ingest.py b/qdl/runtime/stable_ingest.py index 6a60742..b06a735 100644 --- a/qdl/runtime/stable_ingest.py +++ b/qdl/runtime/stable_ingest.py @@ -6,6 +6,7 @@ import hmac import ipaddress import json +import ssl import uuid from dataclasses import dataclass, field from urllib.parse import urlsplit @@ -33,7 +34,7 @@ def _signature(secret: bytes, body: bytes) -> str: def _internal_url(value: str) -> bool: parsed = urlsplit(value) - if parsed.scheme != "http" or not parsed.hostname: + if parsed.scheme not in {"http", "https"} or not parsed.hostname: return False try: return ipaddress.ip_address(parsed.hostname).is_loopback @@ -212,6 +213,7 @@ class StableHttpCanonicalSink: spool: SQLiteDurableSpool timeout_seconds: float = 10.0 client: httpx.AsyncClient | None = None + ssl_context: ssl.SSLContext | None = None _owns_client: bool = field(init=False) def __post_init__(self) -> None: @@ -222,12 +224,17 @@ def __post_init__(self) -> None: or self.timeout_seconds <= 0 ): raise ValueError("stable HTTP sink configuration is invalid") + if any(urlsplit(value).scheme == "https" for value in self.urls) and self.ssl_context is None and self.client is None: + raise ValueError("stable HTTPS sink requires a workload TLS context") + if self.client is not None and self.ssl_context is not None: + raise ValueError("stable HTTP sink client and TLS context are mutually exclusive") self._owns_client = self.client is None if self.client is None: self.client = httpx.AsyncClient( follow_redirects=False, limits=httpx.Limits(max_connections=10, max_keepalive_connections=5), timeout=self.timeout_seconds, + verify=self.ssl_context or True, ) async def publish(self, event: DurableEvent) -> StoredEvent: diff --git a/qdl_sdk/__init__.py b/qdl_sdk/__init__.py index 2270434..e66b085 100644 --- a/qdl_sdk/__init__.py +++ b/qdl_sdk/__init__.py @@ -6,6 +6,7 @@ from qdl_sdk.credentials import ( CallbackCredentialProvider, CredentialProvider, + RotatingJwtCredentialProvider, StaticBearerCredential, ) from qdl_sdk.cursor import CursorCheckpoint, FileCursorStore, MemoryCursorStore @@ -31,6 +32,7 @@ ) from qdl_sdk.projection import market_data_view_from_stream from qdl_sdk.transport import GrpcStreamTransport, RestQueryTransport +from qdl_sdk.tls import WorkloadTlsConfig from qdl_sdk.v1_facade import V1CompatibilityFacade __all__ = [ @@ -58,12 +60,14 @@ "QuantityUnit", "RestQueryTransport", "RecoveryPolicy", + "RotatingJwtCredentialProvider", "SnapshotResponse", "StalePolicy", "StaticBearerCredential", "StreamEvent", "TradeIdentityKind", "WarmupResponse", + "WorkloadTlsConfig", "V1CompatibilityFacade", "WarmupStreamSession", ] diff --git a/qdl_sdk/credentials.py b/qdl_sdk/credentials.py index 4a459b4..609e749 100644 --- a/qdl_sdk/credentials.py +++ b/qdl_sdk/credentials.py @@ -1,7 +1,13 @@ from __future__ import annotations +import asyncio import inspect +import time +import uuid from dataclasses import dataclass +from pathlib import Path + +import jwt from typing import Awaitable, Callable, Protocol @@ -33,3 +39,84 @@ async def get_token(self) -> str: if not isinstance(token, str) or not token.strip(): raise RuntimeError("workload credential provider returned an empty token") return token + + +class RotatingJwtCredentialProvider: + """Signs bounded workload JWTs locally and reloads the private key on refresh.""" + + def __init__( + self, + *, + private_key_file: str | Path, + key_id: str, + algorithm: str, + issuer: str, + audience: str, + subject: str, + environment: str, + roles: tuple[str, ...], + venues: tuple[str, ...] = (), + consumer_manifest_revision: int = 1, + lifetime_seconds: int = 600, + refresh_before_seconds: int = 120, + clock=time.time, + ) -> None: + self.private_key_file = Path(private_key_file).expanduser().resolve() + self.key_id = key_id.strip() + self.algorithm = algorithm.strip().upper() + self.issuer = issuer.strip() + self.audience = audience.strip() + self.subject = subject.strip() + self.environment = environment.strip().lower() + self.roles = tuple(sorted(set(roles))) + self.venues = tuple(sorted({value.upper() for value in venues})) + self.consumer_manifest_revision = int(consumer_manifest_revision) + self.lifetime_seconds = int(lifetime_seconds) + self.refresh_before_seconds = int(refresh_before_seconds) + self._clock = clock + self._token = "" + self._expires_at = 0 + self._lock = asyncio.Lock() + if ( + not self.private_key_file.is_file() + or not all(( + self.key_id, self.issuer, self.audience, + self.subject, self.environment, + )) + or self.algorithm not in {"RS256", "ES256"} + or not self.roles + or self.consumer_manifest_revision < 1 + or not 60 <= self.lifetime_seconds <= 900 + or not 5 <= self.refresh_before_seconds < self.lifetime_seconds + ): + raise ValueError("rotating workload JWT configuration is invalid") + + async def get_token(self) -> str: + now = int(self._clock()) + if self._token and self._expires_at - now > self.refresh_before_seconds: + return self._token + async with self._lock: + now = int(self._clock()) + if self._token and self._expires_at - now > self.refresh_before_seconds: + return self._token + expires_at = now + self.lifetime_seconds + claims = { + "sub": self.subject, + "iss": self.issuer, + "aud": self.audience, + "iat": now, + "exp": expires_at, + "jti": str(uuid.uuid4()), + "environment": self.environment, + "roles": list(self.roles), + "venues": list(self.venues), + "consumer_manifest_revision": self.consumer_manifest_revision, + } + self._token = jwt.encode( + claims, + self.private_key_file.read_bytes(), + algorithm=self.algorithm, + headers={"kid": self.key_id}, + ) + self._expires_at = expires_at + return self._token diff --git a/qdl_sdk/tls.py b/qdl_sdk/tls.py new file mode 100644 index 0000000..8ba1a63 --- /dev/null +++ b/qdl_sdk/tls.py @@ -0,0 +1,50 @@ +from __future__ import annotations + +import ssl +from dataclasses import dataclass +from pathlib import Path + +import grpc + + +@dataclass(frozen=True, slots=True) +class WorkloadTlsConfig: + """Mutual-TLS identity shared by the REST and gRPC transports.""" + + ca_file: str | Path + certificate_file: str | Path + private_key_file: str | Path + + def __post_init__(self) -> None: + paths = self.paths() + missing = [name for name, path in paths.items() if not path.is_file()] + if missing: + raise ValueError( + "Data Layer workload TLS files are unavailable: " + + ",".join(sorted(missing)) + ) + + def paths(self) -> dict[str, Path]: + return { + "ca_file": Path(self.ca_file).expanduser().resolve(), + "certificate_file": Path(self.certificate_file).expanduser().resolve(), + "private_key_file": Path(self.private_key_file).expanduser().resolve(), + } + + def ssl_context(self) -> ssl.SSLContext: + paths = self.paths() + context = ssl.create_default_context(cafile=str(paths["ca_file"])) + context.minimum_version = ssl.TLSVersion.TLSv1_2 + context.load_cert_chain( + certfile=str(paths["certificate_file"]), + keyfile=str(paths["private_key_file"]), + ) + return context + + def grpc_credentials(self) -> grpc.ChannelCredentials: + paths = self.paths() + return grpc.ssl_channel_credentials( + root_certificates=paths["ca_file"].read_bytes(), + private_key=paths["private_key_file"].read_bytes(), + certificate_chain=paths["certificate_file"].read_bytes(), + ) diff --git a/qdl_sdk/transport.py b/qdl_sdk/transport.py index ff8601a..92059f7 100644 --- a/qdl_sdk/transport.py +++ b/qdl_sdk/transport.py @@ -1,7 +1,7 @@ from __future__ import annotations import ipaddress -from collections.abc import AsyncIterator +from collections.abc import AsyncIterator, Sequence from urllib.parse import urlsplit import grpc @@ -11,6 +11,7 @@ from qdl_sdk.credentials import CredentialProvider from qdl_sdk.errors import CursorExpiredError, DataLayerError, SlowConsumerError from qdl_sdk.models import ControlEvent, DataRequirement, Grade, StreamEvent +from qdl_sdk.tls import WorkloadTlsConfig class RestQueryTransport: @@ -21,16 +22,20 @@ def __init__( timeout_seconds: float = 10.0, client: httpx.AsyncClient | None = None, credential_provider: CredentialProvider | None = None, + tls: WorkloadTlsConfig | None = None, ) -> None: if timeout_seconds <= 0: raise ValueError("query timeout must be positive") self.base_url = base_url.rstrip("/") + if client is not None and tls is not None: + raise ValueError("provide either REST client or workload TLS, not both") self._owns_client = client is None self._credential_provider = credential_provider self._client = client or httpx.AsyncClient( base_url=self.base_url, timeout=httpx.Timeout(timeout_seconds), limits=httpx.Limits(max_connections=100, max_keepalive_connections=20), + verify=tls.ssl_context() if tls is not None else True, ) async def warmup(self, requirement: DataRequirement, *, consumer_id: str) -> dict: @@ -139,26 +144,46 @@ def _decode(response: httpx.Response) -> dict: class GrpcStreamTransport: def __init__( self, - target: str, + target: str | Sequence[str], *, credentials: grpc.ChannelCredentials | None = None, + tls: WorkloadTlsConfig | None = None, allow_insecure_loopback: bool = False, credential_provider: CredentialProvider | None = None, ) -> None: - if not target.strip(): - raise ValueError("gRPC stream target is required") - self.target = target + raw_targets = (target,) if isinstance(target, str) else tuple(target) + targets = tuple( + value.strip() + for item in raw_targets + for value in item.split(",") + if value.strip() + ) + if not targets or len(set(targets)) != len(targets): + raise ValueError("gRPC stream targets must be non-empty and unique") + self.targets = targets + self.target = targets[0] + self._target_index = 0 self._credential_provider = credential_provider + if credentials is not None and tls is not None: + raise ValueError("provide either credentials or workload TLS, not both") + credentials = credentials or (tls.grpc_credentials() if tls is not None else None) if credentials is None: - if not allow_insecure_loopback or not self._is_loopback(target): + if not allow_insecure_loopback or not all( + self._is_loopback(value) for value in targets + ): raise ValueError("insecure gRPC is allowed only for explicit loopback tests") - self._channel = grpc.aio.insecure_channel(target) + self._channels = tuple(grpc.aio.insecure_channel(value) for value in targets) else: - self._channel = grpc.aio.secure_channel(target, credentials) - self._subscribe = self._channel.unary_stream( - "/qdl.query.v2.MarketDataStreamService/Subscribe", - request_serializer=query_pb2.SubscribeRequest.SerializeToString, - response_deserializer=query_pb2.SubscribeResponse.FromString, + self._channels = tuple( + grpc.aio.secure_channel(value, credentials) for value in targets + ) + self._subscribes = tuple( + channel.unary_stream( + "/qdl.query.v2.MarketDataStreamService/Subscribe", + request_serializer=query_pb2.SubscribeRequest.SerializeToString, + response_deserializer=query_pb2.SubscribeResponse.FromString, + ) + for channel in self._channels ) async def subscribe( @@ -188,7 +213,8 @@ async def subscribe( ("x-qdl-purpose", RestQueryTransport._purpose(requirement)), ) try: - async for response in self._subscribe(request, metadata=metadata): + subscribe = self._subscribes[self._target_index] + async for response in subscribe(request, metadata=metadata): record = response.record payload = record.WhichOneof("payload") if payload == "control": @@ -202,6 +228,9 @@ async def subscribe( yield StreamEvent(record.logical_offset, record.resume_token, record.event) except grpc.aio.AioRpcError as error: detail = error.details() or "gRPC stream failed" + if error.code() is grpc.StatusCode.UNAVAILABLE and len(self.targets) > 1: + self._target_index = (self._target_index + 1) % len(self.targets) + self.target = self.targets[self._target_index] if error.code() is grpc.StatusCode.OUT_OF_RANGE: raise CursorExpiredError("CURSOR_EXPIRED", detail, retryable=False) from error if error.code() is grpc.StatusCode.RESOURCE_EXHAUSTED: @@ -216,7 +245,8 @@ async def subscribe( raise DataLayerError("DEPENDENCY_UNAVAILABLE", detail, retryable=True) from error async def close(self) -> None: - await self._channel.close() + for channel in self._channels: + await channel.close() @staticmethod def _is_loopback(target: str) -> bool: diff --git a/scripts/build_qdl_sdk_release.py b/scripts/build_qdl_sdk_release.py index 615fa86..2b3abd5 100755 --- a/scripts/build_qdl_sdk_release.py +++ b/scripts/build_qdl_sdk_release.py @@ -22,6 +22,7 @@ "httpx>=0.28.0,<1.0.0", "protobuf>=6.31.1,<7.0.0", "pydantic>=2.0.0,<3.0.0", + "PyJWT[crypto]>=2.13.0,<3.0.0", ) FIXED_TIMESTAMP = (2020, 1, 1, 0, 0, 0) diff --git a/scripts/phase80_generate_tls.sh b/scripts/phase80_generate_tls.sh index cdb45c0..699fa56 100755 --- a/scripts/phase80_generate_tls.sh +++ b/scripts/phase80_generate_tls.sh @@ -22,13 +22,14 @@ issue_certificate() { local principal="$1" local dns_name="$2" local extension_file="${OUTPUT_DIR}/${principal}.ext" + local san_entries="${3:-DNS:${dns_name}}" openssl genrsa -out "${OUTPUT_DIR}/${principal}.key" 2048 >/dev/null 2>&1 openssl req -new -sha256 \ -key "${OUTPUT_DIR}/${principal}.key" \ -subj "/CN=${principal}" \ -out "${OUTPUT_DIR}/${principal}.csr" >/dev/null 2>&1 - printf 'subjectAltName=DNS:%s,DNS:localhost\nextendedKeyUsage=serverAuth,clientAuth\n' "${dns_name}" >"${extension_file}" + printf 'subjectAltName=%s,DNS:localhost\nextendedKeyUsage=serverAuth,clientAuth\n' "${san_entries}" >"${extension_file}" openssl x509 -req -sha256 -days 2 \ -in "${OUTPUT_DIR}/${principal}.csr" \ -CA "${OUTPUT_DIR}/ca.crt" \ @@ -48,9 +49,14 @@ issue_certificate() { for broker in kafka1 kafka2 kafka3; do issue_certificate "${broker}" "${broker}" done -for client in phase8-admin phase8-producer phase8-consumer phase8-core phase8-unauthorized; do +for client in phase8-admin phase8-producer phase8-consumer phase8-core phase8-unauthorized stable-trading-system; do issue_certificate "${client}" "${client}" done +issue_certificate stable-trading-system-jwt stable-trading-system-jwt +openssl pkey -in "${OUTPUT_DIR}/stable-trading-system-jwt.key" -pubout \ + -out "${OUTPUT_DIR}/stable-trading-system-jwt.public.pem" >/dev/null 2>&1 +issue_certificate stable-query query_v2_1 "DNS:query_v2_1,DNS:query_v2_2,DNS:qdl-v2-query" +issue_certificate stable-stream stream_v2_active "DNS:stream_v2_active,DNS:stream_v2_passive,DNS:qdl-v2-stream" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/key.password" printf '%s\n' "${PASSWORD}" >"${OUTPUT_DIR}/store.password" @@ -108,6 +114,10 @@ find "${OUTPUT_DIR}" -maxdepth 1 -name '*.key' \ ! -name 'phase8-producer.key' \ ! -name 'phase8-consumer.key' \ ! -name 'phase8-core.key' \ + ! -name 'stable-query.key' \ + ! -name 'stable-stream.key' \ + ! -name 'stable-trading-system.key' \ + ! -name 'stable-trading-system-jwt.key' \ -delete rm -f "${OUTPUT_DIR}"/*.csr "${OUTPUT_DIR}"/*.ext "${OUTPUT_DIR}"/*.srl diff --git a/scripts/phaseb_prepare_stable_candidate.py b/scripts/phaseb_prepare_stable_candidate.py index 3112846..ed3693d 100755 --- a/scripts/phaseb_prepare_stable_candidate.py +++ b/scripts/phaseb_prepare_stable_candidate.py @@ -50,6 +50,21 @@ def copy_client_identity(source: Path, destination: Path, principal: str) -> Non item.chmod(0o440) +def copy_server_identity(source: Path, destination: Path, principal: str) -> None: + destination.mkdir(parents=True, exist_ok=False) + for source_name, target_name in ( + ("ca.crt", "ca.crt"), + (f"{principal}.crt", "server.crt"), + (f"{principal}.key", "server.key"), + ): + origin = source / source_name + if not origin.is_file(): + raise FileNotFoundError(f"stable TLS source is unavailable: {origin}") + shutil.copyfile(origin, destination / target_name) + for item in destination.iterdir(): + item.chmod(0o440) + + def prepare_candidate( *, rust_image: str, @@ -94,15 +109,30 @@ def prepare_candidate( ("producer", "phase8-producer"), ("core", "phase8-core"), ("projector", "phase8-consumer"), + ("trading-system", "stable-trading-system"), ): copy_client_identity(cert_dir, identities_dir / role, principal) + copy_server_identity(cert_dir, identities_dir / "query", "stable-query") + copy_server_identity(cert_dir, identities_dir / "stream", "stable-stream") + jwt_identity_dir = identities_dir / "trading-system-jwt" + jwt_identity_dir.mkdir(parents=True, exist_ok=False) + for source_name, target_name in ( + ("stable-trading-system-jwt.key", "private.key"), + ("stable-trading-system-jwt.public.pem", "public.pem"), + ): + origin = cert_dir / source_name + if not origin.is_file(): + raise FileNotFoundError(f"stable JWT source is unavailable: {origin}") + shutil.copyfile(origin, jwt_identity_dir / target_name) + for item in jwt_identity_dir.iterdir(): + item.chmod(0o440) schema_digest = hashlib.sha256( (ROOT / "contracts/proto/qdl/marketdata/v2/market_data.proto").read_bytes() ).hexdigest() ingest_secret = secrets.token_urlsafe(48) cursor_secret = secrets.token_urlsafe(48) - jwt_secret = secrets.token_urlsafe(48) + jwt_public_key = (jwt_identity_dir / "public.pem").read_text(encoding="utf-8") compose_cert_dir = (host_cert_dir or cert_dir).resolve() compose_output_dir = (host_output_dir or output_dir).resolve() values = { @@ -112,7 +142,8 @@ def prepare_candidate( {"stable-k1": cursor_secret}, separators=(",", ":") ), "QDL_STABLE_JWT_KEYS_JSON": json.dumps( - {"stable-jwt-k1": jwt_secret}, separators=(",", ":") + {"stable-trading-system-rs256-v1": jwt_public_key}, + separators=(",", ":"), ), "QDL_STABLE_PYTHON_IMAGE": python_digest, "QDL_STABLE_RUST_IMAGE": rust_digest, @@ -124,6 +155,14 @@ def prepare_candidate( "QDL_STABLE_PRODUCER_CERT_DIR": str( compose_output_dir / "identities/producer" ), + "QDL_STABLE_QUERY_CERT_DIR": str(compose_output_dir / "identities/query"), + "QDL_STABLE_STREAM_CERT_DIR": str(compose_output_dir / "identities/stream"), + "QDL_STABLE_TRADING_SYSTEM_CERT_DIR": str( + compose_output_dir / "identities/trading-system" + ), + "QDL_STABLE_TRADING_SYSTEM_JWT_PRIVATE_KEY": str( + compose_output_dir / "identities/trading-system-jwt/private.key" + ), "QDL_STABLE_RUNTIME_DIR": str(compose_output_dir / "runtime"), } env_path = output_dir / "stable.env" @@ -146,6 +185,8 @@ def prepare_candidate( "catalog_revision": catalog.catalog_revision, "acquisition_revision": acquisition.revision, "consumer_count": 5, + "workload_mtls": True, + "workload_identity_count": 3, "secret_values_recorded": False, } manifest_path = output_dir / "candidate-manifest.json" diff --git a/scripts/phasec1_isolated_consumer_acceptance.py b/scripts/phasec1_isolated_consumer_acceptance.py index feb8457..960dc91 100755 --- a/scripts/phasec1_isolated_consumer_acceptance.py +++ b/scripts/phasec1_isolated_consumer_acceptance.py @@ -26,6 +26,7 @@ RestQueryTransport, StalePolicy, StaticBearerCredential, + WorkloadTlsConfig, StreamEvent, market_data_view_from_stream, ) @@ -44,23 +45,23 @@ class VenueCase: VenueCase( "BINANCE", "a953e16e-7138-5562-b5e8-c337a44d0b65", - "alpha.binance.paper.stable", - "spiffe://qdl/paper/alpha-binance-stable", + "trading-system.paper.stable", + "spiffe://qdl/paper/trading-system-stable", "BINANCE_DIRECT", ), VenueCase( "OKX", "fb26214c-7b9b-5961-95b2-55154755af0f", - "alpha.okx.paper.stable", - "spiffe://qdl/paper/alpha-okx-stable", + "trading-system.paper.stable", + "spiffe://qdl/paper/trading-system-stable", "OKX_DIRECT", ), ) def token(subject: str, *, issuer: str, audience: str) -> str: - keys = json.loads(os.environ["QDL_STABLE_JWT_KEYS_JSON"]) - key_id, secret = sorted(keys.items())[0] + private_key_path = Path(os.environ["QDL_STABLE_JWT_PRIVATE_KEY_FILE"]) + key_id = os.environ["QDL_STABLE_JWT_KEY_ID"] now = int(time.time()) return jwt.encode( { @@ -79,8 +80,8 @@ def token(subject: str, *, issuer: str, audience: str) -> str: ], "consumer_manifest_revision": 1, }, - secret, - algorithm="HS256", + private_key_path.read_bytes(), + algorithm="RS256", headers={"kid": key_id}, ) @@ -92,6 +93,7 @@ def client( consumer_id: str, bearer: str, cursor_path: Path, + tls: WorkloadTlsConfig, ) -> AsyncDataLayerClient: credential = StaticBearerCredential(bearer) return AsyncDataLayerClient( @@ -99,10 +101,11 @@ def client( base_url, timeout_seconds=8, credential_provider=credential, + tls=tls, ), stream_transport=GrpcStreamTransport( grpc_target, - allow_insecure_loopback=True, + tls=tls, credential_provider=credential, ), consumer_id=consumer_id, @@ -160,12 +163,13 @@ async def certify_case( issuer: str, audience: str, state_dir: Path, + tls: WorkloadTlsConfig, ) -> dict[str, object]: bearer = token(case.subject, issuer=issuer, audience=audience) bar_requirement = DataRequirement( case.instrument_uid, Feed.BAR, - Grade.ALPHA, + Grade.EXECUTION, "crypto_primary_v2", interval="1m", warmup_limit=5, @@ -180,6 +184,7 @@ async def certify_case( consumer_id=case.consumer_id, bearer=bearer, cursor_path=state_dir / f"{case.venue.lower()}-query-primary.json", + tls=tls, ) secondary = client( base_url=secondary_url, @@ -187,6 +192,7 @@ async def certify_case( consumer_id=case.consumer_id, bearer=bearer, cursor_path=state_dir / f"{case.venue.lower()}-query-secondary.json", + tls=tls, ) try: first = await primary.warmup(bar_requirement) @@ -202,7 +208,7 @@ async def certify_case( trade_requirement = DataRequirement( case.instrument_uid, Feed.TRADE, - Grade.ALPHA, + Grade.EXECUTION, "crypto_primary_v2", warmup_limit=0, max_freshness_ms=15_000, @@ -217,6 +223,7 @@ async def certify_case( consumer_id=case.consumer_id, bearer=bearer, cursor_path=cursor_path, + tls=tls, ) try: async with first_client.warmup_then_stream(trade_requirement) as session: @@ -243,6 +250,7 @@ async def certify_case( consumer_id=case.consumer_id, bearer=bearer, cursor_path=cursor_path, + tls=tls, ) try: async with resumed_client.warmup_then_stream( @@ -290,6 +298,11 @@ async def certify_case( async def run(args: argparse.Namespace) -> dict[str, object]: + tls = WorkloadTlsConfig( + args.tls_ca_file, + args.tls_certificate_file, + args.tls_private_key_file, + ) with tempfile.TemporaryDirectory(prefix="qdl-c1-sdk-") as temporary: state_dir = Path(temporary) results = [] @@ -303,6 +316,7 @@ async def run(args: argparse.Namespace) -> dict[str, object]: issuer=args.issuer, audience=args.audience, state_dir=state_dir, + tls=tls, ) ) return { @@ -317,16 +331,24 @@ async def run(args: argparse.Namespace) -> dict[str, object]: def main() -> int: parser = argparse.ArgumentParser() - parser.add_argument("--primary-url", default="http://127.0.0.1:18201") - parser.add_argument("--secondary-url", default="http://127.0.0.1:18202") + parser.add_argument("--primary-url", default="https://localhost:18201") + parser.add_argument("--secondary-url", default="https://localhost:18202") parser.add_argument("--grpc-target", required=True) parser.add_argument( "--issuer", default="https://identity.qdl.stable.internal" ) parser.add_argument("--audience", default="qdl-v2-stable") + parser.add_argument("--tls-ca-file", required=True) + parser.add_argument("--tls-certificate-file", required=True) + parser.add_argument("--tls-private-key-file", required=True) args = parser.parse_args() - if "QDL_STABLE_JWT_KEYS_JSON" not in os.environ: - raise SystemExit("QDL_STABLE_JWT_KEYS_JSON is required") + required_env = { + "QDL_STABLE_JWT_PRIVATE_KEY_FILE", + "QDL_STABLE_JWT_KEY_ID", + } + missing = sorted(required_env - os.environ.keys()) + if missing: + raise SystemExit(f"required JWT signer environment is missing: {missing}") print(json.dumps(asyncio.run(run(args)), sort_keys=True)) return 0 diff --git a/scripts/rebuild_v2_stable_projection_cache.py b/scripts/rebuild_v2_stable_projection_cache.py index a544fa4..988d327 100755 --- a/scripts/rebuild_v2_stable_projection_cache.py +++ b/scripts/rebuild_v2_stable_projection_cache.py @@ -3,6 +3,7 @@ import argparse import json +import ssl import subprocess import time import urllib.request @@ -166,11 +167,42 @@ def _validate_project(env_file: Path) -> None: raise RuntimeError("compose project is not the isolated stable candidate") -def _wait_http(url: str, deadline: float) -> None: +def _env_value(env_file: Path, key: str) -> str: + prefix = f"{key}=" + matches = [ + line[len(prefix):] + for line in env_file.read_text().splitlines() + if line.startswith(prefix) + ] + if len(matches) != 1 or not matches[0]: + raise ValueError(f"stable env must define exactly one {key}") + return matches[0] + + +def _stable_client_ssl_context(env_file: Path) -> ssl.SSLContext: + identity_root = Path( + _env_value(env_file, "QDL_STABLE_TRADING_SYSTEM_CERT_DIR") + ) + context = ssl.create_default_context(cafile=str(identity_root / "ca.crt")) + context.load_cert_chain( + certfile=str(identity_root / "client.crt"), + keyfile=str(identity_root / "client.key"), + ) + return context + + +def _wait_http( + url: str, + deadline: float, + *, + ssl_context: ssl.SSLContext | None = None, +) -> None: last_error: BaseException | None = None while time.monotonic() < deadline: try: - with urllib.request.urlopen(url, timeout=2) as response: + with urllib.request.urlopen( + url, timeout=2, context=ssl_context + ) as response: if response.status == 200: return except BaseException as error: @@ -298,17 +330,34 @@ def execute_rebuild(env_file: Path, *, timeout_seconds: float) -> dict[str, obje "--to-earliest", "--execute", ) + ssl_context = _stable_client_ssl_context(env_file) _start_services(env_file, *STREAM_SERVICES) - _wait_http("http://127.0.0.1:18210/health/live", deadline) - _wait_http("http://127.0.0.1:18211/health/live", deadline) + _wait_http( + "https://localhost:18210/health/live", + deadline, + ssl_context=ssl_context, + ) + _wait_http( + "https://localhost:18211/health/live", + deadline, + ssl_context=ssl_context, + ) _start_services(env_file, "projector_v2") lag = _wait_bounded_lag(env_file, deadline) _wait_projector_ready(env_file, deadline) _start_services(env_file, *QUERY_SERVICES) - _wait_http("http://127.0.0.1:18201/health/ready", deadline) - _wait_http("http://127.0.0.1:18202/health/ready", deadline) + _wait_http( + "https://localhost:18201/health/ready", + deadline, + ssl_context=ssl_context, + ) + _wait_http( + "https://localhost:18202/health/ready", + deadline, + ssl_context=ssl_context, + ) final_size = int( _compose( diff --git a/tests/test_phaseb_stable_deployment.py b/tests/test_phaseb_stable_deployment.py index 99ab11a..47eb866 100644 --- a/tests/test_phaseb_stable_deployment.py +++ b/tests/test_phaseb_stable_deployment.py @@ -6,7 +6,6 @@ import tempfile import time import unittest -from datetime import datetime, timezone from unittest.mock import AsyncMock, patch from pathlib import Path @@ -814,9 +813,22 @@ def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): with tempfile.TemporaryDirectory(prefix="qdl-phaseb-cert-") as cert_directory: certs = Path(cert_directory) (certs / "ca.crt").write_text("ca", encoding="ascii") - for principal in ("phase8-producer", "phase8-core", "phase8-consumer"): + for principal in ( + "phase8-producer", + "phase8-core", + "phase8-consumer", + "stable-trading-system", + "stable-query", + "stable-stream", + ): (certs / f"{principal}.crt").write_text("crt", encoding="ascii") (certs / f"{principal}.key").write_text("key", encoding="ascii") + (certs / "stable-trading-system-jwt.key").write_text( + "private", encoding="ascii" + ) + (certs / "stable-trading-system-jwt.public.pem").write_text( + "public", encoding="ascii" + ) with tempfile.TemporaryDirectory(prefix="qdl-phaseb-output-") as parent: output = Path(parent) / "candidate" with patch( diff --git a/tests/test_phaseb_stable_edge.py b/tests/test_phaseb_stable_edge.py index a17df9c..19d8f05 100644 --- a/tests/test_phaseb_stable_edge.py +++ b/tests/test_phaseb_stable_edge.py @@ -34,7 +34,7 @@ ProductType, ) from qdl.domain.decimal import CanonicalDecimal -from qdl.query import ConsumerGrade, DataRequirement, FeedType +from qdl.query import ConsumerGrade, DataRequirement from qdl.projection.stable import ( InMemoryStableProjectionTarget, ProjectionCacheMismatch, @@ -1319,6 +1319,8 @@ async def test_stale_projection_epoch_fails_without_broker_checkpoint(self): class StableRuntimeBoundaryTests(unittest.TestCase): def environment(self, root): + for name in ("ca.crt", "workload.crt", "workload.key"): + (root / name).write_text("test", encoding="utf-8") return { "QDL_ENVIRONMENT": "paper", "QDL_CONFIG_REVISION": "phase-b-test-1", @@ -1330,6 +1332,9 @@ def environment(self, root): "QDL_STABLE_AUDIT_PATH": str(root / "state" / "audit.jsonl"), "QDL_STABLE_CONSUMER_MANIFESTS": str(root / "consumer.yaml"), "QDL_STABLE_SOURCE_BINDINGS": str(CATALOG_PATH), + "QDL_STABLE_TLS_CA_FILE": str(root / "ca.crt"), + "QDL_STABLE_TLS_CERT_FILE": str(root / "workload.crt"), + "QDL_STABLE_TLS_KEY_FILE": str(root / "workload.key"), "QDL_STABLE_INTERNAL_INGEST_SECRET": "i" * 32, "QDL_STABLE_REDIS_URL": "redis://qdl-stable-redis:6379/0", "QDL_STABLE_REDIS_PREFIX": "qdl:stable:v2:paper:test", @@ -1346,9 +1351,13 @@ def test_projector_sink_accepts_only_declared_internal_stream_roles(self): secret, object(), client=client, ) self.assertIs(sink.client, client) - with self.assertRaisesRegex(ValueError, "configuration is invalid"): + secure = StableHttpCanonicalSink( + ("https://stream_v2_active:8200",), secret, object(), client=client + ) + self.assertIs(secure.client, client) + with self.assertRaisesRegex(ValueError, "workload TLS context"): StableHttpCanonicalSink( - ("https://stream_v2_active:8200",), secret, object(), client=client + ("https://stream_v2_active:8200",), secret, object() ) with self.assertRaisesRegex(ValueError, "configuration is invalid"): StableHttpCanonicalSink( diff --git a/tests/test_phaseb_stable_rebuild.py b/tests/test_phaseb_stable_rebuild.py index a086720..6e56e64 100644 --- a/tests/test_phaseb_stable_rebuild.py +++ b/tests/test_phaseb_stable_rebuild.py @@ -17,6 +17,8 @@ QUERY_SERVICES, STOP_SERVICES, STREAM_SERVICES, + _env_value, + _stable_client_ssl_context, _start_services, _validate_project, compose_command, @@ -60,6 +62,28 @@ def test_apply_requires_exact_confirmation(self): require_authorization(apply=True, confirm="WRONG") require_authorization(apply=True, confirm=CONFIRM_TOKEN) + def test_tls_identity_is_loaded_from_exact_env_binding(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + identity = root / "identity" + identity.mkdir() + env = root / "stable.env" + env.write_text( + f"QDL_STABLE_TRADING_SYSTEM_CERT_DIR={identity}\n" + ) + self.assertEqual( + _env_value(env, "QDL_STABLE_TRADING_SYSTEM_CERT_DIR"), + str(identity), + ) + with self.assertRaises(FileNotFoundError): + _stable_client_ssl_context(env) + env.write_text( + "QDL_STABLE_TRADING_SYSTEM_CERT_DIR=/first\n" + "QDL_STABLE_TRADING_SYSTEM_CERT_DIR=/second\n" + ) + with self.assertRaisesRegex(ValueError, "exactly one"): + _env_value(env, "QDL_STABLE_TRADING_SYSTEM_CERT_DIR") + def test_compose_command_is_pinned_to_stable_manifest(self): command = compose_command(Path("/tmp/stable.env"), "config") self.assertEqual(command[:2], ["docker", "compose"]) diff --git a/tests/test_phasec2_transport_security.py b/tests/test_phasec2_transport_security.py new file mode 100644 index 0000000..fa5d306 --- /dev/null +++ b/tests/test_phasec2_transport_security.py @@ -0,0 +1,141 @@ +from __future__ import annotations + +import ssl +import tempfile +import time +import unittest +from pathlib import Path +from unittest import mock + +import grpc +import httpx + +from qdl_sdk import GrpcStreamTransport, RestQueryTransport, WorkloadTlsConfig + + +class WorkloadTlsConfigTests(unittest.IsolatedAsyncioTestCase): + def identity(self, root: Path) -> WorkloadTlsConfig: + paths = [] + for name in ("ca.crt", "client.crt", "client.key"): + path = root / name + path.write_text("test-only", encoding="utf-8") + paths.append(path) + return WorkloadTlsConfig(*paths) + + async def test_missing_identity_fails_closed(self): + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + with self.assertRaisesRegex(ValueError, "TLS files are unavailable"): + WorkloadTlsConfig( + root / "missing-ca", + root / "missing-cert", + root / "missing-key", + ) + + async def test_rest_and_grpc_share_one_workload_identity(self): + with tempfile.TemporaryDirectory() as temp: + tls = self.identity(Path(temp)) + context = ssl.create_default_context() + credentials = grpc.ssl_channel_credentials() + with ( + mock.patch.object( + WorkloadTlsConfig, "ssl_context", return_value=context + ) as rest_tls, + mock.patch.object( + WorkloadTlsConfig, + "grpc_credentials", + return_value=credentials, + ) as grpc_tls, + ): + rest = RestQueryTransport("https://qdl-v2-query:8200", tls=tls) + stream = GrpcStreamTransport( + ("qdl-v2-stream-a:8210", "qdl-v2-stream-b:8210"), + tls=tls, + ) + try: + self.assertEqual( + stream.targets, + ("qdl-v2-stream-a:8210", "qdl-v2-stream-b:8210"), + ) + self.assertEqual(stream.target, "qdl-v2-stream-a:8210") + rest_tls.assert_called_once_with() + grpc_tls.assert_called_once_with() + finally: + await rest.close() + await stream.close() + + + async def test_rs256_token_is_cached_rotated_and_verified(self): + from cryptography.hazmat.primitives import serialization + from cryptography.hazmat.primitives.asymmetric import rsa + from qdl.security.policy import ServiceTokenVerifier + from qdl_sdk import RotatingJwtCredentialProvider + + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + private_path = root / "private.key" + private_path.write_bytes(key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.PKCS8, + serialization.NoEncryption(), + )) + public_key = key.public_key().public_bytes( + serialization.Encoding.PEM, + serialization.PublicFormat.SubjectPublicKeyInfo, + ) + now = [int(time.time())] + provider = RotatingJwtCredentialProvider( + private_key_file=private_path, + key_id="workload-rs256-v1", + algorithm="RS256", + issuer="https://identity.qdl", + audience="qdl-v2", + subject="spiffe://qdl/paper/trading-system-stable", + environment="paper", + roles=("market_data_reader", "stream_consumer"), + venues=("BINANCE", "OKX"), + consumer_manifest_revision=1, + lifetime_seconds=600, + refresh_before_seconds=120, + clock=lambda: now[0], + ) + first = await provider.get_token() + self.assertEqual(first, await provider.get_token()) + principal = ServiceTokenVerifier( + issuer="https://identity.qdl", + audience="qdl-v2", + keys_by_id={"workload-rs256-v1": public_key}, + algorithms=("RS256",), + max_lifetime_seconds=900, + ).verify(first, expected_environment="paper") + self.assertEqual( + principal.subject, + "spiffe://qdl/paper/trading-system-stable", + ) + self.assertEqual(principal.venues, frozenset({"BINANCE", "OKX"})) + now[0] += 500 + self.assertNotEqual(first, await provider.get_token()) + + async def test_ambiguous_tls_clients_and_duplicate_targets_are_rejected(self): + with tempfile.TemporaryDirectory() as temp: + tls = self.identity(Path(temp)) + async with httpx.AsyncClient() as client: + with self.assertRaisesRegex(ValueError, "either REST client"): + RestQueryTransport( + "https://qdl-v2-query:8200", client=client, tls=tls + ) + with self.assertRaisesRegex(ValueError, "non-empty and unique"): + GrpcStreamTransport( + ("127.0.0.1:18220", "127.0.0.1:18220"), + allow_insecure_loopback=True, + ) + with self.assertRaisesRegex(ValueError, "only for explicit loopback"): + GrpcStreamTransport( + ("stream-a:8210", "stream-b:8210"), + allow_insecure_loopback=True, + ) + + +if __name__ == "__main__": + unittest.main() From 7039376601cb4cffc34a2c9cffb5b2e63db9646c Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 14:28:49 +0000 Subject: [PATCH 14/21] feat(v2): wire durable Rust authority cutover --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 70 +++++ docker-compose.v2-stable.yml | 119 +++++++- .../v2-production-rust-authority-cutover.md | 100 +++++-- .../0010_authority_dispatcher_security.sql | 20 ++ .../9999_init_authority_dispatcher_role.sh | 31 ++ qdl/control/cutover_packet.py | 272 ++++++++++++++++++ scripts/phase80_generate_tls.sh | 3 +- scripts/phaseb_bootstrap_stable_broker.py | 64 ++++- scripts/phaseb_prepare_stable_candidate.py | 26 +- scripts/phasec3_authority_cutover.py | 171 +++++++++++ scripts/phasec3_authority_db_smoke.sh | 47 +++ scripts/run_authority_outbox_dispatcher.py | 31 ++ tests/test_phaseb_stable_deployment.py | 71 +++++ tests/test_phasec3_authority_runtime.py | 67 +++++ tests/test_phasec3_cutover_packet.py | 136 +++++++++ 15 files changed, 1198 insertions(+), 30 deletions(-) create mode 100644 migrations/postgres/0010_authority_dispatcher_security.sql create mode 100755 migrations/postgres/9999_init_authority_dispatcher_role.sh create mode 100644 qdl/control/cutover_packet.py create mode 100755 scripts/phasec3_authority_cutover.py create mode 100755 scripts/phasec3_authority_db_smoke.sh create mode 100644 tests/test_phasec3_authority_runtime.py create mode 100644 tests/test_phasec3_cutover_packet.py diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 8059d5a..daa75eb 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6234,6 +6234,19 @@ failure. isolated projector group are replaying authentic retained Kafka records because the exact destructive B17 rebuild was not approved. Therefore C.2 is not yet closed and no consumer route or authority was promoted. +- `2026-08-20 C.2 ISOLATED CONSUMER ACCEPTANCE PASS`: the + non-destructive cache generation completed against authentic retained Kafka + bytes. At the final bounded snapshot the six-partition projector lag was + `144`, projector readiness was `READY`, both mTLS query replicas returned + 200 and no new projector error appeared in the last two minutes. The + source-owned `qdl_sdk==2.0.0` acceptance passed for Binance and OKX: + each venue returned five final 1m BARs with `FULL` coverage, identical query + replica fingerprints and authoritative provider identity; cursor resume was + contiguous `982448 -> 982449` for Binance and + `339821 -> 339822` for OKX. A request without a client certificate remained + rejected. Earlier ACL/stream errors were bounded startup/replay history and + were not active at acceptance. V1 stayed HTTP 200, no Trading System route, + order path or authority changed, and all alpha processes remained stopped. - C.2 gates are mTLS positive/negative tests, certificate rotation/reconnect, exact BAR/trade SDK projection, bounded route-manifest parser tests, V1 unmatched-symbol compatibility, authenticated real Binance/OKX adapter @@ -6260,6 +6273,63 @@ freshness/session/contract checks and the source-switch audit is durable. #### C.3 Fast-Track Rust Authority Promotion +**C.3 implementation journal:** + +- `2026-08-20 C.3 DURABLE AUTHORITY RUNTIME WIRING STARTED`: reuse, do not + fork, the accepted Phase 9.2 domain primitives: migrations + `0006/0007/0009`, transactional authority outbox, compacted control event, + Rust `qdl-production-core`, per-target sink fence and W/W+1 handoff. Add + only the missing deployable topology around them: a dedicated isolated + PostgreSQL authority-control database, one least-privilege authority + dispatcher identity, compacted authority/target-checkpoint topics, immutable + production-core configs and three bounded Rust workers behind an explicit + Compose profile. The existing shadow core remains the writer until a + separately approved operator packet fences it. +- Add one source-owned operator command that is plan-only by default and accepts + a versioned immutable packet. It must validate exact slices, candidate/image/ + contract/partition digests, expected state/revision/owner/lease, terminal + checkpoint, zero mismatch/gap canary evidence, hold expiry, Trading System + route and executable rollback before any SQL CAS. Apply requires an exact + confirmation token; transitions execute one slice at a time and stop on the + first failure. No environment-label-only promotion is valid. +- Gates are migration idempotency, DB transaction/outbox atomicity, broker ACK + retry/crash recovery, compacted control rebuild, Rust startup with missing/ + stale/partial authority failure, target fencing, real canary parity, + W/W+1 primary handoff, V1 fallback/return, bounded resources and full + Python/Rust/V1 contract regression. Code/test wiring cannot mutate production + authority; runtime promotion still requires the exact packet and explicit + operator approval named in this section. + +- `2026-08-20 C.3 TOPOLOGY/OPERATOR SLICE PASS`: added a dedicated + non-public PostgreSQL authority database, migration-owned least-privilege + dispatcher role, atomic dispatcher heartbeat, compacted authority/checkpoint + topics, per-principal Kafka ACLs and three bounded + `qdl-production-core` workers behind explicit control/primary profiles. + Stable bundle generation now emits production-core configs and separate + dispatcher/admin credentials outside Git. Added a strict, expiring, + digest-derived plan/apply packet command that validates real-data evidence, + exact route rollback, slice state/revision/owner/lease/digests and uses the + accepted SQL CAS functions one slice per transaction; it is plan-only unless + `--apply --confirm APPLY_C3_` matches the immutable packet. + Focused topology/outbox/packet tests passed 23/23. A network-none/tmpfs + PostgreSQL bootstrap proved all migrations, three SECURITY DEFINER functions, + direct-table UPDATE denial, dispatcher claim permission and migration + idempotency, then auto-removed the test container. Runbook: + [V2 production and Rust authority cutover](docs/runbooks/v2-production-rust-authority-cutover.md). + This code evidence does not authorize a production CAS or consumer restart. + +- `2026-08-20 C.3 FULL REGRESSION/BUNDLE GATE PASS`: full Python + regression passed 546/546 with six explicit environment skips; changed-file + Ruff passed; Rust passed 70/70, `cargo fmt --check` and strict Clippy. + TLS generation emitted the dedicated dispatcher identity, candidate bundle + generation passed with 12 runtime files and no secret values in the public + manifest, and Compose config parsed with both authority profiles. The first + Rust test attempt exhausted a 1 GiB disposable tmpfs during link; rerun with + debug symbols disabled passed in 1.5 GiB and left no build target on disk. + Repository-wide Ruff still reports 63 pre-existing findings outside this + slice; changed files have zero finding. No authority DB/volume, production + CAS, Trading System route, V1 service or provider ownership was mutated. + Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One operator packet may list the complete slice set, image IDs, old/new owners, diff --git a/docker-compose.v2-stable.yml b/docker-compose.v2-stable.yml index 2045674..5d6f91a 100644 --- a/docker-compose.v2-stable.yml +++ b/docker-compose.v2-stable.yml @@ -133,6 +133,33 @@ services: environment: {<<: *kafka-env, KAFKA_NODE_ID: 3, KAFKA_ADVERTISED_LISTENERS: "SSL://kafka3:9092", KAFKA_SSL_KEYSTORE_FILENAME: kafka3.keystore.p12} volumes: ["${QDL_STABLE_CERT_DIR:?set QDL_STABLE_CERT_DIR}:/etc/kafka/secrets:ro", kafka3_data:/var/lib/kafka/data] + stable_authority_db: + image: postgres@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685 + profiles: [stable-authority] + restart: "no" + security_opt: [no-new-privileges:true] + networks: [stable_internal] + environment: + POSTGRES_DB: qdl_authority + POSTGRES_USER: qdl_authority + POSTGRES_PASSWORD: ${QDL_STABLE_CONTROL_DB_PASSWORD:?set QDL_STABLE_CONTROL_DB_PASSWORD} + QDL_STABLE_DISPATCHER_DB_PASSWORD: ${QDL_STABLE_DISPATCHER_DB_PASSWORD:?set QDL_STABLE_DISPATCHER_DB_PASSWORD} + PGDATA: /var/lib/postgresql/data/pgdata + volumes: + - stable_authority_db:/var/lib/postgresql/data + - ./migrations/postgres:/docker-entrypoint-initdb.d:ro + tmpfs: + - /run/postgresql:rw,nosuid,nodev,noexec,size=8m + - /tmp:rw,nosuid,nodev,noexec,size=16m + healthcheck: + test: [CMD-SHELL, "pg_isready -U qdl_authority -d qdl_authority"] + interval: 3s + timeout: 3s + retries: 40 + start_period: 10s + mem_limit: 256m + cpus: 0.50 + stable_redis: image: redis@sha256:dfa18828cbc07b3ae6a95ec7343f6c214fdee2d836197b4be8e9904420762cd8 command: [redis-server, --appendonly, "no", --save, "", --maxmemory, 128mb, --maxmemory-policy, noeviction] @@ -178,12 +205,13 @@ services: - >- install -d -o 10001 -g 10001 -m 0750 /stable-certs/producer /stable-certs/core /stable-certs/projector - /stable-certs/query /stable-certs/stream && + /stable-certs/query /stable-certs/stream /stable-certs/authority-dispatcher && cp /source/producer/* /stable-certs/producer/ && cp /source/core/* /stable-certs/core/ && cp /source/projector/* /stable-certs/projector/ && cp /source/query/* /stable-certs/query/ && cp /source/stream/* /stable-certs/stream/ && + cp /source/authority-dispatcher/* /stable-certs/authority-dispatcher/ && chown -R 10001:10001 /stable-certs && find /stable-certs -type f -exec chmod 0440 {} + restart: "no" @@ -198,6 +226,7 @@ services: - ${QDL_STABLE_PROJECTOR_CERT_DIR:?set QDL_STABLE_PROJECTOR_CERT_DIR}:/source/projector:ro - ${QDL_STABLE_QUERY_CERT_DIR:?set QDL_STABLE_QUERY_CERT_DIR}:/source/query:ro - ${QDL_STABLE_STREAM_CERT_DIR:?set QDL_STABLE_STREAM_CERT_DIR}:/source/stream:ro + - ${QDL_STABLE_AUTHORITY_CERT_DIR:?set QDL_STABLE_AUTHORITY_CERT_DIR}:/source/authority-dispatcher:ro - stable_tls:/stable-certs query_v2_1: @@ -299,6 +328,33 @@ services: stream_v2_active: {condition: service_started} stream_v2_passive: {condition: service_started} + authority_outbox_v2: + <<: *python + profiles: [stable-authority] + command: [python, /app/scripts/run_authority_outbox_dispatcher.py, --poll-seconds, "0.5", --batch-size, "20"] + environment: + <<: *stable-env + QDL_CONTROL_DB_DSN: ${QDL_STABLE_CONTROL_DB_DSN:?set QDL_STABLE_CONTROL_DB_DSN} + QDL_KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092 + QDL_KAFKA_CLIENT_ID: qdl-v2-stable-authority-dispatcher + QDL_KAFKA_CA_LOCATION: /stable-certs/authority-dispatcher/ca.crt + QDL_KAFKA_CERT_LOCATION: /stable-certs/authority-dispatcher/client.crt + QDL_KAFKA_KEY_LOCATION: /stable-certs/authority-dispatcher/client.key + QDL_AUTHORITY_TOPIC: qdl.authority.v1 + QDL_AUTHORITY_HEALTH_FILE: /var/lib/qdl-stable/runtime/authority-dispatcher-health.json + depends_on: + stable_authority_db: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + stable_state_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + healthcheck: + test: [CMD, python, -c, "import json,time; p=json.load(open('/var/lib/qdl-stable/runtime/authority-dispatcher-health.json')); assert p['status']=='READY' and time.time_ns()-int(p['heartbeat_ns'])<5000000000"] + interval: 3s + timeout: 2s + retries: 20 + rust_core: <<: *rust entrypoint: [/usr/local/bin/qdl-realtime-core] @@ -347,6 +403,66 @@ services: kafka2: {condition: service_healthy} kafka3: {condition: service_healthy} + production_core_1: + <<: *rust + profiles: [stable-authority-primary] + entrypoint: [/usr/local/bin/qdl-production-core] + command: [/runtime/production-core-001.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-production-core-001 + QDL_KAFKA_GROUP_ID: qdl-v2-production-core-v1 + volumes: + - stable_tls:/stable-certs:ro + - "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/production-core-001.json:/runtime/production-core-001.json:ro" + depends_on: + authority_outbox_v2: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + production_core_2: + <<: *rust + profiles: [stable-authority-primary] + entrypoint: [/usr/local/bin/qdl-production-core] + command: [/runtime/production-core-002.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-production-core-002 + QDL_KAFKA_GROUP_ID: qdl-v2-production-core-v1 + volumes: + - stable_tls:/stable-certs:ro + - "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/production-core-002.json:/runtime/production-core-002.json:ro" + depends_on: + authority_outbox_v2: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + + production_core_3: + <<: *rust + profiles: [stable-authority-primary] + entrypoint: [/usr/local/bin/qdl-production-core] + command: [/runtime/production-core-003.json] + environment: + <<: *rust-env + QDL_KAFKA_CERT_ROOT: /stable-certs/core + QDL_KAFKA_CLIENT_ID: qdl-v2-production-core-003 + QDL_KAFKA_GROUP_ID: qdl-v2-production-core-v1 + volumes: + - stable_tls:/stable-certs:ro + - "${QDL_STABLE_RUNTIME_DIR:?set QDL_STABLE_RUNTIME_DIR}/production-core-003.json:/runtime/production-core-003.json:ro" + depends_on: + authority_outbox_v2: {condition: service_healthy} + stable_tls_init: {condition: service_completed_successfully} + kafka1: {condition: service_healthy} + kafka2: {condition: service_healthy} + kafka3: {condition: service_healthy} + ingestor_binance_usdm: <<: *rust entrypoint: [/usr/local/bin/qdl-native-raw-ingestor] @@ -493,3 +609,4 @@ volumes: kafka3_data: {labels: {qdl.scope: v2-stable-candidate}} stable_state: {labels: {qdl.scope: v2-stable-candidate}} stable_tls: {labels: {qdl.scope: v2-stable-candidate-secrets}} + stable_authority_db: {labels: {qdl.scope: v2-stable-authority-control}} diff --git a/docs/runbooks/v2-production-rust-authority-cutover.md b/docs/runbooks/v2-production-rust-authority-cutover.md index 8495d13..600842b 100644 --- a/docs/runbooks/v2-production-rust-authority-cutover.md +++ b/docs/runbooks/v2-production-rust-authority-cutover.md @@ -93,6 +93,29 @@ Do not continue while the stable binary still rejects `RUST_CANARY` or `RUST_PRIMARY`, or while authority can be changed by environment variable alone. +Gate 1 is implemented on the feature branch, but no production CAS has +executed. The deployable topology reuses migrations 0006/0007/0009 and adds +migration 0010, a dedicated non-public PostgreSQL authority database, +function-scoped dispatcher role, transactional outbox dispatcher, compacted +authority/checkpoint topics, per-principal ACLs, and three bounded +qdl-production-core workers behind stable-authority and +stable-authority-primary profiles. + +The stable binary reconstructs authority and every target checkpoint before +reading raw input. Missing, partial, stale or wrong-owner state fails closed. +Only RUST_CANARY writes the canary topic; only RUST_PRIMARY writes canonical, +public V2 and legacy compatibility topics. + +Verification completed before immutable build: + +- focused topology/outbox/operator tests: 23/23 passed; +- disposable network-none PostgreSQL bootstrap and least-privilege smoke: pass; +- full Python suite: 546 passed, 6 environment skips; +- Rust workspace: 70 passed; fmt and strict Clippy passed; +- isolated real-provider SDK acceptance: Binance and OKX query/stream/cursor + parity passed while V1 remained HTTP 200. + + ## Gate 2 - Build Immutable Artifacts After Gate 1 is committed and CI-green: @@ -171,29 +194,39 @@ docker compose \ Acceptance: ```bash -curl --fail --silent http://127.0.0.1:18201/health/ready -curl --fail --silent http://127.0.0.1:18202/health/ready -curl --fail --silent http://127.0.0.1:18210/health/live -curl --fail --silent http://127.0.0.1:18211/health/live +TLS_CA="$QDL_RELEASE_ROOT/identities/trading-system/ca.crt" +TLS_CERT="$QDL_RELEASE_ROOT/identities/trading-system/client.crt" +TLS_KEY="$QDL_RELEASE_ROOT/identities/trading-system/client.key" + +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18201/health/ready +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18202/health/ready +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18210/health/live +curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18211/health/live curl --fail --silent http://127.0.0.1:8100/v1/health -# Active/passive means both stream replicas are live and exactly one is ready. -# Select that replica's matching gRPC port for the SDK handoff/reconnect gate. -if curl --fail --silent http://127.0.0.1:18210/health/ready >/dev/null; then - GRPC_TARGET=127.0.0.1:18220 +if curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18210/health/ready >/dev/null; then + GRPC_TARGET=localhost:18220 else - curl --fail --silent http://127.0.0.1:18211/health/ready >/dev/null - GRPC_TARGET=127.0.0.1:18221 + curl --fail --silent --cacert "$TLS_CA" --cert "$TLS_CERT" --key "$TLS_KEY" \ + https://localhost:18211/health/ready >/dev/null + GRPC_TARGET=localhost:18221 fi -set -a -. "$QDL_RELEASE_ROOT/stable.env" -set +a + docker run --rm --network host \ - -e QDL_STABLE_JWT_KEYS_JSON="$QDL_STABLE_JWT_KEYS_JSON" \ - -v "$PWD/scripts/phasec1_isolated_consumer_acceptance.py:/acceptance.py:ro" \ - "$PYTHON_IMAGE" python /acceptance.py --grpc-target "$GRPC_TARGET" + -e QDL_STABLE_JWT_PRIVATE_KEY_FILE=/bundle/identities/trading-system-jwt/private.key \ + -e QDL_STABLE_JWT_KEY_ID=stable-trading-system-rs256-v1 \ + -v "$PWD:/workspace:ro" -v "$QDL_RELEASE_ROOT:/bundle:ro" \ + "$PYTHON_IMAGE" python /workspace/scripts/phasec1_isolated_consumer_acceptance.py \ + --grpc-target "$GRPC_TARGET" \ + --tls-ca-file /bundle/identities/trading-system/ca.crt \ + --tls-certificate-file /bundle/identities/trading-system/client.crt \ + --tls-private-key-file /bundle/identities/trading-system/client.key ``` - Require authentic Binance/OKX data, zero unexplained gap/duplicate/quarantine, bounded broker/projector lag, exact replica results and V1 unchanged. @@ -281,6 +314,39 @@ all required targets through `W` and first publishes at `W+1`. A failed slice enters `BLOCKED` and rolls back under a newer revision without undoing an unrelated healthy slice. +The source-owned command is plan-only by default. Every packet contains exactly +one state step for 1..32 unique slices, expires, binds image/contract/partition/ +route digests, requires clean real-provider evidence and includes an executable +Trading System V1 rollback command. + + python scripts/phasec3_authority_cutover.py \ + --packet /secure/qdl-v2/change/canary-packet.json + +Review the printed APPLY_C3_ token. Apply only the same packet bytes: + + QDL_CONTROL_ADMIN_DSN='postgresql://...' \ + python scripts/phasec3_authority_cutover.py \ + --packet /secure/qdl-v2/change/canary-packet.json \ + --apply --confirm APPLY_C3_ + +Start control services before canary. Start production workers only after the +RUST_CANARY authority event is durable: + + docker compose --env-file "$QDL_RELEASE_ROOT/stable.env" \ + -f docker-compose.v2-stable.yml --profile stable-authority \ + up -d stable_authority_db authority_outbox_v2 + + docker compose --env-file "$QDL_RELEASE_ROOT/stable.env" \ + -f docker-compose.v2-stable.yml \ + --profile stable-authority --profile stable-authority-primary \ + up -d production_core_1 production_core_2 production_core_3 + +The command checks the current DB row under lock and executes one transaction +per slice, stopping on the first stale CAS. Primary and Python restore use the +accepted qdl_transition_authority_v2 handoff; no environment label can promote +authority. + + ## Gate 7 - Close With V1 Hot Fallback After approved Binance/OKX slices are `RUST_PRIMARY`, Trading System uses V2 diff --git a/migrations/postgres/0010_authority_dispatcher_security.sql b/migrations/postgres/0010_authority_dispatcher_security.sql new file mode 100644 index 0000000..243eaa0 --- /dev/null +++ b/migrations/postgres/0010_authority_dispatcher_security.sql @@ -0,0 +1,20 @@ +BEGIN; + +ALTER FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + SECURITY DEFINER + SET search_path = pg_catalog, public; +ALTER FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + SECURITY DEFINER + SET search_path = pg_catalog, public; +ALTER FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + SECURITY DEFINER + SET search_path = pg_catalog, public; + +REVOKE ALL ON FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + FROM PUBLIC; +REVOKE ALL ON FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + FROM PUBLIC; +REVOKE ALL ON FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + FROM PUBLIC; + +COMMIT; diff --git a/migrations/postgres/9999_init_authority_dispatcher_role.sh b/migrations/postgres/9999_init_authority_dispatcher_role.sh new file mode 100755 index 0000000..cfe899f --- /dev/null +++ b/migrations/postgres/9999_init_authority_dispatcher_role.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${POSTGRES_USER:?POSTGRES_USER is required}" +: "${POSTGRES_DB:?POSTGRES_DB is required}" +: "${QDL_STABLE_DISPATCHER_DB_PASSWORD:?QDL_STABLE_DISPATCHER_DB_PASSWORD is required}" + +psql --set=ON_ERROR_STOP=1 \ + --username "${POSTGRES_USER}" \ + --dbname "${POSTGRES_DB}" \ + --set=dispatcher_password="${QDL_STABLE_DISPATCHER_DB_PASSWORD}" <<'SQL' +SELECT format( + 'CREATE ROLE qdl_authority_dispatcher LOGIN PASSWORD %L', + :'dispatcher_password' +) +WHERE NOT EXISTS ( + SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'qdl_authority_dispatcher' +) +\gexec +ALTER ROLE qdl_authority_dispatcher + NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOREPLICATION + PASSWORD :'dispatcher_password'; +GRANT CONNECT ON DATABASE qdl_authority TO qdl_authority_dispatcher; +GRANT USAGE ON SCHEMA public TO qdl_authority_dispatcher; +GRANT EXECUTE ON FUNCTION qdl_claim_authority_outbox(TEXT, INTEGER, INTERVAL) + TO qdl_authority_dispatcher; +GRANT EXECUTE ON FUNCTION qdl_complete_authority_outbox(UUID, TEXT, TEXT, INTEGER, BIGINT) + TO qdl_authority_dispatcher; +GRANT EXECUTE ON FUNCTION qdl_retry_authority_outbox(UUID, TEXT, TEXT, INTERVAL) + TO qdl_authority_dispatcher; +SQL diff --git a/qdl/control/cutover_packet.py b/qdl/control/cutover_packet.py new file mode 100644 index 0000000..9f04f5f --- /dev/null +++ b/qdl/control/cutover_packet.py @@ -0,0 +1,272 @@ +from __future__ import annotations + +import hashlib +import json +import time +import uuid +from dataclasses import dataclass +from typing import Any, Mapping, Sequence + + +_SCHEMA = "qdl.c3.authority-cutover-packet.v1" +_PAIRS = { + "SHADOW_VALIDATE": ("RUST_SHADOW", "VALIDATING"), + "CANARY": ("VALIDATING", "RUST_CANARY"), + "PRIMARY": ("RUST_CANARY", "RUST_PRIMARY"), + "BLOCK_CANARY": ("RUST_CANARY", "BLOCKED"), + "BLOCK_PRIMARY": ("RUST_PRIMARY", "BLOCKED"), + "ROLLBACK_PENDING": ("BLOCKED", "ROLLBACK_PENDING"), + "PYTHON_RESTORE": ("ROLLBACK_PENDING", "PYTHON_PRIMARY"), +} +_HANDOFF_STAGES = {"PRIMARY", "PYTHON_RESTORE"} +_ACTIVE_STAGES = {"CANARY", "PRIMARY"} + + +def _digest(value: Mapping[str, Any]) -> str: + encoded = json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode() + return hashlib.sha256(encoded).hexdigest() + + +def _uuid(value: object, name: str) -> str: + try: + return str(uuid.UUID(str(value))) + except ValueError as error: + raise ValueError(f"{name} must be a UUID") from error + + +def _hex(value: object, name: str, *, image: bool = False) -> str: + text = str(value) + prefix = "sha256:" if image else "" + raw = text.removeprefix(prefix) if prefix else text + if (image and not text.startswith(prefix)) or len(raw) != 64 or any( + char not in "0123456789abcdef" for char in raw + ): + raise ValueError(f"{name} must be a SHA-256 digest") + return text + + +def _exact(value: Mapping[str, Any], fields: set[str], name: str) -> None: + if set(value) != fields: + raise ValueError(f"{name} fields are incomplete or unknown") + + +@dataclass(frozen=True, slots=True) +class CutoverSlice: + transition_id: str + handoff_id: str | None + slice_id: str + expected_state: str + expected_revision: int + expected_owner_id: str + expected_lease_epoch: int + partition_plan_epoch: int + new_state: str + new_owner_id: str + new_lease_epoch: int + terminal_watermark: int | None + prerequisite_bundle_id: str | None + hold_until: str | None + reason: str + + @classmethod + def parse(cls, raw: Mapping[str, Any], *, stage: str) -> "CutoverSlice": + _exact( + raw, + { + "transition_id", "handoff_id", "slice_id", "expected_state", + "expected_revision", "expected_owner_id", "expected_lease_epoch", + "partition_plan_epoch", "new_state", "new_owner_id", + "new_lease_epoch", "terminal_watermark", + "prerequisite_bundle_id", "hold_until", "reason", + }, + "cutover slice", + ) + expected_pair = _PAIRS[stage] + if (raw["expected_state"], raw["new_state"]) != expected_pair: + raise ValueError("cutover slice state pair differs from packet stage") + handoff_id = raw["handoff_id"] + prerequisite = raw["prerequisite_bundle_id"] + if stage in _HANDOFF_STAGES: + handoff_id = _uuid(handoff_id, "handoff_id") + elif handoff_id is not None: + raise ValueError("non-handoff stage cannot carry handoff_id") + if stage in _ACTIVE_STAGES: + prerequisite = _uuid(prerequisite, "prerequisite_bundle_id") + if raw["terminal_watermark"] is None or int(raw["terminal_watermark"]) < 0: + raise ValueError("active Rust stage requires terminal watermark") + if not str(raw["hold_until"] or "").strip(): + raise ValueError("active Rust stage requires hold_until") + elif prerequisite is not None or raw["hold_until"] is not None: + raise ValueError("inactive stage cannot carry prerequisite/hold") + positive = ( + int(raw["expected_revision"]), int(raw["expected_lease_epoch"]), + int(raw["partition_plan_epoch"]), int(raw["new_lease_epoch"]), + ) + if any(value <= 0 for value in positive): + raise ValueError("cutover slice revisions/epochs must be positive") + if ( + raw["expected_owner_id"] != raw["new_owner_id"] + and int(raw["new_lease_epoch"]) <= int(raw["expected_lease_epoch"]) + ): + raise ValueError("authority owner change requires newer lease") + if any( + not str(value).strip() + for value in ( + raw["slice_id"], raw["expected_owner_id"], + raw["new_owner_id"], raw["reason"], + ) + ): + raise ValueError("cutover slice identity/reason is incomplete") + return cls( + transition_id=_uuid(raw["transition_id"], "transition_id"), + handoff_id=handoff_id, + slice_id=str(raw["slice_id"]), + expected_state=str(raw["expected_state"]), + expected_revision=int(raw["expected_revision"]), + expected_owner_id=str(raw["expected_owner_id"]), + expected_lease_epoch=int(raw["expected_lease_epoch"]), + partition_plan_epoch=int(raw["partition_plan_epoch"]), + new_state=str(raw["new_state"]), + new_owner_id=str(raw["new_owner_id"]), + new_lease_epoch=int(raw["new_lease_epoch"]), + terminal_watermark=( + int(raw["terminal_watermark"]) + if raw["terminal_watermark"] is not None else None + ), + prerequisite_bundle_id=prerequisite, + hold_until=( + str(raw["hold_until"]) if raw["hold_until"] is not None else None + ), + reason=str(raw["reason"]), + ) + + +@dataclass(frozen=True, slots=True) +class AuthorityCutoverPacket: + raw: Mapping[str, Any] + packet_id: str + stage: str + actor: str + candidate_digest: str + artifact_image_digest: str + contract_digest: str + partition_plan_digest: str + route_manifest_digest: str + slices: tuple[CutoverSlice, ...] + + @classmethod + def parse( + cls, raw: Mapping[str, Any], *, now_ns: int | None = None + ) -> "AuthorityCutoverPacket": + _exact( + raw, + { + "schema", "packet_id", "stage", "issued_at_ns", "expires_at_ns", + "actor", "change_ticket", "candidate_digest", + "artifact_image_digest", "contract_digest", + "partition_plan_digest", "route_manifest_digest", + "consumer_route", "evidence", "slices", + }, + "cutover packet", + ) + if raw["schema"] != _SCHEMA or raw["stage"] not in _PAIRS: + raise ValueError("cutover packet schema/stage is unsupported") + current = time.time_ns() if now_ns is None else now_ns + issued = int(raw["issued_at_ns"]) + expires = int(raw["expires_at_ns"]) + if issued <= 0 or expires <= issued or not issued <= current < expires: + raise ValueError("cutover packet approval window is invalid") + if not str(raw["actor"]).strip() or not str(raw["change_ticket"]).strip(): + raise ValueError("cutover operator/change ticket is required") + route = raw["consumer_route"] + evidence = raw["evidence"] + if not isinstance(route, Mapping) or not isinstance(evidence, Mapping): + raise ValueError("cutover route/evidence must be objects") + _exact( + route, + { + "consumer_id", "expected_route", "new_route", "rollback_route", + "rollback_command", + }, + "consumer route", + ) + rollback = route["rollback_command"] + if ( + route["consumer_id"] != "trading-system" + or route["rollback_route"] != "V1" + or not isinstance(rollback, Sequence) + or isinstance(rollback, (str, bytes)) + or not rollback + or any(not isinstance(item, str) or not item.strip() for item in rollback) + ): + raise ValueError("cutover Trading System rollback route is invalid") + _exact( + evidence, + { + "provider_provenance", "semantic_mismatches", "open_gaps", + "duplicate_external_effects", "consumer_errors", + }, + "cutover evidence", + ) + if ( + evidence["provider_provenance"] != "REAL" + or any( + int(evidence[name]) != 0 + for name in ( + "semantic_mismatches", "open_gaps", + "duplicate_external_effects", "consumer_errors", + ) + ) + ): + raise ValueError("cutover evidence is not clean/authentic") + values = raw["slices"] + if not isinstance(values, list) or not 1 <= len(values) <= 32: + raise ValueError("cutover packet requires 1..32 slices") + slices = tuple( + CutoverSlice.parse(value, stage=str(raw["stage"])) for value in values + ) + identities = [item.slice_id for item in slices] + if len(identities) != len(set(identities)): + raise ValueError("cutover packet slice IDs must be unique") + packet = cls( + raw=dict(raw), + packet_id=_uuid(raw["packet_id"], "packet_id"), + stage=str(raw["stage"]), + actor=str(raw["actor"]), + candidate_digest=_hex(raw["candidate_digest"], "candidate_digest"), + artifact_image_digest=_hex( + raw["artifact_image_digest"], "artifact_image_digest", image=True + ), + contract_digest=_hex(raw["contract_digest"], "contract_digest"), + partition_plan_digest=_hex( + raw["partition_plan_digest"], "partition_plan_digest" + ), + route_manifest_digest=_hex( + raw["route_manifest_digest"], "route_manifest_digest" + ), + slices=slices, + ) + return packet + + @property + def digest(self) -> str: + return _digest(self.raw) + + @property + def confirmation_token(self) -> str: + return f"APPLY_C3_{self.digest[:16]}" + + def plan(self) -> dict[str, Any]: + return { + "schema": "qdl.c3.authority-cutover-plan.v1", + "packet_id": self.packet_id, + "packet_digest": self.digest, + "confirmation_token": self.confirmation_token, + "stage": self.stage, + "slice_count": len(self.slices), + "slice_ids": [item.slice_id for item in self.slices], + "apply_requested": False, + "production_mutations": 0, + } diff --git a/scripts/phase80_generate_tls.sh b/scripts/phase80_generate_tls.sh index 699fa56..b875ccb 100755 --- a/scripts/phase80_generate_tls.sh +++ b/scripts/phase80_generate_tls.sh @@ -49,7 +49,7 @@ issue_certificate() { for broker in kafka1 kafka2 kafka3; do issue_certificate "${broker}" "${broker}" done -for client in phase8-admin phase8-producer phase8-consumer phase8-core phase8-unauthorized stable-trading-system; do +for client in phase8-admin phase8-producer phase8-consumer phase8-core phase8-unauthorized stable-authority-dispatcher stable-trading-system; do issue_certificate "${client}" "${client}" done issue_certificate stable-trading-system-jwt stable-trading-system-jwt @@ -116,6 +116,7 @@ find "${OUTPUT_DIR}" -maxdepth 1 -name '*.key' \ ! -name 'phase8-core.key' \ ! -name 'stable-query.key' \ ! -name 'stable-stream.key' \ + ! -name 'stable-authority-dispatcher.key' \ ! -name 'stable-trading-system.key' \ ! -name 'stable-trading-system-jwt.key' \ -delete diff --git a/scripts/phaseb_bootstrap_stable_broker.py b/scripts/phaseb_bootstrap_stable_broker.py index ce1830e..203d593 100755 --- a/scripts/phaseb_bootstrap_stable_broker.py +++ b/scripts/phaseb_bootstrap_stable_broker.py @@ -11,11 +11,17 @@ COMPOSE = ROOT / "docker-compose.v2-stable.yml" BOOTSTRAP = "kafka1:9092,kafka2:9092,kafka3:9092" ADMIN_CONFIG = "/etc/kafka/secrets/admin.properties" -TOPICS = ( - "md.raw.stable.v1", - "md.canonical.v2", - "md.quarantine.stable.v1", -) +TOPIC_POLICIES = { + "md.raw.stable.v1": "delete", + "md.canonical.v2": "delete", + "md.quarantine.stable.v1": "delete", + "qdl.authority.v1": "compact", + "qdl.target-checkpoint.v1": "compact", + "md.canary.canonical.v2": "delete", + "md.projector.public.v2": "delete", + "md.projector.legacy.v1": "delete", +} +TOPICS = tuple(TOPIC_POLICIES) def compose(env_file: Path, *arguments: str) -> subprocess.CompletedProcess[str]: @@ -60,7 +66,7 @@ def add_acl( def bootstrap(env_file: Path) -> dict[str, object]: - for topic in TOPICS: + for topic, cleanup_policy in TOPIC_POLICIES.items(): kafka( env_file, "kafka-topics.sh", @@ -71,6 +77,7 @@ def bootstrap(env_file: Path) -> dict[str, object]: "--config", "min.insync.replicas=2", "--config", "unclean.leader.election.enable=false", "--config", "compression.type=producer", + "--config", f"cleanup.policy={cleanup_policy}", ) add_acl( @@ -91,14 +98,47 @@ def bootstrap(env_file: Path) -> dict[str, object]: env_file, "phase8-core", ("READ",), ("--group", "qdl-v2-stable-core-v1"), ) - add_acl(env_file, "phase8-core", ("IdempotentWrite",), ("--cluster",)) + for topic in ("qdl.authority.v1", "qdl.target-checkpoint.v1"): + add_acl( + env_file, "phase8-core", ("READ", "DESCRIBE"), + ("--topic", topic), + ) + for topic in ( + "qdl.target-checkpoint.v1", + "md.canary.canonical.v2", + "md.projector.public.v2", + "md.projector.legacy.v1", + ): + add_acl( + env_file, "phase8-core", ("WRITE", "DESCRIBE"), + ("--topic", topic), + ) add_acl( - env_file, "phase8-core", ("WRITE", "DESCRIBE"), + env_file, "phase8-core", ("READ",), ( - "--transactional-id", "qdl-v2-stable-core-", + "--group", "qdl-v2-production-core-v1-", "--resource-pattern-type", "prefixed", ), ) + add_acl(env_file, "phase8-core", ("IdempotentWrite",), ("--cluster",)) + for transactional_prefix in ( + "qdl-v2-stable-core-", "qdl-v2-production-core-" + ): + add_acl( + env_file, "phase8-core", ("WRITE", "DESCRIBE"), + ( + "--transactional-id", transactional_prefix, + "--resource-pattern-type", "prefixed", + ), + ) + add_acl( + env_file, "stable-authority-dispatcher", ("WRITE", "DESCRIBE"), + ("--topic", "qdl.authority.v1"), + ) + add_acl( + env_file, "stable-authority-dispatcher", ("IdempotentWrite",), + ("--cluster",), + ) for topic in (TOPICS[0], TOPICS[1]): add_acl( env_file, "phase8-consumer", ("READ", "DESCRIBE"), @@ -127,7 +167,11 @@ def bootstrap(env_file: Path) -> dict[str, object]: "replication_factor": 3, "min_insync_replicas": 2, "tls_client_auth": "required", - "principals": ["phase8-producer", "phase8-core", "phase8-consumer"], + "topic_policies": dict(TOPIC_POLICIES), + "principals": [ + "phase8-producer", "phase8-core", "phase8-consumer", + "stable-authority-dispatcher", + ], } diff --git a/scripts/phaseb_prepare_stable_candidate.py b/scripts/phaseb_prepare_stable_candidate.py index ed3693d..c87f730 100755 --- a/scripts/phaseb_prepare_stable_candidate.py +++ b/scripts/phaseb_prepare_stable_candidate.py @@ -17,6 +17,7 @@ from qdl.runtime.stable_deployment import ( StableAcquisitionPlan, stable_authority_record, + write_production_core_bundle, write_stable_runtime_bundle, ) @@ -105,10 +106,18 @@ def prepare_candidate( acquisition=acquisition, authority=authority, ) + bundle_digests.update(write_production_core_bundle( + runtime_dir, + catalog=catalog, + acquisition=acquisition, + raw_authority=authority, + partition_plan_epoch=1, + )) for role, principal in ( ("producer", "phase8-producer"), ("core", "phase8-core"), ("projector", "phase8-consumer"), + ("authority-dispatcher", "stable-authority-dispatcher"), ("trading-system", "stable-trading-system"), ): copy_client_identity(cert_dir, identities_dir / role, principal) @@ -132,6 +141,8 @@ def prepare_candidate( ).hexdigest() ingest_secret = secrets.token_urlsafe(48) cursor_secret = secrets.token_urlsafe(48) + control_db_password = secrets.token_urlsafe(32) + dispatcher_db_password = secrets.token_urlsafe(32) jwt_public_key = (jwt_identity_dir / "public.pem").read_text(encoding="utf-8") compose_cert_dir = (host_cert_dir or cert_dir).resolve() compose_output_dir = (host_output_dir or output_dir).resolve() @@ -151,6 +162,9 @@ def prepare_candidate( "QDL_STABLE_PROJECTOR_CERT_DIR": str( compose_output_dir / "identities/projector" ), + "QDL_STABLE_AUTHORITY_CERT_DIR": str( + compose_output_dir / "identities/authority-dispatcher" + ), "QDL_STABLE_CORE_CERT_DIR": str(compose_output_dir / "identities/core"), "QDL_STABLE_PRODUCER_CERT_DIR": str( compose_output_dir / "identities/producer" @@ -163,6 +177,16 @@ def prepare_candidate( "QDL_STABLE_TRADING_SYSTEM_JWT_PRIVATE_KEY": str( compose_output_dir / "identities/trading-system-jwt/private.key" ), + "QDL_STABLE_CONTROL_DB_PASSWORD": control_db_password, + "QDL_STABLE_DISPATCHER_DB_PASSWORD": dispatcher_db_password, + "QDL_STABLE_CONTROL_DB_DSN": ( + "postgresql://qdl_authority_dispatcher:" + f"{dispatcher_db_password}@stable_authority_db:5432/qdl_authority" + ), + "QDL_STABLE_CONTROL_ADMIN_DSN": ( + "postgresql://qdl_authority:" + f"{control_db_password}@stable_authority_db:5432/qdl_authority" + ), "QDL_STABLE_RUNTIME_DIR": str(compose_output_dir / "runtime"), } env_path = output_dir / "stable.env" @@ -186,7 +210,7 @@ def prepare_candidate( "acquisition_revision": acquisition.revision, "consumer_count": 5, "workload_mtls": True, - "workload_identity_count": 3, + "workload_identity_count": 4, "secret_values_recorded": False, } manifest_path = output_dir / "candidate-manifest.json" diff --git a/scripts/phasec3_authority_cutover.py b/scripts/phasec3_authority_cutover.py new file mode 100755 index 0000000..d7376b9 --- /dev/null +++ b/scripts/phasec3_authority_cutover.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import asyncio +import json +import os +from pathlib import Path +from typing import Any + +from qdl.control.cutover_packet import AuthorityCutoverPacket, CutoverSlice + + +_CURRENT_SQL = """ +SELECT state, authority_revision, owner_id, lease_epoch, partition_plan_epoch, + candidate_digest, artifact_image_digest, contract_digest, + partition_plan_digest +FROM qdl_authority_slices +WHERE slice_id = $1 +FOR UPDATE +""" +_STANDARD_SQL = """ +SELECT * +FROM qdl_transition_authority( + $1::uuid, $2, $3, $4, $5, $6, $7, $8, $9, $10, + $11::uuid, $12::timestamptz, $13, $14 +) +""" +_HANDOFF_SQL = """ +SELECT * +FROM qdl_transition_authority_v2( + $1::uuid, $2::uuid, $3, $4, $5, $6, $7, $8, $9, $10, + $11, $12, $13::uuid, $14::timestamptz, $15, $16 +) +""" + + +def required(name: str) -> str: + value = os.environ.get(name, "").strip() + if not value: + raise RuntimeError(f"required environment variable is missing: {name}") + return value + + +def _assert_current( + row: Any, item: CutoverSlice, packet: AuthorityCutoverPacket +) -> None: + if row is None: + raise RuntimeError(f"authority slice is absent: {item.slice_id}") + expected = { + "state": item.expected_state, + "authority_revision": item.expected_revision, + "owner_id": item.expected_owner_id, + "lease_epoch": item.expected_lease_epoch, + "partition_plan_epoch": item.partition_plan_epoch, + "candidate_digest": packet.candidate_digest, + "artifact_image_digest": packet.artifact_image_digest, + "contract_digest": packet.contract_digest, + "partition_plan_digest": packet.partition_plan_digest, + } + actual = {name: row[name] for name in expected} + if actual != expected: + raise RuntimeError( + f"authority CAS precondition differs for {item.slice_id}: " + f"expected={expected!r} actual={actual!r}" + ) + + +async def _apply_one( + connection: Any, packet: AuthorityCutoverPacket, item: CutoverSlice +) -> dict[str, Any]: + async with connection.transaction(): + current = await connection.fetchrow(_CURRENT_SQL, item.slice_id) + _assert_current(current, item, packet) + if packet.stage in {"PRIMARY", "PYTHON_RESTORE"}: + updated = await connection.fetchrow( + _HANDOFF_SQL, + item.handoff_id, + item.transition_id, + item.slice_id, + item.expected_state, + item.expected_revision, + item.expected_owner_id, + item.expected_lease_epoch, + item.partition_plan_epoch, + item.new_state, + item.new_owner_id, + item.new_lease_epoch, + item.terminal_watermark, + item.prerequisite_bundle_id, + item.hold_until, + packet.actor, + item.reason, + ) + else: + updated = await connection.fetchrow( + _STANDARD_SQL, + item.transition_id, + item.slice_id, + item.expected_state, + item.expected_revision, + item.expected_owner_id, + item.expected_lease_epoch, + item.partition_plan_epoch, + item.new_state, + item.new_owner_id, + item.new_lease_epoch, + item.terminal_watermark, + item.prerequisite_bundle_id, + item.hold_until, + packet.actor, + item.reason, + ) + return { + "slice_id": item.slice_id, + "state": updated["state"], + "authority_revision": updated["authority_revision"], + "owner_id": updated["owner_id"], + "lease_epoch": updated["lease_epoch"], + } + + +async def apply_packet(packet: AuthorityCutoverPacket, dsn: str) -> list[dict[str, Any]]: + try: + import asyncpg + except ImportError as error: + raise RuntimeError("authority cutover apply requires asyncpg") from error + connection = await asyncpg.connect( + dsn=dsn, + command_timeout=15, + server_settings={"application_name": "qdl-c3-authority-cutover"}, + ) + results = [] + try: + for item in packet.slices: + results.append(await _apply_one(connection, packet, item)) + finally: + await connection.close() + return results + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--packet", type=Path, required=True) + parser.add_argument("--apply", action="store_true") + parser.add_argument("--confirm") + args = parser.parse_args() + raw = json.loads(args.packet.read_text(encoding="utf-8")) + packet = AuthorityCutoverPacket.parse(raw) + plan = packet.plan() + if not args.apply: + print(json.dumps(plan, indent=2, sort_keys=True)) + return 0 + if args.confirm != packet.confirmation_token: + raise RuntimeError( + "authority cutover confirmation token differs from immutable packet" + ) + results = asyncio.run( + apply_packet(packet, required("QDL_CONTROL_ADMIN_DSN")) + ) + print(json.dumps({ + **plan, + "apply_requested": True, + "production_mutations": len(results), + "results": results, + }, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/phasec3_authority_db_smoke.sh b/scripts/phasec3_authority_db_smoke.sh new file mode 100755 index 0000000..473d5cd --- /dev/null +++ b/scripts/phasec3_authority_db_smoke.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +NAME="qdl_c3_authority_db_smoke_$$" +IMAGE="postgres@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685" +OWNER_PASSWORD="c3-owner-test-only" +DISPATCHER_PASSWORD="c3-dispatcher-test-only" + +cleanup() { + docker stop "${NAME}" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +docker run --rm -d --name "${NAME}" --network none --tmpfs /var/lib/postgresql/data:rw,nosuid,nodev,size=256m --tmpfs /run/postgresql:rw,nosuid,nodev,size=8m -e POSTGRES_DB=qdl_authority -e POSTGRES_USER=qdl_authority -e POSTGRES_PASSWORD="${OWNER_PASSWORD}" -e QDL_STABLE_DISPATCHER_DB_PASSWORD="${DISPATCHER_PASSWORD}" -e PGDATA=/var/lib/postgresql/data/pgdata -v "${ROOT_DIR}/migrations/postgres:/docker-entrypoint-initdb.d:ro" "${IMAGE}" >/dev/null + +for _attempt in $(seq 1 60); do + if docker exec "${NAME}" pg_isready -U qdl_authority -d qdl_authority >/dev/null 2>&1; then + break + fi + sleep 1 +done +docker exec "${NAME}" pg_isready -U qdl_authority -d qdl_authority >/dev/null + +owner_query() { + docker exec -e PGPASSWORD="${OWNER_PASSWORD}" "${NAME}" psql --set=ON_ERROR_STOP=1 -At -U qdl_authority -d qdl_authority -c "$1" +} + +dispatcher_query() { + docker exec -e PGPASSWORD="${DISPATCHER_PASSWORD}" "${NAME}" psql --set=ON_ERROR_STOP=1 -At -U qdl_authority_dispatcher -d qdl_authority -c "$1" +} + +[[ "$(owner_query "SELECT count(*) FROM pg_proc WHERE proname IN ('qdl_claim_authority_outbox','qdl_complete_authority_outbox','qdl_retry_authority_outbox') AND prosecdef")" == "3" ]] +[[ "$(owner_query "SELECT count(*) FROM pg_proc WHERE proname IN ('qdl_claim_authority_outbox','qdl_complete_authority_outbox','qdl_retry_authority_outbox') AND 'search_path=pg_catalog, public'=ANY(proconfig)")" == "3" ]] +[[ "$(owner_query "SELECT rolsuper::int||':'||rolcreatedb::int||':'||rolcreaterole::int FROM pg_roles WHERE rolname='qdl_authority_dispatcher'")" == "0:0:0" ]] +[[ "$(owner_query "SELECT has_function_privilege('qdl_authority_dispatcher','qdl_claim_authority_outbox(text,integer,interval)','EXECUTE')::int")" == "1" ]] +[[ "$(owner_query "SELECT has_table_privilege('qdl_authority_dispatcher','qdl_authority_event_outbox','UPDATE')::int")" == "0" ]] +[[ "$(dispatcher_query "SELECT count(*) FROM qdl_claim_authority_outbox('c3-smoke',1)")" == "0" ]] + +if dispatcher_query "UPDATE qdl_authority_event_outbox SET status='BLOCKED' WHERE false" >/dev/null 2>&1; then + echo "dispatcher unexpectedly obtained direct table UPDATE" >&2 + exit 1 +fi + +owner_query "\\i /docker-entrypoint-initdb.d/0010_authority_dispatcher_security.sql" >/dev/null + +printf '%s\n' '{"schema":"qdl.c3.authority-db-smoke.v1","status":"PASS","production_mutations":0,"cleanup":"container-auto-remove"}' diff --git a/scripts/run_authority_outbox_dispatcher.py b/scripts/run_authority_outbox_dispatcher.py index 788c0f3..3e5da2c 100755 --- a/scripts/run_authority_outbox_dispatcher.py +++ b/scripts/run_authority_outbox_dispatcher.py @@ -6,6 +6,8 @@ import json import os import socket +import time +from pathlib import Path from qdl.control.authority_outbox import ( AsyncpgAuthorityOutboxRepository, @@ -21,6 +23,27 @@ def required(name: str) -> str: return value +def write_health( + path: Path, *, status: str, published: int, error: str | None = None +) -> None: + if status not in {"STARTING", "READY", "DEGRADED"} or published < 0: + raise ValueError("authority dispatcher health payload is invalid") + payload = { + "schema": "qdl.authority-dispatcher-health.v1", + "status": status, + "heartbeat_ns": time.time_ns(), + "published_last_cycle": published, + "error": error[:1000] if error else None, + } + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(path.suffix + ".tmp") + temporary.write_text( + json.dumps(payload, sort_keys=True, separators=(",", ":")) + "\n", + encoding="utf-8", + ) + temporary.replace(path) + + async def run() -> None: parser = argparse.ArgumentParser() parser.add_argument("--once", action="store_true") @@ -29,6 +52,8 @@ async def run() -> None: args = parser.parse_args() if not 0.05 <= args.poll_seconds <= 60: raise ValueError("poll interval is outside bounds") + health_path = Path(required("QDL_AUTHORITY_HEALTH_FILE")) + write_health(health_path, status="STARTING", published=0) repository = await AsyncpgAuthorityOutboxRepository.connect(required("QDL_CONTROL_DB_DSN")) publisher = KafkaAuthorityPublisher( { @@ -50,10 +75,16 @@ async def run() -> None: try: while True: count = await dispatcher.dispatch_once() + write_health(health_path, status="READY", published=count) print(json.dumps({"event": "qdl_authority_outbox_dispatch", "published": count})) if args.once: return await asyncio.sleep(args.poll_seconds if count == 0 else 0) + except Exception as error: + write_health( + health_path, status="DEGRADED", published=0, error=str(error) + ) + raise finally: await repository.close() diff --git a/tests/test_phaseb_stable_deployment.py b/tests/test_phaseb_stable_deployment.py index 47eb866..bae1df5 100644 --- a/tests/test_phaseb_stable_deployment.py +++ b/tests/test_phaseb_stable_deployment.py @@ -809,6 +809,55 @@ def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): ) self.assertIn("stable_tls:/stable-certs:ro", services[name]["volumes"]) + authority_db = services["stable_authority_db"] + self.assertEqual(authority_db["profiles"], ["stable-authority"]) + self.assertNotIn("ports", authority_db) + self.assertIn( + "./migrations/postgres:/docker-entrypoint-initdb.d:ro", + authority_db["volumes"], + ) + self.assertIn( + "stable_authority_db:/var/lib/postgresql/data", + authority_db["volumes"], + ) + dispatcher = services["authority_outbox_v2"] + self.assertEqual(dispatcher["profiles"], ["stable-authority"]) + self.assertEqual(dispatcher["networks"], ["stable_internal"]) + self.assertNotIn("ports", dispatcher) + self.assertEqual( + dispatcher["environment"]["QDL_AUTHORITY_TOPIC"], "qdl.authority.v1" + ) + self.assertIn( + "/stable-certs/authority-dispatcher/client.crt", + dispatcher["environment"]["QDL_KAFKA_CERT_LOCATION"], + ) + production_names = ( + "production_core_1", "production_core_2", "production_core_3" + ) + self.assertEqual( + { + services[name]["environment"]["QDL_KAFKA_CLIENT_ID"] + for name in production_names + }, + { + "qdl-v2-production-core-001", + "qdl-v2-production-core-002", + "qdl-v2-production-core-003", + }, + ) + for name in production_names: + with self.subTest(production_service=name): + self.assertEqual( + services[name]["profiles"], ["stable-authority-primary"] + ) + self.assertEqual( + services[name]["entrypoint"], + ["/usr/local/bin/qdl-production-core"], + ) + self.assertEqual(services[name]["user"], "10001:10001") + self.assertTrue(services[name]["read_only"]) + self.assertNotIn("ports", services[name]) + def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): with tempfile.TemporaryDirectory(prefix="qdl-phaseb-cert-") as cert_directory: certs = Path(cert_directory) @@ -817,6 +866,7 @@ def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): "phase8-producer", "phase8-core", "phase8-consumer", + "stable-authority-dispatcher", "stable-trading-system", "stable-query", "stable-stream", @@ -851,11 +901,32 @@ def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): self.assertIn( "QDL_STABLE_RUNTIME_DIR=/host/qdl/candidate/runtime", env_text ) + self.assertIn( + "QDL_STABLE_AUTHORITY_CERT_DIR=" + "/host/qdl/candidate/identities/authority-dispatcher", + env_text, + ) + self.assertIn( + "postgresql://qdl_authority_dispatcher:", env_text + ) public_manifest = (output / "candidate-manifest.json").read_text() self.assertNotIn("QDL_STABLE_INTERNAL_INGEST_SECRET", public_manifest) for name in ("core.json", "core-002.json", "core-003.json"): self.assertTrue((output / f"runtime/{name}").is_file()) + for index in range(1, 4): + self.assertTrue( + (output / f"runtime/production-core-{index:03d}.json").is_file() + ) + self.assertTrue( + (output / "runtime/production-core-manifest.json").is_file() + ) self.assertTrue((output / "identities/projector/client.key").is_file()) + self.assertTrue( + ( + output + / "identities/authority-dispatcher/client.key" + ).is_file() + ) if __name__ == "__main__": diff --git a/tests/test_phasec3_authority_runtime.py b/tests/test_phasec3_authority_runtime.py new file mode 100644 index 0000000..15a53a5 --- /dev/null +++ b/tests/test_phasec3_authority_runtime.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import json +from pathlib import Path +import tempfile +import unittest + +from scripts.phaseb_bootstrap_stable_broker import TOPIC_POLICIES +from scripts.run_authority_outbox_dispatcher import write_health + + +ROOT = Path(__file__).resolve().parents[1] + + +class AuthorityRuntimeContractTests(unittest.TestCase): + def test_authority_and_checkpoint_topics_are_compacted(self): + self.assertEqual(TOPIC_POLICIES["qdl.authority.v1"], "compact") + self.assertEqual(TOPIC_POLICIES["qdl.target-checkpoint.v1"], "compact") + self.assertEqual(TOPIC_POLICIES["md.canonical.v2"], "delete") + source = ( + ROOT / "scripts/phaseb_bootstrap_stable_broker.py" + ).read_text(encoding="utf-8") + self.assertIn('"stable-authority-dispatcher"', source) + self.assertIn('"qdl-v2-production-core-"', source) + self.assertIn('"qdl-v2-production-core-v1-"', source) + + def test_dispatcher_role_is_function_scoped_and_not_table_writer(self): + migration = ( + ROOT / "migrations/postgres/0010_authority_dispatcher_security.sql" + ).read_text(encoding="utf-8") + init = ( + ROOT / "migrations/postgres/9999_init_authority_dispatcher_role.sh" + ).read_text(encoding="utf-8") + self.assertEqual(migration.count("SECURITY DEFINER"), 3) + self.assertEqual(migration.count("SET search_path = pg_catalog, public"), 3) + self.assertEqual(migration.count("FROM PUBLIC"), 3) + self.assertIn("NOSUPERUSER NOCREATEDB NOCREATEROLE", init) + self.assertEqual(init.count("GRANT EXECUTE ON FUNCTION"), 3) + self.assertNotIn("GRANT INSERT", init) + self.assertNotIn("GRANT UPDATE", init) + self.assertNotIn("GRANT DELETE", init) + + def test_dispatcher_health_is_atomic_bounded_and_fail_closed(self): + with tempfile.TemporaryDirectory(prefix="qdl-authority-health-") as directory: + path = Path(directory) / "runtime/health.json" + write_health(path, status="READY", published=3) + payload = json.loads(path.read_text(encoding="utf-8")) + self.assertEqual(payload["schema"], "qdl.authority-dispatcher-health.v1") + self.assertEqual(payload["status"], "READY") + self.assertEqual(payload["published_last_cycle"], 3) + self.assertGreater(payload["heartbeat_ns"], 0) + self.assertFalse(path.with_suffix(".json.tmp").exists()) + + write_health( + path, status="DEGRADED", published=0, error="x" * 2000 + ) + degraded = json.loads(path.read_text(encoding="utf-8")) + self.assertEqual(degraded["status"], "DEGRADED") + self.assertEqual(len(degraded["error"]), 1000) + with self.assertRaises(ValueError): + write_health(path, status="UNKNOWN", published=0) + with self.assertRaises(ValueError): + write_health(path, status="READY", published=-1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_phasec3_cutover_packet.py b/tests/test_phasec3_cutover_packet.py new file mode 100644 index 0000000..e495fc8 --- /dev/null +++ b/tests/test_phasec3_cutover_packet.py @@ -0,0 +1,136 @@ +from __future__ import annotations + +import copy +import unittest + +from qdl.control.cutover_packet import AuthorityCutoverPacket +from scripts.phasec3_authority_cutover import _assert_current + + +NOW = 1_800_000_000_000_000_000 +DIGEST = "a" * 64 + + +def packet(stage: str = "CANARY") -> dict: + pairs = { + "SHADOW_VALIDATE": ("RUST_SHADOW", "VALIDATING"), + "CANARY": ("VALIDATING", "RUST_CANARY"), + "PRIMARY": ("RUST_CANARY", "RUST_PRIMARY"), + } + expected, new = pairs[stage] + active = stage in {"CANARY", "PRIMARY"} + return { + "schema": "qdl.c3.authority-cutover-packet.v1", + "packet_id": "10000000-0000-4000-8000-000000000001", + "stage": stage, + "issued_at_ns": NOW - 1, + "expires_at_ns": NOW + 1_000_000_000, + "actor": "operator@example", + "change_ticket": "QDL-C3-001", + "candidate_digest": DIGEST, + "artifact_image_digest": "sha256:" + "b" * 64, + "contract_digest": "c" * 64, + "partition_plan_digest": "d" * 64, + "route_manifest_digest": "e" * 64, + "consumer_route": { + "consumer_id": "trading-system", + "expected_route": "V1", + "new_route": "V2_PRIMARY", + "rollback_route": "V1", + "rollback_command": ["scripts/qdl_route.py", "--route", "V1"], + }, + "evidence": { + "provider_provenance": "REAL", + "semantic_mismatches": 0, + "open_gaps": 0, + "duplicate_external_effects": 0, + "consumer_errors": 0, + }, + "slices": [{ + "transition_id": "20000000-0000-4000-8000-000000000002", + "handoff_id": ( + "30000000-0000-4000-8000-000000000003" + if stage == "PRIMARY" else None + ), + "slice_id": "production/binance/usdm/perpetual/trade/plan-1/btcusdt", + "expected_state": expected, + "expected_revision": 2, + "expected_owner_id": "rust-shadow", + "expected_lease_epoch": 1, + "partition_plan_epoch": 1, + "new_state": new, + "new_owner_id": "rust-canary", + "new_lease_epoch": 2, + "terminal_watermark": 500 if active else None, + "prerequisite_bundle_id": ( + "40000000-0000-4000-8000-000000000004" if active else None + ), + "hold_until": "2026-08-20T12:00:00Z" if active else None, + "reason": "bounded C.3 transition", + }], + } + + +class AuthorityCutoverPacketTests(unittest.TestCase): + def test_plan_is_immutable_and_has_no_mutation(self): + parsed = AuthorityCutoverPacket.parse(packet(), now_ns=NOW) + plan = parsed.plan() + self.assertEqual(plan["stage"], "CANARY") + self.assertEqual(plan["slice_count"], 1) + self.assertEqual(plan["production_mutations"], 0) + self.assertTrue(plan["confirmation_token"].startswith("APPLY_C3_")) + + changed = packet() + changed["change_ticket"] = "QDL-C3-002" + changed_packet = AuthorityCutoverPacket.parse(changed, now_ns=NOW) + self.assertNotEqual( + parsed.confirmation_token, changed_packet.confirmation_token + ) + + def test_unknown_dirty_or_invalid_transition_fails_closed(self): + cases = [] + unknown = packet() + unknown["unexpected"] = True + cases.append(unknown) + dirty = packet() + dirty["evidence"]["semantic_mismatches"] = 1 + cases.append(dirty) + wrong_pair = packet() + wrong_pair["slices"][0]["new_state"] = "RUST_PRIMARY" + cases.append(wrong_pair) + missing_rollback = packet() + missing_rollback["consumer_route"]["rollback_command"] = [] + cases.append(missing_rollback) + for value in cases: + with self.subTest(value=value): + with self.assertRaises(ValueError): + AuthorityCutoverPacket.parse(value, now_ns=NOW) + + def test_primary_requires_handoff_and_exact_database_preconditions(self): + parsed = AuthorityCutoverPacket.parse(packet("PRIMARY"), now_ns=NOW) + item = parsed.slices[0] + row = { + "state": item.expected_state, + "authority_revision": item.expected_revision, + "owner_id": item.expected_owner_id, + "lease_epoch": item.expected_lease_epoch, + "partition_plan_epoch": item.partition_plan_epoch, + "candidate_digest": parsed.candidate_digest, + "artifact_image_digest": parsed.artifact_image_digest, + "contract_digest": parsed.contract_digest, + "partition_plan_digest": parsed.partition_plan_digest, + } + _assert_current(row, item, parsed) + stale = copy.deepcopy(row) + stale["authority_revision"] += 1 + with self.assertRaises(RuntimeError): + _assert_current(stale, item, parsed) + + no_handoff = packet("PRIMARY") + no_handoff["slices"][0]["handoff_id"] = None + with self.assertRaises(ValueError): + AuthorityCutoverPacket.parse(no_handoff, now_ns=NOW) + + +if __name__ == "__main__": + unittest.main() From 5823d642027b7446aa72160aa2ec53c28fdd88f1 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 14:31:12 +0000 Subject: [PATCH 15/21] build(v2): pin Python release base image --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 9 +++++++++ Dockerfile | 4 ++-- tests/test_phaseb_stable_deployment.py | 10 ++++++++++ 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index daa75eb..ce7b072 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6330,6 +6330,15 @@ freshness/session/contract checks and the source-switch audit is durable. slice; changed files have zero finding. No authority DB/volume, production CAS, Trading System route, V1 service or provider ownership was mutated. +- `2026-08-20 C.3 IMMUTABLE BUILD HYGIENE STARTED`: release preflight + found the Python builder/runtime base referenced a mutable tag while Rust + bases were digest-pinned. Pin both Python stages to the locally resolved + official image digest before building the commit-SHA release; verify both + stages use the same digest, rebuild, inspect OCI revision/version/non-root + identity, rerun image-level smoke and retain V1 unchanged. The source/test + slice pins both stages to the same official digest; focused contract tests + passed 14/14, changed-file Ruff and diff checks passed. + Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One operator packet may list the complete slice set, image IDs, old/new owners, diff --git a/Dockerfile b/Dockerfile index dc6654c..ea152a3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.12-slim AS builder +FROM python:3.12-slim@sha256:2c941e860699f878900b0edc2403613c234d4b32eda3cc9fa7036991a2a63c4a AS builder ENV PYTHONUNBUFFERED=1 ENV PYTHONDONTWRITEBYTECODE=1 @@ -17,7 +17,7 @@ RUN python -m venv /opt/venv && \ poetry install --no-root --only main --no-ansi && \ /opt/venv/bin/python -m pip install --no-cache-dir --upgrade "setuptools>=78.1.1" -FROM python:3.12-slim AS runtime +FROM python:3.12-slim@sha256:2c941e860699f878900b0edc2403613c234d4b32eda3cc9fa7036991a2a63c4a AS runtime ARG QDL_UID=10001 ARG QDL_GID=10001 diff --git a/tests/test_phaseb_stable_deployment.py b/tests/test_phaseb_stable_deployment.py index bae1df5..bd7df87 100644 --- a/tests/test_phaseb_stable_deployment.py +++ b/tests/test_phaseb_stable_deployment.py @@ -697,6 +697,16 @@ def test_missing_binding_wrong_provider_kind_and_primary_authority_fail_closed(s class StableComposeAndBundleTests(unittest.TestCase): + def test_python_release_base_is_digest_pinned_in_both_stages(self): + dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8") + pinned = ( + "python:3.12-slim@sha256:" + "2c941e860699f878900b0edc2403613c234d4b32" + "eda3cc9fa7036991a2a63c4a" + ) + self.assertEqual(dockerfile.count(pinned), 2) + self.assertNotIn("FROM python:3.12-slim AS", dockerfile) + def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): raw = (ROOT / "docker-compose.v2-stable.yml").read_text(encoding="utf-8") compose = yaml.safe_load(raw) From 3d3af1c530e1dd52b402294e0bb677eb334a15a2 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 15:53:21 +0000 Subject: [PATCH 16/21] fix(v2): scope initial Rust authority promotion --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 52 +++++++++++++ .../v2/stable-authority-promotion-scope.yaml | 15 ++++ qdl/runtime/stable_deployment.py | 73 ++++++++++++++++++- scripts/phaseb_prepare_stable_candidate.py | 9 +++ tests/test_phaseb_stable_deployment.py | 62 ++++++++++++++++ tests/test_production_catalog.py | 11 ++- 6 files changed, 219 insertions(+), 3 deletions(-) create mode 100644 config/v2/stable-authority-promotion-scope.yaml diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index ce7b072..9dcd2ed 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6339,6 +6339,58 @@ freshness/session/contract checks and the source-switch audit is durable. slice pins both stages to the same official digest; focused contract tests passed 14/14, changed-file Ruff and diff checks passed. +- `2026-08-20 C.3 FINAL IMMUTABLE ARTIFACT GATE PASS`: commit + `5823d642027b7446aa72160aa2ec53c28fdd88f1` produced Python image + `sha256:1758b35646293eca717d269681b867fc485db896a70889bab53df47d8d87345f` + and Rust image + `sha256:1eda689c30484157092cc276a1487d36174acd1a97a353ed792642a6d5512211`. + Both images expose OCI version `2.0.0` and the exact full revision; Python + runs as `qdl:qdl`, Rust as UID/GID `10001:10001`. A network-none Python + image smoke imported the API and source-owned `qdl_sdk==2.0.0`; the Rust + image contains `qdl-production-core`, which failed closed with its usage + error when started without a config. A fresh private bundle generated from + those exact image IDs reported 12 runtime files, `RUST_SHADOW`, + `cutover_authorized=false` and no secret values in its public manifest. + Its manifest SHA-256 is + `a9d2835e86c0f6b2be7f90f7671d2f3d8dc9462da324703658991da774b4b1cb`; + Compose rendered successfully with both `stable-authority` and + `stable-authority-primary` profiles. No container, authority row, consumer + route, V1 service, provider ownership or persistent volume changed. The + next permitted operation is topology/packet preflight; a production CAS and + Trading System restart still require the exact packet approval below. + + +- `2026-08-20 C.3 PROMOTION-SCOPE BLOCKER FOUND; ARTIFACT REVOKED`: packet + preflight inspected the generated production-core configs and found all four + DNSE bindings present alongside the twelve approved Binance/OKX bindings. + This violates the explicit initial-cutover boundary that DNSE remains V1-only + and would make a production worker require DNSE authority/checkpoints even + when the `stable-vn` profile is disabled. The two image IDs above are valid + build evidence but are revoked as cutover artifacts. Fix the generator with + one strict, versioned, explicit authority-promotion binding manifest; filter + both canonical bindings and runtime slices from that manifest, reject empty, + duplicate or unknown bindings, and record its digest in the bundle. Add a + regression proving initial authority contains exactly twelve Binance/OKX + bindings and zero HNX/HOSE/DNSE binding. Re-run focused/full gates and rebuild + one new immutable image pair before topology deployment. V1 and the running + isolated shadow stack remain unchanged while this source-only repair runs. + + +- `2026-08-20 C.3 PROMOTION-SCOPE REPAIR PASS`: added strict manifest + `qdl.v2.authority-promotion-scope.v1`; production-core generation now filters + both canonical bindings and authority slices from its explicit binding IDs, + rejects empty/duplicate/unknown scope and records revision/digest/count in + the public bundle. The initial manifest selects exactly twelve Binance/OKX + trade/quote/final-1m-bar bindings and no DNSE/HNX/HOSE binding. Targeted + contract/bundle/authority tests passed 22/22; full Python passed 543 with six + environment skips; full Rust passed 70/70 with fmt and strict Clippy; isolated + changed-file Ruff passed. All three generated production workers contain + 12 slices, venues `BINANCE,OKX`, zero DNSE subscriptions and common scope + digest `06178202d7ec592c19c41a36c919a13a74971c3e39ed8e67ce9b5de3a978fcd2`. + Compose authority profiles render successfully. Tests used network-none + source mounts and disposable tmpfs/tooling; V1, the running isolated shadow, + Trading System routes, authority state and persistent volumes were unchanged. + Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One operator packet may list the complete slice set, image IDs, old/new owners, diff --git a/config/v2/stable-authority-promotion-scope.yaml b/config/v2/stable-authority-promotion-scope.yaml new file mode 100644 index 0000000..0c90a42 --- /dev/null +++ b/config/v2/stable-authority-promotion-scope.yaml @@ -0,0 +1,15 @@ +schema: qdl.v2.authority-promotion-scope.v1 +revision: 1 +binding_ids: + - binance-spot-btcusdt-bar-1m + - binance-spot-btcusdt-quote + - binance-spot-btcusdt-trade + - binance-usdm-btcusdt-bar-1m + - binance-usdm-btcusdt-quote + - binance-usdm-btcusdt-trade + - okx-spot-btcusdt-bar-1m + - okx-spot-btcusdt-quote + - okx-spot-btcusdt-trade + - okx-swap-btcusdt-bar-1m + - okx-swap-btcusdt-quote + - okx-swap-btcusdt-trade diff --git a/qdl/runtime/stable_deployment.py b/qdl/runtime/stable_deployment.py index 1b6f5d2..350153c 100644 --- a/qdl/runtime/stable_deployment.py +++ b/qdl/runtime/stable_deployment.py @@ -31,6 +31,59 @@ } +@dataclass(frozen=True, slots=True) +class AuthorityPromotionScope: + schema: str + revision: int + binding_ids: tuple[str, ...] + + def __post_init__(self) -> None: + if ( + self.schema != "qdl.v2.authority-promotion-scope.v1" + or self.revision < 1 + or not self.binding_ids + or any(not value.strip() for value in self.binding_ids) + or len(self.binding_ids) != len(set(self.binding_ids)) + ): + raise ValueError("authority promotion scope is invalid") + + @classmethod + def load( + cls, path: str | Path, *, catalog: StableSourceCatalog + ) -> "AuthorityPromotionScope": + payload = yaml.safe_load(Path(path).read_text(encoding="utf-8")) + if not isinstance(payload, dict) or set(payload) != { + "schema", "revision", "binding_ids", + }: + raise ValueError("authority promotion scope fields are incomplete or unknown") + values = payload["binding_ids"] + if not isinstance(values, list) or not values: + raise ValueError("authority promotion scope requires binding IDs") + result = cls( + schema=str(payload["schema"]), + revision=int(payload["revision"]), + binding_ids=tuple(str(value) for value in values), + ) + catalog_ids = {item.binding_id for item in catalog.bindings} + unknown = set(result.binding_ids) - catalog_ids + if unknown: + raise ValueError( + "authority promotion scope contains unknown bindings: " + + ",".join(sorted(unknown)) + ) + return result + + def digest(self) -> str: + payload = { + "schema": self.schema, + "revision": self.revision, + "binding_ids": list(self.binding_ids), + } + return hashlib.sha256( + json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + + @dataclass(frozen=True, slots=True) class StableAcquisitionBinding: binding_id: str @@ -175,14 +228,18 @@ def core_config( authority: Mapping[str, Any], max_events: int = 0, worker_index: int = 1, + binding_ids: frozenset[str] | None = None, ) -> dict[str, Any]: self._validate_authority(authority) if not 1 <= worker_index <= STABLE_CORE_WORKER_COUNT: raise ValueError("stable core worker index is outside the topology bound") source_by_id = {item.binding_id: item for item in catalog.bindings} acquisitions = {item.binding_id: item for item in self.bindings} + selected_ids = frozenset(source_by_id) if binding_ids is None else binding_ids + if not selected_ids or not selected_ids.issubset(source_by_id): + raise ValueError("stable core binding selection is empty or unknown") bindings = [] - for binding_id in sorted(source_by_id): + for binding_id in sorted(selected_ids): source = source_by_id[binding_id] acquisition = acquisitions[binding_id] identity = source.instrument.identity @@ -227,19 +284,26 @@ def production_core_config( *, catalog: StableSourceCatalog, raw_authority: Mapping[str, Any], + promotion_scope: AuthorityPromotionScope, worker_index: int, partition_plan_epoch: int = 1, ) -> dict[str, Any]: self._validate_authority(raw_authority) if partition_plan_epoch < 1: raise ValueError("production partition plan epoch must be positive") + selected_ids = frozenset(promotion_scope.binding_ids) shadow = self.core_config( catalog=catalog, authority=raw_authority, worker_index=worker_index, + binding_ids=selected_ids, ) + source_by_id = {item.binding_id: item for item in catalog.bindings} slices = [] - for source in sorted(catalog.bindings, key=lambda value: value.binding_id): + for source in sorted( + (source_by_id[binding_id] for binding_id in selected_ids), + key=lambda value: value.binding_id, + ): identity = source.instrument.identity native = source.instrument.native_symbol.lower() slice_id = ( @@ -358,6 +422,7 @@ def write_production_core_bundle( *, catalog: StableSourceCatalog, acquisition: StableAcquisitionPlan, + promotion_scope: AuthorityPromotionScope, raw_authority: Mapping[str, Any], partition_plan_epoch: int = 1, ) -> dict[str, str]: @@ -367,6 +432,7 @@ def write_production_core_bundle( acquisition.production_core_config( catalog=catalog, raw_authority=raw_authority, + promotion_scope=promotion_scope, worker_index=worker_index, partition_plan_epoch=partition_plan_epoch, ) @@ -384,6 +450,9 @@ def write_production_core_bundle( "schema": "qdl.v2.production-core-bundle.v1", "partition_plan_epoch": partition_plan_epoch, "worker_count": STABLE_CORE_WORKER_COUNT, + "promotion_scope_revision": promotion_scope.revision, + "promotion_scope_digest": promotion_scope.digest(), + "promotion_binding_count": len(promotion_scope.binding_ids), "files": digests, } encoded_manifest = ( diff --git a/scripts/phaseb_prepare_stable_candidate.py b/scripts/phaseb_prepare_stable_candidate.py index c87f730..6551ad7 100755 --- a/scripts/phaseb_prepare_stable_candidate.py +++ b/scripts/phaseb_prepare_stable_candidate.py @@ -15,6 +15,7 @@ from qdl.runtime.stable_catalog import StableSourceCatalog from qdl.runtime.stable_deployment import ( + AuthorityPromotionScope, StableAcquisitionPlan, stable_authority_record, write_production_core_bundle, @@ -93,6 +94,10 @@ def prepare_candidate( ) acquisition_path = ROOT / "config/v2/stable-acquisition-bindings.yaml" acquisition = StableAcquisitionPlan.load(acquisition_path, catalog=catalog) + promotion_scope = AuthorityPromotionScope.load( + ROOT / "config/v2/stable-authority-promotion-scope.yaml", + catalog=catalog, + ) authority = stable_authority_record( rust_image_digest=rust_digest, capability_manifest=ROOT / "config/v2/stable-capabilities.yaml", @@ -110,6 +115,7 @@ def prepare_candidate( runtime_dir, catalog=catalog, acquisition=acquisition, + promotion_scope=promotion_scope, raw_authority=authority, partition_plan_epoch=1, )) @@ -208,6 +214,9 @@ def prepare_candidate( "runtime_digests": bundle_digests, "catalog_revision": catalog.catalog_revision, "acquisition_revision": acquisition.revision, + "authority_promotion_scope_revision": promotion_scope.revision, + "authority_promotion_scope_digest": promotion_scope.digest(), + "authority_promotion_binding_count": len(promotion_scope.binding_ids), "consumer_count": 5, "workload_mtls": True, "workload_identity_count": 4, diff --git a/tests/test_phaseb_stable_deployment.py b/tests/test_phaseb_stable_deployment.py index bd7df87..372d404 100644 --- a/tests/test_phaseb_stable_deployment.py +++ b/tests/test_phaseb_stable_deployment.py @@ -19,6 +19,7 @@ from qdl.runtime.stable_deployment import ( STABLE_CORE_WORKER_COUNT, + AuthorityPromotionScope, StableAcquisitionPlan, stable_authority_record, write_stable_runtime_bundle, @@ -28,6 +29,7 @@ ROOT = Path(__file__).resolve().parents[1] CATALOG_PATH = ROOT / "config/v2/stable-source-bindings.yaml" ACQUISITION_PATH = ROOT / "config/v2/stable-acquisition-bindings.yaml" +PROMOTION_SCOPE_PATH = ROOT / "config/v2/stable-authority-promotion-scope.yaml" class StableDeploymentContractTests(unittest.TestCase): @@ -36,6 +38,9 @@ def setUp(self) -> None: self.acquisition = StableAcquisitionPlan.load( ACQUISITION_PATH, catalog=self.catalog ) + self.promotion_scope = AuthorityPromotionScope.load( + PROMOTION_SCOPE_PATH, catalog=self.catalog + ) self.authority = stable_authority_record( rust_image_digest="a" * 64, capability_manifest=ROOT / "config/v2/stable-capabilities.yaml", @@ -44,6 +49,53 @@ def setUp(self) -> None: effective_at_ns=time.time_ns(), ) + def test_initial_authority_scope_is_explicit_and_excludes_dnse(self): + expected = { + item.binding_id + for item in self.catalog.bindings + if item.instrument.identity.venue in {"BINANCE", "OKX"} + } + self.assertEqual(set(self.promotion_scope.binding_ids), expected) + self.assertEqual(len(expected), 12) + runtime = self.acquisition.production_core_config( + catalog=self.catalog, + raw_authority=self.authority, + promotion_scope=self.promotion_scope, + worker_index=1, + ) + self.assertEqual(len(runtime["slices"]), 12) + self.assertEqual( + {item["subscription_id"] for item in runtime["slices"]}, + { + item.source_id + for item in self.catalog.bindings + if item.binding_id in expected + }, + ) + self.assertEqual( + {item["venue"] for item in runtime["core"]["bindings"]}, + {"BINANCE", "OKX"}, + ) + + def test_authority_scope_rejects_unknown_and_duplicate_bindings(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "scope.yaml" + path.write_text( + "schema: qdl.v2.authority-promotion-scope.v1\n" + "revision: 1\nbinding_ids: [missing-binding]\n", + encoding="utf-8", + ) + with self.assertRaisesRegex(ValueError, "unknown bindings"): + AuthorityPromotionScope.load(path, catalog=self.catalog) + path.write_text( + "schema: qdl.v2.authority-promotion-scope.v1\n" + "revision: 1\nbinding_ids: [binance-usdm-btcusdt-trade, " + "binance-usdm-btcusdt-trade]\n", + encoding="utf-8", + ) + with self.assertRaisesRegex(ValueError, "scope is invalid"): + AuthorityPromotionScope.load(path, catalog=self.catalog) + def test_all_catalog_bindings_have_one_capability_truthful_acquisition(self): self.assertEqual(len(self.catalog.bindings), 16) self.assertEqual(len(self.acquisition.bindings), 16) @@ -905,6 +957,16 @@ def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): ) self.assertFalse(manifest["cutover_authorized"]) self.assertFalse(manifest["secret_values_recorded"]) + self.assertEqual(manifest["authority_promotion_binding_count"], 12) + self.assertEqual(len(manifest["authority_promotion_scope_digest"]), 64) + production = json.loads( + (output / "runtime/production-core-001.json").read_text() + ) + self.assertEqual(len(production["slices"]), 12) + self.assertEqual( + {item["venue"] for item in production["core"]["bindings"]}, + {"BINANCE", "OKX"}, + ) self.assertEqual((output / "stable.env").stat().st_mode & 0o777, 0o600) env_text = (output / "stable.env").read_text() self.assertIn("QDL_STABLE_CERT_DIR=/host/qdl/certs", env_text) diff --git a/tests/test_production_catalog.py b/tests/test_production_catalog.py index 56525ed..dbdd01f 100644 --- a/tests/test_production_catalog.py +++ b/tests/test_production_catalog.py @@ -11,7 +11,10 @@ ProductionDemandManifest, ) from qdl.runtime.stable_catalog import StableSourceCatalog -from qdl.runtime.stable_deployment import StableAcquisitionPlan +from qdl.runtime.stable_deployment import ( + AuthorityPromotionScope, + StableAcquisitionPlan, +) BINANCE = { @@ -173,9 +176,15 @@ def test_generator_is_deterministic_validated_and_metadata_authentic(self): "approved_by": "production-catalog-test", "effective_at_ns": 1, } + promotion_scope = AuthorityPromotionScope( + schema="qdl.v2.authority-promotion-scope.v1", + revision=1, + binding_ids=tuple(item.binding_id for item in catalog.bindings), + ) runtime = acquisition.production_core_config( catalog=catalog, raw_authority=raw_authority, + promotion_scope=promotion_scope, worker_index=1, ) self.assertEqual(len(runtime["slices"]), 2) From 533f7c606b97f00dbd484fa02746b3a6a5c9dea8 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 15:56:28 +0000 Subject: [PATCH 17/21] docs(v2): record final authority release pair --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 9dcd2ed..9e3ba96 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6391,6 +6391,22 @@ freshness/session/contract checks and the source-switch audit is durable. source mounts and disposable tmpfs/tooling; V1, the running isolated shadow, Trading System routes, authority state and persistent volumes were unchanged. + +- `2026-08-20 C.3 REBUILT RELEASE PAIR PASS`: tested repair commit + `3d3af1c530e1dd52b402294e0bb677eb334a15a2` produced Python image + `sha256:e61c7cb1372071daeb3f9753e616b073b514998845abc61ab168b2cb63617e90` + and Rust image + `sha256:676de79940ed83cc45a8c1490055c8fa69ddc5bcb032af4ab6a4851d25e921b6`. + Both carry exact revision/version labels and retain non-root users. Image-level + network-none smoke imported `qdl_sdk==2.0.0`; `qdl-production-core` remained + fail-closed without config. The fresh bundle binds those exact IDs, reports + `RUST_SHADOW`, `cutover_authorized=false`, scope digest + `06178202d7ec592c19c41a36c919a13a74971c3e39ed8e67ce9b5de3a978fcd2` + and twelve approved bindings; authority Compose profiles render cleanly. + This pair supersedes the revoked `5823d642` pair. No running container or + authority/consumer route changed. Merge/immutable deployment and the exact + operator packet remain the only gates before bounded runtime promotion. + Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One operator packet may list the complete slice set, image IDs, old/new owners, From a4c283c0657205ef63c1bf7aec92f0f6186ebf0c Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 15:57:34 +0000 Subject: [PATCH 18/21] docs(v2): record scoped release cleanup --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 9e3ba96..49bd64c 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6406,6 +6406,9 @@ freshness/session/contract checks and the source-switch audit is durable. This pair supersedes the revoked `5823d642` pair. No running container or authority/consumer route changed. Merge/immutable deployment and the exact operator packet remain the only gates before bounded runtime promotion. + Exact cleanup removed the two revoked `5823d642` image tags and three + disposable test/revoked-bundle paths only; no broad prune, active image, + final release bundle, V1 rollback artifact or volume was removed. Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One From be35aa7389a37b31c21cc2689c25873dcfc7e73d Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 16:04:26 +0000 Subject: [PATCH 19/21] fix(v2): expose stable ingress to governed consumer network --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 28 +++++++++++++++++ docker-compose.v2-stable.yml | 23 +++++++++++--- .../v2-production-rust-authority-cutover.md | 7 +++-- scripts/phaseb_prepare_stable_candidate.py | 8 +++++ tests/test_phaseb_stable_deployment.py | 30 ++++++++++++++++++- 5 files changed, 89 insertions(+), 7 deletions(-) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 49bd64c..ebab503 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6410,6 +6410,34 @@ freshness/session/contract checks and the source-switch audit is durable. disposable test/revoked-bundle paths only; no broad prune, active image, final release bundle, V1 rollback artifact or volume was removed. + +- `2026-08-20 C.2 CONSUMER-NETWORK BLOCKER FOUND`: final deployment + preflight compared Data Layer and Trading System Compose topology. Stable V2 + query/stream roles only join project-private networks and expose loopback host + ports, while Trading System resolves `qdl-v2-query` and + `qdl-v2-stream-a/b` from external `executor_network`; the container cannot + reach host loopback, so a real consumer cutover would fail despite valid SDK + and mTLS tests. Add one explicit generated external-consumer-network setting, + attach only the two query and two stream ingress roles with the frozen DNS + aliases, and keep Kafka/Redis/projector/Rust core off that network. Require + Compose contract tests for aliases/isolation plus existing full regressions. + Rebuild the same-SHA release pair after this bounded topology repair. No + running network/container is changed by the source fix. + + +- `2026-08-20 C.2 CONSUMER-NETWORK REPAIR PASS`: stable bundle generation now + requires a validated external consumer network and records it in private env + plus the non-secret manifest. Only query replicas join it as + `qdl-v2-query`; only active/passive stream roles join as + `qdl-v2-stream-a/b`. Kafka, Redis, projector, Rust shadow/primary cores and + ingestors remain absent from that network. Generated Compose validated + against existing external `executor_network`; focused tests passed 19/19, + full Python passed 543 with six environment skips, changed-file Ruff passed, + and the canonical cutover runbook now requires the network explicitly. + No container was attached, recreated or restarted; port 8100 and Trading + System remained unchanged. Commit and one final same-SHA image rebuild are + required before PR/cutover. + Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One operator packet may list the complete slice set, image IDs, old/new owners, diff --git a/docker-compose.v2-stable.yml b/docker-compose.v2-stable.yml index 5d6f91a..a061bd3 100644 --- a/docker-compose.v2-stable.yml +++ b/docker-compose.v2-stable.yml @@ -232,7 +232,10 @@ services: query_v2_1: <<: *python command: [python, -m, app.entrypoints.query_v2_stable] - networks: [stable_internal, stable_ingress] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-query]} ports: ["127.0.0.1:18201:8200"] environment: <<: *stable-env @@ -250,7 +253,10 @@ services: query_v2_2: <<: *python command: [python, -m, app.entrypoints.query_v2_stable] - networks: [stable_internal, stable_ingress] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-query]} ports: ["127.0.0.1:18202:8200"] environment: <<: *stable-env @@ -267,7 +273,10 @@ services: stream_v2_active: <<: *python command: [python, -m, app.entrypoints.stream_v2_stable] - networks: [stable_internal, stable_ingress] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-stream-a]} ports: ["127.0.0.1:18210:8200", "127.0.0.1:18220:8210"] environment: <<: *stable-env @@ -284,7 +293,10 @@ services: stream_v2_passive: <<: *python command: [python, -m, app.entrypoints.stream_v2_stable] - networks: [stable_internal, stable_ingress] + networks: + stable_internal: {} + stable_ingress: {} + stable_consumer: {aliases: [qdl-v2-stream-b]} ports: ["127.0.0.1:18211:8200", "127.0.0.1:18221:8210"] environment: <<: *stable-env @@ -603,6 +615,9 @@ networks: stable_internal: {internal: true, labels: {qdl.scope: v2-stable-candidate}} stable_ingress: {labels: {qdl.scope: v2-stable-candidate-ingress}} stable_egress: {labels: {qdl.scope: v2-stable-candidate-egress}} + stable_consumer: + external: true + name: ${QDL_STABLE_CONSUMER_NETWORK:?set QDL_STABLE_CONSUMER_NETWORK} volumes: kafka1_data: {labels: {qdl.scope: v2-stable-candidate}} kafka2_data: {labels: {qdl.scope: v2-stable-candidate}} diff --git a/docs/runbooks/v2-production-rust-authority-cutover.md b/docs/runbooks/v2-production-rust-authority-cutover.md index 600842b..cd125eb 100644 --- a/docs/runbooks/v2-production-rust-authority-cutover.md +++ b/docs/runbooks/v2-production-rust-authority-cutover.md @@ -154,12 +154,15 @@ python scripts/phaseb_prepare_stable_candidate.py \ --python-image "$PYTHON_IMAGE" \ --rust-image "$RUST_IMAGE" \ --cert-dir /path/to/approved/phase8-certificates \ - --output-dir "$QDL_RELEASE_ROOT" + --output-dir "$QDL_RELEASE_ROOT" \ + --consumer-network executor_network ``` The manifest must report contract `2.0.0`, authority `RUST_SHADOW`, `cutover_authorized=false`, immutable image IDs, five consumer manifests and -no recorded secret values. +no recorded secret values. `--consumer-network` must name an already-created +external network shared with the sole approved consumer. Only V2 query/stream +ingress joins it; Kafka, Redis, projector and Rust cores remain private. ## Gate 4 - Start Isolated V2 diff --git a/scripts/phaseb_prepare_stable_candidate.py b/scripts/phaseb_prepare_stable_candidate.py index 6551ad7..6b122e4 100755 --- a/scripts/phaseb_prepare_stable_candidate.py +++ b/scripts/phaseb_prepare_stable_candidate.py @@ -4,6 +4,7 @@ import argparse import hashlib import json +import re import secrets import shutil import subprocess @@ -73,6 +74,7 @@ def prepare_candidate( python_image: str, cert_dir: Path, output_dir: Path, + consumer_network: str, rust_image_id: str | None = None, python_image_id: str | None = None, host_cert_dir: Path | None = None, @@ -80,6 +82,8 @@ def prepare_candidate( ) -> dict[str, object]: if output_dir.exists() and any(output_dir.iterdir()): raise FileExistsError("stable candidate output directory must be empty") + if re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}", consumer_network) is None: + raise ValueError("stable consumer network name is invalid") output_dir.mkdir(parents=True, exist_ok=True) runtime_dir = output_dir / "runtime" identities_dir = output_dir / "identities" @@ -154,6 +158,7 @@ def prepare_candidate( compose_output_dir = (host_output_dir or output_dir).resolve() values = { "QDL_STABLE_SCHEMA_DIGEST": schema_digest, + "QDL_STABLE_CONSUMER_NETWORK": consumer_network, "QDL_STABLE_INTERNAL_INGEST_SECRET": ingest_secret, "QDL_STABLE_CURSOR_KEYS_JSON": json.dumps( {"stable-k1": cursor_secret}, separators=(",", ":") @@ -217,6 +222,7 @@ def prepare_candidate( "authority_promotion_scope_revision": promotion_scope.revision, "authority_promotion_scope_digest": promotion_scope.digest(), "authority_promotion_binding_count": len(promotion_scope.binding_ids), + "consumer_network": consumer_network, "consumer_count": 5, "workload_mtls": True, "workload_identity_count": 4, @@ -236,6 +242,7 @@ def main() -> int: parser.add_argument("--python-image", required=True) parser.add_argument("--cert-dir", type=Path, required=True) parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--consumer-network", required=True) parser.add_argument("--rust-image-id") parser.add_argument("--python-image-id") parser.add_argument("--host-cert-dir", type=Path) @@ -246,6 +253,7 @@ def main() -> int: python_image=args.python_image, cert_dir=args.cert_dir, output_dir=args.output_dir, + consumer_network=args.consumer_network, rust_image_id=args.rust_image_id, python_image_id=args.python_image_id, host_cert_dir=args.host_cert_dir, diff --git a/tests/test_phaseb_stable_deployment.py b/tests/test_phaseb_stable_deployment.py index 372d404..a21cc30 100644 --- a/tests/test_phaseb_stable_deployment.py +++ b/tests/test_phaseb_stable_deployment.py @@ -769,7 +769,8 @@ def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): self.assertFalse(compose["networks"]["stable_ingress"].get("internal", False)) for name in ("query_v2_1", "query_v2_2", "stream_v2_active", "stream_v2_passive"): self.assertEqual( - set(services[name]["networks"]), {"stable_internal", "stable_ingress"} + set(services[name]["networks"]), + {"stable_internal", "stable_ingress", "stable_consumer"}, ) self.assertTrue( all(str(port).startswith("127.0.0.1:") for port in services[name]["ports"]) @@ -814,6 +815,30 @@ def test_compose_is_isolated_bounded_nonroot_and_has_no_v1_route(self): self.assertTrue(services[name]["read_only"]) self.assertIn("ALL", services[name]["cap_drop"]) self.assertEqual(services[name]["restart"], "no") + ingress_aliases = { + "query_v2_1": "qdl-v2-query", + "query_v2_2": "qdl-v2-query", + "stream_v2_active": "qdl-v2-stream-a", + "stream_v2_passive": "qdl-v2-stream-b", + } + for name, alias in ingress_aliases.items(): + self.assertEqual( + services[name]["networks"]["stable_consumer"]["aliases"], + [alias], + ) + self.assertEqual( + compose["networks"]["stable_consumer"], + { + "external": True, + "name": "${QDL_STABLE_CONSUMER_NETWORK:" + "?set QDL_STABLE_CONSUMER_NETWORK}", + }, + ) + for name in ( + "kafka1", "kafka2", "kafka3", "stable_redis", "projector_v2", + "rust_core", "rust_core_2", "rust_core_3", + ): + self.assertNotIn("stable_consumer", services[name]["networks"]) self.assertEqual( set(services["ingestor_okx_swap"]["networks"]), {"stable_internal", "stable_egress"}, @@ -952,12 +977,14 @@ def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): python_image="qdl-python:test", cert_dir=certs, output_dir=output, + consumer_network="executor_network", host_cert_dir=Path("/host/qdl/certs"), host_output_dir=Path("/host/qdl/candidate"), ) self.assertFalse(manifest["cutover_authorized"]) self.assertFalse(manifest["secret_values_recorded"]) self.assertEqual(manifest["authority_promotion_binding_count"], 12) + self.assertEqual(manifest["consumer_network"], "executor_network") self.assertEqual(len(manifest["authority_promotion_scope_digest"]), 64) production = json.loads( (output / "runtime/production-core-001.json").read_text() @@ -970,6 +997,7 @@ def test_candidate_bundle_uses_image_ids_and_never_records_secret_values(self): self.assertEqual((output / "stable.env").stat().st_mode & 0o777, 0o600) env_text = (output / "stable.env").read_text() self.assertIn("QDL_STABLE_CERT_DIR=/host/qdl/certs", env_text) + self.assertIn("QDL_STABLE_CONSUMER_NETWORK=executor_network", env_text) self.assertIn( "QDL_STABLE_RUNTIME_DIR=/host/qdl/candidate/runtime", env_text ) From a621e40a2f81187a367dbb0bb84ac842d34fdde5 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 16:07:30 +0000 Subject: [PATCH 20/21] docs(v2): freeze final consumer-ready release pair --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index ebab503..07ac963 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6438,6 +6438,21 @@ freshness/session/contract checks and the source-switch audit is durable. System remained unchanged. Commit and one final same-SHA image rebuild are required before PR/cutover. + +- `2026-08-20 C.2/C.3 FINAL RELEASE ARTIFACT PASS`: topology commit + `be35aa7389a37b31c21cc2689c25873dcfc7e73d` produced Python image + `sha256:89e359ecc731d68db7a1814885023e1ff9f0aea793e668b6298109eb463ff91c` + and Rust image + `sha256:ab57e015da2fb96ef6e4b2180676e0a41b2cc45b64080e820d6a8f29cdab180a`. + Machine-read OCI labels exactly match the Git SHA and version `2.0.0`; users + remain `qdl:qdl` and `10001:10001`. The final bundle manifest digest is + `6a3edff0fdaa690b1fc1237f5678bf8463355bbdc51afb59a018f3e629840425`, + binds `executor_network`, twelve Binance/OKX promotion bindings, zero DNSE, + `RUST_SHADOW` and `cutover_authorized=false`; complete authority Compose + rendering passes. One mistyped preflight revision image was detected by label + comparison and is explicitly not a release artifact or deployed runtime. + This is the only V2 pair eligible for the merge/cutover packet. + Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One operator packet may list the complete slice set, image IDs, old/new owners, From d55ec695e0d957296b508b09eee6b12248ac5f38 Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Thu, 20 Aug 2026 16:08:34 +0000 Subject: [PATCH 21/21] docs(v2): record final artifact cleanup --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 07ac963..32df41b 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -6452,6 +6452,9 @@ freshness/session/contract checks and the source-switch audit is durable. rendering passes. One mistyped preflight revision image was detected by label comparison and is explicitly not a release artifact or deployed runtime. This is the only V2 pair eligible for the merge/cutover packet. + Scoped cleanup then removed the superseded `3d3af1c` pair, the mistyped + Python tag and disposable netfix/test bundles. No broad prune, active + candidate image, final bundle, V1 image or Docker volume was removed. Promote all approved Binance and OKX feed slices in one maintenance window, but execute the CAS internally one slice at a time so a failure is isolated. One