feat: evo snapshot v3 — canonical bounded codec and context-free validation - #7592
feat: evo snapshot v3 — canonical bounded codec and context-free validation#7592PastaPastaPasta wants to merge 5 commits into
Conversation
3779689 to
3398d9a
Compare
…e validation First code PR of the assumeutxo M4 series (dashpay#7579 decomposition): the versioned interchange format for Dash's evo state alongside a UTXO snapshot - canonical serialization, DoS-bounded validating decode, and every validation invariant that needs no chain context. Chain-aware building/validation and dump/load integration follow in the next PRs of the series. Canonical ordering exists because snapshot content is hashed and cross-checked; per-object serializers are reused through a bounded stream wrapper, with bespoke code only at container level (ordering, bounds, per-entry budgets); decode-time checks deliberately stay out of the trusted hot EvoDB deserializers. AssumeutxoData gains the EvoSnapshotHash anchor the format is pinned by. Includes the aggregate rotation skip-list bound (lists accumulate across every quorum index and wrap the combined MN list), the CRangesSet bounded unserializer, and a vendored-immer shift-base ubsan suppression reachable only through the deliberately hash-colliding test fixtures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… sign change The mask for rejecting out-of-range trailing bits promotes through operator~ to a negative int before its implicit conversion back to uint8_t, which clang's implicit-integer-sign-change check reports for every bitset whose size is not a multiple of eight. The evo snapshot unit tests are the first to deserialize such bitsets under the sanitizer job. Same bits, stated explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The codec PR shipped UnserializeBounded without its unit coverage; add the malformed/canonical decode matrix and the round-trip checks from the original series. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d97a8d6 to
5fedfc9
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 34 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (15)
Comment |
|
⛔ Blockers found — Opus deferred (commit 5fedfc9) |
Potential PR merge conflictsThis is advisory only. It does not block CI, but it marks PRs that will likely need a rebase depending on merge order. If this PR merges firstThese open PRs will likely need a rebase:
|
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Preliminary review — Codex only
The bounded codec is well tested, but three in-scope correctness gaps remain: MNHF input ordering is not canonicalized at the trust boundary, context-free MNHF and credit-pool invariants are omitted, and valid commitments for supported overridden LLMQ parameters cannot be decoded. These issues prevent the v3 format from meeting its stated canonical and chain-configuration requirements.
Source: reviewer backend gpt-5.6-sol; final verifier backend gpt-5.6-sol. openclaw-agent/cliproxy/gpt-5.6-sol is orchestration-only and not reviewer evidence.
Validated blockers were found in the Codex precheck. Opus is deferred until a fresh Codex revalidation clears the blocker gate.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet: not run (deferred by blocker gate)
🔴 3 blocking
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `src/evo/snapshot.h`:
- [BLOCKING] src/evo/snapshot.h:567-572: Reject noncanonical MNHF signal ordering
The decoder inserts signals directly into a `std::map`, which normalizes their order and rejects only duplicate keys. The same signal set can therefore be supplied in any permutation, accepted by `Unserialize()`, and reserialized in sorted order. This conflicts with the decoder's `require_canonical_order` contract and with the strict ordering checks applied to the other top-level collections, allowing multiple accepted wire representations for the same snapshot.
- [BLOCKING] src/evo/snapshot.h:488-499: Do not validate commitment sizes against static default LLMQ parameters
`SnapshotLLMQParams()` obtains the compile-time entry from `Consensus::available_llmqs`, and `ReadMinedQuorumCommitment()` requires both commitment bitsets to have exactly that entry's default `size`. However, `-llmqtestparams`, the related regtest overrides, and `-llmqdevnetparams` modify the effective size stored in `Params().GetLLMQ(type)`, which consensus commitment validation uses. A commitment produced under any nondefault supported size is therefore rejected by this decoder: a larger size exceeds the read bound, while a smaller size fails the exact-size comparison. Since this layer is intentionally context-free, it should enforce a format-level maximum and internal bitset consistency, then leave exact sizing to the later chain-aware validation using the effective chain parameters.
In `src/evo/snapshot.cpp`:
- [BLOCKING] src/evo/snapshot.cpp:193-204: Validate credit-pool amounts and MNHF signal values
`Validate()` bounds only the MNHF map's cardinality and performs no semantic validation of the credit-pool amounts. A snapshot with a signal such as `(255, -1)`, a signal height above the snapshot height, or negative/out-of-range `locked`, `currentLimit`, and `latelyUnlocked` values currently passes validation and can receive a canonical snapshot hash. Consensus-produced signals always use bits below `VERSIONBITS_NUM_BITS` and heights between zero and the snapshot height. Credit-pool construction produces money-range nonnegative amounts with `currentLimit <= locked`; enforcing those properties here also prevents an untrusted seeded value from entering later signed credit-pool arithmetic.
| const size_t signal_count{ReadBoundedCompactSize(s, Consensus::MAX_VERSION_BITS_DEPLOYMENTS, "MNHF signals")}; | ||
| for (size_t i{0}; i < signal_count; ++i) { | ||
| std::pair<uint8_t, int> signal; | ||
| s >> signal; | ||
| if (!mnhf_signals.emplace(signal).second) throw std::ios_base::failure("duplicate MNHF signal bit"); | ||
| } |
There was a problem hiding this comment.
🔴 Blocking: Reject noncanonical MNHF signal ordering
The decoder inserts signals directly into a std::map, which normalizes their order and rejects only duplicate keys. The same signal set can therefore be supplied in any permutation, accepted by Unserialize(), and reserialized in sorted order. This conflicts with the decoder's require_canonical_order contract and with the strict ordering checks applied to the other top-level collections, allowing multiple accepted wire representations for the same snapshot.
| const size_t signal_count{ReadBoundedCompactSize(s, Consensus::MAX_VERSION_BITS_DEPLOYMENTS, "MNHF signals")}; | |
| for (size_t i{0}; i < signal_count; ++i) { | |
| std::pair<uint8_t, int> signal; | |
| s >> signal; | |
| if (!mnhf_signals.emplace(signal).second) throw std::ios_base::failure("duplicate MNHF signal bit"); | |
| } | |
| const size_t signal_count{ReadBoundedCompactSize(s, Consensus::MAX_VERSION_BITS_DEPLOYMENTS, "MNHF signals")}; | |
| uint8_t previous_signal_bit{0}; | |
| bool have_previous_signal{false}; | |
| for (size_t i{0}; i < signal_count; ++i) { | |
| std::pair<uint8_t, int> signal; | |
| s >> signal; | |
| if ((have_previous_signal && signal.first <= previous_signal_bit) || | |
| !mnhf_signals.emplace(signal).second) { | |
| throw std::ios_base::failure("noncanonical MNHF signal order"); | |
| } | |
| previous_signal_bit = signal.first; | |
| have_previous_signal = true; | |
| } |
source: ['codex']
| void CEvoSnapshot::Validate(bool require_canonical_order) const | ||
| { | ||
| if (version != EVO_SNAPSHOT_VERSION) throw std::ios_base::failure("unsupported evo snapshot version"); | ||
| if (base_block_hash.IsNull() || mn_list.GetBlockHash() != base_block_hash) { | ||
| throw std::ios_base::failure("evo snapshot base block mismatch"); | ||
| } | ||
| ValidateCanonicalMNInvariants(mn_list); | ||
| if (quorums.size() > Consensus::available_llmqs.size() || | ||
| historical_mn_list_diffs.size() > EvoSnapshotMaxHistoricalMNLists() || | ||
| quorum_modifiers.size() > EVO_SNAPSHOT_MAX_MODIFIERS || | ||
| mnhf_signals.size() > Consensus::MAX_VERSION_BITS_DEPLOYMENTS) { | ||
| throw std::ios_base::failure("oversized evo snapshot collection"); |
There was a problem hiding this comment.
🔴 Blocking: Validate credit-pool amounts and MNHF signal values
Validate() bounds only the MNHF map's cardinality and performs no semantic validation of the credit-pool amounts. A snapshot with a signal such as (255, -1), a signal height above the snapshot height, or negative/out-of-range locked, currentLimit, and latelyUnlocked values currently passes validation and can receive a canonical snapshot hash. Consensus-produced signals always use bits below VERSIONBITS_NUM_BITS and heights between zero and the snapshot height. Credit-pool construction produces money-range nonnegative amounts with currentLimit <= locked; enforcing those properties here also prevents an untrusted seeded value from entering later signed credit-pool arithmetic.
| void CEvoSnapshot::Validate(bool require_canonical_order) const | |
| { | |
| if (version != EVO_SNAPSHOT_VERSION) throw std::ios_base::failure("unsupported evo snapshot version"); | |
| if (base_block_hash.IsNull() || mn_list.GetBlockHash() != base_block_hash) { | |
| throw std::ios_base::failure("evo snapshot base block mismatch"); | |
| } | |
| ValidateCanonicalMNInvariants(mn_list); | |
| if (quorums.size() > Consensus::available_llmqs.size() || | |
| historical_mn_list_diffs.size() > EvoSnapshotMaxHistoricalMNLists() || | |
| quorum_modifiers.size() > EVO_SNAPSHOT_MAX_MODIFIERS || | |
| mnhf_signals.size() > Consensus::MAX_VERSION_BITS_DEPLOYMENTS) { | |
| throw std::ios_base::failure("oversized evo snapshot collection"); | |
| ValidateCanonicalMNInvariants(mn_list); | |
| if (!MoneyRange(credit_pool.locked) || | |
| !MoneyRange(credit_pool.currentLimit) || | |
| !MoneyRange(credit_pool.latelyUnlocked) || | |
| credit_pool.currentLimit > credit_pool.locked) { | |
| throw std::ios_base::failure("invalid evo snapshot credit pool"); | |
| } | |
| const int snapshot_height{mn_list.GetHeightForSnapshotCodec()}; | |
| for (const auto& [bit, height] : mnhf_signals) { | |
| if (bit >= VERSIONBITS_NUM_BITS || height < 0 || height > snapshot_height) { | |
| throw std::ios_base::failure("invalid evo snapshot MNHF signal"); | |
| } | |
| } | |
| if (quorums.size() > Consensus::available_llmqs.size() || | |
| historical_mn_list_diffs.size() > EvoSnapshotMaxHistoricalMNLists() || | |
| quorum_modifiers.size() > EVO_SNAPSHOT_MAX_MODIFIERS || | |
| mnhf_signals.size() > Consensus::MAX_VERSION_BITS_DEPLOYMENTS) { | |
| throw std::ios_base::failure("oversized evo snapshot collection"); | |
| } |
source: ['codex']
| const auto& params{SnapshotLLMQParams(llmq_type)}; | ||
| const size_t total_count{SnapshotCommitmentCount(params, rotation_enabled)}; | ||
| const size_t expected_active{static_cast<size_t>(params.signingActiveQuorumCount)}; | ||
| const size_t active_count{ReadBoundedCompactSize(s, expected_active, "active commitments")}; | ||
| active_commitments.reserve(active_count); | ||
| for (size_t i{0}; i < active_count; ++i) { | ||
| active_commitments.emplace_back(ReadMinedQuorumCommitment(s, params)); | ||
| } | ||
| const size_t safety_count{ReadBoundedCompactSize(s, total_count - expected_active, "safety commitments")}; | ||
| safety_commitments.reserve(safety_count); | ||
| for (size_t i{0}; i < safety_count; ++i) { | ||
| safety_commitments.emplace_back(ReadMinedQuorumCommitment(s, params)); |
There was a problem hiding this comment.
🔴 Blocking: Do not validate commitment sizes against static default LLMQ parameters
SnapshotLLMQParams() obtains the compile-time entry from Consensus::available_llmqs, and ReadMinedQuorumCommitment() requires both commitment bitsets to have exactly that entry's default size. However, -llmqtestparams, the related regtest overrides, and -llmqdevnetparams modify the effective size stored in Params().GetLLMQ(type), which consensus commitment validation uses. A commitment produced under any nondefault supported size is therefore rejected by this decoder: a larger size exceeds the read bound, while a smaller size fails the exact-size comparison. Since this layer is intentionally context-free, it should enforce a format-level maximum and internal bitset consistency, then leave exact sizing to the later chain-aware validation using the effective chain parameters.
source: ['codex']
2 participants
Issue being fixed or feature implemented
Part of the AssumeUTXO M4 decomposition (#7579, now draft — see the series map there). For a Dash node, a UTXO snapshot alone is not enough to operate at the base block: the node also needs the deterministic MN list, quorum commitments, rotation state, credit pool, and MNHF signals that consensus at that height depends on. This PR defines the evo snapshot v3 format: the versioned interchange encoding for that state, its DoS-hardened decoder, and every validation invariant that needs no chain context. It deliberately contains no chain access and no lifecycle wiring — building a snapshot from chain state and validating one against the chain come in the next PR of the series;
dumptxoutset/load integration after that. Reviewing this PR is reviewing the wire format and its trust boundary, nothing else.What was done?
src/evo/snapshot.{h,cpp}: theCEvoSnapshottypes, canonical serialization, bounded validating deserialization, andCEvoSnapshot::Validate()(context-free invariants), plusReconstructHistoricalMNLists(),CanonicalMNListHash(),GetEvoSnapshotHash(), andVerifyEvoSnapshotCbTx()(pure CbTx cross-checks over decoded content).AssumeutxoDatagains anEvoSnapshotHashfield: the hard-coded expected hash of the canonical evo section, the same security anchor rolehash_serializedplays for the UTXO set.CDeterministicMNListgainsApplyDiffForSnapshot()andGetHeightForSnapshotCodec();CRangesSetgains a bounded validating unserializer;OverrideStreamgainsGetStream()for the per-object decode budgets.shift-base), reachable only through the unit tests' deliberately hash-colliding MN fixtures, in the style of the existing vendored-library entries.ReadFixedBitSetwhose trailing-bits mask otherwise trips clang's implicit-sign-change check for any bitset size not a multiple of eight — these tests are the first to decode such bitsets under the sanitizer job.Why a bespoke codec instead of the classes' own serializers (raised in #7579 review): (1) snapshot content is hashed and cross-checked (the completion-time MN-list comparison and CbTx checks), so the encoding must be a pure function of set content — hence canonical proTxHash ordering rather than container iteration order; (2) the snapshot file is untrusted by definition and read once, so its decoder validates and bounds everything, while the EvoDB/P2P deserializers are trusted hot paths that would pay that tax per block; (3) a versioned interchange format must not silently drift when in-memory serialization changes. Note per-object serializers are reused —
CDeterministicMN, commitments, and the credit pool decode through their ownSERIALIZE_METHODSwrapped in a budgeted stream; only the container level (ordering, bounds, budgets) is bespoke.Open question for reviewers (from #7579 feedback on header surface): the codec helpers are
Stream-templated and therefore header-bound; I can move them into anevo::detailnamespace to shrink the nominal API if preferred — say the word and it's a small mechanical commit.How Has This Been Tested?
Full unit suite on a
--enable-werrorbuild, plus the snapshot/netinfo/util suites under a--with-sanitizers=undefined,integerbuild with the repo's ubsan suppressions (which is what surfaced theReadFixedBitSetand immer items above).Breaking Changes
None. The format is new and nothing constructs or consumes it on-chain yet; the
AssumeutxoDatafield is populated with a null placeholder for the existing regtest entries.Checklist: