[codex] add evm-only executor load test harness - #3658
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## codex/sei-v3-evm-only-scaffold #3658 +/- ##
===================================================================
- Coverage 82.29% 66.16% -16.14%
===================================================================
Files 14 27 +13
Lines 2446 5793 +3347
===================================================================
+ Hits 2013 3833 +1820
- Misses 295 1448 +1153
- Partials 138 512 +374
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
bd57e4c to
7c6c44c
Compare
8e28929 to
4ec8da5
Compare
7c6c44c to
6f9547a
Compare
8cc2094 to
a65459e
Compare
6f9547a to
31f16a8
Compare
fc79ac5 to
e312a84
Compare
31f16a8 to
9cb00d6
Compare
e312a84 to
63b8662
Compare
9cb00d6 to
35864cf
Compare
63b8662 to
6a6c193
Compare
35864cf to
3eaa2ab
Compare
6a6c193 to
ef82dfd
Compare
3eaa2ab to
54456cf
Compare
ef82dfd to
2d0570c
Compare
54456cf to
6037fbb
Compare
2d0570c to
fa7c755
Compare
6037fbb to
f72a746
Compare
fa7c755 to
a9debf4
Compare
PR SummaryHigh Risk Overview Custom precompiles are no longer fail-closed placeholders when a registry entry provides a The first full module is an SDK-free staking precompile ( Reviewed by Cursor Bugbot for commit e7fc118. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
Adds a well-structured standalone evmonly-loadtest harness (prepare/execute pipeline, result sinks, metrics) with good test coverage. Two robustness issues surfaced by Codex are confirmed but both are confined to this dev/benchmark tool, so neither blocks merge.
Findings: 0 blocking | 5 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
Non-blocking
cursor-review.mdis empty (Cursor produced no output) andREVIEW_GUIDELINES.mdis empty/missing, so this review proceeded without repo-specific guidelines or a Cursor second opinion.- No direct test exercises the streaming (
runStreaming) path or the enqueue-under-writer-failure path; the confirmed sink race would be a good target for a focused test. - Minor: in
resultSinks.StoreBlockResultthe fallback (non-BlockResultSink) path does not callrelease()ifStoreChangeSet/StoreReceiptserrors, leaking the pooled result. Harmless today because the discard sink never errors, but worth guarding with a deferred release. - 2 suggestion(s)/nit(s) flagged inline on specific lines.
| return err | ||
| } | ||
| select { | ||
| case s.records <- record: |
There was a problem hiding this comment.
[suggestion] Race between this non-blocking send and the writer's failure path. enqueue checks getErr() above, then sends here; meanwhile run() can fail a write, call setErr + releaseQueuedResults, and return. A record that lands in the buffer after the drain but after run() has exited is never written and never released, leaking a pooled BlockResult slot (Codex's high finding — confirmed). Consider selecting on s.done in this first (and the buffered) send, or re-checking getErr()/draining, so a record enqueued after the writer stops is released. In practice the next StoreBlockResult returns the stored error and execution unwinds, so a full hang is unlikely, but the leak/latent race is real.
| } | ||
|
|
||
| func newAppendRLPFile(path string, bufferSize int, syncOnWrite bool) (*appendRLPFile, error) { | ||
| file, err := os.OpenFile(path, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) |
There was a problem hiding this comment.
[suggestion] O_CREATE|O_TRUNC silently truncates pre-existing changesets.rlp/receipts.rlp, and cleanup later os.Removes them — so pointing --persist-dir at a directory containing prior output destroys those files (Codex's medium finding — confirmed). Since this is a load-test tool and the docs already recommend a fresh mktemp -d, this is acceptable, but consider O_EXCL (or rejecting a dir that already contains these files) to avoid clobbering user data by accident.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| func createProfileFile(path string) (*os.File, error) { | ||
| if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { | ||
| return nil, fmt.Errorf("create profile dir for %s: %w", path, err) | ||
| } | ||
| file, err := os.Create(path) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("create profile file %s: %w", path, err) | ||
| } | ||
| return file, nil | ||
| } |
There was a problem hiding this comment.
🔴 The new giga/evmonly/cmd/evmonly-loadtest/main.go fails 'make lint' (golangci-lint v2.8.0, errcheck + gosec) with 15 issues, which AGENTS.md requires to pass and which CI's golangci-lint workflow enforces repo-wide.
Extended reasoning...
AGENTS.md is explicit: make lint runs golangci-lint run (v2.8.0) with errcheck, gosec, govet, staticcheck, and others enabled, and .github/workflows/golangci.yml runs this on every pull request with no path filter and no --new-from-rev/only-new-issues restriction. That means every violation in a new file introduced by this PR fails the same CI job that gates merge, regardless of whether the violation predates this PR (it cannot, since the file is 100% new).
Running golangci-lint run ./giga/evmonly/cmd/evmonly-loadtest/... against this repos own .golangci.yml reproduces exactly 15 issues, all inside main.go:
- errcheck (1):
defer file.Close()at line 464 insidewriteHeapProfile— the deferred close error is silently discarded. - gosec G301 (2):
os.MkdirAll(..., 0o755)at line 421 (createProfileFile) and line 1305 (newFileResultSinks) — gosec wants directory permissions of0o750or less;0o755is world-readable/executable. - gosec G304 (2):
os.Create(path)at line 424 andos.OpenFile(path, ...)at line 1573 — both open a file at a path built from a user-supplied CLI flag (--cpu-profile/--heap-profile/--trace-profile/--persist-dir), which gosec flags as a potential file-inclusion vector since the path is not validated/sanitized before use. - gosec G115 (10): unchecked int/uint64 conversions that could silently wrap on unusual inputs —
maxInt()/prebuildBlockRequestscastingcfg.blocksbetweenuint64andint(lines 677, 680), the Unix-time-to-uint64block timestamp (line 1037), convertinginttx counts touint64metric counters (lines 1864, 1876), convertingtime.Duration.Nanoseconds() int64touint64(lines 1935, 1943), and converting storeduint64nanosecond counters back totime.Duration(int64) for display (lines 2086, 2088, 2106).
None of these are covered by the repos two gosec exclusions in .golangci.yml (.*\.pb\.go$ and the "weak random number generator" text match), and run.tests: false only skips *_test.go, not main.go. I reproduced this directly with golangci-lint run ./giga/evmonly/cmd/evmonly-loadtest/... (v2.5.0 locally, close enough to CIs v2.8.0 that the same stable errcheck/gosec rules apply) and got exit code 1 with exactly these 15 findings (1 errcheck + 14 gosec), matching the report line-for-line.
Step-by-step proof: (1) Check .golangci.yml: linters.enable includes errcheck and gosec; the only exclusions.rules entry is for the gosec weak-RNG message text, and the only excluded path is .*\.pb\.go$. main.go is neither excluded nor a test file. (2) Run golangci-lint run ./giga/evmonly/cmd/evmonly-loadtest/... from the repo root — this uses the exact .golangci.yml that .github/workflows/golangci.yml uses for the CI job named in AGENTS.md as required (make lint). (3) Output: 15 issues, categorized errcheck: 1, gosec: 14, at the exact line numbers listed above. (4) Because main.go is entirely new in this PR (it does not exist on the base branch), all 15 are new violations introduced by this PR, not preexisting debt being surfaced. (5) Since the CI workflow has no diff-scoping (no only-new-issues: true, no revision comparison), this job would fail (exit 1) on this PR as-is, blocking merge per the required-checks convention implied by AGENTS.md.
Fix: check the deferred file.Close() error (e.g., wrap it into the returned error via a named return, matching the pattern already used elsewhere in this file such as profileSession.Close); tighten the two MkdirAll calls to 0o750; either validate/allowlist the profile and persist-dir paths before opening them or add a narrowly-scoped //nolint:gosec with a one-line justification (these are trusted CLI-flag paths for a local load-test tool, not attacker-controlled input); and add explicit bounds checks or //nolint:gosec comments for the ten G115 int/uint64 conversions, since the flagged conversions are either already logically infeasible to overflow on 64-bit platforms or worth an explicit bound (e.g. cfg.blocks vs maxInt()).
There was a problem hiding this comment.
Re-ran golangci-lint v2.5.0 against ./giga/evmonly/cmd/evmonly-loadtest/... on the current HEAD and still get exactly 15 issues (1 errcheck + 14 gosec): the deferred file.Close() in writeHeapProfile is still unchecked, both os.MkdirAll calls (createProfileFile, newFileResultSinks) still use 0o755, os.Create/os.OpenFile on the CLI-flag-derived profile/persist paths are still flagged G304 (plus a new G302 on the 0o644 file mode), and all ten G115 int/uint64 conversions remain. There are zero nolint comments anywhere in main.go, so none of the suggested fixes (check the Close error, tighten permissions to 0750/0600, add narrowly-scoped nolint justifications or bounds checks) were applied. This still fails make lint / the CI golangci-lint workflow.
There was a problem hiding this comment.
Still unresolved as of the latest commit (6eb0196). Re-ran golangci-lint run ./giga/evmonly/cmd/evmonly-loadtest/... against current HEAD and get the identical 15 issues (1 errcheck + 14 gosec) at the same locations reported originally and in the prior follow-up: the deferred file.Close() in writeHeapProfile (line 510) is still unchecked, both os.MkdirAll calls (line 467 createProfileFile, line 1486 newFileResultSinks) are still 0o755, os.Create/os.OpenFile on the CLI-flag-derived profile/persist paths are still flagged G304 (plus G302 on the file's 0o644 mode), and all ten G115 int/uint64 conversions remain unaddressed. The two nolint:gosec comments present in the file (lines 312, 950) predate this PR's diff and cover unrelated conversions, not any of the flagged G115 sites. This still fails make lint / the CI golangci-lint workflow.
There was a problem hiding this comment.
Still unresolved on the latest commit (e7fc118). Re-ran golangci-lint run ./giga/evmonly/cmd/evmonly-loadtest/... against current HEAD and get the identical 15 issues (1 errcheck + 14 gosec) at the same locations: the deferred file.Close() in writeHeapProfile (line 510) is still unchecked, both os.MkdirAll calls (line 467 createProfileFile, line 1486 newFileResultSinks) are still 0o755, os.Create/os.OpenFile on the CLI-flag-derived profile/persist paths are still flagged G304 (plus G302 on the file'''s 0o644 mode), and all ten G115 int/uint64 conversions remain unaddressed. This still fails make lint / the CI golangci-lint workflow.
| erc20TransferSelector = [4]byte{0xa9, 0x05, 0x9c, 0xbb} | ||
| // Minimal ERC20-like runtime for transfer(address,uint256), with balances at | ||
| // storage slot 0 and a standard Transfer(address,address,uint256) log. | ||
| erc20TransferRuntimeCode = common.FromHex("0x60003560e01c63a9059cbb1460145760006000fd5b60243560043533600052600060205260406000208054831060805780548303905580600052600060205260406000208054830190558160005280337fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef60206000a3600160005260206000f35b60006000fd") |
There was a problem hiding this comment.
🟡 The generated ERC20 fixture bytecode (erc20TransferRuntimeCode) has its balance-check and subtraction operand order inverted: the LT guard reverts when amount < senderBalance (i.e. sender has more than enough) instead of reverting on insufficient balance, and the SUB computes amount - senderBalance instead of senderBalance - amount. This is currently masked because erc20TransferWorkload.buildBlock always sets the sender's balance equal to the transfer amount, so the two inversions cancel out; with any other balance the fixture would incorrectly revert or silently underflow, and it can never catch a genuine insufficient-balance case.
Extended reasoning...
I independently disassembled erc20TransferRuntimeCode (offsets in parens are hex) and traced the EVM stack from the transfer(address,uint256) entrypoint at JUMPDEST 0x14:
21 PUSH1 0x24 / CALLDATALOAD -> amount
24 PUSH1 0x04 / CALLDATALOAD -> recipientWord
27 CALLER -> sender
... MSTORE/MSTORE/SHA3 -> senderSlot = keccak256(sender . 0)
41 DUP1 / SLOAD -> senderBalance
43 DUP4 -> dup amount (4th from top)
44 LT -> amount < senderBalance
45 PUSH1 0x80 / JUMPI -> jump to REVERT (offset 0x80) if amount < senderBalance
48 DUP1 / SLOAD -> senderBalance (again)
50 DUP4 -> dup amount
51 SUB -> amount - senderBalance
52 SWAP1 / 53 SSTORE -> senderSlot = amount - senderBalance
Both operations are backwards relative to a correct ERC20 transfer:
- The guard reverts on the wrong condition. It jumps to
REVERTwhenamount < senderBalance— i.e. precisely when the sender has more than enough funds — and falls through (no revert) wheneveramount >= senderBalance, which includes the genuine insufficient-balance case. A correct implementation should revert whensenderBalance < amount. - The debit computes the wrong subtraction.
SUBpops top=amount, second=senderBalanceand computesamount - senderBalance, then stores that as the sender's new balance, instead ofsenderBalance - amount.
The recipient-credit side (SHA3/SLOAD/DUP4/ADD at offsets 67-73) is correct: it computes recipientBalance + amount and stores it.
Both sender-side inversions happen to cancel out in exactly one case: senderBalance == amount. That is precisely what erc20TransferWorkload.buildBlock (main.go) sets up for every generated transaction — it calls state.SetState(w.cfg.erc20Contract, erc20BalanceSlot(sender), common.BigToHash(w.cfg.transferValue)) (main.go:978) and then signs a transfer of w.cfg.transferValue (main.go:1000/921). With senderBalance == amount: the LT guard evaluates amount < amount = false, so it never reverts; and SUB evaluates amount - amount = 0, which happens to equal the correct result (senderBalance - amount = 0 too) — so the existing test (TestERC20TransferWorkloadExecutesAgainstEVMOnlyExecutor) and the EC2 throughput benchmarks in the PR pass by construction, without ever exercising the inverted logic in a way that produces a visibly wrong result.
Concrete proof of the defect with a balance different from the transfer amount, e.g. senderBalance = 100, amount = 30 (the normal, expected real-world case of balance > amount):
- Guard:
amount < senderBalance→30 < 100→ true → jumps toREVERT. A transfer that should succeed (sender has plenty of funds) is wrongly rejected. - If instead
senderBalance = 10,amount = 30(a genuine insufficient-balance case):amount < senderBalance→30 < 10→ false → falls through, no revert.SUBcomputesamount - senderBalance = 30 - 10 = 20and stores20as the sender's new balance — i.e. the contract would let the transfer succeed and give the sender a positive balance out of insufficient funds, instead of reverting on underflow.
Since giga/evmonly/cmd/evmonly-loadtest/README.md explicitly describes this file as a reusable fixture ("Future workloads should add another workload builder... then reuse the same block producer/prepare/executor/metrics pipeline"), this latent inversion would silently break correctness the moment any future workload or config varies balance vs. transfer amount (e.g. sweeping --sender-balance-wei independently of --transfer-value-wei, or a new workload built on top of this contract).
Fix: swap the LT operand order (compare senderBalance < amount to guard correctly) and swap the SUB operand order (senderBalance - amount) so the sender-side debit matches the already-correct recipient-side credit logic.
Severity: this is a synthetic fixture inside a standalone load-test tool (not production/consensus code), and it behaves correctly for its one current caller because balance is always forced equal to the transfer amount, so nothing breaks in this PR as merged. It is a real defect worth a one-line fix but shouldn't block merge.
| func (s *asyncFileResultSinks) run() { | ||
| defer close(s.done) | ||
| for record := range s.records { | ||
| if s.metrics != nil { | ||
| s.metrics.setSinkQueued(len(s.records)) | ||
| } | ||
| if err := s.writeRecord(record); err != nil { | ||
| record.releaseResult() | ||
| s.setErr(err) | ||
| s.releaseQueuedResults() | ||
| return | ||
| } | ||
| record.releaseResult() | ||
| if s.metrics != nil { | ||
| s.metrics.setSinkQueued(len(s.records)) | ||
| } | ||
| } | ||
| if s.metrics != nil { | ||
| s.metrics.setSinkQueued(0) | ||
| } | ||
| } | ||
|
|
||
| func (s *asyncFileResultSinks) releaseQueuedResults() { | ||
| for { | ||
| select { | ||
| case record, ok := <-s.records: | ||
| if !ok { | ||
| return | ||
| } | ||
| record.releaseResult() |
There was a problem hiding this comment.
🟡 In asyncFileResultSinks.run(), after a write error the goroutine drains s.records exactly once via a non-blocking releaseQueuedResults() and then returns, but never closes s.records. Since enqueue() can still pass its getErr()==nil check and successfully send into the still-open buffered channel concurrently with (or just after) that single drain pass, a racing record can be enqueued with no goroutine left to receive it — its BlockResult release()/refcount is never invoked and its changeset/receipt payload is silently never written, even though enqueue() returns nil to the caller. This requires --workers>=2 plus a real write I/O error to trigger.
Extended reasoning...
The bug. asyncFileResultSinks.run() (main.go, around lines 1486-1515) is the sole consumer of the buffered s.records channel. On a writeRecord failure it does:
if err := s.writeRecord(record); err != nil {
record.releaseResult()
s.setErr(err)
s.releaseQueuedResults() // single non-blocking drain pass
return // s.records is NEVER closed here
}releaseQueuedResults() is a select { case <-s.records: ...; default: return } loop — it drains whatever is already buffered at that instant and then returns as soon as the channel is momentarily empty. After that, run() returns and defer close(s.done) fires. Critically, s.records itself is left open; only Close() (called much later, from the main shutdown path) ever does close(s.records).
The race. enqueue() is:
func (s *asyncFileResultSinks) enqueue(ctx context.Context, record resultSinkRecord) error {
if err := s.getErr(); err != nil {
return err
}
select {
case s.records <- record: // non-blocking fast path
...
return nil
default:
}
// slow path: blocking select among records<-, done, ctx.Done()
...
}There is no synchronization between a producer's getErr() check and the consumer's setErr()+drain+return sequence. A concurrent StoreBlockResult/enqueue call (from a different executor worker, hence --workers>=2) can read getErr()==nil a moment before setErr() runs, then successfully complete s.records <- record into the buffered channel's free capacity — a buffered send needs no live receiver — after releaseQueuedResults() has already made its one pass and run() has exited. The slow-path select has the same exposure: Go picks pseudo-randomly among ready cases, so case s.records <- record can win over case <-s.done even though done is also ready.
Why nothing catches it. Once that record lands in the channel, there is no longer a live goroutine reading from s.records, and it is never drained until Close() calls close(s.records) — closing a channel does not deliver buffered-but-unread values to anyone; they're simply discarded when the channel becomes unreachable. So the record's release closure (which holds a retained reference/refcount on a pooled *evmonly.BlockResult) is never invoked, and its changeset/receipt payload is never written to disk — yet enqueue() already returned nil, i.e., success, to the calling executor worker.
Step-by-step proof:
--workers=2,--result-sink=file. Worker A callsStoreBlockResultfor height N; worker B calls it for height N+1 concurrently.- Both go through
resultSinks.StoreBlockResult→asyncFileResultSinks.enqueue. Worker A's record for N is already being processed insiderun()'s loop iteration;writeRecordfor N hits a real I/O error (e.g., disk full, aWrite/Flushfailure on the underlyingos.File). - Worker B's
enqueuecall for N+1 executess.getErr()and observesnil(the error hasn't been set yet). run()proceeds:record.releaseResult()for N,s.setErr(err), thens.releaseQueuedResults()— at this exact instant, worker B's send for N+1 has not yet happened, so the drain'sdefaultcase fires immediately and it returns.run()returns;s.donecloses.- Worker B's
select { case s.records <- record: ...; default: }inenqueuenow executes and succeeds (the buffer has free capacity), soenqueuereturnsnilto worker B. - Record N+1 sits in the buffered channel forever. No goroutine ever reads it again. Its
release()is never called (permanently retaining a pooledBlockResultslot) and its RLP changeset/receipt bytes are never written tochangesets.rlp/receipts.rlp.
Fix. Have run() drain and release all remaining buffered records only after the channel can no longer receive new sends — e.g., have run() (or Close()) close s.records before/as part of the error path so enqueue's sends fail fast via a closed-channel panic-avoidance path, or better, have enqueue() re-check getErr() immediately before/after the send and, on failure, itself call record.releaseResult() for anything it can no longer guarantee will be drained. The simplest robust fix is to make s.done the single source of truth: close s.records from run() itself right before returning on error (guarded so Close()'s later close(s.records) doesn't double-close), so any in-flight enqueue send either lands before the close (and gets drained by a final full drain loop) or observes the closed channel and returns the stored error instead of silently succeeding.
On severity. All three independent verifiers confirm the race is real but converge on nit, and I agree: this is confined to the standalone giga/evmonly/cmd/evmonly-loadtest benchmarking tool, not any consensus/production execution path. It requires a genuine file-write I/O error (rare) racing with a specific timing window, and only manifests with --workers>=2 (default is 1). The claimed "stalls the whole executor pipeline" consequence is overstated — once setErr runs, subsequent enqueue calls fail fast, and the errgroup shuts the pipeline down almost immediately, so at most a handful of records leak rather than an unbounded stall. The dropped changeset/receipt data is also moot in practice since fileResultSinks.Cleanup() removes the persist files on any error or cancellation anyway. The residual real defect is the leaked pooled BlockResult refcount on the error path, which is a minor tooling correctness issue worth fixing but not a merge blocker.
| orderErr := forwardPreparedBlocksInOrder(ctx, unordered, out) | ||
| workerErr := <-done | ||
| if orderErr != nil { | ||
| return orderErr | ||
| } | ||
| return workerErr |
There was a problem hiding this comment.
🟡 In prepareBlocks(), forwardPreparedBlocksInOrder is called with the outer ctx instead of the inner groupCtx that prepare workers run under, so when a worker returns a descriptive error (e.g. a rejected blob tx or a parse failure), the ordering helper only sees the still-live outer ctx and returns a generic "prepared block stream closed before block %d" error instead of the real one. That real error is then discarded in prepareBlocks, so the operator loses the actual root cause of the failure and only sees an uninformative ordering message.
Extended reasoning...
In prepareBlocks() (giga/evmonly/cmd/evmonly-loadtest/main.go:767-772), an inner errgroup.WithContext(ctx) creates groupCtx, and all prepare workers run under that inner context. When a worker fails — for example PrepareBlock rejecting a blob transaction via validateSupportedTx, or a raw tx failing to parse — it returns a descriptive error like "prepare worker 3 prepare block 5: parse tx 2: blob transactions require block-level blob gas accounting". That error cancels groupCtx (not the outer ctx) and, once all workers exit, the helper goroutine closes unordered and stores the real error in workerErr.
However, forwardPreparedBlocksInOrder is invoked with the outer ctx, which is unaffected by the inner cancellation (prepareBlocks has not returned yet). When unordered closes with a gap — i.e. pending holds later blocks that were already prepared and buffered before the failure, but the failed block number never arrived — the drain loop checks ctx.Err() on this still-live outer context, gets nil, and returns the generic fmt.Errorf("prepared block stream closed before block %d", next). Back in prepareBlocks, orderErr != nil takes priority and is returned instead of workerErr, so the real, actionable error message is discarded.
Concrete walkthrough: Suppose prepare-workers=4 and worker 3 is midway through block 5 while workers 0-2 have already produced blocks 1, 2, 4, and 6 (buffered in pending since block 3 has not yet completed). Worker 3 hits a blob tx in block 5 and returns errUnsupportedBlobTx wrapped with context. This cancels groupCtx; the other workers stop; group.Wait() returns the wrapped blob-tx error as workerErr; unordered closes. Meanwhile forwardPreparedBlocksInOrder has already forwarded block 1 and 2, and is waiting at next=3. It sees the channel close, drains pending, finds no entry for next=3 (block 5 never arrived because the worker preparing it failed), checks ctx.Err() on the outer, still-live context (nil), and returns "prepared block stream closed before block 3". prepareBlocks then returns this generic message, and the actual blob transactions require block-level blob gas accounting cause — the very failure mode this PR introduces — never reaches the operator.
This is a real defect in newly added code, but its blast radius is limited: evmonly-loadtest is a standalone benchmarking harness (giga/evmonly/cmd/evmonly-loadtest), not production or consensus-path code, and no incorrect output or silent failure occurs — an error is always still surfaced to the caller, just with a less useful message. The masking also only ever coincides with a genuine workerErr being present (on plain outer-context cancellation, ctx.Err() != nil short-circuits to nil; on normal completion pending is empty), so this cannot produce a false-positive failure — it strictly degrades diagnostics on real failures.
Fix: pass groupCtx (or check workerErr via the done channel) instead of the outer ctx when calling forwardPreparedBlocksInOrder, or simply prefer workerErr over orderErr when both are present, since workerErr carries the actionable root cause.
83967c0 to
205f54c
Compare
There was a problem hiding this comment.
Beyond the inline finding, a candidate issue was also investigated and ruled out this run: the staking precompile's delegate/redelegate/createValidator paths appearing to skip the MaxVotingPowerRatio cap — verified this is not a real bug (the cap is enforced elsewhere in the flow).
Extended reasoning...
This run added one new inline finding (a nit about occWorkerPool serializing OCC execution across --workers in the standalone evmonly-loadtest harness) plus a ruled-out candidate (staking precompile MaxVotingPowerRatio cap). Given this PR's scope (300 files, including consensus-sensitive app.go giga-fallback changes, EVM RPC hardening, and config-management changes) and that prior runs already flagged unresolved issues (lint failures, an ERC20 fixture bytecode inversion, an async sink release race, and a prepareBlocks error-masking bug), this PR clearly still requires human review; this note is purely to record what else was checked this run, not a new verdict.
205f54c to
beef917
Compare
d0ab196 to
5cff00b
Compare
5cff00b to
6eb0196
Compare
| if orderErr != nil { | ||
| return orderErr | ||
| } | ||
| return workerErr |
There was a problem hiding this comment.
Prepare errors masked by ordering
Medium Severity
forwardPreparedBlocksInOrder is driven by the parent ctx rather than the prepare errgroup context. When a prepare worker fails, the unordered stream closes with gaps and the orderer returns a generic missing-block error first, hiding the real prepare failure.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 6eb0196. Configure here.
| CompletionTime: completionTime, | ||
| InitialBalance: amount.String(), | ||
| SharesDst: amount.String(), | ||
| }) |
There was a problem hiding this comment.
Redelegation creation height always zero
Medium Severity
addRedelegation hardcodes CreationHeight to 0 on every entry, while unbonding entries record the block height. Redelegation queries and any height-based logic therefore see incorrect metadata.
Reviewed by Cursor Bugbot for commit 6eb0196. Configure here.
|
|
||
| func addRedelegation(store precompiles.Store, delegator string, srcValidator string, dstValidator string, amount *big.Int, completionTime int64) error { | ||
| record, _, err := getRedelegation(store, delegator, srcValidator, dstValidator) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| record.DelegatorAddress = delegator | ||
| record.ValidatorSrcAddress = srcValidator | ||
| record.ValidatorDstAddress = dstValidator | ||
| record.Entries = append(record.Entries, RedelegationEntry{ | ||
| CreationHeight: 0, | ||
| CompletionTime: completionTime, | ||
| InitialBalance: amount.String(), | ||
| SharesDst: amount.String(), | ||
| }) | ||
| if err := util.SetJSON(store, redelegationKey(delegator, srcValidator, dstValidator), record); err != nil { | ||
| return err | ||
| } | ||
| if err := addStringListItem(store, redelegationsIndexKey(), redelegationID(delegator, srcValidator, dstValidator)); err != nil { | ||
| return err | ||
| } | ||
| return insertRedelegationQueue(store, completionTime, delegator, srcValidator, dstValidator) | ||
| } |
There was a problem hiding this comment.
🟡 In addRedelegation (giga/evmonly/precompiles/staking/state.go:184-206), every redelegation entry hardcodes CreationHeight: 0 instead of taking the current block height as a parameter, unlike the analogous addUnbondingDelegation which correctly threads it through. As a result, every redelegation created via the staking precompile's redelegate() method reports CreationHeight=0 in query responses (redelegations(), validator/delegator redelegation queries) regardless of the actual block height. The fix is a one-line change: add a creationHeight parameter to addRedelegation and pass ctx.Block.Number from the caller, mirroring the unbonding path.
Extended reasoning...
The bug. addRedelegation in giga/evmonly/precompiles/staking/state.go:184-206 builds a new RedelegationEntry for every redelegation created through the staking precompile's redelegate() method, but it hardcodes CreationHeight: 0 (line ~194) rather than deriving it from the block at which the redelegation was initiated. The function signature itself doesn't even accept a height parameter — it only takes (store, delegator, srcValidator, dstValidator, amount, completionTime).
This is inconsistent with the analogous unbonding path: addUnbondingDelegation (state.go:156-179) correctly accepts a creationHeight parameter and sets CreationHeight: saturatingInt64FromUint64(creationHeight). The redelegate() caller in staking.go has ctx.Block.Number readily available — it already uses ctx.Block.Time to compute util.SaturatingCompletionTime(ctx.Block.Time, params.UnbondingTime) for the completion time — but that same call site never threads the block number into addRedelegation.
Why nothing catches it. The e2e test requireStakingRedelegation (executor_test.go) only asserts InitialBalance, SharesDst, and CompletionTime on redelegation entries — it never checks CreationHeight — so this asymmetry with the unbonding path slips through undetected.
Concrete proof. Suppose a delegator calls redelegate(srcValidator, dstValidator, amount) at block height 150 (block time 150). The resulting RedelegationEntry gets CompletionTime = 150 + unbondingTime (correct, derived from ctx.Block.Time) but CreationHeight = 0 (wrong — should be 150, derived from ctx.Block.Number). A client calling redelegations(), validatorDelegations(), or delegatorValidators() and inspecting the entry's creationHeight field would see 0 regardless of whether the redelegation actually happened at block 150, block 10,000, or any other height.
Impact is confined to query-response fidelity. I verified (and all three independent verifiers agree) that RedelegationEntry.CreationHeight is write-only in this package — it is never read by any internal logic. Redelegation maturity and completion are driven entirely by CompletionTime via the time-keyed queue (completeMatureRedelegations/matureTimeQueueIDs), never by CreationHeight. In upstream Cosmos SDK, CreationHeight on a redelegation entry is consumed by the slashing module to determine which entries are subject to a given slashing event — but this precompile's README explicitly documents that slashing is out of scope ("does not model staking rewards, slashing, or jailing"), so there is no slashing consumer to feed incorrect data to either. Pool tokens, validator tokens/shares, delegation balances, and validator-set updates are all computed independently of this field and are unaffected.
Fix. Add a creationHeight uint64 parameter to addRedelegation, set CreationHeight: saturatingInt64FromUint64(creationHeight) (mirroring addUnbondingDelegation), and pass ctx.Block.Number from the redelegate() call site in staking.go. This is a one-line functional change plus a signature update, with no behavioral risk since the field isn't consumed elsewhere.
There was a problem hiding this comment.
The loadtest harness, executor prepare/execute split, and pooled BlockResult refcounting are well built and well tested, but the staking precompile carried in this diff has two real problems: self-undelegation never enforces/jails on MinSelfDelegation (diverging from Cosmos), and hasReceivingRedelegation scans a single global redelegation index on every redelegate, making gas cost grow with chain-wide state. Two of Codex's four findings do not reproduce against sei-cosmos and should be dismissed.
Findings: 3 blocking | 15 non-blocking | 8 posted inline
Blockers
- The staking precompile has no jailing mechanism at all:
Validator.Jailedis read byvalidatorsByPower(endblock.go:182) but is never set to true anywhere in the package. Combined with the missingMinSelfDelegationcheck on undelegate, a validator's declared minimum self-stake is unenforceable after creation. Either wire jailing or document explicitly that jailing/slashing is deliberately out of scope for the evm-only path. - 2 blocking issue(s) flagged inline on specific lines.
Non-blocking
- Codex finding #3 (commission decreases not rate-limited, commission.go:110) is a false positive and should not be actioned:
sei-cosmos/x/staking/types/commission.go:97usesnewRate.Sub(c.Rate).GT(c.MaxChangeRate)with no absolute value, so unbounded decreases are Cosmos behavior. The new code matches upstream. - Codex finding #4 (editValidator compares new minSelfDelegation against total
Tokens, staking.go:519) is also a false positive:sei-cosmos/x/staking/keeper/msg_server.go:170doesmsg.MinSelfDelegation.GT(validator.Tokens)— the same comparison against total tokens including third-party delegations. - The Cursor review pass (
cursor-review.md) produced no output — that file is empty, so this review reflects only Claude's and Codex's findings. validatorDelegationsIndexKey(validator)has the same unbounded-list problem as the redelegation index: everydelegate/undelegatereads, sorts, and rewrites the validator's complete delegator list through the chunkedstorageBackedStore. A validator with many delegators will eventually make delegation to it impossible on gas. Worth addressing alongside the redelegation index (e.g. add prefix iteration toprecompiles.Store).Precompile.RequiredGas(staking.go:136) fully ABI-decodes the input viap.prepare(input)and throws the result away, thenRundecodes it again. On theRunAndCalculateGaspath this doubles decode cost for every staking call.isTransaction(giga/evmonly/precompiles/staking/helpers.go:65) is dead code — nothing in the package references it.metricsServer.stopreturns early ifserver.Shutdownerrors, so<-s.doneis never drained and theServegoroutine leaks. Harmless at process exit but trivial to fix.--result-sink=filealways deletes the changeset/receipt files on close (resultSinks.Closejoinss.close()withs.Cleanup()), including on successful completion. This is intentional per the PR description, but the--persist-dirflag help text ("directory for --result-sink=file append-only changeset and receipt files") reads like the data is retained; worth stating the removal behavior in the flag help and the loadtest README.- No test covers
MinSelfDelegationbehavior at all —guardrails_test.goandstaking_test.gonever reference it. If the jailing semantics are added, please add coverage for undelegating self-stake below the minimum, and foreditValidatorraising the minimum above current tokens. - 6 suggestion(s)/nit(s) flagged inline on specific lines.
| if err := validateDelegationAmount(ctx.Store, delegator, validatorAddress, amount); err != nil { | ||
| return nil, err | ||
| } | ||
| if err := addDelegation(ctx.Store, delegator, validatorAddress, new(big.Int).Neg(amount)); err != nil { |
There was a problem hiding this comment.
[blocker] Self-undelegation can drop the operator below its own MinSelfDelegation with no consequence. Cosmos handles this in sei-cosmos/x/staking/keeper/delegation.go:771-775: when the unbonding delegator is the operator and the remaining self-delegation falls below validator.MinSelfDelegation, the validator is jailed. This path has no equivalent — the validator keeps its status, stays in validatorsByPower, and continues receiving ValidatorUpdates.
Failure: create a validator with minSelfDelegation = 1_000_000 and self-delegate exactly that, then undelegate(self, 999_999). The validator remains bondStatusBonded with 1 usei of self-stake and is still returned by applyAndReturnValidatorSetUpdates, so the advertised minimum is unenforceable after creation. Given the branch commit "staking: align evm-only precompile with Cosmos semantics", this looks unintentional.
| // hasReceivingRedelegation reports whether the delegator has an in-progress | ||
| // redelegation whose destination is validator, mirroring Cosmos | ||
| // HasReceivingRedelegation (used to reject transitive redelegations). | ||
| func hasReceivingRedelegation(store precompiles.Store, delegator string, validator string) (bool, error) { |
There was a problem hiding this comment.
[blocker] hasReceivingRedelegation reads the chain-global redelegation index on every redelegate call. getStringList(store, redelegationsIndexKey()) loads one JSON array containing every in-progress redelegation ID chain-wide, and storageBackedStore.Get expands that into ceil(len/32) SLOADs plus a keccak per chunk. addRedelegation then rewrites the whole array via addStringListItem.
Failure: with N in-progress redelegations chain-wide, each redelegate costs O(N) cold SLOADs. Any account can inflate N cheaply by opening many small redelegations across validator pairs; once N·32 bytes exceeds what fits in the block/tx gas limit, redelegate reverts with out-of-gas for all users — a griefing vector, not just a scaling wart. The same shape applies to addStringListItem(redelegationsIndexKey(), ...) on the write side.
This needs a keyed lookup (e.g. a receiving-redelegation/<delegator>/<dstValidator> counter key, which is O(1) and exactly what Cosmos' HasReceivingRedelegation prefix iterator gives you) rather than a global list scan.
| commissionMaxRate := args[3].(string) | ||
| commissionMaxChangeRate := args[4].(string) | ||
| minSelfDelegation := args[5].(*big.Int) | ||
| pubKey, err := hex.DecodeString(pubKeyHex) |
There was a problem hiding this comment.
[suggestion] createValidator accepts any hex string as the consensus pubkey with no length or type validation, and those raw bytes are persisted into Validator.ConsensusPubkey, keyed into validator-consensus-pubkey/<hex>, and emitted verbatim as precompiles.ValidatorUpdate.PubKey (endblock.go:222).
Failure: createValidator("", ...) succeeds and registers a validator with a zero-length consensus key; createValidator("aabb", ...) registers a 2-byte key. Because the empty key also occupies the single validator-consensus-pubkey/ slot, it additionally blocks any other empty-key registration via errDuplicateConsensusKey, which is a confusing side effect.
Not currently exploitable — nothing consumes BlockResult.ValidatorUpdates yet — but this is state that gets written permanently, and the existing production precompile at least funnels the bytes through ed25519.PubKey. A len(pubKey) != ed25519.PubKeySize check here is cheap insurance before ValidatorUpdates gets wired to consensus.
|
|
||
| func forwardPreparedBlocksInOrder(ctx context.Context, in <-chan preparedBlockEnvelope, out chan<- preparedBlockEnvelope) error { | ||
| next := uint64(1) | ||
| pending := make(map[uint64]preparedBlockEnvelope) |
There was a problem hiding this comment.
[suggestion] pending is an unbounded reorder buffer, which defeats --queue-size backpressure during builder skew. The loop always drains in (out-of-order blocks go straight into pending and the loop continues), so it never applies backpressure while it is waiting for block next.
Failure: with --builders=96, if the goroutine that drew block 1 stalls, the other 95 builders keep drawing new numbers and every prepared block lands in pending — nothing blocks, so RSS grows with txs-per-block × skew. At --txs-per-block=5000 that is ~500 KB of raw RLP plus decoded txs per buffered block. Bounding pending (e.g. block on in once len(pending) reaches queueSize) would keep the documented in-flight bound honest.
| } | ||
| } | ||
|
|
||
| func (s *asyncFileResultSinks) releaseQueuedResults() { |
There was a problem hiding this comment.
[suggestion] releaseQueuedResults drains non-blockingly and the writer goroutine then returns without closing s.records, so a pooled BlockResult can be leaked on the write-error path.
Failure: producer P passes the s.getErr() check at line 1622 (error not yet set), then the writer hits a write error, calls setErr, runs releaseQueuedResults (queue empty), and returns. P's send at line 1626 or 1634 now succeeds into the still-open buffered channel, and nothing will ever drain it — that record's release closure is never invoked, so the lease refcount never reaches zero and the pool slot is permanently lost. Repeated occurrences push every subsequent acquire onto the overflow-allocation path. Recording the error before the final drain, or draining once more after setErr, closes the window.
| return err | ||
| } | ||
| if t.db.GetBalance(from).Cmp(u) < 0 { | ||
| t.db.err = errInsufficientBalance |
There was a problem hiding this comment.
[suggestion] Setting the sticky t.db.err turns a precompile balance shortfall into a fatal block error rather than a reverted transaction: executeTx checks stateDB.Error() (executor.go:275) and executeBlockSequential wraps it into execute tx %d ... %w, failing the entire block. The sticky error is only cleared by reset(source), so it also poisons every later tx in the block.
Today escrow accounting looks internally balanced so this should not fire, but a single accounting bug in completeUnbonding's escrow→delegator transfer would make blocks unproducible instead of just failing one delegation. Returning errInsufficientBalance without touching t.db.err (and letting the EVM frame revert) would be the safer shape.
| } | ||
| data, err := event.Inputs.NonIndexed().Pack(args...) | ||
| if err != nil { | ||
| return |
There was a problem hiding this comment.
[nit] A Pack failure silently drops the log. Combined with Precompile.emit also returning silently when the event name is missing from the ABI, an argument-type mismatch produces a successful staking call with no event and no error — hard to notice and hard to debug. Returning the error (or at minimum having callers surface it) would be better; note the log-gas charge in meteredLogSink.AddLog is also skipped in this case, so gas accounting silently diverges too.
| func blockContext(cfg config, number uint64) evmonly.BlockContext { | ||
| return evmonly.BlockContext{ | ||
| Number: number, | ||
| Time: uint64(time.Now().Unix()), |
There was a problem hiding this comment.
[nit] Block timestamps are stamped with time.Now() inside buildBlock, which runs concurrently across --builders goroutines, so block N can end up with an earlier Time than block N-1. The executor does not validate monotonicity, so this only skews any time-dependent workload (and would matter if a custom-precompile workload using ctx.Block.Time is added later). Deriving Time from number (e.g. a fixed genesis time plus number) would make prebuilt runs deterministic and monotonic.
| MaxVotingPowerEnforcementThreshold: "0.000000000000000000", | ||
| }, nil | ||
| } | ||
|
|
||
| func loadPool(store precompiles.Store) (Pool, error) { | ||
| pool, ok, err := util.GetJSON[Pool](store, poolKey()) | ||
| if err != nil { | ||
| return Pool{}, err | ||
| } | ||
| if ok { | ||
| return pool, nil | ||
| } | ||
| return Pool{NotBondedTokens: "0", BondedTokens: "0"}, nil | ||
| } | ||
|
|
||
| func addPoolBonded(store precompiles.Store, delta *big.Int) error { | ||
| pool, err := loadPool(store) | ||
| if err != nil { | ||
| return err |
There was a problem hiding this comment.
🔴 hasReceivingRedelegation and addRedelegation read/rewrite a single chain-global JSON array ("redelegations/index") holding every in-progress redelegation ID for the whole chain, instead of a delegator/validator-keyed lookup like Cosmos's HasReceivingRedelegation. Every redelegate() call therefore costs O(N) metered SLOAD/SSTORE work where N is the total number of open redelegations chain-wide, so any account can cheaply inflate N with many small redelegations and eventually push everyone else's redelegate() calls out-of-gas. This should be replaced with a keyed per (delegator, dstValidator) lookup (e.g. a counter or presence key) before this precompile is wired to production.
Extended reasoning...
redelegationsIndexKey() (state.go) returns a single fixed key, "redelegations/index", whose value is a JSON array containing the ID (delegator\x00src\x00dst) of every in-progress redelegation for every delegator and validator pair on the chain — it is not scoped to the (delegator, dstValidator) pair being checked.
hasReceivingRedelegation (state.go:249-267) is called from redelegate() to reject transitive redelegations. It loads that entire global list via getStringList(store, redelegationsIndexKey()) and linearly scans every entry looking for one whose delegator/dst match the current call. On the write side, addRedelegation then calls addStringListItem(redelegationsIndexKey(), ...), which re-reads the whole array, appends the new ID, re-sorts, and writes the entire array back via util.SetJSON.
Because this JSON blob is persisted through storageBackedStore, which chunks it into ceil(len/32) 32-byte EVM storage slots, each of those chunks is a separately metered SLOAD or SSTORE (plus a keccak per chunk to derive its slot). So every single redelegate() call does at least two O(N) storage operations, where N is the total count of in-progress redelegations chain-wide — not O(1), and not O(entries for this specific delegator/validator pair) the way Cosmos's HasReceivingRedelegation prefix-scan iterator provides.
This is a genuine scaling and griefing concern, not just an inefficiency: any account can cheaply open many small redelegations across distinct validator pairs (entries persist for the full ~21-day unbonding period before completeMatureRedelegations prunes them), inflating N indefinitely. Once N grows large enough that reading/rewriting the array exceeds the tx or block gas limit, redelegate() starts reverting out-of-gas for every user on the chain, not just the account that inflated N — this is a griefing vector, not merely a self-inflicted cost.
Step-by-step proof:
- Attacker calls
redelegate(srcA, dstB, 1),redelegate(srcC, dstD, 1), ... repeatedly across many distinct validator pairs, using a fresh delegator or validator pair each time so each redelegation gets a unique ID in the global array. - Each call appends one more ~130-byte entry (delegator/src/dst hex addresses plus JSON overhead) to the single global
redelegations/indexarray viaaddStringListItem. - Because entries only get removed 21 days later (
completeMatureRedelegations), the attacker can accumulate an arbitrarily large N within that window at the cost of only the storage-write gas for each individual small redelegation. - Once N is large, any other user's unrelated
redelegate()call must first pay forhasReceivingRedelegation's O(N) read (chunked SLOADs) and thenaddRedelegation's O(N) read + O(N) write (chunked SLOADs and SSTOREs) just to append one more ID — this happens even though their redelegation has nothing to do with the attacker's validator pairs. - Past some N, this O(N) cost exceeds the tx/block gas limit, and
redelegate()reverts out-of-gas for every account on the chain, not just the attacker.
This is new code introduced by this PR (there is no pre-existing global-scan issue to inherit), and it directly undermines the PR's stated goal of "aligning the evm-only precompile with Cosmos staking semantics" — Cosmos's own implementation is keyed and does not have this scaling problem. The fix is straightforward: replace the global-array scan with a keyed presence/counter key such as receiving-redelegation/<delegator>/<dstValidator>, giving O(1) lookup and O(1) insert/remove instead of scanning and rewriting the whole chain-wide list.
One caveat worth noting: at the time of this PR, the giga/evmonly/precompiles/staking package is not yet referenced from app/app.go or any other production wiring (it's exercised only by this package's own tests), so there is no live network today where this can be exploited. However, the flaw is inherent to the data structure this PR introduces for the precompile's core write path, and it should be fixed now — before other logic or on-disk layout builds further on top of the global redelegation index — rather than being carried forward as technical debt into the precompile's production rollout.
|
|
||
| func forwardPreparedBlocksInOrder(ctx context.Context, in <-chan preparedBlockEnvelope, out chan<- preparedBlockEnvelope) error { | ||
| next := uint64(1) | ||
| pending := make(map[uint64]preparedBlockEnvelope) | ||
| for { | ||
| for { | ||
| block, ok := pending[next] | ||
| if !ok { | ||
| break | ||
| } | ||
| if err := sendPreparedBlock(ctx, out, block); err != nil { | ||
| return err | ||
| } | ||
| delete(pending, next) | ||
| next++ | ||
| } | ||
|
|
||
| select { | ||
| case <-ctx.Done(): | ||
| return nil | ||
| case block, ok := <-in: | ||
| if !ok { | ||
| for len(pending) > 0 { | ||
| block, ok := pending[next] |
There was a problem hiding this comment.
🟡 In forwardPreparedBlocksInOrder (giga/evmonly/cmd/evmonly-loadtest/main.go:822-874), the loop always drains the bounded unordered channel into the pending reorder-buffer map with no size cap, even while the head-of-line block is stalled — so a single slow prepare worker lets pending grow to hold every block prepared after it, defeating the --queue-size backpressure the README documents. This only affects the standalone evmonly-loadtest benchmark harness (RSS growth under skew), not correctness or consensus code; fix by blocking on <-in once len(pending) reaches a bound (e.g. queueSize).
Extended reasoning...
forwardPreparedBlocksInOrder (giga/evmonly/cmd/evmonly-loadtest/main.go:822-874) is responsible for re-ordering blocks that come out of the parallel prepareWorkers pool back into sequential order before handing them to the single executor worker. It maintains a pending map[uint64]preparedBlockEnvelope reorder buffer and a next counter. The outer loop first drains every contiguous run starting at next out of pending and forwards it downstream, then falls into a select that unconditionally reads one more block from the bounded in channel (capacity queueSize) and stashes it in pending if its number does not equal next.
The problem is that this drain-from-in step is never gated on the size of pending. As long as in has something to receive, the orderer will pull it out and buffer it, regardless of how large pending has already grown. Because the orderer keeps draining in, the prepareWorkers sending into it are never blocked either, so they keep pulling from the blocks channel and calling PrepareBlock, and the builders goroutines that feed blocks keep advancing nextBlock.Add(1) and producing new raw blocks. In other words, the one place that could apply real backpressure back through the whole pipeline (prepare workers -> builders) declines to do so whenever it happens to be waiting on a stalled head-of-line block.
This directly undermines the documented purpose of --queue-size: the README/README.md advertise it as bounding the number of blocks in flight. In practice, if any single prepare worker is slow or descheduled while holding block next (which does per-tx ECDSA sender recovery — real work with real scheduling variance, especially with many --prepare-workers/--builders), every block prepared after it accumulates in pending with no limit besides the total block count. Each buffered entry holds a full decoded PreparedBlock (recovered senders) plus the underlying raw RLP, so at --txs-per-block=5000 this is on the order of hundreds of KB per stalled block — multiplied by however many blocks pile up during the stall.
Step-by-step reproduction of the mechanism:
- Run with e.g.
--builders=96 --prepare-workers=48 --queue-size=64and prebuilt/streamed blocks. - Suppose the prepare worker handling block 1 gets descheduled or is simply slower than the others (plausible skew given 48 workers racing on CPU-bound sender recovery).
- The other 47 prepare workers keep completing blocks 2, 3, 4, ... and sending them into
unordered(capacityqueueSize). forwardPreparedBlocksInOrders select loop keeps drainingunorderedbecause it never checkslen(pending), so it accepts blocks 2..N intopendingwithout limit as they arrive, well beyond thequeueSizebound that is supposed to cap in-flight blocks.- Meanwhile, since
unorderedkeeps draining, prepare workers are never blocked on their send at line ~800, so they keep pulling new raw blocks fromblocks, and builders keep advancingnextBlockand building more. - RSS grows proportional to
txs-per-block * (skew duration in blocks)until block 1 finally completes and the buffered run is flushed.
This is a real gap in the documented backpressure guarantee, and it was already independently flagged by reviewer seidroid at the same line ("pending is an unbounded reorder buffer, which defeats --queue-size backpressure during builder skew").
Fix: gate the <-in receive in the select on len(pending) < queueSize (or another explicit bound), so once the reorder buffer reaches that bound the orderer stops draining in until it can flush more of pending at next. That causes the unordered channel to fill up, which naturally blocks prepare workers, which naturally blocks builders — restoring end-to-end backpressure through the whole pipeline.
Severity: this is confined to the standalone evmonly-loadtest benchmark harness (cmd/evmonly-loadtest), not consensus or production execution code. Output ordering and correctness are unaffected — only RSS during a manually-run load test can grow beyond what --queue-size implies, and it requires real prepare-worker skew to manifest, which is naturally bounded by the total block count in bounded runs. All three independent verifiers and the reviewer converged on nit for this reason.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 4 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e7fc118. Configure here.
| } | ||
| if err := setHistoricalInfo(ctx.Store, ctx.Block.Number); err != nil { | ||
| return nil, err | ||
| } |
There was a problem hiding this comment.
Mid-tx historical info writes
Medium Severity
createValidator and editValidator call setHistoricalInfo during the transaction. Historical info is supposed to be recorded only in end-block, so same-block queries can observe an incomplete validator snapshot before remaining transactions finish.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e7fc118. Configure here.
There was a problem hiding this comment.
The load-test harness, result-sink plumbing, and the precompile gas meter are careful, well-tested work, but this PR also lands a complete ~2,700-line SDK-free staking precompile that reaches consensus (validator updates) with no consensus-pubkey validation and a byte-store index design whose gas/state cost grows linearly with delegator and redelegation counts. Those need addressing (or the staking half needs splitting out) before merge.
Findings: 4 blocking | 21 non-blocking | 17 posted inline
Blockers
giga/evmonly/precompiles/stakingproducesValidatorUpdates that flow intoBlockResult.ValidatorUpdatesand, eventually, consensus — but nothing validates the consensus public key. Please add explicit key-type/length validation (ed25519, 32 bytes, matching the consensus params'PubKeyTypes) plus tests for empty / wrong-length / duplicate keys. Note the new tests themselves pass 4-byte ("01020304") and 20-byte (operator.Hex()[2:]) "pubkeys", which shows how easily invalid keys get in.- The store index design (
getStringList/addStringListItem/removeStringListItemover whole JSON arrays, chunked into 32-byte storage slots) makes per-call gas and per-block state work grow linearly with the number of delegators per validator, redelegations chain-wide, and historical-info entries. A validator with many delegators will eventually makedelegateunpayable, andtrackHistoricalInforewrites the full validator set every block. This needs a keyed/paginated layout (or an explicit documented bound) before more code is built on this boundary. - 2 blocking issue(s) flagged inline on specific lines.
Non-blocking
cursor-review.mdis empty — the Cursor pass produced no output for this PR, so no findings were merged from it.- Scope: the PR title/description are entirely about the
evmonly-loadtestharness, but ~2,700 of the 7,314 added lines are a brand-new staking precompile, precompile adapter, and dynamic gas meter with consensus-affecting semantics. Consider splitting so the staking module gets a dedicated review; reviewers scanning a "load test harness" PR will not scrutinize it appropriately. - Codex flagged
editValidatorcomparing the newminSelfDelegationagainstvalidator.Tokens(total, including third-party delegations) rather than self-delegation. I disagree — Cosmos SDK'smsgServer.EditValidatorcompares againstvalidator.Tokenstoo, so this matches upstream semantics. No change needed. giga/evmonly/cmd/evmonly-loadtest/README.mdis now out of date: it documents none of--result-sink,--persist-dir,--persist-sync,--persist-buffer-size,--persist-queue-size,--cpu-profile,--heap-profile,--trace-profile,--erc20-contract,--tx-gas-limit,--block-gas-limit,--coinbase,--chain-id, or--disable-gas-price-rule, and still asserts "the executor output is intentionally discarded through mocks," which is only true for--result-sink=discard. It also doesn't mention that the file sink deletes its own output on exit.- Several end-block invariant violations (
errValidatorMissinginapplyAndReturnValidatorSetUpdates, "validator queue contains a validator that is not unbonding" inunbondAllMatureValidators) propagate out ofrunCustomPrecompileEndBlockand fail the whole block. Once this is wired into consensus, any state inconsistency becomes an unrecoverable halt — worth deciding deliberately whether these should be fatal. - Test-coverage gaps worth closing: no test for an invalid/empty consensus pubkey; no test that a mid-
Store.Get/Store.Setout-of-gas leaves no partial state committed; no test asserting that registering a custom precompile disables OCC for all blocks. - 15 suggestion(s)/nit(s) flagged inline on specific lines.
| commissionMaxRate := args[3].(string) | ||
| commissionMaxChangeRate := args[4].(string) | ||
| minSelfDelegation := args[5].(*big.Int) | ||
| pubKey, err := hex.DecodeString(pubKeyHex) |
There was a problem hiding this comment.
[blocker] hex.DecodeString is the only check applied to the consensus pubkey — any byte string (including empty, 4 bytes, or 20 bytes) is accepted, stored on the validator, and later emitted verbatim as ValidatorUpdate.PubKey from EndBlock. The SDK path this replaces goes through ed25519.PubKey + MsgCreateValidator.ValidateBasic + the consensus params' PubKeyTypes check, so it cannot produce a malformed consensus key.
Validate the key type and length here (ed25519 ⇒ exactly 32 bytes) before setValidator, otherwise a single createValidator call can inject an unusable validator-set update. Note the duplicate-key index is keyed on hex.EncodeToString(pubKey), so an empty key also occupies the "" slot. (Also raised by Codex.)
| return util.SetJSON(store, key, items) | ||
| } | ||
|
|
||
| func addStringListItem(store precompiles.Store, key []byte, item string) error { |
There was a problem hiding this comment.
[blocker] Every index read/write here decodes, sorts, re-encodes, and rewrites the entire JSON list through storageBackedStore, which fans out to one gas-metered SSTORE per 32 bytes. For validatorDelegationsIndexKey(validator) that means each delegate costs gas proportional to the validator's current delegator count — so a popular validator eventually becomes undelegatable-to at any block gas limit. redelegationsIndexKey() is worse: a single global list touched by every redelegate.
This needs a keyed layout (one slot per (validator, delegator) membership marker, or an explicit count + indexed entries) rather than a monolithic sorted array. Same applies to historicalInfoIndexKey() (default HistoricalEntries: 10_000).
| // hasReceivingRedelegation reports whether the delegator has an in-progress | ||
| // redelegation whose destination is validator, mirroring Cosmos | ||
| // HasReceivingRedelegation (used to reject transitive redelegations). | ||
| func hasReceivingRedelegation(store precompiles.Store, delegator string, validator string) (bool, error) { |
There was a problem hiding this comment.
[suggestion] hasReceivingRedelegation reads the global redelegation index and linearly scans it on every redelegate call. Beyond the gas cost noted above, this makes every redelegation in the chain read and write the same storage slots — a permanent global serialization point for the OCC conflict tracker once the precompile path is optimistically executed. Consider a per-(delegator, dstValidator) marker key instead.
| record.ValidatorSrcAddress = srcValidator | ||
| record.ValidatorDstAddress = dstValidator | ||
| record.Entries = append(record.Entries, RedelegationEntry{ | ||
| CreationHeight: 0, |
There was a problem hiding this comment.
[suggestion] CreationHeight is hardcoded to 0, so every entry returned by the redelegations query reports height 0. addUnbondingDelegation correctly threads creationHeight through — do the same here (addRedelegation already has ctx.Block.Number available at the call site in redelegate).
| return method.Outputs.Pack(true) | ||
| } | ||
|
|
||
| func (p *Precompile) undelegate(ctx *precompiles.Context, method *abi.Method, args []interface{}) ([]byte, error) { |
There was a problem hiding this comment.
[suggestion] undelegate never re-checks MinSelfDelegation when the caller is the validator operator, so an operator can undelegate their entire self-stake and stay in the active set with a MinSelfDelegation they no longer satisfy. Cosmos handles this in keeper.unbond() by jailing the validator, which giga/evmonly/README.md explicitly says is out of scope — so this is a documented gap rather than an oversight, but the README should state the consequence directly: MinSelfDelegation is only enforced at createValidator/editValidator time and is unenforceable thereafter. (Raised by Codex as High; downgrading given the documented no-jailing scope.)
| if err := completeMatureRedelegations(ctx.Store, ctx.Block); err != nil { | ||
| return nil, err | ||
| } | ||
| if err := trackHistoricalInfo(ctx.Store, ctx.Block); err != nil { |
There was a problem hiding this comment.
[suggestion] trackHistoricalInfo runs unconditionally every block with no gas metering (the end-block storageBackedStore is built with meter: nil), and setHistoricalInfo serializes the entire validator set as JSON into 32-byte storage slots, while pruneHistoricalInfo reads and rewrites a height index that grows to HistoricalEntries (default 10,000). That's a large, unbounded, unmetered per-block state write. Please bound it explicitly or move historical info behind a cheaper representation before this reaches consensus.
| for len(pending) > 0 { | ||
| block, ok := pending[next] | ||
| if !ok { | ||
| if ctx.Err() != nil { |
There was a problem hiding this comment.
[suggestion] This checks the outer ctx, not the inner groupCtx that a failing prepare worker cancels. So when a prepare worker errors, unordered closes with gaps, forwardPreparedBlocksInOrder returns "prepared block stream closed before block N", and prepareBlocks returns orderErr in preference to workerErr — hiding the actual prepare failure. Prefer workerErr when it's non-nil, or plumb groupCtx into the forwarder.
| if _, err := f.writer.Write(payload); err != nil { | ||
| return 0, fmt.Errorf("write record payload for height %d: %w", height, err) | ||
| } | ||
| if err := f.writer.Flush(); err != nil { |
There was a problem hiding this comment.
[suggestion] Flushing on every record makes --persist-buffer-size a no-op and forces one write syscall per record, which directly inflates the sink_write metric the PR uses as its persistence-cost signal. Since Close() already flushes and syncs, drop the per-record Flush (keep it under syncOnWrite).
| } else { | ||
| closeErr = s.close() | ||
| } | ||
| return errors.Join(closeErr, s.Cleanup()) |
There was a problem hiding this comment.
[suggestion] Close always runs Cleanup, so --result-sink=file --persist-dir=... deletes changesets.rlp / receipts.rlp on successful completion too. That makes the persisted output impossible to inspect or replay, which seems at odds with calling it a persistent sink. Consider keeping files on clean exit (removing only on error/signal) or adding a --keep-persist-files flag — and document the behaviour either way.
| func (s *generatedState) GetBalance(addr common.Address) *big.Int { | ||
| if s.frozen.Load() { | ||
| if balance, ok := s.balances[addr]; ok && balance != nil { | ||
| return balance |
There was a problem hiding this comment.
[suggestion] The frozen fast path returns the map's *big.Int directly (and a package-level shared frozenZeroBalance), whereas the unfrozen path returns new(big.Int).Set(balance). Any consumer that mutates the returned value in place would silently corrupt the generated genesis for all subsequent blocks — and frozenZeroBalance is shared across every zero-balance address. Copy here too, or return an immutable type.
There was a problem hiding this comment.
Beyond the inline nit finding, I checked two other candidate issues raised this run and ruled them both out. Self-undelegation not enforcing/jailing below MinSelfDelegation: the README's "Current limitations" section explicitly scopes slashing and jailing out of this precompile, so the missing enforcement matches documented scope rather than being an unintended bug. Prepare-workers/ParseWorkers goroutine multiplication in the loadtest harness: this is a tunable knob in a benchmarking tool, not production code, and the PR's own EC2 tuning notes already exercise and tune around this exact interaction (e.g. "prepare-workers=24 became sender-recovery limited; increasing to 48 recovered throughput").
Extended reasoning...
This PR is large and touches a new SDK-free staking precompile plus a standalone load-test harness, so it warrants human review regardless — the inline nit finding and the still-open lint/gas-griefing/blocker items already flagged by other reviewers in the thread make that clear without restating them here. This note only records two additional candidate issues that finder/verifier agents raised and refuted this run, based on my own reading of the code and README, so they are not re-explored from scratch in a later pass.
| pubKeyHex := args[0].(string) | ||
| moniker := args[1].(string) | ||
| commissionRate := args[2].(string) | ||
| commissionMaxRate := args[3].(string) | ||
| commissionMaxChangeRate := args[4].(string) | ||
| minSelfDelegation := args[5].(*big.Int) | ||
| pubKey, err := hex.DecodeString(pubKeyHex) | ||
| if err != nil { | ||
| return nil, errors.New("invalid public key hex format") | ||
| } | ||
| params, err := loadParams(ctx.Store) | ||
| if err != nil { |
There was a problem hiding this comment.
🟡 createValidator hex-decodes the consensus pubkey with no length or type check before persisting it as Validator.ConsensusPubkey and later emitting it verbatim as precompiles.ValidatorUpdate.PubKey in endblock.go. As a result createValidator("", ...) registers a zero-length consensus key (which also permanently occupies the single validator-consensus-pubkey/ slot, blocking any later empty-key registration), and any short/malformed key like "aabb" is silently accepted.
Extended reasoning...
createValidator (giga/evmonly/precompiles/staking/staking.go:392-442) only checks the hex-decode error on the caller-supplied pubKeyHex:
pubKey, err := hex.DecodeString(pubKeyHex)
if err != nil {
return nil, errors.New("invalid public key hex format")
}There is no subsequent check on len(pubKey) or its structure before the bytes are (a) used as the dedup key via getValidatorByConsensusPubkey/setValidator (keyed into validator-consensus-pubkey/<hex>), and (b) persisted verbatim into Validator.ConsensusPubkey. That same byte slice is later handed straight to precompiles.ValidatorUpdate.PubKey by validatorUpdate() in endblock.go with no re-validation. Concretely, createValidator("", ...) succeeds and registers a validator with a zero-length consensus key, and createValidator("aabb", ...) registers one with a 2-byte key. Because the store key for the empty case collapses to the same validator-consensus-pubkey/ prefix with no hex suffix, it also permanently occupies that slot and would make any other validator's later attempt to register with an empty key fail via errDuplicateConsensusKey — a confusing side effect purely from having no length floor.
The production sei staking precompile funnels the equivalent input through ed25519.PubKey, which implicitly enforces len(pubKey) == ed25519.PubKeySize (32 bytes) before the key is usable. This evm-only path has no equivalent floor, so it diverges from that expectation and lets arbitrary-length, non-cryptographic byte strings become permanent validator state.
One reviewer objected that this is intentional given the test suite (guardrails_test.go, staking_test.go, executor_test.go) deliberately uses 4-byte pubkeys like "01020304", and that TestExecutorStakingPrecompileForwardsPayableValue asserts the 4-byte key flows through unchanged. That's accurate as a description of current test fixtures, but it doesn't establish that unbounded-length/empty keys are an intended contract — the tests use a fixed 4-byte placeholder purely for readability/brevity in assertions, not because 0-length or arbitrary-length keys are meant to be valid; nothing in the ABI, comments, or README documents that consensus pubkeys are permitted to be any length including zero. A len(pubKey) == ed25519.PubKeySize check would only reject genuinely malformed input (empty strings, truncated hex) — it would require updating the existing 4-byte test fixtures to 32-byte ones, which is a reasonable and expected consequence of tightening validation, not evidence the current behavior is by design.
Step-by-step proof:
- Caller A submits
createValidator("", "validator-a", "0.1", "0.2", "0.01", 1)with sufficient self-delegation value. hex.DecodeString("")succeeds and returns a zero-length[]byte{}.getValidatorByConsensusPubkey(store, []byte{})looks up keyvalidator-consensus-pubkey/(hex of empty bytes is the empty string) and finds nothing, so validation passes.- The validator is persisted with
ConsensusPubkey: []byte{}, and thevalidator-consensus-pubkey/slot is now occupied by caller A's operator address. - If caller B later also tries
createValidator("", ...), the same lookup now finds caller A's record and returnserrDuplicateConsensusKey, even though B never intended to collide with A's key — the collision is purely an artifact of the missing length floor. - At end-of-block,
validatorUpdate(validator, power)copiesvalidator.ConsensusPubkeyverbatim intoprecompiles.ValidatorUpdate.PubKey, so this zero-length (or any short) key is exactly what would reach a consensus-integration consumer ofBlockResult.ValidatorUpdatesonce that plumbing exists.
This is genuinely not exploitable today: nothing in this PR wires BlockResult.ValidatorUpdates into actual consensus, so no live system currently consumes malformed keys. That's why I'm filing this as a nit rather than a blocking issue — it's cheap, correct hardening (if len(pubKey) != ed25519.PubKeySize { return nil, errInvalidConsensusPubkeyLength } before the duplicate-check) that should land before this validator-update wiring is trusted by anything downstream, but it does not need to block this PR's merge.
1 participant
Summary
Adds a standalone
evmonly-loadtestcommand that feeds generated EVM-only blocks into the EVM-only executor with generated genesis state, configurable result sinks, and Prometheus/stdout throughput metrics.The executor owns the result-sink boundary.
giga/evmonlyexposesResultSink,BlockResultSink, andWithResultSink, and executor completion invokes the sink for produced outputs before returning. The loadtest harness implements discard/file sink modes through those interfaces.The executor can also use a bounded reusable
BlockResultpool viaBlockResultPoolSize. Pooled results are reference-counted: callers release their returned result withBlockResult.Release(), and async sinks retain/release throughBlockResultSinkafter changesets and receipts are no longer referenced. The loadtest harness enables this by default with--result-pool-size=0, sized for executor workers plus in-flight async sink records; negative disables result pooling.This branch also pipelines stateless sender recovery ahead of execution:
Executor.PrepareBlockdecodes raw tx RLP and recovers senders intoPreparedBlock.Executor.ExecutePreparedBlockexecutes an already prepared block.ExecuteBlockremains the convenience prepare-then-execute path.evmonly-loadtestnow has--prepare-workers; raw blocks flow through parallel prepare workers into an ordered prepared-block queue consumed by the single external executor worker.The harness supports optimistic no-overlap native transfers and ERC20 transfers. It defaults to unique senders and recipients, supports
--prebuild-blocks, and includes a lightweight async file sink via--result-sink=file --persist-dir=<dir>. Persistence records are append-only RLP records for changesets and receipts, and the sink removes files on normal completion, execution errors, andSIGINT/SIGTERM.Metrics include block input/prepared/finished throughput, prepared tx/s, executed tx/s, gas/s, OCC attempts/fallbacks/conflicts, result-sink queue depth, enqueue wait, write time, bytes written, and record counts.
sink_enqueue_waitis the backpressure signal: if it rises, executor workers are blocked on persistence queue capacity.Validation
go test ./giga/evmonly/...DIR=$(mktemp -d)first andremove it with
rmdir "$DIR"after the command succeeds.go run ./giga/evmonly/cmd/evmonly-loadtest \ --metrics-addr= \ --report-interval=0 \ --prebuild-blocks \ --blocks=30 \ --txs-per-block=1000 \ --builders=8 \ --prepare-workers=8 \ --workers=1 \ --executor-workers=12 \ --gas-price-wei=0 \ --min-gas-price-wei=0 \ --queue-size=64 \ --result-sink=file \ --persist-dir="$DIR"Observed locally with pooled async file persistence:
218,638 TPS,prepare_errors=0,errors=0,sink_enqueue_wait=0s, and after closesink_written=60.go run ./giga/evmonly/cmd/evmonly-loadtest \ --metrics-addr= \ --report-interval=0 \ --prebuild-blocks \ --blocks=30 \ --txs-per-block=1000 \ --builders=8 \ --prepare-workers=8 \ --workers=1 \ --executor-workers=12 \ --workload=erc20-transfer \ --gas-price-wei=0 \ --min-gas-price-wei=0 \ --queue-size=64 \ --result-sink=file \ --persist-dir="$DIR"Observed locally with pooled async file persistence:
202,486 TPS,prepare_errors=0,errors=0,sink_enqueue_wait=0s, and after closesink_written=60.EC2 Persistent Runs
Run on commit
fa7c7556busing a temporaryc8i.48xlargeinus-east-1a, SMT disabled withCoreCount=96,ThreadsPerCore=1, Go1.25.6,GOMAXPROCS=96,GOGC=400, one external executor worker, prebuilt raw blocks, unique senders/recipients, zero gas price/min gas price, default result pooling, and--result-sink=file. The temporary instance, key pair, and security group were deleted after collecting logs. Logs were copied locally to/tmp/evmonly-pool-20260702210816/logs.Native transfer tuned command:
GOMAXPROCS=96 GOGC=400 /tmp/evmonly-loadtest \ --metrics-addr= \ --blocks=1000 \ --txs-per-block=5000 \ --prebuild-blocks \ --builders=96 \ --prepare-workers=48 \ --workers=1 \ --executor-workers=40 \ --gas-price-wei=0 \ --min-gas-price-wei=0 \ --report-interval=5s \ --queue-size=64 \ --result-sink=file \ --persist-dir="$DIR"Observed:
ERC20 transfer tuned command:
GOMAXPROCS=96 GOGC=400 /tmp/evmonly-loadtest \ --metrics-addr= \ --blocks=1000 \ --txs-per-block=5000 \ --prebuild-blocks \ --builders=96 \ --prepare-workers=48 \ --workers=1 \ --executor-workers=48 \ --workload=erc20-transfer \ --gas-price-wei=0 \ --min-gas-price-wei=0 \ --report-interval=5s \ --queue-size=64 \ --result-sink=file \ --persist-dir="$DIR"Observed:
Tuning notes from the same EC2 instance:
result-pool-sizeblocks. Commita38d56ec6fixes that.nativeStateDBscratch reuse was tested and removed. It raised CPU use but lowered EC2 throughput to about166k TPS, likely from map-clearing/retained-heap costs.--prepare-workers=24became sender-recovery limited; increasing to48recovered throughput.--prepare-workers=48 --executor-workers=40,185.6k TPSover 2.5M tx. The full 5M-tx confirmation reached196.7k TPS.--prepare-workers=48 --executor-workers=48,173.5k TPSover 2.5M tx. The full 5M-tx confirmation reached181.0k TPS.sink_enqueue_wait=0s.Historical Non-Persistent 199.3k Repro
The earlier 199.3k EC2 benchmark was run before persistent sink and prepared pipeline changes, from commit
4ec8da52c, on ac8i.48xlargeinus-east-1awith SMT disabled, Go1.25.6, one external executor worker, prebuilt blocks, unique senders/recipients, and zero gas price/min gas price.Observed:
EC2 Worker Pool / Pinning Follow-up
A follow-up on temporary commit
63b8662b4bc90936175181b1fff6347c502fb03ftested persistent OCC workers with and without Linux worker pinning on the samec8i.48xlargeshape. The persistent OCC worker pool stayed in this branch; the worker pinning flags/files were removed afterward because they did not improve throughput or CPU utilization.Observed 5M-tx persistent-sink results:
186,607 TPS,sink_enqueue_wait=0s,Percent of CPU: 2404%.186,683 TPS,sink_enqueue_wait=0s,Percent of CPU: 2283%.170,501 TPS,sink_enqueue_wait=0s,Percent of CPU: 2003%.165,290 TPS,sink_enqueue_wait=0s,Percent of CPU: 2009%.Conclusion: the persistent OCC pool removes per-block worker creation overhead, but thread pinning was flat for native transfer and worse for ERC20. CPU remained around 20-24 effective cores on a 96-core instance, so the current branch keeps the simpler unpinned worker-pool model.