feat: add mev cli tests by PoulavBhowmick03 · Pull Request #300 · NethermindEth/pluto

PoulavBhowmick03 · 2026-03-28T07:57:10Z

Closes #235

iamquang95 · 2026-04-02T10:00:05Z

@claude

claude · 2026-04-02T10:00:20Z

Claude finished @iamquang95's task in 2m 58s —— View job

Code Review

Gather context and read changed files
Review crates/cli/src/commands/test/mev.rs
Review supporting changes
Post findings

Summary

This PR ports Charon's MEV relay test suite into the Pluto CLI, adding Ping, PingMeasure, and CreateBlock test cases plus two unit tests. The implementation is largely correct but has one functional parity gap (credential stripping), a concurrency correctness issue in timeout handling, and a structural refactor concern.

Findings

[High] mev_ping_measure_test ignores embedded URL credentials

mev.rs:319 — request_rtt is called with the raw target URL. But request_rtt (mod.rs:699) uses the URL as-is via reqwest, which does not handle user:pass@host in URLs in the same way a manual basic_auth call does.
mev_ping_test (line 287–294) correctly calls parse_endpoint_credentials and apply_basic_auth. mev_ping_measure_test skips this step entirely.
Impact: If a MEV relay endpoint contains embedded credentials (e.g. https://0x<pubkey>@relay.example.com), the PingMeasure test will either fail with a 401 or include credentials in the URL as plain text.
Fix this →

[Medium] Timeout in test_single_mev only emits one failure entry, not N

mev.rs:250–261 — On cancellation, only one TestResult::Fail is pushed for queued_tests[test_counter]. If multiple tests were still queued, the remaining ones (indices test_counter+1..) are silently dropped from the results.
Impact: Score calculation (via calculate_score) and result output will silently omit cancelled tests beyond the first, making the reported score look better than it is and breaking parity with the Go implementation which emits a failure for every remaining test.
Fix this →

[Medium] reqwest::Client is created per request, not reused

mev.rs:285, mev.rs:547, mev.rs:574, mev.rs:616 — A new reqwest::Client is instantiated for every HTTP call. The reqwest docs recommend creating a client once and reusing it, as it manages a connection pool internally.
Impact: Under a load test with --number-of-payloads > 1, connection pooling is never utilized; each round-trip re-establishes TCP connections. RTT measurements may be inflated by connection setup overhead, making results unreliable for CreateBlock.
Recommendation: Thread a shared reqwest::Client through TestMevArgs or pass it as a parameter to the helper functions.

[Low] wiremock removed from dev-dependencies without explanation

crates/cli/Cargo.toml diff — wiremock is removed from [dev-dependencies]. The PR description doesn't mention why.
This is fine if no existing tests in other modules depended on it; confirm no compile errors remain in the other test modules (beacon.rs, peers.rs, etc.).

[Low] create_mev_block bypasses epoch boundary refresh

mev.rs:387 — proposer_duties are fetched once per epoch. Inside create_mev_block, when MevError::StatusCodeNot200 triggers a slot advance (next_slot += 1, line 723), the epoch may roll over, but proposer_duties are not refreshed for the new epoch.
The fallback at line 681 (fetch_proposers_for_epoch) handles the case where the pubkey is not found, so this won't hard-fail — but it will make an extra network call per epoch boundary that crosses inside the retry loop.

[Low] Hardcoded signature in build_blinded_block_payload is intentional but undocumented

mev.rs:765 — The comment-less hardcoded sig_hex is a dummy BLS signature used so the relay returns 400 Bad Request (which request_rtt at line 755 treats as the expected status). This is correct behavior but should have a comment explaining why a dummy/invalid signature is intentional, to avoid future "cleanup" removing it.

Parity Matrix

Component	Go	Rust	Match	Notes
CLI flag `--endpoints`	present	present	yes
CLI flag `--beacon-node-endpoint`	present	present	yes
CLI flag `--load-test`	present	present	yes
CLI flag `--number-of-payloads`	present	present	yes
CLI flag `--x-timeout-ms`	present	present	yes
`Ping` test	present	present	yes
`PingMeasure` test	present	present	partial	credential stripping missing in Rust
`CreateBlock` test	present	present	yes
`format_mev_relay_name`	present	present	yes
Timeout/cancellation per test	emits N failures	emits 1 failure	no	remaining tests silently dropped
Client reuse	shared client	new per request	no	performance gap

Tests

Unit tests added:

test_format_mev_relay_name — covers standard, no-scheme, no-credential, and short-hash cases. Good coverage.
test_get_validator_pk_for_slot — covers found, adjacent, and not-found cases. Good coverage.

No integration tests (requiring live relay) — acceptable given the external dependency.

Clippy/fmt could not be verified (build tools not available in this environment).

Open Questions

Was wiremock removed because beacon.rs/peers.rs tests that used it are also being refactored in a follow-up, or were those tests already removed?
The request_rtt function (mod.rs:691) does not strip URL credentials — is this a known limitation of the shared helper, or should credential handling be added there?

- Follow the `test validator` approach (enum + impl block) - Simplify error handling for timeouts

- Test is already cancelled by caller appropriately

- Workaround for false positive

emlautarom1

I took the liberty of pushing some changes related to CancellationToken, timeouts and the like, that could cause trouble. I did not add them as comments due to the extent of the required changes.

There are a couple of things that require attention, like reusing existing error variants, mismatched features against v1.7.1 and missing tests.

emlautarom1 · 2026-04-03T21:03:56Z

+}
+
+#[cfg(test)]
+mod tests {


There are multiple tests missing from testmev_internal_test.go. Let us known if you're going to include them in this PR or you'll add them in a follow up.

The missing tests will be addressed in a follow-up PR to keep this one focused. We'll add the remaining testmev_internal_test.go cases there.

Follow-up PR created: #329

- Use tokio::time::Instant instead of std::time::Instant in async code - Replace CliError::Other string literals with typed variants TimeoutInterrupted and TestCaseNotSupported - Remove TestResultError from imports; use CliError::TimeoutInterrupted directly - Simplify latest_beacon_block and fetch_proposers_for_epoch to use ? operator, leveraging existing From impls on CliError - Simplify get_block_header error conversions with .map_err(MevError::from) and .map_err(|e| MevError::Cli(e.into())) Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

RUSTSEC-2026-0097: rand 0.8.x unsoundness (transitive dep) RUSTSEC-2026-0098: rustls-webpki URI name constraint bypass (transitive dep) Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

These features are absent from Charon v1.7.1: - Remove x-timeout-ms CLI flag and x_timeout_ms field from TestMevArgs - Remove X-Timeout-Ms and Date-Milliseconds headers from get_block_header - Remove credential parsing (parse_endpoint_credentials / apply_basic_auth) from mev_ping_test, latest_beacon_block, fetch_proposers_for_epoch, and get_block_header - Replace x_timeout_ms-based thresholds in mev_create_block_test with fixed 500ms/800ms constants matching Go reference thresholds Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

varex83agent

All actionable review comments have been addressed. The missing tests (thread #10) will be added in a follow-up PR.

Resolve conflicts: - deny.toml: adopt main's detailed reason comment for RUSTSEC-2026-0097, keep RUSTSEC-2026-0098 - crates/cli/Cargo.toml: add chrono from main - crates/cli/src/commands/test/mod.rs: accept main's refactor (helpers/constants modules) - crates/cli/src/commands/test/mev.rs: update imports for helpers/constants split, use SLOTS_IN_EPOCH.get() for NonZero<u64> Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

@claude-review

* ci: use local /review-pr skill for PR reviews Replace the external code-review plugin with the repo's own /review-pr skill, and upgrade pull-requests permission to write so the skill can post inline review comments via the GitHub API. Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com> * ci: trigger review via @claude-review comment Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com> * ci: use /claude-review as comment trigger Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com> * ci: remove claude-code reviewer trigger, keep only /claude-review comment Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com> --------- Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

* chore: fix cargo deny check Upgrade rustls-webpki 0.103.10 → 0.103.12 (RUSTSEC-2026-0098, RUSTSEC-2026-0099) and multihash 0.19.3 → 0.19.4 (drops yanked core2 0.4.0). * updated rand to 0.9.4 from 0.9.2 * updated other dependencies

…#333) (#334) Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

- Remove unnecessary clones

Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

…#333) (#334) Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

emlautarom1 · 2026-04-16T11:52:16Z

@varex83 I'm a bit confused as to why we have changes to .github/workflows/claude-code-review.yml in this PR. Looks like the agent is adding some additional commits that are not relevant.

emlautarom1

LGTM, let's sort out the agent stuff and then merge.

varex83 · 2026-04-16T11:54:18Z

@varex83 I'm a bit confused as to why we have changes to .github/workflows/claude-code-review.yml in this PR. Looks like the agent is adding some additional commits that are not relevant.

Most likely I've forgot to git stash while changing branches and accidentally pushed the wrong file

Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

PoulavBhowmick03 added 2 commits March 28, 2026 13:09

feat: add mev cli tests

88601b6

clippy

1583cc9

emlautarom1 added 8 commits April 3, 2026 17:08

Remove type aliases for test cases

e7f5fef

- Follow the `test validator` approach (enum + impl block) - Simplify error handling for timeouts

Use JoinSet

c65b0a2

Consistent argument ordering

5b4d55e

Remove select with CT

79dfc87

- Test is already cancelled by caller appropriately

Reorder parameters

dec0653

Remove redundant CancellationTokens

9706161

Pass main CT

9c690e3

Fix clippy lints

0c11a44

- Workaround for false positive

emlautarom1 requested changes Apr 3, 2026

View reviewed changes

varex83agent and others added 3 commits April 15, 2026 11:08

fix(deny): ignore RUSTSEC advisories for transitive deps

e961db0

RUSTSEC-2026-0097: rand 0.8.x unsoundness (transitive dep) RUSTSEC-2026-0098: rustls-webpki URI name constraint bypass (transitive dep) Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

varex83agent approved these changes Apr 15, 2026

View reviewed changes

varex83agent mentioned this pull request Apr 15, 2026

test(cli/mev): add integration tests for MEV test command #329

Open

4 tasks

varex83agent and others added 8 commits April 15, 2026 18:47

ci: fix permissions and pass GITHUB_TOKEN for review comments (#333)

dda7858

Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

chore: fix cargo deny check (#335)

3c8821d

* chore: fix cargo deny check Upgrade rustls-webpki 0.103.10 → 0.103.12 (RUSTSEC-2026-0098, RUSTSEC-2026-0099) and multihash 0.19.3 → 0.19.4 (drops yanked core2 0.4.0). * updated rand to 0.9.4 from 0.9.2 * updated other dependencies

revert: ci: fix permissions and pass GITHUB_TOKEN for review comments (…

d18ae87

…#333) (#334) Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

Revert deny ignore

2b8e09b

Pass CT as value instead of ref

5cb1dda

- Remove unnecessary clones

ci: fix permissions and pass GITHUB_TOKEN for review comments (#333)

be74869

Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

revert: ci: fix permissions and pass GITHUB_TOKEN for review comments (…

7363328

…#333) (#334) Co-authored-by: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

emlautarom1 approved these changes Apr 16, 2026

View reviewed changes

varex83agent and others added 2 commits April 16, 2026 13:55

chore: restore workflow file to match main branch

eb4d1b2

Co-Authored-By: Bohdan Ohorodnii <35969035+varex83@users.noreply.github.com>

fix: revert workflow file changes

3337bd5

varex83 merged commit 8836fb4 into main Apr 16, 2026
8 checks passed

varex83 deleted the poulav/mev_test branch April 16, 2026 15:23

Conversation

PoulavBhowmick03 commented Mar 28, 2026

Uh oh!

iamquang95 commented Apr 2, 2026

Uh oh!

claude bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review

Summary

Findings

Parity Matrix

Tests

Open Questions

Uh oh!

emlautarom1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emlautarom1 Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

varex83agent Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

varex83agent Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

varex83agent left a comment

Choose a reason for hiding this comment

Uh oh!

emlautarom1 commented Apr 16, 2026

Uh oh!

emlautarom1 left a comment

Choose a reason for hiding this comment

Uh oh!

varex83 commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

claude bot commented Apr 2, 2026 •

edited

Loading