fix(evmrpc): stream request-body budget charging to close slowloris gap (PLT-780)

[amir-deris]

Summary

Fixes PLT-780: the EVM HTTP requestSizeLimiter reserved global byte-budget capacity based on the declared Content-Length before the body was read, and held that reservation across the full request. A slow/stalled body (slowloris-style) could pin a large reservation cheaply, exhausting the shared budget and denying the endpoint — including for JWT-protected deployments, since the limiter sat outside the JWT check.

• Charge the global byte budget incrementally as body bytes are read (batched in 64 KiB steps) instead of trusting Content-Length/reserving maxBody up front. A stalled body now pins at most one batch, not its declared size.
• Add body_read_idle_timeout (default 10s): a per-chunk idle guard via http.ResponseController.SetReadDeadline, independent of the overall ReadTimeout, so a stalled read is cut and its reservation released quickly (HTTP 408).
• Reorder the HTTP middleware stack so JWT runs before requestSizeLimiter — unauthenticated clients get 401 without ever touching the shared budget.
• New metrics reasons budget_midread / slow_body on requestRejectedCount for the new rejection paths.

Fixes found while validating

• seiLegacyHTTPGate — always present in the production stack — buffers the whole body via its own io.ReadAll and, on a failed read, was writing its own 400 response with the raw internal error text before the limiter's outcome-based 429/408 could apply. captureResponseWriter now suppresses inner-handler writes once the limiter's outcome is set, so the documented status/message always reaches the client.
• That same gate closes the body immediately after buffering it, before dispatching to the real handler. Because releasing the budget was tied to Body.Close(), this freed the reservation the instant the body finished buffering rather than holding it for the whole request — dropping protection against concurrent slow processing of already-read bodies. Release is now decoupled from Close() and happens once, after the inner handler chain fully returns.

Test plan

• go test ./evmrpc/... and go test ./evmrpc/config/...
• go test ./evmrpc/ -race on the request-limiter/legacy-gate tests
• New regression tests: incremental budget charging, slowloris no longer pins declared size, body read idle timeout (408) + steady slow upload still succeeds, JWT runs before the byte limiter, and budget outcome survives being routed through seiLegacyHTTPGate
• gofmt -s -l / goimports -l clean on all touched files

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Jul 31, 2026, 3:26 PM

[codecov]

Codecov Report

❌ Patch coverage is 90.35088% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.41%. Comparing base (2d2628f) to head (c77c1b3).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
evmrpc/request_limiter.go	89.42%	8 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3836      +/-   ##
==========================================
- Coverage   61.28%   60.41%   -0.88%     
==========================================
  Files        2351     2259      -92     
  Lines      197283   186877   -10406     
==========================================
- Hits       120907   112894    -8013     
+ Misses      65530    63987    -1543     
+ Partials    10846     9996     -850     

Flag	Coverage Δ
sei-chain-pr	71.82% <90.35%> (?)
sei-db	70.41% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
evmrpc/config/config.go	96.90% <100.00%> (+0.04%)	⬆️
evmrpc/metrics.go	96.82% <ø> (ø)
evmrpc/rpcstack.go	79.32% <100.00%> (+0.28%)	⬆️
evmrpc/server.go	89.23% <100.00%> (+0.04%)	⬆️
evmrpc/request_limiter.go	90.59% <89.42%> (-1.41%)	⬇️

... and 93 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

PR Summary

Medium Risk
Changes public HTTP admission control and middleware order for JWT-protected RPC; behavior shifts for slow or concurrent large uploads but is aimed at closing a DoS vector rather than widening limits.

Overview
Closes a slowloris-style gap on EVM HTTP JSON-RPC: requestSizeLimiter no longer reserves max_concurrent_request_bytes from declared Content-Length before the body is read. It now charges the global budget in 64 KiB steps as bytes arrive, returns HTTP 429 when the budget is exhausted mid-read, and pins at most one batch on stalled uploads instead of the full declared size.

Adds body_read_idle_timeout (default 10s, evm.body_read_idle_timeout) with per-chunk idle deadlines via http.ResponseController, returning HTTP 408 and releasing held budget on stall. Rejection metrics gain budget_midread and slow_body on requestRejectedCount.

JWT authentication is moved ahead of requestSizeLimiter so unauthenticated clients get 401 without touching the shared byte budget. A captureResponseWriter ensures limiter 429/408 responses win over seiLegacyHTTPGate error writes, and byte budget is released only after the inner handler chain returns (not when the legacy gate closes the body after buffering).

^{Reviewed by Cursor Bugbot for commit c77c1b3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[seidroid]

The automated review did not complete; see the failing AI Review check for details.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c77c1b3. Configure here.}

[cursor]

Body idle deadline never cleared

High Severity

budgetBody.Read sets a connection SetReadDeadline before every chunk, but neither the EOF path nor Close resets it to zero. With the default 10s body_read_idle_timeout, that deadline stays armed through the rest of the request. Handlers that run longer than 10s after the body is consumed can have their request context cancelled (and keep-alive reuse disrupted), cutting off legitimate slow RPCs such as traces, estimates, and large log queries.

Additional Locations (1)

• evmrpc/request_limiter.go#L159-L169

 

^{Reviewed by Cursor Bugbot for commit c77c1b3. Configure here.}

[claude]

🔴 requestSizeLimiter.ServeHTTP now releases the global byte budget (budgetWrapped.Close()/release()) in plain sequential code after l.inner.ServeHTTP(cw, r) returns, instead of the old defer l.budget.Release(weight). A panic that escapes the inner handler chain (e.g. seiLegacyHTTPGate's JSON batch-merge logic, or the gzip/vhost/cors wrappers, none of which are covered by go-ethereum's per-method-callback recover) skips both cleanup calls, permanently leaking the reserved bytes from the shared max_concurrent_request_bytes semaphore. Repeated panics exhaust the budget and cause legitimate requests to be rejected with 429 until process restart. Fix: wrap the Close()/release() calls in a defer.

Extended reasoning...

The bug. Before this PR, requestSizeLimiter.ServeHTTP released the global byte-budget semaphore with defer l.budget.Release(weight), so the reservation was returned even if the inner handler panicked. This PR replaces incremental up-front reservation with per-chunk charging via budgetBody, and moves the release to plain sequential code in request_limiter.go:88-99:

l.inner.ServeHTTP(cw, r)

if budgetWrapped != nil {
    _ = budgetWrapped.Close()
    budgetWrapped.release()
}

There is no defer and no recover around these two calls. If l.inner.ServeHTTP panics, both Close() and release() are skipped entirely, and the panic unwinds straight past this frame.

Why the panic can actually reach here. The pinned go-ethereum fork's RPC server only recovers panics inside the per-method callback (rpc/service.go, callback.call) — that is, only once a JSON-RPC method handler is actually invoked. Everything upstream of that in the chain this PR wires — seiLegacyHTTPGate (which does its own io.ReadAll plus JSON batch-parsing/merging with slice/map indexing such as entries[idx]/used[idx], with no recover of its own), the gzip/vhost/cors middleware, and rpc.Server's own HTTP dispatch/decode path, are not covered by that recover. A panic anywhere in that surface propagates directly through requestSizeLimiter.ServeHTTP, which also has no recover, up to net/http's per-connection recover — the process survives, but the connection-level recover has no knowledge of the semaphore, so the reservation is never returned.

Why the leak is material, not just theoretical. seiLegacyHTTPGate buffers the whole body via io.ReadAll before it starts its batch-merge logic. budgetBody.Read flushes all unbilled bytes into the semaphore (b.reserved) on EOF, so by the time the gate's merge logic (or anything after the body read) runs, the full body's worth of bytes — up to maxBody (5 MiB by default) — is already charged to the process-global max_concurrent_request_bytes semaphore (128 MiB by default). A single panic there leaks up to 5 MiB; roughly 25 such panics would exhaust the entire budget, after which every legitimate request gets rejected with HTTP 429 until the process restarts. This is a persistent, resource-exhaustion DoS — notably the exact failure class this PR's own description says it is closing (the slowloris/budget-exhaustion gap).

Step-by-step proof:

1. Client sends a JSON-RPC batch request containing a gated sei_* method plus a malformed entry designed to trigger a panic in seiLegacyHTTPGate's merge logic (e.g. an out-of-range index reachable through its entries[idx]/used[idx] bookkeeping) — or any other panic reachable in the gzip/vhost/cors/dispatch layers.
2. requestSizeLimiter.ServeHTTP wraps r.Body in budgetBody, which charges bytes to l.budget (the shared semaphore) as the gate reads the body via io.ReadAll, and flushes the remainder into b.reserved on EOF — up to the full body size.
3. l.inner.ServeHTTP(cw, r) is called, which invokes the gate; the gate's merge logic panics.
4. The panic unwinds through requestSizeLimiter.ServeHTTP without executing budgetWrapped.Close()/release(), because there is no defer.
5. The panic is finally caught by net/http's connection-level recover; the connection is dropped, but l.budget's Weighted semaphore still shows the bytes from step 2 as acquired — permanently, for the life of the process.
6. Repeating steps 1-5 ~25 times (5 MiB each against a 128 MiB budget) exhausts max_concurrent_request_bytes entirely; all subsequent legitimate JSON-RPC requests are rejected with HTTP 429 until the node operator restarts the process.

Why nothing else prevents this. budgetBody.fail() handles the ordinary error paths (budget exhaustion mid-read, idle timeout) by calling b.release() itself, but that only covers errors returned from Read() — not a panic thrown by code that has already consumed the body downstream. The per-method recover() wrappers used elsewhere in evmrpc/*.go (e.g. in tracers.go, send.go, block.go) only protect the individual RPC method being called; they don't run for panics in the HTTP-layer plumbing this PR touches.

The fix is exactly what the old code already did: wrap the cleanup in a defer so the budget is released unconditionally, regardless of whether l.inner.ServeHTTP returns normally or panics, e.g. defer func() { _ = budgetWrapped.Close(); budgetWrapped.release() }() placed immediately after budgetWrapped is constructed.

[claude]

🔴 The SetReadDeadline error branch in budgetBody.Read (evmrpc/request_limiter.go:127-131) is an empty if block containing only a comment, which trips staticcheck's SA9003 check and fails make lint / the golangci CI workflow as committed. Fix by giving the branch an actual no-op statement (e.g. logging the error) instead of just a comment.

Extended reasoning...

What the bug is. In budgetBody.Read (evmrpc/request_limiter.go), the idle-deadline guard is:

if b.idleTimeout > 0 && b.rc != nil {
    if err := b.rc.SetReadDeadline(time.Now().Add(b.idleTimeout)); err != nil {
        // ResponseController may be unavailable on exotic ResponseWriters; proceed
        // without the idle guard rather than failing the request.
    }
}

The inner if err != nil { ... } branch contains nothing but a comment. staticcheck's SA9003 rule treats a comment as a non-statement, so from the analyzer's point of view this branch is empty — a classic "did you forget to handle this error" pattern, and the check fires regardless of the intent conveyed in the comment.

How it manifests / proof. I reproduced this directly against the repo's own toolchain:

$ golangci-lint --version
golangci-lint has version 2.5.0 built with go1.25.1 ...
$ golangci-lint run --enable-only=staticcheck ./evmrpc/
evmrpc/request_limiter.go:128:3: SA9003: empty branch (staticcheck)
                if err := b.rc.SetReadDeadline(time.Now().Add(b.idleTimeout)); err != nil {
                ^
1 issues:
* staticcheck: 1

This is the exact line the PR adds as part of the new budgetBody type backing the streamed body-budget charging feature.

Why nothing else in the repo prevents it. .golangci.yml sets default: none and then explicitly re-enables staticcheck in linters.enable, with no per-check staticcheck.checks override that would disable SA9003, and no path-based exclusion covering request_limiter.go (only generated *.pb.go files and one gosec weak-RNG rule are excluded). request_limiter.go is ordinary non-test source, so run.tests: false in the config doesn't exempt it. AGENTS.md/CLAUDE.md state that make lint (which runs golangci-lint per .golangci.yml) must be clean, and .github/workflows/golangci.yml runs the same golangci-lint version in CI. So this is a genuine, reproducible CI-blocking failure introduced by this PR, not a style nit that a reviewer could wave through.

Impact. As committed, make lint fails locally and the golangci-lint GitHub Actions check fails on the PR, blocking merge until fixed. It's purely a lint/CI-gate issue — there's no runtime behavior bug here (the surrounding comment's stated intent, "proceed without the idle guard rather than failing the request," is a reasonable behavior and doesn't need to change).

Fix. Give the branch a real no-op statement instead of only a comment, e.g.:

if err := b.rc.SetReadDeadline(time.Now().Add(b.idleTimeout)); err != nil {
    logger.Debug("SetReadDeadline unavailable for body read idle timeout", "err", err)
}

or equivalent (_ = err plus a comment, or restructuring to drop the nested if entirely). Either resolves SA9003 while preserving the existing fallback behavior.

[claude]

🔴 budgetBody.Read resets the socket's read deadline to now+body_read_idle_timeout (default 10s) on every chunk via http.ResponseController.SetReadDeadline, which writes directly to the same underlying connection deadline that net/http arms once as the absolute requestStart+ReadTimeout (default 30s) bound for the whole request body read.

Extended reasoning...

budgetBody.Read (evmrpc/request_limiter.go:126-133) calls rc.SetReadDeadline(time.Now().Add(idleTimeout)) unconditionally on every single Read invocation whenever bodyReadIdleTimeout > 0 — which it is by default (BodyReadIdleTimeout: 10 * time.Second in config.go). http.ResponseController.SetReadDeadline is not a separate, layered "idle" deadline; per the Go 1.25.6 net/http source (server.go:499-500), (*response).SetReadDeadline calls c.conn.rwc.SetReadDeadline directly — the exact same socket-level deadline that net/http's readRequest arms once, after headers are parsed, to wholeReqDeadline = t0.Add(Server.ReadTimeout) (server.go:1018-1086). That absolute deadline exists specifically to bound "the maximum duration for reading the entire request, including the body" (the documented semantics of ReadTimeout, echoed in this repo's own evmrpc/config/config.go doc-comment).

Because every chunk read re-arms the deadline to now + idleTimeout instead of computing min(now+idleTimeout, wholeReqDeadline), a client that sends any nonzero amount of body data at an interval shorter than body_read_idle_timeout (e.g. 1 byte every 9 seconds, since the default idle timeout is 10s) keeps pushing the deadline forward indefinitely. Neither guard ever fires: the idle timeout never sees a >10s gap, and the absolute 30s ReadTimeout it silently overwrote can never trip because it no longer exists on the socket. The read is then bounded only by MaxBytesReader's total byte cap (5 MiB by default), which at a 1-byte/9s trickle is on the order of hundreds of days of wall-clock time — effectively unbounded. For that entire duration the connection, its serving goroutine, and one of the max_open_connections slots (default 2000) are pinned.

This is a straightforward regression in the exact direction opposite to what the PR sets out to fix: PLT-780 closed a slowloris gap where a stalled body could pin a large byte-budget reservation cheaply. This change reintroduces a connection/goroutine-lifetime slowloris vector — one that the pre-existing ReadTimeout used to close at a hard 30s ceiling for the body-read phase, and which is now open again, active by default, even in configurations where the byte budget itself is disabled (the wrapping only requires bodyReadIdleTimeout > 0, independent of budget != nil). There is no other mitigation in the stack: per-IP rate limiting isn't wired into this HTTP middleware chain, and even if it were, a rate limiter bounds request arrival rate, not the lifetime of a single already-admitted connection.

The PR's own new tests do not catch this because they build their servers with httptest.NewServer, which never sets http.Server.ReadTimeout. With no absolute deadline present in the test harness, there is nothing for the per-chunk reset to clobber, so TestRequestSizeLimiter_bodyReadIdleTimeout and TestRequestSizeLimiter_steadySlowUploadSucceeds both pass despite the regression. In production, rpcstack.go's Start() always sets http.Server{ReadTimeout: h.timeouts.ReadTimeout}, and CheckTimeouts (evmrpc/endpoints.go) guarantees that value is never below 1s, sanitizing it up to the 30s go-ethereum default when unset — so the absolute deadline is always present in the real server.

Proof walkthrough:

1. Server starts with ReadTimeout = 30s (default) and BodyReadIdleTimeout = 10s (default).
2. A malicious client opens a POST with Content-Length: 5000000 (near the 5 MiB cap) and sends only 1 byte, then goes idle for 9 seconds, then sends 1 more byte, repeating indefinitely.
3. At t=0, after headers are read, net/http arms wholeReqDeadline = t0 + 30s on the socket.
4. At t≈0, requestSizeLimiter wraps the body in budgetBody; the inner handler calls Read, which first calls rc.SetReadDeadline(now + 10s) — overwriting the socket deadline from t0+30s down to t0+10s (this alone already shortens the "whole request" cap, but at least still bounds it).
5. At t≈9s, the 1-byte chunk arrives, Read returns 1 byte, and the next Read call immediately resets the deadline to t≈9s + 10s = t≈19s.
6. This repeats every ~9s forever: t=9s → deadline 19s, t=18s → deadline 28s, t=27s → deadline 37s (already past the original 30s absolute cutoff, which no longer exists), and so on without bound.
7. The connection survives past the documented 30s ReadTimeout and continues indefinitely, limited only by the ~5 MiB MaxBytesReader cap — at 1 byte/9s that is roughly 5×10^6 × 9s ≈ 1.4 years of wall-clock time to exhaust, i.e., practically unbounded for operational purposes.

Fix: compute the idle deadline as min(now+idleTimeout, wholeReqDeadline) rather than unconditionally overwriting the socket deadline forward. Since wholeReqDeadline isn't directly exposed via ResponseController, an equivalent fix is to track an absolute deadline computed once when the wrapped body is constructed (readStart + ReadTimeout, threaded in from config) and cap each per-chunk SetReadDeadline call at min(now+idleTimeout, absoluteDeadline).