Skip to content
52 changes: 36 additions & 16 deletions evmrpc/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -250,19 +250,26 @@ type Config struct {
// batched JSON-RPC call (HTTP and WebSocket). Set to 0 to disable the limit.
BatchResponseMaxSize int `mapstructure:"batch_response_max_size"`

// MaxRequestBodyBytes is the maximum size, in bytes, of a single HTTP
// JSON-RPC request body. Requests larger than this are rejected (HTTP 413)
// before the body is buffered or JSON-decoded. 0 uses the go-ethereum
// default (5 MiB).
// MaxRequestBodyBytes is the maximum size, in bytes, of a single HTTP (:8545)
// or WebSocket (:8546) JSON-RPC request/frame. HTTP requests larger than this
// are rejected (HTTP 413) before the body is buffered or JSON-decoded.
// WebSocket frames exceeding this limit are rejected by the read loop.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] "WebSocket frames exceeding this limit are rejected by the read loop" understates what happens. A frame over the gorilla read limit fails the read, which terminates the read loop and closes the connection (close 1009) — the client loses all in-flight requests and any active eth_subscribe streams, not just the oversized frame. Since operators tune this value to avoid outages, the doc should say the connection is closed.

// 0 uses the go-ethereum default (5 MiB).
MaxRequestBodyBytes int64 `mapstructure:"max_request_body_bytes"`

// MaxConcurrentRequestBytes bounds the total size, in bytes, of HTTP
// JSON-RPC request bodies admitted for processing concurrently, weighted by
// each request's Content-Length. Requests that would exceed the budget are
// rejected fast (HTTP 429) before decode, capping peak memory under load.
// Set to 0 to disable the limit.
// MaxConcurrentRequestBytes bounds the total size, in bytes, of HTTP and
// WebSocket JSON-RPC request bodies admitted for processing concurrently
// (independent budgets per protocol). HTTP uses Content-Length weighting and
// rejects over-budget requests fast (HTTP 429). WebSocket blocks until

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] "bounds the total size, in bytes, of HTTP and WebSocket JSON-RPC request bodies admitted for processing concurrently" reads as size-weighted on both planes, but only HTTP is ("HTTP uses Content-Length weighting"). Per the accounting question on rpcstack.go:392, the WS side looks like it reserves the full frame cap per pending read, making it a concurrency limiter rather than a byte budget. Operators tuning this will mis-predict WS behavior. Please state the WS semantics explicitly - including the derived concurrency cap (budget / max_request_body_bytes) and the wait-timeout duration - here and in the app.toml template text below.

// budget frees or WSAdmissionTimeout elapses. Set to 0 to disable the limit
// on either protocol.
MaxConcurrentRequestBytes int64 `mapstructure:"max_concurrent_request_bytes"`

// WSAdmissionTimeout bounds how long a WebSocket connection waits for
// concurrent-byte budget to free before the next frame is read or committed.
// Zero or negative values use the go-ethereum default (30s).
WSAdmissionTimeout time.Duration `mapstructure:"ws_admission_timeout"`

// MaxOpenConnections caps the number of simultaneously accepted connections
// on the EVM HTTP and WebSocket listeners. The limit is applied per listener
// (HTTP and WS each get their own budget). Excess connections block in the
Expand Down Expand Up @@ -326,6 +333,7 @@ var DefaultConfig = Config{
BatchResponseMaxSize: 25 * 1000 * 1000, // 25MB
MaxRequestBodyBytes: 5 * 1024 * 1024, // 5 MiB (matches go-ethereum rpc default body limit)
MaxConcurrentRequestBytes: 128 * 1024 * 1024, // 128 MiB of request bodies admitted concurrently
WSAdmissionTimeout: 30 * time.Second, // matches go-ethereum rpc defaultWSAdmissionTimeout
MaxOpenConnections: 2000,
}

Expand Down Expand Up @@ -381,6 +389,7 @@ const (
flagBatchResponseMaxSize = "evm.batch_response_max_size"
flagMaxRequestBodyBytes = "evm.max_request_body_bytes"
flagMaxConcurrentRequestBytes = "evm.max_concurrent_request_bytes"
flagWSAdmissionTimeout = "evm.ws_admission_timeout"
flagMaxOpenConnections = "evm.max_open_connections"
)

Expand Down Expand Up @@ -651,6 +660,11 @@ func ReadConfig(opts servertypes.AppOptions) (Config, error) {
return cfg, err
}
}
if v := opts.Get(flagWSAdmissionTimeout); v != nil {
if cfg.WSAdmissionTimeout, err = cast.ToDurationE(v); err != nil {
return cfg, err
}
}
if v := opts.Get(flagMaxOpenConnections); v != nil {
if cfg.MaxOpenConnections, err = cast.ToIntE(v); err != nil {
return cfg, err
Expand Down Expand Up @@ -916,17 +930,23 @@ batch_request_limit = {{ .EVM.BatchRequestLimit }}
# batched JSON-RPC call (HTTP and WebSocket). Set to 0 to disable the limit.
batch_response_max_size = {{ .EVM.BatchResponseMaxSize }}

# max_request_body_bytes is the maximum size, in bytes, of a single HTTP
# JSON-RPC request body. Larger requests are rejected (HTTP 413) before the body
# is buffered or JSON-decoded. Set to 0 to use the default (5 MiB).
# max_request_body_bytes is the maximum size, in bytes, of a single HTTP (:8545)
# or WebSocket (:8546) JSON-RPC request/frame. HTTP larger requests are rejected
# (HTTP 413) before decode; WS frames exceeding this limit are rejected by the
# read loop. Set to 0 to use the default (5 MiB).
max_request_body_bytes = {{ .EVM.MaxRequestBodyBytes }}

# max_concurrent_request_bytes bounds the total size, in bytes, of HTTP JSON-RPC
# request bodies admitted for processing concurrently (weighted by each request's
# Content-Length). Requests that would exceed the budget are rejected fast
# (HTTP 429) before decode, capping peak memory under load. Set to 0 to disable.
# max_concurrent_request_bytes bounds total request bytes admitted concurrently
# on HTTP (:8545) and WebSocket (:8546) (independent budgets per protocol). HTTP
# rejects over-budget requests fast (HTTP 429); WS blocks until budget frees or
# ws_admission_timeout elapses. Set to 0 to disable on either protocol.
max_concurrent_request_bytes = {{ .EVM.MaxConcurrentRequestBytes }}

# ws_admission_timeout bounds how long a WebSocket connection waits for
# concurrent-byte budget before the next frame is read or committed. Zero or
# negative values use the go-ethereum default (30s).
ws_admission_timeout = "{{ .EVM.WSAdmissionTimeout }}"

# max_open_connections caps the number of simultaneously accepted connections on
# the EVM HTTP and WebSocket listeners. Set to 0 to disable the limit.
max_open_connections = {{ .EVM.MaxOpenConnections }}
Expand Down
8 changes: 8 additions & 0 deletions evmrpc/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ type opts struct {
batchResponseMaxSize interface{}
maxRequestBodyBytes interface{}
maxConcurrentRequestBytes interface{}
wsAdmissionTimeout interface{}
maxOpenConnections interface{}
maxTraceStructLogBytes interface{}
traceAllowedTracers interface{}
Expand Down Expand Up @@ -189,6 +190,9 @@ func (o *opts) Get(k string) interface{} {
if k == "evm.max_concurrent_request_bytes" {
return o.maxConcurrentRequestBytes
}
if k == "evm.ws_admission_timeout" {
return o.wsAdmissionTimeout
}
if k == "evm.max_open_connections" {
return o.maxOpenConnections
}
Expand Down Expand Up @@ -252,6 +256,7 @@ func getDefaultOpts() opts {
25 * 1000 * 1000,
int64(5 * 1024 * 1024),
int64(128 * 1024 * 1024),
30 * time.Second,
2000,
uint64(256 * 1024 * 1024),
[]string{"callTracer", "prestateTracer"},
Expand Down Expand Up @@ -519,15 +524,18 @@ func TestReadConfigRequestSizeLimits(t *testing.T) {
require.NoError(t, err)
require.Equal(t, config.DefaultConfig.MaxRequestBodyBytes, cfg.MaxRequestBodyBytes)
require.Equal(t, config.DefaultConfig.MaxConcurrentRequestBytes, cfg.MaxConcurrentRequestBytes)
require.Equal(t, config.DefaultConfig.WSAdmissionTimeout, cfg.WSAdmissionTimeout)

// Custom values (including 0 to use default / disable) flow through.
o := getDefaultOpts()
o.maxRequestBodyBytes = int64(1024)
o.maxConcurrentRequestBytes = int64(0)
o.wsAdmissionTimeout = 10 * time.Second
cfg, err = config.ReadConfig(&o)
require.NoError(t, err)
require.Equal(t, int64(1024), cfg.MaxRequestBodyBytes)
require.Equal(t, int64(0), cfg.MaxConcurrentRequestBytes)
require.Equal(t, 10*time.Second, cfg.WSAdmissionTimeout)
}

func TestReadConfigMaxOpenConnections(t *testing.T) {
Expand Down
19 changes: 14 additions & 5 deletions evmrpc/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ const (
errorClassKey = "error_class"
jsonrpcCodeKey = "jsonrpc_code"
rejectReasonKey = "reason"
protocolKey = "protocol"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Label-name inconsistency: this introduces protocol="http"|"ws", while the sibling metric in this repo already uses plane for the same concept (ratelimiter/registry.go:102rpc_rate_limit_rejected_total{plane}). The PR title/description and the ratelimiter doc tweak in this diff also both say "plane". Pick one name for the dimension across evmrpc metrics so dashboards can join on it.

protocolHTTP = "http"
protocolWS = "ws"
// reject reason values for requestRejectedCount.
rejectReasonOversize = "oversize" // body exceeded max_request_body_bytes
rejectReasonBusy = "busy" // max_concurrent_request_bytes budget exhausted
Expand Down Expand Up @@ -76,7 +79,7 @@ var (
)),
requestRejectedCount: must(rpcTelemetryMeter.Int64Counter(
"evmrpc_requests_rejected_total",
metric.WithDescription("Number of HTTP JSON-RPC requests rejected by pre-decode admission control (labeled by reason)"),
metric.WithDescription("Number of JSON-RPC requests rejected by admission control (labeled by protocol and reason)"),
metric.WithUnit("{count}"),
)),
}
Expand Down Expand Up @@ -166,13 +169,19 @@ func recordHistoricalDebugTraceAttempt(ctx context.Context, endpoint, connection
)
}

// recordRequestRejected counts an HTTP JSON-RPC request dropped by pre-decode
// admission control. reason is one of rejectReasonOversize / rejectReasonBusy.
// No endpoint dimension is recorded: the rejection happens before the JSON-RPC
// method is decoded, so it is not yet known.
func recordRequestRejected(ctx context.Context, reason string) {
metrics.requestRejectedCount.Add(ctx, 1,
metric.WithAttributes(
attribute.String(protocolKey, protocolHTTP),
attribute.String(rejectReasonKey, reason),
),
)
}

func recordWSAdmissionRejected(ctx context.Context, reason string) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The new function has no doc comment, and the diff also deletes the useful comment that used to sit on recordRequestRejected (which reason values are valid, and why there's no endpoint dimension). Since the two functions now differ only by the plane attribute, consider collapsing them into one recordRequestRejected(ctx, plane, reason) and keeping that explanation.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Two doc issues in this hunk. (1) The doc comment removed from recordRequestRejected carried non-obvious information (the enumerated reason values and why there is no endpoint dimension); the new recordWSAdmissionRejected has no comment at all. Please restore/adapt both, matching the commenting density of the rest of this file. (2) reason values now come from two disjoint vocabularies — oversize/busy on HTTP vs the fork's rpc.WSAdmissionReason* strings on WS — while the comment at line 24 still presents oversize/busy as the reject-reason values for this counter. Either map the fork reasons onto the existing constants (preferred: one vocabulary keeps dashboards plane-agnostic) or document the WS values next to line 24.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Two things here:

  1. reason now carries disjoint value sets per plane — HTTP emits oversize/busy from the local constants, while WS passes the fork's string through verbatim (rpc.WSAdmissionReasonBudgetWaitTimeout). Querying evmrpc_requests_rejected_total by reason alone becomes plane-dependent, and any future fork-side reason value silently appears as a new label value. Consider mapping fork reasons onto the existing rejectReason* constants (budget exhaustion is conceptually busy), or documenting the full union of values next to them.
  2. The doc comment removed from recordRequestRejected carried real information ("No endpoint dimension is recorded: the rejection happens before the JSON-RPC method is decoded") that still applies to both recorders. Worth restoring on one of them rather than dropping; recordWSAdmissionRejected currently has no comment at all.

Related: the PR description says rejections on either plane are recorded, but WS oversize frames are dropped by the gorilla read limit, which likely never reaches this admission hook — so plane="ws", reason="oversize" may be unreachable. Worth confirming and adjusting the description if so.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Two things here:

  1. The reason label now carries two disjoint vocabularies. HTTP emits the constants documented at lines 24-26 (oversize, busy); WS emits whatever the fork's hook passes, which is budget_wait_timeout / frame_admission_timeout (and only on context.DeadlineExceeded — see fireAdmissionEventOnBudgetTimeout in the fork). Neither WS value is listed in that const block, and WS never emits oversize, which the new max_request_body_bytes doc comment implies it does. Worth documenting the per-protocol reason sets next to the constants.
  2. The old recordRequestRejected doc comment explaining why there is no endpoint dimension (rejection happens pre-decode) was dropped. Consider keeping one comment above the pair covering both the protocol values and that rationale.

metrics.requestRejectedCount.Add(ctx, 1,
metric.WithAttributes(
attribute.String(protocolKey, protocolWS),
attribute.String(rejectReasonKey, reason),
),
)
Comment on lines 169 to 187

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The WebSocket admission-rejection path forwards raw fork-defined reason strings (e.g. budget_wait_timeout, frame_admission_timeout) into evmrpc_requests_rejected_total's reason label, while the HTTP path and the still-present doc comment only ever use oversize/busy. As a result, any alert or dashboard filtering on reason="busy" (even scoped to plane="ws") will silently miss WS-side budget-exhaustion rejections. Consider mapping the WS reasons onto the existing oversize/busy vocabulary before recording, or updating the doc comment to enumerate the WS-specific values.

Extended reasoning...

recordWSAdmissionRejected (evmrpc/metrics.go:180-187) forwards whatever reason string the vendored go-ethereum fork passes to SetWSAdmissionEventHook straight into the reason attribute of evmrpc_requests_rejected_total. Per the vendored fork (github.com/sei-protocol/go-ethereum@v1.15.7-sei-18, rpc/handler.go), the only values that hook ever fires with are rpc.WSAdmissionReasonBudgetWaitTimeout ("budget_wait_timeout") and rpc.WSAdmissionReasonFrameAdmissionTimeout ("frame_admission_timeout"). Meanwhile the HTTP path, recordRequestRejected in the same file, always records one of the two package constants rejectReasonOversize ("oversize") or rejectReasonBusy ("busy").

The const-block comment directly above those two constants (evmrpc/metrics.go:23-26, "reject reason values for requestRejectedCount") was written when the counter was HTTP-only and still reads as if oversize/busy are the only reason values the metric can take. This PR extends the same counter to the WS plane (via the new plane label) but doesn't reconcile the reason vocabulary between the two planes, and doesn't update or extend that comment to mention the WS-specific reason strings.

Why existing code doesn't prevent it: nothing type-checks or validates the reason string passed into recordRequestRejected/recordWSAdmissionRejected against the documented constants — both functions just accept an arbitrary string and record it as an attribute. The WS hook is wired directly to the fork's reason constants with no translation layer.

Step-by-step proof:

  1. Operator sets up an alert/dashboard query like sum(rate(evmrpc_requests_rejected_total{reason="busy"}[5m])) > 0, intending to catch budget-exhaustion rejections on either plane (optionally scoped further with plane="ws").
  2. A WS client saturates max_concurrent_request_bytes; the fork's admission logic times out waiting for budget and invokes the hook registered in EnableWS (evmrpc/rpcstack.go:392-396) with reason rpc.WSAdmissionReasonBudgetWaitTimeout = "budget_wait_timeout".
  3. recordWSAdmissionRejected records evmrpc_requests_rejected_total{plane="ws", reason="budget_wait_timeout"} — note the value is "budget_wait_timeout", not "busy".
  4. The alert/dashboard query filtering on reason="busy" never matches this data point, even though it is functionally the WS-plane analog of an HTTP busy rejection (budget exhaustion), so the operator never fires the alert they built for exactly this condition.

Impact: this is purely an observability/metrics-contract inconsistency — the counter still increments correctly with a plane label, and admission control itself functions as intended (no incorrect request handling, crash, or data loss). It only affects anyone who queries evmrpc_requests_rejected_total by reason expecting the two documented values to cover both planes. The fix is either (a) map the fork's WS reason strings onto rejectReasonBusy/rejectReasonOversize before calling recordWSAdmissionRejected, unifying the vocabulary across planes, or (b) update the doc comment at metrics.go:23-26 to enumerate the WS-specific reason values alongside oversize/busy so dashboard authors know to account for them.

Expand Down
11 changes: 8 additions & 3 deletions evmrpc/request_limiter.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ import (
// max_request_body_bytes is left at 0 ("use the default").
const defaultMaxRequestBodyBytes int64 = 5 * 1024 * 1024

func effectiveMaxRequestBodyBytes(max int64) int64 {
if max <= 0 {
return defaultMaxRequestBodyBytes
}
return max
}

// requestSizeLimiter is an HTTP middleware that bounds peak decode-time memory by
// admitting JSON-RPC requests *before* the body is buffered or decoded. It enforces:
//
Expand All @@ -34,9 +41,7 @@ type requestSizeLimiter struct {
// global budget. If a positive budget is smaller than maxBody it is raised to maxBody
// so that a single maximum-size request can always be admitted.
func newRequestSizeLimiter(inner http.Handler, maxBody, maxConcurrentBytes int64) http.Handler {
if maxBody <= 0 {
maxBody = defaultMaxRequestBodyBytes
}
maxBody = effectiveMaxRequestBodyBytes(maxBody)
l := &requestSizeLimiter{inner: inner, maxBody: maxBody}
if maxConcurrentBytes > 0 {
if maxConcurrentBytes < maxBody {
Expand Down
1 change: 1 addition & 0 deletions evmrpc/request_limiter_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ func TestRequestSizeLimiter(t *testing.T) {
})

t.Run("zero maxBody uses default cap", func(t *testing.T) {
require.Equal(t, defaultMaxRequestBodyBytes, effectiveMaxRequestBodyBytes(0))
l, ok := newRequestSizeLimiter(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}), 0, 0).(*requestSizeLimiter)
require.True(t, ok)
require.Equal(t, defaultMaxRequestBodyBytes, l.maxBody)
Expand Down
22 changes: 18 additions & 4 deletions evmrpc/rpcstack.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,10 @@ type HTTPConfig struct {

// WsConfig is the JSON-RPC/Websocket configuration
type WsConfig struct {
Origins []string
Modules []string
prefix string // path prefix on which to mount ws handler
Origins []string
Modules []string
prefix string // path prefix on which to mount ws handler
wsAdmissionTimeout time.Duration
RPCEndpointConfig
}

Expand Down Expand Up @@ -381,7 +382,20 @@ func (h *HTTPServer) EnableWS(apis []rpc.API, config WsConfig) error {
// Create RPC server and handler.
srv := rpc.NewServer()
srv.SetBatchLimits(config.batchItemLimit, config.batchResponseSizeLimit)
srv.SetReadLimits(config.readLimit)
readLimit := effectiveMaxRequestBodyBytes(config.readLimit)
if readLimit > math.MaxInt {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This clamp looks copied from the HTTP path (rpcstack.go:331-337), where it's needed because SetHTTPBodyLimit takes an int. Here readLimit is passed to SetReadLimits as an int64 with no conversion, so on 64-bit math.MaxInt == math.MaxInt64 and the branch is dead, while on a 32-bit build it would silently shrink an operator-configured limit. Consider dropping it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This clamp is dead code on 64-bit (math.MaxInt == math.MaxInt64), and unlike the SetHTTPBodyLimit case at line 332 there's no int conversion that requires it — SetReadLimits already takes int64. On 32-bit it would silently lower the operator's configured limit with no log line. Suggest dropping it, or adding a comment if it's deliberate 32-bit defensiveness.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Unlike the HTTP path (line 331-335), which needs this clamp because SetHTTPBodyLimit takes an int, SetReadLimits accepted an int64 directly before this change. If that signature is still int64, this branch is dead on 64-bit (math.MaxInt == math.MaxInt64) and only silently lowers a >2 GiB configured limit on 32-bit. Either drop it or add a one-line note on why it mirrors the HTTP clamp.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] On 64-bit platforms math.MaxInt == math.MaxInt64, so this comparison on an int64 is always false and the clamp is dead code. SetReadLimits already accepted int64 before this change (srv.SetReadLimits(config.readLimit)). If this is deliberate 32-bit safety, a short comment saying so would help; otherwise drop it. Note that silently clamping an operator-configured value is also questionable - a startup config validation error is friendlier than a value that quietly differs from app.toml.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This clamp is dead code: the fork's signature is func (s *Server) SetReadLimits(limit int64), and on 64-bit platforms math.MaxInt == math.MaxInt64, so the branch is unreachable. On a 32-bit build it would silently shrink a legitimately-configured limit for no reason, since the fork stores it as int64 throughout. Suggest dropping lines 386-388.

readLimit = math.MaxInt
}
srv.SetReadLimits(readLimit)
Comment on lines 384 to +389

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The if readLimit > math.MaxInt clamp on the WS read limit in EnableWS (evmrpc/rpcstack.go:383-388) is dead code on every 64-bit build this repo ships, since math.MaxInt == math.MaxInt64 and SetReadLimits already takes an int64 (unlike SetHTTPBodyLimit(int), which genuinely needs the analogous narrowing clamp). On a hypothetical 32-bit build it would silently lower an operator-configured >2GiB limit with no log line. Suggest dropping it, or adding a one-line comment noting it's deliberate 32-bit defensiveness.

Extended reasoning...

What the code does. In EnableWS (evmrpc/rpcstack.go:383-388), the read limit for the WebSocket JSON-RPC server is computed and clamped like this:

readLimit := effectiveMaxRequestBodyBytes(config.readLimit)
if readLimit > math.MaxInt {
    readLimit = math.MaxInt
}
srv.SetReadLimits(readLimit)

This mirrors the pattern used a few lines earlier in EnableRPC for the HTTP body limit:

bodyLimit := config.maxRequestBodyBytes
if bodyLimit > math.MaxInt {
    bodyLimit = math.MaxInt
}
srv.SetHTTPBodyLimit(int(bodyLimit))

Why the HTTP clamp is load-bearing but the WS clamp is not. SetHTTPBodyLimit takes a plain int, so the int64 bodyLimit must be narrowed with an explicit int(...) conversion — without the preceding clamp, a value above math.MaxInt would silently wrap on a 32-bit int platform. SetReadLimits, however, already takes an int64 directly (verified against the vendored fork, github.com/sei-protocol/go-ethereum@v1.15.7-sei-18, rpc/server.go:105: func (s *Server) SetReadLimits(limit int64)). No narrowing conversion occurs, so the clamp guards against nothing on the WS path.

Why this is dead code today. readLimit is an int64, and math.MaxInt is defined as 1<<(intSize-1) - 1, which equals math.MaxInt64 on every 64-bit GOARCH this repo builds and ships for (amd64, arm64, etc.). Since effectiveMaxRequestBodyBytes returns at most whatever an operator configures (or the 5 MiB default), and no realistic config value exceeds math.MaxInt64, the condition readLimit > math.MaxInt can never evaluate true on any binary this repo actually produces. The branch is unreachable dead code in practice.

Step-by-step proof:

  1. Assume a 64-bit build target (the only kind this repo ships). math.MaxInt resolves at compile time to 9223372036854775807, identical to math.MaxInt64.
  2. readLimit is declared as int64 (via effectiveMaxRequestBodyBytes, whose parameter and return type are both int64).
  3. The maximum possible value readLimit can hold is math.MaxInt64 itself (the type's own ceiling) — it can never exceed it.
  4. Therefore readLimit > math.MaxInt (i.e. readLimit > math.MaxInt64) is always false; the assignment readLimit = math.MaxInt never executes.
  5. On a hypothetical 32-bit build, math.MaxInt would instead equal math.MaxInt32 (~2.1 GiB), and the branch would become reachable — silently truncating any operator-configured max_request_body_bytes above ~2 GiB down to ~2 GiB, with no log line to surface that the operator's intended value was altered.

Impact. None on any binary this repository currently produces — this is purely dead code on 64-bit. The only theoretical exposure is on a 32-bit build, where a large max_request_body_bytes setting would be silently reduced without any warning, but Sei does not appear to target 32-bit deployments.

Fix. Either drop the clamp entirely (matching that SetReadLimits never needed narrowing in the first place), or, if 32-bit defensiveness is intentional, add a one-line comment explaining that it mirrors the HTTP path's narrowing requirement even though SetReadLimits itself doesn't need it, so a future reader doesn't mistake it for a copy-paste artifact. This is a small clarity/dead-code cleanup, not a functional defect — no behavior changes on any build this repo ships.

// maxConcurrentRequestBytes is passed through raw; rpc.Server.recomputeWSConcurrentBudget
// raises it to readLimit when smaller, matching
// newRequestSizeLimiter’s max(budget, maxBody) rule on the HTTP protocol.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Two nits in this comment block: it contains a non-ASCII right single quote (U+2019) in newRequestSizeLimiter’s where surrounding comments are ASCII, and it names an unexported fork symbol (rpc.Server.recomputeWSConcurrentBudget) that is invisible from this repo and will rot silently if the fork renames it. The contract itself ("the fork raises the budget to the read limit when it is smaller") is accurate — just state it without the private symbol.

srv.SetWSConcurrentRequestBytes(config.maxConcurrentRequestBytes)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] The WS budget is passed through raw, without the normalization newRequestSizeLimiter applies on the HTTP side (if maxConcurrentBytes < maxBody { maxConcurrentBytes = maxBody }, request_limiter.go:47-49). Since both planes are fed from the same pair of config values, a config with max_concurrent_request_bytes in (0, effective max_request_body_bytes) behaves fine on :8545 but on :8546 every frame larger than the budget can never acquire capacity — semaphore.Weighted.Acquire with n > size blocks until the context is done — so those requests always burn the full admission wait and then fail, instead of being served. Codex flagged the same thing.

Worse, this PR's own TestWSAdmissionHookBudgetWaitTimeout sets SetReadLimits(frameSize) and SetWSConcurrentRequestBytes(frameSize), writes a single payload of exactly frameSize, and expects WSAdmissionReasonBudgetWaitTimeout. That only holds if the weight charged for a max-size frame exceeds a budget equal to the frame cap — i.e. even budget == maxFrame is not admissible, so mirroring HTTP's budget = max(budget, maxBody) may not be sufficient and the required headroom needs to be nailed down.

Please normalize here (or validate in evmrpc/config) so that a single maximum-size frame is always admissible, and add coverage for budget < readLimit.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] SetWSAdmissionTimeout is called only in ws_admission_test.go:139 — never in this production wiring. So the WS budget-wait timeout that the new config docs promise ("WebSocket blocks until budget frees or times out") is whatever the go-ethereum fork happens to default to, is invisible to operators, and can't be tuned. Since that timeout is exactly the knob that bounds how long a WS read loop can stall under budget pressure, please set it explicitly here (even to a named constant) and ideally plumb it through Config alongside max_concurrent_request_bytes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] SetWSAdmissionTimeout is never called on the production path — it appears only in ws_admission_test.go:143. So the wait bound that the config comment promises ("WebSocket blocks until budget frees or times out") is whatever the fork's default happens to be, with no knob and no documented value.

This matters more on WS than HTTP because the failure mode is head-of-line blocking: unlike the HTTP limiter's fast 429, an exhausted WS budget stalls read loops, so one slow/large request can hold up unrelated requests on the same plane. If the fork default is unbounded (or very long), a saturated budget wedges the plane instead of shedding load.

Please set an explicit timeout here — ideally sourced from config alongside the other two knobs — and state the default in the max_concurrent_request_bytes doc comment.

@bdchatham bdchatham Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. The fork default is 30s (defaultWSAdmissionTimeout, fork rpc/handler.go:95), so it is bounded, but it stays undocumented and unreachable from app.toml, and it lands on the same value as wsPingInterval and wsPongTimeout (fork rpc/websocket.go:38-40).

What the wait ends in matters more than how long it is. acquirePreDecode failure returns from the read loop (fork rpc/client.go:730-733) and dispatch calls conn.close() (:664-667), so the connection is torn down with no JSON-RPC error, which is closer to a drop than to shedding.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] The WS budget appears to be a concurrency cap, not a size-weighted budget - please confirm and document.

All three new tests are only consistent with the fork reserving the full readLimit (worst case) per pending read rather than the actual frame size - which makes sense, since a WS frame's length isn't known before it's read:

  • TestWSAdmissionHookBudgetWaitTimeout sends exactly one frame with budget == readLimit == frameSize and expects WSAdmissionReasonBudgetWaitTimeout. With per-frame-size accounting that single frame fits and nothing should ever time out. It only times out if the read loop pre-reserves readLimit for a second, never-sent message while message 1 is still in flight.
  • TestEnableWSConcurrentRequestBytes (budget == readLimit) relies on the two requests serializing, i.e. effective concurrency of 1.
  • TestEnableWSConcurrentBudgetBelowReadLimitAdmitsMaxFrame (budget = readLimit/2, raised to readLimit) again gives concurrency 1.

If that's the implementation, the effective WS limit is max(1, floor(maxConcurrentRequestBytes / effectiveMaxRequestBodyBytes)) in-flight requests plane-wide, independent of actual request sizes. At default config that's 128 MiB / 5 MiB = 25 concurrent WS requests total, down from unlimited today - a real throughput regression for high-throughput WS users. And the two knobs become coupled in a surprising way: bumping max_request_body_bytes to 64 MiB to accept large frames would drop WS concurrency to 2.

Please confirm the accounting, and if it is worst-case-per-read: document the derived concurrency cap on the config field, and consider sizing the WS budget off readLimit (e.g. a minimum multiple) so raising the frame cap can't collapse concurrency.

@bdchatham bdchatham Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed on the mechanism, though framing it as in-flight requests understates the impact. acquirePreDecode runs before the blocking codec.readBatch() (fork rpc/client.go:729-734), so the reservation covers the whole time a connection sits idle, well beyond the window a frame is actually in flight. That makes the unit connections, roughly 25 plane-wide at defaults against max_open_connections = 2000.

Reproduced against v1.15.7-sei-18. 24 idle sockets, victim request served. 25 idle sockets sending zero application bytes, victim reset after the 30s wait. One rpc.Server backs all of :8546, so idle eth_subscribe connections hold their slot for the life of the connection.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The production path never calls SetWSAdmissionTimeout, so the WS budget-wait timeout is whatever the fork defaults to - not configurable and not documented anywhere in config.go or the app.toml template, even though the new config comment tells operators WS "blocks until budget frees or times out". Since this timeout directly determines client-visible latency under load, please either expose it as a config knob or at minimum document the default value alongside max_concurrent_request_bytes.

This also makes TestEnableWSConcurrentRequestBytes brittle: request 2 waits ~200 ms for budget, so the test silently depends on the fork's default timeout exceeding that.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] Enabling the WS budget caps concurrent WS connections at ~25 with default config, and over-cap connections are silently dropped.

I read the enforcement in the fork (sei-protocol/go-ethereum #81/#82, rpc/client.go read + rpc/handler.go). The read loop is:

for {
    if err := h.acquirePreDecode(h.rootCtx); err != nil { c.readErr <- err; return }
    msgs, batch, rawLen, err := codec.readBatch()   // blocks here until the client sends a frame
    ...
    release, err := h.commitFrameBudget(h.rootCtx, rawLen)

acquirePreDecode acquires the full readLimit from the server-wide semaphore.Weighted before readBatch() blocks waiting for the next frame, and holds it for the entire idle period; commitFrameBudget only trues it up to the real frame size once a frame actually arrives.

So every established WS connection permanently pins max_request_body_bytes of the shared budget just by sitting there. With the shipped defaults (5 MiB frame limit, 128 MiB budget) that is 25 connections total, against max_open_connections = 2000. Connection #26 blocks in acquirePreDecode, times out after ws_admission_timeout (30s), and the error goes to c.readErr → read loop returns → connection closed, with no JSON-RPC error (the frameBudgetExceededResponse path only fires on the commitFrameBudget branch). Long-lived, mostly-idle eth_subscribe connections are exactly this workload, so most subscribers would be churned every 30s.

Before this PR SetWSConcurrentRequestBytes was never called (nil budget ⇒ no-op), so this line is what activates the behavior — hence blocking here rather than upstream.

TestEnableWSAdmissionTimeout in this PR encodes the failure mode: one in-flight request, nothing else pending, connection torn down.

Options: leave the WS budget disabled by default (0) until the fork charges only on actual frame size at commit time; or size the WS budget independently against max_open_connections × readLimit instead of reusing the HTTP number; or fix the fork to reserve a small nominal amount pre-read. Whichever route, please also state the client-visible outcome (connection close, not an error response) in the config docs.

srv.SetWSAdmissionTimeout(config.wsAdmissionTimeout)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idle WS ties up byte budget

Medium Severity

Wiring max_concurrent_request_bytes into the WebSocket server enables the fork’s pre-decode byte semaphore on every connection read loop. Reservations are taken before the next frame is read and are not released until that cycle finishes, so idle open sockets can hold budget sized up to the effective read limit even when no frame is in flight. At default settings that caps useful WS concurrency near the budget divided by the read limit, not by actual payload sizes as on HTTP.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 6f5a170. Configure here.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WS timeout zero not normalized

Low Severity

EnableWS normalizes readLimit via effectiveMaxRequestBodyBytes before calling the RPC server, but passes wsAdmissionTimeout through unchanged. Config and TOML docs state zero or negative values should use the 30s go-ethereum default, yet ReadConfig can store an explicit 0 from app.toml, unlike the always-positive default copied from DefaultConfig when the key is omitted.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 6f5a170. Configure here.

srv.SetWSAdmissionEventHook(func(reason string) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] context.Background() here drops any trace/span context, unlike the HTTP path which records against r.Context() (request_limiter.go:58,76). If the fork's hook signature can carry the per-request context, plumb it through; otherwise a short comment explaining why it can't would help.

Also note prod never calls SetWSAdmissionTimeout (only the test does), so the wait timeout the new config comment refers to is whatever the fork defaults to and is not operator-tunable. Worth either exposing it or documenting the value in the toml comment.

Comment on lines +386 to +395

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The shared, process-wide WS concurrent-byte budget (srv.SetWSConcurrentRequestBytes in evmrpc/rpcstack.go:385-390) is enforced by speculatively reserving readLimit bytes for every WS connection's next frame before it even arrives, and that reservation for an in-flight call is only released after its handler returns — so a single slow call (e.g. eth_call near the 60s SimulationEVMTimeout) or simply enough idle connections (only ~25 fit in the default 128MiB budget / 5MiB readLimit, vs MaxOpenConnections=2000) can starve other, unrelated WS connections' lookahead reservation past the 30s wsAdmissionTimeout (never configured by evmrpc) and force-disconnect them entirely via cancelRoot(), killing any active subscriptions on that connection too. This is a materially wider blast radius than the PR description's "requests block until budget frees or time out," which implies only the offending request is affected — please size/document the budget relative to expected connection count, or otherwise bound the lookahead reservation's impact on idle/unrelated connections before merge.

Extended reasoning...

What the bug is. evmrpc/rpcstack.go:385-390 wires config.maxConcurrentRequestBytes into srv.SetWSConcurrentRequestBytes, which the vendored sei-protocol/go-ethereum@v1.15.7-sei-18 fork enforces via a speculative pre-decode reservation. Before the WS read loop even blocks on the next frame, handler.acquirePreDecode (rpc/handler.go:134-144) calls wsConcurrentBudget.Acquire(ctx, h.readLimit) to reserve the full readLimit worth of budget for whatever frame might arrive next — bounded by wsAdmissionTimeout (default 30s, never overridden by evmrpc in production). This reservation is held across the blocking codec.readBatch() wait (rpc/client.go:729-754), meaning even a connection that has sent nothing and is purely listening on an eth_subscribe stream holds readLimit bytes of the budget reserved the entire time it's idle.

The code path that triggers it. The budget itself, s.wsConcurrentBudget, is a single semaphore.Weighted created once per RPC server (rpc/server.go:135-144) and shared across every WS connection process-wide (server.go:178), not per-connection. The weight reserved for an in-flight call is only released after the call's handler returns (release() invoked post-handleCallMsg, handler.go:441), so a single slow method — an eth_call running close to the 60s SimulationEVMTimeout, or a debug_trace* near TraceTimeout — can hold a large chunk of the shared budget for the duration of its execution while unrelated connections keep trying to acquire their own lookahead reservations. When acquirePreDecode hits context.DeadlineExceeded, the error propagates through read() (client.go:730-733) into dispatch(), which calls conn.close(err, lastOp) (client.go:664-667). That reaches handler.closeh.cancelRoot() (handler.go:454-458), and because every subscription's context derives from rootCtx (handler.go:525), canceling it tears down the entire connection and every subscription running on it.

Why existing code doesn't prevent it. Nothing in this PR bounds how many WS connections can be simultaneously admitted relative to the shared budget, nor does it separate 'budget for the currently-processing request' from 'lookahead reservation for connections that are simply idling.' With the shipped defaults (max_concurrent_request_bytes = 128 MiB, effective readLimit = 5 MiB), only ~25 concurrent WS connections can hold a pre-decode reservation at once — far below MaxOpenConnections = 2000 — so under any realistic multi-subscriber load, connections beyond that ~25 (or connections contending with a few slow in-flight calls) will time out and be force-closed, regardless of whether they themselves ever sent an oversized or even a new frame.

Impact. This is a real availability regression on the WS plane, which previously had no concurrent-byte budget at all (just a flat 10 MiB read limit). A well-behaved client with an active eth_subscribe that sends nothing further can be disconnected — and lose its subscription — purely because other connections or a couple of slow requests saturated the shared budget. This contradicts the PR description's framing ('WS requests block until budget frees or time out'), which implies only the offending/oversized request is throttled, not that unrelated idle connections and their subscriptions get torn down.

Step-by-step proof (from the PR's own test). TestWSAdmissionHookBudgetWaitTimeout in evmrpc/ws_admission_test.go demonstrates the mechanism directly: it sets SetReadLimits(frameSize) and SetWSConcurrentRequestBytes(frameSize) (budget == one frame's worth), writes exactly one payload of frameSize bytes that triggers a 200ms sleep inside the handler, and asserts that WSAdmissionReasonBudgetWaitTimeout fires. This happens with only one in-flight call and no second client request at all — the timeout is produced purely by the read loop's own next lookahead acquirePreDecode call trying (and failing) to reserve budget that the first call's handler hasn't released yet. Extrapolate this to production: (1) client A opens a WS subscription and goes idle — its read loop holds a readLimit-sized reservation while blocked in readBatch(); (2) enough other clients connect or one/more clients issue slow eth_calls that collectively saturate the shared 128 MiB budget; (3) client A's next acquirePreDecode cycle (which fires as soon as the previous read completes, or immediately if the connection is still in its very first read) cannot acquire the budget within 30s; (4) conn.close() fires, cancelRoot() cancels client A's context, and its subscription is silently killed — despite client A never sending anything.

Suggested fix. Either exclude idle/no-op lookahead reservations from contending with in-flight active reservations (e.g., only reserve budget once bytes have actually started arriving, rather than before the read even begins), or size guidance/documentation and validation should make explicit that the shared budget must scale with the expected number of concurrent WS connections (budget ≳ readLimit × expected-connections), and that hitting it disconnects the whole connection including subscriptions, not just the one request.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two commits made after this comment (6be7306, a5f1b61) address an unrelated finding (require.IsType -> require.Equal in the test's readJSON helper) and add a comment/test clarifying that SetWSConcurrentRequestBytes is raised to readLimit when smaller. Neither change addresses the concern raised here: SetWSAdmissionTimeout is still never called in the production EnableWS wiring (evmrpc/rpcstack.go) -- only in ws_admission_test.go -- so the 30s fork-default admission wait timeout that bounds how long an idle/unrelated WS connection's lookahead reservation can be starved before cancelRoot() force-disconnects it remains unconfigured and undocumented. The max_concurrent_request_bytes doc comment in evmrpc/config/config.go is unchanged from before this comment was posted, still just saying WebSocket blocks until budget frees or times out with no sizing guidance relative to expected connection count or explanation that hitting the timeout tears down the whole connection (including subscriptions), not just the offending request. The reported issue is still fully applicable.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wsAdmissionTimeout wiring gap is now fixed: config.go adds a WSAdmissionTimeout field (default 30s, documented, exposed via the ws_admission_timeout app.toml key), and evmrpc/server.go -> evmrpc/rpcstack.go:394 now calls srv.SetWSAdmissionTimeout(config.wsAdmissionTimeout) on the production path. That resolves the narrower point from my 10:44:55 follow-up.

However, the original comment's core ask is still open. It's not just about the timeout being configurable -- it's that a saturated shared budget disconnects unrelated idle WS connections (subscriptions and all), which is a materially different failure mode than "the offending request blocks or times out." The suggested fixes were either (a) exclude idle lookahead reservations from contending with active ones, or (b) document the sizing relationship (budget ≳ readLimit × expected-connections) and make explicit that hitting the timeout tears down the whole connection, not just one request.

Neither is present. The MaxConcurrentRequestBytes doc comment (evmrpc/config/config.go:260-265) still only says "WebSocket blocks until budget frees or WSAdmissionTimeout elapses" -- no sizing guidance relative to connection count, and no mention that the wait ending in a timeout closes the connection and drops any active subscriptions on it. The speculative pre-decode reservation mechanism itself is unchanged. Please add that guidance to the config doc (and app.toml template) or bound the lookahead reservation's impact on idle connections before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Agreeing with Codex here: this hook appears to fire only for admission-wait outcomes (the one reason constant referenced anywhere in the tree is rpc.WSAdmissionReasonBudgetWaitTimeout). Frames dropped by SetReadLimits on line 388 don't go through the admission path, so evmrpc_requests_rejected_total{plane="ws",reason="oversize"} would never be emitted — which contradicts the PR description's claim that "rejections on either plane are recorded through evmrpc_requests_rejected_total". Please either extend the fork to signal read-limit rejections through the same hook (ideally with reason == oversize, matching the HTTP vocabulary) or record it here, so operators can distinguish "WS clients are sending oversized frames" from "WS is out of budget". If the fork does already invoke the hook on oversize, a test asserting that would settle it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Related to the accounting question above: if the read loop reserves before a frame exists, this hook fires - and increments evmrpc_requests_rejected_total{plane="ws"} - when no request was actually rejected. TestWSAdmissionHookBudgetWaitTimeout demonstrates exactly that: one request is sent, and the counter records a rejection for a message that was never received. Counting phantom rejections will make the metric unusable for alerting on real client-visible drops. Worth confirming what the client actually observes on a budget-wait timeout (error response? connection close? nothing?) and only recording when a request is genuinely dropped.

// Hook carries no request context, and the fork's own connCtx is context.Background() too.
recordWSAdmissionRejected(context.Background(), reason)
Comment on lines 382 to +397

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WS admission's SetWSAdmissionEventHook only fires for the two concurrent-byte-budget timeout reasons in the go-ethereum fork (WSAdmissionReasonBudgetWaitTimeout / WSAdmissionReasonFrameAdmissionTimeout); an oversize WS frame is instead caught by gorilla's conn.SetReadLimit and just closes the connection without ever invoking the hook. As a result evmrpc_requests_rejected_total{plane="ws"} never gets an oversize-equivalent reason, unlike the HTTP plane which explicitly records rejectReasonOversize — an observability gap relative to the PR's stated goal of tracking rejections on both planes through this counter.

Extended reasoning...

The bug: EnableWS in evmrpc/rpcstack.go wires up srv.SetWSAdmissionEventHook(func(reason string) { recordWSAdmissionRejected(...) }), but in the vendored sei-protocol/go-ethereum fork (rpc/handler.go), that hook is only ever invoked from fireAdmissionEventOnBudgetTimeout, which fires exclusively when context.DeadlineExceeded is hit while acquiring the WS concurrent-byte budget (acquirePreDecodeWSAdmissionReasonBudgetWaitTimeout, commitFrameBudgetWSAdmissionReasonFrameAdmissionTimeout). There is no code path in the fork that fires the hook for an oversize frame.

Where oversize frames actually get rejected: The per-frame size cap set by this same PR (srv.SetReadLimits(readLimit), rpcstack.go:381-391) is enforced purely by gorilla's websocket library via conn.SetReadLimit(readLimit) in newWebsocketCodec (rpc/websocket.go). When an incoming frame exceeds that limit, gorilla's ReadMessage/ReadJSON simply returns a "read limit exceeded" error, which tears down the connection in the read loop — it never touches acquirePreDecode/commitFrameBudget, so admissionEventHook is never called for this case.

Why this is inconsistent with the PR's own stated goal: The PR description says "Rejections on either plane are recorded through evmrpc_requests_rejected_total," and the updated evmrpc/config/config.go doc comment explicitly states "WebSocket frames exceeding this limit are rejected by the read loop" as if it were parallel to the HTTP oversize case. But HTTP's requestSizeLimiter.ServeHTTP (request_limiter.go) explicitly calls recordRequestRejected(rejectReasonOversize) before returning 413 for an oversize body — giving HTTP an observable oversize signal that WS structurally cannot produce with the current wiring.

Step-by-step proof:

  1. Operator sets max_request_body_bytes = 1024 (applies to both planes via effectiveMaxRequestBodyBytes).
  2. A client sends an HTTP POST with a 2048-byte body: requestSizeLimiter.ServeHTTP sees r.ContentLength > l.maxBody, calls recordRequestRejected(ctx, rejectReasonOversize), and returns 413. evmrpc_requests_rejected_total{plane="http",reason="oversize"} increments.
  3. A client sends a WS frame of 2048 bytes over the same-sized limit (srv.SetReadLimits(1024)): gorilla's conn.SetReadLimit(1024) causes the next ReadMessage call inside the fork's read loop to fail with a read-limit error; the connection is closed. admissionEventHook is never invoked, so recordWSAdmissionRejected never runs.
  4. Result: evmrpc_requests_rejected_total{plane="ws"} has no entry for this rejection at all — an operator dashboarding this counter to gauge oversize-frame drop rate on the WS plane sees nothing, even though frames are actively being dropped and connections closed.

Impact and fix: This is purely an observability gap — the oversize WS frame is still correctly rejected (the connection is dropped, which is itself a very visible failure mode to a WS client) — so there's no functional/correctness bug, no crash, no data loss. Closing the gap would require adding a new hook point at the gorilla read-limit boundary inside the vendored go-ethereum fork (e.g. wrapping the read-limit error in the WS read loop to fire admissionEventHook with a new WSAdmissionReasonOversize-style reason), which is beyond the scope of a same-repo fix and would need a fork change. Given the limited blast radius (metrics-only, and the drop is otherwise visible via connection closure), this doesn't need to block merge, but is worth tracking as a known limitation or follow-up fork change.

})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WS reject reasons break metric contract

Low Severity

The WebSocket admission hook forwards fork-specific reason strings (such as budget wait timeout constants) unchanged into evmrpc_requests_rejected_total, while HTTP rejections use the documented oversize and busy values. Alerts or dashboards keyed on reason=busy or reason=oversize will miss WebSocket rejections even though the metric description still describes those reason labels.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 1f9d1f1. Configure here.

logger.Info("Registering apis for evm websocket")
if err := RegisterApis(apis, config.Modules, srv); err != nil {
return err
Expand Down
5 changes: 1 addition & 4 deletions evmrpc/sei_legacy_http.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,7 @@ func wrapSeiLegacyHTTP(inner http.Handler, allowlist map[string]struct{}, maxBod
if allowlist == nil {
return inner
}
if maxBody <= 0 {
maxBody = defaultMaxRequestBodyBytes
}
return &seiLegacyHTTPGate{inner: inner, allowlist: allowlist, maxBody: maxBody}
return &seiLegacyHTTPGate{inner: inner, allowlist: allowlist, maxBody: effectiveMaxRequestBodyBytes(maxBody)}
}

type seiLegacyHTTPGate struct {
Expand Down
5 changes: 3 additions & 2 deletions evmrpc/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ var ConnectionTypeWS ConnectionType = "websocket"
var ConnectionTypeHTTP ConnectionType = "http"

const LocalAddress = "0.0.0.0"
const DefaultWebsocketMaxMessageSize = 10 * 1024 * 1024

type EVMServer interface {
Start() error
Expand Down Expand Up @@ -339,7 +338,9 @@ func NewEVMWebSocketServer(
}

wsConfig := WsConfig{Origins: strings.Split(config.WSOrigins, ",")}
wsConfig.readLimit = DefaultWebsocketMaxMessageSize
wsConfig.readLimit = config.MaxRequestBodyBytes

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This is a user-visible RPC behavior change that isn't called out as such: the WS read limit goes from a hardcoded 10 MiB to max_request_body_bytes, whose default is 5 MiB (config.go:328). Existing WS clients sending 5-10 MiB frames (large eth_sendRawTransaction batches, wide eth_getLogs filter sets) will start being disconnected by the read loop after upgrade unless operators raise the value. Please flag it in the PR description / release notes.

Separately, DefaultWebsocketMaxMessageSize was an exported constant; removing it is a breaking change for anything importing evmrpc. If that's acceptable, fine — just worth being deliberate about.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This silently halves the default WS frame cap: previously a hardcoded 10 MiB, now MaxRequestBodyBytes, whose default is 5 MiB (evmrpc/config/config.go:328). Because a gorilla read-limit violation terminates the connection (close 1009) rather than returning a JSON-RPC error, any existing client sending 5–10 MiB frames (e.g. large batch requests) goes from working to having its socket dropped after this upgrade. Worth an explicit release-note/upgrade-guide entry, and possibly keeping the WS default at 10 MiB unless max_request_body_bytes is set.

Minor cleanup while here: RPCEndpointConfig now has both readLimit and maxRequestBodyBytes fed from the same config.MaxRequestBodyBytes (WS sets only the former, HTTP only the latter). Collapsing them into one field, or documenting which plane reads which, would avoid the next reader wiring the wrong one.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] This silently halves the WS frame ceiling for every operator on default config: MaxRequestBodyBytes defaults to 5 MiB (config.go:328), replacing the previous hardcoded 10 MiB. Two consequences worth explicit sign-off:

  1. Clients currently sending 5–10 MiB WS frames (large batch requests, big eth_call payloads, raw-tx bundles) start failing after upgrade — and because the limit is enforced by the read loop, the whole connection is torn down, not just the one frame. That is a user-visible regression, not a config nicety.
  2. WS frame size is no longer tunable independently of the HTTP body cap. An operator who wants to keep WS at 10 MiB must also raise the HTTP body limit to 10 MiB, which is precisely the memory-amplification the HTTP limiter was added to bound.

Suggest either keeping a dedicated WS knob (defaulting to the old 10 MiB, falling back to max_request_body_bytes when unset), or — if collapsing to one knob is the deliberate call — flagging the reduction prominently in the PR description and release notes so operators can raise the value before upgrading.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] This silently halves the default WS frame limit. wsConfig.readLimit was DefaultWebsocketMaxMessageSize (10 MiB); it now comes from config.MaxRequestBodyBytes, whose default is 5 MiB (evmrpc/config/config.go:328). On upgrade, every node running default config starts rejecting WS frames in the 5-10 MiB range with no config change on the operator's part.

It's worse for operators who already tuned max_request_body_bytes down for the HTTP plane - that value now also clamps WS frames, a coupling they never opted into.

If the reduction is intentional, please call it out explicitly in the PR description / release notes (the PR body currently only says the constant was "replaced"). Otherwise consider either keeping a separate WS read-limit knob or raising the shared default to 10 MiB to preserve existing WS behavior.

@bdchatham bdchatham Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. MaxRequestBodyBytes defaults to 5 MiB (evmrpc/config/config.go:328) where the removed constant was 10 MiB, and gorilla enforces the cap in SetReadLimit (fork rpc/websocket.go:295), so an oversize frame closes the connection with WS 1009 rather than failing the single frame.

The remedy is coupled in a way worth calling out. Raising this to 10 MiB to preserve the old ceiling also halves the WS connection budget, which is the thread on rpcstack.go.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] RPCEndpointConfig now carries both readLimit (WS) and maxRequestBodyBytes (HTTP), fed from the same config.MaxRequestBodyBytes. Two fields holding one config value invite divergence. Consider collapsing to maxRequestBodyBytes for both planes, or at minimum give readLimit (rpcstack.go:67) a comment saying it is max_request_body_bytes applied to WS frames — it is the only field in that struct without one.

wsConfig.maxConcurrentRequestBytes = config.MaxConcurrentRequestBytes
wsConfig.wsAdmissionTimeout = config.WSAdmissionTimeout
wsConfig.batchItemLimit = config.BatchRequestLimit
Comment on lines 338 to 344

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The WS frame-size default silently drops from a hardcoded 10 MiB to 5 MiB (config.MaxRequestBodyBytes' default) because wsConfig.readLimit now derives from MaxRequestBodyBytes instead of the removed DefaultWebsocketMaxMessageSize constant. Since gorilla closes the connection (close 1009) on a read-limit violation rather than returning a JSON-RPC error, any existing WS client sending 5-10 MiB frames will start getting disconnected after upgrading unless the operator explicitly sets max_request_body_bytes; removing the exported constant is also a source-break for external importers of evmrpc.

Extended reasoning...

What changed. Before this PR, evmrpc/server.go hardcoded wsConfig.readLimit = DefaultWebsocketMaxMessageSize, an exported constant equal to 10 * 1024 * 1024 (10 MiB). This PR deletes that constant entirely and replaces the assignment with wsConfig.readLimit = config.MaxRequestBodyBytes (server.go:341-342). MaxRequestBodyBytes is a shared HTTP/WS config knob whose default, per evmrpc/config/config.go's DefaultConfig, is 5 * 1024 * 1024 (5 MiB).

The code path that triggers it. Any node operator who upgrades to this version without explicitly setting max_request_body_bytes in their app.toml inherits the new 5 MiB default in place of the old hardcoded 10 MiB WS ceiling. EnableWS (evmrpc/rpcstack.go) normalizes a zero/negative value through effectiveMaxRequestBodyBytes, which also resolves to the 5 MiB default, then calls srv.SetReadLimits(readLimit). That value is enforced by gorilla's WS read loop (via the vendored go-ethereum fork's SetReadLimits/conn.SetReadLimit), and a read-limit violation there does not surface as a JSON-RPC error response — it terminates the underlying connection with a close code 1009 ("message too big"), silently dropping the client's socket including any active subscriptions.

Why existing code doesn't prevent it. Nothing in this PR (or its config defaults) preserves the old 10 MiB WS ceiling as a WS-specific default; the PR intentionally unifies the HTTP body cap and the WS frame cap under one max_request_body_bytes knob, but doesn't call out that doing so changes the effective default for WS from 10 MiB to 5 MiB. There's also no startup log or metric that flags this to an operator upgrading with a stock config.

Impact. Any existing WS client that was previously relying on the 10 MiB ceiling — e.g. large eth_sendRawTransaction batches, or a wide eth_getLogs-style filter payload in the 5-10 MiB range — will go from working before the upgrade to having its socket forcibly closed after the upgrade, with no in-band error to explain why. The only way to avoid this is for the operator to proactively set max_request_body_bytes back to 10 MiB (or higher) in app.toml, which nothing in the PR prompts them to do. Separately, DefaultWebsocketMaxMessageSize was an exported Go constant; deleting it is a breaking change for any code outside this repo that imports evmrpc and references it directly.

Step-by-step proof.

  1. Pre-PR: operator has no max_request_body_bytes set. NewEVMWebSocketServer sets wsConfig.readLimit = DefaultWebsocketMaxMessageSize = 10*1024*1024. A client sends an 8 MiB WS frame (e.g. a large batch tx submission) — it is accepted and processed.
  2. Operator upgrades to this PR's binary, still with no max_request_body_bytes set (default remains unset/0 in their existing config file).
  3. NewEVMWebSocketServer now sets wsConfig.readLimit = config.MaxRequestBodyBytes, which resolves (via DefaultConfig/effectiveMaxRequestBodyBytes) to 5*1024*1024 (5 MiB).
  4. The same client sends the same 8 MiB frame. Gorilla's read loop in the underlying rpc server detects the frame exceeds the new 5 MiB readLimit and closes the connection with close code 1009, instead of returning a JSON-RPC 413-equivalent error.
  5. The client observes an abrupt disconnect (and loses any live subscriptions on that connection) with no application-level error message indicating the cause, and no upgrade note telling the operator this could happen.

Suggested fix. Add an explicit release-note / upgrade-guide entry calling out that the effective WS frame default has changed from 10 MiB to 5 MiB, and/or default the WS read limit to 10 MiB specifically when max_request_body_bytes is left unset, preserving prior wire behavior while still letting operators opt into the unified single-knob model. This does not need to block merge — the value remains fully operator-tunable via max_request_body_bytes, and the regression only manifests for WS clients already sending 5-10 MiB frames, but it is a real, undocumented behavioral change worth flagging (this was also independently raised inline by the seidroid reviewer during PR review).

🔬 also observed by seidroid

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The WS read-limit wiring is unchanged: evmrpc/server.go:341 still sets wsConfig.readLimit = config.MaxRequestBodyBytes directly, and MaxRequestBodyBytes still defaults to 5 MiB (evmrpc/config/config.go), so an operator upgrading with a stock config still silently drops from the old hardcoded 10 MiB WS ceiling to 5 MiB. The removed DefaultWebsocketMaxMessageSize constant has not been restored. Only doc-comment wording, test cleanup, and an effectiveMaxRequestBodyBytes refactor were added in later commits (3a568d5, 1f9d1f1, a5f1b61, 63410e8) -- none of them touch the actual default value or reintroduce a WS-specific ceiling. This is the same conclusion already reached by two prior follow-ups in this thread (10:44:55 and 10:55:19 for related findings); the concern in this comment remains fully unaddressed.

wsConfig.batchResponseSizeLimit = config.BatchResponseMaxSize
if err := httpServer.EnableWS(apis, wsConfig); err != nil {
Expand Down
Loading
Loading