Inclusion checker floods the warn log: simnet mock omits `execution_optimistic` on block responses

[emlautarom1-agent]

Summary

The networked inclusion checker fails on every due slot: it asks the beacon node whether a slot produced a block, and the response fails to deserialize. At the smoke suite's 1s slots that is roughly one warning per second per node — 30 per 30s window against a Warn Log Rate threshold of 2 — so every pluto smoke scenario currently fails: all_pluto, pluto_dkg, and mixed_2_charon_2_pluto.

WARN pluto_core::tracker::inclusion: Failed to check inclusion slot=169351859
  err=beacon node request failed: JSON deserialization error at path '.':
      missing field `execution_optimistic` at line 1 column 2642

Cause

Both halves of this are pluto's own code; no Charon component is involved.

The server omits the fields. Every smoke node runs its own in-process simnet mock — CHARON_SIMNET_BEACON_MOCK: "true" with an empty CHARON_BEACON_NODE_ENDPOINTS (golden compose file) — so a pluto node never queries a Charon node's beaconmock. The handler mounted at ^/eth/v2/beacon/blocks/[^/]+$ returns an envelope of {version, data} only: defaults.rs#L602-L604, mounted at defaults.rs#L205-L208.

This handler is inconsistent with its own neighbours in the same file, which do emit both fields — e.g. defaults.rs#L110-L111 and defaults.rs#L369-L370. So the mock is internally self-contradictory, and this endpoint is the outlier.

The client requires them. InclusionChecker::block_exists calls get_block_v2, whose generated response struct GetBlockV2ResponseResponse declares execution_optimistic: bool and finalized: bool as plain non-Option fields with no #[serde(default)]. That type is generated at build time by oas3-gen into crates/eth2api/src/types.rs, which is gitignored — hence no permalink. It is non-optional because the vendored spec lists both in required: beacon-node-oapi.json#L16368-L16375.

Note the spec is self-contradictory rather than permissive here: it marks both fields required, while their own descriptions specify what a consumer should do when they are absent ("If the field is not present, assume the False value"). Charon's go-eth2-client shovels unrecognised top-level keys into a metadata map and so cannot fail on omission, but that is a consequence of its decoding style, not a spec guarantee.

A pluto node pointed at a real Charon beaconmock would fail identically — its handler also marshals {version, data} only (charon beaconmock/server.go#L92-L105) — but that path is not exercised by any scenario and is not the cause here.

Rate

One check per due slot, each failing, at 1s slots gives ~1 warning/s/node. Measured in mixed_2_charon_2_pluto: 238 inclusion warnings across the two pluto nodes over a ~113s window, ~1.05/s/node.

Separately, run does not advance checked_slot when block_exists errors, so a failing slot is re-attempted on the next tick. At the smoke suite's 1s slots this changes nothing — the 1s ticker yields a new due slot every tick anyway — but with slot_duration > 1s it multiplies the warning rate by the number of ticks per slot. Worth deciding alongside the decode question rather than treating it as the cause of this flood.

Measured

Two mixed_2_charon_2_pluto runs and one all_pluto run, charon image v1.7.1, pluto:local at 4f60d712, mock VC, 1s slots.

app_log_warn_total on the pluto nodes (no topic label — pluto sets none here) over an ~87–113s observation window:

scenario	node0	node1	node2	node3
mixed_2_charon_2_pluto	(charon)	(charon)	+115	+115
all_pluto	+89	+94	+88	+88

238 of 285 pluto WARN lines in a mixed run are Failed to check inclusion. Alerts fired post-warmup:

mixed_2_charon_2_pluto: Pluto node2 has a high warning rate, Pluto node3 has a high warning rate
all_pluto:              Pluto node0/1/2/3 has a high warning rate

Of the remaining pluto warnings, the only recurring post-warmup one is the downstream consequence, on every proposer duty:

WARN pluto_core::tracker::reporters: Duty failed step=bcast reason=unknown error
  error=bug: missing chain inclusion event duty=169352096/proposer

At 1 per 16s per node that stays within increase[30s] > 2, so it should not by itself fail the gate — but that is inferred from the rate, not yet observed against a fixed inclusion checker.

Attribution

The inclusion checker was wired into the workflow in #570 (0c8d59ff, 2026-08-03 16:12 UTC), the direct child of the smoke suite in #541 (98d60658, 2026-08-03 10:49 UTC). git log -S'InclusionChecker' -- crates/app/ returns only 0c8d59ff.

This was undetectable rather than a missed CI signal: TestSmoke is gated on both -integration and PLUTO_REPO, and .github/workflows/smoke-tests.yml is workflow_dispatch-only with no schedule.

Reproduce

cd test-infra/compose
PLUTO_REPO=$(git rev-parse --show-toplevel) go test ./smoke -v -integration -timeout=35m -run 'TestSmoke/all_pluto$'

Fix directions

Two independent fixes, either of which unblocks the smoke suite. They are not alternatives — the first is a test-fixture bug, the second a robustness question.

1. Emit both fields from the simnet mock's block handler, matching the sibling handlers in the same file. Minimal, and removes the self-contradiction.
2. Decide whether the decoder should tolerate absence, defaulting both to false — either by dropping them from required in the vendored spec and regenerating, or by adding #[serde(default)] in the build.rs post-processing pass. This is the one that matters against third-party beacon nodes. 31 generated structs declare execution_optimistic non-optional and 27 declare finalized, none as Option<bool>; 6 are reachable from pluto code, so this wants an audit rather than a point fix.

Also worth settling: the checked_slot retry policy above, so a persistently failing endpoint cannot scale its warning rate with the tick/slot ratio.

Acceptance

• all_pluto, pluto_dkg and mixed_2_charon_2_pluto pass without exempting pluto jobs or topics from Warn Log Rate.
• A block response without execution_optimistic/finalized decodes, with both defaulting to false.