Persist attester & PTC duties per epoch in the blockdb

[pk910]

Persist attester & PTC duties per epoch in the blockdb

Summary

Slot detail pages resolve attester-committee and PTC (payload-timeliness-committee)
duty mappings from the in-memory EpochStats cache. After an epoch finalizes those
duties are deleted (too expensive to keep), so finalized slot pages can no longer
show which validators were in each attestation/PTC committee.

This PR persists the resolved duties for each finalized epoch in the existing
blockdb (s3 / pebble / tiered) as a new object type, and serves them lazily so
the slot page itself never hits the blockdb on render. Committee membership and PTC
members are fetched on demand when a user expands an attestation's aggregation bits
or a PTC group.

Motivation

• Finalized slot pages currently can't expand attestation committees or PTC votes —
  the duty data is gone once EpochStats is pruned.
• Recomputing duties on the fly requires the full active-validator set (~8 MB) plus a
  reshuffle per epoch — too expensive for a page render.
• The blockdb already stores block bodies, payloads and BALs; duties are a natural fit.

Design

Per-epoch duties object (not per-block)

A per-block section was the first idea but is unworkable: a block at slot S references
attestation slots in [S-32, S-1], which can fall in the previous epoch, and blocks
are written to the blockdb at finalization — by which point the previous epoch is already
finalized and its duties pruned, with no cheap way to reload them mid-finalization.

Storing one object per epoch, written at that epoch's own finalization (when its
EpochStatsValues are still in scope), sidesteps this entirely. Each epoch object is
self-contained and written exactly once.

Storage layout (per backend)

A semantic DutiesEngine interface lets each backend store natively:

• S3 — one packed object per epoch, key …/{firstSlot}_duties. Deterministic layout:
  a 40-byte header + a flat array of 48-bit global validator indices in
  (slotIndex, committee, position) order + a PTC section. Committee byte offsets are a
  pure function of the active-validator count (same SplitOffset math the indexer uses),
  so any (slot, committee) is addressable with a ranged read — no index table.
• Pebble — one key per slot ([ns][firstSlot][type][slotIndex]) so a single slot is a
  point lookup, with no need to load the whole epoch.

The copier converts between the two via the engine-agnostic EpochDuties struct.

48-bit (6-byte) indices are future-safe (protocol indices are uint64) at a third less
storage than full uint64. Estimated overhead at 1M validators: ~6 MB/epoch
(~0.5 TB/yr) — comparable to the Gloas payload+BAL already stored per block.

Lazy, no page-load blockdb hits

The slot page renders committee indices and aggregation bits (both derived from the block
itself) but not validator lists. A cached JSON endpoint resolves duties on expand:

GET /slot/{slot}/duties?committees=M1,M2   -> { validators: [...], names: {...} }
GET /slot/{slot}/duties?ptc=1              -> { validators: [...], names: {...} }

The ChainService accessor decides the source per slot: in-memory EpochStats for
unfinalized epochs (via GetOrLoadValues, which also restores duties from the
unfinalized_duties table during the pruned-but-not-yet-finalized window), and the blockdb
duties object for finalized epochs.

What's included

Storage layer

• blockdb/types/duties_format.go — DUTY object format, encode/decode, deterministic
  SplitOffset-based offsets, per-slot pebble helpers; round-trip tests (incl. 1M-validator
  uneven splits).
• blockdb/types/engine.go — DutiesEngine interface.
• blockdb/{s3,pebble,tiered}/duties.go — backend implementations.
• blockdb/blockdb.go — SupportsDuties() + wrapper methods, engine detection.

Producer

• indexer/beacon/dutiesexport.go — BuildEpochDuties (resolves EpochStatsValues →
  global indices) + writeEpochDutiesToBlockDb.
• Wired into finalization.go and synchronizer.go in the same concurrent write phase as
  block bodies.

Consumer

• services/chainservice_duties.go — GetSlotCommittees / GetSlotPtc (cache-or-blockdb).
• handlers/slot_duties.go — cached /slot/{slot}/duties endpoint.
• handlers/slot.go — eager duty resolution removed from the page build.
• templates/slot/attestations.html, ptc_votes.html — lazy fetch on expand.

Tooling

• cmd/dora-utils/blockdb_copy.go (+ blockdb_copy_duties.go) — duties copy pass (--no-duties).
• cmd/dora-utils/blockdb_sync.go — local duty computation from beacon state at each epoch
  boundary, with --no-duties and --duties-only flags.

Debug / observability

• New block_duties_size column on the epochs table (pgsql + sqlite migrations).
• db.UpdateEpochDutiesSize records the stored size after each epoch's duties are written.
• /debug/cache BlockDB tab gains a Duties row (count / size) alongside Beacon Blocks
  and Exec Data.

Config

• indexer.disableBlockDBDuties (env INDEXER_DISABLE_BLOCK_DB_DUTIES) to opt out.

Incidental fix: PTC participation on small validator sets

When the validator set is small, PTC seats are duplicated (a validator holds several of the
512 seats). The old per-aggregate percentage divided votes by the unique validator count
(lower than the seat count), which produced >100% participation. Participation is now
seat-based — voted seats over PTC_SIZE (512) — so it stays within 0–100%, and the
labels/tooltips reflect the seat model.

Database changes

• Migration 20260623000000_epoch-duties-size.sql (pgsql + sqlite): adds
  epochs.block_duties_size bigint DEFAULT 0. Backfilled going forward as epochs finalize.

Testing

• blockdb/types/duties_format_test.go — encode/decode round-trips for the packed object,
  per-slot pebble blobs, and the 48-bit index list, including uneven committee splits at 1M
  validators (verifies the writer/reader SplitOffset agreement).
• go build ./..., go vet, and go test ./blockdb/... pass.

Notes & follow-ups

• The blockdb-sync CLI computes duties locally from the beacon state (active-validator
  selection + RANDAO-mix seeding). This path could not be runtime-verified here — worth a
  spot-check that its committees match a node's /committees for a sample epoch before
  relying on it for backfills.
• The CLI tool writes duties directly to the blockdb and does not touch the epochs table,
  so objects it writes won't appear in the DB-derived /debug/cache size stats.

🤖 Generated with Claude Code