flashbox-l1: replace lighthouse with sync-proxy - #197
Draft
MoeMahhouk wants to merge 6 commits into
Draft
Conversation
No consensus client runs inside the image anymore. An external sync cluster (CL + reference EL + nginx mirror in our VPC) mirrors its Engine API calls to each flashbox; sync-proxy on the host receives them on :8552 in mirror mode (immediate empty ACK, async forward) and drives the searcher EL on :8551. - build sync-proxy reproducibly via make_git_package (Go), drop the Rust toolchain/C deps and libsnappy that only lighthouse needed - engine-jwt.service fetches the JWT secret shared with the sync cluster from Vault (ENGINE_API_JWT_SECRET) into /tmp/jwt.hex; the container Requires it so the bind mount never sees a missing file. QEMU devmode uses a fixed test secret. sync-proxy itself is JWT-agnostic (pure pass-through). - factor vault_read_secret out of vault.sh so both observability and the JWT fetch share the Vault login - sync-proxy logs go to /persistent/sync_proxy_logs (rotated via /etc/logrotate.d) and are mounted read-only into the container at /var/log/sync-proxy, replacing /var/log/lighthouse sync-proxy is pinned to commit 870715b pending upstream merge of the mirror-mode/metrics PRs.
- drop the CL P2P (9000) in/out rules and the matching netns drops - accept sync-proxy :8552 inbound in ALWAYS_IN (engine drive must survive maintenance mode) from internal source addresses only; the container netns drops :8552 since the podman subnet is internal too - searchersh restart-lighthouse -> restart-sync-proxy, sudoers updated
Scrape sync-proxy metrics locally and derive the remote-written booleans flashbox:searcher_receiving_engine_calls (EL accepted a forwarded newPayload/forkchoiceUpdated in the last 5m) and flashbox:sync_proxy_is_up. Raw syncproxy_* series never leave the box.
systemd opens StandardOutput=append: before running any Exec*, so having ExecStartPre create /persistent/sync_proxy_logs failed with 'Failed to set up standard output', and the container then failed to bind mount the missing directory. sync-proxy-init.service now creates it after /persistent is mounted; sync-proxy.service and searcher-container.service Require it. Found by the QEMU e2e run.
…L P2P toggle refuses to enter production unless the container netns carries the expected DROP rules. Those still named the CL P2P port 9000, which init-container.sh no longer sets, so production toggle would have been refused. Check the tcp/8552 sync-proxy drop instead.
- move vault.sh into the common module: fetch-engine-jwt (flashbox-l1) and flashbox-observability-setup both source it, so it must not depend on the observability module being included - flashbox:searcher_receiving_engine_calls: 'or vector(0)' so the boolean is an explicit 0 before the first-ever successful forward instead of absent - fix the retry-window comment in fetch-engine-jwt, document the container's hard dependency on the JWT fetch and the sync-proxy logrotate policy in the readme, replace the lighthouse example in DEVELOPMENT.md
MoeMahhouk
commented
Aug 25, 2026
Member
Author
There was a problem hiding this comment.
If we decide to just hardcode the JWT, then we can remove such dependency because this would cause the searcher container to simply fail to initialize and load in case the vault is not reachable our timeouts more than 5 minutes.
Hardcoding the JWT secrets would simplify also the changes and reduce the need for the engine-jwt service and vault extraction/parsing additions. CC @ilyaluk @pablin-10 , deferring to you to decide on what is best from operations perspective. We already discussed that this is not a security concern in this particular use-case/product
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the in-image Lighthouse beacon node from flashbox-l1 and replaces it with
sync-proxyin mirror mode. Consensus is now provided by an external sync cluster (CL + reference EL + nginx mirror) in ourown VPC, which mirrors its Engine API calls to every flashbox; sync-proxy on the host receives them on :8552 and drives the searcher EL on :8551 exactly as lighthouse did.
Changes
Build
mkosi.build: Gomake_git_packagebuild of sync-proxy, pinned to commit870715b(tip of the mirror-mode/metrics PR stack: feat: Add optional Prometheus metrics endpoint sync-proxy#24 see below). Drops the Rust toolchain, the C build deps andlibsnappy1v5that only lighthouse needed. Image shrinks 312 MB → 285 MB.JWT
engine-jwt.servicefetches the JWT secret shared with the sync cluster from Vault (ENGINE_API_JWT_SECRET, hex-64, in the existingnode/<suffix>blob) into/tmp/jwt.hex; the container bind mount at/secrets/jwt.hexis unchanged.searcher-container.serviceRequires=it so podman never sees a missing file. QEMU dev images use a fixed test secret. Bounded retry (~5 min), then fail loudly.vault.sh: factorvault_read_secretout ofvault_fetchso observability and the JWT fetch share the Vault login. No behavior change for metrics.Authorizationheader through and the searcher's EL validates it against the same shared secret.Service / logs
sync-proxy.service:-addr 0.0.0.0:8552 -builders http://localhost:8551 -mirror-mode -metrics-addr 127.0.0.1:9106. Logs to/persistent/sync_proxy_logs(rotated via/etc/logrotate.d, which the commonlogrotate.confnow includes) and bind-mountedread-only into the container at
/var/log/sync-proxy, replacing/var/log/lighthouse.sync-proxy-init.service(oneshot) creates the log dir — systemd opensStandardOutput=append:before anyExecStartPre, so the unit can't create its own.Firewall
ALWAYS_INtcp/8552 fromSYNC_PROXY_ALLOWED_SRC(currently10.0.0.0/8, see TODO); always-on because engine drive must survive maintenance mode.Searcher UX
searchershrestart-lighthouse→restart-sync-proxy; sudoers updated.Observability
sync-proxy(keepssyncproxy_.*locally only) and remote-written booleansflashbox:searcher_receiving_engine_calls(EL accepted a forwarded newPayload/forkchoiceUpdated in the last 5m — distinguishes EL down / hanging / rejecting the JWT from healthy) andflashbox:sync_proxy_is_up. promtool-validated.Docs: readme firewall table, netns rules, searcher contract, boot order.
Testing
870715bwith-mirror-mode/-metrics-addr, units wanted byminimal.target, no lighthouse leftovers.engine_newPayloadV1signed with the shared secret (successcounter 0→1), a wrong-secret token was rejected by the EL (http_error0→1), non-engine calls are never forwarded, ACK latency ~1.5 ms, logs on/persistentreadable from the container.Follow-ups / coordination with devops
ENGINE_API_JWT_SECRET(hex-64) to the Vaultnode/<suffix>blob, same value the sync cluster's CL uses.SYNC_PROXY_ALLOWED_SRCinfirewall-configto the sync cluster subnet / mirror IP.mkosi.buildfrom870715bto a release tag once flashbots/sync-proxy PR stack (bounded timeouts, mirror mode, metrics) are merged.restart-lighthouse→restart-sync-proxyand/var/log/lighthouse→/var/log/sync-proxy.