Skip to content

Route snapshots through the snapshot_blob lane (inline ≤1MiB, R2 above)#6

Merged
lightsofapollo merged 2 commits into
mainfrom
fix/snapshot-blob-path
Jun 11, 2026
Merged

Route snapshots through the snapshot_blob lane (inline ≤1MiB, R2 above)#6
lightsofapollo merged 2 commits into
mainfrom
fix/snapshot-blob-path

Conversation

@lightsofapollo

Copy link
Copy Markdown
Owner

Problem

Sharing any file whose snapshot ciphertext exceeded the relay's 256 KiB event cap (HARD_MAX_EVENT_BYTES) was silently broken. publish_snapshot inlined the full SnapshotPlaintext (markdown + anchor index) into the SnapshotCreated event envelope; the relay rejected it with 413; the unclassified poison envelope retried forever at the head of the outbox — blocking the WebRTC signaling envelopes queued behind it, so the reviewer could never join. A ~100 KB markdown file (~1 MB sealed) hit this every time (production logs: webrtc: signaling drain failed: io: relay 413 Payload Too Large).

Fix

Snapshot bytes now travel as kind=snapshot_blob envelopes sealed under the room's snapshotKey (relay cap: policy.maxSnapshotBytes, 5 MiB), per the already-locked spec (relay-spec.md §R2 spillover, crypto-spec.md §Nonce Discipline, decision #14):

  • ≤ 1 MiB sealed — the blob envelope rides the normal outbox, enqueued before the SnapshotCreated event so peers receive bytes-then-pointer in serverSeq order (BlobRef storage=mailbox).
  • > 1 MiB sealed — R2 spillover: presign POST /blobs, PUT nonce||ciphertext||tag (AAD-bound to the wrapper envelope so blob bodies can't be swapped between envelopes), and the outbox envelope carries only the encrypted canonical-JSON BlobRef (storage=r2).

The SnapshotCreated event stays ~800 B on the wire: inline_snapshot is never serialized; encrypted_blob_ref points at the blob by envelopeId.

Receivers persist decrypted blobs (rooms/<room>/blobs/<envelopeId>.bin), resolve R2 refs via the new download-presign endpoint, verify length + content hash against the Ed25519-signed event's BlobRef, and the daemon rehydrates inline_snapshot at the IPC boundary — the Svelte frontend is unchanged.

Relay: implements the spec'd-but-missing download presign — a cap-less GET /v2/rooms/:roomId/blobs/:envelopeId routes to the DO (admission-auth'd; no PoW on GETs per spec) and mints the cap the cap-bearing GET consumes.

Spec amendment (crypto-spec.md): the BlobRef wrapper seals under snapshotKey, not the originally pinned eventKey — the inbound pipeline keys strictly by envelope kind, and a receiver can't know bytes-vs-BlobRef until after decrypting.

Owner side also persists the SnapshotNode (plaintext + BlobRef), making the WebRTC RequestSnapshot recovery path functional (latest_snapshot_for_file previously always returned None in production).

Verification

  • cargo test — 924 tests across binaries, including new coverage: blob envelope round-trip + cross-key isolation + R2 body wrapper-binding, store blob persistence, inline-vs-R2 routing in publish_snapshot (wiremock presign/PUT), inbound persistence rules, and rehydration integrity checks.
  • Relay vitest — 296 tests; 3 new presign-endpoint cases, and the upload/download round-trip now drives the real endpoint instead of minting caps via the test helper.
  • New scripts/test-snapshot-blob-e2e.sh — boots a local relay + owner + reviewer daemons, folder-shares a 28 KB doc (mailbox lane) and a 150 KB doc (R2 lane), and asserts the reviewer renders both with full markdown and no 413 in any log. 5/5 passing.
  • task check:size — release binary 30.25 MiB, under the 32 MiB gate.

Notes / follow-ups

  • The production relay needs a deploy for the R2 download endpoint; files sealing ≤ 1 MiB (including the original failing file) work with the client fix alone.
  • The anchor index inflates snapshots 9–20× depending on block density, so the effective ceiling is ~250–500 KB of markdown. Oversized files now fail loudly at presign instead of poisoning the outbox. Follow-ups worth filing: classify 413 as terminal in the outbox (dead-letter instead of retry-forever), surface share failures in the Share dialog, and consider compressing snapshot blobs / slimming the anchor index.
  • Daemons with pre-fix poisoned outboxes still carry the oversized kind=event envelopes; those rooms need to be stopped/deleted and re-shared.

🤖 Generated with Claude Code

Sharing any file whose snapshot ciphertext exceeded the relay's 256 KiB
event cap (HARD_MAX_EVENT_BYTES) was broken: publish_snapshot inlined the
full SnapshotPlaintext (markdown + anchor index) into the SnapshotCreated
event envelope, the relay rejected it with 413, and the unclassified
poison envelope retried forever at the head of the outbox — blocking
WebRTC signaling behind it, so the reviewer could never join. A ~100 KB
markdown file (~1 MB sealed) hit this every time.

Snapshot bytes now travel as kind=snapshot_blob envelopes sealed under
the room's snapshotKey (relay cap: policy.maxSnapshotBytes, 5 MiB):

- ≤ 1 MiB sealed: the blob envelope rides the normal outbox, enqueued
  BEFORE the SnapshotCreated event so peers receive bytes-then-pointer
  in serverSeq order (BlobRef storage=mailbox).
- > 1 MiB sealed: R2 spillover per relay-spec — presign POST /blobs,
  PUT nonce||ciphertext||tag (AAD-bound to the wrapper envelope so blob
  bodies can't be swapped between envelopes), and the outbox envelope
  carries only the encrypted canonical-JSON BlobRef (storage=r2).

The SnapshotCreated event itself stays small (~800 B): inline_snapshot
is never on the wire (decision #14); encrypted_blob_ref points at the
blob by envelopeId. Receivers persist decrypted blobs in the store
(rooms/<room>/blobs/<envelopeId>.bin), resolve R2 refs via the new
download-presign endpoint, verify length + content hash against the
Ed25519-signed event's BlobRef, and the daemon rehydrates
inline_snapshot at the IPC boundary — the Svelte frontend is unchanged.

Relay: implement the spec'd-but-missing download presign — a cap-less
GET /v2/rooms/:roomId/blobs/:envelopeId now routes to the DO
(admission-auth'd, no PoW per spec for GETs) and mints the download cap
the cap-bearing GET consumes.

crypto-spec amendment: the BlobRef wrapper seals under snapshotKey (not
the originally pinned eventKey) — the inbound pipeline keys strictly by
envelope kind, and a receiver can't know bytes-vs-BlobRef until after
decrypting.

Owner side also persists the SnapshotNode (with plaintext + BlobRef),
making the WebRTC RequestSnapshot recovery path functional.

Verification: cargo test (924 across binaries), relay vitest (296, incl.
3 new presign-endpoint cases + round-trip now driven through the real
endpoint), and a new scripts/test-snapshot-blob-e2e.sh that boots a
local relay + owner + reviewer daemons and proves both lanes end-to-end
(reviewer renders 28 KB and 150 KB docs; no 413 in any log). Release
binary 30.25 MiB, under the 32 MiB gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jun 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
attn Ready Ready Preview, Comment Jun 10, 2026 11:48pm

Request Review

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lightsofapollo lightsofapollo merged commit 2d7ae8a into main Jun 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant