serve capsules by STREAMING — three paths still read a whole capsule into RAM

[MichaelTaylor3d]

Task

Serve capsules by streaming, never by reading the whole capsule into RAM. User directive, 2026-08-22:

"dig-node should be streaming data out of capsules not reading the whole capsule into ram to serve it.
This is antithetical to what the capsule was supposed to support and achieve."

The capsule format was designed for seeking and ranged verification. Any path that materialises a whole
capsule to answer a request is working against the format, and it converts a ~200-byte request into a
~135 MiB allocation.

Measured — and the good news is most of the serve path ALREADY streams

Do not rewrite what is already right:

• dig.getContent / dig.getCapsule seek and return one WINDOW at a time (lib.rs:1347). The
  content serve path is already windowed.
• dig.fetchRange exists with a per-range verification contract (seams::content::range_frame,
  lib.rs:105-106, #1577).
• Peers range-pull rather than whole-pull (lib.rs:2727).

So the format's streaming affordances exist and are used. The defect is the paths that bypass them.

The paths to fix

1. lib.rs:1842 — std::fs::read(&path) of the whole module to extract the data-section blob and decode
   manifests. It is careful about lifetime (it drops the module buffer as soon as the blob exists, and drops
   the blob before rendering) but it still peaks at whole-capsule resident. The data section's offset and
   length are known from the module header — seek to it instead of reading the file to find it.
2. lib.rs:1311 and lib.rs:2115 — two more std::fs::read(&path) of a module. Determine what each
   needs and whether a seek serves it.
3. dig.getMetadata renders a WHOLE data section (lib.rs:1347) and is on the anonymous public-read
   allowlist. Its own doc concedes a hostile capsule could turn a ~200-byte request into a ~100 MB response
   with 3-4 in-RAM copies, and today it is bounded by refusing oversized sections with METADATA_TOO_LARGE.
   That is a cap, not streaming. The doc argues the section is "a whole JSON object, not a byte stream, so
   it cannot be windowed" — assess whether that is genuinely true or an artefact of parsing eagerly; a
   streaming JSON reader would change the answer. If it really cannot be windowed, say so and leave the cap.

Enumerate every read of a capsule/module file rather than fixing the three named — git grep for
fs::read, read_to_end, and any Vec<u8> sized from a module length. The named ones are what one grep
found, not a complete set.

What "done" looks like

• No serve path holds a whole capsule resident. State the new peak-resident figure per path and how you
  measured it, not that it "should be lower".
• Verification is preserved. Content is trusted because it verifies against the merkle root; a streaming
  read must verify incrementally, not defer verification until a whole buffer exists — which would reintroduce
  the allocation it removes. Cite how each streamed path verifies.
• A test that asserts peak resident memory stays bounded while serving a capsule far larger than the
  bound. Prove it red against the current fs::read implementation.

Fix the infra doc, which is miscited and load-bearing

modules/services/rpc.dig.net/infra/variables.tf:75-84 justifies a 2 GiB instance floor with:

"dig-node reads a whole capsule into RAM to serve it (std::fs::read, dig-node-core lib.rs:1004)"

lib.rs:1004 is not a capsule read. It is the staging-directory recursion, which already enforces a byte
budget before reading (max_total_bytes, stat-then-abort). So the sizing rationale cites the wrong line,
and it also overstates the serve path, which windows.

Correct the citation to whichever read actually justifies the floor after this work, or lower the floor if the
answer is that nothing does. An infra sizing comment that cites the wrong line is how a host gets sized for
a defect that was already fixed — or stays undersized for the one that was not.

Supersedes

This replaces the original framing of this ticket, which proposed a memory-derived semaphore with
backpressure. That bounded the symptom — concurrency × 135 MiB — and was the right shape only if the
135 MiB were irreducible. It is not. Stream first. If a bound is still wanted afterwards as
defence-in-depth, it is a much smaller and less costly one, and it can be a follow-up rather than a
precondition.

Why this matters beyond one host

node-rpc.dig.net is both the public bootstrap peer and the CloudFront origin, so peer-port memory pressure
5xxs the browser read tier (dig_ecosystem#923). But every self-hosted node has this shape — that is why
the fix belongs here and not in Terraform.

Sequencing

Do not fold into dig-node PR#245 (client-side bootstrap seed) or PR#295's successors. Own branch. dig-node
has live lanes on crates/dig-wallet/src/sage/* — this is dig-node-core, so no overlap, but use your own
worktree.

Parent

https://github.com/DIG-Network/dig_ecosystem/issues/923

[MichaelTaylor3d]

Re-scoped by user directive — a semaphore was the wrong fix

User, 2026-08-22: "dig-node should be streaming data out of capsules not reading the whole capsule into ram to serve it. This is antithetical to what the capsule was supposed to support and achieve."

The original framing here was a memory-derived semaphore with backpressure. That bounds concurrency × 135 MiB, and it is the right shape only if the 135 MiB is irreducible. It is not. Stream first; a bound afterwards is much smaller and can be a follow-up rather than a precondition.

What measurement changed

Most of the serve path already streams, and that part should not be rewritten:

• dig.getContent / dig.getCapsule seek and return one WINDOW at a time (lib.rs:1347)
• dig.fetchRange exists with a per-range verification contract (lib.rs:105-106, #1577)
• peers range-pull rather than whole-pull (lib.rs:2727)

So the format's streaming affordances exist and are used. The defect is the paths that bypass them — lib.rs:1842, :1311, :2115, plus dig.getMetadata, which renders a whole data section on the anonymous allowlist and is bounded by a cap rather than by streaming.

And the infra doc that justified the sizing is miscited

modules/services/rpc.dig.net/infra/variables.tf:75-84 justifies a 2 GiB floor with "dig-node reads a whole capsule into RAM to serve it (std::fs::read, dig-node-core lib.rs:1004)".

lib.rs:1004 is not a capsule read. It is the staging-directory recursion, which already enforces a byte budget before reading (stat-then-abort against max_total_bytes). So the sizing rationale points at the wrong line and overstates the serve path.

That matters beyond tidiness: an infra comment citing the wrong line is how a host gets sized for a defect that was already fixed — or stays undersized for the one that was not.

Format-level check running in parallel

User, follow-up: "the capsule should support streaming and if it doesnt that needs to be completed. It was an original requirement of dig-store."

A lane is measuring whether the .dig/DIGS format and the dig-store crates expose the streaming primitives this work needs, or whether the format side has to be completed first. If the format cannot stream, that is the prerequisite and this ticket waits on it.

[MichaelTaylor3d]

BLOCKED on dig-store — and dig-node is NOT at fault here

Measured the format. The verdict reverses this ticket's premise about where the defect lives.

dig-node already streams what it can. read_module_window (crates/dig-node-core/src/seams/dig_peer/module_serve.rs:165-189) does File::open + seek + read_exact over the raw bytes, with the rationale stated at :175-178 — a 512 MiB capsule served in 4 MiB windows would otherwise cost ~256 GiB of IO per pull. And get_capsule notes "the whole point is that no buffer of the whole module ever exists" (lib.rs:3245-3247). That is opaque byte transport of an inert file, and it is sound because the receiver verifies the assembled whole against the chain-anchored root.

The three fs::read paths I flagged are the ones that must INTERPRET the capsule — lib.rs:1311 reads then calls serve_blind, which instantiates the wasm. Those cannot be windowed by dig-node at any effort, because the guest needs the blob in wasm linear memory.

Why the format cannot help yet

resource_leaf(ciphertext) = SHA-256(whole resource ciphertext) (digstore-core/src/merkle.rs:140-142). Chunks are not committed individually anywhere in the merkle tree, so there is nothing to check a prefix against. A streaming read that defers verification until the whole resource exists has not removed the allocation — and deferring is not a choice the implementation made, it is the only thing the merkle shape permits.

Also: a capsule is a WASM module, not a data container. Reading any section means parsing the whole wasm binary (digstore-compiler/src/inject.rs:179-206), and seg.data.to_vec() at :193 makes a second full copy.

Corrections to what I wrote on this ticket earlier

1. dig.fetchRange's "per-range verification contract" is not chunk-granular merkle verification and cannot be — no per-chunk commitment exists. Whatever it asserts, it is not the format's merkle tree at range granularity.
2. dig.getContent/getCapsule windowing is not the format streaming. It is dig-node windowing an opaque file. So the affordances do not "exist and get used unevenly" — the transport one exists and is used; the format one does not exist.
3. dig.getMetadata's doc is half wrong and the wrong half does not help. A streaming JSON reader would let it emit incrementally — but getting to the section's bytes at all requires the whole-wasm parse, so streaming the JSON removes maybe one in-RAM copy and none of the ~100 MB. METADATA_TOO_LARGE is the correct remedy given the format; the doc should re-justify it on extraction cost rather than on JSON's shape. Metadata is also unverified plaintext (section 6, outside the merkle root), so emitting it incrementally would be a worse property.

Filed DIG-Network/digs#49 for the dig-store work. This ticket waits on it and should not be worked as a dig-node defect.