Container Dev Mode: host CLI, embedded registry, and VM push path

[jetm]

Container Dev Mode is the inner dev loop for a containerized app on an
immutable Avocado OS device: edit on the host, and only the changed image
layer is hot-reloaded onto the running device — no reflash, no full re-ship.

This adds the host half:

• a per-project content-addressed OCI store, split write/read listeners, and
  a control WebSocket (three-listener model), all over a per-project pinned CA;
• Basic-gated write token vs Bearer read/control token (split-credential model);
• an engine-driver watcher (docker/podman) that syncs only changed layers and
  selects PUSH vs INGEST by host topology, plus a cross-arch guard;
• avocado container dev up/sync/status/down/prune;
• the authenticated VM push path: on the macOS/VM fast path the guest pushes to
  a routable HTTPS write listener whose per-project CA is delivered at up,
  never baked.

Verification: unit + integration suites (container_dev lib, plus
container_dev_e2e, container_dev_security, container_dev_arch) all pass;
the VM write path was validated end-to-end 8/8 against a local QEMU docker+ssh
engine VM. Phase 0 de-risk findings are recorded in
docs/container-dev/phase0-findings.md.

[jetm]

Self-review (cold read of my own +10k PR). The security core is clean: TLS drops the CA key (bootstrap carries only CA cert + read token), tokens are 256-bit CSPRNG, the store closes digest/tag path-traversal, every docker/podman call is argv-only (no shell), and the bind addresses match the threat model. Six correctness/robustness findings inline - the two worth acting on first are the recycled-PID signal hazard in down/sync and the cross-arch guard never being wired into the live up path (confirm whether that's intentionally deferred - if so it shouldn't read as complete).

[jetm]

Independent second review of the increment since f69b3e2 (+444/-55, 4 files), traced cold at 2295751. Ten findings inline, two of them high.

The two I would fix before this merges are both about a guard that does not guard:

• dev.rs:365 - the arch guard this diff wires up is fail-open while the arch book is empty, and reconcile never consults the book at all. Since up spawns the watcher before it bootstraps the device, the empty-book window is the normal case, not an edge: a rebuild pushed before the device ever connects gets recorded into desired state and delivered verbatim on the device's first frame, architecture unchecked. The comment right below claims this decorator prevents exactly that.
• dev.rs:436 - SessionLock::acquire runs after the state write, the token mint, all three binds, the watcher spawn, and both SSH deliveries, so it cannot exclude a second up from any of them. Two of its consequences need no unusual configuration at all: a concurrent status/sync/down in the two-statement window makes acquire fail ENOENT (it lacks .create(true)) after everything is already bound and the device is bootstrapped, and because every read path unlinks the lock inode, mutual exclusion does not survive a single down.

Two findings are backed by running the code rather than reading it. Deleting session.touched = Instant::now(); (registry.rs:407) leaves the test suite at 10/10 passing, so patching_a_session_keeps_it_alive_past_the_ttl asserts nothing about the TTL - it trips the workspace CLAUDE.md rule that every test must fail when the implementation is wrong. And the default-config second-up really does die at EADDRINUSE first (mio sets only SO_REUSEADDR), which is why the destructive half of the lock finding is scoped to the two env overrides rather than claimed outright.

One doc contradiction worth fixing on its own: registry.rs lines 62-63 say only the gap between chunks counts against the TTL, while lines 70-72 say bodies are buffered as Bytes before anything inspects them. The second is what the code does, so the first is wrong, and the commit body repeats it as an absolute ("never evicted mid-push").

Traced and cleared, so nobody re-audits: HelloArchBook's derived Clone does share one map (the Arc is cloned, not the BTreeMap), so the book genuinely is shared between ControlServer and the guard - that was my main suspicion and it is fine. The Arc<dyn DeviceArchBook> double-wrap is the dyn coercion, not redundancy. EngineArchProbe::new(...as_ref()) on a temporary Box is sound because new copies out a &'static str. A refusal from the guard does make do_sync_and_notify skip the notify, as its comment claims. require_basic_write wraps the whole router, so an anonymous request causes no large allocation despite the body limit. The loopback-only doc rewrites are accurate in both directions. read_blob to blob_size is behavior-preserving and the digest is still validated. And I dropped one candidate outright: the new test's Instant::now() - TTL - 1s does not panic on a freshly booted host, since Linux Instant is a signed-seconds Timespec - verified on a box with 3,700s uptime.

Scope note: this was the requested light-to-moderate pass - three finder angles plus four verifiers, no separate gap sweep. avocado-cli has no repo-level CLAUDE.md, so the workspace one governs.

[jetm]

Second-reviewer pass on my own PR, cold, on the change since the last round. Twelve findings inline, most severe first. Four of them are the ones I'd land before anything else: the session lock no longer proving pid ownership (1), the arch filter that only covers one of two notify paths (2), the mid-transfer upload session teardown (3), and registry/uploads/ never being swept (4).

Findings 3, 4, 5, 6 and 12 all trace to the same shift in this round: the blob path moved from buffer-in-memory to stream-to-disk. That was the right move, but the invariants that used to fall out of buffering for free - a request either arrived whole or not at all, an abandoned upload died with the process, a blob could only reach disk if it fit in RAM, and a size cap existed - each needed an explicit replacement, and only the manifest path got one.

Findings 9, 10 and 11 are a set: three tests whose doc comments claim a falsifier they do not have. In each case I checked by making the change the comment says would fail, and the suite stayed green. Worth fixing as a group, because together they mean the lock mechanism and up's guard wiring are effectively untested.

Below the cap - confirmed, lower severity

• store.rs:390 - BlobUpload::finish re-implements blob_path (store.rs:326) inline plus write_blob's exists/create_dir_all/persist tail, so the blob layout now has two authors while only readers go through blob_path. blobs_root is also initialized from self.root and then has "blobs" joined onto it, and format!("sha256:{hex}") hardcodes an algorithm two lines above a parse_digest that accepts any.
• store.rs:341 - parse_digest's doc comment is orphaned onto pub struct BlobUpload, so cargo doc renders the new public type's summary as "Split an OCI digest into its (algorithm, hex) components", and parse_digest at store.rs:402 ends up undocumented.
• store.rs:360/:378 - unbuffered write_all per hyper frame means syscall count per GiB is set by link behavior rather than bounded by code; flush() before persist is a documented no-op on File, and with no sync_all a host crash can leave the blob path present while its tail never reached disk, which has_blob and the HEAD dedup probe both report as complete.
• watcher.rs:601 - record_hello has no production callers left (only watcher.rs:962/990 and six sites in tests/container_dev_arch.rs) and plants holders: 1 with no lease, so its entry can never be released. The struct doc at watcher.rs:579 still says reconnects "overwrite" while record_session six lines below says they refcount.
• ws.rs:441 - the inner Option<DeviceArchLease> is never None, since the only Some(..) return arm always supplies a lease, so if lease.is_some() is a branch that cannot be taken.
• ws.rs:1015 - notify_refuses_an_event_with_no_image_id labels frames.is_empty() "The decisive assertion", but an empty desired digest compares equal to the empty running_digest, so that assertion passes with the digest.is_empty() bail deleted. Only the third assertion is decisive. assert!(result.is_err()) also never checks which error.
• tests/container_dev_arch.rs:323 - pinned_ca_connector is now the fourth byte-identical copy (also container_dev_security.rs:131, container_dev_e2e.rs:353, ws.rs:1064), and tests/common/mod.rs already exists as the shared home.
• dev.rs:949 + watcher.rs:740 - two engine spawns per image per manual sync, to read two fields of the same image inspect object, awaited serially per image.

Checked and refuted

Recording these so nobody re-derives them. Digest verification and path traversal in the new write path are sound: finish compares against the incrementally computed hash before any filesystem move, the dedup branch is only reachable after the hash matched, and parse_digest's permissive algorithm parsing is unreachable because a non-sha256: digest fails equality first. Auth is not bypassable - require_basic_write is a from_fn_with_state layer over the whole router, applied outside routing, so it precedes the /v2/ ping and unrouted methods too. No listener moved this round. Secret file permissions are fine: session.lock is 0644 but always empty (it exists only as a stable flock inode), and persisted blobs keep NamedTempFile's 0600. No DeviceArchLease::drop self-deadlock and no lock-order cycle between image_arches and desired. Instant subtraction in the test helpers does not panic on Linux, and the Windows job is cargo check only. The missing -- before image in resolve_image_id is real in isolation but byte-identical to the pre-existing EngineArchProbe::image_arch shape this round does not touch.

No rust.md violation: edition 2021, anyhow + thiserror matching the CLI row, and no [lints] table, rustfmt.toml, clippy.toml or workspace introduced.

[jetm]

One finding, carried over from avocado-os#46 after tracing it to the code that actually owns it. The os#46 side is resolved; this is the residual half, and it partly corrects what I said there.