Managed Fleet: canonical server, attaching clients, and a self-serve repair hatch

[aarontrowbridge]

Managed Fleet: canonical server, attaching clients, and a self-serve repair hatch

Important

Problem. A user's machines are one logical studio — one canonical chat server, many
panels — but today that topology exists only as ops duct tape. The extension's single
behavior is "spawn a local server", so a client attaching through an SSH tunnel works only
by losing a port race. On 2026-08-07 one fleet failed three times in a day: a silent
three-week fork (279 sessions invisible to the canonical store), a fork reborn in a tunnel
reconnect gap, and a panel stranded dead after a tunnel outage with no self-recovery.
Every failure was found by a human debugging by hand; the product saw nothing.

Approach. Make the fleet a product feature ("Managed Fleet"): three machine-scoped
Server modes (standalone default / server / client); the canonical server as a
product-managed system service that panels always attach to (spawn survives only in
standalone); an extension-owned self-healing Managed Tunnel; an identity handshake plus a
persistent Fleet token on the server; and the repair hatch — Local fallback with Rejoin
merge — so a user keeps working through an outage and loses nothing on reconnect.

Approaches considered. Remote-SSH pivot (zero fleet code, but moves the user's editing
context to the server — rejected as primary); ops-only hardening (keeps the fragility —
rejected); editor-owned canonical server (fleet dies with the editor — rejected);
CLI-daemon tunnel (failures invisible to the panel — rejected; the tunnel is one component
with two launchers instead); mDNS auto-discovery setup (LAN-only — deferred; ssh bootstrap
with manual fallback chosen); anonymous loopback auth (rejected per ADR 0002's threat
model).

Scope. In: server-mode taxonomy + machine-scoped settings; fleet-server enable/disable
commands (service install, stable binary path, upgrade choreography); identity route and
fleet-token enforcement in the vendored server; the rejoin route implementing the guarded
merge; managed tunnel component + extension launcher; client connect flow (ssh bootstrap +
manual); connection state machine + status bar + reconnect command; local fallback + rejoin
flow; Fleet Doctor; golden-fixture tests; migration of the author's own duct-taped fleet.
Out: the tunnel's headless launcher (designed for, built later); systemd service (until a
Linux server appears); mDNS auto-discovery; multi-server fleets.

Assumptions. Fleet users have (or can be walked through) ssh access from client to
server; the canonical store remains SQLite with a single live writer; panels and headless
consumers are the only attach surfaces; Settings Sync stays enabled for users, so all fleet
settings must be machine-scoped by construction.

Acceptance Criteria

1. Never fork. With client mode active and the attach port completely free, booting the
   extension spawns no server process and creates no local store; chat disables with
   actionable copy offering reconnect or Local fallback. (Regression test for the
   2026-08-07 incident class.)
2. Identity verified. Attach succeeds only when the identity handshake returns the
   canonical role and the configured fleet id; a wrong server (any healthy-looking 200) is
   refused with an explicit mismatch error.
3. Self-healing tunnel. Killing the tunnel process produces visible reconnecting
   status, reconnection with bounded backoff, and panel recovery without a window reload.
   Address candidates probe LAN-before-overlay; off-LAN, the overlay candidate connects.
4. Repair hatch closes the loop. Entering Local fallback, creating sessions, and
   reconnecting yields those sessions merged into the canonical store — verified by the
   golden-shard fixtures: no row downgraded (strictly-newer-wins, with a canonical-newer
   fixture proving shard-stale rows are NOT applied, and the equal-timestamp tie-break
   pinned), conflicting event positions skipped and listed in the merge report,
   clock-skewed shards beyond threshold refused, referential integrity clean, and the
   client's shard preserved throughout.
5. Repairable by the user. The Fleet Doctor detects each incident-observed state —
   foreign process on the attach port, stray local store, tunnel down, identity mismatch,
   unmerged fallback data — and offers its one-click fix, with the stray-store fix gated on
   a verified merge (quarantine, never delete). "Disable fleet server" and "Disconnect from
   fleet" restore standalone cleanly.
6. Sync-safe. All fleet settings carry machine scope; an assertion test fails if any
   fleet setting or the fleet token can ride Settings Sync to another machine.
7. Headless server. The canonical service starts at login without any editor, and an
   extension upgrade swaps the service binary at its stable path and returns it to healthy
   without panel intervention on the server machine.
8. CLI parity. Identity and rejoin routes work under standalone opencode serve with no
   extension present (ADR 0002's parity bar).

Key Decisions

• Three Server modes, machine-scoped. standalone (default; the only mode that spawns),
  server (runs the Canonical Server as a system service; its own panel attaches),
  client (never spawns; attaches via Managed Tunnel). One attach path serves both
  non-standalone modes; the spawn path exists only in standalone.
• Canonical Server as product-managed system service. Enabled per machine by an explicit
  command; binary at a stable path; upgrade = replace + restart + health probe. Survives
  editor restarts and reboots.
• Identity handshake before attach. An unauthenticated identity route returns instance
  id, role, fleet id, and version; clients refuse anything but the expected canonical
  identity.
• Fleet token on data routes. Minted at server enable, stored at 0600 (ADR 0001
  discipline), delivered to clients during the ssh bootstrap. Sibling of ADR 0002's per-boot
  password, for the server no extension spawns. Server binds loopback only.
• Managed Tunnel: one component, two launchers. Extension launcher now (spawn, supervise,
  reconnect with backoff, status surface); headless launcher deferred but designed for.
• Local fallback + Rejoin merge in v1. Fallback is a visible, deliberate state; rejoin
  ships the local shard to the server, which merges as single writer under the verified
  policy; unmappable schema drift refuses and preserves rather than merging blind.
• The fork server owns identity and rejoin routes (ADR 0002 seam precedent: server-owned,
  CLI-parity).

Data Contracts

• Identity payload: instance id, role (canonical | standalone), fleet id, version.
  Clients pin the canonical instance id at bootstrap (not just fleet id) and verify it
  in every handshake — a restored or cloned server claiming the same fleet id is refused.
• Fleet token file: 0600 at a fixed per-user path; never in settings, never synced.
• Rejoin merge policy (fixture-locked): id-guarded inserts; strictly-newer-wins per row with
  the re-homed directory excluded and equal timestamps keeping canonical (tie-break pinned
  by fixture); the handshake measures clock skew and refuses merges beyond a defined
  threshold, since the comparator crosses machines' clocks; event stream guarded on
  (aggregate, sequence) position — a position both sides hold with different content is
  skipped-and-reported, never silently renumbered or merged, and every skipped event appears
  in the merge report returned to the client; columns map by name across drift, and drift is
  unmappable — refusing the merge while preserving the shard — when any shard column
  carrying non-default values has no canonical counterpart; one FK-off transaction per shard;
  foreign-key check + quick check after.
• Client settings (all machine-scoped): mode, server ssh target, ordered address candidates,
  fleet id, pinned canonical instance id.
• The Doctor's stray-store fix is gated on the store's merge state: an unmerged Local
  fallback store is quarantined, never deleted, and unjoined fallback data is its own
  reported incident mode.

Constraints & Invariants

• Exactly one canonical server per fleet; the single-writer rule on the store is absolute.
• A client machine must never spawn a server or create a local store outside Local fallback.
• Nothing fleet-related may ride Settings Sync.
• The server binds loopback only; mutation routes refuse otherwise (ADR 0002 graft).
• A merge that cannot be verified does not run; the shard is preserved for manual recovery.

Prior Art

• Client machines silently fork the chat DB: extension always spawns a local server (needs attach-only mode) #279 — attach-only mode, filed from the incident (this issue supersedes
  its scope; Client machines silently fork the chat DB: extension always spawns a local server (needs attach-only mode) #279 is the minimal version of the attach half).
• ADR 0001 (at-rest secret discipline), ADR 0002 (server-route seam; per-boot password;
  loopback graft).
• The 2026-08-07 recovery runbook and golden shards (fleet recovery archive on the author's
  server); the fleet playbook in the Amico skill set (detection signatures recorded there).

Source

Designed 2026-08-07 in a grill-with-docs session following the same-day triple fleet failure
on the author's own machines; terminology landed in CONTEXT.md (Fleet & serving) and the
decision rationale in docs/adr/0005-managed-fleet.md (PR #282).

Notes

• The author's three-machine fleet is the migration first-customer: the Doctor should detect
  today's shim + launchd tunnel + ssh-config duct tape and offer replacement.
• The tunnel component's headless launcher (amico fleet tunnel) serves future panel-less
  consumers (scheduled agentic jobs); design for it, build it when Notturno needs it.
• During this design the team confirmed the target audience is open-source users running
  their own multi-machine setups; the default single-machine experience is unchanged.
• Tracked from review (non-blocking advisories folded in above): pinned instance-id
  verification, clock-skew threshold, skip-and-report for conflicting event positions,
  unmappable-drift refusal rule, expected-downgrade fixture set, Doctor merge-gating.
  Carried forward, not folded: detecting two LIVE servers claiming the same fleet id
  (a canonical-leader lease or enable-time registry) — follow-on once multi-server
  misconfiguration proves worth its own surface.