feat(isolation): implement the RFC 0012 sandbox architecture

[drew]

Summary

Implement RFC 0012 as one combined change. This PR defines the Isolation Backend and adds the OpenShell implementation: the Sandbox Protocol, OpenShellRuntimeBackend, separate openshell-supervisor and openshell-sandbox binaries, transport-neutral JWT authentication, and the Docker, Podman, Kubernetes, and VM integrations.

The supervisor stays outside the agent workload. Each compute driver creates the runtime resources, protected channel, and outer network fence needed by its platform.

Related Issue

Part of #1737. Implements the architecture accepted in #2048.

Changes

• Add the openshell-isolation-interface crate and backend conformance tests.
• Add OpenShellRuntimeBackend and the authenticated OpenShell Sandbox Protocol.
• Add separate supervisor and sandbox runtime binaries.
• Bind gateway and sandbox JWT profiles to sandbox identity, runtime generation, session lineage, credential epoch, and expiry.
• Add process ownership, binary identity, TCP and DNS mediation, loopback connections, and fail-closed reconnect behavior.
• Integrate the architecture with VM, Docker, Kubernetes, and Podman.
• Add TCP and DNS performance harnesses.
• Publish separate glibc supervisor and static musl sandbox artifacts.

Driver placement

• VM: host supervisor, guest sandbox runtime, authenticated vsock, no guest NIC.
• Docker: companion supervisor container, workload container, authenticated Unix socket, network_mode=none on the workload.
• Kubernetes: supervisor Pod, workload Pod, private mTLS Service, and NetworkPolicy fence.
• Podman: separate supervisor and workload containers, authenticated Unix socket, no workload network.

Testing

• mise run pre-commit
• Targeted isolation, authentication, and driver tests
• Driver E2E workflows from the original stack
• Remote CI is the final source of truth for the combined PR.

Checklist

• Follows Conventional Commits
• Authored commits are signed off (DCO)
• Driver-specific implementation stays in its respective driver crate and supporting deployment/tests

Consolidated stack

The following stacked PRs are now merged into this PR:

• feat(vm): run the supervisor outside the guest workload #2945 — VM driver
• feat(docker): isolate workloads behind a companion supervisor #2965 — Docker driver
• feat(kubernetes): isolate workloads behind a dedicated supervisor #3144 — Kubernetes driver
• feat(podman): isolate workloads behind a separate supervisor #3230 — Podman driver
• perf(isolation): add TCP and DNS benchmark harnesses #3229 — performance harnesses

#2941 is superseded by the canonical RFC merged in #2048. #3151 was folded into this PR.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

Label test:e2e applied for b06e0b1. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[drew]

/ok to test

[drew]

/ok to test

[drew]

/ok to test

[drew]

/ok to test

[johntmyers]

gator-agent

Re-check After Author Update

Thanks @drew. I reviewed the session-successor and retry commits at current head 0db1fcf6 against the durable feedback ledger and the VM, Docker, Kubernetes, Podman, and performance layers in the RFC 0012 stack. The signed generation, predecessor, and monotonic rotation now prevent an unrelated same-sandbox bearer from taking over an idle boundary, so GATOR-6143f24e-01 is resolved. The bounded critical-only review found one newly introduced upgrade-path Critical.

Action required: @drew, add a fail-closed compatibility path for persisted sandboxes that have no lineage annotations, plus startup and explicit-start regression coverage, then push an updated head.

Blocking findings:

• GATOR-0db1fcf6-01: the mandatory lineage read strands every pre-existing lifecycle-managed sandbox after a gateway upgrade.

Carried findings:

• None; all five prior findings are resolved or waived.

Gator metadata

• Validation: Maintainer-authored implementation of accepted RFC 0012 (#2048) for #1737; review remains scoped to PR #2942 while checking compatibility with VM #2945, Docker #2965, Kubernetes #3144, Podman #3230, and performance #3229.
• Docs: no new Fern documentation obligation in this authentication-only delta.
• Checks: Current-head E2E, GPU E2E, Helm Lint, Trivy Changes, and DCO are green; Branch Checks remain pending. Pipeline handoff remains suspended while the Critical is open.
• E2E: test:e2e is present and the current-head OpenShell / E2E gate is green; no label, /ok to test, or workflow rerun action is needed this cycle.
• Head SHA: 0db1fcf68fb61b80288ca75dc0fb7a8526b6bcc5
• Base SHA: 5b9daab9351b1e053f9a5e0ce4c899f5d3f674b0
• Merge base SHA: 5b57f0d1549677c7547862d26ff9491285ea0fbc
• Patch ID: 4ee0dd33436f1602aa950752e7e5158af2db202a
• Gator payload: 8
• Review mode: critical_only
• Previous reviewed SHA: 6143f24e2a8269403d1cbca108a722059b4e6b34
• Review budget exhausted: yes
• Maintainer decision required: no — this head introduces a Critical that requires an author fix rather than a Warning-bearing convergence decision.
• Reviewer-quality signals: 1 proposal, 1 validated Critical blocker, 0 duplicate-invariant proposals, 0 waived re-raises, 0 unchanged-code proposals, and 0 missing-reproducer downgrades. The ledger records 6 finding-bearing rounds, 5 prior unique findings, and 30 durable repeated-ID occurrences.
• Next state: gator:in-review

[johntmyers]

gator-agent

Maintainer Convergence Decision

Thanks @drew. I checked current head 25ec09fd against the durable review ledger and the consolidated VM, Docker, Kubernetes, Podman, and performance layers. The bounded critical-only review found no newly introduced Critical defect, but the prior upgrade-path obligation remains unresolved and unwaived.

Root-cause findings:

• GATOR-0db1fcf6-01: PersistedSessionLineage::read still requires every lineage annotation before mint_and_persist_successor can call the compute driver. A persisted sandbox created before lineage was introduced therefore still fails startup reconciliation or explicit start instead of receiving an atomically persisted initial identity. Partial or malformed lineage should remain fail-closed.

Scope growth:

• The five downstream RFC 0012 layers are now consolidated into this PR. I treated that maintainer-authored consolidation as the current architecture context; it does not waive the root-layer upgrade obligation.

Reviewer-quality signals:

• This round proposed 0 findings: 0 duplicate invariants, 0 waived re-raises, 0 unchanged-code proposals, and 0 missing-reproducer downgrades. The ledger records 7 finding-bearing heads and 6 unique findings.

Maintainer action: @drew or @NVIDIA/openshell-maintainers, either implement GATOR-0db1fcf6-01 with startup and explicit-start regression coverage, or explicitly waive/re-scope that finding for this independently mergeable root layer.

Gator metadata

• Validation: Maintainer-authored implementation of accepted RFC 0012 (rfc-0012: Isolation Backend interface #2048) for feat: establish the Isolation Backend interface #1737; review is scoped to PR feat(isolation): implement the RFC 0012 sandbox architecture #2942 while accounting for its consolidated downstream architecture.
• Docs: Prior documentation obligations remain resolved; no additional Fern-doc blocker was identified in this critical-only round.
• Checks: Current-head Branch Checks and OpenShell E2E are running; Helm Lint and GPU E2E pass; Trivy Changes is failing. Pipeline handoff remains suspended while the durable Critical is open.
• E2E: test:e2e is present and the current-head workflow is running; no label, /ok to test, or rerun action is needed this cycle.
• Head SHA: 25ec09fd0ed4b213f0cfefcd1d435ab571dee5f7
• Base SHA: cc780d4e17512e8f804f68c153c033fe6584ae3a
• Merge base SHA: cc780d4e17512e8f804f68c153c033fe6584ae3a
• Patch ID: 6202ba1ff81700955c3f1073e3175cfa3d5c68cc
• Gator payload: 8
• Review mode: critical_only
• Previous reviewed SHA: 0db1fcf68fb61b80288ca75dc0fb7a8526b6bcc5
• Review budget exhausted: yes
• Maintainer decision required: yes — GATOR-0db1fcf6-01 remains unresolved and unwaived.
• Next state: gator:blocked
• Blocked reason: review_convergence_decision_required

[johntmyers]

gator-agent

Maintainer Convergence Decision

I checked current head 5443e21c against the durable review ledger and the consolidated VM, Docker, Kubernetes, Podman, and performance layers. The bounded critical-only review found no newly introduced Critical defect in the Docker local-supervisor-image update, but the prior upgrade-path obligation remains unresolved and unwaived.

Root-cause findings:

• GATOR-0db1fcf6-01: PersistedSessionLineage::read still requires every lineage annotation before mint_and_persist_successor can call the compute driver. A persisted sandbox created before lineage was introduced therefore still fails startup reconciliation or explicit start instead of receiving an atomically persisted initial identity. Partial or malformed lineage should remain fail-closed.

Scope growth:

• None in this head. The one-file Docker development-helper delta builds and selects the matching local supervisor image; it does not change the outstanding gateway upgrade path or the consolidated stack contract.

Reviewer-quality signals:

• This round carried 1 existing finding and proposed 0 new findings: 0 duplicate invariants, 0 waived re-raises, 0 unchanged-code proposals, and 0 missing-reproducer downgrades. The ledger records 7 finding-bearing rounds and 6 unique findings.

Maintainer action: @drew or @NVIDIA/openshell-maintainers, either implement GATOR-0db1fcf6-01 with startup and explicit-start regression coverage, or explicitly waive/re-scope that finding for this independently mergeable root layer.

Gator metadata

• Validation: Maintainer-authored implementation of accepted RFC 0012 (rfc-0012: Isolation Backend interface #2048) for feat: establish the Isolation Backend interface #1737; review is scoped to PR feat(isolation): implement the RFC 0012 sandbox architecture #2942 while accounting for its consolidated downstream architecture.
• Docs: Prior documentation obligations remain resolved; the current development-helper delta adds its operator override inline and creates no additional Fern-doc blocker.
• Checks: Current-head Branch Checks and OpenShell E2E are running; Helm Lint and DCO pass; Trivy Changes is failing. Pipeline handoff remains suspended while the durable Critical is open.
• E2E: test:e2e is present and the current-head workflow is running; no label, /ok to test, or workflow rerun action is needed this cycle.
• Head SHA: 5443e21cab5f9df5bcbbd89b178c1f0e4ed6a24a
• Base SHA: cc780d4e17512e8f804f68c153c033fe6584ae3a
• Merge base SHA: cc780d4e17512e8f804f68c153c033fe6584ae3a
• Patch ID: 8c9d66bea8f2220bd43462d6f57e1c9f5cf4ada5
• Gator payload: 8
• Review mode: critical_only
• Previous reviewed SHA: 0db1fcf68fb61b80288ca75dc0fb7a8526b6bcc5
• Review budget exhausted: yes
• Maintainer decision required: yes — GATOR-0db1fcf6-01 remains unresolved and unwaived.
• Next state: gator:blocked
• Blocked reason: review_convergence_decision_required

[github-actions]

Label test:e2e-kubernetes applied for 5443e21. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA and credential-driver E2E after building the required gateway and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

[johntmyers]

gator-agent

Re-check After Maintainer-Author Update

Thanks @drew. I checked your September 14 reply on GATOR-0db1fcf6-01 and current head 6ba2b161. You explained that pre-RFC-0012 sandboxes lack both lineage and the new supervisor/runtime resources, so synthesizing lineage alone would be unsafe; as a verified maintainer, your explicit fail-closed breaking-upgrade decision waives that migration finding. I also checked the current architecture and Fern docs: they describe planned generation replacement, but they do not yet tell operators that adopting RFC 0012 requires stopping and recreating existing sandboxes.

The bounded critical-only review found no newly introduced Critical defect in the consolidated VM, Docker, Kubernetes, Podman, and performance delta.

Disposition: the prior code finding is waived; the promised operator-facing breaking-upgrade documentation remains outstanding.

Remaining items:

• Add the breaking-upgrade procedure to the relevant Fern operator documentation, including that existing sandboxes must be stopped and recreated and that missing or partial lineage remains fail-closed.

Action required: @drew, please add that operator documentation and push an updated head.

Gator metadata

• Validation: Maintainer-authored implementation of accepted RFC 0012 (rfc-0012: Isolation Backend interface #2048) for feat: establish the Isolation Backend interface #1737; review is scoped to PR feat(isolation): implement the RFC 0012 sandbox architecture #2942 while accounting for its consolidated VM, Docker, Kubernetes, Podman, and performance architecture.
• Docs: missing the maintainer-promised breaking-upgrade procedure for pre-RFC-0012 sandboxes.
• Checks: Current-head Branch Checks and OpenShell E2E are running; Helm Lint, DCO, and GPU E2E pass; Trivy Changes is failing.
• E2E: test:e2e and test:e2e-kubernetes are present. The current-head E2E run is active, but its optional Kubernetes HA and credential-driver jobs were skipped; the obsolete-head rerun was not dispatched, and the authorized current-head rerun must wait until run 34897577924 is rerunnable.
• Head SHA: 6ba2b16120122dc24678743d36ceba2ae17e6bef
• Base SHA: cc780d4e17512e8f804f68c153c033fe6584ae3a
• Merge base SHA: cc780d4e17512e8f804f68c153c033fe6584ae3a
• Patch ID: cfc4d1ae47d40cda69dd573d50f99093a340e947
• Gator payload: 8
• Review mode: critical_only
• Previous reviewed SHA: 0db1fcf68fb61b80288ca75dc0fb7a8526b6bcc5
• Review budget exhausted: yes
• Maintainer decision required: no — the verified maintainer waiver resolves the concrete migration decision; only the stated documentation follow-up remains.
• Reviewer-quality signals: 0 proposals, 0 duplicate-invariant proposals, 0 waived re-raises, 0 unchanged-code proposals, and 0 missing-reproducer downgrades. The ledger records 7 finding-bearing rounds, 6 unique findings, and 34 repeated finding-ID occurrences as durable carries.
• Next state: gator:in-review

[johntmyers]

gator-agent

Re-check After Author Update

Thanks @drew. I checked current head 0d904545 after your Kubernetes RBAC update and the merge of current main. The author-owned change narrows sandbox Service permissions and adds matching Helm assertions; the upstream-only merge changes were excluded from the delta review. The bounded critical-only review found no newly introduced Critical defect, and the consolidated RFC 0012 architecture remains coherent with the VM, Docker, Kubernetes, Podman, and performance layers.

The previously promised breaking-upgrade documentation is still absent from the current Fern/operator docs.

Action required: @drew, please document that adopting RFC 0012 requires operators to stop and recreate existing sandboxes, and that missing or partial lineage remains fail-closed, then push an updated head.

Blocking findings:

• No new findings.

Carried findings:

• None; the prior migration finding is waived. The explicit operator-documentation follow-up remains outstanding.

Gator metadata

• Validation: Maintainer-authored implementation of accepted RFC 0012 (rfc-0012: Isolation Backend interface #2048) for feat: establish the Isolation Backend interface #1737; review is scoped to PR feat(isolation): implement the RFC 0012 sandbox architecture #2942 while accounting for the full six-layer isolation architecture.
• Docs: missing the maintainer-promised breaking-upgrade procedure for pre-RFC-0012 sandboxes.
• Checks: Current-head Branch Checks and OpenShell E2E are pending; Helm Lint, Trivy Changes, DCO, and GPU E2E are green.
• E2E: test:e2e and test:e2e-kubernetes are present. Current-head run 34904575723 is active, so the obsolete-head Label Help rerun instruction does not require a rerun or /ok to test this cycle.
• Head SHA: 0d90454593df9e086e737f16457a7477dce4daac
• Base SHA: 2d5db4c5bd6506eb68668175c9e2707e0403a4e2
• Merge base SHA: 2d5db4c5bd6506eb68668175c9e2707e0403a4e2
• Patch ID: ce527d3d9ba005b0f12e7515f829f01f931630b1
• Gator payload: 8
• Review mode: critical_only
• Previous reviewed SHA: 6ba2b16120122dc24678743d36ceba2ae17e6bef
• Review budget exhausted: yes
• Maintainer decision required: no — the migration behavior was explicitly waived; the promised documentation remains a concrete author obligation.
• Reviewer-quality signals: 0 proposals, 0 duplicate-invariant proposals, 0 waived re-raises, 0 unchanged-code proposals, and 0 missing-reproducer downgrades. The ledger records 8 review rounds, 8 finding-bearing rounds, 6 unique IDs, and 35 repeated ID occurrences as durable carries.
• Next state: gator:in-review