Skip to content

fix: repair guest boot chain and harden host-side sandbox #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dev-dami merged 2 commits into from
Aug 3, 2026
Merged

fix: repair guest boot chain and harden host-side sandbox #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b7f9a9
fix: repair guest boot chain and harden host-side sandbox
dev-dami Aug 3, 2026
e810fde
ci: install musl target and assert the guest agent stays static
dev-dami Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ jobs:
with:
toolchain: stable
components: rustfmt, clippy
target: x86_64-unknown-linux-musl

- name: Verify Formatting
run: cargo fmt --all -- --check
Expand All @@ -42,6 +43,21 @@ jobs:
- name: Run Tests
run: cargo test --all

# The guest agent runs as PID 1 on a rootfs with no libc and no dynamic
# loader. If it ever links dynamically the guest cannot boot at all, and
# no host-side test catches that — so assert it here.
- name: Verify guest agent links statically
run: |
cargo build --release --bin ignite-guest-agent \
--target x86_64-unknown-linux-musl
AGENT=target/x86_64-unknown-linux-musl/release/ignite-guest-agent
file "$AGENT"
if readelf -l "$AGENT" | grep -qi 'interpreter'; then
echo "::error::Guest agent is dynamically linked. The guest rootfs has no dynamic loader, so it would fail to exec as init."
exit 1
fi
echo "Guest agent is statically linked."
Comment on lines +49 to +59

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Make the static-link check fail when readelf fails.

The step pipes readelf -l into grep. The exit status of a pipeline is the status of the last command, so a readelf failure (missing tool, unreadable file) is masked and the check reports success. Enable pipefail and inspect captured output instead.

Also consider matching the segment tag INTERP rather than the descriptive text, because the wording of readelf output can change between binutils versions.

🛡️ Proposed hardening 
       - name: Verify guest agent links statically
         run: |
+          set -euo pipefail
           cargo build --release --bin ignite-guest-agent \
             --target x86_64-unknown-linux-musl
           AGENT=target/x86_64-unknown-linux-musl/release/ignite-guest-agent
           file "$AGENT"
-          if readelf -l "$AGENT" | grep -qi 'interpreter'; then
+          HEADERS=$(readelf -l "$AGENT")
+          if grep -qE 'INTERP|interpreter' <<<"$HEADERS"; then
             echo "::error::Guest agent is dynamically linked. The guest rootfs has no dynamic loader, so it would fail to exec as init."
             exit 1
           fi
           echo "Guest agent is statically linked."
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Verify guest agent links statically
run: |
cargo build --release --bin ignite-guest-agent \
--target x86_64-unknown-linux-musl
AGENT=target/x86_64-unknown-linux-musl/release/ignite-guest-agent
file "$AGENT"
if readelf -l "$AGENT" | grep -qi 'interpreter'; then
echo "::error::Guest agent is dynamically linked. The guest rootfs has no dynamic loader, so it would fail to exec as init."
exit 1
fi
echo "Guest agent is statically linked."
- name: Verify guest agent links statically
run: |
set -euo pipefail
cargo build --release --bin ignite-guest-agent \
--target x86_64-unknown-linux-musl
AGENT=target/x86_64-unknown-linux-musl/release/ignite-guest-agent
file "$AGENT"
HEADERS=$(readelf -l "$AGENT")
if grep -qE 'INTERP|interpreter' <<<"$HEADERS"; then
echo "::error::Guest agent is dynamically linked. The guest rootfs has no dynamic loader, so it would fail to exec as init."
exit 1
fi
echo "Guest agent is statically linked."
🧰 Tools
🪛 zizmor (1.28.0)

[warning] 15-59: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block

(excessive-permissions)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 49 - 59, Update the “Verify guest
agent links statically” workflow step to enable pipeline failure propagation and
capture the `readelf -l "$AGENT"` output before checking it. Make the step fail
when `readelf` cannot inspect the binary, and detect dynamic linking by matching
the stable `INTERP` segment tag rather than descriptive wording.


release:
name: Build & Release Binaries
runs-on: ubuntu-latest
Expand Down
92 changes: 65 additions & 27 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,58 +2,96 @@

This guide tells coding agents how to work safely and effectively in this repository.

## Core Rule
## What This Is

- Use Bun for all package installs and scripts.
- Never use npm.
Ignite is a **Rust cargo workspace** (edition 2024) that executes untrusted
JS/TS inside hardware-virtualized microVMs. There is no Node/Bun build system
here — Bun, Node, Deno, and QuickJS are *guest runtimes* that Ignite downloads
and attaches to VMs as read-only block devices, not tooling for this repo.

## Repository Map

- `packages/core`: sandbox lifecycle, loaders, preflight, execution engine.
- `packages/cli`: `ignite` CLI commands.
- `packages/http`: HTTP server surface for sandbox execution.
- `packages/shared`: shared types/utilities used across packages.
- `packages/runtime-bun`: Bun runtime image files.
- `ignite-shared/`: shared types (`ServiceConfig`, metrics), error enum, validators.
- `ignite-core/`: sandbox lifecycle — preflight, ext4 disk building, host setup,
and the hypervisor backends under `platform/`.
- `ignite-cli/`: the `ignite` command line binary (clap).
- `ignite-http/`: axum REST API server.
- `ignite-guest-agent/`: static PID-1 init that runs *inside* the guest.
- `examples/*`: sample services used for smoke/manual verification.
- `docs/*`: user-facing docs and architecture notes.
- `scripts/*`: release/build helper scripts.

## Local Workflow

1. Install deps: `bun install`
2. Build all packages: `bun run build`
3. Run lint: `bun run lint`
4. Run typecheck: `bun run typecheck`
5. Run tests:
- Unit-only (fast): `bun run test:unit`
- Full suite (requires Docker): `bun run test`
```bash
cargo build --workspace
cargo fmt --all
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
```

CI runs `fmt --check`, `clippy -D warnings`, and `cargo test --all`. All three
must pass.

### Running a service end to end

Requires a Linux host with KVM, `firecracker` on PATH, `e2fsprogs`, and an
**uncompressed ELF `vmlinux`** (not `/boot/vmlinuz-*`, which is a compressed
bzImage Firecracker cannot boot).

```bash
rustup target add x86_64-unknown-linux-musl # guest agent must be static
ignite setup
ignite status # verifies KVM + firecracker
ignite run examples/hello-bun
```

## Change Rules

- Keep changes scoped to the task; avoid drive-by refactors.
- When behavior changes, add or update tests in the relevant package.
- If CLI/API behavior changes, update docs in `README.md` and/or `docs/*`.
- Use existing naming patterns and file structure within each package.
- When behavior changes, add or update tests in the relevant crate.
- If CLI/API behavior changes, update `README.md` and/or `docs/*`.
- Prefer small, composable functions over large rewrites.

## Security Guardrails (Important)

Ignite runs untrusted code. Treat security defaults as product-critical.

- Do not weaken sandbox restrictions (network/filesystem/capabilities) without explicit task requirements.
- If security logic changes, include tests that prove both allowed and blocked behavior.
- Do not weaken sandbox restrictions (network/filesystem/capabilities) without
explicit task requirements.
- If security logic changes, include tests proving both allowed and blocked
behavior.
- Never introduce secrets, tokens, or host-specific paths into committed code.

Three invariants are easy to break by accident:

1. **The guest is untrusted, including the VSOCK peer.** Sandboxed code can open
`AF_VSOCK` to the host itself. Never trust a guest-supplied length prefix
without bounding it, and never assume the guest agent is the only writer.
2. **`service.yaml` is untrusted input.** `service.name` reaches host paths that
get unlinked. Validate it (`validate_service_name`) and sanitize before
interpolating (`sanitize_path_segment`).
3. **The guest rootfs is read-only and has no libc.** The agent cannot `mkdir`
its own mount points — they must exist in the image — and must stay
statically linked against musl.

Do not advertise a security control that is not enforced. If a flag cannot be
honored, fail loudly rather than accepting it silently: `--audit` currently
returns an error for exactly this reason.

## Validation Matrix

- `packages/shared` change: run `bun run build`, `bun run test:unit`.
- `packages/core` change: run `bun run test:unit`; run `bun run test` if execution behavior changed.
- `packages/cli` change: run `bun run test:unit`; manually smoke command paths when possible.
- `packages/http` change: run `bun run test:unit`; verify request/response behavior for changed endpoints.
- `docs`-only change: lint/typecheck optional, tests not required.
- `ignite-shared` change: `cargo test --workspace` (everything depends on it).
- `ignite-core` change: `cargo test -p ignite-core`; manually smoke `ignite run`
if execution behavior changed.
- `ignite-cli` change: `cargo test -p ignite-cli`; smoke the affected command.
- `ignite-http` change: `cargo test -p ignite-http`; verify request/response
behavior for changed endpoints.
- `ignite-guest-agent` change: rebuild for musl and boot a VM — this code runs
as PID 1 and is not covered by host-side tests.
- `docs`-only change: tests not required.

## Commit & PR Hygiene

- Follow Conventional Commits (`feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`).
- Keep PRs focused and include a short verification summary (commands run + results).
- Mention Docker dependency when full test suite could not be run.
- Note when VM-level verification could not be run (no KVM, no kernel image).
40 changes: 29 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@

## Overview

Ignite runs JavaScript/TypeScript code inside isolated, hardware-virtualized microVMs rather than containers. It supports native **Firecracker** on Linux and Apple's **Virtualization.framework** on macOS out of the box, with zero external VM dependencies.
Ignite runs JavaScript/TypeScript code inside isolated, hardware-virtualized microVMs rather than containers. Execution currently requires a **Linux host with KVM**, using **Firecracker** as the hypervisor.

> **Status:** pre-1.0. A macOS backend built on Apple's `Virtualization.framework` is planned but **not implemented** — the platform selector is stubbed and returns a clear error. Bun is the only runtime `ignite setup` provisions today.

It is designed for systems that execute code you do not fully trust:

Expand All @@ -28,18 +30,26 @@ It is designed for systems that execute code you do not fully trust:

## Key Features

- **Dual-Hypervisor Core**: Uses KVM-backed Firecracker on Linux, and native `Virtualization.framework` on macOS.
- **KVM-backed Firecracker**: Each service runs in its own microVM with a separate guest kernel. (A macOS `Virtualization.framework` backend is planned; see Status above.)
- **Host-Reliant Disk Mounts**: The guest microVM has no shell, utilities, or libraries. Service code and language runtimes (Bun, Node, Deno, QuickJS) are compiled on the host and attached as read-only virtual block devices (`/dev/vdb` and `/dev/vdc`).

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Describe runtime provisioning accurately.

The supplied setup flow builds the guest agent, while runtime binaries come from the runtime directory. The phrase “language runtimes ... are compiled on the host” implies a build step that this workflow does not perform. Use “installed or downloaded on the host, then packaged into read-only block devices.”

Proposed wording 
- Service code and language runtimes (Bun, Node, Deno, QuickJS) are compiled on the host and attached as read-only virtual block devices (`/dev/vdb` and `/dev/vdc`).
+ Service code and installed runtime binaries (Bun, Node, Deno, QuickJS) are packaged on the host and attached as read-only virtual block devices (`/dev/vdb` and `/dev/vdc`).

Based on the supplied ignite-core provisioning flow, only the guest agent is built; runtime binaries are read from runtime_src_path.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- **Host-Reliant Disk Mounts**: The guest microVM has no shell, utilities, or libraries. Service code and language runtimes (Bun, Node, Deno, QuickJS) are compiled on the host and attached as read-only virtual block devices (`/dev/vdb` and `/dev/vdc`).
- **Host-Reliant Disk Mounts**: The guest microVM has no shell, utilities, or libraries. Service code and installed runtime binaries (Bun, Node, Deno, QuickJS) are packaged on the host and attached as read-only virtual block devices (`/dev/vdb` and `/dev/vdc`).
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` at line 34, Update the “Host-Reliant Disk Mounts” description in
README.md to state that runtime binaries are installed or downloaded on the host
and then packaged into read-only virtual block devices, while retaining that the
guest agent is built by the setup flow. Remove the claim that Bun, Node, Deno,
and QuickJS are compiled on the host.

- **VSOCK Multiplexing**: Low-latency communication handshakes stream stdout/stderr and exit codes directly back to the host via virtual sockets, bypassing network interfaces.
- **Resource Enforcement**: `memoryMb` and `cpuLimit` are applied to Firecracker machine config, while `timeoutMs` is enforced by a host-side watchdog that force-terminates timed-out VMs.
- **Resource Enforcement**: `memoryMb` and `cpuLimit` are applied to Firecracker machine config, while `timeoutMs` is enforced by a host-side watchdog that force-terminates timed-out VMs. `cpuLimit` is rounded up to whole vCPUs, which is the only granularity Firecracker accepts.
- **Preflight & Metric Timelines**: Sub-millisecond logging of all VM lifecycle transitions (disk format, boot connect, execution, cleanup).

## Quick Start

### 1) Prerequisites

- **Linux**: KVM enabled (`/dev/kvm` accessible) and `e2fsprogs` installed.
- **macOS**: macOS 13 or later.
- Linux with KVM enabled (`/dev/kvm` accessible).
- `firecracker` on your `PATH`.
- `e2fsprogs` (provides `mke2fs`).
- The musl target for the static guest agent: `rustup target add x86_64-unknown-linux-musl`.
- An **uncompressed ELF `vmlinux`**. Distro `/boot/vmlinuz-*` files are
compressed bzImages that Firecracker cannot boot; extract one with the kernel
tree's `scripts/extract-vmlinux`, or use a prebuilt Firecracker kernel, then
point `IGNITE_KERNEL_PATH` (or `--kernel`) at it.

Run `ignite status` to check KVM access and Firecracker availability.

### 2) Build from Source

Expand Down Expand Up @@ -83,12 +93,20 @@ ignite run . --verbose

## Runtime Support

| Runtime | Supported versions | Default |
|---|---|---|
| Bun | `1.0`, `1.1`, `1.2`, `1.3` | `1.3` |
| Node | `18`, `20`, `22` | `20` |
| Deno | `1.40`, `1.41`, `1.42`, `2.0` | `2.0` |
| QuickJS | `2024-01-13`, `2023-12-09`, `latest` | `latest` |
| Runtime | Accepted versions | Default | Provisioned by `ignite setup` |
|---|---|---|---|
| Bun | `1.0`, `1.1`, `1.2`, `1.3` | `1.3` | Yes |
| Node | `18`, `20`, `22` | `20` | No — install manually |
| Deno | `1.40`, `1.41`, `1.42`, `2.0` | `2.0` | No — install manually |
| QuickJS | `2024-01-13`, `2023-12-09`, `latest` | `latest` | No — install manually |

Runtimes are read from `~/.ignite/runtimes/` (override with `IGNITE_RUNTIMES_ROOT`).
A pinned spec such as `bun@1.3` resolves to `runtimes/bun@1.3/` and falls back to
`runtimes/bun/` with a warning. Runtime binaries must be statically linked or
otherwise self-contained: the guest rootfs has no dynamic loader.

Only runtimes you install are available; `ignite setup` currently downloads Bun
only.

## Documentation

Expand Down
34 changes: 31 additions & 3 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ Options:
- `--runtimes-root <path>`: path to custom host language runtimes folder.
- `--vsock-port <port>`: custom host-guest VSOCK communication port.
- `--console-out <path>`: file path to log guest serial console outputs.
- `--audit` / `--audit-output <path>`: **not implemented**; returns an error.
See [Threat Model](./threat-model.md) for the isolation that *is* enforced.

### `ignite preflight <service>`

Expand All @@ -63,9 +65,25 @@ ignite serve [options]
Options:

- `-p, --port <port>`: API port (default `3000`)
- `-h, --host <host>`: host IP to bind (default `localhost`)
- `-h, --host <host>`: host or IP to bind (default `localhost`)
- `-s, --services <path>`: path to services root folder (default `./services`)

Environment:

- `IGNITE_API_KEY`: when set, every endpoint except `/health` requires
`Authorization: Bearer <key>`. **When unset, the server executes services for
any caller that can reach it** and logs a warning at startup — keep the bind
address on localhost in that case.
- `IGNITE_CORS_ORIGINS`: comma-separated list of exact allowed origins
(e.g. `https://app.example.com`). When unset, no CORS headers are sent at all,
which blocks browser-based cross-origin access by default.
- `IGNITE_KERNEL_PATH`, `IGNITE_ROOTFS_PATH`, `IGNITE_RUNTIMES_ROOT`: override
the guest kernel, rootfs image, and runtimes directory.

Requests are rate limited to 60 per minute per client IP, keyed on the real
transport peer address (not the spoofable `X-Forwarded-For` header). If you run
Ignite behind a reverse proxy, apply per-client limits at the proxy.

---

## HTTP REST API
Expand All @@ -77,11 +95,12 @@ Response:
```json
{
"status": "ok",
"version": "0.1.0",
"uptime": 0
"version": "0.9.0"
}
```

`/health` is the only endpoint that does not require authentication.

### `GET /services`

List service folders under the configured services path root.
Expand All @@ -108,6 +127,15 @@ Request body:
}
```

`audit` must be `false` or omitted. Security audit mode is **not implemented**;
sending `true` returns an error rather than silently reporting an audit that
never ran.

`coldStartTimeMs` is present only when the guest runtime emits an
`IGNITE_INIT_TIME:` line on stderr, and `memoryUsageMb` likewise depends on an
`IGNITE_MEMORY_MB:` line. Both are omitted or zero otherwise rather than
estimated.

Response:

```json
Expand Down
12 changes: 7 additions & 5 deletions docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ To minimize vulnerabilities in the guest VM, the root filesystem is built withou
└──────────────────────┘
```

1. **`/dev/vda` (Root filesystem)**: Contains only the statically compiled `/sbin/init` (the `ignite-guest-agent` binary).
1. **`/dev/vda` (Root filesystem)**: Contains only the statically compiled `/sbin/init` (the `ignite-guest-agent` binary, linked against musl) plus empty `/app`, `/runtime`, `/dev`, `/proc`, and `/sys` mount points. Because this disk is attached read-only, those directories must exist in the image — the agent cannot create them at runtime. The kernel is booted with `init=/sbin/init` to match.
2. **`/dev/vdb` (Service filesystem)**: Holds service source code files, formatted dynamically on-the-fly by the host using `mke2fs` without loopback privileges. Mounted read-only at `/app`.
3. **`/dev/vdc` (Runtime engine)**: Holds the selected language runtime (Node, Deno, Bun, or QuickJS) from the host's runtime library folder. Mounted read-only at `/runtime`.

Expand All @@ -41,10 +41,12 @@ ignite-cli/ignite-http
-> Loads service.yaml
-> Runs preflight checks (dependency counts, RAM allocations)
-> Calls mke2fs to format service.ext4 and runtime.ext4 in user-space
-> Binds host VSOCK listener on Unix Socket (/tmp/ignite-vsock-VMID_1052)
-> Spawns hypervisor process (Firecracker or Apple VZ)
-> Hypervisor loads kernel (vmlinux), boots guest VM
-> Guest Agent (init) mounts /dev/vdb to /app and /dev/vdc to /runtime
-> Binds host VSOCK listener on Unix Socket (/tmp/ignite-vsock-<vm>_<port>)
BEFORE starting the VM, so the guest cannot dial out before we listen
-> Spawns hypervisor process (Firecracker)
-> Hypervisor loads kernel (uncompressed ELF vmlinux), boots guest VM
-> Guest Agent (init) mounts devtmpfs, proc, sysfs, then /dev/vdb to /app
and /dev/vdc to /runtime (read-only); a failed disk mount is fatal
-> Guest Agent connects back to host over VSOCK channel (port 1052)
-> Host transmits JSON payload (environment, execution script)
-> Guest Agent executes runtime command inside microVM sandbox
Expand Down
Loading
Loading