Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 32 additions & 6 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,18 @@ jobs:
run: |
version="${GITHUB_REF_NAME#v}"

# The tag is the single source of truth: stamp it into both manifests.
# Cargo.toml's [package] version is the only line starting with
# `version = `; mix.exs has exactly one `version: "..."` (the project
# version). `0,/re/` rewrites just the first match.
sed -i -E "0,/^version = \"[^\"]*\"/s//version = \"${version}\"/" native/suidhelper/Cargo.toml
# The tag is the single source of truth: stamp it into every package
# manifest. `0,/re/` rewrites only the first match, which for each
# Cargo.toml is its own `[package] version` (xtask is intentionally
# left at 0.0.0 — it is publish = false).
for m in native/suidhelper/Cargo.toml native/suidhelper/meta/Cargo.toml native/guest-agent/Cargo.toml; do
sed -i -E "0,/^version = \"[^\"]*\"/s//version = \"${version}\"/" "$m"
done
sed -i -E "0,/version: \"[^\"]*\"/s//version: \"${version}\"/" mix.exs

git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add native/suidhelper/Cargo.toml mix.exs
git add native/suidhelper/Cargo.toml native/suidhelper/meta/Cargo.toml native/guest-agent/Cargo.toml mix.exs

# Idempotent: a re-run after the bump already landed has nothing to commit.
if git diff --cached --quiet; then
Expand Down Expand Up @@ -133,6 +135,18 @@ jobs:
- name: Install dependencies
run: mix deps.get

# The gitignored gRPC/guest-agent/suidhelper artifacts are regenerated by
# Mix compilers on every compile; hex.build compiles, so the runner needs
# their toolchain. Mirrors ci.yml.
- name: Install protoc
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler

- name: Install protoc-gen-elixir
run: mix escript.install hex protobuf 0.17.0 --force

- name: Install musl target for the guest-agent build
run: rustup target add x86_64-unknown-linux-musl

- name: Build package tarball
run: mix hex.build --output "hypervm-${{ needs.prepare-release.outputs.version }}.tar"

Expand Down Expand Up @@ -191,6 +205,18 @@ jobs:
- name: Install dependencies
run: mix deps.get

# The gitignored gRPC/guest-agent/suidhelper artifacts are regenerated by
# Mix compilers on every compile; hex.build compiles, so the runner needs
# their toolchain. Mirrors ci.yml.
- name: Install protoc
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler

- name: Install protoc-gen-elixir
run: mix escript.install hex protobuf 0.17.0 --force

- name: Install musl target for the guest-agent build
run: rustup target add x86_64-unknown-linux-musl

# HEX_API_KEY authenticates non-interactively; --yes skips the prompt.
# Publishes both the package and its docs.
- name: hex publish
Expand Down
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Changelog

All notable changes to this project are documented here. The format is based on
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.1.0]

First public release: a distributed orchestrator for Firecracker microVMs on the
BEAM.

### Added
- Firecracker microVM lifecycle: create, stop, locate, and list VMs across a
cluster, with automatic layer-affinity placement.
- Content-addressed OCI image loading into a shared media store + image database.
- Copy-on-write disk **forking** (`Hyper.Vm.fork/1` and the `ForkVm` gRPC RPC):
colocated thin-snapshot forks with a cluster-wide delta-layer fallback.
- Per-VM egress networking: netns + TAP + nftables NAT.
- Granular per-VM compute metering (CPU-time, from the cgroup) for billing.
- Guest exec over vsock (`Hyper.exec/3`).
- gRPC interface (`hyper.grpc.v0`, UNSTABLE) — off by default, unauthenticated.
- Privileged Rust setuid helper for the host operations the BEAM cannot do safely.
- OpenTelemetry tracing across the subsystems.

[Unreleased]: https://github.com/harmont-dev/hyper/compare/v0.1.0...HEAD
[0.1.0]: https://github.com/harmont-dev/hyper/releases/tag/v0.1.0
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Then load an OCI image and boot it:

```elixir
{:ok, img_id} = Hyper.Img.OciLoader.load("docker.io/library/alpine:3.19")
{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id})
{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id, type: :micro})
{:ok, %{stdout: "hello\n"}} = Hyper.exec(vm, ["/bin/echo", "hello"])
```

Expand Down Expand Up @@ -84,6 +84,7 @@ integrating Hyper.
BEAM-native interfaces, we recognize this may not be ideal for all languages
and existing stacks. For that reason, Hyper has a GRPC interface, so you can
call it from any language you already use.
⚠️ The gRPC API is unauthenticated and off by default — bind it to a trusted network or front it with an authenticating proxy. See [the gRPC guide](docs/grpc.md).

## Docs

Expand Down
26 changes: 26 additions & 0 deletions docs/cookbook/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,32 @@ uid_gid_range = [900000, 999999]
```
<!-- tabs close -->

## VM Networking Configuration

**Required.** Every VM gets NAT'd egress out through a physical host interface,
and a node refuses to start without a `[network]` table (see the
[install guide](install.md#vm-networking-required) for the host prerequisites:
`iproute2`/`nftables` and `net.ipv4.ip_forward=1`). `config.toml`-only — the
setuid helper reads the same `uplink` and `clone_pool` to build each VM's netns,
veth pair, and NAT rules, so it cannot live in `config.exs`.

### `config.toml`

```toml
[network]
# **required**. The physical uplink interface guest egress is NAT'd out through.
# On a single-uplink host, the default-route NIC is the usual choice:
# ip route show default | awk '{print $5; exit}'
uplink = "eth0"

# optional -- defaults shown. The IPv4 CIDR the per-VM /30s are carved from.
clone_pool = "172.31.0.0/16"

# optional -- defaults shown. DNS resolver handed to guests via the kernel
# cmdline (the guest agent writes it to /etc/resolv.conf).
resolver = "1.1.1.1"
```

## gRPC Configuration

Hyper supports a [gRPC](https://grpc.io/) interface enabling you to interface
Expand Down
2 changes: 1 addition & 1 deletion docs/cookbook/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ You can install the required packages by running:
sudo dnf install -y \
coreutils \
device-mapper-persistent-data \
e2fsprogs \
e2fsprogs \
glibc-common \
kernel-modules-extra-$(uname -r) \
lvm2 \
Expand Down
2 changes: 1 addition & 1 deletion docs/cookbook/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ interface, or through [gRPC](../grpc.md):
With the image loaded, and an `img_id` in hand, you can boot it:

```elixir
{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id})
{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id, type: :micro})
```

The VM is scheduled onto the most available node in the cluster, preferring
Expand Down
9 changes: 9 additions & 0 deletions docs/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@ create, stop, locate, and list microVMs.
> **v0 -- unstable.** The contract may change without notice during early
> development. Pin to a commit if you depend on it.

> #### No authentication {: .warning}
>
> The gRPC API has **no authentication**. Any client that can reach the port can
> create and stop VMs and load images. It is **off by default** (`grpc.enabled =
> false`). When you enable it, bind it to loopback or a trusted network, or put
> it behind a proxy that terminates TLS and authenticates callers. TLS (`cred`)
> encrypts the channel but does not authenticate the client. Authentication is
> planned for a later release; the `v0` contract is UNSTABLE.

## Configuration

By default, the gRPC interface is **disabled**. The simplest way to enable it
Expand Down
13 changes: 13 additions & 0 deletions lib/hyper.ex
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,19 @@ defmodule Hyper do
:error, {:erpc, _} -> {:error, :machine_unreachable}
end

@doc """
Fork the running VM `vm_id`: boot a child from a copy-on-write clone of its
disk, colocated when the parent's node has room and re-placed cluster-wide
otherwise. Returns the child VM. See `Hyper.Vm.fork/1` for the semantics.
"""
@spec fork_vm(Hyper.Vm.Id.t()) :: {:ok, Hyper.Vm.t()} | {:error, term()}
def fork_vm(vm_id) when is_binary(vm_id) do
case Hyper.Cluster.Routing.supervisor_of(vm_id) do
nil -> {:error, :not_found}
pid -> Hyper.Vm.fork(pid)
end
end

@doc """
The vm id for a VM handle -- the pid returned by `create_vm/1`. Resolves on the
pid's owning node, so a VM just placed on a remote node is found immediately
Expand Down
8 changes: 8 additions & 0 deletions lib/hyper/grpc/codec.ex
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ defmodule Hyper.Grpc.Codec do
alias Hyper.Grpc.V0.{
CreateVmRequest,
CreateVmResponse,
ForkVmResponse,
GetVmResponse,
GetVmUsageResponse,
ListVmsResponse,
Expand Down Expand Up @@ -75,6 +76,10 @@ defmodule Hyper.Grpc.Codec do
def to_grpc({:created, vm_id, node}) when is_binary(vm_id),
do: %CreateVmResponse{vm_id: vm_id, node: to_string(node)}

@spec to_grpc({:forked, Hyper.Vm.Id.t(), node()}) :: ForkVmResponse.t()
def to_grpc({:forked, vm_id, node}) when is_binary(vm_id),
do: %ForkVmResponse{vm_id: vm_id, node: to_string(node)}

@spec to_grpc({:located, Hyper.Vm.Id.t(), node()}) :: GetVmResponse.t()
def to_grpc({:located, vm_id, node}),
do: %GetVmResponse{vm_id: vm_id, node: to_string(node)}
Expand Down Expand Up @@ -126,6 +131,9 @@ defmodule Hyper.Grpc.Codec do
defp rpc_error(:machine_unreachable),
do: GRPC.RPCError.exception(:unavailable, "VM's host node is unreachable")

defp rpc_error(:node_unreachable),
do: GRPC.RPCError.exception(:unavailable, "VM's host node is unreachable")

defp rpc_error(reason) when reason in [:no_capacity, :exhausted],
do: GRPC.RPCError.exception(:resource_exhausted, "no capacity")

Expand Down
16 changes: 16 additions & 0 deletions lib/hyper/grpc/server.ex
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ defmodule Hyper.Grpc.Server do
alias Hyper.Grpc.V0.{
CreateVmRequest,
CreateVmResponse,
ForkVmRequest,
ForkVmResponse,
GetVmRequest,
GetVmResponse,
GetVmUsageRequest,
Expand Down Expand Up @@ -50,6 +52,20 @@ defmodule Hyper.Grpc.Server do
end
end

@spec fork_vm(ForkVmRequest.t(), GRPC.Server.Stream.t()) :: ForkVmResponse.t()
@decorate with_span("Hyper.Grpc.Server.fork_vm", include: [:vm_id])
def fork_vm(%ForkVmRequest{vm_id: vm_id}, _stream) do
with {:ok, child} <- Hyper.fork_vm(vm_id),
child_id when is_binary(child_id) <- Hyper.id(child) do
Codec.to_grpc({:forked, child_id, node(child)})
else
# The child was placed but its host became unreachable before its id
# could be confirmed — same shape as create_vm/2.
nil -> raise Codec.to_grpc({:error, :machine_unreachable})
{:error, reason} -> raise Codec.to_grpc({:error, reason})
end
end

@spec stop_vm(StopVmRequest.t(), GRPC.Server.Stream.t()) :: Empty.t()
@decorate with_span("Hyper.Grpc.Server.stop_vm", include: [:vm_id])
def stop_vm(%StopVmRequest{vm_id: vm_id}, _stream) do
Expand Down
1 change: 1 addition & 0 deletions native/guest-agent/.gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
/target
*.profraw
3 changes: 3 additions & 0 deletions native/guest-agent/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
name = "hyper-guest-agent"
version = "0.1.0"
edition = "2021"
license = "MIT"
description = "PID-1 guest agent for Hyper microVMs: mounts, resolv.conf, and a vsock exec server."
repository = "https://github.com/harmont-dev/hyper"

[lib]
name = "hyper_guest_agent"
Expand Down
3 changes: 3 additions & 0 deletions native/suidhelper/.gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
/target
/lcov.info
# LLVM source-based coverage drops these at the crate root during instrumented
# test runs (see the xtask coverage pipeline); they are never committed.
*.profraw
27 changes: 27 additions & 0 deletions proto/hyper/grpc/v0/hyper.proto
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,18 @@ service Hyper {
// before its id could be confirmed.
rpc CreateVm(CreateVmRequest) returns (CreateVmResponse);

// Fork a running microVM: snapshot its disk and boot a colocated child from
// the copy-on-write clone, falling back to a cluster-wide placement (via a
// published delta layer) when the parent's node is full. The parent keeps
// running. The fork is crash-consistent (like a power cut) after a best-effort
// guest sync. Returns the new child's `vm_id` and the node it booted on.
//
// Errors:
// NOT_FOUND -- no VM with this `vm_id` is running in the cluster.
// RESOURCE_EXHAUSTED -- no node in the cluster can admit the child.
// UNAVAILABLE -- the parent's host node became unreachable.
rpc ForkVm(ForkVmRequest) returns (ForkVmResponse);

// Stop and tear down a running microVM. Idempotent in effect: tearing down a
// VM that is already stopping is not an error.
//
Expand Down Expand Up @@ -130,6 +142,21 @@ message CreateVmResponse {
string node = 2;
}

// Request to fork a running VM.
message ForkVmRequest {
// The id of the parent VM to fork.
string vm_id = 1;
}

// Result of a successful ForkVm.
message ForkVmResponse {
// The server-minted id for the new child VM; use it in all later RPCs.
string vm_id = 1;

// The cluster node (Erlang node name) the child booted on.
string node = 2;
}

// Request to stop a VM.
message StopVmRequest {
// The id of the VM to tear down.
Expand Down
20 changes: 17 additions & 3 deletions setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ sudo -v
info "OS packages"
sudo apt-get update
sudo apt-get install -y \
coreutils e2fsprogs libc-bin lvm2 skopeo thin-provisioning-tools \
util-linux "linux-modules-extra-$(uname -r)"
coreutils e2fsprogs libc-bin lvm2 nftables iproute2 skopeo \
thin-provisioning-tools util-linux "linux-modules-extra-$(uname -r)"

info "device-mapper modules"
# -a is load-bearing: without it modprobe reads the 2nd+ names as module
Expand All @@ -53,6 +53,11 @@ printf 'dm_snapshot\ndm_thin_pool\nloop\n' \
sudo dmsetup targets | grep -q thin-pool \
|| fail "thin-pool dm target missing after modprobe"

info "IPv4 forwarding (required — host-init only asserts it, never sets it)"
sudo sysctl -w net.ipv4.ip_forward=1 >/dev/null
echo 'net.ipv4.ip_forward=1' \
| sudo tee /etc/sysctl.d/99-hyper-ip-forward.conf >/dev/null

# Ubuntu ships thin_dump as a symlink into pdata_tools; the helper's SafeBin
# rejects symlinks, so install a dereferenced hard copy under the name the
# multi-call binary dispatches on. Skip when a prior run already placed it --
Expand All @@ -72,8 +77,12 @@ if [ -f /etc/hyper/config.toml ]; then
info "/etc/hyper/config.toml exists -- leaving it alone"
else
info "writing /etc/hyper/config.toml"
# A single-uplink host NATs guest egress out of its default-route NIC.
uplink="$(ip route show default | awk '{print $5; exit}')"
[ -n "$uplink" ] \
|| fail "could not detect a default-route interface -- set [network] uplink in /etc/hyper/config.toml by hand (see docs/cookbook/install.md)"
sudo mkdir -p /etc/hyper
sudo tee /etc/hyper/config.toml >/dev/null <<'EOF'
sudo tee /etc/hyper/config.toml >/dev/null <<EOF
work_dir = "/srv/hyper"

[tools]
Expand All @@ -84,6 +93,11 @@ thin_dump = "/usr/local/sbin/thin_dump"
[jails]
uid_gid_range = [900000, 999999]
cgroup = "hyper"

[network]
uplink = "${uplink}"
clone_pool = "172.31.0.0/16"
resolver = "1.1.1.1"
EOF
sudo chown root:root /etc/hyper/config.toml
sudo chmod 0644 /etc/hyper/config.toml
Expand Down
Loading
Loading