diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c6d460d8..c4197f0e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -38,16 +38,18 @@ jobs: run: | version="${GITHUB_REF_NAME#v}" - # The tag is the single source of truth: stamp it into both manifests. - # Cargo.toml's [package] version is the only line starting with - # `version = `; mix.exs has exactly one `version: "..."` (the project - # version). `0,/re/` rewrites just the first match. - sed -i -E "0,/^version = \"[^\"]*\"/s//version = \"${version}\"/" native/suidhelper/Cargo.toml + # The tag is the single source of truth: stamp it into every package + # manifest. `0,/re/` rewrites only the first match, which for each + # Cargo.toml is its own `[package] version` (xtask is intentionally + # left at 0.0.0 — it is publish = false). + for m in native/suidhelper/Cargo.toml native/suidhelper/meta/Cargo.toml native/guest-agent/Cargo.toml; do + sed -i -E "0,/^version = \"[^\"]*\"/s//version = \"${version}\"/" "$m" + done sed -i -E "0,/version: \"[^\"]*\"/s//version: \"${version}\"/" mix.exs git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add native/suidhelper/Cargo.toml mix.exs + git add native/suidhelper/Cargo.toml native/suidhelper/meta/Cargo.toml native/guest-agent/Cargo.toml mix.exs # Idempotent: a re-run after the bump already landed has nothing to commit. if git diff --cached --quiet; then @@ -133,6 +135,18 @@ jobs: - name: Install dependencies run: mix deps.get + # The gitignored gRPC/guest-agent/suidhelper artifacts are regenerated by + # Mix compilers on every compile; hex.build compiles, so the runner needs + # their toolchain. Mirrors ci.yml. + - name: Install protoc + run: sudo apt-get update && sudo apt-get install -y protobuf-compiler + + - name: Install protoc-gen-elixir + run: mix escript.install hex protobuf 0.17.0 --force + + - name: Install musl target for the guest-agent build + run: rustup target add x86_64-unknown-linux-musl + - name: Build package tarball run: mix hex.build --output "hypervm-${{ needs.prepare-release.outputs.version }}.tar" @@ -191,6 +205,18 @@ jobs: - name: Install dependencies run: mix deps.get + # The gitignored gRPC/guest-agent/suidhelper artifacts are regenerated by + # Mix compilers on every compile; hex.build compiles, so the runner needs + # their toolchain. Mirrors ci.yml. + - name: Install protoc + run: sudo apt-get update && sudo apt-get install -y protobuf-compiler + + - name: Install protoc-gen-elixir + run: mix escript.install hex protobuf 0.17.0 --force + + - name: Install musl target for the guest-agent build + run: rustup target add x86_64-unknown-linux-musl + # HEX_API_KEY authenticates non-interactively; --yes skips the prompt. # Publishes both the package and its docs. - name: hex publish diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..b25ab8e2 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,28 @@ +# Changelog + +All notable changes to this project are documented here. The format is based on +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project +adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +## [0.1.0] + +First public release: a distributed orchestrator for Firecracker microVMs on the +BEAM. + +### Added +- Firecracker microVM lifecycle: create, stop, locate, and list VMs across a + cluster, with automatic layer-affinity placement. +- Content-addressed OCI image loading into a shared media store + image database. +- Copy-on-write disk **forking** (`Hyper.Vm.fork/1` and the `ForkVm` gRPC RPC): + colocated thin-snapshot forks with a cluster-wide delta-layer fallback. +- Per-VM egress networking: netns + TAP + nftables NAT. +- Granular per-VM compute metering (CPU-time, from the cgroup) for billing. +- Guest exec over vsock (`Hyper.exec/3`). +- gRPC interface (`hyper.grpc.v0`, UNSTABLE) — off by default, unauthenticated. +- Privileged Rust setuid helper for the host operations the BEAM cannot do safely. +- OpenTelemetry tracing across the subsystems. + +[Unreleased]: https://github.com/harmont-dev/hyper/compare/v0.1.0...HEAD +[0.1.0]: https://github.com/harmont-dev/hyper/releases/tag/v0.1.0 diff --git a/README.md b/README.md index d026d917..4b22f24d 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ Then load an OCI image and boot it: ```elixir {:ok, img_id} = Hyper.Img.OciLoader.load("docker.io/library/alpine:3.19") -{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id}) +{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id, type: :micro}) {:ok, %{stdout: "hello\n"}} = Hyper.exec(vm, ["/bin/echo", "hello"]) ``` @@ -84,6 +84,7 @@ integrating Hyper. BEAM-native interfaces, we recognize this may not be ideal for all languages and existing stacks. For that reason, Hyper has a GRPC interface, so you can call it from any language you already use. + ⚠️ The gRPC API is unauthenticated and off by default — bind it to a trusted network or front it with an authenticating proxy. See [the gRPC guide](docs/grpc.md). ## Docs diff --git a/docs/cookbook/config.md b/docs/cookbook/config.md index a67d7a88..1ede588f 100644 --- a/docs/cookbook/config.md +++ b/docs/cookbook/config.md @@ -108,6 +108,32 @@ uid_gid_range = [900000, 999999] ``` +## VM Networking Configuration + +**Required.** Every VM gets NAT'd egress out through a physical host interface, +and a node refuses to start without a `[network]` table (see the +[install guide](install.md#vm-networking-required) for the host prerequisites: +`iproute2`/`nftables` and `net.ipv4.ip_forward=1`). `config.toml`-only — the +setuid helper reads the same `uplink` and `clone_pool` to build each VM's netns, +veth pair, and NAT rules, so it cannot live in `config.exs`. + +### `config.toml` + +```toml +[network] +# **required**. The physical uplink interface guest egress is NAT'd out through. +# On a single-uplink host, the default-route NIC is the usual choice: +# ip route show default | awk '{print $5; exit}' +uplink = "eth0" + +# optional -- defaults shown. The IPv4 CIDR the per-VM /30s are carved from. +clone_pool = "172.31.0.0/16" + +# optional -- defaults shown. DNS resolver handed to guests via the kernel +# cmdline (the guest agent writes it to /etc/resolv.conf). +resolver = "1.1.1.1" +``` + ## gRPC Configuration Hyper supports a [gRPC](https://grpc.io/) interface enabling you to interface diff --git a/docs/cookbook/install.md b/docs/cookbook/install.md index 79925590..fe555d98 100644 --- a/docs/cookbook/install.md +++ b/docs/cookbook/install.md @@ -45,7 +45,7 @@ You can install the required packages by running: sudo dnf install -y \ coreutils \ device-mapper-persistent-data \ - e2fsprogs \ + e2fsprogs \ glibc-common \ kernel-modules-extra-$(uname -r) \ lvm2 \ diff --git a/docs/cookbook/intro.md b/docs/cookbook/intro.md index 69865537..f6d54265 100644 --- a/docs/cookbook/intro.md +++ b/docs/cookbook/intro.md @@ -65,7 +65,7 @@ interface, or through [gRPC](../grpc.md): With the image loaded, and an `img_id` in hand, you can boot it: ```elixir -{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id}) +{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{img_id: img_id, type: :micro}) ``` The VM is scheduled onto the most available node in the cluster, preferring diff --git a/docs/grpc.md b/docs/grpc.md index eacdf98d..cfac8eee 100644 --- a/docs/grpc.md +++ b/docs/grpc.md @@ -8,6 +8,15 @@ create, stop, locate, and list microVMs. > **v0 -- unstable.** The contract may change without notice during early > development. Pin to a commit if you depend on it. +> #### No authentication {: .warning} +> +> The gRPC API has **no authentication**. Any client that can reach the port can +> create and stop VMs and load images. It is **off by default** (`grpc.enabled = +> false`). When you enable it, bind it to loopback or a trusted network, or put +> it behind a proxy that terminates TLS and authenticates callers. TLS (`cred`) +> encrypts the channel but does not authenticate the client. Authentication is +> planned for a later release; the `v0` contract is UNSTABLE. + ## Configuration By default, the gRPC interface is **disabled**. The simplest way to enable it diff --git a/lib/hyper.ex b/lib/hyper.ex index b0d20a67..4cb03cbc 100644 --- a/lib/hyper.ex +++ b/lib/hyper.ex @@ -166,6 +166,19 @@ defmodule Hyper do :error, {:erpc, _} -> {:error, :machine_unreachable} end + @doc """ + Fork the running VM `vm_id`: boot a child from a copy-on-write clone of its + disk, colocated when the parent's node has room and re-placed cluster-wide + otherwise. Returns the child VM. See `Hyper.Vm.fork/1` for the semantics. + """ + @spec fork_vm(Hyper.Vm.Id.t()) :: {:ok, Hyper.Vm.t()} | {:error, term()} + def fork_vm(vm_id) when is_binary(vm_id) do + case Hyper.Cluster.Routing.supervisor_of(vm_id) do + nil -> {:error, :not_found} + pid -> Hyper.Vm.fork(pid) + end + end + @doc """ The vm id for a VM handle -- the pid returned by `create_vm/1`. Resolves on the pid's owning node, so a VM just placed on a remote node is found immediately diff --git a/lib/hyper/grpc/codec.ex b/lib/hyper/grpc/codec.ex index f2b2f6ce..65f7dd8c 100644 --- a/lib/hyper/grpc/codec.ex +++ b/lib/hyper/grpc/codec.ex @@ -13,6 +13,7 @@ defmodule Hyper.Grpc.Codec do alias Hyper.Grpc.V0.{ CreateVmRequest, CreateVmResponse, + ForkVmResponse, GetVmResponse, GetVmUsageResponse, ListVmsResponse, @@ -75,6 +76,10 @@ defmodule Hyper.Grpc.Codec do def to_grpc({:created, vm_id, node}) when is_binary(vm_id), do: %CreateVmResponse{vm_id: vm_id, node: to_string(node)} + @spec to_grpc({:forked, Hyper.Vm.Id.t(), node()}) :: ForkVmResponse.t() + def to_grpc({:forked, vm_id, node}) when is_binary(vm_id), + do: %ForkVmResponse{vm_id: vm_id, node: to_string(node)} + @spec to_grpc({:located, Hyper.Vm.Id.t(), node()}) :: GetVmResponse.t() def to_grpc({:located, vm_id, node}), do: %GetVmResponse{vm_id: vm_id, node: to_string(node)} @@ -126,6 +131,9 @@ defmodule Hyper.Grpc.Codec do defp rpc_error(:machine_unreachable), do: GRPC.RPCError.exception(:unavailable, "VM's host node is unreachable") + defp rpc_error(:node_unreachable), + do: GRPC.RPCError.exception(:unavailable, "VM's host node is unreachable") + defp rpc_error(reason) when reason in [:no_capacity, :exhausted], do: GRPC.RPCError.exception(:resource_exhausted, "no capacity") diff --git a/lib/hyper/grpc/server.ex b/lib/hyper/grpc/server.ex index c7581ed3..00ad7c87 100644 --- a/lib/hyper/grpc/server.ex +++ b/lib/hyper/grpc/server.ex @@ -15,6 +15,8 @@ defmodule Hyper.Grpc.Server do alias Hyper.Grpc.V0.{ CreateVmRequest, CreateVmResponse, + ForkVmRequest, + ForkVmResponse, GetVmRequest, GetVmResponse, GetVmUsageRequest, @@ -50,6 +52,20 @@ defmodule Hyper.Grpc.Server do end end + @spec fork_vm(ForkVmRequest.t(), GRPC.Server.Stream.t()) :: ForkVmResponse.t() + @decorate with_span("Hyper.Grpc.Server.fork_vm", include: [:vm_id]) + def fork_vm(%ForkVmRequest{vm_id: vm_id}, _stream) do + with {:ok, child} <- Hyper.fork_vm(vm_id), + child_id when is_binary(child_id) <- Hyper.id(child) do + Codec.to_grpc({:forked, child_id, node(child)}) + else + # The child was placed but its host became unreachable before its id + # could be confirmed — same shape as create_vm/2. + nil -> raise Codec.to_grpc({:error, :machine_unreachable}) + {:error, reason} -> raise Codec.to_grpc({:error, reason}) + end + end + @spec stop_vm(StopVmRequest.t(), GRPC.Server.Stream.t()) :: Empty.t() @decorate with_span("Hyper.Grpc.Server.stop_vm", include: [:vm_id]) def stop_vm(%StopVmRequest{vm_id: vm_id}, _stream) do diff --git a/native/guest-agent/.gitignore b/native/guest-agent/.gitignore index ea8c4bf7..6b3e3929 100644 --- a/native/guest-agent/.gitignore +++ b/native/guest-agent/.gitignore @@ -1 +1,2 @@ /target +*.profraw diff --git a/native/guest-agent/Cargo.toml b/native/guest-agent/Cargo.toml index 5392bb3b..a1d38c78 100644 --- a/native/guest-agent/Cargo.toml +++ b/native/guest-agent/Cargo.toml @@ -2,6 +2,9 @@ name = "hyper-guest-agent" version = "0.1.0" edition = "2021" +license = "MIT" +description = "PID-1 guest agent for Hyper microVMs: mounts, resolv.conf, and a vsock exec server." +repository = "https://github.com/harmont-dev/hyper" [lib] name = "hyper_guest_agent" diff --git a/native/suidhelper/.gitignore b/native/suidhelper/.gitignore index 264765f4..34721db3 100644 --- a/native/suidhelper/.gitignore +++ b/native/suidhelper/.gitignore @@ -1,2 +1,5 @@ /target /lcov.info +# LLVM source-based coverage drops these at the crate root during instrumented +# test runs (see the xtask coverage pipeline); they are never committed. +*.profraw diff --git a/proto/hyper/grpc/v0/hyper.proto b/proto/hyper/grpc/v0/hyper.proto index 73380576..a9ca98f6 100644 --- a/proto/hyper/grpc/v0/hyper.proto +++ b/proto/hyper/grpc/v0/hyper.proto @@ -36,6 +36,18 @@ service Hyper { // before its id could be confirmed. rpc CreateVm(CreateVmRequest) returns (CreateVmResponse); + // Fork a running microVM: snapshot its disk and boot a colocated child from + // the copy-on-write clone, falling back to a cluster-wide placement (via a + // published delta layer) when the parent's node is full. The parent keeps + // running. The fork is crash-consistent (like a power cut) after a best-effort + // guest sync. Returns the new child's `vm_id` and the node it booted on. + // + // Errors: + // NOT_FOUND -- no VM with this `vm_id` is running in the cluster. + // RESOURCE_EXHAUSTED -- no node in the cluster can admit the child. + // UNAVAILABLE -- the parent's host node became unreachable. + rpc ForkVm(ForkVmRequest) returns (ForkVmResponse); + // Stop and tear down a running microVM. Idempotent in effect: tearing down a // VM that is already stopping is not an error. // @@ -130,6 +142,21 @@ message CreateVmResponse { string node = 2; } +// Request to fork a running VM. +message ForkVmRequest { + // The id of the parent VM to fork. + string vm_id = 1; +} + +// Result of a successful ForkVm. +message ForkVmResponse { + // The server-minted id for the new child VM; use it in all later RPCs. + string vm_id = 1; + + // The cluster node (Erlang node name) the child booted on. + string node = 2; +} + // Request to stop a VM. message StopVmRequest { // The id of the VM to tear down. diff --git a/setup.sh b/setup.sh index 34fa796c..868f7b91 100755 --- a/setup.sh +++ b/setup.sh @@ -41,8 +41,8 @@ sudo -v info "OS packages" sudo apt-get update sudo apt-get install -y \ - coreutils e2fsprogs libc-bin lvm2 skopeo thin-provisioning-tools \ - util-linux "linux-modules-extra-$(uname -r)" + coreutils e2fsprogs libc-bin lvm2 nftables iproute2 skopeo \ + thin-provisioning-tools util-linux "linux-modules-extra-$(uname -r)" info "device-mapper modules" # -a is load-bearing: without it modprobe reads the 2nd+ names as module @@ -53,6 +53,11 @@ printf 'dm_snapshot\ndm_thin_pool\nloop\n' \ sudo dmsetup targets | grep -q thin-pool \ || fail "thin-pool dm target missing after modprobe" +info "IPv4 forwarding (required — host-init only asserts it, never sets it)" +sudo sysctl -w net.ipv4.ip_forward=1 >/dev/null +echo 'net.ipv4.ip_forward=1' \ + | sudo tee /etc/sysctl.d/99-hyper-ip-forward.conf >/dev/null + # Ubuntu ships thin_dump as a symlink into pdata_tools; the helper's SafeBin # rejects symlinks, so install a dereferenced hard copy under the name the # multi-call binary dispatches on. Skip when a prior run already placed it -- @@ -72,8 +77,12 @@ if [ -f /etc/hyper/config.toml ]; then info "/etc/hyper/config.toml exists -- leaving it alone" else info "writing /etc/hyper/config.toml" + # A single-uplink host NATs guest egress out of its default-route NIC. + uplink="$(ip route show default | awk '{print $5; exit}')" + [ -n "$uplink" ] \ + || fail "could not detect a default-route interface -- set [network] uplink in /etc/hyper/config.toml by hand (see docs/cookbook/install.md)" sudo mkdir -p /etc/hyper - sudo tee /etc/hyper/config.toml >/dev/null <<'EOF' + sudo tee /etc/hyper/config.toml >/dev/null < Hyper.Node.stop_image_vm(parent) end) + + parent_id = Hyper.id(parent) + assert parent_id, "Hyper.id/1 returned nil for a freshly-created VM" + + assert {:ok, %ForkVmResponse{vm_id: child_id, node: child_node}} = + Stub.fork_vm(channel, %ForkVmRequest{vm_id: parent_id}) + + assert is_binary(child_id) and child_id != parent_id + assert child_node != "" + + on_exit(fn -> Stub.stop_vm(channel, %StopVmRequest{vm_id: child_id}) end) + end + defp ensure_node_deps! do if not File.dir?(Path.join(@suite_dir, "node_modules")) do {out, status} = System.cmd("npm", ["ci"], cd: @suite_dir, stderr_to_stdout: true) diff --git a/test/hyper/grpc/codec_fork_test.exs b/test/hyper/grpc/codec_fork_test.exs new file mode 100644 index 00000000..273d3eee --- /dev/null +++ b/test/hyper/grpc/codec_fork_test.exs @@ -0,0 +1,17 @@ +defmodule Hyper.Grpc.CodecForkTest do + use ExUnit.Case, async: true + + alias Hyper.Grpc.Codec + alias Hyper.Grpc.V0.ForkVmResponse + + test "a forked result maps to a ForkVmResponse with the child id and node string" do + assert %ForkVmResponse{vm_id: "child-abc", node: "hyper@host"} = + Codec.to_grpc({:forked, "child-abc", :hyper@host}) + end + + test "node_unreachable maps to an UNAVAILABLE gRPC error" do + err = Codec.to_grpc({:error, :node_unreachable}) + assert %GRPC.RPCError{status: status} = err + assert status == GRPC.Status.unavailable() + end +end