From a29c15b4508ce64cd932fbb4cb716229cfcff0f5 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 7 Jul 2026 10:56:05 +0500 Subject: [PATCH 1/3] design-proposal: tenant external network connectivity via a gateway VM Adds a design proposal for tenant-managed site-to-site connectivity (IPsec or WireGuard) terminated in a KubeVirt VM that NAT-bridges an external site to the tenant's managed-app ClusterIPs, both directions, without granting tenants privileged host-cluster pods. Includes the end-to-end prototype validation results and the relationship to the ClusterMesh/Kilo proposal (#7). Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- .../tenant-site-gateway/README.md | 222 ++++++++++++++++++ 1 file changed, 222 insertions(+) create mode 100644 design-proposals/tenant-site-gateway/README.md diff --git a/design-proposals/tenant-site-gateway/README.md b/design-proposals/tenant-site-gateway/README.md new file mode 100644 index 0000000..4cdd041 --- /dev/null +++ b/design-proposals/tenant-site-gateway/README.md @@ -0,0 +1,222 @@ +# Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard) + +- **Title:** `Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard)` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-07-07` +- **Status:** Draft + +## Overview + +Cozystack tenants need site-to-site VPN connectivity between their workloads and external networks, in both directions: exposing a tenant-managed application to a remote site (inbound), and letting a tenant-managed application reach a service that lives only behind a tunnel (outbound). This must work for any application — databases, message queues, object storage, gRPC, raw UDP — not one special case. + +This proposal terminates the tunnel (IPsec or WireGuard) **inside a KubeVirt VM** rather than in a pod. The privileged dataplane (`NET_ADMIN`, XFRM / a `wireguard` interface, `ip_forward`) lives in a guest kernel isolated by hardware virtualization, so its blast radius is contained by the VM boundary — the host node and other tenants are unaffected. The gateway VM is a normal member of the tenant's pod network, so it reaches any tenant `ClusterIP` Service; it performs all address translation (DNAT/SNAT), and the managed applications are never modified and never speak the VPN — they only ever see ordinary `ClusterIP`s on their own network. We propose shipping this as a Cozystack catalog app, provisionally `packages/apps/site-gateway`. + +The design has been prototyped and validated end-to-end (see [Testing](#testing)). + +## Scope and related proposals + +- **Related — [`cross-cluster-tenant-mesh` (ClusterMesh / Kilo)](https://github.com/cozystack/community/pull/7).** That proposal connects cooperative Kubernetes clusters into a routed WireGuard node-to-node mesh for high-throughput cases (its motivating example is a tenant cluster consuming host Ceph). This proposal is complementary and occupies a different niche — an arbitrary external site (not a Kubernetes cluster), NAT-bridged to managed apps that live in the tenant namespace. See [Alternatives considered](#alternatives-considered) for a side-by-side and a "why two mechanisms" discussion. Notably, PR #7 explicitly defers the "tenant ↔ tenant-namespace applications" and "NAT-egress" integration — precisely the space this proposal fills. +- **Deferred here:** a platform-run (non-VM) managed variant; HTTP/L7 ingress (the existing tenant Ingress / Gateway API already covers public HTTP); preservation of the original client source IP inbound. + +## Context + +Cozystack runs tenant workloads under a hardened networking model: + +- **Namespace hardening.** Cozystack applies Pod Security Standards to tenant namespaces. The platform operator only sets `pod-security.kubernetes.io/enforce=privileged` on a namespace when a platform-declared package requests it. A regular tenant cannot schedule a privileged, `NET_ADMIN`, or `hostNetwork` pod — deliberately, because such a pod executes against the node kernel and could manipulate host networking or observe other tenants' traffic. +- **Default pod network vs. custom VPC.** Managed apps (databases and friends) run as ordinary pods on the **default cluster pod network** and are reached via `ClusterIP` Services + CoreDNS. They are not placed on a custom kube-ovn VPC; VPCs are isolated from the default network and the managed-app operators reconcile over the default network — so a managed app cannot simply be "moved onto a VPC." +- **Dual-homed VMs.** KubeVirt VMs are the exception: every VM always has a `default` pod-network NIC (and can carry additional multus interfaces). A VM is therefore a first-class member of the default pod network. This is the primitive the design leans on. + +### The problem + +A tenant today has no supported way to: + +- **Inbound:** let a remote site reach a tenant-managed app (a Postgres instance, an internal API, an object store) over a tunnel, without exposing it to the public internet. +- **Outbound:** let a tenant app call a service (a legacy database, a licensing server, an on-prem API) that is only routable through a tunnel. + +The naive approaches all require either a privileged host-cluster pod (a tenant-escape vector — defeats the hardening above) or changes to shared, cluster-wide network infrastructure (cross-tenant collision risk). We want a self-service, tenant-scoped mechanism that generalizes to any protocol and touches neither the host kernel nor the managed apps. + +## Goals + +- Give tenants **self-service** site-to-site connectivity, both directions, through a catalog app. +- Support **inbound** exposure of any tenant-managed app to a remote site over the tunnel. +- Support **outbound** connections from any tenant app to services reachable only via the tunnel. +- Be **app-agnostic** — work for any L4 protocol (TCP/UDP, any port) without modifying the target apps. +- Keep the privileged dataplane **contained** in a VM guest; add no privileges to the host cluster and none to managed apps. +- Keep all bridging/NAT **tenant-scoped**; make no changes to shared cluster routers. +- Support **both IPsec and WireGuard** as tunnel backends (interop vs. simplicity — see [Design](#design)). + +### Non-goals + +- **HTTP/L7 ingress.** Public HTTP(S) is served by the existing tenant Ingress / Gateway API; the gateway VM is for site-to-site VPN and non-HTTP protocols. +- **Preserving the original client source IP inbound.** SNAT hides it (see [Failure and edge cases](#failure-and-edge-cases)); source-IP ACLs are out of scope for the first iteration. +- **A platform-managed, always-on VPN service.** This is a tenant-deployed catalog app; a platform-run variant is possible future work. +- **Routing a remote CIDR onto the shared default-VPC router** — explicitly rejected (cross-tenant collisions). + +## Design + +### Principle + +Terminate the tunnel in a KubeVirt VM. The VM guest kernel isolates `NET_ADMIN`/XFRM from the host, so the tenant effectively runs a privileged network appliance without the platform ever granting a privileged pod. The VM is dual-homed on the tenant's default pod network and is the sole holder of the tunnel; it DNAT/SNATs between the tunnel and the tenants' managed-app `ClusterIP`s. + +### Topology + +```mermaid +flowchart TB + ext["External site
(on-prem router / cloud VPN / another org)"] + lb["LoadBalancer VIP
(tunnel UDP ports)"] + subgraph node["Worker node"] + gw["Gateway VM (VyOS)
tunnel termination + DNAT/SNAT + firewall + MSS clamp"] + a["managed app A
(ClusterIP)"] + b["managed app B
(ClusterIP)"] + c["managed app C
(ClusterIP)"] + end + ext -->|"IPsec ESP-in-UDP / WireGuard UDP"| lb --> gw + gw --> a & b & c +``` + +Everything privileged lives inside the VM guest. Managed apps only ever see `ClusterIP`s on their own default network. + +### Inbound path (external site → any tenant app) + +The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. + +1. Remote peer sends tunnel traffic to the gateway's public VIP (a `LoadBalancer` Service on the tunnel's UDP ports). +2. The VIP delivers to the gateway VM's pod NIC; the VM decrypts the tunnel. +3. **DNAT**: the virtual destination (e.g. `10.200.0.10:5432`) → the target app's `ClusterIP:port`. One entry per exposed app — the exposure table. +4. **SNAT (masquerade)**: source → the gateway's own pod IP. Mandatory: the cluster has no route back to the remote peer's inner IP, so without SNAT the app's reply would leave via the default route and be black-holed. +5. The VM forwards onto the pod network; the node resolves the `ClusterIP` to a backend pod; the app sees the connection with the gateway pod IP as client. Replies retrace the path and are re-encrypted back over the tunnel. + +### Outbound path (any tenant app → remote service over tunnel) + +The tenant app never routes to the remote address directly; it talks to a **local Service** whose endpoint is the gateway VM. + +1. The app connects to a local Service name the chart creates (e.g. `remote-db..svc:port`) — the only app-side change is the target hostname. +2. The node routes the `ClusterIP` to the gateway VM. +3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. + +**Rejected alternative — a static route on the shared default-VPC router.** Injecting a tenant's remote RFC1918 CIDR into the cluster-wide default router risks cross-tenant address collisions and leakage. The local-Service approach keeps every remote address strictly inside the tenant's gateway VM. + +### Why it generalizes to all apps + +- The app only ever sees a `ClusterIP` / Service on its own network; it speaks no VPN and needs no route to the far side. +- DNAT is **L4** (TCP and UDP, any port), not L7 — so it works for databases, queues, object storage, gRPC, and raw UDP, unlike an HTTP-only path. +- The gateway VM is the only privileged component, and it is a contained VM guest. + +### Tunnel backend: IPsec or WireGuard + +VyOS terminates both natively and the DNAT/SNAT bridging is identical — only the transport differs. `tunnel.type` selects the backend per peer. + +- **WireGuard — the simpler default on an overlay CNI.** WireGuard is UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP: no ESP, no NAT-T, no forced encapsulation. It is simpler (static keypairs, no IKE) and has smaller header overhead. Preferred for tenant↔tenant and tenant↔Cozystack links. (See the validated finding in [Testing](#testing) that makes this concrete.) +- **IPsec — for interop.** Much external/enterprise gear only speaks IKEv2/IPsec; when the remote site is not under the tenant's control this is often the only option. Supported, with the encapsulation requirement described in [Testing](#testing). + +### Build options + +- **VyOS appliance (recommended).** A single network OS does IPsec, WireGuard, native DNAT/SNAT, and firewalling; no separate proxy needed; configured declaratively via cloud-init. +- **Alpine + libreswan/strongSwan (or wireguard-tools) + haproxy.** A leaner, self-composed image if a network-OS dependency is undesirable; more moving parts. + +## User-facing changes + +A new catalog app, provisionally **`packages/apps/site-gateway`** (VM-backed), appears in the tenant dashboard. The tenant fills a values form; the platform-authored chart renders the gateway VM, its tunnel + NAT + firewall config, the local Services for outbound targets, and a `LoadBalancer` Service for the tunnel's UDP ports. Sketch of the values schema (cozyvalues-gen annotated): + +```yaml +## @param tunnel.type {string} Backend: "wireguard" (default; UDP-native) or "ipsec". +tunnel: + type: wireguard + +## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. +peer: + address: "" + ## backend-specific auth: IPsec IKE/PSK-or-cert; WireGuard peer public key + allowedIPs. + auth: + secretRef: "" + +## @param exposedServices {array} Inbound: {name, listenPort, targetService, port} +exposedServices: [] + +## @param remoteTargets {array} Outbound: {name, localPort, remoteHost, remotePort} +remoteTargets: [] + +## @param resources {object} VM sizing (cpu/memory). +resources: + cpu: "1" + memory: "1Gi" +``` + +No existing app, CRD, or API changes. + +## Upgrade and rollback compatibility + +Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk` and the generated Services), which removes the gateway entirely; nothing else in the tenant is touched. + +## Security + +- **Contained privilege.** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise or misconfiguration inside the gateway cannot manipulate the host node's networking or observe other tenants. The platform never hands a tenant a privileged host-cluster pod. +- **Managed apps untouched.** No new privileges or config on the apps; they stay ordinary pods on the default network. +- **Tenant-scoped bridging.** All translation is NAT inside the VM; nothing changes on the shared cluster router, so there is no cross-tenant surface. +- **Firewall allow-list.** The gateway restricts which tunnel-side sources may reach which exposed ports — exposure is explicit, not "the whole tunnel reaches everything." +- **Tenant-supplied secrets.** PSK / certificate / WireGuard key material is provided via a Secret reference and mounted into the guest; the chart must avoid persisting it in plaintext cloud-init user-data at rest (open question). + +## Failure and edge cases + +- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). +- **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an MSS clamp by default. +- **Native ESP dropped by the CNI overlay (IPsec).** On Cilium/kube-ovn, native ESP (IP proto 50) does not traverse the overlay even pod-to-pod; ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated — see Testing). WireGuard, being UDP-native, is unaffected. +- **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. +- **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. +- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). +- **Single gateway is a per-tenant SPOF** until HA is added (open question). + +## Testing + +The design was **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target. IPsec was validated first; WireGuard is the next backend to validate (open question). All items passed: + +| # | Item | Result | +|---|------|--------| +| 1 | IKEv2 SA establishes both sides | PASS | +| 2 | Inbound: external site → virtual VIP → DNAT/SNAT → managed Postgres (real server response) | PASS | +| 3 | Outbound: gateway client → tunnel → remote listener | PASS | +| 4 | SNAT-required negative test (removing SNAT black-holes the reply) | PASS | +| 5 | MTU / MSS clamp (working tunnel MTU 1320, clamp 1280; multi-MB transfer, no stall) | PASS | + +**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. Consequence: on an overlay CNI the IPsec backend must always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. + +Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows and the SNAT-required and MSS behaviors. + +## Rollout + +- **Phase 1 — IPsec backend** (prototype validated): ship `site-gateway` with the IPsec backend (forced UDP encapsulation) + the inbound/outbound NAT machinery. +- **Phase 2 — WireGuard backend**: validate WireGuard over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. +- **Phase 3 — hardening**: HA (active/standby, shared VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. + +## Open questions + +- **Secret handling.** Best shape for mounting PSK/cert/WireGuard-key material into the guest without persisting it in plaintext cloud-init at rest. +- **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. +- **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. +- **Observability.** Surfacing tunnel state (SA up/down, rekey, counters) through existing dashboards. +- **MTU auto-tuning.** Derive the MSS clamp / tunnel MTU from the detected overlay MTU, or require an explicit value. +- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical DNAT/SNAT bridging before defaulting to it. +- **Relationship to ClusterMesh (PR #7).** Whether to present this and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits. + +## Alternatives considered + +- **Give tenants privileged / `NET_ADMIN` host pods.** Rejected — tenant escape; defeats namespace hardening. The whole design exists to avoid this. +- **Route the remote CIDR into the shared default-VPC router.** Rejected — the router is shared cluster-wide; injecting tenant RFC1918 CIDRs causes cross-tenant collisions and leakage. +- **Put managed apps on a custom kube-ovn VPC.** Not viable — VPCs are isolated from the default network and managed-app operators reconcile over the default network; only VMs are dual-homed. +- **Platform-run kube-ovn VPC-NAT-Gateway with the tunnel.** Possible future managed variant, but more platform work and it couples XFRM to the host kernel + OVS. The VM approach contains the privilege more cleanly and ships sooner. +- **Traefik / L7 proxy.** Limited — HTTP-only; not general across L4 protocols. + +### ClusterMesh / Kilo (PR #7), and why two mechanisms + +[PR #7](https://github.com/cozystack/community/pull/7) pursues an overlapping goal (tenant access to services across a boundary) but is a different shape, and neither mechanism subsumes the other: + +| Axis | ClusterMesh / Kilo (PR #7) | This proposal (gateway VM) | +|------|---------------------------|----------------------------| +| Remote end | A cooperative Kubernetes cluster running Kilo, reached via a kubeconfig | An arbitrary external site — not Kubernetes, no Kilo, no kubeconfig | +| Tenant workload served | The tenant's managed Kubernetes cluster (guest-VM pods) | Managed apps in the host-cluster tenant namespace (ClusterIPs) | +| Transport | WireGuard node-to-node mesh (`mesh-granularity=cross`) | IPsec or WireGuard site-to-site, one VM | +| Address plane | Routed pod-CIDRs, no NAT; disjoint CIDRs required | NAT (DNAT/SNAT) to ClusterIPs; tolerates overlap | +| Throughput | Full node×node mesh — direct pod-IP to many backends (built for Ceph) | Single gateway — app-level flows, not Ceph-scale | +| Privilege | Kilo node `NET_ADMIN`, on the tenant side inside guest-cluster VMs → contained | `NET_ADMIN` inside the gateway VM guest → contained | + +**Kilo cannot terminate a third-party VPN** (both ends must run Kilo with a kubeconfig), and **a single gateway VM cannot serve the Ceph-scale mesh** (throughput/topology incompatible with funneling). They are different layers: a platform storage fabric vs. a tenant-edge VPN concentrator — analogous to a cloud provider shipping both VPC Peering and a VPN Gateway. Both share the principle that matters here: no privileged pods in the shared host namespace; privilege is contained in guest-cluster VMs (Kilo) or a dedicated gateway VM (this proposal). Recommended framing: lead with the gateway VM as the single tenant-facing primitive (covers external sites, tenant↔tenant, and managed-app exposure), and use ClusterMesh only for the high-throughput cross-cluster storage case. From 72db47ee62705a39492f9acde6fe0b92a1fe5110 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 7 Jul 2026 17:36:58 +0500 Subject: [PATCH 2/3] design-proposal: add HA section and address review feedback Adds a High availability section (KubeVirt live-migration for planned maintenance; Service-fronted active/passive and kube-ovn allowed-address-pairs shared-VIP for unplanned failure), with the shared-VIP mechanism validated on a development cluster. Attributes the pod-to-pod drop of non-TCP/UDP/ICMP/SCTP IP protocols (VRRP proto 112, ESP proto 50) to Cilium's conntrack rather than the geneve tunnel, unifying it with the native-ESP finding. Also addresses review feedback: per-target-unique outbound listener ports, a concrete blockSize example, a note that SNAT keeps the gateway anti-spoofing clean, and a stronger secret-handling open question. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- .../tenant-site-gateway/README.md | 32 ++++++++++++++----- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/design-proposals/tenant-site-gateway/README.md b/design-proposals/tenant-site-gateway/README.md index 4cdd041..85878c5 100644 --- a/design-proposals/tenant-site-gateway/README.md +++ b/design-proposals/tenant-site-gateway/README.md @@ -91,8 +91,8 @@ The remote peer targets a virtual "service-exposure" address that the gateway ow The tenant app never routes to the remote address directly; it talks to a **local Service** whose endpoint is the gateway VM. 1. The app connects to a local Service name the chart creates (e.g. `remote-db..svc:port`) — the only app-side change is the target hostname. -2. The node routes the `ClusterIP` to the gateway VM. -3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. +2. The node routes the `ClusterIP` to the gateway VM on a **per-target-unique listener port** (the local Service still exposes the standard port, e.g. 5432, mapped to a unique port on the VM). The uniqueness is required so the VM can tell apart multiple remote targets that share the same destination port. +3. **DNAT**: that unique local listener port → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. **Rejected alternative — a static route on the shared default-VPC router.** Injecting a tenant's remote RFC1918 CIDR into the cluster-wide default router risks cross-tenant address collisions and leakage. The local-Service approach keeps every remote address strictly inside the tenant's gateway VM. @@ -114,6 +114,20 @@ VyOS terminates both natively and the DNAT/SNAT bridging is identical — only t - **VyOS appliance (recommended).** A single network OS does IPsec, WireGuard, native DNAT/SNAT, and firewalling; no separate proxy needed; configured declaratively via cloud-init. - **Alpine + libreswan/strongSwan (or wireguard-tools) + haproxy.** A leaner, self-composed image if a network-OS dependency is undesirable; more moving parts. +### High availability + +HA splits by failure mode: + +- **Planned maintenance (node drain):** KubeVirt **live-migration** relocates the gateway VM with its state intact (conntrack, tunnel SA, pod IP preserved) — no tunnel drop. Preferred for maintenance; needs no standby. (Requires migratable storage — see the storage caveat below.) +- **Unplanned node/VM failure:** needs a standby plus a failover trigger. Two mechanisms, both examined on kube-ovn (see [Testing](#testing) for the shared-VIP validation): + + 1. **Service-fronted active/passive.** The external `LoadBalancer` (tunnel) and internal `ClusterIP` (outbound) Services select whichever gateway pod is active; failover = re-pointing endpoints when the active goes unready. The stable address is the Service VIP (CNI-managed) — no floating L2 VIP, so no port-security interaction at all. Cost: needs a small leader-election/lease agent (or controller) to flip the active endpoint, and failover takes seconds (endpoint reconvergence). CNI-agnostic. + 2. **Shared VIP via kube-ovn allowed-address-pairs (AAP).** A kube-ovn `Vip` plus the `ovn.kubernetes.io/aaps` pod annotation adds *only* the VIP to the port's OVN `port_security` (which stays enforced) — validated: the VIP is reachable, moves between the two gateway pods through OVN on owner change, and any other source address is still dropped, so anti-spoofing is **scoped rather than disabled** and the VM stays fully tenant-controlled (no need to withhold direct VM/cloud-init access). **Caveat:** VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived split-brains and cannot elect a master over the pod network. Root cause (independently verified): the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols pod-to-pod — in policy-enforced tenant pods this is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"), and without a policy it is the OVS datapath; it is **not** geneve/inter-node-specific (same-node pods drop it too). This is the same root cause that makes native ESP need UDP encapsulation. To use this path the election must therefore run over a side-channel (e.g. the gateway pair's own tunnel link) or be driven by a controller; the AAP VIP move itself is proven. Also the `aaps` annotation must be baked into the KubeVirt VM's pod template (the current `VMInstance` chart does not expose it), and the VIP rides the pod's real MAC (keep the VyOS default — no virtual-MAC). + +**Recommendation:** iteration 1 ships a single gateway VM with live-migration for maintenance, and treats automatic unplanned-failure HA as a follow-up — neither mechanism is a clean drop-in today (Service-based needs a controller; AAP needs the VRRP-advert-delivery issue resolved). + +**Storage caveat:** live-migration and single-VM reschedule need migratable/replicated storage — but replicated (DRBD) StorageClasses expose 4K sectors, so the image needs the `blockSize` override to boot (see [Failure and edge cases](#failure-and-edge-cases)). A two-VM active/passive pair can instead use node-local 512-native disks. + ## User-facing changes A new catalog app, provisionally **`packages/apps/site-gateway`** (VM-backed), appears in the tenant dashboard. The tenant fills a values form; the platform-authored chart renders the gateway VM, its tunnel + NAT + firewall config, the local Services for outbound targets, and a `LoadBalancer` Service for the tunnel's UDP ports. Sketch of the values schema (cozyvalues-gen annotated): @@ -158,12 +172,12 @@ Purely additive and opt-in. No migration; existing clusters, manifests, and APIs ## Failure and edge cases -- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). +- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). It also keeps the gateway **anti-spoofing-clean**: every packet it emits onto the pod network carries its own pod IP, so the CNI's port-security never sees a foreign source — the dataplane needs no port-security relaxation (that only arises for shared-VIP HA; see High availability). - **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an MSS clamp by default. -- **Native ESP dropped by the CNI overlay (IPsec).** On Cilium/kube-ovn, native ESP (IP proto 50) does not traverse the overlay even pod-to-pod; ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated — see Testing). WireGuard, being UDP-native, is unaffected. +- **Native ESP dropped pod-to-pod (IPsec).** The CNI drops non-TCP/UDP/ICMP/SCTP IP protocols between pods — native ESP (proto 50) included — so ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated). In policy-enforced tenant pods the dropper is Cilium's conntrack ("Unknown L4 protocol"); it is not geneve-specific (same root cause as the VRRP/proto-112 case — see Testing). WireGuard, being UDP-native, is unaffected. - **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. - **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. -- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). +- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override (e.g. `blockSize.custom.logical: 512` / `physical: 4096` on the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). - **Single gateway is a per-tenant SPOF** until HA is added (open question). ## Testing @@ -180,19 +194,21 @@ The design was **prototyped and validated end-to-end** on a development Cozystac **Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. Consequence: on an overlay CNI the IPsec backend must always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. +**HA mechanism validation (kube-ovn AAP).** Separately validated (kube-ovn v1.15.10): a `Vip` + `ovn.kubernetes.io/aaps` pod annotation shares a VIP across the two gateway VM pods with `port_security` kept **on**. Proven: (1) without AAP the VIP is dropped (GARP/ping blocked); (2) with AAP the VIP is reachable and lands on the active pod; (3) forcing the active down moves the VIP to the standby through OVN; (4) a bogus source address from the VM is still dropped — anti-spoofing is scoped, not disabled. **Not** working: VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived cannot elect over the pod network — the election needs a side-channel or controller (see [High availability](#high-availability)). Independently re-verified with a minimal repro: the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols (112, 50, 47, 4) pod-to-pod, both same-node and cross-node — so this is **not** the geneve tunnel. In policy-enforced tenant pods the dropper is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"); without a policy it is the OVS datapath. This is the same root cause as the native-ESP drop above. An upstream issue, if pursued, targets Cilium (a known conntrack limitation); the OVS-side drop is not yet root-caused. + Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows and the SNAT-required and MSS behaviors. ## Rollout - **Phase 1 — IPsec backend** (prototype validated): ship `site-gateway` with the IPsec backend (forced UDP encapsulation) + the inbound/outbound NAT machinery. - **Phase 2 — WireGuard backend**: validate WireGuard over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. -- **Phase 3 — hardening**: HA (active/standby, shared VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. +- **Phase 3 — hardening**: HA (see [High availability](#high-availability) — Service-based or AAP shared-VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. ## Open questions -- **Secret handling.** Best shape for mounting PSK/cert/WireGuard-key material into the guest without persisting it in plaintext cloud-init at rest. +- **Secret handling.** Deliver PSK/cert/WireGuard-key material to the guest without persisting it in plaintext cloud-init at rest — candidate: mount the referenced `Secret` as a separate config disk (or fetch it on first boot via a small init) instead of inlining it in the VM spec; the post-reconciliation read-access boundary needs to be defined. - **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. -- **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. +- **HA mechanism.** Given VRRP advertisements do not cross the kube-ovn overlay (see Testing), which failover trigger to standardize on — Service/endpoint-based (needs a controller/lease) or AAP shared-VIP with election over a side-channel — and baking the `aaps` annotation into the `VMInstance` chart's pod template. - **Observability.** Surfacing tunnel state (SA up/down, rekey, counters) through existing dashboards. - **MTU auto-tuning.** Derive the MSS clamp / tunnel MTU from the detected overlay MTU, or require an explicit value. - **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical DNAT/SNAT bridging before defaulting to it. From 4672a2ddda23ce833877304c4d777406e760c16a Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 10 Jul 2026 18:08:01 +0500 Subject: [PATCH 3/3] design-proposal: restructure into site-router + site-gateway (routed + NAT) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Split the tenant site-to-site connectivity proposal into two co-equal catalog apps over a shared foundation — site-router (routed, source-IP preserving, kube-ovn) and site-gateway (NAT, CNI-agnostic) — instead of a single NAT gateway with routed as a fallback, and rename the proposal to tenant-site-connectivity. Adds day-2 configuration (VyOS HTTPS API driven by a controller), a low-level-entities/controller section, and publishes the tunnel endpoint via the structured external-exposure primitives (#29) instead of a bespoke LoadBalancer. Tightens secret handling: the chart templates the Secret from values (no tenant-created Secret), the WireGuard key is autogenerated in-chart, and the IPsec PSK is the only genuinely tenant-supplied secret. Reorders rollout to lead with routed + MSS clamping + tunnel observability. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- .../tenant-site-connectivity/README.md | 251 ++++++++++++++++++ .../tenant-site-gateway/README.md | 238 ----------------- 2 files changed, 251 insertions(+), 238 deletions(-) create mode 100644 design-proposals/tenant-site-connectivity/README.md delete mode 100644 design-proposals/tenant-site-gateway/README.md diff --git a/design-proposals/tenant-site-connectivity/README.md b/design-proposals/tenant-site-connectivity/README.md new file mode 100644 index 0000000..9989739 --- /dev/null +++ b/design-proposals/tenant-site-connectivity/README.md @@ -0,0 +1,251 @@ +# Tenant-managed site-to-site connectivity via gateway VMs (`site-router` and `site-gateway`) + +- **Title:** `Tenant-managed site-to-site connectivity via gateway VMs (site-router and site-gateway)` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-07-08` +- **Status:** Draft + +## Overview + +Cozystack tenants need site-to-site connectivity between their workloads and external networks, in both directions — exposing a tenant-managed application to a remote site (inbound), and letting a tenant-managed application reach a service that lives only behind a tunnel (outbound) — and it must work for any application (databases, message queues, object storage, gRPC, raw UDP), not one special case. + +The privileged network dataplane (`NET_ADMIN`, XFRM / a `wireguard` interface, `ip_forward`) is terminated **inside a KubeVirt VM**, not a pod: the guest kernel is isolated by hardware virtualization, so the blast radius is contained by the VM boundary and the platform never grants a tenant a privileged host-cluster pod. The gateway VM is a normal, dual-homed member of the tenant's pod network. + +There are two legitimate ways to bridge such a tunnel to the tenant's workloads, with genuinely different trade-offs, and both are wanted in practice — so this proposal ships them as **two co-equal catalog apps over a shared foundation**: + +- **`site-router`** — *routed* mode. Plain L3 forwarding between the tunnel and the tenant network; the original client source IP is preserved; whole-subnet reachability and ICMP work. Uses kube-ovn routing (kube-ovn-specific). +- **`site-gateway`** — *NAT* mode. The VM DNAT/SNAT-bridges the tunnel to managed-app `ClusterIP`s; CNI-agnostic and fully contained; per-target host:port reachability; the original source IP is not preserved. + +Neither is a fallback for the other — a cluster admin can offer either or both. The design has been prototyped and validated end-to-end (see [Testing](#testing)). (Naming note: these are distinct from the existing `packages/apps/vpn`, which is a client / remote-access VPN — the two apps here are site-to-site network edges.) + +## Scope and related proposals + +- **[#29 structured external exposure](https://github.com/cozystack/community/pull/29).** The tunnel's own external entry point (the UDP listener the remote peer dials) is published through the structured `ServiceExposure` / `ExposureClass` primitives from #29 rather than a bespoke `LoadBalancer` path; that exposure class must support **UDP** (IKE/NAT-T 4500, WireGuard). +- **[#7 ClusterMesh / Kilo](https://github.com/cozystack/community/pull/7).** Complementary and a different niche — a routed WireGuard node-to-node mesh between cooperative Kubernetes clusters. See [Alternatives considered](#alternatives-considered). +- **Deferred here:** a platform-run (non-VM) managed variant; HTTP/L7 ingress (the existing tenant Ingress / Gateway API covers public HTTP); a per-tenant egress IP (future work — the gateway VM is its natural home later). + +## Context + +Cozystack runs tenant workloads under a hardened networking model: + +- **Namespace hardening.** Cozystack applies Pod Security Standards to tenant namespaces. The platform operator only sets `pod-security.kubernetes.io/enforce=privileged` on a namespace when a platform-declared package requests it. A regular tenant cannot schedule a privileged, `NET_ADMIN`, or `hostNetwork` pod — deliberately, because such a pod executes against the node kernel and could manipulate host networking or observe other tenants' traffic. +- **Default pod network vs. custom VPC.** Managed apps (databases and friends) run as ordinary pods on the **default cluster pod network** and are reached via `ClusterIP` Services + CoreDNS. They are not placed on a custom kube-ovn VPC; VPCs are isolated from the default network and the managed-app operators reconcile over the default network — so a managed app cannot simply be "moved onto a VPC." +- **Dual-homed VMs.** KubeVirt VMs are the exception: every VM always has a `default` pod-network NIC (and can carry additional multus interfaces). A VM is therefore a first-class member of the default pod network. This is the primitive the design leans on. + +### The problem + +A tenant today has no supported way to let a remote site reach a tenant-managed app over a tunnel without exposing it to the public internet (inbound), nor to let a tenant app call a service that is only routable through a tunnel (outbound). The naive approaches all require either a privileged host-cluster pod (a tenant-escape vector — defeats the hardening above) or changes to shared, cluster-wide network infrastructure (cross-tenant collision risk). We want a self-service, tenant-scoped mechanism that generalizes to any protocol and touches neither the host kernel nor the managed apps. + +## Goals + +- Give tenants **self-service** site-to-site connectivity, both directions, through catalog apps. +- Support **inbound** exposure of any tenant-managed app and **outbound** reach to tunnel-only services, for any L4 protocol, without modifying the target apps. +- Keep the privileged dataplane **contained** in a VM guest; add no privileges to the host cluster and none to managed apps. +- Offer **both** a routed mode (source-IP-preserving, L3) and a NAT mode (contained, CNI-agnostic) as **co-equal, admin-gateable** choices — not one as a fallback of the other. +- Support **both IPsec and WireGuard** as tunnel backends (interop vs. simplicity — see [Design](#design)). + +### Non-goals + +- **HTTP/L7 ingress.** Public HTTP(S) is served by the existing tenant Ingress / Gateway API. +- **A platform-managed, always-on VPN service.** These are tenant-deployed catalog apps; a platform-run variant is possible future work. +- **A per-tenant egress IP.** Deferred future work; the gateway VM is its natural home later. +- **Routing a remote CIDR onto the shared default-VPC router** — rejected (cross-tenant collisions). Routed mode uses tenant-scoped kube-ovn routes, never the shared router. + +## Design + +### Two apps over one foundation + +Both apps share roughly 80% of their implementation — the VM appliance, the tunnel backend, the #29 exposure of the tunnel endpoint, the day-2 configuration mechanism, secret handling, and packaging. They differ only in what happens to a decrypted packet (L3-forward vs DNAT/SNAT) and in the CNI coupling that entails: + +| Axis | `site-gateway` (NAT) | `site-router` (routed) | +|------|----------------------|------------------------| +| Inner path | DNAT + SNAT → managed-app `ClusterIP` | plain L3 forwarding, no NAT | +| Return path | SNAT masquerade | kube-ovn `ovn.kubernetes.io/routes` annotation | +| CNI coupling | none — any CNI | kube-ovn-specific (routes + port_security) | +| Source IP | not preserved (SNAT) | preserved | +| Reachability | per host:port | whole subnet + ICMP | +| Config surface | `exposedServices` / `remoteTargets` (per-target ports) | `remoteCIDRs` / static routes / BGP | +| Security posture | fully contained | port_security relaxed (scoped is the target state) | +| Topology-advertising apps | need per-member advertised addresses | transparent | + +Why two apps rather than one `mode:` toggle: the two have divergent values schemas (per-target ports vs CIDRs/BGP), asymmetric controllers (routed needs a privileged CNI-mediation controller; NAT needs none), and different security postures a cluster admin may want to gate independently. Sharing the implementation (a common VyOS-driving controller core + image + base chart) keeps this from becoming duplicated code. + +### Shared foundation + +**Principle.** Terminate the tunnel in a KubeVirt VM (contained privilege). The VM is dual-homed on the tenant's default pod network and is the sole holder of the tunnel. + +```mermaid +flowchart TB + ext["External site
(on-prem router / cloud VPN / another org)"] + xp["#29 tunnel exposure
(ServiceExposure, UDP)"] + subgraph node["Worker node"] + gw["Gateway VM (VyOS)
tunnel termination + firewall + MSS clamp
+ (NAT: DNAT/SNAT) / (routed: L3 forward)"] + a["managed app A"] + b["managed app B"] + end + ext -->|"IPsec ESP-in-UDP / WireGuard UDP"| xp --> gw + gw --> a & b +``` + +**Tunnel backend — IPsec or WireGuard.** VyOS terminates both natively and `tunnel.type` selects per peer. **WireGuard is the simpler default on an overlay CNI** — UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP with no ESP, no NAT-T, and no forced encapsulation; static keypairs, no IKE, smaller header overhead (see the validated finding in [Testing](#testing)). **IPsec is for interop** — much external/enterprise gear only speaks IKEv2/IPsec; supported, with the encapsulation requirement in [Testing](#testing). + +**Tunnel entry point.** Published via #29 `ServiceExposure` + `ExposureClass` (UDP), not a bespoke `LoadBalancer` — one shared external-address contract for both apps. + +**Day-2 configuration.** cloud-init provisions the VM only at first boot (KubeVirt bakes it into a static disk that is regenerated only on VM restart), so ongoing changes — add/rotate a peer, add a target or route — are driven by a **platform controller talking to the guest's own management API**. For VyOS the HTTPS API applies an atomic set/delete batch, so a change touches only the affected config and unrelated tunnels stay up. The management-API key is generated at runtime and seeded once via first-boot cloud-init; the API is reachable by the platform only (see [Security](#security)). A first iteration may fall back to "config change = VM restart" if called out explicitly. + +**Controller & low-level entities.** Tenant input stays in the catalog-app values (no new CRD); validation is the values schema plus controller-side reconcile checks surfaced as status conditions. A platform controller reconciles the VM + tunnel and pushes rendered config to the guest. The controller is split behind a small backend interface so the neutral VyOS-driving core (render → push → observe → status) is shared, and each app implements only its own materialization and any CNI mediation. The two apps differ sharply here: `site-router` carries a CNI-mediation controller (below); `site-gateway` needs none. + +**Image.** A VyOS appliance image (IPsec, WireGuard, native DNAT/SNAT, routing, and firewalling in one network OS), published by the platform as a reproducible, KubeVirt-consumable artifact. + +### `site-gateway` — NAT mode + +**Inbound (external site → any tenant app).** The remote peer targets a virtual service-exposure address that the gateway owns and maps to an app `ClusterIP`: + +1. The remote peer sends tunnel traffic to the gateway's #29-exposed UDP endpoint; the VM decrypts the tunnel. +2. **DNAT**: the virtual destination → the target app's `ClusterIP:port` (one entry per exposed app — the exposure table). +3. **SNAT (masquerade)**: source → the gateway's own pod IP. Mandatory: the cluster has no route back to the remote peer's inner IP, so without SNAT the app's reply leaves via the default route and is black-holed. +4. The VM forwards onto the pod network; the node resolves the `ClusterIP` to a backend pod; the app sees the gateway pod IP as client. Replies retrace the path and are re-encrypted. + +**Outbound (any tenant app → remote service over tunnel).** The app talks to a **local Service** whose endpoint is the gateway VM (the only app-side change is the target hostname). The node routes that `ClusterIP` to the gateway on a **per-target-unique listener port** (the local Service still exposes the standard port, mapped to a unique port on the VM), so the VM can disambiguate multiple remote targets that share a destination port; the VM DNATs that unique port to the real remote `IP:port` and SNATs into the tunnel's inner subnet. + +**Why it generalizes.** The app only ever sees a `ClusterIP` on its own network and speaks no VPN; DNAT is L4 (TCP/UDP, any port), so it works for databases, queues, object storage, gRPC, and raw UDP. `site-gateway` is CNI-agnostic and fully contained — SNAT keeps every emitted packet on the gateway's own pod IP, so no port-security relaxation is needed. + +**Topology-advertising apps.** Protocols where a client bootstraps to one endpoint and is then handed the members' own addresses to connect to directly — Kafka (`advertised.listeners`), MongoDB replica-set/sharded, Redis Cluster, Cassandra/Scylla, Elasticsearch with sniffing — are **not** transparent under plain NAT of the bootstrap endpoint: each member needs its own tunnel-reachable address that the app also advertises. Under `site-gateway` this means explicit per-member mapping (a distinct tunnel-side address:port per member, plus the app configured to advertise it); the managed-app operators already do this kind of thing for public exposure, but #29's `target` is per-named-listener, not per-broker/replica, so #29 does not directly cover per-member addressing today. The transparent path for this class is therefore **`site-router` (routed)** — the members' real addresses are directly reachable, so discovery works as long as members advertise a routable ClusterIP. Folding per-member NAT into a tunnel-backed `ExposureClass` is a possible future unification, not a solved design (see [Open questions](#open-questions)). Single-endpoint protocols (Postgres, MySQL, plain Redis, AMQP, ClickHouse client, HTTP/gRPC) are transparent under NAT and need none of this. + +### `site-router` — routed mode + +The decrypted packet is forwarded onto the tenant network **without NAT**; the return path is the kube-ovn `ovn.kubernetes.io/routes` namespace annotation (inherited onto pods at creation by the kube-ovn webhook), so the **client source IP is preserved** and whole-subnet reachability + ICMP work. This is the mode for cases where the remote side authorizes by source IP, or needs to reach a whole subnet, or where the app advertises its own topology (members' real addresses are directly reachable). + +**CNI-mediation controller.** A tenant cannot annotate its own namespace or touch `port_security`, so the platform controller mediates: validate declared remote CIDRs (disjoint from pod/service/join/node networks; cross-tenant overlap is fine — routes are namespace-scoped), set the namespace routes annotation, scope/relax `port_security` on the gateway VM port, and clean everything up on deletion. Because the annotation is applied at pod **creation**, existing tenant workloads gain reachability only after they are rolled; the controller reports the pods whose annotation lags the namespace ("pods pending route") rather than restarting anything. + +**Security posture.** Routed mode relaxes `port_security` on the gateway port. Containment comes from Cilium's sender-side egress enforcement — the endpoint identity is compiled into the per-endpoint eBPF program, so a spoofed source cannot widen the reachable set beyond the tenant tree + world. The target state is **scoped** `port_security` (the declared remote CIDRs added to the port's allowed addresses). Residual risks are documented in [Security](#security). + +**Transport note.** Routed mode carries transparent L3 for TCP/UDP/ICMP/SCTP only — non-standard IP protocols (ESP, GRE, VRRP) still do not cross the pod fabric (same root cause as the ESP finding in [Testing](#testing)). It is kube-ovn-specific and requires the remote CIDR to be disjoint from cluster networks. + +## User-facing changes + +Two catalog apps appear in the tenant dashboard, sharing the tunnel/peer/resources values and differing only in the bridging block. Sketches (cozyvalues-gen annotated): + +```yaml +# site-gateway (NAT) +## @param tunnel.type {string} Backend: "wireguard" (default; UDP-native) or "ipsec". +tunnel: { type: wireguard } +## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. +peer: + address: "" + auth: { secretRef: "" } # IPsec IKE/PSK-or-cert; WireGuard peer public key + allowedIPs +## @param exposedServices {array} Inbound: {name, listenPort, targetService, port} +exposedServices: [] +## @param remoteTargets {array} Outbound: {name, localPort, remoteHost, remotePort} +remoteTargets: [] +## @param resources {object} VM sizing (cpu/memory). +resources: { cpu: "1", memory: "1Gi" } +``` + +```yaml +# site-router (routed) +## @param tunnel.type {string} Backend: "wireguard" (default) or "ipsec". +tunnel: { type: wireguard } +## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. +peer: + address: "" + auth: { secretRef: "" } +## @param remoteCIDRs {array} Remote networks reachable over the tunnel (must be disjoint from cluster networks). +remoteCIDRs: [] +## @param staticRoutes {array} Optional extra static routes. +staticRoutes: [] +## @param bgp {object} Optional BGP peering with the remote side. +bgp: { enabled: false } +## @param resources {object} VM sizing (cpu/memory). +resources: { cpu: "1", memory: "1Gi" } +``` + +A cluster admin can enable/disable `site-router` independently — its kube-ovn coupling and `port_security` relaxation may not be wanted on every cluster. No existing app, CRD, or API changes. + +## Upgrade and rollback compatibility + +Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk`, the generated Services, and — for `site-router` — the namespace annotation / port_security scoping the controller cleans up), which removes the gateway entirely; nothing else in the tenant is touched. + +## Security + +- **Contained privilege (both apps).** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise inside the gateway cannot manipulate the host node's networking or observe other tenants, and the platform never hands a tenant a privileged host-cluster pod. +- **`site-gateway` is fully contained.** SNAT keeps every packet the gateway emits on its own pod IP, so the CNI's port-security never sees a foreign source and needs no relaxation. +- **`site-router` relaxes port_security** on the gateway port (scoped is the target state). Containment is Cilium's sender-side egress enforcement; residual risks to document: intra-namespace identity spoofing (receiver-side identity resolved from source IP) and spoofed-source egress where SNAT is absent. +- **Managed apps untouched**; a firewall allow-list on the gateway makes exposure explicit; all state is tenant-scoped. +- **Secret handling.** A tenant does not create `Secret` objects directly; as with the existing managed apps (`apps/vpn`, `postgres`, `mariadb`) the app chart templates the `Secret` into the tenant namespace from the app values — autogenerating material when it is not supplied and using `lookup` to keep it stable across reconciles. The design minimizes what must be shipped: for **WireGuard** the tenant's private key is generated in-chart (never entered) and only its public key is surfaced for the remote side, while the peer's public key + endpoint are ordinary non-secret values — so no tenant secret is shipped at all. The one genuinely secret tenant input is the **IPsec PSK** (it must match the remote peer, so it cannot be autogenerated); it rides a values field into the chart-templated `Secret`, with the same plaintext-in-values-at-rest exposure the platform is already addressing for database passwords. The management-API key is generated at runtime, never entered. +- **Management API is platform-only.** The guest's config API is network-restricted so only the platform controller can reach it. + +## Failure and edge cases + +- **Missing SNAT → inbound reply black-holed (`site-gateway`).** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). +- **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an **MSS clamp by default**, derived from the detected overlay MTU — clamping is the chosen approach rather than leaving MTU to the tenant. +- **Native ESP dropped pod-to-pod (IPsec).** The CNI drops non-TCP/UDP/ICMP/SCTP IP protocols between pods — native ESP (proto 50) included — so ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated). In policy-enforced tenant pods the dropper is Cilium's conntrack ("Unknown L4 protocol"); it is not geneve-specific. WireGuard, being UDP-native, is unaffected. +- **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs (`site-gateway`). +- **Source IP is lost inbound under NAT** (`site-gateway`) — use `site-router` when the remote side needs it. +- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override (e.g. `blockSize.custom.logical: 512` / `physical: 4096` on the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). +- **A single gateway is a per-tenant SPOF** until HA is added (see [High availability](#high-availability)). + +## Testing + +The design was **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target. The **NAT (`site-gateway`) inbound/outbound paths** were validated with the IPsec backend; the **routed (`site-router`) path and the WireGuard backend** are the next to validate (open questions). All items passed: + +| # | Item | Result | +|---|------|--------| +| 1 | IKEv2 SA establishes both sides | PASS | +| 2 | Inbound: external site → virtual VIP → DNAT/SNAT → managed Postgres (real server response) | PASS | +| 3 | Outbound: gateway client → tunnel → remote listener | PASS | +| 4 | SNAT-required negative test (removing SNAT black-holes the reply) | PASS | +| 5 | MTU / MSS clamp (tunnel MTU 1320, clamp 1280; multi-MB transfer, no stall) | PASS | + +**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. On an overlay CNI the IPsec backend must therefore always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. + +**Non-standard-protocol drop (informs HA).** Independently re-verified with a minimal repro: the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols (proto 112/50/47/4) pod-to-pod, both same-node and cross-node — so it is **not** the geneve tunnel. In policy-enforced tenant pods the dropper is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"); without a policy it is the OVS datapath. This is the same root cause as the native-ESP drop, and it is why keepalived/VRRP cannot elect over the pod network (see [High availability](#high-availability)). Separately, a kube-ovn `Vip` + `ovn.kubernetes.io/aaps` annotation was validated to share a VIP across two gateway pods with `port_security` kept **on** (the VIP moves on owner change and a bogus source is still dropped — anti-spoofing scoped, not disabled). + +Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends and both apps; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows plus the SNAT-required and MSS behaviors. + +## High availability + +Deferred for the first iteration: a **single gateway VM with KubeVirt live-migration** for planned maintenance (conntrack, tunnel SA, and pod IP preserved — no tunnel drop; requires migratable/replicated storage, see the `blockSize` caveat). Automatic **unplanned-failure** HA is a follow-up; the preferred path is **Service-fronted active/passive** because it is CNI-agnostic (no floating L2 VIP, so no port-security interaction) at the cost of a small leader-election/lease agent and seconds-scale endpoint reconvergence. A kube-ovn shared-VIP via allowed-address-pairs was validated to move a VIP with `port_security` kept on, but keepalived-style election cannot run over the pod network (VRRP/proto-112 is dropped — see [Testing](#testing)), so it would need a side-channel or controller to drive the VIP claim. + +## Rollout + +- **Phase 1 — `site-router` (routed), IPsec backend**: the routed dataplane plus its CNI-mediation controller (CIDR validation, routes annotation, scoped `port_security`), with **MSS clamping** and **tunnel-state observability** (SA up/down, rekey, counters) surfaced from the start. Routed is the primary mode tenants ask for. (Note: the end-to-end prototype so far validated the NAT dataplane, so routed carries the first round of validation here.) +- **Phase 2 — `site-gateway` (NAT)**: the inbound/outbound DNAT/SNAT machinery (prototype-validated) with forced UDP encapsulation. +- **Phase 3 — WireGuard backend**: validate over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. +- **Phase 4 — the rest**: HA (Service-fronted active/passive), scoped `port_security` for routed, per-tenant egress IP. + +Ordering leads with routed because it is the primary requested mode; the shared foundation is built once and both apps sit on it. + +## Open questions + +- **IPsec PSK at rest.** WireGuard ships no tenant secret (its key is autogenerated in-chart); the IPsec PSK is the one tenant-supplied secret — decide whether it stays a values field templated into a `Secret` (like DB passwords today — plaintext in values at rest) or gets a write-only / dashboard-mediated affordance. +- **Tunnel exposure via #29.** Confirm the `ExposureClass` UDP support and how the tunnel endpoint address interacts with the tenant's LB pool and quotas. +- **Topology-advertising apps under NAT.** Whether per-member advertised addressing for Kafka / Mongo / Redis-Cluster / Cassandra under `site-gateway` is worth a tunnel-backed `ExposureClass` (needs per-member granularity #29 does not have today) or is simply left to `site-router`. +- **HA mechanism.** Given VRRP does not cross the overlay (see Testing), standardize on Service/endpoint-based failover (needs a controller/lease) vs an AAP shared-VIP with side-channel election. +- **Scoped port_security for `site-router`.** kube-ovn allowed-address-pairs need CIDR support to add the declared remote CIDRs to the port (OVN itself accepts `MAC IP/mask`). +- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical bridging before defaulting to it. +- **Relationship to ClusterMesh (#7).** Whether to present these apps and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits. + +## Alternatives considered + +- **Give tenants privileged / `NET_ADMIN` host pods.** Rejected — tenant escape; defeats namespace hardening. The whole design exists to avoid this. +- **Route the remote CIDR into the shared default-VPC router.** Rejected — the router is shared cluster-wide; injecting tenant RFC1918 CIDRs causes cross-tenant collisions and leakage. (`site-router` instead uses tenant-scoped kube-ovn namespace routes.) +- **Put managed apps on a custom kube-ovn VPC.** Not viable — VPCs are isolated from the default network and managed-app operators reconcile over the default network; only VMs are dual-homed. +- **Platform-run kube-ovn VPC-NAT-Gateway with the tunnel.** Possible future managed variant, but more platform work and it couples XFRM to the host kernel + OVS. The VM approach contains the privilege more cleanly and ships sooner. +- **A single app with a `mode: routed|nat` toggle.** Rejected — the two modes have divergent values schemas, asymmetric controllers (routed needs a privileged CNI-mediation controller, NAT needs none), and different security postures a cluster admin may want to gate independently. Two apps over a shared implementation express this more cleanly than conditional schema + always-on machinery. +- **Traefik / L7 proxy.** Limited — HTTP-only; not general across L4 protocols. + +### ClusterMesh / Kilo (#7), and why two mechanisms + +[#7](https://github.com/cozystack/community/pull/7) pursues an overlapping goal (tenant access to services across a boundary) but is a different shape, and neither mechanism subsumes the other: + +| Axis | ClusterMesh / Kilo (#7) | This proposal (gateway VMs) | +|------|-------------------------|-----------------------------| +| Remote end | A cooperative Kubernetes cluster running Kilo, reached via a kubeconfig | An arbitrary external site — not Kubernetes, no Kilo, no kubeconfig | +| Tenant workload served | The tenant's managed Kubernetes cluster (guest-VM pods) | Managed apps in the host-cluster tenant namespace | +| Transport | WireGuard node-to-node mesh (`mesh-granularity=cross`) | IPsec or WireGuard site-to-site, one VM | +| Address plane | Routed pod-CIDRs, no NAT; disjoint CIDRs required | `site-router`: routed (source-IP preserved); `site-gateway`: NAT to ClusterIPs (tolerates overlap) | +| Throughput | Full node×node mesh (built for Ceph) | Single gateway — app-level flows | +| Privilege | Kilo `NET_ADMIN` inside guest-cluster VMs → contained | `NET_ADMIN` inside the gateway VM guest → contained | + +Kilo cannot terminate a third-party VPN (both ends must run Kilo with a kubeconfig), and a single gateway VM cannot serve the Ceph-scale mesh. They are different layers — a platform storage fabric vs. a tenant-edge VPN concentrator — analogous to a cloud provider shipping both VPC Peering and a VPN Gateway. Both share the principle that matters: no privileged pods in the shared host namespace; privilege is contained in guest-cluster VMs (Kilo) or a dedicated gateway VM (this proposal). diff --git a/design-proposals/tenant-site-gateway/README.md b/design-proposals/tenant-site-gateway/README.md deleted file mode 100644 index 85878c5..0000000 --- a/design-proposals/tenant-site-gateway/README.md +++ /dev/null @@ -1,238 +0,0 @@ -# Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard) - -- **Title:** `Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard)` -- **Author(s):** `@myasnikovdaniil` -- **Date:** `2026-07-07` -- **Status:** Draft - -## Overview - -Cozystack tenants need site-to-site VPN connectivity between their workloads and external networks, in both directions: exposing a tenant-managed application to a remote site (inbound), and letting a tenant-managed application reach a service that lives only behind a tunnel (outbound). This must work for any application — databases, message queues, object storage, gRPC, raw UDP — not one special case. - -This proposal terminates the tunnel (IPsec or WireGuard) **inside a KubeVirt VM** rather than in a pod. The privileged dataplane (`NET_ADMIN`, XFRM / a `wireguard` interface, `ip_forward`) lives in a guest kernel isolated by hardware virtualization, so its blast radius is contained by the VM boundary — the host node and other tenants are unaffected. The gateway VM is a normal member of the tenant's pod network, so it reaches any tenant `ClusterIP` Service; it performs all address translation (DNAT/SNAT), and the managed applications are never modified and never speak the VPN — they only ever see ordinary `ClusterIP`s on their own network. We propose shipping this as a Cozystack catalog app, provisionally `packages/apps/site-gateway`. - -The design has been prototyped and validated end-to-end (see [Testing](#testing)). - -## Scope and related proposals - -- **Related — [`cross-cluster-tenant-mesh` (ClusterMesh / Kilo)](https://github.com/cozystack/community/pull/7).** That proposal connects cooperative Kubernetes clusters into a routed WireGuard node-to-node mesh for high-throughput cases (its motivating example is a tenant cluster consuming host Ceph). This proposal is complementary and occupies a different niche — an arbitrary external site (not a Kubernetes cluster), NAT-bridged to managed apps that live in the tenant namespace. See [Alternatives considered](#alternatives-considered) for a side-by-side and a "why two mechanisms" discussion. Notably, PR #7 explicitly defers the "tenant ↔ tenant-namespace applications" and "NAT-egress" integration — precisely the space this proposal fills. -- **Deferred here:** a platform-run (non-VM) managed variant; HTTP/L7 ingress (the existing tenant Ingress / Gateway API already covers public HTTP); preservation of the original client source IP inbound. - -## Context - -Cozystack runs tenant workloads under a hardened networking model: - -- **Namespace hardening.** Cozystack applies Pod Security Standards to tenant namespaces. The platform operator only sets `pod-security.kubernetes.io/enforce=privileged` on a namespace when a platform-declared package requests it. A regular tenant cannot schedule a privileged, `NET_ADMIN`, or `hostNetwork` pod — deliberately, because such a pod executes against the node kernel and could manipulate host networking or observe other tenants' traffic. -- **Default pod network vs. custom VPC.** Managed apps (databases and friends) run as ordinary pods on the **default cluster pod network** and are reached via `ClusterIP` Services + CoreDNS. They are not placed on a custom kube-ovn VPC; VPCs are isolated from the default network and the managed-app operators reconcile over the default network — so a managed app cannot simply be "moved onto a VPC." -- **Dual-homed VMs.** KubeVirt VMs are the exception: every VM always has a `default` pod-network NIC (and can carry additional multus interfaces). A VM is therefore a first-class member of the default pod network. This is the primitive the design leans on. - -### The problem - -A tenant today has no supported way to: - -- **Inbound:** let a remote site reach a tenant-managed app (a Postgres instance, an internal API, an object store) over a tunnel, without exposing it to the public internet. -- **Outbound:** let a tenant app call a service (a legacy database, a licensing server, an on-prem API) that is only routable through a tunnel. - -The naive approaches all require either a privileged host-cluster pod (a tenant-escape vector — defeats the hardening above) or changes to shared, cluster-wide network infrastructure (cross-tenant collision risk). We want a self-service, tenant-scoped mechanism that generalizes to any protocol and touches neither the host kernel nor the managed apps. - -## Goals - -- Give tenants **self-service** site-to-site connectivity, both directions, through a catalog app. -- Support **inbound** exposure of any tenant-managed app to a remote site over the tunnel. -- Support **outbound** connections from any tenant app to services reachable only via the tunnel. -- Be **app-agnostic** — work for any L4 protocol (TCP/UDP, any port) without modifying the target apps. -- Keep the privileged dataplane **contained** in a VM guest; add no privileges to the host cluster and none to managed apps. -- Keep all bridging/NAT **tenant-scoped**; make no changes to shared cluster routers. -- Support **both IPsec and WireGuard** as tunnel backends (interop vs. simplicity — see [Design](#design)). - -### Non-goals - -- **HTTP/L7 ingress.** Public HTTP(S) is served by the existing tenant Ingress / Gateway API; the gateway VM is for site-to-site VPN and non-HTTP protocols. -- **Preserving the original client source IP inbound.** SNAT hides it (see [Failure and edge cases](#failure-and-edge-cases)); source-IP ACLs are out of scope for the first iteration. -- **A platform-managed, always-on VPN service.** This is a tenant-deployed catalog app; a platform-run variant is possible future work. -- **Routing a remote CIDR onto the shared default-VPC router** — explicitly rejected (cross-tenant collisions). - -## Design - -### Principle - -Terminate the tunnel in a KubeVirt VM. The VM guest kernel isolates `NET_ADMIN`/XFRM from the host, so the tenant effectively runs a privileged network appliance without the platform ever granting a privileged pod. The VM is dual-homed on the tenant's default pod network and is the sole holder of the tunnel; it DNAT/SNATs between the tunnel and the tenants' managed-app `ClusterIP`s. - -### Topology - -```mermaid -flowchart TB - ext["External site
(on-prem router / cloud VPN / another org)"] - lb["LoadBalancer VIP
(tunnel UDP ports)"] - subgraph node["Worker node"] - gw["Gateway VM (VyOS)
tunnel termination + DNAT/SNAT + firewall + MSS clamp"] - a["managed app A
(ClusterIP)"] - b["managed app B
(ClusterIP)"] - c["managed app C
(ClusterIP)"] - end - ext -->|"IPsec ESP-in-UDP / WireGuard UDP"| lb --> gw - gw --> a & b & c -``` - -Everything privileged lives inside the VM guest. Managed apps only ever see `ClusterIP`s on their own default network. - -### Inbound path (external site → any tenant app) - -The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. - -1. Remote peer sends tunnel traffic to the gateway's public VIP (a `LoadBalancer` Service on the tunnel's UDP ports). -2. The VIP delivers to the gateway VM's pod NIC; the VM decrypts the tunnel. -3. **DNAT**: the virtual destination (e.g. `10.200.0.10:5432`) → the target app's `ClusterIP:port`. One entry per exposed app — the exposure table. -4. **SNAT (masquerade)**: source → the gateway's own pod IP. Mandatory: the cluster has no route back to the remote peer's inner IP, so without SNAT the app's reply would leave via the default route and be black-holed. -5. The VM forwards onto the pod network; the node resolves the `ClusterIP` to a backend pod; the app sees the connection with the gateway pod IP as client. Replies retrace the path and are re-encrypted back over the tunnel. - -### Outbound path (any tenant app → remote service over tunnel) - -The tenant app never routes to the remote address directly; it talks to a **local Service** whose endpoint is the gateway VM. - -1. The app connects to a local Service name the chart creates (e.g. `remote-db..svc:port`) — the only app-side change is the target hostname. -2. The node routes the `ClusterIP` to the gateway VM on a **per-target-unique listener port** (the local Service still exposes the standard port, e.g. 5432, mapped to a unique port on the VM). The uniqueness is required so the VM can tell apart multiple remote targets that share the same destination port. -3. **DNAT**: that unique local listener port → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. - -**Rejected alternative — a static route on the shared default-VPC router.** Injecting a tenant's remote RFC1918 CIDR into the cluster-wide default router risks cross-tenant address collisions and leakage. The local-Service approach keeps every remote address strictly inside the tenant's gateway VM. - -### Why it generalizes to all apps - -- The app only ever sees a `ClusterIP` / Service on its own network; it speaks no VPN and needs no route to the far side. -- DNAT is **L4** (TCP and UDP, any port), not L7 — so it works for databases, queues, object storage, gRPC, and raw UDP, unlike an HTTP-only path. -- The gateway VM is the only privileged component, and it is a contained VM guest. - -### Tunnel backend: IPsec or WireGuard - -VyOS terminates both natively and the DNAT/SNAT bridging is identical — only the transport differs. `tunnel.type` selects the backend per peer. - -- **WireGuard — the simpler default on an overlay CNI.** WireGuard is UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP: no ESP, no NAT-T, no forced encapsulation. It is simpler (static keypairs, no IKE) and has smaller header overhead. Preferred for tenant↔tenant and tenant↔Cozystack links. (See the validated finding in [Testing](#testing) that makes this concrete.) -- **IPsec — for interop.** Much external/enterprise gear only speaks IKEv2/IPsec; when the remote site is not under the tenant's control this is often the only option. Supported, with the encapsulation requirement described in [Testing](#testing). - -### Build options - -- **VyOS appliance (recommended).** A single network OS does IPsec, WireGuard, native DNAT/SNAT, and firewalling; no separate proxy needed; configured declaratively via cloud-init. -- **Alpine + libreswan/strongSwan (or wireguard-tools) + haproxy.** A leaner, self-composed image if a network-OS dependency is undesirable; more moving parts. - -### High availability - -HA splits by failure mode: - -- **Planned maintenance (node drain):** KubeVirt **live-migration** relocates the gateway VM with its state intact (conntrack, tunnel SA, pod IP preserved) — no tunnel drop. Preferred for maintenance; needs no standby. (Requires migratable storage — see the storage caveat below.) -- **Unplanned node/VM failure:** needs a standby plus a failover trigger. Two mechanisms, both examined on kube-ovn (see [Testing](#testing) for the shared-VIP validation): - - 1. **Service-fronted active/passive.** The external `LoadBalancer` (tunnel) and internal `ClusterIP` (outbound) Services select whichever gateway pod is active; failover = re-pointing endpoints when the active goes unready. The stable address is the Service VIP (CNI-managed) — no floating L2 VIP, so no port-security interaction at all. Cost: needs a small leader-election/lease agent (or controller) to flip the active endpoint, and failover takes seconds (endpoint reconvergence). CNI-agnostic. - 2. **Shared VIP via kube-ovn allowed-address-pairs (AAP).** A kube-ovn `Vip` plus the `ovn.kubernetes.io/aaps` pod annotation adds *only* the VIP to the port's OVN `port_security` (which stays enforced) — validated: the VIP is reachable, moves between the two gateway pods through OVN on owner change, and any other source address is still dropped, so anti-spoofing is **scoped rather than disabled** and the VM stays fully tenant-controlled (no need to withhold direct VM/cloud-init access). **Caveat:** VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived split-brains and cannot elect a master over the pod network. Root cause (independently verified): the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols pod-to-pod — in policy-enforced tenant pods this is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"), and without a policy it is the OVS datapath; it is **not** geneve/inter-node-specific (same-node pods drop it too). This is the same root cause that makes native ESP need UDP encapsulation. To use this path the election must therefore run over a side-channel (e.g. the gateway pair's own tunnel link) or be driven by a controller; the AAP VIP move itself is proven. Also the `aaps` annotation must be baked into the KubeVirt VM's pod template (the current `VMInstance` chart does not expose it), and the VIP rides the pod's real MAC (keep the VyOS default — no virtual-MAC). - -**Recommendation:** iteration 1 ships a single gateway VM with live-migration for maintenance, and treats automatic unplanned-failure HA as a follow-up — neither mechanism is a clean drop-in today (Service-based needs a controller; AAP needs the VRRP-advert-delivery issue resolved). - -**Storage caveat:** live-migration and single-VM reschedule need migratable/replicated storage — but replicated (DRBD) StorageClasses expose 4K sectors, so the image needs the `blockSize` override to boot (see [Failure and edge cases](#failure-and-edge-cases)). A two-VM active/passive pair can instead use node-local 512-native disks. - -## User-facing changes - -A new catalog app, provisionally **`packages/apps/site-gateway`** (VM-backed), appears in the tenant dashboard. The tenant fills a values form; the platform-authored chart renders the gateway VM, its tunnel + NAT + firewall config, the local Services for outbound targets, and a `LoadBalancer` Service for the tunnel's UDP ports. Sketch of the values schema (cozyvalues-gen annotated): - -```yaml -## @param tunnel.type {string} Backend: "wireguard" (default; UDP-native) or "ipsec". -tunnel: - type: wireguard - -## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. -peer: - address: "" - ## backend-specific auth: IPsec IKE/PSK-or-cert; WireGuard peer public key + allowedIPs. - auth: - secretRef: "" - -## @param exposedServices {array} Inbound: {name, listenPort, targetService, port} -exposedServices: [] - -## @param remoteTargets {array} Outbound: {name, localPort, remoteHost, remotePort} -remoteTargets: [] - -## @param resources {object} VM sizing (cpu/memory). -resources: - cpu: "1" - memory: "1Gi" -``` - -No existing app, CRD, or API changes. - -## Upgrade and rollback compatibility - -Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk` and the generated Services), which removes the gateway entirely; nothing else in the tenant is touched. - -## Security - -- **Contained privilege.** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise or misconfiguration inside the gateway cannot manipulate the host node's networking or observe other tenants. The platform never hands a tenant a privileged host-cluster pod. -- **Managed apps untouched.** No new privileges or config on the apps; they stay ordinary pods on the default network. -- **Tenant-scoped bridging.** All translation is NAT inside the VM; nothing changes on the shared cluster router, so there is no cross-tenant surface. -- **Firewall allow-list.** The gateway restricts which tunnel-side sources may reach which exposed ports — exposure is explicit, not "the whole tunnel reaches everything." -- **Tenant-supplied secrets.** PSK / certificate / WireGuard key material is provided via a Secret reference and mounted into the guest; the chart must avoid persisting it in plaintext cloud-init user-data at rest (open question). - -## Failure and edge cases - -- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). It also keeps the gateway **anti-spoofing-clean**: every packet it emits onto the pod network carries its own pod IP, so the CNI's port-security never sees a foreign source — the dataplane needs no port-security relaxation (that only arises for shared-VIP HA; see High availability). -- **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an MSS clamp by default. -- **Native ESP dropped pod-to-pod (IPsec).** The CNI drops non-TCP/UDP/ICMP/SCTP IP protocols between pods — native ESP (proto 50) included — so ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated). In policy-enforced tenant pods the dropper is Cilium's conntrack ("Unknown L4 protocol"); it is not geneve-specific (same root cause as the VRRP/proto-112 case — see Testing). WireGuard, being UDP-native, is unaffected. -- **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. -- **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. -- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override (e.g. `blockSize.custom.logical: 512` / `physical: 4096` on the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). -- **Single gateway is a per-tenant SPOF** until HA is added (open question). - -## Testing - -The design was **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target. IPsec was validated first; WireGuard is the next backend to validate (open question). All items passed: - -| # | Item | Result | -|---|------|--------| -| 1 | IKEv2 SA establishes both sides | PASS | -| 2 | Inbound: external site → virtual VIP → DNAT/SNAT → managed Postgres (real server response) | PASS | -| 3 | Outbound: gateway client → tunnel → remote listener | PASS | -| 4 | SNAT-required negative test (removing SNAT black-holes the reply) | PASS | -| 5 | MTU / MSS clamp (working tunnel MTU 1320, clamp 1280; multi-MB transfer, no stall) | PASS | - -**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. Consequence: on an overlay CNI the IPsec backend must always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. - -**HA mechanism validation (kube-ovn AAP).** Separately validated (kube-ovn v1.15.10): a `Vip` + `ovn.kubernetes.io/aaps` pod annotation shares a VIP across the two gateway VM pods with `port_security` kept **on**. Proven: (1) without AAP the VIP is dropped (GARP/ping blocked); (2) with AAP the VIP is reachable and lands on the active pod; (3) forcing the active down moves the VIP to the standby through OVN; (4) a bogus source address from the VM is still dropped — anti-spoofing is scoped, not disabled. **Not** working: VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived cannot elect over the pod network — the election needs a side-channel or controller (see [High availability](#high-availability)). Independently re-verified with a minimal repro: the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols (112, 50, 47, 4) pod-to-pod, both same-node and cross-node — so this is **not** the geneve tunnel. In policy-enforced tenant pods the dropper is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"); without a policy it is the OVS datapath. This is the same root cause as the native-ESP drop above. An upstream issue, if pursued, targets Cilium (a known conntrack limitation); the OVS-side drop is not yet root-caused. - -Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows and the SNAT-required and MSS behaviors. - -## Rollout - -- **Phase 1 — IPsec backend** (prototype validated): ship `site-gateway` with the IPsec backend (forced UDP encapsulation) + the inbound/outbound NAT machinery. -- **Phase 2 — WireGuard backend**: validate WireGuard over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. -- **Phase 3 — hardening**: HA (see [High availability](#high-availability) — Service-based or AAP shared-VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. - -## Open questions - -- **Secret handling.** Deliver PSK/cert/WireGuard-key material to the guest without persisting it in plaintext cloud-init at rest — candidate: mount the referenced `Secret` as a separate config disk (or fetch it on first boot via a small init) instead of inlining it in the VM spec; the post-reconciliation read-access boundary needs to be defined. -- **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. -- **HA mechanism.** Given VRRP advertisements do not cross the kube-ovn overlay (see Testing), which failover trigger to standardize on — Service/endpoint-based (needs a controller/lease) or AAP shared-VIP with election over a side-channel — and baking the `aaps` annotation into the `VMInstance` chart's pod template. -- **Observability.** Surfacing tunnel state (SA up/down, rekey, counters) through existing dashboards. -- **MTU auto-tuning.** Derive the MSS clamp / tunnel MTU from the detected overlay MTU, or require an explicit value. -- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical DNAT/SNAT bridging before defaulting to it. -- **Relationship to ClusterMesh (PR #7).** Whether to present this and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits. - -## Alternatives considered - -- **Give tenants privileged / `NET_ADMIN` host pods.** Rejected — tenant escape; defeats namespace hardening. The whole design exists to avoid this. -- **Route the remote CIDR into the shared default-VPC router.** Rejected — the router is shared cluster-wide; injecting tenant RFC1918 CIDRs causes cross-tenant collisions and leakage. -- **Put managed apps on a custom kube-ovn VPC.** Not viable — VPCs are isolated from the default network and managed-app operators reconcile over the default network; only VMs are dual-homed. -- **Platform-run kube-ovn VPC-NAT-Gateway with the tunnel.** Possible future managed variant, but more platform work and it couples XFRM to the host kernel + OVS. The VM approach contains the privilege more cleanly and ships sooner. -- **Traefik / L7 proxy.** Limited — HTTP-only; not general across L4 protocols. - -### ClusterMesh / Kilo (PR #7), and why two mechanisms - -[PR #7](https://github.com/cozystack/community/pull/7) pursues an overlapping goal (tenant access to services across a boundary) but is a different shape, and neither mechanism subsumes the other: - -| Axis | ClusterMesh / Kilo (PR #7) | This proposal (gateway VM) | -|------|---------------------------|----------------------------| -| Remote end | A cooperative Kubernetes cluster running Kilo, reached via a kubeconfig | An arbitrary external site — not Kubernetes, no Kilo, no kubeconfig | -| Tenant workload served | The tenant's managed Kubernetes cluster (guest-VM pods) | Managed apps in the host-cluster tenant namespace (ClusterIPs) | -| Transport | WireGuard node-to-node mesh (`mesh-granularity=cross`) | IPsec or WireGuard site-to-site, one VM | -| Address plane | Routed pod-CIDRs, no NAT; disjoint CIDRs required | NAT (DNAT/SNAT) to ClusterIPs; tolerates overlap | -| Throughput | Full node×node mesh — direct pod-IP to many backends (built for Ceph) | Single gateway — app-level flows, not Ceph-scale | -| Privilege | Kilo node `NET_ADMIN`, on the tenant side inside guest-cluster VMs → contained | `NET_ADMIN` inside the gateway VM guest → contained | - -**Kilo cannot terminate a third-party VPN** (both ends must run Kilo with a kubeconfig), and **a single gateway VM cannot serve the Ceph-scale mesh** (throughput/topology incompatible with funneling). They are different layers: a platform storage fabric vs. a tenant-edge VPN concentrator — analogous to a cloud provider shipping both VPC Peering and a VPN Gateway. Both share the principle that matters here: no privileged pods in the shared host namespace; privilege is contained in guest-cluster VMs (Kilo) or a dedicated gateway VM (this proposal). Recommended framing: lead with the gateway VM as the single tenant-facing primitive (covers external sites, tenant↔tenant, and managed-app exposure), and use ClusterMesh only for the high-throughput cross-cluster storage case.