From ea052aa8d8f2164400dc76189b9c1b29e8a0c4f1 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Mon, 13 Jul 2026 12:50:13 +0200 Subject: [PATCH 1/2] docs(design-proposals): deliver ComputePlane as an extra module wrapping apps/kubernetes Revise the ComputePlane proposal: deliver it as an operator-owned Tenant module (extra/computeplane) that wraps the existing apps/kubernetes chart with fixed values, instead of user-facing preset fields on kind: Kubernetes. Keeps the "not a new kind" conclusion from the #17 review and #26; supersedes the preset-field revision (#27). The underlying isolation mechanism (remote Flux apply, per-VM boundary, single-tenant) is unchanged. Assisted-By: Claude Signed-off-by: Andrei Kvapil --- design-proposals/compute-plane/README.md | 276 +++++++++++------------ 1 file changed, 134 insertions(+), 142 deletions(-) diff --git a/design-proposals/compute-plane/README.md b/design-proposals/compute-plane/README.md index b7d86d6..afb57a5 100644 --- a/design-proposals/compute-plane/README.md +++ b/design-proposals/compute-plane/README.md @@ -5,6 +5,7 @@ - **Author(s):** `@kvaps` - **Date:** `2026-06-23` - **Status:** Draft +- **Revision (this PR):** Supersedes the preset-field revision (#27). ComputePlane is **not** a new top-level kind (agreeing with #26 / the #17 review) **and not** a set of user-facing preset fields on `kind: Kubernetes`. Instead it is a Cozystack-owned **Tenant module** (`packages/extra/computeplane`) that, under the hood, deploys the ordinary `apps/kubernetes` chart with operator-fixed values, sourced through the existing PackageSource "source-only chart" mechanism (the same one NATS and SeaweedFS use — only the wrapped chart comes from `apps/` instead of `system/`). The tenant gets the *same* `kind: Kubernetes` cluster, but owns none of its settings — only the knobs the operator chooses to expose. The mechanism settled in the first revision — remote Flux apply onto a Kamaji+KubeVirt cluster, untrusted code behind a per-VM kernel boundary — is unchanged; this revision is only about the **delivery surface**. ## Overview @@ -12,88 +13,111 @@ Cozystack's tenant model treats a managed application as a single-purpose servic A growing class of applications breaks that property by design: their core feature *is* arbitrary code execution (notebooks, workflow "code" nodes, plugin systems, custom Python components). An operator who wants to *offer* these from the catalog has, today, only one delivery path — deploy them as ordinary pods into the tenant namespace **on the shared management cluster**, where they run on the management nodes' shared host kernel. For code-executing apps that is the unsafe part: a kernel-level container escape — the recurring vulnerability class, e.g. Copy Fail (CVE-2026-31431) — turns the app's untrusted user code into root on a management node and across every co-located tenant. Cloud platforms answer this by running untrusted compute behind a virtualization boundary (the reason Kata, gVisor and sandboxed runtimes exist); Cozystack already runs *tenant-owned* compute that way, on KubeVirt-VM clusters. ComputePlane brings catalog apps onto that same boundary instead of onto the shared management nodes. -This proposal introduces a **ComputePlane**: a Cozystack-managed Kubernetes cluster onto which untrusted-code workloads are placed instead of into the tenant namespace on the management cluster. The name parallels "control plane": the management cluster is the *control plane*; the **ComputePlane** is the separate, Cozystack-managed cluster where (untrusted) workloads actually run. +**ComputePlane is neither a new kind nor a new setting on `kind: Kubernetes`.** From the management plane's view, the cluster a ComputePlane runs on and a regular managed `kind: Kubernetes` are *the same object* — Kamaji control plane, KubeVirt-VM workers whose `virt-launcher` pods sit in the tenant namespace under the existing egress policy, operator-held kubeconfig. So ComputePlane needs no CRD and no new API surface. It is a **packaging** of that existing app: a per-tenant enabler module the operator switches on, which provisions one hardened `kind: Kubernetes` cluster with values the operator controls. + +Two consequences follow, and this is the whole point of the revision: + +- **The user does not order a ComputePlane, and cannot edit it.** Following Cozystack's `apps` vs `extra` split (see [website#594](https://github.com/cozystack/website/pull/594)): `apps` are first-class services a user orders directly (Postgres, Kubernetes, an S3 bucket); `extra` are per-tenant enabler modules that power those services under the hood (`extra/etcd` → `apps/kubernetes`, `extra/seaweedfs` → `apps/bucket`). A ComputePlane is exactly the latter shape — an enabler the tenant switches on, not a cluster they administer. So it belongs in `extra`, and the tenant cannot reach into its configuration. +- **The hardening is tamper-proof by construction.** Because the cluster's values live entirely in an operator-owned chart and the tenant holds no admin kubeconfig, there is no field for the tenant to weaken. "Withhold admin" and "own the config" are the same fact, not two features to reconcile. ComputePlane rests on two principles, and neither is a new isolation boundary it invents: - **Separation of responsibility.** The compute cluster stays under the operator's control: the tenant never touches the management control plane and cannot block platform updates, while the operator's management never breaks the tenant's deployed workloads. This clean split — operator owns the substrate and the hardening, tenant owns their app — is what makes this a *managed service* rather than "provision your own cluster and install it yourself." - **Defense in depth.** Untrusted code runs behind a KubeVirt-VM boundary with its own guest kernel, so a kernel-level escape like Copy Fail is contained to a disposable VM rather than the shared host kernel of a management node. -Both properties come from the managed-`kubernetes` substrate — from the management plane's view a ComputePlane and a regular managed cluster are the same object (Kamaji control plane, KubeVirt-VM workers whose `virt-launcher` pods sit in the tenant namespace under the existing egress policy, operator-held kubeconfig). ComputePlane's own contribution is **packaging**: placement routing that puts a catalog app on that cluster automatically, platform-owned safe defaults the tenant does not assemble, and an operator-retained lifecycle — delivered through the unchanged create-an-app UX, with access proxied back through the normal ingress entry point. (Whether the tenant can *see* the cluster is a separate UX default, not part of the boundary — see [Security](#security).) - The capability is generic and intended to live in Cozystack core as a reusable primitive — not as an LLM-specific feature. Any catalog of code-executing applications (the immediate driver is `cozyllm`; WordPress-with-plugins and future application-platform offerings are the same shape) consumes it through a pluggable interface. ## Scope and related proposals -- **`design-proposals/tenant-module-overrides`** (PR #4): the ComputePlane is delivered as a **Tenant module**, the same mechanism that delivers `etcd` / `monitoring` / `ingress` / `seaweedfs`. Note a deliberate divergence: existing modules today are single scalar toggles in `values.yaml`, and the ComputePlane module follows that scalar shape (see Design §2) rather than introducing a per-module `{ enabled, valuesOverride }` blob. The profile it references is defined once at the tenant/platform level — that single-source-of-truth choice is the resolution to the "two sources of truth" concern that the tenant-module-overrides proposal grapples with, not a dependency on it. -- **`design-proposals/cross-cluster-tenant-mesh`** (PR #7): establishes the trust model for Cozystack-managed clusters — one-way (host → tenant), tenants have no host-cluster API access. The ComputePlane reuses that exact directionality for the **kube-API plane**. It diverges on the **data plane**: rather than a node-to-node mesh, ComputePlane→tenant-service reachability is a narrow, per-service `CiliumNetworkPolicy` (see Design §5). The path already exists — ComputePlane nodes sit on the management Cilium network — so no mesh is required. -- **`design-proposals/kubernetes-nodes-split`** / **`kubernetes-nodes-hybrid-clusters`** (PR #8/#9): the ComputePlane is built on the existing managed-`kubernetes` app (Kamaji control plane + CAPI/KubeVirt nodes). Changes to how `kubernetes` worker nodes are provisioned apply transparently here. -- **Deferred:** per-application ComputePlanes; billing/metering of ComputePlane resource and API consumption; a tenant-facing scoped read/observability view of the ComputePlane; secret delivery of managed-service connection strings into ComputePlane workloads. (Cross-tenant *sharing* of a ComputePlane is **not** deferred — it is rejected by design; see Non-goals.) Each is called out under Open questions. +- **#26 (@myasnikovdaniil)** and the **#17 review (Timofei Larkin)** established that ComputePlane must not be a distinct kind — the cluster it runs on is the same object as a managed `kind: Kubernetes`. This revision keeps that conclusion but delivers the posture as an **operator-owned module wrapping that app**, rather than as user-facing preset fields on the app (see [Alternatives](#alternatives-considered) for why the field model was set aside). +- **[website#594](https://github.com/cozystack/website/pull/594)** (`apps` vs `extra`): the split this design leans on — `apps` = first-class services the user orders; `extra` = per-tenant enabler modules under the hood. ComputePlane is `extra`. +- **`design-proposals/cross-cluster-tenant-mesh`** (PR #7): the trust model for managed clusters (one-way host → tenant, no host kube-API). A *trusted* variant of the module could wire the cluster into that data-plane mesh; the default `sandbox` module deliberately does **not** — only narrow per-service egress (Design §5). +- **`design-proposals/kubernetes-nodes-split`** / **`kubernetes-nodes-hybrid-clusters`** (PR #8/#9): the substrate is the existing managed-`kubernetes` app (Kamaji + CAPI/KubeVirt); node-provisioning changes apply transparently. +- **Deferred:** billing/metering of cluster resource and API consumption; secret delivery of managed-service connection strings into sandbox workloads; the per-instance/label granularity of the visibility control (Design §6). (Cross-tenant *sharing* of a cluster is **not** deferred — it is rejected by design; see Non-goals.) ## Context -Today Cozystack already has every primitive needed *except* the glue that ties them into "deploy this app to a separate, Cozystack-managed cluster the tenant does not administer": - -- **Tenants** (`packages/apps/tenant/`) are the unit of isolation: a hierarchical namespace with its own Cilium network policies, RBAC, and quotas. Cluster services are opt-in **Tenant modules** exposed as single scalar toggles in `values.yaml` — `etcd`, `monitoring`, `ingress`, `gateway`, `seaweedfs` (booleans today) — each rendered as a conditional `HelmRelease` in the tenant namespace. Module enablement is set by the **parent** tenant at child-creation time; module values flow down through a per-namespace `cozystack-values` Secret. - -- **Managed Kubernetes** (`packages/apps/kubernetes/`, user-facing `kind: Kubernetes`) provisions a tenant cluster with a **Kamaji-hosted control plane** (`KamajiControlPlane`, backed by the tenant's `etcd`) and **CAPI + KubeVirt** worker nodes. `values.yaml` exposes `nodeGroups` with `minReplicas` / `maxReplicas` (autoscaling), `instanceType`, `roles`, `resources`, and `gpus` (e.g. `nvidia.com/AD102GL_L40S`). GPU node groups and the cluster-autoscaler are already supported. It also ships the cross-cluster plumbing the ComputePlane reuses: `exposeMethod: Proxied` (management ingress → tenant NodePort), a KubeVirt cloud-controller-manager (`kubevirt-cloud-provider`) that provisions `Service type: LoadBalancer` from the management cluster, and the `kubevirt-csi-driver` for persistent storage. +Today Cozystack already has every primitive needed *except* the glue that ties them into "deploy this catalog app onto a hardened, operator-controlled `kind: Kubernetes` the tenant does not administer": -- **Remote Flux apply already works.** The `kubernetes` app deploys its own in-cluster addons (cert-manager, ingress-nginx, CNI, monitoring agents, …) by creating `HelmRelease` objects *on the management cluster* that carry `spec.kubeConfig.secretRef`, pointing at the freshly-provisioned cluster's admin kubeconfig. Kamaji writes that kubeconfig to a Secret named `-admin-kubeconfig` (key `super-admin.svc`). This is the exact mechanism a ComputePlane needs: a management-side `HelmRelease` whose `kubeConfig` targets the ComputePlane. +- **Tenants** (`packages/apps/tenant/`) are the unit of isolation: a hierarchical namespace with its own Cilium network policies, RBAC, and quotas. Cluster services are opt-in **Tenant modules** (`etcd`, `monitoring`, `ingress`, `seaweedfs`, …), each a chart under `packages/extra/` switched on per-tenant; enablement is set by the **parent** tenant at child-creation time, and module values flow down through a per-namespace `cozystack-values` Secret. ComputePlane is a new module of exactly this shape. +- **Managed Kubernetes** (`packages/apps/kubernetes/`, `kind: Kubernetes`) provisions a tenant cluster with a **Kamaji-hosted control plane** and **CAPI + KubeVirt** worker nodes. `values.yaml` exposes `nodeGroups` (`minReplicas`/`maxReplicas` autoscaling, `instanceType`, `roles`, `resources`, `gpus`) and the addon set. It also ships the cross-cluster plumbing reused here: `exposeMethod: Proxied` (management ingress → tenant NodePort), `kubevirt-cloud-provider` (`Service type: LoadBalancer` from the management cluster), and `kubevirt-csi-driver`. **The module wraps this app unchanged** — it is not a fork or a new component. +- **PackageSource "source-only" charts.** A `PackageSource` component with no `install:` block registers a chart as an available `ExternalArtifact` (named `cozystack---` in `cozy-system`) without installing it; a sibling component *with* `install:` (the `*-rd` resource-definition chart) is what registers the user-facing kind/module. This is how SeaweedFS (`extra/seaweedfs`) and NATS (`apps/nats`) are already delivered. `apps/kubernetes` is already registered source-only in `kubernetes-application`; the ComputePlane package re-declares it as its own source-only component (Design §2) so the module is self-contained. +- **Remote Flux apply already works.** The `kubernetes` app deploys its own addons by creating `HelmRelease`s *on the management cluster* carrying `spec.kubeConfig.secretRef` → the cluster's `-admin-kubeconfig` Secret (key `super-admin.svc`, written by Kamaji). This is exactly the mechanism placement routing uses to put a catalog app onto the ComputePlane. +- **ApplicationDefinition** (`api/v1alpha1/applicationdefinitions_types.go`) maps a user-facing `kind` → a `HelmRelease` via `cozystack-api` (`pkg/registry/apps/application/rest.go`, `ConvertApplicationToHelmRelease()`). `spec.dashboard` already drives UI presentation (incl. `module: true`); the visibility control (Design §6) extends that path. +- **Network isolation** (`packages/apps/tenant/templates/networkpolicy.yaml`): the `-egress` `CiliumClusterwideNetworkPolicy` selects every pod in the tenant namespace — including the KubeVirt `virt-launcher` node pods — and denies egress to the kube-apiserver by default. This is the enforcement point the `sandbox` posture and the scoped data-plane egress (Design §5) build on. -- **ApplicationDefinition** (`api/v1alpha1/applicationdefinitions_types.go`, CRD in `packages/system/application-definition-crd/`) maps a user-facing `kind` to a `HelmRelease`. `spec.release.chartRef` (an `ExternalArtifact`) + `spec.release.prefix` define the chart and the release-name prefix; `spec.application.openAPISchema` validates user input; `spec.dashboard` (with `module: true`) controls UI presentation, including marking a resource as a tenant module. The aggregated `cozystack-api` apiserver (`pkg/registry/apps/application/rest.go`) serves these as `apps.cozystack.io/*` kinds and, on write, converts the resource into a `HelmRelease` via `ConvertApplicationToHelmRelease()`. - -- **Network isolation** (`packages/apps/tenant/templates/networkpolicy.yaml`): per-tenant `CiliumNetworkPolicy` denies pod egress to the kube-apiserver by default. A pod reaches the API only if it carries `policy.cozystack.io/allow-to-apiserver: "true"` (and analogously `allow-to-etcd`). This is the enforcement point the isolation guarantee — and the scoped data-plane egress of Design §5 — build on. - -What does **not** exist yet (grep confirms): anything named "ComputePlane", a separate-cluster placement concept on `ApplicationDefinition`, or a scoped ComputePlane→tenant-service egress policy. The pieces are present; the assembly is new. +What does **not** exist yet: the `extra/computeplane` module chart and its `computeplane-application` PackageSource, a `placement` field on `ApplicationDefinition`, the scoped ComputePlane→tenant-service egress policy, and the `cozystack-api` visibility/mutation control. The substrate is present; the assembly is new. ### The problem -> "I want to offer JupyterHub (or n8n, or ComfyUI, or WordPress) from the Cozystack dashboard. Each of these runs arbitrary user code as a feature. If I deploy them into the tenant namespace on my shared infra cluster, a single container-escape CVE turns a notebook into host root and then into every tenant's secrets. Today my only safe option is to not ship them at all, or to tell users to first provision a full managed Kubernetes cluster and install the app themselves — which is neither the dashboard one-click experience nor something I can bill and operate cleanly." +> "I want to offer JupyterHub (or n8n, ComfyUI, WordPress) from the dashboard. Each runs arbitrary user code as a feature. Deployed as pods in the tenant namespace on the shared management cluster, one container-escape CVE turns a notebook into root on a management node and across every tenant. My only safe options today are to not ship them, or to tell users to provision a full managed cluster and install it themselves — neither is a one-click managed service." -The platform already isolates *tenant-owned* untrusted compute — a managed `kind: Kubernetes` or a VM runs behind the same VM boundary. What it lacks is a way to offer code-executing apps **from the catalog**, as managed services with safe defaults, without dropping them as shared-kernel pods on the management cluster. ComputePlane is that delivery path: the operator keeps control of the cluster (so the app is guaranteed to run and stay maintained), and the untrusted code sits behind the VM boundary — keeping the dashboard one-click UX intact. +The platform already isolates *tenant-owned* untrusted compute (a managed `kind: Kubernetes` runs behind the VM boundary). What it lacks is a way to offer code-executing apps **from the catalog**, as managed services with safe defaults, without dropping them as shared-kernel pods on the management cluster. ComputePlane — a per-tenant `extra` module that provisions a hardened `kind: Kubernetes` and a `placement` field that routes catalog apps onto it — is that delivery path. ## Goals -- A tenant can run an untrusted-code application through the normal dashboard/API flow, with no change to how they create it. -- The application's pods run on a Cozystack-managed cluster — the same VM-isolated substrate as any managed `kind: Kubernetes`: no kubeconfig/ServiceAccount token for and no kube-API path to the management/infra kube-apiserver, and untrusted code confined behind a per-VM guest kernel. This isolation is **inherited from the managed-cluster model, not introduced by ComputePlane**; ComputePlane's job is to route catalog apps onto it automatically instead of onto shared-kernel management nodes. -- The management cluster reconciles workloads *into* the ComputePlane (one-way remote Flux apply); the ComputePlane never receives credentials pointing back. -- The operator retains control of the cluster (the tenant gets no admin/write), so the platform owns and maintains the hardening profile — restricted PSA, egress policy, admission — the way it owns a managed Postgres config: correct by default and not the tenant's to assemble or accidentally undo. This protects the tenant's environment from the tenant's *own* app users (notebook code, LLM-generated code) and keeps the operator's managed guarantees; it is **not** a boundary against the tenant themselves, who could run the same code on an unhardened self-service cluster. Read/observability access for the tenant is a separate, allowable extension. -- A ComputePlane serves **exactly one tenant**; it is never an isolation domain shared between tenants. -- The mechanism is generic (a Cozystack-core primitive) and reusable by any code-executing app catalog through a pluggable interface, not bolted to one product. +- A tenant runs an untrusted-code catalog app through the normal create-an-app flow, unchanged. +- The app's pods run on a managed cluster — the same VM-isolated substrate as any `kind: Kubernetes`: no kube-API path or credentials back to management, untrusted code behind a per-VM guest kernel. **Inherited from the managed-cluster model, not introduced here.** +- The cluster's posture and contents are chosen by the **operator, in the module chart** — not assembled by the tenant, and not multiplied into a combinatorial set of kinds. The tenant sees only the curated knobs the module exposes. +- The operator retains admin of the cluster (the tenant gets no kubeconfig), so platform-owned hardening cannot be stripped — protecting the tenant's environment from the tenant's *own* app users (notebook/LLM-generated code) and keeping the managed guarantee. (Not a boundary against the tenant themselves — see Security.) +- Delivered through the standard `apps`/`extra` packaging, reusing the existing PackageSource + Tenant-module machinery — no new CRD, no new controller, no new reconcile path. +- Generic Cozystack-core primitive, reusable by any code-executing catalog. ### Non-goals -- This proposal does **not** make Kubernetes itself multi-tenant or claim container isolation is sufficient; ComputePlane places code-executing *catalog* apps on the existing VM-isolated managed-cluster substrate rather than as shared-kernel pods on the management cluster. -- It does **not** share a ComputePlane across tenants — parent or child. A ComputePlane is single-tenant **by design** (Design §2 explains why inheriting one re-creates the very escalation it prevents). Sharing the **physical node pool / capacity** across tenants is fine (infra-layer pooling); sharing a **ComputePlane** (a cluster / isolation domain) is not. -- It does **not** present invisibility, or the tamper-proof hardening, as a *platform* security boundary. Withholding tenant admin is about separation of responsibility and keeping the platform-managed hardening intact for the tenant's own benefit (protecting their environment from their app's users); it is not what keeps the management plane safe — see Security. -- It does **not** design billing/metering, nor the secret delivery of managed-service connection strings into ComputePlane workloads. Those are acknowledged as adjacent work. -- It does **not** propose gVisor / sandboxed-runtime isolation as the primary boundary (evaluated under Alternatives; considered too immature to be the trust boundary, and it does not address kernel-panic blast radius). +- Does **not** make Kubernetes multi-tenant or claim container isolation suffices; it places catalog apps on the existing VM-isolated substrate. +- Does **not** share a cluster across tenants (parent or child). Single-tenant **by design** (Design §7). Sharing the **node pool / capacity** is fine; sharing a **cluster** is not. +- Does **not** present invisibility, or the tamper-proof hardening, as a *platform* security boundary (it protects the tenant from their own app's users; it is not what keeps the management plane safe — see Security). +- Does **not** introduce a distinct top-level CRD for ComputePlane, nor new user-facing configuration fields on `kind: Kubernetes` (the central change in this revision — the configuration lives in the operator-owned module chart, not on the API object). +- Does **not** propose gVisor as the *primary* boundary (Alternatives) — though gVisor is a valid *inner* layer for ephemeral per-task sandboxes within a cluster (it sidesteps Kata nested-virt inside KubeVirt VMs), tracked as a future runtime option. ## Design -### 1. The ComputePlane is a managed Kubernetes cluster, owned by Cozystack rather than the tenant +### 1. ComputePlane is an `extra` Tenant module that wraps `apps/kubernetes` -Reuse the existing `kubernetes` app as the cluster substrate. A ComputePlane is a `kind: Kubernetes`-equivalent resource — a Kamaji control plane plus CAPI/KubeVirt worker node groups — provisioned and owned by Cozystack on the tenant's behalf, but **not surfaced to the tenant as a manageable `Kubernetes` resource**: the tenant gets no admin kubeconfig for it (Security → Visibility). Because the worker nodes are KubeVirt VMs (kubelet-in-a-VM), untrusted code runs behind a virtualization boundary, and a kernel panic provoked by the workload takes down a VM, not a physical node. +Reuse the `kubernetes` app as the substrate, unchanged. A ComputePlane is delivered as a new Tenant module, `packages/extra/computeplane`, of the same shape as `extra/etcd` / `extra/seaweedfs`. When a tenant has the module enabled, the tenant chart renders a `HelmRelease` (in the tenant namespace, on the management cluster) whose `chartRef` points at the **source-only** `apps/kubernetes` `ExternalArtifact`, carrying operator-fixed values: + +```yaml +# packages/extra/computeplane — rendered HelmRelease (illustrative) +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: computeplane + namespace: tenant- # management cluster, tenant namespace +spec: + chartRef: + kind: ExternalArtifact + name: cozystack-computeplane-application-kubevirt-kubernetes # apps/kubernetes, re-sourced (Design §2) + namespace: cozy-system + values: # operator-owned; the tenant cannot edit these + # hardened posture: restricted PSA + admission, deny egress → management kube-apiserver, + # scoped per-service egress only, addon set, GPU node groups, autoscaling bounds … + valuesFrom: + - kind: Secret # only the curated knobs the module chooses to surface + name: cozystack-values +``` -To keep the user-facing API unambiguous, the ComputePlane is exposed as a **distinct kind**, `kind: ComputePlane`. It is *not* the same as `kind: Kubernetes`, precisely so users do not confuse "a cluster I administer" with "the Cozystack-managed cluster my apps run on." The name mirrors "control plane" (management cluster) vs. "compute plane" (where workloads run). +Why a module wrapping the app, rather than a new kind or new fields on the app: -GPU and autoscaling come for free: the ComputePlane's `nodeGroups` carry `gpus` and `minReplicas: 0` / `maxReplicas: N`, so the cluster sits idle (one small node for system workloads) until a GPU-hungry app is created, then the cluster-autoscaler adds GPU nodes on demand. +- **Same object, no new surface.** From the management plane a hardened compute cluster and a regular managed cluster share all machinery (Kamaji, KubeVirt nodes, autoscaler, addon Flux apply). A distinct kind, or a `sandbox` field on `kind: Kubernetes`, would add API surface for something that is fully expressible as *a set of values the operator picks*. Keep the app's API untouched; put the choice in the wrapping chart. +- **`apps` vs `extra` says `extra`.** The tenant does not *order* a ComputePlane the way they order a Postgres or a Kubernetes cluster — it is switched on to *enable* a catalog of code-executing apps. Per [website#594](https://github.com/cozystack/website/pull/594) that is precisely an `extra` module. +- **Tamper-proof by construction.** The tenant never receives the admin kubeconfig and never sees a `Kubernetes` CR they can edit, so the hardening in the operator-owned `values` cannot be weakened. There is no user-facing field to reconcile against a profile — the chart *is* the profile. +- **The user still gets a real `kind: Kubernetes`.** It is the same cluster they would get from the catalog, with the same lifecycle, autoscaling and GPU support — they simply do not administer it and can change only what the module exposes. ```mermaid flowchart TB - subgraph user[" "] - U[Tenant user
dashboard / API] - end + U[Tenant user
dashboard / API] subgraph mgmt["Management / infra cluster (control plane)"] - AD["ApplicationDefinition
(kind: JupyterHub)
placement: ComputePlane"] - HR["HelmRelease
spec.kubeConfig.secretRef →
computeplane-admin-kubeconfig"] + MOD["extra/computeplane module (enabled on tenant)
→ HelmRelease chartRef: apps/kubernetes
+ operator-fixed values"] + AD["ApplicationDefinition (kind: JupyterHub)
placement: ComputePlane"] + HR["HelmRelease
kubeConfig.secretRef → computeplane-admin-kubeconfig"] IAP["ingress / identity-aware proxy"] end - subgraph cp["ComputePlane (single-tenant, Cozystack-managed)"] - direction TB - WL["Jupyter / n8n / ComfyUI / WordPress pods
(KubeVirt-VM worker nodes, GPUs here)"] + subgraph cp["ComputePlane = kind: Kubernetes (single-tenant, operator-controlled)"] + WL["Jupyter / n8n / ComfyUI pods
(KubeVirt-VM nodes, GPUs here)"] end - U -->|"creates 'JupyterHub' (unchanged UX)"| AD + MOD --> HR + U -->|"create 'JupyterHub' (unchanged UX)"| AD AD --> HR HR -->|"remote apply (one-way)"| WL U -->|"app access"| IAP @@ -101,156 +125,124 @@ flowchart TB WL -.->|"NO creds, NO kube-API path back"| mgmt ``` -### 2. The ComputePlane is delivered as a Tenant module (single-string profile reference), single-tenant by design +### 2. Packaging: a `computeplane-application` PackageSource that re-sources `apps/kubernetes` -A ComputePlane fits the existing Tenant-module logic exactly: it is a dependency a tenant's apps consume at deploy time, the same way `kubernetes` clusters consume the tenant's `etcd`/`monitoring`/`ingress`. We add a new module to the Tenant chart that mirrors how existing modules are **single scalars** — not a nested `{ enabled, valuesOverride }` object. The field is a single string naming a ComputePlane profile/class: +The module is registered by a `PackageSource` (`packages/core/platform/sources/computeplane-application.yaml`) built like `seaweedfs-application` / `nats-application`, with three components: -```yaml -# packages/apps/tenant/values.yaml (proposed) -computePlane: "" # name of a ComputePlane profile/class; empty = disabled -``` +- `computeplane` → `path: extra/computeplane` — **source-only** (no `install:`); the module chart itself. +- `kubernetes` → `path: apps/kubernetes` — **source-only**, a deliberate **duplicate** of the component already in `kubernetes-application`, so the wrapped chart's `ExternalArtifact` is materialized within this package's own scope and the module is self-contained (it does not depend on the `kubernetes-application` source being independently enabled). +- `computeplane-rd` → `path: system/computeplane-rd` with `install:` — the resource-definition chart carrying the module's `cozyrd` (its `ApplicationDefinition` / dashboard presentation and the tenant-values wiring), the same role `seaweedfs-rd` / `kubernetes-rd` play. -An empty string means disabled; a non-empty value names a ComputePlane **profile/class**. The profile — node groups, GPU types, Kubernetes version, autoscaling bounds — is defined **once** at the tenant/platform level and referenced by name. There is deliberately **no per-module `valuesOverride` blob**: that would re-create the two-sources-of-truth problem (the module spec living both in the profile and inline on the tenant). Keeping the module a bare profile reference makes the profile definition the single source of truth. +The tenant chart gains a `computePlane` toggle (a bool, or a profile name if the operator ships more than one module variant) alongside the existing module toggles; when set by the parent tenant, `packages/apps/tenant/templates/computeplane.yaml` renders the `HelmRelease` above. Kamaji writes the cluster's admin kubeconfig to `computeplane-admin-kubeconfig` (key `super-admin.svc`), which `placement: ComputePlane` apps consume (Design §4). -When set (by the parent tenant, as with all modules today), the Tenant chart renders the ComputePlane `HelmRelease` (using the named profile) into the tenant namespace, and Cozystack records the ComputePlane's admin-kubeconfig Secret reference for use by `placement: ComputePlane` apps in that tenant. +### 3. What the operator exposes vs fixes -**A ComputePlane serves exactly one tenant — there is no parent-walk and no inheritance.** A `placement: ComputePlane` app in a tenant that has its own `computePlane` set deploys into *that* ComputePlane; a tenant with no ComputePlane of its own gets the app **rejected**, not deployed onto an ancestor's ComputePlane. Inheriting a ComputePlane would put a child's untrusted code into the parent's isolation domain — the parent then stands to the child exactly as the management cluster stands to a tenant, and a container escape on the shared ComputePlane lands in the parent's compute environment. That re-creates the same cross-tenant escape one level down the tenant tree — exactly what the substrate's per-tenant isolation otherwise gives you — so it is excluded **by design** rather than blocked-for-now. (Sharing the underlying **node pool / capacity** across tenants is a separate, acceptable infra-layer optimization — comparable to many managed databases sharing instance types; sharing the **cluster / isolation domain** is not.) +Everything security-relevant is **fixed** in the module chart and unreachable by the tenant: the hardened PSA/admission profile, the deny-egress-to-management-kube-API policy, the scoped per-service egress contract (Design §5), the withheld-kubeconfig posture, and the addon set. The operator may **expose** a curated, safe subset as tenant-settable knobs through the module's `values.yaml` (e.g. GPU on/off and type, cluster size / autoscaling bounds, Kubernetes version) — the same "high-level API, operator-controlled" contract every other managed service follows. There is deliberately **no** general `valuesOverride` passthrough onto the wrapped `kind: Kubernetes`: that would re-open the tamper surface and re-create the two-sources-of-truth problem. Different postures (e.g. a future *trusted*/`cluster-meshed` variant) are separate module variants or separate `extra` charts, not a field on the app. -### 3. `placement: ComputePlane` apps deploy *into* the ComputePlane via remote Flux apply +### 4. `placement` routes a catalog app onto the ComputePlane -This is where the design leans entirely on an existing, proven mechanism. An application that must be isolated declares `placement: ComputePlane` on its `ApplicationDefinition`. When a tenant creates the app: +`ApplicationDefinition` gains `placement` = `ManagementPlane` (default) | `ComputePlane`. `ManagementPlane` deploys into the tenant namespace on the management cluster, as today. `ComputePlane` routes the generated `HelmRelease` onto the tenant's ComputePlane by injecting `spec.kubeConfig.secretRef` → `computeplane-admin-kubeconfig` (the module's fixed secret) and `spec.install.createNamespace: true`. Because the module is a single per-tenant enabler, `ComputePlane` resolves unambiguously to *that* tenant's cluster — no cluster name to thread through the app. -1. As today, the `cozystack-api` REST layer converts the resource into a `HelmRelease` on the management cluster. -2. **Unlike** a `placement: ManagementPlane` app, the generated `HelmRelease` carries `spec.kubeConfig.secretRef` pointing at the tenant's ComputePlane admin kubeconfig (the `-admin-kubeconfig` Secret, key `super-admin.svc`, written by Kamaji) — exactly the pattern the `kubernetes` app already uses for its in-cluster addons. It also sets `spec.install.createNamespace: true`, since the target namespace does not exist on a freshly provisioned ComputePlane. -3. Flux on the management cluster therefore applies the chart **into the ComputePlane**, never into the tenant namespace on the management cluster. +*(Optional, advanced — deferred.)* The same routing generalizes to `placement: ` for a tenant that deliberately runs its own `kind: Kubernetes` and wants apps placed there; that path reuses the identical `kubeConfig` injection with the cluster's own `-admin-kubeconfig`. It is not the default delivery surface and is out of scope for v1. -```yaml -# Generated HelmRelease for a placement: ComputePlane app (illustrative) -apiVersion: helm.toolkit.fluxcd.io/v2 -kind: HelmRelease -metadata: - name: jupyterhub- - namespace: tenant- # lives on the management cluster -spec: - chartRef: - kind: ExternalArtifact - name: cozystack-jupyterhub-application - namespace: cozy-system - kubeConfig: # <-- injected only when placement == ComputePlane - secretRef: - name: computeplane--admin-kubeconfig - key: super-admin.svc - install: - createNamespace: true # the target ns does not exist on a fresh ComputePlane - values: { ... } -``` +### 5. Connectivity, remote apply, and inbound access -The routing decision is driven by the `placement` enum on the `ApplicationDefinition` — `ManagementPlane` (default) applies into the tenant namespace on the management cluster as today; `ComputePlane` injects the ComputePlane `kubeConfig.secretRef`. The two values name the two symmetric planes. This keeps the routing policy declarative and out of per-app charts. +- **Remote apply (into the cluster).** For a `placement: ComputePlane` app, `cozystack-api` converts it to a `HelmRelease` on the management cluster carrying `spec.kubeConfig.secretRef` → `computeplane-admin-kubeconfig`; Flux applies the chart **into the ComputePlane**, never into the tenant namespace on management. This is exactly what the first revision implemented. +- **Connectivity to tenant services (`sandbox` data-plane contract).** A notebook/LLM/n8n flow needs the tenant's data ("my Jupyter → my managed Postgres"), which runs in the tenant namespace on the management cluster. The guarantee is "no kube-API access / no creds to escalate," **not** "no packets ever." No mesh is required: the ComputePlane's KubeVirt-VM node pods sit on the management Cilium pod network, so reachability is a **scoped per-service `CiliumNetworkPolicy`** — allow → the tenant's Postgres Service, deny → kube-apiserver (same shape as the existing `policy.cozystack.io/allow-to-apiserver` label). Per-service egress is narrower-by-construction than a node mesh — which matters because the consumer is untrusted code. +- **Inbound access.** Workloads expose themselves via Ingress/Gateway on the ComputePlane; the management ingress proxies to the cluster's `exposeMethod: Proxied` NodePort (or a kubevirt-ccm `Service type: LoadBalancer`). Inbound data path only — no reverse kube-API path, and the tenant never receives cluster credentials. -### 4. Access is proxied back through the tenant's normal entry point +### 6. Tenant-side visibility / mutation control (`cozystack-api` extension) -Workloads expose themselves on the ComputePlane via standard Ingress/Gateway, and the ComputePlane's ingress is wired back to the tenant's existing entry point so the user reaches the app at a normal hostname. Concretely this reuses the managed-`kubernetes` plumbing: the management cluster's ingress proxies to the ComputePlane's `exposeMethod: Proxied` NodePort on the KubeVirt-VM nodes (or to a `Service type: LoadBalancer` that kubevirt-ccm provisions from the management cluster). This is an **inbound data path only** — HTTP(S) app traffic crosses back through the proxy/ingress path; no reverse kube-API path is opened, and the user never receives ComputePlane credentials. +The #17 review's "withhold admin/write, allow scoped read" deserves first-class treatment: it is both a general tenant feature **and** part of the enforcement for a ComputePlane (the tenant should not see or manage the module's underlying `HelmRelease`/cluster object as if it were their own cluster). Let managed apps (and the ComputePlane's backing objects) be marked so a tenant's *regular* users cannot see or manage them while *privileged* subjects (tenant-admin / parent-tenant operator / superadmin) still can. Two separable controls: -### 5. Connectivity to tenant services +- **Visibility** — whether the object appears in `kubectl get` / the dashboard for a subject. +- **Mutation** — whether a subject may create/update/delete it. -The headline workloads are nearly useless in isolation — a notebook, an LLM, an n8n flow exist to reach the tenant's data ("my Jupyter notebook talks to my managed Postgres"). But the tenant's managed Postgres runs as a Service in the tenant namespace **on the management cluster**, so reaching it is a ComputePlane→management flow — the direction Security §2 otherwise restricts. The resolution is to be precise about *which* plane is restricted. +**Enforced in the aggregated `cozystack-api` apiserver** (`pkg/registry/apps/application/rest.go`), which already serves `apps.cozystack.io/*` and converts them to HelmReleases — **not UI-only** (a dashboard filter is bypassable by anyone holding a kubeconfig). Granularity: -**The guarantee is "no kube-API access / no creds to escalate," not "no packets ever."** Those are different planes. A database connection (ComputePlane → `postgres-pod:5432`) is a *data-plane* flow and can be allowed while ComputePlane → kube-apiserver stays denied. +- Per-**kind** ("regular users can't touch `Jupyter`") → plain Kubernetes RBAC (a tenant Role omitting the verbs). +- Per-**instance** / **label-driven** → cannot ride vanilla RBAC (`list`/`watch` can't filter by name/label); needs `cozystack-api` to filter responses by caller identity. Feasible since Cozystack owns the apiserver, but it is apiserver work, not a Role manifest. **(Deferred to a later iteration; v1 ships per-kind only.)** -**No mesh is required.** ComputePlane worker nodes are KubeVirt VMs attached to the management cluster's Cilium pod network (`packages/apps/kubernetes/templates/cluster.yaml` → `networks: - name: default; pod: {}`), so L3 adjacency already exists and is gated by `CiliumNetworkPolicy`. Connectivity is therefore a **scoping** problem, expressed with machinery already in the platform: +Shape (early, names TBD): an annotation/field on `ApplicationDefinition` (likely extending `spec.dashboard`, which already drives presentation / `module: true`) marking an app restricted, plus a tenant-role tier (`regular` vs `privileged`) checked before listing/mutating. -- **Egress to a tenant service** is granted by a narrow, per-service `CiliumNetworkPolicy`: allow ComputePlane workloads → a specific endpoint (the tenant's Postgres Service), deny ComputePlane → kube-apiserver. This is the same shape as the existing `policy.cozystack.io/allow-to-apiserver` label policy, pointed at a data-plane endpoint instead of the API. Per-service egress is narrower by construction than a node-to-node mesh, so untrusted workloads never get broad reach into the infra network. -- **Exposing a ComputePlane workload outward** reuses what the managed `kubernetes` app already ships (Design §4): `exposeMethod: Proxied` and kubevirt-ccm `Service type: LoadBalancer`; persistent storage uses the `kubevirt-csi-driver` path. +### 7. Single-tenant by design -Open: who authorizes each tenant-service endpoint and how the per-service policy is generated (a tenant-scoped allowlist vs. an explicit "expose this service to my ComputePlane" action). The remaining secret-delivery half — getting the Postgres connection string into the workload — is tracked under Open questions; the **network** half is solved by the scoped policy above, and every such opening is narrow and audited by construction. +Each `kind: Kubernetes` is a single tenant's cluster, and the module does not change that: a tenant's ComputePlane serves that tenant only. There is **no module inheritance / parent-walk** — a `placement: ComputePlane` app whose tenant has the module disabled is **rejected**, never routed onto an ancestor's ComputePlane. Inheriting one would put a child's untrusted code into the parent's isolation domain, re-creating the cross-tenant escape one level down. Sharing the underlying **node pool / capacity** across tenants stays fine; sharing a **cluster** does not. -### 6. Pluggable, core-level primitive +### 8. Pluggable, core-level primitive -The ComputePlane (cluster provisioning + module wiring + remote-apply routing) belongs in **Cozystack core**. Consumers — `cozyllm`, a future WordPress catalog, the application-platform work — depend on it through the `ApplicationDefinition` `placement` enum and the module field, without re-implementing remote apply. This keeps the LLM product and any future code-executing catalog on one isolation mechanism. +The module, the `placement` field, and the visibility control live in **Cozystack core**. Consumers (`cozyllm`, a future WordPress catalog, the application-platform work) depend on them by (a) requiring the `computeplane` module on the tenant and (b) setting `placement: ComputePlane` on their code-executing `ApplicationDefinition`s — no re-implementation of remote apply in the app charts. ## User-facing changes -- **Tenant admins (parent tenants):** a new module field `computePlane` (a string naming a profile) when creating/configuring a child tenant, alongside `etcd`/`monitoring`/`ingress`/`seaweedfs`. A non-empty value provisions the cluster. -- **Tenant users:** *no change* to how they create an app. They do not administer the ComputePlane and hold no admin credentials to it; a scoped read/observability view of their own workloads is a possible extension (Security → Visibility), but they never receive a cluster they manage. -- **App authors:** set `placement: ComputePlane` on an `ApplicationDefinition` (default is `ManagementPlane`). No per-app remote-apply plumbing. -- **CRD shape:** a new user-facing kind `ComputePlane` registered via `ApplicationDefinition`, intentionally distinct from `Kubernetes`. New `placement` enum (`ManagementPlane` | `ComputePlane`, default `ManagementPlane`) on `ApplicationDefinitionApplication`. +- **App authors:** set `placement: ComputePlane` on a code-executing `ApplicationDefinition` (default `ManagementPlane`). Optionally mark an app restricted (Design §6). +- **Operators:** enable the `computeplane` module on a tenant (as with any other Tenant module); optionally set the curated knobs the module exposes (GPU, size). No cluster to hand-assemble. +- **Tenant users:** *no change* to creating an app; they don't administer the ComputePlane and (for restricted apps) may not see it. +- **API shape:** new `placement` field on `ApplicationDefinition`; a new `computeplane` Tenant module (`extra/computeplane` + `computeplane-rd` + `computeplane-application` PackageSource); a restricted-app marker + tenant-role tier consumed by `cozystack-api`. **No new CRD, and no new fields on `kind: Kubernetes`.** ## Upgrade and rollback compatibility -- Additive. Existing Tenant manifests and existing apps are unaffected — the `computePlane` module field defaults to empty (disabled), and apps default to `placement: ManagementPlane`, so they keep deploying on the management cluster. -- The new `placement` enum on `ApplicationDefinition` is optional and defaults to `ManagementPlane`; older `ApplicationDefinition`s are valid unchanged. -- Disabling the ComputePlane module / reverting the feature: `placement: ComputePlane` apps stop being routed remotely. Because their workloads live *only* on the ComputePlane, removing the module must be treated like deleting a managed cluster (data on the ComputePlane is lost) — flag this clearly in the module's delete path; it is the one not-cheaply-reversible operation. -- Remote-apply via `spec.kubeConfig` is already a supported Flux feature in use by the `kubernetes` app, so no Flux/CRD upgrade is required. +- Additive and backward-compatible. `placement` defaults to `ManagementPlane`; the `computeplane` module defaults to disabled — so every existing `ApplicationDefinition`, `Tenant`, and `Kubernetes` manifest is valid unchanged. `kind: Kubernetes` is untouched. +- Remote-apply via `spec.kubeConfig` is already a supported Flux feature, and source-only PackageSource components are already how SeaweedFS/NATS ship — no Flux/CRD upgrade is required. +- Disabling the module (removing the ComputePlane) is deleting a managed cluster (its data is lost) — the not-cheaply-reversible operation; gate it accordingly (Design §7 / Failure cases). ## Security -ComputePlane does **not** introduce a new isolation boundary, and the design should not be read as one. From the management plane's point of view, a ComputePlane and a regular managed `kind: Kubernetes` cluster are the same object — Kamaji control plane, KubeVirt-VM workers, operator-held super-admin kubeconfig, and the tenant-namespace egress policy. Its security value is two pre-existing properties of that substrate, plus a managed-service contract on top — not a boundary it invents. +ComputePlane does **not** introduce a new isolation boundary. From the management plane, the cluster it runs on and a regular managed `kind: Kubernetes` are the same object. Its security value is two pre-existing substrate properties plus a managed-service contract — not a boundary it invents. -**Inherited from the managed-`kubernetes` substrate** (each verified by tests): +**Inherited from the managed-`kubernetes` substrate** (each verified by tests): (1) no management credentials in the cluster (the kubeConfig lives management-side); (2) no kube-API path to management — the `-egress` policy already denies the node `virt-launcher` pods egress to the kube-apiserver, while scoped per-service data-plane egress (Design §5) may be granted; (3) the **virtualization boundary** (defense in depth) — a kernel-level escape like Copy Fail (CVE-2026-31431) is contained to a disposable VM, not a management node's shared host kernel; (4) separate identity domain (own Kamaji control plane + RBAC); (5) single-tenant (Design §7); (6) no new tenant-supplied input to the management plane. -1. **No management credentials in the cluster.** The kubeConfig lives management-side and is never copied in — true of any managed cluster. -2. **No kube-API path to the management plane.** The existing `-egress` `CiliumClusterwideNetworkPolicy` already selects every pod in the tenant namespace — including the KubeVirt `virt-launcher` pods that are the cluster's nodes — and denies egress to the management kube-apiserver. Narrow, per-service **data-plane** egress to specific tenant Services may still be granted (Design §5); the kube-API plane stays denied. No ComputePlane-specific policy is required. -3. **Virtualization boundary (defense in depth).** Untrusted code runs on KubeVirt-VM nodes with their own guest kernel, so a kernel-level container escape — the recurring class, e.g. Copy Fail (CVE-2026-31431), which corrupts a *shared* page cache to cross from container to host — is contained to a disposable VM instead of reaching a management node's shared host kernel. This is the reason the platform puts untrusted compute in VMs at all; ComputePlane simply ensures code-executing catalog apps land there rather than as bare pods on the management cluster. -4. **Separate identity domain.** Its own Kamaji control plane and RBAC — identical to any managed cluster. -5. **Single-tenant.** A ComputePlane serves exactly one tenant (Design §2 removes module inheritance so it cannot be shared down the tenant tree). A regular managed cluster is single-tenant too; the design work here is only to ensure the *module* does not re-introduce sharing. -6. **No new tenant-supplied input to the management plane.** Tenants still only write `apps.cozystack.io/*`; placement is decided by platform-owned `ApplicationDefinition` metadata. - -**Separation of responsibility (the managed-service contract).** The compute cluster stays under operator control: the tenant never touches the management control plane and cannot block platform updates, and the operator's management never breaks the tenant's deployed workloads. The operator owns the substrate and the hardening profile, the tenant owns their app. This is what distinguishes ComputePlane from "the tenant provisions their own `kind: Kubernetes` and installs the app" — the operator can guarantee the app runs, keep it patched, and bill for it. +**Separation of responsibility (the managed-service contract).** The cluster stays operator-controlled: the tenant cannot block platform updates, and management never breaks the tenant's app. The operator owns the substrate and hardening; the tenant owns their app. ### What the hardening does and does not protect -The one thing a ComputePlane has that a tenant-run cluster does not is **tamper-proof hardening**: the tenant is not cluster-admin, so platform-applied PSA / network policy / admission cannot be stripped. This is **not** a platform boundary against the tenant, and the doc does not sell it as one. A tenant who actually holds a management-hijacking payload does not run it in the hardened ComputePlane — they provision a regular managed `kind: Kubernetes` (same substrate, one click, but they are admin and the hardening is absent) and run it there. The hardened venue is optional for that attacker, so it adds nothing against them; the substrate (per-VM guest kernel + the `-egress` policy over the VM pods) is what contains arbitrary tenant code, ComputePlane or not. - -The hardening's real and sufficient scope is **intra-cluster**: it protects the tenant's environment from the tenant's *own* app users — JupyterHub students, LLM-generated code, n8n flows — who are confined to whichever cluster the tenant deployed, and from the tenant's own misconfiguration. Combined with separation of responsibility, that is a genuine managed-service guarantee (safe defaults the tenant cannot accidentally undo, on a cluster the operator keeps correct), just not a multi-tenant or management-plane boundary. - -### What ComputePlane actually changes - -It changes **where operator-shipped code-executing catalog apps run**: from shared-kernel pods in the tenant namespace on the management cluster (today's only delivery path) to an operator-controlled VM cluster the tenant does not administer. The isolation that makes that safe is the pre-existing managed-Kubernetes + VM + tenant-egress model; ComputePlane is the routing and the managed, safe-by-default packaging that lets an operator offer such apps from the catalog without putting untrusted user code on the management nodes. There is no known exploit being patched in the management plane — this is about how the operator *delivers* code-executing catalog apps, plus defense in depth for the app's own users. +The one thing a ComputePlane has that a tenant-run cluster does not is **tamper-proof hardening** (the tenant is not cluster-admin and owns none of the values, so PSA / network policy / admission cannot be stripped). This is **not** a platform boundary against the tenant: an attacker holding a management-hijacking payload simply provisions a `standard` managed `kind: Kubernetes` (same substrate, one click, but they are admin and unhardened) and runs it there — the hardened venue is optional for them. The substrate (per-VM guest kernel + `-egress`) is what contains arbitrary tenant code regardless. The hardening's real, sufficient scope is **intra-cluster**: protecting the tenant's environment from the tenant's *own* app users (JupyterHub students, LLM-generated code, n8n flows) and from the tenant's own misconfiguration. The visibility/mutation control (Design §6) is the enforcement layer for that scope. ### Visibility -Tenant *visibility* of the ComputePlane is a separable UX/operability default, not a security mechanism — an opaque cluster is no safer than a visible one. Today Kamaji provisions an admin kubeconfig held by cluster-admins, so operators can debug a stuck workload; a tenant-facing **scoped read/observability** view (logs, events, `describe` of the tenant's own workloads) is a worthwhile extension so users are not operating a black box, and the security model allows it. Full opacity is a default, not a requirement. +Tenant *visibility* is a separable UX/operability default, not a security mechanism — an opaque cluster is no safer than a visible one. A tenant-facing **scoped read/observability** view (logs/events/`describe` of their own workloads) is allowed and worthwhile so users aren't operating a black box; full opacity is a default, not a requirement. ## Failure and edge cases -- **ComputePlane not yet ready when a `placement: ComputePlane` app is created** → the generated `HelmRelease` reconciliation waits on the kubeConfig Secret; Flux surfaces a not-ready condition on the HelmRelease status, same as any dependency-ordering today. -- **ComputePlane kubeconfig Secret missing/rotated** → remote apply fails closed (no fallback to local apply on the management cluster); status reflects the error. Failing closed is the security-correct behavior. -- **App declares `placement: ComputePlane` but the tenant has no ComputePlane** → reject at admission with a clear status error. **Never climb to an ancestor's ComputePlane** — that would breach single-tenant isolation (Design §2). -- **GPU exhaustion** → cluster-autoscaler adds GPU node groups up to `maxReplicas`; beyond that the workload pends, as in any autoscaled cluster. -- **Tenant deletion** → the ComputePlane and its workloads are torn down with the tenant; the remote `HelmRelease`s must be deleted *before* the ComputePlane is deprovisioned. This ordering is enforced by a finalizer on the ComputePlane resource that blocks teardown until its remote `HelmRelease`s are cleaned up — otherwise Flux's `HelmRelease` finalizers would block indefinitely once the target API is gone. +- **ComputePlane not ready when a `placement: ComputePlane` app is created** → the `HelmRelease` waits on the `computeplane-admin-kubeconfig` Secret; Flux surfaces not-ready, as with any dependency ordering. +- **kubeconfig Secret missing/rotated** → remote apply fails closed (no fallback to local apply); the security-correct behavior. +- **`placement: ComputePlane` app but the module is disabled on the tenant** → reject at admission. Never climb to an ancestor's ComputePlane (Design §7). +- **GPU exhaustion** → cluster-autoscaler adds GPU nodes up to the configured `maxReplicas`; beyond that the workload pends. +- **Tenant / module deletion** → remote `HelmRelease`s must be deleted *before* the cluster is deprovisioned (a finalizer on the module blocks teardown until they're cleaned up) — otherwise Flux's HelmRelease finalizers block once the target API is gone. ## Testing -- **Unit:** the app→HelmRelease conversion correctly injects `spec.kubeConfig.secretRef` (and `spec.install.createNamespace`) when `placement == ComputePlane`, and omits the kubeConfig for `placement: ManagementPlane`. -- **Integration (kind, two clusters):** a HelmRelease on cluster A with a kubeConfig for cluster B applies the chart on B and nowhere on A. -- **Security assertions (e2e):** from a pod on the ComputePlane, the management kube-apiserver is unreachable (network) and unauthenticated (no token); a scoped per-service egress policy permits reaching only the allowlisted tenant Service and nothing else; no Secret in the ComputePlane contains management credentials; a workload-triggered node panic is contained to one KubeVirt VM. -- **E2E (real cluster):** create a tenant with the `computePlane` module set, create a JupyterHub app, confirm pods land on the ComputePlane, confirm the app is reachable through the tenant's ingress, confirm the tenant holds no admin kubeconfig for it; confirm a sibling/child tenant cannot deploy onto this ComputePlane. +- **Unit:** app→HelmRelease conversion injects `spec.kubeConfig.secretRef` (+ `createNamespace`) for `placement: ComputePlane` and omits it for `ManagementPlane`; `cozystack-api` hides/blocks restricted apps for `regular` subjects and not for `privileged`. +- **Chart:** `helm template` of `extra/computeplane` renders a `HelmRelease` whose `chartRef` is the source-only `apps/kubernetes` `ExternalArtifact` and whose security-relevant values (PSA, egress deny, withheld kubeconfig) are present and not overridable from tenant input. +- **Integration (kind, two clusters):** a HelmRelease on cluster A with a kubeConfig for B applies on B and nowhere on A. +- **Security (e2e):** from a ComputePlane pod, the management kube-apiserver is unreachable + unauthenticated; only the allowlisted tenant Service is reachable; no Secret holds management creds; a workload-triggered node panic is contained to one VM. +- **E2E (real cluster):** enable the `computeplane` module on a tenant, create a JupyterHub app with `placement: ComputePlane`, confirm pods land on the ComputePlane + the app is reachable via tenant ingress + the tenant holds no admin kubeconfig + a regular tenant user cannot see the restricted app; confirm a sibling/child tenant cannot deploy onto it. ## Rollout -1. **Phase 1 — core primitive.** Ship the ComputePlane as a managed-Kubernetes-backed Tenant module (one ComputePlane per enabling tenant, single-tenant by design), plus the `placement` routing enum in the app→HelmRelease path and the scoped per-service egress policy of Design §5. -2. **Phase 2 — first consumer (`cozyllm`).** Set `placement: ComputePlane` on the code-executing apps (JupyterHub, n8n, ComfyUI, Langflow, code-exec features of Open WebUI); keep vLLM/LiteLLM at the default `placement: ManagementPlane` (inference-only, no code execution). See the cozyllm-specific technical design. -3. **Phase 3 — extensions (deferred).** Tenant-facing scoped read/observability view of the ComputePlane; additional consumers (WordPress / application-platform); billing/metering hooks; secret delivery of managed-service connection strings into ComputePlane workloads. +1. **Phase 1 — core primitive.** The `extra/computeplane` module + `computeplane-rd` + `computeplane-application` PackageSource (source-only `apps/kubernetes` re-source), the `placement` field on `ApplicationDefinition` with `ManagementPlane | ComputePlane` routing, and the per-kind visibility/mutation control. +2. **Phase 2 — first consumer (`cozyllm`).** Require the `computeplane` module and set `placement: ComputePlane` on the code-executing apps (JupyterHub, n8n, ComfyUI, Langflow, code-exec Open WebUI); keep vLLM/LiteLLM on `ManagementPlane`. +3. **Phase 3 — extensions (deferred).** Per-instance/label visibility filtering; a *trusted*/`cluster-meshed` module variant and/or `placement: `; billing/metering; managed-service credential delivery into sandbox workloads; ephemeral per-task runtimes (gVisor RuntimeClass) as a module option. ## Open questions -- **Naming.** `ComputePlane` is chosen (parallels "control plane"). Alternatives considered and set aside: `Compute Dataplane`, `Isolated Compute`. The kind must stay clearly distinct from `kind: Kubernetes` to avoid user confusion. -- **Capacity sharing vs. cluster sharing.** Single-tenant *clusters* are settled (Design §2). The open optimization is sharing the underlying **node pool / capacity** across tenants' ComputePlanes (infra-layer, acceptable) — how that is modelled without ever sharing an isolation domain. -- **Placement target.** The routing mechanism (`HelmRelease` + `kubeConfig.secretRef`) is generic — `placement` could name *any* target cluster, not only a dedicated ComputePlane. Two shapes worth weighing: **(a)** let a tenant target their **own** existing managed `Kubernetes` cluster — appealing (visibility for free, no second cluster) but the tenant holds full admin there, so we cannot guarantee SLA, automate updates, or keep platform hardening tamper-proof; **(b)** a `spec.managedDataplane: true` mode on the `Kubernetes` app where Cozystack withholds the admin kubeconfig — effectively the ComputePlane as a *mode* of the managed-Kubernetes app, recovering the managed guarantees. Either way the dependencies provisioned into such a cluster (ingress-nginx, cert-manager, …) need defining. Open which to adopt; the design should not hard-code ComputePlane as the only non-management target. Framed this way, a ComputePlane is essentially `kind: Kubernetes` + `managedDataplane: true` + a hardened profile + placement routing; keeping it a distinct *user-facing* kind is a UX choice (don't conflate "a cluster I administer" with "the managed environment my apps run on"), even if the implementation is a mode of the managed-`kubernetes` app. -- **Scoped tenant observability.** What read access the tenant gets to their own ComputePlane workloads (logs/events/describe) and how it is delivered (a scoped read-only kubeconfig vs. surfacing through the dashboard) — see Security → Visibility. -- **Per-service authorization (Design §5).** Who authorizes a ComputePlane→tenant-service path and how the `CiliumNetworkPolicy` is generated. -- **Credential delivery.** How a managed Postgres (created in the tenant) delivers its connection secret into a workload running on the ComputePlane (the network reachability is handled by Design §5; this is the remaining secret-plumbing half). -- **Billing.** Metering ComputePlane resources and (for LLM apps) API/token consumption — adjacent product work, noted so the design leaves room for it. +- **Module / chart naming** — `computeplane` for the `extra` chart, `-rd`, and PackageSource. (Note: the name `cozyplane` is already used by the unrelated SDN work on `sdn.cozystack.io`; keep these distinct to avoid confusion.) +- **Module toggle shape** — a bool (`computePlane: true`), or a profile-name string if the operator ships more than one module variant (e.g. `sandbox` vs a future `trusted`). +- **Which knobs the module exposes** vs fixes (Design §3) — GPU/type, size/autoscaling, k8s version are candidates; the security posture is always fixed. +- **Visibility model granularity** — is the visibility/mutation split worth exposing, or is a single `restricted` flag enough for v1? Start per-kind (plain RBAC), defer per-instance/label filtering. +- **Per-service egress authorization** (Design §5) — who authorizes a ComputePlane→tenant-service path and how the `CiliumNetworkPolicy` is generated. +- **Credential delivery** — how a managed Postgres delivers its connection secret into a ComputePlane workload (network reachability handled by §5; this is the secret-plumbing half). +- **Multiple ComputePlanes per tenant** — the default module is a single per-tenant enabler; if a tenant needs several, is that the advanced `placement: ` path, or multiple module instances? ## Alternatives considered -- **Harden containers in the tenant namespace (drop capabilities, no-privilege-escalation, restricted PSA).** Rejected as the primary boundary: hardening does not make container isolation a multi-tenancy boundary, and it breaks the very apps in scope — a heavily-restricted JupyterHub/agent loses its purpose, and many such workloads are stateful and behave badly in locked-down ephemeral sandboxes. -- **gVisor / sandboxed container runtime.** Rejected as the trust boundary for now: incomplete syscall coverage risks breaking workloads, maturity is insufficient to bet the isolation guarantee on, and it does not address the kernel-panic blast radius that VM isolation does. -- **Run each app directly in a VM via cloud-init (no Kubernetes inside).** Rejected: re-creates the lifecycle/reconcile/update machinery Kubernetes already provides ("you've just reinvented Kubernetes"); the kubelet-in-VM model gives the same VM boundary while keeping GitOps lifecycle and autoscaling. -- **Target the tenant's own managed `Kubernetes` cluster instead of a dedicated ComputePlane.** Attractive — no second cluster, and visibility comes for free — but the tenant holds full admin on their own cluster, so the platform cannot guarantee SLA, automate updates, or keep hardening tamper-proof. Captured as a placement-target option (Open questions) rather than the default; the `managedDataplane`-mode variant withholds the admin kubeconfig to recover the managed guarantees. -- **A node-to-node mesh (e.g. Kilo / the cross-cluster-tenant-mesh) for ComputePlane↔tenant connectivity.** Not needed: ComputePlane nodes already share the management Cilium pod network, so reachability is a per-service `CiliumNetworkPolicy` scoping problem (Design §5), which is also narrower than a node mesh — important precisely because the consumer here is untrusted code. -- **A single shared execution cluster for the whole install.** Considered; it concentrates GPU scheduling and reduces idle overhead, but it weakens per-tenant isolation. Rejected in favor of one single-tenant ComputePlane per enabling tenant (shared node-pool capacity remains an acceptable optimization). -- **Per-module `{ enabled, valuesOverride }` shape (à la tenant-module-overrides).** Rejected for this module: an inline override blob would duplicate the ComputePlane spec on every tenant that enables it, re-creating two sources of truth. A single-string profile reference keeps one definition. +- **A distinct `kind: ComputePlane` (the merged first revision).** Rejected per #26 / the #17 review: it is the same object as a managed `kind: Kubernetes`, and it duplicates the CRD/reconcile/RBAC surface for a variation that is fully expressible in packaging. +- **User-facing preset fields (`isolationProfile` × `componentProfile`) on `kind: Kubernetes` (revision #27).** Set aside in favor of the `extra`-module model. Putting the posture on the app object adds user-facing API surface for a choice that is really the operator's, invites a tenant-editable tamper surface (the whole value of a sandbox is that the tenant *cannot* change it), and cuts against the `apps` vs `extra` split ([website#594](https://github.com/cozystack/website/pull/594)) — a ComputePlane is an enabler the tenant switches on, not a cluster they configure. Baking the posture into an operator-owned module chart keeps `kind: Kubernetes` untouched, makes the hardening tamper-proof by construction, and reuses the existing Tenant-module + source-only-PackageSource machinery. (The presets' one advantage — several composable postures per tenant — is retained where it is actually wanted via separate module variants and the deferred `placement: ` path.) +- **Single-string `computePlane:` tenant module rendered inline in `apps/tenant/templates` (first-revision implementation).** Right delivery shape, wrong packaging: the cluster HelmRelease was inlined into the tenant chart rather than shipped as a first-class `extra/computeplane` chart with its own PackageSource. This revision makes it a proper module (like `extra/seaweedfs`). +- **Harden containers in the tenant namespace.** Rejected as the primary boundary: hardening doesn't make container isolation multi-tenant, and it breaks the apps in scope. +- **gVisor / sandboxed runtime as the primary boundary.** Rejected as the *primary* boundary (incomplete syscall coverage, no kernel-panic blast-radius containment) — but valid as an *inner* layer for ephemeral per-task sandboxes within a cluster; tracked as a future runtime option. +- **Run each app directly in a VM via cloud-init.** Rejected: re-invents Kubernetes lifecycle; kubelet-in-VM gives the same boundary with GitOps + autoscaling. +- **A single shared execution cluster for the whole install.** Rejected: weakens per-tenant isolation; shared node-pool capacity remains an acceptable optimization. --- From f17a27bf83218870233725b8144322779b76dc6f Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Mon, 13 Jul 2026 12:59:55 +0200 Subject: [PATCH 2/2] docs(design-proposals): correct the 'not a new kind' framing for the module MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Tenant module is registered via an ApplicationDefinition, so ComputePlane does get its own apps.cozystack.io kind and API — the 'not a new top-level kind' claim carried over from #27 is inaccurate for this delivery. Reframe honestly: no new CRD, no new controller/reconcile path, and no fields on kind: Kubernetes; but a thin ComputePlane module-kind is registered like every other managed service. Assisted-By: Claude Signed-off-by: Andrei Kvapil --- design-proposals/compute-plane/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/design-proposals/compute-plane/README.md b/design-proposals/compute-plane/README.md index afb57a5..400a537 100644 --- a/design-proposals/compute-plane/README.md +++ b/design-proposals/compute-plane/README.md @@ -5,7 +5,7 @@ - **Author(s):** `@kvaps` - **Date:** `2026-06-23` - **Status:** Draft -- **Revision (this PR):** Supersedes the preset-field revision (#27). ComputePlane is **not** a new top-level kind (agreeing with #26 / the #17 review) **and not** a set of user-facing preset fields on `kind: Kubernetes`. Instead it is a Cozystack-owned **Tenant module** (`packages/extra/computeplane`) that, under the hood, deploys the ordinary `apps/kubernetes` chart with operator-fixed values, sourced through the existing PackageSource "source-only chart" mechanism (the same one NATS and SeaweedFS use — only the wrapped chart comes from `apps/` instead of `system/`). The tenant gets the *same* `kind: Kubernetes` cluster, but owns none of its settings — only the knobs the operator chooses to expose. The mechanism settled in the first revision — remote Flux apply onto a Kamaji+KubeVirt cluster, untrusted code behind a per-VM kernel boundary — is unchanged; this revision is only about the **delivery surface**. +- **Revision (this PR):** Supersedes the preset-field revision (#27). ComputePlane is delivered as a Cozystack-owned **Tenant module** (`packages/extra/computeplane`) that, under the hood, deploys the ordinary `apps/kubernetes` chart with operator-fixed values, sourced through the existing PackageSource "source-only chart" mechanism (the same one NATS and SeaweedFS use — only the wrapped chart comes from `apps/` instead of `system/`). Like every Cozystack managed service, the module registers its **own `apps.cozystack.io` kind** (`ComputePlane`, via an `ApplicationDefinition` with `dashboard.module: true`) — its own API endpoint and input schema, served by `cozystack-api` and converted to a HelmRelease. So this is **not** the literal "no new kind" surface #27 aimed for; the honest positioning is: no new **CRD**, no new **controller / reconcile path**, and **no fields added to `kind: Kubernetes`** — the `ComputePlane` kind is a thin operator-owned wrapper over the unchanged `apps/kubernetes`. The tenant gets the *same* `kind: Kubernetes` cluster but owns none of its settings — only the knobs the operator exposes. The isolation mechanism (remote Flux apply onto Kamaji+KubeVirt, untrusted code behind a per-VM kernel boundary) is unchanged from the merged first revision; this revision is about the **delivery surface**. ## Overview @@ -13,7 +13,7 @@ Cozystack's tenant model treats a managed application as a single-purpose servic A growing class of applications breaks that property by design: their core feature *is* arbitrary code execution (notebooks, workflow "code" nodes, plugin systems, custom Python components). An operator who wants to *offer* these from the catalog has, today, only one delivery path — deploy them as ordinary pods into the tenant namespace **on the shared management cluster**, where they run on the management nodes' shared host kernel. For code-executing apps that is the unsafe part: a kernel-level container escape — the recurring vulnerability class, e.g. Copy Fail (CVE-2026-31431) — turns the app's untrusted user code into root on a management node and across every co-located tenant. Cloud platforms answer this by running untrusted compute behind a virtualization boundary (the reason Kata, gVisor and sandboxed runtimes exist); Cozystack already runs *tenant-owned* compute that way, on KubeVirt-VM clusters. ComputePlane brings catalog apps onto that same boundary instead of onto the shared management nodes. -**ComputePlane is neither a new kind nor a new setting on `kind: Kubernetes`.** From the management plane's view, the cluster a ComputePlane runs on and a regular managed `kind: Kubernetes` are *the same object* — Kamaji control plane, KubeVirt-VM workers whose `virt-launcher` pods sit in the tenant namespace under the existing egress policy, operator-held kubeconfig. So ComputePlane needs no CRD and no new API surface. It is a **packaging** of that existing app: a per-tenant enabler module the operator switches on, which provisions one hardened `kind: Kubernetes` cluster with values the operator controls. +**The cluster a ComputePlane runs on is the same object as a regular managed `kind: Kubernetes`** — Kamaji control plane, KubeVirt-VM workers whose `virt-launcher` pods sit in the tenant namespace under the existing egress policy, operator-held kubeconfig. ComputePlane adds no CRD and no controller for that cluster; it is a **packaging** of the existing app — a per-tenant enabler module the operator switches on, which provisions one hardened `kind: Kubernetes` cluster with values the operator controls. Delivered as a Tenant module, ComputePlane *does* register its own `apps.cozystack.io` kind (`ComputePlane`, `module: true`) — its own API endpoint and input schema — the same CRD-free way `SeaweedFS`, `Kubernetes` and every managed service is registered. That kind is a thin wrapper over `apps/kubernetes`, not a second cluster implementation, and it leaves `kind: Kubernetes` itself untouched. Two consequences follow, and this is the whole point of the revision: @@ -29,7 +29,7 @@ The capability is generic and intended to live in Cozystack core as a reusable p ## Scope and related proposals -- **#26 (@myasnikovdaniil)** and the **#17 review (Timofei Larkin)** established that ComputePlane must not be a distinct kind — the cluster it runs on is the same object as a managed `kind: Kubernetes`. This revision keeps that conclusion but delivers the posture as an **operator-owned module wrapping that app**, rather than as user-facing preset fields on the app (see [Alternatives](#alternatives-considered) for why the field model was set aside). +- **#26 (@myasnikovdaniil)** and the **#17 review (Timofei Larkin)** argued ComputePlane should not be a distinct kind — the cluster it runs on is the same object as a managed `kind: Kubernetes`. This revision *partly departs* from that: it does register a `ComputePlane` module-kind (an `ApplicationDefinition`, not a CRD), but keeps the underlying substrate as the unchanged `apps/kubernetes` and puts the config in an operator-owned module rather than in user-facing fields on the app (see [Alternatives](#alternatives-considered) for the trade-off vs #27's field model). - **[website#594](https://github.com/cozystack/website/pull/594)** (`apps` vs `extra`): the split this design leans on — `apps` = first-class services the user orders; `extra` = per-tenant enabler modules under the hood. ComputePlane is `extra`. - **`design-proposals/cross-cluster-tenant-mesh`** (PR #7): the trust model for managed clusters (one-way host → tenant, no host kube-API). A *trusted* variant of the module could wire the cluster into that data-plane mesh; the default `sandbox` module deliberately does **not** — only narrow per-service egress (Design §5). - **`design-proposals/kubernetes-nodes-split`** / **`kubernetes-nodes-hybrid-clusters`** (PR #8/#9): the substrate is the existing managed-`kubernetes` app (Kamaji + CAPI/KubeVirt); node-provisioning changes apply transparently. @@ -68,7 +68,7 @@ The platform already isolates *tenant-owned* untrusted compute (a managed `kind: - Does **not** make Kubernetes multi-tenant or claim container isolation suffices; it places catalog apps on the existing VM-isolated substrate. - Does **not** share a cluster across tenants (parent or child). Single-tenant **by design** (Design §7). Sharing the **node pool / capacity** is fine; sharing a **cluster** is not. - Does **not** present invisibility, or the tamper-proof hardening, as a *platform* security boundary (it protects the tenant from their own app's users; it is not what keeps the management plane safe — see Security). -- Does **not** introduce a distinct top-level CRD for ComputePlane, nor new user-facing configuration fields on `kind: Kubernetes` (the central change in this revision — the configuration lives in the operator-owned module chart, not on the API object). +- Does **not** introduce a new **CRD** or controller for ComputePlane, nor new user-facing configuration fields on `kind: Kubernetes` (the configuration lives in the operator-owned module chart). It **does** register a `ComputePlane` `apps.cozystack.io` module-kind via its `ApplicationDefinition` — the standard CRD-free registration every managed service uses; that is intended, not avoided. - Does **not** propose gVisor as the *primary* boundary (Alternatives) — though gVisor is a valid *inner* layer for ephemeral per-task sandboxes within a cluster (it sidesteps Kata nested-virt inside KubeVirt VMs), tracked as a future runtime option. ## Design @@ -99,7 +99,7 @@ spec: Why a module wrapping the app, rather than a new kind or new fields on the app: -- **Same object, no new surface.** From the management plane a hardened compute cluster and a regular managed cluster share all machinery (Kamaji, KubeVirt nodes, autoscaler, addon Flux apply). A distinct kind, or a `sandbox` field on `kind: Kubernetes`, would add API surface for something that is fully expressible as *a set of values the operator picks*. Keep the app's API untouched; put the choice in the wrapping chart. +- **Same substrate, thin surface.** From the management plane a hardened compute cluster and a regular managed cluster share all machinery (Kamaji, KubeVirt nodes, autoscaler, addon Flux apply). The module registers its own `ComputePlane` kind (like any managed service), but it adds no CRD, no controller and no second cluster implementation — and it leaves `kind: Kubernetes` itself untouched (no `sandbox`/preset fields on it). The variation is *a set of values the operator picks* in the wrapping chart, not new config on the app's API. - **`apps` vs `extra` says `extra`.** The tenant does not *order* a ComputePlane the way they order a Postgres or a Kubernetes cluster — it is switched on to *enable* a catalog of code-executing apps. Per [website#594](https://github.com/cozystack/website/pull/594) that is precisely an `extra` module. - **Tamper-proof by construction.** The tenant never receives the admin kubeconfig and never sees a `Kubernetes` CR they can edit, so the hardening in the operator-owned `values` cannot be weakened. There is no user-facing field to reconcile against a profile — the chart *is* the profile. - **The user still gets a real `kind: Kubernetes`.** It is the same cluster they would get from the catalog, with the same lifecycle, autoscaling and GPU support — they simply do not administer it and can change only what the module exposes. @@ -178,7 +178,7 @@ The module, the `placement` field, and the visibility control live in **Cozystac - **App authors:** set `placement: ComputePlane` on a code-executing `ApplicationDefinition` (default `ManagementPlane`). Optionally mark an app restricted (Design §6). - **Operators:** enable the `computeplane` module on a tenant (as with any other Tenant module); optionally set the curated knobs the module exposes (GPU, size). No cluster to hand-assemble. - **Tenant users:** *no change* to creating an app; they don't administer the ComputePlane and (for restricted apps) may not see it. -- **API shape:** new `placement` field on `ApplicationDefinition`; a new `computeplane` Tenant module (`extra/computeplane` + `computeplane-rd` + `computeplane-application` PackageSource); a restricted-app marker + tenant-role tier consumed by `cozystack-api`. **No new CRD, and no new fields on `kind: Kubernetes`.** +- **API shape:** a new `ComputePlane` `apps.cozystack.io` module-kind (its own `ApplicationDefinition` / input schema, `dashboard.module: true`), served by `cozystack-api` like every managed service — **not a CRD**; a new `placement` field on `ApplicationDefinition`; the `extra/computeplane` + `computeplane-rd` + `computeplane-application` packaging; a restricted-app marker + tenant-role tier consumed by `cozystack-api`. **No new CRD/controller, and no new fields on `kind: Kubernetes`.** ## Upgrade and rollback compatibility @@ -236,7 +236,7 @@ Tenant *visibility* is a separable UX/operability default, not a security mechan ## Alternatives considered -- **A distinct `kind: ComputePlane` (the merged first revision).** Rejected per #26 / the #17 review: it is the same object as a managed `kind: Kubernetes`, and it duplicates the CRD/reconcile/RBAC surface for a variation that is fully expressible in packaging. +- **A distinct, heavyweight `kind: ComputePlane` with its own cluster implementation (the merged first revision).** The module-kind here still registers a `ComputePlane` kind, but as a thin `ApplicationDefinition` that wraps the unchanged `apps/kubernetes` — it does not duplicate the cluster reconcile/RBAC path, which was the #26 / #17-review objection to the first revision. (What #26 wanted — *no* new kind at all, only fields on `kind: Kubernetes` — is the #27 model, set aside in the next bullet.) - **User-facing preset fields (`isolationProfile` × `componentProfile`) on `kind: Kubernetes` (revision #27).** Set aside in favor of the `extra`-module model. Putting the posture on the app object adds user-facing API surface for a choice that is really the operator's, invites a tenant-editable tamper surface (the whole value of a sandbox is that the tenant *cannot* change it), and cuts against the `apps` vs `extra` split ([website#594](https://github.com/cozystack/website/pull/594)) — a ComputePlane is an enabler the tenant switches on, not a cluster they configure. Baking the posture into an operator-owned module chart keeps `kind: Kubernetes` untouched, makes the hardening tamper-proof by construction, and reuses the existing Tenant-module + source-only-PackageSource machinery. (The presets' one advantage — several composable postures per tenant — is retained where it is actually wanted via separate module variants and the deferred `placement: ` path.) - **Single-string `computePlane:` tenant module rendered inline in `apps/tenant/templates` (first-revision implementation).** Right delivery shape, wrong packaging: the cluster HelmRelease was inlined into the tenant chart rather than shipped as a first-class `extra/computeplane` chart with its own PackageSource. This revision makes it a proper module (like `extra/seaweedfs`). - **Harden containers in the tenant namespace.** Rejected as the primary boundary: hardening doesn't make container isolation multi-tenant, and it breaks the apps in scope.