design-proposal: public IPs as a first-class resource (PublicIP / PublicIPClaim / PublicIPClass)

[Timofei Larkin (lllamnyp)]

What

Adds a design proposal under design-proposals/public-ip-claims/ introducing public addresses as a first-class resource: a cluster-scoped PublicIP, a namespaced PublicIPClaim, a PublicIPClass, and a provisioner contract in the shape of CSI.

Status: stub. The problem statement, the primitives survey and the object model are worked out. The controller mechanics, the provisioner contract details and the rollout are deliberately sketched. It is opened early to settle the shape — is an address a resource? — before anyone writes code.

Why

A tenant cannot own a public address. One appears as a side effect of creating a Service type: LoadBalancer, and evaporates when that Service is deleted. So an address cannot be:

• held across a workload rebuild (delete the Service, lose the address — someone else may get it),
• moved from one workload to another during a cutover,
• quota'd (there is no object to count; the only lever is pre-carving a pool per tenant),
• gated (anyone who can create a Service can cause an address to be allocated — RBAC authorizes verbs on resources, not fields),
• enumerated ("which addresses do we own, and who holds them?" has no answer short of listing every Service in the cluster).

The experience we want is the AWS one: allocate an address, keep it, attach it, detach it, attach it to something else, release it when done.

This is a regression, not a wishlist

Cozystack has reserve-then-associate today, via kube-ovn: OvnEip is created independently and later bound by an OvnFip / OvnSnatRule / OvnDnatRule that references it by name. Migrating off kube-ovn drops that capability unless something replaces it. The question is not "should Cozystack gain an AWS-style EIP" but "on what terms does it keep the one it already has."

The model

The PersistentVolume pattern, applied to addresses:

storage	addresses	role
StorageClass	PublicIPClass	which pool, which provisioner, which announcer
PersistentVolume	PublicIP (cluster-scoped)	the address, with a claimRef and a reclaim policy
PersistentVolumeClaim	PublicIPClaim (namespaced)	"give me one" — the whole tenant-facing API
CSI driver	a provisioner contract	how a class's backend allocates, adopts, and pins

Two things fall out of the survey and are worth a reviewer's attention:

• An address source union (fromClass vs providerRef), directly analogous to PV's volume-source union. Self-allocating backends (MetalLB, Cilium, kube-vip) have no reservation concept anywhere, so our provisioner is the IPAM of record. Cloud backends (AWS, GCP) already hold the reservation with a stable handle and their own IAM, so the provisioner must adopt, not allocate. Surviving a backend where allocation isn't ours is the evidence the shape isn't bent around MetalLB.
• A Pin capability, which the entire model depends on: a backend that cannot be told which address to use can never attach a reserved one, so a class over it must reject claims outright rather than allocate an address it can never bind.

The sharp edge (please read the Security section)

The per-backend "pin this IP" annotation (metallb.io/loadBalancerIPs, lbipam.cilium.io/ips, …) is a privilege-escalation surface. A self-allocating backend has no concept of a reservation, so it will hand a tenant an address that another tenant has reserved but not yet attached. That theft window exists on MetalLB and Cilium alike.

The fix is the move Cozystack already makes elsewhere: turn the ungatable field into a reference to an RBAC-gated object. Tenants may only write network.cozystack.io/public-ip-claim (naming an object in their own namespace); the controller writes the raw pin annotation; a ValidatingAdmissionPolicy rejects any other principal writing one. Without that policy the model is advisory, so it belongs in the same release as the CRDs.

Notes for reviewers

• Adjacent to, but not dependent on, design-proposal: structured, additive external exposure for managed applications #29 (structured-external-exposure). That proposal restructures how an application requests exposure; this one is about what an address is. They meet at one point — an exposure entry ought to be able to name a claim instead of implicitly minting an address — but neither needs the other to land first, and this does not assume design-proposal: structured, additive external exposure for managed applications #29's shape.
• ExposureClass is touched on but not depended on. The class idea is right and survives here in some form; binding an address's lifetime to a Service is the part that structurally cannot serve this use case. Open question 4 defers the merge-or-separate call, since that kind's future is being re-examined independently.
• The datapath is out of scope. Whole-IP 1:1 NAT (all ports in, and the workload egressing as that address — today cozy-proxy's nftables rules) is a CNI concern tracked separately. This is the allocation half; that is the forwarding half. Allocation is useful on its own with a stock LoadBalancer Service.
• Nothing upstream is coming. KEP-1880 shipped ServiceCIDR/IPAddress and explicitly named "any generalization onto something like an IPAM API" a non-goal; loadBalancerIP was deprecated in 1.24 with no core replacement. Cluster API's IPAddressClaim/IPAddress is literally this pattern but has only ever been wired to Machine addressing — open question 3 asks whether we adopt it or merely copy the contract.

The test that is the proposal

Claim → bind → associate → delete the Service → the address is still held → associate it to a different workload → the same address comes back.

That single integration test is precisely what cannot pass today.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 748af431-21e2-49d6-b39d-a6bf6afb3aba

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch design/public-ip-claims

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request introduces a design proposal to treat public IPs as first-class resources in Cozystack, modeling them after the Kubernetes PersistentVolume pattern with PublicIP, PublicIPClaim, and PublicIPClass resources. The feedback highlights three key areas for improvement: updating the PublicIPClaim status schema to support dual-stack scenarios by using a list of IP objects, configuring MetalLB's IPAddressPool with autoAssign: false to prevent IP conflicts with standard services, and clarifying how the ValidatingAdmissionPolicy will securely identify the controller principal.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If a Dual stack claim binds to two separate PublicIP objects (as suggested in Open Question 5), the current status schema for PublicIPClaim is insufficient because publicIPName and address are single scalar strings.

To support dual-stack where one claim maps to multiple PublicIP resources, the status should be designed to accommodate multiple names and addresses (e.g., using a list of IP status objects).

Suggested change

	status:
	phase: Bound
	publicIPName: pip-203-0-113-7
	address: 203.0.113.7 # what the tenant reads, and puts in DNS
	status:
	phase: Bound
	publicIPs:
	- name: pip-203-0-113-7
	address: 203.0.113.7

[gemini-code-assist]

There is a potential conflict between the Cozystack provisioner and MetalLB's automatic allocation. If a single IPAddressPool covering the entire range is configured in MetalLB (with autoAssign: true by default), MetalLB will automatically allocate any free IP from this pool to standard Services that do not use claims.

Since MetalLB is unaware of unassociated PublicIP reservations, it could allocate a reserved (but currently unattached) IP to a standard Service, leading to IP conflicts or binding failures when the claim owner later attempts to associate it.

To prevent this, the proposal should specify that the MetalLB IPAddressPool managed by the provisioner must have autoAssign: false configured, or clarify how standard Services are prevented from stealing reserved IPs.

[gemini-code-assist]

Using a ValidatingAdmissionPolicy to reject any principal except the controller from writing the pin annotations is a robust security measure. However, identifying "the controller" securely in a ValidatingAdmissionPolicy can be challenging and brittle if based solely on hardcoded usernames or namespaces.

Consider specifying how the policy will securely identify the controller principal (e.g., by matching a specific ServiceAccount name and namespace, or utilizing a specific SubjectGroup) to ensure it is both secure and maintainable across different deployment environments.

[mattia-eleuteri]

A few reactions grounded in a production Cozystack deployment that is squarely the
world this proposal has to serve: MetalLB-only for LoadBalancer allocation
(Cilium LB-IPAM unused), a single shared, auto-assign pool spanning all
tenants, and cozy-proxy whole-IP 1:1 NAT as the datapath for tenant VMs.

Strong +1 on the shape, and especially on the security move as a standalone win

The PV/PVC/StorageClass framing is the right altitude, and the Security section
describes a hole that is live in our cluster today: our public pool is
auto-assign and cross-tenant, so anyone who can create a Service can write
metallb.io/loadBalancerIPs: <any address in the pool> and be handed an address
another tenant is relying on. The ValidatingAdmissionPolicy that forbids any
principal but the controller from writing a pin annotation is worth landing on
its own, independent of the rest of the model. Please keep it in phase 1 as you
say. One ask: the VAP must explicitly allowlist the platform's own controllers and
the GitOps reconcilers — in our cluster the system istio Services already carry
pool annotations written by non-tenant principals, and a naive policy would break
reconciliation.

The "regression, not a wishlist" framing overstates it for at least one adopter

kube-ovn is installed for us and the OvnEip/IptablesEip CRDs exist, but we run
zero OvnEip instances — the reserve-then-associate capability was never
actually adopted; the 1:1 datapath goes through cozy-proxy instead. So for us this
is a new capability, not one being lost. The motivation stands on its own
merits (hold / move / quota / enumerate / bill); I'd lean less on the regression
argument, since for a real deployment it isn't one.

Suggested additional open question: multiple public addresses per workload

Fine to defer the design, but I'd like it captured explicitly, because it
interacts with the allocation/datapath split. A workload may want more than one
public address (1 v4 + 1 v6, or 2 v4), and the three cases are not the same:

1. Dual-stack (v4 + v6): clean — one dual-stack Service, two PublicIPs, one
   claim. This is essentially open question 5.
2. Two same-family, ingress-only (DNAT N→1): both addresses DNAT to the same
   backend, egress via the primary. The secondaries are entry points, not
   owned addresses.
3. Two same-family, each with full 1:1 whole-IP semantics: the hard one.
   Whole-IP is inherently 1 public ↔ 1 internal address, so N owned addresses
   require N internal addresses on the workload — bringing up an interface (or
   address) per claim, plus guest-side config the platform can't perform
   (cloud-init/DHCP inside the guest). This lands squarely in the deferred datapath
   half.

Two API observations that this open question should keep in mind:

• Association is currently 1 claim ↔ 1 Service (the annotation names a single
  claim; "two Services reference one claim → reject"). Expressing "this VM holds N
  addresses" would need a workload to reference a set of claims. Open questions
  5 and 7 already circle this — they might collapse into one "a workload may bind N
  claims" question.
• The Pin "reject rather than half-support" principle should probably extend to
  this: a backend/datapath that can't attach N owned addresses to one workload
  should be able to say so (a declared capability), so a class refuses such a
  claim rather than reserving an address it can never land.

None of this blocks the core question — "is an address a resource?" is a clear yes,
and the single-address path is well worth landing first. Just flagging the
multi-attach case as an open question so the allocation/datapath boundary is drawn
with it in mind rather than around it.

[Timofei Larkin (lllamnyp)]

Thanks — this is the production grounding the proposal needed, and all three points landed. The revision just pushed reflects them.

VAP allowlist — agreed, and it's now explicit, but admission is only half of it. You're right that a naive "reject everyone but the controller" breaks reconciliation the moment a GitOps reconciler or a system Service writes a pool annotation. The policy now identifies the controller by its ServiceAccount (configured at install, not a hardcoded username) and denies the write only for principals outside an explicit, admin-configurable allowlist (controller SA + GitOps/system SAs and groups). It also covers the pool-select annotation (metallb.io/address-pool), not just the IP pin — naming the reserved pool is as dangerous as pinning an address.

But identity-keyed admission has a ceiling: anyone who can impersonate an allowlisted ServiceAccount — a cluster-admin can — sails past it. Rather than pretend otherwise, the design now stands on two layers. Layer 1 is that admission policy, which reduces the blast radius for tenants and namespaced principals — and it's still worth landing on its own, since the theft window it closes is live in any cluster running a shared auto-assign pool. Layer 2 is reconciliation: the IPAddress inventory is authoritative, so any collision — an impersonated write, a direct MetalLB edit, a misconfigured pool — surfaces as IPAddress.status.phase: Conflict with the offending Service named, and the rightful reservation is never silently overwritten. The cluster-admin case is stated as a trust boundary (they're the cluster's trust root; handing out a reserved address is within their authority, not an escalation), handled by detection rather than a prevention promise the model can't keep.

Regression framing — dropped entirely, and further than "for you it's new." In Cozystack it isn't a regression at all: the platform uses kube-ovn only for subnets and VPCs, never its EIP objects, so there's no reserve-then-associate capability being lost. Anyone driving OvnEip deliberately is doing so outside Cozystack. The proposal now argues purely on merits — hold, move, quota, enumerate, bill.

Multiple addresses per workload — captured, but as explicitly deferred scope, not part of the core model. It's a step beyond "an address is a resource," and it's not clear the platform-owned case is even real. The hard sub-case you name — N owned same-family addresses, each needing its own internal address plus in-guest config the platform can't perform — is the one that looks orthogonal to reservation rather than an extension of it, since those extra addresses tend to live on interfaces the platform neither owns nor sees. So the model keeps 1 claim ↔ 1 Service, and the multi-attach case waits for a concrete platform-owned use case to shape it. Dual-stack (one v4 + one v6) is the one narrow multiplicity the model does admit — the claim status is a list from the start — but that's kept separate from the harder "N owned addresses" case.

Two changes worth flagging for your setup specifically. The kinds are renamed IPAddress/IPAddressClaim/IPAddressClass in ipam.cozystack.io — a class names a pool, and pools aren't necessarily public. And because you run MetalLB with a shared auto-assign pool, note that autoAssign: false on a reserved pool only stops automatic selection (a Service can still name the pool or pin an address), so the proposal adds an allocator-first reconciliation path: MetalLB assigns the moment it sees a Service, before any claim exists, and the controller reconciles those addresses back into the inventory toward eventual consistency rather than assuming every address began life as a claim. One honest gap comes with that: an eager allocator can still hand a plain Service an address that is already reserved-but-unattached, which reconciliation detects but cannot cleanly undo — a real fix likely waits for the per-provider provisioners that own allocation, and it's flagged in the doc as deferred to implementation.