From 59001ba6a21eb965f4c84ff7289537c97a7700cb Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 5 Aug 2026 13:35:22 -0500 Subject: [PATCH 01/15] docs: propose IP classes for workload address allocation Consumers name a class of address on a network interface -- public, tenant, IPv4, IPv6 -- and the platform returns one in the create response and tracks it until release. No pool, prefix length, region, or CIDR appears in a consumer manifest. Covers class configuration and inheritance, per-family resolution, where claims are made in the federated topology, instance address allocation, and a worked two-location example. --- docs/enhancements/ipam-integration.md | 797 ++++++++++++++++++++++++++ 1 file changed, 797 insertions(+) create mode 100644 docs/enhancements/ipam-integration.md diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md new file mode 100644 index 00000000..73fa616d --- /dev/null +++ b/docs/enhancements/ipam-integration.md @@ -0,0 +1,797 @@ +--- +status: provisional +stage: alpha +latest-milestone: "v0.x" +--- + +# Address space consumers can actually ask for + +**Status:** Draft +**Related:** [Federated Deployment Scheduling](federated-deployment-scheduling.md) (how a placement reaches the location that makes the claim) + +--- + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [What it feels like](#what-it-feels-like) + - [User Stories](#user-stories) + - [Notes/Constraints/Caveats](#notesconstraintscaveats) +- [Design Details](#design-details) + - [Class configuration](#class-configuration) + - [Address families](#address-families) + - [Where the address comes from](#where-the-address-comes-from) + - [Instance addresses](#instance-addresses) + - [A workload in two locations](#a-workload-in-two-locations) +- [What this depends on](#what-this-depends-on) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Open Questions](#open-questions) + +## Summary + +A consumer deploying a workload names the **class** of address it should get — +`public-unicast-ipv4`, `tenant-endpoint-ipv6` — and the platform returns one in the +create response, tracked from that moment until it is released. They never name a +pool, a prefix length, a region, or a CIDR. Operators define what each class means, +once, and can change what backs it without touching a consumer's manifest. + +The capability it unlocks is small to describe: **an address a workload keeps.** A +published endpoint that survives a redeploy, a stable outbound address a customer +can allowlist, an inventory an operator can query. + +## Motivation + +Addressing is the kind of decision that gets made implicitly the first time a +workload boots and then has to be migrated. Settling it while it is still a +question of API design costs a field; settling it afterwards costs a renumbering. + +Three things make an address something a consumer can rely on. + +**A way to express intent.** "Give this a public address," "keep this address when +I redeploy," and "make it IPv6" are one-line requests, and the interface a consumer +writes should carry them. The IPv6 point is the sharpest — the platform is +IPv6-first by design, so asking for it should be the easy path. + +**A system of record.** Who holds an address, when they got it, what happens when +they release it, how much of a kind of space is left. An on-call engineer and a +finance owner both ask these, and the answers are cheap to keep while allocations +are being made and expensive to reconstruct from the data plane afterwards. + +**A unit to govern.** Quota, budgets, and utilization want to reason about "public +addresses" as a thing, in terms a consumer would recognise — and a budget +introduced alongside a capability lands very differently from one introduced after +consumption is established. + +### Goals + +- Let a consumer request address space by naming a class, with no knowledge of + pools or topology. +- Let a consumer hold an address across a redeploy. +- Track every address the platform assigns to a workload, from allocation to + release. +- Give operators one inventory: what exists, what is used, who holds it, and how + much of each class is left. +- Make address space a unit that quota and utilization can reason about. + +### Non-Goals + +- **Fabric and infrastructure addressing.** Node loopbacks, routing locators, + underlay links, and the per-site blocks they come from are platform-internal and + have their own plan. They can move onto classes later; nothing here depends on + it. +- **Non-address numbering.** AS numbers, forwarding-instance identifiers, and MAC + assignments are allocatable resources with the same claim semantics, but a class + as designed here is prefix-shaped. They need a sibling model, not this one. +- **Globally-routable or consumer-owned tenant space.** Bring-your-own prefixes + and public tenant address space carry a mandatory validation and export regime + this design does not attempt to express. +- **Anycast.** A single address held by many locations at once is the inverse of + the rule every class here follows. Adding it later is additive: + `public-anycast-ipv4` joins the catalog beside `public-unicast-ipv4`, and nothing + already named changes. +- **Tracking addresses inside an endpoint.** An interface receives a block and + assigns within it, which keeps containers and secondary addresses off the + control-plane path. +- **Programming the data plane.** The platform decides addresses; other systems + install and advertise them. + +## Proposal + +Consumers ask for a *class* of address. Operators define what each class means. +The platform answers immediately and remembers. + +### What it feels like + +Deploying a sandbox that keeps its public address across redeploys: + +```yaml +apiVersion: compute.datumapis.com/v1alpha +kind: Workload +metadata: + name: hello-sandbox +spec: + template: + spec: + runtime: + sandbox: + containers: + - name: app + image: ghcr.io/datum-cloud/hello-unikraft:latest + networkInterfaces: + - network: { name: default } + ipFamilies: [IPv6, IPv4] # dual-stack; IPv6 is primary + reclaimPolicy: Retain # keep the addresses across redeploys + addresses: + - class: public-unicast-ipv4 # a class, never an address + # (omit for ordinary private addressing) + placements: + - name: default + locations: [us-central-1] + scaleSettings: { minReplicas: 1 } +``` + +Three lines are new, and none mention a pool, a prefix length, a CIDR, or which +site serves the location. Asking for both families is one list. An interface that +wants ordinary private addressing names no class at all. + +The result appears on the instance: + +```console +$ kubectl get instance hello-sandbox-default-us-central-1-0 -o yaml +status: + networkInterfaces: + - addresses: + - family: IPv6 address: fd20:a1b:2c3d:1:0:1::/96 primary: true + - family: IPv4 address: 10.128.0.2/32 + external: + - family: IPv4 address: 198.51.100.11 + conditions: + - type: Allocated status: "True" + - type: Programmed status: "True" +``` + +Both conditions matter. `Allocated` means the platform assigned the address; +`Programmed` means the network can carry it. They are separate because allocation +is synchronous and programming is not, and an interface must not report ready on +allocation alone. + +Every address there is a tracked allocation, so the questions above have answers: + +```console +$ datumctl ipam address show 198.51.100.11 + class: public-unicast-ipv4 + claimed by: Instance hello-sandbox-default-us-central-1-0 (uid 4f2a…) + project acme/app-team + policy: Retain — survives instance deletion +``` + +And operators get an inventory: + +```console +$ datumctl ipam class list +NAME FAMILY UNIT BACKED BY USED WORST LOCATION +tenant-endpoint-ipv6 IPv6 /96 12 pools 41,208 <1% +tenant-endpoint-ipv4 IPv4 /32 12 pools 38,104 61% us-central-1 +public-unicast-ipv4 IPv4 /32 1 pool 148 58% us-central-1 +``` + +The last column is the number that matters. A class averaged across locations +always reads healthy; what pages someone is one location filling up, so the view +reports the worst occupant rather than the mean. + +### User Stories + +- **As a developer**, I ask for a public address by naming a class, and I keep it + when I redeploy — so the endpoint I published to customers stays valid. +- **As a developer**, I get IPv6 by default and add IPv4 to the same interface when + I need to reach something that has not moved yet. +- **As a platform operator**, I define what `public-unicast-ipv4` means once — + which space backs it, how it is advertised, what happens on release — and every + team consumes it by name. +- **As a platform operator**, I move a class onto new space by attaching a pool and + draining the old one, with no consumer change. +- **As an operator on call**, I can answer "who has this address" in one command. +- **As a governance owner**, address space appears in quota in terms people + recognise, so it can be budgeted like anything else. + +### Notes/Constraints/Caveats + +- **Allocation is synchronous.** A claim returns its address in the create + response. That is the property the service exists to provide. +- **A consumer never writes an addressing resource.** They name a class on a + workload or network; the platform creates and owns the claim. +- **An interface can hold several families at once**, each a separate address with + its own lifecycle, requested as one list. +- **All requested families must succeed.** A partial result is not published; the + interface reports which family could not be satisfied. +- **The platform is IPv6-first.** An interface that says nothing gets IPv6. +- **A class sets the default reclaim policy; an interface can override it.** +- **A retained address is still held and still counts** against its holder's + budget, or nothing pressures anyone to release it. + +## Design Details + +### Class configuration + +This section defines each field and the rules the allocator applies. The audience +is a platform operator authoring classes. + +Terminology, used consistently: + +- **Class**: the policy object naming a kind of address space. +- **Pool**: a block of capacity offering itself to one or more classes. +- **Claim**: a request for an address of a named class. +- **Allocation**: the record of an address handed out. +- **Context**: the network and location a claim is made for. + +The fields below extend the shipped `IPClass`. Everything already on it — +`provisioner`, `parameters`, `ipFamily`, `strategy`, `allowedPrefixLengths`, +`defaultPrefixLength`, `reclaimPolicy`, `visibility` — keeps its current meaning. + +```yaml +apiVersion: ipam.miloapis.com/v1alpha1 +kind: IPClass +metadata: + # The name consumers write. It carries the address family, and any property + # the consumer chooses between real alternatives — so public-unicast-ipv4 + # rather than public. + name: tenant-endpoint-ipv6 + + annotations: + # Marks this class as the default for its family. A claim naming no class + # gets the default for each family it requested. At most one class per + # family may carry this. + ipam.miloapis.com/is-default-class: "true" + +spec: + # The single address family this class hands out. Required. Immutable. + ipFamily: IPv6 + + # The class whose allocations this one carves from. Empty means allocations + # come from the pools that offer this class via IPPool.spec.classNames. + # Immutable — changing it strands every existing allocation outside its + # declared ancestry. + parentClassName: tenant-subnet-ipv6 + + # How many allocations of this class exist. + # PerClaim one per claim — what an interface address wants + # PerNetwork at most one per network + # PerNetworkLocation at most one per network per location + # Defaults to PerClaim. Immutable. + allocationScope: PerClaim + + # How far the allocator has to look to be sure an address is free. + # Platform compare against every allocation on the platform + # Location compare against allocations in the same location + # Network compare against allocations in the same network + # This is a search scope, not a guarantee — what the guarantee turns out to be + # depends on the parent. Defaults to Platform, the strictest. Immutable. + collisionDomain: Network + + # The sizes a claim of this class may request, and the size used when a claim + # asks for none. A fixed-size class sets min and max equal. + allowedPrefixLengths: { min: 96, max: 96 } + defaultPrefixLength: 96 + + # Positions in the parent this class must not allocate, counted in units of + # this class's own allocation size. Loosening is always safe; tightening + # strands allocations already sitting in newly-reserved positions and warns. + reservations: + leading: 1 # the subnet's gateway and its all-zeros address live here + trailing: 0 + + # What routing does with the address. Advertisement is stated separately for + # inside a location and beyond it, because the two are frequently opposite: + # a per-instance address is a distinct route within its location and must + # never appear outside it, while only the covering block leaves. + routing: + internal: None # None | Host + external: None # None | Aggregate + # An aggregate must be originated with a discard route. A class advertising an + # aggregate it cannot fully resolve blackholes the unallocated space inside it. + + # What happens to the address when its claim goes away. Delete releases it. + # Retain holds it against the claimant's identity so a replacement gets the + # same address back. A claim can override this. + reclaimPolicy: Delete + + # The allocator that satisfies claims of this class. Defaults to the + # platform's own, and is immutable. + provisioner: ipam.miloapis.com/native +``` + +**Pools gain two fields.** `location` names the location a pool serves, so a claim +made in one location reaches that location's space without anyone naming it; a +pool with no location serves everywhere. `parentPoolName` lets pools nest, which is +how a continent's block contains its locations' ranges and stays summarisable as +one route. A pool declaring a location is not eligible for a claim from a different +one, and an unlocated ancestor is never eligible in its child's place. + +**What the collision domain means.** It is how far the allocator looks before +handing an address out, and the parent decides how far is far enough. + +Both endpoint classes in the example below set `collisionDomain: Network`, and they produce +opposite results. An IPv6 endpoint is carved from a `/64` that belongs to one +network and no other, so comparing within that network is the only comparison that +could matter — nothing outside it can allocate from that space, and the result is +unique platform-wide. An IPv4 endpoint is carved from a range every network in the +location shares, so comparing within the network is a genuine narrowing: two +networks reach the same address and both keep it. + +So the field says how hard to look, not how unique the answer is. Setting it wider +than the parent requires is safe and wasteful; setting it narrower is how two +holders end up with one address, which is exactly what IPv4 wants and nothing else +does. + +**What a claim carries.** The collision domain, the allocation scope, and parent +resolution all key off the same two values, so the claim carries them explicitly: + +```yaml +kind: IPClaim +spec: + className: tenant-endpoint-ipv4 + networkRef: { name: default } + location: us-central-1 +``` + +Neither is written by a consumer. The network layer at the location supplies both, +because it is the one thing that knows them: it holds the deployment, so it knows +the location, and it resolved the interface's network reference before claiming. +Both are immutable — a claim whose network or location changed after allocation is +incoherent. + +A name is enough here, and nobody types an identifier. Every claim is already +scoped to the project it was made for, and a network name is unique within a +project, so `default` in one project and `default` in another are different +collision domains without anything extra being carried. That is the same scoping +every pool and allocation already uses. + +The one case a name does not settle is a network deleted and recreated under the +same name. It inherits its predecessor's space and its allocations, which is +usually what someone wants and occasionally not. Deciding otherwise means the +reference carries something stable across recreation rather than a name — worth +settling before retention ships, since that is where it starts to matter. + +A class using `collisionDomain: Network`, or an `allocationScope` naming the +network, cannot be satisfied by a claim that omits the reference. The claim is +rejected rather than falling back to a wider comparison, because a wider comparison +would look correct while refusing addresses the narrow one was meant to allow — and +the error would surface as unexplained exhaustion rather than a missing field. + +**Resolving a parent.** With `parentClassName` empty, the allocator takes every +pool offering this class, discards those whose family differs, discards those +declaring a different location, and picks among the rest by the class's strategy. +With `parentClassName` set, it finds the parent allocation whose scope keys match +this claim's context — for an endpoint claim on network `default` in +`us-central-1`, the `tenant-subnet-ipv6` allocation for that network and location. + +If the parent does not exist, the allocator creates it first, applying the parent +class's configuration. Creation cascades: a claim in a location a network has never +used creates that location's subnet, and a claim on a new network creates the +network's prefix too. + +**Two concurrency rules the cascade requires.** `allocationScope` is a uniqueness +constraint, not a lookup — two simultaneous claims for the same network and +location both observe no subnet and both try to create one, so it needs a partial +unique index on the scope keys, with the loser reading the winner's allocation +rather than failing. And a cascade takes a lock at every level, so the levels must +be locked in a deterministic order; without one, two cascades touching the same +chain from different directions deadlock. Chain depth is capped and cycles are +rejected at class-write time, not at claim time. + +**Rules the allocator enforces.** A class and its parent share an address family. +A class's prefix lengths are longer than its parent's. When a parent is exhausted, +the error names the level that ran out, not the level that was asked for. + +**Class health is computed, never stored.** A class reports whether any pool backs +it and how full its worst location is. Both are aggregates over pool status, read +at query time. They are deliberately not counters maintained during allocation: a +class is backed by many pools, so a class-level counter would be one row every pool +contends on — turning independent claims in different locations into a queue and +destroying the per-pool locking the service depends on. A counter also cannot +express "the worst location," which is the number that matters. + +### Address families + +**A class is single-family. The interface asks for families; the platform picks a +class for each.** + +The tempting alternative is one class spanning both families with sizes declared +per family. It is appealing until checked against the addressing plan, where the +two families are not the same kind of thing. + +A tenant endpoint in IPv6 gets a block carved from **that tenant's own prefix**, +unique to them, which the instance subdivides. In IPv4 it gets a single address +from a **location-wide range every tenant reuses**, with no sub-block, because IPv4 +scarcity makes per-tenant uniqueness impossible at scale. Different parent, +different collision domain, different hierarchy. That is not one class with two +sizes — it is two classes serving the same interface. + +So the interface names families, each family resolves to a class, and where the +consumer names none — the common case — the platform's default for that family +applies. A consumer names a class only for something non-default, and names one per +family if they want that in both. + +**The family belongs in the name.** Naming a class is choosing a family, so +`public-unicast-ipv4`, not `public`. The rule reaches past the family: `unicast` is +there because there is a real alternative a consumer could choose, and the two +behave visibly differently — a unicast address is one per instance per location, an +anycast address is one address live everywhere. Name the property where the +consumer is choosing between real alternatives, and leave it out where there is +only one. Tenant addressing is never advertised, so it carries no routing +qualifier. + +### Where the address comes from + +**One allocator, in the middle, for everything.** Not a copy per location. + +The tempting alternative is pushing allocation out to each location so it keeps +working alone. It does not pay for itself. The high-volume cases that seem to need +it are not ours — pod addresses belong to container networking. What is left is a +few addresses per interface at instance-creation rate. A copy per location would +mean a database at every location, two versions of the truth, and a reconciliation +problem nobody has scoped. + +It would also cost the thing this is for. One allocator is the only way to answer +"who has this address" across the platform, enforce quota on the real resource, and +report utilization honestly. + +The trade is worth stating plainly: **while the central service is unreachable, no +new addresses are handed out.** A location cannot start a new instance. It does not +touch live traffic — existing addresses keep working and routes keep being +advertised — and it is the same dependency instance creation already has on central +quota. If it proves to matter, the answer is a small pre-reserved buffer per +location: a cache, not a second allocator. + +**Claims are made where the work lands.** A consumer declares intent in their +project; that intent travels to the location the workload is placed at, and the +network layer there — which knows the location, the network, and the family — turns +it into a claim. That is why a consumer never writes a location: the system asking +already is one. + +Because that system claims on the consumer's behalf, two things must hold. Its +authority is bounded by the placements actually delivered to it — a claim is valid +only for a network and location it holds a deployment for. And the address is +attributed to the consumer's project, not to the platform identity that made the +call, or quota and ownership both attach to the wrong party. + +### Instance addresses + +Every address on an instance is a real allocation, and that is the change making +the rest of the design work. As things stand the runtime picks the address — under +the sandbox runtime it is simply the container platform's own address — and the +platform records it afterwards, if at all. An address chosen that way cannot be +held across a redeploy or counted against a budget. Reversing it — **the platform +decides the address, and the runtime is told** — is what turns an address into +something a consumer can ask for and keep. + +This is more tractable on the platform's own compute than it would be against a +third party. Every address in play is space the platform already owns, so there is +no external allocator to reconcile with and no address the platform records +without having issued. + +The unit is the interface, not the address. An interface gets a block and assigns +within it, so tracking is one record per interface rather than one per container. + +Three things follow. + +**A claim must be able to ask for a specific address.** A claim asks for a size, not +an address. Handing the same address back to a replacement, and recording an address +already in use, both need a claim that names one. Without it the retention +experience above does not work. + +**The runtime stops choosing.** An instance's address arrives with its interface +configuration rather than being invented at boot, and the container platform's own +address stops standing in for it. That is a change to the runtime contract, not +just to what the platform records. + +**Retention needs an identity that is not a name.** Instance names are composed +from workload, placement, location, and ordinal, and are reused freely — a deleted +workload and a new one under the same name produce identical instance names. So +retention binds to the instance's unique identity, and a late release carrying a +stale identity is rejected rather than honoured. The allocation records that +identity opaquely; nothing about the consumer's type system crosses into the +allocator. + +Retention also needs an expiry. An address held forever against a location's public +range takes that range out of service for everyone, so a retained allocation +carries a lease, keeps consuming its holder's budget while it lives, and can be +force-released by an operator with an audit record. + +### A workload in two locations + +A consumer runs one workload on one network in `us-central-1` and `eu-west-1`, two +replicas each, dual-stack, with a public address per instance. Here is everything +that exists, and where. + +#### The platform, authored once + +Five classes and the pools that back them. Consumers never see these objects; they +see the names. + +```yaml +# The tenant chain. A class names the class it carves from; the top of a chain +# names none and draws from a pool instead. +kind: IPClass +metadata: { name: tenant-network-ipv6 } +spec: + ipFamily: IPv6 + # No parentClassName — the top of a chain draws from the pools that offer it, + # here IPPool/tenant-v6. + allocationScope: PerNetwork # one prefix per network + collisionDomain: Platform + allowedPrefixLengths: { min: 48, max: 48 } + reclaimPolicy: Retain +--- +kind: IPClass +metadata: { name: tenant-subnet-ipv6 } +spec: + ipFamily: IPv6 + parentClassName: tenant-network-ipv6 + allocationScope: PerNetworkLocation # one subnet per network, per location + collisionDomain: Platform + allowedPrefixLengths: { min: 64, max: 64 } + reclaimPolicy: Retain # a location's subnet is never renumbered +--- +kind: IPClass +metadata: + name: tenant-endpoint-ipv6 + annotations: { ipam.miloapis.com/is-default-class: "true" } +spec: + ipFamily: IPv6 + parentClassName: tenant-subnet-ipv6 + collisionDomain: Network # the parent /64 is this network's alone, + # so this is still platform-unique + allowedPrefixLengths: { min: 96, max: 96 } + reservations: { leading: 1 } # the subnet gateway lives in the first block +--- +kind: IPClass +metadata: + name: tenant-endpoint-ipv4 + annotations: { ipam.miloapis.com/is-default-class: "true" } +spec: + ipFamily: IPv4 + # No parentClassName, and that is the whole IPv4 story: there is no per-network + # IPv4 space to carve, so an endpoint draws straight from the location's shared + # range. The IPv6 endpoint above sits three levels down its own chain; this one + # is a chain of one. + collisionDomain: Network # the shared range makes this a real + # narrowing — two networks reach the same + # address and both keep it + allowedPrefixLengths: { min: 32, max: 32 } + reservations: { leading: 2, trailing: 2 } +--- +kind: IPClass +metadata: { name: public-unicast-ipv4 } +spec: + ipFamily: IPv4 + # Also no parentClassName — public addresses come from the location's public + # pool, not from anything the network owns. + collisionDomain: Platform # routable, so unique everywhere + allowedPrefixLengths: { min: 32, max: 32 } + routing: { internal: Host, external: Aggregate } + reclaimPolicy: Retain +``` + +The pools. IPv6 tenant space is one root that networks carve from; IPv4 tenant +space and public space are per-location, nested so a continent summarises as one +route. + +``` +IPPool/tenant-v6 fd20::/20 classNames: [tenant-network-ipv6] + +IPPool/tenant-v4 10.128.0.0/9 +├── IPPool/tenant-v4-americas 10.128.0.0/12 +│ └── IPPool/tenant-v4-us-central-1 10.128.0.0/20 location: us-central-1 +└── IPPool/tenant-v4-emea 10.144.0.0/12 + └── IPPool/tenant-v4-eu-west-1 10.144.0.0/20 location: eu-west-1 + classNames: [tenant-endpoint-ipv4] + +IPPool/public-v4-us-central-1 198.51.100.0/24 location: us-central-1 +IPPool/public-v4-eu-west-1 203.0.113.0/24 location: eu-west-1 + classNames: [public-unicast-ipv4] +``` + +#### The consumer's project + +Two objects, both written by the consumer: + +```yaml +kind: Network +metadata: { name: default } +spec: + ipam: { mode: Auto } +--- +kind: Workload +metadata: { name: hello-sandbox } +spec: + template: + spec: + runtime: { sandbox: { containers: [{ name: app, image: … }] } } + networkInterfaces: + - network: { name: default } + ipFamilies: [IPv6, IPv4] + reclaimPolicy: Retain + addresses: + - class: public-unicast-ipv4 + placements: + - { name: americas, locations: [us-central-1], scaleSettings: { minReplicas: 2 } } + - { name: europe, locations: [eu-west-1], scaleSettings: { minReplicas: 2 } } +``` + +Three more objects appear in the project that the consumer did not write — their +network's own space, and its presence in each location it reaches: + +``` +IPPool/network-default fd20:a1b:2c3d::/48 +├── IPPool/network-default-us-central-1 fd20:a1b:2c3d:1::/64 location: us-central-1 +└── IPPool/network-default-eu-west-1 fd20:a1b:2c3d:2::/64 location: eu-west-1 +``` + +These are project-scoped, so the consumer can see what their network holds and how +much of it is used. The location subnets appear on first use — a location the +workload never runs in never gets one — and are never renumbered afterwards. + +Notice there is no IPv4 equivalent. IPv4 endpoints come from the location's shared +range directly, which is why the two families resolve different classes. + +#### Each location + +The deployment arrives by placement, and everything below it is created there: + +``` +us-central-1 eu-west-1 + WorkloadDeployment/…-americas WorkloadDeployment/…-europe + Instance/…-americas-us-central-1-0 Instance/…-europe-eu-west-1-0 + Instance/…-americas-us-central-1-1 Instance/…-europe-eu-west-1-1 + NetworkInterfaceClaim ×2 NetworkInterfaceClaim ×2 +``` + +Each interface claim carries the consumer's intent — network `default`, families +`[IPv6, IPv4]`, class `public-unicast-ipv4`, retain — and the network layer turns +each into three claims: one per family, plus the named public class. It supplies +the location itself, because it is one. + +#### What comes back + +Twelve allocations, each attributed to the instance holding it: + +``` +us-central-1 + …-americas-us-central-1-0 fd20:a1b:2c3d:1:0:1::/96 10.128.0.2/32 198.51.100.11 + …-americas-us-central-1-1 fd20:a1b:2c3d:1:0:2::/96 10.128.0.3/32 198.51.100.12 +eu-west-1 + …-europe-eu-west-1-0 fd20:a1b:2c3d:2:0:1::/96 10.144.0.2/32 203.0.113.10 + …-europe-eu-west-1-1 fd20:a1b:2c3d:2:0:2::/96 10.144.0.3/32 203.0.113.11 +``` + +Allocation starts at the second block of each subnet because the first holds the +gateway and the subnet's all-zeros address. And **a public address is per instance, +per location** — four replicas means four routable addresses out of two locations' +space, which is a cost and a quota consequence that belongs in the request rather +than in a later discovery. + +The addresses land on each instance and travel back to the project control plane, +which is the only place the consumer looks. + +#### What the shape shows + +The IPv6 address is carved from a `/64` belonging to this network alone; the IPv4 +address comes from a `/20` every network in that location draws from. Same +interface, same request, two genuinely different arrangements — which is why each +family resolves its own class. + +That difference is the collision domain: **an IPv6 endpoint block is unique +platform-wide because the network's prefix is; an IPv4 address is compared only +within its network.** Two networks can hold `10.128.0.2` in `us-central-1` at once +and never meet, because the routing domain separates them — and reaching across +locations comes from that same routing domain, for both families, not from the +prefixes being contiguous. + +It also carries a ceiling worth stating as a product fact: a network cannot exceed +roughly four thousand IPv4 endpoints in one location, because every network draws +from the same location-wide range. IPv6 has no comparable limit. + +Removing a placement releases the addresses its instances held, but not the +location's subnet — that belongs to the network, and other workloads on it draw +from the same subnet. + + +## What this depends on + +Allocating an address is necessary and nowhere near sufficient. These are the pieces +the design assumes and does not provide. Listing them is the point — a design that +quietly assumed them would look finished and behave otherwise. + +**A network needs one routing identity across every location it reaches**, unique +platform-wide, or the two halves of a multi-location workload are unrelated networks +sharing a name. This is the identity scoping route import and export — not the +per-location forwarding-instance identifier, which is deliberately reused in every +location and is a much smaller space. Conflating the two would cap the platform at a +few thousand networks in total rather than per location. + +**A moved instance needs its old route withdrawn before the new one is trusted.** +Each node advertises with a distinct identity, so a route reflector keeps both +advertisements when an instance moves, and traffic splits between the node that has +it and the node that does not. Retention makes this worse by keeping the address +valid across the move. The routes need a sequence number so the newer advertisement +demonstrably wins. + +**Endpoint reachability has a per-node cost quadratic in network size.** Reaching a +remote endpoint installs per-endpoint state on every node that talks to it, and that +state is not reclaimed automatically. It is the real ceiling — far below any +address-space limit — and it needs a stated budget and a cap on endpoints per +network per node. + +**Subnets need programming, not just allocation.** A location's subnet appearing on +first use currently means a record is written; nothing provisions the gateway, the +forwarding instance, or the route-table entry. That is why the interface reports +`Allocated` and `Programmed` separately. + +**The gateway should be a real allocation.** It is currently a hole — space that is +reserved, owned by nothing, and configured by nothing. Making it an allocation held +by the subnet gives it an owner, lets it be programmed, and gives path-MTU discovery +a source address inside the network. Without one, oversized packets are dropped +silently: handshakes succeed and large transfers hang. + +**An endpoint's block is only reachable at its first address.** The block a class +hands an interface is flattened to a single address before distribution, so an +address self-assigned inside it works locally and nowhere else. Until distribution +preserves the block, "assigns within it" is aspirational. + +**Public addresses need a path to the instance** — advertisement, in-location +steering to the node holding it, and translation — and it has to move when the +instance is rescheduled. A released public address also needs a quarantine before +reissue: no route changes when it is handed to someone else, but DNS caches, +customer allowlists, and reputation data all still point the old way. + +**Consuming a class must be a privilege.** Once consumers name classes instead of +pools, the class name is the only authorization boundary left. Naming a class must +be checked, and it must fail closed. + +## Drawbacks + +- **New allocation stops during a central outage.** Covered above; the mitigation, + if measurement justifies it, is a small reserve per location. +- **More concepts.** Consumers gain a name to think about, operators gain a catalog + to curate. Per-family defaults keep the common path free of both. +- **A held address is capacity nobody else can use.** That is the price of an + address that survives a redeploy, and on a finite public range it is the cost that + matters — which is why retention carries a lease rather than lasting forever. +- **A public address per instance per location adds up.** The design makes that + explicit rather than hiding it, but it is a real cost consumers will meet. + +## Alternatives + +- **Allocation at every location.** Rejected: a database per location, two sources + of truth, and it gives up the platform-wide inventory that motivates the work. +- **Delegating pools to locations through the federation layer.** Rejected: it needs + the federation tooling to carry information in a direction it is not built to + carry, and still requires the full service at every location. +- **Leaving instance addresses to the runtime.** Rejected: it is the status quo, + and it is why no one can hold an address across a redeploy or count one against a + budget. +- **A multi-family class with sizes per family.** Rejected: it assumes the families + differ only in size, and they differ in parent, collision domain, and hierarchy. +- **Class-level utilization maintained during allocation.** Rejected: it makes one + row every pool of a class contends on, and cannot express the per-location number + that actually matters. + +## Open Questions + +**What should a network default to?** The platform is IPv6-first, but a network +currently defaults to IPv4 only while interfaces are proposed to default to IPv6. +Those cannot both be right, and the mismatch surfaces as an interface requesting a +family its network does not carry. + +**What is the interface's configured prefix length and next-hop model?** It decides +whether the gateway is on-link, whether reservations protect anything real, and how +much per-node endpoint state each interface costs. Every reservation question +resolves once it is answered. + +**Where does export policy live?** A class is shared by every consumer that names +it, so per-consumer export rules cannot live on one. Anything beyond "advertised or +not" needs a per-holder object the class points at. From 399df9bde41fcd6e20c16b860964d347da37a320 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 5 Aug 2026 13:41:22 -0500 Subject: [PATCH 02/15] docs: add references section and link out to source material Status lives in the frontmatter rather than being repeated in the body. Replaces the single related-doc line with a References section covering the addressing plans this draws on and the systems holding each piece, and links descriptively from the passages that depend on them. --- docs/enhancements/ipam-integration.md | 74 +++++++++++++++++++-------- 1 file changed, 53 insertions(+), 21 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 73fa616d..0923bd67 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -6,11 +6,6 @@ latest-milestone: "v0.x" # Address space consumers can actually ask for -**Status:** Draft -**Related:** [Federated Deployment Scheduling](federated-deployment-scheduling.md) (how a placement reaches the location that makes the claim) - ---- - - [Summary](#summary) - [Motivation](#motivation) - [Goals](#goals) @@ -29,6 +24,7 @@ latest-milestone: "v0.x" - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) - [Open Questions](#open-questions) +- [References](#references) ## Summary @@ -80,8 +76,8 @@ consumption is established. - **Fabric and infrastructure addressing.** Node loopbacks, routing locators, underlay links, and the per-site blocks they come from are platform-internal and - have their own plan. They can move onto classes later; nothing here depends on - it. + are covered by the [fabric addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/fabric.md). + They can move onto classes later; nothing here depends on it. - **Non-address numbering.** AS numbers, forwarding-instance identifiers, and MAC assignments are allocatable resources with the same claim semantics, but a class as designed here is prefix-shaped. They need a sibling model, not this one. @@ -227,7 +223,9 @@ Terminology, used consistently: - **Allocation**: the record of an address handed out. - **Context**: the network and location a claim is made for. -The fields below extend the shipped `IPClass`. Everything already on it — +The fields below extend the +[`IPClass` type the IPAM service already ships](https://github.com/milo-os/ipam/blob/main/pkg/apis/ipam/v1alpha1/types.go). +Everything already on it — `provisioner`, `parameters`, `ipFamily`, `strategy`, `allowedPrefixLengths`, `defaultPrefixLength`, `reclaimPolicy`, `visibility` — keeps its current meaning. @@ -400,8 +398,9 @@ express "the worst location," which is the number that matters. class for each.** The tempting alternative is one class spanning both families with sizes declared -per family. It is appealing until checked against the addressing plan, where the -two families are not the same kind of thing. +per family. It is appealing until checked against the +[tenant addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md), +where the two families are not the same kind of thing. A tenant endpoint in IPv6 gets a block carved from **that tenant's own prefix**, unique to them, which the instance subdivides. In IPv4 it gets a single address @@ -442,15 +441,17 @@ report utilization honestly. The trade is worth stating plainly: **while the central service is unreachable, no new addresses are handed out.** A location cannot start a new instance. It does not touch live traffic — existing addresses keep working and routes keep being -advertised — and it is the same dependency instance creation already has on central -quota. If it proves to matter, the answer is a small pre-reserved buffer per -location: a cache, not a second allocator. +advertised — and it is the same dependency instance creation already has on +[central quota enforcement](quota-enforcement/README.md). If it proves to matter, +the answer is a small pre-reserved buffer per location: a cache, not a second +allocator. **Claims are made where the work lands.** A consumer declares intent in their -project; that intent travels to the location the workload is placed at, and the -network layer there — which knows the location, the network, and the family — turns -it into a claim. That is why a consumer never writes a location: the system asking -already is one. +project; that intent +[travels to the location the workload is placed at](federated-deployment-scheduling.md), +and the network layer there — which knows the location, the network, and the +family — turns it into a claim. That is why a consumer never writes a location: the +system asking already is one. Because that system claims on the consumer's behalf, two things must hold. Its authority is bounded by the placements actually delivered to it — a claim is valid @@ -706,12 +707,14 @@ Allocating an address is necessary and nowhere near sufficient. These are the pi the design assumes and does not provide. Listing them is the point — a design that quietly assumed them would look finished and behave otherwise. -**A network needs one routing identity across every location it reaches**, unique +**A [network](https://github.com/datum-cloud/network-services-operator/blob/main/api/v1alpha/network_types.go) +needs one routing identity across every location it reaches**, unique platform-wide, or the two halves of a multi-location workload are unrelated networks sharing a name. This is the identity scoping route import and export — not the -per-location forwarding-instance identifier, which is deliberately reused in every -location and is a much smaller space. Conflating the two would cap the platform at a -few thousand networks in total rather than per location. +[per-location forwarding-instance identifier](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/srv6.md), +which is deliberately reused in every location and is a much smaller space. +Conflating the two would cap the platform at a few thousand networks in total +rather than per location. **A moved instance needs its old route withdrawn before the new one is trusted.** Each node advertises with a distinct identity, so a route reflector keeps both @@ -795,3 +798,32 @@ resolves once it is answered. **Where does export policy live?** A class is shared by every consumer that names it, so per-consumer export rules cannot live on one. Anything beyond "advertised or not" needs a per-holder object the class points at. + +## References + +**The address space this draws on** + +- [Platform addressing plan](https://github.com/datum-cloud/enhancements/tree/main/architecture/design/network/addressing) + — the overview tying the three plans below together. +- [Tenant addressing](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md) + — the per-network IPv6 prefix and the shared per-location IPv4 range that the + example classes carve from, and the reason the two families need separate classes. +- [Fabric addressing](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/fabric.md) + — the platform-internal blocks this design explicitly leaves alone. +- [SRv6 uSID plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/srv6.md) + — the forwarding-instance identifier that must not be confused with a network's + platform-wide routing identity. + +**The systems that hold the pieces** + +- [IPAM API types](https://github.com/milo-os/ipam/blob/main/pkg/apis/ipam/v1alpha1/types.go) + — `IPClass`, `IPPool`, `IPClaim`, and `IPAllocation` as they exist today, which + the class fields here extend. +- [Network API](https://github.com/datum-cloud/network-services-operator/blob/main/api/v1alpha/network_types.go) + — the network a claim is scoped to, and where a network's addressing intent is + declared. +- [Federated Deployment Scheduling](federated-deployment-scheduling.md) + — how a placement reaches the location whose network layer makes the claim. +- [Quota Enforcement](quota-enforcement/README.md) + — the existing central dependency in the instance-creation path, and where + address space becomes a budgeted unit. From 40f22f53134fc5f72db43d01151d9acae204d9b2 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 5 Aug 2026 17:08:25 -0500 Subject: [PATCH 03/15] docs: generalize class scope so infrastructure addressing fits the model Replaces allocationScope and collisionDomain, two closed enums over network and location, with identity and uniqueWithin -- lists of opaque scope references the allocator indexes without interpreting. The enums needed a new value for every kind of thing that can hold an address, which put node and site concepts inside the allocator; the reference lists carry sites, nodes, and links unchanged. Separates the two questions the enums had merged: identity decides whether a claim gets a new allocation or an existing one, uniqueWithin decides whether two allocations may hold the same address. Reservations now produce real allocations held by the parent, so reserved space has an owner and appears in inventory rather than being a hole, and a reservation need not sit at the edge of its parent. Splits retention's identity in two: the slot the allocation is identified by, which survives replacement, and the instance holding it, which does not -- previously one identity was asked to do both. --- docs/enhancements/ipam-integration.md | 245 +++++++++++++++++--------- 1 file changed, 162 insertions(+), 83 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 0923bd67..dabf0181 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -75,9 +75,14 @@ consumption is established. ### Non-Goals - **Fabric and infrastructure addressing.** Node loopbacks, routing locators, - underlay links, and the per-site blocks they come from are platform-internal and - are covered by the [fabric addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/fabric.md). - They can move onto classes later; nothing here depends on it. + underlay links, and the per-site blocks they come from are platform-internal, + have no consumer, and are covered by the + [fabric addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/fabric.md). + They are out of scope for this document, but not for the model it proposes: they + are a hierarchy of prefixes identified by sites, nodes, and links, which is what + `identity` and `uniqueWithin` are general over. Nothing here needs to change to + carry them, and the fields were checked against that plan rather than only + against consumer addressing. - **Non-address numbering.** AS numbers, forwarding-instance identifiers, and MAC assignments are allocatable resources with the same claim semantics, but a class as designed here is prefix-shaped. They need a sibling model, not this one. @@ -221,13 +226,17 @@ Terminology, used consistently: - **Pool**: a block of capacity offering itself to one or more classes. - **Claim**: a request for an address of a named class. - **Allocation**: the record of an address handed out. -- **Context**: the network and location a claim is made for. +- **Scope**: the references a claim carries, each under a role name — the network + and location it is made for, the interface it is for. A class names the roles it + needs; the allocator indexes their values without interpreting them. The fields below extend the [`IPClass` type the IPAM service already ships](https://github.com/milo-os/ipam/blob/main/pkg/apis/ipam/v1alpha1/types.go). Everything already on it — `provisioner`, `parameters`, `ipFamily`, `strategy`, `allowedPrefixLengths`, -`defaultPrefixLength`, `reclaimPolicy`, `visibility` — keeps its current meaning. +`defaultPrefixLength`, `reclaimPolicy`, `visibility` — keeps its name and its +values. Only `reclaimPolicy` shifts in meaning, and it narrows: it decides how long +an allocation waits, while `identity` decides who it waits for. ```yaml apiVersion: ipam.miloapis.com/v1alpha1 @@ -254,29 +263,40 @@ spec: # declared ancestry. parentClassName: tenant-subnet-ipv6 - # How many allocations of this class exist. - # PerClaim one per claim — what an interface address wants - # PerNetwork at most one per network - # PerNetworkLocation at most one per network per location - # Defaults to PerClaim. Immutable. - allocationScope: PerClaim - - # How far the allocator has to look to be sure an address is free. - # Platform compare against every allocation on the platform - # Location compare against allocations in the same location - # Network compare against allocations in the same network - # This is a search scope, not a guarantee — what the guarantee turns out to be - # depends on the parent. Defaults to Platform, the strictest. Immutable. - collisionDomain: Network + # What makes two claims the same allocation. Names the scope references a + # claim of this class must carry; one allocation exists per distinct + # combination of their values. A second claim presenting the same combination + # receives the allocation that already exists rather than a new one. + # + # The values are opaque {apiGroup, kind, name} references. The allocator + # indexes them and never interprets them, so a class can be identified by + # anything the caller can name. + # + # Defaults to the claim itself, which is one allocation per claim. Immutable. + identity: [interface] + + # What defines one independent address space. Two allocations may hold the + # same address if, and only if, they differ in one of these references. + # Empty means one space platform-wide. + # + # This states the guarantee, and the allocator's search follows from it. + # Defaults to empty, the strictest. Immutable. + uniqueWithin: [network] # The sizes a claim of this class may request, and the size used when a claim # asks for none. A fixed-size class sets min and max equal. allowedPrefixLengths: { min: 96, max: 96 } defaultPrefixLength: 96 - # Positions in the parent this class must not allocate, counted in units of - # this class's own allocation size. Loosening is always safe; tightening - # strands allocations already sitting in newly-reserved positions and warns. + # Positions in the parent this class does not allocate from, counted in units + # of this class's own allocation size. Each becomes a real allocation held by + # the parent — reserved space is inventory, not an invisible hole, so it has + # an owner, appears in utilization, and can be programmed. + # A reservation is held by the parent and excluded from every space carved + # from it, whatever `uniqueWithin` says — one reservation per parent, not one + # per network. + # Loosening is always safe; tightening strands allocations already sitting in + # newly-reserved positions and warns. reservations: leading: 1 # the subnet's gateway and its all-zeros address live here trailing: 0 @@ -291,9 +311,11 @@ spec: # An aggregate must be originated with a discard route. A class advertising an # aggregate it cannot fully resolve blackholes the unallocated space inside it. - # What happens to the address when its claim goes away. Delete releases it. - # Retain holds it against the claimant's identity so a replacement gets the - # same address back. A claim can override this. + # Whether the allocation outlives the claim that created it. Delete releases + # it; Retain keeps it, so the next claim presenting the same identity gets the + # same address back. Which claim counts as "the same" is `identity` above — + # this field only decides how long the allocation waits for it. + # A claim can override this. reclaimPolicy: Delete # The allocator that satisfies claims of this class. Defaults to the @@ -308,44 +330,59 @@ how a continent's block contains its locations' ranges and stays summarisable as one route. A pool declaring a location is not eligible for a claim from a different one, and an unlocated ancestor is never eligible in its child's place. -**What the collision domain means.** It is how far the allocator looks before -handing an address out, and the parent decides how far is far enough. +**Why these are two fields and not one.** They answer different questions about +the same references. `identity` decides whether a claim gets a *new* allocation or +an *existing* one. `uniqueWithin` decides whether two allocations may hold the same +address. A subnet is identified by its network and location, so the second interface +on that network reuses it; an interface address is identified by the interface, so +every interface gets its own. -Both endpoint classes in the example below set `collisionDomain: Network`, and they produce -opposite results. An IPv6 endpoint is carved from a `/64` that belongs to one -network and no other, so comparing within that network is the only comparison that -could matter — nothing outside it can allocate from that space, and the result is -unique platform-wide. An IPv4 endpoint is carved from a range every network in the -location shares, so comparing within the network is a genuine narrowing: two -networks reach the same address and both keep it. +**What `uniqueWithin` means.** Both endpoint classes in the example below set +`uniqueWithin: [network]`, and the field is doing different amounts of work in each. +An IPv6 endpoint is carved from a `/64` that belongs to one network and no other, so +the parent already separates the space and the result is unique platform-wide +regardless. An IPv4 endpoint is carved from a range every network in the location +shares, so the setting is load-bearing: two networks reach the same address and both +keep it. -So the field says how hard to look, not how unique the answer is. Setting it wider -than the parent requires is safe and wasteful; setting it narrower is how two -holders end up with one address, which is exactly what IPv4 wants and nothing else -does. +Setting it wider than the parent requires is safe and wasteful. Setting it narrower +is how two holders end up with one address — which is exactly what IPv4 tenant space +wants, and what nothing else does. -**What a claim carries.** The collision domain, the allocation scope, and parent -resolution all key off the same two values, so the claim carries them explicitly: +**What a claim carries.** `identity`, `uniqueWithin`, and parent resolution all key +off the same references, so the claim carries them by role: ```yaml kind: IPClaim spec: className: tenant-endpoint-ipv4 - networkRef: { name: default } - location: us-central-1 + scope: + network: { apiGroup: networking.datumapis.com, kind: Network, name: default } + location: { apiGroup: networking.datumapis.com, kind: Location, name: us-central-1 } + # Names the interface declared on the slot, not the runtime object rebuilt + # with each instance — see Instance addresses. + interface: { apiGroup: compute.datumapis.com, kind: NetworkInterface, name: … } ``` -Neither is written by a consumer. The network layer at the location supplies both, -because it is the one thing that knows them: it holds the deployment, so it knows -the location, and it resolved the interface's network reference before claiming. -Both are immutable — a claim whose network or location changed after allocation is -incoherent. +None of this is written by a consumer. The network layer at the location supplies +it, because it is the one thing that knows all of it: it holds the deployment, so it +knows the location, and it resolved the interface's network reference before +claiming. The references are immutable — a claim whose network or location changed +after allocation is incoherent. + +Roles are just names a class refers to. The allocator does not know what a network +is, or a location, or an interface; it knows that `tenant-endpoint-ipv4` is +identified by whatever arrives under `interface` and unique within whatever arrives +under `network`. That is what lets a class be identified by something this document +never mentions — which is how the same model covers infrastructure addressing, where +the roles are sites, nodes, and links, without any of those concepts entering the +allocator. A name is enough here, and nobody types an identifier. Every claim is already scoped to the project it was made for, and a network name is unique within a -project, so `default` in one project and `default` in another are different -collision domains without anything extra being carried. That is the same scoping -every pool and allocation already uses. +project, so `default` in one project and `default` in another are different address +spaces without anything extra being carried. That is the same scoping every pool and +allocation already uses. The one case a name does not settle is a network deleted and recreated under the same name. It inherits its predecessor's space and its allocations, which is @@ -353,29 +390,31 @@ usually what someone wants and occasionally not. Deciding otherwise means the reference carries something stable across recreation rather than a name — worth settling before retention ships, since that is where it starts to matter. -A class using `collisionDomain: Network`, or an `allocationScope` naming the -network, cannot be satisfied by a claim that omits the reference. The claim is -rejected rather than falling back to a wider comparison, because a wider comparison +A claim that omits a role its class names in `identity` or `uniqueWithin` is +rejected. It does not fall back to a wider comparison, because a wider comparison would look correct while refusing addresses the narrow one was meant to allow — and the error would surface as unexplained exhaustion rather than a missing field. **Resolving a parent.** With `parentClassName` empty, the allocator takes every pool offering this class, discards those whose family differs, discards those declaring a different location, and picks among the rest by the class's strategy. -With `parentClassName` set, it finds the parent allocation whose scope keys match -this claim's context — for an endpoint claim on network `default` in -`us-central-1`, the `tenant-subnet-ipv6` allocation for that network and location. +With `parentClassName` set, it projects this claim's scope onto the parent class's +`identity` and looks for the allocation with those values — for an endpoint claim on +network `default` in `us-central-1`, the `tenant-subnet-ipv6` allocation identified +by that network and location. If the parent does not exist, the allocator creates it first, applying the parent class's configuration. Creation cascades: a claim in a location a network has never used creates that location's subnet, and a claim on a new network creates the network's prefix too. -**Two concurrency rules the cascade requires.** `allocationScope` is a uniqueness -constraint, not a lookup — two simultaneous claims for the same network and -location both observe no subnet and both try to create one, so it needs a partial -unique index on the scope keys, with the loser reading the winner's allocation -rather than failing. And a cascade takes a lock at every level, so the levels must +**Two concurrency rules the cascade requires.** `identity` is a constraint, not a +lookup — two simultaneous claims for the same network and location both observe no +subnet and both try to create one, so it needs a unique index over the identity +references, with the loser reading the winner's allocation rather than failing. +Because the set of references varies by class, that index is over a canonical +serialization of them rather than a fixed set of columns. And a cascade takes a lock +at every level, so the levels must be locked in a deterministic order; without one, two cascades touching the same chain from different directions deadlock. Chain depth is capped and cycles are rejected at class-write time, not at claim time. @@ -406,7 +445,7 @@ A tenant endpoint in IPv6 gets a block carved from **that tenant's own prefix**, unique to them, which the instance subdivides. In IPv4 it gets a single address from a **location-wide range every tenant reuses**, with no sub-block, because IPv4 scarcity makes per-tenant uniqueness impossible at scale. Different parent, -different collision domain, different hierarchy. That is not one class with two +different uniqueness rule, different hierarchy. That is not one class with two sizes — it is two classes serving the same interface. So the interface names families, each family resolves to a class, and where the @@ -489,13 +528,29 @@ configuration rather than being invented at boot, and the container platform's o address stops standing in for it. That is a change to the runtime contract, not just to what the platform records. -**Retention needs an identity that is not a name.** Instance names are composed -from workload, placement, location, and ordinal, and are reused freely — a deleted -workload and a new one under the same name produce identical instance names. So -retention binds to the instance's unique identity, and a late release carrying a -stale identity is rejected rather than honoured. The allocation records that -identity opaquely; nothing about the consumer's type system crosses into the -allocator. +**Retention needs two identities, not one.** They pull in opposite directions, and +conflating them is why an address either cannot survive a redeploy or cannot be +protected from a late release. + +*What the allocation is identified by* has to survive instance replacement, +because a replacement is the entire point — the new instance must present the same +identity to get the same address back. Instance names are composed from workload, +placement, location, and ordinal, so the name is exactly that: it denotes the slot, +and it is stable across every replacement filling it. This is what the `interface` +reference in a claim's scope denotes — the interface *declared* on that slot, named +the same way, and not the runtime object that is rebuilt with each instance. Every +interface class in this document is identified by it. + +*Who currently holds it* has to change on every replacement, so that a late release +arriving from the instance that was replaced is rejected rather than honoured. That +is the instance's unique identifier, recorded on the allocation and checked on +release. + +The two are recorded opaquely; nothing about the consumer's type system crosses into +the allocator. Note the consequence the slot identity carries: a deleted workload and +a new one under the same name produce identical instance names, so the new one +inherits the old one's retained addresses. That is the same recreation question a +network name raises, and it wants the same answer. Retention also needs an expiry. An address held forever against a location's public range takes that range out of service for everyone, so a retained allocation @@ -522,8 +577,8 @@ spec: ipFamily: IPv6 # No parentClassName — the top of a chain draws from the pools that offer it, # here IPPool/tenant-v6. - allocationScope: PerNetwork # one prefix per network - collisionDomain: Platform + identity: [network] # one prefix per network + uniqueWithin: [] # one space platform-wide allowedPrefixLengths: { min: 48, max: 48 } reclaimPolicy: Retain --- @@ -532,8 +587,8 @@ metadata: { name: tenant-subnet-ipv6 } spec: ipFamily: IPv6 parentClassName: tenant-network-ipv6 - allocationScope: PerNetworkLocation # one subnet per network, per location - collisionDomain: Platform + identity: [network, location] # one subnet per network, per location + uniqueWithin: [] allowedPrefixLengths: { min: 64, max: 64 } reclaimPolicy: Retain # a location's subnet is never renumbered --- @@ -544,7 +599,8 @@ metadata: spec: ipFamily: IPv6 parentClassName: tenant-subnet-ipv6 - collisionDomain: Network # the parent /64 is this network's alone, + identity: [interface] # one block per interface + uniqueWithin: [network] # the parent /64 is this network's alone, # so this is still platform-unique allowedPrefixLengths: { min: 96, max: 96 } reservations: { leading: 1 } # the subnet gateway lives in the first block @@ -559,7 +615,8 @@ spec: # IPv4 space to carve, so an endpoint draws straight from the location's shared # range. The IPv6 endpoint above sits three levels down its own chain; this one # is a chain of one. - collisionDomain: Network # the shared range makes this a real + identity: [interface] + uniqueWithin: [network] # the shared range makes this a real # narrowing — two networks reach the same # address and both keep it allowedPrefixLengths: { min: 32, max: 32 } @@ -571,7 +628,10 @@ spec: ipFamily: IPv4 # Also no parentClassName — public addresses come from the location's public # pool, not from anything the network owns. - collisionDomain: Platform # routable, so unique everywhere + identity: [interface] # the declared interface on the slot, so a + # replacement reclaims it — see Instance + # addresses + uniqueWithin: [] # routable, so unique everywhere allowedPrefixLengths: { min: 32, max: 32 } routing: { internal: Host, external: Aggregate } reclaimPolicy: Retain @@ -685,7 +745,7 @@ address comes from a `/20` every network in that location draws from. Same interface, same request, two genuinely different arrangements — which is why each family resolves its own class. -That difference is the collision domain: **an IPv6 endpoint block is unique +That difference is what `uniqueWithin` states: **an IPv6 endpoint block is unique platform-wide because the network's prefix is; an IPv4 address is compared only within its network.** Two networks can hold `10.128.0.2` in `us-central-1` at once and never meet, because the routing domain separates them — and reaching across @@ -734,11 +794,11 @@ first use currently means a record is written; nothing provisions the gateway, t forwarding instance, or the route-table entry. That is why the interface reports `Allocated` and `Programmed` separately. -**The gateway should be a real allocation.** It is currently a hole — space that is -reserved, owned by nothing, and configured by nothing. Making it an allocation held -by the subnet gives it an owner, lets it be programmed, and gives path-MTU discovery -a source address inside the network. Without one, oversized packets are dropped -silently: handshakes succeed and large transfers hang. +**The gateway still needs programming.** Reservations produce a real allocation held +by the subnet rather than a hole owned by nothing, which gives it an owner, puts it +in inventory, and gives path-MTU discovery a source address inside the network — but +allocating it does not configure it. Without a gateway that answers, oversized +packets are dropped silently: handshakes succeed and large transfers hang. **An endpoint's block is only reachable at its first address.** The block a class hands an interface is flattened to a single address before distribution, so an @@ -778,10 +838,22 @@ be checked, and it must fail closed. and it is why no one can hold an address across a redeploy or count one against a budget. - **A multi-family class with sizes per family.** Rejected: it assumes the families - differ only in size, and they differ in parent, collision domain, and hierarchy. + differ only in size, and they differ in parent, uniqueness rule, and hierarchy. - **Class-level utilization maintained during allocation.** Rejected: it makes one row every pool of a class contends on, and cannot express the per-location number that actually matters. +- **Fixed enums for scope, in place of `identity` and `uniqueWithin`.** Naming the + cases directly — `PerNetwork`, `PerNetworkLocation`, `Platform`, `Network` — reads + more plainly and was the first shape of these fields. Rejected on two counts. It + needs a new enum value for every kind of thing that can hold an address, so + infrastructure addressing would require the allocator to learn what a node and a + site are, which the opaque-reference rule exists to prevent. And the enum hid the + distinction between the two questions: `PerClaim` and `PerNetwork` look like two + settings of one dial, when they are "do not share" and "share on this key". +- **Reserved positions as policy rather than allocations.** Rejected: it leaves + space that is owned by nothing, absent from inventory, and impossible to program, + which is the specific complaint this document makes about the subnet gateway. It + also cannot express a reservation that is not at the start or end of the parent. ## Open Questions @@ -799,6 +871,13 @@ resolves once it is answered. it, so per-consumer export rules cannot live on one. Anything beyond "advertised or not" needs a per-holder object the class points at. +**Is a parent allocation a pool?** This document calls a network's `/48` an +allocation of `tenant-network-ipv6`; the worked example shows it as +`IPPool/network-default`, and the shipped API already nests pools through +`parentPoolRef`. Those want to be one thing — most likely that a pool *is* an +allocation that has children — but until it is settled there are two hierarchies +described where there should be one. + ## References **The address space this draws on** From f39d0b7d8ed5737e481bb820b809a3d902374918 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 5 Aug 2026 17:11:31 -0500 Subject: [PATCH 04/15] docs: expand inline YAML to block style in all examples Flow mappings and sequences read as shorthand and hide field structure. Every manifest now uses one key per line, with comments moved above the fields they describe rather than trailing them. --- docs/enhancements/ipam-integration.md | 172 ++++++++++++++++++-------- 1 file changed, 123 insertions(+), 49 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index dabf0181..7f9d12b6 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -122,16 +122,24 @@ spec: - name: app image: ghcr.io/datum-cloud/hello-unikraft:latest networkInterfaces: - - network: { name: default } - ipFamilies: [IPv6, IPv4] # dual-stack; IPv6 is primary - reclaimPolicy: Retain # keep the addresses across redeploys + - network: + name: default + # dual-stack; IPv6 is primary + ipFamilies: + - IPv6 + - IPv4 + # keep the addresses across redeploys + reclaimPolicy: Retain addresses: - - class: public-unicast-ipv4 # a class, never an address - # (omit for ordinary private addressing) + # a class, never an address + # (omit this whole block for ordinary private addressing) + - class: public-unicast-ipv4 placements: - name: default - locations: [us-central-1] - scaleSettings: { minReplicas: 1 } + locations: + - us-central-1 + scaleSettings: + minReplicas: 1 ``` Three lines are new, and none mention a pool, a prefix length, a CIDR, or which @@ -273,7 +281,8 @@ spec: # anything the caller can name. # # Defaults to the claim itself, which is one allocation per claim. Immutable. - identity: [interface] + identity: + - interface # What defines one independent address space. Two allocations may hold the # same address if, and only if, they differ in one of these references. @@ -281,11 +290,14 @@ spec: # # This states the guarantee, and the allocator's search follows from it. # Defaults to empty, the strictest. Immutable. - uniqueWithin: [network] + uniqueWithin: + - network # The sizes a claim of this class may request, and the size used when a claim # asks for none. A fixed-size class sets min and max equal. - allowedPrefixLengths: { min: 96, max: 96 } + allowedPrefixLengths: + min: 96 + max: 96 defaultPrefixLength: 96 # Positions in the parent this class does not allocate from, counted in units @@ -357,11 +369,20 @@ kind: IPClaim spec: className: tenant-endpoint-ipv4 scope: - network: { apiGroup: networking.datumapis.com, kind: Network, name: default } - location: { apiGroup: networking.datumapis.com, kind: Location, name: us-central-1 } + network: + apiGroup: networking.datumapis.com + kind: Network + name: default + location: + apiGroup: networking.datumapis.com + kind: Location + name: us-central-1 # Names the interface declared on the slot, not the runtime object rebuilt # with each instance — see Instance addresses. - interface: { apiGroup: compute.datumapis.com, kind: NetworkInterface, name: … } + interface: + apiGroup: compute.datumapis.com + kind: NetworkInterface + name: … ``` None of this is written by a consumer. The network layer at the location supplies @@ -572,68 +593,103 @@ see the names. # The tenant chain. A class names the class it carves from; the top of a chain # names none and draws from a pool instead. kind: IPClass -metadata: { name: tenant-network-ipv6 } +metadata: + name: tenant-network-ipv6 spec: ipFamily: IPv6 # No parentClassName — the top of a chain draws from the pools that offer it, # here IPPool/tenant-v6. - identity: [network] # one prefix per network - uniqueWithin: [] # one space platform-wide - allowedPrefixLengths: { min: 48, max: 48 } + # One prefix per network. + identity: + - network + # One space platform-wide. + uniqueWithin: [] + allowedPrefixLengths: + min: 48 + max: 48 reclaimPolicy: Retain --- kind: IPClass -metadata: { name: tenant-subnet-ipv6 } +metadata: + name: tenant-subnet-ipv6 spec: ipFamily: IPv6 parentClassName: tenant-network-ipv6 - identity: [network, location] # one subnet per network, per location + # One subnet per network, per location. + identity: + - network + - location uniqueWithin: [] - allowedPrefixLengths: { min: 64, max: 64 } - reclaimPolicy: Retain # a location's subnet is never renumbered + allowedPrefixLengths: + min: 64 + max: 64 + # A location's subnet is never renumbered. + reclaimPolicy: Retain --- kind: IPClass metadata: name: tenant-endpoint-ipv6 - annotations: { ipam.miloapis.com/is-default-class: "true" } + annotations: + ipam.miloapis.com/is-default-class: "true" spec: ipFamily: IPv6 parentClassName: tenant-subnet-ipv6 - identity: [interface] # one block per interface - uniqueWithin: [network] # the parent /64 is this network's alone, - # so this is still platform-unique - allowedPrefixLengths: { min: 96, max: 96 } - reservations: { leading: 1 } # the subnet gateway lives in the first block + # One block per interface. + identity: + - interface + # The parent /64 is this network's alone, so this is still platform-unique. + uniqueWithin: + - network + allowedPrefixLengths: + min: 96 + max: 96 + # The subnet gateway lives in the first block. + reservations: + leading: 1 --- kind: IPClass metadata: name: tenant-endpoint-ipv4 - annotations: { ipam.miloapis.com/is-default-class: "true" } + annotations: + ipam.miloapis.com/is-default-class: "true" spec: ipFamily: IPv4 # No parentClassName, and that is the whole IPv4 story: there is no per-network # IPv4 space to carve, so an endpoint draws straight from the location's shared # range. The IPv6 endpoint above sits three levels down its own chain; this one # is a chain of one. - identity: [interface] - uniqueWithin: [network] # the shared range makes this a real - # narrowing — two networks reach the same - # address and both keep it - allowedPrefixLengths: { min: 32, max: 32 } - reservations: { leading: 2, trailing: 2 } + identity: + - interface + # The shared range makes this a real narrowing — two networks reach the same + # address and both keep it. + uniqueWithin: + - network + allowedPrefixLengths: + min: 32 + max: 32 + reservations: + leading: 2 + trailing: 2 --- kind: IPClass -metadata: { name: public-unicast-ipv4 } +metadata: + name: public-unicast-ipv4 spec: ipFamily: IPv4 # Also no parentClassName — public addresses come from the location's public # pool, not from anything the network owns. - identity: [interface] # the declared interface on the slot, so a - # replacement reclaims it — see Instance - # addresses - uniqueWithin: [] # routable, so unique everywhere - allowedPrefixLengths: { min: 32, max: 32 } - routing: { internal: Host, external: Aggregate } + # The declared interface on the slot, so a replacement reclaims it — see + # Instance addresses. + identity: + - interface + # Routable, so unique everywhere. + uniqueWithin: [] + allowedPrefixLengths: + min: 32 + max: 32 + routing: + internal: Host + external: Aggregate reclaimPolicy: Retain ``` @@ -662,25 +718,43 @@ Two objects, both written by the consumer: ```yaml kind: Network -metadata: { name: default } +metadata: + name: default spec: - ipam: { mode: Auto } + ipam: + mode: Auto --- kind: Workload -metadata: { name: hello-sandbox } +metadata: + name: hello-sandbox spec: template: spec: - runtime: { sandbox: { containers: [{ name: app, image: … }] } } + runtime: + sandbox: + containers: + - name: app + image: … networkInterfaces: - - network: { name: default } - ipFamilies: [IPv6, IPv4] + - network: + name: default + ipFamilies: + - IPv6 + - IPv4 reclaimPolicy: Retain addresses: - class: public-unicast-ipv4 placements: - - { name: americas, locations: [us-central-1], scaleSettings: { minReplicas: 2 } } - - { name: europe, locations: [eu-west-1], scaleSettings: { minReplicas: 2 } } + - name: americas + locations: + - us-central-1 + scaleSettings: + minReplicas: 2 + - name: europe + locations: + - eu-west-1 + scaleSettings: + minReplicas: 2 ``` Three more objects appear in the project that the consumer did not write — their From d3f4c857bdb77244848d729227a893d9ce4bb37f Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 5 Aug 2026 17:26:00 -0500 Subject: [PATCH 05/15] docs: bind claims to allocations the way PVCs bind to volumes Removes the class field that named what identifies an allocation. A claim binds one allocation and records it, so nothing needs to re-derive which allocation a claim should get -- the claim object is the identity, as it is for PersistentVolumeClaim. Retention follows: an address survives a redeploy because the claim is named for the slot and outlives the instance, not because anything rematches a released address. That removes the window where the address is loose and the second identity introduced to match on. Notes the Released state as the part of the storage model not to copy. Container classes keep a narrower poolPer key, since provisioning one pool per network and location is a real constraint and is not about claims at all. --- docs/enhancements/ipam-integration.md | 257 ++++++++++++++------------ 1 file changed, 137 insertions(+), 120 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 7f9d12b6..25256955 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -79,10 +79,11 @@ consumption is established. have no consumer, and are covered by the [fabric addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/fabric.md). They are out of scope for this document, but not for the model it proposes: they - are a hierarchy of prefixes identified by sites, nodes, and links, which is what - `identity` and `uniqueWithin` are general over. Nothing here needs to change to - carry them, and the fields were checked against that plan rather than only - against consumer addressing. + are a hierarchy of prefixes scoped by sites, nodes, and links — opaque references + like any other, and a long-lived claim per node is the same shape as a long-lived + claim per workload slot. Nothing here needs to change to carry them, and the + design was checked against that plan rather than only against consumer + addressing. - **Non-address numbering.** AS numbers, forwarding-instance identifiers, and MAC assignments are allocatable resources with the same claim semantics, but a class as designed here is prefix-shaped. They need a sibling model, not this one. @@ -231,20 +232,21 @@ is a platform operator authoring classes. Terminology, used consistently: - **Class**: the policy object naming a kind of address space. -- **Pool**: a block of capacity offering itself to one or more classes. -- **Claim**: a request for an address of a named class. +- **Pool**: a block of capacity many claims draw from, offering itself to one or + more classes. +- **Claim**: a long-lived request for an address of a named class, bound to one + allocation for as long as it exists. - **Allocation**: the record of an address handed out. - **Scope**: the references a claim carries, each under a role name — the network - and location it is made for, the interface it is for. A class names the roles it - needs; the allocator indexes their values without interpreting them. + and location it is made for. A class names the roles it needs; the allocator + indexes their values without interpreting them. The fields below extend the [`IPClass` type the IPAM service already ships](https://github.com/milo-os/ipam/blob/main/pkg/apis/ipam/v1alpha1/types.go). Everything already on it — `provisioner`, `parameters`, `ipFamily`, `strategy`, `allowedPrefixLengths`, -`defaultPrefixLength`, `reclaimPolicy`, `visibility` — keeps its name and its -values. Only `reclaimPolicy` shifts in meaning, and it narrows: it decides how long -an allocation waits, while `identity` decides who it waits for. +`defaultPrefixLength`, `reclaimPolicy`, `visibility` — keeps its name, its values, +and its meaning. ```yaml apiVersion: ipam.miloapis.com/v1alpha1 @@ -271,18 +273,13 @@ spec: # declared ancestry. parentClassName: tenant-subnet-ipv6 - # What makes two claims the same allocation. Names the scope references a - # claim of this class must carry; one allocation exists per distinct - # combination of their values. A second claim presenting the same combination - # receives the allocation that already exists rather than a new one. + # Nothing here says which allocation a claim gets. An IPClaim binds to one + # IPAllocation and an IPAllocation to one IPClaim, exactly as a + # PersistentVolumeClaim binds to a PersistentVolume, and the claim object is + # the identity. See "What a claim carries". # - # The values are opaque {apiGroup, kind, name} references. The allocator - # indexes them and never interprets them, so a class can be identified by - # anything the caller can name. - # - # Defaults to the claim itself, which is one allocation per claim. Immutable. - identity: - - interface + # A class that other classes carve from sets `poolPer` instead — see + # tenant-subnet-ipv6 in the worked example. # What defines one independent address space. Two allocations may hold the # same address if, and only if, they differ in one of these references. @@ -323,10 +320,11 @@ spec: # An aggregate must be originated with a discard route. A class advertising an # aggregate it cannot fully resolve blackholes the unallocated space inside it. - # Whether the allocation outlives the claim that created it. Delete releases - # it; Retain keeps it, so the next claim presenting the same identity gets the - # same address back. Which claim counts as "the same" is `identity` above — - # this field only decides how long the allocation waits for it. + # What happens to the allocation when its claim is deleted. Delete releases + # the address; Retain keeps it held so it can be handed back deliberately. + # + # This is not how an address survives a redeploy — a redeploy does not delete + # the claim. See "Instance addresses". # A claim can override this. reclaimPolicy: Delete @@ -342,12 +340,17 @@ how a continent's block contains its locations' ranges and stays summarisable as one route. A pool declaring a location is not eligible for a claim from a different one, and an unlocated ancestor is never eligible in its child's place. -**Why these are two fields and not one.** They answer different questions about -the same references. `identity` decides whether a claim gets a *new* allocation or -an *existing* one. `uniqueWithin` decides whether two allocations may hold the same -address. A subnet is identified by its network and location, so the second interface -on that network reuses it; an interface address is identified by the interface, so -every interface gets its own. +**How a claim finds its allocation.** It does not look one up. A claim binds to one +allocation and an allocation to one claim, each recording the other, and the binding +is made once when the claim is created. This is how a `PersistentVolumeClaim` binds +to a `PersistentVolume`, and it is worth following closely: storage does not +reconstruct which volume a claim should get, and neither should this. The claim +object *is* the identity. + +That decides what the class has to say. Nothing on it selects an allocation for a +claim, because the claim already has one. What remains is what the allocator needs +in order to hand out an address in the first place — which space it comes from, and +what it must not collide with. **What `uniqueWithin` means.** Both endpoint classes in the example below set `uniqueWithin: [network]`, and the field is doing different amounts of work in each. @@ -361,13 +364,24 @@ Setting it wider than the parent requires is safe and wasteful. Setting it narro is how two holders end up with one address — which is exactly what IPv4 tenant space wants, and what nothing else does. -**What a claim carries.** `identity`, `uniqueWithin`, and parent resolution all key -off the same references, so the claim carries them by role: +**What a claim carries.** `uniqueWithin` and parent resolution key off the same +references, so the claim carries them by role: ```yaml kind: IPClaim +metadata: + # Deterministic, derived from the slot and the interface it is for. This name + # is what a replacement instance finds, so it is the durable identity — the + # same construction a StatefulSet uses to give a replaced pod its volume back. + name: hello-sandbox-americas-us-central-1-0-eth0-ipv4 + ownerReferences: + - kind: NetworkInterface + name: … spec: className: tenant-endpoint-ipv4 + # Optional. Names a specific address to bind, the way a PersistentVolumeClaim + # names a volume. Omitted, the allocator chooses. + address: "" scope: network: apiGroup: networking.datumapis.com @@ -377,27 +391,20 @@ spec: apiGroup: networking.datumapis.com kind: Location name: us-central-1 - # Names the interface declared on the slot, not the runtime object rebuilt - # with each instance — see Instance addresses. - interface: - apiGroup: compute.datumapis.com - kind: NetworkInterface - name: … ``` None of this is written by a consumer. The network layer at the location supplies -it, because it is the one thing that knows all of it: it holds the deployment, so it -knows the location, and it resolved the interface's network reference before -claiming. The references are immutable — a claim whose network or location changed -after allocation is incoherent. +it, because it is the one thing that knows it: it holds the deployment, so it knows +the location, and it resolved the interface's network reference before claiming. +The references are immutable — a claim whose network or location changed after +allocation is incoherent. Roles are just names a class refers to. The allocator does not know what a network -is, or a location, or an interface; it knows that `tenant-endpoint-ipv4` is -identified by whatever arrives under `interface` and unique within whatever arrives -under `network`. That is what lets a class be identified by something this document -never mentions — which is how the same model covers infrastructure addressing, where -the roles are sites, nodes, and links, without any of those concepts entering the -allocator. +is, or a location; it knows that `tenant-endpoint-ipv4` is unique within whatever +arrives under `network`. That is what lets a class be scoped by something this +document never mentions — which is how the same model covers infrastructure +addressing, where the roles are sites, nodes, and links, without any of those +concepts entering the allocator. A name is enough here, and nobody types an identifier. Every claim is already scoped to the project it was made for, and a network name is unique within a @@ -411,8 +418,8 @@ usually what someone wants and occasionally not. Deciding otherwise means the reference carries something stable across recreation rather than a name — worth settling before retention ships, since that is where it starts to matter. -A claim that omits a role its class names in `identity` or `uniqueWithin` is -rejected. It does not fall back to a wider comparison, because a wider comparison +A claim that omits a role its class names in `uniqueWithin`, or that its parent +chain needs, is rejected. It does not fall back to a wider comparison, because a wider comparison would look correct while refusing addresses the narrow one was meant to allow — and the error would surface as unexplained exhaustion rather than a missing field. @@ -420,19 +427,26 @@ the error would surface as unexplained exhaustion rather than a missing field. pool offering this class, discards those whose family differs, discards those declaring a different location, and picks among the rest by the class's strategy. With `parentClassName` set, it projects this claim's scope onto the parent class's -`identity` and looks for the allocation with those values — for an endpoint claim on -network `default` in `us-central-1`, the `tenant-subnet-ipv6` allocation identified -by that network and location. - -If the parent does not exist, the allocator creates it first, applying the parent +`poolPer` and looks for the pool with those values — for an endpoint claim on +network `default` in `us-central-1`, the `tenant-subnet-ipv6` pool for that network +and location. + +Note what a parent is. A claim binds an allocation, but the thing many allocations +carve from is a **pool** — an allocation is what one claim holds, a pool is capacity +many claims draw from. So a parent class does not hand out addresses; it provisions +pools, one per distinct combination of its `poolPer` references. That is why +`poolPer` appears only on classes other classes name as a parent, and why it is not +a property of claims at all. + +If the pool does not exist, the allocator creates it first, applying the parent class's configuration. Creation cascades: a claim in a location a network has never used creates that location's subnet, and a claim on a new network creates the network's prefix too. -**Two concurrency rules the cascade requires.** `identity` is a constraint, not a +**Two concurrency rules the cascade requires.** `poolPer` is a constraint, not a lookup — two simultaneous claims for the same network and location both observe no -subnet and both try to create one, so it needs a unique index over the identity -references, with the loser reading the winner's allocation rather than failing. +subnet pool and both try to create one, so it needs a unique index over the +`poolPer` references, with the loser reading the winner's pool rather than failing. Because the set of references varies by class, that index is over a canonical serialization of them rather than a fixed set of columns. And a cascade takes a lock at every level, so the levels must @@ -540,43 +554,46 @@ within it, so tracking is one record per interface rather than one per container Three things follow. **A claim must be able to ask for a specific address.** A claim asks for a size, not -an address. Handing the same address back to a replacement, and recording an address -already in use, both need a claim that names one. Without it the retention -experience above does not work. +an address. Recording an address already in use, and handing a specific one back +deliberately, both need a claim that names one — the `address` field above, playing +the part `volumeName` plays for storage. **The runtime stops choosing.** An instance's address arrives with its interface configuration rather than being invented at boot, and the container platform's own address stops standing in for it. That is a change to the runtime contract, not just to what the platform records. -**Retention needs two identities, not one.** They pull in opposite directions, and -conflating them is why an address either cannot survive a redeploy or cannot be -protected from a late release. - -*What the allocation is identified by* has to survive instance replacement, -because a replacement is the entire point — the new instance must present the same -identity to get the same address back. Instance names are composed from workload, -placement, location, and ordinal, so the name is exactly that: it denotes the slot, -and it is stable across every replacement filling it. This is what the `interface` -reference in a claim's scope denotes — the interface *declared* on that slot, named -the same way, and not the runtime object that is rebuilt with each instance. Every -interface class in this document is identified by it. - -*Who currently holds it* has to change on every replacement, so that a late release -arriving from the instance that was replaced is rejected rather than honoured. That -is the instance's unique identifier, recorded on the allocation and checked on -release. - -The two are recorded opaquely; nothing about the consumer's type system crosses into -the allocator. Note the consequence the slot identity carries: a deleted workload and -a new one under the same name produce identical instance names, so the new one -inherits the old one's retained addresses. That is the same recreation question a -network name raises, and it wants the same answer. - -Retention also needs an expiry. An address held forever against a location's public -range takes that range out of service for everyone, so a retained allocation -carries a lease, keeps consuming its holder's budget while it lives, and can be -force-released by an operator with an audit record. +**The claim outlives the instance.** An address survives a redeploy because nothing +released it — not because anything reconstructs who used to hold it. + +This is how a StatefulSet returns a volume to a replaced pod. The claim is named +deterministically for the slot, the pod is deleted and recreated, and the claim was +never touched, so the binding it holds is still the binding. There is no matching +step to get wrong and no window where the address is loose. + +So the network layer names an interface's claims from the slot, the interface, and +the family — all of which are composed from the workload, placement, location, and +ordinal, and are therefore stable across every replacement filling that slot. A +replacement finds claims that already exist and already hold addresses. Deleting the +workload deletes the claims, through ownership, and that is when `reclaimPolicy` +decides whether the addresses are released or held. + +Two consequences worth stating. A late release arriving from an instance that has +already been replaced is rejected, because the release is checked against the claim +that currently holds the binding rather than against a remembered holder. And a +deleted workload recreated under the same name produces identical claim names, so it +inherits its predecessor's addresses if they were retained — the same recreation +question a network name raises, wanting the same answer. + +A retained allocation still needs an expiry. An address held against a location's +public range takes that range out of service for everyone, so it carries a lease, +keeps consuming its holder's budget while it lives, and can be force-released by an +operator with an audit record. + +The one part of the storage model not to copy is what happens next. A `Retain` +volume whose claim is deleted becomes `Released` and cannot be bound again until +someone clears the stale reference by hand. A retained address must return to a +state something can claim, without an operator in the path. ### A workload in two locations @@ -590,8 +607,8 @@ Five classes and the pools that back them. Consumers never see these objects; th see the names. ```yaml -# The tenant chain. A class names the class it carves from; the top of a chain -# names none and draws from a pool instead. +# The tenant chain. The first two classes are containers: nothing claims them +# directly, and each provisions a pool the next class down carves from. kind: IPClass metadata: name: tenant-network-ipv6 @@ -599,8 +616,8 @@ spec: ipFamily: IPv6 # No parentClassName — the top of a chain draws from the pools that offer it, # here IPPool/tenant-v6. - # One prefix per network. - identity: + # One pool per network. + poolPer: - network # One space platform-wide. uniqueWithin: [] @@ -615,8 +632,9 @@ metadata: spec: ipFamily: IPv6 parentClassName: tenant-network-ipv6 - # One subnet per network, per location. - identity: + # One pool per network, per location. The second interface on this network in + # this location finds the pool the first one caused to be created. + poolPer: - network - location uniqueWithin: [] @@ -634,9 +652,7 @@ metadata: spec: ipFamily: IPv6 parentClassName: tenant-subnet-ipv6 - # One block per interface. - identity: - - interface + # No poolPer — claims bind allocations of this class directly, one per claim. # The parent /64 is this network's alone, so this is still platform-unique. uniqueWithin: - network @@ -658,8 +674,6 @@ spec: # IPv4 space to carve, so an endpoint draws straight from the location's shared # range. The IPv6 endpoint above sits three levels down its own chain; this one # is a chain of one. - identity: - - interface # The shared range makes this a real narrowing — two networks reach the same # address and both keep it. uniqueWithin: @@ -678,10 +692,6 @@ spec: ipFamily: IPv4 # Also no parentClassName — public addresses come from the location's public # pool, not from anything the network owns. - # The declared interface on the slot, so a replacement reclaims it — see - # Instance addresses. - identity: - - interface # Routable, so unique everywhere. uniqueWithin: [] allowedPrefixLengths: @@ -916,14 +926,21 @@ be checked, and it must fail closed. - **Class-level utilization maintained during allocation.** Rejected: it makes one row every pool of a class contends on, and cannot express the per-location number that actually matters. -- **Fixed enums for scope, in place of `identity` and `uniqueWithin`.** Naming the - cases directly — `PerNetwork`, `PerNetworkLocation`, `Platform`, `Network` — reads - more plainly and was the first shape of these fields. Rejected on two counts. It - needs a new enum value for every kind of thing that can hold an address, so - infrastructure addressing would require the allocator to learn what a node and a - site are, which the opaque-reference rule exists to prevent. And the enum hid the - distinction between the two questions: `PerClaim` and `PerNetwork` look like two - settings of one dial, when they are "do not share" and "share on this key". +- **A class field naming what identifies an allocation.** Two shapes of this were + tried: an enum of the cases (`PerClaim`, `PerNetwork`, `PerNetworkLocation`) and + then a list of scope references. Both let the allocator re-derive which allocation + a claim should get. Rejected because nothing needs to re-derive it — a claim binds + an allocation and records it, so the binding is a fact rather than a computation. + Storage settled this a decade ago and neither `PersistentVolumeClaim` nor + `PersistentVolume` carries such a field. The enum had a second problem: it needed + a new value for every kind of thing that can hold an address, which would have put + nodes and sites inside the allocator. +- **Retention by rebinding rather than by not unbinding.** Releasing an address when + an instance is deleted and re-matching it when the replacement appears is how the + design read before. Rejected: it opens a window where the address is loose, needs + a durable identity distinct from the holder to match on, and lands in the state + storage calls `Released`, where a retained volume needs manual intervention before + anything can bind it again. Keeping the claim is strictly simpler. - **Reserved positions as policy rather than allocations.** Rejected: it leaves space that is owned by nothing, absent from inventory, and impossible to program, which is the specific complaint this document makes about the subnet gateway. It @@ -945,12 +962,12 @@ resolves once it is answered. it, so per-consumer export rules cannot live on one. Anything beyond "advertised or not" needs a per-holder object the class points at. -**Is a parent allocation a pool?** This document calls a network's `/48` an -allocation of `tenant-network-ipv6`; the worked example shows it as -`IPPool/network-default`, and the shipped API already nests pools through -`parentPoolRef`. Those want to be one thing — most likely that a pool *is* an -allocation that has children — but until it is settled there are two hierarchies -described where there should be one. +**Should a container class be an `IPClass` at all?** `tenant-network-ipv6` and +`tenant-subnet-ipv6` are never named by a claim; they exist to provision pools. They +share most of a class's fields, which is the argument for one kind — but a consumer +listing classes should not see them, and `poolPer` is meaningless on everything +else. A separate kind, or a marker on the class, may read better than a field that +only applies to half of them. ## References From af0e447c43f44934b8ff6e1bffec61a0ac7e7e45 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 13:14:00 -0500 Subject: [PATCH 06/15] docs: edit for concision against Google tech writing style Applies the Google technical writing guidelines to the IP class proposal. Splits multi-idea sentences so each carries one thought, and cuts the filler and hedging around them. Converts four embedded lists that were buried in prose -- the two pool fields, the two concurrency rules the cascade needs, the three rules the allocator enforces, and the two consequences of the claim outliving the instance -- into real lists. Replaces ambiguous "it" and "that" with the noun they refer to, which was the most frequent problem: several paragraphs opened with a pronoun whose referent was two sentences back. Prefers active voice and a real subject over "there is" constructions. States the audience up front, and drops the restatement of the IPv4 and IPv6 difference in the worked example, which the address families section already makes. No design changes. --- docs/enhancements/ipam-integration.md | 629 +++++++++++++------------- 1 file changed, 314 insertions(+), 315 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 25256955..da0907ff 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -29,37 +29,39 @@ latest-milestone: "v0.x" ## Summary A consumer deploying a workload names the **class** of address it should get — -`public-unicast-ipv4`, `tenant-endpoint-ipv6` — and the platform returns one in the -create response, tracked from that moment until it is released. They never name a -pool, a prefix length, a region, or a CIDR. Operators define what each class means, -once, and can change what backs it without touching a consumer's manifest. +`public-unicast-ipv4`, `tenant-endpoint-ipv6`. The platform returns an address in +the create response and tracks it until release. Consumers never name a pool, a +prefix length, a region, or a CIDR. Operators define each class once and can change +what backs it without touching a consumer's manifest. -The capability it unlocks is small to describe: **an address a workload keeps.** A -published endpoint that survives a redeploy, a stable outbound address a customer -can allowlist, an inventory an operator can query. +The capability this unlocks is **an address a workload keeps**: a published endpoint +that survives a redeploy, a stable outbound address a customer can allowlist, an +inventory an operator can query. + +This document is written for platform operators who author classes and for the +compute and network layers that claim on a consumer's behalf. ## Motivation -Addressing is the kind of decision that gets made implicitly the first time a -workload boots and then has to be migrated. Settling it while it is still a -question of API design costs a field; settling it afterwards costs a renumbering. +Addressing gets decided implicitly the first time a workload boots, and changing it +afterwards means renumbering. Settling it while it is still a question of API design +costs one field. -Three things make an address something a consumer can rely on. +An address a consumer can rely on needs three things. -**A way to express intent.** "Give this a public address," "keep this address when -I redeploy," and "make it IPv6" are one-line requests, and the interface a consumer -writes should carry them. The IPv6 point is the sharpest — the platform is -IPv6-first by design, so asking for it should be the easy path. +**A way to express intent.** "Give this a public address," "keep this address when I +redeploy," and "make it IPv6" are one-line requests, and the interface a consumer +writes should carry them. The platform is IPv6-first by design, so asking for IPv6 +should be the easy path. **A system of record.** Who holds an address, when they got it, what happens when -they release it, how much of a kind of space is left. An on-call engineer and a -finance owner both ask these, and the answers are cheap to keep while allocations -are being made and expensive to reconstruct from the data plane afterwards. +they release it, and how much of a kind of space is left. On-call engineers and +finance owners both ask these questions. The answers are cheap to record during +allocation and expensive to reconstruct from the data plane afterwards. -**A unit to govern.** Quota, budgets, and utilization want to reason about "public -addresses" as a thing, in terms a consumer would recognise — and a budget -introduced alongside a capability lands very differently from one introduced after -consumption is established. +**A unit to govern.** Quota, budgets, and utilization need to reason about "public +addresses" in terms a consumer recognises. A budget introduced alongside a +capability lands better than one introduced after consumption is established. ### Goals @@ -78,22 +80,20 @@ consumption is established. underlay links, and the per-site blocks they come from are platform-internal, have no consumer, and are covered by the [fabric addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/fabric.md). - They are out of scope for this document, but not for the model it proposes: they - are a hierarchy of prefixes scoped by sites, nodes, and links — opaque references - like any other, and a long-lived claim per node is the same shape as a long-lived - claim per workload slot. Nothing here needs to change to carry them, and the - design was checked against that plan rather than only against consumer - addressing. + Fabric addressing is out of scope for this document but not for the model it + proposes: those blocks form a hierarchy of prefixes scoped by sites, nodes, and + links — opaque references like any other — and a long-lived claim per node has the + same shape as a long-lived claim per workload slot. The design was checked against + that plan, and nothing here needs to change to carry it. - **Non-address numbering.** AS numbers, forwarding-instance identifiers, and MAC assignments are allocatable resources with the same claim semantics, but a class - as designed here is prefix-shaped. They need a sibling model, not this one. -- **Globally-routable or consumer-owned tenant space.** Bring-your-own prefixes - and public tenant address space carry a mandatory validation and export regime - this design does not attempt to express. -- **Anycast.** A single address held by many locations at once is the inverse of - the rule every class here follows. Adding it later is additive: - `public-anycast-ipv4` joins the catalog beside `public-unicast-ipv4`, and nothing - already named changes. + as designed here is prefix-shaped. They need a sibling model. +- **Globally-routable or consumer-owned tenant space.** Bring-your-own prefixes and + public tenant address space carry a mandatory validation and export regime this + design does not express. +- **Anycast.** A single address held by many locations at once inverts the rule every + class here follows. Adding it later is additive: `public-anycast-ipv4` joins the + catalog beside `public-unicast-ipv4`, and nothing already named changes. - **Tracking addresses inside an endpoint.** An interface receives a block and assigns within it, which keeps containers and secondary addresses off the control-plane path. @@ -164,11 +164,11 @@ status: ``` Both conditions matter. `Allocated` means the platform assigned the address; -`Programmed` means the network can carry it. They are separate because allocation -is synchronous and programming is not, and an interface must not report ready on -allocation alone. +`Programmed` means the network can carry it. The two stay separate because +allocation is synchronous and programming is not, and an interface must not report +ready on allocation alone. -Every address there is a tracked allocation, so the questions above have answers: +Every address above is a tracked allocation, so the earlier questions have answers: ```console $ datumctl ipam address show 198.51.100.11 @@ -188,8 +188,8 @@ tenant-endpoint-ipv4 IPv4 /32 12 pools 38,104 61% us-central-1 public-unicast-ipv4 IPv4 /32 1 pool 148 58% us-central-1 ``` -The last column is the number that matters. A class averaged across locations -always reads healthy; what pages someone is one location filling up, so the view +The last column is the number that matters. Averaged across locations, a class +always reads healthy; one location filling up is what pages someone. The view reports the worst occupant rather than the mean. ### User Stories @@ -219,17 +219,17 @@ reports the worst occupant rather than the mean. interface reports which family could not be satisfied. - **The platform is IPv6-first.** An interface that says nothing gets IPv6. - **A class sets the default reclaim policy; an interface can override it.** -- **A retained address is still held and still counts** against its holder's - budget, or nothing pressures anyone to release it. +- **A retained address is still held and still counts** against its holder's budget. + Otherwise nothing pressures anyone to release it. ## Design Details ### Class configuration -This section defines each field and the rules the allocator applies. The audience -is a platform operator authoring classes. +This section defines each field and the rules the allocator applies. It is written +for platform operators authoring classes. -Terminology, used consistently: +This document uses the following terms consistently: - **Class**: the policy object naming a kind of address space. - **Pool**: a block of capacity many claims draw from, offering itself to one or @@ -275,8 +275,8 @@ spec: # Nothing here says which allocation a claim gets. An IPClaim binds to one # IPAllocation and an IPAllocation to one IPClaim, exactly as a - # PersistentVolumeClaim binds to a PersistentVolume, and the claim object is - # the identity. See "What a claim carries". + # PersistentVolumeClaim binds to a PersistentVolume. The claim object is the + # identity. See "What a claim carries". # # A class that other classes carve from sets `poolPer` instead — see # tenant-subnet-ipv6 in the worked example. @@ -285,7 +285,7 @@ spec: # same address if, and only if, they differ in one of these references. # Empty means one space platform-wide. # - # This states the guarantee, and the allocator's search follows from it. + # This field states the guarantee; the allocator's search follows from it. # Defaults to empty, the strictest. Immutable. uniqueWithin: - network @@ -299,13 +299,15 @@ spec: # Positions in the parent this class does not allocate from, counted in units # of this class's own allocation size. Each becomes a real allocation held by - # the parent — reserved space is inventory, not an invisible hole, so it has - # an owner, appears in utilization, and can be programmed. - # A reservation is held by the parent and excluded from every space carved - # from it, whatever `uniqueWithin` says — one reservation per parent, not one - # per network. - # Loosening is always safe; tightening strands allocations already sitting in - # newly-reserved positions and warns. + # the parent, so reserved space has an owner, appears in utilization, and can + # be programmed. It is inventory, not an invisible hole. + # + # The parent holds the reservation and excludes it from every space carved + # from that parent, whatever `uniqueWithin` says — one reservation per parent, + # not one per network. + # + # Loosening is always safe. Tightening strands allocations already sitting in + # newly-reserved positions, and warns. reservations: leading: 1 # the subnet's gateway and its all-zeros address live here trailing: 0 @@ -313,7 +315,7 @@ spec: # What routing does with the address. Advertisement is stated separately for # inside a location and beyond it, because the two are frequently opposite: # a per-instance address is a distinct route within its location and must - # never appear outside it, while only the covering block leaves. + # never appear outside it. Only the covering block leaves. routing: internal: None # None | Host external: None # None | Aggregate @@ -333,36 +335,36 @@ spec: provisioner: ipam.miloapis.com/native ``` -**Pools gain two fields.** `location` names the location a pool serves, so a claim -made in one location reaches that location's space without anyone naming it; a -pool with no location serves everywhere. `parentPoolName` lets pools nest, which is -how a continent's block contains its locations' ranges and stays summarisable as -one route. A pool declaring a location is not eligible for a claim from a different -one, and an unlocated ancestor is never eligible in its child's place. +**Pools gain two fields:** + +- `location` names the location a pool serves, so a claim made in one location + reaches that location's space without anyone naming it. A pool with no location + serves everywhere. A pool declaring a location is not eligible for a claim from a + different one, and an unlocated ancestor is never eligible in its child's place. +- `parentPoolName` lets pools nest, so a continent's block contains its locations' + ranges and stays summarisable as one route. **How a claim finds its allocation.** It does not look one up. A claim binds to one allocation and an allocation to one claim, each recording the other, and the binding -is made once when the claim is created. This is how a `PersistentVolumeClaim` binds -to a `PersistentVolume`, and it is worth following closely: storage does not -reconstruct which volume a claim should get, and neither should this. The claim -object *is* the identity. +is made once when the claim is created. A `PersistentVolumeClaim` binds to a +`PersistentVolume` the same way. Storage does not reconstruct which volume a claim +should get, and neither should this: the claim object *is* the identity. -That decides what the class has to say. Nothing on it selects an allocation for a -claim, because the claim already has one. What remains is what the allocator needs -in order to hand out an address in the first place — which space it comes from, and -what it must not collide with. +That settles what the class has to say. Nothing on it selects an allocation, because +the claim already has one. The class carries only what the allocator needs to hand +out an address in the first place: which space it comes from, and what it must not +collide with. **What `uniqueWithin` means.** Both endpoint classes in the example below set -`uniqueWithin: [network]`, and the field is doing different amounts of work in each. -An IPv6 endpoint is carved from a `/64` that belongs to one network and no other, so -the parent already separates the space and the result is unique platform-wide -regardless. An IPv4 endpoint is carved from a range every network in the location -shares, so the setting is load-bearing: two networks reach the same address and both -keep it. +`uniqueWithin: [network]`, and the field does different amounts of work in each. An +IPv6 endpoint is carved from a `/64` belonging to one network and no other, so the +parent already separates the space and the result is unique platform-wide regardless. +An IPv4 endpoint is carved from a range every network in the location shares, so the +setting is load-bearing: two networks reach the same address and both keep it. -Setting it wider than the parent requires is safe and wasteful. Setting it narrower -is how two holders end up with one address — which is exactly what IPv4 tenant space -wants, and what nothing else does. +Setting `uniqueWithin` wider than the parent requires is safe and wasteful. Setting +it narrower is how two holders end up with one address, which IPv4 tenant space wants +and nothing else does. **What a claim carries.** `uniqueWithin` and parent resolution key off the same references, so the claim carries them by role: @@ -393,35 +395,35 @@ spec: name: us-central-1 ``` -None of this is written by a consumer. The network layer at the location supplies -it, because it is the one thing that knows it: it holds the deployment, so it knows -the location, and it resolved the interface's network reference before claiming. -The references are immutable — a claim whose network or location changed after -allocation is incoherent. - -Roles are just names a class refers to. The allocator does not know what a network -is, or a location; it knows that `tenant-endpoint-ipv4` is unique within whatever -arrives under `network`. That is what lets a class be scoped by something this -document never mentions — which is how the same model covers infrastructure -addressing, where the roles are sites, nodes, and links, without any of those -concepts entering the allocator. - -A name is enough here, and nobody types an identifier. Every claim is already -scoped to the project it was made for, and a network name is unique within a -project, so `default` in one project and `default` in another are different address -spaces without anything extra being carried. That is the same scoping every pool and +No consumer writes any of this. The network layer at the location supplies the +scope, because it is the only thing that knows it: it holds the deployment, so it +knows the location, and it resolved the interface's network reference before +claiming. The references are immutable, since a claim whose network or location +changed after allocation is incoherent. + +Roles are only names a class refers to. The allocator does not know what a network or +a location is; it knows that `tenant-endpoint-ipv4` is unique within whatever arrives +under `network`. So a class can be scoped by something this document never mentions. +That is how the same model covers infrastructure addressing, where the roles are +sites, nodes, and links, without any of those concepts entering the allocator. + +A name is enough, and nobody types an identifier. Every claim is already scoped to +the project it was made for, and a network name is unique within a project. So +`default` in one project and `default` in another are different address spaces +without the reference carrying anything extra — the same scoping every pool and allocation already uses. -The one case a name does not settle is a network deleted and recreated under the -same name. It inherits its predecessor's space and its allocations, which is +The one case a name does not settle is a network deleted and recreated under the same +name. The new network inherits its predecessor's space and allocations, which is usually what someone wants and occasionally not. Deciding otherwise means the -reference carries something stable across recreation rather than a name — worth -settling before retention ships, since that is where it starts to matter. +reference carries something stable across recreation rather than a name. Settle this +before retention ships, because that is where it starts to matter. -A claim that omits a role its class names in `uniqueWithin`, or that its parent -chain needs, is rejected. It does not fall back to a wider comparison, because a wider comparison -would look correct while refusing addresses the narrow one was meant to allow — and -the error would surface as unexplained exhaustion rather than a missing field. +The allocator rejects a claim that omits a role its class names in `uniqueWithin` or +that its parent chain needs. Such a claim does not fall back to a wider comparison. A +wider comparison would look correct while refusing addresses the narrow one was meant +to allow, and the error would surface as unexplained exhaustion rather than a missing +field. **Resolving a parent.** With `parentClassName` empty, the allocator takes every pool offering this class, discards those whose family differs, discards those @@ -431,132 +433,135 @@ With `parentClassName` set, it projects this claim's scope onto the parent class network `default` in `us-central-1`, the `tenant-subnet-ipv6` pool for that network and location. -Note what a parent is. A claim binds an allocation, but the thing many allocations -carve from is a **pool** — an allocation is what one claim holds, a pool is capacity -many claims draw from. So a parent class does not hand out addresses; it provisions -pools, one per distinct combination of its `poolPer` references. That is why -`poolPer` appears only on classes other classes name as a parent, and why it is not -a property of claims at all. +Note what a parent is. A claim binds an allocation, but many allocations carve from a +**pool**. An allocation is what one claim holds; a pool is capacity many claims draw +from. So a parent class does not hand out addresses. It provisions pools, one per +distinct combination of its `poolPer` references. That is why `poolPer` appears only +on classes named as a parent, and why it is not a property of claims at all. If the pool does not exist, the allocator creates it first, applying the parent class's configuration. Creation cascades: a claim in a location a network has never used creates that location's subnet, and a claim on a new network creates the network's prefix too. -**Two concurrency rules the cascade requires.** `poolPer` is a constraint, not a -lookup — two simultaneous claims for the same network and location both observe no -subnet pool and both try to create one, so it needs a unique index over the -`poolPer` references, with the loser reading the winner's pool rather than failing. -Because the set of references varies by class, that index is over a canonical -serialization of them rather than a fixed set of columns. And a cascade takes a lock -at every level, so the levels must -be locked in a deterministic order; without one, two cascades touching the same -chain from different directions deadlock. Chain depth is capped and cycles are -rejected at class-write time, not at claim time. - -**Rules the allocator enforces.** A class and its parent share an address family. -A class's prefix lengths are longer than its parent's. When a parent is exhausted, -the error names the level that ran out, not the level that was asked for. - -**Class health is computed, never stored.** A class reports whether any pool backs -it and how full its worst location is. Both are aggregates over pool status, read -at query time. They are deliberately not counters maintained during allocation: a +**The cascade needs two concurrency rules:** + +- **A unique index over the `poolPer` references.** `poolPer` is a constraint, not a + lookup. Two simultaneous claims for the same network and location both observe no + subnet pool and both try to create one; the loser reads the winner's pool rather + than failing. Because the set of references varies by class, the index covers a + canonical serialization of them rather than a fixed set of columns. +- **A deterministic lock order across levels.** A cascade takes a lock at every + level. Without a fixed order, two cascades touching the same chain from different + directions deadlock. + +Chain depth is capped and cycles are rejected at class-write time, not at claim time. + +**The allocator enforces three rules:** + +- A class and its parent share an address family. +- A class's prefix lengths are longer than its parent's. +- When a parent is exhausted, the error names the level that ran out, not the level + that was asked for. + +**Class health is computed, never stored.** A class reports whether any pool backs it +and how full its worst location is. Both are aggregates over pool status, read at +query time. Neither is a counter maintained during allocation, and deliberately so: a class is backed by many pools, so a class-level counter would be one row every pool -contends on — turning independent claims in different locations into a queue and -destroying the per-pool locking the service depends on. A counter also cannot -express "the worst location," which is the number that matters. +contends on. That would turn independent claims in different locations into a queue +and destroy the per-pool locking the service depends on. A counter also cannot express +"the worst location," which is the number that matters. ### Address families **A class is single-family. The interface asks for families; the platform picks a class for each.** -The tempting alternative is one class spanning both families with sizes declared -per family. It is appealing until checked against the +The tempting alternative is one class spanning both families with sizes declared per +family. That alternative holds up until it meets the [tenant addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md), where the two families are not the same kind of thing. A tenant endpoint in IPv6 gets a block carved from **that tenant's own prefix**, -unique to them, which the instance subdivides. In IPv4 it gets a single address -from a **location-wide range every tenant reuses**, with no sub-block, because IPv4 -scarcity makes per-tenant uniqueness impossible at scale. Different parent, -different uniqueness rule, different hierarchy. That is not one class with two -sizes — it is two classes serving the same interface. - -So the interface names families, each family resolves to a class, and where the -consumer names none — the common case — the platform's default for that family +unique to them, which the instance subdivides. In IPv4 it gets a single address from +a **location-wide range every tenant reuses**, with no sub-block, because IPv4 +scarcity makes per-tenant uniqueness impossible at scale. Different parent, different +uniqueness rule, different hierarchy. That is two classes serving one interface, not +one class with two sizes. + +So the interface names families and each family resolves to a class. Where the +consumer names no class — the common case — the platform's default for that family applies. A consumer names a class only for something non-default, and names one per -family if they want that in both. +family to get that in both. -**The family belongs in the name.** Naming a class is choosing a family, so -`public-unicast-ipv4`, not `public`. The rule reaches past the family: `unicast` is -there because there is a real alternative a consumer could choose, and the two -behave visibly differently — a unicast address is one per instance per location, an -anycast address is one address live everywhere. Name the property where the -consumer is choosing between real alternatives, and leave it out where there is -only one. Tenant addressing is never advertised, so it carries no routing -qualifier. +**The family belongs in the name.** Naming a class means choosing a family, so +`public-unicast-ipv4`, not `public`. The rule reaches past the family. `unicast` +appears because a consumer can really choose the alternative, and the two behave +visibly differently: a unicast address is one per instance per location, an anycast +address is one address live everywhere. Name the property where the consumer chooses +between real alternatives, and leave it out where only one exists. Tenant addressing +is never advertised, so it carries no routing qualifier. ### Where the address comes from **One allocator, in the middle, for everything.** Not a copy per location. -The tempting alternative is pushing allocation out to each location so it keeps -working alone. It does not pay for itself. The high-volume cases that seem to need -it are not ours — pod addresses belong to container networking. What is left is a -few addresses per interface at instance-creation rate. A copy per location would -mean a database at every location, two versions of the truth, and a reconciliation -problem nobody has scoped. - -It would also cost the thing this is for. One allocator is the only way to answer -"who has this address" across the platform, enforce quota on the real resource, and -report utilization honestly. - -The trade is worth stating plainly: **while the central service is unreachable, no -new addresses are handed out.** A location cannot start a new instance. It does not -touch live traffic — existing addresses keep working and routes keep being -advertised — and it is the same dependency instance creation already has on -[central quota enforcement](quota-enforcement/README.md). If it proves to matter, -the answer is a small pre-reserved buffer per location: a cache, not a second -allocator. +The tempting alternative pushes allocation out to each location so it keeps working +alone. That alternative does not pay for itself. The high-volume cases that seem to +need it are not ours, because pod addresses belong to container networking; what +remains is a few addresses per interface at instance-creation rate. A copy per +location would mean a database at every location, two versions of the truth, and a +reconciliation problem nobody has scoped. + +It would also cost the thing this design is for. One allocator is the only way to +answer "who has this address" across the platform, enforce quota on the real resource, +and report utilization honestly. + +State the trade plainly: **while the central service is unreachable, no new addresses +are handed out.** A location cannot start a new instance. Live traffic is untouched — +existing addresses keep working and routes keep being advertised — and instance +creation already carries the same dependency on +[central quota enforcement](quota-enforcement/README.md). If the outage window proves +to matter, the answer is a small pre-reserved buffer per location: a cache, not a +second allocator. **Claims are made where the work lands.** A consumer declares intent in their -project; that intent +project. That intent [travels to the location the workload is placed at](federated-deployment-scheduling.md), -and the network layer there — which knows the location, the network, and the -family — turns it into a claim. That is why a consumer never writes a location: the +and the network layer there turns it into a claim, because that layer knows the +location, the network, and the family. A consumer never writes a location, since the system asking already is one. -Because that system claims on the consumer's behalf, two things must hold. Its -authority is bounded by the placements actually delivered to it — a claim is valid -only for a network and location it holds a deployment for. And the address is -attributed to the consumer's project, not to the platform identity that made the -call, or quota and ownership both attach to the wrong party. +That layer claims on the consumer's behalf, so two things must hold: + +- **The placements delivered to it bound its authority.** A claim is valid only for a + network and location it holds a deployment for. +- **The address is attributed to the consumer's project**, not to the platform + identity that made the call. Otherwise quota and ownership both attach to the wrong + party. ### Instance addresses -Every address on an instance is a real allocation, and that is the change making -the rest of the design work. As things stand the runtime picks the address — under -the sandbox runtime it is simply the container platform's own address — and the -platform records it afterwards, if at all. An address chosen that way cannot be -held across a redeploy or counted against a budget. Reversing it — **the platform -decides the address, and the runtime is told** — is what turns an address into -something a consumer can ask for and keep. +Every address on an instance is a real allocation, and that change makes the rest of +the design work. Today the runtime picks the address — under the sandbox runtime, +simply the container platform's own address — and the platform records it afterwards, +if at all. An address chosen that way cannot be held across a redeploy or counted +against a budget. Reversing the order — **the platform decides the address, and the +runtime is told** — turns an address into something a consumer can ask for and keep. -This is more tractable on the platform's own compute than it would be against a -third party. Every address in play is space the platform already owns, so there is -no external allocator to reconcile with and no address the platform records -without having issued. +This is more tractable on the platform's own compute than against a third party. +Every address in play is space the platform already owns, so no external allocator +needs reconciling and the platform never records an address it did not issue. -The unit is the interface, not the address. An interface gets a block and assigns -within it, so tracking is one record per interface rather than one per container. +The unit is the interface, not the address: an interface gets a block and assigns +within it, so tracking costs one record per interface rather than one per container. Three things follow. **A claim must be able to ask for a specific address.** A claim asks for a size, not -an address. Recording an address already in use, and handing a specific one back -deliberately, both need a claim that names one — the `address` field above, playing -the part `volumeName` plays for storage. +an address. Two cases need a claim that names one: recording an address already in +use, and handing a specific address back deliberately. The `address` field above +plays the part `volumeName` plays for storage. **The runtime stops choosing.** An instance's address arrives with its interface configuration rather than being invented at boot, and the container platform's own @@ -566,45 +571,47 @@ just to what the platform records. **The claim outlives the instance.** An address survives a redeploy because nothing released it — not because anything reconstructs who used to hold it. -This is how a StatefulSet returns a volume to a replaced pod. The claim is named -deterministically for the slot, the pod is deleted and recreated, and the claim was -never touched, so the binding it holds is still the binding. There is no matching -step to get wrong and no window where the address is loose. +A StatefulSet returns a volume to a replaced pod this way. The claim is named +deterministically for the slot, the pod is deleted and recreated, and nothing touched +the claim, so the binding it holds is still the binding. No matching step can go +wrong and no window leaves the address loose. So the network layer names an interface's claims from the slot, the interface, and -the family — all of which are composed from the workload, placement, location, and -ordinal, and are therefore stable across every replacement filling that slot. A -replacement finds claims that already exist and already hold addresses. Deleting the -workload deletes the claims, through ownership, and that is when `reclaimPolicy` -decides whether the addresses are released or held. - -Two consequences worth stating. A late release arriving from an instance that has -already been replaced is rejected, because the release is checked against the claim -that currently holds the binding rather than against a remembered holder. And a -deleted workload recreated under the same name produces identical claim names, so it -inherits its predecessor's addresses if they were retained — the same recreation -question a network name raises, wanting the same answer. +the family. All three compose from the workload, placement, location, and ordinal, +and are therefore stable across every replacement filling that slot. A replacement +finds claims that already exist and already hold addresses. Deleting the workload +deletes the claims through ownership, and `reclaimPolicy` then decides whether the +addresses are released or held. + +Two consequences follow: + +- **A late release from an already-replaced instance is rejected.** The release is + checked against the claim that currently holds the binding, not against a + remembered holder. +- **A deleted workload recreated under the same name produces identical claim + names.** It inherits its predecessor's retained addresses — the same recreation + question a network name raises, wanting the same answer. A retained allocation still needs an expiry. An address held against a location's -public range takes that range out of service for everyone, so it carries a lease, +public range takes that range out of service for everyone. So it carries a lease, keeps consuming its holder's budget while it lives, and can be force-released by an operator with an audit record. -The one part of the storage model not to copy is what happens next. A `Retain` -volume whose claim is deleted becomes `Released` and cannot be bound again until -someone clears the stale reference by hand. A retained address must return to a -state something can claim, without an operator in the path. +One part of the storage model should not be copied: a `Retain` volume whose claim is +deleted becomes `Released` and cannot be bound again until someone clears the stale +reference by hand. A retained address must return to a claimable state without an +operator in the path. ### A workload in two locations A consumer runs one workload on one network in `us-central-1` and `eu-west-1`, two -replicas each, dual-stack, with a public address per instance. Here is everything -that exists, and where. +replicas each, dual-stack, with a public address per instance. The sections below +list everything that exists, and where. #### The platform, authored once -Five classes and the pools that back them. Consumers never see these objects; they -see the names. +Five classes and the pools that back them. Consumers see the names, never these +objects. ```yaml # The tenant chain. The first two classes are containers: nothing claims them @@ -670,12 +677,12 @@ metadata: ipam.miloapis.com/is-default-class: "true" spec: ipFamily: IPv4 - # No parentClassName, and that is the whole IPv4 story: there is no per-network - # IPv4 space to carve, so an endpoint draws straight from the location's shared - # range. The IPv6 endpoint above sits three levels down its own chain; this one - # is a chain of one. - # The shared range makes this a real narrowing — two networks reach the same - # address and both keep it. + # No parentClassName, which is the whole IPv4 story: no per-network IPv4 space + # exists to carve, so an endpoint draws straight from the location's shared + # range. The IPv6 endpoint above sits three levels down its chain; this is a + # chain of one. + # Sharing that range makes uniqueWithin load-bearing here — two networks reach + # the same address and both keep it. uniqueWithin: - network allowedPrefixLengths: @@ -776,12 +783,12 @@ IPPool/network-default fd20:a1b:2c3d::/48 └── IPPool/network-default-eu-west-1 fd20:a1b:2c3d:2::/64 location: eu-west-1 ``` -These are project-scoped, so the consumer can see what their network holds and how -much of it is used. The location subnets appear on first use — a location the +These pools are project-scoped, so the consumer can see what their network holds and +how much of it is used. The location subnets appear on first use — a location the workload never runs in never gets one — and are never renumbered afterwards. -Notice there is no IPv4 equivalent. IPv4 endpoints come from the location's shared -range directly, which is why the two families resolve different classes. +Note the missing IPv4 equivalent. IPv4 endpoints come from the location's shared range +directly. #### Each location @@ -796,9 +803,9 @@ us-central-1 eu-west-1 ``` Each interface claim carries the consumer's intent — network `default`, families -`[IPv6, IPv4]`, class `public-unicast-ipv4`, retain — and the network layer turns -each into three claims: one per family, plus the named public class. It supplies -the location itself, because it is one. +`[IPv6, IPv4]`, class `public-unicast-ipv4`, retain. The network layer turns each +into three claims: one per family, plus the named public class. It supplies the +location itself, because it is one. #### What comes back @@ -814,9 +821,9 @@ eu-west-1 ``` Allocation starts at the second block of each subnet because the first holds the -gateway and the subnet's all-zeros address. And **a public address is per instance, -per location** — four replicas means four routable addresses out of two locations' -space, which is a cost and a quota consequence that belongs in the request rather +gateway and the subnet's all-zeros address. Note also that **a public address is per +instance, per location**: four replicas means four routable addresses out of two +locations' space. That cost and its quota consequence belong in the request rather than in a later discovery. The addresses land on each instance and travel back to the project control plane, @@ -825,79 +832,71 @@ which is the only place the consumer looks. #### What the shape shows The IPv6 address is carved from a `/64` belonging to this network alone; the IPv4 -address comes from a `/20` every network in that location draws from. Same -interface, same request, two genuinely different arrangements — which is why each -family resolves its own class. - -That difference is what `uniqueWithin` states: **an IPv6 endpoint block is unique -platform-wide because the network's prefix is; an IPv4 address is compared only -within its network.** Two networks can hold `10.128.0.2` in `us-central-1` at once -and never meet, because the routing domain separates them — and reaching across -locations comes from that same routing domain, for both families, not from the -prefixes being contiguous. +address comes from a `/20` every network in that location draws from. That is what +`uniqueWithin` states: **an IPv6 endpoint block is unique platform-wide because the +network's prefix is; an IPv4 address is compared only within its network.** Two +networks can hold `10.128.0.2` in `us-central-1` at once and never meet, because the +routing domain separates them. Reaching across locations comes from that same routing +domain, for both families, not from the prefixes being contiguous. -It also carries a ceiling worth stating as a product fact: a network cannot exceed -roughly four thousand IPv4 endpoints in one location, because every network draws -from the same location-wide range. IPv6 has no comparable limit. +The shared IPv4 range sets a ceiling worth stating as a product fact: a network cannot +exceed roughly four thousand IPv4 endpoints in one location. IPv6 has no comparable +limit. -Removing a placement releases the addresses its instances held, but not the -location's subnet — that belongs to the network, and other workloads on it draw -from the same subnet. +Removing a placement releases the addresses its instances held, but not the location's +subnet. The subnet belongs to the network, and other workloads on it draw from it. ## What this depends on -Allocating an address is necessary and nowhere near sufficient. These are the pieces -the design assumes and does not provide. Listing them is the point — a design that -quietly assumed them would look finished and behave otherwise. +Allocating an address is necessary and nowhere near sufficient. The pieces below are +what the design assumes and does not provide. Listing them is the point: a design +that quietly assumed them would look finished and behave otherwise. **A [network](https://github.com/datum-cloud/network-services-operator/blob/main/api/v1alpha/network_types.go) -needs one routing identity across every location it reaches**, unique -platform-wide, or the two halves of a multi-location workload are unrelated networks -sharing a name. This is the identity scoping route import and export — not the +needs one routing identity across every location it reaches**, unique platform-wide. +Otherwise the two halves of a multi-location workload are unrelated networks sharing +a name. This identity scopes route import and export. It is not the [per-location forwarding-instance identifier](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/srv6.md), -which is deliberately reused in every location and is a much smaller space. -Conflating the two would cap the platform at a few thousand networks in total -rather than per location. +which is deliberately reused in every location and draws on a much smaller space. +Conflating the two would cap the platform at a few thousand networks in total rather +than per location. **A moved instance needs its old route withdrawn before the new one is trusted.** Each node advertises with a distinct identity, so a route reflector keeps both -advertisements when an instance moves, and traffic splits between the node that has -it and the node that does not. Retention makes this worse by keeping the address -valid across the move. The routes need a sequence number so the newer advertisement -demonstrably wins. +advertisements when an instance moves and traffic splits between the two nodes. +Retention makes the split worse by keeping the address valid across the move. The +routes need a sequence number so the newer advertisement demonstrably wins. **Endpoint reachability has a per-node cost quadratic in network size.** Reaching a -remote endpoint installs per-endpoint state on every node that talks to it, and that -state is not reclaimed automatically. It is the real ceiling — far below any -address-space limit — and it needs a stated budget and a cap on endpoints per -network per node. +remote endpoint installs per-endpoint state on every node that talks to it, and +nothing reclaims that state. This is the real ceiling, far below any address-space +limit, and it needs a stated budget and a cap on endpoints per network per node. **Subnets need programming, not just allocation.** A location's subnet appearing on -first use currently means a record is written; nothing provisions the gateway, the -forwarding instance, or the route-table entry. That is why the interface reports +first use writes a record and nothing more; nothing provisions the gateway, the +forwarding instance, or the route-table entry. That gap is why the interface reports `Allocated` and `Programmed` separately. -**The gateway still needs programming.** Reservations produce a real allocation held -by the subnet rather than a hole owned by nothing, which gives it an owner, puts it -in inventory, and gives path-MTU discovery a source address inside the network — but -allocating it does not configure it. Without a gateway that answers, oversized +**The gateway still needs programming.** Reservations give the gateway an owner, put +it in inventory, and give path-MTU discovery a source address inside the network — but +allocating a gateway does not configure it. Without a gateway that answers, oversized packets are dropped silently: handshakes succeed and large transfers hang. -**An endpoint's block is only reachable at its first address.** The block a class -hands an interface is flattened to a single address before distribution, so an -address self-assigned inside it works locally and nowhere else. Until distribution -preserves the block, "assigns within it" is aspirational. +**An endpoint's block is only reachable at its first address.** Distribution flattens +the block to a single address, so an address self-assigned inside the block works +locally and nowhere else. Until distribution preserves the block, "assigns within it" +is aspirational. -**Public addresses need a path to the instance** — advertisement, in-location -steering to the node holding it, and translation — and it has to move when the +**Public addresses need a path to the instance** — advertisement, in-location steering +to the node holding the address, and translation — and that path must move when the instance is rescheduled. A released public address also needs a quarantine before -reissue: no route changes when it is handed to someone else, but DNS caches, +reissue: no route changes when the address goes to someone else, but DNS caches, customer allowlists, and reputation data all still point the old way. **Consuming a class must be a privilege.** Once consumers name classes instead of -pools, the class name is the only authorization boundary left. Naming a class must -be checked, and it must fail closed. +pools, the class name is the only authorization boundary left. Naming a class must be +checked, and the check must fail closed. ## Drawbacks @@ -905,11 +904,11 @@ be checked, and it must fail closed. if measurement justifies it, is a small reserve per location. - **More concepts.** Consumers gain a name to think about, operators gain a catalog to curate. Per-family defaults keep the common path free of both. -- **A held address is capacity nobody else can use.** That is the price of an - address that survives a redeploy, and on a finite public range it is the cost that - matters — which is why retention carries a lease rather than lasting forever. -- **A public address per instance per location adds up.** The design makes that - explicit rather than hiding it, but it is a real cost consumers will meet. +- **A held address is capacity nobody else can use.** That is the price of an address + that survives a redeploy, and on a finite public range it is the cost that matters. + Retention therefore carries a lease rather than lasting forever. +- **A public address per instance per location adds up.** The design makes the cost + explicit rather than hiding it, but consumers will still meet it. ## Alternatives @@ -926,25 +925,25 @@ be checked, and it must fail closed. - **Class-level utilization maintained during allocation.** Rejected: it makes one row every pool of a class contends on, and cannot express the per-location number that actually matters. -- **A class field naming what identifies an allocation.** Two shapes of this were - tried: an enum of the cases (`PerClaim`, `PerNetwork`, `PerNetworkLocation`) and - then a list of scope references. Both let the allocator re-derive which allocation - a claim should get. Rejected because nothing needs to re-derive it — a claim binds - an allocation and records it, so the binding is a fact rather than a computation. - Storage settled this a decade ago and neither `PersistentVolumeClaim` nor - `PersistentVolume` carries such a field. The enum had a second problem: it needed - a new value for every kind of thing that can hold an address, which would have put - nodes and sites inside the allocator. -- **Retention by rebinding rather than by not unbinding.** Releasing an address when - an instance is deleted and re-matching it when the replacement appears is how the - design read before. Rejected: it opens a window where the address is loose, needs - a durable identity distinct from the holder to match on, and lands in the state +- **A class field naming what identifies an allocation.** Two shapes were tried: an + enum of the cases (`PerClaim`, `PerNetwork`, `PerNetworkLocation`), then a list of + scope references. Both let the allocator re-derive which allocation a claim should + get. Rejected because nothing needs to re-derive it — a claim binds an allocation + and records it, so the binding is a fact rather than a computation. Storage settled + this a decade ago, and neither `PersistentVolumeClaim` nor `PersistentVolume` + carries such a field. The enum had a second problem: it needed a new value for + every kind of thing that can hold an address, which would have put nodes and sites + inside the allocator. +- **Retention by rebinding rather than by not unbinding.** An earlier draft released + an address when an instance was deleted and re-matched it when the replacement + appeared. Rejected: it opens a window where the address is loose, it needs a + durable identity distinct from the holder to match on, and it lands in the state storage calls `Released`, where a retained volume needs manual intervention before anything can bind it again. Keeping the claim is strictly simpler. -- **Reserved positions as policy rather than allocations.** Rejected: it leaves - space that is owned by nothing, absent from inventory, and impossible to program, - which is the specific complaint this document makes about the subnet gateway. It - also cannot express a reservation that is not at the start or end of the parent. +- **Reserved positions as policy rather than allocations.** Rejected: it leaves space + owned by nothing, absent from inventory, and impossible to program — the specific + complaint this document makes about the subnet gateway. It also cannot express a + reservation away from the start or end of the parent. ## Open Questions @@ -953,21 +952,21 @@ currently defaults to IPv4 only while interfaces are proposed to default to IPv6 Those cannot both be right, and the mismatch surfaces as an interface requesting a family its network does not carry. -**What is the interface's configured prefix length and next-hop model?** It decides -whether the gateway is on-link, whether reservations protect anything real, and how -much per-node endpoint state each interface costs. Every reservation question -resolves once it is answered. +**What is the interface's configured prefix length and next-hop model?** The answer +decides whether the gateway is on-link, whether reservations protect anything real, +and how much per-node endpoint state each interface costs. Every reservation question +resolves once it is settled. **Where does export policy live?** A class is shared by every consumer that names it, so per-consumer export rules cannot live on one. Anything beyond "advertised or not" needs a per-holder object the class points at. -**Should a container class be an `IPClass` at all?** `tenant-network-ipv6` and -`tenant-subnet-ipv6` are never named by a claim; they exist to provision pools. They -share most of a class's fields, which is the argument for one kind — but a consumer -listing classes should not see them, and `poolPer` is meaningless on everything -else. A separate kind, or a marker on the class, may read better than a field that -only applies to half of them. +**Should a container class be an `IPClass` at all?** No claim ever names +`tenant-network-ipv6` or `tenant-subnet-ipv6`; they exist to provision pools. They +share most of a class's fields, which argues for one kind. But a consumer listing +classes should not see them, and `poolPer` is meaningless on every other class. A +separate kind, or a marker on the class, may read better than a field that applies to +half of them. ## References From 0799f33cb141dc8fb6c17795b292262b8c0a4499 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 13:16:04 -0500 Subject: [PATCH 07/15] docs: condense the dependencies section into a list Merges the two items that stated the same dependency -- a subnet's record being written without the gateway, forwarding instance, or route-table entry being provisioned -- into one, and drops the restatement of why reservations produce a real allocation, which the reservations field already covers. Converts the remaining seven to a list, since the section is an inventory of assumptions rather than an argument that builds. --- docs/enhancements/ipam-integration.md | 78 +++++++++++---------------- 1 file changed, 32 insertions(+), 46 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index da0907ff..449ca288 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -849,54 +849,40 @@ subnet. The subnet belongs to the network, and other workloads on it draw from i ## What this depends on -Allocating an address is necessary and nowhere near sufficient. The pieces below are -what the design assumes and does not provide. Listing them is the point: a design +Allocating an address is necessary and nowhere near sufficient. The design assumes +each of the following and provides none of them. Listing them is the point: a design that quietly assumed them would look finished and behave otherwise. -**A [network](https://github.com/datum-cloud/network-services-operator/blob/main/api/v1alpha/network_types.go) -needs one routing identity across every location it reaches**, unique platform-wide. -Otherwise the two halves of a multi-location workload are unrelated networks sharing -a name. This identity scopes route import and export. It is not the -[per-location forwarding-instance identifier](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/srv6.md), -which is deliberately reused in every location and draws on a much smaller space. -Conflating the two would cap the platform at a few thousand networks in total rather -than per location. - -**A moved instance needs its old route withdrawn before the new one is trusted.** -Each node advertises with a distinct identity, so a route reflector keeps both -advertisements when an instance moves and traffic splits between the two nodes. -Retention makes the split worse by keeping the address valid across the move. The -routes need a sequence number so the newer advertisement demonstrably wins. - -**Endpoint reachability has a per-node cost quadratic in network size.** Reaching a -remote endpoint installs per-endpoint state on every node that talks to it, and -nothing reclaims that state. This is the real ceiling, far below any address-space -limit, and it needs a stated budget and a cap on endpoints per network per node. - -**Subnets need programming, not just allocation.** A location's subnet appearing on -first use writes a record and nothing more; nothing provisions the gateway, the -forwarding instance, or the route-table entry. That gap is why the interface reports -`Allocated` and `Programmed` separately. - -**The gateway still needs programming.** Reservations give the gateway an owner, put -it in inventory, and give path-MTU discovery a source address inside the network — but -allocating a gateway does not configure it. Without a gateway that answers, oversized -packets are dropped silently: handshakes succeed and large transfers hang. - -**An endpoint's block is only reachable at its first address.** Distribution flattens -the block to a single address, so an address self-assigned inside the block works -locally and nowhere else. Until distribution preserves the block, "assigns within it" -is aspirational. - -**Public addresses need a path to the instance** — advertisement, in-location steering -to the node holding the address, and translation — and that path must move when the -instance is rescheduled. A released public address also needs a quarantine before -reissue: no route changes when the address goes to someone else, but DNS caches, -customer allowlists, and reputation data all still point the old way. - -**Consuming a class must be a privilege.** Once consumers name classes instead of -pools, the class name is the only authorization boundary left. Naming a class must be -checked, and the check must fail closed. +- **A [network](https://github.com/datum-cloud/network-services-operator/blob/main/api/v1alpha/network_types.go) + needs one routing identity across every location it reaches**, unique + platform-wide, or the two halves of a multi-location workload are unrelated + networks sharing a name. This identity scopes route import and export. It is not + the [per-location forwarding-instance identifier](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/srv6.md), + which is deliberately reused in every location; conflating the two would cap the + platform at a few thousand networks in total rather than per location. +- **A moved instance needs its old route withdrawn before the new one is trusted.** + A route reflector keeps both advertisements and traffic splits between the two + nodes. Retention makes the split worse by keeping the address valid across the + move. The routes need a sequence number so the newer advertisement wins. +- **Endpoint reachability costs per-node state quadratic in network size**, and + nothing reclaims it. This is the real ceiling, far below any address-space limit, + and it needs a stated budget and a cap on endpoints per network per node. +- **Subnets and their gateways need programming, not just allocation.** A location's + subnet appearing on first use writes a record and nothing more; nothing provisions + the gateway, the forwarding instance, or the route-table entry. That gap is why the + interface reports `Allocated` and `Programmed` separately. Without a gateway that + answers, oversized packets are dropped silently: handshakes succeed and large + transfers hang. +- **An endpoint's block is only reachable at its first address.** Distribution + flattens the block to a single address, so "assigns within it" stays aspirational + until distribution preserves the block. +- **Public addresses need a path to the instance** — advertisement, in-location + steering, and translation — and that path must move when the instance is + rescheduled. A released public address also needs a quarantine before reissue: + DNS caches, customer allowlists, and reputation data all still point the old way. +- **Consuming a class must be a privilege.** Once consumers name classes instead of + pools, the class name is the only authorization boundary left. The check must fail + closed. ## Drawbacks From 7349987e321c42ba46592443498ea4440887481b Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 13:59:46 -0500 Subject: [PATCH 08/15] docs: cut duplicated material Drops the User Stories section. Its six stories restated the five goals almost one for one; the only capability they carried that Goals did not -- re-backing a class by attaching a pool and draining the old one -- moves into Goals. Collapses the storage analogy to one statement. It was made five times: in the class field comment, in the claim name and address comments, in the section that explains binding, and again in Alternatives. Only the binding section needs it, and the volumeName parallel now sits beside the field it describes. Cuts the paragraph redefining pool and allocation, which the terminology list above it already defines, and shortens the three Alternatives entries that re-argued the body at full length while the other five were one-liners. No design changes. --- docs/enhancements/ipam-integration.md | 264 +++++++++++--------------- 1 file changed, 107 insertions(+), 157 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 449ca288..451f1239 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -12,7 +12,6 @@ latest-milestone: "v0.x" - [Non-Goals](#non-goals) - [Proposal](#proposal) - [What it feels like](#what-it-feels-like) - - [User Stories](#user-stories) - [Notes/Constraints/Caveats](#notesconstraintscaveats) - [Design Details](#design-details) - [Class configuration](#class-configuration) @@ -65,13 +64,14 @@ capability lands better than one introduced after consumption is established. ### Goals -- Let a consumer request address space by naming a class, with no knowledge of - pools or topology. +- Let a consumer request address space by naming a class, with no knowledge of pools + or topology. - Let a consumer hold an address across a redeploy. -- Track every address the platform assigns to a workload, from allocation to - release. -- Give operators one inventory: what exists, what is used, who holds it, and how - much of each class is left. +- Track every address the platform assigns to a workload, from allocation to release. +- Give operators one inventory: what exists, what is used, who holds it, and how much + of each class is left. +- Let an operator move a class onto new space by attaching a pool and draining the + old one, with no consumer change. - Make address space a unit that quota and utilization can reason about. ### Non-Goals @@ -80,11 +80,9 @@ capability lands better than one introduced after consumption is established. underlay links, and the per-site blocks they come from are platform-internal, have no consumer, and are covered by the [fabric addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/fabric.md). - Fabric addressing is out of scope for this document but not for the model it - proposes: those blocks form a hierarchy of prefixes scoped by sites, nodes, and - links — opaque references like any other — and a long-lived claim per node has the - same shape as a long-lived claim per workload slot. The design was checked against - that plan, and nothing here needs to change to carry it. + They are out of scope for this document but not for the model it proposes: they form + a hierarchy of prefixes scoped by sites, nodes, and links — opaque references like + any other. The design was checked against that plan and needs no change to carry it. - **Non-address numbering.** AS numbers, forwarding-instance identifiers, and MAC assignments are allocatable resources with the same claim semantics, but a class as designed here is prefix-shaped. They need a sibling model. @@ -192,21 +190,6 @@ The last column is the number that matters. Averaged across locations, a class always reads healthy; one location filling up is what pages someone. The view reports the worst occupant rather than the mean. -### User Stories - -- **As a developer**, I ask for a public address by naming a class, and I keep it - when I redeploy — so the endpoint I published to customers stays valid. -- **As a developer**, I get IPv6 by default and add IPv4 to the same interface when - I need to reach something that has not moved yet. -- **As a platform operator**, I define what `public-unicast-ipv4` means once — - which space backs it, how it is advertised, what happens on release — and every - team consumes it by name. -- **As a platform operator**, I move a class onto new space by attaching a pool and - draining the old one, with no consumer change. -- **As an operator on call**, I can answer "who has this address" in one command. -- **As a governance owner**, address space appears in quota in terms people - recognise, so it can be budgeted like anything else. - ### Notes/Constraints/Caveats - **Allocation is synchronous.** A claim returns its address in the create @@ -226,10 +209,8 @@ reports the worst occupant rather than the mean. ### Class configuration -This section defines each field and the rules the allocator applies. It is written -for platform operators authoring classes. - -This document uses the following terms consistently: +This section defines each field and the rules the allocator applies, using the +following terms consistently: - **Class**: the policy object naming a kind of address space. - **Pool**: a block of capacity many claims draw from, offering itself to one or @@ -273,13 +254,9 @@ spec: # declared ancestry. parentClassName: tenant-subnet-ipv6 - # Nothing here says which allocation a claim gets. An IPClaim binds to one - # IPAllocation and an IPAllocation to one IPClaim, exactly as a - # PersistentVolumeClaim binds to a PersistentVolume. The claim object is the - # identity. See "What a claim carries". - # - # A class that other classes carve from sets `poolPer` instead — see - # tenant-subnet-ipv6 in the worked example. + # Nothing here says which allocation a claim gets; see "How a claim finds its + # allocation". A class that other classes carve from sets `poolPer` instead — + # see tenant-subnet-ipv6 in the worked example. # What defines one independent address space. Two allocations may hold the # same address if, and only if, they differ in one of these references. @@ -346,14 +323,14 @@ spec: **How a claim finds its allocation.** It does not look one up. A claim binds to one allocation and an allocation to one claim, each recording the other, and the binding -is made once when the claim is created. A `PersistentVolumeClaim` binds to a -`PersistentVolume` the same way. Storage does not reconstruct which volume a claim -should get, and neither should this: the claim object *is* the identity. +is made once when the claim is created — exactly as a `PersistentVolumeClaim` binds +to a `PersistentVolume`. Storage does not reconstruct which volume a claim should +get, and neither should this: the claim object *is* the identity. Its optional +`address` field plays the part `volumeName` plays for storage. -That settles what the class has to say. Nothing on it selects an allocation, because -the claim already has one. The class carries only what the allocator needs to hand -out an address in the first place: which space it comes from, and what it must not -collide with. +So nothing on the class selects an allocation, because the claim already has one. The +class carries only what the allocator needs to hand out an address in the first place: +which space it comes from, and what it must not collide with. **What `uniqueWithin` means.** Both endpoint classes in the example below set `uniqueWithin: [network]`, and the field does different amounts of work in each. An @@ -372,17 +349,15 @@ references, so the claim carries them by role: ```yaml kind: IPClaim metadata: - # Deterministic, derived from the slot and the interface it is for. This name - # is what a replacement instance finds, so it is the durable identity — the - # same construction a StatefulSet uses to give a replaced pod its volume back. + # Deterministic, derived from the slot and the interface it is for. A + # replacement instance finds this name, which makes it the durable identity. name: hello-sandbox-americas-us-central-1-0-eth0-ipv4 ownerReferences: - kind: NetworkInterface name: … spec: className: tenant-endpoint-ipv4 - # Optional. Names a specific address to bind, the way a PersistentVolumeClaim - # names a volume. Omitted, the allocator chooses. + # Optional. Names a specific address to bind. Omitted, the allocator chooses. address: "" scope: network: @@ -395,35 +370,30 @@ spec: name: us-central-1 ``` -No consumer writes any of this. The network layer at the location supplies the -scope, because it is the only thing that knows it: it holds the deployment, so it -knows the location, and it resolved the interface's network reference before -claiming. The references are immutable, since a claim whose network or location -changed after allocation is incoherent. - -Roles are only names a class refers to. The allocator does not know what a network or -a location is; it knows that `tenant-endpoint-ipv4` is unique within whatever arrives -under `network`. So a class can be scoped by something this document never mentions. -That is how the same model covers infrastructure addressing, where the roles are -sites, nodes, and links, without any of those concepts entering the allocator. - -A name is enough, and nobody types an identifier. Every claim is already scoped to -the project it was made for, and a network name is unique within a project. So -`default` in one project and `default` in another are different address spaces -without the reference carrying anything extra — the same scoping every pool and -allocation already uses. - -The one case a name does not settle is a network deleted and recreated under the same -name. The new network inherits its predecessor's space and allocations, which is -usually what someone wants and occasionally not. Deciding otherwise means the -reference carries something stable across recreation rather than a name. Settle this -before retention ships, because that is where it starts to matter. +No consumer writes any of this. The network layer at the location supplies the scope, +because it is the only thing that knows it: it holds the deployment, so it knows the +location, and it resolved the interface's network reference before claiming. The +references are immutable, since a claim whose network or location changed after +allocation is incoherent. + +Roles are only names a class refers to. The allocator does not know what a network or a +location is; it knows that `tenant-endpoint-ipv4` is unique within whatever arrives +under `network`. So a class can be scoped by something this document never mentions — +which is how the same model covers infrastructure addressing, where the roles are +sites, nodes, and links. + +A name is enough, and nobody types an identifier. Every claim is already scoped to the +project it was made for, and a network name is unique within a project, so `default` +in one project and `default` in another are different address spaces. One case a name +does not settle is a network deleted and recreated under the same name: the new +network inherits its predecessor's space and allocations, which is usually what +someone wants and occasionally not. Settle that before retention ships, because that +is where it starts to matter. The allocator rejects a claim that omits a role its class names in `uniqueWithin` or -that its parent chain needs. Such a claim does not fall back to a wider comparison. A -wider comparison would look correct while refusing addresses the narrow one was meant -to allow, and the error would surface as unexplained exhaustion rather than a missing -field. +that its parent chain needs, rather than falling back to a wider comparison. A wider +comparison would look correct while refusing addresses the narrow one was meant to +allow, surfacing as unexplained exhaustion rather than a missing field. **Resolving a parent.** With `parentClassName` empty, the allocator takes every pool offering this class, discards those whose family differs, discards those @@ -433,11 +403,9 @@ With `parentClassName` set, it projects this claim's scope onto the parent class network `default` in `us-central-1`, the `tenant-subnet-ipv6` pool for that network and location. -Note what a parent is. A claim binds an allocation, but many allocations carve from a -**pool**. An allocation is what one claim holds; a pool is capacity many claims draw -from. So a parent class does not hand out addresses. It provisions pools, one per -distinct combination of its `poolPer` references. That is why `poolPer` appears only -on classes named as a parent, and why it is not a property of claims at all. +Note what a parent is. A parent class does not hand out addresses; it provisions +pools, one per distinct combination of its `poolPer` references. That is why `poolPer` +appears only on classes named as a parent, and why it is not a property of claims. If the pool does not exist, the allocator creates it first, applying the parent class's configuration. Creation cascades: a claim in a location a network has never @@ -465,12 +433,12 @@ Chain depth is capped and cycles are rejected at class-write time, not at claim that was asked for. **Class health is computed, never stored.** A class reports whether any pool backs it -and how full its worst location is. Both are aggregates over pool status, read at -query time. Neither is a counter maintained during allocation, and deliberately so: a -class is backed by many pools, so a class-level counter would be one row every pool -contends on. That would turn independent claims in different locations into a queue -and destroy the per-pool locking the service depends on. A counter also cannot express -"the worst location," which is the number that matters. +and how full its worst location is, both as aggregates over pool status read at query +time. Neither is a counter maintained during allocation, deliberately: a class-level +counter would be one row every pool of that class contends on, turning independent +claims in different locations into a queue and destroying the per-pool locking the +service depends on. A counter also cannot express "the worst location," which is the +number that matters. ### Address families @@ -478,44 +446,38 @@ and destroy the per-pool locking the service depends on. A counter also cannot e class for each.** The tempting alternative is one class spanning both families with sizes declared per -family. That alternative holds up until it meets the -[tenant addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md), -where the two families are not the same kind of thing. - +family. It holds up until it meets the +[tenant addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md). A tenant endpoint in IPv6 gets a block carved from **that tenant's own prefix**, -unique to them, which the instance subdivides. In IPv4 it gets a single address from -a **location-wide range every tenant reuses**, with no sub-block, because IPv4 -scarcity makes per-tenant uniqueness impossible at scale. Different parent, different -uniqueness rule, different hierarchy. That is two classes serving one interface, not -one class with two sizes. +unique to them, which the instance subdivides. In IPv4 it gets a single address from a +**location-wide range every tenant reuses**, with no sub-block, because IPv4 scarcity +makes per-tenant uniqueness impossible at scale. Different parent, different uniqueness +rule, different hierarchy — two classes serving one interface, not one class with two +sizes. -So the interface names families and each family resolves to a class. Where the -consumer names no class — the common case — the platform's default for that family -applies. A consumer names a class only for something non-default, and names one per -family to get that in both. +So the interface names families and each family resolves to a class. Where the consumer +names no class — the common case — the platform's default for that family applies. **The family belongs in the name.** Naming a class means choosing a family, so -`public-unicast-ipv4`, not `public`. The rule reaches past the family. `unicast` +`public-unicast-ipv4`, not `public`. The rule reaches past the family: `unicast` appears because a consumer can really choose the alternative, and the two behave -visibly differently: a unicast address is one per instance per location, an anycast +visibly differently — a unicast address is one per instance per location, an anycast address is one address live everywhere. Name the property where the consumer chooses -between real alternatives, and leave it out where only one exists. Tenant addressing -is never advertised, so it carries no routing qualifier. +between real alternatives, and leave it out where only one exists. Tenant addressing is +never advertised, so it carries no routing qualifier. ### Where the address comes from **One allocator, in the middle, for everything.** Not a copy per location. The tempting alternative pushes allocation out to each location so it keeps working -alone. That alternative does not pay for itself. The high-volume cases that seem to -need it are not ours, because pod addresses belong to container networking; what -remains is a few addresses per interface at instance-creation rate. A copy per -location would mean a database at every location, two versions of the truth, and a -reconciliation problem nobody has scoped. - -It would also cost the thing this design is for. One allocator is the only way to -answer "who has this address" across the platform, enforce quota on the real resource, -and report utilization honestly. +alone. It does not pay for itself. The high-volume cases that seem to need it are not +ours, because pod addresses belong to container networking; what remains is a few +addresses per interface at instance-creation rate. A copy per location would mean a +database at every location, two versions of the truth, and a reconciliation problem +nobody has scoped. It would also give up the platform-wide inventory that motivates +the work — the only way to answer "who has this address," enforce quota on the real +resource, and report utilization honestly. State the trade plainly: **while the central service is unreachable, no new addresses are handed out.** A location cannot start a new instance. Live traffic is untouched — @@ -525,8 +487,8 @@ creation already carries the same dependency on to matter, the answer is a small pre-reserved buffer per location: a cache, not a second allocator. -**Claims are made where the work lands.** A consumer declares intent in their -project. That intent +**Claims are made where the work lands.** A consumer declares intent in their project. +That intent [travels to the location the workload is placed at](federated-deployment-scheduling.md), and the network layer there turns it into a claim, because that layer knows the location, the network, and the family. A consumer never writes a location, since the @@ -558,30 +520,25 @@ within it, so tracking costs one record per interface rather than one per contai Three things follow. -**A claim must be able to ask for a specific address.** A claim asks for a size, not -an address. Two cases need a claim that names one: recording an address already in -use, and handing a specific address back deliberately. The `address` field above -plays the part `volumeName` plays for storage. +**A claim must be able to ask for a specific address.** A claim normally asks for a +size. Two cases need one that names an address: recording an address already in use, +and handing a specific address back deliberately. **The runtime stops choosing.** An instance's address arrives with its interface -configuration rather than being invented at boot, and the container platform's own -address stops standing in for it. That is a change to the runtime contract, not -just to what the platform records. +configuration rather than being invented at boot. That is a change to the runtime +contract, not just to what the platform records. **The claim outlives the instance.** An address survives a redeploy because nothing -released it — not because anything reconstructs who used to hold it. - -A StatefulSet returns a volume to a replaced pod this way. The claim is named -deterministically for the slot, the pod is deleted and recreated, and nothing touched -the claim, so the binding it holds is still the binding. No matching step can go -wrong and no window leaves the address loose. +released it, not because anything reconstructs who used to hold it — the same reason a +StatefulSet returns a volume to a replaced pod. No matching step can go wrong and no +window leaves the address loose. -So the network layer names an interface's claims from the slot, the interface, and -the family. All three compose from the workload, placement, location, and ordinal, -and are therefore stable across every replacement filling that slot. A replacement -finds claims that already exist and already hold addresses. Deleting the workload -deletes the claims through ownership, and `reclaimPolicy` then decides whether the -addresses are released or held. +So the network layer names an interface's claims from the slot, the interface, and the +family, all of which compose from the workload, placement, location, and ordinal and +are stable across every replacement filling that slot. A replacement finds claims that +already exist and already hold addresses. Deleting the workload deletes the claims +through ownership, and `reclaimPolicy` then decides whether the addresses are released +or held. Two consequences follow: @@ -592,10 +549,10 @@ Two consequences follow: names.** It inherits its predecessor's retained addresses — the same recreation question a network name raises, wanting the same answer. -A retained allocation still needs an expiry. An address held against a location's -public range takes that range out of service for everyone. So it carries a lease, -keeps consuming its holder's budget while it lives, and can be force-released by an -operator with an audit record. +A retained allocation still needs an expiry, because an address held against a +location's public range takes that range out of service for everyone. It carries a +lease, keeps consuming its holder's budget, and can be force-released by an operator +with an audit record. One part of the storage model should not be copied: a `Retain` volume whose claim is deleted becomes `Released` and cannot be bound again until someone clears the stale @@ -911,25 +868,18 @@ that quietly assumed them would look finished and behave otherwise. - **Class-level utilization maintained during allocation.** Rejected: it makes one row every pool of a class contends on, and cannot express the per-location number that actually matters. -- **A class field naming what identifies an allocation.** Two shapes were tried: an - enum of the cases (`PerClaim`, `PerNetwork`, `PerNetworkLocation`), then a list of - scope references. Both let the allocator re-derive which allocation a claim should - get. Rejected because nothing needs to re-derive it — a claim binds an allocation - and records it, so the binding is a fact rather than a computation. Storage settled - this a decade ago, and neither `PersistentVolumeClaim` nor `PersistentVolume` - carries such a field. The enum had a second problem: it needed a new value for - every kind of thing that can hold an address, which would have put nodes and sites - inside the allocator. -- **Retention by rebinding rather than by not unbinding.** An earlier draft released - an address when an instance was deleted and re-matched it when the replacement - appeared. Rejected: it opens a window where the address is loose, it needs a - durable identity distinct from the holder to match on, and it lands in the state - storage calls `Released`, where a retained volume needs manual intervention before - anything can bind it again. Keeping the claim is strictly simpler. +- **A class field naming what identifies an allocation**, tried first as an enum + (`PerClaim`, `PerNetwork`, `PerNetworkLocation`) and then as a list of scope + references. Rejected: both let the allocator re-derive a binding that is already a + recorded fact, and the enum needed a new value for every kind of thing that can hold + an address, putting nodes and sites inside the allocator. +- **Retention by rebinding rather than by not unbinding.** Rejected: releasing an + address on instance deletion and re-matching it later opens a window where the + address is loose, needs a durable identity distinct from the holder, and lands in + the state storage calls `Released`. - **Reserved positions as policy rather than allocations.** Rejected: it leaves space - owned by nothing, absent from inventory, and impossible to program — the specific - complaint this document makes about the subnet gateway. It also cannot express a - reservation away from the start or end of the parent. + owned by nothing, absent from inventory, and impossible to program, and it cannot + express a reservation away from the start or end of the parent. ## Open Questions From edad97545924132b894b6328d3ee51a4cbd4b49d Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 14:07:17 -0500 Subject: [PATCH 09/15] docs: lead the allocator section with what it asserts Replaces "One allocator, in the middle, for everything. Not a copy per location." The second sentence pre-announced the paragraph directly below it, which introduces the per-location alternative and spends six lines rejecting it. And "in the middle" gestured at a call path it never stated. The lead now says where an address comes from, which is what the section is titled. --- docs/enhancements/ipam-integration.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 451f1239..43523416 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -468,7 +468,8 @@ never advertised, so it carries no routing qualifier. ### Where the address comes from -**One allocator, in the middle, for everything.** Not a copy per location. +**Every address comes from one central allocator**, whatever its class and wherever +the workload runs. The tempting alternative pushes allocation out to each location so it keeps working alone. It does not pay for itself. The high-volume cases that seem to need it are not From f705fefa63b54b47cd3a04210b714e10c607d263 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 14:09:31 -0500 Subject: [PATCH 10/15] docs: state the binding rule directly Replaces "How a claim finds its allocation. It does not look one up." The label asked a question the next fragment answered only by negation, and that fragment leaned on two pronouns whose referents were both in the label rather than the sentence. The lead now states the rule as an assertion, and the point that nothing recomputes the pairing follows it instead of preceding it. Updates the field comment that pointed at the old wording. --- docs/enhancements/ipam-integration.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 43523416..44cd823b 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -254,8 +254,8 @@ spec: # declared ancestry. parentClassName: tenant-subnet-ipv6 - # Nothing here says which allocation a claim gets; see "How a claim finds its - # allocation". A class that other classes carve from sets `poolPer` instead — + # Nothing here says which allocation a claim gets — a claim binds one when it + # is created. A class that other classes carve from sets `poolPer` instead; # see tenant-subnet-ipv6 in the worked example. # What defines one independent address space. Two allocations may hold the @@ -321,12 +321,12 @@ spec: - `parentPoolName` lets pools nest, so a continent's block contains its locations' ranges and stays summarisable as one route. -**How a claim finds its allocation.** It does not look one up. A claim binds to one -allocation and an allocation to one claim, each recording the other, and the binding -is made once when the claim is created — exactly as a `PersistentVolumeClaim` binds -to a `PersistentVolume`. Storage does not reconstruct which volume a claim should -get, and neither should this: the claim object *is* the identity. Its optional -`address` field plays the part `volumeName` plays for storage. +**A claim binds its allocation once, when the claim is created.** From then on the +claim and the allocation each record the other, exactly as a `PersistentVolumeClaim` +binds to a `PersistentVolume`. Nothing recomputes the pairing afterwards: storage does +not reconstruct which volume a claim should get, and neither should this — the claim +object *is* the identity. A claim's optional `address` field plays the part +`volumeName` plays for storage. So nothing on the class selects an allocation, because the claim already has one. The class carries only what the allocator needs to hand out an address in the first place: From df1001adc7cf09983ce1f7edb154e2a8614c1c88 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 14:18:19 -0500 Subject: [PATCH 11/15] docs: make the remaining label-led passages assert Applies the same fix to five more passages that named a topic and then made the reader assemble the claim. Two labels posed a question the text answered only by example or by restating the label: what uniqueWithin means, and what a claim carries. Both now state the rule. Drops the 'Note what a parent is' preamble and the 'State the trade plainly' imperative, and attaches a noun to the 'This' that opened the paragraph on reversing who picks an address. --- docs/enhancements/ipam-integration.md | 45 ++++++++++++++------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 44cd823b..058fdcac 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -332,19 +332,19 @@ So nothing on the class selects an allocation, because the claim already has one class carries only what the allocator needs to hand out an address in the first place: which space it comes from, and what it must not collide with. -**What `uniqueWithin` means.** Both endpoint classes in the example below set -`uniqueWithin: [network]`, and the field does different amounts of work in each. An -IPv6 endpoint is carved from a `/64` belonging to one network and no other, so the -parent already separates the space and the result is unique platform-wide regardless. -An IPv4 endpoint is carved from a range every network in the location shares, so the -setting is load-bearing: two networks reach the same address and both keep it. +**How much `uniqueWithin` does depends on the parent.** Both endpoint classes in the +example below set `uniqueWithin: [network]`. An IPv6 endpoint is carved from a `/64` +belonging to one network and no other, so the parent already separates the space and +the result is unique platform-wide regardless. An IPv4 endpoint is carved from a range +every network in the location shares, so the setting is load-bearing: two networks +reach the same address and both keep it. -Setting `uniqueWithin` wider than the parent requires is safe and wasteful. Setting -it narrower is how two holders end up with one address, which IPv4 tenant space wants -and nothing else does. +Setting it wider than the parent requires is safe and wasteful. Setting it narrower is +how two holders end up with one address, which IPv4 tenant space wants and nothing +else does. -**What a claim carries.** `uniqueWithin` and parent resolution key off the same -references, so the claim carries them by role: +**A claim carries its scope references by role**, since `uniqueWithin` and parent +resolution key off the same ones: ```yaml kind: IPClaim @@ -403,9 +403,9 @@ With `parentClassName` set, it projects this claim's scope onto the parent class network `default` in `us-central-1`, the `tenant-subnet-ipv6` pool for that network and location. -Note what a parent is. A parent class does not hand out addresses; it provisions -pools, one per distinct combination of its `poolPer` references. That is why `poolPer` -appears only on classes named as a parent, and why it is not a property of claims. +A parent class does not hand out addresses. It provisions pools, one per distinct +combination of its `poolPer` references, which is why `poolPer` appears only on +classes named as a parent and is not a property of claims. If the pool does not exist, the allocator creates it first, applying the parent class's configuration. Creation cascades: a claim in a location a network has never @@ -480,8 +480,8 @@ nobody has scoped. It would also give up the platform-wide inventory that motiva the work — the only way to answer "who has this address," enforce quota on the real resource, and report utilization honestly. -State the trade plainly: **while the central service is unreachable, no new addresses -are handed out.** A location cannot start a new instance. Live traffic is untouched — +The trade is plain: **while the central service is unreachable, no new addresses are +handed out.** A location cannot start a new instance. Live traffic is untouched — existing addresses keep working and routes keep being advertised — and instance creation already carries the same dependency on [central quota enforcement](quota-enforcement/README.md). If the outage window proves @@ -512,18 +512,19 @@ if at all. An address chosen that way cannot be held across a redeploy or counte against a budget. Reversing the order — **the platform decides the address, and the runtime is told** — turns an address into something a consumer can ask for and keep. -This is more tractable on the platform's own compute than against a third party. -Every address in play is space the platform already owns, so no external allocator -needs reconciling and the platform never records an address it did not issue. +This reversal is easier on the platform's own compute than it would be against a third +party. Every address in play is space the platform already owns, so no external +allocator needs reconciling and the platform never records an address it did not +issue. The unit is the interface, not the address: an interface gets a block and assigns within it, so tracking costs one record per interface rather than one per container. Three things follow. -**A claim must be able to ask for a specific address.** A claim normally asks for a -size. Two cases need one that names an address: recording an address already in use, -and handing a specific address back deliberately. +**A claim must be able to name a specific address.** A claim normally asks only for a +size, but two cases need one that names an address: recording an address already in +use, and handing a specific address back deliberately. **The runtime stops choosing.** An instance's address arrives with its interface configuration rather than being invented at boot. That is a change to the runtime From 6828399f019bedc18691a09615c9d646980d1151 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 14:21:40 -0500 Subject: [PATCH 12/15] docs: stop re-arguing rejected options in the body The body and Alternatives were making the same case twice. The section on where an address comes from listed a database per location, two sources of truth, and the loss of platform-wide inventory -- which is the Alternatives entry almost word for word. It now keeps only the argument Alternatives does not make: the volume that would justify per-location allocation is not ours, because pod addresses belong to container networking. Class health loses the same duplication, keeping the reason in one sentence rather than three. Address families now opens with the asymmetry between the two families rather than with the single-class option it rejects, since the asymmetry is the reason for the design and the rejection follows from it. --- docs/enhancements/ipam-integration.md | 36 +++++++++++---------------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 058fdcac..3a541c80 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -434,26 +434,22 @@ Chain depth is capped and cycles are rejected at class-write time, not at claim **Class health is computed, never stored.** A class reports whether any pool backs it and how full its worst location is, both as aggregates over pool status read at query -time. Neither is a counter maintained during allocation, deliberately: a class-level -counter would be one row every pool of that class contends on, turning independent -claims in different locations into a queue and destroying the per-pool locking the -service depends on. A counter also cannot express "the worst location," which is the -number that matters. +time. A counter maintained during allocation would serialize independent claims in +different locations behind one row, and could not express "the worst location" anyway. ### Address families **A class is single-family. The interface asks for families; the platform picks a class for each.** -The tempting alternative is one class spanning both families with sizes declared per -family. It holds up until it meets the -[tenant addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md). -A tenant endpoint in IPv6 gets a block carved from **that tenant's own prefix**, -unique to them, which the instance subdivides. In IPv4 it gets a single address from a -**location-wide range every tenant reuses**, with no sub-block, because IPv4 scarcity -makes per-tenant uniqueness impossible at scale. Different parent, different uniqueness -rule, different hierarchy — two classes serving one interface, not one class with two -sizes. +In the +[tenant addressing plan](https://github.com/datum-cloud/enhancements/blob/main/architecture/design/network/addressing/tenant.md), +the two families are not the same kind of thing. A tenant endpoint in IPv6 gets a +block carved from **that tenant's own prefix**, unique to them, which the instance +subdivides. In IPv4 it gets a single address from a **location-wide range every tenant +reuses**, with no sub-block, because IPv4 scarcity makes per-tenant uniqueness +impossible at scale. Different parent, different uniqueness rule, different hierarchy +— so one class spanning both families would be one object describing two things. So the interface names families and each family resolves to a class. Where the consumer names no class — the common case — the platform's default for that family applies. @@ -471,14 +467,10 @@ never advertised, so it carries no routing qualifier. **Every address comes from one central allocator**, whatever its class and wherever the workload runs. -The tempting alternative pushes allocation out to each location so it keeps working -alone. It does not pay for itself. The high-volume cases that seem to need it are not -ours, because pod addresses belong to container networking; what remains is a few -addresses per interface at instance-creation rate. A copy per location would mean a -database at every location, two versions of the truth, and a reconciliation problem -nobody has scoped. It would also give up the platform-wide inventory that motivates -the work — the only way to answer "who has this address," enforce quota on the real -resource, and report utilization honestly. +Pushing allocation out to each location, so a location keeps working alone, does not +pay for itself. The high-volume cases that would justify it are not ours: pod +addresses belong to container networking, and what remains is a few addresses per +interface at instance-creation rate. The trade is plain: **while the central service is unreachable, no new addresses are handed out.** A location cannot start a new instance. Live traffic is untouched — From 14b7e67a0ce83557f6bc9f21bb273efa18041fa9 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 14:24:44 -0500 Subject: [PATCH 13/15] docs: argue claim rate from this platform, not from pod networking The case against a per-location allocator rested on pod addresses belonging to container networking. There are no pods here, so the argument was borrowed rather than true: it described a Kubernetes CNI deployment, not this one. The real argument is already established twice in the document -- the unit is the interface, and an instance assigns its containers from the block its interface holds. So the control plane sees a few claims at instance creation and none during steady state, which is the rate that matters. --- docs/enhancements/ipam-integration.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 3a541c80..c972ff71 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -468,9 +468,10 @@ never advertised, so it carries no routing qualifier. the workload runs. Pushing allocation out to each location, so a location keeps working alone, does not -pay for itself. The high-volume cases that would justify it are not ours: pod -addresses belong to container networking, and what remains is a few addresses per -interface at instance-creation rate. +pay for itself. Only interfaces claim — an instance assigns its containers from the +block its interface already holds — so the control plane sees a few claims when an +instance is created and none while it runs. That is not a rate that needs a local +allocator. The trade is plain: **while the central service is unreachable, no new addresses are handed out.** A location cannot start a new instance. Live traffic is untouched — From 98cd0f8899f104e4a072cd41cd77fa63fab5206b Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 14:36:02 -0500 Subject: [PATCH 14/15] docs: argue claim rate from slot stickiness An address is claimed once for a slot and stays with it, so replacing, rescheduling, or redeploying an instance allocates nothing -- the replacement finds a claim that already holds its address. Instance churn therefore does not become claim churn, and allocation happens only when a slot first appears. This is what the retention design already says; the section on where an address comes from was arguing rate without it, first from pod networking and then from per-interface counting. Both missed that the mechanism keeping an address stable for a consumer is the same one keeping churn away from the allocator. Narrows the outage trade to match. A location cut off from the center can still replace and reschedule instances in existing slots, which is the path that matters during a failure; what pauses is scaling up and deploying something new. --- docs/enhancements/ipam-integration.md | 32 +++++++++++++++++---------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index c972ff71..69f61284 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -468,16 +468,23 @@ never advertised, so it carries no routing qualifier. the workload runs. Pushing allocation out to each location, so a location keeps working alone, does not -pay for itself. Only interfaces claim — an instance assigns its containers from the -block its interface already holds — so the control plane sees a few claims when an -instance is created and none while it runs. That is not a rate that needs a local -allocator. - -The trade is plain: **while the central service is unreachable, no new addresses are -handed out.** A location cannot start a new instance. Live traffic is untouched — -existing addresses keep working and routes keep being advertised — and instance -creation already carries the same dependency on -[central quota enforcement](quota-enforcement/README.md). If the outage window proves +pay for itself, because instance churn does not become claim churn. An address is +claimed once for a slot and stays with it, so replacing, rescheduling, or redeploying +an instance allocates nothing — the replacement finds a claim that already holds its +address. Allocation happens when a slot first appears: a new workload, or a scale-up. +See [Instance addresses](#instance-addresses) for how a slot keeps its address. + +That is what keeps the central allocator off the hot path, and it is worth noticing +that retention pays for itself twice: the mechanism that lets a consumer keep an +address is the same one that stops instance churn from reaching the allocator. + +The trade is narrower than it first looks: **while the central service is unreachable, +a location cannot allocate for a slot it has never seen.** It can still replace and +reschedule instances in slots that already hold addresses, which is the path that +matters during a failure. What pauses is scaling up and deploying something new — the +same dependency instance creation already has on +[central quota enforcement](quota-enforcement/README.md). Live traffic is untouched: +existing addresses keep working and routes keep being advertised. If the pause proves to matter, the answer is a small pre-reserved buffer per location: a cache, not a second allocator. @@ -838,8 +845,9 @@ that quietly assumed them would look finished and behave otherwise. ## Drawbacks -- **New allocation stops during a central outage.** Covered above; the mitigation, - if measurement justifies it, is a small reserve per location. +- **Scaling up and new deployments stop during a central outage.** Replacing an + instance in an existing slot does not, since its address is already claimed. Covered + above; the mitigation, if measurement justifies it, is a small reserve per location. - **More concepts.** Consumers gain a name to think about, operators gain a catalog to curate. Per-family defaults keep the common path free of both. - **A held address is capacity nobody else can use.** That is the price of an address From 8b85f9ebb96d0d670f678f87f85e794f47f7c397 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 6 Aug 2026 14:43:42 -0500 Subject: [PATCH 15/15] docs: account for scale-down releasing a slot's address A claim ends when its slot does, and the document only said that for workload deletion. A scale-down removes slots too, so under the default Delete policy scaling back up allocates different addresses; Retain is what makes a restored slot reclaim what it had. This also corrects the opening example, which labelled reclaimPolicy: Retain as what keeps addresses across redeploys. A redeploy keeps them either way, since it never deletes the claim -- the field comment on reclaimPolicy already said so, and the example contradicted it. Qualifies the claim-rate argument to match: most instance churn does not reach the allocator, but a slot returning after a scale-down does unless its address was retained. --- docs/enhancements/ipam-integration.md | 36 ++++++++++++++++----------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/docs/enhancements/ipam-integration.md b/docs/enhancements/ipam-integration.md index 69f61284..bb4cc62c 100644 --- a/docs/enhancements/ipam-integration.md +++ b/docs/enhancements/ipam-integration.md @@ -127,7 +127,8 @@ spec: ipFamilies: - IPv6 - IPv4 - # keep the addresses across redeploys + # hold the addresses when this scales down, so they come back + # (a redeploy keeps them either way) reclaimPolicy: Retain addresses: # a class, never an address @@ -468,15 +469,16 @@ never advertised, so it carries no routing qualifier. the workload runs. Pushing allocation out to each location, so a location keeps working alone, does not -pay for itself, because instance churn does not become claim churn. An address is -claimed once for a slot and stays with it, so replacing, rescheduling, or redeploying -an instance allocates nothing — the replacement finds a claim that already holds its -address. Allocation happens when a slot first appears: a new workload, or a scale-up. -See [Instance addresses](#instance-addresses) for how a slot keeps its address. +pay for itself, because most instance churn does not become claim churn. An address is +claimed for a slot rather than for the instance filling it, so replacing, rescheduling, +and redeploying allocate nothing — the replacement finds a claim that already holds its +address. Allocation happens when a slot appears that has no claim: a new workload, a +scale-up, or a slot returning after a scale-down released its address. `Retain` removes +that last case, holding the address for the slot that comes back. See +[Instance addresses](#instance-addresses) for what a slot keeps and when. -That is what keeps the central allocator off the hot path, and it is worth noticing -that retention pays for itself twice: the mechanism that lets a consumer keep an -address is the same one that stops instance churn from reaching the allocator. +Retention pays for itself twice, then: the mechanism that lets a consumer keep an +address is the same one that keeps churn away from the allocator. The trade is narrower than it first looks: **while the central service is unreachable, a location cannot allocate for a slot it has never seen.** It can still replace and @@ -538,17 +540,23 @@ window leaves the address loose. So the network layer names an interface's claims from the slot, the interface, and the family, all of which compose from the workload, placement, location, and ordinal and are stable across every replacement filling that slot. A replacement finds claims that -already exist and already hold addresses. Deleting the workload deletes the claims -through ownership, and `reclaimPolicy` then decides whether the addresses are released -or held. +already exist and already hold addresses. + +A claim ends when its slot does. Deleting the workload deletes its claims through +ownership, and so does a scale-down, for every slot it removes. `reclaimPolicy` then +decides what becomes of those addresses: `Delete` releases them, so scaling back up +allocates different ones; `Retain` holds them, so the restored slot reclaims what it +had. **An address survives a redeploy on its own, but surviving a scale-down takes +`Retain`.** Two consequences follow: - **A late release from an already-replaced instance is rejected.** The release is checked against the claim that currently holds the binding, not against a remembered holder. -- **A deleted workload recreated under the same name produces identical claim - names.** It inherits its predecessor's retained addresses — the same recreation +- **Anything recreated under the same name inherits what that name held.** A slot + restored by a scale-up and a workload recreated after deletion both produce + identical claim names, so both reclaim retained addresses — the same recreation question a network name raises, wanting the same answer. A retained allocation still needs an expiry, because an address held against a