support managed control plane anti-affinity groups - #69
Conversation
sudomateo
left a comment
sudomateo
left a comment
There was a problem hiding this comment.
Thanks for working on this! I left some Go doc comment suggestions and asked some questions about adopting and deleting existing anti-affinity groups that I'll want to answer before merging.
| // ControlPlaneAntiAffinityPolicy, when set, makes the controller manage an anti-affinity | ||
| // group for the cluster's control plane machines so that their instances are spread across | ||
| // sleds. The group is created in the cluster's project before the cluster is marked | ||
| // provisioned, joined by every control plane instance at creation, and deleted with the | ||
| // cluster. "allow" permits co-location when the spread cannot be satisfied; "fail" refuses | ||
| // to place the instance instead. Note that the policy is applied when the group is first | ||
| // created: the Oxide API doesn't support updating a group's policy, so changing this field | ||
| // doesn't affect an existing group, and instances only join the group as they are created. | ||
| // https://docs.oxide.computer/guides/deploying-workloads#_affinity_and_anti_affinity |
There was a problem hiding this comment.
| // ControlPlaneAntiAffinityPolicy, when set, makes the controller manage an anti-affinity | |
| // group for the cluster's control plane machines so that their instances are spread across | |
| // sleds. The group is created in the cluster's project before the cluster is marked | |
| // provisioned, joined by every control plane instance at creation, and deleted with the | |
| // cluster. "allow" permits co-location when the spread cannot be satisfied; "fail" refuses | |
| // to place the instance instead. Note that the policy is applied when the group is first | |
| // created: the Oxide API doesn't support updating a group's policy, so changing this field | |
| // doesn't affect an existing group, and instances only join the group as they are created. | |
| // https://docs.oxide.computer/guides/deploying-workloads#_affinity_and_anti_affinity | |
| // ControlPlaneAntiAffinityPolicy specifies the anti-affinity group policy for | |
| // control plane instances. When set, the controller creates an anti-affinity | |
| // group in the cluster's project using the specified policy before marking the | |
| // cluster as provisioned. Each control plane instance joins the group when it | |
| // is created. The group is deleted with the cluster. | |
| // | |
| // Supported policies are: | |
| // | |
| // - "allow": permits an instance to start when anti-affinity cannot be | |
| // achieved. | |
| // - "fail": prevents an instance from starting when anti-affinity cannot be | |
| // achieved. | |
| // | |
| // Changing this field after the anti-affinity group has been created does not | |
| // change the group's policy because Oxide does not support updating it. | |
| // | |
| // https://docs.oxide.computer/guides/deploying-workloads#_affinity_and_anti_affinity |
| // AntiAffinityGroups is the list of anti-affinity group names or IDs that the Oxide instance | ||
| // should use. Anti-affinity groups should be created outside the operator, else instance | ||
| // creation will fail. | ||
| // should use. Anti-affinity groups specified should already be created, else instance | ||
| // creation will fail. A control plane anti-affinity group managed via the OxideCluster's | ||
| // ControlPlaneAntiAffinityPolicy field is joined automatically and doesn't need to be listed | ||
| // here. | ||
| // https://docs.oxide.computer/guides/deploying-workloads#_affinity_and_anti_affinity |
There was a problem hiding this comment.
// AntiAffinityGroups lists the names or IDs of existing Oxide anti-affinity
// groups that the instance joins when it is created. Instance creation fails if
// any referenced group does not exist.
//
// Anti-affinity for control plane instances is configured through
// [OxideClusterSpec.ControlPlaneAntiAffinityPolicy]. Do not list the control
// plane's anti-affinity group here.
//
// https://docs.oxide.computer/guides/deploying-workloads#_affinity_and_anti_affinity
| description: spec defines the desired state of OxideCluster | ||
| properties: | ||
| controlPlaneAntiAffinityPolicy: | ||
| description: |- |
There was a problem hiding this comment.
I'm unsure whether these are generated so if the Go doc comments syntax messes that up let me know.
| // Delete the control plane anti-affinity group unconditionally rather than only when the | ||
| // policy is currently set, so a group created before the policy was unset is cleaned up too. | ||
| // CAPI deletes the cluster's Machines before its infrastructure, so the group has no members | ||
| // by the time this runs. |
There was a problem hiding this comment.
This has the side effect of deleting an anti-affinity group that may have never been managed by CAPI to begin with. LLM scans flagged similar behavior in other integrations but it's a difficult problem to solve without state or resource tags.
- Can you elaborate on the set, unset, delete semantics here? When would a user unset the anti-affinity group setting for the control plane, or are you just accounting for that?
- We can't store the managed resource IDs anywhere, can we?
I'm really just asking to understand, not saying we have to change this. It may be a perfectly fine security boundary to say that we expect to manage resources that are prefixed with capi-.
| // ensureAntiAffinityGroupExists creates or views the control plane anti-affinity group. An | ||
| // existing group is adopted as is: the Oxide API doesn't support updating a group's policy, so | ||
| // drift between the group and ControlPlaneAntiAffinityPolicy isn't reconciled. |
There was a problem hiding this comment.
This is the other side of the delete comment. We could adopt some anti-affinity group that just so happened to be named the same.
jmcarp
left a comment
jmcarp
left a comment
There was a problem hiding this comment.
I haven't looked at the code yet, but this was a feature that I wasn't sure we should implement when I was first looking at anti-affinity support. I'm not against it, but my initial thought was that the provider should only manage the oxide resources that it has to manage. I think the question I want to answer is: what kinds of resources should capox manage, and what kinds should live in terraform or similar?
|
Are you saying that CAPOx should accept a list of existing anti-affinity groups rather than creating one, or not support anti-affinity at all? |
As of |
|
I was under the impression the existing list of anti-affinity groups was only for worker nodes but I may be incorrect there and it's actually for all nodes. I haven't implemented a CAPI provider before nor have I used it much before so I'm not sure what I would want it to manage for me as a customer. This sounds like a good agenda item for our weekly sync. |
3 participants
#67
CAPOx can now create and manage an Oxide anti-affinity group for control plane nodes of a cluster. Users are able to set the policy to allow | fail. If a policy is not specified, then no anti-affinity group is created.