fix(spawner): derive co-location affinity from the pod's username label - #211
Draft
aktech wants to merge 6 commits into
Draft
fix(spawner): derive co-location affinity from the pod's username label#211aktech wants to merge 6 commits into
aktech wants to merge 6 commits into
Conversation
kubespawner 7 computes the hub.jupyter.org/username label with a different
slug truncation than {username} template expansion, so templating the value
into the required pod affinity made it unsatisfiable for usernames that get
a hashed slug (emails): every spawn failed with FailedScheduling on all
nodes. Build the affinity in a modify_pod_hook from the pod's own label so
the values match under any slug scheme.
Covers why home volumes are ReadWriteOnce, the co-location pod affinity that RWO requires, and how to debug spawns that time out unscheduled.
Contributor
|
Docs preview for |
Resolves the overlap with the templated-label fix that landed on main: the affinity value is copied from the pod's rendered username label in modify_pod_hook rather than trusting extra_labels and extra_pod_config template expansions to render identically, which is undocumented kubespawner behavior. Keeps main's slug_scheme pin. Replaces the config-string tests with rendered-manifest tests that assert the affinity value equals the pod's own label under both slug schemes and that node affinity survives the hook.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the templated co-location affinity with a modify_pod_hook that copies the pod's rendered hub.jupyter.org/username label into the affinity value.
The templated approach (main, since #213) writes {username} into both extra_labels and extra_pod_config and relies on kubespawner expanding both identically. That is undocumented internal behavior of exactly the kind that caused the original outage: kubespawner 7 renders its username label and {username} through different code paths (the label ignores slug_scheme and truncates at 32; the template follows slug_scheme and truncates at 48), so any two-sided template equality can silently diverge on an upgrade. Copying the rendered label leaves one computation and nothing to disagree.
Also: preserves node affinity instead of overwriting the pod's affinity object, replaces the config-string tests with rendered-manifest tests (real kubespawner, email usernames, both slug schemes) that fail in a version-bump PR if the invariant breaks, and adds a User Pod Scheduling admin doc.
Refs:
also relevant issue to fix for future: #221