refactor(cni): split galactic-cni into veth/tap master plugins - #303
refactor(cni): split galactic-cni into veth/tap master plugins#303privateip wants to merge 2 commits into
Conversation
…ation
This step rewired galactic-cni/galactic-tap-cni's own IPAM handling
from an in-process ipam.Allocate() call into real CNI IPAM delegation
(ipam.ExecAdd, execing whatever binary "ipam.type" names). The e2e
test's config still had the old "ipam": {"type": "pool"} block from
before this step -- "pool" is no longer a mode selector, it's now
looked up as a literal binary name, so ADD failed with "failed to find
plugin \"pool\" in path [/opt/cni/bin]" (confirmed on PR #303/#304's
own CI run, the first time e2e has actually executed against a live
cluster in this whole plugin-chain-split stack, now that a separate
pre-existing eBPF-artifact-drift issue blocking Build no longer blocks
it).
Fixed to "ipam": {"type": "galactic-ipam", "ipv6_subnet": "..."} --
the explicit contract this step establishes (see internal/cniipam's
doc comment). Also fixed CNI_PATH from /opt/cni/bin to / so
ipam.ExecAdd's lookup can actually find /galactic-ipam: every binary
in the chain lands at the image root
(containers/galactic-cni/Dockerfile), not /opt/cni/bin -- that path
only exists on a real host once installer.Bootstrap's init container
stages it there, which this test's pod never runs. Dropped
GALACTIC_CNI_ENABLE_LOCAL_IPAM=true from the script: it's now inert
for this config (an explicit ipv6_subnet doesn't need the local-IPAM
fallback), though harmless either way since this step doesn't remove
the env var itself.
BGP/SRv6/eBPF publish is still inline in galactic-tap-cni's own cmdAdd
at this point in the split (that only moves out to its own chained
binary in a later step), so the eBPF control daemon startup, bpf-fs
mount, and hostNetwork pod spec stay exactly as they were -- this
config still needs the same BGPRouter fixture and pre-pinned eBPF maps
the previous step's fix already established.
Verification: task lint (0 issues), go build/vet clean, task test:unit
full suite green. Not verified against a live cluster locally;
pushing to let CI's own test-e2e job confirm, same as the previous
step's fix.
"fd00:e2e:tap::/48" isn't a valid IPv6 literal -- "tap" contains 'p', not a hex digit. cniipam's own parseConf validates ipv6_subnet with net.ParseCIDR (see internal/cniipam/config.go's validateIPv6Subnet), so ADD failed with "invalid CIDR value for field 'ipam.ipv6_subnet'" as soon as this step's own IPAM-delegation fix (previous commit) made that field's value actually reach real validation for the first time -- it was never exercised before since PR #303/#304's e2e job never got this far until the eBPF-artifact-drift and wrong-binary-target issues in earlier commits on this branch were fixed. Fixed to "fd00:e2e::/48": drops the invalid "tap" hextet, keeps "e2e" (itself valid hex) as the mnemonic. Verification: task lint (0 issues), go build/vet clean, task test:unit full suite green, plus a plain net.ParseCIDR-equivalent check (Python's ipaddress.ip_network) confirming the literal parses.
|
Went through this one closely. The split looks solid overall: the shared-package extraction (hostconf, crdnames, nadpatch, cniipam, cnibgp) does what the description says, and tests moved with the code they cover. A few things worth looking at before this lands, roughly in order of how much I'd worry about them. First, Second, three files got copy-pasted instead of extracted, which cuts against the stated point of this refactor. The whole reason for pulling out hostconf, crdnames, nadpatch, cniipam, and cnibgp was so cnitap wouldn't duplicate or reach into cni's internals. But A future validation fix or rollback-ordering fix on the veth side won't propagate to any of those tap copies. Given five other packages already went through this same extraction, it seems worth doing the same here rather than letting two copies drift apart. A couple of smaller things:
Given this is a stack, happy to have the duplication cleanup land in a later branch if that's cleaner, but the isTapMode one seems worth addressing here since this branch ships first. |
|
Thanks for the thorough review, @mattdjenkinson — went through all five points against where this stack has landed since. Two are already handled without further changes needed here:
Missing nil- The other three are real and I've landed fixes for them as follow-ups on top of the stack rather than pushing more commits onto this branch, so your review here stays scoped to what you already looked at:
Both are green on |
Sequencing step 0 of the CNI plugin-chain split (design plan in ~/.vaults/notebook/projects/galactic/plan-cni-plugin-chain.md). galactic-cni is now veth-only; a new galactic-tap-cni binary owns tap-mode workloads (Kata, Firecracker, kraftlet/Unikraft). The interface_type config field and its runtime switch are gone entirely from both binaries — which binary runs is now the conflist-authoring decision, matching the design note's own rationale for the split. Getting there required extracting a few things this repo already had duplicated or coupled in ways that would have blocked genuine package separation: - internal/cni/hostconf: HostConf/loadHostConf/conflistEnvelope were duplicated near-verbatim between internal/cni and internal/installer, and hardcoded a match on plugin type "galactic-cni". Generalized into one shared package both binaries (and the installer) now call, with the accepted-type list passed in by the caller so a future binary doesn't need this package touched again. - internal/cni/crdnames: the annotation-key and BGP CRD naming helpers, shared between whatever writes BGP state and whatever reads it back (galactic-ipam's dealloc, until its own local persistence work lands). - internal/cni/nadpatch: NAD annotation logic, identical for both interface types, previously living unexported inside internal/cni. - internal/cniipam: the interface-agnostic IPAM allocation/deallocation logic (ipam_ops.go), now a shared library both master plugins import instead of galactic-tap-cni needing its own copy. Not yet a delegated CNI IPAM plugin of its own — that, plus dropping the CRD read dependency via local marker-file persistence for the IPv6 allocator, is sequencing step 1. - internal/cnibgp: the BGP/SRv6/eBPF publish logic (bgp.go), same shared- library treatment. Its own chain-invoked binary, and the CRD/eBPF-state CHECK logic it doesn't have yet, are sequencing step 2. internal/cni keeps its resourceTracker and CHECK/STATUS shape, just trimmed to veth-only; internal/cnitap gets its own smaller tracker with no guest- netns concerns at all. Taskfile.yaml and the galactic-cni Dockerfile build and stage galactic-tap-cni alongside the existing binaries in the same image, and installer.Bootstrap now copies it to the host too. Existing test coverage moved with the code it tests rather than being dropped: internal/cni/bgp_test.go and bgp_ebpf_test.go became internal/cnibgp's own tests (plus a small resource_ebpf_test.go left behind in internal/cni for resourceTracker's own eBPF-cleanup rollback path, which still lives there), ipam_ops_test.go became internal/cniipam's, nad_test.go became internal/cni/nadpatch's, and the tap-mode cases from cni_test.go became internal/cnitap's own suite. Verified: task lint, task build, and task test:unit are all green. task test:e2e needs a real Kind cluster bring-up (sudo modprobe vrf, kind create cluster) that wasn't attempted in this pass — the static per-node conflist Bootstrap writes is unchanged (still single-entry, node-settings only), so the existing veth-path e2e coverage should be unaffected, but that's unverified here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-cni This step's own split moved tap-mode ADD entirely into galactic-tap-cni's internal/cnitap and removed interface_type from internal/cni's own PluginConf — galactic-cni is veth-only now. This e2e test still invoked /galactic-cni directly with a config carrying interface_type: tap, a field that binary no longer reads at all, so its cmdAdd fell straight into the veth path and produced a 2-interface (host + guest) result instead of the 1-interface (host tap only) result the test's own assertions expect. Fixed to exec /galactic-tap-cni instead, with the config's "type" field updated to match (cosmetic here — this test invokes the binary directly by path, not through a real CNI runtime resolving "type" to a binary — but should still describe reality). Also dropped the srv6_locator field: PluginConf never had a field by that name at any point in this split: it was already dead, ignored JSON before this change and remains so after. Left everything else alone: the eBPF control daemon startup, bpf-fs mount, hostNetwork pod spec, "ipam" block, and GALACTIC_CNI_ENABLE_LOCAL_IPAM are all still exactly as correct for galactic-tap-cni at this point in the plugin-chain split as they were for the old monolithic galactic-cni's own tap branch — BGP/SRv6/eBPF publish is still called inline from cmdAdd here (that only moves out to its own chained binary, galactic-bgp, in a later step), so this test still needs the same BGPRouter fixture (scripts/ci.sh's e2etest case already creates one) and the same pre-pinned eBPF maps. Verification: task lint (0 issues), go build/vet clean, task test:unit full suite green. Not verified against a live cluster locally (no Kind/root here); pushing to let CI's own test-e2e job confirm.
988fbb2 to
4e865d9
Compare
…ation
This step rewired galactic-cni/galactic-tap-cni's own IPAM handling
from an in-process ipam.Allocate() call into real CNI IPAM delegation
(ipam.ExecAdd, execing whatever binary "ipam.type" names). The e2e
test's config still had the old "ipam": {"type": "pool"} block from
before this step -- "pool" is no longer a mode selector, it's now
looked up as a literal binary name, so ADD failed with "failed to find
plugin \"pool\" in path [/opt/cni/bin]" (confirmed on PR #303/#304's
own CI run, the first time e2e has actually executed against a live
cluster in this whole plugin-chain-split stack, now that a separate
pre-existing eBPF-artifact-drift issue blocking Build no longer blocks
it).
Fixed to "ipam": {"type": "galactic-ipam", "ipv6_subnet": "..."} --
the explicit contract this step establishes (see internal/cniipam's
doc comment). Also fixed CNI_PATH from /opt/cni/bin to / so
ipam.ExecAdd's lookup can actually find /galactic-ipam: every binary
in the chain lands at the image root
(containers/galactic-cni/Dockerfile), not /opt/cni/bin -- that path
only exists on a real host once installer.Bootstrap's init container
stages it there, which this test's pod never runs. Dropped
GALACTIC_CNI_ENABLE_LOCAL_IPAM=true from the script: it's now inert
for this config (an explicit ipv6_subnet doesn't need the local-IPAM
fallback), though harmless either way since this step doesn't remove
the env var itself.
BGP/SRv6/eBPF publish is still inline in galactic-tap-cni's own cmdAdd
at this point in the split (that only moves out to its own chained
binary in a later step), so the eBPF control daemon startup, bpf-fs
mount, and hostNetwork pod spec stay exactly as they were -- this
config still needs the same BGPRouter fixture and pre-pinned eBPF maps
the previous step's fix already established.
Verification: task lint (0 issues), go build/vet clean, task test:unit
full suite green. Not verified against a live cluster locally;
pushing to let CI's own test-e2e job confirm, same as the previous
step's fix.
"fd00:e2e:tap::/48" isn't a valid IPv6 literal -- "tap" contains 'p', not a hex digit. cniipam's own parseConf validates ipv6_subnet with net.ParseCIDR (see internal/cniipam/config.go's validateIPv6Subnet), so ADD failed with "invalid CIDR value for field 'ipam.ipv6_subnet'" as soon as this step's own IPAM-delegation fix (previous commit) made that field's value actually reach real validation for the first time -- it was never exercised before since PR #303/#304's e2e job never got this far until the eBPF-artifact-drift and wrong-binary-target issues in earlier commits on this branch were fixed. Fixed to "fd00:e2e::/48": drops the invalid "tap" hextet, keeps "e2e" (itself valid hex) as the mnemonic. Verification: task lint (0 issues), go build/vet clean, task test:unit full suite green, plus a plain net.ParseCIDR-equivalent check (Python's ipaddress.ip_network) confirming the literal parses.
…nadpatch Follow-up to mattdjenkinson's review on #303 (2 of 5 points; the other 3 are handled separately — 1 already resolved upstream in this stack, 1 is now moot, 1 is the dedup fix in a stacked follow-up branch). internal/cni/doc.go still described the package as creating "a veth or tap interface" and listed a tap subpackage as its own, both stale since step 0 of this stack made galactic-cni veth-only and moved tap to its own binary (galactic-tap-cni/internal/cnitap). Rewrote the package doc to describe what internal/cni actually does now (VRF + veth pair + NAD patch only — IPAM, termination routes, and BGP publish are galactic- ipam's/galactic-route's/galactic-bgp's own concerns, chained after this plugin) and to explain why internal/cni/ipam, internal/cni/route, and internal/cni/tap are still physically nested here despite no longer being used by this package itself. Promoted hostconf, crdnames, and nadpatch from internal/cni/* to top-level internal/*, matching the placement cnibgp/cniipam/cnitap/ cniroute already have. All four groups do the same cross-binary- sharing job; leaving hostconf/crdnames/nadpatch nested under internal/cni made it look like internal/cnitap (and cnibgp/cniroute) were reaching into cni's own internals, which is exactly what extracting them in the first place was meant to avoid. Package names are unchanged — every call site is a mechanical import-path update (15 files). internal/cni/ipam, internal/cni/route, and internal/cni/tap are a different category (kernel-primitive packages, like veth) and were not part of this review point, so they're untouched. task lint / task build / task test:unit all clean.
…nimaster Follow-up to mattdjenkinson's review on #303 (the last of 5 points; the other 4 are handled elsewhere — 2 already resolved upstream in this stack, 1 is now moot, 2 landed in fix/cni-review-followups-doc-placement). internal/cnitap reimplemented internal/cni's config.go (parseConf's validation, log setup, status parsing, prevResult validation, base62/ error sanitizing), resource.go (newK8sClient, the tracker/cleanup rollback sequence), and ops_check.go (checkNodeLevelState, cmdStatus, probeAPIServer, validateHostInterface) almost verbatim — none of it is interface-specific. A fix to one wouldn't have propagated to the other. Extracted the interface-agnostic pieces into a new internal/cnimaster package, the same kind of shared-library extraction this stack already did for hostconf/crdnames/nadpatch/cniipam/cnibgp: - PluginConf moves to cnimaster; internal/cni and internal/cnitap each keep `type PluginConf = cnimaster.PluginConf` (mirroring the existing HostConf = hostconf.HostConf alias pattern). - ParseConf, LoadHostConf, ParseStatusConf, ValidatePrevResult(Add), IsValidBase62, SanitizeForError, UnwrapPathError, ParseLogLevel, SetupLogging move to cnimaster.config.go. Each package's own parseConf becomes a two-line wrapper binding ParseConf to its own cniConfig/ ConfFile (these stay per-binary — each resolves its own GALACTIC_CNI_* environment independently). - NewK8sClient (+ the backing scheme) and a new CleanupAttachment (parameterized by interface kind and a Delete func, replacing the copy-pasted rollback sequence) move to cnimaster.resource.go. - CheckNodeLevelState, ValidateHostInterface, ProbeAPIServer(Fn), and a new RunStatus (the full cmdStatus body, since it wasn't interface- specific either) move to cnimaster.check.go. Each package's own cmdStatus becomes a one-line wrapper. - cmdCheck's guest-side validation (checkGuestInterface, validateGuestInterface, validateIPOnInterface) stays in internal/cni only — tap has no guest netns to check, so this piece is genuinely interface-specific and wasn't touched. Test coverage moved with the code: the generic-logic unit tests (TestParseConf, TestIsValidBase62, TestSanitizeForError, TestValidatePrevResult(Add), TestProbeAPIServer*, TestLoadHostConf, TestParseLogLevel, TestLoggingSetup*, TestIPAMBlockPresenceIsTheOnly Trigger) moved from internal/cni/cni_test.go into internal/cnimaster/cnimaster_test.go, since cnitap_test.go never had its own copies to begin with (it was implicitly relying on cni's tests covering logic it silently duplicated — exactly the risk the review comment flagged). internal/cnitap's own TestLoadHostConfMissingFile (testing a function that no longer exists in that package) is removed as now fully redundant with cnimaster's own coverage. Remaining tests in both cni_test.go/cnitap_test.go are either interface-specific (cmdAdd/cmdDel/cmdCheck/buildResult) or thin wiring smoke tests over the shared functions. Net effect: internal/cni and internal/cnitap together lose ~750 lines of near-duplicate implementation+test code; internal/cnimaster adds ~460 (including its own full test suite) — about 250 fewer lines overall, in one place instead of two. task lint / task build / go vet / task test:unit all clean (33/33 packages passing, 0 failures).
Stack (merge bottom to top):
Summary
galactic-cniis now veth-only. A newgalactic-tap-cnibinary owns tap workloads (Kata, Firecracker, kraftlet/Unikraft). Theinterface_typeconfig field and its runtime switch are gone from both binaries.First branch in a stack:
refactor/cni-chain-0-veth-tap-split→-1-galactic-ipam→-2-galactic-bgp→-3-galactic-route→-4-installer-docs. Each branch bases on the previous.New shared packages
Extracted so
internal/cnitapdoesn't duplicate or reach intointernal/cni's unexported internals:internal/cni/hostconf—HostConf/Load/node-name API fallback. Match on plugin type is now caller-supplied, not hardcoded to"galactic-cni".internal/cni/crdnames— BGP CRD naming and annotation-key helpers.internal/cni/nadpatch— NAD annotation logic, identical for both interface types.internal/cniipam— IPAM allocation/deallocation. Shared library for now; becomes its own delegated plugin in step 1.internal/cnibgp— BGP/SRv6/eBPF publish logic. Same treatment; becomes its own chain-invoked plugin in step 2.Verification
task lint✅task build✅ — builds and stagesgalactic-tap-cnitootask test:unit✅task test:e2e— not run. Needs a Kind cluster bring-up (sudo modprobe vrf,kind create cluster) not attempted here.installer.Bootstrap's static per-node conflist is unchanged, so existing veth-path e2e coverage should hold, but that's unverified — flagging for the reviewer.Test coverage moved with the code it tests, not dropped:
bgp_test.go/bgp_ebpf_test.go→internal/cnibgp,ipam_ops_test.go→internal/cniipam,nad_test.go→internal/cni/nadpatch, tap-mode cases fromcni_test.go→internal/cnitap.🤖 Generated with Claude Code