Skip to content

refactor(cni): split galactic-cni into veth/tap master plugins - #303

Open
privateip wants to merge 2 commits into
mainfrom
refactor/cni-chain-0-veth-tap-split
Open

refactor(cni): split galactic-cni into veth/tap master plugins#303
privateip wants to merge 2 commits into
mainfrom
refactor/cni-chain-0-veth-tap-split

Conversation

@privateip

@privateip privateip commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Stack (merge bottom to top):


Summary

galactic-cni is now veth-only. A new galactic-tap-cni binary owns tap workloads (Kata, Firecracker, kraftlet/Unikraft). The interface_type config field and its runtime switch are gone from both binaries.

First branch in a stack: refactor/cni-chain-0-veth-tap-split-1-galactic-ipam-2-galactic-bgp-3-galactic-route-4-installer-docs. Each branch bases on the previous.

New shared packages

Extracted so internal/cnitap doesn't duplicate or reach into internal/cni's unexported internals:

  • internal/cni/hostconfHostConf/Load/node-name API fallback. Match on plugin type is now caller-supplied, not hardcoded to "galactic-cni".
  • internal/cni/crdnames — BGP CRD naming and annotation-key helpers.
  • internal/cni/nadpatch — NAD annotation logic, identical for both interface types.
  • internal/cniipam — IPAM allocation/deallocation. Shared library for now; becomes its own delegated plugin in step 1.
  • internal/cnibgp — BGP/SRv6/eBPF publish logic. Same treatment; becomes its own chain-invoked plugin in step 2.

Verification

  • task lint
  • task build ✅ — builds and stages galactic-tap-cni too
  • task test:unit
  • task test:e2e — not run. Needs a Kind cluster bring-up (sudo modprobe vrf, kind create cluster) not attempted here. installer.Bootstrap's static per-node conflist is unchanged, so existing veth-path e2e coverage should hold, but that's unverified — flagging for the reviewer.

Test coverage moved with the code it tests, not dropped: bgp_test.go/bgp_ebpf_test.gointernal/cnibgp, ipam_ops_test.gointernal/cniipam, nad_test.gointernal/cni/nadpatch, tap-mode cases from cni_test.gointernal/cnitap.

🤖 Generated with Claude Code

@privateip
privateip requested a review from a team as a code owner August 8, 2026 03:10
@privateip
privateip requested a review from jacobsmith928 August 8, 2026 03:10
@privateip
privateip marked this pull request as draft August 8, 2026 03:12
@privateip
privateip marked this pull request as ready for review August 8, 2026 12:20
privateip added a commit that referenced this pull request Aug 8, 2026
…ation

This step rewired galactic-cni/galactic-tap-cni's own IPAM handling
from an in-process ipam.Allocate() call into real CNI IPAM delegation
(ipam.ExecAdd, execing whatever binary "ipam.type" names). The e2e
test's config still had the old "ipam": {"type": "pool"} block from
before this step -- "pool" is no longer a mode selector, it's now
looked up as a literal binary name, so ADD failed with "failed to find
plugin \"pool\" in path [/opt/cni/bin]" (confirmed on PR #303/#304's
own CI run, the first time e2e has actually executed against a live
cluster in this whole plugin-chain-split stack, now that a separate
pre-existing eBPF-artifact-drift issue blocking Build no longer blocks
it).

Fixed to "ipam": {"type": "galactic-ipam", "ipv6_subnet": "..."} --
the explicit contract this step establishes (see internal/cniipam's
doc comment). Also fixed CNI_PATH from /opt/cni/bin to / so
ipam.ExecAdd's lookup can actually find /galactic-ipam: every binary
in the chain lands at the image root
(containers/galactic-cni/Dockerfile), not /opt/cni/bin -- that path
only exists on a real host once installer.Bootstrap's init container
stages it there, which this test's pod never runs. Dropped
GALACTIC_CNI_ENABLE_LOCAL_IPAM=true from the script: it's now inert
for this config (an explicit ipv6_subnet doesn't need the local-IPAM
fallback), though harmless either way since this step doesn't remove
the env var itself.

BGP/SRv6/eBPF publish is still inline in galactic-tap-cni's own cmdAdd
at this point in the split (that only moves out to its own chained
binary in a later step), so the eBPF control daemon startup, bpf-fs
mount, and hostNetwork pod spec stay exactly as they were -- this
config still needs the same BGPRouter fixture and pre-pinned eBPF maps
the previous step's fix already established.

Verification: task lint (0 issues), go build/vet clean, task test:unit
full suite green. Not verified against a live cluster locally;
pushing to let CI's own test-e2e job confirm, same as the previous
step's fix.
privateip added a commit that referenced this pull request Aug 8, 2026
"fd00:e2e:tap::/48" isn't a valid IPv6 literal -- "tap" contains 'p',
not a hex digit. cniipam's own parseConf validates ipv6_subnet with
net.ParseCIDR (see internal/cniipam/config.go's validateIPv6Subnet),
so ADD failed with "invalid CIDR value for field 'ipam.ipv6_subnet'"
as soon as this step's own IPAM-delegation fix (previous commit) made
that field's value actually reach real validation for the first time
-- it was never exercised before since PR #303/#304's e2e job never
got this far until the eBPF-artifact-drift and wrong-binary-target
issues in earlier commits on this branch were fixed.

Fixed to "fd00:e2e::/48": drops the invalid "tap" hextet, keeps "e2e"
(itself valid hex) as the mnemonic.

Verification: task lint (0 issues), go build/vet clean, task test:unit
full suite green, plus a plain net.ParseCIDR-equivalent check
(Python's ipaddress.ip_network) confirming the literal parses.
@mattdjenkinson

Copy link
Copy Markdown

Went through this one closely. The split looks solid overall: the shared-package extraction (hostconf, crdnames, nadpatch, cniipam, cnibgp) does what the description says, and tests moved with the code they cover. A few things worth looking at before this lands, roughly in order of how much I'd worry about them.

First, cmd/galactic-cni/main.go's isTapMode() / CNI_NETNS_OVERRIDE path is dead now, but not in a harmless way. internal/cni doesn't handle tap at all anymore (PluginConf.InterfaceType and the veth/tap switch are gone), yet this main.go still checks for interface_type: "tap" and flips off the netns-mismatch safety check when it sees it. If the external operator that templates per-attachment conflists hasn't been updated yet to route tap workloads to galactic-tap-cni, a config still tagged interface_type: tap will land on galactic-cni, skip the safety check, and get processed as veth anyway, misconfiguring the workload instead of erroring out. I noticed #307 deletes this exact block later as dead code, so you're clearly aware of it, just sequenced for a later branch. Might be worth pulling that deletion forward into this PR, or at least flagging the rollout-ordering risk in the description since this branch ships first.

Second, three files got copy-pasted instead of extracted, which cuts against the stated point of this refactor. The whole reason for pulling out hostconf, crdnames, nadpatch, cniipam, and cnibgp was so cnitap wouldn't duplicate or reach into cni's internals. But internal/cnitap/config.go reimplements parseConf and everything around it (log setup, status parsing, prevResult validation, base62/error sanitizing) as a near copy of internal/cni/config.go. internal/cnitap/resource.go duplicates newK8sClient() and the tracker/cleanup rollback sequence from internal/cni/resource.go, swapping in tap.Delete for veth.Delete and otherwise identical. And internal/cnitap/ops_check.go copies the CHECK/STATUS logic (checkNodeLevelState, checkTerminationRoutes, validateHostInterface, probeAPIServer) verbatim, none of which is interface-type specific.

A future validation fix or rollback-ordering fix on the veth side won't propagate to any of those tap copies. Given five other packages already went through this same extraction, it seems worth doing the same here rather than letting two copies drift apart.

A couple of smaller things:

  • internal/cni/ops_add.go's veth path calls cnibgp.PublishBGPState without the nil-tracker.k8s guard that the tap path still has before its equivalent call. Not reachable today since newK8sClient() can't return a nil client here, but since cnibgp is meant to be a shared library future chain binaries call into directly, might be worth keeping the guard for defense in depth.
  • internal/cni/doc.go still says the package creates "a veth or tap interface" and lists a tap subpackage, both stale now that cni is veth-only.
  • hostconf/crdnames/nadpatch ended up nested under internal/cni/, while cnibgp/cniipam (doing the same cross-binary-sharing job) got promoted to top-level internal/. Just a placement inconsistency, but it makes hostconf etc. look like cni-owned internals that cnitap reaches into, which is the exact thing this PR is trying to get away from.

Given this is a stack, happy to have the duplication cleanup land in a later branch if that's cleaner, but the isTapMode one seems worth addressing here since this branch ships first.

@privateip

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review, @mattdjenkinson — went through all five points against where this stack has landed since. Two are already handled without further changes needed here:

isTapMode()/CNI_NETNS_OVERRIDE dead code — already deleted, in 725ef87 (this stack's own step 4/#307). Confirmed cmd/galactic-cni/main.go no longer has the stdin-peek dance or the isTapMode() helper; the comment now in its place explains why it's gone (interface kind is which binary you invoke now, not a config field either binary branches on).

Missing nil-tracker.k8s guard before cnibgp.PublishBGPState — now moot rather than fixed. e33092c (step 2/#305, extracting galactic-bgp) moved BGP publish entirely out of internal/cni/ops_add.go into its own chain-invoked binary. There's now exactly one call site (internal/cnibgp/ops_add.go), which constructs its own k8sClient immediately before use with an error return on failure — the guard-ordering concern doesn't apply to that shape anymore.

The other three are real and I've landed fixes for them as follow-ups on top of the stack rather than pushing more commits onto this branch, so your review here stays scoped to what you already looked at:

  • docs+refactor(cni): fix stale package doc, promote hostconf/crdnames/nadpatch #315 — fixes the stale internal/cni/doc.go package comment (still described the package as veth-or-tap, still listed tap as its own subpackage) and promotes hostconf/crdnames/nadpatch from internal/cni/* to top-level internal/* to match cnibgp/cniipam/cnitap/cniroute's placement — closing the exact "looks like cnitap reaches into cni's internals" gap you flagged.
  • refactor(cni): dedupe internal/cni vs internal/cnitap into internal/cnimaster #316 (stacked on docs+refactor(cni): fix stale package doc, promote hostconf/crdnames/nadpatch #315) — the internal/cnitap vs internal/cni duplication in config.go/resource.go/ops_check.go. Extracted the interface-agnostic pieces (parseConf's validation, log setup, k8s client construction, CHECK/STATUS node-level checks, prevResult validation) into a new internal/cnimaster package that both master plugins now share, leaving only the genuinely interface-specific bits (which kernel primitive to call, tap's lack of a guest netns to check) in each package. Test coverage moved with the code, including catching that cnitap's own tests had been implicitly relying on cni's tests to cover logic it silently duplicated — exactly the risk you called out.

Both are green on task lint/task build/task test:unit.

mattdjenkinson
mattdjenkinson previously approved these changes Aug 8, 2026
privateip and others added 2 commits August 8, 2026 15:01
Sequencing step 0 of the CNI plugin-chain split (design plan in
~/.vaults/notebook/projects/galactic/plan-cni-plugin-chain.md). galactic-cni
is now veth-only; a new galactic-tap-cni binary owns tap-mode workloads
(Kata, Firecracker, kraftlet/Unikraft). The interface_type config field and
its runtime switch are gone entirely from both binaries — which binary runs
is now the conflist-authoring decision, matching the design note's own
rationale for the split.

Getting there required extracting a few things this repo already had
duplicated or coupled in ways that would have blocked genuine package
separation:

- internal/cni/hostconf: HostConf/loadHostConf/conflistEnvelope were
  duplicated near-verbatim between internal/cni and internal/installer, and
  hardcoded a match on plugin type "galactic-cni". Generalized into one
  shared package both binaries (and the installer) now call, with the
  accepted-type list passed in by the caller so a future binary doesn't
  need this package touched again.
- internal/cni/crdnames: the annotation-key and BGP CRD naming helpers,
  shared between whatever writes BGP state and whatever reads it back
  (galactic-ipam's dealloc, until its own local persistence work lands).
- internal/cni/nadpatch: NAD annotation logic, identical for both interface
  types, previously living unexported inside internal/cni.
- internal/cniipam: the interface-agnostic IPAM allocation/deallocation
  logic (ipam_ops.go), now a shared library both master plugins import
  instead of galactic-tap-cni needing its own copy. Not yet a delegated CNI
  IPAM plugin of its own — that, plus dropping the CRD read dependency via
  local marker-file persistence for the IPv6 allocator, is sequencing
  step 1.
- internal/cnibgp: the BGP/SRv6/eBPF publish logic (bgp.go), same shared-
  library treatment. Its own chain-invoked binary, and the CRD/eBPF-state
  CHECK logic it doesn't have yet, are sequencing step 2.

internal/cni keeps its resourceTracker and CHECK/STATUS shape, just trimmed
to veth-only; internal/cnitap gets its own smaller tracker with no guest-
netns concerns at all. Taskfile.yaml and the galactic-cni Dockerfile build
and stage galactic-tap-cni alongside the existing binaries in the same
image, and installer.Bootstrap now copies it to the host too.

Existing test coverage moved with the code it tests rather than being
dropped: internal/cni/bgp_test.go and bgp_ebpf_test.go became
internal/cnibgp's own tests (plus a small resource_ebpf_test.go left behind
in internal/cni for resourceTracker's own eBPF-cleanup rollback path, which
still lives there), ipam_ops_test.go became internal/cniipam's,
nad_test.go became internal/cni/nadpatch's, and the tap-mode cases from
cni_test.go became internal/cnitap's own suite.

Verified: task lint, task build, and task test:unit are all green. task
test:e2e needs a real Kind cluster bring-up (sudo modprobe vrf, kind create
cluster) that wasn't attempted in this pass — the static per-node conflist
Bootstrap writes is unchanged (still single-entry, node-settings only), so
the existing veth-path e2e coverage should be unaffected, but that's
unverified here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-cni

This step's own split moved tap-mode ADD entirely into
galactic-tap-cni's internal/cnitap and removed interface_type from
internal/cni's own PluginConf — galactic-cni is veth-only now. This
e2e test still invoked /galactic-cni directly with a config carrying
interface_type: tap, a field that binary no longer reads at all, so
its cmdAdd fell straight into the veth path and produced a 2-interface
(host + guest) result instead of the 1-interface (host tap only)
result the test's own assertions expect.

Fixed to exec /galactic-tap-cni instead, with the config's "type"
field updated to match (cosmetic here — this test invokes the binary
directly by path, not through a real CNI runtime resolving "type" to
a binary — but should still describe reality). Also dropped the
srv6_locator field: PluginConf never had a field by that name at any
point in this split: it was already dead, ignored JSON before this
change and remains so after.

Left everything else alone: the eBPF control daemon startup, bpf-fs
mount, hostNetwork pod spec, "ipam" block, and
GALACTIC_CNI_ENABLE_LOCAL_IPAM are all still exactly as correct for
galactic-tap-cni at this point in the plugin-chain split as they were
for the old monolithic galactic-cni's own tap branch — BGP/SRv6/eBPF
publish is still called inline from cmdAdd here (that only moves out
to its own chained binary, galactic-bgp, in a later step), so this
test still needs the same BGPRouter fixture
(scripts/ci.sh's e2etest case already creates one) and the same
pre-pinned eBPF maps.

Verification: task lint (0 issues), go build/vet clean, task test:unit
full suite green. Not verified against a live cluster locally (no
Kind/root here); pushing to let CI's own test-e2e job confirm.
@privateip
privateip force-pushed the refactor/cni-chain-0-veth-tap-split branch from 988fbb2 to 4e865d9 Compare August 8, 2026 19:02
privateip added a commit that referenced this pull request Aug 8, 2026
…ation

This step rewired galactic-cni/galactic-tap-cni's own IPAM handling
from an in-process ipam.Allocate() call into real CNI IPAM delegation
(ipam.ExecAdd, execing whatever binary "ipam.type" names). The e2e
test's config still had the old "ipam": {"type": "pool"} block from
before this step -- "pool" is no longer a mode selector, it's now
looked up as a literal binary name, so ADD failed with "failed to find
plugin \"pool\" in path [/opt/cni/bin]" (confirmed on PR #303/#304's
own CI run, the first time e2e has actually executed against a live
cluster in this whole plugin-chain-split stack, now that a separate
pre-existing eBPF-artifact-drift issue blocking Build no longer blocks
it).

Fixed to "ipam": {"type": "galactic-ipam", "ipv6_subnet": "..."} --
the explicit contract this step establishes (see internal/cniipam's
doc comment). Also fixed CNI_PATH from /opt/cni/bin to / so
ipam.ExecAdd's lookup can actually find /galactic-ipam: every binary
in the chain lands at the image root
(containers/galactic-cni/Dockerfile), not /opt/cni/bin -- that path
only exists on a real host once installer.Bootstrap's init container
stages it there, which this test's pod never runs. Dropped
GALACTIC_CNI_ENABLE_LOCAL_IPAM=true from the script: it's now inert
for this config (an explicit ipv6_subnet doesn't need the local-IPAM
fallback), though harmless either way since this step doesn't remove
the env var itself.

BGP/SRv6/eBPF publish is still inline in galactic-tap-cni's own cmdAdd
at this point in the split (that only moves out to its own chained
binary in a later step), so the eBPF control daemon startup, bpf-fs
mount, and hostNetwork pod spec stay exactly as they were -- this
config still needs the same BGPRouter fixture and pre-pinned eBPF maps
the previous step's fix already established.

Verification: task lint (0 issues), go build/vet clean, task test:unit
full suite green. Not verified against a live cluster locally;
pushing to let CI's own test-e2e job confirm, same as the previous
step's fix.
privateip added a commit that referenced this pull request Aug 8, 2026
"fd00:e2e:tap::/48" isn't a valid IPv6 literal -- "tap" contains 'p',
not a hex digit. cniipam's own parseConf validates ipv6_subnet with
net.ParseCIDR (see internal/cniipam/config.go's validateIPv6Subnet),
so ADD failed with "invalid CIDR value for field 'ipam.ipv6_subnet'"
as soon as this step's own IPAM-delegation fix (previous commit) made
that field's value actually reach real validation for the first time
-- it was never exercised before since PR #303/#304's e2e job never
got this far until the eBPF-artifact-drift and wrong-binary-target
issues in earlier commits on this branch were fixed.

Fixed to "fd00:e2e::/48": drops the invalid "tap" hextet, keeps "e2e"
(itself valid hex) as the mnemonic.

Verification: task lint (0 issues), go build/vet clean, task test:unit
full suite green, plus a plain net.ParseCIDR-equivalent check
(Python's ipaddress.ip_network) confirming the literal parses.
privateip added a commit that referenced this pull request Aug 8, 2026
…nadpatch

Follow-up to mattdjenkinson's review on #303 (2 of 5 points; the other
3 are handled separately — 1 already resolved upstream in this stack,
1 is now moot, 1 is the dedup fix in a stacked follow-up branch).

internal/cni/doc.go still described the package as creating "a veth or
tap interface" and listed a tap subpackage as its own, both stale since
step 0 of this stack made galactic-cni veth-only and moved tap to its
own binary (galactic-tap-cni/internal/cnitap). Rewrote the package doc
to describe what internal/cni actually does now (VRF + veth pair + NAD
patch only — IPAM, termination routes, and BGP publish are galactic-
ipam's/galactic-route's/galactic-bgp's own concerns, chained after this
plugin) and to explain why internal/cni/ipam, internal/cni/route, and
internal/cni/tap are still physically nested here despite no longer
being used by this package itself.

Promoted hostconf, crdnames, and nadpatch from internal/cni/* to
top-level internal/*, matching the placement cnibgp/cniipam/cnitap/
cniroute already have. All four groups do the same cross-binary-
sharing job; leaving hostconf/crdnames/nadpatch nested under
internal/cni made it look like internal/cnitap (and cnibgp/cniroute)
were reaching into cni's own internals, which is exactly what
extracting them in the first place was meant to avoid. Package names
are unchanged — every call site is a mechanical import-path update
(15 files).

internal/cni/ipam, internal/cni/route, and internal/cni/tap are a
different category (kernel-primitive packages, like veth) and were not
part of this review point, so they're untouched.

task lint / task build / task test:unit all clean.
privateip added a commit that referenced this pull request Aug 8, 2026
…nimaster

Follow-up to mattdjenkinson's review on #303 (the last of 5 points; the
other 4 are handled elsewhere — 2 already resolved upstream in this
stack, 1 is now moot, 2 landed in fix/cni-review-followups-doc-placement).

internal/cnitap reimplemented internal/cni's config.go (parseConf's
validation, log setup, status parsing, prevResult validation, base62/
error sanitizing), resource.go (newK8sClient, the tracker/cleanup
rollback sequence), and ops_check.go (checkNodeLevelState, cmdStatus,
probeAPIServer, validateHostInterface) almost verbatim — none of it is
interface-specific. A fix to one wouldn't have propagated to the other.

Extracted the interface-agnostic pieces into a new internal/cnimaster
package, the same kind of shared-library extraction this stack already
did for hostconf/crdnames/nadpatch/cniipam/cnibgp:

- PluginConf moves to cnimaster; internal/cni and internal/cnitap each
  keep `type PluginConf = cnimaster.PluginConf` (mirroring the existing
  HostConf = hostconf.HostConf alias pattern).
- ParseConf, LoadHostConf, ParseStatusConf, ValidatePrevResult(Add),
  IsValidBase62, SanitizeForError, UnwrapPathError, ParseLogLevel,
  SetupLogging move to cnimaster.config.go. Each package's own parseConf
  becomes a two-line wrapper binding ParseConf to its own cniConfig/
  ConfFile (these stay per-binary — each resolves its own GALACTIC_CNI_*
  environment independently).
- NewK8sClient (+ the backing scheme) and a new CleanupAttachment
  (parameterized by interface kind and a Delete func, replacing the
  copy-pasted rollback sequence) move to cnimaster.resource.go.
- CheckNodeLevelState, ValidateHostInterface, ProbeAPIServer(Fn), and a
  new RunStatus (the full cmdStatus body, since it wasn't interface-
  specific either) move to cnimaster.check.go. Each package's own
  cmdStatus becomes a one-line wrapper.
- cmdCheck's guest-side validation (checkGuestInterface,
  validateGuestInterface, validateIPOnInterface) stays in internal/cni
  only — tap has no guest netns to check, so this piece is genuinely
  interface-specific and wasn't touched.

Test coverage moved with the code: the generic-logic unit tests
(TestParseConf, TestIsValidBase62, TestSanitizeForError,
TestValidatePrevResult(Add), TestProbeAPIServer*, TestLoadHostConf,
TestParseLogLevel, TestLoggingSetup*, TestIPAMBlockPresenceIsTheOnly
Trigger) moved from internal/cni/cni_test.go into
internal/cnimaster/cnimaster_test.go, since cnitap_test.go never had
its own copies to begin with (it was implicitly relying on cni's tests
covering logic it silently duplicated — exactly the risk the review
comment flagged). internal/cnitap's own TestLoadHostConfMissingFile
(testing a function that no longer exists in that package) is removed
as now fully redundant with cnimaster's own coverage. Remaining tests
in both cni_test.go/cnitap_test.go are either interface-specific
(cmdAdd/cmdDel/cmdCheck/buildResult) or thin wiring smoke tests over
the shared functions.

Net effect: internal/cni and internal/cnitap together lose ~750 lines
of near-duplicate implementation+test code; internal/cnimaster adds
~460 (including its own full test suite) — about 250 fewer lines
overall, in one place instead of two.

task lint / task build / go vet / task test:unit all clean (33/33
packages passing, 0 failures).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants