feat(containerlab): Add two-node gateway canary - #355
Conversation
a9b4c65 to
bb4683a
Compare
34d6e58 to
16e61ec
Compare
bb4683a to
3688b27
Compare
16e61ec to
fbcc1ef
Compare
3688b27 to
c07772d
Compare
fbcc1ef to
ac48549
Compare
Introduces galactic-gateway as its own binary, hosting the edge XDP NAT+LB gateway engine (previous two branches) and the NetworkGateway/NetworkRule reconcilers (previous branch) separately from galactic-router's tenant BGP process, so a crash on either side no longer takes the other down with it. Tenant BGP (GoBGP + the BGPRouter/BGPPeer/BGPAdvertisement/BGPPolicy/BGPVRFInstance reconcilers) still runs in galactic-router, co-located on the same gateway node. - cmd/galactic-gateway/main.go, root.go: manager setup, flags/env config, RBAC pre-flight -- no BGP runtime/RuntimeManager/BGP-family reconciler here at all. - cmd/galactic-gateway/gateway.go: setupGatewayDatapath, the load/attach/wire-up entry point, moved unchanged from cmd/galactic-router/gateway.go. - internal/config/gateway.go: GatewayConfig (metrics/grpc-health ports distinct from every other galactic-* process on the same hostNetwork node, node name, public interface, SRv6 address). - Taskfile.yaml: builds bin/galactic-gateway alongside the other binaries. Fourth branch in the edge-gateway stack; builds on feat/edge-gateway-03-controllers.
c07772d to
b1e1344
Compare
ac48549 to
ab54bca
Compare
- containers/galactic-gateway/Dockerfile: same shape as galactic-router's (golang builder -> distroless static:nonroot, no shell), regenerating both internal/plumbing/ebpf/prog's and internal/plumbing/ebpf/edgeprog's bpf2go output at build time (via edgepreflight's transitive import of the former), since neither is committed to git. - containers/galactic-router/Dockerfile: no functional change, just points its 'regenerate both bpf2go outputs' comment at galactic-gateway's Dockerfile (which now exists) instead of galactic-cni's (which only regenerates prog, not edgeprog). - containers/galactic-cni/Dockerfile: no functional change. - .github/workflows/publish.yaml: adds publish-galactic-gateway-image and adds it to publish-kustomize-bundles' needs. Does NOT yet stamp config/gateway/base with either image's tag -- that path doesn't exist until the next branch, so referencing it here would 404 the kustomize-bundle job on this branch's own CI. - AGENTS.md: describes the new publish pipeline in place of the now-stale 'no production release image build in this repo' note. Fifth branch in the edge-gateway stack; builds on feat/edge-gateway-04-cmd-wiring. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- config/gateway/serviceaccount.yaml, rbac.yaml: the gateway control
plane's own ServiceAccount/ClusterRole, split out of
config/router/rbac.yaml's single ClusterRole (which used to grant
one galactic-router identity both the BGP-family CRD verbs and
networkgateways/networkrules verbs because both reconciler sets
lived in the same binary -- they no longer do).
- config/gateway/kustomization.yaml: applies serviceaccount+rbac only
(safe/idempotent cluster-wide); deliberately does not include base/,
same exemption as config/fabric/ -- GALACTIC_GATEWAY_SRV6_ADDRESS has
no generic default and must be pinned per gateway node by a further
overlay (see deploy/containerlab/resources/galactic-router-gateway/
for a worked example).
- config/gateway/base/{daemonset.yaml,kustomization.yaml}: the
two-container (galactic-router + galactic-gateway) DaemonSet spec,
gateway-role node affinity, not applied directly.
- config/router/rbac.yaml: trims create/update/patch on
bgpadvertisements/bgpvrfinstances -- those verbs were only ever
needed by NetworkGatewayReconciler/NetworkRuleReconciler, which no
longer live in this binary.
- config/fabric/daemonset.yaml: adds the gateway role's node-affinity
value and NoSchedule toleration, so fabric-router (and therefore the
underlay eBGP session the gateway's XDP datapath depends on) also
runs on gateway-labeled nodes.
- .github/workflows/publish.yaml: now that config/gateway/base exists,
stamp it with both galactic-router's and galactic-gateway's published
tags (that DaemonSet has two containers) -- the previous branch added
the publish-galactic-gateway-image job itself but couldn't reference
this path yet, since it didn't exist until this commit.
- AGENTS.md: describes config/gateway/ in place of the old
config/router/gateway/ design.
Sixth branch in the edge-gateway stack; builds on
feat/edge-gateway-05-images-ci.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…LB gateway
Extends the containerlab lab with iad-gateway1/iad-gateway2, a
dedicated active-active pair of gateway-role nodes exercising
config/gateway/ end to end:
- gvpc.clab.yaml, node_files/iad/config.yaml: two new gateway nodes,
tainted galactic.datumapis.com/node=gateway:NoSchedule, uplinked to
tr3 (node_files/tr3/frr.conf).
- resources/fabric-router/iad/frr.conf.iad-gateway{1,2}: per-node FRR
underlay eBGP config for the new nodes; existing per-site frr.conf
files renamed to frr.conf.<nodename> (dfw-worker, iad-worker,
sjc-worker) to make room for the per-node key scheme fabric-router's
ConfigMap needs once a site has more than one matching node.
- resources/galactic-control/iad/bgppeer-gateway{1,2}.yaml: route
reflector BGPPeer objects for the two new gateway nodes' iBGP
sessions.
- resources/galactic-router-gateway/: per-node overlay instantiating
config/gateway/base/ for iad-gateway1/iad-gateway2 -- BGPPeer,
BGPRouter, NetworkGateway, and the node-pinning patch
(kubernetes.io/hostname, public interface, SRv6 address) each needs.
iad/networkrule-ns60.yaml is a canary NetworkRule pointing at ns60's
nginx pod.
- resources/tenants/ns60/: replaces ns50 (removed) as the gateway
canary's backend tenant -- a plain namespace + pod instead of ns50's
multi-site Deployment, since the gateway canary only needs one
reachable backend address to route traffic to.
- scripts/deploy-fabric.sh, deploy-galactic-router.sh,
deploy-system.sh: wire the new per-node resources into the deploy
pipeline; verify-ns50.sh removed with ns50.
- README.md, docs/tenants.md, docs/verification.md, Taskfile.yaml:
document the new topology and canary.
- deploy-system.sh, Taskfile.yaml: NetworkGateway/NetworkRule CRDs now
fetch from datum-cloud/network like every other BGP CRD, and the
galactic-router/-gateway/-cni image builds no longer need the
'--build-context network=...' pointing at a sibling checkout -- both
were stopgaps for those types not yet being upstream, which is no
longer the case as of this stack's third branch.
Seventh branch in the edge-gateway stack; builds on
feat/edge-gateway-06-config-manifests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
b1e1344 to
6797be1
Compare
ab54bca to
e3bb9c8
Compare
ecv
left a comment
There was a problem hiding this comment.
Holding off on approval here too, for one specific thing rather than the canary.
This PR adds COPY --from=network . /network to containers/galactic-router/Dockerfile, not only the gateway one. That is the image the publish pipeline ships today, and publish.yaml passes no build-contexts input, so the router image build breaks the moment this reaches main. The Dockerfile comment already anticipates this and says callers that do not pass the context must not hit it; publish.yaml is exactly such a caller. Same root cause as #358, and both COPY lines should come out when the replace does.
The lab side is the same coupling wearing different clothes: Taskfile.yaml passes --build-context network=../../../network for three images, and deploy-system.sh reads the new CRDs out of ../../../../network/config/crd. Local dev tolerates that better than CI does, but it means the lab now requires two checkouts in a specific layout with nothing checking for it up front. A clear failure early beats whatever Docker says when the context is missing.
On the canary itself, no objection. The topology, addressing and ASNs read consistently.
One thing worth calling out in the description rather than leaving in the diff: this removes the ns50 tenant entirely, including verify-ns50.sh, and replaces it with ns60. That trades a three-site IPv4 tenant for a single-site IPv6 one. You did update every reference, docs included, so nothing dangles. But anyone who used task deploy:ns50 for IPv4 coverage loses it, and that is the kind of change people should meet in the PR body rather than discover.
Also worth saying plainly, since the task description already half admits it: verify:gateway checks that CRDs are present, not that a packet crossed the datapath. A canary that passes without proving traffic flowed is the failure mode canaries exist to prevent. Fine as a first cut, but it should not stay that way once the lab can actually pass traffic.
ecv
left a comment
There was a problem hiding this comment.
Approving, with one thing that is about merge order rather than about this diff.
The COPY --from=network line this adds to containers/galactic-router/Dockerfile makes the currently-shipping router image unbuildable by the publish pipeline, which passes no build context. That is real, but CI shows it, and it disappears with the same fix as #358. So it is a sequencing constraint, not a reason to hold the branch: this should not land ahead of the module re-pin, and both COPY lines want to come out when the replace does.
Filed the canary's coverage gap as #368. Checking that the resources exist and the pods are up is a reasonable first cut, and it is also the exact shape of pass that the detached-program failure would have sailed through. Worth an assertion that a packet reached a backend and that the rule's counters moved before this is treated as datapath coverage.
The ns50 to ns60 swap I would still put in the PR body. You updated every reference, so nothing dangles, but losing the three-site IPv4 tenant is the kind of thing people should read rather than discover.
Topology, addressing and ASNs all read consistently.
6797be1 to
6f42dd0
Compare
Summary
The manifests from the previous PR need a lab environment to exercise before anyone runs them in a real cluster. This adds a dedicated active-active pair of gateway-role nodes to the containerlab topology, with the FRR underlay, route-reflector peering, and per-node overlay each needs, plus a canary NetworkRule pointed at a live backend pod. Seventh branch in the edge-gateway stack; builds on the config manifests PR.
Test plan
sudo containerlablab deploy brings up the two gateway nodes and their BGP sessionsNote
End-to-end ingress traffic through the datapath doesn't reach a backend in this topology yet — a veth-specific XDP_TX behavior, not a code bug.
Related to #17