Skip to content

feat(containerlab): Add two-node gateway canary - #355

Open
privateip wants to merge 4 commits into
feat/edge-gateway-06-config-manifestsfrom
feat/edge-gateway-07-containerlab
Open

feat(containerlab): Add two-node gateway canary#355
privateip wants to merge 4 commits into
feat/edge-gateway-06-config-manifestsfrom
feat/edge-gateway-07-containerlab

Conversation

@privateip

Copy link
Copy Markdown
Contributor

Summary

The manifests from the previous PR need a lab environment to exercise before anyone runs them in a real cluster. This adds a dedicated active-active pair of gateway-role nodes to the containerlab topology, with the FRR underlay, route-reflector peering, and per-node overlay each needs, plus a canary NetworkRule pointed at a live backend pod. Seventh branch in the edge-gateway stack; builds on the config manifests PR.

Test plan

  • sudo containerlab lab deploy brings up the two gateway nodes and their BGP sessions
  • The canary NetworkGateway/NetworkRule CRDs reconcile without error

Note

End-to-end ingress traffic through the datapath doesn't reach a backend in this topology yet — a veth-specific XDP_TX behavior, not a code bug.

Related to #17

@privateip
privateip requested a review from a team as a code owner August 12, 2026 18:10
@privateip
privateip requested review from gaghan430 and removed request for a team August 12, 2026 18:10
@privateip
privateip force-pushed the feat/edge-gateway-06-config-manifests branch from a9b4c65 to bb4683a Compare August 12, 2026 18:26
@privateip
privateip force-pushed the feat/edge-gateway-07-containerlab branch from 34d6e58 to 16e61ec Compare August 12, 2026 18:26
@privateip
privateip force-pushed the feat/edge-gateway-06-config-manifests branch from bb4683a to 3688b27 Compare August 12, 2026 19:04
@privateip
privateip force-pushed the feat/edge-gateway-07-containerlab branch from 16e61ec to fbcc1ef Compare August 12, 2026 19:04
@privateip
privateip force-pushed the feat/edge-gateway-06-config-manifests branch from 3688b27 to c07772d Compare August 12, 2026 19:18
@privateip
privateip force-pushed the feat/edge-gateway-07-containerlab branch from fbcc1ef to ac48549 Compare August 12, 2026 19:18
Introduces galactic-gateway as its own binary, hosting the edge XDP
NAT+LB gateway engine (previous two branches) and the
NetworkGateway/NetworkRule reconcilers (previous branch) separately
from galactic-router's tenant BGP process, so a crash on either side
no longer takes the other down with it. Tenant BGP (GoBGP + the
BGPRouter/BGPPeer/BGPAdvertisement/BGPPolicy/BGPVRFInstance
reconcilers) still runs in galactic-router, co-located on the same
gateway node.

- cmd/galactic-gateway/main.go, root.go: manager setup, flags/env
  config, RBAC pre-flight -- no BGP runtime/RuntimeManager/BGP-family
  reconciler here at all.
- cmd/galactic-gateway/gateway.go: setupGatewayDatapath, the
  load/attach/wire-up entry point, moved unchanged from
  cmd/galactic-router/gateway.go.
- internal/config/gateway.go: GatewayConfig (metrics/grpc-health ports
  distinct from every other galactic-* process on the same
  hostNetwork node, node name, public interface, SRv6 address).
- Taskfile.yaml: builds bin/galactic-gateway alongside the other
  binaries.

Fourth branch in the edge-gateway stack; builds on
feat/edge-gateway-03-controllers.
@privateip
privateip force-pushed the feat/edge-gateway-06-config-manifests branch from c07772d to b1e1344 Compare August 12, 2026 19:26
@privateip
privateip force-pushed the feat/edge-gateway-07-containerlab branch from ac48549 to ab54bca Compare August 12, 2026 19:26
privateip and others added 3 commits August 12, 2026 15:43
- containers/galactic-gateway/Dockerfile: same shape as
  galactic-router's (golang builder -> distroless static:nonroot, no
  shell), regenerating both internal/plumbing/ebpf/prog's and
  internal/plumbing/ebpf/edgeprog's bpf2go output at build time (via
  edgepreflight's transitive import of the former), since neither is
  committed to git.
- containers/galactic-router/Dockerfile: no functional change, just
  points its 'regenerate both bpf2go outputs' comment at
  galactic-gateway's Dockerfile (which now exists) instead of
  galactic-cni's (which only regenerates prog, not edgeprog).
- containers/galactic-cni/Dockerfile: no functional change.
- .github/workflows/publish.yaml: adds publish-galactic-gateway-image
  and adds it to publish-kustomize-bundles' needs. Does NOT yet stamp
  config/gateway/base with either image's tag -- that path doesn't
  exist until the next branch, so referencing it here would 404 the
  kustomize-bundle job on this branch's own CI.
- AGENTS.md: describes the new publish pipeline in place of the
  now-stale 'no production release image build in this repo' note.

Fifth branch in the edge-gateway stack; builds on
feat/edge-gateway-04-cmd-wiring.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- config/gateway/serviceaccount.yaml, rbac.yaml: the gateway control
  plane's own ServiceAccount/ClusterRole, split out of
  config/router/rbac.yaml's single ClusterRole (which used to grant
  one galactic-router identity both the BGP-family CRD verbs and
  networkgateways/networkrules verbs because both reconciler sets
  lived in the same binary -- they no longer do).
- config/gateway/kustomization.yaml: applies serviceaccount+rbac only
  (safe/idempotent cluster-wide); deliberately does not include base/,
  same exemption as config/fabric/ -- GALACTIC_GATEWAY_SRV6_ADDRESS has
  no generic default and must be pinned per gateway node by a further
  overlay (see deploy/containerlab/resources/galactic-router-gateway/
  for a worked example).
- config/gateway/base/{daemonset.yaml,kustomization.yaml}: the
  two-container (galactic-router + galactic-gateway) DaemonSet spec,
  gateway-role node affinity, not applied directly.
- config/router/rbac.yaml: trims create/update/patch on
  bgpadvertisements/bgpvrfinstances -- those verbs were only ever
  needed by NetworkGatewayReconciler/NetworkRuleReconciler, which no
  longer live in this binary.
- config/fabric/daemonset.yaml: adds the gateway role's node-affinity
  value and NoSchedule toleration, so fabric-router (and therefore the
  underlay eBGP session the gateway's XDP datapath depends on) also
  runs on gateway-labeled nodes.
- .github/workflows/publish.yaml: now that config/gateway/base exists,
  stamp it with both galactic-router's and galactic-gateway's published
  tags (that DaemonSet has two containers) -- the previous branch added
  the publish-galactic-gateway-image job itself but couldn't reference
  this path yet, since it didn't exist until this commit.
- AGENTS.md: describes config/gateway/ in place of the old
  config/router/gateway/ design.

Sixth branch in the edge-gateway stack; builds on
feat/edge-gateway-05-images-ci.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…LB gateway

Extends the containerlab lab with iad-gateway1/iad-gateway2, a
dedicated active-active pair of gateway-role nodes exercising
config/gateway/ end to end:

- gvpc.clab.yaml, node_files/iad/config.yaml: two new gateway nodes,
  tainted galactic.datumapis.com/node=gateway:NoSchedule, uplinked to
  tr3 (node_files/tr3/frr.conf).
- resources/fabric-router/iad/frr.conf.iad-gateway{1,2}: per-node FRR
  underlay eBGP config for the new nodes; existing per-site frr.conf
  files renamed to frr.conf.<nodename> (dfw-worker, iad-worker,
  sjc-worker) to make room for the per-node key scheme fabric-router's
  ConfigMap needs once a site has more than one matching node.
- resources/galactic-control/iad/bgppeer-gateway{1,2}.yaml: route
  reflector BGPPeer objects for the two new gateway nodes' iBGP
  sessions.
- resources/galactic-router-gateway/: per-node overlay instantiating
  config/gateway/base/ for iad-gateway1/iad-gateway2 -- BGPPeer,
  BGPRouter, NetworkGateway, and the node-pinning patch
  (kubernetes.io/hostname, public interface, SRv6 address) each needs.
  iad/networkrule-ns60.yaml is a canary NetworkRule pointing at ns60's
  nginx pod.
- resources/tenants/ns60/: replaces ns50 (removed) as the gateway
  canary's backend tenant -- a plain namespace + pod instead of ns50's
  multi-site Deployment, since the gateway canary only needs one
  reachable backend address to route traffic to.
- scripts/deploy-fabric.sh, deploy-galactic-router.sh,
  deploy-system.sh: wire the new per-node resources into the deploy
  pipeline; verify-ns50.sh removed with ns50.
- README.md, docs/tenants.md, docs/verification.md, Taskfile.yaml:
  document the new topology and canary.
- deploy-system.sh, Taskfile.yaml: NetworkGateway/NetworkRule CRDs now
  fetch from datum-cloud/network like every other BGP CRD, and the
  galactic-router/-gateway/-cni image builds no longer need the
  '--build-context network=...' pointing at a sibling checkout -- both
  were stopgaps for those types not yet being upstream, which is no
  longer the case as of this stack's third branch.

Seventh branch in the edge-gateway stack; builds on
feat/edge-gateway-06-config-manifests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@privateip
privateip force-pushed the feat/edge-gateway-06-config-manifests branch from b1e1344 to 6797be1 Compare August 12, 2026 19:44
@privateip
privateip force-pushed the feat/edge-gateway-07-containerlab branch from ab54bca to e3bb9c8 Compare August 12, 2026 19:44

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding off on approval here too, for one specific thing rather than the canary.

This PR adds COPY --from=network . /network to containers/galactic-router/Dockerfile, not only the gateway one. That is the image the publish pipeline ships today, and publish.yaml passes no build-contexts input, so the router image build breaks the moment this reaches main. The Dockerfile comment already anticipates this and says callers that do not pass the context must not hit it; publish.yaml is exactly such a caller. Same root cause as #358, and both COPY lines should come out when the replace does.

The lab side is the same coupling wearing different clothes: Taskfile.yaml passes --build-context network=../../../network for three images, and deploy-system.sh reads the new CRDs out of ../../../../network/config/crd. Local dev tolerates that better than CI does, but it means the lab now requires two checkouts in a specific layout with nothing checking for it up front. A clear failure early beats whatever Docker says when the context is missing.

On the canary itself, no objection. The topology, addressing and ASNs read consistently.

One thing worth calling out in the description rather than leaving in the diff: this removes the ns50 tenant entirely, including verify-ns50.sh, and replaces it with ns60. That trades a three-site IPv4 tenant for a single-site IPv6 one. You did update every reference, docs included, so nothing dangles. But anyone who used task deploy:ns50 for IPv4 coverage loses it, and that is the kind of change people should meet in the PR body rather than discover.

Also worth saying plainly, since the task description already half admits it: verify:gateway checks that CRDs are present, not that a packet crossed the datapath. A canary that passes without proving traffic flowed is the failure mode canaries exist to prevent. Fine as a first cut, but it should not stay that way once the lab can actually pass traffic.

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, with one thing that is about merge order rather than about this diff.

The COPY --from=network line this adds to containers/galactic-router/Dockerfile makes the currently-shipping router image unbuildable by the publish pipeline, which passes no build context. That is real, but CI shows it, and it disappears with the same fix as #358. So it is a sequencing constraint, not a reason to hold the branch: this should not land ahead of the module re-pin, and both COPY lines want to come out when the replace does.

Filed the canary's coverage gap as #368. Checking that the resources exist and the pods are up is a reasonable first cut, and it is also the exact shape of pass that the detached-program failure would have sailed through. Worth an assertion that a packet reached a backend and that the rule's counters moved before this is treated as datapath coverage.

The ns50 to ns60 swap I would still put in the PR body. You updated every reference, so nothing dangles, but losing the three-site IPv4 tenant is the kind of thing people should read rather than discover.

Topology, addressing and ASNs all read consistently.

@privateip
privateip force-pushed the feat/edge-gateway-06-config-manifests branch from 6797be1 to 6f42dd0 Compare August 13, 2026 01:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants