Add a local two-cluster end-to-end test (no cloud, no GPU)

[ytsarev]

Description of your changes

Modelplane has no end-to-end test in the repo. Unit tests cover what each composition function renders, but nothing checks that the composed resources actually reconcile and serve on a real cluster. The one path that exercises the full flow (publish capacity, register a cluster, deploy a model, serve a request) is getting started, which provisions EKS/GKE/Nebius and needs GPUs, so it can't run on a laptop or gate CI without a cloud account and accelerators. The failures that only surface at runtime (missing RBAC, cross-cluster routing, provider readiness, teardown ordering) have had nothing to catch them.

This adds nix run .#e2e-local: the whole path on two local kind clusters.

• No cloud, no GPU, no credentials. The InferenceCluster registers the workload cluster from a kubeconfig Secret (source: Existing) instead of provisioning EKS/GKE/Nebius; the InferenceClass advertises a fake gpu.example.com device (claim: DRA, vendored dra-example-driver) that binds on a GPU-less node; and the ModelDeployment runs a mock OpenAI server in place of a real engine, so the pod goes Ready without a real model.
• Dev/CI parity. nix run .#e2e-local -- --verify brings it up, waits for the ModelService, and asserts a live 200. You run it locally; CI runs the identical command. The workflow is a thin wrapper (disk, cache, diagnostics, teardown) with no test logic of its own.
• Label-gated. Behind a test-e2e PR label, not every push (a run is 15-20 minutes). No secrets, so it can gate a PR on its own.
• Real orchestration. Two clusters, because the control-plane and workload gateways both install the Gateway API CRDs and collide on one; the full ModelDeployment, ModelReplica, ModelEndpoint, ModelService chain; a DRA ResourceClaim actually bound on the workload cluster.

nix run .#e2e-local -- --verify brings both clusters up, asserts the endpoint serves (it curls from a pod on the control plane and checks the status), and leaves the whole platform queryable, no cloud in sight:

$ nix run .#e2e-local -- --verify
...
==> ModelService address: http://172.19.255.200/ml-team/mock
==> verify attempt 1: HTTP 200
==> End to end OK: http://172.19.255.200/ml-team/mock served HTTP 200

$ kubectl get inferencegateway,inferencecluster,inferenceclass
NAME                                     GATEWAY   ADDRESS          SYNCED   READY
inferencegateway.modelplane.ai/default   Traefik   172.19.255.200   True     True
NAME                                   SOURCE     GATEWAY          SYNCED   READY
inferencecluster.modelplane.ai/local   Existing   172.19.255.100   True     True
NAME                                         SYNCED   READY
inferenceclass.modelplane.ai/synthetic-gpu   True     True

$ kubectl -n ml-team get modeldeployment,modelservice
NAME                                      REPLICAS   SYNCED   READY
modeldeployment.modelplane.ai/mock-demo   1          True     True
NAME                              ADDRESS                              SYNCED   READY
modelservice.modelplane.ai/mock   http://172.19.255.200/ml-team/mock   True     True

Scope. Stops short of real token generation (mock engine), GPU drivers/CUDA (fake DRA), cloud provisioning, and multi-node serving. hack/local-e2e/README.md has the design and a tested/not-tested table; CONTRIBUTING gains a note placing the two layers (unit via nix flake check, integration via nix run .#e2e-local).

Alternatives ruled out.

• A declarative test runner like Chainsaw for the assertions: the hard part is the two-cluster bootstrap, which is irreducible shell a runner would only wrap, so it wasn't worth a third testing idiom for one scenario.
• A small real model on CPU: exercises the inference engine, not Modelplane's orchestration, at a large cost in image size, weights, and runtime.

nix run .#e2e-local -- --verify passes both locally and in CI (green run: https://github.com/ytsarev/modelplane-public/actions/runs/29854197681).

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes. (no functions/ changes)
• Signed off every commit with git commit -s.

[Copilot]

Pull request overview

Adds a laptop/CI-friendly local end-to-end integration test for Modelplane that exercises the real reconciliation + routing path on two kind clusters, without requiring cloud credentials or GPUs.

Changes:

• Introduces nix run .#e2e-local to bootstrap a two-cluster local environment and (optionally) verify the served endpoint returns HTTP 200.
• Adds a label-gated GitHub Actions workflow (test-e2e) that runs the same -- --verify command used locally.
• Documents the new integration-test layer in hack/local-e2e/README.md and CONTRIBUTING.md.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
nix/apps.nix	Adds the e2e-local flake app wrapper that materializes function images and delegates to hack/local-e2e/run.sh.
flake.nix	Exposes the new e2e-local app.
hack/local-e2e/run.sh	Implements two-cluster orchestration and optional --verify live request check.
hack/local-e2e/README.md	Documents design, prerequisites, usage, and caveats for the local e2e flow.
hack/local-e2e/dra-example-driver.yaml	Vendors a fake DRA driver to allow DRA claims to bind on GPU-less kind nodes.
hack/local-e2e/manifests/00-namespaces.yaml	Creates the ml-team namespace for test resources.
hack/local-e2e/manifests/10-inference-gateway.yaml	Defines the control-plane InferenceGateway with MetalLB address pool (subnet-templated by the script).
hack/local-e2e/manifests/20-inference-class.yaml	Adds a synthetic InferenceClass advertising a claimable DRA device.
hack/local-e2e/manifests/30-inference-cluster.yaml	Registers the workload cluster via source: Existing using a kubeconfig Secret.
hack/local-e2e/manifests/40-model-deployment.yaml	Deploys a mock OpenAI-compatible engine to exercise routing without real inference.
hack/local-e2e/manifests/50-model-service.yaml	Fronts the deployment with a single OpenAI-compatible endpoint and publishes .status.address.
CONTRIBUTING.md	Adds guidance distinguishing unit checks (nix flake check) vs integration e2e (nix run .#e2e-local).
.github/workflows/e2e.yml	Adds label-gated CI execution of the local e2e (test-e2e) with diagnostics + teardown.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

hack/local-e2e/run.sh:218

• curl_status assumes the caller-provided pod name is unique, but kubectl run errors are swallowed (|| true). If a previous run crashed and left an e2e-verify-* pod behind, a later run can fail to create the pod and then read stale logs from the old pod, causing false positives/negatives. Generate a unique pod name inside curl_status (or explicitly delete any existing pod) before running curl.

	local pod="$1" url="$2"
	shift 2
	kubectl --context "$cpctx" -n "$ns" run "$pod" --restart=Never \
		--labels=app.kubernetes.io/name=e2e-verify --image="$CURL_IMAGE" \
		--command -- curl -sS --max-time 15 -o /dev/null -w '%{http_code}' "$url" "$@" \

hack/local-e2e/dra-example-driver.yaml:71

• ClusterRoleBinding is cluster-scoped; metadata.namespace should be omitted to avoid confusion and potential schema-validation failures.

kind: ClusterRoleBinding
metadata:
  name: dra-example-driver-role-binding
  namespace: dra-example-driver

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

hack/local-e2e/run.sh:249

• The verify pods use deterministic names (e.g. e2e-verify-oai-1). If a previous run is interrupted before cleanup, re-running can hit an existing pod name; since kubectl run errors are ignored, the subsequent kubectl logs call can read stale output and produce a false 200. Make the pod name unique per script invocation (e.g. suffix with $$) to avoid cross-run collisions.

	code="$(curl_status "e2e-verify-oai-$attempt" "$addr/v1/chat/completions" -H 'content-type: application/json' -d "$oai")"

hack/local-e2e/run.sh:264

• The Anthropic verify step uses a fixed pod name (e2e-verify-anthropic). If a prior run left this pod behind, kubectl run can fail (and is ignored), and kubectl logs may read stale output from the old pod. Suffix the name with $$ (or another unique token) to make runs independent.

mcode="$(curl_status e2e-verify-anthropic "$addr/v1/messages" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d "$ant")"

[haarchri]

we we need the cluster scoped ? - in the apis/mrap.yaml we have

  # provider-helm and provider-kubernetes (serving stack, gateway, caches).
  - releases.helm.m.crossplane.io
  - objects.kubernetes.m.crossplane.io

[ytsarev]

Good catch. Dropped the cluster-scoped forms — the BYO compositions run provider-helm/kubernetes v2, whose MRs are namespaced (.m.crossplane.io), exactly what apis/mrap.yaml activates. The lean MRAP now keeps only *.kubernetes.m.crossplane.io and *.helm.m.crossplane.io.

Verified with a fresh run, locally and in CI — both /v1/chat/completions and /v1/messages still return 200. (2063156)

[ytsarev]

https://github.com/ytsarev/modelplane-public/actions/runs/30088208329/job/89465250927 related CI run

[haarchri]

i wonder if we can run some python test-suites after the project is running in kind - perhaps good for an follow up ?!

[ytsarev]

Yes, I would love to use the consolidated test framework, but upbound-style E2E API is not yet available upstream.

[ytsarev]

Great idea. Opened #368 to track it as a follow-up. I've kept this PR's --verify at a smoke-level 200; behavioral suites against the running control plane are the natural next step (pairs with the CPU-model idea, for something meaningful to assert against). Thanks!

[haarchri]

are we sure that an override of this will also test something we bumped recently ? or is this followup then ?! disaggregation with EPP and then we have multiple container involved eg. https://github.com/modelplaneai/modelplane/blob/main/functions/compose-model-replica/function/routing.py#L60-L61 https://github.com/modelplaneai/modelplane/blob/main/functions/compose-model-replica/function/routing.py#L523 recently bumped because it was not working with Anthropic message style .. eg. #360

[ytsarev]

Good question — split answer:

• The recent Unified-through-EPP routing and the Anthropic /v1/messages surface (feat(docs): Serve the Anthropic Messages API (Claude Code) end to end #360) are exercised: the mock is a Unified engine, so a7d28c3 routes it through the composed InferencePool + endpoint-picker + HTTPRoute, and --verify asserts 200 on both /v1/chat/completions and /v1/messages through that path.
• Disaggregation (PrefillDecode — the pd-sidecar, two-phase engines, disagg EPP config, multiple containers) is not covered here — a single Unified engine doesn't touch it. That's a genuine follow-up, already listed under "Not tested" in the harness README.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

hack/local-e2e/run.sh:248

• The verify pods use deterministic names (e2e-verify-oai-1..10 / e2e-verify-anthropic) while kubectl run errors are ignored. If a previous run leaves any of these pods behind, subsequent runs can fail to create the pod and then read stale logs from the old pod, potentially causing false passes or confusing flakiness. Use a per-run unique suffix in the pod name so each attempt always targets a fresh pod/log stream.

# OpenAI /v1/chat/completions, retried: the address can publish a moment before
# the cross-cluster route is serving, and a slower CI runner widens that gap.
oai='{"model":"'"$svc"'","messages":[{"role":"user","content":"ping"}]}'
code=""
for attempt in $(seq 1 10); do