From 7a54e85584ac1a68776bec2520c8e3cf0cfdb8d9 Mon Sep 17 00:00:00 2001 From: Komh Date: Fri, 24 Apr 2026 01:39:29 +0000 Subject: [PATCH] [networking] Passthrough Ingress Fails Behind a Layer 7 External Load Balancer --- ...Behind_a_Layer_7_External_Load_Balancer.md | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 docs/en/solutions/Passthrough_Ingress_Fails_Behind_a_Layer_7_External_Load_Balancer.md diff --git a/docs/en/solutions/Passthrough_Ingress_Fails_Behind_a_Layer_7_External_Load_Balancer.md b/docs/en/solutions/Passthrough_Ingress_Fails_Behind_a_Layer_7_External_Load_Balancer.md new file mode 100644 index 00000000..52373509 --- /dev/null +++ b/docs/en/solutions/Passthrough_Ingress_Fails_Behind_a_Layer_7_External_Load_Balancer.md @@ -0,0 +1,132 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.0,4.2.x +--- +## Issue + +A client attempting to reach a workload over an `Ingress` (or equivalent route object) configured for TLS **passthrough** receives connection failures — typically `TLS handshake failed`, `unknown SNI`, or an HTTP `404` from the wrong backend — when the cluster sits behind an external Layer-7 load balancer (cloud ALB/NLB in HTTP/HTTPS mode, F5 in "HTTPS" profile, NGINX in reverse-proxy mode, etc.). The same client succeeds when it bypasses the external LB and hits the cluster ingress directly with a `Host:` override, which tells you the path through the cluster itself is fine. + +Indirectly: the workload's own certificate is served on the client's end of the connection, but the SNI extension that reached the cluster proxy either does not match any configured host (hence `404` / `SSL handshake failed`) or was stripped entirely. + +## Root Cause + +A passthrough route is contractual: the cluster ingress proxy does **not** terminate TLS. It looks at the **SNI** extension of the `ClientHello`, picks the backend, and hands the encrypted bytes straight through. This only works if the client's SNI survives every hop between the client and the cluster ingress proxy intact. + +A Layer-7 external load balancer, by definition, terminates TLS at the LB itself: it decrypts, inspects the HTTP layer, and opens a **new** connection toward the backend (the cluster ingress). That new connection carries whatever SNI the LB chooses to set — often the LB's own certificate name, or nothing — and the original client SNI is gone by the time the cluster proxy sees the handshake. The cluster proxy then either routes to a default backend, fails the handshake, or returns `404`. + +This is structural: an L7 LB cannot preserve end-to-end TLS with passthrough semantics. Either the LB must run in L4 (TCP) mode, or the cluster-side must switch off passthrough. + +## Resolution + +ACP's ingress is provided by the **ALB Operator** (`networking/operators/alb_operator`). The same two options apply regardless of the ingress implementation, and the correct choice depends on whether end-to-end TLS is a requirement. + +### Preferred (when end-to-end TLS is required): external LB in L4/TCP mode + +Keep passthrough on the cluster side, and configure the external LB as a pure TCP forwarder on port `443`. It will not terminate TLS, will not inspect headers, and will not alter SNI — the `ClientHello` reaches the cluster ingress unmodified and the cluster proxy routes it correctly by SNI. + +- Cloud NLBs (AWS NLB, GCP TCP LB, Azure Load Balancer) all support a pure L4 listener; use that, not their L7 offering. +- F5/NGINX: configure a Virtual Server with an `fastL4` profile or a `stream` block, not an HTTP virtual server. +- Health checks should be TCP to `:443`, not HTTPS, so the LB does not try to do its own TLS handshake against a backend whose certificate may not match the LB's view of the hostname. + +The downside of L4 mode is that the external LB cannot do per-path routing, WAF inspection, or HTTP-level metrics; those belong at a different layer. + +### Alternative (when L7 external LB is required): switch to edge or reencrypt termination at the cluster + +If the external LB must stay in L7 (WAF rules, per-path routing, centralised access logs), change the cluster side so it is also L7 — terminate TLS at the cluster ingress. Two patterns are supported: + +- **Edge termination.** The cluster ingress presents its own certificate and speaks plain HTTP to the backend pod. This is the simplest option when the in-cluster network is trusted. +- **Reencrypt termination.** The cluster ingress terminates the client-side TLS, then opens a new TLS connection to the backend using either the service-CA certificate or one explicitly configured on the Ingress object. This preserves encryption in the pod-to-pod path. + +Both modes let the external LB and the cluster ingress independently manage their TLS contexts. Neither needs SNI from the hop upstream, so the original termination happens cleanly. + +#### ALB Ingress example: edge termination + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: myapp-edge + namespace: myapp + annotations: + # Bind this Ingress to the named ALB CR. + project.cpaas.io/alb-name: cpaas-alb +spec: + tls: + - hosts: + - myapp.example.com + secretName: myapp-tls + rules: + - host: myapp.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: myapp + port: + number: 80 +``` + +The `tls` block terminates TLS at the ALB using the certificate in `myapp-tls`; the backend is reached as plain HTTP on port 80. Annotations that previously marked the Ingress as passthrough must be removed. + +#### ALB Ingress example: reencrypt (backend HTTPS) + +When the backend pod speaks HTTPS natively — for instance, a Java service presenting a keystore cert — switch the Ingress to talk TLS to the backend as well. The exact annotation name varies by ALB version; the reference is the ALB CR's annotation documentation. The typical pattern is: + +- The Ingress has the same `tls:` block as above (client-side termination). +- An ALB-specific annotation sets the backend protocol to `https` and optionally pins a trusted-CA certificate for the backend's self-signed cert. + +This terminates TLS twice (once at the ALB, once between ALB and backend) but removes the SNI transparency requirement across the external LB hop entirely. + +### Fallback: stock Kubernetes Ingress (no ALB) + +If the cluster runs a plain Kubernetes Ingress controller (NGINX Ingress, Traefik, etc.), the same two choices apply. NGINX has an `ssl-passthrough` annotation that behaves identically to ALB's passthrough; switching it off and supplying `tls.secretName` puts the controller in edge mode. Traefik has `TLSOption` resources for the same purpose. The external-LB-in-L4 pattern is orthogonal and works for any Ingress controller. + +## Diagnostic Steps + +Confirm the Ingress is actually in passthrough mode: + +```bash +kubectl get ingress -A -o json \ + | jq -r '.items[] + | select(.metadata.annotations + | to_entries[]? + | select(.key|test("passthrough|ssl-passthrough"))) + | "\(.metadata.namespace)/\(.metadata.name)"' +``` + +Prove the cluster path works in isolation. Bypass the external LB by resolving the hostname directly to the cluster ingress's public IP: + +```bash +curl -vk --resolve myapp.example.com:443: \ + https://myapp.example.com +``` + +A successful response here and a failure through the LB pinpoints the problem to the LB hop. + +Confirm what the external LB is doing with TLS. On most clouds this is visible in the LB's listener configuration: + +- If the listener uses **HTTPS / HTTP/2** or an L7 product (ALB, Application Gateway, Classic LB `https` listener), the LB is terminating — that matches the failure mode described. +- If the listener is **TCP / TLS passthrough / L4 NLB**, the LB is forwarding — something else is the problem. + +As a quick end-to-end SNI check, capture the `ClientHello` at the cluster ingress (on one of the ingress pod replicas): + +```bash +kubectl exec -it -n -- \ + tcpdump -i any -nn -A -s 0 'tcp port 443' | head -n 50 +``` + +(This requires `tcpdump` in the pod image; otherwise use a debug sidecar or port-forward and capture on the client machine.) If the `SNI=` field in the captured `ClientHello` is empty or matches the external LB's own certificate hostname rather than the client's intended hostname, the diagnosis is confirmed. + +Finally, check the cluster-side proxy's own logs for the handshake attempt: + +```bash +kubectl logs -n --tail=200 | grep -iE "handshake|sni" +``` + +Repeated handshake failures correlated with requests from the LB's source IPs make the last mile of evidence: whatever SNI the LB is offering does not match any configured passthrough host in the cluster.