Validate self-managed PKI end to end

[nv-lacm]

Description

Validate the complete self-managed PKI workflow in a local test environment.

Definition of Done

• The stack renders with cert-manager and OpenBao authentication configured.
• The control-plane attachment and compute-plane registration establish transport trust.
• An LLM workload connects over certificate-validated reverse QUIC.
• Missing or incorrect trust material fails closed.
• Validation results are recorded.

[mikeyrcamp]

Validation update (2026-08-19)

Environment:

• Fresh two-cluster k3d environment: one control plane and one compute plane.
• Source revision: 2a08528429834a61416f831f334c1d4eb76262a3.
• Self-managed stack v0.8.0-dev.178, compute stack v0.2.0-dev.382, request-router chart 1.9.0, NVCA Operator chart 1.21.2, and NVCA 3.2.0.

Passed:

• The stack renders secure request-router configuration with cert-manager and OpenBao. The issuer and router Certificate report Ready=True.
• The mounted server identity contains a leaf plus intermediate certificate. It verifies to the exported root, has the expected service DNS SAN and server-auth EKU, and the router starts without the insecure QUIC flag.
• Compute registration succeeds. The live workload transport profile uses bundle trust, carries a non-empty root bundle and fingerprint, and has insecure QUIC disabled.
• A live reverse-QUIC handshake succeeds with the expected DNS name and returns a two-certificate server chain.
• Fail-closed probes reject an empty trust bundle, an unrelated root, and the correct root with an incorrect server name. None falls back to insecure transport.
• A router built from this main revision picks up a cert-manager renewal without a pod restart. The new identity became active after 44 seconds, the successful reload counter incremented, and the expiry gauge advanced.
• A mismatched certificate/key generation was rejected while the last-known-good identity remained available. The rejection counter incremented, readiness stayed healthy, and the pod UID and restart count did not change.
• Restoring valid material recovered automatically after 63 seconds. The success counter and expiry gauge advanced, with the same pod UID and zero restarts.

Release blocker:

• The artifact selected by request-router chart 1.9.0 predates the server-identity reload implementation merged in feat(stargate): reload mounted TLS server identities without restarts #777.
• After cert-manager issued a new Secret generation, that published process continued serving the previous certificate for 90 seconds. Its pod UID was unchanged and restart count stayed at zero.
• A newer published artifact showed the same behavior and did not expose the reload metrics. The source build from main passed the same live test.

The issue should remain open until:

• An LLM function is created, deployed to the registered compute plane, and successfully invoked through certificate-validated reverse QUIC.
• A request-router artifact containing feat(stargate): reload mounted TLS server identities without restarts #777 is published and the released chart is pinned to it, then the rotation probe is repeated without an image override.

[mikeyrcamp]

End-to-end LLM validation found a remaining blocker, so this issue should stay open.

Validation scope:

• Created an LLM function named issue-52-e2e with model dummy-model using the repository test chart and test image.
• Deployed one instance with a 1-GPU H100 profile.
• Repeated the deployment with an NVCA agent built from the PR feat(self-managed): support a pre-created LLM request-router TLS Secret #966 merge commit.
• Kept the request router on the source build containing PR feat(stargate): reload mounted TLS server identities without restarts #777.

Observed result:

• The function workload, utility pod, and LLM sidecars became Ready, and the control plane reported the deployment as successful.
• The real llm-worker runs in the separately constructed utils pod.
• That container had no STARGATE_TLS_CERT_PATH, no merged CA bundle mount, and no --quic-insecure flag.
• After adding only the test server's actual health path, pylon started and attempted the secure reverse connection. It failed closed with TLS cert required when --quic-insecure is not set.
• The worker therefore never established the secure router connection through the unmodified deployment path.

Root cause in the current merged source:

• MiniService calls prepareTransportTLSForWorkloads for rendered Helm workload objects.
• The utils pod is built separately and contains llm-worker, but it is not included in that injection call.
• Unit coverage exercises a pod containing llm-worker, but it does not cover the real translated Helm LLM layout where the worker is in utils.

Additional end-to-end gaps exposed by the run:

• The repository test server exposes /healthz and /readiness, while pylon probes /health and /v1/health/ready by default. The generated pylon arguments do not carry the test server's health path.
• Deployment readiness does not currently prove that the pylon worker registered with the request router. The deployment was reported successful while secure reverse connection was unavailable.
• The released request-router artifact still needs a build and chart pin containing PR feat(stargate): reload mounted TLS server identities without restarts #777.
• The released self-managed Cassandra migration pin must include migration 8 before the current API can create the LLM function schema without an override.

Required closure evidence:

1. Include the translated utils pod in transport trust injection.
2. Add an integration test for the real Helm LLM layout that asserts the CA bundle, STARGATE_TLS_CERT_PATH, and absence of --quic-insecure on utils/llm-worker.
3. Align or propagate the upstream health endpoint used by the test workload.
4. Prove worker registration and a successful end-to-end invocation over verified QUIC before reporting the deployment ready.
5. Publish and pin the router and migration artifacts needed by the default self-managed path.

The secure failure behavior is correct. Automatic trust propagation for the actual Helm LLM worker path is not complete yet.

[mikeyrcamp]

Validation update from latest origin/main (2026-08-19)

Environment:

• Source revision: d8c4a5b1d9fdf140268a65e31641d5402e115c21.
• Fresh two-cluster local environment with separate control and compute planes.
• NVCA Operator chart 1.21.3, NVCA 3.2.7, request-router chart 1.7.2, and request-router application 0.9.0.

Passed:

• cert-manager and the OpenBao-backed issuer became ready.
• The request-router server identity contained a leaf plus intermediate certificate, verified to the distributed root, contained the expected DNS SAN and server-auth EKU, and ran without insecure QUIC.
• NVCA 3.2.7 fixed the previously reported real-layout trust injection gap. The generated utils/llm-worker received STARGATE_TLS_CERT_PATH, a non-empty registration trust bundle and matching fingerprint, a read-only merged certificate volume, and no --quic-insecure flag.
• With explicit test-only L4 reachability between the clusters, the worker registered successfully. Reverse QUIC negotiated TLS 1.3 with the expected DNS identity.
• A real LLM function deployment registered dummy-model, and invocation through the public LLM endpoint returned the expected response over the verified reverse-QUIC connection.
• Repeating the connection with only the system trust store, which excludes the issued root, failed closed with invalid peer certificate: UnknownIssuer. It did not fall back to insecure transport.

Remaining default-path blockers:

1. Split control-plane and compute-plane installs do not create a worker-facing request-router route. The existing llm.invocation HTTPRoute covers client-to-gateway traffic only. Workers also need a reachable TCP route for registration and discovery and a UDP route for reverse QUIC.
2. The request-router chart does not expose values for the externally reachable gRPC and reverse-tunnel dial addresses. The runtime supports --grpc-pylon-dial-addr and --reverse-tunnel-pylon-dial-addr; the chart must set them while retaining the certificate DNS name as the authority and SNI identity.
3. The published request-router application selected by current main still does not expose the certificate reload metrics used to verify the behavior from feat(stargate): reload mounted TLS server identities without restarts #777. The released chart still needs a pin to an artifact containing that implementation.
4. The default self-managed package still required artifact overrides during this run: the Cassandra image selected by chart 0.19.1 was unavailable, and the selected Cassandra migration did not contain the LLM schema needed by the current API.
5. The released NVCA deployment did not include the upstream readiness-path argument present in current source, so the repository test backend needed a compatibility proxy for its readiness endpoint.

Conclusion:

The PKI and NVCA 3.2.7 trust propagation behavior pass, including successful invocation and fail-closed rejection. Issue #52 should remain open until the split-cluster L4 route contract and the required released artifact pins work from the unmodified default installation path.

[mikeyrcamp]

Released-artifact validation reached a new first-failure boundary on 2026-08-25.

Direct results:

• The released router and gateway-routes charts were used.
• The split-cluster NATS route was enabled declaratively through the documented values-driven workflow. The route reported Accepted=True and ResolvedRefs=True, a compute-side probe received a NATS INFO banner, and the existing NVCA agents recovered without pod deletion.
• A CLI built from exact merged main revision f8b622d1 passed go test ./...; cluster list-registered reported the registered compute plane READY.
• A real LLM function scheduled successfully and its pod reached 3/3 Ready. The credential manager created and refreshed the worker token.

The managed transport-trust criterion is not yet satisfied:

• The live pylon arguments contained neither --stargate-ca-cert-path nor an insecure bypass.
• The worker pod had no transport-trust bundle mount.
• The values-driven compute environment and registration handoff both had agentConfig.mergeConfig=null.
• Pylon logged reverse-registration startup, but router/Stargate evidence did not show inference-server registration, reverse QUIC relay, or an established tunnel.

Source inspection distinguishes the workflows: profile-driven nvcf-cli compute-plane register --control-plane-profile ... renders bundle-mode workload.transportTLS with the public CA PEM and fingerprint, while the Helmfile make register-cluster path used here registers the cluster without consuming that profile. The canonical multi-cluster BDD fixture uses workload.stargateQUICInsecure=true, which is not acceptable evidence for this PKI issue.

Validation stopped before invocation or router failover. The next action is to exercise one complete documented trust-bearing workflow (or explicitly add the exported public bundle/fingerprint contract to the values-driven workflow), then require direct secure registration and tunnel evidence before invoking. No live resource hand-patch or insecure trust bypass was used.

[mikeyrcamp]

Released profile-path validation update (2026-08-25)

Passed:

• Profile-driven compute registration wrote the canonical
  registration-values filename with bundle trust, one public CA certificate,
  the expected fingerprint, no private key, and no insecure QUIC setting.
• The pinned Helmfile 1.1.9, Helm 3.15.4, and helm-diff 3.9.11 toolchain
  completed the no-apply render. The render selected NVCA Operator chart
  1.21.3 from OCI-backed sources and contained no snapshot, latest-tag, or
  local-chart marker.
• The supported compute apply completed. The live Helm values retained the
  trust bundle and fingerprint. The operator rolled successfully, restarted
  NVCA 3.2.7, and the generated agent config contained the bundle trust.

Remaining blocker:

• A pre-existing Ready LLM worker did not adopt transport trust during a
  five-minute condition wait.
• Its pod UID did not change. It had no STARGATE_TLS_CERT_PATH, no
  nvcf-trust-bundle-install init container, no
  nvcf-transport-trust-bundle volume, and no merged-certificate volume.
• No insecure QUIC setting appeared.
• The workload namespace had no generated transport trust ConfigMap.
• The owning ICMSRequest remained RequestCompletionAcknowledged while the
  restarted agent loaded the new transport TLS configuration.

This directly proves propagation through Helm, the operator, and the restarted
agent. It does not prove why the acknowledged workload was not re-rendered. The
current evidence supports the inference that completed requests are not
reconciled when cluster-wide transport trust changes.

Correction to the earlier validation signal: current source does not inject
--stargate-ca-cert-path. The current bundle-trust path injects
STARGATE_TLS_CERT_PATH, an install init container, a trust-bundle ConfigMap
volume, and an in-memory merged-certificate volume. Those are the acceptance
signals going forward.

Validation stopped before worker registration, reverse QUIC, invocation, or
router failover. No workload was deleted, restarted, redeployed, or
hand-patched. The next action is to define and verify the supported behavior
for existing workloads when transport trust changes, then repeat the secure
end-to-end path.

[mikeyrcamp]

Fresh-worker validation update (2026-08-25)

The pre-existing acknowledged worker was removed from this run's acceptance
scope. A newly created LLM function was deployed instead.

Passed on the fresh workload:

• The new pod reached 3/3 Ready after both init containers completed, with
  zero restarts, and the function API reported ACTIVE.
• Managed trust was present through STARGATE_TLS_CERT_PATH, the trust-install
  init container, the source trust ConfigMap volume, and the merged-certificate
  volume. There was no insecure transport marker.
• The workload ConfigMap contained one public certificate and no private key.
  Its canonical nvcf-trust-bundle-v1 fingerprint exactly matched the
  generated registration values and live Helm values.
• Both the request-router and Stargate addresses were non-empty qualified DNS
  authorities, preserving the authority/SNI identity.
• The credential manager obtained, wrote, cached, and scheduled refresh for
  the worker token without exposing it.

New first-failure boundary:

• Pylon logged that Stargate registration started in reverse-tunnel mode, but
  did not log tunnel establishment.

• An authenticated OpenAI chat-completions request using the released gateway
  route's documented LLM Host header reached the LLM API gateway and returned
  HTTP 503.

• The response and correlated API-gateway log both reported:

  rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

• The released request-router pod has the expected TLS certificate/key mount
  and TLS path arguments, while the worker has the matching managed root
  bundle. The present evidence does not yet identify which internal hop is
  serving plaintext to a TLS client.

Validation stopped at this correctly routed pre-failover invocation. No live
resource was patched, no insecure bypass was enabled, and no router pod was
deleted. The next action is to localize the failing client/server hop and make
its released transport configuration consistent, then repeat invocation before
the supported single-router-pod failover test.

[mikeyrcamp]

Root-cause correction for the fresh-worker 503

The failing hop is now localized. The HTTP 503 occurs before the request
router or worker receives the invocation:

1. LLM API Gateway calls the NVCF API AuthLlmInvocation gRPC method.
2. The live gateway configuration enables TLS for that client and targets the
   NVCF API service on port 9090.
3. That service maps directly to the NVCF API pod's plaintext gRPC listener.

A read-only port-forward reproduced both sides:

• A TLS probe failed with tls_validate_record_header: wrong version number.
• A plaintext HTTP/2 probe connected successfully and returned HTTP/2 with
  grpc-status: 13 and grpc-message: Content-Type is missing from the request.

Source and chart inspection agree that the self-managed NVCF API endpoint on
9090 is plaintext and that the released API chart has no gRPC server-certificate
wiring. The retained validation environment omitted
addons.llm.gateway.auth.grpcInsecure: true; both the current self-managed LLM
guide and the multi-cluster BDD fixture include that local-only setting.

This option controls the gateway-to-NVCF authorization RPC. It is separate
from the worker-to-router managed-PKI path and does not disable the fresh
worker's certificate-validated QUIC configuration.

No configuration or live resource was changed during diagnosis. The earlier
statement that the internal hop was not localized is stale. This 503 also does
not prove a reverse-QUIC failure, because authorization stopped the request
before the gateway invoked the request-router upstream. Tunnel establishment
and invocation remain to be tested after the authorization transport modes are
made consistent.

[mikeyrcamp]

Released-artifact validation now proves the positive path with request-router chart 1.10.0 and gateway-routes chart 1.16.0: managed issuer and router certificate Ready, public trust propagated to a fresh worker, certificate-validated reverse QUIC established for all three router authorities, successful invocation, backend-router pod recovery, and successful post-failover invocation. The functional stable-stack run is complete. Keeping this issue open for the remaining released-artifact negative-trust cases and the certificate-rotation-without-restart acceptance described by epic #19; the echo workload capacity test is tracked separately in #1230 and is not part of this issue.

[mikeyrcamp]

Released-artifact closure validation (2026-08-26)

Status: RESOLVED

Validated source revision: 8f8b07f71478c644796746659a82758d487e4df6 (origin/main). The live release used request-router chart 1.10.0 and Stargate application 0.11.1. The already-recorded positive path (managed issuer and identity, trust propagation, fresh-worker registration, three verified reverse-QUIC authorities, invocation, supported router failover/recovery, and post-failover invocation) is direct evidence in the preceding update and was not repeated destructively.

Repository verification on the same revision:

• cargo test --locked -p stargate-tls: 19 passed, 0 failed. This includes empty-trust rejection, projected-volume identity loading, invalid-generation rejection, and new-handshake rotation coverage.
• cargo test --locked -p stargate-k8s-router: 74 passed, 0 failed, 3 performance benchmarks ignored. This includes secure relay SNI preservation and Raw QUIC/WebTransport projected-identity reload tests.

Released fail-closed probes used a temporary wrapper over stargate_tls::build_trusted_quic_client_config_with_alpn and quinn::Endpoint::connect, targeting the released UDP listener. The command form was quic_tls_probe <router-address>:50072 <server-name> with the trust bundle on stdin; no cluster object was changed.

• Empty trust: exit 1, TLS trust bundle contains no certificates.
• Unrelated self-signed root with the correct service name: exit 1, invalid peer certificate: UnknownIssuer.
• Managed root with wrong-name.invalid: exit 1, certificate not valid for name; the peer identity contained only the intended stable-service and wildcard headless-service SANs.
• Managed root with llm-request-router.nvcf.svc.cluster.local: exit 0, verified QUIC handshake succeeded.

Released rotation used SHA-256-verified cmctl v2.5.0 and the supported command:

cmctl renew --namespace nvcf stargate-quic-tls

Direct before/after results:

• Certificate revision advanced 1 -> 2; serial, SHA-256 fingerprint, validity interval, and Secret resource version all changed.
• Every one of the three router replicas logged TLS material reloaded for server_identity with result=success.
• On every replica, stargate_tls_reloads_total{material_type="server_identity",result="success"} changed 0 -> 1; result="rejected" stayed 0.
• stargate_tls_certificate_expiry_seconds{material_type="server_identity"} changed from 1795444603 to 1795533552 on all replicas.
• Each pod UID and container ID was identical before and after; every restart count stayed 0 and all pods remained Running.
• A new verified QUIC handshake after reload returned exit 0. Both registered compute workloads remained Ready with zero restarts.

No workload, router pod, or controller was restarted or deleted. No live resource was patched, no insecure bypass was enabled, and no chart/image override was used. The remaining released-artifact negative-trust and rotation-without-restart acceptance is complete, so this issue can close. Root-CA rotation remains the separate documented rolling Pylon restart case; this validation rotated only the server leaf under the already-trusted issuer.