From b37594b59dcbfbdc5ff0779805331ac446c9fbab Mon Sep 17 00:00:00 2001 From: Komh Date: Fri, 24 Apr 2026 11:25:16 +0000 Subject: [PATCH] [security] "Inspecting the CA Trust Store Inside a Container" --- ...g_the_CA_Trust_Store_Inside_a_Container.md | 164 ++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 docs/en/solutions/Inspecting_the_CA_Trust_Store_Inside_a_Container.md diff --git a/docs/en/solutions/Inspecting_the_CA_Trust_Store_Inside_a_Container.md b/docs/en/solutions/Inspecting_the_CA_Trust_Store_Inside_a_Container.md new file mode 100644 index 00000000..a447e60e --- /dev/null +++ b/docs/en/solutions/Inspecting_the_CA_Trust_Store_Inside_a_Container.md @@ -0,0 +1,164 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.0,4.2.x +--- +## Issue + +A workload running inside a container needs to verify that its base image ships a specific root CA — for example, an external service it calls over HTTPS is signed under `DigiCert Global Root G2`, and the operator wants to confirm the container image already includes that root. The answer determines whether the workload "just works", or whether a custom CA bundle has to be mounted into the pod. + +The question sounds trivial (`cat the trust store`) but the path to the trust store depends on the base image's distribution family, and some images ship multiple trust bundles that the workload's HTTP client picks between by language. + +## Where to Look — By Base-Image Family + +Different Linux distributions place their system-wide CA bundle at different paths. Identify the base image first, then look at the conventional location for that family: + +| Base family | System CA bundle path | +|---|---| +| UBI 8/9, CentOS / Rocky / AlmaLinux, Fedora | `/etc/pki/tls/certs/ca-bundle.crt` (and a trust-database variant `ca-bundle.trust.crt`) | +| Debian / Ubuntu | `/etc/ssl/certs/ca-certificates.crt` (directory `/etc/ssl/certs/` holds per-CA symlinks) | +| Alpine | `/etc/ssl/certs/ca-certificates.crt` (plus a per-CA directory) | +| distroless static | No trust store by default — bundle must be mounted explicitly | +| Windows container | System trust store is in the registry, not the filesystem — different tooling | + +The path drives everything downstream. Most Linux HTTP libraries honour the system path by default: OpenSSL, curl (`--ca-native`), Go's `crypto/tls` (on UBI reads `/etc/pki/tls/certs/ca-bundle.crt`), Python's `ssl` (follows OpenSSL), and so on. A few runtimes maintain their own trust store — Java (`cacerts` keystore), Node.js (can override via `NODE_EXTRA_CA_CERTS`). For those runtimes, the system path is informational but not definitive. + +### Identify the base image + +Inspect the pod's container image reference: + +```bash +kubectl -n get pod -o jsonpath='{.spec.containers[*].image}{"\n"}' +``` + +The image's registry and tag usually reveal the family. If it's ambiguous, inspect the image's metadata: + +```bash +# Inside the running pod: +kubectl exec -n -- cat /etc/os-release +# NAME="Fedora Linux" (or "Ubuntu", "Alpine Linux", ...) +``` + +## Checking for a Specific CA + +Once the trust-store path is known, grep for the root's name: + +```bash +# UBI-family container: +kubectl exec -n -- \ + grep -l 'DigiCert Global Root G2' /etc/pki/tls/certs/* + +# Expected matches: +# /etc/pki/tls/certs/ca-bundle.crt +# /etc/pki/tls/certs/ca-bundle.trust.crt +``` + +Each matched file indicates a place the trust appears — the PEM bundle `ca-bundle.crt` is what most libraries use, and a separate trust-DB `ca-bundle.trust.crt` is consumed by OpenSSL's legacy paths. + +Find the exact certificate block to inspect its fingerprint and validity: + +```bash +kubectl exec -n -- \ + awk '/-----BEGIN CERTIFICATE-----/{flag=1} flag; /-----END CERTIFICATE-----/{flag=0}' \ + /etc/pki/tls/certs/ca-bundle.crt | \ + openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 2>/dev/null +``` + +Compare the fingerprint against the CA's published sha256 fingerprint to confirm the right certificate (not a different CA with a similar name). + +## Adding a Missing CA + +If the root is not in the image's trust store, there are two durable paths: + +### Option A — mount a custom CA bundle as a ConfigMap / Secret + +Keep the extra CA in cluster-managed storage and mount it into the pod alongside the system bundle. Most HTTP clients can be told to read an additional file: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: app-extra-ca + namespace: my-app +data: + extra-ca.pem: | + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app + namespace: my-app +spec: + template: + spec: + containers: + - name: app + image: registry.example.com/app:latest + env: + - name: SSL_CERT_FILE # OpenSSL, Python + value: /etc/pki/ca-trust/extracted/pem/extra-ca.pem + - name: REQUESTS_CA_BUNDLE # Python requests + value: /etc/pki/ca-trust/extracted/pem/extra-ca.pem + - name: NODE_EXTRA_CA_CERTS # Node.js + value: /etc/pki/ca-trust/extracted/pem/extra-ca.pem + volumeMounts: + - name: extra-ca + mountPath: /etc/pki/ca-trust/extracted/pem/extra-ca.pem + subPath: extra-ca.pem + readOnly: true + volumes: + - name: extra-ca + configMap: + name: app-extra-ca +``` + +Tune the env vars to whatever the runtime expects. Some runtimes (Java, .NET) need their own mechanisms — Java uses `-Djavax.net.ssl.trustStore=...`, .NET uses `SslClientAuthenticationOptions.RemoteCertificateValidationCallback` or the system-wide store that has to be updated with `update-ca-trust extract` at image build time. + +### Option B — rebuild the base image with the CA included + +At image-build time, place the CA in the system trust directory and re-run the update command: + +```dockerfile +# UBI-family Dockerfile fragment. +COPY extra-ca.pem /etc/pki/ca-trust/source/anchors/extra-ca.pem +RUN update-ca-trust extract +``` + +After `update-ca-trust extract`, the added CA is in `/etc/pki/tls/certs/ca-bundle.crt` automatically. All runtimes that read the system path pick it up with no pod-side configuration. + +Prefer Option B for workloads where the trusted-CA set is part of the identity of the workload (belongs in the image). Use Option A when the set changes independently of the workload (rotations, per-environment overrides) and mounting from a ConfigMap is cleaner than re-publishing images. + +## Diagnostic Steps + +Verify the trust path the runtime actually uses. From inside the pod, the runtime's own diagnostic is the authoritative answer: + +```bash +# curl — prints the configured CA file. +kubectl exec -n -- curl --version | head +kubectl exec -n -- curl-config --ca + +# Python — exposes the OpenSSL defaults. +kubectl exec -n -- python3 -c "import ssl; print(ssl.get_default_verify_paths())" + +# Java — prints the trust store path. +kubectl exec -n -- java -XshowSettings:properties 2>&1 | grep trustStore +``` + +Compare the runtime's reported path against what the `grep` query inspected — they should match. If the runtime reports a different path (e.g. a custom JVM truststore under the app's installation directory), update the trust there. + +For HTTPS connectivity testing from inside the pod, curl's verbose output shows which CA bundle was used: + +```bash +kubectl exec -n -- curl -v https://metrics-sink.example.com 2>&1 | \ + grep -E 'CAfile|CApath|SSL certificate|verify|error' +``` + +`CAfile: /etc/pki/tls/certs/ca-bundle.crt` plus `SSL certificate verify ok` confirms the runtime read the expected bundle and accepted the remote certificate. A `verify` failure indicates either the remote's chain is missing a root in the bundle (add it) or the remote sent a partial chain (the remote needs to send its intermediates). + +After applying any fix (mount or rebuild), re-run the `curl` probe from the pod; `verify ok` is the end-state.