Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,8 +254,12 @@ SPIFFE JWT-SVIDs for dynamic provider token grants:
3. Redeploy: `mise run helm:skaffold:run`

`ci/values-spire-stack.yaml` configures the local SPIRE trust domain as
`openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
`openshell.local` and adds a `ClusterSPIFFEID` that maps each sandbox pod to
`spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>`,
joining the pod's namespace, name, and `openshell.ai/sandbox-id` annotation into
one opaque final segment. Its `namespaceSelector` is left broad so SVIDs are
issued in all workspace modes; the overlay carries a commented-out stricter
selector for operators who label their managed namespaces.
OpenShell mounts the SPIFFE CSI Workload API socket at
`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token
grants. Supervisor-to-gateway authentication remains on the Kubernetes
Expand Down
21 changes: 17 additions & 4 deletions deploy/helm/openshell/ci/values-spire-stack.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,23 @@ spire-server:
clusterSPIFFEIDs:
openshell-sandboxes:
enabled: true
spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/openshell/sandbox/{{ index .PodMeta.Annotations "openshell.ai/sandbox-id" }}'
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshell
# The final path segment joins namespace, pod-name, and sandbox-id with
# "." into a single opaque token. Treat the whole SPIFFE ID as an opaque
# principal: only the trust domain is authoritative, and the final
# sandbox segment (or any substring of it) must not drive authorization.
# The sandbox-id is appended conditionally, so pods without the
# openshell.ai/sandbox-id annotation render
# spiffe://<td>/openshell/sandbox/<namespace>.<pod-name> with no trailing dot.
spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/openshell/sandbox/{{ .PodMeta.Namespace }}.{{ .PodMeta.Name }}{{ with (index .PodMeta.Annotations "openshell.ai/sandbox-id") }}.{{ . }}{{ end }}'
# Match sandbox pods in shared, managed, and operator workspace modes.
# This CI/dev overlay intentionally leaves namespaceSelector broad so it
# works without requiring namespace labels. To restrict identity issuance
# to OpenShell-managed namespaces, label those namespaces and uncomment
# the namespaceSelector block below.
namespaceSelector: {}
# namespaceSelector:
# matchLabels:
# openshell.ai/managed-by: openshell
podSelector:
matchLabels:
openshell.ai/managed-by: openshell
2 changes: 1 addition & 1 deletion docs/kubernetes/access-control.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Kubernetes sandbox supervisors authenticate back to the gateway as sandbox workl

Dynamic provider token grants can use SPIFFE without changing supervisor-to-gateway authentication. Set `server.providerTokenGrants.spiffe.enabled=true` to mount the SPIFFE CSI Workload API socket into gateway and sandbox pods while keeping the projected ServiceAccount token bootstrap and gateway-minted sandbox JWT path.

Provider token grants require a SPIFFE implementation such as SPIRE and identities for the gateway and sandbox pods. The repository's local SPIRE overlay assigns sandbox IDs from the pod's `openshell.ai/sandbox-id` annotation, but the gateway validation path only requires the supervisor SVID to be valid and in the same SPIFFE trust domain as the gateway SVID. Provider profiles with `token_grant` metadata cause the sandbox supervisor to request JWT-SVIDs and exchange them for upstream OAuth2 access tokens. Token-exchange profiles also require a gateway SPIFFE identity because the gateway brokers the intermediate token exchange with its own JWT-SVID.
Provider token grants require a SPIFFE implementation such as SPIRE and identities for the gateway and sandbox pods. The repository's local SPIRE overlay issues each sandbox a SPIFFE ID of the form `spiffe://<trust-domain>/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>`, joining the sandbox pod's namespace, pod name, and `openshell.ai/sandbox-id` annotation into a single opaque final segment (the sandbox-id is appended only when the annotation is present). Treat the whole ID as an opaque principal: only the trust domain is authoritative, and the final segment must not be parsed or used to drive authorization. The overlay's `podSelector` matches the `openshell.ai/managed-by: openshell` label present on every sandbox pod, and its `namespaceSelector` is left broad so SVIDs are issued in all workspace modes (`Shared`, `Managed`, and `Operator`), whose sandbox pods land in different namespaces. To restrict issuance to OpenShell-managed namespaces, label those namespaces and enable the stricter `namespaceSelector` shown in the overlay; in `Operator` mode the platform team must label the workspace namespaces. The gateway validation path only requires the supervisor SVID to be valid and in the same SPIFFE trust domain as the gateway SVID. Provider profiles with `token_grant` metadata cause the sandbox supervisor to request JWT-SVIDs and exchange them for upstream OAuth2 access tokens. Token-exchange profiles also require a gateway SPIFFE identity because the gateway brokers the intermediate token exchange with its own JWT-SVID.

The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the SPIFFE Workload API, so intermediate token exchange does not require gateway access to the SPIRE OIDC discovery endpoint or its TLS CA.

Expand Down
14 changes: 7 additions & 7 deletions examples/spiffe-token-exchange-demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,15 +166,15 @@ alpha called with path /:
sub: demo-user
aud: alpha, account
scope: alpha profile email
azp: spiffe://openshell.local/openshell/sandbox/<sandbox-id>
client_id: spiffe://openshell.local/openshell/sandbox/<sandbox-id>
azp: spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
client_id: spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>

beta called with path /:
sub: demo-user
aud: beta, account
scope: beta profile email
azp: spiffe://openshell.local/openshell/sandbox/<sandbox-id>
client_id: spiffe://openshell.local/openshell/sandbox/<sandbox-id>
azp: spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
client_id: spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
```

The token issuer logs both token exchange phases:
Expand All @@ -186,9 +186,9 @@ KUBECONFIG="$DEMO_KUBECONFIG" kubectl -n default logs deployment/token-exchange-
Example log lines:

```text
issued intermediate token for user=demo-user audience=spiffe://openshell.local/openshell/sandbox/<sandbox-id>
issued final token for user=demo-user audience=alpha client=spiffe://openshell.local/openshell/sandbox/<sandbox-id>
issued final token for user=demo-user audience=beta client=spiffe://openshell.local/openshell/sandbox/<sandbox-id>
issued intermediate token for user=demo-user audience=spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
issued final token for user=demo-user audience=alpha client=spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
issued final token for user=demo-user audience=beta client=spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
```

## Automated Demo
Expand Down
14 changes: 10 additions & 4 deletions examples/spiffe-token-grant-demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,14 +94,20 @@ Expected output includes endpoint-specific token claims:
alpha called with path /:
aud: alpha, account
scope: alpha profile email
azp: spiffe://openshell.local/openshell/sandbox/<sandbox-id>
azp: spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>

beta called with path /:
aud: beta, account
scope: beta profile email
azp: spiffe://openshell.local/openshell/sandbox/<sandbox-id>
azp: spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
```

The final path segment joins the sandbox pod's namespace, pod name, and
`openshell.ai/sandbox-id` annotation with `.` into a single opaque identifier.
The `token-issuer` accepts any subject under the
`spiffe://openshell.local/openshell/sandbox/` prefix in the gateway trust
domain; the segment must be treated as opaque and not parsed.

The protected services also write proof-of-life logs when they accept a call:

```bash
Expand All @@ -112,8 +118,8 @@ KUBECONFIG=kubeconfig kubectl -n default logs deployment/beta --tail=20
Example log lines:

```text
alpha accepted request path=/ aud="alpha, account" scope="alpha profile email" client_id=spiffe://openshell.local/openshell/sandbox/<sandbox-id>
beta accepted request path=/ aud="beta, account" scope="beta profile email" client_id=spiffe://openshell.local/openshell/sandbox/<sandbox-id>
alpha accepted request path=/ aud="alpha, account" scope="alpha profile email" client_id=spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
beta accepted request path=/ aud="beta, account" scope="beta profile email" client_id=spiffe://openshell.local/openshell/sandbox/<namespace>.<pod-name>.<sandbox-id>
```

## Automated Demo
Expand Down
15 changes: 12 additions & 3 deletions examples/spiffe-token-grant-demo/demo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,20 @@ PROVIDER_NAME="${PROVIDER_NAME:-spiffe-token-demo}"
PROFILE_ID="${PROFILE_ID:-spiffe-token-demo}"
PORT_FORWARD_PORT="${PORT_FORWARD_PORT:-8097}"
GATEWAY_ENDPOINT="${GATEWAY_ENDPOINT:-http://127.0.0.1:${PORT_FORWARD_PORT}}"
GATEWAY_NAME="${GATEWAY_NAME:-}"
KEEP_SANDBOX="${KEEP_SANDBOX:-0}"
ACCESS_TOKEN_SECRET="${ACCESS_TOKEN_SECRET:-$(openssl rand -hex 32)}"

# Optional `--gateway <name>` selector, shared by the OS array and cleanup.
GATEWAY_ARGS=()
if [[ -n "$GATEWAY_NAME" ]]; then
GATEWAY_ARGS=(--gateway "$GATEWAY_NAME")
fi

# Only isolate config when no gateway is named; a named gateway must resolve
# from the caller's real config (registration and stored TLS bundle).
TEMP_CONFIG_HOME=""
if [[ -z "${XDG_CONFIG_HOME:-}" ]]; then
if [[ -z "${XDG_CONFIG_HOME:-}" && -z "$GATEWAY_NAME" ]]; then
TEMP_CONFIG_HOME="$(mktemp -d)"
export XDG_CONFIG_HOME="$TEMP_CONFIG_HOME"
fi
Expand All @@ -27,7 +36,7 @@ PF_PID=""

cleanup() {
if [[ "$KEEP_SANDBOX" != "1" ]]; then
openshell --gateway-endpoint "$GATEWAY_ENDPOINT" sandbox delete "$SANDBOX_NAME" >/dev/null 2>&1 || true
openshell "${GATEWAY_ARGS[@]}" --gateway-endpoint "$GATEWAY_ENDPOINT" sandbox delete "$SANDBOX_NAME" >/dev/null 2>&1 || true
fi
if [[ -n "$PF_PID" ]]; then
kill "$PF_PID" >/dev/null 2>&1 || true
Expand Down Expand Up @@ -89,7 +98,7 @@ sandbox_curl_until() {
exit 1
}

OS=(openshell --gateway-endpoint "$GATEWAY_ENDPOINT")
OS=(openshell "${GATEWAY_ARGS[@]}" --gateway-endpoint "$GATEWAY_ENDPOINT")

printf "\n$ kubectl -n default create secret generic openshell-spiffe-token-demo --from-literal=access-token-secret=*** --dry-run=client -o yaml | kubectl apply -f -\n"
kubectl -n default create secret generic openshell-spiffe-token-demo \
Expand Down
3 changes: 3 additions & 0 deletions examples/spiffe-token-grant-demo/k8s/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ configMapGenerator:
files:
- token-issuer.js
- protected-service.js
- name: openshell-spiffe-token-demo-helper
files:
- spiffe-helper.conf

resources:
- workloads.yaml
8 changes: 8 additions & 0 deletions examples/spiffe-token-grant-demo/k8s/spiffe-helper.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

agent_address = "/run/spire/sockets/spire-agent.sock"
cert_dir = "/target"
svid_file_name = "svid.pem"
svid_key_file_name = "key.pem"
svid_bundle_file_name = "bundle.pem"
40 changes: 35 additions & 5 deletions examples/spiffe-token-grant-demo/k8s/token-issuer.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,15 @@
// SPDX-License-Identifier: Apache-2.0

const http = require("http");
const https = require("https");
const crypto = require("crypto");
const fs = require("fs");

const PORT = Number(process.env.PORT || 8080);
const JWKS_URI =
process.env.SPIRE_JWKS_URI ||
"https://spire-spiffe-oidc-discovery-provider.spire.svc.cluster.local/keys";
const SPIRE_JWKS_CA_FILE = process.env.SPIRE_JWKS_CA_FILE || "";
const SPIRE_ISSUER =
process.env.SPIRE_ISSUER ||
"https://spire-spiffe-oidc-discovery-provider.spire.svc.cluster.local";
Expand Down Expand Up @@ -57,15 +60,42 @@ async function jwks() {
if (cachedJwks && now - cachedJwksAt < 60000) {
return cachedJwks;
}
const response = await fetch(JWKS_URI);
if (!response.ok) {
throw new Error(`JWKS fetch failed with HTTP ${response.status}`);
}
cachedJwks = await response.json();
cachedJwks = await fetchJson(JWKS_URI);
cachedJwksAt = now;
return cachedJwks;
}

function fetchJson(url) {
return new Promise((resolve, reject) => {
const parsed = new URL(url);
const isHttps = parsed.protocol === "https:";
const client = isHttps ? https : http;
const options = {};
if (isHttps && SPIRE_JWKS_CA_FILE) {
options.ca = fs.readFileSync(SPIRE_JWKS_CA_FILE);
}

const req = client.get(parsed, options, (res) => {
const chunks = [];
res.on("data", (chunk) => chunks.push(chunk));
res.on("end", () => {
const body = Buffer.concat(chunks).toString("utf8");
if (res.statusCode < 200 || res.statusCode >= 300) {
reject(new Error(`JWKS fetch failed with HTTP ${res.statusCode}: ${body}`));
return;
}
try {
resolve(JSON.parse(body));
} catch (error) {
reject(error);
}
});
});
req.on("error", reject);
req.setTimeout(10000, () => req.destroy(new Error("JWKS fetch timed out")));
});
}

function hasAudience(payload, expected) {
const aud = Array.isArray(payload.aud) ? payload.aud : [payload.aud];
return aud.includes(expected);
Expand Down
27 changes: 27 additions & 0 deletions examples/spiffe-token-grant-demo/k8s/workloads.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ spec:
value: http://token-issuer.default.svc.cluster.local
- name: SPIRE_JWKS_URI
value: https://spire-spiffe-oidc-discovery-provider.spire.svc.cluster.local/keys
- name: SPIRE_JWKS_CA_FILE
value: /etc/x509/spiffe-bundle/bundle.pem
- name: SPIRE_ISSUER
value: https://spire-spiffe-oidc-discovery-provider.spire.svc.cluster.local
- name: JWT_SVID_AUDIENCE
Expand All @@ -50,10 +52,35 @@ spec:
- name: scripts
mountPath: /demo
readOnly: true
- name: spiffe-bundle
mountPath: /etc/x509/spiffe-bundle
readOnly: true
- name: spiffe-helper
image: ghcr.io/spiffe/spiffe-helper:0.11.0
imagePullPolicy: IfNotPresent
args: ["-config", "/etc/spiffe-helper/spiffe-helper.conf"]
volumeMounts:
- name: spiffe-socket
mountPath: /run/spire/sockets
readOnly: true
- name: spiffe-bundle
mountPath: /target
- name: helper-config
mountPath: /etc/spiffe-helper
readOnly: true
volumes:
- name: scripts
configMap:
name: openshell-spiffe-token-demo-scripts
- name: helper-config
configMap:
name: openshell-spiffe-token-demo-helper
- name: spiffe-socket
csi:
driver: csi.spiffe.io
readOnly: true
- name: spiffe-bundle
emptyDir: {}
---
apiVersion: v1
kind: Service
Expand Down
4 changes: 0 additions & 4 deletions examples/spiffe-token-grant-demo/provider-profile.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,12 @@ endpoints:
tls: none
access: read-write
enforcement: enforce
allowed_ips:
- 10.43.0.0/16
- host: beta.default.svc.cluster.local
port: 80
protocol: rest
tls: none
access: read-write
enforcement: enforce
allowed_ips:
- 10.43.0.0/16
binaries:
- /usr/bin/curl
- /usr/local/bin/curl
Loading