🛰️ relops-bootstrap

Zero-touch provisioning infrastructure for Mozilla RelOps CI workers (macOS today, eventually all hardware).

GCP-native, workload-identity-style secret delivery via SCEP-issued per-host client certs and mTLS at the load balancer. The same primitive — a DEP-enrolled host that holds an SCEP client cert can fetch its role's secrets from a brokered Secret Manager — covers every operator-driven flow:

🚀	first-time provisioning of brand-new hardware out of DEP enrollment
🔄	re-provisioning an existing host via EACS (Erase All Content and Settings — what that actually does, button-press → re-enroll)
🪪	re-keying a host whose cert was compromised or whose role changed
⏳	routine cert rotation — SCEP renews on its own, no operator action

No laptop step. No shared secret copied out of a password manager. No SSH session to type vault.yaml into. Every secret read is logged with the requesting cert's serial number — "who pulled what, when" is a one-liner in Cloud Audit Logs.

Status: Zero-touch and live for the m4 pools (2026-07). One flow, driven by the orchestrator/ CLI (reprovision): quarantine → drain → EACS wipe → DEP re-enroll → SimpleMDM profiles + signed bootstrap PKG → SCEP cert in keychain → auto-mint SecureToken

• escrow Bootstrap Token → bootstrap PKG fetches vault.yaml → puppet → worker re-registered in Taskcluster → auto-unquarantine. Proven end-to-end across the m4 staging pool (macmini-m4-111…115) and verified reprovision-ready on the m4 production pool.

Why the SecureToken mint needs an on-network runner. DEP skips Setup Assistant, so the managed admin holds no SecureToken until a PAM (password) login — and Apple only grants the first SecureToken through an interactive login session, which in this context can only be driven over an on-host SSH session. That single Apple constraint is why the driver has to run inside the datacenter. The runner performs that login automatically (reprovision mint / scripts/mint-securetoken.sh), so it is not a manual per-host step — it's the reason the runner is on-network rather than a Cloud Run job.

One-click from Hangar. The whole reprovision is a button in the hangar fleet dashboard (individual host or a whole pool). Hangar (Cloud Run) can't reach MDC1, so it only queues a job; a Puppet-managed on-network runner (macmini-m4-81, ronin_puppet role gecko_t_osx_1500_m4_reprovision_runner — the orchestrator under a LaunchDaemon) claims it over mTLS, outbound-only, runs reprovision, and streams every phase back into a live view. Nothing dials into the datacenter. The operator never needs a terminal.

---

🏗️ Architecture

                 ┌─────────────────────────────────────────────────┐
                 │   host (m4 Mac Mini, fresh out of DEP)          │
                 │                                                 │
                 │   1. DEP enroll (Setup Assistant skipped) →      │
                 │      managed admin (fixed DEP password)          │
                 │   2. Runner auto-mints the first SecureToken     │
                 │      over an on-host ssh login; BST escrowed      │
                 │   3. SCEP profile → mdmclient → keypair +        │
                 │      cert in System keychain                     │
                 │   4. Signed bootstrap PKG (managed install):     │
                 │      CURL_SSL_BACKEND=securetransport \          │
                 │      curl --cert "<CN>" https://forge/secret/X   │
                 │      (TLS handshake signs via OS network stack,  │
                 │       which holds the keychain sign ACL)         │
                 └─────────────────┬───────────────────────────────┘
                                   │ mTLS
                                   ▼
       ┌──────────────────────  GCP project: relops-bootstrap  ──────────────────────┐
       │                                                                              │
       │   🛡️ HTTPS LB                                                                │
       │      • Cloud Armor (source-CIDR allowlist, MDC1 NAT)                         │
       │      • Trust Config: step-ca root + intermediate (validates client cert)     │
       │      • Server TLS Policy: ALLOW_INVALID_OR_MISSING (so /scep/* still works)  │
       │      • Forwards X-Client-Cert-{Present,Chain-Verified,Leaf,Serial,SPIFFE}    │
       │      • Live at https://forge.relops.mozilla.com (Google-managed cert)        │
       │                                                                              │
       │           │                                          │                       │
       │           ▼                                          ▼                       │
       │   🪪 step-ca                              📜 vault-broker                    │
       │   GCE VM (zonal NEG)                      Cloud Run (serverless NEG)         │
       │   SCEP issuer per puppet role             • Reads X-Client-Cert-Leaf         │
       │   Cert SPIFFE URI =                       • URL-decodes SPIFFE URI from SAN  │
       │     spiffe://relops.mozilla/              • Checks role matches URL path     │
       │       host/<CN>/role/<role>               • Reads vault-<role> secret        │
       │                                           • Per-cert-serial rate limit       │
       │                                           • Logs to Cloud Audit              │
       │                                                                              │
       │                                           🔐 Secret Manager                  │
       │                                              vault-<role> per puppet role    │
       │                                              (mirrors the 1P "RelOps" vault) │
       │                                                                              │
       └──────────────────────────────────────────────────────────────────────────────┘

Why mTLS-at-LB instead of JWT-signed-by-cert?

macOS keychain ACL gates SecKeyCreateSignature to a hard-coded list of network-stack helpers (configd, nehelper, NEIKEv2Provider). A Developer-ID-signed binary running as root in a LaunchDaemon — let alone our bootstrap shell — gets errSecInteractionNotAllowed when trying to sign with a keychain-stored key. We proved this end of every plausible workaround (AllowAllAppsAccess=true, KeyIsExtractable=true, SecItemExport, ACME via the data-protection keychain). The keychain genuinely will not let our code use the SCEP key directly.

But the keychain ACL does allow signing from the OS network stack — which is exactly the path curl takes when invoked with CURL_SSL_BACKEND=securetransport. The TLS handshake's CertificateVerify is signed inside SecureTransport (via the helpers in the allowlist), not in our process. Our code never touches the private key. No PKCS12 extraction. No key on disk. The cert never leaves the keychain.

The LB validates the cert chain via its Trust Config (step-ca root + intermediate) and forwards parsed cert info to the broker as request headers. The broker reads X-Client-Cert-Leaf and parses the SPIFFE URI out of the SAN — we don't trust the LB's built-in SPIFFE parser because it silently drops URI SANs with URL-encoded chars (e.g. Mac%20mini).

---

📂 What's in this repo

.
├── terraform/                    🏗️  GCP infra
│   ├── main.tf, variables.tf, outputs.tf
│   ├── vault_broker.tf           ─ Cloud Run service + IAM
│   ├── step_ca.tf, network.tf    ─ step-ca GCE VM + VPC + firewalls
│   ├── lb.tf                     ─ HTTPS LB + Cloud Armor + custom request headers
│   ├── mtls.tf                   ─ Trust Config + Server TLS Policy
│   ├── trust/{root,intermediate}.pem ─ step-ca PEMs the Trust Config points at
│   ├── iam.tf                    ─ scoped IAM bindings (least privilege)
│   ├── secrets.tf                ─ per-puppet-role secret containers
│   └── artifact_registry.tf      ─ broker image registry
│
├── broker/                       📜  FastAPI vault-broker service
│   ├── app/                      ─ reads X-Client-Cert-* headers, parses leaf,
│   │                               URL-decodes SPIFFE, role-binds, fetches secret
│   ├── tests/                    ─ pytest happy-path + adversarial cases
│   └── Dockerfile, pyproject.toml
│
├── orchestrator/                 🧰  Operator CLI (`reprovision`)
│   └── ... TC (Hawk) + SimpleMDM clients + workflow steps (quarantine→wipe→mint→escrow→wait)
│                                   see orchestrator/README.md for the pipeline + golden paths
│
├── pkg/                          📦  Signed bootstrap PKG builder (current delivery)
│   ├── build.sh                  ─ builds + Developer-ID-signs the component pkg (43AQ936H96)
│   ├── payload/                  ─ /usr/local/sbin/m4-bootstrap.sh + entrypoint LaunchDaemon
│   └── scripts/postinstall       ─ kicks the entrypoint on install
│
│   (provisioner/ — the old Cloud Scheduler → Cloud Run cron reconciler that fired
│    SimpleMDM script-jobs — was DECOMMISSIONED 2026-07. It is superseded by the signed
│    PKG (delivery at DEP convergence) + the Hangar on-network runner (orchestration).
│    Removed from the repo and torn down in GCP; see git history if you need the guard logic.)
│
├── mdm/                          📱  SimpleMDM artifacts
│   ├── scep-relops.mobileconfig.template
│   └── com.mozilla.relops.bootstrap.plist     ─ optional periodic refresh daemon
│
├── scripts/                      🛠️  Operator helpers + worker bootstrap
│   ├── bootstrap-step-ca.sh                  ─ idempotent step-ca init
│   ├── render-scep-mobileconfig.sh           ─ render SCEP profile for SimpleMDM upload
│   ├── simplemdm-m4-no-sip-bootstrap.sh      ─ ⭐ bootstrap script — now shipped as a signed PKG (managed install)
│   ├── mint-securetoken.sh                    ─ automate the SecureToken mint (+ BST escrow) over ssh
│   ├── simplemdm-vault-fetch-snippet.sh      ─ just the fetch block, for splicing
│   ├── relops-bootstrap.sh                   ─ LaunchDaemon variant for periodic refresh
│   ├── install-on-worker.sh                  ─ operator-laptop installer
│   ├── integration-test-broker.py            ─ end-to-end test via real step-ca cert
│   ├── INSTALL-on-worker.md                  ─ install guide + threat-model table
│   └── fetch-vault-mtls.swift                ─ historical URLSession variant (dead end)
│
├── docs/                         📖  rendered walkthrough (index.html + deep-dive.html)
│                                    + reprovision-architecture.html (one-page flow diagram)
│                                    + eacs.html (what EACS actually does, button-press → re-enroll)
├── .github/workflows/test.yml    ✅  CI: pytest + terraform fmt/validate
└── cloudbuild.yaml               🚀  build + push + deploy broker on commit

---

🧭 Bringing it up

One-time operator steps. After this, provisioning a worker is driven by the orchestrator/ CLI (reprovision) — see orchestrator/README.md for the fresh vs reprovision golden paths. The bootstrap ships as a signed PKG.

1️⃣ Create the GCP project + state bucket

gcloud projects create relops-bootstrap --name="RelOps Bootstrap"
gcloud config set project relops-bootstrap
gcloud billing projects link relops-bootstrap --billing-account="$BILLING_ID"

gcloud storage buckets create gs://relops-bootstrap-tfstate \
  --location=us-central1 --uniform-bucket-level-access --public-access-prevention
gcloud storage buckets update gs://relops-bootstrap-tfstate --versioning

2️⃣ Terraform apply

cd terraform
cp terraform.tfvars.example terraform.tfvars
# fill in trusted_source_cidrs (MDC1 worker network) and broker_hostname
terraform init
terraform apply

📤 Outputs you'll need next:

Output	Used for
vault_broker_url	Cloud Run URL of the broker (debug only — production traffic via LB)
step_ca_ip	static external IP of the step-ca VM
broker_lb_ip	static external IP of the LB (point DNS here)
artifact_registry_repo_url	where to docker push the broker image
secret_ids	Secret Manager containers waiting for content

3️⃣ Initialize step-ca

See scripts/README.md — scripts/bootstrap-step-ca.sh is the idempotent operator-side bootstrap. Per-role SCEP provisioners get added afterward with step ca provisioner add scep-<role> --type=SCEP ....

4️⃣ Build + deploy the broker

gcloud builds submit broker \
  --tag us-central1-docker.pkg.dev/relops-bootstrap/vault-broker/vault-broker:latest \
  --project relops-bootstrap
gcloud run deploy vault-broker \
  --image us-central1-docker.pkg.dev/relops-bootstrap/vault-broker/vault-broker:latest \
  --region us-central1 --project relops-bootstrap

(Or push to main and let Cloud Build do it via cloudbuild.yaml.)

5️⃣ Populate per-role vault secrets

For each puppet role, mirror the 1Password entry into Secret Manager:

# Team 1Password vault is named "RelOps" (matches orchestrator/config.py); adjust
# the item/field to your per-role vault.yaml entry.
op read "op://RelOps/Puppet Vault Auth - gecko-t-osx-1500-m4/notesPlain" \
  | gcloud secrets versions add vault-gecko_t_osx_1500_m4 --data-file=-

6️⃣ Upload the SCEP profile to SimpleMDM

scripts/render-scep-mobileconfig.sh produces a .mobileconfig with the step-ca root embedded + SCEP enrollment payload. Upload to SimpleMDM as a Custom Configuration Profile, assign to the device group whose hosts you want to provision via the broker.

7️⃣ Ship the bootstrap as a signed PKG

scripts/simplemdm-m4-no-sip-bootstrap.sh is packaged as a signed PKG (managed install) assigned to the device group, so it lands during DEP convergence and runs on its own — no SimpleMDM script-job, no GCP trigger. It:

1. Waits for admin to hold a SecureToken — the runner mints the first one over an on-host ssh login (DEP skips Setup Assistant, so admin has no token until a PAM login, and Apple only grants the first SecureToken interactively — a script/PKG can't). BST escrow happens alongside the mint.
2. Discovers the SCEP identity in the keychain by issuer DN
3. Fetches the role-scoped vault.yaml from the broker via SecureTransport curl
4. Clones ronin_puppet, sets up ssh-to-localhost, installs the m4-bootstrap-driver LaunchDaemon (which loops run-puppet.sh until the safari semaphores fire)

The mint + BST escrow are driven by reprovision mint / reprovision escrow-bst (the standalone scripts/mint-securetoken.sh), which the on-network runner runs automatically as part of the pipeline — no manual per-host login.

---

🔐 Security model

Layer	Mechanism	Protects against
Cloud Run ingress	INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER (live value: internal-and-cloud-load-balancing)	direct-to-broker traffic that skips the LB (spoofed cert headers). Load-bearing — the broker trusts the LB's chain verdict, so this is what makes header trust safe
HTTPS LB	Cloud Armor source-CIDR allowlist (default-deny; allow 63.245.209.101/32)	off-network attackers — blocked before TLS terminates, on both the broker and step-ca backends
HTTPS LB	mTLS via Trust Config (step-ca root + intermediate)	requests without a valid step-ca-issued cert (chain validated at the LB)
HTTPS LB	LB overwrites the X-Client-Cert-* header names it injects	forged auth headers — an inbound client cannot set X-Client-Cert-Chain-Verified: true itself for the names the LB manages
Broker app	Requires LB-set X-Client-Cert-Present + -Chain-Verified = true, else 401 (does not re-verify the chain)	unauthenticated fetches
Broker app	Role from cert SPIFFE URI must exactly match URL path role, else 403	role escalation / cross-host secret theft
Broker app	Per-cert-serial rate limit, else 429 (in-memory, per-instance — a speed bump, not a global quota)	brute-force enumeration from a single instance
Broker IAM	Per-secret binding to vault-* + step-ca-root-cert only (no project-wide grant)	lateral movement if the broker is compromised
Cloud Audit	Every Secret Manager read logged with cert serial (app log + platform audit log)	after-the-fact incident response
Cert lifecycle	24h cert lifetime (step-ca default; not explicitly pinned in ca.json)	long-lived compromised credentials
Cert private key	macOS System keychain ACL-restricted to network-stack helpers (KeyIsExtractable=false)	key exfiltration by non-OS code

Trust boundary (state this plainly in a review): the broker does no cryptography of its own beyond parsing the leaf to read the role — it trusts the LB's X-Client-Cert-Chain-Verified header. That is safe only because (a) the LB overwrites those header names, and (b) Cloud Run ingress blocks any request that did not transit the LB. Relaxing the ingress setting would turn the header trust into an auth bypass. See SECURITY.md for the command that proves each of these.

---

🌐 Live endpoints

URL	What it is
https://forge.relops.mozilla.com	Single HTTPS LB (Google-managed server cert). URL map: /scep/* (and a now-unused /acme/*) → step-ca backend; everything else (incl. /secret/{role}) → vault-broker backend. The LB requests a client cert on all paths (ALLOW_INVALID_OR_MISSING_CLIENT_CERT) so /scep/* works cert-less; /secret/* authentication is enforced in the broker, not the LB
Cloud Run direct URL	Not reachable from off-LB traffic — ingress is internal-and-cloud-load-balancing, so a direct request is refused at the platform layer (it does not reach the broker) rather than returning a 401

Vault fetch status (m4 role): proven end-to-end. The validating call from a real EACS'd m4 returns the full role-scoped vault.yaml with the broker's strict checks all green.

🛠️ Open work

• 🪪 Per-role expansion — verified live state (2026-07):
  • Live (SCEP provisioner + populated vault-* secret): gecko_t_osx_1500_m4 (scep-no-sip), gecko_t_osx_1500_m4_staging (scep-osx-1500-m4-staging), and gecko_t_osx_1500_m4_no_sip.
  • Scaffolded, not functional: the six gecko_t_linux_* roles have step-ca SCEP provisioners but their vault-* secrets are empty (0 versions) — enrollment would succeed but the fetch has nothing to return. Not usable until the secrets are populated (and the Linux operator-side flow is re-validated — see the Linux notes in docs/).
  • Not started: gecko_t_osx_1400_r8, gecko_t_osx_1015 — empty secret containers, no provisioner yet.
  • Each remaining role is mechanical: new provisioner with a template hardcoding the role in the SPIFFE URI + a "Dev - SCEP - " SimpleMDM profile + a populated Secret Manager secret. scripts/bootstrap-step-ca.sh codifies all eight live SCEP provisioners (2 macOS + 6 Linux), so the CA is reproducible from code; the abandoned acme-no-sip ACME provisioner has been pruned from the live CA.
• 🔁 Cloud Build trigger — cloudbuild.yaml exists, OAuth-flow UI got stuck during initial wire-up; switch to a PAT-based webhook trigger or retry the GitHub App connection. Manual gcloud builds submit works.
• 🔭 Widen trusted_source_cidrs beyond 63.245.209.101/32 if other Mozilla NAT IPs need access (other datacenters, VPN egress, etc.).
• ✅ Bootstrap via signed PKG — DONE (2026-07): the bootstrap is a signed PKG (managed install) that lands at DEP convergence instead of a triggered script-job.
• ✅ Hangar-driven, one-click reprovision — DONE (2026-07): a button in the hangar dashboard queues a job that a Puppet-managed on-network runner claims over mTLS and runs. reprovision runtime-fetches its SimpleMDM / TC / admin creds (Secret Manager / 1Password / host vault.yaml) — no per-session pasting.
• 🔑 Runner cert renewal — the on-network runner uses a step-ca client cert; step-ca isn't reachable from MDC1 yet, so it's refreshed manually. Expose step-ca to MDC1 (like the forge LB) to enable step ca renew --daemon auto-rotation. TC quarantine-worker token still needs periodic re-issue.

---

🤝 Sibling project

hangar — the RelOps fleet dashboard. Same GCP folder, same terraform patterns, same FastAPI stack. This repo borrows hangar's layout conventions.