diff --git a/docs/adr/0014-behavioral-telemetry-ebpf.md b/docs/adr/0014-behavioral-telemetry-ebpf.md index 22f1cb3..5a99b0f 100644 --- a/docs/adr/0014-behavioral-telemetry-ebpf.md +++ b/docs/adr/0014-behavioral-telemetry-ebpf.md @@ -251,3 +251,33 @@ shadow-by-default framing.** The first-party eBPF agent is now the **sole deployed** corroboration source. The port it feeds remains open to any sensor; only the Falco-specific adapter and the cancelled-bake measurement are gone. + +## Amendment (2026-08-08): baked offsets are the decided posture; a load-time BTF preflight replaces manual re-verification + +The agent's eBPF bindings hand-lay kernel struct offsets (verified at kernel 7.0.0) rather +than relocating them via CO-RE. **The original CO-RE language was aspirational: rustc cannot +emit BTF field relocations** (`preserve_access_index` is a pending Rust RFC; bpf-linker emits +func_info BTF for fentry attach but not field-access relocations), so the agent bakes +hand-verified offsets. The `offset_of!` guard catches an inconsistent *edit* but not a +correct-looking layout that is *wrong for a new kernel* — and a stale +`bpf_probe_read_kernel` offset reads **garbage silently**, feeding wrong corroboration +signals into the action bar (only the `bpf_d_path` reads are verifier-checked). + +**The durable guard is a userspace load-time preflight.** Before attach, the loader verifies +every baked field offset and BTF-visible enum value (`LOADING_MODULE`) against the node's +**live BTF** (the same BTF it already reads for fentry attach). On mismatch it attaches only +the struct-free probes (connect/ptrace/module-load), logs each divergent field +(expected-vs-actual — the data needed to update the bindings), and reports degraded via the +existing heartbeat — never crash-loops (degrade-gracefully). A kernel bump can move a field +but can never silently ship wrong reads. Offsets live in **one place** — a const table in +`agent/common`: the eBPF crate's `offset_of!` asserts `bindings == table` at compile time; +the loader asserts `table == node-BTF` at load; transitively `bindings == kernel`. No +`PROTECTOR_*` toggle (a correctness guard, not a feature). + +**Build-time binding generation was considered and rejected** — it bakes the *builder* +kernel's offsets, which diverge from node kernels exactly during a rolling upgrade (the event +this defends against), and cannot run off-fleet. **Real CO-RE is the recorded end state**: +when the Rust toolchain can emit BTF field relocations, the preflight retires with the baked +offsets. Any comments claiming the object is "CO-RE-relocated against node BTF at load" are +corrected to reflect this — baked offsets + a load-time BTF check, not relocation. See +`docs/ideas/ebpf-offset-self-verification.md`. diff --git a/docs/adr/0040-node-scoped-containment-mechanism-escalation.md b/docs/adr/0040-node-scoped-containment-mechanism-escalation.md index 9c4a5bb..04034fb 100644 --- a/docs/adr/0040-node-scoped-containment-mechanism-escalation.md +++ b/docs/adr/0040-node-scoped-containment-mechanism-escalation.md @@ -209,3 +209,28 @@ the control-plane/worker split on the deployed cluster rather than weakening the - [ADR-0038](0038-transitive-internet-exposure-l7-routes.md) / [ADR-0039](0039-downstream-cve-residual-no-llm-reads-code.md) — the exposure-boundary work that strengthens the off-path won't-build. + +## Addendum (2026-08-08): apply-side scope confinement; no in-product approve-to-apply + +§5's "human approval" is pinned down: **protector never grows an in-product approve→apply +path for node containment.** Every operator-reachable surface is read-only by decision +([ADR-0016](0016-severity-vs-urgency.md): presentation is a view, never a decision gate; +[ADR-0031](0031-read-only-mcp-server-tiered-redaction.md): the MCP is read-only), and an +approval control would be exactly the view-becomes-gate shape those ADRs refuse. The durable +act stays where §4 put it — a human, out of band (cordon + runbook). The actuator's `apply` +glue remains in-tree, tested in isolation; any future in-product flow requires its own ADR. +This retires "the future apply call-site" as a design location. + +Consequently the `enforceScope` confinement of the co-resident deny sweep lives at the +**actuation boundary**, not a hypothetical apply flow: the shared full-set sweep stays +unfiltered (the revert path must lift *every* deny protector could ever have placed — +filtering revert would orphan out-of-scope denies), while the **apply-facing subset — the +only set `apply` accepts, by type (a `ScopedDenies` newtype)** — and the proposal-side +eligibility check (`contain_node_in_scope`) both derive from the same +[ADR-0021](0021-two-setting-operating-posture.md) scope match the webhook uses. Unscoped = +full set; scoped = only in-scope co-residents are denied, and the containment is honestly +**weaker**: out-of-scope pods on a contained node stay reachable, because a deny written into +an unauthorized namespace is the enforce-everywhere escape ADR-0021 exists to prevent +(filter, don't refuse-whole). The model-visible fixed proposal note is unchanged — the model +decides *what*; `enforceScope` governs *where-authorized*. See +`docs/ideas/containnode-scope-confinement.md`. diff --git a/docs/ideas/containnode-scope-confinement.md b/docs/ideas/containnode-scope-confinement.md new file mode 100644 index 0000000..da962b5 --- /dev/null +++ b/docs/ideas/containnode-scope-confinement.md @@ -0,0 +1,74 @@ +# Idea brief — Close the ContainNode enforceScope escape at the actuation boundary + +**Status:** settled → ticketed (unblocks the deferred co-resident-deny scope-filter). Load-bearing decision recorded as an +addendum to [ADR-0040](../adr/0040-node-scoped-containment-mechanism-escalation.md). + +## The idea (as posed) +Unblock the deferred scope-filter for node containment: `co_resident_denies` +(`engine/src/engine/respond/actuator/node_containment.rs`) returns one default-deny per +labelled pod co-resident on a contained node, **unfiltered by `enforceScope`**. Applying it +unfiltered would write default-deny `NetworkPolicy`s into namespaces the operator never +authorized — the enforce-everywhere escape [ADR-0021](../adr/0021-two-setting-operating-posture.md) +exists to prevent. The fix was deferred to "the future apply call-site," and the #324 review +ruled the shared helper must **not** be filtered (the revert path uses the full set; filtering +it would orphan out-of-scope denies). + +## Two findings that reframe the unblock +1. **The "apply call-site" will never exist on the current trajectory.** Protector has *no* + approve→apply mechanism for *any* action class: the dashboard is a view that never gates + ([ADR-0016](../adr/0016-severity-vs-urgency.md)), the MCP is read-only by ADR + ([ADR-0031](../adr/0031-read-only-mcp-server-tiered-redaction.md)), and `Decision::Propose` + terminates in a log line. "Human approval" has always meant *the human acts out-of-band + (kubectl)*. `NodeContainmentActuator::apply` is documented as having no `Engine` call site. + Waiting for that call site is waiting forever. +2. **The deny set is not rendered anywhere** (grepped every surface). The `ContainNode` + proposal an operator reads is the menu line + a *fixed-string* note (no names, no counts); + the pre-arm scope preview *excludes* `ContainNode` (it requires `is_additive_live()`); + nothing enumerates the co-resident denies. So there is no "misrepresenting render" to fix + — the set's only non-test consumers are `contain_node_in_scope` (eligibility, already + scope-aware), the revert seam (full set, must stay), and the caller-less `apply`. + +## Recommended approach — scoped-subset function + newtype at the actuator seam +Add `co_resident_denies_in_scope(graph, host, scope)` beside the unfiltered helper: unscoped → +full set (historical meaning); scoped → retain denies where `scope.in_scope(deny)` holds (the +same ADR-0021 namespace-OR-label match the webhook uses). Wrap its result in a `ScopedDenies` +newtype constructible only by that function; change `NodeContainmentActuator::apply` to accept +`&ScopedDenies` (revert keeps `&[Mitigation]`, full set). Rebase `contain_node_in_scope` on the +same function so eligibility and the apply subset are **one scope-match source**, exercised +every pass — the filter is live code, not dead insurance. This makes the invariant structural +("safe by construction, not caller discipline" — mirroring the revert seam's own ownership +gate): the only door into `apply` demands a scope-confined set *by type*, the revert path is +untouched (per #324), and eligibility stops being a parallel scope implementation. + +Approaches rejected: **enumerate + filter the deny set on operator surfaces** (filters a view +that doesn't exist and doesn't close the actuation escape — ADR-0016 says the view is never the +gate); **build the approve-to-apply flow** (a mutating authenticated surface on a +deliberately-read-only product — contradicts ADR-0016/0031; needs its own ADR if ever wanted). + +## Key decisions (see the [ADR-0040](../adr/0040-node-scoped-containment-mechanism-escalation.md) addendum) +1. **No in-product approve→apply path for node containment** — the human act stays out-of-band + (kubectl cordon + runbook). This retires "the future apply call-site" as a design location. +2. **Scoped apply is honest partial containment** — out-of-scope pods on a contained node stay + reachable, because writing into an unauthorized namespace *is* the ADR-0021 escape. Filter, + don't refuse-whole (consistent with the actuator's "partial containment is safer than none"). +3. **Filter home:** a sibling pure function + newtype at the actuator seam; the shared full-set + helper is untouched (revert semantics preserved). +4. **`contain_node_in_scope` derives from the same subset function** — one scope-match source. +5. **The model-visible fixed note is NOT amended** — encoding deployment scope into it would + churn the prompt fingerprint for information the model has no business weighing (the model + decides *what*, `enforceScope` governs *where-authorized*). + +## Risks / open items +- Full-set revert vs filtered apply asymmetry is **safe**: revert deletes only protector-named + objects, so lifting a never-placed deny is a harmless no-op. Invariant: the revert set ⊇ any + set apply could have placed under any historical scope. Holds by construction. +- Scope changed between passes: the subset is computed fresh at each call site from current + scope — no stored stale set. +- Genuinely open: none. Pure-function change with exhaustive unit coverage. + +## HANDOFF TO PLAN-SPRINT +Theme: **"close the ContainNode enforceScope escape at the actuation boundary + record the +no-in-product-approve decision."** One small pure-function engine PR: `co_resident_denies_in_scope` ++ `ScopedDenies` newtype + `apply` takes the scoped set by type + rebase `contain_node_in_scope` ++ the ADR-0040 addendum. No chart/RBAC/config/dashboard surface. The deferred scope-filter ticket closes against it. +Panel: infra. diff --git a/docs/ideas/ebpf-offset-self-verification.md b/docs/ideas/ebpf-offset-self-verification.md new file mode 100644 index 0000000..437fa68 --- /dev/null +++ b/docs/ideas/ebpf-offset-self-verification.md @@ -0,0 +1,89 @@ +# Idea brief — eBPF offset self-verification (load-time BTF preflight) + +**Status:** settled → ticketed. Load-bearing decision recorded as an amendment to +[ADR-0014](../adr/0014-behavioral-telemetry-ebpf.md). + +## The idea (as posed) +Eliminate the agent's **manual eBPF struct-offset maintenance**. The eBPF crate +(`agent/protector-agent-ebpf/src/vmlinux.rs`) hand-lays kernel structs so its field reads +sit at verified kernel-7.0.0 byte offsets, with an `offset_of!` const block as a guard. The +ticket proposed either **real CO-RE** (`preserve_access_index` BTF field relocation) or +**build-time binding generation** from node BTF. + +## The problem, sharpened +The `offset_of!` guard catches an *inconsistent edit* but not a *correct-looking layout that +is wrong for a new kernel*. And the failure modes are worse than "a probe fails to load": +only the `bpf_d_path(&file->f_path)` reads are verifier-checked against kernel BTF (they fail +as `loaded=N<6`); every `bpf_probe_read_kernel` chase (`i_nlink`, `s_magic`, `uid.val`, +`i_ino`, `d_name.name`, `linux_binprm.file`) at a stale offset **silently reads garbage** — +feeding *wrong corroboration signals* (anon-inode exec, tmpfs secret-read, priv-change) into +the action bar. The 6.8→6.11 `struct file` reorg already bit once (`loaded=4/6` fleet-wide), +and two offsets + one enum value are still marked ON-NODE-PENDING in the tree. The fleet is +homogeneous kernel 7.0.0 today, so the risk is **latent** — it bites on the next kernel bump +or a new arch. + +## Assumptions corrected +- **"aya gives us CO-RE" — false.** rustc cannot emit BTF field relocations for Rust struct + accesses; the `preserve_access_index` intrinsics are a pending Rust RFC (rust-lang/rfcs#3966). + bpf-linker emits BTF func_info (which is why fentry attach works) but **not** field-access + relocations. So CO-RE is blocked *upstream*, not a choice we can make today. +- **"Build-time codegen removes the maintenance" — shaky.** It moves the offset bake from a + human to the *builder node's* kernel and silently assumes builder-kernel == every + node-kernel — which breaks precisely during a rolling kernel upgrade, the exact event this + work defends against. It also can't run off-fleet (dev/macOS, generic CI), so a checked-in + fallback survives anyway. It automates the ritual without closing the gap. +- **Dangerous doc-rot found:** `agent/Dockerfile`, `.github/workflows/agent.yml`, and + `docs/ebpf-testing-on-nodes.md` claim the object is "CO-RE-relocated against the node's BTF + at load." That is **factually wrong** — `vmlinux.rs` is the only honest doc. The false + claim invites exactly the complacency this work exists to prevent; correct it. + +## Recommended approach — load-time BTF preflight (keep hand bindings; re-prove them at load) +Keep the curated hand bindings, but before attach the userspace loader verifies each +`(struct, field, expected-offset)` and the `LOADING_MODULE` enum value against the node's +**live BTF** (the loader already reads `aya::Btf::from_sys_fs()` for fentry attach). On +mismatch: log expected-vs-actual per field (the log *is* the regeneration data), attach only +the struct-free probes (connect/ptrace/module-load), and surface degraded via the existing +heartbeat/`probes_loaded`. This is the only option that checks offsets against **the kernel +the probes actually run on**, per-node — so even a heterogeneous mid-upgrade fleet degrades +honestly node-by-node. It converts "silent garbage reads / mystery `loaded=4/6` → human SSH +session" into "the agent log names the exact field, its expected offset, and the kernel's +actual offset." It is also the continuous on-node verification the two PENDING offsets need. + +Approaches rejected: **CO-RE** (blocked upstream — recorded as the *successor* end state for +when the Rust toolchain lands field relocations); **build-time codegen** (bakes the builder +kernel's offsets — automates the wrong thing). + +## Key decisions (see the [ADR-0014](../adr/0014-behavioral-telemetry-ebpf.md) amendment) +1. **Single source of truth:** a const offset+enum table in `agent/common` (shared `no_std`). + The eBPF crate's `offset_of!` consts assert `bindings == table` at compile time; the loader + asserts `table == node-BTF` at load time; transitively `bindings == kernel`. No number lives + in two places. +2. **Fail-closed on struct-reading probes, fail-open on scalar-only probes.** A mismatched + field disables the probes that read structs; connect/ptrace/module-load still attach. The + agent never crash-loops on mismatch — it stays up to report (ADR-0014 degrade-gracefully). +3. **Include the enum check** (`LOADING_MODULE`) — the one constant that is a silent + misclassification today with no verifier backstop. +4. **No toggle** (correctness guard, not a feature — repo convention). +5. **CO-RE is the recorded successor**, revisited when rustc emits BTF field relocations; the + preflight then retires with the baked offsets. **Build-time codegen is rejected.** +6. **Correct the stale "CO-RE-relocated at load" comments** in the same change. + +## Risks / open items +- aya 0.13 `Btf` API surface for member iteration — worst case a small direct parse of + `/sys/kernel/btf/vmlinux` (aya-obj is already in the tree). Bounded; settled in the first + hour of building. +- Anonymous-union members (`i_nlink`): the BTF walk must recurse into anon unions/structs + accumulating offsets — unit-tested against a fixture BTF blob (plain userspace Rust, + off-fleet testable). +- The preflight validates *offsets*, not *semantics* (a field whose meaning changes at the + same offset slips through) — accepted; nothing short of CO-RE + kernel review fixes that. +- Agent-workspace CI is not a required check (known blind spot) — the guard is runtime so it + doesn't depend on CI; flag making the ebpf job required as adjacent. + +## HANDOFF TO PLAN-SPRINT +Theme: **"eBPF offset self-verification — load-time BTF preflight."** One infra PR: the shared +offset/enum table in `agent/common` + the loader preflight (BTF walk, anon-union recursion, +fixture-tested, per-class gating, expected/actual logging, heartbeat surfacing) + resolve the +ON-NODE-PENDING markers + the ADR-0014 amendment + correct the stale CO-RE claims. Panel: +infra (agent workspace / BTF parsing / DaemonSet observability); no product-surface or +data-model change.