Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/adr/0014-behavioral-telemetry-ebpf.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,3 +251,33 @@ shadow-by-default framing.**
The first-party eBPF agent is now the **sole deployed** corroboration source. The port it
feeds remains open to any sensor; only the Falco-specific adapter and the cancelled-bake
measurement are gone.

## Amendment (2026-08-08): baked offsets are the decided posture; a load-time BTF preflight replaces manual re-verification

The agent's eBPF bindings hand-lay kernel struct offsets (verified at kernel 7.0.0) rather
than relocating them via CO-RE. **The original CO-RE language was aspirational: rustc cannot
emit BTF field relocations** (`preserve_access_index` is a pending Rust RFC; bpf-linker emits
func_info BTF for fentry attach but not field-access relocations), so the agent bakes
hand-verified offsets. The `offset_of!` guard catches an inconsistent *edit* but not a
correct-looking layout that is *wrong for a new kernel* — and a stale
`bpf_probe_read_kernel` offset reads **garbage silently**, feeding wrong corroboration
signals into the action bar (only the `bpf_d_path` reads are verifier-checked).

**The durable guard is a userspace load-time preflight.** Before attach, the loader verifies
every baked field offset and BTF-visible enum value (`LOADING_MODULE`) against the node's
**live BTF** (the same BTF it already reads for fentry attach). On mismatch it attaches only
the struct-free probes (connect/ptrace/module-load), logs each divergent field
(expected-vs-actual — the data needed to update the bindings), and reports degraded via the
existing heartbeat — never crash-loops (degrade-gracefully). A kernel bump can move a field
but can never silently ship wrong reads. Offsets live in **one place** — a const table in
`agent/common`: the eBPF crate's `offset_of!` asserts `bindings == table` at compile time;
the loader asserts `table == node-BTF` at load; transitively `bindings == kernel`. No
`PROTECTOR_*` toggle (a correctness guard, not a feature).

**Build-time binding generation was considered and rejected** — it bakes the *builder*
kernel's offsets, which diverge from node kernels exactly during a rolling upgrade (the event
this defends against), and cannot run off-fleet. **Real CO-RE is the recorded end state**:
when the Rust toolchain can emit BTF field relocations, the preflight retires with the baked
offsets. Any comments claiming the object is "CO-RE-relocated against node BTF at load" are
corrected to reflect this — baked offsets + a load-time BTF check, not relocation. See
`docs/ideas/ebpf-offset-self-verification.md`.
25 changes: 25 additions & 0 deletions docs/adr/0040-node-scoped-containment-mechanism-escalation.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,3 +209,28 @@ the control-plane/worker split on the deployed cluster rather than weakening the
- [ADR-0038](0038-transitive-internet-exposure-l7-routes.md) /
[ADR-0039](0039-downstream-cve-residual-no-llm-reads-code.md) — the exposure-boundary work
that strengthens the off-path won't-build.

## Addendum (2026-08-08): apply-side scope confinement; no in-product approve-to-apply

§5's "human approval" is pinned down: **protector never grows an in-product approve→apply
path for node containment.** Every operator-reachable surface is read-only by decision
([ADR-0016](0016-severity-vs-urgency.md): presentation is a view, never a decision gate;
[ADR-0031](0031-read-only-mcp-server-tiered-redaction.md): the MCP is read-only), and an
approval control would be exactly the view-becomes-gate shape those ADRs refuse. The durable
act stays where §4 put it — a human, out of band (cordon + runbook). The actuator's `apply`
glue remains in-tree, tested in isolation; any future in-product flow requires its own ADR.
This retires "the future apply call-site" as a design location.

Consequently the `enforceScope` confinement of the co-resident deny sweep lives at the
**actuation boundary**, not a hypothetical apply flow: the shared full-set sweep stays
unfiltered (the revert path must lift *every* deny protector could ever have placed —
filtering revert would orphan out-of-scope denies), while the **apply-facing subset — the
only set `apply` accepts, by type (a `ScopedDenies` newtype)** — and the proposal-side
eligibility check (`contain_node_in_scope`) both derive from the same
[ADR-0021](0021-two-setting-operating-posture.md) scope match the webhook uses. Unscoped =
full set; scoped = only in-scope co-residents are denied, and the containment is honestly
**weaker**: out-of-scope pods on a contained node stay reachable, because a deny written into
an unauthorized namespace is the enforce-everywhere escape ADR-0021 exists to prevent
(filter, don't refuse-whole). The model-visible fixed proposal note is unchanged — the model
decides *what*; `enforceScope` governs *where-authorized*. See
`docs/ideas/containnode-scope-confinement.md`.
74 changes: 74 additions & 0 deletions docs/ideas/containnode-scope-confinement.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Idea brief — Close the ContainNode enforceScope escape at the actuation boundary

**Status:** settled → ticketed (unblocks the deferred co-resident-deny scope-filter). Load-bearing decision recorded as an
addendum to [ADR-0040](../adr/0040-node-scoped-containment-mechanism-escalation.md).

## The idea (as posed)
Unblock the deferred scope-filter for node containment: `co_resident_denies`
(`engine/src/engine/respond/actuator/node_containment.rs`) returns one default-deny per
labelled pod co-resident on a contained node, **unfiltered by `enforceScope`**. Applying it
unfiltered would write default-deny `NetworkPolicy`s into namespaces the operator never
authorized — the enforce-everywhere escape [ADR-0021](../adr/0021-two-setting-operating-posture.md)
exists to prevent. The fix was deferred to "the future apply call-site," and the #324 review
ruled the shared helper must **not** be filtered (the revert path uses the full set; filtering
it would orphan out-of-scope denies).

## Two findings that reframe the unblock
1. **The "apply call-site" will never exist on the current trajectory.** Protector has *no*
approve→apply mechanism for *any* action class: the dashboard is a view that never gates
([ADR-0016](../adr/0016-severity-vs-urgency.md)), the MCP is read-only by ADR
([ADR-0031](../adr/0031-read-only-mcp-server-tiered-redaction.md)), and `Decision::Propose`
terminates in a log line. "Human approval" has always meant *the human acts out-of-band
(kubectl)*. `NodeContainmentActuator::apply` is documented as having no `Engine` call site.
Waiting for that call site is waiting forever.
2. **The deny set is not rendered anywhere** (grepped every surface). The `ContainNode`
proposal an operator reads is the menu line + a *fixed-string* note (no names, no counts);
the pre-arm scope preview *excludes* `ContainNode` (it requires `is_additive_live()`);
nothing enumerates the co-resident denies. So there is no "misrepresenting render" to fix
— the set's only non-test consumers are `contain_node_in_scope` (eligibility, already
scope-aware), the revert seam (full set, must stay), and the caller-less `apply`.

## Recommended approach — scoped-subset function + newtype at the actuator seam
Add `co_resident_denies_in_scope(graph, host, scope)` beside the unfiltered helper: unscoped →
full set (historical meaning); scoped → retain denies where `scope.in_scope(deny)` holds (the
same ADR-0021 namespace-OR-label match the webhook uses). Wrap its result in a `ScopedDenies`
newtype constructible only by that function; change `NodeContainmentActuator::apply` to accept
`&ScopedDenies` (revert keeps `&[Mitigation]`, full set). Rebase `contain_node_in_scope` on the
same function so eligibility and the apply subset are **one scope-match source**, exercised
every pass — the filter is live code, not dead insurance. This makes the invariant structural
("safe by construction, not caller discipline" — mirroring the revert seam's own ownership
gate): the only door into `apply` demands a scope-confined set *by type*, the revert path is
untouched (per #324), and eligibility stops being a parallel scope implementation.

Approaches rejected: **enumerate + filter the deny set on operator surfaces** (filters a view
that doesn't exist and doesn't close the actuation escape — ADR-0016 says the view is never the
gate); **build the approve-to-apply flow** (a mutating authenticated surface on a
deliberately-read-only product — contradicts ADR-0016/0031; needs its own ADR if ever wanted).

## Key decisions (see the [ADR-0040](../adr/0040-node-scoped-containment-mechanism-escalation.md) addendum)
1. **No in-product approve→apply path for node containment** — the human act stays out-of-band
(kubectl cordon + runbook). This retires "the future apply call-site" as a design location.
2. **Scoped apply is honest partial containment** — out-of-scope pods on a contained node stay
reachable, because writing into an unauthorized namespace *is* the ADR-0021 escape. Filter,
don't refuse-whole (consistent with the actuator's "partial containment is safer than none").
3. **Filter home:** a sibling pure function + newtype at the actuator seam; the shared full-set
helper is untouched (revert semantics preserved).
4. **`contain_node_in_scope` derives from the same subset function** — one scope-match source.
5. **The model-visible fixed note is NOT amended** — encoding deployment scope into it would
churn the prompt fingerprint for information the model has no business weighing (the model
decides *what*, `enforceScope` governs *where-authorized*).

## Risks / open items
- Full-set revert vs filtered apply asymmetry is **safe**: revert deletes only protector-named
objects, so lifting a never-placed deny is a harmless no-op. Invariant: the revert set ⊇ any
set apply could have placed under any historical scope. Holds by construction.
- Scope changed between passes: the subset is computed fresh at each call site from current
scope — no stored stale set.
- Genuinely open: none. Pure-function change with exhaustive unit coverage.

## HANDOFF TO PLAN-SPRINT
Theme: **"close the ContainNode enforceScope escape at the actuation boundary + record the
no-in-product-approve decision."** One small pure-function engine PR: `co_resident_denies_in_scope`
+ `ScopedDenies` newtype + `apply` takes the scoped set by type + rebase `contain_node_in_scope`
+ the ADR-0040 addendum. No chart/RBAC/config/dashboard surface. The deferred scope-filter ticket closes against it.
Panel: infra.
89 changes: 89 additions & 0 deletions docs/ideas/ebpf-offset-self-verification.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Idea brief — eBPF offset self-verification (load-time BTF preflight)

**Status:** settled → ticketed. Load-bearing decision recorded as an amendment to
[ADR-0014](../adr/0014-behavioral-telemetry-ebpf.md).

## The idea (as posed)
Eliminate the agent's **manual eBPF struct-offset maintenance**. The eBPF crate
(`agent/protector-agent-ebpf/src/vmlinux.rs`) hand-lays kernel structs so its field reads
sit at verified kernel-7.0.0 byte offsets, with an `offset_of!` const block as a guard. The
ticket proposed either **real CO-RE** (`preserve_access_index` BTF field relocation) or
**build-time binding generation** from node BTF.

## The problem, sharpened
The `offset_of!` guard catches an *inconsistent edit* but not a *correct-looking layout that
is wrong for a new kernel*. And the failure modes are worse than "a probe fails to load":
only the `bpf_d_path(&file->f_path)` reads are verifier-checked against kernel BTF (they fail
as `loaded=N<6`); every `bpf_probe_read_kernel` chase (`i_nlink`, `s_magic`, `uid.val`,
`i_ino`, `d_name.name`, `linux_binprm.file`) at a stale offset **silently reads garbage** —
feeding *wrong corroboration signals* (anon-inode exec, tmpfs secret-read, priv-change) into
the action bar. The 6.8→6.11 `struct file` reorg already bit once (`loaded=4/6` fleet-wide),
and two offsets + one enum value are still marked ON-NODE-PENDING in the tree. The fleet is
homogeneous kernel 7.0.0 today, so the risk is **latent** — it bites on the next kernel bump
or a new arch.

## Assumptions corrected
- **"aya gives us CO-RE" — false.** rustc cannot emit BTF field relocations for Rust struct
accesses; the `preserve_access_index` intrinsics are a pending Rust RFC (rust-lang/rfcs#3966).
bpf-linker emits BTF func_info (which is why fentry attach works) but **not** field-access
relocations. So CO-RE is blocked *upstream*, not a choice we can make today.
- **"Build-time codegen removes the maintenance" — shaky.** It moves the offset bake from a
human to the *builder node's* kernel and silently assumes builder-kernel == every
node-kernel — which breaks precisely during a rolling kernel upgrade, the exact event this
work defends against. It also can't run off-fleet (dev/macOS, generic CI), so a checked-in
fallback survives anyway. It automates the ritual without closing the gap.
- **Dangerous doc-rot found:** `agent/Dockerfile`, `.github/workflows/agent.yml`, and
`docs/ebpf-testing-on-nodes.md` claim the object is "CO-RE-relocated against the node's BTF
at load." That is **factually wrong** — `vmlinux.rs` is the only honest doc. The false
claim invites exactly the complacency this work exists to prevent; correct it.

## Recommended approach — load-time BTF preflight (keep hand bindings; re-prove them at load)
Keep the curated hand bindings, but before attach the userspace loader verifies each
`(struct, field, expected-offset)` and the `LOADING_MODULE` enum value against the node's
**live BTF** (the loader already reads `aya::Btf::from_sys_fs()` for fentry attach). On
mismatch: log expected-vs-actual per field (the log *is* the regeneration data), attach only
the struct-free probes (connect/ptrace/module-load), and surface degraded via the existing
heartbeat/`probes_loaded`. This is the only option that checks offsets against **the kernel
the probes actually run on**, per-node — so even a heterogeneous mid-upgrade fleet degrades
honestly node-by-node. It converts "silent garbage reads / mystery `loaded=4/6` → human SSH
session" into "the agent log names the exact field, its expected offset, and the kernel's
actual offset." It is also the continuous on-node verification the two PENDING offsets need.

Approaches rejected: **CO-RE** (blocked upstream — recorded as the *successor* end state for
when the Rust toolchain lands field relocations); **build-time codegen** (bakes the builder
kernel's offsets — automates the wrong thing).

## Key decisions (see the [ADR-0014](../adr/0014-behavioral-telemetry-ebpf.md) amendment)
1. **Single source of truth:** a const offset+enum table in `agent/common` (shared `no_std`).
The eBPF crate's `offset_of!` consts assert `bindings == table` at compile time; the loader
asserts `table == node-BTF` at load time; transitively `bindings == kernel`. No number lives
in two places.
2. **Fail-closed on struct-reading probes, fail-open on scalar-only probes.** A mismatched
field disables the probes that read structs; connect/ptrace/module-load still attach. The
agent never crash-loops on mismatch — it stays up to report (ADR-0014 degrade-gracefully).
3. **Include the enum check** (`LOADING_MODULE`) — the one constant that is a silent
misclassification today with no verifier backstop.
4. **No toggle** (correctness guard, not a feature — repo convention).
5. **CO-RE is the recorded successor**, revisited when rustc emits BTF field relocations; the
preflight then retires with the baked offsets. **Build-time codegen is rejected.**
6. **Correct the stale "CO-RE-relocated at load" comments** in the same change.

## Risks / open items
- aya 0.13 `Btf` API surface for member iteration — worst case a small direct parse of
`/sys/kernel/btf/vmlinux` (aya-obj is already in the tree). Bounded; settled in the first
hour of building.
- Anonymous-union members (`i_nlink`): the BTF walk must recurse into anon unions/structs
accumulating offsets — unit-tested against a fixture BTF blob (plain userspace Rust,
off-fleet testable).
- The preflight validates *offsets*, not *semantics* (a field whose meaning changes at the
same offset slips through) — accepted; nothing short of CO-RE + kernel review fixes that.
- Agent-workspace CI is not a required check (known blind spot) — the guard is runtime so it
doesn't depend on CI; flag making the ebpf job required as adjacent.

## HANDOFF TO PLAN-SPRINT
Theme: **"eBPF offset self-verification — load-time BTF preflight."** One infra PR: the shared
offset/enum table in `agent/common` + the loader preflight (BTF walk, anon-union recursion,
fixture-tested, per-class gating, expected/actual logging, heartbeat surfacing) + resolve the
ON-NODE-PENDING markers + the ADR-0014 amendment + correct the stale CO-RE claims. Panel:
infra (agent workspace / BTF parsing / DaemonSet observability); no product-surface or
data-model change.