Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ permissions:

jobs:
migrations-lint:
# Guardrail (JEF-592): migration 0017 shipped a plain (non-CONCURRENTLY)
# Guardrail: migration 0017 shipped a plain (non-CONCURRENTLY)
# CREATE INDEX on a 10.4M-row hot table with nothing flagging it as risky.
# This fails the build on that pattern (and a couple of other
# table-locking-DDL shapes) in any newly-added migration; see
Expand All @@ -37,7 +37,7 @@ jobs:
# Secret via envFrom, so none of it belongs in this workflow and no credential is
# committed here.
#
# The redis backend is deliberately NOT set here any more (JEF-564): sccache selects
# The redis backend is deliberately NOT set here any more: sccache selects
# ONE backend from its env, so setting both would make which one you get an
# implementation detail of sccache's precedence -- the R2 cutover could read as done
# while every build still went to redis. If the pod env is missing, sccache degrades
Expand Down Expand Up @@ -98,10 +98,10 @@ jobs:
cache: npm
cache-dependency-path: ui/package-lock.json
- run: npm ci
# Accessibility floor (JEF-431): eslint-plugin-jsx-a11y gates keyboard-
# Accessibility floor: eslint-plugin-jsx-a11y gates keyboard-
# operable rows, labeled charts, and no silent-live tables from regressing.
- run: npm run lint
# Runtime a11y route-smoke (JEF-442) runs inside `npm test`: each top-level
# Runtime a11y route-smoke runs inside `npm test`: each top-level
# route is mounted (jsdom, API mocked) and scanned by axe-core, failing on
# serious/critical structural violations. Complements the static lint above —
# it catches ARIA that only resolves against the rendered tree.
Expand Down Expand Up @@ -183,8 +183,8 @@ jobs:
# cargo-chef the dep-compile layer cache-hits here until Cargo.lock moves,
# so a cold node skips it. mode=max also caches intermediate (build-stage)
# layers, which is what carries the cook layer — at the cost of a larger
# WAN transfer (see JEF-87 / measure in JEF-82).
# sccache's R2 backend for the in-image cargo build (JEF-584, ADR-0020).
# WAN transfer (measured in practice against the alternative).
# sccache's R2 backend for the in-image cargo build (ADR-0020).
# `secret-envs` (key=envname) reads these straight out of the RUNNER POD's
# env, where the `sccache-r2` Secret is injected via envFrom (cluster repo:
# charts/actions/runners/values-watcher.yaml) — the repo has no Actions secret
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ digest and rolls it. ARM images are required for Raspberry Pi nodes (CI builds t

Traces, logs, metrics (latest-value table **and** time-series charts), OTLP HTTP +
gRPC, service map, retention (global default + optional per-table
spans/logs/metrics windows, JEF-434), metric downsampling/rollups, threshold
spans/logs/metrics windows), metric downsampling/rollups, threshold
alerting (rules + events + email/webhook). Alert rules are **declarative**: a JSON
config (rendered from the chart's `server.alerts` values, env
`WATCHER_ALERTS_CONFIG`) is reconciled into the DB on startup and the
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0007-retention-by-deletion.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ For v0, a background task prunes rows older than `WATCHER_RETENTION_DAYS`
continuous aggregates, which slots in under the same schema ([0001](0001-postgres-only-no-clickhouse.md)).
- Deletes are coarse (whole-row, by age); no per-service policy yet. Spans, logs,
and metric rollups can each be given their own window via
`WATCHER_RETENTION_{SPANS,LOGS,METRICS}_DAYS` (JEF-434, an omitted override
`WATCHER_RETENTION_{SPANS,LOGS,METRICS}_DAYS` (an omitted override
falls back to `WATCHER_RETENTION_DAYS`) — per-service is still out of scope,
since a per-service delete over these tables would need `ctid`-batching like
the raw-metrics prune to avoid the statement-timeout failure mode.
4 changes: 2 additions & 2 deletions docs/adr/0013-auth-at-the-edge.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- Status: Accepted
- Date: 2026-05-30
- Amended: 2026-07-21 (JEF-473 — add origin-side Access JWT verification)
- Amended: 2026-07-21 (add origin-side Access JWT verification)
- Supersedes: [0008](0008-optional-bearer-auth.md)

## Context
Expand Down Expand Up @@ -39,7 +39,7 @@ the chart's token Secret are all gone.
- For environments without Cloudflare, equivalent edge auth (Traefik forwardAuth +
an SSO proxy) is the substitute; the app deliberately holds no auth of its own.

## Amendment — origin-side Access JWT verification (JEF-473, 2026-07-21)
## Amendment — origin-side Access JWT verification (2026-07-21)

Edge-only auth trusts that every request to `/api` and the UI arrived through Access.
An Access-policy slip, a tunnel/ingress misconfig, or direct in-cluster access to the
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0014-self-monitoring-in-process-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
## Context

watcher had no visibility into its own health. A silently stalled retention sweep let
the `metrics` table grow to tens of GB un-paged (JEF-425). We want watcher's own
the `metrics` table grow to tens of GB un-paged. We want watcher's own
operational signals (ingest throughput, drop counts, per-table on-disk bytes,
retention recency, rollup lag, pool utilisation) to be visible and alertable — using
the machinery watcher already has, on a Raspberry Pi, in a single binary.
Expand Down
4 changes: 2 additions & 2 deletions docs/adr/0016-self-log-instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@
## Context

watcher self-instruments its own **traces** (OTLP `SpanExporter` + the
`tracing-opentelemetry` layer) and its own **metrics** (JEF-425 / ADR 0014:
`tracing-opentelemetry` layer) and its own **metrics** (ADR 0014:
`selfmon` hands ops gauges/counters straight to `otlp::store_metrics`, tagged
`service.name=watcher`). But its own **logs** only went to stdout via the `fmt`
layer — there was no `tracing`→logs bridge, so watcher's log lines never landed in
its own `logs` table. You could open a watcher self-trace but couldn't jump to the
correlated self-logs (the span→logs drill, JEF-429).
correlated self-logs (the span→logs drill).

Two shapes were possible, mirroring the ADR 0014 metrics decision:

Expand Down
4 changes: 2 additions & 2 deletions docs/adr/0017-self-trace-instrumentation-in-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ watcher self-instruments its own **metrics** (ADR 0014) and **logs** (ADR 0016)
original network path: an `opentelemetry-otlp` batch `SpanExporter` POSTing OTLP over
HTTP back to watcher's own `:4318/v1/traces`.

That path was **dead** (JEF-462). The batch processor had wedged into a shut-down
That path was **dead**. The batch processor had wedged into a shut-down
state and logged "Spans are being emitted even after Shutdown ... Spans will not be
exported" on every span. Evidence from prod: **0** watcher spans in the `spans` table
ever, while self-metrics and self-logs (in-process) worked; and ~74.8k of the ~75.1k
self-log rows were *that one warning* — the broken exporter flooding the `logs` table
via the JEF-452 self-log capture. Consequences: watcher never appeared in the Services
via the self-log capture. Consequences: watcher never appeared in the Services
pulldown (`/api/services` reads `spans`), no self-traces to correlate, and ~75k junk
rows.

Expand Down
9 changes: 4 additions & 5 deletions docs/adr/0018-read-only-mcp-server-in-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@ watcher already exposes its telemetry through a same-origin query API (`/api/...
consumed by the embedded UI. LLM agents (Claude Code, MCP Inspector, …) increasingly
speak the **Model Context Protocol** (MCP): given an MCP endpoint they can search
traces, read logs/metrics, and inspect services/alerts as tools. We want watcher to
be that endpoint without standing up a second process or duplicating query logic
(JEF-471).
be that endpoint without standing up a second process or duplicating query logic.

Two shapes were possible:

Expand Down Expand Up @@ -50,8 +49,8 @@ mature.
- **Opt-in, default OFF, unauthenticated for now.** `/mcp` mounts only when
`WATCHER_MCP_ENABLED` is truthy. It is deliberately mounted *outside* the edge
auth that fronts the UI/`/api` (Cloudflare Access, ADR 0013): an MCP client is not a
browser and carries no Access cookie. Its own auth is a **separate** ticket
(JEF-472); until that lands the endpoint must not be exposed, so it defaults off and
browser and carries no Access cookie. Its own auth is a **separate** effort
(see ADR 0019); until that lands the endpoint must not be exposed, so it defaults off and
the flag's doc comment says so. The transport's default loopback-only Host allow-list
(a DNS-rebinding guard aimed at locally-run servers reached by a browser) is disabled
here, since watcher's MCP is a server-to-server endpoint reached through a public
Expand All @@ -65,7 +64,7 @@ mature.
- Query behavior stays identical across the HTTP and MCP surfaces because they share
the `query_*` functions; a future change to a clamp or window applies to both.
- The endpoint is inert until an operator sets `WATCHER_MCP_ENABLED` **and** (once
JEF-472 lands) configures its auth. Enabling it before then exposes read access to
ADR 0019's auth lands) configures its auth. Enabling it before then exposes read access to
anyone who can reach the host — the flag default and the ADR make that ordering
explicit, mirroring the "create the Access app first" runbook rule of ADR 0013.
- `rmcp` (and, for tests, its client + `reqwest` 0.13) enters the dependency tree; the
Expand Down
12 changes: 6 additions & 6 deletions docs/adr/0019-mcp-auth-cloudflare-access-oidc.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
- Status: Accepted
- Date: 2026-07-22
- Related: [0013](0013-auth-at-the-edge.md) (edge auth + origin verify), [0018](0018-read-only-mcp-server-in-process.md) (the read-only MCP server)
- Revises: the initial JEF-472 design recorded in this ADR (raw-`Bearer` JWT validation + self-served OAuth metadata), superseded by JEF-493.
- Revises: the initial design recorded in this ADR (raw-`Bearer` JWT validation + self-served OAuth metadata), superseded by the spike finding below.

## Context

Expand All @@ -20,19 +20,19 @@ the client obtains a token from an authorization server and presents it; the res
server validates it. The open question was *who is the authorization server* and *what
does the origin actually receive*.

**Initial design (JEF-472, now revised).** The first cut had watcher itself act as the
**Initial design (now revised).** The first cut had watcher itself act as the
OAuth-aware resource server: it validated the raw `Authorization: Bearer <token>` as a
Cloudflare Access **OIDC** JWT and *self-served* the RFC 9728 protected-resource
metadata (`/.well-known/oauth-protected-resource`) pointing clients at the Access OIDC
authorization server.

**Spike finding (JEF-493).** The mechanism Cloudflare actually provides for this is
**Spike finding.** The mechanism Cloudflare actually provides for this is
Access **Managed OAuth**: Cloudflare is the OAuth authorization server the client needs
(including the dynamic client registration — DCR — that claude.ai's connector performs),
it issues the client an **opaque** access token, resolves that token at its **edge**,
and forwards the origin the standard **`Cf-Access-Jwt-Assertion`** JWT — the *same*
header, issuer, and team JWKS that `/api` already validates (JEF-473). Under this model
the JEF-472 design is wrong in two ways: the origin would receive an *opaque* token in
header, issuer, and team JWKS that `/api` already validates (ADR 0013). Under this model
the initial design above is wrong in two ways: the origin would receive an *opaque* token in
`Authorization: Bearer` (not a JWT — it would fail JWT validation), and OAuth
discovery/metadata is owned by Cloudflare, not the origin.

Expand All @@ -43,7 +43,7 @@ discovery/metadata is owned by Cloudflare, not the origin.
edge sets after resolving the client's opaque Managed-OAuth token — via the shared
[`access_jwt::Verifier`](../../server/src/access_jwt.rs) (RS256 via the team's JWKS,
`iss` = team domain, `aud`, and expiry). This is the **same** assertion model as
JEF-473's `/api` `access_guard`; the header-extraction + verify step is factored into
ADR 0013's `/api` `access_guard`; the header-extraction + verify step is factored into
one shared `check_access_assertion` helper both guards call. The origin only ever
**validates**, never mints (the [0013](0013-auth-at-the-edge.md) invariant), and never
parses the opaque OAuth token.
Expand Down
6 changes: 3 additions & 3 deletions docs/adr/0020-on-ingest-per-series-metric-rollups.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ writes the raw point — there is no background sweep and no `rollup.rs`.
via `ON CONFLICT (name, series_key, bucket) DO UPDATE`, accumulating
`count`/`sum`/`min`/`max`/`avg` (and, for histograms, `bucket_bounds` /
`bucket_counts`, summed element-wise by the `array_sum` aggregate from
`array_add`). The whole batch write goes through `write_with_failover_retry`
(JEF-496), so a Patroni failover retries the statement on a fresh connection.
`array_add`). The whole batch write goes through `write_with_failover_retry`,
so a Patroni failover retries the statement on a fresh connection.
- **Series identity is preserved, not collapsed.** `series_key` is
`metric_series_key(service, attrs)` — `md5(coalesce(service,'') || '|' ||
attrs::text)` (`0009_metric_sql_helpers.sql`) — a stable hash of the service plus
Expand Down Expand Up @@ -80,7 +80,7 @@ current, still-filling bucket.
highest-volume table (see `retention.rs`'s batched delete, added after it once
grew to tens of GB unpruned).
- The extra aggregation work (`GROUP BY` + upsert) now happens inline on every
ingest batch rather than off-peak; batching (JEF-495) and the fixed lock order
ingest batch rather than off-peak; batching and the fixed lock order
keep it from becoming an ingest bottleneck, but it does mean ingest latency and
rollup-write cost are coupled.
- Still the same trend-off as [0011](0011-metric-rollups.md): history beyond the
Expand Down
4 changes: 2 additions & 2 deletions docs/adr/0021-online-ddl-lane.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ advisory lock for the whole run. That makes it the wrong place for heavy or
non-transactional DDL, and we hit **both** failure modes on `metric_series_rollups`
(~10.4M rows / 5.8 GB in production):

- **JEF-548 (first attempt):** `CREATE INDEX CONCURRENTLY` cannot run inside a
- **First attempt:** `CREATE INDEX CONCURRENTLY` cannot run inside a
transaction and deadlocked against the migrator's advisory lock.
- **JEF-580 (second attempt):** a plain, transactional `CREATE INDEX` took a `SHARE`
- **Second attempt:** a plain, transactional `CREATE INDEX` took a `SHARE`
lock on the table and ran past the pool's 60 s `statement_timeout`, so every boot's
migration was cancelled → CrashLoopBackOff, and each attempt re-locked the table and
stalled cluster-wide OTLP ingest for ~25 minutes until the index was built by hand
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ last rollup bucket, so it stays continuous after pruning.
| `WATCHER_RETENTION_METRICS_DAYS` | — | per-table override of `WATCHER_RETENTION_DAYS` for `metric_series_rollups`; unset falls back to the default |
| `WATCHER_METRICS_RAW_DAYS` | `2` | prune age for raw metric points (rollups keep history); `0` = same as retention |
| `WATCHER_ROLLUP_BUCKET_SECS` | `300` | downsample bucket width; `0` disables rollups |
| `WATCHER_MAX_QUERY_HOURS` | `168` (7d), clamped to `[1, 8760]` | max look-back for `/api/traces`, `/api/services`, and `/api/logs`; an explicit `from` (or its absence) is clamped to this ceiling, not honored verbatim (JEF-532, JEF-546) |
| `WATCHER_MAX_QUERY_HOURS` | `168` (7d), clamped to `[1, 8760]` | max look-back for `/api/traces`, `/api/services`, and `/api/logs`; an explicit `from` (or its absence) is clamped to this ceiling, not honored verbatim |
| `WATCHER_ALERT_INTERVAL_SECS` | `30` | how often alert rules are evaluated (min 5) |
| `WATCHER_ALERT_WEBHOOK` | — | optional URL to POST on alert fire/resolve |
| `WATCHER_ALERT_SMTP_HOST` | — | SMTP relay host; setting it enables emailing alert fire/resolve (STARTTLS) |
Expand Down
2 changes: 1 addition & 1 deletion scripts/lint-migrations.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/sh
# CI guardrail (JEF-592): fail the build on migration DDL that can stall the big
# CI guardrail: fail the build on migration DDL that can stall the big
# telemetry tables (spans, logs, metric_series_rollups) at deploy time.
#
# Migration 0017 shipped a plain (non-CONCURRENTLY) `CREATE INDEX` on
Expand Down
6 changes: 3 additions & 3 deletions scripts/start-sccache-docker.sh
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
#!/bin/sh
# Fail-soft sccache backend selection INSIDE the Rust image builds (JEF-584).
# Fail-soft sccache backend selection INSIDE the Rust image builds.
# The in-build twin of .github/scripts/start-sccache.sh — same shape, different
# transport for the config: the CI script reads the runner pod's env directly,
# this one reads BuildKit build secrets mounted at /run/secrets by the RUN that
# calls it.
#
# Adopted verbatim (bar the overridable LOCAL_CACHE_DIR below) from the murmurify
# repo's scripts/start-sccache-docker.sh, which landed this pattern first in
# JEF-589 — same reasoning, same fixtures. Keep them in step when either changes.
# repo's scripts/start-sccache-docker.sh, which landed this pattern first
# same reasoning, same fixtures. Keep them in step when either changes.
#
# Usage:
# sh scripts/start-sccache-docker.sh # start a server, never fail
Expand Down
4 changes: 2 additions & 2 deletions server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ RUN VITE_API_BASE="" npm run build
# chef's `cook` leaves the shared target dir in a state where those extern `.rmeta`
# files don't survive into the final `cargo build`, so sccache fatals ("Failed to
# open file for hashing: …/lib*.rmeta: No such file or directory") and aborts
# (JEF-389, confirmed real and backend-independent). Collapsed to one build stage:
# (confirmed real and backend-independent). Collapsed to one build stage:
# COPY source, one `cargo build`. Migrations are embedded via sqlx::migrate! and
# the UI via rust-embed, so the runtime image needs only the binary + CA certs.
FROM rust:1-bookworm AS build
Expand All @@ -42,7 +42,7 @@ COPY server/migrations ./migrations
COPY server/src ./src
COPY --from=ui /ui/dist /ui/dist
# sccache backend = the shared Cloudflare R2 bucket (cluster repo: charts/sccache,
# ADR-0020, JEF-584), replacing the in-cluster Redis this used to hardcode. The config
# ADR-0020), replacing the in-cluster Redis this used to hardcode. The config
# and the bucket-scoped token arrive as BuildKit build SECRETS — never ENV or a
# build-arg, both of which persist in `docker history` on every image we push to ghcr.
#
Expand Down
4 changes: 2 additions & 2 deletions server/migrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ one of them briefly blocks writes to that table — fine for a genuinely small o
brand-new table, a real stall risk once one of these three has millions of rows
(see `0017_metric_series_rollups_covering_idx.sql`, which hit exactly this).
Online/heavy DDL against `spans`, `logs`, or `metric_series_rollups` belongs in
the JEF-580 online-DDL lane (ADR 0021) — run out-of-band from `sqlx::migrate!`'s
the online-DDL lane (ADR 0021) — run out-of-band from `sqlx::migrate!`'s
advisory lock, not a boot migration.

## `-- no-transaction` migrations: one statement per file
Expand All @@ -26,7 +26,7 @@ transaction (needed for `CREATE INDEX CONCURRENTLY`, which cannot run inside
one). It only supports **one statement per no-transaction file** — split a
multi-step no-transaction change across several numbered migrations instead.

## CI lint: `scripts/lint-migrations.sh` (JEF-592)
## CI lint: `scripts/lint-migrations.sh`

CI runs `scripts/lint-migrations.sh` against every file in this directory and
fails the build on:
Expand Down
Loading