feat(webapp,run-engine): queue metrics and health dashboard#4131
feat(webapp,run-engine): queue metrics and health dashboard#4131ericallam wants to merge 47 commits into
Conversation
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR adds queue-metrics ingestion, storage, query, and UI support. It introduces a Redis/ClickHouse metrics pipeline package, ClickHouse queue-metrics tables and query helpers, run-queue emission hooks, gap-filling support in TSQL, and new webapp admin, dashboard, list, and detail routes. It also adds environment and feature-flag gating, seed tooling, and tests across the pipeline and query layers. Related PRs: None found. Suggested labels: enhancement, area: webapp, area: run-engine, area: internal-packages Suggested reviewers: ericallam, matt-aitken 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
ericallam
force-pushed
the
feat/queue-metrics-and-health
branch
from
a892684 to
9412bf5
Compare
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
ericallam
force-pushed
the
feat/queue-metrics-and-health
branch
from
6432d9f to
3c67a0c
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…signals Gauges are read inside the enqueue/dequeue Lua and returned on the script reply as a 2-tuple; counters are cumulative odometers. The run-queue Redis carries no metrics stream of its own.
Adds an Allocation tab to the Queues page (behind the queue metrics UI flag): overview cards, a burst-aware capacity bar showing each queue allocation and its live usage in a distinct color, an inline-editable limits table with per-queue locks, load-weighted auto-balance, and a review dialog that bulk-applies limits as overrides through the existing concurrency system. The queue list now defaults to Busiest ordering (with Backlog and Name options). ClickHouse ranks queues by activity over the last 15 minutes and returns just the requested page of names, so the cost per page is one small aggregate regardless of environment size; idle queues follow in name order and any failure falls back to name ordering. The classic page keeps plain name order.
The fallback WHERE injection only targeted the top-level SELECT, so a query shaped as an outer aggregation over a FROM subquery failed to compile: the time column only exists inside the subquery. Descend into the subquery so the fallback lands next to the table reference.
Adds two rollups fed from the raw landing table: a per-queue 5-minute tier and an environment-level 1-minute tier (gauges plus TDigest wait quantiles). Ranking now reads the 5m tier and returns the page and the ranked total in one windowed query instead of two scans. The 5m materialized view reads raw rather than cascading off the 10s table: deltaSumTimestamp states hold a single first/last segment, so merging states in an MV's hash-ordered GROUP BY double-counts bridging spans. For the same reason the env tier carries no counter columns, and env-wide counter totals must group by queue before summing.
The built-in queues dashboard's enqueued vs started chart merged counter states across queues, which mixes unrelated cumulative counters and returns wrong totals; it now merges per queue and sums outside. Env header tiles and saturation charts read the environment rollup, so their cost no longer scales with queue count, and coarse-bucket ranges are served from the 5m rollup automatically. Queue list ranking runs as one query, time bounds are aligned to the bucket grid, and repeated auto-refresh reads share ClickHouse query-cache entries.
… rollup The env rollup's win comes from dropping the queue dimension, not from coarser buckets: row count is queue-independent (~8640/day/env), so full 10-second granularity stays cheap at any range. Env header tiles and saturation charts now resolve short-range detail exactly like the per-queue charts, and the current-value tiles read the latest 10-second bucket instead of a minute-wide one.
The simulator's --reset only cleared the raw and 10s tables, leaving stale rows in the 5m and env rollups. It also force-merges the rollups after seeding so current-value widgets read cleanly.
Counter events now emit per queue and op odometer readings with a seeded zero baseline, matching the production emitter, so throughput and started counts reconstruct from simulated data instead of reading zero. Scenario switches prune the previous scenario's queues, a --project flag seeds each scenario into its own project for side-by-side design review, and a new many-queues scenario covers pagination and relevance ranking with one runaway queue, a busy head, a bursty middle, and a sparse tail. Adds --help.
A --usage flag stages plausible running counts in the local run-queue Redis for the seeded queues, so the list's Running column and the Allocation tab's usage bars have data without the run engine. Staged state is reconciled on every run: present with --usage, cleared without. Local Redis hosts only.
The tail query's exclusion list overwrote the search's name filter via object spread, so searching while sorted by activity showed unrelated queues past the ranked head. Combine the conditions with AND instead.
…ot ready Without a readiness guard, every fire-and-forget emit during a metrics Redis outage queued a command in ioredis's in-memory offline queue until rejection. Metrics are loss-tolerant by design, so drop instead; waitUntilReady() lets embedders await the initial connect.
The allocation view keeps manual limit edits, the review dialog, and bulk apply. The one-shot auto-balance button is removed (and the row locks whose only purpose was protecting queues from it); a policy-driven approach can replace it if rebalancing returns.
deltaSumTimestamp states are kept per queue, and merging them across queues silently returns wrong totals, on the dashboard and the public query API alike. Columns can now declare a mergeGroupKey, and the compiler rejects queries that merge such a column without grouping by that key or pinning it to a single value. The error names the column, explains the failure, and includes a corrected example query.
…e calls Short parameter lists on quantilesMerge and quantilesTDigestMerge do execute (the state layout is parameter independent, verified on both ClickHouse versions we run), but they rely on undocumented leniency and make the result-array indexes mean different quantiles per call site. Every merge now uses the stored four-quantile list with indexes re-pointed accordingly; returned values are unchanged.
ericallam
force-pushed
the
feat/queue-metrics-and-health
branch
from
1416ee6 to
cbe5444
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The consumer retries a failed batch with the same insert deduplication token, and deduplicate_blocks_in_dependent_materialized_views re-runs the materialized views on a source-deduplicated insert, relying on each target table's dedup window to drop the duplicate. Only the raw table had one, so retries appended extra copies into every aggregate tier, silently inflating sums and quantiles. All three targets now set non_replicated_deduplication_window, with a regression test inserting the same batch three times.
… cluster slots, and stream caps Counter readings for names past the per-env cardinality cap are dropped instead of merging unrelated odometers under the overflow label (gauges still flow). The odometer key now shares the stream's shard hash tag so the INCR plus XADD script stays in one Cluster slot. The counter stream cap defaults lower when the stream shares the run-queue Redis. The per-bucket counter boundary undercount is documented on the delta columns.
…d live per-key breakdown CK queues now emit two extra gauge fields from the CK-path Lua scripts: the number of concurrency keys with queued runs (ZCARD of the ckIndex) and the head-of-line wait of the most-starved key (now minus the oldest ckIndex score). Both flow through the existing stream into new max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep the 7-field gauge shape. The queue detail page grows a concurrency-keys section for queues with CK activity: charts for backlogged keys and most-starved wait, plus a live per-key table (queued, running, oldest wait) read from the ckIndex zset, most starved first. Per-key history is intentionally not stored: key values are user-controlled, so the per-key dimension stays in Redis where it is bounded by the live backlog. The queue simulator gains a tenant-hotspot scenario that stages the CK gauge columns and a live ckIndex so the per-key table and charts render with data.
…trics history Queues that shard work with concurrencyKey get a per-key history tier. Counter events for CK runs advance a second per-key odometer and carry both readings on ONE stream entry (cum + ck/ckcum), so per-key attribution adds no stream volume; the consumer expands the entry into a base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by (queue, concurrency_key) holds per-key enqueue/started/ack deltas, backlog/running maxes, and wait sums. Rows are activity-bound: a (queue, key, bucket) row exists only when that key had events, so user-controlled key cardinality cannot inflate the table (benchmarked at ~19 bytes per event with reads under 100ms in the worst shapes). The per-queue tiers stay exact: their counter and wait aggregations now consume only base rows, so per-key odometers never merge under one queue_name and waits are never double counted. A per-queue key limiter (default 10k) acts as a safety valve; on overflow the per-key row is dropped while the base row keeps per-queue counts exact. Per-key odometers use a short TTL, which cumulative counters make loss-free. The queue detail page gains a top-keys-by-backlog chart, a key table merging live state (queued, running, oldest wait) with range stats (started, peak backlog, mean delay), and click-through per-key drill-down charts. The new queue_metrics_by_key table is also queryable directly; its delta columns require grouping or pinning BOTH queue and concurrency_key, enforced by the compile-time merge guard which now supports compound keys.
…il page The queue detail page splits into Overview (the existing concurrency, backlog, delay, and throttle charts) and a Concurrency keys tab that holds all per-key content. The tab only appears for queues with key activity. Inside it, the grouped per-key charts now lead (backlog by key plus a new throughput-by-key chart that makes fair-share visible), followed by the two whole-queue health charts retitled to say what they aggregate (keys with queued runs count, most-starved key wait), then the key table and drill-down.
…ery surfaces TableSchema gains a hidden flag: hidden tables stay fully queryable through the engine (the concurrency-keys tab keeps working, tenancy and the merge guard still apply) but are excluded from the query editor autocomplete, the Query page schema and examples panels, the AI query generator context, and the schema API. queue_metrics_by_key is hidden until the per-key surface is settled.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…n re-enqueue The wait metric measured dequeue time minus the original trigger time even when a run re-entered the queue after a waitpoint, checkpoint, or pending version, so the whole wait or checkpoint duration showed up as scheduling delay. Re-enqueues now anchor to the re-enqueue time while first enqueues keep the trigger or delay anchor; queue ordering is unchanged, so re-enqueued runs keep their original position. Nacked runs never left the queue stint and keep the original anchor. Also replaces the :ck: suffix regexes on user-controlled queue names with indexOf slicing (identical semantics) to remove a polynomial regex flagged by code scanning.
Behind the per-org queue metrics UI flag: the task detail page links its queue to the queue detail view and shows live queued/running counts, delay p95, peak backlog, and a queue backlog chart; the run inspector links the queue and concurrency key, and queued, delayed, and pending-version runs get a "Waiting in queue" block with an at-limit callout and per-key counts. Adds optional concurrency-key params to the run engine queue reads.
| }, | ||
| "wait-pct": { | ||
| title: "Scheduling delay p50/p95/p99 (ms)", | ||
| query: `SELECT timeBucket() AS t,\n round(quantilesTDigestMerge(0.5, 0.9, 0.95, 0.99)(wait_quantiles)[1]) AS p50,\n round(quantilesTDigestMerge(0.5, 0.9, 0.95, 0.99)(wait_quantiles)[3]) AS p95,\n round(quantilesTDigestMerge(0.5, 0.9, 0.95, 0.99)(wait_quantiles)[4]) AS p99\nFROM env_metrics\nGROUP BY t\nORDER BY t`, |
There was a problem hiding this comment.
🔴 Dashboard scheduling-delay chart uses the wrong quantile merge function, causing a ClickHouse query error
The scheduling-delay chart widget uses the TDigest variant of the merge function (quantilesTDigestMerge at apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts:715) on a table whose column stores plain quantiles state, so ClickHouse rejects the type mismatch at query time.
Impact: The "Scheduling delay p50/p95/p99" chart on the built-in Queues dashboard fails to render, showing an error instead of data.
Aggregate-state type mismatch between the schema DDL and the TRQL query
The ClickHouse DDL at internal-packages/clickhouse/schema/035_create_queue_metrics_v1.sql:68 defines wait_quantiles as AggregateFunction(quantiles(0.5, 0.9, 0.95, 0.99), UInt32) on queue_metrics_v1. The correct merge function is therefore quantilesMerge.
However, the wait-pct widget query uses quantilesTDigestMerge, which is the merge function for AggregateFunction(quantilesTDigest(...), ...) — the type used by env_metrics_v1, not queue_metrics_v1.
The queue detail page at apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.queues_.$queueParam/route.tsx:221 correctly uses quantilesMerge for the same table, confirming the dashboard widget has the wrong function.
Fix: change quantilesTDigestMerge to quantilesMerge in the wait-pct widget query.
Prompt for agents
In BuiltInDashboards.server.ts, the wait-pct widget query (around line 715) uses quantilesTDigestMerge on the queue_metrics table, but queue_metrics_v1 stores quantiles state (not quantilesTDigest). Change all three occurrences of quantilesTDigestMerge in the wait-pct widget query to quantilesMerge. The env_metrics table correctly uses quantilesTDigest, so its queries (e.g. the p95 header tile) should keep quantilesTDigestMerge.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Summary
Adds per-queue observability to the Queues page: depth (backlog), throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and the scheduling delay (how long runs wait between becoming eligible and actually starting). Each queue shows health at a glance in the list, plus a per-queue detail page with charts, so you can answer "does this queue have enough concurrency to keep up?".
Both the data collection and the dashboard are off by default and gated independently: metric emission is a global switch, and the dashboard is turned on per organization. With both off, the Queues page is unchanged.
Design
Queue operations emit two kinds of signal. Gauges (depth, running, limit, throttled) are read inside the same Redis script that performs the enqueue or dequeue, so the reading is atomic, and returned on the script's reply for the app to forward. Counters (enqueued, started, completed) are cumulative odometers, so a dropped reading self-heals: the next one restates the running total. Both land on one Redis stream on a dedicated metrics instance (falling back to the run queue's Redis when self-hosting), drain through a consumer into ClickHouse (raw, a 10-second-bucket materialized view, and a 30-day aggregate), and the dashboards read the aggregate. The run queue's own Redis carries no metrics stream.
The one change that is live the moment this deploys, independent of both flags, is the enqueue/dequeue script reply shape: those scripts now return a 2-tuple so the gauge reading can ride back to the app. That path is exercised on every queue op, so it is the part of
run-engineworth the closest review.