feat(investigations): report-first detail page and fan-out hub - #357
Merged
Conversation
An investigation is a finding, not a conversation. The Flue transcript was the page; now it is one tab behind the result. Detail page is tabbed — Overview · Evidence n · Hypotheses n · Chat · Transcript — with the header, status chips and actions persisting across all of them, and a one-line CAUSE recap strip so the detail tabs never lose context. Overview stays short and dense: verdict, impact strip, next actions, docked composer. Resolve and Retry moved to the header, beside the subject they act on. The rail drops to checks, run spine, linked records and provenance. The run is modelled as a fan-out: 1-5 diagnosing agents, each assigned a lens from a fixed catalogue, then a validator that promotes one candidate and records why each rival lost. That is the trust payload — it proves the obvious alternative was checked and says why it lost. Diagnosed runs get a "Hypotheses considered" table, live runs get parallel lens lanes, and failed runs get validation_inconclusive: lenses reported but nothing was promoted. Chat and Transcript are deliberately separate: Chat is the user's conversation, Transcript is the agents' own reasoning log, read-only. None of the fan-out is persisted yet. `V2Investigation` models a single pass — one report, one model, one token pair, no steps — so every lens, hypothesis, check and blast-radius value comes from `fanout-placeholder.ts`, a deterministic stub seeded off the investigation id (FNV-1a + LCG; no Math.random, no Date.now, because a lane that renumbered itself on every 3s poll would read as the run changing its mind). That module is the single seam: when the real records land it is the only file that dies. Everything else on the boards reads real wire fields, including several that ship today and rendered nowhere — incident window, related services, affected scope, and the error taxonomy that used to collapse into one generic toast. Two gaps worth knowing: v2 offers no stop endpoint, so a running pass shows Resolve rather than Stop; and the docked follow-up composer hands off to the Chat tab rather than owning a second session, because ChatConversation owns approvals and the failed-send queue.
It was the last child of the scroll area with `mt-auto`, which only reaches the bottom while the tab is shorter than the viewport. On any real diagnosis the content overflows, so the composer sat ~160px below the fold and scrolled away with the page — the opposite of docked, and squashed against the container edge once you did scroll to it. `Content` is a flex column, so the composer is now a `shrink-0` sibling below `Scroll` — the same shape as the sticky header above it. The tab scrolls behind it, it is on screen at every scroll position, and it keeps a 16px gutter. Drops `min-h-full` from the scrolled column with it: that was only there to give `mt-auto` something to push against, and it is the pattern that collapses tall cards inside a scroll area.
…ng policy Foundation for replacing the single diagnostic pass with N lens agents plus a validator. No behaviour changes yet: `fanout_state` defaults to `none` on every row, no lens rows exist, and every read path returns an empty `lensRuns`. - `investigation_lens_runs`: one row per dispatched lens, carrying the candidate claim, the validator's reason for its verdict, per-lens tokens and timings. The `(investigation_id, lens_id)` unique index is the retry-safety key — a replayed Cloudflare Workflow step must upsert its lane, not grow a second one. - Parent row gains `fanout_state`/`fanout_size`/validator fields, all defaulted, so the migration is online-safe and existing rows land on the single-pass path with no data pass. - `fanout-policy.ts` keeps two questions apart that the UI placeholder had conflated. `fanoutSize` is the design's sizing table (how many angles this subject deserves); `shouldFanOut` is the rollout gate (manual starts, or automatic high/critical). An automatic medium alert computes a size of 5 and must still run single-pass — collapsing them renders a Hypotheses tab over an empty array, which is why `hasFanout` has to key off dispatched lenses rather than off the size. - The validator lane is derived from the lens rows rather than stored, so the rail cannot claim a ranking the lanes contradict. - `rowToDocument` takes lens rows as a parameter; `listInvestigations` batches them with one `IN (…)` query, and skips it entirely when nothing on the page fanned out. Verified `Promise.all` over `step.do` really is concurrent before committing to the design: three 4s steps completed in 4.03s under wrangler dev, all starting within 20ms.
Extracts the multi-turn loop out of `runTriageAgent` into `tool-loop.ts`, which the fan-out needed with a different prompt, a narrower allowlist and a deadline — three parameters rather than three copies. `runTriageAgent` keeps its exact behaviour and its tests pass untouched; the final `generateObject` stays with each caller, since a triage pass answers in `AiTriageResult` and a lens answers in `LensCandidate`. Lens passes are headless: no Durable Object, no transcript, and deliberately no `submit_diagnosis` tool. A lens produces a candidate to be ranked, and handing it the tool that publishes a diagnosis would let any one of five rivals declare itself the answer before the validator ran. Two things in `lens-prompt.ts` are load-bearing and easy to undo by accident. The shared preamble comes first and is byte-identical across lenses, so five concurrent passes share a prompt-cache breakpoint — reversing that order is the largest cost regression available at a fan-out of five. And every lens is told, in those words, that reporting no finding is correct; without it the model confabulates and the Hypotheses table's premise that a ruled-out rival was genuinely tested becomes a lie. The validator has no tools at all. It adjudicates text the lenses gathered, and giving it instruments would make it a sixth lens with a casting vote. Promoting nothing is an allowed outcome, and since the schema cannot express "promotedLensId and report are null together", that invariant is enforced after decode. Model tiering: `resolveLensModel` puts lens passes on MAPLE_LENS_MODEL_* and falls back to the triage model when unset, so an unconfigured environment behaves exactly as it does today. The validator deliberately has no knob of its own — it is `resolveTriageModel`, so the ranking cannot silently drift below the lenses it ranks.
`InvestigationFanoutWorkflow` runs claim → N concurrent lens steps → validate → seed-transcript → persist, registered alongside the existing workflows in wrangler.jsonc, alchemy.run.ts and worker.ts. Three engine rules are encoded deliberately, each with a comment saying why: - The deadline is computed inside `claim`, never in the workflow body. A `Date.now()` in the body returns something different on every replay and silently invalidates every cached step downstream. - A lens step never rejects. Its body is a total try/catch and the promise carries a second `.catch`, because `Promise.all` rejects if any member does — which would lose four healthy passes to one bad one. That is the regression the test suite exists for. - Lens step names are frozen at `lens-<lensId>`. An in-flight instance replays cached steps against redeployed code, so a rename orphans the cache, re-runs the model pass and re-bills it. One Effect runtime is built per instance and shared by all five steps. Building `MainLive` is what the dynamic-import workarounds exist to avoid (CF error 10021), and building five concurrently would multiply that against a 30s per-step CPU limit. `applyDiagnosisWrites` is extracted so the workflow's `persist` step and `InvestigationService.submitDiagnosis` produce byte-identical effects — same status transition, same severity application, same deterministically-keyed timeline event. An investigation should not mean different things depending on which path produced it. Autumn metering stays with each caller, and `persist` sums tokens across every lens plus the validator: the idempotency key is the investigation id, so reporting only the validator's would under-bill by the whole fan-out. The validator reads its candidates back from Postgres rather than from the step return values, so a step whose result was lost to a retry boundary still counts — its row was written either way.
…n-out `createAndStartInvestigation` and `restartInvestigation` now branch on `fanoutPlan`. Everything the gate declines keeps the existing single Durable Object turn, so both paths coexist and an unconfigured org sees no change at all. Three things had to move with it, each a bug if left alone: - **The stale sweep now has two budgets.** A healthy five-lens run legitimately outlives the 15-minute single-pass timeout, so the old sweep would mark it `failed` moments before the workflow wrote a diagnosis onto it — leaving `diagnosed` next to a `diagnosis_timeout` error and a failure card over a real finding. In-flight fan-outs get 25 minutes. - **Quota counts passes, not runs.** `autonomous_turns` is incremented by `plan.passCount`, so a five-lens fan-out burns the six model calls it actually costs. `maxPassesPerDay` is a *new* column rather than a reinterpretation of `maxRunsPerDay`: that one is org-configurable and user-visible, and silently changing its unit would turn a configured 20 into about three critical incidents a day with no warning. - **Restart deletes the previous attempt's lanes** and claims a fresh workflow instance id. The unique index on (investigation_id, lens_id) would otherwise make every insert in the retry a no-op and leave the board rendering the old verdicts as if they belonged to the new run. `startFanout` mirrors `sendAutonomousTurn`'s failure discipline exactly, because the caller cannot tell which path it took: a missing binding writes `agent_unavailable` onto the row and fails retryably, a rejected instance becomes `start_failed`. Ships dark behind `ai_triage_settings.fanout_enabled`, default false.
…older `V2Investigation` gains `lens_runs`, `validator` and `fanout`. `lens_runs` is annotated as an evolving shape — the lens catalogue will change, and without that note five string literals are frozen into a public contract. `evidence` stays off the wire: nothing renders it and five evidence blocks per list row would multiply the trace-id decode surface for no gain. `fanout-placeholder.ts` is deleted. What was always a derivation — the tally, the run-spine segment, the checks panel — moved to `lens-derive.ts` and now derives from real rows; what was invented is gone. Lens display copy lives in `lens-catalogue.ts`, keyed by id, because a name is presentation and shipping it over the wire would make a copy edit require an API deploy. `hasFanout` now keys off dispatched lenses rather than `fanout.size`. Those are different questions: an automatic medium-severity alert computes a size of 5 and still runs single-pass, and reading the size would render a Hypotheses tab over an empty array. The blast-radius lane is removed from the impact strip. It was the one number with no possible source — events and users belong to the incident, not the investigation — and a fabricated user count inside an impact strip is the most dangerous thing that could be on the page. Verified against real rows in the browser, not just in tests: the rail reads "1 of 3 held" beside a board showing 1 promoted and 2 ruled out, each lane carrying the validator's own sentence, and the verdict card naming the lens the cause was promoted from.
`chatSessionStub` returns undefined when the binding exists but is not a chat-session namespace, so checking `env.CHAT_SESSION` for truthiness was not the guard it looked like. CI's typecheck caught it; my local run had not.
Two review agents went over the fan-out. These are the findings that made the feature not do what it claims. **The automatic half of the routing rule was never wired.** Fan-out was supposed to run for manual starts *and* high/critical automatic incidents, but every incident-open start goes through `maybeEnqueueTriage`, which did its own insert and `beginTurn` and never consulted `fanoutPlan`. `automatic: true` appeared only in the policy's own unit test, so the severity branch was dead code that passed. It now plans through the same `fanoutPlan` the manual path uses, charges `passCount` to the budget, and dispatches the workflow — and when the binding is missing it records `agent_unavailable` rather than quietly falling back to one agent, because a run planned as a fan-out that ran as a single pass is a lie in the boards. **The rail could still contradict the board, two ways.** `lensTally.reported` counted only lenses that reported a candidate, but a crashed lens becomes a terminal `no_finding` lane and the run ranks anyway — so the spine pulsed "4 of 5 reported" over a published diagnosis, forever. Split into `reported` and `settled`; progress gates on `settled`. And while the validator ran, every un-ranked lane rendered ✗ "Did not hold" under a "0 of 5 held" header — a ruling nobody had made. `pending` is now its own neutral check state. Also: `TriageStrip` had regressed to `fanout.size`, so single-pass runs reported phantom lenses in flight; and `FailedVerdict` claimed "Validator rejected every candidate" for runs the validator never reached, which on a swept-out workflow put three contradictory statements on one card. The stale-sweep budgets now live in one module. They were duplicated, and the copy in `ai-triage-enqueue` still used a flat 15 minutes — it would have failed a healthy five-lens run, which is the exact bug the two budgets exist to prevent. Tests that encoded the buggy tally were corrected rather than deleted, and the uncovered branches they hid — a `no_finding` lane on a ranked run, an un-ranked lane mid-validation — now have their own.
… step that did nothing
**Quota counted the wrong unit.** `autonomousTurns` is incremented by the pass
count, and that pass-sum was compared against `maxRunsPerDay` — silently
reinterpreting a configured 20 runs as 20 passes, i.e. about three critical
incidents a day. Runs and passes are now counted separately, and the window moved
from `createdAt` to `startedAt` so restarting an investigation opened last week
spends today's budget instead of escaping the window entirely.
**Restart could assemble one board out of two runs.** Nothing terminated the
instance being replaced, and lens rows were keyed only by
(investigation_id, lens_id) — so a straggler from attempt 0 wrote its claims and
verdicts into attempt 1's lanes and could publish a diagnosis over a live run.
Restart now terminates the prior instance (best-effort; an already-finished one
cannot be), and lanes carry an `attempt` that is part of the unique index and
every read. Prior lanes are kept rather than deleted: scoping makes them
invisible, and a straggler can now only write where nothing renders it.
**The transcript step was dead.** It called `append({ events: [...] })` with an
`assistant-message` type — `append` takes a single `ChatEventInput` and that union
has no such member. An `as never` cast got it past the compiler and a bare
`catch` swallowed the throw, so every fan-out left the Transcript tab empty and
Chat follow-ups ungrounded, silently, which is the exact failure the step exists
to prevent. It now writes a real turn sequence, is idempotent on a deterministic
message id, and the test exercises the real function against a fake session
rather than stubbing it.
That reconstruction turns telemetry-derived lens claims into *assistant* text,
which a follow-up turn reads as its own prior reasoning — so it is explicitly
marked as a machine-written summary of reported findings, not instructions.
`startFanout` no longer discards why `create()` failed: an id collision means a
live instance already owns the investigation, a network error means retry, and
both used to produce the same unlogged `start_failed`.
…ering gaps **The validator's first ranking rule read "(none given)" on every candidate.** Lens agents produce a `mechanism` — the causal chain, which the prompt asks for explicitly and the validator is told to prefer above confident tone — but there was no column, so `validate` hard-coded null. Every ranking silently degraded to claim-plus-evidence. It is now persisted and read back. **`lens_runs` promised an evolving shape it could not deliver.** The annotation told API consumers to treat `lens_id` as an open string while the schema was a closed `Schema.Literals`, so a server that learned a sixth lens would fail the decode for every deployed client — blanking the detail page and the hub — and it made the client's own unknown-lens fallback unreachable. The wire now decodes tokens openly and the UI humanises what it does not recognise, which is what makes the annotation true. The literal unions stay for everything that writes. **Two metering gaps.** The Autumn idempotency key was the bare investigation id, so a restart reused the first attempt's key and its real, different spend was deduplicated away — every retry after the first was free. The key now carries the attempt, and stays stable within one. Separately, a validator that exhausted its retries killed the instance with nothing metered or written, despite N lens passes having really run; that path now records the failure and bills what was consumed. **Migration snapshot chain repaired.** `db:generate` had overwritten `0030_snapshot.json` in place, destroying the snapshot of an unrelated earlier migration and leaving `prevId` pointing at a file that no longer existed. The original is restored from history, mine renumbered above it, and the chain is continuous again — `db:generate` now reports no pending changes, and the whole chain applies to a fresh database. The OpenAPI example was internally inconsistent (a `diagnosed` investigation with a null report, and a validator noting three rivals beside a single lens) — it is the published contract sample, so it now describes a coherent run.
Makisuo
force-pushed
the
feat/investigations-redesign
branch
from
August 6, 2026 13:05
15eea22 to
14cab55
Compare
… beside it The lens and validator passes had their own `tool-loop.ts`, written before the loop rework landed. Keeping it would have meant two answers to every question the loop already answers — retry policy, context pruning, step budgets, what a given agent may reach for — so it is deleted and the passes run on `runChatTurn`. A lens turns out to be exactly what `AgentDefinition` describes: a self-contained question, a narrow tool allowlist, its own step budget, and no ability to spawn anything further. So the five lenses and the validator are registered agents in `chat/agents.ts` rather than a private catalogue, and their tool allowlists are `PermissionRuleset`s — deny `*`, then allow by name, the same shape as `READ_ONLY_RULESET`, so a mutating tool added next month is denied rather than slipping through. The validator denies everything: it adjudicates text the lenses gathered, and instruments would make it a sixth lens with a casting vote. Structured output arrives the way `submit_diagnosis` already does — a tool supplied through `extraTools` whose `parameters` *is* the schema — rather than a separate forced `generateObject` pass. One fewer model call per lens, and one fewer mechanism doing the same job. `agent-pass.ts` is the only new seam: it drives a registered agent to a typed answer for a workflow. The deadline rides on `isCurrent`, which the loop already checks between steps to stop a superseded turn — the same semantics a lens wants, finish the step you are in and stop. Two behaviours fall out of this that the private loop had to be told: - A lens that never calls its submit tool now returns `None` rather than throwing, so "this lens found nothing" stays a result the boards render instead of an error the workflow has to catch. The step writes a `no_finding` lane. - Lens passes inherit context pruning, which they need more than the attended turn does — warehouse-sized tool payloads with nobody watching them stall. `triage-agent.ts` is restored to exactly its state on main; it keeps its own loop.
Ran the fan-out end to end against a live incident — `MmpNotFoundError` on
subscriptions-api, 12k occurrences — with real agents against real telemetry.
Three lenses dispatched concurrently and made 16, 16 and 22 tool calls; one
reported a candidate, two reached no finding, and the validator promoted nothing.
That is the `validation_inconclusive` path, end to end, for real.
**Cloudflare rejects a colon in a workflow instance id.** Restart built
`${id}:${attempt}` and every restarted fan-out failed with `start_failed`. The
unit tests could not catch it because the harness fakes `create()` — only a real
binding rejects the id. The attempt is now appended with a dash.
Worth noting the diagnosis took one log line, because the previous commit stopped
`startFanout` swallowing the reason: "WorkflowError: Workflow instance has invalid
id" named it outright.
**"Ran for" measured the row's lifetime, not the run.** It read `created_at` to
`updated_at`, so a 20-day-old investigation that had just been restarted reported
a 480-hour run. `started_at` is re-stamped on every restart, so it is the honest
start — and it is now on the wire for that reason.
**"One lenses reported."** Singular when one lens reports, which is exactly the
case this board exists to describe.
Everything else held up under a real run: the lanes moved
queued -> checking -> reported/no_finding concurrently, the rail read "0 of 3
held" beside a board saying none held up, and the spine said "All 3 reported"
rather than wedging at "1 of 3" — which is the `settled` fix working against real
`no_finding` lanes rather than a fixture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An investigation is a finding, not a conversation. The Flue transcript used to be the page; now it is one tab behind the result.
UI only — no schema, no API, no migration.
Detail page
Tabbed: Overview · Evidence n · Hypotheses n · Chat · Transcript. Header, status chips and actions persist across all of them, and the detail tabs carry a one-line
CAUSErecap strip so they never lose context. Overview stays short and dense — verdict, impact strip, next actions, docked composer — and fits above the fold. Resolve/Retry moved to the header, beside the subject they act on. The rail leads with checks, then the run spine, linked records, and a provenance footer.Chat and Transcript are deliberately separate: Chat is the user's conversation with Maple, Transcript is the agents' own reasoning log, read-only.
The run is a fan-out
1–5 diagnosing agents, each assigned a lens from a fixed catalogue (deploy correlation, downstream dependency, resource saturation, traffic shape, config & flags), then one validator that ranks the candidates, promotes one, and records a reject reason for each other. Count comes from severity × signal kind; at a fan-out of one the whole fan-out UI collapses back to a plain ledger.
Validator · blockedlane stating what unblocks it.validation_inconclusive— lenses reported, validator promoted nothing.The placeholder seam
None of the fan-out is persisted.
V2Investigationmodels a single pass: onereport, onemodel, one token pair, no steps. So every lens, hypothesis, check and blast-radius number comes fromfanout-placeholder.ts— a deterministic stub seeded off the investigation id (FNV-1a + LCG; noMath.random, noDate.now, because a lane that renumbered itself on every 3s poll would read as the run changing its mind).That module is the single file that dies when the real records land. Nothing else under
components/investigations/invents data. Everything else reads real wire fields — including several that ship today and rendered nowhere:snapshot.incident_started_at/ended_at,evidence[].relatedServices,report.affectedScope, and theautomation_disabled | agent_unavailable | start_failed | quota | diagnosis_timeouttaxonomy that used to collapse into one generic toast.Reviewed in the browser
Ran the full local stack against real rows (3 diagnosed, 24 failed, plus seeded live runs at fan-out 1 and 5) and walked every board. Four defects found and fixed in the process:
3 of 5 heldwith three green ticks ~300px from "none of them held up". It also always emitted 5 rows regardless of fan-out size, and the two catalogues were in different orders. Checks are now derived from the lens lanes and keyed by lens id, so the rail and the board cannot disagree — structurally, not by copy.Client agent config ✓ changed inside the windownext to an unrelated real diagnosis.not checked — lens ran out of budgetwas hardcoded as the queued copy, so a three-minute-old live run displayed a lens that had already given up.investigatingempty state rendered aStatusMarker— a full-width, left-aligned transcript row — into a slot that vertically centres its child, stranding a progress line mid-pane. It is now anEmptyNoticelike its three sibling states, with abusyflag that keeps the spinner and shimmer.ELAPSEDalso read7.04s:formatDurationgives seconds two decimals, which is right for span durations and false precision for a counter that re-renders every 3s poll. Rounded locally; the shared helper is untouched.Known gaps
ChatConversationowns the session, approvals and failed-send queue, and liftinguseMapleChatout to share one would mean rebuilding all three.Verification
bun typecheckclean; 96 tests pass acrosscomponents/investigationsandcomponents/chat, including a named regression test for the rail/board contradiction.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.