Skip to content

fix(scouts): batch emissions + report lookups in the scout signals panel#2996

Merged
andrewm4894 merged 2 commits into
mainfrom
fix/scouts-batch-emissions
Jun 30, 2026
Merged

fix(scouts): batch emissions + report lookups in the scout signals panel#2996
andrewm4894 merged 2 commits into
mainfrom
fix/scouts-batch-emissions

Conversation

@andrewm4894

Copy link
Copy Markdown
Member

Problem

Opening the Signals section in a scout's detail view was slow. ScoutSignalsSection rendered one RunEmissions per visible run, and each one fired two per-run requests — runs/<id>/emissions/ and runs/<id>/emissions/reports/. With N visible emitted runs that's 2×N requests and N ClickHouse scans on open (each emissions/reports request runs its own unmaterialized JSONExtract scan over document_embeddings).

This is the desktop twin of the web fix in PostHog/posthog#66823, which replaced the same fan-out in the web app's findingsLogic.

Changes

Call the two batched endpoints added by #66823 once for the whole visible-run window:

  • runs/emissions/batch — every visible run's emitted findings in one Postgres query, each row keeps its run_id.
  • runs/emissions/reports/batch — resolves every run's findings to their inbox report in a single ClickHouse round-trip.

Specifically:

  • api-client: new scoutPost helper; replaced listScoutRunEmissions / listScoutEmissionReports with batchScoutRunEmissions / batchScoutEmissionReports (POST a run_ids list, short-circuit on empty, handle array or paginated body).
  • hooks: useScoutRunEmissions / useScoutEmissionReports now take a run-id list, key the query on the sorted ids, and use keepPreviousData so "Show more" doesn't blank the already-rendered cards while the widened window refetches.
  • ScoutSignalsSection: fetches once at the section level, groups emissions by run_id locally; RunEmissions is now presentational (no hooks).

Net effect: 2×N requests / N CH scans → 2 requests / 1 CH scan on panel open.

The desktop window is small (INITIAL_EMITTED_RUNS = 10, "Show more" expands to the full emitted-runs window), well under the backend's 200-id batch cap, so no client-side chunking is needed.

⚠️ Blocked on backend

The runs/emissions/batch and runs/emissions/reports/batch endpoints are added by PostHog/posthog#66823, which is not yet merged. This PR must not ship until #66823 is merged and deployed to US + EU — otherwise the Signals panel will 404 on open. Keeping this as a draft until then.

How tested

Agent (Claude Code), human-directed. Automated only — no manual UI testing.

  • pnpm --filter @posthog/api-client --filter @posthog/ui typecheck clean.
  • Added 4 api-client tests (batched scout emissions): empty short-circuit, request shape + run_ids body, paginated unwrap, non-OK error. Full posthog-client.test.ts passes (34 tests).
  • biome clean on all touched files.

The scout detail Signals section fanned out one emissions request and one
report-link request per visible run (2xN requests, N ClickHouse scans).
Mirror PostHog/posthog#66823: call the new batched endpoints
(runs/emissions/batch, runs/emissions/reports/batch) once for the whole
visible-run window — 2 requests, 1 ClickHouse scan.

- api-client: add scoutPost helper; replace listScoutRunEmissions /
  listScoutEmissionReports with batchScoutRunEmissions /
  batchScoutEmissionReports (run_ids body, empty-list short-circuit).
- hooks: key the batched queries on the sorted run-id window, keep prior
  data while a widened window refetches.
- ScoutSignalsSection: fetch once at the section level, group emissions by
  run_id locally, make RunEmissions presentational.
- tests: cover the empty short-circuit, request shape, paginated unwrap,
  and non-OK error for the batched client methods.
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

React Doctor found no issues in the changed files. 🎉

Reviewed by React Doctor for commit aaab3c6.

@andrewm4894 andrewm4894 marked this pull request as ready for review June 29, 2026 17:59
@andrewm4894 andrewm4894 self-assigned this Jun 29, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d0bb67d35

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/ui/src/features/scouts/hooks/useScoutRunEmissions.ts
@greptile-apps

greptile-apps Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff (1)

  1. packages/ui/src/features/scouts/components/ScoutSignalsSection.tsx, line 99-108 (link)

    P1 Newly-revealed runs show "no signals" while fetching

    When "Show more" widens visibleRuns, keepPreviousData keeps isLoading=false and surfaces the previous (narrower) batch as data. Any run that wasn't in that previous batch returns undefined from emissionsByRunId.get(run.run_id), so RunEmissions reaches the !emissions || emissions.length === 0 branch and displays "No signal details available for this run." — but the run has emitted_count > 0, meaning the user sees incorrect content until the wider batch resolves.

    The fix is to also expose isFetching from the hook and compute a per-run loading flag: emissionsLoading || (emissionsFetching && !emissionsByRunId.has(run.run_id)). That keeps existing cards visible while giving new runs a skeleton instead of a false empty state.

Reviews (1): Last reviewed commit: "fix(scouts): batch emissions + report lo..." | Re-trigger Greptile

Comment thread packages/api-client/src/posthog-client.test.ts Outdated
Comment thread packages/ui/src/features/scouts/components/ScoutSignalsSection.tsx
Address PR review feedback:

- api-client: chunk batchScoutRunEmissions / batchScoutEmissionReports at
  200 ids (parallel POSTs, concatenated) so a window with >200 emitted runs
  no longer fails the whole Signals panel.
- tests: parameterise the shared empty/non-OK/request-shape cases over both
  batch methods, and add a >200-id chunking test.
- ScoutSignalsSection: memoize emittedRuns / visibleRuns so the visibleRunIds
  memo can actually skip recomputation.

Generated-By: PostHog Code
Task-Id: 1dacbd86-8ba2-4213-924a-0ff8495a2f7b
@andrewm4894 andrewm4894 added the Stamphog This will request an autostamp by stamphog on small changes label Jun 29, 2026
@andrewm4894 andrewm4894 enabled auto-merge (squash) June 29, 2026 22:04

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is behaviorally sound and well-tested, but it depends on new batch backend endpoints that can't be verified from the client-side code, and it has no human reviewer sign-off despite being a complex API contract change affecting production UI.

@stamphog stamphog Bot removed the Stamphog This will request an autostamp by stamphog on small changes label Jun 29, 2026
@andrewm4894 andrewm4894 requested a review from a team June 29, 2026 22:11
@andrewm4894 andrewm4894 merged commit d6192f6 into main Jun 30, 2026
26 checks passed
@andrewm4894 andrewm4894 deleted the fix/scouts-batch-emissions branch June 30, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants