fix(webapp): bound logs search memory and fix pagination at scale

[ericallam]

Summary

The logs search page (behind a feature flag) ran ClickHouse out of memory when browsing back over long time ranges. This keeps it within bounded memory and fixes a pagination bug that could skip or duplicate rows at a page boundary.

Fix

Memory: the list query reads in sort-key order, which opens one read stream per part in the window, and on object storage those per-part read buffers dominate peak memory, so it scaled with the number of parts scanned. Two changes bound it:

• The logs ClickHouse client caps the per-part read buffers via new env-tunable settings. The object-storage-only setting is opt-in, so it is never sent to a ClickHouse version that lacks it.
• Recent-first window narrowing: rows come back newest first, so the presenter probes the most recent window and only widens toward the full requested range when a page is short. A busy environment fills a page from a few recent parts instead of scanning the whole range; a quiet one still returns every row in a couple of cheap reads.

Correctness: the keyset cursor ordered on (triggered_timestamp, trace_id), which is not unique because the spans of a trace share both, so rows at a tie could be skipped or duplicated across pages. The cursor and ORDER BY now include span_id, and the cursor is versioned so stale cursors reset to the first page.

Guards: the effective page size is capped, and the existing per-query memory limit lets a pathological wide browse fail with an error instead of taking the node down.

ClickHouse 26.2

The memory fix relies on lazy materialization deferring the wide attributes column to the output rows, which only holds on 26.x. Cloud already runs 26.2, so this moves the dev stack, testcontainers, and CI to match. The ClickHouse test suite passes on 26.2.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: 9f99c92

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 286046b3-9dc7-44c7-a45d-02c56bbd8901

📥 Commits

Reviewing files that changed from the base of the PR and between b49824e and 9f99c92.

📒 Files selected for processing (2)

• apps/webapp/app/env.server.ts
• apps/webapp/app/presenters/v3/LogsListPresenter.server.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• apps/webapp/app/presenters/v3/LogsListPresenter.server.ts

📜 Recent review details

⏰ Context from checks skipped due to timeout. (28)

• GitHub Check: internal / 🧪 Unit Tests: Internal (2, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (5, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (12, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (11, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (9, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (8, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (7, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (4, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (10, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (1, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (6, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (3, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 10)
• GitHub Check: packages / 🧪 Unit Tests: Packages (1, 3)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 10)
• GitHub Check: packages / 🧪 Unit Tests: Packages (3, 3)
• GitHub Check: packages / 🧪 Unit Tests: Packages (2, 3)
• GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
• GitHub Check: typecheck / typecheck
• GitHub Check: Analyze (javascript-typescript)

🧰 Additional context used

📓 Path-based instructions (7)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead

Import from @trigger.dev/sdk when writing Trigger.dev tasks. Never use @trigger.dev/sdk/v3 or deprecated client.defineJob

Files:

• apps/webapp/app/env.server.ts

{packages/core,apps/webapp}/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use zod for validation in packages/core and apps/webapp

Files:

• apps/webapp/app/env.server.ts

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use function declarations instead of default exports

**/*.{ts,tsx,js,jsx}: Prefer static imports over dynamic imports. Only use dynamic import() when circular dependencies cannot be resolved, code splitting is needed for performance, or the module must be loaded conditionally at runtime
Import subpaths only from packages/core (@trigger.dev/core), never import from the root

Files:

• apps/webapp/app/env.server.ts

**/*.ts

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

**/*.ts: When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs
Do not use high-cardinality attributes in OTEL metrics such as UUIDs/IDs (envId, userId, runId, projectId, organizationId), unbounded integers (itemCount, batchSize, retryCount), timestamps (createdAt, startTime), or free-form strings (errorMessage, taskName, queueName)
When exporting OTEL metrics via OTLP to Prometheus, be aware that the exporter automatically adds unit suffixes to metric names (e.g., 'my_duration_ms' becomes 'my_duration_ms_milliseconds', 'my_counter' becomes 'my_counter_total'). Account for these transformations when writing Grafana dashboards or Prometheus queries

Files:

• apps/webapp/app/env.server.ts

apps/webapp/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)

apps/webapp/**/*.{ts,tsx}: Access environment variables through the env export of env.server.ts instead of directly accessing process.env
Use subpath exports from @trigger.dev/core package instead of importing from the root @trigger.dev/core path

Use named constants for sentinel/placeholder values (e.g. const UNSET_VALUE = '__unset__') instead of raw string literals scattered across comparisons

Files:

• apps/webapp/app/env.server.ts

apps/webapp/**/*.server.ts

📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md)

apps/webapp/**/*.server.ts: Never use request.signal for detecting client disconnects. Use getRequestAbortSignal() from app/services/httpAsyncStorage.server.ts instead, which is wired directly to Express res.on('close') and fires reliably
Access environment variables via env export from app/env.server.ts. Never use process.env directly
Always use findFirst instead of findUnique in Prisma queries. findUnique has an implicit DataLoader that batches concurrent calls and has active bugs even in Prisma 6.x (uppercase UUIDs returning null, composite key SQL correctness issues, 5-10x worse performance). findFirst is never batched and avoids this entire class of issues

Files:

• apps/webapp/app/env.server.ts

**/*.{js,ts,tsx,jsx,css,json,md}

📄 CodeRabbit inference engine (AGENTS.md)

Use Prettier for code formatting and run pnpm run format before committing

Files:

• apps/webapp/app/env.server.ts

🧠 Learnings (12)

📚 Learning: 2026-03-22T13:26:12.060Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3244
File: apps/webapp/app/components/code/TextEditor.tsx:81-86
Timestamp: 2026-03-22T13:26:12.060Z
Learning: In the triggerdotdev/trigger.dev codebase, do not flag `navigator.clipboard.writeText(...)` calls for `missing-await`/`unhandled-promise` issues. These clipboard writes are intentionally invoked without `await` and without `catch` handlers across the project; keep that behavior consistent when reviewing TypeScript/TSX files (e.g., usages like in `apps/webapp/app/components/code/TextEditor.tsx`).

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-03-22T19:24:14.403Z

Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3187
File: apps/webapp/app/v3/services/alerts/deliverErrorGroupAlert.server.ts:200-204
Timestamp: 2026-03-22T19:24:14.403Z
Learning: In the triggerdotdev/trigger.dev codebase, webhook URLs are not expected to contain embedded credentials/secrets (e.g., fields like `ProjectAlertWebhookProperties` should only hold credential-free webhook endpoints). During code review, if you see logging or inclusion of raw webhook URLs in error messages, do not automatically treat it as a credential-leak/secrets-in-logs issue by default—first verify the URL does not contain embedded credentials (for example, no username/password in the URL, no obvious secret/token query params or fragments). If the URL is credential-free per this project’s conventions, allow the logging.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-05-18T08:21:27.694Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma error P1001 ("Can't reach database server") in TypeScript, don’t assume a single error shape. Prisma can surface P1001 via two different error classes/fields: `PrismaClientKnownRequestError` exposes it as `err.code === "P1001"` (common during mid-query connection drops), while `PrismaClientInitializationError` exposes it as `err.errorCode === "P1001"` (common on client startup failure). Therefore, predicates should use `err.code === "P1001" || err.errorCode === "P1001"`. Do not flag `err.code === "P1001"` as “unreachable/never matches,” as it is expected in production.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-05-18T08:21:27.694Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma errors for P1001 ("Can't reach database server"), do not assume it only appears under a single property name. Prisma may surface P1001 via either `PrismaClientKnownRequestError` (`err.code === "P1001"`, e.g., mid-query connection drops) or `PrismaClientInitializationError` (`err.errorCode === "P1001"`, e.g., client startup connection failure). To reliably detect the condition, check `err.code === "P1001" || err.errorCode === "P1001"`, and avoid review rules that would incorrectly flag `err.code === "P1001"` as unreachable/never-matching.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-06-13T19:53:13.759Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3937
File: packages/trigger-sdk/skills/realtime-and-frontend/SKILL.md:258-260
Timestamp: 2026-06-13T19:53:13.759Z
Learning: When reviewing code that uses `trigger.dev/react-hooks`’s `useRealtimeRun`, preserve the call signature where the first argument is the full realtime handle object (not `handle.id`). This is intentional to maintain type-safety and is consistent with the official docs; do not suggest changing the first argument from the handle object to `handle.id`.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-06-17T17:13:49.929Z

Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3948
File: apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.bulk-actions.$bulkActionParam/route.tsx:48-62
Timestamp: 2026-06-17T17:13:49.929Z
Learning: In triggerdotdev/trigger.dev, within `dashboardLoader`/`dashboardAction` (or similar context resolver code) whenever you resolve an organization ID from an organization slug for RBAC/enterprise authorization scope, always read from the primary Prisma client (`prisma`), not `$replica`. Using `$replica` can hit replica-lag and cause the RBAC lookup/authorization to run without the correct org scope (bypassing intended role enforcement). Implement the slug→org lookup with `prisma.organization.findFirst(...)` (or equivalent primary-client query) and add an inline comment documenting why the primary client is required (replica lag could lead to unscoped RBAC checks).

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-05-05T09:38:02.512Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3523
File: apps/webapp/app/routes/api.v3.batches.ts:178-181
Timestamp: 2026-05-05T09:38:02.512Z
Learning: When reviewing code that catches `ServiceValidationError` in `*.server.ts` files, do not blindly forward `error.status` to HTTP responses, because SVEs may be thrown with non-default statuses (e.g., 400/500) and forwarding them can cause client-visible behavioral regressions (e.g., surfacing 500s to clients). Prefer a safe default response status of `error.status ?? 422`, but only after confirming via the reachable call graph that the caught `ServiceValidationError` instances are expected to carry those non-default statuses; otherwise, normalize to `422` to avoid unexpected client-visible 5xx behavior.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-05-12T21:04:05.815Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3542
File: apps/webapp/app/components/sessions/v1/SessionStatus.tsx:1-3
Timestamp: 2026-05-12T21:04:05.815Z
Learning: In this Remix + TypeScript codebase, do not flag a server/client boundary violation when a file imports only types from a module matching `*.server`.

Specifically, it’s safe to import types using `import type { Foo } from "*.server"` or `import { type Foo } from "*.server"` because TypeScript erases type-only imports at compile time and they emit no JavaScript, so they won’t cross the Remix server/client bundle boundary.

Only raise the boundary concern for value imports (e.g., `import { Foo }` without `type`, or `import Foo`), since those produce JavaScript output.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-05-20T17:21:18.543Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3678
File: apps/webapp/app/entry.server.tsx:0-0
Timestamp: 2026-05-20T17:21:18.543Z
Learning: In env.server.ts (Zod env schema), any environment variable you plan to access via the typed `env` export (e.g., `env.SENTRY_DSN`) must be explicitly declared in the schema. For `SENTRY_DSN`, include `SENTRY_DSN: z.string().optional()`; otherwise switching from `process.env.SENTRY_DSN` to `env.SENTRY_DSN` will fail TypeScript typechecking.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-06-01T11:37:08.569Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3754
File: apps/webapp/app/env.server.ts:1104-1129
Timestamp: 2026-06-01T11:37:08.569Z
Learning: In apps/*/app/env.server.ts, any new background/periodic worker feature flag should hard-default to "0" (explicit opt-in) rather than inheriting from a parent flag (e.g., avoid defaulting to process.env.TRIGGER_MOLLIFIER_ENABLED ?? "0"). Inheriting can cause the new worker to auto-start on upgrade for deployments that already enabled the parent flag, turning on unexpected background load without an explicit rollout. Each worker component must require its own dedicated env var and default it explicitly to "0" (e.g., TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED defaults to "0" unless explicitly set to enable that worker).

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-06-04T18:16:35.386Z

Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.

Applied to files:

• apps/webapp/app/env.server.ts

📚 Learning: 2026-06-09T17:58:04.699Z

Learnt from: 0ski
Repo: triggerdotdev/trigger.dev PR: 3879
File: apps/webapp/app/models/vercelIntegration.server.ts:619-630
Timestamp: 2026-06-09T17:58:04.699Z
Learning: In this codebase, outbound raw `fetch` calls should typically rely on Node/undici’s default request timeout (about ~300s) rather than adding a per-call `AbortController` + `setTimeout` wrapper inside individual functions (e.g. in files like `apps/webapp/app/models/vercelIntegration.server.ts`). During code review, do not flag the absence of a per-call timeout on a single `fetch` as an issue; if per-call timeouts are needed, they should be implemented via a codebase-wide convention (e.g., a shared fetch wrapper or documented pattern) rather than ad-hoc per-function changes.

Applied to files:

• apps/webapp/app/env.server.ts

🔇 Additional comments (1)

apps/webapp/app/env.server.ts (1)

1659-1672: LGTM!

---

Walkthrough

The pull request upgrades the ClickHouse server image from 25.x to 26.2.19.43 across three CI unit-test workflows, docker-compose.yml, and the ClickHouseContainer testcontainers default. Separately, it fixes logs search memory usage and pagination: three new buffer-size environment variables are added to EnvironmentSchema along with pagination tuning parameters, and these are applied as ClickHouse query settings in getLogsListClickhouseSettings. LogsListPresenter is rewritten with a versioned cursor format (adding spanId as a keyset tiebreaker), a buildProbeFloors helper that generates stepped day windows, and a probe loop that widens the query window until a full page is returned, with keyset ordering on (triggered_timestamp, trace_id, span_id).

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and concisely captures the main change: bounding logs search memory and fixing pagination at scale, which aligns with the PR's primary objectives.
Description check	✅ Passed	The description is largely complete with detailed technical explanations of memory and correctness fixes, but is missing the required checklist completion marks and testing steps section.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/logs-search-memory-pagination

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.