perf(clickhouse): reduce ingestion overhead

[rguliyev]

Low-risk ClickHouse utilization and insert-path optimizations. Single squashed commit, 10 files.

Why

ClickHouse query log showed the top write load coming from sandbox_host_stats (~482k inserts), sandbox_events (~364k), and the OTel metrics_gauge/metrics_sum path (~156k), plus avoidable read scans on hot paths. These are the low-hanging fixes; structural work (sandbox_events ORDER BY swap, projection removal) is deferred to follow-up PRs.

OTel collector (iac/modules/job-otel-collector/configs/otel-collector.yaml)

• batch/clickhouse: flush every 15s (was 5s) with send_batch_max_size: 100000 — fewer, larger inserts into metrics_gauge/metrics_sum.
• filter/external_metrics: narrowed e2b.* → ^e2b\.(sandbox|team)\..*$. All 9 currently emitted e2b.* metrics match, so this is a no-op today; new e2b.* sub-namespaces must be added intentionally. This filter is shared by the ClickHouse, GCP-telemetry, and otel-router pipelines.

Schema migrations (codec-only MODIFY COLUMN, metadata-only, no part rewrites)

• 20260708220300: Delta+ZSTD for timestamps and monotonic cgroup counters, Gorilla+ZSTD for float metric values on team_metrics_gauge_local, team_metrics_sum_local, sandbox_metrics_gauge_local, sandbox_host_stats_local.
• 20260709120000: ZSTD(3) for webhook_deliveries_local bodies/headers (highly compressible JSON, capped at 64KB by the writer).
• Neither migration repeats column types or DEFAULT expressions (per review), so they cannot drift from the original definitions.

Write path

• New kill-switch clickhouse-batcher-async-insert (bool, default true = current behavior): when flipped to false, batched host-stats/events inserts set async_insert=0. The Go batchers already send large Native blocks, so the server-side async buffer only adds a 400ms–4s blocking wait per flush on top of client-side batching.
• Removed the persisted all-zeros baseline row per sandbox start (hoststats_collector.go): deltas are computed against the in-memory prev seed, so the row was write-only noise — one extra insert per sandbox start, and its interval_us=0 was a divide-by-zero hazard for rate queries.
• New NewDriverWithFeatureFlags; idle conn pool cap now comes from LD flag clickhouse-max-idle-conns (int, default 8; was hardcoded 3) to avoid native-protocol handshake churn on bursty batcher flushes. NewDriver keeps its signature (belt imports it) and uses the fallback.

Read path

• QueryLatestMetrics (hot dashboard path) now bounded by maxSandboxTTL (24h) instead of sweeping the table's full retention history for argMaxIf. No sandbox outlives this window, so latest values always fall inside it. Invariant: maxSandboxTTL must stay ≥ the largest tiers.max_length_hours.

Rollout notes (after merge, LD only — no deploys)

1. Flip clickhouse-batcher-async-insert=false.
2. Raise clickhouse-batcher-max-batch-size / clickhouse-batcher-max-delay (existing flags) together with queue size.
3. Watch batcher.items.dropped, ProfileEvents_InsertQuery, InsertedRows, and async-insert metrics before/after.

[gemini-code-assist]

Code Review

This pull request increases the ClickHouse batch processor timeout and batch size in the OpenTelemetry collector configuration, refines the metric filtering regex, and introduces a ClickHouse migration to optimize storage codecs for time-series columns. The reviewer suggested simplifying the migration SQL by omitting the data types and default values in the MODIFY COLUMN statements, as ClickHouse allows modifying codecs directly without repeating these definitions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...es/orchestrator/pkg/sandbox/hoststats_collector.go	0.00%	4 Missing ⚠️
packages/orchestrator/pkg/factories/run.go	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!