OTLP browser telemetry export: follow-up improvements (KERNEL-1596)

[archandatta]

tldr

Follow-up improvements to the in-VM OTLP browser-telemetry exporter (KERNEL-1596), post #308. Independent, reviewable commits. Off-by-default export is preserved; nothing changes for existing sessions unless the new toggle/knobs are set.

What changed

1. Externalize batch knobs — BTEL_OTLP_MAX_QUEUE_SIZE (2048), BTEL_OTLP_EXPORT_INTERVAL (1s), BTEL_OTLP_EXPORT_TIMEOUT (30s). Defaults match the OTel SDK's, so behavior is unchanged unless set; validated as > 0.
2. Per-session export toggle — new export.otlp.enabled on the telemetry API. Export is off by default and gated on the toggle, independent of whether an export destination is configured, so an always-injected relay does not generate work that gets dropped. The writer moves under a runtime-toggleable controller; toggling is best-effort (never fails a telemetry apply).
3. Drop / failure / exported metrics — queue-overflow drops (previously log-only) plus export failures and exported counts as Prometheus counters (kernel_otlp_records_dropped_total, kernel_otlp_export_failures_total, kernel_otlp_records_exported_total), owned above the writer so they stay monotonic across export toggles.
4. Crash/OOM severity — service_crashed and system_oom_kill now map to ERROR in the OTLP converter (were INFO), so consumers can alert on them.
5. Renderer-crash event — detect Inspector.targetCrashed (an "Aw, Snap!" renderer crash that leaves the browser process alive, so no service_crashed fires) and emit a page_crashed event, mapped to ERROR. Categorized as page (produced by the CDP monitor, captured only while the page category is enabled).

Scope notes

• Two of the six ticket items are intentionally not code here: the concurrency-scaled resilience benchmark (measurement, depends on the knobs + drop metric landing first) and the event-reshape-for-relay-routing item (parked pending the durability decision).
• Based on main (which already has Add in-VM OTLP telemetry export sink #308 + the metrics endpoint), keeping main's existing export auth wiring; the x-api-key swap is tracked in Send instance name as x-api-key to the OTLP relay #316.

Generated code (oapi.go)

The oapi.go diff looks large but decomposes as: ~727 lines are the embedded-spec gzip blob (the whole spec is embedded, so any spec change rewrites it — same as #308), ~105 lines are the genuinely new generated types (page_crashed + export config + union accessors), and 4 lines are an oapi-codegen naming side-effect (adding a second standalone page-category event renames the existing BrowserPageTabOpenedEventCategoryPage const to Page). That constant isn't referenced anywhere in-repo, and regenerating main's own spec with the pinned toolchain reproduces main's oapi.go exactly (0 diff), so this is faithful make oapi-generate output.

Testing

• go build ./..., go vet ./..., and the full unit suite (22 packages) pass.
• Unit tests: export toggle (on/off, default-off, no-endpoint no-op, clear-stops-export), config defaults/validation, drop-counting handler, a real-SDK drop test that forces a batch-queue overflow against a local hung listener (so CI catches an SDK drop-message rename), metrics collector, converter severities, and the Inspector.targetCrashed handler.

Live test results

Verified full-stack against the headless image + a real dockerized OpenTelemetry collector, driving the real telemetry API and a real Chromium over CDP:

• Batch knobs: custom BTEL_OTLP_* values surfaced in the boot config (otlp_max_queue_size=512 otlp_export_interval=500ms otlp_export_timeout=30s); export shows available, not started, at boot.
• Export toggle: GET /telemetry → 404 at boot; PUT with export.otlp.enabled:true → 201 echoing it. With export off but page/network capture on, exported_total froze (10 → 10) and the collector received nothing new; re-enabling resumed export (10 → 28), confirming the controller rebuilds the one-shot writer.
• Renderer crash: a real chrome://crash produced a page_crashed record at the collector with category=page, url=chrome://crash/, and SeverityText: ERROR.
• Converter fidelity: network_response carried promoted attributes (http.request.method, url.full, http.response.status_code=200); screenshot/monitor categories never reached the collector.
• Metrics: /metrics served kernel_otlp_records_dropped_total 0, kernel_otlp_export_failures_total 0, kernel_otlp_records_exported_total 28.

🤖 Generated with Claude Code