From fc73fc103c1aa072d03afc840532769d41b16696 Mon Sep 17 00:00:00 2001 From: Markus Hintersteiner Date: Thu, 9 Jul 2026 11:20:09 +0200 Subject: [PATCH 1/5] docs: Add develop-docs with Perfetto profiling pipeline documentation Introduce a develop-docs folder for internal developer documentation, with a README defining the docs conventions (flat structure, prefixes, naming, images, Mermaid) and a first feature doc covering the Android Perfetto profiling pipeline from the SDK through Relay to the monolith. Co-Authored-By: Claude Opus 4.8 (1M context) --- develop-docs/README.md | 104 +++++++++++ develop-docs/feature-perfetto-profiling.md | 206 +++++++++++++++++++++ 2 files changed, 310 insertions(+) create mode 100644 develop-docs/README.md create mode 100644 develop-docs/feature-perfetto-profiling.md diff --git a/develop-docs/README.md b/develop-docs/README.md new file mode 100644 index 00000000000..946a62999be --- /dev/null +++ b/develop-docs/README.md @@ -0,0 +1,104 @@ +# Develop Docs + +This folder holds internal developer documentation for the Sentry Java/Android SDK: +architecture notes, feature deep-dives, design decisions, and cross-module concepts +that don't belong in the public [Sentry docs](https://docs.sentry.io) or in inline +code comments. + +If you are documenting **how** or **why** something works for the people who maintain +this SDK, it goes here. If you are documenting **how to use** the SDK for end users, +it belongs in the public docs instead. + +## Rules + +These rules keep the docs consistent, easy to navigate, and easy to grep. + +### Directory structure + +- Use a **flat directory structure**. All documents live directly in `develop-docs/`. + Do not create nested category folders. +- Group and namespace documents with **filename prefixes** instead of folders. Common + prefixes: + - `general-` — cross-cutting topics (e.g. `general-development.md`, `general-architecture.md`) + - `feature-` — a specific SDK feature (e.g. `feature-errors.md`, `feature-profiling.md`) + - `integration-` — a specific integration or module (e.g. `integration-opentelemetry.md`, `integration-spring.md`) + - `platform-` — platform-specific concerns (e.g. `platform-android.md`, `platform-jvm.md`) + - `process-` — team processes and workflows (e.g. `process-release.md`) + + Add a new prefix only when an existing one clearly does not fit, and keep the list + above up to date. + +### File naming + +- File names are **lowercase**. +- Use **dashes** (`-`) as separators, never underscores or spaces. For example, use + `feature-profiling.md`, not `feature_profiling.md` or `Feature Profiling.md`. +- Use the `.md` extension for all documents. +- Choose short, descriptive names. The prefix conveys the category, so the rest of the + name only needs to describe the topic (`feature-session-replay.md`, not + `feature-session-replay-how-it-works.md`). + +### Images and other assets + +- When a document embeds images (or other binary assets), store them in a **folder with + the same base name** as the document. For example, `feature-profiling.md` keeps its + assets in `feature-profiling/`: + + ```text + develop-docs/ + feature-profiling.md + feature-profiling/ + pipeline.png + overview.svg + ``` + +- Reference assets with **relative paths**: `![Profiling pipeline](feature-profiling/pipeline.png)`. +- Asset file names follow the same rules as documents: lowercase, dashes, descriptive. +- Prefer **vector formats** (SVG) for diagrams and screenshots where practical +- Prefer **Mermaid** over a static image whenever a diagram can be expressed as one + (see below) — it lives in the document, is versioned as text, and is easy to update. + +### Writing style + +- Write in the **present tense** and the **active voice**. Describe how the system + behaves now ("The transport retries failed envelopes"), not how it will or did behave. +- Keep one **top-level `# ` heading** per document (the title), and nest sections with + `##`, `###`, etc. Do not skip heading levels. +- Keep documents focused on a **single topic**. Split large topics into multiple + prefixed documents and link between them rather than growing one giant file. +- Use fenced **code blocks with a language identifier** (```kotlin `, + ` ```bash `) so syntax highlighting works. +- Prefer Kotlin snippets over Java. +- When referencing code, link to the file with a **relative path** (e.g. + `../sentry/src/main/java/io/sentry/Sentry.java`) rather than pasting large excerpts + that fall out of date. +- Avoid pinning content to a specific SDK version or date unless it is genuinely + version-specific; keep docs evergreen. +- Cross-link related documents with relative links (e.g. `[profiling](feature-profiling.md)`). + +### Diagrams with Mermaid + +- Prefer [Mermaid](https://mermaid.js.org/) for diagrams. It renders directly on GitHub + and lives in the document as text, so it versions and reviews like code. +- Embed a Mermaid diagram in a fenced block tagged `mermaid`: + + ````markdown + ```mermaid + flowchart LR + Event[SentryEvent] --> Processor[EventProcessors] + Processor --> Transport + Transport --> Sentry[(Sentry)] + ``` + ```` + +- For complex diagrams, include a link to the [Mermaid Live Editor](https://mermaid.live/) + so reviewers can iterate quickly. +- Fall back to a static image (stored per the asset rules above) only when a diagram + cannot reasonably be expressed in Mermaid. + +## Adding a new document + +1. Pick the right prefix (or introduce a new one and document it above). +2. Create `develop-docs/-.md` with a single `# ` title. +3. If it embeds assets, create the matching `develop-docs/-/` folder. +4. Link to it from related documents so it is discoverable. diff --git a/develop-docs/feature-perfetto-profiling.md b/develop-docs/feature-perfetto-profiling.md new file mode 100644 index 00000000000..5dac7f2a55b --- /dev/null +++ b/develop-docs/feature-perfetto-profiling.md @@ -0,0 +1,206 @@ +# Perfetto profiling on Android + +This document describes how continuous profiling works on Android when the SDK +captures traces through the OS-level [`android.os.ProfilingManager`](https://developer.android.com/reference/android/os/ProfilingManager) +API (available on API 35+), and how a captured **profile chunk** flows all the way +from the device to a downloadable profile in Sentry. + +## What Perfetto is + +[Perfetto](https://perfetto.dev/) is Android's system-wide tracing stack. Its +[stack-sampling profiler](https://perfetto.dev/docs/data-sources/native-heap-profiler) +records periodic samples of native and Java call stacks and writes them to a binary +`.pftrace` trace file (a serialized [Perfetto protobuf](https://perfetto.dev/docs/reference/trace-packet-proto)). +Starting with Android 15, apps can request such traces at +runtime via `ProfilingManager` without root or `adb`, which is what makes on-device +continuous profiling possible. + +Useful Perfetto references: + +- Perfetto docs: https://perfetto.dev/docs/ +- Trace format (`TracePacket` proto): https://perfetto.dev/docs/reference/trace-packet-proto +- Perfetto UI (to open a downloaded `.pftrace`): https://ui.perfetto.dev/ + +## Pipeline overview + +A profile chunk is captured on the device, embedded into a Sentry envelope, expanded and +routed by Relay, and finally symbolicated and stored by the monolith so it can be served +as a flamegraph and downloaded as a raw Perfetto trace. + +```mermaid +flowchart TD + subgraph device["Android device — sentry-java"] + PM[android.os.ProfilingManager] + PP[PerfettoProfiler] + PCP[PerfettoContinuousProfiler] + PC[ProfileChunk] + ENV["Envelope item
[JSON metadata][raw .pftrace]
header: meta_length"] + PM --> PP --> PCP --> PC --> ENV + end + + subgraph relay["Relay (processing mode)"] + SPLIT[Split payload at meta_length] + CONV[Convert Perfetto → Sample v2] + OS1[Upload raw .pftrace to object store] + KAFKA[["Kafka topic: profiles
ProfileChunkKafkaMessage
(Sample v2 + attachment stored_id)"]] + SPLIT --> CONV --> KAFKA + SPLIT --> OS1 + end + + subgraph monolith["Monolith — getsentry/sentry"] + TASK[process_profile_task] + SYM[Symbolicate / deobfuscate] + VR[vroomrs: parse + normalize] + OS2[(Object store)] + SNUBA[(Snuba: function metrics)] + DB[(ProfileChunkAttachment row)] + TASK --> SYM --> VR + VR --> OS2 + VR --> SNUBA + TASK --> DB + end + + ENV -->|envelope| relay + KAFKA --> TASK + OS1 -.stored_id.-> DB + VROOM[getsentry/vroom
serve + merge flamegraphs] + OS2 --> VROOM + SNUBA --> VROOM +``` + +## SDK (getsentry/sentry-java) + +On API 35+, [`AndroidOptionsInitializer`](../sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java) +wires up `PerfettoContinuousProfiler` automatically. On older devices the SDK falls back +to the legacy `Debug`-based [`AndroidContinuousProfiler`](../sentry-android-core/src/main/java/io/sentry/android/core/AndroidContinuousProfiler.java), +gated by the `enableLegacyProfiling` option (manifest key +`io.sentry.profiling.enable-legacy-profiling`, defaults to `true`). Only **continuous +profiling** is supported on the Perfetto path — transaction-based and app-start profiling +are not. + +### Capturing chunks + +Continuous profiling emits a stream of independent [`ProfileChunk`](../sentry/src/main/java/io/sentry/ProfileChunk.java)s +rather than one profile per transaction. `PerfettoContinuousProfiler` drives a chained +loop: each chunk runs for `MAX_CHUNK_DURATION_MILLIS` (60s) via `PerfettoProfiler`, which +calls `ProfilingManager.requestProfiling(PROFILING_TYPE_STACK_SAMPLING, …)` at +`PROFILING_FREQUENCY_HZ` (101 Hz). When a chunk's trace file is ready, a new chunk starts, +so profiling runs continuously. + +A chunk keeps a stable `profilerId` across the session and a per-chunk `chunkId`. When the +OS produces the trace file, the profiler builds a `ProfileChunk` tagged with the Perfetto +content type: + +```kotlin +ProfileChunk.Builder(profilerId, chunkId, measurements, traceFile, timestamp, ProfileChunk.PLATFORM_ANDROID) + .setContentType(ProfileChunk.CONTENT_TYPE_PERFETTO) // "application/x-perfetto-trace" + .build() +``` + +The chunk is captured via `scopes.captureProfileChunk(...)` and sent as its own envelope +with item type [`SentryItemType.ProfileChunk`](../sentry/src/main/java/io/sentry/SentryItemType.java) +(wire name `profile_chunk`). + +### Envelope format and the `meta_length` header + +A legacy chunk base64-encodes its trace into the `ProfileChunk` JSON. A Perfetto chunk is +much larger, so [`SentryClient`](../sentry/src/main/java/io/sentry/SentryClient.java) instead +routes it through the new `SentryEnvelopeItem.fromPerfettoProfileChunk(...)` factory, which +avoids base64 by sending the raw binary alongside the JSON. + +The trick is a single envelope **item** whose payload concatenates the JSON metadata and +the raw `.pftrace` bytes with **no delimiter**: + +```text +[ProfileChunk JSON bytes][raw .pftrace binary bytes] +``` + +A new `meta_length` property on the [envelope item header](../sentry/src/main/java/io/sentry/SentryEnvelopeItemHeader.java) +tells the server where the JSON ends and the binary begins. The standard envelope item +structure (header line + newline + payload) is unchanged; `meta_length` simply subdivides +the payload: + +```text +{"type":"profile_chunk","content_type":"application/x-perfetto-trace","filename":"…","length":,"meta_length":} + +``` + +- `length` — total payload size (JSON + binary), as for any envelope item. +- `meta_length` — byte length of the JSON prefix. It is only known after the payload is + serialized, so the header computes it lazily (via a `Callable`) and omits the + field entirely for non-Perfetto items, keeping the change backward compatible. + +## Relay (getsentry/relay) + +Relay processing is gated behind the `organizations:continuous-profiling-perfetto` feature +flag; without it the chunk is dropped. In processing mode Relay: + +1. **Splits** the compound item payload at `meta_length` into `(metadata JSON, raw profile)` + and reads `content_type: "perfetto"` from the metadata. +2. **Converts** the binary Perfetto trace into the existing **Sample v2** profile JSON + format (`relay_profiling::expand_perfetto(...)`, backed by a checked-in subset of the + Perfetto protobuf definitions). +3. **Uploads** the raw `.pftrace` blob to object store (usecase `profiles`, keyed per + org/project, with an attachment-retention TTL). +4. **Produces** a `ProfileChunkKafkaMessage` to the `profiles` Kafka topic. The message + carries the expanded Sample v2 JSON as `payload` plus an `attachments` array, where each + attachment records: + - `name` (e.g. `profile.perfetto`), + - `content_type` (e.g. `application/x-perfetto-trace`), + - `stored_id` — the object store key of the uploaded raw blob. + +The monolith later uses `stored_id` to fetch the raw trace back. + +## Monolith (getsentry/sentry) + +A profiling task consumes the `profiles` topic. `process_profile_task` (in +`src/sentry/profiles/task.py`): + +1. Unpacks the message and parses the Sample v2 `payload`. +2. Runs **deobfuscation** (Android only for now). +3. Calls `vroomrs` to parse and normalize the chunk + (`vroomrs.profile_chunk_from_json_str(...)`), compresses it, and saves it to the profile + **object store** bucket. +4. Extracts function metrics and emits them to **Snuba** (for querying and flamegraph + metadata). +5. Persists a lightweight **`ProfileChunkAttachment`** DB row for each attachment on the + message — storing `project_id`, `profiler_id`, `chunk_id`, `name`, `content_type`, and + the `stored_id` object store key — so the raw Perfetto trace can be downloaded by ID + rather than exposing the `stored_id` directly. + +`getsentry/vroom` is an extra service on top, which reads the stored chunk and Snuba-indexed metadata to serve and merge multiple profile chunks into a single flamegraph. The API endpoint is served by the monolith, but it just passes the request through to vroom. + +### Perfetto format dispatch (vroom / vroomrs) + +Older Android SDKs emit the legacy Android trace format tagged as a "faulty" `version=2`, +and the pipeline historically keyed off the platform rather than the version. To +distinguish legacy from Sample v2 chunks, `ProfileChunk` carries a dedicated `version` +field, and both `vroom` and `vroomrs` now dispatch on it instead of the platform: + +- Version `""` or `2.android-trace` → legacy Android trace format. +- Any other version → Sample v2. + +## Downloading a Perfetto profile + +The monolith exposes two feature-gated endpoints (both require the +`organizations:continuous-profiling-perfetto` flag): + +- **List attachments** — `GET /organizations/{org}/profiling/chunk-attachments/` + (`sentry-api-0-organization-profiling-chunk-attachments`). Requires a `project` and + `profiler_id`; resolves the visible `chunk_id`s (same logic as the flamegraph) and returns + the matching `ProfileChunkAttachment` metadata. +- **Download** — `GET /projects/{org}/{project}/profiling/chunks/{profiler_id}/{chunk_id}/attachments/{attachment_id}/?download` + (`sentry-api-0-project-profiling-chunk-attachment`). The `?download` param is required; it + streams the raw blob back from object store via the stored `stored_id`. Access requires + the org's configured attachments role, analogous to generic event attachments. + +In the flamegraph UI, a toolbar button (added for continuous profiles when the flag is on +and at least one attachment exists) lists and downloads these traces. + +## References + +- SDK: [sentry-java#5251](https://github.com/getsentry/sentry-java/pull/5251) — Android `ProfilingManager` (Perfetto) support +- Relay: [#5659](https://github.com/getsentry/relay/pull/5659), [#5932](https://github.com/getsentry/relay/pull/5932), [#6099](https://github.com/getsentry/relay/pull/6099), [#6102](https://github.com/getsentry/relay/pull/6102) — Perfetto parsing, pipeline, and object-store routing +- vroom: [#672](https://github.com/getsentry/vroom/pull/672) — version dispatch for Android trace profiles +- vroomrs: [#93](https://github.com/getsentry/vroomrs/pull/93) — accept Android profiles in Sample v2 format +- Monolith: [sentry#118029](https://github.com/getsentry/sentry/pull/118029) (chunk attachments + endpoints), [sentry#118071](https://github.com/getsentry/sentry/pull/118071) (flamegraph download button) From 96c8447c2508c0f51977e6fa2c30540a431c5877 Mon Sep 17 00:00:00 2001 From: Markus Hintersteiner Date: Thu, 9 Jul 2026 11:23:28 +0200 Subject: [PATCH 2/5] docs: Add Kafka message snippet to Perfetto profiling doc Co-Authored-By: Claude Opus 4.8 (1M context) --- develop-docs/feature-perfetto-profiling.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/develop-docs/feature-perfetto-profiling.md b/develop-docs/feature-perfetto-profiling.md index 5dac7f2a55b..01d756b7e49 100644 --- a/develop-docs/feature-perfetto-profiling.md +++ b/develop-docs/feature-perfetto-profiling.md @@ -149,6 +149,23 @@ flag; without it the chunk is dropped. In processing mode Relay: - `content_type` (e.g. `application/x-perfetto-trace`), - `stored_id` — the object store key of the uploaded raw blob. +```json +{ + "organization_id": 1, + "project_id": 42, + "received": 1720000000, + "retention_days": 30, + "payload": "", + "attachments": [ + { + "name": "profile.perfetto", + "content_type": "application/x-perfetto-trace", + "stored_id": "" + } + ] +} +``` + The monolith later uses `stored_id` to fetch the raw trace back. ## Monolith (getsentry/sentry) From d1f4041d5033827154e84587dbafc88f7bb36edc Mon Sep 17 00:00:00 2001 From: Markus Hintersteiner Date: Thu, 16 Jul 2026 12:05:00 +0200 Subject: [PATCH 3/5] Make the rules more concise --- develop-docs/README.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/develop-docs/README.md b/develop-docs/README.md index 946a62999be..1880a4b03db 100644 --- a/develop-docs/README.md +++ b/develop-docs/README.md @@ -15,10 +15,8 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. ### Directory structure -- Use a **flat directory structure**. All documents live directly in `develop-docs/`. - Do not create nested category folders. -- Group and namespace documents with **filename prefixes** instead of folders. Common - prefixes: +- Use a **flat directory structure** for all markdown files, they live live directly in `develop-docs/`. +- All markdown files are prefixed with a group / or namespace. Common prefixes are: - `general-` — cross-cutting topics (e.g. `general-development.md`, `general-architecture.md`) - `feature-` — a specific SDK feature (e.g. `feature-errors.md`, `feature-profiling.md`) - `integration-` — a specific integration or module (e.g. `integration-opentelemetry.md`, `integration-spring.md`) @@ -33,7 +31,7 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. - File names are **lowercase**. - Use **dashes** (`-`) as separators, never underscores or spaces. For example, use `feature-profiling.md`, not `feature_profiling.md` or `Feature Profiling.md`. -- Use the `.md` extension for all documents. +- Use the `.md` extension for all text documents. - Choose short, descriptive names. The prefix conveys the category, so the rest of the name only needs to describe the topic (`feature-session-replay.md`, not `feature-session-replay-how-it-works.md`). @@ -62,6 +60,7 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. - Write in the **present tense** and the **active voice**. Describe how the system behaves now ("The transport retries failed envelopes"), not how it will or did behave. + This way there's no need to update the docs once a feature ships. - Keep one **top-level `# ` heading** per document (the title), and nest sections with `##`, `###`, etc. Do not skip heading levels. - Keep documents focused on a **single topic**. Split large topics into multiple @@ -93,12 +92,10 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. - For complex diagrams, include a link to the [Mermaid Live Editor](https://mermaid.live/) so reviewers can iterate quickly. -- Fall back to a static image (stored per the asset rules above) only when a diagram - cannot reasonably be expressed in Mermaid. +- Fall back to static images (stored per the asset rules above) if mermaid is not practicable. ## Adding a new document 1. Pick the right prefix (or introduce a new one and document it above). -2. Create `develop-docs/-.md` with a single `# ` title. +2. Create the document `develop-docs/-.md` 3. If it embeds assets, create the matching `develop-docs/-/` folder. -4. Link to it from related documents so it is discoverable. From c9dbf3066209f195d79152f43692948924ac0f1a Mon Sep 17 00:00:00 2001 From: Markus Hintersteiner Date: Wed, 12 Aug 2026 10:38:43 +0200 Subject: [PATCH 4/5] docs: Extract ingestion pipeline into general-pipeline.md and address review feedback Co-Authored-By: Claude Opus 5 (1M context) --- develop-docs/README.md | 25 ++++- develop-docs/feature-perfetto-profiling.md | 61 ++++++----- develop-docs/general-pipeline.md | 121 +++++++++++++++++++++ 3 files changed, 179 insertions(+), 28 deletions(-) create mode 100644 develop-docs/general-pipeline.md diff --git a/develop-docs/README.md b/develop-docs/README.md index 1880a4b03db..9b2dbcf4176 100644 --- a/develop-docs/README.md +++ b/develop-docs/README.md @@ -15,7 +15,7 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. ### Directory structure -- Use a **flat directory structure** for all markdown files, they live live directly in `develop-docs/`. +- Use a **flat directory structure** for all markdown files, they live directly in `develop-docs/`. - All markdown files are prefixed with a group / or namespace. Common prefixes are: - `general-` — cross-cutting topics (e.g. `general-development.md`, `general-architecture.md`) - `feature-` — a specific SDK feature (e.g. `feature-errors.md`, `feature-profiling.md`) @@ -26,9 +26,15 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. Add a new prefix only when an existing one clearly does not fit, and keep the list above up to date. +- The prefixes carry the same grouping a nested directory tree would, while keeping every + document one `ls` and one glob away (`develop-docs/feature-*.md`). Nesting becomes worth + the churn once a prefix grows past roughly ten documents — revisit the structure then, + not before. + ### File naming -- File names are **lowercase**. +- File names are **lowercase**, except for this `README.md`, which GitHub renders as the + folder's landing page. - Use **dashes** (`-`) as separators, never underscores or spaces. For example, use `feature-profiling.md`, not `feature_profiling.md` or `Feature Profiling.md`. - Use the `.md` extension for all text documents. @@ -75,6 +81,21 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. version-specific; keep docs evergreen. - Cross-link related documents with relative links (e.g. `[profiling](feature-profiling.md)`). +### Structuring a feature document + +Most feature documents answer the same four questions, and following that order makes them +easier to compare and to keep current: + +1. **Surface area** — where and when the SDK collects the data. +2. **Collection** — how the SDK collects it. +3. **Format** — what the collected data looks like on the wire. +4. **Pipeline** — how the backend ingests, stores, and serves it. + +Do not restate (4) in every document. Describe the shared path once in +[general-pipeline.md](general-pipeline.md) and cover only the deviations a feature +introduces. Omit any of the four that a feature does not have, and keep each as high-level +as the topic allows so the document stays true for longer. + ### Diagrams with Mermaid - Prefer [Mermaid](https://mermaid.js.org/) for diagrams. It renders directly on GitHub diff --git a/develop-docs/feature-perfetto-profiling.md b/develop-docs/feature-perfetto-profiling.md index 01d756b7e49..ad38150ba6b 100644 --- a/develop-docs/feature-perfetto-profiling.md +++ b/develop-docs/feature-perfetto-profiling.md @@ -7,10 +7,12 @@ from the device to a downloadable profile in Sentry. ## What Perfetto is -[Perfetto](https://perfetto.dev/) is Android's system-wide tracing stack. Its -[stack-sampling profiler](https://perfetto.dev/docs/data-sources/native-heap-profiler) -records periodic samples of native and Java call stacks and writes them to a binary -`.pftrace` trace file (a serialized [Perfetto protobuf](https://perfetto.dev/docs/reference/trace-packet-proto)). +[Perfetto](https://perfetto.dev/) is Google's tracing framework for Android and Linux, and +the tooling Android itself is instrumented with. Its +[callstack sampler](https://perfetto.dev/docs/getting-started/cpu-profiling) interrupts the +app at a fixed frequency, records the native and Java call stacks of the running threads, +and writes them to a binary `.pftrace` file (a serialized +[Perfetto protobuf](https://perfetto.dev/docs/reference/trace-packet-proto)). Starting with Android 15, apps can request such traces at runtime via `ProfilingManager` without root or `adb`, which is what makes on-device continuous profiling possible. @@ -18,14 +20,25 @@ continuous profiling possible. Useful Perfetto references: - Perfetto docs: https://perfetto.dev/docs/ +- CPU profiling with Perfetto: https://perfetto.dev/docs/getting-started/cpu-profiling - Trace format (`TracePacket` proto): https://perfetto.dev/docs/reference/trace-packet-proto - Perfetto UI (to open a downloaded `.pftrace`): https://ui.perfetto.dev/ ## Pipeline overview -A profile chunk is captured on the device, embedded into a Sentry envelope, expanded and -routed by Relay, and finally symbolicated and stored by the monolith so it can be served -as a flamegraph and downloaded as a raw Perfetto trace. +Profile chunks travel the standard ingestion path described in +[general-pipeline.md](general-pipeline.md) — SDK envelope, +[Relay](https://develop.sentry.dev/ingestion/relay/) (Sentry's ingestion proxy), Kafka, a +monolith processing task, then storage and a read API. Read that first; the rest of this +document covers only where Perfetto deviates from it. + +The deviations are: + +- The envelope item carries **JSON and raw binary in one payload**, subdivided by a + `meta_length` header rather than base64-encoding the trace ([details](#envelope-format-and-the-meta_length-header)). +- Relay **converts** the Perfetto trace into the existing Sample v2 profile format, and + additionally **keeps the raw `.pftrace`** in the object store so it can be downloaded + later ([details](#relay-getsentryrelay)). ```mermaid flowchart TD @@ -132,8 +145,7 @@ the payload: ## Relay (getsentry/relay) -Relay processing is gated behind the `organizations:continuous-profiling-perfetto` feature -flag; without it the chunk is dropped. In processing mode Relay: +In processing mode Relay: 1. **Splits** the compound item payload at `meta_length` into `(metadata JSON, raw profile)` and reads `content_type: "perfetto"` from the metadata. @@ -170,22 +182,20 @@ The monolith later uses `stored_id` to fetch the raw trace back. ## Monolith (getsentry/sentry) -A profiling task consumes the `profiles` topic. `process_profile_task` (in -`src/sentry/profiles/task.py`): +`process_profile_task` (in `src/sentry/profiles/task.py`) consumes the `profiles` topic. +Because Relay already converted the trace to Sample v2, the task treats a Perfetto chunk +like any other: deobfuscate, hand it to `vroomrs` to parse and normalize +(`vroomrs.profile_chunk_from_json_str(...)`), compress and store it, and emit function +metrics to Snuba. -1. Unpacks the message and parses the Sample v2 `payload`. -2. Runs **deobfuscation** (Android only for now). -3. Calls `vroomrs` to parse and normalize the chunk - (`vroomrs.profile_chunk_from_json_str(...)`), compresses it, and saves it to the profile - **object store** bucket. -4. Extracts function metrics and emits them to **Snuba** (for querying and flamegraph - metadata). -5. Persists a lightweight **`ProfileChunkAttachment`** DB row for each attachment on the - message — storing `project_id`, `profiler_id`, `chunk_id`, `name`, `content_type`, and - the `stored_id` object store key — so the raw Perfetto trace can be downloaded by ID - rather than exposing the `stored_id` directly. +The Perfetto-specific step is the last one: for each attachment on the message the task +persists a lightweight **`ProfileChunkAttachment`** row — `project_id`, `profiler_id`, +`chunk_id`, `name`, `content_type`, and the `stored_id` object store key. The row exists so +the raw trace can be downloaded by ID without exposing the `stored_id`. -`getsentry/vroom` is an extra service on top, which reads the stored chunk and Snuba-indexed metadata to serve and merge multiple profile chunks into a single flamegraph. The API endpoint is served by the monolith, but it just passes the request through to vroom. +Flamegraphs themselves are served by `getsentry/vroom`, which reads the stored chunks and +the Snuba-indexed metadata and merges several chunks into one flamegraph. The endpoint +lives in the monolith and passes the request through. ### Perfetto format dispatch (vroom / vroomrs) @@ -199,8 +209,7 @@ field, and both `vroom` and `vroomrs` now dispatch on it instead of the platform ## Downloading a Perfetto profile -The monolith exposes two feature-gated endpoints (both require the -`organizations:continuous-profiling-perfetto` flag): +The monolith exposes two feature-gated endpoints: - **List attachments** — `GET /organizations/{org}/profiling/chunk-attachments/` (`sentry-api-0-organization-profiling-chunk-attachments`). Requires a `project` and @@ -212,7 +221,7 @@ The monolith exposes two feature-gated endpoints (both require the the org's configured attachments role, analogous to generic event attachments. In the flamegraph UI, a toolbar button (added for continuous profiles when the flag is on -and at least one attachment exists) lists and downloads these traces. +and at least one attachment exists) lists and provides a way to download these traces. ## References diff --git a/develop-docs/general-pipeline.md b/develop-docs/general-pipeline.md new file mode 100644 index 00000000000..03250f42097 --- /dev/null +++ b/develop-docs/general-pipeline.md @@ -0,0 +1,121 @@ +# Ingestion pipeline + +This document describes the path data takes from an SDK to a rendered view in Sentry. It +covers the parts of different payload types, like errors, transactions, logs, replays and +profile chunks. + +## Per data category + +Every payload takes the same four hops — SDK, Relay, a consumer in the monolith, and a read +API — but the topics, processing tasks, and stores differ per category. The diagrams below +show three of them; the hops themselves are described further down. + +### Errors + +```mermaid +flowchart LR + SDK["SDK
captures + batches"] -->|envelope| RELAY + RELAY["Relay
authenticate, normalize,
route"] -->|ingest-events| KAFKA[["Kafka"]] + RELAY -.->|attachments,
minidumps| OS[("Object store")] + KAFKA --> TASK["save_event task"] + TASK --> SYM["Symbolicator
symbolicate, deobfuscate"] + SYM --> TASK + TASK --> NS[("Nodestore
full event body")] + TASK --> SNUBA[("Snuba
searchable columns")] + TASK --> PG[("Postgres
Group / GroupHash rows")] + NS --> READ["Read path
monolith API"] + SNUBA --> READ + PG --> READ + OS --> READ +``` + +### Transactions + +```mermaid +flowchart LR + SDK["SDK
captures spans"] -->|envelope| RELAY + RELAY["Relay
normalize, dynamic sampling,
metric extraction"] -->|ingest-transactions| KAFKA[["Kafka"]] + KAFKA --> CONSUMER["Transaction consumer"] + CONSUMER --> SNUBA[("Snuba
transactions + spans")] + CONSUMER --> NS[("Nodestore
full transaction body")] + SNUBA --> READ["Read path
monolith API"] + NS --> READ +``` + +### Profile chunks + +```mermaid +flowchart LR + SDK["SDK
captures profile chunks"] -->|envelope| RELAY + RELAY["Relay
convert Perfetto → Sample v2"] -->|profiles| KAFKA[["Kafka"]] + RELAY -.->|raw .pftrace blob| OS[("Object store")] + KAFKA --> TASK["process_profile_task"] + TASK --> VRS["vroomrs
parse + normalize"] + VRS --> OS + VRS --> SNUBA[("Snuba
function metrics")] + TASK --> PG[("Postgres
ProfileChunkAttachment rows")] + OS --> VROOM["vroom
serve + merge flamegraphs"] + SNUBA --> VROOM + VROOM --> READ["Read path
monolith API"] + PG --> READ +``` + +## The hops + +### 1. SDK + +The SDK captures data and wraps it in an [envelope](https://develop.sentry.dev/sdk/data-model/envelopes/): +a JSON header followed by one or more items, each with its own header declaring a `type`, +a `length`, and optionally a `content_type`. The envelope is POSTed to the project's +`/api/{project_id}/envelope/` endpoint. + +The item `type` is what routes the payload through everything downstream, so adding a new +kind of data means adding an item type, not a new endpoint. Item payloads are usually JSON; +binary payloads are allowed and are preferable to base64-encoding a large blob into JSON. + +### 2. Relay + +[Relay](https://github.com/getsentry/relay) is Sentry's ingestion proxy — it sits between +the SDK and the rest of the infrastructure and is the first service to inspect a payload. +See the [Relay chapter in develop docs](https://develop.sentry.dev/ingestion/relay/) for +the full picture. + +Relay authenticates the DSN, applies quotas and rate limits, filters and normalizes the +payload, and forwards it. Two behaviours matter when designing a new payload type: + +- Relay may **convert** a payload into a different format before publishing it, so the + format the SDK sends and the format the backend consumes are not necessarily the same. + Whatever Relay publishes is the contract every downstream service depends on. +- Relay runs in two modes. Only **processing mode** (the one Sentry operates) talks to + Kafka and the object store; a self-hosted Relay in proxy mode just forwards envelopes + upstream. + +Relay publishes to a **Kafka topic per data category**. Payloads too large to sit +comfortably in a Kafka message are uploaded to the **object store** instead, and the +message carries a reference to the stored blob rather than the bytes themselves. Event +attachments (minidumps, screenshots, view hierarchies) work this way, and so does the raw +`.pftrace` blob of a Perfetto profile chunk: Relay uploads the trace and puts only its +`stored_id` object store key on the Kafka message. + +### 3. Consumers and processing + +Each topic is consumed by the monolith ([getsentry/sentry](https://github.com/getsentry/sentry)), +which runs a processing task per message. This is where the work that needs Sentry-side +state happens — symbolication and deobfuscation against uploaded debug files, enrichment, +normalization, and quota accounting. + +A task typically writes to more than one store: + +- **Object store** — the payload itself, compressed. Cheap to keep, not queryable. +- **Snuba** — the columns that need to be searched, aggregated, or listed. +- **Postgres** — small metadata rows that the API needs to resolve a request, for example + a row per stored blob so it can be fetched by ID instead of by exposing its storage key. + +### 4. Read path + +The monolith serves the API endpoints. For some categories it does the work itself; for +others it authorizes the request and proxies it to a dedicated service that owns the +heavy read logic. Either way the endpoint is the public surface, and the storage keys and +internal services stay behind it. + +See [feature-perfetto-profiling.md](feature-perfetto-profiling.md) for a worked example. From b6f76e0c4af04edb8e3190bcf70b994cc343fca1 Mon Sep 17 00:00:00 2001 From: Markus Hintersteiner Date: Wed, 12 Aug 2026 12:11:46 +0200 Subject: [PATCH 5/5] docs: Organize develop-docs into category subdirectories Replaces the flat filename-prefix convention with a directory tree, per PR review feedback. Co-Authored-By: Claude Opus 5 (1M context) --- develop-docs/README.md | 99 ++++++++++++------- .../profiling/perfetto.md} | 18 ++-- .../pipeline.md} | 2 +- 3 files changed, 73 insertions(+), 46 deletions(-) rename develop-docs/{feature-perfetto-profiling.md => feature/profiling/perfetto.md} (92%) rename develop-docs/{general-pipeline.md => general/pipeline.md} (98%) diff --git a/develop-docs/README.md b/develop-docs/README.md index 9b2dbcf4176..b9c2c7913ca 100644 --- a/develop-docs/README.md +++ b/develop-docs/README.md @@ -15,48 +15,72 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. ### Directory structure -- Use a **flat directory structure** for all markdown files, they live directly in `develop-docs/`. -- All markdown files are prefixed with a group / or namespace. Common prefixes are: - - `general-` — cross-cutting topics (e.g. `general-development.md`, `general-architecture.md`) - - `feature-` — a specific SDK feature (e.g. `feature-errors.md`, `feature-profiling.md`) - - `integration-` — a specific integration or module (e.g. `integration-opentelemetry.md`, `integration-spring.md`) - - `platform-` — platform-specific concerns (e.g. `platform-android.md`, `platform-jvm.md`) - - `process-` — team processes and workflows (e.g. `process-release.md`) - - Add a new prefix only when an existing one clearly does not fit, and keep the list - above up to date. - -- The prefixes carry the same grouping a nested directory tree would, while keeping every - document one `ls` and one glob away (`develop-docs/feature-*.md`). Nesting becomes worth - the churn once a prefix grows past roughly ten documents — revisit the structure then, - not before. +Documents live in **subdirectories**, one level per level of grouping. Directories are +cheap: reach for a new one as soon as a topic has more than one document, or as soon as you +can name the group. + +Every document sits under one of these top-level categories: + +- `general/` — cross-cutting topics (e.g. `general/architecture.md`, `general/pipeline.md`) +- `feature/` — a specific SDK feature (e.g. `feature/errors/`, `feature/profiling/`) +- `integration/` — a specific integration or module (e.g. `integration/opentelemetry/`, `integration/spring/`) +- `platform/` — platform-specific concerns (e.g. `platform/android/`, `platform/jvm/`) +- `process/` — team processes and workflows (e.g. `process/release.md`) + +Add a new category only when an existing one clearly does not fit, and keep the list above +up to date. + +Below the category, nest by topic and then by sub-topic. A fully grown feature might look +like this: + +```text +develop-docs/ + README.md + general/ + pipeline.md + feature/ + profiling/ + overview.md + perfetto.md + anr.md + symbolication/ + deobfuscation.md +``` + +- Give a directory an `overview.md` once it holds several documents, and link to its + siblings from there. +- Do not create a directory that will only ever hold one document — put the document + directly in the category (`general/pipeline.md`, not `general/pipeline/pipeline.md`). ### File naming - File names are **lowercase**, except for this `README.md`, which GitHub renders as the folder's landing page. - Use **dashes** (`-`) as separators, never underscores or spaces. For example, use - `feature-profiling.md`, not `feature_profiling.md` or `Feature Profiling.md`. + `session-replay.md`, not `session_replay.md` or `Session Replay.md`. - Use the `.md` extension for all text documents. -- Choose short, descriptive names. The prefix conveys the category, so the rest of the - name only needs to describe the topic (`feature-session-replay.md`, not - `feature-session-replay-how-it-works.md`). +- **Do not repeat the path in the file name.** The directories carry the namespace, so the + file name only needs the part that distinguishes it from its siblings: + `feature/profiling/perfetto.md`, not `feature/profiling/perfetto-profiling.md`. +- Choose short, descriptive names (`feature/replay/masking.md`, not + `feature/replay/how-masking-works.md`). ### Images and other assets -- When a document embeds images (or other binary assets), store them in a **folder with - the same base name** as the document. For example, `feature-profiling.md` keeps its - assets in `feature-profiling/`: +- When a document embeds images (or other binary assets), store them in an **`assets/` + folder next to the document**. Documents in the same directory share it: ```text develop-docs/ - feature-profiling.md - feature-profiling/ - pipeline.png - overview.svg + feature/ + profiling/ + perfetto.md + assets/ + pipeline.png + overview.svg ``` -- Reference assets with **relative paths**: `![Profiling pipeline](feature-profiling/pipeline.png)`. +- Reference assets with **relative paths**: `![Profiling pipeline](assets/pipeline.png)`. - Asset file names follow the same rules as documents: lowercase, dashes, descriptive. - Prefer **vector formats** (SVG) for diagrams and screenshots where practical - Prefer **Mermaid** over a static image whenever a diagram can be expressed as one @@ -69,17 +93,18 @@ These rules keep the docs consistent, easy to navigate, and easy to grep. This way there's no need to update the docs once a feature ships. - Keep one **top-level `# ` heading** per document (the title), and nest sections with `##`, `###`, etc. Do not skip heading levels. -- Keep documents focused on a **single topic**. Split large topics into multiple - prefixed documents and link between them rather than growing one giant file. +- Keep documents focused on a **single topic**. Split large topics into several documents + in a shared directory and link between them rather than growing one giant file. - Use fenced **code blocks with a language identifier** (```kotlin `, ` ```bash `) so syntax highlighting works. - Prefer Kotlin snippets over Java. - When referencing code, link to the file with a **relative path** (e.g. - `../sentry/src/main/java/io/sentry/Sentry.java`) rather than pasting large excerpts - that fall out of date. + `../../../sentry/src/main/java/io/sentry/Sentry.java`) rather than pasting large excerpts + that fall out of date. Count the `../` from the document's own directory. - Avoid pinning content to a specific SDK version or date unless it is genuinely version-specific; keep docs evergreen. -- Cross-link related documents with relative links (e.g. `[profiling](feature-profiling.md)`). +- Cross-link related documents with relative links (e.g. + `[the ingestion pipeline](../../general/pipeline.md)`). ### Structuring a feature document @@ -92,7 +117,7 @@ easier to compare and to keep current: 4. **Pipeline** — how the backend ingests, stores, and serves it. Do not restate (4) in every document. Describe the shared path once in -[general-pipeline.md](general-pipeline.md) and cover only the deviations a feature +[general/pipeline.md](general/pipeline.md) and cover only the deviations a feature introduces. Omit any of the four that a feature does not have, and keep each as high-level as the topic allows so the document stays true for longer. @@ -117,6 +142,8 @@ as the topic allows so the document stays true for longer. ## Adding a new document -1. Pick the right prefix (or introduce a new one and document it above). -2. Create the document `develop-docs/-.md` -3. If it embeds assets, create the matching `develop-docs/-/` folder. +1. Pick the right top-level category (or introduce a new one and document it above). +2. Pick or create the topic directory below it. +3. Create the document, naming it for what distinguishes it from its siblings. +4. If the directory now holds several documents, add or update its `overview.md`. +5. If the document embeds assets, put them in an `assets/` folder next to it. diff --git a/develop-docs/feature-perfetto-profiling.md b/develop-docs/feature/profiling/perfetto.md similarity index 92% rename from develop-docs/feature-perfetto-profiling.md rename to develop-docs/feature/profiling/perfetto.md index ad38150ba6b..d4168f8f2a4 100644 --- a/develop-docs/feature-perfetto-profiling.md +++ b/develop-docs/feature/profiling/perfetto.md @@ -27,7 +27,7 @@ Useful Perfetto references: ## Pipeline overview Profile chunks travel the standard ingestion path described in -[general-pipeline.md](general-pipeline.md) — SDK envelope, +[general/pipeline.md](../../general/pipeline.md) — SDK envelope, [Relay](https://develop.sentry.dev/ingestion/relay/) (Sentry's ingestion proxy), Kafka, a monolith processing task, then storage and a read API. Read that first; the rest of this document covers only where Perfetto deviates from it. @@ -83,9 +83,9 @@ flowchart TD ## SDK (getsentry/sentry-java) -On API 35+, [`AndroidOptionsInitializer`](../sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java) +On API 35+, [`AndroidOptionsInitializer`](../../../sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java) wires up `PerfettoContinuousProfiler` automatically. On older devices the SDK falls back -to the legacy `Debug`-based [`AndroidContinuousProfiler`](../sentry-android-core/src/main/java/io/sentry/android/core/AndroidContinuousProfiler.java), +to the legacy `Debug`-based [`AndroidContinuousProfiler`](../../../sentry-android-core/src/main/java/io/sentry/android/core/AndroidContinuousProfiler.java), gated by the `enableLegacyProfiling` option (manifest key `io.sentry.profiling.enable-legacy-profiling`, defaults to `true`). Only **continuous profiling** is supported on the Perfetto path — transaction-based and app-start profiling @@ -93,7 +93,7 @@ are not. ### Capturing chunks -Continuous profiling emits a stream of independent [`ProfileChunk`](../sentry/src/main/java/io/sentry/ProfileChunk.java)s +Continuous profiling emits a stream of independent [`ProfileChunk`](../../../sentry/src/main/java/io/sentry/ProfileChunk.java)s rather than one profile per transaction. `PerfettoContinuousProfiler` drives a chained loop: each chunk runs for `MAX_CHUNK_DURATION_MILLIS` (60s) via `PerfettoProfiler`, which calls `ProfilingManager.requestProfiling(PROFILING_TYPE_STACK_SAMPLING, …)` at @@ -111,13 +111,13 @@ ProfileChunk.Builder(profilerId, chunkId, measurements, traceFile, timestamp, Pr ``` The chunk is captured via `scopes.captureProfileChunk(...)` and sent as its own envelope -with item type [`SentryItemType.ProfileChunk`](../sentry/src/main/java/io/sentry/SentryItemType.java) +with item type [`SentryItemType.ProfileChunk`](../../../sentry/src/main/java/io/sentry/SentryItemType.java) (wire name `profile_chunk`). ### Envelope format and the `meta_length` header A legacy chunk base64-encodes its trace into the `ProfileChunk` JSON. A Perfetto chunk is -much larger, so [`SentryClient`](../sentry/src/main/java/io/sentry/SentryClient.java) instead +much larger, so [`SentryClient`](../../../sentry/src/main/java/io/sentry/SentryClient.java) instead routes it through the new `SentryEnvelopeItem.fromPerfettoProfileChunk(...)` factory, which avoids base64 by sending the raw binary alongside the JSON. @@ -128,7 +128,7 @@ the raw `.pftrace` bytes with **no delimiter**: [ProfileChunk JSON bytes][raw .pftrace binary bytes] ``` -A new `meta_length` property on the [envelope item header](../sentry/src/main/java/io/sentry/SentryEnvelopeItemHeader.java) +A new `meta_length` property on the [envelope item header](../../../sentry/src/main/java/io/sentry/SentryEnvelopeItemHeader.java) tells the server where the JSON ends and the binary begins. The standard envelope item structure (header line + newline + payload) is unchanged; `meta_length` simply subdivides the payload: @@ -220,8 +220,8 @@ The monolith exposes two feature-gated endpoints: streams the raw blob back from object store via the stored `stored_id`. Access requires the org's configured attachments role, analogous to generic event attachments. -In the flamegraph UI, a toolbar button (added for continuous profiles when the flag is on -and at least one attachment exists) lists and provides a way to download these traces. +In the flamegraph UI, a toolbar button (added for continuous profiles when the feature is +enabled and at least one attachment exists) lists and provides a way to download these traces. ## References diff --git a/develop-docs/general-pipeline.md b/develop-docs/general/pipeline.md similarity index 98% rename from develop-docs/general-pipeline.md rename to develop-docs/general/pipeline.md index 03250f42097..6cb0d87f696 100644 --- a/develop-docs/general-pipeline.md +++ b/develop-docs/general/pipeline.md @@ -118,4 +118,4 @@ others it authorizes the request and proxies it to a dedicated service that owns heavy read logic. Either way the endpoint is the public surface, and the storage keys and internal services stay behind it. -See [feature-perfetto-profiling.md](feature-perfetto-profiling.md) for a worked example. +See [feature/profiling/perfetto.md](../feature/profiling/perfetto.md) for a worked example.