diff --git a/develop-docs/README.md b/develop-docs/README.md new file mode 100644 index 00000000000..b9c2c7913ca --- /dev/null +++ b/develop-docs/README.md @@ -0,0 +1,149 @@ +# Develop Docs + +This folder holds internal developer documentation for the Sentry Java/Android SDK: +architecture notes, feature deep-dives, design decisions, and cross-module concepts +that don't belong in the public [Sentry docs](https://docs.sentry.io) or in inline +code comments. + +If you are documenting **how** or **why** something works for the people who maintain +this SDK, it goes here. If you are documenting **how to use** the SDK for end users, +it belongs in the public docs instead. + +## Rules + +These rules keep the docs consistent, easy to navigate, and easy to grep. + +### Directory structure + +Documents live in **subdirectories**, one level per level of grouping. Directories are +cheap: reach for a new one as soon as a topic has more than one document, or as soon as you +can name the group. + +Every document sits under one of these top-level categories: + +- `general/` — cross-cutting topics (e.g. `general/architecture.md`, `general/pipeline.md`) +- `feature/` — a specific SDK feature (e.g. `feature/errors/`, `feature/profiling/`) +- `integration/` — a specific integration or module (e.g. `integration/opentelemetry/`, `integration/spring/`) +- `platform/` — platform-specific concerns (e.g. `platform/android/`, `platform/jvm/`) +- `process/` — team processes and workflows (e.g. `process/release.md`) + +Add a new category only when an existing one clearly does not fit, and keep the list above +up to date. + +Below the category, nest by topic and then by sub-topic. A fully grown feature might look +like this: + +```text +develop-docs/ + README.md + general/ + pipeline.md + feature/ + profiling/ + overview.md + perfetto.md + anr.md + symbolication/ + deobfuscation.md +``` + +- Give a directory an `overview.md` once it holds several documents, and link to its + siblings from there. +- Do not create a directory that will only ever hold one document — put the document + directly in the category (`general/pipeline.md`, not `general/pipeline/pipeline.md`). + +### File naming + +- File names are **lowercase**, except for this `README.md`, which GitHub renders as the + folder's landing page. +- Use **dashes** (`-`) as separators, never underscores or spaces. For example, use + `session-replay.md`, not `session_replay.md` or `Session Replay.md`. +- Use the `.md` extension for all text documents. +- **Do not repeat the path in the file name.** The directories carry the namespace, so the + file name only needs the part that distinguishes it from its siblings: + `feature/profiling/perfetto.md`, not `feature/profiling/perfetto-profiling.md`. +- Choose short, descriptive names (`feature/replay/masking.md`, not + `feature/replay/how-masking-works.md`). + +### Images and other assets + +- When a document embeds images (or other binary assets), store them in an **`assets/` + folder next to the document**. Documents in the same directory share it: + + ```text + develop-docs/ + feature/ + profiling/ + perfetto.md + assets/ + pipeline.png + overview.svg + ``` + +- Reference assets with **relative paths**: `![Profiling pipeline](assets/pipeline.png)`. +- Asset file names follow the same rules as documents: lowercase, dashes, descriptive. +- Prefer **vector formats** (SVG) for diagrams and screenshots where practical +- Prefer **Mermaid** over a static image whenever a diagram can be expressed as one + (see below) — it lives in the document, is versioned as text, and is easy to update. + +### Writing style + +- Write in the **present tense** and the **active voice**. Describe how the system + behaves now ("The transport retries failed envelopes"), not how it will or did behave. + This way there's no need to update the docs once a feature ships. +- Keep one **top-level `# ` heading** per document (the title), and nest sections with + `##`, `###`, etc. Do not skip heading levels. +- Keep documents focused on a **single topic**. Split large topics into several documents + in a shared directory and link between them rather than growing one giant file. +- Use fenced **code blocks with a language identifier** (```kotlin `, + ` ```bash `) so syntax highlighting works. +- Prefer Kotlin snippets over Java. +- When referencing code, link to the file with a **relative path** (e.g. + `../../../sentry/src/main/java/io/sentry/Sentry.java`) rather than pasting large excerpts + that fall out of date. Count the `../` from the document's own directory. +- Avoid pinning content to a specific SDK version or date unless it is genuinely + version-specific; keep docs evergreen. +- Cross-link related documents with relative links (e.g. + `[the ingestion pipeline](../../general/pipeline.md)`). + +### Structuring a feature document + +Most feature documents answer the same four questions, and following that order makes them +easier to compare and to keep current: + +1. **Surface area** — where and when the SDK collects the data. +2. **Collection** — how the SDK collects it. +3. **Format** — what the collected data looks like on the wire. +4. **Pipeline** — how the backend ingests, stores, and serves it. + +Do not restate (4) in every document. Describe the shared path once in +[general/pipeline.md](general/pipeline.md) and cover only the deviations a feature +introduces. Omit any of the four that a feature does not have, and keep each as high-level +as the topic allows so the document stays true for longer. + +### Diagrams with Mermaid + +- Prefer [Mermaid](https://mermaid.js.org/) for diagrams. It renders directly on GitHub + and lives in the document as text, so it versions and reviews like code. +- Embed a Mermaid diagram in a fenced block tagged `mermaid`: + + ````markdown + ```mermaid + flowchart LR + Event[SentryEvent] --> Processor[EventProcessors] + Processor --> Transport + Transport --> Sentry[(Sentry)] + ``` + ```` + +- For complex diagrams, include a link to the [Mermaid Live Editor](https://mermaid.live/) + so reviewers can iterate quickly. +- Fall back to static images (stored per the asset rules above) if mermaid is not practicable. + +## Adding a new document + +1. Pick the right top-level category (or introduce a new one and document it above). +2. Pick or create the topic directory below it. +3. Create the document, naming it for what distinguishes it from its siblings. +4. If the directory now holds several documents, add or update its `overview.md`. +5. If the document embeds assets, put them in an `assets/` folder next to it. diff --git a/develop-docs/feature/profiling/perfetto.md b/develop-docs/feature/profiling/perfetto.md new file mode 100644 index 00000000000..d4168f8f2a4 --- /dev/null +++ b/develop-docs/feature/profiling/perfetto.md @@ -0,0 +1,232 @@ +# Perfetto profiling on Android + +This document describes how continuous profiling works on Android when the SDK +captures traces through the OS-level [`android.os.ProfilingManager`](https://developer.android.com/reference/android/os/ProfilingManager) +API (available on API 35+), and how a captured **profile chunk** flows all the way +from the device to a downloadable profile in Sentry. + +## What Perfetto is + +[Perfetto](https://perfetto.dev/) is Google's tracing framework for Android and Linux, and +the tooling Android itself is instrumented with. Its +[callstack sampler](https://perfetto.dev/docs/getting-started/cpu-profiling) interrupts the +app at a fixed frequency, records the native and Java call stacks of the running threads, +and writes them to a binary `.pftrace` file (a serialized +[Perfetto protobuf](https://perfetto.dev/docs/reference/trace-packet-proto)). +Starting with Android 15, apps can request such traces at +runtime via `ProfilingManager` without root or `adb`, which is what makes on-device +continuous profiling possible. + +Useful Perfetto references: + +- Perfetto docs: https://perfetto.dev/docs/ +- CPU profiling with Perfetto: https://perfetto.dev/docs/getting-started/cpu-profiling +- Trace format (`TracePacket` proto): https://perfetto.dev/docs/reference/trace-packet-proto +- Perfetto UI (to open a downloaded `.pftrace`): https://ui.perfetto.dev/ + +## Pipeline overview + +Profile chunks travel the standard ingestion path described in +[general/pipeline.md](../../general/pipeline.md) — SDK envelope, +[Relay](https://develop.sentry.dev/ingestion/relay/) (Sentry's ingestion proxy), Kafka, a +monolith processing task, then storage and a read API. Read that first; the rest of this +document covers only where Perfetto deviates from it. + +The deviations are: + +- The envelope item carries **JSON and raw binary in one payload**, subdivided by a + `meta_length` header rather than base64-encoding the trace ([details](#envelope-format-and-the-meta_length-header)). +- Relay **converts** the Perfetto trace into the existing Sample v2 profile format, and + additionally **keeps the raw `.pftrace`** in the object store so it can be downloaded + later ([details](#relay-getsentryrelay)). + +```mermaid +flowchart TD + subgraph device["Android device — sentry-java"] + PM[android.os.ProfilingManager] + PP[PerfettoProfiler] + PCP[PerfettoContinuousProfiler] + PC[ProfileChunk] + ENV["Envelope item
[JSON metadata][raw .pftrace]
header: meta_length"] + PM --> PP --> PCP --> PC --> ENV + end + + subgraph relay["Relay (processing mode)"] + SPLIT[Split payload at meta_length] + CONV[Convert Perfetto → Sample v2] + OS1[Upload raw .pftrace to object store] + KAFKA[["Kafka topic: profiles
ProfileChunkKafkaMessage
(Sample v2 + attachment stored_id)"]] + SPLIT --> CONV --> KAFKA + SPLIT --> OS1 + end + + subgraph monolith["Monolith — getsentry/sentry"] + TASK[process_profile_task] + SYM[Symbolicate / deobfuscate] + VR[vroomrs: parse + normalize] + OS2[(Object store)] + SNUBA[(Snuba: function metrics)] + DB[(ProfileChunkAttachment row)] + TASK --> SYM --> VR + VR --> OS2 + VR --> SNUBA + TASK --> DB + end + + ENV -->|envelope| relay + KAFKA --> TASK + OS1 -.stored_id.-> DB + VROOM[getsentry/vroom
serve + merge flamegraphs] + OS2 --> VROOM + SNUBA --> VROOM +``` + +## SDK (getsentry/sentry-java) + +On API 35+, [`AndroidOptionsInitializer`](../../../sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java) +wires up `PerfettoContinuousProfiler` automatically. On older devices the SDK falls back +to the legacy `Debug`-based [`AndroidContinuousProfiler`](../../../sentry-android-core/src/main/java/io/sentry/android/core/AndroidContinuousProfiler.java), +gated by the `enableLegacyProfiling` option (manifest key +`io.sentry.profiling.enable-legacy-profiling`, defaults to `true`). Only **continuous +profiling** is supported on the Perfetto path — transaction-based and app-start profiling +are not. + +### Capturing chunks + +Continuous profiling emits a stream of independent [`ProfileChunk`](../../../sentry/src/main/java/io/sentry/ProfileChunk.java)s +rather than one profile per transaction. `PerfettoContinuousProfiler` drives a chained +loop: each chunk runs for `MAX_CHUNK_DURATION_MILLIS` (60s) via `PerfettoProfiler`, which +calls `ProfilingManager.requestProfiling(PROFILING_TYPE_STACK_SAMPLING, …)` at +`PROFILING_FREQUENCY_HZ` (101 Hz). When a chunk's trace file is ready, a new chunk starts, +so profiling runs continuously. + +A chunk keeps a stable `profilerId` across the session and a per-chunk `chunkId`. When the +OS produces the trace file, the profiler builds a `ProfileChunk` tagged with the Perfetto +content type: + +```kotlin +ProfileChunk.Builder(profilerId, chunkId, measurements, traceFile, timestamp, ProfileChunk.PLATFORM_ANDROID) + .setContentType(ProfileChunk.CONTENT_TYPE_PERFETTO) // "application/x-perfetto-trace" + .build() +``` + +The chunk is captured via `scopes.captureProfileChunk(...)` and sent as its own envelope +with item type [`SentryItemType.ProfileChunk`](../../../sentry/src/main/java/io/sentry/SentryItemType.java) +(wire name `profile_chunk`). + +### Envelope format and the `meta_length` header + +A legacy chunk base64-encodes its trace into the `ProfileChunk` JSON. A Perfetto chunk is +much larger, so [`SentryClient`](../../../sentry/src/main/java/io/sentry/SentryClient.java) instead +routes it through the new `SentryEnvelopeItem.fromPerfettoProfileChunk(...)` factory, which +avoids base64 by sending the raw binary alongside the JSON. + +The trick is a single envelope **item** whose payload concatenates the JSON metadata and +the raw `.pftrace` bytes with **no delimiter**: + +```text +[ProfileChunk JSON bytes][raw .pftrace binary bytes] +``` + +A new `meta_length` property on the [envelope item header](../../../sentry/src/main/java/io/sentry/SentryEnvelopeItemHeader.java) +tells the server where the JSON ends and the binary begins. The standard envelope item +structure (header line + newline + payload) is unchanged; `meta_length` simply subdivides +the payload: + +```text +{"type":"profile_chunk","content_type":"application/x-perfetto-trace","filename":"…","length":,"meta_length":} + +``` + +- `length` — total payload size (JSON + binary), as for any envelope item. +- `meta_length` — byte length of the JSON prefix. It is only known after the payload is + serialized, so the header computes it lazily (via a `Callable`) and omits the + field entirely for non-Perfetto items, keeping the change backward compatible. + +## Relay (getsentry/relay) + +In processing mode Relay: + +1. **Splits** the compound item payload at `meta_length` into `(metadata JSON, raw profile)` + and reads `content_type: "perfetto"` from the metadata. +2. **Converts** the binary Perfetto trace into the existing **Sample v2** profile JSON + format (`relay_profiling::expand_perfetto(...)`, backed by a checked-in subset of the + Perfetto protobuf definitions). +3. **Uploads** the raw `.pftrace` blob to object store (usecase `profiles`, keyed per + org/project, with an attachment-retention TTL). +4. **Produces** a `ProfileChunkKafkaMessage` to the `profiles` Kafka topic. The message + carries the expanded Sample v2 JSON as `payload` plus an `attachments` array, where each + attachment records: + - `name` (e.g. `profile.perfetto`), + - `content_type` (e.g. `application/x-perfetto-trace`), + - `stored_id` — the object store key of the uploaded raw blob. + +```json +{ + "organization_id": 1, + "project_id": 42, + "received": 1720000000, + "retention_days": 30, + "payload": "", + "attachments": [ + { + "name": "profile.perfetto", + "content_type": "application/x-perfetto-trace", + "stored_id": "" + } + ] +} +``` + +The monolith later uses `stored_id` to fetch the raw trace back. + +## Monolith (getsentry/sentry) + +`process_profile_task` (in `src/sentry/profiles/task.py`) consumes the `profiles` topic. +Because Relay already converted the trace to Sample v2, the task treats a Perfetto chunk +like any other: deobfuscate, hand it to `vroomrs` to parse and normalize +(`vroomrs.profile_chunk_from_json_str(...)`), compress and store it, and emit function +metrics to Snuba. + +The Perfetto-specific step is the last one: for each attachment on the message the task +persists a lightweight **`ProfileChunkAttachment`** row — `project_id`, `profiler_id`, +`chunk_id`, `name`, `content_type`, and the `stored_id` object store key. The row exists so +the raw trace can be downloaded by ID without exposing the `stored_id`. + +Flamegraphs themselves are served by `getsentry/vroom`, which reads the stored chunks and +the Snuba-indexed metadata and merges several chunks into one flamegraph. The endpoint +lives in the monolith and passes the request through. + +### Perfetto format dispatch (vroom / vroomrs) + +Older Android SDKs emit the legacy Android trace format tagged as a "faulty" `version=2`, +and the pipeline historically keyed off the platform rather than the version. To +distinguish legacy from Sample v2 chunks, `ProfileChunk` carries a dedicated `version` +field, and both `vroom` and `vroomrs` now dispatch on it instead of the platform: + +- Version `""` or `2.android-trace` → legacy Android trace format. +- Any other version → Sample v2. + +## Downloading a Perfetto profile + +The monolith exposes two feature-gated endpoints: + +- **List attachments** — `GET /organizations/{org}/profiling/chunk-attachments/` + (`sentry-api-0-organization-profiling-chunk-attachments`). Requires a `project` and + `profiler_id`; resolves the visible `chunk_id`s (same logic as the flamegraph) and returns + the matching `ProfileChunkAttachment` metadata. +- **Download** — `GET /projects/{org}/{project}/profiling/chunks/{profiler_id}/{chunk_id}/attachments/{attachment_id}/?download` + (`sentry-api-0-project-profiling-chunk-attachment`). The `?download` param is required; it + streams the raw blob back from object store via the stored `stored_id`. Access requires + the org's configured attachments role, analogous to generic event attachments. + +In the flamegraph UI, a toolbar button (added for continuous profiles when the feature is +enabled and at least one attachment exists) lists and provides a way to download these traces. + +## References + +- SDK: [sentry-java#5251](https://github.com/getsentry/sentry-java/pull/5251) — Android `ProfilingManager` (Perfetto) support +- Relay: [#5659](https://github.com/getsentry/relay/pull/5659), [#5932](https://github.com/getsentry/relay/pull/5932), [#6099](https://github.com/getsentry/relay/pull/6099), [#6102](https://github.com/getsentry/relay/pull/6102) — Perfetto parsing, pipeline, and object-store routing +- vroom: [#672](https://github.com/getsentry/vroom/pull/672) — version dispatch for Android trace profiles +- vroomrs: [#93](https://github.com/getsentry/vroomrs/pull/93) — accept Android profiles in Sample v2 format +- Monolith: [sentry#118029](https://github.com/getsentry/sentry/pull/118029) (chunk attachments + endpoints), [sentry#118071](https://github.com/getsentry/sentry/pull/118071) (flamegraph download button) diff --git a/develop-docs/general/pipeline.md b/develop-docs/general/pipeline.md new file mode 100644 index 00000000000..6cb0d87f696 --- /dev/null +++ b/develop-docs/general/pipeline.md @@ -0,0 +1,121 @@ +# Ingestion pipeline + +This document describes the path data takes from an SDK to a rendered view in Sentry. It +covers the parts of different payload types, like errors, transactions, logs, replays and +profile chunks. + +## Per data category + +Every payload takes the same four hops — SDK, Relay, a consumer in the monolith, and a read +API — but the topics, processing tasks, and stores differ per category. The diagrams below +show three of them; the hops themselves are described further down. + +### Errors + +```mermaid +flowchart LR + SDK["SDK
captures + batches"] -->|envelope| RELAY + RELAY["Relay
authenticate, normalize,
route"] -->|ingest-events| KAFKA[["Kafka"]] + RELAY -.->|attachments,
minidumps| OS[("Object store")] + KAFKA --> TASK["save_event task"] + TASK --> SYM["Symbolicator
symbolicate, deobfuscate"] + SYM --> TASK + TASK --> NS[("Nodestore
full event body")] + TASK --> SNUBA[("Snuba
searchable columns")] + TASK --> PG[("Postgres
Group / GroupHash rows")] + NS --> READ["Read path
monolith API"] + SNUBA --> READ + PG --> READ + OS --> READ +``` + +### Transactions + +```mermaid +flowchart LR + SDK["SDK
captures spans"] -->|envelope| RELAY + RELAY["Relay
normalize, dynamic sampling,
metric extraction"] -->|ingest-transactions| KAFKA[["Kafka"]] + KAFKA --> CONSUMER["Transaction consumer"] + CONSUMER --> SNUBA[("Snuba
transactions + spans")] + CONSUMER --> NS[("Nodestore
full transaction body")] + SNUBA --> READ["Read path
monolith API"] + NS --> READ +``` + +### Profile chunks + +```mermaid +flowchart LR + SDK["SDK
captures profile chunks"] -->|envelope| RELAY + RELAY["Relay
convert Perfetto → Sample v2"] -->|profiles| KAFKA[["Kafka"]] + RELAY -.->|raw .pftrace blob| OS[("Object store")] + KAFKA --> TASK["process_profile_task"] + TASK --> VRS["vroomrs
parse + normalize"] + VRS --> OS + VRS --> SNUBA[("Snuba
function metrics")] + TASK --> PG[("Postgres
ProfileChunkAttachment rows")] + OS --> VROOM["vroom
serve + merge flamegraphs"] + SNUBA --> VROOM + VROOM --> READ["Read path
monolith API"] + PG --> READ +``` + +## The hops + +### 1. SDK + +The SDK captures data and wraps it in an [envelope](https://develop.sentry.dev/sdk/data-model/envelopes/): +a JSON header followed by one or more items, each with its own header declaring a `type`, +a `length`, and optionally a `content_type`. The envelope is POSTed to the project's +`/api/{project_id}/envelope/` endpoint. + +The item `type` is what routes the payload through everything downstream, so adding a new +kind of data means adding an item type, not a new endpoint. Item payloads are usually JSON; +binary payloads are allowed and are preferable to base64-encoding a large blob into JSON. + +### 2. Relay + +[Relay](https://github.com/getsentry/relay) is Sentry's ingestion proxy — it sits between +the SDK and the rest of the infrastructure and is the first service to inspect a payload. +See the [Relay chapter in develop docs](https://develop.sentry.dev/ingestion/relay/) for +the full picture. + +Relay authenticates the DSN, applies quotas and rate limits, filters and normalizes the +payload, and forwards it. Two behaviours matter when designing a new payload type: + +- Relay may **convert** a payload into a different format before publishing it, so the + format the SDK sends and the format the backend consumes are not necessarily the same. + Whatever Relay publishes is the contract every downstream service depends on. +- Relay runs in two modes. Only **processing mode** (the one Sentry operates) talks to + Kafka and the object store; a self-hosted Relay in proxy mode just forwards envelopes + upstream. + +Relay publishes to a **Kafka topic per data category**. Payloads too large to sit +comfortably in a Kafka message are uploaded to the **object store** instead, and the +message carries a reference to the stored blob rather than the bytes themselves. Event +attachments (minidumps, screenshots, view hierarchies) work this way, and so does the raw +`.pftrace` blob of a Perfetto profile chunk: Relay uploads the trace and puts only its +`stored_id` object store key on the Kafka message. + +### 3. Consumers and processing + +Each topic is consumed by the monolith ([getsentry/sentry](https://github.com/getsentry/sentry)), +which runs a processing task per message. This is where the work that needs Sentry-side +state happens — symbolication and deobfuscation against uploaded debug files, enrichment, +normalization, and quota accounting. + +A task typically writes to more than one store: + +- **Object store** — the payload itself, compressed. Cheap to keep, not queryable. +- **Snuba** — the columns that need to be searched, aggregated, or listed. +- **Postgres** — small metadata rows that the API needs to resolve a request, for example + a row per stored blob so it can be fetched by ID instead of by exposing its storage key. + +### 4. Read path + +The monolith serves the API endpoints. For some categories it does the work itself; for +others it authorizes the request and proxies it to a dedicated service that owns the +heavy read logic. Either way the endpoint is the public surface, and the storage keys and +internal services stay behind it. + +See [feature/profiling/perfetto.md](../feature/profiling/perfetto.md) for a worked example.