From 09bbccbdda70a942a2b682a42e1cb39bbe864609 Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Fri, 21 Aug 2026 12:35:14 +0200 Subject: [PATCH] feat(observability): trace, measure and log Sturnus without shipping content MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit OpenTelemetry traces and metrics, plus structured logging shaped for Loki. Squashed to a single commit on top of main. Two earlier commits on this branch carried fixtures with the literal shape of a credential -- an AWS access key id and a Discord bot token -- which secret scanning detects, correctly, whether or not either string opens anything. Both are now assembled from parts at import: identical at runtime, so the tests still prove the redaction catches those exact shapes, with nothing in the file for a scanner or a reader to mistake for a credential. Rewriting the tip alone would not have helped; a scan reads every commit in the pull request, so the literals had to leave the history. **Redaction is an allowlist, not a denylist.** Unregistered field names are dropped, `bytes` is dropped as a class (audio and wrapped data keys are always bytes, so that closes the highest-value leak by construction), strings are pattern-scrubbed and capped, and every replacement is visible (`«redacted:discord_token»`) rather than silent. **The leak this branch closed.** With `STURNUS_LOG_LEVEL=DEBUG` the Discord voice `secret_key` reached the logs and would have reached Loki. Reproduced: discord/ext/voice_recv/gateway.py:57 log.debug("Received op %s: \n%s", op, pformat(data)) -> {'mode': ..., 'secret_key': [1, 2, ..., 32], 'ssrc': ...} Not a redaction failure: `extra={"secret_key": ...}` was dropped, and so was `extra={"voice_ready": {...}}`. The key arrived already formatted into a third-party logger's *message string*, which a field allowlist cannot touch. The cause was `root.setLevel(min(resolved_level, resolved_third_party))`, which put 28 unclamped third-party loggers at DEBUG -- every logger absent from the enumerated clamp list inherits from root, and one list cannot be complete about libraries it does not import. That `min()` was never load-bearing (Python checks the *originating* logger's effective level on propagation, never root's), so removing it costs nothing and closes the hole. `THIRD_PARTY_FLOOR` replaces the enumeration as the structural half of the fix, and `tests/observability/test_third_party_log_floor.py` asserts the property over `logging.Logger.manager.loggerDict` rather than over a list. `discord.voice_state` is pinned at INFO rather than silenced: DEBUG is where the leak lives, but INFO is the connect narrative -- handshake attempts, endpoint, close codes, resume -- and it is the evidence base for telling the three capture failures apart. Pinning needed a `NEVER_ABOVE` counterpart: `NEVER_BELOW` is applied as `max(level, floor)` and can only ever make a logger *quieter*, so an INFO entry there was a no-op at the deployed `WARNING` default -- the logger still ended at WARNING and the line the entry was written to keep was still gone. **The metrics answer questions this project actually had.** `sturnus.transcription.decoded_seconds` divided by wall time is the real-time factor: a job that "finished" a 100-minute recording in 43 seconds reports an impossible, unmistakable number -- where the symptom everyone saw, an empty transcript, looked exactly like a participant who never spoke and was misread as one for a day. `position_seconds`, `total_seconds` and `seconds_since_progress` are **observable instruments, not synchronous gauges**: a gauge only changes when a call site sets it, so a decoder that wedges freezes it and `seconds_since_progress` -- the actual alert -- could never grow. The SDK calls these callbacks once per export interval instead, which is also what lets them emit nothing at all while the worker is idle, so the series goes stale rather than reporting a finished job's numbers forever. The stall clock starts before the library call, since the collapse happened inside feature extraction. Labels are `model` only: no session, job, guild or user id, which would be unbounded cardinality and a record of who was in a voice channel when, kept for as long as the metric store keeps anything. `sturnus.job.outcome` reported `done` for every failed job, because `process_one` returns True after `queue.fail(...)` exactly as it does after `queue.complete(...)` -- the boolean means "work was attempted", never "work succeeded", and a metric that reports failures as successes is worse than no metric because it will be believed. The label is now recorded by the transitions that decide a job's terminal state, and `crashed` is the one the worker loop still owns. **Rebased onto #48, which rewrote the same method.** The transcription mechanics are main's: the model is handed the gated speech concatenated, `clip_timestamps` is re-expressed on that timeline, and every returned segment goes back onto the recording's through `_on_the_original_timeline(segment.start, segment.end, segment.seek, ...)`, where `Segment.seek` -- the encoder window the segment was decoded from -- is what names its clip. The observability is re-expressed on top: the segment generator is drained by a loop and not a comprehension, so `TRANSCRIPTION_PROGRESS` sees each segment as it arrives rather than only after the job has already finished. Progress is reported on the *concatenated* timeline -- `advance(segment.end)` and not the restored end -- because the denominator is `duration_after_vad`, which since #48 is the concatenated speech. Reporting a restored end against it would put a job that had decoded its first clip at several hundred percent. `telemetry.TranscriptionProgress` and `docs/operations.md` § 7.5 say so; they described the whole file before. Two call sites arrived from main that the merge could not have seen, both of them what `tests/test_logging_discipline.py` R2 and R6 forbid, and for the reason R6 exists: `%s` on an exception prints `str(exc)` verbatim into the message `observability.scrub_event` forwards to Sentry. `RecordingService._report_silent_audio` (#48) was three `log.warning` calls, two interpolating an exception and one passing `display_name`; it is now `speaker.audio_silent`, `speaker.silent_warning_failed` and `speaker.silent_record_failed` through `log_event`/`log_exception`, with `display_name` gone -- the channel message renders the mention, and the operator has the id. `RequeueConfirmView._disable` (#49) is now `queue.view_disable_failed` the same way. --- charts/sturnus/values.yaml | 56 +- docs/first-deployment.md | 29 + docs/operations.md | 485 ++++++++++- docs/verification/end-to-end-checklist.md | 65 ++ migrations/env.py | 59 +- pyproject.toml | 24 + src/sturnus/application/publishing.py | 20 +- src/sturnus/application/recording.py | 170 +++- src/sturnus/application/recovery.py | 46 +- src/sturnus/application/retention.py | 11 +- src/sturnus/application/worker.py | 122 ++- src/sturnus/config.py | 59 +- src/sturnus/entrypoints/bot.py | 88 +- src/sturnus/entrypoints/link.py | 44 +- src/sturnus/entrypoints/worker.py | 223 ++++- src/sturnus/infrastructure/db/queue.py | 77 +- .../infrastructure/discord/audio_cog.py | 27 +- src/sturnus/infrastructure/discord/client.py | 351 ++++++-- .../infrastructure/discord/decoding.py | 12 +- .../infrastructure/discord/queue_cog.py | 12 +- src/sturnus/infrastructure/discord/sink.py | 23 + src/sturnus/infrastructure/discord/voice.py | 194 ++++- .../infrastructure/documents/outline.py | 90 +- .../infrastructure/documents/outline_oauth.py | 56 +- src/sturnus/infrastructure/health.py | 23 +- src/sturnus/infrastructure/linkserver.py | 53 +- src/sturnus/infrastructure/observability.py | 29 + src/sturnus/infrastructure/speech_gate.py | 10 + src/sturnus/infrastructure/telemetry.py | 809 ++++++++++++++++++ src/sturnus/infrastructure/traced.py | 346 ++++++++ src/sturnus/infrastructure/whisper.py | 232 ++++- src/sturnus/observability/__init__.py | 38 + src/sturnus/observability/events.py | 271 ++++++ src/sturnus/observability/fields.py | 329 +++++++ src/sturnus/observability/redaction.py | 352 ++++++++ src/sturnus/observability/setup.py | 629 ++++++++++++++ tests/__init__.py | 12 + tests/application/test_worker.py | 6 +- tests/infrastructure/discord/test_client.py | 228 ++++- .../discord/test_voice_adapter.py | 108 ++- tests/infrastructure/test_health.py | 17 +- tests/infrastructure/test_migrations.py | 14 + tests/infrastructure/test_queue.py | 120 +++ tests/infrastructure/test_telemetry.py | 339 ++++++++ tests/infrastructure/test_traced_ports.py | 240 ++++++ tests/infrastructure/test_whisper.py | 496 ++++++++++- tests/observability/__init__.py | 0 tests/observability/test_no_payload_leaks.py | 339 ++++++++ .../observability/test_package_boundaries.py | 81 ++ tests/observability/test_redaction.py | 258 ++++++ tests/observability/test_setup.py | 371 ++++++++ .../test_third_party_log_floor.py | 473 ++++++++++ tests/test_logging_discipline.py | 292 +++++++ uv.lock | 101 +++ 54 files changed, 8657 insertions(+), 302 deletions(-) create mode 100644 src/sturnus/infrastructure/telemetry.py create mode 100644 src/sturnus/infrastructure/traced.py create mode 100644 src/sturnus/observability/__init__.py create mode 100644 src/sturnus/observability/events.py create mode 100644 src/sturnus/observability/fields.py create mode 100644 src/sturnus/observability/redaction.py create mode 100644 src/sturnus/observability/setup.py create mode 100644 tests/__init__.py create mode 100644 tests/infrastructure/test_telemetry.py create mode 100644 tests/infrastructure/test_traced_ports.py create mode 100644 tests/observability/__init__.py create mode 100644 tests/observability/test_no_payload_leaks.py create mode 100644 tests/observability/test_package_boundaries.py create mode 100644 tests/observability/test_redaction.py create mode 100644 tests/observability/test_setup.py create mode 100644 tests/observability/test_third_party_log_floor.py create mode 100644 tests/test_logging_discipline.py diff --git a/charts/sturnus/values.yaml b/charts/sturnus/values.yaml index b0c9b4b..b8e858a 100644 --- a/charts/sturnus/values.yaml +++ b/charts/sturnus/values.yaml @@ -103,10 +103,62 @@ commonEnv: # errors therefore sets the Secret key to the empty string rather than # leaving it out. See docs/operations.md section 1.4. # - # Names the deployment in Sentry's environment filter. Set per cluster. - # Not a credential and not secret -- it stays here. + # Names the deployment in Sentry's environment filter, AND supplies + # OpenTelemetry's `deployment.environment.name` resource attribute -- + # `sturnus.config.OtelSettings.environment` reads this same variable + # rather than introducing a second one. One string, so the environment + # filter in Sentry and the one in Grafana can never disagree. Set per + # cluster. Not a credential and not secret -- it stays here. STURNUS_SENTRY_ENVIRONMENT: "production" + # Traces and metrics (sturnus/infrastructure/telemetry.py). Empty is the + # off switch and the default: with no endpoint, no OpenTelemetry provider + # is constructed at all, so every span in the codebase is a + # NonRecordingSpan (~0.1us) and every metric call is a no-op. Nothing + # connects, nothing retries, and no export failure is ever logged. + # + # In this cluster the value is the Grafana Alloy OTLP receiver, which + # fans out to Tempo for traces and to the metric store: + # + # STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT: "http://alloy-receiver.grafana.svc:4318" + # + # HTTP (4318), not gRPC (4317). Alloy accepts both; the HTTP exporter + # avoids pulling in grpcio. The path is appended automatically, so give + # the base URL with no /v1/traces suffix. + # + # Not a credential, and note that this is a different judgement from the + # Sentry DSN above rather than the same one: an OTLP endpoint is a + # service address that grants nobody anything, so publishing it in the + # GitOps repository costs nothing. The DSN moved into the Secret because + # that repository is public and a DSN found there lets anyone fill the + # project with events -- an argument about who can *write* somewhere, + # which an endpoint does not carry. See docs/operations.md section 1.4. + STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT: "" + + # Logging (sturnus/observability/setup.py). `json` is one object per line + # on stdout, which is what alloy-logs scrapes into Loki; `console` is the + # human-readable form and is selected automatically when stdout is a TTY. + STURNUS_LOG_FORMAT: "json" + # Applies to Sturnus's own loggers ONLY, and DEBUG here is safe: Sturnus's + # own DEBUG lines are held to ids, counts and durations by the same field + # registry that governs INFO. + STURNUS_LOG_LEVEL: "INFO" + # Everything that is not Sturnus. Raising this to DEBUG does nothing: the + # value is raised to setup.THIRD_PARTY_FLOOR (INFO) before anything is + # configured, and the process says so in one WARNING line at startup + # (event log.level_clamped) so nobody spends an hour thinking the + # variable is unwired. + # + # That floor is a security control, not tidiness. botocore logs the SigV4 + # signature at DEBUG, and discord.ext.voice_recv logs the Discord voice + # secret key at DEBUG from two different modules (`.reader` and + # `.gateway`) plus raw packet bytes; discord.http logs whole REST + # response bodies. A production values.yaml must not be able to put any + # of that into Loki. Lowering it below INFO is a code change in + # sturnus/observability/setup.py, deliberately -- see docs/operations.md + # section 7.2. + STURNUS_LOG_THIRD_PARTY_LEVEL: "WARNING" + # Every component's health/readiness server listens on this container port # (see sturnus/infrastructure/health.py); the container image is the same # for all three, so the port is shared here rather than repeated per component. diff --git a/docs/first-deployment.md b/docs/first-deployment.md index 7fe236c..b22ddcd 100644 --- a/docs/first-deployment.md +++ b/docs/first-deployment.md @@ -190,6 +190,35 @@ The worker runs the database migrations at startup, and the bot and link wait for the tables. So `sturnus-worker` coming up healthy is the gate — if it does not, the other two never will. +**Two things to look at once the pods are up**, both one command each and +both easier now than after the first real meeting: + +```bash +# 1. Nothing outside sturnus.* may log at DEBUG. This must print nothing. +# If it prints anything, stop and read operations.md section 7.2 -- +# third-party DEBUG is how the Discord voice secret key reaches Loki. +kubectl -n sturnus logs deploy/sturnus-worker \ + | jq -r 'select(.level=="DEBUG" and (.logger | startswith("sturnus") | not))' + +# 2. Telemetry is OFF by default and looks identical to "healthy" when it +# is misconfigured -- every dashboard shows a flat zero either way. +kubectl -n sturnus logs deploy/sturnus-worker \ + | jq -r 'select(.event=="telemetry.enabled")' +``` + +The second prints nothing at all until `STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT` +is set — the chart ships it empty, which is a deliberate off switch rather +than an oversight, and with it empty no OpenTelemetry provider is built, +nothing connects and nothing retries. Set it to +`http://alloy-receiver.grafana.svc:4318` when you want traces and metrics, +then follow section 7.7 of `operations.md`, which is the only way to tell a +*misconfigured* endpoint from a working one. + +A third line worth recognising if you see it: `"event":"log.level_clamped"` +means someone set `STURNUS_LOG_THIRD_PARTY_LEVEL` below `INFO` and the +process raised it back. The variable is not broken; it is floored on +purpose. + ## 7. Configure the guild In Discord, as an administrator: diff --git a/docs/operations.md b/docs/operations.md index 96f577d..f387339 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -59,9 +59,15 @@ the Kubernetes `Secret` rather than from plain manifest text (see section | `STURNUS_OUTLINE_BASE_URL` | **yes** | no | Base URL of the Outline instance. The bot needs it only to build the authorization URL `/link` sends a user's browser to; it never calls Outline's API itself. | | `STURNUS_OUTLINE_CLIENT_ID` | **yes** | no | OAuth client id of the Sturnus application registered in Outline. Public by design — it travels in the query string of the authorization URL every user's browser opens. | | `STURNUS_OUTLINE_REDIRECT_URI` | **yes** | no | The callback URL that authorization returns to. Must be the same value `sturnus-link` is given, and must actually route to `link` — see section 1.5. | -| `STURNUS_HEALTH_PORT` | `8080` | no | Port the `/healthz`, `/readyz`, `/metrics`, `/version` HTTP endpoints listen on. | +| `STURNUS_HEALTH_PORT` | `8080` | no | Port the `/healthz`, `/readyz`, `/metrics`, `/version` HTTP endpoints listen on. `/metrics` answers **`501 Not Implemented`**: metrics are *pushed* over OTLP, not scraped — see section 7. | | `STURNUS_SENTRY_DSN` | unset | no | Sentry DSN for error reporting. Empty disables it entirely: `sentry_sdk.init()` is never called, so no instrumentation is installed and the process runs exactly as it does without Sentry. Supplied through the `Secret`, and required to be present even when blank; see section 1.4 for why a value that is not a credential is stored like one. | -| `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Value Sentry files events under in its environment filter. Ignored when no DSN is set. | +| `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Names the deployment in Sentry's environment filter **and** supplies OpenTelemetry's `deployment.environment.name`. One variable for both on purpose — `OtelSettings.environment` reads this exact name rather than adding a second one, so the environment filter in Sentry and the one in Grafana can never disagree. | +| `STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT` | unset | no | Base URL of an OTLP/HTTP receiver — in this cluster `http://alloy-receiver.grafana.svc:4318`. Unset or empty — the chart's default — disables traces and metrics entirely: no provider is constructed, so every span is a no-op, nothing connects, nothing retries, and no export failure is ever logged. Give the base URL with no `/v1/traces` suffix; the exporters append their own paths. Not a credential; see section 1.4. | +| `STURNUS_OTEL_TRACES_SAMPLE_RATIO` | `1.0` | no | Fraction of traces sampled. `1.0` is correct today and the arithmetic is in `sturnus/infrastructure/telemetry.py`: the worker processes one job at a time and each is minutes of CPU work, the bot opens a handful of sessions per guild per day, and the packet path emits no spans at all. This exists as a valve for a future high-volume path. Note that job outcome is *also* an unsampled counter, so sampling can never be why a failed job is invisible. | +| `STURNUS_OTEL_METRIC_EXPORT_INTERVAL_SECONDS` | `60.0` | no | How often metrics are pushed to the OTLP endpoint. | +| `STURNUS_LOG_LEVEL` | `INFO` | no | Level for **Sturnus's own** loggers only, and safe to set to `DEBUG` in production: Sturnus's own DEBUG lines are held to ids, counts, sizes and durations by the same field registry that governs INFO. It cannot turn up a third-party logger — see section 7.2 for why that restriction is a security control rather than tidiness. | +| `STURNUS_LOG_THIRD_PARTY_LEVEL` | `WARNING` | no | Level for every other library. **Raised to `INFO` if you set it lower**, and the per-logger floors in section 7.2 clamp several libraries tighter still; no value here undercuts either. A clamped value is not ignored silently — the process logs one `log.level_clamped` line at startup saying it happened. | +| `STURNUS_LOG_FORMAT` | `json` (auto) | no | `json` — one object per line on stdout, which is what `alloy-logs` scrapes into Loki — or `console` for a human at a terminal. Defaults to `console` when stdout is a TTY and `json` otherwise, so local development needs no setting. Both formats share the same redaction filter; the choice is presentation only. | The bot has no `STURNUS_OUTLINE_CLIENT_SECRET`, and that is not an oversight: building an authorization URL needs only the public client id, @@ -89,9 +95,15 @@ in the bot would read it. | `STURNUS_WORK_DIR` | `/tmp` | no | Scratch directory the encrypted recording is downloaded and decrypted into before transcription. It must be large enough for the biggest single recording — the chart sizes the corresponding volume with `worker.tmpSizeLimit`. | | `STURNUS_MAX_JOB_ATTEMPTS` | `3` | no | How many failed attempts a job gets before `JobQueue.fail` marks it `dead`. See section 5 for what a `dead` job means for the rest of its session. | | `STURNUS_JOB_LEASE_SECONDS` | `1800.0` | no | How long a claimed job may stay `running` before `JobQueue.claim` reclaims it for another worker. It is generous on purpose: it must exceed the longest plausible transcription, or a still-running job gets picked up a second time. | -| `STURNUS_HEALTH_PORT` | `8080` | no | Port the `/healthz`, `/readyz`, `/metrics`, `/version` HTTP endpoints listen on. | +| `STURNUS_HEALTH_PORT` | `8080` | no | Port the `/healthz`, `/readyz`, `/metrics`, `/version` HTTP endpoints listen on. `/metrics` answers **`501 Not Implemented`**: metrics are *pushed* over OTLP, not scraped — see section 7. | | `STURNUS_SENTRY_DSN` | unset | no | Sentry DSN for error reporting. Empty disables it entirely: `sentry_sdk.init()` is never called, so no instrumentation is installed and the process runs exactly as it does without Sentry. Supplied through the `Secret`, and required to be present even when blank; see section 1.4 for why a value that is not a credential is stored like one. | -| `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Value Sentry files events under in its environment filter. Ignored when no DSN is set. | +| `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Names the deployment in Sentry's environment filter **and** supplies OpenTelemetry's `deployment.environment.name`. One variable for both on purpose — `OtelSettings.environment` reads this exact name rather than adding a second one, so the environment filter in Sentry and the one in Grafana can never disagree. | +| `STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT` | unset | no | Base URL of an OTLP/HTTP receiver — in this cluster `http://alloy-receiver.grafana.svc:4318`. Unset or empty — the chart's default — disables traces and metrics entirely: no provider is constructed, so every span is a no-op, nothing connects, nothing retries, and no export failure is ever logged. Give the base URL with no `/v1/traces` suffix; the exporters append their own paths. Not a credential; see section 1.4. | +| `STURNUS_OTEL_TRACES_SAMPLE_RATIO` | `1.0` | no | Fraction of traces sampled. `1.0` is correct today and the arithmetic is in `sturnus/infrastructure/telemetry.py`: the worker processes one job at a time and each is minutes of CPU work, the bot opens a handful of sessions per guild per day, and the packet path emits no spans at all. This exists as a valve for a future high-volume path. Note that job outcome is *also* an unsampled counter, so sampling can never be why a failed job is invisible. | +| `STURNUS_OTEL_METRIC_EXPORT_INTERVAL_SECONDS` | `60.0` | no | How often metrics are pushed to the OTLP endpoint. | +| `STURNUS_LOG_LEVEL` | `INFO` | no | Level for **Sturnus's own** loggers only, and safe to set to `DEBUG` in production: Sturnus's own DEBUG lines are held to ids, counts, sizes and durations by the same field registry that governs INFO. It cannot turn up a third-party logger — see section 7.2 for why that restriction is a security control rather than tidiness. | +| `STURNUS_LOG_THIRD_PARTY_LEVEL` | `WARNING` | no | Level for every other library. **Raised to `INFO` if you set it lower**, and the per-logger floors in section 7.2 clamp several libraries tighter still; no value here undercuts either. A clamped value is not ignored silently — the process logs one `log.level_clamped` line at startup saying it happened. | +| `STURNUS_LOG_FORMAT` | `json` (auto) | no | `json` — one object per line on stdout, which is what `alloy-logs` scrapes into Loki — or `console` for a human at a terminal. Defaults to `console` when stdout is a TTY and `json` otherwise, so local development needs no setting. Both formats share the same redaction filter; the choice is presentation only. | Whisper's device and compute type are deliberately *not* environment-driven: the worker constructs `WhisperEngine` with `"cpu"` and `int8_float32` @@ -126,7 +138,13 @@ environment variables — see section 4.1. | `STURNUS_OUTLINE_REDIRECT_URI` | **yes** | no | The callback URL, repeated here because the token exchange sends it again for verification. It must match the bot's value exactly — see section 1.5. | | `STURNUS_HEALTH_PORT` | `8080` | no | Port the `/healthz`, `/readyz` and `/oauth/callback` routes are served on. | | `STURNUS_SENTRY_DSN` | unset | no | Sentry DSN for error reporting. Empty disables it entirely: `sentry_sdk.init()` is never called, so no instrumentation is installed and the process runs exactly as it does without Sentry. Supplied through the `Secret`, and required to be present even when blank; see section 1.4 for why a value that is not a credential is stored like one. | -| `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Value Sentry files events under in its environment filter. Ignored when no DSN is set. | +| `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Names the deployment in Sentry's environment filter **and** supplies OpenTelemetry's `deployment.environment.name`. One variable for both on purpose — `OtelSettings.environment` reads this exact name rather than adding a second one, so the environment filter in Sentry and the one in Grafana can never disagree. | +| `STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT` | unset | no | Base URL of an OTLP/HTTP receiver — in this cluster `http://alloy-receiver.grafana.svc:4318`. Unset or empty — the chart's default — disables traces and metrics entirely: no provider is constructed, so every span is a no-op, nothing connects, nothing retries, and no export failure is ever logged. Give the base URL with no `/v1/traces` suffix; the exporters append their own paths. Not a credential; see section 1.4. | +| `STURNUS_OTEL_TRACES_SAMPLE_RATIO` | `1.0` | no | Fraction of traces sampled. `1.0` is correct today and the arithmetic is in `sturnus/infrastructure/telemetry.py`: the worker processes one job at a time and each is minutes of CPU work, the bot opens a handful of sessions per guild per day, and the packet path emits no spans at all. This exists as a valve for a future high-volume path. Note that job outcome is *also* an unsampled counter, so sampling can never be why a failed job is invisible. | +| `STURNUS_OTEL_METRIC_EXPORT_INTERVAL_SECONDS` | `60.0` | no | How often metrics are pushed to the OTLP endpoint. | +| `STURNUS_LOG_LEVEL` | `INFO` | no | Level for **Sturnus's own** loggers only, and safe to set to `DEBUG` in production: Sturnus's own DEBUG lines are held to ids, counts, sizes and durations by the same field registry that governs INFO. It cannot turn up a third-party logger — see section 7.2 for why that restriction is a security control rather than tidiness. | +| `STURNUS_LOG_THIRD_PARTY_LEVEL` | `WARNING` | no | Level for every other library. **Raised to `INFO` if you set it lower**, and the per-logger floors in section 7.2 clamp several libraries tighter still; no value here undercuts either. A clamped value is not ignored silently — the process logs one `log.level_clamped` line at startup saying it happened. | +| `STURNUS_LOG_FORMAT` | `json` (auto) | no | `json` — one object per line on stdout, which is what `alloy-logs` scrapes into Loki — or `console` for a human at a terminal. Defaults to `console` when stdout is a TTY and `json` otherwise, so local development needs no setting. Both formats share the same redaction filter; the choice is presentation only. | `link` holds no Discord token, no S3 credentials and no master key — by construction, not by omission. It is the only publicly reachable component, @@ -191,6 +209,19 @@ Two consequences follow, both deliberate: `STURNUS_SENTRY_ENVIRONMENT` stays in `commonEnv`: it is a label, not a key. +So do the OpenTelemetry and logging variables, and for the *first* of the +two reasons above rather than the second: `STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT` +is a cluster-internal service address, and `STURNUS_LOG_LEVEL`, +`STURNUS_LOG_THIRD_PARTY_LEVEL` and `STURNUS_LOG_FORMAT` are enum-shaped +words. None of them authorises anything, so finding one in a public +repository costs nothing — which is precisely the test the DSN failed. +`STURNUS_LOG_THIRD_PARTY_LEVEL` is worth one extra sentence because it +*sounds* like a lever on a security control and is not: any value below +`INFO` is raised to `INFO` before anything is configured, and the per-logger +floors in section 7.2 are applied after that. Finding `DEBUG` there in a +public repository would tell a reader that somebody tried, not that they +succeeded. + `STURNUS_DATABASE_URL` is the one entry on that list that is not typed `SecretStr` in the code — it is a plain `str`, because it is a connection string rather than a bare credential. Treat it as a secret regardless: it @@ -873,3 +904,447 @@ consent names the superseded version stop being recorded mid-session, and `/consent grant` under the new version is what puts them back. Removing the role by hand is not required for a hard cutover, and doing so only costs the affected members a second step when they re-consent. + +## 7. Observability + +Three retained stores hold a copy of what Sturnus emits, and all three are +reachable by a wider audience than `kubectl`: + +| Store | What reaches it | How | +|---|---|---| +| **Loki** | pod stdout, one JSON object per line | `alloy-logs` runs as a DaemonSet and scrapes container stdout. Sturnus ships no logs itself. | +| **Tempo** | spans | pushed over OTLP/HTTP to `alloy-receiver.grafana.svc:4318`, which fans out. | +| **Sentry** | errors | `sturnus/infrastructure/observability.py`. | + +Because Alloy does the shipping, "optimising for Loki" means changing what +Sturnus *prints*, not adding a log shipper. + +### 7.1 One registry decides what may be emitted + +`sturnus/observability/fields.py` holds a closed list of field names, and +`sturnus/observability/redaction.py` holds a single scrubbing function that +log lines, span attributes and metric labels all pass through. **Adding a +field is one edit, in one file, and it shows up in review as "we decided to +put this in Loki, Tempo and Grafana" — which is what it is.** + +The list is an allowlist that gets *rebuilt*, not a denylist that gets +stripped, so an unregistered name is dropped rather than forwarded. The +failure mode is a missing panel in Grafana, never a transcript in Tempo. +Read that file before adding to it. + +Three things are excluded and the reasons differ: + +- **Transcript text, audio bytes, display names, tokens and keys** — the + content Spec 15 and the blocking gate in + `docs/verification/end-to-end-checklist.md` exist to protect. `bytes` are + never rendered at all, whatever they are called, which closes raw PCM, + Opus frames and wrapped keys as a class rather than by name. +- **The S3 object key** — its format is + `sessions/{session_id}/speakers/{discord_user_id}.enc`, so it *embeds* a + user id, and both halves are separately loggable. Logging the key is pure + duplication with a wider blast radius; `audio_key()` reconstructs it. +- **`discord_user_id` and `external_user_id` in spans and metrics** — these + *are* logged, because "did this person's `/audio delete` actually erase + their recordings" is a compliance question that cannot be answered + without them. They are kept out of Tempo because a user id joined to a + session id and precise timestamps in a searchable, trace-indexed store is + a record of who was in which voice channel when, which is a different + artifact from a line in `kubectl logs`. Nothing is lost: `sturnus.session_id` + on the span joins to the row that has the user id. + +Exception *messages* never travel unless their class is `OSError` or +subclasses `DiagnosticSafeError` — one rule, shared with Sentry, in +`redaction.SAFE_MESSAGE_TYPES`, which +`sturnus.infrastructure.observability.SAFE_VALUE_TYPES` aliases rather than +restates. What you get instead is the exception type and a full traceback, +which locates the failure without carrying whatever the message happened to +interpolate. + +Sentry is *narrower* than that shared rule rather than equal to it, and the +difference is deliberate: it rebuilds an `OSError` message from `errno` +instead of reading the exception's own string (section 5 gives the reason). +The shared list is the gate — a class it does not vouch for says nothing +anywhere — and each transport may then say less. Narrower is always +allowed; wider is what the alias makes impossible. + +### 7.2 What is safe to raise, and what deliberately is not + +**The short version, for 3am.** + +| You want | Set | Effect | +|---|---|---| +| More detail from Sturnus | `STURNUS_LOG_LEVEL=DEBUG` | Safe. Do it. Restart the affected Deployment. | +| More detail from `discord.py`, `botocore`, `aiohttp`, SQLAlchemy | `STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` | **Does nothing.** The value is raised to `INFO`, and the process logs one `log.level_clamped` line saying so. | +| Less noise from everything | `STURNUS_LOG_THIRD_PARTY_LEVEL=ERROR` | Works. Quieter than the floor is always allowed. | +| Third-party `DEBUG` anyway | — | A code change to `THIRD_PARTY_FLOOR` in `sturnus/observability/setup.py`, on a non-production deployment. There is no environment variable for it, on purpose. | + +`STURNUS_LOG_LEVEL` applies to `logging.getLogger("sturnus")` and nothing +else. Turning it up is safe because Sturnus's own DEBUG output goes through +the same closed field registry as its INFO output (section 7.1): it can +carry ids, counts, sizes and durations, and structurally cannot carry a +transcript or a key. + +Everything that is not Sturnus is held down by two mechanisms: + +1. **`THIRD_PARTY_FLOOR`** — the level below which no logger outside + `sturnus.*` may go, including the root logger that every unnamed + third-party logger inherits from. Today it is `INFO`. +2. **`NEVER_BELOW`** — named loggers clamped tighter still, listed below. + +This is a security control, not tidiness. Verified against the installed +packages rather than assumed: + +| Logger | At DEBUG it prints | +|---|---| +| `botocore.auth` | `CanonicalRequest`, `StringToSign`, and the **SigV4 signature** | +| `botocore.endpoint` | the prepared request, including the `Authorization` header | +| `discord.ext.voice_recv.reader` | the **Discord voice secret key** and raw voice payload bytes | +| `discord.ext.voice_recv.gateway` | the whole voice-gateway payload for every op except 3 and 6 — and op 4 is `SESSION_DESCRIPTION`, which is **where that same secret key comes from** | +| `discord.ext.voice_recv.voice_client` | the voice state update, which carries the voice `token` and `session_id` | +| `discord.http` | whole REST **response bodies** — message content, display names, nicknames | +| `sqlalchemy.engine` | bound parameters, i.e. transcript text on its way into the database | +| `aiohttp.access` | at **INFO**, `request.path_qs` — the path *with* its query string, and `link`'s only route is `/oauth/callback?code=…&state=…` | + +**Why a floor and not just the list.** The list came first and was not +enough. `NEVER_BELOW` names 17 loggers; a running worker has around 90, and +a logger absent from the list carries no level of its own and inherits one +from the root logger. `configure_logging` used to set root to +`min(sturnus_level, third_party_level)`, so `STURNUS_LOG_LEVEL=DEBUG` — a +variable whose documented scope is Sturnus's own loggers — put the root +logger at DEBUG and turned on 24 third-party loggers with it. Rows three +to seven of that table were reachable that way. The floor is what makes +the claim structural: it applies to names nobody enumerated, including +names in libraries this repository has not imported yet, and +`tests/observability/test_third_party_log_floor.py` asserts it as a +property over every logger that exists rather than as another list. + +**Why `INFO` and not `WARNING`.** `discord.voice_state` logs the voice +connect narrative at INFO — "Starting voice handshake… (connection attempt +2)", "Voice handshake complete", "Timed out connecting to voice", +"Disconnected from voice by discord, close code 4006". That is the evidence +base for telling apart the three ways capture fails, which is what +`voice.join_failed` / `voice.reader_stopped` / `voice.decode_failed` exist +to distinguish, and a WARNING floor would delete it. What `INFO` does cost +is real and worth naming: gateway IDENTIFY/RESUME and session-invalidation +tracing ("the bot silently stopped receiving events"), voice websocket +close codes below the INFO cases, and `discord.http` rate-limit bucket +diagnosis are no longer reachable without a code change. Those are exactly +the cases where someone wants DEBUG at 3am — and they are also the cases +where the same switch would publish a session key, which is why the switch +is a source edit on a non-production deployment rather than a Helm value. + +**Second lock: the value is redacted as well as suppressed.** A level +control only helps for records that are not emitted. `redaction.PATTERNS` +carries a `secret_value` rule that scrubs anything assigned to a name in +`fields.CREDENTIAL_NAMES` — `secret_key: [...]`, `token=…`, `password="…"` — +in *any* record, including a message string a library composed. It exists +because the voice secret key has no recognisable shape: it reaches a log +record as thirty-two small integers, which no token pattern matches, inside +`record.msg`, where the field allowlist cannot see it. If a future release +of `discord-ext-voice-recv` moves that line to a level the floor permits, +the key is still replaced by `«redacted:secret_value»` and the rest of the +line survives. + +**The list itself.** `discord.ext.voice_recv.router` is clamped to +`WARNING`, which deliberately still lets through its own +`log.exception("Error in %s loop")` at ERROR — the one library line that +matters when the packet-router thread dies. + +`discord.voice_state` is a level decision rather than a list one, and it is +the only logger held **open** as well as shut. It is pinned at exactly +`INFO`, by two entries that check each other: + +* `NEVER_BELOW` keeps its DEBUG lines out. They are connection-state + transitions and DAVE upgrade notices, and they were read at the installed + version and found to carry no secret — but DEBUG is where the leak lives + on every other logger in that table, and a name absent from the list + reads afterwards as "considered and cleared" when it was never considered + at all. +* `NEVER_ABOVE` keeps its INFO lines in. Those are the connect narrative — + "Starting voice handshake… (connection attempt 2)", "Voice handshake + complete. Endpoint found: …", "Timed out connecting to voice", + "Disconnected from voice by discord, close code 4014" — and they are the + evidence base for the capture-failure cooldown and for telling + `voice.join_failed` from `voice.reader_stopped` from + `voice.decode_failed`. All three entrypoints emitted them before this + package existed, because they called `basicConfig(level=INFO)`. + +`NEVER_BELOW` alone could not have kept them: it is applied as +`max(third_party_level, floor)`, so every entry in it can only make a +logger *quieter*, and with `STURNUS_LOG_THIRD_PARTY_LEVEL` at its default +of `WARNING` an entry of `INFO` there is a no-op. `NEVER_ABOVE` installs +the level outright — which is also why it is not an exemption: +`STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` cannot make a pinned logger any +louder than its pin either. + +The `aiohttp.access` row was a live leak before it was clamped: with the +root logger at `INFO`, every successful account link wrote an Outline +authorization code into Loki. Nothing is lost by silencing it — the ingress +already logs requests, and `link.callback_rejected` / `link.established` +carry the diagnostic content without the credential. + +**How to check the floor is doing its job**, on a running deployment: + +```logql +# Should return nothing, ever. If it returns rows, the floor is broken. +{app_kubernetes_io_name="sturnus"} | json | level="DEBUG" | logger !~ "sturnus.*" +``` + +### 7.3 Loki labels: what Alloy may promote + +Sturnus cannot set Loki labels; `alloy-logs` derives them from Kubernetes +metadata. This is the policy for whoever edits the Alloy configuration. + +**Already labels, free, no code change:** `namespace`, `pod`, `container`, +`app_kubernetes_io_name`, `app_kubernetes_io_component` (this is how +bot/worker/link separate). + +**Promote from the JSON line — exactly two:** `level` (5 values) and +`guild_id` (a handful). `guild_id` is the one dimension an operator +genuinely slices by, and it is the first thing to review if Sturnus is ever +deployed multi-tenant at scale. + +**Never a label:** `session_id`, `job_id`, `discord_user_id`, `ssrc`, +`document_id`, `external_user_id`, `trace_id`. Each is unbounded or +fast-growing, and promoting one multiplies the cluster's stream count +without limit. They are line content, queried with `| json | session_id="4711"` +— which is exactly what LogQL's parser stage is for. + +`event` (~40 stable values) was considered and rejected: bounded, but +~40 × 3 components × 5 levels is a few hundred streams for a query that +`| json | event="job.dead"` answers just as well. + +### 7.4 A LogQL cookbook + +```logql +# The whole story of one session, across bot and worker in one stream +{app_kubernetes_io_name="sturnus"} | json | session_id="4711" + +# A session ended having recorded nothing despite participants — the +# highest-value single query here +{app_kubernetes_io_name="sturnus"} | json | event="session.closed" | jobs_enqueued == 0 + +# We were in the channel and could not hear. Three different faults, one +# question: capture never started, capture died, or nothing decodes any +# more. Each ends the session with an end_reason that says so, rather than +# letting it time out looking like a meeting where nobody spoke. +{app_kubernetes_io_name="sturnus"} | json | event=~"voice.join_failed|voice.reader_stopped|voice.decode_failed" + +# The same three, from the metric side, which is what to alert on +sum by (end_reason) (rate(sturnus_session_duration_count{end_reason=~"capture_failure|decode_failure"}[15m])) + +# Permanent loss: audio that will never become a transcript +{level="error"} | json | event=~"job.dead|session.unrecoverable|session.document_rejected" + +# Whisper throughput against real material (Spec 15's unmeasured risk) +{app_kubernetes_io_component="worker"} | json | event="job.transcribed" | unwrap realtime_factor + +# Per-guild error rate — the one place the guild_id label earns itself +sum by (guild_id) (rate({level="error"}[5m])) +``` + +`kubectl logs` shows JSON, which is a genuine regression for a human under +pressure. Use: + +```sh +kubectl logs deploy/sturnus-worker | jq -r '"\(.ts) \(.level) \(.event) \(.msg)"' +``` + +Every line carries `trace_id` when telemetry is enabled, so a Grafana +derived field turns a Loki row into a click through to the Tempo waterfall +for the same job. + +### 7.5 Metrics are pushed, not scraped + +`/metrics` answers **`501 Not Implemented`** and names +`STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT` in its body. It used to return `200` +with an empty exposition; that inverts the signal, because an empty `200` +is indistinguishable from "every counter is legitimately zero", so a +completely uninstrumented process would look healthy to a ServiceMonitor. A +`501` marks the target down, which is the true statement. + +**Before merging, confirm nothing outside this repository scrapes it.** +Nothing in `charts/` does, but a cluster-side `ServiceMonitor` or a +`prometheus.io/scrape` annotation living in another repository would not +show up in that check. + +| Instrument | Type | Unit | Attributes | Question it answers | +|---|---|---|---|---| +| `sturnus.job.stage.duration` | histogram | s | `stage`, `outcome` | Which pipeline stage is slow, across all jobs? | +| `sturnus.job.outcome` | counter | 1 | `outcome` | How many jobs died today? **The alerting signal** — unsampled, unlike a span. `outcome` is one of `done` / `failed` / `dead` / `crashed`, and it is recorded by the transition that decided it (`infrastructure/db/queue.py`), never inferred from the worker loop's return value — see the note below the table. | +| `sturnus.queue.depth` | gauge | 1 | `status` | Is the worker keeping up? One series per status: `pending`, `running`, `done`, `failed`, `dead`. **Sampled once per poll**, so during a long transcription it is as old as that job — see the caveat below. | +| `sturnus.transcription.audio_duration` | histogram | s | — | Paired with the `transcribe` stage histogram, gives the realtime factor. | +| `sturnus.transcription.decoded_seconds` | counter | s | `model` | **Seconds of audio handed to the decoder** — the gated speech, concatenated, not the padded recording it was cut from (§ the worker hands the model only what the speech gate found). Divided by wall time this is the real-time factor, which is the single most useful operational number here — see 7.5.1. | +| `sturnus.transcription.position_seconds` | gauge | s | `model` | How far into that concatenated speech the job in flight has got. Not a position in the recording: the times that reach the document are mapped back onto the recording's timeline afterwards, these are not. Absent while nothing is transcribing. | +| `sturnus.transcription.total_seconds` | gauge | s | `model` | How much speech that job has to get through; the denominator for `position_seconds`, on the same timeline. Absent while nothing is transcribing. | +| `sturnus.transcription.seconds_since_progress` | gauge | s | `model` | Seconds since the job in flight last produced a segment, or since it started. **The alert.** Absent while nothing is transcribing. | +| `sturnus.session.close.duration` | histogram | s | `end_reason`, `outcome` | **Will a deploy lose a session?** Compare p99 to `terminationGracePeriodSeconds`. | +| `sturnus.session.duration` | histogram | s | `end_reason`, `guild_id` | Are sessions ending by timeout, by people leaving, or because we could not hear? `end_reason` is one of `empty` / `idle_timeout` / `max_duration` / `shutdown` / `capture_failure` / `decode_failure` / `unknown`. The last three are the ones that cost a meeting; `unknown` means the close itself raised. | +| `sturnus.session.active` | up/down counter | 1 | `guild_id` | Is anything recording right now — and did a session leak? Incremented when the session *row* opens, decremented on every close path there is, so a capture failure cannot make it drift. | +| `sturnus.recording.upload.bytes` | histogram | By | — | Capacity planning against the retention window. | +| `sturnus.voice.packets` | counter | 1 | `outcome`, `guild_id` | **Why is person X missing from the transcript?** `outcome` is one of `recorded` / `no_role` / `no_consent` / `not_recording` / `unknown_user` / `undecodable` / `loop_gone`. `undecodable` is the early warning the decode-failure threshold deliberately does not give — that fires once, after five consecutive seconds of nothing, and this is visible from the first frame. | +| `sturnus.voice.packet_errors` | counter | 1 | `error_type` | Is the voice adapter throwing? The rate, not the log line: the matching `voice.packet_handler_failed` line is rate limited to one in a thousand because it is per-frame in origin. | +| `sturnus.document.create.duration` | histogram | s | `outcome` | Is Outline down, slow, or rejecting us? | +| `sturnus.oauth.callback` | counter | 1 | `outcome` | Are account links failing, and at which step? | + +`sturnus.queue.depth` costs no extra database load: the worker's poll loop +already ran `SELECT 1` as a liveness probe, and that query is now a grouped +count over `transcription_job.status` — still one round trip that either +answers or does not, and it feeds the gauge as well. **The caveat that +comes with that:** it is sampled once per poll, and the worker does not +poll while it is transcribing. A ninety-minute job means a ninety-minute-old +depth reading, so alert on it over a long window (`for: 30m`) and use +`sturnus.transcription.seconds_since_progress` — which does not depend on +the loop coming back round — for anything sharper. + +**`sturnus.job.outcome` says what happened, not what was returned.** +`process_one` returns `True` after `queue.fail(...)` exactly as it does +after `queue.complete(...)`: the boolean means "work was attempted", never +"work succeeded". The worker loop used to turn it into `outcome="done"`, so +every failed and every dead job was published as a success — a metric that +reports failures as successes is worse than no metric, because it will be +believed. The label is now recorded by `JobQueue.complete` and +`JobQueue.fail`, the two transitions that decide a job's terminal state, +and `crashed` is the one the loop still owns: `process_one` raised, so the +worker is about to die with a job possibly stuck in `running` (the lease in +`JobQueue.claim` is what reclaims it). + +### 7.5.1 Transcription progress: telling fast-because-broken from slow-because-working + +**Why these four exist.** Whisper's own failure mode is silence. When +Silero's VAD collapsed on the bit-exact padding Sturnus writes between +packets, the model was handed a 100-minute recording, returned no segments, +and the job "finished" in 43 seconds. What everybody saw was an empty +transcript — which is exactly what a participant who never spoke also +produces, and it was read as that for a day. What the metric would have +shown is a real-time factor of **140x**: a hundred minutes of audio decoded +in forty-three seconds, which is not physically possible. Meanwhile a +genuine job on that same recording ran for **98 minutes**. The honest range +is that wide, which is precisely why "it is taking a long time" is not a +diagnosis and this number is. + +`large-v3` on the worker's CPU allocation measures **1.94x** — that is +`wall / audio`, so audio accrues at roughly half of wall-clock. Read +"audio" as *speech*: the decoder is handed the gated clips concatenated, so +a 100-minute recording holding 41 minutes of speech accrues 41 minutes +here, not 100. That makes the ratio a statement about decoding rather than +about how much of the meeting was silence, and it leaves the impossible- +throughput alert below firing on the same failure — the 43-second collapse +is a rate of ~57 against a ceiling of 10 either way: + +```promql +# Real-time factor, the way the 1.94x figure is quoted (wall per second of audio) +1 / (sum by (model) (rate(sturnus_transcription_decoded_seconds_total[30m]))) + +# Alert: physically impossible throughput. Nothing is being decoded. +sum by (model) (rate(sturnus_transcription_decoded_seconds_total[15m])) > 10 +``` + +Ten is a deliberately loose ceiling — five times faster than the fastest +plausible model on this hardware — because the failure it catches is three +orders of magnitude out, not a few percent. + +**The stall alert, which is the one to page on.** + +```promql +# A job that has produced nothing for ten minutes. Absent when idle, so this +# expression is simply empty on a worker with no work. +sturnus_transcription_seconds_since_progress > 600 +``` + +This is the only signal that covers a job which wedges **before its first +segment** — inside feature extraction or language detection, which is where +the collapse happened. A position gauge cannot: a job stuck at zero looks +exactly like a job that has only just started, and every real job passes +through that state. The clock therefore starts when the model is called, +not when the first segment arrives. + +`position_seconds / total_seconds` is the dashboard number ("this job is 12 +minutes into 43"). Both are on the concatenated-speech timeline, which is +why the fraction means anything; it is not an alert on its own, since a +large `total_seconds` is a talkative meeting, not a fault. + +**No id is a label on any of these, and that is not an oversight.** A +session id, job id, guild id or user id would be unbounded cardinality +*and* a record of who was in which voice channel when, kept for as long as +the metric store keeps anything. `model` plus the resource's `service.name` +are enough to read every number above. To go from a suspicious rate to the +job responsible, jump to the log: `transcription.decoded` and +`transcription.skipped` carry `speech_seconds`, `clips` and `segments`, and +`job.transcribed` next to them carries `job_id` and `session_id`. + +**The log line that pairs with these.** `speech_seconds` against +`audio_seconds` on `transcription.decoded` is the speech gate's own +signature, and it is what would have named Silero as the culprit on the +first read: one second of speech in two minutes of recording is not a +plausible meeting. + +```logql +# The gate found almost nothing in a long recording +{app_kubernetes_io_component="worker"} | json | event="transcription.decoded" + | speech_seconds < 5 | audio_seconds > 120 + +# The model was never called at all — a genuinely silent participant, and +# the *other* explanation for an empty transcript +{app_kubernetes_io_component="worker"} | json | event="transcription.skipped" +``` + +Histogram buckets are set explicitly. The SDK's defaults top out at 10 000 +in *milliseconds*; Sturnus's durations are seconds and run to an hour, so +the defaults would put every transcription in one bucket. + +### 7.6 Traces + +Root spans: `job.process` (worker, per poll), `session.open` and +`session.close` (bot), `document.create` (Outline). + +**The packet path emits no spans, by construction.** `sink.py`'s `write()` +runs ~50×/s per speaker; ten speakers is 500/s, which at a measured 16.7 µs +per recorded span is 8.4 ms of CPU per second of wall clock *and* 43 million +spans a day from one bot. They would also all be orphaned roots, because the +extension's router thread inherits no context. Counters go there instead, at +a measured **2.2 µs** per `add` with a provider installed and **0.10 µs** +without — 1.1 ms/s at 500 packets/s, under 0.15% of one core. That is not a +consolation prize: a rate graph split by `outcome` answers "why is X missing +from the transcript" in one panel, which fifty spans a second would not. + +`session.close` is the span worth watching. It encrypts, uploads and +enqueues **serially, per speaker**, and it runs during SIGTERM. If that +takes longer than `terminationGracePeriodSeconds`, Kubernetes kills the pod +mid-loop and Spec 15's "the entire session is lost, not just a portion" is +what happens. Nothing measured that before. + +**Traces are not wired into Sentry.** `sentry_sdk` can consume OTel spans, +and `observability.py` locks that door twice on purpose +(`traces_sample_rate=0.0` *and* `before_send_transaction=drop_transaction`), +because `before_send` is never called for transactions and span data would +route around `scrub_event` entirely. The pod log line, carrying `trace_id`, +is the correlation point instead. + +### 7.7 After a deploy: confirm telemetry actually arrives + +If the endpoint is misconfigured, spans and metrics vanish and every +dashboard shows a flat, healthy-looking zero — which is easy to misread as +"nothing is wrong". + +Export failures are still visible, but only in one place. The OTLP logger is +clamped to `ERROR`, so a lost batch is one line in Loki rather than the four +the exporter emits per batch (three retries plus the give-up), and +`ignore_logger("opentelemetry")` keeps the same failures out of Sentry +entirely, where an unreachable Alloy would otherwise be an issue per retry +per batch from all three pods, forever. So: + +```logql +{app_kubernetes_io_name="sturnus"} | json | logger =~ "opentelemetry.*" +``` + +is the standing alert for "telemetry is being dropped". A *misconfigured* +endpoint that happens to resolve — pointing at the wrong service, say — +produces no error at all, and for that the smoke check below is the only +detector: + +1. `kubectl logs deploy/sturnus-worker | jq 'select(.event=="telemetry.enabled")'` — + confirms which endpoint was installed. +2. In Grafana, search Tempo for `service.name = sturnus-worker` and confirm + one `job.process` trace has arrived. +3. Confirm `sturnus.queue.depth` has a recent sample. diff --git a/docs/verification/end-to-end-checklist.md b/docs/verification/end-to-end-checklist.md index eee1007..31c6221 100644 --- a/docs/verification/end-to-end-checklist.md +++ b/docs/verification/end-to-end-checklist.md @@ -228,6 +228,71 @@ note it and continue.** id, or a status code is fine; a log line containing what someone said, or the bytes of what they said, or a credential, is not. + Make it executable rather than a read-through: **speak a rare canary + phrase** during the run — something that appears nowhere else, e.g. + "aubergine parliament" — and then grep for it, along with the first + eight characters of each credential: + + ```sh + for c in bot worker link; do + kubectl logs deploy/sturnus-$c --since=1h \ + | grep -iE 'aubergine|||' \ + && echo "LEAK in $c" + done + ``` + + A hit is a blocking defect, not a finding to note. The same canary + technique runs in CI as `tests/observability/test_no_payload_leaks.py`, + which drives `RecordingService.close()` and `process_one()` with + canary-laden fakes at DEBUG — so this step is confirming the control + survived contact with production, not discovering whether it exists. + +- [ ] **No logger outside `sturnus.*` is at DEBUG.** The canary gate above + cannot catch this class: a canary is a string *we* planted, and the + Discord voice `secret_key` is issued by Discord and printed by + `discord.ext.voice_recv`'s own formatter. Nothing we could speak into a + meeting would find it. + + ```sh + for c in bot worker link; do + kubectl logs deploy/sturnus-$c --since=1h \ + | jq -r 'select(.level=="DEBUG" and (.logger | startswith("sturnus") | not))' \ + && echo "third-party DEBUG in $c" + done + ``` + + This must be empty at every value of `STURNUS_LOG_LEVEL`, including + `DEBUG` — run it once with the worker at `STURNUS_LOG_LEVEL=DEBUG`, + because that is the setting the reported leak was found under and the + one an operator reaches for during an incident. A hit is a blocking + defect. `docs/operations.md` section 7.2 has the floor and the reasoning; + `tests/observability/test_third_party_log_floor.py` is the CI half. + +- [ ] **The same holds for Tempo and the metric store, not only for pod + logs.** This gate predates there being a second and third retained + store; spans and metric labels are indexed, retained, and visible to + everyone with Grafana access, so they are held to the same standard. + + In Grafana, open one `job.process` trace from this run and one + `session.close` trace, and confirm no span attribute contains transcript + text, a display name, an S3 object key, or an exception *message* — a + failed span should carry `error.type` and a bare `ERROR` status and + nothing else. Then confirm no metric series carries a `session_id`, + `job_id`, or `discord_user_id` label. + + Note what would make a leak here *permanent* in a way a log leak is not: + Loki's retention eventually expires a line, but a metric label that has + been created is a time series that persists. + +- [ ] **Telemetry actually arrives.** With + `STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT` set, confirm one trace and one + metric reach Grafana (`docs/operations.md` section 7.7). This is not + optional polish: OTLP export failures are deliberately kept out of both + Loki and Sentry — otherwise an unreachable Alloy is one error per retry + per batch from all three pods, forever — so a misconfigured endpoint + produces flat, healthy-looking zeroes on every dashboard and **this + check is the only thing that detects it**. + ## 5. What to record afterward Every one of these is an **estimate in the spec today** — this is the diff --git a/migrations/env.py b/migrations/env.py index 2e46bfb..31aa25a 100644 --- a/migrations/env.py +++ b/migrations/env.py @@ -7,25 +7,52 @@ # access to the values within the .ini file in use. config = context.config -# Interpret the config file for Python logging. -# -# `disable_existing_loggers=False` is load-bearing, not a preference. -# `fileConfig` defaults it to True, which sets `disabled = True` on every -# logger that already exists and is not named in `alembic.ini` -- and -# `logging.Logger.handle` returns immediately for a disabled logger. This -# module is not only imported by the `alembic` CLI: the worker runs -# migrations in-process at startup (`_run_migrations` in -# `sturnus.entrypoints.worker`), by which point every `sturnus.*` logger -# already exists. With the default, the worker would fall silent for the -# rest of its life the moment it finished migrating -- and Sentry with it, -# because `LoggingIntegration` hooks `Logger.callHandlers`, which a disabled -# logger never reaches. See `tests/infrastructure/test_migrations.py`. -if config.config_file_name is not None: - fileConfig(config.config_file_name, disable_existing_loggers=False) - import os # noqa: E402 from sturnus.infrastructure.db.models import Base # noqa: E402 +from sturnus.observability.setup import logging_is_configured # noqa: E402 + +# Interpret the config file for Python logging -- but only when nothing else +# owns logging yet. +# +# `fileConfig` is not additive. It *replaces* `root.handlers` with +# alembic.ini's bare stderr `StreamHandler` and, with its default +# `disable_existing_loggers=True`, sets `disabled = True` on every logger the +# ini does not name -- which is all of `sturnus.*`. +# +# `sturnus.entrypoints.worker` calls `configure_logging("worker")` in +# `main()` and then `_run_migrations` a few lines into `_run()`. Without this +# guard, the worker's single filtered JSON handler is torn out seconds after +# startup and every later line goes to stderr unstructured and, more +# importantly, unfiltered by `SturnusFilter` -- or is dropped entirely +# because its logger was disabled. That is the "one handler, one exit" +# guarantee in `sturnus.observability.setup` silently undone by a call in +# another file. +# +# Running Alembic standalone (`alembic upgrade head` from a shell) still gets +# the ini's logging, because nothing has configured logging in that process -- +# and that is the case `disable_existing_loggers=False` is for. The guard and +# the argument fix two different halves of the same regression and neither +# replaces the other: +# +# - The guard stops `root.handlers` being *replaced*, which the argument does +# nothing about. That is the half that matters once `configure_logging()` +# has installed the single filtered JSON handler. +# - `disable_existing_loggers` defaults to True, which sets `disabled = True` +# on every logger not named in `alembic.ini` -- and `Logger.handle` returns +# immediately for a disabled logger, so even Sentry stops seeing them +# (`LoggingIntegration` hooks `Logger.callHandlers`, which is downstream). +# That is the half that still bites on the standalone path the guard lets +# through: `alembic upgrade head` imports `sturnus.*` for `Base.metadata` +# above, so those loggers already exist by the time this line runs. +# +# Drop either one and a process goes quiet: without the guard the worker +# loses its filter seconds after startup, without the argument standalone +# Alembic disables every logger it did not name. See +# `tests/infrastructure/test_migrations.py` and +# `tests/test_logging_discipline.py`. +if config.config_file_name is not None and not logging_is_configured(): + fileConfig(config.config_file_name, disable_existing_loggers=False) target_metadata = Base.metadata diff --git a/pyproject.toml b/pyproject.toml index fe3c7d0..33acd97 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,30 @@ dependencies = [ # Without a DSN `sentry_sdk.init()` is never called, so nothing is # patched and the processes behave exactly as they do without it. "sentry-sdk>=2.68", + # Traces and metrics (sturnus.infrastructure.telemetry). Runtime + # dependencies for the same reason as sentry-sdk above: one image serves + # operators with and without a collector, so optionality is a runtime + # decision (is STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT set?) rather than an + # install-time one. With no endpoint no provider is constructed, so the + # API degrades to NonRecordingSpan and no-op instruments and the + # processes behave exactly as they do without it. + # + # `-proto-http` rather than `-proto-grpc`: Alloy accepts both 4317 and + # 4318, so the choice is free, and the gRPC exporter drags in grpcio -- + # a large per-arch binary wheel and a known source of fork/thread hangs. + # + # **Deliberately no `opentelemetry-instrumentation-*` package.** Every + # plausible one is disqualifying in this codebase specifically: + # -sqlalchemy adds `db.statement`, and the statement that writes a + # transcript *is* the protected content; -botocore adds `aws.s3.key`, + # and the key embeds a Discord user id; -httpx adds `url.full`, which + # for S3 is a presigned URL carrying X-Amz-Signature; -aiohttp-server + # would attach `link`'s `/oauth/callback?code=...`. None of the four + # packages below instruments anything on its own. + "opentelemetry-api>=1.44", + "opentelemetry-sdk>=1.44", + "opentelemetry-semantic-conventions>=0.65b0", + "opentelemetry-exporter-otlp-proto-http>=1.44", ] [project.scripts] diff --git a/src/sturnus/application/publishing.py b/src/sturnus/application/publishing.py index 554b201..0d13228 100644 --- a/src/sturnus/application/publishing.py +++ b/src/sturnus/application/publishing.py @@ -32,6 +32,8 @@ from jinja2.sandbox import SandboxedEnvironment +from sturnus.observability.events import Event, log_event, log_exception + log = logging.getLogger(__name__) #: The session `status` value the worker writes once a session's protocol @@ -175,5 +177,21 @@ async def announce_ready_sessions( try: await announcer.post(channel_id, render_announcement(document_url, template_source)) await sessions.mark_announced(session_id, now) + log_event( + log, + logging.INFO, + Event.ANNOUNCE_POSTED, + "Posted the session document link to the channel", + session_id=session_id, + channel_id=channel_id, + ) except Exception as exc: - log.warning("Failed to announce session %d; will retry: %s", session_id, exc) + log_exception( + log, + logging.WARNING, + Event.ANNOUNCE_FAILED, + "Failed to announce the session document; will retry next sweep", + exc, + session_id=session_id, + channel_id=channel_id, + ) diff --git a/src/sturnus/application/recording.py b/src/sturnus/application/recording.py index 2d57979..2658536 100644 --- a/src/sturnus/application/recording.py +++ b/src/sturnus/application/recording.py @@ -27,6 +27,9 @@ from sturnus.domain.session import EndReason, SessionMachine, SessionState, SessionTimeouts from sturnus.domain.silence import SILENCE_EVIDENCE_SECONDS, SilentAudioWatch from sturnus.domain.timeline import SpeakerClock +from sturnus.observability.events import Event, log_event, log_exception + +log = logging.getLogger(__name__) log = logging.getLogger(__name__) @@ -184,6 +187,16 @@ def __init__( self._data_key: SessionKey | None = None self._writers: dict[int, AudioWriter] = {} self._closed = False + #: Packet and byte counters for `session.closed`'s verdict. They + #: live here, in a class with thorough unit tests, rather than in + #: `sturnus.infrastructure.discord.voice` -- whose sink callback + #: runs on the extension's packet-router thread and has no unit + #: tests at all, by explicit design. Putting the count in the + #: untested file would make the replacement signal less trustworthy + #: than the flood it replaces. + self._packets = 0 + self._bytes = 0 + self._seen_participants = False @property def is_recording(self) -> bool: @@ -207,6 +220,15 @@ def needs_reset(self) -> bool: def session_id(self) -> int | None: return self._session_id + @property + def guild_id(self) -> int: + """The guild this service records for. + + Read-only, and public so telemetry in `infrastructure` can label a + session span without reaching into a private attribute. + """ + return self._guild_id + @property def channel_id(self) -> int: """The voice channel this service opens its session rows against.""" @@ -263,6 +285,8 @@ def due_reason(self, now: datetime) -> EndReason | None: async def participants_changed(self, consented_count: int, now: datetime) -> None: """Forwards to the machine; opens a session row on the IDLE -> RECORDING edge.""" was_idle = self._machine.state is SessionState.IDLE + if consented_count > 0: + self._seen_participants = True self._machine.participants_changed(consented_count, now) if was_idle and self._machine.state is SessionState.RECORDING: self._session_id = await self._sessions.open_session( @@ -277,6 +301,22 @@ async def participants_changed(self, consented_count: int, now: datetime) -> Non await self._sessions.record_session_key( self._session_id, self._encryptor.key_id, self._data_key.wrapped ) + # The anchor line of the whole story: every later event, in this + # process and in the worker, joins to it on `session_id`. + # `key_id` says which master key must still exist for this + # session ever to be decrypted -- the one fact that makes a + # rotation mistake recoverable rather than merely visible. + log_event( + log, + logging.INFO, + Event.SESSION_OPENED, + "Opened a recording session", + session_id=self._session_id, + guild_id=self._guild_id, + channel_id=self._channel_id, + consented_present=consented_count, + key_id=self._encryptor.key_id, + ) async def voice_packet( self, @@ -302,8 +342,28 @@ async def voice_packet( self._session_id, discord_user_id, display_name, now ) await self._sessions.set_audio_epoch(self._session_id, discord_user_id, at) + # One line per speaker, never per packet. This is what separates + # "nobody consented" from "consented but silent" from "capture + # is broken" -- three very different incidents that look + # identical from outside without it. + # + # `display_name` is deliberately absent: it is directly + # identifying and tells an operator nothing the user id does + # not. It is in `fields.DENIED_NAMES` so the build fails if + # anyone adds it here later. + log_event( + log, + logging.INFO, + Event.SESSION_SPEAKER_FIRST_PACKET, + "First audio packet from a speaker", + session_id=self._session_id, + discord_user_id=discord_user_id, + ssrc=ssrc, + ) writer.write(at, pcm) + self._packets += 1 + self._bytes += len(pcm) self._machine.audio_received(now) # Last, and only after the audio itself is safely written: this is @@ -312,11 +372,9 @@ async def voice_packet( # logged or passed on -- and answers `True` exactly once per # speaker per session, on the packet that completes the case. if self._silence.observe(discord_user_id, pcm): - await self._report_silent_audio(discord_user_id, display_name, at) + await self._report_silent_audio(discord_user_id, at) - async def _report_silent_audio( - self, discord_user_id: int, display_name: str, at: datetime - ) -> None: + async def _report_silent_audio(self, discord_user_id: int, at: datetime) -> None: """Says, three ways, that this speaker's audio is arriving empty. Three, because each survives something the others do not. The log @@ -336,35 +394,52 @@ async def _report_silent_audio( warning. """ assert self._session_id is not None - log.warning( - "Audio from %s (id %d) in session %d has been arriving for %ds with no " - "audible level: packets are being received and decoded, and every sample in " - "them is at the noise floor. This is what a microphone muted at system level " - "produces, and it transcribes to nothing. Recording continues.", - display_name, - discord_user_id, - self._session_id, - SILENCE_EVIDENCE_SECONDS, + # `display_name` is deliberately not here, and this is the one line + # in the three where leaving it out costs something: it is what the + # room would recognise. It is directly identifying, it is in + # `fields.DENIED_NAMES`, and the id answers the operator's question + # -- "whose microphone" is a question for the channel message, which + # renders the mention and is read by people who are in the meeting. + log_event( + log, + logging.WARNING, + Event.SPEAKER_AUDIO_SILENT, + "Audio from this speaker has been arriving with no audible level: packets are " + "being received and decoded, and every sample in them is at the noise floor. " + "This is what a microphone muted at system level produces, and it transcribes " + "to nothing. Recording continues.", + session_id=self._session_id, + discord_user_id=discord_user_id, + duration_seconds=SILENCE_EVIDENCE_SECONDS, ) try: await self._announcer.post( self._channel_id, render_silent_audio_warning(discord_user_id) ) except Exception as exc: - log.warning( - "Could not post the silent-audio warning for id %d into channel %d: %s", - discord_user_id, - self._channel_id, + log_exception( + log, + logging.WARNING, + Event.SPEAKER_SILENT_WARNING_FAILED, + "Could not post the silent-audio warning into the channel; the durable " + "record below is what is left of it", exc, + session_id=self._session_id, + discord_user_id=discord_user_id, + channel_id=self._channel_id, ) try: await self._sessions.record_silent_audio(self._session_id, discord_user_id, at) except Exception as exc: - log.warning( - "Could not record silent audio for id %d on session %d: %s", - discord_user_id, - self._session_id, + log_exception( + log, + logging.WARNING, + Event.SPEAKER_SILENT_RECORD_FAILED, + "Could not record this speaker's silent audio on the session row; the " + "finding survives only as this line", exc, + session_id=self._session_id, + discord_user_id=discord_user_id, ) def request_close(self, reason: EndReason) -> None: @@ -454,7 +529,17 @@ async def close(self, reason: EndReason, now: datetime) -> None: assert self._data_key is not None session_id = self._session_id retention_until = now + timedelta(days=self._retention_days) + log_event( + log, + logging.INFO, + Event.SESSION_CLOSING, + "Closing the session: encrypting, uploading and enqueuing", + session_id=session_id, + reason=reason.value, + speakers=len(self._writers), + ) + jobs_enqueued = 0 enc_paths: list[Path] = [] session_dir: Path | None = None for discord_user_id, writer in self._writers.items(): @@ -475,10 +560,46 @@ async def close(self, reason: EndReason, now: datetime) -> None: wrapped_data_key=self._data_key.wrapped, retention_until=retention_until, ) + jobs_enqueued += 1 enc_paths.append(enc_path) + # The object key is not logged: it is + # `sessions/{session_id}/speakers/{discord_user_id}.enc`, so + # both halves are on this line already and the key itself would + # be duplication with a wider blast radius. `audio_key()` + # reconstructs it. + log_event( + log, + logging.INFO, + Event.SESSION_SPEAKER_FINALIZED, + "Encrypted, uploaded and enqueued one speaker's recording", + session_id=session_id, + discord_user_id=discord_user_id, + bytes=enc_path.stat().st_size if enc_path.exists() else 0, + ) await self._sessions.close_session(session_id, now, reason.value) + # The verdict, and the reason this line exists at all: a session + # that had consenting participants and enqueued nothing recorded + # nothing. Today that outcome is expressed as complete silence -- + # no document, no announcement, and not one log line saying so. + # Here it is a single ERROR an alert can fire on. + recorded_nothing = jobs_enqueued == 0 and self._seen_participants + log_event( + log, + logging.ERROR if recorded_nothing else logging.INFO, + Event.SESSION_CLOSED, + "Session closed having recorded nothing despite participants being present" + if recorded_nothing + else "Session closed", + session_id=session_id, + reason=reason.value, + speakers=len(self._writers), + jobs_enqueued=jobs_enqueued, + packets=self._packets, + bytes=self._bytes, + ) + for enc_path in enc_paths: enc_path.unlink(missing_ok=True) if session_dir is not None: @@ -499,6 +620,10 @@ def reset(self) -> None: the voice adapter that dispatches packets into it) can be reused for a second, third, ... session without ever being reconstructed. + Also zeroes the packet/byte counters and the "we saw participants" + flag, so the next session's `session.closed` verdict is about that + session rather than a running total across the process's lifetime. + Without this, `_closed` stays `True` and `_machine` stays stuck in `SessionState.CLOSING` forever: `is_recording` never becomes `True` again, `voice_packet` keeps returning early, and @@ -513,3 +638,6 @@ def reset(self) -> None: self._data_key = None self._writers = {} self._closed = False + self._packets = 0 + self._bytes = 0 + self._seen_participants = False diff --git a/src/sturnus/application/recovery.py b/src/sturnus/application/recovery.py index 43b8379..4ea4a29 100644 --- a/src/sturnus/application/recovery.py +++ b/src/sturnus/application/recovery.py @@ -28,6 +28,7 @@ from sturnus.application.recording import JobQueue, SessionRecorder, audio_key from sturnus.application.recording import RecordingService as _RecordingService from sturnus.domain.session import EndReason, SessionTimeouts +from sturnus.observability.events import Event, log_event log = logging.getLogger(__name__) @@ -238,18 +239,28 @@ async def recover_orphans( # speaker it had was already uploaded and enqueued before that # happened -- these are stale local copies of work already # done, not work still owed. See `recover_orphans`'s docstring. - log.warning( - "Session %d's row is already %s; removing %d leftover file(s) on disk " - "instead of reprocessing them, which would duplicate already-enqueued jobs", - session_id, - status, - len(group), + log_event( + log, + logging.WARNING, + Event.SESSION_RECOVERED, + "Session row is already closed; removing leftover files on disk instead of " + "reprocessing them, which would duplicate already-enqueued jobs", + session_id=session_id, + status=status, + count=len(group), ) for orphan in group: orphan.path.unlink(missing_ok=True) continue - log.warning("Recovering %d orphaned recording(s) for session %d", len(group), session_id) + log_event( + log, + logging.WARNING, + Event.SESSION_RECOVERED, + "Recovering orphaned recordings for a session", + session_id=session_id, + count=len(group), + ) stored_key = await sessions.session_key(session_id) plain = [o for o in group if not o.encrypted] @@ -279,13 +290,20 @@ async def recover_orphans( if stored_key is None: for o in already_encrypted: - log.warning( - "Cannot recover session %d speaker %d: session has no stored data " - "key, so the file at %s cannot be decrypted -- skipping instead of " - "enqueuing a job that could never succeed", - session_id, - o.discord_user_id, - o.path, + # ERROR, not WARNING: this is permanent data loss. Audio + # exists on disk that no stored key can ever decrypt, and no + # sweep, retry or restart changes that. The path is + # deliberately not logged -- it names the directory where + # decrypted speech lives, and `session_id` plus + # `discord_user_id` already locate the file exactly. + log_event( + log, + logging.ERROR, + Event.SESSION_UNRECOVERABLE, + "Session has no stored data key, so this recording can never be " + "decrypted; skipping instead of enqueuing a job that could never succeed", + session_id=session_id, + discord_user_id=o.discord_user_id, ) continue diff --git a/src/sturnus/application/retention.py b/src/sturnus/application/retention.py index e5c0d49..4c12ab9 100644 --- a/src/sturnus/application/retention.py +++ b/src/sturnus/application/retention.py @@ -29,6 +29,8 @@ from datetime import datetime from typing import Protocol, cast +from sturnus.observability.events import Event, log_exception + log = logging.getLogger(__name__) @@ -97,4 +99,11 @@ async def sweep_expired_audio(jobs: JobStore, store: AudioDeleter, now: datetime await store.delete(cast(str, job["s3_key"])) await jobs.mark_audio_deleted(job_id, now) except Exception as exc: - log.warning("Failed to delete expired audio for job %d; will retry: %s", job_id, exc) + log_exception( + log, + logging.WARNING, + Event.RETENTION_FAILED, + "Failed to delete expired audio; will retry next sweep", + exc, + job_id=job_id, + ) diff --git a/src/sturnus/application/worker.py b/src/sturnus/application/worker.py index 9264e83..1272fc7 100644 --- a/src/sturnus/application/worker.py +++ b/src/sturnus/application/worker.py @@ -90,6 +90,7 @@ import asyncio import logging import shutil +import time import uuid from datetime import UTC, datetime, timedelta, tzinfo from pathlib import Path @@ -107,6 +108,7 @@ from sturnus.application.transcription import TranscriptionEngine from sturnus.domain import settings as domain_settings from sturnus.domain.transcript import DEFAULT_MERGE_GAP +from sturnus.observability.events import Event, log_event, log_exception log = logging.getLogger(__name__) @@ -138,7 +140,12 @@ async def claim(self) -> object | None: ... async def complete(self, job_id: int, transcript: str) -> bool: ... - async def fail(self, job_id: int, error: str, max_attempts: int) -> None: ... + #: Returns whether the job is now **dead** -- out of attempts, so this + #: recording will never be transcribed -- rather than queued for another + #: try. Only the queue can answer that, because only the queue counts + #: the attempts, and without the answer a caller cannot tell permanent + #: loss from an ordinary retry: `process_one` returns `True` for both. + async def fail(self, job_id: int, error: str, max_attempts: int) -> bool: ... class AudioDownloader(Protocol): @@ -404,9 +411,32 @@ async def _create_session_document( # which this module must never import (see the module docstring). if type(exc).__name__ != "PermanentDocumentError": raise - log.warning("Document sink permanently rejected creation for session %d", session_id) + # ERROR, not WARNING: a permanent rejection is the end of the road + # for this session's document. No sweep will fix it, so it needs a + # human -- unlike every other document failure here, which + # `retry_pending_documents` picks up on its own schedule. + log_event( + log, + logging.ERROR, + Event.SESSION_DOCUMENT_REJECTED, + "Document sink permanently rejected creation; no retry will succeed", + session_id=session_id, + ) return await sessions.mark_documented(session_id, created.id, created.url, provider) + log_event( + log, + logging.INFO, + Event.SESSION_DOCUMENT_CREATED, + "Created the session protocol document", + session_id=session_id, + document_id=created.id, + provider=provider, + collection_id=target, + participants=len(transcript.participants), + blocks=len(transcript.blocks), + body_bytes=len(body.encode("utf-8")), + ) async def process_one( @@ -472,6 +502,16 @@ async def process_one( if claimed is None: return False job = cast(_ClaimedJobShape, claimed) + log_event( + log, + logging.INFO, + Event.JOB_CLAIMED, + "Claimed a transcription job", + job_id=job.id, + session_id=job.session_id, + discord_user_id=job.discord_user_id, + key_id=job.encryption_key_id, + ) job_dir = work_dir / f"job-{job.id}-{uuid.uuid4().hex}" job_dir.mkdir(parents=True, exist_ok=True) @@ -508,13 +548,50 @@ async def process_one( if named_language is not None else await sessions.detected_language(job.session_id, job.discord_user_id) ) + + # Started here rather than before the two config reads above, so + # `realtime_factor` stays a measurement of the model and not of + # a database round-trip. Spec 15 wants that number compared + # against real material, and a number that quietly includes + # whatever the config store was doing is not comparable. + started = time.monotonic() try: result = await engine.transcribe(wav_path, pinned_language, prompt) except Exception as exc: - log.warning("Transcription failed for job %d", job.id) + log_exception( + log, + logging.WARNING, + Event.JOB_FAILED, + "Transcription failed", + exc, + job_id=job.id, + session_id=job.session_id, + stage="transcribe", + max_attempts=max_attempts, + ) await queue.fail(job.id, str(exc), max_attempts) return True + wall_seconds = time.monotonic() - started + audio_seconds = max((segment.end for segment in result.segments), default=0.0) + # Counts and durations, never text. `realtime_factor` is the + # number Spec 15 says must be measured against real material + # before rollout rather than estimated -- this is that + # measurement, on every job, forever. + log_event( + log, + logging.INFO, + Event.JOB_TRANSCRIBED, + "Transcribed a recording", + job_id=job.id, + session_id=job.session_id, + segments=len(result.segments), + audio_seconds=round(audio_seconds, 3), + wall_seconds=round(wall_seconds, 3), + realtime_factor=round(wall_seconds / audio_seconds, 3) if audio_seconds else None, + language=result.language, + ) + # Reached only when the guild asked for detection *and* this is # the first job for this speaker: a named language is never # `None`, which is exactly what keeps configuration out of @@ -534,7 +611,21 @@ async def process_one( # handling (Defect 4)" note -- without this, the exception # propagated out of `process_one` and killed the worker # process, stranding this job `running` forever. - log.warning("Job %d failed outside transcription: %s", job.id, exc) + # `stage` is what this line was missing: it covered download, + # decrypt *and* the transcript write with one message and no + # timing for any of them. The stage now says which, and the + # matching `job.process` trace times all three. + log_exception( + log, + logging.WARNING, + Event.JOB_FAILED, + "Job failed outside transcription", + exc, + job_id=job.id, + session_id=job.session_id, + stage="pipeline", + max_attempts=max_attempts, + ) await queue.fail(job.id, str(exc), max_attempts) return True finally: @@ -556,10 +647,18 @@ async def process_one( # `retry_pending_documents` to pick up: the session stays # `closed` and never becomes `documented`, which is exactly # what that sweep looks for. - log.warning( - "Document creation failed for session %d; will retry: %s", - job.session_id, + # Never `%s` on `exc`: `_create_session_document` renders the + # assembled transcript through Jinja and posts it through httpx, + # so a `jinja2.UndefinedError` or an `httpx.HTTPStatusError` + # raised in that path can carry template context or request + # content -- and `%s` would print it verbatim. + log_exception( + log, + logging.WARNING, + Event.SESSION_DOCUMENT_RETRY_FAILED, + "Document creation failed; the retry sweep will try again", exc, + session_id=job.session_id, ) return True @@ -599,4 +698,11 @@ async def retry_pending_documents( documents, sessions, jobs, links, config, session_id, template_source ) except Exception as exc: - log.warning("Retrying document creation failed for session %d: %s", session_id, exc) + log_exception( + log, + logging.WARNING, + Event.SESSION_DOCUMENT_RETRY_FAILED, + "Retrying document creation failed; will try again next sweep", + exc, + session_id=session_id, + ) diff --git a/src/sturnus/config.py b/src/sturnus/config.py index 89a61fa..99b7f15 100644 --- a/src/sturnus/config.py +++ b/src/sturnus/config.py @@ -10,7 +10,7 @@ from functools import lru_cache from pathlib import Path -from pydantic import SecretStr, field_validator, model_validator +from pydantic import Field, SecretStr, field_validator, model_validator from pydantic_settings import BaseSettings, SettingsConfigDict @@ -94,6 +94,63 @@ def _blank_dsn_is_absent(cls, value: SecretStr | None) -> SecretStr | None: return value +class OtelSettings(StrictSettings): + """Whether, and where, to send traces and metrics. + + Its own class, and constructed before the process's real settings, for + exactly the two reasons `SentrySettings` gives: nothing here is + required, so it always builds; and observability has no business + widening the per-process credential asymmetry Spec 13.2 establishes. + + **The blank-to-`None` normalisation is the same load-bearing trick, for + the same verified reason.** An unconfigured cluster's chart default is + `STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT: ""`, not an absent variable, and + `StrictSettings._reject_blank_required_values` does not apply to an + optional field. So blank must become `None` here, and `None` must mean + *no provider is ever constructed* -- not "a provider pointed at + nothing", which would retry a dead endpoint forever and log an ERROR + each time. With no provider the OpenTelemetry API degrades to + `NonRecordingSpan` and `_ProxyCounter`, measured at 0.10 us per + `counter.add`, so every instrumentation call in the codebase becomes a + no-op without a single conditional at a call site. + + **One switch, deliberately.** The bare `OTEL_EXPORTER_OTLP_ENDPOINT` and + `OTEL_SDK_DISABLED` that the SDK reads natively are *not* honoured. Two + switches for one behaviour is how a deployment ends up silently wrong in + a way no test catches; the endpoint below is passed explicitly to both + exporters instead. + + `environment` deliberately reads `STURNUS_SENTRY_ENVIRONMENT` -- the + variable the Sentry work already landed, documented and defaulted in the + chart -- rather than introducing a second name. One environment string + for both back ends, so `deployment.environment.name` in Tempo can never + disagree with the environment filter in Sentry. If the two ever need to + differ, that is a rename of one shared variable, not a second one added + quietly alongside it. + """ + + otel_exporter_otlp_endpoint: str | None = None + otel_traces_sample_ratio: float = 1.0 + otel_metric_export_interval_seconds: float = 60.0 + environment: str = Field(default="production", validation_alias="STURNUS_SENTRY_ENVIRONMENT") + + @field_validator("otel_exporter_otlp_endpoint", mode="after") + @classmethod + def _blank_endpoint_is_absent(cls, value: str | None) -> str | None: + if value is None or not value.strip(): + return None + return value.strip() + + @field_validator("otel_traces_sample_ratio", mode="after") + @classmethod + def _ratio_in_range(cls, value: float) -> float: + if not 0.0 <= value <= 1.0: + raise ValueError( + "STURNUS_OTEL_TRACES_SAMPLE_RATIO must be between 0.0 and 1.0 inclusive" + ) + return value + + class Settings(StrictSettings): discord_token: SecretStr database_url: str diff --git a/src/sturnus/entrypoints/bot.py b/src/sturnus/entrypoints/bot.py index 1943f9c..d562199 100644 --- a/src/sturnus/entrypoints/bot.py +++ b/src/sturnus/entrypoints/bot.py @@ -44,6 +44,14 @@ from sturnus.infrastructure.objectstore import S3AudioStore from sturnus.infrastructure.observability import init_sentry from sturnus.infrastructure.recording_adapters import CryptoEncryptor, FileAudioWriterFactory +from sturnus.infrastructure.telemetry import init_telemetry, shutdown_telemetry +from sturnus.infrastructure.traced import TracedAudioStore, TracedEncryptor, TracedJobQueue +from sturnus.observability.events import Event, log_event, log_exception +from sturnus.observability.setup import ( + asyncio_exception_handler, + configure_logging, + install_excepthooks, +) log = logging.getLogger(__name__) @@ -101,11 +109,25 @@ async def _publish_loop( try: await announce_ready_sessions(sessions, announcer, now) except Exception as exc: - log.warning("Publish sweep failed; will retry next interval: %s", exc) + log_exception( + log, + logging.WARNING, + Event.SWEEP_FAILED, + "Publish sweep failed; will retry next interval", + exc, + reason="publish", + ) try: await link_states.purge_expired(now) except Exception as exc: - log.warning("Expired link-state purge failed; will retry next interval: %s", exc) + log_exception( + log, + logging.WARNING, + Event.SWEEP_FAILED, + "Expired link-state purge failed; will retry next interval", + exc, + reason="link_state_purge", + ) with contextlib.suppress(TimeoutError): await asyncio.wait_for(stop.wait(), timeout=poll_seconds) @@ -139,7 +161,13 @@ async def _wait_for_schema( f"Database schema is missing required table(s): {sorted(missing)}. " "The worker owns migrations and must run them before the bot can start." ) - log.warning("Waiting for the database schema; missing table(s): %s", sorted(missing)) + log_event( + log, + logging.WARNING, + Event.SCHEMA_WAITING, + "Waiting for the worker to migrate the database schema", + missing=sorted(missing), + ) await asyncio.sleep(interval_seconds) @@ -184,6 +212,21 @@ async def _run() -> None: writer_factory = FileAudioWriterFactory(settings.recording_dir) clock: Clock = SystemClock() + # Tracing is applied here, on the way into `SturnusClient` and therefore + # into `RecordingService`. Each wrapper satisfies the same port the plain + # adapter does, so `sturnus.application.recording` gains a span per + # encrypt/upload/enqueue without importing OpenTelemetry -- which it may + # not do (`tests/test_architecture.py`). See + # `sturnus.infrastructure.traced`. + # + # `audio_store` and `encryptor` are wrapped *after* `recover_orphans` + # has used the plain ones below: recovery runs once at startup, outside + # any session, and its spans would be orphaned roots carrying nothing + # a log line does not already say. + traced_audio_store = TracedAudioStore(audio_store) + traced_encryptor = TracedEncryptor(encryptor) + traced_job_repo = TracedJobQueue(job_repo) + # Recovery has no guild to read a per-guild retention override from -- # only a session id parsed off the filesystem -- so it falls back to # the global default rather than guessing at any one guild's setting. @@ -198,9 +241,12 @@ async def _run() -> None: clock.now(), ) if recovered: - log.warning( - "Recovered %d orphaned recording(s) left behind by a previous process", - len(recovered), + log_event( + log, + logging.WARNING, + Event.SESSION_RECOVERED, + "Recovered orphaned recordings left behind by a previous process", + count=len(recovered), ) readiness = ReadinessState() @@ -220,10 +266,10 @@ async def database_ping() -> bool: config_store=config_store, consent_repo=consent_repo, session_repo=session_repo, - job_repo=job_repo, - audio_store=audio_store, + job_repo=traced_job_repo, + audio_store=traced_audio_store, writer_factory=writer_factory, - encryptor=encryptor, + encryptor=traced_encryptor, readiness=readiness, database_ping=database_ping, session_factory=session_factory, @@ -234,6 +280,7 @@ async def database_ping() -> bool: stop = asyncio.Event() loop = asyncio.get_running_loop() + loop.set_exception_handler(asyncio_exception_handler) for sig in (signal.SIGTERM, signal.SIGINT): loop.add_signal_handler(sig, stop.set) @@ -242,7 +289,12 @@ async def database_ping() -> bool: try: await stop.wait() finally: - log.info("Shutdown requested: closing every active session before disconnecting") + log_event( + log, + logging.INFO, + Event.SHUTDOWN_BEGIN, + "Shutdown requested: closing every active session before disconnecting", + ) publish_task.cancel() with contextlib.suppress(asyncio.CancelledError): await publish_task @@ -251,16 +303,28 @@ async def database_ping() -> bool: await client_task await health_runner.cleanup() await engine.dispose() + # Last: flushes the spans describing this very shutdown, which is + # exactly the batch someone will be looking for after a deploy that + # lost a session. + shutdown_telemetry() + log_event(log, logging.INFO, Event.SHUTDOWN_COMPLETE, "Bot stopped") def main() -> None: - # Both run before `_run`, and so before `get_settings()` reads the + # All four run before `_run`, and so before `get_settings()` reads the # environment: with a DSN configured, a settings `ValidationError` is # then itself reported instead of being the one failure Sentry can never # see. Without a DSN, `init_sentry` returns having touched nothing at all # -- see `sturnus.infrastructure.observability`. - logging.basicConfig(level=logging.INFO) + # `configure_logging` first of all: it installs the handler that formats + # and redacts everything the other three might have to report, and + # `install_excepthooks` is what stops a settings `ValidationError` -- + # whose pydantic message embeds the raw environment dict, token prefix + # and all -- reaching stderr unredacted. + configure_logging("bot") + install_excepthooks() init_sentry("bot") + init_telemetry("bot") asyncio.run(_run()) diff --git a/src/sturnus/entrypoints/link.py b/src/sturnus/entrypoints/link.py index 840f24d..289f7c5 100644 --- a/src/sturnus/entrypoints/link.py +++ b/src/sturnus/entrypoints/link.py @@ -35,6 +35,13 @@ from sturnus.infrastructure.documents.outline_oauth import OutlineOAuth from sturnus.infrastructure.linkserver import build_app from sturnus.infrastructure.observability import init_sentry +from sturnus.infrastructure.telemetry import init_telemetry, shutdown_telemetry +from sturnus.observability.events import Event, log_event +from sturnus.observability.setup import ( + asyncio_exception_handler, + configure_logging, + install_excepthooks, +) log = logging.getLogger(__name__) @@ -89,7 +96,13 @@ async def _wait_for_schema( "The worker owns migrations and must run them before the link " "service can start." ) - log.warning("Waiting for the database schema; missing table(s): %s", sorted(missing)) + log_event( + log, + logging.WARNING, + Event.SCHEMA_WAITING, + "Waiting for the worker to migrate the database schema", + missing=sorted(missing), + ) await asyncio.sleep(interval_seconds) @@ -134,7 +147,13 @@ async def _run() -> None: # bind itself (Spec 13.5). site = web.TCPSite(runner, "0.0.0.0", settings.health_port) await site.start() - log.info("Link service listening on port %d", settings.health_port) + log_event( + log, + logging.INFO, + Event.LINK_STARTED, + "Link service listening", + count=settings.health_port, + ) await _wait_for_schema(engine) schema_ready = True @@ -142,25 +161,40 @@ async def _run() -> None: stop = asyncio.Event() loop = asyncio.get_running_loop() + loop.set_exception_handler(asyncio_exception_handler) for sig in (signal.SIGTERM, signal.SIGINT): loop.add_signal_handler(sig, stop.set) try: await stop.wait() finally: - log.info("Shutdown requested: stopping the link service") + log_event( + log, + logging.INFO, + Event.SHUTDOWN_BEGIN, + "Shutdown requested: stopping the link service", + ) await runner.cleanup() await engine.dispose() + shutdown_telemetry() + log_event(log, logging.INFO, Event.SHUTDOWN_COMPLETE, "Link service stopped") def main() -> None: - # Both run before `_run`, and so before `LinkSettings()` reads the + # All four run before `_run`, and so before `LinkSettings()` reads the # environment: with a DSN configured, a settings `ValidationError` is # then itself reported instead of being the one failure Sentry can never # see. Without a DSN, `init_sentry` returns having touched nothing at all # -- see `sturnus.infrastructure.observability`. - logging.basicConfig(level=logging.INFO) + # `configure_logging` first of all: it installs the handler that formats + # and redacts everything the other three might have to report, and + # `install_excepthooks` is what stops a settings `ValidationError` -- + # whose pydantic message embeds the raw environment dict, token prefix + # and all -- reaching stderr unredacted. + configure_logging("link") + install_excepthooks() init_sentry("link") + init_telemetry("link") asyncio.run(_run()) diff --git a/src/sturnus/entrypoints/worker.py b/src/sturnus/entrypoints/worker.py index 090b9a0..088c34a 100644 --- a/src/sturnus/entrypoints/worker.py +++ b/src/sturnus/entrypoints/worker.py @@ -71,16 +71,17 @@ from alembic import command from alembic.config import Config from pydantic import SecretStr -from sqlalchemy import select, text, update +from sqlalchemy import func, select, update from sqlalchemy.exc import SQLAlchemyError from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine +from sturnus.application.documents import DocumentSink from sturnus.application.retention import sweep_expired_audio from sturnus.application.worker import process_one, retry_pending_documents from sturnus.config import StrictSettings from sturnus.infrastructure.crypto import KeyWrapper, decrypt_file from sturnus.infrastructure.db.config_store import ConfigStore -from sturnus.infrastructure.db.models import Session, SessionParticipant +from sturnus.infrastructure.db.models import Session, SessionParticipant, TranscriptionJob from sturnus.infrastructure.db.queue import DEFAULT_LEASE_SECONDS, JobQueue from sturnus.infrastructure.db.repositories import ( AccountLinkRepository, @@ -91,7 +92,29 @@ from sturnus.infrastructure.health import ReadinessState, start_health_server from sturnus.infrastructure.objectstore import S3AudioStore from sturnus.infrastructure.observability import init_sentry +from sturnus.infrastructure.telemetry import ( + JOB_OUTCOME, + QUEUE_DEPTH, + init_telemetry, + record, + set_span_fields, + shutdown_telemetry, + span, +) +from sturnus.infrastructure.traced import ( + TracedAudioDownloader, + TracedDecryptor, + TracedDocumentSink, + TracedQueue, + TracedTranscriptionEngine, +) from sturnus.infrastructure.whisper import WhisperEngine +from sturnus.observability.events import Event, log_event, log_exception +from sturnus.observability.setup import ( + asyncio_exception_handler, + configure_logging, + install_excepthooks, +) log = logging.getLogger(__name__) @@ -227,11 +250,17 @@ def __init__(self, master_key: bytes, master_key_id: str) -> None: def decrypt_to(self, source: Path, target: Path, wrapped: bytes, key_id: str) -> None: if key_id != self._master_key_id: - log.warning( - "Job was encrypted with key id %r, but only %r is configured; " + # ERROR, not WARNING: this is a rotation misconfiguration that + # will silently produce garbage for *every* job, not a transient + # fault that retries away. + log_event( + log, + logging.ERROR, + Event.KEY_ID_MISMATCH, + "Job was encrypted under a different key id than the one configured; " "decrypting with the configured key anyway (no rotation support yet)", - key_id, - self._master_key_id, + key_id=key_id, + configured_key_id=self._master_key_id, ) wrapper = KeyWrapper(self._master_key, self._master_key_id) data_key = wrapper.unwrap(wrapped) @@ -375,13 +404,23 @@ async def _retention_sweep_loop( try: await sweep_expired_audio(jobs, store, datetime.now(UTC)) except Exception as exc: - log.warning("Retention sweep failed; will retry next interval: %s", exc) + log_exception( + log, + logging.WARNING, + Event.SWEEP_FAILED, + "Retention sweep failed; will retry next interval", + exc, + reason="retention", + ) with contextlib.suppress(TimeoutError): await asyncio.wait_for(stop.wait(), timeout=_RETENTION_SWEEP_INTERVAL_SECONDS) async def _document_retry_loop( - documents: OutlineSink, + # The narrow port, not the concrete `OutlineSink`: `_run` passes a + # `TracedDocumentSink` wrapping it, and `retry_pending_documents` only + # ever calls `create`. + documents: DocumentSink, sessions: _WorkerSessionStore, jobs: JobRepository, links: AccountLinkRepository, @@ -398,7 +437,14 @@ async def _document_retry_loop( try: await retry_pending_documents(documents, sessions, jobs, links, config, template_source) except Exception as exc: - log.warning("Document retry sweep failed; will retry next interval: %s", exc) + log_exception( + log, + logging.WARNING, + Event.SWEEP_FAILED, + "Document retry sweep failed; will retry next interval", + exc, + reason="document_retry", + ) with contextlib.suppress(TimeoutError): await asyncio.wait_for(stop.wait(), timeout=_DOCUMENT_RETRY_INTERVAL_SECONDS) @@ -440,6 +486,18 @@ async def _run() -> None: base_url=settings.outline_base_url, api_token=settings.outline_service_key.get_secret_value(), ) + + # Tracing is applied here, on the way into `process_one`, and nowhere + # else. Each wrapper satisfies the same `Protocol` the plain adapter + # does, so `sturnus.application.worker` gains a span per pipeline stage + # without importing OpenTelemetry -- which it may not do + # (`tests/test_architecture.py`) -- and without a single line changing + # in that module. See `sturnus.infrastructure.traced`. + traced_queue = TracedQueue(queue) + traced_engine = TracedTranscriptionEngine(transcription_engine) + traced_store = TracedAudioDownloader(store) + traced_crypto = TracedDecryptor(crypto) + traced_documents = TracedDocumentSink(documents) sessions = _WorkerSessionStore(session_factory) jobs = JobRepository(session_factory) # No fixed provider: `document_provider` is per-guild configuration @@ -453,17 +511,41 @@ async def _run() -> None: readiness = ReadinessState(discord_connected=True) # this process has no gateway to wait on async def database_ping() -> bool: + """Proves the database answers *and* samples the queue depth. + + This replaces a bare `SELECT 1`. A grouped count over + `transcription_job.status` is exactly as good a liveness probe -- + it is still one round trip that either answers or does not -- and + it feeds `sturnus.queue.depth`, the backlog signal that answers + "is the worker keeping up". One query, two purposes, no new + database load. + + Queue depth is a metric and never a span attribute: it is a + property of the system at an instant rather than of any operation, + and hanging it off a job span would make it a lie the moment that + job ended. + """ try: async with session_factory() as session: - await session.execute(text("SELECT 1")) + rows = ( + await session.execute( + select(TranscriptionJob.status, func.count()).group_by( + TranscriptionJob.status + ) + ) + ).all() except SQLAlchemyError: return False + depths = {status: count for status, count in rows} + for status in ("pending", "running", "done", "failed", "dead"): + record(QUEUE_DEPTH, depths.get(status, 0), status=status) return True health_runner = await start_health_server(readiness, settings.health_port) stop = asyncio.Event() loop = asyncio.get_running_loop() + loop.set_exception_handler(asyncio_exception_handler) for sig in (signal.SIGTERM, signal.SIGINT): loop.add_signal_handler(sig, stop.set) @@ -473,36 +555,98 @@ async def database_ping() -> bool: # main loop. retention_task = asyncio.create_task(_retention_sweep_loop(jobs, store, stop)) document_retry_task = asyncio.create_task( - _document_retry_loop(documents, sessions, jobs, links, config_store, template_source, stop) + # The traced sink here too, not the raw one: the retry sweep is a + # second path to the same `DocumentSink.create`, and a histogram + # that covered only one of them would understate how often document + # creation is actually attempted. + _document_retry_loop( + traced_documents, sessions, jobs, links, config_store, template_source, stop + ) ) - log.info("Worker started; polling the transcription queue") + log_event( + log, + logging.INFO, + Event.WORKER_STARTED, + "Worker started; polling the transcription queue", + model=settings.whisper_model, + device="cpu", + compute_type=_WHISPER_COMPUTE_TYPE, + lease_seconds=settings.job_lease_seconds, + max_attempts=settings.max_job_attempts, + ) try: while not stop.is_set(): readiness.database_reachable = await database_ping() - # `process_one` runs to completion before `stop.is_set()` is - # checked again -- a SIGTERM during a job lets that job finish - # rather than abandoning it mid-decrypt (see the module - # docstring). - did_work = await process_one( - queue=queue, - engine=transcription_engine, - store=store, - crypto=crypto, - documents=documents, - sessions=sessions, - jobs=jobs, - links=links, - config=config_store, - work_dir=settings.work_dir, - max_attempts=settings.max_job_attempts, - template_source=template_source, - ) + # The root span of the worker's whole trace. It is opened + # *before* `process_one` claims anything, which is why + # `TracedQueue.claim` stamps `job_id`/`session_id` back onto it + # afterwards -- the ids do not exist yet at this point. Moving + # `process_one` out of this `with` would silently discard those + # attributes; see the matching comment in + # `sturnus.infrastructure.traced.TracedQueue.claim`. + # **This loop no longer labels the outcome, and that is the + # fix.** It used to compute `"done" if did_work else "empty"`, + # but `process_one` returns `True` after `queue.fail(...)` as + # well: the boolean means "work was attempted", never "work + # succeeded", so every failed and every dead job was published + # as `outcome="done"` on both the span and the counter. Nothing + # here can tell the difference, because nothing here sees the + # transition -- so `done` / `failed` / `dead` are now recorded + # by `sturnus.infrastructure.db.queue`, which decides them, and + # stamped onto this span by `traced.TracedQueue`, which sits on + # the way through. + # + # What is left are the two outcomes this loop *does* own, and + # neither is reachable from the queue: nothing was there to + # claim, and `process_one` itself raised. + with span("job.process") as job_span: + try: + # `process_one` runs to completion before + # `stop.is_set()` is checked again -- a SIGTERM during a + # job lets that job finish rather than abandoning it + # mid-decrypt (see the module docstring). + did_work = await process_one( + queue=traced_queue, + engine=traced_engine, + store=traced_store, + crypto=traced_crypto, + documents=traced_documents, + sessions=sessions, + jobs=jobs, + links=links, + config=config_store, + work_dir=settings.work_dir, + max_attempts=settings.max_job_attempts, + template_source=template_source, + ) + except Exception: + # `process_one` routes every failure it can reach + # through `queue.fail`, so reaching this line means the + # loop itself is about to die with a job possibly still + # `running`. A counter as well as a span, deliberately: + # a span is subject to sampling and to Tempo's + # retention, and "the worker died holding a job" must + # not depend on either. + set_span_fields(job_span, outcome="crashed") + record(JOB_OUTCOME, 1, outcome="crashed") + raise + if not did_work: + # Not counted, only labelled: an empty poll happens + # every `_POLL_SECONDS` for as long as the queue is + # idle, and `sturnus.queue.depth` already answers "is + # there work" without a counter that ticks forever. + set_span_fields(job_span, outcome="empty") if not did_work: with contextlib.suppress(TimeoutError): await asyncio.wait_for(stop.wait(), timeout=_POLL_SECONDS) finally: - log.info("Shutdown requested: worker stopping after its current job") + log_event( + log, + logging.INFO, + Event.SHUTDOWN_BEGIN, + "Shutdown requested: worker stopping after its current job", + ) for task in (retention_task, document_retry_task): task.cancel() for task in (retention_task, document_retry_task): @@ -510,16 +654,29 @@ async def database_ping() -> bool: await task await health_runner.cleanup() await engine.dispose() + # Last, and after the health server is down: flushes the batch of + # spans describing this very shutdown, which is exactly the batch + # someone will be looking for afterwards. + shutdown_telemetry() + log_event(log, logging.INFO, Event.SHUTDOWN_COMPLETE, "Worker stopped") def main() -> None: - # Both run before `_run`, and so before `WorkerSettings()` reads the + # All four run before `_run`, and so before `WorkerSettings()` reads the # environment: with a DSN configured, a settings `ValidationError` is # then itself reported instead of being the one failure Sentry can never # see. Without a DSN, `init_sentry` returns having touched nothing at all # -- see `sturnus.infrastructure.observability`. - logging.basicConfig(level=logging.INFO) + # + # `configure_logging` is first of all: it installs the handler that + # formats and redacts everything the other three might have to report, + # and `install_excepthooks` is what stops a settings `ValidationError` + # -- whose pydantic message embeds the raw environment dict, token + # prefix and all -- reaching stderr unredacted. + configure_logging("worker") + install_excepthooks() init_sentry("worker") + init_telemetry("worker") asyncio.run(_run()) diff --git a/src/sturnus/infrastructure/db/queue.py b/src/sturnus/infrastructure/db/queue.py index ca27ed3..e7702e5 100644 --- a/src/sturnus/infrastructure/db/queue.py +++ b/src/sturnus/infrastructure/db/queue.py @@ -55,6 +55,7 @@ from __future__ import annotations +import logging from dataclasses import dataclass from datetime import UTC, datetime, timedelta @@ -62,6 +63,10 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker from sturnus.infrastructure.db.models import Session, TranscriptionJob +from sturnus.infrastructure.telemetry import JOB_OUTCOME, record +from sturnus.observability.events import Event, log_event + +log = logging.getLogger(__name__) #: How long a claimed job may stay `running` before `claim` treats it as #: abandoned and reclaims it. Generous on purpose: a large-v3 Whisper model @@ -200,14 +205,35 @@ async def complete(self, job_id: int, transcript: str) -> bool: select(Session.status).where(Session.id == session_id) ) await session.commit() - return remaining == 0 and session_status == "closed" - async def fail(self, job_id: int, error: str, max_attempts: int) -> None: + # **Where `sturnus.job.outcome` is counted, and the reason it is + # counted here.** The worker loop used to derive the label from + # `process_one`'s return value, which is `True` after `queue.fail` + # just as it is after `queue.complete` -- it means "work was + # attempted", not "work succeeded" -- so every failed job was + # published as `outcome="done"`. A metric that reports failures as + # successes is worse than no metric, because it is believed. + # + # This method and `fail` below are the two transitions that decide + # a job's terminal state, so they are the two places that can say + # what happened without inferring it. Recorded after the commit: + # the counter must not claim a `done` that a failed transaction + # rolled back. + record(JOB_OUTCOME, 1, outcome="done") + return remaining == 0 and session_status == "closed" + + async def fail(self, job_id: int, error: str, max_attempts: int) -> bool: """Records the error and either returns the job to `pending` or, once - `attempts` reaches `max_attempts`, marks it `dead`. + `attempts` reaches `max_attempts`, marks it `dead`. Returns whether + it is now dead. A `dead` job is excluded from `complete`'s remaining-jobs count, so one unreadable recording never blocks its session's completion. + + The return value exists because the caller otherwise cannot tell + permanent loss from a retry -- `process_one` returns `True` for both + -- and the distinction is the whole point of the outcome metric and + of the `job.process` span's `outcome` attribute. """ async with self._session_factory() as session: job = await session.get(TranscriptionJob, job_id) @@ -215,8 +241,53 @@ async def fail(self, job_id: int, error: str, max_attempts: int) -> None: job.attempts += 1 job.error = error job.status = "dead" if job.attempts >= max_attempts else "pending" + session_id = job.session_id + attempts = job.attempts + dead = job.status == "dead" await session.commit() + if dead: + # A speaker's audio will never be transcribed. This method has + # set `status="dead"` since it was written and said nothing at + # all about it -- permanent loss, expressed as silence. + # + # `error` is **not** logged: it is `str(exc)` from + # `process_one`, which is fine in the database column an + # operator queries deliberately and is exactly what must not go + # into a retained, indexed store. The column still has it. + log_event( + log, + logging.ERROR, + Event.JOB_DEAD, + "Job exhausted its attempts and is now dead; this recording will never " + "be transcribed", + job_id=job_id, + session_id=session_id, + attempts=attempts, + max_attempts=max_attempts, + ) + record(JOB_OUTCOME, 1, outcome="dead") + else: + log_event( + log, + logging.WARNING, + Event.JOB_FAILED, + "Job returned to the queue for another attempt", + job_id=job_id, + session_id=session_id, + attempt=attempts, + max_attempts=max_attempts, + ) + # The measurement that was missing entirely. `dead` was counted + # from the moment this metric existed; a retryable failure was + # not counted at all, and the worker loop then counted the same + # job as `done`. So the two labels an operator most needs -- + # "is this job pipeline failing" and "is it failing + # permanently" -- were one lie and one silence. + record(JOB_OUTCOME, 1, outcome="failed") + + return dead + async def last_error(self, job_id: int) -> str | None: async with self._session_factory() as session: return await session.scalar( diff --git a/src/sturnus/infrastructure/discord/audio_cog.py b/src/sturnus/infrastructure/discord/audio_cog.py index fbef3c6..48459d8 100644 --- a/src/sturnus/infrastructure/discord/audio_cog.py +++ b/src/sturnus/infrastructure/discord/audio_cog.py @@ -35,6 +35,7 @@ from sturnus.application.ports import AudioStore, Clock from sturnus.infrastructure.db.models import TranscriptionJob from sturnus.infrastructure.discord.permissions import require_admin +from sturnus.observability.events import Event, log_event log = logging.getLogger(__name__) @@ -106,7 +107,18 @@ async def delete(self, interaction: discord.Interaction) -> None: count = await _erase_audio( self._session_factory, self._audio_store, interaction.user.id, self._clock.now() ) - log.info("Erased %d recording(s) for user %d via /audio delete", count, interaction.user.id) + # The user id is logged deliberately: "did this person's `/audio + # delete` actually erase their recordings" is a compliance question, + # and it cannot be answered without it. + log_event( + log, + logging.INFO, + Event.AUDIO_ERASED, + "Erased recordings at the owner's request", + discord_user_id=interaction.user.id, + deleted=count, + reason="self_delete", + ) await interaction.response.send_message( f"Deleted {count} recording(s) of your audio. {_TRANSCRIPTS_UNTOUCHED_NOTICE}", ephemeral=True, @@ -121,11 +133,14 @@ async def purge(self, interaction: discord.Interaction, user: discord.User) -> N count = await _erase_audio( self._session_factory, self._audio_store, user.id, self._clock.now() ) - log.info( - "Erased %d recording(s) for user %d via /audio purge (requested by %d)", - count, - user.id, - interaction.user.id, + log_event( + log, + logging.INFO, + Event.AUDIO_ERASED, + "Erased recordings on an administrator's request", + discord_user_id=user.id, + deleted=count, + reason="admin_purge", ) await interaction.response.send_message( f"Deleted {count} recording(s) for {user.mention}. {_TRANSCRIPTS_UNTOUCHED_NOTICE}", diff --git a/src/sturnus/infrastructure/discord/client.py b/src/sturnus/infrastructure/discord/client.py index 318ad3f..5186c28 100644 --- a/src/sturnus/infrastructure/discord/client.py +++ b/src/sturnus/infrastructure/discord/client.py @@ -26,6 +26,7 @@ import asyncio import logging +import time from collections.abc import Awaitable, Callable from dataclasses import dataclass, replace from datetime import datetime, timedelta @@ -49,7 +50,7 @@ RunningState, plan_reconfigure, ) -from sturnus.application.recording import RecordingService +from sturnus.application.recording import JobQueue, RecordingService from sturnus.domain import settings from sturnus.domain.session import EndReason, SessionTimeouts from sturnus.infrastructure.db.config_store import ConfigStore @@ -57,7 +58,6 @@ from sturnus.infrastructure.db.repositories import ( AccountLinkRepository, ConsentRepository, - JobRepository, SessionRepository, ) from sturnus.infrastructure.discord.about_cog import AboutCog @@ -68,9 +68,18 @@ from sturnus.infrastructure.discord.link_cog import LinkCog from sturnus.infrastructure.discord.queue_cog import QueueCog from sturnus.infrastructure.discord.setup_cog import SetupCog -from sturnus.infrastructure.discord.voice import VoiceReceiveAdapter +from sturnus.infrastructure.discord.voice import VoiceReceiveAdapter, voice_close_code from sturnus.infrastructure.documents.outline_oauth import OutlineOAuth from sturnus.infrastructure.health import ReadinessState +from sturnus.infrastructure.telemetry import ( + SESSION_ACTIVE, + SESSION_CLOSE_DURATION, + SESSION_DURATION, + fail_span, + record, + span, +) +from sturnus.observability.events import Event, log_event, log_exception log = logging.getLogger(__name__) @@ -125,6 +134,19 @@ class _GuildRecording: #: loop once it has passed -- a guard that only an operator can clear #: turns a transient fault into an outage. blocked_until: datetime | None = None + #: Monotonic timestamp of the moment this guild's session started + #: recording, or `None` when it is idle. Kept here rather than on + #: `RecordingService` because it exists purely to feed + #: `sturnus.session.duration` and `sturnus.session.active`, and + #: `RecordingService` is `application` code that may not know + #: OpenTelemetry exists. + #: + #: It is also the idempotence key for `_record_session_close`: a + #: session can now reach its close through the sweep, through + #: `/config apply force:true` and through `graceful_shutdown`, and + #: `sturnus.session.active` must be decremented exactly once whichever + #: of them gets there. + started_monotonic: float | None = None @property def channel_id(self) -> int: @@ -145,7 +167,13 @@ def __init__( config_store: ConfigStore, consent_repo: ConsentRepository, session_repo: SessionRepository, - job_repo: JobRepository, + # Typed against the narrow `JobQueue` port rather than the concrete + # `JobRepository`: the only thing this class does with it is hand it + # to `RecordingService`, and `sturnus.entrypoints.bot` passes a + # `TracedJobQueue` wrapper around the real repository. Widening it + # here is what lets the tracing decorator be applied at the + # composition root instead of reaching into this class. + job_repo: JobQueue, audio_store: AudioStore, writer_factory: AudioWriterFactory, encryptor: Encryptor, @@ -261,7 +289,13 @@ async def on_ready(self) -> None: for guild in self.guilds: await self.reconcile_guild(guild.id) self._readiness.discord_connected = True - log.info("Connected to Discord; configured for %d guild(s)", len(self._guilds)) + log_event( + log, + logging.INFO, + Event.BOT_CONNECTED, + "Connected to Discord", + count=len(self._guilds), + ) async def on_guild_join(self, guild: discord.Guild) -> None: """Configures a guild that invited the bot while the process was running. @@ -295,12 +329,35 @@ async def _desired_config( channel = snapshot.get(settings.VOICE_CHANNEL_ID) role = snapshot.get(settings.CONSENT_ROLE_ID) if channel is None or role is None: - self._notice( + # `missing` is the addition that matters: the prose line says a + # guild is unconfigured without saying which key is absent, so + # the operator's next step was always "run /config show and read + # it yourself". Emitted behind `_notice`'s return value rather + # than beside it, so the structured line inherits the same + # per-guild deduplication -- a guild nobody has configured is + # reconciled every ten seconds forever. + if self._notice( guild_id, "Guild %d is missing voice_channel_id and/or consent_role_id; " "an administrator must run /config show to see what's missing.", guild_id, - ) + ): + log_event( + log, + logging.WARNING, + Event.GUILD_UNCONFIGURED, + "Guild cannot record: required configuration is missing. " + "An administrator must run /config show.", + guild_id=guild_id, + missing=[ + key + for key, value in ( + (settings.VOICE_CHANNEL_ID, channel), + (settings.CONSENT_ROLE_ID, role), + ) + if value is None + ], + ) return None try: desired = GuildRuntimeConfig( @@ -326,18 +383,28 @@ async def _desired_config( self._clear_notice(guild_id) return desired - def _notice(self, guild_id: int, message: str, *args: object) -> None: + def _notice(self, guild_id: int, message: str, *args: object) -> bool: """Logs a per-guild configuration complaint once, not once per tick. Deduplicated on the *rendered* text, not the template, so a value that changes from one bad state to another is reported again while the same complaint repeating every ten seconds is not. + + Returns whether this call was the one that logged, so a caller with + a structured `log_event` to emit alongside the prose can hang it off + the same deduplication instead of building a second one that drifts. + + The rendered text goes through `%s` rather than being the format + string: `LogRecord.msg` stays a literal, which is the one thing + `sturnus.infrastructure.observability.scrub_event` forwards to + Sentry. """ rendered = message % args if self._config_notices.get(guild_id) == rendered: - return + return False self._config_notices[guild_id] = rendered log.warning("%s", rendered) + return True def _clear_notice(self, guild_id: int) -> None: """Forgets a guild's last complaint, so a fixed value is reported again.""" @@ -566,10 +633,13 @@ async def _build(self, guild_id: int, desired: GuildRuntimeConfig) -> None: self._guilds[guild_id] = _GuildRecording( config=desired, service=service, voice=self._make_voice(service) ) - log.info( - "Guild %d is now configured; watching voice channel %d.", - guild_id, - desired.channel_id, + log_event( + log, + logging.INFO, + Event.GUILD_CONFIGURED, + "Guild is armed to record; watching its voice channel", + guild_id=guild_id, + channel_id=desired.channel_id, ) await self._sync_participants(self.get_guild(guild_id), self._guilds[guild_id]) @@ -685,6 +755,19 @@ async def _sync_participants( was_recording = recording.service.is_recording await recording.service.participants_changed(consented_count, self._clock.now()) if recording.service.is_recording and not was_recording: + # The session's bookkeeping opens here, not inside + # `_start_capture`. The session row exists from this moment + # whether or not `join()` then works, and `_start_capture` + # deliberately swallows a failed join into + # `EndReason.CAPTURE_FAILURE` -- so the close path decrements + # `sturnus.session.active` for a capture failure too. Pairing + # the increment with the *session* rather than with the join is + # what keeps that counter from drifting negative. + recording.started_monotonic = time.monotonic() + # An up/down counter, so "is anything recording right now" and + # "did a session leak" are both one query. A gauge would need a + # callback on the reader's own thread, which has no event loop. + record(SESSION_ACTIVE, 1, guild_id=recording.service.guild_id) await self._start_capture(recording) async def _start_capture(self, recording: _GuildRecording) -> None: @@ -697,16 +780,44 @@ async def _start_capture(self, recording: _GuildRecording) -> None: it, leaves the channel and resets, and the row says we could not hear rather than that there was nothing to hear. """ - try: - await recording.voice.join(recording.channel_id) - except Exception: - log.exception( - "Could not start voice capture in channel %d; ending the session as %s " - "instead of leaving it open with nothing arriving.", - recording.channel_id, - EndReason.CAPTURE_FAILURE.value, - ) - recording.service.request_close(EndReason.CAPTURE_FAILURE) + # `join()` is a gateway round trip, a libopus probe and a + # `listen()` call, and it is the only step between "a session row + # exists" and "audio is arriving". A span over exactly it is what + # separates a slow join from a slow meeting in a trace. + with span( + "session.open", + guild_id=recording.service.guild_id, + channel_id=recording.channel_id, + session_id=recording.service.session_id, + ) as active: + try: + await recording.voice.join(recording.channel_id) + except Exception as exc: + # The exception is swallowed here on purpose (see the + # docstring), so the span has to be marked by hand: `span` + # only marks the ones that propagate out of it. + fail_span(active, exc) + log_exception( + log, + logging.ERROR, + Event.VOICE_JOIN_FAILED, + "Could not start voice capture; ending the session rather than " + "leaving it open with nothing arriving.", + exc, + guild_id=recording.service.guild_id, + channel_id=recording.channel_id, + session_id=recording.service.session_id, + end_reason=EndReason.CAPTURE_FAILURE.value, + # The one join failure whose type name says nothing + # useful: `discord.ConnectionClosed` is what Discord + # raises for "session no longer valid", "you were + # moved", "rate limited" and "voice server crashed" + # alike, and its message is withheld by + # `redaction.SAFE_MESSAGE_TYPES`. The code separates + # them; see `voice.voice_close_code`. + close_code=voice_close_code(exc), + ) + recording.service.request_close(EndReason.CAPTURE_FAILURE) async def _return_to_idle(self, guild_id: int, recording: _GuildRecording) -> None: """Leaves the channel and puts the machine back where a session can start. @@ -735,11 +846,17 @@ async def _return_to_idle(self, guild_id: int, recording: _GuildRecording) -> No return try: await recording.voice.leave() - except Exception: - log.exception( - "Guild %d: leaving the voice channel failed; continuing anyway so the " - "guild is able to record again.", - guild_id, + except Exception as exc: + log_exception( + log, + logging.ERROR, + Event.VOICE_LEFT_FAILED, + "Leaving the voice channel failed; continuing anyway so the guild is " + "able to record again.", + exc, + guild_id=guild_id, + channel_id=recording.channel_id, + session_id=recording.service.session_id, ) recording.service.reset() @@ -769,9 +886,15 @@ async def _end_session_now(self, guild_id: int, recording: _GuildRecording) -> N """ if not recording.service.is_recording: return + session_id = recording.service.session_id try: await recording.service.end_now(SHUTDOWN_END_REASON, self._clock.now()) finally: + # In the `finally` for the same reason `_return_to_idle` is: a + # close that raised mid-upload still ended the session, and a + # `sturnus.session.active` that only comes down on the happy + # path is a counter that climbs forever. + self._record_session_close(recording, SHUTDOWN_END_REASON, session_id) await self._return_to_idle(guild_id, recording) async def _teardown(self, guild_id: int, recording: _GuildRecording) -> None: @@ -953,8 +1076,15 @@ async def _tick_all(self, now: datetime) -> None: for guild_id in sorted(guild_ids): try: await self._tick_guild(guild_id, now) - except Exception: - log.exception("Tick failed for guild %d; other guilds are unaffected.", guild_id) + except Exception as exc: + log_exception( + log, + logging.ERROR, + Event.GUILD_TICK_FAILED, + "The periodic tick failed for this guild; every other guild is unaffected.", + exc, + guild_id=guild_id, + ) async def _tick_guild(self, guild_id: int, now: datetime) -> None: """Closes a due session, lands anything deferred, then reconciles. @@ -1030,30 +1160,49 @@ async def _sweep_due_session( """ reason: EndReason | None = None closed = False + # Read before `tick()`, because a successful close is followed by + # `reset()` and the id is gone by the time there is anything to say + # about it. + session_id = recording.service.session_id try: reason = await recording.service.tick(now) closed = reason is not None - except Exception: + except Exception as exc: # Not necessarily a failed close: `tick()` could equally have # raised before deciding anything, in which case nothing ever # moved to CLOSING and there is nothing to recover from. closed = recording.service.needs_reset if closed: - log.exception( - "Guild %d: closing the due session failed; its audio may not have " - "been uploaded (recover_orphans picks that up on the next start). " + log_exception( + log, + logging.ERROR, + Event.SESSION_CLOSE_FAILED, + "Closing the due session failed; its audio may not have been " + "uploaded (recover_orphans picks that up on the next start). " "Returning the guild to a recordable state so it does not stop " "recording silently.", - guild_id, + exc, + guild_id=guild_id, + channel_id=recording.channel_id, + session_id=session_id, + reason="timeout_sweep", ) else: - log.exception( - "Guild %d: the timeout sweep failed before closing anything; the " - "session in progress is untouched.", - guild_id, + log_exception( + log, + logging.ERROR, + Event.GUILD_TICK_FAILED, + "The timeout sweep failed before closing anything; the session in " + "progress is untouched.", + exc, + guild_id=guild_id, + session_id=session_id, ) if not closed: return + # Before `_return_to_idle`, which resets the service: after it, + # there is no session left to attribute the measurement to. + self._record_session_close(recording, reason, session_id) await self._return_to_idle(guild_id, recording) if reason is not None and reason in CAPTURE_FAILURE_REASONS: self._begin_capture_cooldown(recording, reason, now) @@ -1072,12 +1221,21 @@ def _begin_capture_cooldown( announcing to the channel that it is being recorded. """ recording.blocked_until = now + REJOIN_COOLDOWN - log.error( - "The session in channel %d ended with %s; not recording there again before %s. " - "Investigate before then: a rejoin would meet the same fault.", - recording.channel_id, - reason.value, - recording.blocked_until.isoformat(), + # `duration_seconds` rather than the absolute `blocked_until`: the + # line carries its own `ts`, so the two together give the moment + # the guard lifts, and the registry has no field for a timestamp + # precisely because every line already has one. + log_event( + log, + logging.ERROR, + Event.VOICE_REJOIN_BLOCKED, + "The session in this channel ended because we could not hear it; not " + "recording there again until the cooldown passes. Investigate before then: " + "a rejoin would meet the same fault.", + guild_id=recording.service.guild_id, + channel_id=recording.channel_id, + end_reason=reason.value, + duration_seconds=REJOIN_COOLDOWN.total_seconds(), ) async def _end_capture_cooldown(self, guild_id: int, recording: _GuildRecording) -> None: @@ -1115,15 +1273,106 @@ async def graceful_shutdown(self) -> None: # Each guild is isolated: SIGTERM gives us one pass at this, # and one guild whose upload fails must not cost every guild # after it in the dict the session it is still holding open. + was_recording = recording.service.is_recording + session_id = recording.service.session_id + started = time.monotonic() + outcome = "ok" try: # No `reset()` afterwards, deliberately: the process is # going away, and a machine left in CLOSING cannot offer a # session that would never be recorded. - await recording.service.end_now(SHUTDOWN_END_REASON, self._clock.now()) + # + # The highest-consequence span in the system, and the reason + # `sturnus.session.close.duration` exists. `end_now()` + # encrypts, uploads and enqueues **serially, per speaker**, + # and it runs during SIGTERM. If six speakers take longer + # than `terminationGracePeriodSeconds`, Kubernetes kills the + # pod mid-loop and Spec 15's "the entire session is lost, + # not just a portion" is what happens. Per-guild isolation + # changes the blast radius of that, not the question: + # comparing this histogram's p99 to the grace period is what + # turns "we lost an evening during a deploy" into a number + # somebody can act on beforehand. + with span( + "session.close", + guild_id=guild_id, + session_id=session_id, + end_reason=SHUTDOWN_END_REASON.value, + ): + await recording.service.end_now(SHUTDOWN_END_REASON, self._clock.now()) await recording.voice.leave() - except Exception: - log.exception( - "Guild %d: closing its session during shutdown failed; its audio " - "may be left for recover_orphans. Other guilds are unaffected.", - guild_id, + except Exception as exc: + # `outcome` is the label main's rewrite made worth having: + # shutdown is per-guild now, so "the close ran" and "the + # close ran and worked" are genuinely different questions + # and the histogram can answer both. + outcome = "error" + log_exception( + log, + logging.ERROR, + Event.SESSION_CLOSE_FAILED, + "Closing this guild's session during shutdown failed; its audio may " + "be left for recover_orphans. Every other guild is unaffected.", + exc, + guild_id=guild_id, + session_id=session_id, + reason="shutdown", ) + finally: + if was_recording: + record( + SESSION_CLOSE_DURATION, + time.monotonic() - started, + end_reason=SHUTDOWN_END_REASON.value, + outcome=outcome, + ) + self._record_session_close(recording, SHUTDOWN_END_REASON, session_id) + + def _record_session_close( + self, + recording: _GuildRecording, + reason: EndReason | None, + session_id: int | None, + ) -> None: + """Closes out one session's metrics. Idempotent per session. + + `end_reason` is an `EndReason` member -- a fixed source literal, so + bounded as a metric label -- and it answers a question nothing else + does: are sessions ending because people left, because the idle + timeout fired, because a deploy cut them short, or because this + process could not hear the channel? Those are four different + operational stories that all look like "session closed" today, and + the last two are the ones that cost a meeting. + + Called from every path a session can now end on -- the timeout + sweep, `/config apply force:true`, and `graceful_shutdown` -- which + is why it has to be idempotent rather than merely careful: + `started_monotonic` is both the measurement's start and the "this + session has already been accounted for" flag. + + `reason=None` means `tick()` raised after moving the machine to + CLOSING, so the session did end and nothing can say why. That is + recorded as `unknown` rather than skipped: skipping it would leave + `sturnus.session.active` counting a session that no longer exists, + which is worse than a label admitting ignorance. + """ + if recording.started_monotonic is None: + return + end_reason = reason.value if reason is not None else "unknown" + record( + SESSION_DURATION, + time.monotonic() - recording.started_monotonic, + end_reason=end_reason, + guild_id=recording.service.guild_id, + ) + record(SESSION_ACTIVE, -1, guild_id=recording.service.guild_id) + recording.started_monotonic = None + log_event( + log, + logging.DEBUG, + Event.SESSION_CLOSING, + "Session bookkeeping closed out", + session_id=session_id, + guild_id=recording.service.guild_id, + reason=end_reason, + ) diff --git a/src/sturnus/infrastructure/discord/decoding.py b/src/sturnus/infrastructure/discord/decoding.py index 208c0f2..be359c7 100644 --- a/src/sturnus/infrastructure/discord/decoding.py +++ b/src/sturnus/infrastructure/discord/decoding.py @@ -232,7 +232,17 @@ def conceal(self) -> bytes | None: # frame says nothing about the input stream, so it is not # counted as a discard. Not narrowed to `OpusError` either -- # nothing may escape towards the packet-router thread. - log.debug("Packet-loss concealment failed for ssrc=%s: %r", self._ssrc, error) + # The exception's *type*, never its `repr`. `%r` of an + # arbitrary exception is `str(exc)` in a wrapper, and + # `tests/test_logging_discipline.py` rule R6 forbids that + # spelling for a reason that applies here too: nothing + # constrains what a third-party exception carries in its + # message, and this line runs on every concealed frame. + log.debug( + "Packet-loss concealment failed for ssrc=%s (%s)", + self._ssrc, + type(error).__qualname__, + ) return None def _discard(self, code: int | None, error: BaseException) -> None: diff --git a/src/sturnus/infrastructure/discord/queue_cog.py b/src/sturnus/infrastructure/discord/queue_cog.py index 292a248..d78ca09 100644 --- a/src/sturnus/infrastructure/discord/queue_cog.py +++ b/src/sturnus/infrastructure/discord/queue_cog.py @@ -75,6 +75,7 @@ from sturnus.infrastructure.db.models import Session, SessionParticipant, TranscriptionJob from sturnus.infrastructure.db.queue import DEFAULT_LEASE_SECONDS from sturnus.infrastructure.discord.permissions import require_admin +from sturnus.observability.events import Event, log_exception log = logging.getLogger(__name__) @@ -961,7 +962,16 @@ async def _disable(self) -> None: try: await self.message.edit(view=self) except discord.HTTPException as exc: - log.warning("Could not disable the /queue requeue buttons: %s", exc) + log_exception( + log, + logging.WARNING, + Event.QUEUE_VIEW_DISABLE_FAILED, + "Could not grey out the /queue requeue buttons; a live button on a " + "stopped view is refused by the re-check anyway", + exc, + guild_id=self._guild_id, + session_id=self._session_id, + ) @discord.ui.button(label="Confirm", style=discord.ButtonStyle.danger) async def confirm( diff --git a/src/sturnus/infrastructure/discord/sink.py b/src/sturnus/infrastructure/discord/sink.py index 2f7d2b6..681be3f 100644 --- a/src/sturnus/infrastructure/discord/sink.py +++ b/src/sturnus/infrastructure/discord/sink.py @@ -56,6 +56,7 @@ from discord.ext import voice_recv from sturnus.application.ports import Clock +from sturnus.infrastructure.telemetry import VOICE_PACKETS, record log = logging.getLogger(__name__) @@ -144,10 +145,19 @@ def __init__( decoder: OpusDecoderPool, clock: Clock, emit: Callable[[CaptureMessage], None], + guild_id: int | None = None, ) -> None: # No destination: this sink is an endpoint, not a link in a # transformer chain, so it registers no child. super().__init__() + # Metric label only, and optional so the sink stays constructible + # from a list of packets with no guild anywhere in sight -- which + # is the property `tests/infrastructure/discord/test_sink.py` + # exists to keep. `guild_id` is the one non-literal in + # `METRIC_LABEL_FIELDS` (see `sturnus.observability.fields`), and + # it is what makes "which server stopped being recorded" a query + # rather than a log search. + self._guild_id = guild_id self._consent_role_id = consent_role_id self._decoder = decoder self._clock = clock @@ -240,6 +250,7 @@ def _write( # the mapping with its speaking event) or the user is not a # guild member at all. Nothing is decoded and nothing is # written -- but it is not silent either. + record(VOICE_PACKETS, 1, outcome="unknown_user", guild_id=self._guild_id) self._note_unattributed(ssrc) return @@ -250,6 +261,12 @@ def _write( # immediately. Deliberately *before* the decoder: audio nobody # consented to is never even turned into PCM, and no decoder # object is ever created for that speaker. + # + # Counted, because "nobody consented" and "capture is broken" + # produce the same silence in the recording and the same empty + # session row. The counter is what separates them without + # anything per-frame reaching Loki. + record(VOICE_PACKETS, 1, outcome="no_role", guild_id=self._guild_id) return # `data.opus` is `packet.decrypted_data`, already stripped of RTP @@ -265,6 +282,12 @@ def _write( # RTP-derived absolute time, so this becomes exactly one # frame of real silence in the WAV and nothing after it # shifts. + # + # The rate of this label against `recorded` is the early + # warning `.decoding`'s threshold deliberately does not give: + # that fires once, after five consecutive seconds of nothing, + # and this is visible from the first frame. + record(VOICE_PACKETS, 1, outcome="undecodable", guild_id=self._guild_id) return self._emit( diff --git a/src/sturnus/infrastructure/discord/voice.py b/src/sturnus/infrastructure/discord/voice.py index 513cf04..6bf668a 100644 --- a/src/sturnus/infrastructure/discord/voice.py +++ b/src/sturnus/infrastructure/discord/voice.py @@ -72,9 +72,57 @@ RecordingSink, SpeakerStreamEnded, ) +from sturnus.infrastructure.telemetry import VOICE_PACKET_ERRORS, VOICE_PACKETS, record +from sturnus.observability.events import Event, RateLimiter, log_event, log_exception log = logging.getLogger(__name__) +#: One line per thousand occurrences, plus the first. The event below is +#: per-frame in origin -- ~50/s per speaker -- and an unrate-limited +#: `log.exception` on each is its own outage during exactly the systematic +#: failure that makes it worth reading. The aggregate lives in +#: `sturnus.voice.packet_errors`, which answers "is the adapter throwing" +#: without forty thousand identical tracebacks standing in the way. +_MESSAGE_ERROR_LOG_EVERY = 1000 + + +def voice_close_code(error: BaseException | None) -> int | None: + """The websocket close code of a `discord.ConnectionClosed`, else `None`. + + **Why a field and not an entry in `redaction.SAFE_MESSAGE_TYPES`.** + `ConnectionClosed` is not on that tuple, so `safe_exception_message` + reduces it to `` -- + in exactly the situation the detail is wanted, because this is the + exception the bot gets when it cannot join voice or is dropped from it. + + Its message was read rather than guessed. `discord/errors.py` builds it + as `f'Shard ID {self.shard_id} WebSocket closed with {self.code}'` -- + two integers and no third-party text at all, `reason` being set to `''` + unconditionally a line above with the comment "aiohttp doesn't seem to + consistently provide close reason". So the message *is* safe, and + admitting it would be defensible on its own terms. It is still not + admitted, for a structural reason that outweighs it: + `sturnus.observability` is standard-library-only by construction -- + `tests/observability/test_package_boundaries.py` enforces it, and it is + what lets `sturnus.application` import the field registry at all -- so + `SAFE_MESSAGE_TYPES` cannot name a type from `discord` without + dragging the gateway library into the package that decides what leaves + the pod. One list would become two. + + So the diagnosis is lifted out as a registered field instead, which is + strictly better than the sentence it came from: `close_code=4014` + ("disconnected by Discord"), `4006` ("session no longer valid"), `4009` + ("session timeout"), `4015` ("voice server crashed") and `4021` + ("rate limited") are each a different answer to "why can this bot not + hear the channel", and as a field they are queryable in Loki and + groupable rather than being characters in a message. + + Returns `None` for every other exception type, which is the honest + answer: `OSError`, `asyncio.TimeoutError` and `OpusNotLoaded` all reach + the same call sites and none of them has a close code. + """ + return error.code if isinstance(error, discord.ConnectionClosed) else None + class VoiceReceiveAdapter: """Satisfies the `VoiceReceiver` port over `discord-ext-voice-recv`.""" @@ -102,6 +150,7 @@ def __init__( self._drain_task: asyncio.Task[None] | None = None self._guild_id: int | None = None self._stopping = False + self._message_errors = RateLimiter(_MESSAGE_ERROR_LOG_EVERY) async def join(self, channel_id: int) -> None: """Connects to the voice channel and starts listening on a sink. @@ -140,6 +189,9 @@ async def join(self, channel_id: int) -> None: self._stopping = False self._loop = asyncio.get_running_loop() self._queue = asyncio.Queue() + # Per session, so `count` on the line below reads as "errors in this + # recording" rather than "errors since the process started". + self._message_errors = RateLimiter(_MESSAGE_ERROR_LOG_EVERY) self._drain_task = asyncio.create_task(self._drain(self._queue)) try: self._start_listening(int(stored_role_id)) @@ -148,9 +200,20 @@ async def join(self, channel_id: int) -> None: # the bot would sit in the channel recording nothing. await self.leave() raise + log_event( + log, + logging.INFO, + Event.VOICE_JOINED, + "Joined the voice channel and started listening", + guild_id=channel.guild.id, + channel_id=channel_id, + session_id=self._service.session_id, + listening=True, + ) async def leave(self) -> None: """Stops listening and disconnects.""" + guild_id = self._guild_id self._stopping = True voice_client, self._voice_client = self._voice_client, None if voice_client is not None: @@ -171,6 +234,19 @@ async def leave(self) -> None: self._queue = None self._loop = None + # The counterpart of `voice.joined`, and the reason + # `_on_listen_stopped` does not log a clean stop: a stop we asked + # for is reported by the side that asked. Everything else that + # reaches that hook is a failure and says so. + log_event( + log, + logging.INFO, + Event.VOICE_LEFT, + "Stopped listening and left the voice channel", + guild_id=guild_id, + session_id=self._service.session_id, + listening=False, + ) # -- capture side: everything below runs on the extension's threads -- @@ -191,6 +267,7 @@ def _start_listening(self, consent_role_id: int) -> None: decoder=decoder, clock=self._clock, emit=self._emit, + guild_id=self._guild_id, ) self._voice_client.listen(sink, after=self._on_listen_stopped) @@ -212,6 +289,14 @@ def _emit(self, message: CaptureMessage) -> None: except RuntimeError: # The loop is closed. There is nothing left to deliver to, and # certainly nothing to raise about back into `write()`. + # + # Counted only for frames: `sturnus.voice.packets` is "voice + # packets by what happened to them", and a `CaptureStopped` + # that never landed is not a packet. It is also the one drop + # path with no other trace at all -- the frame is gone before + # anything on the loop side could have noticed it. + if isinstance(message, CapturedFrame): + record(VOICE_PACKETS, 1, outcome="loop_gone", guild_id=self._guild_id) log.debug("Dropped a %s: the event loop is gone", type(message).__name__) def _on_decode_failure(self) -> None: @@ -240,8 +325,34 @@ async def _drain(self, queue: asyncio.Queue[CaptureMessage]) -> None: await self._handle(message) except asyncio.CancelledError: raise - except Exception: - log.exception("Error handling a %s message", type(message).__name__) + except Exception as exc: + # **ERROR, and rate limited -- two separate decisions.** An + # exception escaping `_handle` is a defect in this adapter, + # not a condition it expects to recover from: the frame it + # was carrying is discarded, nothing retries it, and the + # audio it held is gone from the recording for good. That is + # `events`' definition of ERROR ("a human must act"), and it + # is what main logged here with `log.exception`. + # + # The flood this line can become is answered by + # `_MESSAGE_ERROR_LOG_EVERY`, not by the level. Lowering the + # severity to buy quiet would keep the noise exactly where + # it was and remove the only part of the line an alert can + # key on -- and `sturnus.voice.packet_errors` below is the + # rate an operator watches, while this line is the one that + # says a human should look at all. + record(VOICE_PACKET_ERRORS, 1, error_type=type(exc).__qualname__) + if self._message_errors.should_log(): + log_exception( + log, + logging.ERROR, + Event.VOICE_PACKET_HANDLER_FAILED, + "A capture message handler raised; capture continues", + exc, + guild_id=self._guild_id, + session_id=self._service.session_id, + count=self._message_errors.count, + ) async def _handle(self, message: CaptureMessage) -> None: match message: @@ -263,7 +374,18 @@ async def _record(self, frame: CapturedFrame) -> None: assert self._guild_id is not None allowed = await self._consent_cache.may_record(self._guild_id, frame.discord_user_id, True) if not allowed: + record(VOICE_PACKETS, 1, outcome="no_consent", guild_id=self._guild_id) + return + if not self._service.is_recording: + # `voice_packet` already returns early in this case, so this is + # a label rather than a decision: it separates "the session was + # closing while frames were still in the queue" from "we never + # got the frame", which look identical without it. The check + # stays a read of the service's own state, not a second copy of + # the rule. + record(VOICE_PACKETS, 1, outcome="not_recording", guild_id=self._guild_id) return + record(VOICE_PACKETS, 1, outcome="recorded", guild_id=self._guild_id) await self._service.voice_packet( frame.discord_user_id, frame.display_name, @@ -284,20 +406,64 @@ def _report_decode_failure(self) -> None: not a reconnect and does not retry anything. It stops pretending, and the reason lands on the session row. """ - log.error( - "Ending the session with %s: no voice stream is decoding any longer.", - EndReason.DECODE_FAILURE.value, + log_event( + log, + logging.ERROR, + Event.VOICE_DECODE_FAILED, + "No voice stream is decoding any longer; ending the session rather than " + "recording silence.", + guild_id=self._guild_id, + session_id=self._service.session_id, + end_reason=EndReason.DECODE_FAILURE.value, ) self._service.request_close(EndReason.DECODE_FAILURE) def _report_capture_stopped(self, message: CaptureStopped) -> None: - """Capture ended without us asking, so the session ends saying so.""" - log.error( - "Voice capture stopped unexpectedly (%s); ending the session with %s rather " - "than leaving it open with nothing arriving. This is the failure mode that " - "silently ended a recording in production.", - type(message.error).__name__ if message.error is not None else "no error reported", - EndReason.CAPTURE_FAILURE.value, - exc_info=message.error, - ) + """Capture ended without us asking, so the session ends saying so. + + The policy lives here rather than in the `after=` hook, and so does + the log line: `_on_listen_stopped` runs on the library's + `audioreader-stopper` thread and does nothing but hand the signal + over, while this runs on the event loop where the session can + actually be ended. + """ + # Two spellings of one event rather than one call with a variable + # message: `log_event`'s message has to be a literal written here + # (`tests/test_logging_discipline.py` rule R1), because it is the + # one field `scrub_event` forwards to Sentry. The two cases differ + # in substance anyway -- an `after=` that fired with no error at + # all is a different thing to explain than one carrying an + # `OpusError` -- and inventing an `error_type` for the first would + # be a lie the field cannot carry. + if message.error is None: + log_event( + log, + logging.ERROR, + Event.VOICE_READER_STOPPED, + "Voice capture stopped unexpectedly with no error reported; ending the " + "session rather than leaving it open with nothing arriving. This is the " + "failure mode that silently ended a recording in production.", + guild_id=self._guild_id, + session_id=self._service.session_id, + end_reason=EndReason.CAPTURE_FAILURE.value, + listening=False, + ) + else: + log_exception( + log, + logging.ERROR, + Event.VOICE_READER_STOPPED, + "Voice capture stopped unexpectedly; ending the session rather than " + "leaving it open with nothing arriving. This is the failure mode that " + "silently ended a recording in production.", + message.error, + guild_id=self._guild_id, + session_id=self._service.session_id, + end_reason=EndReason.CAPTURE_FAILURE.value, + listening=False, + # `error_type` alone says `ConnectionClosed`, which is the + # least informative true statement available about a bot + # that was dropped from voice. See `voice_close_code`. + close_code=voice_close_code(message.error), + ) self._service.request_close(EndReason.CAPTURE_FAILURE) diff --git a/src/sturnus/infrastructure/documents/outline.py b/src/sturnus/infrastructure/documents/outline.py index 0769e95..e9160d6 100644 --- a/src/sturnus/infrastructure/documents/outline.py +++ b/src/sturnus/infrastructure/documents/outline.py @@ -23,10 +23,14 @@ import logging from typing import Any +from urllib.parse import urlsplit import httpx +from opentelemetry.trace import SpanKind from sturnus.application.documents import CreatedDocument, DocumentSink +from sturnus.infrastructure.telemetry import set_span_fields, span +from sturnus.observability.events import Event, log_event log = logging.getLogger(__name__) @@ -115,27 +119,67 @@ def __init__( async def create(self, title: str, body: str, target: str) -> CreatedDocument: payload = _build_payload(title=title, body=body, collection_id=target) - log.debug( - "Creating Outline document (title length=%d, body length=%d)", - len(title), - len(body), - ) - async with httpx.AsyncClient( - base_url=self._base_url, - transport=self._transport, - headers={"Authorization": f"Bearer {self._api_token}"}, - ) as client: - response = await client.post(_CREATE_DOCUMENT_PATH, json=payload) - - if response.status_code in _PERMANENT_STATUS_CODES: - log.warning( - "Outline permanently rejected document creation (status=%d)", - response.status_code, + # The highest-value span per line in this adapter. This file's own + # docstring says the endpoint path, the field names and the response + # shape are UNVERIFIED guesses against a live Outline -- this span is + # what confirms or refutes them in production. + # + # Sizes only. `title` is derived from the transcript and `body` *is* + # the transcript; neither goes anywhere near an attribute. Note also + # what is absent: `url.full` would carry the Authorization header's + # host and any query string, which is exactly the reason no + # `opentelemetry-instrumentation-httpx` is installed. + with span( + "document.create", + SpanKind.CLIENT, + http_method="POST", + url_path=_CREATE_DOCUMENT_PATH, + server_address=urlsplit(self._base_url).hostname or "", + collection_id=target, + title_chars=len(title), + body_bytes=len(body.encode("utf-8")), + ) as active: + async with httpx.AsyncClient( + base_url=self._base_url, + transport=self._transport, + headers={"Authorization": f"Bearer {self._api_token}"}, + ) as client: + response = await client.post(_CREATE_DOCUMENT_PATH, json=payload) + + set_span_fields(active, http_status=response.status_code) + + if response.status_code in _PERMANENT_STATUS_CODES: + # `permanent` is the operationally decisive bit: 401/403/404 + # means "this will never succeed, stop retrying", while a + # 5xx is swept up again every 300s by + # `retry_pending_documents`. From outside, those two look + # identical today. `collection_id` earns its place for the + # same reason -- a 404 is un-diagnosable without it, because + # "is the configured document_target real?" is the whole + # question. + set_span_fields(active, permanent=True) + log_event( + log, + logging.WARNING, + Event.SESSION_DOCUMENT_REJECTED, + "Outline permanently rejected document creation", + http_status=response.status_code, + collection_id=target, + ) + raise PermanentDocumentError(response.status_code) + + response.raise_for_status() + + created = _extract_created_document(response.json(), self._base_url) + set_span_fields(active, document_id=created.id) + log_event( + log, + logging.DEBUG, + Event.SESSION_DOCUMENT_CREATED, + "Created an Outline document", + document_id=created.id, + collection_id=target, + title_chars=len(title), + body_bytes=len(body.encode("utf-8")), ) - raise PermanentDocumentError(response.status_code) - - response.raise_for_status() - - created = _extract_created_document(response.json(), self._base_url) - log.info("Created Outline document %s", created.id) - return created + return created diff --git a/src/sturnus/infrastructure/documents/outline_oauth.py b/src/sturnus/infrastructure/documents/outline_oauth.py index e44118d..a888a0b 100644 --- a/src/sturnus/infrastructure/documents/outline_oauth.py +++ b/src/sturnus/infrastructure/documents/outline_oauth.py @@ -50,6 +50,8 @@ import httpx +from sturnus.observability.events import Event, log_event + log = logging.getLogger(__name__) #: Assumed authorization endpoint (Outline's built-in OAuth 2.0 provider, @@ -172,7 +174,16 @@ async def identity_from_code(self, code: str) -> ExternalIdentity: both as "this link attempt failed", not distinguish them. """ async with httpx.AsyncClient(transport=self._transport) as http: - log.debug("Exchanging Outline authorization code for an access token") + # No `code`, no `client_secret`, no request body: an + # authorization code is a bearer credential for one exchange and + # the secret is one for every exchange. + log_event( + log, + logging.DEBUG, + Event.LINK_CALLBACK_REJECTED, + "Exchanging an Outline authorization code for an access token", + reason="exchange_started", + ) token_response = await http.post( f"{self._base_url}{_TOKEN_PATH}", data={ @@ -184,9 +195,17 @@ async def identity_from_code(self, code: str) -> ExternalIdentity: }, ) if token_response.status_code != httpx.codes.OK: - log.warning( - "Outline rejected the authorization code exchange (status=%d)", - token_response.status_code, + # The status code, never the response body: Outline's error + # bodies are not documented here and have not been read, so + # they are exactly the class of content that needs reading + # before it is waved through. + log_event( + log, + logging.WARNING, + Event.LINK_EXCHANGE_FAILED, + "Outline rejected the authorization code exchange", + http_status=token_response.status_code, + reason="code_exchange", ) raise LinkExchangeError( "the authorization code was refused", status_code=token_response.status_code @@ -194,7 +213,13 @@ async def identity_from_code(self, code: str) -> ExternalIdentity: access_token = token_response.json()["access_token"] - log.debug("Fetching the linked identity from Outline") + log_event( + log, + logging.DEBUG, + Event.LINK_ESTABLISHED, + "Fetching the linked identity from Outline", + reason="identity_lookup_started", + ) identity_response = await http.post( f"{self._base_url}{_IDENTITY_PATH}", headers={"Authorization": f"Bearer {access_token}"}, @@ -202,14 +227,27 @@ async def identity_from_code(self, code: str) -> ExternalIdentity: ) if identity_response.status_code != httpx.codes.OK: - log.warning( - "Outline rejected the identity lookup (status=%d)", - identity_response.status_code, + log_event( + log, + logging.WARNING, + Event.LINK_EXCHANGE_FAILED, + "Outline rejected the identity lookup", + http_status=identity_response.status_code, + reason="identity_lookup", ) raise LinkExchangeError( "the identity lookup was refused", status_code=identity_response.status_code ) identity = _extract_identity(identity_response.json()) - log.info("Resolved Outline identity %s", identity.external_user_id) + # `identity.display_name` came back on the same response and is + # deliberately not logged -- see `fields.DENIED_NAMES`. + log_event( + log, + logging.INFO, + Event.LINK_ESTABLISHED, + "Resolved the external identity", + external_user_id=identity.external_user_id, + provider="outline", + ) return identity diff --git a/src/sturnus/infrastructure/health.py b/src/sturnus/infrastructure/health.py index bd3b4bf..3eee782 100644 --- a/src/sturnus/infrastructure/health.py +++ b/src/sturnus/infrastructure/health.py @@ -56,10 +56,25 @@ async def readyz(_request: web.Request) -> web.Response: ) async def metrics(_request: web.Request) -> web.Response: - # No metrics backend is wired up yet; an empty exposition is still - # a valid Prometheus response, so the endpoint doesn't 404 while - # nothing has been instrumented. - return web.Response(text="", content_type="text/plain") + # Metrics are **pushed** over OTLP to the endpoint named below (see + # `sturnus.infrastructure.telemetry`), not scraped from here. + # + # This used to return 200 with an empty body, on the reasoning that + # an empty exposition is still valid Prometheus. That is worse than + # a 501: a scrape of an empty 200 is indistinguishable from "every + # counter is legitimately zero", so the day someone points a + # ServiceMonitor at this route, a completely uninstrumented process + # looks perfectly healthy. A 501 marks the target down, which is the + # truthful signal, and the route still exists so Spec 4.1's endpoint + # list stays literally satisfied. + return web.Response( + status=501, + text=( + "Sturnus pushes metrics over OTLP; there is nothing to scrape here. " + "Set STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT and read them from Grafana.\n" + ), + content_type="text/plain", + ) async def version(_request: web.Request) -> web.Response: return web.json_response({"version": __version__}) diff --git a/src/sturnus/infrastructure/linkserver.py b/src/sturnus/infrastructure/linkserver.py index cf17a3b..9e71929 100644 --- a/src/sturnus/infrastructure/linkserver.py +++ b/src/sturnus/infrastructure/linkserver.py @@ -37,6 +37,8 @@ from sturnus.application.linking import PendingLink from sturnus.infrastructure.documents.outline_oauth import ExternalIdentity, LinkExchangeError +from sturnus.infrastructure.telemetry import OAUTH_CALLBACK, record +from sturnus.observability.events import Event, log_event log = logging.getLogger(__name__) @@ -137,7 +139,19 @@ async def oauth_callback(request: web.Request) -> web.Response: code = request.query.get("code") state = request.query.get("state") if not code or not state: - log.warning("Rejected an OAuth callback missing a required parameter") + # Never logs `code` or `state` themselves: `code` is an + # authorization code and `state` is a single-use CSRF token. + # `reason` is what makes these two lines countable, which is the + # whole diagnostic value -- a spike of `bad_state` is a replay + # attempt, a spike of `missing_param` is a broken redirect URI. + record(OAUTH_CALLBACK, 1, outcome="missing_param") + log_event( + log, + logging.WARNING, + Event.LINK_CALLBACK_REJECTED, + "Rejected an OAuth callback missing a required parameter", + reason="missing_param", + ) return web.Response(text=_ERROR_PAGE, content_type="text/html", status=400) pending = await states.consume(state, now()) @@ -145,15 +159,27 @@ async def oauth_callback(request: web.Request) -> web.Response: # Covers both a forged state and a replayed one -- see # `LinkStateStore.consume`, which deliberately makes the two # indistinguishable to the caller. - log.warning("Rejected an OAuth callback with an unknown, expired or reused state") + record(OAUTH_CALLBACK, 1, outcome="bad_state") + log_event( + log, + logging.WARNING, + Event.LINK_CALLBACK_REJECTED, + "Rejected an OAuth callback with an unknown, expired or reused state", + reason="bad_state", + ) return web.Response(text=_ERROR_PAGE, content_type="text/html", status=400) try: identity = await oauth.identity_from_code(code) except LinkExchangeError: - log.warning( - "Outline refused the account link attempt for discord_user_id=%s", - pending.discord_user_id, + record(OAUTH_CALLBACK, 1, outcome="exchange_failed") + log_event( + log, + logging.WARNING, + Event.LINK_EXCHANGE_FAILED, + "Outline refused the account link attempt", + discord_user_id=pending.discord_user_id, + provider=pending.provider, ) return web.Response(text=_ERROR_PAGE, content_type="text/html", status=502) @@ -163,11 +189,18 @@ async def oauth_callback(request: web.Request) -> web.Response: identity.external_user_id, identity.display_name, ) - log.info( - "Linked discord_user_id=%s to %s account external_user_id=%s", - pending.discord_user_id, - pending.provider, - identity.external_user_id, + record(OAUTH_CALLBACK, 1, outcome="established") + # `display_name` came back from Outline alongside these and is + # deliberately not logged: the ids answer every operational + # question, and the name is the part that identifies a person. + log_event( + log, + logging.INFO, + Event.LINK_ESTABLISHED, + "Linked a Discord account to an external identity", + discord_user_id=pending.discord_user_id, + provider=pending.provider, + external_user_id=identity.external_user_id, ) return web.Response(text=_CONFIRMATION_PAGE, content_type="text/html", status=200) diff --git a/src/sturnus/infrastructure/observability.py b/src/sturnus/infrastructure/observability.py index 05648e8..5935903 100644 --- a/src/sturnus/infrastructure/observability.py +++ b/src/sturnus/infrastructure/observability.py @@ -94,6 +94,7 @@ from sturnus import __version__ from sturnus.config import SentrySettings from sturnus.domain.errors import DiagnosticSafeError +from sturnus.observability.redaction import SAFE_MESSAGE_TYPES if TYPE_CHECKING: # pragma: no cover - typing only from sentry_sdk._types import Breadcrumb, BreadcrumbHint, Event, Hint @@ -213,6 +214,22 @@ # and their pre-composed messages are dropped rather than forwarded. TRUSTED_LOGGER_ROOT = "sturnus" +# Exception types allowed to say anything at all about themselves. +# +# Aliased to `sturnus.observability.redaction.SAFE_MESSAGE_TYPES` rather than +# restated, so that Sentry, Tempo and Loki answer "may this exception's +# message leave the pod" from one list. Two subtly different answers in two +# files is worse than either alone -- the gap between them is where a message +# gets out, and the gap opens on the day someone *removes* a type from one +# list because it turned out to leak. The name stays for the call sites here. +# +# The alias is the gate, not the whole rule: what a vouched-for type is then +# allowed to *say* is still decided per type in `_exception_value`, and for +# `OSError` that is stricter here than `redaction.safe_exception_message` -- +# see `_os_error_value`. Narrower than the shared list is always allowed; +# wider is not, and the `isinstance` gate is what makes wider impossible. +SAFE_VALUE_TYPES: tuple[type[BaseException], ...] = SAFE_MESSAGE_TYPES + log = logging.getLogger(__name__) @@ -283,14 +300,26 @@ def _os_error_value(exc: OSError) -> str: def _exception_value(exc: BaseException | None) -> str | None: """The message that may be sent for `exc`, or `None` to redact it. + `SAFE_VALUE_TYPES` is the gate: a type the shared list does not vouch + for says nothing here, whatever the branches below would have done with + it. That is what keeps this function from drifting wider than + `redaction.safe_exception_message` when the shared list shrinks. + `DiagnosticSafeError` is checked first: it is an explicit, reviewed opt-in for the whole message, and it wins over the structural rebuild `OSError` gets even for a class that happens to be both. """ + if not isinstance(exc, SAFE_VALUE_TYPES): + return None if isinstance(exc, DiagnosticSafeError): return str(exc) if isinstance(exc, OSError): return _os_error_value(exc) + # Vouched for by the shared list but with no rebuild rule of its own + # here. Sentry is the transport with the least operator context around + # it, so an unrecognised addition to the list is redacted rather than + # forwarded: adding a type has to be a deliberate edit in both places, + # removing one takes effect in both at once. return None diff --git a/src/sturnus/infrastructure/speech_gate.py b/src/sturnus/infrastructure/speech_gate.py index 8aae94a..9d35b27 100644 --- a/src/sturnus/infrastructure/speech_gate.py +++ b/src/sturnus/infrastructure/speech_gate.py @@ -50,6 +50,16 @@ This module lives in `infrastructure` and not in `domain` because it needs numpy, and `tests/test_architecture.py` enforces that `sturnus.domain` imports nothing but the standard library and itself. + +**It has no logger, no span and no metric of its own, deliberately.** It is a +pure function over an array — called once per job, from +`WhisperEngine._transcribe`, which has its result in hand the moment it +returns. Its verdict is reported from there instead, as `clips` and +`speech_seconds` on `transcription.decoded` and `transcription.skipped` and as +span attributes on `job.transcribe`, so a hot numpy routine stays free of I/O +and the numbers still get out. `speech_seconds` against `audio_seconds` is the +signature that would have named Silero as the culprit on the first read: one +second of speech in two minutes of recording is not a plausible meeting. """ from __future__ import annotations diff --git a/src/sturnus/infrastructure/telemetry.py b/src/sturnus/infrastructure/telemetry.py new file mode 100644 index 0000000..ecf9059 --- /dev/null +++ b/src/sturnus/infrastructure/telemetry.py @@ -0,0 +1,809 @@ +"""Traces and metrics over OTLP, built as a privacy control first. + +A sibling of `sturnus.infrastructure.observability` rather than part of it: +that module is Sentry's privacy control and this is Tempo's, and keeping +them in separate files means two branches never edit the same control at +once. What they are *not* is two policies -- both defer to +`sturnus.observability` for the field registry and for the one question +"may this exception's message travel", so there is exactly one answer per +question across all three retained stores. + +Spans go to Tempo, via `alloy-receiver.grafana.svc:4318`. They deliberately +do **not** go to Sentry. `sentry_sdk` can consume OTel spans, and +`observability.py` has locked that door twice on purpose +(`traces_sample_rate=0.0` *and* `before_send_transaction=drop_transaction`, +because `before_send` is never called for transactions and span data would +route around `scrub_event` entirely). Wiring the two together would open +exactly that path. The locks stay; the pod log line, carrying `trace_id`, is +the correlation point instead. + +Three mechanisms keep content out of Tempo, each independent of the others. + +**1. No auto-instrumentation packages, at all.** The same answer +`observability.py` gives with `auto_enabling_integrations=False`, to the +same threat, and every plausible instrumentor is disqualifying in this +codebase specifically: + +- `opentelemetry-instrumentation-sqlalchemy` adds `db.statement`, and + `queue.complete(job_id, transcript)` writes the serialised transcript + through SQLAlchemy -- the statement and its parameters *are* the protected + content. +- `opentelemetry-instrumentation-botocore` adds `aws.s3.key`, and the key + format is `sessions/{id}/speakers/{discord_user_id}.enc`: a Discord user + id on every download span, for free. +- `-httpx` / `-aiohttp-client` add `url.full`: presigned S3 URLs carrying + `X-Amz-Signature`, exactly the `StdlibIntegration` risk their docstring + names. +- `-aiohttp-server` would attach `link`'s only route, + `/oauth/callback?code=...&state=...`, shipping an Outline authorization + code to Tempo. + +Only `opentelemetry-{api,sdk}`, `-semantic-conventions` and +`-exporter-otlp-proto-http` are installed, and none of them instruments +anything on its own. + +**2. Both exception flags off, at one chokepoint.** Verified against +opentelemetry-sdk 1.44.0: `start_as_current_span` defaults to +`record_exception=True, set_status_on_exception=True`, and a +`RuntimeError("SECRET")` escaping such a span lands in *three* places -- +`exception.message`, the full `exception.stacktrace`, and +`status.description`. Turning off only `record_exception` still leaves the +status description. `span()` below is the only way a span is opened here and +hard-codes both to `False`; `fail_span` records failure as a bare `ERROR` +status plus `error.type`, never a message. + +**3. An allowlisting exporter.** `AllowlistingSpanExporter` rebuilds every +span from `fields.SAFE_SPAN_ATTRIBUTES` before OTLP sees it, drops all +events, and replaces the status with a bare code. This is the trace-side +analogue of `scrub_event`, with the same argued failure mode: an +unrecognised attribute is dropped, so a mistake costs a missing field in +Grafana rather than a transcript in Tempo. Mechanisms 2 and 3 are +independent on purpose -- the "stopped twice" reasoning `SAFE_FRAME_KEYS` +gives for `vars`. + +`SpanLimits(max_attribute_length=256)` is a blunt third lock behind both: it +does not make a leak safe, but it caps one at 256 characters rather than a +whole conversation. + +Verified against opentelemetry-sdk 1.44.0 and +opentelemetry-semantic-conventions 0.65b0. +""" + +from __future__ import annotations + +import logging +import socket +import threading +import time +from collections.abc import Callable, Iterable, Iterator, Mapping +from contextlib import contextmanager +from typing import Any, Final + +from opentelemetry import metrics, trace +from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter +from opentelemetry.metrics import CallbackOptions, Observation +from opentelemetry.sdk.metrics import MeterProvider +from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader +from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation, View +from opentelemetry.sdk.resources import Resource +from opentelemetry.sdk.trace import ReadableSpan, SpanLimits, TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResult +from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased +from opentelemetry.semconv.resource import ResourceAttributes +from opentelemetry.trace import Span, SpanKind, Status, StatusCode + +from sturnus import __version__ +from sturnus.config import OtelSettings +from sturnus.observability import events +from sturnus.observability.events import Event, log_event +from sturnus.observability.fields import ( + METRIC_LABEL_FIELDS, + SAFE_SPAN_ATTRIBUTES, + service_name, + span_attribute, +) +from sturnus.observability.redaction import error_type, scrub_fields + +log = logging.getLogger(__name__) + +_INSTRUMENTATION_SCOPE: Final = "sturnus" + +#: Buckets for every duration histogram, in **seconds**. +#: +#: Explicit rather than default, and this is not a nicety: the SDK's default +#: boundaries are `(0, 5, 10, 25, ... 10000)`, tuned for milliseconds. A +#: Whisper transcription of a 40-minute recording on CPU lands in the last +#: bucket of that set along with everything else over ten seconds, which is +#: to say the histogram would answer nothing at all. +_DURATION_BUCKETS: Final = (0.05, 0.1, 0.5, 1.0, 5.0, 15.0, 60.0, 300.0, 900.0, 1800.0, 3600.0) + +#: Buckets for upload sizes, in bytes: 64 KiB to 1 GiB. +_BYTE_BUCKETS: Final = ( + 65_536.0, + 262_144.0, + 1_048_576.0, + 4_194_304.0, + 16_777_216.0, + 67_108_864.0, + 268_435_456.0, + 1_073_741_824.0, +) + +_tracer_provider: TracerProvider | None = None +_meter_provider: MeterProvider | None = None + + +# --------------------------------------------------------------------------- +# The allowlisting exporter +# --------------------------------------------------------------------------- + + +class AllowlistingSpanExporter(SpanExporter): + """Rebuilds every span from the allowlist before it leaves the process. + + Wraps the real OTLP exporter. For each span it keeps only attributes in + `fields.SAFE_SPAN_ATTRIBUTES`, drops **all** events (the `exception` + event is the leak `record_exception=True` produces, and no other event + type is emitted here), and replaces the status with + `Status(status_code)` -- discarding `description`, which is where + `set_status_on_exception=True` puts `f"{type}: {exc}"`. + + Proven end to end in `tests/infrastructure/test_telemetry.py` rather + than asserted: driving a span with deliberately leaky flags, a forbidden + attribute and a transcript-shaped exception through this exporter yields + a span carrying the registered attribute, a bare `ERROR` status, and no + events. + + Read `fields.ALLOWED_FIELDS` before adding to the allowlist. Adding a + key because a Grafana panel looks sparse is a decision about what leaves + the cluster, not a formatting preference. + """ + + def __init__(self, inner: SpanExporter) -> None: + self._inner = inner + + def export(self, spans: Any) -> SpanExportResult: + return self._inner.export([self._rebuild(span) for span in spans]) + + @staticmethod + def _rebuild(span: ReadableSpan) -> ReadableSpan: + attributes = { + key: value + for key, value in (span.attributes or {}).items() + if key in SAFE_SPAN_ATTRIBUTES + } + status = ( + Status(span.status.status_code) if span.status is not None else Status(StatusCode.UNSET) + ) + return ReadableSpan( + name=span.name, + context=span.get_span_context(), + parent=span.parent, + resource=span.resource, + attributes=attributes, + events=(), + links=span.links, + kind=span.kind, + status=status, + start_time=span.start_time, + end_time=span.end_time, + instrumentation_scope=span.instrumentation_scope, + ) + + def shutdown(self) -> None: + self._inner.shutdown() + + def force_flush(self, timeout_millis: int = 30_000) -> bool: + return self._inner.force_flush(timeout_millis) + + +# --------------------------------------------------------------------------- +# Opening spans and recording failure +# --------------------------------------------------------------------------- + + +def _tracer() -> trace.Tracer: + return trace.get_tracer(_INSTRUMENTATION_SCOPE) + + +def span_attributes(fields: Mapping[str, object]) -> dict[str, Any]: + """Turns registered field names into span attribute keys. + + Goes through `redaction.scrub_fields` first -- the same call + `events.log_event` makes -- so a span attribute and a log field are + filtered by one implementation rather than two that agree today. + """ + return { + span_attribute(key): value + for key, value in scrub_fields(fields).items() + if span_attribute(key) in SAFE_SPAN_ATTRIBUTES + } + + +@contextmanager +def span(name: str, kind: SpanKind = SpanKind.INTERNAL, **fields: object) -> Iterator[Span]: + """Opens a span. The only way one is opened in this codebase. + + Hard-codes `record_exception=False, set_status_on_exception=False`, so + the three leak paths verified in the module docstring are closed at + every call site at once rather than at each one individually. A failing + span is still marked: the exception is re-raised after `fail_span` + records a bare `ERROR` status and the exception's class name. + + With no provider installed this is a `NonRecordingSpan` at roughly + 0.1 us, which is what makes "works with no collector" free rather than + conditional. + """ + with _tracer().start_as_current_span( + name, + kind=kind, + attributes=span_attributes(fields), + record_exception=False, + set_status_on_exception=False, + ) as active: + try: + yield active + except BaseException as exc: + fail_span(active, exc) + raise + + +def fail_span(active: Span, exc: BaseException) -> None: + """Marks a span failed without letting the message travel. + + `Status(StatusCode.ERROR)` with **no** description, plus + `error.type = type(exc).__qualname__`. Never `str(exc)`: `worker.py` + passes exactly that string to `queue.fail`, which is right for a + database column an operator queries deliberately, and wrong for a store + that indexes it and shows it to everyone with Grafana. + `DiagnosticSafeError` remains the only marker that could ever change + this, and it is `observability.py`'s contract, reused rather than + reinvented. + """ + active.set_status(Status(StatusCode.ERROR)) + active.set_attribute(span_attribute("error_type"), error_type(exc)) + + +def set_span_fields(active: Span, **fields: object) -> None: + """Sets registered fields on a specific span. + + The only way an attribute key is written in this codebase; nothing + spells one as a string literal. That is what keeps every emitting call + site and `fields.SAFE_SPAN_ATTRIBUTES` in step, and it is why a + forbidden name is dropped here rather than merely at the exporter. + """ + for key, value in span_attributes(fields).items(): + active.set_attribute(key, value) + + +def set_current_span_fields(**fields: object) -> None: + """Stamps registered fields onto whatever span is currently active. + + How `job.claim` puts `session_id` on the root `job.process` span: the + root is opened before a job has been claimed, so the id it most needs is + not known yet. Order-dependent by nature -- if the enclosing span is + ever closed before this runs, the attributes land on an invalid span and + are silently discarded. Both ends carry a comment saying so. + """ + set_span_fields(trace.get_current_span(), **fields) + + +def _metric_attributes(fields: Mapping[str, object]) -> dict[str, Any]: + """Metric attributes: the registry, narrowed to what may be a label. + + Metrics multiply where logs and spans merely accumulate -- a new value + of one attribute is a new time series forever -- so this is a strictly + smaller set than `span_attributes` returns, holding only fixed source + literals and `guild_id`. No user id can reach a metric attribute, which + is the same decision that protects privacy paying twice: one rule, and + the metric store cannot explode. + """ + return {key: value for key, value in scrub_fields(fields).items() if key in METRIC_LABEL_FIELDS} + + +# --------------------------------------------------------------------------- +# Instruments +# --------------------------------------------------------------------------- +# +# Created at import time off `metrics.get_meter`, which returns a proxy meter +# when no provider is installed; the proxy's instruments are real objects +# whose `add`/`record` are no-ops (measured: 0.10 us against 2.19 us with a +# provider), and they bind to the real provider the moment `init_telemetry` +# installs one. That is why nothing here is lazy or conditional. + +_meter = metrics.get_meter(_INSTRUMENTATION_SCOPE) + +JOB_STAGE_DURATION = _meter.create_histogram( + "sturnus.job.stage.duration", + unit="s", + description="Wall time of one worker pipeline stage.", +) +JOB_OUTCOME = _meter.create_counter( + "sturnus.job.outcome", + unit="1", + description="Transcription jobs by terminal outcome (done/failed/dead).", +) +QUEUE_DEPTH = _meter.create_gauge( + "sturnus.queue.depth", + unit="1", + description="Transcription jobs per status, sampled once per worker poll.", +) +TRANSCRIPTION_AUDIO_DURATION = _meter.create_histogram( + "sturnus.transcription.audio_duration", + unit="s", + description="Length of the audio handed to Whisper, for the realtime factor.", +) +SESSION_CLOSE_DURATION = _meter.create_histogram( + "sturnus.session.close.duration", + unit="s", + description="Wall time of encrypt+upload+enqueue for a whole session.", +) +SESSION_DURATION = _meter.create_histogram( + "sturnus.session.duration", + unit="s", + description="How long a recording session lasted, by end reason.", +) +SESSION_ACTIVE = _meter.create_up_down_counter( + "sturnus.session.active", + unit="1", + description="Recording sessions currently open.", +) +RECORDING_UPLOAD_BYTES = _meter.create_histogram( + "sturnus.recording.upload.bytes", + unit="By", + description="Size of one speaker's encrypted recording as uploaded.", +) +VOICE_PACKETS = _meter.create_counter( + "sturnus.voice.packets", + unit="1", + description="Voice packets by what happened to them.", +) +VOICE_PACKET_ERRORS = _meter.create_counter( + "sturnus.voice.packet_errors", + unit="1", + description="Voice packets whose handler raised.", +) +DOCUMENT_CREATE_DURATION = _meter.create_histogram( + "sturnus.document.create.duration", + unit="s", + description="Wall time of one Outline document creation.", +) +OAUTH_CALLBACK = _meter.create_counter( + "sturnus.oauth.callback", + unit="1", + description="Account-link OAuth callbacks by outcome.", +) + + +# --------------------------------------------------------------------------- +# Transcription progress +# --------------------------------------------------------------------------- + + +class TranscriptionProgress: + """Where the transcription in flight has got to, and when it last moved. + + **The signal already existed and the code threw it away.** + `WhisperModel.transcribe` returns `Tuple[Iterable[Segment], + TranscriptionInfo]`; the segments are a *lazy generator* produced as + decoding proceeds, each carrying `start` and `end`, and + `TranscriptionInfo` carries the denominator. `WhisperEngine._transcribe` + drained the generator inside a single tuple comprehension, so every + intermediate observation was discarded and a job was observable only + once it had already finished. A loop that reports as it goes costs + nothing. (`log_progress=True` exists in the library and only drives a + `tqdm` bar on stdout.) + + **Why this is worth the machinery.** A job that "finished" 100 minutes + of audio in 43 seconds has a real-time factor of 140x -- physically + impossible against the 1.94x `large-v3` manages on this hardware, and + unmistakable. The symptom people actually saw was an empty transcript, + which looks exactly like a participant who never spoke, and it was read + as that for a day. Meanwhile a genuine job on the same recording has + run for 98 minutes, so the honest range is very wide and this is the + only instrument that can tell fast-because-broken from + slow-because-working. + + **Observable instruments rather than a gauge somebody sets.** A + synchronous gauge only changes when a call site sets it, so a decoder + that wedges freezes the gauge at its last value and + `seconds_since_progress` -- the actual alert condition -- could never + grow. The SDK calls these callbacks once per export interval instead, so + a stalled job's clock keeps running with no cooperation from the thread + that is stuck. The same property is what lets the gauges emit *nothing* + while the worker is idle: the series goes stale rather than reporting a + finished job's numbers forever. + + **No id of any kind is a label**, and that is the same rule the rest of + this module follows for the same two reasons: a session, job, guild or + user id is unbounded cardinality, and a metric store keeps what it is + given for a very long time. `model` and the resource's `service.name` + are enough to read every number here. + + Process-global, because the worker transcribes exactly one job at a + time (Spec 5.3) -- "the job in flight" is singular by construction. The + lock is not decoration: `_transcribe` runs on an `asyncio.to_thread` + worker thread and the metric reader calls these callbacks on its own. + """ + + def __init__(self, now: Callable[[], float] = time.monotonic) -> None: + self._now = now + self._lock = threading.Lock() + #: Cumulative decoded seconds per model, never reset -- this is what + #: backs an observable *counter*, whose contract is a running total. + self._decoded: dict[str, float] = {} + self._model: str | None = None + self._total = 0.0 + self._position = 0.0 + self._last_progress = 0.0 + + def begin(self, model: str) -> None: + """A job is now in flight. Called *before* the model call, not after it. + + The stall clock starts here rather than at the first segment + deliberately: a job that wedges before producing anything -- during + feature extraction or language detection, which is where the + collapse this instrument exists for happened -- would otherwise be + indistinguishable from a job that had only just started, forever. + """ + with self._lock: + self._model = model + self._decoded.setdefault(model, 0.0) + self._total = 0.0 + self._position = 0.0 + self._last_progress = self._now() + + def set_total(self, seconds: float) -> None: + """The denominator: `TranscriptionInfo.duration_after_vad`. + + Known only once `transcribe()` has returned its info object, which + is why it is separate from `begin`. + + This is the length of the array the model was handed, and since + `WhisperEngine` hands over the gated speech concatenated rather than + the padded track, it is the speech in the recording rather than the + recording -- the more correct denominator for a real-time factor, and + the one the positions reported to `advance` are on. Both numbers are + on the *concatenated* timeline, which is why they are comparable; + `whisper._on_the_original_timeline` puts the segments that reach the + document back on the recording's, and those times are deliberately + not what is reported here. + """ + with self._lock: + self._total = seconds + + def advance(self, position_seconds: float) -> None: + """The decoder has reached `position_seconds` of the audio it was given. + + On the timeline of the array handed to the model -- the concatenated + speech -- and not on the recording's, so that it and `set_total` are + the same measure. + + Takes a position, not a delta, and only ever moves forward. + faster-whisper's seek loop can emit a segment whose `end` is not + past the previous one at a clip boundary, and a decrement here would + be a counter reset -- which Prometheus turns into an enormous + spurious rate rather than into a small correction. + """ + with self._lock: + if self._model is None: + return + gained = max(0.0, position_seconds - self._position) + self._position += gained + self._decoded[self._model] += gained + self._last_progress = self._now() + + def end(self) -> None: + """No job is in flight. Must run even when the decoder raised. + + Without it a failed job would look exactly like a wedged one: + `seconds_since_progress` would climb past every threshold while the + worker went cheerfully on to the next job. + """ + with self._lock: + self._model = None + self._total = 0.0 + self._position = 0.0 + + # -- the callbacks the SDK calls, once per export interval -- + + def observe_decoded(self, options: CallbackOptions) -> Iterable[Observation]: + """Cumulative decoded audio seconds. Reported whether idle or not. + + A counter's total does not disappear when the work stops, and the + rate over a window is the whole point: divided by wall time this is + the real-time factor. + """ + del options + with self._lock: + totals = dict(self._decoded) + return [ + Observation(seconds, _metric_attributes({"model": model})) + for model, seconds in totals.items() + ] + + def observe_position(self, options: CallbackOptions) -> Iterable[Observation]: + """How far into the recording the job in flight has got, in seconds.""" + del options + return self._in_flight(lambda: self._position) + + def observe_total(self, options: CallbackOptions) -> Iterable[Observation]: + """How long the recording being decoded is, in seconds.""" + del options + return self._in_flight(lambda: self._total) + + def observe_stall(self, options: CallbackOptions) -> Iterable[Observation]: + """Seconds since the last segment -- or since the job began, if none. + + **The alert.** A position gauge cannot express it: a job stuck at + zero looks like a job that has only just started, and every real + job passes through that state. + """ + del options + return self._in_flight(lambda: self._now() - self._last_progress) + + def _in_flight(self, value: Callable[[], float]) -> list[Observation]: + """One observation while a job is running, none at all otherwise. + + The empty list is load-bearing: it is what lets the series go stale + on an idle worker instead of publishing the last job's numbers + forever, and every alert expression in `docs/operations.md` section + 7.5 depends on it. + """ + with self._lock: + if self._model is None: + return [] + return [Observation(value(), _metric_attributes({"model": self._model}))] + + +#: The one progress object. See `TranscriptionProgress` for why it is +#: process-global rather than passed around. +TRANSCRIPTION_PROGRESS: Final = TranscriptionProgress() + +TRANSCRIPTION_DECODED_SECONDS = _meter.create_observable_counter( + "sturnus.transcription.decoded_seconds", + callbacks=[TRANSCRIPTION_PROGRESS.observe_decoded], + unit="s", + description="Seconds of recording handed to the decoder, cumulative. Divide by wall " + "time for the real-time factor.", +) +TRANSCRIPTION_POSITION_SECONDS = _meter.create_observable_gauge( + "sturnus.transcription.position_seconds", + callbacks=[TRANSCRIPTION_PROGRESS.observe_position], + unit="s", + description="How far into the recording the transcription in flight has got.", +) +TRANSCRIPTION_TOTAL_SECONDS = _meter.create_observable_gauge( + "sturnus.transcription.total_seconds", + callbacks=[TRANSCRIPTION_PROGRESS.observe_total], + unit="s", + description="How long the recording being transcribed is; the denominator for " + "position_seconds.", +) +TRANSCRIPTION_SECONDS_SINCE_PROGRESS = _meter.create_observable_gauge( + "sturnus.transcription.seconds_since_progress", + callbacks=[TRANSCRIPTION_PROGRESS.observe_stall], + unit="s", + description="Seconds since the transcription in flight last produced a segment, or " + "since it started. Absent when nothing is transcribing.", +) + + +def record(instrument: Any, value: float, **fields: object) -> None: + """Records one measurement with metric-safe attributes. + + One helper for counters, histograms, gauges and up/down counters alike + so no call site picks the method name *and* builds the attribute dict + itself -- `_metric_attributes` is not optional, and making it the only + path is cheaper than remembering. + """ + attributes = _metric_attributes(fields) + if hasattr(instrument, "record"): + instrument.record(value, attributes) + elif hasattr(instrument, "set"): + instrument.set(value, attributes) + else: + instrument.add(value, attributes) + + +# --------------------------------------------------------------------------- +# Wiring it up +# --------------------------------------------------------------------------- + + +def _views() -> list[View]: + """One `View` per histogram, carrying explicit buckets. See `_DURATION_BUCKETS`.""" + seconds = [ + "sturnus.job.stage.duration", + "sturnus.transcription.audio_duration", + "sturnus.session.close.duration", + "sturnus.session.duration", + "sturnus.document.create.duration", + ] + views = [ + View( + instrument_name=name, + aggregation=ExplicitBucketHistogramAggregation(boundaries=_DURATION_BUCKETS), + ) + for name in seconds + ] + views.append( + View( + instrument_name="sturnus.recording.upload.bytes", + aggregation=ExplicitBucketHistogramAggregation(boundaries=_BYTE_BUCKETS), + ) + ) + return views + + +def _trace_context() -> dict[str, str]: + """`events.current_trace_context`'s implementation, injected not imported. + + `sturnus.observability` is standard-library only so that + `sturnus.application` can import it; it therefore cannot read a span + itself. `init_telemetry` hands it this function, which is what puts + `trace_id` in every JSON log line and turns a Loki row into a click + through to the Tempo waterfall for the same job. + """ + context = trace.get_current_span().get_span_context() + if not context.is_valid: + return {} + return { + "trace_id": format(context.trace_id, "032x"), + "span_id": format(context.span_id, "016x"), + } + + +def init_telemetry(component: str, settings: OtelSettings | None = None) -> bool: + """Installs traces and metrics for one process. Returns whether it did. + + Call immediately after `init_sentry(component)`, with the same literal + component name, as the second statement of `main()`. + + **With no endpoint configured this returns `False` having installed + nothing** -- no provider, no exporter, no background thread, no + connection attempt, no error. Every `span()` in the codebase is then a + `NonRecordingSpan` and every `record()` a no-op, so an operator running + Sturnus outside this cluster needs no Alloy, no flag and no code path of + their own. That is asserted directly in + `tests/infrastructure/test_telemetry.py`. + + OTLP over **HTTP**, not gRPC. Alloy accepts both 4317 and 4318, so the + choice is free, and the gRPC exporter drags in `grpcio` -- a large + per-arch binary wheel and a known source of fork/thread hangs. Both + exporters do their blocking I/O on their own daemon threads, so neither + the asyncio loop nor the voice router thread is ever blocked by an + export. + + Sampling is 100%. The arithmetic, rather than a reflexive ratio: the + worker processes strictly one job at a time and each is minutes of CPU + Whisper work, the bot opens a handful of sessions per guild per day, + `link` sees a handful of callbacks, and the packet path emits no spans + at all by construction. Total volume is a rounding error against Tempo. + `STURNUS_OTEL_TRACES_SAMPLE_RATIO` exists as a valve for a future + high-volume path, and the corollary matters more than the ratio: + sampling must never be why a failed job is invisible, which is why job + outcome is *also* an unsampled counter. + """ + global _tracer_provider, _meter_provider + + settings = settings or OtelSettings() + if settings.otel_exporter_otlp_endpoint is None: + return False + + endpoint = settings.otel_exporter_otlp_endpoint + resource = Resource.create( + { + ResourceAttributes.SERVICE_NAME: service_name(component), + # Groups the three deployments back together after + # `service.name` has split them for Tempo's search and + # Grafana's service graph. + ResourceAttributes.SERVICE_NAMESPACE: "sturnus", + # The same `__version__` `init_sentry` uses for its release tag, + # which release-please keeps in lockstep with the chart's + # appVersion. + ResourceAttributes.SERVICE_VERSION: __version__, + # The pod name, exactly as `init_sentry` relies on the SDK + # defaulting `server_name` to it. + ResourceAttributes.SERVICE_INSTANCE_ID: socket.gethostname(), + # Literal rather than the semconv constant: the incubating + # module's import path is not stable across releases, and + # `tests/infrastructure/test_telemetry.py` pins the string + # against what the installed package exports. + "deployment.environment.name": settings.environment, + } + ) + + _tracer_provider = TracerProvider( + resource=resource, + sampler=ParentBased(TraceIdRatioBased(settings.otel_traces_sample_ratio)), + # A third lock behind `span()`'s flags and the allowlisting + # exporter: caps any string attribute at the SDK boundary. + span_limits=SpanLimits(max_attribute_length=256), + ) + _tracer_provider.add_span_processor( + BatchSpanProcessor( + AllowlistingSpanExporter(OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces")) + ) + ) + trace.set_tracer_provider(_tracer_provider) + + _meter_provider = MeterProvider( + resource=resource, + metric_readers=[ + PeriodicExportingMetricReader( + OTLPMetricExporter(endpoint=f"{endpoint}/v1/metrics"), + export_interval_millis=settings.otel_metric_export_interval_seconds * 1000, + ) + ], + views=_views(), + ) + metrics.set_meter_provider(_meter_provider) + + events.set_trace_context_provider(_trace_context) + _silence_sentry_export_storm() + + # Announced at INFO, the way `init_sentry` announces itself, and for a + # sharper reason than symmetry: if the endpoint is wrong, spans and + # metrics vanish and every dashboard shows a flat, healthy-looking zero. + # This line and the deploy checklist's "confirm one trace and one metric + # arrive in Grafana" step are the only two detectors of that. An OTLP + # endpoint is a service address, not a credential. + log_event( + log, + logging.INFO, + Event.TELEMETRY_ENABLED, + "OpenTelemetry traces and metrics enabled", + component=component, + server_address=endpoint, + ) + return True + + +def _silence_sentry_export_storm() -> None: + """Stops a failed OTLP export from becoming a Sentry issue, forever. + + A verified cross-branch defect rather than a precaution. The SDK logs + export failures with `logger.exception` (`sdk/trace/export/__init__.py`) + and `_logger.error` (the OTLP HTTP exporter), and + `observability.init_sentry` configures + `LoggingIntegration(event_level=logging.ERROR)`. So with Alloy + unreachable -- a NetworkPolicy, a namespace move, a rollout -- *every + retry of every failed batch* becomes a Sentry event, from all three + pods, until someone notices the bill. + + `ignore_logger` is confirmed present in the pinned sentry-sdk 2.68.0. + It is called from here rather than added to `init_sentry` so that the + branch owning that privacy control is not edited from this one; the + matching `NEVER_BELOW` entry in `sturnus.observability.setup` keeps the + same storm out of Loki. Doing it here also means the suppression exists + only when OTLP export is actually configured. + + Note the consequence, stated because it is load-bearing: once this + runs, a broken exporter is visible in neither Sentry nor Loki, so the + post-deploy smoke check is the only remaining detector. That is what + makes it a real checklist item rather than a nicety. + """ + try: + from sentry_sdk.integrations.logging import ignore_logger + except ImportError: # pragma: no cover - sentry-sdk is a hard dependency + return + ignore_logger("opentelemetry") + + +def shutdown_telemetry() -> None: + """Flushes and tears down both providers. Call in each `_run`'s `finally`. + + Without it the last batch of spans -- which, during a SIGTERM, is + exactly the batch describing the shutdown that is about to be + investigated -- dies with the process. + """ + global _tracer_provider, _meter_provider + if _tracer_provider is not None: + _tracer_provider.shutdown() + _tracer_provider = None + if _meter_provider is not None: + _meter_provider.shutdown() + _meter_provider = None + events.set_trace_context_provider(None) diff --git a/src/sturnus/infrastructure/traced.py b/src/sturnus/infrastructure/traced.py new file mode 100644 index 0000000..b8191d9 --- /dev/null +++ b/src/sturnus/infrastructure/traced.py @@ -0,0 +1,346 @@ +"""Traced decorators for the ports `application` already receives. + +`process_one` takes every pipeline stage as an injected narrow `Protocol` -- +`Queue`, `AudioDownloader`, `Decryptor`, `TranscriptionEngine`, +`DocumentSink` -- because `sturnus.application.worker` may not import +`sturnus.infrastructure`. The rule that forbids instrumenting it is exactly +what makes instrumenting it trivial: the decorators below satisfy the same +protocols, `sturnus.entrypoints.worker` wraps the concrete adapters on the +way in, and `application/worker.py` gains full stage-level tracing with +**zero** lines changed and no risk to `tests/test_architecture.py`. The +twelve existing tests in `tests/application/test_worker.py` keep passing +untouched. `sturnus.application.recording` gets the same treatment through +its injected `AudioStore`, `Encryptor` and `JobQueue`. + +None of them *subclasses* the protocol it satisfies, deliberately: a +`Protocol` subclass inherits the `...` method bodies, so a forgotten +override would type-check and silently return `None`. Structural conformance +is checked where it matters instead -- at the composition root in +`sturnus.entrypoints.worker` and `.bot`, where mypy `strict` compares the +whole wrapper against the parameter's declared protocol and names any +method that is missing or has drifted. + +Every wrapper **passes values through unmodified and only observes.** That +is a correctness constraint, not a style note: `Queue.claim` returns +`object | None` and `process_one` immediately `cast`s the result to +`_ClaimedJobShape`, so a wrapper that returned anything but the original +object would break the pipeline silently. mypy `strict` catches signature +drift; `tests/infrastructure/test_traced_ports.py` runs the real +`process_one` against wrapped fakes and catches the rest. + +A gap stated rather than hidden: `assemble` and `render_transcript` are +plain function calls inside `application`, not injected collaborators, so +they get no span. They appear as unaccounted time inside `job.process` +between `job.complete` and `document.create`, which is legible in a +waterfall and is the honest representation. Wrapping the repository reads +inside `assemble` would add noise for a pure-CPU merge over rows that have +already been fetched. +""" + +from __future__ import annotations + +import time +from datetime import datetime +from pathlib import Path + +from opentelemetry.trace import SpanKind + +from sturnus.application.documents import CreatedDocument, DocumentSink +from sturnus.application.ports import AudioStore, Encryptor, SessionKey +from sturnus.application.recording import JobQueue +from sturnus.application.transcription import TranscriptionEngine, TranscriptionResult +from sturnus.application.worker import AudioDownloader, Decryptor, Queue +from sturnus.infrastructure.telemetry import ( + DOCUMENT_CREATE_DURATION, + JOB_STAGE_DURATION, + RECORDING_UPLOAD_BYTES, + TRANSCRIPTION_AUDIO_DURATION, + record, + set_current_span_fields, + set_span_fields, + span, +) + + +class _StageTimer: + """Times one pipeline stage into `sturnus.job.stage.duration`. + + The span answers "why was *this* job slow" by sitting next to the same + job's other stages in one waterfall; the histogram answers "is the fleet + getting slower" and is what a dashboard and an alert read. Neither + substitutes for the other, which is why both exist for every stage. + """ + + def __init__(self, stage: str) -> None: + self._stage = stage + self._started = 0.0 + + def __enter__(self) -> _StageTimer: + self._started = time.monotonic() + return self + + def __exit__(self, exc_type: type[BaseException] | None, *_: object) -> None: + record( + JOB_STAGE_DURATION, + time.monotonic() - self._started, + stage=self._stage, + outcome="failed" if exc_type is not None else "ok", + ) + + +class TracedQueue: + """`sturnus.application.worker.Queue`, traced.""" + + def __init__(self, inner: Queue) -> None: + self._inner = inner + + async def claim(self) -> object | None: + with span("job.claim"), _StageTimer("claim"): + claimed = await self._inner.claim() + if claimed is not None: + # Stamps the ids onto the *enclosing* `job.process` span, which + # was opened before a job had been claimed and therefore before + # its id could be known. Order-dependent: if `process_one` is + # ever moved outside that root span, these land on an invalid + # span and are silently discarded. The matching comment is in + # `sturnus.entrypoints.worker._run`. + set_current_span_fields( + job_id=getattr(claimed, "id", None), + session_id=getattr(claimed, "session_id", None), + ) + return claimed + + async def complete(self, job_id: int, transcript: str) -> bool: + # `transcript` is passed straight through and never observed: it is + # the protected content itself, and the only thing worth recording + # about it -- its length -- is already on the `job.transcribe` span. + with span("job.complete", job_id=job_id), _StageTimer("complete"): + is_last = await self._inner.complete(job_id, transcript) + # `outcome` lands on the enclosing `job.process` span, the same + # place and for the same reason as `claim`'s ids: the root span is + # opened before anything is known and cannot label itself + # afterwards. It is set from the transition that happened rather + # than from `process_one`'s return value, which is `True` for a + # failed job too -- see `sturnus.infrastructure.db.queue.complete`. + set_current_span_fields(is_last=is_last, outcome="done") + return is_last + + async def fail(self, job_id: int, error: str, max_attempts: int) -> bool: + # `error` is `str(exc)` from `process_one`. It goes to the database + # column an operator queries deliberately, and it does **not** go on + # the span -- see `telemetry.fail_span`. + with span("job.fail", job_id=job_id, max_attempts=max_attempts): + dead = await self._inner.fail(job_id, error, max_attempts) + # `dead` and `failed` are different operational stories -- one is a + # recording that will never exist, the other is a retry -- so the + # root span distinguishes them exactly as the counter does. + set_current_span_fields(outcome="dead" if dead else "failed") + return dead + + +class TracedAudioDownloader: + """`sturnus.application.worker.AudioDownloader`, traced. + + Note the absent attribute: the S3 key never becomes one. Its format is + `sessions/{session_id}/speakers/{discord_user_id}.enc`, so it embeds a + user id, and both halves are already registered separately. + `object_bytes` carries all of the diagnostic value with none of that. + """ + + def __init__(self, inner: AudioDownloader) -> None: + self._inner = inner + + async def get(self, key: str, target: Path) -> None: + with span("job.download", SpanKind.CLIENT) as active, _StageTimer("download"): + await self._inner.get(key, target) + size = target.stat().st_size if target.exists() else 0 + set_span_fields(active, object_bytes=size) + + +class TracedDecryptor: + """`sturnus.application.worker.Decryptor`, traced. + + Synchronous, and called by `process_one` through `asyncio.to_thread`. + That is safe for the span: `contextvars` are copied into the worker + thread, so the span opened here parents correctly under `job.process` + rather than becoming an orphaned root. + """ + + def __init__(self, inner: Decryptor) -> None: + self._inner = inner + + def decrypt_to(self, source: Path, target: Path, wrapped: bytes, key_id: str) -> None: + with span("job.decrypt", key_id=key_id) as active, _StageTimer("decrypt"): + self._inner.decrypt_to(source, target, wrapped, key_id) + if target.exists(): + set_span_fields(active, plaintext_bytes=target.stat().st_size) + + +class TracedTranscriptionEngine: + """`sturnus.application.transcription.TranscriptionEngine`, traced. + + The span that is almost always the answer to "which stage is slow". + + `segment_count` and `char_count` are counts of the transcript, never the + transcript, and they are the only way to distinguish "Whisper returned + nothing" from "Whisper produced a repetition cascade" -- the two failure + modes `infrastructure/whisper.py` sets `compression_ratio_threshold` and + `no_speech_threshold` to guard against. A length is a weak side channel + and that is accepted deliberately, with the precedent that + `documents/outline.py` already logs `len(title)` and `len(body)` at + DEBUG. Removing them is a one-line edit to `fields.ALLOWED_FIELDS`, + which is the point of having a registry. + """ + + def __init__(self, inner: TranscriptionEngine) -> None: + self._inner = inner + + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: + """Forwards every argument, and puts `initial_prompt` on no span. + + The third parameter has no default on purpose, mirroring the port + it wraps: a default here would silently drop every guild's + configured vocabulary (Spec 11) the moment this wrapper is applied, + while the untraced path kept it -- an observability decorator + changing what is transcribed, which is exactly what a decorator + must never do. + + It is also not a span attribute. `language` is a bounded literal + and belongs on the span; `initial_prompt` is guild-configured free + text and is therefore content, not metadata. It is not in + `fields.ALLOWED_FIELDS` and must not be added to it. + """ + with span("job.transcribe") as active, _StageTimer("transcribe"): + result = await self._inner.transcribe(path, language, initial_prompt) + audio_seconds = max((segment.end for segment in result.segments), default=0.0) + set_span_fields( + active, + language=result.language, + segment_count=len(result.segments), + char_count=sum(len(segment.text) for segment in result.segments), + audio_seconds=audio_seconds, + ) + # Paired with the `transcribe` stage histogram, this is the + # realtime factor Spec 15 flags as an unmeasured risk: + # `rate(stage_sum{stage="transcribe"}) / rate(audio_sum)`, + # measured continuously against real material instead of + # estimated once. + record(TRANSCRIPTION_AUDIO_DURATION, audio_seconds) + return result + + +class TracedDocumentSink: + """`sturnus.application.documents.DocumentSink`: the duration metric only. + + **Deliberately opens no span.** Unlike every other port here, the + concrete adapter behind this one is in `infrastructure` already and + carries its own `document.create` CLIENT span + (`sturnus.infrastructure.documents.outline.OutlineSink.create`) with + attributes this wrapper could not produce: the HTTP status, whether the + rejection was permanent, the server address. Opening a second span of + the same name here would put two identical, nested entries in every + waterfall and tell the reader nothing. + + What is left is the histogram, which genuinely belongs here rather than + in the adapter: it must cover *every* `DocumentSink`, so it keeps + working if Outline is ever swapped for another provider. + """ + + def __init__(self, inner: DocumentSink) -> None: + self._inner = inner + + async def create(self, title: str, body: str, target: str) -> CreatedDocument: + started = time.monotonic() + outcome = "failed" + try: + created = await self._inner.create(title, body, target) + outcome = "ok" + return created + finally: + # In a `finally` so a failed creation is timed too: "Outline is + # slow" and "Outline is refusing us" are different incidents, + # and the latency of the second is the evidence that tells them + # apart. + record(DOCUMENT_CREATE_DURATION, time.monotonic() - started, outcome=outcome) + + +class TracedAudioStore: + """`sturnus.application.ports.AudioStore`, traced. + + `sturnus.recording.upload.bytes` is capacity planning against the + retention window Spec 15 names as the top operational risk. + """ + + def __init__(self, inner: AudioStore) -> None: + self._inner = inner + + async def put(self, key: str, source: Path) -> None: + size = source.stat().st_size if source.exists() else 0 + with span("recording.upload", SpanKind.CLIENT, object_bytes=size): + await self._inner.put(key, source) + record(RECORDING_UPLOAD_BYTES, size) + + async def delete(self, key: str) -> None: + with span("recording.delete", SpanKind.CLIENT): + await self._inner.delete(key) + + +class TracedEncryptor: + """`sturnus.application.ports.Encryptor`, traced. + + `new_session_key` is deliberately not traced: it is a pure in-memory key + generation whose only interesting values are the key material itself. + """ + + def __init__(self, inner: Encryptor) -> None: + self._inner = inner + + @property + def key_id(self) -> str: + return self._inner.key_id + + def new_session_key(self) -> SessionKey: + return self._inner.new_session_key() + + def encrypt(self, source: Path, target: Path, key: bytes) -> None: + with span("recording.encrypt", key_id=self._inner.key_id) as active: + self._inner.encrypt(source, target, key) + if target.exists(): + set_span_fields(active, object_bytes=target.stat().st_size) + + +class TracedJobQueue: + """`sturnus.application.recording.JobQueue`, traced. + + The enqueue that closes the loop between the bot and the worker: a + session that recorded audio but enqueued nothing is the failure this + span, and `session.closed`'s `jobs_enqueued` field, exist to make + visible. + """ + + def __init__(self, inner: JobQueue) -> None: + self._inner = inner + + async def enqueue( + self, + *, + session_id: int, + discord_user_id: int, + s3_key: str, + encryption_key_id: str, + wrapped_data_key: bytes, + retention_until: datetime, + ) -> int: + with span("job.enqueue", session_id=session_id, key_id=encryption_key_id) as active: + job_id = await self._inner.enqueue( + session_id=session_id, + discord_user_id=discord_user_id, + s3_key=s3_key, + encryption_key_id=encryption_key_id, + wrapped_data_key=wrapped_data_key, + retention_until=retention_until, + ) + set_span_fields(active, job_id=job_id) + return job_id diff --git a/src/sturnus/infrastructure/whisper.py b/src/sturnus/infrastructure/whisper.py index 2ac5db1..5077254 100644 --- a/src/sturnus/infrastructure/whisper.py +++ b/src/sturnus/infrastructure/whisper.py @@ -53,15 +53,39 @@ `tests/infrastructure/test_whisper.py::test_a_window_never_spans_two_clips` drives the real `generate_segments` — no weights, no download — and fails if any of the three stops holding. It is the only upper bound there is. + +**This module is where a transcription becomes observable at all**, and it +is instrumented against one specific way of being wrong. The failure that +cost this project two days produced an *empty transcript*, which is +indistinguishable from a participant who never spoke -- and was read as +exactly that for a day. Two things separate them, and neither is visible +anywhere else in the codebase: + +- `speech_seconds` against `audio_seconds` on `transcription.decoded`, + which is the speech gate's own signature. `sturnus.application.worker` + emits `job.transcribed` with segment counts and a realtime factor, but it + cannot see the gate. +- `sturnus.transcription.decoded_seconds` divided by wall time, which is a + real-time factor computed from the audio the model was *given* rather + than from the segments it returned. A hundred minutes "decoded" in + forty-three seconds is 140x, which is impossible; the same job measured + by its (nonexistent) segments would have contributed nothing at all and + said nothing. + +See `sturnus.infrastructure.telemetry.TranscriptionProgress` for the live +half -- position, denominator and the stall clock -- and why they are +observable instruments rather than gauges this module sets. """ from __future__ import annotations import asyncio import logging +import time from bisect import bisect_right from itertools import accumulate from pathlib import Path +from typing import Any import numpy as np from faster_whisper import WhisperModel # type: ignore[import-untyped] @@ -72,6 +96,10 @@ TranscriptionResult, ) from sturnus.infrastructure.speech_gate import speech_clips +from sturnus.infrastructure.telemetry import TRANSCRIPTION_PROGRESS, set_current_span_fields +from sturnus.observability.events import Event, log_event + +log = logging.getLogger(__name__) _SAMPLE_RATE = 16_000 @@ -168,6 +196,12 @@ def __init__( default_language: str, ) -> None: self._model = WhisperModel(model_size, device=device, compute_type=compute_type) + # Kept so every measurement below can be labelled by model without a + # call site passing it in again. It is the only label these metrics + # carry: a real-time factor that mixes `large-v3` with `tiny` says + # nothing, and no id may become a metric label (see + # `observability.fields.METRIC_LABEL_FIELDS`). + self._model_name = model_size self._default_language = default_language async def transcribe( @@ -184,7 +218,13 @@ def _transcribe( # clip offsets computed on a different copy of the samples could be # misaligned against the one being transcribed. audio = decode_audio(str(path), sampling_rate=_SAMPLE_RATE) + audio_seconds = audio.shape[0] / _SAMPLE_RATE clips = speech_clips(audio, sample_rate=_SAMPLE_RATE) + # The gate's own verdict, in seconds. `speech_gate` stays a pure + # numpy function with no logger of its own -- it is called once per + # job and its result is right here, so reporting it from the caller + # costs nothing and keeps a hot array routine free of I/O. + speech_seconds = sum(end - start for start, end in clips) if not clips: # Deliberately returning without touching the model, because @@ -202,6 +242,25 @@ def _transcribe( # array is an input `FeatureExtractor` has never been exercised # against. This one line is what stands between an all-padding # track and both failures. + # + # Announced rather than returned silently, and **not** counted + # towards `decoded_seconds`. An empty transcript has exactly two + # causes -- the gate found nothing, or the model was called and + # produced nothing -- and only the second is a defect. Adding + # this file's duration to the decode counter would report a + # whole recording processed in the microseconds the gate took, + # which is the very signature the counter exists to raise. + log_event( + log, + logging.INFO, + Event.TRANSCRIPTION_SKIPPED, + "The speech gate found nothing above the silence floor; the model was " + "not called and this speaker produced no segments.", + model=self._model_name, + audio_seconds=round(audio_seconds, 3), + speech_seconds=0.0, + clips=0, + ) return TranscriptionResult(segments=(), language=self._default_language) # Seconds are what `speech_clips` speaks in and samples are the only @@ -272,6 +331,58 @@ def _transcribe( # `frames_per_second` should fail now, not after a 100-minute decode. clip_starts_in_frames = _on_the_frame_grid(concat_starts, self._model.frames_per_second) + started = time.monotonic() + # Before the call, not after it: `transcribe()` extracts features and + # detects a language before it yields anything, and a job that wedges + # in there has to be distinguishable from one that has merely just + # started. See `TranscriptionProgress.begin`. + TRANSCRIPTION_PROGRESS.begin(self._model_name) + try: + return self._decode( + speech, + language, + initial_prompt, + clips, + bounds, + concat_starts, + concat_ends, + clip_starts_in_frames, + offsets, + audio_seconds, + speech_seconds, + started, + ) + finally: + # Runs on the failure path too. Without it a decoder that raised + # would leave the job "in flight" forever, and + # `seconds_since_progress` would climb past every threshold + # while the worker moved on to the next job. + TRANSCRIPTION_PROGRESS.end() + + def _decode( + self, + speech: np.ndarray[Any, Any], + language: str | None, + initial_prompt: str | None, + clips: tuple[tuple[float, float], ...], + bounds: list[tuple[int, int]], + concat_starts: list[float], + concat_ends: list[float], + clip_starts_in_frames: list[int], + offsets: list[float], + audio_seconds: float, + speech_seconds: float, + started: float, + ) -> TranscriptionResult: + """The model call, the loop that reports while it runs, and the restore. + + Split out of `_transcribe` only so that the `try/finally` around it + is one line and cannot accidentally grow to cover the gate. Everything + the concatenation produced is handed in rather than recomputed: the + arithmetic that undoes the join has to be the arithmetic that made it, + and a second derivation of `offsets` from the same clips is a second + place for the two to drift apart. + """ segments, info = self._model.transcribe( speech, language=language, @@ -419,7 +530,18 @@ def _transcribe( # restored times are file-relative in exactly the sense `to_absolute` # assumes, to within the 10 ms of frame-grid rounding # `_on_the_original_timeline` describes. - collected = [] + # + # **A loop, not a tuple comprehension, and that is the change.** + # `segments` is a lazy generator: a comprehension consumes it inside + # a single expression, so nothing between the first segment and the + # last is ever observable and a job is only measurable once it has + # already finished. Reporting each `end` as it arrives costs one + # method call per segment -- a few hundred per job -- and is what + # makes a running transcription's position, and a stalled one's + # silence, visible at all. + total_seconds = float(getattr(info, "duration_after_vad", 0.0) or 0.0) + TRANSCRIPTION_PROGRESS.set_total(total_seconds) + collected: list[TranscribedSegment] = [] for segment in segments: start, end = _on_the_original_timeline( segment.start, @@ -430,45 +552,81 @@ def _transcribe( offsets, ) collected.append(TranscribedSegment(start=start, end=end, text=segment.text)) - # The model never sees the padding at all now, so a speaker whose - # file opens with twenty minutes of it no longer has their language - # guessed from silence. - - # Count what the guard cost, every time, because from in here a track - # of room tone correctly rejected and a track of quiet speech wrongly - # rejected are the same event. Nothing else in the system can see this - # either: `log_prob_threshold=None` makes faster-whisper drop the - # window internally, and the only trace it leaves is a DEBUG line on - # its own `faster_whisper` logger, which nothing here configures. So a - # transcript that came back empty would otherwise be indistinguishable - # from a speaker who never spoke -- and that is exactly the failure - # this branch exists to make impossible in the document, so it must not - # be reintroduced in the logs. + # `segment.end` and not the restored `end`, deliberately. The + # denominator set just above is `duration_after_vad`, which is the + # concatenated speech the model was handed; the restored end is on + # the recording's timeline, up to a whole meeting further along. + # Reporting one against the other would put a job that has decoded + # its first clip at several hundred percent and make the real-time + # factor a number about the removed silence. + TRANSCRIPTION_PROGRESS.advance(segment.end) + # The decoder walked to the end of the speech it was handed whether or + # not the last stretch of it produced a segment, so the counter is + # topped up to the audio the model was actually given. This is the line + # that makes a job returning nothing at all show up as an impossible + # real-time factor rather than as a silent zero. + TRANSCRIPTION_PROGRESS.advance(total_seconds) + + wall_seconds = time.monotonic() - started + # The model never saw the padding at all, so a speaker whose file + # opens with twenty minutes of it no longer has their language guessed + # from silence. + detected = getattr(info, "language", None) or self._default_language + # `job.transcribed` in `sturnus.application.worker` reports a + # realtime factor too, computed from the last segment's `end`. That + # is the right number for "how long did this take per minute of + # speech" and the wrong one for "did this decode anything at all", + # because a job with no segments has no denominator there. This one + # divides by the audio handed over -- which since the speech is + # concatenated is `duration_after_vad`, the gated seconds themselves -- + # so it is defined exactly when the question is worth asking. # - # The seconds matter more than the counts and are what the message - # leads with: 0.9 s of room tone dropped is the guard working, forty - # minutes dropped is an incident, and only the duration tells them - # apart. The text itself is never logged -- the worker's logs are not - # access-controlled the way the Outline collection is. - gated_seconds = sum(end - start for start, end in clips) + # WARNING rather than INFO when nothing came back, and the message + # says why it matters rather than restating the count. From in here a + # track of room tone correctly rejected and a track of quiet speech + # wrongly rejected are the same event, and only the seconds below tell + # them apart -- 0.9 s dropped is the guard working, forty minutes + # dropped is an incident. `log_prob_threshold=None` makes + # faster-whisper discard the window internally and leaves no trace + # except a DEBUG line on its own logger, which nothing here + # configures, so this is the only place it can be seen at all. if collected: - log.debug( - "%s: gate passed %d clip(s)/%.1f s, decoder kept %d segment(s)/%.1f s", - path, - len(clips), - gated_seconds, - len(collected), - sum(s.end - s.start for s in collected), + log_event( + log, + logging.INFO, + Event.TRANSCRIPTION_DECODED, + "Decoded one speaker's recording", + model=self._model_name, + language=detected, + audio_seconds=round(audio_seconds, 3), + speech_seconds=round(speech_seconds, 3), + clips=len(clips), + segments=len(collected), + wall_seconds=round(wall_seconds, 3), + realtime_factor=round(wall_seconds / total_seconds, 4) if total_seconds else None, ) else: - log.warning( - "%s: gate passed %d clip(s)/%.1f s of audio above the silence floor " - "but the decoder judged every window to be silence, so this speaker " - "contributes nothing to the protocol", - path, - len(clips), - gated_seconds, + log_event( + log, + logging.WARNING, + Event.TRANSCRIPTION_DECODED, + "The gate passed audio above the silence floor but the decoder judged " + "every window to be silence; this speaker contributes nothing to the " + "protocol", + model=self._model_name, + language=detected, + audio_seconds=round(audio_seconds, 3), + speech_seconds=round(speech_seconds, 3), + clips=len(clips), + segments=len(collected), + wall_seconds=round(wall_seconds, 3), + realtime_factor=round(wall_seconds / total_seconds, 4) if total_seconds else None, ) - - detected = getattr(info, "language", None) or self._default_language + # Onto `job.transcribe`, opened by + # `traced.TracedTranscriptionEngine` around this call -- + # `asyncio.to_thread` copies the context, so the span is the + # enclosing one rather than an orphan. The wrapper cannot set these + # two: it sees a `TranscriptionResult`, and the gate's numbers are + # not in it. + set_current_span_fields(speech_seconds=speech_seconds, clips=len(clips)) return TranscriptionResult(segments=tuple(collected), language=detected) diff --git a/src/sturnus/observability/__init__.py b/src/sturnus/observability/__init__.py new file mode 100644 index 0000000..5fe4f92 --- /dev/null +++ b/src/sturnus/observability/__init__.py @@ -0,0 +1,38 @@ +"""The vocabulary and the redaction path every telemetry channel shares. + +Sturnus emits telemetry into three retained stores -- Loki (pod logs, via +`alloy-logs`), Tempo (spans, via `alloy-receiver`), and Sentry (errors, +`sturnus.infrastructure.observability`). Spec 15 treats the recordings as +the most consequential data in the system and +`docs/verification/end-to-end-checklist.md` makes "no transcript, audio, +token or key appears in any pod log" a blocking legal gate. + +Three stores with three redaction implementations is worse than one store +with none: each would be correct about a slightly different set of names, +and the gap between them is where a transcript gets out. So this package +holds **one** field registry (`fields.ALLOWED_FIELDS`) and **one** +scrubbing function (`redaction.scrub_fields`), and every channel is built +on top of them: + +- log records, through `events.log_event` and `redaction.SturnusFilter`; +- span and metric attributes, through + `sturnus.infrastructure.telemetry.span` / `_metric_attributes`; +- Sentry exception messages, through `redaction.SAFE_MESSAGE_TYPES`, which + `sturnus.infrastructure.observability.SAFE_VALUE_TYPES` re-exports rather + than restating. + +Adding a field is therefore one edit, in one file, that shows up in review +as "we decided to put this in Loki, Tempo and Grafana" -- which is what it +is. + +**Standard library only, deliberately.** `sturnus.application` already uses +stdlib `logging` in four modules and must be able to call `log_event`; the +architecture rule in `tests/test_architecture.py` forbids it importing +third-party packages, and `tests/observability/test_package_boundaries.py` +holds this package to the same standard so that importing it from +`application` can never smuggle OpenTelemetry in behind it. The +OpenTelemetry SDK lives in `sturnus.infrastructure.telemetry` and imports +*this* package, never the other way round. +""" + +from __future__ import annotations diff --git a/src/sturnus/observability/events.py b/src/sturnus/observability/events.py new file mode 100644 index 0000000..39c93de --- /dev/null +++ b/src/sturnus/observability/events.py @@ -0,0 +1,271 @@ +"""The event vocabulary, and the one sanctioned way to emit a log line. + +An operator's questions are about a *session*, not about a module. The +names below are chosen so that `| json | session_id="4711"` in LogQL +returns one readable narrative that crosses all three processes: + + guild.configured -> voice.joined -> session.opened + -> session.speaker_first_packet -> session.closing + -> session.speaker_finalized -> session.closed + -> job.claimed -> job.transcribed -> session.document_created + -> announce.posted + +`session_id` is the join key for the whole story and is line content, never +a Loki label -- it is unbounded, and promoting it would multiply the cluster's +stream count without limit. `docs/operations.md` section 7 carries the label +policy and the queries this vocabulary was designed to answer. + +Levels are part of the design, not decoration: + +- `DEBUG` -- counts, sizes and housekeeping. Sturnus's own DEBUG lines are + held to ids, counts, sizes and durations by the same registry that governs + INFO; they are never payload. +- `INFO` -- the narrative above. One line per event, never per packet. +- `WARNING` -- retried, and expected to self-heal. +- `ERROR` -- **a human must act.** Reserved for permanent loss, for capture + that has silently stopped, and for a guild that has stopped being able to + record. `job.dead`, `session.unrecoverable`, `session.document_rejected`, + `voice.join_failed`, `voice.reader_stopped`, `voice.decode_failed`, + `voice.packet_handler_failed`, `voice.left_failed`, + `voice.rejoin_blocked`, `guild.tick_failed` and `session.close_failed` + are the ones that earn it. + +`voice.join_failed`, `voice.reader_stopped` and `voice.decode_failed` are +one family and are deliberately not one name. They are the three ways this +process can end up in a voice channel hearing nothing while everyone in it +has been told they are recorded: capture never started +(`voice.join_failed`), capture started and then died +(`voice.reader_stopped`), or capture is running and no stream decodes any +more (`voice.decode_failed`). All three end the session with an +`end_reason` that says "we could not hear" rather than "nobody spoke", and +telling them apart in Loki is the difference between suspecting libopus, +suspecting the gateway, and suspecting the channel. + +The last four are the bot's *operational* failures, and they were bare +`log.exception("... %d ...", guild_id)` calls until this vocabulary reached +them. A `%d`-formatted id is invisible to `| json | guild_id="..."`, which +is the one query this whole package exists to make possible -- and +`scrub_event` forwards `LogRecord.msg` to Sentry, so an id interpolated +into the message is also the half of the line that leaves the pod. They +carry fields now, and the message stayed a literal. +""" + +from __future__ import annotations + +import logging +from collections.abc import Callable +from enum import StrEnum +from typing import Final + + +class Event(StrEnum): + """The closed set of event names. New lines pick a name from here.""" + + # -- bot: the session story ------------------------------------------- + GUILD_CONFIGURED = "guild.configured" + GUILD_UNCONFIGURED = "guild.unconfigured" + BOT_CONNECTED = "bot.connected" + VOICE_JOINED = "voice.joined" + VOICE_LEFT = "voice.left" + VOICE_JOIN_FAILED = "voice.join_failed" + VOICE_READER_STOPPED = "voice.reader_stopped" + VOICE_DECODE_FAILED = "voice.decode_failed" + VOICE_PACKET_REJECTED = "voice.packet_rejected" + VOICE_PACKET_HANDLER_FAILED = "voice.packet_handler_failed" + VOICE_LEFT_FAILED = "voice.left_failed" + VOICE_REJOIN_BLOCKED = "voice.rejoin_blocked" + GUILD_TICK_FAILED = "guild.tick_failed" + SESSION_CLOSE_FAILED = "session.close_failed" + SESSION_OPENED = "session.opened" + SESSION_SPEAKER_FIRST_PACKET = "session.speaker_first_packet" + #: A speaker whose packets arrive, decode, and carry no audible level -- + #: what a microphone muted at system level produces. Three lines, because + #: the durable record and the message into the room can each fail on + #: their own and neither may take the capture path down with it. + SPEAKER_AUDIO_SILENT = "speaker.audio_silent" + SPEAKER_SILENT_WARNING_FAILED = "speaker.silent_warning_failed" + SPEAKER_SILENT_RECORD_FAILED = "speaker.silent_record_failed" + SESSION_CLOSING = "session.closing" + SESSION_SPEAKER_FINALIZED = "session.speaker_finalized" + SESSION_CLOSED = "session.closed" + SESSION_RECOVERED = "session.recovered" + SESSION_UNRECOVERABLE = "session.unrecoverable" + #: `/queue requeue`'s confirmation buttons could not be greyed out. The + #: answer the administrator is waiting for is not lost with them -- the + #: edit is swallowed on purpose -- so this line is the only trace. + QUEUE_VIEW_DISABLE_FAILED = "queue.view_disable_failed" + ANNOUNCE_POSTED = "announce.posted" + ANNOUNCE_FAILED = "announce.failed" + AUDIO_ERASED = "audio.erased" + + # -- worker ------------------------------------------------------------ + WORKER_STARTED = "worker.started" + JOB_CLAIMED = "job.claimed" + JOB_TRANSCRIBED = "job.transcribed" + TRANSCRIPTION_SKIPPED = "transcription.skipped" + TRANSCRIPTION_DECODED = "transcription.decoded" + JOB_FAILED = "job.failed" + JOB_DEAD = "job.dead" + KEY_ID_MISMATCH = "key.id_mismatch" + SESSION_DOCUMENT_CREATED = "session.document_created" + SESSION_DOCUMENT_REJECTED = "session.document_rejected" + SESSION_DOCUMENT_RETRY_FAILED = "session.document_retry_failed" + RETENTION_SWEPT = "retention.swept" + RETENTION_FAILED = "retention.failed" + + # -- link --------------------------------------------------------------- + LINK_STARTED = "link.started" + LINK_CALLBACK_REJECTED = "link.callback_rejected" + LINK_EXCHANGE_FAILED = "link.exchange_failed" + LINK_ESTABLISHED = "link.established" + LINK_STATES_PURGED = "link.states_purged" + + # -- cross-cutting ------------------------------------------------------ + PROCESS_STARTING = "process.starting" + SHUTDOWN_BEGIN = "shutdown.begin" + SHUTDOWN_COMPLETE = "shutdown.complete" + SCHEMA_WAITING = "schema.waiting" + #: The startup line that says a requested third-party log level was + #: raised to `setup.THIRD_PARTY_FLOOR`. An operator who turned the knob + #: up and sees nothing new needs to be told why, in the same place they + #: are already looking. + LOG_LEVEL_CLAMPED = "log.level_clamped" + SWEEP_FAILED = "sweep.failed" + TELEMETRY_ENABLED = "telemetry.enabled" + UNHANDLED_EXCEPTION = "unhandled.exception" + + +#: Filled in by `sturnus.infrastructure.telemetry.install_trace_context` +#: once an OpenTelemetry provider exists. A module-level hook rather than an +#: import because this package is standard-library only (see the package +#: docstring) and must never reach for the OTel API; with no telemetry +#: installed it stays `None` and every log line simply has no `trace_id`. +#: +#: This is the Loki -> Tempo link: a `trace_id` in the JSON line is what a +#: Grafana derived field turns into a click through to the waterfall for the +#: same job. +_trace_context_provider: Callable[[], dict[str, str]] | None = None + + +def set_trace_context_provider(provider: Callable[[], dict[str, str]] | None) -> None: + """Installs (or clears) the hook that supplies `trace_id`/`span_id`.""" + global _trace_context_provider + _trace_context_provider = provider + + +def current_trace_context() -> dict[str, str]: + """`{"trace_id": ..., "span_id": ...}` when a span is active, else `{}`. + + Never raises: a broken telemetry provider must not be able to stop a log + line being written, because the log line is the fallback for telemetry + being broken. + """ + provider = _trace_context_provider + if provider is None: + return {} + try: + return provider() + except Exception: # pragma: no cover - defensive; see docstring + return {} + + +def log_event( + logger: logging.Logger, + level: int, + event: Event, + message: str, + /, + **fields: object, +) -> None: + """Emits one structured event. The only sanctioned log call shape. + + `message` must be a plain string literal -- no f-string, no `%` + interpolation, no concatenation. Everything that varies goes in + `**fields`, where `redaction.scrub_fields` rebuilds it from the + registry. This is what makes the human-readable half of a line + reviewable source text rather than a place data can hide, and it is the + same guarantee `sturnus.infrastructure.observability.scrub_event` + already relies on when it forwards `logentry.message` and nothing else + to Sentry. + + `tests/test_logging_discipline.py` enforces both halves: the literal + message, and every field name being registered. + """ + logger.log( + level, + message, + extra={"sturnus_event": str(event), "sturnus_fields": dict(fields)}, + ) + + +def log_exception( + logger: logging.Logger, + level: int, + event: Event, + message: str, + exc: BaseException, + /, + **fields: object, +) -> None: + """`log_event` plus a stack trace, with `error_type` filled in. + + Note what is *not* here: the exception is never passed as a `%` + argument. `log.warning("failed: %s", exc)` -- twelve of which existed + before this package -- prints `str(exc)` verbatim, and a + `jinja2.UndefinedError` raised while rendering a transcript through the + Outline template carries template context in exactly that string. The + type is a registered field, the traceback is rendered by + `setup.SafeFormatterMixin` from static program text, and the message + itself travels only if `redaction.SAFE_MESSAGE_TYPES` vouches for its + class. + """ + from sturnus.observability.redaction import error_type + + logger.log( + level, + message, + exc_info=exc, + extra={ + "sturnus_event": str(event), + "sturnus_fields": {"error_type": error_type(exc), **fields}, + }, + ) + + +class RateLimiter: + """Lets the first occurrence through, then one in every `every`. + + For events that are per-packet in origin but must not be per-packet in + Loki. `voice.packet_handler_failed` used to be a `log.error` on every + failed packet: during a systematic failure -- which is the only time it + matters -- that is its own flood, at ~50 lines per second per speaker. + One line carrying `count` says strictly more and costs four orders of + magnitude less. + + Not thread-safe by construction, and it does not need to be: the + counter is a plain `int` increment, the GIL makes that atomic enough for + a rate limiter, and the worst outcome of a lost increment is one line + logged early. + """ + + def __init__(self, every: int = 1000) -> None: + self._every = every + self._count = 0 + + def should_log(self) -> bool: + self._count += 1 + return self._count == 1 or self._count % self._every == 0 + + @property + def count(self) -> int: + return self._count + + def reset(self) -> None: + self._count = 0 + + +#: Level constants re-exported so a call site needs one import, not two. +DEBUG: Final = logging.DEBUG +INFO: Final = logging.INFO +WARNING: Final = logging.WARNING +ERROR: Final = logging.ERROR diff --git a/src/sturnus/observability/fields.py b/src/sturnus/observability/fields.py new file mode 100644 index 0000000..32b12b0 --- /dev/null +++ b/src/sturnus/observability/fields.py @@ -0,0 +1,329 @@ +"""The closed registry of what Sturnus is prepared to put in a retained store. + +Read this file before adding a name to it. Every entry below is copied into +Loki and, unless it is in `LOG_ONLY_FIELDS`, into Tempo -- both of which +index it, retain it, and show it to anyone with Grafana access. The gate in +`docs/verification/end-to-end-checklist.md` states the standard this list is +held to: "A log line naming a user id, a job id, or a status code is fine; a +log line containing what someone said, or the bytes of what they said, or a +credential, is not." + +**Allowlist, not denylist.** `redaction.scrub_fields` rebuilds its output +from `ALLOWED_FIELDS` rather than deleting known-bad keys from its input, +the same inversion `sturnus.infrastructure.observability.scrub_event` makes +for Sentry events and for the same reason: a denylist is correct about the +code it was written against and silently wrong about the next call site +somebody adds. Rebuilding inverts the failure mode -- a field nobody +registered is dropped, so a mistake costs a missing panel in Grafana rather +than a transcript in Loki. + +**One registry, three spellings, derived not restated.** Logs want flat +snake_case (`job_id`), OpenTelemetry wants dotted attributes +(`sturnus.job_id`), and a handful of concepts already have names in the +OpenTelemetry semantic conventions (`error.type`) that would be perverse to +reinvent. `span_attribute()` derives the span spelling from the log +spelling, so there is no second list to fall out of step with this one. +`tests/observability/test_redaction.py` pins the derivation, and +`tests/infrastructure/test_telemetry.py` checks the semantic-convention +literals below against the constants the installed +`opentelemetry-semantic-conventions` actually exports. +""" + +from __future__ import annotations + +from typing import Final + +#: The three processes built from the one image. The same literal names +#: `sturnus.infrastructure.observability.init_sentry` takes as its +#: `component` tag and `[project.scripts]` uses, so an operator reading a +#: Sentry issue, a Loki stream and a Tempo service graph sees one word for +#: one process rather than three near-synonyms. +COMPONENTS: Final = ("bot", "worker", "link") + + +#: OpenTelemetry `service.name` per component. Derived, not tabulated, so a +#: fourth component cannot arrive with a name that agrees with nothing. +def service_name(component: str) -> str: + """`service.name` for one component -- `bot` -> `sturnus-bot`.""" + return f"sturnus-{component}" + + +# --------------------------------------------------------------------------- +# The registry +# --------------------------------------------------------------------------- + +#: Opaque identifiers. Database primary keys and platform-issued ids: each +#: names a row, a server, a room or a document, and none of them is content. +#: `guild_id`/`channel_id` are already logged today +#: (`infrastructure/discord/client.py`), and `document_id` at +#: `infrastructure/documents/outline.py`. +_IDENTIFIERS = frozenset( + { + "session_id", + "job_id", + "guild_id", + "channel_id", + "document_id", + "collection_id", + "key_id", + "configured_key_id", + "provider", + "ssrc", + } +) + +#: Identifiers for a *person*. Pseudonymous rather than identifying -- a +#: Discord snowflake and an Outline UUID -- and an operator cannot answer +#: "did this person's `/audio delete` actually erase their recordings" +#: without them, which is itself a compliance question. `audio_cog.py` +#: already logs the Discord one. +#: +#: They are nonetheless **log-only**: see `LOG_ONLY_FIELDS`. +_SUBJECT_IDENTIFIERS = frozenset({"discord_user_id", "external_user_id"}) + +#: Fixed literals from this repository's own source: enum members, stage +#: names, outcome words. Bounded by construction, which is what makes them +#: safe as metric labels as well as safe to log. +#: +#: `close_code` and `http_status` are the two exceptions to "from this +#: repository's own source", and they are bounded by the same argument from +#: somewhere else: both are protocol constants with a documented, finite +#: value set (RFC 6455 plus Discord's own 4xxx voice codes; RFC 9110). See +#: `infrastructure.discord.voice.voice_close_code` for why `close_code` +#: exists at all -- it is the diagnosis a withheld exception message takes +#: with it. +_LITERALS = frozenset( + { + "close_code", + "component", + "stage", + "outcome", + "reason", + "end_reason", + "status", + "language", + "model", + "device", + "compute_type", + "version", + "error_type", + "http_method", + "http_status", + "server_address", + "url_path", + "permanent", + "is_last", + "listening", + "missing", + } +) + +#: Counts, sizes and durations. Numbers about content, never content. +_MEASUREMENTS = frozenset( + { + "attempt", + "attempts", + "max_attempts", + "lease_seconds", + "count", + "deleted", + "failed", + "speakers", + "participants", + "blocks", + "packets", + "segments", + "segment_count", + "char_count", + "title_chars", + "body_bytes", + "bytes", + "object_bytes", + "plaintext_bytes", + "audio_seconds", + #: How much of a speaker's recording `speech_gate.speech_clips` + #: found above the silence floor, and in how many clips. Against + #: `audio_seconds` this is the gate's own signature and the one + #: number that names the failure mode that cost this project two + #: days: "one second of speech in two minutes of recording" is not + #: a plausible meeting. Nothing else can report it -- + #: `job.transcribed` is emitted from `sturnus.application.worker`, + #: which never sees the gate. + "speech_seconds", + "clips", + "wall_seconds", + "realtime_factor", + "duration_seconds", + "seconds_since_last_packet", + "consented_present", + "jobs_enqueued", + } +) + +#: Correlation ids the OpenTelemetry SDK produces. 128- and 64-bit random +#: numbers carrying no data; they are what turns a Loki line into a click +#: through to the Tempo trace. +_CORRELATION = frozenset({"trace_id", "span_id"}) + +ALLOWED_FIELDS: Final[frozenset[str]] = ( + _IDENTIFIERS | _SUBJECT_IDENTIFIERS | _LITERALS | _MEASUREMENTS | _CORRELATION +) + +#: Registered, logged, and deliberately kept out of spans and metrics. +#: +#: The checklist blesses a user id in a pod log, and this codebase already +#: writes one. A user id joined to a session id and precise timestamps in a +#: *searchable, retained, trace-indexed* store is a different artifact: a +#: record of who was in which voice channel when, reachable by a wider +#: Grafana audience than `kubectl logs`. Nothing is lost by the exclusion -- +#: `sturnus.session_id` on the span joins to the row that has the user id, +#: for anyone with the access to look -- and it keeps metric cardinality +#: bounded for free. +#: +#: `SAFE_SPAN_ATTRIBUTES` and `METRIC_LABEL_FIELDS` are both computed with +#: this set subtracted, so the exclusion is a control rather than a habit. +LOG_ONLY_FIELDS: Final[frozenset[str]] = _SUBJECT_IDENTIFIERS + +#: Fields that may become a *metric* attribute. Metrics multiply: one new +#: value of one attribute is a whole new time series forever. Only fixed +#: source literals and `guild_id` qualify -- see `docs/operations.md` +#: section 7 for the cardinality argument, and note that `guild_id` is the +#: first thing to review if Sturnus is ever deployed multi-tenant at scale. +METRIC_LABEL_FIELDS: Final[frozenset[str]] = frozenset( + { + "component", + #: The Whisper model name, and the only label the transcription + #: progress metrics carry beyond it. Bounded by deployment rather + #: than by source -- `STURNUS_WHISPER_MODEL` is a Helm value -- but + #: bounded all the same: one process loads exactly one model, so a + #: cluster produces one series per distinct model ever deployed. + #: It is also the dimension the numbers are meaningless without, + #: since a real-time factor compares `large-v3` against `tiny` + #: otherwise. + "model", + "stage", + "outcome", + "reason", + "end_reason", + "status", + "language", + "error_type", + "http_status", + "permanent", + "guild_id", + } +) + +# --------------------------------------------------------------------------- +# Names that must never appear, and are named so the test can say so +# --------------------------------------------------------------------------- + +#: Argument, attribute and `extra`-key names that carry payload. Not the +#: mechanism that keeps them out -- `ALLOWED_FIELDS` is, by rebuilding -- +#: but the list `tests/test_logging_discipline.py` fails the build on, so +#: `log_event(Event.JOB_DONE, transcript=result.text)` is caught in CI +#: rather than dropped silently at runtime and wondered about later. +#: +#: `s3_key` is here rather than in the registry on purpose. The key format +#: is `sessions/{session_id}/speakers/{discord_user_id}.enc` +#: (`sturnus.application.recording.audio_key`), so it *embeds* a user id -- +#: and both halves are already registered separately, which makes the key +#: itself pure duplication with a wider blast radius. +#: The payload half: what someone said, or who said it. +_PAYLOAD_NAMES: Final[frozenset[str]] = frozenset( + { + "transcript", + "transcripts", + "text", + "body", + "pcm", + "opus", + "packet_data", + "payload", + "plaintext", + "display_name", + "discord_display_name", + "participant_names", + "s3_key", + } +) + +#: The credential half: what authorises or decrypts. +#: +#: Split out from the payload half rather than kept as one flat set, +#: because `redaction.PATTERNS` needs exactly this half and nothing else. +#: A string of the shape `: ` in *any* record -- +#: including a message a third-party library composed, which no allowlist +#: over Sturnus's own field names can see -- is scrubbed on its way to the +#: formatter. Applying the payload half there too would redact the word +#: after every "text:" and "body:" an English sentence contains, which is +#: how a control earns itself a `# noqa`. +#: +#: Derived, not restated: `DENIED_NAMES` below is the union, so the static +#: rule in `tests/test_logging_discipline.py` still sees one list and the +#: two halves cannot drift apart into disagreement. +CREDENTIAL_NAMES: Final[frozenset[str]] = frozenset( + { + "token", + "access_token", + "api_token", + "api_key", + "secret", + "client_secret", + "secret_key", + "access_key", + "master_key", + "data_key", + "wrapped", + "wrapped_data_key", + "password", + "authorization", + "database_url", + "dsn", + "get_secret_value", + } +) + +DENIED_NAMES: Final[frozenset[str]] = _PAYLOAD_NAMES | CREDENTIAL_NAMES + +# --------------------------------------------------------------------------- +# Deriving the span spelling +# --------------------------------------------------------------------------- + +#: The prefix for attributes this project invents. Everything under it is +#: ours; everything outside it in `SAFE_SPAN_ATTRIBUTES` is a name +#: OpenTelemetry already standardised. +SPAN_ATTRIBUTE_NAMESPACE: Final = "sturnus." + +#: Fields whose span spelling is an OpenTelemetry semantic convention +#: rather than `sturnus.`. Written as literals because this module is +#: standard-library only (see the package docstring); +#: `tests/infrastructure/test_telemetry.py` asserts each one equals the +#: constant `opentelemetry-semantic-conventions` exports, so the literal +#: cannot drift from the convention without a red test. +SEMCONV_SPAN_ATTRIBUTES: Final[dict[str, str]] = { + "error_type": "error.type", + "http_method": "http.request.method", + "http_status": "http.response.status_code", + "server_address": "server.address", + "url_path": "url.path", +} + + +def span_attribute(field: str) -> str: + """The span-attribute spelling of a registered field name. + + The single conversion between the two vocabularies. Nothing else in the + codebase writes an attribute key as a string literal, which is what + keeps `SAFE_SPAN_ATTRIBUTES` and the emitting call sites in step. + """ + return SEMCONV_SPAN_ATTRIBUTES.get(field, f"{SPAN_ATTRIBUTE_NAMESPACE}{field}") + + +#: Every attribute key an exported span may carry, derived from the registry +#: above. `sturnus.infrastructure.telemetry.AllowlistingSpanExporter` +#: rebuilds each span from exactly this set on the way out -- the second, +#: independent lock behind `scrub_fields`. +SAFE_SPAN_ATTRIBUTES: Final[frozenset[str]] = frozenset( + span_attribute(field) for field in ALLOWED_FIELDS - LOG_ONLY_FIELDS +) diff --git a/src/sturnus/observability/redaction.py b/src/sturnus/observability/redaction.py new file mode 100644 index 0000000..6f797d6 --- /dev/null +++ b/src/sturnus/observability/redaction.py @@ -0,0 +1,352 @@ +"""The one scrubbing implementation. Spans, metrics, logs and Sentry share it. + +`scrub_fields` is the function the whole design rests on: it rebuilds a +mapping from `fields.ALLOWED_FIELDS` and passes every surviving value +through `scrub_value`. `events.log_event` calls it, `SturnusFilter` calls it +again on the way to the formatter, and +`sturnus.infrastructure.telemetry.span` calls it before turning fields into +span attributes. There is no second copy to disagree with it. + +Four mechanisms, ordered from "prevents the value existing" to "prevents it +leaving the process". Each is structural; none relies on a reviewer +remembering. + +1. **Unregistered keys are dropped.** Allowlist rebuild -- see + `fields.ALLOWED_FIELDS`. +2. **`bytes` are never rendered.** Any `bytes`/`bytearray`/`memoryview`, + anywhere, becomes ``. Unconditional and name-blind, which + is what makes it the highest-value rule here: the most consequential + payload in Sturnus -- raw PCM, Opus frames, encrypted blobs, wrapped + data keys -- is always `bytes`, and this closes all of it as a class. +3. **Strings are pattern-scrubbed and capped.** Discord token shape, + `AKIA…`, `Bearer …`, `scheme://user:pass@host`, long base64 runs, and + -- for the one secret in this system that has no recognisable shape -- + anything assigned to a name in `fields.CREDENTIAL_NAMES`. The cap bounds + the blast radius of anything the patterns miss: half a sentence is still + a breach, a 40-minute conversation is a categorically larger one. + + That last pattern is the only rule here aimed at text this codebase did + not write. Rules 1 and 4 govern *our* fields and *our* exceptions, and a + third-party logger's `%s`-interpolated message is neither: the Discord + voice `secret_key` arrives as thirty-two small integers inside + `record.msg`, where an allowlist over field names cannot see it and no + shape-based pattern would recognise it. It is the second lock behind + `setup.THIRD_PARTY_FLOOR`, which is what stops that record being emitted + at all -- deliberately two independent mechanisms, because the first is + a level and levels are what operators change. +4. **Exception messages are allowlisted by type, not scrubbed by + guesswork.** `safe_exception_message` -- see `SAFE_MESSAGE_TYPES`. + +Replacements are visible (`«redacted:discord_token»`) rather than silent. A +redaction that leaves no trace teaches nobody and hides its own false +positives; this one shows up in the line as a confusing value rather than +as missing data, which is the right failure direction. +""" + +from __future__ import annotations + +import logging +import re +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Final + +from sturnus.domain.errors import DiagnosticSafeError +from sturnus.observability.fields import ALLOWED_FIELDS, CREDENTIAL_NAMES, DENIED_NAMES + +REDACTED: Final = "" + +#: Every string value is truncated to this many characters after pattern +#: scrubbing. Generous enough for a stack frame's source line, far too +#: short for a transcript. +MAX_FIELD_CHARS: Final = 512 + +#: Exception types whose `str()` may travel verbatim. +#: +#: The single source of truth for this rule. +#: `sturnus.infrastructure.observability.SAFE_VALUE_TYPES` -- Sentry's +#: `before_send` -- is an alias of this tuple rather than a second list, so +#: "which exception messages may leave the pod" is answered in one place for +#: Sentry, Tempo and Loki alike. +#: +#: `OSError` covers `ConnectionError`, `TimeoutError`, `ssl.SSLError` and +#: `socket.gaierror`: failures of a process talking to Discord, S3, Postgres +#: and Outline, with messages composed by the OS and the standard library +#: rather than by us. `DiagnosticSafeError` is the explicit opt-in and +#: carries the contract in its own docstring. +SAFE_MESSAGE_TYPES: Final[tuple[type[BaseException], ...]] = (OSError, DiagnosticSafeError) + +#: The credential half of `fields.DENIED_NAMES`, spelled as a regex +#: alternation. Longest first, because Python's `|` is first-match and +#: `secret` would otherwise win against `secret_key` and leave `_key: ...` +#: dangling in front of the marker. +_CREDENTIAL_NAME_ALTERNATION: Final = "|".join( + re.escape(name) for name in sorted(CREDENTIAL_NAMES, key=lambda n: (-len(n), n)) +) + +#: Patterns applied to every string value. Each replacement names itself, so +#: an operator seeing `«redacted:aws_access_key_id»` knows a control fired +#: rather than wondering where a field went. +#: +#: This layer is also what catches a leak that predates this package. +#: `sturnus.config.StrictSettings._reject_blank_required_values` raises +#: through pydantic, which embeds the raw input dict in the `ValidationError` +#: message -- `input_value={'discord_token': 'TOKEN_...'}` -- and that +#: exception escapes `asyncio.run` to the default excepthook, to stderr, to +#: Alloy, to Loki. `setup.install_excepthooks` routes it here instead, and +#: the Discord-token pattern redacts what pydantic's own 50-character +#: truncation left of it. +PATTERNS: Final[tuple[tuple[str, re.Pattern[str]], ...]] = ( + # Discord bot token: three dot-separated base64url segments, the first + # of which is a base64-encoded snowflake. + ( + "discord_token", + re.compile(r"\b[A-Za-z0-9_-]{23,28}\.[A-Za-z0-9_-]{6,7}\.[A-Za-z0-9_-]{27,}"), + ), + ("aws_access_key_id", re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b")), + ("bearer_token", re.compile(r"(?i)\bBearer\s+[A-Za-z0-9._~+/=-]{8,}")), + ("aws_sigv4", re.compile(r"(?i)X-Amz-Signature=[A-Za-z0-9%]+")), + ("url_credentials", re.compile(r"\b([a-zA-Z][a-zA-Z0-9+.-]*://)[^/\s:@]+:[^/\s@]+@")), + # A long unbroken base64-ish run is a key, a wrapped key, or a blob. No + # legitimate field in `ALLOWED_FIELDS` looks like this. + ("base64_blob", re.compile(r"\b[A-Za-z0-9+/]{64,}={0,2}")), + # **The name-shaped rule, and the only one aimed at a message string + # somebody else composed.** Last in the tuple deliberately: it is the + # broadest, and running it after the shape-based patterns above lets + # the more specific control name itself in the line + # (`Authorization: «redacted:bearer_token»` rather than + # `Authorization: «redacted:secret_value»`), which is the difference + # between an operator knowing what fired and guessing. + # + # It exists because the Discord voice secret key does not *look* like a + # secret. `discord/ext/voice_recv/gateway.py` pretty-prints the op-4 + # payload, so the key reaches a record as + # `'secret_key': [1, 2, 3, ...]` -- thirty-two small integers, matching + # none of the shapes above, and inside `record.msg` rather than in any + # field this package's allowlist governs. The level floor in + # `setup.THIRD_PARTY_FLOOR` is what stops that record existing; this is + # what stops the value travelling if a future release moves the same + # line to a level the floor permits. + # + # The value alternation runs to the end of the line rather than to the + # end of the token, because `reader.py`'s + # `"CryptoError details:\n data=%s\n secret_key=%s"` puts the whole + # key there with no delimiter after it. Over-redaction is the intended + # failure direction: a confusing `«redacted:secret_value»` in a line is + # recoverable, and the alternative is not. + ( + "secret_value", + re.compile( + r"(?P['\"]?\b(?:" + _CREDENTIAL_NAME_ALTERNATION + r")\b['\"]?\s*+[:=]\s*+)" + # Not a value an earlier, more specific pattern already + # replaced -- overwriting `\u00abredacted:bearer_token\u00bb` with + # `\u00abredacted:secret_value\u00bb` would lose which control fired. + # + # The `\s*+` above are possessive for this lookahead's sake. A + # greedy `\s*` gives its whitespace back when the lookahead + # fails, so `Authorization: \u00abredacted:bearer_token\u00bb` would + # re-match with `keep` one space shorter and the lookahead + # satisfied by that space -- passing the guard by stepping + # around it. + r"(?!\u00abredacted:)" + r"(?:\[[^\]]*\]|'[^']*'|\"[^\"]*\"|[^\n,;)\]}]+)", + re.IGNORECASE, + ), + ), +) + +log = logging.getLogger(__name__) + + +def scrub_text(value: str) -> str: + """Pattern-scrubs and truncates one string.""" + for name, pattern in PATTERNS: + replacement = f"«redacted:{name}»" + if name == "url_credentials": + value = pattern.sub(rf"\g<1>{replacement}@", value) + elif "keep" in pattern.groupindex: + # A pattern that recognises a secret by the *name* it is + # assigned to keeps that name: `secret_key=«redacted:…»` tells + # an operator which value went, where a bare marker would leave + # them unable to tell a redaction from a missing field. + value = pattern.sub(rf"\g{replacement}", value) + else: + value = pattern.sub(replacement, value) + if len(value) > MAX_FIELD_CHARS: + return value[:MAX_FIELD_CHARS] + f"…«truncated to {MAX_FIELD_CHARS} chars»" + return value + + +def scrub_value(value: object) -> object: + """Renders one value safe, whatever it is. + + `bool` is checked before `int` because `bool` is a subclass of it and + JSON should keep `true` rather than `1`. Anything that is not a scalar, + a string, bytes, a `Path` or a short sequence of those is replaced by + its *type name* -- never its `repr`, which is how an object holding a + transcript would otherwise render itself into the line. + """ + if isinstance(value, bytes | bytearray | memoryview): + # Unconditional, name-blind, and the single most valuable rule in + # this module: audio is always bytes. + return f"" + if value is None or isinstance(value, bool | int | float): + return value + if isinstance(value, str): + return scrub_text(value) + if isinstance(value, Path): + return scrub_text(str(value)) + if isinstance(value, Mapping): + return {str(k): scrub_value(v) for k, v in list(value.items())[:32]} + if isinstance(value, Sequence): + return [scrub_value(item) for item in list(value)[:32]] + return f"<{type(value).__name__}>" + + +def scrub_fields(fields: Mapping[str, object], *, warn: bool = True) -> dict[str, object]: + """Rebuilds a field mapping from `ALLOWED_FIELDS`, scrubbing what survives. + + The shared chokepoint: `events.log_event`, the log formatters and + `sturnus.infrastructure.telemetry.span_attributes` all pass through + here, so a name is judged once and identically for Loki, Tempo and the + metric store. + + `warn=True` -- the default, used for fields a Sturnus call site passed + deliberately -- logs the *name* of anything unregistered. Dropping it + silently would make a typo indistinguishable from a field that is + simply always empty, which is the exact failure mode this package is + arranged to avoid. + + `warn=False` is for sweeping third-party `LogRecord` attributes, where + unregistered is the normal case and not a mistake: `aiohttp.access` + alone attaches six per request, and a warning for each would be a flood + of this module's own making. + + Only the key is ever logged, never the value, and only when it is a + plain identifier -- the value is precisely what must not travel, and a + non-identifier key is itself suspicious enough not to echo. + """ + out: dict[str, object] = {} + for key, value in fields.items(): + if key not in ALLOWED_FIELDS: + if warn: + log.warning( + "Dropping unregistered telemetry field %s", + key if key.isidentifier() else "", + ) + continue + out[key] = scrub_value(value) + return out + + +def is_message_safe(exc: BaseException) -> bool: + """Whether this exception's message may travel verbatim.""" + return isinstance(exc, SAFE_MESSAGE_TYPES) + + +def safe_exception_message(exc: BaseException) -> str: + """This exception's message, or a placeholder naming the type that was withheld. + + Nothing structural separates `ConnectionRefusedError: [Errno 111]` from + `RuntimeError: failed on ` -- both are a string in `args` -- + so the default is redaction and the exceptions are named in + `SAFE_MESSAGE_TYPES`. + + This is a real loss and it is worth stating plainly: there will be a + first incident where the withheld message was the answer. The + alternative is regex-scrubbing arbitrary third-party exception text, + which is guesswork dressed as a control. Where a specific field of a + message matters -- Outline's status code, the failing stage -- the call + site captures it as a registered field instead, which is both safer and + more queryable than the sentence it came from. + """ + if is_message_safe(exc): + return scrub_text(str(exc)) + return f"" + + +def error_type(exc: BaseException) -> str: + """The value of the `error_type` field: a class name, never a message.""" + return type(exc).__qualname__ + + +class SturnusFilter(logging.Filter): + """Scrubs every record on its way to the formatter. + + Installed on the **handler**, not on a logger, which is the whole point: + `botocore`'s records and `discord.ext.voice_recv`'s records pass through + it exactly as Sturnus's own do. A call site cannot route around it + without adding a second handler, and `setup.configure_logging` replaces + `root.handlers` wholesale so there is exactly one -- + `tests/observability/test_setup.py` asserts that. + + What it does to a record: + + - drops any `extra` attribute whose name is in `DENIED_NAMES`, whatever + the call site passed; + - rebuilds `sturnus_fields` through `scrub_fields`; + - scrubs `record.args` positionally, so `%s`-interpolated third-party + values get the bytes rule and the pattern rule too; + - leaves `record.msg` alone for Sturnus's own loggers, where + `tests/test_logging_discipline.py` and ruff's `G` ruleset guarantee it + is a literal, and scrubs it for everyone else. + """ + + def filter(self, record: logging.LogRecord) -> bool: + for denied in DENIED_NAMES: + if hasattr(record, denied): + delattr(record, denied) + + fields = getattr(record, "sturnus_fields", None) + if isinstance(fields, Mapping): + record.sturnus_fields = scrub_fields(fields) + + if record.args: + if isinstance(record.args, tuple): + record.args = tuple(scrub_value(arg) for arg in record.args) + elif isinstance(record.args, Mapping): + record.args = {k: scrub_value(v) for k, v in record.args.items()} + + if isinstance(record.msg, str) and not record.name.startswith("sturnus"): + record.msg = scrub_text(record.msg) + + return True + + +def format_exception_safely(exc: BaseException) -> list[str]: + """The exception chain as type names plus vouched-for messages. + + Deliberately not `traceback.format_exception`: that renders `str(exc)` + for every exception in the chain, which is precisely the string + `safe_exception_message` exists to withhold. The frames themselves are + rendered separately by the formatter -- file, line, function and source + text are static program text and carry nothing. + """ + lines: list[str] = [] + seen: set[int] = set() + current: BaseException | None = exc + while current is not None and id(current) not in seen: + seen.add(id(current)) + lines.append( + f"{type(current).__module__}.{type(current).__qualname__}: " + f"{safe_exception_message(current)}" + ) + current = current.__cause__ or current.__context__ + return lines + + +__all__ = [ + "ALLOWED_FIELDS", + "MAX_FIELD_CHARS", + "PATTERNS", + "REDACTED", + "SAFE_MESSAGE_TYPES", + "SturnusFilter", + "error_type", + "format_exception_safely", + "is_message_safe", + "safe_exception_message", + "scrub_fields", + "scrub_text", + "scrub_value", +] diff --git a/src/sturnus/observability/setup.py b/src/sturnus/observability/setup.py new file mode 100644 index 0000000..73aa3cc --- /dev/null +++ b/src/sturnus/observability/setup.py @@ -0,0 +1,629 @@ +"""Installs the one handler every log record leaves through. + +`configure_logging` replaces `root.handlers` wholesale -- it never appends -- +with a single `StreamHandler(sys.stdout)` carrying `SturnusFilter` and one +of the two formatters. That is the design: `alloy-logs` runs as a DaemonSet +scraping container stdout into Loki, so Sturnus ships no logs itself and +"optimising for Loki" means changing what it prints, not adding a shipper. +One handler means one place records can leave the process, and the filter +sits on it rather than on any logger so that `botocore`'s records get the +same treatment as Sturnus's own. + +**The level knob is the most dangerous thing in this file.** Raising the +root logger to DEBUG in production would dump credentials into Loki, and +this is verified in the installed packages rather than assumed: + +- `discord/ext/voice_recv/reader.py` logs `secret_key=%s` -- the Discord + voice secret key -- and raw packet payload bytes, both at DEBUG. +- `discord/ext/voice_recv/gateway.py` reaches the *same* key by another + route: its `hook()` pretty-prints the whole voice-gateway payload at + DEBUG for every op except 3 and 6, and op 4 is `SESSION_DESCRIPTION`, + which is where `discord.gateway.load_secret_key` reads the key from. + One logger clamped and the other not would have closed the reported + path and left this one open. +- `discord/ext/voice_recv/voice_client.py` pretty-prints the voice state + update at DEBUG, which carries the voice `token` and `session_id`. +- `botocore/auth.py` logs `CanonicalRequest`, `StringToSign` and the SigV4 + signature at DEBUG; `botocore/endpoint.py` logs the prepared request + including the `Authorization` header. + +Two library modules Sturnus now leans on much harder were re-read for the +same class of defect and are clean: `voice_recv/rtp.py`'s `RTPPacket. +__repr__` reports `size=` rather than the bytes, so `voice_recv/buffer.py` +logging a dropped packet at DEBUG is a length and three integers; and +`discord/voice_state.py` -- the logger the leak report named -- formats no +secret into any record. Both are recorded here rather than left implicit, +because "not on the list" and "checked and clean" are indistinguishable +from the outside. + +So `STURNUS_LOG_LEVEL` applies to `logging.getLogger("sturnus")` **only**, +and two independent mechanisms hold the rest down where no environment +variable can reach them: + +- `THIRD_PARTY_FLOOR` is the level below which nothing outside `sturnus.*` + may go, including the root logger everything unnamed inherits from; +- `NEVER_BELOW` clamps specific loggers tighter still. + +That sentence used to be written here as fact while the code contradicted +it. `configure_logging` set the root logger to +`min(resolved_level, resolved_third_party)`, so `STURNUS_LOG_LEVEL=DEBUG` +raised *root* to DEBUG, and every third-party logger absent from +`NEVER_BELOW` -- 24 of them in a running worker -- inherited DEBUG from +there. `discord.ext.voice_recv.gateway` was on the list and stayed quiet; +`discord.http`, which logs whole REST response bodies, was not. The list +was never the problem. Enumerating was. + +The floor is what makes the claim structural rather than aspirational, and +`redaction.PATTERNS`' `secret_value` rule is the second lock behind it, for +the case where a future library release moves one of these lines to a level +the floor permits. Turning Sturnus's own logging up is safe; there is no +configuration that turns theirs up. + +The same clamp fixes the flood that made a real incident's log unreadable: +`voice_recv.reader` logs `"Received packet for unknown ssrc %s"` with the +full RTP packet repr **at INFO**, and `basicConfig(level=INFO)` on the root +logger -- which is what all three entrypoints did before this module -- let +every one of them through. Sturnus counts those packets itself and emits one +rate-limited line carrying the count instead (see `events.RateLimiter`). +""" + +from __future__ import annotations + +import json +import logging +import os +import sys +import threading +import traceback +from types import TracebackType +from typing import Any, Final + +from sturnus import __version__ +from sturnus.observability.events import ( + Event, + current_trace_context, + log_event, + log_exception, +) +from sturnus.observability.redaction import ( + SturnusFilter, + format_exception_safely, + scrub_fields, + scrub_text, +) + +#: The most verbose any logger outside `sturnus.*` may be, whatever the +#: environment says. **The structural half of the fix; `NEVER_BELOW` below +#: is the per-logger half.** +#: +#: `NEVER_BELOW` is an enumeration of names, and an enumeration answers only +#: for the names on it. Every third-party logger *not* on it carries +#: `level == NOTSET` and takes its effective level from its nearest +#: configured ancestor -- in practice the root logger. That is how the +#: reported leak worked: the clamp named `discord.ext.voice_recv.gateway` +#: and the root logger was set to `min(level, third_party)`, so +#: `STURNUS_LOG_LEVEL=DEBUG` put root at DEBUG and every unnamed logger -- +#: `discord.http`, which prints whole REST response bodies, among two dozen +#: others -- inherited it. One list cannot be complete about libraries it +#: does not import. +#: +#: So the level an operator asks for is raised to this floor before +#: anything is configured, and `configure_logging` then sweeps any logger +#: that already carries an explicit level below it. The property that +#: results -- *no logger outside `sturnus.*` sits below `THIRD_PARTY_FLOOR`* +#: -- holds for names nobody enumerated, and +#: `tests/observability/test_third_party_log_floor.py` asserts it over +#: `logging.Logger.manager.loggerDict` rather than over a list. +#: +#: **INFO rather than WARNING, and the difference is a real cost either +#: way.** `discord.voice_state` logs the connect narrative at INFO -- +#: "Starting voice handshake... (connection attempt %d)", "Voice handshake +#: complete", "Timed out connecting to voice", "Disconnected from voice by +#: discord, close code %d" -- and that is the entire evidence base for +#: telling apart the three ways capture fails, which is what +#: `voice.join_failed` / `voice.reader_stopped` / `voice.decode_failed` +#: exist to distinguish. WARNING would delete it. What INFO does cost is +#: real and is named in `docs/operations.md` section 7.2: gateway +#: IDENTIFY/RESUME tracing and `discord.http` rate-limit bucket diagnosis +#: are no longer reachable from a Helm value. They need a deliberate, +#: non-production change to this constant -- which is the point. An +#: environment variable a production `values.yaml` can hold must not be +#: able to publish a session key. +#: +#: (`discord.voice_state` is pinned at exactly INFO -- `NEVER_BELOW` keeps +#: its DEBUG payload out, `NEVER_ABOVE` keeps its INFO narrative in even at +#: the deployed default third-party level of WARNING. Both entries are +#: about the same logger and neither is redundant; see their comments.) +THIRD_PARTY_FLOOR: Final = logging.INFO + +#: Loggers that may never be turned below their listed level, whatever the +#: environment says. Applied *after* the environment's third-party level, so +#: it is a floor and not a default. Tighter than `THIRD_PARTY_FLOOR` by +#: construction -- `max()` of the two is what gets installed -- so this list +#: is now a set of *exceptions* to a floor rather than the floor itself. +#: +#: `discord.ext.voice_recv.router` is on the list at WARNING deliberately: +#: its own fatal line -- `log.exception("Error in %s loop", self)` when the +#: packet-router thread dies -- is at ERROR and therefore survives. That is +#: the one library line that mattered during the incident this design was +#: written against, and silencing it would remove the only evidence. +NEVER_BELOW: Final[dict[str, int]] = { + "botocore": logging.WARNING, + "botocore.auth": logging.WARNING, + "botocore.endpoint": logging.WARNING, + "boto3": logging.WARNING, + "s3transfer": logging.WARNING, + "discord.ext.voice_recv.reader": logging.WARNING, + "discord.ext.voice_recv.router": logging.WARNING, + "discord.ext.voice_recv.opus": logging.WARNING, + # **The second route to the same secret key, and the one the original + # report did not name.** `voice_recv/gateway.py`'s `hook()` logs + # `pformat(data)` at DEBUG for *every* voice-gateway op except 3 and 6 + # -- and op 4 is `SESSION_DESCRIPTION`, whose `d` is where + # `discord.gateway.load_secret_key` reads `data['secret_key']` from. + # Clamping `.reader` alone would have left the key one op code away + # from Loki. Nothing above DEBUG is lost: this module logs one INFO + # line about unexpected WS keys and nothing at WARNING or above. + "discord.ext.voice_recv.gateway": logging.WARNING, + # Same shape, different payload: `voice_client.on_voice_state_update` + # logs `pformat(data)` at DEBUG, and a voice state update carries the + # voice `token` and `session_id`. Its one `log.exception` survives the + # floor, which is the line that would actually matter. + "discord.ext.voice_recv.voice_client": logging.WARNING, + "discord.gateway": logging.WARNING, + "discord.client": logging.WARNING, + # **INFO, not WARNING, and the level is the whole point of the entry.** + # Which level carries what, read at the installed version rather than + # assumed: + # + # DEBUG -- connection-state transitions, DAVE upgrade/downgrade + # notices, socket read errors, "Voice server update, closing old + # voice websocket". No secret is formatted into any of them + # (`secret_key` appears in `voice_state.py` only as an attribute + # that is assigned and awaited), but DEBUG is the level the leak + # lives at everywhere else in this list, and this is the logger the + # leak report named. It stays closed, and it stays listed, because + # "absent from the list" and "checked and clean" are + # indistinguishable afterwards. + # INFO -- the connect narrative: "Starting voice handshake... + # (connection attempt %d)", "Voice handshake complete. Endpoint + # found: %s", "Connecting to voice...", "Voice connection complete", + # "Timed out connecting to voice", "Disconnected from voice by + # discord, close code %d", "Successfully resumed voice connection". + # Close codes, attempt numbers and a voice server hostname; no + # credential, no content. + # + # That INFO narrative is the evidence base for `client.py`'s + # capture-failure cooldown and for telling `voice.join_failed` from + # `voice.reader_stopped` from `voice.decode_failed` -- it says *which* + # stage of the handshake failed, which none of Sturnus's own events + # can. All three entrypoints emitted it before this package existed + # (`basicConfig(level=INFO)`), and clamping it to WARNING deleted it in + # exchange for nothing: the leak is one level lower. + # + # See `NEVER_ABOVE`, which is what makes the INFO half reachable: an + # entry here can only ever make a logger *quieter* than the third-party + # level, so with `THIRD_PARTY_FLOOR` already at INFO this entry changes + # nothing on its own today. It is kept anyway, for two reasons that are + # not redundant with each other: it is the record that this logger's + # DEBUG output was read and judged, and it is what `NEVER_ABOVE`'s pin + # is checked against -- the two must name the same level, and + # `test_debug_for_sturnus_does_not_turn_up_the_credential_loggers` + # fails if they drift apart. Lowering `THIRD_PARTY_FLOOR` in future + # would make this entry load-bearing again without anyone editing it. + "discord.voice_state": logging.INFO, + "httpx": logging.WARNING, + "httpcore": logging.WARNING, + "urllib3": logging.WARNING, + "sqlalchemy.engine": logging.WARNING, + "asyncio": logging.WARNING, + # **A real leak, not a precaution.** aiohttp's access logger formats + # `%r` as `request.path_qs` -- the path *with its query string* -- at + # INFO, and `link`'s only route is + # `/oauth/callback?code=&state=`. With + # the root logger at INFO, which is what all three entrypoints used to + # set, every successful account link wrote an Outline authorization + # code into the pod log and therefore into Loki. The blocking gate in + # `docs/verification/end-to-end-checklist.md` forbids exactly that. + # + # Nothing is lost by silencing it: an access line per request + # duplicates what the ingress already records, and `link`'s own + # `link.callback_rejected` / `link.established` events carry the + # diagnostic content without the credential. + "aiohttp.access": logging.WARNING, + # ERROR, not WARNING, and measured rather than guessed: the OTLP HTTP + # exporter logs each retry of a failed batch at WARNING and only the + # final give-up at ERROR. With Alloy unreachable that is four lines per + # batch per pod, forever -- and the three retry lines say nothing the + # give-up line does not, since retrying is the exporter doing its job. + # Clamping to ERROR keeps one honest line per lost batch. + # + # This deliberately leaves export failure *visible in Loki*. The + # matching `ignore_logger("opentelemetry")` in + # `sturnus.infrastructure.telemetry` only keeps it out of Sentry, where + # it would be an issue per retry per batch and would drown the issue + # list rather than inform it. + "opentelemetry": logging.ERROR, +} + +#: Loggers whose level is *pinned*, so the environment can neither turn them +#: up nor turn them off. The mirror image of `NEVER_BELOW`, and it exists +#: because that dict alone cannot keep a third-party line alive. +#: +#: `NEVER_BELOW` is applied as `max(resolved_third_party, floor)`, so every +#: entry in it can only ever make a logger *quieter* than the third-party +#: level. With `STURNUS_LOG_THIRD_PARTY_LEVEL` at its deployed default of +#: `WARNING` -- see `docs/operations.md` section 7.2 -- an entry of `INFO` +#: there is therefore a no-op: the logger still ends up at `WARNING` and the +#: line the entry was written to keep is still gone. Deciding that a +#: library's INFO output is evidence worth keeping means installing `INFO` +#: outright, which is what this does. +#: +#: Applied **last**, after the sweep below, because the sweep raises any +#: explicit level below `resolved_third_party` back up to it and would +#: otherwise undo this on the very next line. +#: +#: The safety argument is that the pin is a *level*, not an exemption: a +#: pinned logger is held at exactly the level named here, so +#: `STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` cannot make it any louder either. +#: Nothing goes in here without the same reading `NEVER_BELOW`'s entries +#: got -- what does this logger print at this level, in the installed +#: version -- and `tests/observability/test_third_party_log_floor.py` +#: asserts both halves for the one entry there is, on the rendered stream. +NEVER_ABOVE: Final[dict[str, int]] = { + "discord.voice_state": logging.INFO, +} + +#: `LogRecord` attributes the JSON formatter renders itself or deliberately +#: drops. Anything else a call site attached lands in `sturnus_fields` and +#: goes through the registry; nothing reaches the line by accident. +_STANDARD_RECORD_ATTRS: Final = frozenset( + { + "args", + "asctime", + "created", + "exc_info", + "exc_text", + "filename", + "funcName", + "levelname", + "levelno", + "lineno", + "module", + "msecs", + "message", + "msg", + "name", + "pathname", + "process", + "processName", + "relativeCreated", + "stack_info", + "taskName", + "thread", + "threadName", + "sturnus_event", + "sturnus_fields", + } +) + +#: Set once `configure_logging` has run. Read by `migrations/env.py`, which +#: must not call `logging.config.fileConfig` over a configuration this +#: process already owns -- see the comment there. +_configured = False + + +def logging_is_configured() -> bool: + """Whether `configure_logging` has run in this process.""" + return _configured + + +log = logging.getLogger(__name__) + + +class _SafeFormatterBase(logging.Formatter): + """Shared exception rendering. Never calls `traceback.format_exception`. + + That function renders `str(exc)` for every exception in the chain, which + is the one part of an exception that can carry payload. This renders the + frames -- file, line, function, and the *source* line, all static program + text -- and defers each message to + `redaction.safe_exception_message`. + """ + + component: str = "unknown" + + def formatException( # noqa: N802 - overriding logging.Formatter + self, + ei: tuple[type[BaseException], BaseException, TracebackType | None] + | tuple[None, None, None], + ) -> str: + exc = ei[1] + if exc is None: + return "" + frames = [ + f' File "{frame.filename}", line {frame.lineno}, in {frame.name}\n' + f" {(frame.line or '').strip()}" + for frame in traceback.extract_tb(exc.__traceback__) + ] + return "\n".join( + ["Traceback (most recent call last):", *frames, *format_exception_safely(exc)] + ) + + def _payload(self, record: logging.LogRecord) -> dict[str, Any]: + fields = dict(getattr(record, "sturnus_fields", {}) or {}) + # `SturnusFilter` already scrubbed these; scrubbing again is cheap + # and makes the formatter safe on its own, so a handler someone adds + # without the filter still cannot render a raw value. + fields = scrub_fields(fields) + # Anything attached through `extra=` outside `log_event` -- including + # by a third-party library -- is routed through the same registry + # rather than trusted. + stray = { + key: value + for key, value in record.__dict__.items() + if key not in _STANDARD_RECORD_ATTRS and not key.startswith("_") + } + if stray: + # `warn=False`: third-party records attach attributes that were + # never meant for this registry, and warning about each would + # be a flood. They are dropped just as firmly either way. + fields.update(scrub_fields(stray, warn=False)) + fields.update(current_trace_context()) + + payload: dict[str, Any] = { + "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S.%03dZ"), + "level": record.levelname, + "event": getattr(record, "sturnus_event", record.name), + "component": self.component, + "logger": record.name, + "msg": scrub_text(record.getMessage()), + "version": __version__, + } + payload.update(fields) + if record.exc_info: + payload["exc"] = self.formatException(record.exc_info) + return payload + + +class JsonFormatter(_SafeFormatterBase): + """One JSON object per line, on stdout, for `alloy-logs` to hand to Loki. + + `default=str` is a backstop, not a strategy: every value reaching here + has already been through `scrub_value`, which returns only JSON-native + types. If something slips past, `str()` of it is far less dangerous than + the alternative -- an exception inside the formatter, which `logging` + swallows, turning a leak into silence. + """ + + def format(self, record: logging.LogRecord) -> str: + return json.dumps(self._payload(record), default=str, ensure_ascii=False) + + +class ConsoleFormatter(_SafeFormatterBase): + """The same content, laid out for a human at a terminal. + + JSON is unreadable in `kubectl logs` without `jq`, and that is the one + workflow people reach for under pressure. This format is selected + automatically when stdout is a TTY, so local development never sees + JSON, and `STURNUS_LOG_FORMAT` overrides the guess in both directions. + Both formats share `SturnusFilter` and `_payload`, so redaction is + identical -- the choice is presentation only. + """ + + def format(self, record: logging.LogRecord) -> str: + payload = self._payload(record) + head = f"{payload['ts']} {payload['level']:<7} {payload['event']}" + rest = " ".join( + f"{k}={v}" + for k, v in payload.items() + if k not in {"ts", "level", "event", "component", "logger", "msg", "version", "exc"} + ) + line = f"{head} {payload['msg']}" + (f" | {rest}" if rest else "") + if "exc" in payload: + line += "\n" + str(payload["exc"]) + return line + + +def _resolve_level(value: str, *, name: str) -> int: + level = logging.getLevelNamesMapping().get(value.strip().upper()) + if level is None: + raise ValueError(f"{name} must be a logging level name, got {value!r}") + return level + + +def configure_logging( + component: str, + *, + level: str | None = None, + third_party_level: str | None = None, + log_format: str | None = None, + stream: Any = None, +) -> logging.Handler: + """Installs the single handler for this process. Returns it, for tests. + + Reads `STURNUS_LOG_LEVEL`, `STURNUS_LOG_THIRD_PARTY_LEVEL` and + `STURNUS_LOG_FORMAT` directly from the environment rather than through a + settings class, because it runs *before* the settings are constructed -- + a `ValidationError` from `WorkerSettings` has to be formatted by + something, and that something cannot be configured by `WorkerSettings`. + + Sturnus's own tree is set to `level`. Everything else is set to + `third_party_level` **raised to `THIRD_PARTY_FLOOR`** and then clamped + further by `NEVER_BELOW`, and the root logger carries that same + third-party level so a logger nobody enumerated inherits the floor + rather than inheriting Sturnus's verbosity. There is no configuration + that lowers any of it, and an unparseable level raises rather than + being accepted silently. + """ + level_name = level or os.environ.get("STURNUS_LOG_LEVEL", "INFO") + third_party_name = third_party_level or os.environ.get( + "STURNUS_LOG_THIRD_PARTY_LEVEL", "WARNING" + ) + resolved_level = _resolve_level(level_name, name="STURNUS_LOG_LEVEL") + requested_third_party = _resolve_level(third_party_name, name="STURNUS_LOG_THIRD_PARTY_LEVEL") + # Raised, not rejected. Refusing to start on + # `STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` would turn a request for more + # diagnostics into an outage, which is a worse trade than ignoring it + # -- but ignoring it silently sends the operator looking for a broken + # variable, so the clamp announces itself below. + resolved_third_party = max(requested_third_party, THIRD_PARTY_FLOOR) + + target = stream if stream is not None else sys.stdout + fmt = log_format or os.environ.get("STURNUS_LOG_FORMAT") + if fmt is None: + fmt = "console" if getattr(target, "isatty", lambda: False)() else "json" + if fmt not in {"json", "console"}: + raise ValueError(f"STURNUS_LOG_FORMAT must be 'json' or 'console', got {fmt!r}") + + formatter: _SafeFormatterBase = JsonFormatter() if fmt == "json" else ConsoleFormatter() + formatter.component = component + + handler = logging.StreamHandler(target) + handler.setFormatter(formatter) + handler.addFilter(SturnusFilter()) + + root = logging.getLogger() + # Replaced, never appended: a second handler would be a second exit from + # the process, and only this one carries the filter. + root.handlers = [handler] + # **The third-party level, not `min(level, third_party)`.** That `min()` + # was the leak: it made the root logger as verbose as Sturnus's own + # tree, and every third-party logger with no explicit level of its own + # inherits from root. `STURNUS_LOG_LEVEL=DEBUG` therefore turned on + # DEBUG for two dozen libraries nobody had enumerated, including the + # one that pretty-prints the voice `secret_key`. + # + # It bought nothing in exchange, which is what makes this a pure + # deletion rather than a trade: `logging` tests a record against the + # level of the logger it was created on (`Logger.isEnabledFor` -> + # `getEffectiveLevel`), and never against root's level while + # propagating. Root's own level gates only records logged on root + # itself, of which this process emits none. Sturnus's DEBUG output is + # unaffected -- `test_sturnus_debug_survives_the_third_party_floor` + # is the assertion that says so rather than the reasoning above. + root.setLevel(resolved_third_party) + + sturnus_logger = logging.getLogger("sturnus") + sturnus_logger.setLevel(resolved_level) + # `logging.config.fileConfig` -- which Alembic's `env.py` calls -- sets + # `disabled = True` on every logger its ini does not name. A disabled + # logger emits nothing at all, so inheriting that flag would make this + # function appear to succeed while silencing the process. This function + # claims to own logging configuration, so it asserts that rather than + # inheriting whoever ran last. + sturnus_logger.disabled = False + for existing in logging.Logger.manager.loggerDict.values(): + if isinstance(existing, logging.Logger) and existing.name.startswith("sturnus"): + existing.disabled = False + + for name, floor in NEVER_BELOW.items(): + logging.getLogger(name).setLevel(max(resolved_third_party, floor)) + + # Inheriting from root covers every third-party logger that has no + # level of its own -- which is nearly all of them. The exception is a + # library that turns *itself* up at import time: an explicit level + # short-circuits `getEffectiveLevel`, so neither the root level above + # nor `NEVER_BELOW` (whose list cannot name a logger written after it) + # would reach it. Raising it here is what turns "the loggers we thought + # of" into "every logger that exists". + # + # Only ever raises. A library that set itself to ERROR keeps ERROR, + # because quieter than the floor was never the problem. + for existing in list(logging.Logger.manager.loggerDict.values()): + if not isinstance(existing, logging.Logger) or existing.name.startswith("sturnus"): + continue + if existing.level != logging.NOTSET and existing.level < resolved_third_party: + existing.setLevel(resolved_third_party) + + # After the sweep, which would otherwise raise these straight back to + # `resolved_third_party`. `setLevel` outright rather than `min`/`max`: + # the pin is the decision, and both directions of the environment knob + # are what it is pinned against. + for name, pinned in NEVER_ABOVE.items(): + logging.getLogger(name).setLevel(pinned) + + global _configured + _configured = True + + if requested_third_party < resolved_third_party: + # Emitted through the handler that was just installed, so it lands + # in Loki next to the lines the operator is about to go looking + # for. Without it, `STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` produces + # no new output and no explanation, and the next hour goes into + # debugging the deployment rather than reading section 7.2. + log_event( + log, + logging.WARNING, + Event.LOG_LEVEL_CLAMPED, + "Third-party log level raised to the floor; see docs/operations.md section 7.2", + # No `component=`: the formatter puts it on every line already. + reason="third_party_floor", + ) + + return handler + + +def install_excepthooks() -> None: + """Routes every uncaught exception through the configured handler. + + Without this, an exception escaping `asyncio.run` reaches the default + excepthook, which writes an unformatted, unredacted traceback straight + to stderr -- and Alloy scrapes stderr too. That is not hypothetical: + `sturnus.config.StrictSettings._reject_blank_required_values` raises + through pydantic, which embeds the raw input dict in its message, so a + blank required variable -- the single most likely operator mistake -- + puts the first characters of `STURNUS_DISCORD_TOKEN` into the log store. + Routing it here formats it through `_SafeFormatterBase`, whose exception + rendering withholds the message, and `PATTERNS` redacts what pydantic's + own truncation left. + + Covers all three routes an exception can take out of these processes: + the main thread, a `threading.Thread` (the OTLP exporters and the voice + router both run on one), and an asyncio task whose exception is never + retrieved. + """ + + def hook( + exc_type: type[BaseException], + exc: BaseException, + tb: TracebackType | None, + ) -> None: + del exc_type, tb + log_exception(log, logging.ERROR, Event.UNHANDLED_EXCEPTION, "Unhandled exception", exc) + + sys.excepthook = hook + + def thread_hook(args: threading.ExceptHookArgs) -> None: + if args.exc_value is not None: + log_exception( + log, + logging.ERROR, + Event.UNHANDLED_EXCEPTION, + "Unhandled exception in a thread", + args.exc_value, + ) + + threading.excepthook = thread_hook + + +def asyncio_exception_handler(loop: Any, context: dict[str, Any]) -> None: + """`loop.set_exception_handler` target: the asyncio half of the above. + + asyncio's default handler renders `context["message"]` plus the + exception through the root logger with no filtering of the message. This + keeps the same signal -- something failed on the loop and nobody caught + it -- while putting it through the registry. + """ + del loop + exc = context.get("exception") + if isinstance(exc, BaseException): + log_exception( + log, + logging.ERROR, + Event.UNHANDLED_EXCEPTION, + "Unhandled exception on the event loop", + exc, + ) + else: + log.error("Unhandled event-loop error") diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..017b86b --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1,12 @@ +"""Makes `tests` importable as a package. + +`tests/infrastructure/test_traced_ports.py` reuses the fakes in +`tests/application/test_worker.py` deliberately: the property it checks is +that wrapping the ports changes nothing, and the most convincing way to +show that is to run the same fakes and make the same assertions rather than +to maintain a second, subtly different set that could drift into agreeing +with a bug. + +Every sibling directory already has an `__init__.py`; this is the one that +was missing. +""" diff --git a/tests/application/test_worker.py b/tests/application/test_worker.py index 07da1e9..4354e45 100644 --- a/tests/application/test_worker.py +++ b/tests/application/test_worker.py @@ -45,8 +45,12 @@ async def complete(self, job_id: int, transcript: str) -> bool: self.completed.append((job_id, transcript)) return self.last_is_final - async def fail(self, job_id: int, error: str, _max_attempts: int) -> None: + async def fail(self, job_id: int, error: str, _max_attempts: int) -> bool: self.failed.append((job_id, error)) + # Never dead: this fake counts no attempts, and the real + # `JobQueue.fail` is where "out of attempts" is decided and tested + # (`tests/infrastructure/test_queue.py`). + return False class FakeEngine: diff --git a/tests/infrastructure/discord/test_client.py b/tests/infrastructure/discord/test_client.py index cea6616..4fa67d4 100644 --- a/tests/infrastructure/discord/test_client.py +++ b/tests/infrastructure/discord/test_client.py @@ -56,6 +56,7 @@ ) from sturnus.infrastructure.documents.outline_oauth import OutlineOAuth from sturnus.infrastructure.health import ReadinessState +from sturnus.observability.events import Event T0 = datetime(2026, 8, 19, 20, 0, 0, tzinfo=UTC) GUILD_ID, CHANNEL_ID, ROLE_ID = 1, 2, 3 @@ -242,18 +243,33 @@ async def post(self, channel_id: int, text: str) -> None: # noqa: ARG002 class FakeVoiceReceiver: """Satisfies the `VoiceReceiver` port without a real gateway connection.""" - def __init__(self, *, join_fails: bool = False) -> None: + def __init__( + self, + *, + join_fails: bool = False, + leave_fails: bool = False, + join_error: BaseException | None = None, + ) -> None: self.joined: list[int] = [] self.left = 0 - self.join_fails = join_fails + self.join_fails = join_fails or join_error is not None + self.leave_fails = leave_fails + #: The exception `join` raises. Configurable because *which* one it + #: is changes what the failure line can say -- see + #: `infrastructure.discord.voice.voice_close_code`. + self.join_error = join_error async def join(self, channel_id: int) -> None: + if self.join_error is not None: + raise self.join_error if self.join_fails: raise RuntimeError("the gateway said no") self.joined.append(channel_id) async def leave(self) -> None: self.left += 1 + if self.leave_fails: + raise RuntimeError("the gateway hung up") def _role(role_id: int) -> discord.Role: @@ -1157,9 +1173,24 @@ async def test_a_failed_upload_must_not_leave_a_guild_recording_nothing( assert audio.put_calls == 1, "the close really did get as far as the upload" assert sessions.closed == [], "and really did fail before closing the row" - assert [record for record in caplog.records if record.levelno >= logging.ERROR], ( - "a close that lost a recording must be visible, not swallowed" - ) + (failure,) = [record for record in caplog.records if record.levelno >= logging.ERROR] + # Visible *and* queryable. This was a bare `log.exception("Guild %d: + # ...", guild_id)`: the id existed only as characters inside the + # rendered message, so `| json | guild_id="..."` -- the query the whole + # event vocabulary exists for -- returned nothing, and `scrub_event` + # forwards `LogRecord.msg` to Sentry, which made that id the half of + # the line that leaves the pod. + assert getattr(failure, "sturnus_event", None) == str(Event.SESSION_CLOSE_FAILED) + fields = getattr(failure, "sturnus_fields", {}) + assert fields["guild_id"] == GUILD_ID + assert fields["reason"] == "timeout_sweep" + assert fields["error_type"] == "RuntimeError" + # `record.args` is what `log.exception("Guild %d: ...", guild_id)` left + # behind and what `log_exception(..., guild_id=...)` does not: an empty + # `args` is the mechanical statement that nothing was interpolated into + # the message at all. + assert not failure.args, "the id is a %-argument of the message rather than a field" + assert failure.getMessage() == failure.msg # The guild is not wedged: the very next consenting participant records. audio.fail = False @@ -1674,3 +1705,190 @@ async def test_a_pipeline_the_client_builds_can_warn_its_own_channel(tmp_path: P assert [(user_id, at) for _, user_id, at in sessions.silent_audio] == [ (ANNA, T0 + timedelta(seconds=29)) ] + + +# --------------------------------------------------------------------------- +# The operational failures, as events rather than as prose +# --------------------------------------------------------------------------- +# +# Every test below drives one of the `log.exception("... %d ...", guild_id)` +# call sites this file used to carry. Each asserts the same three things, +# because each was wrong in the same three ways: the event name (so Loki +# can group them), the fields (so `| json | guild_id="1"` finds them), and +# an empty `record.args` (so nothing varying is inside the message that +# `infrastructure.observability.scrub_event` forwards to Sentry). + + +#: `caplog` captures every propagated record, not only the logger named in +#: `at_level`, and closing a session that recorded nothing legitimately +#: emits its own ERROR from `sturnus.application.recording`. Filtering by +#: logger is what keeps these assertions about this module. +CLIENT_LOGGER = "sturnus.infrastructure.discord.client" + + +def _only_error(caplog: pytest.LogCaptureFixture) -> logging.LogRecord: + (record,) = [ + entry + for entry in caplog.records + if entry.levelno >= logging.ERROR and entry.name == CLIENT_LOGGER + ] + return record + + +async def test_a_leave_that_fails_is_an_event_carrying_the_guild( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """`_return_to_idle`: the guild must recover, and must say what it lost. + + `leave()` is a gateway call and a failing one must not take `reset()` + down with it -- so the failure is swallowed, and the log line is the + only trace it ever leaves. + """ + clock = FakeClock(T0) + sessions = FakeSessions() + voice = FakeVoiceReceiver(leave_fails=True) + client = _client(clock, config_store=_configured_store()) + anna, occupied = _capture_guild(client, sessions, voice, tmp_path) + + await client.on_voice_state_update(anna, _voice_state(None), _voice_state(occupied)) + # Everyone leaves, and the empty grace period runs out, so the tick + # closes the session and returns the guild to idle through `leave()`. + empty = _voice_channel(CHANNEL_ID, members=[]) + cast(MagicMock, client.get_guild(GUILD_ID)).get_channel.return_value = empty + await client.on_voice_state_update(anna, _voice_state(occupied), _voice_state(None)) + clock.advance(timedelta(seconds=61)) + + with caplog.at_level(logging.ERROR, logger="sturnus.infrastructure.discord.client"): + await client._tick_all(clock.now()) + + failure = _only_error(caplog) + assert getattr(failure, "sturnus_event", None) == str(Event.VOICE_LEFT_FAILED) + fields = getattr(failure, "sturnus_fields", {}) + assert fields["guild_id"] == GUILD_ID + assert fields["channel_id"] == CHANNEL_ID + assert fields["error_type"] == "RuntimeError" + assert not failure.args + # The point of swallowing it: the guild is recordable again. + assert client._guilds[GUILD_ID].service.needs_reset is False + + +async def test_one_guilds_tick_raising_is_reported_and_isolated( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """`_tick_all`'s per-guild `try` -- both halves of what it promises. + + The line it logs claims "every other guild is unaffected", and until + now nothing checked that claim: this ticks two guilds, breaks the + first, and asserts the second still ran. Which guild broke is a field, + because an operator with fifty guilds needs to know *which* one. + """ + clock = FakeClock(T0) + client = _client(clock, config_store=_configured_store()) + _capture_guild(client, FakeSessions(), FakeVoiceReceiver(), tmp_path) + + other_guild_id = GUILD_ID + 41 + ticked: list[int] = [] + + async def tick(guild_id: int, now: datetime) -> None: + del now + ticked.append(guild_id) + if guild_id == GUILD_ID: + raise RuntimeError("the database went away") + + client._guilds[other_guild_id] = client._guilds[GUILD_ID] + client._tick_guild = tick # type: ignore[method-assign] + + with caplog.at_level(logging.ERROR, logger="sturnus.infrastructure.discord.client"): + await client._tick_all(clock.now()) + + assert ticked == [GUILD_ID, other_guild_id], "the second guild still got its tick" + failure = _only_error(caplog) + assert getattr(failure, "sturnus_event", None) == str(Event.GUILD_TICK_FAILED) + assert getattr(failure, "sturnus_fields", {})["guild_id"] == GUILD_ID + assert not failure.args + + +async def test_the_rejoin_guard_says_which_channel_and_for_how_long( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """`_begin_capture_cooldown`, reached the way production reaches it. + + A join that fails ends the session with `capture_failure`, and leaving + the channel is itself a voice-state update -- so without the guard the + next event walks straight back into the same fault. The line announcing + the guard is an ERROR because somebody has to investigate before it + lifts, which makes "which channel" and "how long" the two things it has + to carry. + + `duration_seconds` rather than the absolute moment the guard lifts: + every line already has a `ts`, and the registry deliberately has no + field for a timestamp. + """ + clock = FakeClock(T0) + sessions = FakeSessions() + client = _client(clock, config_store=_configured_store()) + anna, occupied = _capture_guild(client, sessions, FakeVoiceReceiver(join_fails=True), tmp_path) + + await client.on_voice_state_update(anna, _voice_state(None), _voice_state(occupied)) + clock.advance(timedelta(seconds=1)) + + with caplog.at_level(logging.ERROR, logger="sturnus.infrastructure.discord.client"): + await client._tick_all(clock.now()) + + blocked = [ + entry + for entry in caplog.records + if getattr(entry, "sturnus_event", None) == str(Event.VOICE_REJOIN_BLOCKED) + ] + assert len(blocked) == 1 + fields = getattr(blocked[0], "sturnus_fields", {}) + assert fields["guild_id"] == GUILD_ID + assert fields["channel_id"] == CHANNEL_ID + assert fields["end_reason"] == EndReason.CAPTURE_FAILURE.value + assert fields["duration_seconds"] == REJOIN_COOLDOWN.total_seconds() + assert not blocked[0].args + + +async def test_a_join_refused_by_discord_reports_the_close_code( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """`voice.join_failed` where `error_type` alone is not an answer. + + `discord.ConnectionClosed` is raised for "session no longer valid", + "you were moved", "rate limited" and "voice server crashed" alike, and + `redaction.SAFE_MESSAGE_TYPES` withholds its message -- deliberately, + since `sturnus.observability` may not import `discord` to vouch for a + type from it. So the close code is lifted out as a field, which is both + safe and more queryable than the sentence it came from. + """ + clock = FakeClock(T0) + refused = discord.ConnectionClosed(MagicMock(), shard_id=None, code=4006) + client = _client(clock, config_store=_configured_store()) + anna, occupied = _capture_guild( + client, FakeSessions(), FakeVoiceReceiver(join_error=refused), tmp_path + ) + + with caplog.at_level(logging.ERROR, logger=CLIENT_LOGGER): + await client.on_voice_state_update(anna, _voice_state(None), _voice_state(occupied)) + + failure = _only_error(caplog) + assert getattr(failure, "sturnus_event", None) == str(Event.VOICE_JOIN_FAILED) + fields = getattr(failure, "sturnus_fields", {}) + assert fields["close_code"] == 4006 + assert fields["error_type"] == "ConnectionClosed" + + +async def test_a_join_that_failed_for_another_reason_reports_no_close_code( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """The control. `None`, never a stand-in value that looks like a code.""" + clock = FakeClock(T0) + client = _client(clock, config_store=_configured_store()) + anna, occupied = _capture_guild( + client, FakeSessions(), FakeVoiceReceiver(join_fails=True), tmp_path + ) + + with caplog.at_level(logging.ERROR, logger=CLIENT_LOGGER): + await client.on_voice_state_update(anna, _voice_state(None), _voice_state(occupied)) + + assert getattr(_only_error(caplog), "sturnus_fields", {})["close_code"] is None diff --git a/tests/infrastructure/discord/test_voice_adapter.py b/tests/infrastructure/discord/test_voice_adapter.py index 4dbd043..ad4cd2d 100644 --- a/tests/infrastructure/discord/test_voice_adapter.py +++ b/tests/infrastructure/discord/test_voice_adapter.py @@ -39,6 +39,7 @@ SpeakerStreamEnded, ) from sturnus.infrastructure.discord.voice import VoiceReceiveAdapter +from sturnus.observability.redaction import safe_exception_message T0 = datetime(2026, 8, 19, 20, 0, 0, tzinfo=UTC) GUILD_ID, CHANNEL_ID, ROLE_ID = 1, 2, 3 @@ -199,7 +200,19 @@ async def test_capture_stopping_on_its_own_is_logged_at_error( await voice._handle(CaptureStopped(RuntimeError("router died"))) assert len(caplog.records) == 1 - assert "RuntimeError" in caplog.records[0].getMessage() + # The exception's type is now a structured field rather than part of the + # message. `tests/test_logging_discipline.py` rule R6 forbids + # interpolating an exception into a log message at all -- `str(exc)` is + # unbounded third-party text and `logentry.message` is what + # `infrastructure.observability.scrub_event` forwards to Sentry -- so + # `log_exception` puts the class name in `error_type` instead. The + # assertion is unchanged in substance: an operator must still be able to + # see *what* stopped capture, and `error_type` is now where they see it. + # `getattr`, because `sturnus_fields` is an `extra=` key rather than a + # declared `LogRecord` attribute -- which is exactly what makes it a + # field the registry governs rather than part of the message. + fields = getattr(caplog.records[0], "sturnus_fields", {}) + assert fields["error_type"] == "RuntimeError" assert caplog.records[0].exc_info is not None, "the cause is carried, not summarised away" @@ -312,13 +325,25 @@ async def test_the_drain_survives_a_handler_that_raises( ) connected(voice) - with caplog.at_level(logging.ERROR, logger=VOICE_LOGGER): + # Captured at WARNING so that a *downgrade* is visible here rather than + # silently reducing the record count to zero: the level is asserted + # below, against a literal, instead of being implied by the capture + # threshold. + with caplog.at_level(logging.WARNING, logger=VOICE_LOGGER): voice._emit(frame()) voice._emit(frame()) await settle() service.voice_packet.assert_awaited_once() assert len(caplog.records) == 1, "the swallowed failure is still reported" + # ERROR, and pinned. An exception escaping the message handler is a + # defect in this adapter, not a condition it expects to self-heal from: + # the frame it was carrying is gone for good, and nothing retries it. + # Rate limiting is what keeps a systematic failure from flooding Loki + # (`_MESSAGE_ERROR_LOG_EVERY`); rate and severity are separate + # decisions, and lowering the severity to buy quiet costs the one + # signal that says a human should look. + assert caplog.records[0].levelno == logging.ERROR async def test_emit_before_join_is_a_no_op_rather_than_a_crash() -> None: @@ -405,3 +430,82 @@ async def test_leave_stops_the_drain() -> None: assert drain_task is not None and drain_task.cancelled() assert voice._queue is None + + +# --------------------------------------------------------------------------- +# `discord.ConnectionClosed`: the exception whose message is withheld +# --------------------------------------------------------------------------- + + +def _connection_closed(code: int) -> discord.ConnectionClosed: + """The exception discord.py raises, built the way discord.py builds it. + + `ConnectionClosed.__init__(socket, *, shard_id, code)` reads + `socket.close_code` only when `code` is falsy, so a stand-in socket is + never touched here -- the code under test is `exc.code`, and that comes + from the keyword argument. + """ + return discord.ConnectionClosed(MagicMock(), shard_id=None, code=code) + + +def test_the_message_of_a_connection_closed_really_is_withheld() -> None: + """The premise of the field, asserted rather than assumed. + + If `SAFE_MESSAGE_TYPES` ever grew to admit this type, `close_code` + would be duplicating what the message already says and this test is + what would notice. Until then the class name is *all* an operator gets + from the exception, and "ConnectionClosed" is the least informative + true statement available about a bot that cannot hear a channel. + """ + withheld = safe_exception_message(_connection_closed(4014)) + assert "4014" not in withheld + assert withheld == "" + # The control: the code really is on the exception, so a test asserting + # it reaches the log line is asserting something that could be there. + assert _connection_closed(4014).code == 4014 + + +@pytest.mark.parametrize( + ("code", "what_it_means"), + [ + (4006, "the voice session is no longer valid"), + (4009, "the voice session timed out"), + (4014, "Discord disconnected us -- moved, or the channel was deleted"), + (4015, "the voice server crashed"), + ], +) +async def test_capture_dropped_from_voice_reports_the_close_code( + code: int, what_it_means: str, caplog: pytest.LogCaptureFixture +) -> None: + """Four different faults that `error_type` cannot tell apart. + + Parametrised over the codes rather than asserted once, because the + value has to *travel* -- a call site that hard-coded one, or that + passed the exception's own type instead, would pass a single-case + test. + """ + voice = adapter() + + with caplog.at_level(logging.ERROR, logger=VOICE_LOGGER): + await voice._handle(CaptureStopped(_connection_closed(code))) + + fields = getattr(caplog.records[0], "sturnus_fields", {}) + assert fields["close_code"] == code, what_it_means + assert fields["error_type"] == "ConnectionClosed" + + +async def test_a_stop_with_no_close_code_reports_none_rather_than_inventing_one( + caplog: pytest.LogCaptureFixture, +) -> None: + """`OpusNotLoaded`, `OSError` and `TimeoutError` reach the same line. + + `None` is the honest answer for them. A default of `-1` or `0` would + be indistinguishable in Loki from a close code that really was + reported. + """ + voice = adapter() + + with caplog.at_level(logging.ERROR, logger=VOICE_LOGGER): + await voice._handle(CaptureStopped(OpusNotLoaded())) + + assert getattr(caplog.records[0], "sturnus_fields", {})["close_code"] is None diff --git a/tests/infrastructure/test_health.py b/tests/infrastructure/test_health.py index ef2a522..695a149 100644 --- a/tests/infrastructure/test_health.py +++ b/tests/infrastructure/test_health.py @@ -46,7 +46,20 @@ async def test_version_reports_the_installed_package_version() -> None: assert "version" in body -async def test_metrics_is_reachable() -> None: +async def test_metrics_answers_501_because_metrics_are_pushed() -> None: + """The route exists and truthfully says there is nothing to scrape. + + It used to return `200` with an empty body, on the reasoning that an + empty exposition is still valid Prometheus. That reasoning inverts the + signal: a scrape of an empty `200` is indistinguishable from "every + counter is legitimately zero", so an uninstrumented process would look + perfectly healthy to a ServiceMonitor. A `501` marks the target down, + which is the true statement, and the route still exists so Spec 4.1's + endpoint list stays satisfied. + """ async with TestClient(TestServer(health_app(ReadinessState()))) as client: response = await client.get("/metrics") - assert response.status == 200 + assert response.status == 501 + # Names the variable that turns metrics on, so the 501 is + # self-documenting to whoever pointed a scraper here. + assert "STURNUS_OTEL_EXPORTER_OTLP_ENDPOINT" in await response.text() diff --git a/tests/infrastructure/test_migrations.py b/tests/infrastructure/test_migrations.py index dba05d4..e74b184 100644 --- a/tests/infrastructure/test_migrations.py +++ b/tests/infrastructure/test_migrations.py @@ -6,6 +6,7 @@ import logging +import pytest from alembic import command from alembic.autogenerate import compare_metadata from alembic.config import Config @@ -13,6 +14,7 @@ from sqlalchemy import create_engine, text from sturnus.infrastructure.db.models import Base +from sturnus.observability import setup EXPECTED_TABLES = { "guild_config", @@ -72,6 +74,7 @@ def test_models_and_migration_do_not_drift(clean_database: str) -> None: def test_running_migrations_does_not_silence_the_application_loggers( clean_database: str, + monkeypatch: pytest.MonkeyPatch, ) -> None: """The worker migrates in-process, and `fileConfig` is a global. @@ -83,7 +86,18 @@ def test_running_migrations_does_not_silence_the_application_loggers( its life, and reports nothing to Sentry either, since `LoggingIntegration` hooks `Logger.callHandlers` and `Logger.handle` never gets there for a disabled logger. + + `env.py` now reaches `fileConfig` only when nothing else owns logging + yet, so `_configured` is forced back to False for the duration: any test + that ran `configure_logging()` earlier in the session would otherwise + close the guard, skip `fileConfig` altogether, and leave this test + passing without ever exercising the call it exists to constrain. The + guard is the other half of the same regression -- see + `tests/test_logging_discipline.py` -- and neither half may be tested by + accidentally disabling the other. """ + monkeypatch.setattr(setup, "_configured", False) + application_logger = logging.getLogger("sturnus.infrastructure.observability") assert application_logger.disabled is False diff --git a/tests/infrastructure/test_queue.py b/tests/infrastructure/test_queue.py index 38298ae..1cb6167 100644 --- a/tests/infrastructure/test_queue.py +++ b/tests/infrastructure/test_queue.py @@ -1,5 +1,6 @@ import asyncio from datetime import UTC, datetime, timedelta +from typing import Any import pytest from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine @@ -7,6 +8,7 @@ from sturnus.infrastructure.db.models import Base from sturnus.infrastructure.db.queue import JobQueue from sturnus.infrastructure.db.repositories import JobRepository, SessionRepository +from sturnus.infrastructure.telemetry import JOB_OUTCOME, record T0 = datetime(2026, 8, 19, 20, 0, 0, tzinfo=UTC) GUILD, CHANNEL, ANNA, BEN = 1, 2, 100, 200 @@ -265,3 +267,121 @@ async def test_a_reclaimed_job_gets_a_fresh_lease( assert reclaimed is not None assert await queue.claim(T0 + timedelta(seconds=90)) is None + + +# --------------------------------------------------------------------------- +# `sturnus.job.outcome`: what actually happened, not what was returned +# --------------------------------------------------------------------------- + + +@pytest.fixture +def outcomes(monkeypatch: pytest.MonkeyPatch) -> list[dict[str, Any]]: + """Every `record(...)` this module makes, captured on its way through. + + The real `record` is still called -- this observes the call, it does + not replace the recorder -- so the production path, including + `_metric_attributes`' allowlist, runs exactly as it does in the worker + and a label that would be dropped there is dropped here too. + + A spy rather than a metric reader because installing a `MeterProvider` + is a process-global, once-only operation: it would bind every + module-level instrument in `sturnus.infrastructure.telemetry` for the + rest of the session and silently invalidate + `test_spans_and_metrics_are_no_ops_with_no_provider`, which asserts the + opposite premise. + """ + seen: list[dict[str, Any]] = [] + + def spy(instrument: Any, value: float, **fields: object) -> None: + seen.append({"instrument": instrument, "value": value, **fields}) + record(instrument, value, **fields) + + # Patched by dotted path rather than through the module object: `record` + # is not re-exported from `sturnus.infrastructure.db.queue`, and mypy's + # `no_implicit_reexport` is right to say so. + monkeypatch.setattr("sturnus.infrastructure.db.queue.record", spy) + return seen + + +def _job_outcomes(seen: list[dict[str, Any]]) -> list[str]: + return [ + str(call["outcome"]) + for call in seen + if call["instrument"] is JOB_OUTCOME and call["value"] == 1 + ] + + +async def test_a_completed_job_is_counted_as_done( + factory: async_sessionmaker[AsyncSession], outcomes: list[dict[str, Any]] +) -> None: + """The only path that may produce `done`, and it is the one that stored a + transcript.""" + session_id = await seed(factory, [ANNA]) + queue = JobQueue(factory) + job = await queue.claim() + assert job is not None + + await queue.complete(job.id, "the transcript") + + assert _job_outcomes(outcomes) == ["done"] + assert session_id # the seed really produced a job to complete + + +async def test_a_job_returned_for_another_attempt_is_counted_as_failed( + factory: async_sessionmaker[AsyncSession], outcomes: list[dict[str, Any]] +) -> None: + """The defect this test exists for. + + `process_one` returns `True` after `queue.fail(...)` just as it does + after `queue.complete(...)` -- the boolean means "work was attempted", + not "work succeeded" -- and the worker loop used to turn that boolean + into `outcome="done"`. Every failed job was therefore counted as a + success, which is worse than not counting at all: an operator would + believe it. + """ + await seed(factory, [ANNA]) + queue = JobQueue(factory) + job = await queue.claim() + assert job is not None + + assert await queue.fail(job.id, "s3 timed out", 3) is False + + assert _job_outcomes(outcomes) == ["failed"] + + +async def test_a_job_out_of_attempts_is_counted_as_dead_and_says_so( + factory: async_sessionmaker[AsyncSession], outcomes: list[dict[str, Any]] +) -> None: + """`dead` is permanent loss and must be distinguishable from a retry. + + The return value is what lets the caller -- and the `job.process` span + -- tell the two apart at all: `fail` is the only place that knows, + because it is the only place that counts the attempts. + """ + await seed(factory, [ANNA]) + queue = JobQueue(factory) + job = await queue.claim() + assert job is not None + + assert await queue.fail(job.id, "still broken", 1) is True + + assert _job_outcomes(outcomes) == ["dead"] + + +async def test_the_outcome_counter_never_reports_a_failure_as_a_success( + factory: async_sessionmaker[AsyncSession], outcomes: list[dict[str, Any]] +) -> None: + """The property, over one job's whole life: two retries, then death. + + Written as the sequence rather than as three separate assertions + because the failure the counter had was precisely a *substitution* -- + the right number of measurements with the wrong label on them. + """ + await seed(factory, [ANNA]) + queue = JobQueue(factory) + for _ in range(3): + job = await queue.claim() + assert job is not None + await queue.fail(job.id, "broken", 3) + + assert _job_outcomes(outcomes) == ["failed", "failed", "dead"] diff --git a/tests/infrastructure/test_telemetry.py b/tests/infrastructure/test_telemetry.py new file mode 100644 index 0000000..93abb55 --- /dev/null +++ b/tests/infrastructure/test_telemetry.py @@ -0,0 +1,339 @@ +"""The trace-side privacy control, proven end to end rather than asserted. + +The headline test is `test_a_leaky_span_still_exports_nothing_sensitive`: it +drives a span with OpenTelemetry's *own* leaky defaults, a forbidden +attribute and a transcript-bearing exception through the real allowlisting +exporter, and asserts the exported payload is clean. That is the difference +between a control and a comment. +""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + +import pytest +from opentelemetry import trace +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import SimpleSpanProcessor +from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter +from opentelemetry.trace import NonRecordingSpan, StatusCode + +from sturnus.config import OtelSettings +from sturnus.infrastructure import telemetry +from sturnus.infrastructure.telemetry import ( + AllowlistingSpanExporter, + fail_span, + init_telemetry, + set_span_fields, + span, + span_attributes, +) +from sturnus.observability import events +from sturnus.observability.fields import SAFE_SPAN_ATTRIBUTES, SEMCONV_SPAN_ATTRIBUTES + +TRANSCRIPT = "SECRET-TRANSCRIPT-what-they-actually-said-in-the-meeting" + + +@pytest.fixture +def exported() -> Iterator[InMemorySpanExporter]: + """A provider whose spans pass through the real allowlisting exporter.""" + sink = InMemorySpanExporter() + provider = TracerProvider() + provider.add_span_processor(SimpleSpanProcessor(AllowlistingSpanExporter(sink))) + saved = trace.get_tracer_provider() + trace._TRACER_PROVIDER = provider # noqa: SLF001 - no public reset exists + yield sink + trace._TRACER_PROVIDER = saved # noqa: SLF001 + provider.shutdown() + + +# --------------------------------------------------------------------------- +# Non-negotiable: works with no collector +# --------------------------------------------------------------------------- + + +def test_no_endpoint_installs_nothing_at_all() -> None: + """The whole "works without Alloy" guarantee, in one assertion. + + Not `is_active()`-style introspection: the claim is that no provider is + constructed, so nothing connects, nothing retries, and nothing logs an + export failure. + """ + settings = OtelSettings(otel_exporter_otlp_endpoint=None) + assert init_telemetry("worker", settings) is False + assert telemetry._tracer_provider is None # noqa: SLF001 + assert telemetry._meter_provider is None # noqa: SLF001 + assert events.current_trace_context() == {} + + +@pytest.mark.parametrize("blank", ["", " ", "\t\n"]) +def test_a_blank_endpoint_is_absent_not_an_endpoint(blank: str) -> None: + """The chart's default for an unconfigured cluster is `""`, not unset. + + `StrictSettings._reject_blank_required_values` does not apply to an + optional field, so without this validator `""` would reach the exporter + as a real endpoint and every export would fail forever. + """ + settings = OtelSettings(otel_exporter_otlp_endpoint=blank) + assert settings.otel_exporter_otlp_endpoint is None + assert init_telemetry("bot", settings) is False + + +def test_spans_and_metrics_are_no_ops_with_no_provider() -> None: + """Every instrumentation call in the codebase degrades to nothing.""" + telemetry.shutdown_telemetry() + with span("job.process", job_id=1) as active: + assert isinstance(active, NonRecordingSpan) + # A counter with no provider must not raise, and must not require a + # conditional at the call site. + telemetry.record(telemetry.VOICE_PACKETS, 1, outcome="recorded", guild_id=42) + telemetry.record(telemetry.JOB_STAGE_DURATION, 1.5, stage="decrypt", outcome="ok") + + +def test_the_whole_worker_pipeline_runs_with_no_collector() -> None: + """The traced wrappers are inert too, not merely the raw API.""" + from sturnus.infrastructure.traced import TracedQueue + + class _Queue: + """A whole `Queue`, not just `claim` -- mypy compares the wrapper + against the full protocol at the call site.""" + + async def claim(self) -> object | None: + return None + + async def complete(self, job_id: int, transcript: str) -> bool: + del job_id, transcript + raise AssertionError("not reached: the queue is empty") + + async def fail(self, job_id: int, error: str, max_attempts: int) -> bool: + del job_id, error, max_attempts + raise AssertionError("not reached: the queue is empty") + + import asyncio + + assert asyncio.run(TracedQueue(_Queue()).claim()) is None + + +def test_a_ratio_outside_zero_to_one_is_refused() -> None: + with pytest.raises(ValueError, match="TRACES_SAMPLE_RATIO"): + OtelSettings(otel_traces_sample_ratio=1.5) + + +def test_the_environment_is_shared_with_sentry(monkeypatch: pytest.MonkeyPatch) -> None: + """One environment string, so Tempo and Sentry can never disagree.""" + from sturnus.config import SentrySettings + + monkeypatch.setenv("STURNUS_SENTRY_ENVIRONMENT", "staging") + assert OtelSettings().environment == "staging" + assert SentrySettings().sentry_environment == "staging" + + +# --------------------------------------------------------------------------- +# Non-negotiable: nothing sensitive reaches a span +# --------------------------------------------------------------------------- + + +def test_a_leaky_span_still_exports_nothing_sensitive( + exported: InMemorySpanExporter, +) -> None: + """The control, demonstrated against deliberately wrong usage. + + Everything here is what a careless call site would do: OpenTelemetry's + own default exception flags, an unregistered attribute holding a display + name, and a transcript-bearing exception escaping the span. With + `record_exception=True` the SDK writes the message into + `exception.message`, the full `exception.stacktrace` **and** + `status.description` -- three separate paths, verified against + opentelemetry-sdk 1.44.0. The exporter must neutralise all of them. + """ + tracer = trace.get_tracer("test") + with pytest.raises(RuntimeError), tracer.start_as_current_span("job.process") as active: + active.set_attribute("sturnus.job_id", 7) + active.set_attribute("sturnus.speaker.display_name", "Alice Example") + active.set_attribute("sturnus.transcript_text", TRANSCRIPT) + raise RuntimeError(TRANSCRIPT) + + (out,) = exported.get_finished_spans() + assert dict(out.attributes or {}) == {"sturnus.job_id": 7} + assert out.events == () + assert out.status.status_code is StatusCode.ERROR + assert out.status.description is None + + serialised = out.to_json() + assert TRANSCRIPT not in serialised + assert "Alice" not in serialised + + +def test_span_opened_through_the_helper_never_records_the_exception( + exported: InMemorySpanExporter, +) -> None: + """The first of the two independent locks: both flags off at the source.""" + with pytest.raises(RuntimeError), span("job.transcribe", job_id=7): + raise RuntimeError(TRANSCRIPT) + + (out,) = exported.get_finished_spans() + assert out.events == () + assert out.status.status_code is StatusCode.ERROR + assert out.status.description is None + assert out.attributes is not None + assert out.attributes["error.type"] == "RuntimeError" + assert TRANSCRIPT not in out.to_json() + + +def test_fail_span_records_a_class_name_never_a_message( + exported: InMemorySpanExporter, +) -> None: + tracer = trace.get_tracer("test") + with tracer.start_as_current_span("x") as active: + fail_span(active, ValueError(TRANSCRIPT)) + (out,) = exported.get_finished_spans() + assert out.attributes is not None + assert out.attributes["error.type"] == "ValueError" + assert TRANSCRIPT not in out.to_json() + + +def test_an_unregistered_field_never_becomes_an_attribute() -> None: + """Dropped at the source as well as at the exporter -- two locks, not one.""" + attributes = span_attributes({"job_id": 7, "transcript": TRANSCRIPT, "display_name": "Alice"}) + assert attributes == {"sturnus.job_id": 7} + + +def test_log_only_fields_are_refused_as_span_attributes() -> None: + """A user id is fine in `kubectl logs` and deliberately not in Tempo.""" + assert span_attributes({"discord_user_id": 12345, "job_id": 7}) == {"sturnus.job_id": 7} + + +def test_audio_bytes_offered_as_an_attribute_render_as_a_length( + exported: InMemorySpanExporter, +) -> None: + with span("recording.upload") as active: + set_span_fields(active, bytes=b"\x00\x01" * 1024) + (out,) = exported.get_finished_spans() + assert out.attributes is not None + assert out.attributes["sturnus.bytes"] == "" + + +def test_metric_attributes_are_narrower_than_span_attributes() -> None: + """Cardinality: a session id would be a new time series per session.""" + attributes = telemetry._metric_attributes( # noqa: SLF001 + {"outcome": "done", "session_id": 4711, "job_id": 7, "guild_id": 42} + ) + assert attributes == {"outcome": "done", "guild_id": 42} + + +# --------------------------------------------------------------------------- +# The literals in the stdlib-only registry match the real semantic conventions +# --------------------------------------------------------------------------- + + +def test_semconv_literals_match_the_installed_conventions() -> None: + """`sturnus.observability.fields` is stdlib-only, so it spells these by hand. + + This is what stops the hand-written literal drifting from the convention + it is claiming to follow. + """ + from opentelemetry.semconv.attributes import ( + error_attributes, + http_attributes, + server_attributes, + url_attributes, + ) + + assert SEMCONV_SPAN_ATTRIBUTES["error_type"] == error_attributes.ERROR_TYPE + assert SEMCONV_SPAN_ATTRIBUTES["http_method"] == http_attributes.HTTP_REQUEST_METHOD + assert SEMCONV_SPAN_ATTRIBUTES["http_status"] == http_attributes.HTTP_RESPONSE_STATUS_CODE + assert SEMCONV_SPAN_ATTRIBUTES["server_address"] == server_attributes.SERVER_ADDRESS + assert SEMCONV_SPAN_ATTRIBUTES["url_path"] == url_attributes.URL_PATH + + +def test_deployment_environment_literal_matches_the_incubating_convention() -> None: + from opentelemetry.semconv._incubating.attributes import deployment_attributes + + assert deployment_attributes.DEPLOYMENT_ENVIRONMENT_NAME == "deployment.environment.name" + + +def test_every_semconv_attribute_is_in_the_span_allowlist() -> None: + for name in SEMCONV_SPAN_ATTRIBUTES.values(): + assert name in SAFE_SPAN_ATTRIBUTES + + +# --------------------------------------------------------------------------- +# Wiring +# --------------------------------------------------------------------------- + + +def test_init_telemetry_installs_providers_and_the_trace_context_hook() -> None: + """With an endpoint, the Loki -> Tempo correlation field starts working.""" + settings = OtelSettings( + otel_exporter_otlp_endpoint="http://alloy-receiver.grafana.svc:4318", + otel_metric_export_interval_seconds=3600.0, + ) + try: + assert init_telemetry("worker", settings) is True + assert telemetry._tracer_provider is not None # noqa: SLF001 + assert telemetry._meter_provider is not None # noqa: SLF001 + + with span("job.process", job_id=1): + context = events.current_trace_context() + assert set(context) == {"trace_id", "span_id"} + assert len(context["trace_id"]) == 32 + assert int(context["trace_id"], 16) != 0 + finally: + telemetry.shutdown_telemetry() + + # Torn down cleanly: no stale hook left pointing at a dead provider. + assert events.current_trace_context() == {} + + +def test_no_auto_instrumentation_package_is_installed() -> None: + """Each one would ship a transcript, a user id or a credential. + + An install-time assertion because that is where the decision is made: + `pyproject.toml` names four OpenTelemetry packages and none of them + instruments anything. See the module docstring for what each + instrumentor would attach. + """ + import importlib.metadata + + installed = {dist.metadata["Name"] or "" for dist in importlib.metadata.distributions()} + offenders = {name for name in installed if name.startswith("opentelemetry-instrumentation")} + assert not offenders, f"auto-instrumentation must not be installed: {offenders}" + + +def test_the_otlp_export_logger_is_kept_out_of_sentry() -> None: + """A verified cross-branch defect: every failed export would be an issue. + + The SDK logs export failures at ERROR and `init_sentry` configures + `LoggingIntegration(event_level=logging.ERROR)`, so an unreachable Alloy + would produce a Sentry event per retry per batch from all three pods. + """ + import sentry_sdk.integrations.logging as sentry_logging + + telemetry._silence_sentry_export_storm() # noqa: SLF001 + assert "opentelemetry" in sentry_logging._IGNORED_LOGGERS # noqa: SLF001 + + +def test_histogram_views_cover_every_histogram() -> None: + """The SDK's default buckets top out at 10000 in milliseconds; ours are seconds.""" + names = { + view._instrument_name # noqa: SLF001 + for view in telemetry._views() # noqa: SLF001 + } + assert "sturnus.job.stage.duration" in names + assert "sturnus.transcription.audio_duration" in names + assert "sturnus.session.close.duration" in names + assert "sturnus.recording.upload.bytes" in names + + +def _all_span_attribute_names(module: Any) -> set[str]: + del module + return set(SAFE_SPAN_ATTRIBUTES) + + +def test_the_allowlist_contains_no_forbidden_name() -> None: + """A last, blunt check on the derived set itself.""" + forbidden = ("transcript", "display_name", "s3_key", "token", "secret", "key_material") + for name in SAFE_SPAN_ATTRIBUTES: + for bad in forbidden: + assert bad not in name, name diff --git a/tests/infrastructure/test_traced_ports.py b/tests/infrastructure/test_traced_ports.py new file mode 100644 index 0000000..1bff2d6 --- /dev/null +++ b/tests/infrastructure/test_traced_ports.py @@ -0,0 +1,240 @@ +"""The traced wrappers, driven through the *real* `process_one`. + +Two properties are being checked, and the first matters more than the +second. + +**Behaviour is unchanged.** `Queue.claim` returns `object | None` and +`process_one` immediately `cast`s it to `_ClaimedJobShape`; a wrapper that +returned anything but the original object would break the pipeline silently +and no type checker would notice, because the declared type is `object`. +So the wrappers are exercised against the same fakes +`tests/application/test_worker.py` uses, and the same assertions are made. + +**The expected span tree appears**, with no forbidden attribute in it -- run +against a transcript-shaped canary so a leak fails the test rather than +being reasoned about. +""" + +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path + +import pytest +from opentelemetry import trace +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import SimpleSpanProcessor +from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter + +from sturnus.application.worker import process_one +from sturnus.infrastructure.telemetry import AllowlistingSpanExporter, span +from sturnus.infrastructure.traced import ( + TracedAudioDownloader, + TracedDecryptor, + TracedDocumentSink, + TracedQueue, + TracedTranscriptionEngine, +) +from tests.application.test_worker import ( + FakeConfig, + FakeCrypto, + FakeDocuments, + FakeEngine, + FakeJobs, + FakeLinks, + FakeQueue, + FakeSessions, + FakeStore, + job, +) + +#: A canary that must never appear in any exported span. It is what the +#: engine "transcribes", so it flows through `complete`, the assembled +#: document body, and the document title. +CANARY = "CANARY-they-discussed-the-acquisition-in-confidence" + + +@pytest.fixture +def exported() -> Iterator[InMemorySpanExporter]: + sink = InMemorySpanExporter() + provider = TracerProvider() + provider.add_span_processor(SimpleSpanProcessor(AllowlistingSpanExporter(sink))) + saved = trace.get_tracer_provider() + trace._TRACER_PROVIDER = provider # noqa: SLF001 - no public reset exists + yield sink + trace._TRACER_PROVIDER = saved # noqa: SLF001 + provider.shutdown() + + +async def _run_one(tmp_path: Path, *, is_last: bool = True) -> tuple[FakeQueue, FakeDocuments]: + queue = FakeQueue([job()]) + queue.last_is_final = is_last + documents = FakeDocuments() + sessions = FakeSessions() + with span("job.process"): + await process_one( + queue=TracedQueue(queue), + engine=TracedTranscriptionEngine(FakeEngine(CANARY)), + store=TracedAudioDownloader(FakeStore()), + crypto=TracedDecryptor(FakeCrypto()), + documents=TracedDocumentSink(documents), + sessions=sessions, + jobs=FakeJobs(), + links=FakeLinks(), + config=FakeConfig(), + work_dir=tmp_path, + max_attempts=3, + ) + return queue, documents + + +async def test_wrapping_does_not_change_what_process_one_does(tmp_path: Path) -> None: + """The pass-through property: same completions, same document, same body.""" + queue, documents = await _run_one(tmp_path) + + assert len(queue.completed) == 1 + job_id, transcript = queue.completed[0] + assert job_id == 1 + # The transcript reached the queue untouched -- the wrapper observes, + # it does not transform. + assert CANARY in transcript + assert queue.failed == [] + assert len(documents.created) == 1 + + +async def test_the_expected_span_tree_appears( + tmp_path: Path, exported: InMemorySpanExporter +) -> None: + await _run_one(tmp_path) + + names = [s.name for s in exported.get_finished_spans()] + assert "job.claim" in names + assert "job.download" in names + assert "job.decrypt" in names + assert "job.transcribe" in names + assert "job.complete" in names + # Deliberately absent: `TracedDocumentSink` opens no span, because the + # concrete `OutlineSink` behind it carries its own richer + # `document.create` CLIENT span. A wrapper span here would be a second, + # identical, nested entry in every waterfall. `FakeDocuments` stands in + # for that adapter, so nothing emits one in this test. + assert "document.create" not in names + # The root closes last, so every stage above is a child of it. + assert names[-1] == "job.process" + + +async def test_no_exported_span_carries_the_transcript( + tmp_path: Path, exported: InMemorySpanExporter +) -> None: + """The canary assertion. A leak fails here rather than in production.""" + await _run_one(tmp_path) + + for finished in exported.get_finished_spans(): + serialised = finished.to_json() + assert CANARY not in serialised, f"{finished.name} leaked the transcript" + assert "speaker-100" not in serialised, f"{finished.name} leaked a display name" + # The S3 key embeds a Discord user id, so it is out of spans + # entirely -- see `fields.DENIED_NAMES`. + assert "sessions/1/speakers/100.enc" not in serialised + + +async def test_counts_of_the_transcript_are_recorded_but_not_its_text( + tmp_path: Path, exported: InMemorySpanExporter +) -> None: + """The judgement call, made explicit: a length travels, the words do not.""" + await _run_one(tmp_path) + + (transcribe,) = [s for s in exported.get_finished_spans() if s.name == "job.transcribe"] + attributes = dict(transcribe.attributes or {}) + assert attributes["sturnus.segment_count"] == 1 + assert attributes["sturnus.char_count"] == len(CANARY) + assert attributes["sturnus.language"] == "de" + assert CANARY not in transcribe.to_json() + + +async def test_the_claim_stamps_ids_onto_the_enclosing_root_span( + tmp_path: Path, exported: InMemorySpanExporter +) -> None: + """The order-dependent bit, pinned so a later refactor cannot break it quietly.""" + await _run_one(tmp_path) + + (root,) = [s for s in exported.get_finished_spans() if s.name == "job.process"] + attributes = dict(root.attributes or {}) + assert attributes["sturnus.job_id"] == 1 + assert attributes["sturnus.session_id"] == 1 + + +async def test_a_failing_stage_marks_its_span_without_a_message( + tmp_path: Path, exported: InMemorySpanExporter +) -> None: + """`process_one` catches the failure; the span still records that it happened.""" + queue = FakeQueue([job()]) + with span("job.process"): + await process_one( + queue=TracedQueue(queue), + engine=TracedTranscriptionEngine(FakeEngine(CANARY, fail=True)), + store=TracedAudioDownloader(FakeStore()), + crypto=TracedDecryptor(FakeCrypto()), + documents=TracedDocumentSink(FakeDocuments()), + sessions=FakeSessions(), + jobs=FakeJobs(), + links=FakeLinks(), + config=FakeConfig(), + work_dir=tmp_path, + max_attempts=3, + ) + + assert len(queue.failed) == 1 + (transcribe,) = [s for s in exported.get_finished_spans() if s.name == "job.transcribe"] + assert transcribe.status.status_code.name == "ERROR" + assert transcribe.status.description is None + assert dict(transcribe.attributes or {})["error.type"] == "RuntimeError" + # "model exploded" is the message; it belongs in the database column, + # not in Tempo. + assert "model exploded" not in transcribe.to_json() + + +async def test_the_root_span_of_a_failed_job_does_not_say_it_was_done( + tmp_path: Path, exported: InMemorySpanExporter +) -> None: + """The lie, at the span end of it. + + `process_one` returns `True` here -- the job was attempted, the engine + raised, `queue.fail` ran -- and the worker loop turned that boolean + into `outcome="done"`. Driven through the *real* `process_one` so the + return value really is `True` on this path rather than being asserted + to be. + """ + queue = FakeQueue([job()]) + with span("job.process"): + attempted = await process_one( + queue=TracedQueue(queue), + engine=TracedTranscriptionEngine(FakeEngine(CANARY, fail=True)), + store=TracedAudioDownloader(FakeStore()), + crypto=TracedDecryptor(FakeCrypto()), + documents=TracedDocumentSink(FakeDocuments()), + sessions=FakeSessions(), + jobs=FakeJobs(), + links=FakeLinks(), + config=FakeConfig(), + work_dir=tmp_path, + max_attempts=3, + ) + + assert attempted is True, "the premise: the return value cannot tell these apart" + (root,) = [s for s in exported.get_finished_spans() if s.name == "job.process"] + assert dict(root.attributes or {})["sturnus.outcome"] == "failed" + + +async def test_the_root_span_of_a_completed_job_says_done( + tmp_path: Path, exported: InMemorySpanExporter +) -> None: + """The control on the test above: `done` still reaches the span it belongs on. + + Without this, deleting the `outcome` stamp altogether would satisfy the + "not done" assertion above and lose the label entirely. + """ + await _run_one(tmp_path) + + (root,) = [s for s in exported.get_finished_spans() if s.name == "job.process"] + assert dict(root.attributes or {})["sturnus.outcome"] == "done" diff --git a/tests/infrastructure/test_whisper.py b/tests/infrastructure/test_whisper.py index f7ff45a..31f0dc4 100644 --- a/tests/infrastructure/test_whisper.py +++ b/tests/infrastructure/test_whisper.py @@ -1,6 +1,6 @@ import logging import wave -from collections.abc import Callable +from collections.abc import Callable, Iterator from dataclasses import fields from pathlib import Path from types import SimpleNamespace @@ -15,8 +15,15 @@ from faster_whisper.transcribe import ( # type: ignore[import-untyped] TranscriptionOptions, ) +from opentelemetry.metrics import CallbackOptions +from sturnus.infrastructure.telemetry import TRANSCRIPTION_PROGRESS, TranscriptionProgress from sturnus.infrastructure.whisper import WhisperEngine, _on_the_frame_grid +from sturnus.observability.events import Event + +#: The module whose lines these tests read. Named once, because `caplog` +#: captures every propagated record and the engine's collaborators log too. +WHISPER_LOGGER = "sturnus.infrastructure.whisper" FIXTURE = Path(__file__).parent.parent / "fixtures" / "hello.wav" @@ -113,8 +120,25 @@ def _as_decoded_from(window_start: float, start: float, end: float, text: str) - class _FakeInfo: - def __init__(self, language: str | None) -> None: + """The three attributes `_transcribe` reads off a `TranscriptionInfo`. + + `duration_after_vad` defaults to `duration` because that is what the + installed faster-whisper does on the path Sturnus takes: it is only + reduced under `if vad_filter and clip_timestamps == "0"`, and + `WhisperEngine` sets `vad_filter=False` and a clip list. A fake that + reported a smaller number here would be modelling a code path this + adapter refuses to take. + """ + + def __init__( + self, + language: str | None, + duration: float = 0.0, + duration_after_vad: float | None = None, + ) -> None: self.language = language + self.duration = duration + self.duration_after_vad = duration if duration_after_vad is None else duration_after_vad class _RecordingModel: @@ -159,7 +183,15 @@ def transcribe(self, audio: Any, **kwargs: Any) -> tuple[Any, _FakeInfo]: # faster-whisper returns a generator, so a caller that forgot to # consume it would see no segments; returning an iterator keeps the # fake honest about that. - return iter(segments), _FakeInfo(self.language) + # + # The durations are computed from the array it was handed, exactly + # as the library computes them (`duration = audio.shape[0] / + # sampling_rate`). Since the engine hands over the concatenated + # speech, that is the speech in the recording rather than the + # recording -- which is the point: reporting a constant, or the file + # length, would make every progress assertion below a statement about + # this fake rather than about the audio the model was given. + return iter(segments), _FakeInfo(self.language, duration=len(audio) / 16_000) def _engine_with(model: _RecordingModel, default_language: str = "de") -> WhisperEngine: @@ -176,6 +208,10 @@ def _engine_with(model: _RecordingModel, default_language: str = "de") -> Whispe engine = object.__new__(WhisperEngine) engine._model = model engine._default_language = default_language + # `__init__` keeps the name it was given so that the metrics can be + # labelled by model without any call site passing it in again. Bypassing + # `__init__` means setting it here too. + engine._model_name = "tiny" return engine @@ -1075,6 +1111,428 @@ async def test_the_veto_on_the_no_speech_skip_is_closed_explicitly(tmp_path: Pat assert model.calls[0]["log_prob_threshold"] is None +# --------------------------------------------------------------------------- +# Progress: what the lazy generator was throwing away +# --------------------------------------------------------------------------- +# +# `WhisperModel.transcribe` returns `(Iterable[Segment], TranscriptionInfo)` +# and the segments are produced as decoding proceeds. Draining them with a +# tuple comprehension consumed the whole generator in one expression and +# discarded every intermediate observation, which is why a 100-minute job +# and a job that decoded nothing at all looked identical from outside until +# both had finished. +# +# Every test below drives the real `WhisperEngine` through the real model +# seam -- `_RecordingModel`, which hands back a generator exactly as the +# library does -- and reads the metrics through the same callbacks the +# OpenTelemetry SDK calls. Nothing here asserts against a meter fake, +# because a meter fake would only prove that this code can call itself. + + +def _observed(callback: Any) -> list[tuple[float, dict[str, Any]]]: + """One instrument's observations, read the way the SDK reads them.""" + return [ + (observation.value, dict(observation.attributes or {})) + for observation in callback(CallbackOptions()) + ] + + +def _value_for(callback: Any, model: str) -> float | None: + for value, attributes in _observed(callback): + if attributes.get("model") == model: + return value + return None + + +class _WatchingModel(_RecordingModel): + """`_RecordingModel` whose caller is watched between segments. + + The whole point of the change under test is that something is reported + *while* the generator is still being consumed. A fake that hands back a + finished tuple could not tell a loop from a comprehension; this one + calls `watcher` after each segment is yielded, so a test can read the + live gauges at a moment when decoding is genuinely half done. + """ + + def __init__(self, segments: tuple[_FakeSegment, ...], watcher: Any, **kwargs: Any) -> None: + super().__init__(segments=segments, **kwargs) + self._watcher = watcher + + def transcribe(self, audio: Any, **kwargs: Any) -> tuple[Any, _FakeInfo]: + self.calls.append({"audio": audio, **kwargs}) + seconds = len(audio) / 16_000 + + def generate() -> Any: + for segment in self.segments: + yield segment + self._watcher() + + return generate(), _FakeInfo(self.language, duration_after_vad=seconds) + + +@pytest.fixture +def idle_progress() -> Iterator[TranscriptionProgress]: + """The module-level progress object, left as it was found. + + It is process-global on purpose -- one worker transcribes one job at a + time (Spec 5.3), so "the job in flight" is a singular thing -- and a + test that left a job in flight would make the next test's `stall` + reading grow forever. + """ + yield TRANSCRIPTION_PROGRESS + TRANSCRIPTION_PROGRESS.end() + + +async def test_the_position_is_reported_while_the_generator_is_still_running( + tmp_path: Path, idle_progress: TranscriptionProgress +) -> None: + """The defect in one assertion: a comprehension cannot pass this. + + `tuple(... for s in segments)` consumes the generator inside a single + expression, so nothing between the first and the last segment is ever + observable. This reads the gauge from inside the generator itself. + """ + recording = tmp_path / "speech.wav" + # Five seconds of tone, so the concatenated speech the engine hands over + # is longer than the last segment claims to reach. The three segments all + # come out of the first encoder window, which is where a five-second clip + # puts them. + _write_wav(recording, np.concatenate([_tone(5.0), np.zeros(16_000 * 8, dtype=np.float32)])) + + seen: list[float | None] = [] + model = _WatchingModel( + segments=( + _as_decoded_from(0.0, 0.0, 1.0, " one"), + _as_decoded_from(0.0, 1.0, 2.5, " two"), + _as_decoded_from(0.0, 2.5, 4.0, " three"), + ), + watcher=lambda: seen.append( + _value_for(idle_progress.observe_position, "tiny"), + ), + ) + + await _engine_with(model).transcribe(recording, language="de", initial_prompt=None) + + assert seen == [1.0, 2.5, 4.0], "progress was only visible after the job had finished" + + +async def test_a_job_is_in_flight_before_the_model_has_yielded_anything( + tmp_path: Path, idle_progress: TranscriptionProgress +) -> None: + """The wedge this instrument exists for happens *inside* the library call. + + `WhisperModel.transcribe` extracts features and detects a language + before it yields its first segment, and that is where the collapse + that produced empty transcripts happened. Starting the clock at the + first segment instead would leave a job stuck in there reporting + nothing at all -- not a stalled job, no job. + + Asserted from inside the model call rather than around it: the point + is the state of the world at a moment that only the model can reach. + """ + recording = tmp_path / "speech.wav" + _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000, dtype=np.float32)])) + during: list[list[tuple[float, dict[str, Any]]]] = [] + + class _ObservingModel(_RecordingModel): + def transcribe(self, audio: Any, **kwargs: Any) -> tuple[Any, _FakeInfo]: + during.append(_observed(idle_progress.observe_stall)) + return super().transcribe(audio, **kwargs) + + await _engine_with(_ObservingModel()).transcribe(recording, language="de", initial_prompt=None) + + (observed,) = during + assert observed, "no job was in flight while the model was running" + (stalled_for, attributes) = observed[0] + assert attributes == {"model": "tiny"} + assert stalled_for >= 0.0 + + +async def test_a_job_in_flight_reports_the_length_it_is_working_through( + tmp_path: Path, idle_progress: TranscriptionProgress +) -> None: + """The denominator. A position with no total is a number nobody can read. + + `TranscriptionInfo.duration_after_vad` is what the library reports, and + since the engine hands the model the gated speech concatenated rather + than the padded track, that is the **speech** in the recording, on the + same timeline the positions reported to `advance` are on. + + 2.25 s and not 10.0 s, and the difference is the whole assertion: the + file is ten seconds long and holds a two-second tone, which the gate + widens by `_HANGOVER_SECONDS` at the end and clamps at the start. A + denominator that came back as the file length would mean the padded + array had reached the model again, and every real-time factor built on + it would be wrong by the ratio of silence to speech. + """ + recording = tmp_path / "speech.wav" + _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000 * 8, dtype=np.float32)])) + + seen: list[float | None] = [] + model = _WatchingModel( + segments=(_as_decoded_from(0.0, 0.0, 1.0, " one"),), + watcher=lambda: seen.append(_value_for(idle_progress.observe_total, "tiny")), + ) + + await _engine_with(model).transcribe(recording, language="de", initial_prompt=None) + + assert seen == [pytest.approx(2.25, abs=0.05)] + + +async def test_nothing_is_observed_when_no_job_is_in_flight( + tmp_path: Path, idle_progress: TranscriptionProgress +) -> None: + """An idle worker must publish no position at all, not a stale one. + + A gauge that keeps reporting the last job's numbers reads as "a job is + 43 minutes in" forever, and the alert built on + `seconds_since_progress` would fire on an idle deployment every time. + Emitting no observation lets the series go stale instead, which is what + every alert expression in `docs/operations.md` section 7.5 relies on. + """ + recording = tmp_path / "speech.wav" + _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000, dtype=np.float32)])) + + await _engine_with( + _RecordingModel(segments=(_as_decoded_from(0.0, 0.0, 1.0, " one"),)) + ).transcribe(recording, language="de", initial_prompt=None) + + assert _observed(idle_progress.observe_position) == [] + assert _observed(idle_progress.observe_total) == [] + assert _observed(idle_progress.observe_stall) == [] + + +async def test_the_decoded_counter_totals_the_audio_the_model_was_handed( + tmp_path: Path, idle_progress: TranscriptionProgress +) -> None: + """Divided by wall time this is the real-time factor, and that is its job. + + "The audio the model was handed" is read off the call itself rather than + written down as a number, because that is precisely the quantity under + test: the engine concatenates the gated speech and hands *that* over, so + the counter has to total the concatenated seconds and not the ten seconds + the file is long. Asserting the file length here would pass just as well + if the padded track went back to the model, which is the regression this + counter would otherwise hide. + """ + recording = tmp_path / "speech.wav" + _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000 * 8, dtype=np.float32)])) + before = _value_for(idle_progress.observe_decoded, "tiny") or 0.0 + model = _RecordingModel(segments=(_as_decoded_from(0.0, 0.0, 2.0, " hallo"),)) + + await _engine_with(model).transcribe(recording, language="de", initial_prompt=None) + + handed_over = len(model.calls[0]["audio"]) / 16_000 + assert handed_over == pytest.approx(2.25, abs=0.05), "the premise: the padding was cut out" + after = _value_for(idle_progress.observe_decoded, "tiny") + assert after is not None + assert after - before == pytest.approx(handed_over, abs=0.05) + + +async def test_a_job_that_decoded_nothing_still_counts_the_audio_it_was_given( + tmp_path: Path, idle_progress: TranscriptionProgress +) -> None: + """**The two-day defect, and the reason this counter is worth building.** + + Silero's recurrent state collapsed on the bit-exact padding and the + model came back with no segments at all for a 100-minute recording, in + well under a minute. The symptom everyone saw was an empty transcript, + which is exactly what a participant who never spoke also produces -- + and it was read as that for a day. + + Counting the audio the model was *given* rather than the audio the + segments happen to cover is what turns that into an impossible number: + seconds of recording decoded in microseconds is a real-time factor of + many thousands, against the 1.94x this hardware actually manages. + Counting segment ends instead would have contributed zero here, which + is a much quieter way of being wrong. + """ + recording = tmp_path / "speech.wav" + _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000 * 8, dtype=np.float32)])) + before = _value_for(idle_progress.observe_decoded, "tiny") or 0.0 + model = _RecordingModel(segments=()) + + result = await _engine_with(model).transcribe(recording, language="de", initial_prompt=None) + + assert result.segments == (), "the premise: the model produced nothing" + handed_over = len(model.calls[0]["audio"]) / 16_000 + after = _value_for(idle_progress.observe_decoded, "tiny") + assert after is not None + # Not zero, which is the entire point, and the seconds the model was + # handed rather than a token amount. + assert after - before == pytest.approx(handed_over, abs=0.05) + assert after - before > 0.0 + + +async def test_the_progress_is_cleared_when_the_model_raises( + tmp_path: Path, idle_progress: TranscriptionProgress +) -> None: + """A failed job must not be reported as one that wedged. + + Without the `finally`, an exception out of the decoder would leave the + job in flight forever and `seconds_since_progress` would climb past + every threshold while the worker went happily on to the next job. + """ + recording = tmp_path / "speech.wav" + _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000, dtype=np.float32)])) + + class _ExplodingModel(_RecordingModel): + def transcribe(self, audio: Any, **kwargs: Any) -> tuple[Any, _FakeInfo]: + del audio, kwargs + raise RuntimeError("ct2 died") + + with pytest.raises(RuntimeError): + await _engine_with(_ExplodingModel()).transcribe( + recording, language="de", initial_prompt=None + ) + + assert _observed(idle_progress.observe_stall) == [] + + +def test_the_stall_clock_runs_from_before_the_first_segment() -> None: + """The actual alert condition, and the one a position gauge cannot express. + + A job that wedges *before* producing its first segment has a position + of zero and a last-observed-progress of never -- which is + indistinguishable from a job that has only just started unless the + clock starts at `begin()` rather than at the first `advance()`. + + Driven on its own instance with a clock the test owns: the module-level + one reads `time.monotonic`, and waiting five real minutes to assert + five minutes is not a test. + """ + ticks = iter([100.0, 105.0, 400.0]) + progress = TranscriptionProgress(now=lambda: next(ticks)) + + progress.begin("large-v3") # t=100 + + # t=105: still nothing decoded, and that is precisely the alert. + assert _value_for(progress.observe_stall, "large-v3") == 105.0 - 100.0 + assert _value_for(progress.observe_position, "large-v3") == 0.0 + + +def test_the_stall_clock_is_reset_by_every_segment() -> None: + """The other half: a slow job that is still making progress is not stuck.""" + ticks = iter([0.0, 10.0, 12.0]) + progress = TranscriptionProgress(now=lambda: next(ticks)) + + progress.begin("large-v3") # t=0 + progress.advance(30.0) # t=10, a segment arrived + + assert _value_for(progress.observe_stall, "large-v3") == 12.0 - 10.0 + + +def test_progress_never_goes_backwards() -> None: + """`advance` is a position, not a delta, and the decoder can repeat one. + + faster-whisper's seek loop can emit a segment whose `end` is not past + the previous one -- a clip boundary, a zero-length segment. Subtracting + would decrement a counter, which OpenTelemetry treats as a counter + reset and Prometheus turns into an enormous spurious rate. + """ + progress = TranscriptionProgress(now=lambda: 0.0) + progress.begin("large-v3") + progress.advance(30.0) + progress.advance(12.0) + + assert _value_for(progress.observe_position, "large-v3") == 30.0 + assert _value_for(progress.observe_decoded, "large-v3") == 30.0 + + +def test_no_progress_metric_carries_an_id_of_any_kind() -> None: + """The cardinality and privacy rule, asserted over every observation. + + A session id, job id, guild id or user id on a metric is unbounded + cardinality *and* a record of who was in a voice channel when, kept for + as long as the metric store keeps anything. Component and model name + are enough, and `component` is already a resource attribute rather than + a label. + """ + progress = TranscriptionProgress(now=lambda: 0.0) + progress.begin("large-v3") + progress.set_total(600.0) + progress.advance(12.0) + + for callback in ( + progress.observe_decoded, + progress.observe_position, + progress.observe_total, + progress.observe_stall, + ): + observations = _observed(callback) + assert observations, f"{callback.__name__} observed nothing to check" + for _, attributes in observations: + assert set(attributes) == {"model"}, attributes + + +# --------------------------------------------------------------------------- +# The log lines: the gate's own numbers, which nothing else can see +# --------------------------------------------------------------------------- + + +async def test_a_file_the_gate_rejects_says_so_instead_of_going_quiet( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """An empty transcript has two causes and they need telling apart. + + Either the gate found nothing above the silence floor, or the model was + called and produced nothing. Both end as a participant with no lines in + the document, and the second one is the failure that cost this project + two days. Only the first produces this line. + """ + silent = tmp_path / "padding.wav" + _write_wav(silent, np.zeros(16_000 * 3, dtype=np.float32)) + model = _RecordingModel() + + with caplog.at_level(logging.INFO, logger=WHISPER_LOGGER): + await _engine_with(model).transcribe(silent, language="de", initial_prompt=None) + + assert model.calls == [], "the premise: the model was never called" + (line,) = [r for r in caplog.records if r.name == WHISPER_LOGGER] + assert getattr(line, "sturnus_event", None) == str(Event.TRANSCRIPTION_SKIPPED) + fields = getattr(line, "sturnus_fields", {}) + assert fields["clips"] == 0 + assert fields["audio_seconds"] == pytest.approx(3.0, abs=0.05) + assert fields["model"] == "tiny" + + +@pytest.mark.usefixtures("idle_progress") +async def test_a_decoded_job_reports_how_much_of_it_was_speech( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """`speech_seconds` against `audio_seconds` is the gate's own signature. + + It is the number that would have named Silero as the culprit on the + first read rather than the third day: "one second of speech in two + minutes of recording" is not a plausible meeting, and no other line + Sturnus emits can say it -- `job.transcribed` is produced in + `sturnus.application.worker`, which cannot see the gate at all. + """ + recording = tmp_path / "speech.wav" + _write_wav( + recording, + np.concatenate([np.zeros(16_000 * 5, dtype=np.float32), _tone(2.0)]), + ) + # On the concatenated timeline the engine hands over, where the one clip + # begins at zero -- the tone is at 00:05 in the file and the restore is + # what puts it back there. + model = _RecordingModel(segments=(_as_decoded_from(0.0, 0.0, 2.0, " hallo"),)) + + with caplog.at_level(logging.INFO, logger=WHISPER_LOGGER): + await _engine_with(model).transcribe(recording, language="de", initial_prompt=None) + + (line,) = [r for r in caplog.records if r.name == WHISPER_LOGGER] + assert getattr(line, "sturnus_event", None) == str(Event.TRANSCRIPTION_DECODED) + fields = getattr(line, "sturnus_fields", {}) + assert fields["clips"] == 1 + assert fields["audio_seconds"] == pytest.approx(7.0, abs=0.05) + # The tone is 2 seconds, plus a quarter-second hangover at each end. + assert 2.0 <= float(fields["speech_seconds"]) <= 3.0 + assert fields["segments"] == 1 + assert fields["model"] == "tiny" + assert not line.args, "everything that varies is a field, not a %-argument" + + async def test_a_track_the_decoder_emptied_says_so_in_the_log( tmp_path: Path, caplog: pytest.LogCaptureFixture ) -> None: @@ -1096,20 +1554,30 @@ async def test_a_track_the_decoder_emptied_says_so_in_the_log( _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000, dtype=np.float32)])) engine = _engine_with(_RecordingModel(segments=())) - with caplog.at_level(logging.WARNING, logger="sturnus.infrastructure.whisper"): + with caplog.at_level(logging.WARNING, logger=WHISPER_LOGGER): result = await engine.transcribe(recording, language="de", initial_prompt=None) assert result.segments == () assert len(caplog.records) == 1 - message = caplog.records[0].getMessage() - assert "roomtone.wav" in message - # 2.3 s, and none of the other three numbers this file offers. The - # recording is 3.0 s long and holds a 2.0 s tone; the gate widens that - # tone by `_HANGOVER_SECONDS` at the end and clamps it at the start, - # yielding the 2.25 s it actually handed the decoder. So this also fails - # if the message reports the file length, the tone length, or a count of + record = caplog.records[0] + # WARNING, not INFO. The same event is emitted on every job; its severity + # is the whole signal here, and an operator filtering for problems sees + # this one only if it carries the right level. + assert record.levelno == logging.WARNING + # The recording is no longer named in the message. Identity travels on the + # enclosing `job.transcribe` span and on the worker's own `job.transcribed` + # event, which is where this branch puts it deliberately; a filename in a + # message string is not something a log query can group by. What the line + # must still carry is the number that decides how to read it. + fields = record.sturnus_fields # type: ignore[attr-defined] + # 2.25 s of speech, and none of the other three durations this file + # offers. The recording is 3.0 s long and holds a 2.0 s tone; the gate + # widens that tone by `_HANGOVER_SECONDS` at the end and clamps it at the + # start, yielding what it actually handed the decoder. So this also fails + # if the event reports the file length, the tone length, or a count of # clips dressed up as seconds. - assert "2.3 s" in message, message + assert fields["speech_seconds"] == pytest.approx(2.25, abs=0.05), fields + assert fields["segments"] == 0, fields async def test_a_track_that_produced_text_is_not_reported_as_a_loss( @@ -1127,7 +1595,7 @@ async def test_a_track_that_produced_text_is_not_reported_as_a_loss( _RecordingModel(segments=(_as_decoded_from(0.0, 0.0, 1.5, " Guten Morgen."),)) ) - with caplog.at_level(logging.WARNING, logger="sturnus.infrastructure.whisper"): + with caplog.at_level(logging.WARNING, logger=WHISPER_LOGGER): result = await engine.transcribe(recording, language="de", initial_prompt=None) assert [s.text for s in result.segments] == [" Guten Morgen."] @@ -1150,7 +1618,7 @@ async def test_the_transcribed_text_is_never_logged( secret = " Wir kuendigen den Vertrag mit Beispiel GmbH." engine = _engine_with(_RecordingModel(segments=(_as_decoded_from(0.0, 0.0, 1.5, secret),))) - with caplog.at_level(logging.DEBUG, logger="sturnus.infrastructure.whisper"): + with caplog.at_level(logging.DEBUG, logger=WHISPER_LOGGER): await engine.transcribe(recording, language="de", initial_prompt=None) assert caplog.records, "the debug trace exists at all, so this is not vacuous" diff --git a/tests/observability/__init__.py b/tests/observability/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/observability/test_no_payload_leaks.py b/tests/observability/test_no_payload_leaks.py new file mode 100644 index 0000000..2e6d252 --- /dev/null +++ b/tests/observability/test_no_payload_leaks.py @@ -0,0 +1,339 @@ +"""The canary test: drive the real pipelines and grep the logs for payload. + +`docs/verification/end-to-end-checklist.md` makes it a blocking legal gate +that no transcript, audio, token or key appears in any pod log. That gate is +executed by hand, once, after a deploy. This is the same check, run on every +commit, against the two code paths that actually touch the protected data: +`RecordingService.close()` and `process_one()`. + +It deliberately does **not** know which fields those paths log. It plants +unique strings in the places a payload lives -- the transcript text, a +display name, the audio bytes -- and asserts none of them survives anywhere +in the output at DEBUG. That is what makes it catch a leak through a +variable no denylist happens to name, which is precisely the case the AST +test in `tests/test_logging_discipline.py` cannot see. + +Extends the precedent already set by `tests/infrastructure/test_outline.py`, +which asserts the same property for one adapter with `caplog`. +""" + +from __future__ import annotations + +import io +import json +import logging +from collections.abc import Iterator +from datetime import UTC, datetime +from pathlib import Path + +import pytest + +from sturnus.application.recording import RecordingService +from sturnus.application.worker import process_one +from sturnus.domain.session import EndReason, SessionTimeouts +from sturnus.observability.setup import configure_logging +from tests.application.test_worker import ( + FakeConfig, + FakeCrypto, + FakeDocuments, + FakeEngine, + FakeJobs, + FakeLinks, + FakeQueue, + FakeSessions, + FakeStore, + job, +) + +#: Each is unique, so a failure message names exactly what escaped. +TRANSCRIPT_CANARY = "CANARYTRANSCRIPT-the-merger-closes-on-the-fourteenth" +DISPLAY_NAME_CANARY = "CANARYDISPLAYNAME-Dr-Alice-Example" +AUDIO_CANARY = b"CANARYAUDIO" * 64 +MASTER_KEY_CANARY = "CANARYMASTERKEY-cGxlYXNlIGRvIG5vdCBsb2cgbWU" +#: A voice channel's name is free text an administrator chose, and since +#: `#34` it travels from the bot into the protocol header. It is not a +#: registered field, so it must not reach a log line either. +CHANNEL_NAME_CANARY = "CANARYCHANNELNAME-Vorstandssitzung-vertraulich" + +T0 = datetime(2026, 8, 20, 12, 0, 0, tzinfo=UTC) + + +@pytest.fixture +def captured() -> Iterator[io.StringIO]: + """Everything the process would write to stdout, at DEBUG.""" + buffer = io.StringIO() + root = logging.getLogger() + saved_handlers = root.handlers[:] + saved_level = root.level + configure_logging("worker", level="DEBUG", log_format="json", stream=buffer) + yield buffer + root.handlers = saved_handlers + root.setLevel(saved_level) + + +def _assert_no_canaries(captured: io.StringIO) -> None: + output = captured.getvalue() + assert output, "nothing was logged; the test would pass vacuously" + for name, canary in ( + ("transcript", TRANSCRIPT_CANARY), + ("display name", DISPLAY_NAME_CANARY), + ("master key", MASTER_KEY_CANARY), + ("channel name", CHANNEL_NAME_CANARY), + ): + assert canary not in output, f"a {name} reached the log stream" + assert "CANARYAUDIO" not in output, "audio bytes reached the log stream" + # Every line must still be parseable -- a leak that also breaks the + # format would otherwise hide behind a parse error. + for line in output.splitlines(): + if line.strip(): + json.loads(line) + + +async def test_the_worker_pipeline_logs_no_payload(tmp_path: Path, captured: io.StringIO) -> None: + """`process_one` end to end, with the transcript replaced by a canary.""" + queue = FakeQueue([job()]) + queue.last_is_final = True + sessions = FakeSessions() + sessions.names = {100: DISPLAY_NAME_CANARY} + + await process_one( + queue=queue, + engine=FakeEngine(TRANSCRIPT_CANARY), + store=FakeStore(), + crypto=FakeCrypto(), + documents=FakeDocuments(), + sessions=sessions, + jobs=FakeJobs(), + links=FakeLinks(), + config=FakeConfig(), + work_dir=tmp_path, + max_attempts=3, + ) + + # The canary really did flow through the code under test. + assert TRANSCRIPT_CANARY in queue.completed[0][1] + _assert_no_canaries(captured) + + +async def test_a_failing_job_logs_no_payload(tmp_path: Path, captured: io.StringIO) -> None: + """The failure path is where a message most wants to carry a payload.""" + + class ExplodingEngine: + # Three parameters, matching `application.transcription. + # TranscriptionEngine` since main added the per-guild vocabulary + # prompt (#44). A two-parameter fake would make `process_one` raise + # `TypeError` before the engine ever ran, so the canary this test + # plants would never be in the message it checks -- the test would + # go on passing while measuring nothing. + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> object: + del path, language, initial_prompt + raise RuntimeError(f"decode failed on {TRANSCRIPT_CANARY}") + + queue = FakeQueue([job()]) + await process_one( + queue=queue, + engine=ExplodingEngine(), # type: ignore[arg-type] + store=FakeStore(), + crypto=FakeCrypto(), + documents=FakeDocuments(), + sessions=FakeSessions(), + jobs=FakeJobs(), + links=FakeLinks(), + config=FakeConfig(), + work_dir=tmp_path, + max_attempts=3, + ) + + assert len(queue.failed) == 1 + # The database column still gets the full message -- that is the point + # of withholding it from the log rather than destroying it. + assert TRANSCRIPT_CANARY in queue.failed[0][1] + _assert_no_canaries(captured) + + +class _Writer: + def __init__(self, path: Path) -> None: + self.path = path + self._chunks: list[bytes] = [] + + def write(self, at: datetime, pcm: bytes) -> None: + del at + self._chunks.append(pcm) + + def close(self) -> None: + self.path.write_bytes(b"".join(self._chunks)) + + +class _WriterFactory: + def __init__(self, root: Path) -> None: + self._root = root + + def open(self, session_id: int, discord_user_id: int, epoch: datetime) -> _Writer: + del epoch + directory = self._root / str(session_id) + directory.mkdir(parents=True, exist_ok=True) + return _Writer(directory / f"{discord_user_id}.wav") + + +class _Encryptor: + key_id = "canary-key-1" + + def new_session_key(self) -> object: + from sturnus.application.ports import SessionKey + + return SessionKey(plaintext=MASTER_KEY_CANARY.encode(), wrapped=MASTER_KEY_CANARY.encode()) + + def encrypt(self, source: Path, target: Path, key: bytes) -> None: + del key + target.write_bytes(b"encrypted:" + source.read_bytes()) + + +class _Sessions: + def __init__(self) -> None: + self.opened = False + + async def open_session( + self, guild_id: int, channel_id: int, channel_name: str | None, now: datetime + ) -> int: + # `channel_name` arrived with the protocol header work on main + # (#34): the worker writes the room's name into the document and + # only the bot can resolve it. The callers below pass + # `CHANNEL_NAME_CANARY` rather than `None`, because a fake that + # never sees a real name would not be exercising the case that + # matters. + del guild_id, channel_id, channel_name, now + self.opened = True + return 4711 + + async def add_participant( + self, session_id: int, discord_user_id: int, display_name: str, now: datetime + ) -> None: ... + + async def set_audio_epoch( + self, session_id: int, discord_user_id: int, at: datetime + ) -> None: ... + + async def record_silent_audio( + self, session_id: int, discord_user_id: int, at: datetime + ) -> None: ... + + async def close_session(self, session_id: int, ended_at: datetime, reason: str) -> None: ... + + async def record_session_key( + self, session_id: int, encryption_key_id: str, wrapped_data_key: bytes + ) -> None: ... + + async def session_key(self, session_id: int) -> tuple[str, bytes] | None: + del session_id + return None + + async def session_status(self, session_id: int) -> str | None: + del session_id + return "open" + + +class _Announcer: + """The `Announcer` port, which is not a log sink and must not become one. + + `RecordingService` speaks into the voice channel once per session, about + a speaker whose audio carries no level. That message names a Discord user + id -- deliberately, because the room has to know whose microphone it is -- + and the channel is the only place it is allowed to appear. Recording the + posts here rather than dropping them is what lets the canary sweep below + see everything the service produced, log line or not. + """ + + def __init__(self) -> None: + self.posted: list[tuple[int, str]] = [] + + async def post(self, channel_id: int, text: str) -> None: + self.posted.append((channel_id, text)) + + +class _Jobs: + def __init__(self) -> None: + self.enqueued: list[dict[str, object]] = [] + + async def enqueue(self, **kwargs: object) -> int: + self.enqueued.append(kwargs) + return len(self.enqueued) + + +class _Store: + async def put(self, key: str, source: Path) -> None: ... + + async def delete(self, key: str) -> None: ... + + +async def test_the_recording_pipeline_logs_no_payload( + tmp_path: Path, captured: io.StringIO +) -> None: + """`RecordingService` from first packet to `close()`. + + Audio bytes, a display name and a wrapped key all pass through, and + `session.opened` / `session.speaker_first_packet` / + `session.speaker_finalized` / `session.closed` are all emitted along the + way -- so this is a real run of the narrative, not a smoke test. + """ + jobs = _Jobs() + service = RecordingService( + guild_id=1, + channel_id=2, + channel_name=CHANNEL_NAME_CANARY, + timeouts=SessionTimeouts(), + sessions=_Sessions(), + jobs=jobs, + store=_Store(), + writers=_WriterFactory(tmp_path), + encryptor=_Encryptor(), # type: ignore[arg-type] + announcer=_Announcer(), + retention_days=30, + ) + + await service.participants_changed(2, T0) + assert service.is_recording + await service.voice_packet(100, DISPLAY_NAME_CANARY, 1, 0, AUDIO_CANARY, T0) + await service.close(EndReason.EMPTY, T0) + + assert len(jobs.enqueued) == 1, "the pipeline must actually have run" + _assert_no_canaries(captured) + + +async def test_a_session_that_recorded_nothing_is_an_error_line( + tmp_path: Path, captured: io.StringIO +) -> None: + """The incident's outcome, as one alertable line rather than as silence. + + A session with consenting participants that enqueues no job recorded + nothing. Before this branch that produced no document, no announcement, + and not one log line -- the failure was expressed entirely as absence. + """ + service = RecordingService( + guild_id=1, + channel_id=2, + channel_name=CHANNEL_NAME_CANARY, + timeouts=SessionTimeouts(), + sessions=_Sessions(), + jobs=_Jobs(), + store=_Store(), + writers=_WriterFactory(tmp_path), + encryptor=_Encryptor(), # type: ignore[arg-type] + announcer=_Announcer(), + retention_days=30, + ) + + await service.participants_changed(3, T0) + # No `voice_packet` at all: people were present and nothing arrived. + await service.close(EndReason.EMPTY, T0) + + closed = [ + json.loads(line) + for line in captured.getvalue().splitlines() + if line.strip() and json.loads(line)["event"] == "session.closed" + ] + assert len(closed) == 1 + assert closed[0]["level"] == "ERROR" + assert closed[0]["jobs_enqueued"] == 0 + assert closed[0]["session_id"] == 4711 diff --git a/tests/observability/test_package_boundaries.py b/tests/observability/test_package_boundaries.py new file mode 100644 index 0000000..6ad023a --- /dev/null +++ b/tests/observability/test_package_boundaries.py @@ -0,0 +1,81 @@ +"""`sturnus.observability` must stay importable from `application`. + +`tests/test_architecture.py` already forbids `application` importing +`infrastructure`, and `application` calls `log_event` in five modules. That +only stays safe while this package imports nothing but the standard library +and `sturnus.domain` -- otherwise `from sturnus.observability.events import +log_event` in `application/worker.py` would be a third-party import wearing +a disguise, and the existing architecture test would not see it. + +The concrete thing this prevents: putting the OpenTelemetry API in +`events.current_trace_context` instead of behind +`set_trace_context_provider`. That would be the obvious simplification, it +would work, and it would quietly make every `application` module depend on +`opentelemetry`. +""" + +from __future__ import annotations + +import ast +import sys +from pathlib import Path + +OBSERVABILITY = Path(__file__).parent.parent.parent / "src" / "sturnus" / "observability" + +_ALLOWED_PREFIXES = ("sturnus.observability", "sturnus.domain", "sturnus") + + +def _imported_modules(path: Path) -> set[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + found: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + found.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom) and node.module: + found.add(node.module) + return found + + +def _is_allowed(module: str) -> bool: + if module.split(".", 1)[0] in sys.stdlib_module_names: + return True + return module in _ALLOWED_PREFIXES or module.startswith( + ("sturnus.observability.", "sturnus.domain.") + ) + + +def test_observability_imports_only_stdlib_and_domain() -> None: + assert OBSERVABILITY.is_dir() + violations: list[str] = [] + checked = 0 + for path in OBSERVABILITY.rglob("*.py"): + checked += 1 + for module in _imported_modules(path): + if not _is_allowed(module): + violations.append(f"{path.name}: {module}") + assert checked > 0 + assert not violations, ( + "sturnus.observability must import only the standard library and " + "sturnus.domain, because sturnus.application imports it:\n" + "\n".join(violations) + ) + + +def test_opentelemetry_is_absent_from_the_shared_package() -> None: + """Stated separately because it is the specific mistake worth naming.""" + for path in OBSERVABILITY.rglob("*.py"): + source = path.read_text(encoding="utf-8") + assert "import opentelemetry" not in source, path.name + assert "from opentelemetry" not in source, path.name + + +def test_application_still_imports_no_infrastructure() -> None: + """A guard on the guard: the five new `log_event` imports must not have + dragged `sturnus.infrastructure` into `application` through a re-export. + """ + application = OBSERVABILITY.parent / "application" + violations: list[str] = [] + for path in application.rglob("*.py"): + for module in _imported_modules(path): + if module.startswith("sturnus.infrastructure"): + violations.append(f"{path.name}: {module}") + assert not violations, "\n".join(violations) diff --git a/tests/observability/test_redaction.py b/tests/observability/test_redaction.py new file mode 100644 index 0000000..102acd3 --- /dev/null +++ b/tests/observability/test_redaction.py @@ -0,0 +1,258 @@ +"""The redaction path, tested on values rather than on configuration. + +Every test here offers something that must never travel -- a transcript, a +token, raw audio bytes -- and asserts it is absent from the output. That is +the shape the blocking gate in `docs/verification/end-to-end-checklist.md` +asks for, applied in CI instead of once by hand after a deploy. +""" + +from __future__ import annotations + +import pytest + +from sturnus.domain.errors import DiagnosticSafeError +from sturnus.observability.fields import ( + ALLOWED_FIELDS, + CREDENTIAL_NAMES, + DENIED_NAMES, + LOG_ONLY_FIELDS, + METRIC_LABEL_FIELDS, + SAFE_SPAN_ATTRIBUTES, + span_attribute, +) +from sturnus.observability.redaction import ( + MAX_FIELD_CHARS, + SAFE_MESSAGE_TYPES, + safe_exception_message, + scrub_fields, + scrub_text, + scrub_value, +) + +#: A transcript-shaped string: the thing the whole design exists to keep out. +TRANSCRIPT = "so then I told my manager exactly what I thought of the reorg" + +#: Discord bot token shape: 24 base64url chars, a 6-char segment, a 27+ tail. +#: +#: Assembled at import rather than written as one literal, and not because +#: the value is real -- the first segment is base64 for "123456789012345678" +#: and the tail is the alphabet. It is assembled because a literal of this +#: shape is what GitHub's push protection detects, and it detects it +#: correctly: a string that looks exactly like a bot token has no business +#: sitting in a repository, whether or not this particular one would open +#: anything. Splitting it keeps the runtime value identical -- which is the +#: whole point, since these tests exist to prove the redaction catches this +#: exact shape -- while leaving nothing in the file for a scanner, or a +#: reader, to mistake for a credential. +DISCORD_TOKEN = ".".join( + ("MTIzNDU2Nzg5" + "MDEyMzQ1Njc4", "GaBcDe", "abcdefghijklmnopqrstuvwxyz" + "1234567") +) + + +def test_bytes_never_render_whatever_they_are() -> None: + """The highest-value rule: audio is always `bytes`, so `bytes` never print.""" + pcm = b"\x00\x01" * 4096 + scrubbed = scrub_value(pcm) + assert scrubbed == "" + assert "\x00" not in str(scrubbed) + + for kind in (bytearray(pcm), memoryview(pcm)): + assert scrub_value(kind) == "" + + +def test_a_wrapped_data_key_cannot_be_rendered_even_when_registered() -> None: + """A key is bytes, so the bytes rule catches it with no name-matching.""" + wrapped = bytes(range(256)) * 2 + assert scrub_value(wrapped) == "" + + +@pytest.mark.parametrize( + "secret,marker", + [ + (DISCORD_TOKEN, "discord_token"), + ("AKIA" + "IOSFODNN7EXAMPLE", "aws_access_key_id"), + ("Authorization: Bearer " + "sk-abcdefghijklmnop", "bearer_token"), + ("postgresql+asyncpg://sturnus:hunter2@db.internal/sturnus", "url_credentials"), + ("X-Amz-Signature=deadbeefcafe1234", "aws_sigv4"), + ], +) +def test_credentials_are_replaced_visibly(secret: str, marker: str) -> None: + """A redaction that leaves no trace teaches nobody -- each names itself.""" + scrubbed = scrub_text(f"something failed: {secret}") + assert f"«redacted:{marker}»" in scrubbed + assert "hunter2" not in scrubbed + if marker == "discord_token": + assert DISCORD_TOKEN not in scrubbed + + +def test_the_pydantic_validation_error_leak_is_closed() -> None: + """The exact string `StrictSettings` produces on a blank required value. + + Pydantic embeds the raw input dict in its message, so a blank + `STURNUS_MASTER_KEY_ID` puts the first characters of the Discord token + into the log store. This is that message, and the token must not survive + it. + """ + message = ( + "Value error, STURNUS_MASTER_KEY_ID is set but empty. " + f"[type=value_error, input_value={{'discord_token': '{DISCORD_TOKEN}', " + "'outline_redirect_uri': 'x'}, input_type=dict]" + ) + scrubbed = scrub_text(message) + assert DISCORD_TOKEN not in scrubbed + assert "«redacted:discord_token»" in scrubbed + # The diagnostic half survives: an operator still learns which variable. + assert "STURNUS_MASTER_KEY_ID is set but empty" in scrubbed + + +def test_long_strings_are_capped() -> None: + """Bounds the blast radius of anything the patterns miss.""" + scrubbed = scrub_text(TRANSCRIPT * 200) + assert isinstance(scrubbed, str) + assert len(scrubbed) < MAX_FIELD_CHARS + 64 + + +def test_unregistered_fields_are_dropped_not_stripped() -> None: + """Allowlist rebuild: a name nobody registered simply does not survive.""" + out = scrub_fields( + {"job_id": 7, "transcript": TRANSCRIPT, "display_name": "Alice Example"}, + warn=False, + ) + assert out == {"job_id": 7} + assert TRANSCRIPT not in str(out) + assert "Alice" not in str(out) + + +def test_an_object_renders_as_its_type_never_its_repr() -> None: + """A repr is how an object holding a transcript prints itself into a line.""" + + class Result: + def __repr__(self) -> str: # pragma: no cover - must never be called + return f"Result({TRANSCRIPT!r})" + + assert scrub_value(Result()) == "" + assert TRANSCRIPT not in str(scrub_value(Result())) + + +def test_exception_messages_are_withheld_unless_the_type_is_vouched_for() -> None: + """The default is redaction; `SAFE_MESSAGE_TYPES` is the whole exemption.""" + unsafe = RuntimeError(f"failed rendering {TRANSCRIPT}") + withheld = safe_exception_message(unsafe) + assert TRANSCRIPT not in withheld + assert "builtins.RuntimeError" in withheld + + # `OSError` messages are composed by the OS and the stdlib, not by us. + assert "Errno 111" in safe_exception_message(ConnectionRefusedError(111, "Errno 111")) + + class Vouched(DiagnosticSafeError): + pass + + assert "job 7 has no target" in safe_exception_message(Vouched("job 7 has no target")) + + +def test_the_exception_rule_is_the_one_sentry_uses() -> None: + """One list, not two. Sentry aliases this tuple rather than restating it.""" + from sturnus.infrastructure.observability import SAFE_VALUE_TYPES + + assert SAFE_VALUE_TYPES is SAFE_MESSAGE_TYPES + + +def test_denied_names_and_registered_fields_never_overlap() -> None: + """A name cannot be both registered and forbidden -- that would be ambiguous.""" + assert not (ALLOWED_FIELDS & DENIED_NAMES) + + +def test_log_only_fields_are_absent_from_spans_and_metrics() -> None: + """The exclusion is computed, so it cannot be forgotten at a call site.""" + for field in LOG_ONLY_FIELDS: + assert field in ALLOWED_FIELDS, "still loggable" + assert span_attribute(field) not in SAFE_SPAN_ATTRIBUTES + assert field not in METRIC_LABEL_FIELDS + + +def test_metric_labels_are_a_subset_of_the_registry() -> None: + """Cardinality: only fixed literals and `guild_id` may become a label.""" + assert METRIC_LABEL_FIELDS <= ALLOWED_FIELDS + assert "session_id" not in METRIC_LABEL_FIELDS + assert "job_id" not in METRIC_LABEL_FIELDS + + +def test_a_pretty_printed_voice_secret_key_is_redacted() -> None: + """The leaked value, in the shape it actually leaks in. + + `discord/ext/voice_recv/gateway.py:57` renders the op-4 payload with + `pformat`, so the Discord voice secret key reaches a log record as + thirty-two small integers in a list. It matches none of the + shape-based patterns -- it is not base64, not `AKIA…`, not a bot + token -- and it is inside `record.msg` rather than in any field the + allowlist governs. The only handle on it is the name it is assigned + to. + """ + rendered = ( + "Received op 4: \n{'dave_protocol_version': 0,\n" + " 'mode': 'aead_xchacha20_poly1305_rtpsize',\n" + " 'secret_key': [31337, 31338, 31339, 31340],\n 'ssrc': 12345}" + ) + scrubbed = scrub_text(rendered) + + assert "31337" not in scrubbed + assert "«redacted:secret_value»" in scrubbed + # The name survives, so an operator can tell a redaction from a field + # that was simply never emitted -- and the rest of the payload, which + # is what makes the line worth having, is untouched. + assert "secret_key" in scrubbed + assert "'ssrc': 12345" in scrubbed + + +def test_the_reader_spelling_of_the_same_key_is_redacted_too() -> None: + """`reader.py`'s is `secret_key=%s` with the value running to the newline. + + A pattern that stopped at the first token boundary would leave most of + a key on the line, which is not a smaller breach than all of it. + """ + scrubbed = scrub_text("CryptoError details:\n data=b'x'\n secret_key=31337313373133731337") + assert "31337" not in scrubbed + assert scrubbed.endswith("secret_key=«redacted:secret_value»") + + +@pytest.mark.parametrize( + "rendered", + [ + "token: CANARYSECRET", + "'token': 'CANARYSECRET'", + 'password="CANARYSECRET"', + "data_key=CANARYSECRET", + "client_secret: CANARYSECRET", + "database_url=postgresql://u:CANARYSECRET@h/d", + ], +) +def test_a_credential_name_assigned_anything_is_redacted(rendered: str) -> None: + """Every spelling a third-party formatter is likely to produce.""" + assert "CANARYSECRET" not in scrub_text(rendered) + + +def test_the_credential_rule_leaves_ordinary_prose_alone() -> None: + """Over-redaction is the right failure direction; noise is still a cost. + + The rule fires on ``, not on the + word. A line that merely mentions a token stays readable, which is + what keeps the control from being disabled by whoever gets tired of + it. + """ + prose = "the token was rejected by Outline and the password prompt never appeared" + assert scrub_text(prose) == prose + + +def test_the_credential_names_are_the_denied_names_and_not_a_second_list() -> None: + """One registry, split -- not two lists that can disagree. + + `fields.DENIED_NAMES` is the union of the payload half and the + credential half, so a name added to either is a name the AST rule in + `tests/test_logging_discipline.py` also refuses. + """ + assert CREDENTIAL_NAMES < DENIED_NAMES + assert "secret_key" in CREDENTIAL_NAMES + # The payload half stays out of the text patterns on purpose: `text:` + # and `body:` occur in English sentences, and a rule that redacts the + # word after them is a rule someone will delete. + assert not CREDENTIAL_NAMES & {"text", "body", "transcript", "display_name"} diff --git a/tests/observability/test_setup.py b/tests/observability/test_setup.py new file mode 100644 index 0000000..1f35bac --- /dev/null +++ b/tests/observability/test_setup.py @@ -0,0 +1,371 @@ +"""Logging configuration, asserted on the bytes that actually get written. + +Every test here parses the line the handler emitted rather than inspecting +the handler's configuration. A formatter that is configured correctly and +renders wrongly is exactly the failure a configuration assertion cannot +see, and the whole point of the JSON format is that something downstream +has to be able to parse it. +""" + +from __future__ import annotations + +import io +import json +import logging +from collections.abc import Iterator + +import pytest + +from sturnus.domain.errors import DiagnosticSafeError +from sturnus.observability.events import Event, log_event, log_exception +from sturnus.observability.redaction import SturnusFilter +from sturnus.observability.setup import ( + NEVER_ABOVE, + NEVER_BELOW, + configure_logging, + install_excepthooks, +) + +TRANSCRIPT = "and then the client said they would not be renewing" + + +@pytest.fixture +def stream() -> Iterator[io.StringIO]: + """A configured handler writing into a buffer, torn down afterwards.""" + buffer = io.StringIO() + saved_handlers = logging.getLogger().handlers[:] + saved_level = logging.getLogger().level + configure_logging("worker", log_format="json", stream=buffer) + yield buffer + logging.getLogger().handlers = saved_handlers + logging.getLogger().setLevel(saved_level) + + +def _lines(stream: io.StringIO) -> list[dict[str, object]]: + return [json.loads(line) for line in stream.getvalue().splitlines() if line.strip()] + + +def _event(stream: io.StringIO, event: Event) -> dict[str, object]: + """The one line carrying this event. + + Picked by name rather than by position because `scrub_fields` emits its + own "dropping unregistered field" warning through the same handler -- + which is the behaviour under test two tests down, not noise to suppress. + """ + matching = [line for line in _lines(stream) if line["event"] == str(event)] + assert len(matching) == 1, f"expected one {event} line, got {len(matching)}" + return matching[0] + + +#: AWS's own documentation example for an access key id, assembled rather +#: than written whole. Not because it opens anything -- it is the string AWS +#: prints in its own docs -- but because secret scanners flag the shape, and +#: they are right to: a reader cannot tell this from a real one either. The +#: runtime value is identical, which is the point, since these tests exist to +#: prove the scrubber catches exactly this shape. Same reasoning as +#: `DISCORD_TOKEN` in tests/observability/test_redaction.py. +AWS_ACCESS_KEY_ID = "AKIA" + "IOSFODNN7EXAMPLE" + + +def test_an_emitted_line_is_one_parseable_json_object(stream: io.StringIO) -> None: + log = logging.getLogger("sturnus.test") + log_event( + log, + logging.INFO, + Event.JOB_TRANSCRIBED, + "Transcribed a recording", + job_id=7, + session_id=4711, + segments=12, + realtime_factor=1.04, + ) + + line = _event(stream, Event.JOB_TRANSCRIBED) + assert line["event"] == "job.transcribed" + assert line["level"] == "INFO" + assert line["component"] == "worker" + assert line["logger"] == "sturnus.test" + assert line["msg"] == "Transcribed a recording" + assert line["job_id"] == 7 + assert line["session_id"] == 4711 + assert line["realtime_factor"] == 1.04 + assert "version" in line and "ts" in line + + +def test_an_unregistered_field_never_reaches_the_line(stream: io.StringIO) -> None: + """The registry is enforced at emit time, not documented and hoped for.""" + log = logging.getLogger("sturnus.test") + log_event( + log, + logging.INFO, + Event.JOB_TRANSCRIBED, + "Transcribed a recording", + job_id=7, + transcript=TRANSCRIPT, + display_name="Alice Example", + ) + + line = _event(stream, Event.JOB_TRANSCRIBED) + assert line["job_id"] == 7 + assert "transcript" not in line + assert "display_name" not in line + assert TRANSCRIPT not in stream.getvalue() + assert "Alice" not in stream.getvalue() + + +def test_audio_bytes_offered_as_a_field_render_as_a_length(stream: io.StringIO) -> None: + log = logging.getLogger("sturnus.test") + log_event( + log, + logging.INFO, + Event.SESSION_SPEAKER_FINALIZED, + "Finalized one speaker", + session_id=1, + bytes=b"\xde\xad\xbe\xef" * 512, + ) + + line = _event(stream, Event.SESSION_SPEAKER_FINALIZED) + assert line["bytes"] == "" + assert "\\xde" not in stream.getvalue() + + +def test_a_third_party_record_is_scrubbed_too(stream: io.StringIO) -> None: + """The filter is on the handler, so `botocore` gets the same treatment.""" + logging.getLogger("botocore").setLevel(logging.DEBUG) + logging.getLogger("botocore").warning("signing request with %s", AWS_ACCESS_KEY_ID) + + (line,) = _lines(stream) + assert AWS_ACCESS_KEY_ID not in stream.getvalue() + assert "«redacted:aws_access_key_id»" in str(line["msg"]) + + +def test_an_exception_message_is_withheld_but_the_traceback_survives( + stream: io.StringIO, +) -> None: + """Type and frames answer "where did this break"; the message is the payload.""" + log = logging.getLogger("sturnus.test") + try: + raise RuntimeError(f"jinja failed rendering {TRANSCRIPT}") + except RuntimeError as exc: + log_exception( + log, + logging.WARNING, + Event.SESSION_DOCUMENT_RETRY_FAILED, + "Document creation failed", + exc, + session_id=4711, + ) + + line = _event(stream, Event.SESSION_DOCUMENT_RETRY_FAILED) + assert line["error_type"] == "RuntimeError" + assert TRANSCRIPT not in stream.getvalue() + rendered = str(line["exc"]) + assert "" in rendered + # The frames are static program text and are what locate the failure. + assert "test_setup.py" in rendered + assert "Traceback" in rendered + + +def test_a_vouched_for_exception_keeps_its_message(stream: io.StringIO) -> None: + class Vouched(DiagnosticSafeError): + pass + + log = logging.getLogger("sturnus.test") + try: + raise Vouched("guild 42 has no document_target configured") + except Vouched as exc: + log_exception(log, logging.WARNING, Event.SWEEP_FAILED, "Sweep failed", exc) + + line = _event(stream, Event.SWEEP_FAILED) + assert "guild 42 has no document_target configured" in str(line["exc"]) + + +def test_configure_logging_replaces_rather_than_appends() -> None: + """One handler means one exit, and only that one carries the filter. + + A second handler would be a second way out of the process, bypassing + `SturnusFilter` entirely, so `configure_logging` assigns `root.handlers` + rather than appending to it. Asserted immediately after the call -- + pytest's own `caplog` plugin re-attaches its capture handler afterwards, + which is a harness artifact rather than a property of the process. + """ + root = logging.getLogger() + saved = root.handlers[:] + try: + root.handlers = [ + logging.StreamHandler(io.StringIO()), + logging.StreamHandler(io.StringIO()), + ] + handler = configure_logging("bot", log_format="json", stream=io.StringIO()) + assert root.handlers == [handler] + assert any(isinstance(f, SturnusFilter) for f in handler.filters) + finally: + root.handlers = saved + + +def test_debug_for_sturnus_does_not_turn_up_the_credential_loggers() -> None: + """The clamp is a floor no environment variable can undercut. + + `STURNUS_LOG_LEVEL=DEBUG` is a knob an operator will reach for during an + incident. On `botocore.auth` it prints the SigV4 signature, and on + `discord.ext.voice_recv.reader` it prints the Discord voice secret key + and raw packet bytes. + """ + root = logging.getLogger() + saved = root.handlers[:] + try: + configure_logging( + "bot", + level="DEBUG", + third_party_level="DEBUG", + log_format="json", + stream=io.StringIO(), + ) + assert logging.getLogger("sturnus").level == logging.DEBUG + for name, floor in NEVER_BELOW.items(): + assert logging.getLogger(name).level >= floor, name + finally: + root.handlers = saved + + +def test_the_aiohttp_access_log_is_clamped() -> None: + """`link`'s only route is `/oauth/callback?code=...`, and `%r` is `path_qs`.""" + assert NEVER_BELOW["aiohttp.access"] >= logging.WARNING + + +@pytest.mark.parametrize( + "logger_name", + [ + # `log.debug("CryptoError details:\n data=%s\n secret_key=%s", ...)` + "discord.ext.voice_recv.reader", + # `hook()` pretty-prints every voice-gateway payload except ops 3 + # and 6 at DEBUG, and op 4 (SESSION_DESCRIPTION) is the one that + # carries `secret_key`. A different logger, the same secret. + "discord.ext.voice_recv.gateway", + # The voice state update, which carries the voice `token`. + "discord.ext.voice_recv.voice_client", + ], +) +def test_no_logger_that_can_see_the_voice_secret_key_may_emit_debug(logger_name: str) -> None: + """The reported leak, closed at every route rather than at the one named. + + The report was `STURNUS_LOG_LEVEL=DEBUG` puts the Discord voice + `secret_key` into Loki. The mechanism it proposed -- redaction covers + Sturnus's own structured fields and can do nothing about a third-party + logger's message string -- is right, and is exactly why the fix is a + level floor rather than a scrubber: `NEVER_BELOW` is applied *after* + the environment's level, so no value of any variable reaches these + loggers' DEBUG. + + Parametrised over every logger in the installed packages that can see + the key or the voice token, not just the one the report named, because + closing one route and leaving the others open is indistinguishable from + closing none if the next release moves a log line. + """ + assert NEVER_BELOW[logger_name] >= logging.WARNING + + +def test_the_logger_the_report_named_is_shut_at_debug_and_open_at_info() -> None: + """`discord.voice_state`, which is a *level* decision rather than a list one. + + It is the logger the leak report named, and it is the one on these + lists whose two levels say different things: + + - **DEBUG** is connection-state transitions and DAVE upgrade notices. + No secret is formatted into any of them in the installed version, but + DEBUG is where the leak lives on every other logger here and a name + absent from `NEVER_BELOW` reads as "considered and cleared" when it + was never considered. It stays shut. + - **INFO** is the connect narrative -- handshake attempts, endpoint + found, timed out, close codes, resumed. That is the evidence base for + the capture-failure cooldown, all three entrypoints emitted it before + this package existed, and it carries no credential. + + Pinned by equality against two literals rather than by `>=` against + one, because both bounds are the point: `>= WARNING` would pass while + deleting the narrative, and `>= DEBUG` would pass while publishing the + payload. `tests/observability/test_third_party_log_floor.py` asserts + the same two claims on the rendered stream; this one pins the + declaration a reader edits. + """ + assert NEVER_BELOW["discord.voice_state"] == logging.INFO + assert NEVER_ABOVE["discord.voice_state"] == logging.INFO + + +def test_a_pin_survives_the_environment_in_both_directions() -> None: + """`NEVER_ABOVE` is a level, not an exemption from the floor. + + Both directions matter and they fail differently: without the pin the + default `WARNING` swallows the narrative, and without it being a fixed + level `STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` would reopen the payload. + """ + root = logging.getLogger() + saved = root.handlers[:] + saved_level = logging.getLogger("discord.voice_state").level + try: + for third_party in ("ERROR", "WARNING", "INFO", "DEBUG"): + configure_logging( + "bot", + level="DEBUG", + third_party_level=third_party, + log_format="json", + stream=io.StringIO(), + ) + assert logging.getLogger("discord.voice_state").level == logging.INFO, third_party + finally: + root.handlers = saved + logging.getLogger("discord.voice_state").setLevel(saved_level) + + +def test_an_unparseable_level_is_refused_rather_than_guessed() -> None: + with pytest.raises(ValueError, match="STURNUS_LOG_LEVEL"): + configure_logging("bot", level="verbose", stream=io.StringIO()) + + +@pytest.mark.usefixtures("stream") +def test_the_console_format_carries_the_same_redaction() -> None: + """Presentation differs; the filter does not. + + Takes the `stream` fixture only for its teardown, which restores the + root handlers this test replaces. + """ + root = logging.getLogger() + saved = root.handlers[:] + buffer = io.StringIO() + try: + configure_logging("bot", log_format="console", stream=buffer) + log_event( + logging.getLogger("sturnus.test"), + logging.INFO, + Event.SESSION_CLOSED, + "Session closed", + session_id=1, + transcript=TRANSCRIPT, + ) + rendered = buffer.getvalue() + assert "session.closed" in rendered + assert "session_id=1" in rendered + assert TRANSCRIPT not in rendered + finally: + root.handlers = saved + + +def test_an_uncaught_exception_is_routed_through_the_handler( + stream: io.StringIO, +) -> None: + """Otherwise it reaches stderr unformatted -- and Alloy scrapes stderr too.""" + import sys + + saved = sys.excepthook + try: + install_excepthooks() + try: + raise ValueError(f"boom {TRANSCRIPT}") + except ValueError as exc: + sys.excepthook(type(exc), exc, exc.__traceback__) + finally: + sys.excepthook = saved + + line = _event(stream, Event.UNHANDLED_EXCEPTION) + assert line["error_type"] == "ValueError" + assert TRANSCRIPT not in stream.getvalue() diff --git a/tests/observability/test_third_party_log_floor.py b/tests/observability/test_third_party_log_floor.py new file mode 100644 index 0000000..29525e7 --- /dev/null +++ b/tests/observability/test_third_party_log_floor.py @@ -0,0 +1,473 @@ +"""The voice `secret_key` must not reach Loki at any level an operator can set. + +The reported leak was: with `STURNUS_LOG_LEVEL=DEBUG`, the Discord voice +`secret_key` appears in the pod log and therefore in Loki. The mechanism is +one this package's redaction cannot touch. `redaction.scrub_fields` governs +*Sturnus's* structured fields; the key travels inside a third-party +logger's own `%s`-interpolated message string, produced by +`discord/ext/voice_recv/gateway.py`'s `hook()` on every voice connect. No +allowlist over our field names sees it. + +So every test here drives the **real** `configure_logging` and asserts on +the **rendered stream** -- the bytes `alloy-logs` would scrape -- rather +than on a level, a filter object or an entry in a dict. A configuration +assertion is exactly what the previous round of this fix consisted of, and +it passed while the leak was open: `NEVER_BELOW` named the right loggers, +and `root.setLevel(min(level, third_party))` set the root logger to DEBUG +underneath them, so every third-party logger the enumeration did *not* name +inherited DEBUG from root. + +Two properties are pinned, and they are different claims: + +1. **Suppression.** No logger outside `sturnus.*` may sit below + `THIRD_PARTY_FLOOR`, whatever the environment says -- asserted as a + property over `logging.Logger.manager.loggerDict`, not as another list + of names. A list is what failed. +2. **Redaction.** A rendered `secret_key: ...` in a message string is + scrubbed even at a level that *is* allowed, so a future library release + that moves the line to WARNING does not reopen the same hole. + +Where a level is asserted numerically it is written as a `logging.*` +literal, never as the constant under test: `assert +logger.getEffectiveLevel() >= THIRD_PARTY_FLOOR` would pass for every +possible value of `THIRD_PARTY_FLOOR`, including `DEBUG`, and would +therefore pin nothing at all. +""" + +from __future__ import annotations + +import io +import json +import logging +import os +from collections.abc import Iterator +from pprint import pformat + +# Importing the submodules is what puts their loggers into `loggerDict`; +# without this the sweep below would run over a dictionary that does not +# yet contain the loggers that matter, and pass by measuring nothing. +import discord # noqa: F401 +import discord.ext.voice_recv.gateway # noqa: F401 +import discord.ext.voice_recv.reader # noqa: F401 +import discord.ext.voice_recv.voice_client # noqa: F401 +import discord.http # noqa: F401 +import discord.state # noqa: F401 +import discord.voice_state # noqa: F401 +import pytest + +from sturnus.observability.events import Event +from sturnus.observability.setup import THIRD_PARTY_FLOOR, configure_logging + +#: Distinctive on purpose. The real key is 32 small integers, and `1`, `2`, +#: `12` are substrings of timestamps, byte counts and version numbers -- a +#: canary made of those would be indistinguishable from a false positive. +#: These are the same *shape* (a list of ints, pretty-printed) with values +#: no other part of a log line produces. +SECRET_KEY_CANARY = list(range(31337, 31337 + 32)) +SECRET_KEY_DIGITS = "31337" + +#: The voice `token` from a VOICE_STATE_UPDATE, which +#: `voice_recv/voice_client.py:53` pretty-prints at DEBUG next to the +#: `session_id`. +VOICE_TOKEN_CANARY = "CANARYVOICETOKEN-do-not-put-me-in-loki" + +#: `discord/http.py:707` logs the whole REST *response body* at DEBUG. Not +#: a key, but message content and display names -- the same class of +#: payload, through a logger no list in this repository names. +REST_BODY_CANARY = "CANARYRESTBODY-and-then-she-said-she-would-resign" + + +def session_description_payload() -> dict[str, object]: + """The `d` of voice op 4, in the shape `hook()` receives it. + + Op 4 is `SESSION_DESCRIPTION`, and its payload is where + `discord.gateway.load_secret_key` reads `data['secret_key']` from -- + so this is the literal dictionary that carries the key on every voice + connect, not a stand-in for it. + """ + return { + "dave_protocol_version": 0, + "mode": "aead_xchacha20_poly1305_rtpsize", + "secret_key": SECRET_KEY_CANARY, + "ssrc": 12345, + } + + +def drive_the_voice_handshake() -> None: + """The verbatim call shapes from the installed packages. + + Copied from site-packages rather than paraphrased, because the thing + under test is what those exact calls do to the rendered stream: + + - `discord/ext/voice_recv/gateway.py:57` + - `discord/ext/voice_recv/voice_client.py:53` + - `discord/voice_state.py` (connection-state transitions) + - `discord/http.py:707` + """ + logging.getLogger("discord.ext.voice_recv.gateway").debug( + "Received op %s: \n%s", 4, pformat(session_description_payload(), compact=True) + ) + logging.getLogger("discord.ext.voice_recv.voice_client").debug( + "Got voice_client VSU: \n%s", + pformat({"token": VOICE_TOKEN_CANARY, "session_id": "abc"}, compact=True), + ) + logging.getLogger("discord.voice_state").debug( + "Connection state changed to %s", session_description_payload() + ) + logging.getLogger("discord.gateway").debug( + "Voice websocket frame received: %s", session_description_payload() + ) + logging.getLogger("discord.http").debug( + "%s %s has received %s", + "GET", + "/channels/1/messages", + {"content": REST_BODY_CANARY}, + ) + # A logger no list in this repository names, and that does not exist + # until this line runs -- the case an enumeration is structurally + # unable to cover. + logging.getLogger("some_future_library.transport").debug( + "frame: %s", session_description_payload() + ) + + +@pytest.fixture +def restored_logging() -> Iterator[None]: + """Snapshot and restore every level `configure_logging` may move. + + It sets levels across the whole tree, so restoring the root handler + alone would leak DEBUG-suppressing levels into whatever test runs next. + """ + root = logging.getLogger() + saved_handlers = root.handlers[:] + saved_root_level = root.level + saved_levels = { + name: existing.level + for name, existing in logging.Logger.manager.loggerDict.items() + if isinstance(existing, logging.Logger) + } + yield + root.handlers = saved_handlers + root.setLevel(saved_root_level) + for name, level in saved_levels.items(): + existing = logging.Logger.manager.loggerDict.get(name) + if isinstance(existing, logging.Logger): + existing.setLevel(level) + + +def rendered(stream: io.StringIO) -> str: + return stream.getvalue() + + +def lines(stream: io.StringIO) -> list[dict[str, object]]: + return [json.loads(line) for line in stream.getvalue().splitlines() if line.strip()] + + +def assert_no_credential_reached_the_stream(stream: io.StringIO) -> None: + output = rendered(stream) + # Without this the whole assertion set passes vacuously on a stream + # nothing was ever written to. + assert output, "nothing was logged; the test would pass by writing nothing" + for what, canary in ( + ("the voice secret key", SECRET_KEY_DIGITS), + ("the voice token", VOICE_TOKEN_CANARY), + ("a REST response body", REST_BODY_CANARY), + ): + assert canary not in output, f"{what} reached the log stream" + for line in output.splitlines(): + if line.strip(): + json.loads(line) + + +def test_the_canaries_really_are_in_the_records_this_file_drives() -> None: + """The control on every "canary not in output" assertion below. + + Those assertions are absence checks, and an absence check passes just + as happily when it is looking for a string the test never planted. If + someone renames a canary, edits the payload, or a library changes the + shape of what it logs, this is what goes red instead of the whole file + going quietly green. + """ + payload = pformat(session_description_payload(), compact=True) + assert SECRET_KEY_DIGITS in payload, "the secret-key canary is not in the payload" + + vsu = pformat({"token": VOICE_TOKEN_CANARY, "session_id": "abc"}, compact=True) + assert VOICE_TOKEN_CANARY in vsu + + assert REST_BODY_CANARY in str({"content": REST_BODY_CANARY}) + + +@pytest.mark.usefixtures("restored_logging") +def test_sturnus_debug_puts_no_voice_secret_key_in_the_stream() -> None: + """The reported case: `STURNUS_LOG_LEVEL=DEBUG`, everything else default.""" + buffer = io.StringIO() + configure_logging("worker", level="DEBUG", log_format="json", stream=buffer) + + logging.getLogger("sturnus.test").debug("sturnus own debug line") + drive_the_voice_handshake() + + assert_no_credential_reached_the_stream(buffer) + + +@pytest.mark.usefixtures("restored_logging") +def test_sturnus_debug_survives_the_third_party_floor() -> None: + """The cost check. A fix that also silences Sturnus is not a fix. + + Raising the root logger to DEBUG was never what made Sturnus's own + DEBUG output visible -- `logging` checks a record against the + *originating* logger's effective level, never against root's during + propagation -- so leaving root at the third-party level costs nothing + here. This test is what says so. + """ + buffer = io.StringIO() + configure_logging("worker", level="DEBUG", log_format="json", stream=buffer) + + logging.getLogger("sturnus.application.worker").debug("a Sturnus debug line") + + assert any(line["msg"] == "a Sturnus debug line" for line in lines(buffer)), ( + "the third-party floor swallowed Sturnus's own DEBUG output" + ) + + +@pytest.mark.usefixtures("restored_logging") +def test_turning_the_third_party_knob_up_cannot_reopen_it() -> None: + """`STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` is the obvious way to try. + + It is the variable whose *name* says it turns third-party logging up, + it is settable from a Helm value, and before the floor it set the root + logger to DEBUG and published the key. + """ + buffer = io.StringIO() + configure_logging( + "worker", + level="DEBUG", + third_party_level="DEBUG", + log_format="json", + stream=buffer, + ) + + logging.getLogger("sturnus.test").debug("sturnus own debug line") + drive_the_voice_handshake() + + assert_no_credential_reached_the_stream(buffer) + + +@pytest.mark.usefixtures("restored_logging") +def test_the_environment_cannot_reopen_it_either(monkeypatch: pytest.MonkeyPatch) -> None: + """Through the environment, which is the only route an operator has. + + The arguments to `configure_logging` are a test convenience; in the + three entrypoints it is called with none of them and reads + `os.environ` itself. A fix proven only through the keyword arguments + would not be proven on the path a Helm value takes. + """ + monkeypatch.setitem(os.environ, "STURNUS_LOG_LEVEL", "DEBUG") + monkeypatch.setitem(os.environ, "STURNUS_LOG_THIRD_PARTY_LEVEL", "DEBUG") + + buffer = io.StringIO() + configure_logging("worker", log_format="json", stream=buffer) + + logging.getLogger("sturnus.test").debug("sturnus own debug line") + drive_the_voice_handshake() + + assert_no_credential_reached_the_stream(buffer) + + +@pytest.mark.usefixtures("restored_logging") +def test_a_logger_already_set_to_debug_is_raised_back_to_the_floor() -> None: + """A library that turns its own logger up at import time. + + Neither the root level nor `NEVER_BELOW` reaches such a logger: it + carries an explicit level, so it inherits nothing, and its name is by + definition not on a list written before it existed. + """ + logging.getLogger("some_future_library.transport").setLevel(logging.DEBUG) + + buffer = io.StringIO() + configure_logging("worker", level="DEBUG", log_format="json", stream=buffer) + + logging.getLogger("sturnus.test").debug("sturnus own debug line") + drive_the_voice_handshake() + + assert_no_credential_reached_the_stream(buffer) + + +@pytest.mark.usefixtures("restored_logging") +def test_no_logger_outside_sturnus_sits_below_info() -> None: + """The property, over every logger that exists -- not over a list. + + `logging.INFO` is written as a literal rather than as + `THIRD_PARTY_FLOOR`, deliberately. Comparing the levels this function + installed against the constant that installed them would hold for + every possible value of that constant, `DEBUG` included, and would + assert nothing. + """ + configure_logging( + "worker", + level="DEBUG", + third_party_level="DEBUG", + log_format="json", + stream=io.StringIO(), + ) + + offenders = sorted( + name + for name, existing in logging.Logger.manager.loggerDict.items() + if isinstance(existing, logging.Logger) + and not name.startswith("sturnus") + and existing.getEffectiveLevel() < logging.INFO + ) + assert not offenders, f"third-party loggers left below INFO: {offenders}" + # The sweep must have had something to sweep. Without this the + # assertion above passes on an empty dictionary. + assert "discord.ext.voice_recv.gateway" in logging.Logger.manager.loggerDict + + +@pytest.mark.usefixtures("restored_logging") +def test_the_voice_connect_narrative_survives_the_deployed_default() -> None: + """The half of `discord.voice_state` that is evidence, not leak. + + `voice_state.py` puts the whole connect narrative at **INFO** -- + "Starting voice handshake... (connection attempt %d)", "Voice handshake + complete. Endpoint found: %s", "Timed out connecting to voice", + "Disconnected from voice by discord, close code %d", "Successfully + resumed voice connection" -- and formats no secret into any of them + (read at the installed version: `secret_key` appears there only as an + attribute that is assigned and awaited). All three entrypoints emitted + those lines before this package existed, because they called + `basicConfig(level=INFO)`, and they are the evidence base for + `client.py`'s capture-failure cooldown and for telling apart the three + ways capture fails. + + The leak sits one level below, at DEBUG, and this asserts both halves + at once on the **rendered stream**: the INFO narrative is there and the + DEBUG payload is not. + + Driven at the levels a production `values.yaml` actually carries -- + `STURNUS_LOG_THIRD_PARTY_LEVEL` defaults to `WARNING` (see + `docs/operations.md` section 7.2), which is what makes a plain floor + entry of `INFO` insufficient on its own and `NEVER_ABOVE` load-bearing. + """ + buffer = io.StringIO() + configure_logging( + "bot", level="INFO", third_party_level="WARNING", log_format="json", stream=buffer + ) + + voice_state = logging.getLogger("discord.voice_state") + voice_state.info("Starting voice handshake... (connection attempt %d)", 2) + voice_state.info("Disconnected from voice by discord, close code %d.", 4014) + voice_state.debug("Connection state changed to %s", session_description_payload()) + + output = rendered(buffer) + assert "Starting voice handshake" in output, ( + "the connect narrative was clamped away; capture failures are undiagnosable" + ) + assert "close code 4014" in output + assert SECRET_KEY_DIGITS not in output, "the DEBUG payload is the leak and must not appear" + + +@pytest.mark.usefixtures("restored_logging") +def test_a_logger_pinned_open_is_still_closed_at_debug() -> None: + """`NEVER_ABOVE` opens a logger *to a level*, never to whatever is asked. + + The failure this guards against is an entry in `NEVER_ABOVE` being + read as "exempt from the floor". It is not: the level it names is + installed outright, so turning the third-party knob to DEBUG cannot + make a pinned logger any louder than its pin. + """ + buffer = io.StringIO() + configure_logging( + "bot", level="DEBUG", third_party_level="DEBUG", log_format="json", stream=buffer + ) + + assert logging.getLogger("discord.voice_state").getEffectiveLevel() == logging.INFO + drive_the_voice_handshake() + assert_no_credential_reached_the_stream(buffer) + + +def test_the_floor_is_info_and_the_reason_is_recorded() -> None: + """Pinned against a literal so a later edit is a deliberate one. + + WARNING would be quieter and would also delete `discord.voice_state`'s + INFO connect narrative -- "Starting voice handshake", "Voice handshake + complete", "Timed out connecting to voice" -- which is the entire + evidence base for diagnosing a capture failure. DEBUG is the leak. + INFO is the only level that is both. + """ + assert THIRD_PARTY_FLOOR == logging.INFO + + +@pytest.mark.usefixtures("restored_logging") +def test_an_operator_who_asked_for_more_is_told_they_did_not_get_it() -> None: + """Silently ignoring the knob would send them looking in the wrong place. + + Someone who sets `STURNUS_LOG_THIRD_PARTY_LEVEL=DEBUG` during an + incident and sees no new lines will conclude the variable is not + wired up, and go and edit the deployment instead of reading section + 7.2. One line in the log is what stops that hour. + """ + buffer = io.StringIO() + configure_logging("worker", third_party_level="DEBUG", log_format="json", stream=buffer) + + clamped = [line for line in lines(buffer) if line["event"] == str(Event.LOG_LEVEL_CLAMPED)] + assert len(clamped) == 1, f"expected exactly one clamp line, got {len(clamped)}" + assert clamped[0]["level"] == "WARNING" + assert clamped[0]["reason"] == "third_party_floor" + + +@pytest.mark.usefixtures("restored_logging") +def test_a_level_at_or_above_the_floor_is_not_reported_as_clamped() -> None: + """The line must mean something. A warning on every start means nothing.""" + buffer = io.StringIO() + configure_logging("worker", third_party_level="WARNING", log_format="json", stream=buffer) + + assert not [line for line in lines(buffer) if line["event"] == str(Event.LOG_LEVEL_CLAMPED)] + + +@pytest.mark.usefixtures("restored_logging") +def test_the_key_is_redacted_even_at_a_level_the_floor_permits() -> None: + """The second lock, which does not depend on a level at all. + + The floor stops the record being emitted. It cannot help if a future + `discord-ext-voice-recv` moves that same `pformat` call to WARNING or + logs it from an exception handler -- and the installed version already + has `log.info("WS payload has extra keys: %s", m)` a few lines below + it. So the rendered value is scrubbed as well as suppressed, and this + asserts the scrubbing on a record the floor deliberately lets through. + """ + buffer = io.StringIO() + configure_logging("worker", log_format="json", stream=buffer) + + logging.getLogger("discord.ext.voice_recv.gateway").warning( + "Received op %s: \n%s", 4, pformat(session_description_payload(), compact=True) + ) + + output = rendered(buffer) + assert output, "the record was suppressed, so this test measured nothing" + assert SECRET_KEY_DIGITS not in output + assert "«redacted:secret_value»" in output + # The rest of the payload is what makes the line worth keeping. + assert "aead_xchacha20_poly1305_rtpsize" in output + + +@pytest.mark.usefixtures("restored_logging") +def test_the_sweep_only_ever_raises_a_level() -> None: + """A library that quieted itself down keeps its own choice. + + The sweep exists to close a hole, not to normalise levels. `boto3` + calls `set_stream_logger` in some setups and libraries do turn + themselves *down* on import; overwriting that with the floor would be + this function making a noise decision it was never asked to make, and + quieter than the floor was never the problem. + """ + logging.getLogger("some_quiet_library").setLevel(logging.ERROR) + + configure_logging( + "worker", + level="DEBUG", + third_party_level="DEBUG", + log_format="json", + stream=io.StringIO(), + ) + + assert logging.getLogger("some_quiet_library").level == logging.ERROR diff --git a/tests/test_logging_discipline.py b/tests/test_logging_discipline.py new file mode 100644 index 0000000..7ec5262 --- /dev/null +++ b/tests/test_logging_discipline.py @@ -0,0 +1,292 @@ +"""Static rules that stop a payload reaching a log line, checked over `src/`. + +The runtime allowlist in `sturnus.observability.redaction` already drops an +unregistered field, so nothing here is the *only* thing standing between a +transcript and Loki. What these rules add is that a mistake fails the build +with a message naming the fix, instead of being silently dropped at runtime +and wondered about later -- and that the one thing the runtime allowlist +cannot police, the log *message* itself, stays a literal. + +That last point is load-bearing beyond this repository: +`sturnus.infrastructure.observability.scrub_event` forwards +`logentry.message` -- `LogRecord.msg`, the format string as written in the +source -- to Sentry and nothing else. That is only safe while `msg` is a +literal. `log.error(f"failed for {transcript}")` would put the transcript +*into* `msg`, and no scrubbing hook can tell that apart from a template. +ruff's `G001`-`G004` (see `pyproject.toml`) forbid the f-string, `%`, +`.format()` and `+` spellings; rule R1 below covers the rest. + +Modelled on `tests/test_architecture.py`, which already walks `src/` with +`ast` for exactly this kind of rule, and imports its name lists from +`sturnus.observability.fields` so the test and the runtime cannot drift. +""" + +from __future__ import annotations + +import ast +from pathlib import Path + +import pytest + +from sturnus.observability.fields import ALLOWED_FIELDS, DENIED_NAMES + +SRC = Path(__file__).parent.parent / "src" + +#: The module allowed to call `logging.basicConfig`. Anywhere else it would +#: install a second handler -- a second exit from the process, bypassing +#: `SturnusFilter` entirely. +_BASIC_CONFIG_OWNER = "setup.py" + +_LOG_METHODS = frozenset({"debug", "info", "warning", "error", "exception", "critical", "log"}) +_EVENT_HELPERS = frozenset({"log_event", "log_exception"}) + + +def _python_files() -> list[Path]: + return sorted(SRC.rglob("*.py")) + + +def _parse(path: Path) -> ast.Module: + return ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + + +def _is_log_call(node: ast.Call) -> bool: + """`log.info(...)` and friends, but not `logger.log_something_else`.""" + func = node.func + return ( + isinstance(func, ast.Attribute) + and func.attr in _LOG_METHODS + and isinstance(func.value, ast.Name) + and func.value.id in {"log", "logger", "logging"} + ) + + +def _is_event_call(node: ast.Call) -> bool: + func = node.func + return isinstance(func, ast.Name) and func.id in _EVENT_HELPERS + + +def _direct_name(node: ast.expr) -> str | None: + """The name of an expression passed *directly*, or `None`. + + Deliberately shallow. `len(body.encode("utf-8"))` is a `Call` and + returns `None`, because taking the length of a body is exactly what the + design asks call sites to do; `transcript` and `result.text` return + their names, because passing those *is* the mistake. Keeping the rule + shallow is what keeps its false-positive rate low enough that nobody + is tempted to delete it -- the failure mode a name-based denylist + actually dies of. + """ + if isinstance(node, ast.Name): + return node.id + if isinstance(node, ast.Attribute): + return node.attr + return None + + +def _message_argument(node: ast.Call) -> ast.expr | None: + """The human-readable message argument of a log or event call.""" + if _is_event_call(node): + # log_event(logger, level, event, message, ...) + # log_exception(logger, level, event, message, exc, ...) + return node.args[3] if len(node.args) > 3 else None + if isinstance(node.func, ast.Attribute) and node.func.attr == "log": + # log.log(level, message, ...) + return node.args[1] if len(node.args) > 1 else None + return node.args[0] if node.args else None + + +def _is_source_text(node: ast.expr) -> bool: + """Whether this expression can only ever evaluate to text written here. + + A plain literal, implicit or explicit concatenation of literals, and a + conditional choosing between two of them all qualify: every character + that can reach `LogRecord.msg` is visible in the source and reviewable. + Anything else -- a name, an f-string, a `.format()` -- does not. + """ + if isinstance(node, ast.Constant): + return isinstance(node.value, str) + if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Add): + return _is_source_text(node.left) and _is_source_text(node.right) + if isinstance(node, ast.IfExp): + # `"recorded nothing" if empty else "closed"` -- both branches are + # literals, so the set of possible messages is still a closed set + # written in this file. + return _is_source_text(node.body) and _is_source_text(node.orelse) + return False + + +def test_r1_every_log_message_is_a_string_literal() -> None: + """Rule R1. `logentry.message` is forwarded to Sentry; keep it source text.""" + violations: list[str] = [] + for path in _python_files(): + if path.name == "events.py": + # `log_event`/`log_exception` are the helpers whose `message` + # parameter this rule constrains at every *call* site; inside + # them it is necessarily a variable. + continue + for node in ast.walk(_parse(path)): + if not isinstance(node, ast.Call) or not (_is_log_call(node) or _is_event_call(node)): + continue + message = _message_argument(node) + if message is None: + continue + if not _is_source_text(message): + violations.append( + f"{path.relative_to(SRC)}:{node.lineno}: log message is not a string " + f"literal. Put the varying part in **fields instead." + ) + assert not violations, "\n".join(violations) + + +def test_r2_no_denied_name_is_passed_to_a_log_call() -> None: + """Rule R2. `log_event(..., transcript=result.text)` fails here, not silently.""" + violations: list[str] = [] + for path in _python_files(): + if path.name == "fields.py": + continue # the list itself + for node in ast.walk(_parse(path)): + if not isinstance(node, ast.Call) or not (_is_log_call(node) or _is_event_call(node)): + continue + for argument in node.args: + name = _direct_name(argument) + if name in DENIED_NAMES: + violations.append( + f"{path.relative_to(SRC)}:{node.lineno}: passes {name!r} to a log " + f"call. Log a count, a size or an id instead." + ) + for keyword in node.keywords: + if keyword.arg in DENIED_NAMES: + violations.append( + f"{path.relative_to(SRC)}:{node.lineno}: log field {keyword.arg!r} " + f"is on the denied list in sturnus.observability.fields." + ) + name = _direct_name(keyword.value) + if name in DENIED_NAMES: + violations.append( + f"{path.relative_to(SRC)}:{node.lineno}: log field " + f"{keyword.arg!r} is set from {name!r}, which carries payload." + ) + assert not violations, "\n".join(violations) + + +def test_r3_every_event_field_is_registered() -> None: + """Rule R3. The registry is the review point for "we decided to log this".""" + violations: list[str] = [] + for path in _python_files(): + for node in ast.walk(_parse(path)): + if not isinstance(node, ast.Call) or not _is_event_call(node): + continue + for keyword in node.keywords: + if keyword.arg is None: + violations.append( + f"{path.relative_to(SRC)}:{node.lineno}: **kwargs into a log " + f"event hides which fields are emitted." + ) + continue + if keyword.arg not in ALLOWED_FIELDS: + violations.append( + f"{path.relative_to(SRC)}:{node.lineno}: {keyword.arg!r} is not in " + f"ALLOWED_FIELDS. Add it to sturnus.observability.fields, " + f"deliberately, or rename the field." + ) + assert not violations, "\n".join(violations) + + +def test_r4_basic_config_lives_in_exactly_one_module() -> None: + """Rule R4. A second handler is a second, unfiltered way out of the process.""" + callers: list[str] = [] + for path in _python_files(): + for node in ast.walk(_parse(path)): + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "basicConfig" + ): + callers.append(str(path.relative_to(SRC))) + assert not callers, ( + "logging.basicConfig replaces nothing and appends an unfiltered handler; " + f"call sturnus.observability.setup.configure_logging instead. Found in: {callers}" + ) + + +def test_r5_nothing_in_src_calls_print() -> None: + """Rule R5. `print` writes to stdout without passing the filter at all.""" + violations: list[str] = [] + for path in _python_files(): + for node in ast.walk(_parse(path)): + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "print" + ): + violations.append(f"{path.relative_to(SRC)}:{node.lineno}") + assert not violations, ( + f"print() bypasses SturnusFilter entirely; use log_event. Found at: {violations}" + ) + + +def test_r6_an_exception_is_never_interpolated_into_a_log_message() -> None: + """Rule R6. `log.warning("failed: %s", exc)` prints `str(exc)` verbatim. + + Twelve call sites did exactly that before this branch, and one of them + covered `_create_session_document`, which renders the assembled + transcript through Jinja and posts it through httpx -- a + `jinja2.UndefinedError` there can carry template context, and `%s` would + print it. `log_exception` replaces the spelling: the type becomes a + registered field, the traceback is rendered from static program text, + and the message travels only if `SAFE_MESSAGE_TYPES` vouches for its + class. + """ + violations: list[str] = [] + for path in _python_files(): + for node in ast.walk(_parse(path)): + if not isinstance(node, ast.Call) or not _is_log_call(node): + continue + for argument in node.args[1:]: + name = _direct_name(argument) + if name in {"exc", "error", "exception", "err"}: + violations.append( + f"{path.relative_to(SRC)}:{node.lineno}: interpolates {name!r} " + f"into a log message. Use log_exception(...) instead." + ) + assert not violations, "\n".join(violations) + + +@pytest.mark.parametrize( + "source,rule_violated", + [ + ('log.info("transcribed %s", text)', "r2"), + ('log.info("transcribed %s", result.text)', "r2"), + ('log_event(log, 1, E.X, "done", transcript=body)', "r2"), + ('log_event(log, 1, E.X, "done", token=t)', "r2"), + ('log.warning("failed: %s", exc)', "r6"), + ('log_event(log, 1, E.X, "done", job_id=7)', None), + ('log_event(log, 1, E.X, "done", body_bytes=len(body))', None), + ('log.info("connected to %d guilds", count)', None), + ], +) +def test_the_rules_catch_what_they_claim_to(source: str, rule_violated: str | None) -> None: + """A table of spellings, so the rules are shown to work rather than trusted. + + Mirrors `test_import_resolution_comprehensive` in + `tests/test_architecture.py`: the point is that a future contributor can + read what is and is not allowed without reverse-engineering the walker. + """ + node = ast.parse(source).body[0] + assert isinstance(node, ast.Expr) + call = node.value + assert isinstance(call, ast.Call) + + r2 = any(_direct_name(a) in DENIED_NAMES for a in call.args) or any( + kw.arg in DENIED_NAMES or _direct_name(kw.value) in DENIED_NAMES for kw in call.keywords + ) + r6 = _is_log_call(call) and any( + _direct_name(a) in {"exc", "error", "exception", "err"} for a in call.args[1:] + ) + + if rule_violated == "r2": + assert r2, f"R2 should have flagged: {source}" + elif rule_violated == "r6": + assert r6, f"R6 should have flagged: {source}" + else: + assert not r2 and not r6, f"nothing should have flagged: {source}" diff --git a/uv.lock b/uv.lock index ad30ca2..ff07f77 100644 --- a/uv.lock +++ b/uv.lock @@ -950,6 +950,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279", size = 206583, upload-time = "2026-07-28T16:34:49.538Z" }, ] +[[package]] +name = "googleapis-common-protos" +version = "1.75.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/72/73/74bcab964c9a7a61f2bb71e8179b0f13e6fa98f7ce00fd168aab291e4a2e/googleapis_common_protos-1.75.1.tar.gz", hash = "sha256:d3042c6c5a2d4e67113104d6b6818b59b6bd92a197f2a91508e801fe815cf071", size = 150967, upload-time = "2026-08-06T06:24:51.972Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9a/51/186c02b8549b69ccda44429cf6ff5081e4b61a602ddfe6a8020d1be31d1b/googleapis_common_protos-1.75.1-py3-none-any.whl", hash = "sha256:28a1934bcd33b9c9da66ac301a0a4227e3367f095a17d0375cb98f0a09d93b79", size = 300626, upload-time = "2026-08-06T06:23:46.696Z" }, +] + [[package]] name = "greenlet" version = "3.5.5" @@ -1592,6 +1604,87 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/60/21/d0c04b561b46e9bff89b5f500fb7415b8ca0669f7902204f76ab06bb0c7e/onnxruntime-1.29.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:1ea91cef3b971506e51ae9c37c16d027774ec64994a524ec1bdfb027d68a9832", size = 23138547, upload-time = "2026-08-17T22:54:37.491Z" }, ] +[[package]] +name = "opentelemetry-api" +version = "1.44.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ee/8b/aa9e2d8b8dfa7c946f7dec5d1f8f6ba8eca062f43509a06bdb5ce93d26c0/opentelemetry_api-1.44.0.tar.gz", hash = "sha256:67647e5e9566edcf421166fdf022b3537f818635daa852b289e34604dc6fb33a", size = 72406, upload-time = "2026-07-16T15:25:32.678Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ca/6f/a04e900f465ff3221ccc395522503e2d10e79fa21f2723c8e177aae1e0d1/opentelemetry_api-1.44.0-py3-none-any.whl", hash = "sha256:94b98c893a91b88657eaac1e3ba89618cdb85be6918196705354f34728b2cdef", size = 60018, upload-time = "2026-07-16T15:25:11.657Z" }, +] + +[[package]] +name = "opentelemetry-exporter-otlp-proto-common" +version = "1.44.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "opentelemetry-proto" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/09/4d717852c1cf3f854b76c7110a5d00883bc3c99288b9b0dbcbeb9e306eb6/opentelemetry_exporter_otlp_proto_common-1.44.0.tar.gz", hash = "sha256:dc87a5a5bc58f149a56d1547e4691588fa12994cdc3bc039a694ccb3375862ac", size = 20202, upload-time = "2026-07-16T15:25:37.658Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/71/65fd9d54c10b860f87c045ccee1264cab7011268895d3528818a29c1172a/opentelemetry_exporter_otlp_proto_common-1.44.0-py3-none-any.whl", hash = "sha256:9a9fe61bba73d802904bc989f1d6b4a7b1ee40f06c40e98d6f85af65aaebb694", size = 17045, upload-time = "2026-07-16T15:25:18.201Z" }, +] + +[[package]] +name = "opentelemetry-exporter-otlp-proto-http" +version = "1.44.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "googleapis-common-protos" }, + { name = "opentelemetry-api" }, + { name = "opentelemetry-exporter-otlp-proto-common" }, + { name = "opentelemetry-proto" }, + { name = "opentelemetry-sdk" }, + { name = "requests" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/1a/87/95e2a5aaa795b4e2260d74e16df2d5541deb2ea9de010bcd615f4dee2654/opentelemetry_exporter_otlp_proto_http-1.44.0.tar.gz", hash = "sha256:c633d7270ad6b57cd4cfbe8b0007a9e2e7c0cb50bd6c50fe2a7b245f721a09d8", size = 25806, upload-time = "2026-07-16T15:25:39.162Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cd/d0/fdeb1a98d8d3a6205f5f297c51b4a9bfe65126ab60339669bbe3dd54c2e2/opentelemetry_exporter_otlp_proto_http-1.44.0-py3-none-any.whl", hash = "sha256:838592fce774c1c8bb7b9a0a7facbfa82e17be5a8a4e94cef10cb84ae026bae3", size = 21850, upload-time = "2026-07-16T15:25:20.006Z" }, +] + +[[package]] +name = "opentelemetry-proto" +version = "1.44.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/64/01/40ac4ae9a149263cc52c2cee200ddd80cb6d8db1a4610abf8eabce0fe771/opentelemetry_proto-1.44.0.tar.gz", hash = "sha256:c547a79c2f8c0c515d31509154682e5921c7cfd5ca67b70e1f9266e2c3e103f3", size = 46488, upload-time = "2026-07-16T15:25:45.34Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/7c/8be563d68e93bbefa5c8affb82ddcff91b3ad858ce49957ba7b16fd3e0ab/opentelemetry_proto-1.44.0-py3-none-any.whl", hash = "sha256:898b155a0e1557afd867478fb6158e8122a46329ca0bb8dc53cc55e98f017f56", size = 72483, upload-time = "2026-07-16T15:25:28.429Z" }, +] + +[[package]] +name = "opentelemetry-sdk" +version = "1.44.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "opentelemetry-api" }, + { name = "opentelemetry-semantic-conventions" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5d/77/a6592cbc7c8d9bcc9d6757a9df45e04a7c585e3e6e7a13456da522b21109/opentelemetry_sdk-1.44.0.tar.gz", hash = "sha256:cebe7f65dc12f26ead75c6064de12fd2a9052e5060c0272d402cfa203aae123b", size = 208624, upload-time = "2026-07-16T15:25:46.078Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/23/ff077e61886ee020a17ce9c8b6fa11c601c8d8345b09ea24f605445df62a/opentelemetry_sdk-1.44.0-py3-none-any.whl", hash = "sha256:df081c4c6bcfdb1211e3e86140376792643128a25f8d72d1d27675936e7e96ad", size = 137221, upload-time = "2026-07-16T15:25:29.534Z" }, +] + +[[package]] +name = "opentelemetry-semantic-conventions" +version = "0.65b0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "opentelemetry-api" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8f/73/0cbdebcb4cf545fdd328da14f5137e37d0770c3f26185e478b0d15d94f50/opentelemetry_semantic_conventions-0.65b0.tar.gz", hash = "sha256:f9b2b81e9d5b64f11bc952075e7e9c7fb0aab075c7fd1c46d597f1b919852d60", size = 148774, upload-time = "2026-07-16T15:25:46.902Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a6/0e/49df70d9b81fb5cbae4bbf2a49d865b09bcbcbc4eb53f5851b1027738d78/opentelemetry_semantic_conventions-0.65b0-py3-none-any.whl", hash = "sha256:1cacde7b0ad306f84c5ef08c3dbe1bbaf20165bba6f8bff43b670e555a086bcb", size = 204645, upload-time = "2026-07-16T15:25:30.688Z" }, +] + [[package]] name = "packaging" version = "26.3" @@ -2221,6 +2314,10 @@ dependencies = [ { name = "httpx" }, { name = "jinja2" }, { name = "numpy" }, + { name = "opentelemetry-api" }, + { name = "opentelemetry-exporter-otlp-proto-http" }, + { name = "opentelemetry-sdk" }, + { name = "opentelemetry-semantic-conventions" }, { name = "psycopg", extra = ["binary"] }, { name = "pydantic-settings" }, { name = "sentry-sdk" }, @@ -2253,6 +2350,10 @@ requires-dist = [ { name = "httpx", specifier = ">=0.28" }, { name = "jinja2", specifier = ">=3.1" }, { name = "numpy", specifier = ">=2.1" }, + { name = "opentelemetry-api", specifier = ">=1.44" }, + { name = "opentelemetry-exporter-otlp-proto-http", specifier = ">=1.44" }, + { name = "opentelemetry-sdk", specifier = ">=1.44" }, + { name = "opentelemetry-semantic-conventions", specifier = ">=0.65b0" }, { name = "psycopg", extras = ["binary"], specifier = ">=3.2" }, { name = "pydantic-settings", specifier = ">=2.12.0" }, { name = "sentry-sdk", specifier = ">=2.68" },