From 12805c8163bf8ad6e3a7b85ee0e3648d563b1b47 Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Thu, 20 Aug 2026 19:33:59 +0200 Subject: [PATCH 1/2] feat(worker): transcribe for the language and the vocabulary of the room Six changes to how a recording is turned into text, all of them aimed at the same thing: a protocol somebody can read afterwards and trust. **The language was the defect, not a setting.** `whisper_default_language` was `en` on a server whose meetings are held in German, and the fallback was only half the problem. With nothing pinned, the engine detected the language per speaker and `set_detected_language` wrote whatever came back -- for the rest of that speaker's session. Detection here is weak in exactly the way that matters: it runs on one speaker's track, which `vad_filter` has already cut down to the fragments where that person actually spoke, so a participant whose first contribution is a three-second "ja, genau" is close to a coin flip between several languages. One unlucky guess was not one bad job, it was every job for that person from then on. So `transcription_language` becomes per-guild configuration (Spec 11), resolved through `ConfigStore` on the path `document_target` and `merge_gap_seconds` already use, defaulting to `de`, with `STURNUS_WHISPER_DEFAULT_LANGUAGE` (now also `de`) left as the floor under it. The order between configuration and detection is the part worth arguing. A configured language wins outright: it is handed to the engine, no detection runs, and nothing is written to `detected_language`. A configured setting a guess may override is a trap, and here it would be a self-locking one -- the guess gets pinned and then beats the configuration on every later job of that session, while the column stops meaning "what the engine detected" and starts meaning "what was configured when this session's first job ran", with no way to tell the two apart in the data. Detection is still reachable, deliberately: `auto` asks for it, and is what a genuinely multilingual guild sets. Without that spelling the detect-and-pin path would have been unreachable in production, since clearing a key restores its default rather than removing it. **`initial_prompt`, carrying the organisation's vocabulary.** Proper nouns are both what a general model reliably gets wrong and what a protocol is read for -- a decision minuted about "Dracula" is not a decision about Ducula. `transcription_prompt` is per-guild for the same reason the language is, and defaults to OneLiteFeather's bird-named projects and their stack, written as an ordinary German sentence so the style it biases towards is punctuated prose as well as the right words. **`condition_on_previous_text=False`.** The library defaults it to `True`, which feeds each segment's own text back in as the next one's prompt, so one hallucination becomes the context every following segment is decoded against -- the cascade `compression_ratio_threshold` and `no_speech_threshold` exist to catch is what that produces. `vad_filter` makes the default worse rather than better here: it removes the silence, so the "previous text" is routinely from minutes earlier and about something else. Per-speaker recordings of a conversation are the case that default suits least. **`beam_size=8`**, over the library's 5. Beam cost is roughly linear in the width, and the trade is CPU seconds the worker has against a wrong word in a document people read instead of having been in the room. **`large-v3` instead of `large-v3-turbo`.** Turbo is a distilled decoder, four layers where large-v3 has thirty-two, and what it gives up is concentrated outside English -- the only place this deployment operates. It buys latency nobody is waiting for: transcription is offline, per speaker, hours after the meeting. **`int8_float32` instead of `int8`.** The weights are quantised to int8 either way; the suffix names the type everything else runs in. Bare `int8` is an alias whose float type CTranslate2 picks for whatever machine it finds itself on, which leaves transcription quality depending on the node the pod was scheduled to, and CTranslate2 falls back silently rather than refusing a compute type it cannot provide. The chart pays for the last two. `worker.resources` works the budget out rather than guessing: ~1.55GB of int8 weights (1.55B parameters against turbo's 809M), ~2.2GB of decoding buffers (cached attention keys and values are held per beam per decoder layer, and this has 32 decoder layers where turbo has 4 -- scaled from faster-whisper's own published CPU benchmark for this architecture at beam 5), ~0.5GB of process. 4Gi request, 6Gi limit; exceeding the limit is an OOMKill mid-transcription. The startup budget doubles with the download, which is ~3.1GB of float16 checkpoint against turbo's ~1.6GB, and nothing answers /healthz until the engine is built. Every parameter that reaches `WhisperModel` or `model.transcribe` is now pinned against a fake in `tests/infrastructure/test_whisper.py`. The existing tests there run real inference and cannot do this job: a wrong `beam_size` still transcribes a two-second fixture perfectly. A blank or whitespace-only `transcription_language` is read as "detect" rather than passed to the engine, which rejects it. `guild_config` is a table operators are told they may edit with SQL, where `ConfigStore.set`'s validation never runs, and one careless UPDATE should not fail every job of a guild. --- charts/sturnus/values.yaml | 61 +++++++- docs/first-deployment.md | 9 ++ docs/operations.md | 71 ++++++++- docs/verification/end-to-end-checklist.md | 5 +- src/sturnus/application/reconfigure.py | 5 +- src/sturnus/application/transcription.py | 12 +- src/sturnus/application/worker.py | 113 ++++++++++++-- src/sturnus/domain/settings.py | 37 +++++ src/sturnus/entrypoints/worker.py | 34 +++- src/sturnus/infrastructure/whisper.py | 47 +++++- tests/application/test_worker.py | 143 ++++++++++++++++- tests/entrypoints/test_worker_entrypoint.py | 60 ++++++- tests/infrastructure/test_config_store.py | 43 ++++++ tests/infrastructure/test_whisper.py | 163 +++++++++++++++++++- 14 files changed, 751 insertions(+), 52 deletions(-) diff --git a/charts/sturnus/values.yaml b/charts/sturnus/values.yaml index f9cba23..0c4e383 100644 --- a/charts/sturnus/values.yaml +++ b/charts/sturnus/values.yaml @@ -267,7 +267,13 @@ worker: replicaCount: 1 env: STURNUS_MODEL_CACHE_DIR: /data/model-cache - STURNUS_WHISPER_MODEL: large-v3-turbo + # Not `large-v3-turbo`: turbo is a distilled decoder, four layers where + # this has thirty-two, and the accuracy it trades away lands outside + # English -- which is the only place these meetings happen. It is a + # latency trade, and nothing here is waiting on latency: transcription + # is offline, per speaker, after the meeting. `worker.resources` below + # is where that decision is actually paid for. + STURNUS_WHISPER_MODEL: large-v3 # Larger than bot/link's: `process_one` downloads the encrypted object and # decrypts it to a WAV before transcribing, both under its work directory # (`sturnus.application.worker.process_one`, which creates a @@ -278,25 +284,60 @@ worker: # Setting STURNUS_WORK_DIR to anything outside this mount points the worker # at the read-only root filesystem, so the two have to move together. tmpSizeLimit: 4Gi + # Sized for large-v3 at int8_float32 with beam_size 8, which is a + # different budget from the large-v3-turbo/beam 5 these numbers were set + # for. Three parts, added up rather than guessed at: + # + # * The weights, resident for the life of the process. 1.55B + # parameters quantised to one byte each is ~1.55GB, against ~0.81GB + # for turbo's 809M. (The checkpoint on the cache volume below is + # float16 and twice that; it is quantised while loading, not kept.) + # * The decoding buffers, which exist only while a job runs and are + # the part that actually scales: the cached attention keys and + # values are held per beam, per decoder layer, and large-v3 has 32 + # decoder layers where turbo has 4. faster-whisper's own published + # CPU benchmark for this architecture (large-v2 -- same 1.55B + # parameters, same layer counts -- int8, beam 5) peaks at ~2.9GB + # all in, i.e. ~1.35GB on top of the weights. beam_size 8 scales + # that share by 8/5, so ~2.2GB. + # * The process itself -- interpreter, numpy, soxr, boto3, SQLAlchemy, + # one decrypted WAV read through it. A few hundred MB; call it 0.5GB. + # + # ~4.25GB at peak. The request is 4Gi (steady state between jobs is + # closer to 2GB -- the second item is transient), and the limit 6Gi. The + # limit is the number worth being generous with: exceeding it is an + # OOMKill in the middle of a transcription, which fails the job, and + # headroom on a node that has the memory costs nothing until it is + # needed. cpu stays at 4: large-v3 needs more *time* than turbo, not + # more parallelism, and the ceiling that time has to stay under is + # STURNUS_JOB_LEASE_SECONDS (1800s), not this. resources: requests: cpu: "4" - memory: 2Gi + memory: 4Gi limits: cpu: "4" - memory: 2560Mi - # The model (faster-whisper large-v3-turbo) downloads on first start on a - # cold cache volume, which takes minutes -- not seconds. A startupProbe - # gives that its own, generous budget: liveness and readiness are not + memory: 6Gi + # The model (faster-whisper large-v3) downloads on first start on a cold + # cache volume, which takes minutes -- not seconds. A startupProbe gives + # that its own, generous budget: liveness and readiness are not # evaluated at all until it succeeds, so a slow download cannot be mistaken # for a hung process and cannot trigger a restart loop. Once the startup # probe succeeds, liveness/readiness take over on their normal cadence. + # The budget is what it is because `sturnus.entrypoints.worker` builds + # the engine -- download, then quantise 1.55B parameters down to int8 -- + # *before* it starts the health server, so nothing answers /healthz until + # that has finished. startupProbe: httpGet: path: /healthz port: health periodSeconds: 10 - failureThreshold: 60 # 10 minutes to cover a cold-cache model download + # 20 minutes. Twice what it was, because large-v3's float16 checkpoint + # is ~3.1GB against large-v3-turbo's ~1.6GB and the quantisation at the + # end of it is longer too. This only ever costs anything on a cold + # cache volume: it is a ceiling, not a wait. + failureThreshold: 120 livenessProbe: httpGet: path: /healthz @@ -309,7 +350,11 @@ worker: periodSeconds: 10 persistence: # Holds the downloaded model weights so a restart doesn't repeat the - # download; ~1.6GB for large-v3-turbo in int8 (Spec 7) plus headroom. + # download. What lands here is the float16 checkpoint as published, + # ~3.1GB for large-v3 (Spec 7) -- the int8 quantisation happens in + # memory at load time and is never written back. Plenty of headroom at + # 10Gi, and worth keeping: a resize is far more disruptive than the + # unused gigabytes are expensive. size: 10Gi storageClassName: "" accessMode: ReadWriteOnce diff --git a/docs/first-deployment.md b/docs/first-deployment.md index 7084b96..7fe236c 100644 --- a/docs/first-deployment.md +++ b/docs/first-deployment.md @@ -201,6 +201,15 @@ In Discord, as an administrator: the target collection id, and `policy_url`, which must point at a real privacy policy naming the retention period. Participants consent to what that document says. + + Two more keys are worth a look here even though they default to + something sensible: `transcription_language` (default `de`) and + `transcription_prompt`, which defaults to OneLiteFeather's own project + names. Both decide what the protocol actually *says*, and both are + cheap to get right now and awkward to notice later — a wrong language + is pinned per speaker for a whole session, and a name the model has + never seen comes out as the nearest word it has. Section 4 of + `operations.md` explains both. 3. Have one person run `/consent grant` and `/link start` to confirm both flows end where they should. diff --git a/docs/operations.md b/docs/operations.md index 6a15964..23961e3 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -83,8 +83,8 @@ in the bot would read it. | `STURNUS_MASTER_KEY_ID` | **yes** | no | Label recorded as `encryption_key_id` when a data key is wrapped. Not key material. | | `STURNUS_OUTLINE_BASE_URL` | **yes** | no | Base URL of the Outline instance the finished protocol is posted to. | | `STURNUS_OUTLINE_SERVICE_KEY` | **yes** | **yes** | Outline API token `OutlineSink` authenticates with when creating documents. Note the name — it is `OUTLINE_SERVICE_KEY`, not an `API_TOKEN` variant. A token that is invalid, lacks access, or points at a collection that does not exist surfaces as `PermanentDocumentError`; see section 5. | -| `STURNUS_WHISPER_MODEL` | `large-v3-turbo` | no | faster-whisper model to load. Larger models are more accurate and markedly slower, and this deployment transcribes on CPU (see the chart's `worker.resources`), so the difference is measured in minutes per recording rather than seconds. | -| `STURNUS_WHISPER_DEFAULT_LANGUAGE` | `en` | no | Language reported when faster-whisper's own detection comes up empty. It matters more than a fallback usually does: the first transcription for a speaker in a session pins that speaker's language, and every later job for them reuses it. | +| `STURNUS_WHISPER_MODEL` | `large-v3` | no | faster-whisper model to load. Larger models are more accurate and markedly slower, and this deployment transcribes on CPU (see the chart's `worker.resources`), so the difference is measured in minutes per recording rather than seconds. It is deliberately not `large-v3-turbo`: turbo is a distilled decoder with four layers instead of thirty-two, and what it gives up is concentrated outside English. Transcription happens offline, per speaker, after the meeting, so the time it costs is time nobody is waiting on — the memory it costs is real, and the chart's `worker.resources` comment works it out. | +| `STURNUS_WHISPER_DEFAULT_LANGUAGE` | `de` | no | Language reported when faster-whisper's own detection comes up empty. This is the floor under the per-guild `transcription_language` (section 4.1), not the usual setting to reach for — it is consulted only for a guild that asked for detection (`transcription_language auto`) and got nothing back. It still matters more than a fallback usually does: the first transcription for a speaker in such a session pins that speaker's language, and every later job for them reuses it. | | `STURNUS_MODEL_CACHE_DIR` | unset | no | Where model weights are cached. When set, the worker exports it as `HF_HOME` before loading the model, so the download lands on a persistent volume; left unset, every cold start re-downloads several gigabytes of weights. | | `STURNUS_WORK_DIR` | `/tmp` | no | Scratch directory the encrypted recording is downloaded and decrypted into before transcription. It must be large enough for the biggest single recording — the chart sizes the corresponding volume with `worker.tmpSizeLimit`. | | `STURNUS_MAX_JOB_ATTEMPTS` | `3` | no | How many failed attempts a job gets before `JobQueue.fail` marks it `dead`. See section 5 for what a `dead` job means for the rest of its session. | @@ -94,11 +94,27 @@ in the bot would read it. | `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Value Sentry files events under in its environment filter. Ignored when no DSN is set. | Whisper's device and compute type are deliberately *not* environment-driven: -the worker constructs `WhisperEngine` with `"cpu"` and `int8` hardcoded, -because Spec 7 sizes this deployment for CPU inference. There is no -`STURNUS_WHISPER_DEVICE` to set — moving to GPU is a code change, not a +the worker constructs `WhisperEngine` with `"cpu"` and `int8_float32` +hardcoded, because Spec 7 sizes this deployment for CPU inference. There is +no `STURNUS_WHISPER_DEVICE` to set — moving to GPU is a code change, not a configuration change. +`int8_float32` rather than plain `int8`: the weights are quantised to int8 +either way, and the suffix names the type everything else runs in. +CTranslate2 treats bare `int8` as an alias and picks that float type for +whichever machine it finds itself on, which would leave transcription +quality depending on the node the pod was scheduled to. It also falls back +silently rather than refusing a compute type it cannot provide, so a wrong +value here costs accuracy with nothing in the logs to say so. + +Neither the decoding parameters (`beam_size`, `condition_on_previous_text`, +the VAD filter and the two hallucination thresholds) is configurable +either. They are quality decisions with one right answer for this workload, +argued in `sturnus/infrastructure/whisper.py` and pinned by +`tests/infrastructure/test_whisper.py`; what *is* per-guild is the language +and the vocabulary, and those are runtime configuration rather than +environment variables — see section 4.1. + ### 1.3 `sturnus-link` (`sturnus.entrypoints.link.LinkSettings`) | Variable | Required | Secret | Purpose | @@ -376,6 +392,40 @@ naming the guild, rather than costing the protocol. /config set timezone Europe/Berlin ``` +Worth setting for the same reason, and for a bigger one: `transcription_language` +decides what language the recordings are transcribed as. It defaults to +`de`. The alternative is not "no language" but detection, and detection is +weak exactly where it is used here — it runs on one speaker's track with +the silence already cut out of it, so a participant whose first +contribution is a three-second "ja, genau" gives it almost nothing to work +with. Whatever it guesses is then pinned for that speaker for the rest of +the session, so one unlucky guess is not one bad job, it is every job for +that person from then on. Naming the language removes the guess. + +``` +/config set transcription_language de +``` + +A guild that genuinely meets in more than one language sets it to `auto`, +which is what asks for detection-and-pinning explicitly. There is no third +state: clearing the key restores the `de` default rather than removing it. + +`transcription_prompt` is the vocabulary Whisper is biased towards while +decoding — Whisper's `initial_prompt`. It defaults to OneLiteFeather's own +project names and stack, written as an ordinary German sentence so the +style it biases towards is punctuated prose as well. Proper nouns are both +what a general model reliably gets wrong and what a protocol is read for: a +decision minuted about the wrong project is worse than no minutes. Set it +if your names are different ones: + +``` +/config set transcription_prompt "Protokoll eines Meetings über Foo, Bar und Baz." +``` + +Keep it a sentence rather than a word list, keep it in the transcription +language, and keep it short — Whisper only sees the last ~224 tokens of it, +and a long prompt bleeds its own wording into the transcript. + Until every required key (`voice_channel_id`, `consent_role_id`, `document_target`, `policy_version`, `policy_url`, `admin_role_id`) is set, the bot logs a warning naming the guild and skips building that guild's @@ -404,9 +454,14 @@ when it detects this. **Live immediately, and never were stale.** `admin_role_id`, `policy_version`, `policy_url` (read per command invocation, and by the -consent cache with a five-second TTL), and `document_target`, -`document_provider`, `merge_gap_seconds` (read per job by the *worker* -process, not the bot at all). +consent cache with a five-second TTL), and `transcription_language`, +`transcription_prompt`, `document_target`, `document_provider`, +`merge_gap_seconds` (read per job by the *worker* process, not the bot at +all). The two transcription keys apply to the next job the worker claims, +which means a session already recording is still transcribed with the new +value — and a job that has already run is not redone. Changing them +because a protocol came out wrong therefore affects the next meeting, not +the one you are looking at. **Deferred until the recording in progress ends.** `voice_channel_id` and `consent_role_id`. These decide which channel a session's row names and diff --git a/docs/verification/end-to-end-checklist.md b/docs/verification/end-to-end-checklist.md index fbf23fa..96b55e4 100644 --- a/docs/verification/end-to-end-checklist.md +++ b/docs/verification/end-to-end-checklist.md @@ -248,7 +248,10 @@ measured, not a plausibility check against the estimate. - [ ] CPU and memory for the bot pod under load: ____________ - [ ] CPU and memory for the worker pod under load (this is the one most likely to differ sharply from the spec's estimate, since transcription - is the heaviest step): ____________ + is the heaviest step). The chart's `worker.resources` comment works out + ~4.25GB at peak for large-v3 at int8_float32 with beam_size 8, on paper + and never yet measured — this is the measurement it is waiting for, and + the limit is 6Gi: ____________ - [ ] **[Plan 4]** CPU and memory for the link-service pod under load: ____________ - [ ] Actual recording size per speaker-hour (extrapolate from this diff --git a/src/sturnus/application/reconfigure.py b/src/sturnus/application/reconfigure.py index e1ffebf..22255e4 100644 --- a/src/sturnus/application/reconfigure.py +++ b/src/sturnus/application/reconfigure.py @@ -59,8 +59,9 @@ class GuildRuntimeConfig: """Everything about a guild that the bot process holds in memory. Deliberately *not* every configuration key: `admin_role_id`, - `policy_url`, `policy_version`, `document_target`, `merge_gap_seconds` - and `document_provider` are read per use (by a permission check, by the + `policy_url`, `policy_version`, `document_target`, `merge_gap_seconds`, + `document_provider`, `transcription_language` and + `transcription_prompt` are read per use (by a permission check, by the consent cache, or by the worker process entirely) and were never stale to begin with. Only what the bot caches needs reconciling. """ diff --git a/src/sturnus/application/transcription.py b/src/sturnus/application/transcription.py index cefb90b..5f39b33 100644 --- a/src/sturnus/application/transcription.py +++ b/src/sturnus/application/transcription.py @@ -30,11 +30,21 @@ class TranscriptionResult: class TranscriptionEngine(Protocol): - async def transcribe(self, path: Path, language: str | None) -> TranscriptionResult: + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: """Transcribe one speaker's recording. `language` pins the language; `None` asks the engine to detect it and report what it found. + + `initial_prompt` is vocabulary and style for the engine to lean + towards — an organisation's project names, the words a general + model has never seen and will otherwise replace with something it + has. It is deliberately a required argument rather than one with a + default: the guild's configured prompt (Spec 11) is worth nothing + if a call site can quietly leave it out, and a caller that really + has no vocabulary to offer says so by passing `None`. """ ... diff --git a/src/sturnus/application/worker.py b/src/sturnus/application/worker.py index 89d4f48..9264e83 100644 --- a/src/sturnus/application/worker.py +++ b/src/sturnus/application/worker.py @@ -13,10 +13,37 @@ transcription can be redone from the original audio; that deletion belongs to the retention sweep (`sturnus.application.retention`), not to this job. -Language pinning (Spec 7): a speaker's first job asks the engine to detect -the language and persists what it found; every later job for that same -speaker passes the stored language back in, so one protocol never mixes -languages mid-session because the engine's guess drifted. +Language (Spec 7, Spec 11). Two things can decide what language a +recording is transcribed as, and the order between them is the whole +point. `transcription_language` is per-guild configuration and wins +outright: when a guild names a language it is handed to the engine, no +detection runs, and *nothing* is written to `detected_language`. Both +halves of that matter. A configured setting that a guess may override is +a trap, and here it would be a self-locking one -- `set_detected_language` +pins the first job's guess for the rest of the session, so the guess would +go on beating the configuration on every later job of that session, and +the column would stop meaning "what the engine detected" and start meaning +"what was configured when this session's first job ran", with no way to +tell the two apart in the data. + +Detection remains available, and is what an unconfigured guild and a guild +that sets the value to `auto` (`sturnus.domain.settings.DETECT_LANGUAGE`) +both get: then, and only then, a speaker's first job asks the engine to +detect the language and persists what it found, and every later job for +that same speaker passes the stored language back in, so one protocol +never mixes languages mid-session because the engine's guess drifted. +That the guess needs pinning at all is the measure of how weak it is: it +is made on one speaker's track, which `vad_filter` has already reduced to +the fragments where that person actually spoke, so a participant whose +first contribution is a three-second agreement is close to a coin flip +between several languages -- and whichever one comes back then governs +every remaining job for them. + +`transcription_prompt` (Spec 11) is the vocabulary the engine is biased +towards while decoding -- an organisation's project names, which is +precisely what a general model has never seen and will replace with +something it has. It is read here, per job, for the same reason the +document settings below are. Dependency-rule note: this module lives in `sturnus.application`, which must never import `sturnus.infrastructure` (tests/test_architecture.py). Every @@ -46,10 +73,13 @@ `links` is typed with this module's own `LinkRepository`, not `assembly`'s `LinkReader`, and `config` (`ConfigReader`) is threaded through alongside -it: `document_target`, `document_provider`, and `merge_gap_seconds` are all -per-guild settings (Spec 11) that this one process cannot resolve until a -session -- and therefore its guild -- is in hand, so they are read inside -`_create_session_document` rather than once at process start. `_BoundLinks` +it: `transcription_language`, `transcription_prompt`, `document_target`, +`document_provider`, and `merge_gap_seconds` are all per-guild settings +(Spec 11) that this one process cannot resolve until a session -- and +therefore its guild -- is in hand. The first two are read in `process_one` +itself, just before the engine is called; the last three inside +`_create_session_document`. None of them is read once at process start, +because one worker serves every guild. `_BoundLinks` adapts one call's resolved provider back down to the plain `LinkReader` shape `assemble` itself calls, so `assemble` stays ignorant of configuration entirely. @@ -211,9 +241,11 @@ async def channel_ref(self, session_id: int) -> tuple[int, int, str | None]: async def guild_id(self, session_id: int) -> int: """The guild a session belongs to. - Needed to resolve per-guild configuration (`document_target`, - `document_provider`, `merge_gap_seconds`, Spec 11) at - document-creation time -- see `_create_session_document`. + Needed to resolve per-guild configuration (Spec 11) twice per job: + `transcription_language` and `transcription_prompt` before the + engine is called (`process_one`), and `document_target`, + `document_provider` and `merge_gap_seconds` when a session's last + job creates the document (`_create_session_document`). """ ... @@ -242,6 +274,32 @@ class _ClaimedJobShape(Protocol): wrapped_data_key: bytes +def _configured_language(configured: str | None) -> str | None: + """The language a guild named, or `None` when it asked for detection. + + Three stored values mean "detect", and the caller has no reason to + tell them apart: `auto` (`sturnus.domain.settings.DETECT_LANGUAGE`), + nothing at all, and blank. The last two are unreachable through + `/config` -- the key has a default and clearing restores it -- but + neither is unreachable in practice: `ConfigReader` is a protocol, and + `guild_config` is a table an operator is told they may edit with SQL + (`docs/operations.md` section 4.1), which `ConfigStore.set`'s + validation never sees. A blank value has to mean *something*, and the + alternative is passing `""` to the engine, which rejects it -- turning + one careless `UPDATE` into every job of that guild failing. + + Surrounding whitespace is stripped for the same reason: `" de "` is + not a language code faster-whisper knows, and a value typed with a + trailing space is not a decision to fail every job. + """ + if configured is None: + return None + named = configured.strip() + if not named or named.casefold() == domain_settings.DETECT_LANGUAGE: + return None + return named + + async def _guild_timezone(config: ConfigReader, guild: int) -> tzinfo: """The timezone the protocol's times are written in (Spec 11). @@ -372,7 +430,11 @@ async def process_one( 1. Claim -- nothing claimed means there is no work; the caller backs off. 2. Download the encrypted object to a scratch directory under `work_dir`. 3. Unwrap the data key and decrypt to a plaintext WAV, still on disk. - 4. Transcribe -- language pinning per Spec 7 (see the module docstring). + 4. Resolve the guild's `transcription_language` and + `transcription_prompt` (Spec 11), then transcribe -- configured + language first, detection and per-speaker pinning only when the + guild asked for it (Spec 7; see the module docstring for the order + and why it is that way round). 5. Store the transcript on the job; ask whether it was the session's last. 6. If it was: assemble every participant's stored transcript into one document (`_create_session_document`, `sturnus.application.assembly. @@ -428,14 +490,37 @@ async def process_one( job.encryption_key_id, ) - pinned_language = await sessions.detected_language(job.session_id, job.discord_user_id) + # Both settings are the guild's (Spec 11), so the guild has to + # be resolved first: one worker process serves all of them and + # only the session names one. Two extra reads per job, against + # a transcription measured in minutes. + guild = await sessions.guild_id(job.session_id) + configured_language = await config.get(guild, domain_settings.TRANSCRIPTION_LANGUAGE) + prompt = await config.get(guild, domain_settings.TRANSCRIPTION_PROMPT) + + # A configured language beats a stored detection outright, and + # the stored detection is not even read when there is one -- + # see this module's docstring for why that order is the point + # rather than a detail. + named_language = _configured_language(configured_language) + pinned_language = ( + named_language + if named_language is not None + else await sessions.detected_language(job.session_id, job.discord_user_id) + ) try: - result = await engine.transcribe(wav_path, pinned_language) + result = await engine.transcribe(wav_path, pinned_language, prompt) except Exception as exc: log.warning("Transcription failed for job %d", job.id) await queue.fail(job.id, str(exc), max_attempts) return True + # Reached only when the guild asked for detection *and* this is + # the first job for this speaker: a named language is never + # `None`, which is exactly what keeps configuration out of + # `detected_language`. Dropping the condition altogether would + # write the configured language into that column on every job + # and pin it there, which is the trap the docstring describes. if pinned_language is None: await sessions.set_detected_language( job.session_id, job.discord_user_id, result.language diff --git a/src/sturnus/domain/settings.py b/src/sturnus/domain/settings.py index 976d3c2..ae5cdc1 100644 --- a/src/sturnus/domain/settings.py +++ b/src/sturnus/domain/settings.py @@ -16,6 +16,17 @@ ADMIN_ROLE_ID = "admin_role_id" MERGE_GAP_SECONDS = "merge_gap_seconds" TIMEZONE = "timezone" +TRANSCRIPTION_LANGUAGE = "transcription_language" +TRANSCRIPTION_PROMPT = "transcription_prompt" + +#: The one value of `TRANSCRIPTION_LANGUAGE` that is not a language: it +#: asks the engine to detect one per speaker and pin what it found for the +#: rest of the session, which is what the worker did unconditionally before +#: this key existed. It exists because `DEFAULTS` below names a language, +#: and `/config clear` restores a default rather than removing it -- without +#: a spelling for "detect", a guild that really does meet in several +#: languages would have no way back to detection at all. +DETECT_LANGUAGE = "auto" DEFAULTS: dict[str, str] = { EMPTY_GRACE_SECONDS: "60", @@ -30,6 +41,32 @@ # not obviously wrong to a reader -- 15:08 looks like a plausible # meeting time whether or not it is the right one. TIMEZONE: "Europe/Berlin", + # Naming the language is worth far more than it looks. The + # alternative is detection, which runs on one speaker's track after + # the silence has been cut out of it, and a speaker whose first + # contribution is "ja, genau" gives it almost nothing to go on -- + # German, Dutch and Danish are all plausible readings of three + # seconds of that. Whatever comes back is then pinned for that + # speaker for the rest of the session, so a single unlucky guess is + # not one bad job, it is every job for that person from then on. + # German is what this deployment's guilds actually meet in; `auto` + # (see `DETECT_LANGUAGE` above) is how one that does not says so. + TRANSCRIPTION_LANGUAGE: "de", + # Whisper's `initial_prompt` biases the decoder towards the + # vocabulary and the style of this text. Proper nouns are where a + # general model fails and where a protocol is judged: a model that + # has never seen "Ducula" will confidently write the nearest word it + # has seen, and a decision recorded about the wrong project is worse + # than no minutes at all. The default is OneLiteFeather's own + # vocabulary -- the bird-named projects and the stack they are built + # on -- written as an ordinary German sentence so the style it biases + # towards is punctuated prose in the language above rather than a + # bare word list. + TRANSCRIPTION_PROMPT: ( + "Protokoll eines OneLiteFeather-Meetings über die Projekte Falco, Otis, " + "Ducula, Pica, Guira, Aves, Sturnus, Cygnus, Apus und Coris sowie über " + "Minestom, Paper, Outline, Harbor, Flux, Kubernetes, Renovate und Gradle." + ), } # No default value, so these must be set before going live. diff --git a/src/sturnus/entrypoints/worker.py b/src/sturnus/entrypoints/worker.py index 4973c77..090b9a0 100644 --- a/src/sturnus/entrypoints/worker.py +++ b/src/sturnus/entrypoints/worker.py @@ -112,9 +112,21 @@ _DOCUMENT_RETRY_INTERVAL_SECONDS = 300.0 #: faster-whisper on CPU (Spec 7 sizes the deployment for CPU, not GPU -- -#: see `charts/sturnus/values.yaml`'s `worker.resources`): int8 is the -#: quantisation the chart's own model-size comment already assumes. -_WHISPER_COMPUTE_TYPE = "int8" +#: see `charts/sturnus/values.yaml`'s `worker.resources`). The weights are +#: still quantised to int8, which is what keeps `large-v3` inside a +#: memory budget a CPU node will actually give it; the `_float32` half +#: names the type everything *else* runs in -- activations, accumulation, +#: the layers that are never quantised. +#: +#: Naming it is the point. Plain `int8` is an alias whose float type +#: CTranslate2 picks for the device it finds itself on, so what the +#: decoder accumulates in is decided by the node the pod landed on rather +#: than by this file; on today's x86 workers the alias resolves to exactly +#: this, and on a machine with bfloat16 support it need not. Transcription +#: quality is not something to leave to the scheduler, and CTranslate2 +#: falls back silently rather than refusing a compute type it cannot +#: provide -- there would be nothing in the logs to say it had happened. +_WHISPER_COMPUTE_TYPE = "int8_float32" #: Package and resource name of the real Outline document template. See #: `_load_template` -- this is the packaged template `process_one` must @@ -161,8 +173,20 @@ class WorkerSettings(StrictSettings): master_key_id: str outline_base_url: str outline_service_key: SecretStr - whisper_model: str = "large-v3-turbo" - whisper_default_language: str = "en" + # `large-v3` rather than `large-v3-turbo`: turbo is a distillation with + # four decoder layers instead of thirty-two, and what it gives up is + # concentrated outside English -- which is the only place this + # deployment operates. It buys speed, and nothing here is waiting: + # transcription runs offline, one speaker's file at a time, after the + # meeting is over. The cost is paid in the chart instead, in memory + # and in a longer first start (`charts/sturnus/values.yaml`). + whisper_model: str = "large-v3" + # The floor under `transcription_language` (Spec 11), which is + # per-guild and normally decides this; reached only when a guild asked + # for detection and the engine's detection came back with nothing. + # `en` here was a real defect, not a harmless default: every guild + # this serves meets in German. + whisper_default_language: str = "de" model_cache_dir: Path | None = None work_dir: Path = Path("/tmp") max_job_attempts: int = 3 diff --git a/src/sturnus/infrastructure/whisper.py b/src/sturnus/infrastructure/whisper.py index d10e4d1..9b5d060 100644 --- a/src/sturnus/infrastructure/whisper.py +++ b/src/sturnus/infrastructure/whisper.py @@ -3,6 +3,14 @@ The library is synchronous and CPU-bound, so every call runs in a worker thread. The model is loaded once and reused; jobs are processed one at a time (Spec 5.3), so no locking is required around it. + +Every decoding parameter below is set explicitly rather than left to the +library's default, and each one is set against a specific way a meeting +protocol goes wrong: silence turning into invented speech, one bad segment +poisoning the segments after it, a project name coming out as a common +word. `tests/infrastructure/test_whisper.py` pins each of them against a +fake model, because none of them is visible in the output of a passing +two-second fixture. """ from __future__ import annotations @@ -29,13 +37,27 @@ def __init__( self._model = WhisperModel(model_size, device=device, compute_type=compute_type) self._default_language = default_language - async def transcribe(self, path: Path, language: str | None) -> TranscriptionResult: - return await asyncio.to_thread(self._transcribe, path, language) + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: + return await asyncio.to_thread(self._transcribe, path, language, initial_prompt) - def _transcribe(self, path: Path, language: str | None) -> TranscriptionResult: + def _transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: segments, info = self._model.transcribe( str(path), language=language, + # Biases the decoder towards the vocabulary and the style of + # this text. It is the only lever Sturnus has on proper nouns, + # and proper nouns are both what Whisper reliably gets wrong + # and what a protocol is read for: a decision about "Ducula" + # is unusable when the sentence says "Dracula". Per-guild + # configuration (`transcription_prompt`, Spec 11) rather than + # a constant here -- the vocabulary that matters is the + # organisation's, and this adapter has no idea whose meeting + # it is transcribing. + initial_prompt=initial_prompt, # Skips the padded silence, which is most of a speaker's file and # would otherwise cost real time and invite hallucinated text. vad_filter=True, @@ -43,6 +65,25 @@ def _transcribe(self, path: Path, language: str | None) -> TranscriptionResult: # long audio (Spec 7). compression_ratio_threshold=2.4, no_speech_threshold=0.6, + # The library defaults this to `True`, which feeds each + # segment's own text back in as the prompt for the next one. + # One hallucinated segment then becomes the context every + # following segment is decoded against, and the cascade the + # two thresholds above exist to catch is exactly what that + # produces. `vad_filter` makes the default worse here rather + # than better: it cuts one speaker's track into fragments with + # every silence removed, so the "previous text" is routinely + # from minutes earlier and about something else entirely -- + # per-speaker recordings of a conversation are the case this + # default is least suited to. + condition_on_previous_text=False, + # Above the library's default of 5. Beam search cost is + # roughly linear in the width and this deployment transcribes + # offline, one speaker's file at a time, hours after the + # meeting -- so the trade is CPU seconds (which the worker has, + # see `charts/sturnus/values.yaml`) against a wrong word in a + # document people read instead of having been in the room. + beam_size=8, ) collected = tuple( TranscribedSegment(start=s.start, end=s.end, text=s.text) for s in segments diff --git a/tests/application/test_worker.py b/tests/application/test_worker.py index 2a1e91d..07da1e9 100644 --- a/tests/application/test_worker.py +++ b/tests/application/test_worker.py @@ -50,17 +50,34 @@ async def fail(self, job_id: int, error: str, _max_attempts: int) -> None: class FakeEngine: - def __init__(self, text: str = "spoken words", fail: bool = False) -> None: + """`detected` is what the engine *reports back*, which is not the same + as what it was asked for: it is deliberately different from the + language the transcription tests configure, so a test asserting that a + configured language is never overwritten by detection cannot pass by + the two happening to agree. + """ + + def __init__( + self, text: str = "spoken words", fail: bool = False, detected: str = "de" + ) -> None: self.text = text self.fail = fail + self.detected = detected self.calls: list[tuple[Path, str | None]] = [] - - async def transcribe(self, path: Path, language: str | None) -> TranscriptionResult: + #: `initial_prompt` from every call, recorded separately from + #: `calls` so the existing `calls[i][1]` language assertions stay + #: as they are. + self.prompts: list[str | None] = [] + + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: self.calls.append((path, language)) + self.prompts.append(initial_prompt) if self.fail: raise RuntimeError("model exploded") return TranscriptionResult( - segments=(TranscribedSegment(0.0, 1.0, self.text),), language="de" + segments=(TranscribedSegment(0.0, 1.0, self.text),), language=self.detected ) @@ -215,6 +232,23 @@ async def get(self, guild_id: int, key: str) -> str | None: return self._values.get((guild_id, key)) +def guild_config(extra: dict[str, str] | None = None) -> FakeConfig: + """A `FakeConfig` for `GUILD` whose document settings are already right. + + The transcription keys (`transcription_language`, `transcription_prompt`) + are absent unless a test names them, which is what keeps the + unconfigured path -- detect once, then pin per speaker -- exercised by + every test that does not care about them. + """ + values: dict[tuple[int, str], str] = { + (GUILD, domain_settings.DOCUMENT_TARGET): "col-default", + (GUILD, domain_settings.DOCUMENT_PROVIDER): "outline", + (GUILD, domain_settings.MERGE_GAP_SECONDS): "15", + } + values.update({(GUILD, key): value for key, value in (extra or {}).items()}) + return FakeConfig(values) + + def job(job_id: int = 1, session_id: int = 1, user_id: int = 100) -> ClaimedJob: return ClaimedJob( id=job_id, @@ -548,6 +582,107 @@ async def test_a_later_job_pins_the_stored_language(tmp_path: Path) -> None: assert engine.calls[0][1] == "de" +async def test_the_guilds_configured_language_is_what_gets_transcribed( + tmp_path: Path, +) -> None: + """`transcription_language` (Spec 11) is passed straight to the engine. + + Naming the language is what stops the engine detecting one, and + detection on a per-speaker track is a coin flip whenever the speaker's + first job is short. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "de"}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.calls[0][1] == "de" + + +async def test_a_configured_language_is_never_pinned_as_a_detection( + tmp_path: Path, +) -> None: + """Writing the configured value into `detected_language` would make the + column mean two different things and would freeze the configuration as + it stood on a session's first job: a guild that corrects the setting + mid-session would keep getting the old language until the session ends. + """ + engine, sessions = FakeEngine(detected="nl"), FakeSessions() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "de"}) + await process_one(**run(tmp_path, engine=engine, sessions=sessions, config=config)) + assert sessions.languages == {} + + +async def test_a_configured_language_wins_over_an_earlier_detection( + tmp_path: Path, +) -> None: + """The stored detection is a guess; the configured value is a decision. + + This is the ordering that stops one bad detection -- pinned by an + earlier job of the same session, before an administrator noticed and + configured the language -- from governing every job after it. + """ + engine, sessions = FakeEngine(), FakeSessions() + sessions.languages[100] = "nl" + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "de"}) + await process_one(**run(tmp_path, engine=engine, sessions=sessions, config=config)) + assert engine.calls[0][1] == "de" + + +async def test_auto_asks_for_detection_and_pins_what_came_back(tmp_path: Path) -> None: + """`auto` is how a genuinely multilingual guild opts back in. + + Without it the detect-and-pin path would be unreachable in production, + since `transcription_language` has a default and clearing the key + restores it rather than removing it. + """ + engine, sessions = FakeEngine(detected="nl"), FakeSessions() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "auto"}) + await process_one(**run(tmp_path, engine=engine, sessions=sessions, config=config)) + assert engine.calls[0][1] is None + assert sessions.languages[100] == "nl" + + +async def test_a_blank_configured_language_asks_for_detection(tmp_path: Path) -> None: + """`guild_config` is a table operators are told they may edit with SQL, + which `ConfigStore.set`'s validation never sees. A blank value has to + mean something, and the alternative -- handing `""` to the engine, + which rejects it -- turns one careless `UPDATE` into every job of that + guild failing. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: " "}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.calls[0][1] is None + + +async def test_a_configured_language_is_stripped_before_the_engine_sees_it( + tmp_path: Path, +) -> None: + """`" de "` is not a language code faster-whisper knows, and a value + typed with a trailing space is not a decision to fail every job. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: " de "}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.calls[0][1] == "de" + + +async def test_the_guilds_vocabulary_prompt_reaches_the_engine(tmp_path: Path) -> None: + """`transcription_prompt` (Spec 11) is per-guild for the same reason + `document_target` is: one worker process serves every guild, and whose + project names matter is not knowable until a session is in hand. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_PROMPT: "Ducula, Guira, Minestom."}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.prompts == ["Ducula, Guira, Minestom."] + + +async def test_a_guild_with_no_prompt_configured_biases_nothing(tmp_path: Path) -> None: + engine = FakeEngine() + await process_one(**run(tmp_path, engine=engine, config=guild_config())) + assert engine.prompts == [None] + + async def test_the_audio_object_is_not_deleted_after_transcription(tmp_path: Path) -> None: """Audio outlives its transcription (Spec 12); the retention sweep deletes it.""" store = FakeStore() diff --git a/tests/entrypoints/test_worker_entrypoint.py b/tests/entrypoints/test_worker_entrypoint.py index 01ee628..d364633 100644 --- a/tests/entrypoints/test_worker_entrypoint.py +++ b/tests/entrypoints/test_worker_entrypoint.py @@ -12,8 +12,14 @@ mention-less fallback. """ +import ctranslate2 # type: ignore[import-untyped] + from sturnus.application.worker import _FALLBACK_TEMPLATE -from sturnus.entrypoints.worker import _load_template +from sturnus.entrypoints.worker import ( + _WHISPER_COMPUTE_TYPE, + WorkerSettings, + _load_template, +) def test_loaded_template_is_not_the_fallback() -> None: @@ -22,3 +28,55 @@ def test_loaded_template_is_not_the_fallback() -> None: def test_loaded_template_renders_outline_mentions() -> None: assert "mention://" in _load_template() + + +def _settings(**overrides: object) -> WorkerSettings: + """`WorkerSettings` with every required field supplied as a literal. + + Passed as arguments rather than through `monkeypatch.setenv` so the + defaults under test are read from the class, not from whatever the + machine running the suite happens to export. + """ + required: dict[str, object] = { + "database_url": "postgresql+asyncpg://u:p@db/sturnus", + "s3_endpoint": "http://s3.invalid", + "s3_bucket": "sturnus-audio", + "s3_access_key": "access", + "s3_secret_key": "secret", + "master_key": "a" * 44, + "master_key_id": "k1", + "outline_base_url": "https://outline.invalid", + "outline_service_key": "token", + } + return WorkerSettings(**{**required, **overrides}) # type: ignore[arg-type] + + +def test_the_default_model_is_the_undistilled_large_one() -> None: + """`large-v3-turbo` is a distilled decoder -- four layers where + `large-v3` has thirty-two -- and the accuracy it gives up is not evenly + spread: it shows up outside English, which is the only place this + deployment operates. Nothing here is latency-sensitive (transcription + happens offline, per speaker, after the meeting), so the distillation + buys time nobody is waiting for. + """ + assert _settings().whisper_model == "large-v3" + + +def test_the_default_language_is_the_one_these_meetings_are_held_in() -> None: + """This is the value that decides what an unconfigured guild gets when + the engine's own detection comes up empty, and `en` on a + German-speaking server was simply wrong. Per-guild configuration + (`transcription_language`, Spec 11) overrides it; this is the floor + under it. + """ + assert _settings().whisper_default_language == "de" + + +def test_the_configured_quantisation_is_one_this_cpu_can_actually_run() -> None: + """CTranslate2 does not refuse a compute type it cannot provide -- it + quietly falls back to one it can, so a wrong value here costs accuracy + or speed with nothing in the logs to say so. `int8_float32` names the + activation type outright instead of leaving it to the `int8` alias, + whose float type CTranslate2 chooses per device. + """ + assert _WHISPER_COMPUTE_TYPE in ctranslate2.get_supported_compute_types("cpu") diff --git a/tests/infrastructure/test_config_store.py b/tests/infrastructure/test_config_store.py index 6f9c65a..56cd243 100644 --- a/tests/infrastructure/test_config_store.py +++ b/tests/infrastructure/test_config_store.py @@ -111,3 +111,46 @@ async def test_snapshot_agrees_with_get_for_every_known_key(store: ConfigStore) async def test_snapshot_is_per_guild(store: ConfigStore) -> None: await store.set(GUILD, settings.VOICE_CHANNEL_ID, "12345", T0) assert settings.VOICE_CHANNEL_ID not in await store.snapshot(GUILD + 1) + + +async def test_the_transcription_language_is_a_key_an_administrator_can_set( + store: ConfigStore, +) -> None: + """`set` rejects every key it does not know, so a setting the worker + reads but `DEFAULTS` never names is one nobody can change: `/config + set` answers "unknown configuration key" while the worker goes on + using the built-in value forever. + """ + await store.set(GUILD, settings.TRANSCRIPTION_LANGUAGE, "en", T0) + assert await store.get(GUILD, settings.TRANSCRIPTION_LANGUAGE) == "en" + + +async def test_clearing_the_transcription_language_restores_german(store: ConfigStore) -> None: + """Clearing restores the default rather than removing the value, which + is why `auto` exists as a spelling (`settings.DETECT_LANGUAGE`) -- there + is no state in which no language is configured. + """ + await store.set(GUILD, settings.TRANSCRIPTION_LANGUAGE, "en", T0) + await store.set(GUILD, settings.TRANSCRIPTION_LANGUAGE, None, T0) + assert await store.get(GUILD, settings.TRANSCRIPTION_LANGUAGE) == "de" + + +async def test_a_guild_that_configured_nothing_still_gets_the_projects_vocabulary( + store: ConfigStore, +) -> None: + """The prompt is only worth having if it carries the names Whisper + actually gets wrong, and a guild that never runs `/config set` is the + normal case -- so the default has to be the real vocabulary, not an + empty string waiting for someone to fill it in. + """ + prompt = await store.get(GUILD, settings.TRANSCRIPTION_PROMPT) + assert prompt is not None + for name in ("Ducula", "Guira", "Minestom", "Outline"): + assert name in prompt + + +async def test_the_transcription_prompt_is_a_key_an_administrator_can_set( + store: ConfigStore, +) -> None: + await store.set(GUILD, settings.TRANSCRIPTION_PROMPT, "Nur eigene Wörter.", T0) + assert await store.get(GUILD, settings.TRANSCRIPTION_PROMPT) == "Nur eigene Wörter." diff --git a/tests/infrastructure/test_whisper.py b/tests/infrastructure/test_whisper.py index ced3134..9727f0c 100644 --- a/tests/infrastructure/test_whisper.py +++ b/tests/infrastructure/test_whisper.py @@ -1,4 +1,20 @@ +"""The faster-whisper adapter, from two sides. + +The `slow` tests at the bottom run real inference and are the only proof +that the adapter produces usable text at all; they download a model, so +they are deselected on pull requests (see `pyproject.toml`'s marker) and +cannot be where a decoding parameter is pinned -- a wrong `beam_size` still +transcribes "hello" perfectly. + +Everything above them therefore drives a fake `WhisperModel` and asserts on +what actually reaches the library. Every one of those parameters is a +quality decision that is invisible in the output of a two-second fixture +and expensive in a real meeting, so each is pinned here rather than trusted +to survive a refactor. +""" + from pathlib import Path +from typing import Any import pytest @@ -7,9 +23,146 @@ FIXTURE = Path(__file__).parent.parent / "fixtures" / "hello.wav" +class _FakeInfo: + """The half of faster-whisper's `(segments, info)` the adapter reads.""" + + def __init__(self, language: str | None) -> None: + self.language = language + + +class _ModelSpy: + """Stands in for `faster_whisper.WhisperModel`, as both class and instance. + + Patched over the name `sturnus.infrastructure.whisper` imported, so + calling it is the construction `WhisperEngine.__init__` performs and + the object it hands back is this same spy -- which keeps the recorded + constructor arguments and the recorded `transcribe` arguments in one + place a test can read. + """ + + def __init__(self) -> None: + self.construction: dict[str, Any] = {} + self.transcription: dict[str, Any] = {} + #: What `info.language` reports; `None` stands for detection that + #: came up empty, which is what the adapter's own default is for. + self.detected: str | None = "de" + + def __call__(self, model_size: str, **kwargs: Any) -> "_ModelSpy": + self.construction = {"model_size": model_size, **kwargs} + return self + + def transcribe(self, path: str, **kwargs: Any) -> tuple[Any, _FakeInfo]: + self.transcription = {"path": path, **kwargs} + return iter(()), _FakeInfo(self.detected) + + +@pytest.fixture +def spy(monkeypatch: pytest.MonkeyPatch) -> _ModelSpy: + model = _ModelSpy() + monkeypatch.setattr("sturnus.infrastructure.whisper.WhisperModel", model) + return model + + +def _engine(compute_type: str = "int8_float32", default_language: str = "de") -> WhisperEngine: + return WhisperEngine( + model_size="large-v3", + device="cpu", + compute_type=compute_type, + default_language=default_language, + ) + + +async def test_the_model_is_built_with_the_quantisation_it_was_given(spy: _ModelSpy) -> None: + """`compute_type` is chosen in `sturnus.entrypoints.worker`, not here. + + It has to arrive at the library unaltered: the adapter has no business + substituting a quantisation, and a silently-dropped one would degrade + every transcription while every test still passed. + """ + _engine() + assert spy.construction == { + "model_size": "large-v3", + "device": "cpu", + "compute_type": "int8_float32", + } + + +async def test_the_vocabulary_prompt_reaches_the_decoder(spy: _ModelSpy) -> None: + """`initial_prompt` is the only lever on proper nouns Sturnus has. + + Project names -- Ducula, Guira, Minestom -- are exactly what Whisper + guesses wrong and exactly what a meeting protocol is read for. Losing + the prompt on the way to the library costs nothing visible and every + name in the document. + """ + await _engine().transcribe(FIXTURE, "de", "Ducula, Guira, Minestom.") + assert spy.transcription["initial_prompt"] == "Ducula, Guira, Minestom." + + +async def test_no_prompt_is_still_a_call_the_library_understands(spy: _ModelSpy) -> None: + """A guild may have no vocabulary worth biasing towards; `None` is that.""" + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["initial_prompt"] is None + + +async def test_a_segment_never_conditions_the_next_one(spy: _ModelSpy) -> None: + """faster-whisper defaults `condition_on_previous_text` to `True`. + + That feeds each segment's text back in as the next one's prompt, so a + single hallucination becomes the context every following segment is + decoded against and the repetition runs away. `vad_filter` (below) + makes it worse rather than better here: one speaker's track is cut + into disconnected fragments with the silence removed, so the + "previous text" a segment gets conditioned on is frequently from + minutes earlier and unrelated. + """ + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["condition_on_previous_text"] is False + + +async def test_the_beam_is_wider_than_the_library_default(spy: _ModelSpy) -> None: + """Costs CPU, which this deployment has, and buys accuracy it does not.""" + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["beam_size"] == 8 + + +async def test_the_hallucination_guards_are_still_in_place(spy: _ModelSpy) -> None: + """The three parameters that keep silence and repetition out (Spec 7). + + Pinned here because they are invisible in a passing test suite: + dropping any of them produces a worker that transcribes perfectly well + in every test and invents speech for a participant who never said a + word in production. + """ + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["vad_filter"] is True + assert spy.transcription["compression_ratio_threshold"] == 2.4 + assert spy.transcription["no_speech_threshold"] == 0.6 + + +async def test_the_pinned_language_reaches_the_library(spy: _ModelSpy) -> None: + """Passing a language is what stops faster-whisper detecting one at all.""" + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["language"] == "de" + + +async def test_detection_is_asked_for_when_no_language_is_pinned(spy: _ModelSpy) -> None: + await _engine().transcribe(FIXTURE, None, None) + assert spy.transcription["language"] is None + + +async def test_a_detection_that_came_up_empty_falls_back_to_the_default(spy: _ModelSpy) -> None: + """`TranscriptionResult.language` is stored and reused for the whole + session (`sturnus.application.worker`), so it may never be `None`. + """ + spy.detected = None + result = await _engine(default_language="de").transcribe(FIXTURE, None, None) + assert result.language == "de" + + @pytest.fixture(scope="module") def engine() -> WhisperEngine: - # `tiny` keeps the test fast; production uses large-v3-turbo (Spec 7). + # `tiny` keeps the test fast; production uses large-v3 (Spec 7). return WhisperEngine( model_size="tiny", device="cpu", compute_type="int8", default_language="de" ) @@ -17,21 +170,21 @@ def engine() -> WhisperEngine: @pytest.mark.slow async def test_transcribes_real_speech(engine: WhisperEngine) -> None: - result = await engine.transcribe(FIXTURE, language="de") + result = await engine.transcribe(FIXTURE, language="de", initial_prompt=None) assert result.segments assert any(segment.text.strip() for segment in result.segments) @pytest.mark.slow async def test_offsets_are_within_the_recording(engine: WhisperEngine) -> None: - result = await engine.transcribe(FIXTURE, language="de") + result = await engine.transcribe(FIXTURE, language="de", initial_prompt=None) for segment in result.segments: assert 0.0 <= segment.start <= segment.end @pytest.mark.slow async def test_detection_reports_a_language(engine: WhisperEngine) -> None: - result = await engine.transcribe(FIXTURE, language=None) + result = await engine.transcribe(FIXTURE, language=None, initial_prompt=None) assert result.language @@ -51,5 +204,5 @@ async def test_silence_yields_no_segments(engine: WhisperEngine, tmp_path: Path) w.setframerate(16_000) w.writeframes(b"\x00" * 16_000 * 3) - result = await engine.transcribe(silent, language="de") + result = await engine.transcribe(silent, language="de", initial_prompt=None) assert [s for s in result.segments if s.text.strip()] == [] From cd8a86b20737e568500782cc3702b3ca7ccc335c Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Thu, 20 Aug 2026 19:50:46 +0200 Subject: [PATCH 2/2] docs(chart): replace the memory arithmetic with a measurement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The resource figures were derived rather than observed, which is the right way to arrive at a first number and the wrong way to leave one. Measured on an x86 CPU pinned to four threads, transcribing 4.7 minutes of German speech: large-v3-turbo int8 beam 5 61.6s 1.96GB large-v3 int8_float32 beam 8 145.0s 4.11GB 4.11GB against the 4.25GB the comment worked out, so the arithmetic holds and the 4Gi request / 6Gi limit stand as chosen. The throughput number is the more useful of the two, because it bounds something the comment could only gesture at: 282s of speech in 145s is 1.94x faster than real time, against 4.6x before. Since `vad_filter` decodes only the speech in a track rather than its padded length, the 1800s job lease covers roughly 58 minutes of one person actually talking. Past that the lease expires mid-job -- harmless at `replicas: 1` with the worker taking one job at a time, since nothing else can claim it, and the reason raising the worker beyond one replica means revisiting the lease first. The same run is also the clearest evidence for the model change itself. The old configuration produced "Der Ender läuft inzwischen auf dem Kluster, aber die Speicherlimits sind noch zu knapp gemessen"; the new one produced "Der Renderer läuft inzwischen auf dem Cluster, aber die Speicherlimits sind noch zu knapp bemessen". --- charts/sturnus/values.yaml | 22 +++++++++++++++++++++- docs/verification/end-to-end-checklist.md | 8 ++++---- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/charts/sturnus/values.yaml b/charts/sturnus/values.yaml index 0c4e383..2f8e590 100644 --- a/charts/sturnus/values.yaml +++ b/charts/sturnus/values.yaml @@ -303,7 +303,14 @@ worker: # * The process itself -- interpreter, numpy, soxr, boto3, SQLAlchemy, # one decrypted WAV read through it. A few hundred MB; call it 0.5GB. # - # ~4.25GB at peak. The request is 4Gi (steady state between jobs is + # ~4.25GB at peak, and measured at **4.11GB** on an x86 CPU pinned to + # four threads, transcribing 4.7 minutes of German speech -- close enough + # to the arithmetic above to trust it for longer jobs, since only the + # second item grows with the audio. For comparison the previous + # configuration (large-v3-turbo, int8, beam 5) peaked at 1.96GB, which is + # why the old 2560Mi limit held. + # + # The request is 4Gi (steady state between jobs is # closer to 2GB -- the second item is transient), and the limit 6Gi. The # limit is the number worth being generous with: exceeding it is an # OOMKill in the middle of a transcription, which fails the job, and @@ -311,6 +318,19 @@ worker: # needed. cpu stays at 4: large-v3 needs more *time* than turbo, not # more parallelism, and the ceiling that time has to stay under is # STURNUS_JOB_LEASE_SECONDS (1800s), not this. + # + # Measured, that ceiling is further away than it looks. The same run + # transcribed 282s of speech in 145s -- 1.94x faster than real time, + # against 4.6x for the old configuration. `vad_filter` means only the + # *speech* in a track is decoded, not its padded length, so 1800s of + # lease covers roughly 58 minutes of one person actually talking. Beyond + # that the lease expires mid-job; with `replicas: 1` (hardcoded in the + # Deployment template) and the worker processing one job at a time, + # nothing else can claim it, so the only consequence today is that a + # restart may reclaim a job that was still running. Raising the worker + # beyond one replica is what would turn this into two workers + # transcribing the same recording, and is the moment to revisit the + # lease. resources: requests: cpu: "4" diff --git a/docs/verification/end-to-end-checklist.md b/docs/verification/end-to-end-checklist.md index 96b55e4..eee1007 100644 --- a/docs/verification/end-to-end-checklist.md +++ b/docs/verification/end-to-end-checklist.md @@ -248,10 +248,10 @@ measured, not a plausibility check against the estimate. - [ ] CPU and memory for the bot pod under load: ____________ - [ ] CPU and memory for the worker pod under load (this is the one most likely to differ sharply from the spec's estimate, since transcription - is the heaviest step). The chart's `worker.resources` comment works out - ~4.25GB at peak for large-v3 at int8_float32 with beam_size 8, on paper - and never yet measured — this is the measurement it is waiting for, and - the limit is 6Gi: ____________ + is the heaviest step). Expect ~4.1GB at peak for large-v3 at + int8_float32 with beam_size 8 — measured off-cluster on four CPU threads, + so this is the confirmation that it holds on a worker node, not a first + look. The limit is 6Gi: ____________ - [ ] **[Plan 4]** CPU and memory for the link-service pod under load: ____________ - [ ] Actual recording size per speaker-hour (extrapolate from this