diff --git a/charts/sturnus/values.yaml b/charts/sturnus/values.yaml index f9cba23..2f8e590 100644 --- a/charts/sturnus/values.yaml +++ b/charts/sturnus/values.yaml @@ -267,7 +267,13 @@ worker: replicaCount: 1 env: STURNUS_MODEL_CACHE_DIR: /data/model-cache - STURNUS_WHISPER_MODEL: large-v3-turbo + # Not `large-v3-turbo`: turbo is a distilled decoder, four layers where + # this has thirty-two, and the accuracy it trades away lands outside + # English -- which is the only place these meetings happen. It is a + # latency trade, and nothing here is waiting on latency: transcription + # is offline, per speaker, after the meeting. `worker.resources` below + # is where that decision is actually paid for. + STURNUS_WHISPER_MODEL: large-v3 # Larger than bot/link's: `process_one` downloads the encrypted object and # decrypts it to a WAV before transcribing, both under its work directory # (`sturnus.application.worker.process_one`, which creates a @@ -278,25 +284,80 @@ worker: # Setting STURNUS_WORK_DIR to anything outside this mount points the worker # at the read-only root filesystem, so the two have to move together. tmpSizeLimit: 4Gi + # Sized for large-v3 at int8_float32 with beam_size 8, which is a + # different budget from the large-v3-turbo/beam 5 these numbers were set + # for. Three parts, added up rather than guessed at: + # + # * The weights, resident for the life of the process. 1.55B + # parameters quantised to one byte each is ~1.55GB, against ~0.81GB + # for turbo's 809M. (The checkpoint on the cache volume below is + # float16 and twice that; it is quantised while loading, not kept.) + # * The decoding buffers, which exist only while a job runs and are + # the part that actually scales: the cached attention keys and + # values are held per beam, per decoder layer, and large-v3 has 32 + # decoder layers where turbo has 4. faster-whisper's own published + # CPU benchmark for this architecture (large-v2 -- same 1.55B + # parameters, same layer counts -- int8, beam 5) peaks at ~2.9GB + # all in, i.e. ~1.35GB on top of the weights. beam_size 8 scales + # that share by 8/5, so ~2.2GB. + # * The process itself -- interpreter, numpy, soxr, boto3, SQLAlchemy, + # one decrypted WAV read through it. A few hundred MB; call it 0.5GB. + # + # ~4.25GB at peak, and measured at **4.11GB** on an x86 CPU pinned to + # four threads, transcribing 4.7 minutes of German speech -- close enough + # to the arithmetic above to trust it for longer jobs, since only the + # second item grows with the audio. For comparison the previous + # configuration (large-v3-turbo, int8, beam 5) peaked at 1.96GB, which is + # why the old 2560Mi limit held. + # + # The request is 4Gi (steady state between jobs is + # closer to 2GB -- the second item is transient), and the limit 6Gi. The + # limit is the number worth being generous with: exceeding it is an + # OOMKill in the middle of a transcription, which fails the job, and + # headroom on a node that has the memory costs nothing until it is + # needed. cpu stays at 4: large-v3 needs more *time* than turbo, not + # more parallelism, and the ceiling that time has to stay under is + # STURNUS_JOB_LEASE_SECONDS (1800s), not this. + # + # Measured, that ceiling is further away than it looks. The same run + # transcribed 282s of speech in 145s -- 1.94x faster than real time, + # against 4.6x for the old configuration. `vad_filter` means only the + # *speech* in a track is decoded, not its padded length, so 1800s of + # lease covers roughly 58 minutes of one person actually talking. Beyond + # that the lease expires mid-job; with `replicas: 1` (hardcoded in the + # Deployment template) and the worker processing one job at a time, + # nothing else can claim it, so the only consequence today is that a + # restart may reclaim a job that was still running. Raising the worker + # beyond one replica is what would turn this into two workers + # transcribing the same recording, and is the moment to revisit the + # lease. resources: requests: cpu: "4" - memory: 2Gi + memory: 4Gi limits: cpu: "4" - memory: 2560Mi - # The model (faster-whisper large-v3-turbo) downloads on first start on a - # cold cache volume, which takes minutes -- not seconds. A startupProbe - # gives that its own, generous budget: liveness and readiness are not + memory: 6Gi + # The model (faster-whisper large-v3) downloads on first start on a cold + # cache volume, which takes minutes -- not seconds. A startupProbe gives + # that its own, generous budget: liveness and readiness are not # evaluated at all until it succeeds, so a slow download cannot be mistaken # for a hung process and cannot trigger a restart loop. Once the startup # probe succeeds, liveness/readiness take over on their normal cadence. + # The budget is what it is because `sturnus.entrypoints.worker` builds + # the engine -- download, then quantise 1.55B parameters down to int8 -- + # *before* it starts the health server, so nothing answers /healthz until + # that has finished. startupProbe: httpGet: path: /healthz port: health periodSeconds: 10 - failureThreshold: 60 # 10 minutes to cover a cold-cache model download + # 20 minutes. Twice what it was, because large-v3's float16 checkpoint + # is ~3.1GB against large-v3-turbo's ~1.6GB and the quantisation at the + # end of it is longer too. This only ever costs anything on a cold + # cache volume: it is a ceiling, not a wait. + failureThreshold: 120 livenessProbe: httpGet: path: /healthz @@ -309,7 +370,11 @@ worker: periodSeconds: 10 persistence: # Holds the downloaded model weights so a restart doesn't repeat the - # download; ~1.6GB for large-v3-turbo in int8 (Spec 7) plus headroom. + # download. What lands here is the float16 checkpoint as published, + # ~3.1GB for large-v3 (Spec 7) -- the int8 quantisation happens in + # memory at load time and is never written back. Plenty of headroom at + # 10Gi, and worth keeping: a resize is far more disruptive than the + # unused gigabytes are expensive. size: 10Gi storageClassName: "" accessMode: ReadWriteOnce diff --git a/docs/first-deployment.md b/docs/first-deployment.md index 7084b96..7fe236c 100644 --- a/docs/first-deployment.md +++ b/docs/first-deployment.md @@ -201,6 +201,15 @@ In Discord, as an administrator: the target collection id, and `policy_url`, which must point at a real privacy policy naming the retention period. Participants consent to what that document says. + + Two more keys are worth a look here even though they default to + something sensible: `transcription_language` (default `de`) and + `transcription_prompt`, which defaults to OneLiteFeather's own project + names. Both decide what the protocol actually *says*, and both are + cheap to get right now and awkward to notice later — a wrong language + is pinned per speaker for a whole session, and a name the model has + never seen comes out as the nearest word it has. Section 4 of + `operations.md` explains both. 3. Have one person run `/consent grant` and `/link start` to confirm both flows end where they should. diff --git a/docs/operations.md b/docs/operations.md index 6a15964..23961e3 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -83,8 +83,8 @@ in the bot would read it. | `STURNUS_MASTER_KEY_ID` | **yes** | no | Label recorded as `encryption_key_id` when a data key is wrapped. Not key material. | | `STURNUS_OUTLINE_BASE_URL` | **yes** | no | Base URL of the Outline instance the finished protocol is posted to. | | `STURNUS_OUTLINE_SERVICE_KEY` | **yes** | **yes** | Outline API token `OutlineSink` authenticates with when creating documents. Note the name — it is `OUTLINE_SERVICE_KEY`, not an `API_TOKEN` variant. A token that is invalid, lacks access, or points at a collection that does not exist surfaces as `PermanentDocumentError`; see section 5. | -| `STURNUS_WHISPER_MODEL` | `large-v3-turbo` | no | faster-whisper model to load. Larger models are more accurate and markedly slower, and this deployment transcribes on CPU (see the chart's `worker.resources`), so the difference is measured in minutes per recording rather than seconds. | -| `STURNUS_WHISPER_DEFAULT_LANGUAGE` | `en` | no | Language reported when faster-whisper's own detection comes up empty. It matters more than a fallback usually does: the first transcription for a speaker in a session pins that speaker's language, and every later job for them reuses it. | +| `STURNUS_WHISPER_MODEL` | `large-v3` | no | faster-whisper model to load. Larger models are more accurate and markedly slower, and this deployment transcribes on CPU (see the chart's `worker.resources`), so the difference is measured in minutes per recording rather than seconds. It is deliberately not `large-v3-turbo`: turbo is a distilled decoder with four layers instead of thirty-two, and what it gives up is concentrated outside English. Transcription happens offline, per speaker, after the meeting, so the time it costs is time nobody is waiting on — the memory it costs is real, and the chart's `worker.resources` comment works it out. | +| `STURNUS_WHISPER_DEFAULT_LANGUAGE` | `de` | no | Language reported when faster-whisper's own detection comes up empty. This is the floor under the per-guild `transcription_language` (section 4.1), not the usual setting to reach for — it is consulted only for a guild that asked for detection (`transcription_language auto`) and got nothing back. It still matters more than a fallback usually does: the first transcription for a speaker in such a session pins that speaker's language, and every later job for them reuses it. | | `STURNUS_MODEL_CACHE_DIR` | unset | no | Where model weights are cached. When set, the worker exports it as `HF_HOME` before loading the model, so the download lands on a persistent volume; left unset, every cold start re-downloads several gigabytes of weights. | | `STURNUS_WORK_DIR` | `/tmp` | no | Scratch directory the encrypted recording is downloaded and decrypted into before transcription. It must be large enough for the biggest single recording — the chart sizes the corresponding volume with `worker.tmpSizeLimit`. | | `STURNUS_MAX_JOB_ATTEMPTS` | `3` | no | How many failed attempts a job gets before `JobQueue.fail` marks it `dead`. See section 5 for what a `dead` job means for the rest of its session. | @@ -94,11 +94,27 @@ in the bot would read it. | `STURNUS_SENTRY_ENVIRONMENT` | `production` | no | Value Sentry files events under in its environment filter. Ignored when no DSN is set. | Whisper's device and compute type are deliberately *not* environment-driven: -the worker constructs `WhisperEngine` with `"cpu"` and `int8` hardcoded, -because Spec 7 sizes this deployment for CPU inference. There is no -`STURNUS_WHISPER_DEVICE` to set — moving to GPU is a code change, not a +the worker constructs `WhisperEngine` with `"cpu"` and `int8_float32` +hardcoded, because Spec 7 sizes this deployment for CPU inference. There is +no `STURNUS_WHISPER_DEVICE` to set — moving to GPU is a code change, not a configuration change. +`int8_float32` rather than plain `int8`: the weights are quantised to int8 +either way, and the suffix names the type everything else runs in. +CTranslate2 treats bare `int8` as an alias and picks that float type for +whichever machine it finds itself on, which would leave transcription +quality depending on the node the pod was scheduled to. It also falls back +silently rather than refusing a compute type it cannot provide, so a wrong +value here costs accuracy with nothing in the logs to say so. + +Neither the decoding parameters (`beam_size`, `condition_on_previous_text`, +the VAD filter and the two hallucination thresholds) is configurable +either. They are quality decisions with one right answer for this workload, +argued in `sturnus/infrastructure/whisper.py` and pinned by +`tests/infrastructure/test_whisper.py`; what *is* per-guild is the language +and the vocabulary, and those are runtime configuration rather than +environment variables — see section 4.1. + ### 1.3 `sturnus-link` (`sturnus.entrypoints.link.LinkSettings`) | Variable | Required | Secret | Purpose | @@ -376,6 +392,40 @@ naming the guild, rather than costing the protocol. /config set timezone Europe/Berlin ``` +Worth setting for the same reason, and for a bigger one: `transcription_language` +decides what language the recordings are transcribed as. It defaults to +`de`. The alternative is not "no language" but detection, and detection is +weak exactly where it is used here — it runs on one speaker's track with +the silence already cut out of it, so a participant whose first +contribution is a three-second "ja, genau" gives it almost nothing to work +with. Whatever it guesses is then pinned for that speaker for the rest of +the session, so one unlucky guess is not one bad job, it is every job for +that person from then on. Naming the language removes the guess. + +``` +/config set transcription_language de +``` + +A guild that genuinely meets in more than one language sets it to `auto`, +which is what asks for detection-and-pinning explicitly. There is no third +state: clearing the key restores the `de` default rather than removing it. + +`transcription_prompt` is the vocabulary Whisper is biased towards while +decoding — Whisper's `initial_prompt`. It defaults to OneLiteFeather's own +project names and stack, written as an ordinary German sentence so the +style it biases towards is punctuated prose as well. Proper nouns are both +what a general model reliably gets wrong and what a protocol is read for: a +decision minuted about the wrong project is worse than no minutes. Set it +if your names are different ones: + +``` +/config set transcription_prompt "Protokoll eines Meetings über Foo, Bar und Baz." +``` + +Keep it a sentence rather than a word list, keep it in the transcription +language, and keep it short — Whisper only sees the last ~224 tokens of it, +and a long prompt bleeds its own wording into the transcript. + Until every required key (`voice_channel_id`, `consent_role_id`, `document_target`, `policy_version`, `policy_url`, `admin_role_id`) is set, the bot logs a warning naming the guild and skips building that guild's @@ -404,9 +454,14 @@ when it detects this. **Live immediately, and never were stale.** `admin_role_id`, `policy_version`, `policy_url` (read per command invocation, and by the -consent cache with a five-second TTL), and `document_target`, -`document_provider`, `merge_gap_seconds` (read per job by the *worker* -process, not the bot at all). +consent cache with a five-second TTL), and `transcription_language`, +`transcription_prompt`, `document_target`, `document_provider`, +`merge_gap_seconds` (read per job by the *worker* process, not the bot at +all). The two transcription keys apply to the next job the worker claims, +which means a session already recording is still transcribed with the new +value — and a job that has already run is not redone. Changing them +because a protocol came out wrong therefore affects the next meeting, not +the one you are looking at. **Deferred until the recording in progress ends.** `voice_channel_id` and `consent_role_id`. These decide which channel a session's row names and diff --git a/docs/verification/end-to-end-checklist.md b/docs/verification/end-to-end-checklist.md index fbf23fa..eee1007 100644 --- a/docs/verification/end-to-end-checklist.md +++ b/docs/verification/end-to-end-checklist.md @@ -248,7 +248,10 @@ measured, not a plausibility check against the estimate. - [ ] CPU and memory for the bot pod under load: ____________ - [ ] CPU and memory for the worker pod under load (this is the one most likely to differ sharply from the spec's estimate, since transcription - is the heaviest step): ____________ + is the heaviest step). Expect ~4.1GB at peak for large-v3 at + int8_float32 with beam_size 8 — measured off-cluster on four CPU threads, + so this is the confirmation that it holds on a worker node, not a first + look. The limit is 6Gi: ____________ - [ ] **[Plan 4]** CPU and memory for the link-service pod under load: ____________ - [ ] Actual recording size per speaker-hour (extrapolate from this diff --git a/src/sturnus/application/reconfigure.py b/src/sturnus/application/reconfigure.py index e1ffebf..22255e4 100644 --- a/src/sturnus/application/reconfigure.py +++ b/src/sturnus/application/reconfigure.py @@ -59,8 +59,9 @@ class GuildRuntimeConfig: """Everything about a guild that the bot process holds in memory. Deliberately *not* every configuration key: `admin_role_id`, - `policy_url`, `policy_version`, `document_target`, `merge_gap_seconds` - and `document_provider` are read per use (by a permission check, by the + `policy_url`, `policy_version`, `document_target`, `merge_gap_seconds`, + `document_provider`, `transcription_language` and + `transcription_prompt` are read per use (by a permission check, by the consent cache, or by the worker process entirely) and were never stale to begin with. Only what the bot caches needs reconciling. """ diff --git a/src/sturnus/application/transcription.py b/src/sturnus/application/transcription.py index cefb90b..5f39b33 100644 --- a/src/sturnus/application/transcription.py +++ b/src/sturnus/application/transcription.py @@ -30,11 +30,21 @@ class TranscriptionResult: class TranscriptionEngine(Protocol): - async def transcribe(self, path: Path, language: str | None) -> TranscriptionResult: + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: """Transcribe one speaker's recording. `language` pins the language; `None` asks the engine to detect it and report what it found. + + `initial_prompt` is vocabulary and style for the engine to lean + towards — an organisation's project names, the words a general + model has never seen and will otherwise replace with something it + has. It is deliberately a required argument rather than one with a + default: the guild's configured prompt (Spec 11) is worth nothing + if a call site can quietly leave it out, and a caller that really + has no vocabulary to offer says so by passing `None`. """ ... diff --git a/src/sturnus/application/worker.py b/src/sturnus/application/worker.py index 89d4f48..9264e83 100644 --- a/src/sturnus/application/worker.py +++ b/src/sturnus/application/worker.py @@ -13,10 +13,37 @@ transcription can be redone from the original audio; that deletion belongs to the retention sweep (`sturnus.application.retention`), not to this job. -Language pinning (Spec 7): a speaker's first job asks the engine to detect -the language and persists what it found; every later job for that same -speaker passes the stored language back in, so one protocol never mixes -languages mid-session because the engine's guess drifted. +Language (Spec 7, Spec 11). Two things can decide what language a +recording is transcribed as, and the order between them is the whole +point. `transcription_language` is per-guild configuration and wins +outright: when a guild names a language it is handed to the engine, no +detection runs, and *nothing* is written to `detected_language`. Both +halves of that matter. A configured setting that a guess may override is +a trap, and here it would be a self-locking one -- `set_detected_language` +pins the first job's guess for the rest of the session, so the guess would +go on beating the configuration on every later job of that session, and +the column would stop meaning "what the engine detected" and start meaning +"what was configured when this session's first job ran", with no way to +tell the two apart in the data. + +Detection remains available, and is what an unconfigured guild and a guild +that sets the value to `auto` (`sturnus.domain.settings.DETECT_LANGUAGE`) +both get: then, and only then, a speaker's first job asks the engine to +detect the language and persists what it found, and every later job for +that same speaker passes the stored language back in, so one protocol +never mixes languages mid-session because the engine's guess drifted. +That the guess needs pinning at all is the measure of how weak it is: it +is made on one speaker's track, which `vad_filter` has already reduced to +the fragments where that person actually spoke, so a participant whose +first contribution is a three-second agreement is close to a coin flip +between several languages -- and whichever one comes back then governs +every remaining job for them. + +`transcription_prompt` (Spec 11) is the vocabulary the engine is biased +towards while decoding -- an organisation's project names, which is +precisely what a general model has never seen and will replace with +something it has. It is read here, per job, for the same reason the +document settings below are. Dependency-rule note: this module lives in `sturnus.application`, which must never import `sturnus.infrastructure` (tests/test_architecture.py). Every @@ -46,10 +73,13 @@ `links` is typed with this module's own `LinkRepository`, not `assembly`'s `LinkReader`, and `config` (`ConfigReader`) is threaded through alongside -it: `document_target`, `document_provider`, and `merge_gap_seconds` are all -per-guild settings (Spec 11) that this one process cannot resolve until a -session -- and therefore its guild -- is in hand, so they are read inside -`_create_session_document` rather than once at process start. `_BoundLinks` +it: `transcription_language`, `transcription_prompt`, `document_target`, +`document_provider`, and `merge_gap_seconds` are all per-guild settings +(Spec 11) that this one process cannot resolve until a session -- and +therefore its guild -- is in hand. The first two are read in `process_one` +itself, just before the engine is called; the last three inside +`_create_session_document`. None of them is read once at process start, +because one worker serves every guild. `_BoundLinks` adapts one call's resolved provider back down to the plain `LinkReader` shape `assemble` itself calls, so `assemble` stays ignorant of configuration entirely. @@ -211,9 +241,11 @@ async def channel_ref(self, session_id: int) -> tuple[int, int, str | None]: async def guild_id(self, session_id: int) -> int: """The guild a session belongs to. - Needed to resolve per-guild configuration (`document_target`, - `document_provider`, `merge_gap_seconds`, Spec 11) at - document-creation time -- see `_create_session_document`. + Needed to resolve per-guild configuration (Spec 11) twice per job: + `transcription_language` and `transcription_prompt` before the + engine is called (`process_one`), and `document_target`, + `document_provider` and `merge_gap_seconds` when a session's last + job creates the document (`_create_session_document`). """ ... @@ -242,6 +274,32 @@ class _ClaimedJobShape(Protocol): wrapped_data_key: bytes +def _configured_language(configured: str | None) -> str | None: + """The language a guild named, or `None` when it asked for detection. + + Three stored values mean "detect", and the caller has no reason to + tell them apart: `auto` (`sturnus.domain.settings.DETECT_LANGUAGE`), + nothing at all, and blank. The last two are unreachable through + `/config` -- the key has a default and clearing restores it -- but + neither is unreachable in practice: `ConfigReader` is a protocol, and + `guild_config` is a table an operator is told they may edit with SQL + (`docs/operations.md` section 4.1), which `ConfigStore.set`'s + validation never sees. A blank value has to mean *something*, and the + alternative is passing `""` to the engine, which rejects it -- turning + one careless `UPDATE` into every job of that guild failing. + + Surrounding whitespace is stripped for the same reason: `" de "` is + not a language code faster-whisper knows, and a value typed with a + trailing space is not a decision to fail every job. + """ + if configured is None: + return None + named = configured.strip() + if not named or named.casefold() == domain_settings.DETECT_LANGUAGE: + return None + return named + + async def _guild_timezone(config: ConfigReader, guild: int) -> tzinfo: """The timezone the protocol's times are written in (Spec 11). @@ -372,7 +430,11 @@ async def process_one( 1. Claim -- nothing claimed means there is no work; the caller backs off. 2. Download the encrypted object to a scratch directory under `work_dir`. 3. Unwrap the data key and decrypt to a plaintext WAV, still on disk. - 4. Transcribe -- language pinning per Spec 7 (see the module docstring). + 4. Resolve the guild's `transcription_language` and + `transcription_prompt` (Spec 11), then transcribe -- configured + language first, detection and per-speaker pinning only when the + guild asked for it (Spec 7; see the module docstring for the order + and why it is that way round). 5. Store the transcript on the job; ask whether it was the session's last. 6. If it was: assemble every participant's stored transcript into one document (`_create_session_document`, `sturnus.application.assembly. @@ -428,14 +490,37 @@ async def process_one( job.encryption_key_id, ) - pinned_language = await sessions.detected_language(job.session_id, job.discord_user_id) + # Both settings are the guild's (Spec 11), so the guild has to + # be resolved first: one worker process serves all of them and + # only the session names one. Two extra reads per job, against + # a transcription measured in minutes. + guild = await sessions.guild_id(job.session_id) + configured_language = await config.get(guild, domain_settings.TRANSCRIPTION_LANGUAGE) + prompt = await config.get(guild, domain_settings.TRANSCRIPTION_PROMPT) + + # A configured language beats a stored detection outright, and + # the stored detection is not even read when there is one -- + # see this module's docstring for why that order is the point + # rather than a detail. + named_language = _configured_language(configured_language) + pinned_language = ( + named_language + if named_language is not None + else await sessions.detected_language(job.session_id, job.discord_user_id) + ) try: - result = await engine.transcribe(wav_path, pinned_language) + result = await engine.transcribe(wav_path, pinned_language, prompt) except Exception as exc: log.warning("Transcription failed for job %d", job.id) await queue.fail(job.id, str(exc), max_attempts) return True + # Reached only when the guild asked for detection *and* this is + # the first job for this speaker: a named language is never + # `None`, which is exactly what keeps configuration out of + # `detected_language`. Dropping the condition altogether would + # write the configured language into that column on every job + # and pin it there, which is the trap the docstring describes. if pinned_language is None: await sessions.set_detected_language( job.session_id, job.discord_user_id, result.language diff --git a/src/sturnus/domain/settings.py b/src/sturnus/domain/settings.py index 976d3c2..ae5cdc1 100644 --- a/src/sturnus/domain/settings.py +++ b/src/sturnus/domain/settings.py @@ -16,6 +16,17 @@ ADMIN_ROLE_ID = "admin_role_id" MERGE_GAP_SECONDS = "merge_gap_seconds" TIMEZONE = "timezone" +TRANSCRIPTION_LANGUAGE = "transcription_language" +TRANSCRIPTION_PROMPT = "transcription_prompt" + +#: The one value of `TRANSCRIPTION_LANGUAGE` that is not a language: it +#: asks the engine to detect one per speaker and pin what it found for the +#: rest of the session, which is what the worker did unconditionally before +#: this key existed. It exists because `DEFAULTS` below names a language, +#: and `/config clear` restores a default rather than removing it -- without +#: a spelling for "detect", a guild that really does meet in several +#: languages would have no way back to detection at all. +DETECT_LANGUAGE = "auto" DEFAULTS: dict[str, str] = { EMPTY_GRACE_SECONDS: "60", @@ -30,6 +41,32 @@ # not obviously wrong to a reader -- 15:08 looks like a plausible # meeting time whether or not it is the right one. TIMEZONE: "Europe/Berlin", + # Naming the language is worth far more than it looks. The + # alternative is detection, which runs on one speaker's track after + # the silence has been cut out of it, and a speaker whose first + # contribution is "ja, genau" gives it almost nothing to go on -- + # German, Dutch and Danish are all plausible readings of three + # seconds of that. Whatever comes back is then pinned for that + # speaker for the rest of the session, so a single unlucky guess is + # not one bad job, it is every job for that person from then on. + # German is what this deployment's guilds actually meet in; `auto` + # (see `DETECT_LANGUAGE` above) is how one that does not says so. + TRANSCRIPTION_LANGUAGE: "de", + # Whisper's `initial_prompt` biases the decoder towards the + # vocabulary and the style of this text. Proper nouns are where a + # general model fails and where a protocol is judged: a model that + # has never seen "Ducula" will confidently write the nearest word it + # has seen, and a decision recorded about the wrong project is worse + # than no minutes at all. The default is OneLiteFeather's own + # vocabulary -- the bird-named projects and the stack they are built + # on -- written as an ordinary German sentence so the style it biases + # towards is punctuated prose in the language above rather than a + # bare word list. + TRANSCRIPTION_PROMPT: ( + "Protokoll eines OneLiteFeather-Meetings über die Projekte Falco, Otis, " + "Ducula, Pica, Guira, Aves, Sturnus, Cygnus, Apus und Coris sowie über " + "Minestom, Paper, Outline, Harbor, Flux, Kubernetes, Renovate und Gradle." + ), } # No default value, so these must be set before going live. diff --git a/src/sturnus/entrypoints/worker.py b/src/sturnus/entrypoints/worker.py index 4973c77..090b9a0 100644 --- a/src/sturnus/entrypoints/worker.py +++ b/src/sturnus/entrypoints/worker.py @@ -112,9 +112,21 @@ _DOCUMENT_RETRY_INTERVAL_SECONDS = 300.0 #: faster-whisper on CPU (Spec 7 sizes the deployment for CPU, not GPU -- -#: see `charts/sturnus/values.yaml`'s `worker.resources`): int8 is the -#: quantisation the chart's own model-size comment already assumes. -_WHISPER_COMPUTE_TYPE = "int8" +#: see `charts/sturnus/values.yaml`'s `worker.resources`). The weights are +#: still quantised to int8, which is what keeps `large-v3` inside a +#: memory budget a CPU node will actually give it; the `_float32` half +#: names the type everything *else* runs in -- activations, accumulation, +#: the layers that are never quantised. +#: +#: Naming it is the point. Plain `int8` is an alias whose float type +#: CTranslate2 picks for the device it finds itself on, so what the +#: decoder accumulates in is decided by the node the pod landed on rather +#: than by this file; on today's x86 workers the alias resolves to exactly +#: this, and on a machine with bfloat16 support it need not. Transcription +#: quality is not something to leave to the scheduler, and CTranslate2 +#: falls back silently rather than refusing a compute type it cannot +#: provide -- there would be nothing in the logs to say it had happened. +_WHISPER_COMPUTE_TYPE = "int8_float32" #: Package and resource name of the real Outline document template. See #: `_load_template` -- this is the packaged template `process_one` must @@ -161,8 +173,20 @@ class WorkerSettings(StrictSettings): master_key_id: str outline_base_url: str outline_service_key: SecretStr - whisper_model: str = "large-v3-turbo" - whisper_default_language: str = "en" + # `large-v3` rather than `large-v3-turbo`: turbo is a distillation with + # four decoder layers instead of thirty-two, and what it gives up is + # concentrated outside English -- which is the only place this + # deployment operates. It buys speed, and nothing here is waiting: + # transcription runs offline, one speaker's file at a time, after the + # meeting is over. The cost is paid in the chart instead, in memory + # and in a longer first start (`charts/sturnus/values.yaml`). + whisper_model: str = "large-v3" + # The floor under `transcription_language` (Spec 11), which is + # per-guild and normally decides this; reached only when a guild asked + # for detection and the engine's detection came back with nothing. + # `en` here was a real defect, not a harmless default: every guild + # this serves meets in German. + whisper_default_language: str = "de" model_cache_dir: Path | None = None work_dir: Path = Path("/tmp") max_job_attempts: int = 3 diff --git a/src/sturnus/infrastructure/whisper.py b/src/sturnus/infrastructure/whisper.py index 841a9b0..75568fb 100644 --- a/src/sturnus/infrastructure/whisper.py +++ b/src/sturnus/infrastructure/whisper.py @@ -4,6 +4,14 @@ thread. The model is loaded once and reused; jobs are processed one at a time (Spec 5.3), so no locking is required around it. +Every decoding parameter below is set explicitly rather than left to the +library's default, and each one is set against a specific way a meeting +protocol goes wrong: silence turning into invented speech, one bad segment +poisoning the segments after it, a project name coming out as a common +word. `tests/infrastructure/test_whisper.py` pins each of them against a +fake model, because none of them is visible in the output of a passing +two-second fixture. + Silence is cut out before the decoder sees it, but *not* by faster-whisper's own `vad_filter`. That option runs Silero, whose recurrent state collapses on the bit-exact zero padding `SpeakerWriter` writes between packets — it @@ -41,10 +49,14 @@ def __init__( self._model = WhisperModel(model_size, device=device, compute_type=compute_type) self._default_language = default_language - async def transcribe(self, path: Path, language: str | None) -> TranscriptionResult: - return await asyncio.to_thread(self._transcribe, path, language) + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: + return await asyncio.to_thread(self._transcribe, path, language, initial_prompt) - def _transcribe(self, path: Path, language: str | None) -> TranscriptionResult: + def _transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: # Decoded here rather than inside `transcribe()` so the gate and the # model measure and seek through the *same* array. Handing the model # the path instead would decode a 100-minute file a second time, and @@ -68,6 +80,16 @@ def _transcribe(self, path: Path, language: str | None) -> TranscriptionResult: segments, info = self._model.transcribe( audio, language=language, + # Biases the decoder towards the vocabulary and the style of + # this text. It is the only lever Sturnus has on proper nouns, + # and proper nouns are both what Whisper reliably gets wrong + # and what a protocol is read for: a decision about "Ducula" + # is unusable when the sentence says "Dracula". Per-guild + # configuration (`transcription_prompt`, Spec 11) rather than + # a constant here -- the vocabulary that matters is the + # organisation's, and this adapter has no idea whose meeting + # it is transcribing. + initial_prompt=initial_prompt, # A flat list of seconds — [start0, end0, start1, end1, ...] — not # a list of pairs and not the dict form, which belongs to # `BatchedInferencePipeline.transcribe`, a different API. @@ -86,6 +108,25 @@ def _transcribe(self, path: Path, language: str | None) -> TranscriptionResult: # filter the decoder's output on it. compression_ratio_threshold=2.4, no_speech_threshold=0.6, + # The library defaults this to `True`, which feeds each + # segment's own text back in as the prompt for the next one. + # One hallucinated segment then becomes the context every + # following segment is decoded against, and the cascade the + # two thresholds above exist to catch is exactly what that + # produces. `vad_filter` makes the default worse here rather + # than better: it cuts one speaker's track into fragments with + # every silence removed, so the "previous text" is routinely + # from minutes earlier and about something else entirely -- + # per-speaker recordings of a conversation are the case this + # default is least suited to. + condition_on_previous_text=False, + # Above the library's default of 5. Beam search cost is + # roughly linear in the width and this deployment transcribes + # offline, one speaker's file at a time, hours after the + # meeting -- so the trade is CPU seconds (which the worker has, + # see `charts/sturnus/values.yaml`) against a wrong word in a + # document people read instead of having been in the room. + beam_size=8, ) # No offset arithmetic here, deliberately. Unlike the `vad_filter` # path, which concatenates the kept audio and repairs the timestamps diff --git a/tests/application/test_worker.py b/tests/application/test_worker.py index 2a1e91d..07da1e9 100644 --- a/tests/application/test_worker.py +++ b/tests/application/test_worker.py @@ -50,17 +50,34 @@ async def fail(self, job_id: int, error: str, _max_attempts: int) -> None: class FakeEngine: - def __init__(self, text: str = "spoken words", fail: bool = False) -> None: + """`detected` is what the engine *reports back*, which is not the same + as what it was asked for: it is deliberately different from the + language the transcription tests configure, so a test asserting that a + configured language is never overwritten by detection cannot pass by + the two happening to agree. + """ + + def __init__( + self, text: str = "spoken words", fail: bool = False, detected: str = "de" + ) -> None: self.text = text self.fail = fail + self.detected = detected self.calls: list[tuple[Path, str | None]] = [] - - async def transcribe(self, path: Path, language: str | None) -> TranscriptionResult: + #: `initial_prompt` from every call, recorded separately from + #: `calls` so the existing `calls[i][1]` language assertions stay + #: as they are. + self.prompts: list[str | None] = [] + + async def transcribe( + self, path: Path, language: str | None, initial_prompt: str | None + ) -> TranscriptionResult: self.calls.append((path, language)) + self.prompts.append(initial_prompt) if self.fail: raise RuntimeError("model exploded") return TranscriptionResult( - segments=(TranscribedSegment(0.0, 1.0, self.text),), language="de" + segments=(TranscribedSegment(0.0, 1.0, self.text),), language=self.detected ) @@ -215,6 +232,23 @@ async def get(self, guild_id: int, key: str) -> str | None: return self._values.get((guild_id, key)) +def guild_config(extra: dict[str, str] | None = None) -> FakeConfig: + """A `FakeConfig` for `GUILD` whose document settings are already right. + + The transcription keys (`transcription_language`, `transcription_prompt`) + are absent unless a test names them, which is what keeps the + unconfigured path -- detect once, then pin per speaker -- exercised by + every test that does not care about them. + """ + values: dict[tuple[int, str], str] = { + (GUILD, domain_settings.DOCUMENT_TARGET): "col-default", + (GUILD, domain_settings.DOCUMENT_PROVIDER): "outline", + (GUILD, domain_settings.MERGE_GAP_SECONDS): "15", + } + values.update({(GUILD, key): value for key, value in (extra or {}).items()}) + return FakeConfig(values) + + def job(job_id: int = 1, session_id: int = 1, user_id: int = 100) -> ClaimedJob: return ClaimedJob( id=job_id, @@ -548,6 +582,107 @@ async def test_a_later_job_pins_the_stored_language(tmp_path: Path) -> None: assert engine.calls[0][1] == "de" +async def test_the_guilds_configured_language_is_what_gets_transcribed( + tmp_path: Path, +) -> None: + """`transcription_language` (Spec 11) is passed straight to the engine. + + Naming the language is what stops the engine detecting one, and + detection on a per-speaker track is a coin flip whenever the speaker's + first job is short. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "de"}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.calls[0][1] == "de" + + +async def test_a_configured_language_is_never_pinned_as_a_detection( + tmp_path: Path, +) -> None: + """Writing the configured value into `detected_language` would make the + column mean two different things and would freeze the configuration as + it stood on a session's first job: a guild that corrects the setting + mid-session would keep getting the old language until the session ends. + """ + engine, sessions = FakeEngine(detected="nl"), FakeSessions() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "de"}) + await process_one(**run(tmp_path, engine=engine, sessions=sessions, config=config)) + assert sessions.languages == {} + + +async def test_a_configured_language_wins_over_an_earlier_detection( + tmp_path: Path, +) -> None: + """The stored detection is a guess; the configured value is a decision. + + This is the ordering that stops one bad detection -- pinned by an + earlier job of the same session, before an administrator noticed and + configured the language -- from governing every job after it. + """ + engine, sessions = FakeEngine(), FakeSessions() + sessions.languages[100] = "nl" + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "de"}) + await process_one(**run(tmp_path, engine=engine, sessions=sessions, config=config)) + assert engine.calls[0][1] == "de" + + +async def test_auto_asks_for_detection_and_pins_what_came_back(tmp_path: Path) -> None: + """`auto` is how a genuinely multilingual guild opts back in. + + Without it the detect-and-pin path would be unreachable in production, + since `transcription_language` has a default and clearing the key + restores it rather than removing it. + """ + engine, sessions = FakeEngine(detected="nl"), FakeSessions() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: "auto"}) + await process_one(**run(tmp_path, engine=engine, sessions=sessions, config=config)) + assert engine.calls[0][1] is None + assert sessions.languages[100] == "nl" + + +async def test_a_blank_configured_language_asks_for_detection(tmp_path: Path) -> None: + """`guild_config` is a table operators are told they may edit with SQL, + which `ConfigStore.set`'s validation never sees. A blank value has to + mean something, and the alternative -- handing `""` to the engine, + which rejects it -- turns one careless `UPDATE` into every job of that + guild failing. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: " "}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.calls[0][1] is None + + +async def test_a_configured_language_is_stripped_before_the_engine_sees_it( + tmp_path: Path, +) -> None: + """`" de "` is not a language code faster-whisper knows, and a value + typed with a trailing space is not a decision to fail every job. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_LANGUAGE: " de "}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.calls[0][1] == "de" + + +async def test_the_guilds_vocabulary_prompt_reaches_the_engine(tmp_path: Path) -> None: + """`transcription_prompt` (Spec 11) is per-guild for the same reason + `document_target` is: one worker process serves every guild, and whose + project names matter is not knowable until a session is in hand. + """ + engine = FakeEngine() + config = guild_config({domain_settings.TRANSCRIPTION_PROMPT: "Ducula, Guira, Minestom."}) + await process_one(**run(tmp_path, engine=engine, config=config)) + assert engine.prompts == ["Ducula, Guira, Minestom."] + + +async def test_a_guild_with_no_prompt_configured_biases_nothing(tmp_path: Path) -> None: + engine = FakeEngine() + await process_one(**run(tmp_path, engine=engine, config=guild_config())) + assert engine.prompts == [None] + + async def test_the_audio_object_is_not_deleted_after_transcription(tmp_path: Path) -> None: """Audio outlives its transcription (Spec 12); the retention sweep deletes it.""" store = FakeStore() diff --git a/tests/entrypoints/test_worker_entrypoint.py b/tests/entrypoints/test_worker_entrypoint.py index 01ee628..d364633 100644 --- a/tests/entrypoints/test_worker_entrypoint.py +++ b/tests/entrypoints/test_worker_entrypoint.py @@ -12,8 +12,14 @@ mention-less fallback. """ +import ctranslate2 # type: ignore[import-untyped] + from sturnus.application.worker import _FALLBACK_TEMPLATE -from sturnus.entrypoints.worker import _load_template +from sturnus.entrypoints.worker import ( + _WHISPER_COMPUTE_TYPE, + WorkerSettings, + _load_template, +) def test_loaded_template_is_not_the_fallback() -> None: @@ -22,3 +28,55 @@ def test_loaded_template_is_not_the_fallback() -> None: def test_loaded_template_renders_outline_mentions() -> None: assert "mention://" in _load_template() + + +def _settings(**overrides: object) -> WorkerSettings: + """`WorkerSettings` with every required field supplied as a literal. + + Passed as arguments rather than through `monkeypatch.setenv` so the + defaults under test are read from the class, not from whatever the + machine running the suite happens to export. + """ + required: dict[str, object] = { + "database_url": "postgresql+asyncpg://u:p@db/sturnus", + "s3_endpoint": "http://s3.invalid", + "s3_bucket": "sturnus-audio", + "s3_access_key": "access", + "s3_secret_key": "secret", + "master_key": "a" * 44, + "master_key_id": "k1", + "outline_base_url": "https://outline.invalid", + "outline_service_key": "token", + } + return WorkerSettings(**{**required, **overrides}) # type: ignore[arg-type] + + +def test_the_default_model_is_the_undistilled_large_one() -> None: + """`large-v3-turbo` is a distilled decoder -- four layers where + `large-v3` has thirty-two -- and the accuracy it gives up is not evenly + spread: it shows up outside English, which is the only place this + deployment operates. Nothing here is latency-sensitive (transcription + happens offline, per speaker, after the meeting), so the distillation + buys time nobody is waiting for. + """ + assert _settings().whisper_model == "large-v3" + + +def test_the_default_language_is_the_one_these_meetings_are_held_in() -> None: + """This is the value that decides what an unconfigured guild gets when + the engine's own detection comes up empty, and `en` on a + German-speaking server was simply wrong. Per-guild configuration + (`transcription_language`, Spec 11) overrides it; this is the floor + under it. + """ + assert _settings().whisper_default_language == "de" + + +def test_the_configured_quantisation_is_one_this_cpu_can_actually_run() -> None: + """CTranslate2 does not refuse a compute type it cannot provide -- it + quietly falls back to one it can, so a wrong value here costs accuracy + or speed with nothing in the logs to say so. `int8_float32` names the + activation type outright instead of leaving it to the `int8` alias, + whose float type CTranslate2 chooses per device. + """ + assert _WHISPER_COMPUTE_TYPE in ctranslate2.get_supported_compute_types("cpu") diff --git a/tests/infrastructure/test_config_store.py b/tests/infrastructure/test_config_store.py index 6f9c65a..56cd243 100644 --- a/tests/infrastructure/test_config_store.py +++ b/tests/infrastructure/test_config_store.py @@ -111,3 +111,46 @@ async def test_snapshot_agrees_with_get_for_every_known_key(store: ConfigStore) async def test_snapshot_is_per_guild(store: ConfigStore) -> None: await store.set(GUILD, settings.VOICE_CHANNEL_ID, "12345", T0) assert settings.VOICE_CHANNEL_ID not in await store.snapshot(GUILD + 1) + + +async def test_the_transcription_language_is_a_key_an_administrator_can_set( + store: ConfigStore, +) -> None: + """`set` rejects every key it does not know, so a setting the worker + reads but `DEFAULTS` never names is one nobody can change: `/config + set` answers "unknown configuration key" while the worker goes on + using the built-in value forever. + """ + await store.set(GUILD, settings.TRANSCRIPTION_LANGUAGE, "en", T0) + assert await store.get(GUILD, settings.TRANSCRIPTION_LANGUAGE) == "en" + + +async def test_clearing_the_transcription_language_restores_german(store: ConfigStore) -> None: + """Clearing restores the default rather than removing the value, which + is why `auto` exists as a spelling (`settings.DETECT_LANGUAGE`) -- there + is no state in which no language is configured. + """ + await store.set(GUILD, settings.TRANSCRIPTION_LANGUAGE, "en", T0) + await store.set(GUILD, settings.TRANSCRIPTION_LANGUAGE, None, T0) + assert await store.get(GUILD, settings.TRANSCRIPTION_LANGUAGE) == "de" + + +async def test_a_guild_that_configured_nothing_still_gets_the_projects_vocabulary( + store: ConfigStore, +) -> None: + """The prompt is only worth having if it carries the names Whisper + actually gets wrong, and a guild that never runs `/config set` is the + normal case -- so the default has to be the real vocabulary, not an + empty string waiting for someone to fill it in. + """ + prompt = await store.get(GUILD, settings.TRANSCRIPTION_PROMPT) + assert prompt is not None + for name in ("Ducula", "Guira", "Minestom", "Outline"): + assert name in prompt + + +async def test_the_transcription_prompt_is_a_key_an_administrator_can_set( + store: ConfigStore, +) -> None: + await store.set(GUILD, settings.TRANSCRIPTION_PROMPT, "Nur eigene Wörter.", T0) + assert await store.get(GUILD, settings.TRANSCRIPTION_PROMPT) == "Nur eigene Wörter." diff --git a/tests/infrastructure/test_whisper.py b/tests/infrastructure/test_whisper.py index 3f6df24..a93b786 100644 --- a/tests/infrastructure/test_whisper.py +++ b/tests/infrastructure/test_whisper.py @@ -20,21 +20,21 @@ def engine() -> WhisperEngine: @pytest.mark.slow async def test_transcribes_real_speech(engine: WhisperEngine) -> None: - result = await engine.transcribe(FIXTURE, language="de") + result = await engine.transcribe(FIXTURE, language="de", initial_prompt=None) assert result.segments assert any(segment.text.strip() for segment in result.segments) @pytest.mark.slow async def test_offsets_are_within_the_recording(engine: WhisperEngine) -> None: - result = await engine.transcribe(FIXTURE, language="de") + result = await engine.transcribe(FIXTURE, language="de", initial_prompt=None) for segment in result.segments: assert 0.0 <= segment.start <= segment.end @pytest.mark.slow async def test_detection_reports_a_language(engine: WhisperEngine) -> None: - result = await engine.transcribe(FIXTURE, language=None) + result = await engine.transcribe(FIXTURE, language=None, initial_prompt=None) assert result.language @@ -56,7 +56,7 @@ async def test_silence_yields_no_segments(engine: WhisperEngine, tmp_path: Path) w.setframerate(16_000) w.writeframes(b"\x00" * 16_000 * 3) - result = await engine.transcribe(silent, language="de") + result = await engine.transcribe(silent, language="de", initial_prompt=None) assert [s for s in result.segments if s.text.strip()] == [] @@ -146,7 +146,7 @@ async def test_a_file_with_only_padding_never_reaches_the_model(tmp_path: Path) model = _RecordingModel() engine = _engine_with(model) - result = await engine.transcribe(silent, language="de") + result = await engine.transcribe(silent, language="de", initial_prompt=None) assert model.calls == [] assert result.segments == () @@ -178,7 +178,7 @@ async def test_the_model_is_given_the_whole_array_and_the_clips_the_gate_found( model = _RecordingModel() engine = _engine_with(model) - await engine.transcribe(recording, language="de") + await engine.transcribe(recording, language="de", initial_prompt=None) assert len(model.calls) == 1 call = model.calls[0] @@ -207,7 +207,7 @@ async def test_silero_is_never_asked(tmp_path: Path) -> None: _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000, dtype=np.float32)])) engine = _engine_with(model := _RecordingModel()) - await engine.transcribe(recording, language="de") + await engine.transcribe(recording, language="de", initial_prompt=None) assert model.calls[0]["vad_filter"] is False @@ -224,7 +224,7 @@ async def test_the_decoder_side_hallucination_guards_stay_set(tmp_path: Path) -> _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000, dtype=np.float32)])) engine = _engine_with(model := _RecordingModel()) - await engine.transcribe(recording, language="de") + await engine.transcribe(recording, language="de", initial_prompt=None) assert model.calls[0]["compression_ratio_threshold"] == 2.4 assert model.calls[0]["no_speech_threshold"] == 0.6 @@ -245,7 +245,7 @@ async def test_offsets_are_returned_unchanged(tmp_path: Path) -> None: model = _RecordingModel(segments=(_FakeSegment(5.25, 6.75, " hallo"),)) engine = _engine_with(model) - result = await engine.transcribe(recording, language="de") + result = await engine.transcribe(recording, language="de", initial_prompt=None) assert [(s.start, s.end, s.text) for s in result.segments] == [(5.25, 6.75, " hallo")] @@ -255,6 +255,163 @@ async def test_an_undetected_language_falls_back_to_the_default(tmp_path: Path) _write_wav(recording, np.concatenate([_tone(2.0), np.zeros(16_000, dtype=np.float32)])) engine = _engine_with(_RecordingModel(language=None), default_language="de") - result = await engine.transcribe(recording, language=None) + result = await engine.transcribe(recording, language=None, initial_prompt=None) assert result.language == "de" + + +# --- Argument-level tests ported from the transcription-quality branch --- +# +# They came with their own fake, `_ModelSpy`, which patches the name the +# module imported and so records the *constructor* arguments as well as the +# transcribe ones. `_RecordingModel` above cannot: it is handed to an engine +# built by `object.__new__`, which never runs `__init__`. The two coexist +# because they answer different questions, not because either is redundant. + + +class _ModelSpy: + """Stands in for `faster_whisper.WhisperModel`, as both class and instance. + + Patched over the name `sturnus.infrastructure.whisper` imported, so + calling it is the construction `WhisperEngine.__init__` performs and + the object it hands back is this same spy -- which keeps the recorded + constructor arguments and the recorded `transcribe` arguments in one + place a test can read. + """ + + def __init__(self) -> None: + self.construction: dict[str, Any] = {} + self.transcription: dict[str, Any] = {} + #: What `info.language` reports; `None` stands for detection that + #: came up empty, which is what the adapter's own default is for. + self.detected: str | None = "de" + + def __call__(self, model_size: str, **kwargs: Any) -> "_ModelSpy": + self.construction = {"model_size": model_size, **kwargs} + return self + + def transcribe(self, path: str, **kwargs: Any) -> tuple[Any, _FakeInfo]: + self.transcription = {"path": path, **kwargs} + return iter(()), _FakeInfo(self.detected) + + +@pytest.fixture +def spy(monkeypatch: pytest.MonkeyPatch) -> _ModelSpy: + model = _ModelSpy() + monkeypatch.setattr("sturnus.infrastructure.whisper.WhisperModel", model) + return model + + +def _engine(compute_type: str = "int8_float32", default_language: str = "de") -> WhisperEngine: + return WhisperEngine( + model_size="large-v3", + device="cpu", + compute_type=compute_type, + default_language=default_language, + ) + + +async def test_the_model_is_built_with_the_quantisation_it_was_given(spy: _ModelSpy) -> None: + """`compute_type` is chosen in `sturnus.entrypoints.worker`, not here. + + It has to arrive at the library unaltered: the adapter has no business + substituting a quantisation, and a silently-dropped one would degrade + every transcription while every test still passed. + """ + _engine() + assert spy.construction == { + "model_size": "large-v3", + "device": "cpu", + "compute_type": "int8_float32", + } + + +async def test_the_vocabulary_prompt_reaches_the_decoder(spy: _ModelSpy) -> None: + """`initial_prompt` is the only lever on proper nouns Sturnus has. + + Project names -- Ducula, Guira, Minestom -- are exactly what Whisper + guesses wrong and exactly what a meeting protocol is read for. Losing + the prompt on the way to the library costs nothing visible and every + name in the document. + """ + await _engine().transcribe(FIXTURE, "de", "Ducula, Guira, Minestom.") + assert spy.transcription["initial_prompt"] == "Ducula, Guira, Minestom." + + +async def test_no_prompt_is_still_a_call_the_library_understands(spy: _ModelSpy) -> None: + """A guild may have no vocabulary worth biasing towards; `None` is that.""" + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["initial_prompt"] is None + + +async def test_a_segment_never_conditions_the_next_one(spy: _ModelSpy) -> None: + """faster-whisper defaults `condition_on_previous_text` to `True`. + + That feeds each segment's text back in as the next one's prompt, so a + single hallucination becomes the context every following segment is + decoded against and the repetition runs away. `vad_filter` (below) + makes it worse rather than better here: one speaker's track is cut + into disconnected fragments with the silence removed, so the + "previous text" a segment gets conditioned on is frequently from + minutes earlier and unrelated. + """ + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["condition_on_previous_text"] is False + + +async def test_the_beam_is_wider_than_the_library_default(spy: _ModelSpy) -> None: + """Costs CPU, which this deployment has, and buys accuracy it does not.""" + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["beam_size"] == 8 + + +async def test_the_hallucination_guards_are_still_in_place(spy: _ModelSpy) -> None: + """The parameters that keep silence and repetition out (Spec 7). + + Pinned here because they are invisible in a passing test suite: + dropping any of them produces a worker that transcribes perfectly well + in every test and invents speech for a participant who never said a + word in production. + + `vad_filter` is asserted **False**, which is the reverse of what this + test said when it was written. Silero was the guard until it was found + to be the defect: its recurrent state collapses on the bit-exact zero + padding `SpeakerWriter` writes between packets, and it reported about + one second of speech in two minutes of a real recording. The silence is + now cut by `sturnus.infrastructure.speech_gate` before the decoder sees + it, so the guard is the clip list plus the two thresholds below, and + turning Silero back on would restore the defect rather than a + safeguard. + """ + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["vad_filter"] is False + # The gate's own output, in the flat [start, end, start, end, ...] form + # faster-whisper expects. Asserted for shape rather than for values -- + # what the gate finds in the fixture belongs to the gate's own tests -- + # but asserted non-empty, because an empty list is read by + # `WhisperModel.transcribe` as "transcribe everything" and would put the + # padding back in front of the decoder. + clips = spy.transcription["clip_timestamps"] + assert clips and len(clips) % 2 == 0 + assert spy.transcription["compression_ratio_threshold"] == 2.4 + assert spy.transcription["no_speech_threshold"] == 0.6 + + +async def test_the_pinned_language_reaches_the_library(spy: _ModelSpy) -> None: + """Passing a language is what stops faster-whisper detecting one at all.""" + await _engine().transcribe(FIXTURE, "de", None) + assert spy.transcription["language"] == "de" + + +async def test_detection_is_asked_for_when_no_language_is_pinned(spy: _ModelSpy) -> None: + await _engine().transcribe(FIXTURE, None, None) + assert spy.transcription["language"] is None + + +async def test_a_detection_that_came_up_empty_falls_back_to_the_default(spy: _ModelSpy) -> None: + """`TranscriptionResult.language` is stored and reused for the whole + session (`sturnus.application.worker`), so it may never be `None`. + """ + spy.detected = None + result = await _engine(default_language="de").transcribe(FIXTURE, None, None) + assert result.language == "de"