feat(worker): transcribe for the language and the vocabulary of the room - #44
Merged
Conversation
Six changes to how a recording is turned into text, all of them aimed at the same thing: a protocol somebody can read afterwards and trust. **The language was the defect, not a setting.** `whisper_default_language` was `en` on a server whose meetings are held in German, and the fallback was only half the problem. With nothing pinned, the engine detected the language per speaker and `set_detected_language` wrote whatever came back -- for the rest of that speaker's session. Detection here is weak in exactly the way that matters: it runs on one speaker's track, which `vad_filter` has already cut down to the fragments where that person actually spoke, so a participant whose first contribution is a three-second "ja, genau" is close to a coin flip between several languages. One unlucky guess was not one bad job, it was every job for that person from then on. So `transcription_language` becomes per-guild configuration (Spec 11), resolved through `ConfigStore` on the path `document_target` and `merge_gap_seconds` already use, defaulting to `de`, with `STURNUS_WHISPER_DEFAULT_LANGUAGE` (now also `de`) left as the floor under it. The order between configuration and detection is the part worth arguing. A configured language wins outright: it is handed to the engine, no detection runs, and nothing is written to `detected_language`. A configured setting a guess may override is a trap, and here it would be a self-locking one -- the guess gets pinned and then beats the configuration on every later job of that session, while the column stops meaning "what the engine detected" and starts meaning "what was configured when this session's first job ran", with no way to tell the two apart in the data. Detection is still reachable, deliberately: `auto` asks for it, and is what a genuinely multilingual guild sets. Without that spelling the detect-and-pin path would have been unreachable in production, since clearing a key restores its default rather than removing it. **`initial_prompt`, carrying the organisation's vocabulary.** Proper nouns are both what a general model reliably gets wrong and what a protocol is read for -- a decision minuted about "Dracula" is not a decision about Ducula. `transcription_prompt` is per-guild for the same reason the language is, and defaults to OneLiteFeather's bird-named projects and their stack, written as an ordinary German sentence so the style it biases towards is punctuated prose as well as the right words. **`condition_on_previous_text=False`.** The library defaults it to `True`, which feeds each segment's own text back in as the next one's prompt, so one hallucination becomes the context every following segment is decoded against -- the cascade `compression_ratio_threshold` and `no_speech_threshold` exist to catch is what that produces. `vad_filter` makes the default worse rather than better here: it removes the silence, so the "previous text" is routinely from minutes earlier and about something else. Per-speaker recordings of a conversation are the case that default suits least. **`beam_size=8`**, over the library's 5. Beam cost is roughly linear in the width, and the trade is CPU seconds the worker has against a wrong word in a document people read instead of having been in the room. **`large-v3` instead of `large-v3-turbo`.** Turbo is a distilled decoder, four layers where large-v3 has thirty-two, and what it gives up is concentrated outside English -- the only place this deployment operates. It buys latency nobody is waiting for: transcription is offline, per speaker, hours after the meeting. **`int8_float32` instead of `int8`.** The weights are quantised to int8 either way; the suffix names the type everything else runs in. Bare `int8` is an alias whose float type CTranslate2 picks for whatever machine it finds itself on, which leaves transcription quality depending on the node the pod was scheduled to, and CTranslate2 falls back silently rather than refusing a compute type it cannot provide. The chart pays for the last two. `worker.resources` works the budget out rather than guessing: ~1.55GB of int8 weights (1.55B parameters against turbo's 809M), ~2.2GB of decoding buffers (cached attention keys and values are held per beam per decoder layer, and this has 32 decoder layers where turbo has 4 -- scaled from faster-whisper's own published CPU benchmark for this architecture at beam 5), ~0.5GB of process. 4Gi request, 6Gi limit; exceeding the limit is an OOMKill mid-transcription. The startup budget doubles with the download, which is ~3.1GB of float16 checkpoint against turbo's ~1.6GB, and nothing answers /healthz until the engine is built. Every parameter that reaches `WhisperModel` or `model.transcribe` is now pinned against a fake in `tests/infrastructure/test_whisper.py`. The existing tests there run real inference and cannot do this job: a wrong `beam_size` still transcribes a two-second fixture perfectly. A blank or whitespace-only `transcription_language` is read as "detect" rather than passed to the engine, which rejects it. `guild_config` is a table operators are told they may edit with SQL, where `ConfigStore.set`'s validation never runs, and one careless UPDATE should not fail every job of a guild.
The resource figures were derived rather than observed, which is the right
way to arrive at a first number and the wrong way to leave one. Measured
on an x86 CPU pinned to four threads, transcribing 4.7 minutes of German
speech:
large-v3-turbo int8 beam 5 61.6s 1.96GB
large-v3 int8_float32 beam 8 145.0s 4.11GB
4.11GB against the 4.25GB the comment worked out, so the arithmetic holds
and the 4Gi request / 6Gi limit stand as chosen.
The throughput number is the more useful of the two, because it bounds
something the comment could only gesture at: 282s of speech in 145s is
1.94x faster than real time, against 4.6x before. Since `vad_filter`
decodes only the speech in a track rather than its padded length, the
1800s job lease covers roughly 58 minutes of one person actually talking.
Past that the lease expires mid-job -- harmless at `replicas: 1` with the
worker taking one job at a time, since nothing else can claim it, and the
reason raising the worker beyond one replica means revisiting the lease
first.
The same run is also the clearest evidence for the model change itself.
The old configuration produced "Der Ender läuft inzwischen auf dem
Kluster, aber die Speicherlimits sind noch zu knapp gemessen"; the new one
produced "Der Renderer läuft inzwischen auf dem Cluster, aber die
Speicherlimits sind noch zu knapp bemessen".
The VAD fix (#45) landed first and rewrote the same method this branch changes, so the two met in `_transcribe`. The fix's structure is the trunk: audio is decoded once in the engine, the stateless gate produces `clip_timestamps`, and an empty clip list short-circuits before the model is touched. This branch's `initial_prompt` is threaded through it. `test_the_hallucination_guards_are_still_in_place` now asserts `vad_filter is False`, the reverse of what it asserted when written. That is not a weakened test: Silero was the guard until it was found to be the defect -- its recurrent state collapses on the bit-exact zero padding `SpeakerWriter` writes, and it reported about a second of speech in two minutes of a real recording. Turning it back on would restore the defect, not a safeguard, so the test now pins the arrangement that replaced it: no Silero, a non-empty clip list, and the two decoder-side thresholds. The two fakes coexist deliberately. `_RecordingModel` is handed to an engine built with `object.__new__`, so it can never see constructor arguments; `_ModelSpy` patches the imported name and records both. They answer different questions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first protocols Sturnus published transcribed badly. This is the configuration side of that: language, vocabulary, decoding, model.
Measured, not argued
Same 4.7 minutes of German speech, x86 CPU pinned to four threads (matching the worker's
cpu: "4"):Three errors in one sentence, all three gone.
4.11 GB against the 4.25 GB the chart comment works out on paper, so the arithmetic holds and the 4Gi request / 6Gi limit stand. 282s of speech in 145s is 1.94× faster than real time, against 4.6× before.
What changed
en. On a German-speaking server that alone accounts for a great deal, and it compounded: with nothing pinned, Whisper guessed,set_detected_languagewrote the guess down, and every later job for that speaker in that session was then bound to it. One unlucky guess was not one bad job, it was all of them. Now a per-guildtranscription_language(defaultde), and a configured language beats a previously stored detection — which is what actually breaks that chain.initial_prompt, per-guild, defaulting to a German sentence naming the bird projects and the stack. Proper nouns are where a general model fails and where a protocol is judged: a decision recorded about "Dracula" instead of "Ducula" is worse than no minutes.condition_on_previous_text=False. The library default feeds each segment's text in as the next one's prompt, so one hallucination becomes the context for everything after it — andvad_filtermakes that worse rather than better here, since it cuts a speaker's track into fragments whose "previous text" is routinely minutes earlier and about something else.beam_size=8,large-v3,int8_float32, resources raised to 4Gi/6Gi,startupProbe.failureThreshold60 → 120.The existing
vad_filter/compression_ratio_threshold/no_speech_thresholdare untouched and now have a regression test.The one addition beyond the brief — worth reviewing
transcription_languageacceptsauto. Without it there would be no state in which detection can run at all:DEFAULTSsuppliesde, and/config clearrestores a default rather than removing it, so the whole detect-and-pin mechanism and thedetected_languagecolumn would have become dead code.autois how a genuinely multilingual guild opts back in. A blank or whitespace value reads the same way —guild_configis a table operators are told they may edit with SQL, whereConfigStore.set's validation never runs, and""handed to faster-whisper fails every job of that guild.When a language is configured, detection does not run and nothing is written to
detected_language: a configured value overridden by a guess would be self-locking, and the column would stop meaning "what the engine detected".Verification
600 tests pass (was 576; the 4
slowmodel-downloading tests were not run and none added), mypy and ruff clean, chart lints and renders. Fifteen mutations, each caught by a named test.One mutation survived and was treated as a finding: the first language guard read
if detecting and pinned_language is None:, where the first conjunct is provably dead. Removed, and the blank-value handling above came out of looking at it.Worth second-guessing
autoas a sentinel — one more concept in exchange for detection staying reachable.STURNUS_JOB_LEASE_SECONDSis 1800s. At the measured 1.94× real time, and sincevad_filterdecodes only the speech in a track rather than its padded length, that covers roughly 58 minutes of one person actually talking. Past that the lease expires mid-job. Harmless today —replicas: 1is hardcoded in the Deployment template and the worker takes one job at a time, so nothing else can claim it, and the only effect is that a restart may reclaim a job that was still running. Raising the worker beyond one replica is what turns this into two workers transcribing the same recording, and is the moment to revisit the lease. Documented in the chart rather than changed here.startupProbebudget is for.