Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,6 +441,51 @@ stored is actually what is running.

## 5. Troubleshooting

**A speaker's audio arrives with no level.** During a meeting the bot may
post, into the recording channel and naming the person:

> Audio is arriving from @Name but at no audible level. The microphone is
> most likely muted at system level. Recording continues.

That message means something narrower than "it is quiet", and the
distinction is the whole point. Three states exist and only the third is
reported:

| What is happening | Reported? |
|---|---|
| No packets at all from that speaker | No — they are not speaking, which is normal |
| Packets arrive but will not decode | No — that is `EndReason.DECODE_FAILURE`, section 5 below |
| Packets arrive, decode, and every sample stays at the noise floor | **Yes** |

It fires after 30 seconds of *received* audio at the noise floor
(`SILENCE_EVIDENCE_SECONDS` in `sturnus/domain/silence.py`), counted in
bytes of PCM rather than wall-clock — somebody who transmits nothing for
half an hour has produced no evidence about their microphone and is never
warned. Any audible packet discards that speaker's evidence and starts
over. Once per speaker per session, never repeated.

The usual cause is a microphone muted below Discord: in the operating
system's mixer, on the device itself, or a hardware mute switch. Discord's
own mute sends no packets at all and therefore never triggers this.

It is also recorded, so it can be seen after the fact:

```sql
SELECT discord_display_name, silent_audio_detected_at
FROM session_participant WHERE session_id = <id>;
```

A row with `silent_audio_detected_at` set will have contributed an empty
or near-empty transcript. That is the signal to look at before concluding
that Whisper failed: a transcript of one short hallucinated phrase over a
long recording is what silence looks like after transcription, not what a
broken model looks like.

Only amplitude is ever measured. No sample is buffered, logged or sent
anywhere, which is why this applies equally to people who have not
consented — a peak is a number about loudness, not about content.


**A job is `dead`.** `transcription_job.status` becomes `dead` once
`attempts` reaches the worker's configured retry limit
(`JobQueue.fail`). A dead job is deliberately excluded from the
Expand Down
46 changes: 46 additions & 0 deletions migrations/versions/0006_participant_silent_audio.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
"""participant silent audio

Revision ID: 0006
Revises: 0005
Create Date: 2026-08-20 00:00:00.000000

Records when a speaker's audio was first seen arriving with no audible
level in it -- packets received and decoded, every sample at the noise
floor. The bot writes it during the session, at the same moment it says so
in the channel, because that message is gone by the next meeting and this
column is what an operator can still read afterwards.

Two live sessions produced empty transcripts from full-length recordings,
and nothing in the database distinguished "we could not hear them" from
"they said nothing" -- both leave exactly the same participant row. This is
the column that answers it.

Nullable, and deliberately not backfilled: being quiet is normal and null
is what nearly every participant will always carry, so there is nothing to
fill in for the sessions that predate this column -- their audio is gone,
and no value could be honestly inferred from what is left.
"""

from typing import Sequence, Union

import sqlalchemy as sa
from alembic import op

# revision identifiers, used by Alembic.
revision: str = "0006"
down_revision: Union[str, Sequence[str], None] = "0005"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
"""Upgrade schema."""
op.add_column(
"session_participant",
sa.Column("silent_audio_detected_at", sa.DateTime(timezone=True), nullable=True),
)


def downgrade() -> None:
"""Downgrade schema."""
op.drop_column("session_participant", "silent_audio_detected_at")
36 changes: 36 additions & 0 deletions src/sturnus/application/publishing.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,42 @@ def render_announcement(
return template.render(document_url=document_url)


#: Wording for the in-meeting warning that a speaker's audio is arriving
#: with nothing audible in it (`sturnus.domain.silence`). Posted publicly
#: into the recording channel and naming the person, which is a deliberate
#: choice over a direct message: whoever is muted at system level is
#: usually the last to notice, and somebody else in the room can help.
#:
#: Every word of it is load-bearing. It states what was observed rather
#: than what somebody did wrong, it names the one cause that explains
#: audio arriving at zero level, and it says the recording continues --
#: without that last sentence the message reads as "you are not being
#: recorded", which is false and would send people out of the meeting to
#: fix something that is not broken.
#:
#: Rendered through the same sandboxed engine as the announcement above,
#: for the same reason (Spec 8.2: one engine for every Discord message
#: this system posts), so the wording can move to a per-guild template
#: later without a code change.
SILENT_AUDIO_WARNING_TEMPLATE = (
"Audio is arriving from <@{{ discord_user_id }}> but at no audible level. "
"The microphone is most likely muted at system level. Recording continues."
)


def render_silent_audio_warning(
discord_user_id: int, template_source: str = SILENT_AUDIO_WARNING_TEMPLATE
) -> str:
"""Renders the public warning for one speaker whose audio carries no level.

`<@id>` is Discord's mention syntax; the id is an integer taken from
the voice packet itself, so nothing user-controlled reaches the
template even though this environment does not autoescape.
"""
template = _build_environment().from_string(template_source)
return template.render(discord_user_id=discord_user_id)


class SessionReader(Protocol):
"""Where announcement candidates are read from and `announced_at` is stamped."""

Expand Down
102 changes: 99 additions & 3 deletions src/sturnus/application/recording.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from __future__ import annotations

import contextlib
import logging
from datetime import datetime, timedelta
from pathlib import Path
from typing import Protocol
Expand All @@ -22,9 +23,13 @@
Encryptor,
SessionKey,
)
from sturnus.application.publishing import Announcer, render_silent_audio_warning
from sturnus.domain.session import EndReason, SessionMachine, SessionState, SessionTimeouts
from sturnus.domain.silence import SILENCE_EVIDENCE_SECONDS, SilentAudioWatch
from sturnus.domain.timeline import SpeakerClock

log = logging.getLogger(__name__)


def audio_key(session_id: int, discord_user_id: int) -> str:
"""Object key for one speaker's recording within a session.
Expand Down Expand Up @@ -53,6 +58,19 @@ async def set_audio_epoch(
self, session_id: int, discord_user_id: int, at: datetime
) -> None: ...

async def record_silent_audio(
self, session_id: int, discord_user_id: int, at: datetime
) -> None:
"""Records that this speaker's audio arrived carrying no audible level.

Written once, at the moment the case is established, so that an
operator reading the session afterwards can tell a broken capture
path from a room in which nobody said anything -- the question two
empty transcripts left unanswerable. The warning posted into the
channel is gone by the next meeting; this is what survives it.
"""
...

async def close_session(self, session_id: int, ended_at: datetime, reason: str) -> None: ...

async def record_session_key(
Expand Down Expand Up @@ -113,9 +131,14 @@ class RecordingService:
packets on an absolute timeline, and one `AudioWriter` per speaker who
has actually spoken. Every collaborator that reaches outward -- the
session and job repositories, the object store, the audio writers, the
encryptor -- is reached through a port or a narrow protocol, so this
class never touches Discord, SQL, S3, the filesystem's audio format, or
a crypto library directly.
encryptor, the announcer -- is reached through a port or a narrow
protocol, so this class never touches Discord, SQL, S3, the
filesystem's audio format, or a crypto library directly.

It speaks into the channel exactly once, and only about a fault it can
see and the room cannot: audio arriving from a speaker with no audible
level in it (`_report_silent_audio`). Everything else it has to say
about a session it says by writing it down.
"""

def __init__(
Expand All @@ -128,6 +151,7 @@ def __init__(
store: AudioStore,
writers: AudioWriterFactory,
encryptor: Encryptor,
announcer: Announcer,
retention_days: int,
channel_name: str | None = None,
) -> None:
Expand All @@ -144,8 +168,18 @@ def __init__(
self._store = store
self._writer_factory = writers
self._encryptor = encryptor
#: The only way this service says anything to the people it is
#: recording. Reached through the same `Announcer` port the link
#: publisher already posts through, rather than a second route out
#: to Discord, so this layer still has no idea Discord exists.
self._announcer = announcer
self._retention_days = retention_days

#: Watches each speaker's amplitude for audio that arrives and
#: decodes but carries nothing (`sturnus.domain.silence`). Rebuilt
#: per session in `reset()`, like `_clock`, so "once per speaker"
#: means once per session.
self._silence = SilentAudioWatch()
self._session_id: int | None = None
self._data_key: SessionKey | None = None
self._writers: dict[int, AudioWriter] = {}
Expand Down Expand Up @@ -272,6 +306,67 @@ async def voice_packet(
writer.write(at, pcm)
self._machine.audio_received(now)

# Last, and only after the audio itself is safely written: this is
# a report about the recording, never a step in making it. The
# watch reads amplitude and nothing else -- no sample is kept,
# logged or passed on -- and answers `True` exactly once per
# speaker per session, on the packet that completes the case.
if self._silence.observe(discord_user_id, pcm):
await self._report_silent_audio(discord_user_id, display_name, at)

async def _report_silent_audio(
self, discord_user_id: int, display_name: str, at: datetime
) -> None:
"""Says, three ways, that this speaker's audio is arriving empty.

Three, because each survives something the others do not. The log
line reaches the operator watching the pod and is the one thing
that cannot itself fail. The channel message reaches the meeting
while it can still act -- at the end it would be worthless, since
the recording is already lost. The participant row outlives both
and is what turns "the transcript is empty" into an answerable
question weeks later.

Each of the two that can fail is guarded on its own rather than
together: a Discord rate limit must not swallow the durable
record, and a database hiccup must not swallow the message that
could still get somebody's microphone fixed. Neither may reach the
caller at all -- `voice_packet` is the capture path, and a warning
that took the recording down with it would be worse than no
warning.
"""
assert self._session_id is not None
log.warning(
"Audio from %s (id %d) in session %d has been arriving for %ds with no "
"audible level: packets are being received and decoded, and every sample in "
"them is at the noise floor. This is what a microphone muted at system level "
"produces, and it transcribes to nothing. Recording continues.",
display_name,
discord_user_id,
self._session_id,
SILENCE_EVIDENCE_SECONDS,
)
try:
await self._announcer.post(
self._channel_id, render_silent_audio_warning(discord_user_id)
)
except Exception as exc:
log.warning(
"Could not post the silent-audio warning for id %d into channel %d: %s",
discord_user_id,
self._channel_id,
exc,
)
try:
await self._sessions.record_silent_audio(self._session_id, discord_user_id, at)
except Exception as exc:
log.warning(
"Could not record silent audio for id %d on session %d: %s",
discord_user_id,
self._session_id,
exc,
)

def request_close(self, reason: EndReason) -> None:
"""Arms an out-of-band close that the next `tick()` acts on.

Expand Down Expand Up @@ -413,6 +508,7 @@ def reset(self) -> None:
assert self._closed, "reset() must only follow close()"
self._machine.reset()
self._clock = SpeakerClock()
self._silence = SilentAudioWatch()
self._session_id = None
self._data_key = None
self._writers = {}
Expand Down
18 changes: 18 additions & 0 deletions src/sturnus/application/recovery.py
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,23 @@ def open(self, session_id: int, discord_user_id: int, epoch: datetime) -> _Alrea
)


class _UnusedAnnouncer:
"""Satisfies `RecordingService`'s constructor without ever posting anything.

Recovery runs at boot, before the Discord client is even constructed,
and only ever calls `close` -- the one method that says nothing to
anybody. The service posts from `voice_packet` alone, which recovery
never reaches, so being asked to post here means the wiring above has
gone wrong and the process would otherwise be quietly announcing into
a channel it has no connection to.
"""

async def post(self, channel_id: int, text: str) -> None:
raise RuntimeError(
f"recovery must not announce anything (channel {channel_id}, {len(text)} chars)"
)


def _service_for_recovery(
sessions: SessionRecorder,
jobs: JobQueue,
Expand All @@ -155,6 +172,7 @@ def _service_for_recovery(
store=store,
writers=_UnusedWriterFactory(),
encryptor=encryptor,
announcer=_UnusedAnnouncer(),
retention_days=retention_days,
)

Expand Down
Loading
Loading