feat(worker): seal a stored protocol under a key the retention sweep cannot take

docs/operations.md

@@ -316,6 +316,18 @@ there isn't one to run. Changing `STURNUS_MASTER_KEY` today makes every
 previously wrapped data key permanently unreadable (see below), it does
 not rotate anything.
 
+**It also wraps things that are not recordings, and one of them outlives
+every recording.** Three kinds of material are sealed under it beyond the
+audio: an export destination's credential and a guild's OAuth client
+secret, both in the database and both bound to their guild and purpose;
+and — since export targets could store a rendered protocol — the
+protocol artefacts themselves in the object bucket. Those artefacts do
+*not* use a session data key. Each carries a data key of its own, wrapped
+under the master key inside the object, because a protocol is deliberately
+not subject to `audio_retention_days` and a key that the retention sweep
+ends the life of would end the protocol's life with it. §6 states that
+rule from the retention side.
+
 **Losing a master key destroys every recording it wrapped.** This is the
 single most consequential operational fact about this system, so it is
 stated here plainly: audio is only ever stored encrypted, and only the
@@ -1298,6 +1310,46 @@ this sweep is the only thing in the system that ends a recording's life.
 partial failure is retried on the next sweep rather than recorded as
 done.
 
+**The sweep does not touch a session's stored protocol, and it must not
+start.** A guild publishing to a `markdown` or `html` export target gets
+an artefact in the same bucket the recordings are in
+(`{prefix}/{session_id}/{target_id}.md`), and nothing in this system ever
+deletes it: not this sweep, not the console, not a re-export, which
+replaces the object at the same address rather than adding one. That is
+the intended rule and not an omission. `audio_retention_days` governs the
+*recording*; the record of the meeting deliberately outlives it, which is
+why a transcript endpoint answers `200` with `audio_available: false`
+rather than `404` once the audio is gone. A protocol swept because its
+recording expired would be a meeting's minutes deleted on the strength of
+a window that was never about them.
+
+Two consequences an operator has to plan for. **These objects accumulate**
+— one per session per object-store destination, tens of kilobytes each,
+and the bucket lifecycle rule that backstops the recordings must not be
+written broadly enough to catch them. And **their retention is a policy
+question this system does not answer**: if a deployment needs stored
+protocols to expire, that is a rule somebody has to state, and there is no
+setting for it today.
+
+They are encrypted, and **not under the recording's key**. Each artefact
+carries a data key of its own, generated when the artefact is written and
+wrapped under `STURNUS_MASTER_KEY` inside the object itself, bound to the
+guild it belongs to. So an artefact opens with the master key and the
+guild alone: it needs no row to have survived, and nothing the sweep
+destroys. Sealing one under the session data key instead would have made
+every stored export unreadable the day its recording's window closed —
+silently, and noticed only by whoever next opened an old document.
+
+**Protocols written before v0.17.0 are plaintext in that bucket.** The
+release that introduced export targets stored them unencrypted; nothing
+sweeps them, so they are still there. They are still served — refusing
+them would turn a link somebody already holds into a 404 over bytes that
+are already written — and each read logs `session.export_unsealed` at
+WARNING, which is how you watch that corpus drain. Re-publishing a session
+replaces its object with a sealed one at the same address; there is no
+bulk migration, deliberately, because it would need to re-render every
+protocol ever published.
+
 Because recordings outlive their transcription by weeks, not minutes, the
 retention period is not merely an implementation detail — **it belongs in
 the privacy policy shown to participants** (the `policy_url` document),

src/sturnus/application/export_formats.py

@@ -94,13 +94,20 @@ class RenderRequest:
 class ExportFormat:
     """One format: how a protocol is rendered, and what carries it away.
 
-    `media_type` and `file_extension` are read only by the object-store
-    family, which has to name the bytes it stores and the object it stores
-    them in. They are on every entry regardless, because "what kind of
-    document is this" is a property of the format rather than of the sink
-    that happens to take it -- `session_document.provider` records the
-    format name, and the console route that serves an artefact back reads
-    its media type from here rather than guessing from the key.
+    `file_extension` names the object an object-store destination stores
+    its bytes in; `media_type` names what those bytes *are* to a reader.
+    They are on every entry regardless, because "what kind of document is
+    this" is a property of the format rather than of the sink that happens
+    to take it -- `session_document.provider` records the format name, and
+    the console route that serves an artefact back reads its media type
+    from here rather than guessing from the key.
+
+    **The stored object does not carry the media type, and that is
+    deliberate.** An object-store artefact is a sealed envelope
+    (`sturnus.infrastructure.documents.artefacts`), so `text/markdown` is
+    true of the document and false of the object; the entry below is the
+    one place that answer lives, and it reaches a browser through the
+    console route rather than through S3 metadata.
     """
 
     name: str

src/sturnus/application/exporting.py

@@ -103,13 +103,25 @@ class Destination:
     configured target and the guild's `document_provider` setting for the
     legacy one, so nothing about what an existing guild writes to that
     column changes.
+
+    `guild_id` is carried rather than looked up because a sink may need to
+    say *whose* artefact this is at the moment it writes one: an
+    object-store destination seals its bytes under a key bound to the
+    guild and the purpose (`sturnus.infrastructure.crypto.secret_context`),
+    and a sink that had to ask a database which guild it was serving would
+    be a sink with a database. It is the same guild for every destination
+    of one publish -- they are that guild's destinations -- which is
+    precisely why it belongs on the destination rather than being threaded
+    through `publish_session` as a second parameter that could disagree
+    with it.
     """
 
     session_id: int
     target_id: int | None
     format: ExportFormat
     target: str
     provider: str
+    guild_id: int
 
 
 @dataclass(frozen=True, slots=True)
@@ -290,6 +302,11 @@ def destinations_for(
                 format=entry,
                 target=target.target,
                 provider=entry.name,
+                # The row's own guild, not one passed in beside it: this
+                # value ends up in the associated data that seals an
+                # object-store artefact, and a mismatch between it and
+                # the row would be an artefact nobody can open again.
+                guild_id=target.guild_id,
             )
         )
     if chosen:

src/sturnus/application/worker.py

@@ -498,6 +498,7 @@ async def _legacy_destination(
         # this is what `session.document_provider` has always been written
         # with, and nothing about an existing guild's rows changes here.
         provider=provider,
+        guild_id=guild,
     )
 
 

src/sturnus/console/adapters.py

@@ -2504,10 +2504,16 @@ async def _read(
         if target_id is not None:
             statement = statement.where(SessionDocumentRow.target_id == target_id)
         async with self._session_factory() as db:
-            exists = await db.scalar(
-                select(select(SessionRow.id).where(SessionRow.id == session_id).exists())
+            # The guild, from the statement that was already establishing
+            # that the session exists. A read model carries it because an
+            # object-store artefact is sealed under a key bound to it
+            # (`sturnus.domain.exports.SessionDocument`), and asking a
+            # second time would be a second query for a column this one
+            # is already looking at.
+            guild_id = await db.scalar(
+                select(SessionRow.guild_id).where(SessionRow.id == session_id)
             )
-            if not exists:
+            if guild_id is None:
                 return None
             rows = await db.scalars(
                 # Publication order, so the list reads as the history it
@@ -2518,6 +2524,7 @@ async def _read(
             return tuple(
                 SessionDocument(
                     session_id=row.session_id,
+                    guild_id=guild_id,
                     target_id=row.target_id,
                     provider=row.provider,
                     document_id=row.document_id,

src/sturnus/console/ports.py

@@ -448,14 +448,28 @@ async def document_of(self, session_id: int, target_id: int) -> SessionDocument
 
 
 class DocumentArtefacts(Protocol):
-    """Where a stored protocol's bytes are read from. `S3DocumentStore`.
+    """Where a stored protocol's bytes are read from. `SealedArtefacts`.
 
     `KeyError` for an object that is not there, which is an ordinary
     outcome rather than a fault: a re-export can move nothing, but a
     destination removed from the bucket by hand leaves the row behind.
+    `sturnus.domain.errors.UnreadableArtefact` for one that is there and
+    does not open, which is not ordinary -- see that class for why the two
+    are different exceptions rather than one.
+
+    `guild_id` is what the artefact's own key is bound to, so it is the
+    caller's to supply and it comes off the `SessionDocument` row rather
+    than out of the object. An envelope carrying the guild it was filed
+    under would authenticate just as happily after being moved to another
+    guild's, which is the move this binding exists to fail.
+
+    The adapter behind this holds the master key and this port says
+    nothing about it: the console decides *who* may read a protocol, and
+    what it takes to turn an object into bytes is not a question a route
+    should be able to ask.
     """
 
-    async def get(self, key: str) -> bytes: ...
+    async def get(self, key: str, *, guild_id: int) -> bytes: ...
 
 
 class AdminDirectory(Protocol):

src/sturnus/console/routes_documents.py

@@ -28,6 +28,15 @@
 distinct one would confirm which sessions the system holds and where a guild
 publishes, to somebody with no business knowing either.
 
+**The bytes in the bucket are sealed, and this is where they are opened.**
+The rule above is what this process enforces; the encryption is a property
+of the object, and the two answer different questions about a bucket
+somebody has a copy of. Which key an artefact is sealed under -- its own,
+never the recording's -- is
+`sturnus.infrastructure.documents.artefacts`' decision and its docstring's
+argument; all that reaches this module is a `guild_id` off the row and a
+second way for the read to fail.
+
 **An Outline document is not served here.** Its bytes live in Outline, and
 the listing carries its URL so the console can link straight out. Only the
 formats whose sink family is the object store have anything for this route
@@ -44,6 +53,7 @@
 
 from sturnus.application.export_formats import OBJECT_STORE_SINK, format_named
 from sturnus.console.ports import DocumentArtefacts, SessionDocumentDirectory
+from sturnus.domain.errors import UnreadableArtefact
 from sturnus.domain.exports import SessionDocument
 from sturnus.observability.events import Event, log_event
 
@@ -171,7 +181,9 @@ async def read_document(request: web.Request) -> web.StreamResponse:
     # media type to `str | None` for a case that cannot happen.
     assert entry is not None
     try:
-        body = await request.app[DOCUMENT_ARTEFACTS].get(document.document_id)
+        body = await request.app[DOCUMENT_ARTEFACTS].get(
+            document.document_id, guild_id=document.guild_id
+        )
     except KeyError:
         # The row outlived its object. Nothing is broken, so this is a 404
         # and not a 500 -- the same reading the audio route gives a
@@ -187,6 +199,26 @@ async def read_document(request: web.Request) -> web.StreamResponse:
             reason="artefact_erased",
         )
         raise _not_found() from None
+    except UnreadableArtefact:
+        # The object is there and does not open: a wrong master key, an
+        # envelope sealed under a guild this row does not name, a body
+        # edited in the bucket. The reader gets the same 404 -- there is
+        # nothing here to serve them either way -- and the log gets a
+        # different `reason`, because "the sweep took it" and "it failed
+        # to authenticate" are different mornings for whoever is on call.
+        # The exception itself is not logged: it is raised from an
+        # `InvalidTag`, whose only useful content is the fact of it.
+        log_event(
+            log,
+            logging.ERROR,
+            Event.CONSOLE_DOCUMENT_REFUSED,
+            "A protocol's object is in the store and did not open",
+            session_id=session_id,
+            target_id=target_id,
+            requested_by=caller,
+            reason="artefact_unreadable",
+        )
+        raise _not_found() from None
 
     log_event(
         log,

src/sturnus/domain/errors.py

@@ -10,6 +10,11 @@
 both are stdlib-only by construction so that every layer, `domain`
 included, can raise them.
 
+`UnreadableArtefact` is here for the same reason in the other direction:
+it is raised by an adapter in `sturnus.infrastructure` and caught by a
+route in `sturnus.console`, and a route reaching into the adapter package
+for an exception type would be the route knowing which adapter it has.
+
 Sturnus records people talking. Spec 12.4 -- "Neither audio data nor
 transcript content appears in logs" -- is the standard the pod logs are held
 to, and `sturnus.infrastructure.observability` holds Sentry to at least the
@@ -85,3 +90,28 @@ class CorruptRecording(Exception):
     from two modules now; see that class's docstring for what the claim
     costs to make honestly.
     """
+
+
+class UnreadableArtefact(Exception):
+    """A stored protocol is there, and these are not bytes to serve.
+
+    Raised where a sealed export artefact fails to open: a wrong master
+    key, an envelope that does not authenticate under the guild it was
+    found filed against, a body somebody edited in the bucket. Every one
+    of those is the same answer for the person asking -- there is no
+    document here -- and the console gives it the same 404 it gives an
+    artefact the object store no longer has.
+
+    **It is not `KeyError`, and that distinction is the reason the class
+    exists.** An object that is missing is an ordinary event with an
+    ordinary cause; an object that is present and will not open is
+    somebody's meeting failing to authenticate, which is the one outcome
+    an operator has to be able to find in a log. Folding it into
+    "artefact erased" would report an integrity failure as a tidy-up.
+
+    A sibling of `CorruptRecording` rather than the same class. That one
+    is about the stored *audio* format and is raised by two readers of it;
+    this is about the artefact envelope, which is a different format with
+    a different lifetime. Naming both "corrupt recording" would make the
+    one search an operator runs return the other's failures.
+    """

src/sturnus/domain/exports.py

@@ -75,9 +75,19 @@ class SessionDocument:
     publishing here", not "forget what was published": the document still
     exists in the other system, and the link is what somebody follows
     when they go looking for last quarter's minutes.
+
+    `guild_id` is the session's guild, joined rather than stored on the
+    row: `session_document` has no such column and does not need one,
+    because a document's guild is its session's and cannot be anything
+    else. It is on the read model because an object-store artefact is
+    sealed under a key bound to that guild, and the reader has to supply
+    that binding from its own context rather than from the object -- an
+    envelope that carried the guild it was filed under would authenticate
+    just as happily after being moved.
     """
 
     session_id: int
+    guild_id: int
     target_id: int | None
     provider: str
     document_id: str

src/sturnus/entrypoints/api.py

@@ -85,6 +85,7 @@
 from sturnus.infrastructure.db.models import GuildOAuthClient as GuildOAuthClientRow
 from sturnus.infrastructure.db.preferences import PreferenceStore
 from sturnus.infrastructure.db.setup_intents import SetupIntentStore
+from sturnus.infrastructure.documents.artefacts import SealedArtefacts
 from sturnus.infrastructure.documents.outline_oauth import OutlineOAuth
 from sturnus.infrastructure.objectstore import S3AudioStore, S3DocumentStore
 from sturnus.infrastructure.observability import init_sentry
@@ -336,11 +337,20 @@ def now() -> datetime:
         # no second class to say so.
         exports=ExportTargetStore(session_factory, keys),
         documents=ConsoleSessionDocuments(session_factory),
-        artefacts=S3DocumentStore(
-            settings.s3_endpoint,
-            settings.s3_bucket,
-            settings.s3_access_key.get_secret_value(),
-            settings.s3_secret_key.get_secret_value(),
+        # `keys` again: a stored protocol is sealed under a data key of
+        # its own, wrapped under this process's master key and bound to
+        # the guild. That this process already holds the master key --
+        # it decrypts audio on the way to the browser -- is why serving
+        # an artefact needs no credential the chart does not already give
+        # it (`charts/sturnus/templates/_helpers.tpl`).
+        artefacts=SealedArtefacts(
+            S3DocumentStore(
+                settings.s3_endpoint,
+                settings.s3_bucket,
+                settings.s3_access_key.get_secret_value(),
+                settings.s3_secret_key.get_secret_value(),
+            ),
+            keys,
         ),
         oauth_clients=ConsoleGuildOAuthClients(oauth_clients, admins),
         setup=ConsoleGuildSetup(session_factory, admins, SetupIntentStore(session_factory)),

src/sturnus/entrypoints/worker.py

@@ -99,6 +99,7 @@
     SessionRepository,
 )
 from sturnus.infrastructure.db.session_documents import SessionDocumentStore
+from sturnus.infrastructure.documents.artefacts import SealedArtefacts
 from sturnus.infrastructure.documents.outline import OutlineSink
 from sturnus.infrastructure.documents.sinks import DocumentSinks
 from sturnus.infrastructure.health import ReadinessState, start_health_server
@@ -619,12 +620,19 @@ async def _run() -> None:
     # The same bucket the recordings are in, through a class that reads and
     # writes whole small objects rather than streaming large encrypted ones
     # -- see `sturnus.infrastructure.objectstore`, which keeps the two
-    # apart on purpose.
-    document_objects = S3DocumentStore(
-        settings.s3_endpoint,
-        settings.s3_bucket,
-        settings.s3_access_key.get_secret_value(),
-        settings.s3_secret_key.get_secret_value(),
+    # apart on purpose -- and sealed on the way in. `keys` again, the same
+    # one the export targets' credentials are wrapped with: one decode of
+    # the master key in this process. An artefact is *not* sealed under
+    # the recording's data key, and `sturnus.infrastructure.documents.
+    # artefacts` is where that decision is argued.
+    document_objects = SealedArtefacts(
+        S3DocumentStore(
+            settings.s3_endpoint,
+            settings.s3_bucket,
+            settings.s3_access_key.get_secret_value(),
+            settings.s3_secret_key.get_secret_value(),
+        ),
+        keys,
     )
 
     # Tracing is applied here, on the way into `process_one`, and nowhere