Discussion issue, applies to every sink backend (fs, mongo, and the PostgreSQL sink proposed in #1687).
Every push row stores the complete diff twice. getDiff writes it to both steps[].logs (via step.log(diff)) and steps[].content (via step.setContent(diff)), and writeAudit persists the whole action document as-is in all backends. Large pushes therefore produce multi-megabyte rows, and everything that touches push documents pays for it:
- The repo activity rollup scans all push rows on every repos page. Indexing can make the scan cheap on PostgreSQL, but it remains O(number of pushes) on every backend, and on mongo/fs it reads the full documents.
- List endpoints and exports ship or filter over fat documents.
- Storage grows with diff size rather than with metadata size.
Ideas worth weighing, none of which are backend-specific:
- Store the diff once, not twice: drop the duplicate from
steps[].logs or make setContent and the log reference one value.
- Move diff content out of the push document into its own store or table, loaded only by the push-detail view.
- A retention or truncation policy for step logs and diff content on old, settled pushes.
- Maintain the repo activity rollup incrementally (a counters document or table updated on push transitions) instead of recomputing it by scanning all pushes.
Filing as a design discussion before any implementation; the immediate, index-level mitigations for the PostgreSQL sink are tracked separately.
Discussion issue, applies to every sink backend (fs, mongo, and the PostgreSQL sink proposed in #1687).
Every push row stores the complete diff twice.
getDiffwrites it to bothsteps[].logs(viastep.log(diff)) andsteps[].content(viastep.setContent(diff)), andwriteAuditpersists the whole action document as-is in all backends. Large pushes therefore produce multi-megabyte rows, and everything that touches push documents pays for it:Ideas worth weighing, none of which are backend-specific:
steps[].logsor makesetContentand the log reference one value.Filing as a design discussion before any implementation; the immediate, index-level mitigations for the PostgreSQL sink are tracked separately.