auto_index watcher leaves orphaned full-size .db.stage.<random> temp files (incl. nested .stage.<A>.stage.<B>), filling the disk

[darkkid0]

Summary

With auto_index = true, the background watcher periodically re-indexes watched projects. Each write of the persisted graph DB leaves behind a full-size .db.stage.<random> temporary file, and in at least one case a nested .db.stage.<A>.stage.<B> copy was also created. For a large repo these temp files are ~1.1 GB each, so a few re-index cycles silently fill the system disk. The temp files are never reclaimed even after a subsequent re-index succeeds and commits a fresh X.db.

This looks like a variant of the re-index write-amplification family (#1083 WAL unbounded, #937 watcher re-index write amplification, #897/#1016 stale WAL/SHM sidecars), but the specific orphaned .db.stage.* temp files (including nested staging) does not appear to have a dedicated report.

Environment

• codebase-memory-mcp 0.10.8
• macOS (Apple Silicon), installed native binary
• CBM_CACHE_DIR / cache root: ~/.cache/codebase-memory-mcp
• Config: auto_index = true (single key/value in _config.db)
• Affected project: <VOLUME>-<PROJECT> — a large private repo whose persisted graph DB is ~1.1 GB (names redacted for privacy)

Evidence (current cache root — repo-identifying names redacted)

$ ls -la *<PROJECT>*
-rw-------  ...  1206976512  8 27 00:32  <VOLUME>-<PROJECT>.db
-rw-------  ...  1206583296  8 27 00:17  <VOLUME>-<PROJECT>.db.stage.mjQU5l
-rw-------  ...  1206583296  8 27 00:17  <VOLUME>-<PROJECT>.db.stage.mjQU5l.stage.VgdLsO
-rw-------  ...       32768  8 27 00:17  <VOLUME>-<PROJECT>.db.stage.mjQU5l.stage.VgdLsO-shm
-rw-------  ...     7932792  8 27 00:17  <VOLUME>-<PROJECT>.db.stage.mjQU5l.stage.VgdLsO-wal

Observations:

1. Orphaned temp, older than the committed DB. The committed X.db has mtime 00:32, but the .stage.mjQU5l temp has mtime 00:17. So a later re-index committed a fresh X.db, yet the earlier 00:17 .stage copy (and its nested copy) were never deleted — pure leak.
2. Nested staging. X.db.stage.mjQU5l.stage.VgdLsO (+ -shm/-wal) means the writer staged its own temp file a second time. This indicates the atomic "write temp → rename" path is itself being re-staged during a re-index, multiplying the on-disk footprint (here: 1.1 GB × 3 copies ≈ 3.3 GB for one project).
3. No live handle. lsof shows no process currently holds these .stage.* files, so they are safe to delete — they are leftovers, not in-flight writes.
4. Smaller watched repos do not carry leftover .stage.* files, consistent with the leak scaling with graph size × re-index frequency.
5. logs/cbm-daemon.log shows the watcher firing watcher.changed repeatedly (strategy=git) for the watched projects.

Suspected mechanism

The persisted graph DB is written atomically: <project>.db.stage.<random> is written, then renamed over <project>.db. With auto_index on, the watcher re-indexes on every git/file change. When re-indexes overlap or a later write lands before an earlier temp is cleaned up, the temp (and worse, a nested temp-of-temp) is left behind and never reclaimed. Because each temp is a full copy of the ~1.1 GB graph, disk usage grows without bound across idle re-index cycles.

Impact

• Unbounded disk consumption (multiple full-size copies of multi-GB graphs) until the system drive is full, which then also breaks daemon startup (cf. Daemon start fails with daemon.ipc.listen_failed stage=pending_publication when /tmp is full; clients get a misleading "active or starting" 30s timeout and MCP hosts silently drop the server #1828 "/tmp is full").
• Silent: no log warning, no auto-cleanup of orphaned stages.

Suggested fixes

1. On commit/rename, unlink any pre-existing <project>.db.stage.* in the cache root (and never let the staging glob match *.stage.* inputs, to prevent nested temp-of-temp re-staging).
2. Enforce a single in-flight index writer per DB (file lock) so overlapping re-indexes don't each emit a temp (also addresses WAL grows unbounded (115 GB in 4.5 h, fills system drive): checkpoint starvation from accumulated per-conversation servers + 11 concurrent index workers on one repo #1083's concurrency).
3. Add a watcher/re-index storage budget (cf. feat(index): enforce opt-in storage resource limits #1725 / Bound indexing resource usage with configurable limits #1347) and warn/abort when staged + committed DBs exceed a threshold.
4. Optionally VACUUM INTO / compact and write directly to a single temp name with a guaranteed unlink on both success and failure paths.

Steps to reproduce

1. codebase-memory-mcp config set auto_index true
2. Index a large repo (graph DB ≥ 1 GB) and leave it watched.
3. Let the watcher trigger several re-indexes (or make file/git changes under the repo).
4. Inspect the cache root — orphaned *.db.stage.<random> files accumulate; under overlap a nested *.db.stage.<A>.stage.<B> appears.

Note on diagnostics

The README asks for logs/trajectory.ndjson, but no trajectory.ndjson / snapshot.json is present in this cache root (likely rotated/absent in 0.10.8 local layout). The file-system evidence above is from the live cache at report time and reproduces the leak deterministically.

Related: #1083, #937, #897, #1016, #1204, #1828