Bug: Deferred large files can starve indefinitely under newest-first scan priority

[IchenDEV]

Summary

Deferred oversized Codex session files can permanently stall at their first-pass progress when the user's daily session volume consistently consumes the full per-refresh byte budget. The bounded scan mechanism introduced in #2452 correctly defers large files and persists resumable progress, but the newest-first priority order combined with high daily throughput creates a starvation condition where deferred files never receive additional budget.

Environment

• CodexBar: 0.55.0 (build 129)
• macOS: 27
• Codex CLI model: gpt-5.6-sol (Ultra mode, frequent sub-agent spawning)
• Daily session volume: ~133 files/day, ~1 GB/day
• Total session corpus: ~979 files across ~/.codex/sessions/ and ~/.codex/archived_sessions/

Observed behavior

Three files have scan_complete = 0 and have not advanced beyond their first-pass progress:

File	Size	Parsed	Progress	Stuck since
rollout-2026-08-19T19-05-32-*.jsonl	1,143 MB	256 MB	22.4%	7 days
rollout-2026-07-14T20-16-33-*.jsonl	28 MB	0.5 MB	1.8%	42 days
rollout-2026-08-26T12-45-09-*.jsonl	3 MB	0.8 MB	26.2%	today (in progress)

The 1.1 GB file is stuck at exactly 268,435,456 bytes = 2^28 = maxFileBytes (256 MiB), confirming the first pass consumed the per-file budget and stopped. Subsequent refreshes have not advanced it despite updated_at_ms showing recent attempts.

The 28 MB file at 0.5 MB parsed for 42 days is especially notable — it is well under maxFileBytes and should complete in a single pass, but the global budget is exhausted by higher-priority (newer) files before it is reached.

Root cause analysis

The starvation occurs due to the interaction of three design choices in CostUsageScanner:

1. Per-file budget (maxFileBytes = 268,435,456): each file reads at most 256 MiB per refresh.
2. Global budget (maxBytesPerRefresh = 536,870,912): all file I/O is capped at 512 MiB per refresh.
3. Newest-first ordering (preferNewestCodexSessionsFirst): files sorted by mtime descending.

With ~133 new session files/day (~1 GB), the global 512 MiB budget is consumed by fresh/recently-modified files on every refresh. Deferred files from older dates are queued but never reached:

Each refresh:
  1. Enumerate files → sort by mtime desc (newest first)
  2. Process new/changed files → consumes ~256-512 MiB of global budget
  3. Reach deferred large file → global budget exhausted → skip
  4. Repeat indefinitely

Suggested fix directions

1. Reserve a deferred-file budget slice. Split maxBytesPerRefresh into e.g. 384 MiB for new/changed files + 128 MiB reserved for deferred-in-progress files. Deferred files with persisted progress > 0 should be served from the reserved slice regardless of priority order. This guarantees forward progress on at least one deferred file per refresh.

2. Age-based priority boost. Track how many refreshes a deferred file has been skipped. After N skips (e.g. 10), promote it above newest-first ordering for one pass. This prevents indefinite starvation while preserving the general newest-first preference.

3. Small-file fast path. Files under maxFileBytes (like the 28 MB file stuck at 1.8%) should bypass the deferred queue entirely and be processed in a single pass, since they cannot exceed the per-file budget. The current code appears to defer them when the global budget is exhausted, even though completing them would cost minimal budget.

Direction 1 is the most straightforward and guarantees bounded completion time for any corpus size.

Diagnostic data

-- From cost-usage.sqlite
SELECT path, size, parsed_bytes, scan_complete, 
       datetime(updated_at_ms/1000, 'unixepoch') as last_update
FROM files WHERE scan_complete = 0;

-- The 1.1 GB file: parsed_bytes = 268435456 = exactly maxFileBytes
-- The 28 MB file: parsed_bytes = 524288, stuck since 2026-07-15
-- Both files have been attempted (updated_at_ms is recent for the 1.1GB one)

File characteristics of the 1.1 GB file:

• 34,338 lines total
• Average line: 48.3 KB
• Max line: 3.3 MB (3 lines > 1 MB in first 200 lines)
• Likely an Ultra-mode session with embedded patches/images

Related issues

• fix: bound Codex cost scans on giant session corpora #2452 — introduced the bounded scan mechanism (the correct fix for CPU storms, but created this starvation edge case)
• Codex cost catch-up never completes when old session files are touched into the active lookback window #3071 / Drain processed zero-row files from Codex cost catch-up #3075 — catch-up queue deadlock for zero-row files (fixed in v0.55.0, different root cause)
• Optimize Codex cost refresh policy for large history windows #1392 — optimize refresh policy for large history windows

Regression test sketch

@Test
func deferredFileReceivesBudgetAfterNewerFilesExhaustGlobalBudget() async throws {
    // 1. Create a large (> maxFileBytes) session file with an old mtime.
    // 2. Create enough newer session files to exhaust maxBytesPerRefresh.
    // 3. Run one refresh → large file should parse first 256 MiB slice.
    // 4. Run a second refresh → large file should advance beyond 256 MiB
    //    (currently fails: progress stays at 268435456).
}

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 26, 2026, 1:50 AM ET / 05:50 UTC.

Summary
Current main still permits an incomplete older session file to be deferred whenever newer dirty files exhaust the shared byte budget. This is a bounded, source-proven scanner fairness bug suitable for a focused repair PR.

Reproducibility: yes. from source: repeatedly present enough newer dirty session data to consume the 512 MiB shared budget while retaining an older incomplete cached file; its scan is deferred before any additional bytes can be parsed.

Regression provenance
Possible regression — probable (reproduction; reviewed change). No predecessor PR is attributed.

Next step
The current scheduler and an adjacent focused test expose a small, mechanical fairness repair with no open PR already owning it.

Review details

Best possible solution:

Schedule at least one persisted incomplete file for a bounded slice before newer dirty files consume the full shared budget, while preserving the existing limits and newest-first behavior for the remaining work.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: repeatedly present enough newer dirty session data to consume the 512 MiB shared budget while retaining an older incomplete cached file; its scan is deferred before any additional bytes can be parsed.

Is this the best way to solve the issue?

Yes. A narrow fairness admission for persisted incomplete files fixes the starvation without removing the performance limits introduced for giant corpora; the regression test should prove fresh work still receives the remaining budget.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The repair must retain the existing per-refresh ceiling and prompt visibility of recent usage; a fairness reservation should be narrow and covered by both fresh-work and resumed-work assertions.

Codex review notes: model internal, reasoning high; reviewed against dfc176cdf945.

Label changes

Label justifications:

• P2: Cost history can remain permanently incomplete for high-volume users, but the failure is bounded to the local cost scanner and has a focused repair path.
• impact:other: The issue causes stale and incomplete local cost-history results, which does not map to the more specific owned impact categories.

Evidence reviewed

Acceptance criteria:

• swift test --filter CostUsagePerformanceGateTests
• make check
• make test

What I checked:

• Current budget and priority behavior: The scanner defaults to a 256 MiB per-file slice, a 512 MiB shared refresh budget, and newest-first ordering; the stated workload can therefore exhaust the shared budget before an older incomplete file is reached. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:167, dfc176cdf945)
• Deferred work has no fairness admission: After candidate order is established, a file whose remaining work has no budget allowance returns deferred; no separate deferred-file allowance or rotation is applied. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:5114, dfc176cdf945)
• Selection preserves newest-first ordering: Bounded candidate selection sorts candidates newest-first before scanning, so persisted incomplete work does not receive priority over newer dirty files. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:3053, dfc176cdf945)
• Existing regression seam demonstrates the first half: The current performance test explicitly asserts that a refresh budget processes the newest dirty file and defers older files, but it does not exercise repeated refreshes with continuing newer work. (Tests/CodexBarTests/CostUsagePerformanceGateTests.swift:1566, dfc176cdf945)
• Feature provenance: The bounded, resumable scan machinery entered through merged PR 2452; its merge commit is the relevant feature-history anchor, while the later merged zero-row queue fix addresses a distinct completion condition. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:209, 53376361a83e)
• Current-main and release check: Current main is after v0.55.0 but has no release tag containing its head; the reported behavior is not fixed by a released or current-main fairness change. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:5462, dfc176cdf945)

Likely related people:

• D4ilyHub: Authored the merged bounded-scan PR that established the per-file/per-refresh budget and resumable progress mechanism implicated here. (role: introduced bounded scan behavior; confidence: high; commits: 53376361a83e; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift)
• Peter Steinberger: Merged the original bounded-scan work and authored the recent adjacent catch-up completion repair. (role: recent scanner contributor and merger; confidence: high; commits: 53376361a83e, 114a6a0921a3; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[steipete]

Fixed on main by #3207, merged as 2a38c2c. Older partial sessions now receive a fair persisted turn within the existing scan limits instead of repeatedly losing to fresh or growing files. The repair preserves compatible cached rows, resume state and correctly scoped totals.

The regression failed against the submitted scheduler, then passed through the production JSONL/SQLite pipeline. Both byte-only and timed fixtures advanced all 514 waiting files and drained to the same 1,034-file totals as an independent unbounded scan. All 249 focused tests and the full 938-selection/79-group suite passed; exact-head macOS/Linux CI and independent review are green. Tests used synthetic temporary files, not your live session corpus.

This is fixed on main, not yet in a new release. Thanks @IchenDEV for the report and original contribution.