status: fix FSMonitor history and clean-proof gaps - #74
Open
ttaylorr-oai wants to merge 11 commits into
Open
Conversation
ttaylorr-oai
marked this pull request as draft
August 26, 2026 18:53
ttaylorr-oai
marked this pull request as ready for review
August 26, 2026 19:17
with_lock__wait_for_cookie() gives a filesystem provider one second to report a synchronization cookie. A healthy FSEvents stream can miss that deadline while macOS is under load. The daemon then returns a trivial response, and status scans the entire index even though event delivery is still making progress. 4b1c56a (fsmonitor: flush pending FSEvents before cookie wait, 2026-07-21) requested an asynchronous flush on every Darwin query but kept the same one-second deadline. f439708 (Revert "fsmonitor: flush pending FSEvents before cookie wait", 2026-08-17) reverted it after a matched 48-query test still saw 12 timeouts in each arm. Avoid restoring that unqualified hot-path request. When the initial Darwin wait expires, request an asynchronous FSEvents flush and wait one more bounded interval. Successful queries retain the original wait and do not issue a flush or extend their deadline. The asynchronous call cannot block on the callback while the client holds main_lock. If the provider stays silent, retain the existing trivial-response fallback after the retry. Add a test-only callback delay to exercise both outcomes: a 1.2-second delay is recovered, while a 2.5-second delay still reaches the bounded fallback.
The daemon currently assumes that each client which advances an FSMonitor token also updates the repository's canonical index. That does not hold for commands using GIT_INDEX_FILE. A private index can advance the daemon past the canonical index's token and cause the canonical index's next query to receive a global invalidation. Keep a deduplicated overflow batch instead of discarding old paths. Clients at the overflow sequence still get an exact delta. Older clients get a conservative union of paths, which may overreport but cannot miss a change. All paths are interned. Keep a pointer-identity hash set with the overflow batch so later compactions hash only newly retired paths, rather than rebuilding a set over the daemon's lifetime history. Add a regression which advances a private index repeatedly, verifies that compaction remains deduplicated, and then checks that a read-only canonical status reports both changed files without a trivial response.
Retired batches are collapsed into a path-only overflow set. That keeps old indexes complete, but it loses the sequence in which each path was last observed. A client that consumed an inode event can therefore see it again after another index compacts the batch list, causing repeated hard-link scans. Unpinned batches have a zero pinned time and are also eligible for compaction immediately despite the default grace period. Do not use unpinned batches as truncation boundaries. Record the newest original batch sequence for every overflow path, and filter overflow responses against the client's requested sequence. The normal batch walk remains unchanged; sequence lookups are confined to overflow responses. Cover both the default retention grace and the cross-index hard-link case. The latter persists a nonzero checkpoint, compacts through a private index, and verifies repeated canonical reads do not rescan or fall back to global invalidation.
The delayed-cookie tests send the v1 timestamp token "0" and only check that the response is nonempty. Both recovery and fallback can satisfy that assertion with the same trivial response, so the tests do not distinguish a rescued cookie from a token-generation reset. Send a deterministic valid v2 token instead. Verify that the 1200ms case preserves its token generation without a global invalidation, while the 2500ms case changes generation and sends the fallback invalidation.
215845a (fsmonitor: preserve authenticated proofs across ordinary commands, 2026-08-15) enabled the clean-status history handoff for merges, but excluded invocations where fast_forward was FF_NO. Requested merge topology does not determine whether the resulting index is semantically safe. A clean non-fast-forward merge can carry the same authenticated FSUC/FSCF state as a fast-forward merge. As a result, --no-ff, --no-ff --no-commit, and merge.ff=false all dropped FSUC and reduced the FSCF flags from 15 to 9 after a clean merge. Each subsequent read-only status invalidated the external history and rescanned the semantic manifest. Enable the handoff for every merge using the canonical index. Conflict handling still invalidates unsafe proofs, and explicit alternate indexes remain excluded. Cover all three non-fast-forward forms, repeated read-only status calls, conflicts, and alternate indexes.
An exact clean status can repair a stale FSMonitor checkpoint or cached stat data while it scans. The repair requires an index write, so the existing issue path leaves no clean sidecar behind. Read-only callers then repeat the full scan until a second writable exact status publishes the proof. After the repair is written and resumable history is durable, install a sidecar bound to the rewritten index. Keep optional-lock-disabled commands read-only, preserve the literal exact-command restriction, and do not extend sidecar support to linked worktrees. Cover repeated read-only scans after a legacy daemon replacement, the single writable index repair in main and linked worktrees, and the next read-only sidecar hit in the main worktree. Keep option-bearing status commands ineligible for proof publication.
A configured pull can discard each layer of authenticated status history even when worktree inputs remain unchanged. Command-scoped protocol and HTTP settings change the config digest, directory events with more than 64 tracked descendants reject the semantic proof, and a fast-forward which adds an indexed directory drops the paired untracked cache. The next status can consequently preload and refresh the full index. Treat command-scoped protocol and HTTP settings as transport-only. For a large directory event, authenticate each distinct attribute source once instead of rejecting the cone outright. When a checkout adds tracked paths, retain the paired untracked cache and replay those additions through its existing invalidation path. Cover configured pulls in main and linked worktrees, large directory events, nested attribute-source changes, and branch switches which add tracked directories. The conservative full-scan fallback remains in place when an attribute source changes.
Configured pulls preserve FSMonitor clean proofs when checkout can authenticate every index change. Tracked policy files were an exception: adding or replacing .gitattributes or .gitignore made the generic semantic transfer reject the whole proof. Later read-only status commands then had to rescan the worktree and could not restore the paired untracked proof. Let checkout retain history across regular policy-file changes that it writes itself. Attribute changes refresh the worktree manifest before the provider boundary is rebound, and fail closed if that refresh cannot authenticate the new sources. Keep the existing untracked-cache invalidation for ignore changes, and transfer that cache only while the full tracked proof remains current. Exercise configured fast-forward pulls in main and linked worktrees. A required-filter control also verifies that changed attributes invalidate the affected tracked entry instead of certifying it.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 27, 2026 04:33
f5f8eaf to
0a572e5
Compare
A clean status proof can survive a pull only when its configuration, tracked-file state, FSMonitor token, and paired untracked cache still describe the resulting worktree. Command-scoped push transport settings were included in the configuration fingerprint. Checkout could also discard the untracked proof for policy-file changes or leave events from its own worktree writes outside the proof. The next diff, write-tree, or status then repeated tracked and untracked work. With optional locks disabled, status could not publish the repair, so each invocation paid the same cost. Treat push.negotiate and remote.*.pushurl like other command-scoped transport settings. Preserve the paired untracked cache across checkout, invalidate only affected policy scopes, and authenticate distinct attribute-source directories before transferring semantic history. For checkout, reset, merge, and sequencer worktree updates, write a provisional index under the existing lock, consume the daemon events caused by the update, and certify the result against that locked index before the final write. This also covers stash cleanup through its hard reset. Alternate indexes, split or sparse indexes, unsafe filter or manifest state, and incomplete stat data still fall back. Cover configured pulls, root and nested policy changes, main and linked worktrees, rebase, reset, stash, checkout, and repeated read-only status.
The Linux listener queued every inotify event as a file pathname. Directory events therefore lacked the trailing slash used by semantic invalidation. After an owned worktree update retained a clean proof, a following read-only status could not close those events against it. The command scanned all tracked entries. With core.preloadIndexBulk enabled, this work appears as statx calls instead of lstat counters. Format Linux worktree events through fsmonitor_format_worktree_paths() and use IN_ISDIR to preserve their directory identity. Advertise directory metadata support and mark Linux tokens so clients replace daemons using the old event format. Concurrent clients can see an expected connection reset while one client replaces a stale daemon. Silence that diagnostic only for gentle IPC reads, which already reconnect, without changing ordinary IPC error handling. Cover both bulk preload modes plus single and concurrent daemon replacement.
Bulk index preload can defer content and conversion checks to the diff that normally follows refresh_index(). repair_fsmonitor_proof() only refreshes the index before deciding whether to persist a clean proof; it does not run that diff. With core.preloadIndexBulk enabled, a pull or rebase that changes .gitattributes or .gitignore can therefore leave tracked entries dirty after the writer reports a successful repair. Repeated read-only status calls cannot persist the missing repairs. Do not request deferred bulk results in the writer-repair path. This keeps the ordinary status and diff bulk path unchanged while forcing the exceptional repair to finish its tracked checks before certifying and writing the proof. Enable bulk preload in the existing fast-forward, policy-file, and sequencer writer tests. They verify targeted refreshes and two subsequent read-only status calls without scans or index writes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
git statusrepairs the canonical index, without extending the path to optional-lock-disabled commands, option-bearing status calls, or linked worktreesController scope
This fast-forwards the existing
tb/codex/fsmonitor-hardlink-inodes-unstabletopic by seven commits after88fab89c5e. This pull request is review-only: approve it for controller admission, but do not merge it intocodex-unstable.Performance
.gitattributesretained the full conservative fallbackgit statusscenarios over 12,000 tracked paths reported zero compactions; wall-clock data was nonstationary and is retained as inconclusiveValidation
At source commit
f5f8eaf9b2:make -j8./t7519-status-fsmonitor.sh(107/107)./t7527-builtin-fsmonitor.sh(178/178)./t7530-status-clean-sidecar.sh(59/59)./t/unit-tests/bin/unit-tests -r clean_status_config(14/14)make -C t test-lintgit diff --checkThe first six commits remain range-diff identical to the validated current-lane replay at
6314bd17c6; the seventh is the configured-pull release-blocker fix described above.