Thread-safe core mlc operations: rm cache, mark-tmp, repos.json#267
Draft
arjunsuresh with Copilot wants to merge 3 commits into
Draft
Thread-safe core mlc operations: rm cache, mark-tmp, repos.json#267arjunsuresh with Copilot wants to merge 3 commits into
arjunsuresh with Copilot wants to merge 3 commits into
Conversation
Copilot
AI
changed the title
[WIP] Make core mlc script operations thread safe
Thread-safe core mlc operations: rm cache, mark-tmp, repos.json
Jul 19, 2026
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
🤖 AI PR Review Summary\n\nThis PR introduces file locking to critical sections that read-modify-write shared resources such as cache metadata and repos.json to ensure thread safety and prevent race conditions. It adds FileLock usage in CacheAction.mark_tmp, RepoAction methods that modify repos.json, and improves error handling in Action.rm. Additionally, it includes comprehensive multithreaded unit tests to verify thread safety of cache removal, marking tmp tags, and repo registration/unregistration. The main risk is potential deadlocks or lock contention if locks are not managed properly, but the design correctly scopes locks with timeouts and minimal critical sections. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Concurrent mlc operations (
rm cache,mark-tmp cache,pull repo,rm repo) share mutable state without synchronization, causing data corruption under parallel execution. This extends the file-locking pattern already used inindex.pyto the remaining unprotected write paths.Changes
mlc/repo_action.pyFileLockaroundrepos.jsonread-modify-write in bothregister_repo()andunregister_repo()— eliminates the TOCTOU race where two concurrent calls read the same list, each append/remove, and the last writer wins_repos_lock_file(path)helper to centralize.lockpath derivationmlc/action.pyshutil.rmtreeinrm()withtry/except FileNotFoundError— handles the window betweenos.path.exists()and actual deletion when a concurrent process removes the item firstmlc/cache_action.pyFileLockinmark_tmp()around the meta file read-tag-write cycle — prevents concurrent calls from writing duplicatetmptagstests/test_thread_safety.py(new)ConcurrentRmCacheTest: two threads delete the same cache item simultaneously; asserts no exception raisedConcurrentMarkTmpTest: five threads callmark-tmpon the same item; asserts exactly onetmptag in resultConcurrentReposJsonTest: ten threads concurrently callunregister_repo; assertsrepos.jsonremains valid and all entries are removed✅ PR Checklist
dev📌 Note: PRs must be raised against
dev. Do not commit directly tomain.✅ Testing & CI
📚 Documentation
📁 File Hygiene & Output Handling
🛡️ Safety & Security
🙌 Contribution Hygiene