Allow independent stores in MemoryFileSystem - #2137
Open
aprylewu wants to merge 2 commits into
Open
Conversation
Signed-off-by: Mingyang Wu <129849514+aprylewu@users.noreply.github.com>
Signed-off-by: Mingyang Wu <129849514+aprylewu@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Memory filesystem instances always share their files and directories, even when callers bypass the instance cache. Add
global_store=Falseso callers can create independent in-memory workspaces with overlapping paths.Each independent instance owns its files and empty directories, has a distinct filesystem token, and bypasses instance caching. The setting also works through fsspec configuration. Pickling an independent filesystem copies its contents and reconnects its file objects to the restored instance.
Default construction keeps the existing global store, instance reuse, and transaction behavior, including writes through
fsspec.open()andfsspec.get_mapper(). A MemoryFS-specific cache metaclass applies the opt-in behavior without changing the common_Cachedimplementation or other backends.Closes #1904; follows the abandoned #1905. The discussion in #1906 suggested disabling MemoryFS caching altogether, but keeping default caching preserves the existing transaction association of the top-level helpers.
Validation on macOS / Python 3.13.5:
git diff --checkpassed.The full Docker/FUSE/downstream suite and documentation build were not run. JSON serialization still records constructor options; it does not copy memory contents.
AI assistance: OpenAI Codex assisted with the implementation, tests, and this description. The validation results above are from local execution.