Skip to content

Allow independent stores in MemoryFileSystem - #2137

Open
aprylewu wants to merge 2 commits into
fsspec:masterfrom
aprylewu:fix/memory-independent-stores
Open

Allow independent stores in MemoryFileSystem#2137
aprylewu wants to merge 2 commits into
fsspec:masterfrom
aprylewu:fix/memory-independent-stores

Conversation

@aprylewu

Copy link
Copy Markdown

Memory filesystem instances always share their files and directories, even when callers bypass the instance cache. Add global_store=False so callers can create independent in-memory workspaces with overlapping paths.

Each independent instance owns its files and empty directories, has a distinct filesystem token, and bypasses instance caching. The setting also works through fsspec configuration. Pickling an independent filesystem copies its contents and reconnects its file objects to the restored instance.

Default construction keeps the existing global store, instance reuse, and transaction behavior, including writes through fsspec.open() and fsspec.get_mapper(). A MemoryFS-specific cache metaclass applies the opt-in behavior without changing the common _Cached implementation or other backends.

Closes #1904; follows the abandoned #1905. The discussion in #1906 suggested disabling MemoryFS caching altogether, but keeping default caching preserves the existing transaction association of the top-level helpers.

Validation on macOS / Python 3.13.5:

  • New tests against original production code: 5 failed, 6 passed; all pass with this change.
  • Eight related test modules: 561 passed, 104 skipped, 1 xfailed. Skips cover unavailable optional FTP/PyArrow/S3/ADL dependencies and shell globstar support.
  • Independent checks covered default transaction commit/rollback, configuration and environment options, subclasses, cache isolation, pickle/deepcopy, and garbage collection.
  • Ruff 0.14.3, formatting, codespell, and git diff --check passed.

The full Docker/FUSE/downstream suite and documentation build were not run. JSON serialization still records constructor options; it does not copy memory contents.

AI assistance: OpenAI Codex assisted with the implementation, tests, and this description. The validation results above are from local execution.

Signed-off-by: Mingyang Wu <129849514+aprylewu@users.noreply.github.com>
Signed-off-by: Mingyang Wu <129849514+aprylewu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

create multiple separate memory filesystems

1 participant