Skip to content

Migrate off unmaintained PyFilesystem2 (fs) and unpin setuptools #6917

Description

@aglinxinyuan

Task Summary

fs (PyFilesystem2) is effectively EOL: 2.4.16 (May 2022) is the last release ever published, upstream master still imports pkg_resources at import time (PyFilesystem/pyfilesystem2#577 and #597 have been open for years), and there is no maintained successor package. Since setuptools 82 removed pkg_resources entirely, import fs hard-crashes there — which is the only reason amber pins setuptools==80.10.2 (#4199, #6412). PR #6880 silences the resulting deprecation warning in CI, so this issue keeps the underlying debt visible (as suggested in this review comment).

Today:  ExecutorManager -> fs.open_fs("temp://") -> pkg_resources -> setuptools held < 82
After:  ExecutorManager -> tempfile.mkdtemp + shutil.rmtree -> fs gone -> setuptools unpinned

ExecutorManager is the only consumer, and fs's TempFS is internally exactly tempfile.mkdtemp + shutil.rmtree, so the swap is behavior-preserving. Full footprint:

File Change
amber/src/main/python/core/architecture/managers/executor_manager.py replace fs.open_fs("temp://") with tempfile.mkdtemp() + shutil.rmtree, preserving lazy creation, sys.path handling, the never-materialized close guard, and the tolerated leak-on-force-kill semantics
amber/src/test/python/core/architecture/managers/test_executor_manager.py two lines read manager.fs.getsyspath(...); no test mocks fs
amber/requirements.txt drop fs==2.4.16 and the setuptools==80.10.2 pin plus its comment (fs is the only distribution in the closure requiring setuptools; pytz's pkg_resources use is a guarded fallback)
amber/LICENSE-binary-python drop fs==2.4.16 and appdirs==1.4.4 (an fs-only transitive); keep six (python-dateutil needs it)

The migration PR should also revert the two warning-filter hunks from #6880 (worker entry point + pytest filterwarnings), which become dead weight once fs is gone.

Implementation notes:

  • Use tempfile.mkdtemp, not tempfile.TemporaryDirectory — its GC/exit finalizer would silently clean up abandoned directories, changing the documented leak-on-force-kill behavior (see the TODO in executor_manager.py).
  • Both license bullets must land in the same PR: stale LICENSE-binary-python entries hard-fail check_binary_deps.py even in PR mode.
  • torch (operator-requirements) still declares setuptools<82 transitively; harmless, and invisible to the license check.

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions