chore(pyamber): silence pkg_resources deprecation warning from fs import#6880
chore(pyamber): silence pkg_resources deprecation warning from fs import#6880aglinxinyuan wants to merge 1 commit into
Conversation
Every spawned Python UDF worker imports fs (PyFilesystem2), whose import of pkg_resources emits a UserWarning on stderr. Filter that one message before the core import chain, and mirror the filter for pytest output. Follow-up item from apache#6796.
Automated Reviewer SuggestionsBased on the
|
There was a problem hiding this comment.
Pull request overview
This PR reduces noisy, non-actionable Python warnings emitted during Amber Python UDF worker startup by adding a targeted warnings filter for the pkg_resources is deprecated as an API... UserWarning that is triggered transitively via fs imports. It improves CI log signal (especially in integration runs that spawn many workers) without suppressing unrelated warnings.
Changes:
- Add a narrowly scoped
warnings.filterwarnings(...)rule in the Python worker entrypoint before the core import chain runs. - Mirror the same warning filter in
pytestconfiguration to keep the test-suite warnings summary quiet and consistent.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
amber/src/main/python/texera_run_python_worker.py |
Installs a targeted UserWarning filter for the pkg_resources deprecation warning before importing core.python_worker (which transitively imports fs). |
amber/pyproject.toml |
Adds a matching pytest filterwarnings entry so test runs don’t surface the same known, non-actionable warning. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6880 +/- ##
=========================================
Coverage 77.31% 77.31%
Complexity 3524 3524
=========================================
Files 1161 1161
Lines 45920 45922 +2
Branches 5099 5099
=========================================
+ Hits 35501 35503 +2
Misses 8839 8839
Partials 1580 1580
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
✅ No material benchmark regressions detected🟢 2 better · 🔴 0 worse · ⚪ 13 noise (<±5%) · 0 without baseline
Baseline detailsLatest main
Raw CSVconfig_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,542.81,200,128000,368,0.225,25799.00,37969.25,37969.25
1,100,10,64,20,2583.89,2000,1280000,774,0.472,128638.30,159174.20,159174.20
2,1000,10,64,20,22123.82,20000,12800000,904,0.552,1102065.07,1155211.62,1155211.62 |
| # at import time (#4199), which emits this deprecation warning on stderr in | ||
| # every spawned worker. It is not actionable on our side, so filter it before | ||
| # the import chain runs. Mirrored for pytest in amber/pyproject.toml. | ||
| warnings.filterwarnings( |
There was a problem hiding this comment.
Narrow, well-scoped filter — no objection to silencing this. One thing worth surfacing: this warning was also the only recurring signal that we're pinned on an EOL fs (2.4.16 is the last release) and holding setuptools below the pkg_resources removal (#4199). Once it's quiet, that debt only lives in this comment. There's no open tracking issue for migrating off PyFilesystem2 / handling the eventual pkg_resources removal — worth opening one and linking it here so the debt stays visible after the reminder is gone. Not blocking.
What changes were proposed in this PR?
Every spawned Python UDF worker prints this on stderr at import time, dozens of times per amber-integration CI run:
Root cause: the worker entry point imports
core.python_worker, which transitively importsfs(PyFilesystem2, used byExecutorManagerfor the tmp filesystem that holds UDF source code);fs/__init__.pyimportspkg_resources, which emits aUserWarning. It is a Pythonwarningswarning, not a loguru log, so the existing log-level knobs (#6797) cannot suppress it.fsstill needspkg_resources— that is why setuptools is range-pinned (#6412) — so the warning is expected and not actionable on our side.amber/src/main/python/texera_run_python_worker.pywarnings.filterwarningsscoped to this one message +category=UserWarning, installed above the core import chainamber/pyproject.tomlfilterwarningsso the suite's warnings summary is quiet tooOnly this exact message is filtered — unrelated warnings still print (verified with an injected control warning). The companion
declare_namespaceDeprecationWarnings fromfsremain, but they only surface in pytest summaries; Python's default filters already hide them in spawned worker processes.Any related issues, documentation, discussions?
Listed as a follow-up item in #6796 (the log-verbosity reduction itself landed via #6797).
How was this PR tested?
texera_run_python_workerprinted theUserWarningbefore the change and is silent after; an injected unrelatedUserWarningstill prints, confirming the filter is not a blanket ignore.pytest src/test/python -m "not integration"fromamber/: 749 passed and thepkg_resourcesUserWarningis gone from the warnings summary (the handful of Iceberg-catalog failures on this Windows machine are identical on a clean tree).ruff checkandruff format --checkpass on the edited file.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Fable 5)