Merge forward 3006.x into 3007.x#69226
Merged
Merged
Conversation
Run `dpkg --configure -a` before installing prerequisites in `install_previous()` to recover from any interrupted dpkg state that would otherwise cause `apt-get` to return exit code 100. Mock `refresh_db` in `test_del_repo` to prevent `apt-get update` from hanging on fake/unreachable repo URLs (e.g. `http://dist.list`), which was causing >90s timeouts in CI.
Fix flaky multimaster reconnection test
- Update python_version to 3.14 in layout.yml.jinja - Add 3.14 to noxfile.py _PYTHON_VERSIONS - Create Python 3.14 requirement files for CI and pkg - Update pre-commit config with 3.14 hooks - Update development setup and troubleshooting docs
- Add missing Changelog, Tools, and Tools VirusTotal 3.14 pre-commit hooks - Update towncrier to 24.8.0 to support Python 3.14 - Regenerate requirements
- Replaced usage of the deprecated and removed `cgi` module with `salt.ext.tornado.httputil._parse_header` in `salt/netapi/rest_tornado/saltnado.py`. - Resolves ModuleNotFoundError when building documentation with Python 3.14.
- Bump pyupgrade to v3.21.2 to fix TypeError on Python 3.14 during pre-commit. - Change python3 to python in setup-relenv to ensure relenv installs into the correct virtual environment on Windows, fixing missing relenv import.
- Update python-tools-scripts to 0.20.5 - Set language_version to python3.10 for all tools hooks to bypass Python 3.14 AST parsing issues
- Remove language_version python3.10 from tools hooks. - ptscripts 0.20.5 already resolves the Python 3.14 AST issue, and pinning the version causes dependency conflicts during hook execution.
- Added `isinstance(attr_value, str)` check to satisfy mypy, as `getattr` could return `Any` or `None`.
- Changed the DEFAULT_REQS_CONFIG requirements files from tools.txt to tools.in. - This prevents pip from attempting to downgrade packages (like urllib3) installed by uv during the initial python-tools-scripts setup, avoiding the OSError caused by uv's non-standard metadata.
…to uv - Added the required dependencies (boto3, pyyaml, jinja2, MarkupSafe, packaging) directly to the `id: tools` hooks in `.pre-commit-config.yaml` via `additional_dependencies`. This allows `pre-commit` (using `uv`) to natively resolve and install everything required by the tools scripts in one pass. - Added a `PRE_COMMIT` environment variable check to `tools/__init__.py` to bypass `ptscripts` attempting to run `pip install -r tools.in` when executed inside `pre-commit`. - This definitively resolves the `OSError` crashes caused by `pip` attempting to downgrade or overwrite packages (like `urllib3` and `six`) that were already installed by `uv` during the hook's initial environment setup.
- Updated lingering hardcoded `3.11` `python-version` inputs to `3.14` in `release.yml`, `depcheck.yml`, `build-docs.yml`, `release-upload-virustotal.yml`, and `ssh-debug.yml`. - Renamed workflow setup steps from `Set up Python 3.11` to `Set up Python 3.14`.
- Removed `backports` from `_SALT_DEPENDENCIES` for Python 3.13+ to avoid Sphinx treating the mocked missing dependency as a fatal warning during HTML docs build.
- Removed the `upload-release-artifacts` job from `staging.yml.jinja`. - Regenerated workflows to apply the change to `staging.yml`.
- Updated the description of the `salt-version` input in the `release-upload-virustotal` pipeline to match the simplified one from the staging pipeline.
- Updated the description of the `salt-version` input in the `release.yml`, `release-upload-virustotal.yml` pipelines and the `staging.yml.jinja` template. - This ensures consistency and matches the detailed formatting instructions (e.g., using the PEP 440 form without a hyphen for prereleases) currently present on the `3008.x` branch.
- Reverted the description of the `salt-version` input in the `release.yml`, `release-upload-virustotal.yml` pipelines and the `staging.yml.jinja` template to the shorter 'Good/Bad' examples format as requested.
`save_load()` catches `psycopg2.IntegrityError` and silently passes. The catch was added in 2015 (issue saltstack#22171) to tolerate duplicate-jid inserts when `master.py` calls `save_load` more than once per jid. On PostgreSQL >= 9.5 the duplicate-jid case is no longer raised at all: `_get_serv` upgrades the SQL to `INSERT ... ON CONFLICT (jid) DO UPDATE`, so the existing row is updated in place. The bare catch is dead code on the modern path. It is not dead on PG < 9.5 -- pgjsonb supports 9.4 (jsonb is from 9.4, ON CONFLICT is from 9.5) -- but on every version the catch is too broad: any `IntegrityError` is dropped, which silently hides FK, NOT NULL and CHECK constraint violations from a misconfigured schema or bad payload. Operators see no symptom, the job return is just gone. Narrow the catch to `psycopg2.errors.UniqueViolation` (sqlstate 23505), which is the only case that saltstack#22171 covered. Log a warning when it triggers so the legacy fallback is at least visible. All other integrity errors now propagate to the master_job_cache caller, which already wraps them with `log.critical` (see master.py:2415- 2426), surfacing the real bug. Add a behavioural test for both branches: a `UniqueViolation` is swallowed; a `ForeignKeyViolation` propagates. Also add an explicit `import psycopg2.errors` to avoid relying on a transitive import. Refs: saltstack#69046
`pgjsonb` defined a module-level `PG_SAVE_LOAD_SQL` string with the
plain-INSERT form, then `_get_serv` rewrote that global to the
ON CONFLICT (UPSERT) form on every connect to PostgreSQL >= 9.5:
if conn.server_version is not None and conn.server_version >= 90500:
global PG_SAVE_LOAD_SQL
PG_SAVE_LOAD_SQL = """INSERT INTO jids ... ON CONFLICT ..."""
`save_load` then read the global. This is hidden coupling -- a
function reads state another function set up -- and the mutation is
one-way: once any connection to a 9.5+ server runs, the global is
permanently UPSERT for the lifetime of the master process. A master
that subsequently talks to a pre-9.5 server (failover, mixed-version
fleet, dev/staging environment) sends UPSERT syntax to a server
that does not understand it and gets a SyntaxError.
Decide the SQL form per-call inside `save_load` from
`cur.connection.server_version`. Drop the module-level global and
the mutation block in `_get_serv`. The UniqueViolation handler from
the previous PR remains: on PG < 9.5 the plain INSERT path can still
hit a duplicate-jid violation (saltstack#22171) and that is tolerated.
Add two behavioural tests asserting the right SQL form is issued
for both server versions, and update the existing save_load tests
to set `cur.connection.server_version` since `save_load` now reads
it directly.
Depends on saltstack#69047 (which narrowed the IntegrityError catch in the
same function); this PR builds on that branch and should be merged
after it.
Refs: saltstack#69052
added 2 commits
May 25, 2026 22:54
CI's Pre-Commit / Run mypy against tools (on Python 3.14) reports [call-overload] for the **dict rmtree call, while my local mypy reported [arg-type] for the same line. Different mypy/typeshed versions express the same incompatibility differently; suppress both error codes.
Test collection failed across all four unit-test groups with "AttributeError: module 'salt.ext' has no attribute 'tornado'" because two 3006.x saltstack#68637 additions auto-merged in: - tests/pytests/unit/test_minion.py: _noop_destroy_async_for_test stub + four tests mocking Minion.destroy_async. 3007.x's _connect_minion and MinionManager.stop_async call minion.destroy() (sync), not destroy_async, so the stub is pointless and the salt.ext.tornado decorator crashes collection. Rewrote the four tests to mock destroy instead. - tests/pytests/unit/transport/test_zeromq.py: four AsyncReqMessageClient shutdown tests that assert internals only present on 3006.x's deferred-shutdown rewrite (_closed, context is None) and import salt.ext.tornado. 3007.x's AsyncReqMessageClient is deprecated and uses different teardown; same obsolescence reasoning as the helper functions dropped from tests/pytests/functional/transport/zeromq/ test_request_client.py earlier in this merge.
added 2 commits
May 26, 2026 00:08
- test_handle_clear_missing_cmd_returns_empty_reply: salt.master.MWorker._handle_clear is async on 3007.x but the 3006.x-authored test called it synchronously and compared the returned coroutine to a tuple. Make the test async and await. - test_pub_async_default_timeout_value: tornado.gen.maybe_future() and the connect_pub Future are built before any event loop is set up; on Py 3.10+ get_event_loop() does not implicitly create one in MainThread, so the Future construction fails with 'RuntimeError: There is no current event loop'. Move the new_event_loop/set_event_loop setup before the with client.LocalClient(...) block so a loop exists when Futures are built.
tests/pytests/functional/transport/zeromq/test_reqchannel_memory.py was added by 3006.x as a regression test for saltstack#68637, which patched AsyncReqMessageClient's deferred-shutdown path to drop FDs synchronously on close. The test asserts VmRSS grows < 512 kB across 400 ReqChannel churn iterations. On 3007.x's rewritten RequestClient (no AsyncReqMessageClient delegation; synchronous, complete-on-return close) the underlying bug is structurally absent (separately audited under this merge), but the 512 kB threshold isn't achievable because Python's pymalloc arenas warm up by ~10 kB/iter in this workload (~4 MB net over 400 iters) before plateauing. The CI failure reports 1.2-4.0 MB growth, matching that pymalloc curve. Same obsolescence rationale as the other saltstack#68637 tests dropped earlier in this merge.
added 3 commits
May 26, 2026 01:49
test_runas_cd_ampersand_dir and test_runas_unpriv_cd_ampersand_dir (the 3006.x saltstack#68448 regression tests for cmd.exe /c compound-command quoting) assumed runas/runas_unpriv returned stdout as str. My merge of salt/utils/win_runas.py kept HEAD's binary 'rb' pipe-drain contract (other tests in the same file already .decode() the bytes), so the marker compares should also use bytes.
tests/pytests/integration/client/test_reqchannel_memory_master.py was added by 3006.x saltstack#68637 as the live-master companion to test_reqchannel_memory.py (which I removed earlier for the same reason). It asserts VmRSS grows < 17408 kB across 80 LocalClient pub iterations against a live master. On 3007.x's RequestClient the 3006.x deferred-shutdown bug is structurally absent, but the threshold was calibrated for that fixed stack. Pymalloc warm-up across 80 publish round-trips against a live master measures ~18-19 MB, above the 17 MB limit, and consistently so.
tests/pytests/scenarios/performance/test_performance.py installs Salt's
pkg requirements into a Docker container before running the perf test.
The auto-merge updated the constraint path (line 232) to the new
linux.lock name but missed the -r requirements file on line 221, so
the in-container pip install ran:
pip install -r /salt/requirements/static/pkg/py3.10/linux.txt
That file no longer exists (renamed to linux.lock as part of the
3006.x .in/.lock requirements restructure brought forward in this
merge). Update line 221 to match line 232.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
What issues does this PR fix or reference?
Fixes
Previous Behavior
Remove this section if not relevant
New Behavior
Remove this section if not relevant
Merge requirements satisfied?
[NOTICE] Bug fixes or features added to Salt require tests.
Commits signed with GPG?
Yes/No