Skip to content

Resolve master_uri in minion engines to fix pillar.data KeyError (#57952)#69781

Open
ggiesen wants to merge 1 commit into
saltstack:3006.xfrom
ggiesen:fix-57952-engine-master-uri
Open

Resolve master_uri in minion engines to fix pillar.data KeyError (#57952)#69781
ggiesen wants to merge 1 commit into
saltstack:3006.xfrom
ggiesen:fix-57952-engine-master-uri

Conversation

@ggiesen

@ggiesen ggiesen commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes KeyError: 'master_uri' when a minion engine calls a salt function that needs the master transport (for example pillar.data), by resolving master_uri in the engine's own process.

What issues does this PR fix or reference?

Fixes #57952

Previous Behavior

A minion forks its engines early in startup -- Minion.__init__ schedules start_engines via io_loop.spawn_callback (salt/minion.py:1538-1541), and add_process forks the engine subprocess right away -- before the minion connects and resolves master_uri into its opts (opts.update(resolve_dns(opts)) at salt/minion.py:875, reached from connect_master). The engine subprocess therefore holds a snapshot of opts that lacks master_uri, and Engine.run() never refreshes it. Any __salt__ call that builds the master transport (pillar.data -> Pillar -> ReqChannel.factory -> AsyncAuth) reads opts["master_uri"] with no default in salt.crypt and raises KeyError: 'master_uri'.

Scheduled jobs (e.g. mine.update) fork after connect, so their opts already have master_uri -- which is why the reporter saw mine.update succeed concurrently while the engine's pillar.data failed.

New Behavior

Engine.run() resolves master_uri in the engine's own process before the execution modules are loaded/used. It is best-effort so it can never block or break engine startup:

  • master engines are skipped (__role != "minion"; master.py starts engines as __role == "master" and needs no master_uri);
  • masterless minions (file_client: local) are skipped (no transport needed);
  • an engine that already has master_uri is left untouched;
  • DNS is resolved once without the minion connect loop's retry (retry_dns=0) and any failure is caught and logged, so a transport-less engine, or one whose master is momentarily unresolvable at boot, still starts -- failing no worse than today (this covers the failover/list-master case, where a single host cannot be resolved yet).

This works on both fork and spawning platforms: on fork the engine's self.opts is the same object the loader closed over; on spawning platforms the execution modules are rebuilt against self.opts after run() begins -- so completing the opts at the top of run() reaches __salt__ either way.

Verified with a real master+minion: an engine calling pillar.data went from KeyError: 'master_uri' to returning the pillar dict.

On the deeper fix

This is a contained fix: the engine completes its own opts. The underlying cause is that a regular minion forks its engines before it connects, so the engine snapshots incomplete opts in the first place. The more architectural fix would be to defer regular-minion engine start until after the first connect (from _post_master_init / tune_in), which is exactly how proxy minions already start their engines (salt/metaproxy/proxy.py:161-166); the engine would then inherit a fully-resolved opts and need no in-engine resolution. I kept it contained here because deferring engine start touches minion startup ordering and would have to preserve behaviour for masterless minions (which never connect) and failover re-spawns. Happy to take that approach instead if you'd prefer the root-cause fix.

Merge requirements satisfied?

  • Docs - n/a
  • Changelog - changelog/57952.fixed.md
  • Tests written/updated - tests/pytests/unit/engines/test_engines.py (resolves for a minion; skips master/already-present/masterless; non-fatal on resolution failure)

Commits signed with GPG?

No

A minion forks its engines early in startup, before it has connected and
resolved master_uri into its opts, so the engine runs against a snapshot that
lacks it. Any __salt__ call needing the master transport (e.g. pillar.data)
then raises KeyError: 'master_uri' in salt.crypt.

Engine.run() now completes the engine's opts by resolving master_uri in the
engine's own process before the execution modules are loaded. It is
best-effort and cannot block or break engine startup: master engines are
skipped by role, masterless minions need no transport, and DNS is resolved
once without the minion connect loop's retry so a transport-less engine still
starts when the master is unresolvable.

Fixes saltstack#57952
@ggiesen ggiesen requested a review from a team as a code owner July 11, 2026 18:18
@dwoz dwoz added the test:full Run the full test suite label Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants