Add dag_bundle_name option to subprocess coordinators - #70805
Open
jason810496 wants to merge 4 commits into
Open
Add dag_bundle_name option to subprocess coordinators#70805jason810496 wants to merge 4 commits into
jason810496 wants to merge 4 commits into
Conversation
jason810496
requested review from
amoghrajesh,
ashb,
ephraimbuddy,
jedcunningham,
kaxil and
uranusjr
as code owners
July 31, 2026 02:27
boring-cyborg
Bot
added
area:coordinator
kaxil
reviewed
Jul 31, 2026
jason810496
marked this pull request as draft
July 31, 2026 15:24
jason810496
force-pushed
the
feature/coordinator/dag-bundle-name
branch
from
August 1, 2026 15:29
1b76bfe to
182919a
Compare
jason810496
marked this pull request as ready for review
August 3, 2026 08:54
1 task
jason810496
force-pushed
the
feature/coordinator/dag-bundle-name
branch
2 times, most recently
from
August 5, 2026 06:34
bc97b03 to
12de628
Compare
1 task
Language-SDK coordinators previously required a hand-managed filesystem root (jars_root/executables_root/bundles_root) for compiled artifacts, kept outside Airflow's Dag-bundle machinery and prone to drifting from the Dags they serve. Allowing artifacts to be located through a configured Dag bundle -- or the task's own bundle when none is named -- lets them be delivered and versioned the same way as everything else.
Language-SDK subprocess coordinators materialized artifact bundles without the version-lock protection the Python task path holds, so bundle cleanup could delete a version a running JVM was still reading from. Resolution now holds the version lock for the subprocess lifetime, mirroring the Python task path, and materialization failures surface in the task log rather than the worker log. Classification moves to the base coordinator so a subclass that adds no explicit-root kwarg falls back to the task-bundle default instead of failing every task, the Java coordinator now requires an explicit entrypoint when it has to scan a whole bundle, and configured-bundle lookup checks names only so an unrelated unimportable bundle no longer breaks coordinator construction.
When a subprocess coordinator locates artifacts through a Dag bundle rather than an explicit root, JavaCoordinator previously rejected a missing main_class outright. That made the co-located default unusable without extra configuration and left the entrypoint no more determinate than an auto-detected scan would be. Auto-detect over the resolved bundle now handles that case, with the remaining cross-bundle ambiguity deferred to an IMPORT_ERROR-stage check under AIP-85. A named Dag bundle resolves to its latest version, so its version was unset and the version lock held around the subprocess silently did nothing, letting bundle cleanup delete artifacts a running task was still reading. The concrete current version is now pinned before locking.
jason810496
force-pushed
the
feature/coordinator/dag-bundle-name
branch
from
August 6, 2026 03:19
12de628 to
c7ab92a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Language-SDK subprocess coordinators (Java, Executable, Node) required a hand-managed filesystem root (
jars_root/executables_root/bundles_root) for compiled artifacts. Users should be able to deploy those artifacts by existing DagBundle mechanism as well.How
dag_bundle_namekwarg on theSubprocessCoordinatorbase, giving three ways to locate artifacts:dag_bundle_nameset: scan that configured bundle's path at its latest version._build_execute_task_commandsignature unchanged:execute_taskstashes the task'sbundle_infofor the duration of the call, and_scan_rootsresolves the scan root on demand.task_runner.initialize_ti_bundle(extracted fromparse) for resolution.dag_bundle_nameare not both set, and that a named bundle is configured via a new lightweightDagBundlesManager.is_bundle_configuredthat checks configuration without constructing the bundle instance.Notes
dag_bundle_name) is pinned to the run'sbundle_info.versionbecause it is the same artifact the run was created from, while a separately named artifact bundle tracks its own latest release independently of any run.Was generative AI tooling used to co-author this PR?