Skip to content

Fix executor_config reference aliasing across tasks and DAGs (#72544) - #72563

Open
abhijeets25012-tech wants to merge 1 commit into
apache:mainfrom
abhijeets25012-tech:fix-executor-config-aliasing
Open

Fix executor_config reference aliasing across tasks and DAGs (#72544)#72563
abhijeets25012-tech wants to merge 1 commit into
apache:mainfrom
abhijeets25012-tech:fix-executor-config-aliasing

Conversation

@abhijeets25012-tech

Copy link
Copy Markdown
Contributor

Description

Fixes #72544

_get_parent_defaults previously used copy.copy (shallow copy) when merging dag.default_args. This caused nested mutable structures, specifically executor_config, to be shared by object reference across all tasks and DAGs inheriting from those shared defaults. Consequently, runtime mutations to task.executor_config (such as setting Kubernetes pod overrides) silently modified sibling tasks and unrelated DAGs.

This PR updates _get_parent_defaults to use copy.deepcopy and explicitly deep-copies executor_config during BaseOperator initialization, ensuring strict reference isolation across task definitions.

Changes Made

  • task-sdk/src/airflow/sdk/bases/operator.py:
    • Replaced copy.copy(dag.default_args) with copy.deepcopy(dag.default_args) in _get_parent_defaults.
    • Updated self.executor_config assignment in BaseOperator.__init__ to use copy.deepcopy(executor_config) to isolate both explicit and inherited configurations.

@abhijeets25012-tech

Copy link
Copy Markdown
Contributor Author

Hi maintainers, this PR is ready for review.

Note: The failing Provider distributions tests / Compat CI jobs appear to be unrelated upstream matrix timeouts/flakes, as the changes in this PR are strictly isolated to task-sdk/src/airflow/sdk/bases/operator.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

executor_config from shared default_args is aliased (not copied) across tasks and DAGs

1 participant