Fix use_job_schedule=False disabling asset-triggered Dag runs#67764
Open
Vamsi-klu wants to merge 1 commit into
Open
Fix use_job_schedule=False disabling asset-triggered Dag runs#67764Vamsi-klu wants to merge 1 commit into
Vamsi-klu wants to merge 1 commit into
Conversation
[scheduler] use_job_schedule is documented to turn off only cron/time-based scheduling, but it gated the entire _create_dagruns_for_dags call, which also creates asset- and partitioned-asset-triggered runs. As a result, setting the flag to False silently stopped all asset-driven scheduling. Move the flag check so it gates only time-based run creation; asset- and partition-triggered runs are now always created, matching the documented behavior. Also update the "Disable the scheduler" maintenance docs, which previously claimed this flag stops all new Dag runs. closes: apache#62929
3bd15a8 to
ff8ce38
Compare
Contributor
Author
|
Gentle ping for SME eyes when you have a moment, @Lee-W @uranusjr 🙏 Small scheduler fix for #62929: @Lee-W — you reviewed the earlier (now closed-for-inactivity) #62931 and called the approach "a valid one", so flagging this here. Includes three regression tests (asset, partition, and a flag-on companion) plus a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / impact
Setting
[scheduler] use_job_schedule=Falsesilently stopped all asset- and partitioned-asset-triggered Dag runs — not just cron/time-based runs. Event- and data-driven pipelines that rely on asset scheduling would stop firing with no error, even though the flag is documented to turn off only cron interval scheduling. After this change the flag disables only time-based (cron) scheduling; asset-, partitioned-asset-, and manually-triggered runs are still created.Why
use_job_schedulegated the entire_create_dagruns_for_dagscall in_do_scheduling, but that method creates three kinds of runs: partitioned-asset (top of the method), time-based, and asset-triggered. Only the time-based path should be governed by the flag. This moves the gate so it wraps only the time-based_create_dag_runs(non_asset_dags, …)call; the orchestrator — and therefore the asset/partition paths — always runs. The "Disable the scheduler" maintenance docs, which previously claimed this flag stops all new Dag runs, are updated to match.How it unblocks
Restores the documented behavior for deployments that turn off cron scheduling while still relying on asset-driven scheduling. Supersedes #62931, which was closed for author inactivity (not rejection) — the approach was never disputed and member Lee-W called it "a valid one".
Tests
Three new regression tests in
test_scheduler_job.py, each verified to fail on the unfixed code: asset-triggered runs created with the flag off, partitioned-asset runs created with the flag off, and a flag-on companion proving the time-based gate still suppresses cron runs. Full scheduler suite and the exact-query-count tests pass; mypy + pre-commit + manual static checks are green.closes: #62929
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines