Skip to content

Fix use_job_schedule=False disabling asset-triggered Dag runs#67764

Open
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:fix/62929-use-job-schedule-asset-runs
Open

Fix use_job_schedule=False disabling asset-triggered Dag runs#67764
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:fix/62929-use-job-schedule-asset-runs

Conversation

@Vamsi-klu
Copy link
Copy Markdown
Contributor

What / impact

Setting [scheduler] use_job_schedule=False silently stopped all asset- and partitioned-asset-triggered Dag runs — not just cron/time-based runs. Event- and data-driven pipelines that rely on asset scheduling would stop firing with no error, even though the flag is documented to turn off only cron interval scheduling. After this change the flag disables only time-based (cron) scheduling; asset-, partitioned-asset-, and manually-triggered runs are still created.

Why

use_job_schedule gated the entire _create_dagruns_for_dags call in _do_scheduling, but that method creates three kinds of runs: partitioned-asset (top of the method), time-based, and asset-triggered. Only the time-based path should be governed by the flag. This moves the gate so it wraps only the time-based _create_dag_runs(non_asset_dags, …) call; the orchestrator — and therefore the asset/partition paths — always runs. The "Disable the scheduler" maintenance docs, which previously claimed this flag stops all new Dag runs, are updated to match.

How it unblocks

Restores the documented behavior for deployments that turn off cron scheduling while still relying on asset-driven scheduling. Supersedes #62931, which was closed for author inactivity (not rejection) — the approach was never disputed and member Lee-W called it "a valid one".

Tests

Three new regression tests in test_scheduler_job.py, each verified to fail on the unfixed code: asset-triggered runs created with the flag off, partitioned-asset runs created with the flag off, and a flag-on companion proving the time-based gate still suppresses cron runs. Full scheduler suite and the exact-query-count tests pass; mypy + pre-commit + manual static checks are green.

closes: #62929


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

@boring-cyborg boring-cyborg Bot added area:Scheduler including HA (high availability) scheduler kind:documentation labels May 30, 2026
[scheduler] use_job_schedule is documented to turn off only cron/time-based
scheduling, but it gated the entire _create_dagruns_for_dags call, which also
creates asset- and partitioned-asset-triggered runs. As a result, setting the
flag to False silently stopped all asset-driven scheduling.

Move the flag check so it gates only time-based run creation; asset- and
partition-triggered runs are now always created, matching the documented
behavior. Also update the "Disable the scheduler" maintenance docs, which
previously claimed this flag stops all new Dag runs.

closes: apache#62929
@Vamsi-klu Vamsi-klu force-pushed the fix/62929-use-job-schedule-asset-runs branch from 3bd15a8 to ff8ce38 Compare May 30, 2026 04:20
@Vamsi-klu Vamsi-klu marked this pull request as ready for review May 30, 2026 04:20
@Vamsi-klu Vamsi-klu requested review from XD-DENG and ashb as code owners May 30, 2026 04:20
@Vamsi-klu
Copy link
Copy Markdown
Contributor Author

Vamsi-klu commented May 30, 2026

Gentle ping for SME eyes when you have a moment, @Lee-W @uranusjr 🙏

Small scheduler fix for #62929: [scheduler] use_job_schedule=False was gating the entire _create_dagruns_for_dags, so it silently disabled asset- and partitioned-asset-triggered runs too — not just cron scheduling. The gate now wraps only the time-based _create_dag_runs(non_asset_dags, …) path, matching the documented behavior.

@Lee-W — you reviewed the earlier (now closed-for-inactivity) #62931 and called the approach "a valid one", so flagging this here. Includes three regression tests (asset, partition, and a flag-on companion) plus a best-practices.rst "Disable the scheduler" doc correction. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

use_job_schedule affects asset based scheduling

1 participant