Skip to content

Allow filtering the Dags list by failed and success runs in any run state filter#69875

Open
yuseok89 wants to merge 1 commit into
apache:mainfrom
yuseok89:feature/67094-any-run-state-filter-all-states
Open

Allow filtering the Dags list by failed and success runs in any run state filter#69875
yuseok89 wants to merge 1 commit into
apache:mainfrom
yuseok89:feature/67094-any-run-state-filter-all-states

Conversation

@yuseok89

@yuseok89 yuseok89 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Closes #67094

The Dags list has two run-state filters. "Last run state" matches on a Dag's latest run, and "Any run state" matches a Dag that has any run in the chosen state. The "Any run state" filter only offered queued and running, so a Dag with failed runs but a green latest run could not be found from the Dags list at all.
This adds failed and success to it, so both filters now cover all four states.

What changed

  • Backend (_AnyDagRunStateFilter). The filter now supports all four run states. Its query moved from dag_id IN (SELECT DISTINCT dag_id FROM dag_run WHERE state = X) to a correlated EXISTS, which the existing (dag_id, state) index resolves per Dag instead of scanning every run in the state. No new index is needed.
  • UI. The "Any run state" selector now lists the same four states as "Last run state".

Performance

Measured on a large dag_run table (a few million rows) across a few thousand Dags(Postgres), filtering by dag_run_state=success (the heaviest state):

  • Old DISTINCT scan of all matching runs: around a second or more
  • New correlated EXISTS: tens of milliseconds

The EXISTS cost grows only with Dag count (one index seek per Dag, stopping at the first match), not with run volume, so it stays fast well beyond typical Dag counts, and the gap widens further as run history grows.

Demo

Screen.Recording.2026-07-14.at.11.09.50.PM.mov

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Opus 4.8

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Jul 14, 2026
@yuseok89 yuseok89 marked this pull request as ready for review July 14, 2026 16:08
@bbovenzi

Copy link
Copy Markdown
Contributor

I think this UX needs to be refined. I don't think any user would realistically search "any dag that has ever had a success", even "any dag that has ever had a failure" could quickly be useless for very old failures. I think we first need a time range filter for use cases like "dags with a failure in the last week", or "dags with any runs in the last 24 hours"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DAG list: state filters should use DAG run counts by state (like 2.x), not latest-run state only

2 participants