Allow filtering the Dags list by failed and success runs in any run state filter#69875
Open
yuseok89 wants to merge 1 commit into
Open
Allow filtering the Dags list by failed and success runs in any run state filter#69875yuseok89 wants to merge 1 commit into
yuseok89 wants to merge 1 commit into
Conversation
Contributor
|
I think this UX needs to be refined. I don't think any user would realistically search "any dag that has ever had a success", even "any dag that has ever had a failure" could quickly be useless for very old failures. I think we first need a time range filter for use cases like "dags with a failure in the last week", or "dags with any runs in the last 24 hours" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #67094
The Dags list has two run-state filters. "Last run state" matches on a Dag's latest run, and "Any run state" matches a Dag that has any run in the chosen state. The "Any run state" filter only offered
queuedandrunning, so a Dag with failed runs but a green latest run could not be found from the Dags list at all.This adds
failedandsuccessto it, so both filters now cover all four states.What changed
_AnyDagRunStateFilter). The filter now supports all four run states. Its query moved fromdag_id IN (SELECT DISTINCT dag_id FROM dag_run WHERE state = X)to a correlatedEXISTS, which the existing(dag_id, state)index resolves per Dag instead of scanning every run in the state. No new index is needed.Performance
Measured on a large
dag_runtable (a few million rows) across a few thousand Dags(Postgres), filtering bydag_run_state=success(the heaviest state):DISTINCTscan of all matching runs: around a second or moreEXISTS: tens of millisecondsThe
EXISTScost grows only with Dag count (one index seek per Dag, stopping at the first match), not with run volume, so it stays fast well beyond typical Dag counts, and the gap widens further as run history grows.Demo
Screen.Recording.2026-07-14.at.11.09.50.PM.mov
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.