Refine L3 promotion criteria for downstream repositories - #102
Open
fffrog wants to merge 5 commits into
Open
Conversation
Contributor
Author
fffrog
commented
Jul 22, 2026
fffrog
commented
Jul 22, 2026
albanD
reviewed
Jul 28, 2026
3 tasks
subinz1
added a commit
to subinz1/test-infra
that referenced
this pull request
Aug 3, 2026
Replace the "Flaky Jobs" stat card with a "Timeout Rate" percentage card on the CRCR downstream repo page, aligning with the L3 promotion criteria proposed in pytorch/rfcs#102 which defines timeout rate < 10% as an infrastructure metric. The timeout rate is computed as timed_out / total_jobs directly in the ClickHouse query, replacing the complex flaky-job subquery. The card shows the percentage with color coding: green (0%), orange (>0%), red (≥10% — exceeds L3 threshold).
subinz1
added a commit
to subinz1/test-infra
that referenced
this pull request
Aug 3, 2026
Update the Pass Rate card color coding to match the L3 promotion criteria from pytorch/rfcs#102 (job pass rate > 90%): - Green: 100% (perfect) - Orange: 90–99.9% (meets L3 threshold) - Red: < 90% (below L3 threshold) Previously the thresholds were ≥95% green, ≥80% orange, <80% red.
3 tasks
atalman
requested changes
Aug 3, 2026
atalman
reviewed
Aug 3, 2026
subinz1
added a commit
to pytorch/test-infra
that referenced
this pull request
Aug 4, 2026
Replace the "Flaky Jobs" stat card with a "Timeout Rate" percentage card on the CRCR downstream repo page, aligning with the L3 promotion criteria proposed in pytorch/rfcs#102 which defines timeout rate < 10% as an infrastructure metric. The timeout rate is computed as timed_out / total_jobs directly in the ClickHouse query, replacing the complex flaky-job subquery. The card shows the percentage with color coding: green (0%), orange (>0%), red (≥10% — exceeds L3 threshold).
subinz1
added a commit
to pytorch/test-infra
that referenced
this pull request
Aug 4, 2026
Update the Pass Rate card color coding to match the L3 promotion criteria from pytorch/rfcs#102 (job pass rate > 90%): - Green: 100% (perfect) - Orange: 90–99.9% (meets L3 threshold) - Red: < 90% (below L3 threshold) Previously the thresholds were ≥95% green, ≥80% orange, <80% red.
atalman
pushed a commit
to pytorch/test-infra
that referenced
this pull request
Aug 4, 2026
…colors with L3 criteria (#8421) ## Summary Two changes to the CRCR per-repo dashboard page (`/crcr/{org}/{repo}`), aligning stat card thresholds with the L3 promotion criteria proposed in [pytorch/rfcs#102](pytorch/rfcs#102). ### 1. Replace Flaky Jobs card → Timeout Rate **ClickHouse query** (`crcr_backend_summary/query.sql`): - Removed the complex flaky-job subquery (correlated subquery scanning the table twice) - Added `timeout_rate = timed_out / total_jobs` **Frontend card**: - Shows percentage (e.g., `0.0%`, `1.2%`) - Sub-text: `X timed out / Y jobs` - Color: green (0%), orange (>0% but <10%), red (≥10% — exceeds L3 threshold) ### 2. Align Pass Rate color thresholds Updated to match L3 criteria (`job pass rate > 90%`): - **Green**: 100% - **Orange**: 90–99.9% (meets L3) - **Red**: < 90% (below L3) Previously: ≥95% green, ≥80% orange, <80% red. ## Context The L3 criteria in [pytorch/rfcs#102](pytorch/rfcs#102) defines: - **Timeout rate < 10%** as an infrastructure reliability signal - **Job pass rate > 90%** as a test quality signal These cards make L3 readiness directly visible on the per-repo dashboard. ## Test plan - [ ] Verify `crcr_backend_summary` query returns `timeout_rate` correctly - [ ] Verify Timeout Rate card renders with proper color coding - [ ] Verify Pass Rate card shows red when < 90%, orange when 90–99.9%, green at 100%
can-gaa-hou
reviewed
Aug 5, 2026
Contributor
Author
|
Hi @albanD @atalman @can-gaa-hou , the new commit is ready, please help to take a look at it when you have a time. thank you in advance. |
albanD
reviewed
Aug 11, 2026
albanD
left a comment
albanD
left a comment
Contributor
There was a problem hiding this comment.
Looks pretty good. Mostly small comments to tune numbers and be more precise.
This was referenced Aug 17, 2026
atalman
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR clarifies the L3 promotion criteria for downstream repositories by: