Skip to content

fix: using confirmed status to compute probation state - #1843

Merged
cka-y merged 1 commit into
mainfrom
fix/confirmed-pass
Sep 15, 2026
Merged

cka-y merged 1 commit into
mainfrom
fix/confirmed-pass

Conversation

@cka-y

@cka-y cka-y commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary:

on_probation and in_grace_period were derived from observed_status. That column is a property of the last run, not of the criterion: a run that reaches no verdict (unknown) leaves the debounced state untouched by design, so any flag built on observed_status moves on a night nobody could measure.

Three concrete bugs came out of that:

  • unknown on top of a confirmed failure reported on_probation: true. Probation is the clean run served after recovering, and a criterion still sitting on a confirmed failure never recovered. observed_status != 'fail' was true for unknown, so the criterion read as recovering.
  • A withdrawn (not_applicable) criterion rolled up into feed-level probation. The nightly job deliberately freezes probation_start rather than clearing it, in case the criterion applies again. observed_status <> 'fail' matched it, so a feed whose only probation sat on a criterion that no longer applies reported on_probation: true with a bogus probation_ends_at — on the feed-detail/report endpoints and in feedsearch.
  • in_grace_period flickered off on a blind night. It required observed_status = 'fail', so an unknown run mid-grace dropped the at-risk flag and nulled grace_period_ends_at, even though the failure streak and its countdown were still running.

Both flags now read only the debounced columns — confirmed_status, first_observed_failure_at, probation_start:

def is_serving_probation(row: SealCriterionOrm) -> bool:
    if row.probation_start is None or probation_period_for(row.criterion) is None:
        return False
    return row.confirmed_status == CriterionStatus.PASS

liquibase/materialized_views/feed_search.sql carries the same predicate for search, so it moves with it: observed_status <> 'fail' becomes confirmed_status = 'pass'.

Related: #1842.

Expected behavior:

Response shapes are unchanged. has_seal and seal_status are unchanged too — roll_up_seal_status only consults on_probation in its confirmed is PASS and on_probation term, and already excludes not_applicable criteria, so neither bug could reach the seal decision. What changes is what gets reported:

Situation Surface Before After
unknown over a confirmed failure criterion on_probation true false
not_applicable with a frozen probation_start feed on_probation / probation_ends_at true / set false / null
not_applicable with a frozen probation_start search reliability_seal.on_probation true false
unknown during a grace period criterion in_grace_period / grace_period_ends_at false / null true / set

The feed-level on_probation was already correct for the first row (roll_up_on_probation cancels on any confirmed failure) — that guard is now redundant rather than load-bearing.

Deploy note: the rebuild-feed-search-mv changeset is runOnChange="true" and drops/recreates feedsearch, so the view is rebuilt on the next Liquibase run. No new changeset file.

Testing tips:

Unit tests, no DB needed:

./scripts/api-tests.sh --test_file tests/unittest/models/test_reliability_criterion_impl.py
./scripts/api-tests.sh --test_file tests/unittest/models/test_feed_reliability_summary_impl.py

New cases:

  • test_an_unevaluable_run_over_a_confirmed_failure_is_not_probation
  • test_an_unevaluable_run_does_not_clear_the_grace_period
  • test_probation_ignored_for_a_withdrawn_criterion (criterion + feed level)
  • test_search_ignores_probation_for_a_withdrawn_criterion — the one that exercises the changed SQL; it needs the test DB and a refreshed feedsearch, and fails against the old predicate.

Two existing tests were rewritten rather than just re-asserted: they built rows the nightly job cannot produce (observed=fail + confirmed=pass + probation_start set — within_grace requires not was_on_probation, so that combination is unreachable). They now use reachable rows.

To check by hand, pick a feed with a criterion on probation, set its latest seal_criterion row to observed_status='unknown' with confirmed_status='fail', and confirm GET /v1/gtfs_feeds/{id} reports that criterion as fail with on_probation: false.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

Generated with Claude Code

@cka-y
cka-y marked this pull request as ready for review September 15, 2026 19:32
@cka-y
cka-y merged commit 3666dbc into main Sep 15, 2026
23 of 24 checks passed
@cka-y
cka-y deleted the fix/confirmed-pass branch September 15, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants