perf(crons): Prefetch some queries in clock tasks consumer - #123067
perf(crons): Prefetch some queries in clock tasks consumer#123067wedamija wants to merge 1 commit into
Conversation
When we're backlogged we are making a lot of queries here attempting to catch up. We can instead preload a bunch of them for the batch and pass them along.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4ce075b. Configure here.
| or monitor_environment.next_checkin_latest > ts | ||
| ): | ||
| return None | ||
| return _mark_environment_missing(monitor_environment, ts) |
There was a problem hiding this comment.
Prefetch snapshot skips live miss guard
Medium Severity
When prefetch is on, mark_environment_missing applies the next_checkin_latest guard to a batch-start snapshot instead of a live row, and mark_checkin_timeout does the same for has_newer_status_affecting_checkin. An ingest check-in that lands after prefetch can advance next_checkin_latest or add a newer OK, but the clock task still writes a miss or incident and can overwrite last_checkin, next_checkin, and status. That window is largest exactly when clock-tasks is backlogged.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 4ce075b. Configure here.
|
|
||
| # If the monitor has had any newer OK/ERROR status check-ins than this | ||
| # timeout, then this timeout cannot affect the status of the monitor. | ||
| if monitor_has_newer_status_affecting_checkins(monitor_environment, checkin.date_added): | ||
| if prefetch is not None: | ||
| has_newer = prefetch.has_newer_status_affecting_checkin( | ||
| monitor_environment.id, checkin.date_added | ||
| ) | ||
| else: | ||
| has_newer = monitor_has_newer_status_affecting_checkins( | ||
| monitor_environment, checkin.date_added | ||
| ) | ||
| if has_newer: |
There was a problem hiding this comment.
Prefetched newer-check-in guard can miss concurrent OK/ERROR
When prefetch is set, has_newer uses a batch-start snapshot instead of the live exists() query, so an OK/ERROR check-in inserted after prefetch still lets this timeout call mark_failed and incorrectly flip the monitor to ERROR.
Evidence
- In this hunk,
prefetch is not Nonereplacesmonitor_has_newer_status_affecting_checkins()withprefetch.has_newer_status_affecting_checkin(). - That helper only reads
ClockTaskPrefetch.newest_status_affecting, populated once inprefetch_clock_tasks()before the thread pool runs. - A concurrent monitor-consumer OK/ERROR for the same environment after prefetch is invisible to the snapshot, so
has_newerstays false. - Execution continues into
mark_failed(), unlike the non-prefetch path which re-queriesdate_added__gt=checkin.date_addedat decision time.
Identified by Warden · sentry-backend-bugs · W8E-ACM


When we're backlogged we are making a lot of queries here attempting to catch up. We can instead preload a bunch of them for the batch and pass them along.