Skip to content

perf(crons): Prefetch some queries in clock tasks consumer - #123067

Open
wedamija wants to merge 1 commit into
masterfrom
danf/crons-prefetch-queries
Open

perf(crons): Prefetch some queries in clock tasks consumer#123067
wedamija wants to merge 1 commit into
masterfrom
danf/crons-prefetch-queries

Conversation

@wedamija

Copy link
Copy Markdown
Member

When we're backlogged we are making a lot of queries here attempting to catch up. We can instead preload a bunch of them for the batch and pass them along.

When we're backlogged we are making a lot of queries here attempting to catch up. We can instead preload a bunch of them for the batch and pass them along.
@wedamija
wedamija requested review from a team as code owners August 28, 2026 21:00
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 28, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4ce075b. Configure here.

or monitor_environment.next_checkin_latest > ts
):
return None
return _mark_environment_missing(monitor_environment, ts)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefetch snapshot skips live miss guard

Medium Severity

When prefetch is on, mark_environment_missing applies the next_checkin_latest guard to a batch-start snapshot instead of a live row, and mark_checkin_timeout does the same for has_newer_status_affecting_checkin. An ingest check-in that lands after prefetch can advance next_checkin_latest or add a newer OK, but the clock task still writes a miss or incident and can overwrite last_checkin, next_checkin, and status. That window is largest exactly when clock-tasks is backlogged.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 4ce075b. Configure here.

Comment on lines 98 to +109

# If the monitor has had any newer OK/ERROR status check-ins than this
# timeout, then this timeout cannot affect the status of the monitor.
if monitor_has_newer_status_affecting_checkins(monitor_environment, checkin.date_added):
if prefetch is not None:
has_newer = prefetch.has_newer_status_affecting_checkin(
monitor_environment.id, checkin.date_added
)
else:
has_newer = monitor_has_newer_status_affecting_checkins(
monitor_environment, checkin.date_added
)
if has_newer:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefetched newer-check-in guard can miss concurrent OK/ERROR

When prefetch is set, has_newer uses a batch-start snapshot instead of the live exists() query, so an OK/ERROR check-in inserted after prefetch still lets this timeout call mark_failed and incorrectly flip the monitor to ERROR.

Evidence
  • In this hunk, prefetch is not None replaces monitor_has_newer_status_affecting_checkins() with prefetch.has_newer_status_affecting_checkin().
  • That helper only reads ClockTaskPrefetch.newest_status_affecting, populated once in prefetch_clock_tasks() before the thread pool runs.
  • A concurrent monitor-consumer OK/ERROR for the same environment after prefetch is invisible to the snapshot, so has_newer stays false.
  • Execution continues into mark_failed(), unlike the non-prefetch path which re-queries date_added__gt=checkin.date_added at decision time.

Identified by Warden · sentry-backend-bugs · W8E-ACM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant