Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

17 changes: 16 additions & 1 deletion qb_site/syncer/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,21 @@ docker compose exec -T web python qb_site/manage.py bootstrap_archive_worklist \
docker compose exec -T web python qb_site/manage.py resync_archive_touched_prs \
--repo leanprover-community/mathlib4 --apply --limit 1000

# Timeline actor-type backfill (design doc 051). Re-resolves each stored
# PRTimelineEvent.github_node_id through GitHub's nodes(ids:) root field to
# fill actor_type / actor_node_id, and fills archive rows' missing
# actor_login in the same pass (fill-only — never overwrites a stored login).
# A full 100-id call costs 1 GraphQL point, so the whole table is ~6k points.
# Resumable and idempotent (target set is `actor_type IS NULL`); repeat runs
# plateau at the genuinely-null-actor population rather than reaching zero.
# Watch progress per repo on the SyncerConvergenceSnapshot admin page:
# `timeline_events_missing_actor_type` (the command's own target set) and
# `timeline_events_untyped_with_login` (converges to ~0).
docker compose exec -T web python qb_site/manage.py backfill_timeline_actor_types \
--repo leanprover-community/mathlib4 --dry-run --limit 500
# Unattended drain (sleeps until resetAt instead of stopping at the floor):
docker compose exec -T web python qb_site/manage.py backfill_timeline_actor_types --wait-for-rate

# App tests
docker compose exec -T web env DJANGO_SETTINGS_MODULE=qb_site.settings.ci python qb_site/manage.py test syncer
```
Expand Down Expand Up @@ -130,7 +145,7 @@ front and expensive to recover from when skipped.
- `syncer.harvest_commit_history` / `syncer.harvest_commit_history_sweep` (optional),
- `syncer.archive_import_tick` → `syncer.archive_import_pr_item` — beat-driven worklist drain for the archive backfill importer (design doc 043). Tick runs every `ARCHIVE_IMPORT_TICK_SECONDS` (default 60s) and gates on `ARCHIVE_IMPORT_ENABLED` so operators can toggle activity without restarting beat. Status surface: `python manage.py archive_import_status [--repo OWNER/NAME] [--errors N]`.
- `syncer.resync_archive_touched_tick` — beat-driven drain for the doc-043 forced-resync remediation. Every `ARCHIVE_RESYNC_TICK_SECONDS` (default 600s) it enqueues up to `ARCHIVE_RESYNC_PER_TICK` (default 0 = disabled) `sync_pr(force=True)` tasks from `archive_touched_resync_targets` (open first, stalest `last_synced_at` first, healed PRs excluded), skipping the tick when the cached GraphQL budget is below `ARCHIVE_RESYNC_MIN_RATE_REMAINING` and deduping against still-queued sync_pr enqueues. Self-completing: returns `status=drained` once the target set is empty; `remaining` in the task result tracks progress, and `syncer.collect_convergence` records the same count as `archive_resync_remaining` on `SyncerConvergenceSnapshot` for admin monitoring.
- `syncer.collect_convergence` — records syncer convergence metrics,
- `syncer.collect_convergence` — records syncer convergence metrics. Includes the doc-051 actor-typing counters: `timeline_events_missing_actor_type` is the `backfill_timeline_actor_types` target set and plateaus at the genuinely-null-actor population, while `timeline_events_untyped_with_login` counts only rows known to have had an actor, so it converges to ~0 and is the standing canary for the fill-empty column allowlist in `sync_timeline_events`,
- `syncer.collect_metrics` — records sync throughput/lag metrics.
- Keep task behavior idempotent and retry-safe; prefer explicit status/reason payloads in return dicts.

Expand Down
10 changes: 9 additions & 1 deletion qb_site/syncer/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -904,20 +904,22 @@ def short_after_sha(self, obj: PRTimelineEvent) -> str: # pragma: no cover - si
"type",
"occurred_at",
"actor_login",
"actor_type",
"label_name",
"assignee_login",
"requested_reviewer_login",
"requested_team_slug",
"short_before_sha",
"short_after_sha",
)
list_filter = ("pull_request__repository", "type")
list_filter = ("pull_request__repository", "type", "actor_type")
search_fields = (
"label_name",
"pull_request__number",
"before_sha",
"after_sha",
"actor_login",
"actor_node_id",
"assignee_login",
"requested_reviewer_login",
"requested_team_slug",
Expand All @@ -933,6 +935,8 @@ def short_after_sha(self, obj: PRTimelineEvent) -> str: # pragma: no cover - si
"label_name",
"assignee_login",
"actor_login",
"actor_type",
"actor_node_id",
"before_sha",
"after_sha",
"extra",
Expand Down Expand Up @@ -1322,6 +1326,8 @@ class SyncerConvergenceSnapshotAdmin(ReadOnlyAdmin):
"archive_completed",
"archive_failed_permanent",
"archive_resync_remaining",
"timeline_events_missing_actor_type",
"timeline_events_untyped_with_login",
)
list_filter = ("repository", "history_cursor_completed", "discovery_continuation_active")
date_hierarchy = "collected_at"
Expand Down Expand Up @@ -1350,6 +1356,8 @@ class SyncerConvergenceSnapshotAdmin(ReadOnlyAdmin):
"archive_completed",
"archive_failed_permanent",
"archive_resync_remaining",
"timeline_events_missing_actor_type",
"timeline_events_untyped_with_login",
"created_at",
)

Expand Down
Loading