Refresh issue timelines when cross-references change (#168)

Iamrodos · Iamrodos · commit 28e8ce9e8ad5 · 2026-07-26T21:03:48.000+10:00
Cross-references do not change an issue's updated_at, so an incremental run never lists the issue and never re-fetches its timeline. An issue can be referenced from another repository years after its last activity and the backup would never see it. On incremental runs with --issue-timeline, ask GraphQL for each item's cross-reference count and newest timestamp, compare both against what is already saved, and re-fetch the timeline of anything that differs. Items the since listing did not return are fetched individually. Both values are needed: the count alone would miss a reference being added and another deleted between runs, and the timestamp alone would miss a deletion. Pull requests are swept too when they are being stored as issues, which is the case unless --pulls is used. They need their own query because the issues connection excludes them, and because totalCount ignores the itemTypes filter on a pull request although it honours it on an issue, so their cross-references are counted from the nodes instead. Where that count may have been truncated, the comparison falls back to the timestamp alone rather than re-fetching on every run. This costs one rate-limit point per 100 items against the GraphQL quota, measured at 6 requests for a repository with 528 issues and pull requests, and stores no new state: the comparison is against the cross-references already in each item's timeline_data. A GraphQL failure warns and leaves the rest of the backup unaffected. Closes #168
diff --git a/README.rst b/README.rst
@@ -366,9 +366,9 @@ Note that timeline entries are not all the same shape as events. ``cross-referen
 
 ``--issue-timeline`` is included in ``--all``. On issues with many cross-references it can be noticeably larger and slower than ``--issue-events`` alone, because each ``cross-referenced`` entry embeds the referencing issue in full. On a heavily cross-referenced issue this can mean several times the API requests and many times the JSON on disk. Ordinary repositories see little difference.
 
-If you enable ``--issue-timeline`` on an existing incremental backup, run once without ``--incremental`` to backfill it. There is no automatic backfill, so an incremental run only revisits recently updated issues and every older issue keeps a JSON file with no ``timeline_data`` until something touches it. The same applies to ``--incremental-by-files``.
+Incremental backups need extra help here, because being referenced from elsewhere does not change an issue's ``updated_at``. An issue can be mentioned from another repository years after its last activity, and the usual ``since`` listing will never report it as changed. On an incremental run with ``--issue-timeline``, github-backup therefore asks GitHub's GraphQL API for each item's cross-reference count and newest timestamp, compares those against what is already saved, and re-fetches the timeline of anything that differs. Pull requests are included in that check when they are being stored as issues, which is the case unless ``--pulls`` is used. The check needs no extra state on disk and also notices references that were later deleted. It costs one rate-limit point per 100 items checked, against GitHub's GraphQL quota, which is separate from the REST one. That is cheap in quota terms even on very large repositories, but it is not instant: a repository with 18,000 issues and pull requests took around 200 points and five minutes for the check alone.
 
-Incremental backups select issues by ``updated_at``, and a cross-reference does not change the referenced issue's ``updated_at``. An issue whose only new activity is being referenced elsewhere is therefore not revisited on that run. It corrects itself the next time the issue is touched, since each issue's timeline is fetched in full, so the only lasting gap is an issue that is referenced and then never touched again. Run occasionally without ``--incremental`` if that matters to you.
+If you enable ``--issue-timeline`` on an existing incremental backup, cross-references on older issues are picked up automatically by that check. Timeline entries of other kinds, such as commits and reviews, are only fetched when the issue itself is updated, so run once without ``--incremental`` if you want those filled in for historical issues too.
 
 
 About pull request reviews
diff --git a/github_backup/github_backup.py b/github_backup/github_backup.py
@@ -18,6 +18,7 @@
 import sys
 import threading
 import time
+import traceback
 from collections.abc import Generator
 from datetime import datetime
 from http.client import IncompleteRead
@@ -33,10 +34,14 @@
     VERSION = "unknown"
 
 from .graphql_queries import (
+    CROSS_REFERENCE_NODE_LIMIT,
+    CROSS_REFERENCE_PAGE_SIZE,
     DISCUSSION_DETAIL_QUERY,
     DISCUSSION_LIST_QUERY,
     DISCUSSION_PAGE_SIZE,
     DISCUSSION_REPLIES_QUERY,
+    ISSUE_CROSS_REFERENCE_COUNT_QUERY,
+    PULL_CROSS_REFERENCE_COUNT_QUERY,
 )
 
 FILE_URI_PREFIX = "file://"
@@ -2575,6 +2580,141 @@ def backup_discussions(args, repo_cwd, repository):
         )
 
 
+def cross_reference_state(timeline_data):
+    """Return (count, newest timestamp) of the cross-references in a timeline."""
+    created = [
+        item.get("created_at")
+        for item in timeline_data or []
+        if item.get("event") == "cross-referenced"
+    ]
+    return len(created), max((c for c in created if c), default=None)
+
+
+def _cross_reference_page(args, repository, query, connection, use_total_count):
+    """Page one GraphQL connection, yielding {number: (count, newest)}.
+
+    totalCount honours the itemTypes filter on issues but not on pull
+    requests, so pull request cross-references are counted from the nodes.
+    """
+    owner, name = _repository_owner_name(repository)
+    state = {}
+    after = None
+    page = 1
+
+    while True:
+        variables = {
+            "owner": owner,
+            "name": name,
+            "after": after,
+            "pageSize": CROSS_REFERENCE_PAGE_SIZE,
+        }
+        if not use_total_count:
+            # Only the pull request query declares this.
+            variables["nodeLimit"] = CROSS_REFERENCE_NODE_LIMIT
+
+        data = retrieve_graphql_data(
+            args,
+            query,
+            variables,
+            log_context="cross-references {0} {1} page {2}".format(
+                repository["full_name"], connection, page
+            ),
+        )
+        repository_data = data.get("repository")
+        if repository_data is None:
+            raise Exception(
+                "Repository {0} not found in GraphQL response".format(
+                    repository["full_name"]
+                )
+            )
+
+        items = repository_data.get(connection) or {}
+        for node in _connection_nodes(items):
+            timeline = node.get("timelineItems") or {}
+            created = [n.get("createdAt") for n in timeline.get("nodes") or [] if n]
+            if use_total_count:
+                count = timeline.get("totalCount", 0)
+            elif len(created) >= CROSS_REFERENCE_NODE_LIMIT:
+                # Counted from the nodes, so it may be truncated. Compare on
+                # the timestamp alone rather than on a count we cannot trust.
+                count = None
+            else:
+                count = len(created)
+            state[node["number"]] = (count, created[-1] if created else None)
+
+        page_info = items.get("pageInfo") or {}
+        if not page_info.get("hasNextPage"):
+            break
+
+        after = page_info.get("endCursor")
+        page += 1
+
+    return state
+
+
+def retrieve_cross_reference_state(args, repository, include_pulls=False):
+    """Ask GitHub for each item's cross-reference count and newest timestamp."""
+    state = _cross_reference_page(
+        args,
+        repository,
+        ISSUE_CROSS_REFERENCE_COUNT_QUERY,
+        "issues",
+        use_total_count=True,
+    )
+    if include_pulls:
+        state.update(
+            _cross_reference_page(
+                args,
+                repository,
+                PULL_CROSS_REFERENCE_COUNT_QUERY,
+                "pullRequests",
+                use_total_count=False,
+            )
+        )
+    return state
+
+
+def find_stale_timeline_issues(args, issue_cwd, repository, include_pulls=False):
+    """Issue numbers whose stored cross-references no longer match GitHub's.
+
+    Incremental runs select issues by updated_at, which a cross-reference does
+    not change, so those issues would otherwise never be revisited.
+    """
+    if not get_graphql_auth(args):
+        # The check needs the GraphQL API, which always requires a token.
+        logger.debug("Skipping cross-reference check, no authentication")
+        return set()
+
+    try:
+        live = retrieve_cross_reference_state(args, repository, include_pulls)
+    except Exception as e:
+        logger.warning(
+            "Unable to check {0} for new cross-references, "
+            "issues referenced since the last backup may be missed: {1}".format(
+                repository["full_name"], e
+            )
+        )
+        logger.debug(traceback.format_exc())
+        return set()
+
+    stale = set()
+    for number, (count, newest) in live.items():
+        existing = read_json_file_if_exists("{0}/{1}.json".format(issue_cwd, number))
+        if existing is None:
+            continue
+        stored = cross_reference_state(existing.get("timeline_data"))
+        changed = stored[1] != newest if count is None else stored != (count, newest)
+        if changed:
+            stale.add(number)
+
+    if stale:
+        logger.info(
+            "Refreshing {0} issues with new cross-references".format(len(stale))
+        )
+        logger.debug("Stale timelines: {0}".format(sorted(stale)))
+    return stale
+
+
 def backup_issues(args, repo_cwd, repository, repos_template):
     has_issues_dir = os.path.isdir("{0}/issues/.git".format(repo_cwd))
     if args.skip_existing and has_issues_dir:
@@ -2590,6 +2730,7 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     _issue_template = "{0}/{1}/issues".format(repos_template, repository["full_name"])
 
     should_include_pulls = args.include_pulls or args.include_everything
+    include_timeline = args.include_issue_timeline or args.include_everything
     issue_states = ["open", "closed"]
     for issue_state in issue_states:
         query_args = {"filter": "all", "state": issue_state}
@@ -2606,6 +2747,30 @@ def backup_issues(args, repo_cwd, repository, repos_template):
 
             issues[issue["number"]] = issue
 
+    stale_timelines = set()
+    incremental_run = args.since or args.incremental_by_files
+    if (
+        include_timeline
+        and incremental_run
+        and any(f.endswith(".json") for f in os.listdir(issue_cwd))
+    ):
+        stale_timelines = find_stale_timeline_issues(
+            args, issue_cwd, repository, include_pulls=not should_include_pulls
+        )
+        for number in sorted(stale_timelines - set(issues)):
+            try:
+                issues[number] = retrieve_data(
+                    args, "{0}/{1}".format(_issue_template, number), paginated=False
+                )[0]
+            except (HTTPError, RepositoryUnavailableError, IndexError) as e:
+                # Deleted or transferred between the sweep and this fetch. The
+                # refresh is opportunistic, so skip rather than fail the run.
+                logger.warning(
+                    "Unable to refresh cross-references for issue {0}: {1}".format(
+                        number, e
+                    )
+                )
+
     if issues_skipped:
         issues_skipped_message = " (skipped {0} pull requests)".format(issues_skipped)
 
@@ -2619,7 +2784,11 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     timeline_template = _issue_template + "/{0}/timeline"
     for number, issue in list(issues.items()):
         issue_file = "{0}/{1}.json".format(issue_cwd, number)
-        if args.incremental_by_files and os.path.isfile(issue_file):
+        if (
+            args.incremental_by_files
+            and os.path.isfile(issue_file)
+            and number not in stale_timelines
+        ):
             modified = os.path.getmtime(issue_file)
             modified = datetime.fromtimestamp(modified).strftime("%Y-%m-%dT%H:%M:%SZ")
             if modified > issue["updated_at"]:
@@ -2636,7 +2805,7 @@ def backup_issues(args, repo_cwd, repository, repos_template):
         if args.include_issue_events or args.include_everything:
             template = events_template.format(number)
             issues[number]["event_data"] = retrieve_data(args, template)
-        if args.include_issue_timeline or args.include_everything:
+        if include_timeline:
             template = timeline_template.format(number)
             # "commented" timeline entries embed the full comment body, which
             # --issue-comments already backs up; storing them twice invites the
diff --git a/github_backup/graphql_queries.py b/github_backup/graphql_queries.py
@@ -2,6 +2,66 @@
 
 DISCUSSION_PAGE_SIZE = 100
 
+CROSS_REFERENCE_PAGE_SIZE = 100
+
+# Cross-references do not change an issue's updatedAt, so the REST listing
+# cannot report them as changed. Count and newest timestamp are compared
+# against what is on disk: the count alone would miss an add paired with a
+# delete, the timestamp alone would miss a delete.
+ISSUE_CROSS_REFERENCE_COUNT_QUERY = """
+query($owner: String!, $name: String!, $after: String, $pageSize: Int!) {
+  repository(owner: $owner, name: $name) {
+    issues(first: $pageSize, after: $after) {
+      nodes {
+        number
+        timelineItems(last: 1, itemTypes: [CROSS_REFERENCED_EVENT]) {
+          totalCount
+          nodes {
+            ... on CrossReferencedEvent {
+              createdAt
+            }
+          }
+        }
+      }
+      pageInfo {
+        hasNextPage
+        endCursor
+      }
+    }
+  }
+}
+"""
+
+# Pull requests are stored as issues when --pulls is not used, and the issues
+# connection above excludes them. This query is shaped differently because
+# totalCount ignores itemTypes on a pull request, unlike on an issue, so the
+# cross-references have to be counted from the nodes themselves.
+CROSS_REFERENCE_NODE_LIMIT = 100
+
+PULL_CROSS_REFERENCE_COUNT_QUERY = """
+query($owner: String!, $name: String!, $after: String, $pageSize: Int!,
+      $nodeLimit: Int!) {
+  repository(owner: $owner, name: $name) {
+    pullRequests(first: $pageSize, after: $after) {
+      nodes {
+        number
+        timelineItems(last: $nodeLimit, itemTypes: [CROSS_REFERENCED_EVENT]) {
+          nodes {
+            ... on CrossReferencedEvent {
+              createdAt
+            }
+          }
+        }
+      }
+      pageInfo {
+        hasNextPage
+        endCursor
+      }
+    }
+  }
+}
+"""
+
 DISCUSSION_LIST_QUERY = """
 query($owner: String!, $name: String!, $after: String, $pageSize: Int!) {
   repository(owner: $owner, name: $name) {
diff --git a/tests/test_issue_timeline.py b/tests/test_issue_timeline.py
