Incremental backup skips some new events

[behrisch]

Our project https://github.com/eclipse/sumo/ has some large meta tickets like eclipse-sumo/sumo#21 where the backup does not pickup the new events anymore. When looking at the log, the issue does not even show up in the list of updated issues. This might be an upstream problem, at least I found no clear definition of what the "since" parameter in the query considers as an update and there is at least one report at https://github.community/t/github-api-how-to-get-issues-since-create-time/13680 which talks about a similar problem.

[Iamrodos]

Dug into this one. It's not the since parameter alone, it's the endpoint. --issue-events reads /issues/{number}/events, and cross-reference events ("mentioned in #X") never appear there, GitHub only exposes them via the timeline API. So full backups miss them too, not just incremental ones.

Have confirmed with a controlled test: mentioning issue A from issue B leaves A's updated_at untouched, A's /events stays empty, and the cross-referenced event appears only in A's /timeline; issues?since=... accordingly never lists A. Same picture on eclipse-sumo/sumo#21 from the original report: /events returns 310 items all from the 2017 repo migration, while /timeline has 2,130 items including 193 cross-references through 2024.

The fix: fetch /issues/{number}/timeline instead of /events in the per-issue loop. I verified the classic event types (labeled, closed, referenced, …) come back byte-identical from both endpoints, so existing event_data consumers are unaffected for those; commented items need filtering out (they embed full comment bodies, already covered by --issue-comments). The newly captured types (cross-referenced etc.) have their own shape — no id, a source object identifying the referencer — so event_data becomes a heterogeneous list, which loose JSON consumers won't notice.

One caveat that survives the fix: incremental runs select issues by updated_at, so an issue whose only new activity is a cross-reference won't be revisited that run. It self-heals on the next touch (the per-issue fetch always rewrites the full history), so the only permanent miss is an issue that's referenced and then never touched again — worth documenting, with an occasional full run as the recommendation, rather than trying to solve.

My recommendation is swap to timeline and pickup all events and live with the incremental nuance.

Happy to put together a PR along these lines with plenty of testing.

[Iamrodos]

Correcting something from my earlier comment: the timeline is not a superset of the events endpoint. That held on the handful of issues I sampled, but across a full backup of this repository, 130 of 254 referenced entries appear only in /events. All of them sit on pull requests, where a commit elsewhere mentions the PR and the timeline carries the PR's own committed entries instead. So swapping --issue-events over to the timeline, which is what I suggested, would have quietly dropped half of those.

#528 takes the additive route instead. A new --issue-timeline flag stores the timeline under its own timeline_data key and leaves event_data exactly as it is, so the two are complementary rather than alternatives and both are worth running.

Your diagnosis about since was right too, and it needed a separate fix. A cross-reference genuinely does not change the referenced issue's updated_at: across 178 cross-referenced issues here, not one had a matching timestamp and 45 had references dated after it. An incremental run therefore never lists the issue and never re-fetches its timeline. So on incremental runs the PR also asks GraphQL for each item's cross-reference count and newest timestamp, and re-fetches anything that disagrees with what is already stored. That costs one rate-limit point per 100 items and needs no extra state on disk.

On sumo#21 specifically, /events returns 310 entries and not one is a cross-reference, while /timeline returns 2,130 including 194 cross-references running through to this year. Those are what the backup has been missing.