Repository navigation
the mirror took the first copy of a day and never looked again - #2820
Open
Flotapponnier wants to merge 1 commit into
Open
Flotapponnier wants to merge 1 commit into
Flotapponnier wants to merge 1 commit into
Conversation
Hyperliquid publishes a day short and sometimes completes it later. 3 October was still a 403 at 22:00 on the 4th and was whole, to 23:59:23, by 01:00 on the 5th. We only got the complete one by luck: the file did not exist at the usual fetch hour, so our first successful download landed after the upstream had finished writing it. 4 October shows the other outcome. It was fetched at 00:59 while still truncated at 12:13:24, and nothing in the harness would ever have gone back for it. syncWindow skipped any day already on disk. That is right for a day that cannot change and wrong for one that can, and the harness had no way to tell the two apart. Recent days are now revalidated instead of skipped. The conditional token is the mirrored file's own mtime, which already carries the CDN's Last-Modified (fetch has stamped it for the settle check since the feed edition), so no new state is stored. Measured on the live CDN: ETag and Last-Modified are both served and If-Modified-Since answers 304, which is what makes re-checking every recent day on every pass affordable. Only days inside the grace window are re-checked. Past it the upstream has never once gone back and filled a day in: 22 September to 2 October all stop near 12:11 UTC and have not moved in the two weeks since, byte for byte identical across re-downloads. Re-checking those would be pure load. A 304 is not a failure and does not count toward the sync failure budget. A 200 on a day we already held means the upstream republished, so the cached parse is of the old half and is dropped. This recovers 4 October if Hyperliquid completes it, and every day after. It does not recover 22 September to 2 October, which are truncated at the source for good; those need the API backfill or they age out of the window in November.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2814, which measured the problem. This one stops us making it worse.
What happens upstream
Hyperliquid publishes a day short and sometimes completes it later:
We only hold the complete 3 October by luck: the file did not exist at the usual fetch hour, so our first successful download landed after the upstream had finished writing it. Our mirror shows it fetched at 00:59 on the 5th.
4 October is the other outcome — fetched at 00:59 while still truncated, and nothing in the harness would ever have gone back for it.
What was wrong on our side
syncWindowskipped any day already on disk. Right for a day that cannot change, wrong for one that can, and the harness had no way to tell them apart.The fix
Recent days are revalidated instead of skipped. The conditional token is the mirrored file's own mtime, which already carries the CDN's
Last-Modified(fetch has stamped it for the settle check since the feed edition), so no new state is stored.Measured against the live CDN:
That 304 is what makes re-checking every recent day on every pass affordable.
Only days inside the grace window (3 by default) are re-checked. Past it the upstream has never once gone back and filled a day in: 22 September to 2 October all stop near 12:11 UTC and have not moved since, byte for byte identical across re-downloads taken a day apart. Re-checking those would be pure load.
A 304 is not a failure and does not count toward the sync failure budget. A 200 on a day we already held means the upstream republished, so the cached parse is of the old half and is dropped.
feedURLTemplatebecomes avarso the revalidation path can run against a local server in tests. Never reassigned in production.What it does and does not recover
userFillsByTimebackfill, or they age out of the 30-day window in November.Verified
go build,go vet,go test ./...,gofmtclean. Two new tests drive the real sequence against an httptest server: first fetch with nothing on disk sends no condition and stores the short body; the second costs a 304 and leaves the file alone; after the upstream republishes with a newerLast-Modified, the mirror takes the longer body. A 403 still reads as absent, not as unchanged.pnpm validate235 specs.