Changelog CDN fetchers: skip unchanged folders via shallow registry maps - #3801
Open
cotti wants to merge 1 commit into
Open
Changelog CDN fetchers: skip unchanged folders via shallow registry maps#3801cotti wants to merge 1 commit into
cotti wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the consumer half of the shallow per-tree registry maps introduced in #3738 (stacked on that PR's branch): the bundle CDN fetcher now consults
bundle/registry.jsonbefore fetching per-product registries, and skips folders whose content demonstrably hasn't changed.Refs: elastic/docs-eng-team#737, elastic/docs-eng-team#688, #3738.
What it does
CdnChangelogFetcherfetches the tree's shallow map (bundle/registry.json, shape{"<product>": "<token>"}) once per fetcher run, memoized per base URI. For each product:bundle/{product}/registry.jsonfetch is skipped and the locally cached registry is reused. Bundle content then resolves through the existing ETag-keyed cache, so an unchanged folder with a warm cache produces zero per-folder requests (asserted in tests).Graceful degradation (non-negotiable, preserved)
A shallow map that is absent (404), unparseable, or fails to fetch degrades to
nullinsideFetchShallowMapAsync— a single debug log, no errors, no warnings — and every per-product registry is fetched exactly as before the map existed. Pre-cutover CDNs and buckets without maps keep working unchanged. Tokens are treated as fully opaque: compared with ordinal string equality only, never parsed.Cache/token bookkeeping design
registry-{product}-{token}in the fetcher's existing memory + disk cache ({ApplicationData}/changelog-bundles/), the same store and conventions the ETag-keyed bundle cache (changelog-{product}-{file}-{etag}) already uses. Embedding the token in the key makes a token mismatch a plain cache miss under the new key — no separate "last-seen token" state to keep consistent.ShallowRegistryReconciler) excludes group manifests from the token digest precisely so rewriting a manifest cannot invalidate consumer caches: same token ⇒ same folder listing ⇒ same derived registry.Deliberately not covered:
CdnChangelogEntryFetcherThe changelog-pool entry fetcher (
changelog/{org}/{repo}/{branch}) has no local cache of any kind — it downloads entries and returns them directly to the bundle command. A matching pool token could therefore never skip anything: there is no cached state a skip could reuse, and honoring the issue's "no behavior change" constraint would require inventing a parallel content store for entry YAML. Per the review direction ("scope the opt-out to whatever cache genuinely exists; correctness over cleverness"), that fetcher is left unchanged; fetchingchangelog/registry.jsonthere would only add a wasted request per run. If a pool-level entry cache is ever introduced, the same token-keyed pattern applies directly.Test plan
All in
CdnChangelogFetcherTests(fakeHttpMessageHandler+MockFileSystem, per the existing pattern):Checks run:
dotnet format(clean),./build.sh build --skip-dirty-check(pass),dotnet publish src/tooling/docs-builder -c ReleaseAOT publish (zero trim/AOT warnings; the shallow map type is registered on the source-generatedChangelogRegistryJsonContext),dotnet testforElastic.Documentation.Configuration.Tests(608 passed) andElastic.Changelog.Tests(865 passed). No CLI surface changes, so nodocs/cli-schema.jsonregen.