Skip to content

Changelog CDN fetchers: skip unchanged folders via shallow registry maps - #3801

Open
cotti wants to merge 1 commit into
changelog-registry-scrubber-ownershipfrom
changelog-shallow-map-consumer
Open

Changelog CDN fetchers: skip unchanged folders via shallow registry maps#3801
cotti wants to merge 1 commit into
changelog-registry-scrubber-ownershipfrom
changelog-shallow-map-consumer

Conversation

@cotti

@cotti cotti commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Implements the consumer half of the shallow per-tree registry maps introduced in #3738 (stacked on that PR's branch): the bundle CDN fetcher now consults bundle/registry.json before fetching per-product registries, and skips folders whose content demonstrably hasn't changed.

Refs: elastic/docs-eng-team#737, elastic/docs-eng-team#688, #3738.

What it does

CdnChangelogFetcher fetches the tree's shallow map (bundle/registry.json, shape {"<product>": "<token>"}) once per fetcher run, memoized per base URI. For each product:

  • Token match + warm cache → the per-product bundle/{product}/registry.json fetch is skipped and the locally cached registry is reused. Bundle content then resolves through the existing ETag-keyed cache, so an unchanged folder with a warm cache produces zero per-folder requests (asserted in tests).
  • Token mismatch, product missing from the map, or no cached state → the flow is byte-identical to today's: fetch the registry, download bundles, and record the fresh registry under the folder's current token for the next run.

Graceful degradation (non-negotiable, preserved)

A shallow map that is absent (404), unparseable, or fails to fetch degrades to null inside FetchShallowMapAsync — a single debug log, no errors, no warnings — and every per-product registry is fetched exactly as before the map existed. Pre-cutover CDNs and buckets without maps keep working unchanged. Tokens are treated as fully opaque: compared with ordinal string equality only, never parsed.

Cache/token bookkeeping design

  • The registry text is cached under key registry-{product}-{token} in the fetcher's existing memory + disk cache ({ApplicationData}/changelog-bundles/), the same store and conventions the ETag-keyed bundle cache (changelog-{product}-{file}-{etag}) already uses. Embedding the token in the key makes a token mismatch a plain cache miss under the new key — no separate "last-seen token" state to keep consistent.
  • This keying is sound because the producer (ShallowRegistryReconciler) excludes group manifests from the token digest precisely so rewriting a manifest cannot invalidate consumer caches: same token ⇒ same folder listing ⇒ same derived registry.
  • The shallow map itself is deliberately never cached to disk — it is the freshness signal — and is fetched at most once per fetcher run regardless of product count.
  • Defense-in-depth: a token that is not a safe single path segment is ignored (treated as "no token") before it can be joined into a cache file path, mirroring the fetchers' existing file-name validation.

Deliberately not covered: CdnChangelogEntryFetcher

The changelog-pool entry fetcher (changelog/{org}/{repo}/{branch}) has no local cache of any kind — it downloads entries and returns them directly to the bundle command. A matching pool token could therefore never skip anything: there is no cached state a skip could reuse, and honoring the issue's "no behavior change" constraint would require inventing a parallel content store for entry YAML. Per the review direction ("scope the opt-out to whatever cache genuinely exists; correctness over cleverness"), that fetcher is left unchanged; fetching changelog/registry.json there would only add a wasted request per run. If a pool-level entry cache is ever introduced, the same token-keyed pattern applies directly.

Test plan

All in CdnChangelogFetcherTests (fake HttpMessageHandler + MockFileSystem, per the existing pattern):

  • Map absent (404) → identical behavior to today (registry + bundles fetched, no diagnostics).
  • Map unparseable → identical behavior, no crash.
  • Token match with warm cache → the fake handler sees only the single map request; results served entirely from cache.
  • Token mismatch → normal fetch, fresh registry recorded under the new token.
  • Token present but cold cache → normal fetch on run one, full skip on a second run sharing the disk cache.
  • Partial run across two products (one matching, one not) → only the changed product's folder is hit; the map is fetched exactly once.
  • Existing ETag-cache tests updated for the one extra map probe per run.

Checks run: dotnet format (clean), ./build.sh build --skip-dirty-check (pass), dotnet publish src/tooling/docs-builder -c Release AOT publish (zero trim/AOT warnings; the shallow map type is registered on the source-generated ChangelogRegistryJsonContext), dotnet test for Elastic.Documentation.Configuration.Tests (608 passed) and Elastic.Changelog.Tests (865 passed). No CLI surface changes, so no docs/cli-schema.json regen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant