Move public changelog registry ownership to the scrubber Lambda

[cotti]

Why

Phase 1 of elastic/docs-eng-team#688. The public registry.json is produced client-side against the private bucket and pass-through copied by the scrubber Lambda, which makes it a log of upload operations rather than a description of the public bucket. Every known consistency gap follows from that: ad-hoc uploads leave it stale (the 2026-07-22 incident), --skip-etag-check re-uploads can never reach a stale public manifest, deletions orphan entries, failed scrubs leave phantom entries, and the recorded ETag is documented as useless.

Changes since the review

The review rejected per-product manifests as a growing inventory with little consumer value, and asked for shallow folder→ETag maps instead. Reworked accordingly — the two trees now part ways:

• bundle/{product}/registry.json stays, reconciler-owned. This is the one place consumers genuinely enumerate: the {changelog} directive and external CDN consumers list a product's bundles, and date-based targets (serverless) are not derivable client-side. Registry-key events never pass through; they only schedule a group reconcile that derives the manifest from public state.
• Pool manifests (changelog/{org}/{repo}/{branch}/registry.json) are no longer reconciled. Release-note discovery starts from PR lists (RFC elastic/docs-eng-team#698), so the scrubber stops owning these. They remain client-authored and are mirrored verbatim (state-decided, so Phase 3's private-manifest cleanup deletes will propagate) until Phase 3 retires them entirely — changelog bundle still enumerates a pool through its manifest today, and 404-probing only works once entries are guaranteed one-per-PR.
• New shallow per-tree maps, maintained by the scrubber: bundle/registry.json and changelog/registry.json (matching the existing singular key layout), shaped {"<folder>": "<token>"}. One map write per touched tree per batch; optimistic concurrency (If-Match/If-None-Match: *, bounded jittered retries, batch-item failure on exhaustion); an absent or unparseable map is rebuilt from one full tree listing (which is also the first-deploy seed path); the map is conditionally deleted when its tree empties.
• One deliberate deviation from the review's literal shape: the map value is a digest over the folder's sorted file/ETag listing, not the ETag of the last-touched file. A last-touched ETag goes stale when an older file is deleted — the newest object, and therefore the value, would not change — so deletions would be invisible to caches. Consumers must treat the value as an opaque change token (covered by Reconcile_TokenChangesWhenAnyFileChangesOrIsDeleted).
• Nothing consumes the shallow maps yet. They are the cache opt-out layer future caching sits on; publishing them costs one small conditional write per touched tree.

What

The scrubber Lambda becomes the sole producer of the public bundle manifests and the shallow maps, computed by reconciling against the current state of the buckets — registry = f(state), never f(event):

• Events are triggers; state decides. The handler ignores the event type: each distinct key in a batch gets one object-level reconcile against the private bucket (present → scrub current content and copy, or mirror verbatim for pass-through pool manifests; absent → delete the public copy), with post-write HEAD validation that redoes the work if the source changed mid-flight. Each distinct bundle/{product}/ group then gets one registry reconcile from the public listing, and each touched tree gets one shallow-map reconcile.
• Group reconcile lists the group's prefix (/-delimited so verbatim branches don't sweep nested pools, paginated), reuses entries whose recorded ETag still matches, recomputes the rest from the scrubbed public YAMLs (amends always recomputed — their target depends on the parent), and writes back with optimistic concurrency. An empty group's manifest is conditionally deleted — absent ≠ empty for consumers. A manifest with a newer schema_version is reported and left untouched.
• producer field on the bundle manifest identifies the reconcile algorithm version; any mismatch — including every legacy pass-through manifest at cutover — forces a full metadata recompute and a write even when entries come out identical. ETags recorded are now the public object's, finally usable by CDN consumers.
• The Lambda's top-level handler logic lives in testable classes in Elastic.Changelog (ScrubberProcessor, RegistryReconciler, ShallowRegistryReconciler, ChangelogContentScrubber); Program.cs is a thin adapter. Per-invocation reconcile metrics are emitted as CloudWatch EMF via a source-generated contract, now including shallow-map write/unchanged counters (the docs-eng-team#688 Phase 0 observability item). The FakeS3 test fake (stateful, MD5 ETags, real conditional PUT/DELETE semantics, delimiter+pagination, interleaving hooks) is harvested from Add changelog registry state discovery and reconciliation commands #3717.

The guarantee is convergence, not atomicity: any successfully processed event repairs all accumulated drift in the group and tree it touches. Consumers' listed-but-missing tolerance stays. The CLI upload path is untouched in this phase (#3760 retires the client-side refresh); rollback = revert the Lambda deploy.

Requires the IAM grants from elastic/docs-infra#360 (s3:GetObject/s3:ListBucket on the public bucket) at deploy time.

Testing

• Elastic.Changelog.Tests: 868/868 passing — including 12 ShallowRegistryReconciler tests (seeding, patching, deletion visibility, nested pools, unparseable-map rebuild, conditional-write races and exhaustion) and new ScrubberProcessor tests for pool pass-through and shallow-map coalescing.
• ./build.sh build --skip-dirty-check clean; scrubber Lambda publishes Native AOT with zero trim warnings.
• No CLI surface change (docs/cli-schema.json untouched); docs rewrite follows in Rewrite changelog registry docs for scrubber Lambda ownership #3761 once this shape settles.

Part of elastic/docs-eng-team#688 (Phase 1).

[Mpdreamz]

This is overly defensive IMO, we use AOT trimming and source generated json contracts heavily. We should not fallback to handcrafted json emission.

These are emitted as cloudwatch metrics, should we push (some of) them as OTEL metrics too?

[Mpdreamz]

Request changes: drop per-product changelogs/<product>/registry.json

Pushing back on the core assumption in this stack — that we need (and should reconcile) a per-product / per-group changelogs/<product>/registry.json as a description of every changelog object under that prefix.

Why we don't need it

Release-note discovery is not "walk the registry and see what's there." The automated release discover moments (see elastic/docs-eng-team#698) always start from a list of PRs:

• Prestage → query for list of PRs
• GitHub release → already have a list of PRs
• Serverless release → query for list of PRs

Given that list, we can hit CloudFront and decide whether a checked-in changelog exists by whether the object returns 404. If it 404s, changelog bundle inlines a changelog for that PR into the bundle anyway.

The changelogs/ tree on S3 is only a mirror of explicitly checked-in changelogs in repositories. Those objects are not mandatory to exist at bundle time. A growing changelogs/<product>/registry.json (or changelogs/products/registry.json) would just accumulate forever for little consumer value — please remove that model and the reconcile/heal machinery built around it.

A lot of the code in this PR (group listing → entry-level reconcile → per-group manifest write/delete, producer/schema cutover heal, etc.) exists to keep that per-product registry correct. With the assumption dropped, that surface can shrink substantially.

What we do want instead

Keep (or introduce) two shallow folder-level registries on S3 for cache opt-out only:

• changelogs/registry.json
• bundles/registry.json (or bundle/registry.json — match existing key layout)

Shape:

{
  "<folder>": "<etag>"
}

Where <etag> is the ETag of the last-touched file under that folder. Caching layers can then bail out eagerly before diving into a per-folder / per-product bundle registry.

Please rework this PR around that thinner contract: scrubber remains the public-bucket writer for changelog objects, but it should maintain the shallow folder→etag map — not a full per-product object inventory.

[cotti]

Agreed on the thinner contract. I'll rework this PR so the scrubber maintains the shallow folder→ETag maps (changelogs/registry.json, bundles/registry.json) for cache opt-out instead of per-product object manifests, and drop the group reconcile/heal machinery that existed to keep those manifests correct. Converting to draft in the meantime.