fix(sitesearch): reconcile ES/OS index mirrors on the write path (#36360)

[fabrizzio-dotCMS]

Problem

A Site Search index is one logical index mirrored across both engines (ES + its .os OpenSearch twin). Reads were made phase-aware in #36797, but the write path can still leave the two engines physically out of sync, so Site Search returns stale or empty results once reads move to OpenSearch. This is issue #36360's write-path half (the read half shipped in #36797).

Failure modes (paths that skip the full-crawl happy path)

1. Forward-only phase change — Phase 0→1/2 does not retroactively build OS twins of indices that already existed in ES.
2. Phase-0 crawl builds an ES-only index; its OS twin never existed.
3. Incremental crawl writes documents in place (no createSiteSearchIndex) — two desync paths: (a) if the OS twin is missing, the raw putToIndex lets OpenSearch auto-create it with a dynamic mapping (keyword→text, breaking aggregations) holding only the delta → partial, wrongly-mapped twin; (b) if the twins already drifted, an incremental only appends the new delta to both and never reconciles the difference.
4. Fire-and-forget shadow create/write — a failed OS create or putToIndex in a dual-write phase is swallowed at WARN. Main source of content drift: the OS twin exists but silently holds fewer docs than ES.
5. Phase-scoped delete — a delete fanned only to the current phase's write providers leaves a twin on the other engine as an orphan after a rollback.

Fix — self-heal on crawl (not a big-bang rebuild on phase change)

• SiteSearchAPI#existsOnAllWriteEngines(name) — whether the index exists on every current write engine (ES checks the plain name, OS the .os twin; router aggregates over writeProviders()).
• SiteSearchAPI#writeMirrorsInSync(name) — existence plus document-count parity across write engines (each leaf counts its own physical index via a match-all rows=0). Catches content drift that existence alone cannot. Count parity is sound here because Site Search is single-writer with immediate refresh and no concurrent crawl on the same index, so copies are quiescent at crawl-planning time.
• Incremental-crawl gate (SiteSearchJobImpl) — an incremental crawl runs only when writeMirrorsInSync is true; a missing twin OR a count mismatch demotes it to a full rebuild, which recreates identical copies (correct mapping) on every engine and re-points the alias. Self-heals both the missing/partial twin and content drift on the next crawl, and removes the dynamic-mapping auto-create path (FM1–FM4).
• SiteSearchAPIImpl#deleteIndex sweeps both engines — primary (current read provider) delete is authoritative and propagates; the other engine is best-effort and failure-tolerant (unreachable engine → WARN, never fails the delete), so a rollback leftover is not orphaned even in a single-provider phase (FM5).

Residual limitation (documented)

Reconciliation is on crawl, so the gap is only closed once a crawl runs in a dual-write phase. In the no-crawl window: in Phase 2 a missing twin is covered by the read fallback (OS errors → read ES), a partial-on-create twin can no longer be created, and content drift heals on the next crawl — but a drifted twin read before that crawl still returns incomplete results silently. Phase 3 is the cliff (ES decommissioned, no fallback). Paired operational rule: crawl each index once before promoting phase. A targeted verify/repair job (detect twins missing or count-divergent → rebuild full) that closes the no-crawl window without depending on a crawl is deferred as a follow-up under #36360.

Tests

• SiteSearchRouterReconciliationTest (unit, mocked leaves, no cluster — runs in surefire): pins existsOnAllWriteEngines aggregation (dual-write vs Phase 0), writeMirrorsInSync (equal counts / count drift / missing twin / Phase 0), and deleteIndex both-engine sweep (authoritative primary, best-effort/tolerant secondary, active-index guard). ✅ 11/11.
• SiteSearchDualWriteRouterIT: fixes pre-existing OS existence assertions that used the logical (untagged) name while the physical twin is .os-tagged since Align OpenSearch Site Search index naming with the .os + versioned-store migration pattern #36672 (only surfaces on a real separate-cluster run, where this IT is gated); adds reconciliation scenarios (missing OS twin flips existsOnAllWriteEngines; content drift flips writeMirrorsInSync; delete sweeps an orphan twin in a single-provider phase).

Docs

docs/backend/OPENSEARCH_MIGRATION.md — new "Site Search mirror reconciliation (self-heal on crawl)" subsection: failure modes, the two fixes, the no-crawl window, and the operational rule.

Notes

• Stacked on fix(sitesearch): resolve aliases & index stats .os-aware in OpenSearch read phases (#36360) #36797 (issue-36360-sitesearch-alias-os-resolution). Based against that branch so the diff shows only the reconciliation changes; GitHub will retarget to main once fix(sitesearch): resolve aliases & index stats .os-aware in OpenSearch read phases (#36360) #36797 merges.
• Fragile zone (two-strikes rule): validated by the router unit test in CI; the dual-cluster IT scenarios require a separate ES+OS setup (local / dual-cluster profile).

🤖 Generated with Claude Code

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[claude]

Claude finished @fabrizzio-dotCMS's task in 9m 39s —— View job

---

Review: PR #36825 — sitesearch mirror reconciliation

• Read the full diff against origin/main
• Review write-path reconciliation (SiteSearchAPIImpl, routers)
• Review incremental-crawl gate (SiteSearchJobImpl)
• Review document-count implementations (ES/OS leaves)
• Check tests + docs
• Post review

The design is sound and the follow-up commits address the earlier count-parity / fail-safe feedback well. The documentCount (_count on ES, track_total_hits on OS), the -1 fail-safe treated as out-of-sync, and the extracted incrementalEligible predicate are all correct. Two small items below; neither blocks merge.

New Issues

• 🟡 Medium: ESSiteSearchAPI.java:49 — the import java.util.stream.Collectors; is now unused. This PR removed its only call site (listIndices() changed .collect(Collectors.toList()) → .toList() at what was line 114), leaving the import dangling. Harmless at runtime but a leftover; per the repo's progressive-enhancement rule it should be dropped. Fix this →

• 🟡 Medium: SiteSearchAPIImpl.java:355-380 (deleteIndex) — behavioral change worth a conscious call. The old router.writeChecked(...) fanned the delete to all write providers and always invoked the shadow even when the primary threw (primary exception recorded, shadow still called, then re-thrown). The new sequential form deletes the primary first and, on failure, throws before the secondary.deleteIndex(...) sweep runs. So a transient primary-delete failure in a dual-write phase now leaves the secondary twin untouched, whereas before it would have been deleted.

  This is arguably more consistent (a failed delete leaves the index present on both engines rather than half-deleted), and the exception surfaces so nothing is silently orphaned — but it's a semantic shift from the prior fan-out. If the intent is "sweep the other engine regardless of primary outcome," the secondary sweep should sit in a finally or run before the re-throw. Assumption: leaf deleteIndex is idempotent (confirmed by the code comment), so a retry recovers. What to verify: whether callers reliably retry a failed deleteIndex, or whether best-effort secondary cleanup should happen even when the primary fails.

Notes (non-blocking)

• The unit test SiteSearchRouterReconciliationTest drives phases 0/1 only. The deleteIndex primary/secondary identity swap (primary == esImpl ? osImpl : esImpl) is only exercised with ES-as-primary; a Phase-2 case (OS primary, ES secondary) would pin the swap in both directions. The IT covers the orphan-sweep behavior, so this is a completeness nit, not a gap.
• writeMirrorsInSync's cross-engine count-parity soundness rests on the documented single-writer/quiescent-at-planning assumption and on ES vs OS reporting identical doc counts for an in-sync mirror. Both are reasonable and clearly documented in the interface javadoc and OPENSEARCH_MIGRATION.md.

[github-actions]

⚠️ AI review failed

Claude did not produce a review — the backend call errored before generating any output (provider: anthropic-bedrock, model: global.anthropic.claude-sonnet-5). This usually means the model has no Bedrock access grant in the target account, or the model ID is invalid — not a problem with this PR.

_{Run: #30587154434}

[fabrizzio-dotCMS]

Addressed the review feedback in a9b8501:

• 🟠 High — count-parity blind above 10k. Confirmed: neither leaf's search() set track_total_hits, so getTotalResults() capped at 10,000 and drift on large mirrors (e.g. ES 15k vs OS 12k → both 10000) slipped through. Added SiteSearchAPI.documentCount(name) — an exact count (dedicated _count on ES, size:0 match-all with track_total_hits:true on OS) — and the parity loop now uses it instead of a search total.
• 🟡 Medium — transient double-failure read as "in sync". documentCount returns -1 on a failed count and writeMirrorsInSync treats -1 as out of sync, so a both-engines-failed 0 == 0 can no longer permit an incremental. Fail-safe is now rebuild.
• 🟡 Medium — duplicated 5-term predicate. Extracted into a single incrementalEligible local in SiteSearchJobImpl, reused by both the decision and the log guard.

Interface javadoc and OPENSEARCH_MIGRATION.md qualified accordingly. SiteSearchRouterReconciliationTest now mocks documentCount and adds drift-above-10k + failed-count cases — 13/13 green.