feat(migration): internal migration-readiness endpoint (content + site search) (#36360)

[fabrizzio-dotCMS]

Problem

Support/QA had no single place to answer "is it safe to change the OpenSearch migration phase right now, and if not, what do I fix?" before promoting. QA Round 2 on #36360 showed the failure mode: promoting a phase while an index's ES and OpenSearch copies were out of sync led to silent empty/partial results, and the downgrade case (Phase-3-only content invisible after a rollback) was undocumented.

This adds an internal, read-only migration-readiness endpoint that condenses that status with actionable recommendations, and retires the role-gated .os reveal in the index portlets in favor of it.

What this adds

GET /api/v1/index/migration/readiness — internal, read-only, advisory. Never mutates anything; the fix is always the operator re-running the crawl/reindex, which self-heals through the existing write-path gates (#36797 read half, #36825 write half).

• Not public. @Hidden (absent from the OpenAPI / API-playground schema) and gated to CMS administrators or members of the migration support role (OS_MIGRATION_INDEX_VISIBILITY_ROLE_KEY, default os_migration_qa); anyone else gets a 403.
• Covers both mirrored families — versioned content indices (working/live) and Site Search indices. content is a keyed object (WORKING / LIVE — a fixed pair); siteSearch is a list (an open set). Each entry is a per-index ES↔OS diff: {indexName, es:{exists,docCount,physicalName}, os:{exists,docCount,physicalName}, verdict, recommendation}, verdict IN_SYNC / MISSING_COUNTERPART / COUNT_DRIFT. physicalName is the full name as stored on each server (cluster-prefixed; .os-tagged on OpenSearch), and the report carries the top-level clusterId.
• Overall verdict: current phase + read/write engines + evaluable; safeToAdvance (toward OpenSearch-only) and safeToRollback (downgrade) with outOfSyncCount, a human summary, and per-index blockers. Field order: clusterId, phase, content, siteSearch, verdict.
• Bare model response — returns the readiness object directly (no ResponseEntityView envelope), serialized by Jackson from typed records.
• Stateless, exact counts. Every field derived at request time. The Site Search half uses SiteSearchAPI.documentCount; the content half reads each engine leaf's getIndicesStats() (index _stats primaries.docs.count) — never a search total (capped at 10,000, which would hide drift on large indices). Both reconcilers query the two engine leaves directly (not the phase-aware router), so both sides show in every phase.
• safeToRollback needs no history: a downgrade routes reads back to Elasticsearch, so it is unsafe when any index's ES copy is behind its OpenSearch counterpart — derivable from the same snapshot, nothing persisted.

Teardown — index portlet visibility is now phase-only (reverts I-4). MigrationIndexVisibility no longer reveals .os by role: .os indices are hidden in Phases 0/1/2 and shown in Phase 3, for everyone. The role key is retained only to gate this endpoint, which is now the single source of truth for migration/QA.

Layers

• com.dotcms.content.index.migration: MirrorStatus (shared per-index diff), SiteSearchMirrorReconciler, ContentIndexMirrorReconciler, MigrationReadinessService (composition + verdict), MigrationReadiness (report DTO). Pure, no mutation.
• com.dotcms.rest.api.v1.index.MigrationReadinessResource: the JAX-RS resource + role gate.

Tests — 23 unit green

• MigrationReadinessServiceTest (8): advance/rollback verdicts across phases 0–3, missing-counterpart and >10k count-drift blocking advance, content-keyed-by-slot / siteSearch-as-list, clusterId.
• ContentIndexMirrorReconcilerTest (5): in-sync, missing OS counterpart, count drift, null/absent slots, physical names.
• MigrationReadinessResourceTest (5): role gate — admin/role-member allowed, non-admin-without-role denied, null user denied, access-lookup failure fails closed.
• MigrationIndexVisibilityTest (5): rewritten to the phase-only contract.

Remaining test (flagging): a container integration test that drives HTTP 403/200 through the JAX-RS stack is not included yet — the gate decision and both reconcilers are unit-covered. Happy to add it if you'd like it before merge.

Notes

• Stacked on fix(sitesearch): reconcile ES/OS index mirrors on the write path (#36360) #36825 (issue-36360-sitesearch-mirror-reconciliation) because it consumes SiteSearchAPI.documentCount from that PR. Based against that branch so the diff shows only the readiness changes; will retarget to main once fix(sitesearch): reconcile ES/OS index mirrors on the write path (#36360) #36825 merges.
• Docs: new "Migration-readiness endpoint" subsection in docs/backend/OPENSEARCH_MIGRATION.md.

🤖 Generated with Claude Code

[claude]

Claude finished @fabrizzio-dotCMS's task in 16m 18s —— View job

---

Review — migration-readiness endpoint (#36360)

• Read the diff vs base branch
• Review new reconcilers + service + resource
• Review visibility teardown + JSP/helper changes
• Post review findings

Reviewed against origin/issue-36360-sitesearch-mirror-reconciliation. The role gate, phase-only visibility teardown, and callers are all consistent; the OS-key strip/tag logic in the content reconciler matches OSIndexAPIImpl.getIndicesStats()'s key format (verified against the impl). No critical/high issues — three non-blocking observations below.

New Issues

• 🟡 Medium: dotCMS/src/main/java/com/dotcms/content/index/migration/ContentIndexMirrorReconciler.java:72 — esImpl.getIndicesStats() is called unguarded, and ESIndexAPI.getIndicesStats() (ESIndexAPI.java:133) is not wrapped in try/catch — performLowLevelRequest throws unchecked when the ES cluster is unreachable. The failure then propagates out of statuses() → evaluate() → the resource, so the endpoint returns 500 instead of a degraded report. This is asymmetric: the OS side (OSIndexAPIImpl.getIndicesStats(), line 475) catches and returns an empty map, and the Site Search half's documentCount returns -1 on failure. For a diagnostic endpoint whose whole purpose is to be usable when things are broken, an unreachable ES cluster is exactly the moment an operator would hit this — and it hard-fails rather than showing "ES copy unavailable." Consider guarding the ES stats call the same way (log + empty map → surfaces as MISSING_COUNTERPART, fail-safe). Fix this →

• 🟡 Medium: dotCMS/src/main/java/com/dotcms/content/index/migration/MigrationReadinessService.java:262 — In the non-dual-write phases (0 and 3), outOfSyncCount is still outOfSync.size(), computed across the raw mirror rows. In Phase 0 the OS counterparts don't exist yet, so every content and Site Search row is MISSING_COUNTERPART and outOfSyncCount reports e.g. 2 (working+live) while safeToAdvance=true and the summary says "nothing to reconcile yet." The payload contradicts itself for an operator reading the JSON (count > 0, empty blockers, "safe"). Consider zeroing outOfSyncCount/scoping it to the evaluable phases, or wording the count so it reads as advisory in phases 0/3.

• 🟡 Medium: dotCMS/src/main/java/com/dotcms/content/index/migration/SiteSearchMirrorReconciler.java:55 — canEvaluate() is a public method that no one calls. The service derives evaluable from phase.isDualWrite() directly (MigrationReadinessService.java:301), not from this method. Dead code — remove it, or wire the service through it so there's a single definition of "evaluable." Fix this →

Everything else checks out: the MigrationIndexVisibility filter(...)/showMigrationIndices() signature change has all callers updated (ESIndexResource, IndexResourceHelper, index_stats.jsp, tests); the role gate fails closed on lookup failure and is unit-covered; -1 failed-count values sort correctly through verdictFor and the esBehindAnywhere rollback check (fail-safe); @Hidden correctly keeps it out of the OpenAPI schema so no @Schema is required. The flagged missing container-level 403/200 integration test is a reasonable follow-up given the gate and reconcilers are unit-covered.
• issue-36360-migration-readiness-endpoint