feat(migration): internal migration-readiness endpoint (content + site search) (#36360)

docs/backend/OPENSEARCH_MIGRATION.md

@@ -353,10 +353,44 @@ closed when a crawl actually runs in a dual-write phase (1 or 2). In the window
   twin that was never rebuilt, and never crawling, is a hard gap.
 
 **Operational rule (pairs with the code):** before promoting the phase — especially into Phase 3 —
-ensure every Site Search index has been crawled at least once so its OS twin exists and is in sync. A
-targeted verify/repair job (detect twins that are missing **or** whose counts diverge → rebuild full)
-that closes the no-crawl window without depending on a crawl is deferred as a follow-up under the same
-issue.
+ensure every Site Search index has been crawled at least once so its OS counterpart exists and is in
+sync. The migration-readiness endpoint below is what tells the operator *which* indices still need
+that crawl, before they change the phase.
+
+#### Migration-readiness endpoint (pre-phase-change advisory)
+
+`GET /api/v1/index/migration/readiness` is an internal, read-only report a support technician runs
+**before changing the migration phase** to see whether it is safe and, if not, what to do. It never
+mutates anything — the fix is always the operator re-running the crawl / reindex, which self-heals
+through the write-path gate above.
+
+- **Not public.** The resource is `@Hidden` (absent from the OpenAPI / API-playground schema) and
+  gated to CMS administrators or members of the migration support role
+  (`OS_MIGRATION_INDEX_VISIBILITY_ROLE_KEY`, default `os_migration_qa`); anyone else gets a 403.
+- **What it reports.** The current phase with its read/write engines and an `evaluable` flag; an
+  overall verdict — `safeToAdvance` (toward OpenSearch-only) and `safeToRollback` (downgrade) with an
+  `outOfSyncCount`, a human `summary`, and per-index `blockers`; and the per-index ES↔OS mirror diff
+  for **both** mirrored families — the versioned content indices (`working`/`live`) and the Site
+  Search indices. `content` is a keyed object by slot (`WORKING` / `LIVE` — a fixed pair);
+  `siteSearch` is a list (an open set). Each entry carries `{indexName, es:{exists,docCount,physicalName},
+  os:{exists,docCount,physicalName}, verdict, recommendation}` — `physicalName` is the full name as
+  stored on each server (cluster-prefixed; `.os`-tagged on OpenSearch) — with verdict `IN_SYNC` /
+  `MISSING_COUNTERPART` / `COUNT_DRIFT`. The top level also carries the `clusterId` embedded in every
+  physical name. The response is the model itself (no `ResponseEntityView` envelope).
+- **Stateless, from live counts.** Every field is derived at request time. Counts are **exact** — the
+  Site Search half uses `SiteSearchAPI.documentCount` and the content half reads each engine leaf's
+  `getIndicesStats()` (index `_stats` `primaries.docs.count`), never a search total (which the ES/OS
+  clients cap at 10,000 and would hide drift on large indices). Both reconcilers query the two engine
+  leaves directly, not the phase-aware router, so the report shows both sides in every phase.
+- **`safeToRollback` needs no history.** A downgrade routes reads back to Elasticsearch, so it is
+  unsafe when any index's ES copy is behind its OpenSearch counterpart (`esDocCount < osDocCount`, or
+  the ES copy missing) — that delta, typically content written while OpenSearch served reads, would be
+  silently absent after the downgrade until a full reindex. That is derivable from the same snapshot,
+  so no per-phase state is persisted.
+
+Because this endpoint is the source of truth for migration/QA, the index portlets no longer reveal
+`.os` indices by role: `MigrationIndexVisibility` is now purely phase-based (hidden in Phases 0/1/2,
+shown in Phase 3, for everyone). The role key is retained only to gate this endpoint.
 
 #### Tag manipulation is the sole responsibility of `IndexTag`
 

dotCMS/src/main/java/com/dotcms/content/index/MigrationIndexVisibility.java

@@ -1,12 +1,6 @@
 package com.dotcms.content.index;
 
 import com.dotcms.content.index.IndexConfigHelper.MigrationPhase;
-import com.dotmarketing.business.APILocator;
-import com.dotmarketing.business.Role;
-import com.dotmarketing.util.Config;
-import com.dotmarketing.util.UtilMethods;
-import com.liferay.portal.model.User;
-import io.vavr.control.Try;
 import java.util.List;
 import java.util.stream.Collectors;
 
@@ -21,79 +15,64 @@
  * (optimize-all, flush-all, {@code indexExists} validation, bulk fix). Filtering inside those
  * methods would silently skip OS indices for those operations in phases 1/2 — a behavioural
  * change disguised as a UI tweak. The complete, phase-correct set must stay intact at the API;
- * only the two display sinks (the maintenance JSP and {@code IndexResourceHelper.indexStatsList})
+ * only the display sinks (the maintenance JSP and {@code IndexResourceHelper.indexStatsList})
  * apply this filter, and only those — see {@code docs/backend/OPENSEARCH_MIGRATION.md}.</p>
  *
- * <h2>Rule</h2>
+ * <h2>Rule — phase-based, for everyone</h2>
  * <ul>
  *   <li>Phase&nbsp;3 (OS-only): OS is the live store, so {@code .os} indices are always visible.</li>
- *   <li>Phases&nbsp;0/1/2: {@code .os} indices are a migration/uniqueness artifact and are hidden,
- *       <em>unless</em> the acting user holds the configured QA/preview role
- *       ({@value #VISIBILITY_ROLE_KEY}, default {@value #DEFAULT_VISIBILITY_ROLE_KEY}).</li>
+ *   <li>Phases&nbsp;0/1/2: {@code .os} indices are a migration/uniqueness artifact and are hidden
+ *       from every user — regular admins never learn a migration is running.</li>
  * </ul>
  *
- * <p>The acting {@link User} is supplied explicitly by each display sink (both are authenticated
- * admin requests where the user is always available) — never resolved from a thread-local inside
- * this policy, so it is safe to unit-test and free of request-context coupling.</p>
+ * <p>The role-gated preview of {@code .os} indices was removed in issue #36360: support and QA now
+ * get migration detail from the dedicated, role-gated migration-readiness endpoint
+ * ({@code /api/v1/index/migration/readiness}), which is the single source of truth. This display
+ * policy is therefore purely phase-based and consults no user or role.</p>
  *
  * <p>OS-origin detection always goes through {@link IndexTag#isTagged(String)}, never
  * {@code name.endsWith(".os")}, per the {@link IndexTag} contract.</p>
  */
 public final class MigrationIndexVisibility {
 
     /**
-     * Config key holding the {@link Role#getRoleKey() role key} whose members may preview
-     * OS-tagged ({@code .os}) indices before Phase&nbsp;3. Defaults to
-     * {@value #DEFAULT_VISIBILITY_ROLE_KEY}.
+     * Config key holding the {@link com.dotmarketing.business.Role#getRoleKey() role key} whose
+     * members may read the role-gated <em>migration-readiness endpoint</em>
+     * ({@code /api/v1/index/migration/readiness}). Defaults to {@value #DEFAULT_VISIBILITY_ROLE_KEY}.
+     * It no longer governs the index portlet display (which is purely phase-based since issue #36360).
      */
     public static final String VISIBILITY_ROLE_KEY = "OS_MIGRATION_INDEX_VISIBILITY_ROLE_KEY";
 
-    /** Default role key allowed to preview migration ({@code .os}) indices. */
+    /** Default role key allowed to read the migration-readiness endpoint. */
     public static final String DEFAULT_VISIBILITY_ROLE_KEY = "os_migration_qa";
 
     private MigrationIndexVisibility() {
         throw new AssertionError("Utility class — do not instantiate");
     }
 
     /**
-     * Whether {@code user} may see OS-tagged ({@code .os}) indices in the current phase.
-     *
-     * @param user the acting user; {@code null} is treated as "not allowed" outside Phase&nbsp;3
-     * @return {@code true} in Phase&nbsp;3, or when {@code user} holds the configured QA role
+     * Whether OS-tagged ({@code .os}) indices are shown in the current phase — only in Phase&nbsp;3,
+     * where OpenSearch is the live store. Before Phase&nbsp;3 they are a migration artifact and stay
+     * hidden from everyone.
      */
-    public static boolean canSeeMigrationIndices(final User user) {
-        if (MigrationPhase.current().isMigrationComplete()) {
-            return true;
-        }
-        if (user == null) {
-            return false;
-        }
-        final String roleKey = Config.getStringProperty(VISIBILITY_ROLE_KEY,
-                DEFAULT_VISIBILITY_ROLE_KEY);
-        if (!UtilMethods.isSet(roleKey)) {
-            return false;
-        }
-        return Try.of(() -> {
-            final Role role = APILocator.getRoleAPI().loadRoleByKey(roleKey);
-            return role != null && APILocator.getRoleAPI().doesUserHaveRole(user, role);
-        }).getOrElse(false);
+    public static boolean showMigrationIndices() {
+        return MigrationPhase.current().isMigrationComplete();
     }
 
     /**
-     * Returns {@code indexNames} with OS-tagged ({@code .os}) entries removed when {@code user}
-     * is not allowed to see them; otherwise returns the list unchanged.
+     * Returns {@code indexNames} with OS-tagged ({@code .os}) entries removed outside Phase&nbsp;3;
+     * in Phase&nbsp;3 (or for a null/empty input) the list is returned unchanged.
      *
-     * @param indexNames the full, phase-correct list of index names; {@code null}/empty is
-     *                   returned as-is
-     * @param user       the acting user
+     * @param indexNames the full, phase-correct list of index names; {@code null}/empty is returned
+     *                   as-is
      * @return a filtered copy, or the original list when no filtering applies
      */
-    public static List<String> filter(final List<String> indexNames, final User user) {
-        if (indexNames == null || indexNames.isEmpty() || canSeeMigrationIndices(user)) {
+    public static List<String> filter(final List<String> indexNames) {
+        if (indexNames == null || indexNames.isEmpty() || showMigrationIndices()) {
             return indexNames;
         }
         return indexNames.stream()
                 .filter(name -> !IndexTag.OS.isTagged(name))
                 .collect(Collectors.toList());
     }
-}
+}

dotCMS/src/main/java/com/dotcms/content/index/migration/ContentIndexMirrorReconciler.java

@@ -0,0 +1,122 @@
+package com.dotcms.content.index.migration;
+
+import com.dotcms.cdi.CDIUtils;
+import com.dotcms.content.elasticsearch.business.ESIndexAPI;
+import com.dotcms.content.elasticsearch.business.IndiciesInfo;
+import com.dotcms.content.index.IndexAPI;
+import com.dotcms.content.index.IndexTag;
+import com.dotcms.content.index.domain.IndexStats;
+import com.dotcms.content.index.migration.MirrorStatus.IndexKind;
+import com.dotcms.content.index.migration.MirrorStatus.Verdict;
+import com.dotcms.content.index.opensearch.OSIndexAPIImpl;
+import com.dotmarketing.business.APILocator;
+import com.dotmarketing.util.Logger;
+import com.dotmarketing.util.UtilMethods;
+import com.google.common.annotations.VisibleForTesting;
+import io.vavr.control.Try;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+
+/**
+ * Content-index half of the migration-readiness report (issue #36360): compares the active versioned
+ * content indices (working and live) against their {@code .os} counterparts across both engines,
+ * mirroring {@link SiteSearchMirrorReconciler} but for the content store. Never mutates anything.
+ *
+ * <h4>How the counts are read (phase-independently)</h4>
+ * <p>{@code IndiciesInfo} always holds the cluster-prefixed, <em>un-tagged</em> Elasticsearch name for
+ * working/live (its backing {@code indicies} table owns only the ES rows — {@code index_version IS
+ * NULL}); the OpenSearch counterpart is that name with the {@code .os} tag. Exact per-engine document
+ * counts come from each engine leaf's {@code getIndicesStats()} — the index {@code _stats}
+ * {@code primaries.docs.count}, an exact total not subject to the 10,000 search hit-count cap. Those
+ * stats maps are keyed by the <em>cluster-stripped</em> name (Elasticsearch un-tagged, OpenSearch
+ * carrying {@code .os}), so each raw name is stripped of the cluster prefix and then, for the
+ * OpenSearch lookup, tagged — the same strip-then-tag order the maintenance JSP uses.</p>
+ *
+ * <p>It queries the two engine leaves directly (never the phase-aware router) so the report shows both
+ * sides regardless of which engine the current phase reads from. Scope is the active working/live
+ * pair; reindex slots are out of scope for this report.</p>
+ */
+public class ContentIndexMirrorReconciler {
+
+    private final IndexAPI esImpl;
+    private final IndexAPI osImpl;
+    private final Supplier<IndiciesInfo> indiciesSupplier;
+
+    public ContentIndexMirrorReconciler() {
+        this(new ESIndexAPI(), CDIUtils.getBeanThrows(OSIndexAPIImpl.class),
+                ContentIndexMirrorReconciler::loadIndiciesQuietly);
+    }
+
+    @VisibleForTesting
+    ContentIndexMirrorReconciler(final IndexAPI esImpl, final IndexAPI osImpl,
+            final Supplier<IndiciesInfo> indiciesSupplier) {
+        this.esImpl = esImpl;
+        this.osImpl = osImpl;
+        this.indiciesSupplier = indiciesSupplier;
+    }
+
+    /** Per-index mirror status for the active working and live content indices. */
+    public List<MirrorStatus> statuses() {
+        final IndiciesInfo info = indiciesSupplier.get();
+        if (info == null) {
+            return List.of();
+        }
+        final Map<String, IndexStats> esStats = esImpl.getIndicesStats();
+        final Map<String, IndexStats> osStats = osImpl.getIndicesStats();
+        final List<MirrorStatus> out = new ArrayList<>(2);
+        addStatus(out, IndexKind.CONTENT_WORKING, info.getWorking(), esStats, osStats);
+        addStatus(out, IndexKind.CONTENT_LIVE, info.getLive(), esStats, osStats);
+        return out;
+    }
+
+    private void addStatus(final List<MirrorStatus> out, final IndexKind kind, final String rawName,
+            final Map<String, IndexStats> esStats, final Map<String, IndexStats> osStats) {
+        if (!UtilMethods.isSet(rawName)) {
+            return;
+        }
+        // IndiciesInfo holds the cluster-prefixed, un-tagged ES name — which IS the full ES physical
+        // name; the OS physical name is that + .os. The stats maps are keyed by the cluster-stripped
+        // name (ES un-tagged, OS carrying .os), so strip for the count lookup, tag for the OS key.
+        final String esPhysical = rawName;
+        final String osPhysical = IndexTag.OS.tag(rawName);
+        final String bare = esImpl.removeClusterIdFromName(rawName);
+        final String osKey = IndexTag.OS.tag(bare);
+
+        final boolean esExists = esStats.containsKey(bare);
+        final long esCount = esExists ? esStats.get(bare).documentCount() : 0L;
+        final boolean osExists = osStats.containsKey(osKey);
+        final long osCount = osExists ? osStats.get(osKey).documentCount() : 0L;
+
+        final Verdict verdict = MirrorStatus.verdictFor(esExists, osExists, esCount, osCount);
+        out.add(new MirrorStatus(bare, kind,
+                new MirrorStatus.EngineCopy(esExists, esCount, esPhysical),
+                new MirrorStatus.EngineCopy(osExists, osCount, osPhysical),
+                verdict, recommend(bare, verdict, osExists)));
+    }
+
+    private static String recommend(final String name, final Verdict verdict, final boolean osExists) {
+        switch (verdict) {
+            case IN_SYNC:
+                return "In sync — no action needed.";
+            case MISSING_COUNTERPART:
+                final String missing = osExists ? "Elasticsearch" : "OpenSearch";
+                return String.format("The %s copy of content index '%s' is missing. Run a full "
+                        + "reindex to rebuild it before promoting to the OpenSearch-only phase.",
+                        missing, name);
+            case COUNT_DRIFT:
+            default:
+                return String.format("The two copies of content index '%s' hold a different number "
+                        + "of documents. Run a full reindex to rebuild the OpenSearch copy before "
+                        + "promoting the phase.", name);
+        }
+    }
+
+    private static IndiciesInfo loadIndiciesQuietly() {
+        return Try.of(() -> APILocator.getIndiciesAPI().loadIndicies())
+                .onFailure(e -> Logger.warn(ContentIndexMirrorReconciler.class,
+                        "Could not load content indices for migration readiness: " + e.getMessage()))
+                .getOrNull();
+    }
+}

dotCMS/src/main/java/com/dotcms/content/index/migration/MigrationReadiness.java

@@ -0,0 +1,60 @@
+package com.dotcms.content.index.migration;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Support-facing ES→OS migration-readiness report (issue #36360): a point-in-time snapshot that a
+ * role-gated support technician reads <em>before</em> changing the migration phase, to see whether it
+ * is safe and, if not, what to do. Read-only and stateless — every field is derived from live index
+ * state at request time, nothing is persisted.
+ *
+ * @param clusterId         the dotCMS cluster id embedded in every physical index name
+ *                          (the {@code <id>} of the {@code cluster_<id>.} prefix); identical for the
+ *                          Elasticsearch and OpenSearch backends
+ * @param phase      the current migration phase and which engine it reads/writes
+ * @param content    the versioned content indices keyed by slot ({@code WORKING} / {@code LIVE}) — a
+ *                   fixed pair, so a keyed object reads naturally
+ * @param siteSearch the Site Search indices as a list — an open set with no natural key, so a list
+ *                   (each entry carries its own {@code indexName})
+ * @param verdict    the overall go/no-go for advancing and rolling back, with reasons
+ */
+public record MigrationReadiness(
+        String clusterId,
+        PhaseInfo phase,
+        Map<String, MirrorStatus> content,
+        List<MirrorStatus> siteSearch,
+        Verdict verdict) {
+
+    /**
+     * @param current      the current phase ordinal (0–3)
+     * @param name         the phase enum name (e.g. {@code PHASE_2_DUAL_WRITE_OS_READS})
+     * @param readEngine   which engine currently serves reads ("Elasticsearch" or "OpenSearch")
+     * @param writeEngines which engines currently receive writes
+     * @param evaluable    whether a cross-engine comparison is meaningful for a forward phase change
+     *                     (only the dual-write phases 1/2); when false the mirror lists are advisory
+     *                     context, not a forward go/no-go
+     */
+    public record PhaseInfo(
+            int current,
+            String name,
+            String readEngine,
+            List<String> writeEngines,
+            boolean evaluable) {}
+
+    /**
+     * @param safeToAdvance  whether it is safe to promote toward the OpenSearch-only phase
+     * @param safeToRollback whether it is safe to downgrade — false when any index's Elasticsearch
+     *                       copy is behind its OpenSearch counterpart, because a downgrade routes reads back
+     *                       to Elasticsearch and would silently drop that delta until a reindex
+     * @param outOfSyncCount how many indices need attention (missing counterpart or count drift)
+     * @param summary        one human-readable sentence describing the overall state
+     * @param blockers       per-index reasons that make advancing unsafe (empty when safe)
+     */
+    public record Verdict(
+            boolean safeToAdvance,
+            boolean safeToRollback,
+            int outOfSyncCount,
+            String summary,
+            List<String> blockers) {}
+}