feat(search): check the index schema on startup and refuse breaking changes

[dschmidt]

Superseded by #3197. Reopened from the upstream repo with identical content and target so it can serve as a native-stack parent.

---

Stacked on #2659. The base tmp/refactor-search-mapping is a temporary upstream snapshot of the #2659 branch so this PR shows only its own commits; it gets retargeted to main (and the temp branch deleted) once #2659 is merged. Both PRs need to ship in the same release. Refs #3092.

Two complementary mechanisms keep the index schema and the code in sync.

Versioned indices (from the base #2659). The index name/path carries the schema version from search.SchemaVersion (OpenSearch opencloud-resource-v3, bleve bleve-v3). An intended breaking schema change is made by bumping the constant: the service targets a fresh, empty index and leaves the old one in place. No migration, no index-to-index reindex; content lives only in the index, the files are the source of truth, so a rescan (opencloud search index --all-spaces) rebuilds everything into the new index.

Startup schema check (this PR). The version bump is manual, so this is the safety net for the schema changing without one. On startup both engines diff the existing (current-version) index schema against the schema generated from code, using one shared recursive classifier:

• equal: start normally.
• additive (new fields with no indexed data): applied in place, no version bump needed. OpenSearch: PUT _mapping with the full code properties. bleve: the code mapping is persisted into the index (SetInternal + reopen), so new fields are typed correctly right away and the next start classifies equal. A startup warning lists the new fields; documents indexed before the upgrade lack them until re-indexed.
• breaking (changed types/analyzers, removed or renamed fields, new fields that already contain data of unknown shape): refuse to start. Because the index is versioned, a released instance never reaches this: an intended break bumps SchemaVersion and gets a fresh index. It fires in development when the mapping is changed in a breaking way without bumping, so the error is developer-facing (bump search.SchemaVersion, or revert the change) and lists the diff that triggered it.

Why this shape:

• The classifier is the only oracle. PUT _mapping is just the apply mechanism: its merge semantics hide removals/renames and it acks in-place updatable param changes, so it must not judge.
• bleve keeps no schema trace for dynamically indexed fields, so the classifier additionally checks idx.Fields() (exact name + path prefix). A field that is explicitly mapped now but already holds dynamically indexed data of unknown shape is breaking rather than a source of silently wrong results, no search beats wrong search.
• No migration framework, no index-to-index reindex, no blue-green: versioning provides the fresh index, a rescan repopulates it.

Known and accepted, no code changes:

• Non-JSON 5xx proxy bodies lose their status code in the opensearch-go error parsing; irrelevant while every non-conflict error is plain fatal.
• Trashed items are missing from the index after a rebuild until they are restored, the rescan walks only the live tree (docs caveat for Search engine schema update documentation // migration #3092).
• The bleve datapath must not be shared between processes; a second opener now fails after 5s (bolt_timeout) instead of hanging forever.

Deliberate consequences of the classifier rules (not bugs):

• Additive upgrades are one-way within a schema version: once the new release widened the index schema (PUT _mapping / persisted bleve mapping), the previous release sees a stored-only field and refuses to start. Roll forward or rebuild; refuse-in-both-directions is what keeps mismatched schemas from ever serving queries.
• Adding a new custom analyzer classifies as breaking even when only a brand-new field uses it: OpenSearch cannot change analysis settings on an open index, and both engines must behave identically.
• Removing or renaming a field in code is breaking by design; tolerating removals would make renames undetectable (a rename is a removal plus an addition).

[codacy-production]

Not up to standards ⛔

🟢 Coverage 62.86% diff coverage · +0.16% coverage variation

Metric	Results
Coverage variation	✅ +0.16% coverage variation (-1.00%)
Diff coverage	✅ 62.86% diff coverage

View coverage diff in Codacy

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (1b697e4)	82737	19353	23.39%
Head commit (a1b0535)	82984 (+247)	19541 (+188)	23.55% (+0.16%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3098)	350	220	62.86%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[Copilot]

Pull request overview

This PR adds a shared, recursive schema-diff classifier and uses it at startup for both OpenSearch and bleve to (1) start normally when schemas match, (2) apply additive mapping changes in-place while warning about missing fields on pre-upgrade documents, and (3) refuse to start on breaking changes with an operator-facing rebuild procedure.

Changes:

• Introduces services/search/pkg/mapping.Classify + shared ErrManualActionRequired/error builder used by both engines.
• Updates OpenSearch startup to create-or-compare index schema, apply additive changes via PUT _mapping, and fail hard on breaking diffs.
• Updates bleve startup to compare stored vs code schema (including “data exists for previously-dynamic fields”), persist additive schema changes into the index, and fail hard on breaking diffs; adds targeted tests for both engines.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
services/search/pkg/opensearch/index.go	Create-or-compare index on startup; classify mapping diff; apply additive changes via PUT _mapping; return shared manual-action error on breaking diffs.
services/search/pkg/opensearch/index_test.go	Expands OpenSearch index-manager tests for idempotency, additive updates, breaking diffs, and transport-error behavior.
services/search/pkg/opensearch/backend.go	Plumbs context + logger through backend creation and schema application/health checks.
services/search/pkg/opensearch/backend_test.go	Updates OpenSearch backend tests to the new NewBackend(ctx, ..., logger) signature.
services/search/pkg/mapping/classify.go	New shared classifier and shared ErrManualActionRequired + operator-facing error message builder.
services/search/pkg/mapping/classify_test.go	Unit tests covering equal/additive/breaking classification cases (including data-aware callback).
services/search/pkg/mapping/bleve.go	Tightens comment to require registering all referenced analyzers (validated by IndexMapping.Validate).
services/search/pkg/command/server.go	Uses bleve classification to warn on additive schema changes; bounds OpenSearch startup checks with a timeout context.
services/search/pkg/bleve/index.go	New bleve open/create logic with schema classification, additive persistence via SetInternal("_mapping"), and breaking-change refusal.
services/search/pkg/bleve/index_test.go	New tests validating bleve’s equal/additive/breaking behaviors plus analyzer-registration validation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

services/search/pkg/opensearch/index.go:224

• propertiesMap claims to return an empty map on missing/empty input, but unmarshalling "null" into a map sets it to nil. Returning a nil map is easy to mishandle later and contradicts the function comment. Normalize empty/null to an empty map and handle unmarshal errors explicitly.

// propertiesMap parses a raw mappings.properties object; missing or empty
// input yields an empty map, which classifies as purely additive.
func propertiesMap(raw string) map[string]any {
	props := map[string]any{}
	_ = json.Unmarshal([]byte(raw), &props)

services/search/pkg/opensearch/index.go:218

• jsonEqual treats missing JSON (empty string from gjson.Result.Raw when a path doesn’t exist) as unequal because json.Unmarshal([]byte(""), …) fails. This can falsely classify settings as drifting (and refuse startup) when both sides are unset. Normalize empty inputs (and optionally null) before unmarshalling.

This issue also appears on line 220 of the same file.

func jsonEqual(a, b string) bool {
	var av, bv any
	if err := json.Unmarshal([]byte(a), &av); err != nil {
		return false
	}

[dschmidt]

Reopened from the upstream repo as #3197 (identical content and target). The head branch now lives in the upstream repo instead of my fork so follow-up PRs can natively stack on it. Continuing there.