From 893870ef24e3322a2a7b5024349367625d6d8ba6 Mon Sep 17 00:00:00 2001 From: Florian Schade Date: Wed, 29 Jul 2026 14:48:43 +0200 Subject: [PATCH 1/4] docs(adr): non-breaking search index changes --- .../0005-non-breaking-search-index-changes.md | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 docs/adr/0005-non-breaking-search-index-changes.md diff --git a/docs/adr/0005-non-breaking-search-index-changes.md b/docs/adr/0005-non-breaking-search-index-changes.md new file mode 100644 index 0000000000..edd8cadbe6 --- /dev/null +++ b/docs/adr/0005-non-breaking-search-index-changes.md @@ -0,0 +1,106 @@ +--- +title: "Non-Breaking Search Index Changes" +--- + +* Status: proposed +* Deciders: [@fschade, @rhafer, @aduffeck, @butonic] +* Date: 2026-07-29 + +Reference: https://github.com/opencloud-eu/opencloud/pull/2659#issuecomment-5103043714 + +## Context and Problem Statement + +The search service runs on one of two engines, bleve or OpenSearch. Both store a mapping +that describes the shape of an indexed document. Whenever we change that shape in an +incompatible way, for example a changed field type or a different analyzer, the index +that is already on disk or in the cluster no longer matches what the code expects. + +Today this stops the instance. The service detects the mismatch on startup, returns an +error and refuses to come up until the operator drops the index and reindexes. Two +consequences follow from that: + +* an upgrade can take an instance down, and it stays down until the operator reacts, +* the change is breaking, so we can only ship it with a major release. + +We already hit this once and solved it by waiting for the next major release. That is +not a process, it just moves the problem. + +## Decision Drivers + +* an upgrade must never keep an instance from starting +* index changes should be shippable in a minor release +* reindexing is expensive, it costs CPU and can run for hours on large data sets, so the + operator has to decide when it happens +* users should not silently lose search results and favorites without knowing why + +## Considered Options + +1. **Keep failing on startup.** Ship index changes only in major releases. Safe in the + sense that nothing is silently empty, but the instance is down and we are blocked by + the release cadence. +2. **Migrate automatically on startup.** The service reindexes itself into the new + mapping before it serves requests. Fully automated, but it blocks the start for an + unbounded time and it takes the timing away from the operator. +3. **Create a new index, keep the old one.** The service creates a fresh index for the + new mapping and starts using it immediately. Reindexing is a separate step, triggered + by the operator. + +## Decision Outcome + +We go with option 3. + +On a breaking index change the search service creates a new index instead of touching +the existing one. For OpenSearch that is a new index name, for bleve a new directory. +The old index is left alone, it is neither read nor deleted, so a rollback to the +previous release still finds its data. The instance starts, everything else keeps +working. + +The price is that the new index starts out empty. Search returns no results and +favorites look empty. From there it fills up again on its own, every upload, move or +metadata change is indexed as it happens, but that only covers what is touched after +the upgrade. Everything older stays invisible until the operator reindexes. Nothing is +lost, the storage is the source of truth, but it is visible to every user. So the manual +step needs to be communicated in two directions: + +* **to the operator**, through the release notes. Every release that changes the index + says so and points to the reindex command. +* **to the users**, through the web UI. The operator can publish a sticky message that + announces the reindex and explains why search and favorites are incomplete during + that time. The operator also decides when to run it, for example outside of working + hours. + +Release notes alone are not enough here. Plenty of instances are upgraded by pulling a +new image tag, and nobody reads a changelog for that. So the service has to say it +itself: when it creates a new index it logs that a reindex is pending, names the command, +and keeps that state readable from the outside, so it also shows up in monitoring. If we +skip this, the realistic path is that the operator learns about it from users who miss +their favorites. + +We deliberately do not ship this batteries included. A migration that runs by itself is +a black box for us, we cannot look into the instance it runs on and we cannot promise it +goes through cleanly. Data volume, cluster sizing and what load is acceptable are things +the operator knows and we do not. So the operator triggers the reindex, watches it, and +can stop and repeat it. We would rather have a manual step that is understood than an +automatic one that fails halfway. + +Old indexes stay around and cost storage. Removing them is the operator's call as well, +we only document how. + +To be clear about what this decision does and does not buy us: the upgrade is no longer +breaking, the instance comes up on its own. But it is still a manual step, and without +the release note and the sticky message it looks like data loss to the user. + +### Implementation Steps + +1. Derive the index name (OpenSearch) and the index path (bleve) from the mapping + version, so a breaking mapping change automatically means a new, empty index. +2. Drop the startup failure path, the mismatch now leads to a new index instead of an + error. +3. Log the pending reindex on startup and expose it as state that can be monitored. +4. Make the reindex fit for a long run. It has to report progress, survive longer than a + fixed timeout, and a single failing space must not kill the whole run. +5. Add the sticky message to the web UI, publishable by the operator. +6. Document the reindex, `opencloud search index --all-spaces`, and the cleanup of old + indexes in the admin docs. +7. Add the manual step to the release notes of every release that bumps the index + version. From a7ecec631c98e5db7e4f37a1069ff6cf034b845b Mon Sep 17 00:00:00 2001 From: Florian Schade Date: Wed, 29 Jul 2026 15:40:38 +0200 Subject: [PATCH 2/4] docs(adr): keep index mismatch detection, drop completion tracking --- .../0005-non-breaking-search-index-changes.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/adr/0005-non-breaking-search-index-changes.md b/docs/adr/0005-non-breaking-search-index-changes.md index edd8cadbe6..b22cc03870 100644 --- a/docs/adr/0005-non-breaking-search-index-changes.md +++ b/docs/adr/0005-non-breaking-search-index-changes.md @@ -71,10 +71,12 @@ step needs to be communicated in two directions: Release notes alone are not enough here. Plenty of instances are upgraded by pulling a new image tag, and nobody reads a changelog for that. So the service has to say it -itself: when it creates a new index it logs that a reindex is pending, names the command, -and keeps that state readable from the outside, so it also shows up in monitoring. If we -skip this, the realistic path is that the operator learns about it from users who miss -their favorites. +itself: when it creates a new index it logs that a reindex is pending and names the +command. If we skip this, the realistic path is that the operator learns about it from +users who miss their favorites. + +This is only about the moment the index is created. Whether a reindex has finished is +not something we track, and the reindex is the operator's job anyway. We deliberately do not ship this batteries included. A migration that runs by itself is a black box for us, we cannot look into the instance it runs on and we cannot promise it @@ -94,12 +96,15 @@ the release note and the sticky message it looks like data loss to the user. 1. Derive the index name (OpenSearch) and the index path (bleve) from the mapping version, so a breaking mapping change automatically means a new, empty index. -2. Drop the startup failure path, the mismatch now leads to a new index instead of an - error. -3. Log the pending reindex on startup and expose it as state that can be monitored. +2. Keep the mismatch detection as it is and only change what an operator gets out of it: + a new index instead of a service that refuses to start. The error path stays, as the + fallback when the new index cannot be created and as a strict mode for development, + where failing loudly on an unversioned mapping change is what we want. +3. Log that a reindex is pending when a new index is created, and name the command. 4. Make the reindex fit for a long run. It has to report progress, survive longer than a fixed timeout, and a single failing space must not kill the whole run. -5. Add the sticky message to the web UI, publishable by the operator. +5. Add the sticky message to the web UI, publishable by the operator. In progress in + https://github.com/opencloud-eu/opencloud/pull/3189. 6. Document the reindex, `opencloud search index --all-spaces`, and the cleanup of old indexes in the admin docs. 7. Add the manual step to the release notes of every release that bumps the index From 0dc20f5002faccf49f610a9fd8895d9a17af2920 Mon Sep 17 00:00:00 2001 From: Florian Schade Date: Wed, 29 Jul 2026 16:21:23 +0200 Subject: [PATCH 3/4] docs(adr): make the configured opensearch index name a prefix --- docs/adr/0005-non-breaking-search-index-changes.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/adr/0005-non-breaking-search-index-changes.md b/docs/adr/0005-non-breaking-search-index-changes.md index b22cc03870..efdaa54bbc 100644 --- a/docs/adr/0005-non-breaking-search-index-changes.md +++ b/docs/adr/0005-non-breaking-search-index-changes.md @@ -81,9 +81,9 @@ not something we track, and the reindex is the operator's job anyway. We deliberately do not ship this batteries included. A migration that runs by itself is a black box for us, we cannot look into the instance it runs on and we cannot promise it goes through cleanly. Data volume, cluster sizing and what load is acceptable are things -the operator knows and we do not. So the operator triggers the reindex, watches it, and -can stop and repeat it. We would rather have a manual step that is understood than an -automatic one that fails halfway. +the operator knows and we do not. So the operator triggers the reindex and watches it. +We would rather have a manual step that is understood than an automatic one that fails +halfway. Old indexes stay around and cost storage. Removing them is the operator's call as well, we only document how. @@ -96,6 +96,9 @@ the release note and the sticky message it looks like data loss to the user. 1. Derive the index name (OpenSearch) and the index path (bleve) from the mapping version, so a breaking mapping change automatically means a new, empty index. + `SEARCH_ENGINE_OPEN_SEARCH_RESOURCE_INDEX_NAME` becomes the prefix of that name + instead of the full name. A configured index can then never be the wrong version, + there is only "this version doesn't exist yet", which is the normal path. 2. Keep the mismatch detection as it is and only change what an operator gets out of it: a new index instead of a service that refuses to start. The error path stays, as the fallback when the new index cannot be created and as a strict mode for development, From e15a888ada2287b943fefc846b6caa29a8e81266 Mon Sep 17 00:00:00 2001 From: Florian Schade Date: Wed, 29 Jul 2026 17:26:54 +0200 Subject: [PATCH 4/4] docs(adr): describe the bleve gap and the actual mismatch reaction --- .../0005-non-breaking-search-index-changes.md | 24 ++++++++++++------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/adr/0005-non-breaking-search-index-changes.md b/docs/adr/0005-non-breaking-search-index-changes.md index efdaa54bbc..78243d12dc 100644 --- a/docs/adr/0005-non-breaking-search-index-changes.md +++ b/docs/adr/0005-non-breaking-search-index-changes.md @@ -15,13 +15,17 @@ that describes the shape of an indexed document. Whenever we change that shape i incompatible way, for example a changed field type or a different analyzer, the index that is already on disk or in the cluster no longer matches what the code expects. -Today this stops the instance. The service detects the mismatch on startup, returns an -error and refuses to come up until the operator drops the index and reindexes. Two -consequences follow from that: +Today this stops the instance, but only on OpenSearch. There the service compares the +stored mapping on startup, returns an error and refuses to come up until the operator +drops the index and reindexes. Two consequences follow from that: * an upgrade can take an instance down, and it stays down until the operator reacts, * the change is breaking, so we can only ship it with a major release. +Bleve is the opposite and just as wrong. It has no such check at all, it opens the index +and keeps answering from the stored mapping without a word, so a mapping change there +goes unnoticed and quietly returns wrong results. + We already hit this once and solved it by waiting for the next major release. That is not a process, it just moves the problem. @@ -99,16 +103,18 @@ the release note and the sticky message it looks like data loss to the user. `SEARCH_ENGINE_OPEN_SEARCH_RESOURCE_INDEX_NAME` becomes the prefix of that name instead of the full name. A configured index can then never be the wrong version, there is only "this version doesn't exist yet", which is the normal path. -2. Keep the mismatch detection as it is and only change what an operator gets out of it: - a new index instead of a service that refuses to start. The error path stays, as the - fallback when the new index cannot be created and as a strict mode for development, - where failing loudly on an unversioned mapping change is what we want. +2. Detect a mismatch on both engines, bleve needs that built first. Change what an + operator gets out of it: log it and keep running instead of refusing to start. A + change that makes the index unusable gets its own version and therefore its own + index, so what still reaches this path is additive, and those fields stay empty until + the next reindex. The error stays as a strict mode for development, where failing + loudly on a mapping change without a version is what we want. 3. Log that a reindex is pending when a new index is created, and name the command. 4. Make the reindex fit for a long run. It has to report progress, survive longer than a fixed timeout, and a single failing space must not kill the whole run. 5. Add the sticky message to the web UI, publishable by the operator. In progress in https://github.com/opencloud-eu/opencloud/pull/3189. 6. Document the reindex, `opencloud search index --all-spaces`, and the cleanup of old - indexes in the admin docs. + indexes in the admin docs. Owned by the docs team. 7. Add the manual step to the release notes of every release that bumps the index - version. + version. Owned by the docs team.