Skip to content

Resolve Elasticsearch exact-match field paths from live index mapping#794

Open
eanzhao wants to merge 1 commit into
feature/routerfrom
fix/2026-05-22_es-exact-match-mapping-truth
Open

Resolve Elasticsearch exact-match field paths from live index mapping#794
eanzhao wants to merge 1 commit into
feature/routerfrom
fix/2026-05-22_es-exact-match-mapping-truth

Conversation

@eanzhao
Copy link
Copy Markdown
Contributor

@eanzhao eanzhao commented May 21, 2026

Problem

Issue #743 — the Lark bot has been unable to reply since 2026-05-18. Root cause confirmed against the code on feature/router:

  • PR Stabilize Elasticsearch projection index mappings #665 added AugmentMetadata, injecting {"type":"keyword"} into in-memory DocumentIndexMetadata for every _id/_key/_hash/… -suffixed string field.
  • BuildExactMatchFieldPathResolver consulted that augmented metadata and, for an augmented-keyword field, emitted the bare field path.
  • Any Elasticsearch index created before 2026-05-18 keeps ES's dynamic default for string fields — text + a .keyword multi-field — and EnsureIndexAsync never reconciles an existing index.
  • So the term query targeted the analyzed text field and returned 0 hits for UUID-shaped values — silently. The relay scope resolver could not resolve apiKeyId → scopeId; every callback returned 401.

Augmented metadata is the code's intent; the index's physical _mapping is the truth. The query path trusted intent.

Solution

QueryAsync now resolves keyword/text field paths from the target index's live _mapping (GET <index>/_mapping, via ElasticsearchIndexLifecycleManager.GetActualFieldMappingsAsync, cached per index for the store lifetime), instead of the code-side augmented metadata.

Reading _mapping is a schema read — no mapping mutation, reindex, document backfill, or event replay. Index creation and descriptor augmentation are unchanged. This implements issue #743 §5.4 standalone, and scopes the 2026-05-15 blueprint hard-constraint #2 to the write/index-init path — recorded in ADR-0025.

Out of scope (still tracked by #743)

Alias indirection, schema fingerprinting, blue-green reindex migration, and a real-ES Testcontainers CI suite (#743 phases P1–P3, P5) — none required to recover the outage or to make the query path drift-tolerant. #743 stays open for that lifecycle epic.

Impact paths

  • …/Stores/ElasticsearchProjectionDocumentStore.csQueryAsync builds the exact-match resolver per query from live mappings.
  • …/Stores/ElasticsearchIndexLifecycleManager.cs — new GetActualFieldMappingsAsync (cached GET _mapping probe).
  • …/Stores/ElasticsearchProjectionDocumentStoreMetadataSupport.cs — new _mapping-response parser.
  • Public API of ElasticsearchProjectionDocumentStore unchanged — GetAsync/UpsertAsync/DeleteAsync untouched; dynamic-scope read models still throw on query. No downstream changes.

Verification

  • dotnet test test/Aevatar.CQRS.Projection.Core.Tests136 passed, 1 skipped (env-gated ES integration test). Includes 3 new regression tests: augmented-keyword field physically text+.keyword → query targets .keyword; probe failure → falls back to declared metadata; repeated queries probe _mapping once.
  • bash tools/docs/lint.sh — passed (44 files, 0 errors).
  • query_projection_priming / projection_state_version / projection_state_mirror_current_state / projection_route_mapping / cqrs_eventsourcing_boundary / committed_state_projection / test_stability guards — all passed.

Docs

  • docs/adr/0025-elasticsearch-exact-match-resolution-reads-index-truth.md — new ADR.
  • docs/design/2026-05-15-…-blueprint.md — revision note linking ADR-0025.
  • src/Aevatar.CQRS.Projection.Providers.Elasticsearch/README.md — new "精确匹配字段路径解析" section.
  • docs/README.md — regenerated index.

🤖 Generated with Claude Code

PR #665's `_id`-suffix -> keyword augmentation made BuildExactMatchFieldPathResolver
trust code intent: for an augmented-keyword field it emitted the bare field path.
Elasticsearch indexes created before 2026-05-18 keep their dynamic `text` +
`.keyword` mapping, so `term` queries hit the analyzed `text` field and returned
0 hits for identifier-shaped values -- the Lark bot outage reported in #743.

QueryAsync now resolves keyword/text field paths from the target index's live
`_mapping` (GET <index>/_mapping, cached per index), falling back to declared
metadata only when the mapping cannot be read. Reading `_mapping` is a schema
read -- no mapping mutation, reindex, backfill, or event replay. Index creation
and descriptor augmentation are unchanged.

ADR-0025 records the decision (it scopes blueprint hard-constraint #2 to the
write/index-init path). Addresses #743; the alias/migration index-lifecycle
phases P1-P3/P5 remain tracked by #743.

Verification:
- dotnet test Aevatar.CQRS.Projection.Core.Tests -- 136 passed, 1 skipped
- tools/docs/lint.sh -- passed
- architecture query/projection guards -- passed

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e6a16eaa42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +303 to +307
foreach (var property in root.EnumerateObject())
{
if (property.Value.ValueKind == JsonValueKind.Object)
{
indexNode = property.Value;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Handle multi-index mapping responses deterministically

TryResolveIndexNode falls back to the first object in the _mapping payload when indexName is not an exact key match, but GET {index}/_mapping can legitimately return multiple concrete indices (for example when {index} is an alias). In that case this code picks an arbitrary index mapping and ignores the others, so exact-match path resolution (field vs field.keyword) can be wrong for part of the searched data and silently drop matches during alias/reindex windows.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant