feat(site-search): allow overriding the index analyzer via SITE_SEARCH_ANALYZER - #36840
feat(site-search): allow overriding the index analyzer via SITE_SEARCH_ANALYZER#36840swicken wants to merge 5 commits into
Conversation
…H_ANALYZER Site Search hardcodes an English-tuned analyzer chain (standard tokenizer + lowercase + asciifolding + English stemmer) into its bundled index mapping, which tokenizes CJK text into single-character unigrams and applies no RTL normalization, with no way to change it short of repackaging the webapp. Add a SITE_SEARCH_ANALYZER Config property (env: DOT_SITE_SEARCH_ANALYZER) naming any analyzer known to the cluster — built-in language analyzers (cjk, arabic, thai, ...) or plugin-provided ones (kuromoji, nori, smartcn). When set, it is applied to the text fields (content, title, description, author) of the site-search mapping at index creation for both the ES and OS providers; the ngram subfield's search_analyzer follows it so queries tokenize consistently. Unset keeps the bundled defaults byte-for-byte. Also fixes the ES path reading its resources via new File(url.getPath()), which breaks when packaged inside a JAR.
…rom review E2E testing against a live stack (DOT_SITE_SEARCH_ANALYZER=cjk, real Site Search job over Japanese file assets) surfaced a gap: the publishers add a content_raw field (raw pre-HTML-strip copy) to every text/* document, but the bundled mapping never declared it. OpenSearch dynamically mapped it with the standard analyzer, and since site search queries use query_string with default_field:*, CJK unigram false positives came back through that field even with the override applied (query 東京 matched a doc containing only 東 and 京 in unrelated words, via content_raw). Declare content_raw explicitly as text in both bundled mappings — same effective analysis as the previous dynamic mapping, so the unset path is unchanged — and include it in the override's text-field list. With the fix, the same E2E run returns only the genuinely matching document. Also fold in review feedback: trim the configured analyzer value (stray whitespace from env files otherwise reaches the cluster as an unknown-analyzer error), guard the JSONObject fork's null-returning toString(), log when the override is applied, correct the javadoc's description of the default analyzers, and extend the unit test with byte-identity and blank-value fallback cases.
|
Claude finished @swicken's task in 2m 36s —— View job Code Review — SITE_SEARCH_ANALYZER override
The design is sound and well-documented. The move to New Issues
No blocking issues. The two non-blocking points raised earlier by the author (mapping key order on the set path; |
…exResources javadoc
|
Tick the box to add this pull request to the merge queue (same as
|
|
Re the two non-blocking notes:
|
…dard, not standard_content
…s for analyzer override
2 participants
Proposed Changes
SITE_SEARCH_ANALYZERConfig property (env:DOT_SITE_SEARCH_ANALYZER) naming any analyzer known to the search cluster — built-in language analyzers (cjk,arabic,thai, ...) or plugin-provided ones (kuromoji,nori,smartcn). When set, it is applied at index creation to the text fields of the site-search mapping (content,content_raw,title,description,author) on both the Elasticsearch and OpenSearch providers, and thecontent.ngramsubfield'ssearch_analyzerfollows it so query-time tokenization stays consistent. Unset keeps the bundled defaults byte-for-byte.content_rawexplicitly in both bundled mappings. The publishers add it to everytext/*document but the mapping never declared it, so it was dynamically mapped with thestandardanalyzer — and since site-search queries usequery_stringwithdefault_field:"*", it silently reintroduced CJK unigram false positives even with the override applied. Declared type matches the previous dynamic mapping, so the unset path is unchanged.SiteSearchIndexResources(also fixes the ES path reading resources vianew File(url.getPath()), which breaks when packaged inside a JAR), with a loudDotSearchExceptionon missing resources or a failed override instead of silently building an index with default analyzers.Why
Site Search hardcodes an English-tuned analyzer chain (standard tokenizer + lowercase + asciifolding + English stemmer). CJK text degrades to single-character tokens (a query for 東京 matches any document containing 東 or 京 anywhere) and RTL languages get no normalization or stemming (كتاب does not match الكتاب). Resolves #36839.
Checklist
Additional Info
Verified end-to-end against a live stack, not just unit tests: built the branch image, booted dotCMS + OpenSearch with
DOT_SITE_SEARCH_ANALYZER=cjk, published Japanese HTML file assets, and ran a real run-now Site Search job:cjkon all text fields, includingcontent_raw.query_string/default_field:"*"shape Site Search uses returns only the document about Tokyo Tower; a control document containing 東 and 京 in unrelated words no longer matches (it did on the default mapping, at an identical score).analyzer [x] has not been configured in mappingserror.Scope notes:
kuromoji/nori/smartcn/icu) additionally require the plugin installed on every node of the ES/OS cluster; built-ins need nothing.content_raw(raw un-stripped HTML, no readers anywhere) roughly 2–5×'s the indexed text per document; making itindex: falseis a good follow-up but changes default-path search behavior, so it is not part of this PR.