[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer by nishant94 · Pull Request #64667 · apache/doris

nishant94 · 2026-06-22T05:53:45Z

What problem does this PR solve?

Issue Number: #64646

Related PR: None

Problem Summary:
Doris has no Japanese-aware tokenizer for the inverted index. Japanese text has no spaces between words, so the existing parsers can't segment it and MATCH / MATCH_PHRASE on Japanese columns end up with poor recall and precision.

This PR adds a built-in kuromoji parser for Japanese, in the same style as the existing Chinese IK analyzer. It's opt-in per column:

 INDEX content_idx (`content`) USING INVERTED
 PROPERTIES("parser" = "kuromoji", "parser_mode" = "search");

After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms.

How it works:

Native C++ under be/src/storage/index/inverted/analyzer/kuromoji/, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix.
- The dictionary is a process-wide singleton loaded once from ${inverted_index_dict_path}/kuromoji. An offline converter compiles raw IPADIC into a compact C++ runtime format (double-array trie + cost matrix + char/unknown tables) at build time, so no binary blob is committed.
- search (default), normal and extended modes are supported. No thrift/proto changes — parser and mode ride as strings in the index properties.

Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses).

Release note

Support Japanese text tokenization in the inverted index via a new kuromoji parser (PROPERTIES("parser"="kuromoji")), with search/normal/extended modes.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)

  CREATE TABLE test_jp (
    id BIGINT,
    content TEXT,
    INDEX idx_content (content) USING INVERTED
      PROPERTIES("parser" = "kuromoji", "parser_mode" = "search")
  ) ENGINE=OLAP
  DUPLICATE KEY(id)
  DISTRIBUTED BY HASH(id) BUCKETS 1
  PROPERTIES("replication_num" = "1");

  INSERT INTO test_jp VALUES
    (1, '東京都に住んでいます'),
    (2, '日本語の形態素解析エンジン');

  -- search-mode decompounding: 東京都 also matches 東京
  SELECT id FROM test_jp WHERE content MATCH '東京';          -- expect: 1
  SELECT id FROM test_jp WHERE content MATCH_PHRASE '形態素解析'; -- expect: 2

  -- inspect segmentation directly
  SELECT TOKENIZE('東京都に住んでいます', '"parser"="kuromoji","parser_mode"="search"');

Behavior changed:
- No.
- Yes. It adds a new opt-in kuromoji parser. Existing parsers and their output are unchanged; the new behavior only applies to indexes that explicitly set parser="kuromoji".
Does this need documentation?
- No.
- Yes. PR Link to Doris-Website.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-06-22T05:53:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

nishant94 · 2026-06-22T06:23:33Z

run buildall

yiguolei · 2026-06-22T06:27:35Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

nishant94 · 2026-06-22T06:31:13Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

@yiguolei The ICU Analyzer is not good as the Kuromoji. There is huge difference between icu and kuromoji when it comes to morphology of the Japanese words. So I think it worth it adding this new parser.

BiteTheDDDDt · 2026-06-22T07:02:18Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

nishant94 · 2026-06-22T07:29:43Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

This is original code but it is modeled on Apache Lucene's kuromoji.

hello-stephen · 2026-06-22T07:56:03Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

nishant94 · 2026-06-22T09:58:03Z

run buildall

hello-stephen · 2026-06-22T12:00:26Z

BE UT Coverage Report

Increment line coverage 82.40% (791/960) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.51% (21439/39327)
Line Coverage	38.17% (205347/537919)
Region Coverage	34.16% (161044/471416)
Branch Coverage	35.14% (70517/200651)

hello-stephen · 2026-06-22T15:49:21Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.50% (21433/39329)
Line Coverage	38.13% (205092/537920)
Region Coverage	34.11% (160793/471446)
Branch Coverage	35.11% (70468/200678)

hello-stephen · 2026-06-22T16:58:18Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.11% (28441/38375)
Line Coverage	58.02% (309954/534209)
Region Coverage	54.69% (258833/473301)
Branch Coverage	56.10% (112608/200725)

hello-stephen · 2026-06-22T17:05:28Z

FE Regression Coverage Report

Increment line coverage 66.67% (6/9) 🎉
Increment coverage report
Complete coverage report

nishant94 · 2026-06-23T06:16:44Z

run buildall

hello-stephen · 2026-06-23T11:38:51Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T11:55:51Z

FE Regression Coverage Report

Increment line coverage 35.29% (6/17) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T14:39:39Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.72% (21523/39332)
Line Coverage	38.18% (205493/538169)
Region Coverage	34.17% (161179/471738)
Branch Coverage	35.13% (70561/200832)

hello-stephen · 2026-06-23T14:44:04Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.19% (28466/38371)
Line Coverage	58.03% (310151/534436)
Region Coverage	54.77% (259367/473580)
Branch Coverage	56.13% (112755/200875)

nishant94 · 2026-06-24T05:18:18Z

run buildall

hello-stephen · 2026-06-24T06:46:43Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-24T10:23:58Z

BE UT Coverage Report

Increment line coverage 84.20% (842/1000) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.58% (21476/39348)
Line Coverage	38.09% (205045/538313)
Region Coverage	34.07% (160754/471838)
Branch Coverage	35.04% (70387/200890)

hello-stephen · 2026-06-24T14:28:27Z

BE Regression && UT Coverage Report

Increment line coverage 84.02% (468/557) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.22% (28491/38387)
Line Coverage	58.10% (310621/534591)
Region Coverage	55.02% (260603/473686)
Branch Coverage	56.29% (113100/200937)

nishant94 · 2026-07-22T10:42:44Z

run buildall

Add a built-in `kuromoji` inverted-index parser that segments Japanese text into morphemes, mirroring the existing Chinese IK analyzer.

- Added `darts.h` to `.clang-format-ignore` and `.licenserc.yaml`. - Improved code formatting in various Kuromoji source files for better readability. - Updated tests files to include necessary headers.

…mposition - Added support for search mode in the Kuromoji Viterbi segmenter, applying penalties for long all-kanji and other tokens to enhance search recall. - Updated the KuromojiMode enumeration to reflect the new search and extended modes. - Modified the KuromojiTokenizer to utilize the new mode functionality. - Added unit tests to validate the behavior of the search mode, ensuring correct segmentation of compounds. - Updated NOTICE.txt to include Apache Lucene as a dependency for the kuromoji analyzer.

…wn words - Implemented functionality in the Kuromoji Viterbi segmenter to decompose unknown (out-of-vocabulary) words into per-character unigrams when in extended mode, aligning with Lucene's JapaneseTokenizer behavior. - Added unit tests to validate the correct segmentation of unknown words in both normal and extended modes, ensuring expected outputs for various input scenarios.

- Modified error messages to include 'kuromoji' parser in the parser mode validation. - Enhanced tests for the Japanese analyzer to assert expected tokenization results.

- Introduced a new configuration option `enable_kuromoji_analyzer` to toggle the Kuromoji analyzer functionality. - Updated unit tests to validate the behavior of the Kuromoji analyzer when enabled and disabled. - Modified tests to enable the Kuromoji analyzer for specific test cases.

- Updated the namespace for Kuromoji components from `doris::segment_v2::kuromoji` to `doris::segment_v2::inverted_index::kuromoji` across multiple files for better organization and clarity.

…ictionary

- Updated the CMake configuration to ensure the required Kuromoji dictionary files are present at build time, failing the build if any are missing. - Modified the KuromojiAnalyzer and KuromojiTokenizer to throw exceptions when the dictionary is not loaded, preventing silent fallbacks to per-codepoint tokenization. - Improved error handling and validation in the dictionary loading process to ensure robust operation. - Updated unit tests to validate the new behavior, ensuring that missing dictionaries trigger appropriate errors.

- Replaced the `ending_at` vector with `end_head` and `end_next` for better memory management and performance during node processing. - Updated node addition and traversal logic to utilize the new data structures, enhancing the segmenter's efficiency in handling word segmentation.

- Changed the values in the `unk.per_category[CAT_DEFAULT]` entry from `{5, 5, 4769, "unk-default"}` to `{2, 2, 4769, "unk-default"}` to correct the test setup.

- Modified CMake configuration to conditionally include the Kuromoji dictionary files only for non-test builds (MAKE_TEST=ON). - Adjusted the custom target for generating the Kuromoji dictionary to reflect the new conditional behavior, ensuring it remains a manual target during unit-test builds.

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

- Introduced a penalty calculation mechanism for segmenting words based on the analyzer mode, improving the accuracy of word segmentation. - Added caching for the current same-category run's byte end to optimize processing of grouped unknown words, reducing unnecessary rescans.

- Moved mecab-ipadic staging into a conditional block based on the BUILD_BE flag.

- Refactored the logic for setting the Kuromoji analyzer mode to ensure it defaults to the search mode when the parser mode is empty or set to coarse granularity.

- Introduced sorting of CSV file paths before processing to ensure consistent order. - Enhanced error handling during lexicon parsing, including detailed error messages for failed parses. - Added sorting of BuilderWord entries by cost and IDs.

- Implemented a new utility function to decode the first UTF-8 code point from a given string view, enhancing character processing capabilities. - Updated the KuromojiTokenizer to skip unknown tokens based on their character category, improving tokenization accuracy.

nishant94 · 2026-07-22T10:43:34Z

run buildall

hello-stephen · 2026-07-22T13:57:08Z

BE UT Coverage Report

Increment line coverage 81.95% (949/1158) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.73% (24165/41859)
Line Coverage	41.52% (238578/574623)
Region Coverage	37.41% (188902/505015)
Branch Coverage	38.53% (84820/220125)

- Introduced a new executable `kuromoji_build_dict` to compile the UTF-8 mecab-ipadic source into binary files. - Added custom commands and targets to generate the Kuromoji dictionary, ensuring it is built as part of the overall project unless in test mode.

nishant94 · 2026-07-23T05:02:15Z

run buildall

hello-stephen · 2026-07-23T06:20:15Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

nishant94 requested review from BiteTheDDDDt, airborne12 and zclllyybb as code owners June 22, 2026 05:53

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from 389fcfb to b79db3c Compare June 22, 2026 09:57

morningman self-assigned this Jun 23, 2026

yiguolei reviewed Jun 24, 2026

View reviewed changes

Comment thread be/src/storage/index/inverted/analyzer/analyzer.cpp

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from db0ee69 to 06b4ef6 Compare June 24, 2026 03:59

nishant94 requested a review from yiguolei June 24, 2026 05:18

yiguolei reviewed Jun 26, 2026

View reviewed changes

Comment thread be/dict/kuromoji/README.md

nishant94 added 23 commits July 22, 2026 16:12

[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer

098f16c

Add a built-in `kuromoji` inverted-index parser that segments Japanese text into morphemes, mirroring the existing Chinese IK analyzer.

Update Kuromoji analyzer files and formatting

47c2041

- Added `darts.h` to `.clang-format-ignore` and `.licenserc.yaml`. - Improved code formatting in various Kuromoji source files for better readability. - Updated tests files to include necessary headers.

Enhance Japanese analyzer tests

df64d0b

- Modified error messages to include 'kuromoji' parser in the parser mode validation. - Enhanced tests for the Japanese analyzer to assert expected tokenization results.

fix indentation issues

2613c3e

Refactor Kuromoji namespace to inverted_index

97d0d98

- Updated the namespace for Kuromoji components from `doris::segment_v2::kuromoji` to `doris::segment_v2::inverted_index::kuromoji` across multiple files for better organization and clarity.

Update README.md to include Apache License information for Kuromoji d…

3c296c9

…ictionary

Fix comment formatting

386b40f

Fix test case for Kuromoji dictionary

c047644

- Changed the values in the `unk.per_category[CAT_DEFAULT]` entry from `{5, 5, 4769, "unk-default"}` to `{2, 2, 4769, "unk-default"}` to correct the test setup.

Enhance Kuromoji dictionary validation and error handling

d4cf17f

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

Enhance inverted index parser mode handling

416bc10

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

Add mecab-ipadic staging in build script

94818f6

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

Refactor Japanese analyzer tests for clarity and consistency

95f158a

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

Refactor mecab-ipadic staging logic in build script

66a0520

- Moved mecab-ipadic staging into a conditional block based on the BUILD_BE flag.

Update Kuromoji analyzer mode handling in inverted index

1661eda

- Refactored the logic for setting the Kuromoji analyzer mode to ensure it defaults to the search mode when the parser mode is empty or set to coarse granularity.

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from d689874 to 9842cf7 Compare July 22, 2026 10:42

Uh oh!

Conversation

nishant94 commented Jun 22, 2026

What problem does this PR solve?

Release note

Uh oh!

hello-stephen commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

yiguolei commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BiteTheDDDDt commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

FE UT Coverage Report

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

FE Regression Coverage Report

Uh oh!

nishant94 commented Jun 23, 2026

Uh oh!

hello-stephen commented Jun 23, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

nishant94 commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

nishant94 commented Jul 22, 2026

Uh oh!

nishant94 commented Jul 22, 2026

Uh oh!

hello-stephen commented Jul 22, 2026

BE UT Coverage Report

Uh oh!

nishant94 commented Jul 23, 2026

Uh oh!

hello-stephen commented Jul 23, 2026

FE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

nishant94 commented Jun 22, 2026 •

edited

Loading