OPENNLP-1929: Regex removal (7/10): Split BasicContextGenerator features on a literal separator

[krickert]

Based on #1275 until that one merges; only the commits of this change: ai-pipestream/opennlp@OPENNLP-1928-regex-removal-trivial...OPENNLP-1929-context-generator-separator

BasicContextGenerator.getContext called String.split(separator), which compiled the separator as a regular expression on each call. The separator is documented as a character, so "|" split on each character, "." matched each character and "a.b".split(".") gave an empty array, and "+" or "(" threw PatternSyntaxException at prediction time.

Changes

• Default constructor: splits on runs of whitespace under the Unicode White_Space property (StringUtil.isUnicodeWhitespace), with a fixed definition that does not follow opennlp.whitespace.mode and does not go through the shared WhitespaceTokenizer.INSTANCE (see OPENNLP-1947). Before: one U+0020 per boundary, with String.split semantics.
  • "a b": before [a, , b], now [a, b]
  • " a": before [, a], now [a]
  • "": before [""], now []
  • "a\tb", "a\nb", "a\u00A0b", "a\u0085b", "a\u2028b": before one predicate, now [a, b]
  • "a\u001Cb" and "a\u200Bb": one predicate in both versions (no White_Space property)
• BasicContextGenerator(String sep): the separator is taken as written. "|" splits on the bar, "." on the period, "+" and "(" no longer throw.
• Empty predicates are never produced: a leading, repeated, or trailing separator gives none, in both constructors.
• Argument validation: a null or empty separator, or one containing an unpaired surrogate (which could split a code point of the input), throws IllegalArgumentException at construction; a null input to getContext throws IllegalArgumentException instead of NullPointerException.
• ContextGenerator.getContext documents its return value.
• Manual: the BasicContextGenerator paragraph in machine-learning.xml states the rules above.

Incompatible change for callers that passed a pattern

Escaping was the only way to split on a regex metacharacter, and those callers now get one predicate per line:

• "\\|" on a|b: before [a, b], now [a|b]
• "\\." on a.b: before [a, b], now [a.b]
• "\\s+" on a b<TAB>c: before [a, b, c], now [a b<TAB>c]
• "[ \t]" and "[,;]": the same way

No caller in the repository, opennlp-eval-tests included, uses the separator constructor; the risk is third-party code. A deprecation path (keeping regex semantics behind @Deprecated(forRemoval = true) and adding a literal replacement) versus a documented break in the 3.0.0 migration notes is open for a decision on dev@; the tests pin the current, literal behavior so the break is visible.

Tests

BasicContextGeneratorTest, 127 cases: literal separators including multi-character and supplementary-plane ones, patterns passed as separators, whitespace characters as separators, name=value predicates, the default split over the Unicode whitespace characters and the format characters that are not whitespace, the same rows under WhitespaceMode.LEGACY, the default split with WhitespaceTokenizer.INSTANCE.setKeepNewLines(true) switched on, and the rejections. The red commits fail on the previous code.

Before merge

• No eval build is needed: no in-repo caller uses either constructor outside tests.
• Author decision: deprecation path for regex separators (dev@), and rebasing onto main without the OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil #1275 commits, which this change does not use.

OPENNLP-1929

[rzo1]

Little time, so here is a GPT 5.6-sol review instead for now

Compatibility note. No additional splitting bug found; please document the intentional constructor behavior changes.

Validation across the combined stack: 1,856 targeted tests, zero failures, one skipped.

[rzo1]

This intentionally changes existing constructor behavior: regex separators such as \s+ become literal, and empty separators now throw. Please include these changes in the migration notes.

[rzo1]

Here are some additional comments. The literal-separator change breaks existing callers silently, so it needs a deprecation path or a documented maintainer decision.

Blocking

1. BasicContextGenerator.java:81. The default constructor splits with WhitespaceTokenizer.INSTANCE, a shared mutable singleton. TokenizerME.java:195-196 calls setKeepNewLines(...) on it on every call, so once any TokenizerME with keepNewLines=true has run, getContext("a\nb\r\nc") returns [a, \n, b, \r, \n, c]. Across threads this is a data race. Scan with a private loop instead, and add a test that sets INSTANCE.setKeepNewLines(true) first.

2. BasicContextGenerator.java:58-66. A regex separator now silently stops splitting. o.split(sep) was a regex split, so escaping was the only way to split on | or ., and exactly those callers break:

   • "\\|" on a|b: old [a, b], new [a|b]
   • "\\." on a.b: old [a, b], new [a.b]
   • "\\s+" on a b\tc: old [a, b, c], new [a b\tc]
   • "[ \t]" and "[,;]" behave the same way

   The whole line becomes one unknown predicate and the model silently returns its prior outcome. This is public maxent API. Keep the String constructor with regex semantics (compile once in the constructor), mark it @Deprecated(forRemoval = true), and add a literal BasicContextGenerator(char) or a named factory as the replacement. Otherwise, get an explicit decision on dev@ and add it to the 3.0.0 migration notes.

3. BasicContextGenerator.java:50-52, :81. The default constructor changes behavior, and the PR description doesn't mention it. It used to split on one U+0020 and now splits on whitespace runs, following the global opennlp.whitespace.mode, and drops empty parts. On 1M fuzz inputs old vs new, 858,399 differ. U+0085 splits under UNICODE but not LEGACY; U+001C–U+001F behave the other way round. "".split(" ") gave [""]; now []. Document the mode dependency (link StringUtil#isWhitespace) and state the change.

4. PR description. It is out of date:

   • "keeps the String.split shape (a leading occurrence gives an empty first element…)" is wrong; the last commit drops every empty part (",a" gives [a]).
   • The default whitespace split and the IAE for null input are not mentioned.
   • "A dot matched no boundary at all" is wrong: . matched every character, so "a.b".split(".") returned [].

   Rewrite it. Retitle, e.g. "OPENNLP-1929: BasicContextGenerator splits on whitespace by default and takes a custom separator literally".

5. machine-learning.xml:82-94. "the same definition of whitespace as the event streams" is false. FileEventStream uses StringTokenizer (ASCII \t\n\r\f), and RealValueFileEventStream/RealBasicEventStream still use split("\\s+") here; only OPENNLP-1933: Regex removal (5/10): Replace per-call String.split/matches/replaceAll patterns #1279 changes them. Event lines also start with an outcome, so "for the line format of the event streams" is misleading. Cut the claim, condense to 2-3 sentences, and mention the whitespace mode. OPENNLP-1933: Regex removal (5/10): Replace per-call String.split/matches/replaceAll patterns #1279 edits the same section, so coordinate the wording.

Minor

• BasicContextGenerator.java:35-36. "Since 3.0.0 the separator is not a regular expression … (OPENNLP-1929)" is version history in Javadoc. Remove it or reduce it to one line. Same at machine-learning.xml:91-93 ("In releases before 3.0.0 …").
• BasicContextGenerator.java:47-48. "as defined by {@link WhitespaceTokenizer}" hides the mode dependency. Link StringUtil#isWhitespace and name the property.
• BasicContextGenerator.java:83-95. Allocates an ArrayList and an array on every call. Add a fast path when indexOf(separator) == -1.
• ContextGenerator.java:24-29. No @return. Add it.
• BasicContextGeneratorTest.java:102-111. A single-value @ValueSource({""}) next to a separate null @Test. Merge into one @NullAndEmptySource test.
• BasicContextGeneratorTest.java:113-119. One test asserts both constructors. Split it or parameterize it.
• BasicContextGeneratorTest.java:33-66. Only "\\s" pins the regex break. Add "\\|", "\\.", Pattern.quote("|") and "\\s+" rows, so the incompatibility is deliberate and visible.
• BasicContextGeneratorTest.java:75-94. No mode-dependent rows (U+0085, U+001C) and no "a\nb" row. Add them, pinning WhitespaceMode with a reset in @AfterEach.
• This PR uses none of OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil #1275's helpers, and WhitespaceMode is already on main. Rebase onto main and drop the OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil #1275 commits.

Verified: the literal path is correct (1M fuzz inputs vs split(Pattern.quote(sep)) with empty parts removed: 0 differences). There is no in-repo caller of either constructor, opennlp-eval-tests included, so no eval build is needed; the risk is third-party code.

[rzo1]

Follow-up: the shared WhitespaceTokenizer.INSTANCE problem behind blocking item 1 is filed as OPENNLP-1947.