OPENNLP-1975: Regex removal (9/10): Correct the output of TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil

[rzo1]

Part 9 of the regex removal (OPENNLP-1926). Split out of #1275. Based on main; part 1 (#1311, OPENNLP-1928) is merged.

These sites lose their regular expressions like the rest of the epic, but the scans follow what the code was meant to do instead of copying the regex quirks, so the output changes. Each corrected case is pinned by a test.

Behavior changes

TokenSampleStream (English). Tokens are split on runs of Unicode White_Space (StringUtil.splitOnUnicodeWhitespace), independent of the whitespace mode; before, split("\\s+") split on ASCII whitespace only. A leading space no longer yields an empty first token with the span (0,0). A line without a token resets the open-quote state. A token holding a letter or digit of any script is a word; before, [A-Za-z0-9] made non-ASCII words attach to the previous token as punctuation.

ParserTool. Brackets that follow each other are each separated: a))b gives a ) ) b, before a ) )b. A line without a bracket is returned unchanged. Whitespace follows the toolkit whitespace mode. Documented in the parser chapter.

ConlluStream. A multiword token id must be two runs of ASCII digits without a leading zero, joined by one hyphen, with the end not smaller than the start. Any other id fails with an InvalidFormatException naming the id. Before, 3- threw ArrayIndexOutOfBoundsException, -4 and 3--4 threw NumberFormatException, and 4-3 and 1-2-3 were read without an error. The text_xx language code is read with a scan, same output. New CoNLL-U section in the corpora chapter.

DownloadUtil. A blank checksum file is an IOException saying so; before, the model was rejected with "Expected: null". The index page links are read with a scan, same output.

FeatureGeneratorUtil. The cp class needs one capital and one period and no other character. A.\n was cp and is ic now. No other token changes class.

Before merge

• Tokenizer eval build that reads through TokenSampleStream.

Commits by @krickert from #1275, regrouped.