fix: use Unicode-aware tokenizer in ROUGE-1 scorer to support non-English eval

[guptaishaan]

The rouge_score library's default tokenizer strips all non-ASCII characters using the regex [^a-z0-9]+, which causes any non-English text (Thai, Chinese, Arabic, etc.) to be reduced to empty token lists. As a result, the ROUGE-1 f-measure always returned 0.0 for non-English text, even when the candidate and reference strings were identical.

The fix introduces _UnicodeFriendlyTokenizer, a custom rouge_score.tokenizers.Tokenizer subclass that tokenizes by simple whitespace splitting and lowercasing. This correctly handles Unicode text from any language while preserving the existing behavior for ASCII text (whitespace-split unigrams). The tokenizers module is now also re-exported from the rouge_scorer dependency shim to keep the import path consistent.

Two regression tests are added to test_final_response_match_v1.py: one asserting that identical Thai text scores 1.0, and one asserting that partially-overlapping Thai text yields the expected partial score.

Fixes #3111.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[ftnext]

It looks like this PR substantially overlaps with #6292, which also introduces a Unicode-aware tokenizer for the ROUGE-1 evaluator.

To me, #6292 appears to preserve the existing ASCII tokenization behavior more carefully and includes broader regression coverage across multiple languages. It may be worth coordinating with that PR or clarifying the advantages of this approach before proceeding.