Document that BpeOptions.ByteLevel requires a PreTokenizer

[zpczpc]

Documents the requirement reported in #7715.

BpeOptions.ByteLevel only documented the byte <-> unicode mapping (for example Space -> 'Ġ').
It did not state that a pre-tokenizer is required. Without one, byte-level encoding maps the space
character to 'Ġ' but does not keep the whitespace attached to the following token, so spaces and
newlines are dropped during encoding and cannot be recovered by decoding - with no exception and
no warning.

This adds a <remarks> block to the ByteLevel property that:

• states that PreTokenizer must be set when ByteLevel is true;
• describes the failure mode (silent whitespace loss, not recoverable on decode);
• points at RegexPreTokenizer built from the GPT-2 pattern as a configuration that round-trips losslessly.

Measured against HuggingFace tokenizers as the reference, a properly configured pre-tokenizer makes
encoding match token-for-token and decoding exact, while without one the lossless round-trip rate drops
to 1/5 on a small mixed-language sample.

This is a documentation-only change: no code and no public API surface change.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.88%. Comparing base (e4dff01) to head (9d393fc).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7716      +/-   ##
==========================================
- Coverage   69.88%   69.88%   -0.01%     
==========================================
  Files        1487     1487              
  Lines      276249   276249              
  Branches    28291    28291              
==========================================
- Hits       193064   193059       -5     
+ Misses      75696    75695       -1     
- Partials     7489     7495       +6     

Flag	Coverage Δ
Debug	69.88% <ø> (-0.01%)	⬇️
production	64.07% <ø> (-0.01%)	⬇️
test	89.82% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs	68.42% <ø> (ø)

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rosebyte]

Could we limit this claim to preserving whitespace during pre-tokenisation? Normalisation and missing vocabulary entries can still change or discard input, so the GPT-2 regex alone cannot guarantee a lossless round trip.

[rosebyte]

Could we describe the default pre-tokeniser’s behaviour rather than make this a requirement of ByteLevel?

[zpczpc]

Thanks for the review — I've reworded the remarks accordingly:

• Dropped the "lossless round trip" claim; the statement is now limited to whitespace during pre-tokenization.
• Replaced the "PreTokenizer must be set" requirement with a description of the default pre-tokenizer (CreateWordOrNonWord), whose pattern doesn't cover whitespace characters.

Please take another look when you have a moment.