Skip to content

Document that BpeOptions.ByteLevel requires a PreTokenizer - #7716

Open
zpczpc wants to merge 2 commits into
dotnet:mainfrom
zpczpc:patch-1
Open

Document that BpeOptions.ByteLevel requires a PreTokenizer#7716
zpczpc wants to merge 2 commits into
dotnet:mainfrom
zpczpc:patch-1

Update ByteLevel remarks per review feedback

9d393fc
Select commit
Loading
Failed to load commit list.
Build Analysis / Build Analysis failed Sep 18, 2026 in 0s

.NET Result Analysis

Details

Tip

To unconditionally bypass the build analysis check (turn it green), you can use the escape mechanism feature. The completion time may vary, potentially taking several minutes, depending on the build analysis workload at the moment.

Build Failures

MachineLearning-CI / Windows_x64_NetFx48 / Windows_x64_NetFx48 Debug_Build / Run Helix Tests

[ 🚧 Create issue against infrastructure team] [ 📄 Create issue in this repo]
    ❌Cmd.exe exited with code '1'.

Known test errors

Was this helpful? Yes No

View more details on Build Analysis