-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
Batch scaling: a flat batch path that shares nothing between workers
#2361
opened Aug 25, 2026 by
ArthurZucker
Collaborator
Loading…
perf(tk-train): 34x BPE training on a 588 MB corpus (9m53s -> 17.6s), byte-identical
#2348
opened Aug 21, 2026 by
ArthurZucker
Collaborator
Loading…
Fix Unigram trainer prune loss using vocab size instead of per-piece alternatives count
#2346
opened Aug 20, 2026 by
mayuriphad
Loading…
fix(normalizer): make NormalizedString::prepend work when normalized is empty
#2339
opened Aug 18, 2026 by
natedemoss
Loading…
fix(bpe): stop build() panicking on a merge longer than any vocab token
#2333
opened Aug 11, 2026 by
axiom-of-choice
Loading…
fix(bpe): stop merging pairs whose count has gone negative
#2327
opened Aug 9, 2026 by
yentur
Loading…
Add missing return type hints to ByteLevelBPETokenizer.from_file, train, train_from_iterator
#2326
opened Aug 9, 2026 by
RudrenduPaul
Loading…
Add missing return type hints to CharBPETokenizer.from_file, train, train_from_iterator
#2325
opened Aug 9, 2026 by
RudrenduPaul
Loading…
[BUG] | The docs listing page for tokenizers is throwing 404
#2311
opened Aug 6, 2026 by
xiaoyaoqilan
Loading…
TARGET POC: bitsplit
run-pipeline-bench
target
#2306
opened Aug 6, 2026 by
ArthurZucker
Collaborator
Loading…
Fix usize underflow in BPE / Strip decoders on empty / short input
#2245
opened Jul 27, 2026 by
weixlu
Loading…
Update fancy-regex dependency to 0.18
#2243
opened Jul 25, 2026 by
musicinmybrain
Contributor
Loading…
Fix panic in Strip decoder on short all-content tokens
#2239
opened Jul 24, 2026 by
v-code01
Loading…
Preserve backslashes in generated Python stub docstrings
#2218
opened Jul 18, 2026 by
cupkk
Loading…
Previous Next
ProTip!
Updated in the last three days: updated:>2026-08-22.