Hugging Face makes BERT tokenizing up to 50x faster

Simon Brandeis of Hugging Face reports that the tokenizer for BERT-style models in the open-source Rust tokenizers library encodes English text about 50x faster than version 0.23, in a community article on the Hugging Face blog dated 15 October 2026. About 8.6x of that came from earlier work; this post explains another 5.7x. The changes are still at pull request stage, ahead of a v1.0.0 release.

Why a 2018 model matters

BERT-like models are the encoders behind embeddings, search, RAG (retrieval-augmented generation), rerankers and classifiers. The author says the three most downloaded models on the Hugging Face Hub all run BERT's normaliser: all-MiniLM-L6-v2 at 218 million monthly downloads, the ms-marco-MiniLM-L6-v2 cross-encoder at 79 million and bge-small-en-v1.5 at 59 million. Tokenizing is the step that turns text into tokens before any model runs, so it sits in front of every indexing job and every query.

What was slow, and the fix

Profiling a warm encode of English text showed 88% of the time went on normalisation, the step that rewrites text into a canonical form. Choosing tokens was only 8%. The post says the fix has three parts: skip runs of ASCII characters, which normalisation never changes, checking 16 bytes at a time with CPU vector instructions; cache, per block of 256 code points, whether normalisation could touch a character; and avoid copying or allocating when the text is already normalised.

The numbers, and who measured them

These are the author's measurements on Hugging Face Jobs, using two AMD EPYC 7R13 hosts and one Intel Xeon Platinum 8488C, one thread per run. On English text, throughput went from 23 MB/s to 151 MB/s and normalisation fell from 88% to 13% of encode time. Russian, Greek and Arabic gain 2.2x to 3.2x despite being only about 11% ASCII. For ModernBERT, which uses standard NFC normalisation, English barely moves (1.06x) while code, Arabic and Russian gain about 2x. The author reports the token ids were identical to main and to 0.23.2 on every corpus, and says speedup ratios agree within a few percent across machines while raw MB/s vary.

Caveats the author lists

• Chinese gains nothing, because BERT pads every Chinese character with spaces anyway.
• In a normaliser-only sweep, 55 of 320 test cells were slower beyond noise; inputs of 32 bytes or less can be slower, down to 0.35x for one accent-stripping case.
• Cased BERT on Chinese is 8% to 15% slower from 64 bytes up.

The post does not say when the change will be released, and it measures tokenizing alone, not end-to-end indexing time, where model inference usually dominates. Teams that tokenize large corpora on CPU, or serve many short queries, should benchmark their own text mix once a release ships.