Perplexity open-sources a Rust tokenizer that is 5× faster than Hugging Face at inference time

Perplexity AI rewrote its Unigram tokenizer from scratch in Rust and open-sourced it in
pplx-garden. Three optimisations drive the gains: a double-array trie replacing
HashMap lookups, per-node bitmap packing into 64-byte cache lines, and 2MB huge
pages to eliminate TLB misses. At 514 tokens, p50 latency drops from 349µs (Hugging
Face) to ~63µs – zero steady-state heap allocations. In production, it cut Perplexity’s
CPU utilisation 5–6× and shaved double-digit milliseconds off reranker latency.

For teams running embedding models, rerankers, or classifiers at scale, CPU
tokenization is an invisible latency tax that standard GPU profiling never surfaces. This
open-source release gives any team a production-proven, drop-in replacement - with
measurable gains in throughput and cost efficiency for high-volume inference
pipelines.