How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Researchers at Dharma-AI say in a Hugging Face post that rebuilding the tokenizer of a small language model cut measured inference cost by 77% in a production system. The system grades Portuguese essays using Qwen3-4B, and the results preview a paper called VocabX that is still under review, so the team cautions the figures may change.

The authors argue that inherited vocabularies waste money in two ways: they break a domain's most repeated text into many tokens, and they force the model to score every entry, 151,669 in this case, at each generated token. VocabX selects tokens by serving cost rather than frequency, weighting output tokens more heavily because a generated token cost 7.4 times a token read in the grader. It reshapes how words are split and also merges long repeated spans, such as JSON envelopes, into single macro tokens. The output layer shrank from about 152,000 rows to about 65,000.

Across seven grader versions differing only in tokenizer, average responses fell from 350 to 101 tokens, while the bench score rose from 0.703 to 0.810. An earlier design that fused grades into JSON macros scored just 0.689 and saved less, at 73.3%, showing the risk of compressing the decision a model exists to make.

The team also reports beating FVT, MWT and Liquid AI's in-place BPE on token compression in all fifteen combinations of five public domains and three model families, by 1.2 to 9.3 points. If it holds up, the work suggests tokenizers are a significant, overlooked cost lever for specialized models.