Generated by Codex with GPT 5.6 Sol High

Techmeme surfaced Rest of World’s October 26 report, “AI models cost more when you don’t use English.” It covers a new Coalition for Language Equity in AI Research study showing that equal-meaning text can consume radically different numbers of tokens depending on its language and script.

That difference is not merely an accounting oddity. Most commercial model APIs charge by the token, and every model has a finite token budget. An inefficient tokenizer therefore makes the same service more expensive, slower, and less able to retain documents or conversation history for some users. CLEAR calls this hidden surcharge the “tokenizer tax.”

The inequality starts before the model reasons

Language models do not directly read words or characters. A tokenizer first converts text into entries from a fixed vocabulary. Common sequences may fit into one token; unfamiliar sequences are broken into fragments, characters, or even bytes. The vocabulary is learned from training data, so languages and scripts that appear often in digitized corpora tend to receive more efficient representations.

CLEAR illustrates the effect with one sentence translated into English, Malayalam, and Amharic. GPT-5’s tokenizer uses 13 tokens for the English version, 26 for Malayalam, and 78 for Amharic. The Amharic sentence is not six times longer or more complicated: each Ge’ez character is split into two byte-level tokens. The extra cost is created by the representation, not the meaning.

The group tested equivalent text from FLORES-200, a benchmark of professionally translated sentences covering roughly 200 languages. For each model family, it concatenated the benchmark sentences by language, counted the resulting tokens, and divided the most expensive language’s count by the least expensive language’s count. Among the 20 languages with the largest web presence, most models’ worst-to-best ratios fell between 1.5 and 3.1. Across the full benchmark, the range widened dramatically: the most equitable tokenizer measured 2.7, while the least equitable reached 17.

Script mattered more than speaker population. Low-resource Latin-script languages can inherit useful vocabulary pieces from English and related European languages. Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki repeatedly appeared among the least efficient scripts. The burden therefore reaches well beyond small language communities: Punjabi, Bengali, Urdu, Burmese, and Amharic together represent hundreds of millions of speakers.

Frontier models made very different design choices

The comparison shows that a large vocabulary helps but does not determine the outcome. Meta’s NLLB-200 translation model achieved the study’s only A grade, with a 2.7 ratio, because its tokenizer was deliberately designed for 200 languages. It is not a fair direct competitor to a general-purpose frontier model—it does not also have to represent programming languages, mathematics, and unusually strong English—but it demonstrates that multilingual efficiency is technically possible.

Google’s Gemini and Gemma family was the strongest general-purpose group on CLEAR’s worst-case metric, with a 5.1 ratio and a B grade. Anthropic’s tested family reached 9.3, Meta’s Llama 4 reached 9.5, and several other families exceeded 12. GPT-5 and gpt-oss were measured at 13.8, enough for an F under CLEAR’s deliberately strict grading rule.

That headline ranking needs a second view. GPT-5 kept 78% of the tested language-script pairs within twice the token count of its most efficient language—the best result among the general-purpose models on that broader-coverage measure. Google’s family reached 74%. In other words, GPT-5 served most languages relatively well but had severe outliers, while the max/min score is designed to focus attention on the single worst-served language. CLEAR also reports a Gini coefficient because no one statistic captures both overall distribution and tail risk.

Newer does not automatically mean fairer. Llama improved from a 14.8 ratio in version 3.3 to 9.5 in version 4, while Alibaba’s Qwen worsened from 10.5 in the tested Qwen 3 family to 13.1 in Qwen 3.6. Those movements reinforce the central claim: tokenization equity is an engineering choice that can improve or regress, not an unavoidable property of language.

Cost is only the most visible penalty

Token-based billing makes the surcharge easy to describe, but context capacity may matter more. A nominal 128,000-token window holds less source material, history, or memory when the tokenizer spends several tokens on each character. A multilingual tutoring system may forget earlier student errors sooner; an agent may fit fewer documents into a retrieval step; and a long-form assistant may need more summarization passes. More tokens also mean more computation and latency, especially where connectivity and compute are already constrained.

The problem cannot be repaired simply by translating prompts into English. That workaround can discard nuance, exclude users who are not fluent in English, and still leaves target-language output subject to the same fragmentation. Nor will lower token prices eliminate unequal context limits. Better corpora can help future tokenizer training, but an existing base model’s vocabulary is largely fixed.

CLEAR proposes two classes of remedy. For future models, labs could publish language-level tokenization scorecards and allocate vocabulary deliberately across languages and scripts. For models already deployed, providers could normalize prices by characters or language rather than charging every language identically per token. Governments and other large buyers could make multilingual cost disclosure part of procurement, creating a business incentive to improve the worst cases.

What the benchmark does not establish

The study measures the token count of one translated sentence corpus. It does not measure answer quality, reasoning accuracy, safety, culturally appropriate language, real application traffic, or the total cost of completing a task successfully. FLORES-200 is useful because it aligns meaning across languages, but its news-style sentences do not represent every conversational, coding, legal, or agentic workload.

The max/min ratio is also intentionally sensitive to one outlier. That makes it a clear alarm for the worst-served script, but a poor standalone description of the typical user experience. The study partly addresses this with the Gini coefficient and the share of languages under a two-times threshold. Model versions and tokenizers continue to change, so the table is a dated snapshot rather than a permanent ranking.

Still, the core result is difficult to dismiss. The same meaning can consume several times more of a model’s billable and contextual capacity solely because of how its script was represented during tokenizer design. CLEAR has published its evaluation code, making the benchmark reproducible and extensible even if its grading choices remain debatable.

The larger lesson is that language access is partly infrastructure. A model may advertise multilingual capability while quietly giving different languages different effective prices, speeds, and memory. Accuracy benchmarks alone will not expose that disparity; model cards and procurement tests need to measure the representation layer too.