The Multilingual Data Gap
AI Engineer path · lesson 56 of 124Compare a language's share of the web against its share of humanity, price the same request in English and Burmese to see the tokenization tax, and find out why refusal behaviour is the first thing that breaks outside English.
Key insight
English being 8× the next language matters more than English being 46%. It means the second tier — Russian, German, Chinese, Japanese, French, Spanish — is already an order of magnitude thinner than the language every model is announced and benchmarked in, and those are the well-served ones.
Why non-English costs more
Sort each effect by whether it hurts quality or the bill.
Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.
Hurts answer quality
Drop items hereRaises cost and latency
Drop items hereTest your knowledge
Approximately what share of Common Crawl is English?
The short version
A library that claims to hold the world's knowledge, where nearly half the shelves are in one language and a hundred others share a single aisle. A reader in that aisle does not just find fewer books — they are also charged by the page, in a language whose pages hold less.
In more depth
The Multilingual Data Gap
Roughly 46% of Common Crawl is English, against about 18% of the world that speaks it. A hundred-plus other languages divide the remainder.
The distribution (Lai et al., 2023)
Only 12 languages reach 1% of the crawl. English at 45.88% is about 8× Russian, the second language. The second tier — Russian, German, Chinese, Japanese, French, Spanish — is already an order of magnitude thinner than the language every model is benchmarked in, and those are the well-served ones.
Under-representation ratio = % world population ÷ % Common Crawl
| Language | Speakers | % pop | % crawl | ratio |
|---|---|---|---|---|
| Punjabi | 113M | 1.41% | 0.0061% | 231.5 |
| Swahili | 71M | 0.89% | 0.0077% | 115.2 |
| Urdu | 231M | 2.89% | 0.0274% | 105.3 |
| Telugu | 95M | 1.19% | 0.0183% | 64.89 |
| Bengali | 272M | 3.40% | 0.0930% | 36.56 |
| English | 1,452M | 18.15% | 45.88% | 0.40 |
Raw share is the wrong measure; the ratio is the planning number.
The tokenization tax
Tokenizers are fitted to the training corpus, so rare languages fragment. Median tokens for the same short text (MASSIVE, 1M texts, 52 languages): English 7, Hindi 32, Burmese 72 — about 10×. That is ~10× the cost, ~10× the decode time, and ~10× less content per context window — which silently truncates retrieval.
What it costs in quality
Safety degrades fastest because refusal comes from a small, English-heavy post-training set — two gaps stacked on the same languages.
What to do
Use a purpose-built model (ChatGLM, YAYI, Llama-Chinese, CroissantLLM, PhoGPT, Jais); ground with retrieval in the target language; treat translate-and-back sceptically (it needs a good model for the same scarce language, and loses information); and evaluate in the language you serve.
Interview Questions
References & Further Reading
Love this visualization?
Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.
100% free • No credit card required