An AI model does not read words. It reads tokens, which are chunks of text mapped to numbers, and it never sees the individual letters inside them. That is the entire answer to why a model can write a working sonnet but fail to tell you how many Rs are in "strawberry." LLM tokenization explained correctly clears up a dozen weird AI behaviors at once: the counting errors, the math slip-ups, the pricing differences between languages, and the strange sensitivity to spacing and punctuation.
Tokenization sits underneath every model you use, from chatbots to coding assistants. Skip past it and you will keep misdiagnosing model failures as intelligence problems when they are actually input format problems.
A token is a chunk of text, usually a piece of a word, sometimes a whole word, sometimes a single character or punctuation mark. Before any text reaches the neural network, a separate program called a tokenizer breaks it into these chunks and converts each one into a number. The model only ever operates on that list of numbers.
Take the word "unbelievable." A common tokenizer might split it into "un," "believ," and "able." Three tokens, one word. Now take a rare or made-up word like "glorbnaxity." The tokenizer has never seen it as a whole unit, so it falls back to smaller and smaller pieces, maybe even individual letters: "g," "lor," "b," "nax," "ity." Five tokens for one word. The model spends more of its limited attention on rare words simply because they cost more tokens to represent.
This matters because the model's entire worldview is built from these chunks. It learns statistical relationships between token IDs, not between letters. It has never directly seen the alphabet in the way you have. It infers spelling indirectly, from patterns across billions of examples, and that inference is shaky.
This is the part that trips people up the most, so it deserves a direct explanation. When you ask a model to count the letters in "strawberry," it does not scan s-t-r-a-w-b-e-r-r-y like you do. It sees whatever tokens the tokenizer produced, maybe "straw" and "berry," two chunks, neither of which comes with a built-in letter count attached. The model has to guess based on patterns it picked up during training, and guessing is exactly where errors creep in.
The same root cause explains bad arithmetic. Numbers get tokenized in chunks too, and the chunking is not consistent. "380" might be one token. "3801" might split into "380" and "1." A model doing multi-digit addition is not applying the carry-the-one method you learned in third grade. It is pattern-matching across token sequences it saw during training, and multi-digit arithmetic on numbers it has not seen before is genuinely hard for it to generalize.
Newer models close some of this gap with better training data, explicit reasoning steps, or tool use that hands math off to an actual calculator. But the underlying architecture has not changed. Any model without tool access is still reasoning over opaque chunks, not digits and letters, and will keep making this exact class of mistake.
Every API call you make gets billed by token count, both the tokens you send and the tokens the model generates back. This is not a minor billing detail. It is the actual unit of cost and speed for the entire industry, and it explains pricing decisions that otherwise look arbitrary.
A context window, the maximum amount of text a model can hold in memory at once, is also measured in tokens, not words or characters. A "128,000 token context window" sounds huge until you remember that dense code or non-English text can burn through tokens far faster than plain English prose. Paste in a long PDF and you might hit the limit sooner than expected, not because the document was too long in words, but because it tokenized inefficiently.
Here is a rough rule of thumb: one token averages about four characters or three-quarters of a word in English. So 1,000 words lands around 1,300 tokens. That ratio is not fixed, though. It shifts hard depending on the content, which is exactly the problem covered next.
Tokenizers are trained on data, and most major tokenizers were trained on datasets dominated by English text. That training bias has a direct financial consequence for anyone working in other languages, and it rarely gets mentioned in pricing pages.
English words often collapse into one or two tokens because the tokenizer has seen them constantly. Languages like Thai, Japanese, or Hindi frequently need three to ten times more tokens to represent the same sentence, because the tokenizer's vocabulary was not built around their character patterns. The result: a company processing customer support tickets in Thai pays several times more per conversation than an identical company handling the same volume in English, for the exact same underlying meaning.
| Language | Relative tokens per word (approx.) | Practical effect |
|---|---|---|
| English | 1.0x baseline | Cheapest, longest effective context |
| Spanish / French | 1.1x to 1.3x | Slightly higher cost and shorter effective context |
| Japanese / Chinese | 2x to 3x | Noticeably higher cost per sentence |
| Thai / Hindi / Amharic | 3x to 10x | Significant cost and context penalty |
This is a real equity problem in the AI industry, not just a technical footnote. Companies serving non-English markets pay a hidden tax baked into the tokenizer, and almost nobody discloses it up front.
Most modern models use a method called Byte Pair Encoding, or a close variant like WordPiece. The idea is simple even though the name sounds academic. Start with individual characters, then repeatedly merge the most frequently occurring pairs into single tokens, and keep doing that until you hit a target vocabulary size, usually somewhere between 30,000 and 100,000 tokens.
This is why common words like "the," "and," or "is" get their own single token, while rare technical terms, brand names, or typos get chopped into fragments. The algorithm learned its merge rules from whatever text corpus it was trained on, and that corpus shapes everything downstream. A tokenizer trained heavily on web text and code will represent programming syntax efficiently and represent, say, medical terminology or a low-resource language poorly.
This also explains why misspellings and unusual formatting confuse models more than people expect. Add extra spaces, use uncommon capitalization, or throw in Unicode lookalike characters, and you push the text into token sequences the model rarely saw during training. It is not being lazy. It is genuinely working with unfamiliar input.
Once you understand tokenization, a few practical habits follow directly, and they are worth adopting immediately.
Do not ask a model to count characters, reverse a string, or do precise letter-level manipulation unless it has code execution available. It will guess, and the guess will often be wrong, not because the model is weak but because the task fights the architecture. Hand that job to a tool instead. If you need exact math on large numbers, ask the model to write and run code rather than compute it directly in its head.
When cost matters, know that verbose formatting, heavy markdown, and unusual symbols all inflate token counts and therefore inflate your bill. Plain, direct prose tokenizes more efficiently than the same idea wrapped in bullet points and bold text. If you are building a product for a non-English market, budget for the token penalty before you finalize pricing, not after your first invoice surprises you.
Tokenization is not a side detail of how language models work. It is the floor everything else stands on, and most of the model's strangest failures trace straight back to it.
Tokenization is the process of breaking text into chunks called tokens, which are then converted into numbers before a language model processes them. The model never sees raw letters or words directly; it only sees these numeric token IDs, which is why its understanding of spelling and letter-level detail is indirect and unreliable.
AI models see text as tokens, which are often multi-letter chunks like "straw" and "berry" rather than individual characters. Since the model never processes letters one by one, it has to infer letter counts from learned patterns instead of directly counting, which produces frequent errors on this specific type of task.
In English, one word averages roughly 1.3 tokens, so 1,000 words typically becomes around 1,300 tokens. This ratio varies significantly by language and content type, with code, rare words, and non-English text often requiring more tokens per word than common English prose.
Yes, nearly every major LLM API charges based on the number of input and output tokens, not on words or characters. This means content that tokenizes inefficiently, such as certain non-English languages or dense code, costs more to process even when the actual meaning or length is comparable to cheaper content.
Most tokenizers were built primarily from English-heavy training data, so their vocabulary represents English words efficiently but breaks other languages into far smaller, less efficient fragments. As a result, languages like Thai or Hindi can require three to ten times more tokens than English to express the same sentence, raising both cost and context window usage.