How many tokens does Vietnamese cost? Measured on six tokenizers — plus a formatting issue that multiplies the AI bill by two and a half

Imagine a development team building an AI assistant to read customer files. They estimate costs by dividing document length by four — the "about four characters per token" rule many English guides mention. The first month's bill could come in at nearly double the estimate. Part of that is because Vietnamese uses more tokens than English. The rest comes from something nobody thought about: some files were extracted from PDFs, and the accented letters in them were stored a different way.

This article measures both with concrete numbers. Every figure below was measured by KonexForge on public tokenizers, using texts we wrote — you can repeat the same measurement on your own data.

What a token is, and why language affects the count

AI models do not read letter by letter but in tokens — pieces of text cut according to a set of rules called a tokenizer. A token can be a whole word, part of a word, or a punctuation mark. API prices are per token, context limits are in tokens, and response speed is measured in tokens too.

Tokenizers are built from training data. Languages that appear often in that data are cut into fewer pieces — a common English word is usually one token. Less common languages are cut into smaller fragments, so the same content takes more tokens. Vietnamese, with its tone marks and modified letters, used to be fragmented considerably by early tokenizers.

How we measured

We wrote eight typical Vietnamese business texts, each with an English translation of the same content: a customer asking about an order, a product description, a return policy, an internal email, a contract clause, instructions for an AI assistant, a revenue report paragraph, and a technical question. 378 Vietnamese words and 288 English words in total.

We then counted tokens with six publicly released tokenizers:

• o200k_base — OpenAI's tokenizer used from GPT-4o and many later models.
• cl100k_base — OpenAI's previous tokenizer, used by GPT-4 and GPT-3.5.
• Llama 3 from Meta.
• Qwen3 from Alibaba.
• Gemma 3, Google's family of open models.
• DeepSeek V3.

Anthropic's Claude models and the newest models from OpenAI and Google do not publish downloadable tokenizers; the end of this article shows how to measure them with the providers' own tools.

Results: how much more Vietnamese costs

Vietnamese tokens divided by English tokens for the same content:

• Gemma 3: 1.23 times
• Llama 3: 1.29 times
• Qwen3: 1.31 times
• o200k (GPT-4o onwards): 1.38 times
• DeepSeek V3: 2.09 times
• cl100k (GPT-4): 2.28 times

The two groups are clearly separated. Newer tokenizers handle Vietnamese fairly efficiently: the same content costs only a quarter to two-fifths more than English. Older ones cost more than double. The ratio is fairly stable across text types — contract clauses and product descriptions cost slightly more, everyday messages and emails slightly less.

Another way to look at it: with o200k, each Vietnamese token corresponds to about 0.75 words — roughly 4 tokens for every 3 words. The English "four characters per token" rule does not apply; for a quick Vietnamese estimate on newer models, use word count × 1.3.

A note on DeepSeek: the figure above is for the published DeepSeek V3 tokenizer. Later versions such as V4.1 may use a different one. That is exactly why you should measure on the model you actually use rather than relying on general figures.

Does typing without diacritics save tokens?

A popular tip: drop Vietnamese diacritics to save money. We measured the same eight texts again with all diacritics removed:

• On older tokenizers — it is cheaper: cl100k falls from 837 to 599 tokens (about 28%), DeepSeek V3 by about 23%.
• On newer tokenizers — it costs more: o200k rises from 501 to 540 tokens, Gemma 3 by about 14%, Llama 3 by about 20%, Qwen3 by about 22%.

The reason: newer tokenizers have learned accented Vietnamese as an ordinary language, so "không" or "được" is a familiar token. "khong" or "duoc" are unfamiliar strings and get cut into smaller pieces. On top of that, dropping diacritics loses meaning: "ban" could be bán (sell), bàn (table), bạn (friend) or bận (busy). The model has to guess, and may guess wrong.

The simple conclusion: with newer models, keep the diacritics — cheaper and more accurate.

The formatting issue that multiplies tokens nearly 2.5 times

This is the most striking finding of the measurement, and could be the rest of the development team's bill from the start of this article.

In Unicode, an accented letter such as "ệ" can be stored in two ways that look identical on screen:

• Precomposed form (NFC): "ệ" is a single character.
• Decomposed form (NFD): "ệ" is three characters — "e", a circumflex and a dot below — combined when displayed.

Keyboards and most software in Vietnam use the precomposed form. But decomposed text turns up more often than people expect: file names created on some Macs, content extracted from some PDFs, data copied from some websites or apps, and some older Vietnamese input methods.

We converted the same eight texts to decomposed form and counted again:

• o200k: from 501 to 1,240 tokens — 2.48 times.
• Llama 3: 2.66 times.
• Gemma 3: 2.18 times.
• cl100k and DeepSeek V3: about 1.8 times.
• Qwen3: unchanged — this tokenizer normalises to precomposed form itself before cutting.

The screen shows exactly the same thing and readers see no difference, but the model receives a string two and a half times longer — and you pay two and a half times as much for that part. The model may also understand it less well, because it rarely saw this form during training.

Some API services may normalise text before counting and some may not — the safest approach is to normalise it yourself before sending. It takes one line of code:

• JavaScript: text.normalize("NFC")
• Python: unicodedata.normalize("NFC", text)

Put this line in exactly one place — the function that calls the API — so all text passes through it, whether from an input box, an uploaded file or content extracted from a PDF. To check existing data, compare the string length before and after normalisation: if they differ, the string is in decomposed form.

What the extra tokens affect

Cost. With newer models, about 1.3 times is an acceptable gap and rarely the deciding factor. For example: a 1,500-token instruction prompt in Vietnamese would be about 1,090 tokens written in English. At 30,000 requests a month and $2 per million input tokens, the difference is about $25 a month — and almost disappears if you cache the repeated prompt. Writing prompts in English is an option, but test it on your evaluation set first: saving $25 is not worth it if the Vietnamese replies become less natural.

Context limits and price thresholds. This is more often overlooked. A document of 80,000 tokens in English becomes about 110,000 tokens for the same content in Vietnamese, at the ratio measured on o200k. With Claude Haiku 5.5, requests over 100,000 tokens are billed entirely at a price five times higher, as discussed in the article on small models. Any threshold estimate based on English documents needs to be multiplied again for Vietnamese.

Speed. Models generate answers token by token. A Vietnamese answer with the same content needs more tokens, so it takes correspondingly longer — about 30% with newer tokenizers.

Measure on the model you actually use

The newest models from Anthropic, OpenAI and Google do not publish downloadable tokenizers, and the labs themselves change tokenizers between generations — Anthropic says Haiku 5.5 uses a new one that produces slightly more tokens for the same content. How to measure:

• Use the usage field in the response: every API call returns the actual number of input tokens. Send the same passage in Vietnamese and in English and compare the two numbers.
• Use a token-counting API: Anthropic and Google both offer a dedicated endpoint for counting tokens without running the model — cheap or free, suitable for measuring in bulk.
• Use five to ten real business passages with English translations rather than a single sample sentence — because the ratio varies with the type of text.

Conclusion

Vietnamese still costs more tokens than English, but with newer models the gap has narrowed to about 1.2–1.4 times — no longer double, as it was a few years ago. Dropping diacritics to save money is a trick from the days of old tokenizers; with new models it costs more and invites misreading.

The biggest waste the measurement found does not come from the language but from formatting: decomposed Vietnamese text can push token counts up nearly 2.5 times on many popular models. It is also the cheapest thing to fix — one line of NFC normalisation before sending.

If you are building AI systems that process Vietnamese text and want to review token costs, normalise input data or re-measure your budget on the model you actually use, contact us or see KonexForge's Development capability.