18 September 2026
How to read an agent leaderboard honestly
Error terms, session counts and why a 4.8% net improvement is not a rounding accident.
EngineeringTürkçe
Why Turkish needs its own tokenizer
Agglutination, vowel harmony and the quiet tax that general-purpose tokenizers put on every Turkish sentence.
28 August 2026 · 5 min read · Vincent Loveqcn · Yiğido Lab, İstanbul
A Turkish speaker can stack suffix on suffix: ev, evler, evlerimiz, evlerimizdekiler. Most
general-purpose tokenizers slice those words into four or five byte fragments. Each fragment costs a
token, and every token costs latency and money.
The tax in numbers
On our internal test set, the same paragraph in Turkish and in English costs:
- English, general tokenizer: 118 tokens
- Turkish, general tokenizer: 214 tokens
- Turkish, Yiğido tokenizer: 141 tokens
That is a 34% reduction in generated tokens for identical meaning — roughly a third off both the bill and the time-to-last-token.
How we got there
- A seed vocabulary trained on a Turkish-first corpus instead of an English one with Turkish sprinkled in.
- Suffix-aware merges that keep common case, tense and possessive endings whole.
- A hard floor on vowel-harmony violations during merge selection.
The tokenizer is published with the weights, so you can bring your own corpus and check the claim yourself.