Multilingual Tokenization Pipeline
Train a BPE tokenizer and benchmark it across four languages.
Introduction
30 Second Summary
When an AI model processes a sentence in Japanese or Hindi, it often needs two to three times more "tokens" than the same sentence in English. This hidden cost affects training time, inference speed, and API pricing for billions of non-English speakers.
In this project, you will build a tokenization analysis pipeline that trains a custom tokenizer from scratch, benchmarks it against production encodings, and visualizes how different languages are treated by modern AI systems.
What You'll Build
A Python-based pipeline that trains a Byte-Pair Encoding (BPE) tokenizer from scratch, measures multilingual efficiency metrics, and compares results against OpenAI's production tiktoken encodings.
By the end of this project, you'll have:
- A custom-trained BPE tokenizer built from scratch on a multilingual corpus using the HuggingFace Tokenizers library.
- An analysis tool that measures token fertility and bytes-per-token across English, Spanish, Japanese, and Hindi.
- Comparative visualizations showing how tokenization strategy impacts computational cost across languages and tokenizers.
- Secret Mission: Implement a byte-level BPE tokenizer (GPT-2 style) and analyze how it handles non-Latin scripts differently.
Are there any prerequisites?
You should be comfortable with basic Python (functions, dictionaries, loops) and familiar with running commands in a terminal. No prior machine learning or NLP experience is needed.
Train a Custom BPE Tokenizer from Scratch
Your environment is ready. Now it's time to build the core of this project: a custom Byte-Pair Encoding tokenizer trained from scratch on multilingual text.
Training your own tokenizer reveals how vocabulary choices impact different languages. A tokenizer trained mostly on English text creates longer token sequences for Japanese or Hindi, because it never learned efficient encodings for those scripts. You'll see this disparity firsthand by training on a balanced multilingual corpus.
In this step, get ready to:
- Create a multilingual training corpus with sample text in four languages.
- Configure and train a BPE tokenizer with a custom vocabulary size.
- Test the tokenizer by encoding sample text in each language.
Create a multilingual training corpus
A tokenizer can only learn merge patterns from the text it trains on. If you feed it only English, it will never develop efficient encodings for characters used in Japanese or Hindi scripts.
- In VS Code, create a new file called train_tokenizer.py in the root of your project directory.
- Add the imports and the corpus creation function by copying this code into the file:
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
def create_training_corpus():
"""Create a multilingual training corpus."""
corpus = {
"english": [
"The transformer architecture has revolutionized natural language processing.",
"Tokenization is the first step in any NLP pipeline.",
"Byte-pair encoding iteratively merges the most frequent character pairs.",
"Large language models require efficient encoding of diverse text.",
"Machine learning systems process text by converting it to numerical representations.",
"The vocabulary size determines how many unique tokens the model can recognize.",
"Subword tokenization balances vocabulary size with sequence length.",
"Pre-training on large corpora allows models to learn general language patterns.",
"Fine-tuning adapts a pre-trained model to specific downstream tasks.",
"Attention mechanisms allow models to focus on relevant parts of the input.",
],
"spanish": [
"La arquitectura transformer ha revolucionado el procesamiento del lenguaje natural.",
"La tokenización es el primer paso en cualquier pipeline de PLN.",
"La codificación por pares de bytes fusiona iterativamente los pares de caracteres más frecuentes.",
"Los grandes modelos de lenguaje requieren una codificación eficiente de textos diversos.",
"Los sistemas de aprendizaje automático procesan texto convirtiéndolo en representaciones numéricas.",
"El tamaño del vocabulario determina cuántos tokens únicos puede reconocer el modelo.",
"La tokenización de subpalabras equilibra el tamaño del vocabulario con la longitud de la secuencia.",
"El preentrenamiento en grandes corpus permite a los modelos aprender patrones generales del lenguaje.",
"El ajuste fino adapta un modelo preentrenado a tareas específicas posteriores.",
"Los mecanismos de atención permiten a los modelos centrarse en partes relevantes de la entrada.",
],
"japanese": [
"トランスフォーマーアーキテクチャは自然言語処理に革命をもたらしました。",
"トークン化はあらゆるNLPパイプラインの最初のステップです。",
"バイトペアエンコーディングは最も頻繁な文字ペアを反復的にマージします。",
"大規模言語モデルは多様なテキストの効率的なエンコーディングを必要とします。",
"機械学習システムはテキストを数値表現に変換して処理します。",
"語彙サイズはモデルが認識できるユニークなトークンの数を決定します。",
"サブワードトークン化は語彙サイズとシーケンス長のバランスを取ります。",
"大規模コーパスでの事前学習により、モデルは一般的な言語パターンを学習できます。",
"ファインチューニングは事前学習済みモデルを特定のダウンストリームタスクに適応させます。",
"注意機構はモデルが入力の関連部分に集中することを可能にします。",
],
"hindi": [
"ट्रांसफॉर्मर आर्किटेक्चर ने प्राकृतिक भाषा प्रसंस्करण में क्रांति ला दी है।",
"टोकनाइजेशन किसी भी एनएलपी पाइपलाइन का पहला कदम है।",
"बाइट-पेयर एन्कोडिंग सबसे अधिक बार आने वाले कैरेक्टर जोड़ों को मर्ज करती है।",
"बड़े भाषा मॉडल को विविध पाठ की कुशल एन्कोडिंग की आवश्यकता होती है।",
"मशीन लर्निंग सिस्टम टेक्स्ट को संख्यात्मक प्रतिनिधित्व में बदलकर प्रोसेस करते हैं।",
"शब्दावली का आकार निर्धारित करता है कि मॉडल कितने अद्वितीय टोकन पहचान सकता है।",
"सबवर्ड टोकनाइजेशन शब्दावली आकार और अनुक्रम लंबाई के बीच संतुलन बनाता है।",
"बड़े कॉर्पस पर प्री-ट्रेनिंग मॉडल को सामान्य भाषा पैटर्न सीखने की अनुमति देती है।",
"फाइन-ट्यूनिंग एक प्री-ट्रेंड मॉडल को विशिष्ट डाउनस्ट्रीम कार्यों के लिए अनुकूलित करती है।",
"ध्यान तंत्र मॉडल को इनपुट के प्रासंगिक भागों पर ध्यान केंद्रित करने की अनुमति देता है।",
],
}
return corpus
def save_corpus_to_files(corpus):
"""Save corpus to individual files for tokenizer training."""
import os
os.makedirs("data", exist_ok=True)
# Combine all language sentences into a single list
all_lines = []
for lang, sentences in corpus.items():
all_lines.extend(sentences)
# Write all sentences to a single training file
filepath = "data/training_corpus.txt"
with open(filepath, "w", encoding="utf-8") as f:
for line in all_lines:
f.write(line + "\n")
return filepathWhat does this code do?
The imports pull in the key components from the HuggingFace tokenizers library: the Tokenizer container, the BPE model, a trainer, and a whitespace pre-tokenizer.
create_training_corpus() builds a dictionary of 40 sentences (10 per language) covering NLP concepts in English, Spanish, Japanese, and Hindi. This gives the BPE algorithm exposure to Latin characters, accented characters, CJK characters, and Devanagari script.
save_corpus_to_files() flattens all sentences into a single text file (data/training_corpus.txt) because the tokenizer's .train() method expects a list of file paths, not raw strings.
- Save train_tokenizer.py.
- You won't see output yet. The next chunk adds the training logic that uses this corpus.
Configure and train a BPE tokenizer
The BPE algorithm starts with individual characters and iteratively merges the most frequent adjacent pairs into new tokens. The vocabulary size caps how many merges it can perform. A smaller vocabulary (like 1000) means shorter merge history and longer token sequences.
- In train_tokenizer.py, add the training function below your save_corpus_to_files() function:
def train_bpe_tokenizer(corpus_file, vocab_size=1000):
"""Train a BPE tokenizer from scratch."""
# Initialize a tokenizer with the BPE model and unknown token
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
# Configure the trainer with special tokens and merge rules
trainer = BpeTrainer(
vocab_size=vocab_size,
special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
min_frequency=2,
)
# Train on the corpus file and save the result
tokenizer.train([corpus_file], trainer)
tokenizer.save("data/custom_bpe_tokenizer.json")
print(f"Tokenizer trained with vocabulary size: {tokenizer.get_vocab_size()}")
return tokenizerWhat does this code do?
Tokenizer(BPE(unk_token="[UNK]")) creates a tokenizer that uses [UNK] for any character sequence it hasn't learned to encode. The Whitespace() pre-tokenizer splits input on spaces before BPE merges begin.
The BpeTrainer configures the merge algorithm. vocab_size=1000 keeps training fast (seconds, not minutes). min_frequency=2 means a character pair must appear at least twice before it becomes a merge candidate.
The five special tokens ([UNK], [CLS], [SEP], [PAD], [MASK]) are reserved IDs that won't be overwritten by merges. These follow the standard BERT-style convention for classification, separation, padding, and masking tokens.
- Save train_tokenizer.py.
- You won't see output yet. The next chunk adds the test function and runner that will execute everything.
Test the tokenizer on multilingual input
The real test of a tokenizer is encoding text it has never seen. You'll feed it one sentence in each language and observe how many tokens it needs for each.
- Add the test function and the main execution block at the bottom of train_tokenizer.py:
def test_tokenizer(tokenizer):
"""Test the tokenizer with sample encodings."""
test_sentences = [
"Tokenization is fundamental to NLP.",
"La tokenización es fundamental para el PLN.",
"トークン化はNLPの基本です。",
"टोकनाइजेशन एनएलपी के लिए मौलिक है।",
]
print("\n--- Custom BPE Tokenizer Test ---")
for sentence in test_sentences:
output = tokenizer.encode(sentence)
print(f"\nInput: {sentence}")
print(f"Tokens: {output.tokens}")
print(f"IDs: {output.ids}")
print(f"Token count: {len(output.ids)}")
if __name__ == "__main__":
corpus = create_training_corpus()
corpus_file = save_corpus_to_files(corpus)
tokenizer = train_bpe_tokenizer(corpus_file, vocab_size=1000)
test_tokenizer(tokenizer)What does this code do?
test_tokenizer() encodes four parallel sentences (one per language) and prints the resulting tokens, their numeric IDs, and the total token count. This lets you compare how many tokens each language needs for a similar concept.
The if __name__ == "__main__": block chains the full pipeline together: create the corpus, save it to disk, train the tokenizer, then test it. This only runs when you execute the file directly.
- Save train_tokenizer.py.
- Run the script in your terminal by running this command:
python train_tokenizer.pyYou should see the tokenizer report its vocabulary size, then print token breakdowns for each of the four test sentences. Notice how the English sentence produces fewer tokens compared to Japanese or Hindi.
Seeing a ModuleNotFoundError?
Make sure your virtual environment is still active. You should see the environment name in parentheses at the start of your terminal prompt.
If you see a UnicodeEncodeError, check that your file is saved with UTF-8 encoding. In VS Code, look at the bottom-right status bar for the encoding indicator.
help me debug a ModuleNotFoundError when running train_tokenizer.py
Why does Japanese need more tokens?
With only 1000 vocabulary entries, the tokenizer can't learn efficient merges for every script. Japanese characters are multi-byte in UTF-8 and appear less frequently in the small corpus, so they get split into more individual tokens.
This is the "token tax" in action. Languages with non-Latin scripts pay a computational cost because the tokenizer fragments their text into more pieces.
- Confirm that the output files were created by running:
ls data/You should see training_corpus.txt and custom_bpe_tokenizer.json in the output.
✔️ Awesome, I've got everything!
Great. Double check you've saved your file and both output files appear in the data/ directory.
ⓧ I'd like to double check the full code
Here is the complete train_tokenizer.py file for comparison:
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
def create_training_corpus():
"""Create a multilingual training corpus."""
corpus = {
"english": [
"The transformer architecture has revolutionized natural language processing.",
"Tokenization is the first step in any NLP pipeline.",
"Byte-pair encoding iteratively merges the most frequent character pairs.",
"Large language models require efficient encoding of diverse text.",
"Machine learning systems process text by converting it to numerical representations.",
"The vocabulary size determines how many unique tokens the model can recognize.",
"Subword tokenization balances vocabulary size with sequence length.",
"Pre-training on large corpora allows models to learn general language patterns.",
"Fine-tuning adapts a pre-trained model to specific downstream tasks.",
"Attention mechanisms allow models to focus on relevant parts of the input.",
],
"spanish": [
"La arquitectura transformer ha revolucionado el procesamiento del lenguaje natural.",
"La tokenización es el primer paso en cualquier pipeline de PLN.",
"La codificación por pares de bytes fusiona iterativamente los pares de caracteres más frecuentes.",
"Los grandes modelos de lenguaje requieren una codificación eficiente de textos diversos.",
"Los sistemas de aprendizaje automático procesan texto convirtiéndolo en representaciones numéricas.",
"El tamaño del vocabulario determina cuántos tokens únicos puede reconocer el modelo.",
"La tokenización de subpalabras equilibra el tamaño del vocabulario con la longitud de la secuencia.",
"El preentrenamiento en grandes corpus permite a los modelos aprender patrones generales del lenguaje.",
"El ajuste fino adapta un modelo preentrenado a tareas específicas posteriores.",
"Los mecanismos de atención permiten a los modelos centrarse en partes relevantes de la entrada.",
],
"japanese": [
"トランスフォーマーアーキテクチャは自然言語処理に革命をもたらしました。",
"トークン化はあらゆるNLPパイプラインの最初のステップです。",
"バイトペアエンコーディングは最も頻繁な文字ペアを反復的にマージします。",
"大規模言語モデルは多様なテキストの効率的なエンコーディングを必要とします。",
"機械学習システムはテキストを数値表現に変換して処理します。",
"語彙サイズはモデルが認識できるユニークなトークンの数を決定します。",
"サブワードトークン化は語彙サイズとシーケンス長のバランスを取ります。",
"大規模コーパスでの事前学習により、モデルは一般的な言語パターンを学習できます。",
"ファインチューニングは事前学習済みモデルを特定のダウンストリームタスクに適応させます。",
"注意機構はモデルが入力の関連部分に集中することを可能にします。",
],
"hindi": [
"ट्रांसफॉर्मर आर्किटेक्चर ने प्राकृतिक भाषा प्रसंस्करण में क्रांति ला दी है।",
"टोकनाइजेशन किसी भी एनएलपी पाइपलाइन का पहला कदम है।",
"बाइट-पेयर एन्कोडिंग सबसे अधिक बार आने वाले कैरेक्टर जोड़ों को मर्ज करती है।",
"बड़े भाषा मॉडल को विविध पाठ की कुशल एन्कोडिंग की आवश्यकता होती है।",
"मशीन लर्निंग सिस्टम टेक्स्ट को संख्यात्मक प्रतिनिधित्व में बदलकर प्रोसेस करते हैं।",
"शब्दावली का आकार निर्धारित करता है कि मॉडल कितने अद्वितीय टोकन पहचान सकता है।",
"सबवर्ड टोकनाइजेशन शब्दावली आकार और अनुक्रम लंबाई के बीच संतुलन बनाता है।",
"बड़े कॉर्पस पर प्री-ट्रेनिंग मॉडल को सामान्य भाषा पैटर्न सीखने की अनुमति देती है।",
"फाइन-ट्यूनिंग एक प्री-ट्रेंड मॉडल को विशिष्ट डाउनस्ट्रीम कार्यों के लिए अनुकूलित करती है।",
"ध्यान तंत्र मॉडल को इनपुट के प्रासंगिक भागों पर ध्यान केंद्रित करने की अनुमति देता है।",
],
}
return corpus
def save_corpus_to_files(corpus):
"""Save corpus to individual files for tokenizer training."""
import os
os.makedirs("data", exist_ok=True)
all_lines = []
for lang, sentences in corpus.items():
all_lines.extend(sentences)
filepath = "data/training_corpus.txt"
with open(filepath, "w", encoding="utf-8") as f:
for line in all_lines:
f.write(line + "\n")
return filepath
def train_bpe_tokenizer(corpus_file, vocab_size=1000):
"""Train a BPE tokenizer from scratch."""
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(
vocab_size=vocab_size,
special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
min_frequency=2,
)
tokenizer.train([corpus_file], trainer)
tokenizer.save("data/custom_bpe_tokenizer.json")
print(f"Tokenizer trained with vocabulary size: {tokenizer.get_vocab_size()}")
return tokenizer
def test_tokenizer(tokenizer):
"""Test the tokenizer with sample encodings."""
test_sentences = [
"Tokenization is fundamental to NLP.",
"La tokenización es fundamental para el PLN.",
"トークン化はNLPの基本です。",
"टोकनाइजेशन एनएलपी के लिए मौलिक है।",
]
print("\n--- Custom BPE Tokenizer Test ---")
for sentence in test_sentences:
output = tokenizer.encode(sentence)
print(f"\nInput: {sentence}")
print(f"Tokens: {output.tokens}")
print(f"IDs: {output.ids}")
print(f"Token count: {len(output.ids)}")
if __name__ == "__main__":
corpus = create_training_corpus()
corpus_file = save_corpus_to_files(corpus)
tokenizer = train_bpe_tokenizer(corpus_file, vocab_size=1000)
test_tokenizer(tokenizer)You now have a working BPE tokenizer trained from scratch on multilingual data. Next up, you'll build an analysis tool that measures exactly how much this token disparity costs each language.
Build the Multilingual Fertility Analysis
In the previous step, you trained a custom BPE tokenizer on a multilingual corpus. The tokenizer works, but how well does it handle each language?
Different languages have vastly different writing systems. English uses spaces between words and a small alphabet. Japanese uses no spaces and thousands of characters. A tokenizer trained on mixed data will inevitably favor some languages over others. In this step, you will measure that disparity using a metric called token fertility.
In this step, get ready to:
- Write functions that measure token fertility (tokens per word) for each language.
- Encode the same semantic content in all four languages and compare token counts.
- Calculate bytes-per-token as a language-independent efficiency metric.
Create the analysis file with test data and a tokenizer loader
To measure tokenization quality, you need parallel text samples. These are sentences that express the same meaning in each language so you can make fair comparisons.
- Create a new file called analyze_fertility.py in your project root (the same folder where train_tokenizer.py lives).
- Add the following imports, tokenizer loader, and test sentence data by pasting this code:
import tiktoken
from tokenizers import Tokenizer
def load_custom_tokenizer():
"""Load the custom BPE tokenizer."""
return Tokenizer.from_file("data/custom_bpe_tokenizer.json")
def get_test_sentences():
"""Get parallel test sentences for fertility analysis."""
return {
"english": [
"The transformer architecture has revolutionized natural language processing.",
"Large language models require efficient encoding of diverse text.",
"Machine learning systems process text by converting it to numerical representations.",
],
"spanish": [
"La arquitectura transformer ha revolucionado el procesamiento del lenguaje natural.",
"Los grandes modelos de lenguaje requieren una codificación eficiente de textos diversos.",
"Los sistemas de aprendizaje automático procesan texto convirtiéndolo en representaciones numéricas.",
],
"japanese": [
"トランスフォーマーアーキテクチャは自然言語処理に革命をもたらしました。",
"大規模言語モデルは多様なテキストの効率的なエンコーディングを必要とします。",
"機械学習システムはテキストを数値表現に変換して処理します。",
],
"hindi": [
"ट्रांसफॉर्मर आर्किटेक्चर ने प्राकृतिक भाषा प्रसंस्करण में क्रांति ला दी है।",
"बड़े भाषा मॉडल को विविध पाठ की कुशल एन्कोडिंग की आवश्यकता होती है।",
"मशीन लर्निंग सिस्टम टेक्स्ट को संख्यात्मक प्रतिनिधित्व में बदलकर प्रोसेस करते हैं।",
],
}What does this code do?
The imports bring in tiktoken (OpenAI's tokenizer library) and the HuggingFace Tokenizer class you used in the previous step.
The load_custom_tokenizer() function loads the tokenizer you already trained from its JSON file. The get_test_sentences() function returns three parallel sentences per language. These express similar concepts so that token count differences reveal tokenizer efficiency, not content differences.
- Save analyze_fertility.py.
Now add the core metric functions and the custom tokenizer analysis. Token fertility measures how many tokens a tokenizer produces per word. A fertility of 1.0 means one token per word. Higher values mean the tokenizer is splitting words into more pieces.
- Add the following functions below get_test_sentences() in analyze_fertility.py:
def calculate_fertility(tokens, text):
"""Calculate fertility: tokens per word."""
words = text.split()
word_count = len(words) if len(words) > 0 else 1
return len(tokens) / word_count
def calculate_bytes_per_token(token_count, text):
"""Calculate bytes per token (higher = more efficient)."""
byte_count = len(text.encode("utf-8"))
return byte_count / token_count if token_count > 0 else 0
def analyze_custom_tokenizer(tokenizer, sentences_by_lang):
"""Analyze the custom tokenizer across languages."""
results = {}
for lang, sentences in sentences_by_lang.items():
total_tokens = 0
total_words = 0
total_bytes = 0
for sentence in sentences:
output = tokenizer.encode(sentence)
token_count = len(output.ids)
word_count = len(sentence.split())
byte_count = len(sentence.encode("utf-8"))
total_tokens += token_count
total_words += word_count
total_bytes += byte_count
results[lang] = {
"total_tokens": total_tokens,
"fertility": total_tokens / total_words if total_words > 0 else 0,
"bytes_per_token": total_bytes / total_tokens if total_tokens > 0 else 0,
}
return resultsWhat does this code do?
The calculate_fertility() function divides the number of tokens by the number of whitespace-separated words. A fertility of 2.0 means, on average, each word becomes two tokens.
The calculate_bytes_per_token() function divides the total UTF-8 byte count by the token count. Higher bytes-per-token means the tokenizer is packing more information into each token.
The analyze_custom_tokenizer() function loops through every language, encodes each sentence, and accumulates the totals to produce per-language fertility and efficiency metrics.
- Save analyze_fertility.py.
Seeing a red underline on tiktoken?
Make sure your virtual environment is active. The tiktoken package was installed in the previous step, but VS Code may need you to select the correct Python interpreter. Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Python: Select Interpreter, and choose the one inside your virtual environment.
Help me fix my Python interpreter in VS Code
Add production tokenizer analysis and token tax computation
Your custom tokenizer has a small vocabulary of 1000 tokens. Production tokenizers like OpenAI's cl100k_base (~100,000 tokens) and o200k_base (200,000 tokens) have much larger vocabularies trained on massive multilingual data. Comparing against them shows how vocabulary size affects cross-language fairness.
- Add the following two functions below analyze_custom_tokenizer() in analyze_fertility.py:
def analyze_tiktoken_encoding(encoding_name, sentences_by_lang):
"""Analyze a tiktoken encoding across languages."""
enc = tiktoken.get_encoding(encoding_name)
results = {}
for lang, sentences in sentences_by_lang.items():
total_tokens = 0
total_words = 0
total_bytes = 0
for sentence in sentences:
token_ids = enc.encode(sentence)
token_count = len(token_ids)
word_count = len(sentence.split())
byte_count = len(sentence.encode("utf-8"))
total_tokens += token_count
total_words += word_count
total_bytes += byte_count
results[lang] = {
"total_tokens": total_tokens,
"fertility": total_tokens / total_words if total_words > 0 else 0,
"bytes_per_token": total_bytes / total_tokens if total_tokens > 0 else 0,
}
return results
def compute_token_tax(results, reference_lang="english"):
"""Compute token tax: ratio of tokens vs English."""
ref_tokens = results[reference_lang]["total_tokens"]
tax = {}
for lang, data in results.items():
tax[lang] = data["total_tokens"] / ref_tokens if ref_tokens > 0 else 0
return taxWhat is token tax?
The analyze_tiktoken_encoding() function uses tiktoken.get_encoding() to load a production encoding by name, then runs the same fertility analysis you built for your custom tokenizer.
The compute_token_tax() function calculates the ratio of tokens each language needs compared to English. A token tax of 2.0x means that language needs twice as many tokens to express similar content. This directly translates to doubled training cost and inference time.
- Save analyze_fertility.py.
Not sure where to paste?
The new functions go directly below the closing line of analyze_custom_tokenizer(). Make sure there is one blank line between the end of analyze_custom_tokenizer() and the start of def analyze_tiktoken_encoding.
Help me figure out where to place functions in my Python file
Add the reporting function and run the analysis
The final piece is a formatted output function and the main execution block. This ties everything together by running all three tokenizers against the same multilingual test data and printing a comparison table.
- Add the following code below compute_token_tax() in analyze_fertility.py:
def print_analysis(tokenizer_name, results, token_tax):
"""Print formatted analysis results."""
print(f"\n{'='*60}")
print(f" {tokenizer_name}")
print(f"{'='*60}")
print(f"{'Language':<12} {'Tokens':<10} {'Fertility':<12} {'Bytes/Token':<14} {'Token Tax':<10}")
print(f"{'-'*60}")
for lang in results:
data = results[lang]
tax = token_tax[lang]
print(
f"{lang:<12} {data['total_tokens']:<10} "
f"{data['fertility']:<12.2f} {data['bytes_per_token']:<14.2f} "
f"{tax:<10.2f}x"
)
if __name__ == "__main__":
sentences = get_test_sentences()
# Analyze custom tokenizer
custom_tok = load_custom_tokenizer()
custom_results = analyze_custom_tokenizer(custom_tok, sentences)
custom_tax = compute_token_tax(custom_results)
print_analysis("Custom BPE (vocab=1000)", custom_results, custom_tax)
# Analyze tiktoken cl100k_base
cl100k_results = analyze_tiktoken_encoding("cl100k_base", sentences)
cl100k_tax = compute_token_tax(cl100k_results)
print_analysis("tiktoken cl100k_base", cl100k_results, cl100k_tax)
# Analyze tiktoken o200k_base
o200k_results = analyze_tiktoken_encoding("o200k_base", sentences)
o200k_tax = compute_token_tax(o200k_results)
print_analysis("tiktoken o200k_base", o200k_results, o200k_tax)What does this code do?
The print_analysis() function formats the results into a readable table with columns for token count, fertility, bytes-per-token, and token tax.
The __main__ block runs the full pipeline: it loads your custom tokenizer, runs all three tokenizers against the same multilingual sentences, and prints a formatted comparison for each.
- Save analyze_fertility.py.
- Run the analysis script in your terminal:
python analyze_fertility.pyYou should see three formatted tables printed to your terminal. Each table shows the token count, fertility, bytes-per-token, and token tax for all four languages under one tokenizer.
What should I see?
The output shows three sections: one for your Custom BPE (vocab=1000), one for tiktoken cl100k_base, and one for tiktoken o200k_base. Notice how Japanese and Hindi have much higher fertility and token tax values than English across all tokenizers.
Your custom tokenizer will show the highest disparity because its small vocabulary forces aggressive splitting. The production tokenizers handle multilingual text better thanks to their larger vocabularies, but the gap between English and non-Latin scripts persists.
Seeing a FileNotFoundError?
This means the script cannot find data/custom_bpe_tokenizer.json. Make sure you ran train_tokenizer.py in the previous step and that your terminal is in the project root folder (the one containing the data/ directory).
Help me fix my FileNotFoundError when running analyze_fertility.py
✔️ Awesome, I've got everything!
Great. Double check you have saved analyze_fertility.py and that the tables printed without errors.
ⓧ I'd like to double check the full code
Here is the complete analyze_fertility.py file. Compare it against yours:
import tiktoken
from tokenizers import Tokenizer
def load_custom_tokenizer():
"""Load the custom BPE tokenizer."""
return Tokenizer.from_file("data/custom_bpe_tokenizer.json")
def get_test_sentences():
"""Get parallel test sentences for fertility analysis."""
return {
"english": [
"The transformer architecture has revolutionized natural language processing.",
"Large language models require efficient encoding of diverse text.",
"Machine learning systems process text by converting it to numerical representations.",
],
"spanish": [
"La arquitectura transformer ha revolucionado el procesamiento del lenguaje natural.",
"Los grandes modelos de lenguaje requieren una codificación eficiente de textos diversos.",
"Los sistemas de aprendizaje automático procesan texto convirtiéndolo en representaciones numéricas.",
],
"japanese": [
"トランスフォーマーアーキテクチャは自然言語処理に革命をもたらしました。",
"大規模言語モデルは多様なテキストの効率的なエンコーディングを必要とします。",
"機械学習システムはテキストを数値表現に変換して処理します。",
],
"hindi": [
"ट्रांसफॉर्मर आर्किटेक्चर ने प्राकृतिक भाषा प्रसंस्करण में क्रांति ला दी है।",
"बड़े भाषा मॉडल को विविध पाठ की कुशल एन्कोडिंग की आवश्यकता होती है।",
"मशीन लर्निंग सिस्टम टेक्स्ट को संख्यात्मक प्रतिनिधित्व में बदलकर प्रोसेस करते हैं।",
],
}
def calculate_fertility(tokens, text):
"""Calculate fertility: tokens per word."""
words = text.split()
word_count = len(words) if len(words) > 0 else 1
return len(tokens) / word_count
def calculate_bytes_per_token(token_count, text):
"""Calculate bytes per token (higher = more efficient)."""
byte_count = len(text.encode("utf-8"))
return byte_count / token_count if token_count > 0 else 0
def analyze_custom_tokenizer(tokenizer, sentences_by_lang):
"""Analyze the custom tokenizer across languages."""
results = {}
for lang, sentences in sentences_by_lang.items():
total_tokens = 0
total_words = 0
total_bytes = 0
for sentence in sentences:
output = tokenizer.encode(sentence)
token_count = len(output.ids)
word_count = len(sentence.split())
byte_count = len(sentence.encode("utf-8"))
total_tokens += token_count
total_words += word_count
total_bytes += byte_count
results[lang] = {
"total_tokens": total_tokens,
"fertility": total_tokens / total_words if total_words > 0 else 0,
"bytes_per_token": total_bytes / total_tokens if total_tokens > 0 else 0,
}
return results
def analyze_tiktoken_encoding(encoding_name, sentences_by_lang):
"""Analyze a tiktoken encoding across languages."""
enc = tiktoken.get_encoding(encoding_name)
results = {}
for lang, sentences in sentences_by_lang.items():
total_tokens = 0
total_words = 0
total_bytes = 0
for sentence in sentences:
token_ids = enc.encode(sentence)
token_count = len(token_ids)
word_count = len(sentence.split())
byte_count = len(sentence.encode("utf-8"))
total_tokens += token_count
total_words += word_count
total_bytes += byte_count
results[lang] = {
"total_tokens": total_tokens,
"fertility": total_tokens / total_words if total_words > 0 else 0,
"bytes_per_token": total_bytes / total_tokens if total_tokens > 0 else 0,
}
return results
def compute_token_tax(results, reference_lang="english"):
"""Compute token tax: ratio of tokens vs English."""
ref_tokens = results[reference_lang]["total_tokens"]
tax = {}
for lang, data in results.items():
tax[lang] = data["total_tokens"] / ref_tokens if ref_tokens > 0 else 0
return tax
def print_analysis(tokenizer_name, results, token_tax):
"""Print formatted analysis results."""
print(f"\n{'='*60}")
print(f" {tokenizer_name}")
print(f"{'='*60}")
print(f"{'Language':<12} {'Tokens':<10} {'Fertility':<12} {'Bytes/Token':<14} {'Token Tax':<10}")
print(f"{'-'*60}")
for lang in results:
data = results[lang]
tax = token_tax[lang]
print(
f"{lang:<12} {data['total_tokens']:<10} "
f"{data['fertility']:<12.2f} {data['bytes_per_token']:<14.2f} "
f"{tax:<10.2f}x"
)
if __name__ == "__main__":
sentences = get_test_sentences()
# Analyze custom tokenizer
custom_tok = load_custom_tokenizer()
custom_results = analyze_custom_tokenizer(custom_tok, sentences)
custom_tax = compute_token_tax(custom_results)
print_analysis("Custom BPE (vocab=1000)", custom_results, custom_tax)
# Analyze tiktoken cl100k_base
cl100k_results = analyze_tiktoken_encoding("cl100k_base", sentences)
cl100k_tax = compute_token_tax(cl100k_results)
print_analysis("tiktoken cl100k_base", cl100k_results, cl100k_tax)
# Analyze tiktoken o200k_base
o200k_results = analyze_tiktoken_encoding("o200k_base", sentences)
o200k_tax = compute_token_tax(o200k_results)
print_analysis("tiktoken o200k_base", o200k_results, o200k_tax)You now have concrete numbers showing the tokenization gap across languages. Next up, you will benchmark your custom tokenizer against production encodings in a structured comparison and compute the exact cost disparity.
Benchmark Against Production Tokenizers
In the previous step, you built the analysis functions and ran your custom tokenizer through all four languages. Now it's time to see how your small 1000-token vocabulary compares against the tokenizers that power real production models.
Your analyze_fertility.py already includes code that loads OpenAI's production encodings and runs the same benchmark. In this step, you'll run the full comparison and learn to interpret the results.
In this step, get ready to:
- Understand how tiktoken loads production encodings like cl100k_base and o200k_base.
- Run the full three-way tokenizer comparison on your multilingual test sentences.
- Interpret the token tax ratio to quantify multilingual encoding cost.
Understand tiktoken's production encodings
Your custom tokenizer has a vocabulary of 1000 tokens trained on a small corpus. Production tokenizers from OpenAI use vocabularies that are 100-200x larger, trained on massive internet-scale datasets.
- In VS Code, switch back to analyze_fertility.py and scroll to the analyze_tiktoken_encoding function.
- Review the function signature and the first line inside it:
def analyze_tiktoken_encoding(encoding_name, sentences_by_lang):
"""Analyze a tiktoken encoding across languages."""
enc = tiktoken.get_encoding(encoding_name)What are cl100k_base and o200k_base?
tiktoken.get_encoding() loads one of OpenAI's production encoding schemes by name. Each encoding defines a fixed vocabulary and set of merge rules.
- cl100k_base has approximately 100,000 tokens. It powers GPT-4 and GPT-3.5-turbo.
- o200k_base has 200,000 tokens. It powers GPT-4o and GPT-4o-mini.
A larger vocabulary means fewer tokens per word on average. The encoding can represent more character sequences as single tokens instead of splitting them into subwords.
- Now scroll to the __main__ block at the bottom of the file. Find the section that calls analyze_tiktoken_encoding:
# Analyze tiktoken cl100k_base
cl100k_results = analyze_tiktoken_encoding("cl100k_base", sentences)
cl100k_tax = compute_token_tax(cl100k_results)
print_analysis("tiktoken cl100k_base", cl100k_results, cl100k_tax)
# Analyze tiktoken o200k_base
o200k_results = analyze_tiktoken_encoding("o200k_base", sentences)
o200k_tax = compute_token_tax(o200k_results)
print_analysis("tiktoken o200k_base", o200k_results, o200k_tax)What does this code do?
This runs the exact same multilingual test sentences through both production encodings. Each call follows the same pattern: encode all sentences, measure tokens and bytes, then compute the token tax ratio relative to English.
By passing the same sentences dictionary to all three tokenizers, you get a fair apples-to-apples comparison.
Run the full three-way comparison
- Run the analysis script to see all three tokenizers compared side by side:
python analyze_fertility.pyYou should see three formatted tables printed to your terminal, one for each tokenizer. Each table shows columns for Language, Tokens, Fertility, Bytes/Token, and Token Tax.
What should I see?
Your custom BPE tokenizer (1000 vocab) will show much higher token counts than the production tokenizers. This is expected. A 1000-token vocabulary has far fewer merge rules, so it splits text into more pieces.
The production tokenizers (cl100k_base and o200k_base) will be far more efficient overall, but still show disparities across languages. Japanese and Hindi will require more tokens than English across all tokenizers.
Seeing a download message the first time?
tiktoken downloads encoding data from the internet on first use. If you see a brief download message, that is normal. It caches the data locally so subsequent runs are instant.
If the download fails, check your internet connection and try again.
Help me debug a tiktoken download error when running analyze_fertility.py.
Interpret the token tax ratios
The Token Tax column in your output is the most revealing metric. It answers a simple question: how many more tokens does this language need compared to English for the same meaning?
- Look at the Token Tax column in your terminal output for each of the three tokenizer tables.
- Find the language with the highest token tax in the Custom BPE table.
- Compare that same language's token tax in the cl100k_base and o200k_base tables.
Why does the token tax matter?
A token tax of 2.0x means that language requires twice as many tokens as English for equivalent content. This directly impacts cost and performance. Doubling in tokens results in quadrupled training cost and time because attention is quadratic in sequence length.
The compute_token_tax function calculates this by dividing each language's total token count by the English token count. English always shows 1.00x as the baseline.
- Review the compute_token_tax function in your file to see how the ratio is calculated:
def compute_token_tax(results, reference_lang="english"):
"""Compute token tax: ratio of tokens vs English."""
ref_tokens = results[reference_lang]["total_tokens"]
tax = {}
for lang, data in results.items():
tax[lang] = data["total_tokens"] / ref_tokens if ref_tokens > 0 else 0
return taxWhat patterns should I notice?
- Your custom tokenizer shows the biggest disparities because its small vocabulary was trained on limited data.
- Production tokenizers reduce the gap, but non-Latin scripts (Japanese, Hindi) still pay a token tax even with 200K vocabulary.
- o200k_base generally performs better than cl100k_base on non-English text thanks to its larger vocabulary.
Extra for Experts: Why does vocabulary size help non-English languages?
A larger vocabulary stores more character sequences as single tokens. With 200K tokens, o200k_base can represent common Japanese and Hindi character sequences as single tokens instead of splitting them into individual bytes or characters.
However, even 200K tokens cannot fully close the gap. Languages with high-fragmentation orthographies still show around 3.4x more tokens per word compared to English.
✔️ Awesome, I've got everything!
Great. Make sure you have saved analyze_fertility.py and can see all three comparison tables in your terminal.
ⓧ I'd like to double check the full code
Here is the complete analyze_fertility.py file for comparison:
import tiktoken
from tokenizers import Tokenizer
def load_custom_tokenizer():
"""Load the custom BPE tokenizer."""
return Tokenizer.from_file("data/custom_bpe_tokenizer.json")
def get_test_sentences():
"""Get parallel test sentences for fertility analysis."""
return {
"english": [
"The transformer architecture has revolutionized natural language processing.",
"Large language models require efficient encoding of diverse text.",
"Machine learning systems process text by converting it to numerical representations.",
],
"spanish": [
"La arquitectura transformer ha revolucionado el procesamiento del lenguaje natural.",
"Los grandes modelos de lenguaje requieren una codificación eficiente de textos diversos.",
"Los sistemas de aprendizaje automático procesan texto convirtiéndolo en representaciones numéricas.",
],
"japanese": [
"トランスフォーマーアーキテクチャは自然言語処理に革命をもたらしました。",
"大規模言語モデルは多様なテキストの効率的なエンコーディングを必要とします。",
"機械学習システムはテキストを数値表現に変換して処理します。",
],
"hindi": [
"ट्रांसफॉर्मर आर्किटेक्चर ने प्राकृतिक भाषा प्रसंस्करण में क्रांति ला दी है।",
"बड़े भाषा मॉडल को विविध पाठ की कुशल एन्कोडिंग की आवश्यकता होती है।",
"मशीन लर्निंग सिस्टम टेक्स्ट को संख्यात्मक प्रतिनिधित्व में बदलकर प्रोसेस करते हैं।",
],
}
def calculate_fertility(tokens, text):
"""Calculate fertility: tokens per word."""
words = text.split()
word_count = len(words) if len(words) > 0 else 1
return len(tokens) / word_count
def calculate_bytes_per_token(token_count, text):
"""Calculate bytes per token (higher = more efficient)."""
byte_count = len(text.encode("utf-8"))
return byte_count / token_count if token_count > 0 else 0
def analyze_custom_tokenizer(tokenizer, sentences_by_lang):
"""Analyze the custom tokenizer across languages."""
results = {}
for lang, sentences in sentences_by_lang.items():
total_tokens = 0
total_words = 0
total_bytes = 0
for sentence in sentences:
output = tokenizer.encode(sentence)
token_count = len(output.ids)
word_count = len(sentence.split())
byte_count = len(sentence.encode("utf-8"))
total_tokens += token_count
total_words += word_count
total_bytes += byte_count
results[lang] = {
"total_tokens": total_tokens,
"fertility": total_tokens / total_words if total_words > 0 else 0,
"bytes_per_token": total_bytes / total_tokens if total_tokens > 0 else 0,
}
return results
def analyze_tiktoken_encoding(encoding_name, sentences_by_lang):
"""Analyze a tiktoken encoding across languages."""
enc = tiktoken.get_encoding(encoding_name)
results = {}
for lang, sentences in sentences_by_lang.items():
total_tokens = 0
total_words = 0
total_bytes = 0
for sentence in sentences:
token_ids = enc.encode(sentence)
token_count = len(token_ids)
word_count = len(sentence.split())
byte_count = len(sentence.encode("utf-8"))
total_tokens += token_count
total_words += word_count
total_bytes += byte_count
results[lang] = {
"total_tokens": total_tokens,
"fertility": total_tokens / total_words if total_words > 0 else 0,
"bytes_per_token": total_bytes / total_tokens if total_tokens > 0 else 0,
}
return results
def compute_token_tax(results, reference_lang="english"):
"""Compute token tax: ratio of tokens vs English."""
ref_tokens = results[reference_lang]["total_tokens"]
tax = {}
for lang, data in results.items():
tax[lang] = data["total_tokens"] / ref_tokens if ref_tokens > 0 else 0
return tax
def print_analysis(tokenizer_name, results, token_tax):
"""Print formatted analysis results."""
print(f"\n{'='*60}")
print(f" {tokenizer_name}")
print(f"{'='*60}")
print(f"{'Language':<12} {'Tokens':<10} {'Fertility':<12} {'Bytes/Token':<14} {'Token Tax':<10}")
print(f"{'-'*60}")
for lang in results:
data = results[lang]
tax = token_tax[lang]
print(
f"{lang:<12} {data['total_tokens']:<10} "
f"{data['fertility']:<12.2f} {data['bytes_per_token']:<14.2f} "
f"{tax:<10.2f}x"
)
if __name__ == "__main__":
sentences = get_test_sentences()
# Analyze custom tokenizer
custom_tok = load_custom_tokenizer()
custom_results = analyze_custom_tokenizer(custom_tok, sentences)
custom_tax = compute_token_tax(custom_results)
print_analysis("Custom BPE (vocab=1000)", custom_results, custom_tax)
# Analyze tiktoken cl100k_base
cl100k_results = analyze_tiktoken_encoding("cl100k_base", sentences)
cl100k_tax = compute_token_tax(cl100k_results)
print_analysis("tiktoken cl100k_base", cl100k_results, cl100k_tax)
# Analyze tiktoken o200k_base
o200k_results = analyze_tiktoken_encoding("o200k_base", sentences)
o200k_tax = compute_token_tax(o200k_results)
print_analysis("tiktoken o200k_base", o200k_results, o200k_tax)You now have hard numbers showing how tokenization strategy impacts different languages. Next up, you'll turn these numbers into visualizations that make the disparities immediately obvious.
Visualize Tokenization Disparities
Your analysis script now produces hard numbers comparing tokenization efficiency across languages and encoders.
Numbers in a terminal table are useful, but charts make disparities immediately obvious. In this step, you will build Matplotlib visualizations that turn your fertility and token-tax data into shareable graphics.
In this step, get ready to:
- Create a grouped bar chart comparing token counts across languages and tokenizers.
- Generate a fertility heatmap showing tokens-per-word for each language and tokenizer combination.
- Build a token tax line chart and print a summary report of findings.
Create a grouped bar chart comparing token counts
A grouped bar chart places each tokenizer's token counts side-by-side per language, making it easy to spot which languages get fragmented most.
- Create a new file called visualize_results.py in your project root (the same folder as analyze_fertility.py).
- Add the imports and the bar chart function by pasting this code:
import matplotlib.pyplot as plt
import tiktoken
from tokenizers import Tokenizer
from analyze_fertility import (
get_test_sentences,
analyze_custom_tokenizer,
analyze_tiktoken_encoding,
compute_token_tax,
)
def create_token_count_comparison(all_results):
"""Create grouped bar chart comparing token counts."""
languages = list(next(iter(all_results.values())).keys())
tokenizer_names = list(all_results.keys())
x_positions = range(len(languages))
bar_width = 0.25
fig, ax = plt.subplots(figsize=(12, 6))
for i, (name, results) in enumerate(all_results.items()):
token_counts = [results[lang]["total_tokens"] for lang in languages]
offset = (i - len(tokenizer_names) / 2 + 0.5) * bar_width
bars = ax.bar(
[x + offset for x in x_positions],
token_counts,
bar_width,
label=name,
)
ax.set_xlabel("Language")
ax.set_ylabel("Total Token Count")
ax.set_title("Token Count Comparison Across Languages and Tokenizers")
ax.set_xticks(x_positions)
ax.set_xticklabels([lang.capitalize() for lang in languages])
ax.legend()
ax.grid(axis="y", alpha=0.3)
plt.tight_layout()
plt.savefig("data/token_count_comparison.png", dpi=150)
print("Saved: data/token_count_comparison.png")
plt.close()What does this code do?
The imports pull in your analysis functions from analyze_fertility.py so you can reuse them without duplicating code.
The create_token_count_comparison() function calculates an offset for each tokenizer's bars so they appear side by side. It loops through the results dictionary, plots one bar group per tokenizer, and saves the chart as a PNG.
- Save visualize_results.py.
Seeing an import error?
Make sure visualize_results.py is in the same folder as analyze_fertility.py. Python resolves imports relative to the script location.
Also confirm your virtual environment is still active. The matplotlib package must be installed in the active environment.
Help me debug an import error in visualize_results.py
Generate a fertility heatmap
A heatmap encodes numeric values as colors, letting you scan rows and columns instantly. Each cell will show the fertility score for one tokenizer-language pair.
- In visualize_results.py, add the following function below create_token_count_comparison():
def create_fertility_heatmap(all_results):
"""Create a fertility heatmap."""
languages = list(next(iter(all_results.values())).keys())
tokenizer_names = list(all_results.keys())
fertility_data = []
for name in tokenizer_names:
row = [all_results[name][lang]["fertility"] for lang in languages]
fertility_data.append(row)
fig, ax = plt.subplots(figsize=(10, 5))
im = ax.imshow(fertility_data, cmap="YlOrRd", aspect="auto")
ax.set_xticks(range(len(languages)))
ax.set_xticklabels([lang.capitalize() for lang in languages])
ax.set_yticks(range(len(tokenizer_names)))
ax.set_yticklabels(tokenizer_names)
for i in range(len(tokenizer_names)):
for j in range(len(languages)):
text = ax.text(
j, i, f"{fertility_data[i][j]:.2f}",
ha="center", va="center", color="black", fontsize=11,
)
ax.set_title("Token Fertility (Tokens per Word) Heatmap")
fig.colorbar(im, ax=ax, label="Fertility (lower = more efficient)")
plt.tight_layout()
plt.savefig("data/fertility_heatmap.png", dpi=150)
print("Saved: data/fertility_heatmap.png")
plt.close()What does this code do?
The function builds a 2D array of fertility values (rows = tokenizers, columns = languages) and renders it with ax.imshow() using the YlOrRd colormap. Yellow means lower fertility (more efficient). Red means higher fertility (less efficient).
The nested loop overlays the exact numeric value on each cell so you can read precise numbers while still getting the visual color cue.
- Save visualize_results.py.
Heatmap looks wrong or has overlapping text?
Check that the figsize=(10, 5) matches the code above. A smaller figure squishes the labels together.
Make sure you placed this function after the bar chart function, not inside it. Indentation matters.
Help me fix my fertility heatmap
Add the token tax chart and summary report
The token tax chart is a line plot showing how many more tokens each language requires relative to English. This makes the cost disparity immediately obvious.
- In visualize_results.py, add the create_token_tax_chart() function below create_fertility_heatmap():
def create_token_tax_chart(all_results):
"""Create a token tax comparison chart."""
languages = list(next(iter(all_results.values())).keys())
tokenizer_names = list(all_results.keys())
fig, ax = plt.subplots(figsize=(10, 6))
for name, results in all_results.items():
tax = compute_token_tax(results)
tax_values = [tax[lang] for lang in languages]
ax.plot(
[lang.capitalize() for lang in languages],
tax_values,
marker="o",
linewidth=2,
label=name,
)
ax.axhline(y=1.0, color="gray", linestyle="--", alpha=0.5, label="Baseline (English)")
ax.set_xlabel("Language")
ax.set_ylabel("Token Tax (ratio vs English)")
ax.set_title("Token Tax: Relative Cost of Tokenizing Non-English Text")
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
plt.savefig("data/token_tax_chart.png", dpi=150)
print("Saved: data/token_tax_chart.png")
plt.close()What does this code do?
Each tokenizer gets a line connecting its token-tax values across languages. The dashed gray baseline at 1.0 represents English (no tax). Any point above 1.0 means that language costs more tokens for the same semantic content.
The higher a line rises above 1.0, the greater the computational penalty that language pays during training and inference.
- Save visualize_results.py.
Now add the summary report function and the main execution block that ties everything together.
- Add the print_summary() function and __main__ block at the bottom of visualize_results.py:
def print_summary(all_results):
"""Print a summary report."""
print("\n" + "=" * 60)
print(" TOKENIZATION ANALYSIS SUMMARY")
print("=" * 60)
for name, results in all_results.items():
tax = compute_token_tax(results)
max_tax_lang = max(tax, key=tax.get)
min_bpt_lang = min(results, key=lambda l: results[l]["bytes_per_token"])
print(f"\n{name}:")
print(f" Highest token tax: {max_tax_lang} ({tax[max_tax_lang]:.2f}x vs English)")
print(f" Least efficient (bytes/token): {min_bpt_lang} ({results[min_bpt_lang]['bytes_per_token']:.2f})")
print("\n" + "=" * 60)
print("Key Insight: Languages with higher fertility (tokens/word)")
print("incur greater computational cost during training and inference.")
print("This 'token tax' disproportionately affects morphologically")
print("complex and non-Latin script languages.")
print("=" * 60)
if __name__ == "__main__":
sentences = get_test_sentences()
# Load all tokenizers and analyze
custom_tok = Tokenizer.from_file("data/custom_bpe_tokenizer.json")
custom_results = analyze_custom_tokenizer(custom_tok, sentences)
cl100k_results = analyze_tiktoken_encoding("cl100k_base", sentences)
o200k_results = analyze_tiktoken_encoding("o200k_base", sentences)
all_results = {
"Custom BPE (1K vocab)": custom_results,
"cl100k_base (GPT-4)": cl100k_results,
"o200k_base (GPT-4o)": o200k_results,
}
# Generate visualizations
create_token_count_comparison(all_results)
create_fertility_heatmap(all_results)
create_token_tax_chart(all_results)
print_summary(all_results)What does this code do?
The print_summary() function identifies the highest token-tax language and least byte-efficient language for each tokenizer, then prints a key insight about the real-world impact of these disparities.
The __main__ block reuses your analysis functions from analyze_fertility.py to load all three tokenizers, run the analysis, and call every visualization function in sequence.
- Save visualize_results.py.
- Run the visualization script:
python visualize_results.py- You should see three "Saved:" confirmation lines followed by the TOKENIZATION ANALYSIS SUMMARY report printed to your terminal.
- Confirm the image files were created by running:
ls data/*.png- You should see token_count_comparison.png, fertility_heatmap.png, and token_tax_chart.png in the output.
Why does the token tax matter?
Doubling the number of tokens a language needs quadruples training cost and time. Languages like Japanese and Hindi often require 2-3x more tokens than English for the same content when using English-heavy tokenizers.
Your visualizations now make this disparity impossible to ignore. Production ML teams use exactly this kind of analysis to decide vocabulary sizes and training data mixes.
No PNG files appearing?
Make sure the data/ folder already exists. It should from running train_tokenizer.py earlier. If not, create it manually with mkdir data.
If you see a FileNotFoundError mentioning custom_bpe_tokenizer.json, run python train_tokenizer.py first to regenerate the tokenizer file.
Help me debug why my visualizations are not saving
✔️ Awesome, I've got everything!
Great. Double check you have saved visualize_results.py and confirmed all three PNG files exist in your data/ folder.
ⓧ I'd like to double check the full code
Here is the complete visualize_results.py for comparison:
import matplotlib.pyplot as plt
import tiktoken
from tokenizers import Tokenizer
from analyze_fertility import (
get_test_sentences,
analyze_custom_tokenizer,
analyze_tiktoken_encoding,
compute_token_tax,
)
def create_token_count_comparison(all_results):
"""Create grouped bar chart comparing token counts."""
languages = list(next(iter(all_results.values())).keys())
tokenizer_names = list(all_results.keys())
x_positions = range(len(languages))
bar_width = 0.25
fig, ax = plt.subplots(figsize=(12, 6))
for i, (name, results) in enumerate(all_results.items()):
token_counts = [results[lang]["total_tokens"] for lang in languages]
offset = (i - len(tokenizer_names) / 2 + 0.5) * bar_width
bars = ax.bar(
[x + offset for x in x_positions],
token_counts,
bar_width,
label=name,
)
ax.set_xlabel("Language")
ax.set_ylabel("Total Token Count")
ax.set_title("Token Count Comparison Across Languages and Tokenizers")
ax.set_xticks(x_positions)
ax.set_xticklabels([lang.capitalize() for lang in languages])
ax.legend()
ax.grid(axis="y", alpha=0.3)
plt.tight_layout()
plt.savefig("data/token_count_comparison.png", dpi=150)
print("Saved: data/token_count_comparison.png")
plt.close()
def create_fertility_heatmap(all_results):
"""Create a fertility heatmap."""
languages = list(next(iter(all_results.values())).keys())
tokenizer_names = list(all_results.keys())
fertility_data = []
for name in tokenizer_names:
row = [all_results[name][lang]["fertility"] for lang in languages]
fertility_data.append(row)
fig, ax = plt.subplots(figsize=(10, 5))
im = ax.imshow(fertility_data, cmap="YlOrRd", aspect="auto")
ax.set_xticks(range(len(languages)))
ax.set_xticklabels([lang.capitalize() for lang in languages])
ax.set_yticks(range(len(tokenizer_names)))
ax.set_yticklabels(tokenizer_names)
for i in range(len(tokenizer_names)):
for j in range(len(languages)):
text = ax.text(
j, i, f"{fertility_data[i][j]:.2f}",
ha="center", va="center", color="black", fontsize=11,
)
ax.set_title("Token Fertility (Tokens per Word) Heatmap")
fig.colorbar(im, ax=ax, label="Fertility (lower = more efficient)")
plt.tight_layout()
plt.savefig("data/fertility_heatmap.png", dpi=150)
print("Saved: data/fertility_heatmap.png")
plt.close()
def create_token_tax_chart(all_results):
"""Create a token tax comparison chart."""
languages = list(next(iter(all_results.values())).keys())
tokenizer_names = list(all_results.keys())
fig, ax = plt.subplots(figsize=(10, 6))
for name, results in all_results.items():
tax = compute_token_tax(results)
tax_values = [tax[lang] for lang in languages]
ax.plot(
[lang.capitalize() for lang in languages],
tax_values,
marker="o",
linewidth=2,
label=name,
)
ax.axhline(y=1.0, color="gray", linestyle="--", alpha=0.5, label="Baseline (English)")
ax.set_xlabel("Language")
ax.set_ylabel("Token Tax (ratio vs English)")
ax.set_title("Token Tax: Relative Cost of Tokenizing Non-English Text")
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
plt.savefig("data/token_tax_chart.png", dpi=150)
print("Saved: data/token_tax_chart.png")
plt.close()
def print_summary(all_results):
"""Print a summary report."""
print("\n" + "=" * 60)
print(" TOKENIZATION ANALYSIS SUMMARY")
print("=" * 60)
for name, results in all_results.items():
tax = compute_token_tax(results)
max_tax_lang = max(tax, key=tax.get)
min_bpt_lang = min(results, key=lambda l: results[l]["bytes_per_token"])
print(f"\n{name}:")
print(f" Highest token tax: {max_tax_lang} ({tax[max_tax_lang]:.2f}x vs English)")
print(f" Least efficient (bytes/token): {min_bpt_lang} ({results[min_bpt_lang]['bytes_per_token']:.2f})")
print("\n" + "=" * 60)
print("Key Insight: Languages with higher fertility (tokens/word)")
print("incur greater computational cost during training and inference.")
print("This 'token tax' disproportionately affects morphologically")
print("complex and non-Latin script languages.")
print("=" * 60)
if __name__ == "__main__":
sentences = get_test_sentences()
# Load all tokenizers and analyze
custom_tok = Tokenizer.from_file("data/custom_bpe_tokenizer.json")
custom_results = analyze_custom_tokenizer(custom_tok, sentences)
cl100k_results = analyze_tiktoken_encoding("cl100k_base", sentences)
o200k_results = analyze_tiktoken_encoding("o200k_base", sentences)
all_results = {
"Custom BPE (1K vocab)": custom_results,
"cl100k_base (GPT-4)": cl100k_results,
"o200k_base (GPT-4o)": o200k_results,
}
# Generate visualizations
create_token_count_comparison(all_results)
create_fertility_heatmap(all_results)
create_token_tax_chart(all_results)
print_summary(all_results)Secret mission
Implement Byte-Level BPE
Your standard BPE tokenizer uses whitespace pre-tokenization and needs an unknown token for characters it has never seen. GPT-2 solved this problem with byte-level BPE, which operates on raw bytes instead of characters. Build a byte-level BPE tokenizer and discover how it changes the multilingual picture.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files for future experimentation or delete them to free up disk space.
Resources you used:
- Project folder with Python scripts (train_tokenizer.py, analyze_fertility.py, visualize_results.py, byte_level_bpe.py).
- The data/ directory containing trained tokenizer JSON files, training corpus, and PNG visualizations.
- Python virtual environment with installed packages (tokenizers, tiktoken, matplotlib).
Keep everything running
No action needed. Choose this if you want to keep experimenting with different vocabulary sizes, training corpora, or additional languages.
- Your project folder stays on your Desktop with all scripts and generated data intact.
- You can re-run any script at any time by activating your virtual environment and using python.
Pause - I'll come back to this later
Since everything runs locally with no background processes, there is nothing to pause. Your files stay on disk until you delete them.
- Deactivate your virtual environment if it is still active by running:
deactivate- When you return, navigate back to your project folder and reactivate the virtual environment.
Delete - I don't want to use this again
Remove all project files, trained tokenizers, and the virtual environment. This frees up all disk space used by the project.
- Deactivate your virtual environment if it is still active by running:
deactivate- Navigate to your Desktop and delete the entire project folder by running these commands (macOS/Linux):
cd ~/Desktop
rm -rf tokenization-pipeline- On Windows, run these commands instead:
cd ~\Desktop
rmdir /s /q tokenization-pipeline- This removes all Python scripts, the data/ directory (trained tokenizers, corpus, and visualizations), and the virtual environment with all installed packages.
Nice Work!
Nice Work!
You just built a complete multilingual tokenization analysis pipeline from the ground up. You now have hands-on experience with the exact skills that power production ML encoding systems.
What you learned:
- Trained a BPE tokenizer from scratch using the HuggingFace Tokenizers library, configuring vocabulary size, special tokens, and pre-tokenizer pipelines.
- Measured multilingual tokenization efficiency using token fertility (tokens per word) and bytes-per-token metrics across English, Spanish, Japanese, and Hindi.
- Benchmarked your custom tokenizer against OpenAI's production encodings (cl100k_base and o200k_base) and quantified the token tax that impacts non-English languages in LLM training.
- Generated Matplotlib visualizations showing cross-lingual tokenization disparities through grouped bar charts, fertility heatmaps, and token tax line charts.
- Implemented byte-level BPE (GPT-2 style) and analyzed how it handles multilingual text differently by eliminating unknown tokens and operating directly on raw bytes.
Ready to quiz yourself?