Compare Text Tokenizers

Encode text with three tokenizers, benchmark speed, and chart token counts.

Introduction

30 Second Summary

Two prompts that look equally long can produce different bills when sent to different models. Measuring your own words reveals the difference.

In this project, you will build a Python script that compares how three tokenizers split the same text. The finished script makes these invisible billing units measurable.

What You'll Build

Run your finished script to watch one paragraph become three different token counts in a saved bar chart.

By the end of this project, you'll have:

  • A token round trip that reveals each ID's text piece before reconstructing the original sentence.
  • A three-tokenizer comparison showing how different model families count the same paragraph.
  • A local speed benchmark that reports tokens per second for a large text file.
  • A saved bar chart that supports a clear prompt cost comparison across the three tokenizers.
  • Secret Mission: An optional challenge to push your token analysis skills further.

Are there any prerequisites?

You need Python, pip, and a paragraph of your own writing. The Hugging Face tokenizers library runs entirely on your laptop with no account or API call required.

Before We Start

Token counts change across model families. A clean local setup lets you measure those differences without mixing packages from other projects.

This step prepares Python 3 for local scripts. You will add Hugging Face Tokenizers plus Matplotlib inside an isolated virtual environment.

In this step, get ready to:
  • Confirm Python 3 runs locally.
  • Create an isolated tokenizer-benchmark workspace.
  • Install the tokenizer plus charting packages.
Confirm Python 3

Python runs every benchmark script in this project. A version check confirms that your terminal can find a Python 3 interpreter.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows).
  • Press Enter to open the terminal.

macOS

  • Check for Python 3 by running this command:
python3 --version

What does this command do?

The command asks the Python 3 interpreter to print its installed version. A version number proves that your terminal can run Python 3.

✔️ I see a Python 3 version

Your Mac can run Python 3 locally. Keep this terminal open for the project setup.

ⓧ Command not found

Your terminal cannot find Python 3 yet. The official installer adds a supported interpreter to your Mac.

  • Visit the official Python downloads page for macOS.
  • Download the latest stable macOS installer.
  • Open the downloaded .pkg file.
  • Complete the installer using its default options.
  • Close Terminal after the installation finishes.
  • Reopen Terminal through macOS search.
  • Confirm Python 3 is available by running the check again:
python3 --version
What confirms the installation?

The reopened terminal loads the updated command path. You should now see a Python 3 version number.

Still cannot find Python 3?

Restart your Mac so the installer changes apply to every terminal session.

If the command still fails, help me troubleshoot the Python installation.

Windows

  • Check for Python 3 by running this command:
python --version

What does this command do?

The command asks the Python interpreter to print its installed version. A Python 3 version number proves that PowerShell can run the interpreter.

✔️ I see a Python 3 version

Your Windows computer can run Python 3 locally. Keep this PowerShell window open for the project setup.

ⓧ Command not found

Windows needs a local Python installation. The official Python Install Manager provides the interpreter used throughout this project.

  • Visit the official Python downloads page for Windows.
  • Download the latest Python Install Manager.
  • Open the downloaded installer.
  • Select Install when prompted.
  • Close PowerShell after the installation finishes.
  • Reopen PowerShell through Windows search.
  • Confirm Python 3 is available by running the check again:
python --version
What confirms the installation?

The reopened PowerShell session loads the installed Python command. You should now see a Python 3 version number.

Still cannot find Python?

Restart Windows so the installer changes apply to every PowerShell session.

If the command still fails, help me troubleshoot the Python installation.

Create the project workspace

The tokenizer-benchmark folder keeps every project file in one place. .venv gives that folder its own Python packages.

Why use a virtual environment?

A virtual environment isolates this project's dependencies from packages installed elsewhere on your computer. The isolation makes your benchmark easier to reproduce.

You will use Visual Studio Code as the code editor. Its command-line tool opens the current terminal folder as an editor workspace.

  • Check whether the Visual Studio Code command-line tool is available by running:
code --version

What does this command do?

The command prints the installed Visual Studio Code version. A version number confirms that the editor can open folders from your terminal.

✔️ I see a version number

Visual Studio Code is ready to open your project folder from the terminal.

ⓧ Command not found

Visual Studio Code needs to be installed or added to your command path. Choose your operating system below.

macOS

One command-path step remains

The application is now installed. The next actions make its terminal command available.

  • Press Cmd+Space to open Spotlight.
  • Type Visual Studio Code.
  • Press Enter to open the editor.
  • Press Cmd+Shift+P to open the Command Palette.
  • Run Shell Command: Install 'code' command in PATH.
  • Close Terminal.
  • Reopen Terminal through macOS search.
  • Confirm the command-line tool is available by running:
code --version
What confirms the setup?

The new terminal session can now find Visual Studio Code. You should see its version number.

Windows

  • Visit the official Visual Studio Code download page.
  • Download the Windows User Installer.
  • Open the downloaded .exe file.
  • Complete the installer using its default options.
  • Close PowerShell.
  • Reopen PowerShell through Windows search.
  • Confirm the command-line tool is available by running:
code --version
What confirms the setup?

The new PowerShell session can now find Visual Studio Code. You should see its version number.

Still cannot find the code command?

Confirm that you reopened the terminal after installing Visual Studio Code. Existing terminal sessions keep their previous command path.

If the command still fails, help me connect Visual Studio Code to my terminal.

macOS

  • Create the tokenizer-benchmark folder on your Desktop by running these commands:
cd ~/Desktop
mkdir tokenizer-benchmark
cd tokenizer-benchmark

What do these commands do?

  • The first command moves your terminal to the Desktop.
  • The second command creates the tokenizer-benchmark folder.
  • The final command moves your terminal inside that folder.
  • Create .venv by running these commands:
python3 -m venv .venv
source .venv/bin/activate

What do these commands do?

Python creates an isolated environment inside .venv. The activation command points this terminal at that environment.

You should see (.venv) at the start of your terminal prompt. That prefix confirms the environment is active.

  • Open the current folder in Visual Studio Code by running:
code .

What does this command do?

The period represents your current terminal folder. Visual Studio Code opens tokenizer-benchmark as the workspace.

You should see tokenizer-benchmark at the top of the Explorer sidebar. Your terminal remains inside the same folder.

Windows

  • Create the tokenizer-benchmark folder on your Desktop by running these commands:
cd ~/Desktop
mkdir tokenizer-benchmark
cd tokenizer-benchmark

What do these commands do?

  • The first command moves PowerShell to the Desktop.
  • The second command creates the tokenizer-benchmark folder.
  • The final command moves PowerShell inside that folder.
  • Create .venv by running these commands:
python -m venv .venv
.\.venv\Scripts\Activate.ps1

What do these commands do?

Python creates an isolated environment inside .venv. The activation command points this PowerShell session at that environment.

You should see (.venv) at the start of your PowerShell prompt. That prefix confirms the environment is active.

  • Open the current folder in Visual Studio Code by running:
code .

What does this command do?

The period represents your current PowerShell folder. Visual Studio Code opens tokenizer-benchmark as the workspace.

You should see tokenizer-benchmark at the top of the Explorer sidebar. PowerShell remains inside the same folder.

Do you see an activation error?

Confirm that your terminal is inside tokenizer-benchmark. The activation path only works from the folder containing .venv.

If activation still fails, help me activate the virtual environment.

Install the project packages

The Tokenizers package turns text into token identifiers. Matplotlib creates the comparison chart later in the project.

This project pins Tokenizers to 1.0.0 so every learner uses the same library release. Matplotlib can use the compatible release selected by pip.

  • Install tokenizers==1.0.0 plus Matplotlib into the active environment by running:
python -m pip install "tokenizers==1.0.0" matplotlib

What does this command do?

  • The python -m pip prefix uses the package installer from your active environment.
  • The version pin installs Tokenizers release 1.0.0.
  • The final package provides the charting tools used later.

You should see installation progress followed by a successful completion message. The packages now belong to tokenizer-benchmark/.venv.

Did the package installation fail?

Check that (.venv) appears at the start of your prompt. Reconnect to the internet before retrying if the download stopped.

If the installation still fails, help me debug the package installation.

Before you check the result, which package do you expect to show version 1.0.0?

  • Display the installed package details by running:
python -m pip show tokenizers matplotlib

What does this command show?

The command prints metadata for both installed packages. Each section includes Name plus Version fields.

You should see Tokenizers at version 1.0.0. You should also see an installed Matplotlib version.

That is the foundation complete. Your isolated Python workspace can now run the tokenizer scripts you build next.

✔️ Awesome, I've got everything!

Your terminal is inside tokenizer-benchmark with .venv active. Visual Studio Code has the same folder open.

Tokenizers 1.0.0 plus Matplotlib are installed locally.

ⓧ I'd like to double check the full setup

Compare your environment with this final setup state.

  • The project folder is named tokenizer-benchmark.
  • The terminal is inside the tokenizer-benchmark folder.
  • Visual Studio Code has tokenizer-benchmark open.
  • The terminal prompt begins with (.venv).
  • Python 3 is available inside the activated environment.
  • The installed package details show Tokenizers version 1.0.0.
  • The installed package details include Matplotlib.

Your local toolchain is ready. Next up, you will encode a sentence and inspect the token identifiers behind it.

Inspect a Tokenizer

Your Python workspace is ready. Now you can see how a sentence becomes the units that a model processes.

A tokenizer uses a fixed vocabulary to split text into pieces. In this step, you will encode one sentence with bert-base-uncased.

In this step, get ready to:
  • Create token_benchmark.py inside the existing tokenizer-benchmark folder.
  • Encode Tokenizers split text into billable pieces. with the pretrained tokenizer.
  • Print the token IDs beside their readable token pieces.
Create the inspection script

Each token receives a numeric ID from the tokenizer's vocabulary. Converting those IDs back to token pieces reveals how the sentence was split.

  • Use your code editor's new-file control to create token_benchmark.py inside the open tokenizer-benchmark folder.
  • Paste the following code into token_benchmark.py.
from tokenizers import Tokenizer

# Keep the inspected text in one reusable value
sample_sentence = "Tokenizers split text into billable pieces."

# Load the pretrained splitting rules
bert_tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# Convert the sentence into token IDs
bert_encoding = bert_tokenizer.encode(sample_sentence)

# Show the IDs in numeric form
print(bert_encoding.ids)

# Show the same IDs as readable pieces
print(bert_tokenizer.convert_ids_to_tokens(bert_encoding.ids))

What does this code do?

  • The sample_sentence variable keeps the text fixed for this inspection.
  • Tokenizer.from_pretrained("bert-base-uncased") loads the pretrained vocabulary plus its splitting rules.
  • bert_tokenizer.encode(sample_sentence) produces an encoding object for the sentence.
  • bert_encoding.ids exposes the numeric IDs produced by the tokenizer.
  • convert_ids_to_tokens() maps those IDs to readable token pieces.
  • Save token_benchmark.py.
  • Confirm your editor no longer shows an unsaved-change indicator beside token_benchmark.py.
Inspect the token IDs and pieces

The first run can pause while the pretrained tokenizer files download. Later runs can load those files from the local cache.

Before you run the script, do you expect both printed lists to contain the same number of entries? Hold that prediction for the next command.

  • Run the saved script from the existing terminal by running this command:
python token_benchmark.py

What should I see?

Once loading finishes, you will see two lists in the terminal. The first list contains numeric token IDs.

The second list contains readable token pieces. Each piece corresponds to the ID at the same position.

That is your first token inspection working. Your script now exposes the numeric form plus the readable pieces of the same sentence.

Script not printing both lists?

  • Confirm the existing .venv environment remains activated if Python cannot import the library.
  • Check that the file is named token_benchmark.py if Python cannot find the script.
  • Retry the first run after restoring your internet connection if the pretrained files cannot load.
  • Compare both print() lines with the full reference below if only one list appears.

Help me troubleshoot my tokenizer inspection.

✔️ Awesome, I've got everything!

Strong result. Your script loads the pretrained tokenizer plus its cached files. It also prints both representations of the encoded sentence.

ⓧ I'd like to double check the full code

Compare your saved token_benchmark.py file with this reference.

from tokenizers import Tokenizer

# Keep the inspected text in one reusable value
sample_sentence = "Tokenizers split text into billable pieces."

# Load the pretrained splitting rules
bert_tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# Convert the sentence into token IDs
bert_encoding = bert_tokenizer.encode(sample_sentence)

# Show the IDs in numeric form
print(bert_encoding.ids)

# Show the same IDs as readable pieces
print(bert_tokenizer.convert_ids_to_tokens(bert_encoding.ids))

Your tokenizer now turns a sentence into inspectable IDs plus readable pieces. Next, you will decode those IDs to test whether the original sentence returns.

Decode the Token IDs

Your tokenizer now turns the sample sentence into token IDs. You can also see the pieces represented by those IDs.

The next part of the round trip converts those IDs back into readable text. In this step, you will decode the IDs before comparing the result with your original sentence.

In this step, get ready to:
  • Decode the existing token IDs into readable text.
  • Compare the decoded sentence with normalized source text.
Add the decoder and comparison

Encoding transforms text into IDs that the tokenizer can process. Decoding follows the same path in reverse.

  • In token_benchmark.py, locate the existing print call for bert_encoding.ids.
  • Find the next print call for bert_tokenizer.convert_ids_to_tokens(bert_encoding.ids).
  • Paste this code directly below the token-pieces print call:
# Decode the IDs into readable text
decoded_sentence = bert_tokenizer.decode(bert_encoding.ids)
print("Decoded text:", decoded_sentence)

# Compare text after removing whitespace differences
print(
    "Round-trip matches:",
    " ".join(decoded_sentence.split()) == " ".join(sample_sentence.split()),
)

What does this code do?

  • bert_tokenizer.decode(bert_encoding.ids) reconstructs readable text from the existing ID list.
  • decoded_sentence stores the reconstructed text for the comparison.
  • split() separates each sentence into words without preserving its current spacing.
  • " ".join(...) rebuilds each sentence with consistent spacing before the equality check.
Verify the round trip

The equality check gives you a stricter signal than reading the decoded sentence by eye. It reveals any text information that changed during encoding.

  • Save token_benchmark.py.

Before you run the script, pause to predict whether the two normalized sentences match.

  • Run the updated script from the terminal in your tokenizer-benchmark folder by running this command:
python token_benchmark.py

You will see a readable decoded sentence. You will also see Round-trip matches: False.

That is the round trip working. Your token IDs now rebuild readable language.

The comparison returns False because bert-base-uncased lowercases its input. Whitespace normalization leaves letter case unchanged.

Decoded text missing?

Check that the decoding code sits below bert_encoding = bert_tokenizer.encode(sample_sentence). The decoder needs those IDs to exist first.

Check that every decoder reference uses bert_encoding.ids. A typo can stop Python before it prints the decoded text.

Help me debug the decoding section in my tokenizer script.

✔️ Awesome, I've got everything!

Your saved script now encodes the sentence before decoding the same token IDs. The terminal also shows the normalized round-trip comparison.

ⓧ I'd like to double check the full code

Compare your token_benchmark.py file with the complete script below.

from tokenizers import Tokenizer

# Keep one sentence constant across every check
sample_sentence = "Tokenizers split text into billable pieces."

# Load the pretrained tokenizer once
bert_tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# Encode the sentence into IDs
bert_encoding = bert_tokenizer.encode(sample_sentence)
print("Token IDs:", bert_encoding.ids)
print(
    "Token pieces:",
    bert_tokenizer.convert_ids_to_tokens(bert_encoding.ids),
)

# Decode the IDs into readable text
decoded_sentence = bert_tokenizer.decode(bert_encoding.ids)
print("Decoded text:", decoded_sentence)

# Compare text after removing whitespace differences
print(
    "Round-trip matches:",
    " ".join(decoded_sentence.split()) == " ".join(sample_sentence.split()),
)

Your script now shows the full trip from text to IDs to readable text. Next up, you will compare one paragraph across three tokenizers to see how model choice changes the count.

Compare Three Tokenizers

Your BERT round trip now proves that real token IDs can reconstruct readable text. The next question is whether another model divides identical text the same way.

A single count only describes one tokenizer. This step measures the same paragraph with GPT-2 before repeating the comparison with RoBERTa.

In this step, get ready to:
  • Prepare one paragraph for all three tokenizers.
  • Load three pretrained tokenizers into one mapping.
  • Print each tokenizer's count for comparison.
Load three tokenizers

A fair comparison keeps the input fixed while changing the tokenizer. A mapping gives each tokenizer a name that the counting loop can use.

  • Switch back to token_benchmark.py in your code editor.
  • Add the following code below the round-trip comparison:
# Use one shared paragraph so every tokenizer receives identical text
paragraph = "Every prompt has a cost because models process text as tokens. Different tokenizers divide the same words differently, so measuring your own text helps you estimate usage before you send it."

# Group the pretrained tokenizers under readable model names
tokenizers = {
    "bert-base-uncased": bert_tokenizer,
    "gpt2": Tokenizer.from_pretrained("gpt2"),
    "roberta-base": Tokenizer.from_pretrained("roberta-base"),
}

What does this code do?

  • The paragraph variable gives every tokenizer identical text. This isolates the effect of the tokenizer.
  • The tokenizers mapping connects each model name to its loaded tokenizer.
  • The mapping reuses bert_tokenizer. This avoids loading BERT twice.
  • Save token_benchmark.py.

A short pause on the first run is expected while the two new pretrained tokenizers become available locally.

  • Rerun token_benchmark.py with the same terminal command from earlier.

You should still see the BERT token IDs plus the decoded sentence. Your terminal should return to its prompt without a loading failure.

Tokenizer load stopped?

Check your internet connection if GPT-2 or RoBERTa is not already available locally. Keep the model names identical to the code above.

If the script still stops, help me troubleshoot the pretrained tokenizer loading step.

Count each tokenizer's output

Each tokenizer uses its own vocabulary plus splitting rules. Counting the resulting IDs makes those model-specific choices visible.

  • Add the counting loop below the tokenizers mapping:
# Store each count under the tokenizer name for side-by-side comparison
token_counts = {}
for name, tokenizer in tokenizers.items():
    encoding = tokenizer.encode(paragraph)
    token_counts[name] = len(encoding.ids)
    print(f"{name}: {token_counts[name]} tokens")

What does this code do?

  • The loop sends the same paragraph through every tokenizer in the mapping.
  • Each encoding.ids list holds the integer IDs created by one tokenizer.
  • The token_counts mapping stores each result under its tokenizer name.
  • The formatted print statement displays one count line per tokenizer.
  • Save token_benchmark.py.

Before you rerun the script, which tokenizer do you think will produce the highest count for the same paragraph?

  • Rerun token_benchmark.py with the same terminal command from earlier.

You'll see three count lines. Their labels are bert-base-uncased, gpt2, and roberta-base.

Each line ends with a token count.

That's your first cross-model comparison: one paragraph now has three measured counts.

Missing one or more counts?

Check that token_counts = {} starts at the left edge of the file. Keep every line inside the for loop indented by four spaces.

If the output still looks incomplete, help me debug the tokenizer counting loop.

✔️ Awesome, I've got everything!

Your saved script now prints comparable counts for all three tokenizers.

ⓧ I'd like to double check the full code

Your token_benchmark.py file should match this cumulative version.

from tokenizers import Tokenizer

# Inspect how BERT turns a short sentence into IDs and pieces
sample_sentence = "Tokenizers split text into billable pieces."
bert_tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
bert_encoding = bert_tokenizer.encode(sample_sentence)
print("Token IDs:", bert_encoding.ids)
print("Token pieces:", bert_tokenizer.convert_ids_to_tokens(bert_encoding.ids))

# Decode the IDs and normalize whitespace for the uncased round-trip check
decoded_sentence = bert_tokenizer.decode(bert_encoding.ids)
expected_sentence = sample_sentence.lower()
normalized_expected = " ".join(expected_sentence.split())
normalized_decoded = " ".join(decoded_sentence.split())
print("Decoded text:", decoded_sentence)
print("Round trip matches:", normalized_decoded == normalized_expected)

# Use one shared paragraph so every tokenizer receives identical text
paragraph = "Every prompt has a cost because models process text as tokens. Different tokenizers divide the same words differently, so measuring your own text helps you estimate usage before you send it."

# Group the pretrained tokenizers under readable model names
tokenizers = {
    "bert-base-uncased": bert_tokenizer,
    "gpt2": Tokenizer.from_pretrained("gpt2"),
    "roberta-base": Tokenizer.from_pretrained("roberta-base"),
}

# Store each count under the tokenizer name for side-by-side comparison
token_counts = {}
for name, tokenizer in tokenizers.items():
    encoding = tokenizer.encode(paragraph)
    token_counts[name] = len(encoding.ids)
    print(f"{name}: {token_counts[name]} tokens")

Your comparison now shows how model choice changes prompt size. Next, you'll benchmark a large text file to calculate token throughput per second.

Benchmark a Large Text File

Your three-tokenizer comparison showed how model families split one paragraph. A performance benchmark needs enough text for the timer to capture meaningful work.

Now you will build a repeatable UTF-8 input with 10,000 copies of that paragraph. The script will measure BERT encoding throughput in tokens per second.

In this step, get ready to:
  • Create a large text file from 10,000 repetitions of your comparison paragraph.
  • Add a timed BERT encoding pass to token_benchmark.py.
  • Calculate BERT encoding throughput from the measured token total.
Create the large benchmark file

A larger input keeps the tokenizer busy long enough to produce a useful duration. Repeating the same paragraph also makes the test easy to reproduce.

  • Create large_text.txt inside tokenizer-benchmark by running this command:
python -c "paragraph = 'Every prompt has a cost because models process text as tokens. Different tokenizers divide the same words differently, so measuring your own text helps you estimate usage before you send it.'; open('large_text.txt', 'w', encoding='utf-8').write((paragraph + '\n') * 10000)"

What does this command do?

  • The command runs a short Python statement without creating another script.
  • The multiplication repeats the paragraph 10,000 times.
  • The encoding setting writes every repetition to large_text.txt as UTF-8 text.
  • Confirm the file contains 10,000 repetitions by running this check:
python -c "text = open('large_text.txt', encoding='utf-8').read(); print(text.count('Every prompt has a cost because models process text as tokens.'))"

What does this check measure?

The opening sentence appears once in each repetition. Counting that sentence verifies the size of the benchmark input.

You should see 10000 in the terminal. Your repeatable benchmark input is ready.

Do you see a different count?

Make sure the terminal is still inside tokenizer-benchmark. Run the creation command again if an older large_text.txt was already in that folder.

Ask for help with the file count.

Add the timing benchmark

The benchmark records the clock immediately before BERT encodes the file. The difference after encoding becomes elapsed_seconds.

  • Switch back to token_benchmark.py in your code editor.
  • Scroll to the end of the existing three-tokenizer comparison.
  • Paste this benchmark section at the bottom of the file:
import time

# Read the large benchmark input from disk
with open("large_text.txt", "r", encoding="utf-8") as file:
    large_text = file.read()

# Measure one BERT encoding pass
start_time = time.perf_counter()
large_encoding = bert_tokenizer.encode(large_text)
elapsed_seconds = time.perf_counter() - start_time

# Calculate throughput from the measured result
large_token_count = len(large_encoding.ids)
tokens_per_second = large_token_count / elapsed_seconds

print(f"\nLarge file token total: {large_token_count:,}")
print(f"Elapsed seconds: {elapsed_seconds:.4f}")
print(f"Tokens per second: {tokens_per_second:,.0f}")

What does this code do?

  • The file-reading block loads the complete contents of large_text.txt into large_text.
  • The two calls to time.perf_counter() surround the BERT encoding pass.
  • The length of large_encoding.ids becomes large_token_count.
  • Dividing that total by elapsed_seconds produces tokens_per_second.
  • Save token_benchmark.py.
Run the benchmark

Before you run the script, do you expect the large BERT encoding pass to finish in under one second or take longer?

  • Run the completed benchmark by using this command:
python token_benchmark.py

What does this command run?

Python runs the earlier token inspection first. It then reaches the new timing section and encodes the complete benchmark file.

Your earlier inspection output appears first. The benchmark ends with labeled values for the large token total, elapsed seconds, and tokens per second.

That is the measurement loop working. Your script now turns one large input into a concrete BERT throughput result.

Benchmark not printing?

Confirm that large_text.txt sits beside token_benchmark.py inside tokenizer-benchmark. A different file location prevents the script from reading the benchmark input.

Check that the timing section appears after bert_tokenizer is created. Ask for help debugging the benchmark.

✔️ Awesome, I've got everything!

Great. Make sure token_benchmark.py is saved and the terminal shows all three benchmark values.

ⓧ I'd like to double check the full code

Compare your complete token_benchmark.py with this version.

from tokenizers import Tokenizer

# Load BERT and inspect how it splits a short sentence
sample_text = "every prompt has a cost."
bert_tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
sample_encoding = bert_tokenizer.encode(sample_text)

print("Token IDs:", sample_encoding.ids)
print("Token pieces:", sample_encoding.tokens)

# Decode the token IDs to verify the round trip
sample_decoded = bert_tokenizer.decode(sample_encoding.ids)
print("Decoded text:", sample_decoded)
print("Round trip matches:", sample_decoded == sample_text)

# Compare one paragraph across three model tokenizers
paragraph = "Every prompt has a cost because models process text as tokens. Different tokenizers divide the same words differently, so measuring your own text helps you estimate usage before you send it."
tokenizers = {
    "bert-base-uncased": bert_tokenizer,
    "gpt2": Tokenizer.from_pretrained("gpt2"),
    "roberta-base": Tokenizer.from_pretrained("roberta-base"),
}

token_counts = {}
for name, tokenizer in tokenizers.items():
    paragraph_encoding = tokenizer.encode(paragraph)
    token_counts[name] = len(paragraph_encoding.ids)
    print(f"{name}: {token_counts[name]} tokens")

import time

# Read the large benchmark input from disk
with open("large_text.txt", "r", encoding="utf-8") as file:
    large_text = file.read()

# Measure one BERT encoding pass
start_time = time.perf_counter()
large_encoding = bert_tokenizer.encode(large_text)
elapsed_seconds = time.perf_counter() - start_time

# Calculate throughput from the measured result
large_token_count = len(large_encoding.ids)
tokens_per_second = large_token_count / elapsed_seconds

print(f"\nLarge file token total: {large_token_count:,}")
print(f"Elapsed seconds: {elapsed_seconds:.4f}")
print(f"Tokens per second: {tokens_per_second:,.0f}")

Your benchmark now measures how quickly BERT processes a large input. Next up, you will turn the tokenizer counts into a chart and compare prompt costs.

Chart Token Costs

Your script now times BERT against the large benchmark file. It also compares one paragraph across three tokenizers.

Rows of terminal output make the three counts difficult to compare at a glance. Matplotlib turns them into a saved chart.

Illustrative price assumptions then convert each number of tokens into a USD estimate. The same paragraph now has a visible cost difference.

In this step, get ready to:
  • Turn token_counts into a labeled bar chart.
  • Save the chart as token_counts.png.
  • Calculate paragraph costs with illustrative input-price assumptions.
Create the token-count chart

A bar chart gives every tokenizer the same visual baseline. Taller bars immediately reveal which tokenizer produces more billable units for the paragraph.

  • In token_benchmark.py, locate the final benchmark output.
  • Add the plotting section below that output by pasting this code:
# Load plotting tools for the final comparison
import matplotlib.pyplot as plt

# Plot one bar for each measured tokenizer count
tokenizer_names = list(token_counts.keys())
counts = list(token_counts.values())
plt.bar(tokenizer_names, counts)
plt.title("Token counts for the same paragraph")
plt.xlabel("Tokenizer")
plt.ylabel("Tokens")
plt.savefig("token_counts.png")

What does this chart code do?

  • The matplotlib.pyplot import provides the plotting functions used by the chart.
  • The tokenizer_names list takes the three labels from token_counts.
  • The counts list supplies the height of each bar.
  • The final line writes the finished chart to token_counts.png.
  • Save token_benchmark.py.

Before you run the script, which tokenizer do you expect to have the tallest bar?

  • Generate the chart by running this command:
python token_benchmark.py

What does this command do?

Python runs token_benchmark.py from top to bottom. The final plotting section uses the counts collected earlier to write the PNG image.

You should see your earlier inspection output followed by the large-file benchmark. You should also see token_counts.png in the tokenizer-benchmark folder.

  • Select token_counts.png in your code editor's file sidebar.

You should see three labeled bars under the title Token counts for the same paragraph. The vertical axis shows the token count.

There it is. Your tokenizer comparison is now an image you can inspect or share.

Chart file missing?

Confirm the plotting section sits after the code that fills token_counts. Check the terminal for a Python error that points to the line that stopped the script.

Make sure you are viewing the tokenizer-benchmark folder in your code editor.

help me debug why token_counts.png was not created

Calculate illustrative token costs

A token count becomes a cost estimate when you divide it by one million. Multiplying that result by an input price gives the estimated paragraph cost.

How should you read these prices?

The prices in this exercise are illustrative USD input-token assumptions. They provide a consistent scenario for comparing the measured counts.

Live vendor pricing sits outside this local benchmark. Replace the assumptions when you evaluate a real service.

  • Add the pricing section below the chart code by pasting this code:
# Store illustrative USD input prices per one million tokens
pricing_per_million = {
    "bert-base-uncased": 1.00,
    "gpt2": 1.50,
    "roberta-base": 1.25,
}

# Estimate the paragraph cost from each measured token count
print("\nIllustrative input price assumptions and paragraph estimates:")
for name, token_count in token_counts.items():
    price = pricing_per_million[name]
    estimated_cost = token_count / 1_000_000 * price
    print(
        f"{name}: {token_count} tokens | "
        f"${price:.2f} per 1M input tokens | "
        f"estimated cost ${estimated_cost:.8f}"
    )

How does the estimate work?

  • The pricing_per_million dictionary pairs each tokenizer with its illustrative USD assumption.
  • The expression token_count / 1_000_000 * price scales the paragraph count to the price for one million input tokens.
  • The final printout places the measured count beside its assumption and estimated cost.
  • Save token_benchmark.py.

Before you run the final version, which tokenizer do you expect to produce the largest estimated cost?

  • Calculate the three estimates by running this command:
python token_benchmark.py

What does the final run prove?

The script uses one paragraph for all three calculations. Any difference in the estimates comes from the tokenizer count and the illustrative price assigned to that tokenizer.

You should see a pricing section with one line for bert-base-uncased, one for gpt2, and one for roberta-base.

Each line should show the measured token count. It should also show the per-million assumption and the estimated paragraph cost.

  • Return to token_counts.png in your code editor.

You should still see all three labeled bars. This confirms the final script preserves the chart while adding the cost calculations.

Cost output missing?

Check that the pricing section appears after the chart section in token_benchmark.py. Confirm each key in pricing_per_million matches a key in token_counts.

help me debug the missing token cost estimates

✔️ Awesome, I've got everything!

Your saved script now produces the chart and the three illustrative cost estimates. Make sure token_benchmark.py is saved.

ⓧ I'd like to double check the full code

Compare your complete token_benchmark.py file with this version.

from tokenizers import Tokenizer
import time

# Load BERT and inspect how it represents a short sentence
bert_tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
sample_text = "tokenizers split text into pieces."
sample_encoding = bert_tokenizer.encode(sample_text)
print("Token IDs:", sample_encoding.ids)
print("Token pieces:", sample_encoding.tokens)

# Decode the token IDs to verify the round trip
 decoded_text = bert_tokenizer.decode(sample_encoding.ids)
print("Decoded text:", decoded_text)
print("Round trip matches:", decoded_text == sample_text)

# Load three tokenizers for the paragraph comparison
tokenizers = {
    "bert-base-uncased": bert_tokenizer,
    "gpt2": Tokenizer.from_pretrained("gpt2"),
    "roberta-base": Tokenizer.from_pretrained("roberta-base"),
}

# Use the same paragraph so each count is directly comparable
paragraph = (
    "Every prompt has a cost because models process text as tokens. "
    "Different tokenizers divide the same words differently, so measuring "
    "your own text helps you estimate usage before you send it."
)
token_counts = {}
for name, tokenizer in tokenizers.items():
    encoding = tokenizer.encode(paragraph)
    token_counts[name] = len(encoding.ids)
    print(f"{name}: {token_counts[name]} tokens")

# Read the larger input used for the throughput benchmark
with open("large_text.txt", "r", encoding="utf-8") as file:
    large_text = file.read()

# Time BERT while it encodes the large file
start_time = time.perf_counter()
large_encoding = bert_tokenizer.encode(large_text)
elapsed_seconds = time.perf_counter() - start_time
large_token_count = len(large_encoding.ids)
tokens_per_second = large_token_count / elapsed_seconds
print(f"Large file tokens: {large_token_count}")
print(f"Elapsed seconds: {elapsed_seconds:.4f}")
print(f"Tokens per second: {tokens_per_second:,.0f}")

# Load plotting tools for the final comparison
import matplotlib.pyplot as plt

# Plot one bar for each measured tokenizer count
tokenizer_names = list(token_counts.keys())
counts = list(token_counts.values())
plt.bar(tokenizer_names, counts)
plt.title("Token counts for the same paragraph")
plt.xlabel("Tokenizer")
plt.ylabel("Tokens")
plt.savefig("token_counts.png")

# Store illustrative USD input prices per one million tokens
pricing_per_million = {
    "bert-base-uncased": 1.00,
    "gpt2": 1.50,
    "roberta-base": 1.25,
}

# Estimate the paragraph cost from each measured token count
print("\nIllustrative input price assumptions and paragraph estimates:")
for name, token_count in token_counts.items():
    price = pricing_per_million[name]
    estimated_cost = token_count / 1_000_000 * price
    print(
        f"{name}: {token_count} tokens | "
        f"${price:.2f} per 1M input tokens | "
        f"estimated cost ${estimated_cost:.8f}"
    )

You have turned an invisible billing unit into a chart and a repeatable cost calculation. Your benchmark now shows how tokenizer choice changes both the count and the estimate.

Secret mission

Recommend a Tokenizer at Scale

Your chart shows the token difference for one paragraph. Scale the results to 100,000 prompts. Then defend the lowest-cost choice under the script's illustrative prices.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project runs locally, so keeping the files creates no ongoing cost. Choose whether to keep the tokenizer-benchmark folder, pause your work, or delete it entirely.

Resources you used:

  • Local tokenizer-benchmark folder containing your Python script, benchmark text, bar chart, and virtual environment.

Keep everything running

No action needed. Choose this if you want to rerun the benchmark or compare more text later.

  • Keep token_benchmark.py so you can run more tokenizer comparisons.
  • Keep token_counts.png as a visual record of the token count differences.
  • Keep large_text.txt for future performance benchmarks.
  • Keep .venv so the installed project packages remain ready to use.

Pause - I'll come back to this later

Stop active processes to free up local resources while keeping every project file intact.

  • Stop any running Python script by pressing Ctrl+C in the terminal.
  • Close the terminal window to end the active .venv session.
  • Close the code editor window when you finish reviewing the project.
  • Leave the tokenizer-benchmark folder unchanged for your next session.

Delete - I don't want to use this again

Deleting the folder permanently removes your script, benchmark text, bar chart, and virtual environment. The commands below target only the tokenizer-benchmark folder from its parent directory.

  • Copy token_counts.png to another folder if you want to keep the chart.
  • Stop any running Python script by pressing Ctrl+C in the terminal.
  • Close the code editor window that has tokenizer-benchmark open.

macOS

  • Delete the project folder and display its parent folder's contents by running these commands:
cd ..
rm -rf tokenizer-benchmark
ls
What do these commands do?
  • The first line moves the terminal out of the project folder.
  • The second line permanently removes the project folder and everything inside it.
  • The final line displays the remaining contents of the parent folder.

You should see the parent folder's contents. The tokenizer-benchmark folder should be absent.

Folder still listed?
  • Close any editor window that still has a file from tokenizer-benchmark open.
  • Move to the parent folder before retrying the deletion command.
  • Help me safely remove my local tokenizer-benchmark folder on macOS.

Windows

  • Delete the project folder and display its parent folder's contents by running these commands:
cd ..
rmdir /s /q tokenizer-benchmark
dir
What do these commands do?
  • The first line moves the terminal out of the project folder.
  • The second line permanently removes the project folder and everything inside it.
  • The final line displays the remaining contents of the parent folder.

You should see the parent folder's contents. The tokenizer-benchmark folder should be absent.

Folder still listed?
  • Close any editor window that still has a file from tokenizer-benchmark open.
  • Move to the parent folder before retrying the deletion command.
  • Help me safely remove my local tokenizer-benchmark folder on Windows.

Nice Work!

Nice Work!

You did it! You built a local Python benchmark that makes tokenization visible through token pieces, round-trip decoding, measured throughput, cost estimates, and a saved chart.

You've learned how to:

  • Inspect token IDs and pieces with the Hugging Face tokenizers library. You also decoded the IDs to confirm that they represented the original text.
  • Compare the same paragraph across three pretrained tokenizers. You also measured BERT's token-processing speed with a large local text file.
  • Turn token counts into a labeled bar chart with Matplotlib. You also translated each count into an estimated input cost using clearly labeled pricing assumptions.
  • Completed an optional Secret Mission to push your token analysis skills further.

Ready to quiz yourself?