Measure Your Own Word Error Rate

August 26, 2026 - Dictate a passage and compute your own Word Error Rate.

Introduction

30 Second Summary

Speech recognition models ship with accuracy numbers measured on standardized benchmarks. Your microphone, your room, and your vocabulary are not a standardized benchmark.

In this project, you will dictate a passage into the Antigravity CLI, capture the raw transcript, and compute your personal Word Error Rate with a Python script. You will compare your result against the published 4.0% streaming benchmark.

What You'll Build

You run a single Python command and see your WER print alongside the published 4.0%, with a word-by-word alignment marking every substitution, insertion, and deletion so you can see exactly where the model stumbled.

By the end of this project, you'll have:

A sealed reference passage with domain jargon that you wrote and closed before any audio existed, preventing the most common methodological error.
A personal WER measurement from one unrehearsed reading on your own microphone, printed beside the published 4.0% streaming benchmark.
A word-level alignment printed by your Python script, tagging every word as correct, substituted, inserted, or deleted.
Secret Mission: Read the same passage a second time to check repeatability, then dictate a jargon-free passage and watch the rate move.

Are there any prerequisites?

None. This project includes full setup instructions for the Antigravity CLI, Python, and every other tool you need.

Before We Start

Before we install anything, let's commit to the exact measurement you're about to make and why your number matters more than the published one. Google's benchmark says 4.0% streaming Word Error Rate for Gemini 3.5 Transcribe, but that number came from a standardized test, not your microphone, your room, or your voice.

Install the Antigravity CLI and Python

Before you measure anything, you need the tools that produce the transcript and compute the score.

The Antigravity CLI handles the dictation, capturing your voice and returning a transcript. Python and a scoring library called jiwer handle the alignment math that turns two text files into a single percentage.

In this step, get ready to:

Install the Antigravity CLI and verify it runs.
Install Python and the jiwer scoring package.
Sign in to Antigravity with your Google account.

Install the Antigravity CLI

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type Terminal (macOS/Linux) or PowerShell (Windows) and press Enter to open it.
Install the Antigravity CLI by running the installer for your operating system:

macOS

curl -fsSL https://antigravity.google/cli/install.sh | bash

Linux

curl -fsSL https://antigravity.google/cli/install.sh | bash

Windows

irm https://antigravity.google/cli/install.ps1 | iex

What does this command do?

The installer downloads the agy binary to your machine and places it on your system PATH so you can run it from any directory.

Confirm the install succeeded by running:

agy --version

✔️ I see a version number

The Antigravity CLI is installed and ready. Move on to the next substep.

ⓧ Command not found

The installer may not have added agy to your PATH automatically.

Close your terminal completely and open a new one.
Run agy --version again.
If it still fails, check the installer output for the directory where agy was placed and add that directory to your PATH manually.

Still stuck after restarting your terminal?

Make sure you ran the correct installer for your operating system. On macOS and Linux, the command starts with curl. On Windows, it starts with irm in PowerShell, not Command Prompt.

Help me add agy to my PATH after installing the Antigravity CLI.

Install Python and jiwer

Python runs the scoring script that computes your rate. The jiwer library does the word-by-word alignment, counting every substitution, insertion, and deletion.

Check whether Python is already installed by running:

macOS

python3 --version

Linux

python3 --version

Windows

python --version

✔️ I see Python 3.x

Python is ready. Skip ahead to the jiwer install below.

ⓧ Command not found

Download Python from the official downloads page.
Run the installer.
On Windows, check the Add Python to PATH box before clicking Install Now.
Close your terminal, open a new one, and run the version check again.

Install the jiwer package by running:

pip install jiwer

You should see output ending with a line that confirms the package was installed along with a version number.

What is jiwer?

jiwer is a Python library that computes Word Error Rate. It aligns your transcript against your reference word by word, then counts every word that was substituted, inserted, or deleted.

You will use it in the final step to turn two text files into a single percentage.

Seeing an install error?

If you see a message about an externally managed environment, you may need to create a virtual environment first. This is common on newer macOS and Linux systems.

If pip is not found, try pip3 instead. On macOS and Linux, the pip3 command is often the default for Python 3.

Help me install the jiwer Python package on my system.

Sign in to Antigravity

The CLI needs your Google account to reach the transcription service. The sign-in flow opens your browser and gives you a code to paste back into the terminal.

Launch the Antigravity CLI by running:

agy

What happens when you run this?

Because this is your first time running agy, it detects that you are not signed in. A browser window opens automatically for Google sign-in.

After you authorize access, the browser shows an authorization code for you to copy and paste back into the terminal.

Sign in with your Google account in the browser window that opens.
Copy the authorization code shown after sign-in.
Return to your terminal and paste the code.
Press Enter to complete sign-in.

Before you press Enter, what do you expect to see once the CLI authenticates?

You should see the interactive agy prompt appear. Look for a model name displayed in the bottom right corner. This confirms the CLI is authenticated and connected.

Not seeing the interactive prompt?

Make sure you copied the full authorization code from the browser. If the browser did not open automatically, check the terminal output for a URL you can open manually.

Help me complete the Antigravity CLI sign-in with my Google account.

Your tools are installed and authenticated. Next, you will write the reference passage and commit your predictions before any audio exists.

Write the Reference and Your Predictions

Your Antigravity CLI is installed and authenticated, and Python is ready with jiwer. Before any audio touches a microphone, you need two things written down: the exact words you are going to read aloud, and three predictions about what will happen when you do.

Writing the reference first prevents the most common methodological error in speech evaluation. If you write or edit your reference after hearing the transcript, you will unconsciously match the two texts and silently lower your WER.

In this step, get ready to:

Write a reference passage with domain jargon in reference.txt.
Record three predictions in predictions.txt.
Seal the reference before any audio exists.

Write your reference passage

The passage you write now becomes the ground truth for every number your script produces. Domain jargon, project names, and acronyms from your own work are the words transcription models have never encountered in their training data.

Jargon is where transcription fails. Including it tests where the model actually struggles, not where it has been optimized.

Open a new terminal tab by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows).
Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir wer-project
cd wer-project

Open any text editor you are comfortable with.
Create a new file and save it as reference.txt inside your wer-project folder on the Desktop.
Write roughly 80 to 100 words of natural prose.
Include two or three terms from your own work: project names, library names, acronyms, or technical jargon.
Save the file and close it.

Why include domain jargon?

Benchmark passages use common words that models have seen millions of times during training. Your jargon has never appeared in their training corpus.

Including terms the model has never seen is what makes your measurement different from the published benchmark. The 4.0% streaming WER was measured on a standard corpus, not on your vocabulary.

Confirm the file exists by running:

ls

You should see reference.txt in the output.

Don't see reference.txt?

Make sure you saved the file inside the wer-project folder on your Desktop, not in a different location.
Check the filename is exactly reference.txt with no extra extension like .txt.txt. Some editors on Windows add a second extension if the file type dropdown is not set to All Files.
Help me find my reference.txt file

Record your three predictions

Predictions written before the experiment force you to commit to specific expectations. Without them, any result feels like "about what I expected."

Create a new file in your text editor and save it as predictions.txt inside your wer-project folder.
Write three predictions, each on its own line:

Does /voice dictation work without configuring an API key? Write yes or no.
Can the tool reach your microphone on the first attempt? Write yes or no.
How close to 4.0% will your WER land? Write a specific number.

Leave space next to each prediction for recording your observation later.
Save and close predictions.txt.

Why write predictions before the experiment?

If you run the experiment first and predict after, your predictions will match the result every time. Writing them before the data exists is the only way to test whether you actually understood what was about to happen.

Confirm both files exist by running:

ls

You should see both reference.txt and predictions.txt listed in the output.

Missing a file?

Some text editors save files in a default Documents folder instead of the folder you specified. Check the file path in your editor's title bar or save dialog.
On Windows, make sure you selected All Files in the save dialog's file type dropdown to avoid a double extension.
Help me find my missing file

Seal the reference

From this point forward, the reference is sealed. You will not open, read, or edit reference.txt again until score.py reads it in the final step.

Close or minimize reference.txt if it is still open in your editor.
Close predictions.txt as well.

Why does sealing matter?

If you glance at the reference while reviewing the transcript later, you will unconsciously correct small differences. That contamination makes your WER look better than it actually is.

Sealing the file before any audio exists is how you keep the measurement honest. The next time these words appear is when your Python script opens them.

Your reference and predictions are locked in. Next up, you will try the Antigravity CLI's /voice dictation cold and find out whether your first two predictions were right.

Try Dictation Cold

Your reference passage is sealed and your three predictions are written down. The experimental controls are in place.

Configuring an API key before trying /voice would guarantee a working run and destroy the evidence about whether a key was needed. The first-run experience, with nothing configured, is itself a data point.

In this step, get ready to:

Try /voice dictation with no prior voice configuration.
Resolve any credential or microphone issues.
Record observations next to predictions 1 and 2 in predictions.txt.

Try /voice with nothing configured

You already wrote a prediction about whether /voice would need an API key. Now you find out.

Open your terminal.
Navigate to the folder containing reference.txt and predictions.txt.
Start agy by running this command:

agy

Type /voice in the prompt box and press Enter. You can also press F5 as a shortcut.
Watch the status line below the prompt.

What should I see?

If /voice connects successfully, the status line transitions from Connecting... to Recording and shows elapsed time.

If it fails, you may see a permissions message or a credentials prompt instead. Both are valid observations.

✔️ The status line says Recording

Dictation started without any additional API key or configuration. That is your first observation.

Press Esc to discard this test recording.
Open predictions.txt in a text editor.
Write your observation next to prediction 1 (for example, "No API key needed, Recording started immediately").
Save and close predictions.txt.

Skip the next substep and jump straight to "Confirm your microphone is working" below.

ⓧ I see an error or a credentials prompt

Your stored credentials may predate voice dictation support. The next substep walks you through the fix.

Resolve credential issues

If /voice reported a permissions error, your stored credentials need to be refreshed. Signing out and back in grants the voice dictation permission.

Type /logout in the prompt box and press Enter.
Complete the Google sign-in flow in your browser.
Paste the code back into the terminal when prompted.
Type /voice and press Enter.
Confirm the status line transitions from Connecting... to Recording.

Is a billing account or card required?

If the sign-in process demands a billing account or credit card, stop and record that in your notes. The free tier appears to require only a Google account sign-in, but this is based on the tier structure and has not been independently confirmed.

Press Esc to discard this test recording.
Open predictions.txt in a text editor.
Write your observation next to prediction 1.
Save and close predictions.txt.

Still seeing a permissions error?

Dictation is not currently supported for business or enterprise accounts. If you are signed in with a business or enterprise Google account, try signing out with /logout and signing back in with a personal Google account.

Why can't I get /voice dictation working after signing in again?

Confirm your microphone is working

The status line says Recording, so the CLI is listening. The next question is whether it can hear you.

Start a new recording by pressing F5 or typing /voice.
Speak a few words clearly into your microphone.
Watch the prompt box for interim transcript text to appear.

✔️ Transcript text appears in the prompt box

Your microphone is working and dictation is transcribing your speech.

Press Enter or F5 to stop recording.
Open predictions.txt.
Write your observation next to prediction 2 (for example, "Microphone worked on first attempt").
Save and close predictions.txt.

ⓧ No text appears when I speak

Your operating system may not have granted microphone access to your terminal app. Press Esc to discard the recording, then grant microphone permission.

macOS

Open System Settings.
Click Privacy & Security.
Click Microphone.
Enable access for your terminal app (for example, Terminal or iTerm).
Quit and reopen your terminal with Cmd+Q to apply the updated permission.

Windows

Open Settings.
Click Privacy & Security.
Click Microphone.
Confirm your terminal app has access and that Let desktop apps access your microphone is enabled.
Close and reopen your terminal to apply the updated permission.

Linux

Check your PulseAudio or PipeWire device settings.
Verify your terminal app has microphone permissions.
Close and reopen your terminal to apply any changes.

After granting permission and reopening your terminal, try again.

Navigate to the folder containing your project files.
Start agy by running this command:

agy

Press F5 or type /voice and press Enter.
Speak a few words into your microphone.
Confirm transcript text now appears in the prompt box.
Press Enter or F5 to stop recording.

Open predictions.txt.
Write your observation next to prediction 2.
Save and close predictions.txt.

Still no transcript after granting permission?

Make sure you fully quit your terminal (not just close the tab) and reopened it. On macOS, use Cmd+Q to ensure the process restarts with the new permission.

Also check that your system default input device is set to the correct microphone under your sound settings.

Why isn't my microphone working with agy /voice dictation?

Before moving on, confirm that predictions.txt has observations written next to both prediction 1 and prediction 2. Prediction 3 (your WER estimate) should still be blank.

Dictation is working and your first two observations are recorded. Next up, you read your sealed reference passage aloud once and capture the raw transcript.

Read the Passage Once

Dictation is live and your microphone is connected. Now you capture the measurement the rest of this project is built on.

The rule for this step is one take. You read your reference passage once, unrehearsed, at your natural pace. Re-recording until the number improves would measure your patience, not the model.

In this step, get ready to:

Read the reference passage aloud using /voice dictation in agy.
Save the raw, unedited transcript to hypothesis.txt.

Read the passage aloud in one take

Your reference passage has been sealed since you wrote it. Now you open it to read from while the Antigravity CLI transcribes your voice in real time.

Open reference.txt in a separate window, or print it out. You need to be able to read from it while agy is recording.
Switch to agy in your terminal.
Press F5 or type /voice and press Enter.
Wait for the status line below the prompt to transition from Connecting... to Recording.
Read the full passage aloud at your natural pace.
Press Enter to stop recording.

Interim text appears in the prompt box as you speak and updates as the transcription processes your audio. After you stop, the full transcript stays in the prompt box.

Why only one take?

Multiple takes let you select the reading that happened to match your reference best. That measures your ability to adapt to the model, not the model's accuracy with your voice.

One unrehearsed reading under your real conditions is the only number nobody can argue you out of.

The transcript does not automatically submit your prompt. It stays in the prompt box for you to copy.

Transcript not appearing?

Make sure the status line showed Recording before you started speaking. If it still says Connecting..., wait for it to transition.
Check that your microphone permission is still granted from the previous step. Restart agy if the permission changed.
Help me troubleshoot why my /voice transcript is not appearing.

Save the transcript to hypothesis.txt

The text in the prompt box is your raw measurement. You need to capture it in a file exactly as the model produced it.

Select all the transcript text in the Antigravity prompt box.
Copy it (Cmd+C on macOS or Ctrl+C on Windows).
Create a new file called hypothesis.txt in your project folder.
Paste the transcript into hypothesis.txt.
Save the file exactly as it appeared. Do not correct spelling, do not fix punctuation, do not rearrange words.

Why save without editing?

Fixing even one misspelled word or adding missing punctuation silently lowers your WER. The scoring script in the next step needs the exact output the model produced, not a cleaned-up version.

Confirm the file exists by running:

ls

You should see hypothesis.txt alongside reference.txt and predictions.txt.

File looks empty?

Return to agy and check whether the transcript is still in the prompt box. If it is, select and copy it again.
Make sure you pasted into hypothesis.txt and saved the file after pasting.
Help me figure out why my hypothesis.txt is empty.

Your transcript is locked in. Next, you'll build the Python script that turns this raw text into a WER number and shows you exactly where the errors landed.

Compute and Verify the Rate

You have a transcript in hypothesis.txt that looks roughly correct. But "roughly correct" is not a measurement.

Word Error Rate counts every substituted, inserted, and deleted word, then divides by the total words in your reference. In this step, you will build a Python script that turns that impression into a single number and sets it beside the published 4.0% streaming benchmark.

In this step, get ready to:

Write score.py to compute WER from your reference and transcript.
Run the script and read a word-level alignment marking every error.
Record your final prediction observation and check the measurement for contamination.

Create the scoring script

The jiwer library you installed earlier has a function called process_words that does the heavy lifting. It takes your reference text and hypothesis text, computes the minimum-edit-distance alignment, and returns the error counts.

Create a file called score.py in your project folder and paste this code:

from jiwer import process_words

def load_text(path):
    with open(path, encoding="utf-8") as f:
        return " ".join(f.read().split())

def main():
    reference = load_text("reference.txt")
    hypothesis = load_text("hypothesis.txt")

    result = process_words(reference, hypothesis)

    wer_pct = result.wer * 100
    benchmark_pct = 4.0

    print(f"Words in reference:  {len(reference.split())}")
    print(f"Substitutions:       {result.substitutions}")
    print(f"Insertions:          {result.insertions}")
    print(f"Deletions:           {result.deletions}")
    print()
    print(f"Your WER:            {wer_pct:.1f}%")
    print(f"Published benchmark: {benchmark_pct}%")
    print()

if __name__ == "__main__":
    main()

What does this code do?

load_text() reads a file and collapses all whitespace into single spaces. This normalises both texts so that line breaks and extra spaces do not count as errors.

process_words() computes the minimum-edit-distance alignment between your reference and hypothesis. It returns the number of substitutions, insertions, and deletions, plus the overall WER as a decimal.

The script multiplies result.wer by 100 to get a percentage and prints it alongside the published 4.0% benchmark from Gemini 3.5 Transcribe.

Save score.py.
Run the script to confirm it loads your files and computes the stats:

python score.py

You should see your word count, substitution/insertion/deletion counts, your WER percentage, and the published 4.0% benchmark.

Seeing a ModuleNotFoundError?

If the error mentions jiwer, the package is not installed in the Python environment you are using. Run pip install jiwer again and confirm it completes without errors.
Make sure reference.txt and hypothesis.txt are in the same folder as score.py. A FileNotFoundError means the script cannot find one of them.
help me debug a FileNotFoundError or ModuleNotFoundError when running score.py

The stats tell you how many errors there are. The next piece shows you exactly which words went wrong.

In score.py, find the last empty print() line, just above if __name__ == "__main__":. Add this alignment loop between them:

    for chunk in result.alignments[0]:
        ref_words = reference.split()[chunk.ref_start_idx:chunk.ref_end_idx]
        hyp_words = hypothesis.split()[chunk.hyp_start_idx:chunk.hyp_end_idx]

        tag = chunk.type
        if tag == "equal":
            for w in ref_words:
                print(f"  C  {w}")
        elif tag == "substitute":
            for r, h in zip(ref_words, hyp_words):
                print(f"  S  {r}  ->  {h}")
        elif tag == "delete":
            for w in ref_words:
                print(f"  D  {w}  ->  ***")
        elif tag == "insert":
            for w in hyp_words:
                print(f"  I  ***  ->  {w}")

What do the alignment tags mean?

C (correct): the word in your reference matched the transcript exactly.
S (substitution): the model heard a different word than what you said.
D (deletion): a word from your reference is missing in the transcript.
I (insertion): the transcript added a word that is not in your reference.

Save score.py.

✔️ Awesome, I've got everything!

Double check you have saved score.py before moving on.

ⓧ I'd like to double check the full code

Your score.py has two sections you pasted above. Scroll up and confirm both are present, in this order:

The first block: from jiwer import process_words at the top, the load_text function, and main() printing WER stats through the last empty print().
The alignment loop: the for chunk in result.alignments[0]: block with all four tag branches (equal, substitute, delete, insert), sitting inside main() before the if __name__ block.

Make sure the alignment loop is indented inside main() (one level of indentation, matching the print lines above it).

Run the script and read the alignment

Before you run this, think back to prediction 3 in predictions.txt. What WER did you estimate? Do you think your rate will land close to the published 4.0%, or further away?

Run the full script by running this command:

python score.py

You should see your WER percentage, the published 4.0% benchmark, error counts, and a word-level alignment with C, S, I, and D tags running down the left side.

Where did the errors land?

Scan the alignment for S, D, and I tags. Were they on your domain jargon, the project names and acronyms that no benchmark corpus contains? That is the gap between a controlled benchmark and your desk.

If every error is on a jargon term, the model handles common English well but struggles with words it has never seen. If errors land on common words too, your room acoustics or microphone may be a factor.

Seeing a KeyError or empty alignment?

If hypothesis.txt is empty, the script has nothing to align. Confirm the file contains the transcript you pasted from the Antigravity CLI.
Check that the alignment loop is indented with four spaces, matching the other lines inside main(). A wrong indent level causes a syntax error or puts the loop outside the function.
help me debug alignment errors in score.py

Record your final observation and check the method

You now have a measured number. The last step is recording it beside prediction 3 and deciding whether to trust it.

Open predictions.txt.
Next to prediction 3, write your measured WER and the published 4.0% benchmark.
Save and close predictions.txt. All three predictions now have observations beside them.

Is your number trustworthy?

If your WER is far better than 4.0%, suspect contamination. The most common cause is a reference written or edited after seeing the transcript. That silently lowers your WER and makes the measurement useless.

If you sealed reference.txt before any audio existed, the number is trustworthy. If your WER matches the published 4.0%, that is a real result and the framing holds. Either outcome is the measurement.

Secret mission

Prove the Measurement Is Sensitive

Your WER is a single number from a single reading. Is it measurement or luck? Read the same passage again to check repeatability, then break it on purpose with a jargon-free passage and watch the rate move.

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files and tools for future experiments or remove them.

Resources you used:

Project files: score.py, reference.txt, hypothesis.txt, predictions.txt, hypothesis_repeat.txt, reference_plain.txt, and hypothesis_plain.txt.
Antigravity CLI credentials stored in ~/.gemini/.
The jiwer Python package.

Keep everything running

No action needed. Choose this if you plan to run more dictation experiments or test other passages.

Your project files stay in the folder where you created them.
The Antigravity CLI (agy) remains installed on your machine. No services are running and nothing costs money.
Your stored credentials in ~/.gemini/ and the jiwer package stay in place.

Pause - I'll come back to this later

There is nothing to pause. This project has no running services or cloud resources.

Your files and tools remain on your machine until you choose to delete them.

Delete - I don't want to use this again

Remove all project files, stored credentials, and the jiwer package to start fresh.

Delete your project files by running:

macOS

rm reference.txt hypothesis.txt predictions.txt score.py hypothesis_repeat.txt reference_plain.txt hypothesis_plain.txt

Windows

Remove-Item reference.txt, hypothesis.txt, predictions.txt, score.py, hypothesis_repeat.txt, reference_plain.txt, hypothesis_plain.txt

Linux

rm reference.txt hypothesis.txt predictions.txt score.py hypothesis_repeat.txt reference_plain.txt hypothesis_plain.txt

Remove the Antigravity CLI's stored credentials by running:

macOS

rm -rf ~/.gemini/

Windows

Remove-Item -Recurse -Force "$env:USERPROFILE\.gemini"

Linux

rm -rf ~/.gemini/

Uninstall the jiwer package by running:

pip uninstall jiwer

When prompted, enter y to confirm the removal.

That's a Wrap!

Nice work. You now have a personal word error rate measured from your own voice, your own microphone, and your own room. That number is yours, and nobody can argue you into a better one.

You've learned how to:

Seal a reference passage with domain jargon before any audio exists, commit three predictions before touching a microphone, and match each one against an observation.
Run a cold /voice dictation session in the Antigravity CLI with no prior configuration, read the passage once unrehearsed, and capture the raw unedited transcript.
Compute your personal Word Error Rate using jiwer, set it beside the published 4.0% streaming benchmark, and read a word-level alignment marking every S, I, and D.
Secret Mission: Proved the measurement was sensitive by showing the rate reproduced on a second reading and moved when you removed the jargon.

Ready to quiz yourself?