Build a Local Voice AI Tutor

Chain Whisper, Ollama, and edge-tts into a real-time voice AI pipeline.

Introduction

30 Second Summary

Voice assistants like Siri and Alexa feel like magic, but under the hood they follow a simple pattern: listen, think, respond. What if you could build your own from scratch, running entirely on your computer?

In this project, you will build a voice AI tutor that holds real spoken conversations with you. You'll chain together three ML models into a real-time pipeline: speech recognition, a large language model, and text-to-speech synthesis.

What You'll Build

A voice-powered AI tutor with an Albert Einstein persona that listens through your microphone, transcribes your speech on your GPU, generates responses via a local LLM, and speaks them back in a natural voice.

By the end of this project, you'll have:

  • A GPU-accelerated speech-to-text pipeline using OpenAI Whisper running on your own hardware.
  • A local LLM conversation engine powered by Ollama with a configurable Einstein persona.
  • A text-to-speech module that speaks responses back using natural-sounding neural voices.
  • Secret Mission: Add multi-persona switching so you can swap characters mid-conversation.

Are there any prerequisites?

You'll need a Windows laptop with an NVIDIA GPU (for running Whisper on your GPU), a working microphone, and an internet connection. You should also have Python 3.13 installed.

No prior experience with ML models or voice processing is required. This project walks you through every step.

Build the Configuration and Project Structure

Your development environment is ready with all the tools installed. Now it is time to give your project a clean structure that separates each piece of the voice AI pipeline into its own module.

Production voice AI systems split their work into distinct stages: listening, thinking, and speaking. You will mirror this pattern by giving each stage its own Python file, plus a central configuration file that stores all the settings in one place.

In this step, get ready to:
  • Create the modular file structure for your voice AI pipeline.
  • Configure persona settings, model names, and audio constants in a central config file.
Create the project files

Each module in your pipeline gets its own file. This keeps your code organized and makes it easy to test each piece independently.

  • In your PowerShell terminal (with your virtual environment still active), create all the project files by running this command:
New-Item -ItemType File -Path "config.py", "stt.py", "llm.py", "tts.py", "main.py", "requirements.txt"

What does each file do?

  • config.py stores all settings (model names, audio parameters, persona definitions).
  • stt.py handles speech-to-text using Whisper.
  • llm.py manages conversation with the LLM via Ollama.
  • tts.py converts text responses into spoken audio.
  • main.py wires all three modules together into the conversation loop.
  • requirements.txt will list your pinned dependencies for reproducibility.
  • Confirm the files exist by running:
ls

You should see all six files listed in the output: config.py, stt.py, llm.py, tts.py, main.py, and requirements.txt.

Seeing an error about files already existing?

If you previously created any of these files, PowerShell will throw an error. Delete the existing file first with Remove-Item config.py (replace the filename as needed), then re-run the New-Item command.

Need more help? Help me troubleshoot file creation errors in PowerShell.

Set up the persona configuration

Your config file defines who your AI tutor is and how each component behaves. You will start with an Albert Einstein persona that speaks with curiosity and uses thought experiments to teach.

  • Open config.py in your code editor and paste in the persona dictionary:
# Persona definitions - each persona has a name, system prompt, and TTS voice
PERSONAS = {
    "einstein": {
        "name": "Albert Einstein",
        "system_prompt": (
            "You are Albert Einstein, the famous theoretical physicist. "
            "You speak with a warm, curious, and slightly playful tone. "
            "You use analogies and thought experiments to explain complex "
            "physics and math concepts. You occasionally reference your own "
            "work on relativity and quantum mechanics. Keep responses "
            "conversational and concise - 2 to 3 sentences for simple "
            "questions, up to a short paragraph for complex ones. "
            "You are tutoring a student who is eager to learn."
        ),
        "voice": "en-US-GuyNeural",
    },
}

What does this code do?

The PERSONAS dictionary holds everything that makes your tutor unique. Each persona entry has three keys:

  • name is the display name printed to the terminal during conversation.
  • system_prompt tells the LLM how to behave. This is the core of prompt engineering for personas.
  • voice selects which neural voice edge-tts uses for speech output.
  • Below the PERSONAS dictionary, add the model and audio configuration constants:
# Default persona to load on startup
DEFAULT_PERSONA = "einstein"

# Model settings
WHISPER_MODEL = "small.en"
OLLAMA_MODEL = "llama3.1:8b"
OLLAMA_URL = "http://localhost:11434/api/chat"

# Audio recording settings
SAMPLE_RATE = 16000
CHANNELS = 1
CHUNK = 1024
SILENCE_THRESHOLD = 500

What do these constants control?

WHISPER_MODEL selects the small.en model. This uses about 2 GB of VRAM and gives strong accuracy for English speech.

OLLAMA_URL points to Ollama's local REST API. Since Ollama runs on your machine, there is no network latency or API key required.

SAMPLE_RATE at 16000 Hz is the standard for speech recognition. SILENCE_THRESHOLD controls how quiet the audio must be before the recorder decides you have stopped talking.

  • Save config.py.

You can verify it loads without errors by running this in your terminal:

python -c "import config; print(config.PERSONAS['einstein']['name'])"

You should see Albert Einstein printed in the terminal.

Seeing a SyntaxError or ModuleNotFoundError?

A SyntaxError usually means a missing comma, parenthesis, or quotation mark in your config.py. Double-check that each string inside system_prompt ends with a space before the closing quote.

A ModuleNotFoundError means Python cannot find the file. Make sure your terminal is in the same folder where config.py lives.

Still stuck? Help me debug my config.py file.

Tab 1

Great. Double-check you have saved config.py and move on to the next step.

Tab 2

Here is the complete config.py file for comparison:

# Persona definitions - each persona has a name, system prompt, and TTS voice
PERSONAS = {
    "einstein": {
        "name": "Albert Einstein",
        "system_prompt": (
            "You are Albert Einstein, the famous theoretical physicist. "
            "You speak with a warm, curious, and slightly playful tone. "
            "You use analogies and thought experiments to explain complex "
            "physics and math concepts. You occasionally reference your own "
            "work on relativity and quantum mechanics. Keep responses "
            "conversational and concise - 2 to 3 sentences for simple "
            "questions, up to a short paragraph for complex ones. "
            "You are tutoring a student who is eager to learn."
        ),
        "voice": "en-US-GuyNeural",
    },
}

# Default persona to load on startup
DEFAULT_PERSONA = "einstein"

# Model settings
WHISPER_MODEL = "small.en"
OLLAMA_MODEL = "llama3.1:8b"
OLLAMA_URL = "http://localhost:11434/api/chat"

# Audio recording settings
SAMPLE_RATE = 16000
CHANNELS = 1
CHUNK = 1024
SILENCE_THRESHOLD = 500

Your project skeleton is in place and your configuration is locked in. Next up, you will bring the speech-to-text module to life so your AI tutor can actually hear you speak.

Build the Speech-to-Text Module

Your development environment is ready with all the tools installed. Now it is time to build the first piece of the voice AI pipeline: the speech-to-text module.

Speech-to-text (STT) is the front door of your voice assistant. It listens through your microphone, detects when you stop speaking, and converts that audio into text that the LLM can understand.

In this step, get ready to:
  • Set up the STT module with required imports and configuration.
  • Implement microphone recording with automatic silence detection.
  • Load the Whisper model on your GPU and transcribe speech to text.
Set up the STT module imports

Your project already has five empty Python files from Step 1. The stt.py file will hold the SpeechToText class responsible for recording audio and converting it to text.

  • Open stt.py in your editor and add the following imports:
import whisper
import torch
import numpy as np
import pyaudio
import wave
import tempfile
import os

from config import WHISPER_MODEL, SAMPLE_RATE, CHANNELS, CHUNK, SILENCE_THRESHOLD

What do these imports do?

  • whisper is OpenAI's speech recognition model that runs on your GPU.
  • torch checks whether CUDA (GPU acceleration) is available.
  • numpy calculates audio amplitude to detect silence.
  • pyaudio opens your microphone as an input stream.
  • wave and tempfile save recorded audio to a temporary WAV file that Whisper can read.
  • The constants from config.py keep all tunable values in one place.
  • Save stt.py.
Implement microphone recording with silence detection

The recording method needs to solve a tricky problem: how does the program know when you have finished speaking? The answer is silence detection.

The code continuously reads audio chunks from your microphone and measures their amplitude. When the amplitude drops below a threshold for roughly 1.5 seconds, the recording stops automatically.

  • In stt.py, below your imports, add the SpeechToText class with the record_audio method by pasting this code:
class SpeechToText:
    def record_audio(self):
        # Open a PyAudio input stream from the microphone
        p = pyaudio.PyAudio()
        stream = p.open(
            format=pyaudio.paInt16,
            channels=CHANNELS,
            rate=SAMPLE_RATE,
            input=True,
            frames_per_buffer=CHUNK,
        )

        print("\nListening... (speak now, silence will end recording)")
        frames = []
        # Number of consecutive silent chunks before we stop
        silence_frames = int(SAMPLE_RATE / CHUNK * 1.5)

        try:
            while True:
                data = stream.read(CHUNK, exception_on_overflow=False)
                frames.append(data)

                # Only check for silence after at least 1 second of audio
                if len(frames) > int(SAMPLE_RATE / CHUNK):
                    recent = frames[-silence_frames:]
                    audio_data = np.frombuffer(b"".join(recent), dtype=np.int16)
                    if np.abs(audio_data).mean() < SILENCE_THRESHOLD:
                        break
        except KeyboardInterrupt:
            pass

        # Clean up the audio stream
        stream.stop_stream()
        stream.close()
        p.terminate()

        # Discard recordings shorter than 0.5 seconds (likely noise)
        if len(frames) < int(SAMPLE_RATE / CHUNK * 0.5):
            return None

        # Save the recorded frames to a temporary WAV file
        temp_file = tempfile.mktemp(suffix=".wav")
        wf = wave.open(temp_file, "wb")
        wf.setnchannels(CHANNELS)
        wf.setsampwidth(2)
        wf.setframerate(SAMPLE_RATE)
        wf.writeframes(b"".join(frames))
        wf.close()

        return temp_file

What does this code do?

This method captures audio from your microphone in a continuous loop. Each iteration reads 1024 frames (about 64ms of audio at 16kHz).

  • exception_on_overflow=False prevents the program from crashing if the buffer fills faster than Python can read it.
  • After 1 second of audio has been collected, it starts checking the most recent 1.5 seconds for silence.
  • If the average amplitude drops below SILENCE_THRESHOLD (500), the loop breaks and recording ends.
  • Recordings shorter than 0.5 seconds are discarded as accidental noise.
  • Valid recordings are saved to a temporary .wav file and the file path is returned.
  • Save stt.py.

Seeing an indentation error?

Make sure record_audio is indented one level inside the class SpeechToText: definition. Python uses indentation to define class membership. Every line inside the method should be indented with 8 spaces (4 for class, 4 for method).

help me fix an indentation error in my Python class

Implement Whisper transcription

OpenAI Whisper is a speech recognition model that can run entirely on your local GPU. The small.en variant uses approximately 2 GB of VRAM and provides strong accuracy for English speech.

  • In stt.py, add the __init__ method inside the SpeechToText class, directly below class SpeechToText: and above record_audio:
    def __init__(self, model_name=WHISPER_MODEL):
        # Check if GPU is available, fall back to CPU
        device = "cuda" if torch.cuda.is_available() else "cpu"
        print(f"Loading Whisper model '{model_name}' on {device}...")
        self.model = whisper.load_model(model_name, device=device)
        print("Whisper model loaded successfully.")

What does this code do?

  • torch.cuda.is_available() checks whether your NVIDIA GPU is accessible for computation.
  • whisper.load_model downloads (on first run) and loads the small.en model into GPU memory.
  • The model stays in memory for the lifetime of the object, so subsequent transcriptions are fast.

Now add the transcribe method that takes the WAV file and returns text.

  • Add this method below record_audio, still inside the SpeechToText class:
    def transcribe(self, audio_file):
        # Skip if no audio was recorded
        if audio_file is None:
            return ""
        # Run Whisper inference on the audio file
        result = self.model.transcribe(audio_file)
        # Delete the temp file after transcription
        os.unlink(audio_file)
        return result["text"].strip()

What does this code do?

  • If audio_file is None (recording was too short), it returns an empty string immediately.
  • self.model.transcribe() runs GPU-accelerated speech recognition and returns a dictionary with a "text" key.
  • os.unlink() deletes the temporary WAV file to avoid filling up disk space over a long conversation.
  • Save stt.py.

Now test the complete module to confirm your microphone and GPU transcription work together.

  • Run this command in your terminal to test the STT module standalone:
python -c "from stt import SpeechToText; s = SpeechToText(); a = s.record_audio(); print('You said:', s.transcribe(a))"
  • When you see Listening..., speak a short phrase into your microphone (e.g. "Hello, can you hear me?").
  • Wait about 1.5 seconds of silence. The recording stops automatically.
  • You should see your spoken words printed as text in the terminal.

Not seeing any transcription output?

  • If you see Loading Whisper model but it hangs, the model is downloading for the first time. This can take a minute on your internet connection.
  • If the recording ends immediately without waiting for speech, your microphone might not be set as the default input device. Check your Windows sound settings.
  • If you see a warning about Triton kernels, ignore it. Triton does not have a Windows binary and the warning is cosmetic. Transcription still works correctly.
  • If it prints an empty line, speak louder or move closer to the microphone. The silence threshold of 500 might be too high for quiet microphones.

help me debug why my Whisper transcription is not producing output

✔️ Awesome, I've got everything!

Great. Double check you have saved stt.py and confirmed it transcribes your speech correctly.

ⓧ I'd like to double check the full code

Here is the complete stt.py file. Make sure yours matches exactly:

import whisper
import torch
import numpy as np
import pyaudio
import wave
import tempfile
import os

from config import WHISPER_MODEL, SAMPLE_RATE, CHANNELS, CHUNK, SILENCE_THRESHOLD


class SpeechToText:
    def __init__(self, model_name=WHISPER_MODEL):
        device = "cuda" if torch.cuda.is_available() else "cpu"
        print(f"Loading Whisper model '{model_name}' on {device}...")
        self.model = whisper.load_model(model_name, device=device)
        print("Whisper model loaded successfully.")

    def record_audio(self):
        p = pyaudio.PyAudio()
        stream = p.open(
            format=pyaudio.paInt16,
            channels=CHANNELS,
            rate=SAMPLE_RATE,
            input=True,
            frames_per_buffer=CHUNK,
        )

        print("\nListening... (speak now, silence will end recording)")
        frames = []
        silence_frames = int(SAMPLE_RATE / CHUNK * 1.5)

        try:
            while True:
                data = stream.read(CHUNK, exception_on_overflow=False)
                frames.append(data)

                if len(frames) > int(SAMPLE_RATE / CHUNK):
                    recent = frames[-silence_frames:]
                    audio_data = np.frombuffer(b"".join(recent), dtype=np.int16)
                    if np.abs(audio_data).mean() < SILENCE_THRESHOLD:
                        break
        except KeyboardInterrupt:
            pass

        stream.stop_stream()
        stream.close()
        p.terminate()

        if len(frames) < int(SAMPLE_RATE / CHUNK * 0.5):
            return None

        temp_file = tempfile.mktemp(suffix=".wav")
        wf = wave.open(temp_file, "wb")
        wf.setnchannels(CHANNELS)
        wf.setsampwidth(2)
        wf.setframerate(SAMPLE_RATE)
        wf.writeframes(b"".join(frames))
        wf.close()

        return temp_file

    def transcribe(self, audio_file):
        if audio_file is None:
            return ""
        result = self.model.transcribe(audio_file)
        os.unlink(audio_file)
        return result["text"].strip()

Your speech-to-text module is working. Next up, you will build the LLM conversation module that takes your transcribed text and generates an Einstein-style response.

Build the LLM Conversation Module

Your speech-to-text module can now transcribe your voice into text. The next piece of the pipeline is the "brain" that understands what you said and generates a response.

In this step, you will build the LLM conversation module that talks to Ollama's local REST API. This module sends your transcribed text to the llama3.1:8b model and returns an in-character response from your Einstein persona.

In this step, get ready to:
  • Build an LLMChat class that communicates with Ollama's REST API.
  • Implement conversation history so the tutor remembers what you said.
  • Test the module standalone and confirm Einstein-style responses.
Set up the Ollama API client

Ollama exposes a REST API at http://localhost:11434/api/chat. Your Python code sends a JSON payload containing the model name, the full conversation history, and a flag to disable streaming. Ollama returns a single JSON object with the assistant's response.

  • Open llm.py in your editor and add the imports and class initializer by pasting this code:
import urllib.request
import json

from config import OLLAMA_MODEL, OLLAMA_URL


class LLMChat:
    def __init__(self, model=OLLAMA_MODEL, system_prompt="", url=OLLAMA_URL):
        self.model = model
        self.url = url
        # Store the system prompt as the first message in conversation history
        self.messages = [{"role": "system", "content": system_prompt}]

What does this code do?

The LLMChat class wraps all communication with Ollama. It uses Python's built-in urllib.request module instead of third-party libraries like requests, keeping your dependency list minimal.

The self.messages list stores the entire conversation history. Ollama's API is stateless, so your client must send all prior messages with every request. The system prompt sits at index 0 and sets the persona for the entire conversation.

You won't see a change yet. The next chunk adds the core chat() and reset() methods that make this class functional.

  • Below the __init__ method, add the chat() and reset() methods by pasting this code:
    def chat(self, user_message):
        self.messages.append({"role": "user", "content": user_message})

        # Build the JSON payload for Ollama's /api/chat endpoint
        payload = json.dumps({
            "model": self.model,
            "messages": self.messages,
            "stream": False,
        }).encode("utf-8")

        req = urllib.request.Request(
            self.url,
            data=payload,
            headers={"Content-Type": "application/json"},
        )

        try:
            with urllib.request.urlopen(req) as response:
                result = json.loads(response.read().decode("utf-8"))
        except Exception as e:
            error_msg = f"Error communicating with Ollama: {e}"
            print(error_msg)
            return "I seem to be having trouble thinking right now. Could you repeat that?"

        assistant_message = result["message"]["content"]
        self.messages.append({"role": "assistant", "content": assistant_message})
        return assistant_message

    def reset(self, system_prompt=None):
        if system_prompt:
            self.messages = [{"role": "system", "content": system_prompt}]
        else:
            self.messages = [self.messages[0]]

What does this code do?

  • chat() appends the user's message to the conversation history, sends the entire history to Ollama as a JSON POST request with "stream": False, and returns the assistant's response. It also appends the assistant's reply to self.messages so the next call includes full context.
  • The try/except block catches network errors gracefully. If Ollama is unreachable, the tutor returns a friendly fallback instead of crashing.
  • reset() clears the conversation history back to just the system prompt. This is useful when switching personas later in the Secret Mission.
  • Save llm.py.

Seeing a syntax error when you save?

Make sure the chat() and reset() methods are indented inside the LLMChat class (one level of indentation). Python relies on whitespace to determine class membership.

help me fix an indentation error in my LLMChat class

✔️ Awesome, I've got everything!

Great. Double check you have saved llm.py.

ⓧ I'd like to double check the full code

import urllib.request
import json

from config import OLLAMA_MODEL, OLLAMA_URL


class LLMChat:
    def __init__(self, model=OLLAMA_MODEL, system_prompt="", url=OLLAMA_URL):
        self.model = model
        self.url = url
        self.messages = [{"role": "system", "content": system_prompt}]

    def chat(self, user_message):
        self.messages.append({"role": "user", "content": user_message})

        payload = json.dumps({
            "model": self.model,
            "messages": self.messages,
            "stream": False,
        }).encode("utf-8")

        req = urllib.request.Request(
            self.url,
            data=payload,
            headers={"Content-Type": "application/json"},
        )

        try:
            with urllib.request.urlopen(req) as response:
                result = json.loads(response.read().decode("utf-8"))
        except Exception as e:
            error_msg = f"Error communicating with Ollama: {e}"
            print(error_msg)
            return "I seem to be having trouble thinking right now. Could you repeat that?"

        assistant_message = result["message"]["content"]
        self.messages.append({"role": "assistant", "content": assistant_message})
        return assistant_message

    def reset(self, system_prompt=None):
        if system_prompt:
            self.messages = [{"role": "system", "content": system_prompt}]
        else:
            self.messages = [self.messages[0]]
Design the Einstein persona prompt

You already configured the Einstein persona in config.py during the previous step. The system_prompt field tells the LLM to respond as Albert Einstein with a warm, curious tone using analogies and thought experiments.

How does the persona work?

When you create an LLMChat instance and pass in the Einstein system_prompt from config.py, that prompt becomes the first message in the conversation history. Every subsequent request to Ollama includes this system message, so the model stays in character for the entire session.

The prompt instructs the model to keep responses concise (2 to 3 sentences for simple questions) and to reference Einstein's own work. This prevents the LLM from generating long, generic answers.

Now test the module standalone to confirm it generates Einstein-style responses.

  • Run this command in your terminal to send a test question to the LLM:
python -c "from llm import LLMChat; from config import PERSONAS; chat = LLMChat(system_prompt=PERSONAS['einstein']['system_prompt']); print(chat.chat('What is gravity?'))"

You should see a response written in Einstein's style. It will use analogies or references to relativity and address you as a curious student.

Why does it take a few seconds?

The first request to Ollama loads the llama3.1:8b model into GPU/CPU memory. Subsequent requests in the same session are much faster because the model stays loaded for 5 minutes by default.

🙋‍♀️ Getting a connection error?

Make sure Ollama is running. Open a separate terminal and run ollama serve if it is not already active. On Windows, Ollama may also be running as a background service after installation.

If you see a timeout, the model may still be loading. Wait 10-15 seconds and try again.

help me debug a connection error when calling Ollama from my Python LLMChat class

Your LLM module is ready. Next up, you will build the text-to-speech module so the tutor can speak its responses back to you.

Build the Text-to-Speech Module

Your STT module captures what the learner says, and your LLM module thinks of a response. Now you need the final piece: a voice that speaks the response back.

In this step, you will build the text-to-speech module using edge-tts. This library connects to Microsoft's neural voice service to generate natural-sounding speech from text, with no API key required.

In this step, get ready to:
  • Implement speech synthesis that generates audio from text using edge-tts.
  • Implement audio playback that converts and plays audio through your speakers.
  • Test the full TTS pipeline by speaking a phrase aloud.
Implement speech synthesis with edge-tts

The TextToSpeech class handles the full journey from text to audible speech. It uses edge-tts to generate an MP3 file, converts that to WAV using ffmpeg, and plays the result through your speakers with PyAudio.

  • In tts.py, add the following code to create the TextToSpeech class with its speak() method:
import edge_tts
import subprocess
import wave
import pyaudio
import tempfile
import os

from config import PERSONAS, DEFAULT_PERSONA


class TextToSpeech:
    def __init__(self, voice=PERSONAS[DEFAULT_PERSONA]["voice"]):
        self.voice = voice

    def set_voice(self, voice):
        self.voice = voice

    def speak(self, text):
        mp3_file = tempfile.mktemp(suffix=".mp3")
        wav_file = tempfile.mktemp(suffix=".wav")

        try:
            # Generate MP3 audio from text using edge-tts
            communicate = edge_tts.Communicate(text, self.voice)
            communicate.save_sync(mp3_file)

            # Convert MP3 to WAV using ffmpeg for PyAudio compatibility
            subprocess.run(
                [
                    "ffmpeg", "-y", "-i", mp3_file,
                    "-ar", "24000", "-ac", "1", "-f", "wav", wav_file,
                ],
                stdout=subprocess.DEVNULL,
                stderr=subprocess.DEVNULL,
            )

            # Play the converted audio through speakers
            self._play_wav(wav_file)
        finally:
            # Clean up temporary files regardless of success or failure
            if os.path.exists(mp3_file):
                os.unlink(mp3_file)
            if os.path.exists(wav_file):
                os.unlink(wav_file)

What does this code do?

The __init__ method loads the default voice (en-US-GuyNeural) from your config.py personas dictionary. The set_voice() helper lets you swap voices later without creating a new instance.

The speak() method orchestrates three stages: generate an MP3 with edge_tts.Communicate, convert it to WAV via ffmpeg, and play the WAV file. The finally block ensures temporary files get deleted even if playback fails.

💡 Why convert MP3 to WAV?

PyAudio works with raw audio streams and does not decode compressed formats like MP3. The ffmpeg conversion step resamples the audio to 24000 Hz mono WAV, which PyAudio can stream directly to your speakers.

Implement audio playback

The _play_wav() method reads the WAV file in small chunks and streams them to your speakers through PyAudio.

  • Add the _play_wav() method directly below the speak() method, inside the TextToSpeech class:
    def _play_wav(self, wav_file):
        wf = wave.open(wav_file, "rb")
        p = pyaudio.PyAudio()
        stream = p.open(
            format=p.get_format_from_width(wf.getsampwidth()),
            channels=wf.getnchannels(),
            rate=wf.getframerate(),
            output=True,
        )

        # Read and play audio in 1024-frame chunks
        data = wf.readframes(1024)
        while data:
            stream.write(data)
            data = wf.readframes(1024)

        stream.stop_stream()
        stream.close()
        p.terminate()
        wf.close()

What does this code do?

The method opens the WAV file and creates a PyAudio output stream matched to the file's sample width, channel count, and frame rate. It then reads 1024 frames at a time and writes them to the stream, which sends audio to your speakers.

Once all frames have been played, the method closes the stream and releases the audio device. The leading underscore in _play_wav signals this is an internal helper that speak() calls, not something external code should use directly.

  • Save tts.py.
  • Test the module by running this command in your terminal:
python -c "from tts import TextToSpeech; tts = TextToSpeech(); tts.speak('Hello, I am your AI tutor.')"

You should hear the phrase "Hello, I am your AI tutor" spoken aloud through your speakers in a natural-sounding male voice.

Not hearing any audio?

  • Make sure your speakers or headphones are connected and volume is turned up.
  • Check that ffmpeg is available by running ffmpeg -version in your terminal. If it is not found, reinstall it.
  • Verify your internet connection is active. edge-tts requires internet access to generate speech audio.

Help me debug why my edge-tts TextToSpeech module is not producing audio

✔️ Awesome, I've got everything!

Great. Double check you have saved tts.py before moving on.

ⓧ I'd like to double check the full code

import edge_tts
import subprocess
import wave
import pyaudio
import tempfile
import os

from config import PERSONAS, DEFAULT_PERSONA


class TextToSpeech:
    def __init__(self, voice=PERSONAS[DEFAULT_PERSONA]["voice"]):
        self.voice = voice

    def set_voice(self, voice):
        self.voice = voice

    def speak(self, text):
        mp3_file = tempfile.mktemp(suffix=".mp3")
        wav_file = tempfile.mktemp(suffix=".wav")

        try:
            communicate = edge_tts.Communicate(text, self.voice)
            communicate.save_sync(mp3_file)

            subprocess.run(
                [
                    "ffmpeg", "-y", "-i", mp3_file,
                    "-ar", "24000", "-ac", "1", "-f", "wav", wav_file,
                ],
                stdout=subprocess.DEVNULL,
                stderr=subprocess.DEVNULL,
            )

            self._play_wav(wav_file)
        finally:
            if os.path.exists(mp3_file):
                os.unlink(mp3_file)
            if os.path.exists(wav_file):
                os.unlink(wav_file)

    def _play_wav(self, wav_file):
        wf = wave.open(wav_file, "rb")
        p = pyaudio.PyAudio()
        stream = p.open(
            format=p.get_format_from_width(wf.getsampwidth()),
            channels=wf.getnchannels(),
            rate=wf.getframerate(),
            output=True,
        )

        data = wf.readframes(1024)
        while data:
            stream.write(data)
            data = wf.readframes(1024)

        stream.stop_stream()
        stream.close()
        p.terminate()
        wf.close()

Your voice AI tutor now has a voice. In the next step, you will wire all three modules together into a real-time conversation loop where you speak, the AI thinks, and it talks back.

Wire the Full Conversation Loop

You have three standalone modules that each handle one piece of the voice AI pipeline. Right now they exist in isolation.

In this step, you will connect them into a single conversation loop inside main.py. The loop records your voice, transcribes it, sends it to the LLM, and speaks the response back to you.

In this step, get ready to:
  • Build the main conversation loop that chains STT, LLM, and TTS together.
  • Add a greeting and graceful exit so the assistant introduces itself and responds to "goodbye".
  • Test the complete pipeline with a multi-turn voice conversation.
Build the main conversation loop

The orchestration code imports all three modules and instantiates them with the Einstein persona configuration. This is where your pipeline becomes a real application.

  • Open main.py and add the imports and module setup by pasting this code:
from config import PERSONAS, DEFAULT_PERSONA, OLLAMA_MODEL, OLLAMA_URL
from stt import SpeechToText
from llm import LLMChat
from tts import TextToSpeech


def main():
    persona = PERSONAS[DEFAULT_PERSONA]

    print("=" * 50)
    print("  Voice AI Tutor")
    print(f"  Persona: {persona['name']}")
    print("  Say 'goodbye' to exit.")
    print("=" * 50)

    # Instantiate the three pipeline modules
    stt = SpeechToText()
    llm = LLMChat(
        model=OLLAMA_MODEL,
        system_prompt=persona["system_prompt"],
        url=OLLAMA_URL,
    )
    tts = TextToSpeech(voice=persona["voice"])

What does this code do?

The imports pull in your three modules (SpeechToText, LLMChat, TextToSpeech) plus the configuration constants from config.py.

The persona dictionary holds Einstein's name, system prompt, and voice setting. Each module is instantiated with the matching persona values so they all work in sync.

Now add the conversation loop directly below the code you just pasted. This is the core listen-think-speak cycle.

  • Add this code below the tts = TextToSpeech(voice=persona["voice"]) line:
    # Main conversation loop: listen, think, speak
    while True:
        audio_file = stt.record_audio()
        user_text = stt.transcribe(audio_file)

        # Skip empty transcriptions (noise or silence)
        if not user_text:
            continue

        print(f"\nYou: {user_text}")

        # Graceful exit when user says goodbye
        if "goodbye" in user_text.lower():
            farewell = (
                "Goodbye! Remember, imagination is more important "
                "than knowledge. Keep questioning everything!"
            )
            print(f"\n{persona['name']}: {farewell}")
            tts.speak(farewell)
            break

        # Send transcription to LLM and speak the response
        response = llm.chat(user_text)
        print(f"\n{persona['name']}: {response}")
        tts.speak(response)


if __name__ == "__main__":
    main()

What does this loop do?

  • stt.record_audio() opens the microphone and waits for you to speak. Silence ends the recording.
  • stt.transcribe() runs Whisper on your GPU to convert the audio to text.
  • if not user_text: continue skips empty results caused by background noise or very short audio.
  • The goodbye check lets you exit the loop gracefully by saying the word "goodbye" anywhere in your sentence.
  • llm.chat() sends your text to Ollama and receives Einstein's response. The response is then spoken aloud by tts.speak().
  • Save main.py.

Seeing a syntax error?

Make sure the while True: block is indented inside the main() function (one level of indentation). The if __name__ block should be at the top level with no indentation.

help me fix an indentation error in my main.py conversation loop

Add the greeting and conversation flow

A good tutor greets you before waiting for questions. The greeting plays as spoken audio when the program starts, so you know the system is ready.

  • In main.py, find the line tts = TextToSpeech(voice=persona["voice"]).
  • Add this greeting code between that line and the while True: loop:
    # Greet the user when the program starts
    greeting = (
        f"Hello! I am {persona['name']}. "
        "What would you like to explore today?"
    )
    print(f"\n{persona['name']}: {greeting}")
    tts.speak(greeting)

What does this code do?

The greeting uses the persona's name so it sounds personal. It prints the greeting to the terminal for debugging AND speaks it aloud through your speakers.

This pattern of printing and speaking every response lets you read what the AI said even if the audio is hard to hear. It also gives you a text log of the full conversation.

  • Save main.py.

Greeting not in the right place?

The greeting code must sit AFTER the three module instantiations (stt, llm, tts) and BEFORE the while True: line. It needs to be at the same indentation level as both sections.

help me position my greeting code correctly in main.py

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py before moving on.

ⓧ I'd like to double check the full code

from config import PERSONAS, DEFAULT_PERSONA, OLLAMA_MODEL, OLLAMA_URL
from stt import SpeechToText
from llm import LLMChat
from tts import TextToSpeech


def main():
    persona = PERSONAS[DEFAULT_PERSONA]

    print("=" * 50)
    print("  Voice AI Tutor")
    print(f"  Persona: {persona['name']}")
    print("  Say 'goodbye' to exit.")
    print("=" * 50)

    stt = SpeechToText()
    llm = LLMChat(
        model=OLLAMA_MODEL,
        system_prompt=persona["system_prompt"],
        url=OLLAMA_URL,
    )
    tts = TextToSpeech(voice=persona["voice"])

    greeting = (
        f"Hello! I am {persona['name']}. "
        "What would you like to explore today?"
    )
    print(f"\n{persona['name']}: {greeting}")
    tts.speak(greeting)

    while True:
        audio_file = stt.record_audio()
        user_text = stt.transcribe(audio_file)

        if not user_text:
            continue

        print(f"\nYou: {user_text}")

        if "goodbye" in user_text.lower():
            farewell = (
                "Goodbye! Remember, imagination is more important "
                "than knowledge. Keep questioning everything!"
            )
            print(f"\n{persona['name']}: {farewell}")
            tts.speak(farewell)
            break

        response = llm.chat(user_text)
        print(f"\n{persona['name']}: {response}")
        tts.speak(response)


if __name__ == "__main__":
    main()
Test the complete pipeline

All three modules are now wired together. Time to run a real voice conversation and confirm everything works end to end.

  • Make sure Ollama is still running. If you closed it, restart it by opening a new PowerShell window and running:
ollama serve
  • In your original PowerShell window (with the virtual environment active), start the voice tutor by running:
python main.py

What should I see?

The terminal prints "Loading Whisper model 'small.en' on cuda..." followed by the banner and Einstein's spoken greeting through your speakers. After the greeting finishes, you will see "Listening..." which means the microphone is active and waiting for you to speak.

  • Ask Einstein a question out loud (for example, "What is the theory of relativity?").
  • Wait for the response to be spoken back to you. You will see both your transcribed text and Einstein's reply printed in the terminal.
  • Have 3-4 exchanges to confirm multi-turn dialogue works (Einstein remembers context from earlier in the conversation).
  • Say "goodbye" to exit the conversation gracefully.

How do I check GPU utilization?

Open Task Manager by pressing Ctrl+Shift+Esc. Click the Performance tab, then select GPU from the sidebar.

You will see spikes in GPU utilization each time Whisper transcribes your speech. The spikes are brief because transcription only takes a moment on the RTX 3060.

🙋‍♀️ No audio coming from speakers?

  • Check that your system volume is not muted and that your default audio output device is set to your speakers or headphones.
  • If you see a "Failed to launch Triton kernels" warning in the terminal, ignore it. This is a cosmetic message on Windows that does not affect transcription.
  • If edge-tts throws a connection error, confirm you have an active internet connection. The TTS service requires internet even though the rest of the pipeline is local.

help me debug why my voice AI tutor is not producing audio output

Now lock down your project dependencies so anyone can reproduce your environment.

  • Open requirements.txt and add the pinned dependency versions:
openai-whisper==20250625
edge-tts==7.2.8
PyAudio==0.2.14

Why pin versions?

Pinning exact versions means anyone cloning your project gets the same packages that you tested with. This prevents mysterious breakage when a library releases a new version with changed behavior.

PyTorch is intentionally excluded because its install command varies by CUDA version and platform. The README should document the PyTorch install separately.

  • Save requirements.txt.

You now have a fully working voice AI tutor running entirely on your own hardware. In the Secret Mission, you will add multi-persona switching so you can talk to different scientists mid-conversation.

Secret mission

Multi-Persona Switching

Your Einstein tutor is impressive, but what if you could swap characters mid-conversation? Say "switch to Marie Curie" and get a chemistry-focused tutor with a completely different voice and personality. Add voice-activated persona switching to your pipeline.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs. However, the Ollama model and Whisper cache take up significant disk space (approximately 6GB total).

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop (voice-ai-tutor with virtual environment and Python files).
  • Ollama llama3.1:8b model (~4.7GB stored locally).
  • Whisper small.en model cache (~1GB in your user cache directory).
  • Ollama application (installed on your system).
  • ffmpeg (installed via Chocolatey or Scoop).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your voice AI tutor or want to build on this project.

  • Your Ollama model stays downloaded and ready for instant use.
  • Your Whisper model stays cached for fast transcription.
  • Your project files stay on your Desktop.

Pause - I'll come back to this later

Free up running processes while keeping your files and models intact for later.

  • If your virtual environment is still active, deactivate it by running:
  • Close any terminal windows running the voice tutor.
  • Ollama runs as a background service on Windows. You can stop it by right-clicking the Ollama icon in your system tray and selecting Quit.
deactivate

Your project folder, models, and all installed tools remain untouched. When you return, reactivate the virtual environment and start Ollama again.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to reclaim your disk space.

  • Remove the Ollama model by running this command in PowerShell:
ollama rm llama3.1:8b
  • Delete the Whisper model cache by running:
Remove-Item -Recurse -Force "$env:USERPROFILE\.cache\whisper"
  • Delete your project folder from the Desktop by running:
Remove-Item -Recurse -Force "$HOME\Desktop\voice-ai-tutor"
  • Uninstall Ollama from your system. Open Settings, then select Apps, then Installed apps. Search for Ollama and click Uninstall.
  • Uninstall ffmpeg. If you installed it with Chocolatey, run:
choco uninstall ffmpeg

If you installed ffmpeg with Scoop instead, run:

scoop uninstall ffmpeg

Nice Work!

Nice Work!

You just built a fully local voice AI tutor that holds real-time spoken conversations using GPU-accelerated ML models on your own hardware.

You've learned how to:

  • Build a complete voice AI pipeline that chains speech-to-text, LLM, and text-to-speech modules into a real-time conversation loop running locally on your GPU.
  • Orchestrate multiple ML models (OpenAI Whisper for transcription, Ollama serving Llama 3.1 for conversation) with GPU acceleration for real-time inference.
  • Create a modular, extensible architecture with persona engineering and natural-sounding text-to-speech via edge-tts that mirrors production voice AI systems.
  • Secret Mission: Added multi-persona switching so you can swap characters mid-conversation by voice command.

Ready to quiz yourself?