Build a Three-Way Voice Conversation
Integrate STT, speaker verification, TTS, and LLM into a local voice pipeline.
Introduction
30 Second Summary
Think about the best conversations you have had. They usually involve more than one perspective, with each voice bringing something different to the table. Now imagine having that kind of multi-perspective discussion with AI assistants who actually speak aloud in distinct voices.
In this project, you will build a Python application that runs a three-way voice conversation between you and two AI assistants. You speak into your microphone, the system transcribes and verifies your identity, then two AI personalities respond with unique synthesized voices.
What You'll Build
A fully local, GPU-accelerated voice conversation system that integrates speech recognition, speaker verification, text-to-speech, and LLM inference into a single interactive pipeline with clean, swappable module interfaces.
By the end of this project, you'll have:
- A modular backend architecture with swappable modules for STT, TTS, LLM, and speaker verification that you can upgrade independently.
- A voice fingerprint enrollment system that verifies your identity using speaker embeddings and cosine similarity, with a trust-all fallback for easy debugging.
- A turn-based three-way conversation loop where two AI assistants respond with distinct synthesized voices and every utterance is labeled in a shared transcript.
- Secret Mission: Add Claude API as a second LLM backend and export the conversation transcript to a timestamped JSON file.
Are there any prerequisites?
This project assumes you have Python 3.12 installed on a Windows desktop with an NVIDIA RTX 5070 GPU (driver 580.88+). You also need Ollama installed with a model pulled, and a Hugging Face account with an access token.
A second machine (like a laptop without a GPU) can optionally run the text-only conversation harness for testing LLM personalities without any audio dependencies.
Build the STT and Speaker Verification Modules
Your environment is set up and every ML model is downloaded. Now it is time to give your conversation system its ears and its identity check.
In this step, you will create the modules that listen to your voice, convert it to text, and verify that the person speaking is actually you. You will also build a central configuration file that every other module in the project will reference.
In this step, get ready to:
- Create a shared configuration file and the speech-to-text module.
- Build a speaker verification system with a trust-all fallback and an enrollment script.
- Create audio recording and playback utilities, then test the full pipeline end-to-end.
Create the configuration and speech-to-text module
Every module in this project needs to agree on shared settings like sample rates, model names, and file paths. A central configuration file keeps all of these in one place so you never have magic numbers scattered through your code.
- Create a new file called config.py in your project folder and paste the following code:
"""Configuration for the three-way voice conversation system."""
# --- LLM Backend ---
LLM_BACKEND = "ollama" # "ollama" or "claude"
OLLAMA_BASE_URL = "http://localhost:11434/v1"
OLLAMA_MODEL = "qwen2.5:7b" # Change to your installed model name
CLAUDE_API_KEY = "your-claude-api-key-here"
CLAUDE_MODEL = "claude-sonnet-4-20250514"
# --- Assistant Personalities ---
ASSISTANT_1_NAME = "Atlas"
ASSISTANT_1_SYSTEM = (
"You are Atlas, a curious and enthusiastic AI assistant. "
"You love exploring ideas and asking follow-up questions. "
"Keep responses concise (2-3 sentences). "
"You are in a three-way conversation with a human and another AI named Sage."
)
ASSISTANT_1_VOICE = "af_heart"
ASSISTANT_2_NAME = "Sage"
ASSISTANT_2_SYSTEM = (
"You are Sage, a calm and thoughtful AI assistant. "
"You provide balanced, well-reasoned perspectives. "
"Keep responses concise (2-3 sentences). "
"You are in a three-way conversation with a human and another AI named Atlas."
)
ASSISTANT_2_VOICE = "af_sarah"
# --- STT ---
WHISPER_MODEL = "base.en"
WHISPER_DEVICE = "cuda"
WHISPER_COMPUTE_TYPE = "float16"
# --- Speaker Verification ---
VERIFICATION_ENABLED = True # Set False for trust-all fallback
VOICEPRINT_PATH = "voiceprint.pt"
VERIFICATION_THRESHOLD = 0.25
# --- Audio ---
SAMPLE_RATE = 16000
SILENCE_THRESHOLD = 0.01
SILENCE_DURATION = 1.5 # seconds of silence before stopping
MAX_RECORD_DURATION = 30 # seconds
# --- TTS ---
TTS_SAMPLE_RATE = 24000
TTS_LANG_CODE = "a" # American EnglishWhat does this configuration do?
This file defines every tuneable setting for the project. Each section controls a different module:
- LLM_BACKEND and OLLAMA_BASE_URL tell the LLM module where to send requests.
- VERIFICATION_THRESHOLD sets how similar a voice must be to your enrolled voiceprint before the system accepts it (0.25 is the SpeechBrain default).
- SAMPLE_RATE of 16000 Hz matches what faster-whisper expects for transcription input.
- TTS_SAMPLE_RATE of 24000 Hz matches Kokoro's output sample rate.
- If your Ollama model name is different from qwen2.5:7b, update the OLLAMA_MODEL value to match the model you have pulled.
- Save config.py.
Not sure what model name to use?
Run ollama list in your terminal to see all models you have pulled. Use the name exactly as it appears in the NAME column.
Help me figure out which Ollama model name to use in my config.
Now you will create the speech-to-text module. This wraps faster-whisper in a simple interface: hand it audio, get back text.
- Create a new file called stt.py in your project folder and paste the following code:
"""Speech-to-text backend using faster-whisper."""
import numpy as np
import soundfile as sf
import tempfile
import os
from faster_whisper import WhisperModel
import config
_model = None
def _get_model():
global _model
if _model is None:
print(" [Loading Whisper model...]")
_model = WhisperModel(
config.WHISPER_MODEL,
device=config.WHISPER_DEVICE,
compute_type=config.WHISPER_COMPUTE_TYPE,
)
return _model
def transcribe(audio):
"""Transcribe a NumPy audio array to text.
Args:
audio: NumPy array of audio samples at 16000 Hz.
Returns:
Transcribed text string.
"""
if len(audio) == 0:
return ""
model = _get_model()
# Write to temp file (faster-whisper reads from file)
tmp_path = os.path.join(tempfile.gettempdir(), "voice_chat_input.wav")
sf.write(tmp_path, audio, config.SAMPLE_RATE)
segments, info = model.transcribe(tmp_path, beam_size=5)
text = " ".join(segment.text.strip() for segment in segments)
os.remove(tmp_path)
return text.strip()What does this code do?
- _get_model() uses a lazy-loading pattern. The model loads into GPU memory only the first time you call transcribe(), then stays cached for all future calls.
- transcribe() writes the NumPy audio array to a temporary WAV file because faster-whisper reads audio from disk. After transcription, the temp file is deleted.
- beam_size=5 tells the decoder to consider 5 candidate sequences at each step, balancing accuracy and speed.
- Save stt.py.
- Verify the module loads correctly by running this command in your terminal:
python -c "import stt; print('STT module loaded OK')"You should see STT module loaded OK printed. The Whisper model does not load yet because of the lazy pattern.
Seeing a ModuleNotFoundError?
Make sure your virtual environment is activated. You should see the environment name in parentheses at the start of your terminal prompt. If not, activate it again before running the import command.
Also check that your terminal's working directory is the project folder where config.py and stt.py live.
Help me fix a ModuleNotFoundError when importing my stt module.
✔️ Awesome, I've got everything!
Great. Make sure both config.py and stt.py are saved before moving on.
ⓧ I'd like to double check the full code
Here is the complete config.py:
"""Configuration for the three-way voice conversation system."""
# --- LLM Backend ---
LLM_BACKEND = "ollama" # "ollama" or "claude"
OLLAMA_BASE_URL = "http://localhost:11434/v1"
OLLAMA_MODEL = "qwen2.5:7b" # Change to your installed model name
CLAUDE_API_KEY = "your-claude-api-key-here"
CLAUDE_MODEL = "claude-sonnet-4-20250514"
# --- Assistant Personalities ---
ASSISTANT_1_NAME = "Atlas"
ASSISTANT_1_SYSTEM = (
"You are Atlas, a curious and enthusiastic AI assistant. "
"You love exploring ideas and asking follow-up questions. "
"Keep responses concise (2-3 sentences). "
"You are in a three-way conversation with a human and another AI named Sage."
)
ASSISTANT_1_VOICE = "af_heart"
ASSISTANT_2_NAME = "Sage"
ASSISTANT_2_SYSTEM = (
"You are Sage, a calm and thoughtful AI assistant. "
"You provide balanced, well-reasoned perspectives. "
"Keep responses concise (2-3 sentences). "
"You are in a three-way conversation with a human and another AI named Atlas."
)
ASSISTANT_2_VOICE = "af_sarah"
# --- STT ---
WHISPER_MODEL = "base.en"
WHISPER_DEVICE = "cuda"
WHISPER_COMPUTE_TYPE = "float16"
# --- Speaker Verification ---
VERIFICATION_ENABLED = True # Set False for trust-all fallback
VOICEPRINT_PATH = "voiceprint.pt"
VERIFICATION_THRESHOLD = 0.25
# --- Audio ---
SAMPLE_RATE = 16000
SILENCE_THRESHOLD = 0.01
SILENCE_DURATION = 1.5 # seconds of silence before stopping
MAX_RECORD_DURATION = 30 # seconds
# --- TTS ---
TTS_SAMPLE_RATE = 24000
TTS_LANG_CODE = "a" # American EnglishHere is the complete stt.py:
"""Speech-to-text backend using faster-whisper."""
import numpy as np
import soundfile as sf
import tempfile
import os
from faster_whisper import WhisperModel
import config
_model = None
def _get_model():
global _model
if _model is None:
print(" [Loading Whisper model...]")
_model = WhisperModel(
config.WHISPER_MODEL,
device=config.WHISPER_DEVICE,
compute_type=config.WHISPER_COMPUTE_TYPE,
)
return _model
def transcribe(audio):
"""Transcribe a NumPy audio array to text.
Args:
audio: NumPy array of audio samples at 16000 Hz.
Returns:
Transcribed text string.
"""
if len(audio) == 0:
return ""
model = _get_model()
# Write to temp file (faster-whisper reads from file)
tmp_path = os.path.join(tempfile.gettempdir(), "voice_chat_input.wav")
sf.write(tmp_path, audio, config.SAMPLE_RATE)
segments, info = model.transcribe(tmp_path, beam_size=5)
text = " ".join(segment.text.strip() for segment in segments)
os.remove(tmp_path)
return text.strip()Create the speaker verification module and enrollment script
Speaker verification answers a simple question: is the person speaking the same person who enrolled earlier? The system does this by comparing speaker embeddings. These are compact numerical fingerprints of a voice.
You will build two verifier classes behind a common interface. The real verifier uses SpeechBrain's ECAPA-TDNN model to extract embeddings and compare them via cosine similarity. The fallback always says "yes" so you can test the full loop without worrying about threshold tuning.
- Create a new file called verifier.py in your project folder and paste the following code:
"""Speaker verification backends."""
import os
import numpy as np
import torch
import torchaudio
import config
class TrustAllVerifier:
"""Fallback verifier that accepts any speaker."""
def enroll(self, audio):
print(" [TrustAll mode: skipping enrollment]")
def verify(self, audio):
return True, 1.0
class EcapaTdnnVerifier:
"""Speaker verification using SpeechBrain ECAPA-TDNN embeddings."""
def __init__(self):
from speechbrain.inference.speaker import EncoderClassifier
print(" [Loading ECAPA-TDNN speaker model...]")
self.classifier = EncoderClassifier.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb",
run_opts={"device": "cuda"},
)
self.voiceprint = None
if os.path.exists(config.VOICEPRINT_PATH):
self.voiceprint = torch.load(config.VOICEPRINT_PATH, weights_only=True)
def _get_embedding(self, audio):
"""Extract speaker embedding from audio array."""
signal = torch.tensor(audio, dtype=torch.float32).unsqueeze(0)
if signal.shape[1] < config.SAMPLE_RATE:
# Pad short audio to at least 1 second
pad = torch.zeros(1, config.SAMPLE_RATE - signal.shape[1])
signal = torch.cat([signal, pad], dim=1)
embedding = self.classifier.encode_batch(signal)
return embedding.squeeze()
def enroll(self, audio):
"""Save a voiceprint from an audio sample."""
embedding = self._get_embedding(audio)
torch.save(embedding, config.VOICEPRINT_PATH)
self.voiceprint = embedding
print(f" [Voiceprint saved to {config.VOICEPRINT_PATH}]")
def verify(self, audio):
"""Verify audio matches the enrolled voiceprint."""
if self.voiceprint is None:
print(" [No voiceprint enrolled, accepting by default]")
return True, 1.0
embedding = self._get_embedding(audio)
similarity = torch.nn.functional.cosine_similarity(
embedding.unsqueeze(0),
self.voiceprint.unsqueeze(0),
)
score = similarity.item()
is_match = score > config.VERIFICATION_THRESHOLD
return is_match, score
def get_verifier():
"""Factory function to get the configured verifier."""
if config.VERIFICATION_ENABLED:
return EcapaTdnnVerifier()
else:
return TrustAllVerifier()What does this code do?
- TrustAllVerifier is a debugging fallback. It accepts any voice so you can test the conversation loop without worrying about verification thresholds.
- EcapaTdnnVerifier loads SpeechBrain's pretrained ECAPA-TDNN model onto GPU. It extracts a 192-dimensional embedding from audio and compares it to your saved voiceprint using cosine similarity.
- The verify() method returns a tuple: whether the speaker matches and the raw similarity score. A score above VERIFICATION_THRESHOLD (0.25) means the voice matches.
- get_verifier() is a factory function that checks config.VERIFICATION_ENABLED and returns the appropriate verifier. Set it to False in config to skip verification entirely.
Why a trust-all fallback?
The ECAPA-TDNN threshold can be sensitive to different microphones and ambient noise levels. If verification keeps rejecting you during development, you can flip VERIFICATION_ENABLED to False and get the full conversation loop running immediately. Once everything else works, turn it back on and tune the threshold.
- Save verifier.py.
Now you need an enrollment script that captures your voice and saves the embedding to disk. This is what creates the voiceprint that the verifier compares against.
- Create a new file called enroll.py in your project folder and paste the following code:
"""Voice enrollment script - records your voice and saves a voiceprint."""
import config
from audio_utils import record_audio
from verifier import EcapaTdnnVerifier
def main():
print("=" * 60)
print(" VOICE ENROLLMENT")
print("=" * 60)
print()
print(" This will record your voice and save a fingerprint.")
print(" Please speak naturally for about 5 seconds.")
print(" Say something like: 'The quick brown fox jumps over")
print(" the lazy dog. I am enrolling my voice now.'")
print()
input(" Press Enter when ready to record...")
audio = record_audio()
if len(audio) < config.SAMPLE_RATE:
print(" [Error: Recording too short. Try again.]")
return
verifier = EcapaTdnnVerifier()
verifier.enroll(audio)
print()
print(" Enrollment complete! Your voiceprint has been saved.")
print(f" File: {config.VOICEPRINT_PATH}")
print()
if __name__ == "__main__":
main()What does this code do?
- The script prompts you to speak for about 5 seconds, then calls record_audio() to capture your voice from the microphone.
- If the recording is shorter than 1 second (fewer samples than SAMPLE_RATE), it rejects the sample and asks you to try again.
- It creates an EcapaTdnnVerifier instance and calls enroll(), which extracts the embedding and saves it to voiceprint.pt.
- Save enroll.py.
You will not be able to run enroll.py yet because it depends on audio_utils.py, which you will create next.
✔️ Awesome, I've got everything!
Great. Make sure both verifier.py and enroll.py are saved before moving on.
ⓧ I'd like to double check the full code
Here is the complete verifier.py:
"""Speaker verification backends."""
import os
import numpy as np
import torch
import torchaudio
import config
class TrustAllVerifier:
"""Fallback verifier that accepts any speaker."""
def enroll(self, audio):
print(" [TrustAll mode: skipping enrollment]")
def verify(self, audio):
return True, 1.0
class EcapaTdnnVerifier:
"""Speaker verification using SpeechBrain ECAPA-TDNN embeddings."""
def __init__(self):
from speechbrain.inference.speaker import EncoderClassifier
print(" [Loading ECAPA-TDNN speaker model...]")
self.classifier = EncoderClassifier.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb",
run_opts={"device": "cuda"},
)
self.voiceprint = None
if os.path.exists(config.VOICEPRINT_PATH):
self.voiceprint = torch.load(config.VOICEPRINT_PATH, weights_only=True)
def _get_embedding(self, audio):
"""Extract speaker embedding from audio array."""
signal = torch.tensor(audio, dtype=torch.float32).unsqueeze(0)
if signal.shape[1] < config.SAMPLE_RATE:
# Pad short audio to at least 1 second
pad = torch.zeros(1, config.SAMPLE_RATE - signal.shape[1])
signal = torch.cat([signal, pad], dim=1)
embedding = self.classifier.encode_batch(signal)
return embedding.squeeze()
def enroll(self, audio):
"""Save a voiceprint from an audio sample."""
embedding = self._get_embedding(audio)
torch.save(embedding, config.VOICEPRINT_PATH)
self.voiceprint = embedding
print(f" [Voiceprint saved to {config.VOICEPRINT_PATH}]")
def verify(self, audio):
"""Verify audio matches the enrolled voiceprint."""
if self.voiceprint is None:
print(" [No voiceprint enrolled, accepting by default]")
return True, 1.0
embedding = self._get_embedding(audio)
similarity = torch.nn.functional.cosine_similarity(
embedding.unsqueeze(0),
self.voiceprint.unsqueeze(0),
)
score = similarity.item()
is_match = score > config.VERIFICATION_THRESHOLD
return is_match, score
def get_verifier():
"""Factory function to get the configured verifier."""
if config.VERIFICATION_ENABLED:
return EcapaTdnnVerifier()
else:
return TrustAllVerifier()Here is the complete enroll.py:
"""Voice enrollment script - records your voice and saves a voiceprint."""
import config
from audio_utils import record_audio
from verifier import EcapaTdnnVerifier
def main():
print("=" * 60)
print(" VOICE ENROLLMENT")
print("=" * 60)
print()
print(" This will record your voice and save a fingerprint.")
print(" Please speak naturally for about 5 seconds.")
print(" Say something like: 'The quick brown fox jumps over")
print(" the lazy dog. I am enrolling my voice now.'")
print()
input(" Press Enter when ready to record...")
audio = record_audio()
if len(audio) < config.SAMPLE_RATE:
print(" [Error: Recording too short. Try again.]")
return
verifier = EcapaTdnnVerifier()
verifier.enroll(audio)
print()
print(" Enrollment complete! Your voiceprint has been saved.")
print(f" File: {config.VOICEPRINT_PATH}")
print()
if __name__ == "__main__":
main()Create audio utilities and test the full pipeline
The last module handles the raw audio I/O: recording from your microphone and playing back through your speakers. It uses silence detection so the recording stops automatically when you stop speaking.
- Create a new file called audio_utils.py in your project folder and paste the following code:
"""Audio recording and playback utilities."""
import numpy as np
import sounddevice as sd
import soundfile as sf
import config
def record_audio():
"""Record audio from microphone until silence is detected."""
print(" [Recording... speak now]")
samplerate = config.SAMPLE_RATE
threshold = config.SILENCE_THRESHOLD
silence_dur = config.SILENCE_DURATION
max_dur = config.MAX_RECORD_DURATION
chunks = []
silence_frames = 0
silence_limit = int(silence_dur * samplerate / 1024)
max_chunks = int(max_dur * samplerate / 1024)
started = False
def callback(indata, frames, time_info, status):
nonlocal silence_frames, started
if status:
print(f" [Audio status: {status}]")
chunks.append(indata.copy())
energy = np.sqrt(np.mean(indata**2))
if energy > threshold:
started = True
silence_frames = 0
elif started:
silence_frames += 1
with sd.InputStream(samplerate=samplerate, channels=1, dtype="float32",
blocksize=1024, callback=callback):
while True:
sd.sleep(100)
if started and silence_frames >= silence_limit:
break
if len(chunks) >= max_chunks:
break
if not chunks:
return np.zeros((0,), dtype="float32")
audio = np.concatenate(chunks, axis=0).flatten()
print(f" [Recorded {len(audio) / samplerate:.1f}s of audio]")
return audio
def play_audio(audio, samplerate=None):
"""Play a NumPy audio array through the speakers."""
if samplerate is None:
samplerate = config.TTS_SAMPLE_RATE
sd.play(audio, samplerate=samplerate)
sd.wait()
def save_audio(audio, path, samplerate=None):
"""Save audio to a WAV file."""
if samplerate is None:
samplerate = config.SAMPLE_RATE
sf.write(path, audio, samplerate)What does this code do?
- record_audio() opens a sounddevice InputStream with a callback that fires for every 1024-sample block. It calculates the RMS energy of each block to detect speech vs. silence.
- Recording starts when energy exceeds SILENCE_THRESHOLD (you begin speaking) and stops after SILENCE_DURATION seconds of consecutive silence (1.5 seconds by default).
- play_audio() sends audio to your speakers and blocks until playback finishes. It defaults to TTS_SAMPLE_RATE (24000 Hz) since it will mostly play synthesized speech.
- save_audio() writes audio to a WAV file for debugging or archiving recordings.
- Save audio_utils.py.
Seeing a sounddevice PortAudio error?
If you see an error about PortAudio or no input device, make sure your microphone is connected and set as the default recording device in Windows Sound Settings. You can check available devices by running:
python -c "import sounddevice; print(sounddevice.query_devices())"Help me fix a sounddevice or PortAudio error on Windows.
All five files are in place. Time to test the full pipeline by enrolling your voice and verifying it works.
- Run the enrollment script by running this command:
python enroll.py- When prompted, press Enter and speak naturally for about 5 seconds. Say something like: "The quick brown fox jumps over the lazy dog. I am enrolling my voice now."
- Wait for the recording to stop automatically (it detects 1.5 seconds of silence).
You should see output ending with Voiceprint saved to voiceprint.pt and "Enrollment complete!" printed.
- Confirm the voiceprint file was created by running:
ls voiceprint.ptYou should see voiceprint.pt listed in the output.
Now test the transcription and verification together. This quick script records you, transcribes your speech, and checks your voiceprint.
- Run this combined test by running the following command:
python -c "
from audio_utils import record_audio
import stt
from verifier import get_verifier
print('Speak a short sentence...')
audio = record_audio()
text = stt.transcribe(audio)
print(f'Transcribed: {text}')
verifier = get_verifier()
is_match, score = verifier.verify(audio)
print(f'Verified: {is_match} (score: {score:.2f})')
"- When you see "Speak a short sentence...", say a few words clearly into your microphone.
You should see your spoken words printed as text after "Transcribed:" and Verified: True with a similarity score above 0.25.
Verification returning False?
If your score is below 0.25, your test recording may sound very different from the enrollment (different volume, background noise, or distance from mic). Try re-enrolling by running python enroll.py again in a quiet environment.
If you want to skip verification for now and move on, set VERIFICATION_ENABLED = False in config.py. This activates the trust-all fallback.
Help me debug why my voice verification score is too low.
✔️ Awesome, I've got everything!
Great. Double check that all five files are saved: config.py, stt.py, verifier.py, audio_utils.py, and enroll.py.
ⓧ I'd like to double check the full code
Here is the complete audio_utils.py:
"""Audio recording and playback utilities."""
import numpy as np
import sounddevice as sd
import soundfile as sf
import config
def record_audio():
"""Record audio from microphone until silence is detected."""
print(" [Recording... speak now]")
samplerate = config.SAMPLE_RATE
threshold = config.SILENCE_THRESHOLD
silence_dur = config.SILENCE_DURATION
max_dur = config.MAX_RECORD_DURATION
chunks = []
silence_frames = 0
silence_limit = int(silence_dur * samplerate / 1024)
max_chunks = int(max_dur * samplerate / 1024)
started = False
def callback(indata, frames, time_info, status):
nonlocal silence_frames, started
if status:
print(f" [Audio status: {status}]")
chunks.append(indata.copy())
energy = np.sqrt(np.mean(indata**2))
if energy > threshold:
started = True
silence_frames = 0
elif started:
silence_frames += 1
with sd.InputStream(samplerate=samplerate, channels=1, dtype="float32",
blocksize=1024, callback=callback):
while True:
sd.sleep(100)
if started and silence_frames >= silence_limit:
break
if len(chunks) >= max_chunks:
break
if not chunks:
return np.zeros((0,), dtype="float32")
audio = np.concatenate(chunks, axis=0).flatten()
print(f" [Recorded {len(audio) / samplerate:.1f}s of audio]")
return audio
def play_audio(audio, samplerate=None):
"""Play a NumPy audio array through the speakers."""
if samplerate is None:
samplerate = config.TTS_SAMPLE_RATE
sd.play(audio, samplerate=samplerate)
sd.wait()
def save_audio(audio, path, samplerate=None):
"""Save audio to a WAV file."""
if samplerate is None:
samplerate = config.SAMPLE_RATE
sf.write(path, audio, samplerate)Your system can now hear, transcribe, and verify who is speaking. Next up, you will build the voice output and conversation intelligence: text-to-speech synthesis with two distinct voices and the LLM backend that powers your AI assistants.
Build the TTS and LLM Modules
In the previous step, you built the input half of your conversation system. Your project can record audio, transcribe it to text, and verify who is speaking.
Now you need the output half. When the AI assistants compose a reply, that text needs to become audible speech. You also need a way to connect to your local LLM so the assistants can actually think. In this step, you will build both the text-to-speech module and the LLM module, then wire them together in a lightweight text-only test harness.
In this step, get ready to:
- Create a TTS backend that synthesizes speech with two distinct voices.
- Create an LLM backend that talks to Ollama via the OpenAI-compatible API.
- Build and run a text-only conversation harness to test the full three-way loop without audio.
Create the text-to-speech backend
For the assistants to speak aloud, you need a module that converts text into audio. Kokoro is a lightweight TTS model that runs locally on your GPU and supports multiple built-in voices.
- Create a new file called tts_backend.py in your project folder and paste the following code:
"""Text-to-speech backend using Kokoro."""
import numpy as np
from kokoro import KPipeline
import config
_pipeline = None
def _get_pipeline():
global _pipeline
if _pipeline is None:
print(" [Loading Kokoro TTS model...]")
_pipeline = KPipeline(lang_code=config.TTS_LANG_CODE)
return _pipeline
def synthesize(text, voice=None):
"""Convert text to speech audio.
Args:
text: The text to synthesize.
voice: Voice identifier (e.g., 'af_heart'). Uses config default if None.
Returns:
NumPy array of audio samples at 24000 Hz.
"""
if voice is None:
voice = config.ASSISTANT_1_VOICE
pipeline = _get_pipeline()
audio_chunks = []
generator = pipeline(text, voice=voice, speed=1.0)
for _, _, audio in generator:
audio_chunks.append(audio)
if not audio_chunks:
return np.zeros((0,), dtype="float32")
return np.concatenate(audio_chunks)What does this code do?
This module follows the same lazy-loading pattern as your STT module. The _get_pipeline() function creates the Kokoro pipeline only on the first call, avoiding repeated model loads.
The synthesize() function feeds text through the pipeline and collects audio chunks from the generator. Kokoro yields audio in segments, so you concatenate them into one continuous NumPy array at 24000 Hz.
- Save tts_backend.py.
- Verify the module loads cleanly by running:
python -c "from tts_backend import synthesize; print('TTS module OK')"You should see TTS module OK printed without errors. The model itself has not loaded yet because of the lazy initialization.
Seeing an import error?
If you see an error about kokoro not being found, make sure your virtual environment is activated. The kokoro package and its misaki[en] dependency should have been installed in Step 1.
Help me debug my Kokoro TTS import error
How do the two voices work?
Your config.py already defines two voice identifiers: af_heart for Atlas and af_sarah for Sage. When the orchestrator calls synthesize() with a different voice parameter, Kokoro generates speech with a distinct vocal character for each assistant.
✔️ Awesome, I've got everything!
Great. Double check you have saved tts_backend.py before moving on.
ⓧ I'd like to double check the full code
"""Text-to-speech backend using Kokoro."""
import numpy as np
from kokoro import KPipeline
import config
_pipeline = None
def _get_pipeline():
global _pipeline
if _pipeline is None:
print(" [Loading Kokoro TTS model...]")
_pipeline = KPipeline(lang_code=config.TTS_LANG_CODE)
return _pipeline
def synthesize(text, voice=None):
"""Convert text to speech audio.
Args:
text: The text to synthesize.
voice: Voice identifier (e.g., 'af_heart'). Uses config default if None.
Returns:
NumPy array of audio samples at 24000 Hz.
"""
if voice is None:
voice = config.ASSISTANT_1_VOICE
pipeline = _get_pipeline()
audio_chunks = []
generator = pipeline(text, voice=voice, speed=1.0)
for _, _, audio in generator:
audio_chunks.append(audio)
if not audio_chunks:
return np.zeros((0,), dtype="float32")
return np.concatenate(audio_chunks)Create the LLM backend and text-only test harness
Your assistants need a brain. Ollama exposes an OpenAI-compatible chat completions endpoint, which means you can use the standard openai Python package to talk to your local model.
- Create a new file called llm_backend.py in your project folder and paste the following code:
"""LLM backend using OpenAI-compatible API (Ollama)."""
from openai import OpenAI
import config
_client = None
def _get_client():
global _client
if _client is None:
_client = OpenAI(
base_url=config.OLLAMA_BASE_URL,
api_key="ollama",
)
return _client
def get_reply(messages, system_prompt, model=None):
"""Get a reply from the LLM.
Args:
messages: List of conversation messages [{"role": ..., "content": ...}].
system_prompt: System prompt for this assistant.
model: Model name. Uses config default if None.
Returns:
Response text string.
"""
if model is None:
model = config.OLLAMA_MODEL
client = _get_client()
full_messages = [{"role": "system", "content": system_prompt}]
full_messages.extend(messages)
response = client.chat.completions.create(
model=model,
messages=full_messages,
)
return response.choices[0].message.contentWhat does this code do?
The _get_client() function creates an OpenAI client that points at your local Ollama instance instead of OpenAI's servers. The api_key="ollama" value is required by the client library but ignored by Ollama.
The get_reply() function prepends a system prompt (the assistant's personality) to the conversation messages, then calls client.chat.completions.create(). This is the same API shape you would use with OpenAI's cloud models, making the backend trivially swappable later.
- Save llm_backend.py.
- Verify the module loads cleanly by running:
python -c "from llm_backend import get_reply; print('LLM module OK')"You should see LLM module OK printed. No connection to Ollama happens yet because of lazy initialization.
Seeing a ModuleNotFoundError for openai?
Make sure your virtual environment is active. The openai package was installed in Step 1. If you are testing on a different machine (like a non-GPU laptop), you only need pip install openai to run the text-only harness.
Help me fix my openai import error
Why use the OpenAI client for a local model?
Ollama implements the same chat completions API contract that OpenAI uses. By coding against that standard interface, you can swap in any provider (OpenAI, Anthropic via an adapter, or another local server) without rewriting your conversation logic. The Secret Mission later takes advantage of this.
✔️ Awesome, I've got everything!
Great. Double check you have saved llm_backend.py before moving on.
ⓧ I'd like to double check the full code
"""LLM backend using OpenAI-compatible API (Ollama)."""
from openai import OpenAI
import config
_client = None
def _get_client():
global _client
if _client is None:
_client = OpenAI(
base_url=config.OLLAMA_BASE_URL,
api_key="ollama",
)
return _client
def get_reply(messages, system_prompt, model=None):
"""Get a reply from the LLM.
Args:
messages: List of conversation messages [{"role": ..., "content": ...}].
system_prompt: System prompt for this assistant.
model: Model name. Uses config default if None.
Returns:
Response text string.
"""
if model is None:
model = config.OLLAMA_MODEL
client = _get_client()
full_messages = [{"role": "system", "content": system_prompt}]
full_messages.extend(messages)
response = client.chat.completions.create(
model=model,
messages=full_messages,
)
return response.choices[0].message.contentNow you need a way to test the conversation logic without any audio or GPU involvement. A text-only harness lets you iterate on assistant personalities quickly from any machine.
- Create a new file called test_text_loop.py in your project folder. Start with the imports and the transcript formatter:
"""Text-only conversation harness (Phase 1).
No audio, no GPU, no TTS/STT required.
Only needs: pip install openai
Plus: Ollama running locally with your model pulled.
This lets you test the full three-way conversation logic,
LLM personalities, and labeled transcript formatting
from any machine (including Windows on ARM with no GPU).
"""
import config
import llm_backend
def format_transcript_for_llm(transcript):
"""Convert labeled transcript to LLM message format."""
messages = []
for entry in transcript:
speaker = entry["speaker"]
text = entry["text"]
if speaker == "Human":
messages.append({"role": "user", "content": f"[Human]: {text}"})
else:
messages.append({"role": "assistant", "content": f"[{speaker}]: {text}"})
return messagesWhat does this code do?
The format_transcript_for_llm() function converts your internal transcript (a list of dictionaries with speaker labels) into the message format the LLM expects. Human messages get the user role, and assistant messages get the assistant role. Each message includes the speaker label in brackets so both assistants know who said what.
- Now add the main() function below the code you just pasted in test_text_loop.py:
def main():
print("=" * 60)
print(" THREE-WAY CONVERSATION (Text Mode)")
print("=" * 60)
print()
print(f" Assistants: {config.ASSISTANT_1_NAME} and {config.ASSISTANT_2_NAME}")
print(f" LLM: {config.OLLAMA_MODEL} via Ollama")
print(f" Mode: Text-only (no audio, no GPU needed)")
print()
print(" Type your messages. Type 'q' to quit.")
print("=" * 60)
transcript = []
while True:
print()
user_text = input(f" [Human] > ").strip()
if user_text.lower() == "q":
break
if not user_text:
continue
transcript.append({"speaker": "Human", "text": user_text})
# --- Assistant 1 Turn ---
print(f"\n [{config.ASSISTANT_1_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_1 = llm_backend.get_reply(messages, config.ASSISTANT_1_SYSTEM)
print(f" [{config.ASSISTANT_1_NAME}]: {reply_1}")
transcript.append({"speaker": config.ASSISTANT_1_NAME, "text": reply_1})
except Exception as e:
print(f" [{config.ASSISTANT_1_NAME} error: {e}]")
# --- Assistant 2 Turn ---
print(f"\n [{config.ASSISTANT_2_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_2 = llm_backend.get_reply(messages, config.ASSISTANT_2_SYSTEM)
print(f" [{config.ASSISTANT_2_NAME}]: {reply_2}")
transcript.append({"speaker": config.ASSISTANT_2_NAME, "text": reply_2})
except Exception as e:
print(f" [{config.ASSISTANT_2_NAME} error: {e}]")
# Print final transcript
print("\n" + "=" * 60)
print(" FULL CONVERSATION TRANSCRIPT")
print("=" * 60)
for i, entry in enumerate(transcript, 1):
print(f" {i}. [{entry['speaker']}]: {entry['text']}")
print("=" * 60)
print(f" Total turns: {len(transcript)}")
print(" Conversation ended.")
if __name__ == "__main__":
main()What does this code do?
The main loop reads your typed input, appends it to the transcript, then asks each assistant for a reply in turn. Atlas responds first (using ASSISTANT_1_SYSTEM as its personality), then Sage responds (with the updated transcript that includes Atlas's reply).
Each assistant sees the full labeled transcript every turn. This means Sage can react to what Atlas just said, creating a natural three-way discussion. When you type q, the loop prints the entire conversation with numbered turns.
- Save test_text_loop.py. You will run it in the next substep to verify everything works together.
✔️ Awesome, I've got everything!
Great. Double check you have saved test_text_loop.py before moving on.
ⓧ I'd like to double check the full code
"""Text-only conversation harness (Phase 1).
No audio, no GPU, no TTS/STT required.
Only needs: pip install openai
Plus: Ollama running locally with your model pulled.
This lets you test the full three-way conversation logic,
LLM personalities, and labeled transcript formatting
from any machine (including Windows on ARM with no GPU).
"""
import config
import llm_backend
def format_transcript_for_llm(transcript):
"""Convert labeled transcript to LLM message format."""
messages = []
for entry in transcript:
speaker = entry["speaker"]
text = entry["text"]
if speaker == "Human":
messages.append({"role": "user", "content": f"[Human]: {text}"})
else:
messages.append({"role": "assistant", "content": f"[{speaker}]: {text}"})
return messages
def main():
print("=" * 60)
print(" THREE-WAY CONVERSATION (Text Mode)")
print("=" * 60)
print()
print(f" Assistants: {config.ASSISTANT_1_NAME} and {config.ASSISTANT_2_NAME}")
print(f" LLM: {config.OLLAMA_MODEL} via Ollama")
print(f" Mode: Text-only (no audio, no GPU needed)")
print()
print(" Type your messages. Type 'q' to quit.")
print("=" * 60)
transcript = []
while True:
print()
user_text = input(f" [Human] > ").strip()
if user_text.lower() == "q":
break
if not user_text:
continue
transcript.append({"speaker": "Human", "text": user_text})
# --- Assistant 1 Turn ---
print(f"\n [{config.ASSISTANT_1_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_1 = llm_backend.get_reply(messages, config.ASSISTANT_1_SYSTEM)
print(f" [{config.ASSISTANT_1_NAME}]: {reply_1}")
transcript.append({"speaker": config.ASSISTANT_1_NAME, "text": reply_1})
except Exception as e:
print(f" [{config.ASSISTANT_1_NAME} error: {e}]")
# --- Assistant 2 Turn ---
print(f"\n [{config.ASSISTANT_2_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_2 = llm_backend.get_reply(messages, config.ASSISTANT_2_SYSTEM)
print(f" [{config.ASSISTANT_2_NAME}]: {reply_2}")
transcript.append({"speaker": config.ASSISTANT_2_NAME, "text": reply_2})
except Exception as e:
print(f" [{config.ASSISTANT_2_NAME} error: {e}]")
# Print final transcript
print("\n" + "=" * 60)
print(" FULL CONVERSATION TRANSCRIPT")
print("=" * 60)
for i, entry in enumerate(transcript, 1):
print(f" {i}. [{entry['speaker']}]: {entry['text']}")
print("=" * 60)
print(f" Total turns: {len(transcript)}")
print(" Conversation ended.")
if __name__ == "__main__":
main()Test both modules independently
With all three files saved, it is time to confirm they work. You will test the LLM conversation loop first (text only), then test the TTS voices on your GPU machine.
- Make sure Ollama is running on the machine you are testing from.
- Run the text-only conversation harness by running:
python test_text_loop.py- Type a message (something like What's your favorite thing about space?) and press Enter.
- You should see labeled replies from both Atlas and Sage, each with a distinct personality.
- Try a few more turns to see the assistants reference each other's replies.
- Type q and press Enter to exit. You should see the full labeled transcript printed.
Seeing a connection error?
If you see a connection refused error, Ollama is not running or is not reachable at http://localhost:11434. Open a separate terminal and start Ollama. If Ollama is running on a different machine, update OLLAMA_BASE_URL in config.py to point at the correct IP address.
If you see a model not found error, make sure OLLAMA_MODEL in config.py matches the exact name of the model you pulled in Step 1.
Help me debug my Ollama connection in test_text_loop.py
Now test the TTS module on your GPU machine. Run the following test script to hear both voices:
- Synthesize and play a test sentence with each voice by running:
python -c "
import tts_backend
from audio_utils import play_audio
print('Testing Atlas voice...')
audio1 = tts_backend.synthesize('Hello, I am Atlas. I love exploring new ideas.', voice='af_heart')
play_audio(audio1)
print('Testing Sage voice...')
audio2 = tts_backend.synthesize('Hello, I am Sage. I prefer thoughtful reflection.', voice='af_sarah')
play_audio(audio2)
print('Both voices working!')
"- You should hear two distinctly different voices speak through your speakers. The first time you run this, Kokoro will download its model weights (this takes a moment).
Not hearing any audio?
Check that your system audio output is not muted and that the correct playback device is selected. If you see a sounddevice error about no output device, verify your speaker or headphone configuration in your Windows sound settings.
If you see an espeak-ng error, make sure you installed espeak-ng in Step 1 and that it is available on your system PATH.
Help me troubleshoot my Kokoro TTS audio playback
Both halves of your pipeline are working. Your assistants can think (LLM) and speak (TTS). Next up, you will wire everything together into the full voice conversation loop where you speak aloud and both assistants respond with their unique voices.
Wire Up the Three-Way Conversation Loop
You have a working STT module, a speaker verifier with your enrolled voiceprint, a TTS engine with two distinct voices, and an LLM backend that talks to Ollama. Each piece works in isolation. Now you need to connect them into a single interactive loop.
In this step, you will build the orchestrator that ties all modules together. The loop records your voice, verifies it is you, transcribes what you said, sends the conversation to both AI assistants, and plays their responses aloud in distinct voices.
In this step, get ready to:
- Create the main orchestrator with imports and a transcript helper function.
- Implement the human turn and both assistant response turns with LLM, TTS, and audio playback.
- Add a clean exit path and run a full multi-turn voice conversation.
Create the orchestrator imports and transcript helper
The orchestrator needs to pull in every module you have built so far. It also needs a helper function that converts your labeled transcript list into the message format the LLM expects.
- Create a new file called main.py in your project folder.
- Add the imports and transcript helper by pasting this code:
"""Three-way voice conversation orchestrator."""
import config
import stt
import llm_backend
import tts_backend
from audio_utils import record_audio, play_audio
from verifier import get_verifier
def format_transcript_for_llm(transcript):
"""Convert labeled transcript to LLM message format."""
messages = []
for entry in transcript:
speaker = entry["speaker"]
text = entry["text"]
if speaker == "Human":
messages.append({"role": "user", "content": f"[Human]: {text}"})
else:
messages.append({"role": "assistant", "content": f"[{speaker}]: {text}"})
return messagesWhat does this code do?
The imports bring in every backend module: config for settings, stt for transcription, llm_backend for AI replies, tts_backend for speech synthesis, plus record_audio and play_audio from your audio utilities.
The format_transcript_for_llm() function converts your labeled transcript (list of dicts with speaker and text) into the OpenAI-style messages format the LLM expects. Human entries become user messages and assistant entries become assistant messages.
You will not see a change yet. The next chunk builds the main function that uses this helper.
Build the main loop with the human turn
Each turn of the conversation follows a pipeline: record your audio, check that it is long enough, verify your voiceprint, transcribe the speech, and append the labeled entry to the shared transcript.
- In main.py, add the main() function below format_transcript_for_llm() by pasting this code:
def main():
print("=" * 60)
print(" THREE-WAY VOICE CONVERSATION")
print("=" * 60)
print()
print(f" Assistants: {config.ASSISTANT_1_NAME} and {config.ASSISTANT_2_NAME}")
print(f" LLM: {config.OLLAMA_MODEL} via Ollama")
print(f" Verification: {'ECAPA-TDNN' if config.VERIFICATION_ENABLED else 'Trust All'}")
print()
print(" Press Enter to speak, or type 'q' + Enter to quit.")
print("=" * 60)
# Initialize backends
verifier = get_verifier()
transcript = []
while True:
print()
user_input = input(" [Press Enter to speak, 'q' to quit] > ")
if user_input.strip().lower() == "q":
break
# --- Human Turn ---
audio = record_audio()
if len(audio) < config.SAMPLE_RATE * 0.5:
print(" [Too short, try again]")
continue
# Verify speaker
is_verified, score = verifier.verify(audio)
if not is_verified:
print(f" [Speaker not verified (score: {score:.2f}). Skipping turn.]")
continue
if config.VERIFICATION_ENABLED:
print(f" [Speaker verified (score: {score:.2f})]")
# Transcribe
text = stt.transcribe(audio)
if not text:
print(" [No speech detected, try again]")
continue
print(f" [Human]: {text}")
transcript.append({"speaker": "Human", "text": text})What's happening here?
- The opening banner prints the active configuration so you can confirm which model and verification mode are in use.
- get_verifier() loads either the ECAPA-TDNN verifier (if VERIFICATION_ENABLED is True) or the trust-all fallback.
- The loop waits for Enter, then records until silence is detected. Audio shorter than 0.5 seconds is rejected as a false trigger.
- The verifier compares your voice embedding against your enrolled voiceprint. If the score falls below the threshold, the turn is skipped entirely.
- Only after verification passes does the STT transcription run. The result is printed and appended to the shared transcript list.
You will not see a change yet. The next chunk adds the assistant response turns.
Add the assistant response turns with TTS playback
After your human turn is recorded and transcribed, both assistants take their turn. Each one receives the full labeled transcript, generates a reply, appends it to the transcript, synthesizes audio with its own voice, and plays it through your speakers.
- In main.py, add this code directly below the transcript.append({"speaker": "Human", "text": text}) line, at the same indentation level (inside the while True loop):
# --- Assistant 1 Turn ---
print(f"\n [{config.ASSISTANT_1_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_1 = llm_backend.get_reply(
messages, config.ASSISTANT_1_SYSTEM
)
print(f" [{config.ASSISTANT_1_NAME}]: {reply_1}")
transcript.append({"speaker": config.ASSISTANT_1_NAME, "text": reply_1})
# Synthesize and play
audio_1 = tts_backend.synthesize(reply_1, voice=config.ASSISTANT_1_VOICE)
play_audio(audio_1)
except Exception as e:
print(f" [{config.ASSISTANT_1_NAME} error: {e}]")
# --- Assistant 2 Turn ---
print(f"\n [{config.ASSISTANT_2_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_2 = llm_backend.get_reply(
messages, config.ASSISTANT_2_SYSTEM
)
print(f" [{config.ASSISTANT_2_NAME}]: {reply_2}")
transcript.append({"speaker": config.ASSISTANT_2_NAME, "text": reply_2})
# Synthesize and play
audio_2 = tts_backend.synthesize(reply_2, voice=config.ASSISTANT_2_VOICE)
play_audio(audio_2)
except Exception as e:
print(f" [{config.ASSISTANT_2_NAME} error: {e}]")What does this code do?
- Each assistant's turn is wrapped in a try/except block so a failure in one backend (network timeout, model error) does not crash the entire loop.
- format_transcript_for_llm() is called fresh each time so the assistant sees everything said so far, including the other assistant's previous reply.
- After the LLM reply is appended to the transcript, tts_backend.synthesize() converts it to audio using the configured voice (af_heart for Atlas, af_sarah for Sage).
- play_audio() blocks until the audio finishes playing, so the assistants speak one at a time.
You will not see a change yet. The next chunk closes the loop and adds the final transcript output.
- Still in main.py, add this code after the while True loop (back at the same indentation as the while keyword itself, inside main()):
# Print final transcript
print("\n" + "=" * 60)
print(" FULL CONVERSATION TRANSCRIPT")
print("=" * 60)
for entry in transcript:
print(f" [{entry['speaker']}]: {entry['text']}")
print("=" * 60)
print(" Conversation ended.")
if __name__ == "__main__":
main()What does this code do?
When you type q and press Enter, the loop breaks and the full labeled transcript prints. Every utterance shows its speaker label so you can review the entire conversation at a glance.
The if __name__ == "__main__" guard means running python main.py starts the conversation loop, but importing the file from elsewhere does not.
- Save main.py.
- Make sure Ollama is running with your model available, then run the full conversation loop:
python main.py- Press Enter when prompted, speak a sentence aloud, then wait. You should hear Atlas respond in one voice, followed by Sage in a different voice.
- Have a 2-3 turn conversation, then type q and press Enter. You should see the full labeled transcript printed with every speaker identified.
Not hearing any audio or seeing errors?
- If you see a speaker verification failure, try setting VERIFICATION_ENABLED = False in config.py to use the trust-all fallback while debugging.
- If you see a connection error from the LLM, confirm Ollama is running and your model name in config.py matches what ollama list shows.
- If the recording never stops, check your microphone is selected as the default input device in Windows Sound settings.
Help me debug my main.py conversation loop
✔️ Awesome, I've got everything!
Great. Double check you have saved main.py and confirmed both assistants respond with distinct voices.
ⓧ I'd like to double check the full code
"""Three-way voice conversation orchestrator."""
import config
import stt
import llm_backend
import tts_backend
from audio_utils import record_audio, play_audio
from verifier import get_verifier
def format_transcript_for_llm(transcript):
"""Convert labeled transcript to LLM message format."""
messages = []
for entry in transcript:
speaker = entry["speaker"]
text = entry["text"]
if speaker == "Human":
messages.append({"role": "user", "content": f"[Human]: {text}"})
else:
messages.append({"role": "assistant", "content": f"[{speaker}]: {text}"})
return messages
def main():
print("=" * 60)
print(" THREE-WAY VOICE CONVERSATION")
print("=" * 60)
print()
print(f" Assistants: {config.ASSISTANT_1_NAME} and {config.ASSISTANT_2_NAME}")
print(f" LLM: {config.OLLAMA_MODEL} via Ollama")
print(f" Verification: {'ECAPA-TDNN' if config.VERIFICATION_ENABLED else 'Trust All'}")
print()
print(" Press Enter to speak, or type 'q' + Enter to quit.")
print("=" * 60)
# Initialize backends
verifier = get_verifier()
transcript = []
while True:
print()
user_input = input(" [Press Enter to speak, 'q' to quit] > ")
if user_input.strip().lower() == "q":
break
# --- Human Turn ---
audio = record_audio()
if len(audio) < config.SAMPLE_RATE * 0.5:
print(" [Too short, try again]")
continue
# Verify speaker
is_verified, score = verifier.verify(audio)
if not is_verified:
print(f" [Speaker not verified (score: {score:.2f}). Skipping turn.]")
continue
if config.VERIFICATION_ENABLED:
print(f" [Speaker verified (score: {score:.2f})]")
# Transcribe
text = stt.transcribe(audio)
if not text:
print(" [No speech detected, try again]")
continue
print(f" [Human]: {text}")
transcript.append({"speaker": "Human", "text": text})
# --- Assistant 1 Turn ---
print(f"\n [{config.ASSISTANT_1_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_1 = llm_backend.get_reply(
messages, config.ASSISTANT_1_SYSTEM
)
print(f" [{config.ASSISTANT_1_NAME}]: {reply_1}")
transcript.append({"speaker": config.ASSISTANT_1_NAME, "text": reply_1})
# Synthesize and play
audio_1 = tts_backend.synthesize(reply_1, voice=config.ASSISTANT_1_VOICE)
play_audio(audio_1)
except Exception as e:
print(f" [{config.ASSISTANT_1_NAME} error: {e}]")
# --- Assistant 2 Turn ---
print(f"\n [{config.ASSISTANT_2_NAME} is thinking...]")
try:
messages = format_transcript_for_llm(transcript)
reply_2 = llm_backend.get_reply(
messages, config.ASSISTANT_2_SYSTEM
)
print(f" [{config.ASSISTANT_2_NAME}]: {reply_2}")
transcript.append({"speaker": config.ASSISTANT_2_NAME, "text": reply_2})
# Synthesize and play
audio_2 = tts_backend.synthesize(reply_2, voice=config.ASSISTANT_2_VOICE)
play_audio(audio_2)
except Exception as e:
print(f" [{config.ASSISTANT_2_NAME} error: {e}]")
# Print final transcript
print("\n" + "=" * 60)
print(" FULL CONVERSATION TRANSCRIPT")
print("=" * 60)
for entry in transcript:
print(f" [{entry['speaker']}]: {entry['text']}")
print("=" * 60)
print(" Conversation ended.")
if __name__ == "__main__":
main()Your entire voice conversation system is now working end-to-end. You speak, the system verifies your identity, transcribes your words, and two AI assistants respond with their own voices and personalities. Next up, the Secret Mission will challenge you to add a second LLM provider and export your conversations.
Secret mission
Add Claude API Backend and Conversation Export
Your voice conversation system works with Ollama, but what if you want to bring in a cloud LLM for richer responses? In this secret mission, you will add the Anthropic Claude API as a second LLM backend and build a JSON export system so you can review past conversations.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Project folder (all Python files, voiceprint.pt, conversation JSON exports).
- Python virtual environment with installed packages (PyTorch, faster-whisper, SpeechBrain, Kokoro, sounddevice, openai, anthropic).
- espeak-ng system application (Windows MSI install).
- Cached ML models in the HuggingFace hub cache (SpeechBrain ECAPA-TDNN, faster-whisper base.en, Kokoro).
Keep everything running
No action needed. Choose this if you are still actively building on the project or plan to continue experimenting with voice conversations.
- Your project files, virtual environment, and cached models stay on disk.
- espeak-ng remains installed as a system application.
- Ollama continues to serve models at http://localhost:11434 whenever it is running.
Pause - I'll come back to this later
Shut down running processes to free up memory. Your files and models stay on disk for when you return.
- If your virtual environment is active, deactivate it by running:
deactivate- If Ollama is running in the background, close it from the system tray (right-click the Ollama icon and select Quit).
- Your project files, cached models, and espeak-ng installation remain untouched for next time.
Delete - I don't want to use this again
Remove all project resources and start fresh. Follow each step below to fully clean up.
- Delete your project folder (the folder containing config.py, main.py, and all other project files). Open a terminal and navigate to the parent folder, then run:
rmdir /s /q your-project-folder-name- This removes the virtual environment, all Python files, voiceprint.pt, and any conversation JSON exports inside the folder.
- Uninstall espeak-ng. Open Settings, select Apps, then Installed apps. Search for eSpeak NG and click Uninstall.
- Delete the cached ML models. These are stored in the HuggingFace hub cache. Run:
rmdir /s /q %USERPROFILE%\.cache\huggingface\hubWhat does this delete?
The HuggingFace cache at C:\Users\username\.cache\huggingface\hub stores all downloaded models from SpeechBrain, faster-whisper, and Kokoro. Deleting it removes several gigabytes of model weights.
If you use other HuggingFace-based projects, this will also remove their cached models. They will re-download automatically next time they run.
Well Done!
Well Done!
Nice work! You just built a fully local, GPU-accelerated voice conversation system that integrates four distinct ML disciplines into a single interactive pipeline.
You've learned how to:
- Build a complete voice conversation pipeline integrating speech-to-text (faster-whisper), speaker verification (SpeechBrain ECAPA-TDNN), text-to-speech (Kokoro), and LLM inference (Ollama) into a turn-based loop.
- Enroll a voice fingerprint and verify speaker identity using cosine similarity on 192-dimensional speaker embeddings.
- Architect a modular backend system with swappable providers behind clean interfaces, letting you switch any component with a single config change.
- Secret Mission: Added Claude API as an alternative LLM backend for hybrid conversations and exported labeled transcripts to timestamped JSON files.
Ready to quiz yourself?