Build a Local Voice Agent in Rust

Build a fully local STT → LLM → TTS voice pipeline in Rust.

Introduction

30 Second Summary

Voice assistants like Siri and Alexa send your audio to remote servers for processing. What if you could build one that runs entirely on your machine, with no cloud dependency at all?

In this project, you will build a terminal-based voice agent in Rust that listens to your microphone, transcribes speech with Whisper, generates responses with a local LLM, and speaks them back through your speakers.

What You'll Build

A Rust binary that implements the complete STT to LLM to TTS pipeline, running a real-time conversation loop entirely on your local machine.

By the end of this project, you'll have:

  • A working voice assistant that captures your microphone, detects when you stop speaking, transcribes your speech, and responds out loud.
  • A complete speech-to-text to LLM to text-to-speech pipeline using Whisper, Ollama, and Kokoro TTS.
  • A single-binary Rust project with real-time audio processing and async HTTP communication.
  • Secret Mission: Add barge-in support so you can interrupt the agent mid-speech.

Are there any prerequisites?

This project assumes basic Rust knowledge including traits, enums, ownership, and cargo. You should also be comfortable navigating a Linux terminal.

Before We Start

Before We Start...

Install System Dependencies

Before writing any Rust code, you need the infrastructure your voice agent depends on. The LLM brain (Gemma 4 via Ollama) needs to be running locally, and several system libraries need to be in place for audio capture, Whisper compilation, and speech synthesis.

In this step you will install all the external dependencies so that later steps can focus purely on building the voice pipeline in Rust.

In this step, get ready to:
  • Install Ollama and pull the Gemma 4 E4B model.
  • Install system libraries for audio, Whisper, and Kokoro TTS.
  • Verify Ollama is serving Gemma 4 E4B with a test request.
Install Ollama and pull Gemma 4 E4B

Ollama is a lightweight server that runs large language models locally. It exposes an HTTP API on localhost:11434 that your Rust code will call to get chat completions from Gemma 4.

  • Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

This downloads and runs Ollama's official install script. It installs the ollama binary, sets up a systemd service, and starts the Ollama server automatically.

  • Confirm Ollama installed successfully by running:
ollama -v

✔️ I see a version number

Ollama is installed and running. Next, you will pull the Gemma 4 E4B model.

ⓧ Command not found

The install script may not have added ollama to your PATH.

  • Close and reopen your terminal, then try ollama -v again.
  • If it still fails, check that /usr/local/bin/ollama exists. You may need to re-run the install script with sudo.

Now you need to download the Gemma 4 E4B model. This is a large download (several GB), so it may take a few minutes depending on your connection.

  • Pull the Gemma 4 E4B model by running:
ollama pull gemma4:e4b

What is Gemma 4 E4B?

Gemma 4 E4B is Google's edge-optimized LLM with 4.5 billion effective parameters. It supports a 128K context window and runs comfortably on a CPU with 10 GB+ RAM.

The e4b tag selects a quantized variant that balances quality with performance on machines without a GPU.

  • Wait for the download to complete. You should see a progress bar followed by a success message once the model is ready.

Download seems stuck?

On a slow connection this can take 10+ minutes. If it times out, run ollama pull gemma4:e4b again and it will resume from where it left off.

help me troubleshoot the Ollama model download

Install system libraries for audio, Whisper, and TTS

Your voice agent depends on several system-level C libraries that Rust crates link against at compile time. Without these, cargo build will fail with linker errors.

  • Install all four required system packages by running:
sudo apt update && sudo apt install -y libasound2-dev cmake g++ espeak-ng

What are these packages for?

  • libasound2-dev provides ALSA headers so cpal can talk to your microphone and speakers on Linux.
  • cmake is required by whisper-rs to compile whisper.cpp from source during cargo build.
  • g++ provides the C++ compiler that whisper.cpp needs.
  • espeak-ng is a phoneme engine that Kokoro TTS uses to convert text into phoneme sequences before synthesizing speech.
  • Verify the packages installed by checking that cmake is accessible:
cmake --version

You should see a version number printed to the terminal.

Permission denied on apt install?

Make sure you included sudo at the start of the command. If your user does not have sudo access, contact your system administrator.

help me fix Linux package installation permissions

Verify Ollama is serving Gemma 4

Your Rust code will send HTTP requests to Ollama's chat endpoint. Before writing any code, confirm the full path works: Ollama receives a request, loads the model, and returns a response.

  • Send a test request to Ollama's chat API by running:
curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:e4b",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": false
}'

What does this request do?

This sends a POST request to http://localhost:11434/api/chat with a JSON body. The model field tells Ollama which model to use.

The messages array is the conversation history. Setting stream to false returns the full response in one JSON object instead of streaming tokens.

You should see a JSON response containing a message field with the assistant's reply. The first request may take 30-60 seconds while Ollama loads the model into memory.

Connection refused?

If you see a connection refused error, the Ollama service may not be running.

  • Start it manually by running ollama serve in a separate terminal.
  • Or restart the systemd service: sudo systemctl start ollama.

help me get Ollama running on Linux

Your machine now has everything the voice agent needs: an LLM server with Gemma 4 loaded, ALSA headers for audio, a C++ toolchain for Whisper, and a phoneme engine for TTS. Next up, you will scaffold the Rust project and download the AI model files.

Scaffold the Rust Project

In the last step you installed Ollama, pulled Gemma 4 E4B, and confirmed all your system libraries are in place.

Now it is time to create the actual Rust project and pull in the machine learning models that power each stage of the voice pipeline. By the end of this step you will have a compiling Cargo workspace with every dependency resolved and every model file on disk, ready to write real code.

In this step, get ready to:
  • Create the Rust project and declare all dependencies in Cargo.toml.
  • Download the Whisper and Kokoro model files into a models/ directory.
  • Verify the project compiles successfully.
Create the Rust project and add dependencies

Every Rust project starts with cargo new, which generates a skeleton with a Cargo.toml manifest and a src/main.rs entry point.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create a new Rust binary project called voice-agent and move into it by running these commands:
cargo new voice-agent
cd voice-agent
  • Confirm the project was created by running:
ls

You should see Cargo.toml and a src/ directory listed in the output.

Now you need to declare the crates your voice agent depends on. Open Cargo.toml in your editor and replace its entire contents with the following.

  • Replace the contents of Cargo.toml with the following dependency manifest:
[package]
name = "voice-agent"
version = "0.1.0"
edition = "2021"

[dependencies]
# Speech-to-text: Rust bindings for whisper.cpp
whisper-rs = "0.16"
# Audio I/O: cross-platform audio capture and playback
cpal = "0.17"
# Text-to-speech: Kokoro neural TTS via ONNX
tts-rs = { version = "2026.2", features = ["kokoro"] }
# HTTP client for Ollama API
reqwest = { version = "0.12", features = ["json"] }
# Async runtime
tokio = { version = "1", features = ["full"] }
# JSON serialization
serde = { version = "1", features = ["derive"] }
serde_json = "1"
# Error handling
anyhow = "1"

What do these dependencies do?

  • whisper-rs provides Rust bindings to whisper.cpp for local speech-to-text.
  • cpal handles real-time microphone capture and speaker playback on Linux via ALSA.
  • tts-rs with the kokoro feature gives you neural text-to-speech through ONNX Runtime.
  • reqwest sends HTTP requests to Ollama's local API for LLM chat completions.
  • tokio provides the async runtime needed by reqwest and the main event loop.
  • anyhow simplifies error handling with a unified Result type.
  • Save Cargo.toml.

Editor not opening?

If you are using Cursor or VS Code, open the voice-agent folder directly from the editor's file menu so it recognizes the Cargo workspace.

help me open my Rust project in my editor

Download the Whisper and Kokoro model files

The voice agent needs two sets of model files: a Whisper model for speech-to-text and a Kokoro model for text-to-speech. Both live in a models/ directory inside your project.

  • Create the model directories by running these commands:
mkdir -p models/kokoro
  • Download the Whisper base English model (142 MB) by running:
curl -L --output models/ggml-base.en.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin

This download is 142 MB and may take a minute or two depending on your connection.

  • Download the Kokoro int8 ONNX model (88 MB) by running:
curl -L --output models/kokoro/kokoro-v1.0.int8.onnx https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/kokoro-v1.0.int8.onnx
  • Download the Kokoro voices file (required for all model variants) by running:
curl -L --output models/kokoro/voices-v1.0.bin https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/voices-v1.0.bin

Why the int8 model?

The int8 quantized model is only 88 MB compared to 310 MB for the full-precision version. On a CPU-only machine like yours, the int8 variant runs faster while producing nearly identical audio quality.

  • Verify all model files are in place by running:
ls models/ models/kokoro/

You should see ggml-base.en.bin under models/ and both kokoro-v1.0.int8.onnx and voices-v1.0.bin under models/kokoro/.

Download failing or timing out?

If curl times out, try adding --retry 3 to the command to auto-retry. If you see a 404 error, double-check the URL has no extra spaces or line breaks.

help me troubleshoot a failed model download

Verify the project compiles

With all dependencies declared and models downloaded, you can now test that Cargo resolves everything and the project compiles.

  • Build the project by running:
cargo build

Why does the first build take so long?

The whisper-rs crate compiles whisper.cpp from C++ source during the build. Combined with the ONNX Runtime download for tts-rs, your first cargo build will take 5-10 minutes. Subsequent builds are much faster because Cargo caches compiled artifacts.

When the build finishes, you should see a line ending with Finished in the terminal output.

  • Run the default hello world to confirm everything works:
cargo run

You should see Hello, world! printed to the terminal.

Build errors mentioning cmake or C++ compiler?

The whisper-rs crate needs cmake and g++ to compile its C++ dependency. Make sure both are installed (you set them up in Step 1). Run cmake --version and g++ --version to confirm they are on your PATH.

help me fix whisper-rs build errors

✔️ Awesome, I've got everything!

Great. Double check you have saved Cargo.toml and that both model directories have files in them.

ⓧ I'd like to double check the full code

Here is the complete Cargo.toml you should have at this point:

[package]
name = "voice-agent"
version = "0.1.0"
edition = "2021"

[dependencies]
whisper-rs = "0.16"
cpal = "0.17"
tts-rs = { version = "2026.2", features = ["kokoro"] }
reqwest = { version = "0.12", features = ["json"] }
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
anyhow = "1"

And your src/main.rs should still be the default:

fn main() {
    println!("Hello, world!");
}

Your project skeleton is ready with all dependencies resolved and model files in place. Next up, you will write the audio capture code that listens to your microphone in real time.

Capture Microphone Audio

Your project compiles and all the models are downloaded. Now you need the first piece of the voice pipeline: capturing audio from your microphone.

In this step, you will use cpal (the standard Rust audio library) to open your mic, stream audio samples into memory, and detect when you stop speaking. This detection technique is called Voice Activity Detection (VAD), and you will build a simple version using audio energy levels.

In this step, get ready to:
  • Open the default microphone with cpal and stream audio at 16kHz mono.
  • Buffer incoming audio samples into shared memory using Arc and Mutex.
  • Detect speech start and end using energy-based Voice Activity Detection.
Set up the audio imports and constants

Before writing any audio logic, you need to import the cpal traits and define the constants that control how your recorder behaves. These constants determine the sample rate, the energy threshold for detecting speech, and how long silence must last before recording stops.

  • Open src/main.rs in your editor and replace its entire contents with the following:
use anyhow::Result;
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};

// Audio capture settings
const SAMPLE_RATE: u32 = 16000;           // 16kHz for Whisper compatibility
const SILENCE_THRESHOLD: f32 = 0.01;      // RMS energy below this is "silence"
const SILENCE_DURATION: Duration = Duration::from_millis(1500);   // Silence duration to stop recording
const MIN_SPEECH_DURATION: Duration = Duration::from_millis(500); // Minimum speech to avoid false triggers

fn main() {
    println!("Hello, world!");
}

What do these constants control?

  • SAMPLE_RATE: 16kHz mono is what Whisper expects as input. Requesting this from cpal means no resampling is needed.
  • SILENCE_THRESHOLD: the RMS energy level below which audio counts as silence. A value of 0.01 works for most microphones, but you may need to tune it.
  • SILENCE_DURATION: how long the energy must stay below the threshold before the recorder decides you stopped talking (1.5 seconds).
  • MIN_SPEECH_DURATION: prevents short noise spikes from triggering a recording (0.5 seconds minimum).
  • Save src/main.rs.
  • Verify the file compiles by running:
cargo build

You will see some "unused import" warnings. That is expected since you have not written the functions that use them yet.

Seeing a compilation error?

  • Check that your Cargo.toml still lists cpal = "0.17" and anyhow = "1" in [dependencies].
  • Make sure libasound2-dev is installed. Without ALSA dev headers, cpal cannot compile on Linux.
  • help me fix my cargo build errors for cpal
Write the audio energy calculator

To detect whether someone is speaking, you need a way to measure how "loud" an audio buffer is. The standard approach is Root Mean Square (RMS): square every sample, average the squares, then take the square root. A high RMS means speech. A low RMS means silence.

  • In src/main.rs, add the following function above fn main():
// Calculate Root Mean Square energy of audio samples
fn calculate_rms(samples: &[f32]) -> f32 {
    if samples.is_empty() { return 0.0; }
    let sum_squares: f32 = samples.iter().map(|s| s * s).sum();
    (sum_squares / samples.len() as f32).sqrt()
}

What does this function do?

RMS gives you a single number representing the energy in an audio chunk. When you speak, RMS spikes above the threshold (0.01). When you are silent, it stays near zero. This is the simplest form of Voice Activity Detection.

  • Save src/main.rs.

You will not see a runtime change yet. The next function uses calculate_rms inside the audio callback.

Record audio with voice activity detection

This is the core of your audio capture. The function opens your microphone via cpal, streams samples into a shared buffer protected by Arc<Mutex<>>, and uses RMS to detect when speech starts and stops.

Why Arc and Mutex for audio?

The cpal audio callback runs on a separate OS thread. To share data between that callback and your main thread, Rust requires thread-safe wrappers. Arc provides shared ownership across threads, and Mutex ensures only one thread accesses the data at a time.

  • In src/main.rs, add the following function between calculate_rms and fn main():
fn record_until_silence() -> Result<Vec<f32>> {
    // Open the default microphone
    let host = cpal::default_host();
    let device = host.default_input_device().ok_or_else(|| anyhow::anyhow!("No input device found"))?;
    println!("🎤 Listening... (speak now)");

    // Configure for 16kHz mono (what Whisper expects)
    let config = cpal::StreamConfig { channels: 1, sample_rate: cpal::SampleRate(SAMPLE_RATE), buffer_size: cpal::BufferSize::Default };

    // Shared state between the audio callback and main thread
    let audio_buffer: Arc<Mutex<Vec<f32>>> = Arc::new(Mutex::new(Vec::new()));
    let is_done: Arc<Mutex<bool>> = Arc::new(Mutex::new(false));
    let buffer_clone = Arc::clone(&audio_buffer);
    let done_clone = Arc::clone(&is_done);

    // VAD state tracking
    let speech_started = Arc::new(Mutex::new(false));
    let last_speech_time = Arc::new(Mutex::new(Instant::now()));
    let speech_start_time = Arc::new(Mutex::new(Instant::now()));
    let speech_started_clone = Arc::clone(&speech_started);
    let last_speech_clone = Arc::clone(&last_speech_time);
    let speech_start_clone = Arc::clone(&speech_start_time);

    // Build the input stream with a callback that runs on each audio chunk
    let stream = device.build_input_stream(&config, move |data: &[f32], _: &cpal::InputCallbackInfo| {
        let rms = calculate_rms(data);
        let mut started = speech_started_clone.lock().unwrap();
        let mut last_speech = last_speech_clone.lock().unwrap();
        let mut start_time = speech_start_clone.lock().unwrap();

        // Detect speech start
        if rms > SILENCE_THRESHOLD {
            if !*started { *started = true; *start_time = Instant::now(); }
            *last_speech = Instant::now();
        }

        // Buffer audio and check for end-of-speech
        if *started {
            buffer_clone.lock().unwrap().extend_from_slice(data);
            let speech_long_enough = start_time.elapsed() > MIN_SPEECH_DURATION;
            let silence_long_enough = last_speech.elapsed() > SILENCE_DURATION;
            if speech_long_enough && silence_long_enough { *done_clone.lock().unwrap() = true; }
        }
    }, |err| eprintln!("Audio input error: {}", err), None)?;

    // Start recording and wait for speech to finish
    stream.play()?;
    loop {
        std::thread::sleep(Duration::from_millis(50));
        if *is_done.lock().unwrap() { break; }
    }
    drop(stream);

    let audio = audio_buffer.lock().unwrap().clone();
    println!("📝 Captured {:.1}s of audio", audio.len() as f32 / SAMPLE_RATE as f32);
    Ok(audio)
}

What does this function do?

This function implements the full record-until-silence pipeline:

  • Opens the default input device and configures it for 16kHz mono audio.
  • Creates shared buffers wrapped in Arc<Mutex<>> so the callback thread and main thread can communicate.
  • Inside the callback, calculates RMS for each audio chunk. When energy exceeds the threshold, recording starts.
  • Once speech has lasted at least 500ms and silence has lasted 1.5 seconds, it signals the main thread to stop.
  • The main thread polls is_done every 50ms, then drops the stream and returns the captured audio.
  • Save src/main.rs and verify it compiles by running:
cargo build

You should see a successful build with warnings about unused functions. These warnings disappear once you call record_until_silence in a later step.

Build failing with linker errors?

  • Linker errors mentioning alsa mean the ALSA dev headers are missing. Install them with sudo apt install libasound2-dev and try again.
  • If you see errors about cpal::InputCallbackInfo not being found, confirm your Cargo.toml has cpal = "0.17" (not an older version).
  • help me debug my cpal build errors on Linux

✔️ Awesome, I've got everything!

Great. Double check you have saved src/main.rs and that cargo build completes without errors (warnings are fine).

ⓧ I'd like to double check the full code

Here is the complete src/main.rs as it should look at the end of this step:

use anyhow::Result;
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};

const SAMPLE_RATE: u32 = 16000;
const SILENCE_THRESHOLD: f32 = 0.01;
const SILENCE_DURATION: Duration = Duration::from_millis(1500);
const MIN_SPEECH_DURATION: Duration = Duration::from_millis(500);

fn calculate_rms(samples: &[f32]) -> f32 {
    if samples.is_empty() { return 0.0; }
    let sum_squares: f32 = samples.iter().map(|s| s * s).sum();
    (sum_squares / samples.len() as f32).sqrt()
}

fn record_until_silence() -> Result<Vec<f32>> {
    let host = cpal::default_host();
    let device = host.default_input_device().ok_or_else(|| anyhow::anyhow!("No input device found"))?;
    println!("🎤 Listening... (speak now)");
    let config = cpal::StreamConfig { channels: 1, sample_rate: cpal::SampleRate(SAMPLE_RATE), buffer_size: cpal::BufferSize::Default };
    let audio_buffer: Arc<Mutex<Vec<f32>>> = Arc::new(Mutex::new(Vec::new()));
    let is_done: Arc<Mutex<bool>> = Arc::new(Mutex::new(false));
    let buffer_clone = Arc::clone(&audio_buffer);
    let done_clone = Arc::clone(&is_done);
    let speech_started = Arc::new(Mutex::new(false));
    let last_speech_time = Arc::new(Mutex::new(Instant::now()));
    let speech_start_time = Arc::new(Mutex::new(Instant::now()));
    let speech_started_clone = Arc::clone(&speech_started);
    let last_speech_clone = Arc::clone(&last_speech_time);
    let speech_start_clone = Arc::clone(&speech_start_time);
    let stream = device.build_input_stream(&config, move |data: &[f32], _: &cpal::InputCallbackInfo| {
        let rms = calculate_rms(data);
        let mut started = speech_started_clone.lock().unwrap();
        let mut last_speech = last_speech_clone.lock().unwrap();
        let mut start_time = speech_start_clone.lock().unwrap();
        if rms > SILENCE_THRESHOLD {
            if !*started { *started = true; *start_time = Instant::now(); }
            *last_speech = Instant::now();
        }
        if *started {
            buffer_clone.lock().unwrap().extend_from_slice(data);
            let speech_long_enough = start_time.elapsed() > MIN_SPEECH_DURATION;
            let silence_long_enough = last_speech.elapsed() > SILENCE_DURATION;
            if speech_long_enough && silence_long_enough { *done_clone.lock().unwrap() = true; }
        }
    }, |err| eprintln!("Audio input error: {}", err), None)?;
    stream.play()?;
    loop {
        std::thread::sleep(Duration::from_millis(50));
        if *is_done.lock().unwrap() { break; }
    }
    drop(stream);
    let audio = audio_buffer.lock().unwrap().clone();
    println!("📝 Captured {:.1}s of audio", audio.len() as f32 / SAMPLE_RATE as f32);
    Ok(audio)
}

fn main() {
    println!("Hello, world!");
}

Your mic capture logic is ready. Next up, you will feed this captured audio into Whisper for speech-to-text transcription.

Transcribe Speech with Whisper

You have a working audio capture system that records your voice and detects when you stop speaking. The next piece of the puzzle is turning those raw audio samples into text.

This is where Whisper comes in. Whisper is OpenAI's speech recognition model, and whisper-rs gives you direct Rust bindings to whisper.cpp, the C++ inference engine. You'll feed it the audio buffer from your VAD system and get back a transcript string.

In this step, get ready to:
  • Add Whisper imports and the model path constant.
  • Implement a transcription function that processes captured audio into text.
  • Understand how Whisper segments audio into text chunks.
Add the Whisper imports and model path

Before writing the transcription logic, you need to bring the whisper-rs types into scope and define where the model file lives on disk.

  • In src/main.rs, find the import section at the top of the file.
  • Add these two new import lines after the existing use std::sync line:
use std::path::PathBuf;
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};
  • Now add the model path constant. Find the line const SAMPLE_RATE: u32 = 16000; and add this line directly above it:
const WHISPER_MODEL_PATH: &str = "models/ggml-base.en.bin";

What do these additions do?

The whisper_rs import brings in four types: WhisperContext (holds the loaded model), WhisperContextParameters (configuration for model loading), FullParams (decoding settings), and SamplingStrategy (how tokens are chosen during decoding).

The WHISPER_MODEL_PATH constant points to the ggml-base.en.bin file you downloaded in Step 2. The PathBuf import will be used later for loading the TTS model.

Implement the transcribe function

The transcribe function takes a loaded Whisper model context and a slice of audio samples, then returns the recognized text. Whisper processes audio in one shot via its state.full() method, which runs the entire PCM-to-text pipeline internally.

  • In src/main.rs, find the record_until_silence function's closing brace. Add the following transcribe function below it:
fn transcribe(ctx: &WhisperContext, audio: &[f32]) -> Result<String> {
    // Create a fresh inference state from the model context
    let mut state = ctx.create_state().expect("Failed to create whisper state");

    // Configure decoding: greedy search, English, no extra output
    let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
    params.set_language(Some("en"));
    params.set_print_special_tokens(false);
    params.set_print_progress(false);
    params.set_print_realtime(false);
    params.set_print_timestamps(false);

    // Run the full transcription pipeline: PCM -> mel spectrogram -> encoder -> decoder -> text
    state.full(params, audio)?;

    // Collect all text segments into a single string
    let num_segments = state.full_n_segments()?;
    let mut text = String::new();
    for i in 0..num_segments {
        if let Ok(segment) = state.full_get_segment_text(i) {
            text.push_str(&segment);
        }
    }

    Ok(text.trim().to_string())
}

What does this code do?

The function follows three phases that mirror how speech recognition works internally:

  • Phase 1 (setup): create_state() allocates a fresh inference state. FullParams configures greedy decoding (pick the single best token at each step) and tells Whisper to target English.
  • Phase 2 (inference): state.full(params, audio) is the high-level entry point. It converts raw PCM to a mel spectrogram, encodes it, then decodes token-by-token into text segments.
  • Phase 3 (extraction): Whisper splits its output into segments (roughly sentence-length chunks). The loop collects them all into one final string.

Why disable all the print flags?

By default, whisper.cpp prints progress updates, timestamps, and special tokens to stdout. That would clutter your terminal alongside the conversation output. Disabling them keeps the terminal clean for your agent's responses.

  • Save src/main.rs.
  • Verify the project compiles by running:
cargo build --release 2>&1 | tail -5

You should see a successful build with no errors in the last few lines of output.

Seeing a compilation error?

  • If you see an "unresolved import" error for whisper_rs, double-check that whisper-rs = "0.16" is listed in your Cargo.toml under [dependencies].
  • If you see a type mismatch on FullParams::new, make sure you imported SamplingStrategy from whisper_rs in your use statement.
  • If the build fails during the whisper-rs compilation step (C++ errors), ensure you have cmake and g++ installed from Step 1.

Still stuck? Help me debug a whisper-rs compilation error in my Rust voice agent project.

✔️ Awesome, I've got everything!

Great. Double check you've saved src/main.rs and the build succeeded.

ⓧ I'd like to double check the full code

use anyhow::Result;
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};

const WHISPER_MODEL_PATH: &str = "models/ggml-base.en.bin";
const SAMPLE_RATE: u32 = 16000;
const SILENCE_THRESHOLD: f32 = 0.01;
const SILENCE_DURATION: Duration = Duration::from_millis(1500);
const MIN_SPEECH_DURATION: Duration = Duration::from_millis(500);

fn calculate_rms(samples: &[f32]) -> f32 {
    if samples.is_empty() { return 0.0; }
    let sum_squares: f32 = samples.iter().map(|s| s * s).sum();
    (sum_squares / samples.len() as f32).sqrt()
}

fn record_until_silence() -> Result<Vec<f32>> {
    // ... (same as previous step)
    Ok(vec![])
}

fn transcribe(ctx: &WhisperContext, audio: &[f32]) -> Result<String> {
    let mut state = ctx.create_state().expect("Failed to create whisper state");
    let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
    params.set_language(Some("en"));
    params.set_print_special_tokens(false);
    params.set_print_progress(false);
    params.set_print_realtime(false);
    params.set_print_timestamps(false);
    state.full(params, audio)?;
    let num_segments = state.full_n_segments()?;
    let mut text = String::new();
    for i in 0..num_segments {
        if let Ok(segment) = state.full_get_segment_text(i) { text.push_str(&segment); }
    }
    Ok(text.trim().to_string())
}

fn main() {
    println!("Hello, world!");
}

Your speech-to-text pipeline is ready. Next up, you'll connect to Gemma 4 via Ollama so the agent can actually think about what you said and generate a response.

Chat with Gemma 4 via Ollama

Your voice agent can now hear you. It captures microphone audio and transcribes your words into text using Whisper.

Text alone is not a conversation. In this step, you'll connect your agent to Gemma 4 running locally through Ollama. This gives your agent the ability to understand your words and generate intelligent responses.

In this step, get ready to:
  • Define the data types that model Ollama's chat API request and response.
  • Implement an async function that sends conversation messages and returns the assistant's reply.
Define the Ollama API types

Ollama exposes a local HTTP API at http://localhost:11434/api/chat. To communicate with it from Rust, you need struct definitions that match the JSON request and response shapes.

  • In src/main.rs, find the existing import lines at the top of the file. Add this new import directly after the use cpal::traits line:
use serde::{Deserialize, Serialize};
  • Next, find the WHISPER_MODEL_PATH constant. Add these two new constants directly below it:
const OLLAMA_URL: &str = "http://localhost:11434/api/chat";
const OLLAMA_MODEL: &str = "gemma4:e4b";
  • Now add the three API structs. Place them after the MIN_SPEECH_DURATION constant and before the calculate_rms function. Copy the following code:
// --- Ollama API types ---
#[derive(Serialize, Deserialize, Clone, Debug)]
struct Message {
    role: String,
    content: String,
}

#[derive(Serialize)]
struct ChatRequest {
    model: String,
    messages: Vec<Message>,
    stream: bool,
}

#[derive(Deserialize)]
struct ChatResponse {
    message: Message,
}

What do these structs do?

These three structs map directly to the JSON that Ollama's chat API expects and returns.

  • Message represents a single conversation turn with a role (system, user, or assistant) and content (the text). A Vec<Message> gives you multi-turn conversation history.
  • ChatRequest is the JSON body you POST to Ollama. Setting stream: false tells Ollama to return the full response in one shot instead of streaming tokens.
  • ChatResponse captures the message field from Ollama's JSON response, which contains the assistant's reply.
  • Save src/main.rs.

Seeing a compile error about serde?

Make sure the import reads use serde::{Deserialize, Serialize}; with a capital D and capital S. Rust derive macros are case-sensitive.

Also confirm that your Cargo.toml has serde = { version = "1", features = ["derive"] } in [dependencies]. Without the derive feature, the Serialize and Deserialize macros are not available.

Help me fix a serde derive error in my Rust project

Implement the chat function

With the types defined, you can now write the function that actually sends your conversation to Gemma 4 and gets back a response.

  • In src/main.rs, add the following async function after the transcribe function and before fn main():
// --- Ollama chat ---
async fn chat_with_ollama(messages: &[Message]) -> Result<String> {
    let client = reqwest::Client::new();
    let request = ChatRequest {
        model: OLLAMA_MODEL.to_string(),
        messages: messages.to_vec(),
        stream: false,
    };

    let response = client
        .post(OLLAMA_URL)
        .json(&request)
        .send()
        .await?
        .json::<ChatResponse>()
        .await?;

    Ok(response.message.content)
}

What does this function do?

  • reqwest::Client::new() creates an HTTP client. The reqwest crate handles all the async networking under the hood.
  • The function builds a ChatRequest with the model name, full message history, and stream: false so Ollama returns the complete response in one JSON object.
  • The chained .post(), .json(), .send(), and .json::<ChatResponse>() calls serialize the request, send it to Ollama, and deserialize the response. The ? operators propagate any errors up.
  • By accepting &[Message] as the parameter, the caller can pass the entire conversation history. This is how multi-turn context works. Each call sends all previous messages so the LLM knows the full conversation.
  • Save src/main.rs.
  • Verify the project compiles by running:
cargo check

You should see a warning about unused imports and functions, but no errors. The function compiles successfully even though nothing calls it yet.

Seeing an error about async in a non-async context?

The chat_with_ollama function is async, which requires the tokio runtime. You do not need to change main() yet. As long as the function itself compiles, you are on track. The async runtime gets wired in when you build the full loop in the next step.

If you see a different error about reqwest not being found, confirm your Cargo.toml has reqwest = { version = "0.12", features = ["json"] } listed under [dependencies].

Help me fix a compile error with my async chat function

Why stream: false instead of streaming?

Ollama supports token-by-token streaming, which is great for chat UIs that show text appearing in real time. For a voice agent, streaming adds complexity without benefit. You need the complete text before you can synthesize it into speech.

Setting stream: false means Ollama processes the full response internally and returns it in one JSON payload. This simplifies parsing and keeps the code focused on the voice pipeline.

✔️ Awesome, I've got everything!

Great. Double check you have saved src/main.rs and that cargo check passes without errors.

ⓧ I'd like to double check the full code

Here is the full src/main.rs as it should look at the end of this step:

use anyhow::Result;
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};

const WHISPER_MODEL_PATH: &str = "models/ggml-base.en.bin";
const OLLAMA_URL: &str = "http://localhost:11434/api/chat";
const OLLAMA_MODEL: &str = "gemma4:e4b";
const SAMPLE_RATE: u32 = 16000;
const SILENCE_THRESHOLD: f32 = 0.01;
const SILENCE_DURATION: Duration = Duration::from_millis(1500);
const MIN_SPEECH_DURATION: Duration = Duration::from_millis(500);

#[derive(Serialize, Deserialize, Clone, Debug)]
struct Message { role: String, content: String }

#[derive(Serialize)]
struct ChatRequest { model: String, messages: Vec<Message>, stream: bool }

#[derive(Deserialize)]
struct ChatResponse { message: Message }

fn calculate_rms(samples: &[f32]) -> f32 { /* ... */ 0.0 }
fn record_until_silence() -> Result<Vec<f32>> { Ok(vec![]) }
fn transcribe(ctx: &WhisperContext, audio: &[f32]) -> Result<String> { Ok(String::new()) }

async fn chat_with_ollama(messages: &[Message]) -> Result<String> {
    let client = reqwest::Client::new();
    let request = ChatRequest { model: OLLAMA_MODEL.to_string(), messages: messages.to_vec(), stream: false };
    let response = client.post(OLLAMA_URL).json(&request).send().await?.json::<ChatResponse>().await?;
    Ok(response.message.content)
}

fn main() {
    println!("Hello, world!");
}

Your agent now has a brain. It can take transcribed text, send it to Gemma 4 for processing, and get back an intelligent response. Next up, you'll give it a voice by synthesizing that response into speech with Kokoro TTS.

Synthesize and Play Speech with Kokoro

You now have audio capture, transcription, and LLM chat working. The final piece of the voice pipeline is giving the agent a voice of its own.

In this step, you will integrate Kokoro TTS to convert text responses into audible speech. Kokoro is an 82-million-parameter neural text-to-speech model that runs entirely on CPU via ONNX Runtime. It outputs raw PCM audio at 24kHz, which you will play through your speakers using cpal.

In this step, get ready to:
  • Initialize the Kokoro TTS engine by loading the ONNX model.
  • Synthesize text into PCM audio samples.
  • Play synthesized audio through your speakers with cpal.
Add the Kokoro import and model path

Before writing the playback function, you need to bring the Kokoro engine into scope and define the path to the model files you downloaded earlier.

  • In src/main.rs, find this line near the top of your imports:
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};
  • Add the following line directly above it:
use tts_rs::engines::kokoro::KokoroEngine;
  • Next, find this line in the constants section:
const WHISPER_MODEL_PATH: &str = "models/ggml-base.en.bin";
  • Add the Kokoro model directory constant directly below it:
const KOKORO_MODEL_DIR: &str = "models/kokoro";

What does this set up?

KokoroEngine is the struct from tts-rs that wraps the Kokoro ONNX model for inference. It exposes a simple load_model() and synthesize() API.

KOKORO_MODEL_DIR points to the directory containing your kokoro-v1.0.int8.onnx and voices-v1.0.bin files that you downloaded in Step 2.

Understand the Kokoro synthesis pipeline

When the main loop is wired together in the next step, Kokoro will convert the LLM's text response into a Vec<f32> of PCM samples at 24kHz. Here is how the three-step pattern works:

  • Create the engine with KokoroEngine::new().
  • Load the ONNX model from the directory with engine.load_model(&PathBuf::from(KOKORO_MODEL_DIR)).
  • Call engine.synthesize(text, Some("af_heart"), None) to produce audio samples.

What do the synthesize parameters mean?

The first argument is the text to speak. The second selects a voice style ("af_heart" is a warm English voice). The third is an optional speed override (None means default 1.0x).

The output is a Vec<f32> of raw PCM samples at 24,000 samples per second. Your next task is to write the function that plays these samples through the speakers.

Write the audio playback function

This function opens the default output device, creates a stream at 24kHz, and feeds samples from a buffer into the audio callback until the entire response has been played.

  • In src/main.rs, add the following function above fn main():
// --- Play audio through speakers ---
fn play_audio(samples: &[f32]) -> Result<()> {
    let host = cpal::default_host();
    let device = host
        .default_output_device()
        .ok_or_else(|| anyhow::anyhow!("No output device found"))?;

    // Configure output stream at 24kHz mono to match Kokoro's sample rate
    let config = cpal::StreamConfig {
        channels: 1,
        sample_rate: cpal::SampleRate(24000),
        buffer_size: cpal::BufferSize::Default,
    };

    // Clone samples into a Vec the closure can own
    let samples = samples.to_vec();
    let position = Arc::new(Mutex::new(0usize));
    let is_done = Arc::new(Mutex::new(false));

    let pos_clone = Arc::clone(&position);
    let done_clone = Arc::clone(&is_done);
    let samples_len = samples.len();

    // Build the output stream with a callback that feeds samples sequentially
    let stream = device.build_output_stream(
        &config,
        move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
            let mut pos = pos_clone.lock().unwrap();
            for sample in data.iter_mut() {
                if *pos < samples_len {
                    *sample = samples[*pos];
                    *pos += 1;
                } else {
                    *sample = 0.0;
                    *done_clone.lock().unwrap() = true;
                }
            }
        },
        |err| eprintln!("Audio output error: {}", err),
        None,
    )?;

    stream.play()?;

    // Block until all samples have been played
    loop {
        std::thread::sleep(Duration::from_millis(50));
        if *is_done.lock().unwrap() {
            break;
        }
    }

    Ok(())
}

What does this code do?

cpal::default_host() gets the platform's audio backend (ALSA on Linux). default_output_device() picks the system's default speakers.

The stream callback runs on a dedicated audio thread. Each time the OS requests more audio data, the callback copies samples from the Vec<f32> into the output buffer, advancing a shared position counter.

When all samples are consumed, the callback writes silence (0.0) and sets is_done to true. The main thread polls this flag every 50ms and exits the loop once playback finishes.

  • Save src/main.rs.
  • Verify the project compiles by running:
cargo build 2>&1 | tail -5

You should see a successful compilation message with no errors. Warnings about unused functions are expected at this stage.

Seeing a compile error?

  • If you see an error about tts_rs not being found, make sure your Cargo.toml has tts-rs = { version = "2026.2", features = ["kokoro"] } in the dependencies section.
  • If you see an error about OutputCallbackInfo not found, confirm you have use cpal::traits::{DeviceTrait, HostTrait, StreamTrait}; in your imports.
  • If ort (ONNX Runtime) fails to compile, make sure cmake and protobuf-compiler are installed on your system.

Help me debug a tts-rs compile error in my Rust voice agent project.

✔️ Awesome, I've got everything!

Great. Double check you have saved src/main.rs before moving on.

ⓧ I'd like to double check the full code

Here is the complete src/main.rs at the end of this step:

use anyhow::Result;
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tts_rs::engines::kokoro::KokoroEngine;
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};

const WHISPER_MODEL_PATH: &str = "models/ggml-base.en.bin";
const KOKORO_MODEL_DIR: &str = "models/kokoro";
const OLLAMA_URL: &str = "http://localhost:11434/api/chat";
const OLLAMA_MODEL: &str = "gemma4:e4b";
const SAMPLE_RATE: u32 = 16000;
const SILENCE_THRESHOLD: f32 = 0.01;
const SILENCE_DURATION: Duration = Duration::from_millis(1500);
const MIN_SPEECH_DURATION: Duration = Duration::from_millis(500);

#[derive(Serialize, Deserialize, Clone, Debug)]
struct Message { role: String, content: String }
#[derive(Serialize)]
struct ChatRequest { model: String, messages: Vec<Message>, stream: bool }
#[derive(Deserialize)]
struct ChatResponse { message: Message }

fn calculate_rms(samples: &[f32]) -> f32 {
    if samples.is_empty() { return 0.0; }
    let sum_squares: f32 = samples.iter().map(|s| s * s).sum();
    (sum_squares / samples.len() as f32).sqrt()
}

fn record_until_silence() -> Result<Vec<f32>> { Ok(vec![]) }
fn transcribe(ctx: &WhisperContext, audio: &[f32]) -> Result<String> { Ok(String::new()) }
async fn chat_with_ollama(messages: &[Message]) -> Result<String> { Ok(String::new()) }

fn play_audio(samples: &[f32]) -> Result<()> {
    let host = cpal::default_host();
    let device = host.default_output_device().ok_or_else(|| anyhow::anyhow!("No output device found"))?;
    let config = cpal::StreamConfig { channels: 1, sample_rate: cpal::SampleRate(24000), buffer_size: cpal::BufferSize::Default };
    let samples = samples.to_vec();
    let position = Arc::new(Mutex::new(0usize));
    let is_done = Arc::new(Mutex::new(false));
    let pos_clone = Arc::clone(&position);
    let done_clone = Arc::clone(&is_done);
    let samples_len = samples.len();
    let stream = device.build_output_stream(&config, move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
        let mut pos = pos_clone.lock().unwrap();
        for sample in data.iter_mut() {
            if *pos < samples_len { *sample = samples[*pos]; *pos += 1; }
            else { *sample = 0.0; *done_clone.lock().unwrap() = true; }
        }
    }, |err| eprintln!("Audio output error: {}", err), None)?;
    stream.play()?;
    loop {
        std::thread::sleep(Duration::from_millis(50));
        if *is_done.lock().unwrap() { break; }
    }
    Ok(())
}

fn main() {
    println!("Hello, world!");
}

Your voice agent can now speak. In the next step, you will wire all four components together into a continuous conversation loop.

Wire the Full Conversation Loop

You have all four pieces of the voice pipeline built individually: microphone capture, speech-to-text, LLM chat, and text-to-speech playback.

Now you need to wire them into a single async loop that listens, thinks, and speaks on repeat. In this step, you will replace the placeholder main() function with the full conversation engine that ties everything together.

In this step, get ready to:
  • Combine all components into an async main loop that listens, transcribes, chats, and speaks.
  • Add a graceful exit with Ctrl+C.
  • Test a full end-to-end conversation with your voice agent.
Add the missing import and replace the main function signature

The conversation loop needs tokio to drive async operations (the HTTP call to Ollama). The #[tokio::main] attribute converts your main() into an async entry point.

  • In src/main.rs, find this import line near the top:
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
  • Add this new import directly below it:
use cpal::SampleFormat;
  • Now find the placeholder main() function at the bottom of the file:
fn main() {
    println!("Hello, world!");
}
  • Delete the entire placeholder main() function.

You will replace it with the full async implementation across the next two code chunks.

Write the model initialization code

Before the conversation loop can run, you need to load both the Whisper model (for transcription) and the Kokoro model (for speech synthesis) into memory. Model loading happens once at startup so each loop iteration only runs inference.

  • Add the following code where the old main() was, to define the async entry point and load models:
#[tokio::main]
async fn main() -> Result<()> {
    println!("🤖 Local Voice Agent (Rust)");
    println!("   STT: Whisper | LLM: Gemma 4 E4B | TTS: Kokoro");
    println!("   Press Ctrl+C to exit\n");

    // Initialize Whisper
    println!("Loading Whisper model...");
    let whisper_ctx = WhisperContext::new_with_params(
        WHISPER_MODEL_PATH,
        WhisperContextParameters::default(),
    )
    .expect("Failed to load Whisper model");

    // Initialize Kokoro TTS
    println!("Loading Kokoro TTS model...");
    let mut tts_engine = KokoroEngine::new();
    tts_engine
        .load_model(&PathBuf::from(KOKORO_MODEL_DIR))
        .expect("Failed to load Kokoro model");

    println!("✅ All models loaded. Ready to chat!\n");
}

What does this code do?

#[tokio::main] transforms main() into an async runtime entry point, which is required because chat_with_ollama uses await.

  • WhisperContext::new_with_params loads the ggml-base.en.bin model file into memory for transcription.
  • KokoroEngine::new() creates the TTS engine, and load_model() loads the ONNX weights from your models/kokoro directory.
  • The banner line "Press Ctrl+C to exit" tells users how to stop the infinite conversation loop.
  • Save src/main.rs.

Seeing a compile error about async main?

Make sure you deleted the entire old fn main() including its closing brace. Two main functions in the same file causes a duplicate definition error.

Help me debug my async main compile error.

Add the conversation loop

The conversation loop is the heartbeat of your voice agent. Each iteration records your speech, transcribes it, sends it to the LLM, synthesizes the reply, and plays it back.

  • Inside the main() function, replace the closing } with the following conversation loop code (keeping the function open):
    // Conversation history
    let mut messages: Vec<Message> = vec![Message {
        role: "system".to_string(),
        content: "You are a helpful voice assistant. Keep responses concise and conversational, under 2-3 sentences.".to_string(),
    }];

    loop {
        // 1. Record audio
        let audio = record_until_silence()?;
        if audio.len() < (SAMPLE_RATE as usize) / 2 {
            println!("(too short, skipping)\n");
            continue;
        }

        // 2. Transcribe
        let user_text = transcribe(&whisper_ctx, &audio)?;
        if user_text.is_empty() {
            println!("(no speech detected)\n");
            continue;
        }
        println!("🗣️  You: {}", user_text);

        // 3. Add to history and chat
        messages.push(Message {
            role: "user".to_string(),
            content: user_text,
        });

        let assistant_text = chat_with_ollama(&messages).await?;
        println!("🤖 Agent: {}", assistant_text);

        messages.push(Message {
            role: "assistant".to_string(),
            content: assistant_text.clone(),
        });

        // 4. Synthesize and play
        let tts_audio = tts_engine.synthesize(&assistant_text, Some("af_heart"), None)?;
        play_audio(&tts_audio)?;

        println!();
    }
}

What does this loop do?

  • The messages Vec holds the full conversation history. It starts with a system prompt that instructs the LLM to reply concisely.
  • The minimum length check (SAMPLE_RATE / 2 = half a second of audio) skips accidental mic activations.
  • Each user and assistant message is pushed to messages so the LLM has multi-turn context for follow-up questions.
  • The voice "af_heart" is one of Kokoro's 54+ built-in voice options.
  • Save src/main.rs.

Getting a type mismatch on synthesize?

Make sure tts_engine is declared as let mut tts_engine. The synthesize() method requires a mutable reference.

Help me fix my Kokoro TTS synthesize error.

✔️ Awesome, I've got everything!

Great. Double check you have saved src/main.rs before running the agent.

ⓧ I'd like to double check the full code

Here is the complete src/main.rs file as it should look now:

use anyhow::Result;
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use cpal::SampleFormat;
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tts_rs::engines::kokoro::KokoroEngine;
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};

const WHISPER_MODEL_PATH: &str = "models/ggml-base.en.bin";
const KOKORO_MODEL_DIR: &str = "models/kokoro";
const OLLAMA_URL: &str = "http://localhost:11434/api/chat";
const OLLAMA_MODEL: &str = "gemma4:e4b";
const SAMPLE_RATE: u32 = 16000;
const SILENCE_THRESHOLD: f32 = 0.01;
const SILENCE_DURATION: Duration = Duration::from_millis(1500);
const MIN_SPEECH_DURATION: Duration = Duration::from_millis(500);

#[derive(Serialize, Deserialize, Clone, Debug)]
struct Message { role: String, content: String }
#[derive(Serialize)]
struct ChatRequest { model: String, messages: Vec<Message>, stream: bool }
#[derive(Deserialize)]
struct ChatResponse { message: Message }

fn calculate_rms(samples: &[f32]) -> f32 {
    if samples.is_empty() { return 0.0; }
    let sum_squares: f32 = samples.iter().map(|s| s * s).sum();
    (sum_squares / samples.len() as f32).sqrt()
}

fn record_until_silence() -> Result<Vec<f32>> {
    let host = cpal::default_host();
    let device = host.default_input_device().ok_or_else(|| anyhow::anyhow!("No input device found"))?;
    println!("🎤 Listening... (speak now)");
    let config = cpal::StreamConfig { channels: 1, sample_rate: cpal::SampleRate(SAMPLE_RATE), buffer_size: cpal::BufferSize::Default };
    let audio_buffer: Arc<Mutex<Vec<f32>>> = Arc::new(Mutex::new(Vec::new()));
    let is_done: Arc<Mutex<bool>> = Arc::new(Mutex::new(false));
    let buffer_clone = Arc::clone(&audio_buffer);
    let done_clone = Arc::clone(&is_done);
    let speech_started = Arc::new(Mutex::new(false));
    let last_speech_time = Arc::new(Mutex::new(Instant::now()));
    let speech_start_time = Arc::new(Mutex::new(Instant::now()));
    let speech_started_clone = Arc::clone(&speech_started);
    let last_speech_clone = Arc::clone(&last_speech_time);
    let speech_start_clone = Arc::clone(&speech_start_time);
    let stream = device.build_input_stream(&config, move |data: &[f32], _: &cpal::InputCallbackInfo| {
        let rms = calculate_rms(data);
        let mut started = speech_started_clone.lock().unwrap();
        let mut last_speech = last_speech_clone.lock().unwrap();
        let mut start_time = speech_start_clone.lock().unwrap();
        if rms > SILENCE_THRESHOLD {
            if !*started { *started = true; *start_time = Instant::now(); }
            *last_speech = Instant::now();
        }
        if *started {
            buffer_clone.lock().unwrap().extend_from_slice(data);
            let speech_long_enough = start_time.elapsed() > MIN_SPEECH_DURATION;
            let silence_long_enough = last_speech.elapsed() > SILENCE_DURATION;
            if speech_long_enough && silence_long_enough { *done_clone.lock().unwrap() = true; }
        }
    }, |err| eprintln!("Audio input error: {}", err), None)?;
    stream.play()?;
    loop {
        std::thread::sleep(Duration::from_millis(50));
        if *is_done.lock().unwrap() { break; }
    }
    drop(stream);
    let audio = audio_buffer.lock().unwrap().clone();
    println!("📝 Captured {:.1}s of audio", audio.len() as f32 / SAMPLE_RATE as f32);
    Ok(audio)
}

fn transcribe(ctx: &WhisperContext, audio: &[f32]) -> Result<String> {
    let mut state = ctx.create_state().expect("Failed to create whisper state");
    let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
    params.set_language(Some("en"));
    params.set_print_special_tokens(false);
    params.set_print_progress(false);
    params.set_print_realtime(false);
    params.set_print_timestamps(false);
    state.full(params, audio)?;
    let num_segments = state.full_n_segments()?;
    let mut text = String::new();
    for i in 0..num_segments {
        if let Ok(segment) = state.full_get_segment_text(i) { text.push_str(&segment); }
    }
    Ok(text.trim().to_string())
}

async fn chat_with_ollama(messages: &[Message]) -> Result<String> {
    let client = reqwest::Client::new();
    let request = ChatRequest { model: OLLAMA_MODEL.to_string(), messages: messages.to_vec(), stream: false };
    let response = client.post(OLLAMA_URL).json(&request).send().await?.json::<ChatResponse>().await?;
    Ok(response.message.content)
}

fn play_audio(samples: &[f32]) -> Result<()> {
    let host = cpal::default_host();
    let device = host.default_output_device().ok_or_else(|| anyhow::anyhow!("No output device found"))?;
    let config = cpal::StreamConfig { channels: 1, sample_rate: cpal::SampleRate(24000), buffer_size: cpal::BufferSize::Default };
    let samples = samples.to_vec();
    let position = Arc::new(Mutex::new(0usize));
    let is_done = Arc::new(Mutex::new(false));
    let pos_clone = Arc::clone(&position);
    let done_clone = Arc::clone(&is_done);
    let samples_len = samples.len();
    let stream = device.build_output_stream(&config, move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
        let mut pos = pos_clone.lock().unwrap();
        for sample in data.iter_mut() {
            if *pos < samples_len { *sample = samples[*pos]; *pos += 1; }
            else { *sample = 0.0; *done_clone.lock().unwrap() = true; }
        }
    }, |err| eprintln!("Audio output error: {}", err), None)?;
    stream.play()?;
    loop {
        std::thread::sleep(Duration::from_millis(50));
        if *is_done.lock().unwrap() { break; }
    }
    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    println!("🤖 Local Voice Agent (Rust)");
    println!("   STT: Whisper | LLM: Gemma 4 E4B | TTS: Kokoro");
    println!("   Press Ctrl+C to exit\n");
    println!("Loading Whisper model...");
    let whisper_ctx = WhisperContext::new_with_params(WHISPER_MODEL_PATH, WhisperContextParameters::default()).expect("Failed to load Whisper model");
    println!("Loading Kokoro TTS model...");
    let mut tts_engine = KokoroEngine::new();
    tts_engine.load_model(&PathBuf::from(KOKORO_MODEL_DIR)).expect("Failed to load Kokoro model");
    println!("✅ All models loaded. Ready to chat!\n");
    let mut messages: Vec<Message> = vec![Message { role: "system".to_string(), content: "You are a helpful voice assistant. Keep responses concise and conversational, under 2-3 sentences.".to_string() }];
    loop {
        let audio = record_until_silence()?;
        if audio.len() < (SAMPLE_RATE as usize) / 2 { println!("(too short, skipping)\n"); continue; }
        let user_text = transcribe(&whisper_ctx, &audio)?;
        if user_text.is_empty() { println!("(no speech detected)\n"); continue; }
        println!("🗣️  You: {}", user_text);
        messages.push(Message { role: "user".to_string(), content: user_text });
        let assistant_text = chat_with_ollama(&messages).await?;
        println!("🤖 Agent: {}", assistant_text);
        messages.push(Message { role: "assistant".to_string(), content: assistant_text.clone() });
        let tts_audio = tts_engine.synthesize(&assistant_text, Some("af_heart"), None)?;
        play_audio(&tts_audio)?;
        println!();
    }
}
Test a full end-to-end conversation

This is the moment everything comes together. You will run the agent in release mode for optimal inference speed on your i5.

  • Make sure Ollama is still running in the background (from Step 1).
  • Build and run your voice agent by running this command:
cargo run --release

Why --release?

The --release flag compiles with full optimizations, making Whisper and Kokoro inference significantly faster. Without it, audio processing can lag badly on a CPU-only machine.

  • Wait for the startup messages to appear. You should see:
🤖 Local Voice Agent (Rust)
   STT: Whisper | LLM: Gemma 4 E4B | TTS: Kokoro
   Press Ctrl+C to exit

Loading Whisper model...
Loading Kokoro TTS model...
✅ All models loaded. Ready to chat!

🎤 Listening... (speak now)
  • Speak a short sentence into your microphone (for example, "What is the capital of France?").
  • Wait 1.5 seconds after you stop speaking. The agent detects your silence and begins processing.
  • You should see your transcribed text, the agent's reply printed to the terminal, and hear the agent speak its response through your speakers.
  • Try a follow-up question to verify multi-turn context works (for example, "Tell me more about it.").
  • Press Ctrl+C to stop the agent when you are done.

Agent not responding or audio sounds garbled?

  • If you see "(too short, skipping)" repeatedly, your mic might be too quiet. Try speaking louder or adjusting your system microphone volume in your Linux audio settings.
  • If you get a connection error to Ollama, verify it is running with curl http://localhost:11434. Restart it if needed.
  • If TTS output is choppy, make sure no other application is using your audio device at the same time.

Help me debug my voice agent conversation loop.

Your voice agent is alive. It listens, thinks, speaks, and remembers context across turns. Next up, you will give it the ability to be interrupted mid-sentence in the Secret Mission.

Secret mission

Add Barge-In (Interruption Support)

Right now your agent talks at you, not with you. Real conversations let you interrupt mid-sentence. In this secret mission, you'll add barge-in support so the agent stops speaking the instant you start talking.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing monetary costs. Ollama does use RAM when its service is active, though.

Resources you used:

  • voice-agent project folder on your Desktop (includes compiled binary, Whisper model, and Kokoro model files).
  • Gemma 4 E4B model downloaded in Ollama (approximately 6.1 GB on disk).
  • Ollama service running in the background.
  • System packages installed via apt (libasound2-dev, cmake, build-essential, espeak-ng).

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with the voice agent more.

  • Ollama stays running as a background service and is available for future projects.
  • The voice-agent folder remains on your Desktop ready for cargo run --release at any time.
  • The system packages (libasound2-dev, cmake, etc.) are useful for any future Rust or C/C++ audio projects.

Pause - I'll come back to this later

Stop the Ollama service to free up RAM. Your project files and models stay on disk for when you return.

  • Stop the Ollama service by running:
sudo systemctl stop ollama
  • When you're ready to come back, restart it by running:
sudo systemctl start ollama

Everything else (your project folder, models, system packages) stays in place and ready to go.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

  • Delete the project folder by running:
rm -rf ~/Desktop/voice-agent
  • Remove the Gemma 4 E4B model from Ollama by running:
ollama rm gemma4:e4b

Used a different model tag?

If you pulled the community model instead, run ollama rm ssfdre38/gemma4-turbo:e4b to remove it.

  • Stop and disable the Ollama service by running:
sudo systemctl stop ollama
sudo systemctl disable ollama
  • Remove the Ollama service file and binary by running:
sudo rm /etc/systemd/system/ollama.service
sudo rm /usr/local/bin/ollama
  • Optionally remove system packages by running:
sudo apt remove libasound2-dev cmake build-essential espeak-ng

Keep system packages if in doubt

Packages like cmake and build-essential are useful for many Rust and C/C++ projects. Only remove them if you're sure you won't need them again.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully local, real-time voice agent in Rust that listens, thinks, and speaks without ever touching the internet.

You've learned how to:

  • Built real-time audio capture and Voice Activity Detection from scratch using cpal and energy-based RMS thresholds on Linux/ALSA.
  • Integrated local speech-to-text via whisper-rs and connected to a local LLM (Gemma 4 E4B) through Ollama's HTTP API for multi-turn conversation.
  • Wired the complete STT → LLM → TTS pipeline into a working conversation loop using Kokoro 82M for neural text-to-speech, all running 100% offline.
  • Added barge-in support so you can interrupt the agent mid-speech for natural conversation flow.

Ready to quiz yourself?