Build a Voice AI Agent on LiveKit

Build a real-time voice assistant on your self-hosted LiveKit server.

Introduction

30 Second Summary

Voice assistants like Siri and Alexa feel like magic, but they rely on massive cloud infrastructure you don't control. What if you could build your own voice AI that runs entirely on your hardware and responds in real-time?

In this project, you will build a real-time voice agent on your self-hosted LiveKit server. You'll speak into your browser, hear an AI respond using your local GPU, and trace the full WebRTC connection lifecycle through server logs.

What You'll Build

You open a browser tab, join a LiveKit room, and have a spoken conversation with an AI agent that transcribes your voice, thinks with a local Gemma model on your GPU, and speaks its answer back to you in real-time.

By the end of this project, you'll have:

  • A real-time voice conversation where you speak into your browser microphone and hear an AI respond instantly, powered by your RTX 4080 running Gemma 4 through Ollama.
  • The ability to read and explain LiveKit server logs, pointing to the exact log line for each stage: signaling handshake, ICE candidate exchange, DTLS encryption, track publication, and media routing.
  • A function tool that the agent calls mid-conversation, proving your agent can take real actions beyond simple chat.
  • Secret Mission: Instrument your agent with observability hooks that capture session transcripts and latency metrics to a local JSON file.

Are there any prerequisites?

This project assumes you already have a LiveKit server running on WSL2 and Ollama running on Windows with a model loaded. You should also have basic familiarity with Python.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

You already have a LiveKit server running on WSL and Ollama with Gemma 4 on your GPU. In this project, you'll connect those pieces into a real-time voice AI agent that you can talk to through your browser. The skills you'll practice are exactly what a Forward Deployed Engineer at LiveKit does for customers every day.

Set Up Your Agent Development Environment

Your LiveKit server is already running on WSL, and Ollama with Gemma 4 is serving inference on your GPU. But right now there is no agent code, no SDK, and no way to bridge speech to your LLM.

In this step you will install Python 3.12, the LiveKit Agents framework and its plugins, and wire up credentials so the agent process can talk to your server, your LLM, and an external speech provider.

In this step, get ready to:
  • Install Python 3.12 on WSL and create a virtual environment for the agent project.
  • Install the LiveKit Agents SDK, plugins, and the LiveKit CLI.
  • Create an ElevenLabs account, gather credentials, and verify Ollama is reachable from WSL.
Install Python 3.12 and create a virtual environment

The LiveKit Agents SDK and its plugin ecosystem target Python 3.12. Your WSL instance may ship with a different default version, so you need to confirm what is installed before creating the project environment.

  • Check which Python version is available on your WSL instance by running:
python3 --version

✔️ I see version 3.12

Python 3.12 is already your default interpreter. Skip straight to creating the virtual environment below.

ⓧ I see a higher version (3.13 or 3.14)

Ubuntu 26.04 defaults to a newer Python. You need to install Python 3.12 alongside it using the deadsnakes PPA. The system python3 stays unchanged.

  • Add the deadsnakes PPA and install Python 3.12 by running:
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.12 python3.12-venv -y
  • Confirm the versioned interpreter works by running:
python3.12 --version

You should see Python 3.12.x in the output.

Unable to locate package python3.12?

Run sudo apt update again after adding the PPA. If the package is still unavailable, confirm your system is Ubuntu 26.04 with lsb_release -a.

Help me troubleshoot this

ⓧ Command not found

Python is not installed at all. Install it from the deadsnakes PPA.

  • Add the deadsnakes PPA and install Python 3.12 by running:
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.12 python3.12-venv -y
  • Confirm the versioned interpreter works by running:
python3.12 --version

You should see Python 3.12.x in the output.

Now create your project directory and virtual environment.

  • Move to your home directory and create the project folder by running:
cd ~
mkdir livekit-agent && cd livekit-agent
  • Create and activate a virtual environment using Python 3.12 by running:
python3.12 -m venv .venv
source .venv/bin/activate
  • Confirm the virtual environment is active by running:
python --version

You should see Python 3.12.x and your prompt should show (.venv) at the beginning.

Why Python 3.12 instead of 3.14?

The LiveKit Agents SDK and its plugin ecosystem are tested against Python 3.12. Running a newer interpreter can introduce compatibility issues with native C extensions used by the audio processing libraries.

Seeing ensurepip is disabled?

This means the python3.12-venv package is missing. Install it with sudo apt install python3.12-venv and try creating the venv again.

Help me fix this venv error

Install the LiveKit Agents framework, plugins, and CLI

The agent needs four Python packages to wire up the voice pipeline, plus a utility for environment variables.

  • Create a requirements.txt file in your livekit-agent folder with the following contents:
livekit-agents
livekit-plugins-openai
livekit-plugins-elevenlabs
livekit-plugins-silero
python-dotenv

What are these packages?

  • livekit-agents is the core framework that handles agent lifecycle, session management, and room connection.
  • livekit-plugins-openai provides the LLM integration. Ollama exposes an OpenAI-compatible API, so this plugin connects to it.
  • livekit-plugins-elevenlabs handles both speech-to-text and text-to-speech via ElevenLabs.
  • livekit-plugins-silero provides Voice Activity Detection (VAD) to detect when you stop speaking.
  • python-dotenv loads credentials from a .env file so you never hardcode secrets.
  • Install all the packages by running:
pip install -r requirements.txt
  • Confirm livekit-agents installed successfully by running:
pip show livekit-agents

You should see package metadata including the name, version, and install location.

Install fails with a build error?

Some plugins compile C extensions. If you see errors mentioning missing headers, install the development package by running sudo apt install python3.12-dev and retry the pip install.

Help me fix this build error

Next, install the LiveKit CLI. This gives you the lk command for generating access tokens and managing rooms from the terminal.

  • Install the LiveKit CLI by running:
curl -sSL https://get.livekit.io/cli | bash
  • Verify the CLI is installed by running:
lk --version

You should see a version number printed. If your shell cannot find lk, open a new terminal tab or run source ~/.bashrc to refresh your path.

Create an ElevenLabs account, configure credentials, and verify Ollama connectivity

Now set up your ElevenLabs account for speech-to-text and text-to-speech.

  • Open https://elevenlabs.io in your browser.
  • Click Sign Up and create a free account. No credit card is required.
  • After signing in, click your profile icon in the bottom-left corner.
  • Select API Keys from the menu.
  • Click Create API Key.
  • Copy the key immediately. You will not be able to see it again after closing the modal.

What does the free tier include?

ElevenLabs free plan gives you 10,000 credits per month. For TTS, 1 credit equals 1 character. A typical agent response of 200 characters costs 200 credits, so you can have roughly 50 full voice interactions per month on the free tier.

Your agent needs credentials for three services: the LiveKit server, ElevenLabs, and Ollama. A .env file keeps these out of your code.

First, find the gateway IP that WSL uses to reach your Windows host. This is the nameserver entry in /etc/resolv.conf.

  • Print your WSL gateway IP by running:
cat /etc/resolv.conf | grep nameserver | awk '{print $2}'

You will see an IP address like 172.x.x.x. Write this down. This is how WSL reaches Windows services like Ollama.

Why not just use localhost?

WSL2 runs inside a lightweight virtual machine with its own network stack. WSL's localhost is NOT the same as Windows' localhost. To reach services running on Windows (like Ollama), you must use the gateway IP from /etc/resolv.conf.

  • Verify that Ollama is reachable from WSL by running:
curl http://$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):11434/api/tags

You should see a JSON response listing your Ollama models, including Gemma 4.

Connection refused from Ollama?

Ollama must be configured to listen on all interfaces, not just localhost. On Windows, set the environment variable OLLAMA_HOST=0.0.0.0 and restart Ollama. This allows WSL to connect over the VM boundary.

Help me fix Ollama connectivity from WSL

Now you have confirmed network connectivity. Time to create the credentials file.

  • Your LiveKit server's API key and secret are in the config file you used to start the server. Find them by checking your LiveKit server config (typically livekit.yaml or the environment variables you passed at startup).
  • Create a .env file in your livekit-agent directory with these contents (replace the placeholder values with your actual credentials):
LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=your-api-key-here
LIVEKIT_API_SECRET=your-api-secret-here
ELEVEN_API_KEY=your-elevenlabs-api-key-here
OLLAMA_BASE_URL=http://172.x.x.x:11434/v1

What do these values mean?

  • LIVEKIT_URL points to your self-hosted server's WebSocket signaling endpoint.
  • LIVEKIT_API_KEY and LIVEKIT_API_SECRET authenticate your agent with the server. These come from your server's config.
  • ELEVEN_API_KEY authenticates with ElevenLabs for STT and TTS.
  • OLLAMA_BASE_URL points to Ollama's OpenAI-compatible endpoint on your Windows host, reachable via the WSL gateway IP at port 11434. The /v1 suffix is required for OpenAI API compatibility.
  • Replace 172.x.x.x in OLLAMA_BASE_URL with the actual IP address you noted from the earlier command.
  • Also create a .env.example file so collaborators know which variables are needed without seeing your actual secrets:
LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=your-api-key-here
LIVEKIT_API_SECRET=your-api-secret-here
ELEVEN_API_KEY=your-elevenlabs-api-key-here
OLLAMA_BASE_URL=http://172.x.x.x:11434/v1
  • Confirm all files are in place by running:
ls -a

You should see .env, .env.example, .venv, and requirements.txt listed.

Before you move on, do a final verification of the full setup. What do you expect to see when you check your Python version and installed packages?

  • Run the final verification commands:
python --version
pip show livekit-agents

You should see Python 3.12.x and the livekit-agents package details confirming installation.

✔️ Awesome, I've got everything!

Your environment is ready. You have Python 3.12 in a virtual environment, all packages installed, the LiveKit CLI available, and credentials configured.

ⓧ I'd like to double check the full code

Here is what your requirements.txt should contain:

livekit-agents
livekit-plugins-openai
livekit-plugins-elevenlabs
livekit-plugins-silero
python-dotenv

And your .env.example should look like:

LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=your-api-key-here
LIVEKIT_API_SECRET=your-api-secret-here
ELEVEN_API_KEY=your-elevenlabs-api-key-here
OLLAMA_BASE_URL=http://172.x.x.x:11434/v1

Your development environment is wired up and every service is reachable. Next up, you will write the agent code and hear your AI respond for the first time.

Build and Run Your Voice Agent

Your development environment is ready with all the packages and credentials in place. Now it's time to write the actual agent that will listen to your voice, think with Gemma 4 on your GPU, and speak back to you.

The challenge in this step is getting all the pieces to talk to each other across the WSL-to-Windows boundary. Your LiveKit server runs on WSL, your Ollama runs on Windows, and the agent process needs to bridge both worlds.

In this step, get ready to:
  • Write an agent file that wires together STT, LLM, TTS, and VAD into a voice pipeline.
  • Run the agent in development mode and connect from your browser.
  • Diagnose and fix a WSL networking issue to get a working voice conversation.
Write the agent imports and configuration

Your agent needs to import the LiveKit Agents framework, the AI provider plugins, and load environment variables. It also needs to know where Ollama is running so it can send LLM requests there.

  • Create a new file called agent.py in your project folder by running:
touch agent.py
  • Open agent.py in your editor and add the following imports and configuration:
import os
from datetime import datetime
from dotenv import load_dotenv

from livekit.agents import Agent, AgentSession, RunContext, WorkerOptions, cli, function_tool
from livekit.plugins import openai, elevenlabs, silero

# Load environment variables from .env
load_dotenv()

# Point at Ollama running on Windows (via WSL gateway IP)
OLLAMA_BASE_URL = "http://localhost:11434/v1"

What does this code do?

The imports pull in the LiveKit Agents framework (Agent, AgentSession, WorkerOptions) and the plugins for each AI provider: openai (used for Ollama's OpenAI-compatible API), elevenlabs (STT and TTS), and silero (voice activity detection).

The OLLAMA_BASE_URL is set to localhost:11434 for now. This seems natural since Ollama is running on "your machine." You will discover shortly why this does not work from inside WSL.

Define the create_agent function

The create_agent function assembles the full voice pipeline: which STT transcribes your speech, which LLM generates a response, and which TTS speaks it back.

  • Add the following function below the OLLAMA_BASE_URL line in agent.py:
def create_agent() -> Agent:
    return Agent(
        instructions=(
            "You are a helpful voice assistant running on a self-hosted LiveKit server. "
            "You are powered by a local Gemma model via Ollama. "
            "Keep your responses concise and conversational since this is a voice interaction. "
            "You have access to tools for getting the current time and system information."
        ),
        stt=elevenlabs.STT(
            model_id="scribe_v1",
            language_code="en",
        ),
        llm=openai.LLM(
            model="gemma3:4b",
            base_url=OLLAMA_BASE_URL,
            api_key="ollama",
        ),
        tts=elevenlabs.TTS(
            model="eleven_flash_v2_5",
        ),
        vad=silero.VAD.load(),
        tools=[],
    )

What does this code do?

  • stt=elevenlabs.STT(model_id="scribe_v1") uses ElevenLabs' cloud STT to transcribe your speech to text.
  • llm=openai.LLM(model="gemma3:4b", base_url=OLLAMA_BASE_URL, api_key="ollama") sends the transcript to Gemma 4 via Ollama's OpenAI-compatible endpoint. The api_key="ollama" is a dummy value since Ollama does not require authentication.
  • tts=elevenlabs.TTS(model="eleven_flash_v2_5") converts the LLM response into speech using ElevenLabs' ultra-low-latency Flash model.
  • vad=silero.VAD.load() runs a lightweight local model that detects when you stop talking, so the agent knows when to respond.
Add the entrypoint and server launcher

The entrypoint is the function LiveKit calls when a participant joins a room and triggers the agent. The cli.run_app call gives your script a CLI interface with built-in dev and start subcommands.

  • Add the following at the bottom of agent.py:
async def entrypoint(ctx: RunContext):
    # Create the agent and start a session in this room
    agent = create_agent()
    session = AgentSession()
    await session.start(agent=agent, room=ctx.room)


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

What does this code do?

When a participant joins, LiveKit dispatches a job to your agent worker. The entrypoint function receives a RunContext with the room the agent should join. It creates the agent, starts a session, and begins listening for audio.

The cli.run_app(WorkerOptions(...)) line turns your script into a CLI tool. Running python agent.py dev starts the agent in development mode with hot-reload and debug logging.

  • Save agent.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py before moving on.

ⓧ I'd like to double check the full code

Your agent.py should contain the three sections you just built, in this order:

  • Imports and OLLAMA_BASE_URL configuration (the first chunk above).
  • The create_agent() function (the second chunk).
  • The entrypoint function and if __name__ == "__main__" block (the third chunk).

Scroll up and confirm each section is present in your file.

Run the agent and hit the networking wall

Time to see if the pieces connect. You will start the agent, connect from your browser, and discover the first real obstacle.

  • Start the agent in development mode by running:
python agent.py dev

What does this command do?

The dev subcommand starts your agent worker with debug-level logging and hot-reload enabled. It reads LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET from your .env file automatically.

You should see log output showing the agent registering with your LiveKit server. The agent is now waiting for someone to join a room.

  • Open a new terminal tab (keep the agent running in the first one).
  • Generate a connection token by running:
lk token create --api-key $(grep LIVEKIT_API_KEY .env | cut -d '=' -f2) --api-secret $(grep LIVEKIT_API_SECRET .env | cut -d '=' -f2) --join --room test-room --identity user1 --valid-for 24h
  • Copy the token that is printed to your terminal.
  • Open your browser and navigate to https://meet.livekit.io.
  • Click the Custom tab.
  • Enter ws://localhost:7880 in the LiveKit URL field.
  • Paste your token in the Token field.
  • Click Connect.

You should see yourself join the room, and moments later the agent appears as a second participant. This is your first visible win. The agent successfully registered with LiveKit and joined the room.

Before you speak, what do you think will happen when the agent tries to send your transcribed speech to Ollama at localhost:11434?

  • Speak a short sentence into your microphone (e.g., "Hello, can you hear me?").
  • Switch back to the terminal running the agent and look at the log output.

You will see a connection error in the logs. The agent cannot reach Ollama at localhost:11434.

Seeing a connection error? That's expected.

WSL2 runs inside a lightweight virtual machine with its own network stack. When your agent code says localhost, it means WSL's localhost, not Windows' localhost where Ollama is actually running.

This is a common stumbling point in WSL2 development and a real networking concept you need as a Forward Deployed Engineer. The fix is to use the WSL gateway IP, which routes traffic from WSL to Windows.

Fix the Ollama URL and confirm your voice conversation works

You already set the correct gateway IP in your .env file back in Step 1 (the OLLAMA_BASE_URL variable). The fix is to read that value instead of hardcoding localhost.

  • In agent.py, find this line near the top:
OLLAMA_BASE_URL = "http://localhost:11434/v1"
  • Replace it with this line that reads from your .env file:
OLLAMA_BASE_URL = os.getenv("OLLAMA_BASE_URL", "http://172.x.x.x:11434/v1")

Why does this fix the problem?

Your .env file already contains OLLAMA_BASE_URL set to the WSL gateway IP (the IP from /etc/resolv.conf that you configured in Step 1). Now os.getenv reads that value, and LLM requests route from WSL through the gateway to Windows where Ollama is listening.

  • Save agent.py.
  • The agent should auto-restart thanks to dev mode's hot-reload. If it does not, press Ctrl+C and run python agent.py dev again.

Before you test again, what do you expect to happen this time when you speak?

  • Go back to your browser tab with meet.livekit.io still connected (or reconnect if it timed out).
  • Speak a sentence like "What can you help me with?"
  • Listen for the agent's voice response.

You should hear the agent respond in a natural voice within a few seconds. You are having a real-time voice conversation with an AI running on your own GPU.

Not hearing a response?

  • Check the agent terminal for errors. If you see a timeout from Ollama, confirm Ollama is still running on Windows by visiting http://localhost:11434 in a Windows browser.
  • If you see an ElevenLabs authentication error, double-check your ELEVEN_API_KEY in .env is correct.
  • If the agent does not appear in the room at all, verify your LiveKit server is still running and that LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET in .env match your server config.

help me debug my LiveKit voice agent not responding

Your voice agent is alive and talking. Next up, you will look under the hood at the LiveKit server logs to understand exactly what is happening during each stage of the connection.

Trace the Architecture Through Server Logs

You can talk to your voice agent. That is a working demo. But to work as a Forward Deployed Engineer, a demo is not enough.

You need to understand what is happening inside the server at every stage of a connection. When a customer asks why audio cuts out, or why their agent takes 3 seconds to respond, the answer lives in the logs. In this step, you will restart your LiveKit server with debug logging and trace a complete WebRTC connection lifecycle from signaling to media delivery.

In this step, get ready to:
  • Restart the LiveKit server with debug-level logging enabled.
  • Trace the signaling, ICE, and DTLS phases in the server logs.
  • Map the agent lifecycle through worker registration, job assignment, and track publication.
Restart the server with debug logging

The LiveKit server supports a --dev flag that sets the log level to debug and uses a human-readable console formatter. This surfaces every internal event the server processes.

  • In the WSL terminal where your LiveKit server is running, press Ctrl+C to stop it.
  • Restart the server with the --dev flag by running this command:
livekit-server --dev --bind 0.0.0.0 --keys "$LIVEKIT_API_KEY: $LIVEKIT_API_SECRET"

What does --dev do?

The --dev flag does three things: sets the log level to debug, switches to a colorful console formatter (instead of JSON), and enables pprof endpoints for profiling. The debug level reveals every signaling message, ICE candidate, and track event that flows through the server.

If you were already running the server with --dev, you already have debug logs. The restart clears old output so you get a clean trace of a single connection.

  • Confirm you see a startup message in the terminal output indicating development mode is active.

Server won't start?

If you see an error about the port already being in use, the previous server process may not have fully stopped. Wait a few seconds and try again, or check for lingering processes with lsof -i :7880.

If you see an error about keys, make sure the colon-space format matches exactly: "yourkey: yoursecret" with a space after the colon.

Help me troubleshoot my LiveKit server not starting in dev mode on WSL

Trace the signaling phase

With debug logs flowing, you will now connect a client and your agent, then read the logs to understand the signaling handshake. This is the first phase of any WebRTC connection.

  • Open a second WSL terminal. Activate your virtual environment and start your agent in dev mode by running:
cd ~/Desktop/livekit-agent
source venv/bin/activate
python agent.py dev
  • Open your browser and connect to the LiveKit room via meet.livekit.io using ws://localhost:7880 as your server URL.
  • Switch back to the LiveKit server terminal (the first one). You will see a flood of debug log lines.

Look for these events in order. They tell the story of how your browser client joins the room:

  • WebSocket connection accepted. The browser opens a WebSocket to the server on port 7880. Look for log lines mentioning a new signal connection or session starting. This is the signaling channel.
  • Join request received. The server processes the client's join request and assigns a participant ID. Look for log entries that reference a participant identity and a room name.
  • SDP offer/answer exchange. The server and client negotiate media capabilities. Look for lines mentioning offer, answer, or SDP. The offer describes what media the client wants to send. The answer confirms what the server can accept.

What is signaling?

Signaling is the initial negotiation phase of WebRTC. It happens over a WebSocket. The browser and server exchange messages about media formats, encryption parameters, and network addresses. No actual audio flows during signaling. Think of it as two people exchanging phone numbers before calling each other.

The SFU (Selective Forwarding Unit) acts as a media router. It does not mix or transcode audio. It receives media from publishers and forwards it to subscribers, making it efficient for multi-party communication.

Trace the ICE and media phase

After signaling completes, the connection moves to the ICE and media phase. This is where actual network connectivity is established and audio starts flowing.

  • In the server logs, look for lines referencing ICE candidates. These are network addresses (IP and port combinations) that the client and server can potentially use to communicate.
  • Find the ICE candidate pair selection. The server and client test multiple candidate pairs and select the best one. In a local setup like yours, this will likely be a direct UDP connection.
  • Identify the DTLS handshake completion. DTLS is the encryption layer for WebRTC media. Once the candidate pair is selected, the two sides perform a DTLS handshake to establish encryption keys. Look for state changes related to DTLS or transport security.
  • Find audio track publication. Once the encrypted channel is established, the browser publishes its audio track. Look for log entries mentioning a track being added, a track type of audio, or an AddTrack request.

What is ICE?

ICE (Interactive Connectivity Establishment) is the process of discovering and testing network paths between two endpoints. Each endpoint gathers candidate addresses from its network interfaces, STUN servers, and TURN servers. The endpoints then test pairs of candidates until they find one that works.

In your local setup, both the browser and server are on the same machine, so ICE resolves quickly using local candidates. In production with clients behind NATs and firewalls, ICE is where most connectivity failures happen.

  • Note the track IDs assigned by the server. Each published track gets a unique identifier that appears throughout the logs whenever that track is forwarded to a subscriber.

Before you continue, predict: when the agent publishes its audio track back to the room, will it go through the same signaling and ICE process as the browser client?

Trace the agent lifecycle

The agent is a special participant. It connects programmatically via the LiveKit Agents SDK rather than through a browser. The server logs reveal how this differs from a browser connection.

  • In the server logs, find where the agent worker registers with the server. The agent process connects and announces itself as available to handle jobs. Look for log entries mentioning worker or agent registration.
  • Find the job assignment. When a participant (you) joins a room, the server dispatches a job to the registered agent worker. Look for log entries referencing job dispatch or assignment.
  • Find where the agent joins the room as a participant. After receiving the job, the agent joins the same room as you. It gets its own participant ID and goes through its own signaling and ICE process.
  • Find the agent subscribing to your audio track. The server routes your published audio to the agent. Look for subscription-related log entries referencing the agent's participant ID subscribing to a track.
  • Find the agent publishing its own audio track. After processing your speech through STT, LLM, and TTS, the agent publishes an audio track back to the room. The server then routes this track to your browser as a subscription.

The full request lifecycle

You can now trace the complete path of a voice interaction through the logs:

  • Your browser captures audio and publishes it as a WebRTC track to the server.
  • The server forwards that audio track to the agent (subscriber).
  • The agent's STT transcribes the audio, the LLM generates a response, and TTS produces audio.
  • The agent publishes the response as a new audio track to the server.
  • The server forwards that audio track back to your browser (subscriber).

The answer to the earlier prediction: yes, the agent goes through the same signaling and ICE process as the browser. It is just another participant from the server's perspective. The server does not treat agent connections differently from browser connections at the transport layer.

You have now traced a complete WebRTC connection lifecycle through the server logs.

You can now read LiveKit's internal state like a book. Next up, you will add function tools to your agent to demonstrate how the framework extends beyond simple conversation.

Add a Function Tool to Your Agent

Your voice agent can hold a conversation, but it has no way to take actions or retrieve real information. Right now, if you ask it "what time is it?" the LLM can only guess.

Real agent deployments need tools that let the LLM call functions and return actual data to the user. In this step, you will define function tools using the LiveKit Agents framework's @function_tool() decorator, register them with your agent, and verify the full tool invocation flow in a live conversation.

In this step, get ready to:
  • Define two function tools that return real system data.
  • Run the agent and trigger a tool invocation by asking a question.
Define function tools using the @function_tool() decorator

The @function_tool() decorator exposes a Python function to the LLM. The LLM reads the function's docstring to decide when to call it, executes the function, and streams the result through TTS back to the user.

  • In agent.py, find the blank line just above def create_agent().
  • Add the following two function tool definitions by pasting this code above def create_agent():
@function_tool()
async def get_current_time(context: RunContext) -> str:
    """Get the current date and time."""
    now = datetime.now()
    return now.strftime("%A, %B %d, %Y at %I:%M %p")


@function_tool()
async def get_system_info(context: RunContext) -> str:
    """Get information about the system this agent is running on."""
    import platform
    return f"Running on {platform.system()} {platform.release()}, Python {platform.python_version()}"

What does this code do?

Each function is decorated with @function_tool() which registers it as a callable tool for the LLM. The docstring (e.g. "Get the current date and time.") is what the LLM reads to decide when to invoke the tool.

Both functions accept a context: RunContext parameter, which gives the tool access to the current agent session. They return a plain string that gets passed back to the LLM, which then speaks it to the user through TTS.

Now you need to tell the Agent to actually use these tools. The agent's tools list is currently empty.

  • In agent.py, scroll down to the create_agent() function.
  • Find this line:
        tools=[],
  • Replace it with:
        tools=[get_current_time, get_system_info],
  • Save agent.py.

Why pass functions to the tools list?

The tools parameter on the Agent tells the LLM which functions are available to call during a conversation. Without registering a tool here, the LLM cannot invoke it even if the decorator is present.

✔️ Awesome, I've got everything!

Great. Make sure you have saved agent.py before continuing.

ⓧ I'd like to double check the full code

Your agent.py should now contain these sections in order:

  • Imports and load_dotenv() at the top.
  • The OLLAMA_BASE_URL variable.
  • The get_current_time function tool with the @function_tool() decorator.
  • The get_system_info function tool with the @function_tool() decorator.
  • The create_agent() function with tools=[get_current_time, get_system_info].
  • The entrypoint() async function.
  • The if __name__ == "__main__" block calling cli.run_app().
Run the agent and trigger a tool invocation

With the tools registered, you can now test the full invocation flow: you speak a question, the LLM decides to call a tool, the tool executes, and the result is spoken back to you.

  • Start the agent in development mode by running:
python agent.py dev
  • Open your browser and connect via meet.livekit.io using ws://localhost:7880 as the server URL.
  • Wait for the agent to join the room as a participant.

Before you speak, think about this: if you ask "what time is it?", do you think the agent will guess a time, or return the actual current time from your system?

  • Speak into your microphone and ask: "What time is it right now?"
  • Listen to the agent's response and watch the terminal where the agent is running.

You should hear the agent respond with the correct current date and time. In your terminal, you will see log output showing the tool invocation: the LLM decided to call get_current_time, the function executed, and the result was streamed back through TTS.

  • Now try asking: "What system are you running on?"

The agent calls get_system_info and reads back details about your WSL environment and Python version.

What is happening in the logs?

When a tool is invoked, the agent logs show a sequence of events. The LLM outputs a tool-call request with the function name. The framework executes the function and returns the string result.

The LLM then incorporates that result into its next response, which is streamed through TTS. This is the same flow that production agents use for booking appointments, querying databases, or calling external APIs.

Agent not calling the tool?

If the agent responds with a guessed time instead of calling the tool, check that tools=[get_current_time, get_system_info] is set in your create_agent() function. An empty tools=[] list means the LLM has no tools available.

If you see an error about the function signature, verify that both functions have context: RunContext as their first parameter and return a str.

Help me debug why my agent isn't calling tools

Secret mission

Add Agent Observability with Data Hooks

Your agent works. But can you prove it works well? In production, LiveKit FDEs instrument every agent deployment with session logging and latency profiling. Your mission: add data hooks that capture transcripts and timing metrics, then use the data to identify your pipeline's bottleneck.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine and uses the ElevenLabs free tier, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Python virtual environment and agent project folder on WSL (local disk only).
  • ElevenLabs free account (no cost, 10,000 credits/month, no credit card required).
  • LiveKit CLI binary installed at /usr/local/bin/lk on WSL.
  • LiveKit server running on WSL (pre-existing, no cloud cost).
  • Ollama running on Windows with Gemma 4 (pre-existing, no cloud cost).

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend your agent further.

  • Your LiveKit server and Ollama stay running. They use no cloud resources and cost nothing.
  • Your ElevenLabs free tier resets to 10,000 credits every month automatically.
  • Your Python virtual environment, project files, and session logs remain on disk in your WSL home directory.
  • The LiveKit CLI remains installed and available for future projects.

Pause - I'll come back to this later

Shut down running processes to free up system memory. Your files stay on disk and everything restarts instantly when you are ready.

  • Stop the LiveKit server by pressing Ctrl+C in the WSL terminal where it is running.
  • Stop Ollama by right-clicking its icon in the Windows system tray and selecting Quit.
  • Deactivate your Python virtual environment by running:
deactivate
  • Your project folder, session logs, and .env file remain untouched on disk.
  • To resume later, start Ollama from the Windows Start menu, restart LiveKit server on WSL, reactivate your virtual environment, and run your agent.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your agent code, session logs, and local environment.

  • Stop the LiveKit server and agent process if they are still running by pressing Ctrl+C in each terminal.
  • Deactivate your virtual environment and delete the entire project folder by running:
deactivate
cd ~
rm -rf ~/Desktop/livekit-agent

What does this remove?

This deletes your agent.py, .env, requirements.txt, README.md, the session_logs/ directory, and the Python virtual environment all at once.

  • Remove the LiveKit CLI binary by running:
sudo rm /usr/local/bin/lk
  • Delete your ElevenLabs account (optional, since it costs nothing). Log in at elevenlabs.io, click your profile icon in the top-right corner, select Settings, scroll down, and click the red Delete Account button. This is permanent and you cannot reuse the same email.

What about LiveKit server and Ollama?

These were pre-existing on your system before this project. If you still use them for other work, leave them installed.

If you want to remove them entirely, refer to their respective documentation for uninstall instructions.

Nice Work!

Nice Work!

You just built a fully self-hosted voice AI agent that runs entirely on your local infrastructure. You can speak into your browser and have a real-time conversation powered by your own GPU.

You've learned how to:

  • Build a real-time voice AI agent using the LiveKit Agents framework with ElevenLabs STT/TTS and a local Ollama LLM running on your GPU.
  • Trace a complete WebRTC connection lifecycle through LiveKit server logs, understanding signaling, ICE negotiation, DTLS encryption, and track routing.
  • Extend the agent with function tools and navigate WSL-to-Windows networking for cross-environment service communication.
  • Secret Mission: Instrumented the agent with observability hooks for production-grade session monitoring and latency profiling.

Ready to quiz yourself?