Build a Voice AI Smart Home Assistant

Build a Python voice assistant that controls smart home devices using GPT-4o.

Introduction

30 Second Summary

Voice assistants like Alexa and Siri feel like magic, but they really just do three things: listen, understand, and act. Today you'll build that same system yourself.

In this project, you will build a voice-controlled smart home assistant in Python. You'll speak commands into your microphone, and your assistant will use AI to understand what you want and control simulated devices in your home.

What You'll Build

A Python voice assistant powered by OpenAI's speech-to-text and LLM APIs that listens to your spoken commands, understands your intent, and controls simulated smart home devices.

By the end of this project, you'll have:

  • A speech-to-text pipeline that captures audio from your microphone and transcribes it using OpenAI's gpt-4o-mini-transcribe model.
  • An LLM-powered intent parser that understands natural language commands and extracts the target device, action, and value.
  • A simulated smart home with lights, a thermostat, and appliances that all track their state across voice commands.
  • A status report voice command that gives you a spoken summary of every device in your home (Secret Mission).

Are there any prerequisites?

No prior experience is needed. This project walks you through installing every tool from scratch, including Python, Homebrew, and all required libraries. You will need a Mac and an OpenAI API key, which costs about $0.05 for testing.

Set Up Python and Your Project

Before you can build a voice assistant, your Mac needs the right tools installed. Python is the programming language you will write your assistant in. OpenAI's API is the AI engine that will power speech recognition and command understanding.

In this step, you will install Python, set up a few system-level dependencies, and create an isolated project folder with everything your assistant needs.

In this step, get ready to:
  • Install Python 3.13 and verify it works.
  • Install Homebrew and PortAudio for microphone access.
  • Create your project folder, virtual environment, and install all required Python packages.
Install Python 3.13

Your Mac may already have an older version of Python, but this project needs Python 3.13. You will download the official installer from python.org, which is the simplest way to get started.

  • Open your web browser and go to python.org/downloads.
  • Click the Download Python 3.13 button.
  • Once the .pkg file has downloaded, double-click it to open the installer.
  • Follow the installer prompts, clicking Continue and then Install when prompted.

After the installer finishes, you need to install SSL certificates so Python can make secure network requests to the OpenAI API.

  • Open Finder.
  • Navigate to /Applications/Python 3.13/.
  • Double-click Install Certificates.command to run it.

A Terminal window will open and run the certificate installation automatically. It closes on its own when finished.

Now verify that Python is installed correctly. Open Terminal by pressing Cmd+Space, typing Terminal, and pressing Enter.

  • Check your Python version by running this command:
python3 --version

I see Python 3.13

You should see something like Python 3.13.x. You are all set to continue.

Command not found

If your Terminal does not recognize python3, the installer may not have completed successfully.

  • Close Terminal and reopen it so it picks up the updated PATH.
  • Run python3 --version again.
  • If it still does not work, re-run the .pkg installer from your Downloads folder.
Install Homebrew and PortAudio

Your voice assistant needs to access your Mac's microphone. The Python library that handles this (PyAudio) depends on a system-level audio library called PortAudio. You install PortAudio using Homebrew, a popular package manager for macOS.

  • Check if Homebrew is already installed by running this command:
brew --version

I see a version number

Homebrew is already installed. Skip ahead to the PortAudio install step below.

Command not found

Homebrew is not installed yet. Install it by running this command in Terminal:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

What happens during the Homebrew install?

The installer will ask for your Mac password and show you what it plans to install. Press Enter to confirm. This process may take a few minutes.

Once the install finishes, close and reopen Terminal so the brew command becomes available.

  • Install PortAudio by running this command:
brew install portaudio

Why install PortAudio first?

On macOS with Apple Silicon (M1/M2/M3/M4 chips), PortAudio must be installed before the PyAudio Python package. PyAudio compiles against PortAudio during installation, so if PortAudio is missing, the install will fail.

Seeing an error with brew install?

If you see command not found: brew, make sure you closed and reopened Terminal after installing Homebrew.

If Homebrew asks you to run extra commands to add it to your PATH, follow those instructions and restart Terminal.

Help me troubleshoot my Homebrew or PortAudio installation on macOS.

Create your project folder and install packages

A virtual environment is an isolated space where Python installs packages just for this project. It keeps your assistant's dependencies separate from anything else on your Mac.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir smart-home-assistant
cd smart-home-assistant
  • Confirm the folder was created by running:
pwd

You should see /Users/your-username/Desktop/smart-home-assistant printed in Terminal.

  • Create a virtual environment by running this command:
python3 -m venv venv
  • Activate the virtual environment by running:
source venv/bin/activate

You should see (venv) appear at the beginning of your Terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

Think of it as a sandbox. Packages you install here only exist inside this project folder. When you are done working, you can deactivate it with deactivate. When you come back, reactivate it with source venv/bin/activate.

  • Install the three Python packages your project needs by running this command:
pip install openai==2.32.0 SpeechRecognition==3.14.1 pyaudio==0.2.14

What are these packages?

  • openai connects your Python code to OpenAI's API for speech transcription and intent parsing.
  • SpeechRecognition captures audio from your Mac's microphone and handles voice activity detection automatically.
  • pyaudio is the low-level audio driver that SpeechRecognition uses under the hood to talk to your microphone hardware.

The install may take a minute. When it finishes, you should see Successfully installed followed by the package names in your Terminal output.

Seeing an error during pip install?

  • If pyaudio fails with a build error, make sure you ran brew install portaudio in the previous substep. PortAudio must be installed before PyAudio.
  • If you see pip: command not found, make sure your virtual environment is active. You should see (venv) at the start of your Terminal prompt.

Help me troubleshoot my Python package installation for this voice assistant project.

  • Verify all packages installed correctly by running:
pip list

You should see openai, SpeechRecognition, and PyAudio listed among the installed packages.

Your development environment is ready. Next up, you will build the smart home device registry that your voice assistant will control.

Build the Smart Home Device Registry

Your Python environment is ready, your project folder is created, and your virtual environment is activated. Now it's time to build the foundation that your voice assistant will control.

Every smart home system needs a registry of devices it can manage. In this step, you'll create a Python class that models lights, a thermostat, and appliances. Each device tracks its own state, so your assistant always knows what's on, off, or set to a specific value.

In this step, get ready to:
  • Define a SmartHome class with six simulated devices.
  • Add methods to control devices and check their status.
  • Test the registry by running commands from the terminal.
Create the SmartHome class

Real smart home platforms like Home Assistant store every device as an entity with a name, type, and current state. You'll build a simplified version of this pattern in Python.

  • Create a new file called devices.py inside your smart-home-assistant folder (the one you created in Step 1).
  • Add the following code to define the SmartHome class and its device registry:
class SmartHome:
    def __init__(self):
        # Registry of all smart home devices and their current states
        self.devices = {
            "living room light": {"type": "light", "state": "off", "brightness": 100},
            "bedroom light": {"type": "light", "state": "off", "brightness": 100},
            "kitchen light": {"type": "light", "state": "off", "brightness": 100},
            "thermostat": {"type": "thermostat", "state": "off", "temperature": 70},
            "coffee maker": {"type": "appliance", "state": "off"},
            "fan": {"type": "appliance", "state": "off"},
        }

What does this code do?

  • SmartHome is a Python class that acts as your device registry. When you create an instance, it sets up six devices automatically.
  • self.devices is a dictionary where each key is a device name (like "living room light") and each value is another dictionary holding the device's type, state, and any extra properties.
  • Lights have a brightness property, the thermostat has a temperature property, and appliances just track on/off state.

You won't see any output yet. The next chunk adds the methods that make this class useful.

Add control and status methods

The device dictionary stores state, but you need methods to actually change that state. You'll add two methods: one to execute commands like "turn on" or "set to 72," and one to report the status of every device.

  • In devices.py, add the execute_command() method directly below the __init__ method, inside the SmartHome class. Copy the following code below the closing brace of the self.devices dictionary:
    def execute_command(self, device, action, value=None):
        # Check if the device exists in our registry
        if device not in self.devices:
            return f"Sorry, I don't know a device called '{device}'."

        dev = self.devices[device]

        # Handle on/off/set actions and update device state
        if action == "on":
            dev["state"] = "on"
            return f"Turned on the {device}."
        elif action == "off":
            dev["state"] = "off"
            return f"Turned off the {device}."
        elif action == "set":
            if dev["type"] == "thermostat" and value is not None:
                dev["temperature"] = value
                dev["state"] = "on"
                return f"Set the {device} to {value} degrees."
            elif dev["type"] == "light" and value is not None:
                dev["brightness"] = value
                dev["state"] = "on"
                return f"Set the {device} brightness to {value} percent."
            else:
                return f"I can't set a value on the {device}."
        else:
            return f"I don't understand the action '{action}'."

What does this method do?

  • execute_command() takes three arguments: the device name, the action to perform, and an optional value (for things like temperature).
  • It first checks whether the device exists in the registry. If not, it returns a friendly error message.
  • For "on" and "off" actions, it flips the device state. For "set" actions, it updates the thermostat's temperature or a light's brightness.
  • Every branch returns a string describing what happened. Your voice assistant will speak this string back to you later.

You won't see a change yet. The next method adds status reporting, and then you'll test everything together.

  • Still inside the SmartHome class, add the get_status() method directly below execute_command():
    def get_status(self):
        # Build a human-readable summary of all devices
        lines = ["Here is your smart home status:"]
        for name, dev in self.devices.items():
            if dev["type"] == "thermostat":
                lines.append(f"  {name}: {dev['state']}, set to {dev['temperature']} degrees")
            elif dev["type"] == "light":
                lines.append(f"  {name}: {dev['state']}, brightness {dev['brightness']} percent")
            else:
                lines.append(f"  {name}: {dev['state']}")
        return "\n".join(lines)

What does get_status() do?

  • This method loops through every device in the registry and builds a formatted summary line for each one.
  • It includes extra detail for thermostats (temperature) and lights (brightness). Appliances just show on or off.
  • The returned string is what your assistant will speak aloud when you ask for a home status report.
  • Save devices.py.

You won't see a change yet. The next substep adds a test block so you can run the file and verify everything works.

Test the device registry

Python's if __name__ == "__main__" pattern lets you add test code that only runs when you execute the file directly. It won't run when other files import this class later.

  • At the very bottom of devices.py, outside the SmartHome class (no indentation), add the following test block:
if __name__ == "__main__":
    # Create a SmartHome instance and test some commands
    home = SmartHome()
    print(home.execute_command("living room light", "on"))
    print(home.execute_command("thermostat", "set", 72))
    print(home.execute_command("coffee maker", "on"))
    print()
    print(home.get_status())

What does this test block do?

  • It creates a SmartHome instance, then runs three commands: turning on the living room light, setting the thermostat to 72 degrees, and turning on the coffee maker.
  • After the commands, it prints get_status() so you can see the full state of every device, including the ones you didn't touch.
  • Save devices.py.
  • Make sure your virtual environment is still active (you should see (venv) at the start of your terminal prompt). Run the test by entering this command:
python3 devices.py

You should see output like this:

Turned on the living room light.
Set the thermostat to 72 degrees.
Turned on the coffee maker.

Here is your smart home status:
  living room light: on, brightness 100 percent
  bedroom light: off, brightness 100 percent
  kitchen light: off, brightness 100 percent
  thermostat: on, set to 72 degrees
  coffee maker: on
  fan: off

Not seeing the expected output?

  • If you see IndentationError, make sure the if __name__ block has no indentation (it sits at the far left, outside the class).
  • If you see NameError: name 'SmartHome' is not defined, check that the if __name__ block is below the class definition, not inside it.
  • If nothing prints at all, make sure you saved the file and that you are running python3 devices.py from inside your smart-home-assistant folder.
  • Still stuck? Help me debug my devices.py output.

Awesome, I've got everything!

Double check you've saved devices.py and move on to the next step.

I'd like to double check the full code

class SmartHome:
    def __init__(self):
        self.devices = {
            "living room light": {"type": "light", "state": "off", "brightness": 100},
            "bedroom light": {"type": "light", "state": "off", "brightness": 100},
            "kitchen light": {"type": "light", "state": "off", "brightness": 100},
            "thermostat": {"type": "thermostat", "state": "off", "temperature": 70},
            "coffee maker": {"type": "appliance", "state": "off"},
            "fan": {"type": "appliance", "state": "off"},
        }

    def execute_command(self, device, action, value=None):
        if device not in self.devices:
            return f"Sorry, I don't know a device called '{device}'."

        dev = self.devices[device]

        if action == "on":
            dev["state"] = "on"
            return f"Turned on the {device}."
        elif action == "off":
            dev["state"] = "off"
            return f"Turned off the {device}."
        elif action == "set":
            if dev["type"] == "thermostat" and value is not None:
                dev["temperature"] = value
                dev["state"] = "on"
                return f"Set the {device} to {value} degrees."
            elif dev["type"] == "light" and value is not None:
                dev["brightness"] = value
                dev["state"] = "on"
                return f"Set the {device} brightness to {value} percent."
            else:
                return f"I can't set a value on the {device}."
        else:
            return f"I don't understand the action '{action}'."

    def get_status(self):
        lines = ["Here is your smart home status:"]
        for name, dev in self.devices.items():
            if dev["type"] == "thermostat":
                lines.append(f"  {name}: {dev['state']}, set to {dev['temperature']} degrees")
            elif dev["type"] == "light":
                lines.append(f"  {name}: {dev['state']}, brightness {dev['brightness']} percent")
            else:
                lines.append(f"  {name}: {dev['state']}")
        return "\n".join(lines)


if __name__ == "__main__":
    home = SmartHome()
    print(home.execute_command("living room light", "on"))
    print(home.execute_command("thermostat", "set", 72))
    print(home.execute_command("coffee maker", "on"))
    print()
    print(home.get_status())

Your smart home device registry is working. You can turn devices on and off, set values like temperature and brightness, and get a full status report. Next up, you'll capture voice commands from your Mac Mini's microphone so the assistant can listen to you.

Capture Voice Commands from Your Microphone

Your smart home device registry is ready. It knows about lights, a thermostat, a coffee maker, and a fan. Now you need a way to actually tell it what to do with your voice.

In this step, you'll build a voice capture module that listens through your Mac's microphone and saves the audio as a file. This file is what you'll send to OpenAI for transcription in the next step.

In this step, get ready to:
  • Set up microphone capture using the SpeechRecognition library.
  • Save captured audio as a WAV file for transcription.
  • Test voice capture by speaking a command from Terminal.
Build the voice capture module

Your assistant needs to listen through the Mac's microphone and turn spoken words into audio data. Python's SpeechRecognition library handles the heavy lifting. It manages microphone access, detects when you start and stop talking, and packages the result as audio data.

  • Create a new file called voice.py inside your smart-home-assistant folder.
  • Add the following code to define your listen() function:
import speech_recognition as sr
import os

# File path for saving recorded audio
AUDIO_FILE = "command.wav"


def listen():
    # Set up the recognizer and microphone
    recognizer = sr.Recognizer()

    with sr.Microphone() as source:
        print("Adjusting for ambient noise... please wait.")
        # Calibrate for background noise before listening
        recognizer.adjust_for_ambient_noise(source, duration=1)
        print("Listening... speak your command now.")

        try:
            # Record up to 8 seconds of speech
            audio = recognizer.listen(source, phrase_time_limit=8)
        except sr.WaitTimeoutError:
            print("No speech detected. Try again.")
            return False

    # Save to WAV file for OpenAI transcription
    with open(AUDIO_FILE, "wb") as f:
        f.write(audio.get_wav_data())

    print(f"Audio saved to {AUDIO_FILE}")
    return True

What does this code do?

  • sr.Recognizer() creates a recognizer object that handles voice activity detection automatically.
  • sr.Microphone() opens your Mac's default microphone as an audio source.
  • adjust_for_ambient_noise() listens for 1 second to calibrate for background noise. This prevents the recognizer from treating room hum or fan noise as speech.
  • recognizer.listen() with phrase_time_limit=8 records up to 8 seconds of speech. Without this limit, the function could hang indefinitely waiting for silence.
  • If no speech is detected, the WaitTimeoutError is caught and the function returns False so the caller knows nothing was captured.
  • The captured audio is saved as a WAV file called command.wav. This file format is what OpenAI's transcription API expects.
  • Save voice.py.

You won't see a change yet. The next chunk adds a test block so you can run the file and try it out.

Test voice capture from Terminal

To verify that your microphone capture works, you need a small test block that runs the listen() function when you execute the file directly.

  • At the bottom of voice.py, add this test block:
if __name__ == "__main__":
    success = listen()
    if success:
        print("Audio captured successfully! Ready for transcription.")
    else:
        print("No audio captured.")

What does this do?

The if __name__ == "__main__": guard makes this code run only when you execute voice.py directly. When other files import voice.py later, this test code is skipped.

Awesome, I've got everything!

Great. Double check you've saved voice.py before running it.

I'd like to double check the full code

import speech_recognition as sr
import os

AUDIO_FILE = "command.wav"


def listen():
    recognizer = sr.Recognizer()

    with sr.Microphone() as source:
        print("Adjusting for ambient noise... please wait.")
        recognizer.adjust_for_ambient_noise(source, duration=1)
        print("Listening... speak your command now.")

        try:
            audio = recognizer.listen(source, phrase_time_limit=8)
        except sr.WaitTimeoutError:
            print("No speech detected. Try again.")
            return False

    # Save to WAV file for OpenAI transcription
    with open(AUDIO_FILE, "wb") as f:
        f.write(audio.get_wav_data())

    print(f"Audio saved to {AUDIO_FILE}")
    return True


if __name__ == "__main__":
    success = listen()
    if success:
        print("Audio captured successfully! Ready for transcription.")
    else:
        print("No audio captured.")

Run this from Terminal, not from an IDE

On macOS, microphone permissions are granted per application. If you run this from inside VS Code or another IDE, the microphone permission may not work correctly.

Always run voice.py from the macOS Terminal app directly.

  • Open Terminal and make sure your virtual environment is active. You should see (venv) at the start of your prompt.
  • If it's not active, navigate to your smart-home-assistant folder and activate it by running:
source venv/bin/activate
  • Run the voice capture script by running this command:
python3 voice.py
  • Wait for the "Listening... speak your command now." message to appear, then say a command like "turn on the living room light."

What about the microphone permission popup?

The first time you run this, macOS will show a popup asking if Terminal can access your microphone. Click OK to allow it. Without this permission, the script cannot capture any audio.

You should see output like this in your Terminal:

Adjusting for ambient noise... please wait.
Listening... speak your command now.
Audio saved to command.wav
Audio captured successfully! Ready for transcription.
  • Confirm a file called command.wav was created in your smart-home-assistant folder by running:
ls command.wav

You should see command.wav in the output. This is your recorded audio, ready to be sent to OpenAI for transcription.

Not seeing any output or getting an error?

  • If you see a PortAudio or PyAudio error, make sure you installed PortAudio with brew install portaudio before installing PyAudio. You may need to reinstall PyAudio after installing PortAudio.
  • If the script hangs without printing anything, you may be running it from an IDE instead of Terminal. Close the IDE and run directly from the macOS Terminal app.
  • If you see "No speech detected", try speaking louder or moving closer to the microphone. Background noise can also interfere with detection.
  • If you denied the microphone permission popup, go to System Settings, then Privacy & Security, then Microphone, and enable access for Terminal.

My voice.py script isn't capturing audio. Help me debug the issue.

Your microphone is capturing audio and saving it as a WAV file. Next up, you'll send that audio to OpenAI's transcription API to turn your spoken words into text.

Transcribe Speech with OpenAI

Your assistant can now listen through the microphone and save audio to a file. The next piece of the puzzle is turning that audio into text so your AI can understand what was said.

You will send the saved WAV file to OpenAI's gpt-4o-mini-transcribe model, which converts speech into text with high accuracy. This is the speech-to-text (STT) stage of your voice assistant pipeline.

In this step, get ready to:
  • Set your OpenAI API key as an environment variable.
  • Create a transcription module that sends audio to OpenAI and returns text.
  • Test the full speech-to-text pipeline from microphone to transcript.
Set your OpenAI API key

The transcription module needs to authenticate with OpenAI's API. You do this by storing your API key in an environment variable that your Python code reads at runtime.

  • Open your browser and go to platform.openai.com/api-keys.
  • Sign in or create an account if you do not have one yet.
  • Click Create new secret key.
  • Copy the key. You will only see it once, so keep it handy.

What is an API key?

An API key is a unique string that identifies your account when making requests to a service. OpenAI uses it to track your usage and bill your account for the tokens your assistant consumes.

  • Back in your Terminal (with your virtual environment still active), set the API key as an environment variable by running this command:
export OPENAI_API_KEY="your-api-key-here"
  • Replace your-api-key-here with the key you just copied from OpenAI.

Pro tip: keep your key alive

The export command only lasts for your current Terminal session. If you close and reopen Terminal, you will need to run it again before running your assistant.

Create the transcription module

This module reads the WAV file that voice.py saved and sends it to OpenAI for transcription. The API returns the spoken words as plain text.

  • Create a new file called transcribe.py inside your smart-home-assistant folder.
  • Add the imports, client setup, and transcription function by copying this code into the file:
import os
from openai import OpenAI

# Path to the WAV file saved by voice.py
AUDIO_FILE = "command.wav"

# Create an OpenAI client using your API key from the environment
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))


def transcribe(audio_path=AUDIO_FILE):
    # Open the audio file and send it to OpenAI for transcription
    with open(audio_path, "rb") as audio_file:
        transcription = client.audio.transcriptions.create(
            model="gpt-4o-mini-transcribe",
            file=audio_file,
        )
    return transcription.text

What does this code do?

  • AUDIO_FILE points to command.wav, the same file that voice.py saves your microphone audio to.
  • client = OpenAI(...) creates an authenticated connection to OpenAI using the API key you just exported.
  • client.audio.transcriptions.create() sends the audio file to the gpt-4o-mini-transcribe model, which returns an object with a .text attribute containing the transcribed words.

You will not see any output yet. The next chunk adds a test block so you can run the file.

  • Now add the test block at the bottom of transcribe.py, below the transcribe() function:

if __name__ == "__main__":
    # Check if an audio file exists before trying to transcribe
    if os.path.exists(AUDIO_FILE):
        text = transcribe()
        print(f"You said: {text}")
    else:
        print(f"No audio file found. Run voice.py first.")

What does this test block do?

  • if __name__ == "__main__": makes this block run only when you execute transcribe.py directly. When other files import the transcribe() function later, this block is skipped.
  • os.path.exists(AUDIO_FILE) checks whether command.wav exists before trying to transcribe it. If you have not recorded audio yet, it reminds you to run voice.py first.
  • Save transcribe.py.

Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

I'd like to double check the full code

import os
from openai import OpenAI

AUDIO_FILE = "command.wav"

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))


def transcribe(audio_path=AUDIO_FILE):
    with open(audio_path, "rb") as audio_file:
        transcription = client.audio.transcriptions.create(
            model="gpt-4o-mini-transcribe",
            file=audio_file,
        )
    return transcription.text


if __name__ == "__main__":
    if os.path.exists(AUDIO_FILE):
        text = transcribe()
        print(f"You said: {text}")
    else:
        print(f"No audio file found. Run voice.py first.")
Test your speech-to-text pipeline

You now have two modules that work together. First, voice.py captures audio from your microphone. Then, transcribe.py sends that audio to OpenAI and returns the text.

  • First, record a voice command by running:
python3 voice.py
  • Wait for the "Listening..." prompt, then speak a clear command like "turn on the living room light."
  • You should see Audio saved to command.wav in the terminal output.
  • Now transcribe the saved audio by running:
python3 transcribe.py
  • You should see your spoken words printed back to you, like You said: Turn on the living room light.

This two-step flow is your working speech-to-text pipeline. You speak, the microphone captures your voice, and OpenAI converts it to text.

Seeing an error about the API key?

  • If you see AuthenticationError, your API key is missing or incorrect. Run echo $OPENAI_API_KEY in Terminal to check if it is set.
  • If it prints nothing, run the export command again. You need to do this in the same Terminal session where you run your scripts.
  • If you see No audio file found, run python3 voice.py first to capture audio before running transcribe.py.
  • Still stuck? Help me debug my OpenAI transcription error.

Why gpt-4o-mini-transcribe instead of Whisper?

OpenAI offers several speech-to-text models. The gpt-4o-mini-transcribe model is newer than the original whisper-1 model and delivers better accuracy with lower word error rates. It costs the same to use in this project and requires no extra setup.

Your assistant can now hear and understand spoken words. Next, you will teach it to figure out what those words actually mean by parsing voice commands into smart home actions.

Parse Commands with GPT-4o-mini

Your speech-to-text pipeline is working. You can capture audio from your microphone and turn it into text using OpenAI's transcription model.

The transcribed text still needs to become structured data so your smart home knows which device to control and what action to take. In this step, you will build an intent parser that uses GPT-4o-mini to extract device, action, and value from any spoken command.

In this step, get ready to:
  • Build an intent parser that turns transcribed text into structured commands.
  • Add validation for commands the parser cannot understand.
  • Test the parser with sample voice commands.
Create intent.py and build the intent parser

Your intent parser takes a plain English sentence like "set the thermostat to 72" and extracts three pieces of information: the device name, the action, and an optional value. You will use GPT-4o-mini to do this extraction by giving it a carefully crafted system prompt that tells it exactly what JSON structure to return.

  • Create a new file called intent.py in your smart-home-assistant folder.
  • Add the imports and device list by copying this code:
import os
import json
from openai import OpenAI

# Reuse the same OpenAI client setup from transcribe.py
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# Device names must match exactly with devices.py
VALID_DEVICES = [
    "living room light",
    "bedroom light",
    "kitchen light",
    "thermostat",
    "coffee maker",
    "fan",
]

What does this code do?

  • json and os are Python standard libraries for parsing JSON strings and reading environment variables.
  • VALID_DEVICES lists every device name from your devices.py registry. The intent parser uses this list to tell GPT-4o-mini which device names are valid.
  • Save intent.py. You will not see any output yet. The next chunk adds the function that uses this setup.
  • Add the parse_intent() function below your VALID_DEVICES list in intent.py:

def parse_intent(text):
    # Build a prompt that tells GPT-4o-mini to extract structured intent
    system_prompt = f"""You are a smart home intent parser. Given a user's voice command, 
extract the intent as a JSON object with these fields:
- "device": one of {VALID_DEVICES}
- "action": one of "on", "off", "set"
- "value": a number if the action is "set", otherwise null

If the command does not match any known device or action, return:
{{"device": null, "action": "unknown", "value": null}}

Respond with ONLY the JSON object, no other text."""

    # Send the transcribed text to GPT-4o-mini for intent parsing
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": text},
        ],
    )

    result_text = response.choices[0].message.content.strip()

    # Parse the JSON response, with a fallback for malformed output
    try:
        intent = json.loads(result_text)
    except json.JSONDecodeError:
        intent = {"device": None, "action": "unknown", "value": None}

    return intent

What does this function do?

  • The system prompt is an f-string that injects your VALID_DEVICES list directly into the instructions for GPT-4o-mini. This tells the model exactly which device names to look for.
  • temperature=0 makes the model's output deterministic. The same input always produces the same parsed intent.
  • The transcribed text is sent as a user message alongside the system prompt using client.chat.completions.create().
  • If the model returns invalid JSON, the try/except block catches the error and returns a fallback dictionary with "action": "unknown".
  • Save intent.py. You cannot test the function on its own yet. The test block in the next substep lets you verify it works.
Handle unrecognized commands

Not every voice command maps to a smart home action. Someone might say "what's the weather like?" or mumble something unintelligible.

Your parser handles these cases with two layers of protection, both already built into the function you just wrote.

How does the validation work?

  • Layer 1: the system prompt. It tells GPT-4o-mini to return {"device": null, "action": "unknown", "value": null} when a command does not match any device or action. This handles off-topic commands like asking about the weather.
  • Layer 2: the try/except block. If GPT-4o-mini returns something that is not valid JSON at all, json.loads() raises a JSONDecodeError. The except block catches it and returns the same fallback dictionary.
Test the parser with sample commands
  • Add a test block at the bottom of intent.py to try the parser with different commands:

if __name__ == "__main__":
    # Test with a variety of commands, including one that should fail
    test_commands = [
        "Turn on the living room light",
        "Set the thermostat to 72 degrees",
        "Turn off the coffee maker",
        "What's the weather like?",
    ]

    for cmd in test_commands:
        print(f"Command: {cmd}")
        result = parse_intent(cmd)
        print(f"Intent: {result}")
        print()

What does this test do?

  • The test runs four commands through parse_intent(): three valid smart home commands and one off-topic question.
  • The last test, "What's the weather like?", should return "action": "unknown" since it does not match any device.
  • Save intent.py.
  • Make sure your OpenAI API key is still set in your terminal session. If you opened a new terminal window since Step 4, set it again by running:
export OPENAI_API_KEY="your-api-key-here"
  • Run the test by running this command:
python3 intent.py

You should see four command-intent pairs printed. The first three show recognized devices and actions, and the fourth ("What's the weather like?") shows "action": "unknown".

Not seeing the expected output?

  • Make sure your virtual environment is activated. Your terminal prompt should show (venv) at the beginning.
  • If you see an authentication error, your OPENAI_API_KEY environment variable may not be set. Check it by running echo $OPENAI_API_KEY in your terminal.
  • If you see a ModuleNotFoundError for openai, make sure you installed the packages inside your virtual environment, not globally.
  • My intent parser is returning unexpected results. Help me debug.

Awesome, I've got everything!

Double check you have saved intent.py.

I'd like to double check the full code

import os
import json
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

VALID_DEVICES = [
    "living room light",
    "bedroom light",
    "kitchen light",
    "thermostat",
    "coffee maker",
    "fan",
]


def parse_intent(text):
    system_prompt = f"""You are a smart home intent parser. Given a user's voice command, 
extract the intent as a JSON object with these fields:
- "device": one of {VALID_DEVICES}
- "action": one of "on", "off", "set"
- "value": a number if the action is "set", otherwise null

If the command does not match any known device or action, return:
{{"device": null, "action": "unknown", "value": null}}

Respond with ONLY the JSON object, no other text."""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": text},
        ],
    )

    result_text = response.choices[0].message.content.strip()

    try:
        intent = json.loads(result_text)
    except json.JSONDecodeError:
        intent = {"device": None, "action": "unknown", "value": None}

    return intent


if __name__ == "__main__":
    test_commands = [
        "Turn on the living room light",
        "Set the thermostat to 72 degrees",
        "Turn off the coffee maker",
        "What's the weather like?",
    ]

    for cmd in test_commands:
        print(f"Command: {cmd}")
        result = parse_intent(cmd)
        print(f"Intent: {result}")
        print()

Your intent parser is ready to understand natural language commands. Next up, you will wire everything together into a voice assistant loop that listens, transcribes, parses, and speaks.

Wire Everything Together into a Voice Assistant Loop

You've built every piece of the voice assistant pipeline individually. A microphone listener, a speech-to-text transcriber, an intent parser, and a smart home device registry are all ready to go.

In this step, you'll create assistant.py, the main file that connects every module into a continuous loop. Speak a command, and your assistant will understand it, act on it, and speak a confirmation back to you through your Mac's speakers.

In this step, get ready to:
  • Create the main assistant file with a text-to-speech helper function.
  • Build a continuous voice command loop that listens, transcribes, parses, and executes.
  • Test the full assistant with real voice commands through your microphone.
Create the assistant file and speak helper

Your assistant needs a way to talk back to you. On macOS, the built-in say command converts any text into spoken audio through your speakers. Python's subprocess module lets you run system commands like this directly from your code.

  • Create a new file called assistant.py inside your smart-home-assistant folder.
  • Add the imports and speak helper function by copying this code into the file:
import subprocess
from devices import SmartHome
from voice import listen
from transcribe import transcribe
from intent import parse_intent


def speak(text):
    # Print the response to the terminal and speak it aloud
    print(f"Assistant: {text}")
    subprocess.run(["say", text])

What does this code do?

  • The imports pull in every module you built in the previous steps: SmartHome for the device registry, listen for mic capture, transcribe for speech-to-text, and parse_intent for understanding commands.
  • speak() prints the response text to your terminal AND uses subprocess.run(["say", text]) to call the macOS say command, which reads the text aloud through your speakers.
  • You won't see a change yet. The next chunk builds the main loop that uses this helper.
Build the main loop with a quit command

The main loop is the heart of your assistant. Each cycle follows the same pipeline: listen for audio, transcribe it to text, parse the intent, execute the command on your smart home, and speak the result.

You also need a way to stop the loop. A quit command lets you say "stop" or "goodbye" to exit gracefully and see a final summary of all your device states.

  • In assistant.py, add the main() function below your speak() function by copying this code:
def main():
    # Set up the smart home device registry
    home = SmartHome()
    speak("Smart home assistant is ready. What would you like to do?")

    while True:
        # Step 1: Listen for a voice command
        success = listen()
        if not success:
            speak("I didn't catch that. Please try again.")
            continue

        # Step 2: Convert speech to text
        text = transcribe()
        print(f"You said: {text}")

        # Check for quit commands
        if any(word in text.lower() for word in ["stop", "goodbye", "quit", "exit"]):
            speak("Goodbye! Here is your final home status.")
            print(home.get_status())
            break

        # Step 3: Extract device, action, and value from the text
        intent = parse_intent(text)
        print(f"Parsed intent: {intent}")

        if intent["action"] == "unknown":
            speak("Sorry, I didn't understand that command. Try something like 'turn on the living room light'.")
            continue

        # Step 4: Execute the command on the smart home
        result = home.execute_command(
            intent["device"],
            intent["action"],
            intent.get("value"),
        )

        # Step 5: Speak the result back to the user
        speak(result)


if __name__ == "__main__":
    main()

What does this code do?

  • SmartHome() creates a fresh device registry with all your simulated devices.
  • The while True loop runs continuously. Each cycle calls listen(), transcribe(), parse_intent(), and execute_command() in sequence to go from voice to action.
  • The quit check uses Python's any() to scan your transcribed text for words like "stop" or "goodbye". If found, it speaks a farewell, prints the full device status summary, and breaks out of the loop.
  • If the intent parser returns "unknown", the assistant tells the user it didn't understand and waits for a new command with continue.
  • if __name__ == "__main__" ensures the loop only starts when you run the file directly with python3 assistant.py.
  • Save assistant.py.

Awesome, I've got everything!

Double check you've saved your file before moving on to testing.

I'd like to double check the full code

import subprocess
from devices import SmartHome
from voice import listen
from transcribe import transcribe
from intent import parse_intent


def speak(text):
    print(f"Assistant: {text}")
    subprocess.run(["say", text])


def main():
    home = SmartHome()
    speak("Smart home assistant is ready. What would you like to do?")

    while True:
        success = listen()
        if not success:
            speak("I didn't catch that. Please try again.")
            continue

        text = transcribe()
        print(f"You said: {text}")

        # Check for quit commands
        if any(word in text.lower() for word in ["stop", "goodbye", "quit", "exit"]):
            speak("Goodbye! Here is your final home status.")
            print(home.get_status())
            break

        intent = parse_intent(text)
        print(f"Parsed intent: {intent}")

        if intent["action"] == "unknown":
            speak("Sorry, I didn't understand that command. Try something like 'turn on the living room light'.")
            continue

        result = home.execute_command(
            intent["device"],
            intent["action"],
            intent.get("value"),
        )

        speak(result)


if __name__ == "__main__":
    main()
Test your voice assistant

Your assistant is ready to go. Time to run it and have a conversation with your smart home.

Why run from Terminal?

On macOS, microphone permissions work most reliably when you run Python scripts directly from Terminal. Running from inside an IDE like VS Code can cause permission issues where the microphone isn't detected.

  • Make sure your virtual environment is still active. You should see (venv) at the start of your Terminal prompt.
  • If you don't see (venv), activate it again by running:
source venv/bin/activate
  • Make sure your OpenAI API key is still set in your terminal session. If you opened a new Terminal window, set it again by running:
export OPENAI_API_KEY="your-api-key-here"
  • Replace your-api-key-here with your actual OpenAI API key.
  • Start the assistant by running this command:
python3 assistant.py

You should hear your Mac say "Smart home assistant is ready. What would you like to do?" through your speakers. The terminal prints the same text.

  • Wait for the "Listening... speak your command now." message to appear in your terminal.
  • Say "turn on the living room light" clearly into your Mac's microphone.

You should see your transcribed text, the parsed intent as a JSON object, and hear the assistant say "Turned on the living room light."

  • Try a few more commands to exercise different parts of your smart home:
  • "Set the thermostat to 72 degrees."
  • "Turn on the coffee maker."
  • "Turn off the living room light."

Each command should produce a spoken confirmation and a printed response in your terminal.

  • When you're done testing, say "goodbye" to exit the loop.

The assistant says "Goodbye!" and prints a full summary of all device states to your terminal. You should see which devices are on and off, along with their current temperature or brightness values.

Assistant not responding to your voice?

  • Make sure you are running the script from Terminal, not from an IDE. On macOS, IDEs may not pass through microphone permissions correctly.
  • Check that Terminal has microphone access. Open System Settings, go to Privacy & Security, select Microphone, and make sure Terminal is toggled on.
  • If the transcription is inaccurate, try speaking more slowly and clearly. Background noise can affect the results.
  • If you see an API key error, make sure you set the environment variable in the same terminal window where you ran the script.
  • My voice assistant isn't hearing me or is returning errors. Help me debug.

Pro tip: try unexpected commands

Try saying something that isn't a smart home command, like "what's the weather?" Your intent parser should return "unknown" and the assistant should ask you to try a different command. This is your error handling in action.

Secret mission

Add a Status Report Command

Your assistant can control devices, but it can't tell you what's happening across your home. What if you could ask "what's the status of my home?" and hear a full spoken summary of every device? In this mission, you'll add a status report feature and expand your smart home with new devices.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs almost entirely locally. The only cost comes from OpenAI API calls, and those only happen when you actively run the assistant.

Resources you used:

  • smart-home-assistant project folder with all Python files.
  • Python virtual environment (venv) with installed packages.
  • PortAudio installed via Homebrew.
  • OpenAI API key set as an environment variable.
  • command.wav temporary audio file.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the project soon.

  • Your project folder and virtual environment stay where they are.
  • Your OpenAI API key remains set in your Terminal session. It resets when you close that Terminal window.
  • No ongoing costs are incurred unless you actively run the assistant and make API calls.

Pause - I'll come back to this later

Deactivate your virtual environment and clean up temporary files. Your project files and installed packages remain intact for when you come back.

  • Deactivate your virtual environment by running:
deactivate
  • Delete the temporary audio file if it exists:
rm -f command.wav
  • When you come back, reactivate the virtual environment with:
cd smart-home-assistant
source venv/bin/activate
export OPENAI_API_KEY="your-api-key-here"

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your project files, virtual environment, and installed dependencies.

  • Deactivate your virtual environment first:
deactivate
  • Delete the entire project folder:
rm -rf smart-home-assistant
  • Optionally uninstall PortAudio if you no longer need it for other projects:
brew uninstall portaudio
  • If you want to revoke your OpenAI API key entirely, visit your OpenAI dashboard and delete the key you created for this project.

Mission accomplished

That's a Wrap!

Nice work! You just built a voice-controlled AI smart home assistant from scratch in Python. You went from an empty project folder to a fully working voice pipeline that listens, understands, acts, and speaks back to you.

Here's what you learned:

  • Built a speech-to-text pipeline that captures microphone audio and transcribes it using OpenAI's gpt-4o-mini-transcribe model.
  • Created an LLM-powered intent parser with gpt-4o-mini that translates natural language into structured smart home commands.
  • Designed a smart home device registry with state tracking, modeled after Home Assistant's entity system and ready to connect to real devices.
  • Added a status report feature that gives you a spoken summary of every device in your smart home on demand.

Ready to quiz yourself?