Build a Cloud-Brained JARVIS

Build a voice task assistant with Ollama Cloud and local privacy controls.

Introduction

30 Second Summary

Talking to your computer should be as simple as saying its name. The challenge is keeping private recordings plus real actions under your control while still getting useful answers.

In this project, you will build a voice-controlled JARVIS task butler that keeps audio processing plus task actions on your computer. Ollama Cloud supplies the language-model responses without requiring a large model download.

What You'll Build

You will say “Hey Jarvis” to wake your assistant before asking it to answer questions or manage your local task list through spoken confirmations.

By the end of this project, you'll have:

  • Wake JARVIS by saying “Hey Jarvis” before seeing the terminal confirm that it heard you.
  • Ask general questions before hearing answers from gemma4:31b without storing the language model on your computer.
  • Manage local tasks by adding, listing, or completing items in tasks.json with confirmation required before each change.
  • Secret Mission: Record every approved task change in a local audit.jsonl action trail.

Are there any prerequisites?

You need a computer with a microphone plus speakers. You also need permission to install software.

An Ollama account with an API key is required. Internet access is needed whenever JARVIS contacts Ollama Cloud.

Before We Start

Before any setup begins, lock in the privacy boundary that guides your JARVIS assistant. This keeps audio processing and task actions on your computer while limiting what can reach the cloud.

Prepare Local Audio and Ollama Cloud Access

Your privacy boundary keeps microphone audio plus assistant actions on your computer. This setup prepares those local components without storing a cloud language model on your disk.

You will prepare Visual Studio Code with Python 3.11.9. You will also build an isolated virtual environment before securely loading an Ollama Cloud credential.

In this step, get ready to:
  • Install Visual Studio Code with Python 3.11.9.
  • Create the project folder with its pinned local voice packages.
  • Load an Ollama Cloud API key into the activated terminal.

Why Ollama Cloud?

Ollama Cloud provides managed model inference through an API. Direct cloud access requires no Ollama app installation.

The language model also stays off your computer. Your local storage remains available for the smaller voice components.

Install the editor and Python runtime

Visual Studio Code gives you one place for project files plus terminal commands. Start by matching the tab that describes your current editor setup.

✔️ Visual Studio Code opens

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Use your application launcher if you use Linux.
  • Type Visual Studio Code into the search field.
  • Press Enter to open the editor.

You should see the Visual Studio Code welcome screen. Your editor is ready for the JARVIS project.

ⓧ Visual Studio Code is not installed

Install the editor using the instructions for your operating system. Visual Studio Code supports macOS plus Windows plus Linux.

macOS

  • Visit the official Visual Studio Code macOS setup guide.
  • Download the disk image.
  • Move Visual Studio Code.app into your Applications folder.
  • Open Visual Studio Code through system search.

Windows

Linux

  • Visit the official Visual Studio Code Linux setup guide.
  • Choose the documented package for your Linux distribution.
  • Complete the package installation.
  • Open Visual Studio Code through your application launcher.

Editor installation not working?

Confirm that you selected the guide for your operating system. Use the official setup overview if your distribution needs a different installation package.

Ask for help with your exact operating system here: Help me install Visual Studio Code.

The project pins Python to 3.11.9 so every package uses the same runtime. The version check separates a ready installation from an older or missing one.

  • Create an integrated terminal from the Terminal menu in Visual Studio Code.
  • Check the Python runtime selected by the terminal by running this command:
python --version

What does this command check?

The --version option prints the version selected by the python command. That selected interpreter creates the project environment later in this step.

✔️ I see the required Python version

Your terminal should report Python 3.11.9. That runtime is ready to create the project environment.

ⓧ I see an older Python version

The selected interpreter is older than Python 3.11.9. Install the pinned release before creating the environment.

  • Visit the official Python 3.11.9 release page.
  • Choose the installer for your operating system.
  • Complete the Python installation.
  • Close the current integrated terminal.
  • Create a fresh integrated terminal from the Terminal menu.
  • Check the selected version again by running:
python --version

What should the new terminal show?

A fresh terminal can detect the newly installed interpreter. The output should identify Python 3.11.9.

ⓧ The Python command is unavailable

The terminal cannot find Python yet. Install Python 3.11.9 before continuing.

  • Visit the official Python 3.11.9 release page.
  • Choose the installer for your operating system.
  • Complete the Python installation.
  • Close the current integrated terminal.
  • Create a fresh integrated terminal from the Terminal menu.
  • Confirm that Python is available by running:
python --version

What confirms the installation?

The command should print Python 3.11.9. This confirms that Visual Studio Code can reach the pinned runtime.

Still seeing the wrong Python version?

Close every existing integrated terminal after installing Python. A fresh terminal reloads the executable paths available to Visual Studio Code.

If the version still differs, ask for operating-system-specific help here: Help me select Python 3.11.9.

Build the isolated local voice environment

The cloud-jarvis folder keeps the assistant files together. Its local environment prevents the pinned voice packages from changing other Python projects on your computer.

  • Switch back to the integrated terminal from the Python check.
  • Create the project on your Desktop by running these commands:
cd ~/Desktop
mkdir cloud-jarvis
cd cloud-jarvis

What do these commands do?

  • The first command moves the terminal to your Desktop.
  • The second command creates the cloud-jarvis folder.
  • The final command moves the terminal into that folder.
  • Click File in the Visual Studio Code menu bar.
  • Click Open Folder.
  • Select the cloud-jarvis folder on your Desktop.
  • Click Open.

You should see cloud-jarvis at the top of the Explorer sidebar. The editor is now anchored to the correct folder.

Cannot find the project folder?

Look for cloud-jarvis directly on your Desktop. Repeat the folder commands from a terminal if it is missing.

Ask for help locating the folder here: Help me find my project folder.

The dependency file pins every package to a known version. This makes your local audio stack reproducible across fresh installations.

  • Click the New File control in the Explorer sidebar.
  • Enter requirements.txt as the file name.
  • Populate requirements.txt by pasting this exact package list:
faster-whisper==1.2.1
numpy==2.3.5
openai==3.42.1
pocketsphinx==5.1.1
pyttsx3==2.99
sounddevice==0.5.6

What does this package list provide?

  • faster-whisper performs local speech transcription.
  • NumPy provides the arrays used for recorded audio.
  • OpenAI Python provides the client that later connects to Ollama Cloud.
  • PocketSphinx detects the local wake phrase.
  • pyttsx3 uses your computer's offline speech engine.
  • sounddevice records microphone audio.
  • Save requirements.txt.
  • Confirm that requirements.txt appears under cloud-jarvis in the Explorer sidebar.

Package file looks different?

Confirm that the file is named requirements.txt with no extra extension. Compare all six lines with the reference tab below.

Ask for a careful comparison here: Help me check my requirements file.

✔️ Awesome, I've got everything!

Your pinned dependency file is saved. Keep it open while you create the isolated environment.

ⓧ I'd like to double check the full code

Compare your full requirements.txt file with this exact reference.

faster-whisper==1.2.1
numpy==2.3.5
openai==3.42.1
pocketsphinx==5.1.1
pyttsx3==2.99
sounddevice==0.5.6

What should match?

Every package name plus version should match the reference exactly. These six pins form the complete dependency file for this step.

Linux may need additional system libraries for microphone input plus offline speech output. Choose your operating system before creating the Python environment.

macOS

The Python packages provide the local audio interfaces used in this project. The final checks in this step confirm that they load on macOS.

Windows

The Python packages provide the local audio interfaces used in this project. The final checks in this step confirm that they load on Windows.

Linux

PocketSphinx uses PortAudio for live microphone input. pyttsx3 uses eSpeak for offline voice output on supported Linux systems.

  • Install the documented Debian-like Linux audio dependencies by running:
sudo apt-get install libportaudio2
sudo apt update && sudo apt install espeak-ng libespeak1

What do these commands install?

The first command installs the PortAudio library used for microphone input. The second command installs the eSpeak libraries used for offline voice output.

Your package manager should return to the prompt after both installations finish. This confirms that the system audio dependencies are available.

Linux packages unavailable?

These package names apply to Debian-like Linux distributions. Use your distribution documentation to locate equivalent PortAudio plus eSpeak packages.

Ask for distribution-specific help here: Help me install the Linux audio libraries.

A virtual environment stores a project-specific Python interpreter plus its packages inside .venv. The folder remains contained within cloud-jarvis.

  • Return to the integrated terminal inside cloud-jarvis.
  • Create the virtual environment by running:
python -m venv .venv

What does this command create?

The venv module creates an isolated Python environment named .venv. It uses the Python 3.11.9 interpreter selected in the terminal.

  • Confirm that .venv appears under cloud-jarvis in the Explorer sidebar.

Virtual environment missing?

Confirm that the terminal is inside the cloud-jarvis folder. Confirm that the earlier version check reports Python 3.11.9.

Ask for help with environment creation here: Help me create .venv.

Activation makes the terminal use the interpreter plus packages inside .venv. Choose the command for your operating system.

macOS

  • Activate .venv in the current terminal by running:
source .venv/bin/activate

What does activation change?

The command updates the current shell so python plus pip use the project environment. It affects only this terminal session.

You should see (.venv) at the start of the terminal prompt.

Windows

  • Activate .venv in the current PowerShell terminal by running:
.venv\Scripts\Activate.ps1

What does activation change?

The command updates the current PowerShell session so python plus pip use the project environment. It affects only this terminal session.

You should see (.venv) at the start of the terminal prompt.

Linux

  • Activate .venv in the current terminal by running:
source .venv/bin/activate

What does activation change?

The command updates the current shell so python plus pip use the project environment. It affects only this terminal session.

You should see (.venv) at the start of the terminal prompt.

Activation did not change the prompt?

Confirm that .venv exists inside cloud-jarvis. Use the command from the tab for your operating system.

Ask for help with your shell here: Help me activate .venv.

The first dependency installation can take several minutes while Python downloads plus prepares the pinned packages. A steady stream of terminal output means the installation is still working.

  • Install every pinned dependency into the activated environment by running:
python -m pip install -r requirements.txt

What does this command install?

The -r option tells pip to read the dependency list from requirements.txt. The activated terminal places every package inside .venv.

The terminal should return to its (.venv) prompt without an installation failure. Your pinned local voice stack is now installed.

Dependency installation failed?

Confirm that (.venv) appears in the terminal prompt. Check the Linux dependency tab if the failure mentions an audio library.

Ask for package-specific help here: Help me fix the dependency installation.

Load the cloud credential and verify the stack

An API key authorizes requests to Ollama Cloud. Loading it into the current terminal keeps the credential out of project files plus source control.

Credentials deserve extra care. The commands below keep the key within one terminal session without printing it during the final check.

  • Visit the Ollama API-key settings.
  • Create an Ollama account if the page prompts you to register.
  • Sign in to your Ollama account.
  • Create a new API key from the API-key settings.
  • Copy the generated key without placing it in any project file.

Keep the key out of your project

Never paste the real key into jarvis.py or any other project file. Keep it out of screenshots plus source control.

The key commands affect only the current terminal. Closing that terminal removes the loaded value from the session.

Choose your operating system to load the copied key into the same terminal where (.venv) is active. Replace the plain placeholder with your real key before running the command.

macOS

  • Load the key into the current terminal by running:
export OLLAMA_API_KEY="your_api_key"

What does this command change?

The command sets OLLAMA_API_KEY for the current shell session. Future Python processes launched from this terminal can read it.

Windows

  • Load the key into the current PowerShell terminal by running:
$env:OLLAMA_API_KEY = "your_api_key"

What does this command change?

The command sets OLLAMA_API_KEY for the current PowerShell session. Future Python processes launched from this terminal can read it.

Linux

  • Load the key into the current terminal by running:
export OLLAMA_API_KEY="your_api_key"

What does this command change?

The command sets OLLAMA_API_KEY for the current shell session. Future Python processes launched from this terminal can read it.

  • Predict whether the local stack check will report readiness before you run it:
python -c "import os, openai, faster_whisper, pocketsphinx, pyttsx3, sounddevice; print('Cloud key and local voice stack ready' if os.getenv('OLLAMA_API_KEY') else 'OLLAMA_API_KEY is missing')"

What does this check prove?

Python imports every installed library from the activated environment. It also confirms that OLLAMA_API_KEY exists without displaying the credential.

You should see Cloud key and local voice stack ready. That result proves the current activated terminal contains the pinned packages plus the cloud credential.

Stack check not ready?

Reload the key in the same terminal if the output reports a missing credential. Repeat the dependency installation if Python reports a missing package.

Ask for help separating those causes here: Help me fix the local stack check.

  • Predict whether your offline voice engine will speak before you run this final check:
python -c "import pyttsx3; pyttsx3.speak('JARVIS setup complete')"

What does the speech check prove?

Python imports pyttsx3 from .venv. Your operating system's offline voice engine then speaks the test phrase.

You should hear JARVIS setup complete through your speakers. Your cloud credential plus local voice stack are ready to support the assistant.

Cannot hear the setup phrase?

Confirm that your speaker volume is audible. Linux learners should also confirm that the documented eSpeak packages finished installing.

Ask for audio-output help here: Help me troubleshoot offline speech.

Your local voice environment is isolated plus ready. Next, you will give JARVIS its cloud brain with a small typed conversation.

Give JARVIS a Cloud Brain

Your local voice stack is ready. JARVIS now needs a model that fits the storage limit you started with.

A typed loop can send short prompts to Ollama Cloud through an OpenAI-compatible API. You will also test what a plain cloud response can change on your computer.

In this step, get ready to:
  • Create a typed loop that connects JARVIS to Ollama Cloud.
  • Send a general question to the cloud-hosted gemma4:31b model.
  • Compare a task response with the local files in cloud-jarvis.
Build the typed cloud client

The installed OpenAI Python client can target Ollama Cloud at https://ollama.com/v1. The API key stays in the current terminal through OLLAMA_API_KEY.

  • Use the file-creation control in the Visual Studio Code sidebar to create jarvis.py directly inside cloud-jarvis.

You should see jarvis.py beside requirements.txt in the project sidebar.

  • Populate jarvis.py by replacing its empty contents with this code:
import os

from openai import OpenAI

MODEL = "gemma4:31b"

api_key = os.getenv("OLLAMA_API_KEY")
if not api_key:
    raise RuntimeError("Set OLLAMA_API_KEY before running JARVIS.")

client = OpenAI(
    base_url="https://ollama.com/v1",
    api_key=api_key,
)


def main() -> None:
    while True:
        user_text = input("YOU: ").strip()
        if user_text.lower() in {"stop", "quit", "exit"}:
            break
        response = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": user_text}],
        )
        print(f"JARVIS: {response.choices[0].message.content}")


if __name__ == "__main__":
    main()

How does this code work?

  • The MODEL constant selects the cloud-hosted model for every request.
  • The os.getenv("OLLAMA_API_KEY") call reads the credential from the activated terminal.
  • The missing-key check stops the program before an unauthenticated request reaches the cloud.
  • The OpenAI client directs requests to Ollama Cloud's compatible endpoint.
  • The main() loop collects one typed message at each YOU: prompt.
  • The Chat Completions call sends each message to gemma4:31b.
  • The final print() call displays the returned cloud response after JARVIS:.
  • Save jarvis.py.

✔️ Awesome, I've got everything!

Your saved jarvis.py now contains the complete typed cloud loop for this step.

ⓧ I'd like to double check the full code

Compare your saved file with this complete reference. The API key itself must remain outside the file.

import os

from openai import OpenAI

MODEL = "gemma4:31b"

api_key = os.getenv("OLLAMA_API_KEY")
if not api_key:
    raise RuntimeError("Set OLLAMA_API_KEY before running JARVIS.")

client = OpenAI(
    base_url="https://ollama.com/v1",
    api_key=api_key,
)


def main() -> None:
    while True:
        user_text = input("YOU: ").strip()
        if user_text.lower() in {"stop", "quit", "exit"}:
            break
        response = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": user_text}],
        )
        print(f"JARVIS: {response.choices[0].message.content}")


if __name__ == "__main__":
    main()

What should match?

  • The first section imports os plus OpenAI.
  • The client section reads OLLAMA_API_KEY from the terminal environment.
  • The final section runs the typed conversation loop.

Before you run this, do you expect JARVIS to answer without downloading an Ollama language model?

  • Start the typed JARVIS loop from the activated terminal by running this command:
python jarvis.py

What does this command do?

Python loads jarvis.py from the current cloud-jarvis folder. The main() loop then displays a YOU: prompt for your first message.

Cloud requests use credits

Each submitted message consumes usage through Ollama Cloud. This smoke test uses two short prompts to keep that usage limited.

  • Enter What is seven times eight? at the YOU: prompt.

You should see a JARVIS: response that gives 56 as the answer.

You have cleared the storage hurdle. The cloud model answered without adding an Ollama language model to your computer.

Cloud answer missing?

  • Reload the key into the current activated terminal if you see Set OLLAMA_API_KEY before running JARVIS..
  • Confirm your computer has an internet connection if the request does not return.
  • Help me troubleshoot my Ollama Cloud request without exposing my API key.
Test the storage boundary

A task request gives you a different kind of test. You can compare the model's response with the folder's actual contents.

Before you send it, do you think a text-only cloud response can create local task data?

  • Enter Add buy milk to my tasks at the next YOU: prompt.

You will see a text response about the request. The loop then returns to the YOU: prompt.

  • Enter exit at the next prompt.

The script stops. Your activated terminal prompt returns.

  • Check the Visual Studio Code project sidebar for tasks.json.

You will still see jarvis.py plus requirements.txt. The tasks.json file is absent.

That is the intended shortfall. JARVIS produced words without changing local task data.

Why was no task file created?

The cloud model received a user message. Its response returned as text through the API.

Local task data requires explicit functions that can read or write a file. Those functions can also enforce permission checks before each change.

  • Return to your Ollama account from earlier.
  • Check how much usage the two short prompts added.

File state looks different?

  • Confirm that tasks.json was not already present before this step.
  • Check that your current jarvis.py matches the full-code reference above.
  • Help me verify why my typed JARVIS changed local files before task tools were added.

Your storage-light cloud brain can now answer typed questions. Next, you will give it narrow task tools with a confirmation gate.

Turn Cloud Decisions Into Local Task Actions

Your typed JARVIS can answer through Ollama Cloud. However, its task responses are still only words because the model cannot change local data.

Tool calling lets the model request narrowly defined Python functions. Your code stays in control by checking each request before it reads or changes the local JSON task file.

In this step, get ready to:
  • Store task information in a local tasks.json file.
  • Give the cloud model three defined task tools.
  • Require typed approval before JARVIS changes local task data.
Build the local task store

The task store uses tasks.json as a readable local record. The file appears when JARVIS executes its first approved task addition.

  • In jarvis.py, select the existing header from import os through the closing parenthesis of client = OpenAI(.
  • Replace the selected header by pasting this code:
import json
import os
from pathlib import Path

from openai import OpenAI

MODEL = "gemma4:31b"
TASKS_FILE = Path("tasks.json")

api_key = os.getenv("OLLAMA_API_KEY")
if not api_key:
    raise RuntimeError("Set OLLAMA_API_KEY before running JARVIS.")

client = OpenAI(
    base_url="https://ollama.com/v1",
    api_key=api_key,
)

What does this code prepare?

  • The json module converts the task list between Python objects and file contents.
  • The Path class gives the program a clear reference to the local tasks.json file.
  • The existing client configuration keeps every model request pointed at https://ollama.com/v1.
  • Place the read-only task functions directly below the client = OpenAI(...) block by pasting this code:
def load_tasks() -> list[dict]:
    if not TASKS_FILE.exists():
        return []
    return json.loads(TASKS_FILE.read_text(encoding="utf-8"))


def save_tasks(tasks: list[dict]) -> None:
    TASKS_FILE.write_text(json.dumps(tasks, indent=2), encoding="utf-8")


def list_tasks() -> str:
    tasks = load_tasks()
    if not tasks:
        return "Your task list is empty."
    return "\n".join(
        f"[{'x' if task['done'] else ' '}] {number}. {task['title']}"
        for number, task in enumerate(tasks, start=1)
    )

How does local persistence work?

  • The load_tasks() function returns an empty list before tasks.json exists. Once the file exists, it loads the saved task objects.
  • The save_tasks() function writes formatted task data with UTF-8 encoding.
  • The list_tasks() function turns each task into a numbered line. An empty marker means the task is still open.
  • Save jarvis.py.
  • Confirm the new storage layer leaves JARVIS runnable by running this command:
python jarvis.py

You should see the YOU: prompt. This proves Python loaded the new imports and task functions without a syntax error.

  • Enter stop to return to the activated terminal.

Script not reaching the prompt?

  • Check that every function begins at the far-left edge of jarvis.py.
  • Check the closing parentheses in list_tasks() against the code above.
  • Ask for help with the storage functions.
  • Place the task-changing functions directly below list_tasks() by pasting this code:
def add_task(task: str) -> str:
    tasks = load_tasks()
    tasks.append({"title": task.strip(), "done": False})
    save_tasks(tasks)
    return f"Added task: {task.strip()}"


def complete_task(task_number: int) -> str:
    tasks = load_tasks()
    index = int(task_number) - 1
    if index < 0 or index >= len(tasks):
        return f"Task {task_number} does not exist."
    tasks[index]["done"] = True
    save_tasks(tasks)
    return f"Completed task {task_number}: {tasks[index]['title']}"

What changes task data?

  • The add_task() function stores a cleaned title with a False completion state.
  • The complete_task() function converts the displayed task number into a zero-based list position.
  • Both functions call save_tasks() after a valid change. This makes approved updates persist between runs.
Define the model's task tools

A tool schema tells the model which function names exist. It also defines the arguments each function accepts.

The allowlist creates a second boundary inside your program. A requested name only runs when it maps to one of your three task functions.

  • Place the tool registries directly below complete_task() by pasting this code:
AVAILABLE_TOOLS = {
    "list_tasks": list_tasks,
    "add_task": add_task,
    "complete_task": complete_task,
}
MUTATING_TOOLS = {"add_task", "complete_task"}
TOOLS = [
    {"type": "function", "function": {"name": "list_tasks", "description": "List all local tasks and their completion state.", "parameters": {"type": "object", "properties": {}, "required": []}}},
    {"type": "function", "function": {"name": "add_task", "description": "Add one task to the local task list.", "parameters": {"type": "object", "properties": {"task": {"type": "string", "description": "The task text to save."}}, "required": ["task"]}}},
    {"type": "function", "function": {"name": "complete_task", "description": "Mark one local task complete by its displayed number.", "parameters": {"type": "object", "properties": {"task_number": {"type": "integer", "description": "The one-based task number shown by list_tasks."}}, "required": ["task_number"]}}},
]

How do the tool boundaries work?

  • The TOOLS list describes the three functions to the cloud model. Each schema restricts the expected argument names and value types.
  • The AVAILABLE_TOOLS mapping connects approved names to local Python functions.
  • The MUTATING_TOOLS set identifies functions that change local data. The read-only list_tasks tool stays outside that set.
  • Save jarvis.py.
  • Check that Python accepts the function schemas by running this command:
python jarvis.py

You should see the YOU: prompt again. The tool definitions now load as valid Python data.

  • Enter stop to return to the activated terminal.

Tool schemas not loading?

  • Check that AVAILABLE_TOOLS appears after all five task functions.
  • Compare every brace in TOOLS with the reference above.
  • Ask for help checking the tool schemas.
Dispatch approved tool calls

The model can now request a task function. Your program still decides whether the function name is allowed and whether a local change needs approval.

After execution, JARVIS sends the result back in a linked tool message. The model uses that evidence to write its final reply.

  • Place ask_jarvis() directly above the existing main() function by pasting this code:
def ask_jarvis(user_text: str) -> str:
    messages = [
        {"role": "system", "content": "You are JARVIS, a concise assistant. Use the supplied task tools whenever the request concerns tasks. Never claim a task changed unless a tool result confirms it."},
        {"role": "user", "content": user_text},
    ]
    response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    assistant_message = response.choices[0].message
    messages.append(assistant_message.to_dict())
    for call in assistant_message.tool_calls or []:
        name = call.function.name
        arguments = json.loads(call.function.arguments)
        function_to_call = AVAILABLE_TOOLS.get(name)
        if function_to_call is None:
            result = f"Blocked unknown tool: {name}"
        elif name in MUTATING_TOOLS:
            approved = input(f"Approve {name.replace('_', ' ')}? Type yes: ").strip().lower() == "yes"
            result = function_to_call(**arguments) if approved else "Cancelled by user."
        else:
            result = function_to_call(**arguments)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": str(result)})
    if not assistant_message.tool_calls:
        return assistant_message.content or "I could not produce a response."
    final_response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    return final_response.choices[0].message.content or "Task request processed."

How does JARVIS control execution?

  • The first cloud request includes TOOLS so the model can choose a task function.
  • Each requested name is looked up through AVAILABLE_TOOLS. An unknown name produces a blocked result.
  • Names in MUTATING_TOOLS pause at a typed approval prompt. Only an exact yes allows the local function to run.
  • The linked tool_call_id connects each local result to the model's original request. The final cloud response can then report what actually happened.
  • Select the existing main() function without selecting the final if __name__ == "__main__": block.
  • Replace the selected function by pasting this code:
def main() -> None:
    while True:
        command = input("YOU: ").strip()
        if command.lower() in {"stop", "quit", "exit"}:
            break
        print(f"JARVIS: {ask_jarvis(command)}")

What changed in the main loop?

The loop now passes every non-shutdown command into ask_jarvis(). This gives general questions the same cloud path while task requests gain controlled local tools.

  • Save jarvis.py.

Before you run the finished version, do you expect JARVIS to change the local task file before you approve the request?

  • Start the tool-enabled assistant by running this command:
python jarvis.py

The YOU: prompt means JARVIS is ready to route a request through the model and its local tool boundary.

  • Enter Add buy milk to my tasks at the YOU: prompt.
  • Enter yes when JARVIS asks you to approve the task addition.
  • Enter List my tasks at the next YOU: prompt.

You should see [ ] 1. buy milk in the response. The approved addition is now backed by the local tasks.json file.

  • Enter stop to finish the test.

Task not appearing?

  • Check that you entered the exact approval word yes.
  • Check that add_task appears in AVAILABLE_TOOLS.
  • Check that the tool result message uses the current call.id value.
  • Ask for help tracing the task request.

That is the control boundary working. Ollama Cloud chooses a defined tool while your local Python code approves the write and stores the result.

✔️ Awesome, I've got everything!

Your approved task changes now persist in tasks.json. Make sure jarvis.py is saved before continuing.

ⓧ I'd like to double check the full code

Compare your jarvis.py with this complete version. It contains the local store, three task schemas, the allowlist, the approval gate, and linked tool results.

import json
import os
from pathlib import Path

from openai import OpenAI

MODEL = "gemma4:31b"
TASKS_FILE = Path("tasks.json")

api_key = os.getenv("OLLAMA_API_KEY")
if not api_key:
    raise RuntimeError("Set OLLAMA_API_KEY before running JARVIS.")

client = OpenAI(
    base_url="https://ollama.com/v1",
    api_key=api_key,
)


def load_tasks() -> list[dict]:
    if not TASKS_FILE.exists():
        return []
    return json.loads(TASKS_FILE.read_text(encoding="utf-8"))


def save_tasks(tasks: list[dict]) -> None:
    TASKS_FILE.write_text(json.dumps(tasks, indent=2), encoding="utf-8")


def list_tasks() -> str:
    tasks = load_tasks()
    if not tasks:
        return "Your task list is empty."
    return "\n".join(
        f"[{'x' if task['done'] else ' '}] {number}. {task['title']}"
        for number, task in enumerate(tasks, start=1)
    )


def add_task(task: str) -> str:
    tasks = load_tasks()
    tasks.append({"title": task.strip(), "done": False})
    save_tasks(tasks)
    return f"Added task: {task.strip()}"


def complete_task(task_number: int) -> str:
    tasks = load_tasks()
    index = int(task_number) - 1
    if index < 0 or index >= len(tasks):
        return f"Task {task_number} does not exist."
    tasks[index]["done"] = True
    save_tasks(tasks)
    return f"Completed task {task_number}: {tasks[index]['title']}"


AVAILABLE_TOOLS = {
    "list_tasks": list_tasks,
    "add_task": add_task,
    "complete_task": complete_task,
}
MUTATING_TOOLS = {"add_task", "complete_task"}
TOOLS = [
    {"type": "function", "function": {"name": "list_tasks", "description": "List all local tasks and their completion state.", "parameters": {"type": "object", "properties": {}, "required": []}}},
    {"type": "function", "function": {"name": "add_task", "description": "Add one task to the local task list.", "parameters": {"type": "object", "properties": {"task": {"type": "string", "description": "The task text to save."}}, "required": ["task"]}}},
    {"type": "function", "function": {"name": "complete_task", "description": "Mark one local task complete by its displayed number.", "parameters": {"type": "object", "properties": {"task_number": {"type": "integer", "description": "The one-based task number shown by list_tasks."}}, "required": ["task_number"]}}},
]


def ask_jarvis(user_text: str) -> str:
    messages = [
        {"role": "system", "content": "You are JARVIS, a concise assistant. Use the supplied task tools whenever the request concerns tasks. Never claim a task changed unless a tool result confirms it."},
        {"role": "user", "content": user_text},
    ]
    response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    assistant_message = response.choices[0].message
    messages.append(assistant_message.to_dict())
    for call in assistant_message.tool_calls or []:
        name = call.function.name
        arguments = json.loads(call.function.arguments)
        function_to_call = AVAILABLE_TOOLS.get(name)
        if function_to_call is None:
            result = f"Blocked unknown tool: {name}"
        elif name in MUTATING_TOOLS:
            approved = input(f"Approve {name.replace('_', ' ')}? Type yes: ").strip().lower() == "yes"
            result = function_to_call(**arguments) if approved else "Cancelled by user."
        else:
            result = function_to_call(**arguments)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": str(result)})
    if not assistant_message.tool_calls:
        return assistant_message.content or "I could not produce a response."
    final_response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    return final_response.choices[0].message.content or "Task request processed."


def main() -> None:
    while True:
        command = input("YOU: ").strip()
        if command.lower() in {"stop", "quit", "exit"}:
            break
        print(f"JARVIS: {ask_jarvis(command)}")


if __name__ == "__main__":
    main()

The file should end with main() calling ask_jarvis(). The approval check belongs inside the tool-call loop.

Your cloud model can now make bounded decisions without gaining unrestricted access to your computer. Next, you will replace typed requests with locally recorded speech and hear JARVIS answer aloud.

Replace the Keyboard With a Storage-Light Voice Loop

Your Ollama Cloud model can now manage local tasks through a permission gate. The assistant still waits for every request at the keyboard.

This step uses sounddevice to capture each request on your computer. faster-whisper transcribes each recording locally before the transcript reaches Ollama Cloud.

In this step, get ready to:
  • Initialize local transcription and offline speech output.
  • Record five-second spoken commands through a push-to-talk loop.
  • Protect local task changes with two-second spoken confirmations.
Initialize the local voice engines

The tiny.en model provides local English transcription. pyttsx3 uses your computer's installed voice engine to speak replies offline.

  • Return to jarvis.py in the editor from earlier.
  • Replace everything from import json through the existing client setup with this initialization code:
import json
import os
import wave
from pathlib import Path

import pyttsx3
import sounddevice as sd
from faster_whisper import WhisperModel
from openai import OpenAI

MODEL = "gemma4:31b"
WHISPER_MODEL = "tiny.en"
SAMPLE_RATE = 16000
COMMAND_SECONDS = 5
CONFIRMATION_SECONDS = 2
AUDIO_FILE = Path("command.wav")
TASKS_FILE = Path("tasks.json")

api_key = os.getenv("OLLAMA_API_KEY")
if not api_key:
    raise RuntimeError("Set OLLAMA_API_KEY before running JARVIS.")

client = OpenAI(base_url="https://ollama.com/v1", api_key=api_key)
voice_engine = pyttsx3.init()
whisper_model = WhisperModel(WHISPER_MODEL, device="cpu", compute_type="int8")

What does this setup do?

  • The wave module supports the temporary command.wav recording.
  • The SAMPLE_RATE value sets microphone recording to 16 kHz.
  • The COMMAND_SECONDS value provides a five-second command window.
  • The CONFIRMATION_SECONDS value provides a two-second approval window.
  • The WhisperModel instance runs tiny.en on the CPU with int8 computation.
  • The voice_engine instance prepares offline speech output.
  • Save jarvis.py.
  • Load the local transcription model by running this command:
python jarvis.py

What does this command prove?

The first launch downloads and caches tiny.en locally. Later launches reuse the cached transcription model.

When initialization finishes, you will see the existing YOU: prompt. That prompt confirms the cloud client and local voice engines loaded together.

  • Type stop at the YOU: prompt.

That first local model load is complete. Your existing task assistant still exits cleanly after the new voice components initialize.

Voice engines not loading?

  • Confirm that you are using the activated virtual environment from the previous step.
  • Check your internet connection if the first tiny.en download does not finish.
  • Confirm that WHISPER_MODEL is set to tiny.en.
  • Ask for help with the local initialization: help me troubleshoot why my JARVIS voice engines do not initialize.
Build the push-to-talk loop

A local WAV file gives the transcriber a short recording to process. The loop removes command.wav after recovering its transcript.

  • Find def ask_jarvis(user_text: str) -> str: in jarvis.py.
  • Paste these local voice helpers directly above ask_jarvis():
def speak(text: str) -> None:
    message = text.strip() or "I do not have a response."
    print(f"JARVIS: {message}")
    voice_engine.say(message)
    voice_engine.runAndWait()


def record_wav(seconds: int) -> None:
    audio = sd.rec(int(seconds * SAMPLE_RATE), samplerate=SAMPLE_RATE, channels=1, dtype="int16", blocking=True)
    with wave.open(str(AUDIO_FILE), "wb") as wav_file:
        wav_file.setnchannels(1)
        wav_file.setsampwidth(2)
        wav_file.setframerate(SAMPLE_RATE)
        wav_file.writeframes(audio.tobytes())


def transcribe_wav() -> str:
    segments, _ = whisper_model.transcribe(str(AUDIO_FILE), language="en", beam_size=5)
    return "".join(segment.text for segment in segments).strip()


def listen(seconds: int) -> str:
    print(f"Listening for {seconds} seconds...")
    record_wav(seconds)
    transcript = transcribe_wav()
    AUDIO_FILE.unlink(missing_ok=True)
    print(f"YOU: {transcript or '[no speech detected]'}")
    return transcript

How does the local voice pipeline work?

  • The speak() function prints each reply before sending it to the offline voice engine.
  • The record_wav() function captures one mono channel with 16-bit samples.
  • The transcribe_wav() function consumes every returned segment to trigger local transcription.
  • The listen() function deletes the temporary recording after transcription.
  • The terminal prints the transcript so you can inspect the exact text sent to Ollama Cloud.
  • Find the existing main() function at the bottom of jarvis.py.
  • Replace the existing main() function with this push-to-talk loop:
def main() -> None:
    while True:
        input("Press Enter to speak...")
        command = listen(COMMAND_SECONDS)
        if command.lower().strip(" .!?") in {"stop", "quit", "exit"}:
            speak("Shutting down.")
            break
        if command:
            speak(ask_jarvis(command))


if __name__ == "__main__":
    main()

What changes in the main loop?

  • The Press Enter to speak... prompt lets you choose when recording starts.
  • The listen(COMMAND_SECONDS) call records one five-second command.
  • The shutdown check handles stop, quit, or exit before any cloud request.
  • The speak() call reads the final response through your computer's voice engine.
  • Save jarvis.py.

Before you start the loop, what do you expect the terminal to show before recording begins?

  • Start the push-to-talk loop by running this command:
python jarvis.py

What does this command start?

You will see Press Enter to speak... after the local model loads. Your operating system may request microphone permission when recording starts.

Each completed recording prints its local transcript. A non-empty transcript becomes the text request sent to the cloud model.

  • Grant microphone access if your operating system requests it.
  • Press Enter to start the five-second recording.
  • Say List my tasks during the recording.

You will see your local transcript after the recording finishes. You will hear JARVIS read the task list through the offline voice engine.

  • Press Enter when the next command prompt appears.
  • Say stop during the recording.

Your first spoken request has completed the full local-audio path. Raw microphone audio stayed on your computer while its transcript powered the cloud request.

Not hearing or transcribing your request?

  • Confirm that your operating system granted microphone access to the terminal or editor.
  • Speak close to the selected microphone during the five-second recording window.
  • Raise your speaker volume if the terminal prints a reply without audible speech.
  • Ask for help with the voice path: help me troubleshoot my local JARVIS recording or speech output.
Use spoken approval for task changes

Mutating tools still need your approval before they change tasks.json. The same local transcription pipeline can capture that decision through a separate two-second recording.

  • Find the existing ask_jarvis() function in jarvis.py.
  • Replace the existing function with this spoken-confirmation flow:
def is_approved(transcript: str) -> bool:
    return bool({"yes", "confirm", "okay", "approve"} & set(transcript.lower().replace(".", "").split()))


def ask_jarvis(user_text: str) -> str:
    messages = [{"role": "system", "content": "You are JARVIS, a concise assistant. Use the supplied task tools whenever the request concerns tasks. Never claim a task changed unless a tool result confirms it."}, {"role": "user", "content": user_text}]
    response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    assistant_message = response.choices[0].message
    messages.append(assistant_message.to_dict())
    tool_calls = assistant_message.tool_calls or []
    if not tool_calls:
        return assistant_message.content or "I could not produce a response."
    last_result = ""
    for call in tool_calls:
        name = call.function.name
        arguments = json.loads(call.function.arguments)
        function_to_call = AVAILABLE_TOOLS.get(name)
        if function_to_call is None:
            result = f"Blocked unknown tool: {name}"
        elif name in MUTATING_TOOLS:
            speak(f"Please confirm {name.replace('_', ' ')}. Say yes or no.")
            result = function_to_call(**arguments) if is_approved(listen(CONFIRMATION_SECONDS)) else "Cancelled by user."
        else:
            result = function_to_call(**arguments)
        last_result = str(result)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": last_result})
    final_response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    return final_response.choices[0].message.content or last_result

How does spoken approval stay safe?

  • The is_approved() function accepts a small local set of approval words.
  • The permission gate still applies only to names in MUTATING_TOOLS.
  • The listen(CONFIRMATION_SECONDS) call records a separate two-second response.
  • A rejected request returns Cancelled by user. without running the local function.
  • An approved result returns to Ollama Cloud through its linked tool message.
  • Save jarvis.py.

Before you run this, do you expect a local task change to happen before JARVIS hears an approval?

  • Start the completed voice loop by running this command:
python jarvis.py

What are you testing?

A mutating request must pass through spoken confirmation before the local task file changes. A read-only request skips that confirmation.

Both request types use local transcription. JARVIS speaks their final responses through the offline voice engine.

  • Press Enter to start a command recording.
  • Say Add review voice loop to my tasks during the five-second recording.
  • Say yes during the two-second confirmation recording.
  • Press Enter when the next command prompt appears.
  • Say List my tasks during the five-second recording.

You will see the command and confirmation transcripts in the terminal. You will hear a task list containing review voice loop.

  • Press Enter when the next command prompt appears.
  • Say stop during the recording.

That is the complete storage-light voice loop working. Spoken commands and approvals stay under local audio control while transcript text drives the cloud model.

Spoken approval not changing the task list?

  • Say an approval word immediately after JARVIS asks for confirmation.
  • Check the printed confirmation transcript for yes, confirm, okay, or approve.
  • Confirm that the new task appears in tasks.json only after an accepted approval word.
  • Ask for help with the permission gate: help me debug spoken approval in my JARVIS task loop.

✔️ Awesome, I've got everything!

Your saved jarvis.py now contains the complete push-to-talk voice loop.

ⓧ I'd like to double check the full code

The complete current version of jarvis.py appears below for comparison.

import json
import os
import wave
from pathlib import Path

import pyttsx3
import sounddevice as sd
from faster_whisper import WhisperModel
from openai import OpenAI

MODEL = "gemma4:31b"
WHISPER_MODEL = "tiny.en"
SAMPLE_RATE = 16000
COMMAND_SECONDS = 5
CONFIRMATION_SECONDS = 2
AUDIO_FILE = Path("command.wav")
TASKS_FILE = Path("tasks.json")

api_key = os.getenv("OLLAMA_API_KEY")
if not api_key:
    raise RuntimeError("Set OLLAMA_API_KEY before running JARVIS.")

client = OpenAI(base_url="https://ollama.com/v1", api_key=api_key)
voice_engine = pyttsx3.init()
whisper_model = WhisperModel(WHISPER_MODEL, device="cpu", compute_type="int8")


def load_tasks() -> list[dict]:
    if not TASKS_FILE.exists():
        return []
    return json.loads(TASKS_FILE.read_text(encoding="utf-8"))


def save_tasks(tasks: list[dict]) -> None:
    TASKS_FILE.write_text(json.dumps(tasks, indent=2), encoding="utf-8")


def list_tasks() -> str:
    tasks = load_tasks()
    if not tasks:
        return "Your task list is empty."
    return "\n".join(f"[{'x' if task['done'] else ' '}] {number}. {task['title']}" for number, task in enumerate(tasks, start=1))


def add_task(task: str) -> str:
    tasks = load_tasks()
    tasks.append({"title": task.strip(), "done": False})
    save_tasks(tasks)
    return f"Added task: {task.strip()}"


def complete_task(task_number: int) -> str:
    tasks = load_tasks()
    index = int(task_number) - 1
    if index < 0 or index >= len(tasks):
        return f"Task {task_number} does not exist."
    tasks[index]["done"] = True
    save_tasks(tasks)
    return f"Completed task {task_number}: {tasks[index]['title']}"


AVAILABLE_TOOLS = {"list_tasks": list_tasks, "add_task": add_task, "complete_task": complete_task}
MUTATING_TOOLS = {"add_task", "complete_task"}
TOOLS = [{"type": "function", "function": {"name": "list_tasks", "description": "List all local tasks and their completion state.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "add_task", "description": "Add one task to the local task list.", "parameters": {"type": "object", "properties": {"task": {"type": "string", "description": "The task text to save."}}, "required": ["task"]}}}, {"type": "function", "function": {"name": "complete_task", "description": "Mark one local task complete by its displayed number.", "parameters": {"type": "object", "properties": {"task_number": {"type": "integer", "description": "The one-based task number shown by list_tasks."}}, "required": ["task_number"]}}}]


def speak(text: str) -> None:
    message = text.strip() or "I do not have a response."
    print(f"JARVIS: {message}")
    voice_engine.say(message)
    voice_engine.runAndWait()


def record_wav(seconds: int) -> None:
    audio = sd.rec(int(seconds * SAMPLE_RATE), samplerate=SAMPLE_RATE, channels=1, dtype="int16", blocking=True)
    with wave.open(str(AUDIO_FILE), "wb") as wav_file:
        wav_file.setnchannels(1)
        wav_file.setsampwidth(2)
        wav_file.setframerate(SAMPLE_RATE)
        wav_file.writeframes(audio.tobytes())


def transcribe_wav() -> str:
    segments, _ = whisper_model.transcribe(str(AUDIO_FILE), language="en", beam_size=5)
    return "".join(segment.text for segment in segments).strip()


def listen(seconds: int) -> str:
    print(f"Listening for {seconds} seconds...")
    record_wav(seconds)
    transcript = transcribe_wav()
    AUDIO_FILE.unlink(missing_ok=True)
    print(f"YOU: {transcript or '[no speech detected]'}")
    return transcript


def is_approved(transcript: str) -> bool:
    return bool({"yes", "confirm", "okay", "approve"} & set(transcript.lower().replace(".", "").split()))


def ask_jarvis(user_text: str) -> str:
    messages = [{"role": "system", "content": "You are JARVIS, a concise assistant. Use the supplied task tools whenever the request concerns tasks. Never claim a task changed unless a tool result confirms it."}, {"role": "user", "content": user_text}]
    response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    assistant_message = response.choices[0].message
    messages.append(assistant_message.to_dict())
    tool_calls = assistant_message.tool_calls or []
    if not tool_calls:
        return assistant_message.content or "I could not produce a response."
    last_result = ""
    for call in tool_calls:
        name = call.function.name
        arguments = json.loads(call.function.arguments)
        function_to_call = AVAILABLE_TOOLS.get(name)
        if function_to_call is None:
            result = f"Blocked unknown tool: {name}"
        elif name in MUTATING_TOOLS:
            speak(f"Please confirm {name.replace('_', ' ')}. Say yes or no.")
            result = function_to_call(**arguments) if is_approved(listen(CONFIRMATION_SECONDS)) else "Cancelled by user."
        else:
            result = function_to_call(**arguments)
        last_result = str(result)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": last_result})
    final_response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    return final_response.choices[0].message.content or last_result


def main() -> None:
    while True:
        input("Press Enter to speak...")
        command = listen(COMMAND_SECONDS)
        if command.lower().strip(" .!?") in {"stop", "quit", "exit"}:
            speak("Shutting down.")
            break
        if command:
            speak(ask_jarvis(command))


if __name__ == "__main__":
    main()

Your storage-light voice loop can now hear commands and protect local task changes. Next, you will give JARVIS a local wake phrase so it can wait for you by name.

Wake JARVIS by Name

Your push-to-talk JARVIS already transcribes requests locally. It also protects task changes with spoken confirmation.

The keyboard still controls when each request begins. This step removes that interruption so JARVIS can wake by name before handling one spoken request.

In this step, get ready to:
  • Prove where the current voice flow depends on the keyboard.
  • Replace that dependency with local wake-phrase activation.
  • Demo conversation, task tools, confirmation, and local shutdown.
Test the keyboard bottleneck

The current loop records speech only after it receives keyboard input. Running it once makes that limitation visible before you replace it.

Before you run this, consider whether JARVIS starts listening without a keyboard press.

  • Start the current push-to-talk assistant by running this command:
python jarvis.py

What does this test prove?

This starts the version from the previous step. Its first input gate sits inside the existing loop.

You will see Press Enter to speak... in the terminal. JARVIS waits there until you touch the keyboard, which is the intended shortfall.

  • Stop the current process by pressing Ctrl+C.

Does the current assistant stop early?

  • Check that you are using the activated terminal from the previous step.
  • Confirm that the terminal still has the Ollama Cloud key loaded for this session.
  • Ask for help with the startup problem through this JARVIS startup troubleshooting prompt.
Add local wake-word detection

PocketSphinx provides the lightweight local activation layer. Its LiveSpeech listener watches for the wake phrase without sending microphone audio to Ollama Cloud.

  • In jarvis.py, replace the import and constant section at the top of the file with this code:
import json
import os
import wave
from pathlib import Path

import pyttsx3
import sounddevice as sd
from faster_whisper import WhisperModel
from openai import OpenAI
from pocketsphinx import LiveSpeech

MODEL = "gemma4:31b"
WHISPER_MODEL = "tiny.en"
WAKE_PHRASE = "hey jarvis"
WAKE_THRESHOLD = 1e-20
SAMPLE_RATE = 16000
COMMAND_SECONDS = 5
CONFIRMATION_SECONDS = 2
AUDIO_FILE = Path("command.wav")
TASKS_FILE = Path("tasks.json")

What does this configuration do?

  • LiveSpeech gives the program access to PocketSphinx microphone keyword detection.
  • WAKE_PHRASE holds the local phrase that activates JARVIS.
  • WAKE_THRESHOLD supplies the keyword threshold used by the local listener.
  • Save jarvis.py.
  • Confirm that the new import and constants load by running this command:
python jarvis.py

What does this check confirm?

Reaching the existing push-to-talk prompt confirms that Python loaded PocketSphinx. It also confirms that the new wake-word values contain valid Python syntax.

You will see the existing Press Enter to speak... prompt again. The imported listener is ready for the final loop.

  • Stop the check by pressing Ctrl+C.

Does PocketSphinx fail to load?

  • Check that the activated virtual environment contains the packages from requirements.txt.
  • Return to the activated terminal from earlier if another Python environment is running the file.
  • Ask for targeted help through this PocketSphinx import troubleshooting prompt.
Build the hands-free loop

The wake listener must release the microphone as soon as it detects the phrase. Closing its iterator gives the five-second command recorder access to the same microphone.

  • In jarvis.py, add this function directly above main():
def wait_for_wake_word() -> None:
    print('Listening for "Hey Jarvis"...')
    listener = iter(
        LiveSpeech(
            keyphrase=WAKE_PHRASE,
            kws_threshold=WAKE_THRESHOLD,
        )
    )
    try:
        next(listener)
    finally:
        listener.close()
    print("Wake word detected.")

How does the listener work?

  • LiveSpeech listens locally for the value stored in WAKE_PHRASE.
  • next(listener) waits until PocketSphinx detects the phrase.
  • finally guarantees that the listener closes after detection.
  • listener.close() releases the microphone before command recording starts.
  • In jarvis.py, locate the current main() function containing input("Press Enter to speak...").
  • Replace that entire function with this hands-free loop:
def main() -> None:
    speak("Cloud JARVIS is online.")
    while True:
        wait_for_wake_word()
        speak("Yes?")
        command = listen(COMMAND_SECONDS)

        if not command:
            speak("I did not hear a command.")
            continue

        if command.lower().strip(" .!?") in {"stop", "quit", "exit"}:
            speak("Shutting down.")
            break

        reply = ask_jarvis(command)
        speak(reply)

What does the final loop do?

  • wait_for_wake_word() keeps activation on the computer.
  • listen(COMMAND_SECONDS) records one five-second command for local transcription.
  • stop, quit, or exit ends the process locally without calling the cloud model.
  • ask_jarvis() sends transcript text through the existing cloud conversation and task-tool path.
  • Save jarvis.py.

Before you run the final assistant, predict what happens after one complete request finishes.

  • Start the wake-word assistant by running this command:
python jarvis.py

What happens at startup?

JARVIS announces that it is online through the offline voice engine. The terminal then shows Listening for "Hey Jarvis"... while PocketSphinx waits locally.

  • Say Hey Jarvis clearly near your microphone.
  • Ask What is seven times eight? after you hear Yes?.

You will see your local transcript. You will hear the cloud model's answer before the terminal returns to Listening for "Hey Jarvis"....

  • Say Hey Jarvis again.
  • Ask What tasks do I have? after the acknowledgement.

You will hear the task list stored in tasks.json. The terminal returns to local wake-word listening after the response.

  • Say Hey Jarvis again.
  • Request Add test wake-word mode to my tasks after the acknowledgement.
  • Say yes when JARVIS asks for confirmation.

You will hear confirmation that the task was added. This proves that wake-word activation still preserves the local approval gate.

  • Say Hey Jarvis one final time.
  • Say stop after you hear Yes?.

You will hear Shutting down. without a cloud response. The terminal then returns to its prompt.

Does JARVIS miss the wake phrase?

  • Speak the phrase clearly with a brief pause before the command.
  • Check that no other application is holding the microphone.
  • Confirm that listener.close() remains inside the finally block.
  • Ask for help through this wake-word troubleshooting prompt.

That is the full hands-free loop working. JARVIS now wakes locally, handles one spoken request, replies aloud, and returns to listening.

✔️ Awesome, I've got everything!

Great work. Save jarvis.py so the wake-word loop is ready for your next session.

ⓧ I'd like to double check the full code

Compare your jarvis.py with this completed wake-word version.

import json
import os
import wave
from pathlib import Path

import pyttsx3
import sounddevice as sd
from faster_whisper import WhisperModel
from openai import OpenAI
from pocketsphinx import LiveSpeech

MODEL = "gemma4:31b"
WHISPER_MODEL = "tiny.en"
WAKE_PHRASE = "hey jarvis"
WAKE_THRESHOLD = 1e-20
SAMPLE_RATE = 16000
COMMAND_SECONDS = 5
CONFIRMATION_SECONDS = 2
AUDIO_FILE = Path("command.wav")
TASKS_FILE = Path("tasks.json")

api_key = os.getenv("OLLAMA_API_KEY")
if not api_key:
    raise RuntimeError("Set OLLAMA_API_KEY before running JARVIS.")

client = OpenAI(
    base_url="https://ollama.com/v1",
    api_key=api_key,
)
voice_engine = pyttsx3.init()
whisper_model = WhisperModel(
    WHISPER_MODEL,
    device="cpu",
    compute_type="int8",
)


def load_tasks() -> list[dict]:
    if not TASKS_FILE.exists():
        return []
    return json.loads(TASKS_FILE.read_text(encoding="utf-8"))


def save_tasks(tasks: list[dict]) -> None:
    TASKS_FILE.write_text(
        json.dumps(tasks, indent=2),
        encoding="utf-8",
    )


def list_tasks() -> str:
    tasks = load_tasks()
    if not tasks:
        return "Your task list is empty."

    lines = []
    for number, task in enumerate(tasks, start=1):
        marker = "x" if task["done"] else " "
        lines.append(f"[{marker}] {number}. {task['title']}")
    return "\n".join(lines)


def add_task(task: str) -> str:
    tasks = load_tasks()
    tasks.append({"title": task.strip(), "done": False})
    save_tasks(tasks)
    return f"Added task: {task.strip()}"


def complete_task(task_number: int) -> str:
    tasks = load_tasks()
    index = int(task_number) - 1
    if index < 0 or index >= len(tasks):
        return f"Task {task_number} does not exist."

    tasks[index]["done"] = True
    save_tasks(tasks)
    return f"Completed task {task_number}: {tasks[index]['title']}"


AVAILABLE_TOOLS = {
    "list_tasks": list_tasks,
    "add_task": add_task,
    "complete_task": complete_task,
}
MUTATING_TOOLS = {"add_task", "complete_task"}
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "list_tasks",
            "description": "List all local tasks and their completion state.",
            "parameters": {
                "type": "object",
                "properties": {},
                "required": [],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "add_task",
            "description": "Add one task to the local task list.",
            "parameters": {
                "type": "object",
                "properties": {
                    "task": {
                        "type": "string",
                        "description": "The task text to save.",
                    }
                },
                "required": ["task"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "complete_task",
            "description": "Mark one local task complete by its displayed number.",
            "parameters": {
                "type": "object",
                "properties": {
                    "task_number": {
                        "type": "integer",
                        "description": "The one-based task number shown by list_tasks.",
                    }
                },
                "required": ["task_number"],
            },
        },
    },
]


def speak(text: str) -> None:
    message = text.strip() or "I do not have a response."
    print(f"JARVIS: {message}")
    voice_engine.say(message)
    voice_engine.runAndWait()


def record_wav(seconds: int) -> None:
    audio = sd.rec(
        int(seconds * SAMPLE_RATE),
        samplerate=SAMPLE_RATE,
        channels=1,
        dtype="int16",
        blocking=True,
    )
    with wave.open(str(AUDIO_FILE), "wb") as wav_file:
        wav_file.setnchannels(1)
        wav_file.setsampwidth(2)
        wav_file.setframerate(SAMPLE_RATE)
        wav_file.writeframes(audio.tobytes())


def transcribe_wav() -> str:
    segments, _ = whisper_model.transcribe(
        str(AUDIO_FILE),
        language="en",
        beam_size=5,
    )
    return "".join(segment.text for segment in segments).strip()


def listen(seconds: int) -> str:
    print(f"Listening for {seconds} seconds...")
    record_wav(seconds)
    transcript = transcribe_wav()
    AUDIO_FILE.unlink(missing_ok=True)
    print(f"YOU: {transcript or '[no speech detected]'}")
    return transcript


def is_approved(transcript: str) -> bool:
    approval_words = {"yes", "confirm", "okay", "approve"}
    spoken_words = set(transcript.lower().replace(".", "").split())
    return bool(approval_words & spoken_words)


def ask_jarvis(user_text: str) -> str:
    messages = [
        {
            "role": "system",
            "content": (
                "You are JARVIS, a concise assistant. "
                "Use the supplied task tools whenever the request concerns tasks. "
                "Never claim a task changed unless a tool result confirms it."
            ),
        },
        {"role": "user", "content": user_text},
    ]
    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        tools=TOOLS,
    )
    assistant_message = response.choices[0].message
    messages.append(assistant_message.to_dict())

    tool_calls = assistant_message.tool_calls or []
    if not tool_calls:
        return assistant_message.content or "I could not produce a response."

    last_result = ""
    for call in tool_calls:
        name = call.function.name
        arguments = json.loads(call.function.arguments)
        function_to_call = AVAILABLE_TOOLS.get(name)

        if function_to_call is None:
            result = f"Blocked unknown tool: {name}"
        elif name in MUTATING_TOOLS:
            speak(f"Please confirm {name.replace('_', ' ')}. Say yes or no.")
            confirmation = listen(CONFIRMATION_SECONDS)
            if is_approved(confirmation):
                result = function_to_call(**arguments)
            else:
                result = "Cancelled by user."
        else:
            result = function_to_call(**arguments)

        last_result = str(result)
        messages.append(
            {
                "role": "tool",
                "tool_call_id": call.id,
                "content": last_result,
            }
        )

    final_response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        tools=TOOLS,
    )
    return final_response.choices[0].message.content or last_result


def wait_for_wake_word() -> None:
    print('Listening for "Hey Jarvis"...')
    listener = iter(
        LiveSpeech(
            keyphrase=WAKE_PHRASE,
            kws_threshold=WAKE_THRESHOLD,
        )
    )
    try:
        next(listener)
    finally:
        listener.close()
    print("Wake word detected.")


def main() -> None:
    speak("Cloud JARVIS is online.")
    while True:
        wait_for_wake_word()
        speak("Yes?")
        command = listen(COMMAND_SECONDS)

        if not command:
            speak("I did not hear a command.")
            continue

        if command.lower().strip(" .!?") in {"stop", "quit", "exit"}:
            speak("Shutting down.")
            break

        reply = ask_jarvis(command)
        speak(reply)


if __name__ == "__main__":
    main()

What should match?

The full file keeps the task tools, spoken confirmation, local transcription, and offline speech from earlier steps. This step adds the PocketSphinx import, wake-word constants, local listener, and hands-free main loop.

Secret mission

Secret Mission: Add a Tamper-Evident Action Trail

Give JARVIS an append-only record of every approved task change. Rejected changes and read-only requests stay out of the local audit file.

Clean Up Your Resources

Clean Up Your Resources

Your code and task data live on your computer. Ollama Cloud charges per token when JARVIS sends a request, so there is no always-on virtual machine to delete.

Cost warning

Each request to gemma4:31b can consume usage credits. There is no always-on virtual machine in this project.

Pause or Delete is the safer choice once testing is finished.

Resources you used:

  • Your local cloud-jarvis/ folder with requirements.txt, jarvis.py, .venv, tasks.json, and audit.jsonl.
  • Your Ollama Cloud API key, which remains valid until you revoke it.

Keep everything running

No action needed. Choose this if you want to keep using your voice assistant or continue developing its local tools.

  • The cloud-jarvis/ folder remains ready with your code and local data.
  • The current terminal keeps OLLAMA_API_KEY loaded until you close it.
  • The API key remains active for future cloud requests.
  • The shared tiny.en transcription cache remains available for future local transcription.

Pause - I'll come back to this later

Shut down JARVIS to prevent new cloud requests. Your code and local records remain available for your next session.

  • Press Ctrl+C in the activated terminal if JARVIS is still listening.
  • Close the current terminal to clear OLLAMA_API_KEY from that session.
  • Leave cloud-jarvis/ in place for your next session.

Delete - I don't want to use this again

Remove the local project files and revoke the cloud credential if you want a clean break. Deleting the folder permanently removes its task history and audit trail.

  • Copy tasks.json or audit.jsonl to a safe location if you want to keep those records.
  • Stop JARVIS by pressing Ctrl+C if it is still listening.
  • Close the current terminal to clear OLLAMA_API_KEY from that session.
  • Close the editor window that has cloud-jarvis/ open.
  • Return to the Ollama API-key settings page.
  • Revoke the API key used for this project.
  • Confirm that the key is no longer active.

The final cleanup is local. Choose your operating system below.

macOS

Finder can remove the project folder without depending on the terminal's current location.

  • Use macOS search to open Finder.
  • Locate the cloud-jarvis/ folder from this project.
  • Move cloud-jarvis/ to the Trash.
  • Open the Trash.
  • Permanently delete only the cloud-jarvis/ folder.
  • Confirm that cloud-jarvis/ no longer appears in its previous location.

Windows

File Explorer can remove the project folder without depending on the terminal's current location.

  • Use Windows search to open File Explorer.
  • Locate the cloud-jarvis/ folder from this project.
  • Move cloud-jarvis/ to the Recycle Bin.
  • Open the Recycle Bin.
  • Permanently delete only the cloud-jarvis/ folder.
  • Confirm that cloud-jarvis/ no longer appears in its previous location.

The shared tiny.en transcription cache stays outside the project folder. Keeping it does not send requests to Ollama Cloud.

Nice Work!

Nice Work!

Mission complete! Your voice-controlled JARVIS task butler is ready. Raw audio stays on your computer while managed cloud inference supplies its language model.

You've learned how to:

  • Built a hands-free voice loop with PocketSphinx wake-word detection. Local faster-whisper transcription turns each command into text. The computer's offline voice engine reads each response aloud.
  • Connected the gemma4:31b model to Ollama Cloud through an OpenAI-compatible API. Cloud inference handles each conversation. No Ollama language model occupies local disk space.
  • Built local task tools that persist task state in tasks.json. A confirmation gate protects every task mutation before execution. Unknown tools remain blocked.
  • Secret Mission: Added an append-only audit trail in audit.jsonl. Each approved mutation creates one complete record for later review. Rejected changes leave no audit record.

Ready to quiz yourself?