Build a Character AI Desktop Agent

Fork an open-source Mac app to run a character AI agent locally.

Introduction

30 Second Summary

AI assistants are everywhere, but they all cost extra subscriptions. What if you could build your own desktop companion that uses the tools you already pay for?

In this project, you will build a character AI desktop agent that lives next to your cursor, listens to your voice, and responds as Spider-Man, Iron Man, or any personality you define. All AI processing runs through your existing Claude Code subscription with zero additional costs.

What You'll Build

A fork of the open-source Clicky desktop agent, modified to use the Claude Agent SDK for AI responses, Kokoro-MLX for natural local speech, and macOS built-in speech recognition for voice input.

By the end of this project, you'll have:

  • A working macOS menu-bar app with a character overlay that follows your cursor and displays streamed AI responses.
  • A Python agent bridge using the Claude Agent SDK that leverages your existing Claude Code subscription for all AI processing.
  • Local text-to-speech powered by Kokoro-MLX running natively on Apple Silicon, replacing paid cloud TTS services.
  • Swappable character personalities (Spider-Man, Iron Man, custom Mentor) with distinct voices and speech patterns.
  • Secret Mission: Add a character selection dropdown in the menu bar so you can switch personalities on the fly.

Are there any prerequisites?

This project requires a Mac with Apple Silicon (M1, M2, M3, or M4) running macOS 14.2 or later. You also need Xcode 15+ installed and the Claude Code CLI already installed and authenticated.

If you do not yet have a Claude Code subscription, you will need one before starting. The project uses your existing subscription for all AI processing.

Before We Start

You are about to build a character AI agent that lives on your desktop, powered entirely by your existing Claude Code subscription. Instead of paying for a separate AI assistant service, you will wire your own from open-source parts.

In this step, you will clone the open-source Clicky desktop agent, set up a Python environment with the Claude Agent SDK and Kokoro-MLX for local text-to-speech, and verify everything connects to your Claude Code subscription.

In this step, get ready to:
  • Clone the Clicky repository and open it in Xcode.
  • Set up a Python environment with the Claude Agent SDK and Kokoro-MLX.
  • Test the Agent SDK connection using your existing Claude Code subscription.
Clone the Clicky repository and open in Xcode

Clicky is an open-source macOS menu-bar app that provides a desktop AI companion with push-to-talk voice input and a cursor overlay. You will fork this as your starting point.

  • Open Terminal by pressing Cmd+Space (macOS) to open Spotlight, then type Terminal and press Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Clone the Clicky repository and move into the project folder by running these commands:
git clone https://github.com/farzaa/clicky.git
cd clicky
  • Confirm the clone succeeded by running:
ls

You should see leanring-buddy (yes, the typo is intentional), worker, and CLAUDE.md in the output.

  • Open the Xcode project by running:
open leanring-buddy.xcodeproj

What does this command do?

The open command tells macOS to launch the file with its default application. Since .xcodeproj files are associated with Xcode, this opens the project directly.

  • In Xcode, look at the file navigator on the left side. Expand the leanring-buddy folder.
  • Verify you can see these key files: leanring_buddyApp.swift, CompanionManager.swift, ClaudeAPI.swift, and ElevenLabsTTSClient.swift.
  • Try building the project by pressing Cmd+B.

Will the build succeed?

The build will likely fail because the project expects API keys and worker URLs that you have not configured yet. This is completely expected. You are going to replace all of those cloud dependencies in the next steps.

Don't see the project files?

Make sure you ran open leanring-buddy.xcodeproj from inside the clicky folder. If the file navigator is hidden, press Cmd+1 to reveal it.

Help me find the project files in Xcode after cloning farzaa/clicky.

Set up Python environment with Claude Agent SDK and Kokoro-MLX

The Claude Agent SDK and Kokoro-MLX are both Python packages. You need Python 3.10 through 3.12 specifically, because Kokoro-MLX does not support versions outside that range.

  • Switch back to your Terminal window.
  • Check your Python version by running:
python3 --version

✔️ I see version 3.10, 3.11, or 3.12

Your Python version is compatible. You are ready to create the virtual environment.

ⓧ I see a different version

macOS often ships with an older or newer Python that does not match the required range. Install Python 3.12 using Homebrew.

  • Install Python 3.12 by running:
brew install python@3.12

After installation, Homebrew installs the binary as python3.12. Use python3.12 instead of python3 for the virtual environment creation in the next step.

Why python3.12 specifically?

On macOS, the generic python3 command may still point to the system version even after you install a newer one via Homebrew. Using the full versioned name python3.12 guarantees you target the correct installation.

ⓧ Command not found

Python is not installed on your system. Install Python 3.12 using Homebrew.

  • Install Homebrew first (if you do not already have it) by visiting https://brew.sh and running the install command shown on the page.
  • Install Python 3.12 by running:
brew install python@3.12

Use python3.12 for the virtual environment creation in the next step.

  • Make sure you are in the clicky folder (~/Desktop/clicky), then create a virtual environment by running:
python3 -m venv venv

What if I installed Python 3.12 via Homebrew?

If you installed Python 3.12 in the previous tab, use python3.12 -m venv venv instead. This ensures the virtual environment uses the correct version.

  • Activate the virtual environment by running:
source venv/bin/activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms you are working inside the isolated environment.

  • Install the Claude Agent SDK by running:
pip install claude-agent-sdk
  • Install Kokoro-MLX and the soundfile library by running:
pip install kokoro-mlx soundfile

What are these packages?

The Claude Agent SDK gives you a Python API to query Claude using your existing Claude Code subscription. No separate API key is needed because it uses the bundled CLI binary that is already authenticated on your machine.

Kokoro-MLX is a text-to-speech engine built specifically for Apple Silicon. It runs the Kokoro-82M model locally using Apple's MLX framework for GPU acceleration.

Seeing permission errors during install?

Make sure (venv) appears in your prompt. If it does not, run source venv/bin/activate again before retrying the install commands.

Help me fix pip install errors in my Python venv for claude-agent-sdk and kokoro-mlx.

Test the Agent SDK connection to your Claude Code subscription

The Agent SDK uses your existing Claude Code authentication. You do not need a separate API key. Let's verify both the SDK and Kokoro-MLX work.

  • Test the Agent SDK by creating a small test script. Run this command to create and execute the test:
python3 -c "
import asyncio
from claude_agent_sdk import query, ClaudeAgentOptions

async def test():
    async for message in query(
        prompt='Say hello in one sentence.',
        options=ClaudeAgentOptions(
            system_prompt='You are Spider-Man. Keep it brief.',
            allowed_tools=[],
            max_turns=1,
            setting_sources=[],
        ),
    ):
        if hasattr(message, 'content'):
            for block in message.content:
                if hasattr(block, 'text'):
                    print(block.text)

asyncio.run(test())
"

You should see a short response from Claude in Spider-Man's voice. This confirms the SDK is connected to your subscription.

What does this test do?

The query() function sends a prompt to Claude using your Claude Code subscription. The ClaudeAgentOptions lets you pass a custom system prompt (the character personality), restrict available tools, and limit the conversation to one turn.

The setting_sources=[] ensures no filesystem settings are loaded, keeping the query isolated.

  • Now test Kokoro-MLX by running this command (the first run downloads a ~350MB model to your HuggingFace cache):
python3 -c "
import os
os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'
from kokoro_mlx import KokoroTTS

tts = KokoroTTS.from_pretrained('mlx-community/Kokoro-82M-bf16')
tts.speak('Hello, I am your character agent.', voice='af_heart')
"

You should hear a natural-sounding voice say the phrase through your speakers. This confirms Kokoro-MLX is working with your Apple Silicon GPU.

Why the PYTORCH_ENABLE_MPS_FALLBACK variable?

This environment variable enables GPU acceleration via Apple's Metal Performance Shaders on M1/M2/M3/M4 chips. Without it, some operations may fall back to CPU and run slower.

No audio or model download stalled?

The first run downloads the mlx-community/Kokoro-82M-bf16 model (~350MB) from HuggingFace. Make sure you have a stable internet connection and wait for the download to complete.

If you hear no audio, check that your Mac's volume is turned up and no headphones are intercepting the output.

Help me troubleshoot Kokoro-MLX audio output on macOS.

Your environment is ready. You have the Clicky codebase in Xcode, a working Agent SDK connection to your Claude Code subscription, and local text-to-speech through Kokoro-MLX. Next up, you will strip the paid cloud dependencies from the Clicky codebase.

Strip Cloud Dependencies

Your environment is ready and the Agent SDK is responding to queries. Now it's time to reshape the Clicky codebase itself.

The original Clicky app relies on three paid cloud services: a Cloudflare Worker proxy for AI and TTS, AssemblyAI for speech-to-text, and ElevenLabs for text-to-speech. You'll use Claude Code to surgically remove each one, leaving a clean shell that compiles but no longer talks to the cloud.

In this step, get ready to:
  • Create a CLAUDE.md project guide so Claude Code understands the codebase.
  • Remove Cloudflare Worker, AssemblyAI, and ElevenLabs dependencies.
  • Verify the project compiles cleanly in Xcode with zero cloud references.
Create a CLAUDE.md and remove the Cloudflare Worker proxy

A CLAUDE.md file lives at the root of your project and teaches Claude Code about your codebase. It describes the architecture, lists key files, and explains coding conventions.

Claude Code reads this file automatically when it starts a session in the directory. The better this file, the more targeted Claude Code's edits become.

  • Create a new file called CLAUDE.md in the root of your clicky project by running:
touch CLAUDE.md
  • Open CLAUDE.md in any text editor and paste the following content:
# Character AI Desktop Agent

## Project Overview
This is a fork of farzaa/clicky, modified to:
1. Use the Claude Agent SDK (Python) as the AI brain instead of a Cloudflare Worker proxy
2. Use Kokoro-MLX (local Python TTS) instead of ElevenLabs
3. Use Apple's built-in SFSpeechRecognizer instead of AssemblyAI
4. Add character personalities (Spider-Man, Iron Man, etc.)

## Architecture
- SwiftUI + AppKit menu bar app (no dock icon)
- Push-to-talk via ctrl+option global hotkey
- Voice input: SFSpeechRecognizer (on-device, Apple Neural Engine)
- AI: Claude Agent SDK via agent_bridge.py (uses existing Claude Code subscription)
- Voice output: Kokoro-MLX via agent_bridge.py (on-device, Apple Silicon MLX)
- Overlay: Full-screen transparent window with character response text

## Key Files
- CompanionManager.swift - Central state machine, coordinates the pipeline
- AgentBridge.swift - NEW: Manages the Python agent_bridge.py subprocess
- agent_bridge.py - NEW: Python script using Claude Agent SDK + Kokoro-MLX
- characters.json - NEW: Character personality definitions
- CLAUDE.md - This file

## Commands
- Build: Cmd+B in Xcode or `swift build`
- Run: Cmd+R in Xcode
- Test Agent: `cd venv && python agent_bridge.py --test "Say hello as Spider-Man"`
- Test TTS: `cd venv && python agent_bridge.py --test-tts "Hello world"`

## Coding Standards
- Swift 5.9+, async/await for all async operations
- No force unwraps (use guard let or if let)
- All subprocess calls must have timeout handling
- Character system prompts live in characters.json, not hardcoded

What does this file do?

The ## Project Overview section tells Claude Code what this project IS. The ## Architecture section maps out the component flow so Claude Code knows how the pieces connect.

The ## Key Files section points Claude Code to the important files. The NEW markers help it understand which files it needs to create versus which already exist.

  • Save CLAUDE.md.
  • Confirm the file exists by running:
ls CLAUDE.md

You should see CLAUDE.md listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before continuing.

ⓧ I'd like to double check the full code

# Character AI Desktop Agent

## Project Overview
This is a fork of farzaa/clicky, modified to:
1. Use the Claude Agent SDK (Python) as the AI brain instead of a Cloudflare Worker proxy
2. Use Kokoro-MLX (local Python TTS) instead of ElevenLabs
3. Use Apple's built-in SFSpeechRecognizer instead of AssemblyAI
4. Add character personalities (Spider-Man, Iron Man, etc.)

## Architecture
- SwiftUI + AppKit menu bar app (no dock icon)
- Push-to-talk via ctrl+option global hotkey
- Voice input: SFSpeechRecognizer (on-device, Apple Neural Engine)
- AI: Claude Agent SDK via agent_bridge.py (uses existing Claude Code subscription)
- Voice output: Kokoro-MLX via agent_bridge.py (on-device, Apple Silicon MLX)
- Overlay: Full-screen transparent window with character response text

## Key Files
- CompanionManager.swift - Central state machine, coordinates the pipeline
- AgentBridge.swift - NEW: Manages the Python agent_bridge.py subprocess
- agent_bridge.py - NEW: Python script using Claude Agent SDK + Kokoro-MLX
- characters.json - NEW: Character personality definitions
- CLAUDE.md - This file

## Commands
- Build: Cmd+B in Xcode or `swift build`
- Run: Cmd+R in Xcode
- Test Agent: `cd venv && python agent_bridge.py --test "Say hello as Spider-Man"`
- Test TTS: `cd venv && python agent_bridge.py --test-tts "Hello world"`

## Coding Standards
- Swift 5.9+, async/await for all async operations
- No force unwraps (use guard let or if let)
- All subprocess calls must have timeout handling
- Character system prompts live in characters.json, not hardcoded

Now you'll start a Claude Code session and use it to rip out the Cloudflare Worker proxy. Claude Code reads CLAUDE.md automatically, so it already understands what you're building.

  • Make sure your terminal is in the clicky project directory, then start a Claude Code session by running:
claude

What is a Claude Code session?

Running claude in your terminal opens an interactive session where you can give Claude Code natural language instructions. It can read, edit, create, and delete files in your project. It asks for permission before making changes.

  • Once the Claude Code prompt appears, paste this prompt to remove the Cloudflare Worker:
Remove the Cloudflare Worker proxy from this project. Specifically:
1. Delete the entire worker/ directory
2. Remove all references to the worker URL in Swift code (look for any Cloudflare Worker endpoints in ClaudeAPI.swift and anywhere else)
3. Remove the API key proxy logic from ClaudeAPI.swift
4. Keep the ClaudeAPI.swift file but strip out the worker-related networking code

The worker is being replaced by a local Python bridge (agent_bridge.py) that talks to the Claude Agent SDK directly.

What will Claude Code do?

Claude Code will scan the project, find the worker/ directory and all Swift files referencing it, then propose deletions and edits. It will ask you to approve each change before applying it.

Review each proposed change and approve it. If Claude Code asks clarifying questions, confirm that you want to remove all Cloudflare Worker references completely.

  • Approve the changes Claude Code proposes.
  • After Claude Code finishes, verify the worker/ directory is gone by running this in Claude Code's shell mode:
! ls worker/

You should see an error saying the directory does not exist. That confirms the removal worked.

Claude Code didn't remove everything?

If you still see worker URLs in the codebase, paste a follow-up prompt asking Claude Code to search for any remaining references: Search the entire project for any remaining references to 'worker' URLs or Cloudflare and remove them.

If Claude Code seems stuck or unresponsive, exit with /exit and restart with claude. It will pick up where it left off because CLAUDE.md persists your project context.

help me remove all Cloudflare Worker references from this Swift project

Replace AssemblyAI with Apple's SFSpeechRecognizer

The original Clicky app uses AssemblyAI for real-time speech transcription over a websocket connection. This costs money and requires an API key.

Apple's built-in SFSpeechRecognizer already exists in Clicky as a fallback provider. You'll promote it to the primary provider and remove AssemblyAI entirely.

  • In your Claude Code session, paste this prompt to swap the speech recognition provider:
Replace AssemblyAI with Apple's built-in SFSpeechRecognizer as the primary transcription provider. Specifically:
1. Remove all AssemblyAIStreamingTranscriptionProvider references and related code
2. Configure the existing Apple Speech (SFSpeechRecognizer) fallback as the PRIMARY and ONLY transcription provider
3. Keep the push-to-talk behavior with the ctrl+option global hotkey exactly as it is
4. Remove any AssemblyAI API key references or websocket connection code

The app already has Apple Speech as a fallback - just make it the default and remove AssemblyAI completely.

Why Apple's SFSpeechRecognizer?

SFSpeechRecognizer runs on-device using Apple's Neural Engine. It's free, private, and already built into the Clicky codebase as a fallback. For a project focused on local-first AI, it's the natural choice.

The push-to-talk hotkey (ctrl+option) stays the same. Only the transcription backend changes.

  • Approve the changes Claude Code proposes.
  • Verify there are no remaining AssemblyAI references by asking Claude Code:
Search the project for any remaining references to "AssemblyAI" or "assemblyai". List them if found.

Claude Code should report zero matches. If it finds leftover references, approve its suggestions to remove them.

Claude Code removed too much?

If Claude Code accidentally removed the push-to-talk hotkey logic or the SFSpeechRecognizer setup, ask it to restore those specifically. The ctrl+option hotkey is defined in CompanionManager.swift and must stay intact.

help me fix my speech recognition setup after removing AssemblyAI

Remove ElevenLabs TTS and verify the build

The last cloud dependency is ElevenLabs, which provides text-to-speech via the Cloudflare Worker. Since the worker is already gone, this code is dead. You'll remove it and leave a placeholder for the Kokoro-MLX bridge you'll build next.

  • In your Claude Code session, paste this prompt to remove ElevenLabs:
Remove ElevenLabs TTS completely from this project. Specifically:
1. Remove all functionality from ElevenLabsTTSClient.swift (you can delete the file or gut its contents)
2. Leave a placeholder method signature where TTS was called so the app still compiles
3. Remove any ElevenLabs API key references
4. The placeholder will be replaced by a Python bridge (agent_bridge.py) using Kokoro-MLX in the next step

After removing, make sure the project compiles without errors. Fix any broken references.
  • Approve the changes Claude Code proposes.
  • After Claude Code finishes, verify the project builds by pressing Cmd+B in Xcode.

What should I see?

Xcode should show a successful build with zero errors. You might see warnings about unused variables. That's fine. The important thing is no errors related to missing Cloudflare Worker URLs, AssemblyAI types, or ElevenLabs methods.

If you run the app (Cmd+R), the menu bar icon appears but the app cannot think or speak yet. That's expected.

  • If the build fails, paste the error messages directly into Claude Code and ask it to fix them.
  • Once the build succeeds, exit Claude Code by typing:
/exit

Build still failing?

The most common issue is a leftover reference to a deleted type or method. Start a new Claude Code session with claude and paste the exact Xcode error message. Claude Code can usually fix it in one shot because CLAUDE.md gives it full project context.

If you see a signing error instead of a code error, go to the project's Signing & Capabilities tab in Xcode and select your personal team under Team.

help me fix Xcode build errors after stripping cloud dependencies from Clicky

The codebase is clean. No Cloudflare Worker, no AssemblyAI, no ElevenLabs. In the next step, you'll wire the Claude Agent SDK as the AI brain and give the app the ability to think again.

Wire the Claude Agent SDK as the AI Brain

Your app can hear you (thanks to SFSpeechRecognizer), but it has no way to think or respond yet.

In this step, you'll build a Python "bridge" that sits between the Swift app and the Claude Agent SDK. The Swift app sends your voice transcript to this bridge, the bridge asks Claude using your existing Claude Code subscription, generates speech with Kokoro-MLX, and returns both text and audio back to Swift.

In this step, get ready to:
  • Create agent_bridge.py with Agent SDK and Kokoro-MLX integration.
  • Use Claude Code to generate AgentBridge.swift for managing the Python subprocess.
  • Test the full voice-to-response pipeline.
Create the agent_bridge.py script

The agent bridge is a long-running Python process. It loads the Kokoro-MLX model once at startup, then listens for JSON commands on stdin. For each command, it queries Claude via the Agent SDK and generates speech from the response.

  • Create a new file called agent_bridge.py in your project's root directory (the same folder that contains CLAUDE.md).
  • Add the file header, imports, and TTS functions by pasting this code:
#!/usr/bin/env python3
"""
Agent Bridge for Character AI Desktop Agent.
Uses the Claude Agent SDK to query Claude with character personalities,
and Kokoro-MLX for text-to-speech generation.

Runs as a long-lived subprocess. Accepts JSON commands on stdin,
returns JSON responses on stdout.
"""

import asyncio
import sys
import os
import json
import tempfile
from pathlib import Path

os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"


def setup_tts():
    """Load Kokoro-MLX model once at startup."""
    from kokoro_mlx import KokoroTTS
    tts = KokoroTTS.from_pretrained("mlx-community/Kokoro-82M-bf16")
    return tts


def generate_speech(tts, text, voice="af_heart", speed=1.0):
    """Generate speech and return path to WAV file."""
    import soundfile as sf

    result = tts.generate(text, voice=voice, speed=speed, sample_rate=24000)

    temp_dir = Path(tempfile.gettempdir()) / "character_agent_tts"
    temp_dir.mkdir(exist_ok=True)
    output_path = temp_dir / "response.wav"

    sf.write(str(output_path), result.audio, result.sample_rate)
    return str(output_path)

What does this code do?

The PYTORCH_ENABLE_MPS_FALLBACK environment variable enables GPU acceleration on Apple Silicon for any PyTorch operations.

The setup_tts() function loads the Kokoro-MLX model from mlx-community/Kokoro-82M-bf16 on HuggingFace. This happens once at startup so subsequent speech generation is fast.

The generate_speech() function takes text, generates audio at 24 kHz, and saves it to a temporary WAV file. The Swift app will play this file.

  • Below the generate_speech function, add the Agent SDK query function:
async def query_agent(user_message, system_prompt):
    """Query Claude via the Agent SDK with a character system prompt."""
    from claude_agent_sdk import query, ClaudeAgentOptions

    full_response = ""

    async for message in query(
        prompt=user_message,
        options=ClaudeAgentOptions(
            system_prompt=system_prompt,
            allowed_tools=[],
            max_turns=1,
            setting_sources=[],
        ),
    ):
        if hasattr(message, "content"):
            for block in message.content:
                if hasattr(block, "text"):
                    full_response += block.text

    return full_response

What does this code do?

This is the core of the bridge. The query() function from claude_agent_sdk returns an async iterator of response messages. It uses your existing Claude Code subscription automatically through the bundled CLI binary.

The ClaudeAgentOptions object configures the request. Setting allowed_tools=[] prevents Claude from using file system tools. Setting max_turns=1 keeps it to a single response. The setting_sources=[] ensures an isolated run without loading filesystem settings.

  • Below query_agent, add the request handler, main loop, and test CLI flags to complete the script:
async def handle_request(tts, request):
    """Handle a single request from the Swift app."""
    user_message = request.get("message", "")
    system_prompt = request.get("system_prompt", "You are a helpful assistant.")
    voice = request.get("voice", "af_heart")
    speed = request.get("speed", 1.0)

    if not user_message:
        return {"error": "empty message"}

    try:
        response_text = await query_agent(user_message, system_prompt)

        if not response_text:
            return {"error": "empty response from Claude"}

        audio_path = generate_speech(tts, response_text, voice=voice, speed=speed)

        return {
            "text": response_text,
            "audio_path": audio_path,
        }
    except Exception as e:
        return {"error": str(e)}


async def main():
    """Main loop: read JSON commands from stdin, process, return results."""
    print("LOADING_MODELS", flush=True)
    tts = setup_tts()
    print("READY", flush=True)

    loop = asyncio.get_event_loop()

    for line in sys.stdin:
        line = line.strip()
        if not line:
            continue

        try:
            request = json.loads(line)
            result = await handle_request(tts, request)
            print(json.dumps(result), flush=True)
        except json.JSONDecodeError:
            print(json.dumps({"error": "invalid JSON"}), flush=True)
        except Exception as e:
            print(json.dumps({"error": str(e)}), flush=True)


if __name__ == "__main__":
    if "--test" in sys.argv:
        idx = sys.argv.index("--test")
        test_msg = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "Hello!"
        result = asyncio.run(query_agent(test_msg, "You are Spider-Man. Keep it to one sentence."))
        print(f"Response: {result}")
    elif "--test-tts" in sys.argv:
        idx = sys.argv.index("--test-tts")
        test_text = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "Hello, I am your character agent."
        tts = setup_tts()
        path = generate_speech(tts, test_text)
        print(f"Generated audio: {path}")
        tts.speak(test_text, voice="af_heart")
    else:
        asyncio.run(main())

What does this code do?

The handle_request() function orchestrates a single interaction: it extracts the message and character settings from the JSON request, queries Claude, generates speech, and returns everything as a JSON response.

The main() function prints LOADING_MODELS and READY as startup signals that the Swift app will listen for. Then it enters a loop reading JSON commands from stdin.

The __main__ block adds two test modes: --test for testing the Agent SDK and --test-tts for testing speech generation. These let you verify each piece independently.

  • Save agent_bridge.py.
  • Verify the Agent SDK connection works by running this test in your terminal:
cd venv && python agent_bridge.py --test "Say hello as Spider-Man"

You should see a one-sentence response from Claude in Spider-Man's voice. This confirms your Claude Code subscription is working through the Agent SDK.

Not getting a response?

Make sure you are authenticated with Claude Code. Run claude in your terminal to verify you can start an interactive session. If you see a login prompt, follow the authentication steps before retrying the test.

Also confirm you activated the virtual environment correctly. The command above uses cd venv && to run from the venv directory where the packages are installed.

Help me debug my agent_bridge.py test

✔️ Awesome, I've got everything!

Great. Double check you have saved agent_bridge.py in the project root.

ⓧ I'd like to double check the full code

#!/usr/bin/env python3
"""
Agent Bridge for Character AI Desktop Agent.
Uses the Claude Agent SDK to query Claude with character personalities,
and Kokoro-MLX for text-to-speech generation.

Runs as a long-lived subprocess. Accepts JSON commands on stdin,
returns JSON responses on stdout.
"""

import asyncio
import sys
import os
import json
import tempfile
from pathlib import Path

os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"


def setup_tts():
    """Load Kokoro-MLX model once at startup."""
    from kokoro_mlx import KokoroTTS
    tts = KokoroTTS.from_pretrained("mlx-community/Kokoro-82M-bf16")
    return tts


def generate_speech(tts, text, voice="af_heart", speed=1.0):
    """Generate speech and return path to WAV file."""
    import soundfile as sf

    result = tts.generate(text, voice=voice, speed=speed, sample_rate=24000)

    temp_dir = Path(tempfile.gettempdir()) / "character_agent_tts"
    temp_dir.mkdir(exist_ok=True)
    output_path = temp_dir / "response.wav"

    sf.write(str(output_path), result.audio, result.sample_rate)
    return str(output_path)


async def query_agent(user_message, system_prompt):
    """Query Claude via the Agent SDK with a character system prompt."""
    from claude_agent_sdk import query, ClaudeAgentOptions

    full_response = ""

    async for message in query(
        prompt=user_message,
        options=ClaudeAgentOptions(
            system_prompt=system_prompt,
            allowed_tools=[],
            max_turns=1,
            setting_sources=[],
        ),
    ):
        if hasattr(message, "content"):
            for block in message.content:
                if hasattr(block, "text"):
                    full_response += block.text

    return full_response


async def handle_request(tts, request):
    """Handle a single request from the Swift app."""
    user_message = request.get("message", "")
    system_prompt = request.get("system_prompt", "You are a helpful assistant.")
    voice = request.get("voice", "af_heart")
    speed = request.get("speed", 1.0)

    if not user_message:
        return {"error": "empty message"}

    try:
        response_text = await query_agent(user_message, system_prompt)

        if not response_text:
            return {"error": "empty response from Claude"}

        audio_path = generate_speech(tts, response_text, voice=voice, speed=speed)

        return {
            "text": response_text,
            "audio_path": audio_path,
        }
    except Exception as e:
        return {"error": str(e)}


async def main():
    """Main loop: read JSON commands from stdin, process, return results."""
    print("LOADING_MODELS", flush=True)
    tts = setup_tts()
    print("READY", flush=True)

    loop = asyncio.get_event_loop()

    for line in sys.stdin:
        line = line.strip()
        if not line:
            continue

        try:
            request = json.loads(line)
            result = await handle_request(tts, request)
            print(json.dumps(result), flush=True)
        except json.JSONDecodeError:
            print(json.dumps({"error": "invalid JSON"}), flush=True)
        except Exception as e:
            print(json.dumps({"error": str(e)}), flush=True)


if __name__ == "__main__":
    if "--test" in sys.argv:
        idx = sys.argv.index("--test")
        test_msg = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "Hello!"
        result = asyncio.run(query_agent(test_msg, "You are Spider-Man. Keep it to one sentence."))
        print(f"Response: {result}")
    elif "--test-tts" in sys.argv:
        idx = sys.argv.index("--test-tts")
        test_text = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "Hello, I am your character agent."
        tts = setup_tts()
        path = generate_speech(tts, test_text)
        print(f"Generated audio: {path}")
        tts.speak(test_text, voice="af_heart")
    else:
        asyncio.run(main())
Use Claude Code to create AgentBridge.swift

The Swift app needs a way to talk to your Python bridge. You'll use Claude Code to generate a Swift class that manages the Python subprocess, sends JSON commands, and reads responses.

  • Open a new terminal tab in your project root directory.
  • Start Claude Code by running:
claude
  • Once Claude Code loads, paste this prompt:
Create a new file AgentBridge.swift in the Xcode project. This class should:

1. Manage agent_bridge.py as a long-running subprocess using Foundation's Process class
2. On start(), launch the Python script at venv/bin/python3 with agent_bridge.py as the argument
3. Wait for LOADING_MODELS then READY on stdout before marking isReady = true
4. Provide sendMessage(message:systemPrompt:voice:speed:) that writes JSON to stdin and reads the JSON response from stdout
5. When the response includes an audio_path, play it via AVAudioPlayer
6. Include stopAudio() and stop() methods
7. Define an AgentBridgeError enum with cases: notReady, startupFailed, invalidResponse, agentError(String), readError
8. Use withCheckedThrowingContinuation for async reading from the pipe
9. Derive pythonPath and scriptPath from Bundle.main.bundlePath dropping the last 3 path components to find the project root

Also update CompanionManager.swift to instantiate AgentBridge, call start() on app launch, and use sendMessage() when the user finishes speaking to display text in the overlay and play audio.

Why prompt Claude Code instead of writing it manually?

Claude Code reads your CLAUDE.md file and understands the project architecture. It will generate code that fits the existing patterns in CompanionManager.swift and wire everything together correctly.

This also means the CompanionManager updates happen automatically. Claude Code sees how the state machine works and integrates the AgentBridge calls at the right points in the pipeline.

  • Review the files Claude Code creates. Allow it to write AgentBridge.swift and modify CompanionManager.swift.
  • Verify AgentBridge.swift includes the start() method that waits for READY and the sendMessage() method that writes JSON to stdin.
  • Exit Claude Code by pressing Ctrl+D or typing /exit.

Check the generated AgentBridge.swift

The generated file should import Foundation and AVFoundation. It should have an AgentBridgeError enum with five cases. The init() should derive pythonPath and scriptPath from the bundle path. If any of these are missing, prompt Claude Code again with the specific fix needed.

✔️ Awesome, I've got everything!

Great. Make sure you have saved all changes and that AgentBridge.swift appears in the Xcode project navigator.

ⓧ I'd like to double check the full code

Your AgentBridge.swift should look like this:

import Foundation
import AVFoundation

/// Manages the Python agent_bridge.py subprocess that runs the Claude Agent SDK
/// and Kokoro-MLX for AI responses and text-to-speech.
class AgentBridge {

    private var process: Process?
    private var stdinPipe: Pipe?
    private var stdoutPipe: Pipe?
    private var isReady = false
    private var audioPlayer: AVAudioPlayer?
    private let pythonPath: String
    private let scriptPath: String

    var isPlaying: Bool {
        return audioPlayer?.isPlaying ?? false
    }

    init() {
        let projectRoot = Bundle.main.bundlePath
            .components(separatedBy: "/")
            .dropLast(3)
            .joined(separator: "/")

        self.pythonPath = "\(projectRoot)/venv/bin/python3"
        self.scriptPath = "\(projectRoot)/agent_bridge.py"
    }

    func start() async throws {
        let process = Process()
        process.executableURL = URL(fileURLWithPath: pythonPath)
        process.arguments = [scriptPath]

        let stdinPipe = Pipe()
        let stdoutPipe = Pipe()
        process.standardInput = stdinPipe
        process.standardOutput = stdoutPipe
        process.standardError = FileHandle.nullDevice

        self.process = process
        self.stdinPipe = stdinPipe
        self.stdoutPipe = stdoutPipe

        try process.run()

        let firstLine = try await readLine(from: stdoutPipe)
        if firstLine == "LOADING_MODELS" {
            let secondLine = try await readLine(from: stdoutPipe)
            guard secondLine == "READY" else {
                throw AgentBridgeError.startupFailed
            }
        }
        isReady = true
    }

    func sendMessage(message: String, systemPrompt: String, voice: String, speed: Double) async throws -> String {
        guard isReady, let stdinPipe = stdinPipe, let stdoutPipe = stdoutPipe else {
            throw AgentBridgeError.notReady
        }

        let command: [String: Any] = [
            "message": message,
            "system_prompt": systemPrompt,
            "voice": voice,
            "speed": speed
        ]

        let jsonData = try JSONSerialization.data(withJSONObject: command)
        var jsonString = String(data: jsonData, encoding: .utf8)!
        jsonString += "\n"

        stdinPipe.fileHandleForWriting.write(jsonString.data(using: .utf8)!)

        let responseLine = try await readLine(from: stdoutPipe)
        guard let responseData = responseLine.data(using: .utf8),
              let response = try? JSONSerialization.jsonObject(with: responseData) as? [String: Any] else {
            throw AgentBridgeError.invalidResponse
        }

        if let error = response["error"] as? String {
            throw AgentBridgeError.agentError(error)
        }

        guard let text = response["text"] as? String else {
            throw AgentBridgeError.invalidResponse
        }

        if let audioPath = response["audio_path"] as? String {
            let audioURL = URL(fileURLWithPath: audioPath)
            audioPlayer = try AVAudioPlayer(contentsOf: audioURL)
            audioPlayer?.play()
        }

        return text
    }

    func stopAudio() {
        audioPlayer?.stop()
    }

    func stop() {
        audioPlayer?.stop()
        process?.terminate()
        process = nil
        isReady = false
    }

    private func readLine(from pipe: Pipe) async throws -> String {
        return try await withCheckedThrowingContinuation { continuation in
            DispatchQueue.global().async {
                let data = pipe.fileHandleForReading.availableData
                if let line = String(data: data, encoding: .utf8)?.trimmingCharacters(in: .whitespacesAndNewlines) {
                    continuation.resume(returning: line)
                } else {
                    continuation.resume(throwing: AgentBridgeError.readError)
                }
            }
        }
    }
}

enum AgentBridgeError: Error, LocalizedError {
    case notReady
    case startupFailed
    case invalidResponse
    case agentError(String)
    case readError

    var errorDescription: String? {
        switch self {
        case .notReady: return "Agent bridge not ready"
        case .startupFailed: return "Agent bridge failed to start"
        case .invalidResponse: return "Invalid response from agent bridge"
        case .agentError(let msg): return "Agent error: \(msg)"
        case .readError: return "Failed to read from agent bridge"
        }
    }
}
Test the integration

All the pieces are connected. Time to verify the full pipeline: speech recognition captures your voice, the Agent SDK generates a response, and the text appears in the overlay.

  • Switch back to Xcode.
  • Press Cmd+B to build the project.
  • Press Cmd+R to run the app.
  • Hold ctrl+option to activate push-to-talk and ask a simple question like "What's the weather like today?"
  • Release the keys and watch for the response text appearing in the overlay window.

What should I see?

The first time you run the app after building, it takes a few seconds to load the Kokoro-MLX model (the LOADING_MODELS phase). After that, responses should appear within 1-3 seconds of releasing the push-to-talk keys.

You should see text streaming into the transparent overlay near your cursor. You should also hear the response spoken aloud through your speakers.

App builds but nothing happens when you speak?

Check that you granted microphone permission. macOS will prompt you on first launch. If you dismissed it, go to System Settings then Privacy & Security then Microphone and enable access for the app.

If text appears but you hear no audio, verify that agent_bridge.py is in the project root (not inside a subdirectory). The scriptPath in AgentBridge derives from the bundle path and expects the file at the root level.

Help me debug why my character agent is not responding to push-to-talk

Your agent now has a brain and a voice. Next up, you'll give it personality by creating character configurations that make it respond as Spider-Man, Iron Man, or any character you choose.

Integrate Kokoro-MLX for Natural Voice Output

Your agent can think and display text, but it still needs a voice. The agent_bridge.py script already includes TTS code that was scaffolded alongside the Agent SDK logic, and AgentBridge.swift already plays audio when it receives a WAV path.

In this step, you will verify each piece of the voice pipeline works independently, then confirm the full loop: speak a question, get an AI response, and hear it spoken back in a natural voice.

In this step, get ready to:
  • Verify Kokoro-MLX generates natural speech from text.
  • Confirm the Swift app plays audio and stops cleanly on new interactions.
  • Test the full voice pipeline end-to-end.
Verify TTS generation in agent_bridge.py

Kokoro-MLX is a text-to-speech engine built specifically for Apple Silicon. It uses Apple's MLX framework to run a neural voice model directly on your GPU with no cloud calls.

Your agent_bridge.py already has two functions that handle TTS. Open the file and locate these functions near the top:

def setup_tts():
    """Load Kokoro-MLX model once at startup."""
    from kokoro_mlx import KokoroTTS
    tts = KokoroTTS.from_pretrained("mlx-community/Kokoro-82M-bf16")
    return tts


def generate_speech(tts, text, voice="af_heart", speed=1.0):
    """Generate speech and return path to WAV file."""
    import soundfile as sf

    result = tts.generate(text, voice=voice, speed=speed, sample_rate=24000)

    temp_dir = Path(tempfile.gettempdir()) / "character_agent_tts"
    temp_dir.mkdir(exist_ok=True)
    output_path = temp_dir / "response.wav"

    sf.write(str(output_path), result.audio, result.sample_rate)
    return str(output_path)

What does this code do?

setup_tts() loads the Kokoro-82M model from HuggingFace cache into memory once when the bridge starts. This avoids reloading the 350MB model on every request.

generate_speech() takes the Agent SDK's text response, runs it through the model, and saves the resulting audio as a WAV file at /tmp/character_agent_tts/response.wav. The voice parameter lets each character use a different Kokoro voice ID.

Now look at the main() function to see how TTS fits into the startup sequence:

async def main():
    """Main loop: read JSON commands from stdin, process, return results."""
    print("LOADING_MODELS", flush=True)
    tts = setup_tts()
    print("READY", flush=True)

Why load once at startup?

The bridge prints LOADING_MODELS to signal the Swift app that model loading is in progress, then READY once complete. The tts object stays in memory for the lifetime of the subprocess. Every subsequent request reuses it, keeping response times fast.

  • Test the TTS in isolation by running this command in your Terminal:
cd venv && python agent_bridge.py --test-tts "Hello world"
  • You should hear a natural-sounding voice say "Hello world" through your speakers.

What does the test command do?

The --test-tts flag runs a standalone test path in agent_bridge.py. It loads the model, generates a WAV file, prints its path, then plays the audio using Kokoro's built-in tts.speak() method.

No audio playing?

  • Check your Mac's volume is turned up and not muted.
  • Make sure your venv is activated. If you see an import error, run source venv/bin/activate first, then try the command again.
  • If you see a model download starting, wait for it to finish. The first run pulls ~350MB from HuggingFace.

Help me debug why Kokoro-MLX test-tts is not producing audio on my Mac

Verify audio playback in the Swift app

The Python side generates audio. Now you need to confirm the Swift side picks it up and plays it. Switch back to AgentBridge.swift in Xcode and find the audio playback logic inside sendMessage():

        if let audioPath = response["audio_path"] as? String {
            let audioURL = URL(fileURLWithPath: audioPath)
            audioPlayer = try AVAudioPlayer(contentsOf: audioURL)
            audioPlayer?.play()
        }

What does this code do?

When the bridge returns a JSON response containing audio_path, the Swift app creates an AVAudioPlayer and immediately starts playback. The text response displays in the overlay at the same time, so the learner sees the words while hearing them spoken.

Now find the stopAudio() method just below sendMessage():

    func stopAudio() {
        audioPlayer?.stop()
    }

Why stop audio on new interactions?

CompanionManager calls stopAudio() when the learner presses the push-to-talk hotkey again. This prevents the previous response from talking over the new one. Without it, you would hear two overlapping voices.

  • Build the project by pressing Cmd+B in Xcode.
  • Confirm the build succeeds with no errors in the status bar.

Build failing on AVFoundation?

  • Verify import AVFoundation is at the top of AgentBridge.swift.
  • Make sure the deployment target is macOS 14.2 or higher in your project settings.

Help me fix AVFoundation build errors in my Xcode project

Test end-to-end voice output

The TTS generates audio. The Swift app plays it. Time to confirm the full pipeline works together.

  • Run the app by pressing Cmd+R in Xcode.
  • Wait for the menu bar icon to appear. The first launch takes a few seconds while Kokoro-MLX loads into memory.
  • Hold ctrl+option to activate push-to-talk.
  • Say a short question like "What is the capital of France?" and release the keys.
  • Watch the overlay. Text should stream in, followed by spoken audio through your speakers.

What should I see and hear?

You should see the text response appear in the overlay near your cursor within 1-3 seconds. A moment later, you hear the response spoken in a natural voice. The voice should sound smooth and human-like, noticeably better than the robotic macOS built-in speech.

  • While the voice is still speaking, press ctrl+option again to start a new interaction.
  • Confirm the previous audio stops immediately when you start speaking again.

Text appears but no voice?

  • Check that the Python venv path is correct. In AgentBridge.swift, the pythonPath must point to your actual venv location.
  • Open Activity Monitor and search for python3. If no Python process is running, the bridge failed to start.
  • Try the standalone test again (--test-tts) to confirm TTS works outside the app. If it works standalone but not in the app, the subprocess communication is the issue.

Help me debug why my agent bridge is not producing voice audio in the desktop app

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

Here is the full agent_bridge.py:

#!/usr/bin/env python3
"""
Agent Bridge for Character AI Desktop Agent.
Uses the Claude Agent SDK to query Claude with character personalities,
and Kokoro-MLX for text-to-speech generation.

Runs as a long-lived subprocess. Accepts JSON commands on stdin,
returns JSON responses on stdout.
"""

import asyncio
import sys
import os
import json
import tempfile
from pathlib import Path

os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"


def setup_tts():
    """Load Kokoro-MLX model once at startup."""
    from kokoro_mlx import KokoroTTS
    tts = KokoroTTS.from_pretrained("mlx-community/Kokoro-82M-bf16")
    return tts


def generate_speech(tts, text, voice="af_heart", speed=1.0):
    """Generate speech and return path to WAV file."""
    import soundfile as sf

    result = tts.generate(text, voice=voice, speed=speed, sample_rate=24000)

    temp_dir = Path(tempfile.gettempdir()) / "character_agent_tts"
    temp_dir.mkdir(exist_ok=True)
    output_path = temp_dir / "response.wav"

    sf.write(str(output_path), result.audio, result.sample_rate)
    return str(output_path)


async def query_agent(user_message, system_prompt):
    """Query Claude via the Agent SDK with a character system prompt."""
    from claude_agent_sdk import query, ClaudeAgentOptions

    full_response = ""

    async for message in query(
        prompt=user_message,
        options=ClaudeAgentOptions(
            system_prompt=system_prompt,
            allowed_tools=[],
            max_turns=1,
            setting_sources=[],
        ),
    ):
        if hasattr(message, "content"):
            for block in message.content:
                if hasattr(block, "text"):
                    full_response += block.text

    return full_response


async def handle_request(tts, request):
    """Handle a single request from the Swift app."""
    user_message = request.get("message", "")
    system_prompt = request.get("system_prompt", "You are a helpful assistant.")
    voice = request.get("voice", "af_heart")
    speed = request.get("speed", 1.0)

    if not user_message:
        return {"error": "empty message"}

    try:
        response_text = await query_agent(user_message, system_prompt)

        if not response_text:
            return {"error": "empty response from Claude"}

        audio_path = generate_speech(tts, response_text, voice=voice, speed=speed)

        return {
            "text": response_text,
            "audio_path": audio_path,
        }
    except Exception as e:
        return {"error": str(e)}


async def main():
    """Main loop: read JSON commands from stdin, process, return results."""
    print("LOADING_MODELS", flush=True)
    tts = setup_tts()
    print("READY", flush=True)

    loop = asyncio.get_event_loop()

    for line in sys.stdin:
        line = line.strip()
        if not line:
            continue

        try:
            request = json.loads(line)
            result = await handle_request(tts, request)
            print(json.dumps(result), flush=True)
        except json.JSONDecodeError:
            print(json.dumps({"error": "invalid JSON"}), flush=True)
        except Exception as e:
            print(json.dumps({"error": str(e)}), flush=True)


if __name__ == "__main__":
    if "--test" in sys.argv:
        idx = sys.argv.index("--test")
        test_msg = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "Hello!"
        result = asyncio.run(query_agent(test_msg, "You are Spider-Man. Keep it to one sentence."))
        print(f"Response: {result}")
    elif "--test-tts" in sys.argv:
        idx = sys.argv.index("--test-tts")
        test_text = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "Hello, I am your character agent."
        tts = setup_tts()
        path = generate_speech(tts, test_text)
        print(f"Generated audio: {path}")
        tts.speak(test_text, voice="af_heart")
    else:
        asyncio.run(main())

And here is the full AgentBridge.swift:

import Foundation
import AVFoundation

/// Manages the Python agent_bridge.py subprocess that runs the Claude Agent SDK
/// and Kokoro-MLX for AI responses and text-to-speech.
class AgentBridge {

    private var process: Process?
    private var stdinPipe: Pipe?
    private var stdoutPipe: Pipe?
    private var isReady = false
    private var audioPlayer: AVAudioPlayer?
    private let pythonPath: String
    private let scriptPath: String

    var isPlaying: Bool {
        return audioPlayer?.isPlaying ?? false
    }

    init() {
        let projectRoot = Bundle.main.bundlePath
            .components(separatedBy: "/")
            .dropLast(3)
            .joined(separator: "/")

        self.pythonPath = "\(projectRoot)/venv/bin/python3"
        self.scriptPath = "\(projectRoot)/agent_bridge.py"
    }

    func start() async throws {
        let process = Process()
        process.executableURL = URL(fileURLWithPath: pythonPath)
        process.arguments = [scriptPath]

        let stdinPipe = Pipe()
        let stdoutPipe = Pipe()
        process.standardInput = stdinPipe
        process.standardOutput = stdoutPipe
        process.standardError = FileHandle.nullDevice

        self.process = process
        self.stdinPipe = stdinPipe
        self.stdoutPipe = stdoutPipe

        try process.run()

        let firstLine = try await readLine(from: stdoutPipe)
        if firstLine == "LOADING_MODELS" {
            let secondLine = try await readLine(from: stdoutPipe)
            guard secondLine == "READY" else {
                throw AgentBridgeError.startupFailed
            }
        }
        isReady = true
    }

    func sendMessage(message: String, systemPrompt: String, voice: String, speed: Double) async throws -> String {
        guard isReady, let stdinPipe = stdinPipe, let stdoutPipe = stdoutPipe else {
            throw AgentBridgeError.notReady
        }

        let command: [String: Any] = [
            "message": message,
            "system_prompt": systemPrompt,
            "voice": voice,
            "speed": speed
        ]

        let jsonData = try JSONSerialization.data(withJSONObject: command)
        var jsonString = String(data: jsonData, encoding: .utf8)!
        jsonString += "\n"

        stdinPipe.fileHandleForWriting.write(jsonString.data(using: .utf8)!)

        let responseLine = try await readLine(from: stdoutPipe)
        guard let responseData = responseLine.data(using: .utf8),
              let response = try? JSONSerialization.jsonObject(with: responseData) as? [String: Any] else {
            throw AgentBridgeError.invalidResponse
        }

        if let error = response["error"] as? String {
            throw AgentBridgeError.agentError(error)
        }

        guard let text = response["text"] as? String else {
            throw AgentBridgeError.invalidResponse
        }

        if let audioPath = response["audio_path"] as? String {
            let audioURL = URL(fileURLWithPath: audioPath)
            audioPlayer = try AVAudioPlayer(contentsOf: audioURL)
            audioPlayer?.play()
        }

        return text
    }

    func stopAudio() {
        audioPlayer?.stop()
    }

    func stop() {
        audioPlayer?.stop()
        process?.terminate()
        process = nil
        isReady = false
    }

    private func readLine(from pipe: Pipe) async throws -> String {
        return try await withCheckedThrowingContinuation { continuation in
            DispatchQueue.global().async {
                let data = pipe.fileHandleForReading.availableData
                if let line = String(data: data, encoding: .utf8)?.trimmingCharacters(in: .whitespacesAndNewlines) {
                    continuation.resume(returning: line)
                } else {
                    continuation.resume(throwing: AgentBridgeError.readError)
                }
            }
        }
    }
}

enum AgentBridgeError: Error, LocalizedError {
    case notReady
    case startupFailed
    case invalidResponse
    case agentError(String)
    case readError

    var errorDescription: String? {
        switch self {
        case .notReady: return "Agent bridge not ready"
        case .startupFailed: return "Agent bridge failed to start"
        case .invalidResponse: return "Invalid response from agent bridge"
        case .agentError(let msg): return "Agent error: \(msg)"
        case .readError: return "Failed to read from agent bridge"
        }
    }
}

Your desktop agent now has a full voice. Next up, you will give it a personality by defining character configurations that control both what it says and how it sounds.

Add Character Personalities

Your agent can hear you, think with Claude, and speak back in a natural voice. The full pipeline works end to end.

Right now, every response sounds the same because the agent uses a generic system prompt and a single default voice. In this step, you will create distinct character personalities that change both how Claude responds and which Kokoro-MLX voice it uses.

In this step, get ready to:
  • Create a character configuration file with personality definitions for Spider-Man, Iron Man, and a Mentor.
  • Wire character loading into the Swift app so the active personality drives both the system prompt and voice.
  • Test each character to confirm distinct responses and voices.
Create the character configuration file

Each character needs a personality definition that tells Claude how to behave and tells Kokoro-MLX which voice to use. Storing these in a JSON file keeps them easy to edit without recompiling the app.

  • Create a new file called characters.json in the root of your clicky project folder (the same level as agent_bridge.py).
  • Paste the following character definitions into the file:
{
  "characters": [
    {
      "id": "spiderman",
      "name": "Spider-Man",
      "voice": "am_adam",
      "speed": 1.1,
      "system_prompt": "You are Spider-Man (Peter Parker). You are helping the user with their computer tasks and coding questions. Respond in character: use NYC slang, make web-slinging references, crack jokes under pressure, and call the user 'buddy' or 'pal'. Keep responses concise (2-3 sentences max). When explaining code, use analogies about swinging between buildings or spider-sense tingling when something is wrong. Stay helpful and accurate despite the playful tone."
    },
    {
      "id": "ironman",
      "name": "Iron Man",
      "voice": "am_michael",
      "speed": 1.0,
      "system_prompt": "You are Tony Stark (Iron Man). You are helping the user with their computer tasks and coding questions. Respond in character: be confident, sarcastic, and technical. Reference your suits, JARVIS, arc reactors, and engineering brilliance. Call the user 'kid' or by an improvised nickname. Keep responses concise (2-3 sentences max). When explaining code, compare it to building suit components or debugging JARVIS. You are genuinely helpful but cannot resist showing off your genius."
    },
    {
      "id": "mentor",
      "name": "The Mentor",
      "voice": "bf_emma",
      "speed": 0.95,
      "system_prompt": "You are a patient, wise coding mentor. You explain things clearly and encouragingly. Use simple analogies from everyday life. When the user makes a mistake, reframe it as a learning opportunity. Keep responses concise (2-3 sentences max). Ask a follow-up question when appropriate to guide the user toward understanding rather than just giving answers. Your tone is warm, calm, and supportive."
    }
  ],
  "default_character": "spiderman"
}

What does this configuration do?

Each character object has five fields that control the agent's behavior:

  • id is the internal key the app uses to track which character is active.
  • voice maps to a Kokoro-MLX voice ID. am_adam is a friendly American male, am_michael is a deeper American male, and bf_emma is a British female voice.
  • speed controls speech rate. Spider-Man talks slightly faster (1.1x) while The Mentor speaks more slowly (0.95x) for a calmer feel.
  • system_prompt is the full personality instruction sent to Claude with every request. It defines tone, catchphrases, and response length.
  • Save characters.json.
  • Confirm the file exists by running this in your terminal from the project root:
cat characters.json

You should see the full JSON output with all three character definitions printed to your terminal.

Seeing a "No such file or directory" error?

Make sure you created characters.json at the root of the clicky project folder, not inside a subfolder. It should sit alongside agent_bridge.py and the .xcodeproj file.

help me find where to put characters.json in my clicky project

Prompt Claude Code to load and apply character personalities

The Swift app needs a way to read characters.json, decode it into Swift types, and pass the active character's personality to the Agent SDK on every request.

  • Switch back to your terminal in the clicky project directory.
  • Launch Claude Code by running:
claude
  • Give Claude Code the following prompt to create the character loading system:
Create a new file CharacterPicker.swift that:
1. Defines a CharacterDefinition struct (Codable, Identifiable) with fields: id (String), name (String), voice (String), speed (Double), system_prompt (String)
2. Defines a CharactersConfig struct (Codable) with fields: characters ([CharacterDefinition]) and default_character (String)
3. Has a loadCharacters() function that reads characters.json from the app bundle first, falling back to the project root directory
4. Includes a CharacterPicker SwiftUI view that shows a Menu dropdown with all characters, a checkmark next to the selected one, and calls an onSelect closure when the user picks a character

Then update CompanionManager.swift to:
1. Load characters.json at startup using loadCharacters()
2. Default to the "spiderman" character
3. Pass the active character's system_prompt and voice to AgentBridge.sendMessage() on every interaction
4. Use the active character's speed value as well

Why a separate CharacterPicker.swift file?

Keeping the data model (CharacterDefinition), the config loader (loadCharacters()), and the UI (CharacterPicker view) in one file makes it easy to add or remove the character system without touching core app logic.

The loadCharacters() function tries the app bundle first, then falls back to the project root. This means the file works both when running from Xcode (bundle) and during development (project root).

  • Review the changes Claude Code proposes and accept them.
  • Verify that Claude Code created CharacterPicker.swift in the Xcode project.
  • Verify that CompanionManager.swift now references loadCharacters() and passes the character's system_prompt, voice, and speed to sendMessage().

Check the CLAUDE.md file

Your CLAUDE.md already lists characters.json under Key Files and mentions character system prompts in Coding Standards. Claude Code uses this context to make better decisions about where to put the code and how to structure it.

  • Build the project in Xcode by pressing Cmd+B.

The project should compile with zero errors.

Build failing after Claude Code's changes?

Common issues include a missing import SwiftUI at the top of CharacterPicker.swift, or the file not being added to the Xcode target. In Xcode, click on CharacterPicker.swift in the sidebar, then check the Target Membership panel on the right to confirm it is checked for your app target.

help me fix build errors after adding CharacterPicker.swift

\u2714\ufe0f Awesome, I've got everything!

Great. Double check you have saved all files before moving on.

\u24e7 I'd like to double check the full code

Here is the expected CharacterPicker.swift file. Claude Code may generate slightly different formatting, but the structure and types should match:

import SwiftUI

struct CharacterPicker: View {
    @Binding var selectedCharacterID: String
    let characters: [CharacterDefinition]
    let onSelect: (CharacterDefinition) -> Void

    var body: some View {
        Menu {
            ForEach(characters, id: \.id) { character in
                Button(action: {
                    selectedCharacterID = character.id
                    onSelect(character)
                }) {
                    HStack {
                        Text(character.name)
                        if character.id == selectedCharacterID {
                            Image(systemName: "checkmark")
                        }
                    }
                }
            }
        } label: {
            HStack(spacing: 4) {
                Image(systemName: "person.circle.fill")
                Text(characters.first(where: { $0.id == selectedCharacterID })?.name ?? "Character")
                    .font(.caption)
            }
            .padding(.horizontal, 8)
            .padding(.vertical, 4)
            .background(Color.blue.opacity(0.2))
            .cornerRadius(8)
        }
    }
}

struct CharacterDefinition: Codable, Identifiable {
    let id: String
    let name: String
    let voice: String
    let speed: Double
    let system_prompt: String
}

struct CharactersConfig: Codable {
    let characters: [CharacterDefinition]
    let default_character: String
}

func loadCharacters() -> CharactersConfig? {
    guard let url = Bundle.main.url(forResource: "characters", withExtension: "json") ??
          findCharactersJSON() else {
        return nil
    }

    do {
        let data = try Data(contentsOf: url)
        return try JSONDecoder().decode(CharactersConfig.self, from: data)
    } catch {
        print("Failed to load characters: \(error)")
        return nil
    }
}

private func findCharactersJSON() -> URL? {
    let projectRoot = Bundle.main.bundlePath
        .components(separatedBy: "/")
        .dropLast(3)
        .joined(separator: "/")
    let path = "\(projectRoot)/characters.json"
    if FileManager.default.fileExists(atPath: path) {
        return URL(fileURLWithPath: path)
    }
    return nil
}
Test each character personality

With the configuration loaded and wired in, each character should produce a distinctly different response style and voice.

  • Run the app in Xcode by pressing Cmd+R.
  • Wait for the menu bar icon to appear (the Kokoro-MLX model takes a few seconds to load).
  • Hold ctrl+option to activate push-to-talk.
  • Ask a simple question like "What does a for loop do?" and release the keys.

The default character is Spider-Man. You should hear a response with NYC slang, web-slinging references, and a slightly faster speaking speed.

How do I test the other characters?

Right now the app defaults to Spider-Man with no UI to switch. To test Iron Man or The Mentor, temporarily change the default_character value in characters.json to ironman or mentor, save the file, and rebuild with Cmd+R.

Spider-Man explains code with web-slinging metaphors and calls you "buddy". Iron Man is sarcastic, references JARVIS, and calls you "kid". The Mentor speaks slowly and asks follow-up questions to guide your understanding.

  • After testing, set default_character back to spiderman in characters.json and save.

Character responding without personality?

If the response sounds generic (no slang, no character traits), check that CompanionManager.swift is actually passing the character's system_prompt to sendMessage(). A common issue is passing an empty string or the old default prompt instead of the loaded character's prompt.

Also confirm that loadCharacters() is not returning nil. Add a print() statement after the load call to verify the characters array has 3 items.

help me debug why my character personality isn't working

Secret mission

Character Selection Dropdown

Your agent works great, but switching characters means restarting the app. What if you could swap between Spider-Man, Iron Man, and The Mentor on the fly from the menu bar?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on your Mac, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your resources as-is, or delete them to reclaim disk space.

Resources you used:

  • The clicky project folder (cloned repo with all your modifications).
  • Python virtual environment (venv folder inside the project).
  • Kokoro-MLX model (~350MB downloaded to your HuggingFace cache).

Keep everything running

No action needed. Choose this if you are still actively using your character agent or want to keep tweaking personalities.

  • The app lives on your Mac and runs locally. It costs nothing to keep.
  • The Kokoro-MLX model stays in your HuggingFace cache (~350MB on disk).
  • Your Python virtual environment stays in the project folder.

Pause - I'll come back to this later

Quit the app and deactivate the virtual environment to free up memory. Your files stay on disk so you can pick up where you left off.

  • Quit the Clicky app if it is running (click the menu bar icon, then close the app).
  • If your terminal has the virtual environment active, deactivate it by running:
deactivate

Everything stays on disk. When you are ready to resume, activate the venv and rebuild in Xcode.

Delete - I don't want to use this again

Remove all project files and the downloaded model to reclaim approximately 400MB of disk space.

  • Quit the Clicky app if it is running.
  • Delete the entire project folder (this removes the venv, source code, and build artifacts) by running:
rm -rf ~/Desktop/clicky
  • Delete the Kokoro-MLX model from your HuggingFace cache by running:
rm -rf ~/.cache/huggingface/hub/models--mlx-community--Kokoro-82M-bf16
  • Delete the temporary TTS audio files by running:
rm -rf /tmp/character_agent_tts

Your Mac is now back to its original state with no project remnants.

That's a Wrap!

That's a Wrap!

Nice work. You just built a fully local character AI desktop agent powered by the Claude Agent SDK and your existing Claude Code subscription.

You've learned how to:

  • Build a character AI desktop agent by forking the open-source Clicky codebase and connecting the Claude Agent SDK as the AI brain using your existing subscription with zero extra API keys or costs.
  • Replace paid cloud dependencies (Cloudflare Worker, AssemblyAI, ElevenLabs) with fully local alternatives: Apple SFSpeechRecognizer for voice input and Kokoro-MLX for natural-sounding text-to-speech on Apple Silicon.
  • Create swappable character personalities with distinct voices, system prompts, and speech patterns that make your AI companion respond in character as Spider-Man, Iron Man, or a custom Mentor.
  • Secret Mission: Added a character selection dropdown in the menu bar for switching personalities on the fly without restarting the app.

Ready to quiz yourself?